DEEPER-GXX: DEEPENING ARBITRARY GNNS
Authors: Lecheng Zheng
Dongqi Fu
Ross Maciejewski
Jingrui He
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Neural Networks
Deep Learning

Abstract: Recently, motivated by real applications, a major research direction in graph neural networks (GNNs) is to explore deeper structures.
For instance, the graph connectivity is not always consistent with the label distribution (e.g., the closest neighbors of some nodes are not from the same category). In this case, GNNs need to stack more layers, in order to find the same categorical neighbors in a longer path for capturing the class-discriminative information. However, two major problems hinder the deeper GNNs to obtain satisfactory performance, i.e., vanishing gradient and over-smoothing. On one hand, stacking layers makes the neural network hard to train as the gradients of the first few layers vanish. Moreover, when simply addressing vanishing gradient in GNNs, we discover the shading neighbors effect (i.e., stacking layers inappropriately distorts the non-IID information of graphs and degrade the performance of GNNs). On the other hand, deeper GNNs aggregate much more information from common neighbors such that individual node representations share more overlapping features, which makes the final output representations not discriminative (i.e., overly smoothed). In this paper, for the first time, we address both problems to enable deeper GNNs, and propose Deeper-GXX, which consists of the Weight-Decaying Graph Residual Connection module (WDG-ResNet) and Topology-Guided Graph Contrastive Loss (TGCL). Extensive experiments on real-world data sets demonstrate that Deeper-GXX outperforms state-of-the-art deeper baselines.

Under review as a conference paper at ICLR 2023
DEEPER-GXX: DEEPENING ARBITRARY GNNs
Anonymous authors
Paper under double-blind review
ABSTRACT
Recently, motivated by real applications, a major research direction in graph neural networks (GNNs) is explore deeper structures. For instance, the graph connectivity is not always consistent with the label distribution (e.g., the closest neighbors of some nodes are not from the same category). this case, GNNs need to stack more layers, in order to find the same categorical neighbors in a longer path for capturing the disssiiiiiiiiitiieee information. However, two major problems hinder the deeper GNNs to obtain satisfactory performance, i.e., vanishing gradient and over-smoothing. On one hand, stacking layers makes the neural network hard to train the gradients of the first few layers vanish. Moreover, when simply addressing vamishing gradient in GNNs, we discover the shading neighbors effect (i.e., stacking layers inappropriately distorts the non-lID information of graphs and degrade the performance of GNNs). On the other hand, deeper GNNs aggregate much more information from common neighbors such that individual node representations share more overlapping features, which makes the final output representations not discriminative (i.e., overly smoothed). In this paper, for the first time, we address both problems to enable deeper GNNs, and propose Deeper- GXX, which consists of the Weight-Decaying Graph Residual Connection module (WDG-ResNe)) and Yopology-Guide experiments on real- world data sets demonstrate that Deeper-GiX outperforms state-of-tt -herttrttt art deeper baselines.
Graph Contrastive Loss (TGCL). Extensive
INTRODUCTION
Graph neural networks (GNNs) have been proven successful at modeling graph data by extracting node hidden representations that are effective for many downstream tasks. general, they are realized by the message passing schema and aggregate neighbor features to obtain node hidden representations (Kipf & Welling, 2017, Hamilton et al.) 2017a| Velickovic et al.], (2018 [Xu al.. 2019). Recently, the surge of big data makes graphs' structural and attribute information much more complex and uncertain, which urges the researchers to make GNNs deeper (i.e., stacking more graph neural layers), in order to capture more meaningful information for better performance. For example, in social media, people from different categories (c.g., occupation, interests, etc.) are often connected (e.g., become friends), and users' immediate neighbor information may not reflect their categorical information. Thus, deepening GNNs is necessary to identify the neighbors from the same category in a longer path (c.g., k-hop neighbors), and to aggregate their features obtain the daas-diisriminative node representations. To demonstrate the benefit of deeper GNNs, we conduct case study shown in Figure [See the detailed experimental setup in Appendix A.I] In Figure Ta we observe that the query node (the diamond in the black dashed circle) cannot rely on its closest labeled neighbor (the red star in the circle) correctly predict its label (the blue). Only by exploring longer paths consisting of more similar neighbors are we able to predict its label as blue. Figure Tb compares the classification accuracy of shallow GNNs and deeper GNNs. We can see that deeper GNNs significantly outperform shallow ones by more than 11,, due their abilities to explore longer paths on the graph. Similar observations of the benefits of deeper GNNs are also found in the missing feature scenario presented in Section 3.3
However, simply stacking layers of GNNs can be problematic, due to vanishing gradient and over- smoothing issues. On one hand, increasing the number of neural layers can induce the hard-to-trait model, where both the training error and test error are higher than shallow ones. This is mainly caused by the vanishing gradient issue (He al. 2016), where the gradient of the first few layers vanish

---

Under review as a conference paper at ICLR 2023
11.3%
0.8
11.6%
Qurry
0.4
0.2
0.0
SGC
Deeper-SGC
GCN
Deeper-GCN
Two groups nodes in the semi- supervisd setting. Stars are labeled, dots are unlaheled, and the diamond is the query node. Euclidean distance between two nodes indicates the edge connection.
(b) Comparison of node classification accuracy
between shallow and deeper GNN models using data on the left. The deeper GNNs are realized by our Deeper-GXX with corresponding backhones.
Figure 1: A toy example to demonstrate the benefit of deeper GNN models.
such that the training loss could not be successfully propagated through deeper models. ResNet (He al. 20166) has been proposed to address this issue. However, we discover that simply combining ResNet with GNNs still leads to the sub-optimal solution: as ResNet stacks layers, the importance of close neighbors' features gradually decreases during the GNN information aggregation process, and the faraway neighbor information becomes dominant. We call this effect as shading neighbors. On the other hand, GNN utilizes the message passing schema to aggregate neighbor features, order to get slass-iisrimiinative node representations. However, by stacking more layers, each node begins to share more and more overlapping neighbor information during the aggregation process and the node representations gradually become indistinguishable (Li al. 2018; Oono & Suzuki 2020). This has been referred to as the over-smoothing issue, and significantly affects the performance of downstream tasks such as node classification and link prediction.
In this paper, we study how to effectively stack GNN layers by addressing shading neighbors and over- smoothing at the same time. First, to address the shading neighbors caused by the direct application of ResNet on GNNs, we propose Weight-Decayin
Graph Residual Connection which learns the weight of each residual connection layer (instead of setting it in ResNet), and further introduces a decaying factor to refine the weight of each layer. Interestingly. we find that the hyperparameter À of the weight decaying factor actually controls the number of effective layers in deeper GNNs based on the input graph inherent property, which is verified in Appendix A.5 Second, for addressing smoothhioin we propose Topology-Guides Graph Contrastive Loss (TGCL)
in the contrastive learning manner (van den Oord et al.) (018). where the hidden representations of the positive pairs should be closer, and those of the negative pairs should be pushed apart. Through theoretical and empirical analysis, we find that TGCL. can be effectively and efficiently realized by only considering 1 1-hop neighbors as the positive pair and all the rest as negative pairs. Combining the proposed WDG-ResNet and TGCL, we propose end-to-end model called Deeper-GXX to help arbitrary GNNs go deeper. Our contributions can be summarized as follows.
We propose Weight-Decaying Graph Residual Connection (WDG-ResNet) to address the shading neighbors effect caused by vanilla ResNet dealing with the vanishing gradient GNNs. We propose Sapology-Guidee Graph Contrastive Loss (TGCL) to address the mmor-shoo problem by encoding the graph topological information to the discriminative node representations. We combine the proposed WDG-ResNet and TGCL into end-to-end model called Deeper-GXX, which is model- model-agnostic and can help arbitrary GNNs go deeper.
Extensive experiments show that Deeper-GXX outperforms state-of-the-ar deeper baselines.
PROPOSED METHOD
In this section, we begin with the overview of Deeper-GXX. Then, we provide the details of the proposed Weight-Decaying Graph Residual Connection (WDG-ResNet) and Topology- -uidddd Graph Contrastive Loss (TGCL), which address shading neighbors and over-smoothing problems, respectively. We formalize the graph embedding problem the context undirected graph g = (V, F,, X), where V consists of n vertices, E. consists of edges, X E R denotes the feature
2

---

Under review as a conference paper at ICLR 2023
Weight- Weight-Decayinng Decaylnng Graph Residual
Topology Guided Graph Contrastive Loss (TGCL)
Connection (WDG ResNet)
Repe
7033)
RELLS
X(2:)
GNN Hidden Layer #1
HC2.3
GNN Hidden Layer Mi-2
GNN Hiddon Laver #E-1
GNN Hidden Lover a!
X(6.))
X
H(G.3)
X(5)))
M/S.))
An Arbitrary GNN
Input Graph Features
Output Graph Representations
Figure 2: An arbitrary GNN with the proposed Deeper-GXX.
matrix and d is the feature dimension. We let A Rax= denote the adjacency matrix and denote A, E IR" as the adjacency vector for node 04. H; E IR" is the hidden representation vector of iz.
2.1 OVERVIEW OF DEEPER-GA. -XXX
The overview of our proposed Deeper-GXX is shown in Figure 2 The green dash line stands for Brigght-ecayyng Graph Residual Connection WWD-ReeNet). In WDG-ResNet, after the current hidden representation (1) is generated by the 1-th layer of arbitrary GNNs, H1)) will be adjusted by its second last layer H (t-2) and the first layer (1) with proper weights. The red dash line stands for Topology- -uidded Graph Contrastive Loss (TGCL). In TGCL, we first need sample positive node pairs and negative node pairs based on the input graph topology such that the hidden representations of positive node pairs get closer and negative ones pushed farther apart. After introducing the TGCL loss to GNNs, the overall loss function Loverall of Deeper-GXX is expressed as follows.
Loverall = LGNN + aLrGct
(1)
where LANN denotes the loss of the downstream task using arbitrary GNN model (e.g., node classification in GCN), LrGCL the TGCL. loss, and a a constant hyperparameter. Deeper-GXX combines WDG-ResNet and TGCL address shading neighbors and over-smoothing problems. The design of WDG-ResNet and TGCL are introduced in the following subsections.
2.2 WEIGHT-DECAYMN GRAPH RESIDUAL CONNECTION (WDG-RESNET)
As we increase the number of layers, one unavoidable problem brought by neural networks is the vanishing gradient, which means that the first several layers of the deeper neural network become hard optimize as their gradients vanish during the training process (He et al.
2016). Currently,
nascent deeper GNN methods (Zhao & Akoglu 2020; Rong et al., 2020 Li et al.
[2099) solve this
problem by adding ResNet (He et al. (0166) on graph neural networks. Taking GCN (Kipf & Welling 2017) as example, the graph residual connection is expressed as follows.
H (ÁA ((-1)) (!-))
where ((2 2) denotes the index of layers, (I-1) and (1-2) are the hidden representations, a(-) is the activation function, (1-1) is the learnable weight matrix, and Å is the re-normalized self- looped adjacency matrix with À = D Ā and Ā = A + In ResNet, the residual connection connects the current layer and its second last layer. Without loss of generality, we assume the last layer of GNNs stacked by ResNet, i.e., is divisible by Then, by extending (1-2) iteratively (i.e., substituting it with its previous residual blocks), the above Eq. could be rewritten as follows.
( αr
(1-2)
()) = (AH (-- (ÁH W(l-3) (1-4)
= -(AA (ÁH W -1) o (AH + +... a (AA(()
Information aggregated
faraway neighhors
aggregated
neighhors
(3)
3

---

Under review as a conference paper at ICLR 2023
According to the GNN theoretical analysis, stacking / layers and getting H in GNNs can be inter- preted as aggregating (-hop neighbors' feature information for the node hidden representations (Xu et al. 2019). As shown in Eq. [ when stack more layers in GNNs, the information collected from faraway neighbors becomes dominant (as there are more terms regarding the information from faraway neighbors), compared with the information collected from the nearest neighbors (e.g., 1-hop or 2-hop neighbors). This phenomenon contradicts the general intuition that the close neighbors of a node carry the most important information, and the importance degrades with faraway neighbors. Formally, we describe this phenomenon as shading neighbors effect when stacking graph neural layers, as the importance of the nearest neighbors diminishing. We show that shading neighbors effect downgrades the GNNs performance in downstream tasks in Section 3.4
To address the shading neighbors effect, we propose (W (WDG-ResNet). Here, we first introduce the formulation and then provide the insights regarding why it can address the problem. Specifically, WDG-ResNet introduces the layer similarity and weight decaying factor follows.
Graph Residual Connection
H (1) a (ÁH ((()),, 1*1-th layer of an arbitrary GNN*
H sim (H F
(0)) -I/A.
+ /*residual connection*/ 1/X,, A (
-/AA. H
cos((1) (0)) (H0)),
where cos(H ((),, H0) = E; | |
( measures the similarity between the 1-th layer and the
1-st layer, and we use the exponential function to map the cosine similarity ranging from {-1, 1) to (e-1,el e)), to avoid the negative similarity weights. The term e -1/2 is the decaying factor to further adjust the similarity weight of ( where À is a constant hyperparameter.
Different from ResNet (He et al. 2016), we add the learnalle similarity sim ()) to essen- tially expand the hypothesis space of deeper GNNs. Additional to that, simply adding vanilla ResNet on GNNs will cause the shading neighbors effect. The introduced decaying factor e -1/a can alleviate this negative effect, because it brings the layer-wise dependency to stacking operations, in order to preserve the graph hierarchical information when GNNs go deeper. Since À is a constant, the value of e 1/> is decreasing while/ / increases. Thus, the later stacked layer is always less important than previously stacked ones by the decaying weight, which addresses the shading neighbors effect. Without the decaying factor, the layer-wise weights independent, and the shading neighbor effect still exists. Moreover, we visualize the layer-wis wise weight distribution of different residual connection methods (including our in Appendix A.4
and their effectiveness in addressing shading neighbors effect
From another perspective, the hyperparameter À of the decaying factor actually controls the number of effective neural layers in deeper GNNs. For example, when À = 10, the decaying factor for the 10-th layer is 0.3679 (i.e., 1);;; but for the 30-th layer, it is 0.0049 (i.e., e))) This decay limits the effective information aggregation scope of deeper GNNs, because the later stacked layers will become significantly less important. Based on this controlling property of X, a natural follow-up question is whether its value depends on the property of input graphs. Interestingly, through our experiments, we find that the optimal X is very close to the diameter of input graphs (if it connected) or the largest component (if it does not have many separate components). This observation verifies our conjecture regarding the property of (i.e., it controls the number of effective layers or number of hops during the message passing aggregation schema of GNNs). Hence, the value of À can be searched around the diameter of the input graph, and we discuss the details in the Appendix A.5
Simplified WDG-ResNet. In the experiments, we observe that if the number of layers of GNNs is too large (e.g., 50 layers or more), the computational cost of adding the similarity function sim HL) () at each graph residual connection layer can be expensive. To accelerate the computation, we formulate a simpler version of Eq. 4 by removing the sim () ()) but keeping the decaying factor. The simplified version of RDG-ResNet is expressed as follows.
( =e V ( + (1-2)
= -1> (AH (1 ()) + +e- (1-2) (AH w((-3) + - (1-4)// O (AH 5) W
+ +... (AH w()) + 2/1
(5)
4

---

Under review as a conference paper at ICLR 2023
Compared with Eq. 4 Eq. gets rid of the exponential cosine similarity measurement, which greatly reduces the computational cost. However, the simplified WDG-Reste RestRe still keeps the decaying factor for the layer-wise dependency, such that the shading neighbors effect can still be alleviated.
GUIDED 2.3 TOPOLOGY- DUUDDEGY-GUIDED GRAPH CONTRASTIVE Loss (TGCL)
To address over-smoothingg current deeper GNN solutions depend on certain hurd-to-acquire prior knowledge (c.g., important hyperparameters or sampling randomness) to get the discriminative node representations (Zhao & Akoglu, 2020; Rong al. 2020; (Zhou et al., 2020). Inspired by these solutions, we are seeking for an effective discriminative indicator that can be easily obtained from the input graph without any prior knowledge. Thus, we propose novel contrastive regularization term, named Kopology-Guidee Graph Contrastive Loss (TGCL), to transfer this hard-to-acquire knowledge into the topology information of the imput graph as follows.
CrCCL -ERN EoyEN; log
0zff(1,, + E Ev,E,, T zx)
(6)
0ij = - dist (A, 4,)/n, 7ik = + dist (A,, Ax)/n
where =((H, () g(-) is an encoder mapping H to another latent space, f(-) is similarity function (e.g.. f(a b) exp( abT
[Tll||a|| D). dist(-) is distance measurement function (e.g.. hamming distance Norouzi et al. 2012) ) NN is the set containing one-hop neighbors of node o; and N; is the complement the set N,.
In Eq. 6 directly connected nodes (14, vj) form the positive pair, while not directly connected nodes (o%, Uk) form the negative pair. The intuition of this equation is to maximize the similarity of the representations of the positive pairs, and to minimize the similarity of the representations of the negative pairs, such that the node representations become discriminative. In the graph contrastive learning setting, designing the positive and negative pairs is very important, because needs to effectively address the over-smoothing issue, and it should also be efficiently obtained given the topology of the input graph. We provide further discussion and theoretical analysis demonstrate the effectiveness of our positive/negative pair sampling strategy for Eq. in Appendix B.2
With Eq. 6 we can analyze the proposed TGCL loss bound terms of mutual information, as stated in Proposition] I In particular, Proposition| I the lower bound of the mutual information between two representations of a neighbor node pair. Therefore, minimizing TGCL is equivalent to maximizing the mutual information of connected nodes by taking graph topology information into consideration.
demonstrates that the TGCL loss for the graph
Proposition 1 Given a neighbor node pair sampled from the graph 9 (V, EV, X), i.e., nodes U; and Uj, we have [z, 2;) 2 LraeL + Ev.~v log(INal), where [(2,, 2,) is the mutual information between two representations of the node pair v; and v.. and LTGCL the topology- guided contrastive loss weighted by hamming distance measurement.
The proof of this proposition can be found in Appendix
B.
3 EXPERIMENTS
In this section, we demonstrate the performance of proposed Deeper-GXX in terms of effective- ness by comparing with state-of-the-art deeper GNN methods and self-ablations. In addition, we conduct case study to show how the increasing number of layers influences the performance of deeper GNNs when the input graph has missing features.
3.1 EXPERIMENT SETUP
Data sets: The Cora (Lu & Getoor 2003) data set is a citation network consisting of 5,429 edges and 2,708 scientific publications from 7 classes. The edge in the graph represents the citation of one paper by another. CiteSeer (Lu & Getoor 2003) data set consists of 3,327 scientific publications which could be categorized into classes, and this citation network has 9,228 edges. PubMed (Namata
5

---

Under review as a conference paper at ICLR 2023
al. 2012) is citation network consisting of 88,651 edges and 19,717 scientific publications from 3 classes. The Reddit (Hamilton et al. 2017b) data set is extracted from Reddit posts, which consists of 4.584 nodes and 19,460 edges. OGB-Arxin (Wang et al. 2020) is citation network, which consists of 1, 1,166,243 edges and 169,343 nodes from 40 classes.
Baselines: We compare the performance of our method with the following baselines including
one vanilla GNN model and four state-of-the-an deeper GNN models: (1) GCN (Kipf & Welling
2017): the vanilla graph convolutional network; (2) GCNII (Chen et al. 2020bb; an extension of
GCN with skip connections and additional identity matrices; (3) DGN (Zhou et al. 2020): the
differentiable group normalization for GNNs to normalize nodes within the same group and separate nodes among different groups; (4) PairNorm (Zhao & Akoglu (2020): GNN normalization layer designed to prevent node representations from becoming too similar; (5) DropEdge (Rong et al.) 2020): GNN-agnostic framework that randomly removes a certain number of edges from the input graph at each training epoch; (6) Deeper- using the simplified WDG-ResNNt in Deeper-GXX
Configurations: In the experiments, we follow the splitting strategy used in (Zhao & Akoglu 2020) by randomly sampling 3% of the nodes as the training samples, 10% of the nodes as the validation samples, and the remaining 87% as the test samples. We set the learning rate to be 0.001 and the optimizer is RMSProp, which is one variant of ADAGRAD (Duchi et al.) 201T). For fair comparison, set the feature dimension of the hidden layer to be 50, the dropout rate to be 0.5, the weight decay rate to be 0.0005, and the total number of iterations to be 1500 for all methods. For Deeper-GXX and Deeper-GXX-.
we sample 10 instances and 5 neighbors for each class from the training set, dist(.) is the hamming distance, and f(-) is the cosine similarity measurement. The experiments are repeated 10 times if not otherwise specified. The code of our algorithm can be found in an anonymous link All of the real-world data sets are publicly available. The experiments are performed on a Windows machine with 16GB RTX 5000 GPU.
The detailed hyperparameters (e.g., X and a) setting for the experimental results in each table can be found in Appendix A.2 Hyperparameter study and efficiency analysis could also be found in Appendix A.5 and A.6 respectively.
3.2 EXPERIMENTAL ANALYSIS
In this subsection, we evaluate the effectiveness of the proposed method on four benchmark data sets by comparing it with state-of-the-arr methods. The base model for all methods we used in these experiments is graph convolutional neural network (GCN) (Kipf & Welling 2017). For fair comparison, we set the numbers of hidden layers to 60 for all methods and the dimension of the hidden layer to be 50. The experiments are repeated 5 times and we record the mean accuracy as well as the standard deviation in Table
Table I: Accuracy of node classification on four benchmark data sets with 60 hidden layers. GCN is used as the backbone for all methods. Method
Cora
CiteSeer
PubMed
Reddit
GCN
0.2962 1 0.0084
0,2125 1 0.0140
0.4172 1 0.0098
0.1140 1 0.0136
PairNorm
0.6759 0.0171
0.4817 + 0.0197
0.7883 0.0101
0.9017 1 0.0241
DropEdge
0.291T + 0.0122
0.2147 + 0.0184
0,4162 + 0.0208
0.10T9 + 0.0324
GCNIT
0.6076 + 0.0050
0.5775 + 0.0027
0.8188 + 0.0030
0.6969 + +.00064
DGN
0.7022 + 0.0079
0.4398 + 0.0118
0.7843 + 0.0032
0.5122 + 0.0143
PairNorm ResNet
0.7394 + 0.0271
0.5544 + 0.0166
0.7995 + 0.0068
0.9385 + 0.0002
DropFdge ResNet
0.1666 + 0.0271
0.1951 + 0.0382
0.5798 + 0.1501
0.09550 + 0.0129
GCNII ResNet
0.7024 + 0.0075
0.6051 + 0.0062
0.8093 + 0.0047
0.7538 + 0.0095
DGN ResNet
0.1543 + 0.0004
0.2104 + 0.0000
0.2086 + 0.0000
0. + 0.0000
Deeper-GXX-S
0.8023 + 0.0117
0.6544 + 0.0099
0.8198 1 0.0012
0.9693 1 0.0036
Deeper-GXX
0,8059 1 0.0028
0.6655 1 0.0117
0.8185 1 0.0016
0.9721 1 0.0011
In the first five rows of Table we observe that when we set 60 hidden layers as the reference,
DropEdge has almost identical performance as the vanilla GCN. PairNorm, GCNII, and DGN increase
the performance by more than 30% in four data sets compared with GCN, The latent reason is that
https://drive google. le com/ fi 1 cbN 741 www.133.33... htt.t/b3.l...hoo. .80khgHHT btN2202
vi view?usp=esn 2osp-sharnn

---

Under review as a conference paper at ICLR 2023
the over-smoothing problem is alleviated to some extent, since these three are deliberately proposed to deal with over-smoothing problem in deeper GNNs. Besides, our proposed method (i.e., Deeper- GXX) and its simpler version (i.e.. Deeper-GXX-St outperform all of these baselines over four data sets. Addition to addressing the over-smoothing problem, another part of our outperformance can be credited to our proposed residual connection. To verify this conjecture, we further incorporate ResNet into PairNorm, DropEdge, DGN, GCNII, and DropEdge. Then, in the sixth to the eighth rows of [ we can observe (1) some baselines like PairNorm and GCNII suffer from the vanishing gradient (e.g., the performance of PairNorm increases by 6% on Cora and 7% on CiteSeer, while the performance of GCNII rises to 75.38% on Reddit and 70.24% on Cora). Also, although GCNII designs a ResNet-like architecture, adding ResNet to GCNII can still boost its performance on three data sets; (2) Compared with their "+ResNet" versions, our Deeper-GXX and Deeper-GXX- S still outperform, which implies that our designed residual connection indeed contributes to the outperformance, and we do the ablation study to quantify each component's 's contribution of Deeper- GXX in Section 3.4; (3) Not all deeper baselines need ResNet. For example, ResSet almost maintains the performance, and ReN+Re drops the performance.
In addition four small data sets, also examine the the node classification performance of Deeper-GXX GXX on a large-scale graph called OGB-Arxiv shown in Figure and Figure In this experiment, fix the feature dimension of the hidden layer as 100, the total iteration is set as 3000 and GCN is chosen as the base model. Due to the memory limitation, we set the number of layers as 10 for all baselines methods in Figure 3b for fair comparison. By observation, we find that (1) in Figure 3h. the performance of Deeper-GXX GXX increases as we increase the number of layers, which verifies our conjuncture that increasing the number of layers indeed leads to better performance in large graphs due to more information aggregated from neighbors; (2) comparing with PairNorm, Deeper-GXX -apper-XX further boosts the performance by more than 5.6% on OGB-Arxiv data set in Figure Bp.
6.75
0.7
6.74
0.735
6.73
10
B4
Paetecm
Depalape
Core
PheMMed
(a)
(c)
Figure 3: (a) Accuracy of Deeper-GXX on OBG-Arxiv data set with different number of layers. (b) Performance (i.e., accuracy) comparison on OGB-Arxiv data set. (c) Accuracy of different base models with 60 hidden layers on four data sets.
In Figure Be, we show the performance of our proposed method with different base models (e.g.. GAT /Velickovic et al. 2018) and SGC (Wu et al. 2019)). In the experiment, we set the numbers of the hidden layers as 60 for all methods and the dimension of the hidden layer as 50. The total number of training iteration is 1500. By observation, we find that both GAT and SGC suffer from vanishing gradient and over-smoothing when the architecture becomes deeper, and our proposed method Deeper- GXX greatly alleviates them and boosts the performance by 40%-60% on average over four data sets. Specifically, compared with the vanilla SGC, our Deeper-GXX boosts its performance by 43% % on the CiteSeer data set and more than 67% on the Reddit data set.
3.3 CASE STUDY: MISSING FEATURE SCENARIO
Why do we need to stack more layers of GNNs? To answer this question, let us first imagine scenario where some values of attributes are missing in the input graph. In this scenario, the shallow GNNs may not work well because GNNs could not collect useful information from the neighbors due to the massive missing values. However, if we increase the number of layers, GNNs are able to gather more information from the k-hop neighbors and capture latent knowledge to compensate for missing features. To verify this, conduct the following experiment: in Cora and CiteSeer data sets (i.c., setting the masked attributes to be 0), gradually increase the number of layers, and record the accuracy for each setting. In this case study, the number of layers
randomly mask p% attributes
7

---

Under review as a conference paper at ICLR 2023
Table 2: Accuracy of node classification two data sets by masking p percent of node attributes. # L denotes the number of layers where a model achieves the best performance.
Node Feature Missing Rale
p 25%
50%
p 75%
data set
Method
Ace
#L
Acc
Acc
#L
GCN ResNet
0.7503 + 0.0101
7
0.7435
0.0048
10
0.7226 + 0.0099
10
PairNorm ResNet
0.7529 + 0.0129
10
0.7482
0.0172
20
0.7262
0.0178
40
Cora
DropEdge+ ResNet
0.7634 = 0.0112
15
0.7611
0.0102
20
0.7297
0.0168
GCNII ResNet
0.2667 = 0.0063
25
0.3351
0.0066
25
0.2914
0.0106
40
DGN w/o ResNet
0.6850 + 0.0184
30
0.6846
0.00147
50
0.6717
0.0156
Deeper-GiXXSS
0.7872 + 0,0128
15
0.7811
0.0117
20
0.7586
0.0121
60
GXX
0.7915 + 0,0060
0.7848
0.0043
20
0.7598
0.0081
60
GCN ResNet
0.6141 + 0.0080
0,5811
0.0093
10
0,5149
0.0173
PairNorm + ResNet
0.6184 + 0.0087
8
0.5947
0.0083
20
0.3176
= 0.0075
10
CiteSeer
DropEdge + ResNet
0.6348 1 0.0156
4
0.6083
1 0.0128
6
0.3240
= 0.0128
10
GCNI ResNet
0.2453 1 0.0045
40
0.2338
= 0.0028
20
0.2403
= 0.0046
25
DGN w/o ResNet
0.4560 1 0.0162
20
0.4593
= 0.0117
15
0.4498 = 0.0292
15
Deeper- GXX-S
0.6508 1 0.0060
10
0.6132
0.0042
15
0.5544
= 0.0138
20
Deeper-GXX
0.6524 + 0,0087
20
0.6169
+ 0.0063
60
0,5576 = 0.0070
50
selected from 3,, 3, 4, 6, 7, 10, 15, 20, 30, 40,50, 60), and the base model GCN. For fair comparison, we add ResNet (He et al. 2016) if it can boost the baseline model's performance by avoiding the vanishing gradient issue. We repeat the experiments five times and record the mean accuracy and standard deviation.
Table [ shows the performance of Deeper-GXX and various baselines with the optimal number of layers denoted as #L, i.e., when the model achieves the best performance. By observation, we find that when the missing rate is 25% shallow GCN with ResNet has enough capacity to achieve the best performance on both CiteSeer and Cora data sets. Compared with GCN, our proposed method further improves the performance by more than 3.83% on the CiterSeer data set and 4.08% on the Cora data set by stacking more layers. However, when observe that most methods tend to achieve the best performance by stacking more layers. Specifically, PairNorm achieves the best performance at 10 layers when 25% features the best performance at 40 layers when 75% features are missing. A similar observation could also be found with GCNII on the Cora data set, DropEdge on CiteSeer data set as well as our proposed methods in both data sets. Overall, the experimental results verify that the more features a data set is missing, the more layers GNNs need be stacked to achieve better performance. Our explanation for this observation is that if the number of layers increases, more information will be collected from the h-hop neighbors to recover the missing information of its 1-hop and 2-hop neighbors.
increase the missing rate to 50% and 75%, we
missing, while it has
3.4 ABLATION STUDY
In this subsection, we conduct an ablation study on Cora to show the effectiveness of WDG- ResNet and TGCL in Table 3 In this experi- ment, we fix the feature dimension of the hidden layer as 50, the total iteration set as 3000, the number of layers is set as 60, the sampling batch size for Deeper-GXX -Xx is 10, and GCN chosen as the base model. In Table 3 Deeper-GXX-T removes TGCL loss, Deeper-GXX-D removes the weight decaying factor in WDG-ResNet and Deeper-GXX GXX-S achieves the simplified WDG-ResNet in Deeper- which removes the similarity measure in WDG-ResNet. In 3 we have the following observations (1) comparing Deeper- GXX with DXx-e,-GXX-T, we find that Deeper-GXX boosts the performance by 1.84% after adding TGCL loss which demonstrates the effectiveness of TGCL address over-smoothing issue: (2) Deeper- outperforms Deeper-GXXX by 5.61 % which shows that Deeper-GX could address the shading neighbors effect by adding the weight decaying factor; (3) comparing Deeper-GXX with Deeper-GXX GXX-S, we verify that adding exponential cosine similarity measure ( ,79)) could further improve the performance by extending the hypothesis space of neural networks.
Table 3: Ablation Study on Cora Data Set w.r.t. Node Classification Accuracy
Method
Accuracy
Deeper-GXL
0.8059 + 0.0028
Deeper-GXX-s
0.8023 + 0.0117
Deeper-GXX-1.
0.7498 + 0.0139
GXX-T
0.7875 + 0.0092
8

---

Under review as a conference paper at ICLR 2023
4 RELATED WORK
Contrastive Learning on Graphs. Recently, contrastive learning attracts researchers' great attention due to its outstanding performance by leveraging the rich unsupervised data. (van den Oord et al. 2018) is one of the earliest works, which proposes Contrastive Predictive Coding framework to extract useful information from high dimensional data with theoretical guarantee. Based on this
work, recent studies (Song& & Ermon 2020; Chuang ct al.) 2020 Khosla et al. [020 Tian et al. 2020
Chen et al. 2020c Zheng et a 22022) reveal surge of research interest in contrastive learning. You
et a 2020) proposes a graph contrastive learning (GraphCL) framework that utilizes different types
of augmentation methods to incorporate various priors and to learn unsupervised representations of graph data. (Qiu et al., 2020) proposes a Graph Contrastive pre-training model named GCC to capture the graph topological properties across multiple networks by utilizing contrastive learning to learn the intrinsic and transferable structural representations. (Zheng et al. (02T)) introduced a weakly supervised contrastive learning framework to tackle the class collision problem by first generating weak label for similar samples and then pulling the similar samples closer with contrastive
regularization. Authors Hassani & Ahmadi (020) aims to learn node and graph level representations by contrasting structural views of graphs. In this paper, we leverage the contrastive learning manner and design positive and negative node pairs such that we can discriminate node representations based on the input graph inherent information, which paves the way for us to design effective deeper GNNs.
Deeper Graph Neural Networks. To effectively make GNNs deeper, many research works focus on addressing the over-smoothing problem. Svor-smog problem of GNNs is formally described by (Li al. 2018) to demonstrate that the final output node representations become indiscriminative after stacking many layers in GNN models. This problem is also analyzed by (Oono & Suzuki 2020)
showing how over-smoothing hurts the node classification performance. To quantify the degree of over-smoothing, different measurements are proposed (Chen et al., [020a Zhao & Akoglu 2020
Liu et al. 2020 Zhou et al. 2020). For example, Mean Average Distance (Chen et al.] 2020a) is
proposed by calculating the divergences between learned node representations. To make GNNs
deeper and maintain performance, some nascent research works are proposed (Kliepera et al.
2019
Chen al.
2020al Zhao & Akoglu, 2020; Rong et al.j [020)] Chen et al.] 2020b]]Liu et al.
2020
Zhou et al.
2020j. Among those, some of them share the same logic of keeping the divergence
between node representations but differ in specific methodologies. For example, PairNorm (Zhao &
Akoglu 2020) introduces a normalization layer to keep the divergence of node representation from the original input node feature. In DGN (Zhou et al. 2020), node representation by deep GNNs is regularized by group-base based mutual information. However, the hyperparameters selection is important and data-driven, which means many efforts need to be paid for the hyperparameter searching. Some methods focus changing the information aggregation scheme to make GNNs deeper, such as APPNP (Klicpera et al. 2019), GCNII (Chen et al. 20205), DropEdge (Rong et al. 2020) and etc. To the best of our know ledge, in dealing with over-smoothing problem, our Deeper-GXX is the first attempt to transfer tord-io--cquir discriminative knowledge into the topology information of the input graph by comparing adjacency vectors of nodes. Also, we analyze another problem (i.e.. vanishing gradient) that hinders GNNs to be effectively deeper and discover the shading neighbors effect caused by simply applying ResNet on GNNs, and propose a GNN- residual connection to avoid this issue and improve the performance of deeper GNNs.
5 CONCLUSION
In this paper, we focus on building deeper graph neural networks to effectively model graph data. To this end, we first provide insights regarding why ResNet not best suited for many deeper GNN solutions, i.c., the shading neighbors effect. Then we propose new residual architecture, Weight- Decaying Graph Residual Connection (WDG-ResNet) to address this effect. In addition, we propose a Topology-guice -ugdeed Graph Contrastive Loss (TGCL) to address the problem of svootsiot,,,ng, where we utilize graph topological information, pull the representations of connected node pairs closer, and push remote node pairs farther apart via contrastive learning regularization. Combining WDG- ResNet with TGCL, an end-to-end model named Deeper-GXt GXX is proposed towards deeper GNNs. We provide the theoretical analysis of our proposed method, and demonstrate the effectivvness of Deeper-GXS GXX by extensive experiment comparing with state-of- of-te-of-the-an ahet verrooosmonggnng algorithms. A case study regarding the missing feature scenario demonstrates the necessity to deepen the GNNs.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. AAAI, 2020a.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In ICML,, 2020b.
Ting Chen, Simon Kormblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ICML, 2020c.
Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debi- ased contrastive learning. In NeurIPS, 2020.
John C. Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 2011.
William L. Hamilton, Zhitao Ying. and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 1024-1034, 2017a.
William I.. Hamilton, Justine Zhang, Cristian Danescu-Niccuessuu-Me Leskovec. Loyalty in online communities. In Proceedings of the Eleventh International Conference on Web and Social Media, ICWSM 2017, Montréal, Québec, Canada, May 15-18, 2017, pp. 540- 543. AAAI Press, 2017b.
Dan Jurafsky, and Jure
Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view representation learning on graphs. In ICML, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kexin Huang and Marinka Zitnik. Graph meta learning via local subgraphs. In NeurIPS, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NeurIPS, 2020.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Johannes Klicpera, Aleksandar Bojehevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Guohao Li, Matthias Müller, Ali K. Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns? In ICCV, 2019.
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In AAAI, 2018.
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In KDD, 2020.
Qing Lu and Lise Getoor. based classification. In Tom Fawcett and Nina Mishra (eds.), Machine Learning, Proceedings of the Twentieth International Conference (ICML 2003), August 21-24, 2003, Washington, DC, USA, pp. 496-503. AAAI Press, 2003.
Galileo Namata, Ben London, Lise Getoor, Bert Huang, and U Edu. Query-driver active surveying for collective classification. In 10th International Workshop on Mining and Learning with Graphs, volume 8, 2012.
Mohammad Norouzi, David J Fleet, and Russ R Salakhutdinov. Hamming distance metric learning. Advances in neural information processing systems, 25, 2012.
10

---

Under review as a conference paper at ICLR 2023
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In ICLR, 2020.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. GCC: graph contrastive coding for graph neural network pre-training. In KDD, 2020.
Yu Rong. Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In ICLR, 2020.
Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. In NeurIPS, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, 2020.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 2018.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
Hongwei Wang and Jure Leskovec. Unifying graph convolutional neural networks and label propaga- tion. CoRR, 2020.
Kuansan Wang, Zhihong Shen, Chiyuan Huang, Chich-Han Wu, Yuxiao Dong, and Anshul Kanakia. Microsoft academic graph: When experts are not enough. Quantitative Science Studies, 1(1): 396-413, 2020.
Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In ICML, 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurlPS, 2020.
Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. ICLR, 2020.
Lecheng Zheng, Jinjun Xiong, Yada Zhu, and Jingrui He. Contrastive learning with complex heterogeneity. In KDD 22: The 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, Washington, DC, USA, August
18, 2022, pp.
ACM, 2022.
Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu. Weakly supervised contrastive learning. CoRR, 2021.
Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu. Towards deeper graph neural networks with differentiable group normalization. In NewrIPS, 2020.

---

Under review as a conference paper at ICLR 2023
A EXPERIMENT
A.1 CASE STUDY: A TOY EXAMPLE
Why do we need to stack more layers of GNNs? To answer this question, we conduct the experiment in toy example. We first the existing package (i.e., draw circle function in the Scikit-learn package) to generate a synthetic data set by setting the number of data points to be 1,000 and the noise level be 0.01. Then, we measure the euclidean distance between each pair of data points, and if the the distance is less than a threshold, then this two data points are connected in a graph. In this way, the adjacency matrix is derived after adding the self loop. Next, we sample 1% data points as the training set, 9% data points as the validation set, and 90% data points as the test set. These data points are visualized in Figure Ta and the experimental results are shown Figure [b In Figure Ta we observe that the query node (the diamond in the red dashed circle) cannot rely on its closest labeled neighbor (the red star in the circle) to correctly predict its label (red or blue). Only by exploring longer paths consisting of more similar neighbors are we able to predict its label as blue. Figure Tb compares the classification accuracy of shallow GNNs and deeper GNNs. We can that deeper GNNs significantly outperform shallow ones by more than 11 %6 due to their abilities to explore longer paths on the graph.
A.2 EXPERIMENTAL SETTING
In this subsection, we provide the detailed experimental setting for each experiment shown in Table
Table 4: Hyperparameters for Deeper-GXX GXX and Deeper-GXXXS shown in Table 1
Method
Deeper-Git GXX
Ceeper-GXX.
Cora
20, a 0.03
a 0.01
CiteSeer
À 10, a 0.02
0.01
PubMed
A =188 a 0.1
a = 0.1
Reddit
A ==20, a 0.02
0.01
A.3 MORE RESULTS IN ABLATION STUDY
In addition to the ablation study in Section 3.4, here in Table 5 we conduct more ablation experiments on CiteSeer, PubMed, Reddit, and OGB-Arxiv data sets, following the same experimental setting in Table 3 to demonstrate the effectiveness of each component in Deeper-GXX by comparing it with three variants.
Table 5: Ablation Study on CiteSeer, PubMed, Reddit, and OGB-Arxiv Data Sets w.r.t. Node Classification Accuracy
Method
CiteSeer
PubMed
Reddit
OGB-Arxiv
Deeper-GXX
0.6655 + 0.0117
0.8185 + 0.0016
0.9721 + 0.0011
0.7401 + 0.0009
Deeper-GXX GXX-S
0.6544 0.0099
0.8198 + 0.0012
0.9693 + 0.0036
0.7382 0.0008
Deeper- Deeper-GXX-D GXX-D
0.6567 = 0.0052
0.8150 = 0.0031
0.9654 + 0.0028
0.7363 = 0.0011
Deeper-GXX-T
0.5750 + 0.0244
0.8098 + 0.0047
0,9397 + 0.0042
0.7335 + 0.0024
A.4 VISUALIZATION OF WEIGHT OF EACH LAYER WITH DIFFERENT WEIGHTING FUNCTIONS
In this subsection, we visualize the weight of each layer with different weighting functions the Cora data set. In this experiment, we fix the feature dimension of the hidden layer to be 50; the total iteration is set be 3000; the number of layers is set to be 60; the sampling batch size for Deeper-Gae GXX is 10; GCN is chosen as the base model; À is set to be 20. In Figure 4 The x-axis the index of each layer and the y-axis the weight for each layer. Deeper-GXX decaying weight factor and only keeps the exponential cosine similarity ( ( (7()) to measure the weight for each layer. Dexper-GXX-S achieves the simplified WeG in Deeper-GXX,
removes the
12

---

Under review as a conference paper at ICLR 2023
Deeper-GXX - -De-per-CXX-D Deeper-CXx-S ResNett
0
0
10
20
30
40
60
Figure 4: Weight visualization. The y-axis represents the weight of each layer, and x-axis represents the index each layer, deeper models
which removes the exponential cosine similarity ( (HO)))))) in Deeper-GXX, By observation, we find that (1) ResNet sets the weight of each layer to be 1, which easily leads to shading neighbors effect when stacking more layers, because the faraway neighbor information becomes more dominant in the GCN information aggregation; (2) without weight decaying factor, the weight for each layer in Deeper-GXX-D fluctuates because they are randomly independent. More specially, the weights for the last several layers (e.g.. L=58 L=60) larger than the weights for the first several layers, which contradicts the intuition that the first several layers should be important than the last several layers; (3) the weights for each layer in both Deeper-GXX and Deeper-GXX-S reduce the number of layers increase, which suggests that both of them could address the shading neighbors effect to some extents; (4) combining the results from Table 3 Deeper- GXX achieves better performance than Deeper- GXX-S, as it imposes larger weights to the first several layers, which verifies that the learnable similarity sim HH)), ( achieves better performance with the enlarged hypothesis space for neural networks.
A.5 HYPERARAMETER ANALYSIS
In this subsection, conduct the hyperparameter analysis of Deeper-GXX, regarding in the weight decaying function of Eq. 4 and a in the overall loss function of Eq]
0.9
0.7
Accuracy
0.8
Accuracy
0.6
0.7
0.5
0.6
0.4
1
10
20
30
40
50
60
20
30
40
50
Lambda
Lambda
(a) Cora data set
(b) CiteSeer data set
0.85
Accuracy
Accuracy
0.8
0.9
0.75
0.8
10
17
20
30
40
50
60
Lambda
Lambda
(c) PubMed data set
(d) Reddit data set
Figure 5: Hyperparameter analysis, i.c., À vs accuracy score on four data sets
To analyze the hyperparameter X, we fix the feature dimension of the hidden layer to be 50, the total iteration is set to be 3000, the number of layers is set to be 60. the sampling batch size for Deeper-GXX is 10, and GCN chosen as the base model. The experiment repeated five times for each configuration. In each sub-figure of Figure the x-axis is the value of X, and the y-axis
13

---

Under review as a conference paper at ICLR 2023
is the accuracy of 60-layer GCN in the above setting. First, observe that it's not true that Deeper-GXX achieves the best performance with a larger X. Specifically, we find that the optimal À 20 Cora data set, the optimal À 10 on CiteSeer data set, the optimal À 18 on PubMed data set, and the optimal À = 20 on Reddit data set. Then, natural questions to ask are (1) what determines the optimal value of X in different data sets? (2) can we gather some heuristics to narrow down the hyperparameter search space to efficiently establish effective GNNs? Here, we provide our discovery. the main text of the paper, we have analyzed that the decaying factor À in Eq. controls the number of effective layers in deeper GNNs by introducing the layer-wise dependency. It means that larger Ả slows down the weight decay and gives considerable large weights more layers such that they can be effective, and the information aggregation scope of GNN extends more multi-hop scope the
neighbors features are collected and aggregated. graph, which is the largest value of the shortest path between any node pairs in the graph. Therefore, the optimal À should be restricted by the input graph, i.c., being close to the input graph diameter. Interestingly. experiments reflect this observation. Combining the optimal À in Figure 5 and the diameter in Table 6 for connected graphs PubMed and Reddit, the optimal is very close to the graph diameter. This also happens to Cora (even though Cora is not connected), because the number of components is not large. As for CiteSeer, the optimal À is less than the diameter of its largest component. possible reason is that CiteSeer has many (i.e., 438) small components, which shrinks the information propagation scope, such that we do not need to stack many layers and we do not need to enlarge the largest diameter (i.e., 28). In general. based on the above analysis, we find the optimal value of À can be searched around the diameter of the input graph.
graph theory, diameter represents
Table Graph Statistics of each Data Set
Cora
Citeseer
PubMed
Reddit
Number of nodes
2,708
3,327
19,717
4,854
Connected graph or not
No
No
Yes
Yes
Number of components
78
438
Diameter of the graph or the largest component
19
28
18
17
To analyze the hyperparameter in Deeper- GXX, we fix the feature dimension of the hidden layer be 50, the total iteration is to be 3000,
0.85
the number of layers is set to be 60, the sampling batch size for Deeper-GXL - 10, GCN is cho- sen as the base model, and the data set is Cora. We gradually increase the value of a and record the accuracy. The experiment is repeated five times in each setting. In Figure 6 the x-axis is and the y-axis is the accuracy score. By observation, when a = 1, the performance is worst and the performance begins to increase by decreasing the value of a. It achieves the best accuracy when a 0.03. The performance starts to decrease again if we further decrease the value of a. Our conjecture is that when a is large, it will dominate the overall objective function, thus jeopardizing the classification performance. Besides, if we set the value of a to be a small number (i.e., 0.001), the performance also decreases. In addition, comparing with the performance without using TGCL. regularization (i.e., = 0), our proposed method with a 0.03 can boost the performance by more than 1.8%, which demonstrates that our proposed TGCL alleviates the issue of oversmoothing to some extent.
0.8
Accuragy
0.75
0.7
0.65
0.6
0.001
0.01
0.03
0.05
0.1
1
Alpha
Figure 6: Hyperparameter analysis, i.c., vs accu- racy score
A.6 EFFICIENCY ANALYSIS
In this subsection, we conduct an efficiency analysis regarding our proposed method in the Cora data set. We fix the feature dimension of the hidden layer to be 50, the total iteration is set to be 1500, the sampling batch size for Deeper-GX GXX and Deeper-GXX- is 10, and GCN is chosen as the base model. We gradually increase the number of layers and record the running time. In Figure 7
14

---

Under review as a conference paper at ICLR 2023
1090
Deeper-GXXS
GCN ResNet
Deeper-GXX
880
Rur ning time
609
100
20
50
50
Number of layers
Figure The number layers vs running time (in seconds) on Cora data set. (Best viewed color)
the x-axis is the number of layers and the y-axis is the running time in second. We observe that the running time of both Deeper-GXX and Deeper-GXX-S is linearly proportional to the number of layers. Comparing the running time of Deeper-GXa GXX, the running time of Deeper- Deeper-GXX-S GXX-S is further reduced after the weighting functtin in Deeper-GXX (e.g., sim(-)) is replaced by a constant.
B ANALYSIS
B. PROOF OF PROPOSITION 1
Proof Following the theoretical analysis in (yan den Oord et al. 2018 Section 2.3, the optimal Thus, the weighted supervised contrastive loss could be P(z,))
value of fzz, zj) is given by P(z,lz))
rewritten follows:
Oig. Fzz,(Zzz2)
LTGLL - EEnay BujeMi log
(z,
EneeN (ik
P(z; Js P(2,)
+ E
U2EN Yek p(an)
P(ec)
Eu Eu,EN, flog
P(2)
Ts P(2,)
P(z,)
P(zzlzi)
E BuyeNi log(1
+
ozg oiy P(zzlz)
Σ
Vikk
P(z+)
v.EN.
Since (vi, Us) is defined as a remote (i.e., negative) node pair, it means that node v; and node vs are not connected in the graph, i.e., Ai,k AR,i = 0. Therefore,
have "Kk (1, 2] for all remote
nodes Uk and osj E (0, for all neighbor nodes vy with hamming distance measurement, which P(as)
leads oz P(z)(z.) P(z)(z.)
P(x,) and tik P(za) 2 p(z)(=.). Thus, have P(zu)
P(zx(z))
P(zz)
P(zulzi)
LIGCL 2 Eu Ee, eNG EN, [log(
(1)
P(zx)
VLEN;
P(z,)
P(zulzi)
~ Buyoy Ee, eN. [log(
(WN LEv.
P(zx) ))
P(z 2:)
P(z₂)
-Enny EeyeNs [log(
P(z, 22.)
P(zz)
2 Eunoy Ee, eNG [log(
) log([
P(z [2))
-II(zi, 2j) + Eg,av log( (WWl)
Finally, have T(2;, 2z) 2 -LTCCL + Byyov log(),?)), which completes the proof.
15

---

Under review as a conference paper at ICLR 2023
B.2 SAMPLING METHOD FOR TGCL
To realize TGCL loss function expressed in Eq. 6 need to get the positive nodes v and neg- ative nodes U/ towards the selected central node v/- To avoid iterating over all existing nodes or randomly sampling several nodes, propose sample positive nodes vj and negative nodes 0k from the star subgraph S; of the central node e4. Moreover, to make the sampling be scal- able and reduce the search space of negative nodes, we propose batch sampling method. As shown in Figure the batch size is controlled by the number of central nodes (i.e., star nodes in the figure). For each central node, the positive nodes are those 1-hop neighbors, and the negative nodes consist of unreachable nodes. In our batch sampling, we strictly constrain that the positive nodes are only from the hhhop neighborhood for the following three reasons: (I) they efficient to be ac- cessed; (2) considering all k-hop neighbors as positive will enlarge the scope of positive nodes and further decrease the intimacy of the directly connected nodes; (3) 1-hop positive nodes in the star subgraph can preserve enough useful information, compared with the positive nodes from the whole graph. For the third point, prove it through the graph inftuence loss (Huang & Zitnik 2020) in Propo- sition [ and the formal definition of graph inffuence loss is given in the following paragraph after Proposition 2
existing edge (positive pair)
non-existing edge (negative pair)
Figure 8: Batch Sampling. Each star node in the figure corresponds to node in Eq.
Proposition 2 (Bounded Graph Influence Loss for Sampling Positive Pairs Locally) Taking GCN as example of GNN, the graph influence loss R(ve) on node v. w.r. positive nodes from the whole graph against positive nodes from the I-hop neighhorhood star subgraph is bounded by R(ve) (n de) [P. where is the number of nodes, de the degree of node ve including
the self-loop, is a constant, P. is the path from center node Ve to a I-hop outside node U, which has the maximal node influence Iugor and [P.] denotes the number of nodes in path P..
Proof 2 According to the assumption of (Wang & Leskovee 2020), o.)) can be identity function and W can be identity matrix. Then, the hidden node representation (of node ve) in the last layer of GCN can be written as follows.
(00) = Deg v.EN
1 - Ac
Then, based on the above equation, can iteratively replace h (000) with its neighbors until the representation h(()) node v, is included. The extension procedure is written as follows.
h (00)
1 Σ Acti A
Σ Ae,h!(o))
De,
Di,
DR.k
v.N.
uyEN;
v.EN.
The above equation suggests that the inftuence from the positive node v, to the center node ve is through the path P = (Vey Vi, Vj...., Uk, Va).
Following the above path formation and assume the edge weight A(i,j) as the positive constant, according to (Huang & Zimik 2020), we can obtain the node inffuence Iec,s of U, on Ve as follows.
I I = (1ah(00) (00) /ahnto)
P
(D
where μ is a constant, D& is the geometric mean of degree of nodes sitting in path P. and is the path from the positive node V, the center node v, that could generate the maximal multiplication of normalized edge weight, [Pl denotes the number of nodes in path P.
The above analysis suggests that the node influence of long-distance positive nodes is decaying.
16

---

Under review as a conference paper at ICLR 2023
Hence, the graph inftuence loss about learning node te from the whole graph positive nodes versus from the 1-hop localized positive nodes can be expressed expressed as follows.
de
As
la(vo) - (ve) Lowes + Lueiez t... +.y Lve do
(D
PPl1
1=1
GM
S (n de)
H4
(DeMs)
where (V( denotes global influence, 1 (Te) is the inftuence for star subgraph, de is the degree of node Ve (including self-loop), and (D (Pa)
is the maximal among all
(D GD )
Specifically, the graph influence loss HHang Zitnik 2020) R(ve) can be expressed R(v,) (v(v)) (v.),. which is determined by the global graph influence on ve (i.e., Io(v))) and the star subgraph influence on te (i.e., (ve)). Then, compute the graph influence (vc),, we need com- pute node influence of each node to node Ue. where node 1 is reachable from node Ve- Based on the
final output node representation vectors, the node influence is expressed as Lv_.y; ||h (00) /0h, and the norm can be any subordinate norm (Wang & Leskovec 2020). Then, 1g (v.) is computed by the L1-norm of the following vector, i.c., (G(2)) ||[ 12.121.......................... 12.......... Lve.v.. ||| Similarly,
can
compute the star subgraph influence Iz (v:) on node Ue- The only difference is that we collect each reachable node by in the star subgraph L (i.e., I-hop neighbours of Uz). Overall, in Proposition 2 show why positive pairs can be locally sampled with the support from graph influence loss of node representation vector output by the GCN final layer.
17


Reviews:
[{"review_id": "pQF-pSYIe0Z", "reviewer": "Reviewer_y1R7", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two tricks to improve GNN performance at large depths: WDG-ResNet (weighted residual connections) and TGCL (additional loss term). Through experiments and ablations they show that a GNN enhanced with both modifications is better-behaved when scaling up depth.", "summary_of_the_review": "The premise of this work is overall sound, and the results look promising, but some extra intuition is needed, and I would welcome some more baseline comparisons. For my initial evaluation I lean slightly to the negative side, but I'm open to discussing with the authors and other reviewers.", "strength_and_weaknesses": "=== Strengths === \n\n(S1): This work explores generalizable modifications to GNNs that could be applicable to a broad range of architectures/problems. \n\n(S2): The authors define the \"shading neigbhors effect\", which they use to motivate their approach better. \n\n \n\n=== Weaknesses === \n\n(W1): The motivation behind the `sim` component of the architecture (Equation 4) is unclear to me, and the gains from adding it border on statistical significance: \n\n- The authors mention that this component \"expands the hypothesis space of GNNs\", but I'm not sure how to understand that. What are we trying to achieve with this term? \n\n- Are we backpropagating through `sim` during training? \n\n- Judging from results in Table 1 and 2, the gain from adding this term is minimal, bordering on statistical significance. I agree there is *some* signal if one integrates the small increments across all the different experiments and setups, but (as the authors note), this component also increases compute requirements and complicates things conceptually. \n\n(W2): Some more comparisons would help to contextualize the results: \n\n- Simplified WDG could be compared to an ablation where the weights are learned scalars, instead of being fixed to `e^{-l/lambda}`. This would show whether the decaying-by-design weighing is indeed what is causing the improvements, or the sheer fact that there is a post-activation coefficient smaller than 1 is already enough. \n\n- The authors compare against ResNet-style GNNs, but there's another approach (also extremely practical and common), which is to concatenate the hidden representations across all layers, project that concatenated vector to a suitable dimension, and use that for the downstream task. This is sometimes referred to as \"Jumping Knowledge\", and obtained by setting `jk=\"cat\"` in `pytorch_geometric`. \n\n \n\n=== Nitpicks === \n\nHere I include some final nitpicks, which did not affect my score; they are here just to help improve the paper. \n\n- \"user’s immediate neighbor information may not reflect his/her categorical information\": use \"their\" instead of \"his/her\" \n\n- \"information collected from faraway neighbors become dominant\": \"become\" -> \"becomes\" \n\n- \"This radioactive decay\": drop the word \"radioactive\" \n\n- \"simpler version of Eq. 4 by fixing the sim\": I think \"fixing\" should rather be \"removing\"? \n\n- Missing space before citation in Section 2.3 \n\n- I'd call Section 3 \"Experiments\" (plural) instead of \"Experiment\" \n\n- \"could be found in Appendix\" (twice): I would say \"can\" instead \n\n- \"shallow GCN with ResNet has enough capability\": \"capability\" -> \"capacity\" \n\n- \"we need more neighbors aggregated for collecting their partial features to compensate for the missing such that stacking GNN layers works\": this sentence seems broken, please rephrase\n\n- \"conduct the ablation study\": \"the\" -> \"an\" \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ncKnRsetj_", "reviewer": "Reviewer_G8Tr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Weight-Decaying Graph Residual Connection and Topology-Guided Graph Contrastive Loss to overcome the over-smoothing issue of GNNs in order to train deeper GNNs. Experiments are conducted on citation networks and social networks such as Cora CiteSeer PubMed Reddit.", "summary_of_the_review": "The major issue of acceptance is the evaluation of the proposed method is done on relatively small datasets. It is not clear how significant it is compared to other SOTA deep GNN methods. I recommend the authors should evaluate their methods on Open Graph Benchmark and Benchmarking Graph Neural Networks to make their claims more conniving.", "strength_and_weaknesses": "Pros:\n\nThe proposed Weight-Decaying Graph Residual Connection is interesting.\n\nCons:\n\nWhy is the graph residual connection expressed as $H^{(l)} = \\sigma(\\hat{A} H^{(l-1)} W^{(l-1)}) + H^{(l-2)}$ in Equation 2? Is it typically $H^{(l)} = \\sigma(\\hat{A} H^{(l-1)} W^{(l-1)}) + H^{(l-1)}$?\n\nThe proposed Topology-Guided Graph Contrastive Loss which considers 1-hop neighbors as the positive pair and the rest as negative pairs has a strong assumption on homophily. I wonder if the proposed method would be helpful for graphs with strong heterophily.\n\nIn my opinion, this work is not that related to contrastive learning on graphs. However, a large paragraph is used to discuss contrastive learning on graphs which is not necessary. I think the related work can be improved to focus more related works on over-smoothing and different techniques in deepening GNNs.\n\nThe experiments are done on relatively small datasets which makes the results less conclusive. I wonder if the authors could perform experiments on recent GNN benchmarks such as Open Graph Benchmark and Benchmarking Graph Neural Networks.\n\n=== post rebuttal ====\n\nThe authors address some of my concerns. I increased my score. But I still have doubts about the claim that the proposed method can work well on graphs with strong heterophily.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "G8_nU51BcT", "reviewer": "Reviewer_YeUe", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of vanishing gradient and over-smoothing in graph neural networks which is highly related to the deep GNN training. This paper proposes two tricks into existing GNN layers named: Weight-Decaying Graph Residual Connection module (WDG-ResNet) and Topology-Guided Graph Contrastive Loss (TGCL). Experiments conducted on four datasets demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "Although the problem studied in the paper is intersting and the paper is well written, my major concerns are the technical contribution (both two tricks) and the experimental results (datasets and baselines). The authors are expected to provide detailed evidences to address the above concerns for increasing the final scores.", "strength_and_weaknesses": "**Strength**\n- The paper is overall well written and easy to follow. The figures and tables are well described.\n- The finding that \"optimal decaying factor is close to the diameter of input graphs (if it is connected) or the largest component (if it does not have many separate components).\" is interesting and may inspire some follow up works.\n- The reproducibility of this paper looks good to me.\n\n**Weakness**\n- The technical contribution is limited. Concerning the two tricks, the first trick has been widely studied in the literature, such as the Jumping Knowledge Networks[1] or the Predict then Propagate [2], the idea is quite similar that by incoporating information from the longer distance nodes while avoiding the domination of them, which is a trade-off between close and distant nodes. This paper propose a decay funnction which is a degraded version of above methods. The second trick is widely studied in contrastive learning that adding weights on the positive and negative pairs by the hardness of sample, where the distance is utilized as hardness here.\n- The critical design is the  learnable similarity between the layers, which distinguish this paper from the ResNet. However, due to the complexity and training efficiency proble, the simiplified WDG-ResNet ignores the learnable parameters and this makes the proposed WDG-ResNet quite similar to the ResNet with only layer-wise constant decays. This is quite weired and the motivation of WDG-ResNet becomes unclear.\n- The experiments are conducted on the four datasets: Cora, Citeseer, Pubmed and Reddit, which is quite small. Furthermore, these datasets are representative ones with high homophily, where neighborhood nodes tends to share similar labels. The performance of the proposed method on large datasets and lower homophily datasets is not convincing.\n- The baselines are weak, many GNN methods such as [1][2] and graph contrastive learning methods are missed.\n\n\n[1] Representation Learning on Graphs with Jumping Knowledge Networks\n[2] Predict then Propagate: Graph Neural Networks meet Personalized PageRank", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "AMDPxMVz62s", "reviewer": "Reviewer_9Psf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes Weight-Decaying Graph Residual Connection (WDG-ResNet) and Topology-Guided Graph Contrastive Loss (TGCL) to address shading neighbors and over-smoothing issues. Based on proposed WDG-ResNet and TGCL, the authors design an end-to-end model called Deeper-GXX to help GNNs perform well with deep layers.", "summary_of_the_review": "Overall, I am leaning towards rejection. My major concern is the novelty and experimental results. If you address my concerns, I will raise my score. \n\n=== post rebuttal ===\n\nThank you for your detailed response on the review. Even though some concerns are addressed, I still have concerns on the paper so that I maintain my score.", "strength_and_weaknesses": "**Strengths**\n\n(1) This paper addresses the over-smoothing issue of graph neural networks, which is one of important topics to graph neural networks with deep layers.\n\n(2) The proposed Deeper-GXX has demonstrated why stacking deep layer is important with the missing feature scenario and shown good performance in missing feature scenario. \n\n**Weakness**\n\n(1) I think that the novelty of this paper is limited. \n\n- The decaying factor of proposed Weight-Decaying Graph Residual Connection seems very similar to $\\alpha$ in APPNP [1].\n- Also, Topology-guided Graph contrastive loss seems similar to self-supervised loss of SuperGAT [2] and PairNorm [3]. Specifically, the intuition of both papers is maximizing similarity between connected nodes and minimizing between unconnected nodes.\n\n(2) Table 1 shows that Deeper-GXX shows good performance when the number of layer is 50. But, GCN shows better performance when the number of layer is lower. To validate the effectiveness of Deeper-GXX, I think comparison with baselines depending on the number of layers is needed.     \n\n(3) From Table 3, we can know that the weight decaying factor mostly contributes to the performance gain. Also, the performance gain of the similarity measure is marginal. It would be better to provide ablation studies on other dataset.\n\n---\n\n[1] Klicpera, Johannes, Aleksandar Bojchevski, and Stephan Günnemann. \"Predict then propagate: Graph neural networks meet personalized pagerank.\" ICLR 2019.\n\n[2] Kim, Dongkwan, and Alice Oh. \"How to find your friendly neighborhood: Graph attention design with self-supervision.\" ICLR 2021.\n\n[3] Zhao, Lingxiao, and Leman Akoglu. \"Pairnorm: Tackling oversmoothing in gnns.\" ICLR 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject