Tackling Oversmoothing of GNNs with Contrastive Learning
Authors: Lecheng Zheng
Dongqi Fu
Jingrui He
Venue: ICLR 2022 Submitted (2022)
Keywords: graph mining
oversmoothing
contrastive learning

Abstract: Graph neural networks (GNNs) integrate the comprehensive relation of graph data and the representation learning capability of neural networks, which is one of the most popular deep learning methods and achieves state-of-the-art performance in many applications, such as natural language processing and computer vision. In real-world scenarios, increasing the depth (i.e., the number of layers) of GNNs is sometimes necessary to capture more latent knowledge of the input data to mitigate the uncertainty caused by missing values.
However, involving more complex structures and more parameters will decrease the performance of GNN models. One reason called oversmoothing is recently proposed, whose research still remains nascent. In general, oversmoothing makes the final representations of nodes indiscriminative to hurt the node classification and link prediction performance.
In this paper, we first survey the current de-oversmoothing methods and propose three major metrics to evaluate a de-oversmoothing method, i.e., constant divergence indicator, easy-to-determine divergence indicator, and model-agnostic strategy. Then, we propose the Topology-guided Graph Contrastive Layer, named TGCL, which is the first de-oversmoothing method maintaining the three mentioned metrics. With the contrastive learning manner, we provide the theoretical analysis of the effectiveness of the proposed method. Last but not least, we design extensive experiments to illustrate the empirical performance of TGCL comparing with state-of-the-art baselines.

Under review as a conference paper at ICLR 2022
TACKLING OVERSMOOTHING OF GNNS WITH CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Graph neural networks (GNNs) integrate the comprehensive relation of graph data and the representation learning capability of neural networks, which is one of the most popular deep learning methods and achieves state-of-the-art performance in many applications, such as natural language processing and computer vision. real-world scenarios, increasing the depth (i.e., the number of layers) of GNNs is sometimes necessary to capture more latent knowledge of the input data to miti- gate the uncertainty caused by missing values. However, involving more complex structures and more parameters will decrease the performance of GNN models. One reason called oversmoothing is recently introduced but the relevant research remains nascent. In general, oversmoothing makes the final representations of nodes indiscriminative, thus deteriorating the node classification and link predie- tion performance. In this paper, we first survey the current de-oversmoothing methods and propose three major metrics to evaluate a de-oversmoothing method, i.e., constant divergence indicator, casy-to-determine divergence indicator, and model-agnostic strategy. Then, we propose the Topology-guice Graph Con- trastive Layer, named TGCL, which is the first de-oversmoothing method main- taining all three mentioned metrics. With the contrastive learning manner, we pro- vide the theoretical analysis of the effectiveness of the proposed TGCL. Last but not least, we design extensive experiments to illustrate the empirical performance of TGCL comparing with state-of tfrtherrrt baselines.
INTRODUCTION
Combining the graph data comprehensive relations with the neural network models representation learning ability, graph neural networks (GNNs) achieve state-of-the-art performances many real- world applications, such document classification, natural language processing, computer vision, and recommender systems (Zhang et al. 2019). GNNs consist of many variant neural network models with different message- passing mechanisms, to name few, such GCN (Kipf & Welling 2017), GrE (Hamilton et al. 2017), GAT (Velickovic et al. 2018), GIN (Xu et al. 2019 and GMNN QQu al. 2019)..
In the complex real-world settings of applying GNNs, not every node is lucky enough to have node labels and/or node features. Hence, increasing the depth (i.e., the number of layers) of GNNs is a viable solution to capture more latent knowledge to reduce the uncertainty caused by missing values (Zhao & Akoglu 2020). However, the number of layers increases, the performance of GNN will decrease to a large degree (Kipf & Welling 2017). The reasons may come from many aspects after involving more parameters like vanishing gradient, overfitting, and oversmoothing Compared with the first two reasons, oversmoothing of GNNs is recently introduced (Li et al.) 2018
Oono Suzuki 2020) and widely discussed (Chen et al. 2020aa Zhao Akoglu 2020 Rong al,, 2020 Chen al. [2220b] Liu al., 2020 Zhou et al. 2020 It is the phenomenon that the learned node representations become indistinguishable as the number the hidden layers increases, thus hurting the performance of down-streaming tasks like node classification and link prediction.
To tackle the oversmoothing problem of GNNs, some nascent research works are proposed (Klicpera et al. 2019 Chen et al. 2020a| Zhao & Akoglu 2020: Rong al. 2020: Chen et al. 2020b Liu et al. 2020; Zhou et al.) 2020). They share the same logic (i.e., keeping the divergence be- tween nodes) but differ in specific methodologies (i.e., rescaling divergences of learned representa-

---

Under review as a conference paper at ICLR 2022
tions (Zhao & Akoglu 2020), adding the divergence regularizer in the learning process (Chen al.
2020a Zhou et al. 2020). changing input graph structures (Chen et al. 2020a Rong et al.) 2020 Chen al. 2020b), or personalizing the information aggregation for each specific node (Klicpera
et al. 2019 Liu et al. 2020). Despite of good performance, some drawbacks still exist in those mentioned solutions. By surveying these SOTA de-oversmoothing strategies, we summarize three major metrics to evaluate a de-oversmoothing strategy: 1) constant divergence indicator, 2) easy- to-determine divergence indicator, and 3) model-agnostic de-oovrsmoothing strategy. (The detailed discussion could be found in Section 2). We find that no prevalent de-oversmoothing methods for GNNs could maintain all of them.
To bridge this gap, we propose a Topology-guided Graph Contrastive Layer (TGCL) with the inspiration from the contrastive learning concept (van den Oord et al.) 2018), where we contrast node topological information to obtain discriminative node representations after many GNN layers. TGCL is the first de-oversmoothing strategy attempting to maintain all three mentioned metrics. Specifically, we set constant and easy-to-determine divergence indicator between nodes, which is purely based on the topology of the input graph. With this divergence indicator, we aim to guide latent representations of neighbor node pairs closer and non-neighbor node pairs farther apart to mitigate the oversmoothing of GNNs. Last but not least, the proposed TGCL is -agnos-ag,ostic, which means TGCL could be incorporated in multiple GNN models. With theoretical proof and empirical analysis, we show that the proposed TGCL. could alleviate the oversmoothing problem of GNNs to large extent.
Our contributions can be summarized as follows:
We survey current de-oversmoothing methods by analyzing the advantages and the dis- advantages of each method and summarize three metrics to evaluate a de-oversmoothing method for GNNs.
We propose topology-guide guideed graph contrastive layer named TGCL to tackle the over- smoothing problem of GNNs, which enjoys all three metries simultaneously. We show the effectiveness of the proposed TGCL. from the theoretical proof and the em- pirical aspect with extensive experiments.
The rest of this paper is organized as follows. After a brief survey of de-oversmoothing methods in Section 2, we introduce the proposed TGCL in Section The empirical evaluation of the proposed TGCL on real-world datasets is presented in Section 4. Then, we review the related work in Section 5 before we conclude the paper in Section
BACKGROUND
As mentioned above, de-oversmoothing methods Kliccera et al., 2019 Chen et al.) 2020a Zhao &
Akoglu 2020 Rong et al.) 2020 Chen al. 2020b; Liu et al. 2020 Zhou et al. 2020) share the same logic of keeping the divergence between node representations but differ in specific methodolo- gies focusing on different merits. By taking the union of the metrics used in different state-of-the- arts, we get three metrics to evaluate a de-oversmoothing algorithm comprehensively.
There three metrics shown in Table 1 including constant divergence indicator, easy-to- determine divergence indicator, and model-agnostic de-oversmoothing strategy. Divergence indi- cator is indispensable for guiding the final node representation similarity based on the specified dis- tance measurement. Several de-oversmoothing methods like (Klicpera et al. 2019 Zhao & Akoglu 2020 Chen al.) 2020b; Liu et al. 2020) achieve the constant divergence indicator, which means the guidance is much more robust and not dependent on the training process of GNNs. However, to guide the node representation similarity reasonably, the divergence indicator is not that easy to be determined. For example, PairNorm (Zhao & Akoglu, 2020) is proposed as normalization layer to keep the divergence of node representation against the original node feature. Instead of adding this regularizer directly to the learning objective of GNN models, PairNorm takes an alternative by rescaling the learned node representations with a constant hyperparameter to keep the original node feature divergence. PairNorm achieves two metries: constant divergence indicator (i.e.. the constant hyperparameter) and model-agnostic strategy (i.e., PairNorm can be added on different GNN models as a layer). However, the selection of that constant hyperparameter heavily depends
2

---

Under review as a conference paper at ICLR 2022
Table I: Comparison of current de-oversmoothing methods
Constant
Easy-to-Detene Determine
Model-Agnostic
Divergence Indicator
Divergence Indicator
Strategy
APPNP (Klicpera et al.
2019
Yes
Yes
MADReg Ade (Chen al. 20/20a
No
Not Sure No
Yes
PairNorm Zhao & Akoglu/2020
Yes
Yes
DropEdge [ong et al. 20220)
No
No
Yes
GCNII Chen et al. 2020b
Yes
Yes
No
DAGNN (Liu et al. 20200
Yes
No
Yes
DGN (Zhou cl al. 2020
No
No
Yes
TGCL (Our Method)
Yes
Yes
Yes
on the prior knowledge of the input graph data, which is hard to determine. (The discussion of other de-oversmoothing methods can be found in Section 5.)
As shown in Table | PairNorm is an effective de-oversmoothing method that maintains two metrics but needs prior knowledge to scale divergence between node pairs. While our proposed TGCL transfers this hard-to-acquire prior knowledge into the topology information of the input graph, where the divergence guidance between nodes constant and easy to be determined. To be specific, our TGCL is the first de-oversmoothing method attempting to maintain these three metrics at the same time. In the next section, we formally introduce the proposed TGCL with theoretical proof for the model effectiveness. Moreover, we prove that the objective of PairNorm is just a special case of our TGCL, which shows the effectiveness of our TGCL from another perspective.
3 PROPOSED METHOD
In this section, we begin with the notions used this paper. Then, we prove that the objective the de-oversmoothing model PairNorm (Zhao & Akoglu 2020) is just special case of our Topology- guided Graph Contrastive Layer (TGCL). After analyzing the limitations of PairNorm, we formally introduce proposed TGCL and show why it could better alleviate the oversmoothing issue with the contrastive learning manner.
3.1 NOTION
Throughout this paper, we regular letters to denote scalars (e.g., a)), boldface lowercase letters to denote vectors (e.g., v), and boldface uppercase letters to denote matrices (e.g., A). We formalize the graph mining problem in the context of undirected graph of n vertices, E consists of m edges, X E Rxx denote the feature matrix and d is the feature dimension. We let E Ra denote the adjacency matrix, D E "x"" denote the diagonal matrix of vertex degrees, and I E " denote the identity matrix. For ease explanation, we denote v; as node IG as the input feature of node i, 2; as the embedding of node by any type of GNNs and A; the adjacency vector for node i. .N. is set that contains the neighbors of node and N. is the complement of Ng, which contains the non-neighbor of node
(V, EX), where V consists
3.2 PRELIMINARY
Each graph convolutional layer can be understood as smoothing operation (Li et al. 2018) but stacking many layers renders the final representation a node indistinguishable from others. There- fore, how recover the divergence between node representations but preserving the shared informa- tion becomes a vital problem in graph mining. In PairNorm (Zhao & Akoglu (220). the divergence between node pairs is based on a hyper-parameter, which requires prior knowledge of the input graph data and is hard to acquire. More specifically, PainNorm is proposed as novel normalization layer prevent all node embeddings from becoming too similar by minimizing the following objective:
Lp
Σ |z - 12,
11 21
|z, z (1A))E
Σ
(1)
V.E4
(LJ)ER
where z; is the node embedding vector of node v; and x; is the original feature vector of node v, In the equation above, the first term the reconstruction error, the second term is responsible for min-
3

---

Under review as a conference paper at ICLR 2022
imizing the difference between two representations of a neighbor node pair, and the last term aims to maximize the difference between two representations of a remote node pair. By reformulating D we could derive upper bound of Lp in the form of contrastive learning loss term as follows:
Eq.
Lp = Σ
2211 +
Σ
2z 2z11²
Σ
Σ
||, 2
NEV
BEV
By
EN;
V.-VV
VLRNN Σ
Σ
|| - wz 112-
Σ
Σ
log(e-11z,--2)
|| 11,1-
log(e Bog(e-1l=:--0 =1:
(2)
V
ByEAG
V.~V
|| x; ||2
Σ
Σ
log(e ||=--;;11²
Σ
log(
Σ e --12z-z211f
(3)
V
uyE.N.
v
|| x/
Σ
log(e ||
||=;- 1og(e-11z:-<5)))
+ v.~V
Σ
Σ
log(
Σ e el
V
V ByENG
V,EN
log( En E 6- 121-=111
Σ Vi
|| xz
Vi-V MyEN;
[log(1 + EvEEN, e e- --
||
xill 1 +
V
Vi~V VBENN;
e 1112 c ==;
Σ V
2| xz 112
Σ
logg(
EnegN, c -
Be MyEN
f(21,
Σ V
||| xzll
[log(
)1 L
(4)
[ zj)
Em,eN, [ Zk.)
Ve-V MyENN
where fz,, 2k) - Here, we apply Jensen's 's inequality derive Eq. 3 as a upper bound of Eq 2 since log(.) is concave. We observe that L1 is a upper bound of PairNorm and we could interpret two regularization terms |z 22 and ||z; z l of PairNorm as a special case of contrastive learning loss term in I, by setting the similarity measurement function f(z;, z1.) to be
=
However, both PairNorm (Eq. D and the upper bound of PairNorm (Eq. 4 only consider the first- order neighhor information but neglect the KC-hop neighbors information. For example, in real- world scenario, we are given a remote pair (Uk, v.). It is highly possible that Vù and vs have the similar representations, if they share the same label information. However, simply minimizing the third term of PairNorm (i.e. -||z; =x will push z; away from zk. resulting in sub-optimal solution. In addition, if we are given two remote pairs (Vk, 2x)) and (U),, e2) such that node 2k, is far from node and node Uk., is near node v; (e.g., 2-hop neighbor), the weight imposed on these two remote pairs should be different as we expect that za, should be more different from z; than 2bg due to the topological information in the graph. However, PairNorm and L2 (Eq. 4 assume that all unconnected node pairs (z; and 2τ) have the same weight by setting the weights to be I for neighbor pairs and remote pairs. Therefore, if the Ko-hop neighbors of z, share the same topological structure of 2; or the same label information, pushing 2, away from the representation of its K--hop neighbors (K 1) and ignoring the different weights for different remote pairs will result in sub-optimal solution. Motivated by these, we propose to utilize the similarity of two adjacency vectors of each node pair and embed the global topological structure information into the representation of each node such that GNNs can derive better discriminative representations for all nodes.
3.3 OVERVIEW OF TGCL
The structure of TGCL is shown in Figure TGCL is model-agnostic, and it can be added before the final output layer of any GNN model. To recover the divergence between node representations, we first need to determine the divergence between different node pairs. In TGCL, we transfer this hard-to-acquing knowledge into the topology information of the input graph, which is constant di- vergence indicator (i.e., not varying with the depth of GNNs), easy to obtain, and purely depends on the adjacency vector of each node. The main idea of TGCL. is encode the topological divergence
4

---

Under review as a conference paper at ICLR 2022
Topolagy
guided Graph Contrastive Layer
(TGCL)
Hidden Layer
Hidden Layer
Hidden Laver
Hideln Layer
Hidden
Input Graph Layer
Layer
Outaut Layer
An Arbitrary GNN
Figure 1: An arbitrary graph neural network with the proposed model-agnostic TGCL.
relationship of any pair of nodes into their final node representations. Specifically, we expect that the representations of two nodes are similar if their adjacency vectors close enough. Otherwise, their representations should be discriminative. Thus, we propose the topology-guided contrastive loss formulated follows:
Oig (Zi,
LTGCL B R Re, EN [log
oij fzi 2j) ) EOLENT Yik fzi, 2x)
(5)
01 - dist( A A,)/n, Vss dist( A Au)/n
where f()) is a similarity function, e.g., f(a, = exp(=="))
the temperature, dist(.) a
distance measurement function, e.g., hamming distance, and the set N₁ contains the non-neighbor nodes of the node The intuition of is that if v; and vj are neighbors, then the similarity of their representations should be as large as possible, while if v; and Uk are two remote nodes (not connected in the graph), the similarity of their representations should be as small as possible, and the magnitude of dissimilarity is determined by how many neighbors these two nodes don't 't share. By adjusting the weights of both remote pairs and neighbor pairs based on the topological information, aim to reduce the negative impact of remote nodes that have similar topological information.
To collaborate with different GNN models, the adaptive loss function Liotal expressed as follows.
Liotal Lugnostic + aLTGCL
where Lagnostic denotes the loss function of an arbitrary GNN model such as GCN (Kipf & Welling 2017), and a constant hyperpararmeter and LTGCL stands for the loss function of our TGCL, which can serve regularizer to alleviate the over-smoothing problem.
3.4 THEORETICAL ANALYSIS OF TOPOLOGY-GUIDED CONTRASTIVE LOSS
In this subsection, we provide an analysis regarding the properties of the proposed contrastive loss.
Lemma Given neighbor node pair sampled from the graph G (V, E, Dú we have 1(z, zz),,,) -LTGCL + Eu,-y log( [Na), where [ 2j) is the mutual information be- tween two representations of the node pair v; and Uj and LTGCI, is the topology-guide contrastive loss weighted by hamming distance measurement.
E, X), i.e., nodes e; and
Following the theoretical analysis in fyan den Oord et al. 2018). the optimal value of Pls, zi), Thus, the weighted supervised contrastive loss could be rewritten as f(z,,2) is given by P(z,)
Proof:
follows:
Tig f21,
LTGCL -B Ee [log
J. f 2j) ) + EniEN; Nix I zx)
P(2, 0i P(x,)
+ +EUENN Vik P(*n)
P(zn(z()
Eo
p(z)(())
Oij P(z;)
P(z,)
P(zzlzi)
= =Bn,vVBoseN [log(1
+
0 P(zz(zz)
V.EN; Σ
"ik
P(zz)
5

---

Under review as a conference paper at ICLR 2022
Since (vi, vk) is defined remote node pair, it means that node vs and node vk connected in the graph, i.c. A(i, k) = A(k, i) = 0. Therefore, we have " E (1,2) for all remote nodes
VR and Djj (0, 1] for all neighbor nodes v with hamming distance measurement, which leads to Pz)) P(z;) and Yik P(zz] 2 P(zulz.) Thus, we have P(LL)
P(z,) 2 [
P(z,)
P(zalzi)
LIGCL 2 Ev [log(
Σ
P(zj(zz)
P(zx)
UNEN:
P(z₂)
P
ENN [log(
P(zx) ))
P(zj
zi)
P(zy)
-B (
[Wi)]]
P(z)|zi)
P(zy)
Ee, E.N. [log(
P(zj(zi) log([<V
1121, 2,) + E E log( (Vg])
Finally, we have 1(z 21) 2 LIGCL + E log(l which completes the proof.
Lemma shows that the copology-guidet contrastive loss for the graph is the lower bound the mu- tual information between two representations of a neighbor node pair. Notice that E log(l the average logarithm of the number of unconnected edges for the nodes the graph, which means that TGCL tends to be better lower bound if imposed on sparser graph.
EXPERIMENT
In this section, demonstrate the performance of our proposed framework in terms of effectiveness by comparing it with state-of-the-aa methods. In addition, we conduct a case study to show how the increase of the number of layers influences the performance of the proposed method.
4.1 EXPERIMENT SETUP
Datasets: Cora dataset is citation network consisting of 2,708 scientific publications in seven classes and 5,429 edges. The edge in the graph represents the citation of one paper to another. CiteSeer dataset consists of 3.327 scientific publications which could be categorized into six classes and this citation network has 9,228 edges. PubMed a diabetes dataset, which consists of 19,717 scientific publications in three classes and 88,651 1 edges. Reddit dataset is extracted from Reddit posts in September 2014, which consists of 4,584 nodes and 19,460 edges. In all experiments, we follow the splitting strategy used in (Zhao & Akoglu (2020) by randomly sampling 3% of the nodes as the training samples, 10% of the nodes as the validation samples, and the rest 87% of the nodes the test samples.
Baselines: We compared the performance of our method with the following baselines: (1) GCN (Kipf & Welling 2017): vanilla graph convolutional network: (2) GCNII (Chen et al. 2020b): an extension of vanilla GCN with skip connections and additional identity matrices; (3) DGN (Zhou
al. 2020): the differentiable group normalization normalize nodes within the same group and separate nodes among different groups; (4) PairNorm (Zhao & Akoglu 2020): a novel normaliza- tion layer designed to prevent all node embeddings from becoming too similar, (5) DropEdge (Rong et al. 2020): novel framework that randomly removes certain number of edges from the input graph at each training epoch to reduce the speed of over-fitting and to prevent the oversmoothing issue. The reason why do not include the de-oversmoothing strategies APPNP (Kliepera al. 2019) and DAGNN (Liu et al. 2020) here is that they replace stacking layers with stacking hops propagation.
Configuration: In all experiments, we set the learning rate to be 0.0005 and the optimizer is Adam (Kingma & Ba 2014). The feature dimension of the hidden layer is 50. The experiments repeated 5 times if not specified. dist(.) is the hamming distance and f(-) is the cosine similarity measurement. All of the real-world datasets are publicly available. The code of our algorithms could

---

Under review as a conference paper at ICLR 2022
Table 2: Accuracy on node classification four benchmark datasets.
Method
Cora
CiteSeer
PubMed
Reddit
GCN
0.6707 + 0.0519
0.5578 + 0.0153
0.7984 = 0.0077
0.7537 = 0.0167
PairNorm
0.7178 + 0.0064
0.5628 + 0.0187
0.7816 + 0.0087
0.7592 = 0.0069
DropEdge
0.7138 + 0.0186
0.5330 + 0.0310
0.8063 + 0.0128
0.7539
0.00
GCNII
0.7179 0.0012
0.5913 + 0.0050
0.8035 = 0.0011
0.7503 = 0.0008
DGN
0.6896 + 0.0035
0.5190 0.014
0.7929 = 0.0018
0.7407 = 0.0321
TGCL
0.7199 + 0.0151
0.5783 =: 0.0191
0.8090 = 0.0065
0.7556 = 0.0132
Res+ResNet
0.7453 + 0.0097
0.6139 + 0.0197
0.8127 + 0.0080
0.7998
0.0181
PairNorm-ResNet
0.7454 + 0.0327
0.6054 + 0.0203
0,8010 = 0.0086
0,8040 = 0.0101
TeCL+ResNet
0.7699 1 0.0113
06125=00011
0.8192 = 0.0013
0.8106 = 0.0118
be found in an anonymous link 16GB RTX 5000 GPU.
The experiments are performed on a Windows machine with a
4.2 EXPERIMENTAL ANALYSIS
In this subsection, we evaluate the effectiveness of the proposed method on four benchmark datasets by comparing it with state-of-the-art methods. The base model for all methods we used in this ex- periment is graph convolutional neural network (GCN). For a fair comparison, we set the numbers of the hidden layers to be 10 for all methods and the dimension of the hidden layer to be 50. The experiments are repeated 5 times and we record the mean accuracy as well as the standard deviation in Table 2 By observation, we could find that proposed method outperforms most baselines over these four datasets without adding ResNet. Though GCNII achieves the best performance in the CiteSeer dataset, it has worse performance in other datasets. When we further incorporate ResNet into the base model (GCN), PairNorm, and our proposed method, we observe the perfor- mance improvement for all methods. In addition, the gap of performance between GCN+ResNet and TGCL+ResNet becomes narrow. Our guess is that as we increase the number of layers, the vanishing gradient problem and the oversmoothing issue coexist GCN based model. Adding ResNet into the base model somehow alleviates the issue of vanishing gradient, thus leading to great performance improvement for GNNs.
To further investigate the impact of oversmoothing issue, we conduct an experiment on the Cora dataset by increasing the number of layers. The x-axis of Figure 2 (a) is the number of layers and the y-axis is the accuracy ResNet, the performance of GCN drops dramatically starting at 10 hidden layers. By comparing the performance of GCN and TGCL, we observe that after utilizing our proposed de-oversmoothing strategy, TGCL boosts the performance by more than 7.5% at 60 hidden layers. After adding ResNet, the performance of GCN+ResNet improves lot due to the alleviation of the vanishing gradient problem, but we could still see more than 4% improvement at 60 hidden layers by our proposed method (TGCL+RReNNet), Combining the experimental results in Table 2 and Figure (a), we find that the oversmoothing issue slightly influences the performance of shallow GCN (when the number of layers less than 10). However, GCN will suffer a lot from the oversmoothing issue, if we increase the number of layers to 20 more (based on the results in Figure 2 (a)). To demonstrate the oversmoothing issue in other types of GNNs, we show the performance of our proposed with different base models (e.g., GAT (Velickovic et al., (018)) and SGC (Wu et al.) 2019))
the test dataset. By observation, we find that without adding
in Figure 2 (b). The experimental setting is the same as the setting in Table 2 Figure 2 (b) shows
that our proposed method outperforms GAT and SGC and thus alleviates the oversmoothing issue.
4.3 CASE STUDY: A MISSING FEATURE SCENARIO
Why do we need a de-oversmoothing strategy, if increasing the number of the layers may result in worse performance? 'To answer this question, let's first imagine a scenario where some values of
attributes are missing in the graph. In this scenario, the shallow GNNs may not work well because
https://drive, google le com/ fwwelll/aao 1ut39t9__5b5357vv htt vie ewwpuspssharing
7

---

Under review as a conference paper at ICLR 2022
200
0.8
101A HHOH
150
6.7
0/9
100
60
10
70 0.0
CheSeerr
PubMed
Redd1
10
38
43
(8)
(b)
(C)
Figure 2: (a) accuracy vs the number of layers on Cora dataset; (b) accuracy of different base models on four datasets; (c) running time (in second) vs the number of layers on Cora dataset. (Best view color)
Table 3: Accuracy of node classification four datasets by masking percent of node attributes. # denotes the number layers where a model achieves the best performance.
Node Feature Missing Rate
p 25%
50%
p 75%
Dataset
Method
Acc
#L
Acc
#L
Acc
#L
GeN+ResNet
0.731 + 0.009
3
0.729 + 0.010
11
0.688 + 0.018
11
Cora
eGCL+ResNet
0.732 0.010
0.751
= 0.016
15
0.717 = 0.031
30
GCN+ResNet
0.615 = 0.013
4
0.575 : 0.017
15
0.524 = 0.005
12
Citeseer
TGCL+ResNet
0.621 0.009
8
0.593 = 0.006
0.562
= 0.039
20
GCN+ResNet
0.807 = 0.006
3
0.783 = 0.005
11
0.737
= 0.002
8
PubMed
NesLe
0.820 0.002
0.823
= 0.005
0.805
= 0.005
20
GCN+ResNet
0.731 - 0.010
3
0.675
= 0.009
0.626 = 0.015
15
Reddit
ResL+R
0.727 0.017
3
0.725
0.010
0.653 = 0.008
GNNs could not collect useful information from the neighbors due to the massive missing values. However, if increase the number of layers, GNNs are able gather more information from the KC-hop neighbors and capture latent knowledge. To verify this, we conduct the following experiment: we randomly mask p% attributes in four datasets, gradually increase the number of layers, and report the performance. In this case study, the number of layers is selected from [2, 3, 4. 5, 6, 7, 8, 9, 10, 11, 12, 13. 14, 15, 20, 25, 30, 40, 50, 60] and the base model is GCN, For fair comparison, we add ResNet (He et al., 2016) avoid the vanishing gradient issue. We repeat the experiments three times and record the mean accuracy and standard deviation.
Table 3 shows the performance of TGCL as well as the number of layers where the model achieves the best performance (denoted as #L). By observation, we can see that when the missing rate is 25%, 3 layers or 4 layers GCN has enough capability to achieve the best performance in all four datasets and our proposed method only slightly improves the performance. However, when we increase the missing rate to 50% and 75%, we observe that both GCN and TGCL achieve the best performance by stacking more layers and our proposed method improves the performance of GCN by 6.8% in the PubMed dataset when 75% attributes are missing. The experimental results verify that the more values a dataset is missing, the more layers GNNs need to stack to achieve better performance. Our guess for this observation that if the number of layers increases, more information will be collected from the K-hop neighbors to recover the missing information of its 1-hop and 2-hop neighbors.
4.4 EFFICIENCY ANALYSIS
In this subsection, we conduct an efficiency analysis regarding proposed method Cora dataset.
We fix the feature dimension the hidden layer to be 50 and we choose GCN as the base model. We gradually increase the number of layers and record the running time. In Figure 2 (c), the x-axis the number of layers and the y-axis is the running time in second. We could see that the running time of proposed method is linearly proportional to the number of layers.
8

---

Under review as a conference paper at ICLR 2022
5 RELATED WORK
In this section, we briefly review the related work on the oversmoothing of GNNs and contrastive learning methods.
5.1 OVERSMOOTIING OF GNNS
Oversmoothing problem of GNNs is formally proposed by (Li et al. 2018) by demonstrating that node representations become indiscriminative after stacking many layers in GNN models. This problem is also analyzed by (Oono & Suzuki 2020) showing how oversmoothing hurts the node classification performance. To quantity the degree of oversmoothing, different mcasurements are proposed (Chen et al.] 2020a] Zhao & Akoglu 2020 Liu et al.) 2020, Zhou et al.) (020). For example, Mean Average Distance (Chen al., 2020a) is proposed by calculating the divergences between learned node representations. To tackle the oversmoothing problem of GNNs, some nascent research works are proposed (Klicpera et al. [2019] Chen et al., 2020a; Zhao & Akoglu 2020 Rong et al., 2020; Chen et al., 2020b; Liu al. 2020: Zhou et al. 2020). They share the same logic of keeping the divergence between node representations but differ in specific methodologies like adding the divergence regularizer in the learning process and changing input graph structures. Taking the union set of these methods' merits, we propose three metrics as shown in Table T to comprehensively evaluate de-oversmoothing method. For example, APPNP (Klicpera al.) (019) personalizes the information propagation for each specific node to tackle the oversmoothing problem. To be specific, APPNP uses the stationary distribution of random walks to propagate information, which is constant and not changing with the depth of GNN models. However, the number of power iterations to get the approximated stationary distribution is hard determine and its effect on alleviating the oversmoothing is not clear. Also, in MADReg (Chen et al.) 2020a), the divergence regularizer is built on the learned node representation, which is varying with the depth of GNN models, and may not be as robust as the constant divergence indicator. To the best of our knowledge, the proposed TGCL is the first dc-oversmoothing method attempting to maintain three metrics at the same time.
Chen
5.2 CONTRASTIVE LEARNING
Recently, contrastive learning attracts researchers' great attention due its prominent performance for unsupervised data. (van den Oord et al.] 2018) is one of the earliest works, which proposes a
Contrastive Predictive Coding framework extract useful information from high dimensional data with a theoretical guarantee. Based on this work, recent studies (Song & Ermon 2020 Chuang
al. 2020; Khosla al., 2020 Tian al. 2020 Chen et al., 2020ch reveal a surge of research interest in contrastive learning. Youu et al. 2020) propose a graph contrastive learning (GraphCL) framework utilize different types augmentations method to incorporate various priors and to learn unsupervised representations of graph data. (Qiu et al.) (020) propose Graph Contrastive pre- training model named GCC to capture the graph topological propertics across multiple networks by utilizing contrastive learning to learn the intrinsic and transferable structural representations. (Has-) sani & Ahmadi, 2020) aims to learn node and graph level representations by contrasting structural views of graphs. In this paper, leverage the topologiaa structure information contrast the node representations to maximize the similarity of two connected nodes and to minimize the similarity of two remote nodes.
CONCLUSION
In this paper, we first survey the current de-oversmoothing methods and take the union of their own merits to propose three metrics to evaluate de-oversmoothing method, i.e., constant divergence strategy, and model-
indicator, casy-to-determine divergence indicator, agnostic strategy. Then, we propose the Topology-guided Graph Contrastive Layer, named TGCL, which is the first neeoovrsmoo method maintaining the three mentioned metrics. With the con- trastive learning manner, we provide the theoretical proof of our proposed TGCL and demonstrate the effectiveness of the proposed method by extensive experiments comparing with state-of of the-art de-oversmoothing algorithms.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Mcasuring and relieving the over-smoothing problem for graph neural networks from the topological view. In The Thirty-Foure Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirry-Second Innovative Applications of Artificial Intelligence Con- ference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 3438-3445, AAAI Press, 2020a.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1725-1735. PMLR, 2020b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey F. Hinton. simple framework for con- trastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1607. PMLR. 2020c.
Ching- Yao Chuang, Joshua Robinson, Yen-Chop Lin, Antonio Torralba, and Stefanie Jegelka. Debiased con- trastive learning. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
William L. Hamilton, Zhitao Ying, and Jure Leskovee. Inductive representation learning large graphs. In Ad- vances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017 7, Long Beach, CA, USA, pp. 1034-1034, 2017.
Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-viww representation learning on graphs. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 41 4116-4126. PMIR, 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 770-778. 778. IEEE Computer Society, 2016.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Prucessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
1412v 69800 2014.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
Johannes Klicpera, Aleksandar Bojehevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Qimai Li, Zhichao Han, and Kiao-Ming Wu. Deeper insights into graph convolutional networks for semi- supervised learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI- 18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposiun on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 3538 3545. AAAI Press, 2018
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp. 338-348. ACM, 2020.
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classifi- cation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang. Hongxia Yang. Ming Ding. Kuansan Wang, and Jie Tang. GCC: graph contrastive coding for graph neural network pre-training. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA, USA, August 23-27, 2020, pp- 1150-1160 160. ACM, 2020.
10

---

Under review as a conference paper at ICLR 2022
Meng Qu, Yoshua Bengio, and Jian Tang. GMNN: graph markov neural networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5241 55200, PMLR, 2019.
Yu Rong. Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpps:://ppeeeeeeeee net/
Jiaming Song and Stefano Ermon. Multi-label contrastive predictive coding. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Yonglong Tian, Dilip Krishman, and Phillip Isola. Contrastive multiview coding. In Computer Vision ECCV 2020 16th European Conference, Glasgow: UK, August 23-28, 2020, Proceedings, Part XI, volume 12356 of Lecture Notes in Computer Science, pp. 776-794. Springer, 2020.
Aäron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, abs//807..tt 00011187.007777 00043,,8 2018.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018.
International Conference on Learning Representations, ICLR 2018, Van-
Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger, Simplifying graph convolutional networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 6861-6871. PMLR, 2019.
Keyulu Xu. Weihua Hu, Jure Leskovec. and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations. ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang. and Yang Shen. Graph contrastive learning with augmentations. Maria-Florina Balcan, and Hsuan- Lin (eds.). Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL 312033 0.t.://0........1......aaaaaaaa.
Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell,
Si Zhang, Hanghang Tong, Jiejun Xu, and Ross Maciejewski. Graph convolutional networks: comprehensive review. Computational Social Networks, 6( 6(1):1-23, 2019.
Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net 2020.
Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha. Rui Chen, and Xia Hu. Towards deeper graph neural networks with differentiable group normalization. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 12, 2020, virtual, 2020.


Reviews:
[{"review_id": "u2ED-h5I5r", "reviewer": "Reviewer_5qKE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\ni)\tIt surveys most of the existing works and provide three metrics to evaluate the oversmoothing problem in GNN.\nii)\tThe paper is well-written and easy to follow.\niii)\tIt provides the theoretical analysis of the proposed loss to show the soundness.\n\nWeakness\ni)\tAs there are lots of existing works such as APPNP, PariNorm, DropEdge etc. Why do authors focus on PairNorm to analyze boundaries? I cannot find the motivation that why choose PairNorm. \nii)\tThe novelty is limited. The authors analyze the boundary of PairNorm and propose the TGCL. \tWhere the majority contribution of this paper is the topology-guided contrastive loss. It is not clear that the meaning of Lemma 1. Why do we need such a lemma? I cannot find any relation between this lemma and the main content.\niii)\tTable 2 shows that the improvements are trivial. Compare with the results in GCNII, TGCL does not outperform it in all the datasets. Moreover, GCNII is outperformed TGCL in CiteSeer, and more stable in all those four datasets than TGCL. For example, GCNII got 0.7179 ± 0.0012 in Cora, TGCL got 0.7199 ± 0.0151. In my vision, TGCL only achieves 0.0002 improvements but introduces larger std to make the result unstable. \niv)\tWhy there are no experiments that use GCNII or other methods in Table 3?\n\n", "summary_of_the_paper": "This paper proposes a method named TGCL which aims to deal with the over smoothing problem in GNN when a number of layers go deeper. Different from existing works, the proposed TGCL considers three different concepts which are Constant\nDivergence Indicator, Easy-to-Determine Divergence Indicator, and Model-Agnostic Strategy. Most of the existing works are only considered two of them. The proposed method can be recognized as an improvement of the PairNorm.\n", "summary_of_the_review": "Overall, this paper provides a new way to define the oversmoothing problem in GNN. But it lacks the novelty and solid experiments. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jrLkIjk3b4m", "reviewer": "Reviewer_R5AE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. The proposed layer TGCL is model-agnostic. \n\n2. The paper is easy to understand.\n\nWeakness:\n1. Incorrect math derivations for upper bound. The inequality after Eq.(3) is incorrect. The term\n$\\sum_{v_i \\in N}\\text{log}(\\sum_{v_k \\notin N_i}e^{-||z_i-z_k||^2})$ can be negative. Thus, \n$\\sum_{v_i \\in N}\\text{log}(\\sum_{v_k \\notin N_i}e^{-||z_i-z_k||^2})$ can be larger than  $\\sum_{v_i \\in N}\\sum_{v_j \\in N_i}\\text{log}(\\sum_{v_k \\notin N_i}e^{-||z_i-z_k||^2})$ and the $\\leqslant$ does not hold. In this sense, the contrastive loss is not the upper bound.\n\n2. Non-rigorous/wrong math derivations for lower bound. In page 6, the proof for your Lemma 1, you use $\\approx$ in the derivations, which is not acceptable to derive a lower bound.\n\n3. No formal/math definition for \"constant divergence indicator\" and \"easy-to-determine divergence indicator\"\n\n4. The paper fails to explain why \"constant divergence indicator\" and \"easy-to-determine divergence indicator\" are a \"must\" for GNNs to overcome oversmoothing\n\n5. Fig.2 (a) only provides plots for weak baselines. GCNII and drop edge should perform much better.", "summary_of_the_paper": "The paper proposes a new layer TGCL for GNNs which is used to tackle the oversmoothing problem. This model satisfies three properties: constant divergence indicator, easy-to-determine divergence indicator, and model-agnostic strategy. They show their model alleviates the oversmoothing problem through experiments on 4 datasets.", "summary_of_the_review": "Overall, based on the weakness mentioned above, I think this paper has severe problems with math derivations and the contribution of this paper is also unclear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "ffbS4h1rLw", "reviewer": "Reviewer_wVaG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- the manuscript is well organized and easy to follow;\n- good analysis on K-hop study in section 4.3, which explains the intuition of the topology guided method;\n- the method is generic and easy to add to baseline GNN models which could serve as a contrastive loss extension;\n\nCons:\n- the three metrics that the author proposed to compare with existing methods are marginal put: most of the baseline model satisfied the third metric(model-agnostic) and as for the second metric: the TGCL method is claimed to be easy to adopt but the loss term definition includes extra parameters like temperature, choice of distance method, similarity function, alpha, etc. The experiment section didn't expand on this to show how the optimal configurations are chosen, given the margin of improvement in Table.2 these parameters might need to be analyzed;\n- Convergence speed compared to baseline methods are not mentioned in the paper especially when loss term is added in the learning process;", "summary_of_the_paper": "In this work, the author analyzed the oversmoothing issue in GNN learning. To compare with the related works, the author proposed three metrics: divergence indicator, the intuition to set its value, and the ease to adopt. The author proposed a new method Topology-guided Graph Conttrasive Layer which achieved de-oversmoothing and maintaining the proposed metrics. Lower bound analysis of the TGCL loss was analyzed and the empirical experiment showed an advantage on four graph datasets.", "summary_of_the_review": "The paper is well written and the method proposed is marginally novel. An extra support section for optimal configuration and convergence analysis would be a great plus to justify the advantage of the method. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZRMoAjk3bQD", "reviewer": "Reviewer_ax27", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nOver-smoothing has been an essential topic in GNN and seriously limiting the development of deep models. In this paper, a summary of the current techniques tackling over-smoothing on GNN is reduced, which could be beneficial for insightful GNN variants design in this community.\n\nThe proposed TGCL is a flexible plugin to include contrastive information on graphs, and the experimental results somehow evaluate the effectiveness of the proposal.  \n\nWeakness:\n\n1.    The literature review of current works tackling over-smoothing is not sufficient, e.g., different architectures of Skip Connections [1], subgraph sampling [2], and DropNode [3]. And the matrices are not clearly explained, studied, or followed in the following content. The formulations of these compared methods need to be illustrated.  \n2.    The discussion of current works is coarse, making the claimed contribution not grounded and the motivation of this work fuzzy. \n3.    The theoretical analysis of TGCL is not consistent with the experiments. For example, the graph sparsity is not studied as discussed in Lemma 1. And the computation cost of TGCL should be high, which should be compared.\n4.    The experiments are not well designed since the basic comparisons of the nascent methods are missing, and the \"repeated 5 times\" seems unreliable. The results are trivial.\n5.    The overall homophily should be explicitly claimed in the preliminaries or background.\n\nRef:\n\n[1] Xu, Kaidi, et al. \"Topology attack and defense for graph neural networks: An optimization perspective.\" arXiv preprint arXiv:1906.04214 (2019).\n\n[2] Zeng, Hanqing, et al. \"Deep Graph Neural Networks with Shallow Subgraph Samplers.\" arXiv preprint arXiv:2012.01380(2020).\n\n[3] Feng W, Zhang J, Dong Y, et al. Graph Random Neural Network for Semi-Supervised Learning on Graphs[J]. arXiv preprint arXiv:2005.11079, 2020.\n", "summary_of_the_paper": "This paper summarizes the current techniques tackling over-smoothing, where three matrices are first introduced to describe better and classify the techniques' characteristics. To this end, TGCL, a model-agnostic regularization term based on contrastive learning, is proposed to deal with over-smoothing, satisfying the three defining characteristics. ", "summary_of_the_review": "Based on the weaknesses of the paper pointed out above, I tend to reject this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new layer for graph neural networks that aims to reduce the oversmoothing issue common to this model type. The reviewers find the paper well organized and easy to follow, and they recognize the importance of the problem that is addressed. However, they also identify critical errors in the mathematical derivations: the authors did not provide a response to the reviews, and hence these errors remain unaddressed. In addition, multiple reviewers indicate they find the experimental evaluation insufficient. For these reasons I'm recommending rejecting this paper.
Decision: Reject