Graph Contrastive Learning with Model Perturbation
Authors: Qiaoyu Tan
Sirui Ding
Ninghao Liu
Soo-Hyun Choi
Li Li
Rui Chen
Xia Hu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Contrastive Learning
Model Perturbation
Graph Augmentation

Abstract: Graph contrastive learning (GCL) has achieved great success in pre-training graph neural networks (GNN) without ground-truth labels. The performance of GCL mainly rely on designing high quality contrastive views via data augmentation. However, finding desirable augmentations is difficult and requires cumbersome efforts due to the diverse modalities in graph data. In this work, we study model perturbation to perform efficient contrastive learning on graphs without using data augmentation. Instead of searching for the optimal combination among perturbing nodes, edges or attributes, we propose to conduct perturbation on the model architectures (i.e., GNNs). However, it is non-trivial to achieve effective perturbations on GNN models without performance dropping compared with its data augmentation counterparts. This is because data augmentation 1) makes complex perturbation in the graph space, so it is hard to mimic its effect in the model parameter space with a fixed noise distribution, and 2) has different disturbances even on the same nodes between two views owning to the randomness. Motivated by this, we propose a novel model perturbation framework -- \textsc{PerturbGCL} to pre-train GNN encoders. We focus on perturbing two key operations in a GNN, including message propagation and transformation. Specifically, we propose \emph{weightPrune} to create a dynamic perturbed model to contrast with the target one by pruning its transformation weights according to their magnitudes. Contrasting the two models will lead to adaptive mining of the perturbation distribution from the data. Furthermore, we present \emph{randMP} to disturb the steps of message propagation in two contrastive models. By randomly choosing the propagation steps during training, it helps to increase local variances of nodes between the contrastive views.  Despite the simplicity, coupling the two strategies together enable us to perform effective contrastive learning on graphs with model perturbation. We conduct extensive experiments on 15 benchmarks. The results demonstrate the superiority of \textsc{PerturbGCL}: it can achieve competitive results against strong baselines across both node-level and graph-level tasks, while requiring shorter computation time. The code is available at \url{https://anonymous.4open.science/r/PerturbGCL-F17D}.

Under review as a conference paper at ICLR 2023
GRAPH CONTRASTIVE LEARNING WITH MODEL PERTURBATION
Anonymous authors Paper under double-blind review
ABSTRACT
Graph contrastive learning (GCL) has achieved great success in pre-training graph neural networks (GNN) without ground truth labels. The performance of GCL mainly rely on designing high quality contrastive views via data augmentation. However, finding desirable augmentations is difficult and requires cumbersome efforts due to the diverse modalities in graph data. In this work, we study model perturbation to perform efficient contrastive learning graphs without using data augmentation. Instead of searching for the optimal combination among perturb- ing nodes, edges attributes, we propose to conduct perturbation on the model architectures (i.e., GNNs). However, it is non-trivial to achieve effective pertur- bations on GNN models without performance dropping compared with its data augmentation counterparts. This is because data augmentation 1) makes com- plex perturbation in the graph space, it hard mimic its effect in the model parameter space with a fixed noise distribution, and 2) has different dis- turbances even on the same nodes between two views owning to the random- ness. Motivated by this, we propose novel model perturbation framework PERTCRBGCL to pre-train GNN encoders. We focus on perturbing two key op- erations in GNN, including message propagation and transformation. Specifi- cally, we propose weightPrune to create dynamic perturbed model to contrast with the target one by pruning its transformation weights according to their mag- nitudes. Contrasting the two models will lead to adaptive mining of the per- turbation distribution from the data. Furthermore, turb the steps of message propagation in two contrastive models. By randomly choosing the propagation steps during training, helps to increase local vari- ances of nodes between the contrastive views. Despite the simplicity, coupling the two strategies together enable us perform effective contrastive learning on graphs with model perturbation. We conduct extensive experiments on 15 bench- marks. The results demonstrate the superiority of PERTURBGCL: it can achieve competitive results against strong baselines across both node-level and graph- level tasks, while requiring shorter computation time. The code available at 4oper.
present randMP dis-
INTRODUCTION
Graph neural networks (GNN) Kipf & Welling 2016a Hamilton et al. 2017 Gilmer et al. 2017)
have become the de facto standard model graph-structured data, such as social networks (Li
& Goldwasser 2019), molecules (Duvenaud et al. 2015). and knowledge graphs (Arora 2020).
Nevertheless. GNNs require task-specific labels supervise the training, which is impractical in many scenarios where annotating graphs is challenging and expensive (Sun et al., (019)) Therefore,
increasing efforts (Hou et al.) 2022. Veličković al. 2018 Hassani Khasahmadi] 2020 Thakoor
al.) 2022) have been made to train GNNs in an unsupervised fashion, so that the pre-trained model or learned representations can be directly applied to different downstream tasks.
Recently, graph contrastive learning (GCL) becomes the state-of-the-art approach for both graph- level You et al. 2020 2021; [Suresh et al., 2021 Xu et al. 2021) and node-level (Qiu et al. 2020
Zhu et al.) 2021b; Bielak al.) 2021; Tha Thakoor et al. 2022 tasks. The general idea of GCL is to
create two views of the original input using data augmentation (Jin et al. (2020). and then encode
them with two GNN branches that share the same architectures and weights (You el. 2020). Then,

---

Under review as a conference paper at ICLR 2023
the model is optimized to maximize the mutual information between the two encoded representations according to contrastive objectives, such as InfoNCE (Oord et al. 2018) or Barlow Twins (Zbontar 202T). As such, the performance of GCL mainly relies on designing high quality contrastive
et al.
views
/Zhng et al., 2021). Recently, intensive studies You et al. 2020 Jin et al. 2020 Han et al.)
2022
has been devoted to exploring effective augmentation strategies for graph data.
Despite their success, finding desirable augmentations requires cumbersome efforts, since the opti- mal augmentations are domain-specific and vary from graph to graph (You et al. 2020 Yin al.) 2022). To tackle this problem, SimGRACE (Xia el al. (022) introduced the idea of model perturba- tion. Instead of searching for the optimal combination among perturbing nodes, edges or attributes in the graph space, SimGRACE conducts perturbation in unified parameter space by adding Gaussian noise to model weights. However, we observe that SimGRACE may lead to sub-optimal represen- tations compared with its data augmentation counterparts because of two reasons. Firstly, the data augmentation in the graph space is rather complicated and beyond Gaussian distribution. As a result, the weight perturbation based on Gaussian noises cannot achieve similar effects data perturbation on representation learning (as illustrated in Section 2.2 p Secondly, the weight perturbation does not
consider local variances among different nodes in graph, since the perturbation is sata-agnostic. Therefore, it still remains an important yet unsolved challenge to develop effective model pertur- bation framework for GCL, so that it can produce effective representations both node and graph learning tasks in a more efficient manner.
To tackle these challenges, in this work, we propose novel framework PerturbGCL to train GNN encoders via model perturbation. Different from SimGRACE (Xia et al. 2022) that only focuses on weight perturbation, we make one step further to disturb the message passing (MP) of GNNs, since it allows to provide local disturbances between contrastive views. Specifically, we present weightPrume to construct a perturbed model by pruning the transformation weights of the target one. Unlike the Gaussian noise in SimGRACE (Xia et al. 2022), the pruned model will co-evolve with the target GNNs, leading to an adaptive mining of the noise perturbation from the data, i.e., data- driven. Furthermore, we propose randMP to offer local disturbances on nodes among contrastive views. It works by conducting times of message propagation steps in each contrastive model, where k is randomly sampled on-the-fly. Informally, performing MP k times can be thought of as conducting convolution on the anchor node's k-hops of neighbors (Gao et al., 20188). On this basis, can learn diverged but correlated representations from the two contrastive models with different k values due to the homophily theory (Altenburger & Ugander, (018). Coupling the two strategies together yields a principled model perturbation solution tailored for GCL, whose effectiveness and efficiency have been empirically verified through our extensive experiments. We summarize our main contributions as follows:
We introduce Perturbed Graph Contrastive Learning (PerturbGCI.), a principled contrastive learn- ing method graphs that works by perturbing GNN architectures. PerturbGCL. is flexible and easy to implement. To the best of our knowledge, PerturbGCL is the first model perturbation work that can achieve promising results on both node and graph learning tasks.
PerturbGCI. innovates to perturb GNN architectures from both the message passing and model weight perspectives via two effective perturbation strategies: randMP and weightPrune. By ap- plying the two strategies jointly in contrastive models, PerturbGCL can be learned to mimic the effect of data augmentation from the model perturbation aspect.
Extensive experiments across 15 benchmark datasets demonstrate the superiority of our proposal. Specifically, PerturbGCL can outperform state-of-the-art baselines without using data augmenta- tion across two evaluation scenarios. Moreover, PerturbGCL is easy to optimize and runs generally faster than the strong GCL baselines.
2 METHODOL.OGY
2.1 NOTATIONS AND PRELIMINARIES
Notations. Let G (V, E X) be undirected graph, where V> is the set of nodes and E is the set of edges. X E R|VI= is the node feature matrix where the i-th row of X denote the F-dimensional feature vector of the i-th node in V. We use f. denote the mapping function that encodes each node E 9 into D-dimensional representation h, E RDD
2

---

Under review as a conference paper at ICLR 2023
Graph Neural Networks. To learn representations graph data, use graph neural networks (GNN) (Kipf & Welling 2016a; Hamilton et al. 2017 Gilmer et al.] (017)) as the encoder fw- Without loss of generality, we present GNN as message passing network:
4 - o(a!-1)w) a! ((--1) = 9 (n ( :
where h. E R. is the intermediate representation of node at the /-the layer, No denotes the direct neighbors of node We use g to denote the message propagation (MP) function, which updates node representations by integrating its neighbors with a transformation function (Wu et al. 2019) w Dx is the transformation weight matrix and is the activation function, such as ReLO.
We often use the final layer's output as node-level representations, i.e., h, h where L is the number layers in a GNN. To get the graph-level representation h& E RDD for graph G, we further aggregate all node-level representations in graph via readout function:
hg READOUT((h) : E V)), 2g h(bb)) MLP(hg)
Here READOUT (-) can be the simple average pooling function more sophisticated ones (Ying 2018 Gao & Ji 2019). h(-) is the projection head, and Zg E RDD denotes the embedding towards loss estimation. In the development of our method, we follow the existing graph con- trastive learning practices and consider three state-of-the-ae GNNs: GCN (Kipf & Welling, 20)
et al.
GIN (uu et al. 2018), and ResGON (Chen et al. 2019).
Graph Contrastive Learning. Graph contrastive learning (GCL) (You et al. 2020) has become an state-of-the-ar approach for pre-training GNNs without ground-truth labels (1 (Liu al., 2021b a) Unlike reconstruction-base. methods (Perozzi et al.) 2014, Kipf & Welling 2016b). GCL is built upon contrastive objective between the so-called positive pairs and negative pairs generated from the original data. Formally, given an anchor node let (Z,; z,)) denote the representations of positive pairs and (Ze, z,))
be the negative pairs. The contrastive loss could be defined as:
LeL
1 V
Σ NEV
- llg
exp(sim(Zu- 2,t) 3/T)
(3)
exp exp(sim(zz z. 3/7) + EneV, exp(sim(z, Zn ZT)/T)²
where 7 the temperature parameter, sim( denotes the similarity function. By minimizing Eq. the GNN encoder will be trained to enforce the similarity of the positive pairs while enlarging the dis- tance of negative pairs in the hidden space. is also worth noting that some GCL variants (Thakoor
et al. 2022 Bielak et al. (202)) do not rely on negative samples. The key question in contrastive learning is how to generate effective positive (or negative) pairs. To this end, graph augmentation
has been adopted as the golden rule in GCL (You et al.] 2020 Jin et al. 2020). Typical graph augmentation techniques include edge perturbation, node masking and attribute masking.
2.2 THE PROPOSED PERTURBGCL FRAMEWORK
Motivation: How to perform model perturbation on GCL? Given a graph 9 = (A, X) where A is the adjacency matrix. For illustration purposes, consider the mapping function fw A, X; W) with one simple GCN (Wu et al. 2019) layer without activation function. Then, the hidden representation is computed by Jw, X; W) ((, X,W, where W is the weight matrix and g(:.:) -) is the message propagation operation defined in Eq. g(A, = D- (A+Ig) X.., (A
where D.; = (A AA+I)); i5 is the degree matrix and lg is the identity matrix. An intuitive solution is adding Gaussian noise to the model weight and derive two contrastive views J', (A, X; W) and fl (A W). as done in SimGRACE (Xia et al. 2022);
X = 9 f (A, (A P(W) ()p())
(4)
where (W)- = W 7/w is the perturbation function model weight, and Aw N'00 82) represents the noise term sampled from Gaussian distribution with zero mean and variance 8², and 77 is hyperparameter to scale the magnitude of the perturbation. Since the learning task is to minimize the distance between the two views, SimGRACE trains the model so that it is robust to the Gaussian noise in the weight.
3

---

Under review as a conference paper at ICLR 2023
Publled
Amazon
Coouthan-C8
CCA-355
DOA-568
0.28
0.30
0.88
SIGGRACE
0.24
0.40
0.22
0:38
0.M4
0.00
-20
-1.0
-1.0
-14
-2.00
-2.25
-2.00
-1.76
-1.00
-1.25
-2.0
-1.8
-1.6
-1.4
-1.2
Cuntee
Cantie
Figure 1: The alignmentt and uniformity performance (L) of SimGRACE and augmentation-basee GCL method (Thakoor et al. 2022) on the same perturbed graphs. Black circles indicate the results of baselines. Orange circles represent the results of SimGRACE. SimGRACE performs worse than standard GCL methods in modeling common perturbations achieved by data augmenta- tion. Detailed experimental setups are listed in Appendix B
However, we found that the contrastive views created by SimGRACE could be sub-optimal com- pared with its tata-augmentation counterparts (see Figure [.. To measure the quality of representa- alignment and uniformity (Wang &
tions learned by GCL models, we consider two popular metries:
Isola 2020 expressed as:
Caligo (Fwi a) A B(x.g)) lh by [2], > 0
(5)
Luniform (wwit) A log E
(a. [e [b, - 1 0
Ppos is the distribution of positive pairs, i.e., augmentations of the same sample, Paata is the data distribution. Calign is used to measure if positive samples stay close in the hidden space. Lunifeem is used to measure if random samples are scattered on the hypersphere of hidden space. In our experiments, we set a =
following (Xia et al. 2022).
Figure I reports the test results of SimGRACE and augmentation- based GCL methods (Zhang et al.) 2021] Thakoor et al.] 2022; Zhu et al.] 2020) on the same perturbed graphs created by random edge and attribute masking. Black circles and rrange circles (0) represent the performance of baselines and SimGRACE, respectively. As can be observed, SimGRACE performs worse than three baselines across different datasets with great margin. These results shed light the bottleneck of SimGRACE in capturing common perturbations achieved by data augmentation. This is because SiEGRACE is limited by the Gaussian noise and cannot handle perturbations in graph space. We thus ask: Can we design more advanced model perturbation strategies so that they can achieve similar functions as data augmentation done in GCL?
What is behind data augmentation in GCL? To answer this question, we start by analyzing the working mechanism behind the standard GCL framework. Let T(.) and q(-) denote augmentation functions on topology structure and node attributes, respectively. The two contrastive representations fl (A, X: W) and f (A, X; W) of standard GCL (You et al. 2020) are defined as:
f X: W) ((T fM (A. W) 9(T (A). ( (
(6)
That is, standard GCL framework learns be robust small disturbances (created by (T'(), (()-) and ((T" (+,aa ())) on the graph. We can easily observe two major properties of data augmentation as follows. O It can disturb different nodes in a graph (different graphs in a graph set) differently. This is because T'-)) and q(-) are random functions, such as edge masking, so they can have distinct effects even for the same input. @ The perturbation distribution incurred by data augmentation (e.g., (T' (r((.)'((-)) q()))) is complicated and often beyond Gaussian noise. As analyzed in Figure ] although SimGRACE is trained to be robust to Gaussian noise some extent, it cannot handle the perturbation in the graph space well. These properties motivate us to design tailored model perturbation strategies from the two aspects.
>> Our PerturbGCL proposal. To birdge the gap, we propose novel model perturbation frame- work called PerturbGCL. Figure provide the overview our framework. In this work, we build PerturbGCL upon the GraphCL pipeline (You al.) 2020), and follows its major components, such as two GNN branches and a non-linear projection head. The main difference is that GraphCL aug- ments the input graph to get two views and then process them with two branches that share the same GNN architecture and weights, while PerturbGCL processes the original input graph with two non- symmetric GNN branches. One branch uses the original GNN model fw. while the other branch
4

---

Under review as a conference paper at ICLR 2023
Message propagation
Weight prune
Projection head
I'' times (A, (A,,,k',
Maximize agreement
AO
E. W m.)
Input graph: G (A.X)
Projection
GNN encoder
head
Embeddiig
may
Figure 2: The overview of the proposed PerturbGCL framework. The original graph is fed into two asymmetric GNN branches: one is the target encoder fw to be trained, and the other the perturbed version l that is pruned from the former online. The two branches share weights for their non- pruned parameters. Either branch has independent message propagation (MP) operations perturbed by random number, i.e., k. to disturb nodes locally. Since the pruned branch is always obtained and updated from the latest target model, the two branches will co-evolve during training.
disturbs the message propagation process and model weights of fw. We conduct perturbation on two major GNN operations, including message passing (MP) and transformation. On this basis, we introduce the following simple yet effective perturbation strategies: randMP and weightPrune, towards effective model augmentation for GNN architectures.
Strategy #1: weightPrune. Recently, model pruning has attracted increasing attention for model compression thanks to the popularity of the lottery ticket hypothesis (Frankle & Carbin 2018). Work (Chen al. 202T) found that GNN can be pruned to a sparse sub-network without significant per- formance drop using rewinding techniques. These observations indicate the latent of pruning practical perturbation approach to GNN's weights. Inspired by this, we propose weightPrune, which creates the perturbed branch by pruning the model parameters of the target encoder. Specifically, assuming W denote the weights the target branch, mw be the mask the pruned branch, which has the same size W. At each iteration, we prune the target branch according to a pre-defined prune ratio 8 according to the magnitude of weight values, i.e., masking weights out if their magni- tudes are ranked below 8. By changing 8, we control the distortion degree of the target branch to a certain extent. After that, the target and perturbed branches will use W and W @ m. respectively as model weights generate representations, which are then fed into the contrastive loss. Since the mask indicator m. is continuously updated from the latest target model, the two branches will co-evolve during training.
Strategy #2: randMP. Message passing is another critical component of the GNN architecture, since it offers the flexibility aggregate features from multi-hop neighbors. Performing MP for times over the graph G is equivalent updating node v's representation based on its k-hop local subgraph. In light of this, & could be naturally regarded as perturbation factor, where different k values generate diverse but semantically correlated representations for the same node. To im- plement randMP, we will randomly sample two k values at each iteration: one is for the target branch (i.e., k') and the other for the perturbed branch (i.e., k²). Formally, if we assume ((A,X) (D } (A + Ig) Xx, In
D (A + D X.. performing MP times gives (AA experiments, we consider sampling k during the training because it may enforce the GNN encoder to learn generable representations invariant to different combinations of local enclosing graphs.
To sum up, different from existing methods (Xia et al. 2022 Thakoor et al. 2022) that disturb model weights with Gaussian noise, we suggest a principled approach to effectively perturb GNN architectures from their message propagation and feature transformation perspectives, via two sim- ple perturbation techniques: randMP and weightPrune. randMP aims to map the same input graph into two semantically similar representations by conducting a random number of message passing steps. Meanwhile, weightPrune targets to increase the diversity of two representations via model pruning. Combining the strategies enables spot the sweet point between the view
5

---

Under review as a conference paper at ICLR 2023
representations (Tian et al. 2020), i.e., correlated but diverged enough. The complete optimization procedure our model is outlined in Algorithm and in Appendix.
2.3 MORE DISCUSSIONS ON PERTURBGCL
PerturbGCL is complementary to other GCL efforts. We focus on training GCL by only using model perturbation. It can be easily combined with existing contrastive learning advances, such as mature graph augmentation techniques YYou al. (020) and the negative-sample free contrastive
loss objectives (Bielak et al. 2022 Thakoor et al. 2022). as we will show in Section 3.4
Computational complexity analysis. addition to saving a lot of time searching for the optimal
data augmentation strategies, we analyze the complexity of PerturbGCL. Given a graph 9 = (V. E) and the GNN encoder fu- The time complexity for most popular GNN architectures (Kipf & Welling 2016a Veličković et al. 2017; Gilmer et al. 2017 is O + (V), where (( and
[[[V]] are mainly caused by the message propagation and feature transformation operations, respec- tively. PerturbGCL performs two encoder computations per update step (one for each GNN branch) plus a node-level projection head. Assuming that the backward pass to be approximately as costly as forward pass and ignoring the cost for weight pruning it small and negligible. Thus the total time complexity per update step for PerturGGCL is 4Cenculer K E| + [V|s) + 2Chead ((V)) Class. where C, are constants depending on architecture of the different components, K the maximum number of MP operations considered (e.g., K = 3), and is the pruning ratio. It is worth noting that forward pass, due weight pruning
although our model at most takes K times MP operations (e.g., 70%), the computation costs for feature transformation and backpropagation are signifi- cantly lower than standard GCL methods. Therefore, the total running cost of PerturbGCL, can be further accelerated in practice. We empirically analyze the efficiency of our model in Section 3.5
3 EXPERIMENTS
In this section, we evaluate the performance of PerturbGCI. Specifically, we first visualize the weight distribution and the alignment between positive pairs in Section 3.1 to investigate what the proposed two strategies actually do. Then we evaluate the effectiveness of PerturbGCL in node classification with several benchmarks and SOTA baselines in Section 3.2 Next, we test the perfor- mance of PerturbGCL in graph classification under both unsupervised and semi-supervised settings in Section 3.3 After that, we evaluate the contributions of different components in Section 3.4 well how it improves the efficiency of GCL against standard methods in Section 3.5 The experiment setting and more experiments are summarized in Appendices experiments, the main observations highlighted.
A and Through the
25-
0044
1.00
2.00
0.
8.20
0.04
0.0
082
000
2.04
0.85
0.01
022
022
003
Figure 3: Visualization of weight distribution (from left to right: initial weights, PerturbGCL w/o. weightPrume, and PerturbGCL) Coauthor-Phy. The x-axis indicates weight values and y-axis the count. Obviously, the number of activated neurons after using weightPrune significantly smaller than others. It shows that weight can regularize the model.
3.1 WHAT ARE weightPrune AND randMP DOING? CASE STUDY
We visualize the weight distribution of PeruturbGCL during training in Figure 3 and Figure g (in Appendix). It shows that ① by continuously pruning the target model along the training, weight- Prune can regularize the target model progressively. From the weight histograms in Figure 3 can see that the bars around zero become higher and higher, which indicates more neurons are inac- tivated in the end. This observation shows the regularization effect of weightPrune. Since effective

---

Under review as a conference paper at ICLR 2023
Publled
Amazon Photo
Coouthrn-S
1,000
Perarbed
Purturbe
0.1175
Forturbed graph
0.16
0.150
0.14
0.300
0.12
0,1100
0825
0.1075
0.08
0.75
Figure 4: The visualization of PerturbGCL w.r.t different k values on the original graphs and the perturbed graphs generated by data augmentation. The x-axis indicates propagation steps and y- axis is the Lalign 4- The gap between the blue and orange lines indicate the generalization ability. Apparently, performing more MP steps will increase the diversity of two positive views since Calign increases, Sweet spots (i.e., minimum performance gap) exist across three scenarios.
Table I: Test accuracy on benchmark datasets in terms of node classification. We report both mean accuracy and standard deviation. A.R. denotes the averaged rank.
Method
Cora
PubMed
Compute
Photo
CS
Phy
J.R.R
GCN
81.5
79.0 79.0
86.51 + 0.54
92.12 + 0.22
93.03 + 0.31
95.65 0.16
8.33
Supervised
GAT
83.0 + 0.7
0.3
86.93 + 0.29
92.56 + 0.35
92.31
0.24
95.47
0.15
8.00
Raw Features
17.9 + 0.4
69.1
0.3
73.81 + 0.00
78.53 + 0.00
90.37
0.00
0.00 91.77
93.58
14.33
DeepWalk
70.7
0.9
89.44
0.22
GAE
71.5 1: 0.4
72.1
1 0.5
85.27 1 0.19
91.62 1 0.13
90.01
0.71
94.92
1 0.07
12.50
DGI
$2.3 +
76.8
0.6
+ 0.47
0.22
02.15
94.51
0.52
11.00
MVGRL
83.5
0.4
80.1
0.7
87.52 0.11
91.74
0.07
92.11
95.33
1133=000 0.03
7.50
GRACE
81.9 1:00:000 0.4
80.6 + 0.4
89.53 - 0.35
92.78
91.12
6.40
GCA
83.4 0.3
80.3 0.4
87.85
0.31
0.09
0..10
96.68
5.33
Unsupervised
BGRL
81.8 = 0.3
80.1 0.5
89.68 = 0.31
92.87
+ 0.227
93.21 0.18
0.12
4.83
GBT
1
92.63
0.11
InfoGCI.
83.5 +: 0.3
79.1 1 0.2
6.00
CCA-SSG
84.2
81.6 0.4
88.74 + 0.28
93.14
93.31 0.22
95.38 0.06
3.17
AFGRL
82.3 0.4
79.7
0.2
89.88
0.33
93.22
0.28
93.27 0.17
0.100
3,83
SimGRACE
78.5 1: 0.3
79.3
0.5
86.42 + 0.35
91.55 1: 0.22
92.37
0.33
94.37 0.15
10.67
PerturbGCL. 83.3 1 0.5 82.10 1. 0.37
88.45
1 0.77
93.62 1 0.40 94.18 0.09 95.85 0.08
2.33
regularization can improve the generalization ability of neural networks (Scholkopt & Smola 2018 believe that why the proposed weightPrune improve the performance.
To investigate the effect of randMP, we report the impacts of different & values on PerturbGCL in terms of the alignment between positive views. From Figure 4 we observe that @ randMP can improve the diversity of contrastive views when l: increases, and sweet points widely exist across three datasets. In Figure 4 with the increase of ks, the generalization gap tends to first decrease to the sweet points and then increase a little bit. It validates the effect of randMP in generating correlated but diverged views. the other hand, it indicates the potential of randMP to improve the generalization ability, i.e., these sweet points.
3.2 CAN PERTURBGCL. PERFORM WELL ON NODE CLASSSTFICATION TASK?
We first examine the effectiveness PerturbGCL in node classification. Results of 15 baseline methods across 6 benchmark datasets are collected in Table D We make the following observa- tions: & PerturbGCI. can achieve better node classification results than SOTA GCL methods without using data augmentation. From Table [ PerturbGCL gains 4 best performances among 6 evaluation scenarios. On average, it ranks 2.33 among 13 augmentation-baat strong methods, such BGRL and CCA-SSG, which indicates the power of model perturbation based contrastive learning. Meanwhile, ① PerturbGCL outperforms the model perturbation baseline SimGRACE with great margins. Among datasets, SimGRACE loses PerturbGCL in all cases. Specifically, PerturbGCL. improves SimGRACE 6. 1.11%, 3.53%, 2.34%, 2.26%, 1.95%, and 1. 1.56% on Cora, PubMed, Computer, Photo, CS, and Phy, respectively. This result is in line with our analysis in Section 2.2
baselines including
3.3 CAN PERTURBGCL GENERALIZE WELL TO GRAPH CLASSIFICATION TASK?
To validate the effectiveness of PerturbGCL on graph classification, we compare it with state-of- the-art graph-level GCL methods different datasets. Table 2 and Table (in Appendix) report

---

Under review as a conference paper at ICLR 2023
Table Test accuracy benchmark datasets in TUdatasets terms of the unsupervised setting for graph classification. means that results are not available in published papers.
Methods
NC11
PROTEINS
DD
MUTAG
COLLAB
RDT-B
RDTMSK
IMDB:
A.R.
Gt.
51.68 +.111
77.34 0.18
41.01 - 0.17
65.87 0.95
9.50
WL
80.01 1 0.50
72.92 1 0.56
1..02.33.00
68.82
0.41
46,06 0.21
72.30 1.3.1.50 3.64
7.50
DGK
80.31 = 0.46
73.30 0,82
87.44
78.04 0.20
41.27
0.18
66.36 + 0.55
6.50
modelvee
64.89 1.61
0T.49 3,57
72.63 + 10:20
10.67
subtvee
02.84 1.47
15,80
0.41
0.42
66.25 1.54
11.50
popk22vc
63.15 9.25
1.03
0.54
IntoGrrahh
1,06 77.57
89.53
T3.013 0..97
GraphCT.
74.30
0.45
0:40
80
1.34
71.36
80.53
0.84
55
71.14
0.64
4.13
JOAD
75.07
0.47
74.35
0.41
77:32
0.54
S7.35
1.02
09.30
50.29
1.35
65.74
0.63
70.21
3.08
5.63
JOADx2
78.36 1. 0.53
74.07
1.10
77.40 1:15
87.67
0.79
69.33
0.34
$6.42
1.45
56,05
0.27
70.83
0.25
4.35
SHnGRACE
79.12 0.48
70.30
0,00
77.44
80001
71.72
0.82
89.51
0.89
60.91
0.34
71.30
0.77
3.13
80.24 + 0.45
76.08 = 0.30
78 78.33 0.57
89.97 + 0.50
76.06 + 0.87
58.88
0.67
66.
0.72
74.14 0.50
2.13
Coauthon-B
Accrrd (5)
83
CZ
Phote
Eooche
The impact of different contrastive objec- tives. Right: Empirical training curves of PerturbGGCL with different values.
Figure 5: Left: Ablation study of PerturbGCL. Middle:
results on unsueprvised and semi-supervised settings, respectively. We observed that ( Pertur- bGCL generally performs better than other baselines across two graph learning tasks. From the unsupervised setting (See Table 2) PerturbGCL achieves the best (or comparable best) results COLLAB and IMDB-B datasets. In
on 6 of 8 datasets, and obtain substantial improvements the semi-supervised setting (See Table Appendix), PerturbGCL. generally performs better than other baselines across comparisons and always ranks top three on all the datasets. These results demonstrate the effectiveness of PerturbGCL on graph learning task.
3.4 ABLATION STUDY
We investigate the contributions of different components in PerturbGCL. Figure and Figure 9 in Appendix report the results on graph and node datasets, respectively. We observe that @ Pertur- bGCL benefits from the combination of randMP with weightPrune. From the figures, Pertur- AGCL.consistentt outperforms two variants (i.e., w/o MP and w/o WP) in all cases, which indicates the reciprocal effects of using randMP and weightPrune together. Morcover, @ replacing weight- Prune with Gaussian noise, PerturbGCL drops significantly. In both node and graph scenarios, PerturbGCL. outperforms the "noise" variant with a great margin. It verifies the effectiveness of the proposed weightPrune strategy.
We also test the results of PerturbGCL under different contrastive objectives, such Barlow Twins (Bielak et al. 2021), Bootstrap (Thakoor et al. 2022), and InfoNCE (You et al. (020)) (Please refer [ details). From [
observe that @ PerturbGCI. performs generally better on InfoNCE and Barlow Twins objectives. Given that InfoNCR is standard contrastive loss and Barlow Twins negative-sample free, PerturbGCL ready to be applied on scenarios with informative negative sample or without negative samples by using different losses.
3.5 CAN PERTURBGCL IMPROVE THE TRAINING EFFICIENCY OF GCL?
We compare the proposed PerturbGCL with strong GCL baselines in terms of the training costs in Table 3 and report the optimization curves Figure 5 (right panel). Experimental configurations are listed in Appendix D We observed that @ using pure model perturbation, PerturbGCL. is training efficient. From Table S we can see that PerturbGCL runs significantly faster per epoch than strong baselines in general, and the performance gap is particularly evident in graph datasets. Besides, PerturbGCL can converge within one hundred epochs in practice as shown in Figure (Right). Thus, the total training time of PerturbGCI. could be further reduced.
8

---

Under review as a conference paper at ICLR 2023
Table 3: Running time per epoch (in seconds). Baseline indicates BGRL and GraphCL for node and graph classification, respectively. All the methods are evaluated on GeForce RTX 2080 Ti GPUs.
Node Benchmark
Graph Benchmark
PubMed
Computer
Photo
CS
NCI1
COLLAB RDT-B RDT-M2K
Baseline
0.14
0.16
0.08
0.21
4.02
10.84
38.35
80.79
PerturbGCL
0.10
0.09
0.05
0.17
1.42
3.01
6.21
12.80
Speedup (vs Baseline)
1.40x
1.77x
1.60x
1.23x
2.83x
3.60x
6.17x
6.31x
3.6
FURTHER ANALYSIS
We finally investigate the sensitivity of PerturbGCI. w.r.t the propagation degree K and prune ratio in Figure TO (Left) of Appendix, the impact of graph augmentation on PerturbGCI. in Figure TT (Middle). and the learning capacity of PerturbGCL. in Figure [0] (Right). We can observe that W PerturbGCL performs stably when K E [1, 3, 5] and E [0.7, 0.9). In Figure 10 left, the performance of PerturbGCL when K = 0.9 (or 0.7) is consistently better than others. PerturbGCL is complementary with advanced graph augmentation. From Figure 10 (Middle), by feeding the augmented graphs as input, PerturbGCL can be further improved. However, the trade-off is that the improvement is not huge but the time search optimal augmentation strategies is costing. Although PerturbGCI is trained based on original graphs, can generalize to perturbed graphs well. As shown in Figure I0 (Right), PerturbGCL has lower Calign and Luniforn values than SimGRACE and strong GCL baseline, which indicates the effectiveness of the proposed model.
4 RELATED WORK
We briefly introduce some related graph contrastive learning methods Xie et al. 2022) and refer readers to (Zhou al. 2020) for a comprehensive review of graph neural networks.
Graph contrastive learning with data augmentation. Similar to contrastive learning on im- ages (Chen et al., 2020). data augmentation is crucial the success of contrastive learning on graphs (GCL), Recently there has been steady progress (You et al.) 20200 [021] Qiu et al.) [2200] Lee et al., 2022: Luo et al. 2022) in designing or identifying informative augmentation strategies to boost the performance of GCI. GraphCI. (You et al. (0220) introduces four augmentation prototypes for graphs, including node dropping, edge perturbation, attribute masking, and subgraph sampling. MoCL {Sun et al.) 2021) and G-Mixup (Han et al.) 2022) propose to utilize domain knowledge such as bioisosteres and graphon to aid augmentation. AutoGCL (Yin et al. 2022), JOAO (You let al. 2021), and GPA (Zhang et al. 2022) suggest leverage extra AutoML techniques (Waring ct al. 2020 to free human labor on augmentation choices. Unlike this line of research, we focus on training GCL methods without explicitly graph augmentation.
Graph contrastive learning without data augmentation. To eliminate the influence of graph aug- mentation GCL, AFGRL (Lee et al. 2022) suggests sampling nodes that share similar semantic information in the hidden space as positive samples. However, it requires non-negligible clustering efforts to spot positive pairs in the learning process. SimGRACE (Xia et al., (022) proposes train GCL by disturbing the model weight using Gaussian noise. However, SimGRACE could limit appli- cations beyond Gaussian distribution, as illustrated Figure In this work, we introduce a tailored model perturbation framework for GNN encoders without constraining the noise distribution.
5 CONCLUSION
In this work, we explore how to perform contrastive learning on graphs without using data aug- mentation, and propose principled framework - PerturbGCCI which is built upon pure model perturbations. Specifically, motivated by the fact that GNN can be divided into message propagation and feature transformation operations, we develop two tailored perturbation strategies: randMP and weightPrume, to effectively disturb GNN's two crucial operations accordingly. We build connections between our model perturbation strategies and estllessabli graph augmentation techniques to understand the working mechanism of PerturbGCL. Through extensive experiments across multi- ple datasets and different graph learning tasks, we show that PerturbGCL can achieve competitive results against strong baselines, while requiring substantially shorter computation time for training.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learn- ing for subgraphs. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pp- 170-182. Springer, 2018.
Kristen Altenburger and Johan Ugander. Monophily in social networks introduces similarity among friends-of-frient ffriendds.. Nature human behaviour, 2(4))22 290, 2018.
Siddhant Arora. A survey on graph neural networks for knowledge graph completion. arXiv preprint arXiv:2007.12374 2020.
Piotr Biclak, Tomasz Kajdanowiez, and Nitesh Chawla. Graph barlow twins: A self-supervised representation learning framework for graphs. arXiv preprint arXiv:2106. 2Xiv:2100.02466, 2021.
Piotr Bielak, Tomasz Kajdanowicz, and Nitesh V Chawla. Graph barlow twins: A self-supervised representation learning framework for graphs. Knowledge- Baasdd Systems, pp. 109631, 2022.
Tianlong Chen, Yongduo Sui, Xuxi Chen, Aston Zhang, and Zhangyang Wang. A unified lottery ticket hypothesis for graph neural networks. In ICML., pp. 1695-1706. PMLR, 2021.
Ting Chen, Song Bian, and Yizhou Sun. Are powerful graph neural nets necessary? a dissection graph classification. arXiv preprint XoX.:v/99500577777 2019.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. ICML, pp. 1597-1607. PMLR, 2020.
David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan Adams. Convolutional networks on graphs for learning molccular fingerprints. Advances in neural information processing systems, 28, 2015.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint 1rXiv:1803.0360 2018.
Hongyang Gao and Shuiwang Ji. Graph u-nets. international conference on machine learning, pp. 2083-2092. PMLR, 2019.
Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. Large-scale learnable graph convolutional networks. In KDD, pp. 1416-1424, 2018.
Justin Gilmer, Samuel Schoenholz, Patrick Riley, Oriol Vinyals, and George Dahl. Neural message passing for quantum chemistry. In International conference on machine learning. pp- 1263-1272. PMIR, 2017.
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 855-864, 2016.
William L Hamilton, Rex Ying. and Jure Leskovec. Inductive representation learning on large graphs. In NeurIPS, pp. 1025-1035, 2017.
Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. arXiv preprint arXiv:2202.07119, 2022.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, pp. 4116-4126. PMLR, 2020.
Zhenyu Hou, Xiao Liu, Yuxiao Dong, Chunjie Wang, Jie Tang, al. Graphmae: Self-supervised masked graph autoencoders. arXiv preprint arXiv:2205. 10803, 2022.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. arXiv preprint 1905. 12265, 2019.
10

---

Under review as a conference paper at ICLR 2023
Wei Jin, Tyler Derr, Haochen Liu, Yiqi Wang, Suhang Wang, Zitao Liu, and Jiliang Tang. Self-supervised learning on graphs: Deep insights and new direction. arXiv preprint 1oXYv:2006.10141 2020.
Thomas N Kipf and Max Welling. Sumi-supe classification with graph convolutional net- works. arXiv preprint arXiv: 1609.02907 2016a.
Thomas N Kipf and Max Welling. 161 2016b.
Variational graph auto-encoders.
arXiv preprint
Namkyeong Lee, Junseok Lee, and Chanyoung Park. Augmentation-free self-supervised learning on graphs. arXiv preprint wxXiv:2112.024777 2021.
Namkyeong Lee, Junseok Lee, and Chanyoung Park. Augmentation-free self-supervised learning on graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp- 7372-7380, 2022.
Chang Li and Dan Goldwasser. Encoding social information with graph convolutional networks forpolitical perspective detection in news media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2594 2604, 2019.
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. TKDE, 2021a.
Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S Yu. Graph self-supervised learning: A survey, arXiv preprint arXiv:2103.00111, 2021b
Youzhi Luo, Michael McThrow, Wing Yee Au, Tao Komikado, Kanji Uchino, Koji Maruhash, and Shuiwang Ji. Automated data augmentations for graph classification. arXiv preprint arXiv:2202.13248, 2022.
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. Image-based rec- ommendations styles and substitutes. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pp. 43-52, 2015.
Christopher Morris, Nils M Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. arXiv preprint vrXiv:2007.08663 2020.
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv preprint arXiv: 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv:1807.03746. 2018.
Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 701-710, 2014.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training. In KDD, pp- 1150-1160, 2020.
Bernhard Scholkopf and Alexander Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2018
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad Collective classification in network data. Al magazine, 29(3):93 93, 2008
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv: /8/ 1086688 2018.

---

Under review as a conference paper at ICLR 2023
Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhor, and Karsten Borgwardt. Ef- ficient graphlet kernels for large graph comparison. In Artificial intelligence and statistics, pp. 488-495. PMLR, 2009.
Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borg- wardt. Weisfeiler-Iehman graph kernels. Journal of Machine Learning Research, 12(9). 2011.
Arnab Sinha, Zhihong Shen, Yang Song, Hao Ma, Darrin Eide, Bo-June Hsu, and Kuansan Wang. An overview of microsoft academic service (mas) and applications. In Proceedings of the 24th international conference on world wide web, pp. 243-246, 2015.
Fan-Yun Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. arXiv preprint
01000, 2019.
Mengying Sun, Jing Xing, Huijun Wang, Bin Chen, and Jiayu Zhou. Mocl: Contrastive learning molecular graphs with multi-level domain knowledge. arXiv preprint arXiv:2106.04509, 2021.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. NeurlPS, 34:15920 15933, 2021.
Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličković, and Michal Valko. Large-scale representation learning on graphs via bootstrapping. In ICLR, 2022.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Advances in Neural Information Processing Systems, 33:6827-68399 2020.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10900 2017.
Petar Veličković, William Fedus, William Hamilton, Pictro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. arXiv preprint 1809, 10341, 2018.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp- 9929-9939, PMLR, 2020.
Jonathan Waring, Charlotta Lindvall, and Renato Umeton. Automated machine learning: Review of the state-of-the-art and opportunities for healthcare. Artificial intelligence in medicine, 104: 101822, 2020.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simpli- fying graph convolutional networks. In ICML. 2019.
Jun Xia, Lirong Wu, Jintao Chen, Bozhen Hu, and Stan Z Li. Simgrace: A simple framework for graph contrastive learning without data augmentation. In Proceedings of the ACM Web Confer- ence 2022, pp. 1070-1079, 2022.
Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Dongkuan Xu, Wei Cheng, Dongsheng Luo, Haifeng Chen, and Xiang Zhang. Infogel: Information- aware graph contrastive learning. Advances in Neural Information Processing Systems, 34, 2021.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? arXiv preprint arXiv:1810.00826 2018
Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1374, 2015.
12

---

Under review as a conference paper at ICLR 2023
Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, and Xiang Zhang. Autogel: Automated graph contrastive learning via learnable view generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 8892-88888888 8900, 2022.
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hier- archical graph representation learning with differentiable pooling, Advances in neural information processing systems, 31, 2018.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. NeurIPS, 33:5812-5821) 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning auto- mated. arXiv preprint arXiv:2106.07594 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Hengrui Zhang, Qitian Wu, Junchi Yan, David Wipf, and Philip S Yu. From canonical correlation analysis to self-supervised graph neural networks. Advances Neural Information Processing Systems, 34:76-8488. 89166-99900 2021.
Xin Zhang, Qiaoyu Tan, Xiao Huang, and Bo Li. Graph contrastive learning with personalized augmentation. arXiv preprint 1xXiv:209.005664 2022.
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica- tions. Al Open, 1:57-81, 81, 2020.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.0 04131, 2020.
Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. An empirical study of graph contrastive learning. 2021a.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW, pp. 2069-2080, 2021b.
13

---

Under review as a conference paper at ICLR 2023
A EXPERIMENT SETUP
In this section, we introduce the datasets used in our experiments. Specifically, we adopt the follow- ing 6 popular node-level datasets and summarize their statistics in Table
They are two widely used citation network datasets (Sen et al. 2008 Nodes represent documents and edges denote citation links. Each node has a sparse bag- of- the-words feature vectors. Labels are defined as the academic topics.
Cora, and PubMed:
A Amazon-Computers and Amazon-Photo: They are two networks of co-purchase relation- ships constructed from Amazon (McAuley et al.) 2015). Nodes indicate goods and edges represent the co-purchase relationships of two products. Each node has a sparse bag-of- words feature encoding products reviews and is labeled with its category. They are widely used for node classification task. Nodes represent authors and edges indicate co-authorship relationships. Each node has a sparse bag-of-wor feature based on paper keywords of the author. The task is to predict the most active research field of authors.
Coauthor-CS and Coauthor-Physics They are two academic networks, which represent co-authorship graphs based the Microsoft Academic Graph from the KDD Cup 2016 challenge (Sinha et al. 2015).
Moreover, we also consider 9 graph-level benchmark datasets to verity the effectiveness of Pertur- bGCL on graph-learning task. Specifically, we use 5 social networks (COLLAB, REDDIT-BINARY, IMDB-BINARY, and GITHUB), and 2 molecules networks (NCI1 and MU- TAG), and 2 bioinformatics networks (PROTEINS and DD) from the benchmark TUDdataset (Mor- ris et al. 2020). Table lists their statistics.
Table 4: Dataset statisties of node-level benchmarks.
Data
# Nodes
# Edges
Features
Split ratio # Classes
Cora
708
5, 429
1, 433
85/5/15
7
PubMed
19, 717
44,338 338
500
85/5/15
3
Amazon-Computer Computers
13, 752
245, 861
767
10
Amazon-Phout Photo
7, 6,0
119, 081
745
8
Coauthor-CS
18,333
81,894 894
6, 808
15
Coauthor-Phyee Physscs
34, 493
247, 962
8,415
5
A.1 SETUP FOR NODE CLASSIFICATION
Dataset. We use 2 Planetoid graphs (Cora and PubMed), and 4 widely used datasets (Shchur al., 2018) Computn-Compuers, Amazon-Photo, Coauthor-CS, and Coauthor-Physics) for experiments. For Cora and PubMed, follow the common semi-supervised practice (Kipf & Welling 2016a) to generate train/val/test data splits without any modifications. For Amazon-Computer,, Amazon- Photo, Coauthor-CS and Coauthor-Physics, since there are no data splits available, similar to BGRL (Thakoor et al. 2022 and GBT [Billak et al. 2022), we generate 20 random train/val/test splits (10%/10%//00%)
Competitors. To have a rigorous and comprehensive comparison with state-of-the-art methods, compare PerturbGCL with 2 classical unsupervised models: DeepWalk (Perozzi al.] 2014) and GAE (Kipf & Welling, 2016b), 9 standard self- models, DGI (Veličkovic et al.
2018. GRACE (Zhu et al. 2020). MVGRL (Hassani & Khasahmadi 2020). GCA (Zhu et al.
2021b), BGRL. TThakoor et al. 2022). GBT (Bielak et al.) (002)), InforGCL (Xu et al. (021]) CCA-SSG (Zhng et al.) 2021). AFGRI (Lee et al. (021). and one model perturbation method
SimGRACE (Xia et al. 2022). We also compare with supervised learning models including
GCN Kipf & Welling 2016a) and GAT /Veličković et al. 2017). The results of baselines are
quoted from (Zhang
2021 Xu et al. 2021 Xia al. 2022 if not specified.
Evaluation protocol. We follow the popular linear evaluation scheme to evaluate the performance of unsupervised models. Specifically, we first pre-train the model the given graph without using ground- truth labels. Then, we freeze the parameters of the encoder and use it to generate node rep- resentations. After that, the generated node representations would be fed into linear classification,
14

---

Under review as a conference paper at ICLR 2023
Table 5: Dataset statistics of graph-level benchmarks. # Label
9
Avg.Edges Nodes
NCII
4, 110
29.87
32.30
PROTEINS
1, 1,3
39.06
72.82
DD
178
284.32
715.66
MUTAG
188
17.93
19.79
COLLAB
5, 5,000
74.49
2, 2,457.78
BMDB-BINARE
1,000
19.77
96.53
REDDIT-BINARY
2,000
429.63
497.75
REDDIT-MULTI-SK
4. 4,999
508.52
594.87
GITHUB
12,
113.79
234.61
2
Table 6: Hyperparameters of PerturbGCL on node classification. We use GCN (Kipf & Welling 201 6ab as the backbone encoder.
Cora
PubMed
Computer
Photo
CS
Phy
layers hidden dim
2
2
2
2
2
512
512
512
512
512
512
learning rate
0.005
0.0005
0.001
0.00001
0.00001
0.00001
propagation step
3
3
2
3
pruning ratio
0.5
0.7
0.7
0.9
0.9
0.9
i..., simple logistic regression model, to make the prediction for each node. It is worth noting that only nodes the training set are used as supervision when training the classifier, and we report the accuracy results testing nodes.
We implement PerturbGCL. with PyTorch and use Adam optimizer to train the model. The graph encoder fw is specified hyperparameters (pruning ratio s and random propagation step K) to tune. For each dataset, we search К [1,2,3] 3] and s E [0.5, 0.7, 0.9). To avoid randomness, we report the mean accuracy with a standard deviation through 10 random initialization. The detailed hyperparameter settings are summarized in Table
standard two-layer GCN model for all the datasets. We have two
A.2 SETUP FOR UNSUPERVISED GRAPH CLASSIFICATION
Dataset. For the unsupervised graph classification task, we adopt benchmark datasets (NCI1, PROTEINS. DD, MUTAG, COLLAB, RDT-B, RDT-MSK, and IMDB-B) for experiments, follow- ing You et al. 2020). There are 20 epochs pre-training under the naive-strategy. After the
Competitors. We compare with the kemne--based methods like graphlet kernel (GL.) (Sher- vashidze et al. 2009). Weisfeiler-Lchman sub-tree kernel (WL) (Shervashidze et al., 2011] }, and deep graph kernel (DGK) (Yanardag & Vishwanathan 2015). and other unsupervised graph rep- resentation meodels like node2vec (Grover & Leskovec (0066), sub2vec (Adhikari et al., 2018).
graph2vec (Narayanan al. 2017). as well as the sfate-of-the-ard GCL methods like MVGRL (Has- sani & Khasahmadi] (020)) InforGraph (Sun et al., 2019), GraphCL (You et al. 2020), JOAO You et al. 2021), and SimGRACE (Xia et al. 2022).
Evaluation protocol. Following GraphCI. You et al. 2020), contrastively train the representation model using unlabeled graph data, and then fix the representation model and train a downstream classifier using labeled data. Specifically, cross validation. For each fold, we employ 90% of the data as the labeled data for training and the remaining 10% the labeled testing data. To avoid randomness, repeatedly experiments for 5 times and report the averaged results.
adopt SVM as the classifier and perform 10-fold
GIN (Xu et al. 2018 as the GNN backbone, and also search the best K and s from [I, 2,33 and (0.5, 0.7, 0.9), respectively. Table reports the parameter configurations for all datasets.
Following GraphCL (You al.) 2020),
15

---

Under review as a conference paper at ICLR 2023
Table 7: Hyperparameters of PerturbGCL on unsupervised graph classification. We GIN Xu et al. 2018) as the backbone encoder.
NCI1
PROTEINS
DD
MUTAG
COLLAB
RDT-B
RDT-MSK
IMDB-B
# layers hidden dim
3
3
32
32
32
32
32
32
32
32
learning rate
0.0001
0.0001
0.0001
0.0001
0.0001
0.0001
0.0001
0.0001
propagation step
2
3
2
2
2
2
pruning ratio
0.9
0.7
0.9
0.7
0.9
0.9
0.5
0.5
Table 8: Hyperparameters of PerturbGCL on semi sumi-supereised graph classification. We use Res- 2019) as the backbone encoder.
GCN (Chen et al.
NCII
PROTEINS
DD
MUTAG
COLLAB
RDT-B
RDT-MSK
GITHUB
# layers hidden dim
5
5
5
3
5
3
3
128
128
128
128
128
128
128
128
learning rate
0.0001
0.0001
0.0001
0.0001
0.0001
0.0001
0.0001
0.0001
propagation step
2
2
2
2
pruning ratio
0.5
0.7
0.9
0.7
0.9
0.7
0.7
0.7
A.3 SETUP FOR SEMI-SUPERVISED GRAPH CLASSIFICATION
Dataset. We perform semi-supervised graph classification task 7 popular benchmark datasets (NCII, PROTEINS, DD, COLLAB, RDT-B, RDT-M5K, and GITHUB) from TUDataset (Morris ct al. 2020). There are 100 epochs pre-training under the default setting.
Competitors. We compare with unsupervised graph representation meodels: GAE Kiipf & Welling 2016b), Infomax (DGI) (Veličković et al. 2018). and ContextPred (Hu et al. 2019 and other state- of-the-ar art GCL methods like InforGraph (Sun et al. 2019), GraphCL You et al. 2020), JOAO (You) et al.
2021 and SimGRACE (Xia et al. 2022).
Evaluation protocol. We employ 10-fold cross validation each dataset. For each fold, 80% the data as the unlabeled data, 10% as labeled training data, and 10% as labeled testing data. For the augmentation only (Augmentations) experiments, we only perform 30 epochs of supervised training with augmentations using labeled data.
Following GraphCL. You et al. 2020), and also search the best K and from (1, 2,3) and (0.5, 0.7, 0.9), respectively. Table 8 reports the parameter configurations for all datasets.
ResGCN (Chen et al. 2019) the GNN backbone,
Algorithm PerturbGCL node level task
Input:
Original graph (V,E, X), GNN encoder f-)()) with weight W, projection head h(-), the maximum propagation step K, and pruning ratio s
2: Initialize the encoder f and set mask indicator ones.
3: for iterate 1, 2, times until convergence do
4: Sample the random propagation steps k,, K3 from the uniform distribution U(1, K)
5: Conduct weight pruning update the mask indicator mae
6: Compute the target representation h, according by performing k' times of g()
7: Get z, h(h,) according to
8:
Compute the perturbed representation h according to using masked weight W m.
I by performing k" times g() and
9: Get 2, = =hhh)
[V EvEY log exp )/T)+ Dur ast exp(sim(z,,225/77 according 11: Optimize for (-),.)(())) to minimize LC
define LOL
exp(sim(2, .2t1:
end for
13: return the pre-trained GNN encoder f.)(.)
16

---

Under review as a conference paper at ICLR 2023
Algorithm 2 PerturbGCL on graph level task
1: Input: Original graph EV,E, X), GNN encoder fie()) with weight W, projection head 1(()), the maximum propagation step K, and pruning ratio 8
2: Initialize the encoder (+) and set mask indicator ones.
3: for iterate 1, 2, times until convergence do
4: for sampling batches Gn from G. where n 1,2,... do
5: Sample the random propagation steps k', k" from the uniform distribution U(1, K)
6:
Conduct weight pruning to update the mask indicator ma
7:
Compute the target representation h, according to 1 by performing k' times of g()
8:
Get 2G (EADOUT(M(b,))e. Compute the perturbed representation h according to by performing k" times of 5()
according to 2
9:
and using masked weight W @ m2
10:
Get G. (EADOOU(H(()))))) )veg.))
exp(sim(Zo.. 2/T)
11:
define LCL [G] Egneg - llog exp(sim((R 26 1/T)+ + Eucg.upte exp(sim(Rc,, 2. 1/T) accord-
ing 3
12:
Optimize fw(()),()) to minimize Lcr.
13:
end for
14:
end for
15:
return The prentrained GNN encoder f(-)
B DETAILS FOR TOY EXAMPLE
To verify the limitation of SimGRACE (Xia et al. 2022 handling perturbation created by data augmentation. We select three popular data augmentation based baselines: GRACE (Zhu et al. 2020), BGRL (Thakoor et al. 2022), and CCA-SSG (Zhang al. 2021). To measure the qualify of the representation models on leanning representations for input data, we adopt the widely used align- ment and uniformity metries (Wang & Isola 2020) for quantitative analysis. According (Wang & Isola 2020), both metrics are the smaller the better.
Evaluation setting. For all methods, we first pre-train them according their own configurations on PubMed, Amazon-Phoot and Coauthor-CS datasets. Then, we use data augmentation strategies to construct two perturbed views. Specifically, we following this detailed empirical study (Zhu al.) 202 and adopt edge perturbation and attribute masking the default perturbation function on the input graph. To have a fair comparison, we fix the random seed and generated two shared perturbed graphs, and then feed the two views into BGRL, GRACE, CCA-SSG, and SimGRACE to obtain node representations for all nodes in the graph. After that, we use the obtained node representations of two views to compute the alignment and uniformity according to Eq. WE repeat the process for 10 times and report the averaged results in Figure
Table 9: Test accuracy on benchmark datasets in TUdatasets in terms of semi-supervised graph classification.
Dataset
NCHI
PROTEINS
DD
COLLAR
RIYT-R
RIYI-MSK
GITHUR
A.R. J
No pre-train
73.72 0.21
70.40
1.54
73.56 = 01.1
73.71 4: 0.27
$6.63 0.27
51.33 0.14
60.87 4: 0.17
9.86
Augmentations
0.332
0.13
0.39
GAE
0..2
70.51 0.17
74.54 = 0.68
75.09
0.11
87.69
0.10
53.58 2 0.13
63.89 2: 0152
7.14
InfoGraph
74.86 0.226
72.27 2 0.10
75.78
0.34
73.76 1 0.29
88.66
0.95
0.31 53.61
65.21 0.88
4.71
73.00+0..30
0.63
73.00
0.37
84.76
0.52
51.23
Infomax
74.86
72.27 0.40
75.78
0.34
73.76
0:29
58.66
0.95
53.61
0.31
65.21 4: 0.88
5.71
Grap&CL
74.63.1..33
0.34 74.17
76.17 1 1.37
74.25 11 0.21
89.
0.19
52.55 0.45
65.81
0.79
3.86
72.10
75.30
88.14
0.25
J0A0v2
74.86 0.39
73.31
0.48
75.81
0.73
53
0.18
88.79
0.66
52.71
0.28
06.60
0.60
3.57
SimGRACE
0.
74.03:1 0.51
76.48
74.71
0.28
0.62 $8.86
0.64 53.97
65.33
3.29
0.42
76.05
74.50 0.41
88.00 0.63
0.12
0.10
17

---

Under review as a conference paper at ICLR 2023
C MORE CONTRASTIVE LOSS FUNCTIONS
Although InfoNCE You et al. 2020 (illustrated in Eq. 3 is the widely used contrastive objective in learning GCL models, Is, some other training objectives have been proposed recently, such Barlow Twins (Bielak et al. 2021), Bootstrap (Thakoor et al. 2022). Different from InfoNCE, the other two contrastive objectives are negative-sample free, they can avoid the efforts identify informative negative samples during the training. Specifically, the core idea of Bootstrap function is to maximize the difference between the positive pairs, defined as:
C
Z
Here h the hidden representation encoded by GNN encoder, and Z, -hhh") where h(-) the prediction head. In cases, since we take the original graph as input, do not have symmetric define of the loss function as done in [Thakoor et al. (022). This might be the reason why our model PerturbGCL performs not good using this objective.
Barlow Twins (Bielak et al. 2021) is recent endeavor to reduce the usage of negative samples. This objective is originally proposed in image domain by (Zbontar al. 2021). The general idea of Barlow Twins is to minimize the redundancy in the hidden dimension. Specifically, given the hidden representation of two views (Z and Z*), first compute the empirical cross-correlation matrix C R DxD as below:
An, Z
C.j E.. (Z..))
En (Z
where n is the the batch indexes and i are the indexes of embeddings. The cross-correlation matrix optimized to be equal to the identity matrix. To be specific, invariance term and 2) the redundancy reduction term. The first one forces the on diagonal elements C.,; to be equal to one, hence making the embeddings invariant to the applied augmentations. The second term optimizes the off-diagonal elements Ct, to be equal to zero this results in decorrelated components of the embedding vectors. Formally, the loss function Lgr computed by:
composed of two parts: 1) the
LBT C.,) + > Ch
The parameter defines the trade-off between the invariance and redundancy reduction terms. In our experiments, we set À following (Bielak et al. 2021
CouthensSS
0.30
DCA-69
DCA-6S GRASE
0.28
Bicaa
0.2
0.50
BiMGERACE PertuoGC
SiGGRAGEE
0224
0/40
PeraebiCL
0.00
0.20
0.2
descirocce
0244
0.4
Penabri0L
5.18
Figure 6: The alignment and uniformity plot for BGRL Thakkoor et al. 2022). SimGRACE Xia et al. 2022). GRACE (Zhu et al., 2020), CCA-SSG (Zhang et al.) [2021) ,, and our PerturbGCL on the same perturbed graphs generated by data augmentation. Black circles indicate the baselines. Orange circles represent the performance of SimGRACE, Red starts (*) are the results of PerturGCL.
D EFFICIENCY ANALYSIS
To evaluate the efficiency of the proposed PerturbGCL, compare it with two strong GCL base- lines: BGRL (Thakoor et al. 2022) on node classification data and GraphCL (You et al. 2020) on require graph classification data. It is worth note that since BGRL and GraphCL. require to search for the
18

---

Under review as a conference paper at ICLR 2023
optimal augmentation strategies via trial-and-ere which is super expensive in practice. To simplify the comparison, we fix their optimal augmentations and only record the running time on the optimal augmentations. For graph-level datastes, GraphCL and PerturbGCL use 3 GIN layers with hidden dimension 32 as the backbone encoder. The propagation step and pruning ratio PerturbGCL. are 0.7. For node-level datasets, BGRL and PerturbGCL. use 2 GCN layers
K 2 and with hidden dimension 512 as the backbone encoder. The propagation step and pruning ratio of PerturbGCL are set as K 3 and 0.9. We conduct experiments on a server with AMD EPYC 7282 16-Core processors, 252 GB memory. and one GeForce RTX 2080 Ti GPUs (24GB). To avoid randomness, the reported results in Table are the averaged performance over 100 training epochs.
Except the improvement in Table 3 another thing we want to mention is that the practical speedup of PerturbGCC. should be significantly higher than the results shown in Table 3 This is mainly because the data augmentation based GCL methods require lot of efforts search for the best augmentation strategies, such as the best augmentation types and their corresponding perturbation ratios.
E
MORE RESULTS
1.0
Spoche
Specht
Specha
Figure 7: Empirical training curves of PerturbGCL on graph benchmarks with different pruning ratios
-0320 -205 -5.00 -0ázs
8000 00s
0250
udis
438. 4000-0.0 220r
-ile vas site
8250
02/s
abx
aite
onis
6:50 62/s
Figure 8: Visualization of second layer weight distribution during training process(from left right: initial weights, PerturbGCL w/o. weightPrune and PerturbGCL) on Coauthor-CS. The x-axis indi- cates weight values and y-axis the corresponding count.
Acouracy (%6)
70
Noise
60
w/a MP w/o WP PerturbGCL
85
PROTEINS
Figure 9: Ablation study of PerturbGCL on node benchmarks.
19

---

Under review as a conference paper at ICLR 2023
0.32
CSA-89
GRAGE
803
BORL
SINORRACE
231
PorturlOCk
0.25
0.24
-2.0
-1.8
-18
-12
Belend
Figure 10: Left: Hyperparameter Analysis on Coauthor-CS CS, Middle: PerturbGCL with data aug- mentation. Right: The aligument and uniformity results of PerturbGCL.
20


Reviews:
[{"review_id": "TYiDgZcQZz3", "reviewer": "Reviewer_SXei", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose a new augmentation based on graph neural networks for graph contrastive learning. They decompose the graph neural networks as two parts: the diffusion part and the transformation part. Based on these, they propose weightPrune for the transformation and randMP for the diffusion part. They call the technique as model pertubation and evaluate it on 15 benchmarks.", "summary_of_the_review": "Between my comments above, I am personally inclined to reject this paper. Of course I am happy to see other different opinions before making a decision.", "strength_and_weaknesses": "Strength:\n1. This paper suggests a possible direction for research. Do not focus on raw graph data like attributes and adjacent matrix for augmentation rather than graph neural networks. And the method is more flexible than simGRACE.\n2. weightPrune is a new way for augmentation in GCL. \nWeakness:\n1.  First of all, this graph augmentation has no theoretical support, at least in my opinion, it should not be difficult to prove the alignment loss bound before the transformation.\n2. Because of the lack of theoretical analysis, there is no way to draw a conclusion as to what this graph augmentation actually does, what is its significance, and what does it tell us about future graph augmentation for models?\n3. Multi-hops diffusion based augmentation is not new, MVGCL has proposed.\n4. In my opinion, weight pruning is also a noise (biased salt noise rather than gaussian). Therefore the paper lacks a comparison with other noise methods, such as Gaussian noise.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OjKH29Wn_L", "reviewer": "Reviewer_4KLm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a model perturbation strategy PERTURBGCL to perform graph contrastive learning without data augmentation. It consists of two key components: message propagation and transformation. They design weightPrune on transformation weight according to magnitudes, which create a dynamic perturbed model to contrast with the target encoder. Then they present randMP on the steps of message propagation. Finally they use above two strategies together to perform effective contrastive learning on graphs.", "summary_of_the_review": "The authors propose a model perturbation strategy PERTURBGCL to perform efficient contrastive learning instead of data augmentation. The method develops more effective model perturbation framework to generate contrastive views. The authors conduct extensive experiments to demonstrate the superiority of PERTURBGCL. However, in this paper, there are still some core points that have no theoretical support, no specific proof or analysis. The survey on the latest model perturbation research for contrastive learning is not sufficient.\n", "strength_and_weaknesses": "Strength:\n1. The paper proposed a novel model perturbation framework, consisits of weightPrune on transformation and randMP on message propagation.\n2. The authors conduct extensive experiments for both node-level classification task and graph-level classification task on benchmark datasets to demonstrate the superiority of PERTURBGCL. \n\nWeaknesses:\n1. The paper only analyzes the flaws of the model perturbed baseline SimGRACE, and the ablation experiment also confirmed that PERTURBGCL is superior to the perturbation of Gaussian noise. However, there is no specific theoretical analysis of the reasons why the design of the two perturbation methods can make contrastive learning obtain better performance.\n2.  Although the paper mentions that the lottery hypothesis: the sparse subnetwork can achieve performance comparable to dense network with fewer model parameters. In my opinion, this advantage comes from iterative magnitude pruning and rewind. The weightPrune is a dynamic pruning with mask updates, and these two elements do not exist. Therefore, please explain what is the source of the advantage of creating perturbed branches with weightPrune.\n3.  What is the difference between k times of message passing and data augmentation at the edge level?\n4.  The author pointed out that weightPrune actually regularizes the weight of the model. Then how about directly increasing the regularization, such as increasing the weight decay?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JqWW4eon-SM", "reviewer": "Reviewer_bYXS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a principled framework, namely PerturbGCL, for augmentation-free graph contrastive learning. Considering that GNN can be divided into the two parts of message propagation and feature transformation operations, the authors develop two tailored perturbation strategies, namely randMP and weightPrune, to effectively disturb GNN. Through extensive experiments on the node classification task and graph classification task, the authors show that PerturbGCL can achieve competitive results against the SOTA graph contrastive learning methods.", "summary_of_the_review": "Please see my comment in 'Strength And Weaknesses'.", "strength_and_weaknesses": "Strength\n\nS1: The proposed method is simple and effective.\n\nS2: The authors verify PerturbGCL on two downstream tasks.\n\nS3: The authors provide the code of PerturbGCL.\n\nWeakness\n\nW1: The paper is not well written. There are lots of syntax errors and unclear representation in this paper.\n1. In Section 2.1 (Page 2), where the i-th row of X denote -> where the i-th row of X denotes; in Page 4, Figure 2 provide the overview our framework.-> Figure 2 provides the overview of our framework; in the title of Figure 4, The gap between the blue and orange lines indicate -> The gap between the blue and orange lines indicates; the vertical spacing between Figure 4 and Table 1 is too small; in Table 1, compute -> computers; In Table 5, |G| -> |D| (D denotes the graph set)\n2. In Table 1, the exact digits of numbers are not the same, e.g., on Cora, the result of PerturbGCL is 83.3 ± 0.5; while on PubMed,  the result of PerturbGCL is 82.10 ± 0.37.\n\nW2: The novelty is limited.\nThe model perturbation or augmentation-free method has been explored in graph contrastive learning by prior work. RandMP and weightPrune are more like two tricks rather than a novel framework.\n\nW3: Lack of comparison with important baselines.\n1. For the node classification task, considering that the design of PerturbGCL is based on the framework of GraphCL, why do not compare PerturbGCL with GraphCL?\n2. Some other graph contrastive learning methods without using data augmentation, such as AFGRL [1], are not compared with PerturbGCL.\n\nOthers.\n1. The abstract is too long. I think that reducing the content of the abstract appropriately is helpful to highlight the core of this work.\n2. In Table 3, the authors only compare PerturbGCL with BGRL or GraphCL on time costs. Why not compare with other graph contrastive learning methods, such as SimGRACE? Since SimGRACE does not use data augmentation, I guess that SimGRACE may be more efficient than BGRL and GraphCL.\n3. In Table 4, the split ratios of Amazon-Computers, Amazon-Photo, Coauthor-CS, and Coauthor-Physics are not given. Also, please check whether the split ratio of Cora and PubMed is right (note that 85+5+15=105, not 100).\n4. In Algorithm 1, the pruning rato s, as one of the inputs, is not used in this algorithm; g() is not given a specific definition or statement.\n5. In Algorithm 2, the pruning rato s, as one of the inputs, is not used in this algorithm; the input data should be a graph set D, not a graph G; in Line 12, p(\\cdot)->h(\\cdot).\n6. The results in Figure 6 are important and I suggest moving Figure 6 into the main body of the paper.\n\n[1] Lee N, Lee J, Park C. Augmentation-free self-supervised learning on graphs[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(7): 7372-7380.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject