Graph Adversarial Self-Supervised Learning
Authors: Longqi Yang
Liangliang Zhang
Wenjing Yang
Venue: NeurIPS 2021 Poster (2021)
Keywords: graph representation learning
adversarial training
self-supervised learning

Abstract: This paper studies a long-standing problem of learning the representations of a whole graph without human supervision. The recent self-supervised learning methods train models to be invariant to the transformations (views) of the inputs. However, designing these views requires the experience of human experts. Inspired by adversarial training, we propose an adversarial self-supervised learning (\texttt{GASSL}) framework for learning unsupervised representations of graph data without any handcrafted views. \texttt{GASSL} automatically generates challenging views by adding perturbations to the input and are adversarially trained with respect to the encoder. Our method optimizes the min-max problem and utilizes a gradient accumulation strategy to accelerate the training process. Experimental on ten graph classification datasets show that the proposed approach is superior to state-of-the-art self-supervised learning baselines, which are competitive with supervised models.

Graph Adversarial Self-Supervised Learning
Longqi Yang
Institute for Quantum Information State Key Laboratory High Performance Computing. College of Computer Science and Technology, National University of Defense Technology. Changsha 410073, China
Defense Innovation Institute, Beijing 100071, China
bangllonggi99nnude eewpundtt.ddd.... edu. cn
Liangliang Zhang*
Wenjing Yang"
Institute of Systems Engineering,
Institute for Quantum Information & State Key Laboratory of High Performance Computing,
AMS, Beijing, China
vermouthloovddhotmiil...
College of Computer Science and Technology, National University of Defense Technology,
Changsha 410073, China wenj ing, yangonudt, edu.
edu. cn
Abstract
This paper studies a long-standing problem of learning the representations whole graph without human supervision. The recent self-supervised learning methods train models to be invariant to the transformations (views) of the inputs. However, designing these views requires the experience of human experts. Inspired by adversarial training, we propose an adversarial self-supervised learning (GASSL) framework for learning unsupervised representations of graph data without any handcrafted views. GASSL automatically generates challenging views by adding perturbations to the input and adversarially trained with respect to the encoder. Our method optimizes the min-mar problem and utilizes gradient accumulation strategy accelerate the training process. Experimental on graph classifi- cation datasets show that the proposed approach is superior to state-of-the-re self-supervised learning baselines, which are competitive with supervised models.
1 Introduction
Learning effective representations of graph-structured data plays an essential role in a variety of real-world applications, including social, biological, molecules, and financial networks (I]. Recently, graph neural networks (GNNs) have emerged as powerful architectures for learning and analyzing graph representations 2 6). GNNs typically learn graph representations in supervised or semi-supervised setting. In practice, obtaining a large number labels often difficult or even impossible, especially in specific areas that very costly, such as in biochemistry. The labeled graphs may be limited, while unlabeled graphs are easy to collect. Self-supervised learning utilizing and shows
unlabeled data has made significant progress in computer vision 7 g 9
10
12
B
great potential exploring unlabeled data to enhance graph deep learning [4
L5
16
18
19 20).
Despite their success, existing self-supervised learning methods rely heavily on handcrafted view, where the view here refers to human-defined data transformations to preserve the invariance their intrinsic properties. In recent years, researchers have designed views of graphs from various levels, including nodes dropping, edge perturbation, attribute masking, subgraph [15), and graph diffusion
"Corresponding authors.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

[4]. However, the handcrafted views require expert knowledge and trial and error but also do not yield consistent performance gains across multiple tasks [[5]. Therefore, how automatically search for augmentations for graph data remains an open problem.
GNNs vulnerable adversarial attacks, as are deep neural networks. Adversarial attacks usually exploit the gradient information to generate imperceptibly small perturbations that alter the model's output. Adding these adversarial samples the training set, i.e., adversarial training, can improve the neural network to generalize to out-of-distribution samples [21 22 231. Adversarial training usually leads trade-off between robustness and generalization. There has been much research on adversarial training for security purposes [24), in particular, it is still unclear how to combine adversarial training self-supervised learning of GNNs to improve the classification accuracy.
In this paper, we are motivated to address the drawbacks mentioned above and propose a self- supervised learning framework train graph neural network without any class labels. We refer to this novel adversarial self-supervised learning approach as Graph Adversarial Self-Supervised Learning (GASSL). GASSL directly maximizes the similarity of a graph and its perturbed adversarial graph, relying on neither negative pairs nor handcrafted augmented views. In the training phase, we use the gradient accumulation strategy 225 21 accelerate the model training. We verify the effectiveness of CASSL on 10 datasets for the graph classification task including the TU datasets [26) and the Open Graph Benchmark (OGB) [27] We conduct extensive experiments across graph datasets by applying classical GNN models (GCN 4 and GIN(S)) as encoders. Our approach automatically generates challenging views to yield performance gains on multiple tasks compared to handcrafted views. The results show that our method outprrorms state-of-the-art graph self-supervised learning and is close the performance of the supervised GNNs.
Our contribution could be summarized as: (I) We propose a self-supervised learning method GASSL for graph representation learning without human supervision. (2) We use adversarial training to automatically generate challenging views for self-supervised learning in place of handcrafted views, which yield performance gains on multiple datasets. (3) We show that GASSL consistently outperforms state-of-the -fhe------e-rrttt self- sul--super models with a significant margin in graph classification tasks. When compared to supervised baselines, GASSL performs on par with or superior the strong baselines.
2
Related work
Graph neural network (GNN) GNN is built on graph structures to learn representation vector H, for each node V. which are formalized as the following function:
H (+++) = COMBINE(
( (GGREGATE( (K) (f {H Vv e((e)))))))
(1)
where H(()) is the embedding of node e at the k-th layer, (w) denotes neighbor set of node
and H (0) = X,, COMBINE and AGGREGATE are functions parameterized by neural networks. After K rounds of message passing, we obtain the final-layer node representations. To obtain the
representation of the entire graph he E Rd, we need the permutation-invarian
READOUT function
as follows: hg READOUT ( (Hg v E V)). Various GNNs have been proposed 4 28 5
with
various pooling
29 30 31,. achieving state-of-the
performance in graph tasks.
Adversarial robustness For adversarial training, more attention is paid to improving model robust- ness and less on improving generalization performance. For image classification tasks, combining a contrast learning framework with perturbation of the input samples is effective. CLAE (32) employs the Fast Gradient Sign Method (FSGM), and RoCL (33 adopts the Projected Gradient Descent (PGD) method to improve the generalization performance of the model. Concurrent to our work, Tamkin et al. (34) proposed a modellaggosste network (ViewMaker) that perturbs the input by adding an lp constraint to produce useful views and has successful applications on image, speech, and time-series data. Kong et al. [24] proposed to perturb the features of the input nodes of GNN for better generalization performance and utilized gradient accumulation strategy to accelerate adversarial training in a supervised learning setting. Out of positive view, Hu et al. [55] proposed to directly learn a set of negative adversaries playing against the self-trained representation.
Graph self-supervised learning Self-supervised learning has recently made new advances in has
graph representation learning, in which contrast lcarning [36] 17 15
37
38
39 40

---

achieved the state-of of art performance. Infograph [7] maximizes the mutual information between the graph-level representation and the representations substructures of different scales (c.g.. nodes, edges, triangles). By doing so, the graph-level representations encode aspects of the data shared across different scales of substructures. GraphCI. [5]] designed four types of graph augmentations to incorporate various priors, including node dropping, edge perturbation, attribute masking, and subgraph. MVGRL [14] utilized graph diffusion for graph augmentation and found no performance gain for more than two views or multi-scales of encoding. GCC [16] performs random walk with restart for each node to sample subgraph as augmentation. GRACE [40] adopt two augmentations, including removing edges and node feature masking. For thorough review, we refer the reader to the recent survey [42], However, one limitation shared by all these successful approaches is the handcrafted view, which is the primary goal of our GASSL how to learn view automatically without resorting to handcrafting or expert domain knowledge.
3 Methodology
We now show how learn the representation of a graph without handcrafted views with domain expert knowledge. Before that, we will briefly introduce adversarial training supervised learning.
A graph can be represented as G = (V, E), where V is the set of V = 7t nodes, the adjacency matrix A E (0, 1]mx., along with the dimensional node attribute X R Our goal is learn from multiple graphs in dataset and predict the property of a single graph G. The learned d dimensional distributed representation hg E Rd applied for downstream tasks (e.g. graph classification task).
Adversarial robustness We start with the definition of adversarial attacks under supervised settings. Given a dataset DD = (X. let x E X and y Y denote a training sample and the corresponding label, respectively. Given a supervised learning model fo X + Y with parameters 9. Traditional adversarial attacks maximize the loss within a certain radius from the sample as follows:
x² =l (af asign (V.: Leu (0, x², ,.))))
(2)
where B(x, e) is the for norm-ball around æ with radius e, and is the projection function for norm-ball, ball, a is the step size, is the attack iterations, sign(-) returns the sign of the vector, LCu is the cross- cross-entropy loss for supervised training. The straightforward way to defend against adversarial attacks is to minimize the loss of adversarial samples. Authors (43 proposed to seek to find optimal parameters 0* to minimize the maximum risk for any within a norm ball as follows:
min R. (2,))/D
max Ler(O, x + +.,y)
(3)
(EB((..c
where 8 is the perturbation of the adversarial example. The conventional adversarial attacks 43 require to have a class label y e Y. which is not applicable to unlabeled data.
3.1 Self-supervised learning on graphs
Self-supervised learning typically design pretext tasks to bring different views of the same instance (positive view) closer and push views of different samples (negative view) farther apart. The simple and performant BYOL [9] does not need to maintain negative views explicitly and depends only on positive views. Inspired by BYOL, we propose GASSL framework (Figure 331) to learn graph representation. GASSL comprises the two networks: the teacher and student networks. The two networks shared the same architecture with different parameters. In detail, the teacher network is defined by a set of weights 0, while the student network using a different set of weights &, which are an exponential moving average of parameters 0. Given a target decay rate 3 E [0, 1], after each training step, we perform the following update, + 3E + (1 3)0. Note that the predictor go is only applied to the teacher network to avoid collapse, leading to an asymmetric architecture.
Encoders. In order to learn the graph representation z, we used GNN (defined in 2 following with a two-layer multi-layer perceptron (MIP) as an encoder f(-). Our framework allows various choices of the network architecture without any constraints. We opt for simplicity and adopt the commonly used GCN [4] and GIN [5].
3

---

perturb on hidden layer
fe
q0
maximize minimize
adv
Z8
perturb on init feature
noise 8
0000
90 (zzav)
moving average
0800
9 (zg)
Z
graph g
ZE
stop gradient
fg
The proposed GASSL consists of two networks, called teacher network (upper) and student network (lower). The encoder of the student network is the moving average of the teacher network. The teacher network has an additional MIP qo to avoid collapse. graph G is processed by the encoder network (Jn) and momentum encoder network (fc). respectively. GASSL alternately performs the outer minimization training and the inner maximization adversarial training.
Figure 1:
Similarity loss Our method relies on the positive view of the input graph. The traditional way to produce positive view need to be designed manually, such as node dropping, edge perturbation, subgraph. We will introduce learn challenging positive view automatically in the next subsection. Here suppose we have obtained the positive view, denoted as g'. We feed g' and G to the teacher network and the student network, and obtained the output representations (0())) and 2ę, respectively, 2/))/1ll We then f2- fo-normalized both (0(z)) and zg to (26)) qo (2(2)) (ellllll0///) (2)) and Eş zę/ z</112ell2. We define the mean squared error as follows,
{go( 2/)); ze) 11ge (2') ( 112 ||
Cogg (10(2))) 2gl1ă
2
We symmetrize the loss La.e by separately feeding G and G' to student network and teacher network compute La.g. At each training step, we optimize the loss as follows,
GASSL B
Lo.E + Lo.s-
(5)
3.2 Graph adversarial self-supervised learning
To apply adversarial training to self-supervised learning, optimize the self-supervised loss CASSL (Eq(5) to replace the supervised learning cross-entropy loss (Eq 3) allowing adversarial training to produce views without labels.
For self-supervised learning, the generated perturbations should be challenging and faitifuul [34] The generated views should be complex and robust enough for the encoder to produce useful representation. We generate perturbations by adversarial training to increase the loss between two networks. The perturbation should not make the encoder task impossible. We accomplish this by constraint the radius of perturbation. We can add perturbations directly to the input node features or the output of the hidden layer of the GNN encoder.

---

Algorithm Graph Adversarial Self-Supervised Learning (GASSL)
Input: Graph G (V, E); input feature matrix X; learning rate 7; ascent steps 7'; ascent step size a: perturbation bound c, decay rate B, (. as objective function.
Initialize 0
for epoch to Nep do 6n c-
go f- 0
for to T do
91-1 + T V C H + 6,-1) #Accumulate gradient of parameters 0
96 V&L & ( H δz-1) #Update the perturbation via gradient ascend
ò, + 6 + 0g8 ||98||
end for
0 - 0 TgT
t S (1 3)0
end for
Perturbation on initial node features It is common practice to add noise the node features X, which we denote the perturbed features as X following the min-max formulation,
X + ô. The adversarial learning objective
min Egnes
max CASSLL [1
(G:X+ +
(6)
Perturbation on hidden layers For GNNs, the features of nodes are aggregated by the neighbor- hoods. Perturbation the hidden layer output affect the nodes with their neighborhoods would produce challenging view. We denote the output of first hidden layer as H(),, the perturbed output is (1) 8. We optimize the following min-mas formulation.
min Eg
GASSL [18
max
(G; (G: + +5)
The problem (6 and problem 7 are similar, we analyze problem the following. For problem 7 the outer "min" of the is non-convex and the inner 'max' non-concave. This saddle-point problem could be reliably solved with stochastic gradient descent (SGD) for outer minimization and projected gradient descent (PGD) for inner maximization [43]. In this work, we take unbounded adversarial attacks instead. The parameter is updated after each step,
8.+1 = 5, + o.g66)))) (=g6)))/ll0))))
(8)
where g(6;) V COSSL (G; H + 6,) the gradient of the loss with respect Ò.
3.3 Acceleration training with gradient accumulation
The computation of 5 is inefficient since 7 7'step updating takes T herward-backward passes, and the SGD takes only one pass through the neural network. We then leverage the 'free' strategy [25] 21 for efficient adversarial training. The idea of 'free' strategy is accumulate gradients of V FoL CASSL in each iteration of inner loop and update the model parameter 0 with the accumulated gradients. During the training procedure, suppose ó, and 0.-1. By taking decent step along the averaged gradients at H (1) + ôō. we approximately optimize the following objective:
inner loop T times, each time computing gradient for
H 8T-1.
min E
Σ t
C GASSL (G: ()) + +6.)
(9)
T
de
Se
The overall procedure is shown in Algorithm
5

---

Experiment
4.1
Datasets
We selected 10 widely used graph classification datasets from TU datasets [26] and Open Graph Benchmark (OGB) (27] For TU datasets, we select three bioinformatics datasets (MUTAG [45), 47 and three social network datasets (COLLAB [48, IMDB- 48). IMDB-MULTI 48) Notably, since the nodes have no features for the social network datasets, the one-hor hot encodings of node degrees features. We classification aceuracy an evaluation metric. For OGB datasets, we selected four of the molecular datasets, including HIV, Tox21, ToxCast, and BBBP. We use the ROC-AUC for an evaluation metric. Statistics are reported in and more details are described in the Appendix.
PTC-MR 46 45 NCI1
Table T
Table I: Statistics of graph classification benchmarks.
Dataset
MUTAG
PTC-MR
IMDB-B
IMDB-M
COLLAB
NC11
HIV
Tox21
ToxCast
BBBB
No. Graphs
188
314
1,000
1,500
5,000
4,110
41,
7,831
8,576
2,039
No. Classes
2
2
2
3
2
12
617
2
No. Nodes
17.9
25.5
19.8
13.0
74.5
29.8
25.51
18.57
18.78
24.06
4.2
Baselines
We select three families of baselines, including graph kernel methods, supervised GNN, unsupervised (self-supervised) methods. The graph kernel methods including shortest path kernel (SP) (4)] Graphlet kernel (GK) [50]. Weisfeiler-
sub-tree kernel (WL) [51). deep graph kernels (DGK)
48), and multi-scale Laplacian kernel (MLG) (52) reported in (17]. The supervised GNN-based
models including GraphSAGE 33 GCN GAT (28), GIN-O and GIN- reported in [5] In addition, GNN incorporates newly developed pooling methods further improve performance on graph classification tasks, and we have selected StructPool [29]. MinCutPool [30], and Grpah Multiset Transformer (GMT) [31]. The unsupervised methods including random walk (53), node2vee (54),
sub2vec [55], and graph2vec [56). The state-of-the-art self-supervised graph representation learning including InfoGraph 17. MVGRL 14] GraphCL
[5] and GCC [16
4.3 Evaluation protocol
For all experiments on the TU dataset, we follow 17 31] and report the mean 10-fold cross-validation accuracy with standard deviation after runs followed by a linear SVM. The linear classifier is trained using cross-vaie validation
training folds of data, and the best mean classification accuracy is reported. For OGB datasets, we evaluate the performance with their original feature extraction and following the original training/validationn validaioon/test dataset splits 27. We train a linear classifier the top of frozen encoder on existing self-supervised learning models
We train the model using Adam optimizer with an initial learning rate of 10-4, and we choose the number of GCN and GIN layers E {2, 3, 4, 5), number of epochs E {20, 40, 100, 200). batch size {32, 64, 128, 256, 512, 1024), and the SVM parameter C' E {10- 1 ..... 10 10 The step size a is set to 8 x 3 the perturbation bound is set to 8 10-3 the embedding dimension is set to 128 (expect HIV set to 512). We also early stopping with the patience of 20, where stop training if there no further improvement on the validation loss during 20 epochs. We conduct all the experiments an Nvidia TITAN Xp.
4.4 The role of adversarial views
Architecture vs. views To illustrate that views generated by adversarial training contribute to graph representation learning, we compare with GraphCL [15] that uses handcrafted views. Our proposed GASSL approach differs from the GraphCL in both the self-supervised learning approach and the view generation. Therefore, we construct new baseline by combining the BYL architecture with GraphCL's view, named GraphCL-BYOL. We follow the setting of GraphCL. and use GIN as the encoder for comparison methods. From the Table 2 we observe that replacing the backbone of GraphCL from SimCLR to BYOL yields consistent performance improvement. We choose the

---

Table 2: Graph classification results (%) test sets. GraphCL-BBOL indicates that the backbone of GraphCL replaced with BYOL. The best result is bolded, and the second is underlined.
Backbone
View
MUTAG
PTC-MR
IMDB-B
IMDB-M
COLLAB
NCTI
GraphC1
SimCLR
Best
86.8
71.1
71.3
77.8
BYOL
NodeDrop
90.4
60.5
73.7
51.6
73.3
78.7
BYOL
EdgePert
90.5
59.9
73.9
50.8
71.2
78.5
GraphCL-BYOL
BYOL
Subgraph
89.1
59.8
72.9
51.2
72.4
78.5
BYOL
AttrMask
89.9
59.6
73.6
50.4
71.4
79.1
BYOL
Best
90.5
60.5
73.9
51.6
73.3
79.1
GASSL (ours)
BYOL
Adversarial
90.9
64.6
80.2
gain from backhone (SimCL.R - BYOL)
3.7
2.8
2.0
1.3
gain from view (GraphCL -> Adversarial)
0.4
4.1
0.3
0.1
4.7
0.9
best result as a baseline and compare it with our approach. Our GASSL obtained boost ranging from 0.1% 4.7% using adversarial training. It is worth mentioning that compared GraphCL, approach improves 6.7% on COLLAB, which backbone contributes 2.0% and view contributes 4.7%. The improvement in classification performance indicates the effectiveness of adversarial training.
Performance step size BBBP
Performance vs. perturbation bound BBBP
72
GASSL(ours)
GAS5L(ours)
ROC-AUC AUC
ROC-AUC
GCN
88
20
caled
coog
dasa
aond
a22
dang
deti
cooe
mntator
Figure 2: Effects of step size a and perturbation bound on BBBP dataset.
Effect of step size a and perturbation bound Step size a and perturbation bound are critical factors of adversarial training. We evaluate the effect of a and on the classification accuracy of the BBBP dataset. We separately vary a and in the range {0.00025, 0.0005, ... 64, 128), while fix the other as 0.008. From Figure 2, the algorithm achieves the best classification performance when S 0.008. As the step size increases, the classification performance gradually decreases. Observing the perturbation bound c, we find that the algorithm performance changes similarly. As mentioned in Section 3.2, the perturbations should be challenging and faithful. An overly large step size a or perturbation bound leads to perturbed samples that deviate too much from the input graph, and the encoder can hardly learn a useful representation. the following experiments, we set a = é = 0.008.
4.5
Comparison with state-of-the-arr
4.5.1 Results on TU datasets
The results shown in Table 3 suggest that GASSL achieves sate--of-the-ar results with respect to unsu- pervised models. For example, on MUTAG it achieves 90.9% accuracy, 1.9% relative improvement over the previous ofate-of-the-art For kernel methods, our approach achieves better performance all datasets. When compared to supervised baselines individually, our model outperforms Graph- SAGE in all datasets and outperforms GCN in 4 out of 6 datasets, e.g., a 4.3% relative improvement on GCN for the MUTAG dataset.
Our approach outperforms the state-of-the-ee contrastive learning approaches. For example, com- pared MVGRI., GASSL has relative improvement of 2.55% on average across all datasets. GASSL outperforms GraphCL and GCC with relative improvement of 4.08% and 5.31%, respectively.

---

Table 3: Graph classification results test sets. The reported results and standard deviation over 5 different runs. The compared numbers are from the corresponding papers under the experiment settings.
Dataset
MUTAG
PTC-MR
IMDB-B
IMDB-M
COLLAB
NCII
SP (49
(49]
85.2 1: 2.4
58.2 1 2.4
35.610.2
38.0 1 0.3
Kemel
GK [50
81.7 + 2.1
57.3
1.4
65.9 + 1.0
43.9 + 0.4
72.8 + 0.3
62.3 0.3
WL (51
80.7 3.0
5.5
72.3 3.4
47.0 0.5
80.0 0,5
DGK (48)
87.4 2.7
2.66
6.66
44.6 1 0.5
73.1 0.3
62.5 1 0.3
MI.G (52
87.9 1.6
63.3
66.6 0.3
41.2 0.0
supervised
GraphSAGEE(3)
7.6
6.79
5.3
50.9 2.2
77.7 = 1.5
GCN (4)
5.8
6.33
74.0 3.4
$1.9 3.8
79.0 1 1.8
80.2 2.0
GIN-O (55)
5.6
64.6
7..1+
52.3 2.8
80.2 1.9
82.7 1.7
GIN-c
6.0
6..7
74.3
52.1
80.1 1.9
1.6
GAT (28
6.1
66.7
70.5 2.3
47.8 3.1
Random Walk
83.7 1.5
1.3
50.7 0.3
34.7 + 0.2
node2vee
(54]
72.6 10.2
8.0
54.9 1.6
sub2vec
61.1 15.8
6..0
$
36.7
52.8
1.5
unsupervace
graph2vec (56
9.6
6.9
7..1
5.94
73.2
1.8
InfoGraph
1.1
1.
73.0
4.5
70.6
73.8
0.7
MVGRL (14)
+.17
62.5
74.2
0.7
51.2+
71.3
+
1.2
75.0
0.7
GraphCL (15)
86.8
71.1 0.4
71.3
11.3-10
1.1
77.8
11.8:01 0.4
GCC ((6))
86.4
1.24
1.99
48.9 1 0.8
75.2
1
0.3
66.9
0.2
GASSL-GCN (ours)
90.4 7.9
62.2 6.0
72.7 0.7
49.6 + 2.3
77.9 + 2.0
77.0
1.9
GASSL-GIN (ours)
90.9 + 7.9
64.6 6.1
74.2 = 0.5
51.7 = 2.5
78.0
+ 2.0
80.2 = 1.9
Table 4: Graph classification results on test sets. The reported results are mean and standard deviation over five different runs. The compared numbers are from the corresponding papers under the same experiment settings. The encoder uses GCN combined with sum pooling, and GASSL-H and GASSL-X -X denote perturbation at the encoder's first hidden layer and input layer, respectively.
Datasct GCN[4] GIN 5
HIV
Tox21
ToxCast
BBBP
76.81 + 1.01
75.04 1 0.80
60.63 1 0.51
65.47 1: 1.73
75.95 + 1.35
73.27 + 0.84
60.83 = 0.46
67.65 + 3.00
StructPool 29
75.85 + 1.81
75.43
0.79
62.17 1.61
67.01 2.65
MinCutPool 30
75.37 + 2.05
75.11 0.69
62.48 = 1.33
65.97 1.13
GMT
77.56 + 1.25
77.30
+ 0.59
65.44 + 0.58
68.31 1.62
GASSL-XX X (ours)
78.67 # 1.23
74.60 = 0.76
61.72 = 0.34
70 70.46 1.21
GASSL- (ours)
78.68 + 1.16
74.59 + 0.81
61.96 = 0.55
70.57 + 1.25
4.5.2 Results OGB datasets
We evaluate our method GASSL on OGB datasets. From Table 4 we observed that perturbing the first hidden layer (GASSL-H) yields a slight performance gain compared to perturbing at the input node features (GASSL-X). Our method outperforms GCN and GIN on all datasets, demonstrating our method's potential to outperform supervised learning larger datasets. Compared with the stronger baselines like structurePool and MinCutPool, which exploit the graph structure information. For StructPool, CASSL has 3% and 3.5% gain for the HIV and BBBP datasets. respectively. GASSL outperforms MinCutPool by 3.2% and 4.5% for the HIV and BBBP datasets, respectively. The performance is similar the Tox21 and ToxCast datasets. Our GASSL performs inferior to GMT on Tox21 and ToxCast and superior to GMT on HIV and BBBP. The above results show that our GASSL method can leam good representation of the graph and outperforms even the state-of-the-an supervised learning methods.
4.6 Ablation studies
Effeet of batch size We analyze the sensitivity of the algorithm the batch size on four OGB datascts. We selected batch size from {32, 64, 128, 256, 512, 1024). From the Table observe

---

Table 5: Effect of batch size the test ROC-AUC (%) four OGB datasets, with GCN as the encoder. and H denoting perturbation on the imput layer and the first hidden layer, respectively.
Dataset
HIV
Tox21
ToxCast
BBBP
Batch size
H
X
H
32
77.6
77.6
74.6
60.8
62.0
66.5
68.9
64
77.1
77.4
72.3
74.6
61.0
61.9
65.3
69.5
128
78.7
78.7
73.6
73.3
60.4
60.9
66.1
70.6
256
75.5
77.0
71.9
73.0
60.9
61.2
66.0
67.9
512
75.9
76.5
71.1
72.3
61.3
60.6
65.5
67.2
1024
75.2
75.1
70.9
71.5
61.2
59.6
61.2
65.4
Average
76.7
76.8(+0.)
72.4
73. ((+8))
60.9
61.0++0.1
65.1
67.1(-20)
that GASSL performs stably under different batch sizes. The performance gradually decreases as the batch size increases. In particular, GASSL performs well for a batch size of 128. The benefit is that GASSL can be trained with fewer resources. Moreover, the perturbation the first hidden layer output consistently leads to better test accuracy than perturbation the input node features.
Table Effect of the number of GNN layers and embedding dimension on the test ROC-AUC (%)
on four OGB datasets, with GCN encoder.
Layers
HIV
Tox21
ToxCast
BBBP
Dimension
HIV
Tox21
ToxCast
BBBP
2
78.7
73.3
60.9
70.6
128
74.8
73.3
60.9
70.6
3
73.8
72.2
60.3
69.6
256
75.5
73.6
61.0
66.5
4
72.1
72.5
59.2
63.0
512
78.7
72.2
61.1
63.8
5
71.8
72.3
60.5
63.8
1024
77.2
72.1
60.0
66.9
Effect of the number of GNN layers We evaluate the effect of the number of layers the classification accuracy using ROC-AUC performance on the OGB dataset, using GCN as the encoder, and selecting the number of layers from (2, 5) respectively. From Table 6 can observe that the performance of GASSL gradually decreases as the number of layers increases, while the best performance is obtained when using 2-layer encoder.
Effect of embedding dimension We test the effect of encoding dimensions on classification accuracy the OGB datasets. We choose the best encoding dimension among {128, 256, 512, 1024). From Table we observe that for HIV, the test accuracy increases as the encoding dimension increases. We set 512 for HIV and 128 for the rest, considering the computational efficiency.
Table Effect of ascent steps T the accuracy and training cost (in seconds) for 200 epochs on MUTAG and IMDB-MUNE datasets.
Encoder
MUTAG
Cost(s)
Speed-up
IMDB
Cost(s)
Speed-up
89.8 + 5.8
37
1x
51.5 + 2.3
310
1x
GIN
89.3 +
1.08x
51.2 2.2
231
1.34x
90.9 + 7.9
30
1.23x
51.7 + 2.5
211
1.46x
Effect of ascent steps We explored the impact of ascent steps T on the performance of graph classification accuracy. We train the model in the same setting and vary T E 2,33, From Table 7 we observed that our method achieves stable performance
test accuracy. When T = 3, for the IMDB-M dataset, there is improvement in test accuracy along with speedup nearly times. The results for other datasets are similar and are detailed in the Appendix.

---

5 Conclusion and Future work
In this paper, we explore a novel problem of how learn graph representations without human super- vision. We propose an adversarial self-supervised learning framework (GASSL) that automatically generates views using adversarial training. Our approach adversarially generates challenging views to train a supervisde model. We obtain performance gain by generating views through adversarial training compared handcrafted views. We use a gradient accumulation training method to improve the training efficiency. We conduct extensive experiments on ten datasets. The results show that our method outperforms state-of-the-art graph self-supervised learning and is close to the performance of the supervised GNNs.
For potential negative societal impact, the graph representations can be extended many fields, such as financial networks, molecular biology. The use of transformations generated by adversarial perturbations does not certainly produce meaningful views. Expert knowledge is also required for domain-speciin applications. The limitation of our approach falls in that it exploits the uniformly norm- norm-bounded perturbation and ignores the distribution of the data. Besides, taking full advantage of the existing expert knowledge is the potential to improve performance.
In the future, we will explore the following directions: (1) Explore how effectively combine adversarial training with existing handcrafted views to enhance performance further. (2) Theoretically analyze the use of adversarial training to improve the performance of downstream tasks. (3) Explore non-uniform norm-bounded perturbations on the graph to generate adversarial samples.
Acknowledgments and Disclosure of Funding
This work was partially supported by the National Natural Science Foundation of China (No. 91948303-1, No. 61803375, No. 12002380. No. 62106278, No. 62101575, No. 61906210, No. 91648204) and the National University of Defense Technology Foundation (No. ZK20-09, No. ZK20-52). We would like to thank the anonymous reviewers for their valuable suggestions.
References
[1] William L. Hamilton, Rex Ying, and Jure Leskovec. Representation learning on graphs: Methods and applications. IEEE Data(base) Engineering Bulletin, 40:52-74, 2017.
[2] Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. In International Conference on Learning Representations, 2016.
[3] William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31 International Conference Neural Information Processing Systems, NIPS' 17, page 1035-1035, Red Hook, NY, USA, 2017. Curran Associates Inc
[4] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference Learning Representations, 2017.
[5] Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanie Jegelka. How powerful are graph neural networks. In International Conference on Learning Representations, 2018
[6] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning, pages 5449-5458, 2018
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML 2020: 37th International Conference on Machine Learning, volume 1, pages 1597-1607, 2020.
[8] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. CoRR, abs//191.00577 2019.
[9] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Ghesh- laghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your
10

---

own latent: A new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, 2020.
[10] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey E. Hinton. Big self-super models are strong semi- supervised learners. In Advances in Neural Information Processing Systems, volume 33, pages 22243- -22255,, 2020.
[11] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Thirty-fourth Conference on Neural Information Processing Systems (NeurIPS), volume 33, pages 99 1912-9924, 2020.
[12] Xinlei Chen, Haoqi Fan, Ross B. Girshick, and Kaiming He. Improved baselines with momen- tumn contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[13] Priya Goyal, Mathilde Caron, Benjamin Lefaudeux, Min Xu, Pengchao Wang, Vivek Pai, Mannat Singh, Vitaliy Liptchinsky, Ishan Misra, Armand Joulin, and Piotr Bojanowski. Self- supervised pretraining of visual features in the wild. arXiv preprint 1rXiv:2103.01988 2021.
[14] Kaveh Hassani and Amir Khasahmadi. Contrastive multi-vie view representation learning graphs. In Proceedings of the I 17th International Conference on Machine Learning (C2ML)020), 06 2020.
[15] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 5803-5815. Curran Associates, Inc., 2020.
[16] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. Gec: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 200 page 1150-1160. New York, NY, USA, 2020. Association for Computing Machinery.
[17] Fan- Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2019.
[18] Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In International Conference Learning Representations (ICLR), 2020.
[19] Yu Rong, Yatao Bian, Tingyang Xu, Weiyang Xie, Ying WEI, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer large-scale molecular data. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 22559-12574
Curran Associates, Inc., 2020.
[20] Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020, WWW '20, page 259-270, New York, NY, USA, 2020. Association for Computing Machinery.
[21] Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. Freelb: Enhanced adversarial training for natural language understanding. In Eighth International Conference Learning Representations, 2020.
[22] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan L. Yuille, and Quoe V. Le. Adversarial examples improve image recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 819-828, 2020.
[23] Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. Smart: Robust and efficient fine-tuning for pre-trained natural language models through princi- pled regularized optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2177-2190, 2020.

---

[24] Kezhi Kong, Guohao Li, Mucong Ding, Zuxuan Wu, Chen Zhu, Bernard Ghanem, Gavin Taylor, and Tom Goldstein. Flag: Adversarial data augmentation for graph neural networks. In arxiv:cs.I.G 2021.
[25] Ali Shafahi, Mahyar Najibi, Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S. Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! In Advances Neural Information Processing Systems, 2019.
[26] Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs, 2020.
[27] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. In Advances in Neural Information Processing Systems, volume 33, pages 12118-22133, 2020.
[28] Petar Velikovi, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Y. Ben- gio. Graph attention networks. In International Conference on Learning Representations, 2018.
[29] Hao Yuan and Shuiwang Ji. Structpool: Structured graph pooling via conditional random fields. In ICLR 2020 Eighth International Conference on Learning Representations, 2020.
[30] Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In ICML 2020: 37th International Conference on Machine Learning, volume 1, pages 874-883, 2020.
[31] Minki Kang Jinheon Baek and Sung Ju Hwang. Accurate learning of graph representations with graph multiset pooling. In ICLR 2021, 2021.
[32] Chih-Hui Ho and Nuno Nvasconcelos. Contrastive learning with adversarial examples. In H. Larochelle, M. Ranzato, R. Hadsell, M. F, Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17081-17093, Curran Associates, Inc., 2020.
[33] Minseon Kim, Jihoon Tack, and Sung Hwang. Adversarial self-supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2983-2994, Curran Associates, Inc., 2020.
[34] Alex Tamkin, Mike Wu, and Noah Goodman. Viewmaker networks: Learning views for unsupervised representation learning. In ICLR 2021: The Ninth International Conference on Learning Representations, 2021.
[35] Qianjiang Hu, Xiao Wang, Wei Hu, and Guo-Jun Qi. Adeo: learning of unsupervised representations from self-trained negative adversaries, 2021.
Adversarial contrast for efficient
[36] Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. Deep graph infomax. In International Conference on Learning Representations, 2018.
[37] Jiaqi Zeng and Pengtao Xie. Contrastive self-supervised learning for graph classification. arXiv preprint arXiv:2009.05923 2020.
[38] Zhen Peng, Yixiang Dong, Minnan Luo, Xiao-Ming Wu, and Qinghua Zheng. Self-supervised graph representation learning via global context prediction. arXiv preprint arXiv:2003.0160) 2020.
[39] Yizhu Jiao, Yun Xiong, Jiawei Zhang, Yao Zhang, Tianqi Zhang, and Yangyong Zhu. Sub-graph contrast for scalable self-supervised graph representation learning. In 2020 IEEE International Conference Data Mining (ICDM), pages 222-231. 2020.
[40] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006. 200,0000,,,,,,,, 2020.
[41] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW 2021: The Web Conference, 2021.
12

---

[42] Yixin Liu, Shirui Pan, Ming Jin, Chuan Zhou, Feng Xia, and Philip S. Yu. Graph self sulf-vuue learning: A survey, 2021.
[43] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
[44] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7472-7482, 09-15 Jun 2019.
[45] Nils Kriege and Petra Mutzel. Subgraph matching kernels for attributed graphs. In Proceedings of the 17th International Conference on Machine Learning (ICML 2012). Morgan Kaufmann, 2012.
[46] C. Helma, R. D. King, S. Kramer, and A. Srinivasan. The predictive toxicology challenge 2000-2001. Bieinformatics, 17(1):107-108, 2001.
[47] Nikil Wale and George Karypis. Comparison of descriptor spaces for chemical compound retrieval and classification. In Sixth International Conference Data Mining (ICDM (06), volume 14, pages 347-375, 2006.
[48] Pinar Yanardag and S.V.N. Vishwanathan. Deep graph kemels. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '15, page 1 1365-1374, New York, NY, USA, 2015. Association for Computing Machinery.
[49] K. M. Borgwardt and H. P. Kriegel. Shortest-path kernels on graphs. In IEEE International Conference on Data Mining, 2006.
[50] Nino Shervashidze, S. N. Vishwanathan, Tobias H. Petri, and Al Kurt Mehlhorn. Efficient graphlet kernels for large graph comparison. Aistats, 5:488-488 495, 2009.
[51] Nino Shervashidze, Pascal Schweitzer, Erik Jan, Van Leeuwen, and Karsten M. Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(3):2539-256 2561,
2011.
[52] Risi Kondor and Horace Pan. The multiscale laplacian graph kernel. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29, pages 2990-2998. Curran Associates, Inc., 2016.
[53] Thomas Gärtner, Peter Flach, and Stefan Wrobel. On graph kernels: Hardness results and efficient alternatives. In Conference On Learning Theory, pages 129-143, 2003.
[54] Aditya Grover and Jure Leskovec. Node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD 16., page 355,864, New York, NY, USA, 2016. Association for Computing Machinery.
[55] Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B. Prakash. Sub2Vec: Feature Learning for Subgraphs, pages 170-182. 06 2018.
[56] Annamalai Narayanan, Chandramohan Mahinthan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. In 13th International Workshop on Mining and Learning with Graphs, 07 2017.
13


Reviews:
[{"review_id": "z7BSkUQfvdC", "reviewer": "Reviewer_rGkH", "summary": "The paper investigates the unsupervised representation of graph data and proposes an adversarial self-supervised learning (GASSL) framework. The paper adopts the self-supervised framework in reference [12] for graph embedding representation, and apply adversarial learning to improve the robustness of the model, following the framework in [26]. A number of experimental results are given to validate the performance of the proposed model. The contribution of the paper is incremental.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem addressed in the paper is significant and interesting to the readers of the conference. The main contribution of the paper is to extend the self-supervised learning framework to graph data and to adopt adversarial learning to improve the robustness of the model. Both self-supervised learning framework and adversarial learning model used in the paper are not original and available in the reference. The novelty of the paper is to combine these two models to learn a robust graph representation. \nThe paper provides a number of experimental results to support that good performance of the proposed approach and ablations experimental results are also given, but it lacks of sufficient evidence that why the proposed method performs well. The reviewer did not see implications from experimental results. In particular, the paper claims that “We use adversarial training to automatically generate challenging views for self-supervised learning in place of handcrafted views, which yield performance gains on multiple datasets.”  From the procedure of adversarial generator, the model is to generate an adversarial attack in the vicinity of the input sample that maximizes its loss function, how to guarantee that the generated sample should be the different view of the same instance? What is the definition of \"challenging views\".\nEnglish writing is clear and easy to understand. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would like to suggest that the authors provide to visualize the feature distribution of graph representation to validate results. \nSome minor comments: \nL84-87. Some paragraphs need to be rewritten. The paper states \" For a thorough review, we refer the reader to the recent survey [50] \", and then gives two references later. \nL104.  \"[26] proposed \" ==> \"Authors [26]  proposed \"\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "30"}, "extra_text": {}}, {"review_id": "OUUgDaMAQ2u", "reviewer": "Reviewer_nLuy", "summary": "This paper presents a self-supervised learning method GASSL for graph representation learning. The authors have designed 2 key points:  a) Utilizing BYOL framework to learn graph representation; b) Using adversarial training to automatically generate challenging views for self-supervised learning. The idea seems to be simple but effective. Sufficient experimental analyses are conducted.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1.\tTo the best of my knowledge, it is the first time that BYOL has been applied for graph self-supervised learning. While this point seems minor, its benefit is greatly supported by the experiments in Table 2 compared to SimCLR.\n\n2.\tDifferent from Euclidean data like images, how to generate valuable positive samples for graphs is non-trivial and central in the domain. Traditional methods like GCC and GraphCL resort to hand-crafted sampling which easily incurs bias during training. This paper proposes to apply adversarial noise to the input features or hidden layers to generate worse-cast positive samples, with the goal of enhancing robustness for BYOL training. The idea is simple yet interesting.\n\n3.\tExperimental evaluations are good on the conducted benchmarks and sufficiently support the claims the authors have made. \n\n\nWeaknesses\n\n1.\tThis paper is almost well written. Yet there still remains certain unclarity. In line 123-124, the authors mentioned that “Our method relies on the positive view of the input graph, which commonly contains: node dropping, edge perturbation, subgraph, and these need to be designed manually.” This statement is quite confusing. Does this mean that the proposed adversarial technique is not self-contained and is performed on the perturbed samples generated by existing data augmentation methods? If so, why has the claim “We use adversarial training to automatically generate challenging views for self-supervised learning in place of handcrafted views, which yield performance gains on multiple datasets.”\n\n2.\tThe authors mentioned that the experiments are conducted on large-scale OGB, which is over-exaggerated. It is true that OGB contains large datasets for benchmarking, but the datasets that applied in this paper are small. Please correct the sentence to make it more professional. \n\n3.\tIt is well known that involving adversarial objective will hinder the training dynamics. Have the authors met this issue and how to address it if so. It is suggested to provide the convergence curves for better clarification.\n\n4.\tRegarding the comparison between initial Perturbation and hidden Perturbation, what about the ablation by considering them both? And what about performing perturbation on other layers (not just the first hidden layer)?\n\n5.\tSeveral important references on graph self-supervised learning are not discussed and cited. See below.\n\n\n[A] Strategies for pre training graph neural networks, ICLR 2020.\n[B] Self-Supervised Graph Transformer on Large-Scale Molecular Data, NIPS 2020.\n[C] Graph Representation Learning via Graphical Mutual Information Maximization, WWW 2020.\n\n\n##########Post Rebuttal###########\nAfter reading the authors' responses carefully (including their replies to my further questions), my concerns are well addressed. I believe the contribution by this paper is novel and valuable, and it does address a crucial issue in graph self-supervised learning. In terms of the writing, I strongly suggest the authors to keep their words and revise their paper according to their responses. \n\nOverall, I decide to increase my score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the weakness part above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~5h"}, "extra_text": {}}, {"review_id": "KbBAtQK1Tkz", "reviewer": "Reviewer_9kp5", "summary": "This paper proposes to utilize the methodology of adversarial training to help the SSL problem in GNN. It establishes some basic formulations for the problem and validate the effectiveness on benchmark datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Pros\n1) This paper is well-written. The idea is presented in a clear and straightforward manner.\n2) The idea aligns with the common understandings about adversarial training. For instance, adversarial training is also applied as an data augmentation method in NLP [1].\n3) The proposed method is evaluated on many benchmark datasets.\n\n## Cons\n1) Although the topic is interesting and insightful, the proposed method is somewhat primitive. \n2) The improvements is relatively marginal. Sometimes the improvement is less than 0.5%, which may well be noise. The variance shown in Table 3 also suggests that the improvement is not very stable.\n3) If I was right, this paper only provide evaluations on the graph classification task but omit the commonly used node classification task. Please explain.\n4) The proposed method should be compared with deep graph infomax. The [44] cited paper in this work.\n5) I am curious about the robustness of the proposed method. Can the author provide corresponding results [2][3]?\n\n\nI may raise my score if the authors can properly answer my questions.\n\n[1] Adversarial Mutual Information for Text Generation\n\n[2] N. Entezari, S. A. Al-Sayouri, A. Darvishzadeh, and E. E. Papalex-akis,  “All  you  need  is  low  (rank)  defending  against  adversarialattacks on graphs,” inWSDM, 2020, pp. 169–177\n\n[3] H.  Wu,  C.  Wang,  Y.  Tyshetskiy,  A.  Docherty,  K.  Lu,  and  L.  Zhu,“Adversarial  examples  on  graph  data:  Deep  insights  into  attackand defense,” inIJCAI, 2019\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "None", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "9DFt4Ro3DA5", "reviewer": "Reviewer_ry4V", "summary": "This paper introduces a novel way to improve unsupervised graph representation learning. The proposed method utilizes the adversarial attacking techniques with a teacher and a student network to refine the generalization of graph representation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ This paper proposes an interesting solution of unsupervised representation learning of the graphs. The work is solid, and it contains some novelty.\n+ The experiments show that their methods were evaluated widely on 10 popular datasets and the results of the experiments showed significant improvement with the existing benchmark results.\n \n \nThis paper contains some uncleared descriptions and concerns. There are a few questions as listed below:\n \n\t1. Why do you add the perturbations to hidden layer 1 instead of other layers or the generated representation? Is there any discussion or comparison about this choice? \n\t2. Eq3. is probably confused to readers. Could you provide more details? \n\t3. In L 125, the definition of G and G' are unclear. What is G and G' exactly? \n\t4. For the experiments in Table 3, why do you compare your unsupervised learning result with supervised and kernel results? \n\t5. Table 4 shows some inferior results on some datasets. Is there any possible analysis and reasons? \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors solved some limitations but it still needs more explorations in the related area.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the problem of self-supervised learning in the context of graph representation learning. It proposes to adopt the technique of adversarial training to automatically augment a training set, and then devises the corresponding adaptation scheme to make adversarial training viable. The authors conducted thorough and insightful experiments (also supplementing some experimental results in the rebuttal) on several benchmark datasets. 

Although adversarial training was originally proposed as a defensive algorithm aiming at increasing the robustness of a certain learning model, it has also been recognized as a variant of data augmentation or hard example mining in many other works. Moreover, employing the philosophy of adversarial training to help generalization has also been discussed in other fields like NLP and style transformation. Thus, it is reasonable to see that the same theory can be verified in the domain of graph learning. Of course, verifying this theory in a new domain like graph learning requires huge efforts and insightful designs, which holds as the main contribution of this work. The proposed method is relatively simple but very effective on the evaluated tasks. In the rebuttal phase, the authors adequately answered the questions from the reviewers, including addressing the issues of writing clarity and evaluations in extra experimental settings. Eventually, the four reviewers reached the consensus of accepting the paper. Therefore, the AC recommends acceptance as poster regarding this submission. 
Decision: Accept (Poster)