Adversarial Graph Augmentation to Improve Graph Contrastive Learning
Authors: Susheel Suresh
Pan Li
Cong Hao
Jennifer Neville
Venue: NeurIPS 2021 Poster (2021)
Keywords: graph contrastive learning
self-supervised graph representation learning

Abstract: Self-supervised learning of graph neural networks (GNN) is in great need because of the widespread label scarcity issue in real-world graph/network data. Graph contrastive learning (GCL), by training GNNs to maximize the correspondence between the representations of the same graph in its different augmented forms, may yield robust and transferable GNNs even without using labels. However, GNNs trained by traditional GCL often risk capturing redundant graph features and thus may be brittle and provide sub-par performance in downstream tasks. Here, we propose a novel principle, termed adversarial-GCL (\textit{AD-GCL}), which enables GNNs to avoid capturing redundant information during the training by optimizing adversarial graph augmentation strategies used in GCL. We pair AD-GCL with theoretical explanations and design a practical instantiation based on trainable edge-dropping graph augmentation. We experimentally validate AD-GCL by comparing with the state-of-the-art GCL methods and achieve performance gains of up-to~14\% in unsupervised, ~6\% in transfer and~3\% in semi-supervised learning settings overall with 18 different benchmark datasets for the tasks of molecule property regression and classification, and social network classification.

Adversarial Graph Augmentation to Improve Graph Contrastive Learning
Susheel Suresh Purdue University
Pan Li" Purdue University panli@purdue edu
Cong Hao Georgia Tech callie. hww@gatech.
suresh43@purdue. edu
Jennifer Neville
Purdue University and Microsoft Research jemeeville@microoffff com
Abstract
Self-supervised learning of graph neural networks (GNN) is in great need because of the widespread label scarcity issue in real-world graph/network data. Graph contrastive learning (GCL), by training GNNs to maximize the correspondence between the representations of the same graph in its different augmented forms, may yield robust and transferable GNNs even without using labels. However, GNNs trained by traditional GCL often risk capturing redundant graph features and thus may be brittle and provide sub-par performance in downstream tasks. Here, we propose a novel principle, termed adversarial -GGL (AD-GCL), which enables GNNs to avoid capturing redundant information during the training by optimizing adversarial graph augmentation strategies used in GCL. We pair AD-GCL with theoretical explanations and design practical instantiation based trainable edge-dropping graph augmentation. We experimentalyy validate AD-GCI by comparing with the state-of-the-art GCL methods and achieve performance gains of up-to 14% in unsupervised. 6% in transfer, and 3% in semi-supervised learning settings overall with 18 different benchmark datasets for the tasks of molecule property regression and classification, and social network classification.
1 Introduction
Graph representation learning (GRL) aims to encode graph-structured data into low-dimensional vector representations, which has recently shown great potential in many applications in biochemistry, physics and social science [3.. Graph neural networks (GNNs), inheriting the power of neural networks 5), have become the almost de facto encoders for GRL 9 GNNs have been mostly studied in cases with supervised end-to-end training (10-16), where a large number of task-specific labels needed. However, in many applications, annotating labels of graph data takes lot of time and resources [1] e.g., identifying pharmacological effect of drug molecule graphs requires living animal experiments [19]. Therefore, recent research efforts are directed towards studying self-supervised learning for GNNs, where only limited or even no labels are needed [18] 31)
Designing proper self seff--suprvised-leaamaee tion of graph-structured data will be captured by GNNs and may heavily impact their performance in downstream tasks. Many previous works adopt the edge-reconstruction principle to match traditional network-embeddeee requirement (32-35). where the edges of the input graph are expected to be recon- Experiments showed that these GNN models learn to over-emphasize node proximity [3 and may lose subtle but crucial structural information, thus
principles for GNNs is crucial, they drive what informa-
structed based the output of GNNs 20 21 36
failing in many tasks including node-role classification [16 35 37 38 and graph classification
Pan Li and Jemnifer Neville co correspond this work. hwtp://///.cccsee///aa..o..o
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Original
AD-GCL: min max max [TC)) where t(G) ~ T(G)
ENN- encoder
Project
SNN augmerter
Learnable Graph Data Augmentation T(-)
Minimize Information
x(-)
Maximize
Information
MLP
Way
May3
M4.4
M4,2
Sample ((C)
GNN- enczder
Project Heod
Hade Lraeddinge
Goge Grbaading
temwel
Edge-dropping Instantiation
D Drasped Enges
The AD-GCL. principle and its instantiation based on learnalle edge-dropping augmentation. AD- GCL contains two components for graph data encoding and graph data augmentation. The GNN encoder f(.) maximizes the mutual information between the original graph G and the augmented graph ((G) while the GNN augmenter optimizes the augmentation T(-) to remove the information from the original graph. The instantiation of AD-GCL proposed this work uses edge dropping: An edge of G is randomly dropped according to Bernoull (wie), where we is parameterized by the GNN augmenter.
Figure 1:
To avoid the above issue, graph contrastive learning (GCL) has attracted more attention recently [18 23 GCL leverages the mutual information maximization principle (InfoMax) (39 that aims to maximize the correspondence between the representations of graph (or node) in its different augmented forms [18 24 25 28 31]. Perfect correspondence indicates that a representation precisely identifies its corresponding graph (or node) and thus the encoding procedure does not decrease the mutual information between them.
However, researchers have found that the InfoMax principle may be risky because it may push encoders to capture redundant information that irrelevant to the downstream tasks: Redundant information suffices to identify each graph to achieve InfoMax but encoding it yields brittle represen- tations and may severely deteriorate the performance of the encoder the downstream tasks [40]. This observation reminds us of another principle, termed information bottleneck (IB) 461. As opposed to InfoMax, IB asks the encoder to capture the minimal sufficient information for the down- stream tasks. Specifically, IB minimizes the information from the original data while maximizing the information that is relevant to the downstream tasks. As the redundant information gets removed, the encoder learnt by IB tends be more robust and transferable. Recently, IB has been applied to GNNs 47 But IB needs the knowledge of the downstream tasks that may not be available.
Hence, a natural question emerges: When the knowledge of downstream tasks are unavailable, how to train GNNs that may remove redundant information? ? Previous works highlight some solutions by designing data augmentation strategies for GCL but those strategies are typically task-related and sub-optimal. They either leverage domain knowledge [25] 28 30), e.g., node centralities in network science molecule motifs in bio-chemistry, depend extensive evaluation the downstream tasks, where the best strategy is selected based on validation performance [24 30).
In this paper, we approach this question by proposing novel principle that pairs GCL with adversarial training, termed AD-GCL, as shown in Fig T We particularly focus on training self supe-super GNNs for graph-level tasks, though the idea may be generalized for node-level tasks. AD-GCL consists of two components: The first component contains a GNN encoder, which adopts InfoMax to maximize the correspondence/mutuall information between the representations of the original graph and its augmented graphs. The second component contains a GNN-based augmenter, which aims to optimize the augmentation strategy to decrease redundant information from the original graph as much as possible. AD-GCL essentially allows the encoder capturing the minimal sufficient information to distinguish graphs in the dataset. We further provide theoretical explanations of AD-GCL. We show that with certain regularization on the search space of the augmenter, AD-GCL can yield a lower bound guarantee of the information related to the downstream tasks, while simultancously holding an upper bound guarantee of the redundant information from the original graphs, which matches the aim of the IB principle. We further give instantiation of AD-GCL: The GNN augmenter adopts task-agnostic augmentation strategy and will learn an input-graph grapperrphhddeeddenntt non-uniform -dgenn---de--ropp probability perform graph augmentation.
Finally, we extensively evaluate AD-GCL on 18 different benchmark datasets for molecule property classification and regression, and social network classification tasks in different setting viz. unsuper-

---

transfer learning (Sec. 5.3) and semi-supervised learning (Sec. 5.4 learning AD-GCL achieves sigmificant performance gains in relative improvement and high mean ranks over the datasets compared to state-of-the-ar baselines. We also study the theoretical aspects of AD-GCI. with apt experiments and analyze the results to offer fresh perspectives (See. 5:2 Interestingly, we observe that AD-GCL outperforms traditional GCL based on non-optimizable augmentation across almost the entire range of perturbation levels.
vised learning (Sec. 5.1.
2 Notations and Preliminaries
We first introduce some preliminary concepts and notations for further exposition. In this work, we consider attributed graphs G = (V, E) where V is node set and E is edge set. G may have node attributes (X, E R v E }} and edge attributes (Xe IR E E} of dimension F. We denote the set of the neighbors of node u as No.
Learning Graph Representations. Given a set of graphs G;, i 1,2, 1,2,... n, in some universe G. the aim is to learn encoder f g - IRd, where ((,, can be further used in some downstream task. We also assume that G,'s are all IID sampled from an unknown distribution Pg defined over G. In downstream task, each G; is associated with a label y/ V. Another model q : RR + y will be learnt to predict Y; based on q(f(G))) We assume (Ge, Y:)'s s are IID sampled from a distribution Fgxy gy where Pyng is the conditional distribution of the graph label in the downstream task given the graph.
Graph Neural Networks (GNNs). In this work, we focus on using GNNs, message passing GNNs in particular [49]. as the encoder f. For a graph G = (V, E)), every node v E V will be paired with a node representation bn initialized h X.. These representations will be updated by GNN. During the hthh iteration, each hy (R-1)) is updated using e's neighbourhood information expressed as,
h
UPDATE()) (1(111) AGGREGATE(") (((h!--),,
E N.)
(1)
where AGGREGATE(() is trainable function that maps the set of node representations and edge attributes Xuv to an aggregated vector, UPDATE(.) is another trainable function that maps both v's current representation and the aggregated vector to v's updated representation. After K iterations of Eq. the graph representation is obtained by pooling the final set of node representations as,
f(G) é he POOL(( V V})
For design choices regarding aggregation, update and pooling functions refer the reader to 3
7
The Mutual Information Maximization Principle. GCL built upon the InfoMax principle 339 which prescribes to leamn encoder that maximizes the mutual information the correspondence between the graph and its representation. The rationale behind GCL is that a graph representation f(G) should capture the features of the graph G so that representation can distinguish this graph from other graphs. Specifically, the objective GCL follows
InfoMax: max ((G),, where G Pg.
f
where I(X1;X, denotes the mutual information between two random variables X1 and X₂ [0
Note that the encoder f(-) given by GNNs is not injective in the graph space G due to its limited expressive power [14 15) Specifically, for the graphs that cannot be distinguished by 1-WL test (5T). GNNs will associate them with the same representations. We leave more discussion on 1-WL test in Appendix C In contrast to using CNNs as encoders, one can never expect GNNs to identify all the graphs in & based their representations, which introduces unique challenge for GCL.
3 Adversarial Graph Contrastive Learning
In this section, we introduce our adversarial graph contrastive learning (AD-GCL) framework and one of its instantiations based on edge perturbation.
3.1
Theoretical Motivation and Formulation of AD-GCL
The InfoMax principle in Eq. could be problematic in practice for general representation learning. Tschannen et al. have shown that for image classification, representations capturing the information
3

---

that is entirely irrelevant to the image labels are also able to maximize the mutual information but
such representations are definitely not useful for image classification [40). similar issue can
also be observed in graph representation learning, as illustrated by 2 We consider binary
graph classification problem with graphs in the dataset ogbg-molbace [52]. Two GNN encoders with exactly the same architecture are trained to keep mutual information maximization between graph representations and the input graphs, but one of the GNN encoders in the same time is further supervised by random graph labels. Although the GNN encoder supervised by random labels still keeps one-to-one correspondance between every input graph and its representation (i.e., mutual information maximization), we may observe significant performance degeneration of this GNN encoder when evaluating it over the downstream ground -tutth labels. More detailed experiment setup is left in Appendix G.1
This observation inspires us rethink what a good graph representation is. Recently, the information bottleneck has applied to learn graph representations 48). Specifically, the objective of graph information bottleneck (GIB) follows
0.80
Test RUC
0.75
GIB: max )ax]( BI f(G)),
(4)
0.70
1
0.65
where (G,
Poxy, 3 is positive constant. Compar- ing Eq. and Eq. 4 may observe the different require- ments between InfoMax and GIB: InfoMax asks for maxi- mizing the information from the original graph, while GIB asks for minimizing such information but simultaneously maximizing the information that is relevant to the down- stream tasks. As GIB asks to remove redundant information, GIB naturally avoids the issue encountered 2 Remov- ing extra information also makes GNNs trained w.r.t. GIB ro- bust to adverserial attack and strongly transferrable [47 48
InfoMax
InfoMaxx
0.60
GT Label Label
Raiddom Label
10
20
40
50
60
Train Epochs
Figure 2: Two GNNs keep the mutual information maximized between graphs and their representations. Simultaneously, they get supervised by ground-truth labels (green) and random labels (blue) respec- tively. The curves show their testing perfor- mance predicting ground-trutt labels.
Unfortunately, GIB requires the knowledge of the class labels from the downstream task and thus does not apply to self-supervised training of GNNs where there are few no labels. Then, the question is how to learn robust and transferable GNNs in a self-supervised way.
To address this, we will develop a GCL approach that uses adversarial learning to avoid capturing redundant information during the representation learning. In general, GCL methods use graph data augmentation (GDA) processes to perturb the original observed graphs and decrease the amount of information they encode. Then, the methods apply InfoMax over perturbed graph pairs (using different GDAs) train encoder f to capture the remaining information.
Definition (Graph Data Augmentation (GDA)). For a graph G E G. T(G) denotes graph data augmentation of G. which is distribution defined over conditioned on G. We use t(G) E 9 to denote sample of TG)).
Specifically, given two ways of GDA Ti and T2, the objective of GCL becomes
GDA-GCL: max ((f (G)); f(t (G))), where G Fg, (.() ~ i (1,2). (5)
In practice, GDA processes are often pre-designed based on either domain knowledge or extensive evaluation, and improper choice of GDA may severely impact the downstream performance [17 24
We will review a few GDAs adopted in existing works in Sec
In contrast to previous predefined GDAs, our idea, inspired by GIB, is to learn the GDA process (over a parameterized family), so that the encoder f can capture the minimal information that sufficient to identify each graph.
AD-GCL: We optimize the following objective, over a GDA family T (defined below).
AD-GCL:
min max I([(G); where G ~ ( TET 1
(6)
Definition 2 (Graph Data Augmentation Family), Let T denote family of different GDAs Tx where & is the parameter in some universe. A T.(+) T is specific GDA with parameter &.
The min-max principle in AD-GCL aims to train the encoder such that even with a very aggressive GDA (i.c., where t(G) is very different from G), the mutual information / the correspondence

---

between the perturbed graph and the original graph can be maximized. Compared with the two GDAs adopted in GDA-GCL (Eq AD-GCL views the original graph G as the anchor while pushing its perturbation T(G) as far from the anchor as it can. The automatic search over T E T saves great deal of effort evaluating different combinations of GDA as adopted in (24).
Relating AD-GCL to the downstream task. Next, we will theoretically characterize the property of the encoder trained via AD-GCL. The analysis here not only further illustrates the rationale of AD-GCL but helps design practical T when some knowledge of Y ís accessible. But note that our analysis does not make any assumption on the availability of Y.
Note that GNN' learning graph representations is very different from CNNs learning image representa- tions because GNNs are never injective mappings between the graph universe G and the representation space Rd. because the expressive power of GNNs is limited by the 1-WL test [1 15 51). So, we need to define a quotient space of G based on the equivalence given by the 1-WL. Definition 3 (Graph Quotient Space). Define the equivalence = between two gruphs G1 G2 if G1, G2 cannot be distinguished by the 1-WL test. Define the quotient space G' =
So every element in the quotient space, i.c., G' E G', is a representative graph from a family of graphs that cannot be distinguished by the I-WL test. Note that our definition also allows attributed graphs
Definition 4 (Probability Measures in g'). Define Pg: over the space g' such that (G') P5 = G') for any G' E g'. Further define Pg xy Y') = P GDA T() defined over G, define distribution on G', (G') EG-re [TG)) = G¹) G' E G.
G', Y = =Y'). Given a
Now, we provide our theoretical results and give their implication. The proof is in the Appendix B
Theorem 1. Suppose the encoder I is implemented by a GNN as powerful as the 1-WL test. Suppose G is countable space and thus G' is a countable space. Then, the optimal solution (S" T to AD-GCL satisfies, letting T (G') = Eowzo (T" (G) G*).
1. (G)); S minTer (("(G);;
(G);;) where t'(G') ~ ('))), (G)))) ~ (G", (G,Y) Paxy and (G", Y) Pgxxy.
2. (() Z ((()(")): = and ~ ~PP'
T(t" (G'); )²)); Y), where 1t (G') (G'),, (G,Y) Poxy
The statement in Theorem guarantees a upper bound of the information that is captured by the representations but irrelevant to the downstream task, which matches our aim. This bound has form very relevant to the GIB principle (Eq When 3 = 1). since minreT I (1' (G'); G') - (((" (G"); Y) Z miny[[( G) (G): ()), where is GNN encoder as powerful the 1-WL test. But note that this inequality also implies that the encoder given by AD-GCL may be worse than the optimal encoder given by GIB (B = 1). This makes sense as GIB has the access to the downstream task Y.
The statement 2 in Theorem [ guarantees a lower bound of the mutual information between the learnt representations and the labels of the downstream task. As long as the GDA family T has a good control, (G'): Y) Z minreT (('); Y) and TGf* Y) thus cannot be too small. This implies that it is better to regularize when learning over T. In our instantiation, based on edge-dropping augmentation (Sec. 3.2) we regularize the ratio of dropped edges per graph.
3.2 Instantiation of AD-GCL via Learnable Edge Perturbation
We now introduce practical instantiation of the AD-GCL principle (Eq. 6 based on learnable edge-dropping augmentations as illustrated in Fig. T (See Appendix for a summary of AD-GCL in its algorithmic form.) The objective of AD-GCL has two folds: (1) Optimize the encoder to maximize the mutual information between the representations of the original graph G and its augmented graph t(G); (2) Optimize the GDA T(G) where t(G) is sampled minimize such a mutual information. We always set the encoder as a GNN fe with learnable parameters O and next we focus on the GDA, To (G) that has learnable parameters Φ.
Learnable Edge Dropping GDA model (-).). Edge dropping is the operation of deleting some edges in a graph. As proof of concept, we adopt edge dropping to formulate the GDA family T. Other types of GDAs such as node dropping, edge adding and feature masking can also be paired with our AD-GCL principle. Interestingly, our experiments, edge-dropping augmentation optimized by AD-GCL has already achieved much better performance than any pre-defined random
5

---

GDAs even carefully selected via extensive evaluation [24] (See See Another reason that supports edge dropping due to our Theorem T statement 2. which shows that good GDAs should keep some information related to the downstream tasks. Many GRL. downstream tasks such as molecule classification only depends on the structural fingerprints that can be represented as subgraphs of the original graph [53]. Dropping a few edges may not change those subgraph structures and thus keeps the information sufficient to the downstream classification. But note that this reasoning does not mean that we leverage domain knowledge to design GDA, as the family T is still broad and the specific GDA still needs to be optimized. Moreover, experiments show that our instantiation also works extremely well on social network classification and molecule property regression, where the evidence of subgraph fingerprints may not exist any more.
Parameterizing (.).). For each G EV, set To (G), T E T as a random graph model (5)5 conditioning on G. Each sample ((G) ~ Ta (G) is a graph that shares the same node set with G while the edge set of ((G) is only subset of E. Each edge E E will be associated with random variable ~ Bernouli(we), where is in t(G) if = 1 and is dropped otherwise.
We parameterize the Bernoulli weights we by leveraging another GNN, i.e., the augmenter, to run G according to Eq 1 of layers, get the final-layer node representations (1r5²) |o V} and set
we = MLP((h (K), :n(K)), where = (u, 2) and {h(K) v E E = GNNaaugme))er(()
To train T(' in an end-to-end fashion, we relax the discrete Pe to be a continuous variable in [0. 1] and utilize the Gumbel-Max reparamctrization trick (56. Specifically, Pe Sigmoid (log log(1 - 8) + we)/T), where Uniform(0.1). As temperature typer-parameter T + 0, Pe gets closer to being binary. Moreover, the gradients aw. dropping based on a random graph model has also been used for parameterized explanations of GNNs (58)
aps are smooth and well defined. This style of edge
Regularizing (-).). As shown in Theorem I a reasonable GDA should keep a certain amount of information related the downstream tasks (statement 2). Hence, we expect the GDAs in the edge dropping family TT not to perform very aggressive perturbation. Therefore, we regularize the ratio of edges being dropped per graph by enforeing the following constraint: For a graph G and its augmented graph t(G), we add EeEB we/l /e/lE) EE| to the objective, where we is defined in Eq 7 indicates the probability that gets dropped.
Putting everything together, the final objective is as follows.
min max fe(())) + Avey EG Σ we/|
where G ~ Pg, (G).
eEE
Note & corresponds to the learnable parameters of the augmenter GNN and MLP used to derive the We's 's and O corresponds to the learnable parameters of the GNN f.
Estimating the objective in Eq In our implementation, the second (regularization) term is easy to estimate empirically. For the first (mutual information) term, we adopt InfoNCE as the estimator (1). which is known to be a lower bound of the mutual information and is frequently used for contrastive learning [40 59 62). Specfically, during the training, given minibatch of m graphs [G+ mm let 24,1
Te G and 212 ((Je((()))) where g(-) is the projection head implemented by a 2-layer MIP as suggested in (62). With sim(;-) denoting cosine similarity, we estimate the mutual information for the mini-batch as follows.
Σ log exp(sim(21.1, 21,))
m
I(fe fe((()) +1
4 Related Work
GNNs for GRL is a broad field and gets a high-level review in the Sec. Here, we focus on the topics that most relevant to graph contrastive learning (GCL).
Contrastive learning (CL) 39 59 60 63 65 was initially proposed to train CNNs for image represen-
tation learning and has recently achieved great success [62 66). GCL applies the idea of CL on GNNs. In contrast to the case CNNs, GCL trained using GNNs posts us new fundamental challenges. An image often has multiple natural views, say by imposing different color filters and so on. Hence,

---

different views of an image give natural contrastive pairs for CL to train CNNs. However, graphs are more abstract and the irregularity of graph structures typically provides crucial information. Thus, designing contrastive pairs for GCL must play with irregular graph structures and thus becomes more challenging. Some works use different parts of a graph to build contrastive pairs, including nodes v.s. whole graphs (18 67 nodes v.s. nodes [68), nodes v.S. subgraphs [1] Other works adopt graph data augmentations (GDA) such as edge perturbation [31] to generate contrastive pairs. Recently. GraphCL (24) gives an extensive study on different combinations of GDAs including node dropping, edge perturbation, subgraph sampling and feature masking. Extensive evaluation is required to determine good combinations. MVGRL (25) and GCA [30] leverage the domain knowledge of network science and adopt network centrality to perform GDAs. Note that none of the above methods consider optimizing augmentations. In contrast, our principle AD-GCL provides theoretical guiding principles to optimize augmentations. Very recently, JOAO (70) adopts a bi-level optimization framework sharing some high-level ideas with our adversarial training strategy but has several differences: 1) the GDA search space in JOAO is set as different types of augmentation with uniform perturbation, such uniform edge/node dropping while we allow augmentation with non-uniform perturbation. 2) JOAO relaxes the GDA combinatorial search problem into continuous space via Jensen's inequality and adopts projected gradient descent to optimize. Ours, instead, adopts Bayesian modeling plus reparameterizatioo tricks to optimize. The performance comparison between AD-GCL and JOAO for the tasks investigated in Sec. 5 is given in Appendix H
Tian et al. [7]] has recently proposed the InfoMin principle that shares some ideas with AD-GCL but there are several fundamental differences. Theoretically, InfoMin needs the downstream tasks to supervise the augmentation. Rephrased in our notation, the optimal augmentation TIM (G) given by InfoMin (called the sweet spot in (71) needs to satisfy T(t (G): I(G and [G) = 0, (G) ~ (G), neither of which are possible without the downstream- task knowledge. Instead, our Theorem [ provides more reasonable arguments and creatively suggests using regularization to control the tradeoff. Empirically, InfoMin is applied to CNNs while AD-GCL applied to GNNs. AD-GCL needs to handle the above challenges due to irregular graph structures and the limited expressive power of GNNs [4 which InfoMin does not consider.
5 Experiments and Analysis
This section devoted to the empirical evaluation of the proposed instantiation of our AD-GCL principle. Our initial focus is on unsupervised learning which is followed by analysis of the effects of regularization. We further apply AD-GCL to transfer and semi-supervised learning. Summary of datasets and training details for specific experiments are provided in Appendix E and G respectively.
5.1 Unsupervised Learning
In this setting, an encoder (specifically GIN (72) is trained with different self-supervised methods to learn graph representations, which are then evaluated by feeding these representations to make prediction for the downstream tasks. We use datasets from Open Graph Benchmark (OGB) (52),
TU Dataset (73] and ZINC (74) for graph-level property classification and regression. More details
regarding the experimental setting are provided in the Appendix G
We consider two types of AD-GCL, where one is with a fixed regularization weight Aveg = (Eq(8), termed AD-GCL-FIX, and another is with Aveg tuned over the validation set among (0.1, 0.5, 1.0, 2.0, 5.0, 10.0), termed AD-GCL-OPT. AD-GCL-FIX assumes any information from the downstream task as unavailable while AD-GCL-OPT assumes the augmentation search space has some weak information from the downstream task. A full range of analysis on how Arey impacts AD-GCI will be investigated in Sec. 5.2 We compare AD-GCL with three unsupervised/self- supervised learning baselines for graph-level tasks, which include randomly initialized untrained GIN (RU-GIN) (72). InfoGraph [18] and GraphCL (24). Previous works [18 24 show that they generally
S.2
outperform graph kernels 77) and network embedding methods [33 34 78 79..
We also adopt GCL with GDA based on non-adversarial edge dropping (NAD-GCL) for ablation study. NAD-GCL drops the edges of a graph uniformly at random. We consider NAD-GCL-FIX and NAD-GCL-OPT with different edge drop ratios. NAD-GCL-Ge of AD-GCL-FIX at the saddle point of the optimization (Eq g while NAD-GCL-OPT optimally tunes the edge drop ratio over the validation datasets to match AD-GCL-OPT. We also adopt fully supervised GIN (F-GIN) to provide an anchor of the performance. We stress that all methods adopt GIN (72 as the encoder. Except F-GIN, all methods adopt downstream linear classifier or regressor
adopts the edge drop ratio
7

---

Dutset
NCT:
PROTENNS
MUDAG
DD
COLLAB
RDT
RDT MSK
IMD9
IMDB M
F-GIN
78.27
72.39 2.76
90.43 4,61
74,82
56.79 204
53.28
71,83 1,93
4
AB-S Roed Ruediiics
RU-GIN
0.10
69.03
87.61 0.39
=..32:0.30
63.05
58.97 0.33
27.52
51.86 0.33
32.81 0:57
InfoCiraph
66.13
72.57 0.65
81.21 4: 1.77
70.35
78.79 2.84
51.11 0.55
71.11
48,66 0.67
GraphC1
24
6..54
72,86 + 1.01
86.20 1.31
74,70=0..0
71,26 + 0.55
62.63 0.99
$3.05 = 0.40
70:80 + 0.77
48,40 0.63
NAT-OCI-RI
60.25
72.81
0.71
88.58
11.55-0.55
71.56 + 0.58
K3,41 + 0.66
52.72 0.71
70.94 + 0.77
48,53 0.47
NAD-GCL-OPT
0..50
73,18 + 0.71
89.05
1.00
74.55 - 0.55
72.04 + 0.67
83,74 0,76
53.43 0.26
71 0,59
49.01 0.90
Ous
GD--CL-FIX
69.67 0.51
73,89
89.25
1.45
74,49
2:
N5.53 :: 0.79
53.00
71.57 4: 1.01
AD-GCL-OPT
69.67 0.51"
7381
0.46"
89.70
1,3
75.30
73.32
N5.53 :: 0.79
54,93
0.43"
72.33 0.56"
49,89 0.66'
Tack
Regrosius (Dowrblicm Classiter Lincer Regerssion + L.2)
Clasdiastioon (Downiirrun Chositia - Logistic Regrasio 1.)
Dataset
molesol
mollipo
molieessly
ZINC 10K
molbace
mothlöp
molcliztox
moboo221
mobider
Mitric
RMSS (shonrd) (1)
MAF(()
ROC-AUC & (shseodl (1)
F-GIN
1,175 0.057
0,987 à 0.018
2:555 à 0,34y
0.254 + 0.08
72.97 + 4.00
65.17
+ 1.48
85.14
2:51
74.91
+:5.1
57.00 + 1.40
ABS Baoeiines
RU-GIN
72
1,706 0.180
L,75 à 0/022
7,525 8: 2.119
0,909 s: 0/022
75,07 +: 2.25
64.68 a 2:46
72.29 à 4.15
71.53
0.74
02.29 4: 1.12
Infolloghh
1,346 +: 0.178
1.005 + 0,023
10005 + 4.810
0,800 + 0.017
74.74 + 3.64
66.33
2.79
5:33
60.74
0.55
60.54 + 0.00
GraphCL 34
4.272 0.089
0,910 0,015
7,679 2,745
0.027 à: 0.013
74.32 2: 2.70
68.22
4 1.89
74.92
à 4.42
72.40
1:00
61.76
1.11
NAD FIX
1,992 a 0.065
0.952 1 0,004
5,540 4 0,877
0.009 a: 0.010
73,50 2.775
06.12
1.50
73.32
2 3.66
71,65
0.094
60.41 # 1.48
NAD-OCL-OPT
1,242 a: 0.006
0,397 à 0,003
5,560 0.877
0.009 a: 0.000
73.09 0t 3.67
67.20
1.88
74.00
0 4.2
71,85
0.0
61.14 1,43
AD-OCL.FIX
1,277 a: 0.087
0.842 : 0,128"
5,150 a: 0.624'
0.578 =: 0.012
76.37 +: 2,03
63.24
1.47
80.77 + 3.92
71.42
4:0.3
63.19 : 0.95
AD OPT OPT
1116 = 0.000
0.822 0.020
4.145 a 0.309
0.541 : 0.004
77.27 + 2.56
69.54
1.92
80.77 4 3.92
72,92 0.66
63.19 a 0,55
Table 1: Unsupervised learning performance for (TOP) biochemical and social network classification TU
datasets (73 (Averaged accuracy 1 std. over 10 runs) and (BOTTOM) chemical molecules property prediction in OGB datasets (52) (mean + std. over 10 runs). Bold/Bold" indicats our methods outperform baselines with 2 0.5/2 2 std respectivcly, Fully supervised (F-GIN) results are shown only for placing GRL methods in perspective. Ablation study (AB-S) results do not count as baselines.
with the same hyper-parameters for fair comparison. Adopting linear models was suggested by [40] which explicitly attributes any performance gain/drop the quality of learnt representations.
Tables T show the results for unsupervised graph level property prediction in social and chemical domains respectively. We witness the big performance gain of AD-GCL opposed to all baselines across all the datasets. Note GraphCL utilizes extensive evaluation to select the best combination of augmentions over a broad GDA family including node-dropping, edge dropping and subgraph sampling. Our results indicate that such extensive evaluation may not be necessary while optimizing the augmentation strategy in adversarial way is greatly beneficial.
We stress that edge dropping is not cherry picked as the search space of augmentation strategies. Other search spaces may even achieve better performance, while an extensive investigation is left for the future work.
Moreover, AD-GCL also clearly improves upon the performance against its non non-adversa counter- parts (NAD-GCL) across all the datasets. which further demonstrates stable and significant advantages of the AD-GCL principle. Essentially, the graph-dependen augmentation learnt by AD-GCL yields much benefit. Finally, we compare AIX with AD-GCL-OPT. Interestingly, two methods achieve comparable results though AD-GCL-OPT is sometimes better. This observation implies that the AD-GCL principle may be robust to the choice of Xreg and thus motivates the analysis in the next subsection. Moreover, weak information from the downstream tasks indeed help with controlling the search space and further betters the performance. We also list the optimal Areg 's of AD-GCL-OPT for different datasets in Appendix F. for the purpose of comparison and reproduction.
5.1.1
Note on the linear downstream classifier
We find that the choice of the downstream classifier can significantly affect the evaluation of the self-supervised representations. InfoGraph (18) and GraphCL (24) adopt a non-linear SVM model as the downstream classifier. Such a non-linear model is more powerful than the linear model we adopt and thus causes some performance gap between the results showed in T and (BOTTOM) and their original results (listed in Appendix G.2.1 as Table 8). We argue that using non-linear SVM model as the downstream classifier is unfair, because the performance of even a randomly initialized untrained GIN (RU-GIN) is significantly improved (comparing results from Table [ (TOP) to Table 8 Therefore, we argue for adopting a linear classifier protocol as suggested by [40]. That having been said, our methods (both AD-GCL-FIX and AD-GCL-OPT) still performs significantly better than baselines in most cases, even when a non-linear SVM classifer is adopted, as shown in Table g Several relative gains are there no matter whether the downstream classifier is simple linear
8

---

3.6
ogog
Arl
cope drop
Exoectd edge
86-
scha-nvliths
Auy
D.1
Auy
5.
0-2
0.2
GOLLAS
Aup 0.3
0.0
Pay
10
20
20
40
100
159
Rog. Strength
Tran Epochs
Fain Epocts
(c)
5.0
Wildertoon AUC--ROC
5.0
2.0
0.5.33
Accuracy
0.74
Veldetton Arcuracy
0.8G
2.003
9.73
Valdasion
0.7
0.8-
NAD: GCL
0.7
0.8
0.5
0.5
0.84
0.6
0.6
0.72
COLLAS
AD
(ours)
0.5
0.5
WAG-50
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.5
0.8
0.0
0.2
0.4
0.6 0.8
0.0
0.2
0.4
0.6
0.8
Edge drop
Edge crop ralio
Edge drop rablo
Edge drop
L (measured at saddle point of Eq (b) Training dynamics of expected drop ratio for Aug. (c) Validation performance for graph classification v.s. edge drop ratio. Compare AD-GCL and GCL with adverrrraai edge dropping. The markers on AD-GCL's performance curves show the Avey used.
Figure 3: (a) Aug v.s. expected edge drop ratio Eg EE
model (Tables I or a non-linear SVM model (Table 8 InfoGraph in 5 over 8 datasets and GraphCL in 6 over 8 datasets. This further provides the evidence for the effectiveness of our method. Details on the practical benefits of linear downstream models can be found in Appendix G.2.1
AD-GCL methods significantly outperform
5.2
Analysis Regularizing the GDA Model
Here, study how different Areg 's impact the expected edge drop ratio of AD-GCL at the saddle point of Eq and further impact the model performance on the validation datasets. Due to the page limitation, we focus on classification tasks in the main text while leaving the discussion on regression 3
tasks in the Appendix F22
Figure shows the results.
As shown in Figure 3 a). large Arey tends to yield a small expected edge drop ratio at the convergent point, which matches our expectation. Aveg ranging from 0. to 10.0 corresponds to dropping almost everything (80% edges) nothing (<10% edges). The validation performance in Figure 3[c) is out of our expectation. We find that for classification tasks, the performance of the encoder is extremely robust different choices of Xreg 's when trained w.r.t. the AD-GCL principle, though the edge drop ratios at the saddle point are very different. However, the non-adversarial counterpart NAD-GCL is sensitive to different edge drop ratios, especially on the molecule dataset (e.g., ogbg-molclitox, agbg-molbbbpp We actually observe the similar issue of NAD-GCL across all molecule datasets (See Appendix F.3 More interesting aspects of our results appear at the extreme cases. When Areg > 5.0, the convergent edge drop ratio is close to O, which means no edge dropping, but AD-GCL still significantly outperforms naive GCL with small edge drop ratio. When A 0.3, the convergent edge drop ratio is greater than 0.6, which means dropping more than half of the edges, but AD-GCL still keeps reasonable performance. We suspect that such benefit comes from the training dynamics of AD-GCL (examples as shown in Figure b))) Particularly, optimizing augmentations allows for non-uniform edge-dropping probability. During the optimization procedure, AD-GCL pushes high drop probability on redundant edges while low drop probability on critical edges, which allows the encoder to differentiate redundant and critical information. This cannot be fully explained by the final convergent edge drop ratio and motivates future investigation of AD-GCL from a more in-depth theoretical perspective.
5.3 Transfer Learning
Next, we evaluate the GNN encoders trained by AD-GCL on transfer learning to predict chemical molecule properties and biological protein functions. We follow the setting [17] and use the same datasets: GNNs are pre-trained on one dataset using self-supervised learning and later fine-tuned on another dataset to test out-of-distribution performance. Here, we only consider AD-GCL-FIX as AD-GCL-OPT is only expected to have better performance. We adopt baselines including no pre-trained GIN (i.e, without self-supervised training on the first dataset and with only fine-tuning), InfoGraph [8] GraphCL [24). three different pre-train strategies in I7 including edge prediction,

---

Pre Train Dunsset
ZINC 2M
Fine- Tune Dataset
BBBP
Tox21
SIDER
ClinTox
BACE
11V
MUV
TaxCast
PP1
Train
70.1
71.8
634
EdgaPred
17
6.
76.0 4: 0.6
00.4
64.1 2
79.9 0:09
76.3
74.1 4:1
64.12 0.5
65.7 1,3
AirMasking
W
64.3 2.8
76.7 1.04
61.0 0.7
71.8 4.1
79.3 1. 1.6
712.11111
74.7 1.14
642
65.2 1.6
ContesaPred
2:
75.7
6.
19.6 1.2
75.8 13.8
63.9
64.41 1.3
InfoGrraph 1k
68.8 1.8.8
75.3 1.0.5
58.4 0.8
69.9 13.0
75,9 1 1.6
76.0 0.7
15.3 1.5
62.7 0.4
64.1 1.5
GraphCL
69.68
60.53
15.99
75.38
1.44
78.47
62.40
AD-OCT-FII
70.01 +1.07
0.82
63.28
0.79
79.78
3.52
78.51 + 0.80
78.28
0.97
72.30
1,61
63.07
0.72
6833
1.36
Our Ranks
Table 2: Transfer learning performance for chemical molecules property prediction (mean ROC-AUC 1 std.
over 10 runs). Bold indicates our methods outperform baselines with 2 0.5 std..
Dataset 0.22
NCII
PROTHINS
DD
COLLAB
RDT-B
RDT-MSK
No Pre-Train
70.40 1.54
73.56
73.71 0.27
86.63 0.27
51.33 0.44
SS-GCN-A
0.33
70.29
0.64
74.30
74.19
1.19-+0.0 0.13
11.74+0.33
52.01 + 0.20
GAR 20
0.224
70.51
74.54+0.66 0.68
75.09 0.19
87.69 0.40
$3.58 0.13
InfoGraph
18
74,86
0.266
72:27
0.40
15.78
73.76 11 0.29
88.66 0,95
53.61 1: 0.31
GraphCL
24
74.63
+
74.17
0.34
76.17
1.37
74.23
+ 0.21
89.11
0.19
52.55 + 0.45
AD-GCL-FIX
75.18 0.31
73:96
+ 0.47
77.91
0.73
75.82 = 0.26
90.10 = 0.15'
53.49 + 0.28
Our Ranks
2
Table 3: Semi-supervised learning performance with 10% labels TU datasets [3] (10-Fold Accuracy (%)+
std over 5 runs). Bold/Bold' indicate our methods outperform baselines with 2 0.5 std/ 2 std respectively.
node attribute masking and context prediction that utilize edge, node and subgraph context respectively. More detailed setup is given in Appendix G
According to Table 2 AD-GCL-FIX significantly outperforms baselines in 3 out of 9 datasets and achieves a mean rank of 2.4 across these 9 datasets which is better than all baselines. Note that although AD-GCL only achieves 5th some datasets, AD-GCL still significantly outperforms InfoGraph [18] and GraphCL [24]. both of which are strong GNN self- self-training baselines. In contrast to InfoGraph [18] and GraphCL (24), AD-GCL achieves some performance much closer to those baselines (EdgePred, AttrMasking and ContextPred) based on domain knowledge and extensive evaluation in [7). This is rather significant our method utilizes only edge dropping GDA, which
again shows the effectiveness of the AD-GCL principle.
5.4 Semi-Supervised Learning
Lastly, we evaluate AD-GCL on semi-supervised learning for graph classification on the benchmark TU datasets (73). We follow the setting (24]: GNNs are pre-trained on one dataset using self- supervised learning and later fine-tuned based on 10% label supervision on the same dataset. Again,
we only consider AD-GCL-FIX and compare it with several baselines in (24 : 1) no pre-trained GCN, which is directly trained by the 10% labels from scratch, 2) SS-GCN GCN-A, a baseline that introduces more labelled data by creating random augmentations and then gets trained from scratch, 3) predictive method GAF (20) that utilizes adjacency reconstruction in the pre-training phase, and GCL methods, 4) InfoGraph [18] and 5) GraphCI. [24). Note that here we have to keep the encoder architecture same and thus AD-GCL-FIX adopts GCN as the encoder. Table 3 shows the results.
AD-GCL-FIX significantly outperforms baselines in 3 out of 6 datasets and achieves a mean rank of 1.5 across these 6 datasets, which again demonstrates the strength of AD-GCL.
6 Conclusions
In this work we have developed theoretically motivated, novel principle: AD-GCL that goes a step beyond the conventional InfoMax objective for self-supervised learning of GNNs. The optimal GNN encoders that are agnostic to the downstream tasks are the ones that capture the minimal sufficient information to identify each graph in the dataset. To achieve this goal, AD-GCL suggests to better graph contrastive learning via optimizing graph augmentations in an adversarial way. Following this principle, we developed practical instantiation based on learnable edge dropping. We have extensively analyzed and demonstrated the benefits of AD-GCL and its instantiation with real- world datasets for graph property prediction in unsupervised, transfer and semi-supervised learning settings.
10

---

Acknowledgments and Disclosure of Funding
We greatly thank the actionable suggestions given by reviewers and the area chair. S.S. and J.N. supported by the National Science Foundation under contract numbers CCF-1918483 and IIS- 1618690, P.L. is partly supported by the 2021 JP Morgan Faculty Award and the National Science Foundation (NSF) award LIDR-2117997
References
[1] A. W. Senior, R. Evans, J. Jumper, J. Kirkpatrick, L. Sifre, T. Green, C. Qin, A. Zidek, A. W. Nelson, A. Bridgland et al., "Improved protein structure prediction using potentials from deep learning," Nature, vol. 577, 7792, pp. 706-710, 2020.
[2] Shlomi, P. Battaglia, and J.-R. Vlimant, "Graph neural networks in particle physics." Machine Learning: Science and Technology, vol. 2, no. 2, p. 021001, 2020.
[3] W. L. Hamilton, "Graph representation learning," Synthesis Lectures on Artificial Intelligence and Machine Learning vol. 14, no. 3, pp. 1-159, 2020.
[4] K. Hornik, M. Stinchcombe, H. White al., "Multilayer feedforward networks are universal approximators. " Neural Networks, vol. 2, no. 5, pp. 359-366, 1989.
[5] G. Cybenko, "Approximation by superpositions of sigmoidal function," Mathematics of control, signals and systems, vol. 2, no. 4, pp. 303-314, 1989.
[6] Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini, "The graph neural network model," IEEE Transactions Neural Networks, vol. 20, no. 1, pp. 61-80, 2008.
[7] I. Chami, S. Abu-El-Haija B. Perozzi, C. Ré, and K. Murphy, "Machine learning graphs: A model and comprehensive taxonomy," arXiv preprint arXiv:2005.03675. 2020.
[8] Z. Zhang, P. Cui, and W. Zhu, "Deep learning on graphs: A survey," JEEE TKDE, 2020.
[9] W. L.. Hamilton, R. Ying, and Leskovec, "Representation learning on graphs: Methods and applications," IEEE Data Engineering Bulletin, vol. 40, no. 3, pp. 52-74, 2017.
[10] N. Kipf and M. Welling, "Semi-superrvised classification with graph convolutional networks," in International Conference on Learning Representations, 2017.
[11] H. Dai, B. Dai, and L. Song, "Discriminative embeddings of latent variable models for structured data," " in International Conference on Machine Learning. PMLR, 2016, pp. 2702-2711.
[12] Veličković, Cucurull, Casanova, Romero, Liò, and Bengio, "Graph attention networks," in International Conference on Learning Representations, 2018.
[13] M. Zhang. Z. Cui, M. Neumann, and Y. Chen, "An end-to-end deep learning architecture for graph classification," the AAAI Conference on Artificial Intelligence, 2018, pp. 4438-4445
[14] K. Xu, W. Hu, Leskovec, and S. Jegelka, "How powerful are graph neural networks?" in International Conference Learning Representations, 2019.
[15] C. Morris, M. Ritzert, M. Fey, W. L. Hamilton, J. E. Lenssen, G. Rattan, and M. Grohe, "Weisfeiler and leman go neural: Higher-order graph neural networks," in the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 4602-4609.
[16] P. Li, Y. Wang, H. Wang, and J. Leskovee, "Distance encoding: Design provably more powerful neural networks for graph representation learning," Advances in Neural Information Processing Systems, vol. 33, 2020.
[17] W. Hu, B. Liu, J. Gomes, M. Zitnik, Liang, V. Pande, and Leskovec, "Strategies for pre-training graph neural networks," International Conference on Learning Representations, 2020.
[18] F-Y. Sun, Hoffmaann, and J. Tang. "Infograph: Unsupervised and supervuper graph-level representation learning via mutual information maximization," arXiv preprint arXiv: 1108.990000000 2019.
[19] H. G. Vogel, Drug discovery and evaluation: pharmacological assays. Springer Science Business Media, 2002.

---

[20] T. N. Kipf and M. Welling, "Variational graph auto auto-encoders," arXiv preprint arXiv:1611.07300
2016.
[21] A. Grover, A. Zweig, and S. Ermon, "Graphite: Iterative generative modeling of graphs," in International Conference on Machine Learning. PMLR, 2019, pp. 2434-2444.
[22] 7. Peng, W. Huang, M. Luo, Q. Zheng. Y. Rong, T. Xu, and J. Huang, "Graph representation learning via graphical mutual information maximization," " in Proceedings of Web Conference 2020, 2020.
[23] P. Veličković, W. Fedus, W. L. Hamilton, P. Liò, Y. Bengio, and R. D. Hjelm, "Deep graph infomax," arXiv preprint arXiv: 10341, 2018.
[24] Y. You, T. Chen, Y. Sui, T. Chen, Z. Wang, and Y. Shen, "Graph contrastive learning with augmentations," Advances in Neural Information Processing Systems, vol. 33, 2020.
[25] K. Hassani and A. H. Khasahmadi, "Contrastive multi-view representation learning on graphs." in International Conference on Machine Learning. PMLR, 2020, pp. 41 41666...
[26] Y. Xie, Z. Xu, Z. Wang, and S. Ji, "Self-supervised learning of graph neural networks: A unified review," arXiv preprini arXiv:2102.1075:: 2021.
[27] Y. Liu, S. Pan, M. Jin, C. Zhou, F. Xia, and S. "Graph self-supervised learning: A survey," arXiv preprini arXiv:2103.000 2021.
[28] S. Zhang, Z. Hu, A. Subramonian, and Y. Sun, "Motif-driven contrastive learning of graph representations." " arXiv preprint 1xXv:20022./2533 2020.
[29] S. Thakoor, C. Tallec, M. G. Azar, Munos, P. Veličković, and M. Valko, "Bootstrapped representation learning on graphs," arXiv preprint arXiv:2102.06554 2021.
[30] Y. Zhu, Y. Xu. F. Yu, Q. Liu, S. Wu, and L. Wang, "Graph contrastive learning with adaptive augmentation." " arXiv preprint 2rXiv:2010.14445 2020.
[31] Qiu, Q. Chen, Y. Dong, Zhang, H. Yang. M. Ding, K. Wang, and J. Tang, "Gee: Graph contrastive coding for graph neural network pre-training," in Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2020, pp. 1150- 1160.
[32] M. Belkin and P. Niyogi, "Laplacian eigenmaps for dimensionality reduction and data represen- tation," " Neural computation, vol. 15, no. 6, pp. 1373-1396, 2003.
[33] B. Perozzi, R. A1-Rfou, and S. Skiena, "Deepwalk: Online learning of social representations," in Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, 2014, pp. 701-710.
[34] A. Grover and J. Leskovec, "node2vec: Scalable feature learning for networks," in the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 855-864.
[35] I.. F. Ribeiro, P. H. Saverese, and D. R. Figueiredo, "struc2vec: Learning node representa- tions from structural identity," in the ACM SIGKDD) International Conference on Knowledge Discovery and Data Mining, 2017, pp. 385-394.
[36] W. Hamilton, Z. Ying, and J. Leskovec, "Inductive representation learning on large graphs," in Advances in Neural Information Processing Systems, 2017.
[37] K. Henderson, B. Gallagher, Eliassi-Rad, H. Tong. S. Basu, Akoglu, D. Koutra, C. Falout- sos, and L. Li, "Rolx: structural role extraction & mining in large graphs," in the ACM SIGKDD international conference on Knowledge discovery and data mining, 2012, pp. 1231-1239.
[38] C. Donnat, M. Zitnik, D. Hallac, and Leskovee, "Learning structural node embeddings via diffusion wavelets." in the ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 1220-1329.
[39] R. Linsker, "Self-organizzttinn in perceptual network," " Computer, vol. 21, no. 3, pp. 105-117, 1988.
[40] M. Tschannen, J. Djolonga, P. K. Rubenstein, S. Gelly, and M. Lucic, "On mutual infor- mation maximization for representation learning," "h in International Conference on Learning Representations, 2020.
12

---

[41] N. Tishby, F. C. Pereira, and W. Bialek, "The information bottleneck method," arXiv preprint physsies00004057, 2000.
[42] N. Tishby and N. Zaslavsky, "Deep learning and the information bottleneck principle," in 2015 IEFE Information Theory Workshop (ITW). IEFE, 2015.
[43] Z. Goldfeld and Y. Polyanskiy. "The information bottleneck problem and its applications in machine learning," IEEE Journal on Selected Areas in Information Theory, 2020.
[44] A. A. Alemi, Fischer. J. V. Dillon, and K. Murphy, "Deep variational information bottleneck," arXiv preprint 17Xiv:1612.004100 2016.
[45] X. B. Peng, A. Kanazawa, S. Toyer, P. Abbeel, and S. Levine, "Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow,"
arXiv preprint arXiv: 1810.00821, 2018.
[46] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick. S. Mohamed, and A. Lerch- ner, "beta-vae: Learning basic visual concepts with a constrained variational framework." in International Conference on Learning Representations, 2017.
[47] T. Wu, H. Ren, Li, and Leskovec, "Graph information bottleneck," " in Advances Neural Information Processing Systems, 2020.
[48] J. Yu, T. Xu, Y. Rong. Y. Bian, J. Huang, and R. He, "Recognizing predictive substructures with subgraph information bottleneck," International Conference on Learning Representations, 2021.
[49] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl, "Neural message passing for quantum chemistry," in International Conference on Machine Learning. JMLR. org. 2017.
[50] T. M. Cover and J. A. Thomas, Elements of Information Theory. John Wiley & Sons, 2012.
[51] B. Weisfeiler and A. Leman, "A reduction of a graph to a canonical form and an algebra arising during this reduction,"
Informatsia, 1968.
[52] W. Hu, M. Fey, M. Zitnik, Y. Dong. H. Ren, B. Liu, M. Catasta, and J. Leskovec, "Open graph benchmark: Datasets for machine learning on graphs," arXiv preprint (eXix:2005.00663 2020.
[53] D. Duvenaud, D. Maclaurin, J.
R. Gómez-Bombarelli, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams, "Convolutional networks on graphs for learning molecular fingerprints," Advances in Neural Information Processing Systems, vol. 2015, pp. 2224-2232, 2015.
[54] E. N. Gilbert, "Random graphs," The Annals of Mathematical Statistics, vol. 30, no. pp. 1 1141-1144, 1959.
[55] P. Erdōs and A. Rényi, "On random graphs i." Publ. Math. Debrecen, vol. 6, pp. 290-297, 1959.
[56) C. J. Maddison, A. Mnih, and Y. W. Teh, "The concrete distribution: A continuous relaxation of discrete random variables," International Conference on Learning Representations, 2017.
[57] E. Jang, S. Gu, and B. Poole, "Categorical reparameterization with gumbel-softmax," in International Conference on Learning Representations, 2017.
[58) D. Luo, W. Cheng, D. Xu, W. Yu, B. Zong. H. Chen, and X. Zhang, "Parameterized explainer for graph neural network," Advances in Neural Information Processing Systems, vol. 33, 2020.
[59] A. v. d. Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding." arXiv preprint arXiv: arXiv:1807.00 007.003748 2018
[60] Y. Tian, D. Krishnan, and P. Isola, "Contrastive multiview coding," arXiv preprint arXiv: 1906605849. 2019.
[61] B. Poole, S. Ozair, A. Van Den Oord, A. Alemi, and G. Tucker, "On variational bounds of mutual information," in International Conference on Machine Learning, 2019.
[62] T. Chen, S. Korbllith, M. Norouzi. and G. Hinton, "A simple framework for contrastive learning of visual representations," in International Conference Machine Learning. PMLR, 2020, pp. 1597-1607.
[63] Becker and F. Hinton, "Self-organizing neural network that discovers surfaces in random- dot stereograms," Nature, vol. 355, no. 6356, pp. 161-163, 1992.
13

---

[64] O. Henaff, "Data-efficient image recognition with contrastive predictive coding," in International Conference on Machine Learning. PMLR, 2020, pp. 4182-4192.
[65] R. D. Hjelm, A. Fedorov, S. Lavoie-Marchildon, K. Grewal, P. Bachman, A. Trischler, and Bengio, "Learning deep representations by mutual information estimation and maximization," in International Conference on Learning Representations, 2019.
[66] T. Chen, S. Korbblith, K. Swersky, M. Norouzi, and G. Hinton, "Big self-supervised models are strong semi-supervised learners," arXiv preprint arXiv:2006. 10029, 2020.
[67] P. Veličković, W. Fedus, W. L. Hamilton, P. Lio, Y. Bengio, and R. D. Hjelm, "Deep graph infomax," arXiv preprint arXiv: 010.888000444 2018.
[68] Z. Peng, W. Huang, M. Luo, Q. Zheng, Y. Rong. T. Xu, and J. Huang, "Graph representation learning via graphical mutual information maximization," in Proceedings The Web Conference
2020, 2020, pp. 259-270.
[69] Y. Jiao, Y. Xiong, J. Zhang, Zhang. Zhang, and Zhu, "Sub-graph contrast for scalable self-supervised graph representation learning." arXiv preprint arXiv:2009. 10273, 2020.
[70] Y. You, T. Chen, Shen, and 7. Wang. "Graph contrastive learning automated," arXiv preprint arXiv:2/06.07594, 2021.
[71] Y. Tian, C. Sun, B. Poole, D. Krishnan, C. Schmid, and P. Isola, "What makes for good views for contrastive learning?" in Advances in Neural Information Processing Systems, 2020.
[72] K. Xu, W. Hu, J. Leskovec, and S. Jegelka, "How powerful are graph neural networks?" in International Conference on Learning Reprexentations, 2019.
[73] C. Morris, N. M. Kriege, F. Bause, K. Kersting, P. Mutzel, and M. Neumann, "Tudataset: A collection of benchmark datasets for learning with graphs," in ICML 2020 Workshop on Graph Representation Learning and Beyond (GRL+ 2020), 2020. [Online]. Available:
www.graphleaming..o
[74] V. P. Dwivedi, C. K. Joshi, T. Laurent, Y. Bengio, and X. Bresson, "Benchmarking graph neural networks," arXiv preprint arXiv:2003. 00982, 2020.
[75] N. M. Kriege, F. D. Johansson, and C. Morris, "A survey graph kernels," Applied Network Science, vol. 5, no. 1, pp. 1-42, 2020.
[76] P. Yanardag and S. Vishwanathan, "Deep graph kernels"" in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2015, pp. 1365-1374.
[77] N. Shervashidze, P. Schweitzer, E. J. V. Leeuwen, K. Mehlhorn, and K. M. Borgwardt, "Weisfeiler-lehman graph kernels," Journal of Machine Learning Research, vol. 12, no. Sep, pp.
2539-2561, 2011.
[78] A. Narayanan, M. Chandramohan, R. Venkatesan, L. Chen, Y. Liu, and S. Jaiswal, "graph2vee: Learning distributed representations of graphs," arXiv preprint arXiv:1707.05005. 2017.
[79] B. Adhikari, Y. Zhang, N. Ramakrishnan, and A. Prakash, "Sub2vec: Feature learning for subgraphs," in Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, 2018, pp. 170-182.
[80] T. M. Cover, Elements of information theory. John Wiley & Sons, 1999.
[81] L. Babai, "Groups, graphs, algorithms: The graph isomorphism problem," in Proc. ICM, vol. 3. World Scientific, 2018, pp. 3303-3320,
[82] H. A. Helfgott, J. Bajpai, and D. Dona, "Graph isomorphisms in quasi qulsi-polynomial time," arXiv preprint arXiv:
2017.
[83] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer. R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, "Scikit-learn: Machine learning in Python," " Journal of Machine Learning Research, vol. 12. pp. 2825-2830, 2011.
[84] C. Zhu, R. H. Byrd, P. Lu, and J. Nocedal, "Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale found--oosstraing (TOMS), vol. 23, no. 4, pp. 550-560, 1997.
optimization," " ACM Transactions on Mathematical Software
[85] R.-E. Fan, K.-W. Chang, C.-J. Hsich, X.RR Wang, and C.-J. Lin, "Liblinear: A library for large linear classification,"" Journal of Machine Learning Research, vol. 9, pp. 1871-1874, 2008
14


Reviews:
[{"review_id": "oEVIWt0Dhqq", "reviewer": "Reviewer_GWqt", "summary": "The paper combines the ideas of adversarial training (on augmentation) and graph contrastive learning (branded as \"AD-GCL\") to further boost the unsupervised performance, targeting at representing less redundant information which is on pair with information bottleneck principle. It is empirically demonstrated with significant performance gains in multiple experiment settings of unsupervised, semi-supervised and transfer learning.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Generally good presentation and solid experiments. I like it in general, with a few comments and suggestions:\n\n(i) (Method): Using learnable edge perturbation to parameterize augmentation is a practical way but might be restricted. Can authors provide some other instantiations to consolidate the proposed idea (e.g. learnable node masking)?\n\n(ii) (Theoretical claim): The claim in Theorem 1 holds the assumption seemingly conflicting with the goal of unsupervised learning, that assuming the GNN is already endowed with good enough distinguishment power, while this is difficult to achieve specifically for certain downstream labels since unsupervised learning does not have labels. More clarification of this claim is needed.\n\n(iii) (Experiment): Table 1&3 show AD-GCL achieves the (nearly) best performance among all datasets (including molecules) but Table 2 shows only 3 out of 8 doing better. Is there any explanation (analysis) for these contradictory observations?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Nothing in particular", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "jJWoPoF_hFT", "reviewer": "Reviewer_PF2i", "summary": "In this paper, the authors try to improve the graph contrastive learning. They point out that the representation learned by maximizing mutual information does not necessarily help for downstream tasks. And they propose adversarial graph augmentation to enhance the InfoMax objective for self-supervised learning of GNNs. They argue that AD-GCL will make GNNs avoid learning the redundant information during the training, which are not beneficial to GraphCL. The author also gave a relatively sufficient theoretical explanation. Extensive experiments are conduct to demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In general, the paper is written well and easy to follow. However, there are still several concerns that the authors need to address:\n\n1. Novelty are limited.\n(1) Recently, there have been many works [2,3] that have studied the use of high quality data augmentation on graph to improve the performance of contrast learning. The novelty of this work that only improves the performance of graph representation learning by how to drop the edges is very limited.\n(2) The method proposed by the author is only to drop the edges, and does not discuss some common graph data augmentation methods, such as dropping nodes, extracting subgraphs and other diverse data augmentation methods mentioned in [1], so I think this data augmentation method is too single, too simple, and very limited in novelty.\n\n2. Questions about the GIB and InfoMax.\n(1) The example of failure given in Figure 2 is not credible, because self-supervised learning does not require label information, but it forces the use of random labels to force self-supervised learning to use wrong information while ensuring maximum mutual information. The label information destroys the learned representation. This is unlikely to happen in practical applications, or this is a very extreme situation. I hope that the author can use a more practical example to illustrate that the maximization of mutual information may not necessarily helpful for downstream tasks.\n(2) I think the core contribution of this work is what the author mentioned in lines 138-140. It does not need to know the label information of downstream tasks and can capture the minimal information for each graph. And the author's proposed method does not need labels for downstream tasks, but the left and right sides of the equation in Theorem 1 both contain label information. How did the author optimize to remove the label influence.\n(3) Will the adversarial data augmentation method proposed by the author have an impact on the maximization of mutual information? In other words, it is impossible to conduct contrastive learning better because of the adversarial examples. I hope the author can give a detial comparison of the accuracy of contrastive learning.\n\n3. Questions about the data augmentation.\n(1) The framework proposed by the author only uses a learning method to drop edges, but it cannot perform data augmentation similar to the other methods mentioned in [1]. \nSimply choosing which edges need to be dropped is definitely not the best choice for graph data.\nFor graph classification tasks, there are many nodes, node features and even some subgraphs can also produce adversarial data augmentation, but the framework only cover on a small part of the search space, which undoubtedly ignored a lot of other possibilities.\n(2) As mentioned in [1], dropping edges is not effective for all datasets. For example, for some molecular data sets such as biochemistry, the connection relationship is a very important property for this kind of data. If you let GNN learn this invariance, it will have a negative impact on downstream tasks. Does the author also find this same rule in the experiment? If found, please give detailed experimental results, if not found, please explain the reasons.\n\n4. Questions about the experiments.\n(1) The paper is missing a lot of baselines[2,3], please compare your experimental results with these methods, and point out the superiority of your method and some of the shortcomings of their methods.\n(2) The paper only conducted experiments on many small data sets. May I ask whether it is also effective for large data sets such as OGB datasets [4]. If so, please give more experimental results on large datasets.\n(3) The paper just performed a lot of experiments on graph classification tasks. May I ask whether it is also effective for node classification or link prediction tasks. If it is effective, please provide some experimental results. If it is not effective, please give an explanation.\n\n[1] Graph contrastive learning with augmentations\n\n[2] Graph Contrastive Learning with Adaptive Augmentation\n\n[3] Graph Contrastive Learning Automated\n\n[4] https://ogb.stanford.edu/", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please refer to the above questions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5h"}, "extra_text": {}}, {"review_id": "ZFIBNxdAdNP", "reviewer": "Reviewer_dRac", "summary": "Current contrastive learning (CL) methods are based on various graph augmentations to generate multiple views for contrast (maximizing MI). To discover the optimal augmentations, the authors propose an adversarial approach to automatically learn views generation that tries to minimize the MI between two views.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed a novel method leveraging adversarial training to learn augmentations for graph contrastive learning. They also introduce edge-dropping augmentation for graph-level tasks as an example to show the theoretical base of the model and the outperforming of their model compared with other popular graph contrastive methods. \n\nThe proposed method is overall novel and interesting. However, there are major concerns in the experimental results. In Table 2, 8, 9 10, the performance gain of the proposed method is not consistent among datasets, especially when comparing to its direct baseline (GraphCL). This makes the author's claim and the proposed method less convincing. The authors may need more experiments to show the effectiveness of their methods on different tasks.\n\nBelow are additional questions in need of clearance.\n\n1, In Section 3.2 (Learnable Edge Dropping GDA model T_\\Phi()), the authors firstly claimed that “structure fingerprints” are important for downstream tasks and then claimed that dropping some edges will not change subgraph structure. The two claims seem conflicting with each other as one type of subgraph can be turned into another type of subgraph by dropping some edges, then the contrastive learning method cannot distinguish these two types of subgraphs. This situation is common in the chemical analysis of molecules, so the authors may need to introduce additional assumptions or proof of this question if they want to mainly illustrate their method by edge-dropping augmentation.\n\n2, In Section 3.2 (Parameterizing T_\\Phi()), the authors called their T as a “graph model”, this was confusing because it was not consistent with the terminology used in Section 3.1, especially in Definition 1 and Definition 2. Do the authors mean “instance of graph augmentation model”?\n\n3, In Section 3.2 (Regularizing T_\\Phi()), the authors proposed to add a regularization term to control the edge perturbation aggressiveness. This is a good idea and classic method in machine learning, but I have two detailed questions here. First, even with the regularization term, it can still have some augmented graphs without any edges in training. Is this situation reasonable or how do the authors prevent it? Second, adding this regularization term was for “not to perform very aggressive perturbation”, but in the Experiment details (Appendix F), the authors showed that aggressive perturbation was somehow better. Let us consider a situation where only the node feature suffices IB or InfoMin target, then the above situation (3.1) is reasonable and can be common sense. The author needs to make it more convincing the necessity of using the regularization term, the edge dropping method, or even GNNs in these experiments.\n\n4, Although the authors showed that even only learnable edge dropping augmentation could lead to SOTA results, the Claims and Proofs in Section 3.1 were not limited in edge dropping and it seems not very easy to analogy edge drooping to other augmentation methods like node dropping and feature masking. The authors maybe need some brief explanation of why this method is not limited to edge-dropping augmentation and suitable tasks for edge-dropping. Also, it would be better to include more experiments to show their learnable augmentation method can be easily transferred to other augmentation methods and suitable tasks for them.\n\n5, Typo at line 137\n\n\n\n===================\n\nI'd like to update my score after reading the authors' responses and comments from other reviewers.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have discussed the limitations and societal impact in the Appendix.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "8JcDZWKodLZ", "reviewer": "Reviewer_uwpc", "summary": "The paper proposes an adversarial graph augmentation technique to boost graph contrastive learning (GCL), with the purpose of avoiding capturing redundant information to be robust. The proposed technique is driven by the information bottleneck principle (IB) with certain theoretical guarantee and empirical verification.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(Originality)\nThe paper looks innovative to me.\n\n(Quality)\nThe overall idea and presentation are clear with extensive experiments on molecule and social network datasets to support rge method.\nTheorem 1 is reasonable and informative to show the bound of information the adversarial technique captures. Nevertheless, the derivation is straightforward (combination of the property of infectivity and mutual information rewriting), and therefore it is suggested to downgrade it into proposition to avoid backfire, while it still reveals the key insight of the proposed AD-GCL method.\n\n(Optional clarification)\nWould you clarify the contributions in this paper between a latest paper \"[Graph Contrastive Learning Automated](https://arxiv.org/abs/2106.07594)\" also leveraging adversarial training on GCL?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not applicable.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper combines the ideas of adversarial training (on augmentation) and graph contrastive learning to further boost the unsupervised performance, targeting at representing less redundant information. It is empirically demonstrated with significant performance gains in multiple experiment settings of unsupervised, semi-supervised and transfer learning.

After the discussion, all reviewers agreed that this paper is above the acceptance threshold. The authors are encouraged to improve this work based on the feedback from the reviewers, especially the following ones.

1. In experiments, compare with a recent work -- "Graph Contrastive Learning Automated", which also leverages adversarial training for GCL. In addition, the authors can compare with the following two works:

     - Graph Contrastive Learning with Adaptive Augmentation

      - Graph Contrastive Learning Automated

2. The performance gain of the proposed method is not consistent among datasets, especially when comparing to its direct baseline (GraphCL). This makes the author's claim and the proposed method less convincing. The authors may need more experiments to show the effectiveness of their methods on different tasks.

Decision: Accept (Poster)