Adversarial Graph Augmentation to Improve Graph Contrastive Learning

Abstract:
Self-supervised learning of graph neural networks (GNN) is in great need because of the widespread label scarcity issue in real-world graph/network data. Graph contrastive learning (GCL), by training GNNs to maximize the correspondence between the representations of the same graph in its different augmented forms, may yield robust and transferable GNNs even without using labels. However, GNNs trained by traditional GCL often risk capturing redundant graph features and thus may be brittle and provide sub-par performance in downstream tasks. Here, we propose a novel principle, termed adversarial-GCL (AD-GCL), which enables GNNs to avoid capturing redundant information during the training by optimizing adversarial graph augmentation strategies used in GCL. We pair AD-GCL with theoretical explanations and design a practical instantiation based on trainable edge-dropping graph augmentation. We experimentally validate AD-GCL2 by comparing with the state-of-the-art GCL methods and achieve performance gains of up-to 14% in unsupervised, 6% in transfer, and 3% in semi-supervised learning settings overall with 18 different benchmark datasets for the tasks of molecule property regression and classification, and social network classification.

heading: 1 Introduction
text: Graph representation learning (GRL) aims to encode graph-structured data into low-dimensional vector representations, which has recently shown great potential in many applications in biochemistry, physics and social science [1–3]. Graph neural networks (GNNs), inheriting the power of neural networks [4, 5], have become the almost de facto encoders for GRL [6–9]. GNNs have been mostly studied in cases with supervised end-to-end training [10–16], where a large number of task-specific labels are needed. However, in many applications, annotating labels of graph data takes a lot of time and resources [17, 18], e.g., identifying pharmacological effect of drug molecule graphs requires living animal experiments [19]. Therefore, recent research efforts are directed towards studying self-supervised learning for GNNs, where only limited or even no labels are needed [18, 20–31].
Designing proper self-supervised-learning principles for GNNs is crucial, as they drive what information of graph-structured data will be captured by GNNs and may heavily impact their performance in downstream tasks. Many previous works adopt the edge-reconstruction principle to match traditional network-embedding requirement [32–35], where the edges of the input graph are expected to be reconstructed based on the output of GNNs [20, 21, 36]. Experiments showed that these GNN models learn to over-emphasize node proximity [23] and may lose subtle but crucial structural information, thus failing in many tasks including node-role classification [16, 35, 37, 38] and graph classification [17].
∗Pan Li and Jennifer Neville co-correspond this work. 2https://github.com/susheels/adgcl
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
To avoid the above issue, graph contrastive learning (GCL) has attracted more attention recently [18, 22, 23, 25–31]. GCL leverages the mutual information maximization principle (InfoMax) [39] that aims to maximize the correspondence between the representations of a graph (or a node) in its different augmented forms [18, 24, 25, 28–31]. Perfect correspondence indicates that a representation precisely identifies its corresponding graph (or node) and thus the encoding procedure does not decrease the mutual information between them.
However, researchers have found that the InfoMax principle may be risky because it may push encoders to capture redundant information that is irrelevant to the downstream tasks: Redundant information suffices to identify each graph to achieve InfoMax, but encoding it yields brittle representations and may severely deteriorate the performance of the encoder in the downstream tasks [40]. This observation reminds us of another principle, termed information bottleneck (IB) [41–46]. As opposed to InfoMax, IB asks the encoder to capture the minimal sufficient information for the downstream tasks. Specifically, IB minimizes the information from the original data while maximizing the information that is relevant to the downstream tasks. As the redundant information gets removed, the encoder learnt by IB tends to be more robust and transferable. Recently, IB has been applied to GNNs [47, 48]. But IB needs the knowledge of the downstream tasks that may not be available.
Hence, a natural question emerges: When the knowledge of downstream tasks are unavailable, how to train GNNs that may remove redundant information? Previous works highlight some solutions by designing data augmentation strategies for GCL but those strategies are typically task-related and sub-optimal. They either leverage domain knowledge [25, 28, 30], e.g., node centralities in network science or molecule motifs in bio-chemistry, or depend on extensive evaluation on the downstream tasks, where the best strategy is selected based on validation performance [24, 30].
In this paper, we approach this question by proposing a novel principle that pairs GCL with adversarial training, termed AD-GCL, as shown in Fig.1. We particularly focus on training self-supervised GNNs for graph-level tasks, though the idea may be generalized for node-level tasks. AD-GCL consists of two components: The first component contains a GNN encoder, which adopts InfoMax to maximize the correspondence/mutual information between the representations of the original graph and its augmented graphs. The second component contains a GNN-based augmenter, which aims to optimize the augmentation strategy to decrease redundant information from the original graph as much as possible. AD-GCL essentially allows the encoder capturing the minimal sufficient information to distinguish graphs in the dataset. We further provide theoretical explanations of AD-GCL. We show that with certain regularization on the search space of the augmenter, AD-GCL can yield a lower bound guarantee of the information related to the downstream tasks, while simultaneously holding an upper bound guarantee of the redundant information from the original graphs, which matches the aim of the IB principle. We further give an instantiation of AD-GCL: The GNN augmenter adopts a task-agnostic augmentation strategy and will learn an input-graph-dependent non-uniform-edge-drop probability to perform graph augmentation.
Finally, we extensively evaluate AD-GCL on 18 different benchmark datasets for molecule property classification and regression, and social network classification tasks in different setting viz. unsuper-
vised learning (Sec. 5.1), transfer learning (Sec. 5.3) and semi-supervised learning (Sec. 5.4) learning. AD-GCL achieves significant performance gains in relative improvement and high mean ranks over the datasets compared to state-of-the-art baselines. We also study the theoretical aspects of AD-GCL with apt experiments and analyze the results to offer fresh perspectives (Sec. 5.2): Interestingly, we observe that AD-GCL outperforms traditional GCL based on non-optimizable augmentation across almost the entire range of perturbation levels.

heading: 2 Notations and Preliminaries
text: We first introduce some preliminary concepts and notations for further exposition. In this work, we consider attributed graphs G = (V,E) where V is a node set and E is an edge set. G may have node attributes {Xv ∈ RF | v ∈ V } and edge attributes {Xe ∈ RF | e ∈ E} of dimension F . We denote the set of the neighbors of a node v as Nv .
Learning Graph Representations. Given a set of graphs Gi, i = 1, 2, ..., n, in some universe G, the aim is to learn an encoder f : G → Rd, where f(Gi) can be further used in some downstream task. We also assume that Gi’s are all IID sampled from an unknown distribution PG defined over G. In a downstream task, each Gi is associated with a label yi ∈ Y . Another model q : Rd → Y will be learnt to predict Yi based on q(f(Gi)). We assume (Gi, Yi)’s are IID sampled from a distribution PG×Y = PY|GPG , where PY|G is the conditional distribution of the graph label in the downstream task given the graph.
Graph Neural Networks (GNNs). In this work, we focus on using GNNs, message passing GNNs in particular [49], as the encoder f . For a graph G = (V,E), every node v ∈ V will be paired with a node representation hv initialized as h (0) v = Xv. These representations will be updated by a GNN. During the kth iteration, each h(k−1)v is updated using v′s neighbourhood information expressed as,
h(k)v = UPDATE (k) ( h(k−1)v , AGGREGATE (k) ({ (h(k−1)u , Xuv) | u ∈ Nv }))
(1)
where AGGREGATE(·) is a trainable function that maps the set of node representations and edge attributes Xuv to an aggregated vector, UPDATE(·) is another trainable function that maps both v’s current representation and the aggregated vector to v’s updated representation. After K iterations of Eq. 1, the graph representation is obtained by pooling the final set of node representations as,
f(G) :≜ hG = POOL ( {h(K)v | v ∈ V } ) (2)
For design choices regarding aggregation, update and pooling functions we refer the reader to [3,7,8].
The Mutual Information Maximization Principle. GCL is built upon the InfoMax principle [39], which prescribes to learn an encoder f that maximizes the mutual information or the correspondence between the graph and its representation. The rationale behind GCL is that a graph representation f(G) should capture the features of the graph G so that representation can distinguish this graph from other graphs. Specifically, the objective of GCL follows
InfoMax: max f I(G; f(G)), where G ∼ PG . (3)
where I(X1;X2) denotes the mutual information between two random variables X1 and X2 [50].
Note that the encoder f(·) given by GNNs is not injective in the graph space G due to its limited expressive power [14, 15]. Specifically, for the graphs that cannot be distinguished by 1-WL test [51], GNNs will associate them with the same representations. We leave more discussion on 1-WL test in Appendix C. In contrast to using CNNs as encoders, one can never expect GNNs to identify all the graphs in G based their representations, which introduces a unique challenge for GCL.

heading: 3 Adversarial Graph Contrastive Learning
text: In this section, we introduce our adversarial graph contrastive learning (AD-GCL) framework and one of its instantiations based on edge perturbation.

heading: 3.1 Theoretical Motivation and Formulation of AD-GCL
text: The InfoMax principle in Eq. 3 could be problematic in practice for general representation learning. Tschannen et al. have shown that for image classification, representations capturing the information
that is entirely irrelevant to the image labels are also able to maximize the mutual information but such representations are definitely not useful for image classification [40]. A similar issue can also be observed in graph representation learning, as illustrated by Fig.2: We consider a binary graph classification problem with graphs in the dataset ogbg-molbace [52]. Two GNN encoders with exactly the same architecture are trained to keep mutual information maximization between graph representations and the input graphs, but one of the GNN encoders in the same time is further supervised by random graph labels. Although the GNN encoder supervised by random labels still keeps one-to-one correspondance between every input graph and its representation (i.e., mutual information maximization), we may observe significant performance degeneration of this GNN encoder when evaluating it over the downstream ground-truth labels. More detailed experiment setup is left in Appendix G.1.
This observation inspires us to rethink what a good graph representation is. Recently, the information bottleneck has applied to learn graph representations [47, 48]. Specifically, the objective of graph information bottleneck (GIB) follows
GIB: max f I(f(G);Y )− βI(G; f(G)), (4)
where (G, Y ) ∼ PG×Y , β is a positive constant. Comparing Eq. 3 and Eq. 4, we may observe the different requirements between InfoMax and GIB: InfoMax asks for maximizing the information from the original graph, while GIB asks for minimizing such information but simultaneously maximizing the information that is relevant to the downstream tasks. As GIB asks to remove redundant information, GIB naturally avoids the issue encountered in Fig.2. Removing extra information also makes GNNs trained w.r.t. GIB robust to adverserial attack and strongly transferrable [47, 48].
Unfortunately, GIB requires the knowledge of the class labels Y from the downstream task and thus does not apply to self-supervised training of GNNs where there are few or no labels. Then, the question is how to learn robust and transferable GNNs in a self-supervised way.
To address this, we will develop a GCL approach that uses adversarial learning to avoid capturing redundant information during the representation learning. In general, GCL methods use graph data augmentation (GDA) processes to perturb the original observed graphs and decrease the amount of information they encode. Then, the methods apply InfoMax over perturbed graph pairs (using different GDAs) to train an encoder f to capture the remaining information. Definition 1 (Graph Data Augmentation (GDA)). For a graph G ∈ G, T (G) denotes a graph data augmentation of G, which is a distribution defined over G conditioned on G. We use t(G) ∈ G to denote a sample of T (G). Specifically, given two ways of GDA T1 and T2, the objective of GCL becomes
GDA-GCL: max f I(f(t1(G)); f(t2(G))), where G ∼ PG , ti(G) ∼ Ti(G), i ∈ {1, 2}. (5)
In practice, GDA processes are often pre-designed based on either domain knowledge or extensive evaluation, and improper choice of GDA may severely impact the downstream performance [17, 24]. We will review a few GDAs adopted in existing works in Sec.4.
In contrast to previous predefined GDAs, our idea, inspired by GIB, is to learn the GDA process (over a parameterized family), so that the encoder f can capture the minimal information that is sufficient to identify each graph.
AD-GCL: We optimize the following objective, over a GDA family T (defined below). AD-GCL: min
T∈T max f
I(f(G); f(t(G))), where G ∼ PG , t(G) ∼ T (G), (6)
Definition 2 (Graph Data Augmentation Family). Let T denote a family of different GDAs TΦ(·), where Φ is the parameter in some universe. A TΦ(·) ∈ T is a specific GDA with parameter Φ.
The min-max principle in AD-GCL aims to train the encoder such that even with a very aggressive GDA (i.e., where t(G) is very different from G), the mutual information / the correspondence
between the perturbed graph and the original graph can be maximized. Compared with the two GDAs adopted in GDA-GCL (Eq.5), AD-GCL views the original graph G as the anchor while pushing its perturbation T (G) as far from the anchor as it can. The automatic search over T ∈ T saves a great deal of effort evaluating different combinations of GDA as adopted in [24].
Relating AD-GCL to the downstream task. Next, we will theoretically characterize the property of the encoder trained via AD-GCL. The analysis here not only further illustrates the rationale of AD-GCL but helps design practical T when some knowledge of Y is accessible. But note that our analysis does not make any assumption on the availability of Y .
Note that GNNs learning graph representations is very different from CNNs learning image representations because GNNs are never injective mappings between the graph universe G and the representation space Rd, because the expressive power of GNNs is limited by the 1-WL test [14, 15, 51]. So, we need to define a quotient space of G based on the equivalence given by the 1-WL test. Definition 3 (Graph Quotient Space). Define the equivalence ∼= between two graphs G1 ∼= G2 if G1, G2 cannot be distinguished by the 1-WL test. Define the quotient space G′ = G/ ∼=.
So every element in the quotient space, i.e., G′ ∈ G′, is a representative graph from a family of graphs that cannot be distinguished by the 1-WL test. Note that our definition also allows attributed graphs. Definition 4 (Probability Measures in G′). Define PG′ over the space G′ such that PG′(G′) = PG(G ∼= G′) for any G′ ∈ G′. Further define PG′×Y(G′, Y ′) = PG×Y(G ∼= G′, Y = Y ′). Given a GDA T (·) defined over G, define a distribution on G′, T ′(G′) = EG∼PG [T (G)|G ∼= G′] for G′ ∈ G′.
Now, we provide our theoretical results and give their implication. The proof is in the Appendix B. Theorem 1. Suppose the encoder f is implemented by a GNN as powerful as the 1-WL test. Suppose G is a countable space and thus G′ is a countable space. Then, the optimal solution (f∗, T ∗) to AD-GCL satisfies, letting T ′∗(G′) = EG∼PG [T ∗(G)|G ∼= G′],

heading: 1. I(f∗(t∗(G));G |Y ) ≤ minT∈T I(t′(G′);G′) − I(t′∗(G′);Y ), where t′(G′) ∼ T ′(G′),
text: t′∗(G′) ∼ T ′∗(G′), (G, Y ) ∼ PG×Y and (G′, Y ) ∼ PG′×Y .
2. I(f∗(G);Y ) ≥ I(f∗(t′∗(G′));Y ) = I(t′∗(G′);Y ), where t′∗(G′) ∼ T ′∗(G′), (G, Y ) ∼ PG×Y and (G′, Y ) ∼ PG′×Y .
The statement 1 in Theorem 1 guarantees a upper bound of the information that is captured by the representations but irrelevant to the downstream task, which matches our aim. This bound has a form very relevant to the GIB principle (Eq.4 when β = 1), since minT∈T I(t′(G′);G′)−I(t′∗(G′);Y ) ≥ minf [I(f(G);G)− I(f(G);Y )], where f is a GNN encoder as powerful as the 1-WL test. But note that this inequality also implies that the encoder given by AD-GCL may be worse than the optimal encoder given by GIB (β = 1). This makes sense as GIB has the access to the downstream task Y .
The statement 2 in Theorem 1 guarantees a lower bound of the mutual information between the learnt representations and the labels of the downstream task. As long as the GDA family T has a good control, I(t′∗(G′);Y ) ≥ minT∈T I(t′(G′);Y ) and I(f∗(G);Y ) thus cannot be too small. This implies that it is better to regularize when learning over T . In our instantiation, based on edge-dropping augmentation (Sec. 3.2), we regularize the ratio of dropped edges per graph.

heading: 3.2 Instantiation of AD-GCL via Learnable Edge Perturbation
text: We now introduce a practical instantiation of the AD-GCL principle (Eq. 6) based on learnable edge-dropping augmentations as illustrated in Fig. 1. (See Appendix D for a summary of AD-GCL in its algorithmic form.) The objective of AD-GCL has two folds: (1) Optimize the encoder f to maximize the mutual information between the representations of the original graph G and its augmented graph t(G); (2) Optimize the GDA T (G) where t(G) is sampled to minimize such a mutual information. We always set the encoder as a GNN fΘ with learnable parameters Θ and next we focus on the GDA, TΦ(G) that has learnable parameters Φ.
Learnable Edge Dropping GDA model TΦ(·). Edge dropping is the operation of deleting some edges in a graph. As a proof of concept, we adopt edge dropping to formulate the GDA family T . Other types of GDAs such as node dropping, edge adding and feature masking can also be paired with our AD-GCL principle. Interestingly, in our experiments, edge-dropping augmentation optimized by AD-GCL has already achieved much better performance than any pre-defined random
GDAs even carefully selected via extensive evaluation [24] (See Sec.5). Another reason that supports edge dropping is due to our Theorem 1 statement 2, which shows that good GDAs should keep some information related to the downstream tasks. Many GRL downstream tasks such as molecule classification only depends on the structural fingerprints that can be represented as subgraphs of the original graph [53]. Dropping a few edges may not change those subgraph structures and thus keeps the information sufficient to the downstream classification. But note that this reasoning does not mean that we leverage domain knowledge to design GDA, as the family T is still broad and the specific GDA still needs to be optimized. Moreover, experiments show that our instantiation also works extremely well on social network classification and molecule property regression, where the evidence of subgraph fingerprints may not exist any more.
Parameterizing TΦ(·). For each G = (V,E), we set TΦ(G), T ∈ T as a random graph model [54, 55] conditioning on G. Each sample t(G) ∼ TΦ(G) is a graph that shares the same node set with G while the edge set of t(G) is only a subset of E. Each edge e ∈ E will be associated with a random variable pe ∼ Bernoulli(ωe), where e is in t(G) if pe = 1 and is dropped otherwise. We parameterize the Bernoulli weights ωe by leveraging another GNN, i.e., the augmenter, to run on G according to Eq.1 of K layers, get the final-layer node representations {h(K)v |v ∈ V } and set
ωe = MLP([h(K)u ;h (K) z ]), where e = (u, z) and {h(K)v | v ∈ V } = GNN-augmenter(G) (7)
To train T (G) in an end-to-end fashion, we relax the discrete pe to be a continuous variable in [0, 1] and utilize the Gumbel-Max reparametrization trick [56, 57]. Specifically, pe = Sigmoid((log δ − log(1 − δ) + ωe)/τ), where δ ∼ Uniform(0,1). As temperature hyper-parameter τ → 0, pe gets closer to being binary. Moreover, the gradients ∂pe∂ωe are smooth and well defined. This style of edge dropping based on a random graph model has also been used for parameterized explanations of GNNs [58].
Regularizing TΦ(·). As shown in Theorem 1, a reasonable GDA should keep a certain amount of information related to the downstream tasks (statement 2). Hence, we expect the GDAs in the edge dropping family T not to perform very aggressive perturbation. Therefore, we regularize the ratio of edges being dropped per graph by enforcing the following constraint: For a graph G and its augmented graph t(G), we add ∑ e∈E ωe/|E| to the objective, where ωe is defined in Eq.7 indicates the probability that e gets dropped.
Putting everything together, the final objective is as follows.
min Φ max Θ I(fΘ(G); fΘ(t(G))) + λregEG [∑ e∈E ωe/|E| ] , where G ∼ PG , t(G) ∼ TΦ(G). (8)
Note Φ corresponds to the learnable parameters of the augmenter GNN and MLP used to derive the ωe’s and Θ corresponds to the learnable parameters of the GNN f .
Estimating the objective in Eq.8. In our implementation, the second (regularization) term is easy to estimate empirically. For the first (mutual information) term, we adopt InfoNCE as the estimator [59–61], which is known to be a lower bound of the mutual information and is frequently used for contrastive learning [40, 59, 62]. Specfically, during the training, given a minibatch of m graphs {Gi}mi=1, let zi,1 = g(fΘ(Gi)) and zi,2 = g(fΘ(t(Gi))) where g(·) is the projection head implemented by a 2-layer MLP as suggested in [62]. With sim(·, ·) denoting cosine similarity, we estimate the mutual information for the mini-batch as follows.
I(fΘ(G); fΘ(t(G))) → Î = 1
m m∑ i=1 log exp(sim(zi,1, zi,2))∑m i′=1,i′ ̸=i exp(sim(zi,1, zi′,2)) (9)

heading: 4 Related Work
text: GNNs for GRL is a broad field and gets a high-level review in the Sec. 1. Here, we focus on the topics that are most relevant to graph contrastive learning (GCL).
Contrastive learning (CL) [39,59,60,63–65] was initially proposed to train CNNs for image representation learning and has recently achieved great success [62,66]. GCL applies the idea of CL on GNNs. In contrast to the case of CNNs, GCL trained using GNNs posts us new fundamental challenges. An image often has multiple natural views, say by imposing different color filters and so on. Hence,
different views of an image give natural contrastive pairs for CL to train CNNs. However, graphs are more abstract and the irregularity of graph structures typically provides crucial information. Thus, designing contrastive pairs for GCL must play with irregular graph structures and thus becomes more challenging. Some works use different parts of a graph to build contrastive pairs, including nodes v.s. whole graphs [18, 67], nodes v.s. nodes [68], nodes v.s. subgraphs [17, 69]. Other works adopt graph data augmentations (GDA) such as edge perturbation [31] to generate contrastive pairs. Recently. GraphCL [24] gives an extensive study on different combinations of GDAs including node dropping, edge perturbation, subgraph sampling and feature masking. Extensive evaluation is required to determine good combinations. MVGRL [25] and GCA [30] leverage the domain knowledge of network science and adopt network centrality to perform GDAs. Note that none of the above methods consider optimizing augmentations. In contrast, our principle AD-GCL provides theoretical guiding principles to optimize augmentations. Very recently, JOAO [70] adopts a bi-level optimization framework sharing some high-level ideas with our adversarial training strategy but has several differences: 1) the GDA search space in JOAO is set as different types of augmentation with uniform perturbation, such as uniform edge/node dropping while we allow augmentation with non-uniform perturbation. 2) JOAO relaxes the GDA combinatorial search problem into continuous space via Jensen’s inequality and adopts projected gradient descent to optimize. Ours, instead, adopts Bayesian modeling plus reparameterization tricks to optimize. The performance comparison between AD-GCL and JOAO for the tasks investigated in Sec. 5 is given in Appendix H.
Tian et al. [71] has recently proposed the InfoMin principle that shares some ideas with AD-GCL but there are several fundamental differences. Theoretically, InfoMin needs the downstream tasks to supervise the augmentation. Rephrased in our notation, the optimal augmentation TIM (G) given by InfoMin (called the sweet spot in [71]) needs to satisfy I(tIM (G);Y ) = I(G;Y ) and I(tIM (G);G|Y ) = 0, tIM (G) ∼ TIM (G), neither of which are possible without the downstreamtask knowledge. Instead, our Theorem 1 provides more reasonable arguments and creatively suggests using regularization to control the tradeoff. Empirically, InfoMin is applied to CNNs while AD-GCL is applied to GNNs. AD-GCL needs to handle the above challenges due to irregular graph structures and the limited expressive power of GNNs [14, 15], which InfoMin does not consider.

heading: 5 Experiments and Analysis
text: This section is devoted to the empirical evaluation of the proposed instantiation of our AD-GCL principle. Our initial focus is on unsupervised learning which is followed by analysis of the effects of regularization. We further apply AD-GCL to transfer and semi-supervised learning. Summary of datasets and training details for specific experiments are provided in Appendix E and G respectively.

heading: 5.1 Unsupervised Learning
text: In this setting, an encoder (specifically GIN [72]) is trained with different self-supervised methods to learn graph representations, which are then evaluated by feeding these representations to make prediction for the downstream tasks. We use datasets from Open Graph Benchmark (OGB) [52], TU Dataset [73] and ZINC [74] for graph-level property classification and regression. More details regarding the experimental setting are provided in the Appendix G.
We consider two types of AD-GCL, where one is with a fixed regularization weight λreg = 5 (Eq.8), termed AD-GCL-FIX, and another is with λreg tuned over the validation set among {0.1, 0.3, 0.5, 1.0, 2.0, 5.0, 10.0}, termed AD-GCL-OPT. AD-GCL-FIX assumes any information from the downstream task as unavailable while AD-GCL-OPT assumes the augmentation search space has some weak information from the downstream task. A full range of analysis on how λreg impacts AD-GCL will be investigated in Sec. 5.2. We compare AD-GCL with three unsupervised/selfsupervised learning baselines for graph-level tasks, which include randomly initialized untrained GIN (RU-GIN) [72], InfoGraph [18] and GraphCL [24]. Previous works [18, 24] show that they generally outperform graph kernels [75–77] and network embedding methods [33, 34, 78, 79].
We also adopt GCL with GDA based on non-adversarial edge dropping (NAD-GCL) for ablation study. NAD-GCL drops the edges of a graph uniformly at random. We consider NAD-GCL-FIX and NAD-GCL-OPT with different edge drop ratios. NAD-GCL-GCL adopts the edge drop ratio of AD-GCL-FIX at the saddle point of the optimization (Eq.8) while NAD-GCL-OPT optimally tunes the edge drop ratio over the validation datasets to match AD-GCL-OPT. We also adopt fully supervised GIN (F-GIN) to provide an anchor of the performance. We stress that all methods adopt GIN [72] as the encoder. Except F-GIN, all methods adopt a downstream linear classifier or regressor
with the same hyper-parameters for fair comparison. Adopting linear models was suggested by [40], which explicitly attributes any performance gain/drop to the quality of learnt representations.
Tables 1 show the results for unsupervised graph level property prediction in social and chemical domains respectively. We witness the big performance gain of AD-GCL as opposed to all baselines across all the datasets. Note GraphCL utilizes extensive evaluation to select the best combination of augmentions over a broad GDA family including node-dropping, edge dropping and subgraph sampling. Our results indicate that such extensive evaluation may not be necessary while optimizing the augmentation strategy in an adversarial way is greatly beneficial.
We stress that edge dropping is not cherry picked as the search space of augmentation strategies. Other search spaces may even achieve better performance, while an extensive investigation is left for the future work.
Moreover, AD-GCL also clearly improves upon the performance against its non-adversarial counterparts (NAD-GCL) across all the datasets, which further demonstrates stable and significant advantages of the AD-GCL principle. Essentially, the input-graph-dependent augmentation learnt by AD-GCL yields much benefit. Finally, we compare AD-GCL-FIX with AD-GCL-OPT. Interestingly, two methods achieve comparable results though AD-GCL-OPT is sometimes better. This observation implies that the AD-GCL principle may be robust to the choice of λreg and thus motivates the analysis in the next subsection. Moreover, weak information from the downstream tasks indeed help with controlling the search space and further betters the performance. We also list the optimal λreg’s of AD-GCL-OPT for different datasets in Appendix F.1 for the purpose of comparison and reproduction.

heading: 5.1.1 Note on the linear downstream classifier
text: We find that the choice of the downstream classifier can significantly affect the evaluation of the self-supervised representations. InfoGraph [18] and GraphCL [24] adopt a non-linear SVM model as the downstream classifier. Such a non-linear model is more powerful than the linear model we adopt and thus causes some performance gap between the results showed in Table 1 (TOP) and (BOTTOM) and their original results (listed in Appendix G.2.1 as Table 8). We argue that using a non-linear SVM model as the downstream classifier is unfair, because the performance of even a randomly initialized untrained GIN (RU-GIN) is significantly improved (comparing results from Table 1 (TOP) to Table 8 ). Therefore, we argue for adopting a linear classifier protocol as suggested by [40]. That having been said, our methods (both AD-GCL-FIX and AD-GCL-OPT) still performs significantly better than baselines in most cases, even when a non-linear SVM classifer is adopted, as shown in
Table 8. Several relative gains are there no matter whether the downstream classifier is a simple linear
e ωe/|E|] (measured at saddle point of Eq.8). (b) Training
dynamics of expected drop ratio for λreg. (c) Validation performance for graph classification v.s. edge drop ratio. Compare AD-GCL and GCL with non-adversarial edge dropping. The markers on AD-GCL’s performance curves show the λreg used.
model (Tables 1) or a non-linear SVM model (Table 8). AD-GCL methods significantly outperform InfoGraph in 5 over 8 datasets and GraphCL in 6 over 8 datasets. This further provides the evidence for the effectiveness of our method. Details on the practical benefits of linear downstream models can be found in Appendix G.2.1.

heading: 5.2 Analysis of Regularizing the GDA Model
text: Here, we study how different λreg’s impact the expected edge drop ratio of AD-GCL at the saddle point of Eq.8 and further impact the model performance on the validation datasets. Due to the page limitation, we focus on classification tasks in the main text while leaving the discussion on regression tasks in the Appendix F.2. Figure 3 shows the results.
As shown in Figure 3(a), a large λreg tends to yield a small expected edge drop ratio at the convergent point, which matches our expectation. λreg ranging from 0.1 to 10.0 corresponds to dropping almost everything (80% edges) to nothing (<10% edges). The validation performance in Figure 3(c) is out of our expectation. We find that for classification tasks, the performance of the encoder is extremely robust to different choices of λreg’s when trained w.r.t. the AD-GCL principle, though the edge drop ratios at the saddle point are very different. However, the non-adversarial counterpart NAD-GCL is sensitive to different edge drop ratios, especially on the molecule dataset (e.g., ogbg-molclitox, ogbg-molbbbp). We actually observe the similar issue of NAD-GCL across all molecule datasets (See Appendix F.3). More interesting aspects of our results appear at the extreme cases. When λreg ≥ 5.0, the convergent edge drop ratio is close to 0, which means no edge dropping, but AD-GCL still significantly outperforms naive GCL with small edge drop ratio. When λreg = 0.3, the convergent edge drop ratio is greater than 0.6, which means dropping more than half of the edges, but AD-GCL still keeps reasonable performance. We suspect that such benefit comes from the training dynamics of AD-GCL (examples as shown in Figure 3(b)). Particularly, optimizing augmentations allows for non-uniform edge-dropping probability. During the optimization procedure, AD-GCL pushes high drop probability on redundant edges while low drop probability on critical edges, which allows the encoder to differentiate redundant and critical information. This cannot be fully explained by the final convergent edge drop ratio and motivates future investigation of AD-GCL from a more in-depth theoretical perspective.

heading: 5.3 Transfer Learning
text: Next, we evaluate the GNN encoders trained by AD-GCL on transfer learning to predict chemical molecule properties and biological protein functions. We follow the setting in [17] and use the same datasets: GNNs are pre-trained on one dataset using self-supervised learning and later fine-tuned on another dataset to test out-of-distribution performance. Here, we only consider AD-GCL-FIX as AD-GCL-OPT is only expected to have better performance. We adopt baselines including no pre-trained GIN (i.e., without self-supervised training on the first dataset and with only fine-tuning), InfoGraph [18], GraphCL [24], three different pre-train strategies in [17] including edge prediction,
node attribute masking and context prediction that utilize edge, node and subgraph context respectively. More detailed setup is given in Appendix G.
According to Table 2, AD-GCL-FIX significantly outperforms baselines in 3 out of 9 datasets and achieves a mean rank of 2.4 across these 9 datasets which is better than all baselines. Note that although AD-GCL only achieves 5th on some datasets, AD-GCL still significantly outperforms InfoGraph [18] and GraphCL [24], both of which are strong GNN self-training baselines. In contrast to InfoGraph [18] and GraphCL [24], AD-GCL achieves some performance much closer to those baselines (EdgePred, AttrMasking and ContextPred) based on domain knowledge and extensive evaluation in [17]. This is rather significant as our method utilizes only edge dropping GDA, which again shows the effectiveness of the AD-GCL principle.

heading: 5.4 Semi-Supervised Learning
text: Lastly, we evaluate AD-GCL on semi-supervised learning for graph classification on the benchmark TU datasets [73]. We follow the setting in [24]: GNNs are pre-trained on one dataset using selfsupervised learning and later fine-tuned based on 10% label supervision on the same dataset. Again, we only consider AD-GCL-FIX and compare it with several baselines in [24]: 1) no pre-trained GCN, which is directly trained by the 10% labels from scratch, 2) SS-GCN-A, a baseline that introduces more labelled data by creating random augmentations and then gets trained from scratch, 3) a predictive method GAE [20] that utilizes adjacency reconstruction in the pre-training phase, and GCL methods, 4) InfoGraph [18] and 5) GraphCL [24]. Note that here we have to keep the encoder architecture same and thus AD-GCL-FIX adopts GCN as the encoder. Table 3 shows the results. AD-GCL-FIX significantly outperforms baselines in 3 out of 6 datasets and achieves a mean rank of 1.5 across these 6 datasets, which again demonstrates the strength of AD-GCL.

heading: 6 Conclusions
text: In this work we have developed a theoretically motivated, novel principle: AD-GCL that goes a step beyond the conventional InfoMax objective for self-supervised learning of GNNs. The optimal GNN encoders that are agnostic to the downstream tasks are the ones that capture the minimal sufficient information to identify each graph in the dataset. To achieve this goal, AD-GCL suggests to better graph contrastive learning via optimizing graph augmentations in an adversarial way. Following this principle, we developed a practical instantiation based on learnable edge dropping. We have extensively analyzed and demonstrated the benefits of AD-GCL and its instantiation with real-world datasets for graph property prediction in unsupervised, transfer and semi-supervised learning settings.
Acknowledgments and Disclosure of Funding
We greatly thank the actionable suggestions given by reviewers and the area chair. S.S. and J.N. are supported by the National Science Foundation under contract numbers CCF-1918483 and IIS1618690. P.L. is partly supported by the 2021 JP Morgan Faculty Award and the National Science Foundation (NSF) award HDR-2117997.

Review summaries:
In this paper, the authors try to improve the graph contrastive learning. They point out that the representation learned by maximizing mutual information does not necessarily help for downstream tasks. And they propose adversarial graph augmentation to enhance the InfoMax objective for self-supervised learning of GNNs. They argue that AD-GCL will make GNNs avoid learning the redundant information during the training, which are not beneficial to GraphCL. The author also gave a relatively sufficient theoretical explanation. Extensive experiments are conduct to demonstrate the effectiveness of the proposed method.