Towards Equivariant Graph Contrastive Learning via Cross-Graph Augmentation
Authors: Zhiyuan Liu
An Zhang
Yu Sun
Yicong Li
Yaorui Shi
Sihang Li
Xiang Wang
Xiangnan He
Tat-Seng Chua
Venue: Submitted to ICLR 2023 (2023)
Keywords: equivariant
self-supervised learning
contrastive learning
graph neural networks

Abstract: Leading graph contrastive learning (GCL) frameworks conform to the invariance mechanism by encouraging insensitivity to different augmented views of the same graph. Despite the promising performance, invariance worsens representation when augmentations cause aggressive semantics shifts. For example, dropping the super-node can dramatically change a social network's topology. In this case, encouraging invariance to the original graph can bring together dissimilar patterns and hurt the task of instance discrimination. To resolve the problem, we get inspiration from equivariant self-supervised learning and propose Equivariant Graph Contrastive Learning (E-GCL) to encourage the sensitivity to global semantic shifts. Viewing each graph as a transformation to others, we ground the equivariance principle as a cross-graph augmentation -- graph interpolation -- to simulate global semantic shifts. Without using annotation, we supervise the representation of cross-graph augmented views by linearly combining the representations of their original samples. This simple but effective equivariance principle empowers E-GCL with the ability of cross-graph discrimination. It shows significant improvements over the state-of-the-art GCL models in unsupervised learning and transfer learning. Further experiments demonstrate E-GCL's generalization to various graph pre-training frameworks. Code is available at \url{https://anonymous.4open.science/r/E-GCL/}

Under review as a conference paper at ICLR 2023
TOWARDS EQUIVARIANT GRAPH CONTRASTIVE LEARNING VIA GGRAPH AUGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Leading graph contrastive learning (GCL) frameworks conform the invariance mechanism by encouraging insensitivity to different augmented views of the same graph. Despite the promising performance, invariance worsens representation when augmentations cause aggressive semantics shifts. For example, dropping the super-node can dramatically change a social network's topology. In this case, encouraging invariance to the original graph can bring together dissimilar pat- terns and hurt the task of instance discrimination. To resolve the problem, we get inspiration from equivariant self-supervised learning and propose Equivari- ant Graph Contrastive Learning (E-GCL) to encourage the sensitivity to global semantic shifts. Viewing each graph transformation others, ground the equivariance principle as a cross-graph augmentation - graph interpolation simulate global semantic shifts. Without using annotation, we supervise the representation of cross-graph augmented views by linearly combining the rep- resentations of their original samples. This simple but effective equivariance principle empowers E-GCL with the ability of cross-graph discrimination. It shows significant improvements over the state-of-the--ar GCL models in unsuper- vised learning and transfer learning. Further experiments demonstrate E-GCL's generalization to various graph pre-training frameworks. Code is available at https:/ //ans////nnnmuuus 4open. wcteeccee/eeecece....//.
INTRODUCTION
Graph contrastive learning (GCL) (You et al., 2020; Suresh et al., 2021; et 2021) is prevailing paradigm for self-supervised learning (Chen et al., 2020; Zbontar et al., 2021) on graph-structured data. It typically pre-trains graph neural network (GNN) (Dwivedi et al., 2020) without labeled data, in an effort to learn generalizable representations and boost the fine- mie-tunin on downstream tasks. The common theme across recent GCL studies is instance discrimination (Dosovitskiy et al., 2014; Purushwalkam & Gupta, 2020)- - viewing each graph as a class of its own, and differing it from other graphs. It galvanizes representation learning to capture discriminative characteristics of graphs.
Towards this end, leading GCL works usually employ two key modules: graph augmentation and contrastive learning. Specifically, graph augmentation adopts the "intra-graph" strategy create multiple augmented views of each graph, such as randomly dropping nodes (You et al., 2020) or adversarially perturbing edges (Suresh et al., 2021). The views stemming from the same graph constitute the positive samples of this class, while the views of other graphs are treated as negatives. Consequently, contrastive learning encourages the agreement between positive samples and the discrepancy between negatives. This procedure essentially imposes "invariance" (Purushwalkam & Gupta, 2020; Dangovski et al., 2022) upon representations - making the anchor graph's 's repre- sentation invariant to its intra-grat graph augmentations (Figure la). Formally, let g be the anchor graph, P be the groups of intra-graph augmentations, and Φ(-) be the GNN encoder. The "invariance to intra- sirragrraph augmentations" mechanism states ((g)
(g)). Yp P the representation Φ(g) is insensitive to the changes in augmentation p, where (p(9) is the action of augmentation p on graph g. We refer to works adopting this mechanism as Invariant Graph Contrastive Learning (I-GCL).
However, we argue that invariance to intra graph augmentations alone insufficient to improve the semantic quality of graph representations and boost the downstream performance:

---

Under review as a conference paper at ICLR 2023
view view2
g
22
0102)
P2
Tp
X
(G)
g
((G)
(a) Invariance.
(b) Equivariance
Figure I: (a) invariance to intra-graph augmentations; (b) equivariance cross-graph augmentations.
Limiting the augmentations to local substructures of an individual graph is aggressive (Purush- walkam & Gupta, 2020; Wang et al., 2022) insofar as the augmented views fragmentarily or even wrongly describe the characteristics of the anchor graph. Take a molecule graph as an example. After randomly dropping some nodes, one view could hold a cyano group (-C=N) that determines the property of molecule hypertoxie, while another could corrupt this functional group. Thus, intra-graph augmentations are inadequate for presenting a holistic view of the anchor graph.
Worse still, aggressive augmentations easily make two positive views far from each other, but the invariance mechanism blindly forces their representations to be invariant. Considering the molecule graph's views again (cf. Figure la), invariance- guided contrastive learning simply maximizes their representation agreement, regardless of the changes in the hypertoxic property. Therefore, it might amplify the negative impact aggressive intra-graph augmentations and restrain representations from reflecting the instance semantics faithfully.
To mitigate these negative influences, we get inspiration from the recent work on equivariant self- supervised learning (E-SSL) (Dangovski al., 2022). It splits the augmentations into two parts, to which representations should be insensitive and sensitive, and then establishes the invariance and cquivariance mechanisms correspondingly. The idea of "cquivariance" is our focus, which makes representations aware of semantic changes caused by certain augmentations H. Here we formulate it as (TT (g)) = T/ (o(g)) Vh E H. where (g) and TV (og)) are the actions of augmentation on graph g and representation ((g), respectively. Jointly learning equivariance to sensitive augmentations H and invariance to insensitive augmentations P promising to shield representations from the harms of aggressive augmentations. Nonetheless, it is hard, without domain knowledge (Dangovski et al., 2022; Chuang et al., 2022) or extensive testing (Dangovski et al., 2022), to tell apart sensitive and insensitive augmentations.
To embody equivariance in GCL, we propose a simple but effective approach of Equivariant Graph Contrastive Learnnng (E-GCL). E-GCL is instantiation of E-SSL for graphs. Unlike previous E-SSL works, E-GCL leaves existing intra-graph augmentations untouched, and creates new augmentations through the ccross-graph" strategy. Concretely, inspired by mixup (Guo & Mao, 2021; Zhang et al., 20,8), the cross-graph augmentation interpolates the raw features of two graphs (i.e., Tr), while employing the same interpolation strategy on the graph labels that are portrayed by graph representations (i.e., ), The augmentations across graphs not only maintain the holistic information on self-discrimination, but also are orthogonal to the intra-graph augmentations. On the top of intra- and cross-graph of augmentations, E-GCL separately builds the invariance and equivariance principles to guide the representation learning. The equivariance to cross- prosp-rraph augmentations diminishes the harmful invariance to aggressive augmentations that change global semantics. Integrating two principles enables representations to be sensitive to global semantic shifts across different graphs and insensitive to local substructure perturbations of single graphs. Experiments show that E- GCL achieves promising performances to surpass current diverse settings. We also demonstrate E-GCL's generalization to various SSL frameworks, including BarlowTwins (Zbontar et al., 2021), GraphCL. (You et al., 2020) and SimSiam (Chen & He, 2021).
GCL models, across
2 PRELIMINARIES: INVARIANT GRAPH CONTRASTIVE LEARNING
We begin by presenting the instance discrimination task and the invariance mechanism of I-GCL, and then introduce two key ingredients: graph augmentations and contrastive learning.
2

---

Under review as a conference paper at ICLR 2023
Instance Discrimination. Let G = (gm | be the set of unlabeled graph instances. We denote graph instance E 9 by (V,E) involving the node set V and the edge set E. This graph structure can be represented as an adjacency matrix A E {0, 1 1)1V where Auu = 1 if the edge (u, v) E from node node holds, otherwise Aue O. Moreover, each node E V could have d1 di-minensoonal features X, E R while each edge (2, u) E & might have d2- dz-dimensional features Cue E R2
On the graph data & without annotations, contrastive self self-supervised learning (SSL) aims pre-train a graph encoder e : G + IR that projects the graph instances to a d-dimensional space, so as to enhance the encoder's representation ability and facilitate its fine-tuning in downstream tasks. Towards this end, a prevailiing task of pre-training is instance discrimination (Dosovitskiy et al., 2014; Purushwalkam & Gupta, 2020; Li et al., 2021)- - treating each graph instance as one single class, and distinguishing it from the other graph instances.
Invariance. A leading solution instance discrimination is to maximize the representation agreement between augmented views of the same graph, while minimizing the representation agreement between views of two different graphs. essentially encourages each instance's ' representation be invariant to the augmentations (Dangovski et al., 2022: Grill et al., 2020; Zbontar et al., 2021). Mathematically, invariance can be described by groups (Dangovski et al., 2022; Kondor & Trivedi, 2018; Maron et al., 2019a). Let P be group of augmentations (aka. transformations). Invariance makes the encoder insensitive to the actions T' : P x 9 + G of the group P on the graphs G. formally:
((g) (Ty(g)), Vp EP Vg E g,
where Tp(g) :=
is an action of applying the augmentation p on the instance Dictating invariance the encoder will output the same representations for the original and augmented graphs Probing into Equation (1), we find two key ingredients: intra-graph augmentation and contrastive learning, and will present their common practices in prior studies.
T(p,g) ==
Intra-graph Augmentation. Typically, the augmentation group P is pre-determined to imply prior knowledge of graph data. Early studies (Hu et al., 2020; Qiu al., 2020; You et al., 2020; Zhu et al., 2020) instantiate augmentations as randomly corrupting the topological structure, node features, or edge features of individual graph instances. For example, AurMasking (Hu et al., 2020) masks node and edge attributes, and applies an objective to reconstruct them. GCC (Qiu et al., 2020) explores random walks over the anchor graph to create different subgraph views. GraphCL (You et al., 2020) systematically investigates the combined effect of various random augmentations. Despite the success, random corruptions are too aggressive to maintain the semantic consistency (Guo & Mao, 2021) between the anchor graph and its augmented views. The invariance principle blindly ignores the semantic shift, thus easily pushing dissimilar patterns together and making pernicious impact on the representation learning. Some follow-on studies (Zhu et al., 2021; Subramonian, 2021; Suresh et al., 2021; Xu et al., 2021) learn augmentations instead to underscore salient substructures, so as to mitigate the semantic shift. For instance, GCA (Zhu al., 2021) applies node centralities to discover important substructures in social networks. MICRO- Graph (Subramonian, 2021) 1) learns chemically meaningful motifs to help the informative subgraph sampling. More recently, AD-GCL (Suresh et al., 1021) adopts the idea of information bottleneck to ad adversarially learn the salient subgraphs.
Contrastive Learning. Upon the augmented views, the contrastive learning objective is to classify whether they come from identical instances. Specifically, it pulls the augmented views derived from the same instance (i.e.. positive samples) together and pushes the views of different instances (i.e. negative samples) apart (Chen et al., 2020; He et al., 2020). The common practices of this objective InfoNCE (van den Oord et al., 2018). NCE (Misra & van der Maaten, 2020), and NT-Xent (Chen et al., 2020). Here we consider the N'T-Xent adopted by GraphCI. Given minibatch of graph instances )
1i=1- it first generates two different augmented views, denoted as {o] lo! = Tp, (g). Pi ~ P
and [g. g² = Tp, (g). P2 ~ PM and then feeds them into the encoder to yield the representations as {z] z = p( N
1 and {z,|z? = p()(g))) where p(-) is an MLP projection head. Formally, the loss of NT-Xent is:
(2))) 1 N log プラ exp(s (z),)()))))) N
exp(s(2), 2;)/T)
(2)
where s(-) is the function of cosine similarity, and is temperature hyperparameter.
In a nutshell, the interplay between intra-graph augmentation and contrastive learning is tailor-made for invariance to make the encoder insensitive to differences between the anchor and augmented
3

---

Under review as a conference paper at ICLR 2023
Invariance
Equivariance
Projector
Projector
Projector
Projector
Represenration
interpolation
GNN encoder
GNN encoder
GNN encoder
GNN encoder
interpolation
Intr-ggraph augmentation
Cross graph augmentation
Figure 2: The framework of E-GCL. The GNN encoder learns invariance for intra-graph augmentation of dropNode and equivariance for cross-graph augmentation of graph interpolation.
views. this work, we explore equivariance on cross-graph augmentations to make the encoder sensitive to the changes in self-discriminativ information.
METHODOLOGY: EQUIVARIANT GRAPH CONTRASTIVE LEARNING
Here present the E-GCL framework, which imposes two principles - invariance to intra-graph augmentations (Figure 2 left) and equivariance to cross-graph augmentations (Figure right) - on the representation learning, aiming to mitigate the potential limitations of I-GCL. Next, we start with the concepts of equivariance and cross graph augmentations.
3.1 EQUIVARIANCE
Inspired by the recent E-SSL studies (Dangovski al., 2022; Chuang et al., 2022), we aim patch invariance's 's potential limitations with equivariance. Mathematically, with group of augmentations H, the encoder d--) is said be H-equivariant w.n.. the actions T G - and T Rd + R of the group H. applied on the graph space G and the representation space Rd if
(Th((g) (o(g))) Vh E H. 4g G,
(3)
where (g) the action applying the transformation on the graph instance g. while (o())))
is the action of h on the representation @(g). H-equivariant requires that, given transformation E H. h's influence on the graph should be faithfully reflected by the change of the graph's representation. Taking Figure 1b as example, given the graph's global semantics are perturbed by graph interpolation, the representation yielded by the equivariant encoder should transform in definite way. Jointly analyzing Equations (1) and (3), can be shown that invariance is special case of equivariance when setting T% as the identity mapping. However, generalizing GCL to equivariance remains unexplored, thus focus of our work.
Furthermore, as suggested in the recent E-SSL studies (Dangovski et al., 2022; Chuang et al., 2022), jointly imposing invariance some transformations P and equivariance to other transformations H is promising to result in better representations than relying solely on one of them. Here we term P. and H insensitive and sensitive transformations, respectively. For example, in computer vision, E-SSL (Dangovski et al., 2022) sets grayscale of images as P, while treating rotations as H; in natural language processing, DiffCSE (Chuang al., 2022) treats the model dropout as P. while using the word replacement as H. Clearly, it is of crucial importance to partition augmentations into P and H. Nonetheless, these studies either conduct extensive testings on the impact of different partitions (Dangovski et al., 2022) which is time-consuming, or exploit domain knowledge to heuristically partition (Chuang et al., 2022) which might generalize poorly to other domains. Hence, is infeasible graph augmentations. Worse still, different graph augmentations stem
to apply these strategies mostly from the perturbation of graph structures, thus highly likely to corrupt the same attributes of graphs. Taking the graph in Figure la example, masking the nitrogen N atom dropping the C' = N bond will both corrupt the cyano group and break the corresponding molecular properties. In nutshell, owing to (1) the common paradigm of structure corruption and (2) the risk of categorizing
4

---

Under review as a conference paper at ICLR 2023
them all as insensitive augmentations, conservatively argue that it is hard to partition graph augmentations into sensitive H. and insensitive parts P.
In this work, leaving partitioning untouched, we remain intra- gtra- augmentations as the insensitive transformations P and propose new augmentations across graphs as the sensitive transformations H.
3.2 CROSS-GRAPH AUGMENTATION
We first introduce graph interpolation (Guo & Mao, 2021) create cross- coss-graph augmentations H. Different from previous work, we propose an extension of graph interpolation for SSL (Section 3.3). We also connect it to group theory and address its limitation of sensitivity to the relative permutation.
Interpolating Graphs as Cross-graph Augmentations. Given two graph instances g E g and g' E G. employ mixup (Zhang et al., 2018), simple yet effective linear interpolation approach, on the input features and class labels, respectively:
0 0= Ag + - >)g', 9 = xy (1 (1- (1-\)y/
where y and s/ separately denote the one-hot encodings to indicate the instance identities of y and g' in the instance discrimination task; À ~ Beta(a, a) E [0. 1] is the interpolation ratio sampled from a Beta distribution, in which a is a hyperparameter. This mixup strategy is initially proposed for supervised learning, aiming to put the interpolated samples in-between different classes and make the decision boundary robust to slightly corrupted samples (Verma et al., 2019; Zhang et al., 2018; 2021). Despite the success of mixing image and text, it is challenging interpolate graphs due the structural differences between graph instances (e.g., varying topologies and sizes).
To this end, we draw inspiration from the recent work (Guo & Mao, 2021) perform linear interpolation between graphs. Specifically, with g (V,E) and g' = E"), we mitigate their structural differences by padding virtual nodes and edges, which are associated with zero features 0. Assuming [V] S. |V"] can be updated as a new graph with [V] nodes, where the original node set V remains unchanged but adds | - | dummy virtual nodes, and the original nodes connect the virtual nodes with dummy virtual edges. Having padded two graphs to the same size, now we can directly add them up. Before the interpolation, we first merge two node and edge sets as the new ones: V = V', E = E E Then, 9 = >g + (1 - A)g' in Equation (4) is achieved by exerting linear interpolation on the adjacency matrices, node features, and edge features:
À = AA + (1- (1-A)A', x = Ax + (1 - A)xx, Tu = λeav + (1- X)e'.. (5) where A and A' are the adjacency matrices of g and g' after padding; x,, and x', are the features of node v e V. which separately come from g and g'; similarly, eae and e',r separately denote the features of edge (u, v) E & from g and g'. Consequently, we generate cross-graph augmentation.
Connecting Cross-graph Augmentations to Groups. In the language of groups, we can describe the cross- gross-graph augmentation in Equation (4) as group of transformations. Given the input (A, 9, g"), we can systemize the graph interpolation as two steps: (1) feature rescaling: ğ = >g. which rescales the node and edge features of with the ratio X; (2) instance composition: ğ = C(g. g') = 9 + ğ', which adds the other rescaled graph g' (1 Ajg'. To construct a closed space for graph interpolation, first define و {g A E [-1, sampling of rescaled graphs. We allow À < 0 to include the inverse elements of graphs. Then, we generate I =< ğ > by combining the graphs in 9 via instance composition. We show that (I, C*) forms a group in Appendix A. is worth noting that each instance can be viewed as transformation to others, i.e., -(g.-) == C.).), such that semantic shifts can be described via algebraic operators.
9} by performing feature rescaling on g to enable direct
Group Averaging for Insensitivity to Relative Permutation. Note that Equation (5) can output different interpolations, when the node orders of one graph or padding positions of dummy nodes change. We ascribe operations on "node orders" and "padding positions" to the "relative permutation" between two input graphs. Unlike images and texts, the canonical permutations (orderings) of nodes in graphs are unlabeled. The default node permutations encode nothing useful about graph semantics (Xu et al., 2019; Hamilton al., 2017). Thus, we enforce insensitivity relative permutations by randomly permuting nodes the bigger graph before graph interpolation. Next, we justify and develop this design with group averaging.
Group averaging can make known architectures invariant to new symmetries (Puny et al., 2022; Yarotsky, 2022). It can be used for strict invariance relative permutations. Let P ~ Sn be random
5

---

Under review as a conference paper at ICLR 2023
permutation, and Tp g be permuting graph g by P. Assuming [g| 2 Ig'). we obtain the graph interpolation as XTp o g + (1 A)g' and its representation as 6(XTp o + (1 A)g"). ). To achieve strict invariance to relative permutations, apply group averaging the permutation operators:
((1
Σ |n PES.
((ATp- + A)g'),
where & is the function of group averaging. & is invariant to relative permutations between, and g', in the sense that (A,, 9, g') Φ Tp g.g') $(), g. Tp o g¹) for all P, P' S.. Intuitively, is achieved by averaging over all relative permutations. See Appendix A.2 for the proof. However, the intractability of averaging over S, naturally arises as a problem. Following Murphy et al. (2019), our random permutation strategy <(XT g + (1- (1-))g') is an unbiased estimator of Φ. Further, this strategy optimizes poc o toward an optima insensitive the relative permutation. By Proposition 1, we conclude that using random permutation is tractable surrogate for optimizing an invariant network to relative permutation. Appendix D.2 shows the influence of sampled permutation numbers. For simplicity, we still use >g + (1 X)g' to represent graph interpolation in the rest text.
Proposition 1. The contrastive loss with poTTpog+ poogppp+1--- (1-A)g') upper bounds the loss of an invariant network to relative permutation [S, Epes, plo(A:TP o gi + Xx)g()).
3.3 EQUIVARIANCE TO CROSS-GRAPH AUGMENTATIONS
Revisiting Equation (4), we can find the two terms of interpolation strategy align with the equivariance mechanism in Equation (3). The semantic change caused by the feature interpolation is equivalently reflected by the label interpolation. Hence, we can instantiate the equivariance mechanism based on the feature and label interpolations. In the SSL setting, interpolate graph representations the alternative for label interpolation. Graph representations are derived from the encoder Φ(-) with a global readout layer to summarize the graphs' global semantics. Hence, shown by the right side of Figure 2, we can parameterize equivariance approximately as:
á(gg + (1 -)g') Aó(g) + A)ogg')
(7)
Minimizing the distance between Equation (7)'s two sides allows the encoder to improve sensitivity to the global semantic shifts caused by cross-graph augmentations.
Although the strict equivariance is hardly guaranteed, experiments show that approaching Equation (7) can boost the performance on downstream tasks (cf. Section 4.2). Furthermore, if the encoder is powerful enough to distinguish the interpolated graphs, it has a deterministic reflection in the repre- sentation space (GG) for a fixed transformation C(g. in graph space G (Non-trivial Equivariance (Dangovski et al., 2022)). See Appendix A.1 for the proof.
Proposition 2. Assuming the encoder can detect the isomorphism of interpolated graphs, there exists a GNN encoder & that is non-trivially equivariant to the graph interpolation transformation.
3.4 IMPLEMENTING E-GCL
This section details our implementation of E-GCL (Figure 2). Specifically, given a minibatch of graph instances {9i J. impose (1) invariance to intra-graph augmentation and (2) equivariance to cross- cross-graph augmentation simultaneously on the shared encoder.
Invariance. For the invariance principle, we follow the I-GCL paradigm to resort to the standard intra-graph augmentations P (e.g., randomly dropping nodes in GraphCL.), and create two augmented views of individual graphs: {gt Consequently, the encoder o6-) brings forth two representation lists: {z! Iz! p(o(g! ) and [x] o( in which p(-) is an MLP projector.
Tp (g),P1 P)N and {g lo Tpp (g), P2 ~ P
Equivariance. For the equivariance principle, we first randomly shuffle the graphs in {o? N J termed as (o2n,) 1 where म : [N] - [N] is the function of random shuffling. Following the left side of Equation (7) and applying random permutations P. ~ Sn for all i e [W], we create the feature interpolations and then generate their representations as {z/12z = p(O(XTp, (1-X)=="))
Meanwhile, according to the right side of Equation (7), we arrive at the representation interpolations as (2. 1 p(A((g!) + 2))())((()))

---

Under review as a conference paper at ICLR 2023
Table Main experiment performances. denotes our reproduced results using the released codes. Other baseline results are borrowed from the original papers. Bold indicates the best performance and underline indicates the second best performance.
(a) Unsupervised learning accuracies (%) on the TU datasets.
Dataset
NCII
PROTEINS
DD
MUTAG
COLLAB
RDT-B
RDT-M5K
IMDB-B
AVG
GAIN
No Pre-train"
72.67at
73.81am
77.01
84, 34.26aus
62.92.am
72.45awr
45.32acm
67.54
69.50
InfoGruph*
78.27-00
74.56-am
77.01
87.87
7
83.22aam
55.82
70.96
74.81
5.31
GraphiLL*
79,25
74.50.ass
78.24
87.6 37.65a
71.59
90,54mmm
55.67
71.38uua
76.10
JOAO*
78.55
74.39_am
77.44mass
88.85.1.1
70.52-4m
88.49
56.04403:
71.44.05
75.72
6.22
AD-GCI.*
72.95-44s
73.62-am
76.06w
89, 29.25.
70.70-a
87.03 18
54.81
71.62mam
74.51
5.01
GraphMAE*
75.00mcs
73.92.aw
76.15
87.17au
72.92:
81.27
49.63atar
71.96uts
73.50
4,00
R-GCL*
78. 78.79um4z
74.60.am
79.14
88.12at.
71.44uae
90.1
55.96uuz
71.84
76.25
6.75
GCL
79.95
75.18.a.
77.83
88.20uae
74.63.a2
90.71.am7
76.93
(b) Transfer learning ROC-AUC (%) scores on the MoleculeNet. GTS denotes GraphTrans.
Dataset
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
AVG
GAIN
No Pre-train*
67.8.1
73.9 .3.9
62.4-a4
58.3
62.64a
73.4
76.541
76.8es
69.0
Infomax*
68.5at.
75. .2.4
62.6an
58.6w0.
71.2as
73.1a19
76.821
74.4at
70.1
1.1
ContextPred
72.2
75.6ias
63.5mn
60.6m9
70.2ea
74.3a1
77.6.as
79.0wa
71.6
2.6
GraphCL
69.7
73. 92n7
62.4ams
60.5a09
76.0228
69.8227
78,5+1a
75.4
70.8
1.8
JOAO
71.4m99
74.32ns
23,2
60.5
81.02t.
73.7210
77.521a
75.5
72.1
*DGCL*
70.5
74.5
63.0ans
59.
78.5+37
71,5
75.9.14
74.0
70.9
1.9
GraphLOG** *
71.0at1
74.9.m
62.8ans
59.7.09
76.9+1.
70.8
75.8+14
82.9.49
71.8
2.8
GraphMAE*
72.2mm9
75.
63.0ma
58.5
80.5
75.7
76.4ms
81.3um
72.8
3.8
RGCL*
71.2,09
75.3ins
61.2
85.0
11.
77.3ins
75.7
72.7
3.7
E-GCL
72.320.
74.9207
64.0aa3
62.8.0s
83.1
78.8
76.3
18.1
73.8
4.8
E-GCL, GTS
72.3,08
77.92ns
66.0ans
62.4at0
80.7a30
79.4.21
77.8at1
79.7
74.5
5.5
Cooperative Game between Invariance and Equivariance. Based on the representations, we optimize the invariance and equivariance losses together with a weighting hyperparameter w E [0,
LEGEL (1 w) [((2))
12 (2 1 tw
{((2)) (2;1
invariance loss
equivariaane JosS
where ((),-) is the loss encouraging the insensitivity between augmented views of the same graph, which is determined by the SSL backbone, such as NT-Xemt (cf. Equation (2)) adopted by the GraphCL. Beyond contrastive learning, E-GCL is also applicable to various other SSL backbones, including Barlow and SimSiam. In nutshell, the invariance loss underscores the insensitivity to intra-graph augmentations, while the equivariance loss induces the sensitivity to cross-graph augmentations. The cooperative game between these two losses helps resolve the potential limitations of the conventional I-GCL paradigm, thus improving the expressive power of the encoder.
4 EXPERIMENT
In this section, we conduct experiments to answer the following research questions: RQI: How effective is the proposed E-GCL in graph representation learning, and how does generalize to existing SSL frameworks? RQ2: What are the properties of E-GCL and the effects of its components?
In Appendix D.I, we present more ablation studies about 1) using G-mixup (Han et al., 2022), 2) interpolating representations at different positions, and 3) interpolating large and small graphs
4.1 EXPERIMENTAL SETUP
Here we briefly introduce the baselines, datasets and evaluations. Details are in Appendix E. For a fair comparison, E-GCL uses the same intra-graph augmentation GraphCL. If not noted, E-GCL employs a BarlowTwins backbone. We study E-GCL's generalization to other SSL frameworks later.
Baselines. We compare E-GCL with the following state-of-the-art graph pre-training methods: Infomax (Veličković et al., 2019), InfoGraph (Sun et al., 2020), ContextPred (Hu et al., 2020), GraphCL (You et al., 2020), JOAO (You et al., 2021), AD-GCL (Suresh et al., 2021), GraphLOG (Xu et al., 2021), GraphMAE (Hou et al., 2022), and RGCL (Li et al., 2022).
Unsupervised Learning evaluates the pre- pre-trained GNNs for prediction on the same dataset. Follow- ing (You et al., 2020), evaluate E-GCL on the eight TU datasets, including biochemical graphs
7

---

Under review as a conference paper at ICLR 2023
Table 2: Generalization to diverse SSL frameworks, Red denotes equivariance improves performance.
(a) Unsupervised learning accuracies (%)
Dataset
NCTI
PROTEINS
DD
MUTAG
COLLAB
RDT-B
RDT-M5K
IMDB-B
AVG
GAIN
GraphCL
79.25 40
74.50
78. 24
87.65 87.65.
71,39
90.54
55.67-24s
71,38:
76.10
+Equivariance
80.22.a.s
74.57am
19.15
89.79
72.75
91.28
55.80aa4
71.70aae
76.91
0.81
Barlow
79.60.a
74.90
77.22
86.92.17
72.94ums
90.11.cas
55.40.am
a1.28
76.05
+Equivariance
79.95
77.83ums
88.20ucas
74.63
90.71azr
56.90
72.02.are
76.93
0.88
(b) Transfer learning ROC-AUC (%) scores. GTS denotes GraphTrans.
Dataset
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
AVG
GAIN
GraphCL
69.7xaz
73.9um
62.4ms
60.5acM
76.0aza
69.8.zy
78.5a1
75.4
70.8
+Equivariance
71.8
75.5
63.5
60.6et4
74.0z21
74.7a07
76.5
78.7s08
71.9
1.1
SimSiam
70.5
74,4.0
63.3mas
60.8
77.4.
70.5ar0
77.3-0
76.2
71.3
+Equivariance
71.4
75.0sos
63.2pas
59,3st9
81 1124
73.5at4
77.6.10
75.8
72.2
0.9
BarlowTwins
70.6
74.3
63.8
61.3ien
82.0Aa
73.0
77.1-1a
73.9kan
72.0
+Equivariance
72.3
74.9
64.0was
62.8e0s
83.1
78.8
76.3e0s
78.1
73.8
1.8
Barlow GTS
71.0Aa
77.0ma
65.0mar
61.9
76.94s
79.4.1x
78. 1-15
77.5
73.4
+Equivariance
72.3
77.9
66.
62,4
80.7am
79.4221
77.8
79.7a24
74.5
1.1
and social networks. Specifically, we pre-train a three-layer GIN (Xu et al., 2019) and feed the generated graph representations into SVMs for evaluation. We report the average values and standard deviations of accuracies (%) of five different runs, each of which corresponds to 10-fold evaluation. Following (You et al., 2021), we report test accuracy of the epoch selected by validation set.
Transfer Learning tests the pre-trained GNN's transferability to downstream tasks. Following (Hu et al. 2020), we use the two million molecule samples from the ZINC15 dataset (Sterling & Irwin, 2015) for pre-training and eight multi-label classification datasets derived from the MoleculeNet (Wu et al. 2018) for fine-tuning. Fine-tuning datasets divided by scaffold split to create distribution shifts among the train/valid/tes sets, so as to provide more realistic estimations of the molecule property prediction performance. Following (Hu et al., 2020), we implement E-GCL with a five-layer GINE. Further, we push the limits of E-GCL with the GraphTrans (Wu et al., 2021) backbone. Graph- Trans stacks four-layer Transformer (Vaswani al., 2017) on top of the GINE learn long range interactions. For evaluation, we pre-train& a model and repeatedly fine-tune it on downstream datasets ten times. We report the averages and standard deviations of ROC-AUC (%) scores. Following (You et al., 2020), we report test set performance of the last epoch.
4.2 MAIN RESULTS (RQ1)
Unsupervised Learning Results. Table Ia presents the unsupervised learning performance in TU datasets. The last column denotes the improvement compared to a randomly initialized GNN. E-GCL achieves the best performance in six out of eight datasets and the top three performances in the other two datasets. It also achieves the best average improvement of 7.43% compared to randomly initialized GNN. We attribute E-GCL's good performances to the equivariance principle of cross- graph augmentation. Other methods apply only the invariance principle of intra-graph augmentation, thus failing to generate representations as discriminative as E-GCL.
Transfer Learning Results. Table 1b presents the fine-tuning performances in transfer learning. E- GCLs have achieved the best performances among all methods. Specifically, E-GCL with GraphTrans achieves the best performances in four out of eight datasets and the best average improvement of 5.5% compared to randomly initialized GNN. When using the GINE backbone, E-GCL continues to show improvements over all the baseline models in average performance. It shows that the equivariant pre-training of cross-graph augmentation gives F-GCLI a better starting point for fine-tuning. Previous models apply only the intra-graph augmentation, which might bring together dissimilar patterns. Notice that, E-GCL and other models' improvements over previous works are not consistent across fine-tuning datasets. We attribute the inconsistency to the Out of Distribution (OOD) evaluation setting in MoleculeNet. The validation set does not overlap the test set, which makes preventing overfitting and underfitting troublesome. We follow the evaluation protocol of previous works (You et al., 2020; Xu et al., 2021), which, however, does not guarantee the convergence of performance. We leave more stable fine-tuning for the OOD evaluation as future work.
In summary, E-GCL establishes new state-of-the-art in unsupervised learning and transfer learning.
8

---

Under review as a conference paper at ICLR 2023
Accurasy (56
77.0
26.5
06.68
76.5%
76.47
76.11
76.75
76.45
76.54
26.55
7663
76.91
76-23
76.65"
76.75
76
76.45
Backhone
Rackbone E.GLL
76.0
E-GLL
0.1
0.4
0.1
04
0.
0.2
0.5
0.4
0.5
1.0
0.1
0.2
0.5
0.4
0.5
1.0
DarlowTwins 0.4)
CraphCL (w 0.3)
BarlowTwins
0.11
GraphCL wia 0.1)
(a) Different Beta(a, a) distributions.
(b) Different equivariance and invariance tradc-off.
Figure 3: Hyper-parameter sensitivity study in unsupervised learning.
Generalization to Different SSL Frameworks. To highlight E-GCL's improvement and generaliza- tion ability, we apply the equivariance principle to three representative SSL frameworks of different flavors: GraphCL GCL, SimSiam (Chen & He, 2021) asymmetric Siamese Networks, and BarlowTwing - decorrelating feature dimensions (Table 2). We observe that equivariance consistently improves about 1% of SSL backbones' average performances in both unsupervised learning and transfer learning. The results demonstrate the effectiveness of equivariance and its generalization ability diverse SSL frameworks and different experimental settings.
4.3 ANALYZING THE PROPERTIES OF E-GCL (RQ2)
Hyper-parameter Sensitivity. Figure 3 presents E-GCL's sensitivity with respect the shape parameter of (et a) distribution and the trade-off factor between invariance and equivariance. Shown by Figure 3a, a = 0.1 gives the best average performance for both the Barlow and GraphCL backbones in unsupervised learning. We also observe that the optimal value differs for SSL backbones (Figure 3b). The best w for BarlowTwins and GraphCL are 0.4 and 0.3, respectively. When 1, the invariance loss vanishes and the performances drop to the lowest. This demonstrates that the invariance mechanism and equivariance mechanism are complementary to each other and their cooperation makes for better graph representation learning.
Training Dynamics of Aligmment and Uni- formity. To understand how E-GCL im- proves over I-GCL, we study their behaviors through the lens of alignment and uniformity losses (Wang & Isola, 2020), which consti- tute the asymptotic objective of contrastive learning. On unit hypersphere, Alignment measures the closeness of the positive pairs and uniformity measures the evenness of the sample distribution. We apply the con- trastive backbone GraphCL with dropNode as the intra graph augmentation and graph interpolation the cross-graph augmenta- tion. Figure 4 shows the losses on each type of augmented samples and their concatenations (i.e., intra+cross aug). Compared to E-GCL cross-aug, E-GCL intra+cross aug has better uniform loss but worse alignment loss. E-GCL achieves much better aligument and uniformity on cross-graph graph augmentations than I-GCL, with a slight sacrifice of alignment on intra- aatra-augmentations explicitly optimize the cross- gross- augmentations and trade-off the optimization of antra-graph aug- mentations. Combining intra- and cross-graph augmentations, E-GCL achieves better alignment and uniformity than I-GCL
0.7
0.6
2
align loss
sug
I-TCL emassnn
E-GCL intr-sug
0.3
cress-ang
0.2
E-GLL inte-cross zug
3,875
3,850
3,825
3,800
3,775
3,750
3.725
3,700 3,675
uniform loss
Figure 4: The alignment and uniformity losses when pre-training with chemical molecules. Losses are eval- uated every 100 pre-train steps and lower numbers are better. Arrows denote the losses' changing directions.
This is in expectation as E-GCL applies equivariance to
which explains the better performance.
CONCLUSION AND FUTURE WORKS
In this paper, we propose Equivariant Graph Contrastive Learning (E-GCL) that combines equivari- ance and invariance to learn better graph representations. encourages the sensitivity to global semantic shifts by grounding the equivariance principle as a cross-graph augmentation of graph interpolation. This equivariance principle protects GNNs from aggressive intra-graph augmentations that can harmfully align dissimilar patterns and enables GNNs to discriminate cross- graph augmented samples. Extensive experiments in unsupervised learning and transfer learning demonstrate E-GCL's significant improvements over state-of-fe tfthe-of-the-rtt methods and its generalization ability to different SSL frameworks. In the future, will explore more groundings of the equivariance principle in graphs.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Waiss Azizian and Mare Lelarge. Expressive power of invariant and equivariant graph neural networks. In ICLR, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variance-ivvarrnnc--ovvaaaan regularization for self-supervised learning. In ICLR, 2022.
Peter Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.0120 2018.
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. CoRR,
2021.
Sourav Chatterjee. Matrix estimation by universal singular value thresholding. The Annals of Statistics, 43(1):177-214, 2015.
Haiwei Chen, Shichen Liu, Weikai Chen, Hao Li, and Randall Hill. Equivariant point network for 3d point cloud analysis. CVPR, pp. 14514-14523, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ICML, pp. 1597-1607, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Yung-Sung Chuang, Rumen Dangovski, Hongyin Luo, Yang Zhang, Shiyu Chang, Marin Soljacic, Shang-Wen Wen Li, Wen-tau Yih, Yoon Kim, and James Glass. DifiCSE: Difference-based contrastive learning for sentence embeddings. In NAACL, 2022.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In JCML, 2016.
Taco S. Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. ICLR, 2018.
Rumen Dangovski, Li Jing, Charlotte Loh, Seungwook Han, Akash Srivastava, Brian Cheung, Pulkit Agrawal, and Marin Soljacic. Equivariant contrastive learning. In ICLR, 2022.
Sander Dieleman, Kyle W. Willett, and Joni Dambre, Rotation-invarian convolutional neural networks for galaxy morphology prediction. CoRR, abs/ 50s/1500.,7777, 2015.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimina- tive unsupervised feature learning with convolutional neural networks. In NeurIPS, pp. 766-774, 2014.
Vijay Prakash Dwivedi, Chaitanya Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. CoRR, abs/2003.00982, 2020.
Matthias Fey and Jan Lenssen. Fast graph representation learning with PyTorch Geometric, In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In NeurIPS, 2020.
Hongyu Guo and Yongyi Mao. Ifmixup: Towards intrusion-free graph mixup for graph classification. arXiv, 2021.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, volume 30, 2017.
Xiaotian Han, Zhimeng Jiang, Ninghao Liu, and Xia Hu. G-mixup: Graph data augmentation for graph classification. CoRR, 2022.
10

---

Under review as a conference paper at ICLR 2023
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9726 9735, 2020.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang. and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In KDD, pp- 594-604, 2022.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitmik, Percy Liang, Vijay Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. ICLR, 2020.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks the action of compact groups. In JCML, volume 80, pp. 2752 22660, 2018.
AA Leman and Boris Weisfeiler. A reduction of a graph to a canonical form and an algebra arising during this reduction. Mauchho--Tchhiichsskaaa Informatsiya, pp. 12-16, 1968.
Haoyang Li, Xin Wang. Ziwei Zhang, Zehuan Yuan, Hang Li, and Wenwu Zhu. Disentangled contrastive learning graphs. In NeurlPS, pp. 21872-21888 2021.
Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In ICML, pp. 13052- 13065, 2022.
Haggai Maron, Heli Ham-H Nadav Shamir, and Yaron Lipman. Invariant and equivariant graph networks. In ICLR, 2019a.
Haggai Maron, Ethan Fetaya, Nimrod Segol, and Yaron Lipman. On the universality of invariant networks. In ICML, 2019b.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext- pretext-invariant represettoonrtrrsssttaaatto In CVPR, pp. 6706-6716, 2020.
Ryan I. Murphy, Balasubramanian Srinivasan, Vinayak A. Rao, and Bruno Ribeiro. Janossy pooling: Learning deep crmmutationninvariann functions for variable-size inputs. In ICLR (Poster). OpenReview.net, 2019.
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. CoRR, abs/1707.05005, 2017.
Omri Puny, Matan Atzmon, Heli Ben-Hamu, Edward J Smith, Ishan Misra, Aditya Grover, and Yaron Lipman. Frame averaging for invariant and equivariant network design. In ICLR, 2022.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. In NeurIPS, 2020.
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang. Ming Ding, Kuansan Wang, and Jie Tang. GCC: graph contrastive coding for graph neural network pre- voo-training. In KDD, pp. 1150-1160, 2020.
Teague Sterling and John J. Irwin. ZINC ligand discovery for everyone. J. Chem. Inf. Model., 55 (11):2324-2333) 2015.
Arjun Subramonian. Motif-driven contrastive learning of graph representations. In AAAI, pp. 15980-15981, 2021.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. In ICLR. 2020.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. In NeurlPS, pp. 15920-15933, 2021.
Yuandong Tian, Xinlei Chen. and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In ICML, pp. 10268-10278, 2021.

---

Under review as a conference paper at ICLR 2023
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. CoRR, 2018
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NIPS, pp. 5998-6008, 2017.
Petar Veličković, William Fedus, William L Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In ICLR, 2019.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Joannis Mitliagkas, David Lopez-Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
Vikas Verma, Meng Qu, Kenji Kawaguchi, Alex Lamb, Yoshua Bengio, Juho Kannala, and Jian Tang. Graphmix: Improved training of gnns for semi- eemi-supereised learning. In AAAI, pp. 10024-10032, 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In ICML, pp. 9929-9939, 2020.
Yifei Wang. Qi Zhang, Yisen Wang. Jiansheng Yang, and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
Yiwei Wang, Wei Wang, Yuxuan Liang, Yujun Cai, and Bryan Hooi. Mixup for node and graph classification. In WWW, pp. 3663-3674, 2021.
Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, and Ion Stoica. Representing long-range context for graph neural networks with global attention. In NeurIPS, pp. 13266-13279, 2021.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing. and Vijay Pande. Moleculenet: benchmark for molecular machine learning. Chemical science, 9(2):513-530, 2018.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? ICLR, 2019.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In ICMI., volume 139, pp. 11548-11555 11558, 2021.
Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. Constructive Approximation, pp. 407-474, 2022.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In NeurlPS, 2020.
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. In ICML, Proccedings of Machine Learning Research, pp. 12121-12132, 2021.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In NeurlPS, 2017.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In ICMI., 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In ICLR, 2018
Yifan Zhang, Bryan Hooi, Dapeng Hu, Jian Liang, and Jiashi Feng. Unleashing the power of contrastive self-supervised visual models via contrast-regularized fine-tuning. In NeurIPS, pp. 2021.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. CoRR, 2020.
Yanqiao Zhu, Yichen Xu, Feng Yu. Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW, pp. 2069-2080, 2021.
12

---

Under review as a conference paper at ICLR 2023
A PROOF
A. PROOFS ON GRAPH INTERPOLATION
Proof of that (Z, C) forms group. We first recall of the definition of (I,C) as follows:
When viewing 9 as the anchor being augmented, we can systemize the mixup as a combination of two steps: (1) feature rescaling: / = Ag. which rescales the node and edge features of g with the ratio A; (2) instance composition: 9 = C o') = ÿ + g', which adds another rescaled graph j'. Let و -ØlA> [0, 11 g) be the enlarged set of graphs via feature rescaling, I =< 9 > be the generated group by combining the graphs in 9 via instance composition. We show that (Z, C) forms group.
Proof. I is set of graphs and C(.,) is binary instance composition operation on I. It suffices to show that (I, C)) is group it satisfies the following conditions:
Associativity Law. Cigi, C(g2, 9a)) = 91 + (02 + 93) (g) + C(C(g) 92) 9a) for all f1, 92, E I. Notice that, both resulting node feature matrices and adjacency matrices of both sides are equivalent.
Existence of Identity. Let be the empty graph with no nodes and no edges, then C(g. e) g = -((e,)) for all g I, then the empty graph is the identity.
Existence of Inverse. For any graph g I, there exists a graph g 1 structure of g but with inverse values of node features and edge weights, aka. Xy- -Xg
I who has the same
and Ag- = AAg such that C(g. g + C.
+
Proposition 2. Assuming the encoder detect the isomorphism of interpolated graphs, there exists a GNN encoder & that is non-trivially equivariant to the graph interpolation transformation.
Proof. For proof, it suffices to show that graph interpolation transformation satisfies the assumption of E-SSL's (Dangovski et al., 2022) Non-trivial Equivariance Proposition. We can then apply the Non-trivial Equivariance Proposition to conclude the proof.
The assumption states that, "given P as the group of our interest, if (T)( (TT) then g g' and p p' for all p.p' P and 9.96 E G." We rewrite the assumption with the graph interpolation formulation of equivariance: for all Ag A)g2. X N9 (1 X')22 E & with X, X [0, 1). if Φ6 (1- = o(X'g (1-X)gg) then either /) >gn = N91, (1 (1-A)g2 = (1-X)g) or 2) Agı (1 X)92, (1 Ag2) = X'g.- We now prove this assumption.
We consider graphs without edge features. We have assumed that GNN encoder can detect graph isomorphism of interpolated graphs. Therefore, we can infer the equivalence of the graphs based on the equivalence of graph embeddings. Let g = Agı + (1 A)g2 and g' X'g + (1 X X'gg2 we
have g = g' because of ((ğ) ((ğ').
Using the Intrusion intrusion-Freeness Theorem in (Guo & Mao, 2021), the graph interpolation transformation invertible and the resulting original graph pair and mixup coefficient unique, aka. the equivalence of the interpolated graphs infers the equivalence of the original graphs and the interpolation coefficient
X. Thus, we have either 1) X = N. 91 g 92 gh. 2) X.g g1 92. 92 = 91. In either case, the assumption holds.
Discussion. Proposition 2 relies on a powerful GNN encoder that can detect the isomorphism of interpolated graphs. This powerful GNN encoder exists both in theory and practice: 1) in theory, the universality of GNNs has been proved that they can approximate any function on graphs (Azizian & Lelarge, 2021); 2) in practice, Puny et al. (2022) have proposed family of universal GNNs. The possible limitation lies in the complexity these methods (Puny et al., 2022; Maron et al., 2019b). We have tested E-GCL with the GIN and Graph architectures. GIN is at most as powerful the
13

---

Under review as a conference paper at ICLR 2023
1-WL test (Leman & Weisfeiler, 1968; Azizian & Lelarge, 2021) and GraphTrans improves its ability to model long range interactions. Our experiments show that GIN and GraphTrans is sufficient to demonstrate the improvement of E-GCL over previous methods. Therefore, we leave the adoption of universal GNNs (Puny et al., 2022) as future work.
Theorem (Guo & Mao, 2021) relies the assumption in either Lemma 2 Lemma 3 of their paper. The assumption of their Lemma states that: "The node feature vectors for all graphs take values from a finite set and the values in the set are linearly independent". This assumption holds strictly for chemical molecules (Hu et al., 2020), in which the node features are atom types that are finite and linearly independent (i.e., you cannot combine two atoms to form another). It is also automatically satisfied by anonymous social networks without any node information, i.e., COLLAB, RDT-B, RDT-M5K, IMDB-B in experiments. For other graphs with continuous node features such as word vectors, Lemma 3 provides a much weaker condition. Lemma 3 requires the linear independence of the entire node feature matrix for each graph in the dataset, which is more likely to hold in practice.
The Intrusion-
A.2 PROOFS ON GROUP AVERAGING
Proof of that (A), g, g¹) is invariant to the relative permutation between g and g' in the sense that (X, 9, g') (X, Tp 9. g') = $(). g, Tp g') for all P. P' Sn.
Proof. For all p E Sn, we have
(A,TTP ((.TT')..g'))
p(T og (1- A)g')
(9)
SSa PES,
1
(10)
S
PES.
Let p = P'1p', , we have P = P'p" and
1
(AA, "g.g')
1 -)g'')
(11)
SSal
p'p"ES,
1
pm pm (1 A)g')
(12)
Saal
p"EP Sn
1
Σ
((T 10g + +1 -)g')
(13)
|Sal
P"ES..
((A,g,9')
(14)
We can similarly prove (), = Φ(A, g, Tp o g').
Proposition 1. The contrastive loss with db AT og (1 (1- >)g') upper bounds the loss of an invariant network to relative permutation I... Σ Epes, p(o(ATTP gi + -x:)g()).
Proof. The loss of NT-Xent is:
1
exp(s(x:) x 221/T)
[((2)) (2))
log
(15)
N
Σ Σj= exp(s (z(2) (zi; 2/)/7)
N
N
N
1
(z)
log Σ J=
exp(s(z),,
(16)
N
1=1
6pox
where lma aims to minimize the distance between positive views.
Let {()1.9) g(), (22.92- 92 ..,, (An, IN g'y) }} be the batch original graph pairs and the mixup coefficients. If use MSE to minimize the distance between the feature interpolation view and
14

---

Under review as a conference paper at ICLR 2023
representation interpolation view and omit the temperature hyperparameter 5, &pss be written as:
1 (As Tr, 94, (.) p(Arb(gg) + (1 {=)o((()))
Epos
Σ
(17)
N
where P3 ~ Sn for all i E [M]. We have
N
Ep [ponl]
EpnSn
(lo(v(A; To 9h. ))) ((Act(s))) +1--)066))1
(18)
[lp(v)); Tp, o )..g)) (()i)(a))) + 1))0()))111 (19)
Sa
PIESn
p(o()i, Tp, o (..9)) (()o(())))))) + - (20)
PJE9.
where the last step is by Jensen's 's inequality.
Notice that, the contrastive loss upper bounds the distance between ((); + (1 ())(((())
and S. Ep,es. p(ab(A, Tp o (()) which is the group averaging of th By the prop- crty of group averaging (Puny et al., 2022; Yarotsky, 2022; Murphy et al., 2019), it follows that [8m] EpyEs, p(,,),, Tp, gi. g()) is invariant to relative permutation as well.
B RELATED WORKS
We have briefly introduced GCL methods in Section 2. connections and differences with E-SSL and IfMixup. Then, we present E-GCL's relations to other graph mixup methods and geometric deep learning.
In this section, we first discuss E-GCL's
E-SSL, E-GCL. is inspired by the pioneer R-SSI. works (Dangovski et al., 2022; Chuang et al., 2022) in CV and NL... Dangovski et al. (2022) find that, when the previous insensitive objective fail on certain augmentations, applying a sensitive objective to the same augmentations can improve performance. Specifically, they apply a sensitive objective on the four-fold rotations of images to improve existing SSL methods. In NLP, Chuang al. (2022) implement the sensitive objective as discriminating word replacement to improve sentence level embedding. Adapting E-SSL for graphs is challenging because existing intra-graph augmentations share the common paradigm of structure corruption, making it hard to categorize them into sensitive and insensitive augmentations. This works is different from previous E-SSL works that we introduce poos-graphh augmentation to create global semantic shifts. By encouraging the sensitivity to cross-graph augmentation, we protect representations from the harmful aggressive intra-graph augmentations.
IMMixup. In this work, we extend ItMixup (Guo & Mao, 2021) for cross-graph augmentation in SSL. IfMixup mitigates the structural differences by padding virtual nodes for graph interpolation. is previously developed for supervised learning. For SSL, we propose to supervise mixed graphs by the interpolation of the original graphs' representations. We also connect graph mixup to group theory and address its limitation of sensitivity to the relative permutation.
Graph Mixup. Graph mixup has been a challenging task due to graphs' irregular structures. GraphMix (Verma et al., 2021) sidesteps the structural differences by mixing only node features. Wang et al. (2021) mixup the graph representations for graph classification. G-Mixup (Han et al., 2022) samples adjacency matrices from the mixed graphons of two classes as graph mixup. We opt out G-Mixup in our method due to its following limitations:
G-mixup does not support node feature mixup. Their paper has no experiments on attributed graphs. Moreover, their instruction for sampling mixed node features is vague: the instruction does not describe the sampling strategy and the used distribution.
G-mixup does not scale to large graphs due to its ((N3) complexity (N is the number of nodes). The high complexity is due to the SVD (Chatterjee, 2015) in graphon estimation. In comparison, IfMixup is scalable to large graphs with linear complexity edge and node numbers O(F + N).
15

---

Under review as a conference paper at ICLR 2023
G-mixup requires class labeling to estimate the graphon in each class. In SSL, class labeling is unavailable. we were to take risks and treated each graph as a class, the obtained graphon would be suboptimal due to the limited sample. In this case, is outperformed by IfMixup (Table 3).
Geometric Deep Learning. Invariance and equivariance have been heavily studied under the scope of geometric deep learning (Bronstein et al., 2021). The goal is to explore geometric symmetries in neural architecture designs for effective weight sharing to reduce sample complexity (Cohen & Welling, 2016). For example, generalizing the convolution operation from the Z² grids to the p4 group makes convolution equivariant to the four-fold rotation (Cohen & Welling, 2016); the message passing operation in GNNs maintains the node-level output equivariant to the permutation group S., (Battaglia et al., 2018). Exploring geometric symmetries like rotation and permutation have greatly improved performances in various applications, including galaxy morphology (Dieleman et al., 2015), point clouds (Chen et al., 2021; Zaheer et al., 2017), and spherical images (Cohen et al., 2018). Our work is different from geometric deep learning because we do not study neural architectures. We study transformations that change the underlying semanties of graphs rather than their "poses" in the geometric space.
C LIMITATIONS
SSL is limited in that it has little knowledge of the downstream tasks. Each type of intra-graph augmentation represents human prior that performs differently on different downstream datasets (Pu- rushwalkam & Gupta, 2020). Our work grounds the equivariance mechanism as domain agnostic cross-graph augmentation to facilitate representations with the sensitivity global semantic shifts.
Previous E-SSL. works (Dangovski et al., 2022; Chuang et al., 2022) in CV and NLP divide existing data augmentations into two sets sensitive augmentations and insensitive augmentations. Our limitation is that we leave the existing intra- sirraggraph augmentations untouched as the insensitive aug- mentations, although the insensitivity to some aggressive intra-graph augmentations might diminish the sensitivity cross-graph augmentations. However, disentangling the aggressive augmentations from the others requires extensive tests or domain knowledge. We leave it as future work. Further, equivariance branch is patch to the limitations of the invariance branch. In future, will explore GCL of the complete focus on equivariance without the invariance branch.
Equivariance is high-level mathematical concept unifying sensitivity and insensitivity. It has promising potential in graph representation learning. Our work is limited that we explore the equivariance simple cross-graph augmentation of graph mixup. We believe there are other cquivariance principles worth exploring.
The limitations and assumptions of the theoretical results have been discussed in Appendix A.
D EXPERIMENT
D.1 ABLATION STUDIES FOR GRAPH INTERPOLATION
Table 3: Unsupervised learning accuracies (%) on the TU datasets.
Dataset
COLLAB
RDT-B
RDT-M5K
IMDB-B
AVG
Barlow
72.94
1
55.40aa.77
71.2840.58
72.40
Barlow Twins G-Mixup (discrete)
71.11at19
91.08an.54
15.90sa11
71.5840.31
72.42
Barlow Twins IfMixup (continuous)
74.63
90.71
56.90sa.29
72.02.0.90
73.57
Comparison with G-mixup. Our E-GCL, framework is agnostic to the graph mixup strategy for cross- graph augmentation. We also exploit G-Mixup (Han et al., 2022) for the cross-graph augmentation. Different from the linear interpolation strategy of IfMixup (Guo & Mao, 2021), G-Mixup yields discrete adjacency matrices by sampling from the mixed graphons of two classes. To adapt G-Mixup their source code and treat each graph class. Grid search is conducted
to the SSL setting, to tune the hyperparameters: a and w. Table shows the performance comparison between different graph mixup strategies. We do not compare performances attributed graph datasets because
16

---

Under review as a conference paper at ICLR 2023
Table 4: Average fine-tuning performances (ROC-AUC (%) in transfer learning downstream datasets.
Interpolation Position
GraphCL
Barlow
No Equivariance
70.8
72.0
Before Projector
71.9
73.8
After Projector
71.8
72.5
Similarity Score
71.5
Table 5: Unsupervised learning accuracies (%) of E-GCL on the TU datasets.
NCII
PROTEINS
DD
MUTAG
COLLAB
RDT-B
RDT-MM
IMDB-J B
AVG
E-GCL
79.95.a2s
78.18
77,83.a.
88. 88.20.a
74.63
90.71amm
36.90
72.02wox
76.93
+Mixup different sizes
78.54.14
74.86uez
a9.05.as
89.74..a
73.70um
90.74uv
56.17
72.30
76.89
G-Mixup does not support mixing node features. We have discussed the limitations of G-mixup in the related works section (Appendix B).
Our linear interpolation strategy substantially outperforms G-mixup in three out of four TU datasets. IfMixup also shows 1.15% better average accuracy than G-mixup. While G-mixup improves over the BarlowTwins baseline in the last three datasets, it performs worse in the COLLAB dataset.
Interpolation Position. The cross-graple augmentation is supervised by the interpolation of the original graphs' representations. We interpolate the representations before the projector to let the gradient mainly optimize the equivariance of the GNN encoder Φ. With this design, the encoder trained approach Equation (7), which is equivariance cross-graph augmentation. However, strict equivariance is hardly achieved by GNNs. Therefore, we have the projector learn to ignore the subtle difference between the two sides of Equation (7) and reduce the task's 's difficulty. Table compares the performances of not using equivariance and the performances of E-GCL when applying representation interpolation at different positions: before projector, after projector, and interpolating the cosine similarity scores. We verify that 1) before the projector is the optimal choice for both GraphCL and BarlowTwins, and 2) using equivariance at positions consistently outperforms No Equivariance. More than better performance, before projector also allows equivariance principle to work with a broader family of SSL frameworks (Zbontar et al., 2021; Bardes et al., 2022).
Influence of Mixing Dummy Nodes. We pad dummy nodes to the original graphs to the same size before mixup. As dummy nodes have zero features, adding them to original graphs does not introduce any noise into the interpolations. We demonstrate the neutral effect of dummy nodes by comparing the performances of mixing graphs with very different sizes and the original E-GCL in Table 5. The performance difference is only 0.04%.
We deliberately create size differences between mixed graphs in the different size experiment. Specifically, we sort the in-hatch graphs by their sizes (91, (0,,92,-.> ---a gN ), such that [g]| S [92| s S. [gv Then, we mix the i-th graph with the (N - + 1)-th graph. In this way, we create size differences in every batch such that the smallest graph 91 will be mixed with the largest graph gN. We use the BarlowTwrl Twins SSL framework for the experiments.
D.2 INFLUENCE OF SAMPLED PERMUTATION NUMBERS
Experimental Settings. the existing experiments (cf. Section 4), we randomly permute one the input graphs before graph interpolation. This strategy is a I-sample estimator of the group averaging and shows improvements over the state-of-the-ee baselines. Here we conduct ablation experiments to show that: 1) using random permutation improves performance; 2) using more samples to approximate group averaging improves performance, and 3) the improvement of using more samples is only marginal and comes at the cost of increased complexity. We conduct unsupervised learning experiments using the TU datasets. We employ E-GCL with BarlowTwins framework, set a = 0.1, and perform 5 experiments of different and max values of the mean accuracies (%) of these different experiments. The forward time is measured across 100 batches of size 128 on the COLLAB dataset.
{0.1, 0.3, 0.4, 0.5} values. We report the average
17

---

Under review as a conference paper at ICLR 2023
Table 6: Unsupervised learning performance in TU datasets. We ablate E-GCL using different numbers of samples to approximate group averaging. We report the average and max of mean accuracies for E-GCL with different w {0.1, 0.2, 0.3, 0.4, 0.5). We a 0.1.
No rand. perm.
I-sample
3-sample
10-sample
Average
76.65
76.70
76.73
76.80
Max
76.85
76,92
76.95
76.97
Forward time (ms)
10.5-24
10.7+2.
15.1
29.4.8.0
Table ACR with GraphCI. backbone. Lower is better
Before pre-training
After pre-training
GraphCL +Equivariance
0.463
0.983
0.424
Experimental Results. Table shows that using random permutation consistently outperforms using random permutation (No rand. perm.). Specifically, the I-sample estimator shows better performance and adds no computational cost compared to No rand. perm.. Further, it shows that using more samples estimate the group averaging improves performance. The 10-samlle estimator gives the best performance. However, the 10-sample estimator's 's improvement is only marginal (0.05% ~ 0.1 1%) compared the 1-sample estimator, but leads to almost three times increase in time complexity. Thus, we recommend the 1-sample estimator in implementation.
D.3 INFLUENCE OF EQUIVARIANCE ON AGGRESSIVE AUGMENTATIONS.
Intra-grap graph augmentations are problematic that they sometimes harmfully enforce insensitivity to semantically shifted graphs (i.e., aggressive augmentations). To patch the problem, cross-graph augmentations always enforce sensitivity to semantically shifted graphs that are generated by graph interpolation. Consequently, the equivariance cross-graph augmentations diminish the harmful invariance of aggressive intra-graph augmentations that change global semantics, leading to better performance.
To justify that equivariance can mitigate the negative effect of aggressive augmentations, we conduct experiments w.r.t Average Confusion Ratio (ACR) (Wang al., 2022). ACR the metric to measure the ratio that the anchor graph's 's nearest neighbors are the views of different graphs, rather than the other views of the same anchor. Higher ACR indicates that the graph representations are less powerful to distinguish different graphs, thus reflecting worse negative influences of aggressive augmentations. We use the GraphCL checkpoints from Table 2b and report the ACR scores on the ZINCI5 dataset.
As shown by Table 7, applying equivariance GraphCL. This demonstrates that the equivariance principle mitigates the negative influences of aggressive augmentations, thus leading to better graph discrimination performance.
rross-graph augmentation improves the ACR for
D.4 RESULTS THE FIRST SUBMISSION.
We include the results from our first submission for your reference (Table 8). We use Table 1 to replace Table 8 to report baseline performances under a consistent evaluation protocol.
E IMPLEMENTATION DETAILS
E. E-GCL PSEUDO CODE
Algorithm presents the pseudo code of E-GCL.
18

---

Under review as a conference paper at ICLR 2023
Table 8: Main experiment performances. denotes results borrowed from (Li al., 2022). * denotes reproduced results using the released codes. Other baseline results are borrowed from (You et al., 2021). Bold indicates the best performance and underline indicates the second best performance.
Unsupervised learning accuracies (%) on the TU datasets.
Dataset
NCTI
PROTEINS
DD
MUTAG
COLLAB
RDT-B
RDT-MSK
IMDB-B
AVG
GAIN
No train
65.40
72.73
75.67
87.39-
65.29
76.86 4035
48.48
69.3742.
70.15
graph2vec
73.22
13.15
75.
86
71.10sas4
InfoGraph
76.20
74. 74.4441
72.85
89.01-..
70.05+111
82 32.50-1
53.46 a
73.03-am
74.02
3.87
GraphCI.
77 77.87.04
34
71.36
29
44
75.71
JOAO
78.36aas
74.0741.0
77.40atar
87 $7.67.am
69.33ums
86.42.1.
56.03
70.83
75.01
4,86
ADGCL"
73.9 73.91amm
73.28ux4
75.79
18.74..15
72.02ance
90.07_aas
54,33ua222
70.21a:es
74.79
4.64
GraphMA
75.00aese
73.92am
76.15
87.17.mm
72.92
81.27-za1
19.6331
71.9622a
73.50
3.35
RGCL"
78.14aus
75.03um4:
78.86ama
$7.66_1m
70.92ame
90.34.a.s
36.38
75.854
76.15
6.00
E-GCL
77.93am4:
75.05ua.
79.07
88.70.az
73.84
91.59.as4
56.48.a33
72. 72.10
76.85
6.70
(b) Transfer learning ROC-AUC (%)
scores on the MoleculeNet. GTS denotes GraphTrans,
Dataset
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
AVG
GAIN
No Pre-train
65.8
74.0
63.4.us
57.3
58.044
71.8
7
70.1
67.0
Infomax
68. 68.8
75.3eas
62.7am4
58.4
69.922.
75.3
76.0aa
75.9at6
70.3
3.3
ContextPred
68.0z2o
75.7a02
63. 9ans
60.9acs
65. 921.
75.8
77.3ain
79.6a1
70.9
3.9
GraphCL
69.7au7
73. 73.9ea7
62.4ats
60.5
76.0s26
69. 1917
78.5
75.4
70.8
3.8
JOAO
71.4299
74.3e06
63.2ams
60.5
81. 31.0e..s
73.7st0
77.5at2
75,5
72.1
5.1
ADGCL
68.3.10
73.6
63.1
59.2
77.6-43
74.947a
75.4
75.0
70.9
3.9
GraphI .OGT
71.0-19
74.6
62.3-us
57.9414
78.7
75.0kxa
75.2
82.6
72.2
5.2
GruphMAE*
72.2an9
75.1204
63,0ana
58.5
80.5210
75.72t2
76,4208
81.3210
72.8
5.8
RGCL
71.4a07
75.2a00
61.4ans
61.4s0s
83.4
76.7
77.9w0
76.0ans
73.2
6.2
E-GCL
72.3.m6
74.9
64.0anz
62.8
83.
78. 8a0.
76.3ama
78.1
73.8
6.8
E-GCL, GTS
72.3aus
77.9tss
66,0ans
62,4sL.
80.7
79.4
77.8
79.7=24
74.5
7.5
E.2
IMPLEMENTATION
We implement GNNs with the PyTorch Geometric library (Fey Lenssen, 2019), which open- source under the MIT license. We conduct experiments using NVIDIA V100 GPU (32 GB memory) on a server with a 40-core Intel CPU.
Baselines. For comparison, we report the performances of the following baseline methods:
graph2vee (Narayanan et al., 2017) treats each graph as a document and employs a document embedding approach to learn graph embeddings.
Infomax (Veličković et al., 2019) maximizes the mutual information between the patch representa- tions that summarize the subgraphs centered around nodes and the global readout of graphs.
ContextPred (Hu et al., 2020) aims to predict the surrounding graph structures using the embed- dings of the local subgraphs. The prediction problem formulated as binary prediction with negative samples to resolve the intractability of predicting graph structures.
InfoGraph (Sun et al., 2020) learns representations by maximizing the mutual information between graph-level representations and graph substructures different scales, e.g., nodes and edges.
GraphCL. (You et al., 2020) explores the combination of intra-graph augmentations, such as node dropping, edge dropping, and subgraph, for contrastive graph representation learning.
AD-GCL (Suresh et al., 2021) trains an augmenter to adversarially drop edges remove redundant information.
GraphL.GG (Xu et al., 2021) utilizes clustering to contrast local instances and the corresponding hierarchical prototypes at every clustering layer.
JOAO (You et al., 2021) aims to automate the selection of graph augmentations via solving a bi-level optimization problem.
GraphMAE (Hou et al., 2022) explores GNN pretraining by reconstructing node features using masked autoencoder.
RGCL (Li et al., 2022) learns a rationale generator to protect the salient features during data augmentation.
19

---

Under review as a conference paper at ICLR 2023
Algorithm PyTorch-ttyle pseudocode for E-GCL
" phi: GNN encoder backbone
zho: MLP projector natwork
loss: loss function that encourages insensitivity positive views
alpha: shape parameter of beta distribution
omega: weight hyper-parameter of equivariance
for in loader:
intxa-graph augmentation
g1 augment (g)
augment (g)
# crosa-graph augmentation
lamb saadon.bata bataaaaaaaaaaaaaaaa alpha, size=len sample interpolation coefficients
perm random. (1en generate permutation graph 1ist
mix_g = miapgraphh_ list (gl, g2 [perm]] lamb) interpolate between gl[i] and g2 [p[ermii)]
invariance loss h1 phi (gl)
b2 =pphii (g2)
asl_ las__osssrriin (zhh rho
# equivariance loss (mix g) feature interpolation output
lamb h1 (1-lamb) h2 [perm]] # representation interpolation output eqv los_ = ssl loss (zho (th (h3), (h4))
and equivariance mechanism loss (1-omega) inv omega eqv_loss
cooperation
invariance
optimization step loss backward
optimizer.step
gra_h list listl, g lists, list):
# minimal graph mixup implementation without edge features mix g list = []]
list): pad node features and adjacency matrices to the aame size N max (size (gl), size(g2))
gl, g2,
in (ip listl, list2,
x1, pad (gi.xx, size=N), xgg2x size=N) shape (8, adj1, adj2 pad_ad) (g1. size-N), pad_adj ad2. size=N) shape (N, N)
node features and adjacency matrices xi (1-lamb) x2 shape (N, adj lamb * adjl (1-1amb) * adj2 # shape = (N, N)
mix
create mix_g Graph (mix mix_ad]
graph
mix list. append return mix g.xgg__. list
We do not compare with DGCL (Li et al., 2021) because their experiments follow different protocol. DGCL selects GNN layers, dimension sizes and batch sizes based on the test set performance on each dataset. However, other baselines and our methods stick to the same GNN configuration for all the datasets. Also, re-implementation hard because the source code has not been released.
Augmentations. Our intra-graph augmentation follows GraphCL (You et al., 2020). We use dropNode for the unsupervised learning experiments and use both dropNode and subgraph for the transfer learning experiments. For the gross-graph augmentation of graph mixup, include the self-loops of virtual nodes in the adjacency matrix due to better empirical performance. Before graph mixup, randomly shuffle the node order of of the input graphs to have random relative permutations between input graphs, which leads to slightly better empirical performance.
Implementation to Different SSL Frameworks. For the Barlow and SimSiam backbones, which negative samples, E-GCL follows strictly to their original loss functions. For the contrastive backbone GraphCL, E-GCL uses both intra- graph augmentations and cross-graph aug- mentations as negative samples to facilitate better cross-graph discrimination. Specifically, we have two types of embeddings for cross-graph augmentations: the feature interpolation embed- ))) and representation interpolation embeddings
dings {x:² x = p(o(Ag) + (1 )) To avoid overfitting to type of cross graph [z p()o(o!) (1 X)6(A22 N)N augmentations, half of each type the anchor graphs and the other half as the negative
20

---

Under review as a conference paper at ICLR 2023
Table Hyper-parameterr in unsupervised learning.
Backbones
learning rate
batch size
weight decay
epochts
W
Prujector dimensions
GraphCL
0.001
128
0
60
0.1
0.2
[32, 32)
BarlowTwins
0.001
128
0
60
0.1
0.4
[32, 128, 128, 128]
Table 10: Hyper-parameters in transfer learning.
Backbones
learning rale
butch size
weight decay
epuchs
W
Projector dimensions
GraphCLL
0,001
256
0
80
1
0.1
[300, 300, 300) 1300, 300, 300, 300]
SimSismn
0.0005
2048
0,00001
100
0.3
BarlowTwing
0,001
2048
0
100
2
0.5
[300,
1200, 1200, 1200]
BarlowTWins, Graphhlrans
0.0001
2048
0
100
2
0.3
1128, 1200, 1200. 1200)
Table 11: GNN configurations in transfer learning.
Model
Training Time
#Parameters
GNN Layers
Transformer Layers
GNN dim
Transformer dim Pooling
GINE
16.3 hours
1.85
300
Mean
GraphTrans
32,5 hours
2.73M
300
128
<CLS>
4-1(i5122) where 1()) is a binary indicator that (/222/122 and m² = 2, samples. Define m = 2
returns 1 when the condition holds and returns 0 otherwise. {m contains half of the feature interpolation embeddings and half of the representation interpolation embeddings; {m N contains the other half. The loss function when using GraphCL backbone be written as:
exp(s(z), xx()/=)
invariance: (((2)) (2)))
log
N 1--1 exp(a(x), 2/177) EE ( exp(a(z),
cquivariance:
(((m)) 2 (an)
exp(s(m, [/r]
log
N
>N 2) (mpss(m).mm)//7) E Je (xp(s(m],x))/)
CEOCL
( (2))))) ((m mm))),))
Notice that, we treat sample pairs that share partial graph identities (e.g., (m3. 2/) and (mf. m² as semi-positives and exclude them from negative samples.
Alignment and Uniformity Loss. We use dropNode the intra-graph augmentation and graph interpolation as the cross-graph augmentation. We pre-train GraphCL for 1000 steps before evaluation. The GraphCL setup follows that in Table 10. We split the original pre-training dataset into two subsets. The 80% subset is used for pre- pre-training, and 51200 samples from the other 20% subset are used for loss evaluation.
E.3 SOPEE-PARAMETERS
Unsupervised Learning. The hyper-parameters are shown in Table 9. We use the same three-layer GIN from (You al., 2020). Following Zbontar et al. (2021), the BarlowTwins backbone uses three- layer MLP projector with hidden dimensions that are four times the input dimension. Following You al. (2020), we use a learning rate 0.01, batch size 128 and no weight decay. For E-GCL, we conduct grid search for a and w in the following ranges We train the GNN for 60 epochs and evaluate the generated embedding using non-linear SVMs every 10 epochs. Following You al. (2020), search for the regularization parameter of SVMs in {0.001, 0.01, 0.1, 1, 10, 100, 1000}. Following (You et al., 2021), we report test accuracy of the epoch selected by validation set.
10. 0.4, 1, 4),
(0.1, 0.2, 0.3. 0.4, 0.5),
Transfer Learning. The Barlow Twins and SimSiam speed up pre- pre-training. The large batch size increases complexity because Barlowwins and SimSiam use no negative samples. Following Zbontar et al. (2021), the BarlowTwill Twins uses three-layer ML.P projector with hidden dimensions that four times the input
are shown in Table 10. We use large batch size 2048 for
21

---

Under review as a conference paper at ICLR 2023
Table 12: Hyperparameters for reproducing baselines. Parentheses include the original range of hyperparameters if they are different from our reproduction.
(a) Hyperparameters for unsupervised learning baselines.
GNN layers
I-dim
Max Epochs
Learning rale
Metric
Epoch selection
InfoGraph
(4.8.12
(100)
1e.3 (1e-2.1e33.Ie-4)
Accuracy
Validation set
GrapECLL
3
60 (20)
Ie-2
Accuracy
Validation set
10A0
3
32
60 (41)
le-3
Accuracy
Validarion set
ADGCL
(1e-2.5e-3.11e3)
Accuracy
Validation set
GrapMMBB
3 (2,355
32 (32,256,512)
60 (300)
1.5e4 (1 (1,54-45---11---
Accuracy (F1)
Validation set (Last spochh)
RGCI.
60
Ie-2
Accuracy
Validarion set
(b) Hyperparameters for transfer learning baselines.
GNN layers
h-dim
Prerzain epochs
Pretmin leamning rite
Fine-tunc epochs
Epoch selection
Infomax
300
100
lo-3
100
Last epoch (Validation sct)
ContextPred
300
le-3
100
epoch (Validation
ADGCI,
300
100 (20,50,80, 100)
le-3
100
Last epoch (Validation set)
GraphLLG
300
local globul
le:3
100
Last epoch
GrapliMAE
300
100
lo.3
100
Last spoch
RGCL
300
100
1o-3
100
Last spoch
Table 13: Statisties of unsupervised learning datasets.
Dataset
Category
#Graphs
#Avg. Node
#Avg. Edges
NCII
Biochemical Molecules
4110
29.87
64.6
PROTEINS
Biochemical Molecules
1113
39.06
145.63
DD
Biochemical Molecules
1178
284.32
1431.32
MUTAG
Biochemical Molecules
188
17.93
39.59
COLLAB
Social Networks
5000
74.49
4914.43
RDT-B
Social Networks
2000
429.63
995.51
RDT-M
Social Networks
4999
508.52
1189.75
IMDB-B
Social Networks
1000
19.77
193.06
Table 14: Statisties of biochemical graph datasets for transfer learning.
Datasets
Utilization
#Graphs
#Avg. Node
#Avg. Edges
ZINC-2M
Pre-training
2000000
26.62
57.72
BBBP
Fine-tuning
2039
24.06
51.91
Tox21
Fine-tuning
7831
18.57
38.59
ToxCast
Fine-tuning
8576
18.78
38.52
SIDER
Fine-tuning
1427
33.64
70.72
ClinTox
Fine-tuning
1477
26.15
55.77
MUV
Fine-tuning
93087
24.23
52.56
HIV
Fine-tuning
41127
25.51
54.94
BACE
Fine-tuning
1513
34.08
73.72
dimension. The SimSiam (Chen & He, 2021) uses two-layer MLP projector and a two-layer MLP predictor. Following Tian al. (2021), we let the predictor network use learning rate that is ten times that for the GNN encoder and the projector. The SimSiam backbone does not perform well with the default learning rate 0.001 and weight decay of Thus, we search for the SimSiam backbone's
learning rate and weight decay values in the following ranges: learning rate = {le 3, 5e 4). weight decay (1e 4, le 5, 5e - 5). We then fix the learning rate and weight decay and add the cquivariance mechanism. For E-GCL, we search for and from a random subset of the following ranges a = {0.1, 0.4, 1, 2, 4}, w = 10.1, 0.2, 0.3, 0.4, 0.5). We do not conduct a grid search because the dataset is large. For muningn the pre-traines trained model dataset. The fine-tuning learning rate is le - for GINE and le 4 for Graph Trans. Following (You et al., 2020), report test performance of the last epoch.
re-trained 100 epochs on the transferred
Table 11 shows the configuration details of our used GNNs, in which the GINE is from (Hu et al., 2020) and the Graph Trans
Trans is from the Molpcha experiment of (Wu et al., 2021).
22

---

Under review as a conference paper at ICLR 2023
Table 15: Complexity of E-GCL and representative GCL baselines. 0(X) O(2BL(NF2 +FF))
GraphCL
HarlowTwing
SimSiam
RGCL
E-GCL
GNN encoding
(
O(X)
O(X)
0(2X)
0(2x)
Loss function
F(
O(BF²)
O(BF)
F(2BPF) 0(2BF))
Table 16: Complexity of graph augmentations.
Drop Node
Drop Edge
Subgraph
G-mixup
Graph Interpolation (Ours)
O(N + B)
O(B)
O(11+DD)N E)
O(N*)
O(NF + EF)
Baseline Hyperparameters. We have re-evaluated some baselines to present consistent experi- mental comparison with E-GCL. In the re-evaluation, we report the test performance selected by validation set for unsupervised learning; When reproducing baselines, we change only the evaluation setting and leave other hyperparameters unchanged much as possible. Table 12 summarizes hyperparameter details. We now describe the process of selecting the baseline hyperparameters. For unsupervised learning, we use the validation set to make decisions when range of hyperparameters are provided in the original paper. We the same set of hyperparameters for all datasets, For transfer learning, we fine-tune the same pre-training checkpoint for all downstream datasets for fair comparison.
report the last epoch test performance for transfer learning.
E.4 DATASET STATISTICS
Table 13 and Table 14 present the statistics of our used datasets.
F COMPLEXITY ANALYSIS
The equivariant principle is computationally affordable. In this section, we analyze the complexity of GCL methods two parts: 1) neural computation, and 2) data augmentation.
Symbols. Formally, we define the symbols follows: B E Z is the batch size; N E Z is the number of nodes a graph; E Zt is the number of edges in a graph; L E Z is the number of GNN layers; & E (0, 1) is the ratio of the cutted subgraph for subgraph augmentation; D Zt is the maximum degree of nodes in graph; F = Z is the dimension of features nodes and edges. For graph interpolation of two graphs, let N, F,, and 1 refer to the values of the bigger graph.
Complexity of Neural Computation. We consider the complexity of GNN encoding and SSL loss function. Here, we analyze the complexity when using the GIN architecture. The E-GCL uses the Barlow Twins framework. Let O(X) = O(2BL(NF2 + EF)) be the complexity of encoding two batches of intra-graph augmentation graphs.
Shown by Table 15, E-GCL's complexity is comparable to RGCL. E-GCL's complexity is at most twice BaalowTwinn The additional O(X) complexity of GNN encoding comes from encoding the cross-graph augmentation batch, whose size is at most the sum of the two intra-graph augmentations batches. Similarly, E-GCL's ' complexity of loss function twice of that of BarlowTwins.
Complexity of Graph Augmentation. Table 16 shows the complexity of some popular graph augmentations (You al., 2020; Han et al., 2022). Note that the complexity of our Graph Interpolation is linear to the graph size times the feature dimension. It is lower than G-mixup and scalable to large graphs with thousands of nodes. Although the complexity of graph interpolation is higher than Drop Node and Drop Edge, it is scalable to large datasets. In practice, PyTorch dataloader with 4 multiprocessing workers can process graph interpolation of 2048 chemical molecules batches without putting GPU on wait.
23


Reviews:
[{"review_id": "mWyLnVVv9D3", "reviewer": "Reviewer_sATH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose an equivariant graph contrastive learning framework that adopts two principles: invariance to intra-graph\naugmentations and equivariance to cross-graph augmentations. They use Mixup as the method for cross-graph augmentation and argue that the cross-graph augmentation captures global semantic shifts and yield better performance.", "summary_of_the_review": "In general, I am not a fan of this paper mainly for the following two reasons: (1) Combining Mixup and contrastive learning is not new as several papers have adopted a similar idea though in the CV domain. (2) The experimental results are not exciting.", "strength_and_weaknesses": "Strength:\n1. Extensive results on multiple datasets.\nWeakness:\n1. Novelty: Though the authors make a great effort in explaining the concept of equivalence, the framework itself boils down to a combination of Mixup and contrastive learning. This combination is not new. For instance, at a high level, [1] and [2] both adopt the combination.\n2. The experimental results are not very convincing. In table 1(a), on the datasets that the proposed method performs the best, the increase is usually below 1% , not to mention that on the remaining 3 datasets, the proposed method underperforms. Similarly, in table 1(b), the proposed method only wins 4/8 datasets and by a small margin.\n\n\nLiterature:\n[1]i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning\n[2]Improving Contrastive Learning by Visualizing Feature Transformation", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "kxm_kftFNvp", "reviewer": "Reviewer_LCfW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a cross-graph augmentation method to simulate global semantic shifts. In particular, the paper first analyzes the limitations of invariant graph contrastive learning (I-GCL). Then the authors explore equivariance for cross-graph augmentation to mitigate the limitations of I-GCL. The authors conduct both unsupervised learning and transfer learning experiments the effectiveness of the proposed method.", "summary_of_the_review": "Overall, this paper takes one the most important research problems of graph learning: graph contrastive. The idea of exploring equivariance cross graph augmentation is interesting. However, I feel this paper is not ready as there are several major concerns regarding the framework design, experiment. Hopefully, the authors can address these concerns and make this paper more solid in the next submission.\n\n##\nScore updated after reading the rebuttal.", "strength_and_weaknesses": "Pros:\n1. The question studied is important and interesting.\n2. Using equivariance can help resolve the sensitive augmentation.\n\nCons:\n1. In Cross-graph augmentation, how do the authors generate labels for instances as\nlabels are unknown in contrastive learning. The authors claim that they mitigate structural differences between two graphs. However, randomly padding virtual nodes and edges may change the latent distribution of two graphs.\n2. In experimental settings, the details of evaluation methods and reported baselines performance are missing. E.g. how do the authors report the baselines performance? Are they reimplementing all baselines or they reporting performances from original papers? Which accuracy is refer to the reported accuracy? The last epoch accuracy or the test accuracy in terms of the best validation accuracy? It is really hard to reproduce the reported performance without these details.\n3. As far as I know, the reported results are from different evaluation methods. InfoMax and ContextPred are reporting the test accuracy of the best validation epoch, while GraphLOG is reporting the performance of the last epoch. The authors need to report results under a same experimental settings.\n4. In results tables, do the authors use a same backbone for all baselines? As authors mentioned, they are using a BarlowTwins as backbone for unsupervised task and employing a GraphTrans for transfer learning task. Do they also using these backbones for other baselines?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "akRl4M9-kR", "reviewer": "Reviewer_itQe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first points out that aggressive augmentations may worsen representation and lead to sub-optimal performance. To mitigate this issue, they propose a novel contrastive learning framework (E-GCL) that employs the equivariance principle to implement cross-graph discrimination. To this end, the authors propose insensitive transformations P and sensitive transformations H for the intra-graph augmentation and cross-graph augmentation, respectively. The author conducted detailed experiments to prove their framework’s effectiveness.", "summary_of_the_review": "This paper is well-written and deals with a non-trivial challenge. The experiments prove the model’s effectiveness. However, theoretical justification for the proposed design and discussion on some experiments are missing. The authors should address these issues. Overall, this paper is above the acceptance borderline if the authors can address all the concerns mentioned above.", "strength_and_weaknesses": "Strength 1: The contribution of this paper is clear; this paper is well written.\nStrength 2: This paper deals with a non-trivial and challenging issue which are neglected by current research. \nStrength 3: The experiment was performed on several datasets. The results show superior performance over existing competitive approaches.\n\nWeakness 1: In the introduction part, the authors point out that ‘Nonetheless, it is hard, without domain knowledge (Dangovski et al., 2022; Chuang et al., 2022) or extensive testing (Dangovski et al., 2022), to tell apart sensitive and insensitive augmentations.’  Indeed, it is an unresolved challenge to tell if a data augmentation will cause global semantic shifts. Learning the semantic shifted view can lead to sub-optimal performance. However, how the equivariant self-supervised learning can mitigate this issue? In the proposed framework, the intra-graph augmentation is kept. Hence, the semantic shifted view can still damage the model.\n\nWeakness 2: In Section 3.2, the author applied the ‘Group Averaging’ to ‘make the interpolations insensitive to relative permutations’. However, in the Section 3.1, the author claims that ‘Equivariance requires that…, any change applied to the graph should be faithfully reflected by the change of representation.’ This is not consistent. Can the author give more discussions on why the model should be insensitive to relative permutations during cross-graph augmentation? And why apply the ‘Group Averaging’ can help improve performance? There is no further experiment or discussion in Section 4 to back up this design.\n\nWeakness 3: In Section 4.3, the authors demonstrate the alignment and uniformity losses of I-GCL and E-GCL. It can be observed that the \"E-GCL with cross-aug\" setting has better alignment and uniformity losses. Can the author give some discussion on this finding? Does this imply if getting rid of the intra-aug, the model may have a better downstream task performance? If possible, the author should conduct ablation study on the \"E-GCL with cross-aug\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-0uxS3vlkB", "reviewer": "Reviewer_f85p", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, an equivariant graph contrastive learning method is proposed to perform cross-graph augmentation, which is reasonable and effective for GCL. Experiments are performed both unsupervised learning and transfer learning tasks to show the model effectiveness.", "summary_of_the_review": "Inspired by the equivariant SSL, this work proposes to enhance graph contrastive learning with cross-graph augmentation under an equivariant SSL framework. Experiments with different tasks show the advantage of the new approach.", "strength_and_weaknesses": "Pros:\n1.\tThe studied problem with equivariant graph contrastive learning is important and new for GCL domain.\n2.\tThe theoretical discussion of graph interpolation is given.\n3.\tComponent-wise model evaluation is provided to show the model effectiveness.\n4.\tHyperparameter sensitivity is conducted in the evaluation section.\n\nCons:\n\nIn the evaluation section, various baselines are compared, such as JOAO, GraphCL, and ADGCL. For the compared various GCL methods, the parameter setting details of those baselines can be described. \n\nAdditionally, the computational complexity of the new E-GCL method can be analyzed compared with representative GCL baselines. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject