Self-attentive Rationalization for Graph Contrastive Learning
Authors: Sihang Li
Yanchen Luo
An Zhang
Xiang Wang
Xiangnan He
Tat-Seng Chua
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Contrastive Learning
Self-supervised Learning
Transformer
Rationalization
self-attention

Abstract: Graph augmentation is the key component to reveal instance-discriminative features of a graph as its rationale in graph contrastive learning (GCL).
And existing rationale-aware augmentation mechanisms in GCL frameworks roughly fall into two categories and suffer from inherent limitations: (1) non-heuristic methods with the guidance of domain knowledge to preserve salient features, which require expensive expertise and lacks generality, or (2) heuristic augmentations with a co-trained auxiliary model to identify crucial substructures, which face not only the dilemma between system complexity and transformation diversity, but also the instability stemming from the co-training of two separated sub-models. 
Inspired by recent studies on transformers, we propose $\underline{S}$elf-attentive $\underline{R}$ationale guided $\underline{G}$raph $\underline{C}$ontrastive $\underline{L}$earning (SR-GCL), which integrates rationale finder and encoder together, leverages the self-attention values in transformer module as a natural guidance to delineate semantically informative substructures from both node- and edge-wise views, and contrasts on rationale-aware augmented pairs.
On real world biochemistry datasets, visualization results verify the effectiveness of self-attentive rationalization and the performance on downstream tasks demonstrates the state-of-the-art performance of SR-GCL for graph model pre-training. 

Under review as a conference paper at ICLR 2023
SELF-ATTENTIVE RATIONALIZATION FOR GRAPH CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Graph augmentation is the key component to reveal instance-discriminaaiee fea- tures of a graph as its rationale in graph contrastive learning (GCL). And existing rationale-aware augmentation mechanisms GCL frameworks roughly fall into two categories and suffer from inherent limitations: (1) non-heuristic methods with the guidance of domain knowledge to preserve salient features, which require expensive expertise and lack generality, or (2) heuristic augmentations with a co- trained auxiliary model identify crucial substructures, which face not only the dilemma between system complexity and transformation diversity, but also the in- stability stemming from the co-training of two separated sub-models. Inspired by recent studies on transformers, Contrastive Learning (SR-GCL), which integrates rationale generator and encoder together, leverages the self-attention values in transformer module a natural guidance to delineate semantically informative substructures from both node- and edge-wise perspectives, and contrasts on rationale-aware augmented pairs. On real-world biochemistry datasets, visualization results verify the effectiveness of self-attentive rationalization, and the performance on downstream tasks demon- strates the state-of of-the-art performance of SR-GCL for graph model pre-training.
propose Self-attentive Rationale guided Graph
INTRODUCTION
Graph augmentation is crucial enabler for graph contrastive learning (GCL) (You et al., 2020; Qiu al., 2020; Zhu et al., 2020). It pre-trains the model to yield instance-diseriminatve repre- sentations by contrasting augmented samples against each other, without hand-annotated labels. To achieve this goal, early studies (You et al., 2020; 2021; Qiu et al., 2020; 7hu et al., 2020) conduct random corruptions in topological structures (i.e., nodes and edges) attributes to construct con- trastive pairs. However, such random corruptions, especially on salient substructures, easily cause semantic gap between two augmented views of the same anchor graph, misguiding the following contrastive optimization procedure (Wang et al., 2021; Li et al., 2022).
To mitigate this, there has been recent interest in rationale discovery (Chang et al., 2020; Suresh et al., 2021; Li et al., 2022) as graph augmentation. We systematize these studies as rationale-aware augmentations, where a rationale exhibits graph's 's instance-discriminate others. The dominant paradigm often consists of two subsequent modules: the rationale discovery function and the rationale encoder, which aim creating the rationale-aware views and yielding their representations to contrast, respectively. To find rationales, early studies turn to domain knowl- edge to highlight the salient parts of graphs (Zhu et al., 2021; Liu al., 2022). For instance, Rong et al. (2020) leverage RDkit (Landrum, 2010), an assistant software of chemistry, to capture cru- cial functional groups with high activity molecule graphs. However, such expertise is expensive or even inaccessible in some scenarios (Tang et al., 2014). Besides, bringing in too much prior knowledge might harm generalization (Wang et al., 2022). To mitigate this problem, recent efforts (Suresh et al., 2021; Li et al., 2022) introduce auxiliary model instead to automatically identify rationales, which is named the rationale generator and co-train with the rationale encoder. In this ad-hoe scheme, however, reveal two inherent limitations:
information from the
Typically, the generator is made for one single transformation of graph data (Suresh et al., 2021; Li et al., 2022), forcing the focus on either node- or edge-wise rationales (e.g., Figures 1(b)

---

Under review as a conference paper at ICLR 2023
(a) Original graph (b) Node-wise views (c) Edge-wise views
(d) Dual views
Figure 1: Rationale-aware graph augmentation preserves
features in original
graph (e.g, g., (a)). Existing frameworks tailor-make an auxiliary model for one single transformation
(e.g., (b) (c)). SR-GCL constructs both node- and adge-wise rationale-aware views (e.g., (d)).
Integrated
Generator
Encoder
Model
(a) Separate generator encoder scheme
(b) Integrated scheme
Figure 2: Separate generator-encodee in (a) may lead to instability and biased representation, while SR-GCL integrates them together in (b) to pursue high-quality rationales and representations.
and I(c).. The lack of view diversity confines the rationale-aware augmentations to one trans- formation, while leaving the trosstrranfooonation untouched. Worse still, might degenerate the effectiveness of contrastive learning, as the studies (Chen et al., 2020; You et al., 2020) empiri- cally show that "no single transformation suffices learn good representations" Here we ascribe this crux of generator to the lack of transformation diversity, and argue that high-performing generator is supposed to be equipped with perspectives of both node and edge (e.g., Figure I(d)).
As illustrated in Figure 2(a), the generator, aiming at discovering rationales, separates from the subsequent encoder, which specializes in encoding them. While conceptually appealing, we hy- pothesize that these separate modules cooperate with each other to pursue high-quality rationales unsmoothly. Because the supervision signal for the generator remotely generated by the con- trastive optimization of the encoder, much of which is weak. Moreover, co-optimizing two sub- models could make the pre-training more complicated and time-consuming but less stable.
To resolve these limitations, we draw inspiration from the transformers (Vaswani et al., 2017) to reshape the generator-encoder scheme. Despite originally being proposed for language (Devlin et al.. 2019) and vision tasks (Dosovitskiy et al., 2021), transformers are attracting a surge of in- terest in graph area (Wu et al., 2021; Chen et al., 2022; Rampásek et al., 2022). At the core the self-attention operation, which models pairwise connections between tokens and yields high-quality representations. We find self-attention de facto a natural mechanism to concurrently discover and condense rationale information from both edge- and node-wise transformations. By prepending special token as its proxy and treating its nodes as other tokens, self-attention able to elegantly indicate the importance of each node and each edge (See Section 2.2 and Figure 4). Sampling nodes and edges based on the importance scores (i.e., heterogeneous transformations) allows us to generate both the node- and edge-wise subgraphs (i.e., rationales) simultaneously. Moreover, self-attention can directly output the rationale representations without additional modules. In stark contrast to the prior generator-encoder scheme, the "self-attentive rationalization" not only accomplishes diverse rationales in one shot, but also integrates the functions of rationale discovery and encoding together.
With the Self-attentive Rationalization as graph augmentation, we incorporate it into GCL and name the framework SR-GCL. Specifically, two augmented views stem from the node- and edge-wise ra- tionales. respectively; subsequently, the contrastive optimization pulls close representations of con- trastive pair augmented from the same anchor graph and pushes away those of different anchors by minimizing contrastive loss. Compared with conventional GCL methods, our SR-GCL collates and conflates the instance distrmiindier information across both node- and edge-wise transformations to construct rationale-aware contrastive pairs from dual perpectives. Henceforth, we find that such strategy improves the generalization performance of pre-trained model on downstream tasks, while simultaneously interpreting the contribution of each node/edge to instance- diiirn-ieeiiiiiitiionn.. Extensive experiments show that SR-GCL sets the new state-of oftte-of-thearr for graph pre- pre-training across a number of biochemical molecule and social network benchmark datasets in (Wu et al., 2018a; Morris et al., 2020). Codes available at https: : //tt://annnn /anonymouu 4open. sci ence/ / r / SR-GGI- EDD3.
2

---

Under review as a conference paper at ICLR 2023
Sample
p(ris)
R,t9l
Node-wise
Node-wien Rationale
Probability
Attention Wasaasaaa Scores
Contrastive Optimization
(2:5)
Anchor Graph
CBB
Sample
Raini
Edge-wiie
Edge-wise
Probsbility
Rationale
GNN
Transformer
Figure 3: SR-GCL framework, which constructs node- and edge-wise rationales (i.e., (R()) by sampling from corresponding probability distributions (i.e., (v) (P((C)) Then contrastive optimization is performed to encourage the agreement between views of the same anchor.
METHODOLOGY
Scrutinizing leading GCL methods (You et al., 2020; 2021; Qiu et al., 2020; Zhu et al., 2020; Suresh et al., 2021; Li et al., 2022), we can summarize the dominant scheme as combination of two subsequent modules: graph augmentation and contrastive optimization. During the pre-training phase, the cooperation these modules enables the backbone encoder to learn how to discover and parameterize the instance-discriminatie information for graph instances. Hereafter, the pre-trained encoder fine-tuned on downstream tasks. See Appendix A.1 for the literature review of GCL.
semantics when augmenting graphs has been founded critical recent studies (Suresh et al., 2021; Li et al., 2022). Here we systemize an augmented view holding such salient semantics rationale (Chang al., 2020; Li et al., 2022), which can help discriminate a graph instance from the other instances. However, these studies focus mainly on discovering homogeneous rationales from single perspective of either node or edge, thereby easily resulting in sub-optimal performance (Chen et al., 2020; You et al., 2020). To explore more effective and diverse rationales, we propose novel transformer-base pre-training framework, SR-GCI. As Figure 3 shows, it uses the graph transformer as the backhone encoder, whose inherent self-attention map allows us to discover heterogeneous rationales from dual perspectives of both node and edge; subsequently, upon the node- and edge-wise rationales, it conducts the contrastive optimization to pre-train the backbone. Appendix presents the detailed algorithm SR-GCL.
Preserving
2.1 BACKBONE MODEL: GRAPH TRANSFORMER
We begin by describing our backbone model, which is the cornerstone of SR-GCL. Distinguishing from the current GCL studies that employ "vanilla" graph neural networks (GNNs) (Thomas et al., 2022) as the backbone model being pre-trained and fine-tuned, (Wu et al., 2021; Rampásek et al., 2022; Chen et al., 2022). Specifically, adopts the transformer architecture (Vaswani al., 2017; Dosovitskiy et al., 2021) process the structural input of graph (See Appendix A.2 for the literature review). It consists of (1) GNN component, which creates node representations of a graph by propagating information between the locally adjacent nodes and then converts them into tokens; and (2) a transformer component, which refines the token represen- tations by applying the self-atter attention mechanism model the global connections between any two nodes. Next, we will elaborate on the graph transformer backbone.
use the graph transformer instead
Formally, let g (V,E) be a graph instance, where V and & are its node and edge sets, respectively. Taking g as input, this component first leverages vanilla GNN fi to yield the node representations X Vixd and then converts them into series of tokens Z E (IV)+))xd
GNN Component:
X X2,
Xayx fi(g), Z [X1, X2, (yyy
(1)
3

---

Under review as a conference paper at ICLR 2023
where N2 Rd is the d-dimensional representation of node V, and all node representations viewed as a sequence. It is worth highlighting that we append a special trainable (CLS) token to this sequence, which serves the graph-level representation for contrastive optimization in pre-training and classificatiin in downstream fine-tuning. That is, (CLS) can be viewed as a virtual supernode connected with all nodes, and expands V to V VU ((CLS)).
Transformer Component: On the tokens, we build a transformer f2 composed of L layers:
z 1 2 M Esss = [2(2,,
where 7- E (R((++))) collects the d-dimensional representations of tokens. Now, we dive into the core of f2 the self-attention mechanism that establishes the pairwise interactions across all tokens so as to enrich representations with the global information. Formally, for the pair of tokens v and u, the attention score d' in the (-th transformer layer can be obtained as follows:
x(z
WW
Wx 2
k(Z,, ,(x))1121_))) exp
22.u Emey. <(Z)
vd
where z!-1 is the representation of token u after (1 ( 1) layers, and z!, is set as Zu in Z at the initial
step; <(:-)) ) is the exponential kernel function with the dot product operation (--)); W Q / E Rdxd
are the trainable query/key projection matrices, Attending to all possible tokens, the representation of token at the 1-th layer can be updated as follows:
2 = Ed W Wyz
NEV-
where ai.,, indicates the contribution of token u V token u; W E Bd is the trainable value projection matrix. After I. layers, the representations are recursively updated to z Equation 2.
Why Graph Transformer? Most vanilla GNN-based backbones responsible solely to the repre- sentation learning, while leaving the graph augmentation step to an additional non non-parametric (e.g., corrupt graphs randomly or based on domain knowledge) or parametric g., generate rationales via an attention or masking network) module. Here we focus on the line of parametric rationale-aware augmentation. Clearly, this augmentation module is disjoint from these conventional backbones. Concretely, graph transformer backbone allows us to integrate the representation learning and ra- tionale generation together. Next, we will introduce simple yet effective rationalization strategy based on the self-attention mechanism of the transformer.
2.2
GRAPH AUGMENTATION SELF-ATTENTIVE R RATIONAL.IZATION
We now present our augmentation strategy - self-attentive rationalization, which exploits the self- attention mechanism to generate rationales from two perspectives of node and edge in one shot.
2.2. SELF-ATTENTION SCORES FOR RATIONALIZATION
10 12
For a graph instance, we reshape the self-attention values in the first transformer layer into a 2-D heat map as illus- trated in Figure 4, which enables to estimate the instance- discriminative power of each node and edge sdge-mod that is, the in- fluence of certain node edge on discriminating the graph instance from all the others. Specifically, for node u E V. its influence score Po is derived from the normalized attention score between it and the (CL.)) token; meanwhile, for edge (v. u) E E, the pairwise interaction between endpoints and u naturally depicts its importance Pe:
K(ZeLs; zu) Pv Sneeyk(zcss 2,r)' Pe Evee ((Zz,',,))) )
1e(Ze, Zu)
18
(5)
where 1o(:.) is the exponential kernel function defined in
Figure 4: Self-attention map.
Equation 3; k(Zeis: 2.) evaluates node v's attribution to the
4

---

Under review as a conference paper at ICLR 2023
graph-level representation, i.e., the (CLS) token; k(z,, Z.)) reflects the volume of information prop- agated on edge e = (v, u), which is further normalized over all edges e' = (v', u²) E E. With the contributions of each node and edge to fettane--discrminattee in hand, probabilistic sampling to generate both node- and edge-wise rationales for contrastive optimization.
further conduct
2.2.2 NODE- & EDGE-WISE WISE RATIONALE GENERATION
We first approximate the distribution of rationale-aware views, and then conduct probabilistic sam- pling to maintain the diversity of views. With a slight abuse of notation, uppercase G and R(G) separately represent a random variable of graph and its rationale, while lowercase g and R(g) are their samples correspondingly, Conditional probabilistic function and conditional probabilistic dis- tribution are denoted by p(-).) and P(-I-), respectively.
Distribution of Node- & Edge-wise Rationales. Given a graph y, the probability distribution of its node-wise rationale-awer aware view P1 approximated as:
Pi (R(()) R(g) G 9)
II p(ulg) DEVR
II (1 p(vl)),)) VEVIVR
(6)
where V and Va are the node sets of g and rationale R(g), respectively; p(ulg) denotes the proba- bility being included into R(g). which quantizes how semantically informative it is. Similarly, the edge-wise probability distribution P2 can be formulated as:
P2 = R(g)G = g)
II p(elg) BEER
II (1 ((elg)))) BEEVER
where and &R are the edge sets of g and R(g), respectively; p(elg) reflects the discriminative power of edge e. And p(v]g) (or p)elg) is parameterized by (or Pe) in Equation 5.
Sampling with Fixed Ratios. To construct anchor graph g's is node-wise rationale-aware view, we sample from Pi (: = g) under the constraint of a fixed node sampling ratio Pe:
(1(g) ~ P1(-IG = g) s.t. VRl = Pu -V).
After obtaining sampled nodes, the edges between them are kept to construct complete subgraph. Analogically, edge-wise rationale-aware view can be generated with another edge sampling ratio Pe:
R2(g) ~ P2(-IG = g) s.t. |ERl = Pe 18),
where the endpoints of sampled edges preserved, while the others discarded.
Without involving any auxiliary models, two rationale-awar views with different transformations (i.e., node- and edge-wise) generated under the guidance of self-attention rationalization.
2.3 RATIONALE REPRESENTATION LEARNING AND CONTRASTIVE OPTIMIZATION
With the rationale-aware views (i.e,, Rí (g)/R2(g)) generated by the the self-attentive rationalization process in hand, now we follow the pipeline of representation learning and contrastive optimization to empower the backbone graph transformer model with instance-diserimnnatee ability.
2.3.1 RATIONALE REPRESENTATION LEARNING
As illustrated in Figure 3, the GNN Si and transformer /2 sequentially process the rationale-aware views as in Equation and Equation 2 to yield their representations:
Z fe(fi (R)
ZÁ= (a(((()))))).
(10)
Subsequently, the (CLS) tokens in z and Z! which we denote as zf,cus and zL,cıs separately, are picked out as the graph-level representations of node- and edge-wise views (i.e., R₁ (g)/R(())
Then we follow the Commonly-adopted projection operation to project them into a latent space through a MLP-based projection head h with lo norm:
Ti = l(z
ra = h((z/.cLs)
The transformer module plays the roles of both the rationale generator, whose attention map guides rationale-awe aware augmentations of both node- and edge-wise views, and the encoder, who generates graph representations for subsequent contrastive optimization.
5

---

Under review as a conference paper at ICLR 2023
H
(a)
(b)
(c)
(d)
(c)
Figure 5: (a): A molecule graph. (b): Random attention. (c): Local attention of adjacency matrix.
(d): Global attention. (e): GraphBigBird, where white entry indicates absence of attention.
2.3.2 CONTRASTIVE OPTIMIZATION
The Info-NCE loss (van den Oord et al., 2018) adopted to maximize the mutual information between node- and edge-wise rationale-aware views of the same anchor graph:
exp (r r2/77)
min L = Egeg (g)), fr.f2.h
l(g) - log
(12)
exp (r1 T (2//) ErER exp (xp(r1rr/7)
where т denotes the temperature parameter, and are the rationale-aware views of the same anchor graph (i.e. positive views derived from Equation 11), and R' summarizes the representations of the other graphs' views in the minibatch (i.e., negative views).
The supervision signal guides the backbone model (i.e., both GNN and transformer module) to learn costance-discrmming transformer enhanced concurrently, thus bring about more accurate self-attentive rationalization.
representations, which indicates that the self-attention mechanism in
Finally, after pre-training, the projection head h will be thrown away. And it's worth mentioning that SR-GCL can be easily adapted to any transformer-based graph model with self-attention mechanism.
2.4 DISCUSSION ON TRANSFORMER COMPLEXITY: GRAPHBIGBIRD
Compared with conventional GNN model with time and space complexity of O(N) w.E. the number of nodes, the graph transformer with self-attention mechanism is empowered with powerful representation ability but quadratic complexity w.zt. the length of token sequence (i.e., the number of nodes) as well. As evaluated in (Wu et al., 2021; Rampásek et al., 2022), such quadratic complex- ity just adds acceptable computation overhead when processing graphs with only dozens of nodes. Nevertheless, in social network datasets, graph instances could hold thousands of nodes, Thus, the graph transformer, especially the self-attention mechanism would not scale well such large-scale graphs, due to the quadratic complexity of pairwise attention. To mitigate this, we get inspiration from the sparse attention mechanism BigBird (Zaheer et al., 2020) the NLP area and propose GraphBigBird tailor-made for the graph transformer.
GrgphBigBird. As illustrated Figure 5, GraphBigBird contains three parts: (I) random attention with hyperpatameter r controlling the sparsity, (2) local attention corresponding to the adjacency matrix of the graph sample, and (3) global attention connecting each token with the (CLS) token. The major difference between BigBird and GraphBigBird lies on the local attention part. Concretely, for a sequence of NLP tokens, BigBird adopts sliding window to capture the proximity between token and its neighboring tokens. GraphBigBird adapts this idea to graph by replacing the sliding window with the adjacency matrix, as Figure 5(c) shows. This simple strategy is effective to depict the local connectivity of graph.
With GraphBigBird, SR-GCL, requiring the graph transformer to be the backbone, is able to break the graph scale limitation of dozens of nodes and generalize domains containing large-scale graph data, to demonstrate its universal effectiveness.
3 EXPERIMENTS
In this section, extensive experiments are conducted on biochemical molecule and social network datasets with transfer and unsupervised learning settings demonstrate the state-of-the-art per-

---

Under review as a conference paper at ICLR 2023
Transfer learning. Test ROC-AUC scores (%) of pre-trained models on downstream datasets. G represents the commonly used GIN model in previous baselines, while T is the graph transformer model containing both GNN and transformer module depicted in this paper. Statistics of G models are from their original papers and those of T models are obtained by replanting the graph transformer model into their official released codes. red and blue indicate the best and the second best performance on each dataset, respectively.
Table 1:
Dutaset
BBBP
Tox21
ToxCast
SIDER
ClinTox
MUV
HIV
BACE
AVG.
GAIN
No Pre-Train
G
65.8 201
74.0
63.4 inn
57.3 116
58.0 144
71.8 ans
75.3 119
70.1 4M4
67.0
T
68. 124294
73,48
62.67am56
19,44
69
12
72.18
74.41
74,23:462
68.24
1.24
Infomax
G
68.8 ios
25.3 1es
62.7 i
58.4 1ea
69.9 1:10
75.3 a24
76.0 1n7
75.9 111
70.3
3.3
69. 15a06
74.62244s
61 85:775
60.14gam
74:78:2266
74.83
74.95 26
76.32=177
70.83
3.83
EdgePred
#24
76.0
64.1 das
60.4 =s7
64.1
#11
76.3
79.6 =12
70.3
69.32a124
76,78
64.43an73
60.15
73.61
12
75.87
78.46:1887
71.37
4.37
AurMasking
G
64.3 #28
76.7
64.2 405
61.0 #a?
71.8
74.7 #1A
77.2
79,3
71.1
4.1
T
66.42
77.22
64.892um
60.88
77:28
74.12
77.23
79.56.2.13
72.20
5.20
ContextPred
G
68.0 #20
75.7 =0.7
63.9 +0.
60.9 =06
65.9 +3.5
75,8 #1.
77.3 #1.0
79.6 =12
70.9
3.9
T
68.52a112
75.29
63.233au4
61.29
74.90
70.92
76.82
80.16:
71.39
4.39
GraphCT.
19s6810s7
73.87.00
62.40.1951
60.53gem
15.99
69.80.3m
78.47.11 15
70.77
3.77
70.412105
73.82
63.06a44:
60.38
77.79
75.62.00v 73.81
78.28:110
71.64
4.64
GraphLoG
72.5 ans
75.7
63.5 1n7
61.2
76.7
76.0
77.8
83.5
73.4
6.4
T
67 59
78.95 so
63 53.663
59.85s2.1
79.10
12
72.54a1 60
83,55
71.88
4.88
AD-GCL
G T
76.54_cxe
63.07,mm
63,28
79.78.1358
12.30_141
78.28.1997
72.72
5.72
67 88:
22.97
63. 49
60. 18:00
78.83 60
75.62a354
75.5421 29
13.37
70.95
3.95
70.22au0
74.98=t.9
62,94
59.97
81.32+229
71.66=1.3
76.73
77,34:044
71.90
4.90
JOAO
70.78an03
75.25 as
63.79ans1
61.04gax7
78.68:2297
75.67
76.98s1 16
17.49
72.46
5,46
RGCL
71.42sus
75,20=t.3
63.33
61.38sa<i
83.38
76.66
77.90sa0
76.03=0077
73.16
6.16
T
71. 21.36
75.72
63.93am3
60.91
s0.02
1..
77.81:005
:9.
73.20
6.20
SR-GCL odge'
T 71. 12
74.98=t.06
64.887ass
61.32s1.90
80,29 00.29:22.6
15.11=143
77.6-1i1 66
79.69
73.19
6.19
SR-GCL node! T 73.01
15.12
65.66
61.21
80.47
77.92
77.89:ass
:1.33::0oz
74.08
7.08
SR-GCL dual
T
72.60
77.30 ccae
15.861nss
61.46g13
81.84.1.50
1..7
78.52.077
74.62
7.62
SR-GCL with edgo-wise perspective, node-wise perspective and dual perspoctives of both node and edge.
formance of SR-GCL. Also, we visualize the rationales to illustrate the effectiveness of the self-
attentive rationalization mechanism. Finally, due to the limitation of space, the analysis for hyper- parameter sensitivity is provided in Appendix G.
3.1 TRANSFER LEARNING
Setup and baselines. Following the commonly adopted transfer learning settings and evaluation metrics in (Hu et al., 2020), the backbone model is pre-trained on large scale label-free dataset ZINC-2M (Sterling& & Irwin, 2015) and fine-tuned on downstream datasets with graph-level classi- fication tasks in MoleculeNet (Wu al., 201 (a) to evaluate the transferability of schemes. SR-GCL. is compared with competitive graph pre-training baselines, including Infomax (Velickovic et al., 2019). EdgePred (Hamilton et al., 2017), AttrMasking Hu et al. (2020), ContextPred (Hu et al., 2020), GraphCL (You et al., 2020), GraphLoG (Xu et al., 2021), AD-GCL (Suresh et al., 2021), JOAO (You al., 2021) and RGCL (Li et al., 2022) The statisties of datasets and briefs of baselines are presented in Appendix C and D, respectively.
Performance on downstream tasks. An MLP-based classifier appended after the pre-trained backbone model and they are subsequently fine-tuned together on downstream graph classification tasks. In Table 1, we present all statistics from their original papers with the backbone model of GIN (Xu et al., 2019), which is denoted as G. To make fair comparison, we also re-run all baselines with the same graph transformer model depicted this paper, which we denote as T. With one pre-trained model, on each downstream dataset, we run fine-tuning procedures 10 times with random seeds from 0 to to collect the test ROC-AUC of the epoch with the highest validation score, the mean and standard deviation statistics of which GraphBigBird is not adopted in this transfer learning experiment since molecule graphs only contain dozens of nodes. The detailed model structures and the computational efficiency comparison of GIN
showcased. Noted that the sparse attention mechanism
7

---

Under review as a conference paper at ICLR 2023
Table 2: Unsupervised learning. Test accuracies (%) on multiple biochemical and social network datasets. Statistics are from their original papers except GraphMAE. red and blue indicate the best and the second best performance on each dataset, respectively.
Dutaset
NCI1
PROTEINS
DD
MUTAG
COLLAB
RDTBB
RDT-M5K
IMDB-B
AVG.
GAIN
No Pre-Train
65 n7
72.73gm0
75.67
87 179
65.29_n
76.861875
48.48g
69.
70.15
graph2vec
:3.22:1.::
73.30
83.15
15
47.86::0s
21.10.um4
InfoGraph
76.20_.10
72.851177
89 01 1::3
70.05 1:33
82.50.1142
53.46.108
73.03 IIIT
74.02
3.87
GraphCL
77.87=041
74.39=245
78.62+0.40
86. 1.33
71.36=1.15
89.53+07-
95.99=022
71.14
75.71
5.56
IOAO
78. 36
14.07
77.40.1115
69.3 n4
86.42.11
56.03
25
75.01
AD-GCL
75. 2oaz
25.04::04s
75.73
74.89:100
92.
56.24
71.49:1u 11./
RGCL
78. 14. 10x
1540
78.86.1048
1.00
10.92..uas
56,33
_1.85_10
GraphMAE
76.12=144
74.69=824
77.15
87.21=031
73.16-245
87.76+1
54.98=1227
71.61
75.33
SR-GCI.
77.76.mm
75.38
78.70.8.7
17.95
76
91.12.10
72.06.077
77:03
7.03
I GraphBigBlrd adopted.
The evaluation metrics and data pre-processing are different from the other basclines, thus reproduced with the same settings others.
and graph transformer are included in Appendix E and F, The overall results are summarized in Table
1, and the following observations can be obtained:
SR-GCL outperforms existing pre-training baselines. On 8 downstream datasets, SR-GCL empowers the backbone graph transformer model with an average ROC-AUC score of 74.62% and performance gain of 6.38% over the same graph transformer model without pre-training. And compared with other competitive pre-training schemes, it achieves the best performance on out of 8 datasets and 3 second bests - the two leading positions on BACE both belong to GraphLoG, thus SR-GCL can be seen the second best pre-training scheme on this dataset. Such clear performance margin between SR-GCL and previous baselines strongly validates its effectiveness framework of self-supervised learning benefiting backbone's transferability.
Benefits of rationale-aware augmentations. Among multiple variants of GCL framework, GraphCL can be viewed as the vanilla one with random augmentations. With the shared mo- tivation of preserving instaneediiscrmminatve substructures rationales in anchor graphs, AD- GCL, RGCL and SR-GCL are proposed, but with different rationale discovery mechanisms. The consistent performance gain of them over GraphCL empirically demonstrates the crucial role of augmentation mechanism in revealing the rationales in anchor graphs.
Effectiveness of the combined design of rationale generator and encoder. RGCL leverages an external auxiliary model as the rationale generator to reveal semantically important nodes for graph augmentation. With the same graph transformer (T) as the backbone, the difference between RGCL(T) and SR-GCL with only node-wise perspective lies on the rationale genera- tor. Compared with the detached design in RGCL,, the self-attentive rationalization mechanism enables SR-Ge SR-GCL to combine the rationale generator and encoder. As shown in Table I, the per- formance gain of SR-GCL with only node-wise perspective, which 74.08%, over RGCL(T), which is 73.20%, verifies our claim that such combination benefits the model.
4. Advantage of diverse perspectives. Despite of the rationale aware augmentations, SR-GCL with only node-/edge-wist perspective has limited diversity of augmented views. The complete SR-GCL with dual perspectives outperforms them, thus verifying the advantage, which is also consistent with the observations in (Chen et al., 2020; You et al., 2020).
The experiments of transfer learning demonstrate the promising performance of SR- GCL as self-supervised GCL scheme. Concretely, the comparisons among GCL variants (e.g., GraphCL, AD-GCL and RGCL) and ablated structures of SR-GCL (e.g., SR-GCL with node- ledge-wise perspective and dual perspectives) verify the effectiveness of our major contributions: (1) combining the rationale generator and encoder and (2) collating and conflating the instance- discriminative information across node- and edge-wise transformations.
Summary.
3.2 UNSUPERVISED LEARNING
We further evaluate SR- GCL in the unsupervised learning settings (Sun et al., 2020), which cover both biochemical and social network datasets from TU datasets (Morris et al., 2020). Several leading
8

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
Figure 6: Self-attentive rationalization. (a): A molecule graph. (b): Corresponding attention map- (c): Node- (top) and edge-wise (bottom) attention, where darker color indicates higher score. (d): The contribution of each atom to Gasteiger partial charge (top) and partition coefficient (bottom).
pre-traing schemes are selected as the baselines, including graph2vec (Narayanan et al., 2017), In- foGraph (Sun et al., 2020), GraphCL (You et al., 2020), JOAO (You et al., 2021), AD-GCL (Suresh et al., 2021), RGCL (Li et al., 2022) and GraphMAE (Hou et al., 2022). The dataset statistics and baseline briefs are available at Appendix C and D, respectively.
GraphBigBird on large-scale graphs. Before presenting the performance comparison, we would like highlight the sparse attention mechanism GraphBigBird to reduce the computational complexity when applying the graph transformer for large-scale graphs. GraphBig as depicted in Section 2.4, is adopted three datasets containing graphs with thousands of nodes: DD, RDT-B and RDT-M. The detailed model structure and hyperparameter settings are included in Appendix E.
Performances on TU datasets. As shown in Table 2, SR-GCL achieves the best average accuracy score of 77.03%. Specifically, it achieves 2 bests and 2 second bests on 4 social network datasets: COLLAB, RDT-B, RDT-M5K and IMDB-B, verifying generalization ability of SR-GCL in various types of domain and the effectiveness of GraphBigBird largely reducing the computational com- plexity while maintaining powerful representation ability. The overall performance of SR-GCL in Table 2 demonstrates its promising performance on both biochemical and social network domains.
3.3 EFFECTIVENESS OF SELF-ATTENTIVE RATIONALIZATION
We pick out a molecule from pre-training dataset ZINC-2M at random for visualization. Figure 6(a) and 6(b) show the original graph and corresponding attention map generated by the pre-trained model, respectively. To be specific, the aforementioned parameterized probability value p(u:(g) and p(ega [g), where U, is the i-th node and Eja the edge between node and k in 6(a), correspond to entry ((CLS) i) and (j, k) in 6(b). To make more intelligible, the attention weights are mapped the molecule graph in 6(c), on top and bottom of which are node- and edge-wise view (i.e., p(ulg) and p(elg)), respectively. 6(d) gives the calculation of each atom to Gasteiger partial charge (top) and partition coefficient (bottom), which indicate the asymmetric distribution of electrons in chemical bonds and the hipophiliityyhydroohiiicii (red/blue shades) of the molecule, respectively. The shade of each atom reflects its impact on chemical properties instance-discriiinatie power. The comparison of 6(c) and 6(d) shows high precision of our proposed self-attentive rationalization, justifying its reliability. We also provide more visualization results in Appendix
CONCLUSION
In this paper, we draw inspiration from the self-attention mechanism in transformers and pro- pose a novel framework for transformer-basee graph models, Self-attentive Rationale guided Graph Contrastive Learning (SR-GCL),, which reshape the separated design of rationale generator and en- coder in existing rationale-aware GCL frameworks. In SR-GCL, the self-attention values in trans- former module are leveraged to guide the construction both node- and edge-wise rationale-aware contrastive pairs, making (1) the graph transformations diverse, and (2) the rationale generator nat- urally integrated with encoder. Visual inspections on real-world molecule datasets demonstrate the effectiveness of self-attentive rationalization proposed by and extensive experiments down- stream tasks show that SR-GCL empowers the backbone model with the ability yielding instance- discriminative representations, thus setting the new state-of-the-ar for graph pre-training.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shiyu Chang, Yang Zhang, Mo Yu, and Tommi S. Jaakkola. Invariant rationalization. ICML, volume 119 of Proceedings of Machine Learning Research, pp. 1448-1458. PMLR, 2020.
Dexiong Chen, Leslie O' Bray, and Karsten M. Borgwardt. Structure-aware transformer for graph representation learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pp- 3469-3489, PMLR. 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton, A simple framework for contrastive learning of visual representations. In ICML, volume 119 of Proceedings of Machine Learning Research, pp. 1597-1607. PMLR, 2020.
Jacob Devlin, Ming-Wer Chang, Kenton Lee, and Kristina Toutanova. BERT:
pre-training of deep
bidirectional transformers for language understanding. In NAACL-HLT (1), 4171-4186, As-
sociation for Computational Linguistics, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR. OpenReview.net, 2021.
Vijay Prakash Dwivedi and Xavier Bresson. A generalization of transformer networks to graphs. CoRR, abs/2012.09699 2020.
William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pp. 1024-1034, 2017.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Zhenyu Hou, Xiao Liu, Yukuo Cen, Yuxiao Dong, Hongxia Yang, Chunjie Wang, and Jie Tang. Graphmae: Self-supervised masked graph autoencoders. In KDD, pp. 594-604, ACM, 2022.
Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. Strategies for pre-training graph neural networks. In ICLR, 2020.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumble-softmax. In International Conference on Learning Representations (ICLR 2017). OpenReview. net, 2017.
Devin Kreuzer, Dominique Beaini, William L. Hamilton, Vincent Létourneau, and Prudencio Tossou. Rethinking graph transformers with spectral attention. In NewrlPS, pp. 21618-21629, 2021.
Greg Landrum. Rdkit, open-source cheminformatics software, 2010. URL https://www rdkit.org/
Sihang Li, Xiang Wang, An Zhang, Yingxin Wu, Xiangnan He, and Tat-Seng Chua. Let invariant rationale discovery inspire graph contrastive learning. In ICML, volume 162 of Proceedings of Machine Learning Research, pp. 13052-13065. PMLR, 2022.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre-
training molecular graph representation with 3d geometry. In ICLR. OpenReview.net, 2022.
Grégoire Mialon, Dexiong Chen, Margot Selosse, and Julien Mairal. Graphit: Encoding graph structure in transformers. CoRR, abs/2106.05667, 2021.
Christopher Morris, Nils M. Kriege, Franka Bause, Kristian Kersting, Petra Mutzel, and Marion Neumann. Tudataset: A collection of benchmark datasets for learning with graphs. CoRR, abs/2007.08663, 2020.
Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. CoRR, abs/1707.05005 2017.
10

---

Under review as a conference paper at ICLR 2023
Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. GCC: graph contrastive coding for graph neural network pre-training. In KDD, pp. 1150-1160. ACM, 2020.
Ladislav Rampásek, Mikhail Galkin, Vijay Prakash Dwivedi, Anh Tuan Luu, Guy Wolf, and Dominique Beaini. Recipe for a general, powerful, scalable graph transformer. CoRR, 12244, 2022.
Yu Rong. Yatao Bian, Tingyang Xu, Weiyang Xie, Ying Wei, Wenbing Huang, and Junzhou Huang. Self-supervised graph transformer on large-scale molecular data. NeurlPS, 2020.
Teague Sterling and John J Irwin. Zinc 15 ligand discovery for everyone. Journal of chemical information and modeling, 55(1 1):2222-23331 2015.
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi- supervised graph-level representation learning via mutual information maximization. In ICLR, 2020.
Susheel Suresh, Pan Li, Cong Hao, and Jennifer Neville. Adversarial graph augmentation to improve graph contrastive learning. NeurlPS, pp. 15920-15933 2021.
Jiliang Tang, Salem Alclyani, and Huan Liu. Feature selection for classification: A review. Data classification: Algorithmss and applications, pp. 37, 2014.
Josephine M. Thomas, Alice Shaallmy--urehh Silvia Beddar- Wietaing, and Clara Holzhüter. Graph neural networks designed for different graph types: A survey. CoRR, abs/2204.03080, 2022.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. CoRR, abs/1 807 00774,478 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NIPS, pp. 5998-6008, 2017.
Petar Velickovie, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and Devon Hjelm. Deep graph infomax. In ICLR (Poster). OpenReview.net, 2019.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In CVPR, pp. 2495- 2504. Computer Vision Foundation IEEE, 2021.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In International Conference on Learning Representations, 2021.
Zhanghao Wu, Paras Jain, Matthew A. Wright, Azalia Mirhoseini, Joseph E. Gonzalez, and Ion Stoica. Representing longerange context for graph neural networks with global attention. In 2021.
NeurIPS, pp.
Zhenqin WU, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh Pappu, Karl Leswing, and Vijay Pande. Moleculenet: benchmark for molecular machine learn- ing. Chemical science, 9(2):513-530) 2018a.
Zhirong Wu, Yuanjun Xiong. Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, pp. 3733-3742. Computer Vision Foundation / IEEE Computer Society, 2018b.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful are graph neural networks? In ICLR, 2019.
Minghao Xu, Hang Wang, Bingbing Ni. Hongyu Guo, and Jian Tang. Self-supervised geaph-level representation learning with local and global structure. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 11548 1558. PMLR, 2021.

---

Under review as a conference paper at ICLR 2023
Xinyi Xu, Cheng Deng, Yaochen Xie, and Shuiwang Ji. Group contrastive self-supervised learning on graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tin-Yan Liu. Do transformers really perform badly for graph representation? In NeurIPS, pp. 28877-28888, 2021.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph 2020.
contrastive learning with augmentations. NeurlPS,
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning auto- mated. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 12121-12132. PMLR, 2021.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Trans- formers for longer sequences. In NeurIPS, 2020.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. CoRR, abs/2006.04131, 2020.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In WWW, pp. 2069-2060 2021.
12

---

Under review as a conference paper at ICLR 2023
A RELATED WORK
A.1 GRAPH CONTRASTIVE LEARNING
As prevalent line of self-supervised learning (SSL) (Chen al., 2020; He al., 2022), GCL has been a rising topic among researchers in the graph area. It pre-trains the backbone model to yield dintimi-isrri representations by contrasting augmented views of graph samples against each other in large label-free datasets, thus benefiting the downstream supervised fine-tuning, The typical paradigm of existing GCL frameworks consists of two modules: (1) graph data augmentation and (2) contrastive optimization.
A.1.1 GRAPH AUGMENTATION
Graph augmentation transforms graph in two correlated but different views. Early studies (You et al., 2020; 2021; Qiu et al., 2020; Zhu et al., 2020) instantiate it by random mechanisms, which cor- rupt the graph structures or attributes with uniform probability. For example, GraphCL (You al., 2020) constructs augmented views with strategies of node dropping, edge perturbation, or attribute masking a random fashion. JOAO (You et al., 2021) adaptively selects graph augmentation strate- gies, but all alternatives are still confined in random mechanisms. GCC (Qiu et al., 2020) augments the r-ego network of nodes by random walks with restarts. And GRACE (Zhu et al., 2020) performs random edge removal and node feature masking to generate diverse views. However, overlooking the importance of keeping saliency properties during augmentation may result in potential loss of semantic information, consequently misleading following contrastive optimization and undermining the goal of
Rationale-guided Augmentation. To tackle this problem, various methods (Zhu al., 2021; Liu et al., 2022) have been proposed to preserve salient semantics in augmented views, which we term rationale-guided augmentation. A research line leverages external domain knowledge to bridge the potential semantic gap between augmented views. For instance, in the social network domain, GCA (Zhu et al., 2021) proposes to utilize node centrality measures to capture crucial connective structures; in the biochemical molecule domain, GraphMVP (Liu et al., 2022) conducts contrast between 2D and 3D conformation structure of molecules. Still, expertise could be expensive even inaccessible in some scenarios (Tang et al., 2014), making their application very limited and the performances of generalizing to unseen domains are easily damaged by these over-specific guidance (Wang et al., 2022). Aforementioned limitations call for rationale-aware but automatic augmentation mechanisms to get rid of the domain knowledge.
Rationale Generator. More recently, GCL frameworks (Suresh et al., 2021; Li et al., 2022; Xu al., 2022) adopt auxiliary models, which named rationale generator, to conduct heuristic ra- tionale discovery. Specifically, AD-GCL (Suresh et al., 2021) hires another graph model as the rationale generator to identify salient edges for edge-wise views. RGCL (Li et al., 2022) captures semantically informative substructures to create node-wise views with GNN-based rationale gen- crator. However, the limited perspective of only node or edge confines the performance of the pre- trained model to sub-optimal position, which is consistent with the observations in (Chen al., 2020; You et al., 2020) "no single transformation suffices to learn good representations". To im- prove the diversity of views, a naive solution is to adopt multiple auxiliary generators, each of which specialises one perspective. Nonetheless, you cannot have your cake and eat it - by this way, more diversity on rationale-awan transformations demands more system complexity (i.e., more auxiliary generators).
Rationale Encoder. Apart from the limited perspectives, existing rationale-aware GCL frameworks separate the rationale generator from the encoder (i.e., the backbone model being pre-trained and fine-tuned). The back-propagatie
of gradients to the encoder is natural when optimizing the con- trastive loss of representations yielded by it. Nevertheless, to enable the propagation to the genera- tor, AD-GCL (Suresh et al., 2021) leverages the Gumbel-Max reparameterization, which may lead to instability stemming from large variance of gradients (Jang et al., 2017). RGCL (Li et al., 2022) mixes the outputs of the encoder and generator to yield the graph representations subsequently be- ing contrasted, resulting in the biased output of the encoder the generator will be discarded after pre- pre-trainn training. Here, we blame the separation of the generator and encoder for the aforementioned instability and biased geaph-level representations.
13

---

Under review as a conference paper at ICLR 2023
A.1.2 CONTRASTIVE OPTIMIZATION
With contrastive pairs in hand, GCL performs instance-discrminaate discrimination task by optimizing contrastive loss, which can be instantiated as NCE (Wu al., 2018b), InfoNCE (van den Oord al., 2018) or NT-Xent (Chen et al., 2020). During this, the agreement between two augmented views of the same anchor graph is encouraged, while the divergence between those different anchors is enforced.
A.2 GRAPH TRANSFORMER
Transformer has achieved remarkable success various domains (Vaswani et al., 2017; Devlin et al., 2019; Dosovitskiy et al., 2021). Researchers also apply it in graph area (Dwivedi & Bresson, 2020; Kreuzer et al., 2021; Mialon et al., 2021; Ying et al., 2021; Wu et al., 2021; Chen et al., 2022; Rampásek et al., 2022), aiming take advantages of self-attention mechanism capture global relations and avoid over-smoothing during the graph representation learning. However, graph is not naturally in agreement with the input format of transformers, but data structure with topology and high regularity. Thus, early studies transform a graph into sequence of nodes and only implicitly incorporate topological information by adding absolute (Dwivedi & Bresson, 2020; Kreuzer al., 2021), relative (Mialon et al., 2021) or structural (Ying et al., 2021) encoding.
More recently, hybrid structures have been proposed to integrate GNN and transformer. In Graph- Trans (Wu et al., 2021), the GNN module first learns local structure to generate token features, and then the transformer module focuses on capturing global relationships. SAT (Chen al., 2022) leverages a GNN to encode structure information, while transformer captures interactions between nodes. More recently, GPS (Rampásek et al., 2022) provides a comprehensive recipe for hybrid models, including transformer structures and encoding methods.
B ALGORITHM OF SR-GCL
Algorithm SR-GCL algorithm
Initialize: dataset {gm m 1, 2, .... M). GNN module fi(), transformer module f2()),
projector h(()), sampling ratio Pe & pe and temperature
2: for all sampled minibatch of data {gn : 1,2, 1...... N) do
3: for 1 to N do ((
4:
Equation
KiZCTS, Zo)
5:
(v(vlgm) EvEv Ki(CLSs Zue)
:0 V/
Equation
K(Zu, Znr.)
6:
(p(elga) = Edee KilZu, 1 k
:6 EE
Equation
7:
Ro(g) ~ Po(.IG = -gn))
S.l. VRl = Pe |V|
Equation & Equation
8:
Re(g) ~ Pe(-GG= = 9n)
s.t. |Ell = Pe 1gl
9:
ra
r = (((a((
10:
end for
11:
for
= N do
12:
R, = [r/: rf : 1, 2,..,,
1, 1,.,, N,....N}
exp r/ T
13:
1 log
P Equation 12
exp (r' ( + Er ER
exp
r/ 1//T
14: end for
15: Update f(-), r()), and h(-) to minimize
Σ E
16: end for
17: return Encoder fa
14

---

Under review as a conference paper at ICLR 2023
C DATASETS
Table 3: Statistics for ZINC-2M and MoleculeNet datasets.
DATASETS
CATEGORY
UTILIZATION
GRAPHS#
AVG.
AVG.
ZINC-2M
BIOCHEMICAL MOLECULES
PRE-TRAINING
2,000,000
26.62
57.72
BBBP
BIOCHEMICAL MOLECULES
FINETUNING
2,039
24.06
51,90
TOX21
BIOCHEMICAL MOLECULES
FINETUNING
7,831
18.57
38.58
TOXCAST
BIOCHEMICAL MOLECULES
FINETUNING
8,576
18.78
38.52
SIDER
BIOCHEMICAL MOLECULES
FINETUNING
1,427
33.64
70.71
CLINTOX
BIOCHEMICAL MOLECULES
FINETUNING
1,477
26.15
55.76
MUV
BIOCHEMICAL MOLECULES
FINETUNING
93,087
24.23
52.55
HIV
BIOCHEMICAL MOLECULES
FINETUNING
41,127
25.51
54.93
BACE
BIOCHEMICAL MOLECULES
FINETUNING
1,513
34.08
73.71
Table 4: Statistics for unsupervised learning TU-datasets.
DATASETS
CATEGORY
GRAPHS#
AvG. N#
AVG. E#
NCII
BIOCHEMICAL MOLECULES
4,110
29.87
32.30
PROTEINS
BIOCHEMICAL MOLECULES
1,113
39.06
72.82
DD
BIOCHEMICAL MOLECULES
1,178
284.32
715.66
MUTAG
BIOCHEMICAL MOLECULES
188
17.93
19.79
COLLAB
SOCIAL NETWORKS
5,000
74.49
2457.78
RDT-B
SOCIAL NETWORKS
2,000
429.63
497.75
RDT-M
SOCIAL NETWORKS
4,999
508.52
594.87
IMDB-B B
SOCIAL NETWORKS
1,000
19.77
96.53
D BASELINES
We compare with following basclines to verify the state-of-the-ar performance of our SR-GCL:
Infomax (Velickovic al., 2019) Infomax conducts mutual information maximization between patches (i.e., subgraphs centered around nodes of interest) and high-level graphs.
EdgePred (Hamilton al., 2017) EdgePred pre-trains a backbone model by generating node representations of previously unseen data with text attributes to make edge predictions.
Attribute Masking (Hu et al., 2020) Attribute Masking masks attributes of node/edge and make backbone predict them based on their neighboring structure.
Context Prediction (Hu et al., 2020) Context Prediction trains the backbone to map nodes ap- pearing in similar contexts to nearby representations by predicting with subgraphs.
GraphCL (You al., 2020) GraphCL is vanilla GCL method, constructing augmented views with strategies of node dropping, edge perturbation or attribute masking in random fashion.
GraphLoG (Xu et al., 2021) GraphLoG preserves the local similarity and leverages prototypes to capture the clusters with global semantics.
AD-GCL (Suresh al., 2021) With GNN-based augmenter, AD-GCL adopts adversarial graph augmentation strategies to minimize redundant information GCL.
JOAO (You et al., 2021) JOAO automatically selects data augmentations from strategy pool to construct contrastive pairs for GCL.
RGCL (Li et al., 2022) RGCL hires an auxiliary model reveal semantically important nodes to construct augmented views for contrastive optimization.
graph2vec (Narayanan et al., 2017) Graph2vec views the graph a document and the rooted sub- graphs around every node as words that compose the document. Then the document embedding neural networks can be adopted to learn representations of graphs
Infograph (Sun et al., 2020) Infograph maximizes the mutual information between the graph-level representation and the representations of substructures.
15

---

Under review as a conference paper at ICLR 2023
GraphMAE (Hou et al., 2022) GraphMAE is masked graph autoencoding framework aiming masked feature reconstruction.
E BACKBONE MODEL ARCHITECTURES
Table Backhone model architectures and hyperparameters
MODELS
GIN (T)
GRAPH TRANS (T)
GRAPH TRANS (U)
GNN TYPE
GIN
GIN
GIN
NEURON#
(300.300,300.300.300)
1300,300,300,300330
[32,32,32]
TRANS IN/OUT DIM
128
32
TRANS PROJ DIM
512
64
TRANS LAYER#
4
TRANS HEADS#
POOLING
MEAN [300,300]
(CI.S)
(CLS)
PROJECTOR NEURON#
[128.128]
[32.32]
OPTIMIZER
ADAM
ADAMW
ADAMW
LEARNING RATE
0.001
0.0001
0.0001
(T) means that the model applied in transfer learning experiments, while (U) indicates unsuper- vised learning experiments. For hyperparameter sensitivity analysis in transfer learning, please refer to Figure 7. And we only adopt GraphBigBird on three datasets containing graphs of thousands of nodes: DD, RDT-B and RDT-M and the default value for sparsity is set 0.01.
F COMPUTATIONAL EFFICIENCY
The training time mainly depends on the type of backbone model. To compare the computational efficiency of commonly adopted GIN model in existing baselines with the graph transformer in our paper, the architectures of which shown in Table 5, we evaluate the number of parameters and the propagation runtime in transfer learning for each mini-batch of 256 samples on NVIDIA GeForce RTX 3090. The statistics in Table 6 show that the transformer module adds overhead of 46.63% to whole contrastive learning procedure and 49.91% % to inference (i.e., forward) time, which we argue, considering the great improvement of representation ability and interpretability, is acceptable.
Table 6: Computational efficiency of backbone models
Backbone
Forward (ms)
Backward (ms)
Cumulative (ms)
Parameter #
GIN
15.55
22.49
38.04.10.07
1.88M
Graph Transformer
23.97
36.51
60.47
2.71M
Meanwhile, we conduct the time complexity comparison between SR-GCL and the vanilla GCL- GraphCL. with the same graph transformer as the backbone model, which is shown as follows:
Table 7: Time complexity comparison between SR-GCL and GraphCL
FRAMEWORKS
GRAPHCL
SR-GCI
DATA AUG.
O(2BplV) log
O V log ea 16 log (8()
RATIONALE
O(B((F] (V]) 4 [V]a4 (V]
BACKBONE FORWARD
O( + LuN (IV)'+ Vid Lgr))
O(B((2)]] + LoNN 2v Ler)
Ler))
CONTRASTIVE Loss
O(B2d)
O(B2d)
where the average numbers of nodes and edges per graph in ZINC-2M datasets are | and [E], re- spectively, B denotes the batch size, denotes the sampling ratio, L denotes the number of layers in backbone encoder, and denote the latent space dimension where contrastive loss is calculated. On single NVIDIA GeForce RTX 3090, the overall training time of GraphCI. and SR-GCL. with 100
16

---

Under review as a conference paper at ICLR 2023
epochs and the same backbone model of graph transformer 25.61 and 30.56 hours, respectively. We argue that the 19.33% training time overhead brought by self-attention rationalization process acceptable, considering the performance gain of SR-GCL compared with GraphCL.
G HYPERPARAMETER SENSITIVITY
MoleculeNet
MoleculeNet
Averagg ROC-AV C-ACC (%)
75
75
Average ROC-AN
73
73
72
71
70
70
0.15
0.2
0.35
0.4
65
(a)
(b) p (%)
Figure 7: Sensitivity w.r.t hyperparameters and
The average performances of SR-GCL for transfer learning 8 downstream MoleculeNet datasets W.E.. hyperparameters т and p are presented in Figure 7. The sharp pike in 7(a) indicates the close relationship between SR-GCL's performance and hyperparameters (Wang & Liu, 2021). Thus, suggest to tune it carefully around 0.1. As for hyperparameter controlling the scale of augmented views, we P. Pe p make the scales of each contrastive pair close. The results show that SR-GCL achieves the best performance when is set to large value around 80%, which is consistent with Li al. (2022).
17

---

Under review as a conference paper at ICLR 2023
H SAMPLES OF SELF-ATTENTIVE RATIONALIZATION
(a)
(b) Cecccc((200C-CC02)(()))
(c) CNCCeIJCOOJocc(Feccc.P
(d) (is-ll00))xCCclcccccc/B100000
(e) co0
(I) COcIce(rr)e(Breec0
Figure 8: Visualization of more molecules
18


Reviews:
[{"review_id": "tcwiF4ygthp", "reviewer": "Reviewer_KsiL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a simple and effective  Self-attentive Rationale guided Graph Contrastive Learning  for graph network learning. The code idea is to learn both node- and edge-wise rationale-aware views. Extensive experimental results show the effectiveness of the proposed method. ", "summary_of_the_review": "The proposed solution for graph contrastive learning is simple and clear. The experimental results are sufficient. ", "strength_and_weaknesses": "Strength:\n1. The writing is good.  It is easy to understand the main idea.\n2.  The experimental analysis is sufficient and in-depth.\n\nWeakness:\n1. More theoretical analysis about the effectiveness of  SR-GCL should be clarified.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lfg5c6crQNM", "reviewer": "Reviewer_PbQf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose a self-attentive rationalization for rationale-aware agmented pairs, which are used in contrastive learning. The efficiency of self-attentive rationalisation is validated by visualisation on real-world biochemistry datasets, and the performance on downstream tasks shows the good performance of SR-GCL for graph model pre-training.", "summary_of_the_review": "Based on my comments, I am inclined to give a rejection right now. But I am happy to refer to other reviewers' comments and the author's response before making a decision. ", "strength_and_weaknesses": "Strength\n1. A rationale finder finder is proposed with plural views instead of either node- or edge-wise rationales finder independently.\n2. In this paper, based on the transformer framework, they propose self-attentive for dual views to guide the rationale finder.\n\nWeakness:\n1. I do not think of “no single transformation suffices to learn good representations‘’. There are many augmentation-free (in raw attributes and the graph) methods working competitively in different benchmarks. I'm curious if augmentation-free is also applicable in specific data.\n2 . In my opinion, the contribution is somewhat limited. It looks like the biggest contribution is a self-attentive augmentation. From a GCL point of view, the contribution seems insufficient. Can this technique be used in the other graphs and tasks? For example node classification and social network.\n3. I'm not sure I understand correctly that sampling makes this model not an end2end optimization. Have you consider using the reparameterization trick in VAE to solve this? or it is unreasonable or useless.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kQVwRVzJI-", "reviewer": "Reviewer_oC1b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use a self-attention mechanism to learn node-level and edge-level importance scores for a graph, and then conduct adaptive sampling considering the node/edge-level scores for GCL. The results have been conducted on 8 molecule datasets.", "summary_of_the_review": "While the method looks good and sensible, whether it can be generalized for other types of graphs (besides molecule) and the running efficiency is not clarified.", "strength_and_weaknesses": "S1. The paper is easy to follow and the idea is clear.\n\nS2. The paper uses case studies to visualize the attention scores to verify the rationality of the learned scores.\n\nW1. The experiments are only conducted on molecule graphs. How about other types of graphs that are widely used in GCL papers (e.g., social networks, Reddit, etc.)? I have personally tested the code for social network datasets without pre-training, and the results show no obvious improvement from baselines such as RGCL. If my test is true, then the practicality of the method is limited, as it needs a large dataset for pre-training, which may not be easy for the graphs except molecule graphs.\n\nW2. Runtime efficiency is not analyzed and experimentally tested", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y97ZNpwEYRG", "reviewer": "Reviewer_K54f", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to mitigate two limitations in the existing graph contrastive learning (GCL) frameworks: i) The lack of view diversity in data augmentation approaches degenerates the effectiveness of self-discriminative contrastive learning. ii) Existing methods utilize separated automated view generator and encoder, which damage the efficiency and efficacy of model training for GCL. To address the limitations, the proposed model employs a self-attention module to generate attention map for view generation and encoding simultaneously, to avoid introducing additional view generator.", "summary_of_the_review": "Considering many recent efforts on graph contrastive learning, this work proposes to leverage self-attention to infer node correlations for graph augmentation. However, the insufficient experiments may not be able to demonstrate the effectiveness of the new introduced approach.", "strength_and_weaknesses": "Strengths:\nThe proposed model utilizes the attention map in self-attention for automatic data augmentation, which improves the model efficiency and potentially improves the model training to be more stable and less prone to overfitting.\nDiverse types of views are generated using a shared view generator in the proposed model, which not only improves the efficiency but also enhance the data augmentation in GCL.\nThe proposed model is evaluated from different dimensions, including illustrative case study on the explainability of the self-attention module.\n\nWeaknesses:\nThe proposed model does not achieve best performance compared to baselines in some circumstances in Table 1.\nThe experiments are only conducted on biochemistry datasets, which cannot show the generality of the proposed model. More datasets such as academic citation networks, social networks and recommendation graphs could be utilized.\nThe authors claim that combining the view generator and the encoder is beneficial for model efficiency, but no experiments targets this point.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HCPA2XnOEuI", "reviewer": "Reviewer_YpSx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose to leverage the attention weights from a pre-trained transformer for guiding how the nodes and edges are sampled during graph contrastive learning. In contrast to previous work that often performs node or edge dropping for creating different views of a graph, the proposed method claims that the stochastic version by sampling via attention weights provides more meaningful views compared to the uniformly sampled ones. Experiments show improvement on MoleculeNet for graph-level classification tasks.", "summary_of_the_review": "- The paper can be largely improved by providing node-level and edge-level classification results on more benchmarks. Also would be great to clarify whether the proposed pipeline requires 2 stages of pre-training.", "strength_and_weaknesses": "Strength:\n- Depending on how the graph structures are constructed, different edges certainly contain different levels of information on how the neighboring nodes are related for different tasks. Leveraging additional information for helping identify the importance of the relationship intuitively should help build a more meaningful corrupted view of graphs for better graph contrastive learning.\n- The proposed modification of conventional graph contrastive learning is straightforward and easy to follow.\n- Figure 3 and Figure 4 are helpful in comprehending how the algorithm works.\n- Sensitivity w.r.t hyper-parameters analysis in Figure 6 is appreciated.\n\nWeaknesses\n- It is not clear how the transformer is obtained for providing the attention weights. If the transformer needs to be pre-trained before the graph contrastive self-supervised pre-training, then the whole process would involve 2 large-scale pre-training stages, one for obtaining the transformer, and another one for the actual graph pre-training. If that is the case, the proposed approach requires significantly more computational cost than the compared baseline methods.\n- As mentioned in the limitation section, the current method has been tested on graph-level classification tasks only. However, node classification or edge classification offer more flexibility for practical use cases, for example social network connectivity identification or fraud detection for anti-money laundering. It is important to have signals on those tasks before we can faithfully validate the proposed method works effectively and consistently.\n- The writing is good in the manuscript. However I would appreciate the authors using terms that are simpler and easier to understand. For example, the authors use the term “rational finder” but in fact it is simply using the attention weight for getting the importance values. That way, junior readers can have a better understanding of the proposed method quicker.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject