G-Censor: Graph Contrastive Learning with Task-Oriented Counterfactual Views
Authors: tianqianjin lin
Yangyang Kang
Zhuoren Jiang
Xurui Li
Changlong Sun
cui huang
Xiaozhong Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: graph contrastive learning
node property prediction
task-oriented counterfactual views

Abstract: Graph Contrastive learning (GCL) has achieved great success in learning representations from unlabeled graph-structure data. However, how to automatically obtain the optimal contrastive views w.r.t specific downstream tasks is little studied. Theoretically, a downstream task can be causally correlated to particular sub-structures in graphs. The existing GCL methods may fail to enhance model performance on a given task when the task-related semantics are incomplete/preserved in the positive/negative views. To address this problem, we propose G-CENSOR, i.e., Graph Contrastive lEarniNg with taSk-oriented cOunteRfactual views, a model-agnostic framework designed for node property prediction tasks. G-CENSOR can simultaneously generate the optimal task-oriented counterfactual positive/negative views for raw ego-graphs and train graph neural networks (GNNs) with a contrastive objective between the raw ego-graphs and their corresponding counterfac-tual views. Extensive experiments on eight real-world datasets demonstrate that G-CENSOR can consistently outperform existing state-of-the-art GCL methods to improve the task performance and generalizability of a series of typical GNNs. To the best of our knowledge, this is a pioneer investigation to explore task-oriented graph contrastive learning from a counterfactual perspective in node property pre- diction tasks. We will release the source code after the review process.

Under review as a conference paper at ICLR 2023
G- G-CENSOR: GRAPH CONTRASTIVE LEARNING WITH TASK-ORIENTED COUNTERFACTUAL VIEWS
Anonymous authors
Paper under double-blind review
ABSTRACT
Graph Contrastive learning (GCL) has achieved great success in learning repre- sentations from unlabeled graph graph-structure data. However, how to automatically obtain the optimal contrastive views w.r.t specific downstream tasks is little stud- ied. Theoretically, a downstream task can be causally correlated to particular sub- structures in graphs. The existing GCL methods may fail to enhance model perfor- mance on given task when the task-related semantics are incomplete/preserved the positive/negative views. To address this problem, we propose G-CENSOR, i.e., Graph Contrastive IEarniNg with taSk-oriented cOunteRfactual views, a model- agnostic framework designed for node property prediction tasks. G-CENSOR can simultaneously generate the optimal task-oriented counterfactual positive/negative views for raw ego- ego-graphs and train graph neural networks (GNNs) with con- trastive objective between the raw ego-graphs and their corresponding counterfac- tual views. Extensive experiments on eight real-world datasets demonstrate that G-CENSOR can consistently outperform existing state-of-the-art GCL methods to improve the task performance and generalizability a series of typical GNNs. To the best of our knowledge, this is pioneer investigation to explore task-oriented graph contrastive learning from a counterfactual perspective in node property pre- diction tasks. We will release the source code after the review process.
INTRODUCTION
Inspired by the convincing success of contrastive learning in the domain of computer vision (Chen et al. 2020: He et al., 2020) and natural language processing (Gao et al. (021). graph contrastive learning (GCL) has become an emerging field that extends the idea graph data (You et al., 2202aa]
Hassani & Ahmadi 2020 Zhu et al., 2021; Li et al. 2022), leading to generalizable, transferable and robust representations from unlabeled graph data You et al. 202T).
Nevertheless, the generation mechanism of contrastive views, which has been recognized an es- sential component in GCL (Zhu et al.] [222 Yin et al.] 2022; You et al. 2021). is still facing the following challenges: (a) Independent of downstream tasks. Although GCL is originally pro- posed for self-supervised learning, how to obtain the optimal positive view when downstream tasks available can be important question Xie al. (2022). However, most prior works, whether based on graph diffusion (Hassani & Ahmadi| (020). uniform sampling (Zhu et al. 2020). or adap- tive sampling (Zhu al. 2021: You al.) (021), ignore the downstream tasks' information. As shown in Figure [ whether a generated view is a appropriate positive view depends critically on the downstream tasks Chen et al. (022)). (b) Fitting spurious correlations. To introduce task in- formation, learnable data augmentation has been investigated to automatically obtain the positive views for downstream tasks (Yin et al.) 2022). While these techniques have achieved promising performance, they are prone to be plagued by spurious correlations between graph structures and downstream tasks like general supervised methods, thus hurting the generalizability of representa- tion model. (c) Difficulty in negative views selection. Beside positive views, negative sampling is also a vital component in GCL. Contrastive learning can benefit from hard negative samples (Joshua et al. 2021). Meanwhile, negative samples, actually similar to the raw instances, lead to per- formance drop (Chuang et al. 2020). Therefore, it can be hard to select suitable negative samples. Some works (He al. 2020) utilize great number of negative samples avoid this trade-off but

---

Under review as a conference paper at ICLR 2023
may cause scalability problems. These challenges become non-trivial with graph data since graph data are far more complex due to the non- -on-Euclidean property (Zhu et al. 2021).
Tesk A
Pesilive
Nogative
Negatäre
Positive
Posttive
Nogative
V
VE
V/3
N
VE
Vd
Task
Negative
Puiitive
Pesitive*
Negative
Positive
Negative
Figure I: An illustration for task-oriented contrastive views. Task to predict whether node is in triangle and task B is to predict the color of node. A task-oriented view is positive if and only it contains the credible evidence for the task label, otherwise it should be negative.
In this paper, we propose a novel model-agnostic framework for node property prediction tasks, namely G-CENSOR, i.e., Graph Contrastive IFarniNg with taSk-oriented cOunteRfactual views. G- CENSOR generates high-quality positive and negative views simultancously from a counterfactual perspective. In other words, task-oriented counterfactual question about the contrastive views could be asked: "Would a judgment on the task label of an ego-graph change if part of the structure of the ego-graph were erased?" The answer should be assigned to a positive view while the swer to a negative view should be yes. Technically, G-CENSOR adopts the learnable view generation approach and leverages an original counterfactual optimization objective decompose an ego-graph into the sub-structures causally correlated to the downstream tasks and the sub-structures spuriously correlated the downstream tasks. This two parts can further be regarded as positive and negative views, respectively. Learning representation model with such contrastive views can enhance both the model's s task performance and generalizability. Notably, G-CENSOR doesn't need to contrast in-batch negative samples, this characteristic can help G-CENSOR get rid of performance-mentt trade-off inherent in most prior GCL methods.
The core contribution of this paper can be three-fold: (a) To the best of our knowledge, this is a pioneer investigation to explore task-oriented graph contrastive learning from a counterfactual perspective in node property prediction tasks. (b) We develop a novel model-agnostic framework, G-CENSOR, an approach to automatically generate both task-oriented counterfactual positive and negative views to enable sufficient and efficient graph contrastive learning. (c) We conduct extensive experiments on eight real-world datasets to demonstrate the superiority of G-CENSOR over existing Mate-offthe-ar GCL methods improve the performance and generalizability of typical GNNs.
2 RELATED WORKS
2.1 GRAPH CONTRASTIVE LEARNING
Inspired by the success of data augmentation and contrastive learning in texts and images to address the data noise and data scarcity issues, Many graph contrastive learning (GCL) frameworks have been proposed lately (Liu et al., 2022 Xie et al. 2022). GCL works usually consist of a graph views generation component construct positive and negative views, and a contrastive objective to discriminate positive pairs from negative pairs (Xie et al., 2022). Most works generate positive views by uniform random transformation, e.g. node dropping. edge perturbation and subgraph sam- pling (Zhu et al. 22200; You et al. (222bb Yu et all.|2222) Zhao et al. 2021 You et al., al., (0200a) Hassani Ahmadi [2200] Sun et al., 2020) Velickovic et al.] 2019). Zhu al. (2021) proposed the adap- tive data augmentation strategies to reflect input graph's intrinsic patterns, i.e., assign larger drop probabilities to unimportant edges. Recently, several works have proposed trainable augmentation strategies You et al. 2021 at. al. 2022 Yin et al. 2022) to learn drop probability distribution over nodes or edges. However, few works have discussed on how to generate an optimal task-triented positive and negative views for graph data, and no learnable augmentation strategy has been pro- posed for node property prediction task. Table the comparison between G-CENSOR and the other state-of-the-art GCL models on 4 different properties.
2

---

Under review as a conference paper at ICLR 2023
Table 1: The comparison between G-CENSOR and the other sfate-of-the-art GCL models.
Property
MVGRL
GRACE
GCA
BGRL
AutoGCL
RGCL
G- G-CENSOR
Primary task
node&graphh mo
node no
node
node
graph
graph
node
Task-uriented
no
no
yes
mo
yes
Positive views
uniform in-batch
unifoomm
aduptive
uniform
learnable in-batch
leurnable learnable&inn-attl
learnable
Negative views
in-batch
in-batch
learnable
2.2 COUNTERFACTUAL GRAPH EXPLANATION
Recently, the research on explainability of GNNs is experiencing a rapid development Yuan et al, 2021). Most of GNN explanation methods (Ying et al. 2019 Luo et al. 2020) focus on identifying a subgraph of the original graph that contributes most to the prediction of a trained GNN. There also works make explanations based retrieving similar existed instances (Faber al. 2020)
Since removing the explanation subgraph from the input graph does not necessarily change the prediction (Baja et al. [02II, some counterfactual GNN explanation techniques (Lucic al. [022 Bajaj et al., (221)) have been proposed. These techniques address a problem that identifies a small subset of edges of the input graph instance such that removing those edges significantly result in an alternative prediction (Lucic et al., 2019; 2022, Bajaj et al. 2021I. While all above works deal with post hoc explanations of estimated GNNs, this study emphasizes ad hoc detection of the causally correlated subgraph with respect to tasks. See Section 4.2 for details.
3 PRELIMINARIES AND PROBLEM FORMULATION
In this section, we provide formal definitions of the optimal task-oriented counterfactual posi- tive/negative views with insightful explanations, and formulate the research problems.
Let G = E) be an undirected graph with nodes v, E V and edges (1; 0j) E E. For each node ti. feature description T., = [x1, Ø2, ....... .z]] and a task label Ye, E (1.....c) are assigned, where d feature dimension and c is the number of classes. A k-hop ego-graph (Daly & Haahr 2007) of node Y; can be defined as G = (Ve, k. Eug vz), where v; indicates the ego, Voz.k Vs the set of all the nodes that are at most hops away from U;, and Ev, S & is the set of interconnected edges between nodes in V G' (V h. E E U;) can be an ego-subgraph of Goyk,, where Fur,k S. Evc,k and V ui.k is nodes involved in F The complement of G',,, w.r.t Go.kk is defined by G' G'..k = (V involved in E Note that G and G must be connected subgraphs thus isolated edges in them will be removed. For brevity, we will omit iú and & in the subsequent sections.
(V . F FoLA:: vi). where F & is Fre. (F and V is the nodes
Definition 1 (Optimal Trik-tri Counterfactual Positive View GT). An ego-subgrah G' is the counterfactual positive view G for an ego-graph instance G if and only E' contains and only contains all the edges that are causally correlated to the task label.
optimal
Definition (Optimal Task-oriented Counterfactual Negative View GT). An ego-subgrah G' the optimal task task-orientedd counterfactual negative view G for ego- ego-graph instance G if and only if G' is the complement of the optimal task-oriented counterfactual positive view G+ w.r. G.
Figure expllans G+ and G from a perspective of ego-graph generation process for node property prediction tasks. In both cases, the relationship between F.' and y can be stable since F,1 is the direct cause or effect of y, i.e., causally-correlltedd to y. The joint distribution of E with y would be different if ego set changes, since E" and y are spuriously correlated(Schöllkon 2022) conditioned on ego, i.e.v. Therefore, according to the Definition I G should be E! v); according to the Definition 2 G should be G (Vo EE.v).
et al. 2012 Joshi
& He
G
Now let ó (G): {G 0, E V} + Re be any GNN that maps an ego-graph G to a probability distri- bution p of the ego over the label space RdC Contrastive learning with task-oriented counterfactual views is defined as:
Definition 3 (Graph Contrastive Learning with Task-oriented Counterfactual Views). Given G with its G and G learning a function & that maximizes the consistency between pair (o(G), (G+)) compared with pair (b(G),(GG)) ((G-))
3

---

Under review as a conference paper at ICLR 2023
VI
E
y
G
E.
V
E
E
V
(a) Graph structure determines label.
(b) Label of ego causes graph structure.
Figure 2: Causal diagrams Pearl 2009) for the 1-hop ego-graph generation process, A solid arrow from represents causal relationship from
b. The two dashed lines with text of GNN
represent that the whole observed graph (G) is used to predict task label (y). (a) The attributes of the ego (v) can cause two kinds edges: E0 (with nodes VO) and E' (with nodes Vl). Only the structure of (F.)) determines the label (y). (b) The label (y) is an inherited property of the ego (v) and this property causes E. (with nodes ) Meanwhile, other properties of would cause (E"). For interpretative examples, refer to Appendix A.1 for the causal diagrams of k-hop (k >> 1) ego-graph generation process, refer to Appedix
As Figure 2 shows. GNNs trained in normal flow may fit the spurious correlations between G and y since they take the whole observed graph G as input. Learning such non-causal association would reduce the reliability and the generalizability of the GNNs. In this study, we train GNNs with the auxiliary graph contrasting learning with task-oriented counterfactual views. Our goal is to make the learned representation to as independent as possible from G so that the model can achieve better generalizability.
Assuming that we can have a e that is sufficient encoder (Chen al. 2020), where an input graph G' can be encoded without information loss. Let R(Ø) represents the empirical risk of estimation and (((G)), y) represents the estimation loss, we can make a following assumption:
Assumption 1. Given a set {(G, y)}, Φ estimated with {(G y)} suffers greater empirical risk compared to & estimated with f(G+, y)} if & is sufficient, i.e. R(O-) = E[L(ă (G-) (.-) > R(Ø+)= = [T (GG()++G+,,)))) (G+),u),, where @ = arg minR(()) and M(b(G), y) can be larg (GG)
In Appedix(A.] we provide an example to explain the reasonableness of Assumption Let p (G; A) be function parameterized by A, which takes G as input and produces the optimal task-oriented positive/negative views. We can enforce yp satisfy the signature Assumption [ Based on the above definitions and assumption, in this paper, investigate the following two research questions:
How to design and learn model-agnostic under the Assumption 1 to conduct graph contrastive learning with task-oriented counterfactual views?
Question 1.
Question 2. Can really enhance é on given tasks with graph contrastive learning with task- oriented counterfactual views produced by y??
METHODOLOGY
To answers the Question 1. we illustrate the proposed framework G- CENSOR in this section. The overall architecture of G-CENSOR is shown in Figure
4.1 BASE GNN MODELS
We select GraphSAGB Hamilion et al. (017) (abbreviated as SAGE), GAT Veličković et al. (2018) and GIN Xu et al. (2199) as base GNN models, i.e., Ø, validate the G-CENSOR, These three models were chosen because they are representative and have been applied in many real scenarios. For details of these three models, refer to Appendix B
Then we can minimize a general prediction loss fpred e.g., the negative log likelihood (NLL), to get the estimated model & as follows:
(o) E(G./VeEv) (NTI,
4

---

Under review as a conference paper at ICLR 2023
Counterfactual Transformation
Counterfactual
Prodiction Loss
Contrastive
(A) Task-ariented Ego-subgraphs
Inper graph
Poutiiv View
Negativ View
aln
(b) Predicted Probabilty Distrimbtion
Graph
The overall architecture of the G-CENSOR. The solid lines represent a normal flow to train GNN with the raw ego-graphs by minimizing a general prediction loss (Section 4.1). The dashed lines represent the flow to train a counterfactual transformation module şp that generates task-oriented counterfactual positiveenegative views by minimizing designed counterfactual loss (Section 4.2) The dotted line is conducting graph contrastive learning with task-oriented counter- factual views (Section 4.3)
Figure 3:
We jointly optimize the three objectives learn & and yp (Section 4.4).
4.2 COUNTERFACTUAL TRANSFORMATION MODULE (CTM)
Figure depicts the proposed counterfactual transformation module for ego-graph, i.e., f., to esti- mate the probability that edge is part of the optimal task-oriented counterfactual positive view G for a G.
de
MLP
MLP2
Edps
(b)
(c)
Figure 4: Counterfactual Transformation Module. (a) Solid blue square original node feature æ, dashed gray square is landing probability of random walk from the ego node, i.e., r, and dotted pink square is whether a node is the ego, i.e., 1,=-- (b) The estimation of the probability that an edge belongs G+ (c) Sampling G according to the estimated probability of all the edges and constructing G based on G Only original node feature æ keeps in both G and G
Representation of an edge. To make an edge as distinguishable as possible, we preserve both the feature and structural information the representation of an edge. Technically, for node u in an ego-graph Ge, we first assign an ego identifier defined as 1 -=v and then construct auxiliary structural information leveraging Distance Encoding (DE) Li al. (2020) defined as landing probabilities of random walks of different lengths from ego v to the node which is denoted by ru- Therefore, new node feature vector for given as
G. x. Ta|G. Luzes where TuIG, = [(AD-1) u, (AD-1)2 12 R....... ... 1AD-1)1
(2)
A is the adjacency matrix for G. such that Au,s matrix for G. where Da is the degree of node u. I indicates the length of random walks, which is set to the same k. Then an edge (u, s) G, is represented
iff (u, <) E E, and D the degree (diagonal)
euc(G,
L.IG.
LujG. L.(G.
(3)
Probability that an edge belongs to G For simplicity, we assume the binary variable 1 (x,s)EE follows Bernoulli distribution, i.e., (u.s)et) Ber(Bucia.) and Bas|G estimated by multiple multilayer perceptrons (MLPs) follows:
5

---

Under review as a conference paper at ICLR 2023
MLP: (eus)..:ddi)).
P ((u, E E.) Ouc|G. =
(4)
m71
where S o² is the sigmoid funtion.
Assuming all the edges are independent of each other, the probability of a sub-egograph Go being G i.e.,. P(G = G,)) can be obtained by
P(G = G.) (, P ((u,s) E F.)) II (U,)E,, VE, (1 ((u, ET))
(5)
Note that once G has been obtained, nature, we adopt the reparameterization trick [ang et al. (2017); Maddison et al. (2017) to enable updating parameters in the MLPs with general gradient-
further construct = G Due to the discrete
optimizer. See Appendix C for details.
Computational Complexity. The complexity for the edge representation euc|G, is O(l -V.l (
due the computation of F.. (Yuster & Zwickk 2005) and the complexity for the probability estimation is O((1+ + m |.ll d²), where we assume the hidden dimension in MLP; is equal to d and h is the number of layers in MLP,. As a comparison, the complexity for baseb flow used in learnable views generation YYin et al. al. 2022 Li et al. 2022) is | Ce k- VV, h- E..l dd²) where C. and Cp are the complexity of message passing and combination respectively. practice, the latter is likely to greater than the former due large C. and C,,. See Appendix D
for the real runtime evaluation.
Optimization objective. To satisfy the signature of G in Definition | the minimum NLL loss on of (G+; ++) can be minimized to encourage G sufficient to predict the correct label y, as follows:
L (1p (+) =Bot~Piat ((ttPP(tt-.) G',) (fpred(0t))
where o+ = arg a+
(o+).
Under Assumption 1 the relationship between G and G can be established by marginal rank loss that enforces Φ (G ( results in a greater empirical risk.
Lrank (,, PCttp(ot=0; max (0, -pread(d ) + (or) + Emargin )] (7)
where Φ arg min
Additionally, to enforce to find the most significant edges, G should be as sparse as possible, which aligns with the findings of previous studies in causal discovery and counterfactual explana- tion (Zheng et al.) 2018, Wachter et al. 2017).A LI regularization Caise is applied on the probability of being selected into E for all edges:
Csize (4) = Busssennt [P ((u, Et)]
(8)
Note that the prior proportion of the causally-correlll part is different in different domains. With a parameter a to control (ys))((p), the counterfactual loss for the counterfactual transformation module can be formulated as:
(e) Lour 2+) + Cranklyp Zt $ + a Leise (1p).
The Differrncc with Counterfactual Explanation. To address task rather than model, Leufs max- imizes the mutual information between G and yo while counterfactual explanation works maxi- mize the mutual information between G and @(G,; : θ). Counterfactual explanation works directly maximize the empirical risk Cprea(0 ), which is helpful to explain estimated model but can sort spurious structures into G since spurious structures is also correlated to label. Therefore, we choose rank loss to ensure consistency with Assumption 1

---

Under review as a conference paper at ICLR 2023
4.3 CONTRASTIVE LEARNING WITH TASK-ORIENTED COUNTERFACTUAL VIEWS
Contrasting the raw ego-graphs to both the positive and negative counterfactual views can inject task knowledge from y into é and lead better model performance and generalizability. An In- foNCE (van den Oord et al. 2018) style loss is formulated as
exp (sim(G G.)/T) exp(sim G )/) exp (G,, G GT)/T) (10)
(a( = (Ia, (7..000005)) log
where (G, G²) 1- o(G')) and JSD is the Jensen-Shannen divergence. 7 is the temperature parameter and plays role controlling the strength of penalties on the task-oriented counterfactual negative views. The larger the is, the smaller the influence of similarity between original ego-graphs and the task-oriented counterfactual negative views is.
Scalability. Similar to BGRL Thakooo et al. 202Tp. this contrastive loss doesn't 't need to contrast in- batch negative samples and has time and space complexities scaling linearly to the batch size. Thus, G-CENSOR can get rid of performance- memory trade-off inherent in most prior GCL. methods.
4.4 JOINT LEARNING GNN AND CTM
For simplicity and efficiency, empirically share with ot and ø and jointly learn ø and oo with a weight parameter 3 for the contrastive learning. Therefore, a final joint loss can formulated as:
Cjoine(0 .p) = Cpred(0) + Les(sp | o) + BLa(
(11)
5
EXPERIMENTS
To answer the Question 2
we conduct extensive experiments on eight real-world datasets from two perspectives: task performance and model generalizability.
5.1
EXPERIMENTAL SETUP
Dataset. Eight open benchmark datasets across three domains are investigated: (a) five citation networks (Bojchevski & Günnemann] 2018) including CoraFull, CoraML, CiteSeer, DBLP and PubMed. (b) two product networks (Shchur et al. 2018) including Computers and Photo. (3) an
image network (Zeng et al. 2020 Flickr. See Appendix E. for more details of datasets.
Baselines. Three types of baselines are compared: (a) base models, i.c., SAGE, GAT and GIN.
(b) GCL methods with uniform adaptive data augmentation, i.c., EVGRL(Hassan & Ahmadi
2020). GRACE(Zhu et al., 2020), GCA (Zhu et al. (021) and BGRL (Taaooor et al.) (202)). (c)
GCL methods with learnable data augmentation, i.e., AUTOGCL (Yin al. 2022) and RGCL Li
ct al. 2022). For more details, refer to Appendix E.2
Implementation. For fair comparison, the architecture of base GNNs and batch training setting were the same for all methods. All the experiments were run 5 times with random seeds from 0 to For more implementation details, refer Appendix E.3
5.2 IMPROVEMENT ON PERFORMANCE
As shown Table 2 G-CENSOR can significantly enhance the performance of base GNNs (best in 23 out of 24 settings). Specifically, G- G-CENSOR improves the accuracy of the base GNNs from 0.4% (GraphSAGE on CiteSeer) to 20.67% (GraphSAGE on CiteSeer). Considering all settings, the average gain of G-CENSOR is around 4.6%. Morever, G-CENSOR can consistently outperform the SOTA methods from 0.06% (GAT on Computers) to 14.59% (GIN on Flickr) except for setting of GAT on Flickr, where G-CENSOR got the second best performance. The average gain of G-CENSOR against SOTA 1.87%. Note that on datasets like Photo, though the SOTA methods have already achieved high performance (>93%), G-CENSOR
still pushes the boundary forward (>94%). All these results can prove the ability of G- CENSOR to enhance model performance. Meanwhile, GCA, approach that adopts adaptive data augmentation and thus enables models learn important

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison in node classification accuracy on test set with independent and identical distribution. Bold and underline indicate the best and second performance, respectively.
Mediod
CiteSeer
Computers
Cora.Full
CocL.ML
DELP
Flickr
PubMeed
BASE
01.25e0.28
85.62=2.05
55.72el.17
83,8361,48
80,97:0.90 99
42 42.58.0.74
02.99+0.35
85,3400,70
MVGRL
65. 68.8321148
00M
ODM
44,3627.12
74.1940.99 977
TLE
OOM
16.1620071
GRACE
99.8310.45
86.67=140
86.1710.88
84.13&1.57
81.7410.71
49 30.0.66 56
91.4811007
56. 96.86100.3 30
GGA
90.20400.88
84.71=1.74
56.8741.34
84.5240.90
91.81-9.56
49.7661100
92.500e1.04
50.85a91.3
BGRL
90,1721.00
86 86:25500.5
50, 25.10g1.25
83.6642.43
81.3540.32
50,6940.25
91.9011.83
86,5021..9
SAGE
RGCL
29.25940877
58.31=12.88
49.00=1.89
80.08+1.66
38.91=7.54
81, $1.2677.22
81.85+0.52
AUTOGCL
90.26660.39
85. 85.73:0.88
56, 30.3050.72
83 83.9820.79
81.3341 121
40.9921.80
92.4029993
86.42401.9
G-rENSOR
91.65e0..6
88.65-8447
58.1089.91
15.19+1.06
84,01-00466
51.14+0.15
94.52-0.25
18.17
90. 19.055
89 39.80.
58.11
$3.5611.39
81 58.200
47.31
92 88.881
$5.25%0.70
MVGRL
88. 39.6440039
0OM
OUM
81.9611.11
21.2550.77
TLE
OUM
86.1040.81
GRACE
89.7720.33
90.15-041
29.2840.74
84.2242.21
82.5240.99
51.56s0.29
93.3220.45
56.3440,82
GGA
00.22+0.71
20.56e0
52 52,6240.52
84.49+0.75
82.0011154
51.91:0.99
93. 93.49H20.08
86.75+11.27
BGRL
90,0%a1.17
89.4000.88 888
58, 1366088778
84.4621.29
81.5220033
50.55e1.12
92.5080070
85.9961).0
GAT
RGCI.
88 18.94+1.56
61 11.64-23.2
45 60+0.92
79.64+1.50
80.06+1 1.5
42.38+0.74
$6.56+1.32
80.83+1 0S
AUTOGIL
20.27+0187
89.54m0880
58.69 69
82,75+0.85
82.1440.66
49.97=0877
93. 164066
86,20H1).42
G-CENSOR
91.26:0.68
90.61:0.34
60.21
85.85
84.06:0.65
61.6660.22
94.13:0.46
87.2840.35
BASE
39.3610.80
82.97=2.20
56.4050.61
81.8420:96
79.5220.74
45.3122.56
58.2511.06
86,0520.41
MYGRI
85,90+1,14
0OM
O0M
71.27+3.52 52
8147H1176
TLE
ODM
85.18
GRACE
88. 88.74
88, 38.35-0086 86
55 9689666
83.01+0:95
79.641 100
44.11=1.19
98.2140.99
86.8++1.21
GGA
89.53a60.22
85 90:1 09
56 56.45600.9
83,77+1.03 1,3
N0.12:0 81
42:76e1.188
92.7561.49
36.802)))551
RGRL
80.93401.85
88.45=1 34
86.8240.85
82.98+1.05
80.66601997
43.49=1.15
91.95e1.47
86. M6..6i+1..6
GIN
RGCL
88, 1.35
H2, 62.95u11.8 80
52, 1930.75
80,4210.83
79.85:0.3 62
43,90s2.66
89, 19.344..06
81.61:0.25
AUTOGCI.
89. 1711,11
88.46-1
55.21 15.29
82.39+1.34
79.91+0.55
37.66e1.98
92.40+1
86.69H0.92
G- G-CENSOR
90.9 00.9610.56
90. 90.36:0.54
58.4650.70
84.52:0.72
83.42:0.33
51.9260177
94.5710.06
87.6660,23
'OM meuns Out Of Memory (2GB) and TLE means Time Limit Exceeded (seconds per epoch >> 1000s)
structures, outperforms other baselines in most settings. This may imply the effectiveness of G- CENSOR to auto-select the task-oriented positive structures. As for learnable data augmentation methods, i.e., AUTOGCL and RGCL, they didn't achieve satisfactory performance on most settings probably because of the task shift from graph classification to node classification.
5.3 IMPROVEMENT ON GENERALIZABILITY
To verify the ability of G-CENSOR to boost the generalizability of GNNs, we further conduct ex- periments on out-of-distribution data split setting based on the confounder discussed in 3 i.e.. the ego. For each dataset, we first run a NODE2VEC (Grover & Leskoved, (016) to get nodes' embeddings and cluster the nodes to two clusters by K-MEANS (Lloyd] 1982). The cluster with larger sample size is randomly divided into training and validation sets and the other is regarded as testing set. As shown in Table 3 G-CENSOR still significantly enhanced the performance of outperrrman base GNNs up to 18.26% with an average improvement of 6.02% and consistently outperformed the SOTA methods up to 7.40% with an average improvement of 1.62%. While a larger performance degradation suggests a larger distribution gap between the test set and the training/validation set, it's observed that base GNNs can benifit more from G-CENSOR compared to SOTA methods. For example, Cora Full, Photo and Flickr are the three datasets with the most significant performance degradation but G-CENSOR outperformed the best SOTA the most on these three datasets. All these results demonstrate the superiority of G-CENSOR to enhance the generalizability of various GNNs.
5.4 SENSITIVITY ANALYSIS
G-CENSOR's 's sensitivity w.r.t. hyperparameters a, 5, and / (with GAT as base model) is evaluated by presenting the median performances in Figure [ It's observed that (a) a plays crucial role in G-CENSOR. Different tasks prefer to different a, e.g., the best a is around 0. on Cora but is around 0.005 on Photo. And increasing constantly can potentially hurt the performance, e.g., the accuracy tends to go down after a exceeds 0.005 on Flickr. This is reasonable since a implies a prior proportion of the nausally-correlatee
part in particular network discussed in Equation 9 (b) G- CENSOR is relatively robust to 3. It's seen that G- G-CENSOR can improve base models' performance under various values of B. For sensitivity analysis with GraphSAGE and GIN, refers to Appendix F
8

---

Under review as a conference paper at ICLR 2023
Table 3: Comparison in node classification accuracy on out-of-distribution test set. Bold and underline indicate the best the second best performance, respectively. A represents the average accuracy degradation of base GNNs on the testing set compared to the accuracy on the validation set. represents the average gain of G-CENSOR against the best SOTA GCL methods.
Machod
CiteSeor
Computers
Full
Cora. ML.
DBLP
Flickr
PubMed
-1.21%
4.76% 0.88%
-6.811 2.89%
-5.18%
15.80%
-1..3% 2.95%
19.19% 3.05%
00091
1.08%
0.54%
BASE
90.29:0.68
82, 89: 99
88.8885888
80.05a4.98
91,22x0.83 83
46 66.66-0.7
75.57a6.13
87. 87.2660.52
MVGRI.
68.41+2.60
GOM
OOM
51.2446.09
86.90 00.00+110 43
T.F
ODM
87 77.6540.75
GRACE
59,7760.48
91.1541.1
83.2413.70
91.8710.52
77.6515.59
GGA
89.77a1.60
88.8543.20
33,78ts404
23.6412.92
91. (1.97: 46
46 36.29H0.4
76112..10 19
88 08.0661..8
BGRL
89.6540.73
88.5843.45
33.9262.19 19
83.1143.58
92.4240.58
46 36.55:0.30
50.2266.55
18.01.
SAGE
RGCL
$9.29g1.69
51, 41.95
25, 25.16111.44
79.6345.02
91.9910.99
36 1..00.0000
49.18.28.8 98
86, 8510.33
AUTOGCL
89.6120.75
86.4487.33
32, 12:1.2333
82.2745.09
91.74a01.00
36 36.0944.84
74.76:1.27
87.01260.58
G-CENSOR
91.15+1.03
92.62:0.23
36.44
84.3643.99
93.3120.30
47.51:0024
84.4222.11
88.7720.39
DASE
89.9810.91
89,9 89.9844.48
36.9911.82
83.3012.90 96
91.7021.13
43.96.4.177
74.7419.61
87.46.1.00 1,06
MVGRL
88.4510.96
00M
0OM
81,9613.96
92,07:1044
87.81100.63
GRACE
58.9341.00
93.1541 108
39.87244.06
84.1942.74
93. 93.1440.32
47. 17.38-0.0
$6.1422.50
87.7420.34
GGA
90.0520.97
93,2720.49
40.28:1.63
84.7552.73
92 92.9610.45
46. 79.00.1
96.803.112
68.2640029
BGRL
90.3740.60 60
91,9660.79
39, 39.4913:42
83.8113.45
92.74.0.65 88
44 44.940068
81.1513.33
88,0410.89
RGCL
88.3721.51
55,5437.86
22.75:3.24
79.01a4.449
92, 02.9080.71
37.6723.99
66.25210.34
86,92211.36
AUTOGCI
80.9640.64
91,97=1.02
37.96+3.01
82.6744.05
92.41-0422
46.30+2.46
35.35e3.06
87,44H))45
G-UENNOH
91,48+0.75
43.29H0.45
40. 10.19::000
85.35+1.84
93.40
47.22
8S. 98.1.33
88.6240.32
BASE
88.9721.17
90.1511.84
31.24
80.5145.39
19.9321.16
44 44.5422477
18.1780161
MVGRL
86.85a1.82
00M
0OM
69.8566.36
92,02:1.24
TLE
OOM
87.6440.75
GRACE
89,57+1.02 02
92.81
328221008 13
82. 32.9843.37
86.45+142
38.11
St. 11.9922.22 22
87. 77.777
GGA
89,2921.33
90.47:2.27
38.3951002
53.1143.55
88.02:1.34
41 11.061111
$1.2124.00
87,9K20100
BGRL
39.0920.49
91.9 91.9640.73
34.7422.55
91.0211.37
38.22.5.15
71.2210.65
19.1940.54
GIN
RGCL AUSOGCL
88.08e1.00
86.98=2.55
31.11=2.97
81.9642.34
02.18-11:3
41 46m3370
30.43e1.63
86.72-1145
87.6%+1.30
90.4860.93
28.2765.45
78.9944
38.5420004
33.57=1.77
77.1241..32
87.1942.11
G-CENSOR
90.5619).45
93.75e0.53
35.26+1.00
83.1683133
98.26511.35
47.82:0.11
15.98
89.1340.34
OOM Out Of Memory (> 32BB) and TLE means Time Limit Exceeded (scconds per opach 1000s)
AVT Eaken
0-0.22e-d11
3-1.
1-1.2
(a) Sensitivity w.r.t hyperparameter and 8 on all i.d split datasets with GAT.
tw0.3 65
3
3-3 100-5
fw1.0r001
Ow Sw0. 15
Oel 2oL.0:0222
(b) Sensitivity w.r.t hyperparameter and 3 on all o.o.d split datasets with GAT.
Figure 5: Sensitivity analysis on hyperparameters with GAT as base model.
6 CONCLUSION
This paper proposes novel graph contrastive learning framework G-CENSOR, approach leverag- ing task-oriented counterfactual views generation, to enhance the performance and generalizability of GNNs on node property prediction tasks without any change the model structure and the in- ference flow. Through extensive experiments with in-depth analysis, we demonstrate the superiority of G-CENSOR. However, the counterfactual data synthesis can further be improved based on coun- terfactual inference, i.e., the three steps of abduction, action and prediction in a structural causal model (Pearl 2009). We will explore it in the future work.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Mohit Bajaj. Lingyang Chu, Zi Yu Xue, Jian Pei, Lanjun Wang, Peter Cho-IHo Lam, and Yong Zhang. Ro- bust counterfactual explanations on graph neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang. and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL bttps: //openrcview not uorum? 1d=WGmLMMb8111
Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsupervised induucttisseddduutve inductive learning via ranking. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URLhttps:
//openreview.o net/ wwtp:ett//orre htr///ormmmmmm wwrrummiddrrlae 1ZdkJ-
Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learn- ing. ICML' IMLR.org. 2020.
Ching- Yao Chuang. Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debi- ased contrastive learning. In Н. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 8765-8775 Cur- Associates, Inc., 2020. URL hhttps::/procceeiingg wttps/eerrr.....p.aaeeeeeeea wttp:ccpppppp/.//o.eaeap 63c3ddcc7b23daa. e 2dc41f9944443000 Pappaee
Djork-Amé Clevert, Thomas Unterthiner. and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representaatioss ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceed- ings, 2016. URL htttttt:///ar v. httpg//ba/111111555553
Elizabeth M Daly and Mads Haahr, Social network analysis for routing in disconnected delay-tolerant manets. In Proceedings of the 8th ACM international symposium on Mobile ad hoc networking and computing, pp. 32-40, 2007.
Lukas Faber, Amin Moghaddam, and Roger Wattenhofer. Contrastive graph neural network explanation, 2020.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embed- dings. In Proceedings of the 202/ Conference on Empirical Methods Natural Language Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 11865///1/22211emmpmmm///
URL
jemnlp main.552.2
Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (eds.), Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Fran-
cisco, CA, USA, August 13-17, 2016, pp. 855-864 ACM, 2016. doi: 10.1145/999972222977544 URL
https:/ ptttps:/dei.coom wecoi..rrg.. wrt.rrgg//// 1t.gg//10..
William L. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, 2017.
Kaveh Hassani and Amir Hosein Khas Ahmadi. Contrastive multi-view representation learning on graphs. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 4116 PMLR, 2020.
URL
mlr. press vv1 htt.///11//aaannneeee
Kaiming He, Hanqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 JEEE/CVF Conference Computer Vision and Pattern Recognition. CVPR 2020. Seattle, WA, USA, June 13-19, 2020, pp.
Computer Vision Foundation IEEE,
2020. doi: 00.1199CYPR4426002220000999
URL
2020 00975
Kurt Homikk Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal ap- proximators. Neural Networks, 2(5):359-366 366, 1989. ISSN 0893-6080. doi: https://doi.org/10..066
0893-6080(39)900220 8. URL httpps:////at. http::://wwwwwwceeeceeeee. wnweeiiiecccccccc rect. com/ http://oo//cceee///rriiieeee 0893608089900222
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with sumbel- In 5th Inter- national Conference on Learning Representations, ICLR 2017. Toulon, France, April 24-26, 2017, Con-
ference Track Proceedings. OpenReview.net, 2017. URI. pectps://openrett ew net. net/forum?id
rkH3y85ee
10

---

Under review as a conference paper at ICLR 2023
Nitish Joshi and He He. An investigation of the (in)effectiveness of counterfactually augmented data. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3668-366 3681, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 256. C mtLlitttps:/allaathollle
org/2022 acl- 1oa.l.oon..255.
Robinson Joshua, Chuang Ching-Yao, Sra Suvrit, and Jegelka Stefanie. Contrastive learning with hard neg- ative samples. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:://opentevieee http://pppeneeeeeee..es/////rrra wwt/forum/ CRIXOQOUTh-
AA Leman and B Weisfeiler. A reduction of graph to canonical form and an algehra arising during this Informatsiya, 2(9):12-16, 1968.
reduction. Mruchho--Tehhicheee
Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovee. Distance encoding: Design provably power- ful neural networks for graph representation learning. Proceedings of the 34th International Conference on Neural Information Processing Systems, 20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 97817 11117799966
Sihang Li, Xiang Wang, An Zhang, Xiangnan Hc, and Tat-Seng Chua. Let invariant rationalc discovery inspire graph contrastive lcarning, In ICML, 2022.
Yixin Liu, Ming Jin, Shirui Pan, Chuan Zhou, Yu Zheng, Feng Xia, and Philip Yu. Graph self-supervised learning: A survey. IEEE Transactions on Knowledge and Data Engineering, pp. 1-1, 2022. doi: 10. 10.1 TKDE:2022.3172903
Stuart P. Lloyd. Least squares quantization in PCM. JEEE Trans. Inf. Theory, 28(2) 136, 1982. doi: 10.1109/TIT. 198211992249921 1056489. URL https:://do doi owg//o.///..o wttpg//...1.///....... 1099/TI1 1982. 1C56489
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LAA, USA, May 6-9, 2019. OpenReview.net, 2019.
net/ for wtt.omnni...kkiiiggggp
Ana Lucie, Harrie Oosterhuis, Hinda Haned, and Maarten de Rijke. Focus: Flexible optimizable counterfactual explanations for ensembles. arXiv preprint 1911 2019.
Ana Lucic, Maartje A. Ter Hoeve, Gabricle Tolomei, Maarten De Rijke, and Fabrizio Silvestri. Cf- gnnexplainer: Counterfactual explanations for graph neural networks. In Gustau Camps-Vall Valls, Francisco R. Ruiz, and Isabel Valera (eds.). Proceedings of The 25th International Conference on Artificial Intel- ligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 4499-4511. PMLR, 28-30 Mar 2022. http:::ts:::/.t.oooot.
mr. www.m..r...ess........ wrwps/psssss..... wttpesssvvS./1I/// w.w.///ccc2i.aaaaa httmmiimmaa...mll.llm..
Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Pa- rameterized explainer for graph neural network. In H. Larochelle. M. Ranzato, R. Hadsell, M. F. Bal- can, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 19620-19631. Curran Associates, Inc., 2020. URL htt http:s::////ccceceeeeeooooo http://www...epppppeceeeeeee///a c3 337500dd3301333ddeeed66666663363 768836666688888888 wwwpp666668ba-aaa..eeoooo Faper. pdf
Andrew Maas, Awni Y. Hamnun, and Andrew Y. Ng. Rectifier nonlinearities improve neural network acoustic models. In in ICML Workshop on Deep Learning for Audio. Speech and Language Processing, 2013.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https:
// /openreview net forum? 1d= (3511gg
Julian J. McAuley and Jure Leskovec. Image labeling network: Using social-network metadata for im- age classification. In Andrew W. Fitzgibbon. Svetlana Lazebnik, Pietro Perona, Yoichi Sato, and Cordelia Schmid (eds.). Computer Vision ECCV 2012 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part IV, volume 7575 of Lecture Notes in Computer Science, pp. 841, Springer, 2012. doi: 10. 0100779978-3-44-3377665 _559. 978 3- 642 33765- 59
Judea Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, USA, 2nd edition, 2009. ISBN 052189560X.
Bernhard Schölkopf. Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. In Proceedings of the 29th International Coference International Conference on Machine Learning, ICMI.12, pp. 459-466, Madison, WI, USA, 2012. Omnipress. ISBN 4781450318851.
11

---

Under review as a conference paper at ICLR 2023
Oleksandr Shehur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. CoRR, abs/1881.05868,, 2018. URL httpp://axx...org//////
05868
Fan-Yun Sun, Jordan Hoffmann, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In 8th International Conference on Learnnng Representatiins, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. Open 2020.
[RLLLtttps///ppezeezzet .nt./ forumm wtw.uummddddrrdddeee
Shantanu Thakoor, Corentin Tallec, Mohammad Gheshlaghi Azar, Rémi Munos, Petar Veličkovié, and Michal Valko. Bootstrapped representation learning on graphs. In ICLR 2021 Workshop on Geometrical and Topo- logical Representation Learning, 2021.
Aätonn van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. C3748
CoRR, abs//107..o/ .0748, 2018, htllhtt:://axxx..rrr///aaa
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lid, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations (ICLR), 2018. URL
htttps:/cgenreevieew net/ forum? d-rJXMpikkk
Petar Velickovic, William Fedus, William L Hamilton, Pictro Lió, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. ICLR (Poster), 2(3)4, 2019.
Sandra Wachter, Brent Mittelstadt, and Chris Russell. Counterfactual explanations without opening the black box: Automated decisions and the gdpr. Horv. JL & Tech., 31:841, 2017.
Yaochen Xie, Zhao Xu, Jingtun Zhang, Zhengyang Wang, and Shuiwang Ji. Self-supervised learning of graph neural networks: A unified review. IEEE Transactions on Puttern Analysis and Machine Intelligence, pp. 1-1, 2022. doi: 10. 109/TAAM..2022..110555
Keyulu Xu. Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful graph neural networks? In International Conference on Learning Representations, 2019. URL http:tss///peeeee.ww...eotc Forum?id-ryGen A5Km
Yihang Yin, Qingzhong Wang, Siyu Huang, Haoyi Xiong, and Xiang Zhang. Autogel: Automated graph contrastive learning via learnable view generators. In Thirty-Sixth AAAI Conference on Artificial Intelligence,
AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 March 1, 2022, pp. 8892 8900. AAAI Press, 2022. CRLibtts:://oj] aaal org/ index. php/
Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generat- ing explanations for graph neural networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. http:/ttts//poceeeeeee werassnnerips.. wwtperpp....pppppr.//////epppeee wwtppppppp//////ee/e.c.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Iisuan-Tien Lin (eds.). Advences in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020. December 6-12, 2020, virtual, 2020a. URL 3fe230348e9al221133313 oaa.a//2e..bb0.11..eeaaaaaaaaaae
html
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph con- trastive learning with augmentations. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 5812-5823. Cur- Associates, Inc., 2020b. URL hhttps://prcceett www.deddinnnnn...nni..eee. www.nnnuri.... pttpppaaap////////11aaaa
Yuning You, Tianlong Chen, Yang Shen, and Zhangyang Wang. Graph contrastive learning automated. Ma- rina Mcila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 22121-12133 PMLR, 2021. URU/tttp://prrcccci.
mlr. press /v1./v133991 v139/you21a, html
Junchi Yu, Tingyang Xu, Yu Rong, Yatao Bian, Junzhou Huang, and Ran He. Graph information bottleneck for subgraph recognition. arXiv preprint arX1v:2010.05563, 2020.
12

---

Under review as a conference paper at ICLR 2023
Hao Yuan, Haiyang Yu, Shurui Gui, and Shuiwang Ji. Explainability in graph neural networks: A taxonomic survey, 2021.
Raphael Yuster and Uri Zwick. Fast sparse matrix multiplication. ACM Trans, Algorithms, 1(1):2-13, jul 2005. ISSN 1549-6325, doi: 10.145/1077444410774666 URL
1077466
Hanqing Zeng, Hongkuan Zhou, Ajitesh Srivastava, Rajgopal Kannan, and Viktor K. Prasanna. Graph- saint: Graph sampling based inductive learning method. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
https::/cpenrewwiew net/ [orumpid-dBeesskpBPP
Tianxiang Zhao, Xiang Zhang, and Suhang Wang. Graphsmote: Imbalanced node classification on graphs with graph neural networks. In Proceedings of the 14th ACM international conference on web search and data mining, pp. 833-841, 2021.
Xun Zheng, Bryon Aragam, Pradeep K Ravikumar, and Eric Xing. Dags with no tears: Continuous optimization for structure learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates, Inc., 2018. URL https:: 133
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.041:: 2020.
Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Jure Leskovec, Marko Grobelnik, Mare Najork, Jie Tang, and Leila Zia (cds.), WWW 21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April /9 23, 2021, pp. 2069 2080. ACM/ IW3C2, 2021. doi: 10. 10.1145/344288111 1.1434442381.349980222 3442381 3449802
URL
Appendices
A EGO-GRAPH GENERATION PROCESS
A.1 INTERPRETATIVE EXAMPLES FOR H-HOP EGO-GRAPH
Example 1 (For Figure Assuming this is a paper impact (y) prediction task defined on citation graph. Generally, the type of v affects the graph structure composed of cited papers (E' with V"), e.g., survey papers are more likely to be cited than general papers. Meanwhile the type of affects the graph structure composed of references (E" with Vº), e.g., survey papers have more references than general papers. While being cited more (E*) ) stably indicating higher impact (y), GNN use all observed structures (G) is prone to learn that papers with more reference (E") tend to get higher impact (y), and this relationship can be considered as a spurious correlation.
Example 2 (For Figure 2b). Assuming this an image category (y) prediction task, e.g. landscape and city, defined
image graph. In this graph, edge exists if two images share some common properties, e.g., location, producer and object (McAuley & Leskoved 2012). (FA with V1) can be images with same objects (related category) and (E" with Vo) can be images with same producers or locations. While category (y) stably indicating specific objects thus generate connections between (y) and (E'), GNN use all observed structures (G) may learn joint probability distribution on category (y) and images sharing same producers or locations (E"). This joint probability distribution can be considered as spurious correlation.
Example 3 (For Assumption] Tp. In Example I all highly-cited papers are necessarily considered as high-impace papers, but there exist high-impact paper without many references. Similarly, in Example an image must have connections to those images that belong to the same category, but an landscape image be produced by photographer who often takes pictures of human models.
13

---

Under review as a conference paper at ICLR 2023
A.2 CAUSAL DIAGRAMS FOR K-HOP EGO-GRAPH GENERATION PROCESS
Let us explain k-hop ego-graph generation process by hierarchical traversal allowing revisiting node or edge. The causal diagrams for k-hop ego-graph are shown in Figure
Figure 6a In this case we assume an edge must have at least one path to the ego, where edges on the path all causal w.r.t a task, when the edge itself is an causal edge w.r.t. the task. This ensures that the causal ego-subgraph is a connected graph. Now if the ego-graph is acyclic, there are three type edges: (a) edges whose path(s) the ego only include causal edges (i.e., edges indicated by the arrows in the upper row), (b) edges whose path(s) to the ego include no causal edges (i.e., edges indicated by the arrows in the bottom row), and (c) edges whose path(s) to the ego include both causal and non-causal edges (i.e., edges formed by the slanted arrows). Edges of the first type can causally-coree correelate since they determine the label y, but edges of the second type can be spuriously correllte since the association path from them the label y exist fork (confounding) patterns joined by v. As for edges of the last type, they and label y can be also confounded by V where E {0, ..................
1}. If the rgo-graph has cycles, which means there exists at least one edge can be included in multiple EJJ, where j E {0, 1}. This edge causally-correlater to the label and only if max(j) = which indicates that this edge is part of causes of the label y, otherwise it is spuriously-correace to the label since can be confounded by v like other normal edges in E1o,
Figure 6b Actually high-order edges in this has no causal relationship the label y and they can be confounded by either or V1.1
Vk.1
22,1
2
V
E
y
V2,0
E
K-
V2,0
2
V
E
V
(a)
V4,
E
y
Vku
E
V2,0
V1.0
L
V
Figure 6: Causal diagrams PPeal 2009) for the k-hop (k: > 1) ego coo-graph generation process. A solid arrow from a b represents causal relationship from b. The two dashed line with text GNN mean that we usually use the whole observed graph (G) to predict task label
B BASE MODELS
GraphSAGE (abbreviated as SAGE). Hamilton et al. Hamilton et al. (017)) proposed three vari- ants and utilize the simple but popular version, SAGE-mean, which directly aggregates neighbors by averaging the embeddings of them. In particular, the strategy of SAGE-mean defined as fol- lows:
h + (W (h Mcan ((ht-))
Yu E (u)))))
where W the linear transformation weight matrix in the k-th layer, h is the embedding of node in the k-th layer and (v) is the neighbors of node v.
GAT. A graph attention network model employs a multi-head attention mechanism on the ag- gregation of neighbors' features, which enables specifying different weights to different neigh-
14

---

Under review as a conference paper at ICLR 2023
bors Veličković al. (2018).
The aggregation strategy is defined as:
h
ma W W h
VuEte)uN(())
m==1
where o² is the ELU Clevert et al. of m-th head in the k-th layer, and
2016) function, Wonk is the linear transformation weight matrix
exp (o² (a
[W h h W -- 1D)
mi O
Σ (ee(()uN())) exp (o² (amk Wmk h.
[ wmmmmpp h ID)
where a² is the LeakyReLU Maas et al. 2013) function and a am is the weight vector of the mi-th head the k-th layer.
GIN. Graph Isomrrphism Network, abbreviated as GIN, captures graph structure differences by summing neighbors' features, which provably maps any two graphs that the Leisae test Leman & Weisfeiler (1968) of isomorphism decides as non-isomorphic, to different embed- dings Xu et al. (2019). The updating strategy is shown below
h - MLPx (L+ 1
Σ
h
YuEN(()
where MLP* is a multi-layer perceptrons Hornik et al. (1989) for the k-th layer, and c² can be a learnable parameter or fixed scalar in the k-th layer.
C REPARAMETERIZATIN TRICK
Following (Maddison et al. 2017), denoting mn Ei= MLP; (enc)G, ; AA) in Equation by Suc[G,> in the training stage, the probability of the edge (u, 5) being part of E is given by
gtrain = log(e) log(1 E) + Sus[G,
where S(x) I+e and U(0,, 1) an independent random variable that obeys a standard uniform distribution. À is the temperature parameter control the approximation. When - O, uc|G, is binarized with
exp (Suc)G.) + exp (Sus)G.)
lim P (otreir. =1) T-30
D REAL RUNTIME EVALUATION
The real runtime of model on dataset is calculated by averaging the minimum one-epoch runtime of the three base GNNs all settings on the dataset (As shown Table 4). Note that all the experi- ments were conducted on the same platform described in Appendix E.3 It's seen that G-CENSOR actually achieves a very fast speed compared to other baselines. This should be attributed to the two reasons mentioned before, i.e., simple edge probability estimator and no need to contrast in-batch samples.
E EXPERIMENTAL SETUP
DATASETS
In the five citation networks. i.e., Cora_Full, CoraMML, CiteSeer, DBLP and PubMed, nodes rep- resent papers and edges represent citation links. Given paper text as bfg-of-wordd node features, the task to predict the topic of paper. In the two product networks, i.e., Computers and Photo,
15

---

Under review as a conference paper at ICLR 2023
Table 4: Real runtime (seconds per epoch) of all the methods on all the datasets.
Dataset
BASE
MVGRL
GRACE
GCA
BGRL
RGCL
AUTOGCL
G-CENSOR
CiteSeer
2.0420.09
2.5020.41
1,7520.10
3.05=0.16
2.80&0.557
1.7120.07
2.0320.13
2.61:0.52
Computers
5.02:0.11
0OM
13.1342.64 64
10.35s0.19
12.16:0.23
53.4112.15
8.16:0.99
9.95a0.22
Com Full
8.41=0.11
0OM
29.4346.99
22.80=4.71
31.25x11.73
31,6+0.47
12.6+1.17
12.31+0.94
Cora ML
2.10
3.25+0.12
2,60#0.46
2.87+0.74
2.71 16
2.12+0.06
2.05+0.07
2.36#0.54
DBLP
3.6620.17
23.3620.53
9.67=0.70
16.17:5.19
9.22+0.55
11.15=0.57
5.0420.15
4.92:0.85
Flickr
9.8120.59
> 10000
28.3543.13
46.36215.96
59.12+36.26
110.66:15.47
17.91=1000
14.332.43
15.35+2.49
PubMed
3.13+0.15
27.12+1.32
8,20+1.23
7.37+1.59
8.10+1.22
10.02+0.19
4,49+0.26
4.56+0042
nodes represent products and edges represent that two goods are frequently bought together. Given product reviews as bag-of-wor of-words node features, the task map goods to their respective product category. In the image network, i.e., Flickr, nodes represent images and edges represent that two images share some common properties (e.g., same geographic location and comments by the same user, etc.). Given bag-of-word representation of the images as node features, the task is to predict the type of an image.
Table Datasets statisties
Dataset
#bodes
#odges
#fcatures
#classes
Cora Full
19.793
126,842
8,710
70
CL
2,995
16,316
2,879
7
CiteScer
4,230
DBLP
17.716
105,734
1.639
4
PubMed
Computers
13,752
491,722
767
10
Photo
7,650
238, 162
745
8
Flickr
89,250
899,756
500
7
E.2 BASELINES
Base models, i.e., SAGE, GAT and GIN, refer to Appendix
GCL methods with uniform or adaptive data augmentation:
1. DVGRL(Hassang Ahmadi 2020): Multi View Graph Representation Learning, an ap- proach contrasting encodings from first-order neighbors and general graph diffusion and also contrasting node and graph encodings across views.
2. Grace(Zhu al., 2020): GRAph Contrastive rEpresentation learning, approach gen- erating two graph views by corruption and learn node representation by maximizing the agreement node representations these two views.
3. GCh et al., 2021): Graph Contrastive representation learning with Adaptive augmen- tation, an approach designing augmentation scheme based on node centrality measures to highlight important connective structures.
4. (hakoor et al., 2021): Bootstrapped Graph Latents, graph representation learn- ing method that learns by predicting alternative augmentations of the input. BGRL uses only simple augmentations and alleviates the need for contrasting with negative examples, and thus scalable by design.
GCL methods with learnable data augmentation:
1. RGCL(Li et al. 2022): Rationale-aware Graph Contrastive Learning, unsupervised approach using a rationale generator to reveal salient structures about graph instance- discrimination as the rationale, and then creating rationale-aware views for contrastive learning. Note that this method, designed for graph property prediction tasks, integrates the views generation module and the inference flow of the predictor. Therefore, gard node property prediction tasks as ego ggo-graph property prediction tasks to adapt to this method.
16

---

Under review as a conference paper at ICLR 2023
2. AutoGCL(Yinner )022): Automated Graph Contrastive Learning, approach employ- ing a set of learnable graph view generators orchestrated by an auto augmentation strategy, where every graph view generator learns probability distribution of graphs conditioned by the input. This method is proposed for graph property prediction tasks. However, it can be directly transferred to node property tasks since its views generator and task predictor are separates.
Note that while our work can be easily enhanced by considering node feature transformation in views generation, we focus structure transformation in this work, thus feature transformation is disabled in all models including G-CENSOR.
E.3 IMPLEMENTATION DETAILS
Experiments were conducted on a Ubuntu 18.04 server with one Nvidia Tesla V100-32G GPU, And the code was implemented using python 3.8 with PyG 2.0.4 and Pytorch 1.11 1 that used CUDA version 1.3. For all datasets, the number of sampled neighbors was set to 64. The batch size was set to 64 for all models and an AdamW optimizer (Loshchilov & Hutter 2019) with learning rate 0.01 was used to train all models.
For fair comparison, the number of layers of base GNNs was set to 2 for all baselines and all contrastive baselines were used as auxiliary task (Xie et al., 2022). The weight of the contrastive loss was searched from 0.1 to 0.9. Moreover, for all basclines with hyperparameters of edge drop probabilities and temperature, we searched edge drop probabilities over [0.1, 0.2, 0.3, 0.4] and searched temperatures over [0.1, 0.2], unless the original paper reported the best choices
the datasets. As for G-CENSOR, m Equation 4 was simply set to 4, & in Equation 9 was searched
from le-5 to le-1, 1, garginn in Equation was searched over [0.5, 0.1], in Equation TO was searched
over [0.05, 0, 0.15, 0.2), and (3 in Equation 11 was searched from 0.1 to 0.5.
F SENSITIVITY ANALYSIS
This section displays the sensitivity analysis on hyperparameters for GraphSAGE and GIN,
17

---

Under review as a conference paper at ICLR 2023
Chiden 14
M
Oiz 197 1
Jazz Pab
falt Palie 14.4
6-03:00011
8=25m002
f=12m002
BeLicreo 25
deLOrw32
(a) Sensitivity hyperparameter and 8 on all Lid split datasets with GraphSAGE.
20221002
$=1,2rw01
Omt 0o
(b) Sensitivity w.r. hyperparameter and on all o.o.d split datasets with GraphSAGE.
fw1.1r011
Aet de
(c) Sensitivity w.r.t hyperparameter and 3 on all i.dd split datasets with GIN.
bote
6-05:7-2.11
f-147-01
6-1.5
del GLL07
(d) Sensitivity w.r.t hyperparameter and 3 on all o.o.d split datasets with GIN.
Figure 7: Sensitivity analysis on hyperparameters with GraphSAGE and GIN.
18


Reviews:
[{"review_id": "UdFo792lyV", "reviewer": "Reviewer_DcTY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of generating optimal task-oriented contrastive views in graph contrastive learning (GCL). Existing GCL methods find this problem challenging when the task-related semantics are incomplete in positive/negative views.  For this problem, the authors propose G-CENSOR, which is a model-agnostic method to generate optimal task-oriented counterfactual views for graphs and GNNs in node property prediction. Experiments are conducted on eight real-world datasets, and the results show that the proposed method outperforms state-of-the-art GCL methods in task performance and generalizability.", "summary_of_the_review": "Overall, the studied problem of this paper is important. The proposed method is novel and sound generally. The experiments include extensive evaluations on benchmark datasets, and the results look encouraging. But my major concern is that the key assumption and corresponding technique design need further justification. That is the main reason I give a weak reject now.", "strength_and_weaknesses": "# Strength\n- The studied problem of optimal task-oriented contrastive view generation is important. \n- The paper is writing in a clear way and easy to follow.\n- The authors provide extensive experiments on benchmark datasets.\n\n# Weakness\n- Assumption 1 assumes that the prediction loss based on $\\phi(G^-)$ should be higher than $\\phi(G^+)$. However, as $G^-$ often has spurious correlations with the label y, the prediction based on $\\phi(G^-)$ is possible to be even more accurate than $\\phi(G^+)$. How can we verify this assumption?\n- Similarly, in the discussion about the difference between counterfactual explanation and this work, the authors mentioned that compared with directing maximizing $\\mathcal{L}_{pred}(\\phi^-)$, the rank loss used in Eq. (7) can mitigate the spurious correlation problem. However, I feel this claim is not convincing enough, and hope the authors can further justify it.\n- The counterfactual transformation module estimates the probability of each edge to be part of the optimal positive view. I wonder whether an edge-level prediction is good enough for view generation,  especially when more higher-level substructure (e.g., triangle) are involved.\n- In Definition 2, I suggest the authors to double check the notation. It seems $\\bar{G}’$ there should be $G’$.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KJEMUpKrTR", "reviewer": "Reviewer_EBjy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a supervised learning framework that utilizes the philosophy of counterfactual learning to enhance the node classification performance. Specifically, this paper utilizes a counterfactual transformation module to construct sub-ego-graphs that are positively correlated to the label information, and explores the complements of these sub-ego-graphs as negative counterparts for contrastive learning, which are negatively correlated with the label information (or not as positively correlated as the former according to the formulation in this paper). The learning framework explores five objectives to simultaneously fulfill the purposes of supervised learning, counterfactual ego-graph construction, and sparsity guarantees of the ego-graphs. The proposed framework improves the performance of the node classification tasks when applied to vanilla GNN backbones.  \n", "summary_of_the_review": "Utilizing the concept from counterfactual learning into the graph machine learning field is interesting and well-motivated. However, several formulations and implementations seem to come from the existing works. Besides, ablation studies on multiple design choices are missing, which made it difficult to judge the research contribution of this work. \n\nTherefore, I think this paper requires substantial revision, and I recommend rejection.\n\n\n===\n\npost rebuttal: I appreciate the detailed response. After reading all the reviews and responses, I decide to keep my score.", "strength_and_weaknesses": "S1:\nThe idea of improving GNNs’ performance on node classification tasks by generating counterfactual examples is an interesting and well-motivated.\n\nW1:\nIt seems like representation of an edge (i.e., Section 4.2) involves some engineering tricks, so ablation studies are needed upon each of these tricks to demonstrate that the actual performance improvements are most from the proposed high-level philosophies. \n\nW2:\nThe high-level idea of the counterfactual transformation module (CTM) is very similar to the augmentation module proposed in ADGCL [1], which may give the readers a feeling that the proposed method re-packages the idea in ADGCL with the concepts of counterfactual learning. I think the authors should discuss [1] in this paper.\n\nW3:\nThe proposed framework involves many sub-modules and objectives, which makes it hard to judge what are the core contribution of the proposed method. I would suggest the authors to conduct extensive ablation studies to demonstrate and justify their design choices. \n\nW4: \n[2] also studied counterfactual data augmentation on graphs. The authors should also include discussion of it.\n\nW5:\nThe experimental setting is not clearly described. According to my own implementation based on DGL, GIN underperforms badly on node-level tasks (e.g., almost 10% worse than vanilla GNNs on benchmarks with public splits), which does not align with the results from this paper. Besides, RGCL and AutoGCL are targeted for the graph classification task and the authors does not specify how they transfer these framework into the settings for the node classification. \n\n[1] Adversarial graph augmentation to improve graph contrastive learning. NeurIPS 21. \\\n[2] Learning from counterfactual links for link prediction. ICML 22.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "G8IWj1PXFt", "reviewer": "Reviewer_VhaN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents graph contrastive learning approach with counterfactual views. Specifically, the authors propose a counterfactual inference objective to generate task-oriented causal views that will be helpful in determining positive and negative samples. Experimental results show superiority over existing graph contrastive learning approaches.", "summary_of_the_review": "As a summary, this paper presents an interesting approach to improve graph contrastive learning by constructing causally correlated positive and negative views. However, it seems that the empirical evaluation is weak and several claims need to be further elaborated or supported by more experimental results. Therefore at this moment I recommend rejection.", "strength_and_weaknesses": "Strengths:\n+ It is a more principled way to construct contrastive views based on counterfactual inference.\n+ The idea is clear and easy to follow.\n\nWeaknesses:\n- Several important baselines should be compared:\n  1. Regarding causal graph contrastive learning: Generating Counterfactual Hard Negative Samples for Graph Contrastive Learning, arXiv 2207.00148\n  2. Regarding learnable view generation: Adversarial Graph Augmentation to Improve Graph Contrastive Learning, NeurIPS 2021\n  3. Regarding task-oriented graph augmentation for contrastive learning: Augmentations in Graph Contrastive Learning: Current Methodological Flaws & Towards Better Practices, WWW 2021\n- Evaluation is not fair as the training stage considers downstream task information. It should be noted that all considered baselines have separated training and test stages. More specifically, their models are trained with unsupervised contrastive objectives at first and then evaluated with another prediction head. In this work, the training stage considers supervision signals, which renders performance comparison unfair. I strongly encourage authors to consider supervised contrastive learning setting (Supervised Contrastive Learning, NeurIPS 2020) for all baselines.\n- Performance improvements are not significant. I note the authors claim their work does not contrast in-batch negative samples, which improves memory efficiency. I would like to see more experiments on large-scale datasets to support this claim, e.g., ogbn-arXiv.\n- The setting for the generalizability experiment is not clear to me. In particular, why clustering size is set to 2 is kind of tricky. Other common ways of splitting datasets can be considered, for example scaffold split in molecular graphs.\n- The authors are encouraged to elaborate on the choice of graph encoders, as in Assumption 1 the graph encoders should be a sufficient encoder. What kind of encoders are considered as sufficient encoders? It feels to me that GIN will result in information loss when applying to node-level tasks.\n\nMinor points:\n- Page 1, a appropriate positive view -> an appropriate positive view\n- Tables 2 and 3, GGA -> GCA\n- Definitions 1 and 2, \"causally correlated\" should be specified in mathematical form to avoid ambiguity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FDhxX0wp-TE", "reviewer": "Reviewer_tYMD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a supervised framework -- G-CENSOR, which utilizes an auxiliary model (i.e., CTM) to estimate the semantic importance of each edge in an ego-graph and transfer them into sampling probabilities to construct positive and negative views. Further, the model is optimized by minimizing three sub-losses: (1) prediction loss i.e., conventional supervised loss, (2) counterfactual loss, which gives the positive view the same supervision signal as the original sample and enlarges the rank gap between positive and negative views, and (3) contrastive loss, an infoNCE style loss with comparison among anchor sample and its augmented views. The experiments results show the effectiveness of G-CENSOR on network datasets.", "summary_of_the_review": "The authors proposed G-CENSOR, a model-agnostic supervised framework on node-level task with the assistance of contrastive learning. And extensive experiment show the effectiveness of it. However, the tech novelty is rather limited and it lacks empirical or theoretical results to demonstrate the effectiveness of the core module -- CTM.   ", "strength_and_weaknesses": "Strength\n(1) Overall, this paper is well-written and the idea is easy to follow.\n(2) Far as I know, this paper is the first to study the construction of semantics-preserved augmented views for GCL on node-level tasks.\n(3) The results showcase the effectiveness of G-CENSOR.\n\nWeakness:\n(1) What is the difference between counterfactual view and augmented view? Please see following Quality for detailed comments.\n(2) Limited technical novelty. Please see following Novelty for detailed comments.\n(3) Lack of empirical validation of CTM. CTM is supposed to discover “edges causally related to the task label”. However, no empirical or theoretical evidence is presented to validate this statement. Concretely speaking, I would prefer to see an ego-graph and CTM’s estimation of each edge’s sampling probability to validate its ability of revealing “edges causally related to the task label”.    \n(4) Miscellaneous minor issues. Several misspelling (a -> an Line 8 Para 2 Page 1) and citation format problem (\\cite or \\citep command).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3nMAOm4tIf", "reviewer": "Reviewer_iyAN", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To answer a pre-posed question about the contrastive views of graph representation learning, this paper studies task-oriented contrastive learning from a counterfactual perspective in node property prediction tasks. A model-agnostic framework, G-Censor was proposed to generate both positive and negative views in graph contrastive learning, the performance of which is evaluated on real-world datasets. Compared with prior methods, the model proposed in this paper has the characteristic to generate learnable views, especially negative views in specific tasks.", "summary_of_the_review": "This paper has a novel way of thinking and perspective on graph contrastive learning in specific tasks, and the formulation of the problem is also very detailed. However, the novelty is limited, and the experiment results are not that convincing compared with the previous work. Hence, I am leaning on the negative side.", "strength_and_weaknesses": "**Strength**\n1. The problem formulation of graph contrastive learning is quite detailed. And there is also a discussion of the causality of the proposed method.\n2. The idea of automatic generation of negative views is practical to get rid of the performance-memory trade-off inherent.\n3. The explanation of each part of the model is clear and the core idea is easy to follow.\n\n**Weakness**\n1. The way in which negative views are generated is sloppy and lacks detailed discussion. More ablation experiments may help explain the trade-off.\n2. In the absence of released code, the experiment details in the task shift of prior methods from graph classification to node classification need to be further explained due to significant discrepancies in results.\n3. The figures and captions in this article are unclear with rough details. Task B in Figure 1 lacks an explanation of how the prediction works. The estimation in Figure (b) is not intuitive.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject