Disentangled Contrastive Learning on Graphs
Authors: Haoyang Li
Xin Wang
Ziwei Zhang
Zehuan Yuan
Hang Li
Wenwu Zhu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Neural Network
Contrastive Learning
Self-supervised Learning
Disentangled Representation Learning

Abstract: Recently, self-supervised learning for graph neural networks (GNNs) has attracted considerable attention because of their notable successes in learning the representation of graph-structure data. However, the formation of a real-world graph typically arises from the highly complex interaction of many latent factors. The existing self-supervised learning methods for GNNs are inherently holistic and neglect the entanglement of the latent factors, resulting in the learned representations suboptimal for downstream tasks and difficult to be interpreted. Learning disentangled graph representations with self-supervised learning poses great challenges and remains largely ignored by the existing literature. In this paper, we introduce the Disentangled Graph Contrastive Learning (DGCL) method, which is able to learn disentangled graph-level representations with self-supervision. In particular, we first identify the latent factors of the input graph and derive its factorized representations. Each of the factorized representations describes a latent and disentangled aspect pertinent to a specific latent factor of the graph. Then we propose a novel factor-wise discrimination objective in a contrastive learning manner, which can force the factorized representations to independently reflect the expressive information from different latent factors. Extensive experiments on both synthetic and real-world datasets demonstrate the superiority of our method against several state-of-the-art baselines.

Disentangled Contrastive Learning on Graphs
Haoyang Li', Xin Wang': Ziwei Zhang', Zehuan Yuan', Hang Li,, Wenwu Zhu
'Tsinghua University, *Bytedance
lihy18@mails. www.:mmmmils..tinnuuua.aa. edu. en, wang, zazhang)@tainghua. edu. (yuanzehuan, lihang. @bytedanee com, wuzhußtsinghua. edua.edddde...... cn
Abstract
Recently, self-supervised learning for graph neural networks (GNNs) has attracted considerable attention because of their notable successes in learning the repre- sentation of graph-structure data. However, the formation of real-world graph typically arises from the highly complex interaction many latent factors. The existing self-supervised learning methods for GNNs are inherently holistic and neglect the entanglement of the latent factors, resulting the learned representa- tions suboptimal for downstream tasks and difficult to be interpreted. Learning disentangled graph representations with self-superviedd learning poses great chal- lenges and remains largely ignored by the existing literature. In this paper, we introduce the Disentangled Graph Contrastive Learning (DGCL) method, which is able learn disentangled graph-leve representations with self self-superviiion. In particular, we first identify the latent factors of the input graph and derive its fac- torized representations. Each of the factorized representations describes a latent and disentangled aspect pertinent to a specific latent factor of the graph. Then we propose novel factor-wise discrimination objective in contrastive learning manner, which can force the factorized representations to independently reflect the expressive information from different latent factors. Extensive experiments both synthetic and real-world datasets demonstrate the superiority of our method against several state-of-the-art baselines.
Introduction
Graph structured data is ubiquitous in the real world, e.g., social networks, biology networks, traffic networks, etc. Recently, graph neural networks (GNNs) have become increasingly prevalent in learning graph representations in supervised manner, demonstrating their strength wide variety of research fields M 3 4]. GNNs require task-dependent annotated labels to learn effective unavailable in practice, thus motivating the
representations, which are extremely scarce, advent of self-supervised graph representation learning.
Contrastive learning, discriminative approach pulling similar samples close and pushing dissimilar samples far away, has become a dominant strategy in self-supervised graph representation learning (5) 8 9 10.101 II.. Despite their notable successes, the existing graph contrastive learning methods generally adopt holistic scheme, i.c., the learned representations characterize graphs as perceptual whole, ignoring the nuances between different aspects of the graph. In fact. the formation of graph typically follows a relational process in the real world, driven by many complex latent factors. For example, in social networks, social group may have several communities originated from different relations (e.g., friends, colleagues, etc.) or interests (e.g., sports, games, etc.) 112. And a molecular graph may consist of various groups of atoms and bonds representing different functional units [13 The complex relations among the multiple latent factors bring an urge for disentangling these factors
"Corresponding authors
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

in contrastive graph representation learning, which remains unexplored by the existing holistic works. As result, the graph representations learned by the existing methods contain mixture of entangled factors, harming interpretability and leading to suboptimal performance for predictive tasks involving whole graph representations.
In this paper, we propose to learn disentangled contrastive graph representation, for the first time. Although disentangled representation learning, which aims to characterize the various underlying explanatory factors behind the observed data in different parts of the factorized representations [4]
15 has been demonstrated to be more explainable [2] and generalizable [14], disentangled graph contrastive learning faces the following two challenges. (1) Tailored graph encoder for disentangled contrastive learning. The graph encoder should be carefully designed so that it can be sufficiently expressive to infer the disentangled latent factors in the graph. (2) Tailored discrimination tasks designed for disentangled graph contrastive learning- Since lask-dependent labels are not available in the self-supervised setting, disentangled graph contrastive learning can only utilize the limited amount of self-superviiion information. This implies that the discrimination tasks should be well-designed for disentangled contrastive representation learning on graphs.
To tackle these challenges, we propose a novel disentangled graph contrastive learning model (DGCL) capable of disentangled contrastive learning on graphs. In particular, we first design disentangled passaggg layer. Each channel is
graph encoder whose key ingredient is a multi multi-channel tailored to aggregate features only from one disentangled latent factor. Then a separate readout operation in each channel summarizes the specific aspect of the graph according to the corresponding latent factor, so as to produce the disentangled graph representation. Next, we conduct contrastive learning in each representation subspace characterized by each factor independently instead of in the whole representation space. This novel factor-wise contrastive approach can ensure that each disentangled factor of the vectorized representations is sufficiently discriminative only under one specific aspect of the whole graph. Thus the representations are encouraged to be disentangled and best characterize the aspect pertinent to a latent factor of the graph. Compared with the existing methods, our proposed DGCL model encodes graph with multiple disentangled representations, making it possible to explore the meaning of each channel, which benefits in more explainability for producing graph representations.
We conduct extensive experiments on both synthetic graph dataset and empirical well-known graph benchmarks. The results show that the representations learned from DGCL, can achieve substantial performance gains on the downstream graph classification task compared with various state-of-the-a baselines.
The contributions of this paper are summarized follows:
We propose disentangled graph contrastive learning model (DGCL), which is able to learn disentangled graph representation via factor-wise contrastive learning. To the best our knowledge, we are the first to study disentangled self-super graph representation learning.
We propose a disentangled graph encoder capture multiple aspects of graphs through learning disentangled latent factors on graphs. We further present the factor-wise contrastive learning approach tailored discrinination tasks in terms of each latent factor independently.
We conduct extensive experiments to verify the efficacy of our proposed model for the graph classification task. The results on several graph classification datasets demonstrate that DGCL achieves state-of-the-ee performance by significantly outperforming the baselines.
We introduce the problem formulation and preliminaries in Section 2 In Section B] describe the details of our proposed method. Section and qualitative comparisons. We review the related work in Section 5 in Section 6
4 presents the experimental results, including quantitative Finally, we conclude our work
2 Problem Formulation and Preliminaries
2.1 Problem Formulation
Let = (G.) be a graph dataset with N graphs. The key of most self-supervised graph representation learning methods, including ours, is to derive a graph encoder f()), which outputs

---

Input Graph Gj
Disentangled Graph Encoder fo()
Factor-wise Contrastive Learning
hack
Ho
Message Passing
MLP
Channel
Disentangled Representationz,
Channel 2
34,1
74,2
2
G
Channel 3
Objective
Graph
Augmentations
Shared Encoder
Feedback
Maximize Agreement Under Each Factor
MLP
RA- AA
2
21
21,3
Message Passing
Disentangled Representations
Extract features specific to each factor
Figure 1: The framework of DGCL Model. (1) The input graph G; undergoes graph augmentations to produce G",, and both of them fed into the shared disentangled graph encoder fo()). (2) In the encoder fo(.), the node features HD are first aggregated by L ppassage-passing layers and then taken the input of multi-channel mressage-passingg layer. (3) Based on the disentangled graph representation zi, the factor-wise contrastive learning aims to maximize the agreement under each latent factor and provide feedback for the encoder improve the disentanglement. This example assumes that there are three latent factors, hence the three channels.
d-dimensional representation z; = f(;)) E Rdd for each input graph, such that Z - M best describes G. In this work, we aim to learn multi-channel graph encoder fo(-) with parameters so that the output can be disentangled representation, i.e. fo(-) [ (.)] where K is the number of channels. To be specific, z; expected to composed of K independent components,
ie.. [z,.1, 2,.,..... k.. where Z fe (G.) R k [1. Ad = d/K, assuming that there are K latent factors behind the graph instances to be disentangled. The b component factor & accurately. We also assume
z is for characterizing the aspect of G; that is pertinent that the value of Z,,k will be merely a white noise vector if the input graph G, does not contain any information of factor k. Note that we focus primarily on undirected graphs in our method, although can be straightforwardly extended to directed graphs.
2.2 Preliminaries on Contrastive Learning
Unlike generative models, contrastive learning is an instance-wise discriminative approach that aims at making similar instances closer and dissimilar instances far from each other representation space [16] [7 It treats each instance in the dataset as a distinct class of its own and trains classifier to distinguish between individual instance classes I8 19]. Given a dataset
{a; J each instance is assigned with unique surrogate label yı since ground-truth labels are given. y, often regarded as the ID of the instance in the dataset, i.c., Ys = So the probability classifier is defined as:
Po(vilz;) = exp Φ(Vi, vy,))
(1)
Σn exp ø( (o(vi, v/g3)
where 0 denotes the parameters of the encoder, Both v, and Vyy are the embeddings from T; which are generated from two different encoders [20], or a shared encoder [21). Before being passed into the encoder, the input x; could undergo data augmentations (2I) which play critical role in defining effective predictive tasks for learning the encoder. é is the similarity function, often adopting cosine similarity with temperature [22]. i.e., Φ(v;, Vyz, Vyy v, assuming the embeddings are (e-normalized. Then the learning objective is to maximize the joint probability
H p(u((x)) over the dataset, namely minimize the negative log-likelibhe likelihood function ΣN l, if let l, = llog p(y)

---

Note that loss E could be NCE loss [18, InfoNCE loss 23 or NT-Xem loss (2I. The encoder will be encouraged learn representation space where samples (e.g.. augmented data) from the same instance (e.g., an image, graph) are pulled closer and samples from different instances are pushed apart [6. For convenience, we follow the settings above in this work.
Disentangled Graph Contrastive Learning
In this section, we present the proposed DGCL model. The framework of DGCL is shown in Figure [ In Section 3.1 we introduce the disentangled graph encoder to identify the complex latent factors and capture multiple aspects of graphs. Then in Section 3.2 propose a factor-wise contrastive learning approach to conduct instance discrimination under each latent factor independently. The objective and the details of the optimization are derived in Section 3.3 and Section 3.4 respectively.
3.1 Disentangled Graph Encoder
The key of the disentangled graph encoder to produce the factorized graph representation z; = [z,,1, 2i, Zi.k) ] for each input graph G2 E G. Based the factorized representation, can infer the related latent factors behind the graph.
Generally, GNNs use the graph structure and node features to learn the representation vector h, of each node o with message-passing mechanism, i.e., iteratively updating the representation of a node by aggregating representations of its neighbors. The propagation of the 4th layer is formulated as:
h COMBINE' (h-, AGGREGATEE ((h E (u))))),
where hy is the representation of node the tth layer and h initialized with node features. N(v) is the neighborhood to node v. We use the term GNN to indicate the message-passing layer in Eq. (2)
Let H' = (h'le v E } be the node embeddings after the tth GNN, where V denotes the node set of the graph. After applying L traditional message-passing layers, we propose graph-disentanglemen layer to learn the disentangled representations. The goal is to extract features specific to each latent factor with a separate channel. Specifically, we adopt K separate channels identify the complex heterogeneous latent factors and capture multiple aspects of the input graph. For each channel, we first utilize a GNN; to propagate information with its own parameters: H 4-1-1 ( where A is the adjacency matrix of the input graph. H is the node embeddings which is only pertinent to the thh latent factor. Then the READOUT function (i.c., pooling function) of each channel used to summarize all the obtained node representations into a fixed-length graph-level representation: hG.. = READOUT, ( Finally, each channel outputs the factorized graph representation with a separate MIP: 2 = MLP (hG....
Compared with the existing graph encoders that are inherently holistic, our disentangled graph encoder consists of K channels, rending the possibility to identify the complex heterogencous latent factors and capture multiple aspects of graphs.
3.2 Disentangled Factor-wise Contrastive Learning
Unlike the existing contrastive learning methods, DGCL designs a novel factor-wise instance dis- criminative task and learns to solve this task under each latent factor independently. This design only makes similar samples closer and dissimilar samples far from each other in the representation space, but also encourages the learned representation to incorporate factor-level information for disentangiement.
Specifically, we assume that the formation of real-world graphs is usually driven by multiple latent heterogeneous factors. So the instance discriminative task should be represented as the expectation of several subtasks under the latent factors:
(e( (HIG..) [po (G,, k)].
Here pa (k)),) is the probability distribution over latent factors for the input graph G,. po po k. denotes the instance discrimination subtask under the th latent factor.

---

Firstly, given the representation of G; derived from the disentangled graph encoder fe(-), we present a prototype-basee method to obtain pa (b]G.), We introduce K latent factor prototypes [c*) l and the probability of the 4th latent factor reflected in G, is parameterized as:
exp (ZZ,, Ci))
po (o(k]
(4)
Σ exp ((zi, k C&)
where á the cosine similarity with temperature T, i.c. ((a, b) COSINE(a, b)/7 and COSINE(a, b) a b/ b/( [|bll])-
Then, we define the instance discrimination subtask under the 1 latent factor as:
exp Φ (Z (.),k)
(5)
=j= exp Φ k, Z (,
where 2 and Z are the disentangled representations produced by the shared graph encoder, and Ys is the unique surrogate label (see Section 2 of the graph G.. In our method, we follow [T8] 19 to implement Y/ as the II) of the graph in the dataset, i.e., Vi i. For notation convenience, we do not distinguish yi and hereafter when there no risk of confusion.
Next, we describe the process to get 2i i.c. Zy in Eq. (5). First, the input graph G; undergoes graph data augmentations to obtain its correlated views G and they form positive pair. Data augmentation expected to create novel and realistically rational data by applying certain transforma- tions that do not affect the label, and plays critical role in defining effective predictive tasks [1 We follow [11 to adopt four types of graph augmentation strategies, including node dropping, edge perturbation, attribute masking, and subgraph sampling. More details of graph augmentations can be found supplementary material. Then, the augmented graph G₁ also fed into the shared disentangled graph encoder fa()) to produce z i.e. Zyy,,'' Given the disentangled representations k and zi,x of the G, and G¹, respectively, conduct factor-wise contrastive learning for each latent factor independently as Eq. (5)
3.3 Evidence Lower Bound (ELBO)
We present the objective of our method. Following the existing methods 18 19 we aim to maximize the joint probability I (G, over the graph dataset G = { We learn the model parameters 8 by maximizing the ligeliidhhhod::
0" arg max
log po(y:|G,) arg max Σ log Epp(K(..) (p,
(6)
However, directly maximizing the likeliihddoodd function is difficult because of the latent factors. Therefore, we instead optimize the evidence lower bound (ELBO) of the log-likelihood function given by Theorem
[
See the supplementary material for the proof.
Theorem 1. The log likelihood function of each graph log (y G+) is lower bounded by the ELBO: i( [ s.)llog [log Pe (yall G2 k)] Dx (4o (k|G,, Ms) (k))).
To make the ELBO as tight as possible, we require that go g (k)G,, Mt) is close to Pa (k.], Mi). whose detailed implementations are provided the next subsection (see Eq. and Eq. 7 In the ELBO C(0, i), pe (3/ G., k) and pe(k have been introduced in Eq. (5) and Eq. 44h respectively, and ge go(k] G.,, Mr) is variational distribution to infer the posterior distribution of the latent factors after observing both G; and its correlated view Gyy-
3.4 Optimization
We introduce variational distribution ( Gt, y6) to infer the posterior probability Po(k)G, y:) that is defined with Bayes' theorem as follows:
(k G. Go G.,
pe (k) =
Σ po (A/ (ol)
5

---

Pe y:) is the probability of the kth latent factor pertinent to both G; and the augmented G simultaneously. Compared with the prior distribution po (k]G.) in Eq. 4. Po Po yi) incorporates more useful information (i.e., factor-wise similarity) from po (yi Gt k). Although both po(k) and po(k G., Vs) are designed to infer the latent factor distribution, po (k:]., is calculated only given the graph Gj, but po G4 Dt) is calculated after observing Gü the augmented version G and their similarities under the specific latent factor.
However, we cannot compute the posterior probability tractably because of the term pa (1i Gi, k). If we directly calculate po po(y: k) according to Eq. (5). all the instances in the dataset G are needed for computing the denominator in Eq. (5), which could be computationally prohibitive 24 I8 [9].
To tackle this obstacle, several strategies are proposed in the literature, including memory bank
[8 20]. dynamic dictionary (251, NT-Xent loss [2I]. Here, we adopt NT-Xent loss on a minibatch
B G. So in practice, the instance discrimination under each latent factor calculated by:
exp (K,,,,,
Po
G1, Po
(8)
Σ E j exp o(z,,k 2y,x)
We approximate the posterior probability po ( wi) with a variational distribution defined as:
po (A) (V,
qo (k(k M))
Σ Po (k Go kt)
Finally, we seek to learn the parameters @ of the disentangled graph encoder. More specifically, we calculate go Gt, ys) and maximize the ELBO over a mini-batch B using mini-batch gradient ascent:
C B) = Σ 1( IEB
(10)
Encourage disentanglement. Note that our objective and its optimization can inherently encourage disentanglement without adding extra regularization term (e.g., minimizing mutual information). The reason is that factorizing the instance discrimination into K factor-wise subtasks will enforce the independence of the learned graph representation z, Besides, qo(k) Gr, y:) is computed based on ht and other K - latent factors. Thus, the graph encoder is forced to preserve exclusive information in each channel to get more accurate approximation to the posterior, a tighter ELBO expected. The strong inductive biases in DGCL encourage to learn disentangled graph representations that match the ground truth factors behind the graphs.
Experiments
We empirically evaluate our proposed method through experiments, and analyze its behavior on synthetic graph dataset to gain deeper insight. Ablation studies are conducted to show the effectiveness of the proposed method. We provide more discussions in supplementary material, including the complexity of our method, the impact of the hyper-parameters, etc.
4.1 Experimental Setup
Datasets. To demonstrate the advantages of our method, we conduct experiments on nine well- known graph classification datasets including four bioinformatics datasets, i.e., MUTAG, PTC-MR, NCII, PROTEINS, and five social network datasets, i.e., COLLAB, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, and REDDIT-MULTT-SK We also adopt a larger graph dataset egbg-molhin from Open Graph BenchMark (OGB) 26 More details are provided in supplementary material.
Baselines. We compare DGCL with the following two groups of basclines. One group of baselines are graph kernels including Shortest Path Kernel (SP) 227. Graphlet Kernel (GK) (28], Weisfeiler- Lehman Sub-tree Kernel (WL) (29] Deep Graph Kernels (DGK) 33), and Multi-Scale Laplacian Kernel (MLG) (31] The other group of baselines are classical unsupervised graph representation learning methods including node2vec [32], sub2vec [33]. graph2vec [34), GVAE [35], and more
recent contrastive graph representation learning methods including InfoGraph [. GCC 10,, MVGRL
9. and GraphCL. [II].
Evaluation. verify the effectiveness our method, we follow the common evaluation protocols in the existing literature [34] 5 30 where graph embeddings are generated from each method

---

Table 1: Graph classification accuracy (%) of DGCL and baselines. In each column, the boldfaced score denotes the best result and the underlined score represents the second-best result. "_" indicates the result is not reported in the paper.
MUTAG
PTC-MR
PROTEINS
NCH1
IMDB-B
IMDB-M
RIYT-B
RIT--MKK
COLLAB
SP
85.242.
58.24244
75.1
73.0+0.22
55.6=0.2
38.0+0.3
64.1+0.1
39.6+0.22
GK
81.742.11
57.3
71.720.6
62.34..3
65.94100
43.910.4
77.3+0.2
41.01022
72.8=0.3
80.743.0
58.0+0.55
72.9+06
80.0+0.5
72.3+34
47.0+055
68.8+0.4
46.14002
DGK
87.422277
60.142..6
73.3=0.8
80.3:00.5
67.0=00.6
44.62:0.5
78.040.4
41.34022
73.1=0.3
MLG
87.9+1166
63.3+11.5
76.14200
80.+113
66.6:10.3
41.2+000
node2vec
72.6+100.2
58.6+-8.D
57.543.6
54.9+11.6
sub2vee
61.1::15.8
60.0:16.4
53.025.6
$2.811.55
15.3:1 1.5
36.74:0.8
71.5:0.4
36.74:0.4
graph2vee
83.24-9.3
60.2446.9
73.342.1
73.2+-1.8
71.1+0.5
50,4+0.9
75.8+110
47.9+0.3
GVAE
87.7:10077
61.21188
70.7=10.7
49.310.4
87.110.1
$2.800.2
InfoGraph
89.0++-111
61,7++1.4
74,4+0.3
76.2+-11
73.0++0.9
49.7
82,5+114
53.5+-10
70.7=1.1
GCC
72.0
49.4
89.8
53.7
78.9
MVGRL
89.7:11.1
62.5:11.7
74.2:10.7
51.2::0.5
84,5:006
GraphCL 86.8:11.3
63.6.1.8
74,420.5
77.910.4
71.1=0.4
50.740.4
89.5
56,011..
71.4=12 81.2-0.3
DGCL
92.1:10.88
65.8:1..5
7..4:00.5
81.9:0.2
75.9::0.7
51.90..4
91.8:10.2
56.1:10.2
and then fed into downstream SVM classifier. We adopt the 10-fold cross validation accuracy, and report the mean accuracy (%) with standard variation after five repeated runs.
Implementation Details. For a fair comparison, the syyeer-arameteryy of the graph augmentations kept consistent with GraphCL. We use GIN [4] as the message- passing layers since it is shown to be one of the most expressive message- passeggg GNNs. Since the ground-truth number of the latent factors is unknown, search the number of channels K from I to 10. More implementation details can be found in supplementary material.
meessag--passing
4.2 Results on Real Benchmark Graphs
The results are reported in Table | We can see that the graph contrastive learning methods generally outperform the graph kernel methods or the classical unsupervised methods, which verify the effectiveness of contrastive learning. Our method DGCL consistently achieves the best performance compared with other contrastive methods (e.g., MVGRL, GrpahCL) and classical unsupervised methods (e.g., graph2vec. GVAE), demonstrating the superiority of our method. For example, our method increases the classification accuracy by 2.4%. 2.2%, and 2.0% compared with the second-best methods on MUTAG, PTC-MR, and RDT-B. respectively. We attribute the results to the fact that these existing methods fail to identify the underlying latent factors which are important in preserving graph properties and can not learn the disentangled representations. In contrast, disentangle graph representations to explicitly consider the entanglement of heterogeneous factors. When compared to graph kernel methods, our method also has the best accuracy all the datasets. Notice that none of these kernel methods is consistently competitive across all of the datasets, as opposed to our method.
Besides the common setting of unsupervised representation learning above, we also consider another setting of semi-supervised representation learning [I] on the ogbg-molhiv from Open Graph Bench- mark 22 to better evaluate our method. Specifically, we first perform pre-training with all training data without labels. Then we conduct fine-tuning on the partial labeled training data and evaluation on the validation/test sets. The task the ogbg-molhiv molhiv dataset is binary classification evaluated by ROC-AUC metric, instead of accuracy in the experiments above. We adopt the provided evaluator and dataset splits for a fair comparison. We compare DGCL with the strong self-supervised bascline GraphCL with 1%, 10%, and 20% label rate for fine-tuning. The results are shown in Figure [ With the increase of the label rate, the results improve for both our method and GraphCL. Our method achieves significant improvement over GraphCL with 1.4%, 3.0%, and 1.4% performance gains at 1%, 10%, and 20% label rate, respectively. The results illustrate that our method is also able to handle large-scale graphs, demonstrating the benefit of learning disentangled graph representations in the contrastive manner.
4.3 Results on Synthetic Graphs
To further investigate the behavior of our method, we generate a synthetic dataset consisting of 1,000 graphs with known latent factors. Specifically, we generate synthetic graphs using the stochastic
7

---

Saogg-mohiiv
Synthetic Dataset
76
64
ROC-AUC (%)
GraphCL
DOCL
73.6
73.2
74.6
Micro-FI (%)
GraphCL
DGCL
72
62
MVGRL
70.6
69.0
68
67.6
60
64
1%
10%
20%
58
1
2
3
4
8 9
10
Label Rate
Number of Channels K
Figure 2: The performance of semi- supervised learning on ogbg-molhiv.
Figure 3: Micro-FI (%) two baselines and our DGCL with different number of channels K.
block model [36] Each graph contains four communities and each community consists of 10 nodes. We define the latent factor the probability p that nodes connected in community. p can take value from {0.2, 0. 0.9}. meaning that there are eight latent factors in the dataset in total. The probability for each community drawn from the eight possible choices without replacement. Two nodes in different communities are connected with probability 0.05. The rows of the adjacency matrices used node features, and the ground-truth communities are used labels, i.e., there are 8 classes and each graph has 4 labels. We train our method and two baseline methods, i.c., MVGRL and GraphCL on the generated synthetic dataset with self-supervision. Then we adopt the SVM classifier on the learned graph embeddings for each method and use the Micro-Fl (%) as the evaluation metric. Other settings are the same Section
We vary the number of channels of our method and report the results in Figure 3 Our method reports better performance than the baselines. We also find that K increases from to 8, the result of DGCL improves, which verifies the importance of disentangling latent factors. After reaching the peak K 8, the performance slightly drops, but in general, our method K is not too large. Our method achieves the best results when K' is equal to the ground-truth number of latent factors, indicating we can model the underlying structure of this simulation dataset.
very sensitive when
Besides the quantitative evaluation, we also provide a qualitative evaluation by plotting the correlation of the latent features in Figure 4 The figure shows the absolute values of the correlation between the elements of 28-dimensional graph representation obtained from MVGRL, GraphCI. and our DGCL (K 8) on the synthetic dataset. We of MVGRL and GraphCL, are entangled. In comparison, the correlation of our DGCL shows eight diagonal blocks, meaning that the channels of DGCL likely extract mutually exclusive information and output disentangled representations.
from the results that the graph representations
4.4 Ablation Studies
We perform ablation studies over the key components of our method to understand their functionalities more deeply. We compare DGCL with the following two variants: (1) Variant 1: it sets po (k)G.)
1/K a uniform distribution of latent factors. (2) Variant 2: it sets K = 1 directly, so that our method will degenerate to the entangled graph contrastive learning model.
Table 2: Ablation studies on the variants of our method. We report the accuracy (%) with standard variation on the datasets. The results of Variant I and drop compared with DGCL, demonstrating the significance to infer latent factors behind the graphs and conduct fastor-wise contrastive learning.
MUTAG
PTC-MR
PROTEINS
NCII
IMDB-B
IMDB-M
RDT-B
RDT-M5K
COLLAB
DGCL
92.1:10.8
65.8:1155
76.4:10.5
81.910.2
75.910.7
51.910.4
91.8:0.2
56.1:10.2
81.2-10.3
Variant
89,340.3
64.3:11.
74.910.2
78.540.5
73.410.5
50.310.2
91.110.7
55.910.3
77.5400
Variant
86.5+10.6
63.5::1.6
73.9+1..6
77.740.6
70.940.5
49,8:10.3
89.710.6
55.7:1.2
71,5400.8
The results of DGCL and variants are shown Table We observe drop in performance of Variant 1, demonstrating the efficacy of inferring the latent factors of the graphs. In variant 2, the latent factors are entangled in the graph representation, making difficulties for characterizing different aspects of the graphs and conducting discrimination tasks in terms of each latent factor independently.

---

100
23
(a) MVGRI.
(b) GraphCI.
(c) DGCL
Figure 4: An analysis of feature correlation on the synthetic graphs with eight latent factors. The figures show the absolute value of the correlations between the elements of the representations learned by MVGRL, GraphCL, and DGCL with eight channels, respectively. We can see that the representations generated from DGCL present more block-wise correlation pattern, indicating that the eight channels of the disentangled graph encoder DGCL are able to capture mutually exclusive information and the latent features have indeed been disentangled.
The deterioration of performance verifies the significance of the proposed factor-wise contrastive learning.
5
Related Work
In this section, we review related works on graph neural networks, contrastive learning on graphs, and disentangled representation learning.
Graph Neural Networks. Graph structured data is ubiquitous in the real world 26
37
38
39 40)
42 have revolutionized the field of graph representation learning. GNNs generally adopt neighborhood aggregation (message passing) paradigm, i.c., the embedding of node is iteratively updated by aggregating embeddings of its neighbors 4 The representation of the whole graph is summarized on node embeddings through the readout function, i.e., graph pooling [4. However, for achieving state-of-the-an performance, most famous GNNs, including their variants, are trained end-to-end with task- specific labels, which could be extremely scarce for some graph datasets. Compared with these supervised models, our model, based on self-supervised contrastive learning, can largely eliminate the over-dependence on the manual labels, which is crucial for graph representation learning.
Recently, graph neural networks (GNNs) p 2 3
Contrastive Learning on Graphs. Recently, contrastive learning, adopting the instance dis- crimination as the pretext task, has become dominant component in self-supervised learning methods (25 21
19 23). Some literatures utilizing contrastive learning for graph data are 43 7 The key of these methods is to maximize the agreement (i.e., similarity) between proper transformations or different views of the input graph. However, the existing graph contrastive learning methods explore general settings where entanglement is severe and do not incor- porate disentangled representation learning. They fail to recognize and disentangle the heterogeneous latent factors behind complex graph data. These holistic methods have limited capacity in preserving detailed graph properties, which easily result in suboptimal representations for downstream tasks.
18
proposed S 10
Disentangled Representation Learning. Disentangled representation learning is to learn factorized representations that identify and disentangle the underlying explanatory factors hidden the observed data [14). The existing efforts about disentangled representation learning are mainly on computer structured data recently 47
vision 44 45 46). has been raising a surge of interest
48 49]
12]. This line of works attempts to learn disentangled representations for graphs but heavily relies on the annotated labels, which largely restricts their applications where labeled data are unavailable or expensive to collect. On the other hand, some works [35] [50] are based on the generative model, namely utilizing Variational Autoencoders (VAEs) on graph for disentanglement, since the hyper- parameter 3 of VAEs can balance the reconstruction and disentanglement [1
52 However, the
reconstruction generative methods could be computationally expensive
53 17 and even introduce
bias that has a negative effect on the learned representation 223. In addition, the reconstruction for

---

graph- structurtd data often involves discrete decisions that are not differentiable [50]. How to learn disentangled representation on graph-structured data with contrastive learning is largely unexplored.
Conclusions
In this paper, we propose a disentangled graph contrastive learning model (DGCL) to solve the problem of disentangled self--uper graph representation learning for the first time. We design disentangled graph encoder with a tailored multi-channel message-passiane layer, which is capable of aggregating features in a disentangled manner. We further propose factor-wise contrastive learning approach to solve instance discrimination task under each latent factor independently, so that the learned representations of DGCL encouraged to not only best describe the graphs but also be disentangled. Therefore, each component of the disentangled representations tends to characterize disentangled aspect of the graph that is pertinent to latent factor. Extensive experiments both synthetic and real-world datasets demonstrate the superiority of our method against several state-of-the-art baselines in self-supervised graph representation learning.
Acknowledgments
This work is supported by the National Key Research and Development Program of China No. 2020AA40106300 and National Natural Science Foundation of China (No. 62050110, No. 62102222).
References
[1] Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard S. Zemel. Gated graph sequence neural networks. In International Conference on Learning Representations, 2016.
[2] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017.
[3] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[4] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations, 2019.
[5] Fan-Yur Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2019.
[6] Petar Velickovic, William Fedus, William Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference Learning Representations, 2019.
[7] Yuxiang Ren, Bo Liu, Chao Huang, Peng Dai, Liefeng Bo, and Jiawei Zhang. Heterogencous deep graph infomax. arXiv:1911.08538, 2019.
[8] Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020, pages 259-270, 2020.
[9] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, pages 4116 4126. PMLR, 2020.
[10] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and lie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 1150-1160, 2020.
10

---

[11] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Neural Information Processing Systems, 2020.
[12] Yiding Yang, Zunlei Feng, Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. In Neural Information Processing Systems, 2020.
[13] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In Neural Information Processing Systems, pages 4800-43 18800-8100 4810, 2018.
[14] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on paltern analysis and machine intelligence, 35(8):
1798-1828, 2013.
[15] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations. arXiv:1812.02230. 2018.
[16] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2021.
[17] Phuc H Le-Khac, Graham Healy, and Alan Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 2020.
[18] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non non-par con-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018
[19] Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. Mi[ce]: Mixture of contrastive experts for unsupervised image clustering. In International Conference on Learning Representations, 2021.
[20] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, pages 776-794. Springer, 2020.
[21] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[22] Feng Wang. Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia, pages 1041-1049, 2017.
[23] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv: /807.03388 2018.
[24] Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Empirical Methods in Natural Language Processing, 2018.
[25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9729 9738, 2020.
[26) Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benehmark: Datasets for machine learning on graphs In Neural Information Processing Systems, 2020.
[27] Karsten M Borgwardt and Hans-Pete Kriegel. Shortest-path kernels on graphs. In IEEE international conference on data mining, pages 8-pp. IEEE, 2005.
[28] Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. Efficient graphlet kernels for large graph comparison, In Artificial intelligence and statistics, pages 488-495. PMLR, 2009,

---

[29] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9). 2011.
[30] Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1365-1374, 2015.
[31] Risi Kondor and Horace Pan. The multiscale laplacian graph kernel. Advances neural information processing systems,
2016.
[32] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864, 2016.
[33] Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learning for subgraphs. In Pacific-Asia Conference Knowledge Discovery and Data Mining, pages 170-182. Springer, 2018.
[34] Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv: 17Xvv:777.005003 2017.
[35] Thomas N Kipf and Max Welling. Variational graph auto-encoders. In NeurlPS Workshop on Bayesian Deep Learning, 2016.
[36] Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social networks, 5(2): 107, 1983.
[37] Haoyang Li, Peng Cui, Chengxi Zang, Tianyang Zhang, Wenwu Zhu, and Yishi Lin. Fates of microscopic social ecosystems: Keep alive or dead? In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages (68-676, 2019.
[38] Haoyang Li, Xin Wang, Ziwei Zhang, Jianxin Ma, Peng Cui, and Wenwu Zhu. Intention-aware sequential recommendation with structured intent transition. IEEE Transactions on Knowledge and Data Engineering, 2021.
[39] Xin Wang, Shuyi Fan, Kun Kuang, and Wenwu Zhu. Explainable automated graph repre- sentation learning with hyperparameter importance. In International Conference Machine Learning, pages 10727-10737. PMLR, 2021.
[40] Yijian Qin, Xin Wang. Peng Cui, and Wenwu Zhu. Gqnas: Graph q network for neural architecture search. In International Conference on Data Mining, 2021.
[41] Fangda Gu, Heng Chang, Wenwu Zhu, Somayeh Sojoudi, and Laurent El Ghaoui. Implicit graph neural networks. Advances in Neural Information Processing Systems, 33:1 198449911999, 2020.
[42] Heng Chang, Yu Rong, Tingyang Xu, Wenbing Huang, Somayeh Sojoudi, Junzhou Huang, and Wenwu Zhu. Spectral graph attention network with fast eigen- aagen-approiimttionn arXiv preprint arXiv:2003.07450, 2020.
[43] Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. arXiv:2012.12533.
[44] Jun- Min-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In Neural Information Processing Systems, pages 517-526, 2018.
[45] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2180-21011 2188, 2016.
12

---

[46] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Disentangled person image generation. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 99-108, 2018.
[47] Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolu- tional networks. In International Conference on Machine Learning, pages 4212-4221, PMLR, 2019.
[48] Yanbei Liu, Xiao Wang, Shu Wu, and Zhitao Xiao. Independence promoted graph disentangled networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4916-4923, 2020.
[49] Zhang Xinyi and Lihui Chen. Capsule graph neural network. In International Conference on Learning Representations, 2018.
[50] Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference on Artificial Neural Networks, pages 412-422. Springer, 2018.
[51] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vac: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2016.
[52] Christopher P Burgess, Trina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 8-vae. NeurIPS Workshop on Learning Disentangled Representations, 2017.
[53] Xiaojie Guo, Liang Zhao, Zhao Qin, Lingfei Wu, Amarda Shehu, and Yanfang Ye. Interpretable deep graph generation with node-edge co-disentanglement. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1697-1707, 2020.
13


Reviews:
[{"review_id": "PJqt16oQwHZ", "reviewer": "Reviewer_T4kE", "summary": "This paper proposed a way of obtaining the disentangled graph representations using graph neural networks and contrastive learning. The learned representation is claimed to capture the heterogeneous latent factors and multiple aspects of graphs. Contrastive learning loss (NT-Xent loss) is employed as the loss function. The authors use the EM algorithm to optimize the ELBO of the likelihood function of observed graphs. The proposed method performs favorably in the benchmark graph datasets compared with other graph representation learning methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduces the idea of disentangled learning to the graph representation. To encourage the representations to have independent subcomponents, the authors use k separate GNNs output channels and use the assignment of channels as the latent factors during modeling. As an unsupervised learning method, contrastive learning is considered. Overall, the setting is novel and the paper proposed complete and practical optimization algorithms. \n\nHowever, I do have the following questions:\n\n1. As disentangled representation is one of the key pursuits of this paper, classic disentangled representation learning methods such as Beta-VAE and factor-VAE are not considered. The sizes of graphs used in the experiments are moderate hence I am not sure it would be too computationally expensive to consider as suggested by the authors. And in fact, for the graph representation learning problem in this paper, no graph reconstruction is even needed.\n\n2. The author proposed the EM-like algorithm to optimize the ELBO. Optimization of ELBO for continuous latent variables is common and easy to implement. What are the motivations for using the discrete channels as the latent variables?\n\n3. Line 148: the paper claims other graph encoders are inherently holistic. What is the definition of being holistic here? Why the proposed method is less holistic than other classic methods such as WL-kernel? \n\n4. In figure 4, the (c) shows a clear block-wise diagonal pattern but at the same time the correlations between each block are not significantly less than (a) and (c). I take a closer look, it seems to be even worse than (b). If the learned representations are well disentangled, I would expect the correlations between each block to be much smaller.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not addressed. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Kr5kfN2hJ4Z", "reviewer": "Reviewer_JYuN", "summary": "This paper proposes a novel disentangled graph contrastive learning model (DGCL) that is the first work to study disentangled self-supervised graph representation learning. The authors address the important problem that the existing self-supervised methods adopt a holistic manner to learn graph representations while largely ignoring the entanglement of the latent factors in real-world graphs. First, the authors design a new disentangled graph encoder to produce the factorized graph representations. This encoder is able to learn disentangled latent factors and capture multiple aspects of graphs. Then, DGCL proposes a novel factor-wise contrastive algorithm to solve the tailored discriminative task in terms of each latent factor independently. It can encourage the learned representations to be expressive and disentangled. To solve the nontrivial inference and learning problems, DGCL introduces a scalable variant of the Expectation-Maximization (EM) algorithm with theoretical contributions. Extensive experiments on both synthetic and real-world datasets demonstrate the superiority of this method over several baselines.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposes DGCL, a novel disentangled graph contrastive learning method for graph representation learning. Existing self-supervised graph representation learning works ignore the entanglement of the latent factors in graphs. This problem can result in the limited capacity in preserving detailed graph properties and harm interpretability. Learning disentangled representation, which is considered an important step to human-like AI, is a promising direction worth investigating particularly in the field of graph representation learning. Learning disentangled graph representation with self-supervised learning is a challenging and significant task, and remains largely unexplored by the existing literature. \n\nThe proposed model is innovative and technically sound. First, the paper proposes a disentangled graph encoder. It can capture multiple aspects of graphs through learning disentangled latent factors and output the disentangled graph representations. The design of the multi-channel message-passing layer is reasonable. The encoder can identify graph factors automatically without having to be designed manually. Then, the paper proposes a novel factor-wise contrastive algorithm to learn informative graph representations and encourage disentanglement. The tailored discrimination task in terms of each latent factor can ensure that each disentangled factor of the representations is sufficiently discriminative only under one specific aspect of the whole graph. So, the learned representations can be disentangled and independently reflect the expressive information from different latent factors.\n\nThe authors provide rigorous theoretical contributions and sufficient empirical evaluations. It is difficult to directly optimize the log-likelihood function because of the latent factors. The paper introduces a variant of the Expectation-Maximization (EM) algorithm to deal with the nontrivial inference and learning of maximizing the evidence lower bound (ELBO). I think this idea is interesting. The experiments are sufficient and convincing. The proposed method DGCL has shown superior performance consistently on both synthetic and real-world datasets, compared to an extensive number of baselines. The results are well-presented both quantitatively and qualitatively and look correct to me. I think the claims are convincingly supported.\n\nBesides, the paper is well written and clearly organized with motivation and approaches comprehensively stated and illustrated. The details of the experiments and implementation for reproducibility are present clearly in the paper. \n\nAdditional suggestions:\n1) The authors have discussed some details about graph contrastive learning baselines. It would be better if the authors provide more descriptions of graph kernel baselines in the experiments.\n2) Typos: Line 198: “nontrival inference and learning” => “nontrivial inference and learning”.\n\nRecommendation: Overall, this paper is novel and makes a non-trivial advance to the field with both solid theoretical contributions and sufficient empirical evaluations. After careful review, I vote for a strong accept with great confidence.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "For completeness, it would be better to provide more descriptions of graph kernel baselines.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15"}, "extra_text": {}}, {"review_id": "8_jW7BYAGzl", "reviewer": "Reviewer_Lf3c", "summary": "This paper studies self-supervised learning for graphs. The key idea is combine\ncontrasting learning with disentanglement. Specifically, the authors define\ndifferent factors to disentangle the latent embedding in different groups Zi's.\nThen the author generates augmented samples from the original graphs, and impose\nfactor-level contrastive loss for learning graph representations. The authors\nhave evaluated the method on several graph classification benchmarks, and shown\nimprovements over several baseline methods.\n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThis paper studies self-supervised learning for graphs. The key idea is combine\ncontrasting learning with disentanglement. Specifically, the authors define\ndifferent factors to disentangle the latent embedding in different groups Zi's.\nThen the author generates augmented samples from the original graphs, and impose\nfactor-level contrastive loss for learning graph representations. The authors\nhave evaluated the method on several graph classification benchmarks, and shown\nimprovements over several baseline methods.\n\nThe method is sound, but with limited novelty. The contrastive learning idea for\ngraph representation learning has already been explored by several works, and\nthe data augmentation techniques are borrowed from existing works [49]. The main extension of this paper is to partition the latent embeddings Zi's into different factors/groups. The technical novelty of this extension is somewhat limited.\n\n\nAnother concern is about generating the positive and negative samples for\ncontrastive learning. The authors follow [49] to generate augmented graphs,\nusing four different types of graph augmentation strategies: node dropping, edge\nperturbation, attribute masking, and subgraph sampling. Such techniques maybe\nfine for general global-level graph data augmentation. However, for the purpose\nof generating factor-wise augmented samples, it is highly likely such\nperturbation may make the generated sample different from the original graph in\nterms of some factors being considered.\n\nThe experimental results is quite positive overall. The authors have compared\ntheir method with graph kernel methods, graph embedding methods and recent\nsemesters on contrastive learning for graphs. They have shown consistent\nimprovements over several benchmarks. However, it will be better if the authors\ncan show what are the latent factors learned by the disentanglement module.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new approach for learning disentangled representations on graphs. Overall, the reviewers were positive about the paper and appreciated the combination of theory and evaluations. One reviewer had concerns about the novelty of the approach. However, after further discussion amongst the reviewers, the consensus moved towards acceptance.
Decision: Accept (Poster)