Disentangled Contrastive Learning on Graphs
Authors: Haoyang Li
Xin Wang
Ziwei Zhang
Zehuan Yuan
Hang Li
Wenwu Zhu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Graph Neural Network
Contrastive Learning
Self-supervised Learning
Disentangled Representation Learning

Abstract: Recently, self-supervised learning for graph neural networks (GNNs) has attracted considerable attention because of their notable successes in learning the representation of graph-structure data. However, the formation of a real-world graph typically arises from the highly complex interaction of many latent factors. The existing self-supervised learning methods for GNNs are inherently holistic and neglect the entanglement of the latent factors, resulting in the learned representations suboptimal for downstream tasks and difficult to be interpreted. Learning disentangled graph representations with self-supervised learning poses great challenges and remains largely ignored by the existing literature. In this paper, we introduce the Disentangled Graph Contrastive Learning (DGCL) method, which is able to learn disentangled graph-level representations with self-supervision. In particular, we first identify the latent factors of the input graph and derive its factorized representations. Each of the factorized representations describes a latent and disentangled aspect pertinent to a specific latent factor of the graph. Then we propose a novel factor-wise discrimination objective in a contrastive learning manner, which can force the factorized representations to independently reflect the expressive information from different latent factors. Extensive experiments on both synthetic and real-world datasets demonstrate the superiority of our method against several state-of-the-art baselines.

Disentangled Contrastive Learning on Graphs
Haoyang Li', Xin Wang': Ziwei Zhang', Zehuan Yuan', Hang Li,, Wenwu Zhu
'Tsinghua University, *Bytedance
lihy18@mails. www.:mmmmils..tinnuuua.aa. edu. en, wang, zazhang)@tainghua. edu. (yuanzehuan, lihang. @bytedanee com, wuzhußtsinghua. edu..edddde...... cn
Abstract
Recently, self-supervised learning for graph neural networks (GNNs) has attracted considerable attention because of their notable successes in learning the repre- sentation of graph-structure data. However, the formation of real-world graph typically arises from the highly complex interaction many latent factors. The existing self-supervised learning methods for GNNs are inherently holistic and neglect the entanglement of the latent factors, resulting the learned representa- tions suboptimal for downstream tasks and difficult to be interpreted. Learning disentangled graph representations with self-superviedd learning poses great chal- lenges and remains largely ignored by the existing literature. In this paper, we introduce the Disentangled Graph Contrastive Learning (DGCL) method, which is able learn disentangled graph-leve representations with self sulf-superviiion. In particular, we first identify the latent factors of the input graph and derive its fac- torized representations. Each of the factorized representations describes a latent and disentangled aspect pertinent to a specific latent factor of the graph. Then we propose novel factor-wise discrimination objective in contrastive learning manner, which can force the factorized representations to independently reflect the expressive information from different latent factors. Extensive experiments both synthetic and real-world datasets demonstrate the superiority of our method against several state-of-the-art baselines.
Introduction
Graph structured data is ubiquitous in the real world, e.g., social networks, biology networks, traffic networks, etc. Recently, graph neural networks (GNNs) have become increasingly prevalent in learning graph representations in supervised manner, demonstrating their strength wide variety of research fields M 3 4]. GNNs require task-dependent annotated labels to learn effective unavailable in practice, thus motivating the
representations, which are extremely scarce, advent of self-supervised graph representation learning.
Contrastive learning, discriminative approach pulling similar samples close and pushing dissimilar samples far away, has become a dominant strategy in self-supervised graph representation learning (5) 8 9 10.101 II.. Despite their notable successes, the existing graph contrastive learning methods generally adopt holistic scheme, i.c., the learned representations characterize graphs as perceptual whole, ignoring the nuances between different aspects of the graph. In fact. the formation of graph typically follows a relational process in the real world, driven by many complex latent factors. For example, in social networks, social group may have several communities originated from different relations (e.g., friends, colleagues, etc.) or interests (e.g., sports, games, etc.) 112. And a molecular graph may consist of various groups of atoms and bonds representing different functional units [13 The complex relations among the multiple latent factors bring an urge for disentangling these factors
"Corresponding authors
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

in contrastive graph representation learning, which remains unexplored by the existing holistic works. As result, the graph representations learned by the existing methods contain mixture of entangled factors, harming interpretability and leading to suboptimal performance for predictive tasks involving whole graph representations.
In this paper, we propose to learn disentangled contrastive graph representation, for the first time. Although disentangled representation learning, which aims to characterize the various underlying explanatory factors behind the observed data in different parts of the factorized representations [4]
15 has been demonstrated to be more explainable [2] and generalizable [14], disentangled graph contrastive learning faces the following two challenges. (1) Tailored graph encoder for disentangled contrastive learning. The graph encoder should be carefully designed so that it can be sufficiently expressive to infer the disentangled latent factors in the graph. (2) Tailored discrimination tasks designed for disentangled graph contrastive learning- Since lask-dependent labels are not available in the self-supervised setting, disentangled graph contrastive learning can only utilize the limited amount of self-superviiion information. This implies that the discrimination tasks should be well-designed for disentangled contrastive representation learning on graphs.
To tackle these challenges, we propose a novel disentangled graph contrastive learning model (DGCL) capable of disentangled contrastive learning on graphs. In particular, we first design disentangled passaggg layer. Each channel is
graph encoder whose key ingredient is a multi multi-channel tailored to aggregate features only from one disentangled latent factor. Then a separate readout operation in each channel summarizes the specific aspect of the graph according to the corresponding latent factor, so as to produce the disentangled graph representation. Next, we conduct contrastive learning in each representation subspace characterized by each factor independently instead of in the whole representation space. This novel factor-wise contrastive approach can ensure that each disentangled factor of the vectorized representations is sufficiently discriminative only under one specific aspect of the whole graph. Thus the representations are encouraged to be disentangled and best characterize the aspect pertinent to a latent factor of the graph. Compared with the existing methods, our proposed DGCL model encodes graph with multiple disentangled representations, making it possible to explore the meaning of each channel, which benefits in more explainability for producing graph representations.
We conduct extensive experiments on both synthetic graph dataset and empirical well-known graph benchmarks. The results show that the representations learned from DGCL, can achieve substantial performance gains on the downstream graph classification task compared with various state-of-the-a baselines.
The contributions of this paper are summarized follows:
We propose disentangled graph contrastive learning model (DGCL), which is able to learn disentangled graph representation via factor-wise contrastive learning. To the best our knowledge, we are the first to study disentangled self-super graph representation learning.
We propose a disentangled graph encoder capture multiple aspects of graphs through learning disentangled latent factors on graphs. We further present the factor-wise contrastive learning approach tailored discrinination tasks in terms of each latent factor independently.
We conduct extensive experiments to verify the efficacy of our proposed model for the graph classification task. The results on several graph classification datasets demonstrate that DGCL achieves state-of-the-ee performance by significantly outperforming the baselines.
We introduce the problem formulation and preliminaries in Section 2 In Section B] describe the details of our proposed method. Section and qualitative comparisons. We review the related work in Section 5 in Section 6
4 presents the experimental results, including quantitative Finally, we conclude our work
2 Problem Formulation and Preliminaries
2.1 Problem Formulation
Let = (G.) be a graph dataset with N graphs. The key of most self-supervised graph representation learning methods, including ours, is to derive a graph encoder f()), which outputs

---

Input Graph Gj
Disentangled Graph Encoder fo()
Factor-wise Contrastive Learning
hack
Ho
Message Passing
MLP
Channel
Disentangled Representationz,
Channel 2
34,1
74,2
2
G
Channel 3
Objective
Graph
Augmentations
Shared Encoder
Feedback
Maximize Agreement Under Each Factor
HBB
MLP
RA- AA
2
21
21,3
Message Passing
Disentangled Representations
Extract features specific to each factor
Figure 1: The framework of DGCL Model. (1) The input graph G; undergoes graph augmentations to produce G",, and both of them fed into the shared disentangled graph encoder fo()). (2) In the encoder fo(.), the node features HD are first aggregated by L ppassage-passing layers and then taken the input of multi-channel mressage-passingg layer. (3) Based on the disentangled graph representation zi, the factor-wise contrastive learning aims to maximize the agreement under each latent factor and provide feedback for the encoder improve the disentanglement. This example assumes that there are three latent factors, hence the three channels.
d-dimensional representation z; = f(;)) E Rdd for each input graph, such that Z - M best describes G. In this work, we aim to learn multi-channel graph encoder fo(-) with parameters so that the output can be disentangled representation, i.e. fo(-) [ (.)] where K is the number of channels. To be specific, z; expected to composed of K independent components,
ie.. [z,.1, 2,.,..... k.. where Z fe (G.) R k [1. Ad = d/K, assuming that there are K latent factors behind the graph instances to be disentangled. The b component factor & accurately. We also assume
z is for characterizing the aspect of G; that is pertinent that the value of Z,,k will be merely a white noise vector if the input graph G, does not contain any information of factor k. Note that we focus primarily on undirected graphs in our method, although can be straightforwardly extended to directed graphs.
2.2 Preliminaries on Contrastive Learning
Unlike generative models, contrastive learning is an instance-wise discriminative approach that aims at making similar instances closer and dissimilar instances far from each other representation space [16] [7 It treats each instance in the dataset as a distinct class of its own and trains classifier to distinguish between individual instance classes [8 19]. Given a dataset
{a; J each instance is assigned with unique surrogate label yı since ground-truth labels are given. y, often regarded as the ID of the instance in the dataset, i.c., Ys = So the probability classifier is defined as:
Po(vilz;) = exp Φ(Vi, vy,))
(1)
Σn exp ø( (o(vi, v/g3)
where 0 denotes the parameters of the encoder, Both v, and Vyy are the embeddings from T; which are generated from two different encoders [20], or a shared encoder [21). Before being passed into the encoder, the input x; could undergo data augmentations (2I) which play critical role in defining effective predictive tasks for learning the encoder. é is the similarity function, often adopting cosine similarity with temperature [22]. i.e., Φ(v;, Vyz, Vyy v, assuming the embeddings are (e-normalized. Then the learning objective is to maximize the joint probability
H p(u((x)) over the dataset, namely minimize the negative log-likelibhe likelihood function ΣN l, if let l, = llog p(y)

---

Note that loss E could be NCE loss [18, InfoNCE loss 23 or NT-Xem loss (2I. The encoder will be encouraged learn representation space where samples (e.g.. augmented data) from the same instance (e.g., an image, graph) are pulled closer and samples from different instances are pushed apart [6. For convenience, we follow the settings above in this work.
Disentangled Graph Contrastive Learning
In this section, we present the proposed DGCL model. The framework of DGCL is shown in Figure [ In Section 3.1 we introduce the disentangled graph encoder to identify the complex latent factors and capture multiple aspects of graphs. Then in Section 3.2 propose a factor-wise contrastive learning approach to conduct instance discrimination under each latent factor independently. The objective and the details of the optimization are derived in Section 3.3 and Section 3.4 respectively.
3.1 Disentangled Graph Encoder
The key of the disentangled graph encoder to produce the factorized graph representation z; = [z,,1, 2i, Zi.k) ] for each input graph G2 E G. Based the factorized representation, can infer the related latent factors behind the graph.
Generally, GNNs use the graph structure and node features to learn the representation vector h, of each node o with message-passing mechanism, i.e., iteratively updating the representation of a node by aggregating representations of its neighbors. The propagation of the 4th layer is formulated as:
h COMBINE' (h-, AGGREGATEE ((h) E (u))))),
where hy is the representation of node the tth layer and h initialized with node features. N(v) is the neighborhood to node v. We use the term GNN to indicate the message-passing layer in Eq. (2.
Let H' = (h'le v E } be the node embeddings after the tth GNN, where V denotes the node set of the graph. After applying L traditional message-passing layers, we propose graph-disentanglemen layer to learn the disentangled representations. The goal is to extract features specific to each latent factor with a separate channel. Specifically, we adopt K separate channels identify the complex heterogeneous latent factors and capture multiple aspects of the input graph. For each channel, we first utilize a GNN; to propagate information with its own parameters: H 4-1-1 ( where A is the adjacency matrix of the input graph. H is the node embeddings which is only pertinent to the thh latent factor. Then the READOUT function (i.c., pooling function) of each channel used to summarize all the obtained node representations into a fixed-length graph-level representation: hG.. = READOUT, (( Finally, each channel outputs the factorized graph representation with a separate MIP: 2 = MLP (hG....
Compared with the existing graph encoders that are inherently holistic, our disentangled graph encoder consists of K channels, rending the possibility to identify the complex heterogencous latent factors and capture multiple aspects of graphs.
3.2 Disentangled Factor-wise Contrastive Learning
Unlike the existing contrastive learning methods, DGCL designs a novel factor-wise instance dis- criminative task and learns to solve this task under each latent factor independently. This design only makes similar samples closer and dissimilar samples far from each other in the representation space, but also encourages the learned representation to incorporate factor-level information for disentangiement.
Specifically, we assume that the formation of real-world graphs is usually driven by multiple latent heterogeneous factors. So the instance discriminative task should be represented as the expectation of several subtasks under the latent factors:
(e( (HIG..) [po (G,, k)].
Here pa (k)),) is the probability distribution over latent factors for the input graph G,. po po k.) denotes the instance discrimination subtask under the th latent factor.

---

Firstly, given the representation of G; derived from the disentangled graph encoder fe(-), we present a prototype-basee method to obtain pa (b]G.), We introduce K latent factor prototypes [c*) l and the probability of the 4th latent factor reflected in G, is parameterized as:
exp (ZZ,, Ci))
po (o(k]
(4)
Σ exp ((zi, k C&)
where á the cosine similarity with temperature T, i.c. ((a, b) COSINE(a, b)/7 and COSINE(a, b) a b/ b/( [|bll])-
Then, we define the instance discrimination subtask under the 1 latent factor as:
exp Φ (Z (.),k)
(5)
=== exp Φ k, Z (,
where 2 and Z are the disentangled representations produced by the shared graph encoder, and ys is the unique surrogate label (see Section 2 of the graph G.. In our method, we follow [T8] 19 to implement Y/ as the II) of the graph in the dataset, i.e., Vi i. For notation convenience, we do not distinguish yi and hereafter when there no risk of confusion.
Next, we describe the process to get 2i i.c. Zy in Eq. (5). First, the input graph G; undergoes graph data augmentations to obtain its correlated views G and they form positive pair. Data augmentation expected to create novel and realistically rational data by applying certain transforma- tions that do not affect the label, and plays critical role in defining effective predictive tasks [1 We follow [11 to adopt four types of graph augmentation strategies, including node dropping, edge perturbation, attribute masking, and subgraph sampling. More details of graph augmentations can be found supplementary material. Then, the augmented graph G₁ also fed into the shared disentangled graph encoder fa()) to produce zi,e. i.e. Zyy,,'' Given the disentangled representations k and zi,x of the G, and G¹, respectively, conduct factor-wise contrastive learning for each latent factor independently as Eq. (5)
3.3 Evidence Lower Bound (ELBO)
We present the objective of our method. Following the existing methods 18 19 we aim to maximize the joint probability I (G, over the graph dataset G = { We learn the model parameters 8 by maximizing the ligeliidh:hod::
0" arg max
log po(y:|G,) arg max Σ log Epp(K(..) (p,
(6)
However, directly maximizing the likeliihldoodd function is difficult because of the latent factors. Therefore, we instead optimize the evidence lower bound (ELBO) of the log-likelihood function given by Theorem
[
See the supplementary material for the proof.
Theorem 1. The log likelihood function of each graph log (y G+) is lower bounded by the ELBO: i( [ s.)llog [log Pe (yall G2 k)] Dx (4o (k|G,, Ms) (k))).
To make the ELBO as tight as possible, we require that go g (k)G,, Mt) is close to Pa (k.], Mi). whose detailed implementations are provided the next subsection (see Eq. and Eq. 7 In the ELBO C(0, i), pe (3/ G., k) and pe(k have been introduced in Eq. (5) and Eq. 44h respectively, and ge go(k] G.,, Mr) is variational distribution to infer the posterior distribution of the latent factors after observing both G; and its correlated view Gyy-
3.4 Optimization
We introduce variational distribution ( Gt, y6) to infer the posterior probability Po(k)G, y:) that is defined with Bayes' theorem as follows:
(k G. Go G.,
pe (k) =
Σ po (A/ (ol)
5

---

Pe y:) is the probability of the kth latent factor pertinent to both G; and the augmented G simultaneously. Compared with the prior distribution po (k]G.) in Eq. 4. Po Po yi) incorporates more useful information (i.e., factor-wise similarity) from po (yi Gt k). Although both po(k) and po(k G., Vs) are designed to infer the latent factor distribution, po (k:],, is calculated only given the graph Gj, but po G4 Dt) is calculated after observing Gü the augmented version G and their similarities under the specific latent factor.
However, we cannot compute the posterior probability tractably because of the term pa (1i Gi, k). If we directly calculate po po(y: k) according to Eq. (5). all the instances in the dataset G are needed for computing the denominator in Eq. (5), which could be computationally prohibitive 24 I8 [9].
To tackle this obstacle, several strategies are proposed in the literature, including memory bank
[8 20]. dynamic dictionary (251, NT-Xent loss [2I]. Here, we adopt NT-Xent loss on a minibatch
B G. So in practice, the instance discrimination under each latent factor calculated by:
exp (K,,,,,
Po
G1, Po
(8)
Σ E j exp o(z,,k 2y,x)
We approximate the posterior probability po ( wi) with a variational distribution defined as:
po (A) (V,
qo (k(k M))
Σ Po (k Go kt)
Finally, we seek to learn the parameters @ of the disentangled graph encoder. More specifically, we calculate go Gt, ys) and maximize the ELBO over a mini-batch B using mini-batch gradient ascent:
C B) = Σ 1( IEB
(10)
Encourage disentanglement. Note that our objective and its optimization can inherently encourage disentanglement without adding extra regularization term (e.g., minimizing mutual information). The reason is that factorizing the instance discrimination into K factor-wise subtasks will enforce the independence of the learned graph representation z, Besides, qo(k) Gr, y:) is computed based on ht and other K - latent factors. Thus, the graph encoder is forced to preserve exclusive information in each channel to get more accurate approximation to the posterior, a tighter ELBO expected. The strong inductive biases in DGCL encourage to learn disentangled graph representations that match the ground truth factors behind the graphs.
Experiments
We empirically evaluate our proposed method through experiments, and analyze its behavior on synthetic graph dataset to gain deeper insight. Ablation studies are conducted to show the effectiveness of the proposed method. We provide more discussions in supplementary material, including the complexity of our method, the impact of the hyper-parameters, etc.
4.1 Experimental Setup
Datasets. To demonstrate the advantages of our method, we conduct experiments on nine well- known graph classification datasets including four bioinformatics datasets, i.e., MUTAG, PTC-MR, NCII, PROTEINS, and five social network datasets, i.e., COLLAB, IMDB-BINARY, IMDB-MULTI, REDDIT-BINARY, and REDDIT-MULTT-SK We also adopt a larger graph dataset egbg-molhin from Open Graph BenchMark (OGB) 26 More details are provided in supplementary material.
Baselines. We compare DGCL with the following two groups of basclines. One group of baselines are graph kernels including Shortest Path Kernel (SP) 227. Graphlet Kernel (GK) (28], Weisfeiler- Lehman Sub-tree Kernel (WL) (29] Deep Graph Kernels (DGK) 33), and Multi-Scale Laplacian Kernel (MLG) (31] The other group of baselines are classical unsupervised graph representation learning methods including node2vec [32], sub2vec [33]. graph2vec [34), GVAE [35], and more
recent contrastive graph representation learning methods including InfoGraph [. GCC 10,, MVGRL
9. and GraphCL. [II].
Evaluation. verify the effectiveness our method, we follow the common evaluation protocols in the existing literature [34] 5 30 where graph embeddings are generated from each method

---

Table 1: Graph classification accuracy (%) of DGCL and baselines. In each column, the boldfaced score denotes the best result and the underlined score represents the second-best result. "_" indicates the result is not reported in the paper.
MUTAG
PTC-MR
PROTEINS
NCH1
IMDB-B
IMDB-M
RIYT-B
RIT--MKK
COLLAB
SP
85.242.
58.24244
75.1
73.0+0.22
55.6=0.2
38.0+0.3
64.1+0.1
39.6+0.22
GK
81.742.11
57.3
71.720.6
62.34..3
65.94100
43.910.
77.3+0.2
41.01022
72.8=0.3
80.743.0
58.0+0.55
72.9+06
80.0+0.5
72.3+34
47.0+055
68.8+0.4
46.14002
DGK
87.422277
60.142..6
73.3=0.8
80.3:00.5
67.0=00.6
44.62:0.5
78.040.4
41.34022
73.1=0.3
MLG
87.9+1166
63.3+11.5
76.14200
808+-13
66.6:10.3
41.2+000
node2vec
72.6+100.2
58.6+-8.D
57.543.6
54.9+11.6
sub2vee
61.1::15.8
60.0:16.4
53.025.6
$2.811.55
15.3:1 1.5
36.74:0.8
71.5:0.4
36.74:0.4
graph2vee
83.24-9.3
60.2446.9
73.342.1
73.2+-1.8
71.1+0.5
50,4+0.9
75.8+110
47.9+0.3
GVAE
87.7:10077
61.21188
70.7=10.7
49.310.4
87.110.1
$2.800.2
InfoGraph
89.0++-111
61,7++1.4
74,4+0.3
76.2+-11
73.0++0.9
49.7
82,5+114
53.5+-10
70.7=1.1
GCC
72.0
49.4
89.8
53.7
78.9
MVGRL
89.7:11.1
62.5:11.7
74.2:10.7
51.2::0.5
84,5:006
GraphCL 86.8:11.3
63.6.1.8
74,420.5
77.910.4
71.1=0.4
50.740.4
89.5
56,011..
71.4=12 81.2-0.3
DGCL
92.1:10.88
65.8:1..5
7..4
81.9:0.2
75.9::0.7
51.90..4
91.8:10.2
56.1:10.2
and then fed into downstream SVM classifier. We adopt the 10-fold cross validation accuracy, and report the mean accuracy (%) with standard variation after five repeated runs.
Implementation Details. For a fair comparison, the syyeer-arameteryy of the graph augmentations kept consistent with GraphCL. We use GIN [4] as the message- passing layers since it is shown to be one of the most expressive message- passeggg GNNs. Since the ground-truth number of the latent factors is unknown, search the number of channels K from I to 10. More implementation details can be found in supplementary material.
meessag--passing
4.2 Results on Real Benchmark Graphs
The results are reported in Table | We can see that the graph contrastive learning methods generally outperform the graph kernel methods or the classical unsupervised methods, which verify the effectiveness of contrastive learning. Our method DGCL consistently achieves the best performance compared with other contrastive methods (e.g., MVGRL, GrpahCL) and classical unsupervised methods (e.g., graph2vec. GVAE), demonstrating the superiority of our method. For example, our method increases the classification accuracy by 2.4%. 2.2%, and 2.0% compared with the second-best methods on MUTAG, PTC-MR, and RDT-B. respectively. We attribute the results to the fact that these existing methods fail to identify the underlying latent factors which are important in preserving graph properties and can not learn the disentangled representations. In contrast, disentangle graph representations to explicitly consider the entanglement of heterogeneous factors. When compared to graph kernel methods, our method also has the best accuracy all the datasets. Notice that none of these kernel methods is consistently competitive across all of the datasets, as opposed to our method.
Besides the common setting of unsupervised representation learning above, we also consider another setting of semi-supervised representation learning [I] on the ogbg-molhiv from Open Graph Bench- mark 22 to better evaluate our method. Specifically, we first perform pre-training with all training data without labels. Then we conduct fine-tuning on the partial labeled training data and evaluation on the validation/test sets. The task the ogbg-molhiv molhiv dataset is binary classification evaluated by ROC-AUC metric, instead of accuracy in the experiments above. We adopt the provided evaluator and dataset splits for a fair comparison. We compare DGCL with the strong self-supervised bascline GraphCL with 1%, 10%, and 20% label rate for fine-tuning. The results are shown in Figure [ With the increase of the label rate, the results improve for both our method and GraphCL. Our method achieves significant improvement over GraphCL with 1.4%, 3.0%, and 1.4% performance gains at 1%, 10%, and 20% label rate, respectively. The results illustrate that our method is also able to handle large-scale graphs, demonstrating the benefit of learning disentangled graph representations in the contrastive manner.
4.3 Results on Synthetic Graphs
To further investigate the behavior of our method, we generate a synthetic dataset consisting of 1,000 graphs with known latent factors. Specifically, we generate synthetic graphs using the stochastic
7

---

Saogg-mohiiv
Synthetic Dataset
76
64
ROC-AUC (%)
GraphCL
DOCL
73.6
73.2
74.6
Micro-FI (%)
GraphCL
DGCL
72
62
MVGRL
70.6
69.0
68
67.6
60
64
1%
10%
20%
58
1
2
3
4
8 9
10
Label Rate
Number of Channels K
Figure 2: The performance of semi- supervised learning on ogbg-molhiv.
Figure 3: Micro-FI (%) two baselines and our DGCL with different number of channels K.
block model [36] Each graph contains four communities and each community consists of 10 nodes. We define the latent factor the probability p that nodes connected in community. p can take value from {0.2, 0. 0.9}. meaning that there are eight latent factors in the dataset in total. The probability for each community drawn from the eight possible choices without replacement. Two nodes in different communities are connected with probability 0.05. The rows of the adjacency matrices used node features, and the ground-truth communities are used labels, i.e., there are 8 classes and each graph has 4 labels. We train our method and two baseline methods, i.c., MVGRL and GraphCL on the generated synthetic dataset with self-supervision. Then we adopt the SVM classifier on the learned graph embeddings for each method and use the Micro-Fl (%) as the evaluation metric. Other settings are the same Section
We vary the number of channels of our method and report the results in Figure 3 Our method reports better performance than the baselines. We also find that K increases from to 8, the result of DGCL improves, which verifies the importance of disentangling latent factors. After reaching the peak K 8, the performance slightly drops, but in general, our method K is not too large. Our method achieves the best results when K' is equal to the ground-truth number of latent factors, indicating we can model the underlying structure of this simulation dataset.
very sensitive when
Besides the quantitative evaluation, we also provide a qualitative evaluation by plotting the correlation of the latent features in Figure 4 The figure shows the absolute values of the correlation between the elements of 28-dimensional graph representation obtained from MVGRL, GraphCI. and our DGCL (K 8) on the synthetic dataset. We of MVGRL and GraphCL, are entangled. In comparison, the correlation of our DGCL shows eight diagonal blocks, meaning that the channels of DGCL likely extract mutually exclusive information and output disentangled representations.
from the results that the graph representations
4.4 Ablation Studies
We perform ablation studies over the key components of our method to understand their functionalities more deeply. We compare DGCL with the following two variants: (1) Variant 1: it sets po (k)G.)
1/K a uniform distribution of latent factors. (2) Variant 2: it sets K = 1 directly, so that our method will degenerate to the entangled graph contrastive learning model.
Table 2: Ablation studies on the variants of our method. We report the accuracy (%) with standard variation on the datasets. The results of Variant I and drop compared with DGCL, demonstrating the significance to infer latent factors behind the graphs and conduct fastor-wise contrastive learning.
MUTAG
PTC-MR
PROTEINS
NCII
IMDB-B
IMDB-M
RDT-B
RDT-M5K
COLLAB
DGCL
92.1:10.8
65.8:1155
76.4:10.5
81.910.2
75.910.7
51.910.4
91.8:0.2
56.1:10.2
81.2-10.3
Variant
89,340.3
64.3:11.
74.910.2
78.540.5
73.410.5
50.310.2
91.110.7
55.910.3
77.5400
Variant
86.5+10.6
63.5::1.6
73.9+1..6
77.740.6
70.940.5
49,8:10.3
89.710.6
55.7:1.2
71,5400.8
The results of DGCL and variants are shown Table We observe drop in performance of Variant 1, demonstrating the efficacy of inferring the latent factors of the graphs. In variant 2, the latent factors are entangled in the graph representation, making difficulties for characterizing different aspects of the graphs and conducting discrimination tasks in terms of each latent factor independently.

---

100
23
(a) MVGRI.
(b) GraphCI.
(c) DGCL
Figure 4: An analysis of feature correlation on the synthetic graphs with eight latent factors. The figures show the absolute value of the correlations between the elements of the representations learned by MVGRL, GraphCL, and DGCL with eight channels, respectively. We can see that the representations generated from DGCL present more block-wise correlation pattern, indicating that the eight channels of the disentangled graph encoder DGCL are able to capture mutually exclusive information and the latent features have indeed been disentangled.
The deterioration of performance verifies the significance of the proposed factor-wise contrastive learning.
5
Related Work
In this section, we review related works on graph neural networks, contrastive learning on graphs, and disentangled representation learning.
Graph Neural Networks. Graph structured data is ubiquitous in the real world 26
37
38
39 40)
42 have revolutionized the field of graph representation learning. GNNs generally adopt neighborhood aggregation (message passing) paradigm, i.c., the embedding of node is iteratively updated by aggregating embeddings of its neighbors 4 The representation of the whole graph is summarized on node embeddings through the readout function, i.e., graph pooling [4. However, for achieving state-of-the-an performance, most famous GNNs, including their variants, are trained end-to-end with task- specific labels, which could be extremely scarce for some graph datasets. Compared with these supervised models, our model, based on self-supervised contrastive learning, can largely eliminate the over-dependence on the manual labels, which is crucial for graph representation learning.
Recently, graph neural networks (GNNs) p 2 3
Contrastive Learning on Graphs. Recently, contrastive learning, adopting the instance dis- crimination as the pretext task, has become dominant component in self-supervised learning methods (25 21
19 23). Some literatures utilizing contrastive learning for graph data are 43 7 The key of these methods is to maximize the agreement (i.e., similarity) between proper transformations or different views of the input graph. However, the existing graph contrastive learning methods explore general settings where entanglement is severe and do not incor- porate disentangled representation learning. They fail to recognize and disentangle the heterogeneous latent factors behind complex graph data. These holistic methods have limited capacity in preserving detailed graph properties, which easily result in suboptimal representations for downstream tasks.
18
proposed S 10
Disentangled Representation Learning. Disentangled representation learning is to learn factorized representations that identify and disentangle the underlying explanatory factors hidden the observed data [14). The existing efforts about disentangled representation learning are mainly on computer structured data recently 47
vision 44 45 46). has been raising a surge of interest
48 49]
12]. This line of works attempts to learn disentangled representations for graphs but heavily relies on the annotated labels, which largely restricts their applications where labeled data are unavailable or expensive to collect. On the other hand, some works [35] [50] are based on the generative model, namely utilizing Variational Autoencoders (VAEs) on graph for disentanglement, since the hyper- parameter 3 of VAEs can balance the reconstruction and disentanglement [1
52 However, the
reconstruction generative methods could be computationally expensive
53 17 and even introduce
bias that has a negative effect on the learned representation 223. In addition, the reconstruction for

---

graph- structurtd data often involves discrete decisions that are not differentiable [50]. How to learn disentangled representation on graph-structured data with contrastive learning is largely unexplored.
Conclusions
In this paper, we propose a disentangled graph contrastive learning model (DGCL) to solve the problem of disentangled self--uper graph representation learning for the first time. We design disentangled graph encoder with a tailored multi-channel message-passiane layer, which is capable of aggregating features in a disentangled manner. We further propose factor-wise contrastive learning approach to solve instance discrimination task under each latent factor independently, so that the learned representations of DGCL encouraged to not only best describe the graphs but also be disentangled. Therefore, each component of the disentangled representations tends to characterize disentangled aspect of the graph that is pertinent to latent factor. Extensive experiments both synthetic and real-world datasets demonstrate the superiority of our method against several state-of-the-art baselines in self-supervised graph representation learning.
Acknowledgments
This work is supported by the National Key Research and Development Program of China No. 2020AA40106300 and National Natural Science Foundation of China (No. 62050110, No. 62102222).
References
[1] Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard S. Zemel. Gated graph sequence neural networks. In International Conference on Learning Representations, 2016.
[2] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017.
[3] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018.
[4] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference Learning Representations, 2019.
[5] Fan-Yur Yun Sun, Jordan Hoffman, Vikas Verma, and Jian Tang. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In International Conference on Learning Representations, 2019.
[6] Petar Velickovic, William Fedus, William Hamilton, Pietro Liò, Yoshua Bengio, and R Devon Hjelm. Deep graph infomax. In International Conference Learning Representations, 2019.
[7] Yuxiang Ren, Bo Liu, Chao Huang, Peng Dai, Liefeng Bo, and Jiawei Zhang. Heterogencous deep graph infomax. arXiv:1911.08538, 2019.
[8] Zhen Peng, Wenbing Huang, Minnan Luo, Qinghua Zheng, Yu Rong, Tingyang Xu, and Junzhou Huang. Graph representation learning via graphical mutual information maximization. In Proceedings of The Web Conference 2020, pages 259-270, 2020.
[9] Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. In International Conference on Machine Learning, pages 4116 4126. PMLR, 2020.
[10] Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and lie Tang. Gee: Graph contrastive coding for graph neural network pre-training. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 1150-1160, 2020.
10

---

[11] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Neural Information Processing Systems, 2020.
[12] Yiding Yang, Zunlei Feng, Mingli Song, and Xinchao Wang. Factorizable graph convolutional networks. In Neural Information Processing Systems, 2020.
[13] Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling. In Neural Information Processing Systems, pages 4800-43 18000-8100 4810, 2018.
[14] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on paltern analysis and machine intelligence, 35(8):
1798-1828, 2013.
[15] Irina Higgins, David Amos, David Pfau, Sebastien Racaniere, Loic Matthey, Danilo Rezende, and Alexander Lerchner. Towards definition of disentangled representations. arXiv:1812.02230. 2018.
[16] Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2021.
[17] Phuc H Le-Khac, Graham Healy, and Alan Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 2020.
[18] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non non-par con-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3733-3742, 2018
[19] Tsung Wei Tsai, Chongxuan Li, and Jun Zhu. Mi[ce]: Mixture of contrastive experts for unsupervised image clustering. In International Conference on Learning Representations, 2021.
[20] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, pages 776-794. Springer, 2020.
[21] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference machine learning, pages 1597-1607. PMLR, 2020.
[22] Feng Wang. Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embedding for face verification. In Proceedings of the 25th ACM international conference on Multimedia, pages 1041-1049, 2017.
[23] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv: /807.03388 2018.
[24] Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Empirical Methods in Natural Language Processing, 2018.
[25] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9729 9738, 2020.
[26) Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benehmark: Datasets for machine learning on graphs In Neural Information Processing Systems, 2020.
[27] Karsten M Borgwardt and Hans-Pete Kriegel. Shortest-path kernels on graphs. In IEEE international conference on data mining, pages 8-pp. IEEE, 2005.
[28] Nino Shervashidze, SVN Vishwanathan, Tobias Petri, Kurt Mehlhorn, and Karsten Borgwardt. Efficient graphlet kernels for large graph comparison, In Artificial intelligence and statistics, pages 488-495. PMLR, 2009,

---

[29] Nino Shervashidze, Pascal Schweitzer, Erik Jan Van Leeuwen, Kurt Mehlhorn, and Karsten M Borgwardt. Weisfeiler-lehman graph kernels. Journal of Machine Learning Research, 12(9). 2011.
[30] Pinar Yanardag and SVN Vishwanathan. Deep graph kernels. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 1365-1374, 2015.
[31] Risi Kondor and Horace Pan. The multiscale laplacian graph kernel. Advances neural information processing systems,
2016.
[32] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855-864, 2016.
[33] Bijaya Adhikari, Yao Zhang, Naren Ramakrishnan, and B Aditya Prakash. Sub2vec: Feature learning for subgraphs. In Pacific-Asia Conference Knowledge Discovery and Data Mining, pages 170-182. Springer, 2018.
[34] Annamalai Narayanan, Mahinthan Chandramohan, Rajasekar Venkatesan, Lihui Chen, Yang Liu, and Shantanu Jaiswal. graph2vec: Learning distributed representations of graphs. arXiv: 17Xvv:777.005003 2017.
[35] Thomas N Kipf and Max Welling. Variational graph auto-encoders. In NeurlPS Workshop on Bayesian Deep Learning, 2016.
[36] Paul W Holland, Kathryn Blackmond Laskey, and Samuel Leinhardt. Stochastic blockmodels: First steps. Social networks, 5(2): 107, 1983.
[37] Haoyang Li, Peng Cui, Chengxi Zang, Tianyang Zhang, Wenwu Zhu, and Yishi Lin. Fates of microscopic social ecosystems: Keep alive or dead? In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages (68-676, 2019.
[38] Haoyang Li, Xin Wang, Ziwei Zhang, Jianxin Ma, Peng Cui, and Wenwu Zhu. Intention-aware sequential recommendation with structured intent transition. IEEE Transactions on Knowledge and Data Engineering, 2021.
[39] Xin Wang, Shuyi Fan, Kun Kuang, and Wenwu Zhu. Explainable automated graph repre- sentation learning with hyperparameter importance. In International Conference Machine Learning, pages 10727-10737. PMLR, 2021.
[40] Yijian Qin, Xin Wang. Peng Cui, and Wenwu Zhu. Gqnas: Graph q network for neural architecture search. In International Conference on Data Mining, 2021.
[41] Fangda Gu, Heng Chang, Wenwu Zhu, Somayeh Sojoudi, and Laurent El Ghaoui. Implicit graph neural networks. Advances in Neural Information Processing Systems, 33:1 198449911999, 2020.
[42] Heng Chang, Yu Rong, Tingyang Xu, Wenbing Huang, Somayeh Sojoudi, Junzhou Huang, and Wenwu Zhu. Spectral graph attention network with fast eigen- aagen-approximttionn arXiv preprint arXiv:2003.07450, 2020.
[43] Shichang Zhang, Ziniu Hu, Arjun Subramonian, and Yizhou Sun. Motif-driven contrastive learning of graph representations. arXiv:2012.12533.
[44] Jun- Min-Ting Hsieh, Bingbin Liu, De-An Huang, Li F Fei-Fei, and Juan Carlos Niebles. Learning to decompose and disentangle representations for video prediction. In Neural Information Processing Systems, pages 517-526, 2018.
[45] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pages 2180-21011 2188, 2016.
12

---

[46] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Disentangled person image generation. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 99-108, 2018.
[47] Jianxin Ma, Peng Cui, Kun Kuang, Xin Wang, and Wenwu Zhu. Disentangled graph convolu- tional networks. In International Conference on Machine Learning, pages 4212-4221, PMLR, 2019.
[48] Yanbei Liu, Xiao Wang, Shu Wu, and Zhitao Xiao. Independence promoted graph disentangled networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 4916-4923, 2020.
[49] Zhang Xinyi and Lihui Chen. Capsule graph neural network. In International Conference on Learning Representations, 2018.
[50] Martin Simonovsky and Nikos Komodakis. Graphvae: Towards generation of small graphs using variational autoencoders. In International Conference on Artificial Neural Networks, pages 412-422. Springer, 2018.
[51] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerehner. beta-vac: Learning basic visual concepts with a constrained variational framework. In International Conference Learning Representations, 2016.
[52] Christopher P Burgess, Trina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Des- jardins, and Alexander Lerchner. Understanding disentangling in 8-vae. NeurIPS Workshop on Learning Disentangled Representations, 2017.
[53] Xiaojie Guo, Liang Zhao, Zhao Qin, Lingfei Wu, Amarda Shehu, and Yanfang Ye. Interpretable deep graph generation with node-edge co-disentanglement. Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1697-1707, 2020.
13


Reviews:
[{"review_id": "kUsf14C9GjH", "reviewer": "Reviewer_mcrT", "summary": "This manuscript proposes a disentangled graph contrastive learning method, which can generate the graph-level disentangled representation. The proposed method contains two main parts: the graph encoder to generate the factors and the factor-wise contrastive learning strategy to learn the parameters of the encoder.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1, This manuscript tries to solve the disentangled graph learning in an unsupervised manner.\n2, The proposed method is evaluated on several datasets and indeed shows its effectiveness.\n\nSome problems:\n1, For a fair comparison, it would be better to also show the number of parameters and the inference time for the proposed method and the comparisons.\n\n2, The work “Factorizable graph convolutional networks” utilizes different GNNs on different factor graphs to generate the factor features, which is similar to the proposed disentangled graph encoder. It is recommended to discuss the connection and difference between these two methods.\n\n3, It seems that the proposed optimization method (disentangled factor-wise contrastive learning) is a general method that can not only be applied to the graph contrastive learning problem (once we can obtain the contrastive pair examples). I’m wondering if there is any special reason to only apply the proposed method to the graph learning problem?\n\n\nMinor issues:\n1, It is recommended to use the same font in the figures like that in the main body, like Fig.2 and Fig.3.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please see the problems in the main review", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "fc0D1R26i6s", "reviewer": "Reviewer_9Pj4", "summary": "The submission presents Disentangled Graph Contrastive Learning (DGCL), a method to learn disentangled representations for graphs. To this end, the submission proposes a multi-channel message passing graph encoder. Each channel is supposed to match one disentangled latent factor. Furthermore, contrastive learning is performed on each latent factor separately. The experiments show that the learned representations work better than several prior works in several datasets when used as input for an SVM.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I found the method proposed by the paper interesting and idea for a disentangled representation of graphs to be novel.\n\nA main issue in the evaluation is that an SVM is used to perform the final task prediction based on the learned graph embeddings. SVMs are a specific kind of model and representations that work well with SVMs might not work well with other models such as neural networks. Hence, I think the paper could be substantially improved if more models are used to perform prediction tasks based on the learned representations. The paper argues that using SVMs is a standard approach, which is also used in prior works. However, this does not mitigate the fact that the evaluation is limited. \n\nFurthermore, fine-tuning with a small set of labeled graphs could have been performed to better evaluate the learned representations. In fact, this would perhaps be the most prominent use case for the learned representations in analogy to other self-supervised representations, e.g. for natural language. Furthermore, successful fine-tuning would be important to evaluate the significance of the results of the submission.\n\nAnother issue is the fact that the learned latent factors in the real world datasets are not analyzed beyond their performance. It would have been very interesting to see 1) what the learned latent factors represent, and 2) if they are really mutually exclusive. This analysis is only performed for the synthetic data. I wonder if the latent factors in the real datasets are similarly mutually exclusive.\n\nI would have preferred to also see results of supervised methods, e.g. in Table 1, to better understand if self-supervised learning alone is already able to learn reasonable good representations.\n\nIn general, I found the paper easy to read and easy to understand, even though a rather complex/technical method is proposed. However, I didn't find the contribution bullet points (lines 77-87) to be helpful and rather repetitive. Some minor paper also also slightly repetitive, e.g. Cosine Similarity is introduced in line 112 and then again in line 165.\n\nOne part that I did not really understand is why the latent factors are mutually exclusive. The paper states this multiple times (e.g. in line 218), but I couldn't identify a single part in the approach that forces the latent factors to be mutually exclusive. Perhaps I missed something here.\n\nAn interesting evaluation on the number of channels is provided in the supplementary material. I found this evaluation very helpful to understand if and how multiple channels improve the representations. I think it would improve the paper if this analysis could be moved into the main part of the paper, since the idea to use multiple disentangled channels is the key contribution of the submission.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "KErXq2qWvy", "reviewer": "Reviewer_vaM6", "summary": "This paper proposes a graph classification model named Disentangled Graph Contrastive Learning (DGCL). The core idea of the paper is to learn a disentangled graph-level representation for each graph in a contrastive self-supervised learning manner. Extensive experiments demonstrate the effectiveness and superiority of the proposed model over the baselines in the paper. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strong points of the paper are as follows:\n\nS1. Current self-supervised learning methods for GNNs neglect the entanglement of the latent factors and thus result in the suboptimal performance for downstream tasks. In the paper, the authors propose to learn disentangled graph-level representations with self-supervision. Then the disentangled representation can be used to boost the performance of the downstream tasks.\n\nS2. The proposed factor-wise contrastive learning approach guarantees the ability of the factorized representations to reﬂect the expressive information from different latent factors independently.\n\nS3. The paper is written and organized in a good manner.\n\nNevertheless, there are also several weak points:\n\nW1. The novelty of this paper is relatively marginal with insignificant technical contribution since the proposed method is mostly a combination of existing modules. The key challenges to the problem are weakly discussed (without sound theoretical or empirical analysis), rendering the necessity of considering disentangled SSL unconvincing.\n\nW2. Unsupervised ablation of GIN should be included in the baselines, since DGCL applies GIN as the backbone model. Also, DGI needs to be compared.\n\nW3. The efficiency and scalability of the proposed model and baselines are not discussed in the paper.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations have been discussed in the supplementary materials.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "JHiXPSDRh4_", "reviewer": "Reviewer_ubo5", "summary": "This paper propose a disentangled graph contrastive learning model to solve the graph representation learning task. The proposed model uses a disentangled graph encoder with a tailored multi-channel message passing layer, where the EM algorithm is used to optimize the parameters of the model. The authors conduct experiment with their models with both synthetic and real-world datasets, and the result validate the effectiveness of the proposed model.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "​1. This paper is well-written and easy to follow. \n\n2. The model proposed model is basically an extension from the GraphCL [49] model, where latent factors for the input graph are introduced for learning the disentangled factor-level information, which is novel but limited. The approach of maximizing ELBO is generally sound. \n\n3. Although the authors provide an analysis of feature correlation with the disentangled latent factors, the proposed approach for learning such latent factors is not well motivated. Say, the factor-level information in Section 3.2 is not well explained.\n\n4. The authors provide a comprehensive ablation study on variants of the proposed methods and show the same trend over different datasets, which is good. However, the codes of the proposed methods are missing, although the authors have ticked on that option in the checklist.\n\n5. Comparing the GraphCL model and other baselines the improvement of the proposed model is impressive. I am wondering whether this approach can be applied to the node-level representation tasks.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors don't describe the limitations of their work.\nSuggestions to improve:\n- It will be good if the authors can make their codes publicly available, as they promised.\n- It will be good if the authors can do some case studies to show the disentanglement on the learned latent factors.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The manuscript has been reviewed by four experienced reviewers, among whom three voted for acceptance and one reviewer (vaM6) voted for borderline reject. The borderline reviewer mainly complained about novelty, some missing experiments, as well as the discussion on scalability and efficiency. Regarding the novelty, in fact, other reviewers indeed rated the proposed approach as being novel; regarding the missing experiments, the authors claimed that the requested experiments were already in the manuscript; regarding the scalability and efficiency, the authors provided a discussion in the rebuttal.

Since reviewer vaM6 only mildly rejects the submission while all other reviewers tend to accept, the AC will follow the majority vote and accept the submission. Still, the authors are strongly recommended to account for the comments from all reviewers in the revised version.


Decision: Accept (Poster)