Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a novel disentangled graph contrastive learning model (DGCL) that is the first work to study disentangled self-supervised graph representation learning. The authors address the important problem that the existing self-supervised methods adopt a holistic manner to learn graph representations while largely ignoring the entanglement of the latent factors in real-world graphs. First, the authors design a new disentangled graph encoder to produce the factorized graph representations. This encoder is able to learn disentangled latent factors and capture multiple aspects of graphs. Then, DGCL proposes a novel factor-wise contrastive algorithm to solve the tailored discriminative task in terms of each latent factor independently. It can encourage the learned representations to be expressive and disentangled. To solve the nontrivial inference and learning problems, DGCL introduces a scalable variant of the Expectation-Maximization (EM) algorithm with theoretical contributions. Extensive experiments on both synthetic and real-world datasets demonstrate the superiority of this method over several baselines.
main_review: The paper proposes DGCL, a novel disentangled graph contrastive learning method for graph representation learning. Existing self-supervised graph representation learning works ignore the entanglement of the latent factors in graphs. This problem can result in the limited capacity in preserving detailed graph properties and harm interpretability. Learning disentangled representation, which is considered an important step to human-like AI, is a promising direction worth investigating particularly in the field of graph representation learning. Learning disentangled graph representation with self-supervised learning is a challenging and significant task, and remains largely unexplored by the existing literature. 

The proposed model is innovative and technically sound. First, the paper proposes a disentangled graph encoder. It can capture multiple aspects of graphs through learning disentangled latent factors and output the disentangled graph representations. The design of the multi-channel message-passing layer is reasonable. The encoder can identify graph factors automatically without having to be designed manually. Then, the paper proposes a novel factor-wise contrastive algorithm to learn informative graph representations and encourage disentanglement. The tailored discrimination task in terms of each latent factor can ensure that each disentangled factor of the representations is sufficiently discriminative only under one specific aspect of the whole graph. So, the learned representations can be disentangled and independently reflect the expressive information from different latent factors.

The authors provide rigorous theoretical contributions and sufficient empirical evaluations. It is difficult to directly optimize the log-likelihood function because of the latent factors. The paper introduces a variant of the Expectation-Maximization (EM) algorithm to deal with the nontrivial inference and learning of maximizing the evidence lower bound (ELBO). I think this idea is interesting. The experiments are sufficient and convincing. The proposed method DGCL has shown superior performance consistently on both synthetic and real-world datasets, compared to an extensive number of baselines. The results are well-presented both quantitatively and qualitatively and look correct to me. I think the claims are convincingly supported.

Besides, the paper is well written and clearly organized with motivation and approaches comprehensively stated and illustrated. The details of the experiments and implementation for reproducibility are present clearly in the paper. 

Additional suggestions:
1) The authors have discussed some details about graph contrastive learning baselines. It would be better if the authors provide more descriptions of graph kernel baselines in the experiments.
2) Typos: Line 198: “nontrival inference and learning” => “nontrivial inference and learning”.

Recommendation: Overall, this paper is novel and makes a non-trivial advance to the field with both solid theoretical contributions and sufficient empirical evaluations. After careful review, I vote for a strong accept with great confidence.
limitations_and_societal_impact: For completeness, it would be better to provide more descriptions of graph kernel baselines.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed a way of obtaining the disentangled graph representations using graph neural networks and contrastive learning. The learned representation is claimed to capture the heterogeneous latent factors and multiple aspects of graphs. Contrastive learning loss (NT-Xent loss) is employed as the loss function. The authors use the EM algorithm to optimize the ELBO of the likelihood function of observed graphs. The proposed method performs favorably in the benchmark graph datasets compared with other graph representation learning methods.
main_review: This paper introduces the idea of disentangled learning to the graph representation. To encourage the representations to have independent subcomponents, the authors use k separate GNNs output channels and use the assignment of channels as the latent factors during modeling. As an unsupervised learning method, contrastive learning is considered. Overall, the setting is novel and the paper proposed complete and practical optimization algorithms. 

However, I do have the following questions:

1. As disentangled representation is one of the key pursuits of this paper, classic disentangled representation learning methods such as Beta-VAE and factor-VAE are not considered. The sizes of graphs used in the experiments are moderate hence I am not sure it would be too computationally expensive to consider as suggested by the authors. And in fact, for the graph representation learning problem in this paper, no graph reconstruction is even needed.

2. The author proposed the EM-like algorithm to optimize the ELBO. Optimization of ELBO for continuous latent variables is common and easy to implement. What are the motivations for using the discrete channels as the latent variables?

3. Line 148: the paper claims other graph encoders are inherently holistic. What is the definition of being holistic here? Why the proposed method is less holistic than other classic methods such as WL-kernel? 

4. In figure 4, the (c) shows a clear block-wise diagonal pattern but at the same time the correlations between each block are not significantly less than (a) and (c). I take a closer look, it seems to be even worse than (b). If the learned representations are well disentangled, I would expect the correlations between each block to be much smaller.

limitations_and_societal_impact: Not addressed.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies self-supervised learning for graphs. The key idea is combine
contrasting learning with disentanglement. Specifically, the authors define
different factors to disentangle the latent embedding in different groups Zi's.
Then the author generates augmented samples from the original graphs, and impose
factor-level contrastive loss for learning graph representations. The authors
have evaluated the method on several graph classification benchmarks, and shown
improvements over several baseline methods.

main_review: 
This paper studies self-supervised learning for graphs. The key idea is combine
contrasting learning with disentanglement. Specifically, the authors define
different factors to disentangle the latent embedding in different groups Zi's.
Then the author generates augmented samples from the original graphs, and impose
factor-level contrastive loss for learning graph representations. The authors
have evaluated the method on several graph classification benchmarks, and shown
improvements over several baseline methods.

The method is sound, but with limited novelty. The contrastive learning idea for
graph representation learning has already been explored by several works, and
the data augmentation techniques are borrowed from existing works [49]. The main extension of this paper is to partition the latent embeddings Zi's into different factors/groups. The technical novelty of this extension is somewhat limited.


Another concern is about generating the positive and negative samples for
contrastive learning. The authors follow [49] to generate augmented graphs,
using four different types of graph augmentation strategies: node dropping, edge
perturbation, attribute masking, and subgraph sampling. Such techniques maybe
fine for general global-level graph data augmentation. However, for the purpose
of generating factor-wise augmented samples, it is highly likely such
perturbation may make the generated sample different from the original graph in
terms of some factors being considered.

The experimental results is quite positive overall. The authors have compared
their method with graph kernel methods, graph embedding methods and recent
semesters on contrastive learning for graphs. They have shown consistent
improvements over several benchmarks. However, it will be better if the authors
can show what are the latent factors learned by the disentanglement module.


limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper introduces a new approach for learning disentangled representations on graphs. Overall, the reviewers were positive about the paper and appreciated the combination of theory and evaluations. One reviewer had concerns about the novelty of the approach. However, after further discussion amongst the reviewers, the consensus moved towards acceptance.
Decision: Accept (Poster)