Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:  
This paper proposes an unsupervised graph-level representation learning method considering global-local disentanglement. Specifically, the authors propose a GL-Disen model based on graph VAE architecture to jointly learn global and local representations for a graph. The global information is shared across the whole graph while the local information varies from patch to patch, corresponding to common and local factors, respectively. Empirical experimental results show that the learned representation achieves superior performance in the downstream graph classification task, and analyses demonstrate the learned representations exhibit some disentangle property.   

Pros:  
1. Unsupervised graph representation learning considering global and local disentanglement seems to be a novel problem.  
2. The proposed method generalizes disentangled VAE into graph data to disentangle common factors from the local ones. The formulations and model descriptions are clear.  
3. Experiments, including both qualitative analysis and quantitative results, demonstrate the effectiveness of the learned global factors in downstream tasks.  

Cons and questions:  
My major concern lies in the insufficiency of experiments. Specifically:  
1. The disentanglement part is modified from Beta-VAE. Since normal VAE is adopted in graphs (e.g., Variational Graph Auto-Encoders by Kipf and Welling), the authors need to compare these methods to demonstrate the improvement is actually from the disentanglement part rather than the VAE structure.   
2. Although the authors demonstrate the effectiveness of disentanglement in downstream tasks (i.e., graph classification), it is unclear whether these global factors have intuitive explanations on some of the datasets, e.g., the showcases of molecular graphs in Duvenaud et al., 2015, or the authors may adopt some synthetic datasets.  
3. Since both the global and local node representations are disentangled, I am curious whether the local node representations can also be validated in some downstream node-level tasks.   
4. Figure 2 in Section 4.2.1 is not entirely convincing since there is no reference line of how much correlation a non-disentangled method will have (e.g., in Ma et al., 2019, the authors compare the disentangled method with GCN).   

Other questions:  
5. How the proposed method can handle the mode collapse problem, i.e., only a few latent factors learn useful information?  
6. As shown in Table 1, though the proposed method outperforms other GNNs, it does not always compare favorably to kernel-based methods such as GCKN. The authors may want to further elaborate on the pros and cons of using GNN vs. kernel-based methods.  
7. There lacks a discussion on the complexity of the proposed method.  
8. The technical contribution is somewhat limited since both Beta-VAE and graph VAE are known in the literature. It would be more interesting if the authors can integrate local-global disentanglement with local neighborhood disentanglement in Ma et al. 2019 to derive a more novel architecture.  

I will be happy to improve my scores if authors can address the above questions.  
  
  
=========  
  
I have updated my score considering the paper has improved its quality after the revision (adding more experiments/baselines, comparison with the literature, etc.).


=========   
New updates: following the new comments of Reviewer 4, I also briefly check the code in the supplementary material and find it indeed seems to have the consistency problem (i.e., not reconstructing graph edges as mentioned in the paper). Thus, I am also wondering how the authors implement Graph-VAE in the rebuttal phase and whether the improvement of their proposed method over Graph-VAE is really from disentanglement or the differences in the autoencoder. Based on this potentially serious problem, I reinstate my original score and think the paper should be clarified before acceptance.
review_title: An interesting idea, but it is unclear whether the improvement really comes from disentanglement 

final_score: rating: 3: Clear rejection
review_content: The authors propose a VAE-type generative model approach to characterize the hidden factors, with a divided focus on the global and local reconstructions. The claim is that the learnt hidden representations are disentangled (which is not defined clearly) using two reconstruction terms. The setting of the problem adopts the graph VAE setting in [1,2] (which I think the authors should mention in the related work), and the ELBO & local aggregation (convolution) approaches used in this paper are relatively standard in the generative modelling and graph representation learning domain. 

Apart from the limited novelty, which would not have affected my evaluation if it solves the problem as claimed, I have several major concerns about this paper:

1. The notion of disentanglement is not well-defined in the first place. In the VAE setting where the hidden factors are stochastic, does disentanglement refer to independence? Or they are orthogonal under a specific measure induced by the graph itself? The claims made by the authors can never be examined rigorously (the visual results do not constitute supportive evidence as I shall discuss later). 

2. There is no guarantee that the so-called global and local factors are not confounded. Both the global and local reconstruction terms involve the two types of factors. Given the high expressivity of deep learning models, the local factors can easily manage both tasks, or the global factors are merely enhancing the signals of the local factors. There no mechanism to prevent the cross-terms during the optimization, so the learning process of the global and local factors confounded as a result of how the authors design the objective function.

3. Unclear interpretation of the visual results. It seems that the visual results showcase a similar pattern among the local and global factors, despite the difference that the signal is stronger for the local factors (which is evident as they play a more critical role in the objective). In the absence of a clear definition of disentanglement, more persuasive numerical results and interpretations are needed.
 

[1] Kipf T N, Welling M. Variational graph auto-encoders[J]. arXiv preprint arXiv:1611.07308, 2016.
[2] Xu, Da, et al. "Generative graph convolutional network for growing graphs." ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.

review_title: Lack of guarantee that the global and local factors are disentangled, unclear definition, limited novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: I think the idea of the paper is interesting. The writing is well and easy to read. However, it does not meet the condition of acceptance from my point of view. I have some concerns with its characterization of the literature.

- Some important related work is missing. It seems authors ignore talking about some literature of unsupervised graph representation learning, such as [1], [2], etc. Also, they do not make a performance comparison with the methods above in experiments.
[1] Contrastive Multi-View Representation Learning on Graphs. ICML 2020
[2] Self-supervised Training of Graph Convolutional Networks. Arxiv 2020

- Disentangling the global and local generative factors graph representation learning is important. However, the authors didn't explain the definition of “Global” and “Local” factors clearly. It would also be better if they can show an example of global/local factors when generating graph.

- The experiments are missing. I have some concerns as follows. What is the best number of generative factors which is important for this method? Can this method occur mode collapse and how to valid or prevent it? How can this method prove that each factor is necessary for the generative process? What is the real meaning of each factor? How about the time/space complexity of this method? More experiments or discussions should be conducted to answer these questions.

Based on the above reasons, this paper can have much more improvement.

review_title: The paper tries to study unsupervised disentanglement learning for graph-level representations. In particular, it focuses on the complex process with global and local generative factors and proposes a VAE based learning algorithm, which argues achieving state-of-the-art performance on the task of unsupervised graph representation learning.

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper, the authors proposed to disentangle the global level information from the local level one to reduce the effect of the irrelevant information. The proposed method outperforms several state-of-the-arts on multiple datasets for graph classification. Overall, I like the idea of applying unsupervised disentangled learning to graph level representation learning. Some concerns are on the experimental study and missing references.

Strong Points:
1. Disentanglement learning is a cutting-edge field and has gained much attention in recent years. It is true that global and local features often entangle together when we learn graph representations. The problem is real and important.

2. The architecture of the model is easy to understand and reasonable.

3. The experimental study is comprehensive, including both qualitative analysis and quantitative analysis. The experimental setup instructions and pseudo-codes are very clear, making the algorithm easy to be reproduced.

Weak Points:
1. Performing experiments only on graph classification tasks weakens the significance of the paper. It is common for graph representation learning methods to be tested on other tasks, such as graph similarity/distance computation and graph-level clustering, in order to draw a general and convincing conclusion.

2. Some important references are missing. The authors should discuss and compare with them.
On graph-level representation learning:
- Bai et al. Unsupervised Inductive Graph-Level Representation Learning via Graph-Graph Proximity. IJCAI 2019.
On disentangled representation learning:
- Yang et al. Factorizable Graph Convolutional Networks. NIPS 2020.
- Guo et al. Interpretable Deep Graph Generation with Node-edge Co-disentanglement. KDD 2020.

3. The paper mentioned that the global and local latent generative factors are sampled from their respective posterior distributions. More details are expected.

review_title: The paper proposed GL-Disen, which is a disentanglement-based unsupervised method for graph representation learning. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this paper, the authors proposed a disentanglement learning based approach for unsupervised graph level representation learning. They assume that disentangled representations which capture these global and local generative factors into independent latent units can be highly beneficial for graph level tasks. The extensive experiments and analysis  show that our method achieves the state-of-the-art performance on the task of unsupervised graph representation learning.

===========
Strengths:
1. The paper is well written and the disentangling factors can benefit the unsupervised graph representation learning.
2. The performance of this work is good compared with the state-of-the-art baselines. The source code is also available.
3. The related work is sufficient to understand the motivation of this work. The 

=====
Weakness:
1. The idea is not very novel. For example, two important assumptions 1) a  global and local factor for graph analysis 2) local latent factors are independent. 
Those two assumptions actually have been explored in unsupervised learning tasks. For example, the follow paper[1] exactly disentangle local and global information into two separate sets of latent variables within the VAE framework.  It seems that migrating this idea under graph is straightforward. The paper is more like a mixture of [1] and (Higgins et al., 2017), and GCN

[1] Charakorn, Rujikorn, et al. "An Explicit Local and Global Representation Disentanglement Framework with Applications in Deep Clustering and Unsupervised Object Detection." arXiv preprint arXiv:2001.08957 (2020).

2. In Figure4, It seems that the GL-Disen global has very good accuracy. The GL-disen global-local combines only outperform GL-Disen global within a very small range of \lambda but with large fluctuation. Does that mean the local factor contribution little to the overall performance?


In Conclude,the authors propose a VAE based learning algorithm to disentangle the global graph-level information. The overall presentation is good. The similar ideas have been explored in unsupervised learning. The novelty of this work is thus not very impressive.
review_title: The authors proposed a disentanglement learning based approach for unsupervised graph level representation learning, which aim to capture the global and local latent factors. 

Meta-review: In this paper, the authors designed a disentanglement mechanism for global and local information of graphs and proposed a graph representation method based on it. I agree with the authors that 1) considering the global and local information of graphs jointly is reasonable and helpful (as shown in the experiments) and 2) disentanglement is different from independence. 

However, the concerns of the reviewers are reasonable --- Eq. (2) and the paragraph before it indeed show that the authors treat the global and the local information independently. Moreover, the disentanglement of the global information (the whole graph) and the local information (the patch/sub-graph) is not well-defined. In my opinion, for the MNIST digits, the angle and the thickness (or something else) of strokes can be disentangled (not independent) factors that have influences on different properties of the data. In this work, if my understanding is correct, the global and the local factors just provide different views to analyze the same graphs and the proposed method actually designs a new way to leverage multi-view information. It is not sure whether the views are disentangled and whether the improvements are from "disentanglement". 

If the authors can provide an example to explain their "disentanglement" simply as the MNIST case does, this work will be more convincing. Otherwise, this work suffers from the risk of overclaiming.
Decision: Reject