Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Response to rebuttal: the authors have drastically improved the quality of the submission with the new experiments and clarifications, I have therefore increased the score to a weak accept.

-------------------------------------------

This paper introduces a non-iid VAE architecture that uses a mixture of gaussian latent space and a global latent variables shared among all the elements of a mini batch to capture global information in correlated datapoints in an unsupervised way.

Overall the paper in well written, and I believe in focuses on two important research directions, namely unsupervised learning of disentangled representations and domain alignment. The model itself is novel and well explained, but I feel the technical explanation is missing intuition on how the model can learn disentanglement in beta from purely random batches, which is not obvious to me. 

My biggest concern is in the experimental section, that I did not find convincing enough for a number of reasons:
1. I find it hard to understand if the improvements come from the introduction of the d or the beta latent variables, or a combination of both. How does the model perform in ablation studies in which you remove just one of this components while leaving the others unchanged? 
2. In the single-datasets experiment in section 4.1 how do you define what constitutes a local vs global factors?  Currently some of the chosen factors in Figure 3 seem quite arbitrary. Why is light a local factor but contrast a global one? Why is hair local but beard global? 
3. The quality of the images is not great to be honest (beta-VAE paper has more convincing ones, just to name a single work), and it is not easy to understand whether the low quality results are due to the fact that as you say you have not validated in depth the networks used or because of flaws in the methodology
4. How would a beta-VAE perform with the same setup of the experiment in 4.1? I would not be surprised if it could capture the same features as your model. It is true as you claim that your method does not require the tuning of the beta hyperparameter in the ELBO, but the UG-VAE needs tuning of the dimensionality of d and beta, and is a more complex architecture than a beta-VAE so it is harder to implement and will take longer to train.
5. It is not clear to me in Figure 4.1. why you are traversing z space in this way, but perhaps I misunderstood what you are doing. How are you guaranteed that you will follow the data manifold? The ML-VAE results might be off just because of this. 
6. I believe the more exciting application of this model would be for domain alignment. Why haven't you focused on more multi-datasets experiments?
7. How would a gm-vae baseline with 2 clusters perform with the same setup of the experiment in section 4.2? 

In its current state I believe this paper is not ready for acceptance, but I hope the authors will be able to clarify some of my concerns in which case I will increase the score.

Minor comment:
* The second paragraph of the introduction is giving a lot of details on related work. I would recommend to move this discussion to the related work section, and leave the introduction for higher level discussions that only aim at giving intuition to the reader.

review_title: Experimental section needs more work

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a novel deep generative model based on noni.i.d. variational autoencoders that captures global dependencies among observations in a fully unsupervised fashion. The proposed model combines a mixture model in the local or data-dependent space and a global Gaussian latent variable, which captures interpretable disentangled representations with no user-defined regularization in the evidence lower bound. The proposed model is being evaluated in two tasks: (1) disentanglement, and (2) domain alignment.

Pros:
(1) The paper is very well-written and easy to understand. Especially the model figures (figure 1 and 2) are very intuitive and makes it much easier to understand the intuition and difference between the proposed model and previous related works.
(2) The paper is trying to handle a very interesting task, which is to relax the assumption in a lot of previous works in VAE field. The i.i.d data assumption destroys the correlation between data points in the same dataset. Relaxing this constraint will enable wider adoption of this line of models, which can be very useful.


Cons:
My major concern on this paper is the quality of the evaluation section.
(1) first of all, there is no quantitative or qualitative comparison between the proposed method and previous works. Although disentanglement is a task that is hard to quantify, it is hard to show that UG-VAE outperforms other methods.
(2) One of the contribution mentioned is that UG-VAE does not need to tune a bete parameter as in beta-VAEs. But tuning a hyper-parameter for the disentanglement task is not necessarily a negative thing, as UG-VAE also needs to set the total number of lusters. When setting K= 10, it is inducing prior knowledge, and not completely unsupervised anymore.
(3) It is not convincing that the global disentanglement features are learned in the global latent variables. From Figure 3, it just shows some dimensions control certain attributes. It is also not clearly discussed what is defined as "global attributes" and waht is "local attributes".
(4) "Composing graphical models with neural networks for structured representations and fast inference" from NeurIPS 2016 also has a mixture model as latent space. It will provide readers more insight if the authors could discuss the similarities and differences between these two papers.
review_title: Interesting Model but Evaluation Can Be Improved

final_score: rating: 5: Marginally below acceptance threshold
review_content: This article introduces a VAE-based method for separating local variation factors from global variation factors in the data in an unsupervised manner. It achieves so by designing a graphical model with a mix of example-local and batch-shared variables, and training it using the ELBO. The article provide an detailed experimental analysis on MNIST and CelebA, and experimental evidence that all parts of the model (notably the discrete d variable) are relevant.

The article provides a well detailed description of the proposed UG-VAE, and how it compare to similar models from the literature (notably ML-VAE, from which it is inspired). The experimental analysis is reasonably convincing, though the interpretation of the provided figures in the text is a little more optimistic than I would agree.

There is however one point in particular I would like to see clarified for this paper to be accepted: the justification of the structural design. I have several questions/remarks regarding it:

1. Equation 8 shows that q(β | X, d) actually depends on the categorical parameters of q(d|Z), rather than the value of d itself. As such, from a correctness perspective, the distribution is q(β | X, Z) (with weight sharing with q(d | Z)): it does not depend on the value of d. I suspect this choice was made to escape the computational cost of marginalizing the whole batch of variables d in the joint distribution for β, but this changes the meaning of the model.

2. The analysis of what features get stored in β versus Z is not very insightful. The text of the paper present this as if it was obvious, but does not seem that obvious to me why "beard" is a global feature but "hair" is a local one. The same non-obviousness applies to all examples.

3. I don't get why the training procedure is supposed to work. It empirically does to some extent, but there is no clear theoretical justification. If the same β is used for the whole batch, given the batch is randomly sampled, what part of the dynamic would indeed drive the model toward extracting some "common" features into this variable?

For this last point in particular, the proposed structure seems pretty close to an other one that would be I believe much more natural: have the global parameter β not shared across the entire batch, but depending on the class d the encoder assigned to the datapoint. Is that something that has been considered, and if yes why was it not satisfactory compared to the proposed model?

I believe it is necessary to clearly address these points for the article to be accepted.

--------

Small remarks & typos:

- Just after equation 9, there is a ^-1 missing on the sigma in the text
- the notation for the KL-divergence is inconsistent between equations 10 and 11 ( D_KL(...) vs KL(...) )
- the text in 4.A about how the z feature is explored in figure 3 is not completely clear. Is the interpolation done only along the diagonal, moving from (μ1 - 3, μ2 - 3, ...) to (μ1 + 3, μ2 + 3, ...) ? If so, why ? If it is rather only moved along one particular dimension, then I suggest rewording this sentence to make it clear.
review_title: Interesting article, but the theoretical justification needs to be clarified

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a deep generative model based on the non i.i.d. VAE framework in an unsupervised version. The model which combines a mixture prior in the local latent space with global latent space has three advantages: First, the latent space can capture interpretable features. Second, the model performs domain alignment. Third, the model can discriminate among their global posterior representations. Although this paper has mild improvement on the basic VAE structure, the model displays a good interpretability power, and the setup of the latent variables are illustrated reasonably in the paper.

Strength:
1. This paper models on non-i.i.d data in an unsupervised version, which provides a flexible model.
2. This model provides a good interpretability power. This paper demonstrates how the features are controlled by the global and local variables, and also it shows the necessary of including the mixture prior in the local space to acquire interpretable information.
3. This model performs domain alignment, the global variable β can capture domain knowledge. 

Critiques:
1. The paper only discussed one prior distribution for the latent variable d and β. The power of choosing other types of prior distribution is unknown.
2. For the domain alignment experiment, only two datasets are used, it will be more convincing to include more datasets.

I have read authors' feedback and will keep my original score.

review_title: A solid paper

Meta-review: This paper aims at learning disentangled representation at different level without the supervision signal of group information. To achieve this, the proposed UG-VAE model uses both global variable $\beta$ to represent common information shared across all data, as well as a mixture of Gaussian prior for the local latent variable $p(z) = \int p(z|d)p(d)d$ where $d$ represents the assignment of the group for a particular datapoint. Experiments considered evaluation on unsupervised global factor learning, domain alignment and a downstream application task on batch classification.

Reviewers agreed that the proposed model seems interesting and novel, however some reviewers raised clarity concerns on how to interpret the learned representation by UG-VAE. Revision has addressed this clarity issue to some extent, although some doubts from some reviewers still exists. Also reviewers raised concerns on less competitive experimental results, and the authors have updated the manuscript with improved results. 

To me the main issues of the experimental section are (1) no quantitative result is provided regarding global factor learning and domain alignment, and (2) there is no other benchmark being studied in the experimental section. In my view, at least some other VAE representation learning baselines can be included in the batch classification section in order to demonstrate the real benefit of learning global factor based representations in downstream tasks. 
Decision: Reject