Venue: ICLR 2018 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ########## UPDATED AFTER AUTHOR RESPONSE ##########

Thanks for the good revision and response that addressed most of my concerns. I am bumping up my score. 

###############################################


This paper presents a Disentangled Inferred Prior (DIP-VAE) method for learning disentangled features from unlabeled observations following the VAE framework. The basic idea of DIP-VAE is to enforce the aggregated posterior q(z) = E_x [q(z | x)] to be close to an identity matrix as implied by the commonly chosen standard normal prior p(z). The authors propose to moment-match q(z) given it is hard to minimize the KL-divergence between q(z) and p(z). This leads to one additional term to the regular VAE objective (in two parts, on- and off-diagonal). It has the similar property as beta-VAE (Higgins et al. 2017) but without sacrificing the reconstruction quality. Empirically the authors demonstrate that DIP-VAE can effectively learn disentangled features, perform comparably better than beta-VAE and at the same time retain the reconstruction quality close to regular VAE (beta-VAE with beta = 1). 

The paper is overall well-written with minor issues (listed below). I think the idea of enforcing an aggregated (marginalized) posterior q(z) to be close to the standard normal prior p(z) makes sense, as opposed to enforcing each individual posterior q(z|x) to be close to p(z) as (beta-)VAE objective suggests. I would like to make some connection to some work on understanding VAE objective (Hoffman & Johnson 2016, ELBO surgery: yet another way to carve up the variational evidence lower bound) where they derived something along the same line of an aggregated posterior q(z). In Hoffman & Johnson, it is shown that KL(q(z) | p(z)) is in fact buried in ELBO, and the inequality gap in Eq (3) is basically a mutual information term between z and n (the index of the data point). Similar observations have led to the development of VAMP-prior (Tomczak & Welling 2017, VAE with a VampPrior). Following the derivation in Hoffman & Johnson, DIP-VAE is basically adding a regularization parameter to the KL(q(z) | p(z)) term in standard ELBO. I think this interpretation is complementary to (and in my opinion, more clear than) the one that’s described in the paper. 

My concerns are mostly regarding the empirical studies: 

1. One of my main concern is on the empirical results in Table 1. The disentanglement metric score for beta-VAE is suspiciously lower than what’s reported in Higgins et al., where they reported a 99.23% disentanglement metric score on 2D shape dataset. I understand the linear classier is different, but still the difference is too large to ignore. Hence my current more neutral review rating. 

2. Regarding the correlational plots (the bottom row of Table 3 and 4), I don’t think I can see any clear patterns (especially on CelebA). I wonder what’s the point of including them here and if there is a point, please explain them clearly in the paper. 

3. Figure 2 is also a little confusing to me. If I understand the procedure correctly, a good disentangled feature would imply smaller correlations to other features (i.e., the numbers in Figure 2 should be smaller for better disentangled features). However, looking at Figure 2 and many other plots in the appendix, I don’t think DIP-VAE has a clear win here. Is my understanding correct? If so, what exactly are you trying to convey in Figure 2? 

Minor comments: 

1. In Eq (6) I think there are typos in terms of the definition of Cov_q(z)(z)? It appears as only the second term in Eq (5). 

2. Hyperparameter subsection in section 3: Shouldn’t \lambda_od be larger if the entanglement is mainly reflected in the off-diagonal entries? Why the opposite? 

3. Can you elaborate on how a running estimate of Cov_p(x)(\mu(x)) is maintained (following Eq (6)). It’s not very clear at the current state of the paper. 

4. Can we have error bars in Table 2? Some of the numbers are possibly hitting the error floor. 

5. Table 5 and 6 are not very necessary, unless there is a clear point. 
review_title: Interesting idea, some concerns on the empirical studies

final_score: rating: 7: Good paper, accept
review_content: This paper describes DIP-VAE, an improvement on the beta-VAE framework for learning a disentangled representation of the data generative factors in the visual domain. The authors propose to augment the standard VAE ELBO objective with an extra term that minimises the covariance between the latents. Unlike the original beta-VAE objective which implicitly minimises such covariance individually for each observation x, the DIP-VAE objective does so while marginalising over x. This difference removes the tradeoff between reconstruction quality and disentangling reported for beta-VAE, since DIP-VAE maintains sensitivity of q(z|x) to each observation x, and hence achieves disentangling while preserving the sharpness of reconstructions.

Pros:
- the paper is well written
- it makes a contribution to an important line or research (unsupervised disentangled representation learning)
- the covariance minimisation proposed in the paper looks like an easy to implement yet impactful change to the VAE objective to encourage disentanglement while preserving reconstruction quality
- it directly compares the performance of DIP-VAE to that of beta-VAE showing significant improvements in terms of disentangling metric and reconstruction error

Cons:
- I am yet to be fully convinced how well the approach works. Table 1 and Figure 1 look good, but other figures are either tangental to the main point of the paper, or impossible to read due to the small scale. For example, the qualitative evaluation of the latent traversals is almost impossible due to the tiny scale of Table 5 (shouldn't this be a Figure rather than a Table?)
- The authors concentrate a lot on the CelebA dataset, however I believe the comparison with beta-VAE would be a lot clearer on the dSprites dataset (https://github.com/deepmind/dsprites-dataset) (the authors call it 2D shapes). I would like to see latent traversals of the best DIP-VAE vs beta-VAE to demonstrate good disentangling and the improvements in reconstruction quality. This (and larger Table 5) would be better use of space compared to Table 2 and Figure 2 for example, which I feel are somewhat tangental to the main message of the paper and are better suited for the appendix. 
- I wonder how the authors calculated the disentanglement metric on CelebA, given that the ground truth attributes in the dataset are often rather qualitative (e.g. attractiveness), noisy (many can be considered an inaccurate description of the image), and often do not align with the data generative factors discoverable through unsupervised modeling of the data distribution
- Table 3 - the legends for the axes are too small and are impossible to read. Also it would be helpful to normalise the scales of the heat plots in the second row.
- Table 3 -  looking at the correlations with the ground truth factors, it seems like beta-VAE did not actually disentangle the latents. Would be nice to see the corresponding latent traversal plots to ensure that the baseline is actually trained well. 

I am willing to increase my score for the paper if the authors can address my points. In particular I would like to see a clear comparison in terms of latent traversals on dSprites between beta-VAE and DIP-VAE models presented in Table 3. I would also like to see where these particular models lie in Figure 1.

---------------------------
---- UPDATE ----------
---------------------------
I have increased my score after reading the revised version of the manuscript.
review_title: Potentially good paper, but the presentation of the results needs improvements

final_score: rating: 6: Marginally above acceptance threshold
review_content: ******
Update: revising reviewer score to 6 after acknowledging revisions and improved manuscript
******

The authors propose a new regularization term modifying the VAE (Kingma et al 2013) objective to encourage learning disentangling representations.
Specifically, the authors suggest to add penalization to ELBO in the form of -KL(q(z)||p(z)) , which encourages a more global criterion than the local ELBOs.
In practice, the authors decide that the objective they want to optimize is unwieldy and resort to moment matching of covariances of q(z) and p(z) via gradient descent.
The final objective uses a persistent estimate of the covariance matrix of q and upgrades it at each mini-batch to perform learning.

The authors use this objective function to perform experiments measuring disentanglement and find minor benefits compared to other objectives in quantitative terms.

Comments:
1. The originally proposed modification in Equation (4) appears to be rigorous and as far as I can tell still poses a lower bound to log(p(x)). The proof could use the result posed earlier: KL(q(z)||p(z)) is smaller than E_x KL(q(z|x)||p(z|x)).
2. The proposed moment matching scheme performing decorrelation resembles approaches for variational PCA and especially independent component analysis. The relationship to these techniques is not discussed adequately. In addition, this paper could really benefit from an empirical figure of the marginal statistics of z under the different regularizers in order to establish what type of structure is being imposed here and what it results in.
3. The resulting regularizer with the decorrelation terms could be studied as a modeling choice. In the probabilistic sense, regularizers can be seen as structural and prior assumptions on variables. As it stands, it is unnecessarily vague which assumptions this extra regularizer is making on variables.
4. Why is using the objective in Equation (4) not tried and tested and compared to? It could be thought that subsampling would be enough to evaluate this extra KL term without any need for additional variational parameters \psi. The reason for switching to the moment matching scheme seems not well motivated here without showing explicitly that Eq (4) has problems.
5. The model seems to be making on minor progress in its stated goal, disentanglement. It would be more convincing to clarify the structural properties of this regularizer in a statistical sense more clearly given that experimentally it seems to only have a minor effect.
6. Is there a relationship to NICE (Laurent Dinh et al)?
7. The infogan is also an obvious point of reference and comparison here.
8. The authors claim that there are no models which can combine GANs with inference in a satisfactory way, which is obviously not accurate nowadays given the progress on literature combining GANs and variational inference.

All in all I find this paper interesting but would hope that a more careful technical justification and derivation of the model would be presented given that it seems to not be an empirically overwhelming change.
review_title: The authors propose an interesting twist on regularizers for decorrelating the posterior dimensions of VAE-type models. 

Meta-review: Thank you for submitting you paper to ICLR. The reviewers are all in agreement that the paper is suitable for publication, each revising their score upwards in response to the revision that has made the paper stronger.

The authors may want to consider adding a discussion about whether the simple standard Gaussian prior, which is invariant under transformation by an orthogonal matrix, is a sensible one if the objective is to find disentangled representations. Alternatives, such as sparse priors, might be more sensible if a model-based solution to this problem is sought.
Decision: Accept (Poster)