Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper extends the Wasserstein Autoencoder (WAE) work by splitting the divergence on the variational marginal into 2 terms, akin to what was done in TC-VAE. This enables directly controlling the explicit contribution of the total correlation term, which is likely to contribute to disentanglement more directly. They explore 2 variations of their model, based on different estimators of the TC term (TCWAE-MWS, using minibatch-weighted sampling; TCWAE-GAN, using a density ratio trick).

Overall, I found this work to be a nicely complete exploration of a simple extension of an existing framework. They mostly rederive existing methods in the WAE framework, but results are promising and the paper addresses several datasets, compares to baselines well and seems well executed. It seems to lack comparison and discussion to a paper which seems directly related [Xiao et al 2019]. But I feel this is still a worthy piece of research to showcase at ICLR.

Questions/comments:
1. A cursory search indicated the following paper which also addresses disentanglement with the Wassertein Total Correlation: [Xiao et al 2019]. They use another estimator of the TC, instead opting for the Kantorovich-Rubinstein formulation.
   1. Can you comment on how their work relates to this current paper?
   2. A direct comparison would be rather interesting, but might be out of scope for a rebuttal.
2. Reconstructions for the TCWAE-MWS appear rather bad (Figures 13-19 in the Appendix ), but Figure 1c doesn’t seem to reflect that, which is slightly surprising.
   1. Could you comment on this discrepancy?
3. Relatedly the TCWAE-GAN disentanglement doesn’t seem particularly exciting (metric-wise), would you still recommend using it instead of TCWAE-MWS? 
   1. It is still a clear improvement over vanilla WAE, so there’s value to the work in this current state; but I’d wonder when one would prefer choosing this versus TCVAE?
4. It might be appropriate to discuss 2-Stage VAE [Dai et al 2019] and associated family of models, which currently obtain really good results on more complex datasets.


References:
*	[Xiao et al 2019] https://arxiv.org/abs/1912.12818
*	[Dai et al 2019] https://arxiv.org/abs/1903.05789

review_title: Official Blind Review #5

final_score: rating: 5: Marginally below acceptance threshold
review_content: This submission proposes to add a KL term to the Wasserstein auto-encoder objective in order to improve its disentanglement capabilities.
This combines the idea of Hoffman & Jonhson (2016) of using a marginal KL term, with the Wasserstein auto-encoder framework.
Challenges regarding the estimation of the KL term are also addressed with two previous works. 
This results in a two regularization parameter objective, whose superiority to existing approaches (using a single parameter) is not clear.

Strengths: WAE with a disentanglement term was as far as I now not attempted before, the authors offer two well justified techniques to do it.

Weaknesses: (1) The work is very iterative, existing approaches are only combined, (2) Superiority to WAE without this term is not surprising, and I failed to see a clear superiority to competing unsupervised disentanglement approaches. (3) Given the emphasis on the Wasserstein distance of the original approach, it is also a bit disappointing to resort to a KL term for disentanglement. (4) Most importantly, comparison to simpler alternative KL (non-marginal) losses is absent as far as I can tell. That was for me the most interesting appeal of the paper.

Overall, I tend to think the paper would require a more exhaustive investigation of disentanglement approaches, contextualized to the Wasserstein distance and issues raised regarding marginal versus non-marginal divergences. I recommend rejection.

On this last point: It remains unclear to me whether the original hypothesis of the paper (page 3), that the index-code MI term of the KL divergence may be detrimental to disentanglement, is supported by the current study, and thus whether the extra technicalities required to eliminate it are worth the effort. Perhaps the authors could elaborate on that with an alternative objective close to the classical KL term, and thus easier to optimize?


review_title: An well written study, but iterative and without very convincing results 

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses disentanglement in the latent space of autoencoders. To this end, it combines ideas from four existing papers, namely the reconstruction loss of the Wasserstein autoencoder, the regularization term decomposition from the total correlation autoencoder, and entropy estimation using minibatch-weighted sampling or the density-ratio trick. This combination certainly makes sense, as it brings together methods that have previously been shown to work well in isolation.

The main part of the paper is devoted to an empirical evaluation of the new autoencoder training procedure. The new method is compared against various baselines in terms of L2 reconstruction error and three disentanglement scores on four toy datasets. In addition, latent space traversals on 3Dchairs and CelebA are shown to qualitatively demonstrate the disentanglement capabilities of the proposed methods.

Unfortunately, the description of the experiments is not very precise. 
* The role of the hyperparameter gamma remains unclear. In the ablation study, the authors simply set gamma=beta without further explanation, and in the comparison, they just state "we first tune gamma" and "for gamma >1, better disentanglement is obtained", again without further explanation.
* In the comparison experiment, they report results for the values of beta that achieve "an overall best ranking on the four different metrics" without explaining what an "overall best ranking" is. Choices like this must not be taken lightly, as the analysis in "Why rankings of biomedical image analysis competitions should be interpreted with care" (Nature Communications 9: 5217, 2018) impressively demonstrates.
* The experiment in figure 2 seems to have three degrees of freedom (the data instance x, the latent index i, and the size of the modification in direction z_i). However, only two degrees of freedom are shown, and it remains unclear from the caption and associated main text, which ones. Moreover, I cannot deduce justification for the statement "all methods .. learn to disentangle, capturing four different factors" from the figure -- I do not see any obvious disentanglement.

The bigger problem with the paper, however, is the question: What have we learned from these experiments? The rankings in table 1 are pretty inconsistent between different metrics, and the corresponding figure 3 appears to be cherry picked, as the ScreamdSprites is the dataset where the proposed methods perform best.

I also do not agree with the claim that "TCWAEs achieve good disentanglement" on real-world datasets. Figure 4 shows severe entanglement between unrelated factors. For example, the size feature for the chairs also changes the type of chair. All features in the CelebA examples have a tendency also to change the background appearance. The gender feature dramatically influences person identity in the MWS results, whereas it does not change the gender at all in the GAN variant. Substantial variations in person identity are also visible in most other examples.

In summary, while the paper provides numbers, it lacks new insight. In light of mathematical proofs indicating that the true generative factors are generally unidentifiable in non-linear unsupervised settings (cf. the work of Aapo Hyvärinen and others), I am skeptical that heuristic trial-and-error investigations of disentanglement like the present one will yield interesting results. In a sense, this is also acknowledged by the authors, who merely state in the conclusion that "our methods achieve competitive disentanglement on toy data sets" -- that's not much, given the effort that went into the experiments.
review_title: Experimenting with disentanglement, the paper merely reports numbers, but lacks new insight.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
The paper is motivated by the need for a better trade-off between the reconstruction and disentanglement performance of an autoencoder. The proposed solution is to use KL as a latent regularizer in the framework of Wassestain autoencoders, which allows for a natural interpretation of total correlation.


The paper reads well, all related work and relevant background concepts are nicely integrated throughout the text. The experiments are exhaustive and the results show competitive performance wrt disentanglement while improving reconstruction/modeling of the AEs.

If a dataset is of dynamical nature, how difficult would it be to extend the current version of TCWAE to dynamical systems? Do the authors have any intuition/hint on what should change to make their method applicable to dynamical setups? Significantly changing the probabilistic model or modifying only the and encoder/decoder architecture could suffice? 

Minor:
- Consider changing the naming of the baselines either in tables or figures to make them consistent  Chen et al (2018) -> TCVAE Kim & Mnih (2018) -> factorVAE.
review_title: Official Blind Review 

Meta-review: There were both positive and negative assessments of this paper by the reviewers: It was deemed a well written paper that explores cleanly rederiving the TC-VAE in the Wasserstein Autoencoder Framework and that has experiments comparing to competing approaches. However, there are two strong concerns with this paper: First, novelty appears to be strongly limited as it appears a rederivation using known approaches. Second, two reviewers were not convinced by the experimental results and do not agree with the claim that the proposed approach is better than competing methods in providing disentangled representations. I agree with this concern, in particular as assessing unsupervised disentanglement models is known to be very hard and easily leads to non-informative results (see e.g. the paper cited by the authors  from Locatello et al., 2019). Overall, I recommend rejecting this paper.
Decision: Reject