Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper presents a deep, latent variable model for unsupervised data modeling problems. The problem with such latent, deep generative models is that they are difficult to train reliably. In this paper, the authors provide an approach based on stacked Wasserstein autoencoders to train deep latent variable models. Experimental results are demonstrated on various image datasets and the latent codes are demonstrated to have an interpretable meaning. 
I like the inference techniques in the paper and like the ideas presented in this paper.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, a hierarchical extension to Wasserstein Autoencoders (WAE) is proposed, where the latent variables are stacked in a multi-layer structure. In the proposed model, the divergence function in WAE is viewed as a relaxed WS distance. Therefore, another layer of WAE can be stacked to minimise the WS distance. In this way, a hierarchical model can be built to learn hierarchical representations.

I think the idea of viewing the divergence in WAE as a relaxed WS distance and then minimising it with another WAE structure is interesting, intuitive and straightforward. However, the advantages of the proposed model over WAE and VLAE (S.Zhao et.al 2017) are less obvious to me. It is a bit hard for me to tell whether the hierarchical latent variables help to improve quantitative results, generate better images, or learn intuitive hierarchical representations, which is the main reason that I go to mild rejection.

For example, I would expect to see similar things as in VLAE, where the representations in different layers capture hierarchical structures or disentanglements. But in the proposed model, it seems to be hard to see the differences between the hierarchical representations such as in Figure 3(b). Also in the two-dimensional visualisation of Figure 3(a), it is a bit hard for me to intuitively understand what the representations really capture. 

From the graphical model point of view, the proposed model is a hierarchical Gaussian model and the inference (although with WAE) is in the flavour of Gibbs sampling, which propagates information layer-wisely from bottom up. Conventionally, a hierarchical Gaussian model is hard to work with many layers such as 5. Therefore, I may suggest improving in case of fewer layers.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper aims to develop a deep generative model, which -unlike VAEs or GANs- comprises a hierarchy of latent variables rather than a direct map from the stochastic latent manifold to the observation space. To this end, the paper builds a training objective based on nesting the Wasserstein distance between the data distribution and its estimation arbitrarily many times. The generated objective corresponds naturally to a deep hierarchical generative model.

The principled approach followed to achieve the objective is solid and elegant. It is also intuitive and matches nicely with some valid observations highlighted in the paper such as insufficiency of by-passing intermediate latent variables (sentence above the Sec 2.3 title).

One major weakness of the paper is that it lacks a sufficient argumentation about how it differentiates from earlier attempts to nest Wasserstein distances. For instance,

Y. Dukler et al., "Wasserstein of Wasserstein Loss for Learning Generative Models", ICML, 2019

Apart from the theoretical argumentation, the paper should also compare their solution to this prior work on a number of benchmarks.

Another major weakness is that the paper lacks a quantitative evaluation scheme for its success. The experiments section starts with the claim that the proposed method "significantly" improves on the WAE, which I fail to see on the plots. 

Lastly, Having said that the proposed method is novel and elegant, it is still a straightforward extension of the existing and well-known Wasserstein Auto-Encoder (WAE) approach. It extends WAEs by repetitively applying the tricks proposed by this earlier work, putting aside some minor additional adjustments.

Minor on style: The abstract does not give any single hint about the methodological novelty of the work.

---
Post-rebuttal: Thanks to authors for their effort for clarifications. Yet, I'm afraid the author response does not touch at all to any of the concerns I have raised. There are well-known ways to compare the success of generative models, FID being one of them as the authors point out. Another could be the test log-likelihood of a synthetic data set the true distribution of which can be predesigned. I understand the issues the authors raise about the difficulties in comparing generative models, but I kindly disagree with the attitude that there are no ways to compare, so we are obliged to live with qualitative comparisons. If a one-score comparison is not enough, the right way to go is to provide multiple scores. If direct metrics are not feasible, one should go for indirect ones, but should still provide outcomes a reader can reproduce.
review_title: Official Blind Review #1

Meta-review: The paper received 6, 3, 1. The main criticism is the lack of quantitative evaluation/comparison. The rebuttal did not convince the last reviewer who strongly argues for a comparison. The authors are encouraged to add additional results and resubmit to a future venue.
Decision: Reject