Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: Summary: In this paper, a diffusion-based VAE is proposed. The authors introduced a non-linear dimension reduction method, diffusion map to standard VAE to encode neighborhood manifold information into the latent space before encoding. They proposed a lower-bound objective, similar to original VAE for content generation. 

Pros: 1) A new VAE method is proposed to incorporate local manifold information into the latent space. 2) A sufficient condition to measure the consensus between latent and input data distributions. 3) Three empirical studies on the visualization of generated images. 

Cons: 
1) The writing could be significantly improved. There are a bunch of grammar errors and confusing notations. For example, “with many default priors the posterior/likelihood pair q(z|x)/p(x|z) can be viewed as an approximate homeomorphism” ->  “with many default priors,  the posterior/likelihood pair q(z|x)/p(x|z) that can be viewed as an approximate homeomorphism”. “In this paper address issues in variational inference and manifold learning” -> “In this paper, we address issues in variational inference ….”. “feedfoward pass” -> “feedforward” …. In algorithm 1, “X is a random batch from X”. Conditional probability are mixed with joint probabilities, “p(y|x) = p(x,y)”. I suggest the authors do careful proofreading.

2) The novelty in this paper is limited.  The diffusion VAE was proposed in Rey’19 https://arxiv.org/abs/1901.08991 with a similar random walk procedure with transition kernels on the manifold of input. However, the authors neither did any comparison with it nor provided convincing advantages over them. 

3) The authors claimed the standard VAE has too many assumptions on the priors, likelihood, and posteriors. However, their framework also assumed Gaussian distribution on posteriors and likelihood, only eliminating the prior distribution, but at the expense of introducing an assumed kernel and eigendecomposition approximation. 

3) The experiments are very limited, containing only 3 visualization results of 3 image generation tasks. The Fig 2 is difficult to read and interpret. How about the log-likelihood estimates from your approach compared with others? 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes a new generative model for unsupervised learning, based on a diffusion random walk principle inspired by the manifold learning literature. The basic idea is to (probabilistically) map points to a latent space, perform a random walk in that space, and then map back to the original space again. Learning of the suitable maps is achieved by casting the problem in a variational inference framework.

The paper is generally well-written, and clearly states out its goals and motivation. Sections 1 - 3 in particular give a nice overview of the broader context of the paper, and its aim of borrowing ideas from manifold learning and variational autoencoders. The particular aim on using concepts from manifold learning to avoid mode collapse - corresponding to the underlying homeomorphism losing its bijectivity - is in particular intriguing.

The method itself is intuitive at a high level, although I did have some difficulty with Section 4:
- in 4.1, one begins by considering the local evidence. This requires drawing a point from U_x, which is defined to a be set. I presume this means one draws uniformly from this set?
- Eqn 3 does not apparently have the same structure as Eqn 2. In particular, the first term in (2) is a function of x, but for (3) it is not a function of x'. How does conditioning affect the ELBO?
- I was not sure how to interpret the statement that pθ(x'|x) ≈ ψ^{-1}(q(z'|x)). Do you mean the distribution is strongly concentrated around this value? Note also an extra "]" in the latter.
- Eqn 5 should presumably be an equality? Also, it was not clear what the d in |.|_d^2 means, and why one does not use ||.||^2.
- At a higher level, given that x' ~ U_x originally, why do we now draw x' ~ p(x'|x)?
- Arriving at 4.2, it was not clear what "The sampling procedure" refers to, i.e., which of the steps in 4.1 it is seeking to specify or augment. It would be useful to clearly lay out the objective function that is being optimised, and how this section fits into that.
- In 4.3, it seemed as if the discussion of the neighbourhood reconstruction error would be better placed in 4.1 itself. It appears to be a justification of the already-derived Eqn 5.
- Algorithm 2, is there a need to introduce Z_t? It is a bit confusing that, e.g., Z_1 is first written to in iteration 1 by g(Z_0, ε), and then by ψ(X_1) in the second iteration.

The authors also claim a contribution to be the identification of a principled measure to identify mismatch between latent and data distributions. This "bi-Lipschitz" property is only introduced in Sec 5.1, and the discussion is not too approachable to someone unfamiliar with the area. In particular:
- it is not clear how precisely the discussion in this section relates to the VDAE algorithm described in the previous section.
- precisely what quantity we are to compute so as to verify this condition remains elusive. The abstract and introduction made me expect that the property is practically verifiable, but it was not clear from this section whether it is so.
- the conclusion or key takeaway of this subsection was unclear. I gather that Jones et al. established the existence of a neighbourhood wherein one can define a bi-Lipschitz mapping to R^d for suitable d. But how does this relate to latent and data space mismatch?

The experiments show that the proposed method can generate meaningful samples for synthetic manifold data, as well as on the MNIST dataset. I would've preferred more discussion of the results in Sec 4.1. I also was hoping for a clearer illustration of mode-collapse problems on standard benchmarks for GANs, with comparison of results to, e.g., those of Wasserstein-GANs (beyond those in Sec 6.1) or other proposals that aim to mitigate mode collapse.

Minor comments:
- Fig 1, the text in the middle panel is hard to read in black and white.
- SpectralNet is mentioned a few times but never formally introduced.
- notationally, Section 4 is a little heavy. I would suggest considering to omit the subscript θ's in the function ψ and its inverse.
- when mentioning the "reparametrisation trick", please provide a citation.
- what is \mathbb{X} in Theorem 2?
- "completementary" -> "complementary"
- " rejection sampling Azadi et al. (2018); Turner et al. (2018)." -> " rejection sampling (Azadi et al. 2018; Turner et al. 2018)."
- some form of Conclusion would be appropriate.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: ** Summary
The paper studies the problem of density estimation and learning accurate generative models. The authors start from the observation that this problem has been approaches either using variational inference models, that scale very well but whose approximations may lead to degenerate results in practice, and diffusion maps, that scale poorly but are very effective in capturing the underlying data manifold. From here, the authors propose integrating the notion of random walk from diffusion maps into VAEs to avoid degenerate conditions. The proposed method is first defined in its generality, a practical implementation is presented, theoretical guarantees are provided, and empirical evidence of its effectiveness is reported.

** Evaluation
The starting point of the paper seems very solid: diffusion maps are capturing the geometry of data very effectively and bringing some of those characteristics into the more scalable approach of VAE is an interesting approach. In the proposed method, this translates into introducing a learned diffusion map from manifold to an Euclidean space into the inference part. As a result, the lower bound optimized by the method now contains local information about the accuracy of the one-step random walk. How this can be translated into a practical implementation is also convincing. My main concern is that the overall method is now approximating many different elements in the original formulation, such as the diffusion map, its inverse, and the covariance of the random walk. Although theory seems to support that as these approximation become more accurate the overall result is reliable, in practice I wonder how they could combine and deteriorate the final result.

The empirical validation is relatively simple but in my opinion it provides enough insights about the advantages of the proposed method compared to VAEs and GANs. More solid and extensive evaluation is definitely needed in the future to have a more thorough comparison and a more careful assessment of the limits of the proposed method, but at this stage, I think the evaluation is sufficient.
review_title: Official Blind Review #3

Meta-review: This paper proposes to train latent-variable models (VAEs) based on diffusion maps on the data-manifold. While this is an interesting idea, there are substantial problems with the current draft regarding clarity, novelty and scalability. In its current form, it is unlikely that the proposed model will have  a substantial impact on the community.
Decision: Reject