Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Summary: This paper devised a framework towards modeling probability distributions in products of spaces with constant curvature and showed how to generalize the VAE to learn latent representations on such product spaces using Gaussian-like priors generalized for this case. Empirically the authors evaluate the VAEs on four different datasets (a synthetic tree dataset, binarized MNIST, Omniglot, and CIFAR-10) for various choices of product spaces (fixed curvature and learnable curvature) and choices of latent space dimensionality. 

Evaluation:
Overall this seems to be a nice work, with balanced discussion of the empirical results, and is clearly written.
--Past works have considered VAEs on single constant curvature spaces and hence it is well-motivated to consider a more flexible model that enables usage of products of such spaces.
--Empirical evaluations seems fair as far as I can tell, but I am not familiar with benchmarks for VAEs. It was interesting to see the variability in best performing models, e.g. cases in which the mixed curvature models did well vs. the Euclidean one.
--Paper is quite readable, though in a few parts seems to delve a bit unnecessarily into geometric formalism/definitions (e.g. I did not really follow or appreciate the relevance of gyrovector distances).
--Main text is 10 pages long and I'm not sure the extra length is necessary.
--I would have appreciated a more clearly delineated discussion on how the technical details of this work overlap with past papers, both those that have investigated product spaces (Gu et al 2019) and single curvature spaces in VAEs (spherical & hyperbolic)? How did the latter approaches deal with modified prior distributions and/or smoothly recovering the Euclidean K=0 limit? As a result, I'm a bit unsure as to the novelty or technical obstacles that are overcome in the proposed framework in comparison to these.
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: Summary: 
This paper is about developing VAEs in non-Euclidean spaces. Fairly recently, ML researchers have developed non-Euclidean embeddings, initially in hyperbolic space (constant negative curvature), and then in product spaces that have varying curvatures. These ideas were developed for embeddings, and recent attempts have been made to build entire models that operate in non-Euclidean spaces. The authors develop VAEs for the product spaces case.

There's largely two aspects here: one is to be able to write down the equivalents for the operations in models (e.g., the equivalent of adding or multiplying matrices and vectors in Euclidean space have to be lifted to other spaces which no longer have a linear structure). The other are VAE-specific choices, particularly choosing a normal distribution on the manifolds. The authors consider several of these choices and then run a variety of experiments on small latent-dimension cases for VAEs. These reveal that sometimes non-Euclidean and in particular product spaces improve performance.


Strengths, Weakness, Recommendation
I like what the authors are trying to do here; embeddings and discriminative models on non-Euclidean spaces have been developed, offer credible benefits, and generative models are the next step. The authors push forward the machinery needed to do this, and the results seem like there's something there.

On the other hand, the entire work seems quite preliminary. It's hard to say what the takeaway is, or any suggestions for users. The paper is written in a pretty frustrating way. There's an enormous amount of stuff in a sprawling appendix (there are 43 results in the first appendix?!), and checking all of these details will take a great deal of time. 

Overall, I recommended weak accept, since a lot of these issues seem like they can be cleaned up.

EDIT: I increased my score based on the authors' response.

Comments:
- The approach taken here is quite similar to another ICLR submission this year, which basically does the same thing but applies these operations to GCNs instead of VAEs.

- A better way to define curvature is just to talk about the sectional curvature, instead of the Gaussian curvature the authors mention at the beginning of section 2. Fortunately for the constant case all of these definitions will be the same.

- It's not quite clear in Section 2.1 why we should care about the fact that you can't fully take K->0 there---why does this hurt anything? You can approximate flat curvature arbitrarily well even without K exactly 0.

- On a similar theme, what's the point of doing the product of {E,S,D,H,P}, instead of just {E,S,H} or {E,D,P}? Seems a bit weird to consider all 5, given the equivalence between S-D and H-P.

- In 2.3, the products of spaces section, the distance decomposition in the 2nd paragraph should have squares (it's an l2): d_M(x,y)^2 = \sum_{i=1}^k d_{M_k_i^n_i)^2(x^i,y^i).

- The discussion in 2.3 should be expanded and made more concrete (some of these you can write out the expressions for), and more pros and cons explained, e.g., which theoretical properties are lost for the wrapped distributions?

- On page 6, I don't understand the first problem with the learnable curvature approach. Why is there no gradient w.r.t to K? Isn't the idea that you'll write this thing as a piecewise function (presumably it's continuous, since that's why the authors built those models that deform to flat), and differentiate the whole thing? Why wouldn't there be a gradient at ELBO(K)? Is it not differentiable at K=0? That doesn't follow directly from just saying the curvature is 0.

-  What's the intuition for the component learning algorithm using 2 dimensions for each of the spaces?

- The experiment section was written in a way where I couldn't understand why the choices being made were there. Why 6 and 12 dimensions here? More clarity here would be great. Also, are there any other models to compare against for these datasets? I'm not a VAE expert; what do other models typically obtain in the authors' regime?

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper introduces a general formulation of the notion of a VAE with a latent space composed by a curved manifold. It follows the current trend of learning representations on curved spaces by proposing a formulation of the latent distributions of the VAE in a variety of fixed-curvature spaces, and introduces an approach to learn the curvature of the space itself. Extensive mathematical derivations are provided, as well as experiments illustrating the impact of various choices of latent manifolds on the performance of the VAE.

I believe this work should be accepted, as while the numerical results are not particularly impressive, it provides some clear foundational work for further exploration of the use of non-euclidean latent spaces in VAEs.

This paper provides extensive and detailed theoretical grounding for their work, ensuring that it is a well-founded extension the VAE formalism. It explores numerous alternatives and compares them, providing detailed experimental results on 4 datasets. The appendices provided a much welcome refreshing on non-euclidean geometry, as well as more details & experimental results.

The paper is already quite dense, especially with the appendices, however there are a few points that could still be detailed in my opinion:

First of all, what were the observation models used for the reconstruction loss in the experiments? I suspect a bernouilli likelhood was used for the binarized dataset, but what about the other ones, and notably CIFAR? Was it a Gaussian observation, a discretized logistic, ...? Was its variance learned? This kind of information is in my opinion crucial for assessing a construction to the latent space of VAE model, as it can have a lot of influence on the kind of information the model will try to store in its latent space.

Secondly, for the model using product of spaces, do you observe some preference of the VAE to store more information in some of the sub-component? This can be explored by comparing the values of the KL term in each of these subspaces.

Third, the VAE with a factorized Gaussian euclidean latent space has a well-known tendency to sparcify its latent representations: unneeded dimensions of the latent space are ignored by the decoder and set to the prior by the encoder. This allows one to not worry too much about the size of the latent space as long as it is "large enough". Does this property remain in curved spaces? Especially in the case the VAE on MNIST with a 72-dimensional latent, as I suspect the 6 and 12 dimensional spaces are not "large enough" for this phenomenon to appear.
review_title: Official Blind Review #3

Meta-review: This paper studies generalizations of Variational Autoencoders to Non-Euclidean domains, modeled as products of constant curvature Riemannian manifolds. The framework allows to simultaneously learn the latent representations as well as the curvature of the latent domain. 

Reviewers were unanimous at highlighting the significance of this work at developing non-Euclidean tools for generative modeling. Despite the somewhat preliminary nature of the empirical evaluation, there was consensus that the paper puts forward interesting tools that might spark future research in this direction. Given those positive assessments, the AC recommends acceptance. 

Decision: Accept (Poster)