Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 1.	The idea of explicitly forcing the encoding space to be flat by putting constraint on metric tensor is simple but neat.
2.	The use of Jacobi regularization in Eq. (9) is effective but the choice of using interpolation to extend this in the entire decoding space is kind of adhoc. Can authors please justify?
3.	Not sure how authors put the Lipschitz continuity constraint on f. Please explain. 
4.	The title of “Flat manifold VAEs” is misleading as it potentially means VAEs for flat manifold 
5.	I wonder what will happen if you put an unfolding constraint in the encoding space like LLE, ISOMAP etc.. The loss function is data driven so this should give atleast similar behavior.
6.	Overall I like the experimental setup, but the tracking experiment is kind of distracting. The authors may want to remove this experiment.
7.	In Fig. 7, the authors have shown with and without  Jacobi normalization which I am really not convinced with, need better explanation.  

review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: Summary of paper:
The paper is concerned with the geometry of latent spaces in VAEs. In particular, it is argued that since geodesics (shortest paths) in the Riemannian interpretation of latent spaces are expensive to compute, then it might be beneficial to regularize the decoder (generator) to be flat, such that geodesics are straight lines. One such regularization is proposed.

Review:
I have several concerns with the paper:

1) Geodesics are never motivated:
The paper provides no motivation for why geodesics are interesting objects in the first place, so it is not clear to me what the authors are even trying to approximate.

2) Under the usual motivation, the work is flawed:
The usual motivation for geodesics is that they should follow the trend of the data (e.g. go through regions of high density). Since no other motivation is provided, I will assume this to be the motivation of the paper as well. The paper propose to use a flexible prior and then approximate geodesics by straight lines. Beyond the most simple linear models, then this cannot work. If the prior is flexible, then straight lines will hardly ever constitute paths through regions of high density. The core idea of the work, thus, seem to be in conflict with itself.

3) A substantial bias is ignored:
The paper consider the Riemannian metric associated with the *mean* decoder. Due to regularization, holes in the data manifold will be smoothly interpolated by the mean decoder, such that geodesics under the associated metric will systematically be attracted to holes in the data manifold. Hauberg discuss this issue in great length here:

  https://arxiv.org/abs/1806.04994

Here it is also demonstrated that geodesics under the mean decoder tend to be straight lines (which is also what the authors observe). Taking the stochasticity of the VAE decoder into account drastically change the behavior of geodesics to naturally follow the trend of the data.

4) Related work is mischaracterized:
Previous work on the geometry of latent spaces largely fall into two categories: those that treat the decoder as deterministic and those that treat it as being stochastic. In the cited papers Arvanitidis et al and Tosi et al consider stochastic decoders, while the other consider deterministic decoders. Given that geodesics have significantly different behavior in the two cases, it is odd that the difference is never discussed in the paper.

5) It is not clear to me what the experiments actually show:

-- I did not understand the sentence (page 5): "The model is more invariant if the condition number is smaller..." What does it mean to be "more invariant" ? And how is invariance (to what) related to the condition number of the metric?

-- Figure 3 show example geodesics, but only geodesics going between clusters (I have no idea how such geodesics should look). If I look at the yellow cluster of Fig3a, then it seems clear  to me that geodesics really should be circular arcs, yet this is being approximated with straight lines. Are the ground truth geodesics circular? At the end, it seems like the shown examples are the least informative ones, and that intra-cluster geodesics would carry much more meaning.

-- What am I supposed to learn from the "Smoothness" experiment (page 7) ? My only take-away is currently that the proposed regularization does what it is asked to do. It is not clear to me if what it aims to do is desirable? Does the experiment shed light on the desirability of the regularizer or is it more of a "unit test" that show that the regularizer is correctly implemented?

-- In the "Geodesic" experiment (page 7) I don't agree with the choice of baseline. If I understand correctly, the baseline approximate geodesics with shortest paths over the neighbor graph (akin to Isomap). However, there is no reason to believe that the resulting paths bare any resemblance to geodesics under the studied Riemannian metric. The above-mentioned paper by Hauberg provide significant evidence that these baseline geodesics are not at all related to the actual geodesics of the studied metric. The only sensible baseline I can think of is the expensive optimization-based geodesics.

== rebuttal ==
I have read the rebuttal and discussed with the authors, and I retain my original score.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: Notes: 

  -This paper suggests the use of VAEs with stronger priors along with more powerful regularization of the decoder (especially its curvature).  This lowering of curvature is seen as "flattening".  

  -Paper uses the normal VAE ELBO.  

  -The normal prior over-regularizes the approximate posterior.  One proposal is to use a "hierarchical prior": integral(p(z|zeta)*p(zeta), zeta) where zeta is a normal distribution.  So basically a function can transform the prior.  

  -Importance weighting with q(z|x) has been proposed as a way to define a valid learning objective for this setting.  

  -Another objective using lagrangian is called "VHP-VAE".  

  -This paper extends VHP-VAE with jacobian regularization, which is approximated (the paper doesn't say so but I think it's a first order taylor expansion).  

  -Paper also uses mixup in the latent space to provide regularization at points farther from the data.  

  -With this mixup objective the mixing is also done to consider extrapolations in addition to interpolations.  

  -The resulting latent space does indeed look much better (Figure 1).  

  -The condition number is also way better (2a, 2b).  

  -In figure 2, the background color indicates the degree of magnification (so the VAE-VHP has greater variability in distances?)  I found this figure a bit hard to itnerpret.  

Comments: 

  -This paper cites Mixup but there are two more papers to consider here: Manifold Mixup (ICML 2019) and Adversarial Mixup Resynthesis (Neurips 2019) which both considered mixing in a latent space.  AMR considered in an autoencoder, and Manifold Mixup is also relevant because its theoretical analysis explicitly considers flattening although in a somewhat difference sense (and both are different from what's done here).  

  -The object tracking experiments don't seem very convincing to me (just looking at table 2 at least).  

Review: 

  This paper considers augmenting the hierarchical VHP-VAE with a criteria in which the jacobian is approximately regularized at interpolations and extrapolations between different points in z space.  The experiments suggest this is an important problem with VHP-VAE and also that it's successfully addressed.  
review_title: Official Blind Review #1

Meta-review: The paper proposes to regularize the decoder of the VAE to have a flat pull-back metric, with the goal of making Euclidean distances in the latent space correspond to geodesic distances. This, in turn, results in faster geodesic distance computation. I share the concern of R2 that this regularization towards a flat metric could result in "biased" geodesic distances in regions where data is scarce. I suggest the authors discuss in the next version of the paper if there are situations where this regularization might have drawbacks and if possible, conduct experiments (perhaps on toy data) to either rule out or highlight these points, particularly about scarce data regions. 
Decision: Reject