Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to use Riemannian geometry to analyze the latent space of variational autoencoders (VAE).  Specifically, the authors define a Riemannian Gaussian distribution on the learned latent space. Based on this, a novel sampling process is proposed in which samples from the uniform distribution are defined on the learned Riemannian latent space. Experiments show that the proposed sampling method achieves competitive results on several benchmark datasets.
strengths_and_weaknesses: Strengths:

1. This paper is well written and easy to follow. The authors clearly explained the mathematics behind the geometrical interpretation of the VAE framework. It is very helpful that the authors give a brief review of Riemannian geometry and the Riemannian Gaussian distribution.

2. Several experiments with different metrics show that the proposed method is effective and robust. 


Weaknesses:

1. While I understand that this work focuses on the sampling process for vanilla VAE without modifying the training procedure, I think this sampling process can be seen as a second stage for estimating the learned latent distribution of vanilla VAE. For comparison, in two-stage VAE [1],  they achieve 34 for FID on CelebA with the second stage of training which is approximating the learned latent distribution by vanilla VAE.  Some discussion/comparison between these methods would be helpful.

[1] Dai, Bin, David Wipf, and Minhao Jiang. "Diagnosing and Enhancing VAE models (ICLR 2019)."
questions: 1. It would be interesting to see some discussion about how the proposed method can benefit other more recent VAE-based methods rather than only the vanilla VAE. 
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper takes a geometric view on the classic VAE model by reinterpreting the latent space through the lens of Riemannian manifolds. It proposes a new sampling technique utilizing the learned manifold and Riemannian metric and empirically shows the method quantitatively improves sample quality on several datasets over various standard VAE baselines.
strengths_and_weaknesses: **Strengths:**
- [**quality**] the elaborate background on Riemannian geometry and extensive appendices nicely introduce and illustrate the proposed solution. The technical derivations are also clearly presented and well written. The empirical results show improvements over classic VAE baselines.
- [**originality**] while there are increasingly many works around enriching the VAE framework by extending it to use more complex distributions, e.g. normalizing flows, or adding distributions defined on manifolds with a non-trivial geometry, e.g. hyperspherical/hyperbolic, the angle proposed in this work adds a Riemannian geometry perspective in a novel and empirically useful manner.

**Weaknesses**:
- [**clarity**] the at times non-standard English makes this paper sometimes challenging to read. Other formatting choices could likely strengthen readability as well, e.g. section 5. (see next section for suggestions on possible sentences, subsections that could perhaps be improved upon).
- [**significance**] as noted by the authors in the introduction, one of the major reasons for the classic VAE’s sustained appeal for modern machine learning tasks is its simplicity and ease of use. This surely contributes to it often still having an edge over more ‘recent methods’, e.g. diffusion based models, with higher quality results. As such, the complexity of the proposed method is likely to limit its adaptation, especially if it can’t be shown to perform on par with state of the art methods.

questions: **Suggestions**:
- 1, line 22-23: perhaps change ‘inducing over-regularization’ with ‘manifold/distribution mismatch’
- 4.3, line 182-185: it is stated that the Beta term to weigh the influence of the KL divergence on the total loss is predominantly seen as a reguralizer, similar to [18]. And that ‘the proposed vision [is kept] and [not amended] for training’. It would increase clarity to briefly touch upon the fact that while the ultimate outcome of the KL-loss as a regularizar is similar, the justification of [18] is very much different.
- 5.1: given the existence of appendix D, perhaps transfer the description of baseline models and benchmark datasets here to focus the section on results discussion. Splitting up the discussion between paragraphs separated by a line of white space would likely make it easier for the reader to grasp the main experimental take-aways.

**Possible Typos / formatting suggestions**:
- Abstract, line 2: *we indeed argue..* → we argue
- 1, line 13: *map into* → map to
- 1, line 17: *these models produce .. samples* → these models produce blurry samples on image generation tasks most of the time
- 1, line 35: *improves significantly…* → improves the generation process from a vanilla VAE significantly
- 1, line 36: *the training ..* → the training. The proposed sampling method outperforms more advanced VAE …
- 4.3, line 133: *voluntarily chosen* → (often) chosen?
- 4.3, line 134: *Hence, the … as follows.* → Hence, the .. as follow:
- 4.3, line 144: *As first approximation and since … the VAE model* → Since …, as a first approximation the VAE model
- 4.3, line 147: *This simplified drastically* → This drastically simplified
- 4.3, line 182: *that, likewise [18],* → that like [18]
- 4.6, line 202: *may be easily appended in Figure* → can be observed/seen in Figure
- 4.6, line 231: *below are presented….* → refer to figure number
- 6, line 309: *Then, we proposed* → We proposed
- 6, line 311: *It showed to be competitive* → The proposed method was empirically shown to be competitive

limitations: The authors are careful to highlight certain practical ‘shortcuts’ of their method where it deviates from theory and the consequences, e.g. section 4.3 starting line 170. They also spent considerable effort in the various appendices, e.g. appendix B and F, to further provide practical implementation details/considerations of their method. While the potential negative societal impact of this specific work seems small, it is not entirely clear why the authors highlight appendix C as answering this question. The only line that seems relevant here is C.3. 100-102?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Authors identify a problem with the standard sampling process of a VAE. They then interpret the mean and variance predicted by a vanilla VAE encoder as the tangent space of a Riemannian Manifold. Under this interpretation, the authors then sample uniformly from the space.
 
Unfortunately, I do not have a good understanding of Riemannian manifolds, so I cannot evaluate the validity of the method described by the authors (if the U_Riem is indeed a valid distribution, for example).
 
However, assuming the validity of this method, it is important to measure empirically how good the samples derived from this method are under several metrics and datasets. The authors perform several experiments to this end in the Experiments section.

strengths_and_weaknesses: 
Strengths:

I am not fully certain of this, but to my knowledge, this is a novel way of interpreting the latent space of the VAE, and leveraging the encoder predictions for improving sampling.

Weaknesses:

The authors would make a better case of their method if they took an _existing_ trained model that already does well, and show that their method of sampling achieves better samples.

Relatedly, the baselines appear to be implemented by the authors. While I understand this is for an “apples to apples” comparison against equivalent latent dimension sizes and architectures, it is hard to know for sure that the baselines will do better under some hyperparameter tuning, thereby weakening the strength of the results

Minor: the language could use a second pass, some grammatical errors:
- Line 17: “Nonetheless, when taken in their simplest version, it was noted that these models produce blurry samples most of the time.”
- Line 20: “Second, the prior distribution used…” run on sentence. -
- Line 34: “We show that this procedure significantly improves the generation..”
- Line 261: “This supports the fact that even the simplest VAE model”
- List is non-exhaustive


questions: 
Have experiments been performed with pre-trained models that already perform comparably with SOTA models, and does the new sampling method work better on top of those results?

limitations: 
A better sampling method for image generation would have wider implications on misinformation in the public sphere. However, I believe the empirical results are not strong enough to substantiate this as a worry for this particular work at the moment.
review_title: null

Meta-review: metareview: The submission proposes a geometrically motivated method to sample from a trained variational autoencoder by defining a Riemannian metric on the latent space and a corresponding distribution to correct the sampling process of VAEs. According to most reviewers and reading the submission, the method is clearly explained and provides a consistent improvement in sample quality according to FID. The authors have mostly addressed the reviewers concerns as well. 
I recommend this paper for acceptance.
confidence: Certain
Decision: Accept