Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper introduces and studies a new type of hyperbolic latent space called Gaussian manifold in the context of variational auto-encoder in order to tackle numerical and sampling issues with existing hyperbolic latent spaces. To make this latent space works in practice, the authors designed a geometric transformation at the last encoder step and the first decoder step; as well as a novel distribution over Gaussian manifold called a pseudo Gaussian manifold normal distribution that makes use of information geometry, easy to sample from, and easy to compute KL divergence. Finally, comprehensive testing on empirical datasets are carried out to highlight the effectiveness of the scheme. 





strength_and_weaknesses: Strengths

1. The proposed method is novel, interesting and makes use of clever engineering details that are rather intuitive.
2. Empirical results clearly indicate that the proposed method is superior in terms of numerical stability to existing hyperbolic latent space methods, over certain datasets (such as binarized-Breakout). In fact, in my opinion, the main selling point of the method should be in terms of numerical stability and the authors may consider bringing the discussion in Appendix E into the main text. 
3. Empirical results in terms of negative test likelihood do not suffer, compared to other methods. 
4. The introduction to hyperbolic latent space is comprehensive and self-contained. Overall I find the language clear and the flow easy to follow, despite small typos. 

Weaknesses

1. Some notations are not clearly defined. In figure 1, the calligraphic E space (I suspect is Euclidean space) is not defined and the calligraphic L^2 space is only defined in passing in subsection 3.3. I suggest that the authors at least make the notation in the figures and its caption self-contained.  What does the triangle in Table 1 mean? How is it different from a circle?
2. Although a lot is mentioned about information geometry and the incorporation of closed form KL, the paper does not clearly mention (or hint at) specific mathematical guarantees for the proposed model. As a result, a lot of the justifications for the proposed method are heuristical and do not form concrete theoretical backings. 
3. Empirical results are rather limited in quantity. There are only a total of 3 datasets and the authors only demonstrate better stability in 1 of these 3 datasets. The proposed methods also do not seem to scale as well as competing methods when increasing the latent space dimension.

clarity,_quality,_novelty_and_reproducibility: 1. Clarity:
The introduction and background sections are well written and comprehensive and the paper as a whole is self contained. However, some notations are not clearly defined, especially in figures. 
2. Quality:
The authors identify several challenges in the current use of hyperbolic latent space for VAE and proceed to tackle them with some success. However, I would like to see more empirical evidence that support the strengths of the method (currently one ⅓ datasets is in favor of the proposed method)
3. Novelty:
Aspects of the new hyperbolic structure have been studied for a while (as a statistical manifold). The novelty comes from the geometric transform and the proposed distribution on the Gaussian manifold.
4. Reproducibility:
Experiments in the papers are described in detail and the code is given in supplementary material. I have not run the code myself due to lack of time but I believe that the results are reproducible, or otherwise can be quickly shown to be wrong. 

summary_of_the_review: Despite the paper containing some interesting and novel ideas on how to make the Gaussian manifold suitable for VAE setting, I find that the theoretical motivation for said ideas is heuristical at best and there is not much empirical evidence to demonstrate the strength of the proposed method. Therefore, I recommend rejection.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, a pseudo Gaussian manifold normal distribution is proposed as the prior of latent representation of data, and the latent representation of data is learned in the framework of VAE, so as to realize a Gaussian manifold learning.
strength_and_weaknesses: Strength：

The idea to learn a new manifold for data is interesting.


Weaknesses:

1. The motivation of the methods in the manuscript is not clear and most hypotheses lack explanation. For example, why use diagnal Gaussian distributions.

2. The authors only introduce different methods by plain descriptions and isolated formulas, and does not analyze or summarize the characteristics of different methods in manifold learning. For example, what scenario does the Riemannian Gaussian distribution apply to, and what scenario does the wrapped normal distribution apply to？

3. The authors directly give the distribution expression of PGM in (5) without explaining its geometric meaning.

4. How to get the  conclusion of ”With this aspect, GM-VAE can be considered as a hierarchical VAE with an additional prior over the Gaussian prior“？

5. The experimental results are not convincing. In Table 2，the proposed method does not perform as well as P-VAE and the authors do not make any anlysis between P-VAE and GM-VAE.

6. For the results of visualization of latent space such as in Figure 2 and Figure 3, how to evaluate them?

7. What's the difference between the proposed GM-VAE and some Gaussian-process-based manifold learning methods, such as GPLVM or VAE-DGP?

clarity,_quality,_novelty_and_reproducibility: The purpose or geometric meaning of the method is not clear.
The idea of this work is somewhat novel.

summary_of_the_review: The manuscript presents an interesting idea, but the details and rationality of the hypothesis are not clear, and the experimental results are difficult to support and explain the hypothesis.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: 
The paper defines PGM-normal, a distribution over the family of diagonal Gaussian distributions, which forms a manifold. With this distribution, they propose a variant of VAE whose latent space is this manifold. They propose two transformations for the VAE to map between Euclidean space and the manifold. Empirically, they show that their method matches the performance of existing hyperbolic VAEs.
strength_and_weaknesses: Weakness: 

(1) The hyperparameter $c$ only changes the scaling of $\mu$ and $\alpha$. It does not influence the representation capability of the VAE, and can be absorbed into the encoder/decoder model. The motivation that "generalization performances of hyperbolic VAEs can be improved with varying curvatures" is weak. ---- by introducing hyperparameters, one always have room for improving the generalization performance. 

(2) Although well-motivated from manifold theory, the distribution over the manifold $\mathbb{R} \times \mathbb{R}_+$  reduces to a Normal distribution and a Gamma distribution. Instead, the properties of the manifolds are only used in the exponential map of the Lorentz model and the isometry between Lorentz space and the Gaussian manifold, the former being a known result in the previous paper.

(3)  It is not well-motivated to model the latent space as a Gaussian distribution. The connection of the proposed VAE to hierarchical VAE is weak.

(4) The experiments. From Table 2 I cannot see any benefit of the proposed method. It only works on Breakout dataset. Besides, why do you need to do binarization instead of modelling the original data?
clarity,_quality,_novelty_and_reproducibility: See above.
summary_of_the_review: The theoretical part is largely known. The motivation of the paper is weak. The experiments only show the proposed method works.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces and studies a new type of hyperbolic latent space called Gaussian manifold in the context of variational auto-encoder in order to tackle numerical and sampling issues with existing hyperbolic latent spaces. To make this latent space works in practice, the authors designed a geometric transformation at the last encoder step and the first decoder step; as well as a novel distribution over Gaussian manifold called a pseudo Gaussian manifold normal distribution that makes use of information geometry, easy to sample from, and easy to compute KL divergence. Finally, comprehensive testing on empirical datasets are carried out to highlight the effectiveness of the scheme.

The proposed method is novel, interesting and makes use of clever engineering details that are rather intuitive. However, the motivation of the methods in the manuscript is not clear and most hypotheses lack explanation. Empirical results are rather limited in quantity and not convincing. It is not well-motivated to model the latent space as a Gaussian distribution. The connection of the proposed VAE to hierarchical VAE is weak.



justification_for_why_not_higher_score: The motivation of the methods in the manuscript is not clear and most hypotheses lack explanation. Empirical results are rather limited in quantity and not convincing. It is not well-motivated to model the latent space as a Gaussian distribution. The connection of the proposed VAE to hierarchical VAE is weak.
Decision: Reject