Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies the geometry of two sub-classes of the Hyperbolic Wrapped Normal distribution (HWN) with the goal deepen the understanding of which family is more appropriate for probabilistic machine learning when the data are in (or can be embedded into) hyperbolic spaces. The first subclass is the diagonal HWN that has been applied lately to different problems in hyperbolic probabilistic machine learning and the second the rotated HWN, that is proposed by the authors as a more appropriate alternative. 

The diagonal HWN draws a sample in two steps. First samples from a diagonal Gaussian in the tangent space at the origin and secondly parallel transports the sample to the new mean and projects it to the manifold via the exponential map. The proposed rotated HWN interpolates an extra step. After sampling at the origin, it rotates the sample by the rotation matrix that brings the origin to the direction of the new mean and then performs last step of the diagonal HWN. The authors provide a proof (and an additional visual explanation) that for both the diagonal HWN and the rotated HWN the principal axes become geodesics that intersect orthogonally on the new mean. From that the authors infer that the rotated HWN is more appropriate (especially for the Poincare model) since the principal axes correspond to the radial and angular variations accordingly. 

They perform 3 experiments on data that have hierarchical structure (and thus better embedded in hyperbolic space) to support their claims. The first is a toy dataset of a noisy binary synthetic tree. The goal is to show that the rotated HWN generrates posteriors that capture better the distances between the nodes as well as their level in the hierarchy (via their norm). In the second task a real language dataset is used and the goal is to embed the words and count their rank. From that task, the authors provide experimental proof that the full HWN is not appropriate for some hierarchical tasks and conjecture that this is due to the difficulty of the optimization problem. The last task is to embed the trees that can describe the evolution of an Atari game. The goal is to show that the angular direction in the rotated HWN is more informative than the one of the diagonal HWN, by describing "sibling states" of equivalent scores. 
strengths_and_weaknesses: Originality: 

(+) To my knowledge the observations and proofs in this paper are original. 

(-) The figures describing the spaces and the experimental evaluation are not novel but they seem standard.

Quality:

(+) High quality in writing and presentation.

Clarity:

(+) Paper is clearly presented.

(-) (Minor) Figure 2 (a): "red" and "blue" are interchanged, since red is the major principal axis. 

(-) In the preliminaries one more sentence about what is a geodesic would be appropriate. 

Significance:

(-) the paper proposes a simple modification to constraint the family of HWN. 

(+) However, this observation aligns well with the intuition gained from the conformality of the map from Euclidean to Poincare disk. It also disentangles better the angular and radial dimensions and the authors provide proofs, figures to build intuition and standard experimental evaluation. 
questions: 1) In Algorithm 1 why do we need to resort to the inner product of the ambient space i.e. a)the normalization of $\mu$ is w.r.t. the Euclidean norm and not the norm induced by the Lorentz inner product, b) the rotation matrix. 

2) The inner product $\langle \cdot,\cdot \rangle_x$ is defined but not used. 
 
4) How are the correlation metrics in experiment 1 computed exactly and why are they more appropriate to present than the ELBO?
limitations: The authors discuss some limitations of the proposed distribution, i.e. that it only works for hyperbolic spaces (although it seems relevant to a lot of conformal maps) and that the full HWN might be better optimized if there were better optimization algorithms (although more samples might still be needed to distinguish the best distribution in the family)

-I would advise the authors to extend their idea to more general families of distributions in hyperbolic space as well as more general conformal maps.

-Moreover, to strengthen the intuition about how the angular and radial dependency decompose more appropriately the authors should provide the marginal distributions and an extensive discussion.  

-(Minor) The authors should mention what exactly is the "hyperbolic VAE" and how it is modified by them in the main text.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this submission the authors introduce a parametric family of probability distribution on the hyperbolic space. They build on the hyperbolic wrapped normal distribution which rely on pushing Gaussian samples with parallel transport to the mean and then along the exponential map onto the manifold. Their main insight is based on the fact that when using a diagonal covariance matrix, the arbitrary choice of basis to sample this Gaussian (in the tangent space at the origin) is carried to the tangent space at the mean. Thus the principles axes of the distribution would not (apart if the mean lies on the x or y axes) be aligned with the "hierarchical depth direction" (which is given by the mean) and the "hierarchical same-level direction) which is the hyperplane orthogonal to the "depth" direction. They highlight that these directions are the "natural" ones over which the uncertainty/variation should be expressed, and therefore proposed to rotate the diagonal covariance matrix so that these are aligned.
They empirically demonstrate that this parametric distribution is better able than the diagonal and full covariance variants in learning latent representation that are better correlated with variables of interest, hence leading to "better" learnt hierarchical representations in that sense.

strengths_and_weaknesses: 
The proposed idea is simple, pretty well presented and well executed, with nice figures to illustrate the model and pretty convincing empirical results.
The paper was mostly pleasant to read and easy to follow.

I don't see any major weakness.
In the following section I make suggestions to improve the organisation of the paper, along with some questions which I hope will lead to some clarification.

questions: 
Global comments:
- I would strongly suggest moving the figures/tables so that they are closer (in the same page) as the related paragraph. At the moment it's not the case, and I believe it would greatly improve the reader's experience.
- As the proposed method is de facto a specific setting of the full covariance matrix, I believe that the investigation of the failure of full covariance model would be worthy of more space. I believe that Figure 3 is useful in that regard but I have a few remarks/questions on it later. It is claimed that this parametrisation lead to unstable optimisation, but it would be nice to dive a bit more, e.g. computing the condition number. Additionally, the isotropic variant where the covariance matrix is given by $\Sigma = \sigma I_d$ is not explored, whilst it would be easy to do so and interesting to see whether it is indeed not flexible enough.


Specific comments:
- typo "Hyperboilc" in openreview title
- Abstract: "RoWN, the newly proposed distribution" -> "newly" perhaps misleading as it's in some ways a specific setting of the full covariance matrix (which now depends on the mean).
- Section 2
    - Sec 2.3: I feel that the sampling process for the HWN would really benefit from having an illustrative figure on the side. This could also be used to illustrate Algorithm 1, which is the core contribution of the paper.
- Section 3
    - Sec 3.1: Proposition 1 is somewhat repeated before *and* after which seems unnecessary.
    - Sec 3.2:
        - Worth noting in Algorithm 1, that to sample the multivariate normal $v \sim \mathcal{N}(0, \hat{\Sigma})$, one can rely on computing $\hat{\Sigma}^{1/2} = R \text{diag}(\sqrt{\sigma})$ (which exist and is unique since $\Sigma$ is positive definite) which can be done easily.
        Figure 3: "different prior choices" -> meant "posterior"? What are the dashed blue lines? Although I do find the upper row visually pretty, I don't understand from which posterior distributions the samples come from (from the ones whose means are represented in red?) nor the signification of the colour.
- Section 4
    - Sec 4.1:
        - I would suggest giving a bit more of background on the model for the noisy synthetic tree, mainly on the loss and likelihood (based on the Hamming distance) and how this plays with the spherical noise since the nodes aren't binary anymore.
        - Also I would be curious to see Table 1 for <4 and >7 depth, are the results similar?
        - Additionally I was wondering whether "better" latent representations (measured via the reported correlations) lead to better likelihood (as estimated via IWAE [1])?
    - Sec 4.2:
        - The HWN (full covariance) dramatically fails, according to the author this due to "unstable optimization". What is the reason why this phenomenon did not occur in the VAE setting from Section 4.1?
        - Table 2: Would suggest adding "metric" and "model" in the table's header
        - Worth reminding what are the MR and mAP metrics for the readers unfamiliar with these? either via their mathematical formula or in an english sentence.
    - Sec 4.3:
        - What is the motivation for the Poincaré norm of the latent representation to be correlated with the score? I'm only partially familiar with breakout, is the score correlated with the number of broken blocks? Isn't the Poincaré norm simply correlated with time then?
- Sec 5:
    - The reader may benefit from starting the related work section with a paragraph on "hierarchical representation learning". These methods are most of the time relying on discrete hierarchical structures, there are quite a few in the Bayesian non-parametrics literature (e.g. [2, 3]).
    - In the "Distributions in the hyperbolic space" paragraph, one could also mention normalizing flow related methods (e.g. [4, 5]).

limitations: I believe so
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors build upon the Hyperbolic Wrapped Normal (HWN) distribution, which puts a distribution on hyperbolic space by transporting a distributed tangent vector from an origin to a mean point mu and taking the exponential map (a diffeomorphism in this case). The authors observe that a standard basis vector in the tangent plane at the origin, transported to point mu and then exp-mapped and projected to a Poincaré disk, is aligned with a standard basis vector in the disk. They posit that the resulting distribution is less desirable than a distribution that is aligned such that one of the tangent vectors points in the disk from mu to the origin. To fix this, they propose the Rotated HWN distribution that is constructed so that the first dimension of the tangent plane points from mu to the origin.

Empirically, the authors show that the RHWN outperforms HWN with diagonal and full covariance matrix on a synthetic task, a word embedding task and a task of embedding Atari rollouts. In the synthetic task, the goal is to have the hamming distance on a binary tree correlate with the (I presume geodesic) distance in embedding space or to correlate tree depth with the "Poincaré norm" (geodesic distance to origin, I presume). In the word embedding task, the goal is to have hypernymy correlate with distance. In the Atari task, the goal is to have distance to the origin correspond with progress in the game.
strengths_and_weaknesses: ## Strengths:
- The paper is very clearly and pleasantly written with many insightful illustrations - critical when dealing with geometry.
- I like the construction of the paper: diagnosing an issue with a prior method, then proposing a simple fix
- The proposed method is simple to describe and implement.
- The proposed method outperforms prior work.

## Weaknesses
I have one major concern with the geometric analysis and some other points.
### Symmetry of the geometry
I am not sure that I agree with the diagnosis. As a geometric method, it shouldn't matter which model of the hyperbolic space we pick. Up to isometry there is only one space of n dimensions with constant -1 curvature. This space, $H_n$, is a homogenous (even symmetric) space meaning that there is an isometry mapping any tangent vector $v$ at point $p$ to any other tangent vector $v'$ at any point $q$ assuming that they have the same norm: $g_p(v, v)=g_q(v', v')$. Thus, I don't really see why in Fig 2 the unrotated image on the top and the rotated image on the bottom are different. I suspect there is an isometry that maps between them. The only reason the top image looks "odd" is the particular choice of coordinates. Similarly, the paper makes repeated reference to the "origin" and "standard basis" of the Poincaré disk, which are also contingent on a particular choice of coordinates. A geometric method shouldn't care about those. Hence, when constructing a single distribution in hyperbolic space, up to isometry, the added rotation in the RHWN effectively does nothing.

The main reason the isometry-symmetry is broken is the particular choice of prior $p(z)$ in the variational inference, taken to be centered around a particular point $\mu_0$, which can then be called the origin. I suppose that's what the authors do. This would be the only reason why the root of a tree should be embedded close to the origin - geometry itself doesn't promote that. Yet the authors make no mention of the prior $p(z)$ in their analysis. I suppose the RHWN works well because first the prior encourages the root to be close to the origin and then it being preferable to have the basis such that one dimension points towards the root.

I feel the analysis can be much improved if these considerations were taken into account. Concretely:
- I'd like to see an argument why the root node should be place at the mean of the prior $p(z)$. For that, I'd also like to know what's special about the root node. If one interprets a (e.g. binary) tree as a graph, then it can be seen as a binary tree with any node as root node. I'd guess that this has to do with the balancing of the resulting tree - or with some notion of distance between that the nodes that's not encoded in an (unweighted) tree.
- Then in the subsequent discussion, it'd be better if references to "standard basis" and "origin" (arbitrary coordinatization) would be made geometrically meaningful relative to the prior origin / placement of the root node.

### Other points
1. Due to my above points, it's not quite clear to me why the $x$ vector in the algorithm should be fixed, so that the first dimension of $v$ is rotated to point to the arbitrary origin. It would be interesting to have it be learned and given by the encoder network. This is somewhat similar to e.g. Householder flows [1]. I'd be great if the authors could compare to such a method. One motivation could be that one wants to align not to the root node, but to the parent node, which may not lie in the direction of the root node. It'd be very interesting if the learned point to rotate to would indeed be the parent.
2. Why would the rotation be particular to a hyperbolic embedding of trees? In other manifolds with a prior with a well-defined mean and exp-mapped normal distributions, one could rotate one tangent vector towards the prior mean. For Euclidean spaces, this would be straightforward, but I could also imagine this working on Lie groups [2]. I'd be very interested to see if that improves the performance. The paper could make a much more general claim if they show improvement across different kinds of manifolds.
3. The rotation matrix (4) is not the only rotation that maps $x$ to $y$. It'd be good if the authors discuss this and say why this particular one (which I believe is the one with the smallest SO(n) distance to the unit) is desirable.
4. The authors use the word "prior" confusingly to not as the distribution $p(z)$ in variational inference, but as the variational family the posterior $q(z|x)$ belongs to. I see how this originates from the idea of "prior" as domain knowledge, but I'd prefer a different wording for that.
5. Due to its simplicity, the proposed method is a quite incremental change over prior work, hence somewhat limited in originality.


# Conclusion
Because I don't think the current analysis is correct without explicit mention of the role of the prior, I propose rejection of the paper. If the authors address those issues satisfactorily, I would recommend acceptance. If the authors could expand the paper by addressing my points (1) and (2), this could become a strong paper.

-- Score updated
-- Score updated again

[1] Tomczak, Jakub M., and Max Welling. "Improving variational auto-encoders using householder flow." (2016).

[2] Falorsi, Luca, et al. "Reparameterizing distributions on lie groups." (2019)
questions: Please see my point above on the isometry symmetries.
limitations: The authors fairly address the limitations of their work.
I disagree though that rotating the tangent space to the root node is applicable only on hyperbolic space, see my comment above.
review_title: null

Meta-review: metareview: The paper proposes the rotated hyperbolic wrapped normal (RoWN) distribution which improves hyperbolic wrapped normal (HWN) distribution in representing hierarchical (implicitly tree structured) datasets e.g., WordNet. The idea is to include a learnable rotation matrix in the generative process. The initial soundness concern raised by one of the reviewers was resolved in the discussion phase. This looks like a relevant contribution to the geometric deep learning community and I would like to recommend acceptance for this paper.
confidence: Certain
Decision: Accept