Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Paper summary:

The paper provides a reformulation of the fundamental operations in Euclidean space that are used in neural networks for the Poincaré ball model of hyperbolic space (and thus hyperbolic space generally). The paper’s reformulation differs from previous reformulations (Ganea et al. 2018) in several ways. For multinomial logistic regression, the excess n-1 degrees of freedom available in previous formulations when defining a Poincaré hyperplane via a point on the hyperplane and a normal vector are eliminated by using a canonical choice of normal vector along with a scalar quantity corresponding to the distance to the hyperplane from the origin. Fully connected (FC) neural network layers are also reformulated in a way that keeps with the interpretation of an affine transformation as returning a point whose individual coordinates are distances to a set of different hyperplanes. On the other hand, the previous reformulation directly used Möbius matrix-vector multiplication, which does not this same interpretation. The paper then gives hyperbolic reformulations of further types of neural network operations in Euclidean space, namely split/concatenation (less computationally intensive than that in previous work), convolution (not present in previous work). Turning its focus to attention models, the paper proves a theorem regarding the equivalence of various hyperbolic midpoints proposed in previous work. Finally, the paper carries out experiments testing each part of its reformulation on appropriate datasets.

------------------------------------------
Strengths and weaknesses:

The paper gave clear motivations for each of part of its hyperbolic reformulation and explained how it differed from previous reformulations. The paper was well-written, and the authors performed regular sanity checks, such as re-deriving the Euclidean case for c -> 0. The experiments were appropriate and demonstrated that the reformulation works as well as previous reformulations (possibly better, with regard to stability). I liked the paper a lot, and I think it will definitely be of interest to people working on non-Euclidean deep learning. I’m assigning a score of 8 (a very good conference paper), and I think that the paper is more or less ready for publication as is. I’ve included a few questions below (to help my own understanding), as well as some typos I spotted whilst reading the paper.

------------------------------------------
Questions and clarification requests:

1) Which of Spivak (1979), Petersen et al. (2006), and Andrews & Hopper (2010) were you working from when writing the brief summary of Riemannian Geometry on page 2? Some of the definitions given like “An n-dimensional manifold M is an n-dimensional topological space that can be linearly approximated to an n-dimensional real space at any point x ∈ M” and “g_{x} is a positive definite symmetric matrix defined on T_{x}M” struck me as unusual (and possibly imprecise).
2) In equation 4, you write q as the exponential of r_{k}[a_{k}], even though a_{k} ∈ T_{q}M, not T_{0}M. Is this correct?
3) At the beginning of section 3, you write “The core concept is re-generalization of ⟨a,x⟩−b type equations with no increase in the number of parameters, which has the potential to replace any affine transformation in a shared manner.” What did you mean by “in a shared manner”?
4) At the end of section 4.1, you write “In particular, our parameter-reduced approach obtains the same level of performance as a conventional hyperbolic MLR in a more stable training, as can be seen from the relatively narrower confidence intervals.” Do you think your approach is genuinely more stable? If so, do you have any intuition as to why performance was more stable?

------------------------------------------
Typos and minor edits:
- Section 1, paragraph 4, sentence 1 – “Despite such progresses” -> “Despite such progress”
- Section 2, Poincaré hyperplane – Ganea et al. 2018 certainly discusses Poincaré hyperplanes, but hyperplanes in Riemannian geometry have been studied a very long time before this.
- Section 3.3, paragraph 1, sentence 5 – “approaches to the” -> “approaches the”
- Section 3.3, last paragraph, last sentence – “treats every inputs fairly” -> “treats every input fairly”
- Section 4.2, paragraph 1, sentence 2 – better to rephrase this
- Section 4.2, paragraph 2, sentence 2 – “our models achieved the almost the same performance as Set Transformers” -> “our models achieved almost the same performance as Set Transformers”
- Section 4.3, paragraph 1, sentence 1 – “we experimented the convolutional sequence to sequence modelling task” -> “we experimented with the convolutional sequence-to-sequence modelling task”
- Table 2 – “on the Euclidean space” -> “on Euclidean space”
- Section 4.3, paragraph 1, sentence 2 – “the hyperbolic version of which have already been verified” -> “the hyperbolic version of which has already been verified”
- Section 4.3, paragraph 1, sentence 6 – “Note that the inputs for the sigmoid functions in Gated Linear Units are logarithmic mapped just as hyperbolic Gated Recurrent Units proposed by” -> “Note that the inputs for the sigmoid functions in Gated Linear Units are logarithmically mapped just like hyperbolic Gated Recurrent Units proposed by”
- Conclusion, final sentence – “for the future researches” -> “for future research” / “for future researchers”
review_title: Review of "Hyperbolic Neural Networks++"

final_score: rating: 7: Good paper, accept
review_content: _Summary_:
Hyperbolic Neural Networks++ extends the existing work of applying hyperbolic manifolds to neural networks. It proposes new ways to reparametrize hyperbolic multinomial logistic regression (MLR) layers to reduce the number of parameters, to generalise fully connected layers  as well as split and concat operations to be more flexible and less computationally expensive. Further they expand hyperbolic neural networks to convolutional layer and attention mechanisms. The evaluation include direct comparison to HNN, and clustering with Transformers and seq-2-seq modeling.

_Strengths_: 
- Paper is well-written and easy to understand. I also appreciate a good overview over hyperbolic geometry and consistent notation.
- The work proposed extends existing architectures with convolutional layers and attention mechanism which is prevalent in many deep architectures nowadays and thus an important contribution.
- The evaluation included a range of different tasks and architectures.

_Weaknesses_:
- The evaluation did not convince me completely of its superiority to HNNs. This is also not really discussed.

_Overall assessment_: I do believe that the extensions and generalization to Hyperbolic Neural Networks are important contributions. Therefore, I am recommending an accept.

_Detailed comments and questions_:
- Reduction of number of parameters needed for hyperbolic MLR: You correctly point out that the current formulation of hyperbolic MLR requires double the number of parameters in comparison to the Euclidean counterpart. This is reduced in this work. However, w.r.t. space complexity both are linear (=O(N)). Has the reduction in number of parameters other implications?
- Concatenation ("heavy computational cost"): Can you quantify that?
- Comparisons to HNN (Table 1): In 1/3 of the results HNN is superior to your approach? Do you have an explanation for that?

_Post-rebuttal_: I do appreciate the authors addressing my comments and updating their paper. Furthermore, the authors also addresses a lot of concerns of my fellow reviewers. As this was a good paper to begin with, I am keeping my score of recommending an accept.
review_title: Official Blind Review #3

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper bolsters the library of hyperbolic neural network building blocks by defining a variety of layers in a natural way. These building blocks include a tighter parameterization of the basic fully connected layer, and the concatenate/split operations which are then used to define a version of hyperbolic multi-head attention.

The experimental section is adequate, covering standard hyperbolic applications such as embedding word hierarchies, as well as interesting tasks to validate the attention and convolutional capabilities of hyperbolic neural networks. Some details could be improved; for example, the BLEU score is difficult to interpret, the architecture and task details in Section 4.2 are sparse, and the first two experiments report only in very low dimensions which has been a toy testbed for hyperbolic models but can be unrealistic. Overall, the model shows general improvement over the baseline hyperbolic neural network models.

Overall, this paper does not present fundamentally new ideas in the development of non-Euclidean geometry for deep learning, but offers solid improvements in various aspects of constructing architecture components for hyperbolic neural networks and is a useful contribution.

review_title: Hyperbolic Neural Networks++

final_score: rating: 7: Good paper, accept
review_content: edit after rebuttal: The idea of the paper was interesting to me but some motivations or choices were unclear. After reading the rebuttal and other reviews, the authors have addressed most of my concerns. Therefore, I am ready to increase my score.

======

This paper introduces an improvement of the "Hyperbolic Neural Networks" (HNN) proposed by Ganea et al. and published at Neurips 2018.

The authors consider in particular the multinomial logistic regression problem. The probability score of a category is determined by a linear separator and a bias term. 
In the vanilla HNN, two sets of parameters, each of dimensionality n (i.e. 2n in total), are learned to adapt the problem to hyperbolic geometry. 
In the submission, the authors propose to learn only n+1 parameters per category. Thanks to this reformulation, the authors propose a new generalization of the fully connected layers of neural networks to hyperbolic geometry. This allows them to define arbitrary dimensional convolutional layers in the Poincare ball model. In Ganea et al., the authors do not exploit fully the hyperbolic geometry when they introduce their bias term, their discriminative hyperplane is instead determined in some tangent space (which is not hyperbolic), and then projected back to the hyperbolic space via the logarithm map. HNN++ proposes instead the exploit the hyperbolic geometry of their discriminator.

Although HNN and HNN++ are both hyperbolic neural networks, I think that the proposed approach should not be called HNN++ as it seems to suggest that this is an incremental extension of HNN.
In general, I think that the paper is hard to read. This is probably due to the 8 page format that makes some parts of the paper very dense. In particular:
- The split operation is not motivated until the multi-head attention part (i.e. 2 subsections after it was defined).
- What does the notation B(n/2,1/2) mean at the end of the "Generalization criterion"?
- The paper contains a lot of equations and formulae, but it would benefit from having a general pipeline explaining how to code exactly the approach. It currently seems hard to reimplement the approach, even with Section E in the Appendix. Will the code be available?

Concerning contributions, I do not think that the reduction of numbers of parameters from 2n to n+1 is a novel contribution, I also do not think it causes problems if the training data is large enough. However, I understand how proposing an approach that better exploits hyperbolic geometry to implement fully connected layers can help. My main concern is that the experiments do not seem to show how the proposed approach improves performance compared to HNN in practice. 

- Section 4.1:
I do not understand the point of the experiments in Section 4.1. The scores obtained by HNN and HNN++ look very similar (are the scores on the right standard errors or standard deviations btw?). 
It is mentioned in the last sentence of Section 4.1 that the "parameter-reduced approach obtains the same level of performance as a conventional hyperbolicMLR in a more stable training." What do the authors mean by more stable? Can they provide more details about the stability? Do they use the same optimizer between HNNs and the proposed approach?

- Why can't HNN be used as a baseline in Sections 4.2 and 4.3? Why is ConvSeq2Seq the only baseline in Table 3?

- Can the authors provide an experimental analysis on the contribution 4 (arbitrary dimensional convolutional layers) in page 2? How can tuning the "arbitrary dimensions" of convolutional layers have impact on performance?

- I have a general question about the gyromidpoints. I understand that the centroid of the squared Lorentzian distance is a minimizer over a sum of squared Lorentzian distances, just like the Euclidean centroid (average vector) would minimize a sum over squared Euclidean distance. However, is this also the case for the other types of gyromidpoints? What is the optimization problem(s) that those gyromidpoints minimize?


In conclusion, I would like to see an application where HNN++ is much better than HNN.


Minor comment: There is a minor mistake in the "Riemannian geometry" paragraph of Section 2. It is mentioned that g_x is a symmetric positive definite matrix. 
This statement is correct for the Beltrami-Klein model because it is the sum of 2 symmetric positive definite matrices. However, this is not correct for the hyperboloid model because g_x is a diagonal matrix (see Section A of Appendix) and it would then be positive definite if and only if all its diagonal elements were positive. 
review_title: How is HNN++ better than HNN in practice?

Meta-review: The paper introduces new methods and building blocks to improve hyperbolic neural networks, including a tighter parameterization of fully connected layers, convolution, and concatenate/split operations to define a version of hyperbolic multi-head attention. The paper is well written and relevant to the ICLR community. The proposed methods offer solid improvements over previous approaches in various aspects of constructing hyperbolic neural networks and also extends their applicability. As such, the paper provides valuable contributions to advance research in learning non-Euclidean representations and HNNs. All reviewers and the AC support acceptance for the paper's contributions. Please consider revising your paper to take feedback from reviewers after reubttal into account.
Decision: Accept (Poster)