Venue: NeurIPS 2022 Workshop NeurReps

Reviews:

final_score: rating: 7: Technically solid paper, with moderate-to-high impact, and good-to-excellent evaluation and reproducibility.
review_content: summary: In order to extract representation from hierarchical or tree-like data, hyperbolic neural networks are studied. Poincar'e or Lorentz coordinate systems suffer from numerical instability, which makes it challenging to construct hyperbolic neural networks with deep hyperbolic layers. Authors examine the critical process of concatenating hyperbolic representations. As an alternative to concatenating in the tangent space, they suggest the Lorentz direct concatenation and show that it is significantly more stable. they offer some explanations and demonstrate the superiority of carrying out direct concatenation in actual tasks.
strengths_and_weaknesses: Strengths:
Validation on existing data sets:
- the MOSES dataset

Weaknesses:
Comparison should be develeloped with other approaches as 
- Equivariant SU(1,1) neural networks:
https://hal.archives-ouvertes.fr/hal-03553274/
https://hal.archives-ouvertes.fr/hal-03309817/
https://link.springer.com/chapter/10.1007/978-3-030-80209-7_62
- Statistics defined in Poincaré Unit Disk:
https://www.sciencedirect.com/science/article/abs/pii/S0169716122000062
https://link.springer.com/chapter/10.1007/978-3-030-80209-7_28

questions: Extend state of the art on Hyperbolic Neural Network by adding references to:
- Equivariant SU(1,1) Neural Network
- Statistics and Machine Learning on SU(1,1) Lie group of Poincaré unit disk
limitations: Compare with SU(1,1) Equivariant Convolution on Fock-Bargmann Spaces
https://hal.archives-ouvertes.fr/hal-03309817/

recommended_decision: 3: Accept
review_title: Lorentz Direct Concatenation for Stable Training in Hyperbolic Neural Networks

final_score: rating: 5: Technically solid paper with some limitations e.g., limited analysis or evaluation.
review_content: summary: This work mainly studies the instability of concatenation in hyperbolic neural networks. With the identified numerical issue in the previously proposed tangent concatenation in the Poincare model and its direct translation to the Lorenz model, the authors propose direct concatenation with the Lorentz model projection onto subspace $\{0\}\times \mathbb{R}^n$, i.e., the Lorentz direct concatenation. The proposal is intuitive, and the authors have verified its numerical stationarity with an experiment using the MOSES dataset, whereas $\beta$-concatenation in the Poincare model and Lorentz tangent concatenation fail to generate meaningful results.

I think the overall proposal and results seem useful.  If the authors can address the weakness, it shall make an interesting workshop discussion.
strengths_and_weaknesses: The strengths are already mentioned in the summary section.

While hyperbolic space can intuitively provide an embedding space for a tree-like hierarchical structure, using hyperbolic operations in earlier layers does not seem to have an equally intuitive motivation. Hyperbolic concatenation is even less motivated. Maybe I have missed some critical discussion in the previous literature. I would highly appreciate that if the authors could provide better guidance to the readers and show why hyperbolic operations and concatenation are crucial in building the hyperbolic representation. Further, I highly suggest the authors provide a solid effort to seek a way to falsify the proposal, e.g., to try hard to show why the proposed method is not useful, e.g., with a strong baseline maybe only using hyperbolic representation at the last layer without concatenation. These efforts may further the understanding.
questions: I have discussed in the previous section.
limitations: Already mentioned in the earlier sections.
recommended_decision: 2: Borderline
review_title: Evaluation

final_score: rating: 7: Technically solid paper, with moderate-to-high impact, and good-to-excellent evaluation and reproducibility.
review_content: summary: A new operation is proposed to construct hyperbolic neural networks that improves numerical stability and hence enables the construction of numerically robust deep and complex neural network architectures.  
strengths_and_weaknesses: Disclaimer: I completely misinterpreted the content of the paper when reading the abstract. I am not an expert in this field at all and my report must be taken at best as an educated guess. I apologize for this.

I am quite partial to the questions being discussed here since they are fundamentally about the stability of numerical computations. The single key idea in this abstract appears to be the "Lorentz direct concatenation" in Eq (1). It is a rather obvious operation, but maybe only in hindsight? If this is indeed a new operation then it should of course be studied and this is done in this paper, some general intuition is given why this new ideas leads to improved stability, and a preliminary application. These experiments appear to be carried out carefully and are presented well but I admit that I did not understand in what sense they are related to numerical stability.

E.g., Re Figure 1 - I did not understand how gradient magnitude is related to numerical stability. Variations in the gradient (i.e. the hessian) is related to conditioning. but this is still not the same as numerical stability. Is it possible that the concepts of conditioning and numerical stability are being confused here? 

questions: see above
limitations: no concern
recommended_decision: 3: Accept
review_title: Making hyperbolic deep networks stable

Decision: Accept (Poster)