Fully Hyperbolic Convolutional Neural Networks for Computer Vision
Authors: Ahmad Bdeir
Kristian Schwethelm
Niels Landwehr
Venue: ICLR 2024 poster (2024)
Keywords: hyperbolic neural networks
hyperbolic image embedding
hyperbolic vision models. hyperboloid representation
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Real-world visual data exhibit intrinsic hierarchical structures that can be represented effectively in hyperbolic spaces. Hyperbolic neural networks (HNNs) are a promising approach for learning feature representations in such spaces. However, current HNNs in computer vision rely on Euclidean backbones and only project features to the hyperbolic space in the task heads, limiting their ability to fully leverage the benefits of hyperbolic geometry. To address this, we present HCNN, a fully hyperbolic convolutional neural network (CNN) designed for computer vision tasks. Based on the Lorentz model, we generalize fundamental components of CNNs and propose novel formulations of the convolutional layer, batch normalization, and multinomial logistic regression. Experiments on standard vision tasks demonstrate the promising performance of our HCNN framework in both hybrid and fully hyperbolic settings. Overall, we believe our contributions provide a foundation for developing more powerful HNNs that can better represent complex structures found in image data. Our code is publicly available at https://github.com/kschwethelm/HyperbolicCV.

Published as a conference paper at ICLR 2024
FULLY HYPERBOLIC CONVOLUTIONAL NEURAL NETWORKS FOR COMPUTER VISION
Ahmad B Kristian Schwethelm 2..1 & Niels Landwehr'
1 Data Science Department, University of Hildesheim
2 Chair for Artificial Intelligence in Medicine, Technical University of Munich (bdeira, schwethelm, landwehr
ABSTRACT
Real-world visual data exhibit intrinsic hierarchical structures that can be repre- sented effectively in hyperbolic spaces. Hyperbolic neural networks (HNNs) are a promising approach for learning feature representations in such spaces. However, current HNNs in computer vision rely on Euclidean backbones and only project features to the hyperbolic space in the task heads, limiting their ability to fully leverage the benefits of hyperbolic geometry. To address this, present HCNN, a fully hyperbolic convolutional neural network (CNN) designed for computer vision tasks. Based on the Lorentz model, generalize fundamental compo- nents of CNNs and propose novel formulations of the convolutional layer, batch normalization, and multinomial logistic regression. Experiments on standard vi- sion tasks demonstrate the promising performance of our HCNN framework in both hybrid and fully hyperbolic settings. Overall, we believe our contributions provide foundation for developing more powerful HNNs that can better repre- sent complex structures found in image data. Our code is publicly available at https: //gi thub. com www.whthhheeee....om..m.... 1m/
INTRODUCTION
Representation learning is fundamental aspect of deep neural networks, as obtaining an optimal representation of the input data is crucial. While Euclidean geometry has been the traditional choice for representing data due to its intuitive properties, recent research has highlighted the advantages of using hyperbolic geometry as geometric prior for the feature space of neural networks. Given the exponentially increasing distance to the origin, hyperbolic spaces can be thought of as continuous versions of trees that naturally model tree-like structures, like hierarchies taxonomies, without spatial distortion and information loss (Nickel & Kiela, 2018; Sarkar, 2012). This is compelling since hierarchies ubiquitous in knowledge representation (Noy & Hafner, 1997), and even the natural spatial representations in the human brain exhibit a hyperbolic geometry (Zhang et al., 2023).
Leveraging this better representative capacity. hyperbolic neural networks (HNNs) have demonstrated increased performance over Euclidean models in many natural language processing (NLP) and graph embedding tasks (Peng et al., 2022). However, hierarchical structures have also been shown to exist in images. Mathematically. Khrulkov et al. (2020) have found high 5-hyperbolicity in the final embeddings of image datasets, where the hyperbolicity quantifies the degree of inherent tree- structure. Extending their measurement to the whole model reveals high hyperbolicity in intermediate embeddings as well (see Appendix D.1). Intuitively, hierarchies that emerge within and across images can be demonstrated on the level of object localization and object class relationships. A straightforward example of the latter is animal classification hierarchy, where species is the lowest tier, preceded by genus, family, order, etc. Similarly, on a localization level, humans are one example: the nose, eyes, and mouth are positioned on the face, which is part of the head, and, ultimately, a part of the body. This tree-like localization forms the basis of part-whole relationships and is strongly believed to be how we parse visual scenes (Biederman, 1987; Hinton, 1979; Kahneman et al., 1992).
In light of these findings, recent works have begun integrating hyperbolic geometry into vision architectures (Mettes et al., 2023; Fang et al., 2023). Specifically, they rely on the Poincaré ball
Equal contribution. Work done while University of Hildesheim.

---

Published as a conference paper at ICLR 2024
Hyperbolic Classification Head
Hyperbolic CNN Encoder
Input x
Class: Elephant
LK
MIR Hyperplane
Figure 1: In contrast to hybrid HNNs that use Euclidean CNN for feature extraction, our HCNN learns features in hyperbolic spaces in every layer, fully leveraging the benefits of hyperbolic geometry. This leads better image representations and performance.
and the Lorentz model as descriptors of hyperbolic space and formalize hyperbolic translations of neural network layers. This is challenging due to ill-defined hyperbolic analogs of, e.g.. addition. multiplication, and statistical measures, Currently, most HNN components are only available in the Poincaré ball as it supports the gyrovector space with basic vector operations. However, due to its hard numerical constraint, the Poincaré ball is more susceptible to numerical instability than the Lorentz model (Mishne al., 2022), which motivates introducing the missing layers for the Lorentz model. Moreover, HNNs in computer vision have been limited to hybrid architectures that might not fully leverage the advantages of hyperbolic geometry as they rely on Euclidean encoders to learn hyperbolic representations. Until now, hyperbolic encoder architectures vision, although prevalent in NLP and graph applications (Peng et al., 2022).
missing in computer
In this work, present HCNN, fully hyperbolic framework for vision tasks that can be used to design hyperbolic encoder models. We generalize the ubiquitous convolutional neural network (CNN) architecture the Lorentz model, extend hyperbolic convolutional layers to 2D, and present novel hyperbolic formulations of batch normalization and multinomial logistic regression. Our methodology is general, and we show that our components can be easily integrated into existing architectures. Our contributions then become three-fold:
1. We propose hybrid (HECNN) and fully hyperbolic (HCNN) convolutional neural network encoders for image data, introducing the fully hyperbolic setting in computer vision.
2. We provide missing Lorentzian formulations of the 2D convolutional layer, batch normaliza- tion, and multinomial logistic regression.
3. We empirically demonstrate the performance potential of deeper hyperbolic integrations in experiments on standard vision tasks, including image classification and generation.
2 RELATED WORK
Hyperbolic image embeddings Previous research on HNNs in computer vision has mainly focused on combining Euclidean encoders and hyperbolic embeddings. This approach involves projecting Euclidean embeddings onto the hyperbolic space in the task heads and designing task-related objective functions based hyperbolic geometry. Such simple hybrid architectures have been proven effective in various vision tasks like recognition (Yu et al., 2022; Khrulkov et al., 2020; Liu et al., 2020; Guo al., 2022), segmentation (Hsu al., 2020; Atigh et al., 2022), ieconstructionnggeerrii (Mathieu et al., 2019; Nagano et al., 2019; Ovinnikov, 2019; Qu & Zou, 2022), and metric learning (Ermolov et al., 2022; Yan et al., 2021; Yue et al., 2023). However, there remains the discussion of whether the single application of hyperbolic geometry in the decoder can fully leverage the present hierarchical information. In contrast, HE/HCNN also learns latent hyperbolic feature representations in the encoder, potentially magnifying these benefits. We also forgo the typically used Poincaré ball in favor of the Lorentz model, as offers better stability and optimization (Mishne et al., 2022). For a complete overview of vision HNNs and motivations, refer to (Mettes et al., 2023; Fang et al., 2023).
Fully hyperbolic neural networks Designing fully hyperbolic neural networks requires generaliz- ing Euclidean network components to hyperbolic geometry. Notably, Ganea et al. (2018) and Shimizu et al. (2020) utilized the Poincaré ball and the gyrovector space to generalize various layers, including fully-connected, convolutional, and attention layers, well operations like split, concatenation,
2

---

Published as a conference paper at ICLR 2024
and multinomial logistic regression (MLR). Researchers have also designed components in the Lorentz model (Nickel & Kiela, 2018; Fan et al., 2022; Chen et al., 2021; Qu Zou, 2022). but erucial components for vision, like the standard convolutional layer and the MLR classifier, are still missing. Among the hyperbolic layer definitions, fully hyperbolic neural networks have been built for various tasks in NLP and graph applications (Peng al., 2022). However, hyperbolic encoder architecture has yet been utilized in computer vision. Our work provides formulations for missing components in the Lorentz model, allowing for hyperbolic CNN vision encoders. Concurrently, van Spengler et al. (2023) proposed fully hyperbolic Poincaré CNN.
Normalization in HNNS There are few attempts at translating standard normalization layers the hyperbolic setting. To the best of our knowledge, there is only a single viable normalization layer for HNNs, ie., the general Riemannian batch normalization (Lou et al., 2020). However, this method is not ideal due to the slow iterative computation of the Fréchet mean and the arbitrary re-scaling operation that is not based on hyperbolic geometry. The concurrent work on Poincaré CNN (van Spengler et al., 2023) only solved the first issue by using the Poincaré midpoint. In contrast, propose an efficient batch normalization algorithm founded in the Lorentz model, which utilizes the Lorentzian centroid (Law et al., 2019) and mathematically motivated re-scaling operation.
The exponential growth of the Lorentz model's volume with respect the radius can introduce numerical instability and rounding errors in floating-pon point arithmetic. This requires many works to rely on 64-bit precision at the cost of higher memory and runtime requirements. To mitigate this, researchers have introduced feature clipping and Euclidean reparameterizations (Mishne et al., 2022; Guo et al., 2022; Mathieu et al., 2019). We adopt these approaches to run under 32-bit floating point arithmetic and reduce computational cost.
Numerical stability of HNNS
BACKGROUND
This section summarizes the mathematical background of hyperbolic geometry (Cannon et al., 2006; Ratcliffe, 2006). The n-dimennsooal hyperbolic space H is Riemannian manifold (M". g with constant negative curvature K < 0, where MM and K represent the manifold and the Riemannian metric, respectively. There are isometrically equivalent models of hyperbolic geometry. We employ the Lorentz model because of its numerical stability and its simple exponential/logarihmie maps and distance functions. Additionally, we use the Poincaré ball for baseline implementations. Both hyper- bolic models provide closed-form formulae for manifold operations, including distance measures, exponential/lggrrimmic maps, and parallel transportation. They are detailed in Appendix A.
Lorentz model The n-dimensional Lorentz model Lj (L", (C*,,0") models hyperbolic geometry on the upper sheet of a two-sheeted hyperboloid L", with origin 0 [V-1/K,0, 0, ... ,0 and embedded in (n + 1)-dimensional Minkowski space (see Figure 2). Based on the Riemannian metric _ = diag( - 1, 1,...,,11
L2
.....,,).. 1), the manifold is defined as
Geodesic
L = (ar m+ (æx
K³ xt 0},
(1)
B2
with the Lorentzian inner product
Figure Comparison of Lorentz and Poincaré model.
(x, y) == - -ey + x xx" diag( -1, 1,1, ... 1)y- (2)
When describing points in the Lorentz model, we inherit the terminology of special relativity and call the first dimension the time component and the remaining dimensions the space component æ.. such that x E L = [ )z and F4 V 1/K.
4 FULLY HYPERBOLIC CNN (HCNN)
We aim to give way to building vision models that can fully leverage the advantages hyperbolic ge- ometry by learning features in hyperbolic spaces. For this, we generalize Fuclidean CNN components
3

---

Published as a conference paper at ICLR 2024
to the Lorentz model, yielding one-to-one replacements that can be integrated into existing architec- tures. In the following, we first define the cornerstone of HCNNs, i.e., the Lorentz convolutional layer, including its transposed variant. Then, introduce the Lorentz batch normalization algorithm and the MLR classifier. Finally, we generalize the residual connection and non-linear activation.
4.1 LORENTZ CONVOLUTIONAL LAYER
Hyperbolic feature maps The convolutional layer applies vector operations an input feature map containing the activations of the previous layer. In Euclidean space, arbitrary numerical values can be combined form vector. However, in the Lorentz model, not all possible value combinations represent a point that can be processed with hyperbolic operations (LL C. 1
We propose using channel-last feature map representations throughout HCNNs and adding the Lorentz model's 's time component as an additional channel dimension. This defines a hyperbolic feature map as an ordered set of n-dimensional hyperbolic vectors, where every spatial position contains a vector that can be combined with its neighbors. Additionally, it offers nice interpretation where an image an ordered set of color vectors, each describing a pixel.
Formalization of the convolutional layer We define the convolutional layer as matrix multiplica- tion between a linearized kernel and concatenation of the values in its receptive field, following Shimizu et al. (2020). Then, generalize this definition by replacing the Euclidean operators with their hyperbolic counterparts the Lorentz model.
Given a hyperbolic input feature map x = {0 E L h, as an ordered set of n-dimensional hyperbolic feature vectors, each describing image pixels, the features within the receptive field of H.W the kernel K mxxxx W are (#w/ LK h, where (h', w²) denotes the starting position and 6 the stride parameter. Now, we define the Lorentz convolutional layer as
H.W
Vk.w LFC(HCat( (*) (#N46) LK
H,W h
where HCat denotes the concatenation of hyperbolic vectors, and LFC denotes Lorentz fully- connected layer performing the affine transformation and parameterizing the kernel and bias, respec- tively (see Appendix A). Additionally, we implement padding using origin vectors, the analog of zero vectors hyperbolic space. The LFC layer is similar to Chen et al. (2021) but does not use normalization is done through the hyperbolic batch normalization formulated below.
Extension to the transposed setting The transposed convolutional layer is usually used in encoder- decoder architectures for up-sampling. A convolutional layer carries out transposed convolution when the correct local connectivity is established by inserting zeros at certain positions. Specifically, when stride 1, then 1 zero vectors are inserted between the features. We refer Dumoulin Visin (2016) for illustrations. Under this relationship, the Lorentz transposed convolutional layer is a Lorentz convolutional layer with changed connectivity through origin padding.
4.2 LORENTZ BATCH NORMALIZATION
Given a batch B of m features I., the traditional batch normalization algorithm (Ioffe & Szegedy, 2015) calculates the mean μ48 Σ 2; and variance o my Σm (a: 23)² across the batch dimension. Then, the features are re-scaled and re-centered using a parameterized variance and mean 3 as follows
BN(x;) = D;
PBB
+ B.
V +
At test time, running estimates approximate the batch statistics. They are calculated iteratively during training: 14 = (1 (1- 7) + Mug and of = (1-1700 no with and f denoting momentum and the current iteration, respectively. We generalize batch normalization to the Lorentz model using the Lorentzian centroid and the parallel transport operation for re-centering, and the Fréchet variance and straight geodesics at the origin's tangent space for re-scaling.
4

---

Published as a conference paper at ICLR 2024
Re-centering To re-center hyperbolic features, it necessary to compute notion of mean. Usually, the Fréchet mean is used (Lou et al., 2020), which minimizes the expected squared distance between a set of points in metric space (Pennec, 2006). Generally, the Fréchet mean must be solved iteratively, massively slowing down training. To this end, we propose to use the centroid with respect to the squared Lorentzian distance, which can be caleulated efficiently in closed form (Law et al., 2019). The weighted Lorentzian centroid, which solves mil >i p (2), ()), with æ; E LR and 2 0, -i=1 V > O, is given by
E Ua 1 V- EI uo.
(5)
In batch normalization, the mean is not weighted, which gives 1 Now, we shift the features from the batch's s mean μg to the parameterized mean /3 using the parallel transport operation PTи #g-88 (æ)). Parallel transport does not change the variance, as it is defined to preserve the distance between all points. Finally, the running estimate is updated iteratively using the weighted centroid with V1 = 7)) and 1/2
Re-scaling For re-scaling, we rely on the Fréchet variance o² RR+ defined as the expected squared Lorentzian distance between point and the mean μ., and given by x² (Kobler et al., 2022). In order to re-scale the batch, features must be moved along the geodesics connecting them their centroid, which is generally infeasible to compute. However, geodesics intersecting the origin are very simple, as they can be represented by straight lines in tangent space T This is reflected by the equality between the distance of point the origin and the length of its corresponding tangent vector (dg(x, 0) logø (x)]].. Using this property, we propose to re-scale features by first parallel transporting them towards the origin PTK ug-ō (log fe (a:) making the origin the new centroid and straightening the relevant geodesics. Then, simple multiplication re-scales the features in tangent space. Finally, parallel transporting to ,3 E L completes the algorithm and yields the normalized features. The final algorithm is formalized
-i=1 d2 (æ:,, p))
PTK Mhe
(log (az)
-0
LBN(x)= exp
PT
7
(6)
Vol
+E
+E
4.3 LORENTZ MLR CLASSIFIER
In this section, we consider the problem of classifying instances that are represented in the Lorentz model. A standard method for multi-class classification is multinomial logistic regression (MLR). Inspired by the generalization of MLR the Poincaré ball (Ganea et al., 2018; Shimizu al., 2020) based on the distance to margin hyperplanes, we derive a formulation in the Lorentz model.
Hyperplane in the Lorentz model Analogous to Euclidean space, hyperbolic hyperplanes split the manifold into two half-spaces, which can then be used separate instances into classes. The hyperplane in the Lorentz model is defined by geodesic that results from the intersection of an n--imensional hyperplane with the hyperboloid the ambient space Ra+1 (Cho al., 2019). Specifically, for L and w E TpL the hyperplane passing through and perpendicular to w is given by
Hw,p (o E L (w, x)c 0}.
(7)
This formulation comes with the non-convex optimization condition (w, w)c > 0, which is unde- sirable in machine learning. To eliminate this condition, we use the Euclidean reparameterization of Mishne et al. (2022), which extend to include the curvature parameter K in Appendix B.1. In short, w is parameterized by a vector ž E ToL* = [0, az/[|>]]], where a E IR and z E R". As E T T. is parallel transported p, which gives
5

---

Published as a conference paper at ICLR 2024
w = PTk PTO
0-p (z) = [sinh(V-Ka]|l=]]] cosh(VVKaaz)]
(8)
Inserting Eq. 8 into Eq. 7, the formula of the Lorentz hyperplane becomes
H2 = tx E LK cosh( cosh(v-Ka)(a v.) - sinh(v || zz = 0},
(9)
where a and z represent the distance and orientation to the origin, respectively.
Finally, we need the distance to the hyperplane to quantify the model's confidence. is formulated by the following theorem, proven in Appendix B.2.
Theorem Given a E R and E IR", the minimum hyperbolic distance from a point æ E L to the
hyperplane Hz, defined in Eq. 9 is given by
cosh (aoblv-Kaa(zeee -inn(v-KKa, |||||
dr(x, Hz,a)
sinh-
(10)
V-K
Vlloosh Ra)= (sinn((/
MLR in the Lorentz model Lebanon & Lafferty (2004) formulated the logits of the Euclidean MLR classifier using the distance from instances to hyperplanes describing the class regions. Specifically, given input I=" and C classes, the output probability of class {l,..., C}} can be expressed as
ply c æ) exp(Uw. (x)), Uw. (x) sign( (We, x>) x))l|/ (elld Hw.), we E ER"
(11)
where Hw. is the decision hyperplane of class c.
We define the Lorentz MLR without loss of generality by inserting the Lorentzian counterparts into Eq. 11. This yields logits given by the following theorem, proven Appendix B.3.
Theorem Given parameters de E R and 2e R", the Lorentz MLR's output logit corresponding to class e and input a E ILk is given by
sign(a)3 sinh (V-
22 (m) V-K
(12)
cosh( V Ka v.)) -inhh Kaa,
V (Kabb(V-K)))3 (aani//a--a-ainiii----------- |
4.4 LORENTZ RESIDUAL CONNECTION AND ACTIVATION
Residual connection The residual connection is a crucial component when designing deep CNNs. As vector addition is ill-defined in the Lorentz model, add the vector's 's space components and concatenate a corresponding time component. This is possible as point x: E 1 can be defined by an arbitrary space component E _n and time component = /I x. 1/K. Our method straightforward and provides the best empirical performance compared to other viable methods for addition implemented, i.e.. tangent space addition (Nickel & Kiela, 2018), parallel transport addition (Chami et al., 2019), Möbius addition (after projecting to the Poincaré ball) (Ganea et al., 2018), and fully-connected layer addition (Chen al., 2021).
Non-linear activation Prior works use non-linear activation in tangent space (Fan et al., 2022), which weakens the model's stability due frequent logarithmic and exponential maps. We propose simpler operation for the Lorentz model by applying the activation function to the space component and concatenating time component. For example, the Lorentz ReLU activation is given by
vI ReLLU 1| 1/K
y
ReL.U(x.)
(13)

---

Published as a conference paper at ICLR 2024
5 EXPERIMENTS
We evaluate hyperbolic models on image classification and generation tasks and compare them against Euclidean and hybrid HNN counterparts from the literature. To ensure fair comparison, in every task, we directly translate a Euclidean baseline to the hyperbolic setting by using hyperbolic modules as one-to-one replacements. All experiments are implemented in PyTorch (Paszke et al., 2019), and we optimize hyperbolic models using adaptive Riemannian optimizers (Bécigneul & Ganea, 2018) provided by Geoopt (Kochurov al., 2020), with floating-point precision set to 32 bits. We provide detailed experimental configurations in Appendix C and ablation experiments in Appendix D.
5.1 IMAGE CLASSIFICATION
Experimental setup We evaluate image classification performance using R8 (He et al., 2015b) and three datasets: CIFAR-10 (Krizhevsky, 2009), CIFAR-100 (Krizhevsky, 2009), and Tmag-ImageNet (Le & Yang, 2015). All these datasets exhibit hierarchical class relations and high hyperbolicity (low 6ret), making the use of hyperbolic models well-motivated.
For the HCNN, we replace all components in the ResNet architecture with our proposed Lorentz modules. Additionally, we experiment with novel hybrid approach (HECNN), where we employ our Lorentz decoder and replace only the ResNet encoder blocks with the highest hyperbolicity (8ret 0.2), i.e., blocks and (see Appendix D.I). To establish hyperbolic baselines the literature (Atigh et al., 2022: Guo et al., 2022) and implement hybrid HNNs with a Euclidean encoder and hyperbolic output layer (using both the Poincaré MLR (Shimizu et al., 2020) and our novel Lorentz MIR). Additionally, we report classification results for the concurrently developed fully hyperbolic Poincaré ResNet (van Spengler al., 2023). For all models, we adopt the training procedure and hyperparameters of Vriri & Taylor (2017), which have been optimized for Euclidean CNNs and yield a strong Euclidean ResNet baseline.
follow
Main results Table 1 shows that hyperbolic models using the Lorentz model achieve the highest accuracy across all datasets, outperforming both the Fuclidean and Poincaré baselines. In contrast, the Poincaré HNNs are consistently worse than the Fuclidean baseline, aligning with the results of Guo et al. (2022). Notably, only in the case of CIFAR-IO, 10,, all models exhibit equal performance, which is expected due the dataset's simplicity. We also notice that the hybrid encoder model outperforms the fully hyperbolic model, indicating that not all parts of the model benefit from hyperbolic geometry. Overall, our findings suggest that the Lorentz model is better suited for HNNs than the Poincaré ball. This may be attributed to the better numerical stability causing fewer inaccuracies (Mishne et al., 2022). Furthermore, we achieve a notable improvement (of up to 1 1.5%) in the accuracy of current HNNs. This shows the potential of using our HCNN components in advancing HNNs.
Table I: Classification accuracy (%) of ResNet- 18 models. We estimate the mean and standard deviation from five runs. The best performance is highlighted in bold (higher is better).
CIFAR-10 (frat = 0.26)
CIFAR-100 (drat = 01:33)
Imag-ema (6xet 0.20)
Euclidean (He al., 2015b)
95.1440.12
77.72+0.15
65.1940.12
Hybrid Poincaré (Guo et al., 2022)
95.04
77.1940.50
64.93
Hybrid Lorentz (Ours)
94.98 12
78.03440.2
65.63
Poincaré ResNet (van Spengler et al., 2023)
94.51
76.60.0.32
62.01
HECNN Lorentz (Ours)
95.1610.11
78.76
65.96
HCNN Lorentz (Ours)
95.1440.08
78.07
65.7140.13
Adversarial robustness Prior works have demonstrated the robustness of hyperbolic models against adversarial attacks (Yue et al., 2023; Guo et al., 2022). We expect better performance for HCNNs/HECNNS due to the bigger effect fully hyperbolic models have the embedding space as can be seen in Figure 3. We believe the benefit could come from the increased inter-class separation afforded by the distance metric which allows for greater slack in the object classification. To study this, we employ the trained models and attack them using FGSM (Goodfellow et al., 2015) and PGD (Madry et al., 2019) with different perturbations. The results in Table 2 show that our HCNN is more
7

---

Published as a conference paper at ICLR 2024
Table Classification accuracy (%) after performing FGSM and PGD attacks on CIFAR-100. We estimate the mean and standard deviation from attacking five trained models (higher is better).
FGSM
PGD
Max. perturbation
0.8/255
1.6/555
3,2/255
0.8/255
1.6/255
3.2/255
Euclidean (He al., 2015b)
66. 06.70_0.2
51.98
39.97
61.43g0:29
49. 19.76
26.30
Hybrid Poincaré (Guo et al, 2022)
64.68_0.40
53.32
37.52 37.5220.4
63.4320.44
48.41 10.0
23. 23.7840.1
Hybrid Lorentz (Ours)
65.27=0.52
53.82+0.49
40.53 50.53
64. 11.15
49.05 19.05ga.68
27. 27.17
HECNN Lorentz (Ours)
66. 66.13=0.41
55.71
42. 22...7
65.01
50.82 30.822n.37
30.344.0.2
HCNN Lorentz (Ours)
66.47.10.27
57.14.1..0
10.5
65.04.0.2s
52.25.0.34
31.77.10.00
robust, achieving up to 5% higher accuracy. In addition, and contrary to Guo et al. (2022), we observe that hybrid decoder HNNs can be more susceptible adversarial attacks than Euclidean models.
Low embedding dimensionality HNNs have shown to be most effective for low-dimensional embeddings (Peng al., 2022). To this end, reduce the dimensionality of the final ResNet block and the embeddings and evaluate classification accuracy on CIFAR-100.
Euclidean
HP
]ECNN HCNN
78
Accuracy
76
74
The results Figure verify the effec- tiveness of hyperbolic spaces with low di- mensions, where all HNNs outperform the Euclidean models. However, our HCNN and HECNN can leverage this advantage best, suggesting that hyperbolic encoders offer great opportunitics for dimensional- ity reduction and designing smaller mod- els with fewer parameters. The high performance of HECNN is unexpected as we hypothesized the fully hyperbolic model perform best. This implies that hybrid encoder HNNs might make better use of the combined characteristics of both Euclidean and hyperbolic spaces.
72
8
16
32
128
Embedding dimensions
Figure 3: CIFAR- 100 accuracy obtained with lower di- mensionalities the final ResNet block.
5.2 IMAGE GENERATION
Experimental setup Variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014) have been widely adopted in HNN research to model latent embeddings in hyperbolic spaces (Nagano al., 2019; Mathieu al., 2019; Ovinnikov, 2019; Hsu al., 2020). HNNs have shown to generate more expressive embeddings under lower dimensionalities which would make them good fit for VAEs. In this experiment, we extend the hyperbolic VAE to the fully hyperbolic setting using proposed HCNN framework and, for the first time, evaluate its performance on image generation using the standard Fréchet Inception Distance (FID) metric (Heusel et al., 2017).
Building on the experimental setting of Ghosh et al. (2019), we test vanilla VAEs and assess generative performance on CIFAR-I0 (Krizhevsky, 2009), CIFAR- 100 (Krizhevsky, 2009), and CelebA (Liu et al, 2015) datasets. We compare Following prior works, the hybrid models only include a latent hyperbolic distribution and no hyperbolic layers. Specifically, employ the wrapped normal distributions in the Lorentz model (Nagano et al., 2019) and the Poincaré ball (Mathieu et al., 2019), respectively.
HCNN-VAE against the Euclidean and two hybrid models.
Main results The results in Table 3 show that our HCNN- -AE outperforms all baselines. Likewise, the hybrid models improve performance over the Euclidean model, indicating that learning the latent embeddings in hyperbolic spaces is beneficial. This is likely due to the higher representation capacity of the hyperbolic space, which is crucial in low dimensional settings. However, our HCNN is better at leveraging the advantages of hyperbolic geometry due to its fully hyperbolic architecture. These results suggest that our method promising approach for generation and for modeling latent structures in image data.
8

---

Published as a conference paper at ICLR 2024
Table 3: Reconstruction and generation FID of manifold VAEs across five runs (lower is better).
CIFAR-10
CIFAR-100
CelebA
Rec. FID
Gen. FID
Gen. FID
Rec. FID
Gen. FID
Fuclidean
21.21
92.40ga.50
63.8
103-54gn.44
54.80
29.25
Hybrid Poincuré (Mathicu al. 2019)
39.85
90. 90.13
62.6140.45
98.19=a.s7
61.62
81.30
Hybrid Lorentz (Nagano et al,, 2019)
59. 20.47
90.91 90.91 #0.84
44..8
08.3420
24.64
82.7820.08
HCNN Lorentz (Ours)
57.78 27.78
29.20
61.41g0.84
100.27
G-1.17 11.17700.5
78.11
1.0
1.0
0.5
A 0.0
-0.5
-0.5
-1.0-
-1.0
2
-1.0
0.55
0.0
0.5
1.0
-1.0
-0.5
0.0
(a) Euclidean (81.02)
(b) Hybrid Lorentz (80.03)
(c) HCNN (74.87)
Figure 4: Embeddings of MNIST dataset 2D latent space of VAEs (with gen. FID). Colors represent golden labels and Lorentz embeddings are projected onto the Poincaré ball for better visualization.
Analysis of latent embeddings The latent embedding space is a crucial component of VAEs as influences how the data's features are encoded and used for generating the output. We visually analyze the distribution of latent embeddings inferred by the VAEs. For this, the models are retrained on the MNIST (Lecun et al., 1998) dataset with an embedding dimension dE = 2. Then, the images of the training dataset are passed through the encoder and visualized as shown in Figure 4.
We observe the formation of differently shaped clusters that correlate with the ground truth labels. While the embeddings of the Euclidean and hybrid models form many clusters that direct towards the origin, the HCNN- VAE obtains rather curved clusters that maintain a similar distance from the origin. The structures within the HCNN's latent space can be interpreted as hierarchies where the distance to the origin represents hierarchical levels. As these structures cannot be found for the hybrid model, results suggest that hybrid HNNs using only single hyperbolic layer have little impact the model's 's Fuclidean characteristics. Conversely, our fully hyperbolic architecture significantly impacts how features are represented and learned, directing the model toward tree-like structures.
CONCLUSION
In this work, proposed HCNN, generalization of the convolutional neural network that learns latent feature representations in hyperbolic spaces. To this end, we formalized the necessary modules in the Lorentz model, deriving novel formulations of the convolutional layer, batch normalization, and multinomial logistic regression. We empirically demonstrated that ResNet and VAE models based on our hyperbolic framework achieve better performance on standard vision tasks than Euclidean and hybrid decoder baselines, especially in adversarial and lower dimensional settings. Additionally, we showed that using the Lorentz model HNNs leads better stability and performance than the Poincaré ball.
However, hyperbolic CNNs are still in their early stages and introduce mathematical complexity and computational overhead. For this, we explored HECNN models with the benefit of targeting only specific parts of the encoder, allowing for faster runtimes and larger models. Moreover, our framework currently relies generalizations of neural network layers that were designed for Euclidean geometry and might not fully capture the unique properties of hyperbolic geometry. Further research needed to fully understand the properties of HCNNs and address open questions such optimization, scalability, and performance on other deep learning problems. We hope our work will inspire future research and development in this exciting and rapidly evolving field.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was performed on the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden-W Württmbert and by the Federal Ministry of Education and Research. Ahmad Bdeir and Kristian Schwethelm were funded by the European Union's Horizon 2020 research and innovation programme under the SustInAfrica grant agreement No 861924. Kristian Schwethelm was also funded by the Deutsche Forsccuugssgemeinschatt (DFG, German Research Foundation)- project number 225197905.
REFERENCES
Mina Ghadimi Atigh, Julian Schoep, Erman Acar, Nanne van Noord, and Pascal Mettes. Hyperbolic image segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 1453-4462, June 2022.
Irving Biederman.
Recognition- coyppoonntb a theory of human image understanding. Psy- chological review, 94 2:115-147, 1987. URL
CorpusID : 8054340.
Gary Bécigneul and Octavian-Eugen Ganea, Riemannian adaptive optimization methods, 2018. URL
https: aaaxiiiiiiiv.///ii wwtp///gg//b////eeeeeeeeeeeea//t 1810.00760.
James W. Cannon, William J. Floyd, Richard Kenyon, and Walter R. Parry. Hyperbolic Geometry, volume 31, MSRI Publications, 2006.
Ines Chami, Rex Ying, Christopher Ré, and Jure Leskovee. Hyperbolic graph convolutional neural networks, 2019, URL https: wwtpiviiiiive//a.a/e.aaaaaaaav.a http://wwwwaagaaaaaa////aa/ooa/t abs//b/s//// 1910. 12933.
Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li,, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. CoRR, abs/2105, 2021. URL https: //tpx//a.vvvvo/////e/ao org/ abs/2108 14686.
Hyunghoon Cho, Benjamin DeMeo, Jian Peng, and Bonnie Berger. Large-margin classification in hyperbolic space. In Kamalika Chaudhuri and Masashi Sugiyama (eds.), Proceedings of the Twenty- Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pp. 1832-1840. PMLR, 16-18 Apr 2019. URL https: https::/prooee httpss//ppoceeeingg.. mlr .rrss / v8 9/ chol 19a html.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large- scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. N0..109//VP..0009 200...500000550684
Terrance DeVries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout, 2017. URL https: / arw.i. org abs / 1100100000000001 04552.
Vincent Dumoulin and Francesco Visin. A guide to convolution arithmetic for deep learning, 2016.
URL https::/aaxiii org/ abs 07285.
Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hyperbolic vision transformers: Combining improvements in metric learning, 2022. URL 10833.
httts:://axive
Xiran Fan, Chun-Hao Yang, and Baba C. Vemuri. Nested hyperbolic spaces for dimensionality reduction and hyperbolic nn design. In 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 356-365, 2022. doi: 10.1 10.100///PRR52688 SEW.0/PRS2228.222200004444
Pengfei Fang, Mehrtash Harandi, Trung Le, and Dinh Phung. Hyperbolic geometry in computer vision: survey, 2023.
Octavian Ganea, Gary Becigneul, and Thomas Hofmann. Hyperbolic neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnet (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Asso- ciates, Inc., 2018. URL https: / wwwp://wwwwnnn.iiin...eeeeoeeooe dbab2ade8 www.08000000s000 30e.000990.aaeeme Obeal 42 P2-Paper.pd
/fiee/
10

---

Published as a conference paper at ICLR 2024
Partha Ghosh, Mehdi S. M. Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders, 2019. URL httpp:://axxx..x./// wttp://wwog//a/////000 12436.
Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations, 2015. URL http:// arxiv. org/ abs 6572.
Jacob Grosek. Fundamental reflection domains for hyperbolic tesselations. In Rose-Hulman Under- gruduate Mathematics Journal, volume 2008. URL //th::::///choarrr rose-huuman edu/ /hmmm/vo 9/ ss/1/
Y. Guo, X. Wang, Y. Chen, and S. X. Yu. Clipped hyperbolic classifiers are super-hyperbolic classifiers. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-10, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society. doi: 10. 1109/CVPR55288.2002 200200000. URL https: doi. ieeeccompttessoolety.y... 1109/ 2022.0022220000002222.0020000002
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/1512.03388, 2015b. URL hwtp:hh//axiii..rr/.e/eeo.e abs/ 111./1112....... 03385.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference Neural Information Processing Systems, NIPS' 17, pp. 6629-6644 Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Geoffrey Hinton. Some demonstrations of the effects of structural descriptions in mental im- agery. Cognitive Science, 3(3):231-250, 1979. ISSN 0364-0213, doi: https://doi.org org//1 10165806640021339990000777 URL https:/ wwwp::/ww...e..eeeeeeeee.e..eeee www/sccinncacecccecccccccccccccc arricilepp11/00040113790000
Joy Hsu, Jeffrey Gu, Gong-Her Wu, Wah Chiu, and Serena Yeung. Capturing implicit hierarchical structure in 3d biomedical images with sele-supervised hyperbolic representations, 2020. URL
https:/ arxiv. org/ abs// 2012. 01 016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on International Conference on Machine Learning Volume 37, ICML' pp. 448 456. JMLR.org. 2015.
Daniel Kahneman, Anne Treisman, and Brian Gibbs. The reviewing of object files: Object- specific integration of information. Cognitive Psychology, 24(2):175-119, 1992. ISSN 0010-0285. doi: http://db.ooggg////0010000000009 com/ htt/:/ccieeeaareeee ccw.:///cooooeerrrrrre///////l/2
URL https: /twp://wwwwwww/w. scwencedireet..........co..o.e..
V. Khrulkov, L. Mirvakhabova, Ustinova, Oseledets, and Lempitsky. Hyperbolic image embeddings. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6417-6427, Los Alamitos, CA, USA, jun 2020. IEEE Computer Society. doi: 10.1109/CVPR442600222000 000024442600222000666555 URL https: //coii lee.oootttttttssecccc..ccoo 10.1109 http://wwR222...0. 2020. 00645.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes, 2013. URL https:
www....og/bb......ototto /111. 6114.
Reinmar Kobler, Jun-ichiro Hirayama, and Motoaki Kawanabe. Controlling the fréchet variance
improves batch normalization the symmetric positive definite manifold. In ICASSP 2022 2022 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pp. 3863-3867, 2022. doi: 10. 0..1099ICASSP433222222291125

---

Published as a conference paper at ICLR 2024
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. pp. 32-33, 2009. URL httpps//wwet
toronto................. http://iiiiaaaaannnnooaatuu-eeee 2009-TR. pdÉ.
Mare Law, Renjie Liao, Jake Snell, and Richard Zemel. Lorentzian distance learning for hyperbolic representations. Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3672-3681. PMLR, 09-15 Jun 2019. URL https: procedd ngs. m1 w.l..pres///ee v law1 9a html..
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
Guy Lebanon and John Lafferty. Hyperplane margin classifiers on the multinomial manifold. In Proceedings of the Twenty- First International Conference on Machine Learning, ICML, '04, pp- 66, New York, NY, USA, 2004. Association for Computing Machinery. ISBN 1581138385 doi: 445/1015533.101533333
URL https:
1ttpg//.114//1113330.0 1015333.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-basee learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2778-23244 1998. doi: 10.1109/5.72671.
Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-Wain Wah Ngo. Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9270-9278, 2020. doi: 10.1 N..119/CVYR4220022200000999
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3730-3738, 2015. doi: 10. 10/1CC.2222............
Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differentiating through the fréchet mean, 2020. URL attps:///arxt http///awwv..orgg//////0000 00335.
at///ba//2/0 /200033300
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019.
Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Continuous hierarchical representations with poincaré variational auto-encoders, 2019. URL https://
arxiv. http://og/bbbbbbbbb.b/ 06033.
Pascal Mettes, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. Hyperbolic deep learning in computer vision: A survey, 2023.
Gal Mishne, Zhengchao Wan, Yusu Wang, and Sheng Yang. The numerical stability of hyperbolic representation learning, 2022. URL http:::: www.v.iiiiiv..iaaaaaaaaaaaaa org/ abs 22112 0018111111..... 00181.
Valter Moretti. The interplay of the polar decomposition theorem and the lorentz group, 2002. URL https::/arxiis orgg alss ht////atthhh/ttt// ph/0211047.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped normal distribution on hyperbolic space for gradient-based learning, 2019. URL https: //atxii.///x...... org/abs/ 1902. 02992.
Maximilian Nickel and Douwe Kiela. Learning continuous hierarchies the lorentz model of hyperbolic geometry, 2018. URL https: wwtp:/iivvrrr.oooooooooo abs 1806. 03417.
Natalya Fridman Noy and Carole D. Hafner. The state of the art in ontology design: A survey and comparative review. Al Magazine, 18(3):53, Sep. 1997. doi: 10. 10.166% URL httpp:://ojsaaa... aaai, wwwpaaaiiigggaaaaaaaaaaaaa wwgg//iaagzzie//nnnaaaoooo pttp:/aaaagg..eeeett.ii.i..e...e vi vi3366
12

---

Published as a conference paper at ICLR 2024
Ivan Ovinnikov. Poincaré wasserstein autoencoder. 2019. doi: 1048550ARRXIV1101.0044. URL https://arxixs org/ als /ts//9/111 01427.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019. URL http::ttpp//... http:://pppprssse nwwrrps pttppperrrccppppp////e/peeppceee 1115--pytoch-aa---iee otg.re/teeeeee.eeeeeeorrrrrrrrer pdf.
Wei Peng, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):10033-10044 2022. doi: 10. 109/TPAMI.2011.3
Xavier Pennec. Intrinsic statistics riemannian manifolds: Basic tools for geometric measurements. Journal of Mathematical Imaging and Vision, 25(1):127-1544 2006.
Eric Qu and Dongmian Zou. Lorentzian fully hyperbolic generative adversarial network, 2022. URL
https: //t.x/aaivvv////v///////ea/.aaav org/ albs///b////t 2t.ps/22000
John G. Rateliffe. Foundations Hyperbolic Manifolds. Springer, 2 edition, 2006.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and
approximate inference in deep generative models, 2014. URL 1401.4082
Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In Marc van Kreveld and Bettina Speckmann (eds.). Graph Drawing, pp. 355-366, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 378-3-644-228878-
Maximilian Seitzer. pytorch- fid: FID Score for
htttps::/ggtthu..omm
Pytorch-fid, August 2020. Version 0.2.1.
Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic neural networks++, 2020. URL https://aet //arivv.. org/ /ww./200000 /205 08210.
Max van Spengler, Erwin Berkhout, and Pascal Mettes. Poincaré resnet, 2023.
Jiexi Yan, Lei Luo, Cheng Deng, and Heng Huang. Unsupervised hyperbolic metric learning. In 2021 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12460-12469,
2021. doi: 11009//VVPR44437.2222000011
Zhen Yu, Toan Nguyen, Yaniv Gal, Lie Ju, Shekhar S. Chandra, Lei Zhang, Paul Bonnington, Victoria Mar. Zhiyong Wang, and Zongyuan Ge. Skin lesion recognition with class-hierarchy regularized hyperbolic embeddings, 2022.
Yun Yue, Fangzhou Lin, Kazunori D Yamada, and Ziming Zhang. Hyperbolic contrastive learning, 2023. URL https: /tt.xvivvv//////////////////aala orgg abs 2200. 01 409.
Huanqiu Zhang, P. Dylan Rich, Albert K. Lee, and Tatyana O. Sharpee. Hippocampal spatial representations exhibit a hyperbolic geometry that expands with experience. Nature Neuroscience, 26(1):131-139, 2023.
13

---

Published as a conference paper at ICLR 2024
A OPERATIONS IN HYPERBOLIC GEOMETRY
A.1 LORENTZ AND POINCARE
Poincaré ball The n-dimensional Poincaré ball B (B", K is defined by 5" for E
R" K| || ? 1} and the Riemannian metric g (A I where Xラ 2(1 + K|
describes the hyperbolic space by an open ball of radius V-1/K, K, see Figure 2.
A.2 LORENTZ MODEL
L
Lix
TyL
T
T,L
(a) Geodesic
(b) Tangent space
(c) Parallel Transport
Figure 5: Illustrations of geometrical operations in the 2-dimensional Lorentz model. (a) The shortest distance between two points is represented by the connecting geodesic (red line). (b) The red line gets projected onto the tangent space of the origin resulting in the green line. (c) The green line gets parallel transported to the tangent space of the origin.
(a) Lorentz rotation
(b) Lorentz boost
Figure 6: Illustration of the Lorentz transformations in the 2-dimensional Lorentz model.
In this section, we describe essential geometrical operations in the Lorentz model. Most of these operations are defined for all Riemannian manifolds and thus introduced for the general case first. However, the closed-form formulae are only given for the Lorentz model. We also provide visual illustrations in Figure 5.
Distance Distance defined as the length of the shortest path between pair of points surface. While in Euclidean geometry, this is straight line, in hyperbolic space, the shortest path is represented by curved geodesic generalizing the notion of straight line. the Lorentz model, the distance is inherited from Minkowski space. Let y IL denote two points in the Lorentz model. Then, the length of the connecting geodesic and, thereby, the distance is given by
1
de(
cosh (K(x,
(14)
V-K
and the squared distance (Law et al., 2019) by
2 K
de y) = || 2112
2(xx, y)
(15)
When calculating the distance of any point E T. the origin O, the equations be simplified to
14

---

Published as a conference paper at ICLR 2024
dc( 0) ||log (z))1
(16)
2 K
d2, (1 Vzx))
(17)
Tangent space The space around each point æ on a differentiable manifold M can be linearly approximated by the tangent space M. It is first-order approximation bridging the gap to Euclidean space. This helps performing Euclidean operations, but it introduces an approximation enror, which generally increases with the distance from the reference point. Let LLk then the tangent space at point a can be expressed as
L E (y. 0}.
(18)
Exponential and logarithmic maps Exponential and logarithmic maps mappings between the manifold M and the tangent space T.M with æ E M. The exponential map exp (z) Ta Lx + L maps tangent vector T 1. the Lorentz manifold by
exp= K (z) cosh(a)) x sinh(a) z a'
with V-K ||2||c = / (2,
(19)
The logarithmic map is the inverse mapping and maps a vector y L to the tangent space of by
cosh (B) (y Bax, with K (x. y) c-
log (y)
(20)
82 1
In the special case working with the tangent space at the origin 0, the exponential map simplifies to
K exp (z)
1
cosh (- sinh( (nnhvV-K|n K |=|
z
(21)
K
||
Parallel transport The parallel transport operation PTK --- (v) maps vector E T& from the tangent space of x M to the tangent space of y M.. It preserves the local geometry around the reference point by moving the points along the geodesic connecting æ and y. The formula for the Lorentz model is given by
PTK (1) (log (y), v) (log (y) + logy (x))
(22)
dc( y)
(,, v)
1
(x + y).
(23)
= (x, 4)
Lorentzian centroid (Law et al., 2019) The weighted centroid with respect to the squared
Lorentzian distance, which solves min Σm (x w), with æ; E L and in 2 0, > 14 > 0, is given by
Σm
(24)
Lorentz average pooling The average pooling layer is implemented by computing the Lorentzian centroid of all hyperbolic features within the receptive field.
15

---

Published as a conference paper at ICLR 2024
Lorentz transformations The set of linear transformations in the Lorentz model called Lorentz transformations. transformation matrix (*+1)x(*+1) that linearly maps IR+ - R"+1 is called Lorentz transformation if and only if (Ax, Ay) (z.z) æ, y E R=+1, The set of matrices forms an orthogonal group (1, n) called the Lorentz group. As the Lorentz model only uses the upper sheet of the two-sheeted hyperboloid, the transformations under consideration here lie within the positive Lorentz group O (1, n) = (A E O(1,n) component 24 of x E Lkk Specifically, here the Lorentz transformations can be formulated as
> 0}, preserving the sign of the time
O*11,n) (A ((n+t)x(xtt] Va E L A K' (Ax)) > 0)}.
(25)
Any Lorentz transformation be decomposed into Lorentz rotation and Lorentz boost by polar decomposition A = RB (Moretti, 2002). The former rotates points around the time axis, using matrices given by
0 R
R
(26)
0
R'R where 0 is zero vector, R R I, and det (R) 1. This shows that the Lorentz rotations for
the upper sheet lie in a special orthogonal subgroup Sot (1, n) preserving the orientation, while E (O(n). On the other side, the Lorentz boost moves points along the spatial axis given a velocity R", ||||| 1 without rotating them along the time axis. Formally, the boost matrices are given
by
Y I
B
(27)
7.0
1+7
with 7
See Figure 6 for illustrations the Lorentz rotation and Lorentz boost.
Lorentz fully-connected layer Recently, Chen et al. (2021) showed that the linear transformations performed in the tangent space (Ganea et al., 2018; Nickel Kiela, 2018) can apply all Lorentz transformations but only special rotation and no boost. They proposed a direct method in pseudo- hyperbolic space' which can apply all Lorentz transformations. Specifically, let E L denote the input vector and W E pmxx+1 E R=+ the weight parameters, then the transformation matrix is given by
([ TT W
1)
We W
1/K
DT
(M) far
(28)
Adding other components of fully-connected layers, including normalization, the final definition of the proposed Lorentz fully-connected layer becomes
(lo - 1/K oi (Wwr
(29)
with operation function
Wv(w) b W (x) + bll
Φ Wa, v) = >a x b')
(30)
where is a learnable scaling parameter and b Ra v/ a denote the bias, activation, and sigmoid function, respectively.
CChen al. (2021) note that their general formula is not fully hyperholic, but relaxation in implementation, while the input and output still guaranteed lie in the Lorentz model.
16

---

Published as a conference paper at ICLR 2024
No Sealing
10
Sealing
10
log
10a
10a
0
20
12 100
80
Figure 7: Growth of the space component's norm | after applying the exponential map to an n-dimensional vector 1, (1....,1) 1) E ToLk with curvature K -1. The y-axis is scaled logarithmically.
In this work, we simplify the layer definition by removing the internal normalization, as we batch normalization. This gives following formula for the Lorentz fully connected layer
Vlkb(Wa + - 1/K (W + 6)
y LFC(2))
(31)
Lorentz direct concatenation (Qu & Zou, 2022) the Lorentz direct concatenation is given by
Given a set of hyperbolic points {a,, L2 1.W
1i=1>
T
N
HCCat(((;)))))
Σ 2
T
T
(32)
K
N
where y E L R1N+1
Wrapped normal distribution Nagano al. (2019) proposed wrapped normal distribution in the Lorentz model, which offers efficient sampling, great flexibility, and closed-form density formulation. be constructed follows:
Sample a Euclidean vector 5 from the Normal distribution N/ Σ).
2. Assume the sampled vector lies in the tangent space of the Lorentz model's origin [0, 5] E ToL%
3. Parallel transport from the tangent space of the origin to the tangent space of a new mean
μa E LL yielding tangent vector u TT
4. Map u to L by applying the exponential map, yielding the final sample L
The distribution is parameterized by a Euclidean variance E p2 and a hyperbolic mean fs Lg-
This method has shown to work well in hybrid HNN settings. However, in our fully hyperbolic VAE, high Euclidean variances destabilize the model. This is because, usually, the VAE's prior is set to standard normal distribution with unit variance ē AoO0, I). However, for high dimensional spaces, this leads to large values after the exponential map. That is why we propose to scale the prior variance follows.
Let E ToL% denote vector in the tangent space of the origin. Then the space component of the hyperbolic vector Lg resulting from the exponential map is given by
17

---

Published as a conference paper at ICLR 2024
2, - (cxpp (b))s
V-K sinh(v -|||]]))
(33)
||0||
This shows that the norm of the space component depends on the sinh function, which grows approximately exponentially with the norm the tangent vector (||z=|l = - sinh(v- I)). The norm of the space component is important as it gets used to calculate the time component VIlz, 1/K, and it indicates how large the values of the hyperbolic points are. Now, assume an n-dimensional vector 1, = (1, .... 1) To resembling the diagonal of the covariance matrix. Applying the exponential map to such vector leads to fast-growing values with respect to the dimensionality n because the norm the tangent vector increases with n2:
|..||
12= VT.
(34)
To work against this, we propose to clip the norm of the prior variance as follows
if v
(35)
otherwise
where s parameterizes the resulting norm. This achieves clipped time component with respect to the dimensionality of 1,, (see Figure 7). Furthermore, it offers nice interpretability using the Fréchet variance. As the distribution has zero mean, the Fréchet variance is given by the distance to the origin, which can be calculated by the norm of the tangent vector. This shows that this method controls the Fréchet variance. In practice, we empirically found s = 2.5 to be good value.
Additionally, to prevent the HCNN- VAE from predicting relatively high variances, the scaling Eq. 35 is applied. In this case, the Fréchet variance is not predefined, as usually o² y 1n. However, introduces scaling operation resembling the variance scaling the prior.
A.3 MAPPING BETWEEN MODELS
Because of the isometry between models of hyperbolic geometry, points in the Lorentz model can be mapped to the Poincaré ball by the following diffeomorphism
PB (x)
(36)
Et
+
K
B PROOFS
B. PROOFS FOR LORENTZ HYPERPLANE
This section contains the proof for the Euclidean reparameterization of the Lorentz hyperplane proposed by Mishne (2022). Unfortunately, the authors only provided proof for the unit Lorentz model, i.c., assuming a curvature of K = -1. However, in general, the curvature can be different as K 0. That is why we reproduce their proof for the general case.
Proof for Eq. 8 Let E R, z E RR" and z T = [0, az/[|=[]] Then, Mishne et al. (2022) parameterize point in the Lorentz model follows
p L == expt
a
[
(37)
V-K cosh(a), sinh(a) a "
(38)
18

---

Published as a conference paper at ICLR 2024
Now, with v-
| 1 Ka get
p
cosh( V -Ka) 1
V-K"
sinh(v- Ka)
V Ka
|
(39)
cosh(V-Ka)) sinh(v
(40)
V-K
This definition gets used to reparameterize the hyperplane parameter w as follows
PT K
(p.z)c x)
= =
(0 p)
K p)
(p. [0, x2²²²²
[0, x|" +
0 +
R (0.p)c
v sinh(V (aah(v-Kaa)laa) Ka)ll=11
[0, =²²
+ coshov-
sinh( (V- Ka)
1
K + 1 K cosh ( v Ka)
V-K K
[0, x|² sinh(v-Ao =1|
1 + cosh -Ka,, sinh( -Ka) 2
+ cosh --vv-KK Ka)
|
sinh² (V-Ka)
sinh(V-Ka)Jle
1 cosh (v -Ka)
cosh? (v -Ka) -1
sinh(V-Ka)lle x 1 cosh (- -a))
2
(sinh(V-Ka)|ll]), cosh( cosh(V-Ka)z)
Proof for Eq. 9 definition becomes
After inserting Eq. 8 into Eq. and solving the inner product, the hyperplane
Hz. = fxx E Li
(anhhhKK(,,,, sinh(V-Kaj))
||||| 0}.
(41)
B.2 PROOF FOR DISTANCE TO LORENTZ HYPEPLANE
Proof for Theorem To proof the distance of point to hyperplanes the Lorentz model, we follow the approach of Cho et al. (2019) and utilize the hyperbolic reflection. The idea is, that hyperplane defines a reflection that interchanges two half-spaces. Therefore, the distance from point æ E IL's to the hyperplane Hm.p can be calculated by halving the distance to its reflection in the hyperplane x - Yo
1 de(æ, H.. dc y.w)
(42)
The hyperbolic reflection is well-known in the literature (Grosek, 2008) and can be formulated as
Fv = x 2(w, x)
(43)
(w, w)
where w the perpendicular vector to the hyperplane and (w, w) > 0. Now, inserting Eq. 43 into Eq. 42 compute the distance to the hyperplane follows
19

---

Published as a conference paper at ICLR 2024
dela, Hw.p)
cosh -1 (K(2) yw))
2V 2
K
2V-K cosh -1 (K x 2(w, c. )c (w, w)
1
(w, k) (w,w)c )c)
cosh - (2K x)) + K(ox
2V- K
(w, 20)
cosh
k + 2K
2 K
(ww, 20)
1
(w, 2))
cosh -1
1+2 V-K
2V-K K
2V
v Viw, w)
1
sinh -1
V
(w, x)/4 | lcc
V-K
which gives the final formula:
1
(V-K (wv ax)
de(x, Hw.p)
sinh
(44)
V-K
||2/|cc
Comparing Eq. 44 to the equation of Cho et al. (2019) shows that the distance formula for hyperplanes in the unit Lorentz model can be extended easily to the general case by inserting the curvature parameter K at places.
Finally, defining w with the aforementioned reparameterizzaiio
w =
PTk P
0
(x)
[sinh( (inhhv-Kaa]|z|]/ cosh(V-Kaaa))
(45)
and solving the inner products, gives our final distance formula
(v-K cosh cosh(V-Ka)(2z - sinh(V ||||| x3
dc(æ, Hz,a) = V-K
sinh -1
(46)
K
V /lloos((VKKa) (sinh( (sinh(V-Ka)|l=]]
B.3 PROOF FOR LOGITS IN THE LORENTZ MLR CLASSIFIER
Proof for Theorem 2
Following Lebanon & Lafferty (2004), given input E R and C' classes,
the Euclidean MLR logits of class c {I, C}} can be expressed as
Uw (x) = sign((w., (w., x)) [ I|d(x, Hw.), we E R"
(47)
where Hm, is the decision hyperplane of class c.
Replacing the Euclidean operations with their counterparts in the Lorentz model yields logits of class for æ E Li as follows
U " P. (x) = (w)): Hwr.p..)
(48)
with w. E LRp.L E Lg. and (w., we) > 0.
Inserting Eq. 44 into Eq. 48 gives general formula without our reparameterization
20

---

Published as a conference paper at ICLR 2024
P. (x) 1 K
(V-K*
wa m) /wollc
sign( (w.((We.
sinh
K
(49)
Now, we reparameterize w with Eq. 45 again, which gives
= (we, x) = coshlv -Kooooooooooeeeeeeee x.) -sins(s sinb(v-Ka),
(50)
-==llellee V [|cchh((
(51)
with a E IR and & E R". Finally, we obtain the equation in Theorem 2:
V-K 3 sinh (v-K²²) 3
Uze oz (x)
(52)
C ADDITIONAL EXPERIMENTAL DETAILS
C.I CLASSIFICATION
Datasets For classification, we employ the benchmark datasets CIFAR-10 (Krizhevsky, 2009), CIFAR-100 (Krizhevsky, 2009), and Tiny-ImageNer (Le & Yang, 2015). The CIFAR-10 and CIFAR- 100 datasets each contain 60,000 32 x 32 colored images from 10 and 100 different classes, respectively. We use the dataset split implemented in Py PoTorch which includes 50,000 training images and 10,000 testing images. Tiny-ImageNe is small subset of the ImageNet (Deng et al., 2009) dataset, with 100,000 images of 200 classes downsized to 64 x 64. Here, we use the official validation split for testing our models.
Settings Table 4 summarizes the hyperparameters we adopt from DeVries & Taylor (2017) to train all classification models. Additionally, we use standard data augmentation methods in training, i..., random mirroring and cropping. Regarding the feature clipping in hybrid HNNs, we tune the feature clipping parameter between 0.8 and 5.0 and find that, for most experiments, the best feature clipping parameter is r = 1. Only the Lorentz hybrid ResNet performs best with r = 4 on and 2 on CIFAR- 100 with lower embedding dimensions. Overall, observe that the hybrid Lorentz ResNet has fewer gradient issues, allowing for higher clipping values. The HCNN-ResNet does not need tuning of any additional hyperparameters.
Table 4: Summary of hyperparameters used in training classification models.
Hyperparameter
Value
Epochs
200
Batch size
128
Learning rate (LR)
le-1
Drop LR epochs
60, 120, 160
Drop LR gamma
0.2
Weight decay
5e-4
Optimizer
(Riemannian)SGD
Floating point precision
32 bit
GPU type
RTX A5000
Num. GPUs
1 or 2
Hyperbolic curvature
21

---

Published as a conference paper at ICLR 2024
C.2 GENERATION
Datasets For image generation, the aforementioned CIFAR-10 10 (Krizhevsky, 2009) and CIFAR- 100 (Krizhevsky, 2009) datasets again. Additionally, we employ the CelebA (Liu et al., 2015) dataset, which includes colored 64 64 images of human faces. Here, the Py PyTorch implementation, containing 162,770 training images, 19,867 validation images, and 19,962 testing images.
Settings For hyperbolic and Euclidean models, we use the same architecture (see Table 5) and training hyperparameterss (see Table 6). We employ vanilla VAE similar to Ghosh al. (2019) as the bascline Euclidean architecture (E- VAE). For the hybrid model, we replace the latent distribution of the E-VAE with the hyperbolic wrapped normal distribution in the Lorentz model (Nagano et al., 2019) and the Poincaré ball (Mathieu et al., 2019), respectively. Replacing all layers with our proposed hyperbolic counterparts yields the fully hyperbolic model. Here, we include the variance scaling mentioned in Section A.2, as otherwise training fails with NaN errors. Furthermore, we set the curvature for the Lorentz model to and for the Poincaré ball -0.1.
We evaluate the VAEs by employing two versions of the FID (Heusel et al., 2017) implemented by Seitzer (2020):
1. The reconstruction FID gives lower bound on the generation quality. It is calculated by comparing test images with reconstructed validation images. As the CIFAR datasets have no official validation set, we exclude fixed random portion of 10,000 images from the training set.
2. The generation FID measures the generation quality by comparing random generations from the models' latent space with the test set.
Table 5: Vanilla VAE architecture employed in all image generation experiments. Convolutional layers have a kernel size of 3 x 3 and transposed convolutional layers of 4 4. and p denote stride and zero padding, respectively. The MLR in the Euclidean model mimicked by 1 x convolutional layer.
Layer
CIFAR-10/100
CelebA
ENCODER:
PROJ +LR CONV 21.
32x32x3
64x64x3
BN RELU
16x16x64
32x32x64
CONV <2
BN + RELU
8x8x128
16x16x 128
x2,
BN RELU
4x4x256
8x8x256
BN RELU
2x2x512
4x4x512
FLATTEN
2048
8192
FC-MEAN
128
64
+ FC- VAR& + SOFTPLUS
128
64
DECODER: SAMPLE
128
64
FC + BN - RELU
32768
32768
RESHAPE
8x8x512
8x8x512
CONVTR25G.22.0 22
BN RELU
16x16x256
16x16x25
CONVTR 22 + BN + RELU
32x32x128
32x32x 128
CONVTREL.2.1
+ BN RELU
64x64 X64
CoNVoL.eL.pl
32x32x64
64x64x64
MLR
32x32x3
64x64x3
22

---

Published as a conference paper at ICLR 2024
Table 6: Summary of hyperparameters used training image generation models.
Hyperparameter
MNIST
CIFAR-10/100
CELEBA
Epochs
100
100
Batch size
100
100
100
Learning rate
5e-4
5e-4
5e-4
Weight decay
0
KL loss weight
0.312
0,024
0.09
Optimizer
(Riem) (Riem))dam
(Riem)
Floating point precision
32 bit
32 bit
32 bit
GPU type
RTX A5000
RTX A5000
RTX A5000
Num. GPUs
2
D ABLATION EXPERIMENTS
D.1 HYPERBOLICITY
Motivating the use HNNs by measuring the o-hyperbolicity of visual datasets proposed by Khrulkov et al. (2020). The idea to first generate image embeddings from a vision model and then quantify the degree of inherent tree-structure. This is achieved by considering 5-slim triangles and determining the minimal value that satisfies the triangle inequality using the Gromov product. The lower 2 0 is, the higher the hyperbol- icity of the dataset. Usually, the scale-invariant value Sret is reported. For further insights refer to Khrulkov et al. (2020).
Table 7: öyet for intermediate em- beddings of ResNet- trained on CIFAR-100.
Encoder Section
Sret
Initial Conv.
0.26
Block
0.19
Block 2
0.23
While Khrulkov al. (2020) studies the hyperbolicity of the final embeddings, we extend the perspective to intermediate embeddings from the encoder, aiming to provide additional motivation for fully hyperbolic models. For this, we first train a Euclidean ResNet-18 classifier using the settings detailed in Appendix C.I. Then, we run the model on the test dataset and extract the embeddings after each ResNet block to assess their d-hyperbolicity. The results in Table show that all intermediate embed- dings exhibit high degree of hyperbolicity. Furthermore, observe difference in hyperbolicity between blocks. This motivates hybrid hyperbolic encoders (HECNNs) with hyperbolic layer only used where hyperbolicity is the highest.
Block 3
0.18
Block
0.21
D.2 HCNN COMPONENTS
In this section, we perform ablation studies to obtain additional insights into our proposed HCNN components. All ablation experiments consider image classification CIFAR-100 using ResNet- 18. We estimate the mean and standard deviation from five runs, and the best performance is highlighted in bold.
Runtime Currently, two major drawbacks of HNNs are relatively high runtime and mem- ory requirements. This is partly due cus- tom Pythonic implementations of hyperbolic net- work components introducing significant com- putational overhead. To study the overhead in practice and assess the efficiency of our imple- mentations, we use PyTorch's 's compile function, which automatically builds a more efficient com- putation graph. We compare the runtime of our Lorentz ResNets with the Euclidean baseline under compiled and default settings in Table 8 using RTX 4090 GPU.
Table 8: Runtime improvement using PyTorch's compile function ResNet-18. Duration of training epoch in seconds.
Euclidean
Hybrid L.
HCNN
HECNN
Default Compiled
7.4
9.2
103.3
65.2
7.1
8.0
62.1
42.5
Difference
4.1%
-13.0%
-39.9%
-35.8%
23

---

Published as a conference paper at ICLR 2024
The results show that hybrid HNNSs only add little overhead compared the significantly slower HCNN. This makes scaling HCNNs challenging and requires special attention in future works. However, we also that hyperbolic models gain much more performance from the automatic compilation than the Fuclidean model. This indicates greater room for improvement in terms of implementation optimizations.
Batch normalization We investigate the ef- fectiveness of our proposed Lorentz batch nor- malization (LBN) compared to other possible methods. Specifically, we train HCNN-ResNets with different normalization methods and com- pare them against a model without normaliza- tion. The results are shown in Table 9.
Table 9: Normalization ablation.
Normalization
Accuracy (%)
None
42. 42.24g3.86
Space bnorm
77.48 48.0.
Tangent bnorm
NaN
Firstly, the batch normalization in tangent space and the Riemannian batch normalization (Lou et al. 2020) lead to infinite loss within the first few training iterations. This could be caused by the float32 precision used in this work. Addi- tionally, the normalization within the Lorentz fully-connected layer (LFC) proposed by Chen al. (2021) (see Section A.2) inhibits learning when not combined with our LBN.
Riemannian bnorm (Lou et al., 2020)
NaN
LFC norm (Chen et al., 2021)
1.00
LBN LFC norm (Chen et al., 2021) LBN
76.98
78.07
Conversely, using LBN alone improves convergence speed and accuracy significantly. getting ap- proximately 36% higher accuracy than the non-normalized model after 200 epochs. Combining LBN and the LFC normalization leads to worse accuracy and runtime, validating our modified LFC (see Section A.2). Overall, this experiment shows the effectiveness of our L.NN and suggests that, currently, there is viable alternative for HNNs using the Lorentz model. However, naive implementation operating on the hyperbolic space component can serve as a good initial baseline, although ignoring the properties of hyperbolic geometry.
Non-linear activation In this work, we propose a method for applying standard activation functions points in the Lorentz model that is more stable and efficient than the usual tangent space activations. Here, we quantify the improvement by comparing HCNN-ResNets using differ- ReLU applications. Furthermore, we consider model without activation functions, as the LFC is non-linear al- ready and might not need such functions.
Table 10: Activation ablation.
Activation
Accuracy (%)
None
52.8540.41
Tangent ReLU
77.43
Lorentz ReLU
78.07
The results in Table 10 show that non-linear activations improve accuracy significantly and are therefore needed in HCNNs. Furthermore, compared to Tangent ReLU, our Lorentz ReLU increases the average accuracy by 0.64%, decreases the runtime of a training epoch by about 16.7%, and is more consistent overall.
Residual connection In this experiment, we compare the effect of different approaches for residual connections on performance. As mentioned in Section 4.4, vector addition is ill-defined in the Lorentz model. As alternatives. we test tangent space addition (Nickel & Kiela, 2018), parallel transport (PT) addition (Chami et al., 2019), 9), Möbius addition (after projecting to the Poincaré ball) (Ganea et al., 2018), fully-connected (FC) layer addition (Chen et al., 2021), and our proposed space component addition. The results are shown in Table 11.
In training, we observe that PT and Möbius addition make the model very unstable, causing an early failure in the training process. This might be because of the float32 precision again. The tangent space addition performs relatively well, but the needed exponential and logarithmic maps add computational overhead (=s 12% higher runtime per training epoch) and some instability that hampers learning. The FC addition and our proposed space addition are very similar and perform best. However, our method simpler and therefore preferred.
24

---

Published as a conference paper at ICLR 2024
Table 11: Residual connection ablation.
Residual connection
Accuracy (%)
Tangent addition (Nickel & Kicla, 2018)
77.73
PT addition (Chami et al., 2019)
NaN
Möbius addition (Ganea et al., 2018)
NaN
FC addition (Chen et al., 2021)
77.93
Space addition
78.07
Initialization Chen et al. (2021) proposed initializing Lorentz fully-connected layers with the uniform distribution (000.22 0.02). As the LFC the backbone of our Lorentz convolutional layer, we test the uniform initialization in HCNNs and compare it to the standard Kaiming initialization (He et al., 2015a) used most Euclidean CNNs. For this, we employ the same ResNet architecture and initialize Lorentz convolutional layers with these two methods.
The results in Table 12 show that the Kaiming initialization is preferred in HCNNs, leading to 0.55% higher accuracy.
Table 12: Initialization ablation.
Initialization
Accuracy (%)
4.00.02, 0.02) (Chen et al., 2021)
77.52
Kaiming (He et al., 2015a)
78.07 20.21
25


Reviews:
[{"review_id": "LHg2t9gW9c", "reviewer": "Reviewer_Nvg1", "summary": "The paper describes layers for neural networks that operate with hyperbolic features, instead of the standard Euclidean vector features. The paper presents Convolution layers, Batch-normalization, activation function, skip connections and formula for hyperbolic decision boundaries and logits. The hyperbolic features are describe via a Lorentz model, which, compared to the Poincare model, had improved numerical stability and convenient closed form expressions for certain operations. The paper then experimentally validates the soundness of their approach in comparison to an Euclidean baseline (same architecture otherwise) and hybrid adaptions that combine Euclidean layers with hyperbolic layers. The results show a modest improvement in performance (though might not be significant), and increase robustness against adversarial attacks. With VAE experiments the authors show improved reconstruction metrics as well as a qualitatively nicer organization of the latent space.", "questions": "1. The contributions state \"superior performance\" without specifying relative to what? Regardless of the comparison, this statement should be nuanced given the discussion above.\n2. Page 3, on the Lorentz model. What is \"the upper sheet\" and what is meant with a \"two-sheeted hyperboloid\"?\n3. In Equation 1 it says $x_t > 0$. Is this because $x_t \\geq \\sqrt{-1/K}$, and if so, why not specify that instead?\n4. Section 4.1 Formalization of the convolution layer... It says \"an affine transformation between a linearized kernel and a ...\" an affine transformation is a transformation of something, not between two things right? One could say the transformation of a vector is parametrized by a kernel (and a bias), or am I misunderstanding something here?\n4. On the same note, equation 3: The definition of LFC seems essential to me, why is it hidden in the appendix and not the main body? This layer is the main work-horse of the neural network, I imagine.\n5. Next, \"the correct local connectivity is established\", what does that mean?\n6. Equation 5, why are there additional | | (vertical bars). I count three of them on each side. Two for the norm, which makes sense, but then another outside of it. Please remove it unless there is a particular reason for it.\n7. Regarding the Multinomial Logistic Regression. Is this common? I thought that usually the outputs are parametrized as multi-class Bernoullis with probabilities obtained by soft-max, not channel-wise exponentials. I.e., typically one considers mutually exclusive classes, no? Does there exists a construction for hyperbolic soft-max?\n8. In 4.4 you mention \"... provides best empirical performance\", that is nice. But what is theoretical lost with the approach of simply adding the spatial parts? Is this mathematically allowed? It feels like a hack that could theoretically break certain properties of the network (not sure about this though). I suppose since addition is not defined you need to define an alternative, and this one seems as good as anything, but until this point the motivation has been primarily the idea of parallel transport and Frechet mean type \"additions\".\n9. In section 5.1 you write \"replace only the resnet encoder blocks with the highest hyperbolicity\", what is \"hyperbolicity? What is $\\delta_{rel}$, and why would different blocks have different hyperbolicity? Many details are missing here.\n10. Also, when working with hybrid models, how does one convert a hyperbolic representation to an Euclidean one? \n11. Main result section: \"This suggests that the Lorentz model is better suited for HNNs than the Poincaré model\", was this expected, why or why not?\n12. It says the Hybrid Poincaré model is inferior to the Euclidean case. But isn't this also the case for the Hybrid Loretnz model on Cifar-10, and for the other tasks the difference seems not significant. As stated in the limitations sections above, I think the results could be more critically assessed and conclusions should be nuanced at certain locations. I might of course be mistaken in my analyses, in which case it means the paper could improve still a bit in terms of presentation and discussion of the results :)\n13. Regarding robustness experiments. Why is this a relevant experiment?\n14. Just above the conclusion \"As these structures cannot be found for the hybrid model, ... little impact ...\" I do not understand this sentence, could this be clarified? Thank you!", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* Although the first part of the methodology section makes too much use of references to the appendix, it is generally clearly written and accessible.\n* The authors provide answers to a so-far unaddressed problem (will the use of hyperbolic representations be useful beyond just as an embedding space, but instead use throughout the network)\n* The paper is self-contained and is accompanied with code", "weaknesses": "1. I find the evidence for the actual usefulness of hyperbolic representations not convincing, in contrast to what the authors present in the abstract and introduction by saying things like \"demonstrate the superiority of our HCNN framework\". This is not to downplay the results, I still think it is valuable that these original approaches are developed and explored, however, I think some conclusions could be nuanced and perhaps more precisely formulate. Which brings me to the next item\n2. It is not always clear what the hypotheses are. It seems like it is \"just interesting\" to try and build a fully hyperbolic network, without clearly specifying why this would be useful. In particular, the experiments should be accompanied with clear hypotheses of what is expected (why would the Lorentz model be better than the Poincare model if both are isometric representations of the same thing? Why would one expect HE or H model be better compared to the other? Why would the H model be more robust to adversarial attacks?). It would help if the experiments are more clearly motivated, and explain to what questions these experiments provide answers\n3. The results, in particular in Table 1, all look the same to me. Almost all results lie within each others uncertainty margins and hence drawing conclusions based on these results is dubious, especially if they as strongly state as on some instances in the paper.\n4. It would be great if some very narrowly defined experiments could be defined to test for desirable properties of the network (if this is possible). Because, the performances are all so close to each other which I think has to do with the fact that almost any neural network can get great results if you engineer it well enough, even if the code contains bugs or design flaws! However, in order to define such experiments one first has to clearly specify what unique properties are obtained by the HCNNs, which I couldn't really get from the paper, nor did I find concrete research questions that are being addressed. \n5. In the Lorentz model section, Figure 4 is reference. It is not nice to refer to a figure which isn't even included in the document. Moreover, the proper way of refering to figures in order of appearance (e.g. first refer to figure 1, then 2 then 3 etc. Not starting with fig 4). I think if this figure is that important it should be included in the main body. Moreover, it would have been nice to include the appendix in the main pdf, not as a separate document.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ZyAFasO5H", "reviewer": "Reviewer_T97r", "summary": "In this paper, the  authors propose the first fully hyperbolic convolutional neural network encoder in computer vision. In particular, the authors introduce Lorentzian formulations of the 2D convolutional layer, batch normalization, and multinomial logistic regression for constructing the fully hyperbolic convolutional neural network. Extensive experiments show that the proposed method is better than existing hybrid hyperbolic convolutional neural network.", "questions": "1. The authors show that fully  hyperbolic neural networks is better than the hybrid version, however, would fully  hyperbolic neural networks incur more parameters than the hybrid version? Also, how about the time complexity comparison？\n\n2. It is not clear what is the real difference made by Lorentzian convolutional layer compared with the standard convolutional layer, more analysis and visualizations are needed to understand the real benefits of the proposed approaches.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduced the Lorentzian formulations of the 2D convolutional layer, batch normalization, and multinomial logistic regression which are new in the literature of hyperbolic neural networks.\n\n2. The authors conducted extensive experiments (image classification, image generation) to demonstrate the effectiveness of the proposed HCNN Lorentz.", "weaknesses": "1. Across all the experiments, the improvement of the proposed HCNN Lorentz/HECNN Lorentz over the standard Euclidean neural network is usually small (65.96 vs. 65.19 on Tiny-ImageNet), so it is not clear what is the practical advantage of the proposed method over standard Euclidean neural network.\n\n2. There is a lack of in-depth analysis of the experimental results. For example, the results show that HECNN is the best when the embedding dimension is low. However, the reason of such an improvement is not clear. Similarly for the image generation task.\n \n3. It is known that hyperbolic space is well suited for hierarchical dataset, however, none of the experiments clearly demonstrate this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qW1mxMAbte", "reviewer": "Reviewer_QXC6", "summary": "This work proposes a fully hyperbolic convolutional network based on the Lorentz model of hyperbolic geometry. Lorentz formulations of Euclidean operations and extensions from the Poincare reformulations of operations such as MLR, FC, and Concat are presented. A series of networks are presented and evaluated in classification tasks, under adversarial attacks and reconstruction tasks. \\\n\nContributions:\nFully Hyperbolic convolutional neural network outperforming Poincare alternatives. \\\n \nReformulation of core operations of convolutional neural networks for the Lorentz model. \\\n \nLorentz batch normalization is proposed based on centroid. \\\n \nLorentz residual and activation functions are proposed based on space-time vector decomposition. \\", "questions": "Do you observe any need for hyperbolic non-linear activation functions given that hyperbolic networks are inherently non-linear? Some works in Poincare ball suggest that it's not essential but marginally improves performance. \\\n \nYou interchangeably reference Hybrid Lorentz as your own work and also reference (Nagano et al., 2019), can you define how yours differs? \\\n \nIn the re-scaling procedure, you assume that the variance direction is along the geodesic intersecting the origin, however, this may not be the case, therefore is not an accurate formulation. Can you elaborate if I have mis-understood. \\\n \nCan you expand on the residual connection and activation section, currently it is unclear how the time and space components (dimensions of the point) can be added without the issues that arise in hyperbolic space? You suggest that the space component lies in Euclidean space, however simply decomposing the hyperbolic vector based of the first dimension to achieve this space and time component does not align to conventional works. I would be grateful if you could elaborate. \\", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Many of the reformulations are sensible and follow well-established reformulations of hyperbolic deep learning that are widely applicable in hyperbolic deep learning in Lorentz models. \\\n \nA fully Lorentz formulation of convolutional networks is missing in the hyperbolic deep learning field and is likely to be an essential component given the improved stability of such models of hyperbolic space. Hence, this work presents a good addition to the literature. \\\n \nUsing Lorentzian centroid in the normalization is sensible and alleviates many of the computation restrictions of the frechet mean on the Poincare model. This normalization scheme seems reasonable and computationally efficient (more empirical studies to show performance would be a nice addition). \\\n \nThe experimentation demonstrates good performance increases under fully Lorentz models, although this could be expanded. \\\n \nThe work for the most part is well written and nicely organized. \\", "weaknesses": "The claims of novelty regarding the first fully hyperbolic and first hybrid CNN for vision tasks do not hold true, you do not cite, nor address the work [1]. Additionally, you reference and discuss a survey of hyperbolic deep learning for vision which contains a significant number of hybrid architectures for vision [2]. \\\n \nThe same issue holds true for batch-normalization regarding missing literature, where [1] also presented a method that does not rely on frechet mean, however, your method is the first to my knowledge to operate in the Lorentz model via such a method. \\\n \nThe experimental ablations and sensitivity are extremely limited, for example, I would expect at the least to see experiments analyzing their performance with and with the batch norm, as there exist many works that suggest batch norm is not necessary [2]. In addition, further analysis into each component, and architectural setting would be a useful insight for those wanting to utilize this work, i.e. initialisations, and curvatures. \\\n \nNo comparison is made to existing hyperbolic CNN operations i.e. Hyperbolic neural networks ++, and Poincare Resnet. Although not operating in the same hyperbolic model which is notably more unstable, a comparison would help demonstrate the benefits of your proposal. \\\n \nYou mention that alternative residual connection methods perform worse, however no results are shown for this analysis, these missing experimentation (even if placed in the supplementary) would provide a more compelling analysis. In addition, a comparison to the Poincare midpoint [1] could be made given the Lorentz model is isometric to Poincare. \\\n \nThe choice of FID is a debated metric, it has been shown that this metric does not appropriately represent the quality of generations [3]. It would be nice to see a series of the generations to compare human perception of the generations. \\\n \nYou refer to the benefit of capturing hierarchies, yet your empirical studies do not support /compare the capturing of hierarchies of any kind to support this claim. \\\n \n[1] van Spengler, M., Berkhout, E. and Mettes, P., 2023. Poincar\\'e ResNet. arXiv preprint arXiv:2303.14027.\n \n[2] Shimizu, R., Mukuta, Y. and Harada, T., 2020. Hyperbolic neural networks++. arXiv preprint arXiv:2006.08210.\n \n[3] Jung, S. and Keuper, M., 2021, December. Internalized biases in fréchet inception distance. In NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)