Robust Hyperbolic Learning with Curvature-Aware Optimization
Authors: Ahmad Bdeir
Johannes Burchert
Lars Schmidt-Thieme
Niels Landwehr
Venue: NeurIPS 2025 poster ()
Keywords: hyperbolic neural networks
deep learning
lorentz metric learning
Primary area: deep_learning

Abstract: Hyperbolic deep learning has become a growing research direction in computer vision due to the unique properties afforded by the alternate embedding space. The negative curvature and exponentially growing distance metric provide a natural framework for capturing hierarchical relationships between datapoints and allowing for finer separability between their embeddings. However, current hyperbolic learning approaches are still prone to overfitting, computationally expensive, and prone to instability, especially when attempting to learn the manifold curvature to adapt to tasks and different datasets. To address these issues, our paper presents a derivation for Riemannian AdamW that helps increase hyperbolic generalization ability. For improved stability, we introduce a novel fine-tunable hyperbolic scaling approach to constrain hyperbolic embeddings and reduce approximation errors. Using this along with our curvature-aware learning schema for Riemannian Optimizers enables the combination of curvature and non-trivialized hyperbolic parameter learning. Our approach demonstrates consistent performance improvements across Computer Vision, EEG classification, and hierarchical metric learning tasks while greatly reducing runtime.

Robust Hyperbolic Learning with Curvature-Aware Optimization
Ahmad Bdeir
Johannes Burchert ISMLL University of Hildesheim
Lars Schmidt-Thieme ISMLL
Department of Data Science University of Hildesheim Hildesheim, Germany ra -hildesheimmmmmmmmmmmm
University of Hildesheim
Hildesheim, Germany burchert@is smll
Hildesheim, Germany
schmidt: tet-thhemm@@@11 smll
Niels Landwehr
Department of Data Science
University of Hildesheim Hildesheim, Germany landwehr@uni -ii www.-hhheeeeem.mmmmmmeem de
Abstract
Hyperbolic deep learning has become growing research direction in computer vision due to the unique properties afforded by the alternate embedding space. The negative curvature and exponentially growing distance metric provide nat- ural framework for capturing hierarchical relationships between datapoints and allowing for finer separability between their embeddings. However, current hy- perbolic learning approaches are still prone to overfitting, computationally expen- sive, and prone to instability, especially when attempting to learn the manifold curvature to adapt tasks and different datasets. To address these issues, paper presents a derivation for Riemannian AdamW that helps increase hyperbolic generalization ability. For improved stability, introduce novel fine-tunable hyperbolic scaling approach to constrain hyperbolic embeddings and reduce ap- proximation errors. Using this along with curvature-aware learning schema for Riemannian Optimizers enables the combination of curvature and non-trivialized hyperbolic parameter learning. Our approach demonstrates consistent performance improvements across Computer Vision, EEG classification, and hierarchical metric learning tasks while greatly reducing runtime. Our code is publicly available http://hww...ooooooccceeeeeoo/oe
1 Introduction
Recently, hyperbolic manifolds have gained attention in deep learning for their ability to model hierarchical and tree-like data structures efficiently. Unlike Euclidean space, hyperbolic space has negative curvature, allowing it to represent exponentially growing distances. This makes these manifolds ideal for tasks like natural language processing, graph representation, and metric learning 333. By embedding data into hyperbolic space, models can capture complex relationships more effectively, and often with fewer parameters [II]. Hyperbolic geometry has also been applied in computer vision, where its ability to better separate embeddings in high-dimensional spaces has shown promise improving tasks like image classification and segmentation
few-shot
learning [25 and feature representation
These works rely two main derivations of the hyperbolic space, the hyperboloid Lorentz space (L) and the Poincaré manifold (P). Typically, the hyperboloid offers better operational stability, as
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

demonstrated by Mishne et al. (29 but lacks clear definitions for basic vector operations such as addition and subtraction. To bridge this gap, recent research has focused on defining Lorentzian variant of common deep learning operations, such as the feed-forward layer 6 g and multinomial linear regression (MIR) 3.
the convolutional
layer
However, the use of these hyperbolic operations comes with challenges. Computations in hyperbolic space are more complex and expensive, and the lack of optimized CUDA implementations drastically slows down training and increases memory requirements. Additionally, optimizing hyperbolic parameters such as class prototypes or batchnorm means can be unstable, especially in low precision floating -oontt environments. This has required research to rely on parameter clamping techniques, which can cause non- non-smooe smooth gradient updates, remain within the accurate representation radius of the manifolds [16] 29)]. The instability is only exacerbated when we incorporate the negative curvature as a learned parameter since it directly affects the embedding space. Finally, in data-scarce scenarios, the higher representational capacity makes the hyperbolic spaces more prone to overfitting, which heavily impacts their generalization ability.
This work addresses key challenges in the Lorentz model. To combat overfitting, we introduce an AdamW-based optimizer that enhances regularization and generalization in hyperbolic learning. For learning instability, we propose an optimization framework that stabilizes the learning of curvature parameters. We also present smoother scaling function to replace weight clipping, ensuring hyperbolic vectors remain within the representational radius. To address computational complexity, we introduce an implementation trick that leverages efficient CUDA-based convolutions for hyperbolic learning. Improved stability not only boosts model performance but also enables lower-precision training, further enhancing efficiency. Our contributions are then four-fold:
1. We propose an alternative schema for Riemannian optimizers that stabilize curvature learning for hyperbolic parameters and a formulation for the Riemannian AdamW
2. We propose the use of maximum distance rescaling function to restrain hyperbolic vectors within the representative radius of accuracy afforded by the number precision, even allowing for fp16 precision.
3. We present LHEIR, HyperMAnt, and HCNN+ as applications of our proposed optimization scheme for the domains of hierarchical metric learning, EEG classification, and computer vision for image classification and generation, respectively.
4. We empirically show the effectiveness of our proposed methods in five domains, hierarchical metric learning, EEG classification, graph embedding. image classification, and image generation to show the effectiveness of our optimizer in different problem settings. We improve performance in all domains with a significant computational speed-up.
2 Related Work
Hyperbolic Embeddings in Deep Learning Initially, many hyperbolic deep learning methods relied on a hybrid model architecture that utilizes Euclidean encoders and hyperbolic decoders (28). Euclidean encoders avoid the high computational complexity of hyperbolic operations, well the lack of well-defined hyperbolic alternatives for Euclidean components. However, this trend has begun to shift towards fully hyperbolic models. Chen et al. [6] propose hyperbolic components for a fully connected linear layer, a graph convolution layer, and an attention layer with the square Lorentzian distance as similarity metric. For the classification head, they learn class prototypes directly on the hyperbolic manifold and use the same distance metric as a loss estimator, following similar work in the literature 22] 19 28 This work was further extended to computer vision by Bdeir al. [3] and van Spengler et al. [40] with both developing fully hyperbolic ResNets, using Riemannian batch normalization layers that rely on learnable mean also embedded and optimized in hyperbolic space. Bdeir et al. 3 attempt to alleviate these issues by including a hybrid encoder that only applies the hyperbolic components in blocks that exhibit higher embedding hyperbolicity. Although this has led to notable performance improvements, both models suffer from upscaling issues. Attempting to apply these approaches to larger datasets or larger architectures becomes less feasible in terms of time and memory requirements. Instead, our approach places greater focus on efficient components to leverage the beneficial hyperbolic properties of the model while minimizing the memory and computational footprint.

---

Curvature Learning Previous work in hyperbolic spaces has explored various approaches to curvature learning. In their studies, Gu et al. [14] and Giovanni et al. [T3] achieve this by using a parametrization that implicitly models variable-curvane 1-curve manifold. This method enables them to simulate K-curve hyperbolic and spherical operations under constant curvature. They apply this method on mixed-curve manifold embeddings where every portion of the embedding belongs to either the Euclidean, spherical, or Poincaré manifold. Other approaches, such the one by Kochurov et al. (20), simply set the curvature to learnable parameter but do not account for the manifold changes while performing the optimization steps with the Riemannian optimizers. This leads to mathematical inconsistencies when updating the hyperbolic parameters, resulting in instability and accuracy degradation. Additionally, some methods, like the one by Kim et al. [19]. store all manifold parameters as Euclidean vectors and project them before use. While this approach partially mitigates the issue of mismatched curvature operations, requires repeated proppaaaiii through hyperbolic mappings. making it computationally expensive and more susceptible to projection errors. Other methods [ 6 40 use a combination of Fuclidean parametrization, and direct hyperbolic parameter learning depending the component and required precision.
embeddings under an explicitly defined
Methodology
3.1 The Lorentz Space
The hyperbolic space is a Riemannian manifold with a constant negative sectional curvature c O. There are many conformal models of hyperbolic space but we focus our work on the hyper- boloid, or Lorentz manifold. The n-dimensional Lorentz model L (C" o is defined with L = {ar [ xa] R=+1| (æx, x) K, 2t > 0} implying a negative curvature of R and the Lorentzian inner product as Riemannian metric g = (az, y) == -ttltt + xx. ys- It then follows that (ar, y) S K which is an important characteristic to note for the stability issues presented later on. The formulation L then presents the Lorentz manifold as the upper sheet of two-sheeted
hyperboloid centered at 0* = [[V,, 0, .J². We inherit the terminology of special relativity and refer the first dimension of a Lorentzian vector as the time component 2y and the remainder of the
vector as the space dimension g. Below are the basic operations presented by the manifold with more complex operations provided in Section A in the supplementary material.
Distance Distance in hyperbolic space is the magnitude of the geodesic forming the shortest path between two points. Let æ; y Lg. then the distance between them is given by dy. (ax, y)
VKacosh (-(a.y)) ) We also define the square distance by Law et al. 22 as df (x., y) = ||
vIR = -2K c-
Exponential and Logarithmic Maps Since the Lorentz space is Riemannian manifold, is locally Euclidean. This can best be described through the tangent space Ta,,, first-order approximation of the manifold a given point æ. The exponential map, exp K (z) : L - L then the operation that maps a tangent vector in TeL onto the manifold through exp K (z) cosh(a) a)x x + sinh(a) & with a = <I/K||=IIL. ||=||c
((z,z) z) The logarithmic map is the inverse of this mapping and can be described as log K (v) - (y-B,),, with 3 k (a. y)e-
acosh(
3.2 Riemannian Optimization Schema
Background Many existing hyperbolic models adopt hybrid learning approaches that combine Euclidean parameterization with direct hyperbolic optimization. Our work focuses on the latter. This motivates our reliance on GeoOpt (20). a widely adopted library for Riemannian optimization based on the definitions by Bécigneul and Ganea 4, whose methodology is also reflected in related work. Here, the curvature K of the hyperbolic space is set as a learnable parameter. However, empirically, we find that naively optimizing this curvature often leads to instability and performance degradation. We argue that this stems from a fundamental oversight: Riemannian optimizers update curvature without adjusting dependent hyperbolic operations, weights, or gradients, creating geometric inconsistencies.
3

---

Given a hybrid model with both Euclidean and hyperbolic parameters, we define the set of learnable parameters as 0 [0r, 0% 0x) where 8₁, are the parameters optimized in Euclidean space, of are
the parameters constrained to a Riemannian manifold M with curvature K and origin 0 and 0g are learnable curvature parameters for the manifold. When optimizing the curvature, refer to of as the hyperbolic parameters defined on the manifold with the value of the curvature parameter at timestep f. It is easy to see here that 0 is dependent on 0k since the hyperbolic parameters defined by their values. Current Riemannian optimization first calculates the Euclidean gradient g through normal backpropagation. If the parameter are currently updating is Euclidean (0g), perform a typical Euclidean update step. If the parameter being updated is hyperbolic (0f ), the optimizer projects the gradient onto the tangent space of the parameter To, in the process egrad2rgrad described in Appendix [A] Momentum vectors typically used with optimizers like Adam and SGD are also initialized and updated the tangent space. Finally, the update step is performed using some form of retraction or exponential map.
Currently, Riemannian optimizers treat 0x as a Euclidean parameter. However, since the curvature defines the geometry of the manifold, changing it during training renders prior projections, gradient momentums, and 0 parameters misaligned with the new curvature. GeoOpt attempts mitigate this instability through an "N-stabilize step." which periodically recomputes the time components of hyperbolic parameters ensure adherence to the manifold. However, this occurs after updates, failing to prevent invalid intermediate states during optimization.
As concrete example, let H and H' be two Lorentz manifolds with curvatures K and K". Let be a learnable model parameter that lies on H. meaning it satisfies (x. x)_ = -K. The hyperbolic distance from this point to itself, de(xx x), should always be O. However, if the optimizer first updates the curvature to K' but does not yet update the parameter x, any subsequent operation will use the new eurvature K" with the old parameter coordinates. The distance calculation becomes:
(x, x)x
K K"
dc(x,x) = v K" acosh
V''
acosh
K"
K is less than 1. The acosh function is undefined for inputs less than 1, leading If K < K", the term K' to NaN values and model crash. If K > K', the calculation does not crash, but the distance is no longer 0. This incorrect distance creates invalid gradients, which destabilizes training. Previous similar works relied rigorous training regimes with separate optimizers for the curvature, very small learning rates, and "burn-in" epochs (7. which we attempt to avoid. Other solutions such as clipping may prevent undefined operations but cannot easily account for the inconsistencies in the second scenario.
Curvature Aware Optimization To address this, we propose a method that groups parameters based on their respective spaces and staggers their updates. Specifically,, at timestep we first isolate 0% 0k--1 and update these parameters first. We then update 0g followed by 0x. At this stage, we have updated all the model parameters, but K, are still defined on the old curvature, so we must use a mapping function update 0% 0%
The N-stabilize step used by GeoOpt can be seen as a pseudo-map, but it alters the relative magnitudes of the hyperbolic parameters 0g and gradients G. as well as the directions of the momentums, which can degrade performance and lead to training instability. As such, we identify two alternate mapping techniques commonly used in the literature and compare their properties. We start with the scaling function used by Skopek et al. (36). Tabaghi et al. (38]. In their works the authors simply multiply 0% Ke-1 by the scaling factor ( V K_; this would automatically map all the parameters from L to Li, and scale all distances between points linearly by that factor. As such, it preserves the relative distances between the hyperbolic parameters.
K. 1'
However, the method itself presents issues during optimization and input scaling. Hyperbolic models suffer from instability and traditionally require higher precision computations to prevent invalid values. Mishne et al. (29 show that the mathematical instability is proportional to the distance between and the hyperbolic embeddings. They are also able to derive the maximum distance allowed before we begin to get undefined operations. By using the scaling factor we also scale da. 0), which could then push outside the stable range. We theorize this why works using this method need to adhere to strict training regimes, including burn-in periods without curvature learning,

---

a separate optimizer for the curvature, and very low learning rates. Clipping these parameters is one solution, but larger I values could make clipping too harsh, and removes the nice property of preserved relative distances.
Additionally, there is the issue of dealing with input projections, almost all hyperbolic learning approaches assume that the Euclidean or input data exists on the tangent plane of the origin and project onto the manifold using the exponential map. This projection preserves the norm of the Euclidean vectors as di, (2, 0). When the input is then projected onto the scaled manifold, the relative norms of the input and the hyperbolic class prototypes, for example, are now completely different, breaking their relationship. One would then have to find a way to scale the inputs while adhering to the maximum stability distance without clipping. This inconsistency is the same for mixed optimization settings where both Euclidean trivialized parametrization and direct hyperbolic parameter learning are used.
An alternative mapping method is presented in Fu et al. [0]. Guo et al. 1.5 and is based on projecting the parameters onto the tangent space the origin 0,-1 of the old curvature using the logarithmic map and then projecting back after the curvature update. We show in Appendix D that this method preserves the distances and angles between the hyperbolic parameters and the origin. These properties are important as they are considered proxies for hierarchy level, and embedding similarity. We also show why the tangent space at the origin is the most mathematically suitable space for this mapping. This method also mitigates the instability caused by scaling, which removes the need for a more complicated training regime and maintains symmetric parameter handling by operating analogously to trivialized parameter learning.
Given the above, our work relies the tangent- based mapping method. We extend it to the optimization process by additionally defining the mapping of parameter gradients and momentums from T - Thke. The entire mapping schema is shown in Algorithm I We additionally perform
empirical experiments to comparing the scaling mapping and the tangent mapping in Appendix important to emphasize that our proposed optimization scheme is compatible with existing curvature learning and meta-learning methods. Rather than being an alternative, it serves as intermediate step for updating manifold parameters during curvature changes, aimed at maintaining learning stability throughout the process.
B It is
3.3 Riemannian AdamW Optimizer
Background The AdamW optimizer was first introduced by Loshchilov and Hutter 26 and relies on an improved application of the L2-regularization factor in the base Adam optimizer. I.2- regularization works on the principle that networks with smaller weights tend to have better generaliza- tion performance than equal networks with higher weight values. Adam applied the L2-regularization during the gradient update step by incorporating it in the loss. However, Loshchilov and Hutter (26) argue that this is inconsistent since the regularization effect is reduced by the magnitude of the gradient norms. Instead, they apply the weight decay directly during the parameter update step. AdamWin is then shown generalize better and lead to better convergence and has become popular choice for many vision tasks.
Given the above, we believe AdamW is significant for hyperbolic learning. Hyperbolic spaces, with their higher representational capacity, are prone to overfitting, especially under data scareity during training (2). Thus, AdamW's enhanced L2-regularization could improve hyperbolic models.
Riemannian AdamW In the following, we derive AdamW for the Lorentz manifold and suggest its extension to the Poincaré ball. The key difference between Adam W and Riemannian Adam lies in direct weight regularization, which is challenging in Lorentz space due to the lack of an intuitive subtraction operation. One option would be following the exponential mapping and retraction typically used for the optimization step. However we propose simpler operation that reduces the need for expensive and less stable parallel transport operations. Specifically, we re-frame parameter regularization in AdamW as a weighted centroid with the origin
o - -~70-1 (1 (1-7A)0t-13 + 7AO
where 7 is the learning rate and À is the weight decay value. We can now directly translate this for hyperbolic parameters 0t, as ( 0) where ná the weighted Lorentz centroid defined in
Law et al. [22] and described in Appendix A We define the centroid weights = [1 7A, 72].
5

---

Algorithm Tangent Based Manifold Mapping
1: Given:
Hyperbolic parameters of the K-curve manifold, Parameter gradients
Tax
2: function MAP PARAMETERS
3:
for each E 0 do
4:
Gump P- (9) (9)
Parallel transport gradient to previous origin b Project parameter onto tangent space > Project back onto updated manifold
log Kc-1 (p) 0,-1
5:
6: - exPo. Ki (z)
7:
To (Guepp)
Transport gradient back for next update
8: end for
9:
end function
By removing the later gradient decay and introducing this operation as seen in Algorithm we adapt AdamW for use in the Lorentz space.
Algorithm 2 Riemannian Adam (RAdam) and Riemannian AdamW (RAdamW)
Require: Manifold M, initial parameters 0_ E EM
Require: Learning rate O, weight decay À 2 0, exponential decay rates B1, B2 E (0,
Require: u-weighted Lorentzian Centroid μr
Require: Small constant 0, max iterations T
Ensure: Optimized parameters E 0:
1: Initialize moment vectors mi
Mo u0 0 E P
2: Initialize timestep 6-0 0
3: for 1 to 7 do
4: 9: + grad ((P--1)
+\ Pt-1
5:
7d, 7>1
6:
Pt (p:: 0])
7:
9t f egrad2rgrad, p; (9t)
See Appendix A
8:
me + 31 m + -1 B1) 9t
9:
UF + 32 11-1 1 + (1 B₂) gt 3 gt
10:
The + m²/(1 (1 - 34)
0N + ut (1 -B2)
Ti t vi
13:
Pe t Retractp.-1 ( (-me)
See Appendix A
my Tpe (1))
15:
end for
return Pt.
3.4 Maximum Distance Rescaling
Background Vectors in the hyperboloid models are defined as æ = [xt, x. E L where I = VI 2/ K K = -1/c and c is the manifold curvature. As such, Lorentzian projections and operations rely on the ability to accurately calculate the corresponding time component x, for the hyperbolic vectors. In their work, Mishne et al. 29 derive a maximum value for the time component Itmax- Values above this push vectors the Lorentz manifold and onto the cone 2
defined by x =x3 One prominent example of instability caused by this is the inner product in
da(a, y) = / Kacosh (=(e_wle where the approximations can lead to undefined mathematical
values by pushing the inverse hyperbolic cosine input to less than one. Based on the above, and given specific K, we can derive maximum representational radius for the model as
K
acosh
Ttmax VK
VK K
(1)

---

Under Float32 precision, and to account for values of K < we use limit value of Flmax 2- 102. When projected onto the tangent space of the origin, this translates to ||oogæ II D 9.1. Vectors outside this radius lead to instability and performance degradation due to inaccurate approximation. This problem is only exacerbated as the dimensionality of the hyperbolic vector increases. Higher dimensional vectors tend to have larger norms which limits hyperbolic models' abilities to scale up.
To constrain hyperbolic vectors within specified maximum distance, either a normalization function or parameter clipping method is required. Parameter clipping can be challenging may lead to information loss and introduce non-smooth gradients. On the other hand, common normalization functions like tanh and the sigmoid function tend saturate quickly, limiting their effectiveness as seen in the sigmoid implementation by Chen et al. [6]
Flexible Scaling Function To address these issues, we introduce a modified scaling function, designed to provide finer control over both the maximum values and the slope of the curve. A visualization of this function is provided Figure and the formulation is presented below:
Frescaled
y |
m tanh |B
atanh(0.99)
(2)
772
where y E IRd, m is our desired maximum value, and controls the slope of the curve. We now have maximum distance value adhere to and a flexible normalizing function. To apply this to the hyperbolic embeddings, we suggest performing the scaling on the tangent plane of the origin. However, this is an expensive operation perform repeatedly, such derive in Section C the equivalent factorized form for the scaling of the space values:
Di JRC
vx
T
(3)
D(e.0)
VR
where D(x, 0) rescaled are the distances obtained by plugging (xx, D) and D
in Equation
2
Omax
Experiments
In order to empirically prove the effectiveness of proposed solutions, apply them to metric learning to test low precision learning, EEG classification for generalization ability in data hungry scenarios, and image classification and generation tasks for component efficiency. We also apply the curvature learning for most scenarios to study the new optimizer scheme and possible benefits. We include more detailed ablations and experiments graph embeddings in the Appendix
4.1 Hierarchical Metric Learning Problem
Problem Setting and Reference Model In their paper Kim et al. 19 take on the problem of hierarchical clustering using unsupervised hyperbolic loss regularizer they name HIER. This method relies on the use of hierarchical proxies as learnable ancestors of the embedded data points in hyperbolic space. In the following experiment, extend HIER to the Lorentz model (LHIER) and compare against the results provided by Kim et [19].
Moving to Lorentz Space To adapt the HIER model to the hyperboloid, we first replace the Fuclidean layer norm and linear layer with Lorentzian norm and linear layer [ We then modify the HIER loss by replacing the Poincaré distance with the Lorentzian distance and optimizing Lorentzian hierarchical proxy parameters directly on the manifold. We use our new optimization schema and apply distance rescaling before layer norm and after the linear layer.
Kim et [99 employ the Euclidean AdamW optimizer with Euclidean parameterizations of hyperbolic proxies. Their experiments use FP16 precision. As the setting is already hyperbolic, significant gains from transitioning to Lorentz space are unlikely. We use this setup to evaluate components' ability to leam curvature in low-precision, unstable environments.
Experimental Goals
7

---

Table Performance of metric learning methods on the four datasets as provided by [9] indicates models using larger input images. Network architectures are abbreviated as, R Res
CUB
Cars
SOP
Methods
Arch.
R@D
R@2
R@D
R@2
R@I
R@ 10
CNN Backbone
NSoftmax
45
R R212
61.3
73.9
84.2
90.4
78.2
90.6
PPrxyNCA++ 39
R R
69.0
79.8
86.5
92.5
80.7
92.0
Hyp 9
R512
65.5
76.2
81.9
88.8
79.9
91.5
HIER [99
R312
70.1
79.4
88.2
93.0
80.2
91.5
LHIER
R
73.4
82.4
90.0
94.0
81.9
93.1
Increase in %
4.71
3.78
2.04
1.08
1,49
1.20
Any improvements likely stem from better curvature adaptation the data and task. We follow the experimental setup in Kim et al. [19]
Results As shown in Table I LHIER Iearns curvature without stability issues. Our approach improves model performance, with recall @ I gains ranging from 1.49% to 4.71%. Additional ablation experiments with different optimizers and configurations are detailed in Table
4.2 EEG Classification Problem
Problem Setting Given EEG recordings labeled into distinct categories, the goal is to classify new recordings based on signal patterns. Each recording is a time series X E CRT, where C' is the number of channels and T is the time length. Each label is {1...., K). Given N labeled recordings ((X1, v1)...., (XN.yN)) from an unknown distribution p. the task is to train a model that maps EEG signals X their correct class.
Reference Model In their work, Pan et al. 32 introduces Matt, a Riemannian model based on the SPD manifold. The model processes data through convolutional denoisers, extracts embedding covariances, and embeds them using a Riemannian attention mechanism before projecting the outputs back into Euclidean space for classification via a linear layer. Our proposed model, HyperMatt, extends this framework by projecting the resulting covariances onto the hyperboloid instead and processing them with hyperbolic attention layer [6]. The outputs are then classified directly on the manifold using the MLR 33
Experimental Goals Hyperbolic spaces are prone to overfitting, especially in EEG classification, where models are trained per subject with limited sessions, leading to data scarcity. The proposed RAdamW optimizer theoretically improves regularization, ensuring smoother convergence. We apply method to the datasets and the evaluation criteria in Pan et al. [32 test this.
Results In Table 2 we show the performance of HyperMAIl compared to the current state-of-the- art baselines, where the results were aggregated from (5] 32 The performance for HyperMArt is measured across 10 runs and we report the mean and standard deviation. Here, our new optimizer achieves state-of-the-art results for SSVEP and ERN, the comparatively smaller datasets, which are more prone to overfitting. We also show the improved performance of our RAdam W optimizer vs the existing RAdam by training the model using both and comparing the results in Table We also achieve these results in significantly less time/epoch compared to MAtt. Specifically for MI 0.077s/epoch vs 3.4s/epoch, for SSVEP 0.08 1/espch vs 4s/epoch, and for ERN s/poohh vs 1. 1.9s/epoch, resulting in an average speed-up by factor of 42.
4.3 Standard Image Classification Problem
Problem Setting and Model In their work, Bdeir et 3 propose fully hyperbolic and hybrid encoder RessNet. To adapt our methods for both models, we introduce the distance rescaling function after every convolution the encoder. Additionally, we attempt improve the performance by performing parametrization trick existing CUDA convolutions to ensure hyperbolic outputs.

---

Table Performance comparison for the EEG datasets MI, SSVEP, and ERN. We report the average accuracy for MI and SSVEP and the AUC for ERN. The best result is highlighted in bold.
Models
MI
SSVEP
ERN
ChallowConvVie
61.8416.39
56.93+6.97
71.86:2.64
EEGNet 23
57.4311.22
53.72+7.23
74.28+22447
SCCNet 42
71.95=5.05
62.1117.70
70.93+2.31
REG-TCNet 17
67.09+-4.66
55.45+7.66
77.05+22.46
TCNct-Fusion 31
56.52+33007
45.00++6.45
70.46+2..44
FB
71.45++:4.45
53.09+5.667
60.47+3.06
MBEEGSE
64.58:16.07
56.45:17.27
75.46:12.34
Inception
62.85
62.71112.95
73.55
MAu
74.71115.01
65.5018..20
76.01112.28
HyperMAtt
74.13+3..09
68.12++2.63
7..98+1.60
Increase in %
-0.78
4.01
2.59
This is done by parametrizing the convolution weight as a rotation operation before passing the input, and then applying boost operation afterwards. The end result is then equivalent the convolution operation by Bdeir et al. S while mitigating computational complexity. Specific implementation details can be found in Section and HCNN+ respectively.
B
We denote Hybrid and Fully hyperbolic models HECNN+
Experimental Goals In their paper, notice lower performance in the fully hyperbolic models compared to the hybrid models and attribute this to instability in training. Additionally, they cite crashes and model divergence when learning the curvature instead of setting it to constant, This would then be an ideal setting to test the optimization schema and rescaling function and their impact on performance and stability.
Results In the ResNet-50 experiments in Table HECNN+ significantly outperforms both the Euclidean model and the base hybrid model across both datasets, demonstrating the positiv effect of curvature learning. We also see ~ 48% reduction in memory usage and ~ 66% reduction in runtime. We attribute this improvement to efficient closed-ouurce CUDA convolution operations we now leverage. One other benefit that we find from learning the curvature is quicker convergence, where the model is able to reach convergence 130 epochs vs the 200 epochs required by a static curve model. ResNet-18 experiments show similar findings and can be found in Appendix P
Table 3: Performance and runtime Analysis for ResNet-50 models. We report classification accuracy (%) and the best performance is highlighted in bold (higher is better). All experiments were conducted on single NVIDIA RTX 4090 GPU and an AMD EPYC 7543 CPU.
CIFAR-100
Tiny- ImageNet (6.el 0.20)
VRAM For Cifar100
lepock For Cifar100
(örei 0.23)
Euclid
78.52
66.23
4.5GB
30s
HECNN
79.83
66.30
15.6GB
300s
HECNN+
80.86
67.18
8.IGB
100s
4.4 VAE Image Generation
Experimental Setup We reproduce the experimental setup from and re-implement the fully hyperbolic VAE using our new efficient convolution and transpose convolution layers. We also use curvature learning with our adjusted Riemannian SGD learning scheme.
Results Our fully hyperbolic VAE implementation outperforms both datasets, as shown in Table 4 while using 2.5x less memory and training 3x faster. This highlights the effectiveness of our curvature learning process and efficient model components.

---

Table 4: Reconstruction and generation FID manifold VAEs across five runs (lower is better).
CIFAR-100
CelebA
Rec. FID
Gen. FID
Rec. FID
Gen. FID
Euclid
63.81 =0.47
103.51ga.84
54.80
79.25 79.25=0.89
Hybrid (IP)
62.64.0.43
98.19 10.57
54.6220061
81.30=0.56
Hybrid (T.)
62. 22.15
98.34.0.62
54.64
82.7820.93
HCNN (T.)
61.44
100.27
20.84
54.17 20.66
78.11 20.95
HCNN+ (IL)
57.69
98.14
52.73 0.227
77.98+0032
Decrease in %
6.10
2.12
2.66
0.17
5 Conclusion
In this work, we propose a robust curvature-aware optimization framework for hyperbolic deep learning, addressing challenges in stability, computational efficiency, and overfitting. By introducing Riemannian AdamW, a novel distance rescaling function, and leveraging efficient CUDA imple- mentations, approach achieved performance improvements in hierarchical metric learning, EEG classification, and image classification tasks, while reducing computational costs, highlighting the practicality of our methods for large-scale applications. These findings emphasize the importance of proper curvature adaptation in hyperbolic learning, paving the way for future research in optimizing hyperbolic embeddings across diverse fields. Further work should be done however address the limitations on theoretical reasoning and efficiency presented in Appendix F
Acknowledgements
Ahmad Bdeir was funded by the European Union's Horizon 2020 research and innovation pro- gramme under the SustInAfrica grant agreement No 861924, as well as Horizon Europe Project BioMonitor4CAP grant agreement No 101081964.
References
[1] G. A. Altuwaijri, G. Muhammad, Altaheri, and M. Alsulaiman. multi-branch convolutional neural network with squeeze-and-excitation attention blocks for eeg-based motor imagery signals classification. Diagnosticx, 12(4):995, 2022.
[2] M. G. Atigh, Schoep, E. Acar, N. van Noord, and P. Mettes. Hyperbolic image segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4453 4462, June 2022.
[3] A. Bdeir, K. Schwethelm, and N. Landwehr. Fully hyperbolic convolutional neural networks for computer vision. The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL httphttp:://g/ https:////ooeerrrrrrr net. htt//forumidd=eeee 1hN5QNh
[4] G. Bécigneul and O.-E Ganea. Riemannian adaptive optimization methods. In International Conference on Learning Representations (ICLR 2019), volume 9, pages 6384-6399, Curran, 2023.
[5] J. Burchert, T. Werner, V. K. Yalavarthi, D. C. de Portugal, M. Stubbemann, and L.. Schmidt- Thieme. Are eeg sequences time series? eeg classification with time series models and joint subject training. arXiv preprint arXiv:2404.06966, 2024.
[6] W. Chen, Han, Y. Lin, H. Zhao, Z. Liu, P. Li, M. Sun, and J. Zhou. Fully hyperbolic neural networks. In S. Muresan, P. Nakov, and A. Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Pupers), ACL. 2022, Dublin, Ireland, May 22-27, 2022, pages Computational Linguisties, 2022. doi: 10.
Association for URL httpps:
//doi org 10. 18653/ v1 v1/vv22222222 ac1/:/222222211122 long 389
10

---

[7] P. Chlenski, Q. Chu, R. R. Khan, K. Du, A. K. Moretti, and I. Pe'er. Mixed-curvatuee decision trees and random forests. 2025.
[8] J. Dai, Y. Wu, Z. Gao, and Y. Jia. A ppperbolic-to-hypebblill graph convolutional network. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR
2021, virtual. June 19-25, 2021, pages 154-163. Computer Vision Foundation / IEEE. 2021. doi: 10.1109/CVPR44447.222...000 thecvf. cww/ Convolut Somal_Neeworr... CVPR_ 2021__pppppee .html
URL http::://ppnaaceeee
Graph_
[9] A. Ermolov, L. Mirvakhabova, V. Khrulkov, N. Sebe, and I. Oseledets. Hyperbolic vision transformers: Combining improvements in metric learning. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[10] X. J., Wu, Q. Sun, C. Ji, S. Wang, J. Tan, H. Peng, and S. Yu. Ace-hgnn: Adaptive curvature exploration hyperbolic graph neural network. In 2021 IEEE international conference on duta mining (ICDM), pages 111-120. IEEE, 2021.
[11] O. Ganea, G. Becigneul, and T. Hofmann. Hyperbolic neural networks. In S. Ben- gio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, ed- itors, Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018 URL hhtpps: htttps:// http://proeeednn..... http://prreeddnngg.eriri........ ccwppps.cpppcp.cpppprrr o000 142 pdf
[12] Z. Gao, Y. Wu, Y. Jia, and M. Harandi. Hyperbolic feature augmentation via distribution estimation and infinite sampling on manifolds. Advances in neural information processing systems, 35:34421 --44445,,, 2022.
[13] F. D. Giovanni, G. Luise, and M. M. Bronstein. Heterogeneous manifolds for curvature-aware graph embedding. In ICLR 2022 Workshop on Geometrical and Topological Representation Learning, 2022. URL URL httthtttt//:oeeeeeeee
best/forummdd=rrtuxsse -axc
[14] A. Gu, F. Sala, B. Gunel, and C. Ré. Learning mixed-curvature representations in product spaces. In International Conference on Learning Representations, 2018. URL httpss//:: semanticscho
org/ Caap/Crrpuss:: 108328651
[15] H. Guo, J. Tang, W. Zeng, X. Zhao, and L. Liu. Multi-modal entity alignment in hyperbolic space. Neurocomputing, 461:598-600 60799996772200 2021.
[16] Y. Guo, X. Wang. Y. Chen, and S. X. Yu. Clipped hyperbolic classifiers are super-hyperbolic classifiers. In 2022 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-10, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society. doi: (21109/CCVRR52688222222222 1109/CVPR52688. 2022.00010
2022.00010.
URL /tt.tppee///.oieeeeeeoooeeeerree hew.:eeeeeccooottsooooeeet.oooo org/10
[17] M. Ingolfsson, M. Hersche, X. Wang, N. Kobayashi, Cavigelli, and L. Benini. Eeg-tcnet: An accurate temporal convolutional network for embedded motor-imagery brain-machine interfaces. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2958 2965. IEEE, 2020.
[18] V. Khrulkov, L. Mirvakhabova, E. Ustinova, Oseledets, and V. Lempitsky. Hyperbolic image embeddings. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[19] S. Kim, B. Jeong, and S. Kwak. HIER: metric learning beyond class labels via hierarchical regu- larization. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver. BC, Canada, June 17-24, 2023, pages 19903- 19912. IEEE, 2023. doi: 10. 109/ CVPR52729.2023.01900 URL https: /ttp://oioiooooo org/ 1 2023. 01906
[20] M. Kochurov, R. Karimov, and S. Kozlukov. Geoopt: Riemannian optimization in pytorch. CoRR, abs/2005.02819, 2020. URL https: htttpp://arxivoorg//// /2055. 02819
11

---

[21] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[22] M. Law, R. Liao, Snell, and R. Zemel. Lorentzian distance learning for hyperbolic representa- tions. In K. Chaudhuri and R. Salakhutdinov, editors. Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3672-3681. PMLR, 09-15 Jun 2019. URL http:://poooeedddee ngs mlr press/ /v97 law19a html
[23] J. Lawhern, A. J. Solon, N. R. Waytowich, S. M. Gordon, C. P. Hung, and B. J. Lance. Eegnet: a compact convolutional neural network for eeg- eeg-based coin-com interfaces. Journal of Neural Engineering, 15(5)0056013, 2018.
[24] Liu. X. Qi, J. Shi, H. Zhang, and J. Jia. Multi-scale patch aggregation (MPA) for simultancous detection and segmentation. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[25] S. Liu, J. Chen, L. Pan, C.-W. Ngo, T.-S. Chua, and Y.-G. Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9270-9278, 2020. doi: 10.1 10.1009/CVPR4440000200.... 202200099,,
[26] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL / met//forun?iddBBgghcccp
[27] R. Mane, E. Chew, K. Chua, K. K. Ang, N. Robinson, A. P. Vinod, S.-W. Lee, and C. Guan. Fbenet: A multi-view convolutional neural network for brain-computer interface. arXiv preprint arXiv:2104.01233, 2021.
[28] P. Mettes, M. G. Atigh, M. Keller-Ressel, J. Gu, and S. Yeung. Hyperbolic deep learning in computer vision: A survey. Int. J. Comput. Vis., 13219):3484-35508 2024. doi: 10.1007/ S1 166--024-0204-555.
http:tts:
org/ 10. 1007 00000///11223302-000055333
[29] G. Mishne, Z. Wan, Wang, and S. Yang. The numerical stability of hyperbolic representation learning. In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 24925-24949) PMLR, 2023. http:tttts///ooeeeeett
mlr. press/ v202 / mi shne html
[30] Moretti. The interplay of the polar decomposition theorem and the lorentz group, 2002. URL https://arxiv org/ abs wbtp//aathhhhh/2210000.
[31] Y. K. Musallam, N. I. AlFassam, G. Muhammad, S. U. Amin, M. Alsulaiman, W. Abdul, H. Altaheri, M. A. Bencherif, and M. Algabri. Electroencephalograahy based motor imagery classification using temporal convolutional network fusion. Biomedical Signal Processing and Control, 69: 102826, 2021.
[32] Y.-T. Pan, J.-L. Chou, and C.-S. Wei. Matt: A manifold attention network for eeg decoding. Advances in Neural Information Processing Systems, 15:1111-311125 2022.
[33] W. Peng, T. Varanka, A. Mostafa, H. Shi, and G. Zhao. Hyperbolic deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):10033110044, 2022. doi: 10. 10.1109/PAMM.201.3113222
[34] E. Qu and D. Zou. Hyperbolic convolution via kernel point aggregation. CoRR, abs/2306.08862. 2023. doi: 1048550/ARXXI22206608666 URL htLphhtss://do http:////oo.oog///////// .rgg/10...... 48550/a555k0 aiv. .2006 08862
[35] R. T. Schirrmeister, J. T. Springenberg, I. D. J. Fiederer, M. Glasstetter, K. Eggensperger, M. Tangermann, F. Hutter, W. Burgard, and T. Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human Brain Mapping, 38(1)):331--5400 2017.
12

---

[36] O. Skopek, O.-E. Ganea, and G. Bécigneul. Mixed-curvature variational autoencoders. In International Conference on Learning Representations, 2020.
[37] H. O. Song, Y. Xiang, S. Jegelka, and S. Savarese. Deep metric learning via lifted structured feature embedding. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[38] Tabaghi, C. Pan, E. Chien, J. Peng, and O. Milenkovic. Linear classifiers in product space forms. arXiv preprint 1rx.x:2102.10204 2021.
[39] E. W. Teh, T. DeVries, and G. W. Taylor. Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis. In European Conference on Computer Vision (ECCV). Springer, 2020.
[40] M. van Spengler, Berkhout, and P. Mettes. Poincaré resnet. IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pages 5396- 5405. IEEE, 2023. doi: 10.1 D01100//CC510000.2000 2023.0009002209999
URL.ttps: //dpi. 10./ 1109/
ICCV51070. 2023.02222000 .04499
[41] Z. Wang, W. Yan, and T. Oates. Time series classification from scratch with deep neural networks: A strong baseline. 2017 International Joint Conference on Neural Networks (IJCNN, pages 1578-1585. IEEE, 2017.
[42] C.-S. Wei, T. Koike-Akino, and Y. Wang. Spatial component-wise convolutional network (secnet) for motor-imagery eeg classification. In 2019 9th International IEEF/EMBS Conference on Neural Engineering (NER), pages 328-331. IEEE, 2019.
[43] Welinder, Branson, T. Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSI) Birds 200. Technical Report CNS-TR-2010-00 California Institute of Technology, 2010.
[44] M. Yang, M. Zhou, R. Ying, Y. Chen, and King. Hyperbolic representation learning: Revisiting and advancing. In Krause, E. Brunskill, K. Cho, B. Engelhardt, Sabato, and J. Scarlett, editors, International Conference Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 99639-39653 PMLR, 2023. http:tts/////roceeeee
mlr press/ v202 yang33y html
[45] A. Zhai and H.-Y. Wu. Classification is a strong baseline for deep metric learning. arXiv preprint arXiv: 18/1.8////... 12649, 2018.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made in our abstract and the contributions are discussed in the methodology section and evaluated with experiments
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of proposed method in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
No theorems or proofs
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we describe all the necessary steps to reproduce our method as well as pseudocode.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: We use only publicly available datasets and have attached our code in the supplemental material. The code will be released GitHub upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.coo
www.bbmmmssoooooeccc...... for more details. Sabllic/guudee//ddeSubbbbb.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https:
/tips..ccppulicc/ wwwcc//gguissseee CodeSubmiss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We do specify the dataset splits and, where applicable, use the same ones already established in the literature. We also provide README in code with all hyperparameters that can be optimized and state which optimizer is being used.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: We provide the standard deviation for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
16

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We specify the runtime per epoch for all models as well as the memory required for c00.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ht
Answer: [Yes]
Justification: We have identified no potential harms or negative social impacts. All data with human subjects is publicly available and commonly used in the literature while being fully anonymized.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The main focus of the paper is a hyperbolic optimizer. There is no inherent social impact because it not domain-specific.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
We do not use any new datasets, and all data is already well established in the literature.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Where applicable, we have credited the original authors in the README of the submitted code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, patasets.paperssee paperswi theode com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [No]
Justification: The experiments on the EEG dataset contain human subjects, however, these dataset are commonly used in the literature and publicly available.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
supplemental material is fine, but if the main contribu-
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [No]
Justification: While the experiments on the EEG datasets contain human subjects, they are commonly used in the literature and fully anonymized. During the collection of the data, there were no discernible risks.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [No]
Justification:
This paper uses LLM exclusively to improve writing and formatting.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standare components.
Please refer to our LLM policy ofttes////eerrrppcccc///oeeeeres for what should or should not be described.
20

---

A Operations in hyperbolic geometry
Parallel Transport A parallel transport operation PT m- (v) describes the mapping of a vector on the manifold from the tangent space of æ E IL to the tangent space of y E L.. This operation is given PTK (v)
(y.0)e
y)
K (w.y))
Lorentzian Centroid [2
Also denoted as pa,, is the weighted centroid between points on Ei= Paa, VI/K|I1 EE ",al
manifold based on the Lorentzian square distance. Given the weights v, μ
Optimization Operations egrad2rgrad converts Euclidean gradient (computed in the ambient space) to a Riemannian gradient by projecting it onto the tangent of the corresponding hyperbolic parameter. For the Lorentz manifold this is then:
egrad2rgrad u) (v, x) x.
K
The retraction maps the updated hyperbolic vector from the tangent space back onto the manifold. In the case of the hyperboloid this the proposed exponential mapping equation.
Lorentz Transformations In the Lorentz model, linear transformations preserving the structure of spacetime are termed Lorentz transformations. A matrix A (x+1)x(n-1) is defined Lorentz transformation it provides linear mapping from I=++1 to Rn++1 that preserves the inner product, i.e., (Axr, Ay) L (ær, v) z for all æ; y R The collection of these matrices forms orthogonal group, denoted (11, n), which is commonly referred to as the Lorentz group.
In this model, we restrict attention to transformations that preserve the positive time orientation, operating within the upper sheet of the two-sheeted hyperboloid. Accordingly, the transformations we consider lie within the positive Lorentz group, denoted O (1, n) A E (11, d11 > O, ensuring preservation of the time component sign za for any a E Lg- Specifically, in this context, Lorentz transformations satisfy the relation
o (1, A ((++1)((x)))) Vo E Li (Ax,Aa):
1 K
(Az)) 0).
(4)
Each Lorentz transformation be decomposed via polar decomposition into Lorentz rotation and a Lorentz boost, expressed as A around the time axis and is defined
RB 300 The rotation matrix designed to rotate points
0
R
(5)
where 0 represents vector, R satisfies R R I, and det 1. This structure shows that Lorentz rotations on the upper hyperboloid sheet belong to a special orthogonal subgroup, (1, which preserves orientation, with R E SO(n).
In contrast, the Lorentz boost applies shifts along spatial axes given velocity vector E with ||a|| < 1, without altering the time axis.
7
B
(6)
770
I+
with | However, this can also be any operation that scales the norms of the space values without changing the vector orientation.
B
Convolution Trick
To address the computational requirements issue, we adopt an alternative definition of the Lorentz Linear layer from Dai et al. (8), which decomposes the transformation into Lorentz boost and
21

---

(a) Tangent space at random point on the manifold
(b) Tangent space at the origin
Figure 1: Tangent planes of a hyperboloid with curvature -1 relative to another hyperboloid with curvature -0.7. Tangential properties between manifolds are better respected the origin where tangents remain parallel.
Lorentz rotation. Using this definition, we replace the matrix multiplication employed by Bdeir et al. 3] for the spatial dimensions and time component projection with learned rotation operation and a Lorentz boost. Additionally, we can achieve the rotation operation using a parameterization of the convolution weights while still relying on the CUDA convolution implementations, significantly improving computational efficiency.
To apply this concept to the convolution operation, the convolution weights, after unfolding, must form a rotation matrix. We define the dimensions of this matrix as (chamnels,, kernel kernelneight) and n' = channelsout respectively. We then use either rotation operation presented above to a norm-po norm-preserving transformation = W T where W R(n'-n), This formula- tion allows us to utilize existing efficient implementations of the convolution operation by directly parameterizing the kernel weights before passing them into the convolutional layer. Finally, formalize the new Lorentz Convolution as:
out DoontzDoossSSaaccccccciiinnnnnnn
where TanhRescaling is the operation described in Eq 3 and RotationConvolution is a normal convo- lution parameterized through the procedure in Algorithm where Transform the norm-preserving transformation above.
Algorithm 3 Lorentz Convolution Parameterization
W e Con Kuitn Keegh
2: function ADAHTWEIGHT
3:
if K Kwidth Keengh Ci < Cout then
4:
W + reshape W, Kwidth Kiength Cins Cout)
5: W core Transform(W) W)
6:
W + reshape( W. Cin: Cout: Kwitths Klengt))
7:
end if
8:
return W
9:
end function
C Scaling Lorentzian Vectors
Tanh Scaling We show the output of the tanh scaling function in Figure 2
By changing the
transformation parameters we are able to fine-tune the maximum output and the slope to match our desired function response.
22

---

Figure 2: The output of the proposed flexible tanh function. Here the maximum value m is set to 9.1 in the vanilla version with alternate value of m=18 and the slope is set 2.6 with alternate value of 3.5
Hyperbolic Scaling We isolate the transformation of the exp K (v) operation on the space values of y as:
1 y
F; V x sinh
K
where y E Rd = logø (æ:). However, at the tangent plane of the origin, the first element yo becomes As such ||//|1.- = |/yl| Ei= v? This gives us:
Cg VV= x sinh 1
(9)
We scale the norm of the Euclidean vector y bay value and find the equivalent value for the hyperbolic space elements:
sinh(
ax|ulls
ax Iy ls
Lilxx VR
Breacoled
VK
VT
(10)
sinh( Vill VK
19 Tulk
VR
VR
Additionally, know that the hyperbolic distance from the origin of the manifold any point is equal to the norm of the projected vector onto the tangent plane. Supposing that we want ((x, 0) D(x, 0) resculed: we get the final equation:
D(s. T)
Die. Ty K
E Precclted
Bg X
(11)
e
D Proofs
In the following, we show that the distances and angles between the origin and the hyperbolic points are preserved. This is relatively trivial but we add for sake of clarity. For this discussion, we work in the Lorentz manifold and repeat the definitions here for easier referencing.
L = := x
Xs] E 1=+1 | (x, x) K, 0}. x,y >c= Ziii + x.y, and the origin is O OO, O.J KK,
also define two hyperboloids H and H' with curvatures K and K" respectively. Let v, be two points on H and v, w E ToH be their logarithmic map onto the tangent space at the origin of H.
23

---

Proof of distance preservation We can show that distances to the origin are preserved because (x, O) collapses to |////lleucssd This gives (x, O) || exclid dx(('' O').
Proof of angle preservation We can similarly show that angles w.r.t to the origin are preserved. The angle between x and y at O is actually computed via the Euclidean inner product in To using
U, W
cos
As such the angle w.r.t. origins also does not change when moving between manifolds of different curvature using this method since v and w don't change.
E
Additional Experiments
E.1
Image Classification
Problem Setting and Reference Model In their work, Bdeir et al. S proposed fully hyperbolic 2D) convolutional layer by breaking down the convolution operation into window-unfolding step followed by a modified version of the Lorentz Linear Layer from Chen et al. [6]. This approach ensured that the convolution outputs remained on the hyperboloid. However, the manual patch creation combined with matrix multiplication made the computation extremely expensive, as prevented the use of highly optimized CUDA implementations for convolutions. As such, the authors proposed a two versions of their hyperbolic ResNet classifier. A fully hyperbolic model with all Lorentz ResNet blocks (HCNN) and a hybrid encoder model with alternating Euclidean and hyperbolic blocks (HECNN).
Table 5: Performance and runtime Analysis for the ResNet-18 models. We report classification accuracy (%) and estimate the mean and standard deviation from five runs. The best performance is highlighted in bold (higher is better).
CIFAR-100 (6.et 0.23)
Tiny-ImageNet (6rel = 0.20)
VRAM For Cifar100
Lepoch For Cifar100
Euclidean
77.7220.15
65.19
1.2GB
12s
Hybrid Poincaré 16
77. 77.1920.60
64.932a..8
Hybrid Lorentz 3
78,03 65.6320.10
Poincaré ResNet 40
76,6020.32
62.01
HECNN (T.)
78.76.0.24
65.96.0.18
4.3GB
100s
HECNN+ (ours)
78.80
65.98
3GB
80s
HCNN K
(L.
78.07 20
65.71=0.13
10GB
175s
H (ours)
78.81
66.12
5GB
140s
Resnet-18 Experiments Table that the new models are able to remain stable while learning the curvature. Additionally, we see significant performance improvements between HCNN and HCNN+, where our proposed architecture now matches the performance of the hybrid encoders. We hypothesize that the improved scaling function helps mitigate the previous performance incon- sistencies. However, the performance difference between the hybrid models is significant, hypothesize this is due to the alternating architecture which could limit the effect and instability of hyperbolic components. Finally, both models manage maintain performance while reducing the memory footprint and runtime by approximately 25 50% and 18 25% respectively.
Ablation Experiments In table 6 we run ablation experiments to verify the effectiveness of our individual components. Specifically, the default case refers the current model with the tanh rescaling function, learn able curvature, and our proposed optimization scheme. In the setting "fixed curve" we use non-learnable cur- vature K 1, and keep the tanh rescaling. The use of the new optimization schema here has effect since
Table 6: Resnet-50 Ablations on Cifar- 1000
CIFAR-100
Default HCNN+ + fixed curve
HCNN+
80.86
79.6
HCNN+ + scaling
80.13
HCNN+ optim scheme
Noa.N
24

---

Table 7: Ablation on components for Metric Learning
CUB
Cars
SOP
Methods
Arch.
R @
R@2
R@1
R@2
R
R@ 10
CNN Backbone
LHIER- RSGD
R
64.2
72.1
73.9
81.4
67.8
73.9
LHIER RAdam
R512
70.1
79.8
87.6
92.0
79.8
90.3
LHIER Fixed Curvature
R512
71.2
80.8
88.7
91.8
79.1
89.7
LHIER Optim Scheme
R512
72.0
LHIER Default
R512
73.4
82.4
90.0
94.0
81.9
93.1
curvature is not learnable which means the staggered updates and parameter projections the setting "no scaling" we continue to learn the curvature with the new optimization schema but do not use the tanh rescaling. And in the setting "no optim scheme", we learn the curvature and use the scaling but do not use the new optimization schema. As we can see, the best results are achieved when all the architectural components are included. In the case of attempting learn the curvature without the proposed optimizer schema, the model breaks completely down due to excessive numerical inaccuracies.
needed. In
E.2
Metric Learning
Problem Setting and Reference Model In their paper [19] rely on a hybrid hyperbolic architecture for modeling hierarchical relationships in image datasets. They include new hyperbolic metric learning loss dubbed HIER which takes into consideration the intrinsic hicrarchy the data. Given triplet of points ri, æj, or where H, and zj determined to be related by reciprocal nearest neighbor measure, and Itk is an unrelated point, the HIER loss calculated as
LIBER (t)
(D Pig) Da(a;, Piyk) + 8]
+ [Dn(sj,Pi) Dn(z,, (isk) + 6]+
(12)
+ [DB(F Pijks DB(*k, Piz) 6]+
where D3 denotes the hyperbolic distance on the Poincaré ball, and Pij is the most likely least common ancestor of points z; and zj. This encourages smaller hyperbolic distance between 2;, Ty. and Pij. and a larger distance with Pijk- The opposite signal is then applied in the case of 2k- the irrelevant data point. Kim al. [19] show substantial performance uplifts for the HIER loss when applied to variety of network architectures. We rely on four main datasets: CUB-200-2011 (43, Cars-196 (2ars))[1] Stanford Online Product (SOP) and In-shop Clothes Retrieval (InShop)]224 Performance is measured using Recall @k k, the fraction of queries with at least one relevant sample in their k-nearest neighbors. All model backbones pre-trained ImageNet for fair comparisons.
Ablation Experiments In table 7 verify the effectiveness of our RAdamW optimizer, along with the additional components proposed. Specifically, "default" refers to the current LHIER model with learnable curvature and the optimization scheme trained using R.damW. The remaining settings are the same as default but differ the mentioned component e.g. "RSGD" is trained with RSGD instead of RAdamW.
E.3 EEG Classification
Ablation Experiments We also train the model with the original Adam Optimizer and include the results in Table This clearly shows the improvement provided by our RAdam W optimizer over standard RAdam.
E.4
Graph Learning
Problem Setting We follow the experimental setup described in Chen et al. 6 for the node classification problem in four popular graph embedding datasets. We choose the node classification
25

---

Table 8: Performance comparison for the EEG datasets MI, SSVEP, and ERN. We report the average accuracy for MI and SSVEP and the AUC for ERN. The best result is highlighted in bold.
Models
MI
SSVEP
ERN
HyperMAtt+ + RAdam
68,53+1.29 74.1242.91
62.42=2.62
74.98 78.01
HyperMAtt + RAdamW
68.10=2.41
task since the task decoder used by the fully hyperbolic graph convolution network (GCN) [ relies on class prototypes learned directly on the hyperboloid. Additionally, the model is already Lorentz-based, this means we do not need to modify it, and we can test our components directly in simple problem setting. Table T summarizes the training conditions for the ablations performed. Here, checked AdamW refers the of the Euclidean AdamW for the trivialized model (hyperbolic parameters learned on the tangent space of the origin) and our proposed hyperbolic RAdam for the others. We keep the same hyperparameters defined in Chen et al. [6 for all training settings. Additionally, for the scalemap- bbsed method we scale the input data to match the new norms of the manifold before projecting.
Results From table 9 we can see that that the model using Learnable curvature, our RAdamW and the Iangent-based scaling achieves the highest performance. It should also be noted that the performance values for the the HyboNet paper are extracted directly from the paper [6] we were able reproduce all results except the Disease dataset where could only achieve score of 91 % using their setting.
Table 9: Results on the node classification task using GCNs.
Disease
Airport
PubMed
Cora
Trivialized
AdamW
Learnable K
Scale Map
HyboNet
96
90.9
80.2
HI
94.09
92.56
76.60
80.80
H2
/
92.89
93.25
77.23
81.5
HB
/
V
94.82
93.6
78.3
82.1
H4
V
/
/
91.94
91.3
77.10
80.3
H5
/
/
94.88
93.55
81.8
F
Limitations
The proposed work still suffers greatly from the computational complexity introduced by hyperbolic operations. The effect of approximations for common operations such as the exponential map and the logarithmic map should be studied to reduce these issues. Additionally, alternatives for moving between manifolds should be researched and correlated with the task at hand and the loss being used. The different properties for the different alternatives could be better leveraged depending on the desired component outputs.
26


Reviews:
[{"review_id": "vtgiC1vper", "reviewer": "Reviewer_1Enn", "summary": "This paper presents a derivation for Riemannian AdamW that helps increase hyperbolic generalization ability. For improved stability, we introduce a novel fine-tunable hyperbolic scaling approach to constrain hyperbolic embeddings and reduce approximation errors.", "questions": "Please see Strengths And Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "​**Strengths**​  \n- ​**Innovation**:  \n  - The proposed method sounds interesting.\n\n\n​**Weaknesses**​  \n- ​**Insufficient Motivation**:  \n  - This paper's claim that existing hyperbolic learning approaches remain prone to overfitting, computationally expensive, and susceptible to instability lacks explanatory justification or supporting references, making it unconvincing. \n  - The manuscript fails to present a rigorous justification for the effectiveness of the proposed methodology in resolving the aforementioned challenges. \n  - Additionally, the proposed work lacks a systematic comparative analysis with state-of-the-art hyperbolic techniques (e.g., HCNN) to explicitly highlight its technical superiority.\n- ​**Incomplete Experimental Comparisons**:  \n  - ​**Outdated Baselines**: Some experiments only compare against outdated methods. Recent hyperbolic methods are omitted.  \n  - ​**Limited Scope**: Complexity and generalization ability are not involved in the experimental analysis.  \n  -  **Poor Performance**: The proposed method yields only marginal performance gains, failing to outperform established baselines across multiple evaluation tasks.", "was_revised": true, "final_justification": "The authors’ rebuttal provides a clear and well-reasoned explanation. However, due to the memory consumption and computational cost of the proposed algorithm, it may be difficult to directly scale it to large models. Overall, the paper presents a solid theoretical exploration. I am willing to adjust my score to Borderline Accept.", "rebuttal": "Thank you for your review. \n\n&nbsp;\n\n### 1. \"This paper's claim that existing hyperbolic learning approaches remain prone to overfitting, computationally expensive, and susceptible to instability lacks explanatory justification or supporting references...\"\n\n**Instability and computational costs:** These are widely recognized challenges with hyperbolic machine learning. They are, for example, discussed in the survey papers linked below as [1] and [2], and there are also full papers dedicated exclusively to the study of instability in hyperbolic learning, see [3]. Additionally, we provide concrete examples of instability in the rebuttals to reviewers k73U and 7vca below. We will discuss this background more clearly in the final version of the paper. For computational cost, we demonstrate significant reductions in runtime and memory in Table 3 of our paper, Table 5 in the appendix, and in the response to Reviewer 7vca. \n\n**Overfitting:** We address overfitting by introducing Riemannian AdamW. As discussed in the paper, the improved L2-regularization of AdamW is designed to enhance generalization. The performance gains we demonstrate in data-scarce settings (e.g., EEG classification in Table 2) are consistent with the expected benefits of this stronger regularization.\n\n&nbsp;\n\n### 2. \"the proposed work lacks a systematic comparative analysis with state-of-the-art hyperbolic techniques (e.g., HCNN) to explicitly highlight its technical superiority” and “Outdated Baselines: Some experiments only compare against outdated methods. Recent hyperbolic methods are omitted.”:\n\nWe actually compare to the work of Bdeir et al. (both HECNN and HCNN) in Table 3\nand Table 4 in the main text and Table 5 in the appendix. To the best of our knowledge, we compare against the most recent and relevant baselines. If there are other specific methods you believe should be included as baselines, we would be happy to discuss them.\n\n&nbsp;\n\n### 3. \" \"Complexity and generalization ability are not involved in the experimental analysis.\"\n\nWe agree that our experimental analysis should explicitly validate our claims of improving computational complexity and generalization. We provide evidence for this in the paper as follows:\n\n**Computational Complexity:** We report a detailed analysis of runtime and memory usage for our method vs. the HECNN baseline in Table 3, showing a ~66% reduction in runtime and a ~48% reduction in VRAM. We provide further analysis in Table 5 of the appendix, and for the VAE models in our response to Reviewer 7vca.\n\n**Generalization Ability:** We address generalization through improved regularization (as mentioned in our first point) and demonstrate its effectiveness with performance gains in data-hungry settings such as EEG classification (Table 2) and hierarchical metric learning (Table 1).\n\n&nbsp;\n\n### 4. \"The proposed method yields only marginal performance gains, failing to outperform established baselines across multiple evaluation tasks.\"\n\nOur results in Tables 1, 2, 3, and 4 show consistent performance improvements over the established baselines across nearly all evaluated tasks and data sets. Could you please clarify which specific task or baseline you are referring to where our method does not show a clear outperformance?\n\n&nbsp;\n\n[1] Peng, Wei, et al. \"Hyperbolic deep neural networks: A survey.\" _IEEE Transactions on pattern analysis and machine intelligence_ 44.12 (2021): 10023-10044.\n[2] Mettes, Pascal, et al. \"Hyperbolic deep learning in computer vision: A survey.\" _International Journal of Computer Vision_ 132.9 (2024): 3484-3508.\n[3] Mishne, Gal, et al. \"The numerical stability of hyperbolic representation learning.\" _International Conference on Machine Learning_. PMLR, 2023."}, {"review_id": "uQZUF3A6sY", "reviewer": "Reviewer_k73U", "summary": "The paper proposes an alternative to Riemannian optimizers, adapted to Hyperbolic Neural Networks on the Lorentz manifold model of hyperbolic space. The optimizer is based on AdamW optimizer, here adapted to computations in the Lorentz model, together with a so-called \"flexible scaling function\" replacing the cutoff or normalization usually done at fixed radius in the manifold's chart centered at the origin, which has to be introduced due to numerical precision reasons. Together these technical tricks allow for better performance on a series of experiments on HNN tasks across varied architectures.", "questions": "1) Line 39-40: About the sentence \"The instability is only exacerbated [...]\"  can you provide a quantitative/empirical proof of this?\n\n2) Line 128-129: \"fundamental oversight: Riemannian optimizers update curvatures without adjusting dependent hyperbolic operations, weights, or gradients, creating geometric inconsistencies\" -- what do you meas by \"geometric inconsistencies\" and why do they matter at all? Please give some concrete examples.\n\n3) Line 244: You say that the flexible scaling function is useful and allows to control the difficulties mentioned in the paragraph of lines 239-243. So can you be more specific or concrete in comparing with cutoff/normalization, to prove the principles that you say the new scaling function helps with? So for example, do you have measures of gradients going to zero under normalization, and not with your new function, or measures of gradient non-smoothness under clipping, compared to your method? This (in the form of an ablation study/comparison on the experiments you already have) would help in order to justify better this method.", "limitations": "I can't imagine potential negative societal impacts, so I think this point is mute for this paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The strength of the paper is shown by the numerical experiments, showing better performance than previous optimizers.\n\nThe weakness is that this is a technical paper, which some reviewers (not me) would consider like an incremental improvement to previous works. I believe that such improvement, be it incremental or not, is important for advancing the field of HNNs.", "was_revised": true, "final_justification": "The authors addressed minor questions that I have, and confirmed my understanding of the paper. I therefore did not change my scores, and am comfortable with them.", "rebuttal": "Thank you for your detailed feedback and insightful questions! We hope we can address any concerns below.\n\n&nbsp;\n\n### Question 1: \"Line 39-40: About the sentence 'The instability is only exacerbated [...]' can you provide a quantitative/empirical proof of this?\"\n### Question 2: \"Line 128-129: '...what do you mean by 'geometric inconsistencies' and why do they matter at all? Please give some concrete examples.\"\n\nWe address these two questions, which are both related to instability, together.\n\nInstability arises when an optimizer updates the manifold's curvature (K) without correctly mapping the parameters on that manifold to the new geometry. This creates \"geometric inconsistencies\" where fundamental mathematical properties of the space are violated during updates, leading to crashes or incorrect gradients.\n\nHere is a concrete example:\n\nFor easier reference we'll repeat some definitions here $\\mathcal{L}^n := [\\mathbf{x}=[x_t, \\mathbf{x_s}]\\in \\mathbb{R}^{n+1}|<\\mathbf{x}, \\mathbf{x}>_{\\mathcal{L}}= -K,x_t>0]$ \n\nwhere $<\\mathbf{x},\\mathbf{y}>_{\\mathcal{L}}= -x_ty_t+\\mathbf{x}_s^T\\mathbf{y}_s$  \n\nLet $H$ and $H'$ be two Lorentz manifolds with curvatures $K$ and $K'$. Let $x$ be a learnable model parameter that lies on $H$, meaning it satisfies $\\langle x, x \\rangle_{\\mathcal{L}} = -K$. The hyperbolic distance from this point to itself, $d_{\\mathbb{L}}(x,x)$, should always be $0$.\n\nIf the optimizer first updates the curvature to $K'$ but does not yet update the parameter $x$, any subsequent operation will use the new curvature $K'$ with the old parameter coordinates. The distance calculation becomes:\n$d_{\\mathbb{L}}(x,x) = \\sqrt{K'} \\cdot \\text{acosh}\\left(\\frac{-\\langle x, x \\rangle_{\\mathcal{L}}}{K'}\\right) = \\sqrt{K'} \\cdot \\text{acosh}\\left(\\frac{K}{K'}\\right)$\n* If $K < K'$, the term $\\frac{K}{K'}$ is less than $1$. The $\\text{acosh}$ function is undefined for inputs less than $1$, leading to \\textit{NaN} values and a model crash.\n\n* If $K > K'$, the calculation does not crash, but the distance is no longer $0$. This incorrect distance creates invalid gradients, which destabilizes training and requires the very strict training regimes (e.g., small learning rates, long warm-ups) that our method seeks to avoid.\n\nThis is why previous similar works rely on rigorous training regimes with separate optimizers for the curvature, very small learning rates, and \"burn-in\" epochs [5,6,7]. Other solutions such as clipping may prevent undefined operations but cannot easily account for the inconsistencies in the second scenario. \n\n&nbsp;\n\n### “Line 244: ...can you be more specific or concrete in comparing with cutoff/normalization, to prove the principles that you say the new scaling function helps with?\"\n\n\n\nEmpirically, when using the scaling function vs the sigmoid function and expmap output clipping used in Chen et al. we noticed a few things:\n* We no longer required strict gradient clipping to 1\n* Slightly faster convergence times (~170 epochs vs 200 epochs for vision tasks when not learning the curvature for example)\n* Much less sensitivity to learning rate and weight decay \n\nFrom a theoretical standpoint, previous work has shown that hard clamping, such as ReLU could lead to \"dead neurons\" during training due to the 0 gradient and non-smooth training which leads to slower convergence [2]. Other works such as [3], show that hard clamping requires specific clamping thresholds to assure convergence and can fail under the standard noise assumptions. \n\nAlternatively, using other activation functions such as sigmoid or vanilla tanh leads to gradient vanishing in the saturation regions as the derivative of the function approaches 0 on either side [1]. \n\nTo try and relate the findings to previous literature, we will include a figure of the gradient distribution for parameters per layer during random training epochs. We did this for our method and other clamping and vanilla scaling approaches and show that other methods have a higher proportion of 0 gradients and more gradient spikes. We have not done extensive empirical ablations however (only for vision task) due to the computation cost but we will try to add more to the appendix.\n\n&nbsp;\n\n[1] Nguyen, Anh, et al. \"An analysis of state-of-the-art activation functions for supervised deep neural network.\" _2021 International conference on system science and engineering (ICSSE)_.\n[2] Horuz, Coşku Can, et al. \"The Resurrection of the ReLU.\" _arXiv preprint arXiv:2505.22074_ (2025).\n\n[3] Koloskova, Anastasia, Hadrien Hendrikx, and Sebastian U. Stich. \"Revisiting gradient clipping: Stochastic bias and tight convergence guarantees.\" _International Conference on Machine Learning_. PMLR, 2023.\n\n[4] Mishne, Gal, et al. \"The numerical stability of hyperbolic representation learning.\" _International Conference on Machine Learning_. PMLR, 2023.\n\n[5] Chlenski, Philippe, et al. \"Mixed-curvature decision trees and random forests.\" Forty-second International Conference on Machine Learning (2024).\n\n[6] Skopek, Ondrej, Octavian-Eugen Ganea, and Gary Bécigneul. \"Mixed-curvature Variational Autoencoders.\" International Conference on Learning Representations.\n\n[7] Chlenski, Philippe, et al. \"Manify: A Python Library for Learning Non-Euclidean Representations.\" arXiv preprint arXiv:2503.09576 (2025)."}, {"review_id": "cXbnhsRTur", "reviewer": "Reviewer_Es5F", "summary": "This paper proposes techniques to improve stability and efficiency in the context of deep hyperbolic neural networks (HNNs). In particular, it addresses challenges of HNNs based on the Lorentz model: overfitting, learning instability, and computational complexity. To this aim, the authors present the following solutions: (1) Extending the optimization method in Fu et al. [9], Guo et al. [14] for dealing with geometric inconsistencies during the update step of hyperbolic parameters (2) Extending AdamW for optimization on Lorentz space (3) A function to rescale points on the Lorentz space that copes with instability an performance degradation due to inaccurate approximation. The authors provide experimental evaluations on hierarchical metric learning, EEG classification, image classification and generation that show the relevance of the proposed techniques.", "questions": "- In Section 3.3, does the issue only apply to hyperbolic space or to other Riemannian manifolds as well ? Can \nthe proposed optimizer be effective for other spaces ? \n\n- Could the authors provide more details on HyperMatt: how covariance matrices are mapped onto the hyperboloid and \nhow the hyperbolic attention layer [6] is used in the next step ?\n\n- Results on EEG signal classification are obtained over 3 runs which does not seem to be the same as the experimental setting \nused by the baseline MAtt (average over 10 runs). Since standard derivations of most methods are high, it is important to see how the proposed method perform for several runs. \n\n- While results on image generation show improvement w.r.t. existing hybrid and fully hyperbolic models, they seem to be largely \noutperformed by state-of-the-art methods on the consider task (e.g. diffusion models). I am wondering if those experiments are worth \ninvestigating for demonstrating the interest of the proposed techniques. How the methods in Table 4 perform w.r.t. different resolutions of input images (e.g. $8 \\times 8$  on CIFAR-100) ?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper deals with some key issues in learning on hyperbolic space (in particular Lorentz space).  \n- Overall, the paper is well-structured and easy to follow\n- The proposed methods improve state-of-the-art HNNs in terms of accuracy and computational efficiency\n\nWeaknesses:\n- Please see the questions below", "was_revised": true, "final_justification": "This paper proposes techniques to address the problems of overfitting, learning instability, and computational complexity when training HNNs with the Lorentz model. The practical benefits of the proposed methods have been demonstrated in four different applications. While in some cases, the proposed methods do not show state-of-the-art results compared to existing works, I think these methods are promising to advance the relevant literature. During the rebuttal, the authors have addressed my concerns appropriately in their answers as I have confirmed in my discussion. I did not update my original score because it is already above 3, and I think the impact of the proposed methods to a broader range of applications is somewhat limited (due to the performance issue mentioned above).", "rebuttal": "Thank you for your thorough review and constructive feedback. \n\n\n### Q1: \"issue only apply to hyperbolic space or to other Riemannian manifolds as well ?\"\n\nWe show a direct example of the instability from optimization in the rebuttal to reviewer K73U, and while this example is specific to the Lorentz manifold, it can be extended to other manifolds. For example, we define two Poincare manifolds $P$ and $P'$ with radii $K$ and $K'$ respectively. Let $x$ be a point on $P$. The distance from x to the origin is defined as $d_\\kappa(\\mathbf{x}, \\mathbf{0}) = K \\cdot \\operatorname{atanh}\\left(\\frac{|\\mathbf{x}|}{K}\\right)$, normally this is well-defined because for any $\\mathbf{x}$ in $P$, $|\\mathbf{x}| < K$. However, if we now update $K \\rightarrow K'$ and $K' < K$, the upper bound of $\\frac{|\\mathbf{x}|}{K}$ is no longer 1 and we get undefined values. For $K' > K$ we get geometric inconsistencies where the distances between points are distorted non-linearly.\n\nEven in spherical manifolds, the operations rely on the property $|\\mathbf{x}| = \\sqrt{K}$ which becomes invalid if we update K without updating the vector and use the new operations anyways. As such, our learning scheme should be able to apply to any Riemannian manifold that has a strict condition on its vectors w.r.t its curvature.\n\n&nbsp;\n\n### Q2. \"Could the authors provide more details on HyperMatt\"\n\nWe first project the covariance matrix into the ambient Euclidean space through the use of the logmap. In order to prevent duplicate values, we take the upper half of the matrix and flatten it, and then we project it onto the hyperboloid using the expmap at the origin. Technically, there is no isometric mapping between the SPD manifold and the Lorentz manifold which is why we leverage the ambient Euclidean space as a middle point for the mapping. We have also tried to use treat the SPD values as Euclidean and directly calculate a time component for them but that yielded worse results.\n\n\n&nbsp;\n\n\n### Q3. \"Results on EEG signal classification are obtained over 3 runs which does not seem to be the same as the experimental setting used by the baseline MAtt (average over 10 runs).\"\n\nThis is an excellent point. We have rerun the EEG classification experiments for 10 runs to align with the baseline's evaluation protocol. We report the updated mean and standard deviation below and have included the original MAtt results for convenience. While the standard deviation has changed slightly, our method's strong performance holds, and we will update the main paper with these new results.\n\n| Model     | MI               | SSVEP            | ERN              |\n| --------- | ---------------- | ---------------- | ---------------- |\n| MAtt      | $74.71 \\pm 5.01$ | $65.50 \\pm 8.2$  | $76.01 \\pm 2.28$ |\n| HyperMAtt | $74.13 \\pm 3.09$ | $68.12 \\pm 2.63$ | $77.98 \\pm 1.6$  |\n\n&nbsp;\n\n### Q4. \"...they seem to be largely outperformed by state-of-the-art methods on the considered task (e.g. diffusion models).\"\n\nThis is a fair observation. While our work provides significant computational efficiency improvements for hyperbolic models, translating the largest state-of-the-art generative models (e.g., diffusion models) is still very challenging computationally. Our goal was to demonstrate the effectiveness of our components in established hyperbolic VAE architectures. We agree that scaling these methods to SOTA generative frameworks such as diffusion models is a very interesting future direction, but we believe the work required would merit a separate, dedicated study."}, {"review_id": "210vVHN0c3", "reviewer": "Reviewer_7vca", "summary": "This paper proposes a curvature-aware optimization framework for hyperbolic deep learning, introducing a Riemannian AdamW optimizer, distance rescaling, and stable curvature learning. The method enhances training stability, generalization, and efficiency for hyperbolic models across diverse tasks. Extensive experiments show consistent performance gains and reduced computational costs in metric learning, EEG classification, and image generation.", "questions": "- The paper highlights improved stability when learning curvature. Please provide concrete cases where baseline methods diverge or crash under curvature learning, and what specific instability symptoms are observed.\n\n- In section 4.3, could the authors clarify whether these CUDA layers will be released publicly? If not, can the efficiency gains be replicated using standard PyTorch ops or approximated with open libraries?\n\n- In Section 3.3, how sensitive is the performance to the weight decay and learning rate values in the Riemannian AdamW optimizer? Have these been tuned per task?\n\n- In Section 3.4, “flexible scaling function”, how does the choice of slope “s” affect training stability or convergence? Was this fixed across all experiments?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed method appears theoretically sound, and addressing important challenges such as instability and overfitting during curvature learning.\n\n- The paper introduces a Riemannian AdamW variant that adapts a widely used optimizer to non-Euclidean space, improving regularization.\n\n- The proposed distance rescaling function helps to keep embeddings within numerically stable regions, and the tangent-space-based parameter remapping strategy prevents inconsistency during curvature updates.\n\n- The method demonstrates \n   + strong empirical performance across tasks like hierarchical metric learning, EEG classification, and image recognition, with consistent improvements over existing baselines.\n   + performance gains under limited data, highlighting good generalization.\n\n- The method reduces memory usage, and converges faster, and improves training stability.\n\n\nWeaknesses:\n- While the method claims to improve training stability under curvature learning, it is not always clear what baseline methods crash or fail, and under what exact conditions.\n\n- In section 4.3, lines 322-323, the reported efficiency gains rely on “closed-source CUDA convolution operations,” which raises concerns about reproducibility and whether others can replicate these results.\n\n- The notation and math in Sections 3.1-3.4 are dense and may be hard to follow without more visual aids or worked examples. This limits clarity.\n\n- In Section 4.4, line 332, the paper reports 2.5x lower memory usage and 3x faster training for the VAE, but does not provide any system details, hardware configuration, or concrete memory/runtime numbers to support this claim.\n\n- The paper proposes several components (optimizer, scaling, mapping), but the contributions of each are spread across different tasks. It’s difficult to assess how much each component helps individually.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal, which successfully addressed all of my initial questions. After carefully reading the other reviewers' feedback, I agree that the paper makes a valuable contribution by providing a robust and efficient optimization framework for hyperbolic networks. Therefore, I am maintaining my initial positive rating.", "rebuttal": "Thank you for your interest in the work!\n\n### Q1. \"Please provide concrete cases where baseline methods diverge or crash under curvature learning, and what specific instability symptoms are observed.\"\n\nInstability arises when an optimizer updates the manifold's curvature (K) without correctly mapping the parameters on that manifold to the new geometry. This creates \"geometric inconsistencies\" where fundamental mathematical properties of the space are violated during updates, leading to crashes or incorrect gradients.\n\nHere is a concrete example:\n\nFor easier reference we'll repeat some definitions here $\\mathcal{L}^n := [\\mathbf{x}=[x_t, \\mathbf{x_s}]\\in \\mathbb{R}^{n+1}|<\\mathbf{x}, \\mathbf{x}>_{\\mathcal{L}}= -K,x_t>0]$ \n\nwhere $<\\mathbf{x},\\mathbf{y}>_{\\mathcal{L}}= -x_ty_t+\\mathbf{x}_s^T\\mathbf{y}_s$  \n\nLet $H$ and $H'$ be two Lorentz manifolds with curvatures $K$ and $K'$. Let $x$ be a learnable model parameter that lies on $H$, meaning it satisfies $\\langle x, x \\rangle_{\\mathcal{L}} = -K$. The hyperbolic distance from this point to itself, $d_{\\mathbb{L}}(x,x)$, should always be $0$.\n\nIf the optimizer first updates the curvature to $K'$ but does not yet update the parameter $x$, any subsequent operation will use the new curvature $K'$ with the old parameter coordinates. The distance calculation becomes:\n$d_{\\mathbb{L}}(x,x) = \\sqrt{K'} \\cdot \\text{acosh}\\left(\\frac{-\\langle x, x \\rangle_{\\mathcal{L}}}{K'}\\right) = \\sqrt{K'} \\cdot \\text{acosh}\\left(\\frac{K}{K'}\\right)$\n* If $K < K'$, the term $\\frac{K}{K'}$ is less than $1$. The $\\text{acosh}$ function is undefined for inputs less than $1$, leading to \\textit{NaN} values and a model crash.\n\n* If $K > K'$, the calculation does not crash, but the distance is no longer $0$. This incorrect distance creates invalid gradients, which destabilizes training and requires the very strict training regimes (e.g., small learning rates, long warm-ups) that our method seeks to avoid.\n\nThis is why previous similar works rely on rigorous training regimes with separate optimizers for the curvature, very small learning rates, and \"burn-in\" epochs [5,6,7]. Other solutions such as clipping may prevent undefined operations but cannot easily account for the inconsistencies in the second scenario. \n\n&nbsp;\n\n### Q2. \"...efficiency gains rely on 'closed-source CUDA convolution operations,' which raises concerns about reproducibility... will be released publicly?\"\n\nThis is a misunderstanding, apologies for the confusion. We do not develop any novel closed-source CUDA operations. Instead, our method provides a novel reparameterization that allows us to leverage the highly optimized, standard PyTorch convolution layers (which use closed-source CUDA implementations under the hood) for hyperbolic operations.\n\nNormally, these standard layers cannot be used as they do not preserve the manifold's geometry. Our work describes the hyperbolic convolution as a sequence of a rotation and a boost. By reparameterizing the PyTorch convolution weights to perform the rotation, we can use the efficient native operation and then apply a standard boost operation. This method is detailed in Appendix B. It still does not achieve the same speeds as the default pytorch convolutions since the parametrization is non-trivial and requires matrix projections and reshaping but it is much faster and more memory efficient than the naive convolution operations used in previous work [1] and is fully reproducible with standard libraries.\n\n&nbsp;\n\n### Q3. \"...does not provide any system details, hardware configuration, or concrete memory/runtime numbers to support this claim.\"\n\nThank you for pointing this out, this should indeed have been given in the paper. All experiments were conducted on a single NVIDIA RTX 4090 GPU with 24GB of VRAM and an AMD EPYC 7543 CPU. We provide the concrete memory and runtime numbers for the VAE experiment below. These results are consistent with the efficiency gains for ResNet-50 reported in Table 3 of the main paper.\n\n| Model Type          | Runtime (s/epoch) | VRAM (GB) |\n| ------------------- | ----------------- | --------- |\n| With Bdeir Conv [1] | 58                | 13.2      |\n| With Our Conv       | 19                | 5.48      |\n\n&nbsp;\n\n### Q4. \"the contributions of each are spread across different tasks. It's difficult to assess how much each component helps individually.\"\n\nWe were unfortunately unable to add these ablations studies to the  main paper due to the page limit, however, the appendix (specifically tables 6, 7, 8, and 9) have extensive experiments with and without each of the included components in all of the experimental settings. If you believe the experiments could benefit from any particular additional ablations, please let us know and we can try to add them in as well.\n\n&nbsp;\n\n### Q5. \"...how sensitive is the performance to the weight decay and learning rate values in the Riemannian AdamW optimizer? Have these been tuned per task?\"\nFor most experiments, we found the optimizer to be robust and simply used the same hyperparameters as the original Euclidean baseline models. The one exception was the EEG classification task, which is known to be sensitive to hyperparameters due to high noise and limited per-subject data. For this task, we performed a small grid search over 3 learning rates and 3 weight decay values to find the optimal configuration, following prior work.\n\n&nbsp;\n\n### Q6. \"...how does the choice of slope 's' affect training stability or convergence? Was this fixed across all experiments?\"\n\nThe slope s was fixed to a value of 3 for all experiments across all tasks and datasets. While we observed in preliminary experiments that performance can degrade with very low (s<1) or very high (s>6) slope values, the model was not overly sensitive to the exact value within a reasonable range. We chose to fix the value rather than tune it per task to avoid introducing an additional hyperparameter to the optimization process and to demonstrate the general robustness of the approach.\n\n&nbsp;\n\n[1] Bdeir, Ahmad, Kristian Schwethelm, and Niels Landwehr. \"Fully hyperbolic convolutional neural networks for computer vision.\" ICLR 2024"}]

Author rebuttal:

Meta-review: The authors propose a robust curvature-aware optimization framework for hyperbolic deep learning, addressing challenges related to stability, computational efficiency, and overfitting. They introduce Riemannian AdamW, which enhances regularization and generalization in hyperbolic learning, a novel distance-rescaling function that ensures hyperbolic vectors remain within the representative radius of accuracy afforded by numerical precision, and a practical trick that enables efficient CUDA implementations. The efficiency of the proposed method is demonstrated through experiments on several tasks.

The reviewers acknowledge that the paper constitutes a meaningful contribution to hyperbolic learning, noting that:
1. the proposed idea is reasonable and sound
2. the method appears to address key issues present in previous approaches
3. the experimental results convincingly support the claims

Most concerns raised during the review process were addressed in the rebuttal phase. Overall, the paper seems as a significant and impactful contribution, providing a robust and efficient optimization framework for hyperbolic networks.

I therefore recommend acceptance and encourage the authors to consider the feedback and update the paper accordingly.
Decision: Accept (poster)