Title: Robust Hyperbolic Learning with Curvature-Aware Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: hyperbolic neural networks

deep learning

lorentz metric learning
Abstract: Hyperbolic deep learning has become a growing research direction in computer vision due to the unique properties afforded by the alternate embedding space. The negative curvature and exponentially growing distance metric provide a natural framework for capturing hierarchical relationships between datapoints and allowing for finer separability between their embeddings. However, current hyperbolic learning approaches are still prone to overfitting, computationally expensive, and prone to instability, especially when attempting to learn the manifold curvature to adapt to tasks and different datasets. To address these issues, our paper presents a derivation for Riemannian AdamW that helps increase hyperbolic generalization ability. For improved stability, we introduce a novel fine-tunable hyperbolic scaling approach to constrain hyperbolic embeddings and reduce approximation errors. Using this along with our curvature-aware learning schema for Riemannian Optimizers enables the combination of curvature and non-trivialized hyperbolic parameter learning. Our approach demonstrates consistent performance improvements across Computer Vision, EEG classification, and hierarchical metric learning tasks while greatly reducing runtime.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a curvature-aware optimization framework for hyperbolic deep learning, introducing a Riemannian AdamW optimizer, distance rescaling, and stable curvature learning. The method enhances training stability, generalization, and efficiency for hyperbolic models across diverse tasks. Extensive experiments show consistent performance gains and reduced computational costs in metric learning, EEG classification, and image generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed method appears theoretically sound, and addressing important challenges such as instability and overfitting during curvature learning.\n\n- The paper introduces a Riemannian AdamW variant that adapts a widely used optimizer to non-Euclidean space, improving regularization.\n\n- The proposed distance rescaling function helps to keep embeddings within numerically stable regions, and the tangent-space-based parameter remapping strategy prevents inconsistency during curvature updates.\n\n- The method demonstrates \n   + strong empirical performance across tasks like hierarchical metric learning, EEG classification, and image recognition, with consistent improvements over existing baselines.\n   + performance gains under limited data, highlighting good generalization.\n\n- The method reduces memory usage, and converges faster, and improves training stability.\n\n\nWeaknesses:\n- While the method claims to improve training stability under curvature learning, it is not always clear what baseline methods crash or fail, and under what exact conditions.\n\n- In section 4.3, lines 322-323, the reported efficiency gains rely on “closed-source CUDA convolution operations,” which raises concerns about reproducibility and whether others can replicate these results.\n\n- The notation and math in Sections 3.1-3.4 are dense and may be hard to follow without more visual aids or worked examples. This limits clarity.\n\n- In Section 4.4, line 332, the paper reports 2.5x lower memory usage and 3x faster training for the VAE, but does not provide any system details, hardware configuration, or concrete memory/runtime numbers to support this claim.\n\n- The paper proposes several components (optimizer, scaling, mapping), but the contributions of each are spread across different tasks. It’s difficult to assess how much each component helps individually.\n\nQUESTIONS:\n- The paper highlights improved stability when learning curvature. Please provide concrete cases where baseline methods diverge or crash under curvature learning, and what specific instability symptoms are observed.\n\n- In section 4.3, could the authors clarify whether these CUDA layers will be released publicly? If not, can the efficiency gains be replicated using standard PyTorch ops or approximated with open libraries?\n\n- In Section 3.3, how sensitive is the performance to the weight decay and learning rate values in the Riemannian AdamW optimizer? Have these been tuned per task?\n\n- In Section 3.4, “flexible scaling function”, how does the choice of slope “s” affect training stability or convergence? Was this fixed across all experiments?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes techniques to improve stability and efficiency in the context of deep hyperbolic neural networks (HNNs). In particular, it addresses challenges of HNNs based on the Lorentz model: overfitting, learning instability, and computational complexity. To this aim, the authors present the following solutions: (1) Extending the optimization method in Fu et al. [9], Guo et al. [14] for dealing with geometric inconsistencies during the update step of hyperbolic parameters (2) Extending AdamW for optimization on Lorentz space (3) A function to rescale points on the Lorentz space that copes with instability an performance degradation due to inaccurate approximation. The authors provide experimental evaluations on hierarchical metric learning, EEG classification, image classification and generation that show the relevance of the proposed techniques.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper deals with some key issues in learning on hyperbolic space (in particular Lorentz space).  \n- Overall, the paper is well-structured and easy to follow\n- The proposed methods improve state-of-the-art HNNs in terms of accuracy and computational efficiency\n\nWeaknesses:\n- Please see the questions below\n\nQUESTIONS:\n- In Section 3.3, does the issue only apply to hyperbolic space or to other Riemannian manifolds as well ? Can \nthe proposed optimizer be effective for other spaces ? \n\n- Could the authors provide more details on HyperMatt: how covariance matrices are mapped onto the hyperboloid and \nhow the hyperbolic attention layer [6] is used in the next step ?\n\n- Results on EEG signal classification are obtained over 3 runs which does not seem to be the same as the experimental setting \nused by the baseline MAtt (average over 10 runs). Since standard derivations of most methods are high, it is important to see how the proposed method perform for several runs. \n\n- While results on image generation show improvement w.r.t. existing hybrid and fully hyperbolic models, they seem to be largely \noutperformed by state-of-the-art methods on the consider task (e.g. diffusion models). I am wondering if those experiments are worth \ninvestigating for demonstrating the interest of the proposed techniques. How the methods in Table 4 perform w.r.t. different resolutions of input images (e.g. $8 \\times 8$  on CIFAR-100) ?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an alternative to Riemannian optimizers, adapted to Hyperbolic Neural Networks on the Lorentz manifold model of hyperbolic space. The optimizer is based on AdamW optimizer, here adapted to computations in the Lorentz model, together with a so-called \"flexible scaling function\" replacing the cutoff or normalization usually done at fixed radius in the manifold's chart centered at the origin, which has to be introduced due to numerical precision reasons. Together these technical tricks allow for better performance on a series of experiments on HNN tasks across varied architectures.\n\nSTRENGTHS AND WEAKNESSES:\nThe strength of the paper is shown by the numerical experiments, showing better performance than previous optimizers.\n\nThe weakness is that this is a technical paper, which some reviewers (not me) would consider like an incremental improvement to previous works. I believe that such improvement, be it incremental or not, is important for advancing the field of HNNs.\n\nQUESTIONS:\n1) Line 39-40: About the sentence \"The instability is only exacerbated [...]\"  can you provide a quantitative/empirical proof of this?\n\n2) Line 128-129: \"fundamental oversight: Riemannian optimizers update curvatures without adjusting dependent hyperbolic operations, weights, or gradients, creating geometric inconsistencies\" -- what do you meas by \"geometric inconsistencies\" and why do they matter at all? Please give some concrete examples.\n\n3) Line 244: You say that the flexible scaling function is useful and allows to control the difficulties mentioned in the paragraph of lines 239-243. So can you be more specific or concrete in comparing with cutoff/normalization, to prove the principles that you say the new scaling function helps with? So for example, do you have measures of gradients going to zero under normalization, and not with your new function, or measures of gradient non-smoothness under clipping, compared to your method? This (in the form of an ablation study/comparison on the experiments you already have) would help in order to justify better this method.\n\nLIMITATIONS:\nI can't imagine potential negative societal impacts, so I think this point is mute for this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents a derivation for Riemannian AdamW that helps increase hyperbolic generalization ability. For improved stability, we introduce a novel fine-tunable hyperbolic scaling approach to constrain hyperbolic embeddings and reduce approximation errors.

STRENGTHS AND WEAKNESSES:
​**Strengths**​  
- ​**Innovation**:  
  - The proposed method sounds interesting.


​**Weaknesses**​  
- ​**Insufficient Motivation**:  
  - This paper's claim that existing hyperbolic learning approaches remain prone to overfitting, computationally expensive, and susceptible to instability lacks explanatory justification or supporting references, making it unconvincing. 
  - The manuscript fails to present a rigorous justification for the effectiveness of the proposed methodology in resolving the aforementioned challenges. 
  - Additionally, the proposed work lacks a systematic comparative analysis with state-of-the-art hyperbolic techniques (e.g., HCNN) to explicitly highlight its technical superiority.
- ​**Incomplete Experimental Comparisons**:  
  - ​**Outdated Baselines**: Some experiments only compare against outdated methods. Recent hyperbolic methods are omitted.  
  - ​**Limited Scope**: Complexity and generalization ability are not involved in the experimental analysis.  
  -  **Poor Performance**: The proposed method yields only marginal performance gains, failing to outperform established baselines across multiple evaluation tasks.

QUESTIONS:
Please see Strengths And Weaknesses

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02