Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings
Authors: Sarang Rajendra Patil
Ashish Parmanand Pandey
Ioannis Koutis
Mengjia Xu
Venue: Accepted by TMLR ()

Abstract: Selective state-space models excel at long-sequence modeling, but their capacity for language representation, in complex hierarchical reasoning -- remains underexplored. Most large language models rely on flat Euclidean embeddings, limiting their ability to capture latent hierarchies. To address this, we propose Hierarchical Mamba (HiM), integrating efficient Mamba2 with hyperbolic geometry to learn hierarchy-aware language embeddings for deeper linguistic understanding. Mamba2-processed sequences are projected to the Poincaré ball or Lorentzian manifold with "learnable" curvature, optimized with a hyperbolic loss. Our HiM model facilitates the capture of relational distances across varying hierarchical levels, enabling effective long-range reasoning for tasks like mixed-hop prediction and multi-hop inference in hierarchical classification. Experimental results show both HiM effectively capture hierarchical relationships across four linguistic and medical datasets, surpassing Euclidean baselines, with HiM-Poincaré providing fine-grained distinctions with higher h-norms, while HiM-Lorentz offers more stable, compact, and hierarchy-preserving embeddings.

Published in Transactions on Machine Learning Research (03/2026)
Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings
Sarang Patil
sp34630njit.edv
Department of Data Science
New Jersey Institute of Technology Nework, NJ 07102, USA
Ashish Parmanand Pandey
op2934 @njitt edu
Department of Data Science
New Jersey Institute of Technology
Nework, NJ 07102, USA
Toannis Koutis
skoutis @njftt. edu
Department of Computer Science New Jersey Institute of Technology
Nework, NJ 07102, USA
Mengjia Xu
mab unfornjit.om edu
Department of Data Science New Jersey Instätute of Technology Nework, NJ 07102, USA
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id- a3g13Fkact
Abstract
Selective state-space models excel at long-sequence modeling, but their capacity for language representation in complex hierarchical reasoning remains underexplored. Most large language models rely flal Fuclidean embeddings, limiting their ability to capture latent hierarchies. To address this, propose Hierarchical Mambo (HiM), integrating efficient Mamba2 with hyperbolic geometry to learn hicrarchy-aware language embeddings for deeper linguistic understanding. Mamnaa-processed sequences are projected onto the Poincaré ball or Lorentzian manifold with "learnable" curvature, optimized with a hyperbolic loss. Our HiM model facilitates the capture of relational distances across varying hierarchical levels, enabling effective long-range reasoning for tasks like mixed-hop prediction and multi- hop inference in hierarchical classification. Experimental results show both HiM variants effectively capture hierarchical relationships across four linguistic and medical datasets, surpassing Euclidean baselines, with HiM-Poincaré providing
distinctions with higher h-norms, while HiM-Lorentz offers more stable, compact, and sewarrhy-preservin smbeediinssfavorrin robustness!
1 Introduction
Large language models (LLMs), such as Transformers (Vaswani al. 2017 and BERT (Devlin el al. 2019), typically encode input sequences into flat Euclidean space. However, they struggle to capture the hicrarchical and tree-like structures inherent in natural language (Chomsky 1965). often leading to distor- tions at different levels of abstraction and specificity (Nickel & Kiela 2017 Ganea et al. 2018). Moreover, transformer- encoders face significant computational overhead due to the quadratic complexity of the attention mechanism (Vaswani et al. 2017). This limitation becomes particularly evident when dealing with
hicrarchical data (c.g., text ontologics, brain connectome [Ramirez et al. 2025
Baker et al. 2024 with
The source code is publicly available https://github. httper///ittuba..caaeerryyeyeet con/Berryyytte/HiM/

---

Published in Transactions on Machine Learning Research (03/2026)
exponentially expanding structure. State-space models, starting with the Structured State Space (S4) model (Gu et al. 2021). have shown exceptional scalability for long-sequence modeling. Mamba's 's selective mecha- nism (Gu & Dao 2023) dynamically prioritizes relevant information, achieving state-of-the-art performance in tasks with long-range rragg dependencies. Mamba2 refines the original Mamba model for long-range sequence tasks by introducing a duality between state-space computations and attention-like operations, enabling the model to function as either an SSM or a structured, "mask-free" form of attention (Dao & Gu 2024).
Recently, leveraging hyperbolic geometry as the latent representation space in machine learning models has shown great promise for learning meaningful hierarchical structures (Nickel & Kiela, 2018 Peng et al., 2021 Petrovski 2024). The Poincaré disk and Lorentz model are two prevalent represcntations of hyperbolic space. The Poincaré disk model is often lavored for its conceptual simplicity (bounded in a unit ball). However, the Lorentz model (with unbounded infinite space) offers a closed-form distance function, but requires careful handling of numerical operations dealing with space-like dimensions and time-like dimension using exponential mapping and logarithmic mapping (Peng et al., 2021). These numerical considcrations are critical because the improper handling of the lime-like coordinate in Lorentz models can lead to manifold violations, requiring specialized projection techniques (Fan et al. 2024 Liang et al. 2024b. Existing hyperbolic LLM architectures (He et al.) 2024b Peng ct al. 2021) often rely on Transformer blocks and apply a simple Poincaré disk model, leading to O(L²) complexity that becomes prohibitive for long sequences typical in deep hierarchies. A key challenge in implementing hyperbolic models is tuning the curvature to maintain numerical stability, particularly in Lorentz parameterization.
In this paper, we introduce hyperbolic Mamba with the Lorentz model, and compare it with its counter- parts Poincaré model and Euclidean model. To address the potential numerical instability in the Lorentz model MMishne et al., 2023), explicitly bound the embedding norms and employ curvature-constraing Maclaurin approximations for hyperbolic operations. HiM aims to achieve high-performance hierarchical classification by preserving relational hierarchies. It demonstrates scalability for processing long sequences without compromising on accuracy or computational efficiency. HiM's novelty lies in integrating a state- space model (Mamba2) with hyperbolic geometry, leveraging Mamba2's (l)) complexity for efficient se- quence modeling while preserving hyperbolic properties for hierarchical representation. Additionally, our HiM incorporates task-spocific hyperbolic losses that explicitly enforce parent-child distance constraints in hyperbolic space, enabling end-to-end hicrarchy learning without Euclidean biases and achieving significant F1 gains on multi-hop inference tasks. To support HiM's framework, we introduce Sentence 16M, a compact, Mamba2-basse large language model with 16 million parameters designed to generate high-quality sentence embeddings.
2 Related Works
Hyperbolic geometry has demonstratod strong potential in modeling hierarchical structures in both shal- low and deep neural networks. Foundational works, such as Poincaré embeddings (Nickel & Kiela| (017) and hyperbolic entailment cones (Ganea et al.) 2018)), showed their effectiveness in capturing hierarchical relationships in taxonomies with shallow neural networks. Morcover, hyperbolic manifolds have also been applied to encode hierarchies in graph-structured data (Liu et al. 2019 Chami et al. 2019). More recent efforts have extended hyperbolic representations to multimodal computer vision tasks, including visual and audio modalities (Yang et al. 2024c| Mandica et al. 2024), further demonstrating their strength in capturing both hicrarchical structure and uncertainty.
However, hyperbolic approaches language modeling remain limited. As an early approach to hyperbolic word embeddings, Dhingra et al. (2018) provided an important early step by reparameterizing Poincaré embeddings for GRU-based sequence modeling. This approach eliminated projection steps and supported both shallow and parametric encoders, but it was ultimately limited by its shallow representations, which restricted its expressive power and ability to capture long-range dependencies. More recent rescarch has exlended these concepts to Iransformers and their variants (He el al.] 2024b Chen el al. 2021 2024). These approaches enable effective prediction of subsumption relations and transitive inferences across hierarchy levels using hyperbolic embeddings, providing a principled framework for encoding syntactic dependencies through geodesic distances. However, Hyperbolic BERT exhibits higher computational cost than standard BERT due to the complexity of hyperbolic operations (Chnn et al. 2024). To improve efficiency, recent
2

---

Published in Transactions on Machine Learning Research (03/2026)
works have explored fine-tuning LLMs directly in hyperbolic space with the Low-Rank Adaptation (LoRA) technique (Hu et al.) 2022). For example, HoRA (Yang et al. 2024ah and HypLoRA (Yang et al. 2024b) apply LoRA to the hyperbolic manifold, allowing parameter-effiiient fine-tuning while capturing complex hicrarchics. These methods show strong gains-up up to 17.3% over Euclidean LoRA. However, these models usually assume a constant curvalure, which may not be optimal for all data, and can suffer from numerical instability due to the exponential and logarithmic mappings required to transition between Euclidean and hyperbolic spaces (López & Strube 2020).
Limitations in Current Approaches and Our Contribution: Despite significant progress, most exist- ing methods either exploit only partial hyperbolic representations (o.g., using adapters or static embeddings) or rely heavily on attention-based trchitectures that scale poorly with long sequences and deep hierarchies. For instance, Poincaré GloVe (Tifrea et al. 2018) is limited to word embeddings, failing to capture dynamic, context-dependent relationships, while Hyperbolic BERT (Chen et al.] 2024) and HiT (He et al.) 2024b) in- troduce significant computational overhead, especially for long sequences. Similarly, probing BERT's embed- dings in a Poincaré ball (Chen et. al.] 2021 to analyze hierarchical structures, but their diagnostie approach does not train new model for hierarchical reasoning tasks like HiM. Methods, such as HoRA (Yang et al, 2024a) and HypLoRA (Yang et al.) 2024b), only introduce hyperbolic geometry through adapter modules added post hoc to standard transformer backbones. These methods inherit the architectural inefficiencies of transformers but cannot fully encode hierarchy directly within the hyperbolic latent space. Building on the strengths and limitations discussed above, we propose Hierarchical Mamba (HiM) as a novel framework for long-range hierarchical reasoning. Our contributions can be summarized as follows:
Direct hyperbolic integration: Unlike HiT (He et al., 2024b), which fine-tunes pretrained Euclidean embeddings (all-MiniLM--G--y) in low-curvature hyperbolic space (K = -1/384), or HoRA AORAAHHyLLRRA (Yang et al. 2024a which adds hyperbolic LoRA adapters post-hoc to frozen Euclidean transformer backbones, HiM trains Mamba2 encoder from scratch directly in hyperbolic space with learnable curvature, operating at strong negative curvature (KC = -1.0) on both Poincaré and Lorentz manifolds without inheriting Euclidean biases.
SentenceMaamba-166
We develop the first Mamba2-based sentence encoder (SontencoMamba- 16M, 16M parameters) with specialized pooling strategies and hyperbolic projection interfaces lai- lored for sentence representation.
Stabilized hyperbolic operations: HiM addresses numerical instability in Lorentzian manifolds using curvature- curvature-bouneed bounded Maclaurin approximations for hyperbolic functions, ensuring robust train- ing for deep hierarchies and higher curvatures.
Hyperbolic losses: Building on HiT's clustering and centripetal losses (He et al. 2024b), we introduce curvaturo-aware dynamic margin scaling that adapts margins proportionally with new radius updated as per learned hyperbolic radius, and variance regularization in the centripetal loss to prevent norm collapse at high curvature.
3 Methodology
Hyperbolic geometry, characterized by negative curvature K. = -1/c, is well-suited for hierarchical data due to its exponential growth properties, modcled using the Poincaré ball or Lorentz model (Nickel & Kicla
2017 2018). Mamba2, a state-space model (SSM), offers efficient sequence modeling with linear complexity, using structured state-space duality to balance SSM and attention-like operations (Dao & Gu 2024
3.1
Hyperbolic Mamba (HiM)
The overall framework of HiM, including the integration of Mamba2 blocks and hyperbolic projections is shown in Figure 1 Firstly, the raw text is tokenized into sequence of tokens; these token IDs are mapped into embedded tokens resulting in token IDs of shape [B, L, DJ, where B is the batch size (B 256), L is the sequence length (L- = 128), and D is the embedding dimension (D = 384), and N is the state dimension (N = -9). These token embeddings are processed through four Mamba2 blocks. Given an input sequence X1.L E gLxD the discrete-time SSM at each timestep t is defined by:
h A,hi- B y= C,h,
(1)
3

---

Projection on Published in Transactions on Machine Learning Research (03/2026)
where h, E R is the hidden state at timestep t, and the selective mechanism makes the matrices A, B, C, input-dependent (see Appendix A.2 for detailed formulation of Mamba2). [AlanisToobato et al. (2016) focuses on efficient embedding of complex networks into hyperbolic space using the network Laplacian, achieving a computational complexity of ((N2) and enabling the analysis of large networks in seconds. This highlights the importance of computational elficiency in scaling hyperbolic models, principle that HiM extends by Mamba2 blocks (Equations 18 and 21 to achieve linear-time complexity OLL, making it particularly suited for long sequences and deep hierarchical language structures. In the Mamba2 blocks, the inputs x, having 384 dimensions are projected into the intermediate state I having 768 dimensions using linear transformation (D: :84 + I: 768).
ScatenceMamba-16M
Hierarchical Mamba (HiM)
Learnable scaling paramter
Sculing
Tanb + L2 Normalization
Normalize to unit sphere
Poincare Manifold
Lorentz Manifold
Mean Pouling
learnable curvature
Mamba2 Block
Mamha2 Block
Mambu2 Block
Lone Minimizzttinn
Hyperbollc Loss Functions
Hyperbollc Model
Train
OUTPUT PROJECTION
Hierarchical Dataset
(Original Distemsion
Mamba2 Block
with dy dymamic margins
SSM Xi B
SiLhSwih
[xx)
Mixed-Hop Prediction Task
Multi-Hop Inference
CoBv
INPUT PROJECTIONS
(latermeddittc Statc)
RMS Normalization
DIMENSIONS (R. D. (256, 138, 38)
Embeddings
Linear Projection
Tokenizer
Nouliacarity (activation, muttiplication)
Input Text
Figure 1: Overview of the Hierarchical Mamba (HiM) model, integrating Mamba2 blocks with hyperbolic projections to the Poincaré ball (via tangent- based mapping) and Lorentzian manifold (via cosine/sine- based mapping), enabling efficient and hicrarchy-awar language embeddings for long-range reasoning tasks.
The x component undergoes a convolution operation with a kernel size of 4. A SiLU activation function follows this operation for non-linearity. In the output projection, the intermediate state dimensions are projected back to their original embedding dimension I: 768 + D 384 for compatibility with downstream tasks. The Sentence.Mamba--166 model, central to HiM, randomly initialized with Kaiming normal weights instead of pretrained weights, enabling it to learn hierarchical structures directly from training data in hyperbolic space without any biases. The SentenceMamba- 16M model is trained using Triplet Contrastive Loss, which brings embeddings of positive pairs closer together while pushing apart embeddings of other sentences in the batch. This loss function has proven effective in prior works involving hierarchical embedding (Schroff et al.] 2015; He et al.) 2024b). After normalizing each embedding to unit length, we measure the pairwise cosine similarity as sim(i,j) = e, e,,, where each embedding e; and e, belong to e E {ci, e,. e. We then calculate the contrastive loss for the batch by constructing a similarity matrix from the similarity scores across nodes. The sentence embeddings are constrained using hyperbolic tangent
4

---

Published in Transactions on Machine Learning Research (03/2026)
activation followed by L2 normalization to ensure numerical stability:
normmalieetann(()))
(2)
where s the mean-pooled embedding from the Mamba2 blocks and normalize()) denotes 1.2 normalization. This operation reduces the sequence to single fixed-size vector, representing the pooled features of the entire imput sequence normalized to unit sphere.
To ensure numerical stability during hyperbolic projection, we apply norm scaling with learnable parameter y;
h ?
for Poincaré,
clamp(u, 8), for Lorentz.
This approach mitigates numerical overflow and enhances training stability. The interplay between the norm scaling parameter 7 and curvature K 1/cc is mathematically significant. When we scale embeddings by yy before projection, it effectively modulates the spread of embeddings in the hyperbolic space. Our model learns both the curvature parameter and scaling factor y that together determine the optimal geometry for representing hierarchical relationships. This dual learning approach provides flexibility in adapting the hyperbolic space lo the structural complexity of the data while maintaining numerical stability. Then the vector h is mapped to a point in hyperbolic space. The general form for Poincaré ball with radius r = vc (curvature K. = 1//c- 1/r2) is:
|b||
h
ep Ve. tanh
(4)
|b]l'
where ||b]] is the norm of embedding vector h. This scaling ensures the vector lies within the unit ball. This yields the final sentence embedding with values constrained between -1 and (indicating a positive negative parent), allowing us to project the embedding onto the Poincaré Ball manifold.
We also project it onto the Lorentzian manifold as it yields richer features in a more convenient. original hyperbolic space. The pooled embeddings are instead mapped to the Lorentzian manifold using:
cosh
cosh
[b
e =
(5)
sinh
Here, ||l]] is the norm distance of embedding vector h, vc is the radius of the hyperbolic space; IC < 0 ensures hyperbolic geometry. For the Lorentz mapping we let = |lbllve in Equation 5 The cosh, sinh functions are the Hyperbolic cosine and sine functions used to compute projections. The first term Ve-cosi.
[Ib]]
V
the Lorentz projection the time-like dimension. The remaining components v sinh
/e [b]] are the [Ibl]
space-like dimension. This step crucial for hyperbolic geometry it ensures the embeddings are bounded, enabling seamless projection onto the Lorentzian manifold.
While the Lorentz projection typically uses exact hyperbolic functions, we stabilize the training even more by approximating cosh and sinh via their Maclaurin (Taylor) expansions for |ж| < 10-3, By substituting truncated polynomial expansions, we limit overflow and hence solve the problem of exploding gradients.
cosh(2) =
+
sinh(z) = 2 +
(6)
2
4
31
5!
Then, the first (time-like) coordinate and the remaining (space-like) coordinates from Equation 5
become:
ve
1
+
er.
Ve
3 + -
h
To fully exploit the hyperbolic structure of our model, we employ an advanced hyperbolic loss function for the HiM model optimization, which is weighted combination of centripetal loss and clustering loss.
5

---

Published in Transactions on Machine Learning Research (03/2026)
These losses enhance the model's ability to effectively learn hicrarchical relationships by optimally posi- tioning and grouping the embeddings in strongly hierarchical structure within the hyperbolic manifold. Detailed equations for our hyperbolic loss are presented below, with the full calculation process provided in Appendix
Clustering Loss: This loss function clusters related entities and distances unrelated ones within the hy- perbolic manifold, promoting the grouping of similar entities while preserving hierarchical separation.
Leluster
Σ
max(de(e,,") - de(e,e e) a,,
(8)
(ejet,e-JED
This loss function ensures that parent entities are positioned closer to the origin of the hyperbolic manifold than their child counterparts. This reflects the natural expansion of hierarchies from the origin to the boundary of the manifold, while preventing norm collapse through variance regularization.
Centripetal Loss:
fentti Σ
max |ee |lelle + -,00- Avar Var( |ee (le)))
(e,e ie JED
Here, (e, et, e-) represent the hyperbolic embeddings of randomly selected anchor node, its positive parent node, and an unrelated negative node, respectively. |/elle de(e,0) measures the distance from the origin to the hyperbolic embedding in the Poincaré and Lorentzian manifold. de(e, measures the distance between hyperbolic embeddings of node e and its positive parent node et. de(e,e") measures the distance between node and a negative node e a and 3 denote margin parameters to enforce centripetal and clustering properties, respectively. Avar is the variance regularization weight set to 0.002 in our experiments.
The Hyperbolic Loss Chyperbolic is defined as the weighted sum of Centripetal Loss Ceentrs and Clustering Loss Celaster:
Liyperbolic WeeCeentri + We Wellsteter;
(10)
where Wee and Wet are weights that control the contribution of each loss component. To the base forms of clustering and centripetal losses, our implementation introduces curvature-aware dynamic margin scaling, where margins a and 3 scale proportionally with the learned radius r = 1/Ve V to maintain geometric 0.0051xr), and variance regularization Cvar -Val((()))le)) added the centripetal loss to prevent norm collapse under strong curvature by encouraging diversity in hyperbolic norms. This loss ensures that the model maintains the hicrarchical structure during training, with parent entities closer to the origin and related entities clustered together. The margins a and / in the clustering and centripetal losses (Equations 9 a S are implemented as dynamic parameters optimized during training to adaptively enforce the hierarchical constraints. The margins for the clustering and centripetal losses are adapted to the hyperbolic gcometry by scaling proportionally with the radius r, ensuring the loss functions remain geometrically consistent. across different curvatures. These scaling factors were determined through empirical validation maintain consistent separation properties the model adapts its curvature during training. The clustering margin is intentionally larger to enforce robust hierarchical separation between related and unrelated entities, while the centripetal margin is smaller to allow fine-grained positioning of parent nodes closer to the origin relative to their children, reflecting the natural expansion of hierarchies in hyperbolic space. In all hierarchical classification tasks, hard negatives were chosen to sharpen the model's discrimination (Schroff et al.] 2015). Rather than randomly sampling unrelated nodes, we select negative examples that are semantically close to the anchor (or positive) embedding space. This training strategy forces the model to learn more subtle hicrarchical distinctions, which is crucial for tasks such as "multi-hop inference". We observe that hard negatives lead to better generalization.
consistency as curvature adapts during training (a 0.255xr,
Following Chami et al. 2019), we optimize the learnable curvature parameter using the AdamW optimizer. This is justified because the curvalure parameter itself is a scalar Euclidean variable controlling the hy- perbolic manifold geometry, making AdamW both theoretically valid and empirically stable. To ensure numerical stability during training in hyperbolic space as the curvature adapts, we implement geometric stabilization technique that periodically projects the model parameters back onto the manifold. Specifically, every 100 optimization steps, this stabilization counteracts numerical drift that can occur during curvature

---

Published in Transactions on Machine Learning Research (03/2026)
optimization, preventing embeddings from violating the constraints of the hyperbolic geometry and ensuring all distance computations remain well-defined throughout training.
4 Experiments
Dataset We compare our proposed HiM models with their Euclidean counterparts, evaluated across four ontology datasets (1.c., DOID, FoodOn, WordNet, and SNOMED-CT) varying in scale and hierarchical com- plexity (1) DOID offers a structured representation of human discases through "is-a" relationships (Schriml et al. 2012). (2) FoodOn is detailed onlology that standardizes food-related terminology, covering ingredi- ents, dishes, and processes for nutritional classification and dietary research. It uses a hierarchical structure and borrows from existing ontologics like LanguaL (Doolcy ct al.) 2018). (3) WordNet is a well-known benchmark that organizes English nouns, verbs, and adjectives into synonym sets connected by hypernym- hyponym relationships (Miller 1995). (4) SNOMED Clinical Terms (SNOMED-CT) is a comprehensive clinical terminology system used in electronic health records (EHRs). It organizes concepts (e.g.: diagnoses, procedures, symptoms) into multiple hierarchies, linked by "is-a" and attribute relationships (Stearns et al., 2001 All datasets are derived from structured taxonomies and can be represented as directed acyelic graphs, where nodes denote entities and edges denote direct subsumption (i.c., parent-child) relations.
Implementation Details We use NVIDIA A100 GPUs with 80GB of memory each, distributed across a single compute node. Our model is implemented using the mamba-ssm library (Dao & Gul 2024). To define and operate over hyperbolic manifolds, we use GeoOpt (Kochurov et al. 2020), while DeepOnto (He et al, 2024a) is employed to process and manage hicrarchical structures in the ontology datascts. We leverage distributed data-parallel training with PyTorch's
DatrribbteDDtaPPaalllll
wrapper (Paszke 2019) Our
models were trained for ten epochs using the AdamW optimizer with a linear warm-up learning rate over the first 100 steps (target learning rate set to le- 4), and weight decay of le 1.-3. The linear warm-up is followed by a constant learning rate 1e-4. 4. The maximum gradient norm is clipped to 1.0. We employ a combination of hyperbolic clustering loss and hyperbolic centripetal loss during pretraining, with weights of 1.0 and 1.0, respectively. Our model incorporates several learnable parameters, such scaling factor Y (initialized to 0.01), curvature K. (initialized to -1.0). We implement dynamic margin parameters for losses o and B, which depend on the updated curvature. We use batch size of 256 per GPU, To regularize the model during training, a dropout rate of 0.2 is applied following each Mamba2 block. The detailed splits for mixed-hop prediction and multi-hop inference tasks, can be found from Table 3 in Appendix C
test
Evaluation Tasks and Metrics We evaluated our HiM models on two key tasks designed to assess its hierarchical reasoning capabilities in ontology completion and knowledge graph inference: (1) multi- hop inference, which involves predicting the existence of indirect relationships (e.g., "dog is a vertebrate") through transitive reasoning. (2) mized-hop prediction, which focuses on estimating hicrarchical distances between entities (e.g., 1-hop vs. 2-hop relations). Both tasks are formulated as classification problems based on hyperbolic distances. Detailed formulations are provided in Appendix [D] We use three metrics for evaluation: Fl score, Precision, and Recall. Among them, the F1 score serves as the primary metric, as it provides a balanced measure of precision and recall, which is critical for hierarchical reasoning tasks. Following prior work on these datasets (He et al. 2024b), we exclude Accuracy due to its vulnerability to class imbalance, where negative samples significantly outnumber positive ones. During training, models are optimized using entity triplets (anchor, positive, negative) under a contrastive learning framework; however, evaluation is performed on entity pairs to directly assess subsumption prediction performance.
5 Results
We compare our proposed HiM models- HiM-Poincaré and HiM-Lorentz Lana-ooonntz-againstt three Euclidean baselines on four hicrarchical datasets for two main downstream tasks: mixed-hop prediction and multi-hop inference.
The Euclidean baselines include: (1) Pretrained SentenceMamba-116M trained on the SNLI dataset (Bow-
man al. 2015); (2) Finetuned (with pretrained weights), which continues training from the pretrained
Datasets are availahle from http:///zanddooorrrrccctt//11111 and see Table 3 in Appendix for details

---

Published in Transactions on Machine Learning Research (03/2026)
checkpoint on hicrarchical datasets; and (3) Trained from scratch, which uses random initialization (Kaim- ing normal for weights, zero for biases) and is trained directly on hierarchical datasets in Euclidean space. Our HiM models share the SentenceMamba--In 16M backbone (=s 16M parameters), but incorporate learnable curvature and are trained from scratch without pretrained weights in hyperbolic space using both Poincaré and Lorentzian manifolds.
5.1 Comparison between HiM models and their Euclidean baselines
We present a comprehensive comparison of HiM-PPincaré, HiM-Lorentz, and their Euclidean baselines, in- cluding the pretrained SentenceMamba-1GMM fine-tuned SentenceMamma-16M (with pretrained weights) and trained SentenceMamba-1ie els were trained with learnable curvature parameter K - -1/c. A deeper curvature (smaller radius r => smaller с => larger/deen curvature X)) allows to exploit the hierarchical structure of the hyperbolic manifold much better, as the hyperbolic embeddings are confined in the conical manifold compact within a smaller radius. The average d-hyperbolicity (Gromov 1987) for cach dataset measures the tree-likeness of the graph by calculating the maximum deviation from the four-point condition. Values closer to 0 indicate a more hierarchical structure (Adcock et al. 2013), making these datasets well-suited for hyperbolic embed- dings. The corresponding 5-hyperbolicity scores for the four datasets are reported in Table [ reflecting a descending order of hierarchy complexity: DOID + SNOMED-CT + WordNet - FoodOn. The experimen- tal results illustrate that HiM-Lorentz model achieves more robust and stable performance (with extremely small variance) in terms of the FI, precision, and recall values for both mixed-hop prediction and multi-hov hop inference tasks across four datasets. Moreover, HiM-Lorentz outperformed the HiM-Poincaré variant on the multi-hop inference task for both the WordNet and SNOMED-CT datasets, both of which are relatively large datasets and exhibit deeper hierarchies characterized by small d-hyperbolicity. However, in the case of FoodOn which also has higher hyperbolicity the Poincaré-based model achieved better performance. To verify that the weak zero-shot performance of the pretrained Euclidean baseline is not merely an effect of small parameter scale, we additionally report inference-only baselines for substantially larger pretrained encoders in Appendix H.1
(trained from scratch on hierarchical datasets) in Table [ Both HiM mod-
To validate our design choices, we conducted comprehensive ablation studies on loss components, pooling strategies, model depth, and negative sampling strategies which are presented in Appendix
5.2 Comparison with Hyperbolic Transformer Baseline
To further evaluate the effectiveness of HiM, we compare it with hyperbolic transformer baseline. The original HiT (He et al.) 2024b) fine-tunes a pretrained model (all-MiniLM--6-v2 22M params) with low curvature (K = 1/384, nearly Euclidcan). To isolate architectural effects and ensure fair comparison under identical training conditions, we implement HiT using the MiniLM-L6-v2 transformer architecture trained from scratch (randomly initialized). We conducted ablations across two different curvatures (K- = -1.0 and -1/d where embedding dimension d 384) and model sizes (16M and 32M paramcters) on both Poincaré and Lorentz manifolds. The results in Table 2 show that HiM consistently outperforms the transformer- based HiT model across both manifolds and most experimental configurations. See Appendix J for detailed model size configurations.
Notably, performance degrades significantly under low curvature settings = -1/d) particularly in the Lorentz manifold, where HiT* shows substantial performance drops. This suggests that stronger hyperbolic curvature (K -1.0) is essential for effective hierarchical modeling. Under our fixed curvature, Poincaré's bounded nature enables more stable norm dispersion and discriminative gradient flow, particularly when combined with variance regularization in our centripetal loss. In contrast, Lorentz embeddings tend to col- lapse toward the hyperboloid's 's shell where time-like distances flatten and norm-based separation weakons. To provide broader context. for our results, we include additional analysis with comparing our model to GPT-40 on zero-shot prompting as well as candidate ranking in Appendix(H.22 and [..3] and also study HiM's computational efficiency in terms of sequence length Appendix HH.4] The success of hyperbolic emboddings in these tasks suggests broader potential for improving hierarchical reasoning in future generative models, where understanding semantic relationships could enhance factual consistency and knowledge grounding.
8

---

Published in Transactions on Machine Learning Research (03/2026)
Table I: Performance comparison of Pretrained, Finetuned (with pretrained weights), Trained from scratch across Euclidean manifold, and HiM models across hyperbolic manifolds on various datasets (with varying average 6-hyperbolicity). Pretrained SentenceMammba-1kk is trained on SNLI; Finetuned (with pretrained weights) continues training from the pretrained checkpoint; Trained from scratch uses random initialization (Kaiming normal for weights, zero for biases). HiM models use learnable curvature for hyperbolic projections. The mean and standard deviation of F1, Precision and Recall scores were computed over five independent runs for each setting. (See details in Appendix F)
Euclidean (K = 0)
Hyperbolic (KC < 0, learnable)
Metric
Trained (pretrained weights) from scratch
Finetuned
Pretrained
HiM-Poincaré HiM-Lorentz
Mixed-hop
Prediction (DOID) Average 5-hyperbolicity = 0.0190
F1
0.135 + 0.022
0.744 + 0.029
0.436 = 0.043
0.795 + 0.019
0.821 + 0.003
Precision
0.087 1 0.003
0.815 + 0.013
0.776 1 0.016
0.812 + 0.020
0.822 1 0.004
Recall
0.390 + 0.207
0.685 0.041
0.305 + 0.040
0.780 1 0.026
0.820 1 0.007
Mixed-hoo hop Prediction (FoodOn)
Average derypeebol = 0.1852
F1
0.125 1 0.046
0.634 + 0.011
0.550 + 0.017
0.836 1:2 0.031
0.827 1 0.002
Precision
0.090 1 0.009
0.760 0.008
0.688 0.008
0.841 0.024
0.852
0.007
Recall
0.330 + 0.232
0.544 + 0.019
0.459 + 0.023
0.831 + 0.033
0.803 + 0.002
Mixed- hop Prediction (WordNet)
Average d-hyperbolicity
0.1438
F1
0.135 + 0.044
0.580 + 0.015
0.615 + 0.009
0.824 + 0.024
0.823 + 0.003
Precision
0.086 + 0.014
0.702 + 0.009
0.755 + 0.018
0.853 + 0.023
0.828 + 0.006
Recall
0.430 + 0.238
0.494 +: 0.018
0.519 + 0.006
0.798 + 0.029
0.815 + 0.004
Mixed- hop Prediction (SNOMED-CT)
: Average d-hyperbolicity = 0.0255
F1
0.129 1 0.017
0.790 0.012
0.672 i 0.009
0.886 1 0.027
0.890
0.004
Precision
0.084 + 0.001
0.950 + 0.002
0.886 + 0.003
0.894 + 0.024
0.901 + 0.006
Recall
0.375 + 0.207
0.673 0.019
0.541 + 0.012
0.877 + 0.032
0.880
0.005
Multi-hop Inference (WordNet)
Average 8- d-hyperbolicins = 0.1431
F1
0.134 + 0.045
0.603
0.009
0.648 0.012
0.865 + 0.026
0.872 + 0.004
Precision
0.086 + 0.016
0.739 + 0.009
0.768 + 0.012
0.867 + 0.023
0.871 + 0.007
Recall
0.431 + 0.240
0.509 0.011
0.560 = 0.013
0.863 + 0.031
0.872
0.005
Multi-hop Inference (SNOMED-CT)
Average 8- d-hypeebolicity
0.0254
F1
0.128 + 0.016
0.754 + 0.025
0.630 = 0.010
0.919 + 0.028
0.920 + 0.003
Precision
0.083 + 0.001
0.969 0.001
0.902 = 0.002
0.917 + 0.024
0.919
0.008
Recall
0.369 + 0.205
0.620 + 0.031
0.483 + 0.011
0.921 + 0.034
0.920 + 0.008
Our results demonstrate that specialized embedding architectures can achieve strong performance on hi- erarchical reasoning tasks with significantly fewer parameters (16M) compared to general-purpose LLMs, offering a complementary approach to scaling.
5.3 Visualization of Hyperbolic Embeddings
To demonstrate how HiM captures hierarchical structure in the learned embeddings, we visualize the hy- perbolic representations on a representative semantic hierarchy with WordNet. The hyperbolic embeddings learned by HiM is presented in Figure 2 which illustrates representative hierarchical path, sport skating skateboarding. The HiM-trained embeddings exhibit tight clustering of semantically related nodes (c.g., skating and sport) in hyperbolic space, indicating enhanced semantic aligmment. Morcover, the embeddings clearly capture the hierarchical structure, as higher-level concepts such as sport are po- sitioned closer to the origin, while more specific concepts like skateboarding are embedded farther from the origin a compact and organized manner. More details and geometric analysis of these hyperbolic embeddings, including quantitative metries comparing h-norms, geodesic distances, and hierarchical depth correlations across both Poincaré and Lorentz manifolds, is provided in Appendix E. Additional examples
9

---

Published in Transactions on Machine Learning Research (03/2026)
Table 2: Fl scores comparing HiM models with HiT across different parameter scales and curvature set- tings on mixed-hop prediction tasks. HiT* (Hyperbolic Transformer with random initialization) uses identical hyperbolic projections, loss functions, and manifolds as HiM, differing only in the use of Transformer archi- tecture instead of Mamba. Mean + standard deviation computed over multiple independent runs. Bold red values indicate the best performance within each row, bold blue indicates second-best. Results demonstrate consistent advantages of the Mamba architecture over Transformers in hyperbolic settings.
Curvature (KC)
Poincaré Manifold
Lorentz Manifold
# Param.
HiM
HiT
HiM
HiT
Werd Dataset
16M
-1.0
0.857
0.007
0.846 0.006
0.851
0.004
0.838 0.007
16M
-1/d
0.791 + 0.012
0.840 + 0.008
0.783 + 0.015
0.523 + 0.025
32M
0.851
0.006
0.836 0.007
0.839 0.005
0.829 = 0.006
32M
-1/d
0.771 + 0.011
0.809 + 0.014
0.817 + 0.014
0.465 + 0.030
DOID Dataset
16M
-1.0
0.904
0.008
0.837 0.007
0.889
0.005
0.844 0.005
16M
-1/d
0.861
+ 0.010
0.825 + 0.009
0.706 = 0.018
0.518 + 0.028
32M
-1.0
0.881 0.007
0.831 0.008
0.871
0.006
0.832 0.007
32M
-1/d
0.837 + 0.009
0.835 + 0.008
0.709 + 0.016
0.527 + 0.026
from FoodOn dataset and out-of-distribution inference are provided in Appendix UMAP visualization methodology.
along with the
speed skating
speed skating
0.6
akntehoarding
rrowing
skating
outdoor
0.5
skating
rowing
skateboarding
0.4
sport
Origin
0.3
outdoor sport
sport
0.1
* Origin
0.0
-0.1
0.00
0.1
Selected Node
Negative Parents Children
Origin (Root)
Selected Node -> Negative Parents
Positive Parent
Selected Node
Parent
Selected Node -> Children
Figure 2: Visualization of HiM's embeddings trained on the WordNet dataset in the Poincaré ball manifold. Left: The full hyperbolic space, illustrating the distribution of entities with parent nodes positioned closer the origin and child nodes exlending loward the boundary, reflecting the exponential expansion of hyperbolic geometry. Right: A zoomed-in view emphasizing the hierarchical structure, such as path sport skating -> skateboarding, Dots represent the entities, with colors indicating hierarchical relationships. For a selected node "skating" denoted by the green dot, the blue node denotes its parent nodes (e.g., sport), and red indicates its hard negatives, such as abblins/coousin (e.g., rowing). Yellow nodes (e.g, skateboarding, speed skating) indicate children nodes of the selected node (skating), meaning the grandchildren nodes of the blue node (sport).
10

---

Published in Transactions on Machine Learning Research (03/2026)
6 Conclusion
By integrating hyperbolic embeddings in the model, HiM successfully captures hierarchical relationships in complex long-range datasets, providing scalable and effective approach for handling long-range dependen- cies. HiM's unique approach, especially in hyperbolic embedding and its SSM incorporation, showcases its strengths hierarchical long-range classification, marking it as a significant advancement in hierarchical learning models. Additionally, we find HiM to be more robust in training, primarily due the Mamba2 blocks' efficient memory usage and the synergy between hyperbolic geomelry and SSM-based sequence mod- eling.
Lorentz embeddings can provide more natural fit for large-scale datascts with intricate hicrarchical patterns compared to other geometries, potentialy ially enhancing performance and interpretabilily. By demonstrating how Lorentzian manifold can be effectively deployed for hyperbolic sentence representations, this paper aims to motivate further exploration of hyperbolic geometry in diverse real-world applications, ultimately broadening the scope and impact of geometry-awwan neural architectures. Investigating HiM's potential for efficient. temporal dependency modeling in intricate long-range hierarchical classification tasks holds significant promise and study its practical applications. Future work could explore integrating CLIP-style pretraining to incorporate multimodal data (e.g., text and images) for tasks like visual question answering, or potentially building on work such as Cobra (Zhao et al. 2025), which demonstrates the potential of extending Mamba models for efficient multimodal language modeling.
Acknowledgments
We would like to acknowledge the funding support from the DOE SEA-CROGS project (DE-SC0023191) and the AFOSR project (FA9550-24-1-02311, We also acknowledge the computing resources provided by the High Performance Computing (HPC) facility NJIT.
References
Aaron B. Adcock, Blair D. Sullivan, and Michael W. Mahoncy. Troe-like structure in large social and information networks. In 2013 IEEE 13th International Conference on Dala Mining, pp. 1 1-10, 2013.
Gregorio Alanis-Lobato, Pablo Mier, and Miguel A Andrade-Navarro. Efficient embedding of complex net- works to hyperbolic space via their laplacian. Scientific reports, 6(1):30108, 2016.
Cole Baker, Isabel Suárez-Méndez. Grace Smith, Flisabeth B Marsh, Michael Funke, John C Mosher, Fer- nando Maestú, Mengjia Xu, and Dimitrios Pantazis. Hyperbolic graph embedding of meg brain networks to study brain altcrations in individuals with subjective cognitive decline. IEEE Journal of Biomedical and Health Informalics, 2024.
Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. The snli corpus. 2015.
Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovee. Hyperbolic graph convolutional neural net- works. Advances in neural information processing systems, 32, 2019.
Boli Chen, Yao Fu, Guangwei Xu, Pengiun Xie, Chuanqi Tan, Mosha Chen, and Liping Jing. Probing bert hyperbolic spaces. In International Conference on Learning Representations, 2021.
Weize Chen, Xu Han, Yankai Lin, Kaichen He, Ruobing Xie, Jie Zhou, Zhiyuan Liu, and Maosong Sun. Hyperbolic pro-trained language model. IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024.
Noam Chomsky. Aspects of the theory of syntax. 1965.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through struc- tured state space duality. arXiv preprint axX::2445.21060, 2024.
11

---

Published in Transactions on Machine Learning Research (03/2026)
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171-4186, 2019.
Bhuwan Dhingra, Christopher J Shallue, Mohammad Norouzi, Andrew M Dai, and George E Dahl. Embed- ding text in hyperbolic spaces. arXiv preprint arXiv: 04306,,43, 2018.
Damion M Dooley, Emma J Grifithh, Gurinder S Gosal, Pier L Buttigieg, Robert Hochndorf, Matthew C Lange, Lynn M Schriml, Fiona SI. Brinkman, and William WL. Hsiao. Foodon: a harmonized food ontology to increase global food traccability, quality control and data integration. npj Science of Food, 2(1):23, 2018.
Xiran Fan, Minghua Xu, Huiyuan Chen, Yuzhong Chen, Mahashweta Das, and Hao Yang. Enhancing hyperbolic knowledge graph embeddings via lorentz transformations. In Findings of the Association for Computational Linguistics ACL 2024, pp. 4575-4589, 2024.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierar- chical embeddings. In International conference on machine learning, pp. 1646 1655. PMLR, 2018
Mikhael Gromov. Hyperbolic groups. In Essays in group theory, pp. 75-263, Springer, 1987.
Albert Gu and Tri Dao. Mamba: Lincar-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.000 00752, 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. Xie preprint orXiv:211 100996 2021.
Yuan He, Jiaoyan Chen, Hang Dong, Tan Horrocks, Carlo Allocca, Taehun Kim, and Brahmananda Sapkota. Deeponto: A python package for ontology engineering with deep learning. Semantic Web, 15(5):1991 2004, 2024a.
Yuan He, Moy Yuan, Jiaoyan Chen, and Ian Horrocks. Language models as hicrarchy encoders. Advances in Neural Information Processing Systems, 1414660-14711 2024b.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 12):3, 2022.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science 8 engineering, ((03)990-99 95, 2007.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Gcoopt: Ricmannian optimization in pytorch. arXiv preprint arXie:2005.022 02819, 2020.
Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vahdat, and Marián Boguná. Hyperbolic geometry of complex networks. Physical Review E Statistical, Nonlinear, and Soft Matter Physics, 82 (3):036106, 2010.
Qiuyu Liang, Weihua Wang, Feilong Bao, and Guanglai Gao. L 2ge: Lorentzian linear graph convolutional networks for node classification. Xix preprint 0t::2403306062 2024.
Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. Advances in neural information processing systems, 32, 2019.
Federico López and Michael Strube. A fully hyperbolic neural model for hierarchical multi-class classi- fication. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, November 2020.
Paolo Mandica, Luca Franco, Konstantinos Kallidromitis, Suzanne Petryk, and Fabio Galasso. Hyperbolic learning with multimodal large language models. arXiu preprint (xXi:2408.05009 2024.
12

---

Published in Transactions on Machine Learning Research (03/2026)
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: S.t::102.03443. 03426, 2018.
George Miller. Wordnet: a lexical database for english communications of the acm 38 (11) 3941. Niemela,
I, 1995.
Gal Mishne, Zhengehao Wan, Yusu Wang, and Sheng Yang. The numerical stability of hyperbolic represen- tation learning. In Proceedings of the 40th International Conference on Machine Learning, Proceedings of Machine Learning Research, pp. 24925-24949. PMLR, 2023.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. Ad- vances in neural information processing systems, 30, 2017.
Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry, In International conference on machine learning, pp. 3779-30 3788.. PMLR, 2018.
A Pasake. Pytorch: An imperative style, high-performance deep learning library. arXiv preprinl arXiv:1912.01 703,, 2019.
Wei Peng, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey, IEEE Transactions on pottern analysis and machine intelligence, 44(12):100231100 2021.
Igor Petrovski. Hyperbolic sentence representations for solving textual entailment. ar preprinl arXiv:2406.154 2024.
Hugo Ramirez, Davide Tabarelli, Arianna Brancaccio, Paolo Belardinelli, Elisabeth B Marsh, Michael Funke, John C Mosher, Fernando Maestu, Mengjia Xu, and Dimitrios Pantazis. Fully hyperbolic neural networks: A novel approach to studying aging trajectories. IEEE Journal of Biomedical and Health Informatics, 2025.
Lynn Marie Schriml, Cesar Arze, Suvarna Nadendla, Yu-Wei Wayne Chang, Mark Mazaitis, Victor Felix, Gang Feng, and Warren Alden Kibbe. Discase ontology: backbone for discase semantic integration. Nucleic acids research, (O(D1):D940-D94)) 2012.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pallern recognilion, pp. 815-823, 2015.
Michael Q Stearns, Colin Price, Kent A Spackman, and Amy Y Wang. Snomed clinical terms: overview of the development process and project status. In Proceedings of the AMA Symposium, pp. 662, 2001.
Alexandru Tifrea, Gary Bécigneul, and Octavian-Engen Ganea. Poincaré glove: Hyperbolic word embed- dings. Xix preprint arXiv:1816 1810006666 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Menglin Yang, Aosong Feng, Bo Xiong, Jiahong Liu, Irwin King, and Rex Ying. Enhancing I1m complex reasoning capability through hyperbolic geometry. In ICML 2024 Workshop on LLMs and Cognition, 2024a.
Menglin Yang, Aosong Feng, Bo Xiong, Jihong Liu, Irwin King, and Rex Ying. Hyperbolic fine-tuning for large language models. ar preprint arXiv:2410.0400 2024b.
Zhe Yang, Wenrui Li, and Guanghui Cheng. SHMamba: Structured hyperbolic state space model for audio- visual question answering. arXiv preprint arXic:2406.0983 09833,, 2024c.
Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to multi-modal large language model for efficient inference. In Proceedings of the AAAT Conference on Artificial Intelligence, volume 39, pp. 10421-10429, 2025.
13

---

Published in Transactions on Machine Learning Research (03/2026)
A Preliminaries
A.1 Hyperbolic Geometry
In hyperbolic geometry, the notion of curvature is commonly represented by negative curvature A = ot where e > 0. Equivalently, one may define a 'radius' r = Vč. A smaller radius r corresponds to a larger and higher negative curvature (K), effectively making the hyperbolic manifold more curved, granting more flexibility to the hicrarchical depth. Conversely, letting r + oo approaches flat (Euclidcan) space since K. -> Basically, r controls the rate of exponential expansion on the hyperbolic manifold. This choice impacts how data at varying levels of abstraction distributes on the manifold and is crucial for tasks requiring fine-grained exponential separation of hierarchical data. By leveraging hyperbolic space, language models can encode features more naturally in a hierarchical branching, keeping more generalized features located near the root of the hierarchy tree, Lee, near the origin of the hyperbolic manifold, and the more specific or complex entities are branched further from the origin towards the margin.
A popular way to realize hyperbolic geometry in an n-dimensional setting is via the Poincaré ball model. Here, the underlying space the open Poincaré unit ball:
B fx E E |x|| < Vcl.
(11)
equipped with a metric tensor that expands distances near the boundary. Concretely, each point x in the ball maintains local geometry that grows increasingly "stretched" as ||x||| approaches radius vc. Formally, the distance between two points x and y in a Poincaré ball is computed by
xx - y dp(x,y) Veevvrereee arcosh ( [1+2 (1 |x| -
(12)
This representation has gained attention in machine learning due to relatively simple reparameteriixatoo for gradient- basde updates, thus facilitating the embedding of hierarchically structured data (Nickel & Kiela 2017).
While the Poincaré ball confines all points within the unit sphere (Equation 11 the Lorentzian mani- fold leverages an (n + 1)ddimension Minkowski space (Equation 13), enabling different perspective on hyperbolic geometry. Specifically, points reside on the "hyperboloid" " defined by:
_ {x E Rn+1 (x,x)M c, 20 > 00,
(13)
where t)s denotes the Minkowski inner product, typically -ool0 + Σn Zzyh-- The hyperbolic distance between two points x and y then appears in the form:
dc(x,y) vc- arcosh (- (x,
(14)
Compared to the Poincaré ball, this approach can sidestep certain numerical instabilities near the boundary because vectors are not constrained to lie within a finite radius. Moreover, Lorentz- based formulations often allow more direct computation of geodesics and exponential maps, making them advantagcous for large-scale hyperbolic embeddings (Nickel & Kiela, 2018). Krioukov et al. (2010) provides a theoretical foundation for the hyperbolic goometry of complex networks, showing that many real-world networks naturally embed into hyperbolic spaces, supporting our choice of the Poincaré and Lorentzian models for hierarchical language embeddings.
We can see how hicrarchics are represented differently in Euclidean space, the Poincaré manifold, and the Lorentzian manifold as illustrated in Figure A. On the left, the hierarchical structure is arranged as a standard tree. While the relationships are maintained, Euclidean space does not naturally encode hicrarchical distances in 2D. In F. the upper-right diagram shows the hicrarchy embedded into the Poincaré ball (the root root/origh /rigin being at the center). The more generalized parent nodes are positioned near origin, and descendant nodes extend outward near the margin. This representation captures the exponential
14

---

Published in Transactions on Machine Learning Research (03/2026)
crab pakes
appeizens
food
dessents
food
cheese calees
appetizers
Hlerarchy within Poincare manifold
desserts
crab
Euclidean Space
cakes
desserts
appetizerr
Food
root
Hierurchy within I. Lorentz manifold
Figure A.1: Illusraaionn of word embeddings in Euclidean (Left) vs. Hyperbolic Spaces for hicrarchical representation in Poincaré (Top right) and Lorentzian Manifolds (Bottom right).
growth of hierarchical structures, where sibling nodes placed far apart in terms of geodesic distance. The lower-right diagram visualizes the same hicrarchy embedded in the Lorentz hyperboloid. The Lorentzian manifold in R=+1 consists of n spatial dimensions and one time-like dimension (zn). The origin is at the center-bottom of the Hyperboloid, and nodes are arranged along the hyperboloid surface. More generalized parent nodes are positioned near the bottom, and the descendants keep extending upward on the cone of Lorentz. Unlike the Poincaré model, which confines embeddings within a finite ball, the Lorentz model represents hierarchies in an unbounded space, making particularly suitable for representing deeply nested hierarchies.
A.2 Mamba2
Mamba2 a state-space model (SSM) introduced by Tri Dao and Albert Gu that refines the original Mamba architecture with improved performance and simplified design (Dao & Gu 2024). Mamba2 builds upon the original Mamba architecture by introducing the State Space Duality (SSD) framework, which establishes theoretical connections between State Space Models (SSMs) and attention mechanisms. Mamba2 achieves 2-8x faster processing while maintaining competitive performance compared to Transformers for language modeling tasks. In order to formulate the overall computation for a single Mamba2 block, let XI:L = [x,,...,XI] be the token (or embedding) sequence for a given input. single Mamba2 block transforms XI.7 into an output sequence YI.L. Each input token embedding x, E RD first normalized via RMSNorm. RMS normalization ensures that the norm of the embeddings remains stable across different inputs, preventing extreme values from causing instability during training.
RMSNorm(x;).
(15)
Given weights W and bias b, project the input into higher-dimennsional space to obtain u.
Win X, + bind
(16)
yielding E RR. is split into two components, x and zi-
x
u1 =
(17)
z
15

---

Published in Transactions on Machine Learning Research (03/2026)
The component z is reserved for the gating mechanism used later in the process. For each time step t, hidden states h, evolve under:
= A + B Y: = C,h.
(18)
where A, is the state transition matrix, B, is the input projection, and C2 the output projection.
For each timestep the state matrices have dimensions A, NRN B, E RxXI, and C, E R where N is the state dimension and I is the intermediate dimension. To enable efficient O(L) complexity, Mamba2 uses structured versions of A, B, C' (e.g., dlagonal-plus-loee plassppuuus---aa--eee forms) and fast transforms (such FFT- based convolution). Mamba2 incorporates a gating mechanism to blend the output of the state-space layer back with the original input, thus forming a residual block:
= og)) Z + XT. = W X1 bg.
(19)
where a()) is typically SILU activation that follows this operation for non-linearity
(x) = I sigmoid(()) sigmoid(s)
(20)
Ite=*
This gating helps regulate the flow of information and provides additional stability during training.
Mamba2 establishes a theoretical framework connecting SSMs and attention mechanisms through "state space duality", allowing the model function either as an SSM or as a structured form of attention via the below formulation:
L = ISS(a) and M Lo (CBB).
(21)
where C is the semiseparable matrix structure derived from the state transition dynamics, denotes the Hadamard (clement-wise) product. Following Dao & Gu (2024), we denote a 1-semiseparable (1-S5) matrix generated by scalar sequence a = (oo, aL-1) 1) as ISS(a). This a lower-triangular matrix L E L.X. where:
L = Cis
II kmj+1
for i>j
(22)
This structure enables efficient O(L) computation via parallel sean operations, which is fundamental to Mamba2's hardware-efficien implementation. This paper introduces a Mamba2- LLM known as SentenceMamba- 16M, a lightweight model with 16M parameters, suitable for resource-efficint and high- quality sentence embedding generation. As we can see in Figure we incorporate four Mamba2 blocks our SentenceMMama--16M for efficient stale-space modeling.
B Hyperbolic Loss Calculations
Depending on our dataset, we can either apply Triplet Loss when we have a triplet relationship in our data and need enforce relative distance constraints apply Contrastive Loss when have pairwise relationships in our data and need to classify pairs similar dissimilar. Based on either triplet relationship data pairwise relationship data, we perform triplet. loss or contrastive loss calculation. Then, we calculate the weighted loss of centripetal loss and clustering loss as the hyperbolic loss. Hyperbolic loss computation framework is illustrated Figure B.2, which shows how these two loss components are weighted and combinod to create the final training objective. Loss minimization involves clustering related entilies and distancing unrelated ones (clustering loss) and tightening parent entities closer to the hyperbolic manifold's origin than their child counterparts (centripetal loss), giving it hierarchical structure.
C Dataset Statistics
To provide a comprehensive overview of the datasets used in our experiments, we detail the size, number of entities (nodes), and /test splits for each dataset in Table These datasets are represented
16

---

Published in Transactions on Machine Learning Research (03/2026)
Clustering Loss
Centripetal Loss
Clustering Triplet Loss
Clustering Contrustive Loss
Centripeta[Triphe Loss
Contripetal Centrustive
If
If
d(child. dichild, non-purent)
dichild..uarent) < positive margin and dichild.oon-parent) negative margin
d'child.origin) deparent.origin)
d(child.origin) d(purent.origin)
(idkganchoopppottivv) d(anchor.eggatvve) + margin)
ReL. U'dlanccoorpeeiiiie) pusitive margin)
ReL.U(margin d(parent.urigin) d(child,origin)
labels Rel.Umargiin
(negutive margin- dianchor.nepative)
d(parent.origin) d(child.origin)
OR
Hyperbolic
OR
Clustering Loss
Centripetal
APPEND
hhpportolic
10 Was Centerr + Welceraatr
Weighted Loss
Hyperbolic Loss
1.oss Minimization
Illerarchical Embeddings
Figure B.2: Calculation of hyperbolic loss from clustering loss and centripetal loss.
as directed acyclic graphs (DAGs), where nodes denote entities (e.g., diseases in DOID, synsets in WordNet) and edges denote direct subsumption relations (is-a). Splits are created by sampling direct (E) and indirect (multi-hopp T) subsumptions, ensuring coverage of both mixed-hop prediction and multi-hop inference tasks.
Table 3: Statistics of hierarchical ontology datasets
Dataset
Entities
DDirectSub
#IndirectSub
Splits (Train (aainn/Va/TTst)
DOID
11,157
11,180
45,383
Mixed-hop: 111K / /3IK /IK
FoodOn
30,963
36,486
438,266
Mixod-hop:: 361K / 261K 261K
Multi-hop: 834K 323K 323K
WordNet (Noun)
74,401
76,850
587,658
Mixed-hop: 751K 365K 365K
Multi-hop:
4,160K /1,758K /1,75KK
SNOMED-CT
364,352
420,193
2,775,696
Mixed-hop: 1,160K/1,758K/1,,77
D Task Formulations
D.1 Multi-Hop inference
Let G (V, denote hierarchical graph, where V represents entities (nodes) and E denotes direct subsumption edges (e.g., parent-child relationships). The transitive closure T of E encompasses all indirect (multi- hop) subsumptions, such as relationships spanning two or more hops (e.g., coanddparentto-graaddeee The multi-hop inference task trains a model fMI on the direct edges E' and evaluates its ability to predict the existence of unscen indirect relations
fMI (V,E) +
(23)
where í approximates T. This binary classification task tests transitive reasoning, such as inforring "dog is a vertebrate" from "dog mammal" and "mammal is a vertebrate." The model computes hyperbolic distances between entity embeddings, with threshold determining relationship existence.
17

---

Published in Transactions on Machine Learning Research (03/2026)
Evaluation uses precision, recall, and FIMI scores over test pairs sampled from T UN,, where N represents negative pairs (non-subsumpttions). Test sets Stest are constructed as:
Stest = ((U,, U;) U5) ETUNINN = 10/7]):
(24)
with negatives, including hard cases like sibling entities (sharing parent but not directly or transitively linked). This assesses fine-grained discrimination across both upward (child-to-ancestor)) and downward (parenl-to-descenddan)) directions, leveraging HiM's hyperbolic embeddings.
D.2 Mixed-Hop prediction
The mixed-hop prediction task evaluates the model's ability to predict the exact number of hops between entities, encompassing both direct (1-hop) and multi-hop (2+ hops) subsumptions. Given a training subset E, the model SMP is trained and lested on:
fMP : (V, - R
(25)
where R= = ET includes all held-out direct and transitive subsumptions, and R approximates R. Unlike multi-hop inference, which focuses on existence, mixed-hop prediction quantifies hierarchical distance (c.g.-
1, 2, or 3 hops), such distinguishing "dog to mamunal" (1 hop) from "dog to vertebrate" (2 hops). This is framed as multi-class classification task, mapping hyperbolic distances to discrete hop counts.
Evaluation employs FIMP scores over test sets:
Stest = {(v,, e5) (Us 15) ( = 10
(26)
where positive pairs from EUT labeled with their true hop distances, and negatives (e.g., siblings unrelated entities) are included at 1:10 ratio. Hard negatives, such as sibling pairs sharing parent Up without subsumption link, enhance the task's difficulty. This bidirectional task also assesses reasoning in both upward and downward directions.
E Learning Interpretable Hierarchical Semantics Through Hyperbolic Geometry
To provide more interpretable results of our HiM models for the hierarchical learning, conducted deeper geometric analysis of the hyperbolic entity embeddings for semantically related WordNet entities under HiM-Poincaré and HiM-Lorentz manifolds (see the visualization of hyperbolic embeddings in Fig- ure 2 Specifically, we computed three key metrics with the learned hyperbolic emboddings: 1) "hyperbolic geodesic distances" between each pair of entities, 2) "h-norm" represents the norm distance from the origin, higher h-norm often indicates a deeper or more specific concept in the hierarchy, 3) "depth" the WordNet tree depth. In both sets of entities, the h-norm correlates strongly with the hicrarchical depth, see For instance, in Table the entity sport (depth 9) has an h-norm of 0.55, while skateboarding (depth 12) has an h-norm of 2. 2.25, reflecting the hierarchical expansion from general to specific concepts. However, a key difference emerges when comparing the two manifolds: HiM-Lorentz consistently produces smaller hyperbolic distances and h-norms compared to HiM-Poincaré. For example, in Table 5 (HiM- Lorentz), the parent-child relationships (e.g., sport - skating - skaleboording) have tighter distances and h-norm gradients compared to Table
Figure E.3
This reduction in distances under the Lorentz manifold is advantageous for hierarchical modeling. The Lorentz model's unbounded nature avoids the boundary constraints of the Poincaré ball, which can lead to numerical instability near the boundary. By mapping embeddings into an unbounded hyperboloid, HiM- Lorentz achieves lighter clustering of related entities (e.g,, between sport and skating: 0.73 h-Norm of HiM-Lorentz vs. 1.67 h-norm of HiM-Poincaré) while maintaining the hierarchical structure. This tighter clustering enhances the model's ability to distinguish fino-grained relationships especially in deeper hicrar- chies, as evidenced by the smaller standard deviations of HiM-Lorentz in performance metries (Table 1
18

---

Published in Transactions on Machine Learning Research (03/2026)
h-norm and Depth
HiM-Poincaré
12.0
2.0
HiM-Lorentz
11.5
Depth
1.5
11.0
h-norm
Depth
10.5
1.0
10.0
0.5
9.0
0.0
sport
outdoor sport
skating
rowing
skateboarding
speed skating
Figure E.3: Alignment between the computed h-norms (derived from hyperbolic embeddings by HiM-Poinaré and HiM-Lorentz) and the actual tree-depth for sports-related entities in the WordNet dataset. As the depth increases from general terms like "sport" to specific ones like "skatcboarding" and "speed skating", both HiM models show increasing h-norms, reflecting the underlying hierarchical structure. While HiM- PRM-Poincaré produces higher h-norms that belter differentiate fine-grained semanlic levels, while HiM-Lorenta yields more compact yet hierarchy preeeriigg embeddings with improved numerical stability. Our results illustrate that both HiM models effectively encode semantic hicrarchy, with Poincaré favoring detail and Lorentz emphasizing robustness.
Table 4: Hyperbolic distances, h-norms, and depths for sports-related entities (from Figure 2 using HiM-
Poincaré, sorted by increasing depth.
sport
outdoor sport
skating
rowing
skateboarding
speed skating
sport
0.00
1.17
1.67
1.62
2.43
2.40
outdoor sport
1.17
0.00
1.90
1.95
2.66
2.62
skating
1.67
1.90
0.00
2.36
3.12
3.07
rowing
1.62
1.95
2.36
0.00
3.10
3.11
skateboarding
2.43
2.66
3.12
3.10
0.00
3.76
speed skating
2.40
2.62
3.07
3.11
3.76
0.00
h-norm depth
0.55
0.97
1.48
1.48
2.25
2.25
9
10
11
11
12
12
F Performance Comparisons between Hyperbolic embeddings and Euclidean embeddings
For mixed-hop prediction (Figure F.4|, HiM-Lorentz achieves better performance on datasets with deeper hierarchies, such DOID (6-hyperbolicity 0.019) and SNOMED-CT (6-hyperbolicity 0.026). This aligns with the Lorentz manifold's ability to handle deeply nested structures more effectively. How- ever, for FoodOn (8-hyperbolicity = 0.185). HiM-Poincaré slightly outperforms HiM-Lorentz, FoodOn's higher d-hyperbolicity indicates a less tree-like structure, suggesting that the Poincaré model's bounded nature may better capture less hierarchical relationships in certain contexts. Both hyperbolic models signif- icantly outperform the Euclidean baselines.
In multi-hop inference (Figure F.5 HiM-Lorentz again demonstrates robust performance, particularly SNOMED-CT and WordNet, which exhibit deeper hierarchies (SNOMED-CT 5-hyperbolieity 0.0254.
19

---

Published in Transactions on Machine Learning Research (03/2026)
Table 5: Hyperbolic distances, h-norms, and depths for sports-related entities (from Figure
using HiM-
Lorentz, sorted by increasing depth.
sport
outdoor sport
skating
rowing
skateboarding
speed skating
sport
0.00
0.55
0.73
0.71
0.87
0.87
outdoor sport
0.55
0.00
0.79
0.87
0.98
0.95
skating
0.73
0.79
0.00
0.95
1.08
1.08
rowing
0.71
0.87
0.95
0.00
1.07
1.07
skateboarding
0.87
0.98
1.08
1.07
0.00
1.20
speed skating
0.87
0.95
1.08
1.07
1.20
0.00
h-norm
0.30
0.48
0.65
0.65
0.81
0.81
depth
9
10
11
12
12
WordNet d-hyperbolicity 0.1431). The smaller standard deviations in HiM-Lorentz's metrics (e.g., 0.003 for SNOMED- CT F1) compared to HiM-Poincaré (0.028) highlight its stability, benefit of the Lorentz manifold's numerical advantages. Notably, HiM-Poincaré achieves a slightly higher recall on SNOMED- CT, suggesting that the bounded nature of the Poincaré ball can occasionally enhance sensitivity. However, the overall F1 score favors HiM-Lorentz, indicating better balance in precision and recall.
A key observation across both tasks is the impact of dataset's tree-like structure as measured by hyperbolicity. Datasets with lower d-hyperbolicity (meaning more tree-like) benefit more from HiM- Lorentz, its unbounded manifold better captures the exponential expansion of deep hierarchies. Tn contrast, FoodOn's higher d-hyperbolicity correlates with HiMM-Pincaré's better performance, suggesting that the choice of manifold may depend on the dataset's s structural properties.
1.0
1.0
0.8
0.8
Precision
0.6
Recall
0.0
0.0
DOID
FaodOn
WordNet
SNOMED-CT
DOID
FoodOn
WordNet
SNOMED-CT
DOID
FoodOn
WordNet
SNOMED-CT
Pretrained
Finetuned
Trained
HiM-Poincaré
HiM-Lorentz
Figure F.4: Comparisons of mixed-hop prediction performance for DOID, FoodOn, WordNet, and SNOMED-CT datasets across our proposed hyperbolic Mamba (HiM) models HiM-Poincaré and HiM- Lorentz and their Euclidean counterparts: Pretrained, Finetuned (with pretrained weights), and Trained from scratch SentenceMamba-116. HiM models trained from scratch hyperbolic space substantially out- perform all Euclidean basclines including the finctuned variant.
Figures F.6 to F.8 illustrate the training dynamics of HiM--Pincaró and HiM-Lorentz across epochs for cach dataset and (ask, plotting Hyperbolic Loss and F1 Score. The Hyperbolic Loss decreases steadily for both models across all datasets, indicating effective optimization hierarchical relationships. HiM-Lorentz often exhibits a slightly faster convergence rate and lower final loss compared HiM-Poincaré, reflecting the Lorentz manifold's suitability for capturing exponential hierarchical expansion. The F1 Score trends mirror the loss behavior, with HiM-Lorentz often achieving slightly better F1 scores.
20

---

Published in Transactions on Machine Learning Research (03/2026)
1.0
1.0
1.0
0.8
0.8
Precision
0.6
Recall
0.6
0.4
0.4
WordNet
SNOMED-CT
WordNet
SNOMED-CT
WordNet
SNOMED-CT
Pretrained
Finetuned
Trained
HiM-Poincaré
HiM-Lorentz
Figure F.5: Comparisons of multi-hop inference performance for WordNet and SNOMED- CE datasets across our proposed hyperbolic Mamba (HiM) Models-HiM-PPinonn and HiM-Lorentz-ani their Euclidean counterparts: Pretrained, Finetuned (with pretrained weights), and Trained from scratch SentenceMamba- 16M. Hyperbolic models demonstrate superior hicrarchical reasoning capabilities compared to all Euclidean variants.
Hyperbolic Loss dold-mixed
F1 Score dold-miixed
Poincaré
HIM Poincaré
0.25
Lorentz
0.80
HMM Lorentz
0.20
0.75
0.70
Score
Losg
0.65
0.60
0.55
0.50
10
Epoch
Epoch
(a) Hyperbolic Loss
(b) F1 Score
Figure F.6: Comparison hyperbolic loss and F1 score on DOID mixed-hop prediction across epochs.
Hyperbolic wordnet-multi
wordnet-mul
0.68
HM Poincart
HIM Poincare
HM Lorentz
HIM Loreniz
0.05
0.84
0.82
0.04
0.80
0.03
0.78
0.76
0.74
0.72
2
10
8
10
Epach
Epoch
(a) Hyperbolic Loss
(b) Fl Score
Figure F.7: Comparison of hyperbolic loss and F'l score WordNet multi-hop inference across epochs.
21

---

Published in Transactions on Machine Learning Research (03/2026)
Hyperbolic Loss snamed mixed
F1 Score shomed- mixed
0.89
0,012
Lorentz
HM Lorenzz
0.010
0.88
0.008
0.87
0.006
F1
0.86
0.002
0.85
0.000
10
Epoch
Epoch
(a) Hyperbolic Loss
(b) F1 Score
Figure F.8: Comparison of hyperbolic loss and F1 score on SNOMED-CT mixed-hop prediction across
the epochs.
G Additional Hierarchical Embedding Visualizations
G.1 FoodOn Ontology: Fruit Hierarchy
Figure G.9 shows HiM's learned embeddings for the FoodOn hierarchy fruit-Onu fruit nut, common hazelnut}.
The model correctly encodes the three-level hicrarchy with monotonically increasing hyperbolic norms "fruit" " is closest to the origin, "nut fruit" at a moderate distance, and leaf nodes "filbert nut" and "common hazelnut" farthest from the origin. This demonstrates the model's ability to learn multi- level taxonomic structure from FoodOn training data.
HiM Trained (Left):
HiM Untrained (Right): The randomly initialized model shows no hierarchical organization; nodes are scattered with inconsistent distance relationships.
G.2 Custom Hierarchy Inference: Figure Concepts
To evaluate gencralization to unseen entities, tested the trained HiM-Poincaré model the conceptual hierarchy from [.I] food-x<dssserts, <ppetizers)~><Chessee cakes, crab cakes). These exact entity names were not present in the FoodOn training ontology, requiring the model to infer hierarchical structure from sentence embeddings. Figure [G.10 shows that the model successfully positions the root concept "food" closest to the origin (norm = 0.30) and mid-level categories ("desserts", "appetizers") at intermediate distances (norms Rs 0.85). The grandchildren nodes achieve similar norms (ss 0.85-0.86) but are distinguished through angular separation, demonstrating that hyperbolic models can leverage both radial and angular dimensions when explicit hicrarchical supervision is unavailable.
G.3 Visualization Methodology for Hyperbolic Embeddings
The 2D visualizations in Figure 2 and Appendix Figures procedure:
G.9
G.10 were generated using the following
(1) Embedding Extraction: For a selected entity and its hierarchical neighbors (parent, children, neg- alives), we extracted their final 384-dimensional hyperbolic embeddings from the trained HiM-Poincaré model.
(2) Dimensionality Reduction: We applied UMAP (MeInnes al. 2018 configured for hyperbolic geometry using the umap-learn library.
22

---

Published in Transactions on Machine Learning Research (03/2026)
HiM Trained (Poincare)
HiM Untrained (Poincare)
Filbert nut
common hazelnut
fruit
nut
fruit
Origin
Origin
fruit
filbert nut
common hazelnut
Solected Node
Children
Node - Parent
Positive Parent
Origin
Node Children
Figure G.9: Comparison of trained vs. untrained HiM embeddings on FoodOn hierarchy fruit--ut fruit->fiibbert, hazelnut}. Trained model (left) preserves hierarchical distances; untrained model (right) shows random placement.
(3) Projection: The 2D UMAP embeddings were re-projected onto the Poincaré disk using the exponential map preserve hyperbolic distances.
(4) Rendering:
Visualizations use polar coordinates where radial position corresponds to hyperbolic norm
and angular position follows UMAP embedding angles, rendered via matplotlib (Hunter 2007
This methodology ensures that the 2D projections respects the hyperbolic geometry, and the hierarchical depth correlates with radial distance from the origin.
H Additional Experimental Analysis
H.1 Larger pretrained LM baselines vs. HiM
To address concerns about whether the weak pretroined Euclidean baseline is primarily an effect of small parameter scale, additionally report zero-shot inference-only results for substantially larger pretrained encoders, evaluated with the similar candidate-rankinn protocol used for our Euclidean baselines. The eval- uation protocol used in both mixed-hop prediction and multi-hop inference evaluates each child against one positive parent and multiple sampled negatives. Under this setting, zero-shot pretrained encoders frequently assign moderately high similarity scores to large subset of candidates, resulting in inflated recall despite poor ranking fidelity. Consoquently, several larger pretrained models exhibit near-saturated recall values while maintaining very low precision and F1 scores, indicating that the high recall is primarily driven by class imbalance rather than meaningful recovery of hierarchical parent relationships.
23

---

Published in Transactions on Machine Learning Research (03/2026)
HiM Trained (Poincare)
crab cakes
appetizers
food
Origin
Origin
Childben
Figure G.10: Out-of-distribution inference on hicrarchy example from Figure A.1 Root and intermediato levels are correctly ordered; leaf-level separation is reduced due to lack of training supervision for more specific entities.
Table 6: Zero-shot inference baselines of larger pretrained encoders on hierarchical prediction/infet /inference tasks, compared with HiM performance. Best per datasct column is bold; sccond-best is underlined (tics are broken deterministically to keep single highlight per colunn). Metric: F1.
Model
Params (M) doid-mixxd foodon-mixed wordnel-mixed wordnet-multi snomed-mixed snomed-mulli
part-base-uncasee
0.155
0.128
0.197
0,201
0.173
0.176
hert-large large- uncased
340
0.165
0.165
0.166
0.168
0.160
0.161
roberta-base
0.135
0.161
0.167
0.170
0.121
roberta-large
355
0.167
0.167
0.167
0.167
0.167
0.167
Al--Minil.M--6ee
22
0.004
0.029
0.003
0.003
0.010
0.009
all-mpnet-base-v2
0.182
0.172
0.173
0.173
0.161
0.161
largrob
355
0.167
0.167
0.167
0.167
0.167
0.167
HiM-Poincaré
16
0.795
0.836
0.824
0.865
0.886
0.919
HiM-Loreutz
0.821
0.827
0.823
0.872
0.890
0.920
Table 8: Same setting as Table 6 metric: Recall. High recall values reflect class imbalance the candidate- ranking protocol (10:1 peggative-tto-ositiie
Model
Params (M)
doid-mixed foodon-mixed wordnet-mixed wordnet-multi suomed-mixed suomed-multi
bert-base-uue buse-nucased
110
0.496
0.107
0.387
0.392
0.578
0.606
lat-llarg--ucceett
0.978
0.972
0.874
0.883
0.927
roberta-base
125
0.556
0.581
0.711
0.721
0.423
0.416
roberta-large
355
0.993
0.993
0.994
0.994
0.992
0.993
Al--M-niLM--66-22
0.002
all-mpnet-bnsee-2
110
0.697
0.667
0.669
0.670
0.588
0.582
lagge-v
355
0.992
0.993
0.994
0.994
0.993
0.993
IHin-Poincaéé
0.863
MoM-Loreutz
16
0.820
0.803
0.815
0.872
0.880
0.920
24

---

Published in Transactions on Machine Learning Research (03/2026)
Table 7: Same setting as Table
metric: Precision.
Params (M)
doid-mixed foodon-mixed
wordnet-multi snomed-mixed snomed-multi
bert-basee uncased
110
0.092
0.159
0.132
0.135
0.101
0.103
port-largo-umeased
0.090
0.090
0.092
0.093
roberta-lsase
125
0.077
0.095
0.095
0.096
0.073
0.073
roberta-large
353
0.091
0.091
0.091
0.091
0.091
0.091
Ml--M-nil.M-L.6-22
0.179
0.080
0.091
all-umpue--uase-ee
110
0.105
0.099
0.099
0.100
0.096
0.095
all-roberta-large-vl
355
0.091
0.091
0.091
0.091
0.091
0.091
HiM-Poincaré
0.812
0.841
0.853
0.867
0.894
0.917
HiM- L
16
0.822
0.852
0.828
0.871
0.901
0.919
H.2 Prompted LLM Experiment (GPT-4o vs. HiM)
To evaluate HiM's performance relative to contemporary large language models, we conducted zero-shot evaluation using GPT-40 on the WordNet mixed-hop prediction lask. This comparison provides insight into how our specialized hyperbolic architecture performs against general-purpose language models that rely on vast pretraining but lack explicit hierarchical inductive biases. We generated 500 binary classification questions following the structure "Is [entity1] subtype/subclass of [entity2]?" sampled from the same test set used for HiM evaluation. The experimental setup mirrored HiM's training regime: for each sampled child node, generated one positive question and ten negative questions (corresponding to HiM's 1 positive parent + 10 hard negatives). GPT-4o was provided with list of 74,401 WordNet entities as context and answered all 500 questions in a single zero-shot prompt without additional training. The results in Table g demonstrate that both HiM variants substantially outperform GPT-4o. GPT-4o performs well on general knowledge hierarchies where concepts like 'dog' + 'animal' are well-represented in large-scale training corpora. HiM (both Poincaré and Lorentz) still outperform GPT-4o by clear margin, even on such general-knowledge hierarchy. The superior performance of HiM highlights the effectiveness of hyperbolic modeling approach for hicrarchical reasoning, even when compared to pretrained LLM with significantly larger parameter counts and extensive pretraining.
Table 9: F1 scores comparing HiM models with GPT-4o on WordNet mixed-hop prediction task
Dataset
HiM
GPT-4o
Poincaré
Lorentz
WordNet-mixed
0.859
0.850
0.750
H.3 Candidate-Selection Ranking Experiment (GPT-40 vs. HiM)
We evaluated GPT-4o using a candidate-seleetion/rannnn methodology using a similar evaluation strategy as HiM. Instead of free-form generation like in Appendix H.2 GPT-4o was presented with a list of candidate positive parents and negatives and tasked to select the correct hierarchical relationship. The comparison is presented Table I0
25

---

Published in Transactions on Machine Learning Research (03/2026)
Table 10: Performance comparison between GPT-4o (candidate-ranking) and HiM models on mixed-hop prediction and multi-hop inference tasks.
Model
Dataset
Precision
Recall
DOID Dataset (Average 8- d-hyperbolicity = 0.0190)
GPT-4o
doid-mixed
0.834
0.834
0.834
HiM-PPimnaré
doid-mixed
0.795
0.812
0.780
HiM-Lorentz
doid-mixed
0.821
0.822
0.820
FoodOn Dataset (Average d-hyperbolicity 0.1852)
GPT-40
foodon-mixed
0.539
0.540
0.539
HiM-Poincaré
foodon-mixed
0.836
0.841
0.831
HiM-Lorentz
foodon-mixed
0.827
0.852
0.803
WordNet Dataset (Mixed-hon -ho,, 6-hyperbolicity
0.1438)
GPT-4o
wordnet-mixed 0.550
0.550
0.550
HiM-PPinnaré wordnet-mixed
0.824
0.853
0.798
HiM-Lorenlz
wordnel-mixed
0.823
0.828
0.815
WordNet Dataset (Multi-hop, 8- d-hyperbolicity = 0.1431)
GPT-4o
wordnet-multi 0.516
0.516
0.516
HiM-Poincaré
wordnet-multi
0.865
0.867
0.863
HiM--Lrrenz
wordnet-multi
0.872
0.871
0.872
GPT-4o achieves competitive performance on the DOID dataset, likely due to its relatively small size and low d-hyperbolicity (0.0190), which indicates a near tree-like structure that aligns well with semantically de- scriptive biomedical terminology present in the model's pretraining corpus. GPT-4o shows perfect Precision- Recall balance across datasets, indicating consistent decision thresholds. However, HiM models substantially outperform GPT-4o on FoodOn and WordNet. While GPT-4o benefits from extensive semantic knowledge, it lacks the geometric inductive bias required to consistently encode hierarchical distances and transitivity con- straints. In contrast, HiM explicitly learns these structural relationships through geometry-aware training, enabling more robust hierarchical reasoning across ontologies, even under a candidate-selection evaluation aligned with embodding- based ranking.
H.4 Computational Efficiency Analysis
To substantiate our claims regarding Mamba's linear complexity advantages, we conducted sequence length scaling study on the WordNet dataset for the mixed-hop prediction task. The results in Table I1 validate Mamba's theoretical linear complexity characteristics. Doubling the sequence length from 128 to 256 tokens results in an exact 2x increase in both FLOPs (4.46G + 8.9G) and MACs (2.23G + 4.45G), while memory consumption remains constant 66.91MB due to the fixed number of model parameters and activations. This linear scaling behavior contrasts sharply with transformer-based architectures, where sequence length increases would result. in quadratic growth in computational requirements.
Table 11:
Sequence length scaling analysis for HiM-16M-Poincaré on WordNet mixed-hop prediction
Sequence Length
FLOPs
MACs
Memory
Training Time
128
4.46
2.23 G
66.91 MB
0:41:32 0:42:13 13
256
8.90
4.45 G
66.91 MB
26

---

Published in Transactions on Machine Learning Research (03/2026)
Ablation Studies
We conducted systematic ablation studies to validate key design choices in HiM architecture. All ablations were performed on WordNet-mixed and DOID-mixed tasks using the default HiM configuration. Results are reported as mean + standard deviation over 5 independent runs.
1.1 Loss Component Ablation
We evaluated six loss configurations with varying margin ratios (a, 3) where margins scale with hyperbolic radius r:
original (0.255 rr 0.0051 - r): Our default balanced configuration
clusterin (0.255 0.0): Clustering loss only (no parent-child ordering)
only centripetal (0.0, 0.0051 - rr): Centripetal loss only (no entity separation)
Ligh_clusterrng (0.510 r, 0.0051 r): Doubled clustering margin
high ceethcipettttlllllel (0.255 r, 0.010.0112--0 r):: Doubled centripetal margin
standard contrastive: Euclidean triplet loss
Results Table 12 demonstrate that the original balanced configuration achieves competitive performance standarddcontrasttvv across both manifolds. Critically, standard contrastive loss performs substantially worse, demonstrating that Euclidean triplet losses fundamentally unsuitable for hyperbolic embeddings due to the mis- match between flat Euclidean geometry and negatively curved hyperbolic space. Using only clustering or ooll_oollltte ceilly__ettipttall may slightly reduces performance, supporting that both components contribute comple- mentarily: clustering enforces entity separation while centripetal preserves parent-child hierarchy. Notably, chol_cruutiiin achieves the best Lorentz performance on WordNet, suggesting that stronger negative sep- aration may benefit deeper hierarchies in an unbounded manifold.
Table 12: Loss component ablation on WordNet-mixod and DOID-mixed. Standard contrastive loss (Eu-
clidean triplet) shows substantial degradation, confirming the necessity of hyperbolic-specific losses.
WordNet-mixed
DOID-mixed
Loss Configuration
Poincaré (F1)
Lorentz (F1)
Poincaré (F1)
Lorentz (F1)
original
0.843 0.007
0.843 = 0.001
0.811 + 0.008
0.823 0.015
only _elu__triiingg
0.842 0.011
0.844 + 0.003
0.786 + 0.021
0.813 0.021
boily_centripeai
0.817 0.005
0.808 = 0.005
0.775 = 0.038
0.790 + 0.010
high cclusteriug
0.833 0.005
0.845 + 0.005
0.797 + 0.010
0.798 + 0.014
high centrripett
0.839 0.011
0.843 + 0.001
0.810 + 0.008
0.809 + 0.025
0.452 = 0.085
0.605 = 0.042
0.197 + 0.020
0.264 + 0.012
1.2 Pooling Strategy Ablation
We compared throe pooling strategics to aggregate Mamba2 sequence outputs: (1) Mcan-pooling: averaging all token embeddings (our default), (2) Max-pooling: element-wise maximum over sequence, (3) Last To- ken: using final position embedding (analogous to transformer CLS tokens, but leveraging Mamba's causal autoregressive accumulation rather than dedicated token).
Results in Table 13 show that mean pooling achieves the best overall performance. Last token pooling performs comparably on WordNet (F1=0.844 for Lorentz), reflecting Mamba's autoregressive nature where final states accumulate the entire soquence information. Max pooling consistently underperforms, likely because max operations can amplify outlier features unsuitable for hierarchical distance computation. These findings validate our default choice of mean pooling.
27

---

Published in Transactions on Machine Learning Research (03/2026)
Table 13: Pooling strategy ablation. Mean pooling achieves best overall performance; last token is competi- tive on WordNet-Lorentz due Mamba's autoregressive state accumulation.
WordNNet-mxxed
DOID-mixed
Pooling
Poincaré (F1)
Lorentz (F1)
Poincaré (F1)
Lorentz (F1)
Mean
0.843 + 0.007
0.843 + 0.001
0.811
0.008
0.823 + 0.015
Max
0.833 + 0.006
0.833 + 0.005
0.794
0.007
0.726 + 0.057
Last Token
0.843 + 0.006
0.844 + 0.008
0.802 0.020
0.806 + 0.011
1.3 Model Depth Ablation
We evaluated 2, 4, and 6 Mamba2 layers while keeping other hyperparameters fixed (d model=384, d state=128, expand=4). As shown in Table 14 the 6-layer configuration underperforms across both datasets. This suggests that excessive depth is unnecessary for capturing hierarchical structure when com- bined with hyperbolic geometry.
Table 14: Model depth ablation. 2-4 layers offer optimal capacity; 6 layers show overfitting moderately- sized ontologics.
WordNet-mixed
DOID-mixed
Layers
Poincaré (F1)
Lorentz (F1)
Poincaré (F1)
Lorentz (F1)
2
0.844
0.007
0.843 + 0.002
0.805 + 0.009
0.802 + 0.026
4
0.843
0.007
0.843 + 0.001
0.811 + 0.008
0.823 +'t 0.015
6
0.834 0.006
0.833 + 0.006
0.799 + 0.005
0.779 + 0.000
1.4 Negative Sampling Ablation
We compared "hard negative mining" (selecting semantically similar siblings/cousing against "random neg- ative sampling" (uniformly sampling entities). Results in Table 15 show mixed outcomes, hard negatives improve WordNet performance, likely due to WordNet's complex multi-level hierarchy requiring fine-grained discrimination. However, on DOID, random negatives perform better, possibly because DOID's is smaller dataset where random negatives themselves provide sufficient separation.
Table 15: Negative sampling ablation. Hard negatives improve complex hierarchies (WordNet) but show mixod results on smaller datascts (DOID).
WordNet-mixed
DOID-mixed
Sampling
Poincaré (F1)
Lorentz (F1)
Poincaré (F1)
Lorentz (F1)
Hard Negative
0.843 + 0.007
0.843 = 0.001
0.808 = 0.005
0.823 + 0.015
Random Negative
0.828 + 0.016
0.841 + 0.009
0.839 = 0.022
0.852 + 0.023
J Model Size Configurations
To investigate the effect of model capacity on hierarchical reasoning performance, we scaled our HiM archi- tecture from 16M to 32M parameters. Table 16 details the complete hyperparameter configuration for both model scales. We kept the base hidden dimension (d_ (d_model 384), state dimension (d_ (d_state 128), and convolution kernel size (d_conv 7) constant maintain consistency in the interfaces and local temporal modeling.
28

---

Published in Transactions on Machine Learning Research (03/2026)
Table 16: Hyperparameter configuration for 16M and 32M parameter models. Scaling achieved through combined depth (n llyer) and width (expand, headdim) increascs.
Hyperparameter
16M
32M
d_ d_model (Hidden Dim)
384
384
n_ __layer (Depth)
3
7
d_state (State Dim)
128
128
d_ conv (Conv Kernel)
7
expand (Expansion Factor)
4
headdim (Head Dim)
96
128
Total Parameters ~66
~32M
K
Code Availability
The source code for the Hicrarchical Mamba (HiM) model publicly available at https://github. com/ BerryByte/HaM, with detailed instructions for spt and execution.
L Declaration of LLM usage
LLMs were only used assist with writing and formatting, not as part of the core methodology. Large Language Models (LLMs) were used in limited capacity during the preparation of this manuscript for grammar checking and text refinement. All technical contributions, results and insights are the original work of the authors.
29


Reviews:
[{"review_id": "D85qtG28WJ", "reviewer": "Reviewer_iYMq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces HiM, a contrastive learning framework for training sentence embedding models based on the Mamba2 architecture, using techniques from hyperbolic geometry to learn hierarchical representations. Concretely, HiM projects sentence embeddings stemming from a Mamba2 language model to either the Poincaré or the Lorentz manifold and computes a contrastive loss which encourages multiple desirable properties in hyperbolic space. For example, a centripetal loss component is responsible for encouraging parent entities to be located closer to the origin of the hyperbolic space than its child entities, overall encouraging embeddings whose hierarchical relationships are reflected in the hyperbolic space. The authors conduct experiments on multiple hierarchical embedding tasks, and ablate their framework by substituting Mamba2 with a transformer model. HiM demonstrates overall strong performance in the 16-32M parameter regime.\n\n### Strengths\n* The motivation of the paper is clear and relevant.\n* The method description (Section 3.1) is particularly clearly written.\n* The main novelty compared to prior work seems to be the integration of the Lorentz manifold projection, which shows overall the best performance. The authors have further investigated ways to overcome training instabilities when training with the Lorentz manifold projection.\n\n### Weaknesses\n* The biggest weakness in my view is the limited novelty of the work. The authors use the idea of hyperbolic projection as well as the loss term from the HiT paper [1], with the only adjustments being the introduction of the Lorentz projection, as well as swapping out the transformer for a Mamba2 model. However, I also want to note that these two changes are demonstrably effective at the 16-32M parameter scale.\n* Another weakness is the lack of depth in the empirical study, in particular the small parameter scale and some open questions about the overall quality of the baseline models. For example, in Table 1, we observe that the performance of the fine-tuned baseline LM dramatically improves over the pre-trained model. This suggests, that the LM simply struggles with domain-transfer to the hierarchical datasets. This could be well explained by the small parameter scale of the baseline LM. While I would not want to ask the authors for training larger models, I would find it reasonable to at least report baseline numbers for larger LMs at inference to ensure that the poor performance of the pre-trained LM is not due to its very small parameter scale.\n* Finally, in Section 5.3, the authors visualize the embeddings of HiM. But as far as I can tell, the authors do not explain *how* the embeddings are visualized. I could also not find clear answers in the linked appendix section.\n* In Section 5, the authors state \"while the fine-tuned SentenceMamba- 16M model is initialized using Kaiming normal initialization for weights and zero initialization for biases, then fine-tuned on hierarchical datasets\". Calling the fine-tuned model \"fine-tuned\" is misleading, as fine-tuning usually refers to further tuning a pre-trained model, not training a model from scratch on a downstream dataset. That being said, it would be interesting to report performance of an actual fine-tuned version of a pre-trained LM.\n* Also in Section 5, the authors state \"To ensure a fair comparison while preserving the hyperbolic nature of the space, we randomly initialize our HiM and evaluate it against a randomly initialized version of HiT (HiT*)\". Does that mean the authors compare to a HiT with initial weights? Why do the authors not compare to HiT with trained weights? In fact, I believe the authors do this, considering the numbers of HiT are at least competitive in some tasks. If this is true, the authors should adjust this sentence, explaining that they retrained HiT.\n\n### References\n[1] He et al., Language Models as Hierarchy Encoders, 2024", "claims_and_evidence_yn": "No", "claims_explanation": "While the authors mostly validate their claims about HiM producing strong hierarchical embeddings compared to the baseline models, I am not convinced by the following claims:\n\n> Unlike prior works using adapters or pre/post-processing, HiM projects sentence-level Mamba2 representations directly into hyperbolic manifolds\n>\n\nIt appears to me that HiT also projects transformer-based LM embeddings into hyperbolic manifolds. The main difference between HiM and HiT seems to be the underlying LM. If the authors find my characterization of their contribution unjustified, I encourage them to explain in more detail how the HiM framework fundamentally differs from HiT.\n\n> We introduce SentenceMamba-16M, a Mamba2-based LLM (16M parameters) trained at sentence-level embeddings on the SNLI dataset\n>\n\nI am not sure I understand this contribution. Mamba2 is an established architecture and the SNLI dataset is an existing dataset. By \"introduce\", do the authors mean that they have trained a Mamba2 model on this dataset? If so, I would hardly count this as a contribution.\n\n> HiM employs weighted clustering and centripetal losses to enforce parent-child separation and compact clustering of related entities with respect to origin, enhancing hierarchical structure preservation in hyperbolic space.\n>\n\nAgain, this loss originates from the HiT paper, which the authors also acknowledge in the method description. If so, this cannot count as a contribution. I ask the authors to clarify how their loss differs in a significant way, warranting a novel contribution.", "audience_yn": "Yes", "audience_explanation": "Absolutely. This work is highly relevant to the language modeling community, as well as to the geometric deep learning community, bringing new insights and ideas to both communities.", "requested_changes": "Critical to my recommendation are the following requested changes or explanations:\n\n1. The authors should either elaborate more on how their contributions differ from prior work, in particular HiT, or adjust the contributions in the paper to reflect the actual contributions. These are, from what I currently understand, the use of Mamba2, and the use of the Lorentz manifold projection.\n2. The authors should adjust the experimental section according to the weaknesses listed above. In particular, the authors should run inference on larger embeddings LMs for Table 1, and explain or adjust statements about fine-tuning and the \"randomly initialized\" HiT model.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bXytVvwcLA", "reviewer": "Reviewer_15H8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes Hierarchical Mamba (HiM), combining Mamba2 state-space sequence modeling with hyperbolic embedding spaces (Poincaré ball and Lorentz manifold) to learn hierarchy-aware sentence embeddings. The method introduces learnable curvature and stabilizes Lorentz hyperbolic operations using bounded norms and curvature-aware approximations. It also proposes a hyperbolic loss (centripetal \\+ clustering) to explicitly enforce parent–child organization in the embedding geometry. Across ontology-based datasets, HiM improves hierarchical reasoning performance on mixed-hop prediction and multi-hop inference over Euclidean baselines.\n\nStrengths:\n\n* Novel integration of SSMs (Mamba2) with hyperbolic geometry for hierarchical reasoning.  \n* Strong gains over Euclidean versions\n\nWeakness:\n\n* Experimental evaluation lacks sufficient ablations to justify multiple design choices (loss terms, mining strategy, architecture/pooling decisions).", "claims_and_evidence_yn": "No", "claims_explanation": "While the paper shows that HiM performs much better than the Euclidean version, the experiments are limited and include very few ablations. Because of this, it is hard to tell which design choices really matter. Many choices are explained using intuition, but there is not enough experimental evidence.\n\n Here are some ablations I suggest:\n\n* Ablation for the proposed hyperbolic loss components (e.g., centripetal vs. clustering loss vs standard contrastive loss)  \n* Ablation comparing hard negative mining against random negative sampling\n\nThere are a number of other ablations as well that are missing. I have mentioned rest in requested changes.", "audience_yn": "Yes", "audience_explanation": "This paper proposes Hierarchical Mamba (HiM), integrating the Mamba2 linear-complexity state-space model with hyperbolic geometry to learn hierarchy-aware sentence embeddings for structured reasoning tasks. The work is timely and relevant to TMLR readers interested in efficient long-sequence modeling, representation learning, and geometry-aware neural architectures, especially given the increasing attention to Mamba-style models in modern ML.", "requested_changes": "* Depth/architecture ablation: number of Mamba2 blocks (e.g., 2 vs 4 vs 6).  \n* Pooling ablation: mean pooling vs alternatives (eg: CLS token)  \n* Report standard deviations for Table 2  \n* Describe how the model is scaled from 16M to 32M (e.g., which parameters change: depth, hidden size, or state dimension  \n* In Appendix A.7.1 (HiM vs. GPT-4o), the LLM baseline is evaluated via free-form generation, which may conflate reasoning errors with formatting/ontology naming issues. A fairer comparison would be to evaluate GPT-4o in a candidate-selection/ranking setting (choose the correct parent/ancestor from a provided shortlist), closer to HiM’s evaluation where candidates are ranked via embedding similarity.", "broader_impact_concerns": "There are no broader impact concerns from this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jZWsWizYoz", "reviewer": "Reviewer_vT5E", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In their work, the authors argue that hyperbolic geometry, in comparison to standard euclidean representations, is better suited for embedding natural language concepts and representing the complex, hierarchical relationships in latent space. In contrast to prior works that leverage simpler, but less expressive hyperbolic geometries, the authors aim to leverage Poincaré and more expressive Lorentz representations while constraining the hyperbolic manifold and tackling possible numerical instabilities. The authors propose the Hierarchical Mamba (HiM) architecture and evaluate on a particular SentenceMamba-16M model which was trained with specific loss terms to tackle long-distance and structured, hierarchical textual relations.\n\nExperiments concern the hyperbolicity of the learned latent space, the eventual obtained F1-scores over structured mixed- and multi-hop reasoning and a qualitative embedding analysis of an actual set of concepts. All presented results are consistent with each other and seem to show a well working of the method.\n\n\n\n**Strengths**\n\n1. Construction of the proposed architecture is clearly described and the need for its individual components is well motivated. Overall, the authors are able to convincingly argue for the utility of hyperbolic latent spaces. The introduced changes, based off of prior findings and their own experimental results seem to agree with the initial motivation. The authors provide a high level of implementation detail and describe the experimental setup clearly. Results are well interpreted and seem to align with the obtained numbers.\n2. The authors are furthermore able to apply task specific centripetal and clustering losses, that enforce the hierarchical structure of natural language texts in latent representations. Even though my expertise for the given domain is limited, presented experiments seem to indicate consistent strong performance improvements under the chosen setups for mixed- and multi-hop reasoning. Comparisons to a transformer-based baseline seems to indicate the better utilization of hyperbolic representations by the Mamba architecture.\n\n\n\n**Weaknesses**\n\n1. The authors motivate the use of hyperbolic latent spaces with the embedding of hierarchical representations from natural language and are able to demonstrate good general performance. In contrast to other experiments, section 5.3 qualitatively discusses hierarchical embeddings on single set of concepts only. The paper might be improved by providing a more diverse set of cases (e.g. including the \"food\" example of Figure 3) and possibly also providing failure or unclear case.\n2. Although the authors focus primarily on textual embeddings, introduction and related work primarily discuss general generative language modeling. Switching from this part to actual experiments and conclusion, I felt like there was a slight mismatch between the initial motivating discussion and the eventual experiments. This might be fixed by relating some of the prior results more closely to the task of text embedding, or by providing a more general outlook of the obtained results onto the more broader task of text generation.\n3. While table 1 seems to indicate a better hyperbolicity of of the Lorentz model latent space, table 2 seems to show consistent stronger F1-scores of the Poincaré model. This seems to oppose to the initial argument of the authors on leveraging more expressive hyperbolic representations.\n\n\n\n**Minor**\n\n* The authors do not report standard deviations/variances for Table 2. Therefore, it is unclear how highlighting for individual results is computed. Some results seem to be made bold even though they are not best in their #parameter/curvature/dataset/...-specific group.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Even though the authors focus on a particular task of text embedding on a specific set of benchmarks, the obtained results show clear indication of a proper working of the method. Experimental training and evaluation details are described clearly. Comparisons across different model sizes, comparison to transformer baselines, as well as the qualitative inspection of embedding space all align in their results.", "audience_yn": "Yes", "audience_explanation": "While my expertise on general SSMs/Mamba-like models is limited, the paper seems to present a sound and well motivated theoretical contribution. The authors are able to make a convincing case towards more expressive hyperbolic space representations and are able to demonstrate the well working of their approach. While the focus on this paper is on text embedding rather than general language modeling, the particular results of SSM benefiting in particular, and in contrast to transformer-based architectures, might be a valuable insight beyond the presented experiments of this paper.", "requested_changes": "In addition to possible changes addressing the mentioned weaknesses, I would like to recommend the following changes:\n\n* While the authors present a formal description of the Mamba2 architecture in the appendix, they continue to use specific formal terms, (e.g. $x'_t$ in the second paragraph of Sec. 3.1), without prior mention in the main text. The authors might improve consistency of the text by repeating some of the key equations (e.g., Eqs. 16 and 17) of the Mamba2 architecture in the main text.\n\n* Eq. (20) includes an undefined \"1SS(a)\" term. They authors should briefly explain its definition.\n\n\n\n**Minor:**\n\n* Figures 9 and 10 seem to use spline interpolation between the points of actual measurement. In general, splines tend to over/undershoot actual measurements, giving a impression of possibly altered minimum/maximum observed loss values than were actually observed. The authors might want to use the same linear interpolation as done in figure 8 to avoid these effects.\n* While the listed references seem to be fine in general, the authors might want to check some of the entries again. In particular, Vaswani is cited as the sole  author of \"Attention is all you need\". Similarly, the authors might want to cite \"Aspects on the theory of syntax\" as its original 1965 version (although the 2014 reprint seems to be more popular by now).\n* Figures 1 and 5-10 in the appendix seems to be included as rasterized images and do not render well. The authors might want improve the paper slightly by including the corresponding figures as a vector graphics, similar to the remaining figures in the paper.", "broader_impact_concerns": "None", "additional_comments": "None", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)