HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts
Authors: Neil He
Rishabh Anand
Hiren Madhu
Ali Maatouk
Smita Krishnaswamy
Leandros Tassiulas
Menglin Yang
Rex Ying
Venue: NeurIPS 2025 poster ()
Keywords: large language models
hyperbolic geometry
mixture-of-experts
Primary area: deep_learning
TL;DR: We introduce HELM, a family of hyperbolic large language models operating fully in hyperbolic space.

Abstract: Frontier large language models (LLMs) have shown great success in text modeling and generation tasks across domains. However, natural language exhibits inherent semantic hierarchies and nuanced geometric structure, which current LLMs do not capture completely owing to their reliance on Euclidean operations such as dot-products and norms. Furthermore, recent studies have shown that not respecting the underlying geometry of token embeddings leads to training instabilities and degradation of generative capabilities. These findings suggest that shifting to non-Euclidean geometries can better align language models with the underlying geometry of text. We thus propose to operate fully in $\textit{Hyperbolic space}$, known for its expansive, scale-free, and low-distortion properties. To this end, we introduce $\textbf{HELM}$, a family of $\textbf{H}$yp$\textbf{E}$rbolic Large $\textbf{L}$anguage $\textbf{M}$odels, offering a geometric rethinking of the Transformer-based LLM that addresses the representational inflexibility, missing set of necessary operations, and poor scalability of existing hyperbolic LMs. We additionally introduce a $\textbf{Mi}$xture-of-$\textbf{C}$urvature $\textbf{E}$xperts model, $\textbf{HELM-MiCE}$, where each expert operates in a distinct curvature space to encode more fine-grained geometric structure from text, as well as a dense model, $\textbf{HELM-D}$. For $\textbf{HELM-MiCE}$, we further develop hyperbolic Multi-Head Latent Attention ($\textbf{HMLA}$) for efficient, reduced-KV-cache training and inference. For both models, we further develop essential hyperbolic equivalents of rotary positional encodings and root mean square normalization. We are the first to train fully hyperbolic LLMs at billion-parameter scale, and evaluate them on well-known benchmarks such as MMLU and ARC, spanning STEM problem-solving, general knowledge, and commonsense reasoning. Our results show consistent gains from our $\textbf{HELM}$ architectures – up to 4\% – over popular Euclidean architectures used in LLaMA and DeepSeek with superior semantic hierarchy modeling capabilities, highlighting the efficacy and enhanced reasoning afforded by hyperbolic geometry in large-scale language model pretraining.

HELM: Hyperbolic Large Language Models via Mixture-of-Curyature Experts
Neil He
Rishabh Anand"
Hiren Madhu
Ali Maatouk
Smita Krishnaswamy Leandros Tassiulas
Menglin Yang'
Rex Ying
Yale University, USA
"Equal Contribution
Open-source code: www/Graph-aaaaaaae htt./Graapaan---aaa aawrrpp-annd-Gemmtrriccooo hamdd--Gemmtri---eaaninnnnann
github.com//enn
Abstract
Frontier large language models (LLMs) have shown great success in text modeling and generation tasks across domains. However, natural language exhibits inherent semantic hierarchies and nuanced geometric structure, which current LLMs do not capture completely owing their reliance on Euclidean operations such as dot- products and norms. Furthermore, recent studies have shown that not respecting the underlying geometry of token embeddings leads to training instabilities and degradation of generative capabilities. These findings suggest that shifting to non-Euclidean geometries can better align language models with the underlying geometry of text. We thus propose to operate fully in Hyperbolic space, known for its expansive, scale-free, and low-distortion properties. To this end, we introduce HELM, family of HypErbolic Large Language Models, offering geometric rethinking of the Transformer basedd LLM that addresses the representational inflexibility, missing set of necessary operations, and poor scalability of existing hyperbolic LMs. We additionally introduce a Mixture-of Offtuureof-Cuuvaaurr Curvature Experts model, HELM-MICF, where each expert operates in distinct curvature space encode more fine-grained geometric structure from text, as well as a dense model, HELM- further develop hyperholic Multi-Head Latent Attention (HMLA) for efficient, reduced-KV K--cche training and inference. For both models, we further develop essential hyperbolic equivalents of rotary positional encodings and root mean square normalization. We are the first to train fully hyperbolic LLMs at billion parimeer scale, and evaluate them on well-known benchmarks such as MMLU and ARC, spanning STEM gobblem-solvigg general knowledge, and commonsense reasoning. Our results show consistent gains from our HELM architectures up to 4% over popular Euclidean architectures used in LLaMA and DeepSeek with superior semantic hierarchy modeling capabilities, highlighting the efficacy and enhanced reasoning afforded by hyperbolic geometry in large-scale language model pretraining.
D. For HEL.M-MICS
1 Introduction
Contemporary Large Language Models (LLMs) [18, 42, 9, 1] fundamentally operate within Euclidean space. This manifests in their reliance on Euclidean operations such dot products and norms applied to token embeddings. However, this architecture presents potential mismatch with the intrinsic structure of natural language data. Existing works have shown that textual data, particularly token inputs to LLMs, exhibit an inherent semantic hierarchy [48, 47, 21, 35], thus requiring
'Correpondence Author; work done while at Yale University, USA.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

space that can naturally accommodate these relationships. An ideal LLM architecture would possess geometric alignment with the underlying structure of the data it aims to represent.
To further illustrate the unique geometry of text data, we show in Figure/ 1 the distribution of Ricci Curvature of token embeddings from popular decoder-only LLMs on 1000 diverse samples from RedPajama [46]². We ob- serve that the vast majority of tokens exhibit a wide range of negative curvatures. This has also been observed by Robinson et al. [38], who investigate the subspace of to- ken embeddings and its inherent Euclidean structure. As Ricci curvature measures the local geometry of man- ifold, these empirical observations suggest hierarchical token structures, while the variation in curvature values suggest complex token geometry that cannot be captured by single curvature approaches. Robinson et al. [38] also show that not respecting the geometry of tokens will harm a Transformer-based LLM's generative capabilities while introducing undue training instabilities. He et al. [21]
Liama 2 7B
Gemma-2
DeepSeek MoE
0.10
Pmbbailitt
2.06
0.62
-1.00 -0.75
-0.50
-0.25
0.00
0.25
0.50
0.7
1.00
OllivierRRici Curvature
Figure Ricci curvature distribution of token embeddings from docoder-only LLMs showing substantial variation of negative curvature, implying higher local by pertbollicity.
also document power law distributions of token frequencies, implying hierarchy among tokens. We thus propose a geometric rethinking of the transformer- based LLM: operate fully in hyperbolic space, where the negative curvature results in exponentially increasing volume w.r.t distance. Hyperbolic space is expansive, scale-free, and yields low distortion, and has shown success in numerous settings, particularly in Transformer architectures [48, 5, 19, 6, 39]. It provides token embeddings more "breathing room" in the network, better aligning with the underlying structure of the text data.
However, previous works that study hyperbolic Transformers and pre-trained models have several major shortcomings: (1) Inflexible geometric spaces. They assign each Transformer block a single hyperbolic manifold, embedding entire token sequences in fixed-curvature spaces. This approach does not align with the observed substantial variant in curvature values as noted above, thus limiting the expressiveness of hidden representations. (2) Lack of essential operations. They omit widely-used LLM components such as rotary positional encoding and RMS normalization, and lack theoretical guarantees of LLM modules in Euclidean settings; (3) Poor scalability. They focus low-dimensional settings and use quadratic hyperbolic self- attentin mechanisms that do not scale comparably modern Euclidean foundation models [18, 10]. In this work, address the limitations of both Euclidean LLMs and prior hyperbolic Transformers through the following contributions:
To alleviate limitation (1), we introduce hyperbolic Mixture-of -orvvvtturrr Experts (MICE), where each expert operates in distinct curvature space, enabling the model to encode fine-grained geometric structure from text. The mixed-curvature strategy employed by MICE captures the range of negative curvatures prevalent among token embeddings, mitigating previous hyperbolic Transformers' representational inflexibility.
To resolve limitation (2), we introduce several novel hyperbolic modules to develop hyperbolic LLMs: Hyperbolic Rotary Positional Encodings (HoPE) and Hyperbolic RMSNorm, bridging the gap between hyperbolic Transformers and modern Fuclidean L.MMs. Additionally, we provide extensive theoretical analysis that provides similar guarantees in the Euclidean case.
To address limitation (3), we propose Hyperbolic Multi-head Latent Attention (HMLA) to perform efficient inference with reduced-footprine KV cache and reduce active memory during training, thereby bridging the scalability gap between previous hyperbolic Transformers and current Euclidean L.L.s.
Finally, we introduce HELM, the first attempt building family of fully HypErbolic Large Language Models. We are the first to train at billion parameter scale and outperform popular Fuclidean architectures on diverse set of benchmarks.
'Ricci Curvature is metric of how strongly or loosely connected region of space is. The more negative the curvature, the more hyperbolic the space is. We describe the metric in Appendix A.I.

---

2 Related Work
Hyperbolic Transformers. Hyperbolic geometry has emerged as a powerful embedding space for representational learning, particularly in domains characterized by hierarchical and scale-free structures [16, 34, 15]. Hyperbolic Neural Networks (HNN) [15] and their extension HNN++ [39] have shown that operating in hyperbolic spaces increases representational capacity. Several works have incorporated hyperbolic geometry into Transformers to better capture semantic relationships. HAT [19]. +NN++ [39], and HyboNet [5] all propose equivalent formulations of hyperbolic attention in different models of hyperbolic space. HypFormer [48] developed several essential modules that were lacking in previous works, showing improved performance for structured and hierarchical datasets. Although work on hyperbolic pre-trained language models exist [6], they ignore essen- tial components required to train large language models, such as normalization layers, residual connections, roatry positional encodings. Furthermore, these works suffer from the limitations mentioned Section I. Previous work have also considered mixed-curvature Transformers [7], however, they only using different curvature values in each attention head and relying on tangent space methods that are prone to mapping errors and numerical stability [5, 48, 20, 3J. Further related works include LResNet [23] who introduces an efficient hyperbolic residual connection method that mitigated numerical instability and tangent space mappings. Some works have devised hybrid approaches by incorporating hyperbolic components to existing pre-trained Euclidean LLMs and vision models [12, 36, 47, 17, 41J. However, these works do not develop and pre-train hyperbolic LLMs from scratch. Our work advances this line of research by developing and scaling hyperbolic architectures to large-scale pretraining setups, and additionally introducing novel components for efficient and expressive language modeling in hyperbolic space.
Open Large Language Models. The recent surge in open LLMs has democratized access to cutting- edge NLP capabilities, enabling broader research and application development. Notable among these is LLaMA [44], which introduced family of efficient and powerful models trained on diverse, large-scale corpora. Llama models employ several optimizations such as rotary positional embeddings and grouped query attention, making them competitive across various downstream tasks. Building on these ideas, Gemma [42] introduced further improvements for better data curation, advanced pertaining techniques, and careful model scaling strategies. In parallel, Mixture-of-Exxeee of-EExeets (MoE) architectures have emerged prominent design to enhance model capacity without a proportional increase in computation cost. DeepSeek-Men dynamically activate subset of experts per input, significantly improving inference throughput compared to other MoE models such as Mixtral [27]. However, all these models are inherently Euclidean, and while effective, may not align well with the geometry of the token subspace.
[11] introduces an efficient routing mechanism to
3 Preliminary
We introduce the relevant background required to understand the building blocks of HELM, including hyperbolic geometry, hyperbolic self-attention, and other useful hyperbolic modules.
3.1 Lorentz Hyperbolic Geometry
There are several isometric models of hyperbolic space employed in previous research [45, 19, 43, 34, 48, 5, 31. In this study, we choose the Lorentz model: its special space-time interactions allow for fully hyperbolic operations, offering enhanced expressiveness and empirical stability from an optimization perspective [5, 48, 33]. Nevertheless, our methods can be easily reformulated for other models of hyperbolic geometry via isometric mappings.
Lorentz model. An re-dimensional Lorentz model, denoted as L K.", is a Riemannian manifold L" equipped with the Riemannian metric tensor g diia(-1) 1,..,, 1) and defined by constant negative curvature K < O. Each point x E T_ has parametrized form [xt, x,)"" where xn E T. is the time-like dimension and x, E R. is the space-like dimension. For points x, y E LLKa, their Lorentzian inner product (x,y)c is given by (x, = -It + x y. x" On y. Hence, the Lorentzian norm |||x|||| = / (xx x) Formally. the point set D = fx E Ra+l (x,x) = 1/K, x2 0}. The origin E L K. is the point V .,....
Tangent space. The tangent space at point E K." is set of points orthogonal to x, defined as T Km = (y E R0-11 (x = 0}. Notably, the tangent space is isomorphic Fuclidean space.
3

---

3.2 Hyperbolic Neural Network Modules
Extensive work has been done to develop hyperbolic neural network modules [48, 5, 3, 45, 23, 15, 39, 19I, which we introduce below.
Lorentz Linear Layer. Given curvatures K1, K2, and parameters W E R(++1)xm and b E R", the
Lorentzian linear transformation with curvature [48] is the map HLT L _ii. + IL K2.m given by.
K2 HLT(X; W. - Ki
VIWTT+bbl' b/22-1111/// 1/K2, W Txx+
x x+bb
(1)
Lorentz Residual Connection. Let x, f(x) E K," where x is an input vector and f(x) is the output of a neural network f. Then, the Lorentzian residual connection [23] is given by,
x B f(x) = 01x + a2y, w./ (v [V-K wxx w2 fx))(x))ll)))) for {0,1),
(2)
where 01, a2 are weights parametrized by constants (W1, w2) E R2? {(0,0)).
Hyperbolic self-attention. Hyperbolic self-attention is defined equivalently in different models through manifold midpoints [39, 19, 5]. Given tokens where x; E K.a, we WK wv g(a+l)xd, then the Lorentzian self-attention [5, 48] is given by
Σj= 4,
exp (q.
Axt V-K|| Σ vilve
where Va E exp (-p(-B2(q k1) /m)
(3)
where de is the Lorentzian distance and Q HXT(X; Wº, b9), HLT(X: W bk), HLT(X; are the queries, keys, and values respectively.
4 Method
In this section, we propose several novel hyperbolic modules that serve as building blocks for HELM. We then introduce the overall architecture of HELM: a Mixture-of- -iee--ff--CC-uurrreee variant, HELM-MICE, and dense variant, HELM-D, -D, for comparison.
4.1 Hyperbolic Rotary Positional Encoding
Previous works that proposed positional encoding methods in hyperbolic space [5, 48, 22] are confined only learning relative encodings. However, contemporary LLMs [18, 10, 4] have instead shifted towards Rotary Positional Encodings (RoPE) [40], a scalable alternative that incorporates aspects from both absolute and relative encoding methods, improving length generalization and downstream performance. We thus propose novel hyperbolic rotary positional encoding (HoPE) construct positional encodings in hyperbolic space, and prove the same theoretical guarantees as RoPE. Given 7' tokens X, where x; E K,dd (d even), let Q, K be the queries and keys as in Equation (3). The hyperbolic rotary positional encoding applied to the i-th token
R. 0
0
0
e R. (z,)(
R
0
0
HoPE(z;))
Ri
(4)
0 R.,.aar
where Ri is the 2D rotation matrix parameterized by angle i0, and can be a query qi or key kj.
Next, we study the theoretical aspects of HoPE; all proofs can be found in Appendix A. First note that since R. e is an Euclidean rotation matrix, it isometrically preserves the (Euclidean) norm of vectors. Given the definition of the Lorentz model (Section 3.1), an equivalent expression for Equation (4) is
(
0
HoPE(z;)
R,,, z,, making HoPE valid Lorentzian rotation operation (sce, e.g., [5]).
Validity. A defining characteristic for the Euclidean RoPE that the inner product of the encoded keys and queries is a function of only the word embeddings and their relative position. Thus, only the relative positional information encoded [40]. For hyperbolic attention Equation (3), the analogous is defined with -dc instead, which we formalize below.

---

Proposition 4.1. Let X be T tokens with x; E L.d.d Let Q, K. be queries and keys as in Equation (3). Then dee (HoPE (q,), HoPE (ks))) = =x(,, g(xa: a b) for some finnttinn g.
HOPE only encodes relative positional information based on Proposition 4.1 similar RoPE, which establishes its validity as a RoPE operation.
Long-term decay. A desiring property of RoPE is long-term decay, where the attention score between a key-query pair decays when the relative position increases. HOPE has the same property as well, as shown by the following proposition.
Proposition 4.2. Let Q, K. be defined in Equation (3), then the negative square Lorentz distance -cc (HoPE (q:) Hk)))(k)) can be upper bounded by f(g, kj)g(i - j) < 0, where I has no dependencies on position, and g depends entirely on relative position and scales inversely w.r.t. i
Thus, HoPE ensures far-apart tokens have weaker connections based on Proposition 4.2, a property not guaranteed by previous learned encoding methods.
Robustness to arbitrary token distances. Barbero et al. [2] recently show that decaying token dependency (Proposition 4.2) may not be the primary reason for RoPE's success, but rather it enables LLMs to attend to specific relative distances. Our formulation of HOPE also ensures this property:
Proposition 4.3. Let Q K be as defined in Equation (3), then HoPE can be maximal at an arbitrary distance, ie., for any relative distance Z, there exists a key kj such that the softmax value is maximum at distance r.
HОPE thus provides hyperbolic transformers the flexibility to decay token dependencies while also ensuring robust attention across arbitrary relative distances.
Positional attention. Barbero al. [2] also prove that using RoPE can help capture purely positional relationships via diagonal attention patterns, where tokens only attend to themselves, and off dff-oiigonal attention patterns, where tokens attend only their preceding neighbor. HOPE also allows for this:
Proposition 4.4. Attention heads with HoPE can learn diagonal or off-diagonal attention patterns.
We provide ablations comparing HoPE prior hyperbolic positional encodings in Appendix D.
4.2 Hyperbolic Mixture-of- -urrv-vturrrr Module
Previous hyperbolic Transformer architectures fix each block to a single hyperbolic manifold, forcing the entire sequence to be embedded with fixed curvature, restricting the flexibility of the hidden representations. Mixture-of-Expertt (MoE) [26] used in Euclidean LLMs [10, 30, 13, 27, 9], where the model selectively activates only subset of specialized "experts" for each token, enables LLMs to learn more diverse data patterns while remaining computationally efficient. However, the experts are still limited one geometric space, restricting the collective granularity the experts can learn. Accommodating variable curvatures calls for a more flexible treatment: we propose the first Mixture- of- Curvaaateeee Experts (MICE) module, where each expert operates on a distinct curvature space. Let x, E L K1.m be the t-th token input, then MiCE : x, LLK. + X2 E IL K,n,, where N, the number of routed experts and N₂ is the number of shared experts. First, we pass X₃ through a gating module to obtain the gating scores for each routed expert, denoted as Jt.; for S Nr, given as,
Hi. ; st, = (xt((x)). 96.3 gt. Σx 96.
[ 8ti 8tJ 0
K+)
(5)
otherwise
Here, St,j is the expert affinity with activation function act, y, is the centroid vector of the i-th routed expert, Topk(S, A) picks the top A values from set S, and K, is the number of activated experts. Then, the token passed through each shared and routed expert. Let HFFN, L - I K... be the routed experts and HFFN,, K. + L K. be the shared experts, defined through hyperbolic feedforward networks. Here, the value of K,,5 and Kg,s can vary for each expert, i.e., each expert lives on a distinct manifold. To align the input's manifold and the experts' manifolds, first we project the tokens to the expert manifolds via s,,; = /K/ Kg, and Fi,; = VK/K,,X, The projected token is passed through each expert and projected back to the input manifold, where we obtain y1,
/Kaa/ /K,,/KHFFN KHFFN,, (St,)) and Z KFFFN, (Fr,)).
5

---

Octput
Merge Experts
Po Podaaa
Routed
Shaned
at
Opeana
Cating Modull
Projet
Hark
Letent
Letret Koyghtee
Quggy
Preettin
Ingutt Tokera
Teiters
(a) MICE module architecture. Routed experts are selected through a gating module. The token are project from input manifold to expert manifold and then passed through each expert. The output of each expert are then project back to the input manifold and merged together through Lorentzian centroid. This modules allows experts to learn from distinct curva- ture spaces to allow for more granularity.
(b) HMLA framework. The embeddings are projected into latent space and then upward projected into queries, keys, and values. Ad- ditional decoupled queries and a shared key are created for hyperbolic positional encoding through HoPF. The queries and keys are con- catenated together before performing hyperbolic self-attention.
Figure Mixture-of-Curvauur Experts (MICE) and hyperbolic Multi-Head Latent Attention (HMLA).
The output of MiCEN; is given by,
No Ve. +
Nr 2
MICE (xx) = X,
(6)
EA Va. ΣE 24.2
The constants V KK.iKK K project from the experts' manifolds to the input manifold, ensuring that the output of each shared and routed expert lives on the same manifold. The outputs are then combined through the Lorentzian centroid [29], before performing Lorentzian residual connection. Note that MICE is indeed a valid hyperbolic module, which we expand on in Appendix
4.3 Efficiency via Hyperbolic Multi-Head Latent Attention
Previous hyperbolic Transformers for natural language applications focus mainly on hyperbolic self- attention, synonymous to naive dot-product attention mechanism in Euclidean LLMs. However, recent Euclidean LLMs have gradually adopted more efficient attention methods for enhanced scalability. The quadratic attention mechanism and the extant hyperbolic self-attention formulation suffer from heavy memory challenges for large-scale training. In this section, we propose hyperbolic Multi-Head Latent Attention (HMLA) to alleviate this bottleneck. Inspired by Euclidean MLA [11, 10], HMLA reduces the size of the KV cache during inference and the active memory consumption during training. We provide a high-level description of HMLA; detailed formulation can be found in Appendix B.2.
Let X K,K,n be the t-token, where n. is the embedding dimension and h is the number of heads. First, we project x, via HLT to latent queries and key-value vectors, obtaining of c.VV of dimen- sions ny. Rike respectively such that ng. Tike < n/h, We then upward-project the latent vectors back to hn heads of dimension n each via HLT, obtaining [kC]les.... [v,iksn, from e and [qC.kks from cf Then the projected keys processed by positional encodings. Following previous works [11, 10], as RoPE incompatible with MIAA due to position coupling, we employ a decoupled HOPE scheme with HMLA. Through HLT, we project cf to additional query vectors, [g and c to shared key, k," of dimension h,, and n,, respectively (n, < nhh,). The queries vectors, gf a. and the key vectors, kG,,- k. are then concatenated together through Lorentzian concatenation [37, 48] for each where obtain the final query and key vectors qt,á, ky.- The atten- tion score and output queries, keys, and values, before concatenating all the heads together with Lorentzian concatenation. The concatenated result is passed through final projection layer with HLT.
computed using Equation (3) with (q..desh... [kg,llesmm: [vG]es lesh. the
Memory complexity, During inference, HMLAA only requires caching the latent key-value pairs. As result, the memory footprint for HMLA is O((RRo + n, L), where L is the number of layers.

---

Hyperboltc Embeed
Laventz RMSNor
itpertalic Alluntann
Lore nou IOMES RAMES Naarn
Pisturenn" Departe (MICE)
Lorenez nMSNNam
Loghts
Hypertolls Allicttann
Hyperbelic e Medworrk
Figure 3: HELM architecture. The input tokens are mapped to hyperbolic word embeddings before being processed by a series of L decoder blocks, comprising an attention block and FFN block. The attention block (blue) can either be hyperbolic self-attention or HMLA, while the FFN block (yellow) can either be HFFN MICE layer. The output of the decoder blocks is mapped to logits. Residual connections are omittted for brevity.
In contrast, the hyperbolic self-attention used in previous hyperbolic Transformers (Fquation (3)) requires storing the full-sized keys and values, resulting in memory complexity of (2),, By choosing R R2 < AıAA, we have +2,) < 2nMp, resulting in a significantly smaller memory footprint while maintaining the same time complexity of O(nnn)²)) Additionally, the latent query projection also results in smaller active footprint during training. This collective mechanism enables far greater scalability.
4.4 Hyperbolic RMSNorm
Previous works have not implemented hyperbolic root mean square normalization (RMSNorm), which is widely used in popular Euclidean LLMs [18, 11, 10] due its stability and robustness in both forward and backward passes. Here, we propose hyperbolic RMSNorm to fill the gap:
RMSNormc(x))
|RMSNrm((x.)) - 1/K, RMSNorm(x.)
Here, RMSNorm denotes the corresponding Euclidean operation. Equation (7) is equivalent to passing RMSNorm into the HRC functions from Hypformer [48]. This formulation is chosen over other methods of defining normalization layers through hyperbolic midpoint and tangent space operations [45. 3] for better numerical stability and computational efficiency.
Invariance to input scaling, Our formulation of hyperbolic RMSNorm is invariant to a scaling of inputs. giving us similar guarantees Euclidean RMSNorm in terms of gradient stability during backpropagation and enhanced robustness to perturbations. We provide proofs for these guarantees in Appendix A.7.
Proposition 4.5. RMSNormc is invariant scaling of inputs during both the forward and backward passes.
4.5 Overall Architecture for Hyperbolic Large Language Models (HELM)
We introduce the framework for hyperbolic LLMs (HELM) based on the modules we introduced and developed in Section 4. In particular, we develop hyperbolic LLMs with Mixture-of Cireof--uurauree Experts (HELM-MICE)) a class of hyperbolic LLMs with MoE modules where each expert functions in a distinct curvature space. We also construct hyperbolic dense LLMs (HELM-D), which resembles classic decoder-only LLMs such as Llama [18].
The overall architecture is as follows (see Figure 3): tokenized text is first mapped to learned hyperbolic word embeddings, which are then passed through series of hyperbolic decoder blocks, each consisting of two components: 1) the attention component, where the embeddings are normalized by a RMSNorm layer, then processed by an attention block such as HMLA or self-attention, and finally added to the embeddings through Equation (2); and 2) the HFFN component, where the processed embeddings are again normalized by RMSNorm before being passed through HFFN block and residually added to the output of the attention block (Equation (2)). For HELM-MICE, the

---

Table 1: Multichoice question question-answeringg answering accuracy (%) of HELM models and their Euclidean counterparts. Bold denotes highest accuracy and underline denotes second-highed HELM models consistently outperform their Euclidean counterparts, with HELM-MICE achieving the highest accuracy overall. The 100M model performances are average over 3 runs.
Model
# Params
CommonsenseeQA
HellaSwag 0-Shot
OpenbookQA 0-Shot
MMLU
ARC-Challenging 5-Shot
Avg
0-Shot
5-Shot
LLAMA
115M
20.9 + 0.3
25.1 + 0.3
25.4 + 0.2
2.5
21.0
23.2 0.2
HELM-D
115M
20.3 0.2
25.9 u.1
27.1
25.6
21.4+0..
24.1
DEEPSBER
T20M
19.3 0.2
25.3
2..0
2.3
22.2
22.2
HELM-MICE
120M
19.7 + 0.3
25.9 + 0.2
27.7++0.4 0.4
24.4 0.2
23.2 0.5
24.1 + 0.1
DEEPSEEKV3
IB
19.5
26.2
27.4
23.6
22.7
23.9
HELM-MICE
IB
19.8
26.5
28.4
25,9
23.7
24.9
HFFN block can either be dense block such HFFN MICE block defined in Section 4.2, where HFNNsG is hyperbolic SwiGLU FFN we built to be consistent with Euclidean LLMs (see Appendix B.3 for details). HELM-D contains only dense HFFN layers. The output of the final decoder block is then normalized once again before projected to logits for next-token prediction.
5
Experiments
We evaluate both HELM variants' ability to answer MCQ questions in two popular benchmarks, MMLU [24] and ARC [8]. Additionally, we train an ablation HELM-MICE with constant curvature across experts, comparing with HELM-MICE models with varying curvature across experts.
Runtime, Memory, and additional experiments. One common concern for hyperbolic models is that they could incur significant computational overhead when compared with their Fuclidean counterparts. HELM models are within 1.55X in runtime and 1.1 in memory usage of the Fuclidean counterparts for both the 100M and 1B models. Additional details are shown in Appendix D. We also show additional ablation and comparison with prior hyperbolic language models, for tasks such as machine translation, in Appendix D.
5.1 Multichoice Benchmarking
We evaluate both HELM-MICE and HELM-D at 100M-parameter scales, across variety of benchmarks spanning STEM problem-sol general knowledge, and commonsense reasoning. The dense models also serve as an ablation comparison with the MICE models. We further scale the HELM-MICE to IB parameters as the smaller HELM-MICH model outperformed HELM-D overall. Additional details regarding implementation and datasets can be found in Appendix B.
Training Setup. We use the LLaMA3. 1-8B tokenizer [18] for all models, with vocabulary size of 128K. For HELM-D, we use hyperbolic self-attention and HFFNsG for the decoder block. We use heads and layers for the 100M model. For HELM-MICE, we use HMLA and mixture of dense and MICE layers, each with 2 active experts and one shared expert. We use 6 heads, 6 layers, and experts per layer for the 100M model, and we use 14 heads, 16 layers, and experts per layer for the 1B model. The experts have curvature initiated uniformly from -0.1 to 2.0.. Additionally, we incorporate the auxiliary- -los--free load balancing scheme and complementary sequence-Ve wise auxiliary loss from DeepSeekV3 [10] to encourage load balancing among the experts. Each model was trained on a cluster of 4 NVIDIA A6000 and 4 NVIDIA A800 GPUs with model and data parallelism, where at most 4 GPUs were used by each model.
We use the English portion of the Wikipedia dataset [14] for training, comprising ~6.4M rows of raw text, or roughly 5B tokens.
Hyperbolic word embedding. Previous works [5, 39] directly map input tokens trained hyperbolic embeddings. However, we experienced model instability when training the 1B models with this method. Therefore, we only train the space-like dimension of the Lorentz word embeddings.
Baselines. We test against two popular Euclidean models: one dense model and one MoE model. For the dense model, we test HELM-D against LLaMA [18]. For the MoE model, we test HELM-MICH
against V3 [10]. We train both baselines from scratch at the same parameter scales as their HELM counterparts, with the same dataset. tokenizer, and training setup.
8

---

Table 2: Ablation accuracy, where we compare HELM-MICE with a variant where all experts have the same curvature value, denoted as MICE-CONST. Bolding denotes the highest accuracy and underline denotes the second-highest. Euclidean DEEPSEEK V3 results are shown for reference. Overall, FELM-MICE consistently achieves the highest accuracy, while both hyperbolic models still outperform the Euclidean counterpart.
Model
Params CommonsenseQA
HellaSwag 0-Shot
OpenbookQA 0-Shot
MMLU S-Shot
ARC-Challenging
Avg
O-Shot
5.Shot
DEEPSEERV3
120M
19.2
25.2
23.4
24.2
21.8
22.8
MIČH-CORST
T20M
20.00
25.5
27.0
23.5
.23
23.7
HELM-MICE
120M
19.7 + 0.3
25.9 + 0.2
27.7 + 0.4
24.4 0.2
23.2
24.1 0.1
Benchmarks. We evaluate on a variety of benchmarks, including STEM and general knowledge reasoning benchmarks such MMLU [24], ARC-Challenging [8], and OpenbookQA [32], and commonsense reasoning benchmarks such as CommonsenseQA [41] succh HellaSwag [50]. For MMLU and ARC, we use O-shot prediction.
5-shot predictions. For Commonsense OpenbookQA, and HellaSwag,
The results are shown in Table 1. We report the accuracy of the models' abilities to answer multiple choice questions from the benchmarks. We mainly focus on comparing models within the same architectural sub-family, i.e., dense models and MoE models are separately tested against each other. Both HELM variants consistently outperform their Fuclidean counterparts. smaller HELM-D model achieves higher accuracy than LLaMA on four out of the five benchmarks, whereas the smaller HELM-MICE model outperforms the smaller DeepSeekV3 model on all five benchmarks. When comparing the ~ 1000Mssale HELM-D D and HELM-MICE models, the latter achieves comparable performance despite using overall significantly fewer active parameters. This reflects the effectiveness of using more flexible geometry. For the larger IB -arrammeterr models, HELM-MICH consistently outperforms the IB DeepSeekV3 model, achieving the highest accuracy overall. While don't provide standard deviation from multiple runs for the IB models is typical of models this size, we provide results of model performance across different stages of training in Appendix D and show that HELM-MICE consistently outperform its Fuclidean counterpart.
Results.
particular, the
In all cases, the hyperbolic LLMs achieve better overall scores across the five benchmarks. The HELM models also always achieve higher accuracy the more difficult reasoning benchmarks, namely MMLU and ARC-Challenging. This suggests better reasoning capability afforded by in- corporating more suitable geometries in the embedding space. Overall, results demonstrate the superiority of hyperbolic LLMs in particular, the Mixture-of- of-Curvaturee Experts framework - in answering complex multiple-choice questions across wide range of domains.
5.2 Ablating Distinct Curvature Learning with HELM-MICE
To assess the effectiveness of each expert operating in distinct curvature space, we train 120M- parameter HELM-MICE model where the curvature of each expert is fixed to 1.0, which we denote as MICE-CONST. Consequently, MICE-CO
embeds the entire token sequence into fixed space of curvature - -1.0 similar to a dense model. MICE-CONST is trained with the same setup as the preceding models. We show the results in Table 2. HELM-MICE outperforms the constant-curvature MICE-CONST out of the 5 benchmarks and achieves the higher overall accuracy, demonstrating the effectiveness of learning more expressive presentation by setting each expert to learn within distinct manifold. Notably, MICE-CONSS still outperforms the Euclidean DeepSeck V3 baseline on all 5 of the benchmarks, further demonstrating the effectiveness of hyperbolic LLMs over their Euclidean counterparts. We don't 't provide statistics for ablation basclines due to the need for excessive compute. Nevertheless, the results remain statistically significant as demonstrated in Table 2.
5.3 Qualitative Studies on Semantic Hierarchy Modeling
In this section, we qualitatively access the ability of HELM to model semantic hierarchy against its Euclidean counterparts. Our investigation of final-layer embedding distributions have found that HELM learns representations where more generic words tend to cluster in areas of smaller norm and more specific words tend to have larger norms. Table 3, we provide case studies for HELMM M IB and Deepseck V3 IB, where we show embedding norm in the final layers for words of varying

---

Table 3:
Case study investigate of embedding norm in the final layer of HELM (IB) and Deepseek
(IB). Top: embedding norm of words of varying levels of specificity; Bottom: embedding norm of question taken from the MMLU benchmark. For HELM-MICE, more generic words are clustered closer to the origin than more specific words, which has a smaller norm than even more specific words. However, this does not necessarily hold for the Deepseek V3 1B model.
Words
HELM-MICE
Deepseck V3
Average Norm
Range
Average Norm
Range
to, in, have, that, and, is, for study, research, subject, papers, category
35,930
35 890-3595551
33,725
33.660-33.80 830
36,080
36.030-336.03
33,735
33.668-33.776
biology, physics, chemistry, mathematics, com- puter science
36.155
36.033-36.270
33,720
33.658-33.776
algehra, geometry, photosyythesis, cellular respira-
36.288
36, 13.-33336
33.741
33.622-33.826
tion, genetics
HELM-MICE
D3epseek
Words
Norm Range
Words
Norm Range
A, How, does, if, there, have, any, with, of
36.03 -36.399
is, connecting, graph, there, edges, complete. have, of
33 93.668-33,766
discrcte, vertices, edges, connecting, pair, graph. complete, many.
36.700-36.717
discrete, 10, how, if. pair, does, with, A, vertioes. 33.772-33.908 any
levels of specificity (top table) and for sample question taken from the MMLU benchmark. For HELM-MiCE, more generic words (e.g., subject) are clustered closer to the origin than more specific words (e.g., biology), which has a smaller norm than even more specific words (e.g., photosynthesis). However, this does not necessarily hold for the Deepseek V3 1B model, demonstrating how HELM- MiCE better handles semantic hierarchies. The hierarchical organization of the space enables the HELM models to sometimes better navigate the embedding space and obtain better performance.
6 Conclusion
In this work, we introduce HELM, a family of fully hyperbolic large language models trained at hundred-mille miilion and billion-paramae parameter scales. Operating entirely in hyperbolic space, HELM models are better aligned with the variable geometrie structure of text and token distributions. We develop MICE modules to construct HELM-MICE variants, enabling fine-grained geometric learning and more expressive and geometrically flexible hidden representations. We further introduce HMLA mechanism to enable HELM models to be memory efficient and improve scalability. We also introduce the HOPE and R.MSNRRM modules, which are fundamental building modern hyperbolic LLMs, and support them with extensive theoretical analysis and guarantees. Trained on 5B tokens, HELM models outperform their Euclidean counterparts across benchmarks STEM reasoning, commonsense reasoning, and general knowledge. Nevertheless, the research presented has a few limitations. Due to computational constraints, our experiments only compare HELM to Euclidean LLMs trained the same 5B tokens, which have less representational capacity when compared to the commercially available LLMs trained on much more extensive data [4, 1, 18, 42, 10]. Additionally, we chose the Wikipedia dataset for its widely accepted reliability. However, the trained models might be under-exposed to areas such as mathematical reasoning as a result. Future work could explore incorporating scaling laws [28, 25] for hyperbolic LLMs across larger compute and data frontiers to investigate their potential.
Acknowledgments
This work was supported in part by the National Science Foundation (NSF) IIS Div Of Information & Intelligent Systems 2403317 and Army Research Office contract W911NFF23- 1-3---0088. We also acknowledge support in part from the Silicon Valley Community Foundation, an Amazon research award, the Yale AI Engineering Research Grant from Yale Office of the Provost, and an LEAP-U Sponsored Research from Samsung Research America. Moreover, this research has greatly benefited from the discussions and research talks held at the IMS-NT Joint Workshop on Applied Geometry for Data Sciences. We also thank Ngoc Bui (Yale Univeristy) for useful feedback and discussion.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, and Petar Veličković. Round and round we go! what makes rotary positional encodings useful?, 2025. URL ptttpp://ariiv
abs/ abs//4a/.....////////////a//aa// 241..06200.0....
[3] Ahmad Bdeir, Kristian Schwethelm, and Niels Landwehr. Fully hyperbolic convolutional neural networks for computer vision. ICI.R, 2024.
[4] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach, GPT- NeoX-20B: An open-source autoregressive language model. In Proceedings of the ACI. Work- shop on Challenges & Perspectives in Creating Large Language Models, 2022.
[5] Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou, Fully hyperbolic neural networks. arXiv preprint arXiv:2105.14663 14686, 2021.
[6] Weize Chen, Xu Han, Yankai Lin, Kaichen He, Ruobing Xie, Jie Zhou, and Zhiyuan Liu. Hyperbolic pre-trained language model. IEEE TASLP, 32, 2024.
[7] Sungjun Cho, Seunghyuk Cho, Sungwoo Park, Hankook Lee, Honglak Lee, and Moontae Lee. Curve your attention: Mixed-curvature transformers for graph representation learning. arXiv preprint arXiv:2309. 0400200009000 2023.
[8] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.0544 054557 2018.
[9] Damai Dai, Chengqi Deng, Chenggang Zhao, R. X. Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y. Wu, Zhenda Xie. Y. K. Li, Panpan Huang, Fuli Luo, Chong Ruan, Zhifang Sui, and Wenfeng Liang. Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models. CoRR, abs/2401.06066, 2024. URL https://arxiv. org aaa/a/bs//200 06066.
[10] DelpSeck-Al. Deepseck-v3 technical report, 2024. URL httphs//aariii..og/aaaaae. 19437.
[11] DeepSeek-AI. Deepseek-v2 A strong, economical, and efficient mixture-of-experts language model, 2024.
[12] Karan Desai, Maximilian Nickel, Tammay Rajpurohit, Justin Johnson, and Shanmukha Ramakr- ishna Vedantam. Hyperbolic image-text representations. In ICML, pages 7694-7731. PMLR, 2023.
[13] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23:1-40, 2022.
[14] Wikimedia Foundation. Wikimedia downloads. URL WRRLhttss://ummp wikimedia..ni.m.mmmmmmmmm.....a. org.
[15] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In NeurlPS, pages 5345-5355 2018.
[16] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In ICML, pages 1646-1655, PMLR, 2018.
[17] Songwei Ge, Shlok Kumar Mishra, Simon Kornblith, Chun-Liang Li, and David Jacobs. Hyper- bolic contrastive learning for visual representations beyond objects. ArXiv, abs/2212.0065 2022.

---

[18] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv:2407.21788, 2024.
[19] Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, et al. Hyperbolic attention networks. In ICLR, 2019.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[21] Neil He, Jiahong Liu, Buze Zhang, Ngoc Bui, Ali Maatouk, Menglin Yang, Irwin King. Melanie Weber, and Rex Ying. Position: Beyond euclidean foundation models should embrace non-euclidean geometries. arXiv preprint 7XXv:25544088966 2025.
[22] Neil He, Menglin Yang, and Rex Ying. Hypercore: The core framework for building hyperbolic foundation models with comprehensive modules. arXiv preprint arXiv:2504.089) 08990,,, 2025.
[23] Neil He, Menglin Yang, and Rex Ying. Lorentzian residual neural networks. In KDD, 2025.
[24]
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In ICLR, 2021.
[25] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optiall large language models, 2022. URL https::/taraatt org/ abs/2205 15556.
[26] Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-8)) 87, 1991. doi: 10. 10.1162//eco.19911.31 79.
[27] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and et al. Mixtral of experts. arXiv preprint 2:Xiv:2401.0400 2024.
[28] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. URL https: http///xxvvvv./// wwt.://////a/////eeeeoeoe /000///0.0000..611....
[29] Mare Law, Renjie Liao, Jake Snell, and Richard Zemel. Lorentzian distance learning for hyperbolic representations. In ICML, pages 3672-3683 PMLR, 2019.
[30] Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding. In International Conference on Learning Represen- tations (ICLR), 2021.
[31] Paolo Mandica, Luca Franco, Konstantinos Kallidromitis, Suzanne Petryk, and Fabio Galasso. Hyperbolic learning with multimodal large language models. arXiv preprint arXiv:2408.05097, 2024.
[32] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, pages 2381-2391 Association for Computational Linguistics, 2018
[33] Gal Mishne, Zhengchao Wan, Yusu Wang, and Sheng Yang. The numerical stability of wwt./oor/////.../eoo//
hyperbolic representation learning, 2024. URL https:/
[34] Maximillian Nickel and Douwe Kiela, Poincaré embeddings for learning hierarchical represen- tations. In NewrIPS, pages 6338-63477 2017.
12

---

[35] Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In ICML, pages 3779-3788, 2018.
[36] Avik Pal, Max van Spengler, Guido Maria D' Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision-language models. ICLR, 2025.
[37] Eric Qu and Dongmian Zou, Autoencoding hyperbolic representation for adversarial generation. arXiv preprint 2rX.v:2201.12825, 2022.
[38] Michael Robinson, Sourya Dey, and Shauna Sweet. The structure of the token space for large language models, 2024. URL
org/ abs/2410 .09993.
[39] Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic neural networks++. In ICLR, 2020.
[40] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint vxXv::214.099844, 2021.
[41] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsense A question answering challenge targeting commonsense knowledge. In NAACL, pages 4149-4158, 2019.
[42] Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint 0rXiv:2403.08225, 2024. URL https:///arii: org/oog///oooo aa///bb//24/// 220//443.0000 08295.
[43] Alexandru Tifrea, Gary Bécigneul, and Octavian-Fugen Ganea. Poincaré glove: Hyperbolic word embeddings. arXiv preprint 18/0. 2018.
[44] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint Stt::2307.09288, 2023.
[45] Max van Spengler, Erwin Berkhout, and Pascal Mettes. Poincaré resnet. CVPR, 2023.
[46] Maurice Weber, Daniel Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexan- drov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. Redpajama: an open dataset for training large language models, 2024. URL /etps://arxiis org/ abs/2411 12372.
[47] Menglin Yang, Aosong Feng. Bo Xiong, Jihong Liu, Irwin King, and Rex Ying. Hyperbolic fine-tuning for large language models. ICML LLM Cognition Workshop, 2024.
[48] Menglin Yang, Harshit Verma, Delvin Ce Zhang, Jiahong Liu, Irwin King, and Rex Ying. Hypformer: Exploring efficient transformer fully in hyperbolic space. In KDD, pages 3770- 3781, 2024.
[49] Yun Yue, Fangzhou Lin, Kazunori D. Yamada, and Ziming Zhang. Hyperbolic contrastive learning. arXiv preprint arXhv:2302.01409, 2023.
[50] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? In ACL, pages 4791-4800, 2019.
[51] Biao Zhang and Rico Sennrich. Root mean square layer normalization, 2019. URL https: //arxiv.orz www.oogg///.........oooo..oo..b. http//aas//.../////// 07467.
13

---

1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Our contributions are listed the abstract and introduction, and we provide empirical and theoretical results to substantiate the claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We provide statements for our theoretical claims in the main text while providing proofs in the Supplementary. We refer the reader to Appendix A for proofs.
14

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We share our source code via an anonymous repository, containing sufficient instructions to set up. pretrain, and evaluate our models on the appropriate dataset(s). You can find our code here. We also provide additional training details and evaluation details in Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (c.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification: We share our source code via an anonymous repository, containing sufficient instructions to set up, pretrain, and evaluate models on the appropriate dataset(s) and benchmarks. Specifically, we rely on the WikiDump dataset [14] for pretraining. Our evaluation is performed on open-source benchmarks such MMLU and ARC. You find our code here: https:
/ / thub. www.ccomm/aaaeeee /raaprraphhhhaa and- See--Geomeeiiiee www.eemtterrr-eeannngnee/aee Learning/heinn
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c ec/
public/ waaccc/ggg//////ee /ttg/gee//ooeeummiieeeeeeeeeeeee CoodeeSumissiioooooo conollcy)) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wwc./uulcc/cococccccccccccccllcl /utcc/iidde///oeuubiiiiii Codeeubmis ss messoooooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide details about our experimental setup, spanning pretraining, model hyperparameters, and evaluation, in Section 5 and Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
We provide standard deviation for results on the smaller models. As for the larger IB models. training these models is highly resource-intensive, making challenging to re-train models and provide relevant statistics. However, we follow standard protocol in evaluating our models on well-known benchmarks. Guidelines:
Justification:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe compute infrastructure in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http::pps..ppppppptteeeeeeeceeee
Answer: [Yes]
Justification: Our work solely introduces and describes methodology for large-scale language modeling, and does not violate the terms in the NeurlPS ethics guideline. We also widely-accepted, open-source libraries and datasets while including relevant citations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
17

---

Justification: This study is foundational research effort to build hyperbolic LLMs, where we focus on introducing new methods. The models are not at the size of the commercially available L.MS and were trained on safe and reliable dataset.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our pretrained models were trained on data extracted from Wikipedia, which is broadly considered reliable and safe. We believe there is low risk of misuse and do not provide any explicit safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (c.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: All assets (c.g., figures and diagrams) used in our work are original. Open source datasets and benchmarks were all properly cited where applicable.
Guidelines:
18

---

The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not provide any new assets.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work neither involves crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work neither involves crowdsourcing nor research with human subjects.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: Our work does not LLMs to create any components of either the manuscript (aside from high-level grammar checks) or the methodology itself.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (ntis:::rr/ss: /00p://////ooooreerrre///2/////0 for what should or should not be described.
20

---

Appendix
A Proofs and Details of Theoretical Results
21
A.I
Ollivier-Ricci Curvature
21
A.2 HoPE is a Lorentz Rotation
22
A.3 Proposition 4.1: HOPE is function of embeddings and relative position only
22
A.4 Proposition 4.2: HOPE decays with increasing relative position
23
A.5 Proposition 4.3: HoPE enables tokens to attend across distances
24
A.6 Proposition 4.4: Attention heads with HoPE learn special positional patterns
24
A.7 Proposition 4.5: Invariance guarantees of Hyperbolic RMSNorm
26
B Additional Details
27
B.1 MICE as a Lorentzian Module
27
B.2 Hyperbolic Multi-Head Latent Attention
27
B.3 Hyperbolic SwiGLU Feedforward Network
28
Training and Evaluation Details
28
C.I Models Setup
29
C,2 Training Details
................................
29
C,3 Evaluation Details
30
Ablation Studies and Additional Experiments
30
D.1 Runtime and memory usage analysis
30
D.2 Comparison with Prior Hyperbolic Language Models ................................
31
D.3 Ablation for HMLA
................................
D.4 Ablation for HOPE
32
D.5
Performance over Training Stages
32
A Proofs and Details of Theoretical Results
A.1 Ollivier-Rieci Curvature
Ricci curvature is a geometric object that measures average geodesic dispersion on a Riemannian manifold, i.e., whether straight paths on surface remain parallel (zero curvature), converge (positive curvature), diverge (negative curvature). Ollivier-Rieci curvature is the discrete analog for graphs that do not have notion tangent structure.
Suppose we have graph GVV For node E V, we define probability measures by:
1
(i,j) E otherwise.
(4(1)
deg(1) 0
For i,j V, the Riccci curvature is given by.
W (1)
(o(()) =
d(i.j)
21

---

Here, W is the 1- I-Wasserstein distance between measures, d4 and Aj. Intuitively, the curvature defined random walks between i and j. If the random walks tend to stay at equal distances. j) = O; if they diverge, x(i,j) < O; and if they converge, k(i, j) > 0. Since curvature is local property Riemannian surface, for graphs, we examine local neighhorhoods with a step size of 1. We thus choose (4(3) Teg(T if (i, E, otherwise 0.
In our preliminary analysis of popular decoder-only LLMs (Figure 1), we draw k-nearest-neighhbrs graphs with the final-layer token embeddings for a collection of prompts from RedPajama [46] to ascertain their geometric structure. We observe high variability of negative curvature values across tokens, hinting at their non-Eudiee -un-uucid nature. Robinson et al. [38] further allude to the non- oon-Euclid token subspace, necessitating language models that can accommodate this unique geometry.
A.2
HOPE is a Lorentz Rotation
Here, we expand on HoPE being a Lorentz rotation. Since R, a Euclidean rotation, we have ||||| |.,e Then, since for any Lorentz vector z E K,"," we have 2, = /11/K + ||zz| E IR. 1 Thus, HOPE does not
Computing (R,,æ2) --1K + [R,ll 1 K affect the time-like dimension of z, so we have,
HoPE(z) = (
0 R1
2,
making HoPE a valid Lorentz rotational operation.
A.3 Proposition 4.1: HOPE is a function of embeddings and relative position only
Proposition. Let X be T tokens with x; E K.d Let Q K be queries and keys as in Equation (3). Then dcg (HoPE (q.), HoPE (ks))) - g(xa: X6; b) for some function g.
Proof. We will denote HoPE(qa), HoPE(k,) as (Xa), (x,) where fqr f& denotes the function that projects the word embeddings to queries and keys, and then applying HOPE. In practice, the projection is done through a hyperbolic linear layer, which we take to be HLT from Section 2. It suffices to prove this proposition for the case of 2, since HOPE does not affect the time-like dimension of the inputs and R; acts independently on each 2D block. First, note that we have
(x f(
2(f (xx))
K
2 2 (x 2(fg(xx) fx(()))),
K
where (c c denotes Lorentzian inner product and (-.-) denotes the regular Euclidean inner prod- uct. Since HOPE is a Lorentz rotation, the term 2 (fg (xx)) is simply 2((qa),, (ks) = (VWax, 1/K / WKx, - 1/K). so we focus on the inner product (Sg (xa)s> s. J& (x))a). Then, by assuming that d = 2, we have (xa)). (fx E ?;;;; hence we can parametrize these vec- tors by their radial and angular components. For simplicity, denote (x,(x))).. ... (xx)), as b respectively. Then, write (a, b) as a function g'. Afterwards, we parametrize the vectors as
a (x a)e (0g(0))))
b px(xs, (x
(8)
g pge"o'
where (')) denote the radial component and 1 (g. denote the angular component. Note that suffice to show that under HOPE, we can express Poy Byg as function of the word embeddings and relative position. To see this, note that by definition of g we have
(x, a) a)px (b,, b) 49g
0, (x a) 0g (x b) 0g-
Now, given the fact that HOPE acts via Fuclidean rotation on the time-like dimension of any vector, we have (x((x,, a) (X,, a'), ipu (xs, b) (xs, 6') for any a' b'. In particular, when
22

---

a' = 8' = 0, HoPE acts via identity since all rotation angles become O. Hence, we have
Sp (Xa, a) |I(g))
(10)
9h (x.. 6) = (l
Furthermore, we have Poy = ( alll (k,) |I tion (9), which is function of just word embeddings. Next, for the angular component, note that given the definition of HoPE, the rotation on any 2D block of the space-like dimension of the
[/ [||] by plugging back into Equa-
input at position p is simply scaling of fixed rotation angle by p. Letting this fixed angle be a, the rotation is precisely po. Therefore, we have
(x,, a) =0, + a0
(11)
(xb. b) = 02 + bos
where 4g. 0y denote the angular components of qa: ky. Next, given Equation (9), we have dy' (a ()) + (0,- 0k). Note that we have eio Wax., |W and e W* xx.
Consequently, 0g
[[ x.
a function of the word embeddings and the relative position a - b. All in all, (x,) (((x)) can be expressed with function (Xx, Xb; b) as desired.
A.4 Proposition 4.2: HOPE decays with increasing relative position
Proposition. Let Q K be as defined in Equation (3), then the negative square Lorentz distance dc (HoPE (qa). HoPE(k,)) can be upper bounded by f f(g, ko)g(a - b) < O. where f has no dependencies on position, and g depends entirely on relative position and scales inversely w.r.. a b.
Proof. For simplicity, we denote qa = =q,ks = k. Recall that & (HoPF (q) (oPP(()))))))
2
2(gtke)) + 2q k,.
K
Next, for simplicity, we denote 9s, k2 as a, b respectively. We group together entries of the queries and keys, where 11] 1 as the 2k-th and the (2k + 1)-th entries of a, b respectively. With this in mind, note that since we take query and key projects to with HLT as given in Section 2, obtain Wºx, x, and b WK To that end,
assert that
a T b
(x.))
(R.sWKK xb)
x Q a,e w Xb
n/2
Re
Σ
2 20 (bb-
((0-))
where Re(x) denotes the real component
C. Now, recall Abel's Transformation, which allows one to rewrite the sum of the product of two sequences the product of the partial sums. Denote one
of the 212)22 [26,2k+1] as AR, and denote the sequence (bb-a)0 as E. With this in mind,
n/2
($) can be written as Re
A+
Ex)
Consequently, we obtain (recall that boundary
k=U
term An/2 = 0)
n/2
a bb
Σ A - E&)
k=0
n/2
Σ A A+41 ER
(Abel Transformation)
a/2
(Ae41 Ex]
k=
23

---

n//
Now, note that the term As+1 Aş has no dependency position.
The sum
Eg scales
inversely with b - a, as shown by Su et al. [40]. To that end, we have
2 2(q(k()) + t.29. K.
de (HoPE HoPE(k)
K
n/2
2 2(gcke) + maxx (anax|(An) (A++1 Ax)l
Σ
Ex|-
K
k=0
Note that As; It, hy depends on only the word embeddings and Eg depends only on the position. Thus, we have the desired result.
A.5 Proposition 4.3: HOPE enables tokens to attend across distances
Proposition. Let Q, be as defined in Equation (3), then HoPE can maximal at an arbitrary distance, i.e., for any relative distance Z, there exists a key kj such that the softmax value is maximum at distance v.
Proof. We first restate Lemma A.1. from Barbero et al. [2]. The remainder of our proof follows similar layout to that of Proposition 3.1. in Barbero et al. [2].
Lemma A.1. (Barbero al. [2]) Consider g Q with g # 0 and n E Z. Then, ng 0 (mod 2x) only when n = O. In particular, this also holds if is algebraic.
Consider a distance We can represent &b as combination of a time and space dimension the Lorentzian manifold, [1h,, t/ IL K,", Using our definition of HOPE in Section 4.1, we have
non-trivial query Q2 = v E L. as well as key K = HHPE(QQ
K, = V R...Rllll
K R2, E ELK.a
Recall that the operator R. is valid Euclidean rotation in T while HOPE remains valid Lorentzian operation. Therefore, R. does not affect the Fuclidean norm in the time dimension as it is an isometry. Instead, we can focus on the space dimension vi, Ra on which we can use the Euclidean dot product. Assume the query is at position i and the key is at some j S i. We then compute the following dot product:
spp HoPE(wa.p) = 16
(i
V
p.B
Σ
(W!))
R
(9!0))
1= m2/2
Cos ((j - r)):).
(12)
12/2
Using Lemma A.1. from Barbero et al. [2], we observe the maximum can be achieved when j j-t - - for i < 0 since using causal masking, j This ensures cos (j i r) cos (0) concluding the proof.
A.6 Proposition 4.4: Attention heads with HOPE learn special positional patterns
Proposition. Attention heads with HoPE can learn diagonal or off-diagonal attention patterns.
Proof. The proof follows similar layout as that of Proposition 5.3. from Barbero et al. [2]. We start with the diagonal case. Suppose Q = K2 = vi E T,K.d, for non-trivial v [1/, v/.). ]. We assume embedding dimension d = 2, i.., only single rotation block RG acts the embeddings.

---

Recall the squared Lorentzian distance for any a, b IL K.d
de (a b)
||a bll2
2aa, b)
K
2
2( 2 + a. bs).
K
Without HoPE,
-2, =
2 + +1.5 v6)
K
+ 2thr 21h 1/0
K
Using HoPE,
d2 R,okk)
200 2 (((,,0)5)) (R,,,
24/2 - -2( ((]_Ri-ioss)
2 K
+ 24/ -211///1 cos ((i i)0)
C [-2 cos ((j - i}01]
2/12
(13)
Using Lemma A.1. from [2], when j = i, we have (j - i)0 = 0 (mod 2r). Next, let us define a
as |C 2/1 This means cos ((j - i)0) = cos (0) 1, and a, a, This gives us the following self attenttention score:
exp (a),))
14,33 = Exes exp (a.,)) + exp (a.,))
exp [C - 211)
Exer exp exp 2/ cos ((k - i)01]) + exp (-|C 2// I2D)
1 Eker exp (-(C 2//1b/ 12 cos ((k- - (--C
1 + Ener exp (-C 2// cos ((k- i)0) + C 21/2).13)
1+ Ener exp 22 ((k i)0) 1))*
for all A i, cos ((k i)0) < This means 2||v/=1/² (cos ((k: - -))0) - 1) < 0. To that end, we obtain
sup || 000 1+ Eker exp (cos ((k: - (()())))) 1)) 1.
This guarantees V4 > 1 - for e > O, where is function of 2//
We now consider the off-diagonal pattern. Set Q s for non-trivial 1a [W,, e [.K.d Set keys K, R1,9 and define a, the off-diagonal input to the softmax when computing 14,1-1 To
25

---

that end, we have
a d2 0 R, 1,,
2 K
+ 2/// 2 ((R (R (Ri-1.oKi))
20/2 -2(( (R
20/ -2 (((( (R (Re,
20/ 2111/1 cos ((i ((i- ((1=)))))
2 K
+ 24? 211
Use the same reasoning from the diagonal case to show that attention head with HoPE can learn off-diagonal patterns. This concludes the proof.
A.7 Proposition 4.5: Invariance guarantees of Hyperbolic RMSNorm
Proposition. RMSNorme invariant scaling of inputs during both the forward and backward passes.
Proof. Euclidean RMSNorm is invariant input-scaling, both during the forward and backward pass. We observe similar guarantees from our formulation of hyperbolic RMSNorm. To that end, we first prove the input-scaling invariance of Euclidean RMSNorm.
Given an input x E Rn and and feed-forward network with parameters R
W RMS( x) Og+b)
m
y =
RMS(a)
m
Here, g is learnable gain parameter, initially set to 1, that re-scales the standardized inputs and is a bias term.
Suppose the weights are scaled by a small factor, W = sW. First, observe that the root mean squared operation, RMS, is input-scaling invariant: RMS(oa) aRMS(a).. It then evident that the final output of RMSNorm is also ocale-invariant:
(W) X
Og+b
RMS((W') x)
SW
Og+h
RMS(OW x)
SWTx x SRMS(WTxX x) Og+b
=y
(14)
A similar argument can be made for scaling of the inputs x. Since hyperbolic RMSNorm uses Fuclidean RMSNorm internally, it offers the same invariance guarantees it operates solely on the space dimension of the Lorentzian input.
26

---

Given an input [2g, X,} E L Kin, K,", we know RMSNorm (x) RMSNorm(x) for some scaling factor 8. As such,
y' = RMMNNrrm_(xx)
[IRRMSNorm((xx) 1/K, RMSNNrr(Oxx.
VIIIMSNorm(x.) (x,)l 1/K, RMSNorm(x.) =
Next, analyze the gradient stability of hyperbolic RMSNNrm. In Euclidean RMSNorm, for given loss L, we are interested in computing three gradients: og bias, and OW for the weights. We compute ög Øb as follows:
AL for the ØL for the gain parameter, 7b
aL and 0L
OL aL 8v ōb ōv öb
0L aL
W X
0g
RMS(W X)
where v denotes the inputs to the activation ø. These gradients are invariant to the scaling of Euclidean inputs x and weights W, trivially for Ø5 and due to the linearity established in Equation (14) for aL Computing SW ōg provides invariance to input scaling as shown by Zhang and Sennrich [51].
aL is more involved due to the quadratic computation in RMS, but also
Given an input = [r, x. K, we know ög 0g and 9W are ocaling-invariant the backward pass since hyperbolic RMSNorm uses Euclidean RMSNorm. Thus, for any scaled hyperbolic input ōx E L K.n we get scaling invariance both in the time and space dimension during the backward pass.
B
Additional Details
B.1 MICE as Lorentzian Module
In this section, we expand on the fact that MICE is indeed a valid hyperbolic module throughout. Note that since Equation (6) consists of the combination of Lorentzian residual connection [23] and Lorentzian centroid [29], it suffices to show that the projection from input manifold to expert manifold, and the reverse projection, fact, it suffices to show that given E L Ki.n we have v /KıKK2 E L Ka.", To see this, note that
valid projections between Lorentz hyperbolic spaces. In
K1
VKn/K2x K2/K2x)
(x,
x)x
K2
K1
1
(x Lim.))
K₂
K1
K₂
Thus, / Lam as desired. Then, each projection via scaling by / KK/K,, and VK/Kr, indeed map the input vector x to the expert manifold, and the projection via / Kg,i/ / and /K,a// K maps the output of the experts back to the input manifold. As a result, every vector in Equation (6) lives on the input manifold, hence the output lives in K.n as desired.
Additionally, note that since the squared Lorentzian distance is given by dx = 2/K 2/K + 20.y/ 2x, ys, it scales inversely w.r.t. x_ y.. As a result, the gating score obtained through Equation (5) is minimizing the the squared hyperbolic distance between the input token vector x, and the vector y, (by viewing centroid vector y, as the space-like dimension of Lorentz hyperbolic vector). Therefore. the gating module in fact hyperbolic module as well.
B.2 Hyperbolic Multi-Heau Latent Attention
In this section, we provide the details for hyperbolic Multi-Head Latent Attention (HMLA). Let X, L K,nh, be the t-token, where is the embedding dimension and h, is the number of heads. Let DKV +)x be the downward projection matrix of the keys and values, and
27

---

w E [(nnk+1)x*, be the downward projection matrix of the query (Mker Mg & nha). We first compress the token into the latent spaces via e HX;; WKV bkV) LLKmko of (x; WQ, E LL We then project the latent query, key, and value vectors back to the higher dimensional spaces. Specifically, let WUV WUK E [ be the upward projection matrix of the keys and values, and let wUQ E 1(nx+))xnh, be the upward projection matrix the query. Then, the final projected keys, values, and queries
[K -.. L 19. Ii.hm!
HLT LV WUK buk): (.I..... - HLI (cKV; WUV bbv)
HLT cf WUQ buq).
(15)
Following previous works [11, 10), as RoPE is incompatible with MLA due to position coupling, we employ decoupled HoPE scheme with HMLA, where we use additional query vectors with shared key. Let WQR E (1 and WKR [ be the upward projection matrix of the decoupled queries and the shared key respectively, where n, is the dimension per head. We apply HoPE to these vectors obtain the resition-encodee vectors
[q:1 ........ :..
HoPE (HLT (c:: W bQR))
k HoPE (HLT (c,; Wk bkR)).
(16)
Then, we obtain the final query and key vectors
d HCat(q : kt, HCat(k(,;
(17)
where HCat denotes hyperbolic concatenation [48, 37J. The attention score is computed based on negative squared Lorentz distance similar to Equation (3)
Σj ani V
exp (- (q:, kt,a)) hn + n4y)
Ek=1 exp (-d2 (qr, k )/ /hn n,) (18)
V- KIII Ex C E,f llle
The final output of HMLA can be expressed as the concatenation of the hyperbolic vector
HMLA X,3 hns n, nry Bgy nkw)) HIT ([Vloul 1/K (//Ko))-1/KKo.)
w bo
(19)
where o, = [or,1..... or and w E (*+((+))xx;; is the out-projeet matrix. HMLA enables HELM models to improve computational efficiency during training and inference compared the regular hyperbolic self-attention in Equation (3).
B.3 Hyperbolic SwiGLU Feedforward Network
In this section, we introduce hyperbolic SwiGLU feedforward networks (FFNs), whose Euclidean formulation is widely used LLMs [18, 10]. This differs from previous FNNs used in hyperbolic Transformers in the need for feature multiplication and activation function [5, 6, 48]. Let IL K,n be the input tokens, W1, W3 E ()++1)xm be the weights of internal projection layers and W E (mm+1)xx be the weight matrix of the outward projection layer. Then, the hyperbolic SwiGLU FNN HFFN IL R. L K.n given by
HFNN sa (x HLT
.2,, b2
(20)
y (HLTx( W,, b,)) 8, HLT(x; W by).
where S denotes SiLU activation using the HRC activation operations from Hypformer [48] and G denotes multiplication on the space dimention of a Lorentz vector, i.e. S, y = x,ys-
C Training and Evaluation Details
In this section we detail the training and evaluation setup for the experiments.
28

---

C.1 Models Setup
Here detail the model setup for all the models we used in the experiments.
HELM-MICE model setup. We follow the notation in Appendix B.2 and Section 4.2. For HELM- MICE, we used HMLA as the attention mechanism in the attention block, MICE as the sparse feedforward network, and HFNN sG as the dense feedforward network. For both sizes, only the first decoder block uses the dense layer and the rest of the blocks using the MICE layer. The MICE layers use HFNN sg as well for its feedforward component. For the dense layer, we set the intermediate dimension to be n2, For the MICE layers, set the intermediate dimension of HFNNsG as n.
For the ~ 100M sized model, we used 6 total layers. 6 heads (n 6) each with n = 64, and we set Akke 64, nr = 16. For MICE layers, we employ 4 experts with 2 active experts per token (N, = 4, K, = 2). We use one shared expert (N, = 1). For the curvatures of the routed experts, we set them to uniform from -0.1 to 2.0. The curvature of the shared expert set to be -1. The curvature of the entire model is set to -1 well. For HMLA layers, in practice, the upward projection matrices do not need to project back to the full dimension of h, Due to compute constraints, we instead employ reduction in dimensionality during the upward projection, where WUK, WUV E (g(ns++1)xx dimensionality of the input with W E (x/2+1) x
and the outward projection matrix Wo projects back to the full
For the 1B sized model, we use 16 total layers, 14 heads (n,, 14) each with 64, and we set fg: = 256, n, = 64. For MICE layers, we employ 4 experts with 2 active experts per token (N, 8, K, = 2). We use one shared expert (N,- = 1). For the curvatures of the routed experts, set them to be uniform from -0.1 to 2.0. The curvature of the shared expert is set to be -1 The curvature of the entire model is set to -1 as well. We do not use the same reduction in dimensionality during upward projection as we did in the ~ 100M case to enable for more expressive attention modules.
HELM-D model setup. For the HELM-D model, we only train the 100M sized model. Here, we use 6 layers, heads each with dimension 64, and we set the intermediate dimension of the HFNN sg feedforward networks to be 4 times the total model dimension. We set the overall curvature of the model to 1. All hyperbolic models are built on top of HyperCore He et al. [22].
Baseline models setup. For the baseline models, we set them up to have identical dimensionality as the HELM models. In particular, for the LLaMA model we train, we use the same number of layers, heads, and dimensionality per head as the feedforward network. For the DeepSeek models we train, we use the same number of layers, heads, dimensionality per head, dimensionality for the feedforward network, dimensionality for the MoE modules, number of routed and shared experts, and the same dimensionality in the MLA layers.
Hyperbolic work embeddings. For the smaller HELM models, we map the input tokens directly to Lorentz. hyperbolic vectors, which are then trained as hyperbolic parameters via Riemannian optimizers. The parameters are initialized via wrapped Gaussian normal distribution on the manifold. However, when training the ~ 1B HELM-MICE model, we found this to cause training instability. As a result, for the larger model, we first map the tokens to the space-like dimension of Lorentz hyperbolic vectors, and then compute the time-like dimension of the vectors afterwards. We found this to stabilize model training.
C.2 Training Details
Dataset. For the training dataset, we use the English portion of the Wikipedia dataset [14]. This dataset consists of ~ 6.4M rows of data. We download the dataset directly from Huggingface. The raw text data is then passed through the LLaMA3.1-88 tokenizer [18], which has a vocabulary size of ~ 128K. We use sequence length of 2048 for all models. Samples longer than 2048 tokens were broken up into multiple samples, with the trailing tailed dropped. The tokenized dataset consist of roughly 4.5B ~ 5B tokens. For training efficiency, as we measured the average number of tokens per sample is ~ 700 across the dataset, we used sample packing with packing ratio of 3.0. Then packed samples shorted than 2048 tokens are then padded on the right.
29

---

Table 4: Runtime and peak memory usage per iteration comparison between HELM variants and Euclidean counterparts. HELM models are within 1.55X in runtime and 1.11X in memory usage of the Euclidean counterparts for both the 100M and 1B models.
Model
# Params
Runtime
Memory Usage
ILaMA
100M
8.4s
23.8 GB
HELM-D
100M
13.1s
24.9 GB
D
100M
11.0s
23.5 GB
HELM-MICE
100M
16.9s
26.3 GB
DEEPSEEKV3
83.5s
33.1 GB
HELM-MICE
IB
119.4s
35.8 GB
Table 5: Performance comparison between a 115M Hypformer model and HELM-D, D, where HELM- D consistently outperforms the baseline.
Model
Params
CommonsenseQA 0-Shot
HelluSwug 0-Shot
OpenbookQA 0-Shot
MMLU 5-Shot
ARC-Challenging 5-Shol
Avg
HYPFORMER
115M
19.4
25.1 25.9 0.1
26.6 27.1 + 0.4
22.9 25.6 0.2
23.6 21.4 + U.3
23.5 24.1 + 0.1
HELM-D
115M
20.3 + 0.2
Pipeline setup. For training, we set up data data-paralles parallelilll with Hugginface Accelerate. We use an effective batch size ~ 2M tokens (including padding). To ensure a fair comparison between the hyperbolic and Euclidean models, we use a learning rate of 2e-4 for all dense models and learning rate 4e-4 for the MoE and MICE models. weight decay rate 0.01 was used for all models. For the HELM- M models and the DeepSeck models, in order to balance the load between each expert, we utilize the auxiliary-loss-ree load balancing strategy and the complementary sequence- eequence-wise wise auxiliary loss during training. The former punishes extreme load imbalance among the experts by dynamically updating a bias term during the gating module, while not needing an explicit auxiliary loss computation for better training efficiency. The latter punishes extreme load imbalance for any particular sequence. All training used a cosine annealing learning rate scheduler with final target learning rate of 0.1x the initial learning rate, with 3% of the gradient update steps used as warmup steps.
Runtime. We empirically observe that HELM models take roughly 1.5 to 1.8 times the training of their Euclidcan counterparts. For example, the larger 1B HELM-MICH model takes roughly 72 hours to train on NVIDIA A800s while the similarly sized DeepSeek model takes roughly 40 hours on the same machine.
C.3 Evaluation Details
We use the Language Model Evaluation Harness library (github. www.//eeetherrAAiinevaaaaaooooo harness) for all evaluations, where the framework prompts the models with the answers choices to each question and picks the one with the highest likelihood value. For OpenbookQA, we convert the answer choices from full sentences to letter choices for all models, to make up for the relatively smaller model and training dataset sizes.
D Ablation Studies and Additional Experiments
In this section, we perform additional experiments such as computational cost analysis, ablation studies to access the effecttivnness of HoPE and HMLA, and comparisons with prior works of hyperbolic language models.
D.1 Runtime and memory usage analysis
While HELM inevitably introduces computational overhead from operations that respect the curvature of the embedding space, our proposed methods are efficient in both runtime and memory usage. HELM models are within 1.55X in runtime and 1.1 memory usage of the Fuclidean counterparts for both the 100M and IB models. In Table 4, we show that runtime and peak memory usage
30

---

Table 7: Ablation accuracy, where we compare HELM- HELM-MICE with a variant using hyperbolic Multi- Head self- AuffAAttetiion instead of HMLA. denoted as MICE- HICE-HMHA. Bolding denotes the highest accuracy and underline denotes the second-highest. Fuclidean DEEPSEFK V3 results are shown for reference. Overall, MECCE consistently achieves the highest accuracy, while both hyperbolic models still outperform the Euclidean counterpart.
Model
CummonsenseQA
HellaSwag 0-Shot
OpenbookQA 0-Shot
MMLU 5-Shot
ARC Challlalggg 5-Shot
Avg
0-Shot
VEEKS 120M
19.2
25.2
23.1
21.2
21.8
22,8
MICE- HMHA
120M
19.3
26.0
23.8
25.3
23.7
iexibHHELM-MCC
120M
19.7
25.9 0.2
27.7=
21.1
23.2 = 0.5
24.1 0.1
Table 8: Ablation accuracy, where we compare HELM with variants using learned relative positional encoding instead of HoPE, denoted as HELM-D-L and HELM-MICE-L. Bolding denotes the highest accuracy and underline denotes the second-highest. Euclidean results are shown for reference. Overall, HELM-MICU and HELM-D consistently achieves the higher accuracy, while both hyperbolic models still outperform the Euclidean counterpart.
Model
# Params
CommonsenseQA 0-Shot
HellaSwag 0-Shot
OpenbookQA 0-Shot
MMLU 5-Shot
ARC-Challenging 5-Shot
Avg
LLAMA
115M
21.1
25.3
25.3
23.8
21.0
23.3
HELM-D-L
115M
19.7
25.5
28.6
23.0
21.8
23.7
HELM-D
115M
20.3 0.2
25. + 0.1
27.1 = 0.4
25.6 = 0.2
2.3
24.1 24.1:00 0.1
DEEPSEEK V3
120M
19.2
25.2
25.4
24.2
21.8
22.8
HELM-
120M
19.0
25.5
27.0
23.0
25.7
24.0
HELM-MICE
120M
19.7 = 0.3
25.9 + 0.2
27.7 = 0.4
24.4 = 0.2
23.2 4: 0.5
24.1 # 0.1
comparisons between HELM and the Euclidean baselines for one training iteration (roughly 2M tokens). The results shown are for our exact experimental setup ran on A100 GPUs, where we show the worst runtime and memory usage across the ranks. The results are averaged over 10 runs, and standard deviations are not shown since they are within 0.2 of the results.
D.2 Comparison with Prior Hyperbolic Language Models
We performance additional experiments to com- pare the HELM architecture against prior works of hyperbolic language models and Transformers. As the majority of prior hyperbolic Transformers lacked essential components common in modern LLMs such as LayerNorm or positional encoding, we train a small version of HELM-D and com- pare the performance against these models in the machine translation task. The setup of our exper- iment is identical that of Chen al. [5]. Hyp- former Yang et al. [48], on the other hand, does possess all components, As result, we compare the performance of HELM-D with Hypformer by training a 100M model the same setup.
Table 6: Machine translation BLEU score.
HELM-D outperforms prior works of hyper- bolic Transformers.
Model
IWSLT'14
WM1'14
HAT Gulcehre et al. [19]
23.7
21.8
HNN+ Shimizu al. [39]
22.0
25.5
HyyoNett Chen al.
25.9
26.2
HELM-D
26.3
26.5
D.3 Ablation for HMLA
Past works have found that the Euclidean case, Multi-Head Latent Attention can achieve comparable and in some cases even superior performance compared to regular Multi-Head Attention [10]. Here we assess the effectiveness of HMLA against hyperbolic Multi-Head celf-Attention. We train HELM-MICE with the same setup. where we replace the HMLA layers with a hyperbolic Multi- Head self-Attention layer as given in Equation (3). We denote the this model as MICE- MICE-HMHA. HMHA. The results are shown in Table 7. HELM-MICH outperforms achieving the same accuracy for task, with the MICE-HMHA achieving better accuracy in the last task. The results demonstrate the effectiveness of HELM-MIC' while significantly reducing the memory footprint of the KV-cchhe. Both hyperbolic models still outperform the Fuclidean model, demonstrating the effectiveness of HELM in general.
in 3 out of the 5 tasks,
31

---

Table Multi-choice answer aceuracy for HELM-MICE 1B and DeepSeek 1B across training
stages. HELM-MEN M-MMMEE demonstrates consistent improvement over its Euclidean counter part.
Model
Token Count
CommonsenseeQ
HellaSwag
OpenbonkQA
MMLU
ARC-Challenging
Avg
D (1B)
4B
18.8
25.1
26.4
23.5
22.1
23.2
HELM-MiCE
27.0
V (IB)
4.58
19.0
26.2
27.2
23.6
22.6
23.7
MELM--MCCE (IB)
4.5B
19.7
26.4
27.6
25.5
23.
24.5
DeepscekVS (IB)
5B
19.3
26.2
27.4
23.6
22.7
23.9
HELM-MiCE (IB)
5B
19.8
26.5
28.4
25.9
23.7
21.9
D.4
Ablation for HOPE
In this section we assess the effectiveness of HoPE against other hyperbolic positional encoding methods, namely the learned relative positional encoding from Hypformer [48]. We devise a variant of HELM-D, denoted as HELM-D-L.
variant HELM-MICE, denoted HELM-MICE-L, where each model uses the learned positional encoding instead of HOPE. The results are shown Table 8. Overall both HELMM MEE.M-MICE and HELM-- outperform their counterparts that use learned positional encoding instead of HOPE. Interestly, however, HELM-MICED and HELM-D- outperformed HELM-MICE and HELM-D respectively on the ERC-Challenging benchmark, possibly due to better alignment with reasoning prompts with non-uniform encodings. Nevertheless, the results demonstrate the effectiveness HoPE over learned positional encodings in 4 out of the tasks.
D.5 Performance over Training Stages
While we don't provide standard deviation from multiple runs for the 1B models is typical for model of this size, provide the model performance of HELM-MICE and DeepSeek IB in Table 9. HELM-MICE demonstrates consistent improvement over its Euclidean counter part, suggesting this improvement could sustain when the training corpus scales.
32


Reviews:
[{"review_id": "0Pd4JOzxl3", "reviewer": "Reviewer_1cTB", "summary": "The paper introduces HELM, a family of fully hyperbolic large language models that re-implements every key Transformer component—positional encoding (HOPE), normalization (Hyperbolic RMSNorm), attention (HMLA), and feed-forward blocks—inside Lorentz space. It further proposes Mixture-of-Curvature Experts (MICE) so different experts operate at distinct curvatures, and scales these ideas to 100M – 1B parameters trained on 5B Wikipedia tokens. Across five reasoning benchmarks (MMLU, ARC-Chal., CommonsenseQA, HellaSwag, OpenBookQA) HELM-MICE beats matched Euclidean baselines and shows the largest gains on the hardest tasks.", "questions": "1. Are each expert’s curvature values fixed or learnable, and how sensitive are results to that choice?\n2. Do you expect the hyperbolic–Euclidean gap to widen, stay flat, or shrink when trained on ≥100 B tokens? And what stability issues could emerge at that scale?\n3. Could you provide low-dimensional projections (e.g., Poincaré-disk plots) of the learned embeddings to show how curvature and expert routing organize semantic structure?\n4. What motivated using the Lorentz representation over alternatives like the Poincaré ball or Klein model, and did you observe concrete numerical or training-stability advantages?", "limitations": "Yes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper tackles the mismatch between language’s hierarchical structure and Euclidean embeddings by operating wholly in hyperbolic space and letting different experts learn at different curvatures.\n2. HOPE, Hyperbolic RMSNorm, and Lorentz-residual connections give HELM feature-parity with modern LLMs, while accompanying proofs ensure these modules retain the stability and relative-position guarantees of their Euclidean counterparts.\n3. By combining HMLA with MICE, the authors train the first billion-parameter fully hyperbolic LLM; the models consistently top equally-sized Euclidean (dense and MoE) baselines, especially on MMLU and ARC-Challenging, demonstrating that the geometric shift yields real, if modest, gains.\n4. The paper is well-written and easy to follow.\n\n## Weaknesses\n\n1. All models are trained on only 5B tokens, so absolute accuracy remains low and  improvements could shrink or vanish at larger data scales—making it unclear whether hyperbolic benefits persist in realistic pre-training regimes.\n2. HELM introduces non-trivial engineering overhead (Lorentz ops, curvature routing, expert load-balancing), yet the paper gives no timing or memory benchmarks against standard implementations, leaving practitioners uncertain about the real compute-efficiency trade-offs.\n3. The paper does not provide error analysis, case studies, or interpretive examples to show how hyperbolic geometry changes model behavior or what kinds of errors it corrects", "was_revised": true, "final_justification": "I am satisfied with the authors' response. The efficiency aspect is addressed and the interpretability examples are convincing. I still have some doubts in the scaling behavior of this model. But I think based on compute budget and current results, the authors has shown the best they could achieve. I would recommend the acceptance of the paper.", "rebuttal": "Thanks for your valuable comments! We’ll denote the reviewer's concerns with Q and our responses with R.\n\n**Q:** HELM introduces non-trivial engineering overhead (Lorentz ops, curvature routing, expert load-balancing), yet the paper gives no timing or memory benchmarks against standard implementations, leaving practitioners uncertain about the real compute-efficiency trade-offs.\n\n**R:** While HELM inevitably introduces computational overhead from operations that respect the curvature of the embedding space, our proposed methods are efficient in both runtime and memory usage. HELM models are within 1.55X in runtime and 1.11X in memory usage of the Euclidean counterparts for both the 100M and 1B models. In the table below, we show that runtime and memory comparisons between HELM and the Euclidean baselines for one training iteration (roughly 2M tokens). The results shown are for our exact experimental setup ran on 4 A100 GPUs, where we show the worst runtime and memory usage across the ranks. The results are averaged over 10 runs, and standard deviations are not shown since they are within 0.2 of the results. We will include the results in the revision. \n\n| Model | Runtime  | Memory Usage |\n| --- | --- | --- |\n| LLaMA 100M | 8.4s | 23.8Gb |\n| HELM-D 100M | 13.1s | 24.9GB |\n| DeepseekV3 100M | 11.0s | 23.5Gb |\n| HELM-MiCE 100M | 16.9s | 26.3Gb |\n| DeepseekV3 1B | 83.5s | 33.1Gb |\n| HELM-MiCE 1B | 119.4s | 35.8Gb |\n\nAdditionally, there has been very little effort to develop optimized tools and libraries for hyperbolic foundation models [23]. In contrast, there is an abundance of the same for Euclidean models (e.g., LLM-Foundry, Flash Attention, DeepSpeed). Development of comparable hyperbolic libraries would further reduce the gap between HELM models and Euclidean baselines.\n\nFinally, while there is overhead for models of the **same size**, non-Euclidean models require **fewer dimensions** to embed complex structures when compared to Euclidean models [a,b]. This shows promise for non-Euclidean models to match the performance of Euclidean models with **fewer parameters** to build parameter-efficient foundation models despite the additional overhead. There is also potential to follow scaling law relationships between parameters and model performance. On the other hand, Euclidean models face heavy dimension-distortion tradeoffs [c].\n\n**Q:** The paper does not provide error analysis, case studies, or interpretive examples to show how hyperbolic geometry changes model behavior or what kinds of errors it corrects. Could you provide low-dimensional projections (e.g., Poincaré-disk plots) of the learned embeddings to show how curvature and expert routing organize semantic structure? \n\n**R:** Prior works of hyperbolic language and vision models have found that hyperbolic models better learn the hierarchies in semantic structures [13, 47, 35], where more generic words/concepts tend to cluster in areas of smaller norm and more specific words/concepts tend to have larger norms. Our investigation of final-layer embedding distributions has found that HELM learns representations in a similar way. Below are examples of how the 1B HELM-MiCE model organizes semantics as compared to the 1B DeepseekV3 model:\n\n|  | HELM-MiCE | HELM-MiCE | DeepseekV3 | DeepseekV3 |\n| --- | --- | --- | --- | --- |\n| Words | Average Norm | Range | Average Norm | Range |\n| to, in, have, that, and, is, for | 35.930 | 35.890~35.951 | 33.725 | 33.660~33.800 |\n| study, research, subject, papers, category | 36.080 | 36.030~36.033 | 33.735 | 33.668~33.776 |\n| biology, physics, chemistry, mathematics, computer science | 36.155 | 36.033~36.270 | 33.720 | 33.658~33.776 |\n| algebra, geometry, photosynthesis, cellular respiration, genetics | 36.288 | 36.133~36.484 | 33.741 | 33.622~33.826 |\n\nFor HELM-MiCE, more generic words (e.g., subject) are clustered closer to the origin than more specific words (e.g., biology), which have a smaller norm than even more specific words (e.g., photosynthesis). However, this does not necessarily hold for the DeepseekV3 1B model, demonstrating how HELM-MiCE better handles semantic hierarchies. We will provide low-dimensional visualizations of these embeddings in our revision.\n\nLet us now consider tokens from questions from MMLU that HELM-MiCE correctly answers but DeepseekV3 gets wrong. We examine their final embedding norms:\n\n| **HELM-MiCE** | **HELM-MiCE** | **DeepseekV3** | **DeepseekV3** |\n| --- | --- | --- | --- |\n| Words | Norm Range | Words | Norm Range |\n| A, How, does, if, there, have, is, any, with, of | 36.031~36.396 | is, a, connecting, graph, there, edges, complete, have, of,  | 33.668~33.768 |\n| discrete, vertices, edges, connecting, pair, graph, complete, many , 10 | 36.506~36.717 | discrete, 10, how, if, pair, does, with, A, vertices, any | 33.772~33.908 |\n\nFor HELM-MiCE, general words have a smaller norm (e.g., how, if) while the more specific “graph theory” keywords are further away from the origin (e.g., connecting, graph). However, for DeepseekV3, general and specific words are sometimes intertwined in the embedding space. The hierarchical organization of the space enables the HELM models to sometimes better navigate the embedding space and obtain better performance.\n\n**Q:** Are each expert’s curvature values fixed or learnable, and how sensitive are results to that choice?\n\n**R:** For our experiments with MiCE, the experts’ curvatures are trainable and initiated uniformly from -0.1 to -2.0 (e.g., with 4 experts, the curvature values were -0.1, -0.733, -1.367, -2.0). We did not observe the curvature to converge to values that were far from their initial values (within $\\pm0.1$ of the initial values). This is presumably due to the wide range of initial curvatures, which we have selected to better capture the geometric variation from token to token. Our ablations in Table 2 demonstrate the effectiveness of variable curvatures for each expert.\n\n**Q:** All models are trained on only 5B tokens...do you expect the hyperbolic–Euclidean gap to widen, stay flat, or shrink when trained on ≥100 B tokens? And what stability issues could emerge at that scale?\n\n**R:**  The results for the 1B models from checkpoints taken at 4B and 5B tokens demonstrate that HELM-MiCE consistently outperforms the Euclidean baselines, as shown below:\n\n|  | Token Count | CommonsenseQA | HellaSwag | OpenbookQA | MMLU | ARC-Challenging | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| DeepseekV3 (1B) | 4B | 18.8 | 25.1 | 26.4 | 23.5 | 22.1 | 23.2 |\n| **HELM-MiCE (1B)** | **4B** | **19.5** | **26.0** | **27.0** | **25.6** | **22.9** | **24.3** |\n| DeepseekV3 (1B) | 4.5B | 19.0 | 26.2 | 27.2 | 23.6 | 22.6 | 23.7 |\n| **HELM-MiCE (1B)** | **4.5B** | **19.7** | **26.4** | **27.6** | **25.5** | **23.1** | **24.5** |\n| DeepseekV3 (1B) | 5B | 19.5 | 26.2 | 27.4 | 23.6 | 22.7 | 23.9 |\n| **HELM-MiCE (1B)** | **5B** | **19.8** | **26.5** | **28.4** | **25.9** | **23.7** | **24.9** |\n\nAs we scale up data to 5B, the gap between HELM and Euclidean baselines did not show any behavior or signs of diminishing, and the outperformance remained consistent. While one can only fully answer the reviewer’s question by increasing the training corpus to 100B tokens, these results give us more confidence in HELM to outperform the Euclidean baselines at larger scales. Our work presents a stable foundation to achieve training hyperbolic LLMs at such scales, paving the way for future explorations. It is worth noting that Euclidean space will always incur significant distortion when embedding tree-like data regardless of dimensionality and token count [b], while hyperbolic space can achieve arbitrarily low distortion [a]. Thus, there is a theoretical upper bound on the performance of the Euclidean baselines at larger scales, which is not present for HELM models.\n\nIn regard to stability, we have employed several techniques to ensure training stability and stable convergence of the losses. Notably, as described in section 5, our modified hyperbolic token embedding method – where the embeddings are learned along the space dimension – improves stability over prior methods. \n\n**Q:** What motivated using the Lorentz representation over alternatives like the Poincaré ball or Klein model, and did you observe concrete numerical or training-stability advantages?\n\n**R:** The Lorentz hyperboloid comes with a **wide availability of fully hyperbolic operations in the Lorentz space**. These are hyperbolic operations that are performed directly on the manifold without requiring repeated mapping to and from the tangent space, and are not generally available in the Poincaré ball or Klein model. Many prior works have found that these Lorentz operations have several advantages, including better computational efficiency, higher expressiveness, and fewer mapping errors [24, 48, 6, 4]. These prior works have found that the fully hyperbolic operations concretely improve numerical stability, whereas tangent space mappings can result in NaN/overflow [24, 4]. Our models are built on top of these efforts and operations (e.g., fully hyperbolic linear layer and residual connection).\n\nOur proposed methods also take advantage of properties that are specific to the Lorentz space, such as formulating of HoPE based on Lorentzian rotations and the interactions between the time and space dimensions. These properties enable us to prove theoretical guarantees for HoPE and hyperbolic RMSNorm that are comparable to the Euclidean counterparts.\n\n***References***: \n[a] Sarkar, R. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pp. 355–366. Springer, 2011.\n\n[b] Lee, J. R., Naor, A., and Peres, Y. Trees and markov convexity, 2007.\n\n[c] Matousek, J. Lectures on Discrete Geometry. Springer, 2002."}, {"review_id": "qTbILEr5TB", "reviewer": "Reviewer_2Div", "summary": "This paper introduces a new family of Large Language Models (i.e. HELM) that can operate entirely in hyperbolic space. In particular, the authors propose architectural interventions to obtain a set of hyperbolic transformer models, including a scalable hyperbolic mixture-of-curvature expert model where each expert operates using a different curvature to better leverage the geometric structure of the space. The authors show that their architectural interventions allow training fully hyperbolic LLMs of up to a billion parameters, demonstrating competitive performance with Euclidean counterparts on different downstream tasks.", "questions": "- How do you ensure that the reported results and improvements are statistically significant? \n- Could you provide qualitative examples of how your model improves upon Euclidean ones? In particular, how does HELM capture and leverage hierarchical information to deal with the natural language tasks you evaluate on?\n- How does HELM empirically compare with previous hyperbolic transformers on different tasks? And why were previous hyperbolic approaches not considered in the experiments?", "limitations": "No. The limitations are only briefly discussed in section 6 and should be better elicited in the experimental design through an in-depth error analysis and qualitative evaluation (please see weaknesses).", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Reason to accept:**\n- The paper is generally well-written and easy to follow. The motivation for exploring hyperbolic representations for LLMs is clearly articulated, as well as its potential impact.\n- The paper performs an important technical and architectural contribution, introducing novel components for building LLMs operating in hyperbolic space and introducing a new model (i.e., HELM). This is an important contribution that can support future research in the field.\n- Experiments performed on different natural language tasks demonstrate that HELM is competitive with existing LLMs of the same size operating in Euclidean space.\n\n**Reason to reject:**\n\nWhile I find the architectural contribution compelling, I believe the experiments are currently insufficient for acceptance. Further experiments and analyses are required to validate the impact of their contribution. Specifically:\n\n- The difference in performance with Euclidean models is minimal and not supported by statistical significance analyses, which, I think, given the relatively small difference in performance, is necessary. \n- While the authors perform a comparison with Euclidean models of the same size, there is a missing comparison with previous Hyperbolic transformers such as HAT, HNN++, and HyboNet, HypFormer. I believe such a comparison is necessary (even considering different tasks and setups, such as relation prediction) to fully validate the quality of their contribution.\n- Beyond quantitative comparison with Euclidean models, I believe the paper should include a more in-depth qualitative/error analysis to understand the real impact and limitations of the hyperbolic representation and the introduced architecture. One of the expected impacts of hyperbolic space should be an improved ability to capture hierarchical relationships. However, from the current empirical evaluation, this is not at all clear.", "was_revised": true, "final_justification": "The authors have partially addressed some of the concerns in the original review, providing additional results expanding the breadth and scope of the evaluation. \n\nHowever, the concerns regarding the marginal improvement over existing baselines remain.", "rebuttal": "Thanks for your valuable comments! We’ll denote the reviewer's concerns with Q and our responses with R.\n\nWe appreciate that Reviewer 2Div finds our method novel and compelling. The concerns raised are mostly due to reservations about the significance of our experimental results. We focus our rebuttal below on solidifying our experiments and including the reviewer’s feedback in our new experimental results. \n\n**Q:** The difference in performance with Euclidean models is minimal and not supported by statistical significance analyses, which, I think, given the relatively small difference in performance, is necessary. How do you ensure that the reported results and improvements are statistically significant?\n\n**R:** We appreciate the reviewer raising this important point. To further strengthen our results, we performed additional training runs for the DeepseekV3 and HELM-MiCE at the 100M, where the statistics are computed over 3 runs as shown below:\n\n|  | CommonsenseQA   | HellaSwag  | OpenbookQA  | MMLU  | ARC-Challenging | Avg |\n| --- | --- | --- | --- | --- | --- | --- |\n| DeepseekV3 (100M) | $19.3\\pm0.2$ | $25.3\\pm 0.1$ | $24.0\\pm0.4$ | $23.9\\pm0.3$ | $22.2\\pm0.3$ | $22.9\\pm0.1$ |\n| HELM-MiCE (100M) | $19.7\\pm0.3$ | $25.9\\pm0.2$ | $27.7\\pm0.4$ | $24.4\\pm0.2$ | $23.2\\pm0.5$ | $24.1\\pm0.1$ |\n\nThe results show that HELM-MiCE outperforms the Euclidean DeepseekV3 model with statistical significance. We will include these results, along with similar statistics for the 100M dense models, in the revision. \n\nAs for going up to the 1B scale parameters, performing multiple training runs is rather compute-heavy, requiring thousands of GPU hours for multiple training sessions of both our model *and* the baselines. As performing multiple runs at the scale of billions of parameters becomes extremely expensive, it is typically evaluated for a single run across prior literature, even those that were previously accepted by NeurIPS[49,2], popular open LLMs[19, 11], and LLMs of similar size[52]. To showcase the significance and consistency of the results, though, we provide the results at different checkpoints to showcase how the performance gap is consistent between the two and remains beneficial to our framework:\n\n|  | Token Count | CommonsenseQA | HellaSwag | OpenbookQA | MMLU | ARC-Challenging | Avg |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| DeepseekV3 (1B) | 4B | 18.8 | 25.1 | 26.4 | 23.5 | 22.1 | 23.2 |\n| **HELM-MiCE (1B)** | **4B** | **19.5** | **26.0** | **27.0** | **25.6** | **22.9** | **24.3** |\n| DeepseekV3 (1B) | 4.5B | 19.0 | 26.2 | 27.2 | 23.6 | 22.6 | 23.7 |\n| **HELM-MiCE (1B)** | **4.5B** | **19.7** | **26.4** | **27.6** | **25.5** | **23.1** | **23.5** |\n| DeepseekV3 (1B) | 5B | 19.5    |  26.2 | 27.4 | 23.6  | 22.7 | 23.9 |\n| **HELM-MiCE (1B)** | **5B** | **19.8**    |  **26.5** | **28.4** | **25.9** | **23.7** |  **24.9** |\n\nThe results demonstrate that the HELM-MiCE model consistently outperforms the Euclidean DeepseekV3 model across training stages, ranging from 4 to 5 billion tokens. We will incorporate these statistics in the revision.\n\n**Q:** While the authors perform a comparison with Euclidean models of the same size, there is a missing comparison with previous Hyperbolic transformers such as HAT, HNN++, and HyboNet, HypFormer. I believe such a comparison is necessary (even considering different tasks and setups, such as relation prediction) to fully validate the quality of their contribution. How does HELM empirically compare with previous hyperbolic transformers on different tasks? And why were previous hyperbolic approaches not considered in the experiments? \n\n**R:** We’d like to thank the reviewer for bringing attention to these prior works. Of these prior works for hyperbolic Transformers, three of which (namely HAT, HNN++, and HyboNet) each lacked hyperbolic counterparts of essential modules in modern LLMs. In comparison, all of these modules are available in HELM. We summarize the modules below:\n\n- HAT focuses on proposing hyperbolic self-attention mechanism without the rest of the hyperbolic Transformer architecture (e.g., LayerNorm, residual connection, positional encoding, linear layers), which is used in HELM-D where the operations are mathematically equivalent (as proven in HNN++)\n- HNN++ lacks positional encoding, layer normalization, and residual connection\n- HyboNet lacked layer normalization, and its positional encoding and residual connection could only be used in conjunction with its linear layers, as they are added as bias terms in the linear layers, greatly limiting the usability of these methods.\n\nGiven these limitations, we have not considered these models in LLM baselining of HELM. For completeness, we compare our HELM-D model against these prior works in the machine translation tasks from HyboNet, where the choice of using the dense version of HELM comes from the fact that the prior works were all dense models. The results are shown below. The results for the baselines are taken from [6], and the original study did not show statistical information.\n\n|  | IWSLT’14 | WMT’14 |\n| --- | --- | --- |\n| HAT | 23.7 | 21.8 |\n| HNN++ | 22.0 | 25.5 |\n| HyboNet | 25.9 | 26.2 |\n| **HELM-D** | **26.3** | **26.5** |\n\nAs for Hypformer, which does have all the required modules for language modeling, the differences that set HELM-D apart from Hypformer are:\n\n- Their positional encoding methods (relative for the former and our proposed HoPE for the latter).\n- Our proposed hyperbolic RMSNorm and the hyperbolic SwiGLU feedforward network (Appendix B.3.)\n\nFor the former, we have provided an ablation study between the positional encoding method proposed in Hypformer and HoPE in our appendix (Table 4), where we created an ablation model HELM-D-L that replaced HoPE with Hypformer’s relative encoding method. HELM-D with HoPE consistently outperformed HELM-D-L. For the latter, we recognize that a direct comparison between HELM and Hypformer would be informative. Thus, we compare a 100M Hypformer model against HELM-D 100M below, where the choice of HELM variant is based on the fact that Hypformer is a dense model:\n\n|  | CommonsenseQA | HellaSwag | OpenbookQA | MMLU | ARC-Challenging | Avg |\n| --- | --- | --- | --- | --- | --- | --- |\n| Hypformer (100M) | 19.4 | 25.1 | 26.6 | 22.9 | 23.6 | 23.5 |\n| **HELM-D (100M)** | **20.1** | **25.9** | **27.0** | **24.7** | **23.5** | **24.0** |\n\nAs shown above, the HELM-D model consistently outperforms Hypformer. Additionally, in comparison to Hypformer, the ablation model HELM-D-L still outperformed Hypformer in 4 out of the 5 tasks (Table 2), highlighting the effectiveness of the hyperbolic RMSNorm and hyperbolic SwiGLU feedforward network.\n\n**Q:** Could you provide qualitative examples of how your model improves upon Euclidean ones? In particular, how does HELM capture and leverage hierarchical information to deal with the natural language tasks you evaluate on?\n\n**R:** Prior works of hyperbolic language and vision models have found that hyperbolic models better learn the hierarchies in semantic structures [13, 47, 35], where more generic words/concepts tend to cluster in areas of smaller norm and more specific words/concepts tend to have larger norms. Our investigation of final-layer embedding distributions has found that HELM learns representations in a similar way. Below are examples of how the 1B HELM-MiCE model organizes semantics as compared to the 1B DeepseekV3 model:\n\n|  | HELM-MiCE | HELM-MiCE | DeepseekV3 | DeepseekV3 |\n| --- | --- | --- | --- | --- |\n| Words | Average Norm | Range | Average Norm | Range |\n| to, in, have, that, and, is, for | 35.930 | 35.890~35.951 | 33.725 | 33.660~33.800 |\n| study, research, subject, papers, category | 36.080 | 36.030~36.033 | 33.735 | 33.668~33.776 |\n| biology, physics, chemistry, mathematics, computer science | 36.155 | 36.033~36.270 | 33.720 | 33.658~33.776 |\n| algebra, geometry, photosynthesis, cellular respiration, genetics | 36.288 | 36.133~36.484 | 33.741 | 33.622~33.826 |\n\nAs we can see, for HELM-MiCE, more generic words (e.g., subject) are clustered closer to the origin than more specific words (e.g., biology), which has a smaller norm than even more specific words (e.g., photosynthesis). However, this does not necessarily hold for the DeepseekV3 1B model, demonstrating how HELM-MiCE better handles semantic hierarchies. We will provide a low-dimensional visualization of these embeddings in our revision.\n\nThis difference manifests when embedding questions HELM-MiCE correctly answers, but DeepseekV3 struggles with. For instance, below is a question in the MMLU benchmark categorized by its final embedding norm:\n\n| **HELM-MiCE** | **HELM-MiCE** | **DeepseekV3** | **DeepseekV3** |\n| --- | --- | --- | --- |\n| Words | Norm Range | Words | Norm Range |\n| A, How, does, if, there, have, is, any, with, of | 36.031~36.396 | is, a, connecting, graph, there, edges, complete, have, of,  | 33.668~33.768 |\n| discrete, vertices, edges, connecting, pair, graph, complete, many , 10 | 36.506~36.717 | discrete, 10, how, if, pair, does, with, A, vertices, any | 33.772~33.908 |\n\nAs we can see, for HELM-MiCE, general words have a smaller norm (e.g., how, if) while the more specific graph theory words are further away from the origin (e.g., connecting, graph). However, for DeepseekV3, general and specific words are sometimes intertwined in the embedding space. The hierarchical organization of the space enables the HELM models to sometimes better navigate the embedding space and obtain better performance."}, {"review_id": "cfVAx7260s", "reviewer": "Reviewer_cdXH", "summary": "The paper proposes shifting the geometry of LLM from Euclidean space to hyperbolic space. Therefore, the paper introduces HELM: a family of hyperbolic large language models. The paper also introduces a Mixture of Curvature experts model. In HELM-MICE, each expert operates in a distinct curvature space. The paper also develops hyperbolic multi-head latent attention.", "questions": "Please check weaknesses.", "limitations": "The limitations are addresses in the paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength:\nThe paper has a valid motivation. There is a clear explanation of the motivation in the introduction, starting from the distribution of the token embeddings. The motivation and research question are valid and novel. \n\nThe paper is the first to train a hyperbolic LLM at this scale. \n\nThe paper compares two existing models and shows the performance improvement. \n\n\n# Weakness:\n\nThe paper provides numbers to compare with the existing models and shows that the performance improves. However, as the main motivation is to choose a geometry that better matches hierarchical nature, I would like to see a more detailed analysis of the models. It is still not completely clear why hyperbolic geometry results in a better performance. What has improved in the hyperbolic model? Which type of errors have decreased? In addition, adding qualitative analysis would bring benefits to better understand why the hyperbolic model is performing better. \n\nI have a concern about the scalability of the proposed approach. However,  as this is the first step towards hyperbolic LLMs, the current scale is okay. \n\n## Related work:\nMissing related work: Hyperbolic space is beneficial in a lot of domains. To motivate the paper and cover the existing works, I would highly recommend citing the latest surveys: \n\n@article{peng2021hyperbolic,\n  title={Hyperbolic deep neural networks: A survey},\n  author={Peng, Wei and Varanka, Tuomas and Mostafa, Abdelrahman and Shi, Henglin and Zhao, Guoying},\n  journal={IEEE Transactions on pattern analysis and machine intelligence},\n  volume={44},\n  number={12},\n  pages={10023--10044},\n  year={2021},\n  publisher={IEEE}\n}\n\n@article{mettes2024hyperbolic,\n  title={Hyperbolic deep learning in computer vision: A survey},\n  author={Mettes, Pascal and Ghadimi Atigh, Mina and Keller-Ressel, Martin and Gu, Jeffrey and Yeung, Serena},\n  journal={International Journal of Computer Vision},\n  volume={132},\n  number={9},\n  pages={3484--3508},\n  year={2024},\n  publisher={Springer}\n}\n\nLines 57-66 talk about existing hyperbolic transformers; however, there are no citations provided. I would highly recommend adding citations, especially at the end of each limitation.\n\nThere is a research line of mixed curvature hyperbolic models. I would suggest explaining a bit and adding to the related work, and elaborating differences.\n\nAs the paper is using RoPE a lot, it would be beneficial to add that to the preliminaries as well. \n\n## Equations: \nWhat is y in equation 2?\n\nXj is not defined in Proposition 4.1. \n\nThe equation between lines 886 and 887 is not clear to me. Can the authors provide steps on how they yield that? The same holds for the equation between lines 912 and 913. \n\nProposition 4.2 talks about negative square Lorentz distance. Why is there no power 2 in the distance then?\n\nI have a problem understanding equation 4 and where it comes from. Maybe the authors can elaborate by explaining RoPE and the HoPE.\n\n## Writing:\nAs an overall suggestion, the writing is good; however, it needs proofreading and changing the text to make it easier to grasp.\n\nLines 16 and 17 suddenly introduce the dense model, HELM-D. It is confusing in the abstract.\n\nThe abstract talks about both models. What are both models?\n\nLine 73 uses Hyperbolic RMSNorm. I would recommend using the full name first and introducing the abbreviation. \n\nLines 121-122 talk about several isometric models of hyperbolic space; however, it is not yet clear to the reader what the hyperbolic space is. As it is preliminary, I would recommend adding a short sentence about negative curvature and the difference from Euclidean geometry. I know that it is already added in the Lorentz model section. However, it is more beneficial to add a general sentence to introduce hyperbolic space first. \n\nAdd citations for the Lorentz model section as well. \n\nIs Figure 2 referenced and explained in the text?\n\n\nLine 286, please first explain what MCQ questions are to increase the readability of the paper. \n\nThe paper needs also qualitative analysis to explore the role of geometry on the LLMs. \nLine 294: Appendix B -> Appendix C", "was_revised": true, "final_justification": "As the rebuttal has answered my questions, I increase my rating to accept.", "rebuttal": "Thanks for your valuable comments! Thanks for the valuable edit suggestion for our writing as well, we will incorporate the edits into our revision. We also appreciate the reviewer for pointing out the additional related works, which we will incorporate into our revision as well. We’ll denote the reviewer's concerns with Q and our responses with R. \n\n**Q:** It is still not completely clear why hyperbolic geometry results in a better performance. What has improved in the hyperbolic model? Which type of errors have decreased? In addition, adding qualitative analysis would bring benefits to better understand why the hyperbolic model is performing better.\n\n**R:** Prior works of hyperbolic language and vision models have found that hyperbolic models better learn the hierarchies in semantic structures [13, 47, 35], where more generic words/concepts tend to cluster in areas of smaller norm and more specific words/concepts tend to have larger norms. Our investigation of final-layer embedding distributions have found that HELM learns representations in a similar way. Below are examples of how the 1B HELM-MiCE model organizes semantics as compared to the 1B DeepseekV3 model:\n\n|  | HELM-MiCE | HELM-MiCE | DeepseekV3 | DeepseekV3 |\n| --- | --- | --- | --- | --- |\n| Words | Average Norm | Range | Average Norm | Range |\n| to, in, have, that, and, is, for | 35.930 | 35.890~35.951 | 33.725 | 33.660~33.800 |\n| study, research, subject, papers, category | 36.080 | 36.030~36.033 | 33.735 | 33.668~33.776 |\n| biology, physics, chemistry, mathematics, computer science | 36.155 | 36.033~36.270 | 33.720 | 33.658~33.776 |\n| algebra, geometry, photosynthesis, cellular respiration, genetics | 36.288 | 36.133~36.484 | 33.741 | 33.622~33.826 |\n\nAs we can see, for HELM-MiCE, more generic words (e.g., subject) are clustered closer to the origin than more specific words (e.g., biology), which has a smaller norm than even more specific words (e.g., photosynthesis). However, this does not necessarily hold for the DeepseekV3 1B model, demonstrating how HELM-MiCE better handles semantic hierarchies. We will provide low-dimensional visualization of these embeddings in our revision.\n\nThis difference manifests when embeddings questions HELM-MiCE correctly answer but DeepseekV3 struggles. For instance, below is a question in the MMLU benchmark categorized by its final embedding norm:\n\n| **HELM-MiCE** | **HELM-MiCE** | **DeepseekV3** | **DeepseekV3** |\n| --- | --- | --- | --- |\n| Words | Norm Range | Words | Norm Range |\n| A, How, does, if, there, have, is, any, with, of | 36.031~36.396 | is, a, connecting, graph, there, edges, complete, have, of,  | 33.668~33.768 |\n| discrete, vertices, edges, connecting, pair, graph, complete, many , 10 | 36.506~36.717 | discrete, 10, how, if, pair, does, with, A, vertices, any | 33.772~33.908 |\n\nAs we can see, for HELM-MiCE, general words have a smaller norm (e.g., how, if) while the more specific graph theory words are further away from the origin (e.g., connecting, graph). However, for DeepseekV3, general and specific words are sometimes intertwined in the embedding space. The hierarchical organization of the space enables the HELM models to sometimes better navigate the embedding space and obtain better performance.\n\n**Q:** The reviewer has concerns about the scalability of the proposed approach.\n\n**R:** We appreciate the reviewer recognizing that this is the first work to train hyperbolic LLMs and hyperbolic models at the scale of billions of parameters and tokens. To address the reviewer’s concerns, we compare the runtime and memory consumption of HELM models and the Euclidean baselines, which we will include in the revision. We found that our proposed methods are efficient and within 1.55X in runtime and 1.11X in memory usage of the Euclidean counterparts. In the table below, we show that runtime and memory comparisons between HELM and the Euclidean baselines for one training iteration (roughly 2M tokens). The results shown are for our exact experimental setup ran on 4 A100 GPUs, where we show the worst runtime and memory usage across the ranks. The results are averaged over 10 runs, and standard deviations are not shown since they are within 0.2 of the results.\n\n| Model | Runtime  | Memory Usage |\n| --- | --- | --- |\n| LLaMA 100M | 8.4s | 23.8Gb |\n| HELM-D 100M | 13.1s | 24.9GB |\n| DeepseekV3 100M | 11.0s | 23.5Gb |\n| HELM-MiCE 100M | 16.9s | 26.3Gb |\n| DeepseekV3 1B | 83.5s | 33.1Gb |\n| HELM-MiCE 1B | 119.4s | 35.8Gb |\n\nAs we can see, HELM does not incur significant training overhead across the different model sizes. Additionally, while some efforts have tried to develop tools for hyperbolic foundation models [23], there currently still lacks libraries that optimize training hyperbolic foundation models, whereas there are many that are optimized for training Euclidean models (such as LLM-Foundry, Flash Attention, Deepspeed). Development of comparable hyperbolic libraries would further reduce the gap between HELM models and the Euclidean baselines. \n\nFinally, while there is overhead for models of the **same size**, non-Euclidean models require **fewer dimensions** to embed complex structures when compared to Euclidean models [a,b]. This enables the potential for non-Euclidean models to match the performance of Euclidean models with **fewer parameters** to build parameter-efficient foundation models, countering the computational overhead from non-Euclidean operations and offering the potential to continue the scaling law relationship between parameters and model performance. On the other hand, Euclidean models face heavy dimension-distortion tradeoffs [c].\n\n[a] Sarkar, R. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pp. 355–366. Springer, 2011.\n\n[b] Lee, J. R., Naor, A., and Peres, Y. Trees and markov convexity, 2007.\n\n[c] Matousek, J. Lectures on Discrete Geometry. Springer, 2002.\n\n**Q:** What is y in equation 2?\n\n**R:** The y here is f(x), we will fix all of our typos in the revision.\n\n**Q:** Xj is not defined in Proposition 4.1?\n\n**R:** The i and j subscripts in the proposition index through the tokens, queries, and keys. We will make this clearer in the revision.\n\n**Q:** Confusion regarding equations on lines 886 and 887 and lines 912 and 913?\n\n**R:** The equation on these utilizes the simplified squared Lorentz distance formula from [28]\n\n**Q:** Why is there no power 2 in the distance in Proposition 4.2?\n\n**R:** We apologize for our typo; there should be an exponential in the distance\n\n**Q:** Where did equation 4 come from?\n\n**R:** In the case of the 2D Euclidean RoPE, the formulation depends on properties of Euclidean rotation operations. Let $R_{\\theta_i}$ denote a 2D rotational matrix of angle $\\theta_i$ One property of this matrix is that $R_{\\theta_i}^TR_{\\theta_j} = R_{\\theta_i-\\theta_j}$ for some other angle $\\theta_j$. This identity allows relative position information to be encoded directly into the dot-product-attention: $(R_{\\theta_i}q_i)^T(R_{\\theta_j}k_j) = (q_i^T)(R_{\\theta_i-\\theta_j}k_j)$. The properties of rotational operations also enable other theoretical guarantees of RoPE. The higher-dimensional RoPE operations generalize this by putting the 2D matrices on the diagonal. Our proposed HoPE operation takes inspiration from RoPE and is based on Loretzian rotational operations (see [6]). This formulation enables HoPE to encode relative positional information similar to RoPE and achieve similar theoretical guarantees."}, {"review_id": "7CWSEEQmHh", "reviewer": "Reviewer_Ar71", "summary": "This paper propose a new hyperbolic language model called HELM, which extends a Lorentz space Transformer architecture with modifications inspired by modern Euclidean Transformers, including hyperbolic rotary position encoding (HoPE), mixture-of-curvature (MiCE), hyperbolic multi-head latent attention (HMLA) and hyperbolic RMSNorm activations. These modifications are essential for hyperbolic Transformers to catch up on efficiency and capacities with their Euclidean counterpart. Experimental results show that HELM models with up-to 1B parameters out-perform Eucliean baselines on several multi-choice benchmarks when trained on 5B tokens.", "questions": "1. What are the expected benefits of hyperbolic models for frontier models? The hyperbolic models are motivated by the distortions observed in the Euclidean embedding space, but are there any indications of this distortion negatively affecting the frontier models in any tangible ways?\n2. Is it possible to retain some experts in flat space in the MiCE structure? Euclidean space is already demonstrated as a solid foundation for numerous models. It would be interesting if MiCE can retain some computations in flat spaces to maximize its flexibility.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The modifications brought by HELM are essential for building hyperbolic LLMs. Compared to the barebone hyperbolic models in previous works, this paper presents a full-fledged hyperbolic model recipe that can serve as the basis to pre-train large scale, high performing hyperbolic LLMs.\n2. The modifications are inspired by their Euclidean counterparts, and the adaptions to Lorentz space are intuitive and well-justified.\n3. The mixture-of-curvature architecture extends the popular MoE architecture, and exploits the unique feature of hyperbolic space where different experts can resides in spaces with different curvatures.\n\nWeakness:\n1. The empirical results are limited to small scale experiments and the coverage of the benchmarks are limited.\n\n1a. The model size studied ranges from 100m to 1B in the current experiments. I would like to see results on 7B/14B models. The motivation for hyperbolic models is the distortion of embeddings in Euclidean space, but it is unclear to me whether this distortion would manifest in meaningful way when we scale up the model size.\n\n1b. The current pre-training dataset only contains 5B tokens from English Wikipedia. It would better to see more tokens in pre-training by using open-source datasets like C4. It is unclear whether the models have been trained to saturation and whether the advantages of hyperbolic model would still stand given more training budgets. Based on my personal experience, pre-training would not saturate for at least 100B tokens for 1B sized models.\n\n1c. The paper only has results on multi-choice benchmarks. It would be better to also include some instruction following, math and coding benchmarks.\n\n2. It is unclear how the HELM model compares to the Euclidean models efficiency wise in terms of both memory and computation.", "was_revised": true, "final_justification": "My main concern about HELM is their scalability in terms of both model size and training data size. Overall this work brings an interesting model architecture for foundation models and I would like to see it published in the conference.", "rebuttal": "Thanks for your valuable comments! We’ll denote the reviewer's concerns with Q and our responses with R. We appreciate the reviewer finding our architectural and theoretical contributions compelling. The main concern of the reviewer focuses on the scalability of the HELM models, which we address below.\n\n**Q:** Regarding the size of the trained models and training corpora:\n\n- The model size studied ranges from 100m to 1B in the current experiments. I would like to see results on 7B/14B models…would distortion manifest in meaningful way?\n- The current pre-training dataset only contains 5B tokens from English Wikipedia….pre-training would not saturate for at least 100B tokens for 1B sized models.\n\n**R:** Thank you for the feedback and for sharing your personal experience! We agree that the models can benefit from increased parameter count and going beyond 100B tokens combining datasets like C4. Given how computationally demanding this is, however, we focused on the high-quality tokens in Wikipedia to showcase the merits of our architecture in providing a first step towards making such large-scale hyperbolic models a reality.\n\nAs the first work to take a fully hyperbolic approach to LLMs and pretraining at the scale of billions of parameters and tokens, our work focuses on demonstrating the effectiveness of HELM under *computational expenses that are reasonable within the academic research context*. When going from 100M parameters to 1B, we found that the advantages of hyperbolic LLMs persist when compared to Euclidean baselines. Our architecture and implementation, such as the theoretical guarantees and hyperbolic word embedding mentioned in Sec. 5, enables the training stability needed to be the first to train hyperbolic models at this scale. \n\nWe also do reasonably expect that the advantage of hyperbolic LLMs would not diminish with increasing parameter counts. Previous works have shown theoretically that Euclidean space will **always incur significant distortion when embedding tree-like data regardless of dimensionality and token count** [b], while hyperbolic space can achieve arbitrarily low distortion [a]. Thus, there is a theoretical upper bound on the performance of the Euclidean baselines at larger scales, which is not present for HELM models. We plan on releasing well-documented code, and we hope our architecture helps pave the path for 7B/14B models on the 100B of tokens scale to be trained in the future, in settings that are not limited by the computational resources of academic research.\n\n**Q:** The paper only has results on multi-choice benchmarks...better to also include some instruction following, math and coding benchmarks.\n\n**R:** Thank you for your valuable suggestion. We agree with the reviewer that additional benchmarks, such as instruction following, would be interesting. Due to limited computational resources in academic research settings, we focused on pretraining on the high-quality tokens from English Wikipedia, instead of much larger-scale data or post-training, such as instruction fine-tuning. We chose the benchmarks due to the simplicity of evaluation on multi-QA as they align with the Wikipedia tokens. Thus, they better reflects the true performance of the learned model, instead of random noise. These benchmarks are also wide used by previously accepted by NeurIPS[49,2], popular open LLMs[19, 11], and LLMs of similar size[52]. To achieve meaningful results in math or coding would require much larger computational resources that would be out of our compute abilities. For this reason, we focused on benchmarks that models can, in fact, answer based on the high-quality Wikipedia tokens it has been trained on. \n\n**Q:** It is unclear how the HELM model compares to the Euclidean models efficiency wise in terms of both memory and computation.\n\n**R:**  Our proposed methods are efficient in both runtime and memory usage. HELM models are within 1.55X in runtime and 1.11X in memory usage of the Euclidean counterparts for both the 100M and 1B models. In the table below, we show that runtime and memory comparisons between HELM and the Euclidean baselines for one training iteration (roughly 2M tokens). The results shown are for our exact experimental setup ran on 4 A100 GPUs, where we show the worst runtime and memory usage across the ranks. The results are averaged over 10 runs, and standard deviations are not shown since they are within 0.2 of the results. We will include the results in the revision. \n\n| Model | Runtime  | Memory Usage |\n| --- | --- | --- |\n| LLaMA 100M | 8.4s | 23.8Gb |\n| HELM-D 100M | 13.1s | 24.9GB |\n| DeepseekV3 100M | 11.0s | 23.5Gb |\n| HELM-MiCE 100M | 16.9s | 26.3Gb |\n| DeepseekV3 1B | 83.5s | 33.1Gb |\n| HELM-MiCE 1B | 119.4s | 35.8Gb |\n\nAdditionally, there has been very little effort to develop optimized tools and libraries for hyperbolic foundation models [23]. In contrast, there is an abundance of the same for Euclidean models (e.g., LLM-Foundry, Flash Attention, DeepSpeed). Development of comparable hyperbolic libraries would further reduce the gap between HELM models and Euclidean baselines.\n\nFinally, while there is overhead for models of the **same size**, non-Euclidean models require **fewer dimensions** to embed complex structures when compared to Euclidean models [a,b]. This shows promise for non-Euclidean models to match the performance of Euclidean models with **fewer parameters** to build parameter-efficient foundation models despite the additional overhead. There is also potential to follow scaling law relationships between parameters and model performance. On the other hand, Euclidean models face heavy dimension-distortion tradeoffs [c].\n\n**Q:** What are the expected benefits of hyperbolic models for frontier models? Are there any indications of this distortion negatively affecting the frontier models in any tangible ways?\n\n**R:** The advantage of hyperbolic space is that it could better capture the hierarchical and scale-free properties in texts and language, as shown by prior works. To the best of our knowledge, there currently lacks conclusive work directly connecting numerical distortion to downstream performance in foundation models. Some previous works have explored how using manifolds that more accurately estimate the structure of the data can enhance performance in tasks related to GNNs and word embeddings (e.g., [d]).\n\nNevertheless, our investigation into the final embedding distribution of our pre-trained HELM models suggests that hyperbolic LLMs can better represent the semantic hierarchies than the Euclidean baselines. Below are examples of how the 1B HELM-MiCE model organizes semantics as compared to the 1B DeepseekV3 model:\n\n|  | HELM-MiCE | HELM-MiCE | DeepseekV3 | DeepseekV3 |\n| --- | --- | --- | --- | --- |\n| Words | Average Norm | Range | Average Norm | Range |\n| to, in, have, that, and, is, for | 35.930 | 35.890~35.951 | 33.725 | 33.660~33.800 |\n| study, research, subject, papers, category | 36.080 | 36.030~36.033 | 33.735 | 33.668~33.776 |\n| biology, physics, chemistry, mathematics, computer science | 36.155 | 36.033~36.270 | 33.720 | 33.658~33.776 |\n| algebra, geometry, photosynthesis, cellular respiration, genetics | 36.288 | 36.133~36.484 | 33.741 | 33.622~33.826 |\n\nAs we can see, for HELM-MiCE, more generic words (e.g., subject) are clustered closer to the origin than more specific words (e.g., biology), which has a smaller norm than even more specific words (e.g., photosynthesis). However, this does not necessarily hold for the DeepseekV3 1B model, demonstrating how HELM-MiCE better handles semantic hierarchies. We will provide a low-dimensional visualization of these embeddings in our revision.\n\nThis difference manifests when embeddings questions HELM-MiCE correctly answer but DeepseekV3 struggles. For instance, below is a question in the MMLU benchmark categorized by its final embedding norm:\n\n| **HELM-MiCE** | **HELM-MiCE** | **DeepseekV3** | **DeepseekV3** |\n| --- | --- | --- | --- |\n| Words | Norm Range | Words | Norm Range |\n| A, How, does, if, there, have, is, any, with, of | 36.031~36.396 | is, a, connecting, graph, there, edges, complete, have, of,  | 33.668~33.768 |\n| discrete, vertices, edges, connecting, pair, graph, complete, many , 10 | 36.506~36.717 | discrete, 10, how, if, pair, does, with, A, vertices, any | 33.772~33.908 |\n\nFor HELM-MiCE, general words have a smaller norm (e.g., how, if) while the more specific graph theory words are further away from the origin (e.g., connecting, graph). However, for DeepseekV3, general and specific words are sometimes intertwined in the embedding space. The hierarchical organization of the space enables the HELM models to sometimes better navigate the embedding space and obtain better performance.\n\n**Q:** Is it possible to retain some experts in flat space in the MiCE structure? \n\n**R:** Thank you for your valuable suggestions! We agree that incorporating Euclidean experts into MiCE is an exciting future direction. Our current implementation of MiCE initializes the experts with a variety of curvature values (from -0.1 to -2.0), where the lower curvature values can simulate a flatter space (like Euclidean space). To truly extend MiCE to Euclidean space, the projection coefficients in equation Sec. 4.2 can be derived from log and exp mappings. As the tangent space of hyperbolic spaces are universally Euclidean, Eq.6 can be extended to incorporate Euclidean space by lifting points to and from the tangent spaces. We will add these discussions to the revision.\n\n***References:*** \n\n[a] Sarkar, R. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing, pp. 355–366. Springer, 2011.\n\n[b] Lee, J. R., Naor, A., and Peres, Y. Trees and markov convexity, 2007.\n\n[c] Matousek, J. Lectures on Discrete Geometry. Springer, 2002.\n\n[d] Gu, A., Sala, F., Gunel, B., and Re, C. Learning mixed-curvature representations in product spaces. ICLR, 2019."}]

Author rebuttal:

Meta-review: The submission presents HELM, a new class of Large Language Models designed to operate in hyperbolic space rather than traditional Euclidean geometry. By adapting key components like Mixture of Experts and Multi-head Latent Attention to this geometry, HELM aims to overcome limitations such as training instability and reduced generative performance. Experimental results show modest improvements over standard Euclidean-based models.

During the discussion with authors and reviewers, the issues that still remained unanswered are:

1. Scalability of the model (R-1cTB and R-Ar71): It is unclear how this model would scale with data and model parameters. 
2. Marginal improvements (R-2Div): Experiments show marginal improvements over the baselines.
3. Comparisons of Hyperbolic and Euclidean norms (R-cdXH): Authors provided an unconvincing and/or unclear answer.

To solve some of the issues above required more time and resources (e.g., 1 and 2). While many issues remained open, the majority of reviewers agreed that the approach is novel and can be of interest to the community. Because of the submissions’ novelty, I am recommending its acceptance to the program.
Decision: Accept (poster)