Title: HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language models

hyperbolic geometry

mixture-of-experts
Abstract: Frontier large language models (LLMs) have shown great success in text modeling and generation tasks across domains. However, natural language exhibits inherent semantic hierarchies and nuanced geometric structure, which current LLMs do not capture completely owing to their reliance on Euclidean operations such as dot-products and norms. Furthermore, recent studies have shown that not respecting the underlying geometry of token embeddings leads to training instabilities and degradation of generative capabilities. These findings suggest that shifting to non-Euclidean geometries can better align language models with the underlying geometry of text. We thus propose to operate fully in $\textit{Hyperbolic space}$, known for its expansive, scale-free, and low-distortion properties. To this end, we introduce $\textbf{HELM}$, a family of $\textbf{H}$yp$\textbf{E}$rbolic Large $\textbf{L}$anguage $\textbf{M}$odels, offering a geometric rethinking of the Transformer-based LLM that addresses the representational inflexibility, missing set of necessary operations, and poor scalability of existing hyperbolic LMs. We additionally introduce a $\textbf{Mi}$xture-of-$\textbf{C}$urvature $\textbf{E}$xperts model, $\textbf{HELM-MiCE}$, where each expert operates in a distinct curvature space to encode more fine-grained geometric structure from text, as well as a dense model, $\textbf{HELM-D}$. For $\textbf{HELM-MiCE}$, we further develop hyperbolic Multi-Head Latent Attention ($\textbf{HMLA}$) for efficient, reduced-KV-cache training and inference. For both models, we further develop essential hyperbolic equivalents of rotary positional encodings and root mean square normalization. We are the first to train fully hyperbolic LLMs at billion-parameter scale, and evaluate them on well-known benchmarks such as MMLU and ARC, spanning STEM problem-solving, general knowledge, and commonsense reasoning. Our results show consistent gains from our $\textbf{HELM}$ architectures – up to 4\% – over popular Euclidean architectures used in LLaMA and DeepSeek with superior semantic hierarchy modeling capabilities, highlighting the efficacy and enhanced reasoning afforded by hyperbolic geometry in large-scale language model pretraining.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a new hyperbolic language model called HELM, which extends a Lorentz space Transformer architecture with modifications inspired by modern Euclidean Transformers, including hyperbolic rotary position encoding (HoPE), mixture-of-curvature (MiCE), hyperbolic multi-head latent attention (HMLA) and hyperbolic RMSNorm activations. These modifications are essential for hyperbolic Transformers to catch up on efficiency and capacities with their Euclidean counterpart. Experimental results show that HELM models with up-to 1B parameters out-perform Eucliean baselines on several multi-choice benchmarks when trained on 5B tokens.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The modifications brought by HELM are essential for building hyperbolic LLMs. Compared to the barebone hyperbolic models in previous works, this paper presents a full-fledged hyperbolic model recipe that can serve as the basis to pre-train large scale, high performing hyperbolic LLMs.\n2. The modifications are inspired by their Euclidean counterparts, and the adaptions to Lorentz space are intuitive and well-justified.\n3. The mixture-of-curvature architecture extends the popular MoE architecture, and exploits the unique feature of hyperbolic space where different experts can resides in spaces with different curvatures.\n\nWeakness:\n1. The empirical results are limited to small scale experiments and the coverage of the benchmarks are limited.\n\n1a. The model size studied ranges from 100m to 1B in the current experiments. I would like to see results on 7B/14B models. The motivation for hyperbolic models is the distortion of embeddings in Euclidean space, but it is unclear to me whether this distortion would manifest in meaningful way when we scale up the model size.\n\n1b. The current pre-training dataset only contains 5B tokens from English Wikipedia. It would better to see more tokens in pre-training by using open-source datasets like C4. It is unclear whether the models have been trained to saturation and whether the advantages of hyperbolic model would still stand given more training budgets. Based on my personal experience, pre-training would not saturate for at least 100B tokens for 1B sized models.\n\n1c. The paper only has results on multi-choice benchmarks. It would be better to also include some instruction following, math and coding benchmarks.\n\n2. It is unclear how the HELM model compares to the Euclidean models efficiency wise in terms of both memory and computation.\n\nQUESTIONS:\n1. What are the expected benefits of hyperbolic models for frontier models? The hyperbolic models are motivated by the distortions observed in the Euclidean embedding space, but are there any indications of this distortion negatively affecting the frontier models in any tangible ways?\n2. Is it possible to retain some experts in flat space in the MiCE structure? Euclidean space is already demonstrated as a solid foundation for numerous models. It would be interesting if MiCE can retain some computations in flat spaces to maximize its flexibility.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes shifting the geometry of LLM from Euclidean space to hyperbolic space. Therefore, the paper introduces HELM: a family of hyperbolic large language models. The paper also introduces a Mixture of Curvature experts model. In HELM-MICE, each expert operates in a distinct curvature space. The paper also develops hyperbolic multi-head latent attention.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength:\nThe paper has a valid motivation. There is a clear explanation of the motivation in the introduction, starting from the distribution of the token embeddings. The motivation and research question are valid and novel. \n\nThe paper is the first to train a hyperbolic LLM at this scale. \n\nThe paper compares two existing models and shows the performance improvement. \n\n\n# Weakness:\n\nThe paper provides numbers to compare with the existing models and shows that the performance improves. However, as the main motivation is to choose a geometry that better matches hierarchical nature, I would like to see a more detailed analysis of the models. It is still not completely clear why hyperbolic geometry results in a better performance. What has improved in the hyperbolic model? Which type of errors have decreased? In addition, adding qualitative analysis would bring benefits to better understand why the hyperbolic model is performing better. \n\nI have a concern about the scalability of the proposed approach. However,  as this is the first step towards hyperbolic LLMs, the current scale is okay. \n\n## Related work:\nMissing related work: Hyperbolic space is beneficial in a lot of domains. To motivate the paper and cover the existing works, I would highly recommend citing the latest surveys: \n\n@article{peng2021hyperbolic,\n  title={Hyperbolic deep neural networks: A survey},\n  author={Peng, Wei and Varanka, Tuomas and Mostafa, Abdelrahman and Shi, Henglin and Zhao, Guoying},\n  journal={IEEE Transactions on pattern analysis and machine intelligence},\n  volume={44},\n  number={12},\n  pages={10023--10044},\n  year={2021},\n  publisher={IEEE}\n}\n\n@article{mettes2024hyperbolic,\n  title={Hyperbolic deep learning in computer vision: A survey},\n  author={Mettes, Pascal and Ghadimi Atigh, Mina and Keller-Ressel, Martin and Gu, Jeffrey and Yeung, Serena},\n  journal={International Journal of Computer Vision},\n  volume={132},\n  number={9},\n  pages={3484--3508},\n  year={2024},\n  publisher={Springer}\n}\n\nLines 57-66 talk about existing hyperbolic transformers; however, there are no citations provided. I would highly recommend adding citations, especially at the end of each limitation.\n\nThere is a research line of mixed curvature hyperbolic models. I would suggest explaining a bit and adding to the related work, and elaborating differences.\n\nAs the paper is using RoPE a lot, it would be beneficial to add that to the preliminaries as well. \n\n## Equations: \nWhat is y in equation 2?\n\nXj is not defined in Proposition 4.1. \n\nThe equation between lines 886 and 887 is not clear to me. Can the authors provide steps on how they yield that? The same holds for the equation between lines 912 and 913. \n\nProposition 4.2 talks about negative square Lorentz distance. Why is there no power 2 in the distance then?\n\nI have a problem understanding equation 4 and where it comes from. Maybe the authors can elaborate by explaining RoPE and the HoPE.\n\n## Writing:\nAs an overall suggestion, the writing is good; however, it needs proofreading and changing the text to make it easier to grasp.\n\nLines 16 and 17 suddenly introduce the dense model, HELM-D. It is confusing in the abstract.\n\nThe abstract talks about both models. What are both models?\n\nLine 73 uses Hyperbolic RMSNorm. I would recommend using the full name first and introducing the abbreviation. \n\nLines 121-122 talk about several isometric models of hyperbolic space; however, it is not yet clear to the reader what the hyperbolic space is. As it is preliminary, I would recommend adding a short sentence about negative curvature and the difference from Euclidean geometry. I know that it is already added in the Lorentz model section. However, it is more beneficial to add a general sentence to introduce hyperbolic space first. \n\nAdd citations for the Lorentz model section as well. \n\nIs Figure 2 referenced and explained in the text?\n\n\nLine 286, please first explain what MCQ questions are to increase the readability of the paper. \n\nThe paper needs also qualitative analysis to explore the role of geometry on the LLMs. \nLine 294: Appendix B -> Appendix C\n\nQUESTIONS:\nPlease check weaknesses.\n\nLIMITATIONS:\nThe limitations are addresses in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new family of Large Language Models (i.e. HELM) that can operate entirely in hyperbolic space. In particular, the authors propose architectural interventions to obtain a set of hyperbolic transformer models, including a scalable hyperbolic mixture-of-curvature expert model where each expert operates using a different curvature to better leverage the geometric structure of the space. The authors show that their architectural interventions allow training fully hyperbolic LLMs of up to a billion parameters, demonstrating competitive performance with Euclidean counterparts on different downstream tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Reason to accept:**\n- The paper is generally well-written and easy to follow. The motivation for exploring hyperbolic representations for LLMs is clearly articulated, as well as its potential impact.\n- The paper performs an important technical and architectural contribution, introducing novel components for building LLMs operating in hyperbolic space and introducing a new model (i.e., HELM). This is an important contribution that can support future research in the field.\n- Experiments performed on different natural language tasks demonstrate that HELM is competitive with existing LLMs of the same size operating in Euclidean space.\n\n**Reason to reject:**\n\nWhile I find the architectural contribution compelling, I believe the experiments are currently insufficient for acceptance. Further experiments and analyses are required to validate the impact of their contribution. Specifically:\n\n- The difference in performance with Euclidean models is minimal and not supported by statistical significance analyses, which, I think, given the relatively small difference in performance, is necessary. \n- While the authors perform a comparison with Euclidean models of the same size, there is a missing comparison with previous Hyperbolic transformers such as HAT, HNN++, and HyboNet, HypFormer. I believe such a comparison is necessary (even considering different tasks and setups, such as relation prediction) to fully validate the quality of their contribution.\n- Beyond quantitative comparison with Euclidean models, I believe the paper should include a more in-depth qualitative/error analysis to understand the real impact and limitations of the hyperbolic representation and the introduced architecture. One of the expected impacts of hyperbolic space should be an improved ability to capture hierarchical relationships. However, from the current empirical evaluation, this is not at all clear.\n\nQUESTIONS:\n- How do you ensure that the reported results and improvements are statistically significant? \n- Could you provide qualitative examples of how your model improves upon Euclidean ones? In particular, how does HELM capture and leverage hierarchical information to deal with the natural language tasks you evaluate on?\n- How does HELM empirically compare with previous hyperbolic transformers on different tasks? And why were previous hyperbolic approaches not considered in the experiments?\n\nLIMITATIONS:\nNo. The limitations are only briefly discussed in section 6 and should be better elicited in the experimental design through an in-depth error analysis and qualitative evaluation (please see weaknesses).", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces HELM, a family of fully hyperbolic large language models that re-implements every key Transformer component—positional encoding (HOPE), normalization (Hyperbolic RMSNorm), attention (HMLA), and feed-forward blocks—inside Lorentz space. It further proposes Mixture-of-Curvature Experts (MICE) so different experts operate at distinct curvatures, and scales these ideas to 100M – 1B parameters trained on 5B Wikipedia tokens. Across five reasoning benchmarks (MMLU, ARC-Chal., CommonsenseQA, HellaSwag, OpenBookQA) HELM-MICE beats matched Euclidean baselines and shows the largest gains on the hardest tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper tackles the mismatch between language’s hierarchical structure and Euclidean embeddings by operating wholly in hyperbolic space and letting different experts learn at different curvatures.\n2. HOPE, Hyperbolic RMSNorm, and Lorentz-residual connections give HELM feature-parity with modern LLMs, while accompanying proofs ensure these modules retain the stability and relative-position guarantees of their Euclidean counterparts.\n3. By combining HMLA with MICE, the authors train the first billion-parameter fully hyperbolic LLM; the models consistently top equally-sized Euclidean (dense and MoE) baselines, especially on MMLU and ARC-Challenging, demonstrating that the geometric shift yields real, if modest, gains.\n4. The paper is well-written and easy to follow.\n\n## Weaknesses\n\n1. All models are trained on only 5B tokens, so absolute accuracy remains low and  improvements could shrink or vanish at larger data scales—making it unclear whether hyperbolic benefits persist in realistic pre-training regimes.\n2. HELM introduces non-trivial engineering overhead (Lorentz ops, curvature routing, expert load-balancing), yet the paper gives no timing or memory benchmarks against standard implementations, leaving practitioners uncertain about the real compute-efficiency trade-offs.\n3. The paper does not provide error analysis, case studies, or interpretive examples to show how hyperbolic geometry changes model behavior or what kinds of errors it corrects\n\nQUESTIONS:\n1. Are each expert’s curvature values fixed or learnable, and how sensitive are results to that choice?\n2. Do you expect the hyperbolic–Euclidean gap to widen, stay flat, or shrink when trained on ≥100 B tokens? And what stability issues could emerge at that scale?\n3. Could you provide low-dimensional projections (e.g., Poincaré-disk plots) of the learned embeddings to show how curvature and expert routing organize semantic structure?\n4. What motivated using the Lorentz representation over alternatives like the Poincaré ball or Klein model, and did you observe concrete numerical or training-stability advantages?\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-11"}