Title: Hyperbolic Fine-Tuning for Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Hyperbolic Space

Low-Rank Adaptation

Embedding Space
Abstract: Large language models (LLMs) have demonstrated remarkable performance across various tasks. However, it remains an open question whether the default Euclidean space is the most suitable choice for LLMs.
In this study, we investigate the geometric characteristics of LLMs, focusing specifically on tokens and their embeddings.
Our findings reveal that token frequency follows a power-law distribution, where high-frequency tokens (e.g., the, that ) constitute the minority, while low-frequency tokens (e.g., apple, dog) constitute the majority. Furthermore, high-frequency tokens cluster near the origin, whereas low-frequency tokens are positioned farther away in the embedding space.
Additionally, token embeddings exhibit hyperbolic characteristics, indicating a latent tree-like structure within the embedding space.
Motivated by these observations, we propose **HypLoRA**, an efficient fine-tuning approach that operates in hyperbolic space to exploit these underlying hierarchical structures better.
HypLoRA performs low-rank adaptation directly in hyperbolic space, thereby preserving hyperbolic modeling capabilities throughout the fine-tuning process.
Extensive experiments across various base models and reasoning benchmarks, specifically arithmetic and commonsense reasoning tasks, demonstrate that HypLoRA substantially improves LLM performance.


Peer reviews:
comments: SUMMARY:
The paper explores the suitability of hyperbolic geometry for representing token embeddings in large language models (LLMs), motivated by the hierarchical structure commonly found in language data. It first identifies that token embeddings naturally display hyperbolic characteristics, such as a tree-like structure, where high-frequency (general) tokens cluster near the origin, and low-frequency (specific) tokens are positioned farther away, following a power-law distribution.

To better exploit this hierarchical structure, the authors propose HypLoRA, a new fine-tuning method that integrates hyperbolic geometry directly into the low-rank adaptation process, unlike traditional methods that use Euclidean geometry. The key idea of HypLoRA is to perform parameter-efficient updates directly on the hyperbolic manifold, avoiding the limitations of using standard Euclidean operations.

Through extensive experiments on arithmetic and commonsense reasoning benchmarks, the paper demonstrates that HypLoRA significantly improves the performance of existing LLMs compared to conventional Euclidean fine-tuning methods, confirming the advantage of explicitly modeling token hierarchies in hyperbolic space.

STRENGTHS AND WEAKNESSES:
**Strengths:**

* Extensive experiments across diverse benchmarks and models
* Integration of hyperbolic geometry into low-rank adaptation (HypLoRA)

**Weaknesses:**

* Incremental improvements may not justify increased complexity and computational overhead in practical deployments.

QUESTIONS:
NA

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the geometric properties of token embeddings in LLMs. This paper first performs a detailed analysis and find that LLM embeddings exhibit strong hyperbolic properties. Based on this observation, the authors propose a new PEFT method, HypLoRA, to perform low-rank adaptation directly on a hyperbolic manifold. Through extensive experiments on multiple benchmarks, the authors show that HypLoRA outperforms LoRA across a variety of models.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n1. Novel and insightful perspective: The main strength of this paper lies in its novel and insightful perspective. Rather than simply proposing a new technique, the authors provide a compelling motivation by first analyzing the underlying geometric structure of LLM embeddings.\n\n2. Robust performance improvement: The proposed method, HypLoRA, shows consistent and robust performance improvement over the standard LoRA baseline. This is demonstrated across multiple base models (LLaMA, Gemma, etc.) and various inference benchmarks.\n3. Solid theoretical foundation: The research in this paper is well supported by theoretical analysis.\n\n## Weaknesses:\n\n1. Lack of pre-fine-tuning benchmarks: The experimental results table lacks the performance of the base model before any fine-tuning (i.e., zero-shot or few-shot performance). Including the performance of the base model allows the reader to measure the absolute improvement brought by PEFT itself. This is crucial to understand the overall impact of the two PEFT methods and to justify the need for fine-tuning in the first place.\n\n2. Motivation is disconnected from experimental analysis: The experimental analysis can be better aligned with the core motivation of the paper. The motivation of this paper is based on the idea that hyperbolic geometry can better model the hierarchical tree-like structures present in language. However, the evaluation section focuses mainly on showing its accuracy improvement on downstream tasks. The paper would be significantly strengthened if qualitative or quantitative analysis could be included to show how HypLoRA actually changes the embedding space to better represent these hierarchical structures.\n\nQUESTIONS:\n1. As shown in Table 3, the LLaMA3-8B model fine-tuned with LoRA achieves 69.8% accuracy on GSM8K. However, the official technical report for LLaMA-3 (https://ai.meta.com/blog/meta-llama-3/) shows an 8-shot CoT accuracy of GSM8K is about 79.6%. Can you explain the evaluation settings that lead to this result? This significant difference raises potential concerns about the strength of the LoRA baseline used for comparison. If the baseline is not optimally tuned, the gains reported by HypLoRA may be partially due to a suboptimal starting point rather than simply the advantages of the hyperbolic method.\n\n2. Regarding the stability of fine-tuning: It is well known that the performance of LoRA can be sensitive to hyperparameter choices (e.g., learning rate, rank \"r\"). The paper compares the final performance of LoRA and HypLoRA, but a deeper analysis of training stability would be helpful. Can you comment on the stability of HypLoRA compared to LoRA? Are the reported results based on the average of multiple runs with different random seeds to explain the variance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HypLoRA, a low-rank adaptation technique designed for large language models (LLMs) operating in hyperbolic space. The authors begin by empirically demonstrating that token embeddings exhibit hierarchical structure, supported by measurements of hyperbolicity. Next, they explore the relationship between the distribution of the token frequency and hyperbolic geometry. Finally, they introduce a low-rank transformation directly formulated in the Lorentz model of hyperbolic space and present experimental results validating its effectiveness.\n\nSTRENGTHS AND WEAKNESSES:\n## Strength\n\nLLM attracts huge attention in recent research, and using the hyperbolic geometry tool to analyze LLM is well-motivated and interesting for both communities. The paper lays out a step-by-step analysis, justifying the finding of the hierarchical structure of the token and using the hyperbolic geometry tool to enhance the low-rank transformation. I’m not from the LLM background and could not judge the significance from this aspect, but I believe integrating and using the hyperbolic tools in a new domain and analyzing the geometric insights should be appreciated.\n\n## Weakness\n* The explanation of the “cancellation effect caused by the exponential and logarithmic maps” is unclear. If operations are done in hyperbolic space and then mapped to or from the tangent space, the overall transformation is not necessarily the same. So why and how does $\\log_o^K(\\exp_o^K(x))$ appear here? This needs more clarification.\n* The motivation for the design of the proposed direct Lorentz low-rank transformation is not well explained.  While the authors want to explore the tool of hyperbolic geometry for LLM, there are a few questions raised in the design when examining the hidden states of LLM. First, why not transform the hidden states of LLMs from Euclidean space to hyperbolic space and stay in hyperbolic space instead of transforming the state back to Euclidean space? Second, why is transforming the space-like dimensions desired and not another alternative way? \n* In the quantitative analysis of hyperbolicity (lines 194–196), the authors mention using GNNs to embed random graphs with known hyperbolicity. However, it’s not clear which GNN model is used or how the experiment is set up. Appendix C does not provide enough detail. \n* In lines 199–202, the authors describe computing hyperbolicity “at the prompt level”, but it is not clear to me. What is the prompt level?  Did you compute the hyperbolicity of x? What is the representation you computed for hyperbolicity?  What exactly is measured and how? This part is hard to follow.\n* While the authors briefly mentioned the linear transformation in lines 266-268, the description is vague and lacks explanation.\n* In line 314, the paper refers to Appendix G for a “comprehensive comparison with additional methods,” but Appendix G contains GPU usage details. The comparison is either missing or mislabeled.\n* The paper uses the Lorentz model for the main transformation, but when discussing the connection between power-law distribution and hyperbolic geometry, it suddenly switches to the Poincaré disk model without any introduction or reasoning. For readers unfamiliar with hyperbolic geometry, this may cause confusion and feel abrupt.\n* The connection between the power-law distribution and hyperbolic geometry was not clear. First, the author uses the Poincaré disk model, which is in 2D, and the token embedding in hyperbolic dimension is not; how to generalize it for higher dimensions is unclear. Second, the derivation was not clear on how to consider token embedding with polar coordinates. Also, the authors use “radial coordinate (correlating with token frequency) ”  and “angular coordinate (encoding semantic similarity)”, and then state radial distribution, but it’s unclear how it is derived. Each statement in lines 224-225 was just given without any explanation or derivation. \n* In Equations 6, 7, and the following lines, the symbol $\\sim$ is used without explanation of what kind of approximation or relation it indicates.\n* In line 263, the paper refers to Appendix E.1 for the stereographic projection formulas, but the actual definitions of $\\Pi_{E\\rightarrow H}$ and $\\Pi_{H\\rightarrow E}$ are in Appendix E.2.\n* Appendix F consists of the transformation analysis. However, it was not clearly referred to in the main text, and the context of the properties introduced in Appendix F is unclear, e.g., using proof to introduce a property\n* There is no proof of Proposition 1, and the explanation of the dependency on the L2 norm was not clear. Following the authors pointing to Appendix F.1, it was hard to follow how “the norm-dependent, higher-order modification enables HypLoRA to capture hierarchical relationships”\n\nQUESTIONS:\nSee weakness above.\n\nLIMITATIONS:\nIn the paper checklist, the authors mention that limitations are discussed in the Conclusion (Section 6). However, the point raised there is a general observation about hyperbolic methods. The paper does not clearly identify under what conditions the method might fail or perform poorly. A more detailed discussion of the method’s limitations (such as sensitivity, dataset characteristics) would help clarify the scope and applicability of the approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first provided an empirical observation showing that the frequency of tokens tends to be inversely proportional to the norm of their embedding vectors in several LLMs on various datasets, indicating an underlying hierarchical structure. The paper then measures the hyperbolicity for the token embeddings of various LLMs to validate that conjecture. The paper then proposed to fine-tune LLMs by performing low-rank transformation of the input of each layer on the hyperbolic manifold. Extensive experiments are conducted on several reasoning datasets with various pretrained models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The consideration of transforming the token embeddings in the hyperbolic space is reasonable and interesting. \n2. The suggestion of an underlying hyperbolic structure of token embeddings from pretrained models are informative. \n3. The usage of the direct Lorentz Low-rank Transformation is interesting.\n\nWeakness:\nThe effectiveness and the motivation of the proposed method, HypLoRA, seem a bit unclear. \n\nPlease see the section of Questions.\n\nQUESTIONS:\n1. Comparing the final representations between HypLoRA and Lora, eqns (30)-(32) and eqns (49)-(52), it seems that instead of injecting $BA$ matrix into the original weight, HypLoRA injects $c BA$ where $c$ is some constant that depends on the input norm $\\lVert x \\rVert^p$ for some integer $p$. In the end, the representative power of LoRA and HypLoRA seems to be equivalent, i.e. the solution sets of HypLoRA is covered by LoRA. I am wondering what might be the advantage of doing all the transformations from and to the manifold, compared with directly injecting, e.g. $\\lVert x \\rVert BA$, and optimize over B and A? \n\n2. Assuming the tokens follow the hyperbolic structure, it seems natual to do any of the following (or simultaneously):\n\n    1) Fine-tune the embedding space by applying riemmanian optimization algorithms on the hyperbolic manifold, and fine-tune the other parameters separately either in the eculidean space or via the proposed linear transformation on the hyperbolic manifold.\n    2) Train a hyperbolic network that does all the transformations on the hyperbolic space before the final layers.\n    \n\nFor now, HypLoRA freezes the embedding space and the pretrained weights, and injects a learnable linear transformation (by first projecting to the hyperbolic manifold and projecting back to the euclidean space), and merges it with the original eucliean linear transformation. I am wondering what is the logic behind and why it offers more effective training procedure, considering the similar amount of parameters of other fine-tuning algorithms.  \n\n3. The frequency of the tokens seem to be largely depend on the corpus or the context considered. I am wondering how the observation that the frequency of tokens tends to be inversely proportional to the norm can help for fine-tuning on new datasets. \n\n4. I am wondering how you choose or estimate the curvature constant $K$ and how it affects the final result.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores whether operating in Euclidean space is optimal for token embeddings in large language models. The authors find that token embeddings exhibit non-Euclidean characteristics: high-frequency tokens cluster near the origin, and embeddings display strong hyperbolicity, suggesting a latent tree-like structure. To exploit these properties, they propose HypLoRA, a novel method for low-rank fine-tuning directly in hyperbolic space. Experiments show that HypLoRA improves LLM performance on some benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The proposed method is interesting and novel, leveraging the geometric properties of token embeddings.\n- The experimental evaluation is reasonably comprehensive, covering multiple models and datasets to support the claims.\n- The paper is generally well-written and well-structured with clear explanations.\n\n\nWeaknesses:\n- The analysis of hierarchical structure and hyperbolicity appears to focus predominantly on the input embeddings. It would strengthen the work to examine whether these non-Euclidean properties persist or transform across deeper layers of the model.\n- Additional discussion in the Questions section below\n\nQUESTIONS:\n- Do the authors have any insights or empirical results regarding how the hierarchical structure and hyperbolic properties of embeddings evolve across deeper layers of the model, as well as with the token context? It could also be interesting to see how such behavior may differ when using HypLoRA as opposed to other parameter efficient finetuning methods.\n- Is there any relationship between the degree of hyperbolicity in an LLM’s embeddings and the amount of performance improvements observed with HypLoRA?\n- Could the insights from this work be applied beyond the low-rank adapter setting, for example in other fine-tuning strategies or pretraining?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}