Hyperbolic Fine-Tuning for Large Language Models
Authors: Menglin Yang
Ram Samarth B B
Aosong Feng
Bo Xiong
Jiahong Liu
Irwin King
Rex Ying
Venue: NeurIPS 2025 spotlight ()
Keywords: Large Language Models
Hyperbolic Space
Low-Rank Adaptation
Embedding Space
Primary area: deep_learning
TL;DR: A novel method to efficiently fine-tune Large Language Models in hyperbolic space, unlocking their latent tree-like structures and significantly boosting complex reasoning performance by adapting directly on the hyperbolic manifold.

Abstract: Large language models (LLMs) have demonstrated remarkable performance across various tasks. However, it remains an open question whether the default Euclidean space is the most suitable choice for LLMs.
In this study, we investigate the geometric characteristics of LLMs, focusing specifically on tokens and their embeddings.
Our findings reveal that token frequency follows a power-law distribution, where high-frequency tokens (e.g., the, that ) constitute the minority, while low-frequency tokens (e.g., apple, dog) constitute the majority. Furthermore, high-frequency tokens cluster near the origin, whereas low-frequency tokens are positioned farther away in the embedding space.
Additionally, token embeddings exhibit hyperbolic characteristics, indicating a latent tree-like structure within the embedding space.
Motivated by these observations, we propose **HypLoRA**, an efficient fine-tuning approach that operates in hyperbolic space to exploit these underlying hierarchical structures better.
HypLoRA performs low-rank adaptation directly in hyperbolic space, thereby preserving hyperbolic modeling capabilities throughout the fine-tuning process.
Extensive experiments across various base models and reasoning benchmarks, specifically arithmetic and commonsense reasoning tasks, demonstrate that HypLoRA substantially improves LLM performance.

Hyperbolic Fine-Tuuiig for Large Language Models
Menglin Yang 1.2 Ram Samarth B B², Aosong Feng', , Bo Xiong Jiahong Liu Irwin King", Rex Ying" HKUSTTGZZ): ²KKUST; IIdian Institute of Science;
Yale University; Stanford University; Thee Chinese University of Hong Kong
menglin. yangSoutlook, com, rom,ramammartiin wamsamarthbbeissccooo in, aosong www.gg.engggale..om edu,
xi ziongbo@stanford edu, {jhliu22, king} cuhk.
rex. mex.yinngyale... edu Code: https://github. com/marlin codes/HypL.oe
Project! cttp:://hyperroolicleammm github. wtwwook//ypllorr..zz....
Abstract
Large language models (LLMS) have demonstrated remarkable performance across various tasks. However, it remains an open question whether the default Fuclidean space the most suitable choice for L.Ms.. In this study, we investigate the geometric characteristics of LLMs, focusing specifically tokens and their em- beddings. Our findings reveal that token frequency follows power-law distribution, where high-frequency tokens (e.g., "the," " "that") constitute the minority, while low-frequency tokens (e.g., "apple," "dog") constitute the majority. Furthermore, high-frequency tokens cluster near the origin, whereas low-frequency tokens are positioned farther away in the embedding space. Additionally, token embeddings exhibit hyperbolic characteristics, indicating a latent tree-like structure within the embedding space. Motivated by these observations, we propose HypL an efficient fine-tuning approach that operates in hyperbolic space to exploit these underlying hierarchical structures better. HypLoRA performs low-rank adaptation directly in hyperbolic space, thereby preserving hyperbolic modeling capabilities throughout the fine-tuning process. Extensive experiments across various base mod- els and reasoning benchmarks, specifically arithmetic and commonsense reasoning tasks, demonstrate that HypLoRA substantially improves LLM performance.
Introduction
Large language models (LLMs) such as GPT-4 [1), LLaMA [2), Gemma [3], and Qwen [4] have demonstrated remarkable capabilities in understanding and generating human-like text [5, 6. 7]. Despite their impressive capabilities, these models often rely on Euclidean geometry for token representation, which may inadequately capture the inherently complex and hierarchical nature of real-world data structures [8, 9, 10, 11, 12, 13]. Consider how words naturally organize into nested categories with varying levels of abstraction: abstract concepts like "fruit" occupy higher positions in the semantic hierarchy, while specific instances such as "apple" or "banana" populate the lower levels. Representing such structures effectively crucial for understanding the semantics of language in L.LMs.
Recent advancements suggest that non-Euclidean geometries, particularly hyperbolic spaces [11, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23), offer promising alternatives for modeling hierarchical data. Hyperbolic space, distinguished by its negative curvature, is especially well-suited for representing tree-like hierarchical data due to its exponential volume growth and geometric prior. This geometric property makes hyperbolic space particularly capable for tasks involving complex, hierarchically structured information.
"Conresooningg author: Menglin Yang.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

GSNESK
GSMIK
10
AQUA
AQUA
101
Y-1.87
1.89
Oruat tor sall)
Dauze (ing solld
18
18
10
101
39
04
06
10
10
12
35
0.3 0.6
0.5
0.6
Tokes Fraquency Dog scai))
Tokes Noets
Tokea Froquenes Oog sciic
Takas Norm
Figure 1: Token frequency distribution and token frequency vs. norm analysis for GSM8K (Group 1) and AQuA (Group 2) datasets in LLaMA3-8B. For each group, the left panels show the token frequency distributions (power-law distribution), while the right panels illustrate the relationship between token frequency and the corresponding norms. This visualization reveals the underlying geometric structure of the token embeddings. For additional data analysis and visualizations, please refer to Appendix A.
Proposed Analysis Framework. In this work, we first delve into how LLMs interact with token embeddings and explore the extent to which these embeddings exhibit non-Euclidean characteristics. We approach this from both a global and local perspective. At the global level, we analyze the overall distribution of tokens by frequency and investigate how these frequencies are arranged across the embedding space. At the local level, we measure the hyperbolicity of the metric space spanned by each input prompt, where the hyperbolicity serves as a proxy for evaluating the distance or dissimilarity between the underlying embedding structure and a tree-like hierarchy [24, 25, 16].
Our analysis in Section 4 reveals several key insights. Globally, token frequency follows power- law distribution, where high-frequency tokens (e.g., "the," "that") constitute the minority, while low- frequency tokens (e.g.. "apple," "dog") constitute the majority. Power-law distributions are consistent with, and can naturally arise from, underlying hierarchical or branching generative mechanisms [12, 26, 27).2 Besides, high-frequency tokens (e.g., abstract concepts, function words) tend to be located near the origin of the embedding space, while low-frequency tokens (c.g., specific terms) are farther away, as demonstrated in Table 1. Locally, demonstrates that L.M token embeddings in each prompt exhibit significant tree-like properties.
investigation of hyperbolicity (6 values) in Table 2
Based on our findings above, a natural consideration is to develop hyperbolic LLMs that explic- itly incorporate a hyperbolic inductive bias?. However, training LLMs from scratch resource- intensive [29, 30, 31J. As more resource-efficient alternative, we propose build the first low-rank adaptation fine-tuning method in hyperbolic space. This approach is particularly advantageous given that existing LLMs are all Euclidean, and not all downstream tasks require hyperbolic geometry in their tine-tuning. By employing hyperbolic adapters on Euclidean LLMs for specific tasks, we can leverage the benefits of both geometries while maintaining computational efficiency.
Challenges. Adapting LLMs non-Euclidean embedding spaces with classic techniques, i.e., apply- ing exponential and logarithmic maps within tangent space [32, 33, 34, 35, 36] for weight adaptation problematic in this case. This approach fails to fully capture the hyperbolic geometry, as the expo- nential and logarithmic maps are mutually inverse and can be canceled with consecutive operations*. Consequently, the inherent properties of the hyperbolic space are not effectively preserved, limiting the potential benefits of incorporating non-Euclidean geometries into the adaptation process.
Proposed Method. To address this limitation, introduce HypLoRA to perform low-rank adapta- tion directly on the hyperbolic manifold without transformation to the tangent space, thus preserving hyperbolic modeling capabilities and counteracting the cancellation effect. HypLoRA integrates hy- perbolic geometry into existing LLMs, implicitly introducing high-order interactions and accounting
PPower-law scaling alone does not uniquely identify the underlying structure or mechanism; additional evidence is needed to support a hierarchical interpretation [28].
'The connection hetween power-law distribution and hyperbolic geometry is elaborated in Section 4.3.
*The cancellation effect occurs because standard hyperbolic neural network [33, 32] approaches apply transformations in the tangent space the origin, requiring the sequence: Euclidean embedding - exponential map to hyperbolic space + logarithmic map to tangent space - linear transformation - exponential map back to hyperbolic space - projection to Euclidean space. When these operations are chained together, the maps are mutually inverse and effectively cancel out, reducing the entire sequence approximately the original Euclidean transfomation BAx without preserving the beneficial hyperbolic geometry.

---

for token hierarchies, enabling them to benefit from hyperbolic characteristies while minimizing additional computational costs.
To summarize, our main contributions are twofold: (1) We conduct a comprehensive investigation into the geometric characteristics of token embeddings in L.Mss, revealing their inherent tree-like structure and strong hyperbolic properties. (2) We propose HypLoRA, parameter-efficient fine-tuning method that integrates hyperbolic geometry into LLMs while keeping aligned with the Euclidean LLM framework. We conduct extensive experiments on various models and different tasks, specifically arithmetic reasoning and commonsense reasoning, demonstrating clear advantages over competitive baselines.
2 Related Work
Hyperbolic Representation Learning and Foundation Models. Hyperbolic geometry has been
successfully applied to various neural network architectures and models [19. 21, 18], including shal- low hyperbolic neural networks [15, 33, 37, 38, 39, 40], hyperbolic CNNs [41, 42, 43], hyperbolic GNNs [32, 44, 45, 46], and hyperbolic attention networks or Transformers [47, 37, 38, 48]. These models leverage the inductive biases hyperbolic geometry to achieve remarkable performance on various tasks and applications [32, 22, 23, 49, 16, 17, 50, 51, 52, 53]. Recent efforts have focused on adapting LLMs and CLIP [54] to hyperbolic spaces. Key advancements include developing more expressive hyperbolic image-text representations [55], enabling compositional entailment learning for deeper vision-language understanding [56], designing safety-aware hyperbolic frameworks for content moderation [57], and creating core modules facilitate the construction of novel hyper- bolic foundation models [58]. While these adaptations show promise, training LLMs from scratch remains computationally expensive [59, 60]. The computational complexity increases further when considering Riemannian optimization [59, 60, 61] and additional hyperbolic operations, like Möbius addition.
Geometric Analysis of Language Model Embeddings. Prior work has made important observations about the geometry of embeddings that helped shape and motivate research. Reif al. [62] demonstrated that BERT embeddings contain distinct syntactic and semantic subspaces and showed evidence of tree-like parse structures, while Gao al. [63] revealed that token embeddings tend to cluster in a narrow cone during training, leading to representation degeneration. Building on these geometric insights, Rudman et al. [64] introduced IsoScore to formally quantify how uniformly embeddings utilize the ambient vector space. Additionally, Puccetti et al. [65] analyzed outlier dimensions in Transformers and showed their correlation with token frequencies. While these works provide crucial foundations for understanding embedding geometry, our work differs in that we specifically quantify and leverage the natural hyperbolicity of token embeddings.
Earameter-Efficient Fine-Tunine Tuning (PEFT) and LoRA. Fine-tuning LLMs [66, 1, 2] for downstream tasks poses significant challenges due to their massive number of parameters. To address this issue, PEFT methods have been proposed, which aim to train a small subset of parameters while achieving comparable or even better performance compared full fine-tuning. PEFT methods can be broadly categorized into prompt-based methods [67, 68, 69], adapter-based methods [70, 71], and reparameterizzation based methods [31, 72. 73]. Among these, the reparameterization based LoRA [31] has gained significant attention due to its simplicity, effectiveness, and compatibility with existing model architectures. Variants of LoRA, such as LoRA+174], DoRA [75], and AdaLoRA [76], have been proposed to improve its performance and efficiency. Recent research has also investigated ensembles of multiple LoRAs [77, 78] and quantization techniques [79, 80, 81]. The proposed method is foundational algorithm that is orthogonal to existing approaches and can potentially be combined with various LoRA variants to exploit their complementary strengths and achieve superior
performance.
3 Preliminary
This section introduces the key concepts used in our study, including the Lorentz model of hyperbolic geometry and the LoRA adapter.
Hyperbolic Geometry. Unlike flat Fuclidean geometry, hyperbolic geometry is characterized by constant negative curvature. We utilize the Lorentz model, also known as the hyperboloid model due
3

---

to its ability to effectively capture hierarchical structures and maintain numerical stability [14, 82]. The Lorentz model in n dimensions with curvature (K/K(K 0) defined as:
L = fx R"+! (x,x) = - K, > 0},
(1)
where ( e is the Lorentzian inner product, given by: (x,y)c Tufo EE= Ziyi
Tangent Space. In the Lorentz model Ck. the tangent space point is denoted Tx T It is defined as the set of all vectors u that are orthogonal to x under the Lorentzian inner product:
TxCa = {u [ IR=+1 (u,x)
0}. =0.
(2)
To facilitate projection between the hyperboloid and its tangent spaces at x, one can utilize two critical mappings: the exponential and logarithmic maps. The exponential map at x, denoted exp projects vector from the tangent space Tx C back onto the hyperboloid. Conversely, the logarithmic map, denoted logx maps a point the hyperboloid to the tangent space x. The detailed formulas given in Appendix C.
LoRA Adapter. The LoRA adapter provides an efficient approach for modifying LLMs with minimal computational overhead. Instead of retraining the entire model, LoRA focuses on adjusting specific components within the model's 's architecture to transform practice, LoRA targets the weight matrices found in each Transformer layer of an LLM. Typically, the weight W of the Transformer, which resides in the dimensions g is adapted through low-rank approximation. This is achieved by introducing an additional term, AW, to the original weight matrix:
input x E Rd into an output z E TR* In
z = (LORA()) Wx + AWx = Wx + BAx.
Here, A E R2'dd and B E Rxr represent two smaller, learnable matrices where r is the rank of these matrices, which is significantly less than either d or This design choice ensures that r & min(d, k). thereby reducing the complexity of the model adaptation. During the fine-tuning process, only the matrices A and B are adjusted, while the pre-existing weights W are kept frozen. This method significantly decreases the number of parameters that need to be trained, from d k to (d+ .r, enhancing the efficiency the fine-tuning process. As a result, LoRA enables the targeted adaptation of LLMs, allowing them to transform an input x into an output z while maintaining high performance and adapting to new tasks or datasets with a fraction of the computational resources typically required.
4 Investigation
In this section, we present an in-depth investigation of token embeddings in LLMs from both global and local perspectives. Our goal is to uncover the geometric structures underlying pretrained token representations, specifically examining the global distribution of token frequencies and their spatial arrangement, as well as the local hyperbolicity of token embeddings across various datasets.
4.1 Global Token Statisties
We begin by investigating the global distribution of token frequencies in the context of arithmetic reasoning datasets, focusing on datasets such as GSM8K [83]. AQuA [84], MAWPS [85], and SVAMP [86]. We also provide a broader analysis across different types of datasets and LLMs in Appendix A. Figure I (left) presents the distribution of token frequencies, with power-law exponent r R 1.9, as estimated by the powerlaw aww package [87]. In such distributions, the exponent y controls how quickly token frequencies decline: smaller values of y (closer to 1) indicate decay where frequent tokens dominate, while larger values signify a sharper decline, with most tokens being rare.
gradual
This pawer-law behavior consistent with the tree-like hierarchical nature of language [11, 16, 88, 52, 23J. frequency tokens represent specific or rare terms. This pattern aligns with hierarchical organization of the token space: abstract, high-frequency tokens cluster near the origin, while specific terms are positioned farther out, mirroring how general concepts sit at the core of semantic hierarchy with specialized terms at the periphery.
tokens often correspond to more abstract or general concepts, while low-

---

Table Mean, Minimum, and Maximum frequeney and norm values of token embedding different base models and groups. Group 1: to, in, have, that, and, is, for, Group 2: how, much, many, time, cost, Group 3: animal, fruit, number, color, size, Group dog, cow, apple, banana, 380, 480, purple, red, medium, small, large.
Model
Group
Frequency (Mean [Min~-Max])
Norm (Mean [Min~Maxx]
Group
4934.4 [1838 8G39]
3.160 [3.060 3.299]
Group
2709.4 [474- ~ 6681]
3.561 [3.488 ~ 3.627
Gemma-7B
Group
292.0 [34 ~ 1191]
3.765 [3.623 ~ 3.887]
Group
114.3 [25 ~ 284]
3.998 [3.660 4.520]
Group
4993.9 [1838 ~ 8547]
0.951 [0.793 ~ 1.060]
LLaMA-7B
Group
2712.6 [474 6683]
1.222 11.118 1.299]
Group
299.8 [34 1200]
1.325 [1.274 1.428)
Group
139.1 [26 ~ 286]
1.364 [1 [1.326 1.417]
Group
4937.4 [1838 ~ 86477
0.353 [0.330 0.396]
Group 2
2710.0 [474 ~ 6683)
0.456 [0.394 ~ 0. 0.499)
LL
Group
292.6 [34 ~ [191]
0.499 (0.452 0.549]
Group
97.1 [13 284]
0.569 (0.499 0.675]
Group1
4993.9 [1838 ~ 8547]
1.027 [0.833 1.255]
Group
2712.6 [474 ~ 0683]
1.429 [1.346 1.489]
LLaMA-13B
Group
299.8 [34 ~ 1200]
1.494 [1.453 ~ 1.532]
Group
139.1 1 [26 ~ 286]
1.501 [1.470 ~ 1.526]
Empirical Observation. To better understand the relationship between token frequency and their spatial arrangement within the embedding space, we calculate the average token frequency function of their distance from the origin. As shown in Figure (right), high-frequency tokens (c.g.. "the," "that") tend to have smaller norms, while low-frequency tokens (c.g.. "apple," "dog") have larger norms. Table presents representative tokens across different frequencies and norm ranges within the embedding space of different base models. We categorize tokens into four groups based on their linguistic function and specificity: Group contains and), Group 2 contains common question/quantity words (e.g.. how, much, many), Group 3 contains general category nouns (e.g., animal, fruit, color), and Group 4 contains specific instances (e.g., dog, apple, purple).
function words (e.g., to, is,
The results presented in Table I demonstrate several critical findings. First, we observe a statistically significant separation between functional/abstract words (Group 1) and specific terms (Group 4) across all models, with Group 1 consistently exhibiting the smallest embedding norms and highest frequencies, while Group 4 shows the largest norms and lowest frequencies. Second, the relative ordering of groups remains consistent across all examined models, with Group 1 < Group 2 < Group 3 < Group 4 in terms of embedding norms, despite absolute magnitude variations. Most notably, even across different architectural families (LLaMA vs. Gemma), the hierarchical organiza- tion principle remains preserved, though with different absolute scales, where Gemma-7B exhibits systematically larger embedding norms (mean Group 1 norm: 3. 160) compared to LLaMA models (mean Group 1 norm: 0.353 1.027), yet maintains the same relative hierarchical structure.
Conclusion (1) These findings suggest that the spatial organization of token embeddings reflects the inherent hierarchical relationships in language, supporting the hypothesis that token embedding in LLMs exhibits a tree-like structure, with spatial positioning aligned with token frequency and specificity. It is worth noting, however, that a power-law distribution of token frequency alone does not guarantee the emergence of a hierarchical token embedding, as it also depends on the training objectives. Our analysis demonstrates that the hierarchy is strongly correlated with token frequencies, which can be understood through the lens of I.L.M'' tokenization and co-occurrence pattern learning during training [89]. While the exact mechanisms underlying this relationship require further investigation in future work, the spatial distribution of token embeddings remains crucial as it provides the primary motivation for our methodological approach.
4.2 8- ò-Hyperbolicity of Local Token Embeddings
To rigorously quantify the hierarchical nature of token embeddings, we further examine the 8- hyperbolicity of the space spanned by the token embedding. 8-hyperbolicity, introduced by Gro- mov [90], is measure that captures the degree to which a metric space deviates from exact tree
5

---

Table 2: Comparison of 5-Hyperbolicity across various metric spaces and datasets. The left table provides reference values for baseline metric spaces, allowing for a clearer interpretation of hyperbol- icity in the analyzed datasets in the right table.
Metric Space
Hyperbolicity())
Hyperbolicity(0)
MAWPS
SVAMP
GSM8K
AQuA
Sphere Space
0.99 0.01
LLaMA-7B
+ 0.02
0.09 0.01
0.10 + 0.01
0.10 0.01
Random Graph
0.62 0.34
LLaMA-13B
0.08 + 0.01
0.09 + 0.01
0.09 0.00
0.10 0.01
PubMed Graph
0.40 0.45
Gemma-7B
0.11 + 0.01
0.11
1:0000000
0.11+0.00
0.12 0.01
Scale-free Graph
0.00
LLaMA3-8B
0.06 -: 0.01
0.07
0.01
0.07 0.01
0.08 0.01
Tree Graph
0.00
Averuge
0.08 + 0.01
0.09 0.01
0.09 0.01
0.10
0.01
structure. Lower values of 8 imply a space more similar to a perfect tree, while higher values indicate deviation from a tree-like structure.
We compute 9-hyperbolicity using the four-point condition, which compares the Gromov products between any four points o, b, c, and wo in the metric space. Specifically, the hyperbolicity is defined as:
la, c/w 2 min((a, b)w, [b, clw) 5,
(4)
where the Gromov product (a, bl is:
1
[a, bl = 2 (d(a, w) d(b, w) d(a,
(5)
Quantitative Analysis. To measure the hyperbolicity of token embeddings. we apply this algorithm to various open-souree LLMs. Following the methodologies proposed by Khrulkov al. [16] and Cetin et al. [17], we estimate 5-hyperbolicity using the efficient algorithm introduced by Fournier et al. [91]. To ensure scale invariance, we normalize by the diameter of the embedding space, diam (X). yielding a relative measure: 8rel diam(X) This relative measure ranges from 0 to 1,
26
with values closer to indicating a highly hyperbolic (tree-like) structure, and values near indicating non-hyperbolic, flat structure. We employ Euclidean distance as measure of the shortest distance, maintaining the same computational paradigm as previous works [16, 17]. To further validate the correctness of this approach, generate a series of random graphs with predefined hyperbolicity, embed them using a two-layer graph neural network (GNN) [92]. and then compute the hyperbolicity. Details of this process provided in Appendix B. Our experiments reveal positive correlation between the hyperbolicity of the embeddings and the original graphs. Consequently, we utilize this method as a proxy for estimating the hyperbolicity of token embeddings. In our analysis, hyperbolicity at the prompt level, treating each token within a prompt as a point in the metric space spanned by the embeddings. By averaging the hyperbolicity across all prompts, we assess the overall hyperbolic structure of token embeddings in each dataset.
calculate
Conclusion (2) Our results, as shown in Table 2, reveal that token embeddings exhibit significant
hyperbolicity, suggesting that the embedding space has a strong tree-like structure. This observation further corroborates our findings from the global token statistics, where the arrangement of tokens in the embedding space mirrors hierarchical relationships seen in language data. We also provide the hyperbolicity analysis of the final hidden layer in Appendix A.3.
4.3 Connection between Power-law Distribution and Hyperbolic Geometry
Having established both the global power-law distribution (Section 4.1) and local tree-like geometry (Section 4.2) of token embeddings, we now examine the theoretical connection between these two observations.
The observation of a power-law distribution in token frequencies, as discussed in Section 4, is not merely statistical curiosity. It has deep connections to the underlying geometry of the data, particularly to hyperbolic spaces, which are well-suited for representing hierarchical structures [11, 12, 93, 94]. For instance, Nickel and Kiela [11] highlighted that the existence of power-law degree distributions can often be traced back to hierarchical structures. Similarly, Ravasz and Barabási [88] established that the scaling law P(k) ~ h- can signify the co-existence of a hierarchy of nodes with varying degrees of clustering. Krioukov et al. [12] further strengthened this connection by showing that the exponent of the power-law degree distribution is a function of the hyperbolic space curvature. Building on this geometric understanding, Papadopoulos et al. [94] demonstrated that

---

complex (scale-free) network topologies naturally emerge when networks grow within an underlying hyperbolic metric space, and importantly, that the resulting hyperbolic embedding of these dynamic scale-free networks facilitates highly efficient greedy forwarding.
To formalize this connection with hyperbolic geometry, can consider embedding tokens in hyperbolic space. A common model for hyperbolic space is the Poincaré disk model (HH2) with curvature K -1.. In such a space, both the cireumference ((r) and area A(r) of a circle of radius r exhibit exponential growth:
(r) 2n sinh(r) ~ e² as r - 00,
(6)
A(r) 2T (c - e" as r 00.
(7)
If we consider token embeddings in a hyperbolic space with polar coordinates (r, 0), where r E R+ the radial coordinate (correlating with token frequency) and E [0, 2T] the angular coordinate (encoding semantic similarity), the radial distribution of tokens follows p(r) e-Sr where Ç <> 0 relates to the hyperbolic curvature K. The frequency function k((r) for tokens at radius r is then given by k(r) ~ e eT. Given k(r) ~ e", we have r ~ - In k. and thus dr the radial distribution p(r) ~ e Sr ~ K.,,, this yields:
~ k Combined with
dk
dr
P(k) ~ p(r)
~ 15 k- ~ (1-5)
(8)
dk
Following the parameterization of Krioukov al. [12], the power-law exponent y relates to the curvature parameter via 2/Ç + 1 in the context of complex networks. This relationship underscores the theoretical connection between the power-law behavior observed in token frequencies and the inherent hyperbolic geometry of the embedding space. Since hyperbolic models such as the Poincaré ball model and the Lorentz, model are isometric, this conclusion can be extended to other hyperbolic models.
Hyperbolic space offers distinct advantages for modeling language hierarchies, especially when addressing the structural and spatial constraints of token co-occurrence: (1) Separation of Low- Frequency Tokens. Tokens with low frequencies, which typically represent more specific or granular concepts. require clear separation from each other to maintain semantic clarity. (2) Proximity to High-Frequency Hypernyms. Simultancously, these low-frequency tokens should remain close to their corresponding suited for capturing these dual constraints due to exponential volume growth, which inherently supports hierarchical structure and allows for ample separation of specific entities while keeping them close to their parent categories. This contrasts with Euclidean space, where such arrangements can lead to crowding or distortion of distances.
hypernyms or function words. Hyperbolic space is uniquely
Overall Conclusion. Through these analyses, demonstrate that token embeddings in LLMs exhibit hierarchical organization and significant hyperbolicity. This understanding not only sheds light on the geometric nature of token embeddings but also motivates the development of methods that can better capture and preserve these underlying geometric properties."
5 Hyperbolic Fine- Tine-Tunigg for LLMs
The core technique the LoRA adapter involves matrix transformations. The conventional approach to implementing these transformations in the Lorentz model of hyperbolic geometry is through operations in the tangent space, while maintaining the learnable weights Fuclidean space [33, 32).
"The derivation uses the Poincaré ball model for its intuitive geometrie interpretation. However, all models of hyperbolic space, including the Poincaré ball, the Lorentz (hyperboloid) model, and the Klein model, are isometrically equivalent, preserving geodesic distances under explicit diffeomorphisms [95, 96]. Since method (Section 5) operates in the Lorentz model, the theoretical connections established here between power-law distributions and hyperbolic geometry remain fully applicable.
'While our analysis reveals consistent hierarchical patterns across multiple LLMs, several limitations should be noted. First, our investigation focuses on arithmetic reasoning and commonsense datasets (please check Appendix A details); the generalizability to other domains (e.g.. code, multilingual text) requires further validation. Second, the relationship between token frequency and embedding norm, while strong, is correlational rather than causal. Third, our 5- h-hpeebbolicity measurements are computed at the prompt level: corpus-level analysis may yield different insights.
7

---

Table 3: Comparison of various LLMs arithmetic reasoning tasks. The percentage following each dataset indicates the proportion of prompts relative to the total number of inference prompts. M.AVG represents the micro-average accuracy (since the datasets imbalanced). For more adapter comparisons, please see Appendix F.
Base Model
PEFT Method
Params (%)
MAWPS(8.5%)
SVAMP(35.6%)
GSM8K(46699%
AQuA(9.0%)
MLAG
GPT-3.5
None
56.4
LLaMA-78
LoRA
0.83
81.9
18.2
38.3
18.5
13.7
HypLoRA (Ours)
0.83
LLaMA-138
HypLoRA (Ours)
LoRA
0.67
83.5
54.7
18.5
18.5
51.0
Gemma-7B
LoRA
0.79
91.6
76.2
06.3
28.9
68.6
HypLoRA (Ours)
0.79
LLMMA3-88
LoRA
0.70
92.7
78.9
70.8
30.4
71.9
HypLoRA (Ours)
0.70
80.5
74.0
Gomma3-4B
LoRA
L01
90.8
77.3
72.3
50.8
73.7
HypLoRA (Ours)
Qwen2.5-7B
LoRA
0.71
90.8
81.4
78.6
68.1
80.8
HypLoRA (Ours)
0.71
However, this approach presents a significant challenge for our application. Since the hidden states of LLMS exist Euclidean space, we would need to project these states to hyperbolic space and subsequently map them back the tangent space. This process results in consecutive logarithmic and exponential mappings (logo (exp² (x))). which effectively cancel each other out, reducing the method to the original LoRA approach and nullifying any benefits from hyperbolic geometry.
Direct Lorentz Low-Rank Transformation (LLR). To overcome this limitation, propose direct Lorentz Low-Rank Transformation (LLR) that operates directly on the hyperbolic space without relying on tangent space mappings. This approach allows to perform low-rank adaptation while preserving the advantages of hyperholic geometry:
z E (X = Wx + A
= E
I (LLR(BA, II exp (x²)),
where LLR. represents the direct Lorentz Low-Rank Transformation that operates directly on the hyperbolic representation x
I7
exp (x
LLR(BAX*)) -(v BBx K BAx""),,
(10)
where x is the space-like component of x i..., x =x without the first time-like dimension (oo1] The operators II and I1 represent projections from local tangent space to hyperbolic space (e.g., exponential map) and from hyperbolic space to local tangent space (e.g.. logarithmic map). respectively. The detailed formulas are provided in Appendix C. It can verified that LLR(BA, E L", ensuring that our transformation remains within the Lorentz model of hyperbolic space. This transformation primarily affects the space-like dimensions, functioning similarly pseudo-Lorentz rotation [37]. The linear transformation is inspired by hyperbolic neural networks [37, 48, 97]. For efficient integration with LLMs, the transformation removes normalization and non-linear activation terms in [37], varying curvatures in [48], and orthogonal constraints in [97]. Our main contribution lies in applying hyperbolic low-rank adaptation for LLMs, while the specific linear transformation itself is flexible, and other transformations the manifold could also be compatible with our approach.
H
By adapting in the hyperbolic domain, HypLoRA captures more complex hierarchical relationships than traditional Euclidean-baue methods, as detailed in Proposition 1. Additionally, the low-rank nature of the adaptation matrices A and B promotes parameter efficiency, making HypLoRA well- suited for LLMs.
Time Complexity Analysis. HypLoRA has similar theoretical time complexity as the Euclidean LoRA, which O(r- (d k:)), where d and represent the input and output dimensions, respectively. However, in practical implementation, HypLoRA introduces additional computations due to the space mapping. These additional operations, nevertheless, can completed within O(N) where N the number of input tokens.

---

Table 4: Comparison of various LLMs on commonsense reasoning tasks. These datasets contain
relatively similar amounts of data, so we use AVG to represent the average accuracy.
Base Model
PEPT Method
Purams
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-
ARC-
OBQA
AVG
GPT-3.5
None
None
73.1
85.1
68.5
78.5
06.1
89.8
79.9
71.8
77.0
LLAMA3-BB
HypLoRA (Ours)
LaRA
0.70
0.70
74.1
70.8
85.2
87.6
79.9
80.6
91.7
94,5
84.7
$4.3
84.2
90.4
71.2
81.2
85.2
79.0
84.8
90.8
Gemmu3-4B
LoRA
1.04
68.1
53.2
77:2
88.8
80.5
81.5
69.8
83.6
79.5
HypLoRA (Ours)
1.01
70.0
84.3
79.2
91.5
80.3
89.1
75.9
86.4
82.5
Qwen2.5-78
LakA
0.71
73.4
89.5
79.5
93.6
84.1
87.0
86.2
HypLoRA (Ours)
Proposition 1. Let E Rd denote the input token embeddings. The HypLoRA adaptation, applied to x, involves sequence of projection into hyperbolic space, a Direct Lorentz Low-Rank Transformation (LLR), and projection back to Euclidean space. Due to the non non-linear nature of these hyperbolic operations, the effective transformation applied by HypLoRA introduces higher-order terms with respect to As detailed in Appendix E, these terms exhibit explicit dependency on the 1.2 norm, [|x|22 2. of the imput embeddings. This norn-dependent, higher-order modification enables HypLoRA to capture hierarchical relationships in the embedding space, thereby achieving natural alignment with the underlying hyperbolic geometry of the token representations.
5.1 Experimental Settings
Datasets. Following the experiment setup outlined in [98], we utilize two high-quality datasets, Math10K and Commonsensel 70K, tailored for mathematical and commonsense reasoning, respec- tively. Math10K consists of training data from GSM8K [83], MAWPS, MAWPS-singe [85], and 1,000 samples from AQuA [84], augmented with ChatGPT-generated step-by-step rationales to reinforce reasoning capabilities. The test set includes GSM8K, AQuA, MAWPS, and SVAMP [86], ensuring no overlap with the training data. Commonsense 70K is constructed by reformatting samples from BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA using standardized templates that outline the task, content, and answer, resulting in 170K training samples. The test datasets drawn from the same sources, with strict separation from training samples. For fine-tuning methods, we compare with LoRA [31] and also make a comparison with other adapters in Appendix F, which also includes training details.
5.2 Experimental Results
Table summarizes our key experimental outcomes on arithmetic reasoning tasks, while Table presents results for commonsense reasoning benchmarks. Our primary comparison contrasts LoRA and HypLoRA to demonstrate the effectiveness of the proposed approach, with additional baselines provided in Appendix F.
Arithmetic Reasoning Performance. On arithmetic reasoning tasks, as indicated by results in Table 3. HypLoRA shows notable efficacy, especially on datasets recognized for their complexity, such as GSM8K, AQuA, and SVAMP. These datasets demand robust multi-step reasoning and nuanced understanding of numerical and textual relationships. For instance, on Qwen2.5-7B. HypLoRA achieves a substantial + +7.5 percentage point improvement in M.AVG (88.3% vs. 80.8%). with notable gains of +7.8% on SVAMP and +9.3% on GSM8K. The enhanced performance of HypLoRA in these areas aligns with design; by operating in hyperbolic space, can better model the hierarchical structure of problems and distinguish subtle yet critical differences in input embeddings. This is further corroborated by the theoretical analysis (Appendix E), which posits that HypLoRA introduces higher-order, norm norm-dependent terms. These terms allow the model to develop more refined sensitivity to token importance and inter-token relationships.
Commonsense Reasoning Performance. The robust performance of HypLoRA extends to common- sense reasoning, as detailed in Table 4. For the Gemma3-4B model, HypLoRA achieved an average accuracy of 82.5% across all datasets, surpassing LoRA's 79.5%. Similarly, on the Qwen2.5-7B model, HypLoRA obtained an average 87.0% compared to LoRA's 85.2%. These improvements are distributed across various commonsense benchmarks, including notable gains on datasets like ARC-c and OBQA for Gemma3-4B, and ARC-e, ARC-e, and OBQA for Qwen2.5-7B. Commonsense reasoning often relies on understanding implicit relationships and contextual nuances, which may not

---

Table 5: Results for varying cur- vature on the Gemma3-4B model
Figure 2: GPU (A100) usage during inference
Gemma 78
LLaMA3 8B
Dataset
K=0.5
K=1.0
GPU Hours
15-
HypLoRA DoRA
GPU Hours
10
HyploRA
DoRA
MAWPS
88.2
91.9
10-
SVAMP
83,9
80.3
6-
GSM8K
76.1
73.8
AQuA
53.5
52.7
M.AVG
77.8
73.8
GSMBK SVAMP
AQUA
MAWPS
GSMBK
AQUA MAWPS
always be explicitly hierarchical but still benefit from the richer representational capacity offered by hyperbolic geometry. The ability of HypLoRA to better discern these subtleties, likely due to the mechanisms described in Proposition contributes these observed performance gains, showcasing the broad applicability of hyperbolic fine-tuning.
The Impact of Curvature on Performance. Curvature in hyperbolic space is key hyperparameter in HypLoRA, directly affecting its capacity to model underlying structures and geometries. To evaluate its impact, we experiment with learnable curvature initialized with different curvature values on the Gemma3-4B model, as shown in Table 5, where the curvature is defined as -1/K. Our results demonstrate that curvature does influence model performance. For Gemma-7B and Gemma3-4B, a curvature value of 0.5 consistently yields the best overall performance across both arithmetic and commonsense reasoning benchmarks. Similarly, for LLaMA3-SB, 0.5 proves optimal. In commonsense reasoning benchmarks, a curvature of 1.0 performs best for LLaMA3-8B and Qwen2.5-7B.
Efficiency. In Section 5, we analyze the time complexity our approach, which remains consistent with that of LoRA. However, during actual inference, HypLoRA incurs additional computational overhead due to operations such as projections. These operations introduce some additional runtime, particularly for larger models. The GPU hours for inference four datasets are presented in Figure 2. Despite this overhead, our method demonstrates improved efficiency when compared to the previous competitive model, DoRA. Notably, HypLoRA still outperforms DoRA in terms of both runtime and overall efficiency. Besides, all models can be fine-tuned in approximately one hour for optimal training efficiency.
6 Conclusion
In this work, we investigated the non-Euclidean geometric properties inherent in LLMs, confirming their strong hyperbolic characteristics, which suggest underlying hierarchical structures. Building on these insights, we introduced HypLoRR,, hyperbolic low-rank adaptation technique. HypLoRA performs fine-tuning directly on the hyperbolic manifold. Extensive experiments show that HypLoRA arithmetic reasoning and commonsense tasks. By
significantly improves LLM performance leveraging the hyperbolic structure of the data, HypLoRA enhances the model's ability to capture and utilize intricate relationships, leading to better reasoning capabilities.
Broader Impact. Enhancing cessoning-orienter LLMs can help education, scientific assistance, and safer decision-support systems, but the same improvements may also accelerate misuse (e.g.. automating complex disinformation or amplifying biased advice) and increase energy consumption due to added hyperbolic projections. We therefore advocate releasing checkpoints and code with usage guidelines (as our public repo), tracking compute budgets when scaling HypLoRA further.
Acknowledgment
The authors would like express their sincere gratitude to the anonymous reviewers and the Area Chair for their valuable comments and insightful suggestions, which have greatly improved this work. We thank our colleagues and lab mates at HKUST(GZ), IISc, Yale, Stanford, and CUHK for motivating discussions on geometric learning for LLMs, and we acknowledge the administrators of the institutional GPU clusters that enabled the large-scale experiments. Part of this research was conducted while Menglin Yang was at Yale University, whose support and computing resources were instrumental in shaping the initial ideas of this project.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. GPT-4 Technical Report. arXiv preprint arXiv:2303.08772 087744 2023.
[2] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint Stti:2302.11977., 2023.
[3] Google Deepmind Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint wxXiv:2433008295 2024.
[4] An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:2412.15115, 2024.
[5] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a gencral-purpose natural language processing task solver? arXiv preprint urXiv:2302.064476. 2023.
[6] Yongliang Shen, Kaitao Song. Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. HuggingGPT: Solving Al tasks with ChatGPT and friends in Hugging Face. In Conference Neural Information Processing Systems (NeurIPS), volume 36, 2024.
[7] Jiahong Liu, Zexuan Qiu, Zhongyang Li, Quanyu Dai, Wenhao Yu, Jieming Zhu, Minda Hu, Menglin Yang. Tat-Seng Chua, and Irwin King. A survey of personalized large language models: Progress and future directions. arXiv preprint arXiv:2502.11528 2025.
[8] Michael M Bronstein, Joan Bruna, Yann LeCun, Arthur Szlam, and Pierre Vandergheynst. Geometric deep learning: going beyond euclidean data. IEEE Signal Processing Magazine, 34(4):18-42, 2017.
[9] Gregor Bachmann, Gary Bécigneul, and Octavian Ganea. Constant curvature graph convolu- tional networks. In International Conference on Machine Learning (ICML), pages 486-496. PMLR, 2020.
[10] Atsushi Suzuki, Atsushi Nitanda, Jing Wang, Linchuan Xu, Kenji Yamanishi, and Marc Cavazza. Generalization eтror bound for hyperbolic ordinal embedding. In International Conference on Machine Learning (ICML), pages 10011-10021. PMLR, 2021.
[11] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical rep- resentations. In Conference on Neural Information Processing Systems (NeurIPS), pages 6338-6347,200 2017.
[12] Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vahdat, and Marián Boguná. Hyperbolic geometry of complex networks. Physical Review Statistical, Nonlinear, and Soft Matter Physics, 82(3):036106, 2010.
[13] Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. International Symposium on Graph Drawing, pages 355-366. Springer, 2011.
[14] Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International Conference on Machine Learning (ICML), pages 3779-3788. 2018.
[15] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference Machine Learning (ICML), pages 1646-1655. PMLR, 2018.
[16] Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In IEEE/VV Conference on Computer Vision and Pattern Recognition (CVPR), pages 6418-6422 2020.

---

[17] Edoardo Cetin, Benjamin Chamberlain, Michael Bronstein, and Jonathan J Hunt. Hyperbolic deep reinforcement learning. arXiv preprint arXiv:2210.01542. 2022.
[18] Wei Peng, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyper- bolic deep neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[19] Menglin Yang, Min Zhou, Zhihao Li, Jiahong Liu, Lujia Pan, Hui Xiong, and Irwin King. Hyperbolic graph neural networks: A review of methods and applications. arXiv preprint 1rXiv:2202.13852 2022.
[20] Jiahong Liu, Xinyu Fu, Menglin Yang, Weixi Zhang, Rex Ying, and Irwin King. Client-specific hyperbolic federated learning. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), 2024.
[21] Pascal Mettes, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. Hyperbolic deep learning in computer vision: A survey. arXiv preprint arXiv:2305.06611 I, 2023.
[22] Menglin Yang, Min Zhou, Jiahong Liu, Defu Lian, and Irwin King. Href: Enhancing collabo- rative filtering via hyperbolic geometric regularization. In The Web Conference (WWW), pages 2462-2471, 2022.
[23] Menglin Yang. Zhihao Li. Min Zhou, Jiahong Liu, and Irwin King. Hicf: Hyperbolic informative collaborative filtering. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages
2022.
[24] Michele Borassi, Alessandro Chessa, and Guido Caldarelli. Hyperbolicity measures democracy in real-world networks. Physical Review E, 92(3):032812, 2015.
[25] W Sean Kennedy, Onuttom Narayan, and Iraj Saniee. On the hyperbolicity of large-scale networks. arXiv preprint 1007,0:337.0033 0.00000,01,..0 2013.
[26] Albert-Lászıó Barabási, Zoltán Dezsö, Erzsébet Ravasz, Soon-Hyung Yook, and Zoltán Oltvai. Scale-free and hierarchical structures in complex networks. In AIP Conference Proceedings, volume 661, pages 1-16. American Institute of Physics, 2003.
[27] Enrique Alvarez-Lacalle Beate Dorow, J-P Eckmann, and Elisha Moses, Hierarchical struc- tures induce long-range dynamical correlations in written texts. Proceedings of the National Academy of Sciences, 103(21):7966-7611 2006.
[28] Kai Nakaishi, Ryo Yoshida, Kohei Kajikawa, Koji Hukushima, and Yohei Oseki. Rethink- ing the relationship between the power law and hierarchical structures. arXiv preprint arXiv:2505. 2025.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051001 2017.
[30] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory opti- mizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16. IEEE, 2020.
[31] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, arXiv preprint Xrx: 100.:/20000008, 2021.
[32] Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolu- tional neural networks. In Conference on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[33] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In Conference on Neural Information Processing Systems (NeurIPS), volume 31, 2018.
[34] Menglin Yang, Min Zhou, Hui Xiong, and Irwin King. Hyperbolic temporal network embed- ding, IEEE Transactions Knowledge and Data Engineering (TKDE), 2022.
12

---

[35] Sungjun Cho, Seunghyuk Cho, Sungwoo Park, Hankook Lee, Honglak Lee, and Moontae Lee. Curve your attention: Mixed-curvature transformers for graph representation learning, arXiv preprint arXiv:2309.
2023.
[36] Xingcheng Fu, Yisen Gao, Yuecen Wei, Qingyun Sun, Hao Peng, Jianxin Li, and Xianx- ian Li. Hyperbolic geometric latent diffusion model for graph generation. arXiv preprini arXiv:2405.03 1888 2024.
[37] Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Fully hyperbolic neural networks. arXiv preprint arXiv:2105. 14686, 2021.
[38] Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic neural networks++. arXiv preprint arXiv:2006.082/11 2020.
[39] Xiaomeng Fan, Yuwei Wu, Zhi Gao, Mehrtash Harandi, and Yunde Jia. Curvature learning for generalization of hyperbolic neural networks. International Journal of Computer Vision (IJCV), pages 1-37, 2025.
[40] Yidan Mao, Jing Gu, Marcus C Werner, and Dongmian Zou. Klein model for hyperbolic neural networks. arXiv preprini arXiv:2410. (601:1330...813,, 2024.
[41] Ahmad Bdeir, Kristian Schwethelm, and Niels Landwehr. Hyperbolic geometry in computer vi- sion: novel framework for convolutional neural networks. arXiv preprint arXiv:2303. 15919, 2023.
[42] Max van Spengler, Erwin Berkhout, and Pascal Mettes. Poincaré resnet. In JEEE/CVF International Conference on Computer Vision (ICCV), pages 5419-5428, 2023.
[43] Raiyan R Khan, Philippe Chlenski, and Itsik Pe'er. Hyperbolic genome embeddings. arXiv preprint arXiv:2507 2025.
[44] Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. In Confer- ence on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[45] Yiding Zhang, Xiao Wang. Chuan Shi, Xunqiang Jiang, and Yanfang Ye. Hyperbolic graph attention network. IEEE Transactions on Big Data, 8(6): 2021.
[46] Menglin Yang, Min Zhou, Lujia Pan, and Irwin King. khgen: Fike-likeness modeling via continuous and discrete curvature learning. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 2965-2977 20772977,20 2023.
[47] Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, et al. Hyperbolic attention networks. arXiv preprint arXiv:/8055 /80500976, 2018.
[48] Menglin Yang, Harshit Verma, Delvin Ce Zhang, Jiahong Liu, Irwin King, and Rex Ying. Hypformer: Exploring efficient transformer fully in hyperbolic space. arXiv preprint arXiv:2407.01290. 2024.
[49] Jianing Sun, Zhaoyue Cheng, Saba Zuberi, Felipe Pérez, and Maksims Volkovs. HGCF: Hyperbolic graph convolution networks for collaborative filtering. In The Web Conference (WWW), pages 593-601, 2021.
[50] Zhenzhen Weng, Mehmet Giray Ogut, Shai Limonchik, and Serena Yeung. Unsupervised discovery of the long-tail in instance segmentation using hierarchical self-supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2603- 2612, 2021.
[51] Bo Xiong, Michael Cochez, Mojtaba Nayyeri, and Steffen Staab. Hyperbolic embedding infer- ence for structured multi-label prediction. In Conference on Newral Information Processing Systems (NeurIPS), volume 35, pages 33016-33028, 2022.
13

---

[52] Menglin Yang, Min Zhou, Marcus Kalander, Zengfeng Huang, and Irwin King. Discrete- time temporal network embedding via implicit hierarchical learning in hyperbolic space, ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 1975-1985, 2021.
[53] Zhi Gao, Yuwei Wu, Yunde Jia, and Mehrtash Harandi. Curvature generation in curved spaces for few-shot learning. In IEEE/CVE International Conference on Computer Vision (ICCV), pages 8691-8700, 2021.
[54] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
[55] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Ramakrishna Vedantam. Hyperbolic image-text representations, In International Conference Machine Learning (ICML), 2023.
[56] Avik Pal, Max Spengler, Guido Maria D' Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision- language models. arXiv preprint arXiv:2410.06911 2024.
[57] Tobia Poppi, Tejaswi Kasarla, Pascal Mettes, Lorenzo Baraldi, and Rita Cucchiara. Hyperbolic swfety-aware vision-language models. In /EEE/CVF Conference Computer Vision and Patlern Recognition (CVPR), 2025.
[58] Neil He, Menglin Yang, and Rex Ying. Hypercore: The core framework for building hyperbolic foundation models with comprehensive modules. In International Conference on Learning Representations (ICLR), 2025.
[59] Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch. arXiv preprint urXiv:2005.02819 2020.
[60] Steven Thomas Smith. Optimization techniques on riemannian manifolds. arXiv preprint arXiv:1407.5965, 2014.
[61] Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. arXiv preprint arXiv:1810.00760, 2018.
[62] Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. Visualizing and measuring the geometry of bert. In Conference on Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[63] Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie- Yan Liu. Representation degeneration problem in training natural language generation models. arXiv preprint 1rXi: 12009, 2019.
[64] William Rudman, Nate Gillman. Taylor Rayne, and Carsten Eickhoff. Isoscore: Measuring the uniformity of embedding space utilization. arXiv preprint arXiv::2 108.07344, 2021.
[65] Giovanni Puccetti, Anna Rogers, Aleksandr Drozd, and Felice Dell' Orletta. Outliers dimen- sions that disrupt transformers are driven by frequency. arXiv preprint arXiv:2205. 1380, 2022.
[66] OpenAl Foundation. Introducing chatgpt.
Novem-
ber 2022.
[67] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2 (eXXv:210400869) 14400091,,112000 2021.
[68] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190, 2021.
14

---

[69] Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, et al. Exploring universal intrinsic task subspace via prompt tuning. arXiv preprint arXiv:21 10.07867, 2021.
[70] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International Conference on Machine Learning (ICML), pages 2790-271 20997. PMLR, 2019.
[71] Yaoming Zhu, Jiangtao Feng, Chengqi Zhao, Mingxuan Wang, and Lei Li. Counter- interference adapter for multilingual machine translation. arXiv preprint arXiv:2104.08154 2021.
[72] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, arXiv preprint arXiv:2012.13255: 2020.
[73] Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James Clark, and Mehdi Rezagholizadeh, Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint 1xXiv:2212.10650 2022.
[74] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402. 12354, 2024.
[75] Shih- Yang Liu, Chien- Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu- Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.093:2 2024.
[76] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for pfrimeter-efficient fine-tuning. In International Conference on Learning Representations (ICLR), 2023.
[77] Xi Wang, Laurence Aitchison, and Maja Rudolph. Lora ensembles for large language model fine-tuning, arXiv preprint 2rxiv:2310.0003:: 2023.
[78] Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. Mini-ensemble low-rank adapters for parameter-efficien fine-tuning, arXiv preprint arXiv:2402.17263. 2024.
[79] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2024.
[80] Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhensu Chen, Xiaopeng Zhang, and Qi Tian. Qa-lora: Quantiaation-wwrre low-rank adaptation of large language models. arXiv preprint arXiv:2309.14711 14717, 2023.
[81] Yixiao Li, Yifan Yu, Chen Liang, Pengeheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine finn--tning--aee awaree quantization for large language models. arXiv preprint arXiv:2310.00 10.00699, 2023.
[82] Gal Mishne, Zhengchao Wan, Yusu Wang, and Sheng Yang. The numerical stability of hyperbolic representation learning. In International Conference on Machine Learning (ICML), pages 24925-249499 PMLR, 2023.
[83] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichir Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv: 10. J4168, 2021.
[84] Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by ratio- nale generation: Learning to solve and explain algebraic word problems. arXiv preprint 705.44/705..4146, 2017.
[85] Rik Mawps: A math word problem repository. In Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 1152-1157, 2016.
Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi.
15

---

[86] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are nip models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191, 2021.
[87] Jeff Alstott, Ed Bullmore, and Dietmar Plenz. Powerlaw: a python package for analysis of heavy-tailed distributions. PloS one, x1))885777 2014.
[88] Erzsébet Ravasz and Albert-Lászió Barabási. Hierarchical organization in complex networks. Physical review E, 67(2):02612,, 2003.
[89] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th annual meeting of the association for computational linguistics (ACL), pages 1715-1725, 2016.
[90] Mikhael Gromov. Hyperbolic groups. In Essays in group theory, pages 75-263. Springer, 1987.
[91] Hervé Fournier, Anas Ismail, and Antoine Vigneron. Computing the gromov hyperbolicity of discrete metric space. Information Processing Letters, 115(6-8)5576-579) 2015.
[92] TN Kipf. supervied classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 2016.
[93] Dmitri Krioukov, Fragkiskos Papadopoulos, Amin Vahdat, and Marián Boguná. Curvature and temperature of complex networks. Physical Review E, 80(3):035101, 2009.
[94] Fragkiskos Papadopoulos, Dmitri Krioukov, Marián Boguná, and Amin Vahdat. Greedy forwarding in dynamic scale-free networks embedded in hyperholic metric spaces. In Infocom, pages 1-9. IEFE, 2010.
[95] James W Cannon, William J Floyd, Richard Kenyon, Walter Parry, et al. Hyperbolic geometry. Flavors of geometry, 31(59-115):2, 1997.
[96] John G Ratcliffe. Foundations of hyperbolic manifolds, volume 149. Springer, 2006.
[97] Jindou Dai, Yuwei Wu, Zhi Gao, and Yunde Jia. A hyperbolic-to-byperbbhll graph convo- lutional network. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 154-163, 2021.
[98] Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing, and Soujanya Poria. LIm-adapters: An adapter family for parameter-effiieen fine-tuning of large language models. arXiv preprint arXiv:2304.0199 2023.
[99] Prithviraj Sen, Galilco Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 29(3):93-93, 2008
[100] Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. Exploring network structure, dynamics, and function using networkx. In Gäel Varoquaux, Travis Vaught, and Jarrod Millman, editors, SciPy2008, pages 11-5.. Pasadena, CA USA, 2008
[101] Ondrej Skopek, Silavian-Euger Ganea, and Gary Bécigneul. Mixed-curvature variational autoencoders. arXiv preprint arXiv:1911.084/1 2019.
[102] Philip McCord Morse and Herman Feshbach. Methods of theoretical physics. Technology Press, 1946.
[103] Valter Moretti. The interplay of the polar decomposition theorem and the lorentz group. arXiv preprint pl02-17/0211047 2002.
[104] Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint 2000022400.05229, 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper's s abstract and introduction explicitly state the main claims, including the contribution the proposed method and analysis, and its scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses limitation in the Conclusion (Section 6).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
17

---

The paper presents Proposition I regarding HypLoRA, introducing higher- order terms. states that the details and derivation of these terms are provided in Appendix F.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Section 5.1 "Experimental Settings" describes the datasets, models, and implementation details, including optimizer (AdamW), etc.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The datasets used (e.g., GSM8K, AQUA, various commonsense reasoning benchmarks) are publicty available and cited.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc
wwtssSubiisssooooiiiiooooooooooe onnooiccy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //npss cccppblic//aeecccccccccccccc htallli//uiieeee http//uudeseeeeeee https//ooeSSumiiiiieee.e wwwsaabmisssonPPaa ssioonolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5.1 details the experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification; Table 2, which presents hyperbolicity across various metric spaces and datasets, includes error bars reported as mean and standard deviation. Tables 3 and 4 report mean accuracies over three runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides implementation details in Section 5.1, including the GPU type (NVIDIA A A100) used for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: http://uwwppscccplli///eieeeeeee htt.bl/icc/hhiiccccieeeeeeeee
Answer: [Yes]
Justification: Based on the content of the paper, the research focuses on investigating geometric properties of LLMs and proposing new fine-tuning method. The datasets used are standard benchmarks in the field. There is no indication of activities that would violate the NeurlPS Code of Ethics, such as plagiarism, falsification of data, unethical use of human subjects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Conclusion part.
20

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., dep.oymmly of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: does not introduce new large-scale pretrained models novel scraped datasets that would pose a high risk for misuse requiring specific safeguards beyond those applicable to the original models it builds upon.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators/original owners of assets (datasets like GSM8K, MAWPS, AQUA; models like LLaMA, Gemma; and software like the Powerlaw Package are credited via citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
21

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The datasets are existing benchmarks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information
supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research does not appear to involve human subjects in a way that would necessitate IRB approval, as per the justification for question 14.
Guidelines:
22

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [No]
Justification: No, except for writing, editing, or formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /tt///eeerrp///e/ee/eee/ee www.c///o////neeeecceeeoooo /0tp:////////ooeeeeee//////////0 for what should or should not be described.
23

---

Appendix Contents
More Investigation Results
25
A.1 Token Frequency and Norm Distribution on Mathematical Reasoning
25
A.2 Token Frequency and Norm Distribution on Commonsense Reasoning
25
A.3 Hyperbolicity the Final Hidden Layer of LLMs
27
Hyperbolicity on Different Metric Spaces
28
Exponential and Logarithmic Maps
28
C.1 Exponential Map
28
C.2 Logarithmic Map
29
C.3 Notation in the Main Text
29
Lorentz Transformation
29
D.I Lorentz Boost
29
D.2 Lorentz Rotation
29
Transformation Analysis
30
Full Comparison
32
F.1
Implementation Details
32
F2
Comparison on Mathematical Reasoning
33
F3
Comparison Commonsense Reasoning
34
E44
GPU Usage
34
G Case Study
35
24

---

A More Investigation Results
A.1 Token Frequency and Norm Distribution on Mathematical Reasoning
To provide a comprehensive understanding of the geometric properties of token embeddings across different mathematical reasoning tasks, we extend our analysis beyond the GSM8K dataset presented in the main text include AQuA and MAWPS datasets. This broader investigation allows to validate the consistency of our findings across diverse mathematical problem types and complexity levels. The AQuA dataset presents algebraic word problems that require multi-step reasoning and equation solving, while MAWPS focuses on elementary arithmetic word problems with varying structural complexity. By analyzing token distributions across these complementary datasets, we can assess whether the observed power-law behavior and hierarchical token organization represent universal properties of mathematical reasoning tasks or are specific to particular problem domains.
SVAMP
AQUA
MAWPS
10
y 1.95
y 1.89
y 1.90
Count (lgg soals)
102
Coust (log scalo)
104
101
(log
101
101
00.00
101
10P
10
100
10P
10/
100
10
10P
Token Frequency (log scale)
Token Frequency (log stale)
Token Frequency (log scale)
SVAMP
10a
AQuA
MAWPS
Tokon Froqueec (log scale)
Frequmcy (log scall)
Teken Frequeccy (log scalo)
10
102
10a
10
10t
101
10
0.5
0.4
0.5
0.6
0.7
0.5
0.4
0.5
0.6
0.7
0.5
0.4
0.5
0.5
0.7
Token
Token
Token Norm
Figure 3: Token frequency distribution (top row) and token frequency vs. norm (bottom row) across different mathematical reasoning datasets in LLaMA3. The top row shows the power-law distribution of token frequencies with the decay rate (7) annotated for each dataset. The bottom row illustrates the relationship between token frequency and token norm, binned and colored by frequency, where higher token norms correspond to lower frequencies.
Our extended analysis, illustrated in Figure 3, reveals remarkably consistent patterns across all three mathematical reasoning datasets. The power-law exponents remain stable within a narrow range (7 [1.89, 1.95)). indicating that the hierarchical structure of mathematical language is preserved regardless of the specific problem type or complexity level. The relationship between token frequency and embedding norms shows consistent inverse correlation across all datasets, with high-frequency mathematical operators and common function words clustering near the origin, while domain-specific mathematical terms and numerical values positioned at greater distances. This consistency strengthens our hypothesis that mathematical reasoning tasks inherently exhibit hyperbolic characteristics in their token embedding spaces, providing strong empirical support for the effectiveness of hyperbolic fine-tuning approaches like HypLoRA in mathematical domains.
A.2 Token Frequency and Norm Distribution on Commonsense Reasoning
To demonstrate the generalizability of our findings beyond mathematical reasoning, we conduct comprehensive analysis of token distributions across six diverse commonsense reasoning datasets: ARC-Challenge, ARC-Fasy, BoolQ, HellaSwag, PIQA, and SIQA. These datasets span wide range of commonsense reasoning tasks, from factual knowledge retrieval (ARC datasets) and yes/no question answering (BoolQ) to physical commonsense (PIQA) and social understanding (SIQA).
25

---

ARC Cac-Challenge
ARC-Easy
boolq
Count (log scale)
10a
y=1.93
(log scale!
10a
y= 1.89
Count (log scale:
10a
y 1.94
10?
10?
10?
10a
104
10a
10a
10a
10
10a
10a
10a
10
100
10
102
10a
10-
10a
10
10
Frequency scale)
Token Frequency (log scale)
Token Frequency (log scale)
hell aswag
social |
1.94
1.94
10a
1.76
Count (log scale)
10a
Count (loa scale)
102
20a
10a
10.
Count
20
201
10a
109
10"
10
16a
10
10'
102
10
10
101
10a
Token Frequency (log scale)
Token Frequency scale)
Token Frequency scale)
ARC-Challenge
ARC-Challenge
boolg
Token Frequency (log scale)
Frequency (log scale)
(log scale)
10a
10a
10a
10a
10a
Taken
10a
10p
10a
0.1
0.6
0.4
0.6
0.4
0.6
hella swag
piqa
social |
Token Frequency (log scale)
10
frequency (log scale)
Taken [log scale)
10
10a
10?
102
10a
104
10a
10
0.4
0.6
0.4 0.5
0.7
0.4
0.6
Token Norm
Token Norm
Token Nerm
Figure 4: Token frequency distribution (top rows) and token frequency vs. norm (bottom two rows) across different commonsense reasoning datasets in LLaMA3. The top two rows show the power-law distribution of token frequencies with the decay rate (γ) annotated for each dataset. The bottom two rows illustrate the relationship between token frequency and token norm, binned and colored by frequency, where higher token norms correspond to lower frequencies.
26

---

Table 6: Relative 5-hyperbolicity (mean std.) of the final hidden layer in Gemma-7B across math (AQuA, GSM8K) and commonsense (ARC-Challenge, WinoGrande, OpenBookQA) datasets comparing the frozen base model, LoRA, DoRA, and HypLoRA.
Dataset
Base Model
LoRA
DoRA
HypLoRA
AQuA
0.31 0.04
0.24 1 0.05
0.23 + 0.05
0.22 - 0.03
GSM8K
0.28 0.04
0.21 0.05
0.21 - 0.05
0.20 0.03
ARC-Challess
0.30 0.03
0.35 - 0.03
0.36 4: 0.02
0.25 - 0.02
Winogrande
0.22 0.04
0.32
0.27 0.02
0.27 0.02
OpenbookQA
0.30 0.03
0.35 + 0.03
0.38 + 0.02
0.25 0.02
Table 7:
Relative 5-hyperbolicity (mean + std.) of the final hidden layer in Gemma3-4B for the same five datasets, contrasting the base model with LoRA, DoRA, and HypLoRA.
Dataset
Base Model
LoRA
DoRA
HypLoRA
AQuA
0.17 = 0.03
0.17 + 0.03
0.19 + 0.02
0.11 + 0.01
GSM8K
0.16 = 0.03
0.20 0.00
0.19 + 0.03
0.11 + 0.02
ARC-Challenge
0.17 = 0.02
0.21 + 0.01
0.17 + 0.02
0.20 + 0.02
Winogrande
0.16 + 0.02
0.16 0.02
0.21 + 0.01
0.12 01011
OpenbookQA
0.17 0.03
0.16 0.02
0.17 + 0.03
0.11 + 0.01
This diverse collection allows us to investigate whether the hyperbolic characteristics observed in mathematical reasoning extend to broader domains of human knowledge and reasoning. The inclusion of both challenging AARC-Challenge HellaSwag) and more accessible (ARC-Easy, BoolQ) datasets enables us to examine how task difficulty influences the underlying geometrie structure of token embeddings.
The results presented in Figure demonstrate that the power-law distribution of token frequencies and the inverse relationship between frequency and embedding norms persist across all commonsense reasoning datasets, with power-law exponents ranging from 7 = 1.76 to 7 = 1.94. Notably, the Social IQA dataset exhibits a slightly lower exponent (7 = 1.76). suggesting that social reasoning tasks may have a somewhat different hierarchical structure, possibly due to the more nuanced and context-dependerr nature of social interactions compared to factual or physical reasoning. Despite this variation, the overall pattern remains consistent: abstract concepts and function words maintain smaller norms and higher frequencies, while specific entities, proper nouns, and domain-speciffe terminology are positioned at greater distances from the origin.
A.3 Hyperbolicity in the Final Hidden Layer of LLMs
In this part, we further present the analysis of the hyperbolicity of the hidden states in Tables 6 and Table Considering five distinct reasoning datasets, including two mathematical reasoning datasets (AQuA and GSM8K) as well as three commonsense reasoning datasets (ARC-Challenge, Winogrande, and OpenbookQA), we observe that the base models consistently exhibit less hyperbolic structure (i.e., higher 8 values) in their final hidden layer representations compared to their initial token embeddings.
LoRA and DoRA generally reduce the ó values, while the proposed HypLoRA method mostly achieves even lower values, indicating higher degree of hyperbolicity in the learned representations. This effect is observed across most datasets in both model families. These empirical findings complement our analysis of the initial token embeddings: while the pretrained models begin with a latent hierarchical structure, as evidenced by hyperbolicity in the input layer, fine-tuning methods can either preserve or distort this property. The consistently lower values HypLoRA provide strong empirical evidence that our method actively preserves and enhances the hierarchical structure of the representations throughout the model, aligning the final contextualized embeddings with the geometric biases that are beneficial for reasoning.
27

---

B Hyperbolicity on Different Metric Spaces
Table 2 presents the hyperbolicity values in both continuous (i.e., sphere space) and discrete metric spaces (i.e., tree, scale-free, and random graphs). We employ a consistent processing method similar that used in Section 4 for embedding spaces. Specifically, we sample 1,000 four-tuples, compute the 5 value for each, and then take the maximum value.
Embeedding Iypurbuuliity
12
Graph Hyperbolicity
Empirical correlation between the ground-truth d-hyperbolicity several reference graphs (tree, scale-free, PubMed, dense, sphere) and the 6 measured after embedding them with two-layer GCN into Euclidean space; each point averages 1,000 sampled quadruples.
Figure 5:
For the sphere space, we use a two-dimensional model and calculate hyperbolicity based on geodesic distances. The PubMed graph is sourced from Sen et al. [99]. The tree and dense graphs generated using NetworkX [100]. For these graphs, we remove isolated nodes before performing our calculations to ensure consistency. We the shortest-patt distance each graph the distance measure, analogous to the concept of geodesics continuous spaces.
In this study, we utilize the Euclidean distance to compute the hyperbolicity of token embeddings, following the approach proposed by [16]. To further assess the validity of this method, we embed graphs with varying degrees of hyperbolicity into Euclidean space using a two-layer GCN and compute hyperbolicity based on the distances between embeddings. The results, presented in Figure 5, indicate positive correlation between the hyperbolicity of the original graphs and that of the embeddings, although the values do not exactly coincide. Building on this observed relationship. calculate the hyperbolicity of token embeddings as proxy for estimating their underlying geometric structure. In this context, lower hyperbolicity values suggest a more tree-like geometric configuration.
C Exponential and Logarithmic Maps
The exponential and logarithmic maps are fundamental tools for navigating between the tangent space and the hyperbolic manifold. These maps enable to perform computations in the familiar Euclidean tangent space while preserving the geometric properties of hyperbolic space.
C.1 Exponential Map
The exponential map expx K. Tx - C*k projects a tangent vector E TxxC at point x onto the hyperboloid C3. Geometrically. it maps v to the point expx (v) == =1)), where is the unique geodesic satisfying (0) x and (0) v.
The exponential map is given by:
M VK
lille VK
expx K (v) cosh
+ VK sinh
(11)
|
where ||vl|c / (v. v)c the norm of the tangent vector under the Lorentzian inner product.
At origin o = (vK,0, O..,0 for tangent vector = (0, u) where u IR", the exponential map simplifies to:
exp (v)= = (VK cosh
|
K sinh
|u
u
(12)
VK
V
[u]
28

---

C.2 Logarithmic Map
The logarithmic map logx 6 - Txhx is the inverse of the exponential map. It projects a point y 2 back to the tangent space at x:
cosh -1 (x,y)c
(x. (x,
K (y) logx
y
(13)
(x.))2
These maps satisty the inverse relationships: log (exp
/ (v)) = v and expx " (logx (o)) = y.
C.3 Notation in the Main Text
In the main text, we use the shorthand notation Ho and II to denote general projection operators between Fuclidean space IR" and hyperbolic space C The exponential and logarithmic maps described above represent one valid instantiation of these operators:
xxxp(x) (x) == expo ((0xx)). IT (y) log (y)
(14)
(15)
where (0,x) Rl+d denotes the vector obtained by prepending
to E R. and (-)pa denotes the restriction to the last d coordinates (i.e., removal of the first coordinate). However, other diffeomorphisms [101] between Euclidean and hyperbolic spaces. The choice of projection method can be adapted based on computational efficiency and numerical stability requirements, while the core principle of our approach, performing the low-rank transformation directly on the hyperbolic manifold, remains unchanged.
Important Observation. applied consecutively at the same base point without intermediate operations on the manifold, they effectively cancel each other out. For example, log (exp (v)) = v. This is why the conventional tangent-space approach for hyperbolic neural networks [33, 32] does not directly benefit LLM adaptation, where the hyperbolic geometry is effectively bypassed. Our Direct Lorentz Low-Rand Transformation (LLR) addresses this limitation by operating directly on the hyperbolic manifold between the projection steps, ensuring that the geometric properties hyperbolic space are preserved and utilized.
Regardless of the specific projection method used, when these maps
D Lorentz Transformation
In the context of special relativity, Lorentz transformations linear mappings that preserve the spacetime interval between events, ensuring the constancy of the speed of light across all inertial frames. These transformations can be categorized into two primary types: Lorentz boosts and Lorentz rotations [102, 103].
D.1 Lorentz Boost
A Lorentz boost corresponds to transformation between two inertial reference frames moving at a constant relative velocity. Given velocity vector R." with magnitude ||v|| < 1, the Lorentz boost matrix B mixes time and space coordinates:
?
TV ?
B
(16)
-v
I+
14y
where 7
is the Lorentz factor.
/1- [
D.2 Lorentz Rotation
A Lorentz rotation involves only the rotation of spatial coordinates while preserving the time coordi-
nate:
R
(17)
R
29

---

where R E (O(n) is spatial rotation matrix.
Our Spatial-like Transformation. In Direct Lorentz Low-Rank Transformation (LLR), apply transformations exclusively to the spatial components while maintaining the constraint of the Lorentz manifold. Given a point x" (mo xx E L transformation is:
LLR (VI BAx
(18)
where we transform the spatial component x and recompute the time component to maintain the Lorentz constraint z - |x, K.
This can be decomposed into two sequential transformations:
yd = (y0 H .y.) (v ||Ax 11 + K, Ax
(19)
z = (20 x.. (20", (V By K, By,
(20)
Interpretation as a Constrained Lorentz Rotation. Our transformation can be viewed as a special
case of Lorentz rotation where: (1) We apply linear transformation to the spatial coordinates:
(2) We recompute the time component to preserve the manifold constraint: x + x + BAx H
/ K. This approach differs from a standard Lorentz rotation in two ways (see also [37]): (1) the spatial transformation BA is not necessarily orthogonal (i.e., BA SO(n)); (2) the time component is not preserved but rather recomputed to maintain the manifold constraint.
In matrix form, transformation can be expressed as:
VIIBAX! PPK
O'
|xf 1i+K
(21)
x
0
BA
The key property that this transformation preserves the Lorentz manifold structure: if x
C
then LLR(BA,XX x C as verified by:
(=0 )= - 1|2 - || 11 1
(22)
This spatial-like transformation approach allows us to leverage the low-rank structure of BA while maintaining the geometric properties of the hyperbolic space, providing computationally efficient method for hyperbolic low-rank adaptation.
E
Transformation Analysis
This section provides detailed analysis of how HypLoRA differs from standard LoRA by examining the higher-order terms introduced through hyperbolic geometry.
Proof. Let E Rdd be input token emhedding. Let A R and B Rexr be low-rank matrices with rank r < min (d, k). Consider the d-dimensional hyperbolic space C (Lorentz model) with curvature = 1//K, where K O.
Our goal is to analyze how the HypLoRA update differs from the LoRA update and to understand the impact of token norms |x|| on the higher-order terms introduced by HypLoRA.
Mapping the Input Embedding to Hyperbolic Space. Following previous work [32], we interpret the Euclidean token embedding x as an element in the tangent space at the origin o of the hyperbolic space - The tangent vector given by (0,x) ToCk- The exponential map exp projects onto the hyperbolic space:
&
(VK cosh
|ivile VK
||vllc VK
expo (v)
KK sinh
(23)
|
where ||vlcc denotes the Minkowski norm. Since v (0, x) and ||v||c simplifies to:
||x||| the exponential map
(xpk(v) = (VK cosh
[x]
K sinh
[x|
x
(24)
VK
K
|x|
30

---

| v K
Approximations. For small
let
|x||
can use the Taylor series expansions:
2 VK
2
cosh
2 +
sinh
(25)
2
K
6K3/22
Applying these to the exponential map of x:
||x|2
"
K
(26)
2vK'
|x 12
space
x
+
6K
(27)
Applying Low-Rank Transformations to the Approximated Embedding. Using the approximated Opace we apply the transformations.
First transformation:
|x|2 6K x) = -Ax +
||x|12
y
H space
Au
space
A
x
6K
Ax.
(28)
Second transformation:
||x||2 6k
Z
space
By
space
BAx +
BAx.
(29)
Compute the time component after the transformations:
20
K
H space
(30)
Approximating the Logarithmic Map. We map the transformed hyperbolic point z = (zf back the tangent space at the origin using the logarithmic map log
H
2 Sspace)
H
z H space
LQU logo (z VK arcosh
(31)
VK
2
K
H spacee
1
Using the approximation z
v K
and for small 8
we have:
2vK
2K
H
space
arcosh
~ arcosh +8) ~0 200
(32)
K
(= )2
(33)
Therefore, the logarithmic map simplifies to:
AQ
(34)
Comparing HypLoRA and LoRA Updates. The HypLoRA update is:
|x|
AQHYP vo BAx +
BAx.
(35)
6K
The LoRA update is:
AQLORA BAx.
(36)
The difference between the updates is:
|||| B.A..
Hyp AQLORA
(37)
6K
31

---

|
Impact of Token Norms on Higher-Order Terms. The higher-order term to ||x||? Since ||x|| reflects the specificity of the token in the hierarchical structure (larger norms
BAx is proportional
6K
correspond to more specific tokens), this term becomes significant for tokens representing specific concepts.
Impact on Attention Scores. The HypLoRA attention scores are computed as:
ScoreSHypLaRA (Q + AQH (K + AKHyp) T
(38)
vd
where ArH is derived similarly.
The difference in attention scores includes higher-order terms dependent on ||x|
A Scores = ScoresHypLoRA ScorcsLoRA
(39)
These higher-order terms enable HypLoRA to capture more complex hicrarchical relationships, particularly for tokens with larger norms.
Remark 1. Alignment with Token Hierarchy: The higher-order terms in HypLoRA's 'ss updates are proportional to ||x||3 correlating with the specificity of tokens in the hierarchical structure. As result, HypLoRA places greater emphasis on more specific tokens, enhancing its ability to model detailed relationships.
Role of Curvature C: The curvature = -1/K scales the higher-order corrections. Smaller K (larger negalive curvature) amplifies these terms, aligning with the hyperbolic nature of token embedddings In practice, the curvature parameter K can taned to ensure this condition is satisfied for typical token embedding norms.
By incorporating these higher-order terms, HypLoRA leverages the inherent hierarchical and hyperbolic structure of token embeddings. This leads to improved perfor- mance, especially on problems requiring complex reasoning, explaining why the proposed method performs better on more challenging datasets.
Effectiveness of HypLoRA:
F
Full Comparison
While the main body of our paper focuses on comparing HypLoRA against the standard LoRA baseline demonstrate the core effectiveness of our hyperbolic fine-tuning approach, this section provides comprehensive evaluation against broader range of eefffiitr-effcientt fine-tuning methods, such Prefix tuning [68], Series and Parallel adapters [70], and DoRA [75], providing a more complete picture of HypLoRA's 's performance relative to the current landscape of efficient fine-tuning techniques. This extended comparison validates that our improvements are not merely due to increased model capacity or specific architectural choices, but rather stem from the fundamental advantages of incorporating hyperbolic geometry into the adaptation process.
F.1 Implementation Details
To ensure consistency and comparability, our experimental setup closely followed the training config- urations outlined in Hu et al. [98]. Across all fine-tuning tasks, we employed the AdamW optimizer with learning rate of 3 x 10 and trained for total of three epochs. LoRA modules (and conse- quently, HypLoRA adapters) were integrated into both the Multi-Head Attention (MHA) and MLP layers of the foundation models. A key hyperparameter for HypLoRA is the curvature K (defining the hyperbolic curvature as 1/K), which was initialized by searching the set {0.5, 1.0). For evaluation, final scores across the datasets, thereby giving equal weight to each individual prompt, regardless of the varying number of questions per dataset (e.g., 319 in GSM8K versus 238 in MAWPS).
micro- averaged for arithmetic reasoning and averaged for commonsense reasoning
For baseline methods, we adopted the following approach: results for Prefix tuning [68], Series adapters, and Parallel adapters [70] are directly cited from Hu et al. [98] to ensure fair comparison
32

---

Table 8: Comprehensive comparison of parameter-efficien fine-tuning methods on mathematical reasoning tasks. Results marked with are from [98], while i indicates our reproduced results. The percentage following each dataset name indicates the proportion of prompts relative to the total number of inference prompts. M.AVG represents the micro-average accuracy across all datasets. Best results for each model are highlighted in bold. OOT indicates out-of-time during training.
Base Model
PEFT Method
MAWPS(B.5%)
SVAMP(35.6%)
GSM8K(46.9%)
AQuA(9.0%)
M.AVG
GPT-3.5
None
87.4
69.9
50.4
38.9
62.3
None
51.7
32.4
15.7
16.9
24.8
Prefix
63.4
38.1
24.4
14.2
31.7
Scries*
77.7
52.3
33.3
15.0
42.2
LLaMA-7B
Paralle
82.4
49.6
35.3
18.1
42.8
LoRA*
79.0
52.1
37.5
18.9
44.6
LoRAf
81.9
48.2
38.3
18.5
43.7
DoRA
80.0
48.8
39.0
16.4
43.9
HypLoRA (Ours)
39.1
44.4
None
65.5
37.5
32.4
15.0
35.5
Prefix*
66.8
41.4
31.1
15.7
36.4
Series*
78.6
50.8
44.0
22.0
47.4
LLaMA-13B
Parallet
81.1
55.7
43.3
20.5
48.9
LoRA#
83.6
54.6
47.5
50.5
LoRA
34.7
48.5
51.0
DoRA
83.0
54.6
OOT
18.9
NA
HypLoRA (Ours)
83.2
34.8
49.0
21.5
51.5
None
76.5
60.4
38.4
25.2
48.3
LoRA
91.6
76.2
66.3
28.9
68.6
Gemma-7B
90.7
79.2
68.3
71.0
HypLoRA (Ours)
89.5
78.7
60.5
32.7
71.2
Nome
79.8
50.0
54.7
21.0
52.1
LoRA
92.7
78.9
70.8
30.4
71.9
LLaMA3-8B
DoRA
90.3
79.8
73.3
21.3
72.4
HypLoRA (Ours)
91.6
80.5
74.0
34.2
74.2
LoRA
90.8
77.3
72.3
50.8
73.7
Gemma3-4B
DoRA
89.5
78.8
68.5
52.4
72.5
HypLoRA (Ours)
88.2
83.9
76.1
53.2
77.8
LoRA
90.8
84.4
78.6
68.1
80.8
Qwen2.5-7B
DoRA
92.8
87.4
80.4
64.2
82.5
HypLoRA (Ours)
91.2
92.2
87.9
71.6
88.3
under identical experimental conditions. For LoRA and DoRA, conducted independent reimple- mentations following their respective original papers and parameters [31, 75] to enable rigorous and controlled comparisons.
F.2
Comparison on Mathematical Reasoning
Looking at the mathematical reasoning comparison table, several key experimental findings emerge regarding HypLoRA's performance across different model architectures and datasets. The results demonstrate that HypLoRA consistently outperforms standard LoRA across multiple model families, with particularly notable improvements more challenging datasets. For the Gemma-7B model, Hy- PLoRA achieves a micro-averaged accuracy of 71.2%, surpassing LoRA's 68.69 68.6%. For LLaMA3-8B, HypLoRA reaches 74.2% compared to LoRA's 71.9%. The improvements are especially pronounced on the AQuA dataset, which requires complex algebraic reasoning. Specifically, HypLoRA shows gains of 3.8 percentage points over LoRA on Gemma-7B (32.7% 28.9%) and 3.8 points LLaMA3-8B (34.2% vs 30.4%). This pattern suggests that HypLoRA's hyperbolic geometry is particularly effective for problems requiring multi-step reasoning and understanding of hierarchical mathematical relationships.
The consistency of improvements across different model architectures further validates the general- izability the hyperbolic approach. While HypLoRA shows competitive performance on simpler datasets like MAWPS, the performance advantages become more significant on challenging datasets like GSM8K and AQuA, which demand sophisticated reasoning capabilities. For instance, on GSM8K. HypLoRA achieves 69.5% accuracy Gemma-7B versus 66.3% for LoRA, and 74.0%
33

---

Table 9: Extended commonsense reasoning accuracy (%) for GPT-3.5 and for LoRA, DoRA, and HypLoRA on LLaMA3-888 Gemma3-4B and Qwen2.5-74 B... Columns correspond to BoolQ, PIQA, SIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA; the rightmost column reports the macro average across the eight benchmarks.
Model
PEFT Method
Paraus
BoolQ
HellaSwag
WinoGrande
ARC-
GPT:3.5
None
None
73.1
85.4
68.5
73.5
66.1
89.8
79.9
74.8
77.0
LoRA
0.70
70.8
85.2
79.9
91.7
81.3
81.2
71.2
79.0
80.8
LLMM3388
DoRA
85.2
HypLoRA (Ours)
0.70
74.1
87.6
80.6
94.5
84.7
90.4
81.2
85.2
84.8
LoRA
1.44
68.1
83.2
77-2
88.0
80.5
84.5
69.3
$3.6
79.5
Gemma3-4B
HypLoRA (Ours)
1.01
70.0
84.3
79.2
91.5
80.3
89.1
75.9
86.4
82.5
LoRA
0.71
73.4
89.5
79.5
93.6
84.1
92.8
82.0
87.0
85.2
Qwen2.5.7B
DaRA
0.72
71.7
88.7
92.4
82.4
85.1
HypLoRA (Ours)
0.71
72.8
89.3
95.5
87.5
on LLaMA3-8B versus 70.8% for LoRA. These correspond gains of 3.2 points over LoRA on both Gemma-7B and LLaMA3-8B. Notably, the most recent models, HypLoRA demonstrates substantial improvements: on Gemma3-4B, HypLoRA achieves 77.8% M.AVG compared to LoRA's 73.7% (+4.1 points), and on Qwen2.5-7B, HypLoRA reaches 88.3% versus LoRA's 80.8% (+7.5 points). The fact that HypLoRA maintains superior performance across both older (LLaMA-7B. ALaMA-13BI and newer (LLaMA3-8B Gemma3-4B, Qwen2.5-7B) model architectures demon- strates the robustness of incorporating hyperbolic inductive biases into parameter-efficin aine-tnnng, regardless of the underlying model's specific architectural details training paradigms.
F33 Comparison Commonsense Reasoning
HypLoRA demonstrates substantial improvements over standard LoRA across diverse commonsense reasoning benchmarks, as shown in Table 9. The commonsense reasoning tasks evaluated include BoolQ (yes/no question answering), PIQA (physical commonsense inference), SIQA (social inter- action reasoning). HellaSwag (commonsense natural language inference), WinoGrande (pronoun disambiguation), ARC-e and ARC-c (science question answering with easy and challenging diffi- culty), and OBQA (open book question answering). These benchmarks collectively assess the model's ability to understand implicit knowledge, contextual nuances, and real-world reasoning patterns. The consistent performance gains across all three model architectures and eight diverse benchmarks indicate that the hierarchical inductive bias introduced by hyperbolic geometry effectively captures the implicit relational structures underlying commonsense reasoning
F.4
GPU Usage
Table 10 presents a comprehensive comparison of memory usage across different fine-tuning methods for both LLaMA3 8B and Gemma3-4B models. The results demonstrate that HypLoRA maintains comparable memory efficiency to the baseline LoRA method. Specifically, HypLoRA with stereo- graphic projection achieves identical memory consumption LoRA (30.12 12 GB for LLaMA3-8B and 14.61 GB for Gemma3-4B), while HypLoRA A with exponential/logrrthmii maps introduces only minimal overhead of 0.02 GB for LLaMA3-SB and 0.01 GB for Gemma3-4B. Notably, both Hy- oLRA variants significantly outperform DoRA in terms of memory efficiency, with DoRA requiring 30.23 GB and 14.62 GB, respectively. These results confirm that hyperbolic adaptation approach does not compromise memory efficiency while delivering superior performance improvements, mak- ing HypLoRA a practical choice for resource-constrained environments where both performance gains and memory conservation are critical considerations.
Table 10: Allocated Memory Usage Comparison for Fine-tuning Methods
LLaMA3-8B
Gemma3-4B
Method
Param %)
Memory (GB)
Param (%)
Memory (GB)
LoRA
0.7002
30. 12
1.0371
14.61
DoRA
0.7098
30.23
1.0516
14.62
HypLoRA
0.7099
30.14
1.0518
14.62
34

---

Table 11:
Example problem with solutions from LoRA and HypLoRA (Case 1)
Question
To have the car for the weekend, Wilson's is report card needs to show that he received 80 or higher in his math class. His math grade is calculated by the average of 5 test scores. On those tests, he received: 65, 94, 81, 86, and 74. What is his current math grade? Answer: 80.0
LoRA
To find Wilson's current math grade, need to take the average of his test scores. First, we add up all test scores:
+ 380. Then, we divide the sum by the number of tests: 380 = 5 = 76. Therefore, Wilson's current math grade is 76. Answer: 76.
HypLoRA
find Wilson's current math grade, we need to take the average his 5 test scores. do this, we add up the scores and divide by 5: 65 + 94 + 81 + 86 + 74 = 400. Then, we divide by 5: 400 + 5 = 80. Therefore, Wilson's 's current math grade is 80. Answer: 80.
G
Case Study
In this section, we present three examples demonstrate the advantages of incorporating HypLoRA over standard LoRA fine-tuning.
Table 12: Example problem with solutions from LoRA and HypLoRA (Case 2)
Question
It takes 240 minutes of walking to break in new pair of shoes. Jason wants to try out for the track team in three weeks. If he can walk days a week break in the new shoes, how long does have to spend walking each day? Answer: 20.0 minutes
LoRA
To find out how long Jason has to spend walking cach day, we need to divide the total number of minutes it takes to break in the shoes by the number of days he has to do it. Jason has 240 minutes divided by days per week. This results in 60 minutes per day. Answer: 60.
HypLoRA
First, we need to find how many minutes Jason needs to walk in total to break in the new shoes: 240 minutes. Next, we find how many minutes Jason needs to walk each week over three weeks: 240 minutes divided by 3 weeks equals 80 minutes per week. Then, to find how many minutes he needs to walk each day: 80 minutes per week divided by days equals 20 minutes per day. Answer: 20.
Table 13: Example problem with solutions from LoRA and HypLoRA (Case 3)
Question
Gissela, Gordy, and Gary are truck drivers. Gissela has truck large enough haul 4,000 pounds of gravel. Gordy's truck can haul 800 pounds more than Gissela's truck. When Gary brings his truck and joins Gissela and Gordy, the three trucks combined can haul total of 11,600 pounds of gravel. How many pounds of gravel can Gary's 's truck carry? Answer: 2,800 pounds
LoRA
Let's start by assigning variables to represent the amount of gravel each truck can haul: Gissela's truck hauls 4,000 pounds. Gordy's 's truck hauls 4, 000 + 800 4, 800 pounds. Now, we set up equation for the total amount of gravel the three trucks where is the amount Gary's truck hauls. Simplifying the equation: 9, 800 + 11, 600, we subtract 9, 800 from both sides to get x = 11, 600 9, 800 1, 800. Therefore, Gary's truck
haul: 4, 000 800 x 11, 600,
hauls 1,800 pounds. Answer: 1,800.
HypLoRA
Gissela's 's truck can haul 4,000 pounds of gravel. Gordy's truck can haul 800 more pounds than Gissela's 's truck, so it can haul 4,800 pounds of gravel. Together, Gissela and Gordy's 's trucks can haul 8,800 pounds of gravel. If the three trucks combined can haul 11,600 pounds, then Gary's truck can haul 11, 600 8, 800 2, 800 pounds of gravel. Answer: 2,800.
These examples demonstrate how HypLoRA consistently provides more accurate reasoning compared to LoRA across different types of mathematical problems. In Case 1, LoRA drops 20 points when summing the five scores (reporting 380 instead of 400) and therefore produces the wrong average. This seemingly small arithmetic lapse aligns with the observation that LLMs often rely on high-level pattern similarity rather than exact computation [104]. By preserving greater separation among numerically close but semantically distinct tokens (e.g., 380 vs. 400), the hyperbolic representation in HypLoRA keeps the sequence of operations faithful and recovers the correct average.
In Case 2, LoRA immediately divides 240 minutes by the four weekly walking days, yielding 60 minutes per day and ignoring that the 240-m minute budget must be spread over three weeks. HypLoRA
35

---

correctly reasons in stages: divide 240 by 3 weeks, then by 4 days per week, recovering the required 20 minutes per day and showing stronger temporal reasoning.
In Case 3, LoRA actually sets up the correct balance equation 4,000+4,,800 + = 11, 11,600 but subtracts 9,800 from 1 11,600 rather than 8,800, reporting
1,800. HypL.oRA carries the subtraction through correctly and outputs the true 2,800 pounds. Together, these examples illustrate how the hyperbolic geometry employed by HypLoRA enables better handling of multi-step reasoning, maintaining both semantic context and numerical consistency in mathematical problem-solving scenarios.
Overall, these cases highlight consistent trend: LoRA frequently derails on either single arithmetic step (Cases and 3) or latent multi-hop dependency (Case 2), whereas HypLoRA preserves each intermediate calculation, keeps quantities well separated in representation space, and consequently delivers the correct final answers. These qualitative observations complement the quantitative gains reported in the main paper.
36


Reviews:
[{"review_id": "ctQ8fdax3Z", "reviewer": "Reviewer_uPaF", "summary": "This paper explores whether operating in Euclidean space is optimal for token embeddings in large language models. The authors find that token embeddings exhibit non-Euclidean characteristics: high-frequency tokens cluster near the origin, and embeddings display strong hyperbolicity, suggesting a latent tree-like structure. To exploit these properties, they propose HypLoRA, a novel method for low-rank fine-tuning directly in hyperbolic space. Experiments show that HypLoRA improves LLM performance on some benchmarks.", "questions": "- Do the authors have any insights or empirical results regarding how the hierarchical structure and hyperbolic properties of embeddings evolve across deeper layers of the model, as well as with the token context? It could also be interesting to see how such behavior may differ when using HypLoRA as opposed to other parameter efficient finetuning methods.\n- Is there any relationship between the degree of hyperbolicity in an LLM’s embeddings and the amount of performance improvements observed with HypLoRA?\n- Could the insights from this work be applied beyond the low-rank adapter setting, for example in other fine-tuning strategies or pretraining?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The proposed method is interesting and novel, leveraging the geometric properties of token embeddings.\n- The experimental evaluation is reasonably comprehensive, covering multiple models and datasets to support the claims.\n- The paper is generally well-written and well-structured with clear explanations.\n\n\nWeaknesses:\n- The analysis of hierarchical structure and hyperbolicity appears to focus predominantly on the input embeddings. It would strengthen the work to examine whether these non-Euclidean properties persist or transform across deeper layers of the model.\n- Additional discussion in the Questions section below", "was_revised": true, "final_justification": "My concerns and questions have been mostly addressed through the hyperbolicity analysis of the final layer embeddings and the comparison between accuracy improvements and the degree of hyperbolicity. Moreover, this geometric perspective holds potential for broader applications in training, beyond just parameter-efficient fine-tuning.", "rebuttal": "Thank you for your thorough review and constructive feedback. In the follows, we use “Q” to denote any question, weakness, or issue raised, and “A” to denote our response.\n\n---\n\n**Q1:**  The analysis of hierarchical structure and hyperbolicity appears to focus predominantly on the input embeddings. It would strengthen the work to examine whether these non-Euclidean properties persist or transform across deeper layers of the model. (as well as with the token context) . It could also be interesting to see how such behavior may differ when using HypLoRA as opposed to other parameter efficient finetuning methods.\n\n**A1:** We thank the reviewer for this excellent suggestion. To extend our hyperbolicity analysis beyond the input embeddings, we further examine the relative hyperbolicity of the final hidden layer representations of Gemma 7B and Gemma 3-4B across five datasets, two focused on math reasoning (AQuA, GSM8K) and three on commonsense reasoning (ARC-Challenge, Winogrande, OpenbookQA). We compared the base models against versions fine-tuned with LoRA, DoRA, and our proposed HypLoRA across a diverse set of five reasoning datasets.\n\nFirst, as a baseline, the hyperbolicity of the initial, non-contextualized token embeddings is presented in Table 1.\n\n**Hyperbolicity of Inital Token Embeddings (Table1)**\n\n| Method | AQuA | GSM8K | ARC-Challenge | Winogrande | OpenbookQA |\n| --- | --- | --- | --- | --- | --- |\n| Gemma 7B | 0.12 ± 0.01 | 0.11 ± 0.01 | 0.12 ± 0.01 | 0.11 ± 0.01 | 0.12 ± 0.01 |\n| Gemma3 4B | 0.19 ± 0.01 | 0.19 ± 0.02 | 0.16 ± 0.02 | 0.17 ± 0.01 | 0.16 ± 0.02 |\n\n**Hyperbolicity of Final Hidden Layer Embeddings in Gemma 7B: (Table2-1)**\n\n| Dataset | Base LLM | LoRA | DoRA | HypLoRA |\n| --- | --- | --- | --- | --- |\n| AQuA | 0.31 ± 0.04 | 0.24 ± 0.05 | 0.23 ± 0.05 | 0.22 ± 0.03 |\n| GSM8K | 0.28 ± 0.04 | 0.21 ± 0.05 | 0.21 ± 0.05 | 0.20 ± 0.03 |\n| ARC-Challenge | 0.30 ± 0.03 | 0.35 ± 0.03 | 0.36 ± 0.02 | 0.25 ± 0.02 |\n| Winogrande | 0.22 ± 0.04 | 0.32 ± 0.02 | 0.27 ± 0.02 | 0.27 ± 0.02 |\n| OpenbookQA | 0.30 ± 0.03 | 0.35 ± 0.03 | 0.38 ± 0.02 | 0.25 ± 0.02 |\n\n**Hyperbolicity of Final Hidden Layer Embeddings in  Gemma3 4B: (Table2-2)**\n\n| Dataset | Base LLM | LoRA | DoRA | HypLoRA |\n| --- | --- | --- | --- | --- |\n| AQuA | 0.17 ± 0.03 | 0.17 ± 0.03 | 0.19 ± 0.02 | 0.11 ± 0.01 |\n| GSM8K | 0.16 ± 0.03 | 0.20 ± 0.03 | 0.19 ± 0.03 | 0.11 ± 0.02 |\n| ARC-Challenge | 0.17 ± 0.02 | 0.21 ± 0.01 | 0.17 ± 0.02 | 0.20 ± 0.02 |\n| Winogrande | 0.16 ± 0.02 | 0.16 ± 0.02 | 0.21 ± 0.01 | 0.12 ± 0.01 |\n| OpenbookQA | 0.17 ± 0.03 | 0.16 ± 0.02 | 0.17 ± 0.03 | 0.11 ± 0.01 |\n\nThe results for the final hidden layer embeddings are shown in Table 2. Our analysis reveals several key findings: \n\n- First, the final hidden states of the base models exhibit a lower degree of hyperbolicity (i.e., larger hyperbolicity value than in the initial embeddings). \n- Second, HypLoRA consistently learns final representations with a higher degree of hyperbolicity (lower δ values) across almost all datasets and models. This provides strong empirical evidence that our method actively preserves and enhances the hierarchical structure of the representations throughout the model, aligning the final contextualized embeddings with the geometric biases that are beneficial for reasoning.\n\n---\n\n**Q2:** Is there any relationship between the degree of hyperbolicity in an LLM’s embeddings and the amount of performance improvements observed with HypLoRA? \n\n**A2:** We observe that as dataset complexity increases, the final hidden layer exhibits correspondingly higher relative hyperbolicity value.\n\nFor instance, AQuA, a particularly challenging dataset, shows a relative hyperbolicity of 0.31 in Gemma 7B and 0.17 in Gemma 3-4B. When these models were fine-tuned with HypLoRA, the degree of hyperbolicity increased (i.e., the relative hyperbolicity value decreased). In Gemma 7B, the drop from 0.31 to 0.22 led to a significant improvement in accuracy, reaching 46.5%, compared to LoRA’s 34.3%, which had a relative hyperbolicity of 0.24. \n\nSimilar trends are observed in Gemma 3-4B as well. Moreover, in commonsense reasoning tasks, the effect is even more pronounced: while LoRA and DoRA tend to reduce the degree of hyperbolicity relative to the base LLM, HypLoRA increases it. As a result, the performance improvement with HypLoRA is both evident and consistent.\n\n---\n\n**Q3:** Could the insights from this work be applied beyond the low-rank adapter setting, for example in other fine-tuning strategies or pretraining?  \n\n**A3:** The insights from HypLoRA have broad applicability beyond low-rank adaptation. The core principle—using norm-dependent scaling (||x||²) to automatically emphasize semantically specific tokens—can enhance any parameter-efficient fine-tuning method including adapters, prefix-tuning, and prompt-tuning, replacing their uniform scaling with hierarchically-aware adaptations that align with natural language structure.\n\nFor pretraining, the potential is also transformative. Since our analysis reveals pretrained embeddings naturally develop hyperbolic characteristics, incorporating hyperbolic inductive biases from the start—through curvature-aware attention, hyperbolic positional encodings, or learnable curvature parameters—could accelerate the emergence of beneficial hierarchical structure. \n\nRather than discovering these geometric properties implicitly through millions of training steps, models could be architected to respect linguistic hierarchy from initialization, potentially reducing training costs while improving the quality of learned representations and enabling more interpretable, structured language understanding."}, {"review_id": "iKmIIiSwki", "reviewer": "Reviewer_8MJ1", "summary": "This paper first provided an empirical observation showing that the frequency of tokens tends to be inversely proportional to the norm of their embedding vectors in several LLMs on various datasets, indicating an underlying hierarchical structure. The paper then measures the hyperbolicity for the token embeddings of various LLMs to validate that conjecture. The paper then proposed to fine-tune LLMs by performing low-rank transformation of the input of each layer on the hyperbolic manifold. Extensive experiments are conducted on several reasoning datasets with various pretrained models.", "questions": "1. Comparing the final representations between HypLoRA and Lora, eqns (30)-(32) and eqns (49)-(52), it seems that instead of injecting $BA$ matrix into the original weight, HypLoRA injects $c BA$ where $c$ is some constant that depends on the input norm $\\lVert x \\rVert^p$ for some integer $p$. In the end, the representative power of LoRA and HypLoRA seems to be equivalent, i.e. the solution sets of HypLoRA is covered by LoRA. I am wondering what might be the advantage of doing all the transformations from and to the manifold, compared with directly injecting, e.g. $\\lVert x \\rVert BA$, and optimize over B and A? \n\n2. Assuming the tokens follow the hyperbolic structure, it seems natual to do any of the following (or simultaneously):\n\n    1) Fine-tune the embedding space by applying riemmanian optimization algorithms on the hyperbolic manifold, and fine-tune the other parameters separately either in the eculidean space or via the proposed linear transformation on the hyperbolic manifold.\n    2) Train a hyperbolic network that does all the transformations on the hyperbolic space before the final layers.\n    \n\nFor now, HypLoRA freezes the embedding space and the pretrained weights, and injects a learnable linear transformation (by first projecting to the hyperbolic manifold and projecting back to the euclidean space), and merges it with the original eucliean linear transformation. I am wondering what is the logic behind and why it offers more effective training procedure, considering the similar amount of parameters of other fine-tuning algorithms.  \n\n3. The frequency of the tokens seem to be largely depend on the corpus or the context considered. I am wondering how the observation that the frequency of tokens tends to be inversely proportional to the norm can help for fine-tuning on new datasets. \n\n4. I am wondering how you choose or estimate the curvature constant $K$ and how it affects the final result.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The consideration of transforming the token embeddings in the hyperbolic space is reasonable and interesting. \n2. The suggestion of an underlying hyperbolic structure of token embeddings from pretrained models are informative. \n3. The usage of the direct Lorentz Low-rank Transformation is interesting.\n\nWeakness:\nThe effectiveness and the motivation of the proposed method, HypLoRA, seem a bit unclear. \n\nPlease see the section of Questions.", "was_revised": true, "final_justification": "After the rebuttal, it seems that the authors further clarify the effectiveness of the proposed fine-tuning approach. Given the interesting connections between hyperbolic structure and token embeddings, and the potential broader impact, I increased my score to 5.", "rebuttal": "Thank you for your thorough review and constructive feedback. In what follows, we use “Q” to denote each question, weakness, or issue raised and “A” for our response.\n\n---\n\n**Q1**: Representative power of LoRA and HypLoRA and the advantage of HypLoRA\n\n**A1**:  Thank you for your thorough observations and constructive feedback. The main differences lies in the higher-order interaction term( $||\\mathbf{x}||^2BA\\mathbf{x}$), which represents a qualitatively different type of transformation that cannot be replicated and covered by standard LoRA.\n\nThe higher-order term ($||\\mathbf{x}||^2BA\\mathbf{x}$) creates a multiplicative coupling between the input embedding norm and the low-rank transformation matrices, fundamentally changing how the adaptation interacts with different tokens. This is not simply a scaled version of BAx—it's an entirely different functional form that introduces **input-dependent adaptation strength**. For example, for tokens with larger norms (more specific concepts like \"dog\", \"banana\" with norm ~0.569 from Table 1), the term ($||\\mathbf{x}||^2BA\\mathbf{x}$) provides significantly stronger modifications, while for tokens with smaller norms (general function words like \"to\", \"in\" with norm ~0.353), the adaptation is naturally attenuated. **This creates an adaptive mechanism where the fine-tuning automatically emphasizes tokens that are more semantically specific and likely crucial for downstream task performance**.\n\n---\n\n**Q2**: why not applying riemmanian optimization && does all the transformations on the hyperbolic space before the final layers\n\n**A2**: Thanks for your critical suggestions. While these are theoretically appealing alternatives, they face significant practical limitations that motivated our design choices. Fine-tuning embeddings with Riemannian optimization on hyperbolic manifolds would require substantial computational overhead and careful handling of the manifold constraints, while training fully hyperbolic networks from scratch is prohibitively expensive for large-scale LLMs and would sacrifice the valuable pretrained representations that already capture hierarchical structure (as shown in our Section 4 analysis). \n\nOur HypLoRA approach balances theoretical rigor with practical feasibility: we preserve the pretrained Euclidean embeddings and weights that already exhibit hyperbolic characteristics, while selectively applying hyperbolic transformations only to the low-rank adaptation matrices A and B. **This hybrid design leverages the best of both worlds—maintaining compatibility with existing pretrained models while capturing hyperbolic inductive biases through the learnable curvature parameter $K$ and norm-dependent scaling terms.**\n\n---\n\n**Q3**：The frequency of the tokens seem to be largely depend on the corpus or the context considered. I am wondering how the observation that the frequency of tokens tends to be inversely proportional to the norm can help for fine-tuning on new datasets. \n\n**A3**: While it is true that absolute token frequencies are corpus-dependent, our core argument relies on the stable, underlying hierarchical structure of the embedding space, **which is a fundamental property learned during the LLM's large-scale pre-training, not just on the specific fine-tuning dataset**. Our cross-model and cross-dataset analysis in Appendices A and B demonstrates that this principle is a remarkably consistent feature across different model families and task domains. \n\nThis pre-existing geometric organization is precisely what makes our approach effective for fine-tuning on new datasets. Fine-tuning does not occur in a vacuum; it leverages the rich geometric landscape established during pre-training. When HypLoRA is applied to a new task, its norm-dependent update mechanism automatically exploits this latent structure. The higher-order term, $||\\mathbf{x}||^2BA\\mathbf{x}$, naturally applies a stronger adaptation to the tokens that the pre-trained model has already encoded as more specific (i.e., those with larger norms). These are often the most informative tokens for mastering the nuances of the new task.\n\n---\n\n**Q4**: I am wondering how you choose or estimate the curvature constant and how it affects the final result.\n\n**A4**：We treat curvature $C=-1/K$ as a learnable hyperparameter to allow the model to adapt to the specific geometric needs of each task. During our experiments, we initialize $K$ by searching over the candidate set {0.5, 1.0, 2.0} and subsequently make it trainable during the fine-tuning process. This approach allows the model to automatically discover a suitable curvature, effectively balancing the hyperbolic inductive bias with the geometric characteristics of a given dataset."}, {"review_id": "SDGllNz47F", "reviewer": "Reviewer_AwM6", "summary": "This paper proposes HypLoRA, a low-rank adaptation technique designed for large language models (LLMs) operating in hyperbolic space. The authors begin by empirically demonstrating that token embeddings exhibit hierarchical structure, supported by measurements of hyperbolicity. Next, they explore the relationship between the distribution of the token frequency and hyperbolic geometry. Finally, they introduce a low-rank transformation directly formulated in the Lorentz model of hyperbolic space and present experimental results validating its effectiveness.", "questions": "See weakness above.", "limitations": "In the paper checklist, the authors mention that limitations are discussed in the Conclusion (Section 6). However, the point raised there is a general observation about hyperbolic methods. The paper does not clearly identify under what conditions the method might fail or perform poorly. A more detailed discussion of the method’s limitations (such as sensitivity, dataset characteristics) would help clarify the scope and applicability of the approach.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strength\n\nLLM attracts huge attention in recent research, and using the hyperbolic geometry tool to analyze LLM is well-motivated and interesting for both communities. The paper lays out a step-by-step analysis, justifying the finding of the hierarchical structure of the token and using the hyperbolic geometry tool to enhance the low-rank transformation. I’m not from the LLM background and could not judge the significance from this aspect, but I believe integrating and using the hyperbolic tools in a new domain and analyzing the geometric insights should be appreciated.\n\n## Weakness\n* The explanation of the “cancellation effect caused by the exponential and logarithmic maps” is unclear. If operations are done in hyperbolic space and then mapped to or from the tangent space, the overall transformation is not necessarily the same. So why and how does $\\log_o^K(\\exp_o^K(x))$ appear here? This needs more clarification.\n* The motivation for the design of the proposed direct Lorentz low-rank transformation is not well explained.  While the authors want to explore the tool of hyperbolic geometry for LLM, there are a few questions raised in the design when examining the hidden states of LLM. First, why not transform the hidden states of LLMs from Euclidean space to hyperbolic space and stay in hyperbolic space instead of transforming the state back to Euclidean space? Second, why is transforming the space-like dimensions desired and not another alternative way? \n* In the quantitative analysis of hyperbolicity (lines 194–196), the authors mention using GNNs to embed random graphs with known hyperbolicity. However, it’s not clear which GNN model is used or how the experiment is set up. Appendix C does not provide enough detail. \n* In lines 199–202, the authors describe computing hyperbolicity “at the prompt level”, but it is not clear to me. What is the prompt level?  Did you compute the hyperbolicity of x? What is the representation you computed for hyperbolicity?  What exactly is measured and how? This part is hard to follow.\n* While the authors briefly mentioned the linear transformation in lines 266-268, the description is vague and lacks explanation.\n* In line 314, the paper refers to Appendix G for a “comprehensive comparison with additional methods,” but Appendix G contains GPU usage details. The comparison is either missing or mislabeled.\n* The paper uses the Lorentz model for the main transformation, but when discussing the connection between power-law distribution and hyperbolic geometry, it suddenly switches to the Poincaré disk model without any introduction or reasoning. For readers unfamiliar with hyperbolic geometry, this may cause confusion and feel abrupt.\n* The connection between the power-law distribution and hyperbolic geometry was not clear. First, the author uses the Poincaré disk model, which is in 2D, and the token embedding in hyperbolic dimension is not; how to generalize it for higher dimensions is unclear. Second, the derivation was not clear on how to consider token embedding with polar coordinates. Also, the authors use “radial coordinate (correlating with token frequency) ”  and “angular coordinate (encoding semantic similarity)”, and then state radial distribution, but it’s unclear how it is derived. Each statement in lines 224-225 was just given without any explanation or derivation. \n* In Equations 6, 7, and the following lines, the symbol $\\sim$ is used without explanation of what kind of approximation or relation it indicates.\n* In line 263, the paper refers to Appendix E.1 for the stereographic projection formulas, but the actual definitions of $\\Pi_{E\\rightarrow H}$ and $\\Pi_{H\\rightarrow E}$ are in Appendix E.2.\n* Appendix F consists of the transformation analysis. However, it was not clearly referred to in the main text, and the context of the properties introduced in Appendix F is unclear, e.g., using proof to introduce a property\n* There is no proof of Proposition 1, and the explanation of the dependency on the L2 norm was not clear. Following the authors pointing to Appendix F.1, it was hard to follow how “the norm-dependent, higher-order modification enables HypLoRA to capture hierarchical relationships”", "was_revised": true, "final_justification": "The authors addressed my comments during the rebuttal and discussion. I believe this paper offers valuable insights and potential applicability for the hyperbolic community. Therefore, I recommend acceptance.", "rebuttal": "Thank you for your constructive feedback. In the follows, \"Q” denotes any question raised, and “A”  denotes response.\n\n---\n\nQ1: Explanation of cancellation effect\n\nA1:  The cancellation effect occurs because standard hyperbolic neural network [1,2] approaches apply transformations in the tangent space at the origin, requiring the sequence : Euclidean embedding → exponential map to hyperbolic space → logarithmic map to tangent space → linear transformation → exponential map back to hyperbolic space → projection to Euclidean space. \n\nWhen these operations are chained together, the maps are mutually inverse and effectively cancel out, reducing the entire sequence to approximately the original Euclidean transformation BAx without preserving the beneficial hyperbolic geometry.  Logrithmic map is the inverse of exponentail map.\n\nThis is why we developed Direct Lorentz Low-rank Transformation (LLR) that operates directly on the hyperbolic manifold without intermediate tangent space mappings, preserving the hyperbolic structure. \n\n[1] Hyperbolic Graph Convolutional Neural Networks\n\n[2] Hyperbolic Neural Networks\n\n---\n\nQ2: Motivation of direct LLR; the reason of transforming the space-like dimensions \n\nA2: Regarding the decision to transform the hidden states back to Euclidean space, this is a fundamental requirement for our method's design as a parameter-efficient adapter. HypLoRA is intended to fine-tune existing, pre-trained LLMs, whose core components—such as the attention mechanism, MLP layers, and residual connections—are frozen and operate exclusively on Euclidean vectors. For our adapter to be compatible with this architecture, its output must be projected back into Euclidean space **so it can be integrated into the model's residual stream**. This \"project-transform-project back\" approach allows us to leverage the benefits of hyperbolic geometry in a localized, \"plug-and-play\" manner for specific tasks, avoiding the prohibitive cost and complexity of redesigning and retraining the entire language model in a new geometry.\n\nFor the second question about transforming space-like dimensions, as detailed in Section 5 and Appendix D, our spatial-like transformation can be interpreted as a constrained Lorentz rotation that preserves the hyperbolic manifold structure while enabling low-rank adaptation. The space-like components contain the semantic information analogous to the original token embeddings, while the time component is recomputed to maintain the Lorentz constraint (x₀² - ||xs||² = K). This design is inspired by established hyperbolic neural network architectures and allows us to apply linear transformations BA directly to the meaningful semantic dimensions while preserving hyperbolic geometry.\n\n---\n\nQ3:  Details of GNN models \n\nA3: We use two-layer GCN to construct a GNN models. \n\n---\n\nQ4: Meaning of “prompt level”\n\nA4: Prompt-level means that for each input prompt (e.g., a mathematical problem from GSM8K), we extract the token embeddings for all tokens within that specific prompt, treating each token embedding as a point in the metric space. \n\nThe \"prompt level\" refers to this per-prompt analysis rather than computing hyperbolicity across the entire vocabulary or dataset. We repeat this process for all prompts in each dataset and then average the resulting δ values to obtain the overall hyperbolicity measure reported in Table 2. We will revise this section in the manuscript to detail this procedure more clearly.\n\n---\n\nQ5: Linear transformation in lines 266-268\n\nA5: Thanks for your comment. We direct your attention to the comprehensive mathematical formulation provided throughout Section 5 and Appendix D for the complete technical details.\n\nOur Direct Lorentz Low-rank Transformation (LLR) operates directly on the hyperbolic manifold using the formula LLR(BA, x^H) = (√(||BAx^H_s||² + K), BAx^H_s), where x^H_s represents the space-like component of the hyperbolic representation and BA are the low-rank adaptation matrices (Equation 9). This transformation applies the linear adaptation BA to the semantic space-like dimensions while recomputing the time component to preserve the Lorentz manifold constraint x₀² - ||xs||² = K. As detailed in Appendix D, this can be interpreted as a constrained Lorentz rotation that maintains hyperbolic geometry while enabling parameter-efficient adaptation. \n\nThe linear transformation is inspired by established hyperbolic neural networks, which allows us to capture the hierarchical inductive bias of hyperbolic geometry while maintaining the efficiency and compatibility.\n\n---\n\nQ6: GPU usage comparison is either missing or mislabeled.\n\nA6: We apologize for this mislabeling error. \n\n---\n\nQ7: The reason of using Poincaré disk model for discussing the connection between power-law distribution and hyperbolic geometry.\n\nA7: Thanks for your comments. While our main methodology employs the Lorentz model throughout due to its numerical stability and advantages for performing transformations within neural networks, the theoretical connection in Section 4.3 utilizes the Poincaré disk model because it provides a more intuitive geometric framework for explaining the relationship between power-law distributions and hyperbolic curvature. \n\nThe Poincaré disk model's polar coordinate system (r, θ) naturally aligns with the hierarchical organization we observe in token embeddings, where radial distance from the origin correlates with token specificity as demonstrated in our empirical analysis (Figure 1, Table 1). \n\nImportantly, the Lorentz and Poincaré disk models are isometric; they are different representations of the same underlying hyperbolic geometry and share equivalent geometric properties. Therefore, the choice of the Poincaré disk model for the conceptual derivation in Section 4.3 does not affect the mathematical validity of the connection we draw between power-law distributions and hyperbolic space.\n\nTo prevent any potential confusion for readers, we will revise the manuscript accordingly.\n\n---\n\nQ8: Connection between the power-law distribution and hyperbolic geometry\n\nA:  Thanks for your detailed and constructive feedback. 2D Poincaré disk model offers an intuitive visualization of exponential volume growth—a key property that generalizes to n-dimensional hyperbolic spaces. \n\nThe generalization to higher dimensions follows standard hyperbolic geometry principles where the exponential volume growth property (C(r) ∼ e^r and A(r) ∼ e^r as r → ∞) holds in n-dimensional hyperbolic space, not just 2D. \n\nOur core argument builds upon the established work of Krioukov et al. [19], which demonstrates that embedding random geometric graphs in an n-dimensional hyperbolic space naturally gives rise to scale-free networks with a power-law degree distribution. The polar coordinate system is a conceptual framework used in that work to model this phenomenon.  Our assignment of \"radial coordinate correlating with token frequency\" is motivated by our empirical findings in Section 4.1. The angular coordinate encoding semantic similarity follows from the hierarchical tree-like structure we demonstrate through δ-hyperbolicity analysis (Table 2). \n\n\n---\n\nQ9: In Equations 6, 7, and the following lines, the symbol is used without explanation of what kind of approximation or relation it indicates.\n\nA9: The symbol \"∼\" indicates asymptotic equivalence as r approaches infinity, meaning the ratio of the left-hand side to the right-hand side approaches 1 in the limit. We will make it clear in the improved version. \n\n---\n\nQ10: In line 263, the paper refers to Appendix E.1 for the stereographic projection formulas, but the actual definitions of and are in Appendix E.2.\n\nA10: Sorry for the typo. Line 263 should refer to Appendix E.2.\n\n---\n\nQ11 Missing referred information of Appendix F in main text\n\nA: The purpose of Appendix F is to provide the formal mathematical derivation that substantiates Proposition 1. \n\nSpecifically, it demonstrates how the non-linear projections inherent in HypLoRA introduce higher-order, norm-dependent terms into the adaptation, which is the core mechanism behind its improved performance.  Thanks for the reviewer’s comments. We will revise the main text to more clearly introduce the purpose of the appendix when it is referenced. \n\n---\n\nQ12: Proof of Proposition 1\n\nA:  The analysis in Appendix F serves as the mathematical derivation that substantiates Proposition 1. The core of the argument is that the sequence of non-linear projections to and from hyperbolic space results in an effective transformation that is dependent on the input token's L2 norm. As shown in Appendix F.1 (Eq. 30), the final update term from HypLoRA can be approximated as $\\Delta Q^{\\text{Hyp}} \\approx BAx + \\frac{||x||^2}{6K} \\cdot BAx$. The second term, $\\frac{||x||^2}{6K} \\cdot BAx$, is the higher-order modification introduced by our method. This term is directly proportional to the square of the L2 norm ($\\|x\\|^2$) of the input token embedding $x$. This explicit dependency is the key mechanism that allows HypLoRA to perform a more nuanced adaptation than standard LoRA, which only applies the linear transformation $BAx$.\n\nThe link between this norm-dependent modification and \"capturing hierarchical relationships\" stems directly from our empirical findings in Section 4.1. In our analysis (Figure 1, Table 1), we demonstrate that token embeddings are organized hierarchically, where token specificity correlates with L2 norm: general, high-frequency tokens (e.g., \"to\", \"and\") have smaller norms, while specific, low-frequency tokens (e.g., \"banana\", \"purple\") have larger norms. Because the higher-order term in HypLoRA is proportional to $\\|x\\|^2$, our method applies a larger, more significant adaptation to these specific, large-norm tokens. This allows the model to differentially weight the importance of tokens based on their position in the semantic hierarchy, enabling a more refined and context-aware fine-tuning."}, {"review_id": "2gWePw5XlH", "reviewer": "Reviewer_uuh3", "summary": "This paper studies the geometric properties of token embeddings in LLMs. This paper first performs a detailed analysis and find that LLM embeddings exhibit strong hyperbolic properties. Based on this observation, the authors propose a new PEFT method, HypLoRA, to perform low-rank adaptation directly on a hyperbolic manifold. Through extensive experiments on multiple benchmarks, the authors show that HypLoRA outperforms LoRA across a variety of models.", "questions": "1. As shown in Table 3, the LLaMA3-8B model fine-tuned with LoRA achieves 69.8% accuracy on GSM8K. However, the official technical report for LLaMA-3 (https://ai.meta.com/blog/meta-llama-3/) shows an 8-shot CoT accuracy of GSM8K is about 79.6%. Can you explain the evaluation settings that lead to this result? This significant difference raises potential concerns about the strength of the LoRA baseline used for comparison. If the baseline is not optimally tuned, the gains reported by HypLoRA may be partially due to a suboptimal starting point rather than simply the advantages of the hyperbolic method.\n\n2. Regarding the stability of fine-tuning: It is well known that the performance of LoRA can be sensitive to hyperparameter choices (e.g., learning rate, rank \"r\"). The paper compares the final performance of LoRA and HypLoRA, but a deeper analysis of training stability would be helpful. Can you comment on the stability of HypLoRA compared to LoRA? Are the reported results based on the average of multiple runs with different random seeds to explain the variance?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths:\n\n1. Novel and insightful perspective: The main strength of this paper lies in its novel and insightful perspective. Rather than simply proposing a new technique, the authors provide a compelling motivation by first analyzing the underlying geometric structure of LLM embeddings.\n\n2. Robust performance improvement: The proposed method, HypLoRA, shows consistent and robust performance improvement over the standard LoRA baseline. This is demonstrated across multiple base models (LLaMA, Gemma, etc.) and various inference benchmarks.\n3. Solid theoretical foundation: The research in this paper is well supported by theoretical analysis.\n\n## Weaknesses:\n\n1. Lack of pre-fine-tuning benchmarks: The experimental results table lacks the performance of the base model before any fine-tuning (i.e., zero-shot or few-shot performance). Including the performance of the base model allows the reader to measure the absolute improvement brought by PEFT itself. This is crucial to understand the overall impact of the two PEFT methods and to justify the need for fine-tuning in the first place.\n\n2. Motivation is disconnected from experimental analysis: The experimental analysis can be better aligned with the core motivation of the paper. The motivation of this paper is based on the idea that hyperbolic geometry can better model the hierarchical tree-like structures present in language. However, the evaluation section focuses mainly on showing its accuracy improvement on downstream tasks. The paper would be significantly strengthened if qualitative or quantitative analysis could be included to show how HypLoRA actually changes the embedding space to better represent these hierarchical structures.", "was_revised": true, "final_justification": "The authors' response addressed my concerns regarding the performance of the base model and baseline. They also provided additional experimental results supporting their motivation. Regarding parameter stability, the authors provided variance results, which showed that their method does exhibit significant volatility, but the improvement is stable and therefore acceptable. I believe the method proposed in this paper has potential for further research, and therefore this paper can be accepted by the main conference.", "rebuttal": "Thank you for your thorough review and constructive feedback. In what follows, we use “Q” to denote each question, weakness, or issue raised and “A” for our response.\n\n---\n\nQ1: Base models’ Performance\n\nA1: To address the reviewer’s concerns and in the limit of time, we report the arithmetic reasoning performance across four base LLMs, and on Gemma 3-4B for the commonsense reasoning evaluations.\n\nArithmetic reasoning task:\n\n| Model | MAWPS | SVAMP | GSM8K | AQuA |\n| --- | --- | --- | --- | --- |\n| LLaMA-7B | 51.7 | 32.4 | 15.7 | 16.9 |\n| LLaMA-13B | 65.5 | 37.5 | 32.4 | 15.0 |\n| Gemma-7B | 76.5 | 60.4 | 38.4 | 25.2 |\n| Gemma-3 4B | 54.6 | 42.2 | 39.2 | 23.2 |\n\nCommonsense reasoning task:\n\n| Model | BoolQ | PIQA | SIQA | winogrande | ARC-Challenge | ARC-Easy | OpenbookQA |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Gemma3-4B | 58.41 | 77.31 | 63.10 | 14.92 | 61.43 | 72.011  | 54.2 |\n\nThese baseline results clearly demonstrate the value of fine-tuning. More importantly, these benchmarks further highlight the contribution of our proposed method, HypLoRA. Compared to both the base LLM and the LoRA baseline, HypLoRA consistently outperforms across most evaluations in both arithmetic and commonsense reasoning tasks.\n\n---\n\nQ2: Motivation is disconnected from experimental analysis: The experimental analysis can be better aligned with the core motivation of the paper. \n\nA2:  Thanks for your suggestion. In this response, we present a quantitative study of the embeddings finetuned using HypLoRA. We extend our hyperbolicity analysis beyond the input embeddings. We examine the relative hyperbolicity of the final hidden layer representations of Gemma 7B and Gemma 3-4B across five datasets, two focused on math reasoning (AQuA, GSM8K) and three on commonsense reasoning (ARC-Challenge, Winogrande, OpenbookQA). We conduct this analysis on both the base models and their fine-tuned counterparts using three PEFT methods: LoRA, DoRA, and HypLoRA.\n\nFirst, as a baseline, the hyperbolicity of the initial, non-contextualized token embeddings is presented in Table 1.\n\n**Hyperbolicity of Inital Token Embeddings (Table1)**\n\n| Method | AQuA | GSM8K | ARC-Challenge | Winogrande | OpenbookQA |\n| --- | --- | --- | --- | --- | --- |\n| Gemma 7B | 0.12 ± 0.01 | 0.11 ± 0.01 | 0.12 ± 0.01 | 0.11 ± 0.01 | 0.12 ± 0.01 |\n| Gemma3 4B | 0.19 ± 0.01 | 0.19 ± 0.02 | 0.16 ± 0.02 | 0.17 ± 0.01 | 0.16 ± 0.02 |\n\n**Hyperbolicity of Final Hidden Layer Embeddings in Gemma 7B: (Table2-1)**\n\n| Dataset | Base LLM | LoRA | DoRA | HypLoRA |\n| --- | --- | --- | --- | --- |\n| AQuA | 0.31 ± 0.04 | 0.24 ± 0.05 | 0.23 ± 0.05 | 0.22 ± 0.03 |\n| GSM8K | 0.28 ± 0.04 | 0.21 ± 0.05 | 0.21 ± 0.05 | 0.20 ± 0.03 |\n| ARC-Challenge | 0.30 ± 0.03 | 0.35 ± 0.03 | 0.36 ± 0.02 | 0.25 ± 0.02 |\n| Winogrande | 0.22 ± 0.04 | 0.32 ± 0.02 | 0.27 ± 0.02 | 0.27 ± 0.02 |\n| OpenbookQA | 0.30 ± 0.03 | 0.35 ± 0.03 | 0.38 ± 0.02 | 0.25 ± 0.02 |\n\n**Hyperbolicity of Final Hidden Layer Embeddings in  Gemma3 4B: (Table2-2)**\n\n| Dataset | Base LLM | LoRA | DoRA | HypLoRA |\n| --- | --- | --- | --- | --- |\n| AQuA | 0.17 ± 0.03 | 0.17 ± 0.03 | 0.19 ± 0.02 | 0.11 ± 0.01 |\n| GSM8K | 0.16 ± 0.03 | 0.20 ± 0.03 | 0.19 ± 0.03 | 0.11 ± 0.02 |\n| ARC-Challenge | 0.17 ± 0.02 | 0.21 ± 0.01 | 0.17 ± 0.02 | 0.20 ± 0.02 |\n| Winogrande | 0.16 ± 0.02 | 0.16 ± 0.02 | 0.21 ± 0.01 | 0.12 ± 0.01 |\n| OpenbookQA | 0.17 ± 0.03 | 0.16 ± 0.02 | 0.17 ± 0.03 | 0.11 ± 0.01 |\n\nThe results for the final hidden layer embeddings are shown in Table 2-1 and .Table 2-2.  Our analysis reveals several key findings: First, the final hidden states of the base models exhibit notable hyperbolicity, though it is generally less pronounced (i.e., higher $\\delta$ values) than in the initial embeddings. Second, HypLoRA consistently learns final representations with a higher degree of hyperbolicity (lower δ values) across almost all datasets and models. This provides strong empirical evidence that our method actively preserves and enhances the hierarchical structure of the representations throughout the model, aligning the final contextualized embeddings with the geometric biases that are beneficial for reasoning. We also observe that as dataset complexity increases, the final hidden layer exhibits correspondingly higher relative hyperbolicity value.\n\nFor instance, AQuA, a particularly challenging dataset, shows a relative hyperbolicity of 0.31 in Gemma 7B and 0.17 in Gemma 3-4B. When these models were fine-tuned with HypLoRA, the degree of hyperbolicity increased (i.e., the relative hyperbolicity value decreased). In Gemma 7B, the drop from 0.31 to 0.22 led to a significant improvement in accuracy, reaching 46.5%, compared to LoRA’s 34.3%, which had a relative hyperbolicity of 0.24. Similar trends are observed in Gemma 3-4B as well. Moreover, in commonsense reasoning tasks, the effect is even more pronounced: while LoRA and DoRA tend to reduce the degree of hyperbolicity relative to the base LLM, HypLoRA increases it. As a result, the performance improvement with HypLoRA is both evident and consistent.\n****\n\n---\n\nQ3: As shown in Table 3, the LLaMA3-8B model fine-tuned with LoRA achieves 69.8% accuracy on GSM8K. However, the official technical report for LLaMA-3 (https://ai.meta.com/blog/meta-llama-3/) shows an 8-shot CoT accuracy of GSM8K is about 79.6%. Can you explain the evaluation settings that lead to this result? This significant difference raises potential concerns about the strength of the LoRA baseline used for comparison. If the baseline is not optimally tuned, the gains reported by HypLoRA may be partially due to a suboptimal starting point rather than simply the advantages of the hyperbolic method. \n\nA3: We follow the experimental setup outlined in (Hu et al.)[1], which has been adapted by various works. For arithmetic and commonsense reasoning tasks, we finetune the LLMs using Math10k and  Commense170k datasets, respectively proposed in (Hu et al.)[1]. For the PEFT baselines, we adopt standard hyperparameters: lora rank 32, lora alpha of 64 and apply it at QKVUD. We use a batch size of 16, micro-batch size of 4, a learning rate of 3e-4, and train for 3 epochs with a maximum sequence length of 256 tokens. These settings are consistent with Hu et al[1]*.*, DoRA, and are also similar to those used in the S2FT[2] framework.  Notably, similar LoRA performance on GSM8K can be observed in the S2FT paper. Importantly, the 8-shot accuracy reported in the official LLaMA-3 blog refers to the chain-of-thought prompting, which is not directly comparable to our setting, where the model is fine-tuned using Math10K, a dataset that includes GSM8K, AQuA, and MAWPS. Thus, drawing a direct comparison between these two settings would not be valid, as they represent different evaluation paradigms.\n\n[1]Hu, Z., Wang, L., Lan, Y., Xu, W., Lim, E.P., Bing, L., Xu, X., Poria, S. and Lee, R.K.W., 2023. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. EMNLP 2023\n\n[2]Yang, X., Leng, J., Guo, G., Zhao, J., Nakada, R., Zhang, L., Yao, H. and Chen, B., 2024. S $^{2}$FT: Efficient, scalable and generalizable LLM fine-tuning by structured sparsity. *Advances in Neural Information Processing Systems*, *37*, pp.59912-59947.\n\n---\n\nQ: Regarding the stability of fine-tuning: It is well known that the performance of LoRA can be sensitive to hyperparameter choices (e.g., learning rate, rank \"r\"). The paper compares the final performance of LoRA and HypLoRA, but a deeper analysis of training stability would be helpful. Can you comment on the stability of HypLoRA compared to LoRA? Are the reported results based on the average of multiple runs with different random seeds to explain the variance? \n\nA:  We thank the reviewer for the suggestion. The reported results were from the best single run across different curvature values (0.5, 1.0, 2.0). To address stability concerns and in limit of time, we report mean and std over 3 runs for LLaMA-3B (LoRA vs. HypLoRA) and for Gemma3-4B (HypLoRA). While HypLoRA shows slightly higher variance in some cases, it consistently outperforms LoRA on most arithmetic tasks. Improving stability is an exciting direction for future work.\n\n| Model | MAWPS | SVAMP | GSM8K | AQuA |\n| --- | --- | --- | --- | --- |\n| LLaMA3-8B (LoRA) | 91.04 ± 0.14 | 80.75 ± 0.45 | 70.88 ± 0.99 | 41.77 ± 0.49 |\n| LLaMA3-8B (HypLoRA) | 91.46 ± 0.54  | 79.27 ± 1.26 | 72.93 ± 0.27 | 44.75 ± 0.60 |\n| Gemma 3 4B (HypLoRA) | 91.28 ± 0.46 | 77.77 ± 1.17 | 67.62 ± 1.22 | 46.89 ± 1.19 |\n\n---"}, {"review_id": "Bvh8flCZ7c", "reviewer": "Reviewer_2bQs", "summary": "The paper explores the suitability of hyperbolic geometry for representing token embeddings in large language models (LLMs), motivated by the hierarchical structure commonly found in language data. It first identifies that token embeddings naturally display hyperbolic characteristics, such as a tree-like structure, where high-frequency (general) tokens cluster near the origin, and low-frequency (specific) tokens are positioned farther away, following a power-law distribution.\n\nTo better exploit this hierarchical structure, the authors propose HypLoRA, a new fine-tuning method that integrates hyperbolic geometry directly into the low-rank adaptation process, unlike traditional methods that use Euclidean geometry. The key idea of HypLoRA is to perform parameter-efficient updates directly on the hyperbolic manifold, avoiding the limitations of using standard Euclidean operations.\n\nThrough extensive experiments on arithmetic and commonsense reasoning benchmarks, the paper demonstrates that HypLoRA significantly improves the performance of existing LLMs compared to conventional Euclidean fine-tuning methods, confirming the advantage of explicitly modeling token hierarchies in hyperbolic space.", "questions": "NA", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n* Extensive experiments across diverse benchmarks and models\n* Integration of hyperbolic geometry into low-rank adaptation (HypLoRA)\n\n**Weaknesses:**\n\n* Incremental improvements may not justify increased complexity and computational overhead in practical deployments.", "was_revised": true, "final_justification": "The authors have addressed my concern on the increased complexity and computational overhead.", "rebuttal": "Q: Incremental improvements may not justify increased complexity and computational overhead in practical deployments. \n\nA: We thank the reviewer for raising the practical consideration of the trade-off between performance gains and computational overhead. We agree that any added complexity must be justified by meaningful improvements. \n\nWhile HypLoRA introduces hyperbolic projection operations, we have taken care to ensure its practical overhead is minimal and justified by its performance benefits.  In terms of model complexity, the number of additional trainable parameters in HypLoRA is negligible, remaining nearly **identical** to that of the standard LoRA method (**as shown in Tables 3 and 7**). \n\nRegarding computational overhead, our analysis in Section 5.2 shows that while inference time is slightly increased compared to LoRA, HypLoRA remains more efficient than other competitive methods like DoRA (Figure 2). Furthermore, memory consumption is comparable to LoRA (Table 7) and in the table below, ensuring our method remains a practical choice for deployment.\n\n**GPU Allocated Memory (Finetuning, in GB)**\n\n| Model | Method | Allocated Memory (GB) |\n| --- | --- | --- |\n| **Qwen2.5 - r32** | LoRA | 28.63 |\n|  | DoRA | 28.64 |\n|  | HypLoRA (stereographic) | 28.63  |\n|  | HypLoRA (exp/log) | 28.63 |\n| **Gemma3 - 4B** | LoRA | 14.61 |\n|  | DoRA | 14.62 |\n|  | HypLoRA (stereographic) | 14.61 |\n|  | HypLoRA (exp/log) | 14.61  |\n| **LLaMA3 - 8B** | LoRA | 30.13 |\n|  | DoRA | 30.14 |\n|  | HypLoRA (stereographic) | 30.13  |\n|  | HypLoRA (exp/log) | 30.13  |\n\nBesides, the performance improvements offered by HypLoRA are not merely incremental, but are particularly significant on more challenging reasoning tasks. For instance, on the complex GSM8K and AQUA arithmetic benchmarks, HypLoRA delivers substantial gains over LoRA across multiple models, such as a +9.8% and +12.2% improvement with Gemma-7B, respectively (Table 3). Similarly, on commonsense reasoning, we observe consistent gains of +1.8-3.0% on average (Table 4). This demonstrates that the benefits of explicitly modeling the latent hyperbolic geometry are most pronounced where complex, hierarchical relationships must be understood. Therefore, we believe the slight and manageable increase in computational cost is a worthwhile trade-off for the significant and targeted performance enhancements HypLoRA provides on difficult reasoning tasks."}]

Author rebuttal:

Meta-review: This paper proposes HypLoRA, a method for fine-tuning large language models that exploits the latent hierarchical structure in token embeddings. The motivation is based on the observation that token embeddings display hyperbolic characteristics, reflecting a tree-like structure. Standard low rank adaptation in Euclidean space does not preserve this property, while HypLoRA applies a Lorentz transformation directly in hyperbolic space, resulting in norm-dependent updates.

The strengths are a novel and well-motivated idea, a technically sound method with minimal additional cost, and thorough experiments across multiple models and reasoning benchmarks. Reviewers appreciated the additional analysis in the rebuttal, including evidence that hyperbolicity persists in hidden layers, ablations confirming the role of the mapping operations, and clarifications on baselines and efficiency. These responses resolved the main concerns, and all reviewers now support acceptance.

The weaknesses were mostly related to clarity in the initial submission, the strength of baseline comparisons, and a gap between the geometric motivation and the evaluation. Some reviewers also noted that the performance gains, while consistent, are moderate and that fine-tuning stability could be improved. Overall, this is a solid and creative contribution that connects hyperbolic geometry with parameter-efficient fine-tuning and demonstrates consistent benefits.
Decision: Accept (spotlight)