Title: Hierarchical Hyperbolic Embedding: Enhancing Long Context Comprehension of Large Language Models via Fusing Local and Global Features in Hyperbolic Space
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, Positional Embedding, Hyperbolic Space

Abstract: Extending the context length of Large Language Models (LLMs) by improving Rotary Position Embedding (RoPE) has become a trend. While existing works mainly address RoPE’s limitations within attention mechanism, this paper provides an analysis across nearly all parts of LLMs, uncovering their adverse effects on length generalization for RoPE-based attention. RoPE mapping knowledge to polar coordinates has the following drawbacks: 1) it cannot support exponential growth 2) it can not represent the complex hierarchical knowledge contained in the long context, 3)coordinates lead to distortions, in polar coordinates, points on different radii are ignored if they have the same Angle. Building on our observations, we propose HHPE(Hierarchical Hyperbolic Positional Embedding), which enhances long context comprehension of Large Language Models via fusing local and global features in hyperbolic space. HHPE can naturally represent hierarchical structure with equidistant embedding capability and exponential growth. Experiments across various model scales show that HHPE can maintain a more stable perplexity. Several analyses and ablations bring further support to our method and theoretical modeling.

TL;DR: Enhances long context comprehension of Large Language Models via fusing local and global features in hyperbolic space