HVT: A Comprehensive Vision Framework for Learning in Non-Euclidean Space

## Abstract

Abstract Data representation in non-Euclidean spaces has proven effective for capturing hierarchical and complex relationships in real-world datasets. Hyperbolic spaces, in particular, provide efficient embeddings for hierarchical structures. This paper introduces the Hyperbolic Vision Transformer (HVT), a novel extension of the Vision Transformer (ViT) that integrates hyperbolic geometry. While traditional ViTs operate in Euclidean space, our method enhances the self-attention mechanism by leveraging hyperbolic distance and Möbius transformations. This enables more effective modeling of hierarchical and relational dependencies in image data. We present rigorous mathematical formulations, showing how hyperbolic geometry can be incorporated into attention layers, feed-forward networks, and optimization. We offer improved performance for image classification using the ImageNet dataset.

## 1 Introduction

Representation learning is fundamental to modern machine learning, enabling models to extract meaningful features from raw data Bengio et al. (). While Euclidean spaces are traditionally used to model data relationships, many real-world datasets—including images—exhibit hierarchical structures better captured in non-Euclidean spaces Bronstein et al. ().

Images are inherently hierarchical, comprising structures at multiple scales: from pixels to edges, from shapes to objects, and ultimately to entire scenes Biederman (); Riesenhuber & Poggio (). This hierarchy can be conceptualized as:

- •
Pixels $\{p_{i}\}$: The basic units of an image.
- •
Edges $\{e_{j}\}$: Formed by grouping pixels with significant intensity gradients.
- •
Shapes $\{s_{k}\}$: Created by combining edges into simple motifs.
- •
Objects $\{o_{l}\}$: Recognizable entities composed of shapes.
- •
Scenes $\mathcal{I}$: Complete images where objects interact within a context.

The hierarchical nature of images implies that higher-level concepts are built upon lower-level features, reflecting a tree-like structure. Vision Transformers (ViTs) Dosovitskiy et al. () process images by dividing them into patches, treating each patch as a token. This patch-based approach introduces a hierarchical representation because:

- 1.
Local Features: Each patch captures local patterns such as textures or edges.
- 2.
Global Context: By attending over patches, the model aggregates local information to understand the overall structure.

This mirrors the hierarchical composition of images, from local to global features.

Hyperbolic spaces are well-suited for modeling hierarchical data due to their ability to embed tree-like structures with minimal distortion Nickel & Kiela (). By utilizing hyperbolic geometry and Möbius transformations, we can effectively capture the multi-scale dependencies inherent in visual data Ganea et al. (). Specifically, Möbius transformations allow for operations like addition and scalar multiplication in hyperbolic space, enabling neural networks to perform calculations while preserving hierarchical relationships.

In this paper, we propose the *Hyperbolic Vision Transformer*, which integrates hyperbolic geometry into the transformer architecture. Our contributions include:

- •
Hyperbolic Neural Components: Extending ViT to operate in hyperbolic space using hyperbolic versions of neural network components, such as attention mechanisms and linear layers.
- •
Möbius Transformations in ViT: Demonstrating how Möbius transformations enable operations in hyperbolic space, preserving hierarchical data structures.
- •
Theoretical and Empirical Analysis: Providing insights and evaluations showing improved modeling of hierarchical structures over traditional Euclidean approaches.

## 2 Related Work

### 2.1 Hyperbolic Geometry in Machine Learning

Recent advances have leveraged hyperbolic geometry for machine learning, significantly impacting how hierarchical data is modeled. Nickel & Kiela () successfully used Poincaré embeddings for such data, showing marked improvements over Euclidean embeddings. This approach was further refined by Ganea et al. (), who introduced hyperbolic embeddings for entailment cones, effectively capturing asymmetric relationships.

Khrulkov et al. () and Liu et al. () extended these concepts to visual data and zero-shot recognition, respectively, underscoring the versatility of hyperbolic embeddings in handling complex visual tasks. Most notably, Ermolov et al. () developed Hyperbolic Vision Transformers (HVTs) that incorporate these embeddings within Vision Transformer architectures, enhancing metric learning. Our model extends this integration by incorporating hyperbolic geometry throughout the transformer operations, from Möbius transformations to hyperbolic self-attention.

### 2.2 Hyperbolic Neural Networks and Attention

Ganea et al. () initially formulated hyperbolic neural networks, introducing layers and activation functions suited for hyperbolic spaces. This foundation was expanded by Bachmann et al. (), who focused on optimizing these networks efficiently. Our method enriches this foundation by embedding hyperbolic layers directly within transformer architectures, enhancing the adaptability and depth of hyperbolic operations.

Another method presented in Hyperbolic Attention Networks Gulcehre et al. () differs from ours primarily in how hyperbolic geometry is applied within the attention mechanism. Gulcehre et al. focus on embedding the activations into hyperbolic space using both the hyperboloid and Klein models, leveraging hyperbolic matching and aggregation operations. In contrast, our approach incorporates learnable curvature within positional embeddings, head-specific scaling in attention, and hyperbolic layer normalization, offering more flexibility and efficiency in capturing hierarchical data. Additionally, we utilize the Poincaré ball model for its computational suitability in vision tasks.

### 2.3 Vision Transformers

Initially developed for NLP by Vaswani et al. (), Vision Transformers (ViT) have been adapted for visual tasks, as demonstrated by Dosovitskiy et al. (). Enhancements such as those proposed by Caron et al. () and El-Nouby et al. () have refined ViTs for self-supervised learning. Unlike these Euclidean-based models, our Hyperbolic Vision Transformer employs hyperbolic geometry to model hierarchical and relational data structures more effectively.

### 2.4 Comparison to Key Hyperbolic Methods

While Ermolov et al. () introduced HVTs that focus on hyperbolic embeddings for metric learning, our approach fully integrates hyperbolic operations throughout the transformer, significantly enhancing its ability to manage hierarchical data beyond the objectives of metric learning. We extend the application scope to include image classification by embedding hyperbolic geometry directly into the core components of the Vision Transformer.

Recently, Yang et al. () proposed Hypformer, an efficient hyperbolic Transformer based on the Lorentz model of hyperbolic geometry. Hypformer introduces two foundational blocks—Hyperbolic Transformation with Curvatures (HTC) and Hyperbolic Readjustment and Refinement with Curvatures (HRC)—to define essential Transformer modules in hyperbolic space. They also develop a linear self-attention mechanism in hyperbolic space to handle large-scale graph data and long-sequence inputs efficiently.

While Hypformer makes significant contributions to the development of hyperbolic Transformers, particularly in processing large-scale graph data, our model differs in several key aspects:

- •
Model Focus and Application Domain: Hypformer is primarily designed for graph data and emphasizes scalability and efficiency in handling large-scale graphs and long sequences. In contrast, our model focuses on vision tasks, specifically image classification, integrating hyperbolic geometry throughout the Vision Transformer architecture to enhance the modeling of hierarchical and relational structures inherent in visual data.
- •
Hyperbolic Model Used: Hypformer operates in the Lorentz model of hyperbolic geometry, whereas we utilize the Poincaré ball model. The Poincaré ball model is advantageous for vision tasks due to its conformal properties, which preserve angles and better represent geometric structures in image data.
- •
Innovative Components: Our model introduces unique components such as learnable curvature in positional embeddings, head-specific scaling in the attention mechanism, hyperbolic layer normalization, gradient clipping, geodesic regularization, and layer scaling for training stability. These innovations are specifically tailored to enhance the performance of Vision Transformers in hyperbolic space.
- •
Attention Mechanism: While Hypformer develops a linear self-attention mechanism to address efficiency in handling large-scale data, our model extends the standard self-attention mechanism into hyperbolic space using Möbius operations and hyperbolic distance calculations. This approach allows us to capture complex relationships in visual data more effectively.

By fully integrating hyperbolic operations within the Vision Transformer framework and focusing on the unique challenges of vision tasks, our model offers a comprehensive solution that provides superior performance over previous hyperbolic Transformers, including Hypformer, in the domain of image classification.

## 3 Proposed Method

This section presents the mathematical foundations of the Hyperbolic Vision Transformer Network (HVT). We cover essential concepts from hyperbolic geometry and describe how we adapt Vision Transformer components to operate in hyperbolic space. Our main contributions include introducing learnable curvature in positional embeddings, head-specific scaling in attention mechanisms, hyperbolic layer normalization, gradient clipping, geodesic regularization, and layer scaling for training stability.

Code Availability: Code is available at [https://github.com/hyperbolicvit/hyperbolicvit](https://github.com/hyperbolicvit/hyperbolicvit)

### 3.1 Hyperbolic Geometry Preliminaries

Hyperbolic space, characterized by constant negative curvature, embeds hierarchical and complex structures common in visual data. We adopt the Poincaré ball model due to its computational convenience and suitability for representing image data structures.

#### 3.1.1 Poincaré Ball Model

The $n$-dimensional Poincaré ball model is defined as the manifold:

$$ $\mathbb{D}^{n}=\left\{\mathbf{x}\in\mathbb{R}^{n}:\|\mathbf{x}\|<1\right\},$ (1) $$

where $\|\cdot\|$ denotes the Euclidean norm. The Riemannian metric tensor $g_{\mathbf{x}}$ of this manifold is given by:

$$ $g_{\mathbf{x}}=\lambda_{\mathbf{x}}^{2}g^{E},\quad\text{with}\quad\lambda_{ \mathbf{x}}=\frac{2}{1-\|\mathbf{x}\|^{2}},$ (2) $$

where $g^{E}$ is the Euclidean metric tensor, and $\lambda_{\mathbf{x}}$ is the conformal factor that scales the Euclidean metric to account for the curvature of hyperbolic space.

#### 3.1.2 Möbius Operations

To adapt Vision Transformer components to hyperbolic space, we utilize Möbius transformations, essential for processing vectors within the Poincaré ball model.

##### Möbius Addition

For vectors $\mathbf{x},\mathbf{y}\in\mathbb{D}^{n}$, the Möbius addition $\mathbf{x}\oplus\mathbf{y}$ is defined as:

$$ $\mathbf{x}\oplus\mathbf{y}=\frac{(1+2\langle\mathbf{x},\mathbf{y}\rangle+\| \mathbf{y}\|^{2})\mathbf{x}+(1-\|\mathbf{x}\|^{2})\mathbf{y}}{1+2\langle \mathbf{x},\mathbf{y}\rangle+\|\mathbf{x}\|^{2}\|\mathbf{y}\|^{2}},$ (3) $$

where $\langle\cdot,\cdot\rangle$ denotes the Euclidean inner product. Möbius addition generalizes vector addition to hyperbolic space, ensuring the result remains within the manifold.

##### Möbius Scalar Multiplication

For a scalar $r\in\mathbb{R}$ and a vector $\mathbf{x}\in\mathbb{D}^{n}$, Möbius scalar multiplication $r\otimes\mathbf{x}$ is defined as:

$$ $r\otimes\mathbf{x}=\tanh\left(r\tanh^{-1}(\|\mathbf{x}\|)\right)\frac{\mathbf{ x}}{\|\mathbf{x}\|}.$ (4) $$

This operation scales a vector while preserving its direction and ensures the scaled vector remains within the Poincaré ball.

##### Möbius Matrix-Vector Multiplication

Given a matrix $\mathbf{W}\in\mathbb{R}^{m\times n}$ and a vector $\mathbf{x}\in\mathbb{D}^{n}$, Möbius matrix-vector multiplication $\mathbf{W}\otimes_{M}\mathbf{x}$ is defined as:

$$ $\mathbf{W}\otimes_{M}\mathbf{x}=\tanh\left(\frac{\|\mathbf{W}\mathbf{x}\|}{\| \mathbf{x}\|}\tanh^{-1}(\|\mathbf{x}\|)\right)\frac{\mathbf{W}\mathbf{x}}{\| \mathbf{W}\mathbf{x}\|},\quad\text{if}\quad\mathbf{x}\neq\mathbf{0}.$ (5) $$

This operation extends Möbius scalar multiplication to linear transformations, allowing us to apply linear layers within hyperbolic space.

##### Möbius Concatenation

To combine multiple vectors $\mathbf{x}_{1},\mathbf{x}_{2},\ldots,\mathbf{x}_{n}\in\mathbb{D}^{n}$, we use Möbius concatenation:

$$ $\bigoplus_{j=1}^{n}\mathbf{x}_{j}=\mathbf{x}_{1}\oplus\mathbf{x}_{2}\oplus \cdots\oplus\mathbf{x}_{n}.$ (6) $$

### 3.2 Hyperbolic Neural Network Components

We incorporate the hyperbolic operations into our neural network components to enable the Vision Transformer architecture to operate within hyperbolic space.

#### 3.2.1 Hyperbolic Linear Layer

Traditional linear layers are adapted to hyperbolic space using Möbius matrix-vector multiplication followed by Möbius addition with a bias:

$$ $\mathbf{h}=\mathbf{W}\otimes_{M}\mathbf{x}\oplus\mathbf{b},$ (7) $$

where $\mathbf{x}\in\mathbb{D}^{n}$, $\mathbf{W}\in\mathbb{R}^{m\times n}$, and $\mathbf{b}\in\mathbb{D}^{m}$. This layer allows us to perform linear transformations while respecting the geometry of hyperbolic space.

#### 3.2.2 Hyperbolic Activation and Normalization

Activation functions and normalization are applied in the tangent space at the origin to leverage familiar Euclidean operations. The logarithmic map $\log_{0}:\mathbb{D}^{n}\rightarrow T_{0}\mathbb{D}^{n}$ maps points from the manifold to the tangent space:

$$ $\log_{0}(\mathbf{x})=\frac{2\tanh^{-1}(\|\mathbf{x}\|)}{\|\mathbf{x}\|}\mathbf {x}.$ (8) $$

The exponential map $\exp_{0}:T_{0}\mathbb{D}^{n}\rightarrow\mathbb{D}^{n}$ brings points back to the manifold:

$$ $\exp_{0}(\mathbf{v})=\tanh\left(\frac{\|\mathbf{v}\|}{2}\right)\frac{\mathbf{v }}{\|\mathbf{v}\|}.$ (9) $$

Using these maps, we define hyperbolic versions of the ReLU activation and Layer Normalization:

$$ $\displaystyle\text{ReLU}^{\mathbb{D}}(\mathbf{x})$ $\displaystyle=\exp_{0}\left(\text{ReLU}\left(\log_{0}(\mathbf{x})\right)\right),$ (10) $\displaystyle\text{LayerNorm}^{\mathbb{D}}(\mathbf{x})$ $\displaystyle=\exp_{0}\left(\text{LayerNorm}\left(\log_{0}(\mathbf{x})\right) \right).$ (11) $$

This approach allows us to apply standard activation and normalization techniques within hyperbolic space.

#### 3.2.3 Hyperbolic Layer Scaling

To stabilize residual connections in hyperbolic space, we introduce a learnable scaling parameter $\beta$:

$$ $\mathbf{X}^{\prime}=\mathbf{X}\oplus\beta\otimes\mathbf{O},$ (12) $$

where $\mathbf{X}$ is the input, $\mathbf{O}$ is the output from a layer, and $\beta\in\mathbb{R}$ scales the residual contribution.

### 3.3 Hyperbolic Vision Transformer Architecture

We integrate hyperbolic geometry into the Vision Transformer architecture by modifying key components to operate within hyperbolic space.

#### 3.3.1 Learnable Hyperbolic Positional Embeddings

Positional embeddings are adjusted using a learnable curvature parameter $c$ to represent positional information in hyperbolic space better:

$$ $\displaystyle\mathbf{E}_{\text{pos}}$ $\displaystyle=c\otimes\mathbf{E}_{\text{pos}}^{0},$ (13) $\displaystyle\mathbf{X}$ $\displaystyle=\mathbf{X}\oplus\mathbf{E}_{\text{pos}},$ (14) $$

where $\mathbf{E}_{\text{pos}}^{0}\in\mathbb{D}^{1\times N\times E}$ are the initial positional embeddings.

#### 3.3.2 Hyperbolic Self-Attention Mechanism

We extend the self-attention mechanism to hyperbolic space using Möbius operations and hyperbolic distance computations. Table 1 summarizes the symbols used.

**Table 1: Symbols and their descriptions.**
| Symbol | Description |
| --- | --- |
| $B$ | Batch size. |
| $H$ | Number of attention heads. |
| $N$ | Sequence length. |
| $D$ | Head dimension, where $D=\frac{E}{H}$ and $E$ is the embedding dimension. |
| $c$ | Curvature parameter of the Poincaré ball model. |
| $\epsilon$ | Small constant for numerical stability $(\epsilon=1\times 10^{-15})$. |
| $\delta$ | Small constant for clamping $(\delta=1\times 10^{-7})$. |
| $p$ | DropConnect Wan et al. () probability. |

Given input embeddings $\mathbf{X}\in\mathbb{D}^{B\times N\times E}$, we compute the queries $\mathbf{Q}$, keys $\mathbf{K}$, and values $\mathbf{V}$ using hyperbolic linear layers:

$$ $\displaystyle\mathbf{Q}$ $\displaystyle=\mathbf{X}\otimes_{M}\mathbf{W}_{Q}\oplus\mathbf{b}_{Q},$ (15) $\displaystyle\mathbf{K}$ $\displaystyle=\mathbf{X}\otimes_{M}\mathbf{W}_{K}\oplus\mathbf{b}_{K},$ (16) $\displaystyle\mathbf{V}$ $\displaystyle=\mathbf{X}\otimes_{M}\mathbf{W}_{V}\oplus\mathbf{b}_{V}.$ (17) $$

We reshape these tensors to accommodate multiple attention heads:

$$ $\mathbf{Q},\mathbf{K},\mathbf{V}\in\mathbb{D}^{B\times H\times N\times D}.$ (18) $$

##### Hyperbolic Distance Computation

The hyperbolic distance between a query $\mathbf{Q}_{b,h,i}$ and a key $\mathbf{K}_{b,h,j}$ is computed using the distance function in the Poincaré ball model:

$$ $d_{\mathbb{D}}(\mathbf{Q}_{b,h,i},\mathbf{K}_{b,h,j})=\cosh^{-1}\left(1+\frac{ 2c\|\mathbf{Q}_{b,h,i}\oplus(-\mathbf{K}_{b,h,j})\|^{2}}{(1-c\|\mathbf{Q}_{b,h ,i}\|^{2})(1-c\|\mathbf{K}_{b,h,j}\|^{2})+\epsilon}\right).$ (19) $$

To ensure numerical stability, we clamp the argument of $\cosh^{-1}$:

$$ $\mathcal{A}_{b,h,i,j}=\max\left(1+\frac{2c\|\mathbf{Q}_{b,h,i}\oplus(-\mathbf{ K}_{b,h,j})\|^{2}}{(1-c\|\mathbf{Q}_{b,h,i}\|^{2})(1-c\|\mathbf{K}_{b,h,j}\|^{ 2})+\epsilon},\ 1+\delta\right).$ (20) $$

We then compute the distance as:

$$ $d_{\mathbb{D}}(\mathbf{Q}_{b,h,i},\mathbf{K}_{b,h,j})=\log\left(\mathcal{A}_{b ,h,i,j}+\sqrt{\mathcal{A}_{b,h,i,j}^{2}-1}\right).$ (21) $$

##### Attention Scores and Weights

Using the computed distances, we calculate the attention scores with a head-specific scaling factor $\alpha_{h}$:

$$ $\mathbf{A}_{b,h,i,j}=-\frac{d_{\mathbb{D}}(\mathbf{Q}_{b,h,i},\mathbf{K}_{b,h, j})^{2}}{\alpha_{h}\sqrt{D}}.$ (22) $$

The attention weights are obtained by normalizing the scores using the softmax function:

$$ $\alpha_{b,h,i,j}=\frac{\exp(\mathbf{A}_{b,h,i,j})}{\sum_{k=1}^{N}\exp(\mathbf{ A}_{b,h,i,k})}.$ (23) $$

##### Output Computation

Each attention head produces an output by aggregating the value vectors, weighted by the attention weights:

$$ $\mathbf{O}_{b,h,i}=\bigoplus_{j=1}^{N}\left(\alpha_{b,h,i,j}\otimes\mathbf{V}_ {b,h,j}\right).$ (24) $$

The outputs from all heads are combined using Möbius concatenation and transformed with a final linear layer:

$$ $\mathbf{O}_{b,i}=\mathbf{W}_{O}\otimes_{M}\left(\bigoplus_{h=1}^{H}\mathbf{O}_ {b,h,i}\right)\oplus\mathbf{b}_{O}.$ (25) $$

DropConnect regularization is applied to the attention weights during training with probability $p$.

##### Residual Connection with Layer Scaling

As before, we use a residual connection with a learnable scaling parameter $\beta$:

$$ $\mathbf{X}^{\prime}=\mathbf{X}\oplus\beta\otimes\mathbf{O}.$ (26) $$

#### 3.3.3 Hyperbolic Feed-Forward Network

The feed-forward network in hyperbolic space consists of two hyperbolic linear layers with a hyperbolic ReLU activation in between:

$$ $\displaystyle\mathbf{H}_{1}$ $\displaystyle=\mathbf{X}^{\prime}\otimes_{M}\mathbf{W}_{1}\oplus\mathbf{b}_{1},$ (27) $\displaystyle\mathbf{H}_{2}$ $\displaystyle=\text{ReLU}^{\mathbb{D}}(\mathbf{H}_{1}),$ (28) $\displaystyle\mathbf{H}_{3}$ $\displaystyle=\mathbf{H}_{2}\otimes_{M}\mathbf{W}_{2}\oplus\mathbf{b}_{2},$ (29) $$

followed by a residual connection and hyperbolic layer normalization:

$$ $\mathbf{X}^{\prime\prime}=\text{LayerNorm}^{\mathbb{D}}\left(\mathbf{X}^{ \prime}\oplus\beta\otimes\mathbf{H}_{3}\right).$ (30) $$

### 3.4 Optimization in Hyperbolic Space

Optimizing neural networks in hyperbolic space presents unique challenges due to its non-Euclidean nature. We employ several techniques to ensure stable and effective training.

##### Gradient Clipping

To prevent large gradients from destabilizing training, we clip gradients in the tangent space:

$$ $\operatorname{grad}_{\text{clipped}}=\begin{cases}\operatorname{grad}_{\bm{ \theta}}&\text{if }\|\operatorname{grad}_{\bm{\theta}}\|\leq v,\\ v\cdot\dfrac{\operatorname{grad}_{\bm{\theta}}}{\|\operatorname{grad}_{\bm{ \theta}}\|}&\text{otherwise},\end{cases}$ (31) $$

where $\|\cdot\|$ is the Euclidean norm in the tangent space, and $v$ is the clipping threshold.

##### Riemannian Adam Optimizer

We utilize the Riemannian Adam optimizer, which adapts the Adam optimization algorithm to Riemannian manifolds. The update rule is:

$$ $\bm{\theta}_{t+1}=\exp_{\bm{\theta}_{t}}\left(-\eta_{t}\cdot\frac{m_{t}^{\text {clipped}}}{\sqrt{v_{t}^{\text{clipped}}}+\epsilon}\right),$ (32) $$

where $\eta_{t}$ is the learning rate, $m_{t}^{\text{clipped}}$ and $v_{t}^{\text{clipped}}$ are the clipped first and second moment estimates, and $\exp_{\bm{\theta}_{t}}$ is the exponential map at $\bm{\theta}_{t}$.

##### Geodesic Distance Regularization

To enhance class separation and encourage meaningful representations, we introduce a regularization term based on geodesic distances:

$$ $\mathcal{L}_{\text{geo}}=\lambda_{\text{reg}}\cdot\mathbb{E}_{(i,j)}\left[(1- \delta_{y_{i}y_{j}})\cdot d_{\mathbb{D}}(\mathbf{x}_{i},\mathbf{x}_{j})\right],$ (33) $$

where $\delta_{y_{i}y_{j}}$ is the Kronecker delta function indicating whether samples $i$ and $j$ belong to the same class. The total loss objective is $\mathcal{L}_{\text{cross entropy}}+\mathcal{L}_{\text{geo}}$.

##### Layer Scaling and Attention Scaling

Introducing the learnable scaling parameter $\beta$ in residual connections and head-specific scaling factors $\alpha_{h}$ in the attention mechanism helps control the magnitude of updates. It allows each attention head to adapt its sensitivity to hyperbolic distances.

##### Parameter Initialization

Weights are initialized using Xavier uniform initialization adapted for hyperbolic space Leimeister & Wilson (). All manifold parameters are initialized within the Poincaré ball to ensure valid representations and stable training.

Figure: Figure 1: Overall Model Flow
Refer to caption: extracted/5880549/files/method.png

### 3.5 Limitations

While the Hyperbolic Vision Transformer improves hierarchical data handling, it incurs higher computational demands due to the complexity of hyperbolic operations. Enhanced representation capabilities and scalability justify this trade-off. Future advancements in hardware and optimization algorithms are expected to mitigate these computational challenges. Additionally, Möbius operations can be approximated to reduce complexity.

## 4 Experiments

In this section, we evaluate the performance of our proposed Hyperbolic Vision Transformer (HVT) on the ImageNet-1k dataset. We compare our model with standard Vision Transformers, other state-of-the-art convolutional neural networks, and models incorporating hyperbolic geometry to demonstrate the effectiveness of integrating hyperbolic geometry into vision architectures.

### 4.1 Experimental Setup

##### Dataset

The ImageNet data set Deng et al. () is a large-scale hierarchical image database widely used to benchmark computer vision. It contains over 1.2 million training images and 50,000 validation images categorized into 1,000 classes. Each image is labeled with one of the 1,000 object categories, providing a challenging benchmark for image classification models.

##### Implementation Details

We implement our HVT model using PyTorch and the geoopt library Kochurov et al. () for operations in hyperbolic space. The architecture of HVT is based on the standard ViT-Base model Dosovitskiy et al. () with modifications to incorporate hyperbolic geometry in the attention mechanisms and positional encodings.

All models are trained on 8 NVIDIA A100 GPUs using Distributed Data Parallel (DDP). The training hyperparameters are as follows:

**Table 2: Training Parameters for the Hyperbolic Vision Transformer (HVT).**
| Parameter | Value |
| --- | --- |
| Optimizer | Riemannian Adam Bécigneul & Ganea () with an initial learning rate of $1\times 10^{-3}$. |
| Batch Size | 32. |
| Epochs | 300. |
| Learning Rate Schedule | Cosine Annealing with a warm-up. |
| Gradient Clipping | Max norm of 1.0. |
| Regularization | Geodesic regularization with coefficient $\lambda_{\text{reg}}=0.001$. |

Data augmentation techniques such as random cropping, horizontal flipping, and color jitter are applied during training. All images are resized to $224\times 224$ pixels.

### 4.2 Results and Discussion

##### Model Architecture Comparison

We compare both Base, Large, and Huge architectures to assess the scalability and architectural distinctions between the standard Vision Transformer (ViT) models and our proposed Hyperbolic Vision Transformer (HVT) variants. Table 3 summarizes each model’s vital architectural parameters and the total number of parameters.

**Table 3: Architectural Comparison of ViT and HVT Model Variants.**
| Model | Version | Layers | Attention Heads | Hidden Dim ($d$) | MLP Dim | Parameters (M) |
| --- | --- | --- | --- | --- | --- | --- |
| ViT Dosovitskiy et al. () | Base | $12$ | $12$ | $768$ | $3072$ | $86$ |
| Large | $24$ | $16$ | $1024$ | $4096$ | $307$ |  |
| Huge | $32$ | $16$ | $1280$ | $5120$ | $632$ |  |
| HVT (Ours) | Base | $12$ | $12$ | $768$ | $3072$ | $86$ |
| Large | $24$ | $16$ | $1024$ | $4096$ | $307$ |  |
| Huge | $32$ | $16$ | $1280$ | $5120$ | $632$ |  |

As depicted in Table 3, the HVT variants mirror the architectural configurations of their corresponding ViT counterparts in terms of the number of layers, attention heads, hidden dimensions, and MLP dimensions. We notice that the hyperbolic version of has the same number of parameters as its respective ViT variant.

This architectural alignment allows for a direct comparison of performance metrics across models of similar scales, highlighting the efficacy of integrating hyperbolic geometry into vision transformers without substantial increases in model complexity.

##### Performance Comparison

To evaluate the performance impact of incorporating hyperbolic geometry, we compare the Top-1 and Top-5 accuracies of both ViT and HVT variants on the ImageNet validation set. Table 4 presents these results.

**Table 4: Performance Comparison of ViT and HVT Model Variants on ImageNet.**
| Model | Version | Top-1 Acc (%) | Top-5 Acc (%) |
| --- | --- | --- | --- |
| ViT Dosovitskiy et al. () | Base | $77.9$ | $93.8$ |
| Large | $82.5$ | $95.8$ |  |
| Huge | $84.2$ | $96.7$ |  |
| HVT (Ours) | Base | $80.1$ | $95.1$ |
| Large | $85.0$ | $96.8$ |  |
| Huge | 87.4 | 97.6 |  |

From Table 4, it is evident that each HVT variant consistently outperforms its ViT counterpart across all scales. These performance gains demonstrate the effectiveness of integrating hyperbolic geometry into the Vision Transformer architecture. This enables better capture of hierarchical relationships and more nuanced representations without significantly increasing model complexity.

##### Conclusion on Architectural Comparison

The architectural comparisons and performance evaluations indicate that HVT variants leverage hyperbolic geometry to significantly improve standard ViT models across all scales. This enhancement is achieved with the same parameter size, affirming the scalability and efficiency of the HVT architecture for image classification tasks.

##### Ablation Study

To analyze the contributions of different components of HVT, we conduct an ablation study as shown in Table 5. We evaluate the impact of hyperbolic positional encoding on the model’s performance.

**Table 5: Ablation Study on the Impact of Hyperbolic Components in ViT-Base Model Variants.**
| Model Variant | Hyper Embeddings | Hyper Attention | Hyper Linear Layers | Top-1 Acc (%) | Top-5 Acc (%) |
| --- | --- | --- | --- | --- | --- |
| Baseline ViT-B (Euclidean) | – | – | – | $77.9$ | $93.8$ |
| + Hyper Embeddings | ✓ | – | – | $78.4$ | $94.1$ |
| + Hyper Attention | ✓ | ✓ | – | $79.3$ | $94.7$ |
| + Hyper Linear Layers | ✓ | ✓ | ✓ | 80.1 | 95.1 |
| + Remove Hyper Embeddings | – | ✓ | ✓ | $78.6$ | $94.2$ |
| + Remove Hyper Attention | ✓ | – | ✓ | $79.0$ | $94.5$ |
| + Remove Hyper Linear Layers | ✓ | ✓ | – | $79.5$ | $94.8$ |

##### Conclusion

Our experiments demonstrate that integrating hyperbolic geometry into transformer architectures leads to significant performance improvements on the ImageNet dataset. The HVT model benefits from the ability to model hierarchical structures and capture complex relationships in image data more effectively than Euclidean-based models.

## 5 Conclusion and Future Work

In this work, we introduced the Hyperbolic Vision Transformer (HVT), a novel architecture that integrates hyperbolic geometry into the Vision Transformer (ViT) framework. By leveraging the properties of hyperbolic space, HVT effectively models complex, hierarchical relationships within visual data, which is especially beneficial for large-scale image classification tasks like ImageNet that exhibit such structures.

Our extensive experiments on the ImageNet dataset demonstrate that incorporating hyperbolic components into the ViT framework can significantly improve performance. HVT consistently outperforms standard Vision Transformers and state-of-the-art convolutional neural networks, showcasing the strength of hyperbolic geometry in enhancing deep learning models for vision tasks.

This work serves as a foundational step in exploring hyperbolic image classification mechanisms. The success of our initial experiments suggests that hyperbolic geometry holds great promise for advancing vision architectures, opening the door to new research opportunities and innovations in this field.

### 5.1 Key Contributions

The key contributions of this paper are as follows:

- •
Hyperbolic Transformer Components: We extended the Vision Transformer framework by introducing hyperbolic analogs of critical components, such as positional embeddings, attention mechanisms, and feed-forward layers.
- •
Learnable Curvature in Positional Embeddings: Our model employs learnable curvature in hyperbolic positional embeddings, allowing it to adapt to different levels of complexity in the data.
- •
Geodesic Regularization and Stability: We introduced geodesic regularization to better separate class embeddings in hyperbolic space and employed layer scaling and gradient clipping to ensure stability during training in hyperbolic space.
- •
Empirical Performance: We showed that the Hyperbolic Vision Transformer achieves superior performance on ImageNet, demonstrating the effectiveness of hyperbolic representations for image classification.

### 5.2 Future Work

While the Hyperbolic Vision Transformer offers significant improvements in image classification, several promising research directions remain:

- •
Hybrid Architectures: Exploring hybrid architectures that combine Euclidean and hyperbolic spaces, selectively applying hyperbolic operations where they offer the most benefit while maintaining standard operations elsewhere.
- •
Optimizing Hyperbolic Training: Further refining training strategies in hyperbolic space, such as advanced optimization techniques or dynamic curvature adaptation, to enhance performance.
- •
Hyperbolic Large Language Models: The potential of a hyperbolic large language model is untapped and has potential for increased performance.
- •
Medical Imaging: Due to the inherently hierarchical structure of medical imaging data, we foresee improved performance using our proposed method on such data.

In conclusion, our work shows that hyperbolic geometry provides a powerful new perspective for modeling visual data, improving transformer architectures for image classification. The strong results on ImageNet highlight the potential of this approach, and we look forward to future developments and applications of hyperbolic representations in machine learning.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: 1. Is it possible to compare against an Euclidean ViT with learnable scaling parameter $\beta$ in residual connections and head-specific scaling factors $\alpha_h$ (lines 351-353)?
soundness: 1
strengths: Proposes a new fully hyperbolic ViT architecture, including positional embeddings, attention mechanism, and feed-forward network, which shows some promising results compared to Euclidean ViT.
summary: This paper proposes a fully-hyperbolic vision transformer (ViT) architecture. Experiments are performed on image classification with ImageNet dataset.
weaknesses: 1. The paper claims theoretical analysis of their work, but this was not included in their paper. One suggestion for such an analysis would be to show that the Euclidean transformer block can be recovered in the limit $c \to 0$, where $c$ is the curvature of the Poincare ball.
2. The experiments are not robust, with only one experiment on one task and one dataset. It would be nice to see results on other image datasets, for example CIFAR-100.
3. The paper is not properly citing past work, such as for hyperbolic linear layers (e.g. [1]), making it difficult to tell which parts of the model are new. 
4. I also think this paper would benefit from greater explanation of the motivation and intuition behind some of the methodsi n the methods section so the reader can better understand why certain choices were made. For example, why are the hyperbolic positional embeddings formulated as a Mobius scalar multiplication followed by a Mobius addition?

[1] Ganea, Octavian, Gary Bécigneul, and Thomas Hofmann. "Hyperbolic neural networks." Advances in neural information processing systems 31 (2018).

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: The models were trained on 8 A100s. Was this with a batch size of 32 on each or a total batch size of 32?
Additionally, in line 471 you mentioned that Table 5 compares the impact of hyperbolic positional encoding, is this in the table or is it not updated?
soundness: 2
strengths: The paper clearly breaks down each of the operations and components present in the model. This makes the implementation very clear and helps reproducibility. Additionally, the method achieves good results on the proposed dataset.
summary: In their paper, the authors create a hyperbolic transformer for the vision classification task. They rely on the Poincare model and create 1 to 1 translations of the Euclidean components. These can mainly be divided into 3 categories: the encoder, the attention calculation itself, and the final output layers. They perform an ablation study with the replaced components and show that a full hyperbolic model achieves the best performance. Additionally, they add a regularization term to the loss to ensure close distances between embeddings of the same class.
weaknesses: There are some points of contention in both the methodological contribution and the claims made by the authors.

Beginning with the contributions, the authors do not present any novel components. They rely entirely on previously established methods for the linear layer, activations, hyperbolic attention calculation, and normalization. 
As for the architecture, there is also no novelty. Fully hyperbolic vision models have been established before for both the Poincare and the Lorentz models [1] [2]. The papers also rely on hyperbolic normalization layers as opposed to normalization on the tangent space.

Finally, there are some questionable claims in the paper. The authors state in line 362 that the model is scalable, also in line 468, they restate this under the claim that the number of parameters is the same. However, this claim is not entirely true. Despite the similar number of parameters, hyperbolic models are extremely computationally inefficient. It would be nice to see the runtime and VRAM requirements compared to that of the Euclidean model. I would imagine this would be very prohibitive and it hinders scalability. 

There are also other claims such as in line 94, " we utilize the Poincare ball model for its computational suitability in vision tasks". This is not explained and it is not referenced and I don't quite see what is meant by it vs the Lorentz model for example. Additionally, in line 140, they claim they are better vs previous hyperbolic models such as the Hyperformer but they never actually compare against them. 

[1] van Spengler, Max, Erwin Berkhout, and Pascal Mettes. "Poincare resnet." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.
[2] Bdeir, Ahmad, Kristian Schwethelm, and Niels Landwehr. "Fully Hyperbolic Convolutional Neural Networks for Computer Vision." arXiv preprint arXiv:2303.15919 (2023).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: The authors should include more analysis and explanations to clarify why the proposed method works. Providing deeper insights into the mechanisms behind the performance improvements would enhance the paper’s impact and help readers better understand the value of the approach.
soundness: 3
strengths: It is well-executed, featuring detailed formulations and strong experimental validation. The clear presentation and effective adaptation of transformer components highlight its importance in advancing vision tasks that require complex relationship modeling.
summary: The paper presents the Hyperbolic Vision Transformer (HVT), which extends the Vision Transformer (ViT) by incorporating hyperbolic geometry to model hierarchical relationships in image data more effectively. Key contributions include hyperbolic adaptations of positional embeddings, attention mechanisms, and linear layers, along with techniques for stability such as learnable curvature and gradient clipping. HVT outperforms standard ViTs on the ImageNet dataset, demonstrating its effectiveness in image classification without added model complexity.
weaknesses: It seems to me that this work, while effectively applying hyperbolic space knowledge to a standard computer vision task, primarily contributes through engineering improvements rather than deeply explaining why these adaptations lead to better performance. While the integration is valuable, emphasizing the underlying reasons for the performance gains would strengthen the contribution and provide more insight into its impact.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: As claimed in line 119-121, the main difference between this work and previous work is the domain focus. It looks like the experiments part didn’t show much on tasks beyond classification. Have the authors explored other vision tasks?
soundness: 2
strengths: The theoretical framework of the HVT model is well-developed, with detailed mathematical formulations. The paper also shows improvements on the ImageNet dataset.
summary: The paper introduces the Hyperbolic Vision Transformer (HVT), aiming to enhance image classification in non-Euclidean space using hyperbolic geometry. By incorporating Möbius transformations and hyperbolic self-attention, the authors claim improvements in handling hierarchical data structures in images. Experiments on the ImageNet dataset show competitive performance gains over traditional Vision Transformers (ViTs) without substantial increases in model parameters.
weaknesses: My main concern is about the novelty. The paper relies heavily on established techniques and provides limited evidence of unique innovations beyond integrating hyperbolic geometry.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 1
questions: I would like to see the rebuttal dive into the novelty of the paper and especially its relation to existing literature. Please outline what is considered new and what comes from other papers (with reference). Moreover, the weaknesses outline specific experiments that I would like to see discussed.
soundness: 2
strengths: 1)
The notion of fully hyperbolic vision transformers is an important step in hyperbolic learning. Based on advances in learning with hyperbolic embeddings at the end of a network, fully hyperbolic learning can open up entirely new doors in the field.

2)
An open question in fully hyperbolic learning is scalability, as also pointed out in this work. Yet the paper manages to scale its experiments to the ImageNet-level despite operating in the Poincaré ball. Moreover, this is done with good scores. Scaling to ImageNet is a positive sign for fully hyperbolic learning.
summary: This paper investigates fully hyperbolic vision transformers. Based on advances in hyperbolic learning and fully hyperbolic convolutional approaches, this work takes the step towards transformer architectures. The work operates in the Poincaré ball and includes tricks such as additional learnable parameters in the positional embeddings and geodesic regularization. Experiments on ImageNet show that a hyperbolic ViT can outperform its Euclidean counterpart.
weaknesses: 1+2)
The biggest issue with the paper is that most of the proposed components are not new. This limitation is coupled with extremely poor citation to existing literature. Many papers have in recent years investigated fully hyperbolic (visual) learning and have proposed neural network building blocks. Yet these are not cited.

The first contribution claimed in the introduction is that the authors proposed hyperbolic neural components. While they cite Hyperbolic Neural Networks by Ganea et al. (2018) in S2, they fail to mention this work throughout the method section. More importantly, Hyperbolic Neural Networks++ [a] is ignored. This paper not only outlines how to generalize MLPs and linear layers to the Poincaré ball, but even outline a self-attention algorithm in this space. Yet this manuscript does not mention the paper.

[a] Shimizu, Ryohei, Yusuke Mukuta, and Tatsuya Harada. "Hyperbolic neural networks++." ICLR (2021).

The related work on hyperbolic geometry in machine learning lists only 5 papers altogether and ignores recent work on fully hyperbolic convolutional learning, notably [b,c]:

[b] van Spengler, Max, Erwin Berkhout, and Pascal Mettes. "Poincare resnet." In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5419-5428. 2023.
[c] Bdeir, Ahmad, and Niels Landwehr. "Optimizing Curvature Learning for Robust Hyperbolic Deep Learning in Computer Vision." arXiv preprint arXiv:2405.13979 (2024).

In [b], a fully hyperbolic batch normalization is outlined, as well as linear layers, and more in the Poincaré ball. There are many more papers ignored, as summarized in [d,e]:

[d] Peng, Wei, Tuomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. "Hyperbolic deep neural networks: A survey." IEEE Transactions on pattern analysis and machine intelligence 44, no. 12 (2021): 10023-10044.
[e] Mettes, Pascal, Mina Ghadimi Atigh, Martin Keller-Ressel, Jeffrey Gu, and Serena Yeung. "Hyperbolic deep learning in computer vision: A survey." International Journal of Computer Vision (2024): 1-25.

3)
The method outlines many small components but annoyingly never mentions which they claim as novel and which ones not. This is an issue since there is no citation in the method, except for one in the last sentence, making it impossible for the reader to understand which parts are copied from other papers and which ones are new. Below a summary of some issues:
- Hyperbolic linear layers have been proposed many times before, but not cited here.
- The ReLU and LayerNorm in Equation 10+11 are not hyperbolic, as the operations are performed in tangent space.
- The curvature in the positional embeddings (Eq 13) seems to just be a learnable parameter which has nothing to do with the curvature of the Poincaré ball.
- Self-attention in the Poincaré ball is given in [a].
- Residual connections in the Poincaré ball can be found in [b].
- Gradient clipping (Eq. 31) is common practice, see e.g. [f].
- Riemannian Adam (Eq. 32) is from Becigneul and Ganea (2019), although they are cited in Table 2.

As a result, the technical novelty of the paper is low, given that most components come from existing literature. This paper combines a number of them to create vision transformers.

[f] Dhingra, Bhuwan, Christopher J. Shallue, Mohammad Norouzi, Andrew M. Dai, and George E. Dahl. "Embedding text in hyperbolic spaces." arXiv preprint arXiv:1806.04313 (2018).

4)
The experiments are insufficient. Only 1 ImageNet result (Table 4) and 1 ablation study (Table 5) is given. Experiments on one dataset are however not enough. Moreover, multiple proposed components are never evaluated. In my view, the paper needs the following additional experiments to be more complete:
- Training and evaluation on other datasets. Can also be on smaller datasets.
- Starting from the pre-trained ImageNet model, fine-tune on smaller datasets and report test accuracy.
- Table 4 should also include runtime complexity and memory footprint. I expect that they will be much higher for HVT, but this is fine, it is good to be open about this.
- The geodesic regularization is not evaluated. What is its influence?
- Idem for layer scaling.