HyperVQ: MLR-based Vector Quantization in Hyperbolic Space
Authors: Nabarun Goswami
Yusuke Mukuta
Tatsuya Harada
Venue: Accepted by TMLR ()

Abstract: The success of models operating on tokenized data has heightened the need for effective tokenization methods, particularly in vision and auditory tasks where inputs are naturally continuous. A common solution is to employ Vector Quantization (VQ) within VQ Variational Autoencoders (VQVAEs), transforming inputs into discrete tokens by clustering embeddings in Euclidean space. However, Euclidean embeddings not only suffer from inefficient packing and limited separation—due to their polynomial volume growth—but are also prone to codebook collapse, where only a small subset of codebook vectors are effectively utilized. To address these limitations, we introduce HyperVQ, a novel approach that formulates VQ as a hyperbolic Multinomial Logistic Regression (MLR) problem, leveraging the exponential volume growth in hyperbolic space to mitigate collapse and improve cluster separability. Additionally, HyperVQ represents codebook vectors as geometric representatives of hyperbolic decision hyperplanes, encouraging disentangled and robust latent representations. Our experiments demonstrate that HyperVQ matches traditional VQ in generative and reconstruction tasks, while surpassing it in discriminative performance and yielding a more efficient and disentangled codebook.

Published in Transactions on Machine Learning Research (04/2025)
HyperVQ: MLR-based Vector Quantization in Hyperbolic Space
Nabarun Goswami The University of Tokyo
Yusuke Mukuta
makatto@mmi...ooookoooo....
The University of Tokyo, RIKEN
Tatsuya Harada The University of Tokyo, RIKEN
harada@@mitta--tkkgo..oo.o
Reviewed on OpenReview: https: // openreview net/ forum? id= NgJggULL9
Abstract
The success of models operating on tokenized data has heightened the need for effective tokenization methods, particularly in vision and auditory tasks where inputs are naturally contimuous. common solution to employ Vector Quantization (VQ) within VQ Vari- ational Autocncoders (VQVAEs), transforming inputs into discrete tokens by clustering embeddings in Euclidean space. However, Euelidean embeddings not only suffer from ineffi- cient packing and limited separation due their polynomial volume growth-bit are also prone to codebook collapse, where only small subset of codebook vectors are effectively utilized. To address these limitations, we introduce HyperVQ, novel approach that formu- lates VQ as a hyperbolic Multinomial Logistic Regression (MLR) problem, leveraging the exponential volume growth in hyperbolic space to mitigate collapse and improve cluster sep- arability. Additionally, HyperVQ represents codebook vectors as geometric representatives of hyperbolic decision hyperplanes, encouraging disentangled and robust latent representa- tions. Our experiments demonstrate that HyperVQ matches traditional VQ in generative and reconstruction tasks, while surpassing it in discrininative performance and yielding more efficient and disentangled codebook.
1 Introduction
Tokenization is fundamental component in modern data-processing pipelines, driven in part by the success of transformer-basee models. While discrete data (c.g., text) can be tokenized directly, continuous data such as images and audio require specialized approaches, typically through vector quantization. Methods like VQVAE (Van Den Oord et al. 2017)) generate discrete "tokens" from continuous inputs for generative
modeling, and they also underpin various discriminative applications (Yang et al. 2023 Baevski et al. 2020b
A persistent challenge in vector quantization is codebook collapse, where only limited subset of codebook vectors is utilized. In this scenario, few indices dominate the assignments while many remain "dead," receiving gradient updates. This underutilization reduces the representational capacity of the model, ultimately impairing both generative and discriminative performance.
Recent research has highlighted the benefits of learning represenlations in non-Euclidean spaces, particularly hyperbolic spaces. Hyperbolic geometry naturally fosters compact embeddings due to its exponential volume growth with radius (Peng et al.] 2021). Although hyperbolic spaces are often associated with hicrarchical data, their advantages extend to general-purper representations. Unlike Euclidean spaces, which exhibit polynomial volume growth, hyperbolic spaces promote well-separated clusters and reduce the risk of codebook

---

Published in Transactions on Machine Learning Research (04/2025)
collapse. By allowing embeddings to spread efficiently within bounded domain, the negative curvature inherent in hyperbolic spaces benefits tokenization even for non-hierarchical data such as images and audio. Compared to spherical manifolds (with positive curvature) or more complex non-Euclidean geometries (mixed curvature or SPD manifolds), hyperbolic spaces offer both intuitive exponential volume expansion and simpler analytical tools, making them particularly suitable for our proposed HyperVQ.
In this work, we exploit the properties of hyperbolic spaces to develop a robust and efficient tokenization method that addresses codebook collapse and enhances latent disentanglement. We reformulate the nearest neighbor search in vector quantization as multinomial logistic regression in hyperbolic space, where quan- tized vectors serve as representative points of hyperbolic decision hyperplancs. Our approach, HyperVQ, differs from Gumbel Vector Quantization (GumbelVQ) [Baevski et al. 2020a), which uses the straight- through estimator (Bengio et al. 2013) to predict codebook vector indices. The key innovation lies in our choice of latent space and the adoption of geometrically constrained codebook vectors, climinating the need for a separate codebook matrix. Leveraging the exponential volume growth of hyperbolic space, HyperVQ not only mitigates codebook collapse but also promotes implicit disentanglement of the latent space. More- over, selecting representative point the decision hyperplane the codebook vector enhances robustness to noise and outliers, making the training process less sensitive to codebook initialization. Our experimental results validate that the proposed method enhances discriminative performance while preserving generative quality.
The contributions of this work are summarized as follows:
First Realization of Hyperbolic Vector Quantization: We introduce vector quantization method for hyperbolic spaces critical building block in modern deep learning architectures.
Geometrically Constrained Codebook Vectors: Our formulation yields enhanced disentangle- ment and compact latent representations while mitigating codebook collapse.
Improved Discriminative Performance: We demonstrate that Hy maintains generative performance comparable to standard VQ approaches and achieves superior discriminative capabili-
lies.
The remainder of the paper is organized as follows: Section 2
reviews related work; Section 3 provides
necessary preliminaries; Section | details our hyperbolic vector quantization formulation; and Section 5 presents empirical evaluations.
2
Related Work
2.1 Discrete Representation Learning
Discrete representation of data is essential for various purposes, including compression and tokenization. While vector quantization |Gray] 1984) has been critical component of classical signal processing methods. it has gained significant traction in the deep learning community. First proposed in (Van Den Oord et al. 2017), the VQVAE applies vector quantization to the latent space of a VAE, transitioning from a continuous to discrete latent space. This approach offers advantages, such as learning powerful priors on the discrete space, as demonstrated by PixcICNN (Van den Oord et al. 2016).
More recently, tokenized image synthesis methods like VQGAN EEser al. 2021 VQGAN+CLIP (Crow-
son et al.) 2022). and VQ-Diffusion (Gu et al.) 2022) have shown impressive results in image generation. While discrcte representation has excelled in generative tasks, vector quantization has also been integrated into discriminative lasks. For instance, Yang et. al. (2023) proposed combining quantized representation with unquantized ones for inality-iddeeendent representation learning, and Lu et al. (2023) introduced a hicrarchical vector quantized transformer for unsupervised anomaly detection. Voctor quantization' 's appli-
cability extends beyond computer vision domains such audio (Baevski al. 2020b Zeghidour et al.
2021 Baevski et al. 2020a).
2

---

Published in Transactions on Machine Learning Research (04/2025)
While several works have explored enhancing vector quantization operations to address issues like codebook collapse (Larccuci et al. 2020: Roy et al. 2018 Takida et al.) 2022), to the best of our knowledge, no work has explored simultaneously improving the generative and discriminative capabilities of quantized representations.
2.2 Hyperbolic Deep Learning
Non-Euclidean geometry offers a promising approach to uncover inherent geometrical structures in high- dimensional data. Hyperbolic spaces, known for their exponential volume growth with respect to radius, induce low-distortion embeddings and provide better model interpretation (Peng et al., 2021). Hyperbolic embeddings exhibit. robustness to noise and outliers and demonstrate better generalization capabílities with
reduced overfitting, computational complexity, and data requirements (Nickel & Kiela 2017 Khrulkov et al. 2020).
Varions works, including (Ganea et al. 2018 Shimizu et al., 2021), ) lay the mathematical foundation for creating neural networks in hyperbolic space. Building upon these foundations, methods have been proposed address lasks such as image segmentation (Atigh el al. 2022), audio source separation (Petermann el al. 2023), image-text representation learning (Desai et al.) 2023). and variational autoencoders (Mathieu al, 2019
Skopck al. 2020).
While solid foundation of hyperbolic neural network building blocks have been laid, there still lacks principled and effective vector quantization method utilizing the hyperbolic space since it is such an important mechanism of several modern neural network architectures across domains. In this work, we explore the definition of a VQ method for the hyperbolic space which can improve feature disentanglement while maining the geometric properties of the quantized latents.
3 Preliminaries
In this section, we provide a concise overview of non-Euclidean geometry and relevant theory crucial for understanding the foundations of our work. While we touch upon the basics, for an in-depth exploration, we recommend referring to (Ganca et al. 2018). Additionally, introduce the concepts of vector quantization and vector quantized variational autoencoders, laying the groundwork for the subsequent discussions.
3.1 Hyperbolic Geometry
3.1.1 Riemannian Manifold
A manifold is an n-dimensional topological space M, such that Vz E M, the tangent space, T.M of M at 2, is Euclidean, R". A manifold paired with a group of Riemannian metric tensors, Ir T.M x T.M + R", is called a Riemannian manifold (M,g).
3.1.2 Hyperbolic Space and Poincaré Ball Model
A Riemannian manifold is called a hyperbolic space if its scctional curvature is negative and constant everywhere. There are several models to repre- sent the n-dimensional hyperbolic space with con- stant negative curvature, such as the Hyperboloid model, the Poincaré ball model, the Beltrami-KKil model, etc. In this work, we use the Poincaré ball model, which is defined by the pairing of the man- ifold T
T.P
d. (x,
pn
y exp (v)
Manifold
log (y) To Tangent Space
{xr E IR" ||भ||| < } and the metric gh = X2 where te is the radius of the ball, 5t is the Fuclidean metric on R and A,- 2(1 [|]
Figure 1: Illustratoon of the Poincaré ball model and
its associated exponential and logarithmic maps
3

---

Published in Transactions on Machine Learning Research (04/2025)
is called the conformal factor between the two met- rics. Two metries are conformal if they define the same angles.
3.1.3 Gyrovector Spaces and Poincaré Operations
Gyrovector spaces provide algebraic structure to the hyperbolic space, which is analogous to the vector space structure of the Euclidean space. Several operations on the Poincaré ball model P. are defined under this framework such as Möbius addition Ge, distance between points on the manifold de(x, v), exponential and logarithmic maps. We refer the reader to (Ganea et al., 2018) for a detailed explanation of these operations. Here we present the definitions of the exponential and logarithmic maps which are pertinent our work. The following operations are defined under the framework of gyrovector spaces and considering the Poincaré ball model p.:.
tanh VC //Azlleel
v
exp_ (b) = x tBe
(1)
2
Vellu||
- +.
log (v) Vcte
tanh
(vcll-x te Ul)
(2)
||- +
y
The exponential and logarithmic maps allow us move between the tangent space (Euclidean) and the manifold shown in Figure
3.1.4 Poincaré Hyperplanes, MLR and Unidirectional MLR
In (Ganea el al. 2018), the authors generalized the concept of a hyperplane in Euclidean space to hyperbolic space, by defining it the set of all geodesics containing an arbitrary point p e Pa and orthogonal a tangent vector E Tp (0).
Ha.p {x E IP (-p a) = 0}
(3)
However, as shown (Shimizu al. 2021). this definition causes over-parameetrriztion of the hyperplane. They instead proposed a unidirectional hyperplane formulation by fixing the choice of the point on the manifold to be parallel to the normal vector. This is done by introducing new scalar parameter r& R which allows the definition of the bias vector in terms of Fk and the normal vector dk: as,
D = expo
(4)
and correspondingly, the hyperplane can be defined in terms of q as,
u fx E IP (-qi 3 I. ax) 0}
(5)
Following this new formulation, the hyperbolic multinomial logistic regression (MLR), which is a general- ization of the Euclidean MLR formulation (from the perspective of distances to margin hyperplancs), to perform classification lasks hyperbolic space was defined. The unidirectional hyperbolic MLR formulation given K classes, ls (1, K} and for all x I defined as follows:
Uk = p(u = k x))
(6)
sign ((-gk gk O. x, ax)) ||ax|| de (z, Har
where de is the distance between a point and a Poincaré Hyperplane Shimizu et al. 2021
3.2 Vector Quantization and Limitations in Euclidean VQVAEs
Vector Quantization (VQ) is a widely used data compression and signal processing technique wherein a continuous input signal ze E N is approximated by mapping it to the nearest vector from a finite codebook C (C1, .z...., Ck}. Formally,
== q(ze) arg min min Z - Cxlls f. = =g- -Ck:
(7)
4

---

Published in Transactions on Machine Learning Research (04/2025)
where & is the index of the nearest codebook vector. VQ is central to applications such as image and speech compression, clustering, and quantization tasks, by reducing data dimensionality while preserving essential information.
Built atop this idea, the Vector Quantized Variational Autoencoder (VQVAE) introduces a discrete latent space between the encoder and decoder of standard Variational Autoencoder. The encoder outputs a continuous latent vector 2e, which is quantized into 24 via VQ, and subsequently passed to the decoder for imput reconstruction. Although this discrete mapping is non-differentiable, end-to-end training is facilitated by ttraight-thoouhh estimator that bypasses the discontimuity. The training objective typically includes commitment loss:
L= logo p( 2q) + ||sg[=]] =0112 + 3 B11 - sgl
where sgl:] is the stop-gradient operator and / weights the commitment term. Variants of VQVAE have explored alternative codcbook selection methods, such as replacing the nearest-neighhor lookup with a Gumbel Softmax sampling mechanism (Bengio et al. 2013 Baevski et al. 2020a
Despite their widespread applicability, Euclidean VQVAEs can suffer from codebook collapse, wherein only small subset of codebook vectors is effectively utilized. Concretely, let B = {ze F==1 be batch of encoded samples, and define the codebook usage distribution
(1) N
1 Σ
((C))
(a(22 =k),
(9)
where I(-) is the indicator function and q()) is the nearest-neighbor assignment. Codebook collapse arises when p(C)) R 0 for most k, meaning the majority of samples map to a small subset of codebook vectors. This tendency is exacerbated by the polynomial volume growth in Euclidean space, limiting how effectively clusters can be separated as the model's capacity or data complexity grows. Consequently, the encoder outputs can concentrate around a few dominant codebook vectors, reducing the representational capacity of the model and negatively impacting both its generative and discriminative performance.
Our work addresses these shortcomings by exploring hyperbolic formulation of vector quantization. By exploiting the exponential volume growth of hyperbolic space, we seek to better distribute latent representa- tions across codebook vectors and mitigate collapse, thereby enhancing the overall robustness and efficiency of the learned discrete representations.
4 Method
4.1 Overview
In an n--imenniinall Euclidean space F", the volume of an n-ball of radius r grows polynomially as Ve(r) x r", making it necessary to expand substantially to maintain clear separation among multiple clusters. By contrast, in n-dimensional hyperbolic space (P2), the volume of ball of radius satisfies
Vn(rr) & sinh"-I(r) xe (x-r))r for large r.
(10)
Hence, if we consider clusters each modeled an n-ball, their total volumes scale
ViotaEE x kr (Euclidean space),
(11)
VeotalHH X (n-1r) (Hyperbolic space).
(12)
Because the entire hyperbolic space P lies within unit ball of Euclidean radius 1/vc, distances near the boundary grow exponentially, allowing more efficient packing of clusters within a bounded region while preserving separation. In Euclidean space, by contrast, one must increase the radius r significantly to maintain cluster separation, resulting in less efficient packing. Additionally, the exponential volume growth also encourages balanced codebook usage. For a more detailed intuitive and theoretical discussion, refer appendix A
5

---

Published in Transactions on Machine Learning Research (04/2025)
Encoder
41
Q1
lngi(n)
Decoder
Zq
4g
20
Figure 2: Illsstatioon of VyperVQ in VQVAE setting. The key steps the projection of the euclidean embeddings to the hyperbolic space, classification of the projected embeddings to one of K classes, and utilization of the representative point the hyperplane, Qs the codebook vector which is projected back to the Euclidean space.
Motivated by this property, we formulate the Vector Quantization (VQ) operation as a multinomial logistic regression (MLR) directly in hyperbolic space. Specifically, rather than assigning latent vectors the nearest codcbook entry a Euclidean sense, we perform classification P. using hyperbolic MLR. To further promote disentanglement,
propose using the representative points of hyperbolic decision hyperplanes codebook vectors. Unlike arbitrarily learned embedding matrix, these representative points capture the geometric structure more faithfully and encourage distinct separations in the latent space.
Figure 2 iilustrates the overall pipeline of an encoder-decoder framework augmented with our Hyper The encoder and decoder remain in Euclidean space, while the codebook lookup (i.c., quantization) occurs in Pn. Algorithm [ provides a task-agnostic training procedure, and Section 4.2 details the hyperbolic MLR formulation and the geometric constraints that define our codebook vectors. Different tasks may employ diverse encoder-decoder architectures, but the quantization mechanism in hyperbolic space stays the same, exploiting the exponential volume growth to achieve compact and efficient clustering of embeddings.
In the following subsections, we provide a detailed description of HyperVQ.
4.2 HyperVQ
4.2.1
Hyperbolic MLR for Vector Quantization
Given an encoder output 2e Ted we first obtain its hyperbolic embedding 2n exp6(=a), where exp6(.) is the exponential map at the origin Equation ( We then trcat the codebook selection problem as a classification ( (Baevski el al. 2020a) in IP via hyperbolic multinomial logistic regression (MLR). Let la (zx) be the logit for codebook index & (1,2.... .., K}. To assign 24 to one of the K codebook vectors, apply Gumbel Softmax over these logits:
exp(((a(Zz) + gw)/T)
(a(#)) Σ (L, + 95)/T)*
(13)
where (g1.....kx' 9 are ii.d. samples from the Gumbel distribution (Jang al. 2017) and 7 is a temperature parameter. The codebook index chosen by
k* arg max (x()))
(14)
To enable end-to-end training, we use the straight throuuh estimator (Bengio et al, 2013 when backprop- agating through the discrete choice. This ensures gradients pass from the decoder's reconstruction loss (or downstream objectives) back to the hyperbolic logits (x(-). We rely on the unidirectional MLR formulation (discussed in Equation 6 for calculating these logits in a parameter-effiieet manner.

---

Published in Transactions on Machine Learning Research (04/2025)
4.2.2 Geometrically Constrained Codebook Vectors
The selection of codebook vectors a critical factor in vector quantization. Traditional methods include choosing cluster centroids via online K-Means (as in standard VQVAE) or learning an embedding matrix (as in Gumbel Vector Quantization). While K-Means centroids reflect intrinsic geometry in Euclidean space, direct extensions to hyperbolic space require iteratively computing central mcasures (Lou et al. 2020), which is computationally expensive and numerically sensitive near the Poincaré ball boundary. On the other hand, embedding matrices optimized purely by reconstruction objective may overfit to imput patterns, reducing diversity among codebook entries and weakening disentanglement.
To overcome these limitations, we assign each codebook entry to be the representative point hyperbolic decision hyperplane, denoted by gl. In particular, 9 encodes both the position and orientation of the k-th hyperplane in Pe (see Section 3.1.4 for its unidirectional definition). Unlike centroid-based approach that averages embeddings, Qk remains stable under outliers and preserves local geometric structure. More- over, the decision hyperplane formulation naturally enforces distinct regions in the latent space, improving disentanglement across different codebook entries.
Since the decoder typically operates in Euclidean space, we apply the logarithmic map logō to project q from Pa back to Rd Referring equation for Ou = expb(r> (ax]): we have
2g logó (g)) logo cxp6 (rx(ax])) rk [ax].
(15)
where Tk is a scalar and ak E R" represents the normal vector in the tangent space. In essence, each hyporbolic codebook vector in Euclidean space becomes the product rx(ox). This mapping incorporates the hyperbolic geometry (via тı and ax) without requiring computationally heavy centroid calculations.
An important consequence of using qu is that the decoder is forced to reconstruct from geometrically mean- ingful and relatively disentangled features. As a result, each codebook entry captures distinct direction and radius in the langent. space, and the exponential growth property of IPn helps mainlain clear separations between codebook vectors. This structure not only mitigates codebook collapse but also enhances discrimi- native performance, since the latent space is partitioned by well-defined hyperplanes rather than overlapping or redundant embedding vectors.
Algorithm HyperVQ Training
1: Initialization: Initial network parameters of Encoder (0g), Decoder (0p), and Quantizer (fq = {ak; rkk for k = 0,1, K and temperature, T E [Tmax; Tmin], and decay factor 5
2: Training:
3: for each iteration, j do
sample data, x
2e Encoder(x:)
2A expε(ze)
logits = unidirectional mlr(z)) (Equation (6)
8:
ko = argnax((uubbe_ssttaaa(ooog,,,
9.
= Fk [ax] (Equation (15
10:
T = max (Tmax 87, Tmin)
11:
y = Decoder(z,))
12:
Compute objective function:
13:
Update fg 0g, 0q by minimizing L with gradient descent
14: end for
5
Experiments
To assess the offectiveness of our proposed method, we conducted experiments across diverse tasks, encom- passing image reconstruction, generative modeling, image classification, and feature disentanglement. The implementation details and results of each experiment. elaborated upon in the following subsections.

---

Published in Transactions on Machine Learning Research (04/2025)
5.1 Implementation Details
Our method was implemented using the PyTorch library Passkk et al. 2019 Model training con- ducted on 4 A100 GPUs utilizing the Adam optimizer (Kingma & Ba 2014), with learning rate set at 3e-4 and a batch size of 128 per GPU, unless otherwise specified. For hyperbolic functions, we employed the geoopt library (Kochurov et al. 2020 and referenced the official implementation of hyperbolic neural + (Shimizu et al. 2021
5.2 Reconstruction and Generative Modeling
To assess the generative modeling capabilities of the proposed HyperVQ method, we conducted experiments on the Cifar100 (Krizhevsky] 2009) and ImageNet (Russakovsky et al. 2015 datasets. We trained VQVAEs with the original formulation, referred to as KmeansVQ, and our proposed hyperbolic formulation, termed as HyperVQ, for varying numbers of codebook vectors K. The encoder comprises an initial convolutional encoder block, followed by a stack of residual blocks, before being passed through the quantization layer. The decoder is the inverse of the encoder, with the residual blocks and the convolutional encoder block replaced by transposed convolutional decoder block for upsampling. The reconstruction results presented in Table [ By comparing the reconstruction mean squared error (MSE) of the two methods, we observed that our proposed method performs slightly better or on par with the original formulation.
Table 1: Comparison of reconstruction mean squared error (MSE) for KmeansVQ and HyperVQ Cifar100 (test) and ImageNet (validation) datasets for different codebook sizes, K.
Cifar100 (32x32) J
ImageNet (128x 128) J
K
512
256
128
512
256
128
KmeansVQ
.264+.003
.644+.003
.250=:.003
.175:.000
.199:.001
.225+.001
HyperVC VQ
.216::002
.241#.003
.2561.003
.175:.001
.202::.001
.217:.001
Following this, we trained a generative 15-layer GatedPixelCNN (Van den Oord et al. 2016 model using the quantized representations obtained from VQVAE models trained on the ImageNet dataset with K 512 codebook vectors for 50 epochs.
Table 2: Generative modeling performance of CNNedPixelCNN with VQVAE quantized representations on ImageNet, comparing VyperVQ and the original formulation terms of Fréchct Inception Distance (FID) and Inception Score (IS)
FID 4
IS
KmeansVQ
143.78
5.50 + 0.07
VQ
130.31
5.77
0.05
As shown in Table 2 HyperVQ demonstrates comparative generative modeling performance compared to the original formulation, measured by the Fréchet Inception Distance (FTD) (Heusel al.) 2017) and Ineop- tion Score (IS) (Salimans et al., 2016). Additionally, Figure 3a displays some reconstructions achieved with HyperVQ the ImageNet validation set (128x128), while Figure 33b presents samples generated by the GatedPixelCNN model. We reiterate here, that he goal of this experiment is to show the effectiveness of HyperVQ and not to achieve stato-of-the-art.
5.3 Image Classification
To substantiate our claim that HyperVQ learns a more disentangled representation, thereby enhancing discrininative performance, conducted experiments the Cifar100 dataset. In these experiments,

---

Published in Transactions on Machine Learning Research (04/2025)
(a) HyperVQ reconstruction
(b) Samples from GatedPixeLCNN trained on top of HyperVQ encodings
Figure 3: Image reconstruction and generative modeling results on ImageNet.
initially trained VQVAEs with various quantization methods and a varying number of codebook vectors, K. Subsequently, we employed the pre-trained VQVAF as a feature extractor and trained a lightweight classifier atop it, consisting and two fully connected layers to obtain the logits. Only the classifier block is trained using the cross-entropy loss, while keeping the parameters of the convolutional encoder, resicuall blocks, and the quantization layer fixed. The goal of this experiment is to show the effectiveness of HyperVQ emboddings and not to achieve state-of-the-arl accuracy.
convolutional layer with ReLU activation, followed by a global average pooling layer
For this experiment, in addition to Kmeans VQ and HyperVQ, we also included GumbelVQ for comparison. All models underwent training for 500 epochs, utilizing the same scttings as described Section 5.1
Table 3: Evaluation of discriminative performance using pre-trained encoder and quantizer a VQVAE as feature extractors on the Cifar test set. Additionally, codebook usage as measured by average perplexity is reported in the last column.
K
512
256
128
64
Perplexity (K 512)
Kmeans
30.04++0.00
29.54++0.88
30.23:40.90
29.58++0.88
58.09
GumbelVQ
25.99-+0.86
26.93++0.87
26.94+0.87
27.4570..8
169.59
HyperVQ
31.59:10.91
31.06:10.91
30.63:10.90
30.14:10.90
220.22
The results presented in Table 3 indicate that Hy consistently outperforms other methods in terms of classification accuracy, irrespective of the number of codebook vectors.
To gain insights into the reasons behind this improved performance, we visualize the codebook vectors 4 and compare the codebook usage for reconstruction based on the Pk log log where Pk is the empirical probability of selecting the k-th codebook entry, the test set shown in Table g Perplexity quantifies the effective number of utilized codes, with higher values indicating more uniform usage. Additionally, we treat the codebook vector as a latent representation with x spatial dimensions and decode it using the corresponding pretrained VQVAE decoder for visualization purposes.
learned by different methods in Figure perplexity, defined as = exp ΣK
K
Insights drawn from the codebook visualization and perplexitics yield several interesting observations. Firstly, it is evident that Kmeans VQ exhibits low codebook usage, as illustrated in Figure 4a where substantial portion of the codebook vectors appears to be invalid and unused. While GumbelVQ demonstrates higher
9

---

Published in Transactions on Machine Learning Research (04/2025)
(a) Vm
(b) GumbelVQ
(c) HyperVQ
Figure 4: Visualization of the codebook vectors for VQVAE trained on Cifar100 with different quantization methods for K 512.
codebook usage compared to KmeansVQ and learns valid codebook vectors, the vectors themselves lack strong discriminative qualities. This limitation stemm from their optimization for reconstruction, leading to redundancy in the codebook visualizations (Figure 4bJ, ultimately resulting in lower classification accuracy compared to Kmenns
Contrastingly, the codebook vectors learned by Hy exhibit both the highest perplexity (Table 3 and less redundancy (Figure 4el. This characteristic enhances the discriminative ability of the quantized embeddings, translating into the highest classification accuracy among the methods considered.
5.3.1 Disentanglement and Cluster Quality
To assess whether HyperVQ is capable of learning a more disentangled representation, we trained Hyper- VQVAE model on the simple MNIST dataset. We maintained the latent dimension as 3, with 16 codcbook vectors; for easy visualization without any post-processing. For comparison, we also trained a Kmeans VQ- VAE model with the same configuration. To facilitate visualization, we sampled 1000 embeddings and plotted them against 3 codebook vectors each for Hyper VQ and KmeansVQ.
(a) Hyper VQ (xe)
(b) HyperVQ (2h, and 2q)
(c) Kmeans VQ (ze)
Figure 5: Vizualization and comparison the embeddings learnt by the Kmeans VQVAB and HyperVQVAL
models on MNIST dataset. Figure 5a shows the pre-pojection Euclidean embeddings learnt by the Hyper and Figure 5b shows the projected embeddings along with the decision hyperplanes and quantized vectors. Figure 5c shows the embeddings learnt by the Kmeans VQ model,
As depicted in Figure 5 the clusters of the HyperVQVAR are more compact and disentangled than those of the VQVAE. This phenomenon arises because the hyperbolic MLR formulation encourages embeddings to be highly localized within the regions enclosed by the decision hyperplanes, thus inducing implicit disentan- glement of the latent space.
Apart from the visualization, we also performed cluster quality assessment by computing the Silhoutte (Rousseeuw) 1987) and Davies- Davies-Bouldin Index (Davies & Bouldin 1979). To test the robustness,
10

---

Published in Transactions on Machine Learning Research (04/2025)
Table 4: Cluster quality and robustness assessment under standard and noisy conditions, comparing Silhou- ette score and Davies-Bouldin Index for KmeansVQ and HyperVQ.
Standard Conditions
Noisy Conditions
Silhouette Scoref
DB Index.]
Silhouette Scoret
DB Index]
KmeansVC VQ HyperVQ
0.487 0.565
0.732 0.553
0.310 0.564
0.915
0.560
applied random noise in the form of random rotations, flip, and gaussian noise and computed the above scores. From Table | we can see that HyperVQ more robust towards noisy inputs and maintains its cluster compactness even under noisy conditions and in general performs better than KmeansVQ.
5.4 Quality Independent Representation Learning
We also applied HyperVQ the VQ-SA model (Yang et al. 2023 This model introduces a VQ layer and self-attention layer, combining quantized and unquantized features before the final classifier of a pretrained backbone network. This augmentation aims to enhance robustness to corruptions by learning innltt--iddeeeddent features. The VQ-SA model undergoes training the ImageNet dataset, incorporating corruptions from the ImageNet-C dataset (Hendryeks & Dictterich/ 2019) for data augmentation. In the original work, they employed K = 10000 and trained for 300 epochs. In our case, to conserve computational resources, we used = 1024 and trained for 50 epochs. We also trained the original VQ-SA model under the same settings for fair comparison. Classification accuracy is reported on the clean validation set ImageNet,
Table 5: Classification accuracy on ImageNet and ImageNet-C for the VQ-SA method with different quan- tization methods.
Clean t
Known f
Unknown
mCE 4
ResNet50
75.704
37.39
48.94
76.43
VQ-SA
70.178
58.64
53.55
52.65
HypSrVQ-SA
74.344
62.30
56.09
47.61
along with accuracy on known and unknown corruptions in the ImageNet-Ci dataset. Additionally, we report the mcan corruption error (mCE) (Hendrycks Dictterich, (019). From the Table we can see that the hyperVQ-SA method significantly outperforms the original VQ-SA.
5.5 Application to Speech Self- Supervied Learning
To explore whether HyperVQ provides performance gains beyond the domain of computer vision, we applied it to the way2vec-2.0 (Baevski et al. 2020b) pre-training framework. In wav2vec-2.0 pre-training, intermedi- ate convolutional features quantized, and the transformer backbone predicts the quantization IDs from masked continuous convolutional features. The model is optimized using contrastive loss. In our experi- ments, we replaced the standard vector quantization module in wav2vec-2.0 with HyperVQ and conducted pre-training on the LibriSpeech 960h dataset. As shown in Figure 0 HyperVQ achieves consistently higher codebook usage, which is reflected in the higher perplexity and resulting a lower validation contrastive loss, thereby demonstrating the generalizability and efficacy of our proposed method. These results indicate that HyperVQ's structured representation contributes to more efficient learning dynamics, reinforcing its applicability in diverse domains beyond vision.
11

---

Published in Transactions on Machine Learning Research (04/2025)
600
4.0
3.5
400
wav2vec2 Codebook Usage
wav2vec2 hyperVO Codebook Usege
Loos
wav2vec2 Validation Contrastive Loss
3.0
300
wav2vec2 wav2vec2-hyperVV hyperVO Validation Contrastive Loss
2.5
100
2000
4000
6000
8000
10000
1200C
14000
Step x100
Figure 6: Application of hyperVQ to wav2vee 2.0 pre-training
5.6 Ablation Study
We conducted ablation study to understand the individual and combined impacts of the hyperbolic Multinomial Logistic Regression (MLR) formulation and the use of the representative point on the decision hyperplane as the codebook vector in our proposed method.
Our evaluation compared the performance of our HyperVQ method against traditional vector quan- tization approaches such Kmeans VQ and Gum- belVQ, as well study: HyperKmeansVO. which uses codebook vec- tors in hyperbolic space learned with the KmeansVQ mechanism and distances induced by the Poincaré ball model; and HlpeeEEbb.attV which applics the hyperbolic MLR formulation but uses the em- bedding matrix as the codebook vectors, similar GumbelVQ but with logits derived from the hyper- bolic MLR.
Table 6: Comparison of Classification Accuracy CIFAR-100 for various vector quantization methods with codebook size of 512.
variants designed for this
Accuracy (%)
Kmeans VQ HyperKmeansVC
30.04
22.42++0.86 ..2
GumbelVQ
25.99
HyperEmbMat VQ
28.28-+0.88
HQ
31.59+10.91
The findings from our study, as depicted in Ta- ble Table [6] highlight the benefits incorporating the hyperbolic MLR formulation with the embedding matrix, which notably improves classification accuracy over the GumbelVQ model. Additionally, leveraging the representative point on the decision hyperplane as the codebook vector further enhances classification performance, establishing our HyperVQ method as the superior approach among those tested.
6 Limitations and Conclusion
We proposed an altornative formulation for vector quantization utilizing hyperbolic space, demonstrating its effectiveness in enhancing the discriminative power of quantized embeddings while preserving the generative capabilities of the VQVAE. The hyperbolic MLR formulation representation in the pre-quantized Euclidean latent space. Through various experiments, we illustrated that Vy can serve as a drop-in replacement for multiple methods across domains, improving their performance.
shown to encourage more compact
However, it important note that the projections and from the hyperbolic space exhibit sensitivity numerical instability, particularly near the boundary of the Poincaré ball which can be circunvented mild radius cap (Ganea et al. 2018). However, this sensitivity still raises concerns, especially when
12

---

Published in Transactions on Machine Learning Research (04/2025)
employing low-precision methods, such 16-bit Doating-point numbers, during model training, as it may lead to overflow and instability. Consequently, this limitation could restrict the application of HyperVQ in very large models, which are typically trained using low or mixed precision. Further analysis of the stability of HyperVQ under low-prccision training is left for future work.
Broader Impact Statement
Our work advances the methodology of vector quantization by leveraging hyperbolic space, a novel approach that can improve codebook utilization and representation disentanglement in generative models. Although primarily theoretical in its contribution, this method has practical implications for improving performance in diverse domains, including image and speech processing. However, we acknowledge that as with any advancement in generative modeling, there exists a potential risk for misusc. Improved generative models may inad veadeeleetlly facilitate the creation of synthetic content that could be exploited for disinformation deepfakes, or other unethical applications. We encourage the research community to adopt robust ethical guidelines and deploy additional safeguards when utilizing such technologics. On the positive side, our approach may enhance model efficiency and robustness, contributing to fields where reliable data representation is critical. We believe these considerations provide balanced view of the potential societal impacts of our work.
Acknowledgement
This work was partially supported by JST Moonshot R&D Grant Number JPMJPS2011, CREST Grant Number JPMJCR2015 and Basic Research Grant (Super AI) of Institute for AI and Beyond of the University of Tokyo.
References
Mina Ghadimi Atigh, Julian Schoep, Erman Acar, Nanne Van Noord, and Pascal Mettes. Hyperbolic image segmentation. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4453-4462, 2022.
Alexei Baevski, Steffen Schneider, and Michael Auli. vq-wav2vec: Self-supervised learning of discrete speech representations. In International Conference on Learning Representations, 2020a. URL https:
Alexci Bacvski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli, wav2vec 2.0: framework for self-supervieed learning of speech representations. Advances in neural information processing systems, 33: 12449 12460, 2020b.
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013.
Herman Chernoff. measure of asymptotic efficiency for tests of a hypothesis based on the sum of observa- tions. Ann. Math. Statist., 23(4)44 507, 1952. URL h mttpa/tttttttttttt/////7723000
Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff. Vqgan-clip: Open domain image generation and editing with natural language guidance. In European Conference Computer Vision, pp. 88-105. Springer, 2022.
David L. Davies and Donald W. Bouldin. cluster separation mcasure. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAM 227, 1979. doi: 10.1 1099/PAAM.1197007660000
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakrishna Vedan- tam. Hyperbolic image-text representations. In International Conference on Machine Learning, pp. 7694- 7731. PMLR, 2023.
13

---

Published in Transactions on Machine Learning Research (04/2025)
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resohttion image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12873- 12883, 2021.
Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018.
Gray. Vector quantization. IEEE ASSP Magazine, 1(2):4-20 29, 1984. doi: N0.1109/AASSP.198..
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceodings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10696 10706, 2022.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. Proceedings of the International Conference on Learning Representations, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Fric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. Tn International Conference Learning Representations, 2017. URL httpp:://openneeiii..neeee/////e id rdrreee8
Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky, Hyper- bolic image embeddings. In Proceedings of the IEEE/ /CVF Conference on Computer Vision and Pattern Recognition, pp. 1418-6428, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.60N% 2014.
Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
Adrian Lańcucki, Jan Chorowski, Guillaume Sanchez, Ricard Marxer, Nanxin Chen, Hans JGA Dolfing, Sameer Khurana, Tanel Alumãe, and Antoine Laurent. Robust training of vector quantized bottleneck models. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-7. IEEE, 2020.
Aaron Lou, Isay Katsman, Qingxuan Jiang, Serge Belongie, Ser-Nam Lim, and Christopher De Sa. Differ- entiating through the fréchet mean. In International Conference Machine Learning, pp. 6393-6443. PMLRR 2020.
Ruiying Lu, YuJie Wu, Long Tian, Dongsheng Wang, Bo Chen, Xiyang Liu, and Ruimin Hu. Hierarchical vector quantized transformer for multi-class unsupervised anomaly detection. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL http:::t////ooerreiiee httt:e///ppaaeeeeeeeeeee/rrrotdp
Emile Mathieu, Charline Le Lan, Chris J Maddison, Ryota Tomioka, and Yee Whye Teh. Continuous hierar- chical representations with poincaré variational auto-encoders. Advances in neural information processing systems, 32, 2019.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems, 30, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems
14

---

Published in Transactions on Machine Learning Research (04/2025)
32. pp. 8024-8035. Curran Associates, Inc., 2019.
URL cttp:///ppers.nnuuziiee www..ie..iippp.. /cpppppp deep-learning
Wei Peng, Thomas Varanka, Abdelrahman Mostafa, Henglin Shi, and Guoying Zhao. Hyperbolic deep neural networks: A survey, IEEE Transactions on pattern analysis and machine intelligence, 44(12))10023 10044, 2021.
Darius Petermann, Gordon Wichern, Aswin Subramanian, and Jonathan Le Roux. Hyperbolic audio source separation. In ICASSP 2023-2023 TEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Peter J. Roussceuw. Silhoucttes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53 65, 1987. ISSN 0377-0427. doi: https://
doi.org/ (0..org110.1011 10161.0...111111111111111111 (03777-442778))991255 0377042787901257/
Aurko Roy, Ashish Vaswani, Arvind Neelakantan, and Niki Parmar. Theory and experiments on vector quantized autocncoders. CoRR, abs/1805.11063, 2018.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei-Fei. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (LJCV), 115(3):211 252, 2015.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
Ryohei Shimizu, YUSUKE Mukuta, and Tatsuya Harada. Hyperbolic neural networks In International Conference on Learning Representations, 2021. URL http:://ppapeeeeeeeeeeeeerrraaaa
Ondrej Skopek, Octavian-Fugen Ganea, and Gary Bécigneul. Mixed-curvature variational autoencoders. Tn Sth International Conference Learning Reprosentations (ICLR (vittal.. International Conference on Learning Representations, 2020.
Yuhla Takida, Takashi Shibuya, WeiHsiing Liao, Chieh-Hsii Lai, Junki Ohmura, Toshimitsu Uesaka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji. Sq-vae: Variational bayes on discrete representation with self-annealed stochastic quantization. arXiv preprint arXiv:2205.07547, 2022.
Aaron Van den Oord, Nal Kalehbrenner, Lasse Espcholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelcnn decoders. Advances in neural information processing systems, 29, 2016.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Zhou Yang, Weisheng Dong, Xin Li, Mengluan Huang, Yulin Sun, and Guangming Shi. Vector quantization with self-attention for quality-indeppndent representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 24438-24448, 2023.
Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec. IEEE/AC ACM Transact ions on Audio, Speech, and Language Processing, 30: 495-507, 2021.
A Intuitive and Theoretical Justification for Mitigating Codebook Collapse via Hyperbolic Geometry
Hyperbolic geometry mitigates codebook collapse by leveraging its exponential volume growth. Below, we unify an intuitive explanation with a theoretical argument, showing how negative curvature promotes more uniform cluster usage than Fuclidean space.
15

---

Published in Transactions on Machine Learning Research (04/2025)
A.1 Exponential vs. Polynomial Volume Growth
Let Ω C pe be an n-dimensional hyperbolic region (e.g., Poincaré ball) with radius R. Up to lower-order factors, the volume of an n-ball in hyperbolic space, By (R), grows as
(ol(Bu = an en-e)_
where an depends on the dimension. By contrast, in Euclidean space the volume of an n-ball of radius R. (g(R) is
Vol(Be(R) = R",
which grows polynomially in R. Consequently, for large radii, hyperbolic volumes dominate exponentially, setting the stage for more "room" at the boundaries.
A.2 Codeword Partitions and Volume Fractions
Now let K codewords partition Q into regions {Ri,..., Rkh, defined by hyperbolic decision hyperplanes. Each codeword & covers portion of Ω with volume
Vol(Rx);
its volume fraction is
Vol(R2)
Pk
Vol(22)
Even a moderate extent for R& implies non-negligible Pt- In Euclidean space, by contrast, polynomial growth yields smaller fractions for similarly sized partitions, especially at higher dimension or larger radius.
A.3 Chernoff Bounds and Underutilization Probability
Suppose we collect N samples (2(")...,, (1) in Ω and let Xş be the number assigned to codeword Under idealized model where data drawn i.i.d. from 2, B[Xx] N pk- Chernoff bounds Chernoff 1952h imply that for 0 << 8 1,
......2, (N))
Pr Pr[x, s
pk] s exp
8a P
Hence, larger Pk& (driven by exponential volume expansion) makes N p bigger and the probability of severe underutilization exponentially smaller. This effect dircctly combats "codeword collapse" (dead nearly unused codewords).
A.4 Why Hyperbolic Partitions Tend to be More Uniform
Boundary Sensitivity and Partition Variance. In hyperbolic space, the volume of a region grows so fast with radius that small shifts in a cluster boundary can produce large changes absolute volume. However, negative curvature also "pushes" these boundaries outward a way that resists any single cluster from monopolizing the space, bocause other clusters can still expand at the perimeter. By contrast, Euclidean geomel.ry, boundary shifts can easily let one cluster gain (or lose) substantial fraction without the same "pushback." Consequently, the variance in partition sizes (relative to the mean size) tends to be lower in hyperbolic space, favoring more uniform distribution among clusters.
Analogy: The "Expanding Pie". Imagine ② as a pie shared among K people (codewords):
Euclidean Pie (Polynomial Growth): If one person grabs a large central slice, there isn't much more pie near the edge to compensate. Others end up with smaller slices.
Hyperbolic Pie (Exponential Growth): Even if someone starts in the center, the pie "expands" quickly outward that there still ample portion for others. Negative curvature effectively keeps adding new "pie" at the boundary, preventing single cluster from hoarding too much volume.
Hence, hyperbolic geometry inherently biases the system toward more balanced clusters.
16

---

Published in Transactions on Machine Learning Research (04/2025)
A.5 Encoder-Generann Data and Practical Considerations
In real scenarios, latent embeddings come from encoder (a neural network) that may deviate from i.i.d. or uniform sampling. Nevertheless, hyperbolic geometry's exponential expansion still amplifies outward spread of the data. This property inherently stabilizes codeword usage, making collapse or extreme imbalance less likely than in polynomial-growe growth Euclidean counterparts.
Summary. Combining the exponential vs. polynomial volume contrast with Chernoff bounds provides a high-level theoretical explanation for why hyperbolic geomelry naturally promoles more uniform codeword utilization:
Exponential volume expansion larger volume fractions рs-
Higher pk lower probability of underutilizatio by Chernoff concentration.
Negative curvature and boundary sensitivity + partitions remain closer to uniform even under moderate shifts, reducing variance in cluster sizes.
Thus, hyperbolic geometry yields an effective "self-balancing" mechanism, mitigating both codebook collapse compared to Euclidean settings.
A.6 Numerical Experiment
dact 15-25, 10-00000 8x c+5
612
000
0.4
1.01
1.05
(a) Euclidean
(b) Hyperbolic (uniform)
(c) Hyperbolic (boundary)
Figure 7: Visualization of the codebook assignment. ratios for toy numerical experiment.
To illustrate how hyperbolic geometry promotes more balanced codeword usage relative Euclidean geom- etry, we conduct a toy experiment under controlled conditions. The key difference lies in how place the codewords or hyperplanes before adding noise to randomize their positions.
Setup and Codeword Initialization.
Euclidean Codewords (Centers): We define a grid in the region [-c, c] and pick K grid points to serve as codewords. This ensures a roughly central arrangement. To avoid perfect symmetry, we add small isotropic Gaussian noise to each coordinate. Hence, each codeword "close to" its grid location but not placed exactly on lattice point.
Hyperbolic Codewords (Unidircctional Hyporplancs): We work in the Poincaré ball model. We place K hyperplanes around the center at evenly spaced angles (e.g., 0g = 2mk/K), then intro- duce small random offsets each hyperplane's angle, bias, and scaling parameters.
Data Sampling and Assignment.
We sample N points:
Uniformly in box [-R,R] We assign each point the codeword that is nearest in Euclidean distance.
Euclidean:
17

---

Published in Transactions on Machine Learning Research (04/2025)
2. Hyperbolic: Uniformly in the unit Poincaré ball, and concentrating near the boundary (e.g. |||||| E 1)). We assign each point to the hyperplane/codewwor whose hyperbolic MLR logit is largest.
We measure the assignment ratio for each codeword (i.c. the fraction of data points assigned to it) and plot in bar charts.
Observations Figures ra 7c compare the resulting codeword usage. In the Euclidean setting, a few centers often capture disproportionately large fractions of points. By contrast, the hyperbolic setting do not allow any single hyperplane to dominate pariittionn stay relatively balanced. As discussed in Section A.3 the exponential volume growth boosts each codeword's volume /ssignmmnt ratio, sharply reducing the probability of codebook collapse, corroborating our thcoretical justification.
18


Reviews:
[{"review_id": "vePzru4UNE", "reviewer": "Reviewer_6v75", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThis paper continues the trend of adapting Euclidean models to hyperbolic geometry. The proposed method seems to learn good representations. \n\n**Weakness**\n\n1. Mathematical concern. \n\nSection 3.1.1. - Local neighborhoods can be approximated by Euclidean space. The tangent space is Euclidean. \n\nSection 3.1.2. - $c$ is not the radius. The radius is $\\frac{1}{\\sqrt{c}}$ based on the equation. Hence there is a mismatch. The $c$ should also feature in the Reimannian metric. Specifically, $\\lambda_x = \\frac{2c}{1 - c \\|x\\|^2}$. This differ from Ganea et al 2018 by a factor of c. Based off of this the exp and log need to checked. \n\nSection 3.1.4 - What is $z_h$ in Eqaution (6)? How are the logits $\\ell_j$ calculated? It says that equation (6) is used, but that equation is not clear. For the logits, should the $x$ also be a $z_h$?\n\nWhat are \"we propose using the representative points of hyperbolic decision hyperplanes\" (bottom of page 5). Again in section 4.2.2. this is referenced as the $q_k$ from the hyperbolic regression problem. However, how are there $q_k$ choosen?\n\n2. Experimental Questions. \n\nDoes Table 1 represent training data reconstrution MSE or test data reconstruction MSE? What are the errorbars or the uncertaininity in these numbers?\n\nThe generative model, (and the decoder) are Euclidean models, do we just treat the hyperbolic embeddings as Euclidean vectors?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a new method for quantizing continuous data using hyperbolic gemetry. In particular, the paper suggests, training an encoder, then doing classification in hyperbolic space to learn the codebook representation. The autoencoder is trained using reconstruction error. \n\nThe idea of using hyperbolic space is interesting, the representations learned to appear to be more diverse and contain more information. \n\nOverall, I am for eventual acceptance of the paper, however, the current version has a variety of mathematical issues and needs a lot more experimental details.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like the authors to answer the questions in the weakness section. In general, I would like the paper to contain more experimental details and be more precise regarding the mathematical definitions and the model.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "l5T79KvGjJ", "reviewer": "Reviewer_gHht", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The overall idea of using hyperbolic space for more efficient tokenization in the context of vector quantization is interesting and has merit; to the best of my knowledge, this is the first time hyperbolic space has been used in this way.\n\n2. The perplexity/codebook utilization aspects of the model seem to be quite promising; in particular, I like the visualizations of the codebook vectors in Figure 4 (where it clearly seems the model improves over at the very least KmeansVQ).\n\n## Weaknesses\n\n1. Results-wise, I do not believe a very convincing case has been made for the utility of the model, i.e. there does not seem to be a particular \"essential\" application whose needs are served by this model design and by many downstream metrics HyperVQ does not seem to give a considerable improvement over KmeansVQ (e.g. see Table 1 for 256 and 128 codebook vectors) by mean difference. Perhaps most crucially, for almost all results, standard deviations are not given! This make it very hard to assess statistical significance of the results in e.g. Table 3.\n\n2. Although this idea is interesting at a high-level, I think it is currently under-explored. I would have liked to see a more thorough investigation and some variation on the implementation of the VQ approach that perhaps goes a little outside of the \"VAE\" setting the paper builds on directly. For example, instead of using hyperbolic MLR for quantization and the VQVAE loss, how would a more straightforward quantization approach work, where we perform quantization by selecting the closes codebook vector by hyperbolic distance? This seems like it would make for a simple and interesting baseline that may help better justify some of the design decisions currently being made.\n\n## Verdict\n\nThis paper presents a vector quantization approach using hyperbolic space in the VQVAE framework. To the best of my knowledge, this is a reasonably novel application of hyperbolic space and seeks to improve the issues of codebook collapse and enhances disentanglement. I believe the claims made in the paper are somewhat well-supported by the empirical results as of right now, but will be better supported once standard deviations are provided for most numerical results (see requested changes below). I also believe this paper is of interest to at least some portion of TMLR's audience, specifically those who work on either vector quantization or hyperbolic machine learning. As a consequence, I recommend an accept rating for this paper pending the completion of the requested changes below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors harness the properties of hyperbolic space to develop an efficient tokenization method that improves the issue of codebook collapse and enhances disentanglement. The authors propose formulating the nearest neighbor search problem in vector quantization (VQ) as a multinomial logistic regression in hyperbolic space, with quantized vectors service as representative points of hyperbolic decision hyperplanes. They call this approach \"HyperVQ\" and note that the key distinctions of their method with respect to prior work lies in the choice of the latent space and the selection of geometrically constrained codebook vectors, rather than optimizing a separate codebook matrix. The paper conducts a variety of experiments to demonstrate that HyperVQ achieves generative performance on par with standard VQ approaches and also achieves improved discriminative capabilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have highlighted my requested changes below and have specified for each whether they are critical for securing my recommendation for acceptance:\n\n(1) I am requesting one main change in terms of results (critical for securing acceptance): please provide standard deviations for results so that the reader may assess statistical significance! This would be particularly useful in highlighting when even a small mean difference between two results is actually quite significant statistically. \n\n(2) Although the writing in the paper is understandable and generally clear, it can and should be improved prior to publication. I am requesting the following changes to the writing in the paper (critical for securing acceptance):\n\n- Section 2.1: The citations \"Gray (1984)\", \"Esser et al. (2021)\", \"Crowson et al. (2022)\" should all be parenthetical (\\citep). This is a persistent issue throughout the paper, that is, in-text citations are frequently used when parenthetical citations should be used instead. Please fix this in all places where it occurs.\n\n- Section 2.1: \"Yang et al. Yang et al. (2023)\" -> \"Yang et al. (2023)\" (duplicate text)\n\n- Section 2.1: \"Lu et al. Lu et al. (2023)\" -> \"Lu et al. (2023)\" (duplicate text)\n\n- Section 2.2: \"Since VQ is such an important mechanism of several modern neural network architectures across domains\" -> This is not a complete sentence, please fix. An easy fix may be merging this with the previous sentence.\n\n- Section 3: \"relevant theories\" -> \"relevant theory\"\n\n- Section 3.1.1: \"a n-dimensional\" -> \"an n-dimensional\"\n\n- Section 3.1.2: \"A Riemannian manifold is called a hyperbolic space if its sectional curvature is negative everywhere.\" -> This is incorrect. The sectional curvature must be both negative **and constant** everywhere.\n\n- Section 3.1.3: Please move the word \"Operations\" back onto the same line with the rest of the title.\n\n- Section 3.1.4: \"an unidirectional hyperplane\" -> \"a unidirectional hyperplane\"\n\n- Figure 2, caption: \"on the hyperplane, $q_k$ as the codebook vector\" -> \"on the hyperplane, $q_k$, as the codebook vector\"\n\n- Section 4.1, final paragraph: Delete the extra space before \"Figure 2\".\n\n- Algorithm 1: The algorithm doesn't include the loss by which the parameters are updated via gradient descent. Either include these details or consider moving the algorithm into the appendix.\n\n- Section 5.3: \"aconsisting of a convolutional layer\" -> consisting of a convolutional layer\"\n\n- Section 5.3: Please define perplexity (i.e., you are likely using this term to mean exponentiated cross-entropy here).\n\n- Section 5.4: \"outperforms the original VQ-SA\" -> outperforms the original VQ-SA.\"", "broader_impact_concerns": "This work concerns an alternative formulation for vector quantization using hyperbolic space; given the fairly theoretical nature of this modification, I do not believe a broader impact statement specifically concerning this novelty is required. That being said, one of the main applications of the work is to generative modelling; it would be beneficial for the paper to give a brief blurb regarding the potential negative societal consequences of generative machine learning in the final section (Section 6).", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nzNmjcXPzu", "reviewer": "Reviewer_Z9Sf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strength\nThe paper is easy to read, and it’s intuitive to propose a hyperbolic variant VQ method, given the popularity of hyperbolic geometry in the recent machine learning community. \n\n# Weakness\n- The paper does not clearly explain what is meant by “polynomial volume growth in Euclidean space.” It remains ambiguous as to what this growth is measured relative to and how it integrates into the overall methodology.\n- While the paper focuses on hyperbolic spaces, it does not adequately justify why hyperbolic geometry is more suitable for addressing codebook collapse compared to other non-Euclidean spaces (e.g., spherical spaces, mixed-curvature spaces, or SPD manifolds).\n- The discussion on “index collapse” is vague. The paper did not clearly distinguish between index collapse and codebook collapse, and it does not explain why existing methods that address index collapse were not considered as competing approaches.\n- The hyperbolic embedding induces low-distortion embedding and captures the geometric structure well only for hierarchical or tree-like structures and not for any type of data. The authors did not provide the motivation of VQ in this perspective.\n- There is a noticeable redundancy between the introduction and related work sections. The introduction should focus on high-level motivations, while technical details should be confined to the related work section.\n- Given the known numerical instability of the Poincaré ball model, the paper does not discuss why alternative hyperbolic models were not considered or how the performance might be impacted if another model were used.\n- The importance of incorporating MLR in the proposed method is not well justified. Why not directly use the hyperbolic representation/hyperplane in VQ-VAE? or why not consider VQ in a hyperbolic VAE? why not using other classification method in hyperbolic spaces?\n- The paper does not provide theoretical guarantees—such as theorems or propositions—supporting the effectiveness of the proposed method in mitigating codebook collapse.\n- Mapping the Euclidean representation to the hyperbolic embedding at the origin may not necessarily capture well the hierarchical structure; how does this choice affect the proposed method?\n- Although the paper references numerous works related to vector quantization in the introduction and related work sections, the experimental evaluation includes only a limited set of competing methods.\n- Missing reference for numerical instability in hyperbolic spaces\n- d_c in Eq. (6) is not defined", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel vector quantization technique that operates in hyperbolic space. Specifically, the paper presents a hyperbolic variant vector quantization method based on hyperbolic multinomial logistic regression and the Poincaré ball model.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should clarify the definition of \"polynomial volume growth in Euclidean space\" and justify why hyperbolic space is favored over other non-Euclidean spaces for addressing codebook collapse, while also distinctly differentiating between index collapse and codebook collapse. Additionally, the paper would benefit from elaborating on the motivation for using hyperbolic embeddings to capture hierarchical data, polishing the introduction and related work sections to reduce redundancy, and addressing the numerical instability of the Poincaré ball model by either justifying its use or comparing it with alternative models. Moreover, it is essential to clarify the role of multinomial logistic regression within the proposed method and examine the implications of mapping Euclidean representations to the hyperbolic space at the origin. Finally, providing theoretical guarantees for mitigating codebook collapse, expanding the experimental evaluation to include a broader set of competing methods and updating the references to support claims regarding numerical instability will further solidify the work.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: We concur with the reviewers' unanimous support for acceptance. The paper is expected to be of broad interest to the TMLR audience, and the central claim of the viability of a hyperbolic-based quantisation strategy has been well demonstrated on a range of datasets. The authors did a good job of addressing several questions and suggestions from reviewers, which have strengthened the paper.
Decision: Accept (poster)