Dimensional Collapse in VQVAEs: Evidence and Remedies
Authors: Jiayou Zhang
Yifan Shen
Guangyi Chen
Le Song
Eric P. Xing
Venue: NeurIPS 2025 poster ()
Keywords: vector quantization
representation collapse
Primary area: deep_learning
TL;DR: We identify dimensional collapse in VQVAEs, where the codebook's effective dimensionality is surprisingly low, and investigate its implications and potential remedies.

Abstract: Vector-Quantized Variational Autoencoders (VQVAEs) have enabled strong performance in generative modeling by mapping continuous data to learnable codes.
In this work, we identify a surprising yet consistent phenomenon that we term \emph{dimensional collapse}: despite using high-dimensional embeddings, VQVAEs tend to compress their representations into a much smaller subspace, typically only 4 to 10 dimensions.
We provide an in-depth analysis of this phenomenon and reveal its relation to model performance and learning dynamics.
Interestingly, VQVAEs naturally gravitate toward this low-dimensional regime, and enforcing higher-dimensional usage (e.g., via rank regularization) could lead to degraded performance.
To overcome this low-dimensionality limitation, we propose \textbf{Divide-and-Conquer VQ (DCVQ)}, which partitions the latent space into multiple low-dimensional subspaces, each quantized independently.
By design, each subspace respects the model’s preference for low dimensionality, while their combination expands the overall capacity.
Our results show that DCVQ overcomes the inherent dimensional bottleneck and achieves improved reconstruction quality across image datasets.

Dimensional Collapse in VQVAEs: Evidence and Remedies
Jiayou Zhang', Yifan Shen', Guangyi Chen', Le Song 1., Eric Xing!.2.3 "GenBio Al *Carnegie Mellon University
*MBZUAI
{jiayou zhang, yifan. shen, guangyi chen, le. le.song, eric. (eg..xing)mmbbzua. ac. ae
Abstract
Vector-Quantizze Quantized Variational Autoencoders (VQVAEs) have enabled strong per- formance in generative modeling by mapping continuous data to learnable codes. In this work, identify surprising yet consistent phenomenon that term di- mensional collapse: despite using high- high-dimensional embeddings, VQVAEs tend to compress their representations into much smaller subspace, typically only 4 10 dimensions. We provide an in-depth analysis of this phenomenon and reveal its rela- tion to model performance and learning dynamics. Interestingly, VQVAEs naturally gravitate toward this low-dimessional regime, and enforcing higher-dimensiin usage (e.g., via rank regularization) could lead to degraded performance. To overcome this low-dimensionality limitation, we propose Sovide-and-Conquee VQ (DCVQ), which partitions the latent space into multiple dimensional subspaces, each quantized independently. By design, each subspace respects the model's preference for low dimensionality. while their combination expands the overall capacity. Our results show that DCVQ overcomes the inherent dimensional bottleneck and achieves improved reconstruction quality across image datasets.
Introduction
Learning discrete representations has become key building block in modern generative modeling, with Vector- cucioo-Quattzzeed Variational Autoencoders (VQVAEs) [18) standing out as popular approach. VQVAEs have demonstrated strong empirical success across domains such as images, videos, audio, and protein structures 4 I7 20,
2 19 6), enabling efficient and flexible generative pipelines. At the core of VQVAEs is the idea of discretizing a continuous latent space into learnable codes with code associated with an embedding.
Prior work has primarily focused codebook collapse, where only subset of codebook entries utilized. contrast, we uncover a different and pervasive phenomenon, which we term dimen- sional collapse: despite using high-dimensional embeddings, VQVAEs tend to compress their representations into a much smaller subspace, typically around to 10 dimensions (Figure This behavior consistently appears across wide range of pretrained VQVAE models spanning domains and architectures.
To better understand this behavior, we conduct large-scale controlled experiments varying dataset, architecture, and hyperparameters. We observe a surprising U-shaped relationship between effective dimensionality and validation loss: performance improves as the dimension increases, but worsens beyond certain optimal point (Figure [T)). Notably, the optimum lies a low-dimensional regime, highlighting the model's inherent preference and limitation in its expressiveness. A natural goal is to shift this optimum toward higher dimensionality and lower loss (Figure Tb).
Achieving this goal requires understanding why the model favors low- -iw-dimesiional solutions. We find that collapse originates in the quantizer early in training. with the commitment loss pulling the encoder to match this structure and reinforcing reiffreinn loop. Beyond training dynamics, we
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Valldation Loss
Hyperparameter (eg. commitment weight)
Validatton Loss
VQVAE
DCVQ
Codebook
Effective Dimension
Effective Dimension
(b)
(c)
Figure Illustration of dimensional collapse in VQVAEs. (a) Codebook entries lie in a low- dimensional subspace despite being in high-dimensional embedding space. (b) Validation loss follows U-shaped curve as a function of effective dimension, with the optimum a low dimension d² Hyperparameters such the commitment loss weight control the effective dimension. (c) Our method, DCVQ, shifts the optimum
higher effective dimension d' and achieves lower loss.
show that this behavior stems from structural bias of quantization itself, which favors high-variance directions and contracts the representation' 's effective dimension. While rank regularization can counteract this bias by increasing dimensionality, it consistently degrades performance, indicating that naively enforcing rank is ineffective.
Motivated by these insights, propose Seiiee-andConquuee VQ (DCVQ), simple yet effective architectural modification that addresses collapse at its source: the quantizer. DCVQ partitions the latent space into multiple low- www-dimeniional subspaces, each quantized independently. This design respects the model's natural preference for low-rank structure while scaling capacity through parallel subspaces. As result, DCVQ breaks the intrinsic low-dimensional limit and achieves improved reconstruction quality and effective dimensionality across datasets.
Contributions. Our main contributions are:
We identify and systematically characterize dimensional collapse in VQVAEs, where latent representations occupy a surprisingly low-dimensional subspace.
We show that collapse arises not only from training dynamics but from structural bias of quantization itself, which favors high-variance directions and contracts the representational spectrum.
Through controlled experiments, reveal U-shaped relationship between effective dimension and performance, and show that rank regularization fails to resolve collapse, often degrading performance.
We propose DCVQ, which partitions the latent space into multiple independently quantized low-dimensional subspaces, effectively breaking the U-shape limit.
We validate DCVQ across datasets and architectures, showing consistent gains in recon- struction quality and dimensional utilization.
2
Background and problem setup
In this section, we formalize the VQVAE framework, introduce key concepts related to dimensional collapse and describe how estimate the effective dimensionality.
2.1 Vector-Quantized Variational Autoencoders (VQVAEs)
VQVAE consists of three main components: an encoder E, a codebook C = {e; [K and a decoder
D. Given an input x, the encoder produces a continuous latent vector = = {(x) E Rd where d the background dimensionality. This vector is then discretized by replacing it with its nearest codebook entry:
where h = arg min ex Ck=,
If the encoder outputs multiple latent vectors (e.g., patch embeddings). quantization applied independently to each. The decoder reconstructs the input from the quantized latent: & = D(2).

---

The training objective combines reconstruction loss (e.g., mean squared error) and a commitment loss that encourages the encoder output to stay close the selected codebook vector 2. When using exponential moving average (EMA) updates [18], the codebook embeddings are updated separately from gradient descent and thus not appear the training loss. The resulting training loss is:
L = x 21 + = sg[2]]l2
where sg[-] denotes the stop-gradient operator, and | controls the strength of the commitment.
2.2 Dimensional collapse in VQVAEs
While the background dimensionality is fixed by model design, the effective dimensionality refers to the number of directions in latent space that are meaningfully utilized by the model. In practice, we observe that VQVAEs often use only a low-dimensional subspace of the full latent space, phenomenon we refer to as dimensional collapse.
estimate the effective dimensionality using principal component analysis (PCA). The explained variance ratio X, of each principal component indicates how much variance is captured along that direction. We define the effective dimension (Eff. Dim) as the minimum number of principal components required to explain a fixed proportion (e.g.. 99%) of the total variance: Effective Dim = min d'
Formally, given the code vectors C {er
A3 > 0.99
j=1
Unless otherwise noted, we use 99% variance threshold throughout our experiments.
Explained Variance (VQGAN)
Analyzing dimensional collapse
Cumulative Variance
1.0
In this section, we perform a comprehensive analysis of dimensional collapse in Dector-Quantized VAEs (VQVAEs). We begin by surveying broad set of pre- trained models and observe that collapse emerges as general and robust phenomenon, across modalities and architectures.
0.8
0.6
Effective dim
(explains 99% variance)
0.4
0.2
We then conduct large-scale controlled experiments to gain deeper insight into the relationship between effective dimensionality and model performance. These studies reveal consistent U-shaped curve: while extremely low-dimensional codes hurt expres- siveness, foreing higher dimensionality beyond a low- dimensional sweet spot also leads to performance degradation.
0.0
0
200 Principal Components
100
Figure Dimensional collapse in VQGAN (trained on ImageNet). Over 99% of variance is captured by only 4 principal components, despite 256-dimensional background space. To explain this behavior, we analyze training dynam- ies and show that dimensional collapse originates early in the quantization layer, then propagates to the encoder through the commitment loss. This insight is supported by temporal measurements and correlation analysis of key hyperparameters.
Finally, we evaluate whether rank regularization, an intuitive fix, can resolve collapse. Our experi- ments show that while such regularization increases effective dimensions, it also harms performance, highlighting that collapse reflects the quantizer's 's training dynamics. These findings motivate the need for architectural solution, which present in Section
3.1 Collapse in pretrained models
To assess the generality of dimensional collapse, we begin by analyzing diverse collection of pretrained VQ models spanning multiple domains, including images, video, and protein structures. For each model, we extract the latent codebook embeddings and compute their effective dimensionality via Principal Component Analysis (PCA), defined as the number of principal components required to explain 99% of the total variance.
3

---

80
Type
Modality garal
We start with VQGAN, one of the most widely used VQVAE variants. As shown in Figure 2 despite a 256-dimensional background space, over 99% of the variance captured by just four principal compo- nents, which is an extreme case of collapse. The sharp slope in the cumulative variance curve highlights how aggressively the model compresses information into a narrow subspace.
VOVAE
Image
RQVAE
Video
DCVQ (Ours)
Protein
60
Effective Dim
40
ROAE
30
DCVO
Figure 3 extends this observation across a wider range of models and modalities. Both vanilla VQ- VAE and RQVAE (9] consistently exhibit low effec- tive dimensionality- simensionalltyyooten often below even when the background dimension is 256. This pattern holds across domains, indicating that the collapse is not datasct-specific but instead reflects architectural training-level constraints.
PC10 112d1
Nuskō1-236
Naskan
10
VOGAN-1GRRE
Fole
VQGAN
ViSe:GPT
50
100
150
200
250
Background Dim
Effective vs. background dimension- ality across pretrained VQ models. Vanilla VQVAE and RQVAE consistently underuti- lize their latent space. Multiple models lie (256, 4); to improve visibility, their positions are slightly jittered.
Figure 3:
3.2 VASs low-dimensional preference: U-shaped performance curve
To systematically study the conditions under which dimensional collapse arises and affects performance,
we conduct large-scale controlled experiment. We train 512 VQVAES from scratch under diverse combinations of dataset, architecture, and hyperparameter settings, allowing us to isolate key factors influencing effective dimensionality.
3.2.1 Experimental Setup
Datasets. We use CIFAR-10 [ and CelebA
[12], following standard preprocessing pipelines. CelebA images are resized and center-cropped to 64 x resolution. All images are normalized using dataset-specific mean and standard deviation. Our implementation builds on S and (2
Architectures. We evaluate both CNN-based and ViT-based encoder-decoder architectures. These are based on VQGAN 4 and VīT-VQGAN [3], respectively. We vary the encoder scale factor (f), background dimension, and codebook size. All models apply L2 normalization before quantization.
Hyperparameters. Table I summarizes the search space. To vary the background dimension without changing the encoder, we insert a projection layer after the encoder and a corresponding inverse projection before the decoder. This setup enables isolate the codebook background dimensionality from the encoder and decoder dimensionality.
Hyperparameter
Values Sampled
Codebook size
512, 1024, 2048, 8192
Background dimension
4, 6, 8, 16, 32, 64, 128, 256
Commitment loss weight
0.1, 0.2, 0.5, 0.8, 1.0, 1.2, 1.5, .... 5.,5..,5555, 6.0
EMA decay
0.5, 0.8, 0.9, 0.95, 0.98
Rotation trick
Enabled / Disabled
Dead-code restart threshold
0.008, 0.032, 0.125, 0.5
Table 1: Summary of hyperparameters explored in the large-scale controlled study.
3.2.2 Discovering the U-shaped performance curve
We next examine how effective dimensionality influences reconstruction performance across models. Using the 512 configurations described above, train all models for 100 epochs under consistent training settings to ensure comparability across architectures and hyperparameters.
Figure 4 plots the validation loss as function of effective dimensionality for different datasets (CIFAR-10 10, CelebA), model types (CNN, ViT), and encoder scale factors (ʃ = 4, 16). Each point represents a trained model, with color indicating codebook size.

---

Validation Lass Effective Dimension Colored by Codebook Size
CelebA, CNN,
CelebA, VIT, fm
CalebA, CRN, f-16
Cale DA, VIT, F-15
6.3100
6,066
0:2075
0.05
6.04
0.3050
3 2.044
0.8
0.23
Pagen
0.02
0.302
10
15
20
20
Effedive
Effective
Effective D
CIFAR1O, CNN, fmnd
CIFARIO, VIT, find
CIFARIO, CNN, fx166
CIFAR10, WT. fm16
i
0.35
0.15
0.30
4.5
0.55
0.4
10
20
30
52
5
10
10
15
20
30 Eflection
Effective
Effecitve Dim
Effection D'm
Figure 4: Validation loss (reconstruction MSE) versus effective dimension across datasets, architec- tures, and scale factors. Color represents codebook size. Within each color group, one can observe a U-shaped curve, indicating consistent relationship between effective dimension and performance.
Across most settings, we observe clear U-shaped pattern: models with very low effective dimen- sionality perform poorly, while performance improves as dimensionality increases up to a certain point. Beyond this point, performance begins to degrade, resulting distinct U-shaped curve. For example, on CIFAR- 10 (ViT, f = 4), models with effective dimensions in the 4-10 range achieve the best performance; higher dimensions offer no gain or even degrade results.
This U-shape suggests the existence of an intrinsic "sweet spot" in latent space usage, typically low but not minimal, where the model achieves optimal reconstruction fidelity. For each model, this corresponds to fixed effective dimension d' that it consistently converges to when properly trained. natural question then arises: how can we shift this optimal effective dimension higher to enhance the model's expressiveness? To answer this question, we first investigate the cause of the collapse.
3.3 Understanding the cause of collapse
Effective Dim vs. Background Dim
Upprerr
Upt
88
18
OFAR10, CNK, fet
CIRAR10, VIT. fed
CIRARL 6, CHN, fw
CIFAK10, VIT, f-
60
8
LOOAA
15
32
200
153
200
250
52
320
1.50
200
250
50
202
150
200
250
30
120
150
200
250
Backgraundd sunggrunnddddo
Background
Background
Background Dim
Figure 5: Effective dimension versus background dimension across datasets, architectures, and scale factors. The dashed line indicates the upper bound where effective dimension equals background dimension. As background dimension increases, effective dimension tends to plateau, indicating that higher latent capacity is not utilized.
We begin by examining whether dimensional collapse is simply consequence of limited latent size. As shown in Figure [] when the background dimension is small (c.g.. 3-6), the effective dimension scales linearly. However, once the background dimension becomes sufficiently large, the effective dimension saturates and diverges from the upper bound. This decoupling suggests that collapse is not caused by latent capacity itself, but rather emerges from internal training dynamics. Notably, this saturation effect occurs across datasets and architectures, implying a general phenomenon.
To gain deeper understanding of where and when collapse originates, we analyze the training-time dynamics of effective dimensionality. As shown in Figure 6 collapse occurs rapidly- wiithin the first 10k training steps-an and then plateaus. More importantly, observe that the codebook consistently
5

---

Background Codebook Encoder Effective Dimensionality Over Training Steps
Background
Background
200
doiie
200
Gooe dec
negative codeboot dalaprer more veverey
dode dec
regubre codebeat colacees more sovemy
negative codebcor calacrer more reverey
-264
6033 Lab32
(403)
Training Stop
Training Step
Training Step
Figure 6: Evolution of effective dimensionality over training steps for different background dimen- sions (d 64, 128, 256). Top row: effective dimensionality of the codebook embeddings (deode). Bottom row: difference between codebook and encoder dimensionality (deode denc)), where negative values indicate that the codebook collapses more severely than the encoder. Collapse occurs rapidly in early training and then stabilizes, with the encoder gradually adapting to the collapsed codebook.
collapses earlier than the encoder. The bottom row shows the dimensionality gap (deode dene it initially negative, revealing that the quantizer underutilizes dimensions first. Over time, the encoder's dimensionality drops as conforms to the collapsed codebook.
These results indicate that collapse is not a failure of encoder expressiveness, but is instead induced by early-stage quantizer behavior. This in turn suggests that collapse originates from the training dynamics of the quantization mechanism itself.
To further validate this hypothesis, analyze correlations between hyperparameters and final effective dimensionality (Table 2). Among all factors, the commitment loss weight shows the strongest and most consistent negative correlation across background dimensions. This aligns with the temporal findings: the commitment loss acts as the conduit through which early quantizer collapse propagated to the encoder.
Taken together, these results support a coherent causal explanation: dimensional collapse originates in the quantizer due to unstable training dynamics, and is reinforced by the commitment term that pulls encoder outputs toward a degenerate codebook.
Table 2: Pearson correlation between effective dimensionality and hyperparameters, grouped by background dimension. Commitment loss weight shows the strongest and most consistent correlation.
Hyperparameter
Background Dim
Avg.
16
32
64
128
256
Commitment Loss Weight
-0.45
-0.72
-0.79
-0.68
-0.50
-0.62
-0.40
-0.60
Codebook Size
0.57
0.19
0.17
0.49
0.39
0.55
0.59
0.42
Rotation Trick
0.29
0.19
0.13
-0.02
0.41
0.39
0.44
0.26
Code Restart Threshold
-0.10
-0.09
0.06
-0.09
-0.08
-0.01
0.15
-0.02
EMA Decay
-0.04
0.03
0.10
0.07
0.05
0.03
-0.03
0.03
3.4 Quantization bias as structural cause of collapse
While the previous analysis shows that collapse originates in the quantizer, the underlying cause remains to be explained. We hypothesize that this behavior is not incidental but arises from inherent bias the quantization process itself. Vector quantization in VQ-VAE can be viewed as an online form of k-means, with codebook entries centroids. Classical analyses relate k-means to PCA by showing that the continuous relaxation of the k-means objective a trace maximization problem whose global optima given by the top eigenvectors of the covariance/Gran matrix; equivalently, principal components arise as continuous solutions to the discrete cluster-indicator problem 27 3. This linkage implies that K-means effectively operates within the reading-varianes
dene is computed from the encoder outputs of the current batch.

---

subspace, which helps explain our empirical finding that low-variance directions are attenuated after quantization. This bias naturally leads contraction of the representation' 's effective dimension.
Figure illlsttraess this behavior
in controlled syntheric setting.
Eijenva Preservation Rarle
Project cnto Components
We generate high-dimensional
Cancola
Data
Gaussian data with a prescribed covariance spectrum and apply k-means clustering learn 512 centroids, then compare the co- variance eigenvalues of the data and centroids to quantify the change in effective rank (code provided in Sec. G). The ac- companying visualization shows that smaller eigenvalues shrink markedly after clustering, while projection onto the first and last principal components reveals that centroids spread along the high-variance direction but col- lapse along the low-variance axis. This geometric contraction mirrors the theoretical link between k-means and PCA-where the relaxed and together they
Ax10-1
Dxte-
Flos Priedpa Companens Doghh inrlones
Egenssue LaB to leaa
Figure 7: Synthetic illustration of quantization bias via k-means clustering. (a) The ratio between centroid and data eigenvalues (log scale) shows that clustering preserves high-variance directions while attenuating low-variance ones. (b) A projection onto the first and last principal components confirms this effect visually: centroids (red) align with the high-variance axis, whereas variation along the low-variance direction largely lost.
k-means solution lies in the subspace spanned by leading eigenvectors [27 3
indicate that quantization inherently biases representations toward low-dimensional, high-variance subspaces, offering a structural explanation for dimensional collapse.
In summary, both theoretical and empirical evidence suggest that dimensional collapse is not a pathological optimization artifact but structural outcome of the quantization process itself. By favoring directions of high variance, quantization inherently compresses the representational spectrum, even when the encoder produces high-rank features.
3.5 Failure of Rank Regularization
Given that collapse originates in the quantizer and is reinforced by training dynamics, one might wonder whether it can be mitigated by explicitly encouraging high-rank encoder outputs. To investi- gate this, tested wide range of rank-promoting strategies applied directly the encoder outputs, including KoLco regularization [16] [4], Barlow Twins (25), VICReg [[]. Spectrum Hinge, and Rank Hinge. These methods address collapse through embedding spreading, redundancy reduction. covariance spectrum shaping, and rank thresholding. Notably, KoLco, Barlow Twins, and VICReg originate from the self-supervised learning literature, where they are designed to prevent represen- tational collapse. A summary of these methods and their corresponding objectives is provided in Table 7
Table 3 summarizes the quantitative effects of these regularizers on both CIFARIO and ImageNet. Across all methods, stronger regularization consistently increases the effective dimensionality of encoder features but also leads to higher reconstruction loss, confirming that the apparent rank expansion occurs primarily before quantization and does not translate into better post-quantization performance.
These results reinforce a key insight: simply increasing the effective dimension does not improve performance. On the contrary, forcing high-rank outputs from the encoder undermines the model's natural dimensional biases and yields lower-quality latent representations. Importantly, encoder-side regularization fails to address the root cause of collapse: the behavior of the quantization layer.
This limitation highlights the need to intervene directly at the quantizer level, motivating strategy that respects the model's low-dimensional preference while expanding the latent capacity. We introduce such approach, DCVQ, in the next section.
7

---

Table 3: Validation reconstruction loss and effective dimension under different rank-promoting regularizers, Results are reported for CIFARIO (CNN, 4=4) and ImageNet- Ik (VIT, f-16). For each regularizer, the lowest validation loss and the highest effective dimension are marked with "*"
CIFAR10
ImageNet-1k
Regularizer
Weight Val Loss
Eff. Dim
Weight Val Loss Eff. Dim
None
0.081
9
0.001
0.081*
9
KoL.o
0.01
0.082
10
0.1
0.106
27
0.0001
0.081*
10
0.0001
0.173*
14
0.001
0.093
31
0.001
0.186
41
Barlow Twins
0.01
0.105
78
0.01
0.205
91*
0.1
0.087
49
0.1
0.182
0.1
0.081
0.1
0.172
9
0.080
10
0.164*
VICReg
10
0.082
11
10
0.169
13
100
0.085
12a
100
0.180
20
0.1
0.080*
9
0.1
0.158
9
Spectrum Hinge
0.081
9
1
0.168
11
(hinge=0.01)
10
0.089
49
10
0.181
63
100
0.129
111
100
0.247
137*
0.1
0.080
Spectrum Hinge
0.081
10
0.171
(hinge=0.1)
10
0.084
11
10
0.183
15
100
0.085
12
100
0.186
17
0.001
0.081
9*
0.001
0.173
9*
Rank Hinge (hinge=64)
0.01
0.081
9*
0.01
0.173
9
0.1
0.080
9*
0.1
0.173
9*
0.081
9*
0.173*
0.001
0.080*
9*
0.001
0.166
9*
Rank Hinge
0.01
0.082
9*
0.01
0.163*
(hinge=128) 128)
0.1
0.080
9*
0.1
0.168
ge
0.081
9*
0.166
9s
DCVQ: Divide-anddconquer quantization for dimensional collapse
4.1
Motivation
Our analysis dimensional collapse reveals a surprising yet consistent trend: model performance, as measured by reconstruction quality, first improves and then degrades as the effective latent dimension increases. Critically, the optimal performance typically emerges at low effective dimension.
This finding suggests that VQVAEs intrinsically prefer operating in low-dimensional latent subspaces. Forcing the model uniformly utilize higher-dimennsonall latent space, such as through explicit rank regularization, would degrade the performance. Therefore, more natural strategy is to embrace the -in-dimeniional preference rather than fighting against
However, solely operating at such low effective dimensions severely limits the total information capacity of the latent space. To balance these competing needs-respnettng low-dimensional structure while scaling total capacity-we propose DCVQ.
4.2 DCVQ architecture
Divide-and-Congee Vector Quantization (DCVQ) is simple architectural modification VQVAE that addresses dimensional collapse by splitting the latent space into multiple independently quantized

---

subspaces. This design reflects the intrinsic preference of VQVAEs for low-dimensional structure while scaling total capacity through parallelism.
Effestne Ohes
Ellcctie Dimi
Split
Quantize
Merge
Figure 8: DCVQ divides the encoder output into multiple low-dimensional subspaces and quantizes each independently. The quantized subspaces are then merged via a direct sum (concatenation) and passed to the decoder. This divide-and saneee--n--onquee strategy enables high total capacity while preserving the model's preference for low-dimensional structure.
Divide Step. Given an encoder output & E R
Conquer Step. Each subspace 2" is quantized
DCVQ partitions into N equally sized sub- spaces, each of dimension d,, with d = N dg:
separately using its own codebook C,:
Quantize (2tizeee2",,,,) ,C;))
12
M, 2' Z'R R
The final latent representation is the concatena- tion of the quantized subspaces:
This decomposition enables each subspace to op- erate independently in a oiw-dimesiional regime close to the model's optimal effective dimension (dg d²). Based on prior observations, we use d = 8.
N
Rd.
Concatenation forms a direct sum of subspaces. By linear algebra, the rank of the concatenated space equals the sum of the ranks of the individual subspaces. Thus, stacking more quantizers leads to linear increase in effective dimension. This allows the model to achieve high expressivity without violating its low-rank preference.
Experiments
5.1 Comparison with vanilla VQVAES
We compare DCVQ with stan-
Validation vs. Effective Dimension
dard VQVAE models across datasets (CelebA, CIFAR10) and architectures (CNN, ViT), with results shown in
fm16
0.03
0.04
Figure 9 Here, we fix the codebook size 512 entries for all models and consider only the setting with f = 16. We vary the subspace dimensionality d, between 8 and 32, with d, 8 assumed be near the optimal for in- dividual codebooks.
0.02
6x200
15x0d
15H333
0.02
2-04
40320
16h9J
30420d
Effective Dim
Effective Dim
CIFAR10,
CIFAR1O,
0.6
Across all settings, we observe that DCVQ achieves lower validation loss significantly higher effective dimen- sions, demonstrating that our goal in Figure is realized: the performance curve shifts toward both higher dimen- sionality and lower loss. This con- firms that DCVQ overcomes the low- dimensional bottleneck of vanilla VQ- VAEs and scales more effectively, The gray points represent vanilla VQVAEs, which consistently
0.4
0.4
4+300
Bx30c
0.2
0.2
5400
10x00
2+80
40
60
50
100
25
50
75
300
Effective Dim
Effective Dim
Method
DCVQ-8d
DCVQ-32d
VOVAE
Figure 9: Comparison with vanilla VQVAEs. Text annota- tions are N d, (i.e., #Quantizers Subspace Dim)

---

operate in a narrow low- low-dimensional regime. In contrast, DCVQ variants span a broader range of effective dimensions while achieving better loss. Notably, models using d, 32 exhibit higher validation loss than those using d, = 8, supporting the choice of d, = 8 as a near-optimal subspace dimension. These results demonstrate that DCVQ enables higher-capacity representations without sacrificing reconstruction quality.
5.2
Comparison with ROVAES
Since DCVQ employs
(1) #Quantizers
rFID VS. +Quantizers
HFD (+) Dim
multiple codebooks, we compare it against Residual Quantization (RQ, also known as RQVAE) [9) a method that also leverages multiple quantizers.
(hecche Dm
+
We evaluate reconstruction performance ImageNet- 256 using reconstruction Fréchet Inception Distance (rFID) the metric. All models use convolutional encoder-decoder architec- ture from the RQ-VAH codebase. For DCVQ, the latent vector is partitioned into subspaces of fixed dimension (dg {8, 64)), each quantized independently. We vary the number of quantizers N E {8, 16, 32), ensuring that the total codebook dimensionality for RQ matches that of DCVQ-8d (i.e., d = 8NN.. The codebook size fixed at 16,384 across all settings.
10
60
4Quantieer
#Quantzers
Effective Dim
nQ
DO10-35
DOV0
Figure 10: Effective dimension and reconstruction FID (rFID) across different quantizer counts. We vary the number of quantizers while keeping other hyperparameters, such as the dimensionality of each quantizer, fixed to study how reconstruction quality evolves.
Figure shows that DCVQ achieves significantly higher effective dimensionality than RQ as the number of quantizers increases, indicating more expressive latent codes. DCVQ-&d and DCVQ-64d converge to similar effective dimensions despite differing background sizes, consistent with our earlier observation that beyond a threshold, background dimension no longer constrains capacity.
In terms of rFID, DCVQ consistently outperforms RQ when N > 16. While RQ quickly saturates, DCVQ continues to improve with more quantizers, especially with d, 8, achieving higher effective dimension and better reconstruction quality. These results confirm that DCVQ not only expands latent capacity effectively but also translates that capacity into meaningful performance gains.
Conclusion and Discussion
We study dimensional collapse in VQVAEs, phenomenon where models utilize only small subspace (typically 4-10 dimensions) regardless of high background dimension. This limits expressiveness, as increasing the background dimension yields diminishing gains in effective usage.
Our analysis reveals that collapse originates from early codebook degeneration and cannot be resolved by encoder-side regularization alone. To address this limitation, we propose Divide-and-Conquer VQ (DCVQ), which splits the latent space into low-dimensional subspaces. each independently quantized. This respects the model's s low- -iwddimensional preference while expanding total expressivity. achieving both low background dimension and high utility.
Experiments across benchmarks show that DCVQ consistently improves both effective dimensionality and reconstruction quality. These results confirm that DCVQ not only expands latent capacity effectively, but also translates that capacity into meaningful performance gains.
Limitations. This work is primarily empirical and does not yet offer theoretical explanation for codebook collapse. Future work should aim to provide theoretical grounding and extend DCVQ to other modalities.
10

---

References
[1] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variaaceiivrraacc--vvaaaan regular- ization for self-supervised learning. In ICLR, 2022.
[2] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00341, 2020.
[3] Chris Ding and Xiaofeng He. K-means clustering via principal component analysis. In Proceedings of the twenty-first international conference Machine learning, page 29, 2004.
[4] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12873 12883, 2021.
[5] Christopher Fifty, Ronald G Junkins, Dennis Duan, Aniketh Iyengar. Jerry W Liu, Ehsan Amid, Sebastian Thrun, and Christopher Ré. Restructuring vector quantization with the rotation trick. ArXiv, abs/2410.06424 2024.
[6] Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, et al. Simulating 500 million years of evolution with language model. Science, page eads0018, 2025.
[7] Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Understanding dimensional collapse in contrastive self-supervised learning. arXiv preprint arXiv:21 10.0938,, 2021.
[8] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[9] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523-11532. 2022.
[10] Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Bhiksha Raj, and Zhe Lin. Imagefolder: Autoregressive image generation with folded tokens. arXiv preprini arXiv:2410.01756 2024.
[11] Xiang Li, Kai Qiu, Hao Chen, Jason Kuen, Jiuxiang Gu, Jindong Wang, Zhe Lin, and Bhiksha Raj. Xq-gan: An open-source image tokenization framework for autoregressive generation. arXiv preprini arXiv:24/2.0176. 2024.
[12] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730 3738, 2015.
[13] Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: Vq-vae made simple, 2023.
[14] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 2023.
[15] Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high- fidelity images with vq-vae-2. In Advances in neural information processing systems, pages 2019.
[16] Alexandre Sablayrolles, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Spreading vectors 2018.
for similarity search. arXiv preprint
[17] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. 2024.
[18] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.

---

[19] Michel van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. Biorxiv, pages 2022-02, 2022.
[20] Jacob Walker, Ali Razavi, and Aäron van den Oord. Predicting video with vqvae. arXiv preprint arXiv:2103.0195, 2021.
[21] Phil Wang. vector-quantizz-pytorre
http://ggttuuu...mmmmieceeee pactor-quantize-pyooooo 2021. Accessed: 2025-04-25
[22] Dongchao Yang, Songxiang Liu, Rongjie Huang, Jinchuan Tian, Chao Weng, and Yuexian Zou,
Hifi-codec: Group-residual vector quantization for high fidelity audio codec. 2023.
[23] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2 10.04627, 2021.
[24] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10459-10469, 2023.
[25] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International conference machine learning, pages 12310-12320. PMLR, 2021.
[26] Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi. Soundstream: An end-to-end neural audio codec, 2021.
[27] Hongyuan Zha, Xiaofeng He, Chris Ding, Ming Gu, and Horst Simon. Spectral relaxation for k-means clustering. Advances in neural information processing systems, 14, 2001.
[28] Yongxin Zhu, Bocheng Li, Yifci Xin, and Linli Xu. Addressing representation collapse in vector quantized models with one linear layer. 2024.
12

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] is generally preferable to "[No]" it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract and introduction accurately summarize the paper's key contri- butions: (1) the discovery and analysis of dimensional collapse in VQVAEs, and (2) the proposal of Divide-and- -nn--uee VQ (DCVQ) overcome this limitation. These contribu- tions are supported by empirical analysis in Section 3 and the proposed method and results in Section and 5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
13

---

Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: A dedicated "Limitations" paragraph is included at the end of Section 6. We acknowledge that our study is primarily empirical and does not provide a theoretical explanation for codebook collapse. We also note that the applicability of DCVQ modalities beyond vision remains unexplored and is left for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-tex. system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper is primarily empirical and does not include formal theoretical results, assumptions, or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
14

---

Answer: [Yes]
Justification: We provide detailed descriptions of model architecture, training procedures, datasets, and evaluation metrics Section and 4 and Appendix. These include hyperparam- eter settings, codebook configuration, and data preprocessing steps, sufficient reproduce the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will include anonymized code and detailed instructions to reproduce our main experimental results in the supplemental material before the final appendix submission deadline. The code will cover both our proposed method (DCVQ) and baseline comparisons.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co co/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
15

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ht.ps..cc/ubblcc/uuddeeee Ssw.aeeesesoooSSesessiisassiaase wwwpeeeSSuiiissssiniiii ssiiioicccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide the key experimental settings the main paper (Section 3 and 5), including datasets used, model configurations, and evaluation metrics. Full training and testing details, including data splits, hyperparameters, and optimizer configurations, will be included in the submitted anonymized code and supplemental materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: While we do not explicitly report error bars or confidence intervals, we conduct large-scale controlled experiment involving 512 independently trained VQVAEs across diverse set conditions (Section 3.2). This extensive coverage enables us to systematically analyze the statistical trends and variability of dimensional collapse. The number of runs and control over variables provide strong empirical grounding for the observed U-shaped performance patterns.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
16

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We will provide full details compute resources, training time per experiment, and total compute cost in the Appendix of the supplementary material. All experiments were conducted NVIDIA A100 GPUs with S0GB memory, though the experiments can be reproduced on GPUs with 40GB memory with mild modifications.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics and confirm that our research fully complies. Our experiments only publicly available datasets (CIFAR-10 10, CelebA, and ImageNet) under standard preprocessing. No human subjects, sensitive data, or person ally identifiable information are involved.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Our work focuses on understanding and improving the latent representation properties of VQVAFs, foundational generative modeling technique. We do not target any specific downstream application or deployment. While improved generative models could, in theory, contribute to misuse, our contribution abstract and technical in nature. Therefore, we do not explicitly discuss broader societal impacts in the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Our paper does not release any models or datasets that pose a high risk of misuse. We work with standard, publicly available datasets (CIFAR-10. CelebA, ImageNet) and do not release any pretrained generative models or scraped data. Therefore, no special safeguards necessary.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use only publicly available datasets and open-source code, and all assets are properly credited and cited in the paper. Datasets: We use CIFAR-10 and CelebA, both publicly available for academic use, and ImageNet (under its standard academic research terms). Codebases: (I) lucidrains/vectann http//eeoor-uuuatizeeeee pytorch: MIT License (2) baakaobrain//q-va---eeeeeee transformer.. Apache 2.0 License (3) cf cfiity/rotation. arraaiiootriikk: No explicit license, but official code release accompanying an ICLR paper; used un- http//aaaiinnnaaanneeeeeeeeeeeee
der academic fair use (4)
MIT License (5) MIT License. We cite all these assets in the main
CompVis/ traming-trrassfe paper and respect their licenses and terms of use. No proprietary or restricted resources are used.
18

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. comddataets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
The paper does not introduce new assets such as datasets or pretrained mod- method (DCVQ), and the accompanying code will be released in anonymized form as supplemental material for reproducibility purposes, but it is not considered structured new asset release.
Justification:
els. We propose
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve any crowdsourcing experiments or research with human subjects. All experiments are conducted on publicly available datasets, and no new data was collected from human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
19

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
This paper does not involve research with human subjects crowdsourced data collection. All experiments are conducted on publicly available datasets (CIFAR-10 and CelebA) under standard academic usage terms.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We did
any large language model (LLM) component of the methods, experiments, or contributions in this research. Any language assistance was limited standard editing and had no impact on the scientific content or originality.
Guidelines:
The answer NA means that the core method development in this research does not involve L.MMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

Contents
Introduction
2
Background and problem setup
2.1
Vector-Quantized Variational Autoencoders (VQVAES)
Dimensional collapse in VQVAES
3
Analyzing dimensional collapse
3.1
Collapse in pretrained models
3.2
VOVAEs's low-dimensional preference: U-shaped performance curve
3.2.1
Experimental Setup
3.2.2
Discovering the U-shaped performance curve
3.3
Understanding the cause of collapse
Quantization bias as a structural cause of collapse
3.5
Failure of Rank Regularization
DCVQ: Divide-and-conquer quantization for dimensional collapse
Motivation
DCVQ architecture
Experiments
Comparison with vanilla VQVAES
Comparison with ROVAES
10
Conclusion and Discussion
10
Related work
22
A.I
VQVAE and its variants
22
Multi-codehook methods
22
A.3
Codebook collapse and dimensional collapse
22
Case study: visualization of each effective dimension
23
Purpose
23
Methodology
...........
23
Visualization setup
23
B.4 Findings
23
Implementation details
25
Codebase
25
................................
C.2
Training infrastructure
................................
25
Datasets
................................
25
C.4
Hyperparameters for CelebA and CIFAR 10
................................
25
C.S
Hyperparameters for ImageNet
25
C.6
Architecture Details for CelebA and CIFAR 10
25
Architecture Details for ImageNet
26
21

---

C.8 GPU usage
27
D Details of the rank regularizers
27
Additional results on Pearson correlations with effective dimensionality
29
Computational cost of DCVQ
32
Synthetic Experiment Demonstrating Quantization Bias
33
A Related work
A.1 VQVAE and its variants
Vector- Cestor-uuantized Variational Autoencoders (VQVAEs) [8]]] have emerged as cornerstone in discrete representation learning, enabling high-fidelity generative modeling across vision [15] audio (26). video [20]. and structural biology [99 (). The central idea is to discretize continuous latent space
into learnable embeddings, typically via nearest-neighbor assignment to a codebook.
Successive works have aimed to scale and stabilize this framework. VQVAE-2 [15] introduced hierarchical quantization, while VQVAE [9] proposed recursive residual quantization. Innovations
like grouped quantization [22], the rotation trick [5], and joint codebook updates in SimVQ 28
further refine training dynamics and representation efficiency. Lookup-Free Quantization (LFQ) [24
and Finite Scalar Quantization (FSQ) B offer scalar-level discretization with extremely low latent
dimensions (e.g., 6-8), improving code utilization but at the cost of expressivity.
Despite extensive progress in improving generation quality and mitigating codebook collapse, our work focuses on dimensional collapse, more subtle and previously overlooked issue.
A.2 Multi-codebook methods
Several recent works have proposed architectural strategies that superficially resemble our approach by partitioning the latent space into independently quantized subspaces. For example, XQ-GAN and IMAGEFOLDER [T] [0] adopt product quantization to split the latent space into low-dimensional branches, each quantized separately. However, their primary goal is align tokens with spatial or semantic features to facilitate autoregressive modeling and reduce sequence length, rather than addressing the underlying structure of the latent space. In contrast, work is driven by systematic analysis of dimensional collapse in VQVAES, where high-dimensional embeddings are compressed into narrow subspaces, limiting expressivity.
Other methods, such as RQVAE 9 and grouped quantization [22], improve expressivity by intro- ducing multi-stage groupwise quantization to better approximate the encoder outputs. Visual AutoRegressive modeling (VAR) [17 further complements this line by redefining generation as coarse-to-fine finee, multi-scale prediction task, achieving state-of-the-art results in efficiency and quality. While these approaches adopt multi-codebook designs to enhance generation, our proposed DCVQ leverages them as means address latent space underutilization.
A.3 Codebook collapse and dimensional collapse
Codebook collapse- the underutilization of codebook entries-har long been recognized key limitation in VQVAES [8]. Numerous approaches have been proposed to mitigate this issue, includ- ing dead-code replacement (26), exponential moving average (EMA) updates [15), and architectural modifications that improve gradient flow and codebook dynamics (28, S]. Interestingly, several
works [13] 23 have shown that reducing the embedding dimension can lead to improved code usage, suggesting a link between codebook utilization and the underlying structure of the latent space.
However, most of these efforts focus on usage statistics such as codebook perplexity or entropy, without examining the geometry of the latent space itself. more fundamental dimee-dimensional collapse-ha has only recently begun to attract attention. This refers to the observation that, despite
22

---

operating in high- -igh-dimensional latent spaces, VQVAEs often encode information in much lower- dimensional subspace. While the term "dimensional collapse" has previously been used in contrastive self- self-supervised learning 7 to describe degeneracy in feature representations, its occurrence in VQ-based generative models has been systematically studied.
B
Case study: visualization of each effective dimension
B.1
Purpose
This section aims to provide an intuitive understanding of what is encoded in each effective dimension of the VQVAE codebook. By visualizing the role of individual principal components (PCs), we explore how semantic information is distributed across the most significant axes of variation in the learned discrete latent space.
B.2 Methodology
We perform principal component analysis (PCA) on the codebook embeddings to identify the most informative directions. Specifically, we construct series of reduced codebooks that contain only the first k principal components 1 to 6). Each level of this PCA-reduced codebook represents an increasingly complete approximation of the full latent space.
For given input image, extract its discrete token indices using the encoder. We then replace the standard codebook embeddings with their PCA-reduced counterparts before decoding. This allows us visualize how reconstructions evolve as more effective dimensions are included.
We conduct the experiment on VQGAN. Notably, our analysis reveals that over 99% of the variance in the codebook is captured by the first 4 principal components. Thus, we focus up to the first 6 PCs to analyze marginal contributions beyond the main informative axes.
B.3 Visualization setup
The visualizations are arranged in two rows for each example:
Top row: reconstructions using progressively richer PCA embeddings- from PC1 only up PC6. The baseline input and full VQGAN reconstruction are also included for reference.
Bottom row: the latent activations corresponding each individual PC, rendered as grayscale images, to show spatial distribution and intensity.
B.4 Findings
From the visualizations in Figure 11 we observe the following:
PC1 captures the global layout or coarse structure of the image. For example, the silhouette contour of person is often visible even with only the first PC.
PC2 and PC3 encode finer details such as texture and local contrast, improving visual fidelity and definition.
PC4 and PCS introduce color and tone variations, filling stylistic and chromatic informa- tion.
PC6 contributes marginally, with minimal visible structure in either the reconstruction or the latent visualization, indicating that it carries negligible additional semantic content.
This experiment supports the interpretation that VQVAE codebooks suffer from a dimensional bottle- neck, where only a few directions in the latent space carry most of the meaningful information. This visualization method complements quantitative measures by providing direct human-interpretable evidence of dimensional collapse.
23

---

Ript
FGD Laki
1V0 pXrt
PC4
Holloyel
2WV
PolLaW
Figure 11: Reconstructions and principal component visualizations for selected examples.
24

---

C
Implementation details
C.1
Codebase
Our implementation is based on several publicly available GitHub repositories:
21
(MIT License)
(Apache 2.0 License)
Stiftyyrrotation_trl.. trick |
(No explicit license; used under academic fair use as per
official ICLR release)
thuan2123/enhanciiii trrmssforrrrrsssserrrs
(MIT License)
hampViis/tamnnggrraaffoooo 4
(MIT License)
Specifically, the training script for VQVAE on CelebA and CIFAR-10 was adapted from cfiftryrottation.triikk and the ViT architecture implementation was taken from 10aamz2123///nhannnnnnttaaaaaaa
The CNN-based encoder/decoder backbone was based on CompVis/taming- www.srorrrrso....oooooooo The RR-VAE training procedure and the training script for ImageNet were derived from kakaobrain/ze rew aaraaetraansi transformo We re- re-trained RQVAE on our own setup and observed that the performance closely matched the results reported in their paper.
C.2 Training infrastructure
All experiments were conducted on NVIDIA A100 GPUs with 80GB of memory. Training was performed single GPU per Although A100s were used, models did not fully utilize the available memory, making it feasible to train them on lower-end GPUs as well. We employed Weights and Biases (wandb) for experiment tracking and conducted uniform random hyperparameter sweeps. Reported training times (in GPU-hours) are taken directly from the wandb platform.
C.3 Datasets
We conduct experiments on three datasets: ImageNet-256, CelebA-64, and CIFAR-10. ImageNet images are resized to 256x256 resolution. CelebA images are resized to 64 64, and CIFAR-10 consists of 32x32 images. These datasets cover a range of visual complexity and resolution, allowing us evaluate the model's behavior across diverse settings.
C.4 Hyperparameters for CelebA and CIFAR-10
For the experiments on CclebA and CIFAR-10 presented in Section 3 and Section of the main text, conducted hyperparameter searches over selected variables described in Table I (main text). Other hyperparameters were held fixed across runs. The full configuration used these experiments is detailed in Table We custom learning rate scheduler that linearly warms up, then decays via cosine annealing to fixed minimum learning rate.
C.5 Hyperparameters for ImageNet
For the ImageNet experiments in Section 5, we adopt the default hyperparameters provided by the RQVAE codebase, which could be found in Table
C.6 Architecture Details for CelebA and CIFAR-10
We use a standard VQ-VAE framework composed of an encoder, a vector quantizer, and decoder. The encoder and decoder architectures differ depending the model configuration:
CNN-based: We adopt convolutional encoder-decoder architecture inspired by VQGAN. The model downsamples the input by factor of 4 or 16, using 3 or 5 convolutional blocks respectively. Each block contains residual layers, and the number of channels determined by a base width of 128 multiplied by channel multiplier. The channel multiplier is set to [1, 2, 4) for a downsampling factor of 4, and [1, 1, 4) for a factor 16. The encoder output is projected to a fixed dimensionality of 256 using convolutional layer.
25

---

Hyperparameter
Value
Batch size
32
Optimizer
Adam
Learning rate
0.0001
Weight decay
0.0001
Epochs
100
Model type
VQVAE with rotation trick
Codebook type
cosine
Warmup iterations
3000
Decay iterations
50000
Stochastic sampling of codes
False
Dropout
Seed
Table 4: Fixed hyperparameter configuration for the experiments Section 3.
Hyperparameter
Value
Batch size
32
Epochs
10
Optimizer
Adam
Learning rate
4.0e-05
Betas
(.95)
Weight decay
0.0
Learning rate scheduler
fixed learning rate
Discriminator loss
Hinge
Discriminator start epoch
0
Discriminator weight
0.75
Generator loss
Vanilla
Perceptual loss weight
1.0
Table 5: Non-architectural hyperparameters used in the experiments in Section 5.
We employ Vision Transformer (ViT) encoder-decoder architecture for image tokenization and reconstruction. The input image is partitioned into non-overlapping patches of size where denotes the patch size (e.g., f 4 or 16). Each patch is embedded via a convolutional projection into a 768-dimensional vector. Fixed 2D sinusoidal positional embeddings added to the patch sequence, which is then processed by Transformer encoder comprising 12 layers, each with 12 self-attention heads and an MLP of width 3072. The encoded patch representations Transformer decoder to reconstruct the image. This architecture enables global context modeling and adaptive spatial compression.
ViT-based:
quantized and then passed to symmetric
In both architectures, linear projection maps the encoder output the quantizer input space, which allows tunable codebook dimensionality. After quantization, the embeddings are projected back before being passed to the decoder. The quantizer is based cosine similarity and updated using exponential moving average (EMA).
C.7 Architecture Details for ImageNet
For experiments on ImageNet, the settings are similar but we only have CNN-based backbone with 8. We follow the RQVAE architecture with the following configuration:
CNN-based (ImageNet):: We utilize deep convolutional encoder-decoder architecture designed for high-resolution inputs. The encoder comprises stages, each with residual blocks, totaling 12 residual layers. A base channel size of 128 is used with channel multi-
pliers [1, 1, 2, 2, 4, 4), reaching up to 512 channels in deeper layers. The spatial resolution is reduced by factor of 8. The encoder output is projected to 256-dimensional latent
26

---

space. A commitment loss with weight 0.25 is applied to encourage consistency between the encoder output and the quantized representation.
C.8 GPU usage
Table 6 summarizes the GPU usage for the experiment in Section 3. The GPUs we use were A100 80GB.
Sweep Name
Dataset & Model
GPU Days
sweep hal___eeeba__eeeeeeeeeeeeeeeeeee cmn cmn_f=4 f=4
CelebA CNN
47
sweep_celcba__it vit f=4
CelebA VIT
87
sweep_ sweep_cifar_ent ciffrrccfnnnanc f=4
CIFAR-10 CNN
15
@weep_cifart vit f=4
CIFAR-10 VIT
19
sweep celeba cmn f=16
CelebA CNN
34
sweep_celeba_vite vit f=16
CelebA VIT
32
sweep_cifar_cnn_f== 1=66
CIFAR-10 CNN
15
sweep_cifar_vit_f-16
CIFAR-10 + ViT
17
Table 6: GPU time (in GPU-days) consumed by each sweep. Each sweep includes 64 individual runs, totaling 512 runs across all configurations. The sweeps vary by dataset (CelebA or CIFAR-10), model type (CNN VIT), and downsampling factor (f = 4 or f 16).
D Details of the rank regularizers
Table 7 summarizes the regularizers in Table
27

---

Table 7: Rank-promoting regularization methods applied to encoder outputs. All operate 62- normalized features æ E nR where n the batch size and is the feature dimension. We denote by (I..;) the feature-wise standard deviation of by C(x:) the covariance matrix of x, and by (A: ) the singular values obtained from its singular value decomposition (SVD). Batch-noue natch-normalized is denoted as BN
Method
Description
Objective
KoLeo
Encourages uniform spreading of features latent space by maximiz- ing pairwise distances.
log min || xzy jzz
14
Barlow
Promotes feature decorrelation by penalizing off-ddagonna covariance and enforcing unit variance.
(C(a )et - -1)2 +
511
Σ
C(x BN
1006
Twins 225
VICReg []
Maintains feature variance while reducing covssccovrri between dimensions.
max (0, 1 o(x.i)
1
(x))
25d
Spectrum Hinge
Prevents small singular values by applying hinge threshold
max(0, 7 A;)
E x² 0, 0.99
Rank Hinge
Enforces minimum effective rank r by penalizing when leading (r 1) components explain over 99% of variance.
max
Σイ x
28

---

E Additional results on Pearson correlations with effective dimensionality
In Table of the main text, report the average Pearson correlation between effective dimensionality and various hyperparameters, aggregated over different background dimensionalities. Figure T2 provides more detailed breakdown of these correlations across individual datasets, architecture, scale factors, and background dimensions.
To further disentangle the effect of the Rotation Trick, Tables 8 a 9 analysis separately for cases where the Rotation Trick is enabled and disabled, respectively. The results confirm that the overall trends remain consistent.
report the same correlation
Table 8: Correlation coefficients between hyperparameters and model performance when the Rotation
Trick is enabled.
Hyperparameter
6
8
16
32
64
128
256
Avg.
Commitment Loss Weight
-0.67
-0.82
-0.93
-0.84
-0.69
-0.50
-0.53
-0.71
Codebook Size
0.73
0.20
0.15
0.55
0.18
0.77
0.53
0.45
Code Restart Threshold
-0.23
-0.43
-0.22
-0.07
0.04
-0.20
-0.03
-0.14
EMA Decay
0.13
0.21
-0.03
0.17
-0.03
-0.06
0.12
0.06
Table 9: Correlation coefficients between hyperparameters and model performance when the Rotation
Trick is disabled.
Hyperparameter
8
16
32
64
128
256
Avg.
Commitment Loss Weight
0.14
-0.44
-0.79
-0.69
-0.75
-0.56
-0.73
-0.57
Codebook Size
0.69
0.17
0.29
0.48
0.38
0.76
0.81
0.52
Code Restart Threshold
0.13
0.33
0.05
0.08
-0.01
-0.21
-0.15
0.02
EMA Decay
-0.17
0.39
0.06
-0.11
-0.05
0.24
0.22
0.08
29

---

Correlation of Codebook Size with Effective Dim
Average correlation: r= 0.42
CelebA, CNN, 4=4
ColebA, VIT, f=4
CelebA, CNN, f-16
CelebA, VIT, f=16
10.0
Effective Dim
Effectivo 10
Effective
20
Effective
20
5.0
20
10
2.5
53
+
5/
30
5.5
3
3
3
+
3
Background
Background Dim
Background Dim
Background CIFARIO.
CIFARIO, CNN,
CIFARIO, VIT,
CIFARIO,
f=16
Dim
7003
20
Effartive Dim
20
Effertive Dim
40
ti
Effertive
20
10
10
10
4
256
4
30 4 0 4
Background Dim
Background Dim
Background Dim
Background Dim
Codebook Size
512
1024
2048
8192
Correlation of EMA Decay with Effective Dim
Average correlation: r= = 0.03
CelebA, CNN, 4=4
CelebA, VIT, f=4
CelebA, CNN, f-16
CelebA, VIT, f=16
10.0
Effactive Dim
7.5
Effective
Effective
20
5.0
10
2.5
53.
0/
4
256
53
6/
256
53
0.
256
00
256
Background
Background
Background
Background
CIFAR1O, CNN. fm4
CIFAR1O, VIT,
CIFARIO,
f=16
CIFARIO. f=16
40
Effoctive Dim
15
Effective
Effoctive
Effective
20
10
4+
4
Background Dim
Background Dim
Background Dim
Background Dim
EMA Decay
0.95
0.98
Correlation of Code Restart Threshold with Effective Dim
Average correlation: r == -0.02
CelebA, CNN, fm4
CelebA, VIT, fw4
CelebA, CNN. f-16
CalebA, VIT. f=16
10.0
Effective Dim
Efferctive Dim
Dim
7.5
20
Effective
5.0
10
2.5
53
0/
30
53
06
3/
250
53
0.
36
230
33 06
30
238
Background Dim
Background Dim
Background Dim
Background Dim
CIFAR1O, CNN,
CIFARIO, VIT,
CIFARID, CNN,
CIFAR1O,
20
40
15
Effective
20
Effective
10
Effective
10
Effective
10
60
3
3/
4
6
256
1/
6
8
Background Dim
Background Dim
Background Dim
Background Dim
Code Restart Threshold
0.00
00000
0.032
0.125
0.5
Figure 12: Correlation between effective dimensionality and various hyperparameters (part 1).
30

---

Correlation Commitment Loss Weight with Effective Dim
Average correlation: r -0.60
CelebA, CNN, f=4
CelebA, VIT, 4=4
CelebA, CNN, f-16
CelebA, VIT, f=16
ta
Effective Dim
Effectuc Dim
7.5
10
Effective
20
Effective
20
5.0
10
2.5
6m
55 66
99.60
Background Dim
Background Dim
Background Dim
Background Dim
CIFAR1O, CNN, fm4
CIFAR10, ViT, f=4
CIFAR10, CNN, f=
CIFAR1O, VIT, f-16
Effective Dim
20
15
25
Effective
Effective
20
10
10
53 6/b
+
0/
30
4
256
59
6.6
3/
4+
258
6.
36
4
5
Background Dim
Background Dim
Background Dim
Background Dim
Commitment Loss Weight
1.5
0.2
0.8
2.0
3.0
4.0
5.0
6.0
Correlation of Rotation Trick with Effective Dim
Average correlation: 0.26
CalebA,
fw4
CalebA CNN.
CalebA, VIT,
Effective Dim
Effactiv Dim
7.5
10
Effactive
20
Effactive
20
10
2.5
53
+
53.09 3/
4
30
4t
258
Background Dim
Background Dim
Background Dim
Background Dim
CIFAR10, CNN, fm4
CIFAR10, VIT, fm4
CIFAR10, CNN, f=16
CIFAR10, VIT, f-16
40
Effoctive Dim
15
Effoctive Dim
20
Effective Dim
15
20
10
20
33 0/
23
0/
+
12t
246
5/60
3. 0.
Background Dim
Background Dim
Background Dim
Background Dim
Rotation Trick
Disabled
Enabled
Figure 12 (continued): Additional correlation plots.
31

---

Computational cost of DCVQ
To assess the computational overhead of DCVQ, we compare its training time with that of standard VQVAE under identical conditions. All models are trained on CIFAR-10 10 using the same CNN architecture for 100 epochs on a single NVIDIA A GPU. DCVQ, each subspace quantizer operates independently, allowing for efficient parallel processing. The results are summarized in Table I0
Table 10:
Comparison of training time between DCVQ and vanilla VQVAE CIFAR-10. The total latent dimensionality is matched in each pair of experiments.
Total dim
Type
Dim per quantizer
Num. quantizers
Training time
32
VQVAE
5h 34m 41s
32
DCVQ
8
4
5h 34m 35s
64
VQVAE
5h 40m 41s
64
DCVQ
8
8
5h 40m 17s
256
VQVAE
6h 17m 57s
256
DCVQ
32
8
6h 13m 39s
The training times of DCVQ and vanilla VQVAE are nearly identical across all settings when the total latent dimensionality is matched. This demonstrates that the additional quantization operations introduced by DCVQ incur negligible computational overhead.
32

---

G Synthetic Experiment Demonstrating Quantization Bias
The following PyTorch code reproduces the synthetic k-means experiment shown in Figure
import torch
import matplotlib pyplot plt
from sklearn cluster import KMeans
def gen_data (N, D, p):
eigvals (10 ** torch randn (D)) pow (p). sort (descending- [0]
U,
torch. linalg avd (torch randn (N, D), full_matrices-fall
= U a torch. diag sorochhssrt(()))) 1) eigvals)) a Vt
return x, eigvals
11
def centroid_eigvals (x, n_clusters=512):
12
kmeans KMeans (L.chusters=n randon_state=0 n_init=10) kmeans. fit (X. mumpy
- torch. cov (torch. tensor (kmeans wwweeaaeeeeasssseeee.eeeo... cluster.centers...r
return torch. tensor (kmeans centers.,, torch winall.nnn..... (cov [[]]
16
torch. manual seed (0)
N. D. 4096, 64, 512
x, target_eigs gen_data (N, D, p-0.3)
20
centroids, cengroid_eigs eigvalid n_clusters=K) ratio centroid_eigs tagg
21
fig, axs plt. subplota 2, figsize=(10, 4))
24
25
axs [0] plot (ratio numpy O, ?o-", "alor=/tabbbuue" markersize=4)
U, 8, Vt torch linalg (X, full mull_matrices-Pals))
28
x_2d
Vt[[o, -1] :l.T
29
C_2d centroids Vt[lo, -11, :].T
30
axs [1] all.scatterr... (:_ 01, x_2d[; 11, 2=5, alpha=0.3, color='tab gray
label='Data')
[1] acatter (c_ ol, c_2a(: 11, ==30, color='tab:res label=) Centroids
32
plt. show
Code Synthetic experiment showing eigenvalue suppression under k-means quantization.
33


Reviews:
[{"review_id": "nHRPYIBKSg", "reviewer": "Reviewer_enVD", "summary": "This paper revolves around VQ-VAE (and vector quantization overall), and centers around the claim that such models hardly use the full size of their code vectors: the linear dimension for such trained models is incredibly small compared to the feature dimension. When trying to regularize for higher rank features, performance drops off notably. The authors then offer a fix that applies to all VQ models: splitting into subspaces, then quantizing at each individually low-dimensional space. This keeps VQ in a low-dimensional regime while allowing for higher model capacity.", "questions": "1. Why does the KoLeo regularization give a strong statement regarding restriction to a higher rank set for optimization? That is, why are we not concerned that performance degredation could be due to the effect of this regularizer's gradient structure on optimization?\n2. What other methods for rank maximization/thresholding would be viable to test against as well?\n3. What are the visual effects of more/less subspaces, and higher/lower dimensions for each of the subspaces?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper generally is very readable: the claim is easy to understand, and the experiments are controlled and interpretable. The proposed fix is simple and intuitive to the proposed issue, and demonstrates notable performance increase on the demonstrated experiments.\n\nI have some curiosities around the experiments and the drawn conclusions. Most notably, the claim that \"higher rank results in worse performance\" would need more diverse experiments than training on a single regularization term. It is unclear to me why model performance can be disentangled from potential learning instability from the regularizer, or if this results from the loss \"focusing too much\" on the regularizer (instead of e.g. using a hinge loss, where features are thresholded to some minimum rank). If the authors have an explanation for why this is not a concern, or if there are alternative methods for rank promotion that can be tried which still degrade performance, this will greatly increase the strength of their claim given the lack of theoretical evidence. Since this is a more fundamental VQ fix rather than a specific CV application, it's more excusable that experiments are restricted to now older datasets like CIFAR10 and CelebA, but in the absence of theory, it would be notably strengthened with more modern examples (e.g. ImageNet).", "was_revised": true, "final_justification": "The main concern was around experimental verification of the rank claim, which is now much more robust. I have raised my score accordingly.", "rebuttal": "# Response to Reviewer enVD\nWe greatly appreciate the reviewer’s thorough evaluation and insightful feedback. We address each of your points in detail below.\n\n## Response to Q1\n\nWe acknowledge the concern that KoLeo’s performance drop may not reflect true higher-rank incompatibility. To test this hypothesis, we conducted experiments with multiple alternative rank-promoting methods: Barlow Twins, VICReg, SingHinge (our hinge loss on singular values of covariance), and RankHinge (as suggested by the reviewer). All regularizers were applied on the unquantized latents. Due to the time limit, we only tested the CNN architecture with $f=4$. Detailed implementation is provided later in this response.\n\n**Table 1: CIFAR10 performance with different regularizers**\n\n| Regularizer Type|   Regularizer Weight |   Val Loss  |   Effective Dim  |\n|-----------------|--------------|-------------|------------------|\n| None|  -   |   0.081 |9 |\n| KoLeo   |   0.001  |   0.081 |9 |\n| KoLeo   |   0.01   |   0.082 |   10 |\n| KoLeo   |   0.1|   0.106 |   27 |\n| BarlowTwins |   0.0001 |   0.081 |   10 |\n| BarlowTwins |   0.001  |   0.093 |   31 |\n| BarlowTwins |   0.01   |   0.105 |   78 |\n| BarlowTwins |   0.1|   0.087 |   49 |\n| VIC |   0.1|   0.081 |9 |\n| VIC |   1  |   0.080 |   10 |\n| VIC |  10  |   0.082 |   11 |\n| VIC | 100  |   0.085 |   12 |\n| SingHinge(0.01) |   0.1|   0.080 |9 |\n| SingHinge(0.01) |   1  |   0.081 |9 |\n| SingHinge(0.01) |  10  |   0.089 |   49 |\n| SingHinge(0.01) | 100  |   0.129 |  111 |\n| SingHinge(0.1)  |   0.1|   0.080 |9 |\n| SingHinge(0.1)  |   1  |   0.081 |   10 |\n| SingHinge(0.1)  |  10  |   0.084 |   11 |\n| SingHinge(0.1)  | 100  |   0.085 |   12 |\n| RankHinge(64)   |   0.001  |   0.081 |9 |\n| RankHinge(64)   |   0.01   |   0.081 |9 |\n| RankHinge(64)   |   0.1|   0.080 |9 |\n| RankHinge(64)   |   1  |   0.081 |9 |\n| RankHinge(128)  |   0.001  |   0.080 |9 |\n| RankHinge(128)  |   0.01   |   0.082 |9 |\n| RankHinge(128)  |   0.1|   0.080 |9 |\n| RankHinge(128)  |   1  |   0.081 |9 |\n\n(Val Loss reports validation reconstruction loss only, excluding the regularization term. SingHinge($h$) applies a hinge loss on singular values with threshold $h$ (e.g., 0.01 or 0.1). RankHinge($r$) applies a hinge loss on feature rank with threshold $r$ (e.g., 64 or 128). Regularization weights were chosen to yield performance comparable to the no-regularization baseline.)\n\nFrom these results, we find a consistent trend: increasing the strength of KoLeo, Barlow Twins, VICReg, or SingHinge increases the effective dimension but also degrades performance once it grows beyond the baseline level (≈9). This pattern holds across very different gradient structures, suggesting the effect is not an artifact of KoLeo but a fundamental property of the VQ quantization bottleneck.\n\nRankHinge is a special case. While the codebook effective dimension remains flat (~9) regardless of weight, we want to emphasize that this is not due to insufficient hyperparameter tuning: the unquantized latents show clear growth in dimensionality, reaching levels similar to KoLeo (shown below). Even with regularization weights up to 1000, the effective dimension stays at 9 while validation loss worsens (not shown), underscoring a regularizer limitation rather than a tuning issue.\n\n**Table 2: RankHinge regularization effective dimension vs. unquantized effective dimension**\n| Regularizer Type   |   Regularizer Weight |   Val Loss  |   Effective Dim  |   Effective Dim (before Quant) |\n|----------------|--------------|-------------|------------------|--------------------------------|\n| None    | - |0.081 |  9 | 27 |\n| KoLeo   | 0.001 |0.081 |  9 | 19 |\n| KoLeo   | 0.01  |0.082 | 10 | 76 |\n| KoLeo   | 0.1   |0.106 | 27 |123 |\n| RankHinge(64)  | 0.001 |0.081 |  9 | 17 |\n| RankHinge(64)  | 0.01  |0.081 |  9 | 33 |\n| RankHinge(64)  | 0.1   |0.08  |  9 | 52 |\n| RankHinge(64)  | 1     |0.081 |  9 | 54 |\n| RankHinge(128) | 0.001 |0.08  |  9 | 20 |\n| RankHinge(128) | 0.01  |0.082 |  9 | 15 |\n| RankHinge(128) | 0.1   |0.08  |  9 |128 |\n| RankHinge(128) | 1     |0.081 |  9 |133 |\n\n\n## Response to \"it would be notably strengthened with more modern examples (e.g. ImageNet)\"\nWe appreciate the reviewer’s suggestion to include more modern datasets. We wanted to clarify that Section 5.2 includes experiments on ImageNet for our method and RQVAE. Here, we add comparison with vanilla VQVAEs and other regularizers on ImageNet:\n\n**Table 3: ImageNet-1k performance with different methods**\n| Type|   Regularizer Weight |   Val Loss  |   Effective Dim  |\n|-----------------|--------------|-------------|------------------|\n| BarlowTwins |   0.0001 |   0.173 |   14 |\n| BarlowTwins |   0.001  |   0.186 |   41 |\n| BarlowTwins |   0.01   |   0.205 |   91 |\n| BarlowTwins |   0.1|   0.182 |   58 |\n| VIC |   0.1|   0.172 |9 |\n| VIC |   1  |   0.164 |   11 |\n| VIC |  10  |   0.169 |   13 |\n| VIC | 100  |   0.18  |   20 |\n| SingHinge(0.01) |   0.1|   0.158 |9 |\n| SingHinge(0.01) |   1  |   0.168 |   11 |\n| SingHinge(0.01) |  10  |   0.181 |   63 |\n| SingHinge(0.01) | 100  |   0.247 |  137 |\n| SingHinge(0.1)  |   1  |   0.171 |   11 |\n| SingHinge(0.1)  |  10  |   0.183 |   15 |\n| SingHinge(0.1)  | 100  |   0.186 |   17 |\n| RankHinge(64)   |   0.001  |   0.173 |9 |\n| RankHinge(64)   |   0.01   |   0.173 |9 |\n| RankHinge(64)   |   0.1|   0.173 |9 |\n| RankHinge(64)   |   1  |   0.173 |9 |\n| RankHinge(128)  |   0.001  |   0.166 |9 |\n| RankHinge(128)  |   0.01   |   0.163 |9 |\n| RankHinge(128)  |   0.1|   0.168 |9 |\n| RankHinge(128)  |   1  |   0.166 |9 |\n| DCVQ(4x8d)  |   - |   0.115 |   20 |\n| DCVQ(8x8d)  |   - |   0.105 |   36 |\n\nSimilar to CIFAR10, we observe that increasing the regularization strength leads to higher effective dimensions but also worse validation loss. RankHinge maintains a low effective dimension while showing no performance degradation.\n\nPlease notice that the results presented here are of a different setting compared to Section 5.2. Here, we didn't train the models to converge due to time constraints. Instead, we stopped training after 21k steps, at which point the effective dimensions had stabilized. Another difference is the use of a smaller parameter size. We plan to include full convergence results in the camera-ready version.\n\n## Response to Q2\n\nIn the above tables, we explored a diverse set of regularization strategies, which can be grouped into three categories:\n- Embedding spreading (e.g., KoLeo): Encourages the embeddings to be spread out in the latent space.\n- Redundancy reduction methods (e.g., Barlow Twins, VICReg): These encourage feature decorrelation and variance, commonly used in self-supervised learning.\n- Covariance spectrum shaping (e.g., SingHinge): These directly regularize the singular values of the feature covariance matrix to promote higher rank.\n- Rank thresholding (e.g., RankHinge): Inspired by the reviewer’s suggestion, this penalizes low-rank representations only when effective dimension falls below a target threshold.\n\n\n## Response to Q3\n\nWe observe that using more subspaces generally helps preserve high-frequency details in the reconstructions (e.g., hair). However, increasing the dimensionality of each subspace beyond our recommended 8 dimensions often harms these details. Due to rebuttal constraints, we are unable to upload visual examples at this stage, but we will include detailed qualitative comparisons and reconstructions in the camera-ready version.\n\n## Implementation Details for Regularizers\n\nFor **Barlow Twins** and **VICReg**, which were originally designed for contrastive learning with augmented view pairs, data augmentation is not needed in our rank-promotion setting. Instead, we form input pairs by duplicating the same latent vector. All other hyperparameters follow the original codebase.\n\n### Barlow Twins Loss\n\nGiven normalized latent vectors $ x, y \\in \\mathbb{R}^{B \\times D} $, where $B$ denotes the total number of patches across GPUs and $D$ the feature dimension, and letting $\\text{BN}$ denote 1D batch normalization, the Barlow Twins loss is defined as:\n\n$$\nC = \\frac{1}{B} \\, \\text{BN}(x)^\\top \\text{BN}(y) \\\\\\\\\n\\mathcal{L}_{\\text{Barlow}} = \\sum_i (C_{ii} - 1)^2 + 0.0051 \\sum_{i \\neq j} C_{ij}^2\n$$\n\n### VICReg Loss\n\nGiven $x, y, B, D$ defined similarly, the VICReg loss combines variance and covariance regularization:\n\nThe variance term:\n\n$$\n\\mathcal{L}_{\\text{var}} \n= \\frac{1}{D} \\sum_{i=1}^D \\frac{\\max(0, 1 - \\sigma(x_{\\cdot i})) + \\max(0, 1 - \\sigma(y_{\\cdot i}))}{2}\n$$\n\nwhere $\\sigma$ denotes the standard deviation.\n\nThe covariance term:\n\n$$\n\\mathcal{L}_{\\text{cov}} = \\frac{1}{D} \\sum_{i \\neq j} (\\text{Cov}(x)_{ij}^2 + \\text{Cov}(y)_{ij}^2)\n$$\n\nwith $\\text{Cov}(x) = \\tfrac{1}{B-1}(x - \\mu(x))^\\top(x - \\mu(x))$, where $\\mu(x)$ is the batch mean.\n\nThe total VICReg loss is then:\n\n$$\n\\mathcal{L}_{\\text{VICReg}} = \\mathcal{L}_{\\text{var}} + \\tfrac{1}{25} \\cdot \\mathcal{L}_{\\text{cov}}\n$$\n\n### SingHinge\n\nFor latent representations $z \\in \\mathbb{R}^{B \\times D}$, let $s_i$ denote the singular values of $\\text{Cov}(z)$. With a threshold $h$, the SingHinge loss is defined as\n\n$$\n\\mathcal{L}_{\\text{SingHinge}(h)} = \\frac{1}{D} \\sum_i \\max(0, h - s_i)\n$$\n\n### RankHinge\n\nThe intuition behind RankHinge is to ensure that at least $r$ singular values are necessary to capture 99% of the variance. To achieve this, we penalize the case where the first $r-1$ singular values already explain $\\geq 99\\%$ of the variance. Formally, let\n\n$$\nE_{r-1} = \\sum_{i=1}^{r-1} \\frac{s_i}{\\sum_j s_j}\n$$\n\ndenote the cumulative variance explained by the top $r-1$ singular values. The RankHinge loss is then defined as\n\n$$\n\\mathcal{L}_{\\text{RankHinge}(r)} = \\max(0,\\, E_{r-1} - 0.99)\n$$\n\n**References:**\n\n[1] Barlow Twins: Self-Supervised Learning via Redundancy Reduction\n\n[2] VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning"}, {"review_id": "JEEXRoPL2v", "reviewer": "Reviewer_TLST", "summary": "This paper identifies a previously unrecognized issue in vector quantized variational autoencoders (VQVAEs), termed dimensional collapse, wherein the learned codebooks tend to collapse onto a limited number of principal components, even when the dimensionality of the code vectors is large. The authors support their findings through extensive experiments, showing that the phenomenon arises from the quantization step during the early training phase. To address this issue, the latent representation is divided into smaller vectors, each of which is independently quantized. This technique enhances both the effective dimensionality of the codebooks and the overall reconstruction quality.", "questions": "1. The detailed explanation of the projection layer following the encoder may help clarify the architectures. Is this layer also trained during the training?\n2. In Figure 9, the compared models employ multiple codebooks. Could the authors clarify how the effective dimension was computed in this multi-codebook setting?\n3. In the experimental results shown in Figures 4 and 5, the effective dimension remained below 20 even when hyperparameters such as $f$ and background dimension were varied. However, in Figure 9, effective dimensions around 100 were reported. Could the authors briefly explain which aspect of the proposed method leads to this discrepancy?\n4. While I agree that training dynamics of the quantizers may cause dimension collapse, I am curious about the authors’ thoughts on why this phenomenon occurs. Since the codebook is updated as cluster centers of the encoding vectors, implying that encoding vectors are mapped near a low-dimensional subspace, any hypothesized reasoning behind this behavior would be very interesting.\n5. I am also curious about the potential relationship between the effective dimension and the intrinsic dimension of the data. Could the authors comment on this?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. The writing is well-written, with clear and easy-to-follow logical flow.\n2. The authors effectively design experiments that validate their hypotheses, and their results provide clear evidence, supported by sound interpretation.\n3. The authors propose a simple yet effective method to address the identified issue.\n\n### Weaknesses\n1. The encoder scale factor $f$ controls the number of encoding vectors and likely influences the expressive capacity of the latent representation. It may also be related to the effective dimension of the model. Although the experiments include comparisons with $f=4$ and $f=16$, the main text does not provide a detailed interpretation of how this parameter affects the results. Including such an analysis would help clarify the role of $f$ and its relationship to the model's representational capacity.\n2. The proposed method shows improvements in reconstruction quality, but does not discuss the generation procedure or the quality of generated samples. Including such analysis would help assess the model as a generative model.\n3. I would like to offer a few comments regarding the figures and tables, which may help improve the clarity and presentation of the results.\n    1. In Figure 4, the size of the circles is not clearly distinguishable, which makes it difficult to interpret the relationships among background dimension, effective dimension, and validation loss.\n    2. In Figure 5, using a logarithmic scale on the x-axis may improve readability. Also, it would be helpful to clarify what the colors represent.\n    3. The notation $d_{\\text{eff}}$​ is defined in the text, but it does not appear in the figures or results. Referring to it explicitly in the relevant graphs would help improve clarity. Similarly, in Table 1, it would be helpful to indicate how each hyperparameter corresponds to components in the formulation presented in Section 2.1.\n    4. In Figure 7, the legend or caption does not explain the meaning of the different line styles (e.g., circle line for validation loss, square line for effective dimension). Providing this information would help readers interpret the figure correctly.\n    5. For completeness and consistency, it may be useful to test the model under the case $f = 4$ as well.\n    6. In Figure 9, it is unclear which parameter(s) of RQVAE were varied to generate the graph, as this is not described in the main text. Additionally, since DCVQ can be interpreted as increasing the codebook size and decreasing $f$ (which corresponds to increasing the number of encoding vectors), it would be beneficial to include the standard RQVAE with these settings as a baseline for comparison.\n    7. In Table 2, reporting regression coefficients instead of correlations may better reveal the influence of each latent variable. For interpretability, it may also be helpful to exclude the binary variable associated with the rotation trick when fitting the regression model.\n4. Some additions to the manuscript could improve clarity and completeness.\n    1. Table 1 includes terms such as “rotation trick” and “dead-code restart threshold,” but these are not explained in the main text. Providing brief explanations would help make the paper more self-contained.\n    2. Including training details such as the optimizer, learning rate, batch size, and number of epochs (perhaps in an appendix) would help improve the reproducibility of the experiments.", "was_revised": true, "final_justification": "The authors have addressed most of my initial concerns and provided compelling empirical evidence relevant to my question. The identified issue in this paper has the potential to be influential in this field, and its validity is supported by well-designed experiments.", "rebuttal": "# Response to Reviewer TLST\n\nWe are truly grateful to the reviewer for taking the time to carefully read our work and provide such constructive suggestions. Your feedback has been instrumental in refining our manuscript. Below, we respond to each of your comments and questions.\n\n## Response to Weakness 1\n\nWe omitted detailed comparisons earlier because $f$ strongly affects validation loss, making figures hard to interpret. Table 1 now reports the values from Figure 7, grouped by scale factor $f$. Results show that effective dimension is not systematically influenced by $f$, while validation loss is. Under KoLeo, effective dimension varies with $f$, but without a consistent trend—likely due to regularization-induced instability.\n\n**Table 1. Comparison of different $f$**\n| Dataset   | Architecture   |   Val Loss |   Effective Dim |   KoLeo loss weight |\n|:----------|:---------------|-----------:|----------------:|----------:|\n| CIFAR10   | CNN(f=16)  |  0.480 |   8 | 0 |\n| CIFAR10   | CNN(f=4)   |  0.081 |   9 | 0 |\n| CIFAR10   | ViT(f=16)  |  0.480 |   9 | 0 |\n| CIFAR10   | ViT(f=4)   |  0.084 |   9 | 0 |\n| CIFAR10   | CNN(f=16)  |  0.482 |   9 | 0.001 |\n| CIFAR10   | CNN(f=4)   |  0.081 |   9 | 0.001 |\n| CIFAR10   | ViT(f=16)  |  0.479 |   9 | 0.001 |\n| CIFAR10   | ViT(f=4)   |  0.084 |   9 | 0.001 |\n| CIFAR10   | CNN(f=16)  |  0.492 |  10 | 0.01  |\n| CIFAR10   | CNN(f=4)   |  0.082 |  10 | 0.01  |\n| CIFAR10   | ViT(f=16)  |  0.480 |   9 | 0.01  |\n| CIFAR10   | ViT(f=4)   |  0.084 |   9 | 0.01  |\n| CIFAR10   | CNN(f=16)  |  0.523 |  15 | 0.1   |\n| CIFAR10   | CNN(f=4)   |  0.106 |  27 | 0.1   |\n| CIFAR10   | ViT(f=16)  |  0.494 |  11 | 0.1   |\n| CIFAR10   | ViT(f=4)   |  0.090 |  12 | 0.1   |\n| CelebA| CNN(f=16)  |  0.033 |   7 | 0 |\n| CelebA| CNN(f=4)   |  0.003 |   7 | 0 |\n| CelebA| ViT(f=16)  |  0.034 |  10 | 0 |\n| CelebA| ViT(f=4)   |  0.004 |   9 | 0 |\n| CelebA| CNN(f=16)  |  0.033 |   7 | 0.001 |\n| CelebA| CNN(f=4)   |  0.003 |   7 | 0.001 |\n| CelebA| ViT(f=16)  |  0.034 |  11 | 0.001 |\n| CelebA| ViT(f=4)   |  0.004 |  10 | 0.001 |\n| CelebA| CNN(f=16)  |  0.034 |   9 | 0.01  |\n| CelebA| CNN(f=4)   |  0.003 |   8 | 0.01  |\n| CelebA| ViT(f=16)  |  0.036 |  14 | 0.01  |\n| CelebA| ViT(f=4)   |  0.004 |  11 | 0.01  |\n| CelebA| CNN(f=16)  |  0.043 |  41 | 0.1   |\n| CelebA| CNN(f=4)   |  0.005 |  84 | 0.1   |\n| CelebA| ViT(f=16)  |  0.038 |  60 | 0.1   |\n| CelebA| ViT(f=4)   |  0.007 |  37 | 0.1   |\n\n## Response to Weakness 2\n\nWe thank the reviewer for this valuable comment. We agree that evaluating the generation procedure and sample quality is essential. These experiments are ongoing, and we plan to include results soon.\n\n## Response to Weakness 3\n\nWe thank the reviewer for the detailed feedback on our figures and tables.\n1. Fig. 4: We will improve circle size visibility in the revision.\n2. Fig. 5: All points are plotted with the same color; the perceived differences arise from overlapping points. We will revise the plotting style to eliminate this artifact.\n3. Notation / Table 1: We apologize for the oversight. The effective dimension $d_eff$ was denoted as Effective Dim in the figures. We will revise 2.1 to make it clearer.\n4. Fig. 7: We will remove line styles, as colors suffice.\n5. $f=4$ (Fig. 9): Omitted due to space, but we will add it.\n6. Fig. 9 (10?) (RQVAE parameters): The varied parameter was the number of quantizers; we will revise the caption/description. We also plan to include the standard VQVAE as a baseline.\n7. Table 2:\n (a) We use correlations instead of regression coefficients since hyperparameters vary in scale (e.g., EMA decay vs. codebook size), making regression coefficients non‑comparable; correlations are scale‑independent.\n (b) We agree the binary “Rotation Trick” should be excluded and will present correlations separately for enabled/disabled cases. Please check the updated Table 2.1 and Table 2.2. Conclusions remain unchanged.\n\n**Table 2.1 Correlation coefficients when Rotation Trick is enabled**\n| Hyperparameter | Background Dim   |   Correlation |\n|:-----------------------|:-----------------|--------------:|\n| Commitment Loss Weight | 6| -0.67 |\n| Commitment Loss Weight | 8| -0.82 |\n| Commitment Loss Weight | 16   | -0.93 |\n| Commitment Loss Weight | 32   | -0.84 |\n| Commitment Loss Weight | 64   | -0.69 |\n| Commitment Loss Weight | 128  | -0.50 |\n| Commitment Loss Weight | 256  | -0.53 |\n| Commitment Loss Weight | Average  | -0.71 |\n| Codebook Size  | 6|  0.73 |\n| Codebook Size  | 8|  0.20 |\n| Codebook Size  | 16   |  0.15 |\n| Codebook Size  | 32   |  0.55 |\n| Codebook Size  | 64   |  0.18 |\n| Codebook Size  | 128  |  0.77 |\n| Codebook Size  | 256  |  0.53 |\n| Codebook Size  | Average  |  0.45 |\n| Code Restart Threshold | 6| -0.23 |\n| Code Restart Threshold | 8| -0.43 |\n| Code Restart Threshold | 16   | -0.22 |\n| Code Restart Threshold | 32   | -0.07 |\n| Code Restart Threshold | 64   |  0.04 |\n| Code Restart Threshold | 128  | -0.20 |\n| Code Restart Threshold | 256  | -0.03 |\n| Code Restart Threshold | Average  | -0.14 |\n| EMA Decay  | 6|  0.13 |\n| EMA Decay  | 8|  0.21 |\n| EMA Decay  | 16   | -0.03 |\n| EMA Decay  | 32   |  0.17 |\n| EMA Decay  | 64   | -0.03 |\n| EMA Decay  | 128  | -0.06 |\n| EMA Decay  | 256  |  0.12 |\n| EMA Decay  | Average  |  0.06 |\n\n**Table 2.2 Correlation coefficients when Rotation Trick is disabled**\n| Hyperparameter | Background Dim   |   Correlation |\n|:-----------------------|:-----------------|--------------:|\n| Commitment Loss Weight | 6|  0.14 |\n| Commitment Loss Weight | 8| -0.44 |\n| Commitment Loss Weight | 16   | -0.79 |\n| Commitment Loss Weight | 32   | -0.69 |\n| Commitment Loss Weight | 64   | -0.75 |\n| Commitment Loss Weight | 128  | -0.56 |\n| Commitment Loss Weight | 256  | -0.73 |\n| Commitment Loss Weight | Average  | -0.57 |\n| Codebook Size  | 6|  0.69 |\n| Codebook Size  | 8|  0.17 |\n| Codebook Size  | 16   |  0.29 |\n| Codebook Size  | 32   |  0.48 |\n| Codebook Size  | 64   |  0.38 |\n| Codebook Size  | 128  |  0.76 |\n| Codebook Size  | 256  |  0.81 |\n| Codebook Size  | Average  |  0.52 |\n| Code Restart Threshold | 6|  0.13 |\n| Code Restart Threshold | 8|  0.33 |\n| Code Restart Threshold | 16   |  0.05 |\n| Code Restart Threshold | 32   |  0.08 |\n| Code Restart Threshold | 64   | -0.01 |\n| Code Restart Threshold | 128  | -0.21 |\n| Code Restart Threshold | 256  | -0.15 |\n| Code Restart Threshold | Average  |  0.02 |\n| EMA Decay  | 6| -0.17 |\n| EMA Decay  | 8|  0.39 |\n| EMA Decay  | 16   |  0.06 |\n| EMA Decay  | 32   | -0.11 |\n| EMA Decay  | 64   | -0.05 |\n| EMA Decay  | 128  |  0.24 |\n| EMA Decay  | 256  |  0.22 |\n| EMA Decay  | Average  |  0.08 |\n\n## Response to Weakness 4\n1. The definitions of the rotation trick and dead‑code restart threshold are provided in Related Work (Appendix A). We will expand on these explanations in the revision.\n2. Training details are included in Appendix C.\n\n## Response to Questions\n1. The projection layer following the encoder is a trainable linear layer. This allows us to map the latents into arbitrary background dimensions. Similar tricks could be found in Improved VQGAN (RQVAE).\n2. For RQVAE, the final representation is obtained by summing multiple codes, while the representational space is determined by the set of codes prior to summation. We concatenate all code vectors into a joint matrix and compute its effective dimension. In contrast, our method defines the space as a direct sum of subspaces, so its effective dimension equals the sum of the effective dimensions of the individual subspaces.\n3. In Figure 9, effective dimensions of around 100 correspond to the DCVQ method. The figure illustrates how increasing the number of quantizers raises the effective dimension. For example, the notation $16 \\times 8d$ indicates 16 quantizers, each with dimensionality $8d$.\n4. Our hypothesis is that quantization biases towards high‑variance directions. Since quantization can be viewed as a variant of online K‑Means, we can illustrate this phenomenon in a controlled K‑Means experiment. Specifically, we generate synthetic data with a covariance matrix of desired eigenvalues, apply K‑Means clustering, and then examine the eigenvalues of the covariance of the cluster centroids. We find that the smaller eigenvalues shrink more severely after clustering, while the larger ones are preserved. As a result, the effective dimension of the centroids (i.e., the codebook) is reduced compared to that of the original data. The following snippet demonstrates this behavior. To develop a theoretical explanation for the rank‑vanishing behavior observed in K‑Means, one could connect the observations to existing frameworks, such as (1) Spectral Relaxation for K‑Means Clustering and (2) K‑Means Clustering via Principal Component Analysis. These studies suggest that K‑Means exhibits PCA‑like behavior, preferentially capturing the principal components of the data.\n5. We assume that the effective dimension is more like an artifact of the quantization process, since no such collapse is observed in standard autoencoders.\n\n```python\nimport torch\nimport matplotlib.pyplot as plt\nfrom sklearn.cluster import KMeans\n\ndef gen_data(N, D, p):\n    eigvals = (10 ** torch.randn(D)).pow(p).sort(descending=True)[0]\n    U, _, Vt = torch.linalg.svd(torch.randn(N, D), full_matrices=False)\n    X = U @ torch.diag(torch.sqrt((N - 1) * eigvals)) @ Vt\n    return X, eigvals\n\ndef centroid_eigvals(X, n_clusters=512):\n    kmeans = KMeans(n_clusters=n_clusters, random_state=0, n_init=10)\n    kmeans.fit(X.numpy())\n    cov = torch.cov(torch.tensor(kmeans.cluster_centers_).T)\n    return torch.linalg.svd(cov)[1]\n\ntorch.manual_seed(0)\n\nN, D, K = 4096, 64, 512\nX, target_eigs = gen_data(N, D, p=0.3)\ncentroid_eigs = centroid_eigvals(X, n_clusters=K)\nratio = centroid_eigs / target_eigs\n\nplt.plot(ratio.numpy(), 'o-')\nplt.axhline(1.0)\nplt.xlabel('Eigenvalue Index (Largest to Smallest)')\nplt.ylabel('Centroid / Data Eigenvalue')\nplt.yscale('log')\nplt.show()\n```"}, {"review_id": "KZ62ZJy809", "reviewer": "Reviewer_21RK", "summary": "The paper uncovers an interesting phenomenon where VQ-VAEs tend to compress their representations into a much smaller subspace, despite being trained with high-dimensional embeddings. Through a thorough analysis, the authors demonstrate how this compression relates to model performance and learning dynamics. To address the limitations imposed by this low-dimensional bottleneck, they propose the Divide-and-Conquer VQ (DCVQ) method, which partitions the latent space into multiple low-dimensional subspaces that are quantized independently. The combined effect of these subspaces significantly expands the overall representational capacity. Experimental results show that DCVQ effectively overcomes the dimensionality bottleneck and achieves improved reconstruction quality across various image datasets.", "questions": "It is necessary to include comparisons with VQGAN-FC (which performs dimensionality reduction and normalization), while also controlling for the same amount of information bits in the codebooks.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n- The identification of the dimension collapse problem in VQ-VAEs is insightful and worthy of further exploration.\n- The experiments and analyses are comprehensive and well-executed.\n\n### Weaknesses\n- I had hoped the authors would uncover the fundamental cause of dimension collapse and provide a thorough explanation, possibly from the perspective of optimization dynamics. Unfortunately, the paper only observes that the quantizer collapses first, which then causes the encoder to collapse, without deeper theoretical insight.\n- The proposed Divide-and-Conquer method still relies on low-dimensional codebook subspaces, making the approach fundamentally similar to previous dimensionality reduction techniques (e.g., Improved VQGAN), lacking a truly novel solution.", "was_revised": true, "final_justification": "This paper raises an important and novel issue but is not technically solid.", "rebuttal": "# Response to Reviewer 21RK\n\nWe sincerely thank the reviewer for the thoughtful review and for the positive assessment of our work. We greatly appreciate your constructive feedback, and we have carefully considered each of your comments in detail. Below we provide clarifications and additional context.\n\n## Response to \"I had hoped the authors would uncover the fundamental cause of dimension collapse\"\n\nWe appreciate this insightful question, as understanding the fundamental cause of dimension collapse is indeed central to advancing this line of work. While a full theoretical characterization remains an open challenge, our experiments provide evidence for a plausible underlying mechanism.\n\nAs discussed in the paper, the quantizer is the first module to collapse. We attribute this not to incidental factors but to an inherent bias of the quantization process. Specifically, VQ‑VAE’s quantization can be viewed as an online K‑Means procedure. A synthetic experiment can illustrate that K‑Means systematically reduces the effective dimension of the data: the learned centroids bias towards with the high‑variance directions of the data covariance matrix. Concretely, the ratio between centroid eigenvalues and data eigenvalues stays close to 1 for the largest eigenvalues but drops significantly for smaller ones. Thus, high‑variance components are well preserved, whereas low‑variance components are suppressed, leading to a reduced effective dimension in the codebook compared to the original data.\n\nThe behavior could be illustrated by the snippet below. The experiment constructs synthetic data with a controlled covariance spectrum, applies K‑Means clustering, and compares the eigenvalue decay before and after quantization. The results consistently show that smaller eigenvalues shrink more severely, confirming the bias towards high‑variance directions.\n\nOur findings are also supported by prior theoretical work linking K‑Means and PCA, including [1] Spectral Relaxation for K‑Means Clustering and [2] K‑Means Clustering via Principal Component Analysis. These works suggest that clustering inherently emphasizes dominant principal components, providing a theoretical lens through which to interpret the observed rank vanishing.\n\nWhile a rigorous proof remains an exciting direction for future research, we believe that the empirical evidence presented here, together with prior literature, strongly indicates that dimension collapse arises as a structural property of quantization.\n\n```python\nimport torch\nimport matplotlib.pyplot as plt\nfrom sklearn.cluster import KMeans\n\ndef gen_data(N, D, p):\n    eigvals = (10 ** torch.randn(D)).pow(p).sort(descending=True)[0]\n    U, _, Vt = torch.linalg.svd(torch.randn(N, D), full_matrices=False)\n    X = U @ torch.diag(torch.sqrt((N - 1) * eigvals)) @ Vt\n    return X, eigvals\n\ndef centroid_eigvals(X, n_clusters=512):\n    kmeans = KMeans(n_clusters=n_clusters, random_state=0, n_init=10)\n    kmeans.fit(X.numpy())\n    cov = torch.cov(torch.tensor(kmeans.cluster_centers_).T)\n    return torch.linalg.svd(cov)[1]\n\ntorch.manual_seed(0)\n\nN, D, K = 4096, 64, 512\nX, target_eigs = gen_data(N, D, p=0.3)\ncentroid_eigs = centroid_eigvals(X, n_clusters=K)\nratio = centroid_eigs / target_eigs\n\nplt.plot(ratio.numpy(), 'o-')\nplt.axhline(1.0)\nplt.xlabel('Eigenvalue Index (Largest to Smallest)')\nplt.ylabel('Centroid / Data Eigenvalue')\nplt.yscale('log')\nplt.show()\n```\n\n## Response to \"The proposed Divide-and-Conquer method still relies on low-dimensional codebook subspaces ... lacking a truly novel solution.\"\n\nWe thank the reviewer for this comment. We would like to clarify that Improved VQGAN does not perform dimensionality reduction; instead, it aggregates high-dimensional codebook vectors, which unfortunately still leads to collapsed representations. Our proposed DCVQ is explicitly designed to mitigate this issue, enabling more effective utilization of the latent space.\n\nWe also wish to emphasize that, beyond proposing a method, our work makes a contribution by identifying and systematically analyzing the collapse phenomenon itself — an aspect that, to our knowledge, has not been studied in depth in prior literature.\n\n\n## Repsonse to Q1: \"It is necessary to include comparisons with VQGAN-FC ..., while also controlling for the same amount of information bits in the codebooks.\"\n\nWe greatly appreciate this suggestion. While we did not explicitly label it as “VQGAN‑FC,” we believe the requested comparison is already covered in our experiments. Specifically, in §3.2.1 we introduced a projection layer after the encoder and a corresponding inverse projection before the decoder to vary the background dimension while keeping the encoder architecture unchanged. This setup is equivalent to the Factorized Codes approach.\n\nIn Table 1, we report results across a wide range of background dimensions (3, 4, 6, 8, 16, 32, 64, 128, 256), directly testing different levels of factorization. Furthermore, Figure 5 shows how varying the background dimension affects the effective dimensionality, thereby addressing the reviewer’s concern.\n\nRegarding normalization, we note that all models in our experiments apply L2 normalization before quantization (as described in §3.2.1). The only exception is in §5, where Residual VQ does not support L2 normalization; in this case, we disabled it for our method as well to ensure fairness.\n\nFinally, we ensured comparability by controlling the total information content: the number of tokens and codebook sizes were kept consistent across methods, ensuring that the effective information capacity was comparable.\n\n**References:**\n\n[1] Spectral Relaxation for K-means Clustering\n\n[2] K-means Clustering via Principal Component Analysis"}, {"review_id": "RR5oNtHJC7", "reviewer": "Reviewer_JcVR", "summary": "This paper investigates a consistent phenomenon known as \"dimensional collapse\" in VQVAEs, where high-dimensional embeddings are effectively compressed into a much lower-dimensional subspace in practice. The authors provide a thorough empirical analysis of this phenomenon across various backbones and modalities, exploring its implications for model performance and learning dynamics. To address the limitations imposed by low-dimensional usage, the paper proposes Divide-and-Conquer VQ (DCVQ), which partitions the latent space into independently quantized low-dimensional subspaces, thereby expanding overall model capacity. Experimental results demonstrate that DCVQ overcomes the dimensional bottleneck and improves reconstruction quality on several image datasets.", "questions": "1. Why is there no dedicated Related Work section? How does this work relate to previous studies on dimensional collapse in self-supervised and representation learning?\n2. Have the authors considered comparing DCVQ with other methods that address dimensional collapse, such as those based on feature covariance regularization or network weight constraints?\n3. Can the authors provide experimental results on larger datasets to demonstrate the scalability and robustness of the proposed approach?\n4. What is the computational overhead introduced by DCVQ’s divide-and-conquer scheme, especially in terms of quantization time? Are there trade-offs in efficiency?", "limitations": "1. The lack of engagement with prior literature on dimensional collapse and related phenomena is a major limitation, as it reduces the paper's impact and novelty from a broader perspective.\n2. Methodological comparisons are insufficient. The effectiveness and efficiency of DCVQ would be better validated with comparisons against a wider range of established techniques.\n3. The experimental scope is limited to relatively small datasets, raising concerns about the generality and practicality of the proposed approach for real-world, large-scale tasks.\n4. The time or computational overhead incurred by the DCVQ method is not discussed or measured, which is important for assessing its applicability in resource-constrained scenarios.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The authors conduct a comprehensive empirical study of the dimensional collapse phenomenon in VQVAE models, covering multiple architectures and datasets. This extensive analysis helps raise awareness of an often-overlooked property of VQ-based models.\n2. The observation that effective rank and model performance are not strictly correlated is novel and merits further investigation.\n3. The proposed DCVQ method is intuitive and directly addresses the identified limitation, showing tangible improvements in reconstruction accuracy.\n\nWeaknesses:\n\n1. The paper lacks a Related Work section, and overall citation coverage is insufficient. The phenomenon of dimensional collapse has been studied extensively in representation learning and self-supervised learning literature. The omission of this context makes it difficult to position the paper relative to prior research.\n2. The comparative analysis is limited. While the paper uses rank regularization to encourage higher-dimensional usage, other established approaches, such as network weight constraints and feature covariance regularization, are not discussed or empirically compared.\n3. The experimental evaluation is restricted to relatively small datasets. It would significantly strengthen the work to include results on larger-scale datasets such as ImageNet-100.\n4. The paper does not provide a detailed assessment of the computational cost introduced by the DCVQ approach, particularly regarding the time overhead from multiple quantizations.", "was_revised": true, "final_justification": "The new experiments have addressed my primary concerns", "rebuttal": "# Response to Reviewer JcVR\nWe greatly appreciate the reviewer’s thorough evaluation and insightful feedback. Your comments have been invaluable in helping us improve the clarity and rigor of our work. We address each of your points in detail below.\n\n## Response to Q1\n\nWe thank the reviewer for this important comment. Due to page limits, we placed the Related Work section in the appendix, but we will make this clearer in the revised version. We also appreciate the observation that “dimensional collapse has been studied extensively in representation learning and self-supervised learning literature.” Accordingly, we now include more regularizers inspired by self-supervised methods (see our later response).\n\nWe would like to clarify, however, that the collapse we study in VQVAEs differs from that in self-supervised settings: while the representational collapse of self-supervised arises from forcing invariant output, the collapse in VQVAEs is due to the quantization mechanism, which are inherently different.\n\n## Response to Q2\n\nWe thank the reviewer for pointing out the limited scope of the comparative analysis. To address this, we extended our study beyond rank regularization to include several established approaches, such as Barlow Twins and VICReg from the self-supervised learning literature. Though originally designed for contrasting augmented views, these methods can be adapted to our setting for avoiding representational collapse. We also present a custom regularizer (SingHinge) designed to encourage singular values of the feature covariance matrix to be above a certain threshold. Detailed implementations of these regularizers are provided later in this response.\n\n\n**Table 1: CIFAR10 performance with different regularizers**\n\n| Regularizer Type|   Regularizer Weight |   Val Loss  |   Effective Dim  |\n|-----------------|--------------|-------------|------------------|\n| None|  -   |   0.081 |9 |\n| KoLeo   |   0.001  |   0.081 |9 |\n| KoLeo   |   0.01   |   0.082 |   10 |\n| KoLeo   |   0.1|   0.106 |   27 |\n| BarlowTwins |   0.0001 |   0.081 |   10 |\n| BarlowTwins |   0.001  |   0.093 |   31 |\n| BarlowTwins |   0.01   |   0.105 |   78 |\n| BarlowTwins |   0.1|   0.087 |   49 |\n| VIC |   0.1|   0.081 |9 |\n| VIC |   1  |   0.080 |   10 |\n| VIC |  10  |   0.082 |   11 |\n| VIC | 100  |   0.085 |   12 |\n| SingHinge(0.01) |   0.1|   0.080 |9 |\n| SingHinge(0.01) |   1  |   0.081 |9 |\n| SingHinge(0.01) |  10  |   0.089 |   49 |\n| SingHinge(0.01) | 100  |   0.129 |  111 |\n| SingHinge(0.1)  |   0.1|   0.080 |9 |\n| SingHinge(0.1)  |   1  |   0.081 |   10 |\n| SingHinge(0.1)  |  10  |   0.084 |   11 |\n| SingHinge(0.1)  | 100  |   0.085 |   12 |\n\n(All regularizers were applied on the unquantized latents. Due to the time limit, we only tested the CNN architecture with $f=4$. Val Loss reports validation reconstruction loss only, excluding the regularization term. SingHinge($h$) applies a hinge loss on singular values with threshold $h$ (e.g., 0.01 or 0.1). Regularization weights were chosen to yield performance comparable to the no-regularization baseline.)\n\nFrom these results, we observe a consistent pattern: stronger regularization increases the effective dimensionality but also leads to performance degradation once the dimension grows beyond the baseline level (≈9). Importantly, this trend appears robust across diverse regularization strategies, suggesting that the limitation is not specific to a single method or optimization artifact, but instead reflects a fundamental property of the VQ quantization bottleneck.\n\n## Response to Q3\n\nWe appreciate the reviewer’s suggestion to strengthen our evaluation with larger-scale datasets. We would like to clarify that Section 5.2 already includes ImageNet-1k experiments for our method and RQVAE. To further address the reviewer’s concern, we extended the evaluation to include vanilla VQVAEs and additional regularizers on ImageNet-1k:\n\n**Table 2: ImageNet-1k performance with different methods**\n| Type|   Regularizer Weight |   Val Loss  |   Effective Dim  |\n|-----------------|--------------|-------------|------------------|\n| BarlowTwins |   0.0001 |   0.173 |   14 |\n| BarlowTwins |   0.001  |   0.186 |   41 |\n| BarlowTwins |   0.01   |   0.205 |   91 |\n| BarlowTwins |   0.1|   0.182 |   58 |\n| VIC |   0.1|   0.172 |9 |\n| VIC |   1  |   0.164 |   11 |\n| VIC |  10  |   0.169 |   13 |\n| VIC | 100  |   0.18  |   20 |\n| SingHinge(0.01) |   0.1|   0.158 |9 |\n| SingHinge(0.01) |   1  |   0.168 |   11 |\n| SingHinge(0.01) |  10  |   0.181 |   63 |\n| SingHinge(0.01) | 100  |   0.247 |  137 |\n| SingHinge(0.1)  |   1  |   0.171 |   11 |\n| SingHinge(0.1)  |  10  |   0.183 |   15 |\n| SingHinge(0.1)  | 100  |   0.186 |   17 |\n| DCVQ(4x8d)  |   - |   0.115 |   20 |\n| DCVQ(8x8d)  |   - |   0.105 |   36 |\n\nConsistent with our CIFAR10 findings, we observe that increasing the strength of Barlow Twins, VICReg, or SingHinge raises the effective dimensionality but eventually degrades validation performance once dimensionality exceeds the baseline regime. This trend suggests that the trade-off between dimensional usage and reconstruction quality is robust across both small-scale and larger-scale datasets.\n\nPlease note that, due to time constraints, the results presented here are from training runs stopped early (21k steps), before full convergence. At this stage, the effective dimensions had plateaued, making it possible to compare the influence of regularizers. Differences from Section 5.2 also include the use of smaller parameter sizes. We plan to provide full-convergence results in the camera-ready version.\n\n## Response to Q4\n\nWe thank the reviewer for raising the important point about the computational cost of DCVQ. We emphasize that the computational overhead of DCVQ is minimal, as the processing of each subspace is parallelized. To quantify this, we report the actual training times of DCVQ compared to vanilla VQVAE on CIFAR10 using a CNN architecture with $f=16$, trained for 100 epochs on an A100 GPU:\n\n| Total dim | Type        | Dim per quantizer  | Num quantizers | Training time |\n|-----------|-------------|--------------------|----------------|---------------|\n| 32        | VQVAE       | -                  | -              | 5h 34m 41s    |\n| 32        | DCVQ        | 8                  | 4              | 5h 34m 35s    |\n| 64        | VQVAE       | -                  | -              | 5h 40m 41s    |\n| 64        | DCVQ        | 8                  | 8              | 5h 40m 17s    |\n| 256       | VQVAE       | -                  | -              | 6h 17m 57s    |\n| 256       | DCVQ        | 32                 | 8              | 6h 13m 39s    |\n\nFrom these results, we observe that the training time of DCVQ is essentially the same as that of vanilla VQVAE when the total dimensionality is matched. This indicates that the additional quantizations in DCVQ introduce negligible overhead.\n\n\n## Implementation Details for Regularizers\n\nFor **Barlow Twins** and **VICReg**, which were originally designed for contrastive learning with augmented view pairs, data augmentation is not needed in our rank-promotion setting. Instead, we form input pairs by duplicating the same latent vector. All other hyperparameters follow the original codebase.\n\n### Barlow Twins Loss\n\nGiven normalized latent vectors $ x, y \\in \\mathbb{R}^{B \\times D} $, where $B$ denotes the total number of patches across GPUs and $D$ the feature dimension, and letting $\\text{BN}$ denote 1D batch normalization, the Barlow Twins loss is defined as:\n\n$$\nC = \\frac{1}{B} \\, \\text{BN}(x)^\\top \\text{BN}(y) \\\\\\\\\n\\mathcal{L}_{\\text{Barlow}} = \\sum_i (C_{ii} - 1)^2 + 0.0051 \\sum_{i \\neq j} C_{ij}^2\n$$\n\n### VICReg Loss\n\nGiven $x, y, B, D$ defined similarly, the VICReg loss combines variance and covariance regularization:\n\nThe variance term:\n\n$$\n\\mathcal{L}_{\\text{var}} \n= \\frac{1}{D} \\sum_{i=1}^D \\frac{\\max(0, 1 - \\sigma(x_{\\cdot i})) + \\max(0, 1 - \\sigma(y_{\\cdot i}))}{2}\n$$\n\nwhere $\\sigma$ denotes the standard deviation.\n\nThe covariance term:\n\n$$\n\\mathcal{L}_{\\text{cov}} = \\frac{1}{D} \\sum_{i \\neq j} (\\text{Cov}(x)_{ij}^2 + \\text{Cov}(y)_{ij}^2)\n$$\n\nwith $\\text{Cov}(x) = \\tfrac{1}{B-1}(x - \\mu(x))^\\top(x - \\mu(x))$, where $\\mu(x)$ is the batch mean.\n\nThe total VICReg loss is then:\n\n$$\n\\mathcal{L}_{\\text{VICReg}} = \\mathcal{L}_{\\text{var}} + \\tfrac{1}{25} \\cdot \\mathcal{L}_{\\text{cov}}\n$$\n\n### SingHinge\n\nFor latent representations $z \\in \\mathbb{R}^{B \\times D}$, let $s_i$ denote the singular values of $\\text{Cov}(z)$. With a threshold $h$, the SingHinge loss is defined as\n\n$$\n\\mathcal{L}_{\\text{SingHinge}(h)} = \\frac{1}{D} \\sum_i \\max(0, h - s_i)\n$$\n\n**References:**\n\n[1] Barlow Twins: Self-Supervised Learning via Redundancy Reduction\n\n[2] VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning"}]

Author rebuttal:

Meta-review: This paper identifies and presents a study of dimensional collapse in VQ-GANs. Applying simple rank regularization leads to poor performance, so this paper proposes an alternative method for spanning the space better. The reviewers all praised the paper for the importance of the tackled problem, clarity, and simplicity. Most concerns raised in the reviews have been addressed in the rebuttal. For all the above reasons, I recommend accepting this paper as poster contribution to NeurIPS 2025.
Decision: Accept (poster)