Title: Dimensional Collapse in VQVAEs: Evidence and Remedies
Venue: NeurIPS 2025
Accepted: yes
Keywords: vector quantization

representation collapse
Abstract: Vector-Quantized Variational Autoencoders (VQVAEs) have enabled strong performance in generative modeling by mapping continuous data to learnable codes.
In this work, we identify a surprising yet consistent phenomenon that we term \emph{dimensional collapse}: despite using high-dimensional embeddings, VQVAEs tend to compress their representations into a much smaller subspace, typically only 4 to 10 dimensions.
We provide an in-depth analysis of this phenomenon and reveal its relation to model performance and learning dynamics.
Interestingly, VQVAEs naturally gravitate toward this low-dimensional regime, and enforcing higher-dimensional usage (e.g., via rank regularization) could lead to degraded performance.
To overcome this low-dimensionality limitation, we propose \textbf{Divide-and-Conquer VQ (DCVQ)}, which partitions the latent space into multiple low-dimensional subspaces, each quantized independently.
By design, each subspace respects the model’s preference for low dimensionality, while their combination expands the overall capacity.
Our results show that DCVQ overcomes the inherent dimensional bottleneck and achieves improved reconstruction quality across image datasets.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates a consistent phenomenon known as \"dimensional collapse\" in VQVAEs, where high-dimensional embeddings are effectively compressed into a much lower-dimensional subspace in practice. The authors provide a thorough empirical analysis of this phenomenon across various backbones and modalities, exploring its implications for model performance and learning dynamics. To address the limitations imposed by low-dimensional usage, the paper proposes Divide-and-Conquer VQ (DCVQ), which partitions the latent space into independently quantized low-dimensional subspaces, thereby expanding overall model capacity. Experimental results demonstrate that DCVQ overcomes the dimensional bottleneck and improves reconstruction quality on several image datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The authors conduct a comprehensive empirical study of the dimensional collapse phenomenon in VQVAE models, covering multiple architectures and datasets. This extensive analysis helps raise awareness of an often-overlooked property of VQ-based models.\n2. The observation that effective rank and model performance are not strictly correlated is novel and merits further investigation.\n3. The proposed DCVQ method is intuitive and directly addresses the identified limitation, showing tangible improvements in reconstruction accuracy.\n\nWeaknesses:\n\n1. The paper lacks a Related Work section, and overall citation coverage is insufficient. The phenomenon of dimensional collapse has been studied extensively in representation learning and self-supervised learning literature. The omission of this context makes it difficult to position the paper relative to prior research.\n2. The comparative analysis is limited. While the paper uses rank regularization to encourage higher-dimensional usage, other established approaches, such as network weight constraints and feature covariance regularization, are not discussed or empirically compared.\n3. The experimental evaluation is restricted to relatively small datasets. It would significantly strengthen the work to include results on larger-scale datasets such as ImageNet-100.\n4. The paper does not provide a detailed assessment of the computational cost introduced by the DCVQ approach, particularly regarding the time overhead from multiple quantizations.\n\nQUESTIONS:\n1. Why is there no dedicated Related Work section? How does this work relate to previous studies on dimensional collapse in self-supervised and representation learning?\n2. Have the authors considered comparing DCVQ with other methods that address dimensional collapse, such as those based on feature covariance regularization or network weight constraints?\n3. Can the authors provide experimental results on larger datasets to demonstrate the scalability and robustness of the proposed approach?\n4. What is the computational overhead introduced by DCVQ’s divide-and-conquer scheme, especially in terms of quantization time? Are there trade-offs in efficiency?\n\nLIMITATIONS:\n1. The lack of engagement with prior literature on dimensional collapse and related phenomena is a major limitation, as it reduces the paper's impact and novelty from a broader perspective.\n2. Methodological comparisons are insufficient. The effectiveness and efficiency of DCVQ would be better validated with comparisons against a wider range of established techniques.\n3. The experimental scope is limited to relatively small datasets, raising concerns about the generality and practicality of the proposed approach for real-world, large-scale tasks.\n4. The time or computational overhead incurred by the DCVQ method is not discussed or measured, which is important for assessing its applicability in resource-constrained scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper uncovers an interesting phenomenon where VQ-VAEs tend to compress their representations into a much smaller subspace, despite being trained with high-dimensional embeddings. Through a thorough analysis, the authors demonstrate how this compression relates to model performance and learning dynamics. To address the limitations imposed by this low-dimensional bottleneck, they propose the Divide-and-Conquer VQ (DCVQ) method, which partitions the latent space into multiple low-dimensional subspaces that are quantized independently. The combined effect of these subspaces significantly expands the overall representational capacity. Experimental results show that DCVQ effectively overcomes the dimensionality bottleneck and achieves improved reconstruction quality across various image datasets.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The identification of the dimension collapse problem in VQ-VAEs is insightful and worthy of further exploration.\n- The experiments and analyses are comprehensive and well-executed.\n\n### Weaknesses\n- I had hoped the authors would uncover the fundamental cause of dimension collapse and provide a thorough explanation, possibly from the perspective of optimization dynamics. Unfortunately, the paper only observes that the quantizer collapses first, which then causes the encoder to collapse, without deeper theoretical insight.\n- The proposed Divide-and-Conquer method still relies on low-dimensional codebook subspaces, making the approach fundamentally similar to previous dimensionality reduction techniques (e.g., Improved VQGAN), lacking a truly novel solution.\n\nQUESTIONS:\nIt is necessary to include comparisons with VQGAN-FC (which performs dimensionality reduction and normalization), while also controlling for the same amount of information bits in the codebooks.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies a previously unrecognized issue in vector quantized variational autoencoders (VQVAEs), termed dimensional collapse, wherein the learned codebooks tend to collapse onto a limited number of principal components, even when the dimensionality of the code vectors is large. The authors support their findings through extensive experiments, showing that the phenomenon arises from the quantization step during the early training phase. To address this issue, the latent representation is divided into smaller vectors, each of which is independently quantized. This technique enhances both the effective dimensionality of the codebooks and the overall reconstruction quality.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. The writing is well-written, with clear and easy-to-follow logical flow.\n2. The authors effectively design experiments that validate their hypotheses, and their results provide clear evidence, supported by sound interpretation.\n3. The authors propose a simple yet effective method to address the identified issue.\n\n### Weaknesses\n1. The encoder scale factor $f$ controls the number of encoding vectors and likely influences the expressive capacity of the latent representation. It may also be related to the effective dimension of the model. Although the experiments include comparisons with $f=4$ and $f=16$, the main text does not provide a detailed interpretation of how this parameter affects the results. Including such an analysis would help clarify the role of $f$ and its relationship to the model's representational capacity.\n2. The proposed method shows improvements in reconstruction quality, but does not discuss the generation procedure or the quality of generated samples. Including such analysis would help assess the model as a generative model.\n3. I would like to offer a few comments regarding the figures and tables, which may help improve the clarity and presentation of the results.\n    1. In Figure 4, the size of the circles is not clearly distinguishable, which makes it difficult to interpret the relationships among background dimension, effective dimension, and validation loss.\n    2. In Figure 5, using a logarithmic scale on the x-axis may improve readability. Also, it would be helpful to clarify what the colors represent.\n    3. The notation $d_{\\text{eff}}$​ is defined in the text, but it does not appear in the figures or results. Referring to it explicitly in the relevant graphs would help improve clarity. Similarly, in Table 1, it would be helpful to indicate how each hyperparameter corresponds to components in the formulation presented in Section 2.1.\n    4. In Figure 7, the legend or caption does not explain the meaning of the different line styles (e.g., circle line for validation loss, square line for effective dimension). Providing this information would help readers interpret the figure correctly.\n    5. For completeness and consistency, it may be useful to test the model under the case $f = 4$ as well.\n    6. In Figure 9, it is unclear which parameter(s) of RQVAE were varied to generate the graph, as this is not described in the main text. Additionally, since DCVQ can be interpreted as increasing the codebook size and decreasing $f$ (which corresponds to increasing the number of encoding vectors), it would be beneficial to include the standard RQVAE with these settings as a baseline for comparison.\n    7. In Table 2, reporting regression coefficients instead of correlations may better reveal the influence of each latent variable. For interpretability, it may also be helpful to exclude the binary variable associated with the rotation trick when fitting the regression model.\n4. Some additions to the manuscript could improve clarity and completeness.\n    1. Table 1 includes terms such as “rotation trick” and “dead-code restart threshold,” but these are not explained in the main text. Providing brief explanations would help make the paper more self-contained.\n    2. Including training details such as the optimizer, learning rate, batch size, and number of epochs (perhaps in an appendix) would help improve the reproducibility of the experiments.\n\nQUESTIONS:\n1. The detailed explanation of the projection layer following the encoder may help clarify the architectures. Is this layer also trained during the training?\n2. In Figure 9, the compared models employ multiple codebooks. Could the authors clarify how the effective dimension was computed in this multi-codebook setting?\n3. In the experimental results shown in Figures 4 and 5, the effective dimension remained below 20 even when hyperparameters such as $f$ and background dimension were varied. However, in Figure 9, effective dimensions around 100 were reported. Could the authors briefly explain which aspect of the proposed method leads to this discrepancy?\n4. While I agree that training dynamics of the quantizers may cause dimension collapse, I am curious about the authors’ thoughts on why this phenomenon occurs. Since the codebook is updated as cluster centers of the encoding vectors, implying that encoding vectors are mapped near a low-dimensional subspace, any hypothesized reasoning behind this behavior would be very interesting.\n5. I am also curious about the potential relationship between the effective dimension and the intrinsic dimension of the data. Could the authors comment on this?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper revolves around VQ-VAE (and vector quantization overall), and centers around the claim that such models hardly use the full size of their code vectors: the linear dimension for such trained models is incredibly small compared to the feature dimension. When trying to regularize for higher rank features, performance drops off notably. The authors then offer a fix that applies to all VQ models: splitting into subspaces, then quantizing at each individually low-dimensional space. This keeps VQ in a low-dimensional regime while allowing for higher model capacity.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper generally is very readable: the claim is easy to understand, and the experiments are controlled and interpretable. The proposed fix is simple and intuitive to the proposed issue, and demonstrates notable performance increase on the demonstrated experiments.\n\nI have some curiosities around the experiments and the drawn conclusions. Most notably, the claim that \"higher rank results in worse performance\" would need more diverse experiments than training on a single regularization term. It is unclear to me why model performance can be disentangled from potential learning instability from the regularizer, or if this results from the loss \"focusing too much\" on the regularizer (instead of e.g. using a hinge loss, where features are thresholded to some minimum rank). If the authors have an explanation for why this is not a concern, or if there are alternative methods for rank promotion that can be tried which still degrade performance, this will greatly increase the strength of their claim given the lack of theoretical evidence. Since this is a more fundamental VQ fix rather than a specific CV application, it's more excusable that experiments are restricted to now older datasets like CIFAR10 and CelebA, but in the absence of theory, it would be notably strengthened with more modern examples (e.g. ImageNet).\n\nQUESTIONS:\n1. Why does the KoLeo regularization give a strong statement regarding restriction to a higher rank set for optimization? That is, why are we not concerned that performance degredation could be due to the effect of this regularizer's gradient structure on optimization?\n2. What other methods for rank maximization/thresholding would be viable to test against as well?\n3. What are the visual effects of more/less subspaces, and higher/lower dimensions for each of the subspaces?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}