Title: Information-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variables
Venue: NeurIPS 2025
Accepted: yes
Keywords: Information theoretic generalization analysis

IT analysis

generalization error analysis

VQ-VAE

vector quantized VAE
Abstract: Latent variables (LVs) play a crucial role in encoder-decoder models by enabling effective data compression, prediction, and generation. Although their theoretical properties, such as generalization, have been extensively studied in supervised learning, similar analyses for unsupervised models such as variational autoencoders (VAEs) remain insufficiently underexplored. In this work, we extend information-theoretic generalization analysis to vector-quantized (VQ) VAEs with discrete latent spaces, introducing a novel data-dependent prior to rigorously analyze the relationship among LVs, generalization, and data generation. We derive a novel generalization error bound of the reconstruction loss of VQ-VAEs, which depends solely on the complexity of LVs and the encoder, independent of the decoder. Additionally, we provide the upper bound of the 2-Wasserstein distance between the distributions of the true data and the generated data, explaining how the regularization of the LVs contributes to the data generation performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work provides an information-theoretic (IT) analysis of the generalisation capacity and data generation quality of VQ-VAEs. \n\nAfter showing that the basic IT bound cannot be directly applied to such (unsupervised) setting if one wants to analyse separately the latent variables (LVs) and decoder parameters, the authors propose a new error bound which explicitly encodes the dependence between the latent variables and supersample indexes to remove this dependency. An analysis of this new bound reveals that the generalisation gap solely depends on the complexity of the LVs and encoders' parameters. However, the proposed bound contains the (non-converging) empirical KL divergence, and a new (converging) bound is proposed that removes any dependency on the empirical KL divergence. \n\nAdditionally, an upper bound on the 2-Wasserstein distance between the true and generated data is derived to assess the data generation quality, and here again, the authors show that the LVs significantly impact this.\n\n Overall, this paper demonstrates the importance of LVs' complexity (and their regularisation) for generalisation and data generation quality, providing some insights into why some training techniques worked well in VQ VAEs.\n\nSTRENGTHS AND WEAKNESSES:\n___Quality:___\n\n[+] The paper is sound, and the claims are well-supported. I did not find any issue with the proposed proofs. However, I am an expert in VAEs, not IT, and I may have missed something.\n\n[+] The limitations of the work are clearly stated and extensively discussed in the conclusion and appendix.\n\n[-] It would have been nice to see more of the experimental part in the main paper.\n\n\n___Clarity:___\n\n[+] The paper is well-written and easy to follow\n \n[+] The maths are clear and easy to follow; each variable is defined before use, and a notation table is provided in the appendix.\n\n[+] The main theoretical results are generally followed by one or two sentences summarising the key points in layperson's terms. This is greatly appreciated as it avoids breaking the flow of the paper and helps the reader check that their understanding is correct.\n\n[-] Some of the notations are difficult to distinguish and may gain in readability if slightly updated (see comment 2).\n\n[-] There are a few typos (see comment 3).\n\n___Significance:___\n\n[+] Formally showing that the generalisation and data generation quality strongly depends on the LVs regularisation is important as it helps understand why some practical settings of VQ VAEs worked well and opens some avenues for improving the capabilities of these models.\n\n[-] It is a shame that the result from Thm. 3 cannot be used as a generalisation score due to its computational cost, as this would have increased the significance of this work.\n\n___Originality:___\n\n[+] This paper provides a new understanding of why some empirical techniques used for VQ VAEs worked well.\n\n[+] While such analysis was previously done in a supervised setting, this has not been attempted in unsupervised learning (as far as I know, but as I said before, I am not an expert in IT and may have missed an existing pocket of literature). Although the current work is limited to a discrete latent space, it still advances our understanding of unsupervised models.\n\n[-] The results are not very surprising (the importance of LV regularisation was already observed empirically) and are restricted to VQ VAEs.\n\nQUESTIONS:\n[1] Could the authors provide a more detailed explanation of the supersample setting from Section 2.3? From what I understand, each entry is an $x$ drawn iid from $\\\\mathcal{X}$ and then one uses the associated index to select the $x$ from the correct column when one is in test or training setup. But then $\\\\mathcal{A}(\\\\mathcal{X}_U)$ would only use one element per row, and so $l_0$ would be an nx1 matrix. However, it is stated that $l_0$ is an nx2 matrix, so there is something that I did not understand correctly about how $\\\\mathcal{X}_U$ is created.\n\n[2] The $\\\\tilde{\\\\cdot}$ and $\\\\bar{\\\\cdot}$ are sometimes hard to distinguish (e.g. l. 1044). Could the authors replace the $\\\\tilde{\\\\cdot}$ with something more distinctive like $\\\\breve{\\\\cdot}$, for example?\n\n[3] l.5 I do not think that the authors truly mean \"insufficiently underexplored\". I suggest replacing it with either \"underexplored\" or \"insufficiently explored\". \nl. 378 \"that enabling\" should probably be replaced by \"enabling.\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper presents a theoretical analysis of VQ-VAEs by extending information-theoretic generalization bounds to discrete latent spaces. Introducing a new data-dependent prior within a supersample framework, the authors derive decoder-independent generalization error bounds for reconstruction loss, emphasizing the critical role of the encoder and latent variables in both generalization and data generation. They further propose a permutation symmetric prior to ensure asymptotic convergence of the bounds and provide upper bounds on the 2-Wasserstein distance between true and generated data distributions.

STRENGTHS AND WEAKNESSES:
### Strengths
- The paper demonstrates a solid and deep understanding of VQVAE from a mathematical and theoretical perspective.
- Highlights the crucial role of the encoder and latent variables in both reconstruction and generation, offering guidance for future model design.

### Weaknesses
- Despite the extensive mathematical theory and statistical error bounds presented, it is difficult to see how this work offers meaningful practical contributions to the community. At least from my perspective, the authors do not leverage their theoretical insights to propose improvements to VQ or demonstrate tangible benefits. Including more evaluations or experiments could strengthen the paper’s overall persuasiveness.
- The theoretical results, while novel, may be challenging to translate directly into practical improvements or model design guidelines.
- The focus on decoder-independence, while theoretically elegant, might overlook decoder contributions in real-world scenarios.

QUESTIONS:
None

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops an information-theoretic framework to analyze the generalization behavior of vector-quantized variational autoencoders (VQ-VAEs), focusing on the role of discrete latent variables (LVs). By extending the supersample-based generalization bounds from supervised learning to the unsupervised setting, the authors derive decoder-independent upper bounds on the reconstruction loss. They propose a novel permutation symmetric prior, allowing them to establish asymptotically vanishing generalization bounds under certain regularity conditions. Additionally, the authors provide an upper bound on the 2-Wasserstein distance between the true and generated data distributions, offering theoretical insight into data generation quality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper provides a rigorous, information-theoretic treatment of generalization for VQ-VAEs.\n\n- The results show that generalization and data generation quality depend primarily on the encoder and latent structure, not the decoder. This insight is useful for guiding model design.\n\n- The introduction of a permutation symmetric setting provides a technically novel way to construct a data-dependent prior that leads to bounds with asymptotic convergence.\n\nWeaknesses:\n\n- Despite asymptotic guarantees, the derived bounds are too loose to be practical, often several orders of magnitude larger than observed generalization gaps.\n\n- While the bounds are decoder-independent and theoretically elegant, they do not translate into actionable training strategies or practical diagnostics for improving models.\n\n- The paper's notation and technical detail can be hard to follow. Key intuitions behind the bounds and assumptions could be better emphasized, especially for a broader NeurIPS audience.\n\nQUESTIONS:\nI want to be upfront that I am not an expert in the topic of this paper.\n\n- Have you considered experiments that modify encoder or latent complexity and measure their correlation with your theoretical quantities? That could help demonstrate utility beyond the current generalization gap plots.\n\n- You mention this as future work, are there concrete barriers (e.g., measure-theoretic issues or lack of symmetry) that prevent your approach from being directly applicable to standard VAEs?\n\n- Given the decoder’s role is marginalized in generalization bounds, could the authors provide guidelines for determining when a decoder is “sufficiently expressive” to avoid affecting generalization?\n\nLIMITATIONS:\nYes, the authors are upfront with the limitations. I appreciate it very much.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an information-theoretic analysis of VQ-VAEs in a supersampling setting, aiming to understand how latent codes influence both generalization and generation. The authors: (1) derive new generalization bounds that evaluate reconstruction loss without invoking the decoder; (2) introduce a permutation-symmetric supersample setting, which lets them control the generalization gap by regularizing the encoder’s capacity; (3) provide an upper bound on the 2-Wasserstein distance between the true and generated data distributions, offering insight into the model’s generative quality.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nhe paper addresses a critical question in the theory of unsupervised learning, specifically regarding VQ-VAE, and its finding that the generalization bound is independent of the decoder is particularly interesting.\n\n**Weakness**\n\nMy main concern is the lack of direct practical implications. While the insights into the generalization of VQ-VAE are interesting, the theoretical results do not offer clear guidance for the design and training of VQ-VAEs. \n\nThere are a few points I would like more discussion and clarification on, as listed below:\n\n- Table 4: *\"The results are consistent with Theorem 4, which suggests that when using a stochastic mechanism for latent variables (LVs), the upper bound of the generalization gap is independent of decoder complexity, but depends on $d_z$.\"* This result is both interesting and valuable. However, it’s also possible to increase the complexity or expressiveness of LVs by increasing the codebook size rather than $d_z$. Could the authors provide any discussion or insight on when it is preferable to increase $d_z$ versus the codebook size?\n\n- Figure 2: I believe the reconstruction error should be included in the analysis to observe whether increasing decoder size leads to better training error. This would ensure that the model is well-trained.\n\n- Figure 2: What are the implications of the behavior of the first LK term?\n\n- The finding regarding the 2-Wasserstein distance between the true and generated data distributions is not entirely novel. Reference [1] shows that the Wasserstein distance between the training and generated data distributions is equivalent to the reconstruction error and an arbitrary distance (e.g., KL divergence, Wasserstein) between the encoder output and the prior. Reference [2] discusses the consistency of the Wasserstein distance. I believe that combining the results from [1] and [2] aligns with the findings presented in Section 5, albeit in a more general framework. Reference [3] also addresses generalization between the true and generated data distributions. Could the authors provide further discussion on how these results compare?\n\n- Additionally, the authors mention that \"designing a prior that reduces this bound could lead to improved data generation accuracy.\" Since discrete latents learning and generalization are typically performed in two stages—first learning the latent variables, followed by learning the prior using autoregressive models like transformers, which effectively fit the encoder output—I'm unclear on the practical benefits and implications of the results presented in this section for data generation performance. Could the authors elaborate on this further?\n\n\n**Discussion:** While the generalization gap is independent of decoder, it should still be considered in the analysis, as it affects the overall performance of the VQ-VAE (e.g., training loss). Once the decoder is considered, my understanding is that there is a relationship between decoder size and latent size ($d_z$). As shown in Figure 6, latent size ($d_z$) is now correlated with the generalization gap. Therefore, generalization performance (not just the gap) is not independent of the decoder. Focusing solely on the generalization gap may lead to incorrect conclusions.\n\nOverall, I do believe the authors have done valuable work, and this research will benefit the community working with VQ-VAE models.\n\n[1] Vo, Vy, et al. \"Parameter estimation in DAGs from incomplete data via optimal transport ICML 2024 \n\n[2] D. Pollard, \"Quantization and the method of k-means,\" in IEEE Transactions on Information Theory, vol. 28, no. 2, pp. 199-205, March 1982, doi: 10.1109/TIT.1982.1056481.\n\n[3] Vuong, Tung-Long. \"Task-Driven Discrete Representation Learning.\" AISTAT 2025.\n\nQUESTIONS:\nPlease see strengths and weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}