Vector Quantization By Distribution Matching

Authors: Xianghong Fang, Litao Guo, Hengchao Chen, Yuxuan Zhang, Song Dingjie, Ye xin Liu, Harry Yang, Ser-Nam Lim, Benyou Wang, Qiang Sun
Primary area: generative models
Keywords: Vector Quantization, Distribution Matching, Criterion Triple, Wasserstein Distance

Abstract:
The success of autoregressive models largely depends on the effectiveness of vector quantization, a technique that compresses and discretizes continuous features by mapping them to the nearest code vectors within a learnable codebook. Two critical issues in existing vector quantization methods are training instability and codebook collapse. Training instability arises from the gradient gap during both forward and backward gradient propagation, especially in the presence of significant quantization errors, while codebook collapse occurs when only a small subset of code vectors are utilized during training.
A closer examination of these issues reveals that they are primarily driven by a mismatch between the distributions of the features and code vectors, leading to unrepresentative code vectors and significant data information loss during compression. To address this, we employ the Wasserstein distance to align these two distributions, achieving near 100\% codebook utilization and significantly reducing the quantization error. Both empirical and theoretical analyses validate the effectiveness of the proposed approach.

Decision: reject
Reviewer scores: 6, 5, 3, 5

Reviews and discussion:
<thread>
creation date:
1730102634130

thread_type:
official_review

<thread_object>
creator: reviewer_yhdn

<content>
summary:
In the paper, the authors highlighted that a low utilization rate of the codebook will impair the training stability  of Vector Quantization (VQ), and subsequently proposed to enhance this utilization rate by aligning the distributions of feature vectors with those of the codebook.

strengths:
1) To enhance the utilization rate of codebook, this paper proposed to align the distributions of feature vectors with those of the codebook.

2) Gaussian distributions are adopted  to model the feature and code vectors.

 3) Wasserstein  distance is used d to measure the distance between distributions.

weaknesses:
1) Using Gaussian distributions to model the feature and code vectors is not reasonable, as these vectors typically belong to diverse classes with more intricate distributions. From a statistical perspective, modeling the data with Gaussian mixtures is more appropriate. Consequently, it is reasonable to raise doubts regarding the effectiveness of the proposed distribution matching method.

2) Improving the utilization rate of  code vectors in a  codebook (or called dictionary) is not a novel problem, which has been early studied in dictionary learning.

questions:
1) Is the low utilization rate of code vectors a result of the gradient backpropagation problem (as argued by the authors in line 049), or is it due to the insufficient diversity of training samples relative to the large size of the codebook? I lean towards the latter explanation, and therefore consider the low utilization rate is *not* a pivotal issue that significantly impacts the final performance of vector quantization.

2) In Lines 062-068, the authors should  delve into the reasons behind the low code vector utilization rate observed in existing methods. For instance, in my view, the codebook initialization with k-means (Zhu et al. 2024) should work well in distribution matching, but performed worse in the paper. Could this be attributed to the issue of codebook utilization rate, or other factors, such as differences in network structures?

3) A codebook with a high utilization rate should closely match the distribution of feature vectors. Therefore, is it necessary to incorporate an additional distribution matching approach based on the Wasserstein distance?

4) As previously mentioned, modeling the entire set of feature vectors using a single Gaussian distribution is not reasonable. Instead, Gaussian mixtures are recommended as a more suitable alternative.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730108115127

thread_type:
official_review

<thread_object>
creator: reviewer_9xez

<content>
summary:
Existing vector quantization methods in autoregressive models suffer from training instability and codebook collapse due to a mismatch between feature and code vector distributions, causing significant data loss. To address the problem, the authors propose to use the Wasserstein distance to align these distributions, achieving high codebook utilization and reduced quantization error.

strengths:
The Gaussian modeling-based distribution matching is proposed to enhance the codebook utilization rate, then improving the VQ performance.

weaknesses:
As discussed in the paper, several methods have been introduced to enhance the codebook utilization rate. The paper's contribution appears to be more incremental than highly innovative. Additionally, the proposed distribution matching method simply assumes that all feature vectors adhere to a Gaussian distribution. This assumption may not be perfect for feature vectors of various patterns.

questions:
1) There are simpler methods for matching  distributions, like the k-means, which does not require prior assumptions about the data distribution. This raises the question of whether  the Gaussian modeling-based distribution matching is  necessary, especially given its higher complexity.


2) Is Criterion 2 (Codebook utilization rate) necessary for optimization?  This criterion seems to be well satisfied if Criterion 3 (Codebook perplexity ) is satisfied.

3) The paper devotes considerable effort (in sections 2.3 and 2.4) to demonstrating the advantage of distribution matching (between feature vectors and code vectors) in enhancing quantization accuracy. This relationship is straightforward and readily understandable. However, the authors are encouraged to provide a more detailed examination of the convergence properties of the proposed three criteria.


4)  Does the multi-scale VQ cost (namely Eq. (5)) really work?  By 4.2, it is easy to derive that $z_q=z_1+(\hat{z}_T-z_T)$, implying that the features we adopt for codebook optimization essentially share the same resolution  with  $ z_T$, rather than multi-scale. For the operation on $z_i$, ($1<i<T$), the method implicitly assume that the feature vectors nearby in space are similar to each other, and then share the same quantization values (code vectors). However, this assumption is not appropriate for the low-resolution features with high variance.


5) In Figures 2 and 6, the sub-figures about  ${e_k}$  and ${\gamma}$ should be put in the rectangle of “vector quantizer”.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730456176153

thread_type:
official_review

<thread_object>
creator: reviewer_tvxj

<content>
summary:
The paper tackles key issues in vector quantization (VQ) within autoregressive models, specifically addressing training instability and codebook collapse. The authors identify these issues as a mismatch between the distributions of features and code vectors, leading to significant information loss during compression. They propose aligning the codebook distribution with the feature distribution using quadratic Wasserstein distance to improve three criteria they introduce—Quantization Error, Codebook Utilization Rate, and Codebook Perplexity. The paper validates the effectiveness of these metrics through empirical analysis and further substitutes vanilla VQ with a multi-scale VQ structure. Experimental results demonstrate near 100% codebook utilization, reduced quantization error, and performance improvements.

strengths:
1. The paper thoroughly analyzes the issues with VQ, particularly the training instability due to gradient discontinuities (addressed via the Straight-Through Estimator, though still prone to instability), and the low codebook utilization due to limited active code vectors.
2. The introduction of three specific criteria—Quantization Error, Codebook Utilization Rate, and Codebook Perplexity—adds quantifiable methods for assessing VQ performance and measuring improvements objectively.
3. The alignment of the feature distribution with the codebook distribution via quadratic Wasserstein distance is well-motivated theoretically and effectively addresses the identified issues.
4. The paper validates its proposed approach through experiments, showing substantial improvements in codebook utilization (nearly 100%) and other metrics, supporting the contributions' significance.
5. The multi-scale VQ approach, combined with Wasserstein distance loss, shows promise for practical applications, especially in visual generative tasks.

weaknesses:
1. While Gaussian sampling effectively validates the proposed metrics, testing on a broader range of data distributions would enhance the generalizability of the findings, particularly for distributions commonly encountered in autoregressive models or visual tasks. Notably, VQ-VAE does not assume a Gaussian distribution in its latent space, making evaluating performance across diverse data types essential to reflect real-world applications better.
2. Although the metrics (Quantization Error, Codebook Utilization Rate, and Codebook Perplexity) are proposed and demonstrated through Gaussian sampling, the paper lacks validation on diverse and realistic data, especially those commonly used in autoregressive and generative tasks. The reliance on Gaussian sampling limits the generalizability and applicability of the metrics, making it uncertain how they would perform in varied contexts.
3. The paper's focus on aligning feature and codebook distributions with Wasserstein distance for improved quantization seems limited in scope and impact. The primary advantage is codebook utilization. However, the broader implications or potential applications in complex or large-scale generative tasks (e.g., real-world autoregressive models) are not clearly demonstrated.
4. While the paper achieves nearly 100% codebook utilization, the practical significance of this improvement remains unclear. It's difficult to determine if the proposed method advances the field without a comprehensive comparison to other quantization methods that might achieve similar outcomes or more straightforward approaches to training stabilization.
5. While multi-scale VQ improves vanilla VQ, the paper lacks detailed theoretical grounding or motivation for why a multi-scale approach is advantageous, specifically in Wasserstein distance.

questions:
1. Could you clarify the theoretical motivation behind using multi-scale VQ over vanilla VQ and its role in improving Wasserstein alignment?
2. Have you tested your metrics on real-world datasets beyond Gaussian sampling to validate their general applicability?
3. Can you clarify the practical implications of 100% codebook utilization, especially when compared to alternative methods in vector quantization?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730833825023

thread_type:
official_review

<thread_object>
creator: reviewer_bt9x

<content>
summary:
To address the training instability and codebook collapse in vector quantization, the authors align the distribution of features and code vectors by including an additional quadratic Wasserstein distance term which regularizes their disparity under a Gaussian assumption. This method leads to 100% codebook utilization and better image reconstruction/generation quality.

strengths:
1. The Wasserstein VQ algorithm is efficient and has a mathematical foundation
1. Empirical experiments indicate improvement in reconstruction quality

weaknesses:
1. The presentation is not clear enough. See the questions below, some of which are crucial.
1. The improvement in reconstruction quality is marginal. The authors also didn't explain how they picked the additional hyperparameter $\alpha_3$, which can raise suspicion of $p$-hacking.
1. Insufficient ablation experiments. The authors should study how the choice of distance measure (Wasserstein vs KL divergence), $\alpha_3$, and other knobs affect the performance of the proposed method.
1. Editorial:
    1. On line 93, it's inaccurate to say "codeword IDs $r \in \mathbb{R}^{h \times w}$", because the components in $r$ are discrete.
    1. When presenting the experiment in Figure 3, the authors should point the reader to Appendix E.1, which includes details of the experimental setting.
    1. On line 280, consider calling the optimal set of code vectors $\{e_K^*\}_{k=1}^K$ to avoid confusion.
    1. There is an extra "Then" in Lemma 3 on line 310.
    1. In Section 4.1 on line 370, it's clearer to use the existing notations $z_i$ and $e_i$, instead of defining new ones $z_e$ and $z_q$.
    1. In Section 4.2 on line 418, the footnote $ ^3$ looks like a cubic power notation.
    1. In Section 6 on line 534, there is a typo in "alignment".

questions:
1. The authors mentioned training instability on line 107, but it's unclear to me how Wasserstein VQ can help stabilize the learning. Is the argument that a smaller VQ reconstruction error makes $z_i$ and $e_i$ closer in expectation, and thus corresponds to better learning stability?
1. For "Criterion 2 (Codebook Utilization Rate)" defined on line 156, how does it depend on $N$?
1. For "Criterion 3 (Codebook Perplexity)" defined on line 180, do we want a high codebook perplexity? I am a little confused because in language modeling we want the "perplexity" to be low.
1. The interpretation of Figure 3 in lines 203 through 213 is not deep enough. In particular, why do $\mathcal{E}$, $\mathcal{U}$, and $\mathcal{C}$ exhibit different behavior when the feature covers the codebook vs vice versa?
1. In Section 2.4, the theorems all require bounded support, but Section 3.1 has the Gaussian assumption, while Gaussian does not have a bounded support. It's fine to use the theorems as an intuitive argument, but the authors should at least explain why their theoretical results "somehow" apply to Gaussian when the assumption is violated.
1. What's the motivation for the Gaussian assumption in Section 3.1? What's the real distribution of the features and code vectors in the experiments?
1. On line 307, it is claimed that KL divergence between Gaussian distributions has no analytical formula, but there is one on [Wikipedia](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence#Multivariate_normal_distributions). It would be great if the authors can check if that is correct and explain why the quadratic Wasserstein distance is a better measure than the KL divergence.
1. In Section 3.2 and Figure 5, it is unclear what "training steps" mean. By inspection, we can see Equation (3) has an optimal solution at $\hat{\mu}_2^* = \hat{\mu}_1$ and $\hat{\Sigma}_2^* = \hat{\Sigma}_1$. We can then transform the code vectors such that $\hat{\mu}_2 = \hat{\mu}_2^*$ and $\hat{\Sigma}_2 = \hat{\Sigma}_2^*$ with a linear transformation. I don't think there is anything to be trained.
1. In Section 3.2 on line 346, what's the motivation for sampling the code vectors such that $\hat{\mu}_2 \ne \hat{\mu}_1$, which is clearly suboptimal? Moreover, how does Wasserstein VQ compare against other methods when $\hat{\mu}_2 = \hat{\mu}_1$ at initialization?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732593361070

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General commention to all reviewers

comment:
Thank you to all the reviewers for your valuable feedback. In response to the reviewers’ feedback, we have made major revisions to the paper, summarized as follows:

- **Distributional matching analyses of existing VQ methods**  We have developed an atomic experimental setting to examine these VQ approaches through the lens of distributional matching, which is detailed in Section 3.2.
- **Visual explanations of the two criteria** To enhance clarity and understanding, we have incorporated visual elucidations to illustrate Criterion 2 and Criterion 3, as shown in Figure 11 of Appendix H.1.
- **Visual illustrations of the limitation of $k$-means based VQ approach** We offer visual illustrations to elucidate why k-means-based VQ methods fall short in achieving distributional matching, as depicted in Figure 12 in Appendix H.3.
- **Improved theoretical guarantees and discussions.** We improve our Theorem 2 by removing the assumption of bounded support, making it applicable to Gaussian distributions. In addition, we explicitly discuss the implication of Theorem 1.

Moreover, we acknowledge certain misunderstandings in the reviewer’s comments and are willing to clarify the following three points.

**G1 The main contribution of this paper lies in addressing codebook collapse or low codebook utilization rate**

Our paper aims to present an optimal VQ strategy tailored for the machine learning community, rather than merely addressing codebook collapse. In our paper, we examine two distinct issues in VQ: training stability and codebook collapse from the perspective of distributional matching, and empirically and theoretically demonstrate that identical distributions yield the optimal solution in VQ. Therefore, we not only achieve 100% codebook utilization, but also establish a stable VQ system with minimal information loss. 

We also emphasize that we are the pioneers in presenting this optimal VQ solution to the machine learning community, despite its simplicity and intuitiveness.

**G2 $k$-means-based VQ methods can also achieve distributional matching.** 

The limitation of $k$-means-based VQ methods, notably the significant issue of codebook collapse, especially when the codebook size $K$ is large, is widely recognized in numerous research studies. To help reviewers better understand this limitation, we offer visual illustrations to elucidate why
$k$-means-based VQ methods fall short in achieving distributional matching in Appendix H.3. Furthermore, in Section 3.2,  we have developed an atomic experimental setting to demonstrate that Vanilla VQ and methods based on $k$-means and $k$-means++ fail to align the distributions of features and codebooks. Additionally, we show that without explicit distributional matching regularization, the effectiveness of the existing VQ methods heavily depends on codebook initialization, which can become problematic when the feature distribution is unknown and dynamically changing during training.

**G3 The motivation of experimental setting and several insights in Section 3.2**

We recognize that it is nearly impossible to provide an atomically fair experimental setup to compare the performance of various VQ methods based on the criterion triple in image reconstruction experiments, as thoroughly discussed in **R 1.8.** This challenge motivates us to establish a genuinely atomic and fair comparison setting, which is the focal point of Section 3.2. Within this setting, we constrain the feature distribution across different VQ methods to a fixed Gaussian distribution. Although feature distributions are usually intricate and dynamic in real-world applications, this simplified setting yields valuable insights.

- Vanilla VQ and methods based on $k$-means and $k$-means++ demonstrate inferior performance due to their inability to align the distributions of features and codebooks, as evidenced by the experimental results depicted in Figure 5.
- In the absence of explicit distributional matching regularization, the effectiveness of existing VQ methods is significantly contingent upon codebook initialization. Conversely, with the incorporation of explicit distributional matching regularization, the performance of our proposed Wasserstein VQ is robust and independent of codebook initialization.
- Despite the inherent arbitrariness of the codebook distribution during training, our proposed quadratic Wasserstein distance, based on the Gaussian distribution assumption, efficiently aligns the distributions and attains superior VQ performance.

Our code is available in the supplementary material, and we have added the experimental code for Section 3.2. All experimental results are reproducible, and we promise that our code will be made publicly available. Once again, we appreciate the reviewers’ thoughtful comments, and we have carefully addressed each of them in our revised paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732753692519

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking Forward to Deeper Discussions with All Reviewers

comment:
Dear Area Chairs and All Reviewers,

Thank you to all the reviewers for your valuable feedback. We have dedicated more than two weeks to carefully address the comments of each reviewer, by providing a comprehensive 14-page response that includes 33 detailed answers. We recognize that not all reviewers may fully grasp the motivation and contributions of our paper, and thus, we have provided extensive explanations to address every concern raised. If any further concerns arise, please do not hesitate to bring them forth; we are eager to engage in deeper discussions with all reviewers.

Best Regards,
1683 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732846178442

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Public Comment: Reviewer YHdN Request Us To Provide a Theoretical Proof for Accurate Density Matching In Deep Neural Networks

comment:
Dear Program Chairs, Senior Area Chairs, Area Chairs, All Reviewers, and All Readers,

In our paper, we empirically and theoretically demonstrate that an identical distribution between feature vectors and code vectors yields the optimal solution in Vector Quantization (VQ). To achieve this distributional matching, we propose a quadratic Wasserstein distance metric based on the Gaussian distribution hypothesis. Notably, we are the pioneers in presenting this optimal VQ insight to the machine learning community.

However, **Reviewers YHdN** contends that the actual feature distribution is not Gaussian, and thus argues that our paper suffers from logical flaws, deeming our contribution as incremental. Furthermore, they request us to provide a theoretical proof for accurate density matching using the Wasserstein distance. 

First, we clarify **the substantial challenges involved in precise density matching.**

- Achieving precise density matching is already a highly challenging problem in statistical theory. When considering its application in deep neural networks, where the feature distribution is entirely unknown, it becomes virtually an insurmountable task. 
- Representative research paper, such as Variational Autoencoders (VAEs) [1], utilize the Kullback-Leibler (KL) divergence to approximate two distributions. However, explicit evidence for precise density matching is not provided. Generative Adversarial Networks (GANs) [2] introduce an adversarial training framework to achieve density matching between generated and real images, while diffusion models[5] propose a multi-step denoising approach to accomplish the same task. Neither of these methods, however, offers a detailed explanation or theoretical proof of their density matching capabilities. All of these works are highly influential and extensively cited in the machine learning field.

Second, we further clarify **certain plausibility of employing the Gaussian distribution**. 

- We acknowledge that relying on the assumption of a Gaussian distribution may have undesired effects.  However, this assumption is commonly employed in various practices. For instance, the Fréchet inception distance (FID), a widely used metric for evaluating generated images from generative models, assumes that both the generated images and the ground-truth images follow a Gaussian distribution. Despite its limitations, FID remains a popular choice for measuring the distance between images in different domains. Additionally, the Gaussian distribution assumption is extensively utilized in numerous classical research works, including Variational Autoencoders (VAEs) [1], Generative Adversarial Networks (GANs) [2], Stable Diffusion (SD) [3], and Stable Diffusion XL (SDXL) [4].
- The central limit theorem possibly plays a role in ensuring that learned feature vectors and code vectors follow a Gaussian distribution provided a large sample size and a large codebook size.
- The real distribution of the feature and code vectors would be exceedingly intricate and often defies precise mathematical formulation. Our selection of the Gaussian assumption is primarily motivated by the Gaussian distribution’s inherent simplicity. Within the specific context of VQ, the feature vectors $z_e = E(x) \in \mathbb{R}^{h\times w \times d}$ extracted from the encoder typically reside in a moderately small dimension space (d=8). In Vanilla VQ setting, if $n$ image samples are used in one training step, there are $n * h * w = 8192$ feature vectors for calculating the Wasserstein distance. Under the Multi-VQ setting, each GPU processes $n * h * w = 24576$ feature vectors in a single training step. Given that the number of feature vectors is sufficiently large while the dimensionality is moderately small, adopting the Gaussian assumption is not particularly problematic and provides a reasonable approximation.
- The effectiveness of the Gaussian assumption is also evident in our experimental results. For instance, in Section 3.2, despite initializing the codebook with a standard Gaussian distribution, the optimization process of the codebook distribution remains inherently arbitrary during training. Nevertheless, our quadratic Wasserstein distance, which is based on the Gaussian distribution assumption, consistently achieves superior VQ performance.

[1] Auto-Encoding Variational Bayes, ICLR 2014
[2] Generative Adversarial Nets, NeurIPS 2014
[3] High-Resolution Image Synthesis with Latent Diffusion Models, CVPR 2022
[4] SDXL: Improving Latent Diffusion Models for Hight-Resolution Image Synthesis
[5] Denoising Diffusion Probabilistic Models

We sincerely appreciate the time and effort each reader has dedicated to reviewing our work. We strongly encourage readers to provide their comments and offer their perspectives on the reasonableness of the reviewers’ assessments and requests. Your insights are invaluable to us as we strive for continuous improvement in our research.

Best Regards,
1683 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732922517177

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
1683 Authors Publicly Question the Objectivity and Impartiality of Reviewer YHdN’s Comments (1/2)

comment:
Dear Program Chairs, Senior Area Chairs, Area Chairs, All Reviewers, and All Readers,

Leveraging the transparency of the OpenReview platform, 1683 authors have publicly questioned the objectivity and impartiality of Reviewer YHdN’s comments, citing their subjective perspectives and several factual errors.

We commence by summarizing the **discussion concerning novelty and contribution**.

- **Reviewer Comment**: Improving the utilization rate of code vectors in a codebook (or called dictionary) is not a novel problem, which has been early studied in dictionary learning.
- **Our Response**: The contribution of our paper is to present an optimal VQ strategy tailored for the machine learning community, rather than merely addressing codebook collapse, as detailed in **R 4.3**.
- **Round 1 Reviewer Comment**: The technical contribution of improving code utilization rate. This technique has been developed in the area of dictionary learning. The contribution is incremental, rather than novel.
- **Our Round 1 Response**: We have clarified this comment in **R 4.3**. Our contribution is to provide an optimal VQ strategy tailored for the machine learning community, rather than merely addressing codebook collapse. Could you identify any existing VQ paper that approaches the problem from the perspective of optimality? 
- **Round 2 Reviewer Comment** The authors claimed that their major contribution is providing an optimal VQ strategy for the machine learning community. If this claim holds true, the paper should progress in the following way. Initially, the authors should mathematically define what is the optimal VQ, given a distribution of feature vectors. Subsequently, they must demonstrate that the VQ involved in deep networks can converge towards this optimal VQ, utilizing the BP algorithm under the constraint of Wasserstein distance-based distribution matching. Regrettably, the two crucial elements are not found in Theorems 1 and 2.
- **Our Round 2 Response** **1)** In Section2, based on two critical issues of VQ, we introduce a criterion triple that an optimal VQ must satisfy. We then empirically and theoretically demonstrate that an identical distribution between feature vectors and code vectors yields the optimal solution in VQ. Thus, we provide **a mathematically rigorous definition of optimal VQ**.  **2)** In Section 3.1, we propose the Wasserstein distance to achieve distributional matching under the Gaussian distribution assumption. While we acknowledge the difficulty in providing a theoretical proof for optimal solutions in deep neural networks due to the NP-hard nature of accurate density matching, we note that influential works such as VAEs use KL divergence andGANs use adversarial training to approximate or match distributions. Similarly, diffusion models employ multi-step denoising techniques. None of these methods, however, offer explicit evidence or a detailed theoretical proof of their density matching capabilities **3)** The reviewer **was unable to find any existing VQ papers** that address the problem from an optimality perspective and raised demands that the authors consider unreasonable. As the saying goes, Rome was not built in a day. We are confident that our insights will inspire future research to refine and enhance density matching, ultimately leading to improved VQ performance.
- **The Round 3 Reviewer Comment**  The history of optimal VQ theory dates back to the 1950s at Bell Laboratories, where researchers conducted studies to enhance signal transmission through suitable discretization techniques[4]. These studies have investigated the optimal VQ issue from the traditional rate-distortion perspective, offering a more comprehensive and in-depth analysis compared to the theorems presented in the paper. This suggests that the innovation in this paper is quite limited.
- **Our Round 3 Response** **1)** Early studies on VQ were primarily confined to numerical computation and signal processing, e.g., [4] by the reviewer. In the context of deep learning tasks, the pioneering work is the VQ-VAE  **2)** By the reviewer’s assessment of novelty and contributions, the seminal works on diffusion models [1][2] appear quite limited, considering the extensive study of diffusion processes within the statistical community. Specifically, [2] employs score matching, a technique introduced in 2005 by [3]. We kindly ask the reviewer why this work was accepted as an oral presentation at NeurIPS 2019 and has received significant citations. Similarly, VQ has been extensively studied throughout the 20th century, then why VQVAE is often regarded as the pioneering work in the field of visual generative models.

[1] DDPM: Denoising Diffusion Probabilistic Models NeurIPS 2020 [2] Generative Modeling by Estimating Gradients of the Data Distribution, NeurIPS 2019 [3] Estimation of Non-Normalized Statistical Models by Score Matching, JMLR 2005  [4] Introduction to vector quantization and its applications for numerics
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a vector quantization (VQ) method to deal with the  training instability and codebook collapse issues. The main idea is to introduce a distribution-matching approach based on the Wasserstein distance.

Strengths:
1. Novelty: The use of Wasserstein distance  (of course, under Gauss assumption) for VQ optimization is an interesting idea for aligning feature and code vector distributions.
2. Theoretical Analysis: The authors presented some mathematical analysis demonstrating improvements. 

Weaknesses:

1. The strong Gaussian distribution assumption might be impractical for high-dimensional data in practical applications, as noted by multiple reviewers. The authors’ rebuttal did not adequately address this concern. 

I would like to add  a few more comments on Gaussian assumption since the authors mentioned diffusion models. In the original work of diffusion models, in fact the reverse Gauss distribution assumption is asymptotically correct when the time interval approaches zero, which in fact theoretically justifies the use of Gaussian distribution in the reverse process.  In current paper. I fully understand that Gaussian might be a natural choice due to its simplicity, but more justification is needed. For example, using the illustration of Gaussian setting, the authors argued that their method works independently of codebook initialization. How can this be justified for non-Gaussian features using Wasserstein distance as an approximation? 


2. The novelty is marginal. The authors’ rebuttal, while clarifying some distinctions, did not convincingly establish the novelty of their method. 

Regarding the critics of reviewer YHdN on prior works of optimality of VQ, I think his/her main concern was that what is the main difference between the previous well-studied traditional rate-distortion perspective of optimality and the newly stated optimality in current paper? For down-stream applications, what fundamental differences do the two different views lead to, and why should there are such differences? 

3. The method provides some marginal gains in reconstruction quality compared to previous methods. 

Overall, while the paper introduces a very interesting perspective, but the weaknesses mentioned above make the contributions insufficient for acceptance at this stage. The authors’ responses did not fully resolve the reviewers' concerns  about the practical benefits and robustness of their method. I have to recommend rejection.  The decision does not  indicate the idea itself but more on the improvement to be done to make it sufficient to get published.