Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: As a researcher working in the similar field, I definitely appreciate the author's efforts to solve a long-standing problem faced by many using a method that is reasonable according to the nature of the discretization process. The author seek to solve the problem that long sequences of discrete making it very difficult for transformer or other method to model in an autoregressive manner, which leads to difficulty in generative task. The author use a non-linear combination of two set of tokens , the primary and the secondary , to make the output representation semi-continuous and with higher capacity, using a MLP projector and dropout mask. This strategy can be intuitive understood as estimating the posterior of representation distribution given the latent consisting of two different tokens ,in a mathematically non-tight way.  
strength_and_weaknesses: The authors solve a real problem and pain point confront by many in discrete token based generative model for vision task, using a semi-continuous represention consisting of a primary and a secondary token. They empirically show using different generative task to show their approach improves performance from different aspects. The problem setting is clear, the approach at a high level is great and the results are solid 

However, the reason why the dropout mask and multi-hypothesis version is use is not probably reasoned. There are many alternative possibilities including a small Bayesian neural network , learnable dropout such as Contextual Dropout or GFlowOut. The reason to use the current masking method to be properly explained. I understand it does not have to be the most sophisticated method in the literature but at least the author need to explain why it cannot be simpler.
clarity,_quality,_novelty_and_reproducibility: The manuscript is well written, the quality of method , experiment and images are high , the method is very novel and , more importantly , clear. The work is original and needed by the community urgently. 

 
summary_of_the_review: This is a great piece of work due to its timing in solving an urgent problem in the research field as well as in the industry. The quality of the work is high , the method is novel and clean. One thing I like to point out is that figure is difficult to understand , not sure it is helping or making it more difficult for the readers. I would suggest improve it
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper:  - This work proposes a Masked Vector Quantization (MVQ) framework to boost per-code capacity, targeting at representing images with shorter sequence or fewer codebook entries. 

 - In the framework of MVQ, each instance is mapped into three components, including a primary code, a secondary code and a mask. The computational infeasibility is mitigated by Multiple Hypotheses Dropout. 

 - The authors demonstrate that with MVQ, improvements on FIDs over VQGAN are observed, especially when the number of tokens per instance and codebook size is reduced. The authors also show some properties of learned codes. 
strength_and_weaknesses: #### **Strength**

 - The topic of designing a better vector quantization system is important, which has recently been adopted in a variety of tasks, including generative modeling. 

 - Vector quantization regimes face a compression-fidelity trade-off, whereas a longer sequence reduces quantization error and increases modeling difficulty. This work proposes to overcome this issue through a novel VQ system.


#### **Weaknesses**

 - Despite including novel masking operations on the code, the motivation of this work is closely related to Product Quantization[1]. Discussion and comparison against relevant quantization methods should be provided.

 - Some details are not self-contained, including the designs of MVQGAN. For example, the authors refer to VQVAE-2 as the architectures of encoder and decoder, and "All architectures use a Transformer as the categorical posterior model as proposed in Rombach et al. (2021)", where **all** is quite confusing. The authors need to provide more details to make the work more self-contained.

The scope of experiments is unable to demonstrate the effectiveness of such modeling power. 

 - Since the major target of vector quantization is not to do compression, the reconstruction quality, errors, or sampling speed, cannot reflect the learned token representations as reliable. In fact, there exist many adaptations that reduce quantization error, while resulting in a worse token representation that might be more difficult for the probabilistic model to learn. 

 - Applying an aggressive downsampling rate such as $f$=16 for small images used in the experiments makes less sense, as the obtained sequence length is already short, e.g. 4x4 . A shorter sequence and extremely small codebook hinders the contribution of this work, of which a non-auto-regressive is able to model well(e.g. MaskGIT[2] for 16x16 tokens on ImageNet). 



[1] Product quantization for nearest neighbor search. T-PAMI 2011
[2] MaskGIT: Masked Generative Image Transformer. CVPR 2022
clarity,_quality,_novelty_and_reproducibility: Some parts of this work is not clear and hard to follow.
summary_of_the_review:  - Although this work is studying this important question of vector quantization, lack of solid experiments, such as the feasibility of training a better generative model with this MVQ system, and improvements on longer sequences and larger dataset, are unable to show the effectiveness of this method. Note that the target of VQ is not to solely do better compression, but for handling complex signals that existing models are unable to do well.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper presents Masked Vector Quantization (MVQ), which is an extension of hierarchical Vector Quantized Variational AutoEncoder (VQ-VAE [a]). MVQ introduces mask configuration on the secondary code vector and demonstrates its effectiveness on image generation with shorter sequences (for example, less than 20) per sample.

The paper uses a training schema named Multiple Hypotheses Dropout (MH-Dropout) to learn the mask configuration. The key idea is to sample a number of binary masks (with Dropout) and take the best one that minimizes the latent reconstruction. 


[a] Razavi, Ali, Aaron Van den Oord, and Oriol Vinyals. "Generating diverse high-fidelity images with vq-vae-2." Advances in neural information processing systems 32 (2019).
strength_and_weaknesses: ### Strength
* [S1]: The approach is well presented, figures are clear and contain enough detailed information to present the approach 
* [S2]: Generating long sequences is indeed computationally expensive (see supplementary material Table 6), the problem is well motivated
* [S3]: The related work is completed, and to the best of my knowledge, all the related fields are discussed. 
* [S4]: Representations learned by MVQ with short sequences are interesting (Sec. 4.4) 

### Weakness 
* [W1]: Although the proposed method (MH-Dropout) shows improvement in image generation with shorter sequences, the reason why it works remains unexplained. I think the paper lacks a discussion on the intuition and the explainability of the proposed approach. 

* [W2]: The proposed pipeline is conceptually similar to VQ-VAE2. In fact, in 4.1 Setup, the paper precises that it replaces the top-bottom hierarchical VQ codebooks in VQ-VAE2 with MVQ, which consists of injecting masks into the top-level codes if I understand correctly. In such a case, it would be better for the paper to discuss clearly the difference to VQ-VAE2 and clarify its own technical contribution. 

* [W3]: The generation results are weak (Table 1, Figure 7). I am fully convinced that the community needs faster methods, however, I am not sure the community benefits from approaches that sacrifice too much on the generation quality.  

* [W4]: The transformer part is not self-contained. It is not clear whether the primary and the secondary codes are in different scales and how to conduct generation for hierarchical codes. 
 
[a] Razavi, Ali, Aaron Van den Oord, and Oriol Vinyals. "Generating diverse high-fidelity images with vq-vae-2." Advances in neural information processing systems 32 (2019).
clarity,_quality,_novelty_and_reproducibility: ### Clarity, Quality, and Novelty 

* If I understand correctly, the proposed method is partly based on the previous work VQ-VAE2[a]. The difference to VQ-VAE2 is the mask configuration added into the top-level codes. Although it shows improvement in the generation tasks with shorter sequences, the technical novelty seems limited. 

### Reproducibility
Not enough implementation details are provided in the paper, for example, in VQ-VAE2, it trains with lots of tricks to address the issue of the codebook collapse: EMA to update the codebook, codebook reset, etc. I am not sure whether these are used in the paper. Personally, it would be hard to reproduce the experiments merely with the description available in the paper. 
summary_of_the_review: As detailed in Section **Strength and Weakness**, I think improving the generation efficiency is important for autoregressive generation. The problem in the paper is well-motivated. 

However, the current manuscript could be further improved. Particularly, I think it lacks a part to explain the intuition behind the proposed approach and why the proposed approach works. In addition, the difference to VQ-VAE2 is not clear and the experimental results are weak.  

Considering the above concerns, my current rating is to reject the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents Masked Vector Quantization (MVQ), which is an form of of hierarchical Vector Quantized Variational AutoEncoder (VQ-VAE. The paper uses a training schema named Multiple Hypotheses Dropout (MH-Dropout) to learn the secondary codebook. The key idea is to sample a number of binary masks (with Dropout) and take the best one that minimizes the latent reconstruction.

Strength
The problem setup is well-motivated and presentations are clear.

Weakness
The paper lack a stronger theoretical justification on the multiple hypotheses learning approach. While this could be potentially made up by empirical evidence, we do not see strong results demonstrating the method in challenging scenarios. In fact, we think the number of tokens used in experiments are too small, resulting in poor generation quality.
justification_for_why_not_higher_score: The significance is not high due to limited experimental study in the regime of small number of tokens.
Decision: Reject