Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The author's introduce a new VQ-GAN model with three improvements, spatial normalization of the quantized vectors, MaskGIT for quicker autoregressive reconstruction, and multichannel feature quantization.  They show better reconstruction performance with similar code size to other models and methods.
strengths_and_weaknesses: This is obviously of interest to the NeurIPS community and the results are impressive.  The implication that normalization the quantized code vectors adds substantial improvement is interesting, and definitely opens up interesting areas to follow up on.  The use of MASKGit seems to me to be not a large contribution of this work, and while interesting, can be downplayed a bit in comparison to the normalization and use of multichannel quantization.

The weaknesses are that the paper is not always careful in its comparison to other methods (figure 3 doesn't show the code size or latent size in comparison between several different methods so it is fairly difficult to compare across methods).  More careful use of common tools from compression literature (on rate-distortion) would help clarify some of the comparisons across methods.
questions: Other papers have found that generalized normalization functions can learn as powerful representations as much deeper traditional neural network architectures without normalization.  Do the author's think this may be part of the power of the addition of normalization as they state it?  Or is it something specific to normalization before the MASKGit?
limitations: The authors address stock concerns about implications but I believe that larger implications are raised by the image of the child in Figure 4 (4th column).  The child's face has changed significantly from the original, but unlike traditional encoding techniques, does not betray any indication to the downstream user that the image has lost information or is in any sense "uncertain".  Traditional artifacts in simpler compression techniques may look bad to the eye but they at least faithfully convey to the user when information has been lost.  I think not enough concern is paid in this paper and in this literature to technologies that produce confident and clear images that are not what was captured and encoded on the other side, and that may fool the end user otherwise.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a new VQ-based image synthesis method. Based on MaskGIT, the paper proposes spatially conditional normalization and uses multichannel representation to improve the reconstructed image quality of the tokenization stage. The proposed spatially conditional normalization modulates the quantized vectors for a better reconstruction performance by inserting spatially variant information to the VQ decoder. The multichannel representation subdivides the encoded continuous latent along the channel dimension into multiple chunks and quantizes them with a shared codebook, which further improves the reconstruction performance by increasing the latent size.  For the generation stage, the paper modifies MaskGIT to sample the multichannel latent. Experimental results on two benchmark datasets show the proposed image synthesis method is efficient and effective for generating diverse and high-quality images.
strengths_and_weaknesses: Strengths:
+ The paper is technically sound.
+ The paper is well structured.
+ The citations are extensive.

Weaknesses:
- The novelty is somewhat limited. The idea to improve the reconstruction ability of the tokenization stage is original and interesting. However, the proposed method heavily relies on existing techniques such as multichannel representation and MaskGIT.
- The performance on image synthesis is not good enough. As shown in Table 3, the proposed Mo-VQGAN performs worse than MaskGIT for class-conditional image generation on ImageNet in terms of complexity and quality metrics. It weakens the contribution since the proposed Mo-VQGAN is based on MaskGIT.
- The evaluation is somewhat limited. The main contribution of the paper is incorporate the spatially conditional normalization to modulate the quantized vectors. Figure 6(a) has shown that spatially conditional normalization can improve the reconstruction performance (rFID), especially for Fourier features. However, improving the reconstruction performance (rFID) does not necessarily improve the generation performance (FID), as shown in Figure 6(c). In my opinion, the experimental evaluation should report FID with or without the proposed spatially conditional normalization.

questions: 1.Section 4.3 says that MaskGIT and VIT-VQGAN performs better on ImageNet because they “use more GPUs for longer training”. I’m interested in the difference of training budget. Can the proposed Mo-VQGAN surpass MaskGIT given the same training budget?
2.It would be better to conduct more ablation experiments on the generation performance. Which initialization of the spatially conditional normalization is better for the generation? Why is the spatially conditional normalization only used in the first three blocks of the VQ decoder?

limitations: The paper discusses an interesting limitation of the proposed method. The model sometimes generates images with a high-frequency appearance without the structure information, which may be attributed to the generation of multichannel representation. I think the choice of mask scheduling function in MaskGIT may be not optimal for the multichannel representation. Maybe the multi-channels of the same location should be masked and generated together, and the sampling can rely on the probability product of the multi-channels.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper discusses a method for generating, reconstructing images using quantized representations. The key difference wrt prior work is that they modulate the quantized representations, that is, they propose to use AdaIN-like modulation of the quantized features. They claim that without this, the results are often repetitive. A further smaller contribution is to use several channels of quantized features, ie when the image is encoded they split features into 4 blocks along the channel dimension and quantize them with the same dictionary. 

They show better reconstruction results and comparable or worse results on image synthesis. 
strengths_and_weaknesses: The paper is well written and addresses a challenging, very competitive problem. The key contribution---modulating quantized vectors---is interesting although perhaps not sufficient on its own right. Improving stage-1 training is a very important problem, which potentially can improve many applications, including image synthesis, in- and out-painting, text-to-image, video synthesis. The paper, however, needs to be improved to be able to claim that their contribution lead to these improvements.

**Motivation**: The paper is motivated by the claim that quantization results into repetitive structures. I wonder if the authors could support this statement somehow, since it's not easily observable in images if that's the case. Furthermore, if that's one of the motivations, it should be supported by numerical experiments, especially to show that the contributions improve the situation. It's not totally clear how do to this, but a method based on image autocorrelation might do the trick. The only support for this claim in the paper is fig 3. However, I cannot say it shows any repetitive content, besides, this image *can* be repetitive. Finally, even if there is repetitive content, is it due to the exactly the same tokens (a visualization of token indices will help here), or because the decoder collapses. Currently, the support behind the claim is not sufficient. Modulation can "just" improve expressivity of the the tokens, allowing the generator to have shorter tokens. 

**Methodology**: from a technical standpoint modulation of the tokens is a simple extension of the original quantized schema. The proposed multi-channel quantization is interesting too, but I cannot say that both of these contributions combined together bring the paper above the bar.

**Significance**: Improving only the first stage makes the paper less interesting. Why did previous papers gain attention? I believe because they had good encoder-decoder frameworks, which offered rich latent spaces. These latent spaces can be used to solve a variety of generative tasks, such as reconstruction, generation, completion, outpainting, text-to-image and even text-to-video. If the current paper improves stage-1 it can perhaps also show better results at stage-2? In terms of stage-2 the paper reports image synthesis comparisons, in which the numbers are either on par or worse than the state-of-the-art. The authors say that their model is smaller, but the model size is not the main contribution of the work. I believe, there are many ways of making the model even smaller. If you make the model larger will it be better? In the supplement they show class-to-image generation results. According to table 3, there is no improvement over MaskGIT in terms of all of the scores. MaskGIT is even a smaller model. So in terms of significance of the proposed contributions it's hard to tell whether stage-1 improvements lead to improvements in downstream tasks. The numbers show the opposite. An intuition could be that the proposed framework reconstructs images better than others, because it provides larger, more expressive latent space, at the cost of poorer structure. Somehow the authors admit this in the limitations paragraph. It would be great if the paper could prove otherwise. 
questions: Please see above
limitations: It would be great to better understand what the proposed stage-1 can do and what it cannot do. Like can it do outpainting, superresolution and other downstream applications. Reconstruction per se is a less interesting application.
review_title: null

Meta-review: metareview: The three reviewers had significantly diverging final opinions (strong accept, borderline accept and weak reject). The authors addressed many of the concerns in their rebuttal. I read the paper carefully, and I agree with the concerns from one reviewer about why the improvements in stage-1 do not lead to significant improvements in stage-2. I think this concern needs to be properly addressed, because otherwise it is unclear what the benefit of this approach would be for real applications. While previous work has shown that improved stage-1 performance leads to improved stage-2 performance, why was it not replicated in this situation? I also found the analysis of why the spatially conditioned normalization improves reconstruction to be lacking. If the "jagged" structures are addressed by this work, then understanding why with simple examples would have shed more insight into the technical contribution. However, in summary, I think this paper is slightly above the acceptance bar, and addressing the above concerns is recommended for the final version.
confidence: Less certain
Decision: Accept