Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper achieves state-of-the-art generation quality on the ImageNet and CC-3M text2image synthesis task by proposing a new transformer architecture that models spatial attention and depth attention of Residual Quantization, and iteratively revising the generated image by randomly masking the output image and re-synthesizing the masked region. 
strengths_and_weaknesses: Strength. 

* The paper achieves SOTA result on ImageNet synthesis, while making the inference time faster than previous methods including VQGAN. 
* I find the idea of revising the generated image very interesting. The paper adds the notion of corse-to-fine refinement to transformer-based autoregressive models, in both the depth transformer of RQ and the revise step. There existed conceptual gap between AR and Diffusion-based models, in that AR models performed iterative generation spatially, and the diffusion-based models performed iterative generation in a coarse-to-fine manner. The proposed method developed a model that can control the iterative refinement in both axes independently, by changing T_draft and M. 
* I like ablations and investigations including Figure 4. 
* There is adequate explanation of previous literature, including Section A.1. 

Weakness

* I found the exposition a bit difficult to follow. The paper introduces many new notations like T, D, M, and Pi. It would be nice if they are accompanied by textual description rather than just referring to them as symbols. 
* The paper puts emphasis on parallel decoding of features, but in the end, the best performing model is using T_draft = 64, which doesn't use parallel decoding in the draft phase. This is confusing in some sense. Moreover, the paper could discuss more on its similarity to MaskGIT. 
questions: Why does [Draft only] achieve such bad FID? It looks like FID ranges from 50+ to 15 in Figure4(a). Shouldn't it still achieve similar FID as MaskGIT or RQ-Transformer? It would be nice if Contextual RQ-Transformer is also evaluated in Table 1 without the revise step. To me, Contextual RQ-Transformer sounds like the architecture described in the upper portion of Figure 2 without the revise step. So it is unclear if [Contextual RQ-Transformer] includes the revise step in Table 1. 
limitations: The paper does not seem to discuss the societal impact of the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposed a two-stage framework for image synthesis, which consists of RQ-VAE and contextual RQ-Transformer. The proposed framework is inherited from the prior work [23] but modifies the original auto-regressive pipeline with a bidirectional pipeline borrowed from BERT.
strengths_and_weaknesses: Pros. 
+ The results are good. Compared with RQ-VAE, the proposed method achieves better FID with smaller model sizes. But the results are not as good as GAN-based approaches such as styleGAN-XL.
+ The writing is easy to follow.

Cons. 
- Lack of in-depth analysis and strong motivation for the proposed method. Both RQ-VAE [23] and the proposed method used an RQVAE as a tokenizer. The difference lies in the second-stage transformer that produces the RQ tokens. In [23], the tokens are generated in an auto-regressive way, while in this submission the tokens are generated via masked image modeling as BERT. By combining RQVAE and BERT, the proposed method is claimed to perform better than existing approaches. However, it is still unclear how the bidirectional masked token modeling outperformed autoregressive modeling. More ablation studies are preferred.
- Some details are missing. Implementation details on text-conditional image synthesis are missing.
- The paper reported class-conditional results on ImageNet. How about unconditional image synthesis performance?  It would be better to provide results on diverse datasets such as Imagenet and FFHQ. Also, the reported results are in a resolution of 256x256, and one concern is the quality in higher resolutions such as 1024x1024.

questions: Please address my concerns shown in the weaknesses.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a Draft-and-Revise framework with a novel contextual RQ-Transformer module for the task of image generation. I think the proposed model is an extension of the previous method "RQ-VAE + RQ-Transformer" (Lee et al., Autoregressive Image Generation using Residual Quantization, CVPR 2022). Specifically, the newly proposed model changes the paradigm of AR model that gradually generate an image by attending to only precedent codes generated. It introduces random masking and gradually generate an image by attending to contextual codes. In addition, the original Spatial Transformer is replaced by Bidirectional Spatial Transformer and a draft-and-revise strategy is introduced to further improve the performance.
strengths_and_weaknesses: Strengths:

+ The idea is interesting, it draws from CMLMs (Ghazvininejad et al., Mask-predict: Parallel decoding of conditional masked language models, EMNLP-IJCNLP, 2019) and skillfully combines with previous work to form an effective image generation model. 
+ The experiments verify the effectiveness of the proposed method in both class-conditional and text-conditional image generation tasks.
+ The presentation is clear and well organized.

Weaknesses:

+ The effectiveness of the proposed model seems to largely depend on the existing model "RQ-VAE + RQ-Transformer" (Lee et al., Autoregressive Image Generation using Residual Quantization, CVPR 2022), the experimental results also seem to illustrate this point. 
+ In addition, the technical contribution of this method seems to be insufficient, mainly using the technical route of the previous method.
questions: See weaknesses above.
limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper pointed out the limitation of the autoregressive image generation using the existing transformer based on the vector quantization methods and proposed the contextual RQ-transformer is trained to infer the masked code stacks by bidirectional self-attention like BERT in order to solve it. Furthermore, they suggested the two-phase decoding, Draft to fill the masked codes and Revise to increase the image quality, in order to break the trade-off between quality and diversity. They demonstrated their significant performances in class-conditional/text-conditional image generation as well as class-conditional image inpainting experiments.
strengths_and_weaknesses: ### Strengths
- Overall clear presentation and discussion on the key argument/idea. 
- The justifications and motivations are clearly established. Especially, the draft-and revise process that imitates the image generation process of a human expert is interesting. 
- Overall, both qualitative and quantitative performance improvements over existing methods are convincing in various experiments.

### Weaknesses
- Novelty: It seems only to the extent of introducing the limitation and solution proposed by MaskGIT[6] to RQ-Transformer. Especially, for introducing the bidirectional transformer to the VQ scheme, the most novel part is the mask generation algorithm; MaskGIT first proposed this using VQVAE. However, this paper simply applies it to RQ-VAE in the draft phase and does not present any special novelty.
 For the above reasons, the comparison with MaskGIT is indispensable to verify their novelty. However, they hardly mentioned MaskGIT in both Related work and Experiment sessions, while highlighting only the merits in comparison with the autoregressive method (i.e., VQGAN) and the Diffusion methods.  
  > In Tabel 1, MaskGIT shows almost similar scores compared to Contextural RQ-Transformer, although it uses fewer parameters. However, there is a lack of analysis with MaskGIT.  

  > They highlight their inference speed compared with VQGAN at the 'trade-off between quality and sampling speed' sub-session. However, as far as I know, MaskGIT requires less iteration compared to Contextural RQ-Transformer in creation (Maskgit needs 16 iterations for a generation while Contextural RQ-Transformer needs T_draft = 64, T_revise = 2, M = 2.)    

 I agree that the improvement of performance through several experiments is convincing. However, I believe that in-depth comparison and analysis in terms of motivation or performance with MaskGIT are essential to demonstrate their novelty.

- Revise phase: Another novelty of the paper is the Revise phase. However, it lacks a clear rationale for why the performance improves because they only perform a simple random uniform sampling. In the Revise phase, what percentage of the code to revise is an important factor (i.e., the number of codes for re-masking), but there is no mention of it. Please provide more details about the Revise phase to improve the novelty.
questions: Overall, the proposed method is good, and the performance has also been well proven through many experiments. However, it is questionable that the novelty, which is the most important, is not enough and does not explicitly specify the differentiation from MaskGIT, which is the most like the problem raised. Being able to clearly explain the weakness mentioned above will be the biggest key to the final rating. 
limitations: Please see the weaknesses and questions.
review_title: null

Meta-review: metareview: A new transformer method for image generation is discussed. Reviewers appreciated the results but raised concerns regarding exposition, some questionable ablations, limited novelty and relation to prior work (MaskGIT). The rebuttal was able to address some concerns. In a discussion reviewers generally kept their rating but raised concerns regarding novelty and ablations again. AC thinks the paper just barely made the cut and strongly encourages authors to further improve the ablations in the camera ready version to further strengthen the paper. AC also recommended senior ACs to look at this decision and possibly revise.
confidence: Less certain
Decision: Accept