Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer
Authors: Doyup Lee
Chiheon Kim
Saehoon Kim
Minsu Cho
Wook-Shin Han
Venue: NeurIPS 2022 Accept (2022)
Keywords: image generation
discrete image representation
transformer

Abstract: Although autoregressive models have achieved promising results on image generation, their unidirectional generation process prevents the resultant images from fully reflecting global contexts. To address the issue, we propose an effective image generation framework of \emph{Draft-and-Revise} with \emph{Contextual RQ-transformer} to consider global contexts during the generation process. As a generalized VQ-VAE, RQ-VAE first represents a high-resolution image as a sequence of discrete code stacks. After code stacks in the sequence are randomly masked, Contextual RQ-Transformer is trained to infill the masked code stacks based on the unmasked contexts of the image. Then, we propose the two-phase decoding, Draft-and-Revise, for Contextual RQ-Transformer to generates an image, while fully exploiting the global contexts of the image during the generation process. Specifically. in the \emph{draft} phase, our model first focuses on generating diverse images despite rather low quality. Then, in the \emph{revise} phase, the model iteratively improves the quality of images, while preserving the global contexts of generated images. In experiments, our method achieves state-of-the-art results on conditional image generation. We also validate that the Draft-and-Revise decoding can achieve high performance by effectively controlling the quality-diversity trade-off in image generation.

Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer
Doyup Lee"
Chiheon Kim Kakao Brain
Kakao Brain
doyup wwwekkaaabbaii.ccoom
chiheon www.kkaakbbaaiinccooomm com
Saehoon Kim
Minsu Cho
Wook-Shin Han POSTECH
Kakao Brain
POSTECH
shkim@kakaobrain..cmm com
mscho@postech..cc ac.kr
www.ndddbab...stecch..
Abstract
Although autoregressive models have achieved promising results on image gener- ation, their unidirectional generation process prevents the resultant images from fully reflecting global contexts. To address the issue, we propose an effective image generation framework of Sogfl-and-Revisee with Contextual RQ-transformer to generalized VQ-VAE,
consider global contexts during the generation process. RQ-VAE first represents high-resolution image sequence of discrete code stacks. After code stacks in the sequence are randomly masked, Contextual RQ- Transformer is trained to infill the masked code stacks based on the unmasked contexts the image. Then, we propose the two-phase decoding, Draft-and- Revise, for Contextual RQ- To-Trsssfrrmer to generate image, while fully exploiting the global contexts the image during the generation process. Specifically, the draft phase, model first focuses generating diverse images despite rather low quality. Then, in the revise phase, the model iteratively improves the quality images, while preserving the global contexts of generated images. In experiments, our method achieves state-of-the-art results on conditional image generation. We also validate that the Braft-and-Revise decoding can achieve high performance by effectively controlling the quality-diversiy trade-off in image generation.
Introduction
Learning discrete representations of images enables autoregressive (AR) models to achieve promising re- sults image generation. Here, image is encoded into a feature map, which is represented as sequence of discrete codes [13
code stacks 2 Then,
an AR model generates a sequence of codes in the raster scan order and decodes the codes into image. Consequently, AR models show high performance and scalability large-scale datasets 13
Figure 1: Examples of image inpainting by an AR model (middle) and ours (right).
23 26
Despite the promising results of AR models, we pos- tulate that the ability of AR models is limited due the lack of considering global contexts in the generation process. Specifically, since AR models generate images by sequentially predicting the
Equual contribution Cooresponding author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

next code and attending to only precedent codes generated, they neither exploit the later part of the generated image nor consider the global contexts during generation. For example. Figuref (middlee) shows that an AR model fails to generate coherent image, when it is asked to inpaint the masked region of Figure [] (left) with a school bus. Such a failure due to the inability of AR models to refer to the context of traffic lane on the right side of the masked region.
To address this issue, we propose an effective image generation framework, Draft-and-Revise, with a contextual transformer to exploit the global contexts of images. Given randomly masked image, the contextual transformer is first trained infill the masks by bidirectional self eeff-attention similarly to BERT 8 To fully leverage the contextual prediction in generation, we propose Draft-and- Revise decoding which has two phases, draft and revise, imitating the image generation process of human expert who draws a draft first and iteratively revises the draft to improve its quality. In the draft phase, the model first infills an empty image to generate draft image with diverse contents despite the rather low-quality. In the revise phase, the visual quality of the draft is iteratively improved, while the global contexts of the draft are preserved and exploited. Consequently, our Draft-and- Draft-and-Revise Revise with contextual transformer effectively generates high-quality images with diverse contents.
We use residual-quantize VAE (RQ-VAE) [3] to implement our image generation framework, since RQ- VAE generalizes rector-quantized VAE (VQ-VAE) [3 by representing an image as a sequence of code stacks instead of a sequence of codes. Then, we propose Contextual RQ-Transformer as a contextual transformer for masked code stack modeling of RQ- VAE. Specifically, given a sequence of randomly masked code stacks, Contextual TQ-Trasfformer first uses a bidirectional transformer to capture the global contexts of unmasked code stacks. Based on the global contexts, the masked code stacks are predicted parallel, while the codes in each masked code stack are sequentially predicted. In experiments, our Draft-and -evit- framework with Contextual RQ- Transforfrrm achieves state-of- the-art results on conditional image generation and remarkable improvements on image inpainting. In addition, we demonstrate that Draft-and- decoding can effectively control the quality-diversity trade-off in image generation to achieve high performance.
The main contributions of this paper are summarized as follows. 1) We propose an intuitive and powerful framework, Contextual RQ- T--Tansffrrmer, for masked code stack modeling of RQ-VAE based on bidirectional transformer. 2) We propose a novel two-phase decoding, Draft-and soaff-and-Revise, for bidirectional transformers to fully exploit the global contexts during image generation and achieve state-of-the-arr results on class- and text-conditional image generation benchmarks. 3) An extensive ablation study validates the effectiveness of Traft-and-Revis decoding on controlling the quality-diversity trade-off and its capability to generate high-quality images with diverse contents.
2 Related Work
Discrete Representation for Image Generation By representing an image as sequence of codes,
VQ-VAE [33] becomes an important part for high-resolution image generation [6] [O. 15 23 26
33),
but suffers from low quality of reconstructed images. However, VQGAN [I3 significantly improves the perceptual quality of reconstructed images by adding the adversarial and perceptual losses into the training objective of VQ-VAF. As a generalized approach of VQ- VAE and VQGAN, RQ-VAR (23 represents an image as a sequence of code stacks, which consists of ordered codes, and reduces the sequence length, while preserving the reconstruction quality. Then, EQ-Transformer (23) achieves high performance with lower computational costs on generating high-resolution images. However, as an AR model of RQ- RQ- Transsfforerr cannot capture the global contexts of generated images.
Generation Tasks with Bidirectional Transformers To overcome the limitation of AR models on unidirectional architecture, bidirectional transformers have been studied for generative tasks. After bidirectional transformer' is trained to infill random mask as the masked token modeling of BERT 8 an effective decoding method has been proposed for bidirectional transformers to generate
texts [1 30). images O 35), or videos [16]. For image generation, M6-UFC (35) first incorporates
the masked token modeling in the codes of VQGAN [13 for image generation. MaskGIT 66) proposes the confidence- based decoding to achieve high performance on ImageNet [7]. Similar to M6-UFC (35
and MaskGIT (6), we also incorporate the masked token modeling of RQ- VAE in a bidirectional transformer for image generation. However, instead of the menfidence-baseed decoding 6 35
we propose a novel two-phase decoding for bidirectional transformer to achieve ofate-of-the-art performance and validate its effectiveness. Recently, discrete diffusion models [ 4 12 [5] also uses

---

Tokenizing (RQ-VAE)
Random Masking
Contextual RQ-Transformer
Bedrectoonal
Unddrretional
Rawe
Encoderr
Bidirectional Spatial Transformar
RGB Image
Stacked Code Map [H W D)
Dravi-and-Revise Decoding
Draft
Revise
2
Figure 2: The overview of Draft-and-Revin Revise framework with Contextual RQ-Transformer. Our framework exploits global contexts of images to generate high-quality images with diverse contents.
bidirectional transformers to generate an image, and the training with an absorbing state [1 is also the same to infill randomly masked sequence [4 [6]. However, different from the reverse process of diffusion models, our decoding method has explicit two phases to generate high-quality images with diverse contents.
Draft-and-Rewe Revise Framework for Effective Image Generation
In this section, we propose our Draft-and-Revis Revise framework for effective image generation using bidirectional contexts of images. We first review RQ-VAE (3 as generalization of VQ-VAE. Then, we propose Contextual RQ-Transformer which is trained to infill a randomly masked sequence of code stacks of RQ-VAE by understanding bidirectional contexts of unmasked parts in the sequence. Lastly, we propose traft-and-revic decoding for a bidirectional transformer to effectively generate high-quality images exploiting global contexts of images. Figure provides the overview of our proposed framework, including Contextual RQ-Transformer and Draft-and-Revie decoding.
3.1 Residual-Quantized Variational Autoencoder (RQ-VAE)
VAE 23 represents an image as sequence of code stacks. Let a codebook C = {(ks c(k)) include pairs of a code and its code embedding e(k) E R"= where K = |C| is the codebook size and n, is the dimensionality of ekk.. Given vector Rm= Q(2; C) is defined the code of
((z; C) = arg min | e(k)ll
Then, RQ with depth D represents vector as a code stack which consists of codes:
RQ(z; D) = (kI. kk) E [K]P;
(2)
where ka is the d-th code of z. Specifically, RQ first initializes the 0-th residual vector as rv = z, and then recursively discretizes residual vector ra-1 and computes the next residual vector r,
ka Q(ra-niC),
Fd ra-1 e(ka),
for d [D]. Finally, z approximated by the sum the D code embeddings == >D e (aa). We remark that RQ is generalized version of VQ, RQ with D = is equivalent VQ. For D > 1. RQ conducts finer approximation of z as the quantization errors are sequentially reduced as increases. Here, the coarse-to-fin fine approximation ensures the D codes to be sequentially dependent.
RQ- RQ-VAE represents an image as map code stacks. Specifically, given image first converted to low-resolution feature map Z = E(X) E H HW xn=, and then each feature vector Zine at spatial position (h, w) is discretized into a code stack by RQ with depth As result, we get map of code stacks S E [[] Further details of RQ-VAE are referred Appendix.

---

3.2 Contextual Transformer for Image Generation with Global Contexts
As a bidirectional transformer for KAE, we propose Contextual RQ- Te-Trnsfformer for image generation based on a contextual understanding of images. First, we adopt the pretraining of BERT [8] to formulate a masked code stack modeling of VAE. Then, we introduce how Contextual RQ-
Transformer infills the randomly masked code stacks after reading the given contextual information.
3.2.1 Masked Code Stack Modeling of RQ-VAE
By adopting the pretraining of BERT (.
formulate the masked code stack modeling of RQ-VAE
with contextual transformer to generate an image by iterative mask-infilling as non-AR models [4].
We first convert the map S [K] H Hx D into sequence of code stacks S' E [K] NK] using the raster-scan ordering, where N = HW and S (S.1 S/DD) E [K]] for n E [M]. We denote
S' as S for the brevity of notation. mask vector m is defined as a binary vector m E {0. 1) to
indicate the spatial positions be masked. Then, the masked sequence Sim of S by is defined
Sn { [MASK]] if m, = 1
m, = 0
(S (m)m)
where [MASK] is mask token to substitute for Snd if m, 1. Given random mask vector m ~ g( (m), the masked code stacks given Sim are modeled as
D
II (().)S = II
II =ld=l
P
pS SSn (mm):
(5)
where (m)) is a mask distribution where the masking portion Σ ===1 / in (0. 1] as well as the masking positions are randomly chosen. Instead of fixing the portion to 15% as in BERT, training a model with a random masking portion from (0, 1] enables the model to generate new images based on various masking patterns including m,,
for all n. We explain the details of q(m) Section 3.2.3
The left-hand side of Eq. implies that all masked code stacks can be decoded in parallel based on the contexts of S mm If D 1. Eq. 5 becomes equivalent conventional masked token modeling of texts [8] and images 66| 16 where a single token at each masked position is predicted. For D > 1, the D codes of S., are autoregressively predicted,
they sequentially computed in Eq. 3 for coarse- to-fine approximation and hence well-suited for an AR prediction. We show that the effectiveness of our framework is generalizable regardless of D in Appendix B. In addition, the parallel decoding can control the trade-off between the quality and sampling speed of image generation.
3.2.2 Contextual NQ-Transformer
We modify the previous TQ-Transformer 23 for masked code stack modeling with bidirectional contexts in 5 Contextual EQ-Transformer consists of Bidirectional Spatial Transformer and Depth Transforme:: Bidirectional Spatial Transformer understands contextual information in the unmasked code stacks using bidirectional self-attentions, and Depth Transformer infills the masked code stacks in parallel, by autoregressively predicting the D codes at each position.
Bidirectional Spatial Transformer Given a masked sequence code stacks S bidirectional spatial transformer first embeds the masked sequence im using the code embeddings of RQ- KQ-VAE.am as
Ed e(Snd) m, =0 @[MASK]
Un, = (n)
(6)
if m, = 1
where (E)(n) is an embedding for position n. and @[MASK] E "= is a mask embedding. Then, the bidirectional self-attention blocks, papatial extracts the context vector h, to predict code stack Sn.
(h).-.. hx) espatial (uj. ... "u......uu. u.).
The code stack at masked position S, (Sa1, ... S.D)) is autoregressively predicted, The input of depth transformer (Vnd) D is defined as
Depth Transformer
hn Σ Ed=1 e(Sn))
ifd
V Vnd = PE +
(8)
if

---

can Algorithm UPDATE of S
Require: A sequence of code stacks S, partition II (m
.m"), model 0
1: for / 1,... do
2: Sample Sn ~ Pe
Vn : m
update the codes masked positions
3: end for
4: return
Algorithm 2 Draft-and- Revise decoding
Require: Partition sampling distributions Parah and Prevs the number of revision iterations
/+ draft phase */
1: Sempty < ([MASK]
MMSK])
initialize empty code map
2: Sample II III T draf
3: Sdralt +- UPDATE((SEM)) Sempty 01
b generate draft code map
/+ revision phase +/
4: S S
5: for m 1, .M do
6: Sample II ~ p(II; revise)
7: Sth UPDATE(S II:0)
iteratively revise the code map
8: end for
9:
return S
where (d) the positional embedding for depth Then, depth transformer gdepth which consists of causal attention blocks, outputs the logit Pnd to predict the d-th code S at position n as
Pnd pdepth (Vnl,"' Vnd) and po SSnd k kS S S
We remark that the architecture of Contextual RQTaassffreerr subsumes bidirectional transformers. Specifically, RQ-Transformer with the depth D is equivalent to a bidirectional transformer since the depth transformer becomes a multilayer perceptron with layer normalization (2].
3.2.3 Training of Contextual RQ-Transformer
For the training of Contextual RQ- Transfforrrrrr let define mask distribution ((m) with mask scheduling function y. Following previous approaches [4] [6], the scheduling function y is chosen to be decreasing and to satisfy (0) 1 and ((1) O. Then, mask g(m) is specified as
T ~ Unif 1)) and m Unif((m: |m| [n(r) -NH))
(10)
where m
ENE[N] m, the count of masked positions. Finally, the training objective of Contextual RR-araasffrmrr is to minimize the negative ligelikel masked code stacks:
C (m)
Es
Σ
Σ
log po S S
(11)
Ln:m
d=1
3.3 Draft-and-Revise Two-Phase Decoding with Global Contexts of Generated Images
We propose novel decoding algorithm, Draft-and-Reviss for Contextual RQ-Transformer to effectively generate high-quality images with diverse visual contents. We remark that our effective decoding method is required for bidirectional transformers fully exploit the global contexts of images and achieve high performance of image generation. Here, we introduce the details of our BaaftanddRRvvin decoding and then explain how the two-phase decoding of effectively control the quality-diversity trade-off of generated images.
We define partition II (m'.... .m as collection of pairwise disjoint T mask vectors to cover all spatial positions, where Σ m for n E [N]. partition is sampled from the distribution pII;; T), which is the uniform distribution over all balanced partitions with size T:
PII: Unif (III (m' mmm.....m' m Vt [T]))
(12)

---

Table 1: FIDs, ISs, Precisions, and Recalls for class-conditional generation on ImageNet 7 denotes the use of pretrained classifier for rejection sampling, gradient guidance, or training.
Params
W x
FID4
ist
Precision?
Recallf
BigGAN- deep 5
112M
6.95
202.6
0.87
0.23
StyleGAN XL
29
166M
2.3
262.1
0.78
0.53
ADM 9
554M
10.94
101.0
0.69
0.63
ADM-G 0
608M
4.59
186.7
0.82
0.52
ImageBART
3.5B
16x16x1
21.19
VQ-Diffusion
518M
16x16x1
11.89
LDM-8 27
395M
32x32
15.51
79.03
0.65
0.63
LDM-8-G 27
506M
32x32
7.76
209.52
0.84
0.35
MaskGIT 6
227M
16x16x1
6.18
182.1
0.80
0.51
VQ-GAN [3
1.4B
16x
15.78
74.3
RQ-Transfommer
1.4B
8x8x4
8.71
119.0
0.71
0.58
ro-Transformer
3.8B
8x8x4
7.55
0.73
0.58
RQ- '
2
3.8B
8x8x4
3,80
323.7
0,82
0.50
Contextual RQ- To-Trrnsformer
333M
8x8x4
5,45
172.6
0.81
0,49
Contextual RQ-Transformer
821M
8x8x4
3.45
221.9
0.82
0,52
Contextual RQ-Transformer
1.4B
8x8x4
3.41
224.6
0.79
0.54
Validation Data
1.62
234.0
0.75
0.67
We first define a procedure UPDATE(S, IT) to update the sequence S as described in Algorithm which updates S,, with m 1 for (T]. Then, Draft-and-Revés decoding in Algorithm generates a draft from the empty sequence of code stacks and improves the quality of the draft.
Draft phase the draft phase, our model gradually infills the empty sequence of code stacks to generate draft image, considering the global contexts of infilled code stacks. Let Sempty be empty sequence of code stacks with St [MASK]] for all n. Given partition size Taran our model generates a draft image as
Sdraft = UPDATE cmmpyy II; 0) where II ~ p(I;; Tdma..
(13)
Revise phase The generated draft Sdmalt is repeatedly revised to improve the visual quality of the image, while preserving the overall structure of the draft. Given partition size Trevise and the number of updates M, the draft S = Sdraft is repeatedly updated M times as
S UPDATE(S m II; where II p(II; Treviee for m = 1, -..... M.
(14)
Note that Draft-and-Revis is a tailored method, since can adopt any mask-infilling-basee generation method 4 6 for UPDATE in Algorithm For example, confidence-basee decoding [6] 16), which iteratively updates from high-confidence to low-confidence predictions, can be used for UPDATE. However, we find that confidence-baute oversimplified contents, since model tends to predict simple visual patterns with high confidence. In addition, confidence-based decoding often leads to biased unmasking patterns, which are not used in training. We attach the detailed discussion about confidence-bant use uniformly random partition II in UPDATE as the most simplified rule, leaving investigations on sophisticated update methods as future work.
decoding generates low vivviivessity images with
decoding in Appendix Thus, we
We postulate that our Draft-and-Revisee can generate high-quality images with diverse contents by explicitly dividing two phases. Specifically, a model first generates draft images with diverse visual contents despite the rather low quality of drafts. After semantically diverse images are generated as drafts, we use sampling strategies such temperature scaling [19] and classifier-fre guidance [20] in the revise phase to improve the visual quality of the drafts, while preserving the major semantic contents in drafts. Thus, our method can improve the performance image generation by effectively controlling the quality-diversity trade-off. In addition, intuitive and resembles the image generation process of human experts, who repeatedly refine their works to improve the quality after determining the overall contents first.
emphasize that the two-phased decoding is

---

Figure 3: The examples of generated 256x 256 images of our model trained on (Top) ImageNet and (Bottom) CC-3M. The used text conditions are "Sunset over the skyline of a (beach, city).". "an avocado {in the desert, the seashore)." and painting of (dog, cat) with sunglasses.".
Experiments
In this section, show that Draft-and- Revise with Contextual RQ-Transformer outperform previous approaches for class- and text-conditional image generation. In addition, we conduct an extensive ablation study to understand the effects of Draft-and-Reviss decoding on the quality and diversity of generated images, and the sampling speed. We use the publicly released RQ-VAE [23 represent
resolution of images Contextual RQ-Transformer have the same model size as the previous RQ-Transformer [2 For training, the quarter-perion cosine is used as the mask scheduling function y Eq. 10 following the previous studies 66 [4]. We include the implementation details in Appendix.
8x8x4 codes. For a fair comparison,
make
4.1 Class-conditional Image Generation
We train Contextual RQ- T0-rrsssfrrerr with 333M, 821M, and 1.4B parameters on ImageNet 7 for class-conditional image generation. For Draft-and-Revise decoding, we use Taran = 64, Trevise 2, and M We temperature scaling [19] and classifier-free guidance (20) only in the revise phase, while none of the strategies are applied in the draft phase. Fréchet Inception Distance (FID) [18]
Inception Score (IS) (28) and Precision and Recall 22 are used for evaluation measures.
Table D shows that Contextual RQ- TQ-Transformer significantly outperforms the previous approaches. Notably, Contextual NQ-Transformer with 333M parameters outperforms RQ- 20-Transffommrrs with 1.4B and 3.8B parameters on all evaluation measures, despite having only about 4.2 and 11.4x fewer parameters. In addition, the performance is improved as the number of parameters increases to 821M and 1.4B. Our model with 333M parameters is competitive with MaskGIT [6. and hence we conduct additional analysis the effect of confidence-basee decoding in Appendix C. Contextual RQ-Transformer can achieve the lower FID score without a pretrained classifier than ADM-G and 3.8B parameters of Transfoofrrr with the of pretrained classifier. StyleGAN-XI. also a pretrained classifier during both training and image generation and achieves the lowest FID in Table However, our model with 1.4B parameters has higher precision and recall than StyleGAN- XL, implying that our model generates images of better fidelity and diversity without pretrained classifier. Our high performance without classifier is remarkable, since the gradient guidance and rejection sampling are the tailored techniques to the model- based evaluation metrics in Table [] Considering that the performance marginally improved as the number of parameters increases from 821M to 1.4B, an improved RQ-VAL VAE can boost the performance of Contextual RQ- T0aassfoorrrr, since the reconstruction quality determines the best results of generated images.
4.2 Text-conditional Image Generation
We train our model with 333M and 654M parameters on CC-3M 32 for text-to-image (T2I) generation described in Appendix A. A text condition is encoded into 32 tokens using Byte Pair Encoding [31 34]. We report CLIP-score 25 with ViT-B/32 III as the correspondence between
and images.

---

Draft only
15.0
50
Revise
40
Precision
0.8
0.6
Draft only Revise, M -11 Revise,
Precision Recall
30
20
0.7
0.5
10
2.5
16
32
64
5
8
32
64
(M)
(a)
(b)
(c)
Figure 4: Ablation study of Draft-and- Revise decoding. (a) FIDs with the partition sizes of the draft phase Tran- (b) Precision and recall with the revision iterations M. (c) FID with revise.
Figure 5: Examples of generated images in the draft phase (left) and revise phases at each iteration
M = 1,2,3,4, 4. The draft images are generated with Tiratt 8 (top) and Taran 64 (bottom).
Contextual RQ-Trassforrer in Table 2 out-
Table FIDs and CLIP scores on the valida- tion dataset of CC-3M [32 for T2I generation.
performs the previous T2I generation models. Contextual RQ-Transformer with 333M param- eters achieves better FID than RQ-Transformer with 654M parameters, and outperforms Im- ageBART and LDM-4, although our model has 12x fewer parameters than ImageBART. When we increase the number of parameters to 654M, model achieves state-of-the-art FID CC- 3M. In Figure 3 our model generates images with unseen texts in CC-3M.
Params
FIDJ
CLIP-st
VQ-GAN [13
600M
28.86
0.20
ImageBART
2.8B
22.61
0.23
LDM-4 27
645M
17.01
0.24
RQ- RQ-Transformer 23
654M
12.33
0.26
Ours
333M
10.44
0.26
Ours
654M
9.80
0.26
4.3 Conditional Image Inpainting
We conduct conditional image inpainting where a model infills a masked area according to the given condition and contexts. Figure[sshow shows example of image inpainting by RQ-Transformer (middle) and Contextual TQ-nnansformer (right), when the class-condition is school bus. RQ- CQ-Transformer cannot attend to the right and bottom sides of the masked area and fails to generate a coherent image with given contexts. However, our model can complete the image to be coherent with given contexts by exploiting global contexts. We attach more examples of image inpainting Appendix.
4.4 Ablation Study on Draft-and-Revin
We conduct an extensive ablation study demonstrate the effectiveness Braft-and-Revis decoding of our framework. We use Contextual RQ-Transformer with 821M parameters trained on ImageNet.
Quality improvement of draft images the revise phase Figure shows the effects of T Taran on draft images and their quality improvement in the revised phase with Trevis and M = 2.

---

In the draft phase, FID is improved as Taran increases from 4 to 64, At each inference, Contextual RQ- Transformer generates a/Tarann code stacks in parallel, starting with the empty sequence. Thus, the model with a large Taran generates small number of code stacks at each inference and can avoid generating incoherent code stacks in the early stage of the draft phase. Although FIDs in the draft phase are worse due to low precision, we intend to increase the diversity of generated images, since the precision is significantly improved in the revise phase as shown in Figure([jb)
Efffect of revision iteration M and partition size Tresise in the revise phase Figure (b) shows the effects of the number of updates M in the revise phase on the quality and diversity of generated images. Since the quality-diversity trade-off exists as the updates are repeated, we select M = 2 as the default hyperparameter to balance the precision and recall, considering that the increase of precision starts to slow down. Interestingly. Shows shows that the overall contents remain unchanged even after M > 2. Thus, we claim that Draft-and- Revise decoding does not harm the perceptual diversity of generated images throughout the revise phase despite the consistent deterioration of recall.
Figure ()) shows the effects of Tevise on the quality of generated images. The FIDs are significantly improved in the revise phase regardless of the choice of Tevise. but increasing T Tevie slightly dete- riorates FIDs. We remark that some code stacks of a draft can be erroneous due to its low quality, and a model with large Trevise slowly updates small number of code stacks at once the revise phase. Therefore, the updates with large Tevise can be more influenced by the erroneous code stacks. Although Trevise = updates half of an image at once, our draft-and-revise decoding successfully improves the quality of generated images, while preserving the global contexts of drafts, as shown in Figure 5 The study on self-supervised learning [17] also reports similar results, where a masked auto-encoder reconstructs the global contexts of an image after masking half of the image.
Quality-diversity control of Drevt-and-Revise Our Draft-and-Reve decoding can effectively control the quality-diversity trade-off in generated images. Table 3 shows FID, precision (P), and recall (R) according to the use of classifier-free guidance (20] with scale of
Table 3: The effects of classifier-free
guidance on the image generation.
Draft
Revise
FID
P
R
1.8, while applying temperature scaling with 0.8 only to the revise phase. Contextual RQ-rransformer without the guidance already outperforms -Q-Transformer with 3.8B parameters and demonstrates the effectiveness of our framework. When the guidance is used for both draft and revise phases, the precision dramatically increases but the recall decreases to 0.33. Consequently. FID becomes worse due to the lack of diversity in generated images. However, when the guidance is applied only the revise phase, our model achieves the lowest FID, as the quality and diversity are well-balanced. Thus, the explicitly separated two phases of Draft-and-Revic can effectively control the issue of quality-diversil trade-off by generating diverse drafts and then improving their quality.
5.78
0.72
0.58
3.45
0.82
0.52
8.90
0.92
0.33
Trade-off between quality and sampling speed Table 4 shows that our framework can control the trade-off between the quality and sampling speed according to Taran after we
Table 4: Comparison of FID and the sampling speed of image generation.
fix Previse = 2 and M = 2. We generate 5,000 samples with batch size of 100 as the previous study (23). Contextual RQ-Transformer with Taaaf = 8 outperforms both FID and sampling speed of VQGAN and RQ-Transformer with 1.4B parameters. Although the sampling speed becomes slow with increased Taran. the FID scores are consistently im- proved. We remark that the sampling speed with Tara = 64 about 3x slower than RQ-Transformer, but our model outperforms 3.8B parameters of Transfoofrr with rejec- tion sampling in Table The results show that our framework has inexpensive costs to generate high-quality images, considering that rejection sampling requires generating up 20x more samples.
FID
s/sample
VOGAN RQ- Transformer
15.78
0.16
8.71
0.04
Contextual RQ- RQ-Transformer
Tanat
5.41
0.03
Tànn 32
3.73
0.06
Tarm = 64
3.45
0.10

---

Conclusion
In this study, we have proposed Draft-and-RR Contextual Transfforrrr. After an image is represented sequence of code stacks, Contextual RQ- -Q-Transformer is trained to infill a randomly masked sequence. Then, Traft-and-Revine decoding is used generate high-quality images by first generating draft image with diverse contents and then improving its visual quality based on the global contexts of the draft. Consequently, we can achieve state-of-the-art results on ImageNet and CC-3M, demonstrating the effectiveness of our framework.
for an effective image generation framework with
Our study has two main limitations to be further explored. Firstly, Draft-and- Revise decoding always updates all code stacks in the revise phase, although some code stacks might not need an update. In future work, a selective method can be developed to improve the efficiency of the revise phase by a sophisticated approach. Secondly, our generative model is not validated various downstream tasks. Since masked token modeling is successful self-supervised learning for texts g and images [ 17], unified model for both generative and discriminative tasks [21] is worth exploration for future work.
Acknowledgements
This work was supported by Institute of Information & communications Technology Planning (valuation(IITP) grant funded by the Korea govemmenn(MSTT) (No.2018-0-0139)) Development of a Conversational, Self-tuning DBMS, 35%; No. No.2021-0-00533 Visual Common Sense, 30%) and the National Research Foundation of Korea(NRF) grant funded by the Korea govemment(MSIT) (No. TRI-2022RII22B3000000 35%)
References
[1] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-17993, 2021.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1eXiv:1607) 06450, 2016.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEit: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022.
[4] Sam Bond- Bond-Taylor, Peter Hessey, Hiroshi Sasaki, Toby Breckon, and Chris G Willcocks. Un- leashing transformers: Parallel token prediction with discrete absorbing diffusion for fast high- resolution image generation from vector rector-quantized codes. arXiv preprint arXiv:2111.12701, 2021.
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019.
[6] Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[7] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, lece, 2009.
[8] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, 2019.
[9] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
10

---

[10] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang. et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
[11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[12] Patrick Esser, Robin Rombach, Andreas Blattmann, and Bjorn Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
[13] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12873-12883, 2021.
[14] Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Mask-predice Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNIP), pages 61 6122-1121, 2019.
[15] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2022.
[16] Ligong Han, Jian Ren, Hsin- Ying Lee, Francesco Barbieri, Kyle Olszewski, Shervin Minaee, Dimitris Metaxas, and Sergey Tulyakov. Show me what and tell me how: Video synthesis via multimodal conditioning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377 2021.
[18] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[19] Geoffrey Hinton, Oriol Vinyals. Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv: 503.000311,,2210 2015.
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
[21] Saehoon Kim, Sungwoong Kim, and Juho Lee. Hybrid generative-ccontrattie representation learning. arXiv preprint arXiv:2 106.06/62, 2021.
[22] Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[23] Doyup Lee, Chiheon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[24] Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[25] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748-87666 PMIR, 18-24 Jul 2021.
11

---

[26] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ily Sutskever. Zero-shot text-to-image generation. Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821-8831. PMIR, 18-24 Jul 2021.
[27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[28] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett editors, Advances in Neural Information Processing Systems, volume 29, Curran Associates, Inc., 2016.
[29] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. arXiv preprint arXiv:2202.00277. 2022.
[30] Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled denoising autoencoders for text generation. In International Conference Learning Representations, 2022.
[31] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 1715-1725, 2016.
[32] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymedd, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, Melbourne, Australia, July 2018. Association for Computational Linguistics.
[33] Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learning. In Guyon, U. Luxburg, Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[34] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[35] Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang. Jingren Zhou, and Hongxia Yang. UFC-BERT: Unifying multi-modal controls for conditional image synthesis. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
12


Reviews:
[{"review_id": "xFkVuioVwkI", "reviewer": "Reviewer_EFXa", "summary": "This work proposed a two-stage framework for image synthesis, which consists of RQ-VAE and contextual RQ-Transformer. The proposed framework is inherited from the prior work [23] but modifies the original auto-regressive pipeline with a bidirectional pipeline borrowed from BERT.", "questions": "Please address my concerns shown in the weaknesses.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros. \n+ The results are good. Compared with RQ-VAE, the proposed method achieves better FID with smaller model sizes. But the results are not as good as GAN-based approaches such as styleGAN-XL.\n+ The writing is easy to follow.\n\nCons. \n- Lack of in-depth analysis and strong motivation for the proposed method. Both RQ-VAE [23] and the proposed method used an RQVAE as a tokenizer. The difference lies in the second-stage transformer that produces the RQ tokens. In [23], the tokens are generated in an auto-regressive way, while in this submission the tokens are generated via masked image modeling as BERT. By combining RQVAE and BERT, the proposed method is claimed to perform better than existing approaches. However, it is still unclear how the bidirectional masked token modeling outperformed autoregressive modeling. More ablation studies are preferred.\n- Some details are missing. Implementation details on text-conditional image synthesis are missing.\n- The paper reported class-conditional results on ImageNet. How about unconditional image synthesis performance?  It would be better to provide results on diverse datasets such as Imagenet and FFHQ. Also, the reported results are in a resolution of 256x256, and one concern is the quality in higher resolutions such as 1024x1024.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "nhQZe62x73P", "reviewer": "Reviewer_uiaY", "summary": "This paper presents a Draft-and-Revise framework with a novel contextual RQ-Transformer module for the task of image generation. I think the proposed model is an extension of the previous method \"RQ-VAE + RQ-Transformer\" (Lee et al., Autoregressive Image Generation using Residual Quantization, CVPR 2022). Specifically, the newly proposed model changes the paradigm of AR model that gradually generate an image by attending to only precedent codes generated. It introduces random masking and gradually generate an image by attending to contextual codes. In addition, the original Spatial Transformer is replaced by Bidirectional Spatial Transformer and a draft-and-revise strategy is introduced to further improve the performance.", "questions": "See weaknesses above.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The idea is interesting, it draws from CMLMs (Ghazvininejad et al., Mask-predict: Parallel decoding of conditional masked language models, EMNLP-IJCNLP, 2019) and skillfully combines with previous work to form an effective image generation model. \n+ The experiments verify the effectiveness of the proposed method in both class-conditional and text-conditional image generation tasks.\n+ The presentation is clear and well organized.\n\nWeaknesses:\n\n+ The effectiveness of the proposed model seems to largely depend on the existing model \"RQ-VAE + RQ-Transformer\" (Lee et al., Autoregressive Image Generation using Residual Quantization, CVPR 2022), the experimental results also seem to illustrate this point. \n+ In addition, the technical contribution of this method seems to be insufficient, mainly using the technical route of the previous method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lj7kKfhDM2o", "reviewer": "Reviewer_7o7S", "summary": "The paper achieves state-of-the-art generation quality on the ImageNet and CC-3M text2image synthesis task by proposing a new transformer architecture that models spatial attention and depth attention of Residual Quantization, and iteratively revising the generated image by randomly masking the output image and re-synthesizing the masked region. ", "questions": "Why does [Draft only] achieve such bad FID? It looks like FID ranges from 50+ to 15 in Figure4(a). Shouldn't it still achieve similar FID as MaskGIT or RQ-Transformer? It would be nice if Contextual RQ-Transformer is also evaluated in Table 1 without the revise step. To me, Contextual RQ-Transformer sounds like the architecture described in the upper portion of Figure 2 without the revise step. So it is unclear if [Contextual RQ-Transformer] includes the revise step in Table 1. ", "limitations": "The paper does not seem to discuss the societal impact of the paper. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength. \n\n* The paper achieves SOTA result on ImageNet synthesis, while making the inference time faster than previous methods including VQGAN. \n* I find the idea of revising the generated image very interesting. The paper adds the notion of corse-to-fine refinement to transformer-based autoregressive models, in both the depth transformer of RQ and the revise step. There existed conceptual gap between AR and Diffusion-based models, in that AR models performed iterative generation spatially, and the diffusion-based models performed iterative generation in a coarse-to-fine manner. The proposed method developed a model that can control the iterative refinement in both axes independently, by changing T_draft and M. \n* I like ablations and investigations including Figure 4. \n* There is adequate explanation of previous literature, including Section A.1. \n\nWeakness\n\n* I found the exposition a bit difficult to follow. The paper introduces many new notations like T, D, M, and Pi. It would be nice if they are accompanied by textual description rather than just referring to them as symbols. \n* The paper puts emphasis on parallel decoding of features, but in the end, the best performing model is using T_draft = 64, which doesn't use parallel decoding in the draft phase. This is confusing in some sense. Moreover, the paper could discuss more on its similarity to MaskGIT. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TKPjeuNuTlX", "reviewer": "Reviewer_bucK", "summary": "This paper pointed out the limitation of the autoregressive image generation using the existing transformer based on the vector quantization methods and proposed the contextual RQ-transformer is trained to infer the masked code stacks by bidirectional self-attention like BERT in order to solve it. Furthermore, they suggested the two-phase decoding, Draft to fill the masked codes and Revise to increase the image quality, in order to break the trade-off between quality and diversity. They demonstrated their significant performances in class-conditional/text-conditional image generation as well as class-conditional image inpainting experiments.", "questions": "Overall, the proposed method is good, and the performance has also been well proven through many experiments. However, it is questionable that the novelty, which is the most important, is not enough and does not explicitly specify the differentiation from MaskGIT, which is the most like the problem raised. Being able to clearly explain the weakness mentioned above will be the biggest key to the final rating. ", "limitations": "Please see the weaknesses and questions.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Overall clear presentation and discussion on the key argument/idea. \n- The justifications and motivations are clearly established. Especially, the draft-and revise process that imitates the image generation process of a human expert is interesting. \n- Overall, both qualitative and quantitative performance improvements over existing methods are convincing in various experiments.\n\n### Weaknesses\n- Novelty: It seems only to the extent of introducing the limitation and solution proposed by MaskGIT[6] to RQ-Transformer. Especially, for introducing the bidirectional transformer to the VQ scheme, the most novel part is the mask generation algorithm; MaskGIT first proposed this using VQVAE. However, this paper simply applies it to RQ-VAE in the draft phase and does not present any special novelty.\n For the above reasons, the comparison with MaskGIT is indispensable to verify their novelty. However, they hardly mentioned MaskGIT in both Related work and Experiment sessions, while highlighting only the merits in comparison with the autoregressive method (i.e., VQGAN) and the Diffusion methods.  \n  > In Tabel 1, MaskGIT shows almost similar scores compared to Contextural RQ-Transformer, although it uses fewer parameters. However, there is a lack of analysis with MaskGIT.  \n\n  > They highlight their inference speed compared with VQGAN at the 'trade-off between quality and sampling speed' sub-session. However, as far as I know, MaskGIT requires less iteration compared to Contextural RQ-Transformer in creation (Maskgit needs 16 iterations for a generation while Contextural RQ-Transformer needs T_draft = 64, T_revise = 2, M = 2.)    \n\n I agree that the improvement of performance through several experiments is convincing. However, I believe that in-depth comparison and analysis in terms of motivation or performance with MaskGIT are essential to demonstrate their novelty.\n\n- Revise phase: Another novelty of the paper is the Revise phase. However, it lacks a clear rationale for why the performance improves because they only perform a simple random uniform sampling. In the Revise phase, what percentage of the code to revise is an important factor (i.e., the number of codes for re-masking), but there is no mention of it. Please provide more details about the Revise phase to improve the novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept