Locally Hierarchical Auto-Regressive Modeling for Image Generation
Authors: Tackgeun You
Saehoon Kim
Chiheon Kim
Doyup Lee
Bohyung Han
Venue: NeurIPS 2022 Accept (2022)

Abstract: We propose a locally hierarchical auto-regressive model with multiple resolutions of discrete codes. In the first stage of our algorithm, we represent an image with a pyramid of codes using Hierarchically Quantized Variational AutoEncoder (HQ-VAE), which disentangles the information contained in the multi-level codes. For an example of two-level codes, we create two separate pathways to carry high-level coarse structures of input images using top codes while compensating for missing fine details by constructing a residual connection for bottom codes. An appropriate selection of resizing operations for code embedding maps enables top codes to capture maximal information within images and the first stage algorithm achieves better performance on both vector quantization and image generation. The second stage adopts Hierarchically Quantized Transformer (HQ-Transformer) to process a sequence of local pyramids, which consist of a single top code and its corresponding bottom codes. Contrary to other hierarchical models, we sample bottom codes in parallel by exploiting the conditional independence assumption on the bottom codes. This assumption is naturally harvested from our first-stage model, HQ-VAE, where the bottom code learns to describe local details. On class-conditional and text-conditional generation benchmarks, our model shows competitive performance to previous AR models in terms of fidelity of generated images while enjoying lighter computational budgets.

Locally Hierarchical Auto-Regressive Modeling for Image Generation
Tackgeun You
Saehoon Kim
tackgeun.c yww.posouuuppocccccomoo kc
shkim@kakaobrain com
Chiheon Kim*
Doyup Lee
Bohyung Han 1.2.3
chiheon, www.kaaaobrriinnccmm..m com
doyup.Lee@kakaoraate com
bhhan&snu, ac
'ECE PIPAI, AAIIS, Scoul National University, Korea
*Kakao Brain, Korea
$CSE, POSTECH, Korea
Abstract
We propose locally hierarchical auto-regressive model with multiple resolutions discrete codes. In the first stage of our algorithm, we represent an image with a pyramid of codes using Hierarchically Quantized Variational AutoEncoder (HQ- VAE). which disentangles the information contained in the multi-level codes. For example of two-level codes, create separate pathways to carry high-level coarse structures of input images using top codes while compensating for missing fine details by constructing residual connection for bottom codes. An appropriate selection of resizing operations for code embedding maps enables top codes to capture maximal information within images and the first stage algorithm achieves better performance on both vector quantization and image generation. The second stage adopts Hierarchically Quantized Transformer (iQ--ransformer) to process a sequence of local pyramids, which consist of a single top code and its corresponding bottom codes. Contrary to other hicrarchical models, we sample bottom codes in parallel by exploiting the conditional independence assumption on the bottom codes. This assumption naturally harvested from our first-stage model, HQ-VAR HQ-, where the bottom code learns to describe local details. On class-conditional and text-conditional generation benchmarks, our model shows competitive performance to previous AR models in terms of fidelity of generated images while enjoying lighter computational budgets.
Introduction
Deep generative models 22 3 have become the fundamental basis for synthesizing diverse images of high-resolution from various forms of input supervision, including class conditions 2 4) text prompts [5] 6), and segmentation maps [7] to name a few. Most approaches in this paradigm roughly categorized into two groups depending on whether they explicitly maximize data likelihood or not. GANs implicitly estimate underlying data distribution to generate novel and crispy samples, but it typically suffers from a lack of diversity D. In contrast, it is widely accepted that likelihood- models are capable of generating diverse examples, compared to GANs, since they aim to cover all modes of a data distribution g This work focuses on auto-regressive models among many likelihood-based approaches motivated by the existing works validating scalability [5] 9
Auto-Regressive (AR) models factorize a data likelihood into product conditionals the pre- defined generation order. Since AR models already have mode-coverage property, we only need
"This work was done during internship of this author at Kakao Brain.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(909)
(885))
17180
R248
(4)
Illustraioon of generated images (the last row) by our algorithm trained on ImageNet, where the first row shows the examples synthesized by the top codes only while the second row presents the images from the bottom codes only.
Figure I:
to make minimal changes in architectures and optimization methods for the applications to various domains including audio, language, and images. However, regardless of the modality, training AR models in a high-dimensional space is computationally expensive since each conditional needs to consider many past observations. In context of image generation, VQ-VAE [IO] introduces two-stage learning process, where the first stage represents an image by discrete code map while the second stage constructs AR priors [ on discrete sequences. To make this two-stage framework effective, sequence of the discrete codes has to hold sufficient information for high- quality reconstruction an original image while the sequence length should be reasonably short for efficient training of AR models.
In this study, we postulate that leveraging the multi-scale structure is simple but effective way for representing an image as discrete codes with AR modeling. We aim to disentangle the information residing in the hierarchy and design an efficient model to generate unseen images by exploiting the multi-level structure maximally. To achieve this goal, we propose locally hierarchical auto-regressive modeling by extending two-stage AR modeling. We first extend VQ- VAE to produce two-level pyramid of discrete codes, where the top code contains high-level coarse information and the bottom encodes the low-level fine details. Using these locally hicrarchical discrete representations, our efficient AR model generates novel example. Specifically, the top codes are sequentially generated predefined order, and multiple bottom codes associated with a top code are decoded in parallel. The main contributions of the proposed model are summarized below:
We propose HQ- VAEV (Hierarchically Quantized Variational Auto-Encoder). which encodes an image using pyramid of two-level codes through hierarchical vector quantization. To improve the utility of the two-level codes, we disentangle the information contained in the codes, where top codes capture high-level structural information in images while bottom codes carry the remaining low-level details.
We introduce HQ-Transformer (Hierarchically Quantized Transformer). which is an AR model to generate images based on hierarchical discrete codes. We predict a sequence of locally hierarchical codes efficiently by maximizing its likelihood based on the conditional independence assumption imposed on bottom codes.
The proposed algorithm achieves competitive performance on class- and text-conditional image generation tasks while yielding high throughput in sample generation, compared to various types of generative models.
Figure T fillustrates several examples of generated images by the proposed algorithm. We also present the synthesized images based only on top and bottom codes together, which clearly exhibits the proper disentanglement of image contents and low-level details.

---

2
Related Works
This section describes existing approaches closely related our algorithm and highlights the differ- ences between the prior works and ours.
Discrete representations for autoregressive models Learning AR priors based on discrete repre- sentations turn out to be effective in language modeling (9 [2]. VQ-VAE [0] pioneers the two-stage image generation process, where the first stage represents an image by a set of discrete codes and the next stage constructs AR priors 2 II] on the discrete sequences. Following VQ- VQ-VAE, many approaches aim to further reduce the sequence length and improve the architecture [33 14] or training objectives 77] of the first stage. Specifically, VQ-GAN shortens the sequence length by using the perceptual and adversarial losses and encoding the semantic information in the codes effectively. However, these approaches still exhibit the fundamental trade-off between code sequence length and generated image quality due to a lack of expressiveness of the single-scale vector quantization with the reduced length of the code sequence.
Hierarchical representations for autoregressive models The construction of hierarchical codes an effective way to control the fundamental trade-off related the length of the code sequence. There exist several image representation and generation methods based on hierarchical discrete codes 15 16]. VQ- VAE-2 8 and HAM [5] make use of the hierarchical relationship between the codes for unseen image generations in a coarse-to-fine manner. In particular, to tackle the trade- off issue, VQ-VAE-2 8 allocates heavier model for processing coarse top codes and a lighter model for dealing with fine bottom codes. RQ-Transformer [1 encodes image into stack of code maps by iteratively quantizing residual vectors between latent features and code embeddings, effectively reducing the spatial resolution of the code maps without degrading reconstruction quality. Hierarchical VQ-VAF [T6] encodes structural and textual information of an image using different types of codes by training two separate VQ-VAEs where one learns with the original images and the other employs the residuals between the original and reconstructed images. The main idea of Hierarchical VAE is about how to disentangle the information across the code levels but not about how to train AR models based on this set of codes. Our approach constructs a pyramid of discrete codes and exploits the codes using AR models to generate
images.
iteratively denoise sample from white noise by seeing the contexts in a bidirectional manner. Most diffusion models have been implemented based on the U-Net [20) style architecture, meaning that the multi-scale feature of an image has been naturally incorporated. However, they suffer from low throughput in the generation process because, even though the inference requires a lot of iterative steps, the intermediate computation results between the denoising steps are not reusable, unlike AR models.
Diffusion models Diffusion models 3 18 19)
3 Locally Hierarchical Auto-Regressive Models
This section describes the details of the proposed method for image generation. Our approach exploits two levels of codes-top and bottom-for the representation of an image contrary to existing two- stage approaches based on AR models. The proposed architecture encourages the top code to model the overall structure of an image while facilitating the bottom code to complement the remaining details of the image. To this end, we propose Hierarchically Quantized Variational AutoEncoder (HQ-VAE) for the image vector quantization in the first stage followed by -O--ransformer to learn the distribution over sequence of the top and bottom codes in the second stage. HQ-Tranaee Tro-nsssformer achieves better perceptual quality and sampling speed using the underlying hierarchical structure. Figure illustrates the two-stage procedure of our algorithm based AR model.
3.1 Stage 1: Hierarchically Quantized VAE (HQ-VAE)
We present the architecture and the optimization method of the proposed VAE, for which we first review the vector quantization technique and then discuss our hierarchical quantization module.
3.1.1
Notations for vector quantization
Vector quantization 21 maps a d-dimensional continuous feature vector to a discrete code, which given by identifying the nearest code embedding of the feature vector in codebook. The codebook
3

---

STAGE (HQ-VAE)
STAGE (HO-Trande Transformer)
Bidirectional Attantion Blocks
Encoder
Mixed Attention Blocks
Downsample
Ag
h
A
Image (256x256)
Quantizer
Prediction Head
exty
Main Transformer
Transformer
Upsample
Decodler
Quantizer
sos
U2
PE
PE
PE
PE
PruM
Paus
Reconstruction
5
Suxi
Input Embedding Transformer
5
Pazy
Prusi
Top code (2x2)
Bottom code (4xx4)
C
bull
Figure 2: Conceptual illustration of the overall architecture: (a) components of HQ-VAE with residual and resampling operations encode different information levels in the top and bottom codes, and (b) components of HQ- Transssfossms to train an AR prior on the top and bottom codes.
collection of tuples C (ss e²) where e, Rd denotes the embedding of code and ne is the size of the codebook C. Supposing that Zij is a d-dimensional vector at a spatial location (i.j) in a feature map E RR. I code 8ij is given by
B1j = (z.y C) = argmin Zig - C[s]]l+
where C[.] returns the code embedding of an input code and (-; C) denotes the quantization process based on a codebook C. Note that, since every element a feature map has its own code, the spatial resolution of the code map, denoted by is the same as that of the feature map.
3.1.2 Hierarchical vector quantization
Our approach adopts hierarchical vector quantization scheme encode input data using two levels of discrete codes top and bottom denoted by and b, respectively. Ideally, the combination of the two-level codes is expected maintain the full representation power of original feature map. In addition, we desire the two kinds of codes to have sufficiently disentangled and heterogeneous information; for example, top codes hold the high-level concept of inputs while bottom ones represent remaining low-level details. To these ends, our network creates two separate pathways designated for the top and bottom codes and makes the branch for bottom codes constrained to have residual information. In specific, our hierarchical vector quantization transforms feature map |g into a tuple of two code maps (t, b), where t zixl and E zrixrl with an integer scaling factor E (1.
Our scheme first captures the high-level information of feature map by quantizing downsampled version using the top codes and computing their embeddings as follows:
top Downsample(z:r),
taj = - (Z CLop)
e = (topl
where (ooP denotes the codebook of top codes and Downsample(.) is generic pooling operation to reduce the spatial resolution feature map. Given the top code map t, the bottom codes and its embeddings derived by
z bot = z Upsample(e"P;:))
bij = huotttt bot abot
e,y = cho
(3)
where ghot the codebook of bottom codes and Upsample(-) any upsampling operation to increase the spatial resolution of feature map. Figure [left) illustrates this hierarchical quantization procedure. Note the residual vector between the original feature map and the upsampled top code embedding map, denoted by bot, involves sampling and quantization errors. The bottom codes with their embeddings should compensate for the errors and improve the reconstruction quality the original feature map when combined with the tep-level information.

---

D
Di (ee)
Encoder
Encoder
Die
Die
Aug. Pool
PleelUnShuffle
8x8x
8x8x4X
D D(ee ebo)))
Quantizer
D( e
Quantizer
Nearest Interp.
Quantizer
Pisetshuflle
Quantizer
16 16nk
16x 16xk
Decoder
Decoder
original
original
(a) Average pooling and nearest interpolation
(b) Pixel-unshuffle and pixel-shuffle
Figure 3: Comparison different implementations of resampling operations in our hicrarchical vector quantization procedure. When the average pooling and nearest interpolation used for resizing as in (a), the reconstructed images from the top codes only are too blurry. On the other hand, resizing by pixel-unshuffle and pixel-shuffle operations leads to significantly better reconstruction quality in the same condition as shown (b).
One question is which operations are appropriate for Downsample()) and Upsample(-). We observe that non-learnable layers give favorable solutions for the resizing operations compared to naïve learnable counterparts. Straightforward choices for those operations are the average pooling and the interpolation with nearest neighbors. Despite the reasonable performance given by this strategy, Figure 3al exhibits that the overall structure reconstructed by the top code embeddings is below expectation. Instead, it turns out that a combination of pixel-unshufffe and pixel-shuffle operations [22] effectively pushes more information into the top codes while making the bottom codes focus local details. Given the feature map z E Rrixrixd and a scaling factor r, the pixel-unshuffe returns a tensor in the shape of / x x r2d, encoding high-level information of the original feature map. To compute the residual, we pixel-shuffle the top code embedding and restore the original dimensionality of Figure Jllearly demonstrates that the top codes given by the pixel-unshuffle followed by the quantization hold the primary information of an input image more than the ones given by the average pooling.
operation
3.1.3 HQ-VAE: architecture and optimization
Overall structure HQ- consists of three parts: encoder E(-), decoder D(-). and hierarchical quantizer ((::r), as shown in Figure left), where is the additional scaling factor for the top code. Given an image x, an encoder E(.) extracts the feature map z. Then, the hierarchical quantizer H((::)) is applied to the feature map, providing the top and bottom codes (t, b) and their associated code embeddings: elu and bot Now, the decoder D(-) takes the top and bottom code embeddings to reconstruct the input image.
To train three parts in HQ- VAE, our objective function is composed of the reconstruction loss introduced in VQ-VAR (10 as well as adversarial and perceptual losses to improve the fidelity of reconstructed images 7 Here, we only present the reconstruction loss as below:
Objective function
(x; D, x "T bot DI
Σ [[lsg(z') - e'112 Bl(se(e") z (4)
VE (top.bbb)))
where sg(-) refers to a stop-gradient operation, and is 0.25 suggested in The first term measures &g distance between the reconstructed and original images. The second term enforces both the top and bottom code embedding to located close the inputs of the quantizers. To update the codebook, we make use of exponential moving average (EMA) to accelerate convergence as discussed in
5

---

Implementation details We adopt the same network structures of the encoder and the decoder as VQ-GAN 7 except two minor modifications. To increase throughput training, we set the stride of the first convolution layer in the encoder and the last transposed convolution layer in the decoder as 2, instead of in the original implementation. We replace the batch normalization [23] in the discriminator by the group normalization [24 to achieve consistent results regardless of batch sizes.
3.2 Stage 2: Hierarchically Quantized Transformer (HQ-Transformer)
This section presents HQ-Transformer (see Figure (right)) to autoregressively model the top and bottom codes by fully exploiting their spatial relationships obtained from the first stage.
3.2.1 Formulation
Our objective is to maximize the log-likelihood of tuple of t and b. Given the scaling factor for the top, the resizing operation aligns single top code t; with r² bottom codes b(2,)), where 1 < h < r². Figure 2 iillustrates transformer part), where all bottom codes aligned with a single top code t, are b, = (b(,,)) b(a,r))). The likelihood of the top and bottom codes is given by
example of top and bottom codes with = (in the input embedding
1 Po (li, bilt,,t bei),
Po (t, b) Po (4 40, b(1.)) (2)
II ==1
(5)
where the inequality symbol (< i) denotes collection of all indices less than i. The conditional is further factorized into
Po (4, b,ltc bz.) = =P (b, 14, tec, bes) Pe (t. bes).
(6)
The last term in (6) maximized by predicting the current position given the previous top and bottom codes. By assuming that the bottom codes are conditionally independent given the associated top code, the first term of the right-hand side is formulated more efficiently as follows:
Pe (ba tt, tet bat) A
II Po (b(a,)) It, trci, ber),
Note that the assumption we made here reasonable because the bottom codes are supposed to contain local details while the top codes are enforced by our stage model, HQ-VAE, to encode global landscape.
3.2.2
Model
HQ-Transformer consists of three sub-modules based on the transformer architecture, which are referred to as Input Embedding Transformer (IET), Main Transformer (MT), and Prediction Head Transformer (PHT), as illustrated in Figure 2 right). IET generates an embedding to be used an input for MT, and the hidden state given by MT is fed into PHT to generate top and bottom codes, which are given back to IET in the next step. We discuss each of the three modules in the rest of this section.
Similar to GPT-like architectures 9 25
Input Embedding Transformer (IET) IET transforms tuple of (f:, b;) into single embedding vector u, Vi. To implement this, we first construct local embedding by summing up the code embedding and the positional encoding, which is given by
toplin + T [1] PPE
m;
= C
bot = botin (v.k)
+ PE [A],
for le = 1,
(5.k)
where e Japin and botio (a,k) are the learnable code embedding of t, and b(g,e)) respectively, and PE
and PE denote the learnable positional encoding for the top and bottom Note that the code
*P is introduced distinguish the top code from the bottom ones.

---

embeddings are learned within this IET module, by which we construct the codebooks Cropin and Choi,in These local embeddings are refined by bidirectional transformer,
Top hot u;
bot (2, IET
top m
m
hot (c,1)
bot m (1.m2)
(9)
where the structure is the same as GPT-2 9 and we employ single GPT-2 block composed of a self- attention followed by MLP block. Then, the final input embedding u, for the main Transformer block is simply given by averaging all updated local embeddings as
u; = Average(u, top (t.)))
hot u (2,r2)
(10)
Main Transformer (MT) This module updates the outputs of IET by a stack of NMT causal self-attention and MLP blocks:
ho, h1, ha-1 MSOS u1 U2. Upa
(11)
where the sequence length is equal to the number of top codes and the start-of- token (SOS) employed as the first input of the MT(-) function. This function also has the same structure as GPT-2.
Prediction Head Transformer (PHT) This module predicts the top and bottom codes from the hidden representation derived by the main transformer, where the bottom codes are generated in parallel based on their associated top code and a hidden representation from MT. In specific, we first define the input sequence of PHT below:
h; + Pe (k)
k= if 15k5
(12)
tophead PP (A)
where e; jop.head denotes the learnable code embedding of defined within PHT and P denotes the learnable positional encoding for top and associated bottom codes defined within PHT. This sequence then updated to produce the conditional probabilities of the top and bottom codes as follows:
P.+1 P
PV
(0) v
(13)
where each output probability is defined as
(eltten,bl) P {PP (b(,,) tei, bes)
if k= 0,
(14)
if 1sk
Note that we predict the bottom code distributions in parallel. In PHT, we employ self-attention top code and multiple bottom codes, which conditioned on
block of mixture of the positions the top position while the attention between bottom positions is unconditional. Compared to the full autoregressive modeling on the bottoms, such a parallel decoding scheme for bottom codes leads to speed-up in sampling and even better image generation quality. We will discuss its benefits in the experiments section.
3.2.3 Computational complexity of HQ-Transformer
Given the top code t zxx! and the bottom code b E grrxrl, where is scaling factor for the quantization of top codes, the length the top codes Lop 12 and the number codes local patch Lical = + yd because there exist , bottom codes associated with a single top code. The time complexities of IET, MT, and PHT are then given by IN O and T respectively, where NETT NMT. and Neir are the numbers of attention blocks in individual modules. The time complexity of MT dominates those of the other two modules because NMT larger than NIET and NрHт- Because the code map size of VQ-GAN is equal to the bottom code resolution of IQ--rranfformer the computational complexity of VQ-GAN, O L2g is higher than the complexity of HQ- TrQ-rrasformer, O (op

---

Table 1: Plass-conditional image generation performance on ImageNet in terms of FID, precision. recall, and throughput. The values in parentheses are obtained by rejection sampling.
Parameter
Code
Throughput
Models
FID 4
Precision
Recall f
size
resolution
images/sec
ADM 126
55TM
10.94 (4.59)
0.69 (0.2)
0.63 (0.52)
0.12 (0.07)
IniageBART
3.5
16'x 16
(7.44
VQ-Diffusion 19
370 M
32 32
11.89 (5.32)
2.66 (0.13)
I.DM-4 [8
400 M
64 x 64
10.56 (3.60)
0.71 (0.87)
0.62 (0.48)
0.70 (0.30)
MaskGIT 28
228 M
16 16
6.18 (4.02)
0.80
0.51
11.04 (0.55)
VQ-VAE-2 88
13.5
32 x 32 X 64
<- 10)
0.36
0.57
ViT-VOCAA
133
1.6
(3.04)
VQ-GAN 7
1.4.B
16
6.76 (0.34)
RO-Transformer
17
1.4
8 x 8 x
8.71 (3.89)
0.71
0,58
37.03 (1.85)
RQ-Transformer
3.8 B
8 x x 4
7.55 (3.80)
0.73 (0.82)
0.58 (0.50)
23.22 (2.90)
HQ-TVVE
530 M
X 8 + 16 X 16"
9.36 (4.98)
0.69 (0.78)
0.55 (0.51)
78.87 (19.72)
HQ-TVAE (M)
M
X + 16 x 16
8.46 (4.47)
0.71 (0.79)
0.56 (0.52)
70.00 (17.50)
HQ-TVAR (L.
1.4
8
16
7.15 (4.35)
0.73 (0.81)
0.55 (0.50)
49.44 (12.36)
Table 2: Text-conditional image generation performance on the CC3M validation set.
Models
Parameter size
Code map resolution
FID
CLIP-score
Throughpur
ImageBART [27
2.8
x 16
22.61
0.23
1.75
LDM (18]
645 M
64 x 64
17.01
0.24
3.9
VQ-GAN 7
1.5
16 x 16
28.86
0.20
2.69
RQ- Trrnssnffrrmrr
17
654 M
8 8
12.33
0.26
46.47
(IQTTVA
379
8
16 16
12.86
0.26
554.88
3.3 Sampling from HQ-Transformer and HQ-VAE
For inference, we first generate discrete codes using HQ-Transformer and synthesize new images using HQ-VAE based on the codes. We begin with the start-of-sentence (SOS) token and obtain its hidden representation from MT in -ro-rrnsformer, where the class-conditional SOS is employed in the case of class-cone conditionii generation. Then, PHT takes the hidden representation as its input embedding at the first step and predicts top code distribution. We sample top code from the distribution and forward it to the next step of PHT to obtain the bottom codes, which are actually acquired in parallel by forwarding the top code embedding the multiple next steps. The top and bottom codes are given to IET, which outputs the embedding for the input to MT. We repeat this procedure in auto-regressive manner until we obtain all the top and bottom codes. Once completing the generation of the codes, we finally produce a synthetic image using the decoder of HQ-VAE based on the discrete codes.
4 Experiments
We evaluate our models, referred as HQ-TVAE hereafter, on class- and text-conditional image generation tasks.
4.1 Conditional Image Generation
Experiment settings We train our models on 1.2M images in the train split of ImageNet [2 for class-conditional image generation. For text-conditional tasks, we employ 15M image-text pairs in Conceptual Caption (CC) [30] and Conceptual- 12M [31]. In the training phase of both cases, we resize the short axis of an image to 256 and randomly crop it to acquire 256 256 image. In testing, we also resize the short axis to 256 and crop the image at the center.
The scaling factor in HQ- VAE is set to 2, i.e., r = 2, to produce the top and bottom codes of 8 x 8 and 16 x 16 resolution by default, respectively. We test three versions of -o-Transformer by varying hyperparameters: (a) NMr = 12, and NPHr = 4 for the smallest model, (b) NMT = 24 and Neur = 4 for the mid-size model, and (c) NMT = 42, and NEнт = for our largest model. In all cases, NIET = 1 and other parameters related to the network size are fixed. We describe more details about the network architecture in the supplementary document. As evaluation metrics, we adopt Fréchet Inception Distance (FID) 32 and precision & recall (PR) 33 for class-conditional generation, and
8

---

(a) Generated samples
(b) Top code-conditional image generation
Figure 4: Class-conditional image generation results. Images constructed by both top and bottom codes for ImageNet class lorikeet (90) are illustrated in (a). Given image generated from the top code only, the left-most image in (b) (with red bounding box), which corresponds to the upper-right image (a) (with green bounding box), HQ-Transte Trannsfffrrm samples several different bottom codes visualized at the first row in (b) and generates realistic images with diverse details as shown in the second row in (b).
Table 3: Comparison image reconstruction quality on the ImageNet validation set. RQ-VAE" is a modified version of its original model obtained by adopting the same architecture used in HQ-E
Model
Code map resolution
rFID
Codebook size
Effective code usage
DALL-E E SS]
32 x 32
32.01
8192
AAgVAE-SL SL
32 x 32
3,28
8192
VIT-YQGAN
32 x
1.28
8192
VQ-GAN
16 16
4,90
973 16353 8192 + 8192
16384
RQ-VAE VAE 17
8 x
3.20
16384
RO- 17
8 x 5
3.37
16384
16322
HQ-VAE (16 x 16)
8 x
16 16
2.61
8119 + 8192 8192 + 8192
HQ-VAE (32 32)
16 x
+
32
0.57
8192 + 8192
FID and CLIP-score 34 for the text-conditional generation task. We measure the throughput of sample generation on single Tesla A100 GPU.
Class-conditional image generation Table 1 presents that our largest model, HQ-TVAE (L), achieves the best FID (7.15) compared to other two-stage AR models with similar sequence lengths such as VQ-GAN (16 x 16) and TQ-Transformer (8 x 8 x 4). We observe that HQ-TVAE reports the highest throughput in sample generation than other models because it generates x2 bottom codes in parallel given an associated top code. Compared to ADM [26), our smallest model (S) generates images of similar quality while achieving 650 times throughput. We also conduct rejection sampling in all the models based on ResNet- 101 35 with an acceptance rate of 0.25 and report the results within the parentheses in Table [ Figure demonstrates the qualitative results of our approach, which also include the generated images based only on top codes to visualize the representation disentanglement. Interestingly. contrary to typical generative models, HQ-TVAE restricts the diversity of generated images within details by sampling only bottom codes from TrO-Trasffrrmer with fixed
top codes.
presents FIDs and CLIP-scores of several models measured on the CC3M validation split, and the throughput in sample generation. The results show that our model is comparable to RQ- Re-aransffrrrr in terms of FID (12.86 vs. 12.33) and CLIP--coree (0.26 vs. 0.26) while achieving 18%p higher throughput than RO-aransformer.
Text-conditional image generation Table 2
4.2 Analysis of VAE and HQ-Transformer Models
Comparison with other stage 1 methods Table 3 compares HQ- VAE implementations with two different feature map sizes (shown in the parentheses) from the encoder with the existing stage models, where we measure FIDs between the original and reconstructed images, referred to as rFID. table reports the utilization of codebook(s) for the representations of the images
The last column in the ImageNet validation dataset, and our algorithm turns out to make full use of the codebooks. rFID from HQ- VAE x 16 x 16) better than the baselines with the similar sequence length, x 4) and RQ- VAE (8 x 8 x 5). In addition, HQ- VAE (16 16 + 32 x 32)
e.g., RQ-VAE (8

---

Table 4: Ablation study on HQ- VAE.
Table 5: Ablation study on HQ- Transformer
Downsampling op.
Epochs
rFID
Decoding policy
Label type
FID J
Conv2 + Perceptual loss
15
3.72
Global conditioning
Oue-hot
23.25
Avg. Pool. -
13
3.67
Locally séquential conditionig
113
One-hot
PixelUnshuffle PixaUnshuffle
15
3.14
Locally hierarchical conditioning
10.01
"50
2.61
Localfy hiträrchical conditionngg
Sf 15 9.36
Table 6: Performance of HQ-TVAE with more than two levels of codes. The performance is evaluated on the ImageNett validation split. HQ HAES and HQ-Transformers are trained for 15 and 100 epochs respectively for a fair comparison.
Code map resolution
dFI
Code utilization
FID
Precision t
Recall
x 4 x
18.65
7524 8192
37,59
0.49
0.38
x 1444888 8 x
x616
2.48
1954 + 7763 + 8192
15.66
0.61
0.54
888+ T6
3.14
8008 + T992
13.39
0.67
0.52
8x81 F 16 x 16
1 32
0.65
2656 + 8192 8192
11.10
0.64
0.58
outperforms ViT-VQGAN (32 x 32) although we admit that ViT-VQGAN can be implemented more efficiently based on the hierarchical code maps.
Ablation study on HQ-VAE T supports the rationale of our choice, showing that resizing with pixel-unshuffle and -shuffle outperforms the average pooling followed by the nearest interpolation by 0.53 in FID since it prevents information loss during the downsampling operation. We also observe that longer training boosts rFID performance from 3.14 to 2.61. Note that learnable resizing turns out to be not effective, where we adopt simple network with single convolution and transposed convolution layers with kernel size and stride 2, for downsampling and upsampling, respectively. We impose the same loss function on the reconstructed images using only the top codes to avoid top code collapse. Refer to the supplementary document for a more detailed analysis.
Ablation study on HQ-Transformer Table 5 compares the effect of decoding policies and high- lights the benefit of the proposed locally hicrarchical decoding. VQ- VAE-2 [8] first generates the entire top codes and then derives the bottom codes by one with global conditioning sequence of the top codes. RQ-Transformer 17 sequentially generates top code and the relevant bottom codes, locally conditioned on the generated codes from the previous steps. We observe that our decoding strategy achieves better FID than the sequential methods since error accumulation during sampling is alleviated by our parallel decoding of bottom codes. We also validate the effects of soft-labeling as an alternative to one-hot labeling in the cross-entropy loss function for training HQ- ir-Transformer as proposed in TQ-arasfformer [17.. The use of soft labels with temperature T in our case improves performance as similarly observed in [17]
the
Multi-level extensions Table 6 present that three-level architectures outperform the two-level counterparts. For these experiments, we train two different combinations of three-level codes {(4 x 4). (8 x 8), (16 x 16)} and {(8 8), (16 x 16), (32 x 32)). The results show that the use of three-level codes is helpful for improving performance at the early stage of training (15 epochs for training HQ- VAE). However, they fail to achieve additional gains with more iterations probably due to the lower usage of top codes as shown in the "code utilization" column of the table. Detailed architectures for multi-level extensions are described in the supplementary document.
5 Conclusion
We presented two-stage auto-regressive model, which consists of HQ-VAL and HQ- Transsfforrrr, to generate images based on hierarchically quantized codes. HQ- VAE represents an image by two-level discrete codes with different spatial resolutions, where it encodes the high-level structure of image using the top codes and the low-level information using the bottom codes. HQ-Transformer models the multi-level codes efficiently and the combination of the two modules successfully generates unseen images. Experiments confirm the capability and potential of our model for high-quality and high-resolution image generation with a fraction of computational cost compared to existing methods.
Acknowledgement This work was partly supported by IITP grant (N0.2022-0-00959 (Part 2) Few-Shot Learning of Causal Inference in Vision and Language for Decision Making, 2021-4 01343, Artificial Intelligence Graduate School Program (Seoul National University)) and NRF grant [No. 2022R122111 A5A708390811 11 Trustworthy Artificial Intelligence] funded by the Korean government (MSIT).
10

---

References
Goodfellow. LJ., Pouget-Abadie, Mirza, M., Xu, B.. Warde-Farley, D., Ozair, S., Courville, A., Bengio, Generative adversarial networks. NeurIPS, (2014)
[2] van den Oord, A., Kalchbrenner, N., Vinyals, O., Espeholt, L.. Graves, A., Kavukcuoglu, K.: Conditional image generation with pixelenn decoders. NeurIPS. (2016)
[3] Ho, J., Jain, A., Abbeel, P:: Denoising diffusion probabilistie models, NeurIPS, (2020)
[4] Mirza, M., Osindero, Conditional generative adversarial nets. arXiv, (2014)
Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., Sutskever, I.: Zero-shot text-to-imaae generation. ICML. (2021)
[6] Ramesh, A., Dhariwal, Nichol, A., Chu, C., Chen, M.: Hierarchical lext-conditional image generation with clip latents. In arXiv, (2022)
[7] Esser, P., Rombach, R., Ommer, B.: Taming transformers for high-resolution image synthesis. CVPR. (2021)
[8] Razavi, A., van den Oord, A., Vinyals, O.: Generating diverse high-fidelity images with vq-vae-2. In NeurIPS. (2019)
[9] Radford, A., Wu, J., Child, R., Luan, D., Amodci, D., Sutskever, I.: Language models are unsupervised multitask learners. (2018)
[10] van den Oord, A., Vinyals, O., Kavukcuoglu, K.; Neural discrete representation learning, In NeurIPS. (2017)
[11] Chen, X., Mishra, N., Rohaninejad, M., Abbeel, P:: Pixelsnail: An improved autoregressive generative model. In ICML. (2017)
[12] Radford, A., Narasimhan, K., Salimans, T., Sutskever, L: Improving language understanding by generative pre-training. (2018)
[13] Yu, Li, X., Koh, J.Y., Zhang, H., Pang. Qin, Ku, A., Xu, Y., Baldridge, Wu, Vector-quantized image modeling with improved vggan. In ICLR, (2022)
[14] Kim. T. Song, Lee. S., Kim, S., Seo, Lee, S., Kim, S.H., Lee, H., Bae, L-verse: Bidirectional generation between image and text. In CVPR. (2022)
[15] Fauw, J.D., Dicleman, S., Simonyan, K.: Hicrarchical autoregressive image models with auxiliary decoders. arXiv preprint arXiv:1903. 1903.0493 (2019)
[16] Peng. J., Liu, D., Xu, S., Li, H.: Generating diverse structure for image inpainting with hierarchical vq-vae. In CVPR. (2021)
[17] Lee, D., Kim, C., Kim, S., Cho, M., Han, W.S.: Autoregressive image generation using residual guantization. In CVPR. (2022)
[18] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent difffusion models. In CVPR. (2022)
[19] Gu, S., Chen, D., Bao, J., Wen, F., Zhang, B., Chen, D., Yuan, L., Guo, B.: Vector quantized diffusion model iixaaee-mmage synthesis. In CVPR. (2022)
[20] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In MICCAL (2015)
[21] MacQueen. Some methods classification analysis of multivariate observations. In Berkeley Symposium on Mathematical Statistics and Probability, 1967: 281-297. (1967)
[22] Shi, W., Caballero, J., Huszár, F., Totz, J., Aitken, A.P., Bishop, R., Rueckert, D., Wang, Z.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In CVPR. (2016)
[23] loffe, S.. Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In ICML. (2015)

---

[24] Wu, Y., He, K.: Group normalization. In ECCV. (2018)
[25] Chen, M., Radford, A., Child, R., Wu, J., Jun, H., Luan, D., Sutskever, I: Generative pretraining from pixels. In ICML. (2020)
[26] Dhariwal, P., Nichol, A: Diffusion models beat gans image synthesis. In NeurIPS, (2021)
[27] Esser, P., Rombach, R., Blattmann, A., Ommer, B.; Imagehart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. In NeurIPS. (2021)
[28] Chang, H., Zhang. H., Jiang, L., Liu, C., Freeman, W.T.: Maskgit: Masked generative image transformer. In CVPR. (2022)
[29] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.; Imagenet: A large-scale hicrarchical image database. In CVPR. (2009)
[30] Sharma, P., Ding. N., Goodman, S., Soricut, R.:
Conceptual captions: A cleaned, hypernymed, image alt-text text dataset for automatic image captioning. In ACL. (2018)
[31] Changpinyo, S.. Sharma, P., Ding, N., Soricut, R.: Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR. (2021)
[32] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurlPS, (2017)
[33] Sajjadi, M.S.M., Bachem, O., Lucic, M., Bousquet, O., Gelly, S.: Assessing generative models via precision and recall. In NeurIPS, (2018)
[34] Radford, A., Kim, J.W. Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., Sutskever, L: Learning transferable visual models from natural language supervision. In ICML. (2021)
[35] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In CVPR. (2016)
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information how to answer these questions. For each question, change the default [TODO] to [Yes], [No], ; or [N/A]. You are strongly encouraged include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit, In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] We diseuss the limitation of our models on the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] We discuss it in the supplementary material
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
12

---

(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL)? [Yes] We report in both the paper and the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We report in both the paper and the supplementary material.
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [No] The evaluation protocol for conditional generative models does not report the error bars. Instead, we fix the random seed and make experiments reproducible.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We report in both the paper and the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite and report in the supplementary material.
(b) Did you mention the license of the assets? [Yes] We report in the supplementary
material.
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "jhnE0kFnqt6", "reviewer": "Reviewer_4ntj", "summary": "The paper proposes a hierarchical autoregressive transformer model for image generation. Based on a quantizing autoencoder (VQGAN/VQ-VAE) the paper splits the quantized representation into two groups called top and bottom codes. The top codes have a smaller spatial resolution **r** (e.g., 8x8) and encode high-level, abstract information. The bottom codes have a larger spatial resolution **2r** (e.g., 16x16) and encode more detailed texture/high-frequency information. Each bottom code is considered to be only dependent on one top code, i.e., each top code is \"responsible\" for 4 bottom codes. During generation, the top codes are sampled autoregressively, however, the bottom codes can be sampled in parallel (conditioned on the respective top code) which significantly boosts the sampling speed. Results on class- and text-conditional image generation show similar or improved results compared to recent SOTA baselines while having a significantly higher sampling throughput than other autoregressive models.", "questions": "I would also be interested in the model's performance on unconditional generation or other conditioning, e.g., label map to image or similar.\n\nAlso, do you think the performance or resolution could be further improved by adding even more hierarchical layers of quantized codes?", "limitations": "One of the main limitations still remains as the top codes still have to be sampled autoregressively. Other approaches, e.g. MaskGIT, sample all codes in parallel and then iteratively resample code entries with low probability which overall also reduces the numbe of model evaluation steps. I wonder if approaches like that could be combined with the current approach?\n\nAlso, according to table 3 the size of the latent representation does affect the reconstruction capability so it is not clear how well this scales to larger resolutions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper tackles an important problem of autoregressive image generation models, i.e., their sampling speed. The proposed approach makes the assumption that the high-level image content can be modeled via a relatively small spatial resolution while detailed texture information can be modeled locally without relying on information from far away (in pixel space). This hierarchical approach allows faster sampling speeds due to parallel generation of the texture codes while the high-level image codes (which still have to be sampled autoregressively) can be represented in a smaller spatial dimension.\n\nOverall the paper is well written and motivated. I believe the methodology section could be slightly improved to make it easier to understand. Maybe Fig 2 can be slightly extended to show more details of the transformer. I believe the encoder/decoder structure is quite clear and easier to understand from the textual description.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N1hGkg6YF5c", "reviewer": "Reviewer_skYU", "summary": "In this paper, a hierarchical method for synthesizing natural images based on discrete representations is presented. More precisely, a combination of the previous two-stage hierarchical VQVAE-2 [1] and residual vector quantization (RQ-Transformer [2]) is first used to learn a hierarchical two-stage discrete latent space (called HQ-VAE). In a second phase, as usual, a generative prior is learned over the latent space. For this purpose, the \"HQ-Transformer\" is introduced, which, given a top code, predicts all corresponding local bottom codes in parallel. Experiments show good results in class-conditional modeling of ImageNet and in text-image synthesis of conceptual captions with improved sampling speed over related two-stage discrete methods.", "questions": "Please see the above section for questions. ", "limitations": "Yes, limitations and societal implications are discussed, albeit very briefly.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The proposed approach is simple (which is good!).\n- In my opinion, the parallel local decoding is interesting and quite promising for further upscaling of the model.\nSide question: for the parallel decoding scheme, is it possible to make use of a resampling strategy that uses the most confident codes only akin to [3]?\n- The study on avg.-pooling vs channel-resampling is nice and the paragraph on using learnable down/upsampling layers here should be moved to the main paper.\n\n**Weaknesses**\n\n- The main drawback of the paper is it somewhat incremental nature over RQ-Transformer[2] and VQVAE-2 [1]. The main novelty seems to be the parallel decoding scheme,  while the increase in sample quality is limited and still contains lots of perceptual artifacts. I am curious about the other reviewers' opinion on this.\n- While the comparisons to other \"off-the-shelf\" models in Tab.1-3 is interesting, I think the more interesting comparison would be to compare to a VQVAE-2 style hierarchical code (which can be trained with VQGAN reconstruction losses) and the strictly hierarchical autoregressive bottom -> top sampling scheme proposed there **in the same setting** (16x16 + 8x8, 8192+8182). In a related manner, what are the autoregressive decoding parameters used for the comparison in Table 5? \n- For text-to-image: My understanding is that the model is directly conditioned on text embeddings obtained from BPE. Does the model further improve by deploying a fixed or learnable (potentially pretrained) text encoder here?\n\n_Minor:_\n- some of the samples in Fig.7 show lots of artifacts which are often associated with an overly reliance on perceptual losses such as LPIPS. Does weighing this term down in the reconstruction loss improve upon this? Related: Does the proposed HQ-VAE structure make training the discriminator more difficult?\n- Is it possible to add more levels to the model (e.g. training a top-middle-bottom model)? Can this be used to train models with even higher receptive field in the bottom codes (e.g., for further improving sampling speed)?\n- l. 196: And.we --> And we \n- l. 199: casual self-attention --> causal self-attention\n\n_Summary:_\n\nThis paper presents a simple and straightforward extension of RQ-Transformer and VQVAE-2 to learn a hierarchical two-stage discrete generative model of images. I like the general setup, the parallel (local) decoding strategy is good and a reasonable assumption, and although it is a straightforward combination of existing approaches, I think this approach has some potential to produce really good samples at a decent sampling speed. Further questions: How well does the model scale to even higher resolutions? Training the model on 32x32 and 16x16 latent codes would be interesting. In my opinion, the main drawback of the paper are both its limited increase in quality of the samples and somewhat limited novelty. This is why I currently view the paper as a borderline submission, but I am happy to be convinced otherwise by the authors and/or other reviewers.\n\n\n_Literature_:\n- [1]: Generating Diverse High-Fidelity Images with VQ-VAE-2, Razavi et al\n- [2]: Autoregressive Image Generation using Residual Quantization, Lee et al\n- [3]: MaskGIT: Masked Generative Image Transformer, Chang et al", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2T1-D8QTgpu", "reviewer": "Reviewer_CKx9", "summary": "This paper focused on the problem of improving computational efficiencies of auto-regressive (AR) models on high-resolution images. The authors proposed a hierarchical-quantized VAE which models the images at different scales and a Transformer model for the encoded sequence at each scale. Evaluations on class-conditional and text-conditional tasks between the proposed method and the baselines were shown.", "questions": "It would be helpful if the authors could address the issues and concerns listed above.", "limitations": "The authors provide some discussions on the limitation regarding the independence assumption. More discussions on how the proposed method would behave when the target resolution increases would be helpful. \n\nThe authors also provided discussion on potential negative social impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is generally nicely written and well-organized. In addition, the paper contains a good coverage of the related background and training details (architecture selection, hyper-parameter settings).\n2. The paper included thorough ablation studies to validate the design decisions.\n\n## Weaknesses\n\n1. The proposed methodology seems incremental: representing images using VQ-VAE in a coarse-to-fine manner was previously explored (VQ-VAE2[28], HAM[10], Hierarchical VQ-VAE[22], RQ-VAE[18]). In particular, RQ-VAE[18] also disentangles the information among different code levels (depths) without specialized modules. In addition, [18] proposed RQ-Transformer which also models codes at different levels in an AR fashion. In terms of computation complexity, RQ-Transformer[18] has the same dominating factor as the proposed method as well.\n2. There are some concerns regarding the generalizability of the proposed method:\n    1. The empirical validation is insufficient for disentanglement of the latent code as only 2 scales were explored. It is also not clear how the proposed approach could be extended to more than 2 scales. Since the claimed contribution involves the disentanglement of information into different codes, having a general principle that could be extended to an arbitrary number of scales is important and more empirical evidence should be present.\n    2. It is not clear whether the conditional independence assumption would still hold when $r>2$ which would be important as the target image resolution increases (e.g. generating images with 512x512 resolution and above).\n3. The formulation in equation 6 rewrites the joint probability into the product of two terms. During inference, the proposed method maximizes the second term first (by drawing the best $t_i$) and then maximizes the first term based on the chosen $t_i$. This is a sub-optimal strategy as maximizing the second term and then maximizing the first term does not equal maximizing the joint probability. Since the paper presents this as an inference strategy, the corresponding drawbacks/tradeoffs of this approach should be analyzed and discussed.\n4. There are some important details missing:\n    1. How does the HQ-Transformer generate diverse samples when the top code is fixed? Does it sample $\\bf{b}_0$ only or does it apply stochastic sampling for all $\\bf{b}_n$ like RQ-Transformer[18]?\n    2. There are some missing throughput results in table 1.\n5. Some minor issues\n    1. In equation 11: there is a duplicate $u_1$, it should be $u_0$ instead?\n    2. What were the batch sizes used to time the throughput in Tables 1 & 2 for different models?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept