MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation
Authors: Chuanxia Zheng
Long Tung Vuong
Jianfei Cai
Dinh Phung
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vector Quantization
Generative Models
Transformer Generation

Abstract: Although two-stage Vector Quantized (VQ) generative models allow for synthesizing high-fidelity and high-resolution images, their quantization operator encodes similar patches within an image into the same index, resulting in a repeated artifact for similar adjacent regions using existing decoder architectures. To address this issue, we propose to incorporate the spatially conditional normalization to modulate the quantized vectors so as to insert spatially variant information to the embedded index maps, encouraging the decoder to generate more photorealistic images. Moreover, we use multichannel quantization to increase the recombination capability of the discrete codes without increasing the cost of model and codebook. Additionally, to generate discrete tokens at the second stage, we adopt a Masked Generative Image Transformer (MaskGIT) to learn an underlying prior distribution in the compressed latent space, which is much faster than the conventional autoregressive model. Experiments on two benchmark datasets demonstrate that our proposed modulated VQGAN is able to greatly improve the reconstructed image quality as well as provide high-fidelity image generation.

MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation
Chuanxia Zheng
Long Tung Vuong VinAI 1ongvt.44@gail... com
Monash University @buanxiazheng@gmil.coo com
Jianfei Cai
Dinh Phung
Monash University Jianfei. Cai @monash.edee
Monash University
dinh. Saang.phun@@@e monash.edu
Abstract
Although two-stage Vector Quantized (VQ) generative models allow for synthesiz- ing high-fidelity and high-resolution images, their quantization operator encodes similar patches within an image into the same index, resulting repeated arti- fact for similar adjacent regions using existing decoder architectures. To address this issue, we propose to incorporate the spatially conditional normalization to modulate the quantized vectors to insert spatially variant information to the embedded index maps, encouraging the decoder to generate more photorealistic images. Morcover, capability of the discrete codes without increasing the cost of model and codebook. Additionally, to generate discrete tokens the second stage, adopt Masked Generative Image Transformer (MaskGIT) to learn an underlying prior distribution in the compressed latent space, which is much faster than the conventional au- toregressive model. Experiments on two benchmark datasets demonstrate that our proposed modulated VQGAN is able greatly improve the reconstructed image quality as well as provide high-fidelity image generation.
multichannel quantization to increase the recombination
Introduction
The vision community has rapidly improved image synthesis results on quality, diversity and res- olution over a short period of time. particular, many powerful baseline frameworks have been introduced, such as Generative Adversarial Networks (GAN) [13,, Variational AutoEncoder (VAE) 233. Flow-based Generators
and Diffusion Probabilistic Models (DPM) 35 These methods are conceptually intuitive, leading to an explosion of image synthesis works that push the boundaries of image generation quality and diversity [14
20
21 37 4
Figure 1: 256 256 image samples generated by the proposed MoVQ, with model trained on FFHQ.
Among these existing image synthesis frameworks, Vector Quantized-Variaaionaal
Variational AutoEncoder (VQ- VAE) 39 a very popular nova, which learns a compressed discrete representation for each image
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

at the first stage and subsequently learns the underlying prior distribution in the discrete latent space at the second stage. Unlike conventional GANs that pursue fragile balance in a minimax game. resulting in unstable training, the optimization target of VQ- VAE is definite negative log- log-likelihood (NIL)) objective of the training data. By maximizing the log-likelihood over all training examples, theoretically VQ-VAE is able to cover all modes of the data, and thus bypass the "mode collapse" issue in GAN. On the other hand, compared with VAE [23]. VQ-VAE maps an image into a palette of latent discrete codes in higher resolution with spatial structure information and learns the composition of the codes from the data itself, which overcomes the long-dragged image quality issue of VAE in image synthesis. These advantages of VQ-VAE have led to remarkable image synthesis results as evident by its recent extensions, such as VQ- VAE-2 [I]. DALL-E [30], VQGAN [TT] ImageBART [TO,, LDMs 32 VIT-VQGAN 45), IQ-VAE 24], MaskGIT
2 and DALL-E-2 29
Despite the great performance, the VQ-VAE or VQGAN pipeline also has its shortcomings. For example, its second stage is typically modelled as a sequence generation process in an autoregressive way (39 i.e. generating each discrete latent code one by one at different spatial locations, which is very time-consuming for inference. MaskGIT [2] nicely addresses this issue by predicting multiple tokens based on the prediction confidence at each step, which greatly reduces the number of steps needed in the autoregressive token generator.
In contrast, in this paper we focus on improving the stage-1 learning of the VQ-based image synthesis pipeline. Specifically, we notice another inherent shortcoming of the VQ-based methods, i.. they often generate repeated artifact patterns in image synthesis (see Fig. 3 which is because the quantization operator drops away some variances and embeds similar patches into the same quantization index. To address this, motivated by the spatially conditional normalization and position embedding literature 117 (2 20 41, we re-design the decoder architecture in a way that utilizes a spatially conditional normalization layer to modulate the activations using the quantized vectors, which essentially adds spatially variant information to the discrete representation. Moreover, we leverage a multichannel representation technique, introduced in [47 for image completion, to keep the codebook size manageable while with sufficient representation capability for image generation. Lastly, with a better quantizer, we follow MaskGIT [2] to speed up the prior distribution training over the compact discrete representations in the second stage [45].
In summary, our main contributions are as follows:
We point out an inherent shortcoming of the VQ-based image synthesis pipeline: having repeated artifacts in semantically similar nearby regions, and solve it by introducing a spatially conditional normalization to provide spatially variant information for different locations.
With the help of our proposed spatially conditional normalization, we further incorporate the multichannel representation that dramatically improves the reconstructed image quality using the same encoder-decoder layers as in VQGAN.
When sampled from learned prior on all directions for multi-locations and multi-channels, experimental results on two benchmark datasets show that our synthesized samples hold not only high quality but also large diversity.
2
Related Work
VQ-based image synthesis. Our model is derived from VQGAN [I. which is built upon the two- stage VQ-VAE [39] first, a quantizer with encoder-decoder architecture is trained embed images into compact sequences using discrete tokens from a learned codebook, and then a prior network is learned model the underlying distribution in the discrete space. Unlike the minimax game in popular GAN models [13 [ 20 (1). the VQ-based generator is trained by optimizing negative log-likelihood over all examples in the training set, leading to stable training and bypassing the "mode collapse" issue. Driven by these advantages, many image synthesis models follow the two-stage paradigm, such as image generation [3] 45 2 24 [66. image- to image translation [I] IC 32], text-to-image synthesis 30 29 10 77, conditional video generation 28 42 44, and image completion [II [0 47]. Apart from VQGAN, the most related works also include ViT-VQGAN (45) and VAE (24 that aim to train better quantizer in the first stage. Compared to them, our model is simple and efficient, yet effective to improve the image quality, without adding the computational cost on higher resolution representations with much larger model (45) or more stages of recursive quantization [4]

---

Spatially conditional normalization, also called adaptive instance normalization, has been widely used in image synthesis task 17
Among them, the spatially conditional input target domain images [8]. semantic maps Comparing our approach to these works, our spatially conditional input is the embedded discrete features, which contain automatically learned
18
27
20
contains various variants, such as style images
9 17
[27], learned vectors [20] and random heatmaps
41
compact contents.
To the best of our knowledge, this is the first work to modulate quantized vectors and use multichannel quantization on the VQ-based image generation framework. In the following sections, we will describe the modulated quantized vectors and multichannel quantization and discuss their advantages and [24] in detail.
over the concurrent models such as [32 45
Stage Modulated VQ Encoder Decoder Training
Multi-Chaneels Masked Token Predication
VQ
Z4
Conv
g
Taunfforee
Bidresstoonl
Ouaicc
Decader
Vorm
Original
Reconstruction
MoVQ
Masked Tokeus
Prodicated Tokens
Figure 2: Left: The quantizer architecture of our proposed MoVQ. We incorporate the spatially conditional normalization layer into the decoder, where the two convolution layers predict modulation parameters 7 and B in point- point-wise wise way to modulate the learned discrete structure information. Right: Masked image generation. Here, a bidirectional transformer applied estimate the underlying prior distribution on the discrete representation with multiple channels.
3 Method
The proposed approach, illustrated in Fig. 2 follows a two-stage recipe that embeds images using learnable codebook (see Fig. left), and then tames bidirectional transformer estimate the underlying prior distribution over the discrete latent space (see Fig. right.. The key observation we found that better quantizer will naturally lead to better image reconstruction and generation quality [45] Hence, our main goal in this work is to improve the quantization in the first stage, yet without suffering from awkward computational cost as in concurrent works 45 24
3.1 Modulating Quantized Vector
Background. Given an image æ e Hxw'3, vanilla VQ-VAE: learn a discrete codebook to repre- sent observations collection of codebook entries 2g E fx where n, is the dimensionality of quantized vectors in the codebook. In this way, each image can be equivalently represented as compact sequence with w indices of the codevectors zq. Formally, the observed image is reconstructed by:
a = Ga(z,) (o() Go (T))
(1)
In particular, an encoder (-) first embeds an image x into a continuous vector 2, and the quantization operator q()) is then conducted to transfer the continuous feature into the discrete space by looking up the closest codebook entry zμ for each spatial grid feature æig within a:
2q
(2) arg minll&3 zull.
(2)
2L62
where Z E R. Xne is the codebook that consists of K entries with n, dimensions. The quantized
vector 2n is finally transmitted to a decoder Go(-) for rebuilding the original image. The overall models and the codebook can be learned by optimizing the following objective:
L(E Go, Z) | a|l + [[s)(({(()) - =n 1 3 [[gg[=,)] Exp
Here, sg denotes the stop-gradient operator, and /3 is a hyperparameter for the third commitment loss. The first term is a reconsiruction loss to estimate the error between the observed and the reconstructed é. The second term is the codebook loss to optimize the entries in the codebook. We use the released VQGAN implementation our baseline.

---

(a) VOGAN M 16 16 / 16384
(b) ROVAE 24
(c) Ours
(d) Original
8 x 16/ 16384
16 x 16 x 4/ 1024
Figure 3: Reconstructions from different models. The numbers denote the represented latent size and learned codebook sizes, respectively. Compared to the latest state-of-the-art RQVAE [24). our model dramatically improves the image quality in the first stage under the same compression ratio.
Spatially conditional normalization. The quantization operator is lossy BT and similar patches are often embedded as the same codebook indices, resulting in a repeated artifact when they are synthesized through existing decoder architectures (see Fig. 3 (a)). As opposed to existing methods that directly feed discrete feature maps into the decoder as activations, in this work, we introduce a new spatially conditional normalization layer to propagate the embedded contents to the activations. The key motivation behind this is to add spatial variants to the discrete maps, such that the model can generate plausible and diverse results, even for the same quantization index in neighboring regions.
The structure of our modulated decoder illustrated in Fig. 2 (left), with an activation F normalized by conventional normalization, and then modulated by the learned scale and bias calculated from the embedded vector. Specifically, the activation F of the i-th layer in the decoder Go is given by:
(-((FFF)) o(Fi-1) + dg(2)),
F' = (24)
())
where FX-1 is the intermediate feature map, which can be initialized as the positional embedding
112 40). learned constant [20]. or Fourier features [36 [9] for 8=9 in the decoder. μ(-) and (-)
respectively denote the functions for calculating the mean and standard deviation of the activation.
There are many choices of the normalization. Following the baseline VQGAN [T], here we directly the Group Normalization
43) (-) and (-)() are two learned affine transformations, which are implemented as 1 x convolutional filters in our setting, to convert the discrete representation 2g to the scaling and bias values. Note that, the output of Φ (-) and (ba()) hold the same resolution the current activation Fi-1, which injects spatial variances into the discrete feature.
In fact, this spatially conditional normalization is derived from the adaptive instance normalization (AdaIN), which has been applied for many image synthesis approaches, such as style transfer [7], image-to-image translation 18 277, and image generation (20 41 Compared to these methods, our spatially conditional map is quantized map, which contains learned compact contents. As such modulation operator injects the spatially variant information, encourages the same quantization entries to generate plausible and diverse results different locations.
Multichannel representation. Unlike existing quantizers that map an image into a single-channel index map, here convert it into multichannel index map with the shared codebook to further improve the image quality, similar to [47. This is inspired by the conventional GAN setting, where 1 x ne random vector is able to generate photorealistic images with reasonable structure 13 20 21,, showing that values across the channels contain abundant information. In practice, first subdivide the encoded continuous feature E Rhxwxn, along the channel dimension into multiple chunks, i.e. 8 = {g(1) .... 2(c)), J. g(e) E /efxwxn./c Each of these chunks is then quantized based on the equation to the closest codevectors in the codebook.
For 256 x 256 images, we downsample it by fixed factor of 16 to 16 16 features. Unless otherwise noted, in this work we employ e = parallel chunks, and our equivalent sequence representation is 16 x 16 x 4, which is 192x times smaller than the original image. Note that, in such way the codevector dimensionality in the codebook will be reduced to ny nz/c. Due to this reduced dimension of each codevector, the total computational cost is similar to that of the single-channel

---

representation with fully dimensionality. In fact, our codebook 2 E RKX", has a smaller size with the number of dimensions to be n R2//c 256/4 64.
While a higher resolution representation, e.g. 32 x 32 x 1 or 64 x 64 x 1, can also improve the reconstruction quality in 30 stage due to the longer sequence e h x w (see Sec. 3.2 for details). In contrast, our multichannel representation maintains smaller sequence length, i.e. 16 16, although each sequence token now is recomposed by four pieces, each of which corresponds to one quantization index. With such an exponential combination capacity of K for each spatial grid feature 2gj. the multichannel representation power becomes much larger than the K entries in the original codebook.
45 32
the computational cost will be expensive for the second
3.2 Modeling Prior Distribution
While decoder can invert the discrete image embeddings 2g to produce images x, we need to tame a model to estimate the underlying prior distribution over the discrete space to enable image generation. Since our main goal to improve the codebook learning stage, we directly employ the existing setup for the second stage, including the conventional autoregressive model and the latest masked generative image transformer (MaskGIT) 2
Autoregressive token generation. After embedding an image into an index sequence
{8), Sh [hxw] of the codebook entries 2g. the image generation can be naturally formulated as an autoregressive next-symbol predication problem. particular, we maximize the likelihood function:
p(s) ((s)bee))
where p(s:(<<v) is the probability of having the next symbol 8; given all the previous symbols 8ci- Then, the training objective for the second stage is equal to minimize the negative log-likelihood of the whole sequence, i.e. L Ex~p(=) logr(s)))
There are many choices of autoregressive generation models, such as CNN-based PixeICNN and cunsformer-basee GPT [3]. While they sequentially produce diverse and reasonable results based on the previously generated results, the sampling process is very slow.
Masked token generation. To produce the index sequence efficiently, we follow the latest MaskGIT to simultancously predict all tokens in parallel. Inspired by BERT (5), the model aims to estimate the distribution of all indices based on the visible indices in fixed length:
p(s) Il p(ss(sm),
(6)
where 8m denotes conditional representation with partially visible codebook indices. In practice,
the masked map 8m, with random mask ratios during training, is fed into multi-layer bidirectional transformer to estimate the possible distribution of each masked indices, where the negative log- likelilood objective as above also calculated between the ground-truth one-hot index and predicted index. At inference time, the model runs fixed steps and predicts all tokens simultaneously in parallel based on the previous produced visible tokens at each iteration. Then, the most confident tokens are selected out and the remaining masked tokens will be predicted in the next step condition previous predicted tokens. We refer readers to MaskGIT [2 for more details.
Multichannel token generation. Since our MoVQ embeds an image into a multichannel sequence {s1, Shxwh,, ). where s; E {0..... |Z| - 1} for each position, we inversely concatenate chunks along the channel as one token for the input, and predict e indices for the output. practice, we independently mask each index for various channels and positions. Compared to MaskGIT, the predicated indices our model condition on three directional dependencies, iee visible indices from different positions and channels, resulting in larger diversity due to the combination capability.
5

---

Table Quantitative reconstruction results on the validation splits of ImageNet [33 (50,000 images)
and FFHQ (20] (10,000 images). denotes the model we trained with the publicly available code.
"Num 2" the number of codevectors the codebook.
Model
Dataset
Latent Size
Num Z
PSNR 1
SSIM
LPIPS
rFID
VOGAN 11
16x 16
1024
22.24
0.6641
0.1175
4.42
ViT-VQGAN
45
32x32
8192
3.13
FFHQ
RQ-VAE 24
8x8x4
2048
22.99
0.6700
0.1302
7.04
RQ-VAR
24
16x 16x4
2048
24.53
0.7602
0.0895
3.88
Mo-VQGAN (Ours)
16x16x4
1024
26.72
0.8212
0.0585
2.26
VQGAN
11
16x16
1024
19.47
0.5214
0.1950
6.25
VQGAN
II
16x16
16384
19.93
0.5424
0.1766
3.64
ViT-VQGAN
45
ImageNet
32x32
8192
1.28
RQ-VAE
24
8x8x16
16384
1.83
Mo- VQGAN (Ours)
16x16x4
1024
22.42
0.6731
0.1132
1.12
Figure 4: Top: original 256 x 256 x images, bottom: reconstructed images from MoVQ with
16 16 x latent representation in a discrete space. Zoom in to see the details.
4 Experiments
4.1
Experimental Details
Datasets. To evaluate the proposed method, we instantiated MoVQ on both unconditional and clndsicon image generation tasks, with FFHQ [20] and ImageNet (33] respectively. The training and validation setting followed the default one our baseline model VQGAN [1. We trained all the models on 256 x 256 images.
Evaluation metrics. For image generation, we used the two most common evaluation metrics Fréchet Inception Distance (FID) [15] and Inception Score (IS) 34 to evaluate the quality and diversity between generated images and ground truth images. Since we focused on the first stage, also evaluated the quality between reconstructed images and original images. Except for the rFID score, we additionally reported the traditional patch-level image quality metrics, including peak signal-to-noise ratio (PSNR), structure similarity index (SSIM), and the latest learned feature-level LPIPS 466 for the paired reconstructed images and ground truth images.
4.2 Image Quantization
Network structures and implementation details. For each dataset. only trained single scale quantizer with a codebook E 1024x61 i.e. 1024 codevectors each with 64 dimensions, 256 256 images for all experiments. Our encoder-decoder pipeline is built upon the original VOGANT except the only difference that we replaced the original Group Normalization with the
htttss//gghhbboccom//oo/vv/iiiin transformrsss

---

Table 2: FID comparison for unconditional image generation on FFHQ (0 with 256 resolution.
"# Params" is the model size and VQ-based models hold an encoder model and prior model, respectively. "# steps" denotes the number of steps run for generating sample.
Model
# Params
# Steps
FID
BigGAN
164M
1
12.4
StyleGAN2
21
30M
1
3.8
VQGAN (w/ top-k sampling)
72. 1M 801M
256
11.4
ImageBART
10
9.57
RQVAE
24
100M + 355M
256
10.38
ViT-VQGAN
45
599M + 1697M
1024
5.3
Mo-VOGAN (Ours)-auto
82.7M + 307M
1024
8.52
Mo-VQGAN (Ours)-mask
82.7M 307M
8
8.78
proposed spatially conditional normalization layer. We applied the new normalization layer in the first three blocks of the decoder. Following the default setting in VQGAN, images are always downsampled by fixed factor of 16, i.e. from 256 x 256 16 x 16 x 4. We set hyper-paramett models with a batch size of 48 across 4 Tesla V100 GPUs with 40 epochs for this stage.
a grid of tokens with the size of following the baseline VQGAN work, and we trained all
We compare MoVQ with the state-of-the-ar methods for image reconstruction in Table T All instantiations of model outperform the state-of-the-art methods under the same compression ratio (192x). This includes the concurrent works ViT-VQGAN (45) and RQ- VAE [24). which utilize higher resolution representation and recursively residual representation, respectively. Without bells and whistles, though we use a much smaller number of parameters (82.7M), similar to VQGAN 17), MoVQ outperformers ViT-VQGAN 45). which employs a larger transformer model (599M) on higher resolution representation (32 X 32) for the first stage. The concurrent RQ- RQ-VAE work [24] also represents an image with multiple channels by recursively calculating the residual information between quantized vectors and their continuous ones, which requires much more embedding times. Furthermore, the entries in RQ- RAE are not equally important because the residual representation highlights the codevectors in the first round, leaving the other codevectors to capture the small residual information. contrast, our codevectors share the same significance in all channels, resulting in larger representation capability. More importantly, our model dramatically improves the reconstructed images quality on all metrics, suggesting the reconstructed images are closer to the original inputs, which contributes to more downstream image interpolation tasks.
Note that, our learned codebook contains only 1024 codevectors with 64 dimensionality, but interest- ingly outperforms other methods using larger codebook sizes. This suggests that a better quantizer can improve the codebook usage, and is not necessary greedily increase the codebook size.
The qualitative results are visualized in Figs. 3 and MoVQ achieves impressive results under various conditions. In Fig. 3] compare MoVQ with the baseline model VQGAN [11] and the concurrent model RQ- VAE [24]. VQGAN holds repeated artifacts on the similar semantic patches, such the grass and trees. RQ-VAE improves the visual appearance, but exhibits systematic artifacts with lossy information. Our Mo shows no such artifacts, providing much more realistic details.
4.3 Image Generation
Network structures and implementation details. After embedding images a sequence, transformer is implemented to estimate the underlying prior distribution in the second stage. Here, we used the same configuration for all models: 24 layers, 16 attention heads, 1024 embedding dimensions and 4096 hidden dimensions. The network architecture is build upon the VQGAN baseline, except that we predicted a 16 16 4 x 1024 tensor in the final layer, where 16 x 16 4 is the number of predicted indexes, and 1024 is the number of entries in our learned codebook. Here, we set the image generation two scenarios:
(1) an "auto" scenario, in which the training and inference are totally similar with the VQGAN baseline, and (2) "mask" scenario, in which the training and inference
7

---

Table 3: Quantitative comparison for class-conditional image generation on ImageNet [33
Model
# Params
# Steps
FID 4
IS f
Prec f
Rec t
BigGAN-deep
160M
6.95
198.2
0.87
0.28
DCTransformer 25
738M
> 1024
36.51
0.36
0.67
Improved DDPM 226
280M
250
12.26
0.70
0.62
VQ-VAE-2 31
13.5B
5120
31.11
445
0.36
0.57
ADM [6]
554M
250
12.94
101.0
0.69
0.63
VQGAN
1.4B
256
15.78
78.3
RQ-VAE
24
3.8B
1024
7.55
134.0
MaskGIT
2
227M
6.18
182.1
0.80
0.51
VIT-VQGAN
45
(w/ rejection sampling)
2.2B
1024
4.17
175.1
Mo-YQGAN (Ours)-auto
389M
1024
7.13
138.3
0.75
0.57
Mo-VQGAN (Ours)-mask
389M
12
7.22
130.1
0.72
0.55
Unless otherwise noted, default samples tokens as in MaskGIT for much faster sampling. The syper-parameters follow the defaults setting as in VQGAN, and we trained all models with batch size of 64 across 4 Tesla V100 GPUs with 200 epochs.
is inspired by MaskGIT 22
The unconditional generation results are compared in Table 2 While the original autoregressive sampling in our baseline VQGAN ( provides tiny better generation results, the sampling time is much more expensive. Following MaskGIT, we not set any special sampling strategies such as top-k and top-p sampling heuristics. We produce 60,000 samples at the inference time. Our model outperforms most VQ-based methods, with smaller parameters and faster inference time. While our model performs worse than the concurrent work ViT-VQGAN, they use much larger model (5 times than ours) in the second stage with longer training, which is nor a directly fair comparison. A few generated samples are visualized in Figures (see the Appendix for more). MoVQ generates photorealistic high-fidelity images, with a large diversity.
We also compare the proposed MoVQ with state-of-the-art models for class-conditional image generation on ImageNet 256 256. Our model generates 50 samples for each category, 50,000 samples in total, for the quantitative evaluation. As shown in Table the proposed Mo significantly improve the performance over the baseline model VQGAN IT. Compared with the concurrent work RQ-VAE (24), which also employs multi-channel representation, our performance is competitive with them, even without the specially designed EQ-transformer for further predicting multichannel indexes. As MaskGIT and VIT- use more GPUs for longer training, it is not directly fair comparison with our MoVQ. Figures 5 show a few generated samples. Note that, the proposed MoVQ appears to be able generate high frequency details, such the eyes of the bird.
4.4 Discussion
We run a few ablations to analyze the proposed MoVQ. The results are summarized in Fig. 6
Architecture. Fig. (a) reports MoVQ with various decoder settings. Compared with the baseline model VQGAN (A), although used the same settings for the network architecture, the multichannel representation (B) naturally leads to significant improvement by decomposing and recomposing the
features in channel. For the spatially conditional normalization (C), we explored three most commonly used embedding functions as the initial feature F", including positional embedding [12] 40), learned constant (20), and Fourier features [36] 19.. For positional embedding, we implemented sine and cosine functions of different frequencies on different locations and channels. The learned constant is implemented as learned parameters with the same size as the embedded discrete feature. However, the improvement from both methods is limited. We believe key reason is the lack of special identity information for the reconstruction task, while they can provide high frequency signals. contrast, each Fourier feature is calculated from the corresponding discrete feature, which contains the identity feature, as well as the high-frequency signals, resulting in a significant improvement.
8

---

Goldfish
Goldfinch
Goose
Flamingo
Silky Terrier
Tibetan Mastiff
Red Panda
Grey Fox
Figure 5: Generated 256 256 images by our MoVQ for class-conditional generation on ImageNet.
Codebook size. Vanilla VQGAN shows a larger codebook leads to much better representation. However, this is not true in our experimental results. In Fig. (b) we compare the results with different codebook sizes. As shown in the figure, the difference between the 1024 codebook entries and 16384 entries in our setting is negligible. This suggests that once a better quantizer is designed. is sufficient to represent vast amounts of images using small codebook, which makes the model easier to train. Besides, we found low codebook usage in current VQ models, while our design can somehow improve the codebook usage. We plan to investigate this issue in future work.

---

Methods
iFID
FID
Methods Num Z rFID
Reconstraction
15
Generation FID
Baseline VQGAN
4.42
11.4
VQGAN 1024
6.25
+ multichannel x4
3.78
10.6
VQGAN 16384
3.64
w/ sinusiods
3.52
9.17
w/ learned constants
3.48
8.86
MoVQ
1024
1.12
w/ Fourier features
2.26
8.78
MoVQ
16384
1.05
channels
(a) Initial spatial features 1o for the spatially conditional normalization on FFHQ.
(b) Different numbers of entries in the codebook ImageNet.
(c) Reconstruction and genera- tion FID of representation with different channels on FFHQ.
Figure 6: Ablations MoVQ for different network architectures, codebook sizes and latent maps.
Latent size. Vector quantization is compression system, and we exploit the influence of different compress ratios by using different latent sizes. In order not dramatically increase the computational cost, we mainly compare the performance of the latent representation with different numbers of channels Fig. (c). The reconstruction performance naturally benefits from more channels due to the lower compression ratio. However, we note that not all configurations automatically benefit from more channels on image generation, as we need to predict more information in the second stage. Thus, we select trade-off between the reconstruction and the generation. It is worth to note that this an initial step toward combining spatial vectors the likelihood model with the channel information as in GAN for the generation task.
5 Conclusion
In this work, we introduced MoVQ, a new model that is simple, efficient yet effective for generating diverse and plausible images using a better quantizer with a powerful transformer to estimating the prior. Our encoder and decoder architectures are kept simple as the baseline framework VQGAN, except that incorporate spatially conditional normalization and multichannel latent maps into the VQGAN method. Experimental results show that MoVQ significantly outperforms the state-of-the-art VQ models image modeling, yet without increasing computational cost. With better quantizer, we show that the fidelity of our unconditional samples and class-conditional samples are better than the existing methods.
In the future, we would like to explore the semantic meaning of each entry in the learned codebook. As our model not only generates high fidelity image the sate-of-the-are GAN, but also provides much better reconstructed images, we are excited about the future of VQ-based models and plan to apply them to more image inversion, interpolation and translation tasks.
Limitation. Although our MoVQ dramatically improves the image representation quality than the state-of-the-art under the same compression ratio, the model sometimes generates images with high-frequency appearance, while the structure information is missing. We believe this is partially due the multichannel representation employed in our model. Therefore, better generation model for modeling multichannel indexes needs to be further studied.
Acknowledgements:
This research was supported by Monash FIT research Grant.
10

---

References
[1] A. Brock, J. Donahue, and K. Simonyan. Large scale gan training for high fidelity natural image synthesis. In Proceedings of the International Conference on Learning Representations (ICLR), 2018.
[2] H. Chang, H. Zhang. L. Jiang, Liu, and W. T. Freeman. Maskgit: Masked generative image transformer. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2022.
[3] M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and Sutskever. Generative pretraining from pixels. In International Conference on Machine Learning (ICML), pages 1691-1703, PMLR, 2020.
[4] A. Cherepkov, A. Voynov, and A. Babenko. Navigating the gan parameter space for semantic image editing, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3671-3680, 2021.
[5] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, 2019.
[6] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021.
[7] M. Ding, Z. Yang. W. Hong, W. Zheng, C. Zhou, D. Yin, Lin. X. Zou, Z. Shao, H. Yang. et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34, 2021.
[8] L. Dinh, D. Krueger, and Y. Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: 1410.85/6, 2014.
[9] V. Dumoulin, J. Shlens, and Kudlur. learned representation for artistic style. In Proceedings of the International Conference on Learning Representations (ICLR), 2016.
[10] Esser, R. Rombach, A. Blattmann, and B. Ommer. Imagebart: Bidirectional context with multinomial diffusion for autoregressive image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, 2021.
[11] P. Esser, R. Rombach, and B. Ommer. Taming transformers for resouutesotttin image synthesis. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 12873-12883, 2021.
[12] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin. Convolutional sequence to sequence learning. In International Conference on Muchine Learning, pages 1253-1252. PMLR, 2017.
[13] I. Goodfellow, Pouget-Abadie, M. Mirza, Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems (NeurIPS), pages 2672-2680, 2014.
[14] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems (NeurIPS), pages 5767-5777, 2017.
[15] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference Neural Information Processing Systems (NeurlPS), pages 0026,6637,22 2017.
[16] M. Hu, Y. Wang, T.-J. Cham, J. Yang, and P. Suganthan. Global context with discrete diffusion in vector quantised modelling for image generation. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
11

---

[17] X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). pages 1501-1510, 2017.
[18] X. Huang, M.-Y. Liu, S. Belongie, and Kautz. Multimodal unsupervised image-to-image translation. In Proceedings of the European conference on computer vision (ECCV), pages 172-189, 2018.
[19] T. Karras, M. Aittala, S. Laine, E. Härkönen, J. Hellsten, J. Lehtinen, and T. Aila. Alias- free generative adversarial networks. In Advances Neural Information Processing Systems (NeurIPS), volume 34, 2021.
[20] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4401-4410, 2019.
[21] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 81 11-8819, 2020.
[22] D. P. Kingma and P. Dhariwal. Glow: Generative flow with invertible Ixl convolutions. In Advances in Neural Information Processing Systems (NeurlPS), pages 10215-10224, 2018.
[23] D. P. Kingma and M. Welling, Auto-encoding variational bayes. In editor, editor, Proceedings of the International Conference Learning Representations (ICLR), 2014.
[24] D. Lee, C. Kim, S. Kim, M. Cho, and W.-S. Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[25] C. Nash, J. Menick, S. Dieleman, and P. Battaglia. Generating images with sparse represen- tations. In Proceedings of International Conference on Machine Learning (ICML), pages 7958-7968. PMLR, 2021.
[26] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In Pro- ceedings of International Conference on Machine Learning (ICML), pages 8162-8171. PMLR, 2021.
[27] T. Park, M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2337-2346, 2019.
[28] R. Rakhimov, D. Volkhonskiy, A. Artemov, D. Zorin, and E. Burnaev. Latent video transformer. In 16th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, VISIGRAPP 2021, pages 101-112. SeiTePress, 2021.
[29] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[30] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss. A. Radford, M. Chen, and Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages PMLR, 2021.
[31] A. Razavi, A. Van den Oord, and O. Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
[32] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[33] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3):211-252 2015.
12

---

[34] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans. In Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS), 2016.
[35] J. Sohl-Dickstein, E. A. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference Machine Learning (ICML), pages 2256-2265, 2015.
[36] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich- Keil, N. Raghavan, U. Singhal, R. Ra- mamoorthi, J. Barron, and R. Ng. Fourier features let networks learn high frequency functions low dimensional domains. In Advunces in Neural Information Processing Systems (NeurIPS),
pages 7537-7547, 2020.
[37] A. Vahdat and J. Kautz. NVAE: A deep hierarchical variational autoencoder. In Advances in Neural Information Processing Systems (NewrIPS), 2020.
[38] A. Van den Oord, N. Kalchbrenner, L. Espeholt, O. Vinyals, A. Graves, et al. Conditional image generation with pixelcnn decoders. Advances in Neural Information Processing Systems (NeurlPS), volume 29, 2016.
[39] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 30, 2017.
[40] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[41] Wang, C. Yang, Y. Xu, Y. Shen, H. Li, and B. Zhou. Improving gan equilibrium by raising spatial awareness. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[42] C. Wu, J. Liang, L. Ji, F. Yang, Y. Fang, D. Jiang, and N. Duan. N:"" uwa: Visual synthesis pre-training for neural visual world creation. arXiv preprint arXiv:2 2021.
[43] Y. Wu and He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018
[44] W. Yan, Y. Zhang, P. Abbeel, and A. Srinivas. Videogpt: Video generation using vq-vae and transformers. arXiv preprint arXiv:2104. 10157, 2021.
[45] J. Yu, X. Li, Y. Koh, H. Zhang, R. Pang. J. Qin. A. Ku, Y. Xu, J. Baldridge, and Y. Wu. Vector-quantized image modeling with improved VQGAN. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[46] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition (CVPR), pages 586-595, 2018
[47] C. Zheng, G. Song, T.-J. Cham, Cai, D. Phung, and L. Luo. High-quality pluralistic image completion via code shared vegan. arXiv preprini 224.12220. 2022.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. . You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license the code and datasets? [No] The code and the data are proprictary.
13

---

Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "wCX7KXajPC1", "reviewer": "Reviewer_GwN2", "summary": "The author's introduce a new VQ-GAN model with three improvements, spatial normalization of the quantized vectors, MaskGIT for quicker autoregressive reconstruction, and multichannel feature quantization.  They show better reconstruction performance with similar code size to other models and methods.", "questions": "Other papers have found that generalized normalization functions can learn as powerful representations as much deeper traditional neural network architectures without normalization.  Do the author's think this may be part of the power of the addition of normalization as they state it?  Or is it something specific to normalization before the MASKGit?", "limitations": "The authors address stock concerns about implications but I believe that larger implications are raised by the image of the child in Figure 4 (4th column).  The child's face has changed significantly from the original, but unlike traditional encoding techniques, does not betray any indication to the downstream user that the image has lost information or is in any sense \"uncertain\".  Traditional artifacts in simpler compression techniques may look bad to the eye but they at least faithfully convey to the user when information has been lost.  I think not enough concern is paid in this paper and in this literature to technologies that produce confident and clear images that are not what was captured and encoded on the other side, and that may fool the end user otherwise.  ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is obviously of interest to the NeurIPS community and the results are impressive.  The implication that normalization the quantized code vectors adds substantial improvement is interesting, and definitely opens up interesting areas to follow up on.  The use of MASKGit seems to me to be not a large contribution of this work, and while interesting, can be downplayed a bit in comparison to the normalization and use of multichannel quantization.\n\nThe weaknesses are that the paper is not always careful in its comparison to other methods (figure 3 doesn't show the code size or latent size in comparison between several different methods so it is fairly difficult to compare across methods).  More careful use of common tools from compression literature (on rate-distortion) would help clarify some of the comparisons across methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "l35Xob3uDP6", "reviewer": "Reviewer_D1KP", "summary": "The paper presents a new VQ-based image synthesis method. Based on MaskGIT, the paper proposes spatially conditional normalization and uses multichannel representation to improve the reconstructed image quality of the tokenization stage. The proposed spatially conditional normalization modulates the quantized vectors for a better reconstruction performance by inserting spatially variant information to the VQ decoder. The multichannel representation subdivides the encoded continuous latent along the channel dimension into multiple chunks and quantizes them with a shared codebook, which further improves the reconstruction performance by increasing the latent size.  For the generation stage, the paper modifies MaskGIT to sample the multichannel latent. Experimental results on two benchmark datasets show the proposed image synthesis method is efficient and effective for generating diverse and high-quality images.", "questions": "1.Section 4.3 says that MaskGIT and VIT-VQGAN performs better on ImageNet because they “use more GPUs for longer training”. I’m interested in the difference of training budget. Can the proposed Mo-VQGAN surpass MaskGIT given the same training budget?\n2.It would be better to conduct more ablation experiments on the generation performance. Which initialization of the spatially conditional normalization is better for the generation? Why is the spatially conditional normalization only used in the first three blocks of the VQ decoder?\n", "limitations": "The paper discusses an interesting limitation of the proposed method. The model sometimes generates images with a high-frequency appearance without the structure information, which may be attributed to the generation of multichannel representation. I think the choice of mask scheduling function in MaskGIT may be not optimal for the multichannel representation. Maybe the multi-channels of the same location should be masked and generated together, and the sampling can rely on the probability product of the multi-channels.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper is technically sound.\n+ The paper is well structured.\n+ The citations are extensive.\n\nWeaknesses:\n- The novelty is somewhat limited. The idea to improve the reconstruction ability of the tokenization stage is original and interesting. However, the proposed method heavily relies on existing techniques such as multichannel representation and MaskGIT.\n- The performance on image synthesis is not good enough. As shown in Table 3, the proposed Mo-VQGAN performs worse than MaskGIT for class-conditional image generation on ImageNet in terms of complexity and quality metrics. It weakens the contribution since the proposed Mo-VQGAN is based on MaskGIT.\n- The evaluation is somewhat limited. The main contribution of the paper is incorporate the spatially conditional normalization to modulate the quantized vectors. Figure 6(a) has shown that spatially conditional normalization can improve the reconstruction performance (rFID), especially for Fourier features. However, improving the reconstruction performance (rFID) does not necessarily improve the generation performance (FID), as shown in Figure 6(c). In my opinion, the experimental evaluation should report FID with or without the proposed spatially conditional normalization.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "WlNdWjaJ6BG", "reviewer": "Reviewer_hVkj", "summary": "The paper discusses a method for generating, reconstructing images using quantized representations. The key difference wrt prior work is that they modulate the quantized representations, that is, they propose to use AdaIN-like modulation of the quantized features. They claim that without this, the results are often repetitive. A further smaller contribution is to use several channels of quantized features, ie when the image is encoded they split features into 4 blocks along the channel dimension and quantize them with the same dictionary. \n\nThey show better reconstruction results and comparable or worse results on image synthesis. ", "questions": "Please see above", "limitations": "It would be great to better understand what the proposed stage-1 can do and what it cannot do. Like can it do outpainting, superresolution and other downstream applications. Reconstruction per se is a less interesting application.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and addresses a challenging, very competitive problem. The key contribution---modulating quantized vectors---is interesting although perhaps not sufficient on its own right. Improving stage-1 training is a very important problem, which potentially can improve many applications, including image synthesis, in- and out-painting, text-to-image, video synthesis. The paper, however, needs to be improved to be able to claim that their contribution lead to these improvements.\n\n**Motivation**: The paper is motivated by the claim that quantization results into repetitive structures. I wonder if the authors could support this statement somehow, since it's not easily observable in images if that's the case. Furthermore, if that's one of the motivations, it should be supported by numerical experiments, especially to show that the contributions improve the situation. It's not totally clear how do to this, but a method based on image autocorrelation might do the trick. The only support for this claim in the paper is fig 3. However, I cannot say it shows any repetitive content, besides, this image *can* be repetitive. Finally, even if there is repetitive content, is it due to the exactly the same tokens (a visualization of token indices will help here), or because the decoder collapses. Currently, the support behind the claim is not sufficient. Modulation can \"just\" improve expressivity of the the tokens, allowing the generator to have shorter tokens. \n\n**Methodology**: from a technical standpoint modulation of the tokens is a simple extension of the original quantized schema. The proposed multi-channel quantization is interesting too, but I cannot say that both of these contributions combined together bring the paper above the bar.\n\n**Significance**: Improving only the first stage makes the paper less interesting. Why did previous papers gain attention? I believe because they had good encoder-decoder frameworks, which offered rich latent spaces. These latent spaces can be used to solve a variety of generative tasks, such as reconstruction, generation, completion, outpainting, text-to-image and even text-to-video. If the current paper improves stage-1 it can perhaps also show better results at stage-2? In terms of stage-2 the paper reports image synthesis comparisons, in which the numbers are either on par or worse than the state-of-the-art. The authors say that their model is smaller, but the model size is not the main contribution of the work. I believe, there are many ways of making the model even smaller. If you make the model larger will it be better? In the supplement they show class-to-image generation results. According to table 3, there is no improvement over MaskGIT in terms of all of the scores. MaskGIT is even a smaller model. So in terms of significance of the proposed contributions it's hard to tell whether stage-1 improvements lead to improvements in downstream tasks. The numbers show the opposite. An intuition could be that the proposed framework reconstructs images better than others, because it provides larger, more expressive latent space, at the cost of poorer structure. Somehow the authors admit this in the limitations paragraph. It would be great if the paper could prove otherwise. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept