MaskBit: Embedding-free Image Generation via Bit Tokens
Authors: Mark Weber
Lijun Yu
Qihang Yu
Xueqing Deng
Xiaohui Shen
Daniel Cremers
Liang-Chieh Chen
Venue: Accepted by TMLR ()

Abstract: Masked transformer models for class-conditional image generation have become a compelling alternative to diffusion models. Typically comprising two stages - an initial VQGAN model for transitioning between latent space and image space, and a subsequent Transformer model for image generation within latent space - these frameworks offer promising avenues for image synthesis. In this study, we present two primary contributions: Firstly, an empirical and systematic examination of VQGANs, leading to a modernized VQGAN. Secondly, a novel embedding-free generation network operating directly on bit tokens - a binary quantized representation of tokens with rich semantics. The first contribution furnishes a transparent, reproducible, and high-performing VQGAN model, enhancing accessibility and matching the performance of current state-of-the-art methods while revealing previously undisclosed details. The second contribution demonstrates that embedding-free image generation using bit tokens achieves a new state-of-the-art FID of 1.52 on the ImageNet $256\times256$ benchmark, with a compact generator model of mere 305M parameters. The code for this project is available on https://github.com/markweberdev/maskbit.

Published in Transactions on Machine Learning Research (12/2024)
MaskBit: Embedding-free Image Generation via Bit Tokens
Mark Weber
mark-cs.wwberrBtam de
Technical University of Munich, MCML
Lijun Yu
Carnegie Mellon University
Qihang Yu ByteDance
Xueqing Deng
ByleDance
Xiaohui Shen
ByteDance
Daniel Cremers
Technical University of Munich, MCML
Liang-Chis Chich Chen
ByteDance
Reviewed on OpenReview: https: // openreview. net/ net/ forum? id= NYe2JuN3v3
Abstract
Masked transformer models for class-conditional image generation have become a compelling alternative to diffusion models. Typically comprising two stages an initial VQGAN model for transitioning between latent space and image space, and a subsoquent Transformer model for image generation within latent space these frameworks offer promising avenues for im- age synthesis. In this study, present two primary contributions: Firstly, empirical and systematic examination of VQGANs, leading to modernized VQGAN, Secondly, novel ambedding-free free generation network operating directly on bit tokens binary quantized representation of Lokens with rich semantics. The first contribution furnishes transparent, reproducible, and seghpeerforiig VQGAN model, enhancing accessibility and matching the performance of current stato-of-the-art methods while revcaling previously undisclosed details. The second contribution demonstrates that embedding-fin image generation using bit tokens achieves
state-of-the-art FID of 1.52 the ImageNet. 256 x 256 bench- mark, with compact generator model of mere 305M parameters. The code for this project is available on https://github http://hhhhb..oommakkeoooorr/aa/

---

Published in Transactions on Machine Learning Research (12/2024)
Figure 1: Generated images by the proposed MaskBit. We showcase samples from MaskBit trained Net at 256 256 resolution.
1
Motivation
Masked transformer models for class-conditioned image (Chang al. 2022 Yu et al. 2024a) or text-to- image CChang et al., 2023) generation have emerged as strong alternatives to aulo-regressive models (Esser
et al. 2021 Yu et al. 2022a Lee et al. 2022a and diffusion models (Ho al. 2020 Rombach et al. 2022
These masked transformer methods typically employ two-stage framework: discrete tokenizer (c.g., VQGAN (Esser et al. 2021)) projects the imput from image space to a discrete, compact latent space, while a transformer model (Vaswani et al. 2017 Devlin et al. 2018) serves as the generator to create images in latent space from a masked token sequence. These methods achieve performance comparable to state-of- the-art auto-regressive and diffusion models, but with the advantage of requiring significantly fewer sampling steps and smaller model sizes, resulting in substantial speed-ups.
Despite the succoss of masked transformer frameworks, the development details of strong tokenizer have been largely overlooked. Moreover, one key aspect of training modern VQGAN-based tokenizers (Chang et al. 2022 Yu et al.) [023 2024a) the perceptual loss remains undiscussed. Since latent space-based generation rolics on encoding and decoding the input image with the VQGAN tokenizer, the final imago quality influenced by both the generator network and the VQGAN. The importance a publicly available, high-performance VQGAN model cannot be overstated, yet the most widely-used model and code-base still originate from the original work (Esser et al.) (021) developed over three years ago. Although stronger, closed-source VQGAN variants exist (Chang al. 2022 Yu et al.) 2023 2024a), their details are not fully shared in the literature, creating a signilicant performance gap for researchers without access to these advanced tokenizers. While the community has made attempts (Besnier & Chen 2023 Luo et al. 2024] to reproduce these works, none have matched the performance (For both reconstruction and generation) reported in (Chang et al. 2022 Yu et al. 2023 2024a).
In this paper, we undertake systematic step-by-step study to elucidate the architcctural design and training process necessary to create modernized VQGAN model, referred to VQGAN+. We provide a detailed ablation of key components in the VQGAN design, and propose several changes to them, including model and discriminator architecture, perceptual loss, and training recipe. As result, we significantly enhance the VQGAN model, reducing the reconstruction FID from 7.94 (Esser et al. 2021) to 1.66, marking an impressive improvement of 6.28. Moreover, our modernized VQGAN+ improves the generation performance and establishes a strong, reproducible foundation for future generation frameworks based on VQGAN.
Furthermore, we delve into embedding-free tokenization Yuu et al. 2024a Mentzer
2024). specifically
implementing Tookup-Free Quantization (IFQ) (Yu et al. 2024a within improved tokenizer framework.
2

---

Published in Transactions on Machine Learning Research (12/2024)
Our resulting method, employs binary quantization process by projecting latent embeddings into K di- mensions and then quantizing them based on their sign values. This process produces bit. tokens, where each token is represented by K bits. We empirically observe that this representation captures high-level structured information, with bit tokens in close proximity being semantically similar. This insight leads us to propose a novel embedding- tokens, climinating the need for learning new embeddings (from VQGAN token indices to new embedding values) as required in traditional VQGAN-based generators (Esser et al., 2021 Chang et al. 2022 Yu et al,, 2024a]. Consequently, MaskBit achieves state-of-the-ae performance, with FID score of 1.52 the pop- ular ImageNet 256 x 256 image generation benchmark using compact generator model of 305M parameters. All implementations will be made publicly available to support further research in this field.
generation model, MaskBit, which directly generates images using bil
Our contribution can be summarized as follows:
We study the key ingredients of modern closed-source VQGAN tokenizers, and develop an open- source and high-performing VQGAN model, called VQGAN+, achieving a significant improvement of 6.28 rFID over the original VQGAN developed three years ago.
2. Building on our improved tokenizer framework, we leverage Lookup-Free Quantization (LFQ). We observe that embodding- free bit token representation exhibits highly structured semantics, important for generation.
3. Motivated by these discoveries, we develop a novel embedding-free generation framework, Mask which achieves state-of-the-art performance on the ImageNet 256 x 256 image generation benchmark.
2 Revisiting VQGAN
Image generation methods (Essor et al., 2021 Rombach et al., 2022 that operate in a latent space rather than in the pixel space require networks to project images into this latent space and then back to the pixel space. Such networks usually trained with a reconstruction objective, as is common in VAF-inspired 2021). Over time, the training frameworks for these networks have become increasingly complex [Esser et al. 2021 Yu et al. 2022a Chang et al. 2022:
methods (Kingma & Welling, 2014 Oord et al. 2017 Esser al.
Lee et al. 2022a Yu et al.] 2023 2024a). Despite this complexity, the exploration and optimization of these frameworks remain significantly underrepresented in the scientific literature. In particular, issues in reproduction arise when crucial details are not transparently discussed, making a comparison under fair and consistent conditions hard. Moreover, fully disclosing all details enables the community to scrutinize and validate the true advancements brought by new designs. In response to this challenge, we conduct systematic study on the essential modifications to the popular baseline For this study, we use the term "VQGAN" to refer to any network of this lype and "Taming- VQGAN" specifically to refer to the network published by (Esser et al. [222]). In the following, we provide an empirical roadmap for developing a modernized VQGAN, named VQGAN+, aiming to bridge the performance gap and make advanced image generation techniques more accessible.
VQGAN (Esser et al. 2021
2.1 Preliminaries
Variational Autoencoders (VAFs) (Kingma & Welling, 2014) compress high-dimensional data into low- dimensional representations using an encoder network and reverse this process with decoder network. VAEs are typically trained with reconstruction loss, such as the La loss. However, minimizing the L2 dis- tance alone is insufficient for achieving visually satisfying results. Therefore, perceptual loss term (Johnson ct al. 2016) based on the LPIPS metric (Zhang ct al. 2018) is used to reduce noise and improve image qual- ity. Veclor-Quantized VAEs (VQ-VAEs) (Oord et al. 2017) incorporate a learnable codebook that acts as a lookup table between the encoder and decoder. The codebook is trained with two Le losses: the commitment loss, which reduces the distance of the encoder output to the codebook entries, and the codebook loss, which minimizes the distance from the codebook entries to the encoder output. VQGANs (Esser et al. [021]] build top of TQ-VAEs by incorporating discriminator network to add adversarial objective to the training
3

---

Published in Transactions on Machine Learning Research (12/2024)
7.94
7
7.15
5
rFID
4.02
3
2.40
2.17
2.00
1.89
1.66
1.61
Taming VQGAN (128ch)
Symmetric design (64ch)
Capacity increase (128ch)
Perceptual loss:
Discriminator
Discriminator update #2
EMA
Longer training
Embedding free
update
ResNet
Figure 2: Detailed roadmap to build a modern VQGAN+. This overview summarizes the performance gains achieved by each proposed change to the architecture and training recipe. The reconstruction FID (xFID) computed against the validation split of ImageNet al resolution of 256. The popular and open- source "aming-VQGAN VQGAN [Esser al., 2021 serves as the baseline and starting point.
Tramework (Goodfellow et al. 2014). The discriminalor's goal to distinguish reconstructed images from original ones, while the VQ- VQ-VAE is trained to fool the discriminator.
2.2 The Experimental Setup
We follow standard practices to train and evaluate the network on ImageNet (Deng et al. 2009 Unless specified otherwise, all networks are trained with a batch size of 256 for 300,000 iterations. Evaluation is performed using the reconstruction Fréchet Inception Distance (rFID) Heusel et al. 2017 where lower values indicate better performance. We use Taming- point, due to its thorough study and open-source availability, ensuring reproducibility of results. Tarnnng- VQGAN an encoder-decode network that uses residual blocks (He et al.) 2016) with convolutions for low-level feature processing and interleaved blocks of convolutions and self-attention (Vaswani al.) 2017) for high-level feature processing. In experiments, embeddings with an output stride of 16, resulting in 16 x 16 latent embeddings for input images of 256 256 pixels. The learned vocabulary consists of 1,024 entries, each being 256-dimensional vector.
(Esser et al. 2021 as our bascline and starting
focus encoder networks that produce latent
2.3 VQGAN+: Modern VQGAN
Starting with the baseline Taming VQGAN (Esser al. 2021) that attains 7.94 rFID on ImageNet- 256 256 (Deng et al.] 2009), we provide a step-by-step talk-through and ablation of all changes in the following paragraphs. summary of the changes and their effects is given in Fig.
Basic Training Recipe and Architecture. The initial modifications the Haming-VQGAn baseline are follows: (1) removing attention blocks for purely convolutional design, (2) adding symmetry to the generator and discriminator, and (3) updating the learning rate scheduler. Removing the attention layers, as adopted in recent methods (Chang et al. 2022 Yu et al. 2023 2024a). reduces computational complexity without sacrificing performance. We also observed discrepancies ween the open-source implementation of Taming-VQGAN (Esser et al. 2021 and its description in the publication. The implementation uses two residual blocks per stage in the encoder but three per stage the decoder. For our experiments, we adopt a symmetric architecture with two residual blocks per stage in both the encoder and decoder, as described in the publication. Additionally, we align the number of base channels in the generator and discriminator.
4

---

Published in Transactions on Machine Learning Research (12/2024)
Specifically, we reduce the channel dimension of the generator from 128 to 64 to match the discriminator. Here, base channels refer to the feature dimension in the first stage of the network. The third modification involves the learning rate scheduler. We replace constant learning rate with a cosine-learning rate scheduler with warmup [Loshchilov & Hutter 2016). Specifically, we use a warmup phase of 5,000 iterations to reach the initial learning rale of le 4 and then gradually decay it to 10% of the initial learning rale over time.
Result: 7.15 rFID (an improvement of 0.79 rFID)
Increasing Model Capacity. Next, we increase the number of base channels from 64 128 for both generator and discriminator. This adjustment aligns the generator's capacity with that of the Taming- VQGAN bascline, except for the absence of attention blocks. As result, the generator uses 17 million fewer parameters overall compared to the Taming-VQGAN baseline, while the number of parameters in the discriminator increases by only 8.3 million.
+ Result: 4.02 rFID (an improvement of 3.13 rFID)
Perceptual Loss. The perceptual loss Johnson et al. 2016 Zhang et al. 2018 plays a crucial role in further reducing the rFID score. In Taming- VQGAN, the LPIPS (hang et al. 2018 score obtained by the
LPIPS VGG (Simonyan & Zisserman 2015) network minimized to improve image decoding. However, we noticed in the partially open-source code of recent work on image generation @Yu al., 2023 the
usage of a ResNet50 (He et al. 20166) network instead to compute the perceptual loss. Although confirmed by the authors of (Yu et al. 2023 2024a), this change has so far not been discussed in the literature. We believe this change was already introduced in carlier publications of the same group (Chang et al. 2022,, but never transparently documented. Following the open-source code of (Yu et al., 2023), we apply an Le loss on the logits of a pretrained ResNet50 using the original and reconstructed images. While incorporating intermediate features into this loss can improve reconstruction further, it negatively affects generation performance in our experiments. Therefore, we compute the loss solely on the logits.
Result: 2.40 rFID (an improvement of 1.62 rFID)
Discriminator Update. The original PatchGAN discriminator (Esser et al. 2021 employs 4 x 4 convo- lutions and batch normalization [loffe & Szegedy 2015), resulting in output resolution of 30 x 30 from 256 x 256 input and utilixing 11 million parameters (with 128 base channels). We propose to replace the 4 xx4 4 convolution kernels with 3 x 3 kernels and switch to group normalization (Wu & He 2018), producing a 16 x 16 output resolution to align the output stride between the generator and discriminator. These ad- justments reduce the discriminator's parameter count to 7.4 million. In the second update, following prior work YYu al. 2024a),, we replace average pooling for downsampling with a precomputed 4 Gaussian blur kernel using stride of (Zhang) 2019). Additionally, we incorporate LeCAM loss (Tenng al., 2021 to stabilize adversarial training.
Result: 2.00 rFID (an improvement of 0.4 rFID)
The Final Changes. Our last addition to the training framework is the use of Exponential Moving Average (EMA) (Polyak & Juditsky 1992). We found that EMA significantly stabilizes the training and improves convergence, while also providing a small performance boost. Orthogonal to the reconstruction performance, we also add an entropy loss (Yu et al.) 2024a) to ease the learning in the generation phase. To achieve the final performance of VQGAN+, we increase the number training iterations from 300,000 to 1.35 million iterations, following common practices YYu et al. 2024a Gao et al., 2023).
Result: 1.66 rFID (an improvement of 0.34 rFID)
2024a Mentzer al. 2024 have shown that An Embedding-Free Variant. Recent works Yu el. al. removing the lookup table from the quantization Oord et al. (017)) improves scalability. Following the Lookup-Frce Quantization (LFQ) approach (Yu et al. 2024a), we implement a binary quantization process by projecting the latent embeddings to K dimensions (K = 12 in this experiment) and then quantizing them based on their sign values. Due to this binary quantization, we can directly interpret the obtained quantized
5

---

embedding for each token, as the Published in Transactions on Machine Learning Research (12/2024)
input image
2nd
3rd
tth
5th
6th
reconstructed image
7th
8th 9th 10th
11th
12th
reconstructed images with ith bit flipped
Figure 3: Bit tokens exhibit structured semantic representations. We visualize images obtained after corrupting tokens via bit flipping. Specifically, we utilize our method to encode images into bit tokens, where each loken is represented by K-bits (K = 12 in this example). We then flip the i-th bit for all the bit tokens and reconstruct the images as usual. Interestingly, the reconstructed images from these bit-flipped tokens remain semantically consistent to the original image, exhibiting only minor visual modifications such changes in texture, exposure, smoothness, color palette, or painterly quality.
representations their token numbers. We use the standard conversion between base-2 and base-10 integers (e.g., 1001, = 910). In VQGAN+, the token is merely the index number of the embedding table; however, in this case, the token itself contains its corresponding representation, eliminating the need for an additional embedding. Implicitly, this defines a non-learnable codebook of size 2 Given the compact nature of binary quantization with only K feature dimensions, we do not anticipate significant gains reconstruction. However, will empirically demonstrate the clear advantages of this representation in the generation stage.
Result: 1.61 rFID (an improvement of 0.05 rFID)
3 MaskBit: A New Embedding- Image Generation Model
Since the typical pipeline of VQGAN-based image generation models (Chang et al. 2022 Yu et al. 2024a includes two stages with Stage-1 already discussed in the previous section, we focus the Stage-I1 in this section. Stage-II refers to transformer network that learns to generate images in the VQGAN's latent space. During Stage-II training, the tokenizer produces latent representations of images, which are then masked according to a prodefined schedule (Chang et al. 2022h. The masked tokens are fed into the Stago-II transformer network, which is trained with a categorical reconstruction objective for the masked tokens. This process resembles masked language modeling, making language models such as BERT (Devlin et al., 2018) suitable for this stage. These transformer models "relearn" input tokens themselves are just arbitrary index numbers.
Bit Tokens Are Semantically Structured. Following study of embedding-fre free tokenization, which eliminates the lookup table from Stage-1 (i.e., removing the need for embedding tables in VQ-VAE (Oord ct al.) 2017)) by directly quantizing the tokens into K-bits, we study the achieved represcntation more carefully. As shown in Yuu el al.) 2024a), this representation yields high-fidelity image reconstruction results, indicating that images can be well represented and reconstructed using bit tokens. However, prior arts (Yu ct al. 20244) have used different token spaces for Stago-I and Stago-II, intuitively allowing the Stago-II token focus more on semantics, while enabling the Stage-I token space capture all visual details necessary for faithful reconstruction. This raises the question:

---

Published in Transactions on Machine Learning Research (12/2024)
Stage-l
Stage-Il
input
reconstruction
ingut:
masked
predicted tokens
tokena
tokens
transforo er
encoder
decoder
VQGAN
lopicup
Lookup
index sharing
legend
codebook embedddings
codebook
Index masked
embeddiiggs Stage-ll
Stage-I
:
Input
reconstruction
input: bit tokens
masked bit tokene
predicted bit tokens
bit tokens
encoder
decoder
Laonstone
MaskBit (ours)
legend
token sharing
: (masked)
Figure 4: High-level comparison of the architectures. Our training framework comprises two stages for image generation. In Stage-l, an encoder-decoder network compresses images into a latent representation and decodes them back. Stage-II masks the tokens, feeds them into a transformer and predicts the masked tokens. Most prior art uses VQGAN-hased methods (top) that learn independent embedding tables in both stages. In VQGAN-based methods, only indices of embedding tables are shared across stages, but not the embeddings. In MaskBit, however, neither Stage-I nor Stage-II utilizes embedding tables. The Stage-I predicts bit tokens by using binary quantization shared bit tokens into groups, masks and feeds them into a transformer to predict the masked bit tokens. An illustration of grouping and masking of bit tokens can be found in the Appendix
the encoder output directly. The Stage-II partitions the
Are bit lokens also good representation for generalion, and can bil lokens eliminate the need for "re-learning" new embeddings in the typical Stage-Il II network?
To address this, first analyze the bit tokens' learned representations for image reconstruction. Specifically, given latent representation with 256 tokens from model (where each token is represented by K-bits, and each bit by either -1 or 1), we conduct a bit flipping experiment to show how the structure of the token space relates to the content. This test involves flipping the i-th bit (i.e., swapping -1 and 1) for all 256 bit tokens and decoding them as usual to produce images. The bit-flipped tokens each have a Hamming distance of from the original bit tokens. Surprisingly, we find that the reconstructed images from these bit bit-flipped tokens are still visually and semantically similar the original images. This indicates that the representation of bit tokens has learned structured semantics, where neighboring tokens (within Hamming distance of 1) are semantically similar to each other. We visualize example of this in Fig. 3 and provide more examples in Sec. E. We note that the structural differences in the latent space between bit tokens and learnable embeddings also have implications on the generator's representation, which we describe in Appendix D Since the generation model is usually semantically conditioned on external input such as classes, the model needs to learn the semantic relationships of tokens. Given this evidence the inherent structured semantic representation in bit tokens, we believe that the learned rich semantic structure in the bit token representation can thus be good representation for the Stage-ll generation model.
Proposing MaskBit. This motivates us to propose MaskBit for Stage-II, transformer-bace generative model that generates images directly using bit tokens, removing the need for the embedding lookup table

---

Published in Transactions on Machine Learning Research (12/2024)
required by existing methods (Chang et al. 2022 Yu et al. 2024a). Our training framework is illustrated in Fig. 4 Prior art using VQGAN approaches share only indices of the embedding tables between the two stages, but learn independent embeddings. Taking advantage of the built-in semantic structure of bit tokens, the proposed MaskBit can share the tokens directly between Stage-I and Stage-II. We discuss more detailed differences between the two approaches Sec. D Next, we provide details on bit token masking.
Masked Bits Modeling. The Stage-ll training follows the masked modeling framework (Devlin et al, 2013), where a certain number of tokens are masked (i.e., replacod with a special mask token) before being fed into the transformer, which is trained to recover the masked tokens. This approach requires an additional entry in the embedding table to learn the embedding vector for the special mask token. However, this presents challenge for an embedding-fiee setup, where images are generated directly using bit tokens without embedding lookup. Specifically, it raises the question of how to represent the masked bit tokens in the new framework. To address this challenge, we propose straightforward yet effective solution: using zeros to represent the masked bit tokens. In particular, a bit token t is represented as t E {-1,1)K (i.e., K-bits, with each bit being either -1 or 1), while we set all masked bits to zero. Consequently, these masked bit tokens do not contribute to the image representation.
Masked "Groups of Bits" Modeling. During training, the network is supervised with a categorical cross-entropy loss applied to all masked tokens. With increasing number of bits, the categorical cross-entropy gets computed over an exponentially increasing distribution size. Given that bit tokens capture channel- wise binary quantization, we explore masking "groups of bits". Specifically, for each bit token t E {] we split it into N groups to E (-1,1)K/N Vn E bits. During the masking process, each group of bits can be independently masked. Consequently, a bit token may be partially masked, allowing the model to leverage unmasked groups to predict the masked bits, easing the training process. An illustration of the grouping and masking procedure can be found in Appendix C During the inference phase, the sampling procedure allows to sample some groups and use their values to guide the remaining samplings. However, this approach increases the number of bit token groups to be sampled, posing challenge during inference due to the potential for poorly chosen samples. Empirically, we found that using two groups yields the best performance, striking a good balance.
NN, with cach group contains K/N consecutive
4 Experimental Results for Image Generation
Experimental Setup. We evaluate the proposed MaskBit on class- coiss-ooniiionall image generation. Specifi- cally, the network generates a total of 50,000 samples for the 1,000 ImageNet (Deng et al. 2009) classes. All Stago-II generative networks are trained and evaluated on ImageNet resolutions of 256 x 256. We report the main metric Fréchet Inception Distance for generation (gFID) (Heusel et al., 2017), along with the side metric Inception Score (IS) (Salimans et al., 2016). All scores are obtained using the official evaluation suite and reference statistics by ADM (Dhariwal & Nichol 2021). We train all Stage- networks for 1,080 epochs and report results with classifier-free guidance (Ho & Salimans
2022
More training and inference details
can be found in Sec. B
4.1 Main Results
Tab. 1 summarizes the generation results ImageNet 256 x 256. We make the following observations.
VQGAN+ as a New Strong Baseline for Image Tokenizers. By carefully revisiting the VQGAN design in Sec. - we develop VQGAN+, a solid new baseline for image tokenizers that achieves rFID of 1.66, marking a significant improvement of 6.28 rFID over the original Taming- VQGAN model [Esee et al.
2021). Using VQGAN+ in the generative framework of MaskGIT (Chang et al., 2022) results in a gFI of 2.12, outperforming several recent advanced VQGAN- methods, such as ViT VOQAV (Yu et al. 2022a
(by 0.92 gFID), RQ- Transformer (Lee et al.) 2022a) (by 1.68 gFID), and even the original MaskGIT
Chang
ct al.) 2 (by 1.90 gFID). This result shows that the improvements we propose to the Stago-I model,
indeed transfer to the generation model. Furthermore, our result surpasses several recent diffusion-basod
generative models that utilize the heavily optimized VAF (Kingma & Welling 2014) by Stable Diffusion (AT
8

---

Published in Transactions on Machine Learning Research (12/2024)
Table 1: ImageNet 256 x 256 generation results. t: Tokenizers trained on Openlmages (Kuznetovaa ct al. 2020). I: Tokenizers trained on OpenImages, ": Concurrent methods. All results use classifier-fres guidance or rejection sampling, but not both. #Params: Generator's parameters. #Steps: Sampling steps.
(Schuhmann al. 2022)
Tokenizer
Generator
Model
#tokens
codebook
gFID4
IST
# #Params
#Steps
continuous latent representation
LDM-4-G
Rombach et al.
2022
4096x3
3.60
247.7
400M
250
GIVT-L-A
Tschannen et a.
2024
256x
2.59
1.67B
256
U-VIT
[Bao et al.
2023
1024x4 4
2.29
263.9
501M
50
7T--
Peobles & Xic
2023
1024x4
2.27
278.2
675M
250
MDT
Gao et al.
2023 $
1024x4
1.79
283.0
676M
250
DiMR
Liu et al.
2024
1024x4
1.70
289.0
505M
250
MDTv2
Gao et a..
2024 2024
1024x4
1.58
314.7
676M
250
MAR-H
Li
256x 16
1.55
303.7
943M
256
discrete latent representation
Taming- VQGAN
[sser et al.
2021
256
1024
5.88
304.8
1.4B
256
MaskGIT Chang et al. 2022
256
1024
4.02
355.6
177M
8
RQ- JQ-Transformer [Lee et al. 2022al
256
16384
3.80
323.7
3.8B
64
ViT-VQGAN Yu et al. 2022a
1024
8192
3.04
227.4
1.7B
1024
LlanaGen-3B
Sun el al. 2024
576
16384
2.18
263.3
3.1B
576
TYTok-S- --8 YYu et al.
2024c
128
4096
1.97
281.8
287M
64
VAR Tian et al. 2024
680
4096
1.92
350.2
2.0B
10
MAGVIT-v2 YYu al.,
2024a)
256
262144
1.78
319.4
307M
64
MaskGIT (w/ our VQGAN+)
256
4096
2.12
300.8
310M
64
MaskBit (ours)
256
4096
1.65
341.8
305M
64
MaskBit (ours)
256
16384
1.62
338.7
305M
64
MaskBit (ours)
256
16384
1.52
328.6
305M
256
2022), including LDM (Rombach al.) 2022). U-VIT (Bao et al. 2023), and DiT (Peebles & Xie 2023
We believe these results with VQGAN- establish new solid baseline for the community.
Bit Tokens as a Strong Representation for Both Reconstruction and Generation. As discussed in previous sections, bit tokens encapsulate structured semantic representation that retains essential infor- mation for image generation, forming the foundation of MaskBit. Consequently, MaskBit delivers superior performance, achieving a gFID of 1.65 with 12 bits and 1.62 with 14 bits, surpassing prior works. No- tably, MaskBit outperforms the recent state-of-the-art MAGVIT-v2 (Yu et al. 2024a) while utilizing an implicit codcbook that is 16x to 64x smaller. When the number of sampling steps is increased from 64 to 256 matching the steps used by auloregressive models MaskBit sets a new state-of-the
with a gFID of 1.52, outperforming concurrent works (Gao et al., 2024 Li al.) 2024 Liu al. 2024 Sun al. 20244 Tian et al. 2024 Yu et al., 2024c), while using compact generator model of just 305M parameters. Re- markably, MaskBit, which operates with discrete tokens, surpasses methods relying continuous tokens, such MDTv2 (Gao et. al. 2024) and MAR-H [Li et al. 2024
Comparison under Similar Generator Parameters. In Tab. 2 we present performance comparison across models of similar sizes. Previous works have demonstrated that incrcasing model size generally lcads to superior performance (Li et. al.) 2024 Sun et. al.) 2024 Tian el al.) 2024). Therefore, to provide a more comprehensive comparison, we only compare models using a similar number of parameters, but vary codcbook size and sampling steps. As shown in the table, MaskBit not only achieves ttato-of-the-an results overall but also outperforms all other models under similar conditions. is worth noting that all models in the comparison larger Stage-T (reconstruction) tokenizer than ours.

---

Concurrent methods. All results use Published in Transactions on Machine Learning Research (12/2024)
Table 2: Comparison using similar generator model capacity (~ 300M) on ImageNet 256 x 256. t: Tokenizers trained on Openlmages (Kuznetsova al. 2020). classifier free guidance. #Steps: Sampling steps.
Tokenizer #tokens
Generator # #arrams
Model
# Params
codebook
gFID
1St
# #Steps
VAR TTan et al. 2024
109M
680
4096
3.30
274.4
310M
10
TYTok-S-128 YYu al,
2024c
72M
128
4096
1.97
281.8
287M
64
MaskBit (ours)
54M
256
4096
1.65
341.8
305M
64
LlamaGen-L (Sun et al.
2024
72M
256
16384
3.80
263.3
343M
256
MaskBit (ours)
54M
256
16384
1.52
328.6
305M
256
MAGVIT-v2 (uu et al.
2024a
116M
256
262144
1.78
319.4
307M
64
MaskBit (ours)
54M
256
262144
1.67
331.6
305M
64
Table 3: Ablation studies on ImageNet 256
256 generation. Our final setting is labeled in gray. To computations, shorter training schedule has been used for this study.
(a) Embedding-freee free Schemes
(b) Number of Groups
Tokenizer
Generator
Groups
gFID
embedding-frree free
embedding- free
gFID4
1
1.94
X
2.12
2
1.82
1.95
4
2.01
1.82
X
2.83
Qualitative Results. We present visualizations of the generated 256 x 256 images using the proposed MaskBit in Fig. 5 The results show the diversity of classes represented in ImageNet. MaskBit is able to generate high- fidelityy outputs. We note that these generated results inherit the dataset bias of ImageNet. As classification dataset, the images are centered around objects showing the respective class. Thus, this dataset bias influences the kind of images that can be generated. Methods used in production environments such Stable Diffusion
general large-scale datasets such LAION- 5B (Schuhmann et al. 2022 remove such dataset bias.
AT 2022 therefore train
4.2 Ablation Study
Unless stated otherwise, we use fewer training iterations for the ablation studies due to limited resources and computational costs.
Embedding-free Stages Bring Improvements. In Tab. 3a we investigate embedding--ree designs in both Stage-I (tokenizer) and Stage-II (generator). To ensure fair comparison, all models use the same mum- ber of tokens, codebook size, and model capacity. In the first row, employ VQGAN+ with traditional Stage-II model, as in MaskGIT (Chang et al. 2022), achieving gFID of 2.12. Using bit tokens in Stago-I improves performance to gFID of 1.95, highlighting the effectiveness of structured semant representation of bit tokens. Further enabling our Stage-ll model MaskBit (d.e., both stages now are embedding ( achioves the best gFID of 1.82, demonstrating the advantages embedding tokens. For completeness, we also explore VQGAN+ with an embedding- free Stage-II model, yielding a gFID 2.83. This variant performs worse due to the lack of the semantic structure the tokens.
image generation from bit
Two Groups of Bits Lead to Better Performance. Tab. 3b presents the performance of MaskBit using a different numbers groups. As shown in the table, when using two groups, each consisting of six consecutive bits, MaskBit achieves the best performance. This configuration outperforms single group
10

---

Published in Transactions on Machine Learning Research (12/2024)
Figure 5: Visualization of generated 256 x 256 images. MaskBit demonstrates the ability to produce high-fidelity images across diverse range of classes.
11

---

Published in Transactions on Machine Learning Research (12/2024)
Table 4: Ablation on the effect of different number of bits and different number of sampling steps on ImageNet 256 256. Table (a) uses 64 sampling steps, and Table (b) uses a 14 bit model.
(a) Number of Bits
(b) Number of Sampling Steps
Bits
#tokens
codebook
gFID
1St
Sampling Steps
gFID
ISt
10
256
1024
1.68
353.1
64
1.62
330.7
12
256
4096
1.65
341.8
128
1.56
341.9
14
256
16384
1.62
330.7
256
1.52
328.6
16
256
65536
1.64
339.6
18
256
262144
1.67
331.6
because it allows the model to leverage information from the unmasked groups. However, when using four groups, the increased difficulty in sampling during inference due the need for four times more sampling detracts from performance. Still, all group configurations outperform strong baseline using VQGAN+
well as modern approaches like ViT-VQGAN Yu et al. 2022a) and DīT-XL (Peebles & Xie, 2023 showing the potential of grouping. We also conduct another experiment, where apply 2 groups to VOGAN+. The performance drops form 2.12 to 4.4 gFID, due to the missing semantic structure in the tokens.
MaskBit with 14 Bits Yields Better Performance. In Tab. 4a we train the Stage-II models with our final training schedule. In each setting, MaskBit leverages different number of bits. Empirically, we find that using 14 bits works best on ImageNet. We note that all settings outperform prior methods (e.g., Peebles (eebles & Xie (2023) ; Yu et al. (2024a)) gFID and the performance differences are marginal across different number of bits. While using 14 bits is sufficient for ImageNet, more general and large-scale datasets such as LAION- 5B (Schuhmann et al. 2022 might benefit. from using more bits (Le., larger implicit vocabulary sizes).
MaskBit Scales with More Sampling Steps. MaskBit follows the non-autoregressive sampling paradigm (Chang ct al.
2022 Yu et al., 2023), enabling flexibiiltty in the number of sampling steps dur- ing inference (up to 256 sleps in our ImageNell 256x256 experiments). Unlike auloregressive models (Esser 2024). this approach allows for fewer forward passes through the Stage-Il genera- tive model, reducing computational cost and inference time. However, increasing MaskBit's 's sampling steps to match those of antoregressive models can also improve performance. As shown in Tab.[b] even with steps, MaskBit surpasses prior work and achieves an excellent balance between compute efficiency and performance. Scaling further to 256 steps sets new state-of-the-art gFID of 1.52.
et al. 2021 Sun et al.
5 Related Work
Image Tokenization. Images represented by raw pixels are highly redundant and commonly compressed to
a latent space with autoencoders (Hinton & Salakhutdinov 2006 Vincent et al. 2008). ), since the early days
of deep learning. Nowadays, image tokenization still plays a crucial role in mapping pixels into discrete (Oord
let al.
2017) or continuous (Kingma & Welling 2014
Higgins et al.
2017) representation suitable for gener-
ative modeling. We focus on the stream of vector quantization (VQ) tokenizers, which proves successful
state-of-the-art transformer generation frameworks [Esse al. 2021 Chang al. 2022 Yu et al.) b022b
2024a
Lezama, et al. 2022 2023 Lee et al.. 2022a Starting from the seminal work VQ-VAE (Ord et al.
2017
several key advancements have been proposed, such as the perceptual loss (Zhang et al. 2018) and
discriminator loss Karras et al. 2019) from Esser et al. 2021) or better quantization methods (Zheng et al.
2022 Lee et al. 2022a Zheng & Vedaldi 2023 Mentzer et al. 2024 Yu et al.) (024a), which significantly
improve the reconstruction quality. Nonetheless, modern transformer generation models (Chang et al. 2022
Yu et al. 2023 2024a) utilize an even stronger VQ model, with details not fully discussod in the literature, resulting a performance gap to the wider research community. In this work, we conducted a systematic study to modernize the VQGAN model, aiming demystifying the process to obtain strong tokenizer.
12

---

Published in Transactions on Machine Learning Research (12/2024)
Image Generation. Although various types of image generation models exist, mainstream state-of-the-at
methods are usually either diffusion-based (Dhariwal & Nichol 2021
Chen et al.
2023 Rombach et al.
2022 Peebles & Xie 2023 Li et al.
2023 Hoogeboom et al. 2023 auto seutoorrgressivebaaseee based
Chen et al. 2020
Esser et al. 2021 Yu ct al., 2022a;
Gafni et al. 2022 Tscannen et al.) 2024 Yu et al.
2024b) or masked-
translormer-bose Chang al. 2022 Lee et al.
2022b Yu et al. 2024a Lezama el al.
2022 2023 Among
methods exhibit competitive performance, while bringing a substantial speed-up, thanks a much fewer sampling steps. This research line begins with MaskGIT (Chang et al., 2022), which generates images from masked sequence in non-antoregrressvv manner where at each step multiple token can be predicted (Devlin et al. 2018). Follow-up work has successfully extend the idea to the video generation domain Yu et al.) 2023] Gupta et al., 2023), large-scale text-to-image generation (Chang eet al. 2023), along with competitive performance to diffusion models (Yu el al., 2024a) for image generation.
them, the maaskkodtrannsformr--aaaaee
Still, these works only utilize the tokens from the tokenizer to learn new embeddings in the transformer network, while we aim at an embedding-five generator framework, making better use of the learned semantic
structure from the tokenizer.
6 Conclusion
We conducted systematic experiments and provided a thorough, step-by-step study towards a modernized VQGAN+. Together with bit tokens, we provide two strong tokenizers with fully reproducible training recipes. Furthermore, we propose an embedding- generation model MaskBit, which makes full use of the rich semantic information from bit tokens. Our method demonstrates hate-of-theean results on the challenging 256 x 256 class-conditional ImageNet benchmark.
References
Stability AI. Stable diffusion vae, 2022. URL https::/huggingffee http:///w..o./.aaaaaaaaaa/a/raae
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusson models. In CVPR, 2023.
Victor Besnier and Mickael Chen. A pytorch reproduction of masked generative image transformer. arXiv preprint ar Xrx::22310.11 14400, 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In CVPR, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming Hsuan Yang, Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transformers. In ICML, 2023.
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Nya Sutskever. Gener- ative pretraining from pixels. In ICML, 2020.
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. In ICLR, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NewrIPS, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022.
Shanghua Gao, Pan Zhou, Mling-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer strong image synthesizer. In TCCV, 2023.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Mdtv2: Masked diffusion transformer is 2024.
a strong image synthesizer. ar preprint
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 2014.
Yuchao Gu, Xintao Wang, Yixiao Ge, Ying Shan, and Mike Zheng Shou. Rethinking the objectives of vector-quantized tokenizers for image synthesis. In CVPR, 2024.
Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Mccra Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprial arXiv:2312. 26666222 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. NeurIPS, 2017.
Trina Higgins, Loic Mauthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthew M Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vac: Learning basic visual concepts with a constrained variational framework. In ICLR, 2017.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786), 2006.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprinl ar vrXic:2207.22988 2022.
Jonathan Ho, Ajay Jain, and Picter Abbool. Denoising diffusion probabilistic models. NewrIPS, 2020.
Emiel Hoogeboom, Jonathan Heek, and Tim Salimans. simple diffusion: End-to-ond diffusion for high resolution images. In ICML, 2023.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelorating deep network training by reducing internal covariate shift. In ICML, 2015.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In ECCV, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- Tust,, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
Doyup Lee, Chihcon Kim, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In CVPR, 2022a.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and WOOK SHIN HAN. Draft-and-revise: Effective image generation with contextual tq-transformer. NeurIPS, 2022b.
José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token-critic. In ECCV, 2022.
14

---

Published in Transactions on Machine Learning Research (12/2024)
José Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, and Irfan Essa. Predictor-corrector sampling for discrete diffusion models. In ICLR, 2023.
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. NewrIPS, 2024.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Qihao Liu, Zhanpeng Zeng, Ju He, Qihang Yu, Xiaohui Shen, and Liang- Chieh Chen. Alleviating distortion in image generation via multi-resolution diffusion models. NeurIPS, 2024.
Πya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In ICLR, 2016.
lly Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Zhuoyan Luo, Fengyuan Shi, Yixiao Ge, Yujiu Yang, Limin Wang, and Ying Shan. Open-magvit2: An open- source project toward democratizing auto-regressive visual generation. arXiv preprint orXiv:2400.04411 044004 2024.
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite scalar quantization: VQ-VAE made simple. Tn ICLR, 2024.
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. NewrlPS, 2017.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV, 2023.
Boris T Polyak and Anatoli B Juditsky. Acceleration of stochastic approximation by averaging. SIAM journal on control and optimization, 30(4), 1992.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
Tim Salimans, lan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NeurIPS, 2016.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 2022.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06522 06525, 2024.
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. NewrIPS, 2024.
Michael Tschannen, Cian Eastwood, and Fabian Mentzer. Givt: Generative infniite-vocabulary transformers. ECCV, 2024.
Hung- Yu Tseng, Lu Jiang, Ce Liu, Ming-Hsuan Yang, and Weilong Yang. Regularizing generative adversarial networks under limited data. In CVPR, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Mllia Polosukhin. Attention is all you need. NewrIPS, 2017.
15

---

Published in Transactions on Machine Learning Research (12/2024)
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and PeerreAnttine Manzagol. Extracting and composing robust features with denoising autoencoders. In ICML, 2008
Yuxin Wu and Kaiming He. Group normalization. In ECCV, 2018.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vegan. In TCLR, 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexan- der Ku, Yinfei Yang, Bureu Karagol Ayan, et al. Scaling autoregressive models for content-rich text-to- image generation. Xiv preprint or Xix::2206. 10789, 2(3), 2022b.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Mang-ssunn&a Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In CVPR, 2023.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion-tokonizerr is key to visual generalion. In ICLR, 2024a.
Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieli Chen. Randomized autoregressive visual generation. orXiv preprint crXio:2411.00776, 2024b.
Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chiel Chen. An image is worth 32 tokens for reconstruction and generation. NewrIPS, 2024c.
Richard Zhang. Making convolutional networks shift-invariant again. In ICML, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable eflectiveness of deep features as a perceptual metric. In CVPR, 2018.
Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In ICCV, 2023.
Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. NewrIPS, 2022.
16

---

Published in Transactions on Machine Learning Research (12/2024)
Appendix
In the appendix, we provide additional information as listed below:
Sec.
provides the dataset information and licenses.
Sec.
provides all implementation details.
Sec.
C
provides a visual explanation on masking groups of bit tokens.
Sec.
D
provides more discussion on bit tokens.
Sec.
E
provides more reconstructed images for the bit flipping experiments ImageNet and COCO.
Sec.
provides zero-shot reconstruction results on COCO.
Sec.
provides ncarest-neighbor analysis on ImagetNet.
Sec. discusses the method's limitations.
Sec. discusses the broader impacts.
A Datasets
ImageNet:
ImageNet (Deng et al. 2009) is one the most popular benchmarks in computer vision. has been used to benchmark image classification, class-conditional image generation, and more.
License: Custom License, non-commercial. https://image net. oet.arr/aaccesssrremmmnt
Dataset website: httpss//amaae-nett.oo
B Implementation Details
In the following, we provide implementation details for the purpose of reproducibility:
B.1 Stage-l
For data augmentation, we closely follow the prior arts (Yu et al. 2023 2024a). Specifically, we use random cropping of 80%-100% of the original image with random aspect ratio of 0.75-1.35. After that, resize the input to 256 256 and apply horizontal flipping.
Base channols: 128
Base channel multiplier per stage: [1,1,2,2,4]
Residual blocks per stage:
Adversarial loss enabled at iteration: 20000
Discriminator loss weight: 0.02
Discriminator loss: hinge loss
Perceptual loss weight: 0.1
LeCam regularization weight: 0.001
L2 reconstruction weight: 4.0
Commitment loss weight: 0.25
17

---

Published in Transactions on Machine Learning Research (12/2024)
Entropy loss weight: 0.02
Eutropy loss temperature: 0.01
Gradient clipping by norm: 1.0
Optimizer: AdamW Loshchilov & Hutter 2019
Betal: 0.9
Beta2: 0.999
Weight decay: le-4
LR scheduler: cosine
Base LR: lc-4
LR warmup iterations: 5000
Training iterations: 1350000
Total Batchsize: 256
GPU: A100
B.2 Stage-Il
For data augmentation, closely follow the prior arts (Yu et al. 2023
2024a). Specifically, we random
cropping of 80%-100% of the original image. After that, we resize the input to 256 x 256 and apply horizontal flipping.
Hidden dimension: 1024
Depth: 24
Attention heads: 16
MLP dimension: 4096
Dropout: 0.1
Mask schodule: arccos
Class label dropout: 0.1
Label smoothing: 0.1
Gradient clipping by norm: 1.0
Optimizer: AdamW [Loshchilov & Hutter 2019
Betal: 0.9
Beta2: 0.96
Weight decay: 0.045
LR scheduler: cosine
Base LR: 1e-4
LR warmup iterations: 5000
18

---

Published in Transactions on Machine Learning Research (12/2024)
Training iterations: 1350000
Total Batchsize: 1024
GPU: A100
We use 32 A100 GPUs for training Stage-I models. Training time varies with respect to the number of training iterations between 1 (300K iterations) and 4.5 (1.35M iterations) days. We note that the Stage-I model can also be trained with fewer GPUs without any issues, but this will accordingly increase the training duration. Stage-II models are trained with 64 A100 GPUs and take 4.2 days for the longest schedule (1.35M iterations).
B.3 Sampling Procedure
Our sampling procedure closely follows prior works for non-autooregresive image generation (Gao et al. 2023 Chang et al. 2022 Yu et al. 2024a 2023 Chang et al.; 2023). By default, the Stage-II network generates
images in 64 steps. Each step, a number of tokens is unmasked and kept. The number of tokens to keep is determined by an arccos scheduler (Besnier & Chen 2023 Chang et al. 2022). Additionally, we use
classifier-free guidance and a guidance scheduler (Gao et al. 2023 Yu et al., 2023). We refer to the codebase
available on /https://githhei httpa////woeaaeeeeeerrrrr/aaaaaa model.
for the detailed sampling parameters for each
B.4 Sensitivity to Sampling Parameters
In this subsection, detail the specific values for the sampling hyper-parameters to ensure reprodueibility, However, the reported FID scores can be obtained by large range of typer-parameters Due to the inherent randomness in the sampling procedure, results might vary by +0.02 FID. For the resolution of 256 x 256, we verified several choices of paramoters obtaining similar results: temperature E [7.3,8 8.3), guidance E [5.8, 7.8] and "scale pow" m E 3.55.
C
Visualization of Masking Groups of Bit Tokens
Fig. 6 provides a visualization of the approach, masking groups bit tokens, presented in this paper. When using a single group, bit token will either be masked completely or remain unchanged after the masking is applied. When using more than single group, the bit token is split into groups of consecutive bits. Each of these groups can be independently masked during the masking procedure, leading to some cases where the bit tokens are partially masked as shown in the illustration. Notably, within each group of bits, the masking can only be applied to all bits in the group.
D More Discussion of Bit Tokens
D.1 Connection of Semantic Structure and Generator Performance
Prior work Gu et al. 2024) studying the tokenizer objective has found that a strong semantic representation in the latent space important for good generation performance. This observation was made by using different variations of the LPIPS perceptual loss. The experiments presented this work See. 2 support the same observation. Specifically, VQGAN+ uses ResNet logits-based perceptual loss which found to work well for generation. However, as mentioned in the paragraph "Perceptual Loss", adding intermediate activations of ResNet to the perceptual loss decreased the generation performance even though it helps the reconstruction performance.
Given the insights on the importance of semantics, this work studies the latent space bit tokens and the connection the image content. The key finding is that bit tokens exhibit this uniquely constrained structure by design, which leads a structured semantic representation. In the next paragraph, we discuss how this structured representation affects the internal generation model feature representation per token.
19

---

Published in Transactions on Machine Learning Research (12/2024)
mesked bit tokens using
bit tokens
group
takens
legend
bits
bit tokens
(maskod)
tokens
mesked bit toke ns using groups
bits
Figure 6: Visualization of the process for masking of groups of bit tokens. The figure visualizes the masking procedure of bit tokens (12 bits) when using either one group (top row) or two groups (boltom row). When using a single group, a bit token can either be fully masked or fully mon-masked, When using two groups, each bit token split into two consecutive groups (f.e., the first group contains the first 6 bits and the second group has the last bits). These two groups can be independently masked. This leads to the case where bit Loken be partially masked, as shown in the illustration.
We refer the experimental results presented in Sec. 4.1 for quantitative study on how much benefit bit tokens with this structured semantic representation bring to the generation performance.
D.2 Implications of Different Token Spaces on the Generator
In prior work, only the token indices were shared between Stage-I and Stage-II models. In all cases, the Stage-IT model relearns an embedding table. Considering our setting of 12 bits, a MaskGIT Stage-IT model would learn new embedding table with 2 4096 entries. Each of these entries is an (arbitrary) vector in I004 with 1024 being the hidden dimension. We note that this approach is equivalent to replacing the embedding table with a linear layer processing one-hot encoded token indices. The linear layer would in this case do the same mapping as the embedding table and have the same number of parameters, i.e., 4096 x 1024, omitting the bias for simplicity.
In MaskBit, instead of token indices, the bit tokens are shared and processed directly. MaskBit also uses a lincar layer to map the model's internal representation, but this mapping is from 12 dimensions (when K 12) to 1024, which corresponds learning just. 12 embedding vectors. Thus, the learnable mapping is reduced by a factor of 341. This is possible as MaskBit does not use one-hot encoded representation, but bit token representation with 12 bits. The linear layer therefore adds/subtracts the learnable weights depending on the bit coefficients +1. It follows that the internal representation per token in MaskBil is not arbilrary in R1021 but significantly constrained. These constraints require a semantically meaningful structure in the bit tokens, while MaskGIT the representation for each token index is independent and arbitrarily learnable.
Example: Given two bit tokens 4 and la with only difference in the last bit,
20

---

Published in Transactions on Machine Learning Research (12/2024)
Table 5: Zero-shot reconstruction evaluation on COCO with 256 x 256 resolution.
Model
zero-shot rFID 4
Taming- VQGAN [Esse et al. 2021
16.3
VQGAN+
8.7
Embedding-fiee variant
8.3
+1
f =
t2
In MaskBit, the only difference when processing the tokens in the first linear layer that the last weight vector, corresponding the last bit in the tokens, is either added (for (2) or subtracted (for 41). Thus, the obtained results are not independent of each other and arbitrary, which different from prior work.
E More Reconstruction Results of Flipping Bit Tokens
We present additional reconstruction results from the bit flipping analysis. As shown in Fig. 7 the recon- structed images from these bit-flipped tokens remain visually and semantically similar to the original images, consistent with the findings in the main paper.
Furthermore, we repeat this experiment in zero-shot manner on COCO Lin et al. 2014 with the same model only trained on ImageNet. The results are visualized in Fig. demonstrating that the bit token's structured semantic representation also holds for COCO images.
F Zero-Shot Reconstruction Results on COCO
In Tab.5 5 we provide zero-shol evaluation of the presented Lokenizers of Sec. 2 on COCO (Lin al. 2014]
show how the modifications made the orginal VQGAN (Esser al.) 2021 impact the performance on more complex datascts. For this experiment,
the same models for the experiments in Sec. [ that were only trained ImageNet (Deng al., 2009). All images resized with their shorter edge to 256 and then center cropped before feeding them into the models. The results show that the modifications made to obtain VQGAN+ and its embedding variant also achieve improvements on more complex datasets in the zero-shot evaluation setting.
G Nearest Neighbor Analysis of Generated Images
Following the idea presented in Esser et al. (2021). we conduct nearest neighbor analysis. In this analysis, generated image used and compared to all images in the training set. For this comparison, we use two distance measures: LPIPS and Hamming distance. The LPIPS metric used to measure perceptual similarity of two images, while the Hamming distance takes the bit tokens of the encoded images and computes the bit-wise distance. In Fig, g and Fig. 10 we visualize the 10 nearest neighbors for each generated image and distance measure. The results show that the model is not just memorizing samples from the training set, but generates indeed new images. Since LPIPS and Hamming distance are two different measures, the obtained nearest neighbors also differ. For example, considering the neighbors to the generated images of the parrot or the space rocket Fig. g we observe that the nearest samples according to LPIPS focus more on staying true the reference color palette, while this is less the case for the closest samples according to the Hamming distance. This finding aligns with the observations in the bit-flipping experiment,
21

---

Published in Transactions on Machine Learning Research (12/2024)
where with small changes in the Hamming distance, the visual attributes can vary quite bit, while the semantic content stays the same.
H Limitations
In this study, the focus has been non-atttoregreesive, loken-based transformer models for generation. Duc to the computational costs, we did not explore using the modernized VQGAN model (VQGAN+) in conjunction with diffusion models. For similar reasons, MaskBit. was only trained on ImageNet, thus inheriting the dataset bias, i.e., focus on a single foreground object. This makes difficult to do have a qualitative comparison to methods trained on large-scale datascts such as LAION (Schuhmann et al., 2022). We also note, that the standard benchmark metric FID has received criticism due to it measuring only distribution similarities. However, due to the lack of better and established alternatives, this study uses FID for its quantitative evaluation and comparison.
Broader Impacts
Generative image models have diverse applications with significant social impacts. Positively, they enhance creativity in digital art, entertainment, and design, enabling easier and potentially new forms of expression, art and innovation. However, these models pose risks, including the potential for misuse such as deepfakes, which can spread misinformation and facilitate harassment. Additionally, generative models can suffer from social and cultural biases present in their training datasets, reinforcing negative stereotypes. We notice that these risks have become dilemma for open-source and reproducible research. The scope of this work however is purely limited to class-conditional image generation. In class-conditional generation, the model will only be able to generate images for predefined, public and controlled set of classes. In this case, the images and class set are coming from ImageNet. As result, we believe that this specific work does not pose the aforementioned risks and is therefore safe to share.
22

---

Published in Transactions on Machine Learning Research (12/2024)
input image
1
nd
3
4th
5th
6th
reconstructed image
7th
10
11
12th
reconstructed images with 1thh bit flipped
input image
nd
3rd
4th
5th
tth
reconstructed image
7th
8th
9th
10th
th
12th
reconstructed images with 1th bit flipped
input image
nD
3
tth
reconstructed image
th
8th
h0th
th
12th
reconstructed images with ith bit flipped
input image
3rd
5th
reconstructed
7th
8th 9th 10th
11
12
image
reconstructed images with 1th bit flipped
Figure 7: More examples demonstrating the structured semantic representation in bit tokens. Similar to the observation in the main paper, the reconstructed images from these bit-flipped tokens remain semantically similar to the original image, exhibiting only minor visual modifications such as changes in texture, exposure, smoothness, color palette, or painterly quality.
23

---

Published in Transactions on Machine Learning Research (12/2024)
Input image
1st
2nd
3d
4th
5
6th
reconstructed image
7sh
8th
10th necoustddtted images with 1sh bit flipped
9th
11
12th
Input image
1st
2nd
3
4th
5th
6th
reconstructed image
8th
9th
10th
11
12th
reconstructed images with 7th bit flipped
Input image
reconstructed
7th
8th
9th
10th
11
12th
image
reconstructed images with 1s bit flipped
Input image
1st
2nd
3d
4th
5th
6th
reconstructed image
gth
12th
reconstructed images bit flipped
Figure 8: Examples demonstrating the structured representation of bit tokens in a zero-shot sctting. The structured semantic representation can also be found when running the tokenizer in a zero-shot manner on the more complex dataset COCO (Lin al. 2014
24

---

Published in Transactions on Machine Learning Research (12/2024)
Nearest Neighbors LPIPS
Generated Image
Nearest Neighbors based Hamming distance
Nearest Neighbors based LPIPS
Generated Image
Nearest Neighbors based on Hamming distance
Figure 9: Nearest Neighbors of two generated Images. The figure shows the ten nearest. neighbors in the ImageNet training set terms LPIPS (top) and Hamming distance (bottom).
25

---

Published in Transactions on Machine Learning Research (12/2024)
Nearest Neighbors LPIPS
Generated Image
Nearest Neighbors based Hamming distance
Nearest Neighbors based LPIPS
Generated Image
Nearest Neighbors based on Hamming distance
Figure 10: Nearest Neighbors of two generated Images. The figure shows the ten nearest. neighbors in the ImageNet training set in terms of LPIPS (top) and Hamming distance (bottom).
26


Reviews:
[{"review_id": "ozJ7alSiLa", "reviewer": "Reviewer_UHqj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\n- Section 2 effectively outlines the step-by-step improvements to VQ-GAN, making it easier to understand the impact of each modification.\n\n- They plan to release the code for all experiments, which will be valuable for the deep generative modeling community, as there are few works that publish reproducible code.\n\n- The empirical results, especially the FID score achieved by MaskBit, are impressive and set a new benchmark for the field.\n\n**Weaknesses**\n\n- Although making the code available to reproduce the experiments is a key contribution of this work, the reviewers currently do not have access to it.\n\n- Regarding Figure 3, readers may not be convinced that this robustness property occurs exclusively with bit-token latents. The authors simply state, 'We note that this behavior is not present in VQGAN methods.' Even if that is true, it remains unclear why only bit tokens exhibit robustness against latent perturbations.\n\n- The discussion on the semantic structure of bit tokens from a robustness perspective is not well connected to their ability to provide a good representation for generation. Moreover, the current experiments do not clearly demonstrate how this affects generation performance.\n\n- The inference procedure of MaskBIT is not clearly explained.\n\n- It is unclear from the experiments how much the bit-token latents and bit grouping each contribute to MaskBit's improvement over MaskGIT.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents VQGAN+, an improved version of the VQGAN model through step-by-step examination of VQGANs, which not only modernizes the existing VQGANs but also enhances transparency and reproducibility. It also introduces MaskBit, which utilizes an embedding-free mechanism based on bit tokens. It achieves state-of-the-art results on ImageNet with fewer parameters, demonstrating the effectiveness of the bit token representation.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- I strongly recommend that the authors provide the code, at least to the ACs and reviewers, during the discussion period, as I believe it is the most significant contribution of this paper.\n\n- The authors should demonstrate that VQ-GAN is not robust against latent manipulation. However, it may be challenging to make a fair comparison between bit latents and VQ latents, as changing a code of VQ latent to a random one affects the latent representation more significantly than simply flipping a bit in MaskBit's latent.\n\n- I suggest that the authors explore bit latents trained on datasets other than ImageNet to verify whether the properties of bit tokens are universal. \n\n- I suggest that the authors include illustrations, pseudo-code, or mathematical formulas for the training and inference processes of MaskBit.\n\n- Figure 4 could be more informative by including notations and terminology used throughout this paper.\n\n- The phrase \"14 Bits are Enough for ImageNet\" may be misleading and could confuse readers. A more precise expression is recommended.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "nMAAXyEHC6", "reviewer": "Reviewer_r6hD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n* Open-source and reproducible implementation of look-up free quantization autoencoders and masked transformers, reaching SOTA performance on ImageNet. \n\n* Very clear, detailed, and reproducible experimental setting. The improvements on both VQGAN++ and bit-masked transformers are strong and can serve as a new baseline for practitioners. \n\nWeaknesses: \n\n* The distinction between MAGVIT-v2 (Yu et al.) and MaskBit is not clear from reading the paper. Indeed, it is stated in Page 6: \"This motivates us to propose MaskBit for Stage-II, a transformer-based generative model that generates images directly using bit tokens, removing the need for the embedding lookup table as\nrequired by existing methods (Chang et al., 2022; Yu et al., 2024a)\". However, Yu et al. precisely propose lookup-free quantization, which does not require embedding lookup table as far as I understand. The fact that (Yu et al., 2024a) propose lookup-free quantization is even acknowledged by the authors in other parts of the paper. Thus, how can we compare both approaches? What are the differences in MaskBit? \n\n* The main contribution, regarding the difference with MAGVIT-v2, seems to be the Group of Bits modelling. However, its impact seems weak, with only 0.12 improvement of FID evaluated on a single dataset and without any confidence interval on the measure. I doubt that the improvement given by group of bits is really robust. The slight variation of gFID could be \"in the noise\". I understand that launching several trainings is too costly, but at least several sampling of generated/reference images is doable and can give a gross idea of the standard deviation of gFID.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "* Propose group of bits masking as an extension of the framework proposed in (Yu et al., 2024), based on lookup-free quantization and masked modelling. \n\n* Demonstrate that SOTA results can be obtained on ImageNet from lookup-free quantization + (group of bits) masked modelling. \n\n* Provide open-source implementation and reproducible experiments of SOTA image tokenizers and masked transformers for image generation on ImageNet.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* To me, the most important point concerns the explanation of contributions (cf Weaknesses). How does MaskBit differ from MAGVIT-v2, which propose lookup-free quantization and masked modelling? In the abstract, it is written \"The second contribution demonstrates that embedding-free image generation using bit tokens achieves a new state-of-the-art FID of 1.52 on the ImageNet 256×256 benchmark\". I am completely ok with this. However, in other parts of the paper, it is not that clear anymore. For example, in the contributions page 2: \"We develop a novel embedding-free generation framework, MaskBit.\" Again (cf Weaknesses), how is MaskBit a novel embeding-free generation framework?", "broader_impact_concerns": "The broader impact concerns is reasonable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5CIkRaawNj", "reviewer": "Reviewer_eg8T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### **Strengths**\n\nMy overall impression of the paper is quite positive. The problem is well motivated and is grounded in an area that I think tends to get inadequate attention in the generative modeling literature, namely the nature and design of the data representation formed in \"Stage I\" versus the far more commonly explored improvements to the *generative* component of the problem in \"Stage II.\" The writing and presentation are generally strong throughout the paper, with the section discussing the development of VQGAN+ and its improvements to the baseline being especially well presented. The introduction of masking into lookup-free quantization (LFQ), while straightforward, is clean and elegant. The ImageNet results are compelling, and the results about the semantic structuring of the representations are quite interesting (although on that latter point, please see below). The authors are also doing a useful service to the research community by making their implementation publicly available.\n\n### **Weaknesses**\n\nThe authors acknowledge various potential limitations of having trained and tested only on ImageNet, and I appreciate their rationale for not extending the work to larger-scale data sets. Nevertheless, I do have a few lingering questions about whether the customizations that resulted in the superior performance of VQGAN+ are more specifically suited to ImageNet and its characteristics (i.e. \"focus on a single foreground object\") or if they can be expected to provide similar benefits on image data sets with more varied structure, which are the type more likely to be leveraged in current applications. For similar reasons, I wonder whether the findings about the bit tokens' semantic structuring will also hold on more complex data sets. (I suspect so, but I can't be sure.) \n\nThis is the only area where there may be a slight mismatch between the paper's claims and the evidence provided, but I do not believe it is to the point where I would say that the authors have not satisfied the claims-and-evidence criterion. The paper's explicitly claimed contributions are delivered on, and it's only *implicit* claims (such as the implied generality from referring to VQGAN+ as \"a modern VQGAN\") that are perhaps not completely established ... at least, not yet. This issue aside, I think that the paper makes a useful contribution as it stands, and the authors' public release of their implementation will help settle some of these remaining questions in follow-up work. \n\nMy other concerns about the paper are relatively minor and mostly amount to areas that I believe would benefit from clarification or could otherwise be strengthened. A few examples are:\n* Given the exclusive focus on ImageNet, it would be helpful to have a nearest-neighbor analysis of generative output versus training data, such as reported in [1]. There is also an interesting opportunity to conduct a neighbor analysis by comparing the LFQ encodings of data versus generated samples, as opposed to the LPIPS-based approach of [1]. This would allay some concerns about whether the optimizations presented in this work are targeting the ImageNet data specifically or can effectively generalize beyond them.\n* In testing the benefits of perceptual loss, the authors mention that \"incorporating intermediate features into this loss can improve reconstruction further, [but] it negatively affects generation performance.\" Does this mean higher FID in generation or a greater tendency to memorize the training data? In Section 2.3, every design choice is made to improve reconstruction FID, so I got curious when something (i.e. intermediate features) could have improved rFID further but was ruled out due to undesired effects on the downstream generation task. Presumably, the effects on generation of every choice made in Section 2.3 were tested, and if so, the authors might consider including those results. (There are some choices for how best to present them, and the large amount of real estate currently devoted to Figure 2's bar graph might be a good spot.)\n* In Section 2.2, the authors mention that \"[u]nless specified otherwise, all networks are trained with a batch size of 256 for 300,000 iterations.\" This seems clear enough, so am I correct in assuming that all of the listed improvements (other than longer training) were the result of training from scratch in the new configuration and not from a previous checkpoint?\n* The semantic structuring results in Section 3 are interesting, but I have to confess that I remain unsure as to exactly how they were obtained. The procedure itself seems clear enough: You have 12 bits, and you flip one of them across all bit tokens and decode the results to see what happens. This gives you the 12 variations on reconstruction, such as what is shown in Figures 3 and 6, indicating that \"the reconstructed images from these bit-flipped tokens are still visually and semantically similar to the original images.\" Here's where I got confused:\n> We note that this behaviour is not present in VQGAN methods. Conducting the same experiment with VQGAN+ leads to non-meaningful output, sharing no semantic or visual similarities with the original image.\n\nIf it's not VQGAN and not VQGAN+, what's doing the decoding for reconstruction? I may be missing something obvious here, but I remain stuck on it. And since it is one of the highlighted findings of the paper and the motivation for MaskBit, I want to make sure I understand it.\n\n[1] Esser, P., Rombach, R., and Ommer, B. (2021). Taming Transformers for High-Resolution Image Synthesis.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel approach to token-based class-conditional image generation through embedding-free binarized bit tokens, which the authors claim contain interesting semantic structure, unlike the more semantically opaque representations found in the lookup tables standard to other token-based approaches. The paper also examines the design space of an exemplar token-based model, VQGAN, and proposes a series of changes that lead to VQGAN+, an updated model that shows substantially better reconstruction performance relative to VQGAN, and achieves state-of-the-art performance on the ImageNet $256 \\times 256$ generation benchmark. The scheme for binarizing tokens according to sign value also allows for a simple way to perform bit masking through zeroing out groups of bits, which allows for separate training of a Transformer-based, LLM-style generative image model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The bulleted items listed above should be fairly easy to address, and I hope that the authors discuss them in the rebuttal and consider incorporating them into the paper for the benefit of other readers.\n\nThe larger, looming question I have regarding the potential influence of ImageNet on the results is harder to conclusively address without conducting larger-scale experiments that I concede would place an unnecessary burden on the authors. As I mentioned above, I believe that the paper makes a worthy contribution even if it leaves that issue somewhat unresolved. At a minimum, I think the nearest-neighbor analysis mentioned above would help push this issue to the side somewhat. Beyond that, I am primarily interested in what is made of this issue during the rebuttal discussion with the authors and from the input from the other reviewers.", "broader_impact_concerns": "The authors have adequately addressed any potential broader impact concerns in their statement in the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is recognized for clear motivation, strong writing, and significant contributions to both token-based image generation and latent representation design.
Public release of code and reproducible experiments is praised, addressing community needs.
Performance results on ImageNet are compelling, establishing new benchmarks.

Key Weaknesses and Authours' rebuttal:

1) Limited Dataset Scope: Reviewers noted that experiments are limited to ImageNet, raising questions about generalizability to datasets with more complex structures (e.g., COCO or FFHQ).
Authors provided zero-shot evaluations on COCO, demonstrating some transferability.
2) Distinction from MAGVIT-v2: Reviewers initially found the differences between MaskBit and MAGVIT-v2 unclear.
Authors clarified the unique aspects, such as MaskBit's shared representations and masking scheme.
3)Robustness of Improvements: A 0.12 improvement in gFID was deemed marginal without confidence intervals.
The authors addressed this with additional evaluations showing low standard deviations, confirming the reliability of improvements.
4) Additional Visual Analysis: Suggestions included nearest-neighbor analysis of generated images to training data.
The authors added these results, showing non-memorization of training samples.
5) Terminological and Procedural Clarifications: Requests were made for clearer explanations of semantic structure, decoding procedures, and grouped bit modeling.
Authors updated sections and illustrations for clarity.

Most reviewers leaned towards acceptance, highlighting reproducibility, community benefit, and solid contributions. The AE suggested the acceptance.
Decision: Accept (poster)