Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99%
Authors: Lei Zhu
Fangyun Wei
Yanye Lu
Dong Chen
Venue: NeurIPS 2024 poster (2024)
Keywords: Image quantization
image generation
Primary area: machine_vision

Abstract: In the realm of image quantization exemplified by VQGAN, the process encodes images into discrete tokens drawn from a codebook with a predefined size. Recent advancements, particularly with LLAMA 3, reveal that enlarging the codebook significantly enhances model performance. However, VQGAN and its derivatives, such as VQGAN-FC (Factorized Codes) and VQGAN-EMA, continue to grapple with challenges related to expanding the codebook size and enhancing codebook utilization. For instance, VQGAN-FC is restricted to learning a codebook with a maximum size of 16,384, maintaining a typically low utilization rate of less than 12% on ImageNet. In this work, we propose a novel image quantization model named VQGAN-LC (Large Codebook), which extends the codebook size to 100,000, achieving an utilization rate exceeding 99%. Unlike previous methods that optimize each codebook entry, our approach begins with a codebook initialized with 100,000 features extracted by a pre-trained vision encoder. Optimization then focuses on training a projector that aligns the entire codebook with the feature distributions of the encoder in VQGAN-LC. We demonstrate the superior performance of our model over its counterparts across a variety of tasks, including image reconstruction, image classification, auto-regressive image generation using GPT, and image creation with diffusion- and flow-based generative models.

Scaling the Codebook Size of VQGAN to 100,000 with a Utilization Rate of 99 %
Lei Zhu'
Fangyun Wei Yanye Lu' Dong Chen?
PPeknng University Microsoft Research Asia
zhulei@stu, edu.
edu. cn fawe@microsoft. com
yanye. lu@pku. edu. cn
doch@microsoft.com
Ours
Generation-DiT
1.0
Codebook Utilization Rate
Best-rFID: 2.62
Classification 1
Concration-Sit 1
Best-rFID: 3.41
0.8
13.4
13.7
0.6
73.6
10.3
VOGAN-EMA
0.4
Reconstruction J
9.1
Generation-LDM
0.2
VOGAN-FC
16.3
Best-rFID:4.29
0.0
1024
16384
50000
100000
Codebook Size
Generation-GPT J
(a) Codebook size utilization rate.
(b) Evaluation downstream tasks.
Figure 1: (a) Two enhanced versions of VQGAN 11, namely VQGAN-FC (Factorized Codes) and VQGAN-EMA (Exponential Moving Average), experience a decline codebook utilization rate and performance Codebook), effectively leverages an extremely large codebook, persistently maintaining a utilization rate of up to 99% and achieving higher performance. We highlight the best reconstruction rFID for each model. (b) Comparison among three models across various tasks. For image generation, we evaluate the applications of these three VQGAN variants GPT 22, LDM
their codebook sizes expand. In contrast,
method, VQGAN-LC (Large
3 DiT and SiT [
Abstract
In the realm of image quantization exemplified by VQGAN, the process encodes images into discrete tokens drawn from codebook with predefined size. Recent advancements, particularly with LLAMA 3, reveal that enlarging the codebook significantly enhances model performance. However, VQGAN and its derivatives, such as VQGAN-FC (Factorized Codes) and VQGAN-EMA, continue to grapple with challenges related to expanding the codebook size and enhancing codebook utilization. For instance, VQGAN-FC restricted to learning a codebook with of less
maximum size of 16,384, maintaining typically low utilization than 12% on ImageNet. In this work, we propose a novel image quantization model named VQGAN-LC (Large Codebook), which extends the codebook size 100,000, achieving an utilization rate exceeding 99%. Unlike previous methods that optimize each codebook entry, our approach begins with a codebook initialized with 100,000 features extracted by pre-trained vision encoder. Optimization then focuses on training a projector that aligns the entire codebook with the feature distributions of the encoder in VQGAN-LC. We demonstrate the superior perfor- mance our model over its counterparts across a variety of tasks, including image reconstruction, image classification, augr-segressive image generation using GPT, and image creation with diffusion- and flow-based generative models.
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Table I: We conduct a comparative analysis of our VQGAN-LC against two advanced variants of VQGAN [1 namely VQGAN-FC and VQGAN-EMA, focusing on the effects of enlarging their codebook sizes from 1,024 to 100K. The only difference among the three models lies in the initialization and optimization of the codebook. The evaluation covers both reconstruction and generation using the latent diffusion model (LDM) B
on the ImageNet dataset.
Reconstruction (rFID)
Generation with LDM 3
(FID)
Method
1,024
16,384
50K
100K
1,024
16,384
50K
100K
VQGAN-FC
4.82
4.29
4.96
4.65
10.81
9.78
10.37
10.12
VQGAN-EMA
4.93
3.41
3.88
3.46
10.16
9.13
9.29
9.50
VQGAN-LC (Ours)
4.97
3.01
2.75
2.62
9.93
8.84
8.61
8.36
1 Introduction
refers to the process of encoding an image into a set of discrete representations, also known image tokens, each derived from a codebook of pre-defined size. VQGAN [1] stands out as a prominent architecture, with an encoodr--uuantizr--eeee playing pivotal role in various applications, including: (1) training a GPT [2]
Image quantization
structure, image tokens to create images; (2) serving as an autoencoder in latent diffusion models (LDMs) 3 4
8 9 10
and generative models 112
1
14 with flow matching [5] 15 and (3) funetioning within large where its encoder processes input images and its decoder
multi-modality models 16
18 19], 17
assists in image generation.
In contrast to natural languages, which typically maintain a static vocabulary, image quantization models necessitate codebook of pre-defined size to convert images into discrete image tokens. The nature of image signals-comples. complex and continuous-mann form of lossy compression that is generally more severe than converting them into continuous feature maps. The capability of these models to represent images largely depends on the codebook size. Previous studies, such VQGAN [1] its improved versions, including VQGAN with exponential moving average (EMA) update (VQGAN-EMA) and VQGAN using factorized codes (VQGAN-FC), and its predecessors, like VQVAE 6 and VQVAE-2 77, have demonstrated that they can only learn a codebook with a maximum size of 16,384. These models often face unstable training or performance saturation issues when the codebook size is further increased, as shown in Table []] Additionally, they typically exhibit a low codebook utilization rate-for instance, under 12% in VQGAN-FC, as shown in Figure andddicaaiin that significant portion of the codebook remains unused, thereby diminishing the model's representational capacity. Furthermore, studies on large language models suggest that employing a tokenizer with an expanded vocabulary significantly enhances model efficacy. For example, the technical report for LLAMA - shows, "LLAMA 3 uses a tokenizer with a vocabulary of 128K tokens that encodes language much more efficiently, which leads to substantially improved model performance."
translating images into token maps a
In this study, investigate the scalability of codebook size in VQGAN and the improvement of its codebook utilization rate, thereby substantially enhancing the representational capabilities of VQGAN. Typically, as shown in Figure 2a), image quantization models like VQGAN are structured with an encooer-quaantzzr--eeood
architecture, where the quantizer is connected to codebook. For a given image, the encoder produces feature map that the quantizer then converts into a token map. Each token in this map corresponds to an entry in the codebook, based on their cosine similarity. This token map is subsequently used by the decoder to reconstruct the original image.
Generally, the codebook in VQGAN begins with a random initialization. Each entry (a.k.a. a token embedding) in the codebook is designated as trainable and undergoes optimization through either gradient descent 6 i 20 21 (Figure([bb) or an exponential moving average (EMA) update 22 (Figure ()) during the training phase. Nevertheless, in each iteration, only small amount of token embeddings, corresponding to the token maps of the current training batch, are optimized. As training progresses, these frequently optimized token embeddings gradually align more closely with the distributions of the feature maps generated by the encoder, compared to those less frequently or never optimized (referred to as inactive token embeddings). Consequently, these inactive token embeddings
httpp:://a //aa: wwwa. wwt.eccoobbbleeee wowm/blog//etaaa.//eooooo 11ama-33

---

are excluded from the training process and subsequently remain unused during the inference phase, resulting in poor codebook utilization.
Our approach deviates from conventional image quantization models by initiating with a codebook composed of N frozen features, sourced from pretrained image backbone like the CLIP-vision- encoder [23]. and utilizing datasets like ImageNet [24]. A projector is then employed to transition the entire codebook into a latent space, producing token embeddings. During the training process, it is the projector that is optimized, not the codebook itself, which distinguishes our method from traditional models. By optimizing the projector, we adapt the aggregate distribution of the codebook entries to align with the feature maps generated by the encoder. This contrasts with methods like VQGAN M.. where adaptations are made limited number of codebook entries to match the feature map distributions during each iteration. Our simple quantization technique ensures that almost all token embeddings (over 99%) remain active throughout the training phase. The process is depicted in Figure dd).
Our newly developed quantizer can be integrated directly into the existing VQGAN architecture, replacing its standard quantizer without requiring any changes to the encoder and decoder. This innovative quantizer enables the expansion of the codebook sizes up 100,000, while maintaining an impressive utilization rate of 99%. By comparison, the conventional VQGAN is limited to a codebook size of 16.384 with utilization rate of only 1 11.2% when applied to ImageNet. The advan- tages of a larger codebook with enhanced utilization are demonstrated across various applications, including image reconstruction, image classification, augo-segressive image generation using GPT, and image creation with di diffusion models and flow matching. Figure lillustrates the performance of our improved VQGAN, termed VQGAN-LC (Large Codebook), compared to its counterparts.
|
2 Related Work
Image Quantization. Image quantization focuses compressing an image into discrete tokens derived from a codebook [6] T [22 20 25 26 (2). VQVAE (6) introduces a method of quantizing patch-level features using the nearest codebook entry, with the codebook learned jointly with the encoder-decoder structure through reconstruction loss. VQVAE2 (7] enhances this by employing exponential moving average updates and multi-scale hierarchical structure improve quantization performance. VQGAN 1 further refines VQVAE by integrating adversarial and perceptual losses, enabling more accurate and detailed representations. ViT-VQGAN [1) replaces the CNN-based encoder-decoder (28) with a vision transformer and shows that using a factorized code mechanism with l2 regularization can improve codebook utilization. Reg-VQ (2] introduces prior distribution regularization to prevent collapse and low codebook utilization. Additionally, some approaches use codebook reset strategies to reset unused codebook entries during training [26] 30 31 [2] or utilizing stochastic quantization to enhance utilization rates [26] 29]. In contrast to these methods, the proposed VQGAN-LC initializes codebook entries using a pre-trained vision encoder on target datasets, ensuring nearly full codebook utilization throughout the training process and allowing for scaling up the codebook size to more than 100K.
Tokenized Image Synthesis. Substantial advancements have been achieved in the realm of image synthesis, particularly through image quantization techniques. Initial efforts such as PixeIRNN [33] employs LSTM networks (34] to autoregressively model dependencies between quantized pixels. Building upon this, the groundbreaking VQVAE [ introduces the quantization of image patches into discrete tokens, significantly enhancing generation capabilities when paired with PixeICNN (35). The iGPT [36] further advances the field by leveraging the powerful Transformer [37 for sequence modeling of VQ-VAE tokens. Recently, there has been a shift towards using non-autoregressive Transformers for image synthesis 112 1 14 M. which provide efficiency improvements over traditional
seae--saannbeeeee generation methods. Innovative approaches such as discrete diffusion models, including D3PMs (38] and VQ-Diffusion [39]. utilize discrete diffusion processes to model
the distribution of image tokens. Additional diffusion- based techniques [5] 4 5 40 41 compress images into latent representations using quantizers, thereby reducing both training and inference costs. Moreover, image quantizers can enhance large language models for both image synthesis and
understanding 17 16 18
42 [9]. Our work introduces a superior image quantizer, further refining
the image synthesis process.
3

---

VOGAN
Encoder
Quantizer
Decoder
Feature Map
Codebook
Token Map
Reconstruction
(a) Architecture of VQGAN.
Gradient Descent
EMA Update
Argmin
Angetiin
Gradient
EMA
Update
Random Initiallzation
Random Initialization
(Trainable)
(Updatrable)
(b) Codebook optimization in VQGAN(-FC).
(c) Codebook update in VQGAN-EMA.
Ours
Class Centers
Gradient
Argmin
Pre- trained
Projector
Vision Encoder
Trainable
Patch Level
Image Dataset
Feature Clustering
Frozen
(d) Codebook initialization and quantization process our VQGAN-LC.
Figure 2: (a) The enooder-quantizer-dcooor structure of VQGAN, with a codebook linked to the quantizer. (b) The codebook optimization strategy employed in VQGAN and VQGAN-FC. (c) The codebook update mechanism utilized VQGAN-EMA. (d) The codebook initialization and quantization process implemented in our VQGAN-LC.
Method
3.1 Preliminary
VQGAN. Let = (b, E R denote a codebook containing N entries, with each entry b, being D-dimensional trainable embedding with random initialization. As shown in Figure[Ja) a). VQ-GAN adopts an mncoder-quaniize--eeoodd structure. In this setup, an encoder processes image X of height H and width W to generate feature map E Rhx=xD with (h, w)) representing the latent dimensions. Subsequently, the quantizer maps Z to a token map 2, where each token 2 is an entry in B based on the cosine distance between Z and B. Finally, the decoder reconstructs the original image from the token map 2. The entire network is optimized using combination of losses, expressed as follows:
C
XXX - +allsallse
(2)
211 +Cp + LCAN, 2 B1 sg
(I)
CR
La
where sg(+) denotes the stop-gradient operation. The terms LR. LQ- Lp and LGAN represent the reconstruction loss, quantization loss, VGG-based perceptual loss []. and GAN loss [1] respectively. (lyper-parameter and / are set to 1.0 and 0.33 by default. As shown in Figure 2[b). we refer the codebook optimization strategy used in the original VQGAN as "gradient descent".
VQGAN-FC. VQGAN faces significant challenges with inefficient codebook utilization. To address this issue, the factorized code (FC) mechanism, initially proposed by ViT-VQGAN (1], is employed. We refer to VQGAN integrated with the FC mechanism VQGAN-FC. The key differences between VQGAN and VQGAN-FC are two-fold: 1) linear layer added project the encoder feature Z E x into low-dimensional feature Z' E ThxxxDD where D' & D; the codebook B. consisting of N D' -ddimensionnal trainable embeddings, is randomly initialized. Consequently, the

---

Codebook Utilization Rate
VQGAN-LC (Ours)
0.8-
VQGAN-ICC (Ours)
0.6-
VOGAN-EMA
0.4
VQGAN-EMA
0.2
VOGAN-FC
0.2
0.0
2
8
10
12
16
18
Training Epoch
VQGAN-FC
Figure 3: (Left) The codebook utilization rate over the training epoch. A codebook entry is considered utilized for the epoch it is used at least once. (Right) The average utilization frequency of each codebook entry over all epochs, with each pixel representing single entry. All models adopt codebook with a size of 100K and use images with resolution of 256 x 256 on ImageNet.
quantization loss in Eq. is reformulated as:
cQ= =
[allsg(( 2 calallsgt
Z'Il (>g(( (Z') 211.
However, as illustrated in Figure [a), the utilization rate of VQGAN-FC is only 11.2% on ImageNet when the codebook size is configured 16,384, and increasing the size of the codebook fails to enhance performance as demonstrated in Table
VQGAN-EMA. As depicted in Figure (c). this variant of VQGAN adopts exponential moving average (EMA) strategy to optimize the codebook. Specifically, Let B B denote the set of token embeddings used for all token maps in the current training batch. The set B updated through the EMA mechanism using the corresponding encoder features Z in each iteration. As result, the codebook does receive any gradients. Therefore, the quantization loss in Eq. i defined as:
LQ a |sg( 2)
Our results, highlighted in Figure indicate that VQGAN -MAA outperforms VQGAN-FC on various downstream tasks, leading to enhanced utilization of the codebook. However, expanding the codebook size continues pose significant challenge for VQGAN-EMA, detailed in Table
3.2 VQGAN-LC
Analysis of VQGAN-FC and VQGAN-EMA. In these enhanced versions of VQGAN, the codebook is initialized randomly. During each iteration, only a small subset of entries related to the current training batch are optimized. As a result, the frequently optimized entries become more aligned with the feature map distributions generated by the encoder, while the less frequently optimized entries remain underutilized. Consequently, a significant portion of the codebook remains unused during both the training and inference stages. Figure shows the codebook utilization rate over the training epoch and visualizes the utilization frequency each codebook entry once training is completed.
Overview. We present VQGAN-LC (Large Codebook), which allows for the expansion of the codebook sizes of up 100,000 while achieving a remarkable utilization rate of 99%. As illustrated in Figure d). our method diverges from VQGAN-FC and VQGAN-EMA in its design of the quantizer. into latent space, aligning the distributions of the feature maps generated by the encoder. This approach allows us scale the codebook size effectively without modifying the encoder and decoder, achieving extremely high utilization rate and resulting in superior performance across various tasks, as shown in Figure I TTallc[aan Figure It is important to note that increasing the codebook size incurs almost no additional computational cost.
maintain static codebook and train a projector to map the entire codebook
Codebook Initialization. To initialize a static codebook, we first utilize pre-trained vision encoder (e.g., CLIP with a ViT backbone) to extract patch-level features from the target dataset (e.g., ImageNet)
containing M images. This extraction results in set of features denoted as F {F(())
5

---

R R.,....M
fi=..==I.=-1 where FliJ) represents D-dimensional patch-level feature at location (i.j) in the m-th image, and (h, w) indicate the spatial dimensions F. Subsequently, we apply K-means clustering to F, resulting in N cluster centers (with a default value of N 100, 000). These cluster JR=1. where Cn the n-th center. Our codebook then
centers form the set C {Co E RD JN initialized using C.
Quantization. Unlike VQGAN, VQGAN-FC and VQGAN-EMA, which optimize the codebook directly, our approach involves training a projector P(-). implemented as a simple linear layer, to align the static codebook B with the feature distributions generated by the encoder E(() of our denote the projected codebook. For a given input VQGAN-LC. Let P(B)= = {b, E "DD) J
image X, the quantizer transforms the feature map Z = { E IR into a token map 2.
This quantization process can be expressed as == argmin| Z (id) blll-
6 EB'
Loss Function. We employ the same loss function specified in [ However, the key distinc- tion that our codebook remains frozen, while the newly introduced projector P(-) undergoes optimization.
3.3 Evaluation of Image Quantization Models
We evaluate the performance of VQGAN-FC, VQGAN-EMA, and our proposed VQGAN-LC across image reconstruction, image classification and image generation tasks.
Image Reconstruction. Images are processed through the encoder, quantizer, and decoder to produce reconstructed images. These reconstructed images are then compared to their original images using the rFID metric as the evaluation criterion.
Image Classification. Initially, the encoder and quantizer convert each image into token map. Subsequently, we utilize a ViT-B model [37], pre-trained with MAE 43 to train on all token maps for the purpose of image classification. Top-1 accuracy is used as the evaluation metric.
Image Generation. Image quantization models can be integrated with different image genera- tion frameworks, such as auto-regressive causal Transformers (GPT (2),, latent diffusion models (LDM 3) diffusion Transformers (DiT [4), and flow-based generative models (SiT [S), to facilitate image creation.
GPT. The encoder and quantizer transform each image into a token map 2, which is then flattened into a token sequence. Ultimately, GPT is trained on the collection of these token sequences.
LDM. It progressively adds noise onto the encoder feature Z. The training objective is to denoise and reconstruct Z. During the inference phase, the output from LDM is inputted into the quantizer and decoder of image quantization models to generate images.
DIT. This model is variant of LDM, distinguished by its use of a Transformer architecture the backbone. The incorporation of image quantization models into DiT follows the same approach as their integration into LDM.
SiT. This method presents a flow-based generative framework utilizing the DiT backbone. The integration of image quantization models in SiT follows the same methodology as in LDM and DiT.
4 Experiments
4.1 Setup
Implementation Details of Image Quantization. All image quantization models, including VQGAN, VQGAN-FC, VQGAN-EMA, and our proposed VQGAN-LC, utilize the same encoder and decoder of the original VQGAN. The input images are processed at a resolution of 256 x 256 pixels. The encoder (U-Net (X downsamples the input image by factor of 16, yielding a feature map Z
with dimensions of 16 x 16. The quantizer then converts this feature map into token map of the same size, which is subsequently fed into the decoder (U-Net) for image reconstruction. In our observations, the optimal codebook size for VQGAN, VQGAN-FC. and VQGAN-EMA is 16,384, whereas for our VQGAN-LC, the optimal codebook size is 100,000. Training is conducted on the

---

Table 2: Reconstruction performance on ImageNet-IK. The term "# Tokens" refers to the number of tokens used represent an image. The codebook utilization rate is computed across all training images. The FC and EMA mechanisms are originally introduced by ViT- (21 and VQVAE ( 7 respectively. important computational cost.
that increasing the codebook size incurs almost no additional
Method
# Tokens
Codebook Size
Utilization (%)
rFID
LPIPS
PSNR
SSIM
DQVAE 20
256
1,024
4.08
DF-VQGAN
46
256
12,288
5.16
DIVAE
4
256
16,384
4.07
RQVAE
22
256
16,384
3.20
ROVAE
16,384
2.69
RQVAE
1,024
16,384
1.83
DF-VQGAN [46
1,024
8,192
1.38
256
16,384
3.4
5.96
0.17
23.3
52.4
VQGAN
256
50,000
1.1
5.44
0.17
22.5
52.5
256
100,000
0.5
5.44
0.17
22.3
52.5
256
16,384
11.2
4.29
0.17
22.8
54.5
VQGAN-FC 21
256
50,000
3.6
4.96
0.15
23.1
54.7
256
100,000
1.9
4.65
0.15
22.9
55.1
16,384
3.41
23.5
VQGAN-EMA 7
256
50,000
40.2
3.88
0.14
23.2
55.9
256
100,000
24.2
3.46
0.13
23.4
56.2
256
16,384
99.9
3.01
0.13
23.2
56.4
256
50,000
99.9
2.75
0.13
23.8
58.4
VQGAN-LC (Ours)
256
100,000
99.9
2.62
0.12
23.8
58.9
1,024
100,000
99.5
1.29
27.0
71.6
datasets, utilizing 32 Nvidia V100 GPUs. For ImageNet-IK: we train for 20 epochs, whereas for FFHQ, we train for 800 epochs. The Adam optimizer [45] is used, starting with an initial learning rate 5e-4, This learning rate follows a half-cycle cosine decay schedule after a linear warm-up phase of 5 epochs.
ImageNet-1 1K 24 and FFHQ
the CLIP image encoder 23 with ViT-L/14 backbone, adding an additional 4 x 4 average pooling layer, to extract patch-level features from images in the training split of the target dataset (either ImageNet or FFHQ). These features are then clustered into N groups using the K-Means algorithm with CUDA acceleration. The cluster centers constitute the codebook. By default, N configured to 100,000. We specify the codebook entries to have a dimension of 8.
Codebook Initialization of Our VQGAN-LC. Unless otherwise specified,
Image Generation Models. For LDM 3. DiT 4 and SIT [ adopt their original architectures. For generation using GPT [2], we follow VQGAN M. using a causal Transformer decoder with 24 layers, 16 heads per attention layer, a latent dimension of 1,024 and total of 404M parameters. For ImageNet, we employ class-conditional generation, whereas for FFHQ, we use unconditional generation. In LDM. DiT, and SiT, classifier-free guidance generation. More implementation details can be found in Section A
implemented for class-conditional
Evaluation. In the image reconstruction task, evaluate performance using rFID, LPIPS, PSNR. and SSIM metrics on the validation sets of ImageNet and FFHQ. For image classification, we measure the top-1 accuracy using 50K generated images compared against the ImageNet training set. For FFHQ, the FID score determined using 50K generated images in comparison with the combined training and validation sets FFHQ. The codebook utilization rate is also reported for comparison, which is calculated as the ratio of active entries (tokens/codes) to the total size of the codebook.
ImageNet. For image generation, we calculate the FID score ImageNet
4.2 Main Results
Image Reconstruction. Tables and present the reconstruction performance for ImageNet and FFHQ, respectively. We make three key observations: Our method consistently achieves
7

---

Table 3: Reconstruction performance on FFHQ.
Method
# Tokens
Codebook Size
Utilization (%)
iFID
LPIPS
PSNR
SSIM
ROVAR
2,048
7.04
0.13
22.9
VQWAR
48
256
1,024
4.20
0.12
22.5
66.5
MQVAR
49
256
1,024
78.2
4.55
VOGAN
256
16,384
2.3
5,25
0.12
24.4
63,3
VQGAN-FC
256
16,384
10.9
4.86
0.11
24.8
64.6
VQGAN-EMA
256
16,384
68.2
4.79
0.10
25.4
66.1
VQGAN-LC (Ours)
256
100,000
99.5
3.81
0.08
26.1
69.4
Table 4: Image generation ImageNet- IK.
Method
Tokens
Codebook Size
Utilization (%)
FID
RQTransformer (GPT-480M) 22
256
16,384
15.7
ViT-VQGAN (GPT-650M) 21
256
8,192
11.2
DQTransformer (GP7-355M)
20
640
1,024
7.34
DQTransfommer (GPT-655M)
640
1,024
5.11
ViT-VQGAN (GPT-650M)
1,024
8,192
8.81
Stackformer (GPT-65IM)
49
1,024
1,024
6,04
LDM B
1,024
16,384
8.11
with GPT-404M 2
VQGAN FC 21
256
16,384
VQGAN EMA
256
16,384
83.1
16.3
VQGAN-LC (Ours)
256
16,384
99.9
16.1
VQGAN-LC (Ours)
256
100,000
97.0
15.4
with SiT-XL 5
VQGAN-FC 21
256
16,384
11.2
10.3
VQGAN-EMA
256
16,384
83.1
9.31
VQGAN-LC (Ours)
256
16,384
99.9
9.06
VQGAN-LC (Ours)
256
100,000
99.6
8.40
with DIT-XL f
VQGAN FC 21
256
16,384
11.2
13.7
VQGAN EMA
256
16,384
85.3
VQGAN-IC (Ours)
256
16,384
99.9
11.2
VQGAN-IC (Ours)
256
100,000
99.4
10.8
with LDM (3)
VQGAN-FC -C 21
256
16,384
11.2
9.78
VQGAN-EMA 7
256
16,384
83.1
9.13
VQGAN-LC (Ours)
256
16,384
99.9
8.36
VQGAN-LC (Ours)
256
100,000
99.4
8.36
VQGAN-LC (Ours)
1,024
100,000
99.4
4.81
codebook utilization rate over 99% across all codebook sizes on both datasets. 2) The reconstruction
performance improves consistently with the scaling of codebook size using our method. 3) Increasing the codebook size (e.g., VQGAN-LC with codebook size of 100,000 and 256 tokens), and the number of tokens to represent an image (e.g., RQVAE with a codebook size of 16,384 and 512 tokens) both enhance performance, with the former introducing almost no additional computational cost compared to the latter.
Image Generation. Table 4 shows the results of class-conditional image generation on ImageNet. All models (GPT, LDM, DIT, and SiT) demonstrate improved performance with the integration of our VQGAN-LC. regardless of their underlying architectures, which include auto-regressive causal Transformers, diffusion models, diffusion models with Transformer backbones, and flow-based generative models. The diversity the generated images increases due to utilization codebook, which has a size of up to 100,000 and utilization rate exceeding 99%. Table 5 displays the unconditional generation results on the FFHQ dataset. Notably, DiT and SiT, which use the Transformer architecture, require more extensive training data for optimizing diffusion- and flow-
large

---

Table 5: Image generation on FFHQ.
Method
# Tokens
Codebook Size
Utilization (%)
FID
Stackformer (GP7-307M)
1,024
7.67
DQTransformer (GPT-308M)
20
640
1,024
4.91
Stackformer (GPT:307M) 49
1,024
1,024
6.84
Stackformer (GPT-651M)
49
1,024
1,024
5.67
ViT-VOGAN (GPT-650M)
1.024
8,192
3.13
LDM 3
4.096
8,192
4.98
with LDM 3
VQGAN-FC
256
16,384
11.2
13.2
VQGAN-EMA
256
16,384
68.2
12.5
VQGAN-LC (Ours)
256
100,000
99.7
12.3
with GPT (404M)
VQGAN-FC
256
16,384
10.9
3.23
VQGAN-EMA
256
16,384
68.2
4.87
VQGAN-LC (Ours)
256
100,000
99.1
2.61
Table 6: Ablation study of using various codebook initialization strategies on ImageNet.
Strategy
Dataset
Model
Utilization (%)
rFID
LPIPS
PSNR
SSIM
Random Initialization
5,4
108.7
0.46
18.2
36.4
Random Selection
ImageNet
VIT-I
99.8
2.95
0.12
23.8
58.9
K-Means Clutering
ImageNet
ResNet-50
99.8
2.71
0.12
23.7
K-Means Clutering
ImageNet
ViT-B
99.9
2.70
0.12
23.8
58.7
K-Means Clutering
ImageNet
ViT-L
99.9
2.62
0.12
23.8
58.9
based generative models. Given that FFHQ significantly smaller than ImageNet, we limit our training FFHQ GPT and LDM.
Image Classification. Section 3.3 we discuss the training of an image classifier on dataset containing tokenized images. We fine-tune three ViT-B [37] models, pre-trained by MAE (43, using the tokenized images produced by the top-performing VQGAN-FC, VQGAN-EMA, and our proposed VQGAN-LC, on ImageNet. Both VQGAN-FC and VQGAN-EMA demonstrate optimal reconstruction performance when utilizing a codebook with 16,384 entries. As illustrated in Figure (b), method achieves top-1 accuracy of 75.7 on ImageNet, surpassing VQGAN-FC and VQGAN-EMA by margins 1.6 and 1.9, respectively.
Visualizations. Section C presents images generated by GPT incorporating our VQGAN-LC.
LDM 3, DiT and SiT (5),
4.3 Ablation Studies
Unless otherwise specified, we evaluate reconstruction performance on ImageNet across all studies.
Codebook Initialization. In Section 3.2 we describe the default codebook initialization approach. This involves using CLIP vision encoder with ViT-L backbone extract patch-level features from ImageNet, followed by K-means clustering algorithm to generate N cluster centers, resulting in the static codebook B. In Table 6, we evaluate two factors: 1) employing a CLIP vision encoder with different backbones (c.g., VIT-B (37] and ResNet-50 (50)) to extract patch-level features; and 2) utilizing non-clustering strategies to initialize the static codebook, including random initialization and random selection, where N features randomly chosen from all patch-level features of ImageNet. Our findings are threefold: 1) random initialization leads to extremely poor performance since the codebook remains frozen in VQGAN-LC; 2) using CLIP vision encoder with ViT-L backbone outperforms those using ViT-B and ResNet-50 backbones; and 3) K-means clustering produces more robust codebook.
Codebook Size. In Table 7 we incrementally increase the codebook size in our VQGAN-LC from 1,000 to 200,000. The performance shows minimal improvements beyond a codebook size of

---

Table 7: Ablation study of using different codebook sizes on ImageNet.
Codebook Size
Utilization (%)
rFID
LPIPS
PSNR
SSIM
1,000
100.0
4,98
0.17
55.3
10,000
99.8
3.80
0.14
23.3
57.2
50,000
99.9
2.75
0.13
23.8
58.4
100,000
99.9
2.62
0.12
23.8
58.9
200,000
99.8
2.66
0.12
23.9
59.2
Table 8: The computational cost of VQGAN-LC with different codebook sizes.
Codebook Size
MACs
Model Size
16,384
195.08G 195.70G
71.71M 71.72M
100,000
Table 9: Ablation study on codebook transferability. The term "source dataset->tareet dataset" indicates that the codebook initialized using the source dataset, while our VQGAN-LC trained on the target dataset.
Setting
Utilization (%)
rFID
LPIPS
PSNR
SSIM
FFHQ-FFHQ
99.5
3.81
0.08
26.1
69.4
ImageNet-FFHQ
99.4
4.08
0.08
26.1
69.4
ImageNet-
99.9
2.62
0.12
23.8
58.9
ImageeIm 99.9
2.91
0.12
23.8
58.9
100,000 speefia--ye only 0.01 PSNR and 0.03 SSIM gains observed, when the codebook size reaches 200,000. Therefore, we consistently use codebook size 100,000 in all experiments. The codebook utilization rate consistently exceeds 99% across all configurations.
Computational Cost. The primary inference cost of VQ-GAN is attributed to the encoder and decoder. Increasing the codebook size N incurs minimal additional cost since the matrix multiplication between F and is negligible compared to the encoder and decoder processing time. Table X we present the caltiply-accumulater (MACs) and model sizes for VQGAN-LC models with codebook sizes of 16,384 and 100,000, respectively, when inferring an image of size 256 x 256.
Codebook Transferability. In our standard setup, both the codebook initialization and the VQGAN- LCC training are conducted using the same dataset, either ImageNet or FFHQ. In Table we examine the transferability of the codebook by initializing it with one dataset and training our VQGAN-ICC on a different dataset. Our findings indicate that our method exhibits significant codebook transferability, highlighting the robustness of our codebook initialization process.
5 Conclusion
In this work, we introduce a novel image quantization model, VQGAN-LC, which extends the codebook size 100,000, achieving a utilization rate exceeding 99%. Our approach significantly outperforms prior models like VQGAN, VQGAN-FC and VQGAN-EMA, across various tasks including image reconstruction, image classification and image synthesis using numerous generation models such as auto-regressive causal Transformers (GPT), latent diffusion models (LDM), diffusion Transformers (DiT), and flow-based generative models (SiT), while incurring almost no additional costs. Extensive experiments on ImageNet and FFHQ verify the effectiveness of our approach.
6 Acknowledgements
This work was supported in part by the Natural Science Foundation of China under Grant 623B2001, Grant 6239431 I, and Grant 82371112 12; in part by Beijing Natural Science Foundation under Grant 7210008; and in part by the High-Grade, Precision and Advanced University Discipline Construction Project of Beijing under Grant BMU2024GJJXK004
10

---

References
[1] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[2] Alec Radford, Jeffney Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAI blog. 1(8), 2019.
[3] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[4] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195 4205, 2023.
[5] Nanye Ma, Mark Goldstein, Michael S Alhergo, Nicholas M Boffi, Fric Vanden- Fijjden,, and Saining Xic. Sit: Exploring flow and diffusion-busee -asssdd generative models with scalable interpolant transformers. arXiv preprint or Xiv:2401 087440, 2024.
[6] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[7] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vac-2. Advances in neural information processing systems, 32, 2019.
[8] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subhiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners. Advances in neural information processing systems, 13:1877-1 2020.
[10] OpenAL. Gpt-4 technical report, 2023.
[11] Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexci A Efros. Sequential modeling enables scalable learning for large vision models. arXiv preprint arXiv:2312.00778 2023.
[12] Huiwen Chang. Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[13] Tianhong Li, Huiwen Chang, Shlok Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 2142-2152, 2023.
[14] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. arXiv preprint arX1v:2404.02905, 2024.
[15] Quan Dao, Hao Phung, Binh Nguyen, and Anh Tran. Flow matching in latent space. arXiv preprint arXiv:2307.0800 008988 2023.
[16] Lijun Yu, Yong Cheng, Zhiruo Wang, Vivek Kumar, Wolfgang Macherey, Yanping Huang, David A Ross, Irfan Essa, Yonatan Bisk, Ming-Hsuan Hsuan Yang. et al. Spae: Semantic pyramid autoencoder for multimodal generation with frozen llss. arXiv preprint arXiv:2306.17842 2023.
[17] Hao Liu, Wilson Yan, and Pieter Abbeel. Language quantized autoencoders: Towards unsupervised text-image alignment. arXiv preprint a0X99 2023.
[18] Lei Zhu, Fangyun Wei, and Yanye Lu. Beyond text: Frozen large language models in visual signal comprehension. arXiv preprint arXiv:2403.0733. 07874, 2024.
[19] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aninuddha Kemhhavi. Unified-io: unified model for vision, language, and multi-modal tasks. In The Eleventh International Conference on Learning Representations, 2022.
[20] Mengqi Huang. Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22596 22605, 2023.
11

---

[21] Jiahui Yu, Xin Li. Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu. Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vggan. arXiv preprint urXiv:2110.0667,. 2021.
[22] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15523-11532 2022.
[23] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. International conference on machine learning, pages 8748-8763, PMLR, 2021.
[24] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai and Fei-Fei Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255, leee, 2009.
[25] Chuanxia Zheng. Tung-Long Vuong, Jianfei Cai, and Dinh Phung. Movq: high-fidelity image generation. Advances in Neural Information Processing Systems, 35:23412-23425, 2022.
Modulating quantized vectors
[26] Will Williams, Sam Ringer. Tom Ash, David MacLeod, Jamie Dougherty, and John Hughes. Hierarchical quantized autoencoders. Advances in Neural Information Processing Systems, 33:4524 4535, 2020.
[27] Jialun Peng. Dong Liu, Songcen Xu, and Houqiang Li. Generating diverse structure for image inpainting with hierarchical vq-vae. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10775-10784, 2021.
[28] Olaf Ronneherger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assistr Cetrventtoo-MMICC.. 2015: 18th international conference. Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[29] Jiahui Zhang, Fangneng Zhan, Christian Theobalt, and Shijian Lu. Regularized vector quantization for tokenized image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18467-18466 2023.
[30] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and llya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.003: 00341, 2020.
[31] Chuanxia Zheng and Andrea Vedaldi. Online clustered codebook. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22798-22807, 2023.
[32] Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight-through estimator: Overcoming optimization challenges in vector quantized networks. In International Conference on Machine Learning, pages 14096-1411 14113. PMLR, 2023.
[33] Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference machine learning, pages 1747-1756. PMLR, 2016.
[34] Alex Graves and Alex Graves. Long short-term memory. Supervised sequence lahelling with recurrent neural networks, pages 37-45, 2012.
[35] Aaron Van den Oord, Nal Kalehbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with pixelenn decoders. Advances in neural information processing systems, 29, 2016.
[36] Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pages (691-1703, PMI.R, 2020.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention you need. Advances neural information processing systems, 30, 2017.
[38] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete spaces. Advances in Neural Information Processing Systems, 34:17981- 17993, 2021.
12

---

[39] Shuyang Gu, Dong Chen. Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings the IEEE/CVF Conference on Computer Vixion and Puttern Recognition, pages 10696-10706, 2022.
[40] Zhicong Tang. Shuyang Gu, Jianmin Bao, Dong Chen, and Fang Wen. Improved vector quantized diffusion models. arXiv preprint urXiv:2205.16000 2022.
[41] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.
[42] Chen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang, Alan Yuille, and Jiahui Yu. De-diffusion makes text strong cross-modes modal interface. arXiv preprint arXiv:231 1.00018 2023.
[43] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. arXiv:2111.0633 2021.
[44] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401 4410, 2019.
[45] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[46] Minheng Ni, Xiaoming Li, and Wangmeng Zuo. Nuwa-lip: language-guided image inpainting with defect- free vggan. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 141922 2023.
[47] Jie Shi, Chenfei Wu, Jian Liang, Xiang Liu, and Nan Duan. Divae: Photorcalistic images synthesis with denoising diffusion decoder. arXiv preprint arXiv:2206.003880 2022.
[48] Tung-Lome Vuong. Trung Le, He Zhao, Chuanxia Zheng. Mehutash Harandi, Jianfei Cai, and Dinh Phung. Vector quantized wasserstein auto-en arXiv preprint arXiv:2302.05// 059117, 2023.
[49] Menggi Huang, Zhendong Mao, Quan Wang, and Yongdong Zhang. Not all image regions matter: Masked vector quantization autoregressive image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2002 2011, June 2023.
[50] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778 2016.
13

---

Table 10: The impact of maintaining static codebook and incorporating projector on ImageNet.
Static
Projector
Utilization (%)
rFID
LPIPS
PSNR
SSIM
3.9
18.6
0.36
19.2
40.4
99.1
2.65
0.12
23.7
58.0
99.9
2.62
0.12
23.8
58.1
Table 11: Ablation study on the dimension of the projected codebook ImageNet.
Dimension
Utilization (%)
rFID
LPIPS
PSNR
SSIM
99.8
2.66
0.12
23,9
59.2
16
99.8
2.36
0.12
23.8
58.8
32
99.8
2.75
0.13
23.6
58.7
128
99.8
2.37
0.12
23.6
58.3
256
99.8
2.49
0.12
23.9
59.4
512
99.9
2.76
0.12
23.5
58.1
More Implementation Details
We train GPT with a batch size of 1024 across 32 Nvidia V100 GPUs. The Adam optimizer is employed with an initial learning rate of 4.5e and the model is trained for 100 epochs. Moreover, linear decay schedule is used for adjusting the learning rate. 5-epoch linear warm-up phase is adopted. Top-k sampling is adopted for auto-regressive generation, where is set as 10% of the vocabulary size.
GPT.
LDM. The implementation utilizes four UNet layers with channel dimensions of (256, 1024, 1024, 256). Conditions integrated through cross-attention mechanism each UNet layer. The model is trained using the Adam optimizer with an initial learning rate of 4.5e and batch size of 448, distributed across 8 Nvidia V100 GPUs. The training process is conducted over 100 epochs. The classifier-free guidance scale is set as 1.4 for class-conditional generation.
DIT. We employ the 28-layer DiT-XL model with a patch size of 2, consisting of 675 million parameters. The model features 16 attention heads and embedding dimension of 1152. To handle class conditions, we utilize the AdaLN- with initial learning rate of 4.5e- -, and the model is trained for 400,000 iterations on the ImageNet dataset. The training is performed with a batch size 256 across 8 Nvidia V100 GPUs. The classifier-free guidance scale set as 8 for class-conditional generation.
block. For optimization, the Adam optimizer used
SIT. We utilize SiT-XL as our flow-based generative model, mirroring the architecture DiT-XL. For optimization, the Adam optimizer is employed with initial learning rate of 4.5e The model undergoes training for 400,000 iterations on the ImageNet dataset. The training process conducted with batch size of 256, distributed across 8 Nvidia V V100 GPUs. The classifier-fre guidance scale set to 8 for class-con generation.
B More Experiments
Projector and Static Codebook. As described in Section 3.2
codebook is initialized using
CLIP vision encoder to extract patch-level features on ImageNer. During the training of our VQGAN- LC, we optimize a projector to map the entire codebook to latent space. Table I0 illustrates the significance of tuning the projector the static codebook by comparing alternatives: !) omitting the projector; and 2) making each entry in the initialized codebook trainable. Our results show that incorporating the projector markedly improves performance, whereas making the codebook entries trainable has minimal impact.
default strategy with two
Dimension of the Projected Codebook. In our VQGAN-LC, the projector transforms the codebook into latent codebook B', with each entry in B' having dimension denoted as D'. Table [] shows the results of varying D' from 8 to 512. We find that our model's performance remains stable regardless of the value of D'. consistently achieving a codebook utilization rate of over 99% in all configurations.
14

---

Codebook Size (1024)
Codebook Size (16384)
Codebook Size (50,000)
Codebook Size ( (100,000)
VQGAN-EMA VOGAN-FO
Uniecatinn
11.2% Uniication
100% Unilization
23,
Unlization
Unlization
VQGAN-LC (Ous)
100% Unlization Rate
99. 9% Utilzation Rat
99.9% Utilication Rate
99.9% Uniliation Rate
Figure 4: Visualization of the active and inactive codes for three models (VQGAN-FC, VQGAN- EMA, and our VQGAN-LC) using t-SNE.
Visualization of Active and Inactive Codes. Figure the distribution of the active and inactive codes for three models: VQGAN-FC, VQGAN-EMA, and our VQGAN-LC. The visualization is created using t-SNE. Active codebook entries are highlighted in green, while inactive ones are shown in blue. As the codebook size increases, more codes tend to be inactive in both VQGAN-FC and VQGAN-EMA models.
Token Replacement. VQGAN, VQGAN-FC, and VQGAN-EMA all utilize a codebook of 16,384 entries achieve optimal performance. In contrast, our VQGAN-LC can scale the codebook size up to 100,000 while maintaining an exceptionally high utilization rate of over 99%, allowing each token to represent more detailed visual elements. This verified through an ablation study: for each input image, we use VQGAN-FC (-EMA/-LC)'s encoder convert the image into token map. We then replace each token in the map with the M nearest entry from its codebook. The modified token map is fed into the decoder for reconstruction. Figure 5 shows the PSNR results subset of ImageNet, using images from 100 randomly selected categories, and visualizes the results of our VQGAN-LC, VQGAN-FC, and VQGAN-EMA when M is set to 1, 50, 100, and 1000.
C Visualizations
In Figures 6 present the class-conditional generation results at resolution of 256 256 for our
VQGAN- LC with GPT (2). LDM [3], DiT []. and SiT [5), respectively, using 256 (16 x 16) tokens on ImageNet. Additionally, Figure TO illustrates the results of VQGAN-LC with LDM using 1024 (32 x 32) tokens on ImageNet. Figure IT shows the unconditional generation results at a resolution of 256 x 256 for VQGAN-LC with LDM the FFHQ dataset, utilizing 256 (16 x 16) tokens. The introduction a large-scale codebook facilitates the generation of images with diverse poses, intricate textures, and complex backgrounds.
D Limitations and Broad Impacts
We present image quantization technique called VQGAN-LC, which expands the codebook size to 100,000 with a utilization rate of 99%. This approach has the potential to enhance any downstream applications that involve image quantization models. However, VQGAN-LC trained on the ImageNet and FFHQ datasets, which restricts downstream applications, like image generation, producing images from the limited categories found in these datasets. While training VQGAN-LC on larger datasets like LAION-5B may improve its utility in downstream applications, it would also be more costly and computationally demanding. Furthermore, please refrain from using this model to generate malicious or inappropriate content. It is intended solcly for positive and constructive purposes in research and creativity.
15

---

24-
22
Ours
EMGAN-EMA
PSNR
20
18
VQGAN-FC
16
14-
2
20
50
100
1000
Replacement with the M Nearest Entry
Image
M-1
M-50
M-100
M-1000
Raw Image
M-1
M-50
M-110
M-1000
VOGIN-FMA VQDANRC
VOGAN PC
VOGANAFE
VOGAN
VQUAN-EMA
VOCAN-EMA VQGANSE
VOGAN
Figure 5: For a given image, we employ an image quantization model (VQGAN-FC, VQGAN-EMA, our VQGAN-LC) transform into token map. Fach token in this map is then substituted with the th nearest entry from the codebook. This altered token map is subsequently fed into the decoder for reconstruction. (Top) PSNR for each configuration. (Bottom) Reconstruction visualizations for the three models.
16

---

Australian terrier (193)
Rapeseed (984)
Agaric (992)
Figure 6: Qualitative results of class- canss-conditiona generation using our VQGAN-LC with GPT 2 on ImageNet, utilizing 256 (16 x 16) tokens. We display the category name and corresponding category ID for each group.
Volcano (980)
Arctic Fox (279)
Pug (254)
Figure 7: Qualitative results of class-conee conditioni generation using our VQGAN-LC with LDM B on ImageNet, utilizing 256 (16 x 16) tokens and classifier- free guidance scale of 1.4. We display the category name and corresponding category ID for each group.
Lake Shore (975)
Geyser (974)
Space Shuttle (812)
Figure 8: Qualitative results of class-conditional generation using our VQGAN-LC with DiT 4 on ImageNet, utilizing 256 (16 x 16) tokens and classifier-free guidance scale of 8.0. We display the category name and corresponding category ID for each group.
17

---

Cliff (972)
Balloon (417)
Guinea Pig (338)
Figure 9: Qualitative results of class-conditional generation using our VQGAN-LC with SiT [S] on ImageNet, utilizing 256 (16 16) tokens and classifier-free guidance scale of 8.0. We display the category name and corresponding category ID for each group
Valley (979)
Red Panda (387)
Macaw (88)
Figure 10: Qualitative results class-conditional generation using our VQGAN-LC with LDM B on ImageNet, utilizing 1024 (32 32) tokens and classifier-free guidance scale of 1.4. We display the category name and corresponding category ID for each group.
Figure 11: Qualitative results unconditional generation using our VQGAN-LC with LDM on FFHQ, utilizing 256 (16 x 16) tokens.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper introduces novel approach named VQGAN-LC, which scales the codebook size VQGAN to 100,000, with a utilization rate of 99%.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification:
No theory assumptions and proofs are introduced in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide all the information including training and inference to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
The code will made publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ Swwelees/eoSSSSeeeeeeeeeee
wwt.Suubmisiioooiiioooooooio for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we specify the training and test details, hyperparameters, optimizers and other implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: we report rFID, LPIPS, PSNR and SSIM for image reconsturction, top-1 accuracy for image classification, and FID for image generation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We use Nvidia V100 GPUs across experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
we strictly comply with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
we discuss the societal impacts in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: this is discussed in the appendix.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: use public datasets like ImageNet and FFHQ.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: no new assets introduced.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing experiments and research with human subjects are included in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No such potential risks in this work.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.


Reviews:
[{"review_id": "Hboq1asf6O", "reviewer": "Reviewer_Kj4D", "summary": "The VQGAN-LC (Large Codebook) model tackles the challenges of expanding codebook size and utilization in image quantization. Unlike its predecessors, which struggled with limited codebook sizes and low utilization rates, VQGAN-LC increases the codebook size to 100,000 and achieves over 99% utilization. This model starts with features extracted by a pre-trained vision encoder and optimizes a projector to align these features with the encoder's distributions. VQGAN-LC demonstrates superior performance in tasks such as image reconstruction, classification, and generative image models compared to earlier methods like VQGAN-FC and VQGAN-EMA.", "questions": "1. What is the definition of active or inactive for the utilization of the codebook? How to mesure it?\n2. Can the tokenizer be integrated into large language models (GPT-like) to train a multimodal large language model?", "limitations": "The main limitation is also mentioned in the weakness.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper aims to address a very important topic, which may be generalized to other modalities, such as speech and video.\n2. The structure and presentation of this paper is straightforward and easy to understand.\n3. The experiments are comprehensive and persuasive, which shows obvious improvement over other SOTA methods.", "weaknesses": "1. My main concern is doubt about the scalability of this method. It appears that this paper does not provide enough evidence to show that it can still be useful for open-domain image reconstruction and generation under billion-level image datasets, since all experiments are carried out on small-scale datasets and low-resolution images (256x256).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Kj4D,\n\nThanks for your valuable comments.\n\n**Q1: Reconstruction and Generation on Billion-Level Datasets**\n\n- Firstly, we want to highlight that we adhere to the methodologies established in previous works such as VQGAN [1] and RQTransformer [2], conducting our experiments on the commonly used benchmarks ImageNet-1K and FFHQ for fair comparison. Training a standard VQGAN and our VQGAN-LC on ImageNet-1K, which consists of 1.28M images, requires 66 and 72 hours, respectively, using 32 V100 GPUs. Training a VQGAN or its variants on a significantly larger dataset like LAION-400M, which is nearly 300 times the size of ImageNet, could take up to 19,800 hours (825 days) with the same resources. This extensive training duration is prohibitively expensive, and we hope the reviewer understands the heavy training cost involved. Most previous studies validate their methods on ImageNet-1K. Only a few works can afford training on exceptionally large datasets. For example, Muse [3] trained a VQGAN-based generation model on LAION-400M, which took over a week using a 512-core TPU-v4.\n\n- However, to verify the scalability of our method within the limited rebuttal period, we train our VQGAN-LC on a combination of ImageNet-1K, which includes 1.28M images, and a subset (termed LAION-1M) containing an equivalent number of images sampled from LAION-400M. We report reconstruction performance using the rFID metric in the following table. Our approach demonstrates the benefits of data scalability: training on larger datasets significantly reduces the rFID scores across validation sets of various benchmarks. We will further verify large-scale training in our revision.\n\n| Dataset                   | Codebook Size | Utilization (\\%) | ImageNet (val) | LAION (val) | FFHQ (val) |\n|------|------|--------|---------|--------|--------|\n| ImageNet        | 100,000      | 99.4 | 2.62          | 5.73    | 7.29        |\n| ImageNet + LAION-1M | 100,000  | **99.6** |**2.36**      | **3.87**     | **6.86**      |\n\n**Q2: Experiments on High-Resolution Images**\n\nBelow, we present the reconstruction performance on ImageNet, which contains images with a resolution of $512 \\times 512$ pixels. Without altering the network structure, each image is quantized into a $32 \\times 32$ token map (i.e. 1024 tokens). It is worth noting that the optimal codebook size for the baseline models, VQGAN-FC and VQGAN-EMA, remains 16,384, whereas for our VQGAN-LC, it is 100,000. Our model consistently surpasses all baselines in high-resolution settings, as evidenced by various metrics such as rFID, LPIPS, PSNR, and SSIM. We will include this experiment in our revised version.\n\n| Method       | # Tokens | Codebook Size | Utilization (\\%) | rFID | LPIPS | PSNR | SSIM |\n|---------|-----|---------|--------|----|-------|------|------|\n| VQGAN-FC     | 1024     | 16,384        | 11.1             | 2.15 | 0.13  | 25.8 | 72.8 |\n| VQGAN-EMA    | 1024     | 16,384        | 85.3             | 1.76 | 0.12  | 26.6 | 74.4 |\n| VQGAN-LC (Ours) | 1024 | 100,000       | **99.7**         | **1.51** | **0.11** | **27.1** | **77.4** |\n\n**Q3: Defining and Measuring Active and Inactive Tokens**\n\nVQGAN, its variants, and our VQGAN-LC use a codebook for image quantization. These models convert each image into a token map, with each token corresponding to a codebook entry. After training, we quantize all images from the training set into token maps. Codebook entries that are never used, often due to suboptimal training strategies, are designated as inactive tokens (codes). In contrast, codebook entries used at least once to represent an image in the training set are classified as active tokens (codes).\n\nThe codebook utilization rate is calculated as the ratio of active entries (tokens/codes) to the total size of the codebook. Figure 3 illustrates the codebook utilization rate across training epochs and the average utilization frequency. Figure 4 visualizes active and inactive tokens using t-SNE.\n\n**Q4: Integrating this Work into GPT-like LLMs for Training Multimodal LLMs**\n\nThis is an excellent suggestion! GPT-like autoregressive models can predict subsequent tokens using a causal Transformer decoder. By integrating an image quantization model, such as our VQGAN-LC, we can enable large multimodal models to generate both language and image tokens. These generated image tokens are then processed by the image quantization model's decoder to produce realistic images.\n\nOn the one hand, some existing works have already explored this direction by utilizing an image quantization model to develop multimodal LLMs. For instance, VideoPoet [4] employs MAGVIT-v2 [5] as the quantization model to generate content in both language and visual modalities. However, training VideoPoet requires extensive resources, and due to limited rebuttal time and computational resources, we were unable to conduct such experiments.\n\nOn the other hand, our work, like prior research on image quantization, aims to develop an improved quantization model. Image understanding and generation are central to multimodal LLMs. We compare our VQGAN-LC with several models in both understanding and generation tasks. In Figure 1.b, we demonstrate the image classification for the understanding task. In Tables 4 and 5, we showcase the image generation capability by applying our VQGAN-LC to various generation models, including autoregressive image generation (GPT), and diffusion- and flow-based generative models (LDM, DiT, and SiT).\n\nWe will include these discussions in our revision.\n\n**Reference**\n\n[1] Taming transformers for high-resolution image synthesis, CVPR 2021.\n\n[2] Autoregressive image generation using residual quantization, CVPR 2022.\n\n[3] Muse: Text-to-image generation via masked generative transformers, ICML 2023.\n\n[4] VideoPoet: A Large Language Model for Zero-Shot Video Generation, ICML 2024.\n\n[5] Language Model Beats Diffusion-Tokenizer is Key to Visual Generation, ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5iQ2FTiRQ1", "reviewer": "Reviewer_c7tj", "summary": "The paper introduces VQGAN-LC (Large Codebook), a novel image quantization model that significantly extends the codebook size and enhances codebook utilization. Traditional models like VQGAN-FC are limited in codebook size and utilization rates, with a maximum size of 16,384 and utilization rates typically below 12%. In contrast, VQGAN-LC expands the codebook size to 100,000 and achieves utilization rates exceeding 99%. Instead of optimizing each codebook entry individually, the model trains a projector that aligns the entire codebook with the encoder's feature distributions. This approach allows for a significant increase in codebook size and utilization without a corresponding increase in computational cost.", "questions": "How does  Utilization Rate effect the performance? There seems to be a strong correlation for large codebook but for codebook sized 16,384 with “83% and 68%” Utilization achieve similar results compared to 99%Utilization.  In some cases lower Utilization has slightly better performance as well?\n\nhow does increasing codebook effect the inference time and memory footprint of the network. The paper claims in section 4.3 line#241 that large codebook incurs almost no additional computational cost. This is not obvious. How is computational cost independent of the codebook size?\n\nCan you explain more about the perceptual loss used in eq.(1)\n\nDoes high utilization rate is any relationship with compressibility of the model? Intuitively lower utilization rates suggest that the model can be further compressed/ optimized? Will high utilization rates mean that it decreases the compressibility of the model?", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The most important contribution of this work is a trainable projector that maps the codebook entries to a latent space. The approach ensures nearly full codebook utilization throughout the training process, allowing for the codebook size to expand to over 100,000 entries while maintaining an impressive utilization rate of 99%.\n\nThe proposed method can be easily incorporated into existing VQGAN architecture. \n\nThe ability to scale the codebook size to up to 100,000 entries without incurring significant additional computational cost is critical.", "weaknesses": "In line 211 of page 6 ​​authors mention that the optimal codebook size of the proposed method is 100,000 which is intuitively correct. However, authors should also provide a comparison of smaller codebook sizes with other methods. In Table 3, Table 4, and Table 5  there is no comparison of VQGAN-LC with other methods utilizing a similar codebook size. \n\nQuantization error constitutes a principal component of the loss function equation; however, in the two variants, Factorized Codes (FC) and Exponential Moving Average (EMA), distinct formulations of this loss are observed. The rationale provided may be comprehensible to those familiar with the VQGAN architecture; nonetheless, the explanations offered in the document lack sufficient clarity for those less acquainted with this framework.\n\nMisc:\nMention the batch size for the experiments in the experimental setup section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer c7tj,\n\nThanks for your valuable comments.\n\n**Q1: Comparison of Smaller Codebook Sizes with Other Methods**\n\nTable 1 in the main paper compares our VQGAN-LC with baseline models VQGAN-FC and VQGAN-EMA across various codebook sizes (**1,024**, **16,384**, **50K**, and **100K**). The evaluation encompasses both reconstruction and generation using the latent diffusion model (LDM) on the ImageNet dataset. The results show that VQGAN-FC and VQGAN-EMA perform optimally at a codebook size of 16,384. In contrast, our VQGAN-LC supports codebook sizes up to 100,000 while maintaining an impressive utilization rate of 99\\%.\n\nBased on your suggestions, we conducted additional experiments with our VQGAN-LC using a smaller codebook size of 16,384. All other configurations remained the same as in the main paper (e.g., representing an image with 256 tokens and an input image resolution of $256 \\times 256$). We compared our model with baseline models and report: 1) reconstruction performance on ImageNet and FFHQ, using metrics such as rFID, LPIPS, PSNR, and SSIM (**the first table in global text response**); 2) image generation results on ImageNet, using various generation models including GPT, LDM, SiT, and DiT, with FID as the evaluation metric (**the second table in global text response**). Despite utilizing a smaller codebook of size 16,384, our VQGAN-LC surpasses the baseline models in both reconstruction and generation tasks. This experiment will be incorporated into our revised version.\n\n**The two tables can be found in the global text response.**\n\n**Q2: More Explanations for VQGAN-FC and VQGAN-EMA**\n\nThank you for the suggestions. Due to the rebuttal's length constraints, we regret that we cannot provide further details at this time. However, we will include more comprehensive explanations of the VQGAN-FC and VQGAN-EMA formulations in our revised version.\n\n**Q3: Batch Sizes**\n\nThe specifics of the batch sizes are provided in Appendix A of the main paper. Specifically, the batch size for training our VQGAN-LC is 256. For the training of GPT, LDM, DiT, and SiT, the batch sizes are 1024, 448, 256, and 256, respectively.\n\n**Q4: Impact of Utilization Rate on Performance**\n\nThe performance of an image quantization model is determined by the number of active codes, which is the product of the utilization rate and the codebook size. Active codes are the ones that participate in converting images into token maps, thus defining the discrete representation space. For example, a model with a codebook size of 1024 and a utilization rate of 99\\% has $1024 \\times 0.99 = 1014$ active codes. In contrast, a model with a codebook size of 16,384 and a utilization rate of 83\\% has $16,384 \\times 0.83 = 13,599$ active codes. Although the first model has a higher utilization rate, its performance may be inferior to the latter model due to the smaller number of active codes. The superior performance of our VQGAN-LC model can be attributed to its ability to expand the codebook size up to 100,000 while maintaining a high utilization rate of 99\\%. This high efficiency ensures that almost no codes are wasted, leading to improved training efficiency.\n\n**Q5: Inference and Memory Costs**\n\nAll VQGAN models, including our VQGAN-LC and its variants, follow an encoder-quantizer-decoder architecture. In this work, we utilize the same encoder and decoder across all models, including ours. Let $F \\in \\mathcal{R}^{16 \\times 16 \\times C}$ represent the feature generated by the encoder, where $C$ is the feature dimension. Let $B \\in \\mathcal{R}^{N \\times C}$ represent the codebook with size $N$. The quantizer performs a matrix multiplication between $F$ and $B$ to transform $F$ into a token map, where each element corresponds to an entry in the codebook. The primary inference cost is attributed to the encoder and decoder. Increasing the codebook size $N$ incurs minimal additional cost since the matrix multiplication between $F$ and $B$ is negligible compared to the encoder and decoder processing time. Below, we present the multiply-accumulates (MACs) and model sizes for our VQGAN-LC models with codebook sizes of 16,384 and 100,000, respectively, when inferring an image of size $256 \\times 256$.\n\n| Codebook Size | MACs   | Model Size |\n|---------------|--------|------------|\n| 16,384        | 195.08G| 71.71M     |\n| 100,000       | 195.70G| 71.72M     |\n\n**Q6: Perceptual Loss**\n\nPerceptual loss is widely used in image generation. Unlike the traditional reconstruction loss, which calculates the mean squared error (MSE) directly between the raw and reconstructed images in the pixel space, perceptual loss computes the MSE between the feature representations of the raw and reconstructed images. These feature representations are extracted using a pre-trained VGG network, and the loss is applied in the feature space rather than the pixel space. We will provide further details about the perceptual loss in our revision.\n\n**Q7: Relationship between Utilization Rate and Compressibility**\n\nIn image quantization models designed for generative purposes, the primary aim is to convert images into discrete tokens with minimal information loss, enabling the generation of diverse and realistic images. As discussed in **Q5**, the VQGAN family primarily focuses on the number of active codes in the codebook. Typically, an image is represented by a fixed set of tokens across all VQGAN models, including ours, resulting in a constant level of compressibility. However, the number of active codes determines the span of the discrete space. A higher number of active codes indicates a more powerful representation capability and finer-grained representations within the image quantization model. This, in turn, can lead to the generation of more diverse and realistic images for downstream generative models such as GPT, LDM, SiT, and DiT. We will incorporate these discussions into our revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HrS4knjXz6", "reviewer": "Reviewer_igUX", "summary": "This study introduces VQGAN-LC (Large Codebook), an innovative image quantization model that significantly extends the codebook size to 100,000, achieving a utilization rate of 99%. Unlike previous methods that optimize each codebook entry individually, VQGAN-LC initializes its codebook with 100,000 feature centers from a pre-trained vision encoder. These codebook vectors are then frozen for the remainder of the process. The optimization focuses on training a projector to align the frozen codebook with the feature distributions of the encoder in VQGAN-LC. This approach ensures that nearly all token embeddings remain active throughout the training phase.", "questions": "(1)\tWhen the codebook size increases significantly, I think that discretization might lose its meaning as the intuition behind discrete representation learning can be thought of as learning to represent the data space with a latent space as small as possible for a lower computational load. I’m curious about your justification for why we should increase the number of representations in the codebook instead of decreasing them, apart from better reconstruction performance.\n\n(2)\tOne of my questions pertains to the span of the codebooks. Can you conduct a study or experiment to illustrate the coverage and distribution of the codebook entries? For instance, creating a t-SNE map by normalizing all codebooks and visualizing them in the same space could provide valuable insights into the codebook feature space learned by your model. This visualization would help us understand the diversity and quality of the representations within the codebook and how effectively it spans the feature space.", "limitations": "The authors address limitations and societal impact in Section D in Supplementary Material.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1)\tExtensive number of experiments. Datasets and metrics are diverse. \n\n(2)\tExperiments are suitable for demonstrating the effectiveness of the proposed method.\n\n(3)\tSolid improvement in benchmarks.", "weaknesses": "(1)\tThe paper primarily focuses on scaling the codebook size and improving utilization rates in VQGAN models. While this is a noteworthy improvement, it can be seen as an incremental advance rather than a novel approach. The method relies heavily on existing architectures and concepts, particularly those established by prior works such as VQGAN, VQGAN-EMA, and VQGAN-FC.\n\n(2)\tThe related work section fails to comprehensively cover the breadth of existing research focused on improving codebook usage and addressing the codebook collapse problem. Numerous significant contributions in this area are overlooked, which diminishes the depth and rigor of the literature review. This lack of thoroughness may lead to an incomplete understanding of the current state of the field and the novelty of the proposed approach. You may cite some recent papers for the codebook utilization problem, such as [1], [2], or other top papers that are found on arxiv by searching keywords: discrete VAEs codebook collapse.\n\n(3)\tThere is no hypothesis to explain why the proposed method works. The paper does not address how the method increases the utilization rate and why this increase leads to improved results. This lack of theoretical foundation makes it difficult to understand the underlying mechanisms driving the observed performance gains.\n\n(4)\tThe paper lacks experiments and explanations regarding the quality of the codebooks and how well they span the representational space. While the method claims high utility even on ResNet50, it is crucial to study the coverage and distribution of the codebook entries. Experiments should be conducted to demonstrate how the codebook spans the feature space and ensure that the high utilization rate translates to meaningful and diverse representations. Without such analysis, it is difficult to ascertain the true effectiveness and robustness of the proposed method.\n\n(5)\tIn Preliminary D, a fixed step size (M=1, 50, 100, 1000) is used for the Mth nearest replacement. However, the ratio of the step size to the codebook size is not consistent. For example, if the codebook size is 16,000 in the baseline methods and a step size of M=1000 is used, it would be more meaningful to compare with a distance of 100,000/16 when the codebook size is 100,000. This is because, relatively, in VQGAN-FC, the replacement is made with a vector that is 1 step away in the codebook, whereas in the baseline methods, the replacement is made with a vector that is 6 steps farther away. \n\n[1] Huh, M., Cheung, B., Agrawal, P., & Isola, P. (2023). Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks. International Conference on Machine Learning.\n\n[2] Takida, Y., Ikemiya, Y., Shibuya, T., Shimada, K., Choi, W., Lai, C.-H., … Mitsufuji, Y. (2024). HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes. Transactions on Machine Learning Research", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer igUX,\n\nThanks for your valuable comments.\n\n**Q1: Dependence on Established Architectures**\n\nVQGAN and VQVAE are foundational works that introduced the encoder-quantizer-decoder framework for image quantization. Subsequent works, such as RQ-VAE, SQ-VAE, Reg-VQ, ViT-VQGAN, and the two references [1,2] you mentioned, adhere to this encoder-quantizer-decoder architecture without significant alterations. These works primarily focus on learning a discrete representation space that closely approximates the pixel space while minimizing information loss. To this end, they implement various improvements, such as: 1) increasing representation capability through multiple tokens for each image patch or replacing the traditional CNN backbone with a ViT backbone; 2) enhancing codebook utilization via stochastic quantization or reinitializing inactive codes during training.\n\nThis paper builds upon the encoder-quantizer-decoder architecture established by VQGAN and VQVAE, similar to most previous works. However, existing research has not investigated the potential of learning an extremely large codebook while maintaining high codebook utilization. We undertake pioneering efforts to explore this possibility. Our experiments demonstrate that our VQGAN-LC can achieve this, establishing a novel approach for learning a robust discrete representation space that closely approximates the pixel space with minimal information loss.\n\nPlease see **Q6** in the **global text response** for the significance of increasing the codebook size.\n\n**Q2: Discussion of Recent Papers**\n\nThank you for highlighting these two excellent papers [1,2]. After a thorough review, we have identified several commonalities between the works we have discussed (CVQ-VAE and RegVQ) and the papers [1,2] you mentioned: 1) Both CVQ-VAE and [1] adopts the concept of online clustering to enhance codebook utilization; 2) Both RegVQ and [2] use stochastic quantization to prevent codebook collapse. We will incorporate a detailed discussion of the works [1-2] you mentioned in our revised version.\n\n**Q3: Explanation of the Motivations and Underlying Mechanisms**\n\nIn L52-61, we examine why prior VQGAN variants struggle to achieve a high codebook utilization rate when the codebook size is significantly increased and the drawbacks of random codebook initialization. This setup leads to only a small portion of codes being optimized in each iteration. This observation inspired us to develop VQGAN-LC, which, instead of optimizing a small set of codes per iteration, initializes the codebook with a pre-trained visual encoder and optimizes the entire codebook distribution directly, as detailed in L62-72. In L146-152 and Figure 3, we analyze the codebook utilization rate throughout the training epochs for VQGAN-FC, VQGAN-EMA, and our VQGAN-LC, and the models' average utilization frequency across all epochs. Figure 4 presents t-SNE maps of the codebook used in our model and the baseline models. This analysis demonstrates that, during the training phase of image quantization models, prior works like VQGAN-FC and VQGAN-EMA have a significant number of codebook entries that do not receive any supervision signals, resulting in suboptimal representation capabilities. \n\nFor further discussion, please see **Q6** in the **global text response**.\n\n**Q4: Codebook Coverage of Representational Space and T-SNE Visualization**\n\nFigure 4 in the main paper (Appendix B) showcases the active and inactive codes from the codebook for three models (VQGAN-FC, VQGAN-EMA, and our VQGAN-LC) at different codebook sizes (1,024, 16,384, 50,000, and 100,000), using t-SNE maps. Active codes are those that contribute to converting images into token maps, thereby defining the discrete representational space. Compared to the baseline models (VQGAN-FC and VQGAN-EMA), our method successfully expands the codebook size to 100,000 while maintaining a 99\\% utilization rate, resulting in a significantly large number of active codes.\n\nAdditionally, in **Figure 1 of the global rebuttal PDF file**, we provide t-SNE visualizations of the active codes from the codebook for the three models (VQGAN-FC, VQGAN-EMA, and our VQGAN-LC) as well as a combined t-SNE visualization for these models in the same space.\n\n**Q5: The M-th Nearest Replacement**\n\nWe conjecture that you may refer to Figure 4 in the main paper (Appendix B) rather than \"Preliminary D\". To address your concerns, we performed the M-th nearest replacement on three models: two baselines, VQGAN-FC and VQGAN-EMA, and our model, VQGAN-LC. Each model uses a codebook of the same size, 100,000, to ensure a fair comparison. Please see **Figure 2 in the global rebuttal PDF file** for the results. This experiment illustrates that learning a large codebook with an extremely high code utilization rate enhances the representation capability and provides finer-grained representations within the image quantization model.\n\n**Q6: Why Increase the Number of Representations in the Codebook**\n\nThe comprehensive answer to this question can be found in the **global text response**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their constructive comments. 

In this global response, we provide the performance tables for **Reviewer c7tj**, and address some comments from **Reviewer igUX**. 

**Reviewer-c7tj-Q1: Comparison of Smaller Codebook Sizes with Other Methods**

Reconstruction performance on FFHQ and ImageNet:

| Method             | # Tokens | Codebook Size | Utilization (\%) | rFID | LPIPS | PSNR | SSIM |
|-|-|-|-|-|-|-|-|
| **FFHQ**   |          |               |                  |      |       |      |      |
| VQGAN              | 256      | 16,384        | 2.3              | 5.25 | 0.12  | 24.4 | 63.3 |
| VQGAN-FC           | 256      | 16,384        | 10.9             | 4.86 | 0.11  | 24.8 | 64.6 |
| VQGAN-EMA          | 256      | 16,384        | 68.2             | 4.79 | 0.10  | 25.4 | 66.1 |
| VQGAN-LC (Ours)    | 256      | 16,384        | 99.9         | 4.22 | 0.09  | 25.7 | 68.0 |
| VQGAN-LC (Ours)    | 256      | 100,000       | 99.5             | **3.81** | **0.08** | **26.1** | **69.4** |
| **ImageNet** |        |               |                  |      |       |      |      |
| VQGAN              | 256      | 16,384        | 3.4              | 5.96 | 0.17  | 23.3 | 52.4 |
| VQGAN-FC           | 256      | 16,384        | 11.2             | 4.29 | 0.17  | 22.8 | 54.5 |
| VQGAN-EMA          | 256      | 16,384        | 83.2             | 3.41 | 0.14  | 23.5 | 56.6 |
| VQGAN-LC (Ours)    | 256      | 16,384        | 99.9         | 3.01 | 0.13  | 23.2 | 56.4 |
| VQGAN-LC (Ours)    | 256      | 100,000       | 99.9         | **2.62** | **0.12** | **23.8** | **58.9** |

Generation results on ImageNet:

| Method             | # Tokens | Codebook Size  | GPT | SiT | DiT |  LDM |
|-|-|-|-|-|-|-|
| VQGAN-FC           | 256      | 16,384         | 17.3   | 10.3   | 13.7   | 9.78   |
| VQGAN-EMA          | 256      | 16,384         | 16.3   | 9.31   | 13.4   | 9.13   |
| VQGAN-LC (Ours)    | 256      | 16,384         | 16.1   | 9.06   | 11.2  | 8.84   |
| VQGAN-LC (Ours)    | 256      | 100,000        | **15.4**   | **8.40**   | **10.8**   | **8.36**   |

**Reviewer-igUX-Q6: Why Increase the Number of Representations in the Codebook**

In light of GPT's notable success in auto-regressive language generation, image quantization models, such as VQGAN, have been developed to facilitate GPT-style image generation. Given the inherently continuous nature of image signals, directly applying GPT to image generation presents substantial challenges. To address this, image quantization techniques have been introduced, effectively transforming images into token maps. This conversion enables the training of GPT models to generate images by predicting flattened token maps in an auto-regressive manner. This approach not only leverages the strengths of GPT in handling sequential data but also facilitates the efficient and coherent synthesis of high-quality images, bridging the gap between language and visual data processing.

The primary objective of GPT-style image generation using image quantization models is to produce high-quality, diverse, and realistic images, rather than minimizing the size of the codebook used to represent images. In our experiments, we discovered that training our VQGAN-LC with a codebook size of 100,000 on ImageNet incurs only a 1\% additional cost compared to training a model with a codebook size of 16,384. The extra inference cost for converting an image into a token map is negligible, adding just less than 1\% to the total cost. Training a GPT model (480M) using a codebook size of 100,000 incurs only 1.3\% cost than using one with a codebook size of 16,384. Below, we present the multiply-accumulates (MACs) and model sizes for our VQGAN-LC models with codebook sizes of 16,384 and 100,000, respectively, when inferring an image of size $256 \times 256$.

| Codebook Size | MACs   | Model Size |
|-|-|-|
| 16,384        | 195.08G| 71.71M     |
| 100,000       | 195.70G| 71.72M     |

Previous baseline models, such as VQGAN-FC and VQGAN-EMA, have shown that increasing the codebook size significantly improves performance. For example, **the original VQGAN demonstrated that expanding the codebook size from 1,024 to 16,384 reduced the reconstruction FID from 7.94 to 4.98.** However, these VQGAN variants have not explored the utilization of a codebook larger than 16,384. As illustrated in Figure 1 of the main paper, our re-implementation of two VQGAN baselines indicates that these models do not benefit from a codebook larger than 16,384 using their codebook optimization strategies.

In this work, we successfully expand the codebook size to 100,000 while maintaining a codebook utilization rate of over 99\%. In Figure 1 and Tables 2-5, our VQGAN-LC and all baseline models utilize the same encoder-decoder network and generation models (such as GPT, LDM, SiT, and DiT), differing only in codebook size and optimization strategy. The optimal codebook size for VQGAN-FC and VQGAN-EMA is 16,384, while for our VQGAN-LC, it is 100,000. Our model consistently outperforms the baseline models in image classification, image reconstruction, and image generation using various generative models, highlighting the potential of a larger, well-optimized codebook. This is because a larger codebook allows for a more detailed representation of the input data, capturing subtle variations and intricate details that smaller codebooks may miss.

The improvement in GPT generation performance with an increased codebook size is not unique to image generation. Studies on LLMs suggest that employing a tokenizer with an expanded vocabulary significantly enhances model efficacy. For example, the technical report (https://ai.meta.com/blog/meta-llama-3/) for LLAMA 3 shows, "LLAMA 3 uses a tokenizer with a vocabulary of 128K tokens that encodes language much more efficiently, which leads to substantially improved model performance."

We will include these discussions in our revision.

Meta-review: All reviewers suggested acceptance, altough without much enthusiasm. The paper's contribution is more of an engineering effort, but the practical results are good.
Decision: Accept (poster)