Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99%

Authors: lei  zhu
fangyun  wei
yanye  lu
dong  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 12:19:42

Abstract: In the realm of image quantization exemplified by VQGAN, the process encodes images into discrete tokens drawn from a codebook with a predefined size. Recent advancements, particularly with LLAMA 3, reveal that enlarging the codebook significantly enhances model performance. However, VQGAN and its derivatives, such as VQGAN-FC (Factorized Codes) and VQGAN-EMA, continue to grapple with challenges related to expanding the codebook size and enhancing codebook utilization. For instance, VQGAN-FC is restricted to learning a codebook with a maximum size of 16,384, maintaining a typically low utilization rate of less than 12% on ImageNet. In this work, we propose a novel image quantization model named VQGAN-LC (Large Codebook), which extends the codebook size to 100,000, achieving an utilization rate exceeding 99%. Unlike previous methods that optimize each codebook entry, our approach begins with a codebook initialized with 100,000 features extracted by a pre-trained vision encoder. Optimization then focuses on training a projector that aligns the entire codebook with the feature distributions of the encoder in VQGAN-LC. We demonstrate the superior performance of our model over its counterparts across a variety of tasks, including image reconstruction, image classification, auto-regressive image generation using GPT, and image creation with diffusion- and flow-based generative models.

Main paper: 
 introduction: 
 Image quantization [1, 6, 7] refers to the process of encoding an image into a set of discrete representations, also known as image tokens, each derived from a codebook of a pre-defined size. VQGAN [1] stands out as a prominent architecture, with an encoder-quantizer-decoder structure, playing a pivotal role in various applications, including: (1) training a GPT [2, 8, 9, 10, 11] on image tokens to create images; (2) serving as an autoencoder in latent diffusion models (LDMs) [3, 4] and generative models [12, 13, 14] , with flow matching [5, 15] ; and (3) functioning within large multi-modality models [16, 17, 18, 19] , where its encoder processes input images and its decoder assists in image generation.In contrast to natural languages, which typically maintain a static vocabulary, image quantization models necessitate a codebook of a pre-defined size to convert images into discrete image tokens. The nature of image signals-complex and continuous-makes translating images into token maps a form of lossy compression that is generally more severe than converting them into continuous feature maps. The capability of these models to represent images largely depends on the codebook size. Previous studies, such as VQGAN [1] , its improved versions, including VQGAN with exponential moving average (EMA) update (VQGAN-EMA) and VQGAN using factorized codes (VQGAN-FC), and its predecessors, like VQVAE [6] and VQVAE-2 [7] , have demonstrated that they can only learn a codebook with a maximum size of 16,384. These models often face unstable training or performance saturation issues when the codebook size is further increased, as shown in Table 1 . Additionally, they typically exhibit a low codebook utilization rate-for instance, under 12% in VQGAN-FC, as shown in Figure 1 (a)-indicating that a significant portion of the codebook remains unused, thereby diminishing the model's representational capacity. Furthermore, studies on large language models suggest that employing a tokenizer with an expanded vocabulary significantly enhances model efficacy. For example, the technical report for LLAMA 3 2 shows, "LLAMA 3 uses a tokenizer with a vocabulary of 128K tokens that encodes language much more efficiently, which leads to substantially improved model performance."In this study, we investigate the scalability of codebook size in VQGAN and the improvement of its codebook utilization rate, thereby substantially enhancing the representational capabilities of VQGAN. Typically, as shown in Figure 2 (a), image quantization models like VQGAN are structured with an encoder-quantizer-decoder architecture, where the quantizer is connected to a codebook. For a given image, the encoder produces a feature map that the quantizer then converts into a token map. Each token in this map corresponds to an entry in the codebook, based on their cosine similarity. This token map is subsequently used by the decoder to reconstruct the original image. are excluded from the training process and subsequently remain unused during the inference phase, resulting in poor codebook utilization.Our approach deviates from conventional image quantization models by initiating with a codebook composed of N frozen features, sourced from a pretrained image backbone like the CLIP-visionencoder [23] , and utilizing datasets like ImageNet [24] . A projector is then employed to transition the entire codebook into a latent space, producing token embeddings. During the training process, it is the projector that is optimized, not the codebook itself, which distinguishes our method from traditional models. By optimizing the projector, we adapt the aggregate distribution of the codebook entries to align with the feature maps generated by the encoder. This contrasts with methods like VQGAN [1] , where adaptations are made to a limited number of codebook entries to match the feature map distributions during each iteration. Our simple quantization technique ensures that almost all token embeddings (over 99%) remain active throughout the training phase. The process is depicted in Figure 2(d) .Our newly developed quantizer can be integrated directly into the existing VQGAN architecture, replacing its standard quantizer without requiring any changes to the encoder and decoder. This innovative quantizer enables the expansion of the codebook to sizes up to 100,000, while maintaining an impressive utilization rate of 99%. By comparison, the conventional VQGAN is limited to a codebook size of 16,384 with a utilization rate of only 11.2% when applied to ImageNet. The advantages of a larger codebook with enhanced utilization are demonstrated across various applications, including image reconstruction, image classification, auto-regressive image generation using GPT, and image creation with diffusion models and flow matching. Figure 1 illustrates the performance of our improved VQGAN, termed VQGAN-LC (Large Codebook), compared to its counterparts. 
related work: 
 Image Quantization. Image quantization focuses on compressing an image into discrete tokens derived from a codebook [6, 1, 22, 20, 25, 26, 27] . VQVAE [6] introduces a method of quantizing patch-level features using the nearest codebook entry, with the codebook learned jointly with the encoder-decoder structure through reconstruction loss. VQVAE2 [7] enhances this by employing exponential moving average updates and a multi-scale hierarchical structure to improve quantization performance. VQGAN [1] further refines VQVAE by integrating adversarial and perceptual losses, enabling more accurate and detailed representations. ViT-VQGAN [21] replaces the CNN-based encoder-decoder [28] with a vision transformer and shows that using a factorized code mechanism with l 2 regularization can improve codebook utilization. Reg-VQ [29] introduces prior distribution regularization to prevent collapse and low codebook utilization. Additionally, some approaches use codebook reset strategies to reset unused codebook entries during training [26, 30, 31, 32] or utilizing stochastic quantization to enhance utilization rates [26, 29] . In contrast to these methods, the proposed VQGAN-LC initializes codebook entries using a pre-trained vision encoder on target datasets, ensuring nearly full codebook utilization throughout the training process and allowing for scaling up the codebook size to more than 100K.Tokenized Image Synthesis. Substantial advancements have been achieved in the realm of image synthesis, particularly through image quantization techniques. Initial efforts such as PixelRNN [33] employs LSTM networks [34] to autoregressively model dependencies between quantized pixels. Building upon this, the groundbreaking VQVAE [6] introduces the quantization of image patches into discrete tokens, significantly enhancing generation capabilities when paired with PixelCNN [35] . The iGPT [36] further advances the field by leveraging the powerful Transformer [37] for sequence modeling of VQ-VAE tokens. Recently, there has been a shift towards using non-autoregressive Transformers for image synthesis [12, 13, 14, 11] , which provide efficiency improvements over traditional raster-scan-based generation methods. Innovative approaches such as discrete diffusion models, including D3PMs [38] and VQ-Diffusion [39], utilize discrete diffusion processes to model the distribution of image tokens. Additional diffusion-based techniques [3, 15, 4, 5, 40, 41] compress images into latent representations using quantizers, thereby reducing both training and inference costs. Moreover, image quantizers can enhance large language models for both image synthesis and understanding [17, 16, 18, 42, 19] . Our work introduces a superior image quantizer, further refining the image synthesis process. 3 Method 
preliminary: 
 VQGAN. Let B = {b n ∈ R D } N n=1 denote a codebook containing N entries, with each entry b i being a D-dimensional trainable embedding with random initialization. As shown in Figure 2 (a), VQ-GAN adopts an encoder-quantizer-decoder structure. In this setup, an encoder processes an image X of height H and width W to generate a feature map Z ∈ R h×w×D , with (h, w) representing the latent dimensions. Subsequently, the quantizer maps Z to a token map Ẑ, where each token in Ẑ is an entry in B based on the cosine distance between Z and B. Finally, the decoder reconstructs the original image from the token map Ẑ. The entire network is optimized using a combination of losses, expressed as follows:EQUATIONwhere sg(•) denotes the stop-gradient operation. The terms L R , L Q , L P and L GAN represent the reconstruction loss, quantization loss, VGG-based perceptual loss [1] , and GAN loss [1] , respectively. Hyper-parameters α and β are set to 1.0 and 0.33 by default. As shown in Figure 2 (b), we refer to the codebook optimization strategy used in the original VQGAN as "gradient descent".VQGAN-FC. VQGAN faces significant challenges with inefficient codebook utilization. To address this issue, the factorized code (FC) mechanism, initially proposed by ViT-VQGAN [21] , is employed. We refer to VQGAN integrated with the FC mechanism as VQGAN-FC. The key differences between VQGAN and VQGAN-FC are two-fold: 1) a linear layer is added to project the encoder featureZ ∈ R h×w×D into a low-dimensional feature Z ′ ∈ R h×w×D ′, where D ′ ≪ D; 2) the codebook B, consisting of N D ′ -dimensional trainable embeddings, is randomly initialized. Consequently, the quantization loss in Eq. 1 is reformulated as:EQUATIONHowever, as illustrated in Figure 1 (a), the utilization rate of VQGAN-FC is only 11.2% on ImageNet when the codebook size is configured to 16,384, and increasing the size of the codebook fails to enhance performance as demonstrated in Table 1 .VQGAN-EMA. As depicted in Figure 2 (c), this variant of VQGAN adopts an exponential moving average (EMA) strategy to optimize the codebook. Specifically, Let B ⊂ B denote the set of token embeddings used for all token maps in the current training batch. The set B is updated through the EMA mechanism using the corresponding encoder features Z in each iteration. As a result, the codebook does not receive any gradients. Therefore, the quantization loss in Eq. 1 is defined as:EQUATIONOur results, highlighted in Figure 1 , indicate that VQGAN-EMA outperforms VQGAN-FC on various downstream tasks, leading to enhanced utilization of the codebook. However, expanding the codebook size continues to pose a significant challenge for VQGAN-EMA, as detailed in Table 1 .Analysis of VQGAN-FC and VQGAN-EMA. In these enhanced versions of VQGAN, the codebook is initialized randomly. During each iteration, only a small subset of entries related to the current training batch are optimized. As a result, the frequently optimized entries become more aligned with the feature map distributions generated by the encoder, while the less frequently optimized entries remain underutilized. Consequently, a significant portion of the codebook remains unused during both the training and inference stages. Figure 3 shows the codebook utilization rate over the training epoch and visualizes the utilization frequency of each codebook entry once training is completed.Overview. We present VQGAN-LC (Large Codebook), which allows for the expansion of the codebook to sizes of up to 100,000 while achieving a remarkable utilization rate of 99%. As illustrated in Figure 2 (d), our method diverges from VQGAN-FC and VQGAN-EMA in its design of the quantizer. We maintain a static codebook and train a projector to map the entire codebook into a latent space, aligning the distributions of the feature maps generated by the encoder. This approach allows us to scale the codebook size effectively without modifying the encoder and decoder, achieving an extremely high utilization rate and resulting in superior performance across various tasks, as shown in Figure 1 , Table 1 and Figure 3 . It is important to note that increasing the codebook size incurs almost no additional computational cost.Codebook Initialization. To initialize a static codebook, we first utilize a pre-trained vision encoder (e.g., CLIP with a ViT backbone) to extract patch-level features from the target dataset (e.g., ImageNet) containing M images. This extraction results in a set of features denoted asF = {F (i,j) m ∈ R D } h, w,M i=1,j=1,m=1, where F (i,j) m represents a D-dimensional patch-level feature at location (i, j) in the m-th image, and ( h, w) indicate the spatial dimensions of F . Subsequently, we apply K-means clustering to F, resulting in N cluster centers (with a default value of N = 100, 000). These cluster centers form the setC = {c n ∈ R D } N n=1, where c n is the n-th center. Our codebook B is then initialized using C.Quantization. Unlike VQGAN, VQGAN-FC and VQGAN-EMA, which optimize the codebook directly, our approach involves training a projector P (•), implemented as a simple linear layer, to align the static codebook B with the feature distributions generated by the encoderE(•) of our VQGAN-LC. Let B ′ = P (B) = {b ′ n ∈ R D ′ } N n=1denote the projected codebook. For a given input image X, the quantizer transforms the feature map Z = E(X) ∈ R h×w×D ′ into a token map Ẑ. This quantization process can be expressed asẐ := argmin b ′ n ∈B ′ ||Z (i,j) -b ′ n ||.Loss Function. We employ the same loss function as specified in Eq.1. However, the key distinction is that our codebook B remains frozen, while the newly introduced projector P (•) undergoes optimization.We evaluate the performance of VQGAN-FC, VQGAN-EMA, and our proposed VQGAN-LC across image reconstruction, image classification and image generation tasks.Image Reconstruction. Images are processed through the encoder, quantizer, and decoder to produce reconstructed images. These reconstructed images are then compared to their original images using the rFID metric as the evaluation criterion.Image Classification. Initially, the encoder and quantizer convert each image into a token map. Subsequently, we utilize a ViT-B model [37] , pre-trained with MAE [43] , to train on all token maps for the purpose of image classification. Top-1 accuracy is used as the evaluation metric.Image Generation. Image quantization models can be integrated with different image generation frameworks, such as auto-regressive causal Transformers (GPT [2] ), latent diffusion models (LDM [3] ), diffusion Transformers (DiT [4] ), and flow-based generative models (SiT [5] ), to facilitate image creation.GPT. The encoder and quantizer transform each image into a token map Ẑ, which is then flattened into a token sequence. Ultimately, GPT is trained on the collection of these token sequences.LDM. It progressively adds noise onto the encoder feature Z. The training objective is to denoise and reconstruct Z. During the inference phase, the output from LDM is inputted into the quantizer and decoder of image quantization models to generate images.DiT. This model is a variant of LDM, distinguished by its use of a Transformer architecture as the backbone. The incorporation of image quantization models into DiT follows the same approach as their integration into LDM.SiT. This method presents a flow-based generative framework utilizing the DiT backbone. The integration of image quantization models in SiT follows the same methodology as in LDM and DiT.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: we strictly comply with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
setup: 
 Implementation Details of Image Quantization. All image quantization models, including VQGAN, VQGAN-FC, VQGAN-EMA, and our proposed VQGAN-LC, utilize the same encoder and decoder of the original VQGAN. The input images are processed at a resolution of 256 × 256 pixels. The encoder (U-Net [28] ) downsamples the input image by a factor of 16, yielding a feature map Z with dimensions of 16 × 16. The quantizer then converts this feature map into a token map Ẑ of the same size, which is subsequently fed into the decoder (U-Net) for image reconstruction. In our observations, the optimal codebook size for VQGAN, VQGAN-FC, and VQGAN-EMA is 16,384, whereas for our VQGAN-LC, the optimal codebook size is 100,000. Training is conducted on the ImageNet-1K [24] and FFHQ [44] datasets, utilizing 32 Nvidia V100 GPUs. For ImageNet-1K, we train for 20 epochs, whereas for FFHQ, we train for 800 epochs. The Adam optimizer [45] is used, starting with an initial learning rate of 5e -4 . This learning rate follows a half-cycle cosine decay schedule after a linear warm-up phase of 5 epochs.Codebook Initialization of Our VQGAN-LC. Unless otherwise specified, we use the CLIP image encoder [23] with a ViT-L/14 backbone, adding an additional 4 × 4 average pooling layer, to extract patch-level features from images in the training split of the target dataset (either ImageNet or FFHQ). These features are then clustered into N groups using the K-Means algorithm with CUDA acceleration. The cluster centers constitute the codebook. By default, N is configured to 100,000. We specify the codebook entries to have a dimension of 8.Image Generation Models. For LDM [3] , DiT [4] and SiT [5] , we adopt their original architectures.For generation using GPT [2] , we follow VQGAN [1] , using a causal Transformer decoder with 24 layers, 16 heads per attention layer, a latent dimension of 1,024 and a total of 404M parameters.For ImageNet, we employ class-conditional generation, whereas for FFHQ, we use unconditional generation. In LDM, DiT, and SiT, classifier-free guidance [3] is implemented for class-conditional generation. More implementation details can be found in Section A.Evaluation. In the image reconstruction task, we evaluate performance using rFID, LPIPS, PSNR, and SSIM metrics on the validation sets of ImageNet and FFHQ. For image classification, we measure the top-1 accuracy on ImageNet. For image generation, we calculate the FID score on ImageNet using 50K generated images compared against the ImageNet training set. For FFHQ, the FID score is determined using 50K generated images in comparison with the combined training and validation sets of FFHQ. The codebook utilization rate is also reported for comparison, which is calculated as the ratio of active entries (tokens/codes) to the total size of the codebook.Image Reconstruction. Tables 2 and 3 present the reconstruction performance for ImageNet and FFHQ, respectively. We make three key observations: 1) Our method consistently achieves a codebook utilization rate of over 99% across all codebook sizes on both datasets.2) The reconstruction performance improves consistently with the scaling of codebook size using our method. 3) Increasing the codebook size (e.g., VQGAN-LC with a codebook size of 100,000 and 256 tokens), and the number of tokens to represent an image (e.g., RQVAE with a codebook size of 16,384 and 512 tokens) both enhance performance, with the former introducing almost no additional computational cost compared to the latter.Image Generation. Table 4 shows the results of class-conditional image generation on ImageNet. All models (GPT, LDM, DiT, and SiT) demonstrate improved performance with the integration of our VQGAN-LC, regardless of their underlying architectures, which include auto-regressive causal Transformers, diffusion models, diffusion models with Transformer backbones, and flow-based generative models. The diversity of the generated images increases due to the utilization of a large codebook, which has a size of up to 100,000 and a utilization rate exceeding 99%. based generative models. Given that FFHQ is significantly smaller than ImageNet, we limit our training on FFHQ to GPT and LDM.Image Classification. In Section 3.3, we discuss the training of an image classifier on a dataset containing tokenized images. We fine-tune three ViT-B [37] models, pre-trained by MAE [43] , using the tokenized images produced by the top-performing VQGAN-FC, VQGAN-EMA, and our proposed VQGAN-LC, on ImageNet. Both VQGAN-FC and VQGAN-EMA demonstrate optimal reconstruction performance when utilizing a codebook with 16,384 entries. As illustrated in Figure 1 (b), our method achieves a top-1 accuracy of 75.7 on ImageNet, surpassing VQGAN-FC and VQGAN-EMA by margins of 1.6 and 1.9, respectively.Visualizations. Section C presents images generated by GPT [2] , LDM [3] , DiT [4] , and SiT [5] , incorporating our VQGAN-LC.Unless otherwise specified, we evaluate reconstruction performance on ImageNet across all studies.Codebook Initialization. In Section 3.2, we describe the default codebook initialization approach. This involves using a CLIP vision encoder with a ViT-L backbone to extract patch-level features from ImageNet, followed by a K-means clustering algorithm to generate N cluster centers, resulting in the static codebook B. In Table 6 , we evaluate two factors: 1) employing a CLIP vision encoder with different backbones (e.g., ViT-B [37] and ResNet-50 [50] ) to extract patch-level features; and 2) utilizing non-clustering strategies to initialize the static codebook, including random initialization and random selection, where N features are randomly chosen from all patch-level features of ImageNet. Our findings are threefold: 1) random initialization leads to extremely poor performance since the codebook remains frozen in our VQGAN-LC; 2) using a CLIP vision encoder with a ViT-L backbone outperforms those using ViT-B and ResNet-50 backbones; and 3) K-means clustering produces a more robust codebook.Codebook Size. In Table 7 , we incrementally increase the codebook size in our VQGAN-LC from 1,000 to 200,000. The performance shows minimal improvements beyond a codebook size of 100,000-specifically, only 0.01 PSNR and 0.03 SSIM gains are observed, when the codebook size reaches 200,000. Therefore, we consistently use a codebook size of 100,000 in all experiments. The codebook utilization rate consistently exceeds 99% across all configurations.Computational Cost. The primary inference cost of VQ-GAN is attributed to the encoder and decoder.Increasing the codebook size N incurs minimal additional cost since the matrix multiplication between F and B is negligible compared to the encoder and decoder processing time. Table 8 , we present the multiply-accumulates (MACs) and model sizes for our VQGAN-LC models with codebook sizes of 16,384 and 100,000, respectively, when inferring an image of size 256 × 256.Codebook Transferability. In our standard setup, both the codebook initialization and the VQGAN-LC training are conducted using the same dataset, either ImageNet or FFHQ. In Table 9 , we examine the transferability of the codebook by initializing it with one dataset and training our VQGAN-LC on a different dataset. Our findings indicate that our method exhibits significant codebook transferability, highlighting the robustness of our codebook initialization process. 
conclusion: 
 In this work, we introduce a novel image quantization model, VQGAN-LC, which extends the codebook size to 100,000, achieving a utilization rate exceeding 99%. Our approach significantly outperforms prior models like VQGAN, VQGAN-FC and VQGAN-EMA, across various tasks including image reconstruction, image classification and image synthesis using numerous generation models such as auto-regressive causal Transformers (GPT), latent diffusion models (LDM), diffusion Transformers (DiT), and flow-based generative models (SiT), while incurring almost no additional costs. Extensive experiments on ImageNet and FFHQ verify the effectiveness of our approach.  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: we discuss the societal impacts in the appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: this is discussed in the appendix.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: we use public datasets like ImageNet and FFHQ.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
: 
 Table 1 : We conduct a comparative analysis of our VQGAN-LC against two advanced variants of VQGAN [1] , namely VQGAN-FC and VQGAN-EMA, focusing on the effects of enlarging their codebook sizes from 1,024 to 100K. The only difference among the three models lies in the initialization and optimization of the codebook. The evaluation covers both reconstruction and generation using the latent diffusion model (LDM) [3] on the ImageNet dataset. 
 
 Appendix: Projector and Static Codebook. As described in Section 3.2, the codebook is initialized using a CLIP vision encoder to extract patch-level features on ImageNet. During the training of our VQGAN-LC, we optimize a projector to map the entire codebook to a latent space. Table 10 illustrates the significance of tuning the projector on the static codebook by comparing our default strategy with two alternatives: 1) omitting the projector; and 2) making each entry in the initialized codebook trainable.Our results show that incorporating the projector markedly improves performance, whereas making the codebook entries trainable has minimal impact.Dimension of the Projected Codebook. In our VQGAN-LC, the projector transforms the codebook B into a latent codebook B ′ , with each entry in B ′ having a dimension denoted as D ′ . Table 11 shows the results of varying D ′ from 8 to 512. We find that our model's performance remains stable regardless of the value of D ′ , consistently achieving a codebook utilization rate of over 99% in all configurations.Size (1024) Token Replacement. VQGAN, VQGAN-FC, and VQGAN-EMA all utilize a codebook of 16,384 entries to achieve optimal performance. In contrast, our VQGAN-LC can scale the codebook size up to 100,000 while maintaining an exceptionally high utilization rate of over 99%, allowing each token to represent more detailed visual elements. This is verified through an ablation study: for each input image, we use VQGAN-FC (-EMA/-LC)'s encoder to convert the image into a token map. We then replace each token in the map with the M th nearest entry from its codebook. The modified token map is fed into the decoder for reconstruction. Figure 5 shows the PSNR results on a subset of ImageNet, using images from 100 randomly selected categories, and visualizes the results of our VQGAN-LC, VQGAN-FC, and VQGAN-EMA when M is set to 1, 50, 100, and 1000.In Figures 6 7 8 9 , we present the class-conditional generation results at a resolution of 256 × 256 for our VQGAN-LC with GPT [2] , LDM [3] , DiT [4] , and SiT [5] , respectively, using 256 (16 × 16) tokens on ImageNet. Additionally, Figure 10 illustrates the results of VQGAN-LC with LDM using 1024 (32 × 32) tokens on ImageNet. Figure 11 shows the unconditional generation results at a resolution of 256 × 256 for VQGAN-LC with LDM on the FFHQ dataset, utilizing 256 (16 × 16) tokens.The introduction of a large-scale codebook facilitates the generation of images with diverse poses, intricate textures, and complex backgrounds.We present a new image quantization technique called VQGAN-LC, which expands the codebook size to 100,000 with a utilization rate of 99%. This approach has the potential to enhance any downstream applications that involve image quantization models. However, VQGAN-LC is trained on the ImageNet and FFHQ datasets, which restricts downstream applications, like image generation, to producing images from the limited categories found in these datasets. While training VQGAN-LC on larger datasets like LAION-5B may improve its utility in downstream applications, it would also be more costly and computationally demanding. Furthermore, please refrain from using this model to generate malicious or inappropriate content. It is intended solely for positive and constructive purposes in research and creativity. Volcano (980) Arctic Fox (279) Pug (254) Geyser (974) Space Shuttle (812) Lake Shore (975) Guidelines:V Q G A N -E M A V Q G A N -F C• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.https://ai.meta.com/blog/meta-llama-3/ 


Decision: All reviewers suggested acceptance, altough without much enthusiasm. The paper's contribution is more of an engineering effort, but the practical results are good.

Reviews: review: paper_summary: paper_summary: The VQGAN-LC (Large Codebook) model tackles the challenges of expanding codebook size and utilization in image quantization. Unlike its predecessors, which struggled with limited codebook sizes and low utilization rates, VQGAN-LC increases the codebook size to 100,000 and achieves over 99% utilization. This model starts with features extracted by a pre-trained vision encoder and optimizes a projector to align these features with the encoder's distributions. VQGAN-LC demonstrates superior performance in tasks such as image reconstruction, classification, and generative image models compared to earlier methods like VQGAN-FC and VQGAN-EMA.
strength_weakness: strength_weakness: 1. My main concern is doubt about the scalability of this method. It appears that this paper does not provide enough evidence to show that it can still be useful for open-domain image reconstruction and generation under billion-level image datasets, since all experiments are carried out on small-scale datasets and low-resolution images (256x256).
questions: questions: 1. What is the definition of active or inactive for the utilization of the codebook? How to mesure it?
2. Can the tokenizer be integrated into large language models (GPT-like) to train a multimodal large language model?
limitations: limitations: The main limitation is also mentioned in the weakness.
paper_summary: paper_summary: The paper introduces VQGAN-LC (Large Codebook), a novel image quantization model that significantly extends the codebook size and enhances codebook utilization. Traditional models like VQGAN-FC are limited in codebook size and utilization rates, with a maximum size of 16,384 and utilization rates typically below 12%. In contrast, VQGAN-LC expands the codebook size to 100,000 and achieves utilization rates exceeding 99%. Instead of optimizing each codebook entry individually, the model trains a projector that aligns the entire codebook with the encoder's feature distributions. This approach allows for a significant increase in codebook size and utilization without a corresponding increase in computational cost.
strength_weakness: strength_weakness: In line 211 of page 6 ​​authors mention that the optimal codebook size of the proposed method is 100,000 which is intuitively correct. However, authors should also provide a comparison of smaller codebook sizes with other methods. In Table 3, Table 4, and Table 5  there is no comparison of VQGAN-LC with other methods utilizing a similar codebook size. 

Quantization error constitutes a principal component of the loss function equation; however, in the two variants, Factorized Codes (FC) and Exponential Moving Average (EMA), distinct formulations of this loss are observed. The rationale provided may be comprehensible to those familiar with the VQGAN architecture; nonetheless, the explanations offered in the document lack sufficient clarity for those less acquainted with this framework.

Misc:
Mention the batch size for the experiments in the experimental setup section
questions: questions: How does  Utilization Rate effect the performance? There seems to be a strong correlation for large codebook but for codebook sized 16,384 with “83% and 68%” Utilization achieve similar results compared to 99%Utilization.  In some cases lower Utilization has slightly better performance as well?

how does increasing codebook effect the inference time and memory footprint of the network. The paper claims in section 4.3 line#241 that large codebook incurs almost no additional computational cost. This is not obvious. How is computational cost independent of the codebook size?

Can you explain more about the perceptual loss used in eq.(1)

Does high utilization rate is any relationship with compressibility of the model? Intuitively lower utilization rates suggest that the model can be further compressed/ optimized? Will high utilization rates mean that it decreases the compressibility of the model?
limitations: limitations: NA
paper_summary: paper_summary: This study introduces VQGAN-LC (Large Codebook), an innovative image quantization model that significantly extends the codebook size to 100,000, achieving a utilization rate of 99%. Unlike previous methods that optimize each codebook entry individually, VQGAN-LC initializes its codebook with 100,000 feature centers from a pre-trained vision encoder. These codebook vectors are then frozen for the remainder of the process. The optimization focuses on training a projector to align the frozen codebook with the feature distributions of the encoder in VQGAN-LC. This approach ensures that nearly all token embeddings remain active throughout the training phase.
strength_weakness: strength_weakness: (1)	The paper primarily focuses on scaling the codebook size and improving utilization rates in VQGAN models. While this is a noteworthy improvement, it can be seen as an incremental advance rather than a novel approach. The method relies heavily on existing architectures and concepts, particularly those established by prior works such as VQGAN, VQGAN-EMA, and VQGAN-FC.

(2)	The related work section fails to comprehensively cover the breadth of existing research focused on improving codebook usage and addressing the codebook collapse problem. Numerous significant contributions in this area are overlooked, which diminishes the depth and rigor of the literature review. This lack of thoroughness may lead to an incomplete understanding of the current state of the field and the novelty of the proposed approach. You may cite some recent papers for the codebook utilization problem, such as [1], [2], or other top papers that are found on arxiv by searching keywords: discrete VAEs codebook collapse.

(3)	There is no hypothesis to explain why the proposed method works. The paper does not address how the method increases the utilization rate and why this increase leads to improved results. This lack of theoretical foundation makes it difficult to understand the underlying mechanisms driving the observed performance gains.

(4)	The paper lacks experiments and explanations regarding the quality of the codebooks and how well they span the representational space. While the method claims high utility even on ResNet50, it is crucial to study the coverage and distribution of the codebook entries. Experiments should be conducted to demonstrate how the codebook spans the feature space and ensure that the high utilization rate translates to meaningful and diverse representations. Without such analysis, it is difficult to ascertain the true effectiveness and robustness of the proposed method.

(5)	In Preliminary D, a fixed step size (M=1, 50, 100, 1000) is used for the Mth nearest replacement. However, the ratio of the step size to the codebook size is not consistent. For example, if the codebook size is 16,000 in the baseline methods and a step size of M=1000 is used, it would be more meaningful to compare with a distance of 100,000/16 when the codebook size is 100,000. This is because, relatively, in VQGAN-FC, the replacement is made with a vector that is 1 step away in the codebook, whereas in the baseline methods, the replacement is made with a vector that is 6 steps farther away. 

[1] Huh, M., Cheung, B., Agrawal, P., & Isola, P. (2023). Straightening Out the Straight-Through Estimator: Overcoming Optimization Challenges in Vector Quantized Networks. International Conference on Machine Learning.

[2] Takida, Y., Ikemiya, Y., Shibuya, T., Shimada, K., Choi, W., Lai, C.-H., … Mitsufuji, Y. (2024). HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes. Transactions on Machine Learning Research
questions: questions: (1)	When the codebook size increases significantly, I think that discretization might lose its meaning as the intuition behind discrete representation learning can be thought of as learning to represent the data space with a latent space as small as possible for a lower computational load. I’m curious about your justification for why we should increase the number of representations in the codebook instead of decreasing them, apart from better reconstruction performance.

(2)	One of my questions pertains to the span of the codebooks. Can you conduct a study or experiment to illustrate the coverage and distribution of the codebook entries? For instance, creating a t-SNE map by normalizing all codebooks and visualizing them in the same space could provide valuable insights into the codebook feature space learned by your model. This visualization would help us understand the diversity and quality of the representations within the codebook and how effectively it spans the feature space.
limitations: limitations: The authors address limitations and societal impact in Section D in Supplementary Material.

Comments: title: Response to rebuttal
Response to Reviewer c7tj
comment: Thank you for the detailed response.

I am satisfied with the authors response and I have increased my score accordingly.
I have read the rebuttal. The authors’ responses are mostly satisfactory, particularly with addition of the new experiments. However, I still believe this work still lacks a theoretical foundation. It is a strong experimental paper. 
I will increase my score accordingly.
Dear Reviewer c7tj,

Thank you for your question. We appreciate the opportunity to clarify this point.

In our rebuttal, we highlighted that the main goal in GPT-style image generation using image quantization models is to produce high-quality, diverse, and realistic images. Achieving this often involves expanding the codebook (increasing its size or employing better optimization strategies), using more tokens to represent an image (e.g., from 256 to 1024 tokens), or adopting a more advanced backbone (e.g., replacing CNN with ViT).

Several statements from vector quantization models [1][2][3] support our argument. VQGAN [1], VQ-VAE-2 [2], and Reg-VQ [3] all follow a two-step process for image generation: first, an image quantization model is trained to convert an image into a sequence of tokens, and then a GPT is trained to model these discrete tokens. These statements include:

- In VQGAN [1], the authors state, "using **transformers** to represent images as a distribution over latent image constituents requires us to push the limits of compression and **learn a rich codebook**."

- In VQ-VAE-2 [2], the authors state, "we **scale and enhance the autoregressive priors** used in VQ-VAE to generate synthetic samples of **much higher coherence and fidelity** than possible before."

- In Reg-VQ [3], the authors state, "we can observe that **the performance of regularized quantization (Reg-VQ) improves clearly** with **the increasing of codebook size**."


A consistent theme across these works is **their emphasis on enhancing the quality of the generated images**. They adopt different strategies, such as expanding the codebook size or increasing the number of tokens used for image representation, to achieve this objective. It is important to highlight, as discussed in response to "Q5: Inference and Memory Costs", that expanding the codebook size results in minimal additional costs, while increasing the number of tokens used to represent an image substantially raises the GPT training and inference expenses.


Thank you once again. We look forward to continuing our discussions with you.


[1] Taming Transformers for High-Resolution Image Synthesis, CVPR 2021.

[2] Generating Diverse High-Fidelity Images with VQ-VAE-2, NeurIPS 2019.

[3] Regularized Vector Quantization for Tokenized Image Synthesis, CVPR 2023.
I have read the rebuttal and I am satisfied with the Authors response. 

However I have one additional question: in the main rebuttal authors have mentioned 

"The primary objective of GPT-style image generation using image quantization models is to produce high-quality, diverse, and realistic images, rather than minimizing the size of the codebook used to represent images."

Can you please provide any references for this statement