Image Understanding Makes for A Good Tokenizer for Image Generation

Authors: luting  wang
yang  zhao
zijian  zhang
jiashi  feng
si  liu
bingyi  kang
  bytedance
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 13:04:22

Abstract: Modern image generation (IG) models have been shown to capture rich semantics valuable for image understanding (IU) tasks. However, the potential of IU models to improve IG performance remains uncharted. We address this issue using a token-based IG framework, which relies on effective tokenizers to project images into token sequences. Currently, **pixel reconstruction** (e.g., VQGAN) dominates the training objective for image tokenizers. In contrast, our approach adopts the **feature reconstruction** objective, where tokenizers are trained by distilling knowledge from pretrained IU encoders. Comprehensive comparisons indicate that tokenizers with strong IU capabilities achieve superior IG performance across a variety of metrics, datasets, tasks, and proposal networks. Notably, VQ-KD CLIP achieves $4.10$ FID on ImageNet-1k (IN-1k). Visualization suggests that the superiority of VQ-KD can be partly attributed to the rich semantics within the VQ-KD codebook. We further introduce a straightforward pipeline to directly transform IU encoders into tokenizers, demonstrating exceptional effectiveness for IG tasks. These discoveries may energize further exploration into image tokenizer research and inspire the community to reassess the relationship between IU and IG. The code is released at https://github.com/magic-research/vector_quantization.

Main paper: 
 introduction: 
 Image understanding (IU) and image generation (IG) have been the core pursuits of computer vision research for a long time. Thanks to the progress in generative models [35, 37, 15, 36, 48] and network architectures [43, 9] , IG has witnessed remarkable advancements in recent years. These advancements spurred extensive research on leveraging powerful IG models for IU tasks (Fig. 1 ). Studies have shown that IG models can benefit IU tasks in various ways, including data augmentation through synthetic data generation [51, 46, 45] , improved representation learning [28, 16, 52] , and utilizing intermediate features from IG models for solving perception tasks [53, 26] . However, the reciprocal question remains largely uncharted: how might IU models aid IG tasks?The primary focus of this paper lies in the AutoRegressive (AR) IG framework, which is gaining considerable attention for its excellence in generating high-quality images and videos [48, 50, 20] . This framework operates in a two-stage process. The first stage learns a tokenizer to map images into sequences of discrete tokens. Subsequently, the second stage trains a proposal network to model the token sequences. As underlined by prior research [49, 4] , the quality of the tokenizers significantly influences overall IG performance. Meanwhile, tokenizers and IU encoders adhere to a similar structure as they both aim to map images into latent representations, either discrete or continuous. As a result, the token-based IG framework provides an optimal environment for Generator 𝒛 ∼ 𝒩 0, 𝑰Representation Learning Data Augmentation Intermediate Feature ⋯ ?Predictor 𝒚 ∈ ℝ 𝒉×𝒘×𝒄Figure 1 : Extensive studies have tried to adopt IG models for IU. However, few attempts have been made to use IU models in IG.investigating the relationship between IU and IG. Through comprehensive studies, we demonstrate that existing IU models from representation learning can be useful in generative models, even if they are not specifically designed for the IG task.Our study involves training three components within the AR framework: tokenizer, decoder, and proposal network. Traditionally, pixel reconstruction has been the dominant objective for training tokenizers, such as VQGAN [10] and FSQ [25] . To the best of our knowledge, we are the first to systematically demonstrate that feature reconstruction (VQ-KD [27] ) achieves better IG performance. 1This approach distills knowledge from pretrained IU encoders to tokenizers. Therefore, the training strategy of the IU encoder is crucial for the performance of the tokenizer. In this regard, we investigate four representative IU encoders: ViT [9] , CLIP [29] , DINO [3] , and MAE [13] . Following VQGAN [10] , we train decoders to restore pixel values from discrete tokens, and proposal networks (AR or NAR) that can model the distribution of image tokens. The models are then evaluated using various metrics, including codebook usage, Fréchet Inception Distance (FID) [14] , Inception Score (IS) [34] , perplexity (PPL), etc.Initially, we compare the above tokenizers on IN-1k for class-conditional IG. VQ-KD achieves 4.10 FID AR , outperforming VQGAN (15. 78 FID AR ) by a large margin. FSQ experiments confirm that the superiority of VQ-KD is not solely attributable to the specific quantization operation or high codebook usage. More generally, VQ-KD consistently outperforms across different proposal networks, datasets, and tasks.We analyze VQ-KD from multiple perspectives. By visualizing the codebook, we discover that codes from VQ-KD carry more semantics than VQGAN, which makes them easier to model and subsequently improve the IG quality. Building upon this insight, we propose a straightforward pipeline to efficiently transform IU encoders into tokenizers, outperforming VQ-KD on the MS-COCO dataset. We also find that tokenizers with weaker IU capabilities require larger proposal networks for effective AR modeling and show less robustness to variations in the training images. Finally, we conduct qualitative analysis to present the visual results.In sum, the key insights from our study include the following: 1) This research is the first to demonstrate that IU models can substantially enhance IG through VQ-KD; 2) Tokenizers with strong IU capabilities consistently outperform conventional VQGAN-based methods across various metrics, datasets, tasks, and network architectures; 3) The VQ-KD codebook encapsulates more semantics than VQGAN, contributing to the superiority of VQ-KD in IG.We believe these findings can benefit future research on image tokenizers and provoke further discussion on the relationship between IU and IG. 
related work: 
 Image Tokenization. Vector Quantization (VQ) [11] is originally developed for data compression.To circumvent posterior collapse in the VAE [19] framework, VQ-VAE [41] adopts VQ for image tokenization. Subsequently, VQGAN [10] introduces adversarial and perceptual losses to enhance the quality of the generated images. Vanilla VQGAN suffers from limitations like low codebook usage, limited semantic representation ability, and the trade-off between modeling efficiency and image quality. To address these challenges, researchers have focused on improving the codebook. ViT-VQGAN [47] adopts a ViTbased [9] autoencoder to create more expressive code vectors. SeQ-GAN [12] improves the perceptual loss and decoder to balance between semantic compression and detail preservation. SQ-VAE [40] improves VQ-VAE with stochastic quantization and a trainable posterior categorical distribution. VQ-WAE [44] builds upon SQ-VAE by encouraging the discrete representation to be a uniform distribution via a Wasserstein distance. HQ-VAE [39] employs random re-initialization of inactive code vectors. CVQ-VAE [55] selects encoded features as anchors to update dead codes. VQ-KD [27] adopts knowledge distillation instead of image reconstruction as the objective to train VQ-VAE. LFQ [49] and FSQ [25] adopt bounded scalar quantization techniques from neural compression to harness the potential of extra-large codebooks.Furthermore, several works explore the potential of multiple codebooks. VQ-VAE-2 [32] extends VQ-VAE to a multi-scale hierarchical organization. RQ-VAE [21] and MoVQ [54] aim to represent each feature as a stack of tokens, where RQ-VAE adopts an iterative way to factorize features into a series of residuals and MoVQ models features across multiple channels via specialized modulation.Token-based Image Generation. Inspired by the success of GPT [30, 31, 2] , VQ-VAE and most of its derivative works [10, 54] adopt AR transformers to model the token sequence. This approach leverages techniques from text generation to enhance IG performance. However, the decoding time of AR models scales linearly with the length of the token sequence. To accelerate decoding, MaskGIT [5] introduces a bidirectional transformer, referred to as the NAR proposal network.Given the versatility of token-based modeling, both AR and NAR proposal networks can be easily extended to conditional IG scenarios. For instance, VQGAN uses a class token as the condition in its AR proposal network for class-conditional IG. With an NAR proposal network, MUSE [4] adopts text embeddings to predict masked image tokens in Text-to-Image generation. 
token-based image generation: 
 We start with the two-stage IG framework in Sec. 3.1. Subsequently, Sec. 3.2 details the architecture and training protocol for the tokenizers under consideration. Sec. 3.3 explains the evaluation benchmark. Sec. 3.4 further outlines our main observations derived from the IN-1k experiments. Lastly, we validate the observations under different settings in Sec. 3.5.We illustrate the two-stage IG framework in Fig. 2 . Given an image I ∈ R H×W ×3 , the encoder E converts this image into a feature map x ∈ R h×w×d , where (h, w) = (H/f, W/f ) and f is a downsample factor. Let a codebook C be a set of N code vectors {c i } N i=1 ∈ R N ×d , where each code vector c i ∈ R d corresponds to a specific code i. A quantizer Q then maps x into a sequence of codes z = {z i } L i=1 , where L = h × w defines the sequence length and z i is an integer that falls within the range [1, N ] . Let c zi denote the code vector that corresponds to code z i . Similarly, C(z) = {c zi } L i=1 ∈ R L×d represents a sequence of code vectors associated with the code sequence z. The encoder E, quantizer Q, and codebook C collectively form an image tokenizer T .The proposal network P models the distribution over z, where the distribution is denoted as p(z). Early proposal networks are implemented as an AR transformer, which sequentially models p(z i |z 1:i-1 ) and formulates p(z) as h×w i=1 p(z i |z 1:i-1 ). While the AR transformers can be trained in parallel, it has to sequentially decode z during inference, which renders it inefficient. Therefore, NAR proposal networks are being prevalent [5] , which typically adopt bidirectional transformers to model z. We denote the two types of proposal networks as P AR and P NAR , respectively. Finally, a decoder D maps the code vectors to the pixel space. In training, D takes C(z) as input and learns to reconstruct the original image I as Î ∈ R H×W ×3 . In inference, a sequence of codes ẑ is sampled from p(z), translated to C(ẑ), and then fed into D to generate an image Ĩ.In this paper, we focus on three types of image tokenizers: VQGAN [10] , FSQ [25] , and VQ-KD [27] .Let x i,j ∈ R d be a vector in the feature map x. As shown in Fig. 3a , to quantize x i,j , the VQGAN tokenizer looks up the codebook C for the closest code vector in terms of Euclidean distance:z i,j = arg min z ∥x i,j -c z ∥ 2 .(EQUATIONSince the quantization process is non-differentiable, VQGAN adopts the Straight-Through Estimator (STE) [1] to optimize the encoder E, which copies gradients from C(z) to x. As a result, the codebook C receives no gradient. To optimize C, VQGAN introduces a quantization loss L Q :EQUATIONwhere sg[•] denotes the stop-gradient operation and β is the loss weight. The first term is the codebook loss, which optimizes the codebook. The second term is the commitment loss to make sure the encoder E commits to a code vector [41] . Therefore, the overall loss for VQGAN is defined as:EQUATIONwhere L rec is the reconstruction loss between image I and reconstruction Î, which includes ℓ 1 loss, perceptual loss, and adversarial loss.Based on VQGAN, FSQ introduces a simpler image tokenizer, without the need for codebook lookup and quantization loss. As shown in Fig. 3b , FSQ adopts finite scalar quantization to quantize . Formally, the reconstruction loss is defined as:EQUATIONwhere cos(•, •) represents cosine similarity.In this study, we examine VQ-KD using four types of pretrained teachers, including fullysupervised, text-supervised, contrastive, and Masked Image Modeling (MIM). We use VQ-KD CLIP and VQ-KD DINO to represent VQ-KD tokenizers trained with CLIP [29] and DINO [3] teachers, respectively. VQ-KD MAE and VQ-KD ViT represent tokenizers trained with MAE [13] and ViT [9] teachers. The latter two teachers are pretrained on IN-1k utilizing a ViT-B/16 architecture.We detail how we fairly compare different tokenizers for token-based IG here.For each tokenizer, we train a proposal network P and a decoder D to constitute an image generator.In training, the tokenizer is frozen to ensure fairness. Thus, P and D can be trained in parallel. We follow VQGAN [10] to train the AR proposal network and the decoder. The NAR proposal network is trained following MAGE [22] . Implementation details can be found in Appendix B.Our benchmark adopts various metrics to comprehensively evaluate the image tokenizers. Given an image tokenizer, we assess the effectiveness of its encoding process by evaluating the codebook usage. To assess the generative capabilities of the image tokenizers, we evaluate IS [34] and FID [14] on the generated images Ĩ. We assess the reconstruction capabilities of the image tokenizers by reporting the reconstruction FID (rFID). In addition, we present the PPL scores to appraise the AR modeling proficiency of the image tokenizers. A low PPL score implies that P AR easily models z. Details about the evaluation metrics can be found in Appendix C.We evaluate the class-conditional IG performance of VQGAN, FSQ, and VQ-KD tokenizers on IN-1k. The results in Tab. 1 leads to the following observations. (11.40) . This trend can be attributed to the superior capability of supervised models in capturing semantics compared to the unsupervised ones.The superiority of VQ-KD holds across proposal networks. As seen in Tab. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: The paper does not include theoretical results.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
analysis: 
 In this section, we analyze tokenizers based on feature reconstruction from various perspectives.We delve into the superiority of VQ-KD by visualizing its codebook. From IN-1k, we randomly choose four categories: golden retriever, pirate ship, valley, and sea anemone. For each image belonging to these categories, we deploy t-SNE [42] to project the feature map x and the code vectors C(z) into a two-dimensional space.x is colored according to the image category and C(z) is illustrated in black. As depicted in Fig. 4 , the feature space of VQ-KD shows superior organization compared to VQGAN. In the VQ-KD feature space, x from the same category are grouped together. This implies that each code in the VQ-KD codebook conveys clear semantics. As codes with similar semantics are likely to present concurrently in an image, it becomes easier for the proposal network to model the code sequence z. Conversely, each code in the VQGAN codebook is shared by multiple categories, resulting in semantics ambiguity. Hence, as illustrated in Tab. 1, the PPL metric for VQGAN is higher than VQ-KD, even though its codebook usage is considerably lower. To better harness the semantics in IU encoders, we propose a straightforward pipeline that transforms IU encoders into tokenizers via feature clustering. Given a pretrained IU model T ′ , we employ it to encode the feature map x T and subsequently utilize a clustering approach [55] We investigate the impact of strong data augmentation on the AR modeling performance of tokenizers. Specifically, we employed a strong random crop, where the crop scale ranges from 0.08 to 1.0, introducing greater variability into the training data. As shown in Tab. 5, all tokenizers exhibit worse FID AR metrics than their counterparts in Tab. 1. Interestingly, tokenizers with stronger IU capabilities demonstrate greater robustness to the strong data augmentation. For instance, VQ-KD ViT experiences a minor increase in FID AR of just 1.92 (from 11.40 to 13.32), whilst VQ-KD MAE records a considerable leap of 9.78 (from 26.85 to 36.63). The size and dimension of the codebook exert a significant influence on the IG performance of tokenizers [10, 47] . Tab. 7a showcases the performance of VQ-KD CLIP with varying codebook sizes. Large codebooks aid the tokenizers in representing fine-grained semantics, contributing to a consistent decrease in the rFID metric from 6.59 to 4.53. The IS metric also shows favor towards larger codebooks, with size 2 14 leading to the highest IS metric of 131.51. However, choosing the correct code from a large codebook is harder than from a small codebook, hindering P AR from achieving lower FID scores with larger codebooks.Tab. 7b demonstrates the influence of codebook dimension. High-dimensional codes carry more information but lead to lower codebook usage. As a result, the rFID metric initially drops from 4.96 to 4.64, then increases drastically to 6.80. Similar to Tab. 7a, the FID and IS metrics favor different codebook dimensions. FID favors 128-dimensional codebooks, where codebook usage is relatively low. In contrast, IS favors 32-dimensional codebooks, possibly due to a superior diversity.The reconstruction quality of various tokenizers is demonstrated in Fig. 5 . Original images are displayed in the first row. Regions where VQGAN and FSQ fail to reconstruct are highlighted with red boxes. In contrast, VQ-KD reconstructions are visually more accurate. Since the IU encoder in Cluster emphasizes encoding semantics over visual details, Cluster fails to preserve all visual details during reconstruction. Nonetheless, the reconstruction results of Cluster still appear more natural than VQGANand FSQ, especially in the highlighted areas. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: The model architecture is demonstrated in sec. 3.1. The implementation details are illustrated in appendix B. The code is released at https://github.com/ magic-research/vector_quantization. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines: 
conclusion: 
 In this paper, we show that image understanding (IU) models can be useful in image generation (IG). Through comprehensive studies, observe that the VQ-KD tokenizers significantly enhance generation quality over VQGAN, irrelevant of the quantization operation and codebook utilization. Within the VQ-KD tokenizers, stronger IU capabilities tends to deliver superior IG performance.Further verification shows that the superiority of VQ-KD holds across proposal networks, datasets, and tasks. Lastly, we analyze VQ-KD from multiple angles, including clustering pretrained models as tokenizers, scaling up the proposal network, influence of strong data augmentation, large teacher models in VQ-KD, and codebook size and dimensions.rFID is defined as the FID between I and their reconstructed counterparts Î. Obtaining a low rFID score requires that the image tokenizer encode sufficient visual details within the codes C(z) to enable accurate reconstruction by the decoder.The PPL score is defined as:EQUATIONwhere z denotes a sequence of codes offered by the tokenizer, L represents the length of z, and p(z) embodies the distribution modeled by the AR proposal network P AR .Both reconstruction and AR modeling serve as two pivotal capabilities in an image generator. We anticipate that these metrics will lead to a more thorough insight into the generative capacities of image tokenizers.• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper mainly uses the BEiT v2 repository, which is cited in the paper and mentioned in the code.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: The VQ-KD tokenizers are designed to mimic the IU encoders, yielding superior quantitative results compared to traditional tokenizers like the VQGAN and FSQ. Nonetheless, qualitative analysis suggest that the VQ-KD may modify visual details during the pixel reformation process, thereby posing challenges for tasks such as image editing.This paper explores the question how might image understanding (IU) models aid image generation (IG) tasks. We envision that our findings will motivate research on image tokenizers and prompt the community to reconsider the correlation between IU and IG.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.VQ-KD was initially proposed for image pretraining. 


Decision: This paper received positive scores of (5, 5, 6, 6) from the reviewers. Reviewers unanimously found the application of image understanding models to enhance image generation intriguing. While concerns were raised regarding the generalizability and qualitative results, the authors successfully addressed these concerns in their rebuttal. The AC, having carefully considered the paper, reviews, and rebuttal, concurs with the reviewers' recommendation for acceptance. The authors are encouraged to incorporate the reviewers' feedback into their final submission.

Reviews: review: paper_summary: paper_summary: This paper explores using image understanding (IU) models to aid image generation (IG) performance. To verify the hypothesis, the authors focus on the different tokenizers and introduce feature reconstruction (VQ-KD) as a training objective for image tokenizers, distilling knowledge from pre-trained IU encoders. This paper compares VQ-KD tokenizers with conventional methods like VQGAN and FSQ across various metrics, datasets, tasks, and proposal networks. The results show that tokenizers with strong IU capabilities, particularly VQ-KD, outperform traditional methods.
strength_weakness: strength_weakness: - While the paper examines four types of IU encoders, it could benefit from investigating a broader range of IU models to strengthen its conclusions.
- The paper could include more detailed ablation studies to isolate the impact of different components in the VQ-KD approach.
- As in the limitation, maybe it is better to introduce the fidelity-related metrics to indicate that this work has a better generation ability, but may result in lower fidelity.
- Some visual comparisons are not obvious. It is better to give a close-up to see the details.
questions: questions: 1. How does the proposed method perform on more diverse and challenging datasets beyond ImageNet-1k and MS-COCO, such as medical imaging or satellite imagery? This may show the generalization ability of the proposed model.
2. The description of VQGAN 'pixel reconstruction' seems not suitable, as VQGAN adopts the perceptual loss which may also introduce the semantic constraint.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces a novel framework that leverages the rich semantic capabilities of Image Understanding models for Image Generation tasks. By employing a token-based generation framework and a feature reconstruction objective, the paper trains tokenizers capable of mapping images into token sequences. Compared to traditional pixel reconstruction methods, this approach demonstrates good performance across various metrics.
strength_weakness: strength_weakness: 1. The paper lacks theoretical support for the VQ-KD tokenizer.
2. The experiments are limited to the ImageNet and COCO dataset, which may not be sufficient to demonstrate the effectiveness of the method. Consider adding higher resolution and higher quality datasets, like the LAION-Aesthetics
3. There is a lack of comparison and discussion regarding computational efficiency and computational load.
questions: questions: Please see weaknesses
limitations: limitations: 1. Experiments dataset is limited 
2. Generalizability was not enough explored and discussed
paper_summary: paper_summary: This work focuses on the connection between image understanding (IU) and image generation (IG). The authors introduce a token-based IG framework and a novel feature reconstruction objective for tokenizer training. They introduce an extra feature reconstruction loss to distill semantic knowledge from pretrained IU model to tokenizer. They demonstrate superior IG performance with such tokenizers having strong IU capabilities, as evidenced by various metrics, datasets, tasks, and networks.
strength_weakness: strength_weakness: 1. **Excessive and unjustified claims.**
    - An important contribution stated by this work is that it is the first to combine IU with IG, which might not hold true. I am not very familiar with related works in this specific direction, but I can mention one from ICLR 2024 [a]. They not only use IU as a tokenizer but also reduce the token number in a dynamic way.
1. **Observations in Sec. 3.4 needs further justification** For example,
    - More experiments are needed to verify that *The superiority of VQ-KD is irrelevant to the quantization operation and codebook usage.*
    - The observation that *Tokenizers with stronger semantic understanding tend to deliver superior IG performance* are not fully supported  by the experimental results.

1. **More visualization results are necessary.**
    - The paper is almost completely backed up by quantitative results. This may not be favorable for an image generation method. Many more examples are needed to enhance the persuasiveness of the experiment. For example, results on MSCOCO (Tab. 3), qualitative comparison with different IU backbone, etc.

1. **Lack of qualitative analysis for the learned codebook.**
    - There is no visualization of what is encoded in the VQ-KD and why it is superior to the original VQGAN. The authors could refer to [a] for some examples.

1. **Others**
    - Please use `Tab. X` and `Fig. X` in main text.
    - What is $D$ is used in Eq (4)? 

[a] Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization. Jin et al. ICLR2024.
questions: questions: In summary, this is a work featuring good motivation and quantitative results. However, I have the following concerns:
- As I am not familiar  with specific related works, I am unable to offer a competent evaluation of the quantitative results and assess the significance of these outcomes. I would prefer to see the comments from more expert reviewers before arriving at a final decision.
- The comparison with related works appears insufficient. There is almost no comparison made with state-of-the-art image generation models.
- Why are there so few qualitative results? For an image generation paper, visual examples are of considerable importance.
- Visual analysis of learned codebook is also missing.
limitations: limitations: Although the authors discuss the limitations in a few sentences, I believe more should be expounded.
paper_summary: paper_summary: This paper shows that image understanding models can be helpful in image generation and that a stronger IU model can result in better IG performance. To demonstrate it, this paper sets multiple experiments from many perspectives (e.g. different datasets and codebook sizes) and gives out reasonable analysis.
strength_weakness: strength_weakness: 1.The visualization of reconstruction of VQ-KD and Cluster seems worse than that of VQGAN and FSQ in both paper and appendix from my point of view. Maybe more visual results should be included.

2.This paper seems to claim that IU model can help IG, but this paper's experiments and analysis are limited to token based tasks. I think this paper should include more IG methods like diffusion, VAE with some little experiments or analysis if possible. Or this paper should emphasize the range is limited to token based tasks.

3.Various experiments have been done across different settings and perspectives, but I think this paper lacks of methods.Besides VQGAN, FSQ and VA-KD, more related methods should be included.

4.The structure of this paper is a little confusing, with experiments, analysis and descriptions mixed together.
questions: questions: 1.Why doesn't a larger teacher model always lead to better results? Following the theory of this paper, ViT-G/14 should perform best.

2.The experiment about different codebook sizes and dimensions seems to reveal that when the IU model gets stronger, the IG may not always be better. Can I think that to make better IG results, we don't have to use an IU model with too strong capability? I need detailed analysis.
limitations: limitations: From this paper's limitations, although VQ-KD performs better due to its stronger capability in image understanding, it is not suitable for all scenarios.

Comments: comment: Hi Authors,

I do not have other questions. Thanks for addressing my concerns.
Dear Reviewer,

Thank you for the suggestions that help us improve the paper. As the deadline for discussion is approaching, please let us know if you have any additional questions. We genuinely hope you can consider raising the score if we have satisfactorily addressed your concerns.

Thanks again,

The Authors
Dear Reviewer,

Thank you for the suggestions that help us improve the paper. As the deadline for discussion is approaching, please let us know if you have any additional questions. We genuinely hope you can consider raising the score if we have satisfactorily addressed your concerns.

Thanks again,

The Authors
Thank you so much for your detailed response and raising your score.
Dear Reviewer,

Thank you for your feedback on our rebuttal. We appreciate your thorough review and would like to ensure we address any remaining concerns you may have. Could you kindly specify which kind of explanation would be most helpful in better supporting the effectiveness of the VQ-KD tokenizer?

Thank you once again for your valuable insights.

Best regards,
The Authors
I have carefully read your rebuttals and the rebuttals have solved my questions. The new visualizations can reflect that IU model helps IG tasks well. The experiments on diffusion models can also demonstrate this paper's conclusion(though diffusion is not actually what this paper discusses). The comparisons and linear probing experiments about IU capability are clear. And it is good for you to do such detailed experiments during this time. I think this paper is worth accepted.
I really appreciate the author's additional experiments on additional datasets, which better prove the advantages of the method. The author addressed most of my concerns, but I still have some concerns about the explanation of VQ-KD tokenizer.
I think it is better to keep the original rating.
Dear Reviewers,

This is a reminder that the Reviewer-Author discussion period is nearing its end. We encourage you to participate in the discussion and provide your valuable feedback to the authors. Your comments are greatly appreciated and will contribute to the quality of the submission.

AC