Title: Image Understanding Makes for A Good Tokenizer for Image Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Image Tokenizer

Vector Quantization

Image Generation
Abstract: Modern image generation (IG) models have been shown to capture rich semantics valuable for image understanding (IU) tasks. However, the potential of IU models to improve IG performance remains uncharted. We address this issue using a token-based IG framework, which relies on effective tokenizers to project images into token sequences. Currently, **pixel reconstruction** (e.g., VQGAN) dominates the training objective for image tokenizers. In contrast, our approach adopts the **feature reconstruction** objective, where tokenizers are trained by distilling knowledge from pretrained IU encoders. Comprehensive comparisons indicate that tokenizers with strong IU capabilities achieve superior IG performance across a variety of metrics, datasets, tasks, and proposal networks. Notably, VQ-KD CLIP achieves $4.10$ FID on ImageNet-1k (IN-1k). Visualization suggests that the superiority of VQ-KD can be partly attributed to the rich semantics within the VQ-KD codebook. We further introduce a straightforward pipeline to directly transform IU encoders into tokenizers, demonstrating exceptional effectiveness for IG tasks. These discoveries may energize further exploration into image tokenizer research and inspire the community to reassess the relationship between IU and IG. The code is released at https://github.com/magic-research/vector_quantization.

Image Understanding Makes for A Good Tokenizer for Image Generation

Abstract

Modern image generation (IG) models have been shown to capture rich semantics valuable for image understanding (IU) tasks. However, the potential of IU models to improve IG performance remains uncharted. We address this issue using a token-based IG framework, which relies on effective tokenizers to map images into token sequences. Currently, pixel reconstruction ( e.g ., VQGAN) dominates the training objective for tokenizers. In contrast, our approach adopts the feature reconstruction objective, where tokenizers are trained by distilling knowledge from pretrained IU encoders. Comprehensive comparisons indicate that tokenizers with strong IU capabilities achieve superior IG performance across a variety of metrics, datasets, tasks, and proposal networks. Notably, VQ-KDCLIP achieves 4 . 10 FID on ImageNet-1k. Visualization suggests that the superiority of VQ-KD can be partly attributed to the rich semantics within the VQ-KD codebook. We further introduce a straightforward pipeline to directly transform IU encoders into tokenizers, demonstrating exceptional effectiveness for IG tasks. These discoveries may energize further exploration into image tokenizer research and inspire the community to reassess the relationship between IU and IG. The code is released at https://github.com/magic-research/vector_quantization .

1 Introduction

Image understanding (IU) and image generation (IG) have been the core pursuits of computer vision research for a long time. Thanks to the progress in generative models [15,35-37,48] and network architectures [9,43], IG has witnessed remarkable advancements in recent years. These advancements spurred extensive research on leveraging powerful IG models for IU tasks (Fig. 1). Studies have shown that IG models can benefit IU tasks in various ways, including data augmentation through synthetic data generation [45, 46, 51], improved representation learning [16, 28, 52], and utilizing intermediate features from IG models for solving perception tasks [26,53]. However, the reciprocal question remains largely uncharted: how might IU models aid IG tasks?

The primary focus of this paper lies in the AutoRegressive (AR) IG framework, which is gaining considerable attention for its excellence in generating high-quality images and videos [20, 48, 50]. This framework operates in a two-stage process. The first stage learns a tokenizer to map images into sequences of discrete tokens. Subsequently, the second stage trains a proposal network to model the token sequences. As underlined by prior research [4, 49], the quality of the tokenizers significantly influences overall IG performance. Meanwhile, tokenizers and IU encoders adhere to a similar structure as they both aim to map images into latent representations, either discrete or continuous. As a result, the token-based IG framework provides an optimal environment for investigating the relationship between IU and IG. Through comprehensive studies, we demonstrate

Generator

𝒛 ∼ 𝒩 0, 𝑰

Image Generation (IG)

⋯

?

Predictor

𝒚 ∈ ℝ 𝒉×𝒘×𝒄

Image Understanding (IU)

that existing IU models from representation learning can be useful in generative models, even if they are not specifically designed for the IG task.

Our study involves training three components within the AR framework: tokenizer, decoder, and proposal network. Traditionally, pixel reconstruction has been the dominant objective for training tokenizers, such as VQGAN [10] and FSQ [25]. To the best of our knowledge, we are the first to systematically demonstrate that feature reconstruction (VQ-KD [27]) achieves better IG performance. 1 This approach distills knowledge from pretrained IU encoders to tokenizers. Therefore, the training strategy of the IU encoder is crucial for the performance of the tokenizer. In this regard, we investigate four representative IU encoders: ViT [9], CLIP [29], DINO [3], and MAE [13]. Following VQGAN[10], we train decoders to restore pixel values from discrete tokens, and proposal networks (AR or NAR) that can model the distribution of image tokens. The models are then evaluated using various metrics, including codebook usage, Fr´ echet Inception Distance (FID) [14], Inception Score (IS) [34], perplexity (PPL), etc .

Initially, we compare the above tokenizers on IN-1k for class-conditional IG. VQ-KD achieves 4 . 10 FIDAR, outperforming VQGAN ( 15 . 78 FIDAR) by a large margin. FSQ experiments confirm that the superiority of VQ-KD is not solely attributable to the specific quantization operation or high codebook usage. More generally, VQ-KD consistently outperforms across different proposal networks, datasets, and tasks.

We analyze VQ-KD from multiple perspectives. By visualizing the codebook, we discover that codes from VQ-KD carry more semantics than VQGAN, which makes them easier to model and subsequently improve the IG quality. Building upon this insight, we propose a straightforward pipeline to efficiently transform IU encoders into tokenizers, outperforming VQ-KD on the MSCOCO dataset. We also find that tokenizers with weaker IU capabilities require larger proposal networks for effective AR modeling and show less robustness to variations in the training images. Finally, we conduct qualitative analysis to present the visual results.

In sum, the key insights from our study include the following: 1) This research is the first to demonstrate that IU models can substantially enhance IG through VQ-KD; 2) Tokenizers with strong IU capabilities consistently outperform conventional VQGAN-based methods across various metrics, datasets, tasks, and network architectures; 3) The VQ-KD codebook encapsulates more semantics than VQGAN, contributing to the superiority of VQ-KD in IG.

We believe these findings can benefit future research on image tokenizers and provoke further discussion on the relationship between IU and IG.

2 Related Work

Image Tokenization. Vector Quantization (VQ) [11] is originally developed for data compression. To circumvent posterior collapse in the VAE [19] framework, VQ-VAE [41] adopts VQ for image tokenization. Subsequently, VQGAN [10] introduces adversarial and perceptual losses to enhance the quality of the generated images.

Vanilla VQGAN suffers from limitations like low codebook usage, limited semantic representation ability, and the trade-off between modeling efficiency and image quality. To address these

1

2

⋮

𝑁

Decoder

𝒟

3

2

2

3

4

1

3

5

2

4

6

1

5

6

1

6

AR / NAR

Proposal Network

𝒫

Image

𝑰

Reconstruction

෠

𝑰

𝑪 𝒛

Tokenizer

𝒯

Encoder

ℰ

Quantizer

𝒬

Codebook

𝑪

𝑥

𝒛

Decoder

𝒟

1

1

3

2

6

1

5

6

5

4

4

3

2

6

3

2

Generation

෨

𝑰

𝑪 ො 𝒛

ො 𝒛

challenges, researchers have focused on improving the codebook. ViT-VQGAN [47] adopts a ViTbased [9] autoencoder to create more expressive code vectors. SeQ-GAN [12] improves the perceptual loss and decoder to balance between semantic compression and detail preservation. SQVAE [40] improves VQ-VAE with stochastic quantization and a trainable posterior categorical distribution. VQ-WAE [44] builds upon SQ-VAE by encouraging the discrete representation to be a uniform distribution via a Wasserstein distance. HQ-VAE [39] employs random re-initialization of inactive code vectors. CVQ-VAE [55] selects encoded features as anchors to update dead codes. VQ-KD [27] adopts knowledge distillation instead of image reconstruction as the objective to train VQ-VAE. LFQ [49] and FSQ [25] adopt bounded scalar quantization techniques from neural compression to harness the potential of extra-large codebooks.

Furthermore, several works explore the potential of multiple codebooks. VQ-VAE-2 [32] extends VQ-VAE to a multi-scale hierarchical organization. RQ-VAE [21] and MoVQ [54] aim to represent each feature as a stack of tokens, where RQ-VAE adopts an iterative way to factorize features into a series of residuals and MoVQ models features across multiple channels via specialized modulation.

Token-based Image Generation. Inspired by the success of GPT [2, 30, 31], VQ-VAE and most of its derivative works [10, 54] adopt AR transformers to model the token sequence. This approach leverages techniques from text generation to enhance IG performance. However, the decoding time of AR models scales linearly with the length of the token sequence. To accelerate decoding, MaskGIT [5] introduces a bidirectional transformer, referred to as the NAR proposal network.

Given the versatility of token-based modeling, both AR and NAR proposal networks can be easily extended to conditional IG scenarios. For instance, VQGAN uses a class token as the condition in its AR proposal network for class-conditional IG. With an NAR proposal network, MUSE [4] adopts text embeddings to predict masked image tokens in Text-to-Image generation.

3 Token-Based Image Generation

We start with the two-stage IG framework in Sec. 3.1. Subsequently, Sec. 3.2 details the architecture and training protocol for the tokenizers under consideration. Sec. 3.3 explains the evaluation benchmark. Sec. 3.4 further outlines our main observations derived from the IN-1k experiments. Lastly, we validate the observations under different settings in Sec. 3.5.

3.1 Two-Stage Image Generation

We illustrate the two-stage IG framework in Fig. 2. Given an image I ∈ R H × W × 3 , the encoder E converts this image into a feature map x ∈ R h × w × d , where ( h, w ) = ( H/f,W/f ) and f is a downsample factor. Let a codebook C be a set of N code vectors { c i } N i =1 ∈ R N × d , where each code vector c i ∈ R d corresponds to a specific code i . A quantizer Q then maps x into a sequence

Encoder

ℰ

Decoder

𝒟

Image

𝑰

Reconstruction

෠

𝑰

𝑥1,1

⋯

𝑥𝑖,𝑗

⋯

𝑥ℎ,𝑤

𝑐

𝑧 1,1

⋯

𝑐

𝑧 𝑖,𝑗

⋯ 𝑐𝑧ℎ,𝑤

𝑐

1

𝑐𝑁

𝑥𝑖,𝑗

𝑐

2

ℒ𝒬

ℒrec

Vector Quantization

(a) VQGAN

Encoder

ℰ

Decoder

𝒟

-0.9

-1.2

-0.2

𝑥1,1

⋯

𝑥𝑖,𝑗

⋯

𝑥ℎ,𝑤

-1.0

-0.5

-0.0

𝑐

𝑧 1,1

⋯

𝑐

𝑧 𝑖,𝑗

⋯ 𝑐𝑧ℎ,𝑤

Finite  Scalar  Quantization

Image

𝑰

Reconstruction

෠

𝑰

ℒrec

-0.5

0.0

0.5

1.0

𝑥 𝑖,𝑗,2

𝑥 𝑖,𝑗,3

𝑥 𝑖,𝑗,1

(b) FSQ

𝑥1,1

𝒯

⋯

𝑥

𝑖,𝑗

𝒯

⋯

𝑥ℎ,𝑤

𝒯

Encoder

ℰ

Teacher

𝒯′

𝑥1,1

⋯

𝑥𝑖,𝑗

⋯

𝑥ℎ,𝑤

𝑐

𝑧 1,1

⋯

𝑐

𝑧 𝑖,𝑗

⋯ 𝑐𝑧ℎ,𝑤

ℒrec

Vector Quantization

Image

𝑰

𝑐

1

𝑐𝑁

𝑥𝑖,𝑗

𝑐

2

ℒ𝒬

(c) VQ-KD

of codes z = { z i } L i =1 , where L = h × w defines the sequence length and z i is an integer that falls within the range [1 , N ] . Let c z i denote the code vector that corresponds to code z i . Similarly, C ( z ) = { c z i } L i =1 ∈ R L × d represents a sequence of code vectors associated with the code sequence z . The encoder E , quantizer Q , and codebook C collectively form an image tokenizer T .

The proposal network P models the distribution over z , where the distribution is denoted as p ( z ) . Early proposal networks are implemented as an AR transformer, which sequentially models p ( z i | z 1: i -1 ) and formulates p ( z ) as ∏ h × w i =1 p ( z i | z 1: i -1 ) . While the AR transformers can be trained in parallel, it has to sequentially decode z during inference, which renders it inefficient. Therefore, NAR proposal networks are being prevalent [5], which typically adopt bidirectional transformers to model z . We denote the two types of proposal networks as P AR and P NAR, respectively.

Finally, a decoder D maps the code vectors to the pixel space. In training, D takes C ( z ) as input and learns to reconstruct the original image I as ˆ I ∈ R H × W × 3 . In inference, a sequence of codes ˆ z is sampled from p ( z ) , translated to C (ˆ z ) , and then fed into D to generate an image ˜ I .

3.2 Image Tokenizers

In this paper, we focus on three types of image tokenizers: VQGAN [10], FSQ [25], and VQKD [27].

Let x i,j ∈ R d be a vector in the feature map x . As shown in Fig. 3a, to quantize x i,j , the VQGAN tokenizer looks up the codebook C for the closest code vector in terms of Euclidean distance:

Since the quantization process is non-differentiable, VQGAN adopts the Straight-Through Estimator (STE) [1] to optimize the encoder E , which copies gradients from C ( z ) to x . As a result, the codebook C receives no gradient. To optimize C , VQGAN introduces a quantization loss L Q :

where sg [ · ] denotes the stop-gradient operation and β is the loss weight. The first term is the codebook loss, which optimizes the codebook. The second term is the commitment loss to make sure the encoder E commits to a code vector [41]. Therefore, the overall loss for VQGAN is defined as:

where L rec is the reconstruction loss between image I and reconstruction ˆ I , which includes ℓ 1 loss, perceptual loss, and adversarial loss.

Based on VQGAN, FSQ introduces a simpler image tokenizer, without the need for codebook lookup and quantization loss. As shown in Fig. 3b, FSQ adopts finite scalar quantization to quantize

each channel of x i,j into a finite set of scalars. Since the quantization process involves no trainable parameter, FSQ can be trained with solely the reconstruction loss L rec ( I , ˆ I ) .

Unlike VQGAN and FSQ, which are designed for IG, VQ-KD was originally presented in BEiT v2 to provide supervision for IU models. As shown in Fig. 3c, VQ-KD is trained to reconstruct the feature map x T encoded by a pretrained teacher T ′ . Formally, the reconstruction loss is defined as:

where cos( · , · ) represents cosine similarity and σ is a feature adapter. σ is implemented as a decoder, which maps C ( z ) to the same feature space as x T .

In this study, we examine VQ-KD using four types of pretrained teachers, including fullysupervised, text-supervised, contrastive, and Masked Image Modeling (MIM). We use VQ-KDCLIP and VQ-KDDINO to represent VQ-KD tokenizers trained with CLIP [29] and DINO [3] teachers, respectively. VQ-KDMAE and VQ-KDViT represent tokenizers trained with MAE [13] and ViT [9] teachers. The latter two teachers are pretrained on IN-1k utilizing a ViT-B/16 architecture.

3.3 Benchmark

We detail how we fairly compare different tokenizers for token-based IG here.

For each tokenizer, we train a proposal network P and a decoder D to constitute an image generator. In training, the tokenizer is frozen to ensure fairness. Thus, P and D can be trained in parallel. We follow VQGAN [10] to train the AR proposal network and the decoder. The NAR proposal network is trained following MAGE [22]. Implementation details can be found in Appendix B.

Our benchmark adopts various metrics to comprehensively evaluate the image tokenizers. Given an image tokenizer, we assess the effectiveness of its encoding process by evaluating the codebook usage. To assess the generative capabilities of the image tokenizers, we evaluate IS [34] and FID [14] on the generated images ˜ I . We assess the reconstruction capabilities of the image tokenizers by reporting the reconstruction FID (rFID). In addition, we present the PPL scores to appraise the AR modeling proficiency of the image tokenizers. A low PPL score implies that P AR easily models z . Details about the evaluation metrics can be found in Appendix C.

3.4 Main Observation

We evaluate the class-conditional IG performance of VQGAN, FSQ, and VQ-KD tokenizers on IN-1k. The results in Tab. 1 leads to the following observations.

VQ-KD significantly enhances generation quality over VQGAN. Equipped with either AR or NAR proposal networks, VQ-KD tokenizers consistently outperform VQGAN and FSQ, as evidenced by superior FID and IS metrics. In particular, VQ-KDViT attains an FIDAR of 11 . 40 and an FIDNAR of 8 . 45 , both less than half of those from VQGAN ( 24 . 11 FIDAR and 20 . 03 FIDNAR).

Tab. 2 presents a system-level comparison between VQ-KDCLIP and other class-conditional IG models on IN-1k at a resolution of 256 × 256 . With a 1.4B AR proposal network, VQ-KDCLIP achieves an FID of 4 . 10 , surpassing prior AR, NAR, and several VQ-based diffusion models.

The superiority of VQ-KD is irrelevant to the quantization operation and codebook usage. Both VQ-KDCLIP and FSQ record 100 . 0% codebook usage and 4 . 96 rFID, but VQ-KDCLIP achieves lower FIDAR and higher ISAR scores. Moreover, VQ-KD proves robustness to high codebook usage, with the PPL metric of most VQ-KD tokenizers surpassing VQGAN. In contrast, FSQ lags behind VQGAN in terms of PPL, suggesting that the high codebook usage of FSQ hinders P AR from modeling the code sequence z . As demonstrated in Sec. 4.1, this difference is likely due to the rich semantics in the VQ-KD feature map.

Tokenizers with stronger semantic understanding tend to deliver superior IG performance. Considering the FID and IS metrics, we find that VQ-KD tokenizers with supervised teachers (CLIP and ViT) consistently surpass those with unsupervised teachers (DINO and MAE). While VQ-KDDINO achieves the lowest rFID and PPL, its 13 . 15 FIDAR is worse than VQ-KDCLIP ( 11 . 78 ) and VQ-KDViT ( 11 . 40 ). This trend can be attributed to the superior capability of supervised models in capturing semantics compared to the unsupervised ones.

3.5 Further Verification

The superiority of VQ-KD holds across proposal networks. As seen in Tab. 1, all VQ-KD tokenizers surpass VQGAN and FSQ in the FIDNAR and ISNAR metrics. In particular, VQ-KDViT scores the lowest FIDNAR at 8 . 45 and VQ-KDCLIP scores the highest ISNAR at 121 . 33 . In contrast, VQGAN only achieves an FIDNAR of 20 . 03 and an ISNAR of 48 . 30 . It is also worth mentioning that VQ-KDCLIP and VQ-KDViT show slightly better performance than VQ-KDDINO and VQ-KDMAE, further supporting our conclusion that superior semantic understanding in supervised models plays a significant role in enhancing the quality of IG.

The superiority of VQ-KD holds across datasets. We conduct unconditional IG experiments on the MS-COCO dataset [23], which contains images of greater complexity than IN-1k. As demonstrated in Tab. 3, VQ-KDDINO achieves a FIDAR metric of 17 . 55 , significantly outperforming VQGAN( 38 . 43 ) and FSQ ( 44 . 64 ). Since the ViT teacher in VQ-KDViT is pretrained on IN-1k, the rFID and FIDAR metrics of VQ-KDViT are slightly inferior to VQ-KDDINO. Note that the PPL metric of VQGAN is misleadingly favorable, due to its low codebook usage ( 2 . 4% ).

The superiority of VQ-KD holds across tasks. Text-to-Image (T2I) experiments are conducted on the MS-COCO Captions dataset [6]. As shown in Tab. 3, the FIDT2I metric of VQ-KD tokenizers range from 11 . 17 to 15 . 60 , while VQGAN and FSQ only achieves 24 . 11 and 23 . 36 , respectively.

4 Analysis

In this section, we analyze tokenizers based on feature reconstruction from various perspectives.

4.1 Codebook Visualization

We delve into the superiority of VQ-KD by visualizing its codebook. From IN-1k, we randomly choose four categories: golden retriever, pirate ship, valley, and sea anemone. For each image belonging to these categories, we deploy t-SNE [42] to project the feature map x and the code vectors C ( z ) into a two-dimensional space. x is colored according to the image category and C ( z ) is illustrated in black. As depicted in Fig. 4, the feature space of VQ-KD shows superior organization compared to VQGAN. In the VQ-KD feature space, x from the same category are grouped together. This implies that each code in the VQ-KD codebook conveys clear semantics. As codes with similar semantics are likely to present concurrently in an image, it becomes easier for the proposal network to model the code sequence z . Conversely, each code in the VQGAN codebook is shared by multiple categories, resulting in semantics ambiguity. Hence, as illustrated in Tab. 1, the PPL metric for VQGAN is higher than VQ-KD, even though its codebook usage is considerably lower.

VQGAN

60

codebook

60

codebook

goldenretriever

goldenretriever

40

pirate ship

valley

40

pirate ship

valley

seaanemone

seaanemone

20

20

0

0

-20

-20

-40

-40

-60

-40

-20

0

20

40

60

-60

-40

-20

0

20

40

60

4.2 Clustering Pretrained Models as Tokenizers

To better harness the semantics in IU encoders, we propose a straightforward pipeline that transforms IU encoders into tokenizers via feature clustering. Given a pretrained IU model T ′ , we employ it to encode the feature map x T and subsequently utilize a clustering approach [55] to acquire N clusters. The cluster centroids constitute a codebook C . T ′ remains frozen during training, which significantly accelerates the training process. As shown in Tab. 4, ClusterViT presents 13 . 40 FIDAR, 10 . 58 FIDT2I, and 0 . 245 CLIP score on MS-COCO, outperforming all tokenizers in Tab. 3. This suggests that pretrained models with simple feature clustering can become good tokenizers. However, the cluster-based tokenizers behave worse in terms of rFID, since they encode little appearance detail in x T , which is essential for exact reconstruction. As a result, their FID and IS metrics on IN-1k are marginally weaker than those of their VQ-KD counterparts.

4.3 Scaling Up the Proposal Network

We examine the IG performance of tokenizers with a large-scale proposal network. Following VQGAN, we adopt GPT2 XL as P AR, which comes with 1 . 4 B parameters. In line with Tab. 1, VQ-KDViT leads with 9 . 23 FIDAR, while VQ-KDCLIP achieves the highest IS metric at 150 . 63 . Upon comparing with Tab. 1, tokenizers with stronger IU capabilities exhibit less improvement in the FIDAR metric. For instance, VQ-KDMAE improves significantly from 26 . 85 to 17 . 11 , while VQ-KDCLIP reveals a marginal enhancement from 11 . 78 to 11 . 27 . This suggests that a smallscale P AR is sufficient for tokenizers with strong IU capabilities, whereas those with weaker IU abilities benefit from a largescale P AR.

4.4 Influence of Strong Data Augmentation

We investigate the impact of strong data augmentation on the AR modeling performance of tokenizers. Specifically, we employed a strong random crop, where the crop scale ranges from 0 . 08 to 1 . 0 , introducing greater variability into the training data. As shown in Tab. 5, all tokenizers exhibit worse FIDAR metrics than their counterparts in Tab. 1. Interestingly, tokenizers with stronger IU capabilities demonstrate greater robustness to the strong data augmentation. For instance, VQ-KDViT experiences a minor increase in FIDAR of just 1 . 92 (from 11 . 40 to 13 . 32 ), whilst VQ-KDMAE records a considerable leap of 9 . 78 (from 26 . 85 to 36 . 63 ).

4.5 Large Teacher Models in VQ-KD

We incorporate OpenCLIP [17] models of varying sizes as teacher models to train the VQ-KD tokenizers. As illustrated in Tab. 6, the FIDAR metric sees a reduction from 10 . 31 to 8 . 70 when the size of the OpenCLIP model escalates from ViTL/14 to ViT-G/14. Given that larger OpenCLIP models inherently possess stronger IU capabilities, these findings further corroborate the superiority of image tokenizers with more potent IU capabilities.

Original

VQGAN

FSQ

Cluster

4.6 Codebook Size and Dimension

The size and dimension of the codebook exert a significant influence on the IG performance of tokenizers [10, 47]. Tab. 7a showcases the performance of VQ-KDCLIP with varying codebook sizes. Large codebooks aid the tokenizers in representing fine-grained semantics, contributing to a consistent decrease in the rFID metric from 6 . 59 to 4 . 53 . The IS metric also shows favor towards larger codebooks, with size 2 14 leading to the highest IS metric of 131 . 51 . However, choosing the correct code from a large codebook is harder than from a small codebook, hindering P AR from achieving lower FID scores with larger codebooks.

Tab. 7b demonstrates the influence of codebook dimension. High-dimensional codes carry more information but lead to lower codebook usage. As a result, the rFID metric initially drops from 4 . 96 to 4 . 64 , then increases drastically to 6 . 80 . Similar to Tab. 7a, the FID and IS metrics favor different codebook dimensions. FID favors 128 -dimensional codebooks, where codebook usage is relatively low. In contrast, IS favors 32 -dimensional codebooks, possibly due to a superior diversity.

4.7 Qualitative Analysis

The reconstruction quality of various tokenizers is demonstrated in Fig. 5. Original images are displayed in the first row. Regions where VQGAN and FSQ fail to reconstruct are highlighted with red boxes. In contrast, VQ-KD reconstructions are visually more accurate. Since the IU encoder in Cluster emphasizes encoding semantics over visual details, Cluster fails to preserve all visual details during reconstruction. Nonetheless, the reconstruction results of Cluster still appear more natural than VQGANand FSQ, especially in the highlighted areas. Fig. 6 and Fig. 7 further illustrates the AR and NAR generation results, showcasing the superior visual performance of VQ-KD and Cluster.

VQGAN

FSQ

Cluster

VQGAN

FSQ

Cluster

5 Conclusion

In this paper, we show that image understanding (IU) models can be useful in image generation (IG). Through comprehensive studies, observe that the VQ-KD tokenizers significantly enhance generation quality over VQGAN, irrelevant of the quantization operation and codebook utilization. Within the VQ-KD tokenizers, stronger IU capabilities tends to deliver superior IG performance. Further verification shows that the superiority of VQ-KD holds across proposal networks, datasets, and tasks. Lastly, we analyze VQ-KD from multiple angles, including clustering pretrained models as tokenizers, scaling up the proposal network, influence of strong data augmentation, large teacher models in VQ-KD, and codebook size and dimensions.

References

Yoshua Bengio, Nicholas L´ eonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv:1308.3432 , 2013.

Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In NeurIPS , 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Herv´ e Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In ICCV , 2021.

Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, MingHsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: Text-To-Image Generation via Masked Generative Transformers. In ICML , 2023.

Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked Generative Image Transformer. In CVPR , 2022.

Xinlei Chen, Hao Fang, Tsung-yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C Lawrence Zitnick. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv preprint arXiv:1504.00325 , 2015.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A largescale hierarchical image database. In CVPR , 2009.

Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. In NeurIPS , volume 11, pages 8780-8794, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR , 2021.

Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for High-Resolution Image Synthesis. In CVPR , 2021.

R. Gray. Vector quantization. IEEE ASSP Mag. , 1(2), 1984.

Yuchao Gu, Xintao Wang, Yixiao Ge, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis. arXiv preprint arXiv:2212.03185 , 2022.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ ar, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In CVPR , 2022.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In NeurIPS , 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In NeurIPS , 2020.

Drew A. Hudson, Daniel Zoran, Mateusz Malinowski, Andrew K. Lampinen, Andrew Jaegle, James L. McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. SODA: Bottleneck Diffusion Models for Representation Learning. arXiv preprint arXiv:2311.17901 , 2023.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. https://doi.org/10.5281/zenodo. 5143773 , 2021.

Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. In ICLR , 2015.

Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. ICLR , 2014.

Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos´ e Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, Yong Cheng, Ming-Chang Chiu, Josh Dillon, Irfan Essa, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, David Ross, Grant Schindler, Mikhail Sirotenko, Kihyuk Sohn, Krishna Somandepalli, Huisheng Wang, Jimmy Yan, Ming-Hsuan Yang, Xuan Yang, Bryan Seybold, and Lu Jiang. VideoPoet: A Large Language Model for Zero-Shot Video Generation. arXiv preprint arXiv:2312.14125 , 2023.

Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook Shin Han. Autoregressive Image Generation using Residual Quantization. In CVPR , 2022.

Tianhong Li, Huiwen Chang, Shlok Kumar Mishra, Han Zhang, Dina Katabi, and Dilip Krishnan. MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis. In CVPR , 2023.

Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll´ ar. Microsoft COCO: Common Objects in Context. In ECCV , 2014.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2019.

Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite Scalar Quantization: VQ-VAE Made Simple. In ICLR , 2024.

Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava. Diffusion Models Beat GANs on Image Classification. arXiv preprint arXiv:2307.08702 , 2023.

Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers. arXiv preprint arXiv:2208.06366 , 2022.

Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion Autoencoders: Toward a Meaningful and Decodable Representation. In CVPR , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML , volume 139, 2021.

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training. 2018.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. arXiv preprint arXiv:2007.07582 , 2019.

Ali Razavi, A¨ aron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. In NeurIPS , 2019.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR , 2022.

Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In NeurIPS , 2016.

Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learning using Nonequilibrium Thermodynamics. In ICML , 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In ICLR , 2020.

Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In NeurIPS , 2019.

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In CVPR , 2016.

Yuhta Takida, Yukara Ikemiya, Takashi Shibuya, Kazuki Shimada, Woosung Choi, Chieh-Hsin Lai, Naoki Murata, Toshimitsu Uesaka, Kengo Uchida, Wei-Hsiang Liao, and Yuki Mitsufuji. HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes. arXiv preprint arXiv:2401.00365 , 2023.

Yuhta Takida, Takashi Shibuya, Wei Hsiang Liao, Chieh Hsin Lai, Junki Ohmura, Toshimitsu Uesaka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji. SQ-VAE: Variational Bayes on Discrete Representation with Self-annealed Stochastic Quantization. Proc. Mach. Learn. Res. , 162, 2022.

Aaron Van Den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS , 2017.

Laurens Van Der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. JMLR , 9, 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurIPS , 2017.

Tung-Long Vuong, Trung Le, He Zhao, Chuanxia Zheng, Mehrtash Harandi, Jianfei Cai, and Dinh Phung. Vector Quantized Wasserstein Auto-Encoder. ICML , 2023.

Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. DatasetDM: Synthesizing Data with Perception Annotations Using Diffusion Models. In NeurIPS , 2023.

Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models. In NeurIPS , 2023.

Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-Quantized Image Modeling With Improved Vqgan. In ICLR , 2022.

Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. TMLR , 2022.

Lijun Yu, Jos´ e Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong, Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language Model Beats Diffusion - Tokenizer is Key to Visual Generation. arXiv preprint arXiv:2310.05737 , 2023.

Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, ShangWen Li, Susan Zhang, Gargi Ghosh, Yaniv Taigman, Maryam Fazel-Zarandi, Asli Celikyilmaz, Luke Zettlemoyer, and Armen Aghajanyan. Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning. arXiv preprint arXiv:2309.02591 , 2023.

Manlin Zhang, Jie Wu, Yuxi Ren, Ming Li, Jie Qin, Xuefeng Xiao, Wei Liu, Rui Wang, Min Zheng, and Andy J. Ma. DiffusionEngine: Diffusion Model is Scalable Data Engine for Object Detection. arXiv preprint arXiv:2309.03893 , 2023.

Zijian Zhang, Zhou Zhao, and Zhijie Lin. Unsupervised Representation Learning from Pretrained Diffusion Probabilistic Models. In NeurIPS , volume 35, 2022.

Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing Text-to-Image Diffusion Models for Visual Perception. In ICCV , 2023.

Chuanxia Zheng, Jianfei Cai, Long Tung Vuong, and Dinh Phung. MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation. In NeurIPS , 2022.

Chuanxia Zheng and Andrea Vedaldi. Online Clustered Codebook. In CVPR , 2023.

A Datasets

The experiments are conducted on two image datasets: ImageNet-1k (IN-1k) [7] and MSCOCO [23]. The IN-1k dataset contains approximately 1 . 28 million training images and 50 , 000 validation images across 1 , 000 diverse categories. The MS-COCO dataset comprises 82 , 783 images for training and 40 , 504 for validation. Each image is annotated with several captions.

For a given image, we first resize its shorter side to s pixels, where s symbolizes the input size. Subsequently, a central crop is performed to derive an image fragment sized s × s pixels. Our default augmentation strategy incorporates a random crop (between 0.8 and 1.0) partnered with random horizontal flipping.

B Implementation Details

The image tokenizers under consideration generate token sequences of length 256 upon a 256 × 256 input image. All tokenizers remain frozen throughout the training of the decoder and the proposal networks. Experiments are performed using 8 A100 80 GB GPUs. The approximate training times for the VQ-KD tokenizer is around 30 hours, the decoder requires roughly 68 hours, while the AR proposal network and NAR proposal network necessitate about 29 hours and 72 hours, respectively. In total, a single experiment takes approximately 200 hours training.

The CNN-based VQGAN tokenizers, with 27 . 9 M parameters, are trained following the identical procedure employed for decoders. The codebook dimension of VQGAN is 256 . FSQ levels L are set to (8 , 8 , 5 , 5 , 5) , equivalent to codebook size 8 , 000 .

As per BEiT v2 [27], an AdamW optimizer is utilized to train the VQ-KD tokenizers. The learning rate warms up linearly to 10 -4 for 25 , 000 steps, subsequently decaying to 10 -5 under a cosine schedule. Unless specifically stated, VQ-KD tokenizer is trained with an input size of 224 × 224 and codebook dimension of 32 .

VQGAN [10] is adopted for training both the decoder and the AR proposal networks. Both D and P AR training span 260 , 000 steps with a collective batch size of 96 for IN-1k and 24 for MS-COCO. The decoder is a CNN-based VQGAN decoder, consisting of 40 . 5 M parameters. The decoders utilize the Adam [18] optimizer with learning rates set at 5 . 4 × 10 -5 , β 1 = 0 . 5 , and β 2 = 0 . 9 . Their discriminators are also trained via Adam optimizer, employing learning rates of 4 . 32 × 10 -4 , while keeping the β constants identical. Subsequent training of AR proposal networks relies on the AdamW [24] optimizer with β 1 = 0 . 9 , β 2 = 0 . 98 , and a 0 . 2 weight decay. An initial learning rate of 10 -4 is set, after which it decays to 0 on a cosine schedule. The AR proposal network is a GPT-2 Medium [31], with 335 Mparameters.

We follow MAGE [22] for training NAR proposal networks. P NAR is trained for 300 epochs with a collective batch size of 512 on ImageNet-1k. NAR proposal networks are trained with the AdamW optimizer with β 1 = 0 . 9 , β 2 = 0 . 95 , and a 0 . 05 weight decay. The learning rate warms up linearly to 3 × 10 -4 throughout 10 epochs before decaying to 0 following a cosine schedule. The encoder of P NAR is a ViT-B/16, with 86 Mparameters.

C Evaluation

Codebook usage is defined as the proportion of codes from the codebook that have been used at least once when encoding the dataset. A low value for codebook usage might be an indication of the 'codebook collapse' issue.

IS provides a measure of both the fidelity and diversity of ˜ I . However, IS significantly relies on the classification capabilities of a pretrained Inception-v3 model [38]. Complex images are likely to be misinterpreted as lacking fidelity by IS. Therefore, we limit the use of IS to IN-1k experiments only.

To circumvent the limitations of IS, FID computes the statistical distance in the Inception-v3 feature space between the real images I and the generated images ˜ I . A lower FID score indicates that ˜ I is statistically similar to I .

rFID is defined as the FID between I and their reconstructed counterparts ˆ I . Obtaining a low rFID score requires that the image tokenizer encode sufficient visual details within the codes C ( z ) to enable accurate reconstruction by the decoder.

The PPL score is defined as:

where z denotes a sequence of codes offered by the tokenizer, L represents the length of z , and p ( z ) embodies the distribution modeled by the AR proposal network P AR.

Both reconstruction and AR modeling serve as two pivotal capabilities in an image generator. We anticipate that these metrics will lead to a more thorough insight into the generative capacities of image tokenizers.

D Limitations

The VQ-KD tokenizers are designed to mimic the IU encoders, yielding superior quantitative results compared to traditional tokenizers like the VQGAN and FSQ. Nonetheless, qualitative analysis suggest that the VQ-KD may modify visual details during the pixel reformation process, thereby posing challenges for tasks such as image editing.

E Broader Impacts

This paper explores the question how might image understanding (IU) models aid image generation (IG) tasks . We envision that our findings will motivate research on image tokenizers and prompt the community to reconsider the correlation between IU and IG.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The contributions and scope of the paper are summarized in the last paragraph of sec. 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of the work is discussed in appendix D.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The model architecture is demonstrated in sec. 3.1. The implementation details are illustrated in appendix B. The code is released at https://github.com/ magic-research/vector_quantization .

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is released at https://github.com/magic-research/vector_ quantization . The data used are publicly available.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Implementation details are illustrated in appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Given the massive amount of experiments conducted in this paper, providing error bars would be computationally prohibitive.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Compute resources are described in appendix B.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper conforms with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Potential societal impacts of the work are discussed in appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper mainly uses the BEiT v2 repository, which is cited in the paper and mentioned in the code.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows that image understanding models can be helpful in image generation and that a stronger IU model can result in better IG performance. To demonstrate it, this paper sets multiple experiments from many perspectives (e.g. different datasets and codebook sizes) and gives out reasonable analysis.\n\nSTRENGTHS:\n1.This paper has done experiments across various perspectives, making a solid demonstration.\n\n2.The results of different VQ-KD models outperforming VQGAN and FSQ support this paper's standpoint.\n\n3.The visualization of codebooks is very concise, clear and well drawn.\n\n4.The idea of training a clustering based tokenizer from an IU model is interesting, which can also illustrate the standpoint.\n\nWEAKNESSES:\n1.The visualization of reconstruction of VQ-KD and Cluster seems worse than that of VQGAN and FSQ in both paper and appendix from my point of view. Maybe more visual results should be included.\n\n2.This paper seems to claim that IU model can help IG, but this paper's experiments and analysis are limited to token based tasks. I think this paper should include more IG methods like diffusion, VAE with some little experiments or analysis if possible. Or this paper should emphasize the range is limited to token based tasks.\n\n3.Various experiments have been done across different settings and perspectives, but I think this paper lacks of methods.Besides VQGAN, FSQ and VA-KD, more related methods should be included.\n\n4.The structure of this paper is a little confusing, with experiments, analysis and descriptions mixed together.\n\nQUESTIONS:\n1.Why doesn't a larger teacher model always lead to better results? Following the theory of this paper, ViT-G/14 should perform best.\n\n2.The experiment about different codebook sizes and dimensions seems to reveal that when the IU model gets stronger, the IG may not always be better. Can I think that to make better IG results, we don't have to use an IU model with too strong capability? I need detailed analysis.\n\nLIMITATIONS:\nFrom this paper's limitations, although VQ-KD performs better due to its stronger capability in image understanding, it is not suitable for all scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the connection between image understanding (IU) and image generation (IG). The authors introduce a token-based IG framework and a novel feature reconstruction objective for tokenizer training. They introduce an extra feature reconstruction loss to distill semantic knowledge from pretrained IU model to tokenizer. They demonstrate superior IG performance with such tokenizers having strong IU capabilities, as evidenced by various metrics, datasets, tasks, and networks.\n\nSTRENGTHS:\n- The paper is well-written, systematically organized, and straightforward to follow.\n- The idea to merge the image understanding and generation model is sensible and is shown to be effective.\n- Plenty of quantitative experiments are carried out to validate the superiority of the proposed method.\n\nWEAKNESSES:\n1. **Excessive and unjustified claims.**\n    - An important contribution stated by this work is that it is the first to combine IU with IG, which might not hold true. I am not very familiar with related works in this specific direction, but I can mention one from ICLR 2024 [a]. They not only use IU as a tokenizer but also reduce the token number in a dynamic way.\n1. **Observations in Sec. 3.4 needs further justification** For example,\n    - More experiments are needed to verify that *The superiority of VQ-KD is irrelevant to the quantization operation and codebook usage.*\n    - The observation that *Tokenizers with stronger semantic understanding tend to deliver superior IG performance* are not fully supported  by the experimental results.\n\n1. **More visualization results are necessary.**\n    - The paper is almost completely backed up by quantitative results. This may not be favorable for an image generation method. Many more examples are needed to enhance the persuasiveness of the experiment. For example, results on MSCOCO (Tab. 3), qualitative comparison with different IU backbone, etc.\n\n1. **Lack of qualitative analysis for the learned codebook.**\n    - There is no visualization of what is encoded in the VQ-KD and why it is superior to the original VQGAN. The authors could refer to [a] for some examples.\n\n1. **Others**\n    - Please use `Tab. X` and `Fig. X` in main text.\n    - What is $D$ is used in Eq (4)? \n\n[a] Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization. Jin et al. ICLR2024.\n\nQUESTIONS:\nIn summary, this is a work featuring good motivation and quantitative results. However, I have the following concerns:\n- As I am not familiar  with specific related works, I am unable to offer a competent evaluation of the quantitative results and assess the significance of these outcomes. I would prefer to see the comments from more expert reviewers before arriving at a final decision.\n- The comparison with related works appears insufficient. There is almost no comparison made with state-of-the-art image generation models.\n- Why are there so few qualitative results? For an image generation paper, visual examples are of considerable importance.\n- Visual analysis of learned codebook is also missing.\n\nLIMITATIONS:\nAlthough the authors discuss the limitations in a few sentences, I believe more should be expounded.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework that leverages the rich semantic capabilities of Image Understanding models for Image Generation tasks. By employing a token-based generation framework and a feature reconstruction objective, the paper trains tokenizers capable of mapping images into token sequences. Compared to traditional pixel reconstruction methods, this approach demonstrates good performance across various metrics.\n\nSTRENGTHS:\nThe paper's strength lies in its innovative fusion of image understanding with image generation frameworks, offering a fresh perspective that transcends traditional pixel-based approaches. The originality is evident in the novel application of feature reconstruction for tokenizer training, drawing knowledge from pre-trained image understanding models, which is a creative synthesis of existing concepts. The significance of this work is underscored by its potential to redefine tokenizer research and enhance generation performance across various metrics, as demonstrated through rigorous empirical validation on the ImageNet-1k dataset. The clarity of the paper is reflected in its well-structured presentation and articulate explanation of complex concepts, making the methodology and results easily comprehensible. Overall, the quality of the research, its original problem formulation, and the clarity of the findings contribute to the paper's impact and broad applicability.\n\nWEAKNESSES:\n1. The paper lacks theoretical support for the VQ-KD tokenizer.\n2. The experiments are limited to the ImageNet and COCO dataset, which may not be sufficient to demonstrate the effectiveness of the method. Consider adding higher resolution and higher quality datasets, like the LAION-Aesthetics\n3. There is a lack of comparison and discussion regarding computational efficiency and computational load.\n\nQUESTIONS:\nPlease see weaknesses\n\nLIMITATIONS:\n1. Experiments dataset is limited \n2. Generalizability was not enough explored and discussed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores using image understanding (IU) models to aid image generation (IG) performance. To verify the hypothesis, the authors focus on the different tokenizers and introduce feature reconstruction (VQ-KD) as a training objective for image tokenizers, distilling knowledge from pre-trained IU encoders. This paper compares VQ-KD tokenizers with conventional methods like VQGAN and FSQ across various metrics, datasets, tasks, and proposal networks. The results show that tokenizers with strong IU capabilities, particularly VQ-KD, outperform traditional methods.\n\nSTRENGTHS:\n+ The paper is the first to demonstrate that image understanding models can substantially enhance image generation.\n+ To verify the hypothesis, the analyses of different tokenizers and training objectives are reasonable. \n+ The authors conduct extensive experiments across different metrics, datasets, tasks, and network architectures to validate their findings.\n+ The paper gives detailed visualizations and analyses that help in understanding why the proposed model outperforms existing methods.\n+ Experiments show the usage of VQ-KD tokenizers outperforms conventional methods, achieving state-of-the-art results on several benchmarks.\n\nWEAKNESSES:\n- While the paper examines four types of IU encoders, it could benefit from investigating a broader range of IU models to strengthen its conclusions.\n- The paper could include more detailed ablation studies to isolate the impact of different components in the VQ-KD approach.\n- As in the limitation, maybe it is better to introduce the fidelity-related metrics to indicate that this work has a better generation ability, but may result in lower fidelity.\n- Some visual comparisons are not obvious. It is better to give a close-up to see the details.\n\nQUESTIONS:\n1. How does the proposed method perform on more diverse and challenging datasets beyond ImageNet-1k and MS-COCO, such as medical imaging or satellite imagery? This may show the generalization ability of the proposed model.\n2. The description of VQGAN 'pixel reconstruction' seems not suitable, as VQGAN adopts the perceptual loss which may also introduce the semantic constraint.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-17"}