Image Understanding Makes for A Good Tokenizer for Image Generation
Authors: Luting Wang
Yang Zhao
Zijian Zhang
Jiashi Feng
Si Liu
Bingyi Kang
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Tokenizer
Vector Quantization
Image Generation
Primary area: generative_models

Abstract: Modern image generation (IG) models have been shown to capture rich semantics valuable for image understanding (IU) tasks. However, the potential of IU models to improve IG performance remains uncharted. We address this issue using a token-based IG framework, which relies on effective tokenizers to project images into token sequences. Currently, **pixel reconstruction** (e.g., VQGAN) dominates the training objective for image tokenizers. In contrast, our approach adopts the **feature reconstruction** objective, where tokenizers are trained by distilling knowledge from pretrained IU encoders. Comprehensive comparisons indicate that tokenizers with strong IU capabilities achieve superior IG performance across a variety of metrics, datasets, tasks, and proposal networks. Notably, VQ-KD CLIP achieves $4.10$ FID on ImageNet-1k (IN-1k). Visualization suggests that the superiority of VQ-KD can be partly attributed to the rich semantics within the VQ-KD codebook. We further introduce a straightforward pipeline to directly transform IU encoders into tokenizers, demonstrating exceptional effectiveness for IG tasks. These discoveries may energize further exploration into image tokenizer research and inspire the community to reassess the relationship between IU and IG. The code is released at https://github.com/magic-research/vector_quantization.

Image Understanding Makes for A Good Tokenizer for Image Generation
Luting Wang*t
Yang Zhao
Zijian Zhang'
Jiashi Feng'
Si Liu
Bingyi Kang" 18
ByteDance
Abstract
Modern image generation (IG) models have been shown to capture rich semantics valuable for image understanding (IU) tasks. However, the potential of IU mod- els to improve IG performance remains uncharted. We address this issue using token-based IG framework, which relies on effective tokenizers to map images into token sequences. Currently, pixel reconstruction (e.g.. VQGAN) dominates the training objective for tokenizers. In contrast, our approach adopts the fea- ture reconstruction objective, where tokenizers are trained by distilling knowledge from pretrained IU encoders. Comprehensive comparisons indicate that tokeniz- ers with strong IU capabilities achieve superior IG performance across a variety of metrics, datasets, tasks, and proposal networks. Notably, VQ-KDeLIP achieves 4.10 FID on ImageNet-1k. Visualization suggests that the superiority of VQ-KD can be partly attributed to the rich semantics within the VQ-KD) codebook. We further introduce straightforward pipeline directly transform IU encoders into tokenizers, demonstrating exceptional effectiveness for tasks. These discover- ies may energize further exploration into image tokenizer research and inspire the community to reassess the relationship between IU and IG. The code is released at http:///ght..ttooooooooar.arrrrr
Introduction
Image understanding (IU) and image generation (IG) have been the core pursuits of computer vision research for a long time. Thanks the progress in generative models [15 37(48) and network ar- chitectures [9]3). IG has witnessed remarkable advancements in recent years. These advancements spurred extensive research on leveraging powerful IG models for IU tasks (Fig. D Studies have shown that IG models can benefit IU tasks in various ways, including data augmentation through
synthetic data generation [45| 46 51), improved representation learning [16 28 52], and utilizing
intermediate features from IG models for solving perception tasks 226 53 However, the reciprocal
question remains largely uncharted: how might IU models aid IG tasks?
The primary focus of this paper lies in the AutoRegressive (AR) IG framework, which is gaining considerable attention for its excellence in generating high-quality images and videos [20 48
This framework operates in a two-stage process. The first stage learns tokenizer to map images into sequences of discrete tokens. Subsequently, the second stage trains a proposal network to model the token sequences. As underlined by prior research 4144) the quality of the tokenizers significantly influences overall performance. Meanwhile, tokenizers and IU encoders adhere to a similar structure as they both aim to map images into latent representations, either discrete or continuous. As a result, the loken-based IG framework provides an optimal environment for investigating the relationship between IU and IG. Through comprehensive studies, we demonstrate
*Work done during an internship at Byte Email: wangluting buaa.cdu.cn *Coresponding authors. Email: liusi @ buaa. cdu.en. bingyikang @ bytedance.com 'qqual contribution.
$Project lead. lcad.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Representation Leamnigg
Image Generation (IG)
Image Understanding (IU)
Data A Agmentation Intermediate Feature
Generator
Predictor
2 N0.
y E Rhxwxc
Figure I:
Extensive studies have tried to adopt IG models for IU. However, few attempts have been made to use IU models in IG.
that existing IU models from representation learning can be useful in generative models, even if they are not specifically designed for the IG task.
Our study involves training three components within the AR framework: tokenizer, decoder, and proposal network. Traditionally, pixel reconstruction has been the dominant objective for training tokenizers, such as VQGAN [10] and FSQ [25] To the best of our knowledge, we are the first to sys- tematically demonstrate that feature reconstruction (VQ-KD (27) achieves better IG performance, This approach distills knowledge from pretrained IU encoders to tokenizers. Therefore, the training strategy of the IU encoder is crucial for the performance of the tokenizer. In this regard, we inves- tigate four representative IU encoders: ViT (9), CLIP [29). DINO [3] and MAE [13]. Following VQGAN [10). we train decoders to restore pixel values from discrete tokens, and proposal networks (AR NAR) that model the distribution of image tokens. The models are then evaluated using
various metrics, including codebook usage, Fréchet Inception Distance (FID) [4]. Inception Score
(IS) 34 perplexity (PPL), etc.
Initially, we compare the above tokenizers on IN-1k for class-conditional IG. VQ-KD achieves 4. 4.10 FID AR. outperforming VQGAN (15.78 FID by a large margin. FSQ experiments confirm that the superiority of VQ-KD is not solely attributable to the specific quantization operation or high code- book usage. More generally, VQ-KD consistently outperforms across different proposal networks, datasets, and tasks.
We analyze VQ-KD from multiple perspectives. By visualizing the codebook, we discover that codes from VQ-KD carry more semantics than VQGAN, which makes them easier to model and subsequently improve the IG quality. Building upon this insight, we propose straightforward pipeline to efficiently transform IU encoders into tokenizers, outperforming VQ-KD on the MS- COCO dataset. We also find that tokenizers with weaker IU capabilities require larger proposal networks for effective AR modeling and show less robustness to variations in the training images. Finally, we conduct qualitative analysis to present the visual results.
In sum, the key insights from our study include the following: 1) This research is the first demon- strate that IU models can substantially enhance IG through VQ-KD; 2) Tokenizers with strong IU capabilities consistently outperform conventional VQGAN- methods across various metrics, datasets, tasks, and network architectures; 3) The VQ-KD codebook encapsulates more semantics than VQGAN, contributing to the superiority of VQ-KD in IG.
We believe these findings can benefit future research on image tokenizers and provoke further dis- cussion the relationship between IU and IG.
2 Related Work
Image Tokenization. Vector Quantization (VQ) 11 is originally developed for data compression.
To eircumvent posterior collapse in the VAE [19 framework, VQ-VAE (4T] adopts VQ for image tokenization. Subsequently, VQGAN [T0] introduces adversarial and perceptual losses to enhance the quality of the generated images.
Vanilla VQGAN suffers from limitations like low codebook usage, limited semantic representa- tion ability, and the trade-off between modeling efficiency and image quality. To address these
VQQKD was initially proposed for image pretraining.

---

TokenizerJ
Proposa Network-
Frocker
AR/ NAR
DeooderD
C(2)
Image I
Generation T
Codebook
Quantizer
2
DeoderD
C(z)
Reconstruction
Figure 2: The token-based IG framework. Solid and dashed lines represent training and inference pipelines, respectively. During training, the tokenizer T tokenizes an image into discrete codes z. A proposal network P is trained to model the distribution p(z), while a decoder D learns to reconstruct During inference, we sample codes from P, which guides 'D to perform generation.
challenges, researchers have focused on improving the codebook. ViT-VQGAN 47 adopts a VIT- based autoencoder to create more expressive code vectors. SeQ-GAN I2 improves the per- ceptual loss and decoder to balance between semantic compression and detail preservation. SQ- VAE [40] improves VO-VAE with stochastic quantization and trainable posterior categorical dis- tribution. VQ-WAE (44) builds upon SQ-VAE by encouraging the discrete representation to a uniform distribution
Wasserstein distance. HQ-VAE (39 employs random re-initialization of inactive code vectors. CVQ- VAE (55) selects encoded features as anchors to update dead codes. VQ-KD (27) adopts knowledge distillation instead of image reconstruction the objective train VQ-VAE. LFQ 49] and FSQ [25] adopt bounded scalar quantization techniques from neural com- pression harness the potential of extra-large codebooks.
Furthermore, several works explore the potential multiple codebooks. VQ-VAE- VAE-2 32 extends
VQ-VAE to multi-scale hierarchical organization. RQ-VAE (21) and MoVQ [54] aim represent each feature as a stack of tokens, where RQ- VAE adopts an iterative way to factorize features into series of residuals and Mo models features across multiple channels via specialized modulation.
330311, 31J. VQ- and most Token-based Image Generation. Inspired by the success of GPT 22 30
of its derivative works (1) adopt AR transformers model the token sequence. This approach leverages techniques from text generation to enhance IG performance. However, the decoding time of AR models scales linearly with the length of the token sequence. To accelerate decoding, MaskGIT [5] introduces a bidirectional transformer, referred to as the NAR proposal network.
Given the versatility of token- based modeling, both AR and NAR proposal networks can be easily extended to conditional scenarios. For instance, VQGAN uses class token as the condition in its AR proposal network for condiiconiiiional IG. With an NAR proposal network, MUSE (4] adopts text embeddings to predict masked image tokens in Text-to-Image generation.
Token-Based Image Generation
We start with the two-stage IG framework in Sec. 3.1 Subsequently, Sec. 3.2 details the architecture and training protocol for the tokenizers under consideration. e the evaluation bench- mark. Sec. (344 further outlines our main observations derived from the IN-1k experiments. Lastly, we validate the observations under different settings Sec. 3.5
3.1 Two-Stage Image Generation
We illustrate the two-stage IG framework in Fig. 2 Given an image I E BHxxXxx the encoder E converts this image into feature map æ Ii wxxx where (h, w) (H/f, W/f) and f is downsample factor. Let codebook C be set of N code vectors JN E R Nxd where each code vector E d corresponds to specific code i. A quantizer Q then maps into a sequence

---

Vasur Qurtialiss
Pazze Sokaa
Ouattionion
Veclor Quitaatinn
Lyec
Ray
Encoder
DecaderDD
Encoder
Decoder
Encoder
Teacher
Erue
Image
Reconstructon.
Image
Reconstraction
Image
(a) VQGAN
(b) FSQ
(c) VQ-KD
Figure 3: The architecture and training objective of different image tokenizers.
of codes { where L h x defines the sequence length and is an integer that falls within the range [1, N.. Let Cz, denote the code vector that corresponds to code 2, Similarly, C(z) (ce, 1 E R 1.xd represents a sequence of code vectors associated with the code sequence 2. The encoder ε, quantizer Q, and codebook C collectively form an image tokenizer T.
The proposal network P models the distribution p(z). Early proposal networks are implemented as an AR transformer, which sequentially models p(=1211--1) and formulates p(z) as I p(=)/21=--1). While the AR transformers can be trained in parallel. it has to sequentially decode during inference, which renders it inefficient. Therefore, NAR proposal networks are being prevalent (5). which typically adopt bidirectional transformers to model z. We denote the two types proposal networks as PAR and PNAR, respectively.
2, where the distribution denoted as
Finally, a decoder 'D maps the code vectors to the pixel space. In training, D takes C(z) input xw and learns to reconstruct the original image I as I "xx x3 In inference, sequence of codes is sampled from p(z), translated to C(2), and then fed into D generate image I.
3.2 Image Tokenizers
In this paper, we focus on three types of image tokenizers: KD (27].
VQGAN 10
FSQ 25
and VQ-
Let Rijj E Rd be a vector the feature map r. As shown Fig. 3a to quantize Tıjj, the VQGAN tokenizer looks up the codebook C for the closest code vector in terms of Euclidean distance:
Zij arg minl|mag - cz||2-
Since the quantization process dinfeiiferentiable,, VQGAN adopts the Straight-Through Estimator (STE) to optimize the encoder ε, which copies gradients from C(z) to x. As a result, the codebook C receives no gradient. optimize C, VQGAN introduces a quantization loss Lo:
Le |sg[ C(2)1l3 B||x - sg( sg(C C(z)]l|,
where sg[:] denotes the stop-gradient operation and book loss, which optimizes the codebook. The second term is the commitment loss to make sure the encoder commits to a code vector [41]. Therefore. the overall loss for VQGAN is defined as:
loss weight. The first term code-
L= Lree iI,
where Lree is the reconstruction loss between image I and reconstruction which includes E loss, perceptual loss, and adversarial loss.
Based on VQGAN, FSQ introduces simpler image tokenizer, without the need for codebook lookup and quantization loss. As shown in Fig. 3b FSQ adopts finite scalar quantization quantize

---

Table Comparison between image tokenizers on IN-Ik.
Codebook Usage (%)
PAR
PNAR
Tokenizer T
rFID +
PPL +
FIDAR AR
ISAR
FIDNAR
ISNAR
VOGAN
4.9
5.09
116.75
24.11 11
39.52
20.03
48.30
FSQ
100.0
4.96
791.56
40.17
26.40
29.78
33.63
VQ-KDcLIP
100.0
4.96
53.73
11.78
128.18
9.51
121.33
VQ-KDviT 100.0
3.69
89.30
11.40
107.56
8.45
108.75
VQ-KDDINO 100.0
3.41
74.07
13.15
80.89
10.21
91.39
VQ-KDMAE 100.0
4.93
280.06
26.85
40.03
16.11
59.05
each channel øiy into a finite set of scalars. Since the quantization process involves no trainable parameter, FSQ can be trained with solely the reconstruction loss Lie (I, I).
Unlike VQGAN and FSQ, which are designed for IG, VQ-KD originally presented in BETT v2 to provide supervision for IU models. As shown in Fig. 3d VQ-KD is trained to reconstruct the feature map x² encoded by pretrained teacher T'. Formally, the reconstruction loss is defined
Lre = cos (a(C(z)),,²))
where cos(,;) :) represents cosine similarity and a a feature adapter. a is implemented as a decoder, which maps C(z) to the same feature space as x²"
In this study, we examine VQ-KD using four types of pretrained teachers, including fully- supervised, text-supervised, contrastive, and Masked Image Modeling (MIM). We use VQ-KDCLIP and VQ-KDDINo represent VQ-KD tokenizers trained with CLIP [29] and DINO (3) teachers, respectively. VQ-KD and VQ-KD represent tokenizers trained with MAE [13] and ViT 9 teachers. The latter two teachers are pretrained on IN-1k utilizing ViT-B/16 architecture.
3.3 Benchmark
We detail how we fairly compare different tokenizers for
IG here.
For each tokenizer, we train proposal network P and a decoder 'D to constitute an image generator. In training, the tokenizer frozen ensure fairness. Thus, P and D can be trained parallel. We follow VQGAN [10] to train the AR proposal network and the decoder. The NAR proposal network trained following MAGE 22). Implementation details can be found in Appendix B
Our benchmark adopts various metrics to comprehensively evaluate the image tokenizers. Given an image tokenizer, assess the effectiveness of its encoding process by evaluating the codebook
usage. To assess the generative capabilities of the image tokenizers, we evaluate IS [34] and FID [14 on the generated images I. We assess the reconstruction capabilities of the image tokenizers by reporting the reconstruction FID (rFID). In addition, we present the PPL scores appraise the AR
modeling proficiency of the image tokenizers. A low PPL score implies that PAR easily models z. Details about the evaluation metrics can be found in Appendix C
3.4 Main Observation
We evaluate the class-conditional IG performance VQGAN, FSQ, and VQ-KD tokenizers on IN-1k. The results in Tab. Tleads to the following observations.
VQ-KD significantly enhances generation quality over VQGAN. Equipped with either AR or NAR proposal networks, VQ-KD tokenizers consistently outperform VQGAN and FSQ, as evi- denced by superior FID and IS metries. In particular, VQ-KDvit attains an FIDAR of 11.40 and an FIDNAR of 8.45, both less than half of those from VQGAN (24.11 FIDAR and 20.03 FIDNAR).
Tab. presents system-level comparison between VQ-KDeLIP and other class-conditional IG mod- els on IN-Ik at resolution of 256 x 256. With 1.4B AR proposal network, VQ-KDeLp achieves an FID of 4.10, surpassing prior AR. NAR, and several VQ-based diffusion models.
5

---

The superiority of VQ-KD is irrel- evant to the quantization operation and codebook usage. Both VQ-KDcLIP and FSQ record 100.0% codebook usage and 4.96 rFID, but VQ-KD achieves lower FIDAR and higher IS scores. Moreover, VQ-KD proves robustness to
Table 2: System level comparison IN-1k.
Model
Architecture
#params
FID +
VQGAN 10
AR
1.4B
15.78
RQ-VAE 21
AR
1.4B
8.71
ViT-VQGAN
AR
1.7B
5.30
high codebook usage, with the PPL met-
MoVQ [54
NAR
307M
8.78
ric of most VQ-KD tokenizers surpass-
MaskGIT
NAR
227M
6.18
ing VQGAN. In contrast, FSQ lags be-
FSQ [25
NAR
227M
4.53
hind VQGAN in terms of PPL, suggest-
LDM-8-G 33 CVQ-VAE
Diffusion
506M
7.76
ing that the high codebook usage FSQ hinders PAR from modeling the code quence 2. As demonstrated in Sec. 1.1 this difference is likely due to the rich semantics in the VQ-KD feature map.
Diffusion
400M
6.87
VQ-KDeLI 27 (ours)
AR
1.4B
4.10
Tokenizers with stronger semantic understanding tend to deliver superior IG performance. Considering the FID and IS metrics, we find that VQ-KD tokenizers with supervised teachers (CLIP and ViT) consistently surpass those with unsupervised teachers (DINO and MAE). While VQ-KDDINO achieves the lowest rFID and PPL, its 13.15 FIDAR is worse than VQ-KDCLIP (11.78) and VQ-KD (11.40). This trend can be attributed to the superior capability of supervised models in capturing semantics compared to the unsupervised ones.
3.5 Further Verification
The superiority of VQ-KD holds across proposal networks. As seen in Tab. all VQ-KD tok- enizers surpass VQGAN and FSQ in the FIDNAR and ISNAR metrics. In particular, VQ-KDvar scores the lowest FIDNAR at 8.45 and VQ-KDCLIP scores the highest ISNAR at 121.33. In contrast, VQ- GAN only achieves an FIDNAR of 20.03 and an ISNAR of 48.30. It is also worth mentioning that VQ-KDerIP and VQ-KDvi show slightly better performance than VQ-KDpINo and VQ-KDMAE. further supporting our conclusion that superior semantic understanding in supervised models plays significant role in enhancing the quality of IG.
Table 3: Comparison between image tokenizers on MS-COCO. T21 experiments conducted the MS-COCO Captions dataset.
Codebook Usage (%)
PAR
Tokenizer T
rFID 4
FIDr21 +
PPL.
FIDAR +
VQGAN
2.4
16.21
47.89
38,43
24.11
FSQ
100.0
4.62
1040.02
44.64
23.36
VQ-KDCLIP
82.2
5.48
72.31
29.80
11.17
VQ-KD
100.0
3.70
117.10
23.51
15.49
VQ-KD
100.0
2.69
129.93
17.55
11.50
VQ-KDMAE 100.0
3.51
317.98
44.01
15.60
The superiority of VQ-KD holds across datasets. We conduct unconditional IG experiments the MS-COCO dataset [23]. which contains images of greater complexity than IN-Ik. As demon- strated in Tab. 3 VQ-KD achieves a FIDAR metric of 17.55, significantly outperforming VQ- GAN (38.43) 43) and FSQ (44.64). Since the ViT teacher in VQ-KDwir is pretrained on IN- 1k, the rFID and FID AR metrics of VQ-KDwi slightly inferior to VQ-KD Note that the PPL metric of VQGAN is misleadingly favorable, due to its low codebook usage (2.4%).
The superiority of VQ-KD holds across tasks. Text-to-Image (T21) experiments are conducted on the MS-COCO Captions dataset (6). As shown Tab. 3 the FIDra metric of VQ-KD tokenizers range from 11.17 to 15.60, while VQGAN and FSQ only achieves 24.11 11 and 23.36, respectively.

---

Analysis
In this section, we analyze tokenizers based on feature reconstruction from various perspectives.
4.1 Codebook Visualization
We delve into the superiority of VQ-KD by visualizing its codebook. From IN-Ik, we randomly choose four categories: golden retriever, pirate ship, valley, and sea anemone. For each image belonging to these categories, we deploy t-SNE [42] to project the feature map I and the code colored according the image category and C(z) is illustrated in black. As depicted in Fig. 4 the feature space of VQ-KD shows superior organization compared to VQGAN.
vectors C(z) into two-dimensional space.
the VQ-KD feature space, z from the same category are grouped together. This implies that each code in the VQ-KD codebook conveys clear semantics. As codes with similar semantics are likely to present concurrently an image, becomes easier for the proposal network to model the code sequence 2. Conversely, each code in the VQGAN codebook is shared by multiple categories, resulting in semantics ambiguity. Hence, as illustrated in Tab. [ the PPL metric for VQGAN is higher than VQ-KD, even though its codebook usage is considerably lower.
VOGAN
KD
60
gotden retrever prote ship valley sea anomone
golden retr ever pirace ship valley
(500 oremone
3
-10
Figure 4: Codebook visualization of VQGAN and VQ-KDvi- Best viewed in color.
4.2 Clustering Pretrained Models as Tokenizers
To better harness the semantics in IU encoders, we propose straightforward pipeline that transforms
IU encoders into tokenizers via feature clustering. Given a pretrained IU model T we employ
encode the feature map xT and subsequently utilize clustering approach (55) acquire N
clusters. The cluster centroids constitute codebook C. T'' remains frozen during training, which significantly accelerates the training process. As shown in Tab. 4. Clustervı presents 13.40 FIDAR- 10.58 FIDra and 0.245 CLIP score on MS-COCO, outperforming all tokenizers Tab. 3 This suggests that pretrained models with simple feature clustering can become good tokenizers. How- ever, the cluster-based tokenizers behave worse in terms of rFID, since they encode little appearance detail in x which is essential for exact reconstruction. As result, their FID and IS metrics on IN- are marginally weaker than those of their VQ-KD counterparts.
Table 4: Performance of cluster- based tokenizers.
IN-
MS-COCO
Encoder
rFID +
FIDAR 4
FIDNAR +
rFID 4
FIDAR +
FID +
CLIP
9.13
14.81
12.83
7.28
20.03
12.82
VIT
4.78
11.87
8.59
4.59
13.40
10.58
DINO
5.16
14.53
11.23
4.02
25.35
7.66
MAE
15.15
38.72
34.26
10.08
62.17
18.85

---

4.3 Scaling Up the Proposal Network
We examine the IG performance of to- kenizers with a large-scale proposal net- work. Following VQGAN, we adopt GPT- XL as PAR. which comes with 1.4B pa- rameters. In line with Tab, [ VQ-KDwr leads with 9.23 FRDAR, while VQ-KDeuI achieves the highest IS metric at 150.63.
Table 5: AR modeling with a large-scale proposal net- work or strong data augmentation.
GPT-2 XI
Strong Aug.
Tokenizer T
FIDAR +
ISAR
FIDAR +
ISAR
Upon comparing with Tab. [ tokenizers with stronger IU capabilities exhibit less improvement the FIDAR metric. For in- stance, VQ-KDMAE improves significantly from 26.85 to 17.11, while VQ-KDeLIp re- veals marginal enhancement from 11.78 to 11.27. This suggests that a small- scale PAR sufficient for tokenizers with strong IU capabilities, whereas those with weaker IU abilities benefit from large-
VQGAN
17.13 25.87
59.19
32.76
26.76
FSQ
49.59
52.17
18.09
VQ-KDCLIP
11.27
150.63
15.24
90.41
VQ-KD Vir
9.23
146.00
13.32
81.44
VQ-KDDINO
9.50
120.26
19.39
52.74
VQ-KDMAE
17.11
69.03
36.63
28.49
ClustercuIP
14.00
110.26
17.22
83.72
scale PAR-
4.4 Influence of Strong Data Augmentation
We investigate the impact of strong data augmentation the AR modeling performance of tok- enizers. Specifically, we employed a strong random crop, where the crop scale ranges from 0.08 to 1.0, introducing greater variability into the training data. As shown Tab, [5] all tokenizers exhibit worse FRD metrics than their counterparts in Tab. Interestingly, tokenizers with stronger IU ca- pabilities demonstrate greater robustness to the strong data augmentation. For instance, VQ-KD experiences a minor increase in FIDAR of just 1.92 (from 11.48 11.40 to 13.32), whilst VQ- records a considerable leap of 9.78 (from 26.85 36.63)
4.5 Large Teacher Models in VQ-KD
We incorporate OpenCLIP (17 models of varying sizes as teacher models to train the VQ-KD tokenizers. As illustrated in Tab. 6 FIDAR metric sees a redue- tion from 10.31 to 8.70 when the size of OpenCLIP model escalates from ViT- to14 ViT-G/14. Given that larger Open- CLIP models inherently possess stronger IU capabilities, these findings further cor- roborate the superiority of image tokeniz- with more potent IU capabilities.
Table 6: Effect of different teachers in VQ-KD.
OpenCLIP
rFID
PPL
FID
ISAR
ViT-L/14
4.03
80.56
10.31
146.21
ViT-H/14
3.60
97.32
9.64
161.13
ViT-G/14
3.80
77.79
8.70
152.71
Table 7: Effect of codebook size and dimension. Experiments are conducted on VQ-KDeLP
(a) Codebook size.
(b) Codebook dimension.
Codebook
Codebook C
rFID
FIDAR
ISAR
rFID 1
FIDAR AR
IS
Size (loga)
Usage (%)
Dim
Usage (%)
10
100.0
6.59
11.65
114.90
16
100.0
4.94
12.19
124.71
11
100.0
5,99
10.98
119.72
32
100.0
4.96
11.78
128.18
12
100.0
5.64
11.71
123.42
64
97.4
4.64
11.00
126.60
100.0
4,96
11.78
128.18
128
89.6
5.03
10.50
119.78
93.7
4.53
11.61
131.51
256
48.7
6.80
12.08
103.07

---

Original
VOGAN
FSQ
VQ-DD
Cluster
Figure 5: Reconstruction results of different image tokenizers.
4.6 Codebook Size and Dimension
The size and dimension of the codebook exert a significant influence the IG performance of kenizers [10] Tab. 7a showcases the performance of VQ-KDCLP with varying codebook sizes. Large codebooks aid the tokenizers in representing fine-grained semantics, contributing to consis- tent decrease in the rFID metric from 6.59 to 4.53. The IS metric also shows favor towards larger codebooks, with size 24 leading to the highest IS metric of 131.51. However, choosing the correct code from large codebook is harder than from a small codebook, hindering PAr from achieving lower FID scores with larger codebooks.
Tab. 76 demonstrates the influence of codebook dimension. figh-dimensional codes carry more information but lead lower codebook usage. As a result, the rFID metrie initially drops from 4.96 to 4.64, then increases drastically to 6.80. Similar to Tab. 7a the FID and IS metrics favor different codebook dimensions. FID favors 1 128-dimensional codebooks, where codebook usage is relatively low. In contrast, IS favors 12-dimensional codebooks, possibly due to superior diversity.
4.7 Qualitative Analysis
The reconstruction quality of various tokenizers is demonstrated in Fig. 5 Original images are displayed in the first row. Regions where VQGAN and FSQ fail reconstruct are highlighted with red boxes. In contrast, VQ-KD reconstructions are visually more accurate. Since the IU encoder in Cluster emphasizes encoding semantics over visual details, Cluster fails to preserve all visual details during reconstruction. Nonetheless, the reconstruction results of Cluster still appear more natural than VQGANand FSQ, especially the highlighted areas. Fig. fig. Fig, further illustrates the AR and NAR generation results, showcasing the superior visual performance of VQ-KD and Cluster.

---

VQGAN
FSQ
VQ-KD
Cluster
Figure 6: Class-conditional AR generation results of different image tokenizers.
VQGAN
FSQ
VQ-DD
Cluster
Figure 7: Unconditional NAR generation results of different image tokenizers.
Conclusion
In this paper, show that image understanding (IU) models be useful in image generation (IG). Through comprehensive studies, observe that the VQ-KD tokenizers significantly enhance generation quality over VQGAN, irrelevant of the quantization operation and codebook utilization. Within the VQ-KD tokenizers, stronger IU capabilities tends to deliver superior IG performance. Further verification shows that the superiority of VQ-KD holds across proposal networks, datasets, and tasks. Lastly, we analyze VQ-KD from multiple angles, including clustering pretrained models tokenizers. scaling up the proposal network, influence of strong data augmentation, large teacher models in VQ-KD, and codebook size and dimensions.
10

---

Acknowledgement
This research supported in part by National Key R&D Program of China (2022ZD01 15502), National Natural Science Foundation of China (NO. 62122010, U23B2010), Zhejiang Provincial Natural Science Foundation of China under Grant No. LDT23F02022F02, Key Research and Devel- opment Program of Zhejiang Province under Grant 2022C01082, "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No. 2024C01161).
References
[1] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv: 2013.
[2] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learn- ers. NeurlPS, 2020.
[3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging Properties in Self-Supervised Vision Transformers. In ICCV, 2021.
[4] Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming- Hsuan Yang, Kevin Murphy, William T. Freeman, Michael Rubinstein, Yuanzhen Li, and Dilip Krishnan. Muse: TexI-To-lmage Generation via Masked Generative Transformers. In ICML, 2023.
[5] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. MaskGIT: Masked Generative Image Transformer. In CVPR, 2022.
[6] Xinlei Chen, Hao Fang, Tsung-yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C Lawrence Zitmick. Microsoft COCO Captions: Data Collection and Evaluation Server. arXiv preprint arXiv: 1500:0504000055, 2015.
[7] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. CVPR, 2009.
[8] Prafulla Dhariwal and Alex Nichol. Diffusion Models Beat GANs on Image Synthesis. In NeurIPS, volume 11, pages 8780-8794, 2021.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2021.
[10] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming Transformers for High- Righ-Resolution Image Synthesis. In CVPR, 2021.
[11] R. Gray. Vector quantization. IEEE ASSP Mag., 1(2), 1984.
[12] Yuchao Gu, Xintao Wang. Yixiao Ge, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Re- thinking the Objectives of Vector- citto-Quantized Tokenizers for Image Synthesis. arXiv preprint (tt::22//.031888 2022.
[13] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. In CVPR, 2022.
[14] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to Local Nash Equilibrium. In NeurIPS, 2017.
[15] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In NewrIPS, 2020.

---

[16] Drew A. Hudson, Daniel Zoran, Mateusz Malinowski, Andrew K. Lampinen, Andrew Jaegle, James L. McClelland, Loic Matthey, Felix Hill, and Alexander Lerchner. SODA: Bottleneck Diffusion Models for Representation Learning. arXiv preprint arXiv:231 10.::23/1.122 171001,, 2023.
[17] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Ro- han Taori, Achal Dave, Vaishaal Shankar. Hongseok Namkoong, John Miller, Hannaneh Ha- jishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip.
5143773 2021.
[18] Diederik Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization, In ICLR, 2015.
[19] Diederik P. Kingma and Max Welling, Auto-encoding variational bayes. ICLR, 2014.
[20]
Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, Yong Cheng, Ming-Chann Chiu, Josh Dillon, Irfan Essa, Agrim Gupta, Meera Hahn, Anja Hauth, David Hendon, Alonso Martinez, David Minnen, David Ross, Grant Schindler, Mikhail Sirotenko, Kihyuk Sohn, Kr- ishna Somandepalli, Huisheng Wang, Jimmy Yan, Ming-Hsuan Yang. Xuan Yang. Bryan Sey- bold, and Lu Jiang. VideoPoet: A Large Language Model for Zero-Shot Video Generation. arXiv preprint arXiv:2312.1412. 2023.
[21] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook Shin Han. Autoregressive Image Generation using Residual Quantization. In CVPR, 2022.
[22] Tianhong Li, Huiwen Chang, Shlok Kumar Mishra, Han Zhang, Dina Katabi, and Dilip Kr- ishnan. MAGE: MAsked Generative Encoder to Unify Representation Learning and Image Synthesis. In CVPR, 2023.
[23] Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft COCO: Common Objects in Context. In ECCV, 2014.
[24] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[25]
Fabian Mentzer, David Minnen, Eirikur Agustsson, and Michael Tschannen. Finite Scalar Quantization: VQ- VAE Made Simple. In ICLR, 2024.
[26) Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava. Diffusion Models Beat GANs on Image Classification. arXiv preprint arXiv:2307.08702 2023.
[27] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. BETT v2: Masked Image Modeling with Vector-Quantiza Visual Tokenizers. arXiv preprint arXiv:2208. 06366, 2022.
[28] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Diffusion Autoencoders: Toward a Meaningful and Decodable Representation. In CVPR, 2022.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In ICML, volume 139, 2021.
[30] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Understanding by Generative Pre-Training. 2018
[31] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. arXiv preprint arXiv:2007.00 07582, 2019.
[32] Ali Razavi, Aäron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-V-.-2. In NeurIPS, 2019.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR, 2022.
[34] Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved Techniques for Training GANs. In NeurIPS, 2016.
[35] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Un- supervised Learning using Nonequilibrimm Thermodynamics. In ICMI. 2015.
12

---

[36] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. In ICLR, 2020.
[37] Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In NeurlPS, 2019.
[38] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the Inception Architecture for Computer Vision. In CVPR, 2016.
[39] Yuhta Takida, Yukara lkemiya, Takashi Shibuya, Kazuki Shimada, Woosung Choi, Chieh-Hsin Lai, Naoki Murata, Toshimitsu Uesaka, Kengo Uchida, Wei-Hsiang Liao, and Yuki Mitsu- fuji. HQ-VAE: Hierarchical Discrete Representation Learning with Variational Bayes. arXiv preprint arXiv:2401. 00365, 2023.
[40] Yuhta Takida, Takashi Shibuya, Wei Hsiang Liao, Chich Hsin Lai, Junki Ohmura. Toshim- itsu Uesaka, Naoki Murata, Shusuke Takahashi, Toshiyuki Kumakura, and Yuki Mitsufuji. SQ-VAE: Variational Bayes Discrete Representation with Self-annealed Stochastic Quanti- zation. Proc. Mach. Learn. Res., 162, 2022.
[41]
Aaron Van Den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurlPS, 2017.
[42]
Laurens Van Der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. JMLR, 9, 2008.
[43]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurlPS, 2017.
[44]
Tung-Long Vuong, Trung Le, He Zhao, Chuanxia Zheng, Mehrtash Harandi, Jianfei Cai, and Dinh Phung. Vector Quantized Wasserstein Auto-Encoder, ICML, 2023.
[45] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. DatasetDM: Synthesizing Data with Perception Anno- tations Using Diffusion Models. In NeurlPS, 2023.
[46] Lihe Yang, Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models. In NeurlPS, 2023.
[47] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-Quantized Image Modeling With Improved Vggan. In ICLR, 2022.
[48] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Va- sudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling Autoregressive Models for Content-Rich Text-to-Image Generation. TMLR, 2022.
[49] Lijun Yu, José Lezama, Nitesh B. Gundavarapu, Luca Versari, Kihyuk Sohn. David Minnen, Yong Cheng. Agrim Gupta, Xiuye Gu, Alexander G. Hauptmann, Boqing Gong. Ming-Hsuan Yang, Irfan Essa, David A. Ross, and Lu Jiang. Language Model Beats Diffusion Tokenizer is Key to Visual Generation. arXiv preprint arXiv:2310.0573: 2023.
[50] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, Candace Ross, Adam Polyak, Russell Howes, Vasu Sharma, Puxin Xu, Hovhannes Tamoyan, Oron Ashual, Uriel Singer, Shang- Wen Li. Susan Zhang, Gargi Ghosh, Yaniv Taigman, Maryam ZaraldZ Asli Celikyilmaz, Luke Zettlemoyer, and Armen Aghajanyan. Scaling Autoregressive Multi-Modal Models: Pre- training and Instruction Tuning. arXiv preprint arXiv:2300.000 029991 2023.
[51] Manlin Zhang, Jie Wu, Yuxi Ren, Ming Li, Jie Qin, Xuefeng Xiao, Wei Liu, Rui Wang, Min Zheng, and Andy J. Ma. DifflssionEngine: Diffusion Model is Scalable Data Engine for Object Detection. arXiv preprint arXiv:2309. 00893,08830...0.. 2023.
[52] Zijian Zhang, Zhou Zhao, and Zhijie Lin. Unsupervised Representation Learning from Pre- trained Diffusion Probabilistic Models. In NeurlPS, volume 35, 2022.
[53] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing Text-to-Image Diffusion Models for Visual Perception. In ICCV, 2023.
[54] Chuanxia Zheng, Jianfei Cai, Long Tung Vuong, and Dinh Phung. MoVQ: Modulating Quan- tized Vectors for
Image Generation. In NewrlPS, 2022.
[55] Chuanxia Zheng and Andrea Vedaldi. Online Clustered Codebook. In CVPR, 2023.
13

---

A Datasets
The experiments are conducted on two image datasets: ImageNet-Ik (IN-Ik) 7 and MS- COCO 23 The IN-IK dataset contains approximately 1.28 million training images and 50, 000 validation images across 1, 000 diverse categories. The MS-COCO dataset comprises 82, 783 im- ages for training and 40, 504 for validation. Each image is annotated with several captions.
For a given image, we first resize its shorter side to s pixels, where symbolizes the input size. Subsequently, central crop is performed to derive an image fragment sized x pixels. Our default augmentation strategy incorporates a random crop (between 0.8 and 1.0) partnered with random horizontal flipping.
B Implementation Details
The image tokenizers under consideration generate token sequences of length 256 upon a 256 x 256 input image. All tokenizers remain frozen throughout the training of the decoder and the proposal networks. Experiments are performed using 8 A100 80GB GPUs. The approximate training times for the VQ-KD tokenizer is around 30 hours, the decoder requires roughly 68 hours, while the AR proposal network and NAR proposal network necessitate about 29 hours and 72 hours, respectively. In total, a single experiment takes approximately 200 hours training-
The CNN-based VQGAN tokenizers, with 27.9M parameters, are trained following the identical procedure employed for decoders. The codebook dimension of VQGAN is 256. FSQ levels set to (8, 8, 5, 5, 5),,, equivalent to codebook size 8, 000.
As per BEiT v2 (27), AdamW optimizer is utilized to train the VQ-KD tokenizers. The learning rate warms up linearly to 10 for 25, 000 steps, subsequently decaying to 10 under a cosine schedule. Unless specifically stated, VQ-KD tokenizer is trained with input size of 224 224 and codebook dimension of 32.
VQGAN [T0) is adopted for training both the decoder and the AR proposal networks. Both D and PAR training span 260, 000 steps with collective batch size of 96 for IN-1K and 24 for MS- COCO. The decoder is a CNN-based VQGAN decoder, consisting of 40.5M parameters. The decoders utilize the Adam 18 optimizer with learning rates set at 5.4 10-5, 3, 0.5, and B3 = 0.9. Their discriminators are also trained via Adam optimizer, employing learning rates of 4.32 x 10-4 while keeping the /3 constants identical. Subsequent training of AR proposal networks relies on the AdamW [24] optimizer with 31 0.9, 82 0.98. and 0.2 weight decay. An initial learning rate of 10 is set, after which decays to 0 on a cosine schedule. The AR proposal network is GPT-2 Medium (II. with 335M parameters.
We follow MAGE [2] for training NAR proposal networks. PNAR trained for 300 epochs with a collective batch size of 512 on ImageNet- kk. NAR proposal networks are trained with the AdamW optimizer with 31 = 0.9, 32 = 0.95, and a 0.05 weight decay. The learning rate warms up linearly to 3 x 10-4 throughout 10 epochs before decaying to O following a cosine schedule. The encoder of PNAR is ViT-B/16, with 86M parameters.
C Evaluation
Codebook usage is defined as the proportion of codes from the codebook that have been used at least once when encoding the dataset. A low value for codebook usage might be an indication of the "codebook collapse' issue.
IS provides measure of both the fidelity and diversity of i. However, IS significantly relies on the classification capabilities of a pretrained Inception-v3 model (38]. Complex images are likely to be misinterpreted as lacking fidelity by IS. Therefore, we limit the use of IS to IN-1k experiments only.
To circumvent the limitations of IS, FID computes the statistical distance the Inception-v3 feature space between the real images and the generated images Î. lower FID score indicates that Î is statistically similar to I.
14

---

rFID is defined as the FID between I and their reconstructed counterparts I. Obtaining a low rFID score requires that the image tokenizer encode sufficient visual details within the codes C(z) to enable accurate reconstruction by the decoder.
The PPL score is defined as:
L Σ
log pl (1)
PPL. = exp
(5)
L
where z denotes sequence of codes offered by the tokenizer, L represents the length of z, and p(z) embodies the distribution modeled by the AR proposal network PAR-
Both reconstruction and AR modeling serve as two pivotal capabilities in an image generator. We anticipate that these metries will lead to more thorough insight into the generative capacities of image tokenizers.
D Limitations
The VQ-KD tokenizers are designed to mimic the IU encoders, yielding superior quantitative results compared to traditional tokenizers like the VQGAN and FSQ. Nonetheless, qualitative analysis suggest that the VQ-KD may modify visual details during the pixel reformation process, thereby posing challenges for tasks such as image editing.
E Broader Impacts
This paper explores the question how might image understanding (IU) models aid image generation (IG) tasks. We envision that our findings will motivate research on image tokenizers and prompt the community to reconsider the correlation between IU and IG.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions and scope of the paper summarized in the last paragraph sec. 0
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in appendix D
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specificattin, asymptotic approximations only holding locally). The au- thors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't i acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theo- rems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be comple- mented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The model architecture demonstrated in sec. 3.1 The implementation details are illustrated in appendix B The code is released at https: //tp:://gtttt.....o com/
Justification:
seggicrreeeacchveccteee http://etoorruuuiiiaiiiiiii
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all sub- missions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results way produce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which thors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models. it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The code is released at haatts///gahuuu.coc////iireeeees quantization The data used are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
wttpg/uddde//odeSubm Comeeuumieiiiececcceeee ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Implementation details are illustrated in appendix B Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [No]
Justification: Given the massive amount of experiments conducted in this paper, providing error bars would be computationally prohibitive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should prefer- ably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources Question:
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Compute resources are described in appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't t make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics Ethics/https: http:////wrrppppppppppp//////aaa
Answer: [Yes]
Justification:
The paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential socictal impacts the work are discussed in appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact spe- cific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitiga- tion strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by re- quiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided relcasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper mainly uses the BEiT v2 repository, which is cited in the paper and mentioned in the code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use the pack- wwwrswittthhd..... com /ataastt has
age should be provided. For popular datasets, batasets..paperswtt wwwpppppppwaiieee curated licenses for some datasets. Their licensing guide can help determine the li- cense of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the per include the full text instructions given to participants and sereenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, cura- tion, or other labor should be paid at least the minimum wage the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equiva- lent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "xgnmipk9oT", "reviewer": "Reviewer_cC74", "summary": "This paper explores using image understanding (IU) models to aid image generation (IG) performance. To verify the hypothesis, the authors focus on the different tokenizers and introduce feature reconstruction (VQ-KD) as a training objective for image tokenizers, distilling knowledge from pre-trained IU encoders. This paper compares VQ-KD tokenizers with conventional methods like VQGAN and FSQ across various metrics, datasets, tasks, and proposal networks. The results show that tokenizers with strong IU capabilities, particularly VQ-KD, outperform traditional methods.", "questions": "1. How does the proposed method perform on more diverse and challenging datasets beyond ImageNet-1k and MS-COCO, such as medical imaging or satellite imagery? This may show the generalization ability of the proposed model.\n2. The description of VQGAN 'pixel reconstruction' seems not suitable, as VQGAN adopts the perceptual loss which may also introduce the semantic constraint.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The paper is the first to demonstrate that image understanding models can substantially enhance image generation.\n+ To verify the hypothesis, the analyses of different tokenizers and training objectives are reasonable. \n+ The authors conduct extensive experiments across different metrics, datasets, tasks, and network architectures to validate their findings.\n+ The paper gives detailed visualizations and analyses that help in understanding why the proposed model outperforms existing methods.\n+ Experiments show the usage of VQ-KD tokenizers outperforms conventional methods, achieving state-of-the-art results on several benchmarks.", "weaknesses": "- While the paper examines four types of IU encoders, it could benefit from investigating a broader range of IU models to strengthen its conclusions.\n- The paper could include more detailed ablation studies to isolate the impact of different components in the VQ-KD approach.\n- As in the limitation, maybe it is better to introduce the fidelity-related metrics to indicate that this work has a better generation ability, but may result in lower fidelity.\n- Some visual comparisons are not obvious. It is better to give a close-up to see the details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful feedback. Please note our top-level comment. Below we address specific questions.\n\n# Generalizability\n\nPlease see our top-level comment. We examine VQ-KD ConvNext and observe good image generation abilities. More types of IU encoders will be added to our revised manuscript. We also conduct VQGAN and VQ-KD CLIP experiments on medical imaging (SA-Med2D-20M) and satellite imagery (SATIN) datasets, where VQ-KD CLIP consistently outperforms VQGAN. Visualizations are shown in the attached PDF file. These results confirm the generalization ability of our conclusions across different models and datasets. \n\n# Ablation Study\n\nPlease refer to our top-level comment, where we replace the K-Means module in VQ-KD with the quantization loss used in VQGAN. A slight performance drop is observed, but the overall performance of VQ-KD still surpasses VQGAN by a large margin.\n\n# Fidelity-Related Metrics\n\nSome results in Tab. 7 of the main text shows that the generation ability of VQ-KD can surpass VQGAN even if its rFID metric is relatively lower. To further support this observation, we conduct experiments with a VQ-KD CLIP model that is not well-trained. As shown in the following table, while the rFID metric of this VQ-KD CLIP model is worse than VQGAN, its PPL and FID metrics still outperform VQGAN by a large margin.\n\n|  | rFID $\\downarrow$ | PPL $\\downarrow$ | FID $\\downarrow$ |\n|:-:|---|---|---|\n| VQGAN | 5.09 | 116.75 | 24.11 |\n| VQ-KD CLIP (codebook size 1024) | 6.59 | 21.28 | 11.65 |\n| VQ-KD CLIP (codebook dim 256) | 6.80 | 16.44 | 12.08 |\n| VQ-KD CLIP (not well-trained) | 5.26 | 62.77 | 11.79 |\n\n# Qualitative results\n\nPlease see our top-level comment. We present more qualitative results in the attached pdf file.\n\n# VQGAN description\n\nThanks for pointing out this ambiguity. We will clarify this in our revised manuscript to avoid confusion. VQGAN adopts perceptual loss to enhance the perceptual quality of reconstructed images. Given a real image $\\mathbf{I}$ and a reconstructed image $\\hat{\\mathbf{I}}$, the perceptual loss encodes both images and outputs the distance between the image features. While semantic constraint is introduced, the primary goal of perceptual loss is to achieve better pixel reconstruction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m5ZBjnwb9h", "reviewer": "Reviewer_aFBf", "summary": "This paper introduces a novel framework that leverages the rich semantic capabilities of Image Understanding models for Image Generation tasks. By employing a token-based generation framework and a feature reconstruction objective, the paper trains tokenizers capable of mapping images into token sequences. Compared to traditional pixel reconstruction methods, this approach demonstrates good performance across various metrics.", "questions": "Please see weaknesses", "limitations": "1. Experiments dataset is limited \n2. Generalizability was not enough explored and discussed", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper's strength lies in its innovative fusion of image understanding with image generation frameworks, offering a fresh perspective that transcends traditional pixel-based approaches. The originality is evident in the novel application of feature reconstruction for tokenizer training, drawing knowledge from pre-trained image understanding models, which is a creative synthesis of existing concepts. The significance of this work is underscored by its potential to redefine tokenizer research and enhance generation performance across various metrics, as demonstrated through rigorous empirical validation on the ImageNet-1k dataset. The clarity of the paper is reflected in its well-structured presentation and articulate explanation of complex concepts, making the methodology and results easily comprehensible. Overall, the quality of the research, its original problem formulation, and the clarity of the findings contribute to the paper's impact and broad applicability.", "weaknesses": "1. The paper lacks theoretical support for the VQ-KD tokenizer.\n2. The experiments are limited to the ImageNet and COCO dataset, which may not be sufficient to demonstrate the effectiveness of the method. Consider adding higher resolution and higher quality datasets, like the LAION-Aesthetics\n3. There is a lack of comparison and discussion regarding computational efficiency and computational load.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments. Please note our top-level comment. Below we address specific questions.\n\n# Theoretical Support\n\nWhile it is hard to theoretically prove the superiority of the VQ-KD tokenizer, we hypothesize that superiority is because each token generated by VQ-KD contains rich semantics, rather than simply appearance information. The hypothesis is supported by the finding that the PPL AR metric of most VQ-KD tokenizers are lower than VQGAN and FSQ in Tab. 1 of the main text. Lower PPL AR metric suggests that the proposal network can easily model the token sequence generated by VQ-KD. Moreover, Fig. 4 in the main text demonstrates that images belonging to the same category are encoded with similar tokens. Please also refer to Fig. 2 in the attached PDF file for a qualitative analysis of the semantics encoded within each VQ-KD token.\n\n# Generalizability\n\nPlease see our top-level comment. We perform experiments on challenging datasets including LAION-Aesthetics. Visualizations are shown in the attached PDF file. The results demonstrate that our conclusion generalizes well to various datasets.\n\n# Computation cost\n\nWe present the computation cost for each training stage of VQGAN and VQ-KD in the following tables. All experiments are conducted on IN-1k using 8 A100-80G GPUs. In sum, VQGAN takes 81 hours to train and VQ-KD CLIP takes 89 hours to train. The training cost for VQ-KD CLIP is only 10% higher than VQGAN.\n\n| VQGAN | #epochs | CUDA Memory (GB) | Training Time (hours) |\n|---|---|---|---|\n| Stage 1: Tokenizer | 20 | 25.6 | 42 |\n| Stage 2: AR Proposal Network | 20 | 17.9 | 39 |\n\n| VQ-KD | #epochs | CUDA Memory (GB) | Training Time (hours) |\n|---|---|---|---|\n| Stage 1: Tokenizer | 100 | 10.7 | 14 |\n| Stage 2: Pixel Decoder | 20 | 18.2 | 36 |\n| Stage 3: AR Proposal Network | 20 | 17.8 | 39 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XXtnuKZQRt", "reviewer": "Reviewer_3Kzg", "summary": "This work focuses on the connection between image understanding (IU) and image generation (IG). The authors introduce a token-based IG framework and a novel feature reconstruction objective for tokenizer training. They introduce an extra feature reconstruction loss to distill semantic knowledge from pretrained IU model to tokenizer. They demonstrate superior IG performance with such tokenizers having strong IU capabilities, as evidenced by various metrics, datasets, tasks, and networks.", "questions": "In summary, this is a work featuring good motivation and quantitative results. However, I have the following concerns:\n- As I am not familiar  with specific related works, I am unable to offer a competent evaluation of the quantitative results and assess the significance of these outcomes. I would prefer to see the comments from more expert reviewers before arriving at a final decision.\n- The comparison with related works appears insufficient. There is almost no comparison made with state-of-the-art image generation models.\n- Why are there so few qualitative results? For an image generation paper, visual examples are of considerable importance.\n- Visual analysis of learned codebook is also missing.", "limitations": "Although the authors discuss the limitations in a few sentences, I believe more should be expounded.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written, systematically organized, and straightforward to follow.\n- The idea to merge the image understanding and generation model is sensible and is shown to be effective.\n- Plenty of quantitative experiments are carried out to validate the superiority of the proposed method.", "weaknesses": "1. **Excessive and unjustified claims.**\n    - An important contribution stated by this work is that it is the first to combine IU with IG, which might not hold true. I am not very familiar with related works in this specific direction, but I can mention one from ICLR 2024 [a]. They not only use IU as a tokenizer but also reduce the token number in a dynamic way.\n1. **Observations in Sec. 3.4 needs further justification** For example,\n    - More experiments are needed to verify that *The superiority of VQ-KD is irrelevant to the quantization operation and codebook usage.*\n    - The observation that *Tokenizers with stronger semantic understanding tend to deliver superior IG performance* are not fully supported  by the experimental results.\n\n1. **More visualization results are necessary.**\n    - The paper is almost completely backed up by quantitative results. This may not be favorable for an image generation method. Many more examples are needed to enhance the persuasiveness of the experiment. For example, results on MSCOCO (Tab. 3), qualitative comparison with different IU backbone, etc.\n\n1. **Lack of qualitative analysis for the learned codebook.**\n    - There is no visualization of what is encoded in the VQ-KD and why it is superior to the original VQGAN. The authors could refer to [a] for some examples.\n\n1. **Others**\n    - Please use `Tab. X` and `Fig. X` in main text.\n    - What is $D$ is used in Eq (4)? \n\n[a] Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization. Jin et al. ICLR2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and thoughtful comments. Please note our top-level comment. Below we address specific questions.\n\n# Related works\n\nOur main conclusion is that IU models can aid IG tasks, which have not been explored before. While LaViT adopts a pretrained ViT encoder in the tokenizer, they did not perform in-depth analysis regarding the relationship between IU and IG. We will modify our claim and clarify that we are the first to demonstrate that IU models can substantially enhance IG through VQ-KD.\n\n# Main observations\n\nWe perform an ablation study on the quantization operation of VQ-KD and show the results in our top-level comment. The results suggest that VQ-KD can optimize the quantizer in the same way as VQGAN, while achieving better IG performance. \n\nAs for codebook usage, Tab. 7 (b) in the main text shows that when the codebook dimension of VQ-KD increases to 256, which is the same as VQGAN, the codebook usage of VQ-KD drops to 48.7%. However, VQ-KD still achieves 12.08 FID AR, which is significantly better than the 24.11 FID AR metric of VQGAN.\n\nTo support the conclusion that Tokenizers with stronger semantic understanding tend to deliver superior IG performance, we further conduct linear probing experiments on IN1k:\n\n|  | PPL AR $\\downarrow$ | FID AR $\\downarrow$ | Top-5 Acc. |\n|---|---|---|---|\n| VQ-KD CLIP | 53.73 | 11.78 | 75.11 |\n| VQ-KD ViT | 89.30 | 11.40 | 64.88 |\n| VQ-KD DINO | 74.07 | 13.15 | 54.17 |\n| VQ-KD MAE | 280.06 | 26.85 | 41.39 |\n\nAs shown in the above table, VQ-KD CLIP and VQ-KD ViT achieve the highest Top-5 Acc. scores, while VQ-KD MAE achieves the worst Top-5 Acc metric. This trend in the Top-5 Acc. metric is roughly the same as the trend in the FID AR metric, suggesting that tokenizers with stronger IU capabilities tend to behave better in IG tasks.\n\n# Qualitative results\n\nPlease see our top-level comment. We present more visualizations and quantitative analysis for the VQ-KD codebook in the attached PDF file.\n\n# Presentation\n\nThanks for the suggestions. We are revising our manuscript and will use Tab./Fig. for cross-reference. In Eq. (4), the symbol $\\mathcal{D}$ represents the decoder of VQ-KD, which maps the code vectors $\\mathbf{C}(z)$ to the feature space of teacher $\\mathcal{T}'$. We apologize for the confusion and will provide additional information in the main text.\n\n# Comparison Experiments\n\nWe adopt a stronger recipe to train the AR proposal network, with larger batch size (32 per GPU), more training epochs, and classifier-free-guidance (CFG). As shown in the following table, VQ-KD CLIP outperforms prior AR and NAR methods, and is comparable to some Diffusion-based methods.\n\n|  | Architecture | FID |\n|---|---|---|\n| RQVAE [3] | AR | 10.38 |\n| ViT-VQGAN [4] | AR | 4.17 |\n| MoVQ [5] | NAR | 8.78 |\n| MaskGIT [6] | NAR | 4.51 |\n| FSQ [7] | NAR | 4.53 |\n| ADM [8] | Diffusion | 4.59 |\n| LDM-4-G [9] | Diffusion | 3.60 |\n| CVQ-VAE [10] | Diffusion | 6.87 |\n| DiT-XL/2-G [11] | Diffusion | 2.27 |\n| VQ-KD CLIP | AR | 4.10 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b3GROZRGOV", "reviewer": "Reviewer_WgNi", "summary": "This paper shows that image understanding models can be helpful in image generation and that a stronger IU model can result in better IG performance. To demonstrate it, this paper sets multiple experiments from many perspectives (e.g. different datasets and codebook sizes) and gives out reasonable analysis.", "questions": "1.Why doesn't a larger teacher model always lead to better results? Following the theory of this paper, ViT-G/14 should perform best.\n\n2.The experiment about different codebook sizes and dimensions seems to reveal that when the IU model gets stronger, the IG may not always be better. Can I think that to make better IG results, we don't have to use an IU model with too strong capability? I need detailed analysis.", "limitations": "From this paper's limitations, although VQ-KD performs better due to its stronger capability in image understanding, it is not suitable for all scenarios.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.This paper has done experiments across various perspectives, making a solid demonstration.\n\n2.The results of different VQ-KD models outperforming VQGAN and FSQ support this paper's standpoint.\n\n3.The visualization of codebooks is very concise, clear and well drawn.\n\n4.The idea of training a clustering based tokenizer from an IU model is interesting, which can also illustrate the standpoint.", "weaknesses": "1.The visualization of reconstruction of VQ-KD and Cluster seems worse than that of VQGAN and FSQ in both paper and appendix from my point of view. Maybe more visual results should be included.\n\n2.This paper seems to claim that IU model can help IG, but this paper's experiments and analysis are limited to token based tasks. I think this paper should include more IG methods like diffusion, VAE with some little experiments or analysis if possible. Or this paper should emphasize the range is limited to token based tasks.\n\n3.Various experiments have been done across different settings and perspectives, but I think this paper lacks of methods.Besides VQGAN, FSQ and VA-KD, more related methods should be included.\n\n4.The structure of this paper is a little confusing, with experiments, analysis and descriptions mixed together.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and valuable comments. Please note our top-level comment. Below we address specific questions.\n\n# Qualitative results\n\nPlease see our top-level comment. We present more qualitative results in the attached pdf file. Note that image reconstruction capabilities and image generation capabilities may not be aligned. Our primary focus is on image generation.\n\n# Generalizability\n\nThe primary focus of this paper lies in the AutoRegressive (AR) IG framework (as mentioned by line 27 in the main text). We are revising our manuscript to further clarify the scope of our work. \n\nWe also conduct preliminary experiments on Diffusion Models and observe that IU capabilities may enhance their performance. We initially follow LDM [9] to train a VQGAN and a VQ-KD CLIP model. Both models adopted 8-dimensional codebooks of size 16384. Subsequently, we trained diffusion models for 20 epochs on IN-1k for conditional image generation. We summarize the performance of VQGAN, VQ-KD CLIP, and their corresponding diffusion models below:\n\n| Tokenizer | Codebook Usage (%) | rFID | Diffusion Model FID |\n|---|---|---|---|\n| VQGAN | 2.0 | 5.70 | 31.67 |\n| VQ-KD | 100.0 | 5.47 | 31.31 |\n\nWe find that the FID score of the VQ-KD model (at 31.31) is marginally lower than the FID score of the VQGAN model (at 31.67). Yet, it is important to note that the current VQ-KD and Cluster methods are tailored for token-based generators, because token-based frameworks are more apt for exploring the interplay between Image Understanding (IU) and Image Generation (IG). Consequently, these methods may not be directly applicable to continuous VAE models. We remain on the lookout for alternative frameworks that could effectively incorporate IU to assist continuous VAE models.\n\n# Comparison Experiments\n\nCurrently, image tokenization methods can be roughly separated into two groups: vector-quantization-based and scalar-quantization-based. VQGAN adopts vector quantization, while FSQ is one of the latest image tokenizers that adopt scalar-quantization. Other works like SQ-VAE and CVQ-VAE introduce various improvements to VQGAN, but share the same architecture as VQGAN, so their impact on our final conclusion is little.\n\nFor comparison with related works, we adopt a stronger recipe to train the AR proposal network, with larger batch size (32 per GPU), more training epochs, and classifier-free-guidance (CFG). As shown in the following table, VQ-KD CLIP outperforms prior image tokenizers:\n\n| Image Tokenizer | FID |\n|---|---|\n| RQVAE [3] | 10.38 |\n| ViT-VQGAN [4] | 4.17 |\n| MoVQ [5] | 8.78 |\n| MaskGIT [6] | 4.51 |\n| FSQ [7] | 4.53 |\n| VQ-KD CLIP | 4.10 |\n\n# Presentation\n\nOur work studies the relationship between IU and IG. We adopt this organization to clearly express our findings. Any better suggestions are welcome.\n\n# Larger Teacher Models\n\nIn Tab. 6 of the main text, as the OpenCLIP teacher gets larger, the FID AR metric consistently decreases. The FID AR metric evaluates the similarity between generated images and reference images (usually the IN1k val split), so we use FID AR as the primary metric. Other metrics including rFID, PPL, and IS AR, evaluate the reconstruction quality, quality of token sequence modeling, and diversity of the generated images. These metrics may be subject to noise disturbance, so a larger teacher does not guarantee better metrics. Nonetheless, larger teachers do tend to yield better metrics.\n\n# Codebook Sizes and Dimensions\n\nLarger codebook sizes or dimensions do not necessarily lead to stronger IU models. For instance, in Tab. 7 (b) of the main text, increasing the codebook dimension leads to a significant drop in codebook usage, which limits the IU capabilities of VQ-KD. \n\nTo further investigate the relationship between IU and IG capabilities, we conduct linear probing experiments on IN1k, where tokenizers with higher Top-5 Acc. metrics tend to achieve better FID AR metrics:\n\n|  | PPL AR $\\downarrow$ | FID AR $\\downarrow$ | Top-5 Acc. |\n|---|---|---|---|\n| VQ-KD CLIP | 53.73 | 11.78 | 75.11 |\n| VQ-KD ViT | 89.30 | 11.40 | 64.88 |\n| VQ-KD DINO | 74.07 | 13.15 | 54.17 |\n| VQ-KD MAE | 280.06 | 26.85 | 41.39 |\n\nThe above results suggest that tokenizers with stronger IU capabilities tend to behave better in IG tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers, 

We would like to thank you all for providing constructive feedback that helps us improve the paper. We are encouraged by the reviews:

- "The paper is the first to demonstrate that image understanding models can substantially enhance image generation." (Reviewer cC74)
- "The clarity of the paper is reflected in its well-structured presentation and articulate explanation of complex concepts." (Reviewer aFBf)
- "Plenty of quantitative experiments are carried out to validate the superiority of the proposed method." (Reviewer 3Kzg)
- "The visualization of codebooks is very concise, clear and well drawn." (Reviewer WgNi)

We've devoted considerable effort to enhancing our manuscript, and addressing the valuable feedback you've provided. Here, we summarize the key revisions including qualitative results, generalizability, and ablation study. For a more detailed discussion, we encourage you to review our responses to individual reviewer comments.

# Qualitative results

In the attached PDF file, we present more visualizations to demonstrate the superior image generation (IG) capabilities of VQ-KD. The comparison between reconstructed images of VQGAN and VQ-KD CLIP is shown in Fig. 1, with the key regions highlighted. We present qualitative analysis on the codebooks of VQGAN and VQ-KD in Fig. 2. Fig. 3 contains the images generated by VQ-KD CLIP. Lastly, Fig. 4 to Fig. 6 illustrates the reconstructed and generated images on LAION-Aesthetics, SA-Med2D-20M [1], and SATIN [2] datasets.

# Generalizability

## More IU Encoders

In the manuscript, we examined four types of IU encoders (CLIP, DINO, ViT, and MAE), representing fully-supervised, text-supervised, contrastive, and MIM models. To further strengthen our conclusions, we introduce ConvNext as a representative of convolutional IU encoders. Specifically, we use a ConvNext Base model pretrained on IN1k as the teacher to train VQ-KD. The results are shown in the following table, where VQ-KD ConvNext consistently outperforms VQGAN. 

|  | rFID | PPL AR | FID AR | IS AR |
|---|---|---|---|---|
| VQGAN | 5.09 | 116.75 | 24.11 | 39.52 |
| VQ-KD ConvNext | 3.57 | 22.20 | 9.68 | 208.10 |

## More Datasets

From the dataset perspective, in addition to IN-1k and MS-COCO, we also perform experiments on three challenging datasets: LAION-Aesthetics, SA-Med2D-20M [1], and SATIN [2]. LAION-Aesthetics is a subset of LAION 5B with high visual quality. SA-Med2D-20M is a large benchmark dataset in the field of medical imaging. SATIN is a metadataset containing 27 constituent satellite and aerial image datasets spanning 6 distinct tasks. Visualizations are shown in the attached pdf file. The tables below present the rFID and FID AR metrics for quantitative comparison. The reference images for FID evaluation are randomly sampled from each dataset.

| LAION-Aesthetics | rFID | FID AR |
|---|---|---|
| VQGAN | 5.98 | 21.19 |
| VQ-KD CLIP | 5.40 | 10.31 |

| SA-Med2D | rFID | FID AR |
|---|---|---|
| VQGAN | 10.64 | 20.46 |
| VQ-KD CLIP | 9.90 | 18.38 |

| SATIN | rFID | FID AR |
|---|---|---|
| VQGAN | 9.75 | 60.89 |
| VQ-KD CLIP | 9.21 | 55.99 |

# Ablation Study

In the main text, we conducted ablation studies of VQ-KD and find that VQ-KD achieves good IG performance with a wide range of teacher models, codebook sizes, and codebook dimensions. In the table below, we further ablate the quantization operation of VQ-KD. As Sec. 3.2 in the main text mentions, VQGAN introduces a quantization loss to optimize the codebook $\mathbf{C}$. In contrast, VQ-KD adopts K-Means to update $\mathbf{C}$. We replace the K-Means module in VQ-KD with quantization loss and observe a slight performance drop. However,  the performance of VQ-KD CLIP w/o K-Means still outperform VQGAN by a large margin, affirming our findings that IU models can be helpful to the IG task.

|  | Codebook Usage | Codebook PPL | rFID | PPL AR | FID AR | IS AR |
|---|---|---|---|---|---|---|
| VQGAN | 4.9 | 5.96 | 5.09 | 116.75 | 24.11 | 39.52 |
| VQ-KD CLIP | 100.0 | 8.93 | 4.96 | 53.73 | 11.78 | 128.18 |
| VQ-KD CLIP w/o kmeans | 100.0 | 8.73 | 5.65 | 43.99 | 12.07 | 72.43 |

---

[1] Sa-med2d-20m dataset: Segment anything in 2d medical imaging with 20 million masks. arXiv preprint arXiv:2311.11969.

[2] Satin: A multi-task metadataset for classifying satellite imagery using vision-language models. arXiv preprint arXiv:2304.11619.

[3] Autoregressive Image Generation using Residual Quantization, CVPR, 2022.

[4] Vector-Quantized Image Modeling With Improved VQGAN, ICLR, 2022.

[5] MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generation, NeurIPS, 2022.

[6] MaskGIT: Masked Generative Image Transformer, CVPR, 2022.

[7] Finite Scalar Quantization: VQ-VAE Made Simple, ICLR, 2024.

[8] Diffusion models beat GANs on image synthesis. NeurIPS. 2021.

[9] High-resolution image synthesis with latent diffusion models. CVPR. 2022.

[10] Online Clustered Codebook, CVPR, 2023.

[11] Scalable diffusion models with transformers. CVPR. 2023.

Meta-review: This paper received positive scores of (5, 5, 6, 6) from the reviewers. Reviewers unanimously found the application of image understanding models to enhance image generation intriguing. While concerns were raised regarding the generalizability and qualitative results, the authors successfully addressed these concerns in their rebuttal. The AC, having carefully considered the paper, reviews, and rebuttal, concurs with the reviewers' recommendation for acceptance. The authors are encouraged to incorporate the reviewers' feedback into their final submission.
Decision: Accept (poster)