UniTok: a Unified Tokenizer for Visual Generation and Understanding
Authors: Chuofan Ma
Yi Jiang
Junfeng Wu
Jihan Yang
Xin Yu
Zehuan Yuan
BINGYUE PENG
XIAOJUAN QI
Venue: NeurIPS 2025 spotlight ()
Keywords: Visual Tokenizer
Multimodal Generation and Understanding
Primary area: deep_learning
TL;DR: This paper introduces a unified visual tokenizer to facilitate unification of visual generation and understanding within a single autoregressive framework.

Abstract: Visual generative and understanding models typically rely on distinct tokenizers to process images, presenting a key challenge for unifying them within a single framework. Recent studies attempt to address this by connecting the training of VQVAE (for autoregressive generation) and CLIP (for understanding) to build a unified tokenizer. However, directly combining these training objectives has been observed to cause severe loss conflicts. In this paper, we show that reconstruction and semantic supervision do not inherently conflict. Instead, the underlying bottleneck stems from limited representational capacity of discrete token space. Building on these insights, we introduce UniTok, a unified tokenizer featuring a novel multi-codebook quantization mechanism that effectively scales up the vocabulary size and bottleneck dimension. In terms of final performance, UniTok sets a new record of 0.38 rFID and 78.6\% zero-shot accuracy on ImageNet. Besides, UniTok can be seamlessly integrated into MLLMs to unlock native visual generation capability, without compromising the understanding performance. Additionally, we show that UniTok favors cfg-free generation, reducing gFID from 14.6 to 2.5 on ImageNet 256$\times$256 benchmark. All codes and models have been made publicly available.

UniTok: A Unified Tokenizer for Visual Generation and Understanding
Chuofan Ma 1.2
Yi Jiang
Junfeng Wu
Jihan Yang'
Xin Yu!
Zehuan Yuan 2-
Bingyue Peng?
Xiaojuan Qi
'The University of Hong Kong
"ByteDance Inc.
Huazhong University of Science and Technology
Abstract
Visual generative and understanding models typically rely on distinct tokenizers to process images, presenting key challenge for unifying them within single frame- work. Recent studies attempt to address this by connecting the training of VQVAE (for autoregressive generation) and CLIP (for understanding) to build a unified tok- enizer. However, directly combining these training objectives has been observed to cause severe loss conflicts. In this paper, we show that reconstruction and semantic supervision do not inherently conflict. Instead, the underlying bottleneck stems from limited representational capacity of discrete token space. Building on these insights, we introduce UniTok, unified tokenizer featuring novel multi-codebook quantization mechanism that effectively scales up the vocabulary size and bottle- neck dimension. In terms of final performance, UniTTok sets new record of 0.38 rFID and 78.6% zero-shor accuracy on ImageNet. Besides, UniTok can be seam- lessly integrated into MLLMs unlock native visual generation capability, without compromising the understanding performance. Additionally, we show that UniTok favors efg-free generation, reducing gFID from 14.6 2.5 on ImageNet 256 benchmark. GitHub: https: / //g//gtuuul thub. http//Founatiion// www.ooniiiinnnooninnnnoo.oo UniTok
VC4 acora
f6
2audin
Vislon
becoder
Decader
Eyqusr
Seap 'T|
Aypp '"|
Vision
Vision Encodor
Vision Encodien
Encoder
VQVAE
VQVAF-CLIP
UniTok (Ours)
Reconstriction and Understanding Performance
Figure 1: The major challenge in unified tokenizer training. CLIP supervision cannot easily incorporated into VQVAE training This provides only marginal improvements in understanding performance, while drastically degrading reconstruction FID.
Introduction
The advent of GPT-4o [33] highlights the immense potential of Multimodal Large Language Models (MLLMs) with native visual generation capabilities [12, 46, 62, 73,58) 58]. These unified models offer precise control in multimodal interactions, enabling exceptional fluency in tasks such as multi-turn image editing and visual in-context learning. However, fundamental dilemma remains in the choice of visual tokenizers for unified MILMs ee.g, the CLIP [38, 71] tokenizer excels in multimodal understanding but complicates generative modeling due to its high-dimensional, continuous feature
Corresponding authors; 1: Project lead.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

space; Conversely, the discrete VQVAE [8] tokenizer fits autoregressive generation but struggles to capture essential semantics for understanding [62].
In this work, we aim to design a unified visual tokenizer to bridge the gap in multimodal generation and understanding. Intuitivelyy, this can be achieved by integrating CLIP supervision into VQVAR training, resulting in a discrete tokenizer capturing both fine- tiee-graind details and high-level semantics. However, we empirically find this training recipe confronts severe convergence issues [61] and largely falls behind the CLIP baseline in multimodal understanding (Figure 1). While prior studies commonly attribute these challenges to conflicts between semantic and pixel-Ievel feature learning [61, 37, 58], recent progress in visual generation suggests the opposite, showing semantic regularization could benefit tokenizers in reconstruction-ories Do reconstruction and semantic losses truly conflict in tokenizer training?
training [63, 4, 19J. Such disparity motivates us question:
To study the problem, we conduct a comprehensive ablation the unified tokenizer training paradigm (Figure 3), which yields several intriguing findings: First, we show that removing reconstruction supervision, which leads to a vector-quantized CLIP model, does not improve understanding per- formance compared to the unified tokenizer. This observation indicates that the performance gap between unified and CLIP tokenizers mainly arises from vector quantization, rather than conflicts between learning objectives; Further analysis reveals that this gap is driven by two key factors: token factorization, which projects tokens into lower-dimension discretization. These operations are essential for vector quantization but inevitably compromise the expressiveness of visual tokens. We thus argue that the primary bottleneck of unified tokenizers lies in the limited representational capacity of discrete token space.
space for code index lookup [65], and
In light of the issue, consider expanding the vocabulary size and latent code dimension, which allows for a closer approximation of the continuous feature space. However, extensive studies have shown that doing so could result in low codebook utilization [74, 65] and diminishing performance gains [67]. To address this, we introduce multi-codebook quantization to partition the visual token into several chunks, each discretized using small, separate sub-codebook, akin the multi-head attention mechanism [54]. This design exponentially scales the vocabulary size with the number of sub-codebooks, while avoiding the optimization problems of large monolithic codebooks. Besides, we replace traditional linear projection layers with adapted attention modules for token factorization, which is observed to consistently improve training stability and understanding performance.
Building upon these techniques, we train a unified tokenizer called UniTok to bridge visual generation and understanding. Through extensive experiments, we demonstrate that UniTTok achieves comparable or even better performance to domain-specific tokenizers: On ImageNet evaluation, UniTok records an impressive 0.38 reconstruction FID and 78.6% zero-shot accuracy at 256 resolution; In building unified MILMs, UniToo enables the MLLM with native visual generation capabilities while maintaining decent understanding performance. It outperforms the Liquid [59] baseline with a VQGAN tokenizer by 5.5% on VQAv2 [13], 9.2% on TextVQA [42], and 339 points on MME [64]; In addition, we demonstrate that semantic supervision leads to improved latent space structure for autoregressive generation, i.e., for closs-conditional image generation on ImageNet 256x256, UniTook significantly reduces generation FID without classifier-free guidance from 14.6 to 2.5 under the LlamaGen [43] framework, which aligns with recent findings in diffusion modeling [19, 4, 63].
2 Related Work
Image Tokenization for Generation. In the domain of visual generation, image tokenization plays an important role in encoding raw pixels into compact latent features for generative modeling [53, 39J. Among variety of tokenizers, the vector-quantized tokenizer [53] is favored for its discrete latent space and compatibility with autoregressive or masked generative models [48, 43, 3, 66]. The pioneering work VQVAE [53] initially introduced the concept of discretizing continuous tokens by mapping them to the nearest neighbors in learnable codebook. Built on this, VQGAN [8] added perceptual loss [72] and discriminator loss [16] to improve the reconstruction quality. ViT-VQGAN [65] subsequently advanced the framework with the transformer architecture. In recent literature, considerable efforts have been devoted to developing better quantization methods such as residual quantization [18] and lookup-free quantization [67], which also constitute focal point of this paper.
Image Tokenization for Understanding. The unprecedented success of large language models (LLMs) [57, 1, 51, 47] has catalyzed the development of multimodal large language models (MLLMs)

---

[28, 25,31]. As critical component of MLLMs, the selection of an effective vision tokenizer has been the subject extensive study [55, 49]. common choice of the vision tokenizer the pretrained CLIP model [38], which undergoes alignment with language during its pretraining phase. While self-supervised learning models, such as DINOv2 [34], are shown be advantageous at region-level tasks [30]. However, these tokenizers predominantly encode images into a continuous feature space, presenting challenges for uniformly modeling both vision and text tokens. To address this, some works have explored discretizing CLIP tokens [10] or employing VQVAE encoders [27, 62]. Yet, these methods have been observed to substantially impair understanding performance of MLLMs.
Unified Vision-Language Models. The rise MLLMs is not limited to the realm of visual under- standing. Recent advancements have witnessed an increasing focus unifying visual generation and understanding within one MILM [7, 60, 46, 73, 62, 50, 21 Specifically, line of works employs continuous visual tokenizers for image encoding, and leverages pretrained diffusion models for image synthesis [7. 11, 44J. This approach inevitably increases model complexity and disconnects the visual sampling process from the MLLM. In contrast, another stream of research adopts VQVAE models to encode images into discrete tokens [46, 56, 62, 61, 59]. These tokens are subsequently modeled using the same cross-entropy loss that is applied to text tokens, facilitating unified approach to multimodal learning. However, as reconstruction orreettd VQVAE does not naturally align with the LLM token space, these models typically suffer from degraded visual comprchension capabilities. Our research aligns with the second approach, with particular focus on the tokenizer design that is suitable for both generation and understanding tasks.
3 Method
In this section, we introduce UniTok, unified tokenizer well-suited for both visual generation and understanding tasks. We start with unified training recipe that integrates reconstruction (VQVAE) and semantic (CLIP) supervisions (Section 3.1). However, find that simply combining both training objectives leads to severe performance degradation, which can be mainly attributed limited representational capacity of discrete tokens (Section 3.2). To this end, we propose multi-codebook quantization and attention projection to enhance the latent feature space and derive unified visual representations (Section 3.3). An overview of the framework is presented in Figure 2.
Reconstruction Loss
VIE
Vision Encoder
Vision Decoder
Multi-codebook Quantization
painting depicting the countryside scanery.
Text Encoder
Continuous token Discrete token
Contrastive Loss
Figure 2: An overview of UniTok. The tokenizer is trained to reconstruct the input image while aligning its discrete latent features with the text caption. For vector quantization, each visual token split into multiple chunks, which then undergo code index lookup corresponding sub-codebooks.
3.1 Unified Supervision
Visual generative and understanding models typically impose distinct demands on the visual tokeniz- ers. For instance, generation emphasizes precise encoding of the visual signals, whereas understanding prioritizes capturing high-level semantics. To accommodate both requirements, we jointly train the tokenizer with (i) a VQVAE-bart b-ssded reconstruction loss to preserve low-level information, and (ii) an image-text contrastive loss that enhances high-level semantics of the features.
To be specific, the VQVAF-based loss term Lreson consists of pixel-level reconstruction loss LR, perceptual loss Lp based on the LPIPS metric [72], discriminator loss CG to enhance reconstruction

---

fidelity [16], and a vector quantization loss Lvo to minimize distance between the encoder output and its nearest code entry. It denoted as;
freccm = LR + AveLvo + XpLp + AaLa:
where À is the weight factor for the corresponding loss term. The image-text contrastive loss term Coomra is basically the same as in CLIP [38]. Therefore, the final loss term can be written as:
L = Lrecon + Acontra Ceontra.
We simply choose Acoura in this paper.
3.2
Quantization Bottleneck
Despite being augmented with CLIP supervision, we find that the unified tokenizer exhibits unsatis- factory performance in visual understanding tasks, significantly lagging behind the commonly used CLIP tokenizer. To figure out the underlying cause of this underperformance, we break down the key components involved in training unified tokenizer, illustrated in Figure 3. Starting with the CLIP baseline, we provide step-by-ster stes walk- tulk-hhhruuhh of all changes following paragraphs.
Vision Encader
Vssion Decoder
TV Text Encoder
Fremm
VGA Soote
60
58
PC
Boawine
Factorization
Décretiration
Reconstrucioo
*Mullicodebook
Attn. Projection
Figure 3: Roadmap from CLIP to UniTok. It is observed that major degradation in understanding performance comes from token factorization and discretization, rather than reconstruction supervision. The proposed multi-codebbok quantization and attention projection effectively address this by scaling up the vocabulary size and bottleneck dimension. The VQA score is measured using the average score across the VQAv2, GQA, TextVQA, and POPE benchmarks. All tokenizers are trained from scratch on 512m image-text pairs from DataComp.
Factorization. Modern VQ-tokenizers typically project continuous tokens to lower-dimensional latent space for code index lookup (e.g. from 768-d to 8-d), known as token factorization [65]. This increases the relative density of codes by compressing the latent code space, thereby reducing quantization error. To evaluate the impact of factorization in CLIP training, we add two linear projection layers on top of the CLIP vision encoder (right before average pooling), which transforms tokens from 768-d to 16-d and then back to 768-d. Notably, vector quantization and reconstruction supervision are not included at this stage. Surprisingly, it turns out that this channel compression operation significantly compromises the expressiveness of tokens, leading to severe performance degradation in downstream VQA tasks.
Discretization. Based on the implementation described above, we further introduce vector quan- tization to CLIP training, which maps factorized tokens to their nearest code entries. Compared to language tokenizers with vocabularies exceeding 200k entries, the vocabulary size of modern VQ-tokenizers is markedly smaller (i.c., typically ranging from 4k to 16k). Mapping continuous tokens to such a small codebook results in considerable information loss. This is validated in our experiment, which demonstrates that discretizing the factorized tokens with a 16k codebook causes an average accuracy drop of 2.1 in VQA tasks.
Reconstruction Supervision. Finally, we integrate reconstruction losses into the training process to build a unified tokenizer, outlined in Section 3.1. Previous literature suggests that loss conflict between VQVAE and CLIP a major cause of performance degradation in joint training [61]. We observe a similar phenomenon where joint training results in sub-optimal ImageNet zero-shot classi- fication accuracy and reconstruction FID compared to specialized training. However, surprisingly, find that this degradation has negligible impacts on downstream understanding performance.

---

Moreover, the degradation in classification accuracy and reconstruction FID diminishes after we improve the quantization methods (detailed in the next section). Based on these observations, we speculate that the perceived loss conflict is only superficial issue, and the primary cause of the underperformance lies in the limited representational capacity of discrete tokens.
3.3 UniTok
A straightforward solution to breaking the quantization bottleneck could be increasing the codebook size and the latent code dimension. However, current studies on VQVAE tokenizers suggest that there is diminishing gain in scaling and the performance saturates after the codebook size reaches 16k 167, 43]. Continuing expansion results in substantial portion of codes being rarely used or becoming 'dead' during training. which negatively impacts downstream task performance [65]. To address this, we propose multi-codebook quantization and attention projection the following paragraphs.
Multi-codebook quantization (MCQ) discretizes the latent tokens with a set of independent code- books. Specifically, the latent vector f E Id is first evenly split into n chunks {fi, f2, ... fn], where fa E R The subsequent quantization process is denoted as:
= Concat (2 f1). (2). f2) Q (Zn,
(3)
where ff is the discretized latent vector, Q is the code index lookup operation, and Z, is i-th sub- codebook. Compared to conventional quantization methods, the proposed MCQ effectively scales up the vocabulary size. For instance, by increasing the number of sub-codebooks from to 4, and suppose each sub-codebook contains 16k code entries, the theoretical vocabulary size exponentially increases from 2 to 26 (i.e.. there are up to 2 possible combinations of codes for each token). As the size of each individual codebook remains constant, it circumvents the optimization problem associated with large codebooks. Besides, the dimensionality of the latent codes also scales proportionally with the number of codebooks (i.e., increasing from 16-d to 64-d this case), which further enhances the representational capacity of discrete representations.
Discussions. MCQ shares similar concept with residual quantization (RQ) [18] in using multiple codes to quantize token, but differs fundamentally in design philosophy: RQ follows a coarse-to- fine quantization order, whereas MCQ adopts a divide-and-conquer strategy. This distinction gives MCQ unique advantages when operating in high-dimensional latent spaces, where codes tend to become increasingly sparse. For instance, with a latent dimension of 64-d, we observe that MCQ's quantization loss is 15 to 45 times lower than that of RQ. This is because MCQ partitions the original latent space into multiple oow-dimeniional subspaces for quantization. Our ablation study in Table further confirms the superiority of MCQ in unified tokenizer training.
Attention projection. Existing VQ methods usually employ linear convolutional projection layers for token factorization. But as shown in Figure 3, this over-simplified design fails to preserve rich semantics when compressing the feature dimensions, leading degraded understanding performance. To alleviate this problem, we suggest adapting the multi-head attention modules for factorization. Specifically, instead of concatenating features from multiple heads after the attention calculation, we replace the concatenation operation with average pooling to realize channel compression. Figure 6 provides a detailed illustration of the adaptation. Despite simplicity, we find this design effectively strengthens the representational power of factorized tokens and stabilizes training.
3.4 Unified MILM
We proceed to develop unified multimodal model with UniTok. Particularly, we leverage the unified framework introduced in Liquid [59], which models (discrete-valuee) vision and language sequences with universal next-token prediction loss. But instead of learning the visual codebook from scratch, we reuse code embeddings of UniToo by projecting them to the MLLM token space with an MLP projector. Notably, despite UniTok encodes an image into H W x K codes (where K represents the number of sub-codebooks). we simplify this for MLLM input by merging every K consecutive codes into a single visual token. Similarly, when it comes to visual token prediction, we make each token autoregressively predict the next K codes, using depth transformer head as implemented in NQ-Transformer [18] and VII.A-U [61]. This design maintains efficiency for visual generation the context of multi-codebooks.
5

---

Experiments
4.1 Implementation Details
Tokenizer Setup. Leading VQVAR tokenizers predominantly adopt the CNN architecture, while ViT is preferred in CLIP training for its scalability. To take advantage of both, we choose hybrid architecture, ViTamin-L/161 [5], to instantiate UniTok. We configure UniTTok with eight sub-codehooks, each containing 4.096 code entries and latent dimension set to 8-d (the global latent dimension is thus 64-d). The discriminator is initialized with pretrained DINOv2-S [34]. We train the tokenizer for one epoch the public dataset DataComp-Ia IB [9] consisting of 1.28B image-text pairs, with all images resized to 256 x 256 resolution and global batch size of 16k. The learning rate is set to le-3 for the tokenizer and 2e-4 for the discriminator. Besides, we prepare two settings for evaluation: one with pretrained CLIP weight initialization and one with random initialization (the default setting).
MILMM Setup. We instantiate unified MILMM described in Section 3.4 with the Llama-2-7B base model [52]. Following Liquid, we first pretrain the model on a mix of multimodal data, which is composed of 10M language data from DCLM [22], 30M internal MicJJourny--ttlee synthetic data, and 30M re-captioned image-tex( pairs from COYO [32] and Laion [41]. Subsequently, we finetune the model on 1.5M text-to-image data and 1.5M multimodal instruction tuning data introduced in Mini-Gemini [23]. Specifically, the learning rate is set to 5e-5 in the pretraining stage and 2e-5 in the finetuning stage. For visual understanding evaluation, we report results on standard VQA benchmarks including VQAv2 [13], GQA [14]. VQA [42]. POPE [24], MME [64], and MM- Vet [70]. For visual generation evaluation, we report results on GenAl-Bence [26] and MJHQ-30K [20].
4.2 Tokenizer Comparison
We benchmark UniTok on ImageNet using two pri- mary metrics: Fréchet Inception Distance (FID) evaluate reconstruction quality, and top-1 1 zero-shot accuracy to assess image-text aligmment. The results are presented in Table 1. To provide a fair compari- with tokenizers trained small datasets, we also train a version of UniTok on OpenImages [17] solely with reconstruction supervision. It can be seen that UniTok excels in reconstruction quality compared to both unified and domain-specifet tokenizers, record- ing an impressive 0.38 rFIDD on ImageNet with 16x downsampling ratio. As discrete tokenizer, UniTok even surpasses the continuous VAE tokenizer from Stable Diffusion v2.1 [40], showcasing the superi- ority of the proposed multi-codebook quantization. For the perception performance, we observe that ran- domly initialized UniTook demonstrates suboptimal zero-shot classification accuracy. This is expected as current training schedule (i.e., one epoch on 1.28B samples) is insufficient for CLIP training to fully con- verge. It can be seen that initializing the model with pretrained CLIP weights largely alleviates the prob- lem, boosting the zero-shot accuracy from 70.8% to 78.6%. In complement to quantitative results, we pro- vide examples of reconstructed images in Figure 4.
Table 1: Comparison on ImageNet reconstruc- tion FID and zero-shot classification accuracy. rFID is measured at 256x 256 resolution with 16x downsample ratio. indicates model us- ing pretrained CLIP weights for initialization. indicates model trained on Openlmages.
Method
#Tokens rFID
Accuracy
VQVAE Model
VQ-GAN" [8]
256
4.98
RQ-VAE
256
1.30
VAR" [48]
680
0.90
UniTok
256
0.33
CLIP Model
CLIP [38]
256
76,2
SigLIP [71]
256
80.5
ViTamin [S]
256
81.2
Unified Model
TokenFlow [37]
680
1.37
VILA-U* [61]
256
1.80
73.3
Tni
256
0.41
70.8
UniTok!
256
0.38
78.6
4.3
Caas-Condiiional Image Generation
Recent studies on diffusion models indicate that injecting semantics into VAE training leads to better- structured latent space, considerably enhancing guidance-free free generation performance [63, 4, 19]. To evaluate whether UniToo possesses similar properties, we test within the LlamaGen framework for class-conditional image generation. As shown in Table 2, Uni reduces the FID by 12.11 compared to the VQGAN baseline in CFG-free generation, under the same generator setup. This implies that UniTok learns more structured code distribution, benefiting autoregressive modeling.

---

Liquid while Original Images
EDMARO JONES MESICE
LA
Reconstructed Images
Figure 4: Qualitative results on image reconstruction resolution of 256 256.
Table 2: Class-conditional image generation results on ImageNet 256x256. VQGAN tokenizer from LlamaGen. :; Images are generated at 3 384 resolution and then resized to 256 256 for evaluation. Pre." precision; 'Rec.: recall; 'CFG': classifier- freeiierr-reeguuue greggudance....
Generation w/o CFG
Generation w/
Tokenizer
FiD
Generator
#Params.
gFID
1St
Pre.
Rec.
gFID
1St
Pre.
Rec.
Diffusion Models
SD-VAE [40]
0.61
DIT [35]
675M
9.62
121.5
0.67
0.67
2.27
278.2
0.83
0.57
VAVAE [63]
0.28
LightningDTT
675M
2.17
205.6
0.77
0.65
1.35
295.3
0.79
0.65
Masked Generative Models
LFQ [67]
0.9
MAGVIT-v2 [67]
307M
3.07
213.1
1.91
324.3
TYTok-L [69]
2.21
MaskGIT [3]
177M
3.15
173.0
2.77
199.8
Autoregressive Models
VQGAN!
2.19
LlamaGen* [43]
1.4B
14.65
86.3
0.63
0.68
2.34
253.9
0.81
0.60
UniTok (Ours)
0.41
LlamaGen [43]
1.4B
2.51
216.7
0.82
0.57
2.77
227.5
0.81
0.57
4.4 Unified Understanding and Generation
Understanding Performance. We evaluate the understanding performance of UniTok diverse VQA benchmarks in Table 3. Our unified MILLM showcases clear advantages when compared to other unified models that also utilize discrete visual tokenizer. Specifically, UniTTok significantly outperforms the Chameleon model, which relies on a traditional VQVAE tokenizer, by 7.2% higher accuracy on VQAv2. Additionally, surpasses VILA-U, another model with a unified tokenizer, by 3.3% in accuracy on the TextVQA benchmark and by notable margin of 112 points on the MMME-Perception scores. Furthermore, we can see that UniTok largely narrows the performance gap with MLLMs that incorporate continuous visual tokenizers. These strong results confirm the candidacy of UniTok unified visual tokenizer for multimodal models.
Generation Performance. Table presents the text-to-image generation performance our unified MILM the GenEval benchmark. We show that UniTok not only outperforms most of the unified MILMs, but also demonstrates competitive performance against domain experts (diffusion models) trained billions of images. Besides, UniToo achieves non-trivial improvements using exactly the same set of text-to-image training data, highlighting the importance of unified tokenizer. We also provide results on GenAl-Bench in Table 10 and Table 11 in Appendix.
further evaluate the quality of images generated by our model on the MJHQ-30K benchmark, details of which are presented in Table 5. Notably, this benchmark primarily relies on the FID score
7

---

Table 3: Comparison with unified multi-modal large language models on VQA benchmarks.
Method
LLM
Token Type
Res.
VQAv2
GQA
TexIVQ
POPE
MME
MM-Vel Vet
Emu [45]
Llama-13B
Continuous
224
52.0
LaVIT [1S]
Llama-7B
Continuous
224
66.0
46.8
DreamLLM
Vicuna-7B
Continuous
224
72.9
41.8
26.6
Unified-OO [29]
6.8B from scratch
Continuous
384
79.4
87.7
Janus [58]
DeepSeck-1.3 1.3B
Continuous
384
77.3
59.1
87.0
1338
34.3
CM3Leon [68]
7B from scratch
Discrete
256
47.6
LWM [27]
Llama-2-7B
Discrete
256
55.8
44.8
18.8
75.2
Show-o [62]
Phi-1.5-1.3B
Discrete
256
59.3
48.7
73.8
948
Chameleon [46]
34B from scratch
Discrete
512
69.6
Liquid [59]
Gemma-7B
Discrete
512
71.3
58.4
42.4
81.1
1119
VII.A-U [6]]
Llama-2-7B
Discrete
256
75.3
58.3
48.3
83.9
1336
27.7
UniTok
Alama-2-78
Discrete
256
76.8
61.1
51.6
83.2
1448
33.9
Table 4: Comparison with other visual generation methods the GenEval benchmark.
Method
Type
#Dtt
Single Ohj.
Two Ohj.
Counting
Colors
Position
Color Attri
Overall
SD v2.1 [39]
Diffusion
2000M
0.98
0.51
0.44
0.85
0.07
0.50
SD-XL [36]
Diffusion
2000M
0.98
0.74
0.39
0.85
0.15
0.55
DALL-E [2]
Diffusion
0.96
0.87
0.47
0.83
0.43
0.45
0.67
36M
0.95
0.52
0.49
0.82
0.11
0.28
0.53
Show-o [62]
Discrete Diff.
2.0B
0.98
0.80
0.66
0.84
0.31
0.50
0.68
LWM [27]
Autoregressive
0.93
0.41
0.46
0.79
0.09
0.15
0.47
Janus [58]
Auloregressive
0.97
0.68
0.30
0.84
0.46
0.42
0.61
Liquid [59]
Auloregressive
30M
0.98
0.73
0.32
0.76
0.17
0.37
0.55
Tok
Autoregressive
30M
0.99
0.71
0.36
0.79
0.26
0.45
0.59
for evaluation, high-resolution images are preferred because they potentially capture more fine-grained details. Despite this makes FID across different reso- lutions less comparable, we show that our model achieves impressive performance smallest resolution, showcasing its ability to generate high-quality, detail-rich images.
Table 5: Results on MJHQ-30K.
Method
Type
Res.
FIDJ
SD-XI [36]
Ditfusion
1024
9.55
the the
PixArt [6]
Diffusion
1024
6.14
Playground [20]
Diffusion
1024
4.48
Liquid [59]
Autoregressive
512
5.47
Janus [58]
Autoregressive
384
10.10
We present some examples of the images generated by our model in Figure 5, using text prompts sampled from MJHQ-30K. The visualization results demon- strate our model is capable of synthesizing photo- realistic and visually appealing images. Moreover,
LWM [27]
Autoregressive
256
17.77
Show-o [62]
Discrete Diff'
256
15.18
VILA-U [61]
Autoregressive
256
12.81
UniTok
Autoregressive
256
7.46
the model is able to comprehend wide spectrum of concepts, such as 'Vincent van Gogh painting style" and 'bitcoin', and flexibly combine these concepts to synthesize creative images.
4.5 Ablation Studies
Impact of Supervision Types. ablate the impact of contrastive and reconstruction losses in UniTok training, we conduct experiments tokenizers trained with different supervision types, as shown in Table 6. It is worth noting that all the tokenizers are vector-quantized even though some do not have reconstruction supervision. First, show that reconstruction-orieee significantly lags behind tokenizers with contrastive supervision in visual understanding performance. This observation evidences the limitations of traditional VQVAE. Second, we demonstrate that reconstruction and contrastive training objectives do not inherently conflict, or can be addressed by enhancing discrete feature space. With multi-codebook quantization, the jointly trained tokenizer not only exhibits understanding performance on par with the tokenizer trained solely with contrastive loss, but also slightly improves generation performance
tokenizer
the reconstruciion-orinneed tokenizer.
MCQ v.s. RQ. Following discussions in Section 3.3, we provide an apple-to-apple comparison between muwti-codebout guantization and residual quantization in Table 7. For fair comparisons,

---

B
Figure 5:
Images generated in resolution of 256 256 with unified MILM.
Table 6: Impact of different supervision types downstream generation and understanding perfor- mance. The rFID and gFID are measured on the ImageNet (256 x 256) validation set. LlamaGen-L [43] is adopted as the generator for gFID evaluation.
Generation
Understanding
Supervision
IFID +
gFID
VQAv2
GQA
SciQA
TextVQa
POPE
MME
Contrastive
68.95
56.89
65.64
49.89
82.34
1373
Reconstruction
0.82
3.59
56.33
47.53
63.26
43.65
77.09
Recon. + Contra.
0.72
3.26
69.14
56.06
65.25
49.22
81.42
1333
directly implement RQ on our codebase, keeping all the training settings the same as UniTok. Both tokenizers are trained on 5 512M subset of DataComp- IB. Notably, unlike MCQ, RQ by default uses a shared large codebook. To keep the global codebook size the same as UniToo, we set the codebook size of RQ 32768. It can be that RQ demonstrates inferior reconstruction performance and lower classification accuracy compared to MCQ under the high bottleneck dimension (64-d) setting.
Number of Sub-Codebooks. To gain deeper insights into multi-codebook quantization, evaluate how tokenizer performance changes with the number of sub-codebooks in Table 8. Specifically, the size of codebook is denoted as A x B, where A is the number of sub-codebook and B is the size of cob-codebbok. For rFID evaluation, we train the tokenizer solely with reconstruction loss OpenImages [17], and evaluated it on ImageNet (256 x 256) validation set. While for ImageNet zero-shot accuracy evaluation, the tokenizer is trained on DataComp- 128m subset using only contrastive loss. Given constant global codebook size,
that increasing the number of sub- codebooks consistently improves reconstruction FID and classification accuracy. This indicates that MCQ generally benefits vector-quantized models, independent of the training objectives.
Table 7: MCQ v.s. RQ.
Table 8: Ablation number of sub-codebooks.
Metbod Code Shape Code Dim.
=FID. Accuracy
2xx192/22 Codebook/ / Vocabulary 1x "63
4x:096/22 /2
8x:0088/ /
RQ
16x16x8
64
3.46
58.8
#iD
1.50
0.98
0.54
0.33
MCQ
16x16x8
63.7
Accuracy
41.0%
43.9%
44.7%
46.1%
CLIP Weight Initialization. We notice that higher ImageNet accuracy does not guarantee superior downstream performance. In Table 9, we ablate the impact CLIP weight initialization on visual understanding performance. Specifically, we adopt the classic LLaVA framework for evaluation,

---

replacing the original CLIP tokenizer with UniTok while keeping all other the training settings unchanged. One tokenizer is initialized with the pretrained ViTamin-L-256 [5] weights. while the other is randomly initialized. To our surprise, UniTok that is trained from scratch surpasses the one initialized with pretrained CLIP weights, despite the latter actually achieves better zero- shot classification accuracy. This suggests downstream VQA performance may not be highly correlated with ImageNet classification accuracy. More importantly, also implies that CLIP weight initialization may serve negative prior for unified tokenizers, as the unified visual feature space could drastically differ from CLIP feature space.
Table Comparison of different initialization methods under the LLaVA framework. f indicates the model uses CLIP weights for initialization. We highlight the default setting of UniTok in gray.
Tokenizer
VQAv2
GQA
TextVQA
POPE
MME
UniTokt
69.9
56.2
49.3
81.2
1331
UniTok
72.4
58.2
51.6
82.4
1392
5 Limitations and Conclusion
This paper studies unified visual tokenization for generation and understanding, which serves as the cornerstone of unified multimodal large language models. We investigate the training paradigm of unified tokenizers and identify that the current challenge in unification mainly arises from the limited representational power of discrete tokens. To address this limitation, quantization and attention projection to build a unified tokenizer called UniTok. We show that UniTok excels downstream visual generation and understanding tasks. The ablation study further reveals that discriminative and generative representation learning does not inherently conflitt. We hope our findings could inspire future research in this domain.
introduce multi-codebook
However, due to limited computational resources, UniTok is only trained for one epoch, which is not sufficient for CLIP-based semantic representation learning. We believe extending the training schedule could further benefit the tokenizer, especially understanding performance.
10

---

Acknowledgments This work has been supported by the National Key R&D Program of China (Grant No. 2022YFB3608300, Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422, 17212923, 17215025) Themebased Research (Grant No. T45-701/22-R) and Shenzhen Science and Technology Innovation Commission (SGIX202205301 I 1405040). We sincerely thank Seng Ye for the insightful discussions and valuable contributions to this project.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXiv23303.08774, 2023.
[2] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science, https://edn. openai. com/papersddallee.. pdf. 2(3):8, 2023.
[3] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11315-11325, 2022.
[4] Hao Chen, Yujin Han, Fangyi Chen. Xiang Li, Yidong Wang. Jindong Wang. Ze Wang. Zicheng Liu, Difan Zou, and Bhiksha Raj. Masked autoencoders are effective tokenizers for diffusion models. arXiv preprint ar arXiv:2502.0344. 2025.
[5] Jieneng Chen, Qihang Yu, Xiaohui Shen, Alan Yuille, and Liang-Chieh Chen. Vitamin: Designing scalable vision models in the vision-language era. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12954-12966, 2024.
[6] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.0042
2023.
[7] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang. Liang Zhao, Jian- jian Sun, Hongyu Zhou, Haoran Wei, et Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint :rrXiv::309 /499, 2023.
[8] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883, 2021.
[9] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jicyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
[10] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.0804 080441 2023.
[11] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint urXiv:2404.14390 2024.
[12] Google. Experiment with gemini 2.0 flash native image generation, 2025.
[13] Yash Goyal, Tejas Khot, Douglas Summers-Star Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904 6913, 2017.
[14] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6700-67000 2019.
11

---

[15] Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Bin Chen, Chenyi Lei, An Liu, Chengru Song, Xiaoqiang Lei, et al. Unified language-viste pretraining with dynamic discrete visual tokenization. arXiv preprint arXiv:2309.04660 2023.
[16] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410 2019.
[17] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International journal of computer vision, 128(7): 1956 2020.
[18] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523-11532 2022.
[19] Xingjian Leng, Jaskirat Singh, Yunzhong Hou, Zhenchang Xing, Saining Xie, and Liang Zheng. Repa-e: Unlocking vae for end-to-end tuning with latent diffusion transformers. arXiv preprint arXiv:2504.10483 2025.
[20] Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation. arXiv preprint arXiv:2402.17245, 2024.
[21] Hao Li, Changyao Tian, Jie Shao, Xizhou Zhu, Zhaokai Wang, Jinguo Zhu, Wenhan Dou, Xiaogang Wang. Hongsheng Li, Lewei Lu, et al. Synergen-vl: Towards synergistic image under- standing and generation with vision experts and token folding. arXiv preprint ar arXiv:2412.09604, 2024.
[22] Jeffrey Li, Alex Fang, Georgios Smyriis, Maor Ivgi. Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, et al. Datacomp-Im: In search of the next generation of training sets for language models. arXiv preprint arXiv:2406.1179, 2024.
[23] Yanwei Li, Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 18814, 2024.
[24] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object ballucination in large vision- vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[25] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IÉEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26689 21689,,26999...1 2024.
[26] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-texx generation. In European Conference on Computer Vision, pages 366-384. Springer, 2025.
[27] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention. arXiv preprint 200202240..08868, 2024.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[29] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26455, 2024.
[30] Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, and Xiaojuan Qi. Groma: Localized visual tokenization for grounding multimodal large language models. In European Conference on Computer Vision, pages 417-435. Springer, 2025.
12

---

[31] Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint arXiv:2403.0961/ 2024.
[32] Byeon Minwoo, Park Beomhee, Kim Haecheon, Lee Sungjun, Baek Woonhyuk, and Kim. Sachoon. Coyo-700m: Image-text pair dataset., 2022.
[33] OpenAl. Introducing 4o image generation, 2025.
[34] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint urXiv:2304.07193. 2023.
[35] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[36] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
[37] Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint 241.0029,,,0 2024.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer: High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[41] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-2522 2022.
[42] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[43] Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 2.Xiv:24006066555, 2024.
[44] Quan Sun, Yufeng Cui, Xiaosong Zhang. Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14398-14409, 2024.
[45] Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222. 2023.
[46] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405008818, 2024.
[47] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint 2rX.n:2403.08295, 2024.
13

---

[48] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive mod- eling: Scalable image generation via sext-scale prediction. arXiv preprint arXiv:2404.02903 2024.
[49] Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprini arXiv:2406. 16860, 2024.
[50] Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal under- standing and generation via instruction tuning. arXiv preprint 2rXiv:2412.14166 2024.
[51] Hugo Touvron, Thibaut Lavril, Gautier Izacard. Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1397, 2023.
[52] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.09225. 2023.
[53] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[55] Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankanhalli, and Ying Shan. What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305.12221 2023.
[56] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869, 2024.
[57] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
[58] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma. Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint arXiv:2410. 13848, 2024.
[59] Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable and unified multi-modal generators. arXiv preprint arXiv:2412.04332. 2024.
(60] Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. arXiv preprint 2ttiv:23000051111 2023.
[61] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409.044 04499,9442,, 2024.
[62] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang. and Mike Zheng Shou. Show-o: One single trans- former to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528 2024.
[63] Jingfeng Yao, Bin Yang, and Xinggang Wang. Reconstruction vs. generation; Taming optimiza- tion dilemma in latent diffusion models. arXiv preprint 2510012501 2025.
[64] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint 13..v:23006.3549) 2023,
14

---

[65] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. fector-quantized image modeling with improved vqgan. arXiv preprint arXiv:21 11.0100.44222 2021.
[66] Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10459-10469, 2023.
[67] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion-tokenizer is key to visual generation. arXiv preprint arXiv:2310.05733 2023.
[68] Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, al. Scaling autoregressive multi-modal models:
Pretraining and instruction tuning. arXiv preprint arXiv:2309.0233 02..99,,,.9...... 2(3), 2023.
[69] Qihang Yu, Mark Weber, Xueqing Deng, Xiaohui Shen, Daniel Cremers, and Liang-Chieh Chen. An image is worth 32 tokens for reconstruction and generation. Advances in Neural Information Processing Systems, 37:118940-128966 2024.
[70] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308. 200.000900 2023.
[71] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19955-11986, 2023.
[72] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018.
[73] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408. 1tt1::400..11399, 2024.
[74] Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vqgan to 100,000 with utilization rate of 99%. arXiv preprint arXiv:2406. 1837, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We have included the motivation, empirical findings, technical innovations, and major achievements of our work in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please see "Limitations and Conclusion' section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have included the implementation details of our tokenizer and MLLM (such as training data, hyperparameter configurations, testing framework, etc) in the 'Experiments' part.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Most code and data for this paper are open-source, except the unified MILMM training data, which includes some internal, non-public data. However, our main focus tokenizer training- only publicly available data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
public/guides/Codeee httpe//oeeuttissssoooooiiiieeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips. cctpublic/guiii httpa///uiiiee/oeeuuubbiiieeee wwwesSubmisiiooPoo...
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please see our code for the full set of hyperparameter configurations.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We mainly evaluate our methods using ImageNet FID and zero-shot accuracy. metrics known for low variance. Following prior work, we omit error bars in our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: It takes roughly 50 hours with the equivalent computing power of 256 A100 GPUs train the tokenizer.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: h/tpprip////r/p.....e cc/public/Eate
Answer:
[Yes]
Justification: I have read and confirmed that this paper complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., - if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
19

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: We will add safeguards in our future release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We strictly followed the licenses use other assets (like DataComp) with credits.
Justification:
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy btttp:://eeurps... for what should or should not be described.
22

---

A Attention Projection Modules
Figure 6 illustrates the adaptations made to the tradition MHA module, which enables channel compression and expansion in token factorization.
NxC
Avg
Concat
Attention
Attention
Nxcoxe
Nxe
Nxhove
NxC
Lineur
Lineur
Lineur
Linear
Norm
Norm
Norm
Norm
NxC
Nkc
Channel Compression Block
Channel Expansion Block
Figure 6: Modified attention blocks for factorization. Modules in yellow indicate a change in the number of channels. C and stand for the channel dimension, h is the number of heads in the multi-head attention module. C' c.
B More Generation Results
We provide the results on GenAl-Bench in Table 10 and Table 11. UniTok consistently delivers superior generation performance on this benchmark.
Table 10: Comparison with other visual generation methods on GenAl-Bench (basic prompts).
Relation
Method
Type
# Fraining Images
Attributet
Scenet
Overall
Spatial
Action
Part
SD v2.1 [39]
Diffusion
2000M 2000M
0.80
0.79
0.76
0.77
0,80
0.78
SD-XL [36]
Diffusion
0.84
0.84
0.82
0.83
0.89
0.83
Midjoumey v6
Diffusion
0.88
0.87
0.87
0.87
0.91
0.87
DALL-E [2]
Diffusion
0.91
0.90
0.92
0.89
0.91
0.90
Show-o [62]
Discrete Diff.
36M
0.72
0.72
0.70
0.70
0.75
0.70
LWM [27]
Autoregressive
0.63
0.62
0.65
0.63
0.70
0.63
VILA-U [61]
Autoregressive
15M
0.78
0.78
0.77
0.78
0.79
0.76
Liquid [59]
Autoregressive
30M
0.84
0.86
0.81
0.83
0.91
0.83
UniTok
Autoregressive
30M
0.85
0.87
0.86
0.86
0.89
0.85
Table 11: Comparison with other visual generation methods on GenAl-Bench (advanced prompts).
Logicalt
Method
Type
#Training Images
Countt
Differt
Compare
Overallf
Negate
Universal
SD [39]
Diffusion
2000M
0.70
0.54
0.64
0.62
SD-XL [36]
Diffusion
2000M
0.71
0.73
0.69
0.50
0.66
0.63
Midjournyy v6
Diffusion
0.78
0.78
0.79
0.50
0.76
0.69
DALI.-F [2]
Diffusion
0.82
0.78
0.82
0.48
0.80
0.70
Show-o [62]
Discrete Diff.
36M
0.70
0.62
0.71
0.51
0.65
0.60
LWM [27]
Autoregressive
0.59
0.58
0.54
0.49
0.52
0.53
VILA-U [61]
Autoregressive
15M
0.70
0.71
0.74
0.53
0.66
0.64
Liquid [59]
Autoregressive
30M
0.76
0.73
0.74
0.46
0.74
0,65
UniTok
Autoregressive
30M
0.76
0.76
0.79
0.46
0.73
0.67
23

---

C Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There many potential societal consequences of our work, none which we feel must be specifically highlighted here.
24


Reviews:
[{"review_id": "RwooPQRDsi", "reviewer": "Reviewer_rTfH", "summary": "This paper proposes studying a visual tokenizer, **UniTok**, which can be effectively used for both generation and understanding tasks in MLLMs. The authors conduct extensive experimental analysis to investigate the importance of multi-codebook quantization and attention projection in their model design.", "questions": "Please see **Cons**", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Pros\n* The writing is clear and easy to follow.\n* The authors have conducted comprehensive experiments to analyze the bottlenecks of the tokenizer in unifying different tasks.\n* The tokenizer demonstrates strong empirical performance in both generation and MLLM understanding tasks.\n\n\n## Cons\nThese are not necessarily weaknesses, but addressing them may help further strengthen the paper:\n* Table 6 shows that supervision from reconstruction loss and contrastive loss have different impacts on downstream tasks: adding contrastive loss further improves generation, but adding reconstruction loss often decreases performance on understanding tasks. Do the authors have any insights into why this occurs?\n* Given the rapid development of this field, the authors might also consider including additional related work for discussion, such as [1].\n\n[1] Hansen-Estruch, Philippe, et al. \"Learnings from Scaling Visual Tokenizers for Reconstruction and Generation.\" *arXiv preprint* arXiv:2501.09755 (2025).", "was_revised": true, "final_justification": "I have read the authors’ rebuttal. The paper presents a solid empirical study on using a visual tokenizer to unify visual understanding and generation tasks. I will maintain my initial score.", "rebuttal": "Dear Reviewer rTfH,\nThank you for your constructive feedback and thoughtful evaluation of our work. We hope that our responses below will address your questions and concerns.\n\n**Q1: How does contrastive loss improve generation?**  \n\nIn UniTok, the contrastive loss (CLIP supervision) injects semantic information into the code embeddings, which is conceptually similar to the MAE loss in MAETok [1] and the VF loss in VA-VAE [2]. Both of these works have demonstrated that **semantic guidance leads to more structured latent distributions**, which in turn facilitate diffusion-based generative modeling. Our experimental results in Tables 2 and 6 further confirm that this benefit extends to discrete latent spaces and autoregressive generative models.\n\n[1] Yao, Jingfeng, Bin Yang, and Xinggang Wang. \"Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Chen, Hao, et al. \"Masked autoencoders are effective tokenizers for diffusion models.\" Forty-second International Conference on Machine Learning. 2025.\n\n**Q2: How does reconstruction loss impact understanding?**\n\nIntuitively, reconstruction loss encourages the tokenizer to preserve low-level details (e.g., texture) when encoding images. Since the total representation capacity of discrete tokens is limited, this creates a trade-off — allocating more \"storage space\" to low-level details leaves less capacity for encoding high-level semantics, which can lead to degraded understanding performance. However, when the discrete latent space is enlarged (i.e., by using multi-codebook quantization), this effect becomes minor and has only a mild impact on understanding performance.\n\n**Q3: Related work ViTok**\n\nThank you for the suggestion. ViTok presents a comprehensive study on the scaling properties of VAEs, offering valuable insights into VAE design. We appreciate your recommendation and will include a discussion of this work in our revised manuscript."}, {"review_id": "eIENc40hGA", "reviewer": "Reviewer_Qdyc", "summary": "The paper introduces UniTok, a unified tokenizer designed to bridge the gap between visual generation and understanding. UniTok features a novel multi-codebook quantization mechanism that increases the tokenizer’s capacity and enables it to scale effectively. It achieves a record low reconstruction FID of 0.38 and 78.6% zero-shot accuracy on ImageNet. UniTok can be seamlessly integrated into multimodal large language models, enabling visual generation capabilities without compromising performance in understanding tasks.", "questions": "1) Can the tokenizer be enhanced by captioning loss (e.g., COCA)? This may enhance the downstream VQA task too.", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n1) The paper is well-written and easy to read, with a logical flow that addresses both the technical details and experimental results effectively.\n2) The paper presents a well-structured approach to visual tokenization, demonstrating solid experimental results. The multi-codebook quantization significantly improves performance over traditional VQ-based methods.\n3) UniTok represents a important step forward in solving the problem of unified tokenization for both visual generation and understanding.\n4) The use of multi-codebook quantization to effective. \n\nWeakness:\n1) The vision encoder is trained in 256x256 resolution. Will the LMM tasks in Table 6 need higher resolution? Can the author align the resolution when training the tokenizer?", "was_revised": true, "final_justification": "I carefully read the authors' response, and my concerns are addressed. Thus I raise my score.", "rebuttal": "Dear Reviewer Qdyc,\n\nWe greatly appreciate your insightful comments and support for our work. We hope that our responses below will address your questions.\n\n**Q1: Resolution used in LMM evaluation (Table 6)**\n\nIn Table 6, we evaluate the model at 256×256 resolution for all tasks, consistent with the resolution used during UniTok training. We have also tested UniTok at 384×384 and 512×512 resolutions within the LLaVA framework. We found that UniTok generalizes well to 384×384 resolution without finetuning, likely due to its hybrid architecture (CNN + ViT). In particular, accuracy on the TextVQA benchmark—which benefits from higher resolution input—increases by 2%. However, further increasing the input resolution to 512×512 yields little additional improvement. We believe that finetuning UniTok at higher resolutions is necessary to make the most of high-resolution inputs.\n\n**Q2: Could UniTok be enhanced by incorporating captioning loss?**\n\nThat's an excellent point. In addition to the image-text contrastive loss, image captioning loss is generally considered beneficial and is widely adopted in CLIP training, as demonstrated by works such as CoCa, BLIP2, and SigLIP2. Since UniTok inherits the CLIP training paradigm, we believe that incorporating captioning loss could further enhance UniTok's performance. Besides, TexTok [1] has shown that image captions can complement visual tokens to achieve better reconstruction quality and higher compression rates. Therefore, leveraging generated captions during the image decoding process could potentially yield additional improvements.\n\n[1] Zha, Kaiwen, et al. \"Language-guided image tokenization for generation.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "ebk1zoX6XW", "reviewer": "Reviewer_uzhd", "summary": "This manuscript focuses on developing effective tokenizer for unified understanding and generation. The authors analyze the potential problems of joint training with reconstruction and semantic supervision. Experiments show that the challenge lies in fabrication and discretization. To improve the vocabulary size and representation, the authors propose multi codebook quantization (MCQ) and utilize modified multi-head attention to replace the linear projection. The proposed method achieves excellent performance on both understanding and generation tasks.", "questions": "* The authors claim that MCQ can also boost the dimension of the vocabulary. However, the authors do not analyze or compare with the method that directly increases the dimension of a single codebook.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Strengths:\n1. Tokenizer is a critical part for unified multi-modal models. This work propose an effective tokenizer for balancing understanding and generation performance. Comparison with most state-of-the-art methods demonstrate its effectiveness. \n\n2. The motivation of addressing the poor representation of discrete tokenizer is clear. The proposed methods seem to reasonable.\n\n3. The paper is well-written and easy to follow.\n\n- Weaknesses:\n1. The experiment results seem to inconsistent with the analysis. Specifically, the authors claim that the MQC is mainly designed to expand the vocabulary size. However, from Table 8 it can be seen that the performance increases as the number of codebook groups increases while the total vocabulary size remains unchanged. It is more like multi-head attention that utilizes more sub-space.\n\n2. Figure 3 utilizes VQA score for analysis and draw the conclusion that the challenge of unified tokenizer is not the joint training. However, the authors claim in L144: \"We observe a similar phenomenon where joint training results in sub-optimal ImageNet zero-shot classification accuracy and reconstruction FID compared to specialized training.\" It seems that the conclusions are inconsistent and are related to the tasks.", "was_revised": true, "final_justification": "The rebuttal has well addressed my concerns. Thus, I would improve my rating to \"Accept\".", "rebuttal": "Dear Reviewer uzhd,\n\nThank you for your constructive feedback and thoughtful evaluation of our work. We hope our point-to-point response will address your concerns.\n\n**Q1: Clarification on the vocabulary size in Table 8**\n\n- First, we would like to clarify that the row of `Codebook` in Table 8 denotes the codebook size rather than the vocabulary size. (We apologize for any confusion caused by the notation.) In UniTok, the vocabulary size is exponentially larger than the codebook size. For example, UniTok uses 8 codebooks with 4096 code entries each, resulting in a total codebook size of 4096 × 8 = 32,768. In contrast, the vocabulary size is 4096^8 = 2^96, i.e., there are up to 4096^8 possible combinations of codes for each token.\n\n- We have extended Table 8 below to illustrate how both the vocabulary size and latent code dimension increase with the number of codebooks. As shown, increasing the number of codebooks simultaneously scales up the vocabulary size and latent dimension, which consistently leads to improved performance. We will revise Table 8 in our manuscript to make this relationship clearer.\n\n> | Codebook | 1×16384 | 2×8192 | 4×4096 | 8×2048 |\n> |:------------:|:----------:|:---------:|:---------:|:--------:|\n> | Vocabulary | 2^14 | 2^26 | 2^48 | 2^88 |\n> | Latent Dim. | 8 | 16 | 32 | 64 |\n> | rFID ↓ | 1.50 | 0.98 | 0.54 | 0.33 |\n> | Accuracy | 41.0% | 43.9% | 44.7% | 46.1% |\n\n**Q2: Inconsistency between observations in L144 and the conclusion**\n\nAlthough joint training can initially lead to a degradation in classification accuracy and reconstruction FID (L144), we observe that this degradation diminishes after improving the quantization methods (L147). For example, Table 6 shows that joint training actually results in better rFID and gFID. This supports our conclusion that VQVAE training and CLIP training do not inherently conflict; rather, the underlying issue is that discrete tokens lack sufficient capacity to simultaneously encode both low-level details and high-level semantics.\n\n**Q3: The impact of increasing the latent code dimension (single codebook setting)**\n\n- Thank you for your suggestion. The table below presents a comparison between the multi-codebook and single codebook settings, where the latent code dimension is scaled up to 64d. Due to time constraints, we trained the tokenizers on OpenImages using only the reconstruction loss. As shown, solely increasing the latent dimension of a single codebook results in lower codebook utilization rate (even with entropy loss) and degraded rFID.\n\n> | Codebook Size | Vocabulary Size | Latent Dim. | Codebook Utilization Rate | rFID ↓ |\n> |:-----------------:|:-------------------:|:-------------:|:------------------------------:|:-------:|\n> | 1×32768 | 2^14 | 64 | 82% | 2.20 |\n> | 8×4096 | 2^88 | 64 | 100% | 0.33 |\n\n- Besides, as mentioned in Lines 45-47, previous works have also reported similar effects when scaling the latent dimension. Specifically, Table 4 in ViT-VQGAN [1] provides a detailed and quantitative ablation study supporting this observation.\n\n[1] Yu, Jiahui, et al. \"Vector-quantized image modeling with improved vqgan.\" arXiv preprint arXiv:2110.04627 (2021)."}, {"review_id": "ULDQGPZE65", "reviewer": "Reviewer_emrz", "summary": "This work studies how to design a unified discrete tokeniser for both image understanding and generation tasks. It identifies the bottle necks of directly combining CLIP and VQVAE: improper compression from the network architecture, a single codebook. They made some improvements to enhance the representative capacity of the discrete token space. Experimental results verify the validity of the proposed method.", "questions": "Please see the weakness.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength\n1. This paper studies an interesting topic and reveals that under proper design, the discrete tokenizer can achieve good performance on both understanding and generation tasks.\n2. The paper is well oragnzied and  easy to follow.\n\nWeakness\n1. It seems that the comparison in Table 7 is unfair. RQ also support multi codes/groups, therefore, 1×32768  (K=1)is not fair for using the RQ baseline. Can you show the results for RQ(K=4 or 8?)?\n2. The baseline in Table 2 is a bit old and weak.  Vavae [1] achieves much better FID (2.17 wo CFG /1.35 w CFG, only 675M parameters) than the cited baselines. If compared with this baseline, the performance of this work is not that good.\n3. It's a bit strange to do the main understanding experiment using Liquid, which is not published yet by the DDL of NeurIPS.  VILA-U is a better choice (it provides the data and code).  Moreover, this setting can provide a controlled experiment for training data, which is well recognized critical for final performance. \n4. Does the performance in Table 3 come from better training data?\n5. Minor: It's better to include the performance on the GenEval [2] benchmark, because many unified models also cover it.\n\n[1] Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models, CVPR25\n[2] GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment, NeurIPS", "was_revised": true, "final_justification": "I maintain my initial score, as some of my concerns have been addressed.", "rebuttal": "Dear Reviewer emrz,\n\nThank you for your constructive feedback and thoughtful evaluation of our work. We hope our point-to-point response will address your concerns.\n\n**Q1: Fair comparison with RQ.**\n\nWe apologize for any misconceptions caused. In Table 7, we do implement RQ with the multi-code setting (i.e., using 8 codes to represent a single token), consistent with the MCQ approach. For RQ, the codes are generated by recursively quantizing each token, so the number of codes is determined by the quantization depth (D) rather than the number of codebooks (K). Following the official RQ implementation, we set D = 8 and K = 1 for the experiments in Table 7. This results in 16 × 16 × 8 codes per image, matching the configuration used in MCQ. We will clarify this point in the revised version of our paper.\n\n**Q2: Comparison with VAVAE.**\n\nThank you for the suggestion. We will include VAVAE for comparison in Table 2. However, we would like to clarify that VAVAE should not be considered a baseline for UniTok, as it is a continuous VAE tokenizer evaluated within the diffusion framework, whereas UniTok is a discrete VQVAE tokenizer evaluated under the autoregressive framework. This fundamental difference makes the two approaches less directly comparable. Instead, LlamaGen is a widely adopted baseline for discrete tokenizers. In Table 2, we strictly follow the LlamaGen implementation and only substitute the tokenizer with UniTok for evaluation. In this context, we believe that using LlamaGen as a baseline provides a more fair and meaningful comparison.\n\n**Q3: Why choose Liquid rather than VILA-U as the MLLM framework for evaluation?**\n\n- First, we would like to clarify that the earliest release of Liquid was on December 5, 2024, on arXiv, which is more than five months before the NeurIPS 2025 submission deadline.\n- Besides, the training data of VILA-U contains 15M internal data that are not publicly available. This makes it impossible to conduct controlled experiments with VILA-U.\n- We attempted to request access to the internal data used in both Liquid and VILA-U. However, only the authors of Liquid granted us access. Therefore, we selected Liquid as our evaluation codebase.\n\n**Q4: Does the performance in Table 3 come from better training data?**\n\nWe agree that data quality has a significant impact on final performance. However, most unified models today use different data recipes for training and often involve internal data (e.g., as in Chameleon, VILA-U, Show-o, Janus, Liquid, among others). This makes direct, apple-to-apple comparisons impractical. Nonetheless, we believe the tokenizer innovation in UniTok is the primary contributor to the final performance, as demonstrated in Figure 3.\n\n**Q5: Evaluating UniTok on the GenEval benchmark.**\n\nThank you for the suggestion. The table below presents a comparison of unified models on the GenEval benchmark. The overall trend is largely consistent with the results observed on GenAI-Bench, except that Show-o achieves better performance by scaling up the text-to-image training data to 2 billion samples. Notably, UniTok achieves non-trivial improvements over Liquid while using exactly the same set of text-to-image training data.\n\n| Method | # Data | Single Obj. | Two Obj. | Counting | Colors | Position | Color Attri. | Overall ↑ |\n|:---------|:-------:|:-------------:|:----------:|:----------:|:---------:|:---------:|:------------:|:----------:|\n| LWM | -- | 0.93 | 0.41 | 0.46 | 0.79 | 0.09 | 0.15 | 0.47 |\n| Chameleon | -- | -- | -- | -- | -- | -- | -- | 0.39 |\n| Show-o | 35M | 0.95 | 0.52 | 0.49 | 0.82 | 0.11 | 0.28 | 0.53 |\n| Show-o | 2.0B | 0.98 | 0.80 | 0.66 | 0.84 | 0.31 | 0.50 | 0.68 |\n| Janus | -- | 0.97 | 0.68 | 0.30 | 0.84 | 0.46 | 0.42 | 0.61 |\n| Liquid | 30M | 0.98 | 0.73 | 0.32 | 0.76 | 0.17 | 0.37 | 0.55 |\n| UniTok | 30M | 0.99 | 0.71 | 0.36 | 0.79 | 0.26 | 0.45 | 0.59 |\n\n#Data: Number of text-to-image data samples used in pretraining"}]

Author rebuttal:

Meta-review: UniTok studies a unified visual tokenizer that works for both generation and understanding by diagnosing that the apparent conflict between reconstruction and semantic supervision stems from the limited capacity of discrete token spaces rather than the losses themselves. The paper proposes multi codebook quantization with an attention based projection to expand both the effective vocabulary and the latent bottleneck, and shows state of the art image generation and recognition outcomes, including a reconstruction FID of 0.38 and 78.6 percent zero shot ImageNet accuracy, while enabling native generation inside MLLMs without harming understanding. It further reports large gains in CFG free generation, cutting gFID on ImageNet 256 by an order of magnitude, and backs the claims with broad ablations and analyses. Strengths are a clear problem formulation, a principled design that links analysis to method, extensive experiments across both modalities, and compelling practical integration into an MLLM. Weaknesses raised in review include fairness and coverage of baselines such as RQ settings and VAVAE, resolution alignment for LMM evaluation, reliance on Liquid rather than VILA U for some comparisons, possible data confounds in one table, and a request to include GenEval; there were also questions about why contrastive supervision aids generation and how reconstruction affects understanding capacity. Overall the AC find the technical contribution substantial and clean, the empirical evidence strong, and the paper well written and easy to follow.

During rebuttal the authors answered the main points thoroughly. For Reviewer rTfH, they explained that contrastive supervision injects semantics into code embeddings and that reconstruction can trade off high level capacity when the discrete bottleneck is too tight, which their larger multi codebook space alleviates; Reviewer rTfH kept a borderline accept after acknowledging the responses. For Reviewer Qdyc, they clarified that all LMM results used 256 by 256 to match training and reported successful generalization to 384 by 384 with a two point gain on TextVQA and little benefit at 512 by 512; Reviewer Qdyc raised to strong accept. For Reviewer uzhd, they disambiguated that Table 8 varied codebook groups while exponentially growing the combinatorial vocabulary and latent dimension, added an ablation showing that simply widening a single codebook to 64d reduces utilization and hurts rFID, and reconciled the joint training discussion by showing improved outcomes once quantization is fixed; Reviewer uzhd raised to accept. For Reviewer emrz, they clarified the multi code setting used for RQ, argued why VAVAE is not a directly comparable discrete baseline while agreeing to add it, justified choosing Liquid due to data access constraints, and provided a GenEval table where UniTok improves over Liquid under the same training data. Weighing these, the most important reasons for acceptance are that the paper isolates a widely observed failure mode to a concrete capacity bottleneck, proposes a simple and general fix that materially advances both generation and understanding in one tokenizer, and validates it across strong baselines with careful ablations and practical MLLM integration. Hence, the AC recommends acceptance.
Decision: Accept (spotlight)