Title: UniTok: a Unified Tokenizer for Visual Generation and Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Visual Tokenizer

Multimodal Generation and Understanding
Abstract: Visual generative and understanding models typically rely on distinct tokenizers to process images, presenting a key challenge for unifying them within a single framework. Recent studies attempt to address this by connecting the training of VQVAE (for autoregressive generation) and CLIP (for understanding) to build a unified tokenizer. However, directly combining these training objectives has been observed to cause severe loss conflicts. In this paper, we show that reconstruction and semantic supervision do not inherently conflict. Instead, the underlying bottleneck stems from limited representational capacity of discrete token space. Building on these insights, we introduce UniTok, a unified tokenizer featuring a novel multi-codebook quantization mechanism that effectively scales up the vocabulary size and bottleneck dimension. In terms of final performance, UniTok sets a new record of 0.38 rFID and 78.6\% zero-shot accuracy on ImageNet. Besides, UniTok can be seamlessly integrated into MLLMs to unlock native visual generation capability, without compromising the understanding performance. Additionally, we show that UniTok favors cfg-free generation, reducing gFID from 14.6 to 2.5 on ImageNet 256$\times$256 benchmark. All codes and models have been made publicly available.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies how to design a unified discrete tokeniser for both image understanding and generation tasks. It identifies the bottle necks of directly combining CLIP and VQVAE: improper compression from the network architecture, a single codebook. They made some improvements to enhance the representative capacity of the discrete token space. Experimental results verify the validity of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. This paper studies an interesting topic and reveals that under proper design, the discrete tokenizer can achieve good performance on both understanding and generation tasks.\n2. The paper is well oragnzied and  easy to follow.\n\nWeakness\n1. It seems that the comparison in Table 7 is unfair. RQ also support multi codes/groups, therefore, 1×32768  (K=1)is not fair for using the RQ baseline. Can you show the results for RQ(K=4 or 8?)?\n2. The baseline in Table 2 is a bit old and weak.  Vavae [1] achieves much better FID (2.17 wo CFG /1.35 w CFG, only 675M parameters) than the cited baselines. If compared with this baseline, the performance of this work is not that good.\n3. It's a bit strange to do the main understanding experiment using Liquid, which is not published yet by the DDL of NeurIPS.  VILA-U is a better choice (it provides the data and code).  Moreover, this setting can provide a controlled experiment for training data, which is well recognized critical for final performance. \n4. Does the performance in Table 3 come from better training data?\n5. Minor: It's better to include the performance on the GenEval [2] benchmark, because many unified models also cover it.\n\n[1] Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models, CVPR25\n[2] GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment, NeurIPS\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript focuses on developing effective tokenizer for unified understanding and generation. The authors analyze the potential problems of joint training with reconstruction and semantic supervision. Experiments show that the challenge lies in fabrication and discretization. To improve the vocabulary size and representation, the authors propose multi codebook quantization (MCQ) and utilize modified multi-head attention to replace the linear projection. The proposed method achieves excellent performance on both understanding and generation tasks.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths:\n1. Tokenizer is a critical part for unified multi-modal models. This work propose an effective tokenizer for balancing understanding and generation performance. Comparison with most state-of-the-art methods demonstrate its effectiveness. \n\n2. The motivation of addressing the poor representation of discrete tokenizer is clear. The proposed methods seem to reasonable.\n\n3. The paper is well-written and easy to follow.\n\n- Weaknesses:\n1. The experiment results seem to inconsistent with the analysis. Specifically, the authors claim that the MQC is mainly designed to expand the vocabulary size. However, from Table 8 it can be seen that the performance increases as the number of codebook groups increases while the total vocabulary size remains unchanged. It is more like multi-head attention that utilizes more sub-space.\n\n2. Figure 3 utilizes VQA score for analysis and draw the conclusion that the challenge of unified tokenizer is not the joint training. However, the authors claim in L144: \"We observe a similar phenomenon where joint training results in sub-optimal ImageNet zero-shot classification accuracy and reconstruction FID compared to specialized training.\" It seems that the conclusions are inconsistent and are related to the tasks.\n\nQUESTIONS:\n* The authors claim that MCQ can also boost the dimension of the vocabulary. However, the authors do not analyze or compare with the method that directly increases the dimension of a single codebook.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper introduces UniTok, a unified tokenizer designed to bridge the gap between visual generation and understanding. UniTok features a novel multi-codebook quantization mechanism that increases the tokenizer’s capacity and enables it to scale effectively. It achieves a record low reconstruction FID of 0.38 and 78.6% zero-shot accuracy on ImageNet. UniTok can be seamlessly integrated into multimodal large language models, enabling visual generation capabilities without compromising performance in understanding tasks.

STRENGTHS AND WEAKNESSES:
**Strengths**

1) The paper is well-written and easy to read, with a logical flow that addresses both the technical details and experimental results effectively.
2) The paper presents a well-structured approach to visual tokenization, demonstrating solid experimental results. The multi-codebook quantization significantly improves performance over traditional VQ-based methods.
3) UniTok represents a important step forward in solving the problem of unified tokenization for both visual generation and understanding.
4) The use of multi-codebook quantization to effective. 

Weakness:
1) The vision encoder is trained in 256x256 resolution. Will the LMM tasks in Table 6 need higher resolution? Can the author align the resolution when training the tokenizer?

QUESTIONS:
1) Can the tokenizer be enhanced by captioning loss (e.g., COCA)? This may enhance the downstream VQA task too.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03

comments: SUMMARY:
This paper proposes studying a visual tokenizer, **UniTok**, which can be effectively used for both generation and understanding tasks in MLLMs. The authors conduct extensive experimental analysis to investigate the importance of multi-codebook quantization and attention projection in their model design.

STRENGTHS AND WEAKNESSES:
## Pros
* The writing is clear and easy to follow.
* The authors have conducted comprehensive experiments to analyze the bottlenecks of the tokenizer in unifying different tasks.
* The tokenizer demonstrates strong empirical performance in both generation and MLLM understanding tasks.


## Cons
These are not necessarily weaknesses, but addressing them may help further strengthen the paper:
* Table 6 shows that supervision from reconstruction loss and contrastive loss have different impacts on downstream tasks: adding contrastive loss further improves generation, but adding reconstruction loss often decreases performance on understanding tasks. Do the authors have any insights into why this occurs?
* Given the rapid development of this field, the authors might also consider including additional related work for discussion, such as [1].

[1] Hansen-Estruch, Philippe, et al. "Learnings from Scaling Visual Tokenizers for Reconstruction and Generation." *arXiv preprint* arXiv:2501.09755 (2025).

QUESTIONS:
Please see **Cons**

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03