UniTok: A Unified Tokenizer for Visual Generation and Understanding
Authors: name: Chuofan Ma; name: Yi Jiang; name: Junfeng Wu; name: Jihan Yang; name: Xin Yu; name: Zehuan Yuan; name: Bingyue Peng; name: Xiaojuan Qi
Published: 2025-02-27 17:47:01

Abstract: The representation disparity between visual generation and understanding
imposes a critical gap in integrating these capabilities into a single
framework. To bridge this gap, we introduce UniTok, a discrete visual tokenizer
that encodes fine-grained details for generation while also capturing
high-level semantics for understanding. Despite recent studies have shown that
these objectives could induce loss conflicts in training, we reveal that the
underlying bottleneck stems from limited representational capacity of discrete
tokens. We address this by introducing multi-codebook quantization, which
divides vector quantization with several independent sub-codebooks to expand
the latent feature space, while avoiding training instability caused by
overlarge codebooks. Our method significantly raises the upper limit of unified
discrete tokenizers to match or even surpass domain-specific continuous
tokenizers. For instance, UniTok achieves a remarkable rFID of 0.38 (versus
0.87 for SD-VAE) and a zero-shot accuracy of 78.6% (versus 76.2% for CLIP) on
ImageNet. Our code is available at https://github.com/FoundationVision/UniTok.