Image Tokenizer Needs Post-Training
Authors: name: Kai Qiu, name: Xiang Li, name: Hao Chen, name: Jason Kuen, name: Xiaohao Xu, name: Jiuxiang Gu, name: Yinyi Luo, name: Bhiksha Raj, name: Zhe Lin, name: Marios Savvides

Recent image generative models typically capture the image distribution in a
pre-constructed latent space, relying on a frozen image tokenizer. However,
there exists a significant discrepancy between the reconstruction and
generation distribution, where current tokenizers only prioritize the
reconstruction task that happens before generative training without considering
the generation errors during sampling. In this paper, we comprehensively
analyze the reason for this discrepancy in a discrete latent space, and, from
which, we propose a novel tokenizer training scheme including both
main-training and post-training, focusing on improving latent space
construction and decoding respectively. During the main training, a latent
perturbation strategy is proposed to simulate sampling noises, \ie, the
unexpected tokens generated in generative inference. Specifically, we propose a
plug-and-play tokenizer training scheme, which significantly enhances the
robustness of tokenizer, thus boosting the generation quality and convergence
speed, and a novel tokenizer evaluation metric, \ie, pFID, which successfully
correlates the tokenizer performance to generation quality. During
post-training, we further optimize the tokenizer decoder regarding a
well-trained generative model to mitigate the distribution difference between
generated and reconstructed tokens. With a sim400M generator, a discrete
tokenizer trained with our proposed main training achieves a notable 1.60 gFID
and further obtains 1.36 gFID with the additional post-training. Further
experiments are conducted to broadly validate the effectiveness of our
post-training strategy on off-the-shelf discrete and continuous tokenizers,
coupled with autoregressive and diffusion-based generators.