Aligning Visual Foundation Encoders to Tokenizers for Diffusion Models
Authors: name: Bowei Chen; name: Sai Bi; name: Hao Tan; name: He Zhang; name: Tianyuan Zhang; name: Zhengqi Li; name: Yuanjun Xiong; name: Jianming Zhang; name: Kai Zhang
Published: 2025-09-29 17:57:39

Abstract: In this work, we propose aligning pretrained visual encoders to serve as
tokenizers for latent diffusion models in image generation. Unlike training a
variational autoencoder (VAE) from scratch, which primarily emphasizes
low-level details, our approach leverages the rich semantic structure of
foundation encoders. We introduce a three-stage alignment strategy: (1) freeze
the encoder and train an adapter and a decoder to establish a semantic latent
space; (2) jointly optimize all components with an additional semantic
preservation loss, enabling the encoder to capture perceptual details while
retaining high-level semantics; and (3) refine the decoder for improved
reconstruction quality. This alignment yields semantically rich image
tokenizers that benefit diffusion models. On ImageNet 256times256, our
tokenizer accelerates the convergence of diffusion models, reaching a gFID of
1.90 within just 64 epochs, and improves generation both with and without
classifier-free guidance. Scaling to LAION, a 2B-parameter text-to-image model
trained with our tokenizer consistently outperforms FLUX VAE under the same
training steps. Overall, our method is simple, scalable, and establishes a
semantically grounded paradigm for continuous tokenizer design.