FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion
  Model
Authors: name: ZiDong Wang; name: Zeyu Lu; name: Di Huang; name: Cai Zhou; name: Wanli Ouyang; name: and Lei Bai
Published: 2024-10-17 15:51:49

Abstract: Nature is infinitely resolution-free. In the context of this
reality, existing diffusion models, such as Diffusion Transformers, often face
challenges when processing image resolutions outside of their trained domain.
To address this limitation, we conceptualize images as sequences of tokens with
dynamic sizes, rather than traditional methods that perceive images as
fixed-resolution grids. This perspective enables a flexible training strategy
that seamlessly accommodates various aspect ratios during both training and
inference, thus promoting resolution generalization and eliminating biases
introduced by image cropping. On this basis, we present the Flexible
Vision Transformer (FiT), a transformer architecture specifically designed for
generating images with unrestricted resolutions and aspect ratios. We
further upgrade the FiT to FiTv2 with several innovative designs, includingthe
Query-Key vector normalization, the AdaLN-LoRA module, a rectified flow
scheduler, and a Logit-Normal sampler. Enhanced by a meticulously adjusted
network structure, FiTv2 exhibits 2times convergence speed of FiT. When
incorporating advanced training-free extrapolation techniques, FiTv2
demonstrates remarkable adaptability in both resolution extrapolation and
diverse resolution generation. Additionally, our exploration of the scalability
of the FiTv2 model reveals that larger models exhibit better computational
efficiency. Furthermore, we introduce an efficient post-training strategy to
adapt a pre-trained model for the high-resolution generation. Comprehensive
experiments demonstrate the exceptional performance of FiTv2 across a broad
range of resolutions. We have released all the codes and models at
https://github.com/whlzy/FiT to promote the exploration of diffusion
transformer models for arbitrary-resolution image generation.