I2V-Adapter: A General Image-to-Video Adapter for Video Diffusion Models
Authors: name: Xun Guo; name: Mingwu Zheng; name: Liang Hou; name: Yuan Gao; name: Yufan Deng; name: Chongyang Ma; name: Weiming Hu; name: Zhengjun Zha; name: Haibin Huang; name: Pengfei Wan; name: Di Zhang
Published: 2023-12-27 19:11:50

Abstract: In the rapidly evolving domain of digital content generation, the focus has
shifted from text-to-image (T2I) models to more advanced video diffusion
models, notably text-to-video (T2V) and image-to-video (I2V). This paper
addresses the intricate challenge posed by I2V: converting static images into
dynamic, lifelike video sequences while preserving the original image fidelity.
Traditional methods typically involve integrating entire images into diffusion
processes or using pretrained encoders for cross attention. However, these
approaches often necessitate altering the fundamental weights of T2I models,
thereby restricting their reusability. We introduce a novel solution, namely
I2V-Adapter, designed to overcome such limitations. Our approach preserves the
structural integrity of T2I models and their inherent motion modules. The
I2V-Adapter operates by processing noised video frames in parallel with the
input image, utilizing a lightweight adapter module. This module acts as a
bridge, efficiently linking the input to the model's self-attention mechanism,
thus maintaining spatial details without requiring structural changes to the
T2I model. Moreover, I2V-Adapter requires only a fraction of the parameters of
conventional models and ensures compatibility with existing community-driven
T2I models and controlling tools. Our experimental results demonstrate
I2V-Adapter's capability to produce high-quality video outputs. This
performance, coupled with its versatility and reduced need for trainable
parameters, represents a substantial advancement in the field of AI-driven
video generation, particularly for creative applications.