FreeInit: Bridging Initialization Gap in Video Diffusion Models
Authors: name: Tianxing Wu; name: Chenyang Si; name: Yuming Jiang; name: Ziqi Huang; name: Ziwei Liu
Published: 2023-12-12 18:59:16

Abstract: Though diffusion-based video generation has witnessed rapid progress, the
inference results of existing models still exhibit unsatisfactory temporal
consistency and unnatural dynamics. In this paper, we delve deep into the noise
initialization of video diffusion models, and discover an implicit
training-inference gap that attributes to the unsatisfactory inference quality.
Our key findings are: 1) the spatial-temporal frequency distribution of the
initial latent at inference is intrinsically different from that for training,
and 2) the denoising process is significantly influenced by the low-frequency
components of the initial noise. Motivated by these observations, we propose a
concise yet effective inference sampling strategy, FreeInit, which
significantly improves temporal consistency of videos generated by diffusion
models. Through iteratively refining the spatial-temporal low-frequency
components of the initial latent during inference, FreeInit is able to
compensate the initialization gap between training and inference, thus
effectively improving the subject appearance and temporal consistency of
generation results. Extensive experiments demonstrate that FreeInit
consistently enhances the generation results of various text-to-video
generation models without additional training.