FreeLong: Training-Free Long Video Generation with SpectralBlend
  Temporal Attention
Authors: name: Yu Lu; name: Yuanzhi Liang; name: Linchao Zhu; name: Yi Yang
Published: 2024-07-29 11:52:07

Abstract: Video diffusion models have made substantial progress in various video
generation applications. However, training models for long video generation
tasks require significant computational and data resources, posing a challenge
to developing long video diffusion models. This paper investigates a
straightforward and training-free approach to extend an existing short video
diffusion model (e.g. pre-trained on 16-frame videos) for consistent long video
generation (e.g. 128 frames). Our preliminary observation has found that
directly applying the short video diffusion model to generate long videos can
lead to severe video quality degradation. Further investigation reveals that
this degradation is primarily due to the distortion of high-frequency
components in long videos, characterized by a decrease in spatial
high-frequency components and an increase in temporal high-frequency
components. Motivated by this, we propose a novel solution named FreeLong to
balance the frequency distribution of long video features during the denoising
process. FreeLong blends the low-frequency components of global video features,
which encapsulate the entire video sequence, with the high-frequency components
of local video features that focus on shorter subsequences of frames. This
approach maintains global consistency while incorporating diverse and
high-quality spatiotemporal details from local videos, enhancing both the
consistency and fidelity of long video generation. We evaluated FreeLong on
multiple base video diffusion models and observed significant improvements.
Additionally, our method supports coherent multi-prompt generation, ensuring
both visual coherence and seamless transitions between scenes.