VideoFactory: Swap Attention in Spatiotemporal Diffusions for
  Text-to-Video Generation
Authors: name: Wenjing Wang; name: Huan Yang; name: Zixi Tuo; name: Huiguo He; name: Junchen Zhu; name: Jianlong Fu; name: Jiaying Liu
Published: 2023-05-18 11:06:15

Abstract: We present VideoFactory, an innovative framework for generating high-quality
open-domain videos. VideoFactory excels in producing high-definition
(1376x768), widescreen (16:9) videos without watermarks, creating an engaging
user experience. Generating videos guided by text instructions poses
significant challenges, such as modeling the complex relationship between space
and time, and the lack of large-scale text-video paired data. Previous
approaches extend pretrained text-to-image generation models by adding temporal
1D convolution/attention modules for video generation. However, these
approaches overlook the importance of jointly modeling space and time,
inevitably leading to temporal distortions and misalignment between texts and
videos. In this paper, we propose a novel approach that strengthens the
interaction between spatial and temporal perceptions. In particular, we utilize
a swapped cross-attention mechanism in 3D windows that alternates the "query"
role between spatial and temporal blocks, enabling mutual reinforcement for
each other. To fully unlock model capabilities for high-quality video
generation, we curate a large-scale video dataset called HD-VG-130M. This
dataset comprises 130 million text-video pairs from the open-domain, ensuring
high-definition, widescreen and watermark-free characters. Objective metrics
and user studies demonstrate the superiority of our approach in terms of
per-frame quality, temporal correlation, and text-video alignment, with clear
margins.