Video-Infinity: Distributed Long Video Generation
Authors: name: Zhenxiong Tan; name: Xingyi Yang; name: Songhua Liu; name: Xinchao Wang
Published: 2024-06-24 01:56:12

Abstract: Diffusion models have recently achieved remarkable results for video
generation. Despite the encouraging performances, the generated videos are
typically constrained to a small number of frames, resulting in clips lasting
merely a few seconds. The primary challenges in producing longer videos include
the substantial memory requirements and the extended processing time required
on a single GPU. A straightforward solution would be to split the workload
across multiple GPUs, which, however, leads to two issues: (1) ensuring all
GPUs communicate effectively to share timing and context information, and (2)
modifying existing video diffusion models, which are usually trained on short
sequences, to create longer videos without additional training. To tackle
these, in this paper we introduce Video-Infinity, a distributed inference
pipeline that enables parallel processing across multiple GPUs for long-form
video generation. Specifically, we propose two coherent mechanisms: Clip
parallelism and Dual-scope attention. Clip parallelism optimizes the gathering
and sharing of context information across GPUs which minimizes communication
overhead, while Dual-scope attention modulates the temporal self-attention to
balance local and global contexts efficiently across the devices. Together, the
two mechanisms join forces to distribute the workload and enable the fast
generation of long videos. Under an 8 x Nvidia 6000 Ada GPU (48G) setup, our
method generates videos up to 2,300 frames in approximately 5 minutes, enabling
long video generation at a speed 100 times faster than the prior methods.