Collaborative Video Diffusion: Consistent Multi-video Generation with
  Camera Control
Authors: name: Zhengfei Kuang; name: Shengqu Cai; name: Hao He; name: Yinghao Xu; name: Hongsheng Li; name: Leonidas Guibas; name: Gordon Wetzstein
Published: 2024-05-27 17:58:01

Abstract: Research on video generation has recently made tremendous progress, enabling
high-quality videos to be generated from text prompts or images. Adding control
to the video generation process is an important goal moving forward and recent
approaches that condition video generation models on camera trajectories make
strides towards it. Yet, it remains challenging to generate a video of the same
scene from multiple different camera trajectories. Solutions to this
multi-video generation problem could enable large-scale 3D scene generation
with editable camera trajectories, among other applications. We introduce
collaborative video diffusion (CVD) as an important step towards this vision.
The CVD framework includes a novel cross-video synchronization module that
promotes consistency between corresponding frames of the same video rendered
from different camera poses using an epipolar attention mechanism. Trained on
top of a state-of-the-art camera-control module for video generation, CVD
generates multiple videos rendered from different camera trajectories with
significantly better consistency than baselines, as shown in extensive
experiments. Project page: https://collaborativevideodiffusion.github.io/.