Cavia: Camera-controllable Multi-view Video Diffusion with
  View-Integrated Attention
Authors: name: Dejia Xu; name: Yifan Jiang; name: Chen Huang; name: Liangchen Song; name: Thorsten Gernoth; name: Liangliang Cao; name: Zhangyang Wang; name: Hao Tang
Published: 2024-10-14 17:46:32

Abstract: In recent years there have been remarkable breakthroughs in image-to-video
generation. However, the 3D consistency and camera controllability of generated
frames have remained unsolved. Recent studies have attempted to incorporate
camera control into the generation process, but their results are often limited
to simple trajectories or lack the ability to generate consistent videos from
multiple distinct camera paths for the same scene. To address these
limitations, we introduce Cavia, a novel framework for camera-controllable,
multi-view video generation, capable of converting an input image into multiple
spatiotemporally consistent videos. Our framework extends the spatial and
temporal attention modules into view-integrated attention modules, improving
both viewpoint and temporal consistency. This flexible design allows for joint
training with diverse curated data sources, including scene-level static
videos, object-level synthetic multi-view dynamic videos, and real-world
monocular dynamic videos. To our best knowledge, Cavia is the first of its kind
that allows the user to precisely specify camera motion while obtaining object
motion. Extensive experiments demonstrate that Cavia surpasses state-of-the-art
methods in terms of geometric consistency and perceptual quality. Project Page:
https://ir1d.github.io/Cavia/