4Real-Video: Learning Generalizable Photo-Realistic 4D Video Diffusion
Authors: name: Chaoyang Wang; name: Peiye Zhuang; name: Tuan Duc Ngo; name: Willi Menapace; name: Aliaksandr Siarohin; name: Michael Vasilkovsky; name: Ivan Skorokhodov; name: Sergey Tulyakov; name: Peter Wonka; name: Hsin-Ying Lee
Published: 2024-12-05 18:59:41

Abstract: We propose 4Real-Video, a novel framework for generating 4D videos, organized
as a grid of video frames with both time and viewpoint axes. In this grid, each
row contains frames sharing the same timestep, while each column contains
frames from the same viewpoint. We propose a novel two-stream architecture. One
stream performs viewpoint updates on columns, and the other stream performs
temporal updates on rows. After each diffusion transformer layer, a
synchronization layer exchanges information between the two token streams. We
propose two implementations of the synchronization layer, using either hard or
soft synchronization. This feedforward architecture improves upon previous work
in three ways: higher inference speed, enhanced visual quality (measured by
FVD, CLIP, and VideoScore), and improved temporal and viewpoint consistency
(measured by VideoScore and Dust3R-Confidence).