Gaussian Variation Field Diffusion for High-fidelity Video-to-4D
  Synthesis
Authors: name: Bowen Zhang, name: Sicheng Xu, name: Chuxin Wang, name: Jiaolong Yang, name: Feng Zhao, name: Dong Chen, name: Baining Guo

In this paper, we present a novel framework for video-to-4D generation that
creates high-quality dynamic 3D content from single video inputs. Direct 4D
diffusion modeling is extremely challenging due to costly data construction and
the high-dimensional nature of jointly representing 3D shape, appearance, and
motion. We address these challenges by introducing a Direct 4DMesh-to-GS
Variation Field VAE that directly encodes canonical Gaussian Splats (GS) and
their temporal variations from 3D animation data without per-instance fitting,
and compresses high-dimensional animations into a compact latent space.
Building upon this efficient representation, we train a Gaussian Variation
Field diffusion model with temporal-aware Diffusion Transformer conditioned on
input videos and canonical GS. Trained on carefully-curated animatable 3D
objects from the Objaverse dataset, our model demonstrates superior generation
quality compared to existing methods. It also exhibits remarkable
generalization to in-the-wild video inputs despite being trained exclusively on
synthetic data, paving the way for generating high-quality animated 3D content.
Project page: https://gvfdiffusion.github.io/.