GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields
  through Efficient Dense 3D Point Tracking
Authors: name: Weikang Bian, name: Zhaoyang Huang, name: Xiaoyu Shi, name: Yijin Li, name: Fu-Yun Wang, name: Hongsheng Li

4D video control is essential in video generation as it enables the use of
sophisticated lens techniques, such as multi-camera shooting and dolly zoom,
which are currently unsupported by existing methods. Training a video Diffusion
Transformer (DiT) directly to control 4D content requires expensive multi-view
videos. Inspired by Monocular Dynamic novel View Synthesis (MDVS) that
optimizes a 4D representation and renders videos according to different 4D
elements, such as camera pose and object motion editing, we bring pseudo 4D
Gaussian fields to video generation. Specifically, we propose a novel framework
that constructs a pseudo 4D Gaussian field with dense 3D point tracking and
renders the Gaussian field for all video frames. Then we finetune a pretrained
DiT to generate videos following the guidance of the rendered video, dubbed as
GS-DiT. To boost the training of the GS-DiT, we also propose an efficient Dense
3D Point Tracking (D3D-PT) method for the pseudo 4D Gaussian field
construction. Our D3D-PT outperforms SpatialTracker, the state-of-the-art
sparse 3D point tracking method, in accuracy and accelerates the inference
speed by two orders of magnitude. During the inference stage, GS-DiT can
generate videos with the same dynamic content while adhering to different
camera parameters, addressing a significant limitation of current video
generation models. GS-DiT demonstrates strong generalization capabilities and
extends the 4D controllability of Gaussian splatting to video generation beyond
just camera poses. It supports advanced cinematic effects through the
manipulation of the Gaussian field and camera intrinsics, making it a powerful
tool for creative video production. Demos are available at
https://wkbian.github.io/Projects/GS-DiT/.