Controlling Space and Time with Diffusion Models
Authors: name: Daniel Watson; name: Saurabh Saxena; name: Lala Li; name: Andrea Tagliasacchi; name: David J. Fleet
Published: 2024-07-10 17:23:33

Abstract: We present 4DiM, a cascaded diffusion model for 4D novel view synthesis
(NVS), conditioned on one or more images of a general scene, and a set of
camera poses and timestamps. To overcome challenges due to limited availability
of 4D training data, we advocate joint training on 3D (with camera pose), 4D
(pose+time) and video (time but no pose) data and propose a new architecture
that enables the same. We further advocate the calibration of SfM posed data
using monocular metric depth estimators for metric scale camera control. For
model evaluation, we introduce new metrics to enrich and overcome shortcomings
of current evaluation schemes, demonstrating state-of-the-art results in both
fidelity and pose control compared to existing diffusion models for 3D NVS,
while at the same time adding the ability to handle temporal dynamics. 4DiM is
also used for improved panorama stitching, pose-conditioned video to video
translation, and several other tasks. For an overview see
https://4d-diffusion.github.io