DepthCrafter: Generating Consistent Long Depth Sequences for Open-world
  Videos
Authors: name: Wenbo Hu; name: Xiangjun Gao; name: Xiaoyu Li; name: Sijie Zhao; name: Xiaodong Cun; name: Yong Zhang; name: Long Quan; name: Ying Shan
Published: 2024-09-03 17:52:03

Abstract: Despite significant advancements in monocular depth estimation for static
images, estimating video depth in the open world remains challenging, since
open-world videos are extremely diverse in content, motion, camera movement,
and length. We present DepthCrafter, an innovative method for generating
temporally consistent long depth sequences with intricate details for
open-world videos, without requiring any supplementary information such as
camera poses or optical flow. DepthCrafter achieves generalization ability to
open-world videos by training a video-to-depth model from a pre-trained
image-to-video diffusion model, through our meticulously designed three-stage
training strategy with the compiled paired video-depth datasets. Our training
approach enables the model to generate depth sequences with variable lengths at
one time, up to 110 frames, and harvest both precise depth details and rich
content diversity from realistic and synthetic datasets. We also propose an
inference strategy that processes extremely long videos through segment-wise
estimation and seamless stitching. Comprehensive evaluations on multiple
datasets reveal that DepthCrafter achieves state-of-the-art performance in
open-world video depth estimation under zero-shot settings. Furthermore,
DepthCrafter facilitates various downstream applications, including depth-based
visual effects and conditional video generation.