Video Depth without Video Models
Authors: name: Bingxin Ke; name: Dominik Narnhofer; name: Shengyu Huang; name: Lei Ke; name: Torben Peters; name: Katerina Fragkiadaki; name: Anton Obukhov; name: Konrad Schindler
Published: 2024-11-28 14:50:14

Abstract: Video depth estimation lifts monocular video clips to 3D by inferring dense
depth at every frame. Recent advances in single-image depth estimation, brought
about by the rise of large foundation models and the use of synthetic training
data, have fueled a renewed interest in video depth. However, naively applying
a single-image depth estimator to every frame of a video disregards temporal
continuity, which not only leads to flickering but may also break when camera
motion causes sudden changes in depth range. An obvious and principled solution
would be to build on top of video foundation models, but these come with their
own limitations; including expensive training and inference, imperfect 3D
consistency, and stitching routines for the fixed-length (short) outputs. We
take a step back and demonstrate how to turn a single-image latent diffusion
model (LDM) into a state-of-the-art video depth estimator. Our model, which we
call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator
that is derived from a single-image LDM and maps very short video snippets
(typically frame triplets) to depth snippets. (ii) a robust, optimization-based
registration algorithm that optimally assembles depth snippets sampled at
various different frame rates back into a consistent video. RollingDepth is
able to efficiently handle long videos with hundreds of frames and delivers
more accurate depth videos than both dedicated video depth estimators and
high-performing single-frame models. Project page: rollingdepth.github.io.