Marigold-DC: Zero-Shot Monocular Depth Completion with Guided Diffusion
Authors: name: Massimiliano Viola; name: Kevin Qu; name: Nando Metzger; name: Bingxin Ke; name: Alexander Becker; name: Konrad Schindler; name: Anton Obukhov
Published: 2024-12-18 00:06:41

Abstract: Depth completion upgrades sparse depth measurements into dense depth maps
guided by a conventional image. Existing methods for this highly ill-posed task
operate in tightly constrained settings and tend to struggle when applied to
images outside the training domain or when the available depth measurements are
sparse, irregularly distributed, or of varying density. Inspired by recent
advances in monocular depth estimation, we reframe depth completion as an
image-conditional depth map generation guided by sparse measurements. Our
method, Marigold-DC, builds on a pretrained latent diffusion model for
monocular depth estimation and injects the depth observations as test-time
guidance via an optimization scheme that runs in tandem with the iterative
inference of denoising diffusion. The method exhibits excellent zero-shot
generalization across a diverse range of environments and handles even
extremely sparse guidance effectively. Our results suggest that contemporary
monocular depth priors greatly robustify depth completion: it may be better to
view the task as recovering dense depth from (dense) image pixels, guided by
sparse depth; rather than as inpainting (sparse) depth, guided by an image.
Project website: https://MarigoldDepthCompletion.github.io/