GeometryCrafter: Consistent Geometry Estimation for Open-world Videos
  with Diffusion Priors
Authors: name: Tian-Xing Xu, name: Xiangjun Gao, name: Wenbo Hu, name: Xiaoyu Li, name: Song-Hai Zhang, name: Ying Shan

Despite remarkable advancements in video depth estimation, existing methods
exhibit inherent limitations in achieving geometric fidelity through the
affine-invariant predictions, limiting their applicability in reconstruction
and other metrically grounded downstream tasks. We propose GeometryCrafter, a
novel framework that recovers high-fidelity point map sequences with temporal
coherence from open-world videos, enabling accurate 3D/4D reconstruction,
camera parameter estimation, and other depth-based applications. At the core of
our approach lies a point map Variational Autoencoder (VAE) that learns a
latent space agnostic to video latent distributions for effective point map
encoding and decoding. Leveraging the VAE, we train a video diffusion model to
model the distribution of point map sequences conditioned on the input videos.
Extensive evaluations on diverse datasets demonstrate that GeometryCrafter
achieves state-of-the-art 3D accuracy, temporal consistency, and generalization
capability.