MonST3R: A Simple Approach for Estimating Geometry in the Presence of
  Motion
Authors: name: Junyi Zhang; name: Charles Herrmann; name: Junhwa Hur; name: Varun Jampani; name: Trevor Darrell; name: Forrester Cole; name: Deqing Sun; name: Ming-Hsuan Yang
Published: 2024-10-04 18:00:07

Abstract: Estimating geometry from dynamic scenes, where objects move and deform over
time, remains a core challenge in computer vision. Current approaches often
rely on multi-stage pipelines or global optimizations that decompose the
problem into subtasks, like depth and flow, leading to complex systems prone to
errors. In this paper, we present Motion DUSt3R (MonST3R), a novel
geometry-first approach that directly estimates per-timestep geometry from
dynamic scenes. Our key insight is that by simply estimating a pointmap for
each timestep, we can effectively adapt DUST3R's representation, previously
only used for static scenes, to dynamic scenes. However, this approach presents
a significant challenge: the scarcity of suitable training data, namely
dynamic, posed videos with depth labels. Despite this, we show that by posing
the problem as a fine-tuning task, identifying several suitable datasets, and
strategically training the model on this limited data, we can surprisingly
enable the model to handle dynamics, even without an explicit motion
representation. Based on this, we introduce new optimizations for several
downstream video-specific tasks and demonstrate strong performance on video
depth and camera pose estimation, outperforming prior work in terms of
robustness and efficiency. Moreover, MonST3R shows promising results for
primarily feed-forward 4D reconstruction.