MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion

## Abstract

Abstract Estimating geometry from dynamic scenes, where objects move and deform over time, remains a core challenge in computer vision. Current approaches often rely on multi-stage pipelines or global optimizations that decompose the problem into subtasks, like depth and flow, leading to complex systems prone to errors. In this paper, we present Mo tio n DU St3R (MonST3R), a novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes.
Our key insight is that by simply estimating a pointmap for each timestep, we can effectively adapt DUSt3R’s representation, previously only used for static scenes, to dynamic scenes.
However, this approach presents a significant challenge: the scarcity of suitable training data, namely dynamic, posed videos with depth labels.
Despite this, we show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics, even without an explicit motion representation.
Based on this, we introduce new optimizations for several downstream video-specific tasks and demonstrate strong performance on video depth and camera pose estimation, outperforming prior work in terms of robustness and efficiency.
Moreover, MonST3R shows promising results for primarily feed-forward 4D reconstruction.
Interactive 4D results, source code, and trained models will be available at: https://monst3r-project.github.io/ .

## 1 Introduction

Despite recent progress in 3D computer vision, estimating geometry from videos of dynamic scenes remains a fundamental challenge.
Traditional methods decompose the problem into subproblems such as depth, optical flow, or trajectory estimation, addressed with specialized techniques, and then combine them through global optimization or multi-stage algorithms for dynamic scene reconstruction (Luiten et al., ; Kumar et al., ; Bârsan et al., ; Mustafa et al., ).
Even recent work often takes optimization-based approaches given intermediate estimates derived from monocular video (Lei et al., ; Chu et al., ; Wang et al., ; Liu et al., ; Wang et al., ). However, these multi-stage methods are usually slow, brittle, and prone to error at each step.

While highly desirable, end-to-end geometry learning from a dynamic video poses a significant challenge, requiring a suitable representation that can represent the complexities of camera motion, multiple object motion, and geometric deformations, along with annotated training datasets.
While prior methods have centered on the combination of motion and geometry, motion is often difficult to directly supervise due to lack of annotated training data.
Instead, we explore using *only* geometry to represent dynamic scenes, inspired by the recent work DUSt3R (Wang et al., ).

For static scenes, DUSt3R introduces a new paradigm that directly regresses scene geometry.
Given a pair of images, DUSt3R produces a pointmap representation - which associates every pixel in each image with an estimated 3D location (*i.e*., $xyz$) and aligns these pair of pointmaps in the camera coordinate system of the first frame. For multiple frames, DUSt3R accumulates the pairwise estimates into a global point cloud and uses it to solve numerous standard 3D tasks such as single-frame depth, multi-frame depth, or camera intrinsics and extrinsics.

We leverage DUSt3R’s pointmap representation to directly estimate geometry of dynamic scenes.
Our key insight is that pointmaps can be estimated per timestep and that representing them in the same camera coordinate frame still makes conceptual sense for dynamic scenes. As shown in Fig. 1, an estimated pointmap for the dynamic scene appears as a point cloud where dynamic objects appear at multiple locations, according to how they move.
Multi-frame alignment can be achieved by aligning pairs of pointmaps based on static scene elements.
This setting is a generalization of DUSt3R to dynamic scenes and allows us to use the same network and original weights as a starting point.

One natural question is if DUSt3R can already and effectively handle video data with moving objects. However, as shown in Fig. 2, we identify two significant limitations stemming from the distribution of DUSt3R’s training data.
First, since its training data contains only static scenes, DUSt3R fails to correctly align pointmaps of scenes with moving objects; it often relies on moving foreground objects for alignment, resulting in incorrect alignment for static background elements.
Second, since its training data consists mostly of buildings and backgrounds, DUSt3R sometimes fails to correctly estimate the geometry of foreground objects, regardless of their motion, and places them in the background. In principle, both problems originate from a domain mismatch between training and test time and can be solved by re-training the network.

However, this requirement for dynamic, posed data with depth presents a challenge, primarily due to its scarcity. Existing methods, such as COLMAP (Schönberger & Frahm, ), often struggle with complex camera trajectories or highly dynamic scenes, making it challenging to produce even pseudo ground truth data for training.
To address this limitation, we identify several small-scale datasets that possess the necessary properties for our purposes.

Our main finding is that, surprisingly, we can successfully adapt DUSt3R to handle dynamic scenes by identifying suitable training strategies designed to maximally leverage this limited data and finetuning on them.
We then introduce several new optimization methods for video-specific tasks using these pointmaps and demonstrate strong performance on video depth and camera pose estimation, as well as promising results for primarily feed-forward 4D reconstruction.

The contributions of this work are as follows:

- •
We introduce Motion DUSt3R (MonST3R), a geometry-first approach to dynamic scenes that directly estimates geometry in the form of pointmaps, even for moving scene elements. To this end, we identify several suitable datasets and show that, surprisingly, a small-scale fine-tuning achieves promising results for direct geometry estimation of dynamic scenes.
- •
MonST3R obtains promising results on several downstream tasks (video depth and camera pose estimation).
In particular, MonST3R offers key advantages over prior work: enhanced robustness, particularly in challenging scenarios; increased speed compared to optimization-based methods; and competitive results with specialized techniques in video depth estimation, camera pose estimation and dense reconstruction.

Figure: Figure 2: Limitation of DUSt3R on dynamic scenes. Left: DUSt3R aligns the moving foreground subject and misaligns the background points as it is only trained on static scenes. Right: DUSt3R fails to estimate the depth of a foreground subject, placing it in the background.

## 2 Related Work

Structure from motion and visual SLAM.
Given a set of 2D images, structure from motion (SfM) (Schönberger & Frahm, ; Teed & Deng, ; Tang & Tan, ) or visual SLAM (Teed & Deng, ; Mur-Artal et al., ; Mur-Artal & Tardós, ; Engel et al., ; Newcombe et al., ) estimate 3D structure of a scene while also localizing the camera.
However, these methods struggle with dynamic scenes with moving objects, which violate the epipolar constraint.

To address this problem, recent approaches have explored joint estimation of depth, camera pose, and residual motion, optionally with motion segmentation to exploit the epipolar constraints on the stationary part.
Self-supervised approaches (Gordon et al., ; Mahjourian et al., ; Godard et al., ; Yang et al., ) learn these tasks through self-supervised proxy tasks.
CasualSAM (Zhang et al., ) finetunes a depth network at test time with a joint estimation of camera pose and movement mask.
Robust-CVD (Kopf et al., ) jointly optimizes depth and camera pose given optical flow and binary masks for dynamic objects.
Our approach directly estimates 3D structure of a dynamic scene in the pointmap representation without time-consuming test-time finetuning.

Representation for static 3D reconstruction.
Learning-based approaches reconstruct static 3D geometry of objects or scenes by learning strong 3D priors from training datasets.
Commonly used output representations include point clouds (Guo et al., ; Lin et al., ), meshes (Gkioxari et al., ; Wang et al., ), voxel (Sitzmann et al., ; Choy et al., ; Tulsiani et al., ), implicit representation (Wang et al., ; Peng et al., ; Chen & Zhang, ), *etc*.

DUSt3R (Wang et al., ) introduces a pointmap representation for scene-level 3D reconstruction. Given two input images, the model outputs a 3D point of each pixel from both images in the camera coordinate system of the first frame.
The model implicitly infers camera intrinsics, relative camera pose, and two-view geometry and thus can output an aligned points cloud with learned strong 3D priors.
However, the method targets only static scenes.
MonST3R shares the pointmap representation of DUSt3R but targets scenes with dynamic objects.

Learning-based visual odometry.
Learning-based visual odometry replaces hand-designed parts of geometry-based methods (Mur-Artal et al., ; Mur-Artal & Tardós, ; Engel et al., ) and enables large-scale training for better generalization even with moving objects.
Trajectory-based approaches (Chen et al., ; Zhao et al., ) estimate long-term trajectories along a video sequence, classify their dynamic and static motion, and then localize camera via bundle adjustment.
Joint estimation approaches additionally infer moving object mask (Shen et al., ) or optical flow (Wang et al., ) to be robust to moving objects while requiring their annotations during training.
In contrast, our method directly outputs dynamic scene geometry via a pointmap representation and localizes camera afterwards.

Monocular and video depth estimation.
Recent deep learning works (Ranftl et al., ; ; Saxena et al., ; Ke et al., ) target zero-shot performance and with large-scale training combined with synthetic datasets (Yang et al., ; ) show strong generalization to diverse domains. However, for video, these approaches suffer from flickering (temporal inconsistency between nearby estimates) due to their process of only a single frame and invariant training objectives.

Early approaches to video depth estimation (Luo et al., ; Zhang et al., ) improve temporal consistency by fine-tuning depth models, and sometimes motion models, at test time for each input video.
Two recent approaches attempt to improve video depth estimation using generative priors. However, Chronodepth (Shao et al., ) still suffers from flickering due to its window-based inference, and DepthCrafter (Hu et al., ) produces scale-/shift-invariant depth, which is unsuitable for many 3D applications (Yin et al., ).

4D reconstruction.
Concurrently approaches (Lei et al., ; Chu et al., ; Wang et al., ; Liu et al., ) introduce 4D reconstruction methods of dynamic scenes.
Given a monocular video and pre-computed estimates (*e.g*., 2D motion trajectory, depth, camera intrinsics and pose, *etc*.), the approaches reconstruct the input video in 4D space via test-time optimization of 3D Gaussians (Kerbl et al., ) with deformation fields, facilitating
novel view synthesis in both space and time.
Our method is orthogonal to the methods and estimate geometry from videos in a feed-forward manner. Our estimates could be used as initialization or intermediate signals for these methods.

## 3 Method

### 3.1 Background and baselines

Model architecture.
Our architecture is based on DUSt3R (Wang et al., ), a ViT-based architecture (Dosovitskiy et al., ) that is pre-trained on a cross-view completion task (Weinzaepfel et al., ) in a self-supervised manner.
Two input images are first individually fed to a shared encoder.
A following transformer-based decoder processes the input features with cross-attention.
Then two separate heads at the end of the decoder output pointmaps of the first and second frames aligned in the coordinate of the first frame.

Baseline with mask.
While DUSt3R is designed for static scenes as shown in Fig. 2, we analzye its applicability to dynamic scenes by using knowledge of dynamic elements (Chen et al., ; Zhao et al., ).
Using ground truth moving masks, we adapt DUSt3R by masking out dynamic objects during inference at both the image and token levels, replacing dynamic regions with black pixels in the image and corresponding tokens with mask tokens.
This approach, however, leads to degraded pose estimation performance (Sec. 4.3), likely because the black pixels and mask tokens are out-of-distribution with respect to training.
This motivates us to address these issues in this work.

### 3.2 Training for dynamics

Main idea.
While DUSt3R primarily focuses on static scenes, the proposed MonST3R can estimate the geometry of dynamic scenes over time.
Figure. 1 shows a visual example consisting of a point cloud where dynamic objects appear at different locations, according to how they move.

Similar to DUSt3R, for a single image $\mathbf{I}^{t}$ at time $t$, MonST3R also predicts a pointmap $\mathbf{X}^{t}\in\mathbb{R}^{H\times W\times 3}$.
For a pair of images, $\mathbf{I}^{t}$ and $\mathbf{I}^{t^{\prime}}$, we adapt the notation used in the global optimization section of DUSt3R.
The network predicts two corresponding pointmaps, $\mathbf{X}^{t;t\parbox{4.26773pt}{ \leavevmode\hbox to5.67pt{\vbox to3.2pt{
\pgfpicture\makeatletter\hbox{\hskip 1.19998pt\lower-1.59998pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{{
{\pgfsys@beginscope\pgfsys@setlinewidth{0.32pt}\pgfsys@setdash{}{0.0pt}
\pgfsys@roundcap\pgfsys@roundjoin{}
{}{}{}
{}{}{}
\pgfsys@moveto{-1.19998pt}{1.59998pt}\pgfsys@curveto{-1.09998pt}{0.99998pt}{0.
0pt}{0.09999pt}{0.29999pt}{0.0pt}\pgfsys@curveto{0.0pt}{-0.09999pt}{-1.09998pt
}{-0.99998pt}{-1.19998pt}{-1.59998pt}\pgfsys@stroke\pgfsys@endscope}}
}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{0.0pt}
\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope
\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0.45999pt}{0.0pt}
\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{
\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$ and $\mathbf{X}^{t^{\prime};t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to
0.4pt{\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$, with confidence map, $\mathbf{C}^{t;t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{
\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$ and $\mathbf{C}^{t^{\prime};t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to
0.4pt{\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$
The first element $t$ in the superscript indicates the frame that the pointmap corresponds to, and $t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{\pgfpicture
\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope
\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}$
indicates that the network receives two frames at $t,t^{\prime}$ and that the pointmaps are in the coordinate frame of the camera at $t$.
The key difference from DUSt3R is that each pointmap in MonST3R relates to a single point in time.

**Table 1: Training datasets used fine-tuning on dynamic scenes. All datasets provide both camera pose and depth, and most of them include dynamic objects.**
| Dataset | Domain | Scene type | $\#$ of frames | $\#$ of Scenes | Dynamics | Ratio |
| --- | --- | --- | --- | --- | --- | --- |
| PointOdyssey (Zheng et al., ) | Synthetic | Indoors & Outdoors | 200k | 131 | Realistic | 50% |
| TartanAir (Wang et al., ) | Synthetic | Indoors & Outdoors | 1000k | 163 | None | 25% |
| Spring (Mehl et al., ) | Synthetic | Outdoors | 6k | 37 | Realistic | 5% |
| Waymo Perception (Sun et al., ) | Real | Driving | 160k | 798 | Driving | 20% |

Training datasets.
A key challenge in modeling dynamic scenes as per-timestep pointmaps lies in the scarcity of suitable training data, which requires synchronized annotations of input images, camera poses, and depth.
Acquiring accurate camera poses for real-world dynamic scenes is particularly challenging, often relying on sensor measurements or post-processing through structure from motion (SfM) (Schönberger et al., ; Schönberger & Frahm, ) while filtering out moving objects.
Consequently, we leverage primarily synthetic datasets, where accurate camera poses and depth can be readily extracted during the rendering process.

For our dynamic fine-tuning, we identify four large video datasets: three synthetic datasets - PointOdyssey (Zheng et al., ), TartanAir (Wang et al., ), and Spring (Mehl et al., ), along with the real-world Waymo dataset (Sun et al., ), as shown in Tab. 1.
These datasets contain diverse indoor/outdoor scenes, dynamic objects, camera motion, and labels for camera pose and depth.
PointOdyssey and Spring are both synthetically rendered scenes with articulated, dynamic objects; TartanAir consists of synthetically rendered drone fly-throughs of different scenes without dynamic objects; and Waymo is a real-world driving dataset labeled with LiDAR.

During training, we sample the datasets asymmetrically to place extra weight on PointOdyssey (more dynamic, articulated objects) and less weight on TartanAir (good scene diversity but static) and Waymo (a highly specialized domain). Images are downsampled such that their largest dimension is 512.

Training strategies.
Due to the relatively small size of this dataset mixture, we adopt several training techniques designed to maximize data efficiency. First, we only finetune the prediction head and decoder of the network while freezing the encoder.
This strategy preserves the geometric knowledge in the CroCo (Weinzaepfel et al., ) features and should decrease the amount of data required for fine-tuning.
Second, we create training pairs for each video by sampling two frames with temporal strides ranging from 1 to 9. The sampling probabilities increase linearly with the stride length, with the probability of selecting stride 9 being twice that of stride 1.
This gives us a larger diversity of camera and scene motion and more heavily weighs larger motion. Third, we utilize a Field-of-View augmentation technique using center crops with various image scales. This encourages the model to generalize across different camera intrinsics, even though such variations are relatively infrequent in the training videos.

### 3.3 Downstream applications

Instrinsics and relative pose estimation.
Since the intrinsic parameters are estimated based on the pointmap in its own camera frame $\mathbf{X}^{t;t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{
\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$, the assumptions and computation listed in DUSt3R are still valid, and we only need to solve for focal length $f^{t}$ to obtain the camera intrinsics $\mathbf{K}^{t}$.

To estimate relative pose $\mathbf{P}=[\mathbf{R}|\mathbf{T}]$, where $\mathbf{R}$ and $\mathbf{T}$ represent the camera’s rotation and translation, respectively, dynamic objects violate several assumptions required for the use of an epipolar matrix (Hartley & Zisserman, ) or Procrustes alignment (Luo & Hancock, ).
Instead, we use RANSAC (Fischler & Bolles, ) and PnP (Hartley & Zisserman, ; Lepetit et al., ).
For most scenes, where most pixels are static, random samples of points will place more emphasis on the static elements, and relative pose can be estimated robustly using the inliers.

Confident static regions.
We can infer static regions in frames $t,t^{\prime}$ by comparing the estimated optical flow with the flow field that results from applying only camera motion from $t$ to $t^{\prime}$ to the pointmap at $t$.
The two flow fields should agree for pixels where the geometry has been correctly estimated and are static.
Given a pair of frames $\mathbf{I}^{t}$ and $\mathbf{I}^{t^{\prime}}$, we first compute two sets of pointmaps $\mathbf{X}^{t;t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{
\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}},\mathbf{X}^{t^{\prime};t\parbox{
4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{\pgfpicture\makeatletter
\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}
\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$ and $\mathbf{X}^{t;t^{\prime}\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to
0.4pt{\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t},\mathbf{X}^{t^{\prime};t^{\prime}\parbox{
4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{\pgfpicture\makeatletter
\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}
\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t}$.
We then use these pointmaps to solve for the camera intrinsics ($\mathbf{K}^{t}$ and $\mathbf{K}^{t^{\prime}}$) for each frame and the relative camera pose from $t$ to $t^{\prime}$, $\mathbf{P}^{t\rightarrow t^{\prime}}=[\mathbf{R}^{t\rightarrow t^{\prime}}|
\mathbf{T}^{t\rightarrow t^{\prime}}]$ as above.
We then compute the optical flow field induced by camera motion, $\textbf{F}_{\textrm{cam}}^{t\rightarrow t^{\prime}}$, by backprojecting each pixel in 3D, applying relative camera motion, and projecting back to image coordinate,

$$ $\textbf{F}_{\textrm{cam}}^{t\rightarrow t^{\prime}}=\pi(\textbf{D}^{t;t\parbox {4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{\pgfpicture\makeatletter \hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0} \pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ } \pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{ \pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{} {{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{ 0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{ \pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0 .45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss} \pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}} \lxSVG@closescope\endpgfpicture}}}t^{\prime}}\mathbf{K}^{t^{\prime}}\mathbf{R} ^{t\rightarrow t^{\prime}}{\mathbf{K}^{t}}^{-1}\hat{\mathbf{x}}+\mathbf{K}^{t^ {\prime}}\mathbf{T}^{t\rightarrow t^{\prime}})-\mathbf{x},$ (1) $$

where $\mathbf{x}$ is a pixel coordinate matrix, $\hat{\mathbf{x}}$ is $\mathbf{x}$ in homogeneous coordinates, $\pi(\cdot)$ is the projection operation $(x,y,z)\rightarrow(x/z,y/z)$, and $\textbf{D}^{t;t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{
\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$ is estimated depth extracted from the point map $\mathbf{X}^{t;t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{
\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$.
Then we compare it with optical flow (*i.e*., $\textbf{F}_{\textrm{est}}^{t\rightarrow t^{\prime}}$) computed by an off-the-shelf optical flow method (Wang et al., ) and infer the static mask $\mathbf{S}^{t\rightarrow t^{\prime}}$ via a simple thresholding:

$$ $\mathbf{S}^{t\rightarrow t^{\prime}}=\left[\alpha>||\textbf{F}_{\textrm{cam}}^ {t\rightarrow t^{\prime}}-\textbf{F}_{\textrm{est}}^{t\rightarrow t^{\prime}}| |_{\text{L1}}\right],$ (2) $$

with a threshold $\alpha$, $||\cdot||_{\text{L1}}$ for smooth-L1 norm (Girshick, ), and $[\cdot]$ for the Iverson bracket.
This confident, static mask is both a potential output and will be used in the later global pose optimization.

### 3.4 Dynamic global point clouds and camera pose

Even a short video contain numerous frames (*e.g*. a 5-second video with 24 fps gives 120 frames) making it non-trivial to extract a single dynamic point cloud from pairwise pointmap estimates across the video.
Here, we detail the steps to simultaneously solve for a global dynamic point cloud and camera poses by leveraging our pairwise model and the inherent temporal structure of video.

Figure: Figure 3: Dynamic global point cloud and camera pose estimation. Given a fixed sized of temporal window, we compute pairwise pointmap for each frame pair with MonST3R and optical flow from off-the-shelf method. These intermediates then serve as inputs to optimize a global point cloud and per-frame camera poses. Video depth can be directly derived from this unified representation.
Refer to caption: x3.png

Video graph.
For global alignment, DUSt3R constructs a connectivity graph from all pairwise frames, a process that is prohibitively expensive for video.
Instead, as shown on the left of Fig. 3, we process video with a sliding temporal window, significantly reducing the amount of compute required.
Specifically, given a video $\mathbf{V}=[\mathbf{I}^{0},\ldots,\mathbf{I}^{N}]$, we compute pointmaps for all pairs $e=(t,t^{\prime})$ within a temporal window of size $w$, $\mathbf{W}^{t}=\{(a,b)\mid a,b\in[t,\ldots,t+w],a\neq b\}$ and for all valid windows $\mathbf{W}$.
To further improve the run time, we also apply strided sampling.

Dynamic global point cloud and pose optimization.
The primary goal is to accumulate all pairwise pointmap predictions (*e.g*., $\mathbf{X}^{t;t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{
\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}},\mathbf{X}^{t^{\prime};t\parbox{
4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{\pgfpicture\makeatletter
\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@rgb@stroke{0}{0}{0}
\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}{0}\pgfsys@invoke{ }
\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$) into the same global coordinate frame to produce world-coordinate pointmap $\mathbf{X}^{t}\in\mathbb{R}^{H\times W\times 3}$.
To do this, as shown in Fig. 3, we use DUSt3R’s alignment loss and add two video specific loss terms: camera trajectory smoothness and flow projection.

We start by re-parameterizing the global pointmaps $\mathbf{X}^{t}$ with camera parameters $\mathbf{P}^{t}=[\mathbf{R}^{t}|\mathbf{T}^{t}],\mathbf{K}^{t}$ and per-frame depthmap $\mathbf{D}^{t}$, as $\mathbf{X}_{i,j}^{t}:={\mathbf{P}^{t}}^{-1}h({\mathbf{K}^{t}}^{-1}[i\mathbf{D}
_{i,j}^{t};j\mathbf{D}_{i,j}^{t};\mathbf{D}_{i,j}^{t}])$, with $(i,j)$ for pixel coordinate and $h()$ for homogeneous mapping.
It allows us to define losses directly on the camera parameters. To simplify the notation for function parameters, we use $\textbf{X}^{t}$ as a shortcut for $\mathbf{P}^{t},\mathbf{K}^{t},\mathbf{D}^{t}$.

First, we use the alignment term in DUSt3R which aims to find a single rigid transformation $\mathbf{P}^{t;e}$ that aligns each pairwise estimation with the world coordinate pointmaps, since both $\mathbf{X}^{t;t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{
\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$ and $\mathbf{X}^{t^{\prime};t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to
0.4pt{\pgfpicture\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{
\pgfsys@beginscope\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}}$ are in the same camera coordinate frame:

$$ $\mathcal{L}_{\textrm{align}}(\mathbf{X},\sigma,\mathbf{P}_{W})=\sum_{W^{i}\in W }\sum_{e\in W^{i}}\sum_{t\in e}||\mathbf{C}^{t;e}\cdot(\mathbf{X}^{t}-\sigma^{ e}\mathbf{P}^{t;e}\mathbf{X}^{t;e})||_{1},$ (3) $$

where $\sigma^{e}$ is a pairwise scale factor. To simplify the notation, we use the directed edge $e=(t,t^{\prime})$ interchangeably with $t\parbox{4.26773pt}{ \leavevmode\hbox to4.67pt{\vbox to0.4pt{\pgfpicture
\makeatletter\hbox{\hskip 0.2pt\lower-0.2pt\hbox to0.0pt{\pgfsys@beginscope
\pgfsys@invoke{ }\definecolor{pgfstrokecolor}{rgb}{0,0,0}
\pgfsys@color@rgb@stroke{0}{0}{0}\pgfsys@invoke{ }\pgfsys@color@rgb@fill{0}{0}
{0}\pgfsys@invoke{ }\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to
0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{{}{{}}{}
{{}{}}{}{}{}{}{}{{}}\pgfsys@moveto{0.45999pt}{0.0pt}\pgfsys@lineto{4.26791pt}{
0.0pt}\pgfsys@stroke\pgfsys@invoke{ }{{}{{}}{}{}{{}}{{{}}{{{}}{
\pgfsys@beginscope\pgfsys@invoke{ }\pgfsys@transformcm{-1.0}{0.0}{0.0}{-1.0}{0
.45999pt}{0.0pt}\pgfsys@invoke{ }\pgfsys@invoke{ \lxSVG@closescope }
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope}}{{}}}}
}
\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope{}{}{}\hss}
\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hss}}
\lxSVG@closescope\endpgfpicture}}}t^{\prime}$.

We use a camera trajectory smoothness loss to encourage smooth camera motion by penalizing large changes in camera rotation and translation in nearby timesteps:

$$ $\mathcal{L}_{\text{smooth}}(\mathbf{X})=\sum_{t=0}^{N}\left(\left\|{\mathbf{R} ^{t}}^{\top}\mathbf{R}^{t+1}-{\bm{I}}\right\|_{\text{f}}+\left\|{\mathbf{R}^{t }}^{\top}(\mathbf{T}^{t+1}-\mathbf{T}^{t})\right\|_{2}\right),$ (4) $$

where the Frobenius norm $\|\cdot\|_{\text{f}}$ is used for the rotation difference, the L2 norm $\|\cdot\|_{2}$ is used for the translation difference, and ${\bm{I}}$ is the identity matrix.

We also use a flow projection loss to encourage the global pointmaps and camera poses to be consistent with the estimated flow for the confident, static regions of the actual frames.
More precisely, given two frames $t,t^{\prime}$, using their global pointmaps, camera extrinsics and intrinsics, we compute the flow fields from taking the global pointmap $\mathbf{X}^{t}$, assuming the scene is static, and then moving the camera from $t$ to $t^{\prime}$.
We denote this value $\textbf{F}_{\textrm{cam}}^{\textrm{global};t\rightarrow t^{\prime}}$, similar to the term defined in the confident static region computation above.
Then we can encourage this to be close to the estimated flow, $\textbf{F}_{\textrm{est}}^{t\rightarrow t^{\prime}}$, in the regions which are confidently static $\mathbf{S}^{\textrm{global};t\rightarrow t^{\prime}}$ according to the global parameters:

$$ $\mathcal{L}_{\text{flow}}(\mathbf{X})=\sum_{W^{i}\in W}\sum_{t\rightarrow t^{ \prime}\in W^{i}}||\mathbf{S}^{\textrm{global};t\rightarrow t^{\prime}}\cdot( \textbf{F}_{\textrm{cam}}^{\textrm{global};t\rightarrow t^{\prime}}-\textbf{F} ^{t\rightarrow t^{\prime}}_{\textrm{est}})||_{1},$ (5) $$

where $\cdot$ indicates element-wise multiplication.
Note that the confident static mask is initialized using the pairwise prediction values (pointmaps and relative poses) as described in Sec. 3.3.
During the optimization, we use the global pointmaps and camera parameters to compute $\textbf{F}_{\textrm{cam}}^{\textrm{global}}$ and update the confident static mask.

The complete optimization for our dynamic global point cloud and camera poses is:

$$ $\hat{\mathbf{X}}=\operatorname*{arg\,min}_{\mathbf{X},\mathbf{P}_{W},\sigma} \mathcal{L}_{\textrm{align}}(\mathbf{X},\sigma,\mathbf{P}_{W})+w_{\text{smooth }}\mathcal{L}_{\textrm{smooth}}(\mathbf{X})+w_{\text{flow}}\mathcal{L}_{ \textrm{flow}}(\mathbf{X}),$ (6) $$

where $w_{\text{smooth}},w_{\text{flow}}$ are hyperparameters. Note, based on the reparameterization above, $\hat{\mathbf{X}}$ includes all the information for $\hat{\mathbf{D}},\hat{\mathbf{P}},\hat{\mathbf{K}}$.

Video depth.
We can now easily obtain temporally-consistent video depth, traditionally addressed as a standalone problem.
Since our global pointmaps are parameterized by camera pose and per-frame depthmaps $\hat{\textbf{D}}$, just returning $\hat{\textbf{D}}$ gives the video depth.

## 4 Experiments

MonST3R runs on a monocular video of a dynamic scene and jointly optimizes video depth and camera pose.
We compare the performance with methods specially designed for each individual subtask (*i.e*., depth estimation and camera pose estimation), as well as monocular depth methods.

### 4.1 Experimental Details

Training and Inference.
We fine-tune the DUSt3R’s ViT-Base decoder and DPT heads for 25 epochs, using 20,000 sampled image pairs per epoch.
We use the AdamW optimizer with a learning rate of $5\times 10^{-5}$ and a mini-batch size of 4 per GPU.
Training took one day on $2\times$ RTX 6000 48GB GPUs. Inference for a 60-frame video with $w=9$ and stride 2 (approx. 600 pairs) takes around 30s.

Global Optimization.
For global optimization Eq. 6, we set the hyperparameter of each weights to be $w_{\text{smooth}}=0.01$ and $w_{\text{flow}}=0.01$.
We only enable the flow loss when the average value is below $20$, when the poses are roughly aligned.
The motion mask is updated during optimization if the per-pixel flow loss is higher than $50$.
We use the Adam optimizer for 300 iterations with a learning rate of $0.01$, which takes around 1 minute for a 60-frame video on a single RTX 6000 GPU.

### 4.2 Single-frame and video depth estimation

Baselines.
We compare our method with video depth methods, NVDS (Wang et al., ), ChronoDepth (Shao et al., ), and concurrent work, DepthCrafter (Hu et al., ), as well as single-frame depth methods, Depth-Anything-V2 (Yang et al., ) and Marigold (Ke et al., ).
We also compare with methods for joint video depth and pose estimation, CasualSAM (Zhang et al., ) and Robust-CVD (Kopf et al., ), which address the same problem as us.
This comparison is particularly important since joint estimation is substantially more challenging than only estimating depth.
Of note, CasualSAM relies on heavy optimization, whereas ours runs in a feed-forward manner with only lightweight optimization.

**Table 2: Video depth evaluation on Sintel, Bonn, and KITTI datasets. We evaluate for both scale-and-shift-invariant and scale-invariant depth. The best and second best results in each category are bold and underlined, respectively.**
|  |  |  | Sintel | Bonn | KITTI |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Alignment | Category | Method | Abs Rel $\downarrow$ | $\delta$<$1.25\uparrow$ | Abs Rel $\downarrow$ | $\delta$<$1.25\uparrow$ | Abs Rel $\downarrow$ | $\delta$ <$1.25\uparrow$ |
| Per-sequence scale & shift | Single-frame depth | Marigold | 0.532 | 51.5 | 0.091 | 93.1 | 0.149 | 79.6 |
| Depth-Anything-V2 | 0.367 | 55.4 | 0.106 | 92.1 | 0.140 | 80.4 |  |  |
| Video depth | NVDS | 0.408 | 48.3 | 0.167 | 76.6 | 0.253 | 58.8 |  |
| ChronoDepth | 0.687 | 48.6 | 0.100 | 91.1 | 0.167 | 75.9 |  |  |
| DepthCrafter (Sep. 2024) | 0.292 | 69.7 | 0.075 | 97.1 | 0.110 | 88.1 |  |  |
| Joint video depth & pose | Robust-CVD | 0.703 | 47.8 | - | - | - | - |  |
| CasualSAM | 0.387 | 54.7 | 0.169 | 73.7 | 0.246 | 62.2 |  |  |
| MonST3R | 0.335 | 58.5 | 0.063 | 96.4 | 0.104 | 89.5 |  |  |
| Per-sequence scale | Video depth | DepthCrafter (Sep. 2024) | 0.692 | 53.5 | 0.217 | 57.6 | 0.141 | 81.8 |
| Joint depth & pose | MonST3R | 0.345 | 56.2 | 0.065 | 96.3 | 0.106 | 89.3 |  |

**Table 3: Single-frame depth evaluation. We report the performance on Sintel, Bonn, KITTI, and NYU-v2 (static) datasets. MonST3R achieves overall comparable results to DUSt3R.**
|  | Sintel | Bonn | KITTI | NYU-v2 (static) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Abs Rel $\downarrow$ | $\delta$<$1.25\uparrow$ | Abs Rel $\downarrow$ | $\delta$<$1.25\uparrow$ | Abs Rel $\downarrow$ | $\delta$ <$1.25\uparrow$ | Abs Rel $\downarrow$ | $\delta$ <$1.25\uparrow$ |
| DUSt3R | 0.424 | 58.7 | 0.141 | 82.5 | 0.112 | 86.3 | 0.080 | 90.7 |
| MonST3R | 0.345 | 56.5 | 0.076 | 93.9 | 0.101 | 89.3 | 0.091 | 88.8 |

Benchmarks and metrics.
Similar to DepthCrafter, we evaluate video depth on KITTI (Geiger et al., ), Sintel (Butler et al., ), and Bonn (Palazzolo et al., ) benchmark datasets, covering dynamic and static, indoor and outdoor, and realistic and synthetic data.
For monocular/single-frame depth estimation, we also evaluate on NYU-v2 (Silberman et al., ).

Our evaluation metrics include absolute relative error (Abs Rel) and percentage of inlier points $\delta<1.25$, following the convention (Hu et al., ; Yang et al., ).
All methods output scale- and/or shift- invariant depth estimates.
For video depth evaluation, we align a single scale and/or shift factor per each sequence, whereas the single-frame evaluation adopts per-frame median scaling, following Wang et al. ().
As demonstrated by Yin et al. (), shift is particularly important in the 3D geometry of a scene and is important to predict.

Results. As shown in Tab. 2, MonST3R achieves competitive and even better results, even outperforming specialized video depth estimation techniques like DepthCrafter (a concurrent work).
Furthermore, MonST3R significantly outperforms DepthCrafter (Hu et al., ) with scale-only normalization.
As in Tab. 3, even after our fine-tuning for videos of dynamic scenes, the performance on single-frame depth estimation remains competitive with the original DUSt3R model.

### 4.3 Camera pose estimation

Baselines.
We compare with not only direct competitors (*i.e*., CasualSAM and Robust-CVD), but also a range of learning-based visual odometry methods for dynamic scenes, such as DROID-SLAM (Teed & Deng, ), Particle-SfM (Zhao et al., ), DPVO (Teed et al., ), and LEAP-VO (Chen et al., ).
Notably, several methods (*e.g*., DROID-SLAM, DPVO, and LEAP-VO) require ground truth camera intrinsic as input and
ParticleSfM is an optimization-based method that runs $5\times$ slower than ours.
We also compare with the “DUSt3R with mask” baseline in Sec. 3.1 to see if DUSt3R performs well on dynamic scenes when a ground truth motion mask is provided.

Benchmarks and metrics.
We evaluate the methods on Sintel (Butler et al., ) and TUM-dynamics (Sturm et al., ) (following CasualSAM) and ScanNet (Dai et al., ) (following ParticleSfM) to test generalization to static scenes as well.
On Sintel, we follow the same evaluation protocol as in Chen et al. (); Zhao et al. (), which excludes static scenes or scenes with perfectly-straight camera motion, resulting in total 14 sequences.
For TUM-dynamics and ScanNet, we sample the first 90 frames with the temporal stride of 3 to save compute.
We report the same metric as Chen et al. (); Zhao et al. (): Absolute Translation Error (ATE), Relative Translation Error (RPE trans), and Relative Rotation Error (RPE rot), after applying a Sim(3) Umeyama alignment on prediction to the ground truth.

Results. In Tab. 4, MonST3R achieves the best accuracy among methods to joint depth and pose estimation and performs competitively to pose-only methods even without using ground truth camera intrinsics.
Our method also generalizes well to static scenes (*i.e*., ScanNet) and shows improvements over even DUSt3R, which proves the effectiveness of our designs (*e.g*., Eq. 6) for video input.

**Table 4: Evaluation on camera pose estimation on the Sintel, TUM-dynamic, and ScanNet. The best and second best results are bold and underlined, respectively. MonST3R achieves competitive and even better results than pose-specific methods, even without ground truth camera intrinsics.**
|  |  | Sintel | TUM-dynamics | ScanNet (static) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Category | Method | ATE $\downarrow$ | RPE trans $\downarrow$ | RPE rot $\downarrow$ | ATE $\downarrow$ | RPE trans $\downarrow$ | RPE rot $\downarrow$ | ATE $\downarrow$ | RPE trans $\downarrow$ | RPE rot $\downarrow$ |
| Pose only | DROID-SLAM^∗ | 0.175 | 0.084 | 1.912 | - | - | - | - | - | - |
| DPVO^∗ | 0.115 | 0.072 | 1.975 | - | - | - | - | - | - |  |
| ParticleSfM | 0.129 | 0.031 | 0.535 | - | - | - | 0.136 | 0.023 | 0.836 |  |
| LEAP-VO^∗ | 0.089 | 0.066 | 1.250 | 0.068 | 0.008 | 1.686 | 0.070 | 0.018 | 0.535 |  |
|  | Robust-CVD | 0.360 | 0.154 | 3.443 | 0.153 | 0.026 | 3.528 | 0.227 | 0.064 | 7.374 |
| Joint depth | CasualSAM | 0.141 | 0.035 | 0.615 | 0.071 | 0.010 | 1.712 | 0.158 | 0.034 | 1.618 |
| & pose | DUSt3R w/ mask^† | 0.417 | 0.250 | 5.796 | 0.083 | 0.017 | 3.567 | 0.081 | 0.028 | 0.784 |
|  | MonST3R | 0.108 | 0.042 | 0.732 | 0.063 | 0.009 | 1.217 | 0.068 | 0.017 | 0.545 |
| ^∗ requires ground truth camera intrinsics as input,<br>^† unable to estimate the depth of foreground object. |  |  |  |  |  |  |  |  |  |  |

### 4.4 Joint dense reconstruction and pose estimation

Fig. 4 qualitatively compares our method with CasualSAM and DUSt3R on video sequences for joint dense reconstruction and pose estimation on DAVIS (Perazzi et al., ).
For each video sequence, we visualize overlayed point clouds aligned with estimated camera pose, showing as two rows with different view points for better visualization.
As discussed in Fig. 2, DUSt3R struggles with estimating correct geometry of moving foreground objects, resulting in failure of joint camera pose estimation and dense reconstruction.
CasualSAM reliably estimates camera trajectories while sometimes failing to produce correct geometry estimates for foreground objects.
MonST3R outputs both reliable camera trajectories and reconstruction of entire scenes along the video sequences.

Figure: Figure 4: Qualitative comparison. Compared to CasualSAM and DUSt3R, our method outputs both reliable camera trajectories and geometry of dynamic scenes. Refer to Fig. A5 for more results.
Refer to caption: x4.png

### 4.5 Ablation Study

**Table 5: Ablation study on Sintel dataset. For each category, the default setting is underlined, and the best performance is bold.**
|  |  | Camera pose estimation | Video depth estimation |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Variants | ATE $\downarrow$ | RPE trans $\downarrow$ | RPE rot $\downarrow$ | Abs Rel $\downarrow$ | $\delta$ <1.25 $\uparrow$ |
| Training<br>dataset | No finetune (DUSt3R) | 0.354 | 0.167 | 0.996 | 0.482 | 56.5 |
| w/ PO | 0.220 | 0.129 | 0.901 | 0.378 | 53.7 |  |
| w/ PO+TA | 0.158 | 0.054 | 0.886 | 0.362 | 56.7 |  |
| w/ PO+TA+Spring | 0.121 | 0.046 | 0.777 | 0.329 | 58.1 |  |
| w/ TA+Spring+Waymo | 0.167 | 0.107 | 1.136 | 0.462 | 54.0 |  |
| w/ all 4 datasets | 0.108 | 0.042 | 0.732 | 0.335 | 58.5 |  |
| Training<br>strategy | Full model finetune | 0.181 | 0.110 | 0.738 | 0.352 | 55.4 |
| Finetune decoder & head | 0.108 | 0.042 | 0.732 | 0.335 | 58.5 |  |
| Finetune head | 0.185 | 0.128 | 0.860 | 0.394 | 55.7 |  |
| Inference | w/o flow loss | 0.140 | 0.051 | 0.903 | 0.339 | 57.7 |
| w/o static region mask | 0.132 | 0.049 | 0.899 | 0.334 | 58.7 |  |
| w/o smoothness loss | 0.127 | 0.060 | 1.456 | 0.333 | 58.4 |  |
| Full | 0.108 | 0.042 | 0.732 | 0.335 | 58.5 |  |

Table 5 presents an ablation study analyzing the impact of design choices in our method, including the selection of training datasets, fine-tuning strategies, and the novel loss functions used for dynamic point cloud optimization. Our analysis reveals that: (1) all datasets contribute to improved camera pose estimation performance; (2) fine-tuning only the decoder and head outperforms alternative strategies; and (3) the proposed loss functions enhance pose estimation with minimal impact on video depth accuracy.

Discussions. While MonST3R represents a promising step towards directly estimating dynamic geometry from videos as well as camera pose and video depth, limitations remain.
While our method can, unlike prior methods, theoretically handle dynamic camera intrinsics, we find that, in practice, this requires careful hyperparameter tuning or manual constraints.
Like many deep learning methods, MonST3R struggles with out-of-distribution inputs, such as open fields. Expanding the training set is a key direction to make MonST3R more robust to in-the-wild videos.

## 5 Conclusions

We present MonST3R, a simple approach to directly estimate geometry for dynamic scenes and extract downstream information like camera pose and video depth. MonST3R leverages per-timestep pointmaps as a powerful representation for dynamic scenes.
Despite being finetuned on a relatively small training dataset, MonST3R achieves impressive results on downstream tasks, surpassing even state-of-the-art specialized techniques.

#### Acknowledgments

We would like to express gratitude to Youming Deng, Hang Gao, and Haven Feng for their helpful discussions and extend special thanks to Brent Yi, Chung Min Kim, and Justin Kerr for their valuable help with online visualizations.

## Appendix A More qualitative results

### A.1 Depth

For more thorough comparisons, we include additional qualitative examples of video depth, comparing our method against DepthCrafter, a concurrent method specifically trained for video depth. We include comparisons on the Bonn dataset in  Fig. A1 and the KITTI dataset in Fig. A2.
In these comparisons, we show that after alignment, our estimates are much closer to the ground truth than those of DepthCrafter.

Figure: Figure A1: Video depth estimation comparison on Bonn dataset. Evaluation protocol is per-sequence scale & shift. We visualize the prediction result after alignment. Note, in the first row, our depth estimation is more aligned with the GT depth (*e.g*., the wall) compared to DepthCrafter’s.
Refer to caption: x5.png

Figure: Figure A2: Video depth estimation comparison on KITTI dataset. Evaluation protocol is per-sequence scale & shift. For each case, the upper row is for input frame and depth prediction; the lower row is for ground truth depth annotation and error map. Prediction result is after alignment.
Refer to caption: x6.png

### A.2 Camera pose

We present additional qualitative results for camera pose estimation. We compare our model with the state-of-the-art visual odometry method LEAP-VO and the joint video depth and pose optimization method CasualSAM.
Results are provided for the Sintel dataset in Fig. A3 and Scannet dataset in Fig. A4.
In these comparisons, our method significantly outperforms the baselines for very challenging cases such as “temple_3” and “cave_2” in Sintel and performs comparable to or better than baselines in the rest of the results like those in the Scannet dataset.

Figure: Figure A3: Camera pose estimation comparison on the Sintel dataset. The trajectories are plotted along the two axes with the highest variance to capture the most significant motion. The predicted trajectory (solid blue line) is aligned to match the ground truth trajectory (dashed gray line). Our MonST3R is more robust at challenging scenes, “temple_3” and “cave_2” (the last two rows).
Refer to caption: x7.png

Figure: Figure A4: Camera pose estimation comparison on the Scannet dataset. The trajectories are plotted along the two axes with the highest variance to capture the most significant motion.
Refer to caption: x8.png

### A.3 Joint depth & camera pose results

We present additional results for joint point cloud and camera pose estimation, comparing against CasualSAM and DUSt3R. Fig. A5 shows three additional scenes for Davis: mbike-trick, train, and dog.
For mbike-trick, CasualSAM makes large errors in both geometry and camera pose; DUSt3R produces reasonable geometry except for the last subset of the video which also results in poor pose estimation (highlighted in red); and ours correctly estimates both geometry and camera pose.
For train, CasualSAM accurately recovers the camera pose for the video but produces suboptimal geometry, misaligning the train and the track at the top right. DUSt3R both misaligns the track at the top left and gives poor camera pose estimates. Our method correctly estimates both geometry and camera pose.
For dog, CasualSAM produces imprecise, smeared geometry with slight inaccuracies in the camera pose. DUSt3R results in mistakes in both the geometry and camera pose due to misalignments of the frames, while our method correctly estimates both geometry and camera pose.

Figure: Figure A5: Qualitative comparison on Davis. Compared to CasualSAM and DUSt3R, our method outputs both reliable camera trajectories and geometry of dynamic scenes.
Refer to caption: x9.png

### A.4 Pairwise pointmaps

In Fig. A6, we also include visualizations of two input frames and estimated pairwise pointmaps, the direct output of the trained models, for both DUSt3R and MonST3R. Note, these results do not include any optimization or post-processing. Row 1 demonstrates that even after fine-tuning, our method retains the ability to handle changing camera intrinsics. Rows 2 and 3 demonstrate that our method can handle “impossible” alignments that two frames have almost no overlap, even in the presence of motion, unlike DUSt3R that misaligns based on the foreground object. Rows 4 and 5 show that in addition to enabling the model to handle motion, our fine-tuning also has improved the model’s ability to represent large-scale scenes, where DUSt3R predicts to be flat.

Figure: Figure A6: Qualitative comparison of feed-forward pairwise pointmaps prediction.
Refer to caption: x10.png

### A.5 Static/dynamic mask

We present the result of static/dynamic mask estimation based on the optical flow, as discussed in Sec. 3.3. As shown in Fig. A7, our MonST3R achieves overall plausible results while DUSt3R fails to provide accurate motion mask due to error in camera pose and depth estimation.

Figure: Figure A7: Qualitative comparison of static/dynamic mask. We visualize both the continuous error map (upper row) and binary static/dynamic mask (lower row). The threshold $\alpha$ is fixed.
Refer to caption: x11.png

## Appendix B More quantitative results

### B.1 Ablation on traning/inference window size

In this section, we present additional quantitative results on the impact of different training and inference window sizes, as shown in Tab. A1. The results demonstrate that performance generally improves as the inference window size increases, up to the size of the training window.

Moreover, our proposed stride-based sampling provides a better trade-off between window size and computational cost. For instance, for the training window size of 7 or 9, the inference configuration “Window size 7 (stride 2)” outperforms “Window size 4” while consuming a similar amount of memory (19.9 GB vs. 20.1 GB). Additionally, for a training window size of 9, “Window size 9 (stride 2)” achieves better performance than “Window size 7” while reducing memory consumption by 20%, highlighting the efficiency of our design.

**Table A1: Ablation study on different training/inference window sizes on the Sintel dataset. Each cell displays two values: ATE $\downarrow$ / Abs Rel $\downarrow$, corresponding to camera pose and video depth estimation, respectively. The cells where the inference window size exceeds the training window size are highlighted in grey. The default setup is underlined, and the best results are in bold. GPU memory consumption for each inference setup is listed in the leftmost column.**
|  |  | Training window size |  |  |
| --- | --- | --- | --- | --- |
| Memory (GB) | Inference video graph | Size 5 | Size 7 | Size 9 |
| 17.3 | Window size 3 | 0.191 / 0.442 | 0.182 / 0.413 | 0.163 / 0.383 |
| 20.1 | Window size 4 | 0.178 / 0.431 | 0.166 / 0.406 | 0.148 / 0.362 |
| 17.2 | Window size 5 (stride 2) | 0.145 / 0.439 | 0.140 / 0.411 | 0.137 / 0.367 |
| 23.5 | Window size 5 | 0.139 / 0.406 | 0.132 / 0.399 | 0.133 / 0.355 |
| 19.9 | Window size 7 (stride 2) | 0.180 / 0.409 | 0.140 / 0.372 | 0.121 / 0.359 |
| 29.5 | Window size 7 | 0.174 / 0.389 | 0.136 / 0.351 | 0.113 / 0.346 |
| 23.2 | Window size 9 (stride 2) | 0.177 / 0.380 | 0.156 / 0.387 | 0.108 / 0.345 |

### B.2 Ablation on loss weight sensitivity

**Table A2: Ablation study on loss weight sensitivity. The table shows the effect of varying the loss weights $w_{\text{smooth}}$ and $w_{\text{flow}}$ on camera pose and video depth estimation. The default setup is underlined, and the best results are in bold.**
|  |  | Camera pose estimation | Video depth estimation |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| $w_{\text{smooth}}$ | $w_{\text{flow}}$ | ATE $\downarrow$ | RPEtrans $\downarrow$ | RPErot $\downarrow$ | Abs Rel $\downarrow$ | $\delta$ <1.25 $\uparrow$ |
| 0.01 | 0.001 | 0.118 | 0.045 | 0.716 | 0.335 | 58.2 |
| 0.01 | 0.005 | 0.109 | 0.042 | 0.715 | 0.336 | 58.2 |
| 0.01 | 0.01 | 0.108 | 0.042 | 0.732 | 0.335 | 58.5 |
| 0.01 | 0.05 | 0.115 | 0.044 | 0.831 | 0.329 | 59.5 |
| 0.01 | 0.1 | 0.118 | 0.049 | 0.838 | 0.330 | 59.3 |
| 0.001 | 0.01 | 0.110 | 0.048 | 0.869 | 0.332 | 58.4 |
| 0.005 | 0.01 | 0.109 | 0.044 | 0.844 | 0.334 | 58.3 |
| 0.01 | 0.01 | 0.108 | 0.042 | 0.732 | 0.335 | 58.5 |
| 0.05 | 0.01 | 0.127 | 0.045 | 0.779 | 0.342 | 57.9 |
| 0.1 | 0.01 | 0.138 | 0.049 | 0.799 | 0.346 | 57.3 |

In the main paper, our global optimization objective (see Eq. 6) is a combination of three different loss terms, controlled by two hyperparameters. Here, we evaluate the sensitivity of the results to variations in these two weights.

From the results in Tab. A2, it can be seen that the weight of the optical flow loss ($w_{\text{flow}}$) does not significantly impact the overall performance. Varying the flow loss weight sometimes leads to slightly better results in other metrics.
The weight of the camera trajectory smoothness constraint ($w_{\text{smooth}}$) exhibits more noticeable effects. When it is set to a lower value, the difference in performance remains small, though the RPE performance drops noticeably.
However, when the weight is set too high, performance degrades, likely due to the over-constraining of the camera trajectory.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. In the depth comparison in figure A1, the depth colormaps effectively hides the error, it's better to also add error colormaps. Additionally, in all the visualized frames, the model all predicts closer depth compared to ground truth on the dynamic parts. Is this specific to this dataset or is it a general bias of the model? What could be the possible reasons?
2. In lines 261-269, to calculate induced flow from camera motions, the method needs to estimate the intrinsics separately from the pointmaps. I wonder if using dataset intrinsics would degrade the performance?
3. Another related question on intrinsics, in lines 456-458, the relax of requiring intrinsics can be seen as a strength, however DUSt3R is not able to include known intrinsics. Is it considered to include the input intrinsics during the method development?
4. As we see from DUSt3R to MASt3R, the pointmaps serve as general and powerful representation for static geometry, as multiple downstream task can be estimated based on them. However, it's questionable whether it fit best for dynamic scenes. Here, an off-the-shelf flow estimator is needed to help obtain the static/dynamic mask, are flow fields themselves better suited for representing dynamics?
soundness: 3
strengths: 1. The method motivates the problem well on joint pose and geometry estimation on dynamic scenes by identifying the limitations in the current geometric learning method DUSt3R which is trained on static scenes.
2. With carefully selected datasets for dynamic scenes, together with a window-based training strategy, the scarcity of dynamic training data for finetuning is dealt with.
3. Finetuning results validate the proposed idea, with only the decoder and prediction head tuned, the model is capable of predicting pointmaps for the dynamic regions, showing sharping improvement compared to the DUSt3R with mask baseline.
4. The method achieves SoTA results on video depth estimation task, and competitive results on video pose estimation. The selected baselines are thorough, including baselines specialized in each task.
summary: The paper first identifies the limitations in joint pose and geometry estimation methods on dynamic scenes, and proposes to finetune based on the recent geometric learning model DUSt3R, representing the dynamic scenes with per-timestamp pointmaps. During global alignment for the pairwise inference results, trajectory smoothness loss and flow projection loss are added to the alignment term to adapt the optimization on sequential dynamic inputs. Experimental results show that the proposed method achieves state-of-the-art performance on the video depth estimation task and remains competitive in camera pose estimation, while jointly estimating both.
weaknesses: 1. The loss used in fine-tuning is not detailed in the methods section. In line 294, "static mask is both a potential output and will be used in the later global pose optimization", the word "potential" here is confusing, is the static mask also used during finetuning?
2. Although a sliding window technique is used, the runtime reported is 30s for pairwise inference and 1min for global optimization given the 60 frames (around 2.5s) video input, which is slow.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: See weakness
soundness: 4
strengths: Clarity and Readability: The paper is well-written, with a clear statement of the motivation, methodology, and challenges, making it easy to follow.

Effective Demonstration: The video demonstration is impressive, showcasing efficient 4D reconstruction with significant potential for downstream applications.

Sufficient Novelty: While the proposed approach builds on DUSt3R, extending it to dynamic scenes is non-trivial. The authors clearly outline the challenges and propose effective solutions to address them.

Convincing Evaluation: The evaluation is thorough, comparing the method to different categories of existing techniques, with both quantitative and qualitative results presented.
summary: This paper extends DUSt3R, originally a feed-forward method for static scene reconstruction, to handle dynamic scenes by reconstructing scene geometry at each timestep. The initial performance of DUSt3R was poor due to its training on static datasets only. To improve results, this paper identifies and fine-tunes DUSt3R on several dynamic datasets. Consequently, the proposed method delivers competitive results in video depth estimation, camera pose estimation, and dense reconstruction. The approach is efficient and demonstrates results that rival those of specialized techniques.
weaknesses: 1. Mention of Related Work: For video depth estimation, self-supervised monocular depth methods such as Manydepth [a] and SC-DepthV3 [b] should be mentioned. SC-DepthV3, in particular, provides video depth estimation results on the Bonn dataset, which would make for a relevant comparison.

[a] The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth, CVPR 2021.

[b] SC-DepthV3: Robust Self-Supervised Monocular Depth Estimation for Dynamic Scenes, TPAMI 2024.

2. Robustness in Dynamic Regions: The statement “For most scenes, where most pixels are static, random samples of points will place more emphasis on the static elements” is generally true, but there are edge cases that require more robust approaches. For example, in the Bonn dataset, moving objects such as humans can occupy over 50% of the image. Potential solutions could include masking out dynamic regions using segmentation masks or re-running relative pose estimation after identifying dynamic regions.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: See Weaknesses
soundness: 4
strengths: Clearly identifies and addresses key shortcomings of DUSt3R when applied to dynamic scenes
- This work identifies non-obvious weaknesses of DUSt3R in foreground depth prediction, in addition to more predictable poor dynamic correspondence performance. Experiments show clear improvement accordingly
- Getting MonST3R to work with dynamic points is nontrivial. Careful finetuning choices in data and network, along with regularizations to handle a point representation taking into account time must have been necessary to achieve such strong performance
- Comprehensive ablations clearly show impact of several contributions in data, training and inference. Notably, PointOdyssey (dynamic) is essential, and fine-tuning specifically the decoder and head is critical.

Effective and efficient performance on an important task
- SOTA performance on depth and camera pose estimation across standard dynamic video datasets: Sintel, TUM-Dynamics, Bonn, KITTI, NYU-v2.
- Impressive the method is still competitive on static scenes ScanNet, showing this fine-tuning does not sacrifice original DUSt3R results.

My rating of 8 (Accept) is because of strong experiment results (SOTA Depth and Pose), resulting from clearly identified weaknesses (static and distant training data), from a strong starting point (DUSt3R); weaknesses are minor.


***Post-discussion Update*** I leave this paper at an 8. The rebuttal well addresses weaknesses in this paper. I do not feel it is worthy of a 10 because the contributions are not exceptional, i.e. it consists mostly of fine-tuning on synthetic data for a sizeable but not massive gain; but results, writing and method are still very strong and the paper should be accepted.
summary: This paper approaches the task of obtaining depth and camera pose given dynamic video. To do so, it begins from the DUSt3R architecture and pretraining. The paper argues optimizing dynamic scenes requires good estimation of foreground depth, and alignment based on dynamic objects; both of which DUSt3R struggles on due to static training data of mostly backgrounds. To this end, the paper fine-tunes on dynamic videos, as well as proposing slightly different optimization criteria. It shows state-of-the-art results on depth and camera pose estimation on several standard dynamic video benchmarks.
weaknesses: (Minor) Camera Pose Estimation suite could be more comprehensive
- E.g. COLMAP or masked COLMAP baseline would be good benchmarks.
- Why are DROID-SLAM, DPVO and ParticleSfM not run on TUM, and DROID-SLAM and DPVO not on ScanNet? 
- LEAP-VO is highly competitive with the proposed method. This is a minor weakness given this method also estimates depth and improves on DUSt3R, but should be considered when analyzing e.g. if this paper should be considered for strong accept.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakness
soundness: 3
strengths: - MonST3R successfully adapts the DUSt3R's pointmap representation to dynamic scenes. By estimating a pointmap for each timestep and representing them in the same camera coordinate frame, it can handle the movement of objects.
- Data Efficiency: Trains well with limited data via strategies.
- Good Performance: Strong in video depth and pose estimation.
summary: This paper presents MonST3R, a method for estimating geometry in dynamic scenes. It adapts the DUSt3R's pointmap representation to handle moving objects. Key points include: using suitable synthetic and real datasets for training despite data scarcity, introducing new optimization methods for downstream tasks like depth and pose estimation, demonstrating strong performance compared to prior works, and achieving good results in 4D reconstruction.
weaknesses: - There are also some high-quality synthetic datasets in the video field, such as DynamicStereo and IRSDataset. The author could consider expanding the scale of the training data to explore better model performance.
- There are doubts about the metrics used by the author when comparing video depth estimation. When the delta metric is below 0.7 or lower, it is usually considered that the prediction has completely collapsed. However, as far as the reviewer knows, the video depth estimation methods compared by the author have good performance and can provide basically correct depth estimation results at least on the Sintel dataset. Some metrics, such as NVDS, seem to be inconsistent with the report in the paper.
- It is necessary to explore the robustness of DUSt3R, that is, the failure cases when the scene is too dynamic.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weakness part.
soundness: 3
strengths: 1. The proposed problem of reconstructing static scenes and estimating per-timestep geometry of dynamic objects is compelling and relevant.
2. The paper presents a reasonable and well-structured solution to the defined problem.
3. The method achieves state-of-the-art performance, demonstrating its effectiveness.
summary: The paper introduces MonST3R, a novel geometry-first approach for estimating 3D structure from dynamic scenes in videos. Unlike existing methods that rely on multi-stage pipelines prone to errors, MonST3R adapts the DUSt3R pointmap representation, originally designed for static scenes, to handle moving objects by estimating pointmaps at each time step. This adaptation is achieved through fine-tuning on carefully selected synthetic and real-world datasets that provide depth and camera pose annotations, despite the challenge of limited data availability. The method demonstrates significant improvements in video depth and camera pose estimation, surpassing traditional optimization-based methods in robustness and efficiency. MonST3R is capable of generating reliable 4D reconstructions in a predominantly feed-forward manner, enhancing performance in various video-specific tasks.
weaknesses: 1. The complete optimization in Eq. 6 minimizes the loss to obtain optimal values for X, P_W, and \sigma. However, it seems that the smoothness loss in Eq. 4 is minimized specifically for R and T. The input notation L_{smooth}(X) might be misleading—should it be L_{smooth}(R,T) instead?
2. The notation for L_{flow}(X) also seems ambiguous. Would L_{flow}(F) or L_{flow}(F,S) be more appropriate to reflect the actual inputs?
3. There is no explanation provided for the notation global. Clarifying what the global parameters are and how S^{global} is derived would be helpful. Specifically, is S^{global} in Eq. 5 an extension of Eq. 2 that uses F_{cam}^{global,t→t′} and F_{est}^{global,t→t′}
4. The proposed network is trained on diverse datasets, including PointOdyssey, TartanAir, Spring, and Waymo Perception, but the authors rely on a pretrained optical flow model. This could introduce a domain gap between the depth network and the optical flow network. Were there any issues related to domain gaps during the experiments?
5. It would be beneficial if the authors provided depth evaluation results for static scenes and dynamic objects separately. This would clarify whether the performance improvement is due to coherent depth estimation on static scenes or the enhanced depth quality of dynamic objects.
6. Do the authors use zero-shot models like Marigold and Depth-Anything? The use of the Waymo dataset (similar to KITTI) and the PointOdyssey dataset (with properties akin to Sintel) might explain why the proposed method outperforms these models. Can the authors specify which components of their approach are most critical for achieving this performance boost?
7. Overall, the technical parts is not clearly written.

Meta-review: The paper present an approach to depth and pose estimation from monocular video.
It extends DUST3R to handle dynamic scenes by fine-tuning on appropriate datasets and tailoring its global pose optimization to dynamic scenes. Specifically, a sliding-window based optimization improves run-time and losses encouraging smooth camera trajectories and consistency with optical flow improve accuracy.
The method compares favourably against several representative baselines for video depth and pose estimation. After clarifications provided in the rebuttal, all reviewers unanimously recommended to accept the paper. The AC agrees with this assessment.

Decision: Accept (Spotlight)