MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion
Authors: Junyi Zhang
Charles Herrmann
Junhwa Hur
Varun Jampani
Trevor Darrell
Forrester Cole
Deqing Sun
Ming-Hsuan Yang
Venue: ICLR 2025 Spotlight (2025)
Keywords: 3D computer vision
structure from motion
depth estimation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We adapt pointmaps for dynamic scenes and do a primarily feed forward estimation of geometry for videos, resulting in SOTA performance for several downstream tasks including estimating camera pose and video depth.

Abstract: Estimating geometry from dynamic scenes, where objects move and deform over time, remains a core challenge in computer vision. Current approaches often rely on multi-stage pipelines or global optimizations that decompose the problem into subtasks, like depth and flow, leading to complex systems prone to errors. In this paper, we present Motion DUSt3R (MonST3R), a novel geometry-first approach
that directly estimates per-timestep geometry from dynamic scenes. Our key insight is that by simply estimating a pointmap for each timestep, we can effectively adapt DUSt3R’s representation, previously only used for static scenes, to dynamic scenes. However, this approach presents a significant challenge: the scarcity of suitable training data, namely dynamic, posed videos with depth labels. Despite this, we show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics, even without an explicit motion representation. Based on this, we introduce new optimizations for several downstream video-specific tasks and demonstrate strong performance on video depth and camera pose estimation, outperforming prior work in terms of robustness and efficiency. Moreover, MonST3R shows promising results for primarily feed-forward 4D reconstruction. Interactive 4D results, source code, and trained models are available at: https://monst3r-project.github.io/.

Published as a conference paper at ICLR 2025
MONST3R: A SIMPLE APPROACH FOR ESTIMATING GEOMETRY IN THE PRESENCE OF MOTION
Junyi Zhang'
Charles Herrmann 2
Junhwa Hur Varun Jampani
Trevor Darrell
Forrester Cole²
Deqing Sun
Ming- Ming-Hsuan Hsuan Yang 2,4,a
UC Berkeley
Google DeepMind
3 Stability AI *UC Merced
Video Depth
Camera Intrinsics
Video Input
Dynamic Point Cloud & Camera Pose
Dynamic Static Mask
Figure 1: MonST3R processes a dynamic video to produce a time-varying dynamic point cloud, along with per-frame camera poses and intrinsics, in predominantly feed-forward manner. This representation then enables the efficient computation of downstream tasks, such as video depth estimation and dynamic/static scene segmentation.
ABSTRACT
Estimating geometry from dynamic scenes, where objects move and deform over time, remains core challenge computer vision. Current approaches often rely multi-stage pipelines or global optimizations that decompose the problem into subtasks, like depth and flow, leading complex systems prone to errors. In this paper, we present Motion DUSt3R (MonST3R). novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes. Our key in- sight is that by simply estimating pointmap for each timestep, can effectively adapt DUSt3R's ' representation, previously only used for static scenes, to dynamic scenes. However, this approach presents significant challenge: the scarcity of suitable training data, namely dynamic, posed videos with depth labels. Despite this, show that by posing the problem eral suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics, even without an explicit motion representation. Based on this, we introduce new optimizations for several downstream video-specific tasks and demonstrate strong performance on video depth and camera pose estimation, outperforming prior work in terms of robust- ness and efficiency. Moreover, MonST3R shows promising results for primarily feed-forward 4D reconstruction. Interactive 4D results, source code, and trained models are available at: https: htt=ss://most..rppreres projectt.................or..... wwreect..gttttt github.. io/.
fine-tuning task, identifying sev-
INTRODUCTION
Despite recent progress in 3D computer vision, estimating geometry from videos of dynamic scenes remains a fundamental challenge. Traditional methods decompose the problem into subproblems
Project lead, "Equal contribution

---

Published as a conference paper at ICLR 2025
such depth, optical flow, or trajectory estimation, addressed with specialized techniques, and then combine them through global optimization or multi-stage algorithms for dynamic scene reconstruc- tion (Luiten et al., 2020; Kumar et al.. 2017; Bårsan et al., 2018; Mustafa et al., 2016). Even recent work often takes optimization-banl
approaches given intermediate estimates derived from monocu- lar video (Lei et al., 2024; Chu al., 2024; Wang al., 2024b; Liu et al., 2024; Wang et al., 2024a). However, these multi-stage methods are usually slow, brittle, and prone to error at each step.
While highly desirable, end-to-end geometry learning from a dynamic video poses a significant challenge, requiring a suitable representation that can represent the complexities of camera motion, multiple object motion, and geometric deformations, along with annotated training datasets. While prior methods have centered on the combination of motion and geometry. motion is often difficult to directly supervise due to lack of annotated training data. Instead, we explore using only geometry to represent dynamic scenes, inspired by the recent work DUSt3R (Wang et al., 2024c).
For static scenes, DUSt3R introduces a new paradigm that directly regresses scene geometry. Given pair of images, DUSI3R produces a pointmap representation which associates every pixel in each image with an estimated 3D location (i.e., zyz2 and aligns these pair of pointmaps in the camera coordinate system of the first frame. For multiple frames, DUS(3R accumulates the pairwise estimates into a global point cloud and uses it to solve numerous standard 3D tasks such as single- frame depth, multi-frame depth, or camera intrinsics and extrinsics.
We leverage DUS(RR' s pointmap representation to directly estimate geometry of dynamic scenes. Our key insight is that pointmaps be estimated per timestep and that representing them in the same camera coordinate frame still makes conceptual sense for dynamic scenes. As shown in Fig. 1, an estimated pointmap for the dynamic scene appears as a point cloud where dynamic objects appear at multiple locations, according to how they move. Multi-frame alignment can be achieved by align- ing pairs of pointmaps based on static scene elements. This setting a generalization of DUSt3R to dynamic scenes and allows us to use the same network and original weights as a starting point.
One natural question if DUSt3R can already and effectively handle video data with moving ob- jects. However, as shown in Fig. 2, we identify two significant limitations stemming from the distribution of DUSt3R's s training data. First, since its training data contains only static scenes, DUSt3R fails to correctly align pointmaps of scenes with moving objects; it often relies on moving foreground objects for alignment, resulting in incorrect alignment for static background elements. Second, since its training data consists mostly of buildings and backgrounds, DUSt3R sometimes fails to correctly estimate the geometry of foreground objects, regardless of their motion, and places them in the background. In principle, both problems originate from domain mismatch between training and test time and can be solved by re-training the network.
However, this requirement for dynamic, posed data with depth presents a challenge, primarily due to its scarcity. Existing methods, such as COLMAP (Schönberger & Frahm, 2016), often struggle with complex camera trajectories highly dynamic scenes, making it challenging to produce even pseudo ground truth data for training. To address this limitation, we identify several small-scale datasets that possess the necessary properties for our purposes.
Our main finding is that, surprisingly, we can successfully adapt DUSt3R to handle dynamic scenes by identifying suitable training strategies designed maximally leverage this limited data and fine- tuning on them. We then introduce several new optimization methods for video-specific tasks using these pointmaps and demonstrate strong performance on video depth and camera pose estimation, as well as promising results for primarily feed- nerward 4D reconstruction.
The contributions of this work are as follows:
We introduce Motion DUS(3R (MonST3R), a geometry-first approach to dynamic scenes that directly estimates geometry in the form of pointmaps, even for moving scene elements. To this end, we identify several suitable datasets and show that, surprisingly, a small-scale fine-tuning achieves promising results for direct geometry estimation of dynamic scenes. MonST3R obtains promising results on several downstream tasks (video depth and cam- era pose estimation). In particular, MonST3R offers key advantages over prior work: enhanced robustness, particularly in challenging scenarios; increased speed compared to optimization- based methods; and competitive results with specialized techniques in video depth estimation, camera pose estimation and dense reconstruction.
2

---

Published as a conference paper at ICLR 2025
Issue 1: Alignment based dynamic object
Issue 2: F Foreground depth not estimated correctly
DUSI3R X
MonS13R
DUSI3R X
MonST3R
Limitation of DUS(3R on dynamic scenes. Left: DUSI3R aligns the moving foreground subject and misaligns the background points as it is only trained on static scenes. Right: DUSI3R fails to estimate the depth of foreground subject, placing it in the background.
Figure 2:
2 RELATED WORK
Structure from motion and visual SLAM. Given set of 2D images, structure from motion (SfM) (Schönberger & Frahm, 2016; Teed & Deng, 2018; Tang & Tan, 2018) or visual SLAM (Teed & Deng, 2021; Mur-Artal al., 2015; Mur-Artal & Tardós, 2017; Engel al., 2014; Newcombe et al., 2011) estimate 3D structure of a scene while also localizing the camera. However, these methods struggle with dynamic scenes with moving objects, which violate the epipolar constraint.
To address this problem, recent approaches have explored joint estimation of depth, camera pose, and residual motion, optionally with motion segmentation exploit the epipolar constraints on the stationary part. Self-supervised approaches (Gordon et al., 2019; Mahjourian et al., 2018; Go- dard al., 2019; Yang al., 2018) learn these tasks through self-supervised proxy tasks. Casual- SAM (Zhang et al., 2022) finetunes depth network at test time with joint estimation of camera pose and movement mask. Robust-CVD (Kopf al., 2021) jointly optimizes depth and camera pose given optical flow and binary masks for dynamic objects. Our approach directly estimates 3D structure of a dynamic scene in the pointmap representation without time-consuming test-time finetuning.
Representation for static 3D reconstruction. Learning-basee approaches reconstruct static 3D geometry of objects scenes by learning strong 3D priors from training datasets. Commonly used output representations include point clouds (Guo et al., 2020; Lin et al., 2018), meshes (Gkioxari al., 2019; Wang al., 2018), voxel (Sitzmann et al., 2019; Choy et al., 2016; Tulsiani al., 2017), implicit representation (Wang et al., 2021a; Peng et al., 2020; Chen & Zhang, 2019), etc.
DUSt3R (Wang et al., 2024c) introduces pointmap representation for scene-level 3D reconstruc- tion. Given two input images, the model outputs a 3D point of each pixel from both images in the camera coordinate system of the first frame. The model implicitly infers camera intrinsies, rela- tive camera pose, and two-view geometry and thus can output an aligned points cloud with learned strong 3D priors. However, the method targets only static scenes. MonST3R shares the pointmap representation of DUSt3R but targets scenes with dynamic objects.
Learning-based visual odometry. Learning- based visual odometry replaces hand-designed parts of geometry- based methods (Mur-Arte Artal et al., 2015; Mur-Artal & Tardós, 2017: Engel et al.. 2017) and enables large-scale training for better generalization even with moving objects. Trajectory-base approaches (Chen et al., 2024; Zhao et al., 2022) estimate long-term trajectories along video se- quence, classify their dynamic and static motion, and then localize camera via bundle adjustment. Joint estimation approaches additionally infer moving object mask (Shen et al., 2023) or optical flow (Wang et al., 2021b) to be robust to moving objects while requiring their annotations during training. In contrast, method directly outputs dynamic scene geometry via a pointmap represen- tation and localizes camera afterwards.
Monocular and video depth estimation. Recent deep learning works (Ranftl al., 2020; 2021; Saxena et al., 2024; Ke et al., 2024) target zero-shot performance and with large-scale training com- bined with synthetic datasets (Yang al., 2024a;b) show strong generalization diverse domains.
3

---

Published as a conference paper at ICLR 2025
Table I:
Training datasets used fine-tuning dynamic scenes. All datasets provide both camera pose and depth, and most of them include dynamic objects.
Dataset
Domain
Scene type
of frames
# of Scenes
Dynamies
Ratio
PointOdyssey (Zheng al., 2023)
Synthetic
Indoors Outdoors
200k
131
Realistic
50%
TartanAir (Wang al., 2020)
Synthetic Synthetic
Indoors & Outdoors
1000k
163
None
25%
Spring (Mehl et al,, 2023)
Outdoors
6k
37
Realistic
5%
Waymo Perception (Sun et al., 2020)
Real
Driving
160k
798
Driving
20%
However, for video, these approaches suffer from flickering (temporal inconsistency between nearby estimates) due to their process of only a single frame and invariant training objectives.
Early approaches to video depth estimation (Luo al., 2020; Zhang al., 2021) improve temporal consistency by fine-tuning depth models, and sometimes motion models, at test time for each input video. Self-supervised methods (Watson al., 2021; Sun al., 2023) are also explored to enhance temporal coherence without explicit annotations. Two recent approaches attempt to improve video depth estimation using generative priors. However, Chronodepth (Shao et al., 2024) still suffers from flickering due to its window-based inference, and DepthCrafter (Hu et al., 2024) produces seale-/shift-invarfe depth, which is unsuitable for many 3D applications (Yin al., 2021).
4D reconstruction. Concurrently approaches (Lei et al., 2024; Chu et al., 2024; Wang et al., 2024b; Liu et al., 2024) introduce 4D reconstruction methods of dynamic scenes. Given a monocular video and pre-computed estimates (e.g., 2D motion trajectory, depth, camera intrinsics and pose, etc.), the approaches reconstruct the input video in 4D space via test-time optimization of 3D Gaus- sians (Kerbl et al., 2023) with deformation fields, facilitating novel view synthesis in both space and time. Our method is orthogonal to the methods and estimate geometry from videos in a forw-forward manner. Our estimates could be used as initialization or intermediate signals for these methods.
3 METHOD
3.1 BACKGROUND AND BASELINES
Model architecture. Our architecture is based on DUSt3 (Wang et al., 2024c), ViT-based archi- tecture (Dosovitskiy et al., 2021) that is pre-trained on a cross-view completion task (Weinzaepfel et al., 2023) in a self-supervised manner. Two input images are first individually fed to a shared encoder. A following transformer- bassed decoder processes the input features with cross-attention. Then two separate heads at the end of the decoder output pointmaps of the first and second frames aligned in the coordinate of the first frame.
Baseline with mask. While DUSt3R is designed for static scenes as shown in Fig. 2, we analzye its applicability to dynamic scenes by using knowledge of dynamic elements (Chen et al., 2024; Zhao et al., 2022). Using ground truth moving masks, we adapt DUSt3R by masking out dynamic objects during inference at both the image and token levels, replacing dynamic regions with black pixels in the image and corresponding tokens with mask tokens. This approach, however, leads to degraded pose estimation performance (Sec. 4.3), likely because the black pixels and mask tokens are -atto--distribution with respect to training. This motivates us to address these issues in this work.
3.2 TRAINING FOR DYNAMICS
Main idea. While DUSt3R primarily focuses on static scenes, the proposed MonST3R can estimate the geometry of dynamic scenes over time. Figure. shows a visual example consisting of a point cloud where dynamic objects appear at different locations, according to how they move.
Similar to DUSt3R, for a single image I' at time t, MonST3R also predicts a pointmap X xx For a pair of images, I' and I' we adapt the notation used in the global optimization section of DUS(3R. The network predicts two corresponding pointmaps, Xitte and s'et'" with confidence map, Crast" and Ct' stot'' The first element in the superseript indicates the frame that the pointmap corresponds to, and tet' indicates that the network receives two frames at t' and that the pointmaps are in the coordinate frame of the camera at t. The key difference from DUSt3R is that each pointmap in MonST3R relates to single point in time.
4

---

Published as a conference paper at ICLR 2025
Training datasets. A key challenge in modeling dynamic scenes per-timestep pointmaps lies in the scarcity of suitable training data, which requires synchronized annotations of input images, camera poses, and depth. Acquiring accurate camera poses for real-world dynamic scenes is partic- ularly challenging, often relying on sensor measurements or post-processing through structure from motion (SfM) (Schönberger et al., 2016; Schönberger & Frahm, 2016) while filtering moving objects. Consequently, we leverage primarily synthetic datasets, where accurate camera poses and depth can be readily extracted during the rendering process.
For our dynamic fine-tuning, we identify four large video datasets: three synthetic datasets PointOdyssey (Zheng et al., 2023), TartanAir (Wang et al., 2020), and Spring (Mehl et al., 2023), along with the real-world Waymo dataset (Sun et al., 2020), as shown in Tab. 1. These datasets con- tain diverse indoor/outdoor scenes, dynamic objects, camera motion, and labels for camera pose and depth. PointOdyssey and Spring are both synthetically rendered scenes with articulated, dynamic objects; TartanAir consists of synthetically rendered drone fly-throughs of different scenes without dynamic objects; and Waymo is a real-world driving dataset labeled with LiDAR.
During training, we sample the datasets asymmetrically to place extra weight on PointOdyssey (more dynamic, articulated objects) and less weight on TartanAir (good scene diversity but static) and Waymo (a highly specialized domain). Images are downsampled such that their largest dimen- sion is 512.
Training strategies. Due to the relatively small size of this dataset mixture. we adopt several train- ing techniques designed to maximize data efficiency. First, we only finetune the prediction head and decoder of the network while freezing the encoder. This strategy preserves the geometric knowledge in the CroCo (Weinzaepfel et al., 2022) features and should decrease the amount of data required for fine-tuning. Second, we create training pairs for each video by sampling two frames with temporal strides ranging from to 9. The sampling probabilities increase linearly with the stride length, with the probability of selecting stride 9 being twice that of stride 1. This gives camera and scene motion and more heavily weighs larger motion. Third, we utilize a Field-of ofeldoo--Ve View augmentation technique using center crops with various image scales. This encourages the model to generalize across different camera intrinsics, even though such variations are relatively infrequent in the training videos. We train the model with the same confidence-awee regression loss DUS13R.
larger diversity of
3.3 DOWNSTREAM APPLICATIONS
Instrinsics and relative pose estimation. Since the intrinsic parameters are estimated based on the pointmap in its own camera frame X the assumptions and computation listed in DUSt3R are still valid, and we only need to solve for focal length ft to obtain the camera intrinsics K'.
To estimate relative pose P = [' where R and T represent the camera's 's rotation and translation, respectively, dynamic objects violate the assumptions for methods relying on correspon- dences between two views, e.g., epipolar matrix (Hartley & Zisserman, 2003) with 2D and Procrustes alignment (Luo & Hancock, 1999) with 3D correspondences. Instead, we leverage per-pixel 2D-3D correspondences within the same view and use PnP (Lepetit et al., 2009) to recover the relative pose:
T arg min R.T
x E VEI
(K"
[RX] ste''
+ +T))l
where x is the pixel coordinate matrix and (T-)) is the projection operation (Ir, (2.2) 2) - (x/2, v/z), To improve the robustness to outliers, we use RANSAC (Fischler & Bolles, 1981) and define valid correspondences by taking threshold of the estimated confidence mask, I = {i C d' a).
Confident static regions. We can infer static regions in frames t, t' by comparing the estimated optical flow with the flow field that results from applying only camera motion from to t' to the pointmap at f. The two flow fields should agree for pixels where the geometry has been correctly estimated and are static. Given a pair of frames I' and Id,, we first compute two sets of pointmaps X e' x''stee stct and xtit'ct a'''s We then use these pointmaps to solve for the camera intrinsics (K' and Kt' ) for each frame and the relative camera pose from t to t', pi-t' above. We then compute the optical flow field induced by camera motion, Fi-t', by backprojecting
(e-++|T== as
5

---

Published as a conference paper at ICLR 2025
Lulign
Window
Accomulate
+
x 'x
Leluw
+
8. R
Sumooth
Variables Optimization
X X
Fest
Fiee
Input Video
Pairwise Pointmup Optical
Accumalated Intermediatess
Parameters
Figure 3: Dynamic global point cloud and camera pose estimation. Given fixed sized of tem- poral window, we compute pairwise pointmap for each frame pair with MonST3R and optical flow from off-the-shelf method. These intermediates then serve as inputs to optimize global point cloud and per-frame camera poses. Video depth can be directly derived from this unified representation.
each pixel in 3D, applying relative camera motion, and projecting back to image coordinate,
pt-t' cam
T (D K' R - K* & K T x,
(2)
where & is the pixel coordinate matrix x in homogeneous coordinates, and Distee' estimated depth extracted from the point map Xtiee", Then we compare it with optical flow (i.e., F(x"") computed by an off-the-shelf optical flow method (Wang et al., 2024d) and infer the static mask S via simple thresholding:
Fest |1:11
with threshold a, |I-IIu for smooth-LI norm (Girshick, 2015), and [-] for the Iverson bracket. This confident, static mask is both a potential output and will be used in the later global pose optimization.
3.4 DYNAMIC GLOBAL POINT CLOUDS AND CAMERA POSE
Even short video contain numerous frames (e.g. a 5-second video with 24 fps gives 120 frames) making non-trivial extract single dynamic point cloud from pairwise pointmap estimates across the video. Here, we detail the steps to simultaneously solve for global dynamic point cloud and camera poses by leveraging our pairwise model and the inherent temporal structure of video.
Video graph. For global alignment, DUSI3R constructs a connectivity graph from all pairwise frames, process that is prohibitively expensive for video. Instead, shown the left of Fig. 3, we process video with a sliding temporal window, significantly reducing the amount of compute required. Specifically, given video V I .......I I", we compute pointmaps for all pairs e (t, f') within a temporal window of size w, W = (a, b) a, E [t...., w). a o b} and for all valid windows W. To further improve the run time, we also apply strided sampling.
Dynamic global point cloud and pose optimization. The primary goal is to accumulate all pairwise pointmap predictions (e.g., t Xt':t stct" into the same global coordinate frame to produce world- coordinate pointmap X "x3x3 To do this, as shown in Fig. 3, use DUSt3R's s aligmment loss and add two video specific loss terms: camera trajectory smoothness and flow projection.
We start by re-parameterizingg the global pointmaps X with camera parameters P [R K and fermframe depthmap D', as X = pr- h(K' k (DDjssDD
Di,))) with (i.j) for pixel coordinate and h() for homogeneous mapping. allows us to define losses directly on the camera parameters. To simplify the notation for function parameters, we use as shortcut for pK D
First, use the alignment term in DUSt3R which aims find single rigid transformation pt: that aligns each pairwise estimation with the world coordinate pointmaps, since both Xhee and t' are in the same camera coordinate frame:
Lalign XX Pw)
Σ Σ ΣΣN [|tti (X' o" o x 12
(4)
W eEW xEe
where of pairwise scale factor. To simplify the notation, we use the directed edge = (t, t') interchangeably with tel'.

---

Published as a conference paper at ICLR 2025
We use camera trajectory smoothness loss to encourage smooth camera motion by penalizing large changes in camera rotation and translation in nearby timesteps:
Ssmooth X)
Σ
(RRTR* -
[ R R
=|,, + - -
(5)
|| ll2 is used for the where the Frobenius norm || ||r is used for the rotation difference, the L2 norm translation difference, and I is the identity matrix.
We also use a flow projection loss to encourage the global pointmaps and camera poses to be con- sistent with the estimated flow for the confident, static regions of the actual frames. More precisely, given two frames t, l', using their global pointmaps, camera extrinsics and intrinsics, we compute the flow fields from taking the global pointmap X assuming the scene is static, and then moving the camera from to t'. We denote this value Fglobal:t->t' similar to the term defined in the confi- dent static region computation above. Then we can encourage this to be close to the estimated flow, Ftait' in the regions which are confidently static Salobal:t-st according the global parameters:
Lrow (X) = Σ Σ
(Fglobala->t
F est
cam
W EW
where indicates element- wise multiplication. Note that the confident static mask is initialized using the pairwise prediction values (pointmaps and relative poses) as described in Sec. 3.3. During the optimization, we use the global pointmaps and camera parameters to compute Fglobal and update the cam
confident static mask. Please refer Appendix D for more details on Lsmooh and Lnow-
The complete optimization for our dynamic global point cloud and camera poses is:
X arg min Calign (X, o, Pw) Wsmooth Camooth (X) + Witow (X), P.P
where Wsmooth Wftow are hyperparameters. Note, based on the reparameterization above, X includes all the information for D, P.K.
Video depth. We can now easily obtain lemporally-consisten video depth, traditionally addressed as a standalone problem. Since our global pointmaps are parameterized by camera pose and per- frame depthmaps D just returning D gives the video depth.
4 EXPERIMENTS
MonST3R runs on a monocular video of a dynamic scene and jointly optimizes video depth and camera pose. We compare the performance with methods specially designed for each individual subtask (i.e,, depth estimation and camera pose estimation), as well as monocular depth methods.
4.1 EXPERIMENTAL DETAILS
Training and Inference. We fine-tune the DUSt3R's ViT-Base decoder and DPT heads for 25 epochs, using 20,000 sampled image pairs per epoch. We use the AdamW optimizer with learning rate of x 10-1 and a mini-batch size of 4 per GPU. Training took one day on 2x RTX 6000 48GB GPUs. Inference for a 60-frame video with w 9 and stride 2 (approx. 600 pairs) takes around 30s.
Global Optimization. For global optimization Eq. (7), we set the hyperparameter of each weights to be Wsmooth = 0.01 and Whow = 0.01. We only enable the flow loss when the average value is below 20, when the poses are roughly aligned. The motion mask is updated during optimization if the per-pixel flow loss is higher than 50. We use the Adam optimizer for 300 iterations with learning rate of 0.01, which takes around minute for a 60-frame video on a single RTX 6000 GPU.
4.2 SINGLE-FRAME AND VIDEO DEPTH ESTIMATION
Baselines. We compare method with video depth methods, NVDS (Wang et al., 2023), Chron- oDepth (Shao et al., 2024), and concurrent work, DepthCrafter (Hu et al., 2024), as well as single- frame depth methods, Depth-Anything-V- (Yang et al., 2024b) and Marigold (Ke al., 2024).
7

---

Published as a conference paper at ICLR 2025
Table 2: Video depth evaluation on Sintel, Bonn, and KITTI datasets. We evaluate for both scale- mud-ssift-invarian and invariant depth. The best and second best results each category are bold and underlined, respectively.
Sintel
Bonn
KITTI
Alignment
Cutegory
Method
Abs Rel 1 8-1..25 T |lbs Rel 1 6<1.25 1
Abs Rel 1 6 <1.25 f
Single-framd frame depth
Marigold
0.532
$1.5
0.091
93.1
0.149
79.6
Depth-Anything-22
0.367
55.4
0.106
92.1
0.140
80.4
48.3
0.167
76.6
0.253
Per-sequence scale & shift
Video depth
Ceroh
0.687
48.6
0.100
91.1
0.167
75.9
DepthCrafter (Sep. 2024)
0.292
69.7
0.075
97.1
0.110
88.1
Rubust-CVD
0.703
47.8
Joint video depth pose
CasualSAM
0.387
54.7
0.169
73.7
0.246
62.2
MonST3R
0.335
58.5
0.063
96.4
0.104
89.5
Per-sequence scale
Video depth Joint depth & pose MonST3R
DepthCrafter (Sep. 2024)
0.692
53.5
0.217
57.6
0.141
81.8
0.345
56.2
0.065
96.3
0.106
89.3
Table 3: Single-frame depth evaluation. We report the performance on Sintel, Bonn, KITTI, and NYU-v2 (static) datasets. MonST3R achieves overall comparable results to DUSt3R.
Sintel
Bonn
KITTI
NYU-v2 (static)
Method
Abs Rel k,2225 f Ahs Rel 4 8<1.25
Abs Rel 4. 8 <1.25 1| Abs Rel 4 å <1.25 f
DUSGR
0.424
58.7
0.141
82.5
0.112
86.3
0.080 0.091
90.7
MonST3R
0.345
56.5
0.076
93.9
0.101
89.3
88.8
We also compare with methods for joint video depth and pose estimation, CasualSAM (Zhang al., 2022) and Robust-CVD (Kopf et al., 2021), which address the same problem as us. This comparison particularly important since joint estimation is substantially more challenging than only estimat- ing depth. Of note, AA relies on heavy optimization, whereas ours runs in a feed-forward manner with only lightweight optimization.
Benchmarks and metrics. Similar to DepthCrafter, we evaluate video depth on KITTI (Geiger al., 2013), Sintel (Butler et al., 2012), and Bonn (Palazzolo et al., 2019) benchmark datasets, cov- ering dynamic and static, indoor and outdoor, and realistic and synthetic data. For monocular/single- frame depth estimation, we also evaluate on NYU-v2 (Silberman et al., 2012).
Our evaluation metrics include absolute relative error (Abs Rel) and percentage of inlier points 8 1.25, following the convention (Hu et al., 2024; Yang al., 2024b). All methods output scale- and/or shift- invariant depth estimates. For video depth evaluation, we align single scale and/or shift factor per each sequence, whereas the single-frame evaluation adopts per-frame median scaling, following Wang et al. (2024c). As demonstrated by Yin et al. (2021), shift is particularly important scene and is important to predict.
in the 3D geometry
Results. As shown in Tab. 2, MonST3R achieves competitive and even better results, even out- performing specialized video depth estimation techniques like DepthCrafter (a concurrent work). Furthermore, MonST3R significantly outperforms DepthCrafter (Hu al., 2024) with scale-only normalization. As in Tab. 3, even after our fine-tuning for videos of dynamic scenes, the perfor- mance on single-frame depth estimation remains competitive with the original DUS(3R model.
4.3 CAMERA POSE ESTIMATION
Baselines. We compare with not only direct competitors (i.e., CasualSAM and Robust-CVD), but also a range of learning-based visual odometry methods for dynamic scenes, such as DROID- SLAM (Teed & Deng, 2021), Particle-SIM (Zhao al., 2022), DPVO (Teed et al., 2024), and LEAP-VO (Chen et al., 2024). Notably, several methods (e.g.. DROID-SLAM, DPVO, and LEAP- VO) require ground truth camera intrinsic input and ParticleSfM is an optimization-
method that runs 5x slower than ours. We also compare with the "DUSt3R with mask" " baseline in Sec. 3.1 to see if DUSt3R performs well on dynamic scenes when ground truth motion mask is provided.
Benchmarks and metrics. We evaluate the methods on Sintel (Butler et al., 2012) and TUM- dynamics (Sturm et al., 2012) (following CasualSAM) and ScanNet (Dai et al., 2017) (following
8

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation on camera pose estimation on the Sintel, TUM-dynamic, and ScanNet. The best and second best results are bold and underlined, respectively. MonST3R achieves competitive and even better results than pose-specific methods, even without ground truth camera intrinsics.
Sintel
TUM-dynumics
ScanNet (stutic)
Category Method
ATE RPE trans + RPE
ATE + RPE trans & RPE
ATE + RPE trans + RPE rot
DROID-SLAM'
0.175
0.084
1.912
DPVO
0.115
0.072
1.975
Pose only
ParticleSIM
0.129
0.031
0,535
0.136
0.023
0.836
LEAP-VO'
0.089
0.066
1,250
0.046
0.027
0.385
0.070
0.018
0.535
Robust-CVD
0.360
0.154
3.443
0.189
0.071
3.681
0.227
0.064
7.374
Joint depih CasualSAM
0.141
0.035
0.615
0,045
0.020
0.841
0.158
0.034
1.618
& pose
DUSB3 w/ mask?
0.417
0.250
5.796
0,127
0.062
3.099
0.081
0.028
0.784
MonST3R
0.108
0.042
0.732
0.074
0.019
0.905
0.068
0.017
0.545
requires ground truth camera intrinsies as input. unable to estimate the depth of foreground object.
ParticleSfM) to test generalization static scenes well. On Sintel, follow the same evaluation protocol as in Chen et al. (2024); Zhao et al. (2022), which excludes static scenes or scenes with perfectly-straight camera motion, resulting in total 14 sequences. For TUM-dynamics and ScanNet, we sample the first 90 frames with the temporal stride of 3 to save compute. We report the same metric as Chen et al. (2024); Zhao et al. (2022): Absolute Translation Error (ATE), Relative Trans- lation Error (RPE trans), and Relative Rotation Error (RPE rot), after applying a Sim(3) Umeyama alignment on prediction the ground truth.
Results. In Tab. 4, MonST3R achieves the best accuracy in Sintel and ScanNet among methods to joint depth and pose estimation and performs competitively to pose-only methods, even without using ground truth camera intrinsics. Our method also generalizes well to static scenes (i.e., Scan- Net) and shows improvements over even DUSt3R, which proves the effectiveness of our designs (e.g., Eq. (7)) for video input.
4.4 JOINT DENSE RECONSTRUCTION AND POSE ESTIMATION
Fig. qualitatively compares our method with CasualSAM and DUSt3R on video sequences for joint dense reconstruction and pose estimation DAVIS (Perazzi et al., 2016). For each video sequence, we visualize overlayed point clouds aligned with estimated camera pose, showing as two rows with different view points for better visualization. As discussed in Fig. DUS(3R struggles with estimating correct geometry of moving foreground objects, resulting in failure of joint camera pose estimation and dense reconstruction. CasualSAM reliably estimates camera trajectories while sometimes failing to produce correct geometry estimates for foreground objects. MonST3R outputs both reliable camera trajectories and reconstruction of entire scenes along the video sequences.
4.5 ABLATION STUDY
Table 5 presents an ablation study analyzing the impact of design choices in our method, including the selection of training datasets, fine-tuning strategies, and the novel loss functions used for dy- namic point cloud optimization. Our analysis reveals that: (I) all datasets contribute to improved camera pose estimation performance; (2) fine-tuning only the decoder and head outperforms alter- native strategies; and (3) the proposed loss functions enhance pose estimation with minimal impact on video depth accuracy.
Discussions. While MonST3R represents promising step towards directly estimating dynamic ge- ometry from videos as well as camera pose and video depth, limitations remain. While our method can, unlike prior methods, theoretically handle dynamic camera intrinsics, we find that, in prac- tice, this requires careful hyperparameter tuning or manual constraints. To trade off compute and performance, our global alignment applies a relatively small size of the sliding window, making it vulnerable to long-term occlusion. Additionally, Like many deep learning methods, MonST3R struggles with direction to make MonST3R more robust in-the-wild videos.
inputs, such as open fields. Expanding the training set is a key

---

Published as a conference paper at ICLR 2025
Input Video
CasualSAM
DUSt3R
MonST3R (Ours)
Figure 4: Qualitative comparison. Compared to CasualSAM and DUSt3R, our method outputs both reliable camera trajectories and geometry of dynamic scenes. Refer to Fig. A5 for more results.
Table 5:
Ablation study Sintel dataset. For each category, the default setting is underlined, and the best performance bold.
Camera estimation
Video depth estimation
Variants
ATE
RPE trans
RPE rot
Abs <11.5
No linetune (DUSt3R weights)
0.354
0.167
0,996
0.482
56.5
PO
0.220
0.129
0.901
0.378
53.7
Training dataset
PO+TA
0.054
0.362
PO+TA+Spring
0.121
0.046
0.777
0.329
w/ S Waymo
0.167
0.107
1.136
0.462
54.0
w/ all 4 datasets
0.108
0.042
0.732
0.335
58.5
Full model finetune
0.110
0.738
0.352
55.4
Training strategy
Finetune decoder & head
0.108
0.042
0.732
0.335
58.5
Finetunc head
0.185
0.128
0.860
0.394
55.7
w/o flow loss
0.339
57.7
w/o static region mask
0.132
0.049
0.899
0.334
58.7
Inference
w/o smoothness loss
0.127
0.060
1.456
0.333
58.4
Full
0.108
0.042
0.732
0.335
58.5
CONCLUSIONS
We present MonST3R, simple approach to directly estimate geometry for dynamic scenes and ex- tract downstream information like camera pose and video depth. MonST3R leverages per- cer-timestep pointmaps powerful representation for dynamic scenes. Despite being finetuned relatively small training dataset, MonST3R achieves impressive results on downstrcam tasks, surpassing even state-of-the. oottt---ttttttt specialized techniques.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We would like express gratitude Youming Deng, Hang Gao, and Haven Feng for their helpful discussions and extend special thanks to Brent Yi, Chung Min Kim, and Justin Kerr for their valuable help with online visualizations. This work is supported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior/ Interior Business Center (DOI/IBC) contract number 140D0423C0074. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC or the U.S. Government.
REFERENCES
Ioan Andrei Bårsan, Peidong Liu, Mare Pollefeys, and Andreas Geiger. Robust dense mapping for large-scale dynamic environments. In ICRA, pp. 7510-7517, 2018. 2
Daniel J. Butler, Jonas Wulff, Garrett B. Stanley, and Michael J. Black. A naturalistic open source movie for optical flow evaluation. ECCV, pp. 61 625, 2012.
Weirong Chen, Le Chen, Rui Wang, and Mare Pollefeys. LEAP-VO: Long-term effective any point tracking for visual odometry. CVPR, pp. 19844-19853, 2024. 3, 4, 8,
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. CVPR, pp. 5939-5948, 2019.
Christopher B. Choy, Danfci Xu, JunYoung Gwak, Kevin Chen, and Silvio Savarese. 3D-R2N2: A unified approach for single and multi-view 3D object reconstruction. In ECCV, pp. 628-644, 2016. 3
Wen-Hsuan Chu, Lei Ke, and Katerina Fragkiadaki. DreamScene4D: Dynamic multi-object scene generation from monocular videos. arXiv preprint arXiv:2405.02288 2024. 2,
Angela Dai, Angel X. Chang, Manolis Savva, Macicj Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotall 3D reconstructions of indoor scenes. In CVPR, pp. 5828- 2019, 8
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021. 4
Jakob Engel, Thomas Schöps, and Daniel Cremers. LSD-SLAM: Large-scale direct monocular SLAM. In ECCV, pp. 834-849, 2014. 3
Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct sparse odometry. IEEE TPAMI, 40(3): 611 611-625, 2017.
Martin A Fischler and Robert C. Bolles. Random sample consensus: paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24 (6):381-3955 1981.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. IJRR, 32(11):1231-1237, 2013.
Ross Girshick. Fast R-CNN. In ICCV, pp. 1440-1448, 2015.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh R-CNN. In ICCV, pp. 9785-9795, 2019. 3
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self- supervised monocular depth estimation. In ICCV, pp. 2019.
11

---

Published as a conference paper at ICLR 2025
Ariel Gordon, Hanhan Li, Rico Jonschkowski, and Anelia Angelova. Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras. In ICCV, pp. 1977-8986, 2019.
Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Liu, and Mohammed Bennamoun. Deep learning for 3D point clouds: A survey. IEEE TPAMI, (5112):438843666 2020. 3
Richard Hartley and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge
university press, 2003. 5
Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. DepthCrafter: Generating consistent long depth sequences for open-world videos. arXiv preprint arXiv:2409.02095, 2024. 7,
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In CVPR, pp. 9492-9502, 2024. 7.7
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM TOG, 42(4): 2023. 4
Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In CVPR, pp. 161 1-121,,,, 2021. 3,
Suryansh Kumar, Yuchao Dai, and Hongdong Li. Monocular dense 3D reconstruction of a complex dynamic scene from two perspective frames. ICCV, pp. 4649 4657, 2017.
Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Danilidiis. MoSca: Dynamic gaussian fusion from casual videos via 4D motion scaffolds. arXiv preprint arXiv:240..17421, 2024. 2,
Vincent Lepetit, Francese Moreno-Noguer, and Pascal Fua. EPnP: An accurate O(n) solution the PnP problem. IJCV, 81:155-166 2009.
Chen-Hsuan Lin, Chen Kong, and Simon Lucey. Learning efficient point cloud generation for dense 3D object reconstruction. In AAAI, 2018.
Qingming Liu, Yuan Liu, Jiepeng Wang, Xianqiang Lv, Peng Wang, Wenping Wang, and Junhui Hou. MoDGS: Dynamic gaussian splatting from causually-capturel monocular videos. arXiv preprint arXiv:2406.004. 2024. 2,
Jonathon Luiten, Tobias Fischer, and Bastian Leibe. Track reconstruct and reconstruct to track. IEEE Robotics and Automation Letters, 5(2): 1803:100-18 2020. 2
Bin Luo and Edwin R. Hancock. Procrustes alignment with the EM algorithm. In International Conference on Computer Analysis of Images and Patterns, pp. 623-631, 1999.
Xuan Luo, Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation. ACM Transactions on Graphics (ToG), 39(4):1111, 2020. 4
Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego- motion from monocular video using 3D geometric constraints. In CVPR, pp. 5667-5675, 2018.
Lukas Mehl, Jenny Schmalfuss, Azin Jahedi, Yaroslava Nalivayko, and Andrés Bruhn. Spring: A high-resolution high-detail dataset and benchmark for scene flow, optical flow and stereo. In CVPR, pp. 4981-4991 2023. 4,
Raul Mur-Artal and Juan D. Tardós. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics. 33(5): 2017,
Raul ArtaArtal, Jose Maria Martinez Montiel, and Juan D Tardos. ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Transactions Robotics, (1(5):1147-116 2015.
12

---

Published as a conference paper at ICLR 2025
Armin Mustafa, Hansung Kim, Jean- Yves Guillemaut, and Adrian Hilton. Temporally coherent 4D reconstruction of complex dynamic scenes. In CVPR. pp. 4660 46690 2016.
Richard A Newcombe, Steven J. Lovegrove, and Andrew J. Davison. DTAM: Dense tracking and mapping in real-time. In ICCV, pp.
2011.
Emanuele Palazzolo, Jens Behley, Philipp Lottes, Philippe Giguere, and Cyrill Stachniss. Refusion: 3d reconstruction in dynamic environments for RGB-D cameras exploiting residuals. In IROS, pp. 7855-7865 2019. 8
Songyou Peng, Michael Niemeyer, Lars Mescheder, Mare Pollefeys, and Andreas Geiger. Convolu- tional occupancy networks. In ECCV, pp. 523-540, 2020. 3
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine-Hornung A benchmark dataset and evaluation methodology for video object segmenta- tion. In CVPR, pp. 724-732, 2016.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE TPAMI,
44(3):1633-1637 2020. 3
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pp. 12179-12188, 2021. 3
Saurabh Saxena, Charles Hermaan, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David J. Fleet. The surprising effectiveness of diffusion models for optical flow and monoc- ular depth estimation. NeurIPS, 2024.
Johannes Lutz Schönberger and MinMichaeel Frahm. 2016. 2, 3,5
revisited. In CVPR,
Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016. 5
Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning temporally consistent vidco depth from video diffusion priors. arXiv preprint 04f1::2406.014933, 2024. 4,7
Shihao Shen, Yilin Cai, Wenshan Wang, and Sebastian Scherer. Dytan Joint refinement of visual odometry and motion segmentation in dynamic environments. In JCRA, pp. 4048-4444 4055, 2023. 3
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and sup- port inference from RGBD images. In ECCV, pp. 746-760, 2012.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. VexpVsxels: Learning persistent 3D feature embeddings. In CVPR, pp. 2437-2446, 2019.
Jürgen Sturm, Nikolas Engelhard, Felix Endres, Wolfram Burgard, and Daniel Cremers. A bench- mark for the evaluation of RGB-D - SLAM systems. In IROS, pp. 573-580, 2012.
Libo Sun, Jia-Wang Bian, Huangying Zhan, Wei Yin, Ian Reid, and Chunhua Shen. Sc-depthv3: Robust self-supervised monocular depth estimation for dynamic scenes. IEEE TPAMI, 2023. 4
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 4.
Chengzhou Tang and Ping Tan. BA-Net: Dense bundle adjustment network. arXiv preprint
arXiv:1806.04800 08807 2018.
Zachary Teed and Jia Deng. Deep V2D: Video to depth with differentiable structure from motion. arXiv preprint arXiv: arXiv: 1842.04655 2018.
13

---

Published as a conference paper at ICLR 2025
Zachary Teed and Jia Deng. DROID-SLAM: Deep visual SLAM for monocular, stereo, and RGB-D
cameras. NewrIPS, pp. 16558- -6558,,66,9, 2021. 3,
Zachary Teed, Lahav Lipson, and Jia Deng. Deep patch visual odometry. NeurIPS, 2024.
Shubham Tulsiani, Tinghui Zhou, Alexei A. Efros, and Jitendra Malik. Multi-view supervision for hinlleview reconstruction via differentiable ray consistency. In CVPR, pp. 2626-2634, 2017. 3
Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2Mesh: Generating 3D mesh models from single RGB images. ECCV, pp. 52-67, 2018.
Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2 10689, 2021a.
Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. arXiv preprint arXiv:2407.1376. 2024a. 2
Shizun Wang, Xingyi Yang, Qiuhong Shen, Zhenxiang Jiang, and Xinchao Wang. GFlow: Recov- ering 4D world from monocular video. arXiv preprini urXiv:2405. 18426, 2024b. 2, 4
Shuzhe Wang. Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. DUSt3::
Geometric 3D vision made easy. In CVPR, pp. 20697-20709, 2024c. 2, 3, 4,
Wenshan Wang, Delong Zhu, Xiangwei Wang, Yaoyu Hu, Yuheng Qiu, Chen Wang, Yafei Hu, Ashish Kapoor, and Sebastian Scherer. TartanAir: A dataset to push the limits of visual SLAM. In IROS, pp. 4909-4916, 2020. 5.
Wenshan Wang, Yaoyu Hu, and Sebastian Scherer. TartanVO, A generalizable learning-based VO. In CoRI, pp. 1761-1772, 2021b. 3
Yihan Wang, Lahav Lipson, and Jia Deng. SEA-RAFT: Simple, efficient, accurate RAFT for optical flow. In ECCV, 2024d. 6
Yiran Wang, Min Shi, Jiaqi Li, Zihao Huang, Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. Neural video depth stabilizer. ICCV, pp. 9466-9476, 2023.
Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel Brostow, and Michael Firman. The temporal opportunist: Self-supervised multi-frame monocular depth. In CVPR, pp. 1164-1174, 2021.
Philippe Weinzaepfel, Vincent Leroy, Thomas Lucas, Romain Brégier, Yohann Cabon, Vaibhav Arora, Leonid Antsfeld, Boris Chidlovskii, Gabriela Csurka, and Jérôme Revaud. CroCo: Self- supervised pre-training for 3D vision tasks by cross-view completion. NeurIPS, pp. 3502-3516, 2022. 5
Philippe Weinzaepfel, Thomas Lucas, Vincent Leroy, Yohann Cabon, Vaibhav Arora, Romain Brégier, Gabriela Csurka, Leonid Antsfeld, Boris Chidlovskii, and Jérôme Revaud. CroCo v2: Improved cross-view completion pre-training for stereo matching and optical flow. In ICCV, pp- 17969-17980, 2023. 4
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, pp.
2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything V2. arXiv preprint XXXv:2406.094144, 2024b. 3, 8
Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, and Ram Nevatia. Every pixel counts: Unsu- pervised geometry learning with holistic 3D motion understanding. In ECCV workshops, 2018.
Wei Yin, Jianming Zhang, Oliver Wang. Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from a single image. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 204-213, 2021. 4,
14

---

Published as a conference paper at ICLR 2025
Zhoutong Zhang, Forrester Cole, Richard Tucker, William T Freeman, and Tali Dekel. Consistent depth of moving objects in video. ACM Transactions on Graphics (ToG), 40(4):1-12, 2021. 4
Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T.
Freeman. Structure and motion from casual videos. In ECCV, pp. 20-37, 2022. 8
Wang Zhao, Shaohui Liu, Hengkai Guo, Wenping Wang, and Yong-Jin Liu. ParticleSfM: Exploiting dense point trajectories for localizing moving cameras in the wild. In ECCV, pp. 523-542, 2022.
3, 4, 8, 9
Yang Zheng, Adam W. Harley, Bokui Shen, Gordon Wetzstein, and Leonidas Guibas. PointOdyssey: A large-scale synthetic dataset for long-term point tracking. In ICCV, 2023. 4,5
15

---

Published as a conference paper at ICLR 2025
A MORE QUALITATIVE RESULTS
A.1 DEPTH
For more thorough comparisons, we include additional qualitative examples of video depth, com- paring our method against DepthCrafter, a concurrent method specifically trained for video depth. We include comparisons on the Bonn dataset Fig. Al and the KITTI dataset in Fig. A2. In these comparisons, we show that after alignment, our estimates are much closer to the ground truth than those of DepthCrafter.
Input Frames
GT Depth
DepthCrafter
MonST3R (Ours)
Figure Al: Video depth estimation comparison on Bonn dataset. Evaluation protocol is per- sequence scale shift. We visualize the prediction result after alignment. Note, in the first row, our depth estimation is more aligned with the GT depth (e.g., the wall) compared to DepthCrafter's.
A.2 CAMERA POSE
We present additional qualitative results for camera pose estimation. We compare our model with the state-of-the-art visual odometry method LEAP-VO and the joint video depth and pose optimiza- tion method CasualSAM. Results are provided for the Sintel dataset in Fig. A3 and Scannet dataset in Fig. A4. In these comparisons, our method significantly outperforms the baselines for very chal- lenging cases such as "temple 3" and "cave 2" in Sintel and performs comparable to better than baselines in the rest of the results like those in the Scannet dataset.
A.3 JOINT DEPTH & CAMERA POSE RESULTS
We present additional results for joint point cloud and camera pose estimation, comparing against CasualSAM and DUSt3R. Fig. A5 shows three additional scenes for Davis: mbike-trick. train, and dog. For mbike-trick trick, CasualSAM makes large errors in both geometry and camera pose; DUSt3R produces reasonable geometry except for the last subset of the video which also results in poor
16

---

Published as a conference paper at ICLR 2025
DepthCrafter (Prediction Error)
MonST3R (Ours)
Input Frames/ GT Depth
(Prediction / 'Error)
Figure A2: Video depth estimation comparison on KITTI dataset. Evaluation protocol is per- sequence scale & shift. For each case, the upper row for imput frame and depth prediction; the lower row is for ground truth depth annotation and error map. Prediction result is after alignment.
pose estimation (highlighted in red); and ours correctly estimates both geometry and camera pose. For train, CasualSAM accurately recovers the camera pose for the video but produces suboptimal geometry, misaligning the train and the track the top right. DUSt3R both misaligns the track the top left and gives poor camera pose estimates. Our method correctly estimates both geometry and camera pose. For dog, CasualSAM produces imprecise, smeared geometry with slight inaccuracies in the camera pose. DUSt3R results in mistakes in both the geometry and camera pose due to misalignments of the frames, while our method correctly estimates both geometry and camera pose.
A.4 PAIRWISE POINTMAPS
In Fig. A6, we also include visualizations of two input frames and estimated pairwise pointmaps, direct output of the trained models, both DUSt3R and MonST3R. Note, these results do not include any optimization method retains the ability to handle changing camera intrinsics. Rows 2 and 3 demonstrate that our method handle "impossible" alignments that two frames have almost overlap, presence of motion, unlike DUSt3 that misaligns based on the foreground object. Rows and show that in addition enabling the model to handle motion, fine-tuning also has improved the model's ss ability to represent large-scale scenes, where DUSt3R predicts to be flat.
post-processing. Row demonstrates that even after fine-tuning,
in the
17

---

Published as a conference paper at ICLR 2025
Mle_3
Do. oTP
FORCH
XPH
02
-40
Creak Iuh hoddac
400
DMQ
LEAP-VO
CAsuaISAM
MonST3R (Ours)
Figure A3: Camera pose estimation comparison on the Sintel dataset. The trajectories are plotted
along the two axes with the highest variance to capture the most significant motion. The predicted trajectory (solid blue line) is aligned match the ground truth trajectory (dashed gray line). Our
MonST3R is more robust at challenging scenes, "temple 3 and "cave 2" (the last two rows).
Seene 726
144
Scone 722
Poose
Scene 717
Scene 715
LEAP-VO
CasualSAM
MonST3R (Ours)
Figure A4: Camera pose estimation comparison on the Scannet dataset. The trajectories are plotted along the two axes with the highest variance to capture the most significant motion.
18

---

Published as a conference paper at ICLR 2025
Input Video
CasualSAM
DUS13R
MonST3R (Ours)
Figure A5: Qualitative comparison on Davis. Compared to CasualSAM and DUSt3R, our method outputs both reliable camera trajectories and geometry of dynamic scenes.
19

---

Published as a conference paper at ICLR 2025
Pairwise Input
DUSI3R
MonST3R (Ours)
Figure A6: Qualitative comparison of feed-forward pairwise pointmaps prediction.
20

---

Published as a conference paper at ICLR 2025
Input Frames
CasualSAM
DUS(3R
MonST3R (Ours)
Qualitative comparison of static/dynamic mask. We visualize both the continuous error map (upper row) and binary static/dynamic mask (lower row). The threshold a is fixed.
Figure A7:
A.5 STATIC/DYNAMIC MASK
We present the result of static/dynamic mask estimation based the optical flow, discussed in Sec. 3.3. As shown Fig. A7, our MonST3R achieves overall plausible results while DUSt3R fails to provide accurate motion mask due to error in camera pose and depth estimation.
B MORE QUANTITATIVE RESULTS
B. ABLATION ON TRANING//NFERENCE WINDOW SIZE
In this section, present additional quantitative results the impact of different training and inference window sizes, as shown Tab. A1. The results demonstrate that performance generally improves the inference window size increases, up the size of the training window.
21

---

Published as a conference paper at ICLR 2025
Table Al: Ablation study on different training/inference window sizes on the Sintel dataset. Each cell displays two values: ATE / Abs Rel L. corresponding to camera pose and video depth estimation, respectively. The cells where the inference window size exceeds the training window size are highlighted in grey. The default setup is underlined, and the best results are in bold. GPU memory consumption for each inference setup is listed in the leftmost column.
Training window size
Memory (GB)
Inference video graph
Size
Size
Size 9
17.3
Window size 3
0.191 / 0.442
0.182 0.413
0.163 / 0.383
20.1
Window size 4
0.178 0.431
0.166 0.406
0.148 0.362
17.2
Window size 5 (stride 2)
0.145 0.439
0.411
0.137 / 0.367
23.5
Window size 5
0.139 / 0.406
0,132 0.399 0.399
0.133 / 0.355
19.9
Window size 7 (stride 2)
0.180 / 0.409
0. 0.140/ 0.372
0.121 / 0.359
29.5
Window size 7
0.174 / 0.389
0.136 0.351
0.113 / 0.346
23.2
Window size 9 (stride 2)
0.177 / 0.380
0.156/ 0.387
0.108 / 0.345
Moreover, proposed stride-based sampling provides better trade-off between window size and computational cost. For instance, for the training window size of 7 or 9, the inference configuration "Window size 7 (stride 2)" outperforms "Window size 4" while consuming similar amount of memory (19.9 GB vs. 20.1 GB). Additionally, for a training window size of 9, "Window size (stride 2)" achieves better performance than "Window size 7" while reducing memory consumption by 20%, highlighting the efficiency of our design.
B.2 ABLATION ON LOSS WEIGHT SENSITIVITY
Table A2: Ablation study on loss weight sensitivity. The table shows the effect of varying the loss weights Wsmooth and Wnow camera pose and video depth estimation. The default setup is underlined, and the best results are in bold.
Camera pose estimation
Video depth estimation
Wsmooth
Wilo
ATE L
RPEmnass J
RPE,tt J
Abs Rel J
1.225 1
0.01
0.001
0.118
0.045
0.716
0.335
58.2
0.01
0.005
0.109
0.042
0.715
0.336
58.2
0.01
0.01
0.108
0.042
0.732
0.335
58.5
0.01
0.05
0.115
0.044
0.831
0.329
59.5
0.01
0.1
0.118
0.049
0.838
0.330
59.3
0.001
0.01
0.110
0.048
0.869
0.332
58.4
0.005
0.01
0.109
0.044
0.844
0.334
58.3
0.01
0.01
0.108
0.042
0.732
0.335
58.5
0.05
0.01
0.127
0.045
0.779
0.342
57.9
0.1
0.01
0.138
0.049
0.799
0.346
57.3
In the main paper, our global optimization objective (see Eq. (7)) is combination of three different loss terms, controlled by two hyperparameters. Here, we evaluate the sensitivity of the results to variations in these two weights.
From the results Tab. A2, it can be seen that the weight of the optical flow loss (Wnow) does not significantly impact the overall performance. Varying the flow loss weight sometimes leads to slightly better results in other metrics. The weight of the camera trajectory smoothness constraint (Wsmooth) exhibits more noticeable effects. When it is set to a lower value, the difference in perfor- mance remains small, though the RPE performance drops noticeably. However, when the weight is set too high, performance degrades, likely due to the over-constraining of the camera trajectory.
22

---

Published as a conference paper at ICLR 2025
C FULLY FEED-FORWARD RECONSTRUCTION
The MonST3R (or DUST3R) model predicts pairwise pointmaps in the coordinate frame of the first image, which can be seen as the anchor frame. enable faster and fully feed-forwar reconstruction from monocular video input, we construct image pairs that align all the 7 frames to the same anchor frame (e.g., the first frame or the middle frame), denoted as {fanchor t- alignment ensures that the predicted point cloud of each frame shares the same camera coordinate system as the anchor frame and can be treated as global point cloud, i.e., X Xii f-t
E 1...., T). This
Figure A8: Real-time reconstruction result. By aligning all the frames to the middle frame, MonST3R enables real-time reconstruction from monocular video input. More video results at our webpage: https:: /monst -eeeer-pppeee pppper. github. ceiiio//aggge /pwgea.a..... html.
This approach significantly enhances runtime performance, achieving approximately 40 FPS on single RTX4090 GPU. Moreover, it has the potential to enable real-time reconstruction in stream- ing fashion, by adaptively updating the anchor frame. We provide qualitative examples in Fig. A8.
Currently, this method has certain limitations. The reconstruction quality is sensitive to the choice of the anchor frame, and since each frame is aligned independently to the anchor, some artifacts believe (e.g., shifting) may occur due to the lack of global information sharing. Nonetheless. this approach is a promising direction for achieving streaming, real-time, and fully feed-forward reconstruction from monocular video input.
23

---

Published as a conference paper at ICLR 2025
D DETAILS ON GLOBAL OPTIMIZATION
D.1 DETAILS ON LsMOOTH
The camera trajectory smoothness loss encourages smooth transitions between consecutive camera poses by penalizing large changes in rotation and translation. For frame t, given the rotation R and translation T the smoothness loss defined as:
Smooth RR T)
(|
[|RST" R + | -T'll2)
where I the identity matrix, norm.
||
||| denotes the Frobenius norm, and ||2 denotes the Euclidean
Since R. and T parameterize the global point cloud X along with the depth map D and camera in- trinsics K, we simplify the notation by writing the smoothness loss Lemooth (X) Lsmoath (R, T) for brevity.
D.2 DETAILS ON LLoOw
The flow projection loss ensures consistency between the camera-induced flow and the estimated optical flow for regions identified as static. It is defined as follows:
Lroo cam: S)
Σ Σ
Selobaltt
(Fglobalat -F II,
(9)
cam
WOEW (-r'EW'
p is the flow induced by camera motion from frame to frame t', and F is where cam the estimated optical flow obtained from an off-the-shelf method. The mask Salobalat>e" indicates regions that are confidently static.
The camera- induued flow Felobal;t++e is computed using the global camera parameters, intrinsics, and depth map follows:
cam
F pelobal:t--' Cam
= (D'K"R'R* K' k" (T" -T)) -x,
(10)
where is pixel coordinate matrix, & is in homogeneous coordinates, and x(-) represents the projection operation from homogeneous to image coordinates.
To derive the confident statie mask Selobal:>>1" we first initialize per-frame mask S with all the sampled pairs:
S'
Σ Sti Σ Sti -t
(11)
2|
MEN.
PEN.
where N,, = (t' 1' is sampled). This initialization averages the pair-wise static masks from all sampled pairs involving frame t. For robustness, we also update the mask with global parameters, and derive the final mask as:
Selobalic+t' [
F est
(12)
cam
where V denotes the logical "or" operator, and a is predefined threshold.
Since both F Fram and S are derived from the global point cloud X (which includes R, T. D, and K), we express the flow loss as Cnow (X) := Lnow (F cam- S) for brevity.
24


Reviews:
[{"review_id": "m72rackV87", "reviewer": "Reviewer_b6SW", "summary": "The paper first identifies the limitations in joint pose and geometry estimation methods on dynamic scenes, and proposes to finetune based on the recent geometric learning model DUSt3R, representing the dynamic scenes with per-timestamp pointmaps. During global alignment for the pairwise inference results, trajectory smoothness loss and flow projection loss are added to the alignment term to adapt the optimization on sequential dynamic inputs. Experimental results show that the proposed method achieves state-of-the-art performance on the video depth estimation task and remains competitive in camera pose estimation, while jointly estimating both.", "questions": "1. In the depth comparison in figure A1, the depth colormaps effectively hides the error, it's better to also add error colormaps. Additionally, in all the visualized frames, the model all predicts closer depth compared to ground truth on the dynamic parts. Is this specific to this dataset or is it a general bias of the model? What could be the possible reasons?\n2. In lines 261-269, to calculate induced flow from camera motions, the method needs to estimate the intrinsics separately from the pointmaps. I wonder if using dataset intrinsics would degrade the performance?\n3. Another related question on intrinsics, in lines 456-458, the relax of requiring intrinsics can be seen as a strength, however DUSt3R is not able to include known intrinsics. Is it considered to include the input intrinsics during the method development?\n4. As we see from DUSt3R to MASt3R, the pointmaps serve as general and powerful representation for static geometry, as multiple downstream task can be estimated based on them. However, it's questionable whether it fit best for dynamic scenes. Here, an off-the-shelf flow estimator is needed to help obtain the static/dynamic mask, are flow fields themselves better suited for representing dynamics?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method motivates the problem well on joint pose and geometry estimation on dynamic scenes by identifying the limitations in the current geometric learning method DUSt3R which is trained on static scenes.\n2. With carefully selected datasets for dynamic scenes, together with a window-based training strategy, the scarcity of dynamic training data for finetuning is dealt with.\n3. Finetuning results validate the proposed idea, with only the decoder and prediction head tuned, the model is capable of predicting pointmaps for the dynamic regions, showing sharping improvement compared to the DUSt3R with mask baseline.\n4. The method achieves SoTA results on video depth estimation task, and competitive results on video pose estimation. The selected baselines are thorough, including baselines specialized in each task.", "weaknesses": "1. The loss used in fine-tuning is not detailed in the methods section. In line 294, \"static mask is both a potential output and will be used in the later global pose optimization\", the word \"potential\" here is confusing, is the static mask also used during finetuning?\n2. Although a sliding window technique is used, the runtime reported is 30s for pairwise inference and 1min for global optimization given the 60 frames (around 2.5s) video input, which is slow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O6fjpDQGLN", "reviewer": "Reviewer_GUwz", "summary": "This paper extends DUSt3R, originally a feed-forward method for static scene reconstruction, to handle dynamic scenes by reconstructing scene geometry at each timestep. The initial performance of DUSt3R was poor due to its training on static datasets only. To improve results, this paper identifies and fine-tunes DUSt3R on several dynamic datasets. Consequently, the proposed method delivers competitive results in video depth estimation, camera pose estimation, and dense reconstruction. The approach is efficient and demonstrates results that rival those of specialized techniques.", "questions": "See weakness", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Clarity and Readability: The paper is well-written, with a clear statement of the motivation, methodology, and challenges, making it easy to follow.\n\nEffective Demonstration: The video demonstration is impressive, showcasing efficient 4D reconstruction with significant potential for downstream applications.\n\nSufficient Novelty: While the proposed approach builds on DUSt3R, extending it to dynamic scenes is non-trivial. The authors clearly outline the challenges and propose effective solutions to address them.\n\nConvincing Evaluation: The evaluation is thorough, comparing the method to different categories of existing techniques, with both quantitative and qualitative results presented.", "weaknesses": "1. Mention of Related Work: For video depth estimation, self-supervised monocular depth methods such as Manydepth [a] and SC-DepthV3 [b] should be mentioned. SC-DepthV3, in particular, provides video depth estimation results on the Bonn dataset, which would make for a relevant comparison.\n\n[a] The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth, CVPR 2021.\n\n[b] SC-DepthV3: Robust Self-Supervised Monocular Depth Estimation for Dynamic Scenes, TPAMI 2024.\n\n2. Robustness in Dynamic Regions: The statement “For most scenes, where most pixels are static, random samples of points will place more emphasis on the static elements” is generally true, but there are edge cases that require more robust approaches. For example, in the Bonn dataset, moving objects such as humans can occupy over 50% of the image. Potential solutions could include masking out dynamic regions using segmentation masks or re-running relative pose estimation after identifying dynamic regions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SiEsOgCR4o", "reviewer": "Reviewer_2c91", "summary": "This paper approaches the task of obtaining depth and camera pose given dynamic video. To do so, it begins from the DUSt3R architecture and pretraining. The paper argues optimizing dynamic scenes requires good estimation of foreground depth, and alignment based on dynamic objects; both of which DUSt3R struggles on due to static training data of mostly backgrounds. To this end, the paper fine-tunes on dynamic videos, as well as proposing slightly different optimization criteria. It shows state-of-the-art results on depth and camera pose estimation on several standard dynamic video benchmarks.", "questions": "See Weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Clearly identifies and addresses key shortcomings of DUSt3R when applied to dynamic scenes\n- This work identifies non-obvious weaknesses of DUSt3R in foreground depth prediction, in addition to more predictable poor dynamic correspondence performance. Experiments show clear improvement accordingly\n- Getting MonST3R to work with dynamic points is nontrivial. Careful finetuning choices in data and network, along with regularizations to handle a point representation taking into account time must have been necessary to achieve such strong performance\n- Comprehensive ablations clearly show impact of several contributions in data, training and inference. Notably, PointOdyssey (dynamic) is essential, and fine-tuning specifically the decoder and head is critical.\n\nEffective and efficient performance on an important task\n- SOTA performance on depth and camera pose estimation across standard dynamic video datasets: Sintel, TUM-Dynamics, Bonn, KITTI, NYU-v2.\n- Impressive the method is still competitive on static scenes ScanNet, showing this fine-tuning does not sacrifice original DUSt3R results.\n\nMy rating of 8 (Accept) is because of strong experiment results (SOTA Depth and Pose), resulting from clearly identified weaknesses (static and distant training data), from a strong starting point (DUSt3R); weaknesses are minor.\n\n\n***Post-discussion Update*** I leave this paper at an 8. The rebuttal well addresses weaknesses in this paper. I do not feel it is worthy of a 10 because the contributions are not exceptional, i.e. it consists mostly of fine-tuning on synthetic data for a sizeable but not massive gain; but results, writing and method are still very strong and the paper should be accepted.", "weaknesses": "(Minor) Camera Pose Estimation suite could be more comprehensive\n- E.g. COLMAP or masked COLMAP baseline would be good benchmarks.\n- Why are DROID-SLAM, DPVO and ParticleSfM not run on TUM, and DROID-SLAM and DPVO not on ScanNet? \n- LEAP-VO is highly competitive with the proposed method. This is a minor weakness given this method also estimates depth and improves on DUSt3R, but should be considered when analyzing e.g. if this paper should be considered for strong accept.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hqoHh3mEwX", "reviewer": "Reviewer_CcMU", "summary": "This paper presents MonST3R, a method for estimating geometry in dynamic scenes. It adapts the DUSt3R's pointmap representation to handle moving objects. Key points include: using suitable synthetic and real datasets for training despite data scarcity, introducing new optimization methods for downstream tasks like depth and pose estimation, demonstrating strong performance compared to prior works, and achieving good results in 4D reconstruction.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- MonST3R successfully adapts the DUSt3R's pointmap representation to dynamic scenes. By estimating a pointmap for each timestep and representing them in the same camera coordinate frame, it can handle the movement of objects.\n- Data Efficiency: Trains well with limited data via strategies.\n- Good Performance: Strong in video depth and pose estimation.", "weaknesses": "- There are also some high-quality synthetic datasets in the video field, such as DynamicStereo and IRSDataset. The author could consider expanding the scale of the training data to explore better model performance.\n- There are doubts about the metrics used by the author when comparing video depth estimation. When the delta metric is below 0.7 or lower, it is usually considered that the prediction has completely collapsed. However, as far as the reviewer knows, the video depth estimation methods compared by the author have good performance and can provide basically correct depth estimation results at least on the Sintel dataset. Some metrics, such as NVDS, seem to be inconsistent with the report in the paper.\n- It is necessary to explore the robustness of DUSt3R, that is, the failure cases when the scene is too dynamic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ceYJGrM6Gt", "reviewer": "Reviewer_VTp6", "summary": "The paper introduces MonST3R, a novel geometry-first approach for estimating 3D structure from dynamic scenes in videos. Unlike existing methods that rely on multi-stage pipelines prone to errors, MonST3R adapts the DUSt3R pointmap representation, originally designed for static scenes, to handle moving objects by estimating pointmaps at each time step. This adaptation is achieved through fine-tuning on carefully selected synthetic and real-world datasets that provide depth and camera pose annotations, despite the challenge of limited data availability. The method demonstrates significant improvements in video depth and camera pose estimation, surpassing traditional optimization-based methods in robustness and efficiency. MonST3R is capable of generating reliable 4D reconstructions in a predominantly feed-forward manner, enhancing performance in various video-specific tasks.", "questions": "See the weakness part.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed problem of reconstructing static scenes and estimating per-timestep geometry of dynamic objects is compelling and relevant.\n2. The paper presents a reasonable and well-structured solution to the defined problem.\n3. The method achieves state-of-the-art performance, demonstrating its effectiveness.", "weaknesses": "1. The complete optimization in Eq. 6 minimizes the loss to obtain optimal values for X, P_W, and \\sigma. However, it seems that the smoothness loss in Eq. 4 is minimized specifically for R and T. The input notation L_{smooth}(X) might be misleading—should it be L_{smooth}(R,T) instead?\n2. The notation for L_{flow}(X) also seems ambiguous. Would L_{flow}(F) or L_{flow}(F,S) be more appropriate to reflect the actual inputs?\n3. There is no explanation provided for the notation global. Clarifying what the global parameters are and how S^{global} is derived would be helpful. Specifically, is S^{global} in Eq. 5 an extension of Eq. 2 that uses F_{cam}^{global,t→t′} and F_{est}^{global,t→t′}\n4. The proposed network is trained on diverse datasets, including PointOdyssey, TartanAir, Spring, and Waymo Perception, but the authors rely on a pretrained optical flow model. This could introduce a domain gap between the depth network and the optical flow network. Were there any issues related to domain gaps during the experiments?\n5. It would be beneficial if the authors provided depth evaluation results for static scenes and dynamic objects separately. This would clarify whether the performance improvement is due to coherent depth estimation on static scenes or the enhanced depth quality of dynamic objects.\n6. Do the authors use zero-shot models like Marigold and Depth-Anything? The use of the Waymo dataset (similar to KITTI) and the PointOdyssey dataset (with properties akin to Sintel) might explain why the proposed method outperforms these models. Can the authors specify which components of their approach are most critical for achieving this performance boost?\n7. Overall, the technical parts is not clearly written.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)