Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenes
Authors: Yihong Sun
Bharath Hariharan
Venue: NeurIPS 2023 poster (2023)
Keywords: monocular
depth estimation
dynamical scenes
motion segmentation
self-supervised
TL;DR: We improve unsupervised monocular depth estimation for dynamical scenes by modeling 3D independent flow and motion segmentation.

Abstract: Unsupervised monocular depth estimation techniques have demonstrated encouraging results but typically assume that the scene is static. These techniques suffer when trained on dynamical scenes, where apparent object motion can equally be explained by hypothesizing the object's independent motion, or by altering its depth. This ambiguity causes depth estimators to predict erroneous depth for moving objects. To resolve this issue, we introduce Dynamo-Depth, an unifying approach that disambiguates dynamical motion by jointly learning monocular depth, 3D independent flow field, and motion segmentation from unlabeled monocular videos. Specifically, we offer our key insight that a good initial estimation of motion segmentation is sufficient for jointly learning depth and independent motion despite the fundamental underlying ambiguity. Our proposed method achieves state-of-the-art performance on monocular depth estimation on Waymo Open and nuScenes Dataset with significant improvement in the depth of moving objects. Code and additional results are available at https://dynamo-depth.github.io.

Dynamo-Depth: Fixing Unsupervised Depth Estimation for Dynamical Scenes
Yihong Sun Corelll University yihongêcs. sohooglcs.cornell.... cornell. edu
Bharath Hariharan Cornell University cow.eeel...cooeeeeeeeecccc..
Abstract
Unsupervised monocular depth estimation techniques have demonstrated encour- aging results but typically assume that the scene is static. These techniques suffer when trained dynamical scenes, where apparent object motion can equally be ex- plained by hypothesizing the object's independent motion, or by altering its depth. This ambiguity causes depth estimators to predict erroneous depth for moving objects. To resolve this issue, we introduce Dynamo-Depth, an unifying approach that disambiguates dynamical motion by jointly learning monocular depth, 3D independent flow field, and motion segmentation from unlabeled monocular videos. Specifically, we offer our key insight that a good initial estimation of motion seg- mentation is sufficient for jointly learning depth and independent motion despite the fundamental underlying ambiguity. Our proposed method achieves state-of-the- thetth performance on monocular depth estimation on Waymo Open [34] and nuScenes [3] Dataset with significant improvement in the depth moving objects. Code and additional results are available at https: htttp:://dyaamm/e ddynamo-depth gi thub.io.
Introduction
Embodied agents acting in the real world need to perceive both the 3D scene around them well as how objects around them might behave. For instance, a self-driving car executing a lane change will need to know where the nearby cars are and how fast they are moving. Instead of relying expensive sensors such as LiDAR to estimate this structure, a promising alternative is to use commodity cameras. This has motivated long line of work monocular depth estimation using neural networks.
While neural networks can learn to estimate depth, the dominant training approach requires expensive 3D sensors in the training phase. This limits the amount of training data we can capture, resulting in downstream generalization challenges. One would prefer to train these depth estimators without supervision, for e.g., using unlabeled videos captured from a driving car. This is possible to do if the videos are produced by camera moving in static scene, since the apparent motion (i.c., optical flow) of each pixel is then inversely proportional to the depth of the pixel. This provides useful supervisory signal for learning depth estimation and has been used effectively the past [48, 36).
However, for unsupervised approaches to learn depth, dynamic scenes with moving objects pose challenge. Here, the apparent pixels motion on the image plane combined effect of camera ego-motion (or rigid scene motion) and the independent motion of objects in the scene. Methods that ignore the presence of moving objects and treat the scene as static will then learn erroneous depth, where the depths of moving objects are altered to match their observed motion, shown in Figure (b). Past works have tried to overcome this issue by having another module predict independent object motion [31, 24, 15], but the presence of multiple equivalent solutions makes training difficult: even with sophisticated regularizers, the training can converge to the degenerate solution of either predicting the whole scene as static with incorrect depth, predicting the entire scene moving flat canvas. This results in depth error on moving objects at least times as large as the error on static
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Input
Static Scene Constraint
Dynamo-Depth (Ours)
Input Fremes
Depth
(c) Rigid
Monocular Depth
Rigid Independent Flow
Figure Visualization of erroneous cases of infinite object depth and floating objects that arise from the static scene constraint. The motion of the black SUVs and white van are explained via (b) incorrect depth predictions to generate (c) the correct rigid flow for reconstruction under static scene constraint. In contrast, Dynamo-Dephr predicts (d) correct monocular depth by explicitly disentangling (f) the 3D independent flow field from (c) the camera ego-motion induced rigid flow.
background even for saate-of-the-de methods. Since navigating safely around moving objects is primary challenge in self-driving, poor depth estimation of moving objects is particularly problematic.
In this paper, we propose a new architecture and training formulation for this unsupervised learning problem that substantially improves depth accuracy for moving objects. Our key insight is that need a good initial estimate for which pixels are independently moving (a "motion mask"). With this motion mask hand, one easily train depth estimation using the static pixels, and account for independent motion to correctly estimate the depth of the moving objects. However, the challenge obtaining this motion mask in the first place: how do identify which pixels independently moving? First, we introduce two separate network modules that predict complete 3D scene flow and camera ego-motion. The difference between these then be used identify the moving objects. Second, we introduce a novel initialization scheme that stops depth training early to prevent the depth estimator from learning erroneous depth for the moving objects. This allows the motion estimation networks to learn the correct scene flow. In sum, we make the following contributions:
1. We propose new approach, Dynamo-Depth. for learning depth estimation for dynamic scenes solely from unlabeled videos. In addition to monocular depth, Dynamo-Depth also predicts camera ego-motion, 3D independent flow, and motion segmentation.
2. We propose between the rigid flow field induced by camera ego-motion and complete scene flow field to facilitate learning and regularization.
architecture that identifies independent object motion via the difference
3. We propose a novel motion initialization technique that learns an early estimation of motion segmentation to explicitly resolve the ambiguity and disentangle independent motion from camera ego-motion (Figure 1). This allows the entire architecture to learn end-to-end without any additional supervision from auxiliary model predictions or ground-tree -tuth labels.
4. We evaluate on both nuScenes [3] and Waymo Open [34] and demonstrate not only state-of- t-th-- art performance in terms of overall depth accuracy and error, but also show large improvements on moving objects by up to 62% relative gain in accuracy and 68% relative reduction error, while achieving up to 71.8% in F1 score in motion segmentation without any supervision.
2 Related Work
2.1 Unsupervised Monocular Depth Estimation
The framework for jointly learning monocular depth and ego-motion from unlabeled consecutive frames was first explored in [48, 36), where the objective was posed as novel-view synthesis problem. Here, the monocular depth estimated from the target frame is coupled with the predicted ego-motion

---

to warp the reference frame to be photometrically consistent with the target frame. Following these works, 3D geometric consistency [28] and depth prediction consistency [1] were imposed across consecutive frames to further constrain the unsupervised learning objectíve. Furthermore, discrete volumes [19], uncertainty estimation [30], optical flow based pose estimation [47], minimal projection
loss with auto-masking [10], and multiple input frames during inference [40, 25] were also proposed
to improve depth predictions and mitigate influence of occlusion and relative stationary pixels.
Unsupervised Scene Flow. In addition to predicting depth from a single frame, a closely related task of scene flow estimation obtaining both the 3D structure and 3D motion from two temporally consecutive images can also be learned by leveraging the same novel-view reconstruction objective. When trained on sequences of vecro--rames, scene flow can be estimated from both stereo [18, 41] and monocular [16, 17] inputs during inference. Additionally, DRAFT [13] learned from monocular sequences only by utilizing synthetic data and multiple consecutive frames during inference.
2.2 Mitigating Static Scene Constraint
In essence, the reconstruction objective encourages pixels to be predicted at an appropriate distance away to explain its apparent motion on the image plane, when coupled with the predicted ego-motion This assumes that all pixel motion can be sufficiently modeled by depth and camera ego-motion alone, which implies underlying static scene. However, since dynamical objects are common in the wild, such static scene assumption is often violated. To disambiguate independent motion, recent works were proposed leverage stereo-view information [38, 26, 11], additional input modalities [45, 8). synthetic data [13], supervised segmentation task [21]. and auxiliary monocular depth network [33]. Extending from these methods, additional works proposed to mitigate the negative effects by modeling the behavior of dynamical objects without the use of additional modalities and labels.
Modeling Independent Motion via Optical Flow. Although an explainability mask [48] can be used to ignore the independently moving regions, recent works exploited the jointly learned optical flow to refine the depth prediction under non-rigid scene settings. This includes predicting residual optical flow [43, 35] to account for dynamical objects and enforcing forward-backe backward flow consistency [49] to discount dynamical regions. EPC++ [27] directly modeled the dynamical objects by integrating its depth, ego-motion, and optical flow predictions. Additionally, CC [3 proposed a coordinated learning framework for joint depth and optical flow estimation assisted by a segmentation branch.
Modeling Independent Motion via 3D Non- Non-Rigid Flow Field. Instead of modeling dynamical objects via 2D optical flow, additional works proposed to explicitly model their 3D independent flow field to encourage accurate depth learning for dynamical objects. To facilitate independent motion learning, various methods [4, 12, 22, 23, 2, 32] leveraged semantic priors from off-the- shelf pretrained detection or segmentation models estimate regions that are "possibly moving." Additionally, Dyna- Mena-eethFFrmerr [46] used multi-frame information to compute the depth map, while iteratively refining the residual 3D flow field via a jointly trained motion network. Finally, Li el al. [24] proposed to jointly learn monocular depth, ego-motion, and residual flow field via sparsity loss and RM-Depth [15] improved upon it via an awtler--aware regularization loss. Unlike [24, 15] where the residual/indeeendent flow is predicted directly, we propose to model the 3D independent flow field via the complete scene flow, which facilitates training and effectively enforces sparsity.
Motivation and background
When a camera moves in rigid scene, the apparent motion of image pixels is entirely explained by (a) their 3D position relative to the camera, and (b) the motion of the camera. This fact is used by unsupervised techniques that learn to predict depth from unlabeled videos captured moves through the scene (as in driving footage). Concretely, one network predicts depth from a target frame and another predicts the motion of the camera from the source frame to the target frame. The two predictions are then used to reconstruct a target frame from the source frame, and a reconstruction loss would serve as the primary training objective.
camera
What happens if there are moving objects in the scene? Clearly, depth and camera-motion are not sufficient to explain the apparent image motion of these objects, resulting in high reconstruction loss for these dynamical regions. principle, the motion of these objects are unpredictable from a single frame, and even though the network sees high loss for these objects, the best it can do is to
3

---

fit the average case of a static object. As such, in principle, moving objects should not be concern for learning- bauning-basedd based techniques. However, practical driving scenes, there is large class moving objects for which the network can in fact minimize the reconstruction loss by predicting an incorrect depth estimate. We describe this class below.
Epipolar ambiguity in driving scenes:
Consider the simple case of pinhole camera moving forward along the 7 axis (i.e., along its viewing direction). This is in fact the common case of a car driving down the road. In this case, the image location (z, y) and corresponding optical flow (i, 3) for 3D point (X, Y, 2) in the world would be:
X Z
Y Z
-X
7 Z I
-Y% 7 1
x=
2 =
2
z
Z
(I)
Consider what happens when this 3D point is on an object moving collinear to the camera (e.g., cars in front of the ego-car, or oncoming traffic). In this case, there are two contributions to 2: the camera motion and the independent motion of the object. Thus, given ego-motion, one can produce the same optical flow by adding the appropriate amount of independent motion to Z, changing the depth Z. For example, an object can appear to move faster if it either closer and static (i.e., Z is smaller) or it is farther and moving towards the camera (i.e., 2 is larger). This is a special case of epipolar ambiguity [42]. This implies that it is possible model independent motion under the static scene constraint using an erroneous depth estimation. Unfortunately, there are statistical regularities that enable such erroneous learning.
Statistical regularities:
In self-driving scenes, there often enough information in object appearance to predict its motion even from single frame. The high capacity monocular depth network can learn recover this information, and then alter its depth prediction to better reconstruct the target frame. For instance, in Figure (b) the depth network may have learnt to recognize the backs of cars on the same nearby lanes. In the training data, cars seen from the back often travel in the same direction with similar speed the ego car, and thus have a very small optical flow in consecutive frames. The depth network therefore predicts that the car should be very far away; thus correctly predicting its optical flow, but incorrectly predicting its depth. similar effect happens when the white van recognized in its frontal view on the opposite lane, and the depth network predicts it to be closer to the ego-car. This allows the depth network to model the statistically- moving towards the camera. Furthermore, the speed of the object can also be roughly estimated from object identity (pedestrian < cyclist < car) and background context (residential city < highway). Therefore, by recognizing the statistical regularities that give away object motion, the depth model can model the behavior of dynamical objects by predicting incorrect depth values to minimize the reconstruction objective.
higher optical flow as a result of the object
This suggests that simply training a depth network on dynamic scenes is not good solution, and an alternative framework is needed. However, we do note that learning these statistical regularities about object motion is much harder than learning the depth of the static background, since the background is a lot more consistent in its optical flow across videos and frames. Thus, we observe that this kind of overfitting to dynamical objects only happens later in training; fact we use below.
4 Method
Following past works, we formulate the learning objective as a novel-view synthesis where the target frame reconstructed from the source frame. As shown in Section 3, modeling the rigid flow induced by camera motion alone is not sufficient for modeling the behavior of dynamical objects and causes performance degradation for moving objects. We address this by learning separate 3D flow field that captures the independent motion of dynamical objects.
4.1 Architecture
The proposed architecture (Figure 2) contains two components that collaborate together to explain the apparent pixels motion the image plane. Section 1.1.1 details the rigid motion networks that model

---

Depth
dt
FR
lt
i
T >
Complete
Fc
=
Mask
M
Fi FR
I
Figure 2: Overview of the proposed Dynamo-Depth. The target frame I reconstructed from source frame I by composing the 3D rigid flow FR (predicted via Depth D & Pose P) and 3D independent flow F1 (predicted via Complete Flow C Motion Mask M).
the 3D rigid flow field (motion induced by camera while assuming static scene), which includes the depth network 2D and ego-motion pose network P. Section 4.1.2 details the independent motion networks that model the 3D independent flow field (motion of dynamical objects), which includes the complete flow network C and motion mask network M.
4.1.1 Rigid Motion Networks: Depth D & Pose
Given a source frame Ia and a target frame It, we compute the 3D rigid flow induced by camera motion, (R(P:), for each target pixel Pı as follows. We first obtain the monocular depth map de by passing I to the depth network D (Eq. 2). We pass I, and I, to the ego-motion pose network P to obtain the relative camera pose Tı- (rotation and translation) (Eq. 3). We then the depth and inverse camera intrinsics K-1 to back-project every target pixel pt to compute its 3D location P,, and transform Pt into its corresponding 3D location, P.. in the source frame via T (Eq. 4). The "rigid flow" induced by camera motion is then simply (R(P:) = P. P..
D #xx33 "W
D(L,) T P P 1o) FR(P:) P. P. R
(2)
P Hxwx3 x lIx Xxxxxx 3x4
(3)
P. - (p.) Pt. P. TT
(4)
4.1.2 Independent Motion Networks: Complete Flow C & Motion Mask M
In addition the rigid flow FR. also wish to model object motion via the 3D independent flow, denoted Fj. With independent motion explicitly modeled by F,, reformulate Pb as P. (F. FR(P:) + Pt) integrate the contribution of the independent flow vector F1 (pc). However, directly estimating F₁ via network turns out to be difficult for two reasons:
(a) Learning. During training, the network will need to fill in the missing independent motion vector that is appropriate for the current separately predicted monocular depth and ego-motion for a correct reconstruction: "moving target" during training. Furthermore, it is intuitively much more difficult to learn and predict independent motion directly when the apparent motion the input frames consists of both rigid and independent motion entangled together, especially in the ambiguous scenarios illustrated in Section 3.
(b) Regularization. Adding to this complexity, (p:) should be non-zero if and only if the pixel (p()) independently moving. One could regularize this network encourage sparsity, but early on when depth and ego-motion predictions are noisy, high sparsity term would encourage this
Fc. FR, F1 and F1 FR are 3D flow fields visualized as optical flow on image plane. 2 denotes the conversion from Cartesian homogeneous coordinate.
5

---

network to turn off entirely. At the other end of the spectrum, weak sparsity regularization allows independent flow Fy to be active on static regions of the scene, which would "explain away" the rigid motion FR(P:) when computing Pa and corrupt the learning signal for depth estimation. Also, conventional sparsity regularization Fr (e.g.. L1/2 sparsity loss (24]) would dampen the magnitude of predicted motion, which reduces Fy's s capacity to model fast or far-away moving objects.
To address these issues, we propose to decompose the prediction of independent flow Fi in two: (1) predicting whether pixel (p:) independently moving and (2) predicting how is moving. For (1), we model the probability of independent motion as a binary motion segmentation task where motion mask network M takes the consecutive frames I and I, and predict a per-pixel motion mask M as shown in Eq. 5. For (2), instead of directly predicting the independent motion, we predict the complete scene flow (p())) via complete flow network C shown in Eq. 6.
M: H3xxxx3 X I HxxW33 + IR Hx
M (a)
(5)
C: C:RHw X R Hxxx33 x3 - R HxWx3
Fc Cc
(6)
From these predictions, we formulate the independent flow field as F1 = M - Fn), i.e., the residual flow between Fc and FR gated by the motion mask M. Intuitively, computing P. = (Fr(p:) + FR(Pt) + Pr) can be thought of as combining the contribution of the complete flow (p( and rigid flow (P() as a weighted average via (pr) (Eq. 8).
Ff M (Fo FR)
P. = (p.) + FR(P() PP = M Fc(p:) + (1 - (p:) FR(P:) + Pt
(8)
This formulation directly resolves the two aforementioned issues. For (a), C has simpler learning objective of modeling the full scene flow with per-pixel predictions. i.c. P. Fp.) + P.. For (b), the sparsity regularization can be applied to M alone without any impact to the learning of C and its ability to learn the complete scene flow.
Finally, we compute the reconstruction fx by sampling I, at the corresponding pixel coordinate P. via p. = KP, for every target pixel Pt.
4.2 Loss Function
We present the overall loss function in Eq. 9. The photometric loss Lrecon in Eq. 10 serves as the main learning objective, which evaluates the reconstruction i, via SSIM [39] and L1 weighted by
L = Lyecon + Lg + Tohe + .m Lm + 7gLg
(9)
Lipeoun
a 5 - (1-
SL, (.) + (1 4 fell:
(10)
The smoothness loss Lgg in Eq. 11 regularizes the smoothness of the predicted depth map de, complete flow Fc. and motion mask M. The edge-aware smoothness loss 1,(2, I) = |82= exp((16__1)) |8,2
is weaker around pixels with high color variation. We also use the mean- normalized inverse depth d' to discourage shrinking of the estimated depth [37].
Lg = reals (rsals(d)) (d, It) + "scla(Fc.) It) + 7 It)
(11)
The motion consistency loss Le in Eq. 12 computes the flow discrepancy FD(p) between the complete scene flow (p(p) and rigid flow (P( for static pixels. The probability of pixel being static is approximated by (p)).
La HW Ea- (1-M(p))- FD(p), Fb(p) = [( FR(P)lI
(12)
The motion sparsity loss J.m in Eq. 13 considers all pixels with flow discrepancy FD lower than the mean value over the frame as putative background, and encourages the motion mask to be 0 by using the cross entropy against label O. which is denoted as g(-).
Lm = ((( : Vp s.t. FD( < mean mean(Fp)))
(13)
Finally, the thove-ground loss L.g in Eq. 14 penalizes projected 3D points below the estimated ground plane (via RANSAC) where de denotes the mean-normallizd inverse depth of the ground plane.
1
Lg HW
Σ (eLU(dd(p)) d)(p))
(14)

---

4.3 Motion Initialization
Since the reconstruction loss is the main learning signal in the unsupervised approach, the gradient from the reconstruction loss between It and I at pixel Ps. would propagate through the computed sample coordinate p.g KP, (P)) + FR(P:) + Pi). Here, although it may be ideal jointly train all models end-to-end from start to finish, in practice, updating the rigid flow vector FR(P:) )))) and the independent flow vector Fr (pt) jointly from the start would cause the learning problem to be ill-posed, since each component has the capacity to overfit and explain away the other's prediction. Due to the ambiguity between camera motion and object motion as discussed in Section 3, the image reconstruction error alone is not a strong enough supervisory signal to enforce the accurate prediction of all underlying sub-tasks jointly, as family of solutions of depth and object motion would all satisfy the training objective, ranging from predicting the whole scene as static with incorrect depth to predicting the scene as moving objects on a flat canvas.
Here, we offer our key insight that a good initial estimate of motion segmentation M would allow the system to properly converge without arising degenerate solutions. As shown in Eq. 8, P. = (Pt). (P(P:) + (1 (p:)) FP(P:) + P.. For any pixel pt, P. (p:):) + P. if M (p:) is near 1, and approximates FR(P,) - P, if (p.) is near O. From this, is clear that a good motion mask would route the back- pack-propggating gradient to the complete flow network C via Fo if ps is moving and route the gradient to the rigid motion networks D and P via FR(De) if ps. is static.
Of course, the question is how one can initialize good motion mask, since it requires identifying the moving objects. Ideally, moving objects would be identified as the pixels that are poorly reconstructed based on depth and ego-motion alone, but discussed in Section 3, the depth network has the capacity to alter its depth predictions and reconstruct even the moving objects. Nevertheless, we observe that overfitting the moving objects in this way is diffieult, and only happens in later iterations. In earlier training iterations, reconstruction errors for dynamical pixels are ignored (Appendix B.5).
To ensure good motion mask M, we first train a depth network assuming a static scene, but stop updates to the depth network at an early stage. We then initialize the independent motion networks C and M from a coarse depth sketch predicted by this frozen early-stage depth network. Thus, without access to any labels or auxiliary pretrained networks, we obtain an initialization for motion mask M that helps in disambiguating and accurately estimating the rigid flow FR and independent flow F7.
5 Experiments
Dataset. While Dynamo-Deyy Depth is proposed for general applications where camera motion is infor- mative for scene geometry, we focus attention on the challenging self-driving setting, where the epipolar ambiguity and the statistical regularities mentioned in Section that lead to erroneous depth predictions are highly prevalent. Specifically, we evaluate on three datasets- Waymo Open [34], nuScenes [3], and KITTI [9] with Eigen split [6].
On all datasets, we only use the unlabeled video frames for training; ground truth LiDAR depth is only used for evaluation. We evaluate both overall depth estimation accuracy, as well as accuracy computed separately for static scene and moving objects. For the latter evaluation, we use Waymo Open and nuScenes and identify moving objects by rectifying the panoptic labels with 3D box labels to obtain masks for static/moving objects. It is worth quantifying the number of moveable objects per frame in each dataset. Waymo Open has a mean of 12.12 with a median of 9. In nuScenes, the mean is 7.78 with median of 7. In KITTI, due to lack of per-frame labels, we approximate using its curated object detection dataset, which only has a mean of 5.26 with a median of 4. Additional dataset information and distribution histograms are found in Appendix C.
Model and Training setup. The proposed method trained on four NVIDIA 2080 Ti with a total batch size of 12 and an epoch size of 8000 sampled batches. Adam optimizer [20] is used with an initial learning rate of 5e-5 and drops to 2.5e-5 after 10 epochs. Motion Initialization lasts 5 epochs and takes place after the depth network and complete flow network have been trained for 1 epoch each. After initialization, the system is trained for 20 epochs, totalling approximately 20 hours. The hyperparameter values are the same for all experiments and are provided in Appendix 1. along with the rest of the details of the model architecture. To demonstrate the generality of the proposed framework, we adopt two architectures for the depth network D, one with ResNet18 [14] backbone
7

---

Table 1: Depth evaluation on the KITTI (K), nuScenes (N), and Waymo Open (W) Dataset, IM stands for independent motion where X denotes lack of independent motion modeling. Sem indicates the amount of semantic information given during training, where 'm' indicates mask-level supervision and 'b' indicates box-level supervision. Manual replication with released code is indicated by 1.
Error metric (L)
Accuracy metric (t)
IM Sem
Abs Rel
Sq Rel
RMSE
RMSE log
< 1.25
8 < 1.25
ó 1.25*
Monodepth2 [10]
X
0.115
0.882
0.190
0.879
0,961
0.982
LiteMono [44]]
x
0.101
0.729
4.454
0.178
0.897
0.983
Struct2Depth [4]
m 0.141
1,0226
5.290
0.2115
0.816
0,945
0.979
Dyna-DM [32]
m 0.115
0.785
4.698
0.192
0.871
0,959
0.982
Lee et al. [23]
b K 0.114
0.876
4.715
0.191
0.872
0,955
0.981
SGDepth [21]
m K
0.113
0.835
4.693
0.191
0.879
0,961
0,981
Boulahbal et al. [2]
m K
0,110
0.719
4.486
0.184
0.878
0.964
0.984
GeoNet [4:3]
K 0.155
1.296
5.857
0.233
0.793
0.931
0,973
EPC++ |27]
K 0.141
1,029
5,350
0.211
0.816
0.941
0.976
CC|31]
K 0.140
1,070
5.326
0.217
0.826
0.941
0.975
TrianFlow [47]
K 0.113
0.704
4,581
0.184
0.871
0.961
0,984
ei al. [24]
K
0.130
0.950
5.138
0.209
0.843
0.948
0,978
RM-Depth [15]
K
0.107
0.687
4.476
0.181
0.883
0.964
0,984
Dynamo- (MD2)
K 0.120
0.864
4.850
0.195
0.858
0.956
0,982
Dyname-Depth
K
0.112
0.758
4.505
0.183
0.873
0.959
0,984
Monudepthh2 [10]
x
N
0.425
16.592
10.040
0.402
0.723
0.837
0,887
LiteMono [44]
x
N
0.419
15.578
9.807
0.449
0.720
0.831
0,879
Dynamo-Depee Depth (MD2)
N
0.193
2.285
7.357
0.287
0.765
0.885
0,935
Dynamo-Deve Depth
0.179 N
2,118
7.050
0.271
0.787
0.896
0.940
Monodepth22 [10]
X
W
0.173
2.731
7.708
0).227
0.797
0.930
0.968
I LiteMona [44]
X
W
0.158
2.305
7.394
0.210
0.816
0.944
0.976
Struct2Depth [4]
W
0.180
1.782
8.583
0.244
ei al. [24]
W 0.157
1,531
7.090
0.205
Lee al. [23]
b
W 0.148
1.686
7.420
0.210
Li al. [24]
W 0.162
1.711
7.833
0.223
Dynamo-Depth (MD2)
W 0.130
1.439
6.646
0.183
0.851
0.959
0.985
Dynamo-Deple
W 0.116
1.156
6.000
0.166
0.878
0.969
0.989
from Monodepth2 [10] and another with CNN/Transformer hybrid backbone from LiteMono [44], denoted as Dynamo-Depth (MD2) and Dynamo-Deper respectively. To be commensurate with baselines, the encoders initialized with ImageNet [5] pretrained weights.
Metrics. Depth performance reported via commonly used metrics proposed in [7], including error metrics (A (Abs Rel, Sq Rel, RMSE, and RMSE log) and 3 accuracy metrics (8 < 1.25, 5 < 1.252. and 1.253). We also report precision-recat curve for evaluating binary motion segmentation.
5.1 Monocular Depth Estimation
As shown in Table 1, our proposed approach outperforms prior arts on both nuScenes and Waymo Open across all metrics, with over 57% and 21% reduction in overall Abs Rel for nuScenes and Waymo Open, respectively. This improvement is significant: we repeat the experiment for Dynamo- Depth on Waymo Open times and obtained 95% confidence interval of 0. 1.9 + 0.003 for Abs Rel and 0.874 + 0.004 for 8 < 1.25. This suggests the challenge and importance of estimating accurate depth for moving objects in realistic scenes. On KITTI, our approach is competitive with prior art. However, KITTI has fewer moveable objects, which better conforms to the static scene constraint and diminishes the importance modeling independent motion. For instance, LiteMono [44], which does not model independent motion, demonstrates superior performance on KITTI but drastically underperforms in nuScenes and Waymo Open, with both datasets having many more moving objects.
To further evaluate the effectiveness of our approach in modeling independent motion, we split nuScenes and Waymo Open into static background, static objects and moving objects and evaluate depth estimation performance and LiteMono [44], Table 2, we compare our approach against the respective baseline with the same depth architecture. For simplicity, report Abs Rel and 8 < 1.25. Notably, by explicitly modeling
each partition. As we adopt our depth model from Monodepth2 [10]

---

Table 2: Depth evaluation with semantic split on the nuScenes (N) and Waymo Open (W) Dataset. S.B., S.O. and M.O. denotes the partition of pixels that are statie background, static moveable object and moving object, respectively. Manual replication with released code is indicated by
Abs Rel (4)
8 < 1.25 (t)
All
S.B.
S.O.
M.O.
All
S.B.
S.O.
M.O.
Monodepth2 [10]
N
0.425
0,447
0.232
0.418
0.723
0.735
0.704
0.570
Dynamo-Depth (MD2)
N
0.193
0.196
0.196
0.228
0.765
0.761
0.759
0.684
A (%)
-54.6
-56.2
-15.5
-45.5
+5.8
+3.5
+7.8
+20.0
LiteMono [44]
N
0.419
0.431
0.248
0.502
0.720
0.734
0.718
0.517
Dynamo-Depp
-57.3 N
0.179
0.184
0.182
0.198
0.787
0.781
0.774
0.753
A (%)
-57.3
-26.6
-60.6
+9.3
+6.4
+7.8
+45.6
Monodepth2 [10]
W
0.173
0.152
0.215
0.749
0.797
0.810
0.683
0,416
Dynamo-Depth (MD2)
W
0.130
0.122
0.175
0.234
0.851
0.862
0.778
0.674
A (%)
-24.9
-19.7
-18.6
-68.8
+6.8
+6.4
+13.9
+62.0
Lite [44]
0.158
0.140
0.179
0.599
0.816
0.827
0.733
0.506
Dynamo-Depth
W
0.116
0.110
0.155
0.194
0.878
0.891
0.812
0.750
A (%)
-26.6
-21.4
-13,4
-67.6
+7.6
+7.7
+10.8
+48.2
Waymo Open Dataset
nuScenes
KITTI
(a) Input Frames
(b) Monocular Depth
(c) Rigid Flow
(d) Independent Flow (e) Motion Mask
Figure 3: Qualitative results of our proposed approach that learns from (a) unlabeled video sequences
and predicts (b) monocular depth, (c) rigid flow, (d) independent flow, and (e) motion segmentation.
independent motion, observe consistent and significant improvement moving objects (M.O.) across both architectures and both datasets, with over 48% relative improvement in accuracy and 67% relative reduction in error for both architectures on Waymo Open. Additionally, a substantial improvement on both static moveable objects (S.O.) and static background (S.B.) for both datasets and both architectures. Interestingly. the large reduction error on static background in nuScenes does not match the corresponding improvement in accuracy, as the large errors on background mainly come from night-time instances where all methods fail, but differ in their respective artifacts (discussed further in Appendix D.2).
see
All motion segmentation in (e) are uniformly visualized with range of [0, 1J, and corresponding video sequences that contain each example found in https: //yyn//ynooo///////// depth cet.giittuu github.55 io.

---

Table 3: Depth ablation results on the Waymo Open Dataset with Dynamo-Depth.
Motion
ImageNet Pretraining
Abs Rel (L)
< 1.25 (t)
Le
I
Initialization
All
S.B.
S.O.
M.O.
All
S.B.
S.O.
M.O.
x
0.525
0.489
0.625
0.447
0.254
0.274
0.210
0.325
0.522
0.487
0.621
0.444
0.256
0.275
0.212
0.326
0.266
0.237
0.167
0.211
0.816
0.844
0.792
0.743
X
0.136
0.132
0.177
0.201
0.833
0.839
0.778
0.730
0.116
0.110
0.155
0.194
0.878
0.891
0.812
0.750
In sum, by explicitly modeling independent motion, we are able to outperform the respective baselines and achieve state-of-the-arr performances on both Waymo Open and nuScenes Dataset. Qualitative results are found in Figure and Figure 3, where "Static Scene Constraint" refers to performance of LiteMono [44] in Figure
5.2 Motion Segmentation and Method Ablation
Figure 4 evaluates the quality of the jointly learned binary motion mask network using precision and recall. We observe high precision with increasing recall, achieving a Fl score of 71.8% on Waymo Open [34]. Note that this segmentation quality is achieved without any labels. On nuScenes, Dynamo-Depth achieves F1 score of 42.8% with 57.2% on day-clear conditions and 21.1% on all other conditions. Notably, the motion mask net- work is able to segment out small dynamical ob- jects, such as the cyclist in the first row of Figure
0.8
Precision
0.6-
Abl L_c
Abl Mot Init
Abl L_g
0.4
Abl IN Pret
Ours
0.2
In addition, we show our ablation results on Waymo Open in Table 3. First, we found that ablating the motion consistency loss removes the regularization on the complete flow prediction, which causes the degenerate solution of moving objects on a flat canvas, as indicated by the poor segmentation per- formance in Figure 4. Similarly, by ablating motion initialization, initial independent motion prediction lead to large reconstruction errors, which diverges depth training. In addition, we observe that ablating the above ground prior preserves the performance of motion segmentation and depth of moving objects, but degrades the performance of depth on static background. Intuitively, this prior regularizes the structural integrity of the ground plane. Finally, we observed that the original set of hyperparameter values leads to training divergence when ablating the ImageNet pretrained weights. To compensate, we increase the initial number of epochs for depth learning from I to 2 (marked as x(2 without performing any hyperparameter sweeps.
0.2
0.6
Recall
Figure 4: Precision-Recall curve for motion segmentation on the Waymo Open Dataset.
6 Conclusion
In this work, we identify the effects of dynamical objects in unsupervised monocular depth estimation with static scene constraint. To mitigate the negative impacts, we propose to jointly learn depth, ego-motion, 3D independent motion and motion segmentation from unlabeled videos. Our key insight is that a good initial estimation of motion segmentation encourages joint depth and independent motion learning and prevents degenerate solutions from arising. As result, our approach is able to achieve state-of-the-art performance on both Waymo Open and nuScenes Dataset.
Limitations and Societal Impact. Our work makes brightness consistency assumption, where the brightness of pixel will remain the same. Although common in optical flow, this assumption limits the our method's i ability to model scenes with dynamical shadows, multiple moving light sources, and lighting phenomenons (e.g., lens flare). In addition, our work does not introduce any foreseeable societal impacts, but will generally promote more label-efficient and robust computer vision models.
10

---

7 Acknowledgement
This research is based upon work supported in part by the National Science Foundation (IIS-2144117 and IIS-2107161). Yihong Sun is supported in part by an NSF graduate research fellowship.
References
[1] Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and Ian Reid. Unsupervised scale-consistent depth and ego-motion learning from monocular video. Advances in neural information processing systems, 32, 2019. 3
[2] Houssem Eddine Boulahbal, Adrian Voicila, and Andrew I Comport Instance-aware multi- object self-superviiion for monocular depth prediction. IEEE Robotics and Automation Letters, 7(4):
2022. 3,8 8
[3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving- In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020. 1, 7,
[4] Vincent Casser, Soeren Pirk, Reza Mahjourian, and Anelia Angelova. Depth prediction without the sensors: Leveraging structure for unsupervised learning from monocular vidcos. In Pro- ceedings of the AAAl conference on artificial intelligence, volume 33, pages 8001-8008, 2019. 3, 8
[5] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009. 8
[6] David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE international conference on computer vision, pages 2650-2658, 2015. 7, 17
[7] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network. Advances in neural information processing systems, 27, 2014. 8
[8] Stefano Gasperini, Patrick Koch, Vinzenz Dallabetta, Nassir Navab, Benjamin Busam, and Federico Tombari. R4dyn: Exploring radar for self-supervised monocular depth estimation of dynamic scenes. In 2021 International Conference on 3D Vision (3DV), pages 751-760. IEEE, 2021. 3, 17,
[9] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(1 32(11):231-1233 2013. 17
[10] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self-supervised monocular depth estimation. Proceedings of the IEEE/CVF international conference computer vision, pages 3828-3838, 2019, 3, 8, 9, 15, 16, 18, 19
[11] Juan Luis Gonzalez and Munchurl Kim. Plade-net: Towards pixel-level accuracy for self- supervised single-view depth estimation with neural positional encoding and distilled matting loss. Proceedings of the IEEE/CVE Conference on Computer Vision and Patler Recognition, pages 6851-6860, 2021. 3, 17
[12] Ariel Gordon, Hanhan Li, Rico Jonschkowski, and Anelia Angelova. Depth from videos the wild: Unsupervised monocular depth learning from unknown cameras. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 8977-8986. 2019. 3
[13] Vitor Guizilini, Kuan-Hui Lee, Rareş Ambruş, and Adrien Gaidon. Learning optical flow, depth, and scene flow without real-world labels. IEEE Robotics and Automation Letters, 7(2):349133499 2022. 3, 17, 19

---

[14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition, 2016. 7, 15
[15] Tak-Wai Hui. Rm-depth: Unsupervised learning of recurrent monocular depth in dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1675-1684, 2022. 3,
[16] Junhwa Hur and Stefan Roth. Self-supervised monocular scene flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7396-7405,
2020. 3, 17, 19
[17] Junhwa Hur and Stefan Roth. Self-supervised nulti-frame monocular scene flow. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2684-2694, 2021. 3, 19
[18] Yang Jiao, Trac D Tran, and Guangming Shi. Effiscene: Efficient per-pixel rigidity inference for unsupervised joint learning of optical flow, depth, camera pose and motion segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5538-5547, 2021. 3, 17, 19
[19] Adrian Johnston and Gustavo Carneiro. Self-supervised monocular trained depth estimation using self-attention and discrete disparity volume. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 4756-4765, 2020.
[20] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. 7
[21] Marvin Klingner, Jan-Aike Termöhlen, Jonas Mikolajezyk, and Tim Fingscheidt. Self- supervised monocular depth estimation: Solving the dynamic object problem by semantic guidance. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part XX 16, pages 582-600. Springer, 2020. 3,8
[22] Seokju Lee, Sunghoon Im, Stephen Lin, and In So Kweon. Learning monocular depth in dy- namic scenes via instance-aware projection consistency. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 1863- 1872, 2021.
[23] Seokju Lee, Francois Rameau, Fei Pan, and So Kweon. Attentive and contrastive learning for joint depth and motion field estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021. 3,8
[24] Hanhan Li, Ariel Gordon, Hang Zhao, Vincent Casser, and Anelia Angelova. Unsupervised monocular depth learning in dynamic scenes. In Conference on Robot Learning, pages 1908- 1917. PMLR, 2021. 1, 3, 6, 8
[25] Zhenyu Li, Zehui Chen, Xianming Liu, and Junjun Jiang. Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation. Machine Intelligence Research, pages 1-18, 2023. 3, 17
[26] Liang Liu, Guangyao Zhai, Wenlong Ye, and Yong Liu. Unsupervised learning of scene flow estimation fusing with local rigidity. In IJCAI, pages 876-882, 2019. 3, 17
[27] Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ram Nevatia, and Alan Yuille. Every pixel counts++: Joint learning of geometry and motion with 3d holistic understanding. IEEE transactions on pattern analysis and machine intelligence, 42(10):2624 2644- 2019. 3, 8
[28] Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised learning of depth and ego-motion from monocular video using 3d geometric constraints. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5667-5675, 2018. 3
[29] Jieru Mei, Alex Zihao Zhu, Xinchen Yan, Hang Yan, Siyuan Qiao, Liang-Chied Chen, and Henrik Kretzschmar. Waymo open dataset: Panoramic video panoptic segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27,
2022, Proceedings, Part XXIX, pages 53-72. Springer, 2022. 17
12

---

[30] Matteo Poggi, Filippo Alcotti, Fabio Tosi, and Stefano Mattoccia. On the uncertainty of self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3227-3237, 2020. 3
[31] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, and Michael Black. Competitive collaboration: Joint unsupervised learning of depth, camera motion, optical flow and motion segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12240-12249, 2019. 1,3,8 8
[32] Kieran Saunders, George Vogiatzis, and Luis J Manso. Dyna-dm: Dynamic object- aware sule-supe monocular depth maps. In 2023 IEEE International Conference on Autonomous
Robot Systems and Competitions (ICARSC), pages 10-16. IEEE, 2023. 3, 8
[33] Libo Sun, Mia-Wang Bian, Huangying Zhan, Wei Yin, Ian Reid, and Chunhua Shen. Sc-depthv3: Robust self- supervisee monocular depth estimation for dynamic scenes. IEEE Transactions on Pattem Analysis and Machine Intelligence, 2023. 3, 17
[34] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaík, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2020. 1, 10, 17
[35] Madhu Vankadari, Stuart Golodetz, Sourav Garg, Sangyun Shin, Andrew Markham, and Niki Trigoni. When the sun goes down: Repairing photometric losses for all-day depth estimation. In Conference on Robot Learning, pages 1992-2003, PMLR, 2023. 3
[36] Sudheendra Vijayanarasinhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, and Ka- terina Fragkiadaki. Sfm-net: Learning of structure and motion from video. arXiv preprint
arXiv:170.007804 2017. 1.2
[37] Chaoyang Wang, José Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning depth from monocular videos using direct methods. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2022-2030, 2018.
[38] Yang Wang, Peng Wang, Zhenheng Yang, Chenxu Luo, Yi Yang, and Wei Xu. Unos: Unified unsupervised optical-flow and stereo-depth estimation by watching videos. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8071-8081, 2019. 17
[39] Zhou Wang, Alan C Bovik, Hamid Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004. 6
[40] Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel Brostow, and Michael Firman. The temporal opportunist: Self-supervised multi-frame monocular depth. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1164-1174, 1174, 2021. 3,
[41] Xuezhi Xiang, Rokia Abdein, Ning Lv, and Jie Yang. Self-supervised learning of scene flow with occlusion handling through feature masking. Pattern Recognition, 139:109487, 2023. 3, 17, 19
[42] Gengshan Yang and Deva Ramanan. Learning to segment rigid motions from two frames. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1266-1275, 2021.
[43] Zhichao Yin and Jianping Shi. Geonet: Unsupervised learning of dense depth, optical flow and camera pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1983-1992, 2018. 3, 8
[44] Ning Zhang, Francesco Nex, George Vosselman, and Norman Kerle. Lite-mono: A lightweight cnn and transformer architecture for self-supervised monocular depth estimation. arXiv preprint arXiv:2211.13202 2022. 8, 9, 10, 15, 16, 18, 19
13

---

[45] Sen Zhang, Jing Zhang, and Dacheng Tao. Towards scale-aware, robust, and generalizable unsupervised monocular depth estimation by integrating imu motion dynamics. European Conference Computer Vision, pages 143-160. Springer, 2022. 3, 17
[46] Songchun Zhang and Chunhui Zhao. Dyna-depthformer: Multi-frame transformer for self- supervised depth estimation in dynamic scenes. arXiv preprint arXiv:2301.0887, 2023. 3,
17
[47] Wang Zhao, Shaohui Liu, Yezhi Shu, and Yong-Jin Liu. Towards better generalization: Joint depth-pose learning without posenet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 9151-9161, 2020. 8,8
[48] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1851-1858, 2017. 2,
[49] Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In Proceedings of the European conference computer vision (ECCV), pages 36-53, 2018. 3
14

---

A Visualizations
In Figure show qualitative comparison between LiteMono [44] and our Dynamo-Depth. We select absolute relative difference (Abs Rel in quantitative results) as the measure of error, as shown in Eq. 15 with predicted depth d. ground truth depth d', and pixel coordinate p.
[d(p) - (x)l/d (p), if is a LiDAR point 0,
Abs Rel(p)
(15)
otherwise
Frame
Depth
Abs Rel
Lite-Mono
Dynamo-Depth
Lite-Mono
Dynamo-Depi
Waymo Open Datasel
nuScenes
KITTI
(a) Input Frames
(b) Monocular Depth
Monocular Depth
Abs Rel Emrrr
(e) Abs Rel Error
Figure 5: Qualitative comparison between LiteMono [44] and our proposed Dynamo-Denn The input frame is shown in column (a). Column (b) and (c) show the depth predictions and column (d) and (e) show the absolute relative difference for each LiDAR point with fixed error range [o, 1].
B Implementational Details
B.1 Hyperparameters
Please refer to Table for the hyperparameters used for training. The values are found via Waymo Open and kept the same for other datasets. The trade-off coefficient and depth smoothness coefficient "ed are set to 0.85 and 0.001, respectively, as [10]. The motion smoothness coefficient be consistent with Ysd and
Yse is set to 0.001 coefficient 7sm is set to a much higher value of 0.1 to remove sporadic false positives induced by and buildings. Since both rigid motion FR and complete
not tuned further. The mask smoothness
high frequency image features such motion Fc have relatively small magnitudes, we set the motion consistency coefficient " to be 5.0 properly enforce consistency. Due to the high loss values outputted by binary cross entropy, we the mask sparsity coefficient to be 0.04 to accommodate. The above ground coefficient Tn is set to 0.1 after searching across the values 0.01, 0.1, and 1.0.
B.2 Architectural Details
The depth network D adopted from two architectures, one with ResNet18 [14] backbone from Monodepth2 [10] and another with a CNN/Transfommer hybrid backbone from LiteMono [44], denoted
15

---

Table 4: List of hyperparameters used.
Hyperparameter
Value
a 0.85
Yad
0.001
Yee
0.001
0.1
T/e 5.0
7m 0.04
Y
0.1
as Dynamo-Depi (MD2) and Dynamo-Depth, respectively. For both versions of our model, the pose network P is adopted from Monodepth2 [10]. The encoders of the complete flow network C and motion mask network M are shared and have the same architecture as P. To obtain per per-pixel predictions, the features of the shared encoder are fed into the the separate decoders with the same architecture as the depth decoder in Monodepth2 [10] and with output per-pixel dimension of 3 and for C and M, respectively.
B.3 Training Details
For notational simplicity, Figure 2 only considers the case of single source frame La, while in practice, Dynamo-Depth is trained with two source frames: one frame previous and one frame after the target 4. Here, the respective reconstructions are rectified via minimal projection loss proposed in [10] obtain Í.. To enforce wewardd-ackwart consistency, the independent motion networks C and M predict single independent flow field Fy that is used to warp both source frames into the target frame, with the appropriate sign change.
Following Monodepth2 [10] and LiteMono [44], we perform auto-masking to facilitate learning during depth initialization. Afterwards, auto-masking is turned off the apparently static pixels may belong to dynamical objects traveling with the ego-camera, in which case auto-masking would mask out the corresponding reconstruction loss essential for independent motion learning.
There is a linear weight ramp for all loss terms that include Fc and M the beginning of each training stage, with the length of 2666 iterations for Waymo and nuScenes, and 8000 iterations for KITTI due to the limited number of moving objects.
B.4 Ground Plane Estimation
Concretely, 3D projected points (P; in Eq. 4) belonging to the bottom half of the image are sampled to construct the ground plane di via RANSAC (5 points are selected iteration, for a total of 100 iterations). This assumption does break under cases of traffic jams, but due the rare occurrence of these scenarios for this learning task assuming moving camera, the negative effects are limited.
B.5 Difficulties in Overfitting to Dynamical Objects
In Figure 6, we show qualitative example where a depth network learned with static scene constraint (LiteMono [44] in this case) would ignore the high reconstruction loss in earlier iterations in (b) and overfit to the moving objects via erroneous depth predictions only in later iterations in (c)-(f).
Input
Static Scene Constraint
Epachs Figure 6: Visualization of the overfitting phenomenon as shown via LiteMono [44], where erroneous depth for the black SUV and the oncoming car is only learned in later iterations, in (c)-(f).
Imput
(b) Epoch
[c) Epochs
Epochs
Epochs
16

---

C Dataset Details
We provide additional information of the three datasets used for evaluation.
For the Waymo Open Dataset [34], 76,852 front camera image-triplets from the provided train set containing 798 video sequences used for training while 2,216 front camera images uniformly sampled from the provided validation set containing 202 video sequences are used for evaluation. During training, frames are downsampled 480 320. We rectify the panoptic labels [29] with 3D box labels to obtain the corresponding masks for static and moving objects.
For the nuScenes Dataset [3], 79,760 front camera -mage-triplets from the provided train containing 700 video sequences are used for training while 6019 front camera images from the official validation set containing 150
KITTI
nuScenes
0.12
Fraction of Frames
Waymo
video sequences are used for evaluation (of
which 4449 are day-clear). Following [8], we
0.08
consider sequence to be in the day-clear sub-
if its description does not contain "night"
0.06
or "rain". During training. frames are down- sampled 512 288. We rectify the panop- tic LiDAR labels with annotated 3D bounding boxes to obtain motion attributes for LiDAR points.
0.02
Number of Movable Objects
Notably, our method learns to explicitly model independent motion, there is no filter- ing based on camera velocity recordings when constructing the training dataset.
Figure 7:
Histogram of the number of moveable objects per frame for all three dataset.
For the KITTI Dataset [9], follow the Eigen split [6] with 39,180 image- ttipeets used for training and 697 images used for evaluation. During training, frames are downsampled to 640 x 192.
Finally, we plot the histogram of the number of moveable objects per frame for all three datasets. Shown in Figure 7, Waymo Open Dataset has the most moveable objects with a mean 12.12, followed by nuScenes with a mean of 7.78. Due to the lack of per- per-frame object labels in KITTI, we evaluate on its curated object detection dataset, which has mean of 5.26. We note that the real distribution should be much tighter near 0, compared to the detection dataset with all images containing at least moveable object.
Table 5: Additional depth evaluation on the KITTI Dataset. Stereo Train and Stereo Infer. stand for the use of stereo-view during training and inference, respectively. Multi-Frann denotes the use of multiple frames as inputs during inference. Add. Sup. stands for additional supervision during training where I, S, and stand for IMU-level, synthetic-data, and pretrained depth model supervision, respectively.
Stereo
Multi- Add.
Error metric (i)
Accuracy metric (1)
Train Infer. Frame Sup.
Abs Rel
Sq Rel
RMSE
RMSE log
5 < 1.25
5 << 1.25?
ó 1.25
UnOS [38]
0.049
0.515
3.404
0.121
0.965
0.984
11.992
EffiSceee [18]
0.049
0.522
3.461
0.120
0,961
0,984
0,992
Xiung et af. [41]
0.048
0.487
3.447
0.117
0,964
0.985
0.992
Liu et al. [26] (stereo)
0.051
0.532
3,780
0.126
0.957
0.982
0.991
Liu al. [26] (mono)
0.108
5.528
0.863
0,948
0,980
Hur ef al. [16]
0.125
0.978
4.877
0,208
0,851
0.950
0.978
PLADE-Net [1]]
0.089
0.590
4.008
0.172
0.900
0.967
0.985
DRAFT [13]
0.097
0.647
3.991
0.169
0.899
0,968
0.984
DoreetrrrrFFrrer [46]
0.094
0.734
4,442
0.169
0.893
0.967
0,983
Mepth [40]
0.090
0.713
4.261
0.170
0,914
0.966
0.983
DepthFormer [25]
0.090
0.661
4.149
0.175
0.905
0.967
0.984
Zhang et al. [45]
0.108
0.761
4.608
0.187
0.883
0,962
0.982
SC-DepthV3 [33]
D
0.118
0.756
4.709
0.188
0.864
0.960
0.984
Dynamo-Depth
0.112
0.758
4.505
0.183
0.873
0.959
0.984
17

---

Table 6: Depth evaluation on the nuScenes Dataset with day-clear split. IM stands for independent motion where denotes lack of independent motion modeling. Sup. indicates additional supervision given during training, where "V" indicates camera velocity recordings and 'R' indicates radar scans. Manual replication with released code indicated by 1.
Error metric (L)
Accuracy metric (t)
IM Sup.
Abs Rel
Sq Rel
RMSE
RMSE log
á << 1.25
1.25²
1.25
Monodepth21 [10]
X
0.157
2.432
6.943
0.248
0.844
0,940
0.968
LiteMonof [44]
0.161
2.618
6.818
0.251
0.846
0.939
0,966
R4Dyn-L [8]
V.R
0.130
1.658
6.536
0.858
Dynamo-Depth (MD2)
0.149
1.792
6,536
0.236
0.830
0.938
0.972
Dynamo-Depth
0.132
1.571
6.158
0.216
0.856
0.951
0.977
Table 7: Depth evaluation with semantic split on the nuScenes Dataset with day-clear split. S.B., S.O. and M.O. denotes the partition of pixels that are static background, static moveable object and moving object, respectively. Manual replication with released code indicated by t.
Abs Rel (4)
8 < 1.25 (1)
All
S.B.
S.O.
M.O.
All
S.B.
S.O.
M.O.
Monodepth21 [10]
0.157
0.150
0.214
0.324
0.844
0.850
0.774
0.626
Dynamo-Depth (MD2)
0.149
0.146
0.186
0.197
0.830
0.834
0.779
0.737
A (%)
-5.1
-2.7
-13.1
-39.2
-1.7
-1.9
+0.6
+17.7
LiteMono [44]
0.161
0.151
0.232
0.373
0,846
0.853
0.781
0,572
Dynamo-Dees -Dethh
0.132
0.130
0.168
0.167
0.856
0.857
0.795
0.804
A (%)
-18.0
-13.9
-27.6
-55.2
+1.2
+0.5
+1.8
+40.6
D Additional Results
D.1 Additional KITTI Comparisons
Please refer to Table 5 for additional comparisons on KITTI, where the compared methods leverage either additional modalities additional supervision beyond the of semantic information as shown in Table 1.
D.2 Additional nuScenes Comparisons
Please refer to Table 6 and Table 7 for depth performance comparisons on the nuScenes val- idation day-clear subset, along with precision- recall curve for motion segmentation Fig- ure 8. Note that R4Dyn-L [8] is only trained on the day-clear subset with velocity filtering, while the rest of the methods
1.0
0.8
all trained with the entire dataset without any filtering. Furthermore, Table 7 still demonstrates the trend as observed in Table 2, where the a sig-
Precision
0.6
other
0.4
day-clear
nificant improvement on moving objects are observed in terms of both accuracy and er- ror. Finally, as shown in Figure 9, both meth- ods fail under low-light conditions where the brightness consistency assumption no longer holds. Due to the rigid scene constraint, arti- facts appear the respective depth prediction in (b), which causes much larger enror com- pared to the prediction in (d), as discussed Section 5.
0.0
0.2
0.4
0.6
0.8
1.0
Recall
Figure 8: Precision- Recall curve for motion segmen- tation the nuScenes Dataset, which is partitioned into the day-clear and other subsets.
18

---

Input
Static Scene Constraint
Dynamo- -Dant (Ours)
(5) Input Frames
[b) Manocular Depth
(c) Rigid Flow
(d) Monocular Depth
(e) Rigid Flow
Independent Flow
Figure 9: Visualizing predictions under night-time conditions in the nuScenes dataset.
D.3
Odometry Evaluation
Please refer to Table for odometry evaluation for Dynamo-Depth and Dynamo-Depth (MD2) on both nuScenes day-clear subset and Waymo Open Dataset.
Table 8: Odometry results on the nuScenes (day-clear subset) and Waymo Open Dataset. Results show the average absolute trajectory error over all overlapping 5-frame snippets and 100-frame snippets in the test sequences, and standard deviation, in meters. For consistency, the first 100 test sequences for both datasets are evaluated. Manual replication with released code is indicated by 1
Dataset Speed (m/s)
nuScenes
Waymo
5.657 + 4.188
6.978
6.028
Snippet Length (# frames)
100
5
100
Snippet Duration (seconds)
0.3
7.4
0.4
9.9
Snippet Counts
22820
13320
19364
9864
Monodepth2f [10]
0.016 + 0.017
0.097 0.069
0.018 0.029
0.143 + 0.194
Dynamo-Depth (MD2)
0.019 + 0.029
=.112:0116
0.015
# 0
LiteMono [44]
0.018 + 0.018
0.090 + 0.065
0.044
0.156
0.194
Dynamo-Depa
0.017 + 0.019
0.081 = 0.070
0.037
0.107 + 0.164
D.4 Scene Flow Comparisons
We were able compare against the mentioned related works in unsupervised scene flow estimation due to the large difference in methodology (stereo-view during training [18, 41, 16, 17J) and lack of codebase for reproducibility [13].
E Limitations
As shown in Figure 10, dynamical shadows and moving light sources in low-light conditions violate the brightness consistency assumption, which results in erroneous predictions of motion segmentation and monocular depth, respectively.
(a) Input Frames
(b) Monocular Depth
(c) Rigid Flow
(d) Independent Flow
(e) Motion Mask
Figure 10: Limitations of Dynamo-Depth
19


Reviews:
[{"review_id": "SAW6dRwzDU", "reviewer": "Reviewer_SBjP", "summary": "This paper deals with monocular depth estimation by optical flow estimation.\nThe key is to decompose the flow as ego-motion-induced flow and independent flow.\nTo implement the idea, depth is predicted on the target image, and camera pose change from target to source is also estimated.\nThen a flow network predicts the flow from the target to the source, and a motion mask is also predicted.\nThen the final flow that warps the source to the target is recomposed by an alpha combination of the independent flow from the flow network and the rigid flow computed from the depth and the pose change. \nExperiments are performed on Kitti, nuScenes, and Waymo Open.", "questions": "1. \"Prima facie, the epipolar ambiguity should not be that much of a problem for a learning-based depth estimation system. This is because independent object motion should be generally inconsistent between different videos, and the depth estimator network cannot recover this information from a single frame alone. Thus, in a hypothetical world filled by moving spheres, the motion information is unobtainable from a single frame and the depth estimation network would indeed regress to the correct depth.\" can you explain a bit what is the underlying reasoning that the ambiguity should not be a problem? The writing is not explicit, and I have to guess the meaning of words, e.g., \"the depth estimator network cannot recover this information\" what does \"this information\" mean? why if the motion is unbotainable then the depth estimation network would indeed regress to the correct depth?\n2. please clarify the major contribution of this work.", "limitations": "The authors claimed no societal impacts.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Explicit modeling of the ego-motion of the ego car and the independent motion of the moving car can help improve depth estimation.\nSince the disentanglement can help optimize the right part of the parameters, otherwise, if there is no explicit modeling, then the network can tradeoff the accuracy on the depth to get an accurate optical flow due to ambiguities.", "weaknesses": "The idea of modeling independent motion and ego motion for depth estimation, flow estimation, and moving object estimation is not quite new. For example, this idea has appeared in the following papers:\n- \"UnOS: Unified Unsupervised Optical-flow and Stereo-depth Estimation by Watching Videos\" CVPR 2019.\n- \"GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose\" CVPR 2018\n- \"Towards scale-aware, robust, and generalizable unsupervised monocular depth estimation by integrating IMU motion dynamics\" ECCV 2022\n- \"Dyna-DM: Dynamic Object-aware Self-supervised Monocular Depth Maps,\" 2023 IEEE International Conference on Autonomous Robot Systems and Competitions (ICARSC)\n- \"Learning Optical Flow, Depth, and Scene Flow Without Real-World Labels\" ICRA 2023\n- \"Instance-Aware Multi-Object Self-Supervision for Monocular Depth Prediction\" RAL 2022, seems to have better results on Kitti.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and for agreeing with the motivation of our work. We list our response to your questions and concerns below. \n\n---\n### Modeling independent motion and ego-motion is not new\nCompared to the mentioned papers, our method learns from **monocular** videos only **without any labels or semantic knowledge** from pretrained models and jointly predicts monocular depth, ego-motion, 3D independent flow, and motion segmentation to disambiguate between the rigid and non-rigid parts of a dynamical scene. The following bullet points detail the differences in the methodology and the table of results are provided in Table 8 (Rebuttal PDF). \n\n- UnOS (Wang et al.) uses **stereo input** for both training and inference, which is a different task from our method which learns from monocular videos only.\n\n- DRAFT (Guizilini et al.) utilizes **additional synthetic data with synthetic ground truth**, whereas our method only learns from the unlabeled monocular videos. Additionally, **multiple frames** are used to predict monocular depth for DRAFT, where our approach only uses a single frame.\n\n- Zhang et al. uses **additional inertial measurement unit (IMU) measurements** for training. In comparison, our method only has access to monocular videos and explicitly models independent motion field and motion segmentation to aid depth learning. We achieve comparable results on KITTI without accessing additional supervisions.\n\n- Dyna-DM (Saunders et al.) uses a **background mask from an off-the-shelf pretrained Mask R-CNN** to reason with dynamical objects. In comparison, our method does not use any semantic knowledge from pretrained networks and reason about dynamical objects solely by disentangling the underlying motion. Quantitatively, our approach consistently outperforms Dyna-DM on KITTI.\n\n- Boulahbal et al. [2] uses a **frozen pretrained segmentation network trained on Cityscape** to predict piecewise rigid pose for each dynamic object, whereas our method does not leverage any semantic knowledge from pretrained networks. Our method is comparable with [2], despite not obtaining object information from pretrained networks.\n\n- GeoNet [28] (Yin et al.) only **models moving objects in 2D** via optical flow, in contrast, we model the behavior of moving objects via a 3D motion field. Modeling the 3D motion allows our model to accurately learn the depth of independently moving objects by explaining its overall flow as a combination of independent motion and ego-motion, rather than ignoring the dynamical pixels in the 2D case. In addition, the consistency mask predicted by [28] also includes occlusions and non-Lambertian surfaces, which is different from motion segmentation in our case. Quantitatively, our method significantly outperforms [28] on KITTI.\n\nThank you for pointing out the references, the appropriate discussion and comparison will be incorporated in the final draft.\n\n---\n### Clarify Contribution\nWe list the following four main contributions for our work:\n- Our method learns from **monocular** videos only **without any labels or semantic knowledge** from pretrained models and jointly predicts monocular depth, ego-motion, 3D independent flow, and motion segmentation.\n\n- We reformulate the independent 3D motion field as the difference between the rigid flow induced by ego-motion and complete scene flow to enable proper regularization and facilitate learning.\n\n- We propose a **novel motion initialization** technique that resolves the ambiguity between camera motion and independent motion that frequently appears in complex dynamical scenes. This initialization allows the proposed architecture to be jointly trained without any labels.\n\n- Our proposed method improves unsupervised monocular depth performance for dynamical scenes, especially for moving objects, as benchmarked on both Waymo Open and nuScenes.\n\n---\n### Clarify “Prima facie…” on L115-L120\nHere is a rephrasing of that paragraph:\n\nThe depth network gets a single frame as input and must predict where objects are in the 3D scene. The training objective is that when the camera moves, the next frame must be correctly reconstructed. \n\nFor independently moving objects, even in the ideal scenario when the depth work predicts the right depth for these dynamical objects, their predicted position in the next frame will still be incorrect due to their own motion. However, the same object will move one way in one video, another way in another video and be static in a third. Since the moving instances are unpredictable and cannot be modeled by the depth network from a single frame, the reconstruction error caused by their independent motion cannot be avoided. \n\nTherefore, if there is no a priori bias in the direction or speed of object movement in the data, then the best that the depth network can do is to fit the static instances well and thus learn to predict the correct depth. This is what we mean by the depth network not being able to recover “this information” which is the object's motion.\n\nOur point is that unfortunately, in self-driving scenes, there are statistical regularities in object motion. For example, a car viewed from the back will mostly be moving away from the camera. This means that the depth network has the ability to predict this movement from a single frame and alter its depth prediction (by placing the car farther away than the true depth) as shown in Figure 1(a) to reconstruct the next frame correctly. Conversely, when one sees the frontal view of a car, it is likely that the car is moving towards the camera and the depth network can predict the car to be closer to explain the pixel motion, as shown in Figure 1(b). Additionally, the speed of the object can be roughly estimated from object identity (pedestrian < cyclist < car) and background context (residential < city < highway).\n\nWe hope this helps and will update the phrasing.\n"}, {"review_id": "XFaRqmozKZ", "reviewer": "Reviewer_9riS", "summary": "This paper presents a method for monocular depth estimation in dynamic scenes, employing an unsupervised learning approach. While prior work utilizes networks for depth estimation, pose estimation, flow estimation, and mask estimation to decompose object motion and agent motion, the proposed method in this paper similarly adopts these network structures. Distinguishing itself from earlier works, this paper introduces an initial estimation of motion segmentation for the joint learning of depth and independent motion. The main contribution lies in providing an effective initial estimation of motion segmentation, which is critical for determining which pixels in a scene are moving independently. An ablation study demonstrates the effectiveness of the proposed initialization technique. Furthermore, the method achieves state-of-the-art performance on the Waymo and nuScenes datasets.", "questions": "- Is the proposed network trained separately for each dataset (KITTI, Waymo, nuScenes), resulting in distinct sets of parameters as displayed in Table 1? Alternatively, is there a single trained network that is tested across all three datasets?\n- The 'Model and Training setup' subsection appears to describe a training schedule. Is this training schedule uniformly applied across all datasets (KITTI, Waymo, nuScenes), or is it adapted according to specific dataset properties such as size and resolution?", "limitations": "Authors adequately addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The proposed method significantly outperforms the state-of-the-art in the Waymo and nuScenes datasets.\n- The inclusion of an ablation study effectively illustrates the significance of the proposed initialization techniques.\n- The paper introduces a simple yet effective technique that enhances the performance of depth estimation.\n- Comprehensive training details are provided, along with evaluations concerning various regions, including static backgrounds, static movable objects, and moving objects.", "weaknesses": "- The proposed method fails to surpass the performance of existing approaches on the KITTI dataset.\n- The paper primarily focuses on the initialization technique for motion segmentation, which appears overly simplistic and may limit the scope of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for finding the proposed method effective and our evaluation convincing. We list our response to your questions and concerns below.\n\n---\n### Fail to Surpass Existing SOTA on KITTI\nThank you for pointing this out. Although our approach performs only comparably to existing SOTA on KITTI, the lack of moving objects in the KITTI dataset makes it a suboptimal benchmark for evaluating monocular depth estimation for dynamical scenes. In fact, on KITTI, LiteMono [29], without modeling any independent motion, is able to outperform Li et. al [17] and RM-Depth [11], which both explicitly model independent motion. We also offer further insights into this in our manuscript (L237-L239) and supplementary materials (Figure 6). This view is corroborated by *Reviewer EvaZ*, who pointed out that “KITTI dataset has been overfitted, and it contains fewer moving objects than the other two datasets.”\n\n---\n### Limited Scope\nThe initialization strategy of motion segmentation does not just impact the motion segmentation, but in fact is crucial for all parts of the pipeline. As depicted in Section 4.4, the joint learning of depth, ego-motion, and independent motion is inherently underdetermined, as a family of solutions of depth and object motion would produce accurate reconstruction. Therefore, the proposed initialization of motion segmentation serves as a gradient router that assigns rigid pixels to be modeled by depth and ego-motion via $F_R$ and assigns non-rigid pixels to be modeled by the independent flow field via $F_C$ (Eq. 15). Thus, without using any labels, rigid motion and independent motion can be disambiguated and jointly learned without over-explaining and overfitting to their counterparts when modeling dynamical scenes.\n\nMore broadly, while the idea of improving initialization may appear simplistic, this is not the case for underconstrained problems with whole families of solutions. For such problems, we believe that good initialization strategies are not just an engineering trick of limited scope, but are in fact a fundamental component of the solution and deserve careful investigation. \n\nFurthermore, by focusing our attention on the initialization, we are able to create a “simple, yet effective” approach as you point out. We are encouraged that you agree with us that simplicity is a virtue. We could have added more complexity into the architecture or the training paradigm, but we believe that this would have obscured the fundamental insight.\n\nWe appreciate your opinion and we will further discuss this point in the introduction and Section 4.4.\n\n---\n### Experimental Setup\nThe models are trained individually per each dataset, with identical hyper-parameters and the same training paradigm applied uniformly (with exception of input image dimension detailed in L232-L236). The hyperparameters and training paradigm are found on Waymo and directly applied to KITTI and nuScenes without further tuning. We believe that the motion initialization improves training stability and allows generalization across different data distributions with a uniform setup. Further clarification and discussion will be added in the experiment section.\n"}, {"review_id": "UpNpC7R9Rr", "reviewer": "Reviewer_EvaZ", "summary": "This paper proposes a self-supervised depth estimation method. It addresses dynamic objects in the scene by jointly learning monocular depth, 3D independent flow field, and motion segmentation from unlabeled videos. The paper writing is very good, and the motivation/insights are clearly stated. The method is evaluated in KITTI, Waymo, and nuSecens datasets. Overall, I think that it is a high-quality paper.", "questions": "NA", "limitations": "NA", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper is well-written, and it is easy to read.\n\nThe results are evaluated in Waymo and nuSecens datasets. This is very good because the KITTI dataset has been overfitted, and it contains fewer moving objects than the other two datasets.\n\nThe Sec 3 is very informative. The discussion on motivations makes the paper easy to understand.  \n\nThe evaluation results and ablation studies clearly demonstrate the efficacy of the proposed method.\n", "weaknesses": "\nThere is no obvious weakness to me, but I think that it would be better if the authors could show visual odometry results. The nuSences and Waymo datasets provide camera poses, so it is not hard to do.\n\n[a] also proposes a method for dynamic objects, but it uses a different solution. The comparison is not required, but it would be better if making a discussion on different ideas for addressing dynamic objects.\n\n[a] SC-DepthV3: Robust Self-supervised Monocular Depth Estimation for Dynamic Scenes, arXiv 2022\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your support and for finding our paper well-written and the proposed method effective. We are encouraged by your remarks on our evaluations. We share your concern regarding the overfitting and fewer-moving-object issue that makes KITTI a less effective benchmark compared to Waymo and nuScenes for unsupervised monocular depth estimation in dynamical scenes. In the following sections, we list our responses to your questions.\n\n---\n### Visual Odometry Results\nTable 9 (Rebuttal PDF) compares Monodepthv2, LiteMono, and our proposed methods on their visual odometry performance on both Waymo Open and nuScenes Dataset. For consistency, camera poses from the first 100 test sequences for both datasets as used for evaluation. As shown, our approach achieves comparable results and we will incorporate these results in the final version.\n\n---\n### Additional Reference \nThanks for pointing out [a], as it is relevant to our research direction. Similar to works [4, 9, 15] where semantic knowledge is injected via pretrained segmentation networks to mitigate the negative influence of dynamical objects, [a] utilizes a depth prior injected via a pretrained depth network to do the same. The pretrained depth network in [a] is trained on a large dataset with ground truth depth values, while in contrast, our approach uses no ground truth depth at all. Although KITTI is a suboptimal benchmark for evaluating monocular depth estimation for dynamical scenes, we report the performance of [a] on KITTI, for a consistent comparison with the rest of the mentioned references. We will also discuss [a] in the related works.\n"}, {"review_id": "AxLPxQSPef", "reviewer": "Reviewer_mnoy", "summary": "This paper aims to estimate the depth map in a dynamic scene environment.\nTo this end, the paper proposed 1) two separate module architectures that estimate rigid scene flow and residual scene flow and 2) a motion initialization method to enable a stable learning process.\nHowever, the technical and performance comparisons with the previous self-supervised depth estimation, self-supervised scene flow estimation, self-supervised motion segmentation, and self-supervised optical flow are insufficient.\nThe current manuscript needs a lot of modification.\n", "questions": "My major questions are shown in the weakness part.", "limitations": "This paper must include in-depth technical and performance comparisons with the recent self-supervised depth, scene flow, optical flow, and motion segmentation literature.\nThe current manuscript doesn't support the proposed method's originality or superiority without the comparisons.\n", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "- The proposed method achieves better performance in nuScenes and Waymo datasets. However, the comparison group is too limited to judge the superiority of the proposed method.", "weaknesses": "W1. Technical and performance comparisons with the recent self-supervised scene flow estimation\n\nThe proposed method needs to describe its originality and superiority compared to recent scene flow estimation on the KITTI, nuScene, or Waymo datasets.\n\n- Xiang, Xuezhi, et al. \"Self-supervised learning of scene flow with occlusion handling through feature masking.\" Pattern Recognition 139 (2023): 109487.\n- Jiao, Yang, Trac D. Tran, and Guangming Shi. \"Effiscene: Efficient per-pixel rigidity inference for unsupervised joint learning of optical flow, depth, camera pose and motion segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n- Hur, Junhwa, and Stefan Roth. \"Self-supervised monocular scene flow estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n- Hur, Junhwa, and Stefan Roth. \"Self-supervised multi-frame monocular scene flow.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n- Guizilini, Vitor, et al. \"Learning optical flow, depth, and scene flow without real-world labels.\" IEEE Robotics and Automation Letters 7.2 (2022): 3491-3498.\n\nW2. Technical and performance comparisons with the recent self-supervised motion segmentation\n\nThe proposed method needs to describe its originality and superiority compared to recent motion segmentation on the KITTI, nuScene, or Waymo datasets.\n\n- Liu, Liang, et al. \"Unsupervised Learning of Scene Flow Estimation Fusing with Local Rigidity.\" IJCAI. 2019.\n- Jiao, Yang, Trac D. Tran, and Guangming Shi. \"Effiscene: Efficient per-pixel rigidity inference for unsupervised joint learning of optical flow, depth, camera pose and motion segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n- Xiang, Xuezhi, et al. \"Self-supervised learning of scene flow with occlusion handling through feature masking.\" Pattern Recognition 139 (2023): 109487.\n\nW3. Technical and performance comparisons with the recent self-supervised optical flow estimation \n\nThe proposed method needs to describe its originality and superiority compared to recent optical flow estimation on the KITTI, nuScene, or Waymo datasets.\n\n- Teed, Zachary, and Jia Deng. \"Raft: Recurrent all-pairs field transforms for optical flow.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.\n- Jonschkowski, Rico, et al. \"What matters in unsupervised optical flow.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer International Publishing, 2020.\n- Zhao, Wang, et al. \"Towards better generalization: Joint depth-pose learning without posenet.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\nW4. Technical and performance comparisons with the recent self-supervised depth estimation \n\nThe proposed method needs to describe its originality and superiority compared to recent self-supervised depth estimation on the KITTI, nuScene, or Waymo datasets.\n\n- Watson, Jamie, et al. \"The temporal opportunist: Self-supervised multi-frame monocular depth.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n- PLADE-Net: Towards Pixel-Level Accuracy for Self-Supervised Single-View Depth Estimation with Neural Positional Encoding and Distilled Matting Loss, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n- Guizilini, Vitor, et al. \"Multi-frame self-supervised depth with transformers.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to point out the references!  We outline the differences between our approach and each of the references below. We have added many of these references to Table 8 (see below for details) and we also discuss how we propose to add additional evaluations to our final camera-ready.\n\n---\n### Depth Estimation \n\nAmong the three mentioned works, PLADE-Net uses **stereo-view during training**, while ManyDepth and DepthFormer uses **multiple frames during inference**. In comparison, our depth model is trained from monocular videos only and takes a single frame as input during inference. The following bullet points detail the differences in comparison to our proposed work.\n\n- PLADE-Net uses **stereo input** during training, which is a different task from our method which learns from monocular videos only. \n\n- ManyDepth (Watson et al.) is proposed as a cost volume based approach to fuse temporal information from **multiple frames as input**, whereas our method only requires a single frame. To alleviate the effect of moving objects, ManyDepth trains a single-frame inference-time monocular depth network and leverages it as a teacher network. However, the teacher network would be susceptible to the same issues as we detailed in Section 3, making it unreliable in more complex scenes (Figure 1). \n\n- DepthFormer (Guizilini et al.) uses **multiple frames as input**, whereas our method only requires a single frame.\n\nNevertheless, for all the above mentioned papers that evaluate depth estimation performance, we have now added their results into Table 8 (Rebuttal PDF). We will include this new comparison and the above discussion in the final camera ready. \n\n---\n### Scene Flow Estimation\nAll five mentioned works leverage either stereo information or additional synthetic data, while our model learns from **monocular** videos only **without any labels or semantic knowledge** from pretrained models and predicts complete scene flow along with monocular depth, independent motion field, and motion segmentation. Please refer to the following bullet points for additional discussions.\n- Xiang et al. uses **stereo information** during both training and inference, while our approach has no access to stereo information.\n\n- Similarly, EffiScene (Jiao et al.) uses **stereo information** during both training and inference, while our method predicts depth from monocular view only.\n\n- Hur et al. 2020 uses **stereo information during training**, while our method is strictly monocular.\n\n- Hur et al. 2021 builds on top of Hur et al. 2020 and also uses **stereo information during training**.\n\n- DRAFT (Guizilini et al.) **utilizes additional synthetic data with synthetic ground truth**, whereas our method only learns from the unlabeled monocular videos.\n\nIn spite of the different problem setup, we include depth results from all of these papers (with the exception of Hur et al. 2021, which does not report depth) in Table 8. We will add these to the main paper.\n\nWe will also include scene flow comparisons to these papers in the camera ready. However, we believe that this comparison must be done on Waymo and NuScenes because of the lack of moving objects in KITTI as pointed out by *Reviewer EvaZ*, L237-L239 in the manuscript, and Fig. 6 in the Rebuttal PDF. We will perform this comparison and add results to the main paper.\n\n---\n### Motion Segmentation\nAll three mentioned works leverage **stereo information** during training, with EffiScene (Jiao et al.)  and Xiang et al. using **stereo information** during inference. This gives significant information to depth without needing to learn from camera motion during training. Once depth can be estimated, the independent motion that is entangled with rigid-motion, especially collinear with camera motion, can be disambiguated as well. In contrast, in our work, we are able to learn object motion and motion segmentation jointly with depth from monocular videos only without using any labels. \n\nWe will incorporate additional discussions on this in the final version. In addition, we will also include quantitative comparisons where possible. Once again, we believe that this must be done on Waymo and NuScenes as discussed above.\n\n---\n### Optical Flow \n\nFor optical flow, we would like to note that optical flow is not a focus of our work. We have no network that directly predicts optical flow, and we have no loss functions that encourage accurate optical flow to emerge (e.g. forward-backward consistency). Although we can obtain an indirect optical flow from our model by combining the outputs of our various modules, we do not claim that this will yield state-of-the-art on optical flow benchmarks. \n\n"}, {"review_id": "XNMacajVsY", "reviewer": "Reviewer_9pRa", "summary": "This paper introduces a new unsupervised approach and a new network architecture for monocular depth estimation.\n\nThe paper is motivated by the necessity of new algorithms capable of learning to predict depth from unlabelled data while being able to perform reliable predictions in the regions of moving objects, which is particularly critical for autonomous driving applications.\n\nTo tackle this problem, the authors proposed a new architecture that separates the observable flow field as rigid regions and independently moving regions. A training scheme is also proposed to effectively learn this disentanglement guided by a motion masking that is also learned in a self-supervised manner.\n\nThe proposed approach is evaluated on three datasets and achieves state of the art results on nuScenes and Waymo Open, specifically improving by 50% on moving objects.", "questions": "How the ground plane d^g_t used in Eq. 13 is defined? The authors mentioned that it is estimated via RANSAC, but is this assuming that the ground is mostly empty and visible? What happens in traffic jam situations? Please clarify this part of the method and provide what are the assumptions for the ground plane estimation.\n\nTypos:\nL48 \"learning erroneous depth for the moving objects and allow.\"\nL51 \" achieve achieve\"\nL198: F_D instead of F_d ?", "limitations": "Limitations were briefly discussed in the paper and no major concerns are remaining.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The main insight in the proposed method is to decompose the flow field used to obtain depth supervision as rigid flow F_R, which goal is to explain the flow field due to the camera motion as if the scene where rigid, and independent flow F_I, which models the flow field resulting from moving objects. Decomposing these two factors is not trivial and the paper proposes an interesting solution to the problem that does not require annotated labels.\n\nThe insights presented about the later overfitting of depth estimation models to dynamically moving objects is a very interesting finding and supports the main idea in the paper. The authors successfully explored this observation to propose a new training scheme for learning the motion mask without requiring segmentation labels.\n\nThe results are convincing both quantitatively and qualitatively, as provided in the supplementary material, specially for moving objects. The paper is well written and relatively easy to follow.\n", "weaknesses": "As in many previous works, the proposed approach targets the depth estimation problem as a novel-view synthesis problem. The particular case of moving objects is then tackled by a separated 3D flow field that captures the independent motion of moving objects, which is coherently discussed in the paper. Although this setup seems to be generic, the proposed approach is only evaluated in the context of autonomous driving. Therefore, it is not evidenced in the paper if the proposed formulation would adapt well to different contexts if a moving camera. For instance, if the camera is moving in a way that results in no parallax effect, could this framework diverge to a solution with most of the pixels being segmented as moving objects?\n\n\nThe main contribution of the paper is in the network architecture and specially in the training strategy. However, the idea of braking the scene rigidity assumption by estimating a motion field for moving objects was already explored in [11] and the notion of sparse moving objects in the scene as a prior was also explored in [17]. The main differences with these two works in special are only briefly mentioned in the paper.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your support and constructive comments! We are encouraged that you find the paper well-written, the insights interesting, and the results convincing! We list our response to your questions below.\n\n---\n### Contexts Beyond Self-Driving\nOur approach would work well in other general settings where camera motion is informative for scene geometry. Even when there are some videos without parallax effects, if **the majority of the dataset contains meaningful camera motion**, then the model would recover from the ambiguity in these edge cases and work as intended.\n \nIf the camera motion results in **no parallax effect at all** for the entire dataset, all scenes would become ambiguous and the geometry becomes impossible to recover. To clarify, the camera motion that does not produce parallax effect would correspond to rotation about the pin-hole. In this case, for our proposed approach, due to the sparsity regularization term (Eq. 13), it is more likely for the model to predict every point in the scene to be static without any independent motion (assuming the apparent motion is ego-motion-induced only). \n\nRegarding our choice of self-driving data, to the best of our knowledge, we could not find any large real dataset that captures dynamical scenes in the wild for training. Therefore, we focus our attention on the challenging self-driving setting, where the epipolar ambiguity and the statistical regularities that lead to erroneous depth predictions are highly prevalent.\n\n- Epipolar Ambiguity: Collinearly moving objects are common, since all agents in nearby lanes are moving in the same or opposite direction as the camera. \n\n- Statistical Regularities: From one frame, an object’s appearance can indicate its direction of motion (frontal vs. rear view) and a rough estimate of speed (car vs. cyclist).\n\nWe will incorporate discussions regarding different contexts and a more general setting in the final version.\n\n---\n### Comparison against Li et. al [17] and RM-Depth [11]\nOur proposed method differs from both Li et. al [17] and RM-Depth [11] in the following ways:\n\n- Our method learns a separate motion segmentation network, whereas Li et. al [17] and RM-Depth [11] keep it implicit. By explicitly predicting dynamical regions as opposed to implicitly thresholding the independent motion field in [11,17], our method has a better capacity for learning semantically meaningful motion segmentation, instead of modeling low-level pixel motion outliers.\n\n- Unlike [11,17] where the residual/independent motion field is predicted directly, we model the independent motion field $F_I$ by learning the complete scene flow $F_C$. Unlike the independent motion field $F_I$, which also requires accurate ego-motion for correct reconstruction, the complete scene flow $F_C$ is much easier to learn since it is independent of the predicted ego-motion, removing its dependence on the quality of the ego-motion prediction during training. Intuitively, it is much more difficult to learn and predict independent motion directly when the observed motion in the input consists of both rigid and non-rigid motion entangled together.\n\nSpecifically compared to Li et. al [17]:\n\n- Li et. al [17] applies a $L_{1/2}$ sparsity loss on the independent motion field which directly penalizes large motion that is common in the real world, whereas we apply the sparsity loss on the motion segmentation network without suppressing the independent motion field. \n\n- As a result, we achieve superior performance on Waymo against Li et. al [17], reducing the Absolute Relative difference from 0.162 to 0.116.\n\nSpecifically compared to RM-Depth [11]:\n\n- RM-depth only reports performance on KITTI and Cityscapes without releasing code for reproducibility so we cannot compare against its ability in modeling moving objects in more chaotic settings such as Waymo or nuScenes. (We reached out to the author but our request for a code base for only evaluation purposes was denied.)\n\nThank you for pointing this out, as we will include additional discussion in the final draft.\n\n---\n### Ground Plane Estimation\nConcretely, 3D projected points ($P_t$ in Eq.5) belonging to the bottom half of the image are sampled to construct the ground plane $d^g_t$ via RANSAC (5 points are selected per iteration, for a total of 100 iterations). It is true that traffic jams break this assumption, but due to the rare occurrence of these scenarios for this learning task assuming a moving camera, the negative effects are limited. Thank you and we will clarify this point in the final version.\n\n---\n### Typos\nThank you for pointing out these typos and we will correct them in the final version."}]

Author rebuttal:
We thank the reviewers for their insightful comments and positive feedback. We are encouraged that they found our work well-written (*Reviewer EvaZ, 9pRa*), well-motivated (*Reviewer SBjP*), effective (*Reviewer 9riS, EvaZ, 9pRa*), and with proper evaluations (*Reviewer 9riS, EvaZ*).

We appreciate all of the paper references mentioned in the reviews. To be clear, in our work, we aim to solve **unsupervised monocular depth estimation in dynamical scenes**. At training time, we only have access to a set of monocular videos and assume no labels of any kind (synthetic or real), no stereo information, no other input modalities, and no auxiliary pretrained models. At inference time, monocular depth is predicted from a single frame without any stereo or sequences of frames.

We are encouraged that the mentioned works address dynamical scenes in some capacities, however, most are tackling different problem setups that make it easier to disentangle and mitigate the presence of dynamical objects.

- ***stereo-view during training***. Stereo-view gives significant information to depth without needing to learn from camera motion during training. Once depth can be estimated, the independent motion that is entangled with rigid-motion, especially collinear with camera motion, can be disambiguated as well. In contrast, in our work, we are able to learn depth jointly with object motion from monocular videos only without using any labels.

- ***additional modalities during training***. More broadly, access to additional modalities can reduce the learning complexity by adding additional information to the observed videos. For instance, camera motion can be found via an inertial measurement unit (IMU), without being estimated from the apparent motion in the videos. In our setting, the apparent motion cues are the sole input and supervising signal for learning.

- ***auxiliary models during training***. Auxiliary models pretrained on instance segmentation or monocular depth estimation in a supervised manner can provide additional information for learning to model dynamical scenes, such as providing the region containing moveable objects or a depth prior for dynamical objects. In comparison, our method learns depth, independent motion, and motion segmentation solely from observing monocular videos without any additional pretrained models.

- ***labels during training***. With access to ground truth labels, the learning problem can, to some extent, rely on the supervised objective, which is much easier than our learning objective that is purely based on image reconstruction. In cases of ground truth depth labels (real or synthetic), similar to stereo-view during training, independent motion can be directly disambiguated given the known depth information.

- ***stereo-view during inference***. Scene geometry and depth information can be extracted via simple correspondence, which is a much easier setup compared to monocular input during inference.

- ***multi-frame during inference***. Additional consecutive frames would introduce much more information regarding scene geometry that is not retrievable from a single frame as input during inference.

Nevertheless, we provide the results for comparison in Table 8 (Rebuttal PDF) and we discuss the mentioned works further in the individual responses. 
Since previous works have been primarily focused and reported on KITTI, we report comparisons on KITTI, even though KITTI has fewer moving objects compared to Waymo and nuScenes (pointed out by *Reviewer EvaZ*, L237-L239 in manuscript and Figure 6 in supplementary materials). For additional qualitative results in dynamical scenes, please refer to our submitted supplementary materials containing 12 video visualizations of our model’s predictions.

We look forward to your response and we are happy to address any additional questions or concerns!

Meta-review: The paper received conflicting reviews, which could not be reconciled during the post-rebuttal discussion. Two (experienced) reviewers strongly support the paper, and a third one is also positive. On the contrary, two reviewers argue against the paper. The AC has read the paper and checked the discussion, and found that some of their negative arguments are not valid. Most importantly, the correct observation that "modelling independent motion and ego motion for depth estimation, flow estimation, and moving object estimation is not quite new" does not imply a lack of novelty: that the high-level ideas have appeared in other works could be said about the overwhelming majority of NeurIPS publications. Also the fact that foundation models are available does not imply that one must use them and that starting from "only" ImageNet pre-training is wrong. The AC concluded that while the contribution of the paper is not ground-breaking, it is sufficiently interesting to warrant publication.
Decision: Accept (poster)