SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
L _ {s} = \gamma_ {s d} l _ {s} \left(\boldsymbol {d} _ {t} ^ {*}, \boldsymbol {I} _ {t}\right) + \gamma_ {s c} l _ {s} \left(\boldsymbol {F} _ {\boldsymbol {C}}, \boldsymbol {I} _ {t}\right) + \gamma_ {s m} l _ {s} \left(\boldsymbol {M}, \boldsymbol {I} _ {t}\right) \tag {11}
$$
The motion consistency loss  $L_{c}$  in Eq. 12 computes the flow discrepancy  $F_{D}(p)$  between the complete scene flow  $F_{C}(p)$  and rigid flow  $F_{R}(p)$  for static pixels. The probability of pixel  $p$  being static is approximated by  $(1 - M(p))$ .
$$
L _ {c} = \frac {1}{H W} \sum_ {p} (1 - M (p)) \cdot F _ {D} (p), \quad F _ {D} (p) = \left\| \boldsymbol {F} _ {\boldsymbol {C}} (p) - \boldsymbol {F} _ {\boldsymbol {R}} (p) \right\| _ {1} \tag {12}
$$
The motion sparsity loss  $L_{m}$  in Eq. 13 considers all pixels with flow discrepancy  $F_{D}$  lower than the mean value over the frame as putative background, and encourages the motion mask to be 0 by using the cross entropy against label  $\vec{0}$ , which is denoted as  $g(\cdot)$ .
$$
L _ {m} = g \left(\left\{\boldsymbol {M} (p): \forall p \text {s . t .} F _ {D} (p) \leq \operatorname {m e a n} \left(F _ {D}\right) \right\}\right) \tag {13}
$$
Finally, the above-ground loss  $L_{g}$  in Eq. 14 penalizes projected 3D points below the estimated ground plane (via RANSAC) where  $d_t^g$  denotes the mean-normalized inverse depth of the ground plane.
$$
L _ {g} = \frac {1}{H W} \sum_ {p} \operatorname {R e L U} \left(d _ {t} ^ {g} (p) - d _ {t} ^ {*} (p)\right) \tag {14}
$$
4.3 Motion Initialization
Since the reconstruction loss is the main learning signal in the unsupervised approach, the gradient from the reconstruction loss between  $I_{t}$  and  $\hat{I}_{t}$  at pixel  $p_t$  would propagate through the computed sample coordinate  $\vec{p_s} \equiv K\hat{P_s} \equiv K(F_I(p_t) + F_R(p_t) + P_t)$ . Here, although it may be ideal to jointly train all models end-to-end from start to finish, in practice, updating the rigid flow vector  $F_{R}(p_{t})$  and the independent flow vector  $F_{I}(p_{t})$  jointly from the start would cause the learning problem to be ill-posed, since each component has the capacity to overfit and explain away the other's prediction. Due to the ambiguity between camera motion and object motion as discussed in Section 3, the image reconstruction error alone is not a strong enough supervisory signal to enforce the accurate prediction of all underlying sub-tasks jointly, as a family of solutions of depth and object motion would all satisfy the training objective, ranging from predicting the whole scene as static with incorrect depth to predicting the scene as moving objects on a flat canvas.
Here, we offer our key insight that a good initial estimate of motion segmentation  $M$  would allow the system to properly converge without arising degenerate solutions. As shown in Eq. 8,  $\hat{P}_s = M(p_t) \cdot F_C(p_t) + (1 - M(p_t)) \cdot F_R(p_t) + P_t$ . For any pixel  $p_t$ ,  $\hat{P}_s \approx F_C(p_t) + P_t$  if  $M(p_t)$  is near 1, and approximates  $F_R(p_t) + P_t$  if  $M(p_t)$  is near 0. From this, it is clear that a good motion mask would route the back-propagating gradient to the complete flow network  $\mathcal{C}$  via  $F_C$  if  $p_t$  is moving and route the gradient to the rigid motion networks  $\mathcal{D}$  and  $\mathcal{P}$  via  $F_R(p_t)$  if  $p_t$  is static.
Of course, the question is how one can initialize a good motion mask, since it requires identifying the moving objects. Ideally, moving objects would be identified as the pixels that are poorly reconstructed based on depth and ego-motion alone, but as discussed in Section 3, the depth network has the capacity to alter its depth predictions and reconstruct even the moving objects. Nevertheless, we observe that overfitting to the moving objects in this way is difficult, and only happens in later iterations. In earlier training iterations, reconstruction errors for dynamical pixels are ignored (Appendix B.5).
To ensure a good motion mask  $M$ , we first train a depth network assuming a static scene, but stop updates to the depth network at an early stage. We then initialize the independent motion networks  $\mathcal{C}$  and  $\mathcal{M}$  from a coarse depth sketch predicted by this frozen early-stage depth network. Thus, without access to any labels or auxiliary pretrained networks, we obtain an initialization for motion mask  $M$  that helps in disambiguating and accurately estimating the rigid flow  $F_{R}$  and independent flow  $F_{I}$ .
Dataset. While Dynamo-Depth is proposed for general applications where camera motion is informative for scene geometry, we focus our attention on the challenging self-driving setting, where the epipolar ambiguity and the statistical regularities mentioned in Section 3 that lead to erroneous depth predictions are highly prevalent. Specifically, we evaluate on three datasets - Waymo Open [34], nuScenes [3], and KITTI [9] with Eigen split [6].
On all datasets, we only use the unlabeled video frames for training; ground truth LiDAR depth is only used for evaluation. We evaluate both overall depth estimation accuracy, as well as accuracy computed separately for static scene and moving objects. For the latter evaluation, we use Waymo Open and nuScenes and identify moving objects by rectifying the panoptic labels with 3D box labels to obtain masks for static/moving objects. It is worth quantifying the number of moveable objects per frame in each dataset. Waymo Open has a mean of 12.12 with a median of 9. In nuScenes, the mean is 7.78 with a median of 7. In KITTI, due to lack of per-frame labels, we approximate using its curated object detection dataset, which only has a mean of 5.26 with a median of 4. Additional dataset information and distribution histograms are found in Appendix C.
Model and Training setup. The proposed method is trained on four NVIDIA 2080 Ti with a total batch size of 12 and an epoch size of 8000 sampled batches. Adam optimizer [20] is used with an initial learning rate of 5e-5 and drops to 2.5e-5 after 10 epochs. Motion Initialization lasts 5 epochs and takes place after the depth network and complete flow network have been trained for 1 epoch each. After initialization, the system is trained for 20 epochs, totalling approximately 20 hours. The hyperparameter values are the same for all experiments and are provided in Appendix B.1, along with the rest of the details of the model architecture. To demonstrate the generality of the proposed framework, we adopt two architectures for the depth network  $\mathcal{D}$ , one with a ResNet18 [14] backbone
Table 1: Depth evaluation on the KITTI (K), nuScenes (N), and Waymo Open (W) Dataset.  $IM$  stands for independent motion where  $\mathbf{x}$  denotes a lack of independent motion modeling.  $Sem$  indicates the amount of semantic information given during training, where 'm' indicates mask-level supervision and 'b' indicates box-level supervision. Manual replication with released code is indicated by  ${}^{ \dagger  }$  .
IM Sem D	Error metric (↓)	Accuracy metric (↑)
Abs Rel	Sq Rel	RMSE	RMSE log	δ < 1.25	δ < 1.252	δ < 1.253
Monodepth2 [10]	X	K	0.115	0.882	4.701	0.190	0.879	0.961	0.982
LiteMono [44]	X	K	0.101	0.729	4.454	0.178	0.897	0.965	0.983
Struct2Depth [4]	m K	0.141	1.026	5.290	0.215	0.816	0.945	0.979
Dyna-DM [32]	m K	0.115	0.785	4.698	0.192	0.871	0.959	0.982
Lee et al. [23]	b K	0.114	0.876	4.715	0.191	0.872	0.955	0.981
SGDepth [21]	m K	0.113	0.835	4.693	0.191	0.879	0.961	0.981
Boulahbal et al. [2]	m K	0.110	0.719	4.486	0.184	0.878	0.964	0.984
GeoNet [43]	K	0.155	1.296	5.857	0.233	0.793	0.931	0.973
EPC++ [27]	K	0.141	1.029	5.350	0.216	0.816	0.941	0.976
CC [31]	K	0.140	1.070	5.326	0.217	0.826	0.941	0.975
TrianFlow [47]	K	0.113	0.704	4.581	0.184	0.871	0.961	0.984
Li et al. [24]	K	0.130	0.950	5.138	0.209	0.843	0.948	0.978
RM-Depth [15]	K	0.107	0.687	4.476	0.181	0.883	0.964	0.984
Dynamo-Depth (MD2)	K	0.120	0.864	4.850	0.195	0.858	0.956	0.982
Dynamo-Depth	K	0.112	0.758	4.505	0.183	0.873	0.959	0.984
Monodepth2† [10]	X	N	0.425	16.592	10.040	0.402	0.723	0.837	0.887
LiteMono† [44]	X	N	0.419	15.578	9.807	0.449	0.720	0.831	0.879
Dynamo-Depth (MD2)	N	0.193	2.285	7.357	0.287	0.765	0.885	0.935
Dynamo-Depth	N	0.179	2.118	7.050	0.271	0.787	0.896	0.940
Monodepth2† [10]	X	W	0.173	2.731	7.708	0.227	0.797	0.930	0.968
LiteMono† [44]	X	W	0.158	2.305	7.394	0.210	0.816	0.944	0.976
Struct2Depth [4]	m W	0.180	1.782	8.583	0.244	-	-	-
Li et al. [24]	m W	0.157	1.531	7.090	0.205	-	-	-
Lee et al. [23]	b W	0.148	1.686	7.420	0.210	-	-	-
Li et al. [24]	W	0.162	1.711	7.833	0.223	-	-	-
Dynamo-Depth (MD2)	W	0.130	1.439	6.646	0.183	0.851	0.959	0.985
Dynamo-Depth	W	0.116	1.156	6.000	0.166	0.878	0.969	0.989
from Monodepth2 [10] and another with a CNN/Transformer hybrid backbone from LiteMono [44], denoted as Dynamo-Depth (MD2) and Dynamo-Depth, respectively. To be commensurate with baselines, the encoders are initialized with ImageNet [5] pretrained weights.
Metrics. Depth performance is reported via commonly used metrics proposed in [7], including 4 error metrics (Abs Rel, Sq Rel, RMSE, and RMSE log) and 3 accuracy metrics ( $\delta < 1.25$ ,  $\delta < 1.25^2$ , and  $\delta < 1.25^3$ ). We also report precision-recall curve for evaluating binary motion segmentation.
5.1 Monocular Depth Estimation
As shown in Table 1, our proposed approach outperforms prior arts on both nuScenes and Waymo Open across all metrics, with over  $57\%$  and  $21\%$  reduction in overall Abs Rel for nuScenes and Waymo Open, respectively. This improvement is significant: we repeat the experiment for Dynamo-Depth on Waymo Open 3 times and obtained a  $95\%$  confidence interval of  $0.119 \pm 0.003$  for Abs Rel and  $0.874 \pm 0.004$  for  $\delta < 1.25$ . This suggests the challenge and importance of estimating accurate depth for moving objects in realistic scenes. On KITTI, our approach is competitive with prior art. However, KITTI has fewer moveable objects, which better conforms to the static scene constraint and diminishes the importance in modeling independent motion. For instance, LiteMono [44], which does not model independent motion, demonstrates superior performance on KITTI but drastically underperforms in nuScenes and Waymo Open, with both datasets having many more moving objects.
To further evaluate the effectiveness of our approach in modeling independent motion, we split nuScenes and Waymo Open into static background, static objects and moving objects and evaluate depth estimation performance on each partition. As we adopt our depth model from Monodepth2 [10] and LiteMono [44], in Table 2, we compare our approach against the respective baseline with the same depth architecture. For simplicity, we report Abs Rel and  $\delta < 1.25$ . Notably, by explicitly modeling
Table 2: Depth evaluation with semantic split on the nuScenes (N) and Waymo Open (W) Dataset. S.B., S.O. and M.O. denotes the partition of pixels that are static background, static moveable object and moving object, respectively. Manual replication with released code is indicated by  $\dagger$ .
D	Abs Rel (↓)	δ < 1.25 (↑)
All	S.B.	S.O.	M.O.	All	S.B.	S.O.	M.O.
Monodepth2† [10]	N	0.425	0.447	0.232	0.418	0.723	0.735	0.704	0.570
Dynamo-Depth (MD2)	N	0.193	0.196	0.196	0.228	0.765	0.761	0.759	0.684
Δ (%)	-54.6	-56.2	-15.5	-45.5	+5.8	+3.5	+7.8	+20.0
LiteMono† [44]	N	0.419	0.431	0.248	0.502	0.720	0.734	0.718	0.517
Dynamo-Depth	N	0.179	0.184	0.182	0.198	0.787	0.781	0.774	0.753
Δ (%)	-57.3	-57.3	-26.6	-60.6	+9.3	+6.4	+7.8	+45.6
Monodepth2† [10]	W	0.173	0.152	0.215	0.749	0.797	0.810	0.683	0.416
Dynamo-Depth (MD2)	W	0.130	0.122	0.175	0.234	0.851	0.862	0.778	0.674
Δ (%)	-24.9	-19.7	-18.6	-68.8	+6.8	+6.4	+13.9	+62.0
LiteMono† [44]	W	0.158	0.140	0.179	0.599	0.816	0.827	0.733	0.506
Dynamo-Depth	W	0.116	0.110	0.155	0.194	0.878	0.891	0.812	0.750
Δ (%)	-26.6	-21.4	-13.4	-67.6	+7.6	+7.7	+10.8	+48.2
(a) Input Frames
Figure 3: Qualitative results of our proposed approach that learns from (a) unlabeled video sequences and predicts (b) monocular depth, (c) rigid flow, (d) independent flow, and (e) motion segmentation.
(c) Rigid Flow
(d) Independent Flow
(e) Motion Mask
independent motion, we observe a consistent and significant improvement on moving objects  $(M.O.)$  across both architectures and both datasets, with over  $48\%$  relative improvement in accuracy and over  $67\%$  relative reduction in error for both architectures on Waymo Open. Additionally, we see a substantial improvement on both static moveable objects  $(S.O.)$  and static background  $(S.B.)$  for both datasets and both architectures. Interestingly, the large reduction in error on static background in nuScenes does not match the corresponding improvement in accuracy, as the large errors on background mainly come from night-time instances where all methods fail, but differ in their respective artifacts (discussed further in Appendix D.2).
<sup>3</sup>All motion segmentation in (e) are uniformly visualized with a range of [0, 1], and corresponding video sequences that contain each example are found in https://dynamo-depth.github.io.
Table 3: Depth ablation results on the Waymo Open Dataset with Dynamo-Depth.
Lc	Motion Initialization	Lg	ImageNet Pretraining	Abs Rel (↓)	δ < 1.25 (↑)
All	S.B.	S.O.	M.O.	All	S.B.	S.O.	M.O.
X	X	X	X[2]	0.525	0.489	0.625	0.447	0.254	0.274	0.210	0.325
0.522	0.487	0.621	0.444	0.256	0.275	0.212	0.326
0.266	0.237	0.167	0.211	0.816	0.844	0.792	0.743
0.136	0.132	0.177	0.201	0.833	0.839	0.778	0.730
0.116	0.110	0.155	0.194	0.878	0.891	0.812	0.750
In sum, by explicitly modeling independent motion, we are able to outperform the respective baselines and achieve state-of-the-art performances on both Waymo Open and nuScenes Dataset. Qualitative results are found in Figure 1 and Figure 3, where "Static Scene Constraint" refers to performance of LiteMono [44] in Figure 1.
5.2 Motion Segmentation and Method Ablation
Figure 4 evaluates the quality of the jointly learned binary motion mask network using precision and recall. We observe a high precision with increasing recall, achieving a F1 score of  $71.8\%$  on Waymo Open [34]. Note that this segmentation quality is achieved without any labels. On nuScenes, Dynamo-Depth achieves a F1 score of  $42.8\%$  with  $57.2\%$  on day-clear conditions and  $21.1\%$  on all other conditions. Notably, the motion mask network is able to segment out small dynamical objects, such as the cyclist in the first row of Figure 3.
In addition, we show our ablation results on Waymo Open in Table 3. First, we found that ablating the motion consistency loss removes the regularization on the complete flow prediction, which causes the degenerate solution of moving objects on a flat canvas, as indicated by the poor segmentation performance in Figure 4. Similarly, by ablating motion
initialization, initial independent motion prediction lead to large reconstruction errors, which diverges depth training. In addition, we observe that abating the above ground prior preserves the performance of motion segmentation and depth of moving objects, but degrades the performance of depth on static background. Intuitively, this prior regularizes the structural integrity of the ground plane. Finally, we observed that the original set of hyperparameter values leads to training divergence when abating the ImageNet pretrained weights. To compensate, we increase the initial number of epochs for depth learning from 1 to 2 (marked as  $\pmb{x}^{[2]}$ ) without performing any hyperparameter sweeps.
Figure 4: Precision-Recall curve for motion segmentation on the Waymo Open Dataset.
In this work, we identify the effects of dynamical objects in unsupervised monocular depth estimation with static scene constraint. To mitigate the negative impacts, we propose to jointly learn depth, ego-motion, 3D independent motion and motion segmentation from unlabeled videos. Our key insight is that a good initial estimation of motion segmentation encourages joint depth and independent motion learning and prevents degenerate solutions from arising. As a result, our approach is able to achieve state-of-the-art performance on both Waymo Open and nuScenes Dataset.
Limitations and Societal Impact. Our work makes a brightness consistency assumption, where the brightness of a pixel will remain the same. Although common in optical flow, this assumption limits the our method's ability to model scenes with dynamical shadows, multiple moving light sources, and lighting phenomenons (e.g., lens flare). In addition, our work does not introduce any foreseeable societal impacts, but will generally promote more label-efficient and robust computer vision models.
This research is based upon work supported in part by the National Science Foundation (IIS-2144117 and IIS-2107161). Yihong Sun is supported in part by an NSF graduate research fellowship.
[45] Sen Zhang, Jing Zhang, and Dacheng Tao. Towards scale-aware, robust, and generalizable unsupervised monocular depth estimation by integrating imu motion dynamics. In European Conference on Computer Vision, pages 143-160. Springer, 2022. 3, 17
[46] Songchun Zhang and Chunhui Zhao. Dyna-depthformer: Multi-frame transformer for self-supervised depth estimation in dynamic scenes. arXiv preprint arXiv:2301.05871, 2023. 3, 17
[47] Wang Zhao, Shaohui Liu, Yezhi Shu, and Yong-Jin Liu. Towards better generalization: Joint depth-posed learning without posenet. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9151–9161, 2020. 3, 8
[48] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1851-1858, 2017. 1, 2, 3
[49] Yuliang Zou, Zelun Luo, and Jia-Bin Huang. Df-net: Unsupervised joint learning of depth and flow using cross-task consistency. In Proceedings of the European conference on computer vision (ECCV), pages 36-53, 2018. 3
A Visualizations
In Figure 5, we show qualitative comparison between LiteMono [44] and our Dynamo-Depth. We select absolute relative difference (Abs Rel in quantitative results) as the measure of error, as shown in Eq. 15 with predicted depth  $d$ , ground truth depth  $d^{*}$ , and pixel coordinate  $p$ .
$$
\operatorname {A b s} \operatorname {R e l} (p) = \left\{ \begin{array}{l l} | \boldsymbol {d} (p) - d ^ {*} (p) | / d ^ {*} (p), & \text {i f} p \text {i s a L i D A R p o i n t} \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {15}
$$
Figure 5: Qualitative comparison between LiteMono [44] and our proposed Dynamo-Depth. The input frame is shown in column (a). Column (b) and (c) show the depth predictions and column (d) and (e) show the absolute relative difference for each LiDAR point with a fixed error range of  $[0,1]$ .
B Implementational Details
B.1 Hyperparameters
Please refer to Table 4 for the hyperparameters used for training. The values are found via Waymo Open and kept the same for other datasets. The trade-off coefficient  $\alpha$  and depth smoothness coefficient  $\gamma_{sd}$  are set to 0.85 and 0.001, respectively, as in [10]. The motion smoothness coefficient  $\gamma_{sc}$  is set to 0.001 to be consistent with  $\gamma_{sd}$  and was not tuned further. The mask smoothness coefficient  $\gamma_{sm}$  is set to a much higher value of 0.1 to remove sporadic false positives induced by high frequency image features such as trees and buildings. Since both rigid motion  $F_{R}$  and complete motion  $F_{C}$  have relatively small magnitudes, we set the motion consistency coefficient  $\gamma_{c}$  to be 5.0 to properly enforce consistency. Due to the high loss values outputted by binary cross entropy, we set the mask sparsity coefficient to be 0.04 to accommodate. The above ground coefficient  $\gamma_{g}$  is set to 0.1 after searching across the values 0.01, 0.1, and 1.0.
B.2 Architectural Details
The depth network  $\mathcal{D}$  is adopted from two architectures, one with a ResNet18 [14] backbone from Monodepth2 [10] and another with a CNN/Transformer hybrid backbone from LiteMono [44], denoted
Table 4: List of hyperparameters used.
Hyperparameter	Value
α	0.85
γsd	0.001
γsc	0.001
γsm	0.1
γc	5.0
γm	0.04
γg	0.1
as Dynamo-Depth (MD2) and Dynamo-Depth, respectively. For both versions of our model, the pose network  $\mathcal{P}$  is adopted from Monodepth2 [10]. The encoders of the complete flow network  $\mathcal{C}$  and motion mask network  $\mathcal{M}$  are shared and have the same architecture as  $\mathcal{P}$ . To obtain per-pixel predictions, the features of the shared encoder are fed into the the separate decoders with the same architecture as the depth decoder in Monodepth2 [10] and with output per-pixel dimension of 3 and 1 for  $\mathcal{C}$  and  $\mathcal{M}$ , respectively.
For notational simplicity, Figure 2 only considers the case of a single source frame  $I_{s}$ , while in practice, Dynamo-Depth is trained with two source frames: one frame previous and one frame after the target  $I_{t}$ . Here, the two respective reconstructions are rectified via a minimal projection loss proposed in [10] to obtain  $\hat{I}_t$ . To enforce forward-backward consistency, the independent motion networks  $\mathcal{C}$  and  $\mathcal{M}$  predict a single independent flow field  $F_{I}$  that is used to warp both source frames into the target frame, with the appropriate sign change.
Following Monodepth2 [10] and LiteMono [44], we perform auto-masking to facilitate learning during depth initialization. Afterwards, auto-masking is turned off as the apparently static pixels may belong to dynamical objects traveling with the ego-camera, in which case auto-masking would mask out the corresponding reconstruction loss essential for independent motion learning.
There is a linear weight ramp for all loss terms that include  $F_{C}$  and  $M$  at the beginning of each training stage, with the length of 2666 iterations for Waymo and nuScenes, and 8000 iterations for KITTI due to the its limited number of moving objects.
B.4 Ground Plane Estimation
Concretely, 3D projected points  $(P_{t}$  in Eq. 4) belonging to the bottom half of the image are sampled to construct the ground plane  $d_t^g$  via RANSAC (5 points are selected per iteration, for a total of 100 iterations). This assumption does break under cases of traffic jams, but due to the rare occurrence of these scenarios for this learning task assuming a moving camera, the negative effects are limited.
B.5 Difficulties in Overfitting to Dynamical Objects
In Figure 6, we show a qualitative example where a depth network learned with static scene constraint (LiteMono [44] in this case) would ignore the high reconstruction loss in earlier iterations in (b) and overfit to the moving objects via erroneous depth predictions only in later iterations in (c)-(f).
(b) After 1 Epoch
(c) After 2 Epochs
Figure 6: Visualization of the overfitting phenomenon as shown via LiteMono [44], where erroneous depth for the black SUV and the oncoming car is only learned in later iterations, in (c)-(f).
(d) After 3 Epochs
(e) After 5 Epochs
(f) After 10 Epochs
We provide additional information of the three datasets used for evaluation.
For the Waymo Open Dataset [34], 76,852 front camera image-triplets from the provided train set containing 798 video sequences are used for training while 2,216 front camera images uniformly sampled from the provided validation set containing 202 video sequences are used for evaluation. During training, frames are downsampled to  $480 \times 320$ . We rectify the panoptic labels [29] with 3D box labels to obtain the corresponding masks for static and moving objects.
For the nuScenes Dataset [3], 79,760 front camera image-triplets from the provided train set containing 700 video sequences are used for training while 6019 front camera images from the official validation set containing 150 video sequences are used for evaluation (of which 4449 are day-clear). Following [8], we consider a sequence to be in the day-clear subset if its description does not contain "night" or "rain". During training, frames are downsampled to  $512 \times 288$ . We rectify the panoptic LiDAR labels with annotated 3D bounding boxes to obtain motion attributes for LiDAR points.