Deconstructing Self-Supervised Monocular Reconstruction: The Design Decisions that Matter
Authors: Jaime Spencer
Chris Russell
Simon Hadfield
Richard Bowden
Venue: Accepted by TMLR ()

Abstract: This paper presents an open and comprehensive framework to systematically evaluate state-of-the-art contributions to self-supervised monocular depth estimation. This includes pretraining, backbone, architectural design choices and loss functions. Many papers in this field claim novelty in either architecture design or loss formulation. However, simply updating the backbone of historical systems results in relative improvements of 25%, allowing them to outperform most modern systems. A systematic evaluation of papers in this field was not straightforward. The need to compare like-with-like in previous papers means that longstanding errors in the evaluation protocol are ubiquitous in the field. It is likely that many papers were not only optimized for particular datasets, but also for errors in the data and evaluation criteria. To aid future research in this area, we release a modular codebase (https://github.com/jspenmar/monodepth_benchmark), allowing for easy evaluation of alternate design decisions against corrected data and evaluation criteria. We re-implement, validate and re-evaluate 16 state-of-the-art contributions and introduce a new dataset (SYNS-Patches) containing dense outdoor depth maps in a variety of both natural and urban scenes. This allows for the computation of informative metrics in complex regions such as depth boundaries. 

Published in Transactions on Machine Learning Research (12/2022)
Deconstructing Self- Monocular Reconstruction: The Design Decisions that Matter
Jaime Spencer
ixpencermartnndssrree.....uu
CVSSP, University of Surrey
Chris Russell
cmruss@amazon..co
Amazon
Simon Hadfield
s. hadffeldüssrreey.ccc.. uk
CVSSP, University of Surrey
Richard Bowden
www.den@ssrree..ccooo
CVSSP, University of Surrey
Reviewed on OpenReview: https: //
// operreveew.eeee... net/ forum? id- GFK1Fhe87F
Abstract
This paper presents an open and comprehensive framework to systematically evaluate state- of-the-art contributions to self-supervised monocular depth estimation. This includes pre- training, backbone, architectural design choices and loss functions. Many papers in this field claim novelty in either architecture design or loss formulation. However, simply up- dating the backbone of historical systems results in relative improvements of 25%, allowing them to outperform most modern systems. A systematic evaluation papers this field was not straightforward. The need to compare like-with-like previous papers means that longstanding errors in the evaluation protocol are ubiquitous in the field. It is likely that many papers were not only optimized for particular datasets, but also for errors in the data and evaluation criteria. To aid future research in this area, release a mod- benchmark), allowing for easy
ular codebase (https://github. http:/gttthuubccomjjsssss ccom:/sppeaar//mooooeeeeoo evaluation of alternate design decisions against corrected data and evaluation criteria. We re-implement, validate and ro-evaluate 16 state-of-the--e tho-art contributions and introduce a new datasel (SYNS-Patches) containing dense outdoor depth maps in variely of both natural and urban scenes. This allows for the computation of informative metries in complex regions such as depth boundaries.
1 Introduction
Depth estimation fundamental low-level computer vision task that allows us to estimate the 3-D world from its 2-D projection(s). It is a core component cnabling mid-level tasks such as SLAM, visual localization object detection. More recently, it has heavily impacted fields such as Augmented Reality, aulonomous vehicles and robotics, knowing the real-world geometry of scene is crucial for interacting with both virtually and physically.
This interest. has resulted in a large influx of researchers hoping to contribute to the field and compare with previous approaches. New authors are faced with a complex range of established design decisions, such as the choice of backbone architecture & pretraining, loss functions and regularization. This further complicated by the fact that papers are written be accepted for publication. As such, they often emphasize the theoretical novelty of their work, over the robust. design decisions that have the most impact on performance.

---

Published in Transactions on Machine Learning Research (12/2022)
Aboltel Improvament (S0
0.20
Hazettd
Abstte
0.72
Honotopers5
Kumistee'2
Deinsts Aostrt
paintiinn
Festherts raco
caneuto (211 comeer
desg
Honotoytt Fumiitey322
Destutt Reet334
Slsedeentinn
Horott Mesodeeiii resturt oo
Aesthppt (210
(207
Calyore 2212
carnec FPA
ret meldngpoe
expite poteen
(a) Sot.A Performance per Baseline
(b) SotA Improvement per Baseline
Abultel Imprevement (N)
0.12
AboRel
0.31
0.59
0.06
NOMM 10
Basher 151
Auttec.ia P
Busienjo Pf
Anste stit
natter opta
Govtent
Peskee 18
Ruster 181
Nateeda Pr
Comvieti
Gentent
Rumienn
Backhone Performance
Backbone Improvement
Figure 1: Quantifying Sot Contributions. (a) Performance at the time es. our re-implementation using common design decisions (lower is better). Kuznictsov et al. (2077) and SIM-Lerruer (Zhou et al. 2017) have much higher crror as they do not use stereo data training. (b) Original papers AbsRel relative performance improvement w.r.t. Garg et al. (206) us. real improvements observed obtained by ablating the backbone architecture.
reaainn//valuttnng baselines in a fair and comparable manner (higher is better). (c) Performance
(d) Backhone relative performance improvements (w.r.t. ResNet-18 from
scratch) outweigh those provided by most recent contributions. See section Section 4.2 for more details of design choices.
This paper offers a chance to step back and re-evaluate the state self-supervisod monocular depth estima- tion. We do this via an exlensive baseline study by carefully re-implementing popular Slate-of-the-Ar Art (SoLA) algorithms from scratct Our modular codebase allows to study the impact of each framework component and ensure all approaches are trained in fair and comparable way.
Figure D compares the performance reported by recent SotA against that obtained by the same technique our updated benchmark. Our reimplementation improves performance for all evaluated baselines. However, the relative improvement resulting from each contribution is siguificantly lower than that reported by the original publications. In many cases, is likely this is the result of arguably unfair comparisons against outdated baselines. For instance, as seen Figures Tc & ld simply modernizing the choice of backbone in these legacy formulations results performance gains of 25%. When applying these common design decisions to all approaches, it appears that 'legacy' formulations are still capable of outperforming many recent methods.
As part of our unified benchmark, we proposc novel evaluation dataset in addition to the exclusively used urban driving datasets. This new dataset (SYNS-Patches) contains 1175 images from a wide variely of urban and natural scenes, including categories such as urban residential, woodlands, indoor, industrial and more. This allows us to evaluate the generality of the learned depth models beyond the restricted automotive domain that is the focus of most papers. To summarize, the contributions of this paper are:
1. We provide modular codebase containing modernized baselines that easy to train and extend. This encourages direct like-with-like comparisons and better research practices.
2. We ro-evaluate the updated basclines algorithms consistently using higher-quality corrected ground- Iruth on the existing benchmark dataset. This pushes the field away from commonly used flawed benchmarks, where errors are perpetuated for the sake of compatibility.
Code publicly available https://githinn httpp:///gttuu...cc//eseaaa/a// http:////ggggeee//oooo//ttoeeeea benchnark
2

---

Published in Transactions on Machine Learning Research (12/2022)
3. In addition to democratizing code and evaluation the common Kitti benchmarks, propose novel lesting dataset (SYNS-Patches) containing both urban and natural scenes. This focuses on the ability to generalize to a wider range of applications. The dense nature of the ground-truth allows us to provide informative metrics in complex regions such as depth boundarics.
4. We make these resources available to the wider research community, contributing to the further advancement of self-supervised monocular depth estimation.
2
Related Work
We consider self-supervised approaches that do not use ground-truth depth data at training time, but instead learn to predict depth as a way to estimate high-fidelity warps from one image to another. While all approaches predict depth from a single image, they can be categorized based on the additional frames used to perform these warps. Stereo-supervised frameworks directly predict metric depth given known stereo baseline. Approaches using only monocular video additionally need to estimate Visual Odometry (VO) and only predict depth up to an unknown scale. These depth predictions are scaled during evaluation and aligned with the ground-truth. However, monocular methods are more flexible, since they do not require a stereo pair. Despite having their own artifacts, they do not share stereo occlusion artifacts, making video a valuable cue complementary stereo.
2.1 Stereo
Garg et al. 2016) introduced view synthesis as a proxy task for self-supervised monocular depth estimation. The predicted depth map was used synthesize the target view from its stereo pair, and optimized using Ln reconstruction loss. An additional smoothness regularization penalized all gradients in the predicted depth. Monodepth (Godard et al. 2017) used Spatial Transformer Networks (Jaderberg et al. 2015 to per- form view synthesis in fully-differentiabbe manner. The reconstruction loss was improved via weighted L and SSIM (Wang et al. 2004) photometric loss, while the smoothness regularization was softened in regions with strong image gradients. Monodepth additionally introduced a virtual stereo consistency term, forcing the network to predict both left and right depths from a single image. 3Net (Poggi et al. 2018 extended Monodepth to a trinocular setting by adding an extra decoder and treating the input as the central im- age in three-camera rig. SuperDepth (Pillai et al. 2019) replaced Upsample-Com Conv blocks with sub-pixel convolutions (Shi et al. 2016), resulting in improvements when training with high-resolution images. They additionally introduced a differentiable stereo blending procedure based on test-time stereo blending (Go- dard et al. 2017). FAL-Net (Gonzalez Bello & Kim 2020) proposed a discrete disparity volume network, complemented by a probabilistic view synthesis module and an occlusion-aware reconstruction loss.
Other methods complemented the self-supervised loss with (proxy) ground-truth depth regression. Kuzni- etsov et al. (2017) introduced a reverse Huber (berHu) regression loss (Zwald & Lambert-Taccoix 2012 Laina 2016 using the ground-truth sparse Light Detection and Ranging (LiDAR). Meanwhile, SVSM Luo et al.
et al.]
2018 proposed two-stage pipeline in which virtual stereo view first synthesized from sel sell-supeviied depth network. The larget image and systhesized view were then processed in stereo matching cost volume trained on the synthetic Flying[Things3] dataset (Mayer et al. 2016). Similarly, DVSO (Rui et al. 2018) and MonoResMatch (Tosi et al. 2019) incorporated stereo matching refinement networks to predict a residual disparity. These approaches used praxy ground-truth -ruth depth maps from di- rect stereo Simultancous Localization and Mapping (SLAM) [Wang et al.] (017) and hand-crafted stereo matching (Hirschumülerr) 2005), respectively. DVSO (Rui et al.) 20018) introduced an occlusion regulariza- tion, encouraging sharper predictions that prefer background depths and second-order disparity smoothness. DepthHints (Watson et al.) 2019) introduced proxy depth supervision into Monodepth2 (Godard et all 2019), also obtained using SGM (Hirschüüller) 2005). The proxy depth maps were computed as the fused minimum reconstruction loss from predictions with various hyperparaneters. As with the automasking pro- cedure of Monodepth2 [Godard et al.) 2019), the proxy regression loss was only applied to pixels where the hint produced lower photometric reconstruction loss. Finally, PLADE-Net (Gonzalez Bello & Kim (021) expanded FAL-Net (Gonzalez Bello & Kim 2020) by incorporating positional encoding and proxy depth regression using a matted Laplacian.
3

---

Published in Transactions on Machine Learning Research (12/2022)
2.2 Monocular
SIM-Learner (Zhou et al. 2017) introduced the first approach supervised only by stream of monocular images. This replaced the known stereo baseline with an additional network to regress VO between consec- utive frames. An explainability mask was predicted to reduce the effect of incorrect correspondences from dynamic objects and occlusions. Klodt & Vedaldi (2018) introduced uncertainty (Kendall & Gal 2017) alongside proxy SLAM supervision, allowing the network to ignore incorrect predictions. This uncertainly formulation also replaced the explainability mask from SfM-Learner. DDVO (Wang et al. 2018 introduced differentiable DSO module (Engel al.) 20018) to refine the VO network prediction. They further made the observation that the commonly used edge-aware smoothness regularization (Godard et al. 2017 suffers from
degenerate solution in monocular framework. This was accounted for by applying spatial normalization
prior to regularizing.
Subsequent approaches focused on improving the robustness of the photometric loss. Monodepth2 (Godard et al. 2019) introduced several simple changes to explicitly address the assumptions made by the view synthesis framework. This included minimum reconstruction filtering to reduce occlusion artifacts, alongside static pixel automasking via the raw reconstruction loss. D3VO (Yang et al., 2020) additionally predicted affine brightness transformation parameters (Engel et al. 2018) for each support frame. Meanwhile, Depth- VO-Feat (Zhan et al., 2018) observed that the photometric loss was not always reliable due to ambignous matching. They introduced an additional Seature-based reconstruction loss synthesized from a pretrained dense feature representation (Weerasekera et al. 2017). DeFeat-NNtt (Spencer et al.) 2020) extended this concept by learning dense features alongside depth, improving the robustness to adverse weather conditions and low-light environments. Shu et al. (2020) instcad trainod an autoencoder regularized to learn discrimative features with smooth second-order gradients.
Mahjourian et al. (2018) incorporated explicit geometric constraints via Iterative Closest Points using the predicted aligmment pose and the mean residual distance. Since this process was non-differentiable, the gradients were approximated. SC-SIM-Learner (Bian et al. 2019) proposed an end-to-end differentiable geometric consistency constraint by synthesizing the support depth view. They included a variant of the absolute relative loss constrained to the range [0, 1], additionally used as automasking for the reconstruction loss. Poggi et al.] (2020) compared various approaches for estimating uncertainty in the depth prediction, including dropout, ensembles, student-teacher training and more. Johnston & Carneiro (2020) proposed to use a discrete disparity volume and the variance along each pixel to estimate the uncertainty of the prediction. During training, the view reconstruction loss was computed using the Expected disparity, i.e. the weighted sum based on the likelihood of each bin.
Recent approaches have focused on developing architectures to produce higher resolution predictions that do not suffer from interpolation artifacts. PackNet (Guizilini et al., (2200) proposed an encoder-decoder network using 3-D (un)packing blocks with sub-pixel convolutions (Shi et al., (0066). This allowed the network to encode spatial information in an invertible manner, used by the decoder to make higher quality predictions. However, this came at the cost of a tenfold increase in parameters. CADepth (Yan et al. 2021 proposed a Structure Perception self-attention block as the final encoder stage, providing additional context to the decoder. This was complemented by a Detail Emphasis module, which refined skip connections using channel-wise attention. Mcanwhile, Zhou et al.] (2021) replaced the commonly used ResNet encoder with HRNet due to its suitability for dense predictions. Similarly, concatenation skip connections were replaced with a phannel/spatial attention block. HR-Depth (Lyu et al. 2021) introduced a highly efficient decoder based on SqueezeExcitation blocks (Hu et al. 2020) and progressive skip connections.
While each of these methods reports improvements over previous approaches, they are frequently not di- reetly comparable. Most approaches differ in the number of training epochs, pretraining datascts, backbone architectures, post-processing, image resolutions and more. This begs the question as to what percentage of the improvements are due to the fundamental contributions of each approach, and how much is unacoountd for in the silent changes. In this paper, we aim to answer this question by first studying the effect of chang- ing components rarely claimed as contributions. Based on the findings, we train recent SotA methods in a comparable way, further improving their performance and evaluating each contribution independently.
4

---

Published in Transactions on Machine Learning Research (12/2022)
(a) Iuput image
(b) Raw GT
(c) Improved G
Figure 2: Inaccurate Touund-Truth. The original Kitti Geiger al 2013 ground-truth data used by previous monocular depth benchmarks (Eigen & Fergus 2015) is inaccurate and contains crrors, especially at object boundaries. Note the back-
ground bleoding in the highlighted region.
Uhrig et al.
2018 correct this by accumulating LiDAR data over multiple frames.
0.10
Eigan
Eigen Benchmark
0.05
0.08
0.06
0.00
Donsty
0.03
0.04
0.04
0.02
0.02
0.02
0.00
Depth
Figure Depth Distribution. show the distribution of depths KE, KEB SYNS-Patchos. SYNS-Patches contains more varied depth values due to the indoors scenes. all cases the maximum depth clamped to 100 meters during evaluation.
3 Benchmark Datasets
The objective of this paper is to critically examine recent SotA contributions to monocular depth learning. One of the biggest hurdles to overcome is the lack of informative benchmarks due to erroneous evaluation procedures. By far, the most common evaluation dataset the field is the Kitti Eigen (KE) split (Eigen) & Fergus 2015). Unfortunately, has contained critical errors since its creation, the most egregious being the inaccuracy of the ground-truth depth maps. The original Kitti data suffered from artifacts due to the camera and LiDAR not being identically positioned. As such, each sensor had slightly different viewpoints, each with slightly different occlusions. This was exacerbated by the sparsily of the LiDAR, resulting in the background bleeding into the foreground. An example of this can be seen in Figure 2 Furthermore, the conversion to depth maps omitted the transformation to the camera reference frame and used the raw LiDAR depth instead. Finally, the Squared Relative error was computed incorrectly without the squared term in the denominator.
Although these errors have been noted by previous works, they have nevertheless been propagated from method to method up to this day due to the need to compare like-with-lik when reporting results. It is much easier to simply ignore these errors and copy the results from existing papers, than it is to correct the evaluation procedure and re-evaluate previous baselines. We argue this behavior should be corrected immediately and provide an open-source codebase to make the transition simple for future authors. Our benchmark consists of two datasets: the Kitti Eigen- Benchmark (KEB) split (Uhrig et al. 2018 and the newly introduced SYNS-Patches datasct.
3.1 Kitti Eigen-Benchhard Benchmark
Chrig et al. (2018) aimed to fix the aforementioned errors in the Kitti (Geiger et al. 2013) dataset. This was done by accumulating LiDAR data over +5 frames create denser ground-truth and removing occlusion artifacts via SGM (Hirschüüller) 2005) consistency checks. The final KEB split represents the subset of KE with available corrected ground-truth depth maps. This consists of 652 of the original 697 images.
5

---

Published in Transactions on Machine Learning Research (12/2022)
Figure 4: SYNS-Patches. Top: Diverse testing images. Middle: Donse depth maps. Bottom: Log-depth Canny edges.
Table SYNS-Patches Scenes. show the distibution of images per scene the proposed dataset. This evaluates the
model's capability generalize beyond purely automotive data.
Agriculture
Natural
Woodland
Residential
Industry
Misc
Recreation
Transport
Total
315
183
148
144
123
107
72
62
21
1,175
Similarly, we report the well-established image-based metries from the official Kitti Benchmark. While some of these overlap with KE, we avoid saturated metrics such as < 1.25 or the incorrect SqRel errof] We also report pointcloud- based reconstruction metrics proposed by Örnek et al. (2022). They argue that image- basod depth metries are insufficient to accurately evaluate depth estimation models, since the true objective is reconstruet the 3-D scene. Further delails can be found in Section B.1 Nevertheless, the Kitti dalaset is becoming saturated and more varied and complex evaluation framework is required.
3.2 SYNS-Patches
The second dataset in our benchmark is the novel SYNS-Patches, based SYNS (Adams et al. 2016). The original SYNS is composed of 92 scenes from 9 different categories. Each scene contains a panoramic HDR image and an aligned dense LiDAR scan. This provides a previously unseen dataset to evaluate the generalization capabilities of the trained baselines. We extend depth estimation to a wider variety of environments, inchuding woodland & natural scenes, industrial estates, indoor scenes and more. SYNS provides dense outdoor LiDAR scans. Previous dense depth datasets (Koch et al. 2018) are typically limited to indoor scenes, while outdoor datasets (Geiger et al. 2013: Guizilini et al.] 2020) are sparsc. Those dense depth maps allow us to compute metrics largeting high-intersst regions such as depth boundaries.
Our dataset is generated by sampling 18 undistorted patches per scene, performing full horizontal rotation roughly at eye level. To make this dataset more amenable to the transition from Kitti, we maintain the same aspect ratio and extract patches of size 376 1242. We follow the same procedure on the LiDAR to extr aligned dense ground-truth depth maps. Ground-truth depth boundaries are obtained via Canny edges in the dense log-depth map. After manual validation and removal of data with dynamic object artifacts, the final test set contains 1,175 of the possible 1,656 images. We show the distributions of depth values in Figure 3 the images per category in Table 1 and some illustrative examples in Figure 4 For each dataset, we additionally compute the percentage of image pixels with ground-truth depth values. The original Kitti Eigen has a density of only 4.10%, while the improved Kitti Eigen- Benchmark has 15.28%. Mcanwhile, SYNNSPPtcches has ground-truth for 78.30% of the scene, demonstrating the high-quality of the data.
We report image-based metrics from Uhrig al. (2018) and pointcloud- metrics from Örnek et al. (2022). For more granular results, compute these metries only al. depth boundary pixels. Finally, compute the edge- accuracy and completeness from Koch et al. (2018) using the Chamfer pixel distance to/from predicted and ground-truth depth edges. Further datasct crcation details can be found in B.3 Note that SYNS-Patches is purely a lesting dataset never used for training. This represents completely unseen environments that test the generalization capabilities of the trained models.
Section
The squared was missing from the denominator.

---

Published in Transactions on Machine Learning Research (12/2022)
4
The Design Decisions That Matter
Most contributions to sclf-supervised monocular depth cstimation focus on altcrations to the view synthesis loss GGdardd et al.
2017
2019 Zhan et al.] 2018), additional geometric consistency (Mahjourian et al. 2018
regularization (Rui al. 2018 and the introduction of proxy supervised losses (Kuzni- etsov et al. 2017 Tosi et al. 2019 Watson et al. 2019). In this paper, return to first principles and study the effect of changing components in the framework rarely claimed as contributions. We focus on practical changes that lead to significant. improvements, raising the overall baseline performance and providing a solid platform on which to evaluate recent SotA models.
Bian al. 2019)
Since this paper primarily focuses on the benchmarking and evaluation procedure, we provide a detailed review of monocular depth estimation and each contribution as supplementary material in Section A We encourage readers to the field to refer to this scction for additional details. It is worth reiterating that the depth estimation network only ever requires a single image as its input, during both training and evaluation. However, methods that use monocular video sequences during training require an additional relative pose regression network. This replaces the known fixed sterco baseline used by sterco-trained models. Note that this pose network is only required during training to perform the view syuthesis and compute the photometric loss. As such, it can be discarded during the depth evaluation. Furthermore, since all monocular- trained approaches use the same pose regression system, it is beyond the scope of this paper to evaluate the performance of this component.
4.1
Implementation details
We train these models the common Kitti Eigen-Zhou (KEZ) training split (Zhou et al. 2017), D, containing 39,810 frames from the KE split where static frames are discarded. Most previous works perform their ablation studies on the KE test set, where the final models are also evaluated. This indirectly incorporates the testing data into the hyperparameter optimization cycle, which can lead to overfitting to the test set and exaggerated performance claims. We instead use a random set of 700 images from the KEZ validation split with updated ground-truth depth maps (Uhrig et al. 2018). Furthermore, we report the amage-basch metries from the Kitti Benchmark and the pointcloud-based metrics proposed by Örnek et al. (2022), as detailed in Section B.2 For ease of comparison, Image-based ordering uses AbsRel, while pointeloud-based uses the F-Score.
add the performance rank ordering of the various methods.
Models were trained for 30 epochs using Adam with a base learning rate of 1e-4, reduced to le-5 halfway through training. The default DepthNet backbone a pretrained ConvNeXt-T (Liu et al.) 2022), while PoseNet uses a pretrained ResNet-18 (He et al.) 2016). We use an image resolution of 192 x 640 with batch size of 8. Horizontal flips and color jittering are randomly applied with a probability of 0.5.
We adopt the minimum reconstruction loss and static pixel automasking losses from Monodepth2 Godard et al. 2019). due to their simplicity and effectiveness. We use edge-aware smooth regularization Godard et al. 2017 with a scaling factor of 0.001. These losses are computed across all decoder scales, with the intermediate predictions upsampled to match the full resolution. We train in a Mono+ sctting, using monocular video and stereo pair support frames. To account for the inhereutly random optimization procedure, each model variant is trained using three random seeds and mean performance is reported. We emphasize that the training code has been publicly released alongside the benchmark code ensure the reproducibility of our results and to allow future rescarchers to build off them.
4.2 Backbone Architecture & Pretraining
Here evaluate the performance of recent SotA backbone architectures and their choice of pretraining,
Results can be found in Table 2 & Figure 5 We test ResNet (He et al., 2016). ResNeXt (Xic et al.
2017j, MobileNet-v3 (Howard et. al. 2019), EfficientNet (Tan & Le 2019)), HRRNet (Wang el al. 2021 | and
ConvNeXt (Liu et al. 2022). ResNet variants are trained either from scratch or using pretrained supervised weights from Wightman (2019). ResNeXtt100 variants additionally include fully supervised (Wightman) 2019),
and self-supervised (aalniz et al.] 2019) weights. All remaining backbones are pretrained by default.
7

---

Published in Transactions on Machine Learning Research (12/2022)
Table 2: Backhone Ablation. We study the effect of various backbone architectures & pretraining methods. PT indicates of pretrained ResNet weights. All other baselines pretrained by default. ConvNeXt provides the best performance, followed by HRNet-W66d. ResNeXt be further improved by using self- /eeakyy-superriee weights. Frames per second measured NVIDIA GeForce RTX 3090 with image of size 192 640.
Image- based
Pointcloud bassed
KEZ (val)
MParam
FPSt
#
MAEJ
RMSE4
AbeRely
LogS14
#
Chamfer
F-Scoref
IoUt
ResNet-18
3.38
16
50.71
ResNet-18 PT
14.33
238.2
1.23
3.06
6.29
9.03
10
0.49
54.35
38.79
ResNet-101
51.51
79.63
13
1.34
3.27
6.76
10.33
12
0.53
53.53
37.80
ResNet-101 PT
51.51
79.72
7
1.16
2.95
5.71
8.94
0.47
57.25
41.35
ResNeXt-101
95.76
72.07
1.29
3.17
6.55
9.74
11
0.51
53.80
38.16
ResNeXt-101 PT
95.76
71.98
1.09
2.64
5.74
7.77
0.44
55.10
39.55
ResNeXt-101 SSL
95.76
71.50
1.08
2.62
5.65
7.74
55.56
ResNeXt-101 SWSL
56.40
MobileNet-v3-S
2.20
158.3
17
1.95
1.27
10.25
14.74
17
0.76
45.13
30.35
MobileNet-v3-LL
6.67
137.8
1.27
3.05
6.72
9.00
13
0.51
52.87
37.46
EftiientNet-B0
5.84
99.35
1.29
3.04
6.95
8.90
14
0.52
52.14
37.05
EffiiientNetB11
19.41
54.28
15
1.33
3.11
7.20
9.22
15
0.53
51.77
36.83
HRNet-W18
16.05
29.65
1.14
2.81
5.86
8.17
55.34
39.76
HRt
122.81
57.05
CouvNeXt-T
31.3
147.5
3
1.03
2.39
5.30
7.63
0.43
57.97
12,26
ConvNeXt-B
92.65
98.09
1
0.97
2.49
4.98
7.40
0.40
60.63
44.89
ConvNeXt-L
203.27
72.44
2
0.98
5.19
7.07
0.41
58.17
42.49
40
F-Score
improvement (%)
35
LogSI
30
AbsRel
25
MAE
20
15
10
5
ResNet-18
ResNet-101
ResNet-18 PT
ResNet-101 101 PT
ResNext-101 PT
ResNext-101 SWSL
HRNet-W64
ConvNeXt-B
ConvNeXt-L
Figure Backbone Ablation. We show the relative performance improvement in F-Score, MAE, LogSI and AbsRtll obtained by different backbone architectures and pretraining methods. Most existing papers limit their backbone pretrained ResNet- resulting limited improvements. Full results in Table 2
As seen in Table 2 ConvNeXt variants outperform all other backbones, with HRNeI-WWG following closely behind. Within lighter backbones, i.e. 20 MParams, we find HRNet- W11 to be the most effective, greatly outperforming mobile backbones such MobileNet-v3 and EfficientNet. Regarding pretraining, all ResNet backbones show significant improvements when using pretrained weights. ResNeXt variants are further improved by using pretrained weights from self-supervised weakly-supervised training (Yalniz et al. 2019

---

Published in Transactions on Machine Learning Research (12/2022)
A summary of these results can be found in Figure 5 showing the relative improvement over ResNet-18 (from scratch) in F-Score, MAE, LogSI and AbsRel metries. The vast majority of monocular depth approaches simply stop at pretrained ResNet-18 backbone (~20% improvement). However, replacing this with well- engineered modern architecture such as ConvNeXt or HRNet results in an additional 15% improvement. As such, the remainder of the paper will use the Con-N backbone when comparing baselines.
4.3 Depth Regularization
The second study evaluates the performance of various commonly used depth regularization losses. We focus on variants of depth spatial gradient smoothing such as first-order (Garg et al. 2016),, edge-aware (Godard
et al. 2017], second-order (Rui et al., 2018) and Gaussian blurring. We additionally Lest two variants of occlusion regularization (Rui et al. 2018). favoring either background or foreground depths.
Results are shown in Table 3 We evaluate these models on the proposed KEZ validation split, as well as the KE test set commonly used by previous papers. Once again, the KEZ split is used to limit the chance of overfitting to the target KEB and SYNS-Patches test sets. When evaluating the updated ground-truth, using no additional regularization produces the best results. All variants of depth smoothness produce slightly inferior results, all comparable to each other. Incorporating occlusion regularization alongside smoothness regularization again leads to a decrease in performance. Mcanwhile, on the inaccurate ground-truth (Eigen & Fergus 2015), smoothhess constraints produce slightly better results. We believe this is due to the regularization encouraging oversmoothing in boundaries, which mitigates the effect of the incorrect ground- truth boundaries shown in Figure 2 As such, this is overfitting to errors in the evaluation criteria, rather than improving the actual depth prediction. Mcanwhile, the large overlapping receptive fields of modern architectures such as ConvNeXt are capable of implicitly providing the smoothness required by neighbouring depths. Once again, these results point towards the need for an up-to-date benchmarking procedure that is based on reliable data and more informative metrics.
Table 3: Depth Regularization Ablation. We study the effect of adding smoothness (edge-aware, trat-/second-order w/wo Gaussian blurring) and occlusion regularization (prefer background /fooegrroudd.. background/foreground)) Whilst all methods typically use these regularizations, omitting them provides the hest performance when evaluating the corrected ground-truth. Meanwhile, these regularizations provide slight improvements the outdated Kitti Figen split. This is likely due to oversmonthing to account. for the inaccurate boundaries shown in Figure
Image- based
Pointeloud- based
KEZ (val)
#
MAF4
RMSF4
AbsRe44
LogS14
#
Chamfer4
F-Scoref
ToUt
No Regularization
1.63
3.68
7.86
11.35
0.66
50.25
34.68
First-order
3
1.65
3.64
8.12
11.32
6
0.67
49.30
33,90
Fist-order Blur
3.66
8.19
11.36
4
0.67
49.39
33.96
Second-order
2
11.25
0.67
49.32
33.93
Second-order Blur
1.65
3.66
8.14
11.27
2
0.67
19.46
31.05
Occlusion (BG)
7
1.66
3.65
8.27
11:38
7
0.68
48.86
33.52
Occlusion (FG)
1.65
3.65
8.19
11.23
3
0.67
49.40
34.02
KE (test)
#
AbsRtl4
SqRtel
RMSEL
LogRMSE4
< 1.25'
5 < 1.25²
<< 1.25
No Regularization
0.1002
0.7580
4.5833
0.1905
0.8865
0.9589
0.9798
First-order
0.0993
0.7386
4.5274
0.1873
0.8870
0.9608
0.9810
Fist-order Blur
0.1013
0.7600
0.8849
Second-order
0. 0.04
0.7567
4.5260
0.1877
0.8871
0.9609
0.9808
Second-order Blur
0.1003
0.7661
4.5512
0.1881
0.8867
0.9607
0.9806
Occlusion (BG)
0.1004
0.7573
4.5454
0.1872
0.8850
0.9607
0.9811
Occlusion (FG)
0.1004
0.7623
4.5326
0.1872
0.8870
0.9611
0.9810

---

Published in Transactions on Machine Learning Research (12/2022)
Table State-of-the-AAt Summary. We summarize the settings used by each evaluated method in the henchmark. Con- tributions made by each method indicated by either bold font asterisk. Please that these settings do exactly reflect the original implementations by the respective authors, since have introduced changes for the sake of comparability and performance improvement.
Docoder
Proxy
Min+ Mask
Virtual
Blend
Smooth
SiM-Learner
M
Monodepth
Explainability
Klodt
M
Monodepth
Uncertainty
Monodepth2
Monodepth
Johnston
Discrete
HR-Depth
HRt-Depth
Garg
S
Monodepth
Monodepth
Monodepth
SuperDepth
Sub-Pixel
Depth-VO-Feat
Monodepth
DepthNet
Monodepth2
Monodepth
PeatDepth
MS
Monodepth
AutoEne
CADepth
M8
CADepth
DiffNet
MS
DiffNet
HR-Depth
MS
HR-Depth
Kuznietsov
SD*
Monodepth
berHu
DVSO
SD*
Monodepth
berHu
MoncResMMtthh
SD*
Monodepth
berHu
DepthHints
MSD
Monodepth
LogL1
Ours
MS
HR-Depth
Ours (Min)
MS
Hlt-Depth
Ours (Proxy)
MSD*
HR-Depth
LogL1
Our Proxy)
MSD*
HR-Depth
LogL1
5
Results
This section presents the results obtained when combining recent SotA approaches with our proposed design changes. Once again, the focus lies in training all baselines in fair and comparable manner, with the aim of finding the effectiveness of each contribution. For completeness and comparison with the original papers, we report results on the KE split in Section 5.2 However, it is worth reiterating that we strongly believe this eoaluation should not be used by fulure authors.
5.1
Evaluation details
We cap the maximum depth to 100 meters (compared to the common 50m (Garg et al. 2016 80m Zhou et al. 2017)) and omit border cropping (Garg et al.) 2016) & stereo-blending post-processing Godard et al. 2017 J. Again, we show the rank ordering based on image-based (AbsRel), pointcloud- based (F-Score) and edge- (F-Score) metrics. Stereo-supervised (S or MS) approaches apply fixed scaling factor to the prodiction, due the known bascline between the cameras during training. Monocular-supervisie (M) methods instead apply per-image median scaling to align the prediction and ground-truth.
We largely follow the implementation details outlined in Section which is replaced with ConvNeXt-B (Liu et al. 2022). As a bascline, all methods use the standard DispNet decoder (Mayer et al.] 2016) (excluding methods proposing new architectures). the SSIM+L1 photometric loss (Godard et al. (217)), edge-aware smoothness regularization (Godard et al. 2017) and upsampled multi- scale losses (Godard et al.) 2019). We settle on these design docisions due to their popularity and prevalence in all recent approaches. This allows to minimize the changes between legacy and modern approaches and foeus on the contributions of each method.
except for the backbone architecture,
10

---

Published in Transactions on Machine Learning Research (12/2022)
The specific settings and contributions of each approach can be found in Table For the full details please refer to the original publication, the review in Section 2 or the public codebase. We label the training supervision as follows: M = Monocular video synthesis, S = Stereo pair synthesis & D* = Proxy depth regression. In this case, all proxy depth maps used for regression (D*) are obtained via the hand-crafted stereo disparity algorithm SGM (Hirschmüller 2005). We further improve their robustness via the min roconstruction fusion proposed by Depth Hints (Watson et al. 2019).
Table 5: Kitti Bigen Evaluation. Rosults reported the original publications (top) those obtained by updated baselines (bottom). The proposed basclincs outperform those provided by the original authors in every casc, most notably in the mono Zhou 2017) and (Garg al. 2016) baselines. For instance 1.25 accuracy improved by raw 10% in both cases, while RbsRel error decreased by such, models with the proposed changes represent the new A.t
Original results
Train
AbsRel4
SqRely
RMSE4
LogRMSE4
< 1.25'
1.25"*
< 1.25"*
SFM-Learner
M
18
0.1830
1.5950
6.7090
0.2700
0.7340
0.9020
0,9590
Klodt
M
17
0,1800
1.9700
6,8550
0,2700
0.7650
0.9130
0.9620
Mouodepth2
M
13
0.1150
0,9030
4.8630
0.1930
0.8770
0,9590
0.9810
Johnston
0.8610
4.6990
0.9620
Hlt-Depth
0.1090
0.7920
4.6320
0.9620
0.9830
Garg
S
16
0.1520
1.2260
5.8490
0.2460
0.7840
0.9210
0.9670
Monodepth
8
14
0.1330
1.1420
5.5330
0.2300
0.8300
0.9360
0,9700
SuperDepth
11
0.1120
0,8750
4.9580
0,2070
0.8520
0,9470
0,9770
Depth-VO-Feat
1.1320
5.5850
0,9330
Monodepth2
0,8180
4.7500
0.9570
FeatDepth
MS
0.0990
0.6970
4.4270
0.1840
0.8890
0,9630
0.9820
CADepth
MS
3
0.1020
0.7520
4.5040
0.1810
0.8940
0.96410
0,9830
DiENet
MS
2
0.1010
0.7490
4.4450
0.1790
0.8980
0.9650
0,9830
HR-DDepth
MS
7
0.1070
0.7850
4.6120
0.1850
0.8870
0.9620
0.9820
Kuznietsov
SD*
12
0.1130
0.7410
4.6210
0.1890
0.8620
0,9600
0.9860
DVSO
SD*
0.1070
0.8520
4.7850
0.1990
0.8660
0.9500
0.9780
Monchh
SD*
10
0.1110
0.8670
4.7140
0.1990
0.8640
0.9540
0.9790
DepthHluts
MSD
0.1050
0.7690
4.6270
0.1890
0.8750
0.9590
0.9820
Our implementation
Train
AbsRelj
SqRel
RMSEL
LogRMSEL
< 1.25
8 < 1.253
<< 1.25
SIM-Learner
M
18
0.1374
1.6426
5.3609
0.2201
0.8562
0.9467
0.9732
Klodt
M
17
0.1325
1.4591
5.2738
0.2178
0.8604
0.9482
0.9740
Monodepth2
M
16
0.1 145
0.9374
4.9131
0.1946
0.8767
0.9589
0.9803
Johnston
M
15
0.1140
0.8577
4.7707
0.1912
0.8772
0.9606
0.9816
HR-Depth
M
14
0.1114
0.9026
4.8276
0.1911
0.8817
0.9609
0.9815
Garg
0.1007
0.8180
4.6693
0.1922
0.8859
0.9575
0.9788
Monodepth
S
0.1003
0.7938
1.6148
0.1909
0. 18888
0.9573
0.9793
SuperDepth
S
10
0.1030
0.8122
4.6907
0.1939
0.8828
0.9562
0.9786
Depth-VO-Feat
MS
9
0.1012
0.7802
4. 4.6676
0.1954
0.8800
0.9551
0,9780
Monodepth2
MS
5
0.0994
0.7491
4.5438
0.1879
0.8883
0.9607
0,9806
FeatDepth
0.0986
0.7410
4.5521
0.1880
0.8868
0.9603
0,9805
CADepth
0.7313
4.5117
0.9621
DiffNet
MS
0.0983
0.7377
4.5139
0.1852
0.8900
0.9622
0.9814
HR-Depth
MS
0.0974
0.7306
4.4875
0.1854
0. 8921
0.9621
0.9811
Kuznietsov
SD*
18
0.1098
0.9175
4.7278
0.1846
0.8797
0.9625
0.9832
DVSO
SD*
12
0.8226
4.6427
0.9594
Moschh
SD*
0.1048
0.8266
4. 1.6263
0.1881
0.8835
0.9588
0.9806
DepthHints
MSD
0.0997
0.7958
4.5112
0.1834
0.8924
0.9631
0.9820
11

---

Published in Transactions on Machine Learning Research (12/2022)
5.2 Kitti Eigen
We first validate the effectiveness of our improved implementations on the KE split. As discussed, we strongly believe that this dataset encourages suboptimal design decisions, and future work should not evaluate it. We report the original metrics from Eigen & Fergus (0015), detailed in Section B.1 Results can be found in Table 5 alongside results from the original published papers. As seen, the models trained with our design decisions improve over each of their respective baselines. This is particularly noticeable in the early baselines (Garg et al., 2016 Zhou et al.) 2017), which have remained unchanged since publication. This again highlights the importance of providing up-to-date baselines that are trained a comparable way.
5.3 Kitti Besen-Benchhark
We report results on the KEB split, described in Sections 3.1 & B.2 To re-iterate, this is an updated evaluation using the corrected depth maps from [hrig et al. (001)). From these images, we select a subset of 10 intercsting images to evaluate the qualitative performance of the trained models. Results can be found in Table G & Figure 7 As shown, when training and evaluating in comparable manner, the improvements provided by recent contributions are significantly lower than those reported in the original papers. In fact, we find the seminal stereo method by Garg et al. (2016) to be one of the top performers across all metries. Most notably, it outperforms all other methods in 3-D pointcloud-basse metrics, indicating that the reconstructions are the most accurate. Incorporating the discussed design decisions along with SotA contribution provides the best image-based based performance, However, these contributions were not developed to optimize pointcloud roconstruction. As seen, purely monocular approaches porform worse than stereo-based methods, despite the median scaling aligning the predictions the ground truth. However, incorporating the contributions from Monodepth2 (Godard al.) 2019) results in significant improvements over SfM-Learner.
Qualitative depth visualizations can be found in Figure Once again, the seminal Garg al. (2066) and SfM- Learner (Zhou el al. 2017) baselines are drastically improved w.r.t. the original implementation. However, SIM-Leaner predictions are still characterized by artifacts common to purely monocular supervision. Most notably, objects moving at similar speeds to the camera are predicted as holes of infinite depth, due to the fact that they appear static across images. Static pixel automasking from Monodepth2 (Godard et al. 2019)
GT (Interp)
STM- S Learner
Garg
Monodepth2 (MS)
Dett-Depth (MS)
Dipthifints (MS)
Figure Kitti Visualization. Baseline models Gaar 2016 Zhou al. 2017 greatly improved from their original implementations. Incorporating the minimum reconstruction loss improves accuracy on thin structures and prevents holes infinite depth. Full results Table
automasking from Monodepth2 (Codard et. al. 2019
12

---

Published in Transactions on Machine Learning Research (12/2022)
Table 6: Kitti Keeee-Benchhark Evaluation. When training in comparahle conditions, the sterro baseline
Garg et. al.
2016 is of the top performing methods. 'Tee minimum reprojection and automasking losses [Godard 2019 help
improve performance and mitigate monocular supervision artefacts. This is further improved via high-resolution decoder (Lyu)
2021 and praxy depth supervision (Watson 2019). However, these contributions only improve image-based depth accurate 3-D pointeloud reconstructions.
metrics, but not result.
Image-basor based
Pointcloudd-bsse
KEB (test)
Train
MAE)
RMSE4
AbsRte.
LogS14
#
Chamfer4
F-Scoref
loUf
SfM-Learner
M
1.98
0.73
Klodt
M
21
1.96
4.54
10.49
15.86
0.72
45.26
30.40
Monodepth2
M
18
1.81
1.11
8.82
13.10
18
0.71
46.64
31.62
Johnston
M
19
1.83
3.99
8.85
12.89
20
0.71
45.72
30.78
HR-Depth
M
17
1.80
4.04
8.65
12.75
17
0.69
47.35
32.10
Garg
1.60
0.60
53.28
37.33
Monodepth
1.61
3.72
7.73
11.57
0.64
51.25
35.45
SuperDepth
1.64
3.77
7.81
11.63
2
0.63
52.30
36.40
Depth-VO-Feat
MS
4
1.63
3.72
7.70
11.64
3
0.62
52.01
36.15
Monodepth2
MS
10
1.61
3.62
7.90
10.99
7
0.64
50.50
31.98
FeatDepth
MS
1.60
3.60
7.80
11.01
0.65
49.99
34.51
CADepth
0.66
34.06
DiffNet
MS
7.97
0.65
31.23
HR-Deptth
MS
3
1.58
3.56
7.70
10.68
5
0.62
51.49
35.93
Kuznietsov
SDY*
20
1.82
3.98
9.32
11.80
19
0.71
45.80
30.63
DVSO
SD*
13
1.66
3.77
8.05
11.32
0.67
49.
34.18
MouoBessMathh
SD*
DepthHints
MSD
15
1.63
3.62
8.10
10.94
0.66
33.80
Ours
MS
1.59
3.64
7.63
11.15
4
0.62
51.74
35.97
Ours (Min)
MS
1.58
3.53
7.79
10.58
0.63
50.45
35.01
Ours (Proxy)
MSD
1.57
3.50
7.73
10.59
0.64
34.09
Oure (Min+Proxy)
MSD
1.61
0.65
33.90
Improvement (%)
0
-10
-20
F-Score
LogSI
-30
AbsRel
MAE
40
Garg
Monodepth
Kuznletsov
SfM- SAM-Learner
Klodt
DVS0
Depth-VO-Fead -eat
SuperDepth
MonoResMatch
Monodepth2 (M)
Monodepth2 (MS)
Depth Hints
Johnston
FeatDepth
CADepth
DiffNet
HR- IR-Depth (M)
HR DR-Depth (MS)
Figure 7: Kitti Bigen-Bennchmate Improvement. When training and evaluating in fair conditions, many contributions
do not result. in relative improvements w.r.t. the Garg et al. 2016) stereo baseline. Most notably, all monoeular-supervieel approaches perform siguificantly worse despite the per-image median scaling. Full results in Table
13

---

Published in Transactions on Machine Learning Research (12/2022)
fixes most of these artifacts. Similarly, the minimum reconstruction loss leads to more accurate predictions for thin objects, such traffic signs and posts. DepthHints (Watson et al. 2019) and HR-Depth (Lyu et al. 2021) independently improve the quality of the predictions via proxy depth supervision and high-resolution decoder, respectively. However, these contributions do not significantly improve the 3-D reconstructions.
5.4 SYNS-Patches
Finally, we evaluate the basclines on the SYNS-Patches dataset. As discussed in Sections 3.2 this dataset consists of 1175 images from a variety of different scenes, such as woodlands, natural scenes and areas. It worth noting that we evaluate the models from previous section
urban rrsidential/industiie without re-training or fine-tuning. As such, SYNS-Patches represents a dataset completely unseen during training. This allows to evaluate the robustness of the learned representations new unseen environment types. We select subset of images per category to evaluate the qualitative performance. To make results more comparable, all models evaluated using the monocular protocol, where each predicted depth map is aligned to the ground-truth using per-image median scaling. We reuse the metrics from the KEB split and additionally report edge-based accuracy and completion from Koch et al. (2018). Finally, we also compute the F-Score only at depth boundary pixels, reflecting the quality of the predicted discontinuities.
Results for this evaluation can be found in Table As seen, performance decrcases drastically for all approaches, showing that models do not transfer well beyond the aulomotive domain. A further decrease in F-Score can be seen when evaluating only depth edges, indicating this as common source of error. Once again, models incorporating recent contributions provide SotA performance in traditional image-based
Table 7: SYNS-Patches Evaluation. Overall performance is drastically reduced when evaluating outside the automotive training domain. Methods using minimum reconstruction loss and automasking improve image-based based metrics, while Garg et al. (2016) still provides some of the top 3-D) pointeloud reconstructions. Predicted edge boundaries typically accurate (Fdgee Aon px), but incomplete (Edge-Comp >5 px).
Image- Based
Polutcloud-based
Edgo-based
$ SYN8-Patches
Train
#
MAF4
RMSF4
AbsRely
#
F-Scoref
ToU'
#
F-Scoref
Arc
Comp4
SfM-Learner
M
5.43
9.25
31.58
11.79
6.43
8.47
3.46
36.12
Klodt
M
20
5.40
9.20
31.20
21
12.00
6.57
19
8.48
3.44
35.22
Monodepth2
M
13
5.33
9.02
30.05
20
12.08
6.62
21
8.46
3.30
37.01
Johuston
M
10
5.21
8.92
29.72
18
12.16
6.66
18
8.60
3,23
42.82
HR-Depth
8
5.26
8.95
29.53
5
13.37
7.40
9.16
3.07
30.03
Garg
S
5.29
9.20
30.73
13.488
7.45
9.53
3.37
26.79
Monodepth
8
21
5.29
9.20
31.27
19
12.14
6.67
17
8.69
3.57
61.15
SuperDepth
S
16
5.26
9.08
30.83
13
12.87
7.10
11
9.01
3.40
10.10
Depth-VO-Feat
MS
17
5.30
9.17
30.83
16
12.43
6.82
15
8.77
3.50
38.49
Monodepth22
MS
6
5.18
8.91
29.04
13.18
7.27
8.95
3.38
32.69
FeatDepth
5.16
CADepth
MS
5.22
8.97
12.83
7.06
8.70
3.42
35.89
DiHNet
MS
2
5.16
8.91
28.80
13.16
7.26
8.81
3.45
39.46
HR-Depth
MS
5
5.13
8.85
28.91
1
13.79
7.65
5
9.21
3.25
28.33
Kuzuletsov
SD*
5.47
9.50
31.08
13.15
7.26
9.11
3.39
47.13
DVSO
SD*
5.18
MotchesMatch
SD*
14
5.24
9.07
30.28
12.73
7.01
9.03
3.47
51.03
DepthHiuts
MSD*
18
5.33
9.07
30.90
12
12.91
7.11
10
9.01
3.24
26.21
Ours
MS
12
5.20
9.03
29.93
4
13.38
7.39
9.28
3.31
34.36
Ours (Min)
5.11
28.59
Oure (Proxy)
MSD*
3
5.11
9.23
3.16
30.60
Ours "Mi++Proxy"
MSD*
4
5.08
8.71
28.91
6
13.223
7.30
8
9.11
3.16
31.32
14

---

Published in Transactions on Machine Learning Research (12/2022)
Learner
Garg
Monodepth2 (MS)
HRR-Depth (MS)
DepthHints (MS)
Figure SYNS Visualization. evidenced by Table models trained on Kitti not transfer well natural scenes, such
as woodlands. Similar to Kitti, we the contributions from Mondepth2 (Godard al. 2019) improve prediction accuracy in challenging areas such as thin structures and object boundaries. Full results in Table
depth metrics. However, Garg et al. (2016) consistently remains one of the top performers in 3-D pointcloud reconstruction and edge-based metrics. In general, predicted edges are typically accurate (~3 px error).
However, there are many missing edges, as reflected by the large edge completeness error (~26 px error).
Qualitative dopth visualizations can be found in Figure Similar to Kitti, Monodepth2 (Godard et al. 2019) and its successors (Watson al., 2019 Lyu al.; 2021 greatly improve performance on thin structures. However, there is still room for improvement, shown by the railing prediction in the second image. This is reflected by the low 3-D reconstruction metrics. Furthermore, all methods perform significantly worse in natural and woodlands scenes, demonstrating the need for more varied training data.
6 Conclusion
This paper has presented a detailed ablation procedure critically evaluate the current SotA self- supervised monocular depth learning. We independently reproduced 16 recent baselines, modernizing them with sensible design choices that improve their overall performance. When benchmarking on level playing field, show how many contributions do not provide improvements over the legacy stereo baseline. Even in the case where they do, the change in performance is drastically lower than that claimed by the original publications. This results in new SotA models that set the bar for future contributions. Furthermore, this work has shown how, in many cases, the silent changes that are rarely claimed as contributions can result in equal or greater improvements than those provided by the claimed contribution.
Regarding future work, we identify two main research directions. The first of these is the generalization capabilities of monocular depth estimation. Given the results on SYNS-Patches, it is obvious that purely automotive data is not sufficient to generalize to complex natural environments, ronments. As such, it would be of interest to explore additional sources of data, such as indoor sequences, natural scenes or even synthetic data. The second avenue should focus on the accuracy on thin objects and depth discontinuities, which are challenging for all existing methods. This is reflected in the low F-Score and Edge Completion metrics these regions. To aid future research and encourage good practices, we make the proposed codebase publicly available. We invite authors contribute to it and use it a platform train and evaluate their contributions.
even other urban envi-
15

---

Published in Transactions on Machine Learning Research (12/2022)
Acknowledgments
This work was partially funded by the EPSRC under grant agreements EP EP//016317/1 & EP GP/S035761/L.
References
Wendy Adams, James H Elder, Erich W Graf, Julian Leyland, Arthur J Lugtigheid, and Alexander Muryy. The Southampton- York Natural Scenes (SYNS) dataset: Statistics of surface attitude. Scientific Reports, doi: (..1038/srep3550))
6(1):35805, 2016. ISSN 2045-2322.
Jiawang Bian, Zhichao Li, Naiyan Wang, Huangying Zhan, Chunhua Shen, Ming-Ming Cheng, and lan Reid. Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video. In Advances in Neural Information Processing Systems, volume 32, 2019.
David Eigen and Rob Fergus. Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture. In International Conference Computer Vision, pp. 2650-2658, 2015. doi: 10.1109 /1110//CCV.... 2015.304.
Jakob Engel, Vladlen Koltun, and Daniel Cremers. Direct Sparse Odometry. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(3):611-6255 2018. doi:
Ravi Garg, Vijay Kumar, Gustavo Carneiro, and lan Reid. Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue. In European Conference Computer Vision, pp. 740-756, 2016.
TSBN 978-3-319-46484-8.
A Geiger, P Lenz, C Stiller, and R Urtasun. Vision meets robotics: The KITTI dataset. International Journal of Roboties Research, 32(11): 2013. doi: 10.1777/22733644114492977
Clement Godard, Oisin Mac Aodha, and Gabriel J. Brostow. Unsupervised Monocular Depth Estimation with Legt-Right Consistency. Conference Computer Vision and Pattern Recognition, pp. 2017. ISSN 1063-6919. doi: N0.1199/CVPR.2017.000
Clement Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging Into Self-Supervised Monocular Depth Estimation. International Conference Computer Vision, 2019-0ctob:3527 3837, 2019. doi: 10.1109 /ICCV 2019.00393.
Juan Luis Gonzalez Bello and Munchurl Kim. Forget About the LiDAR: Self-Supervised Depth Estimators with MED Probability Volumes. In Advances in Neural Information Processing Systems, volume 33, pp. 12626-12637, 2020.
Juan Luis Gonzalez Bello and Munchurl Kim. PLADE-Net: Towards Pixel-Level Accuracy for Self- Supervised Single-Vier Depth Estimation with Neural Positional Encoding and Distilled Matting Loss. In Conference on Computer Vision and Pattern Recognition, pp. 6847-6856, 2021. doi: 10.1109 D01109/CVPR46437. 2021.00678.
Vitor Guizilini, Ambrus Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3D packing for self- supervised monocular depth estimation. Conference on Computer Vision and Pattern Recognition, pp. 2482-24911 2020. ISSN 10636919. doi:
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016. ISBN 97734688855 doi:
N0.1109/CVPR.2066.20
Heiko Hirschmüller. Accurate and Efficient Stereo Processing by Clni-Global Matching and Mutual Infor- mation. Conference on Computer Vision and Pattern Recognition, II:8078814 jun 2005. ISSN 1063-6919. doi: 10.1109/CVPR.220
Andrew Howard, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chicl Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, Yukun Zhu, Ruoming Pang. Hartwig Adam, and Quoe Le. Searching for MobileNetV3. Tn International Conference on Computer Vision, pp.
2019. doi: 10.1109/ICCV.2019000
16

---

Published in Transactions on Machine Learning Research (12/2022)
Jie Hu, Li Shen, Samuel Albanic, Gang Sun, and Enhua Wu. aaneezo-and-Excitattie Networks. IEEE Transactions on Patlern Analysis and Machine Intelligence, 42((0):2011-200 2020. TSSN 0162-8828.
doi: 1AL10//TPAML220199113377
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukeuoglu. Spatial Transformer Net- works. In Advances in Neural Information Processing Systems, volume 28, 2015.
Adrian Johnston and Gustavo Carneiro. Self-Supervised Monocular Trained Depth Estimation Using Self- Attention and Discrete Disparity Volume. In Conference on Computer Vision and Pattern Recognition, pp. 4755-4764, 2020. doi: N01109/CVPR42600 010000..220004455
Alex Kendall and Yarin Gal. What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? In Advances in Neural Information Processing Systems, volume 30, 2017.
Maria Klodt and Andrea Vedaldi. Supervising the New with the Old: Learning SFM from SFM. In European Conference on Computer Vision, pp. 713 728, 2018. ISBN 978-3-030-01249-6.
Tobias Koch, Lukas Liebel, Friedrich Fraundorfer, and Marco Körner. Evaluation of CNN-Based Single- Image Depth Estimation Methods. In European Conference on Computer Vision Workshops, pp. 331-348, 2018. ISBN 978-3-030-11015--1
Yevhen Kuznielsov, Jörg Stückler, and Bastian Leibe. Semi-Supervised Deep Learning for Monocular Depth Map Prediction. In Conference on Computer Vision and Pattern Recognition, pp. 2215-2223, 2017. doi: (0.1009/CVPR.2117.0
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. International Conference on 3D Vision, pp. 239-248, 2016. doi: 10.1109/ ////333V2206630
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, Saining Xie, and Face- book AI Rescarch. A ConvNet for the 2020s. arXiv preprint, 2022.
Yue Luo, Jimmy Ren, Mude Lin, Jiahao Pang, Wenxiu Sun, Hongsheng Li, and Liang Lin. Single View Stereo Matching. Conference on Computer Vision and Pattern Recognition, pp. 155 163, 2018. ISSN 10636919. doi: ML1109/CVPR.2018
Xiaoyang Lyu, Liang Liu, Mengmeng Wang, Xin Kong, Lina Liu, Yong Liu, Xinxin Chen, and Yi Yuan. HR-Depth: High Resolution Self- Self-Supervised Monocular Depth Estimation. AAAI Conference on Artificial Intelligence, 35(3):2204-230)) 2021.
Reza Mahjourian, Martin Wicke, and Anelia Angelova. Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints. Conference on Computer Vision and Pattern Recognition, pD. 3667-5675, 2018. doi: N0.100//CVP.22118.000
Nikolaus Mayer, Eddy Ilg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. Conference on Computer Vision and Pattern Recognátion, pp. 4040-4048, 2016. ISSN 1063-
6919. doi: 10.1109/CVDD CVPR.2006..23
Evin Pinar Örnek, Shristi Mudgal, Johanna Wald, Yida Wang, Nassir Navab, and Federico Tombari. From 2D to 3D: Re-thinking Benchmarking of Monocular Depth Prediction. arXiv preprini, 2022.
Sudeep Pillai, Rareş Ambruş, and Adrien Gaidon. SuperDepth: Self-Supervised, Super- Resolved Monocular Depth Estimation. In International Conference on Robotics and Automation, pp. 2.111009/CCAA2219.8877000
2019. doi:
Mattco Poggi, Fabio Tosi, and Stefano Mattoccia. Learning Monocular Depth Estimation with Unsupervisod Trinocular Assumptions. In International Conference on 3D Vision, pp. 324-333, 2018. doi: 10.1109/ 3DV EDVV2018.00045.
17

---

Published in Transactions on Machine Learning Research (12/2022)
Matteo Poggi, Filippo Alcotti, Fabio Tosi, and Stefano Mattoccia. On the Uncertainty of Self- Seff-Supervied Monocular Depth Estimation. In Conference on Computer Vision and Patlern Recognätion, pp- 3224-3234 2020. doi: 10.1109 10.1109/CVPR22000 0T0T:42600.220.003222
Fitsum A Reda, Guilin Liu, Kevin Shih, Robert Kirby, Jon Barker, David Tarjan, Andrew Tao, and Bryan Catanzaro. SDC-Net: Video Prediction Using Opatially-Displaced Convolution. In European Conference on Compuler Vision, pp. 747 763, 2018. ISBN 978-3-030-01234--.
Rui, Stückler Jörg, Cremers Daniel Yang Nan, and Wang. Deep Virtual Sterco Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry. In European Conference on Computer Vision, pp. 835-852 2018. ISBN 978-3-030-01237-3.
Wenzhe Shi, Jose Caballero, Ferenc Huszar, Johannes Totz, Andrew Aitken, Rob Bishop, Daniel Rueckert, and Zchan Wang. Real-Time Single Image and Vidco Super-Rcsolution Using an Efficient Sub-Pixel Convolutional Neural Network. Conference on Computer Vision and Pattern Recognition, 2016-Dccem: 1874-1883, 2016. TSSN 1063-6919. doi: 10.1109/CVPR.20162201
Chang Shu, Kun Yu, Zhixiang Duan, and Kuiyuan Yang. Feature- Loss for Self-supervised Learning of Depth and Egomotion. In European Conference on Computer Vision, pp. 572-588, 2020. ISBN 978-3- 030-58529-7.
Jaime Spencer, Richard Bowden, and Simon Hadfield. DeFeat-Net: General monocular depth via simulta- ncous unsupervised representation learning. In Conference on Computer Vision and Pattern Recognition, pp. 14390- 14401, 2020. doi: 10.1109/ N0.1109/CVPR42600 NSPPR42600.200.001
Mingxing Tan and Quoc Le. ElficientNet: Rethinking Model Scaling for Convolutional Neural Networks. In International Conference on Machine Learning, volume 97, pp.
2019.
Fabio Tosi, Filippo Aleotti, Matteo Poggi, and Stefano Mattoccia. Learning monocular depth estimation infusing traditional stereo knowledge. Conference on Computer Vision and Pattern Recognition, 2019- June:9791 9801, 2019. ISSN 10636919. doi: 10.
Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger, Sparsity Invariant CNNs. International Conference on 3D Vision, pp. 11-20, 2018. ISSN 2475-7888, doi: 10.1109/ 3DV.2017.00012.
Chaoyang Wang, Jose Miguel Buenaposada, Rui Zhu, and Simon Lucey. Learning Depth from Monocular Videos Using Direct Methods. Conference on Computer Vision and Pottern Recognation, pp. 2030- 2018. ISSN 10636919. doi: www.109/CVVR.2088..00
Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang, Wenyu Liu, and Bin Xiao. Deep High-Resohution Representation Tearning for Visual Recognition. IEFE Trunsactions on Pallern Analysis and Machine Intelligence, 43
(10):3349-3366) 2021. ISSN 0162-8828. doi: 1111009/TAMMI CLLMI.2020.211 298.299.6686
Rui Wang, Martin Schwörer, and Daniel Cremers. Stereo DSO: Large-Scale Direct Sparse Visual Odometry with Stereo Cameras. In International Conference on Computer Vision, pp. 3931,, 2017. doi: 10.
(009/ICCV.2017.40
Zhou Wang, A C Bovik, H R. Sheikh, and E P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600- 60,6 2004. doi: 10.1109/TIP. 2003.819861.
Jamie Watson, Michael Firman, Gabriel Brostow, and Daniyar Turmukhambetov. Self-supervised monocular depth hints. International Conference on Computer Vision, 2019-Octob:2162 2171, 2019. ISSN 15505499. doi:
2019.0002...
Chamara Saroj Weerasekera, Ravi Garg, and Ian D Reid. Learning Deeply Supervised Visual Descriptors for Dense Monocular Reconstruction. CoRR, abs/1711.0, 2017.
18

---

Published in Transactions on Machine Learning Research (12/2022)
Ross Wightman. PyTorch Image Models, 2019.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. Conference on Computer Vision and Pattern Recognition, 2017-Jamaa:99775596 2017. doi: 10.1109/CVPR... 201...22.........
I. Zeki Yalniz, Jegou Hervé, Chen Kan, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXic preprint, 2019. ISSN 2331-8422.
Jiaxing Yan, Hong Zhao, Penghui Bu, and YuSheng Jin. Channel-Wise Attention- Based Network for Self- Supervised Monocular Depth Estimation. In International Conference on 3D Vision, pp. 464 473, 2021.
Nan Yang, Lukas von Stumberg, Rui Wang, and Daniel Cremers. D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry. In Conference Computer Vision and Pattern Recognition, pp. 1278-1289, 2020. doi: 10.1109 10009//2R42600000 CVVPR42600.202000003333 2020.00136.
Huangying Zhan, Ravi Garg, Chamara Saroj Weerasekera, Kejie Li, Harsh Agarwal, and Ian M. Reid. Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Re- construction. Conference Computer Vision and Pattern Recognition, pp. 340-349, 2018. doi: 10.1109/CVPII 20103.20...000.
Hang Zhou, David Greenwood, and Sarah Taylor. Self-Supervised Monocular Depth Estimation with Internal Feature Fusion. In British Machine Vision Conference, 2021.
Tinghui Zhou, Malthew Brown, Noah Snavely, and David G. Lowe. Unsupervised Learning of Depth and Ego-Motion from Video. Conference on Computer Vision and Pattern Recognition, pp. 1612-6619, 2017. ISSN 1063-6919. doi: 10.1109/CVPR.2011
Laurent Zwald and Sophie Lambert-Lacroix. The BerHu penalty and the grouped effect. Xi preprint, 2012.
19

---

Published in Transactions on Machine Learning Research (12/2022)
A Monocular Depth Overview
We provide a review of self-supervised monocular depth estimation, which serves as an introduction to the field while providing in-depth details about the implemented basclinc models. We also provide some practical implementation details that are frequently omitted. For the purpose of this paper, we limit our benchmark to methods that perform only depth estimation along with optional VO prediction. We do not include methods that additionally learn other tasks, such as semantic segmentation, optical flow or surface normals.
A.1 DepthNet
The core component of the framework is the depth prediction network, composed of a fully-convolutional encoder-decoder architecture with skip connections. This network produces multi-scale dense depth predic- tions.
Encoder. Any architecture producing a multi-scale feature representation can be used as the encoder. In practice, this is implemented in a highly flexible way by using the timm library (Wightman 2019 containing pretrained models for recent Sot.A classification architectures. As part of the ablation presented in Section [ we perform an in-depth study of the most effective backbone architocture and its protraining method. We find ConvNeXt. (Liu al. 2022 to provide the best performance overall.
Depth Decoder. The commonly used dense decoder is inspired by the DispNet architecture (Myer et al, 2016). It contains five upsampling stages, each composed of two Conv-ELU blocks with nearest- neighbur upsampling and a concatenated skip connection from the corresponding encoder stage. Each stage makes an initial depth prediction at its reduced resolution, used as additional supervision during training. These intermediate predictions are omitted from the following network and loss equations for the sake of brevity and clarity. In practice, DepthNet predicts the inverse depth (i.e. disparity) due its increased stability. As such, this prediction is obtained via convolution followed by sigmoid activation.
The depth prediction process can be summarized as
Ż = b (I)
where D is the full DepthNet architecture and 2 is the predicted sigmoid disparity. This is converted into a scaled depth prediction through
D
(2)
aZ b
where a & b are constants such that the final depth is in the range [0.1, 100). It is worth noting this scale is arbitrary and does not correspond to metric depth.
Virtual Stereo. Monodepth (Godard et al. 2017) rotuodd the concept of virtual stereo prediction where the network is forced to predict both the left and right disparity from only the left input image. This is achieved by adding an extra output channel to the disparity prediction at each encoder scale. The original Monodepth was trained using only images from the left viewpoint, meaning the virtual disparity would always correspond to the right view.
To make this procedure more flexible and allow for training with either stereo viewpoint, we adopt a procedure similar to 3Net (Poggi et al. 2018). We assume the input to the network is the central viewpoint in a three- camera rig and predict two virtual disparities, corresponding to the left/right cameras. During training, we select the "real virtual" disparity, i.e. if the network was given the right image we sample the virtual left disparity, and vice versa. It is worth noting this does not require an additional decoder as in Poggi et al. (2018), since simply add two extra output channels to each network prediction.
Predictive Mask. Training in a purely monocular sctting (Zhou et al. 2017: Klodt & Vedaldi 2018 is susceptible to artifacts not present during stereo training. Dynamic objects moving independently from the rest of the scene are unaccountd for in the predicted PoseNet motion, resulting in incorrect synthesized views even the depth prediction correct. If the object is moving at similar speeds to the camera, it will appear as static between images, resulting in predictions of infinite depth. Meanwhile, objects moving towards the camera will seem closer than they are and depth will be underestimated.
20

---

Published in Transactions on Machine Learning Research (12/2022)
To mitigate these effects, SfM-Learner Zhou et al. 2017 introduced additional decoder to predict explainability mask in the range [0. 1.. This mask was trained to downweight the photometric loss in unreliable regions, such as dynamic objects or specularities and regularized using a binary cross-entropy loss pushing all mask values towards 1. Klodt & Vodaldi 2018) instead adopted the uncertainty formulation introduced by Kendall & Gal (2017). The nelwork is therefore trained to predict the log variance uncertainly values.
associated with each photometric error pixel. In this case, the mask restricted to positive uncertainty
Following recent implementations Godard et al. 2019 these masks are predicted for each of the support images via an additional DepthNet decoder. Details on integrating these masks into the photometric loss are provided in Section A.4
A.2 PoseNet
Encoder. Similar to DepthNet, the encoder can be casily replaced with any desired architecture by leveraging the pretrained models from [Wightman (019)). However, PoseNet is typically restricted to a pretrained ResNet-18 for efficiency. VO requires us to predict the relative motion between frames. As such, we perform an independent prediction for each support frame by concatenating them channel-wise with the target frame. We modify the first convolutional layer of the architecture accordingly, ensuring the pretrained weights are correctly duplicated and scaled. As is common, we force the network to make forward-motion prediction by swapping the image order if necessary.
Decoder. Since VO is a holistic regression task (it produces one output for the whole image), we use a simple decoder. The encoder features are projected to lower dimensionality via a 1 x convolution and processed by two Conv-ReLU blocks. The final prediction is obtained by applying a convolution without an activation. As is common practice, we scale the network predictions by a factor of 0.01 to improve stability and convergence. The network produces a 6-D output, formed by a translation and axis-angle rotation. The rotation vector is converted into indicating the angle and the direction corresponding to the axis of rotation.
3 transform matrix via the Rodrigues formula, with the magnitude
Overall, the process for obtaining the predicted motion P. P + between frames I, & I. is simplified as
P = 4p (I, + k)),
(3)
where @ represents channel-wise concatenation and 4 is the resulting PoseNet.
A.3 View Synthesis
The core component of all self-supervised monocular depth approaches is the ability to synthesize the target image from a set of adjacent support frames. Given the scene depth and camera. locations, a point in the target image Pa can be reprojected onto any of the available frames as
P' P+A KP,+14 (p.) P.:.
(4)
where K represents the shared camera. intrinsic parameters, D. (Pt) is the predicted depth at the given point and P. + is the predicted camera motion matrix (or the known stereo baseline). Using these correspondences, is possible to synthesize a support frame aligned to the target image by bilinearly sampling each image pixel. This is given by
---== = 1 (p'(+)),
(5)
where (1) represents bilinear sampling using Spatial Transformer Networks (Jaderberg et al. 2015
A.4 Losses
Photometric. The main loss the photometric error. This measures the difference between the raw target frame I, and the senthesized view from the support frame I' + & generated by 5 Tn the most simple case,
21

---

Published in Transactions on Machine Learning Research (12/2022)
L1 loss can be used. However, this is typically complemented by an SSIM (Shi et al. 2016) loss improve its robustness. This is defined
+ (1-Aastm) Li (I,1'), sem (I,I')
photoo (1,1') Assim
(6)
where Assim represents the SSIM weight. This photometric loss can be additionally downweightod when using a predictive mask. When using the explainability mask Mr (Zhou al. 2017) this simply defined
Phooto (I. r1) = Mg O Sphoto (I.1),
while the uncertainty mask M (Klodt & Vedaldi 2018 affects it as
Cphoto (I, r) exp (-M @ phooto (I, + My.
(8)
where @ represents the Hadamard product.
Reconstruction. The photometric loss aggregated by the reconstruction loss. The base case simply averages the photometric loss over each support frame and each image pixel, defined as
Frac (I+) = >I Shhoto (I, ((,k() P
where D > represents the average summation. Monodepth2 (Godard al. 2019 showed the benefit of instead selecting the minimum loss per-pixel over the various support trames. Intutively, this selects correct pixel-wise correspondences, instead of averaging out incorrect errors caused by occlusions. This is simplified as
yyec (I,) =
Σ min Cphoto +n)..
(10)
Monodepth2 additionally proposed an automatic masking procedure. Static pixels are removed from the loss if the photometric error for the original support frame lower than the synthesised view loss. This is given by
min Sphoto (T, < min Lphoto (I,, Te
Ms [
(11)
where [-] represents the Iverson brackets and Mig the resulting mask selecting non-static pixels. Note that the second term in the equation uses the original support frame It k-
Feature reconstruction. As noted by previous works (Zhan et al. 2018 Spencer al. 2020 Shu et al. 2020), the image-based based reconstruction loss might still produce ambiguous correspondences. This is especially the case in low-light environments that contain multiple light sources and complex reflections. In this case, it can be beneficial to incorporate an additional frasurebbased reconstruction loss, defined as
Ljen (Fz) = D min Lphoto (Fr, F' F)
(12)
Fi- = Fe+ (p'1-n),
=
(13)
where F', & are the synthesized support features using the previously computed reprojection correspon- dences from 4 Note that in this case the L, SSTM photometric loss can be replaced with the L2 distance between feature embeddings. In practice, instead of introducing an additional pretrained dense feature network, we propose to re-use the low-level features from the depth encoder.
Depth regression. To complement the self-supervised reconstruction losses, previous approaches have intro- duced an additional proxy supervised regression loss. This can prevent local minima during the optimization process. When computing the loss between disparities we opt for the standard L loss. However, when in
22

---

Published in Transactions on Machine Learning Research (12/2022)
depth space other well-established losses. The first of these is the reverse Huber (berHu) loss
Zwald
& Lambert-Tacroix 2012 Laina et. al. 2016). defined as
C1 D.
D)
where L, (D. 8
HuerHuu (.D)
=
D.D
+
(14)
otherwise
20
where D represents the (proxy) ground truth depth, D is the network prediction and the margin threshold is adaptively set per-batch as = 0.2 max L; (D.D). DepthHinnts (Watson et al. 2019) argues the berHu loss
is better suited towards regressing ground-truth LiDAR and proposes to use the Log-I1 loss when regressing hand-crafted stereo disparities, given by
LiogLi (D.D) = log (1 [1+ L1 (D.D))
(15)
Virtual stereo consistency. Following Monodepth (Godard et al. 2017 use the virtual stereo consis- tency loss, forcing the network to predict consistent scene depth for both viewpoints. This is achieved by applying the view synthesis module to the disparities, as opposed to the images. This results in
Ssteere (Z.,Z.++)) -1 Li (2 2 +)).
(16)
-'-x= = = + (p'
(17)
where z' Z'(4k & is the warped disparity corresponding to the virtual stereo prodiction from the network.
A.5 Regularization
Disparity smoothness. In general, expect pixel's disparity to be similar to that of its ncighbours. Garg
al. 2016 introduced simple regularization constraint encouraging this by penalizing all gradients in the disparity map, defined as
Ssmooth
7 =
az (p)
(18)
where OZ are the spatial gradients of the mean- normalzede disparity, computed in the x & y directions independently. We omit these dircctions from the equations for clarity. Godard et al.] (2017) softened this constraint by allowing disparity gradients proportional to the image gradients at that pixel location. This is known as the edge-aware smoothness regularization, given by
L Lamooth
(2)
Σ
oz (p) exp (- (p))..
(19)
Rui et al. 2018 instead enforce a smooth change in gradients, É.e. socond-order smoothness. This is defined as
Ssmooth
i = Σ 822 (p) exp (-10)(pp)),
(20)
where á2 represents the second-order gradients in each spatial direction. In this benchmark we add the option to apply Gaussian smoothing prior to computing the disparity sroparity/image gradients.
Occlusions. DVSO (Rui et al.) 2018) noted that the smoothness constraints can result in oversmoothed predictions. To counteract this, they proposed occlusion regularization term that penalizes the sum of disparities in the scene. This should allow the network to predict sharper boundaries at occlusions, while favouring background disparities. This is simply defined as
Loo 7
I Ż (p).
(21)
23

---

Published in Transactions on Machine Learning Research (12/2022)
Masks. Using the explainability mask (Zhou al. 2017] lets the network downweigh the photometric loss in regions where it believes correspondences may be incorrect uninformative. This occurs with dynamic object moving independently from the scene. To avoid the degenerate case where the mask ignores all pixels, an additional binary cross-cntropy regularization pushes all values towards 1.
Benchmark Datasets
Complementing Section we provide additional details for cach of dataset. This includes the various metrics defined by the Kitti splits, well as the creation procedure for the SYNS-Patches dataset.
B.1 Kitti Eigen
As discussed in the main paper, we strongly encourage future authors to avoid this evaluation, due the inaccurate ground-truth. However, we provide details for comparison with previous publications the community transitions to the proposed benchmark. The KE split (Eigen & Fergus 2015 defines the following metrics:
AbsRel. Measures the mcan relative error (%) as
y
(22)
where Σ i= E represents the average summation.
SqRel. Measures the relative square error (%)
j
y
(23)
Note that this is incorrectly computed, as the squared term is missing from the denominator.
RMSE. Measures the root mean square error (meters) as
yl
(24)
LogRMSE. Measures the root mean square log error (log (etees) as
log (0) log (y)1²
(25)
Threshold accuracy. Measures the threshold accuracy (%) 5 1.25% where & E (1,2,3) and
i y
8 max
(26)
y 0
B.2 Kitti Eigen-Benchmark
As replacement for KE, propose to use the updated and corrected ground truth from Uhrig et al. (0088 This benchmark defines the following metrics:
MAE. Measures the mean absolute error (meters) as
| - 3/
(27)
RMSE. Measures the root mean square error (meters) as
(28)
24

---

Published in Transactions on Machine Learning Research (12/2022)
InvMAE. Mcasures the mean absolute inverse error (1/meters) as
1
(29)
1
InvRMSE, Measures the root mean square inverse error (1/meters) as
2
(30)
LogMAE. Measures the mean absolute log error (oggmmeers) as
log (0) log (y)l
(31)
LogRMSE. Measures the root mean square log error (log-metersy as
log (ÿ) log (y)
(32)
LogSI. Measures the root mean scale invariant log error
Figenn & Fergus 2015)
(log meters) as
Σ log (ÿ) log (y))
Σ log (6) log (1)
(33)
Note that the second term in this metric is directional, i.e. the prediction is rewarded if it is consistently incorrect in the same direction. This accounts for the case where the prediction is correct, but scaled differently to the ground truth.
AbsRel. Measures the mean relative error (%) as
y
(34)
SqRel. Measures the mean relative square error (%) as
Σ |0 y/
(35)
y²
The metrics described above measure the error the predicted 2-D depth map. However, the true object of monocular depth estimation accurately reconstruct the 3-D world. As such, Örnek al. (2022)
proposed to instead evaluate depth prediction using well-established 3-D metries, which measure the fidelity of the reconstructed pointcloud. We report the following 3-D metrics:
Chamfer. Mcasures the Chamfer distance between the reconstructed pointclouds (m) as
e min lq
âll
Σ
min | q
åll.
(36)
qEQ QEQ
qEp
qEQ
where Q & Q represent the ground-truth and predicted pointclouds and & å are 3-D points in those pointclouds, respectively.
Precision. Measures the number of prodicted points (%) within a distance threshold to the ground-truth surface as
P
min qq - all < <6]
(37)
qEQ
qEQ
25

---

Published in Transactions on Machine Learning Research (12/2022)
(a) Sigma 01
Sigma
(c) Sigma 03
(d) Sigma 01 Log
(e) Sigma 02 Log
(f) Sigma 03 Log
Sigma Inv
(h) Sigma Inv
Sigma Inv
(j) Sigma 01
(k) Sigma 02
(I) Sigma 03
(m) Sigma 01 Log
(n) Sigma 02 Log
Sigma Log
(p) Sigma 01 Inv
(q) Sigma 02 Inv
(r) Sigma 03 Inv
Figure SYNS-Patches Dataset. Extracted ground-truth depth boundaries raw aaw/loginn depth maps using increusing Caussian blurring sigmas. Sigma log-depth provides good balance of edges-to-noisc ratio.
Recall. Measures the number of ground-truth points (%) within distance threshold 5 the predicted surface as
x qEQ
<o] 8
R
min l|q all <
(38)
QEQ
As per Örnek et al. (022)) we set the threshold for correctly reconstructed point to 5 0.1, d.e. 10 cm.
26

---

Published in Transactions on Machine Learning Research (12/2022)
F-Score.
Also known the Dice coefficient. Mcasures the harmonic mean of precision and recall (%) as
P.R
F= P+R
(39)
ToU. Also known the Jaccard Index between two pointclouds. Measures the volumetric quality of a 3-D reconstruction (%) as
P.R IoU P+R-
(40)
B.3 SYNS-Patcher
We generate SYNS-Patches, based on SYNS (Adams et al., 2016), by sampling undistorted patchos from the spherical panorama image every 20 azimuth degrees at constant elevation of zero. In other words, we perform full horizontal rotation roughly at eye level, sampling images per scene.
The original panoramic data collected using scanning camera rotating around its vertical axis, with each scan taking approximately 5-10 minutes. The LiDAR was captured after the HDR panorama following similar procedure, resulting in dense depth scans. Since the panorama. is not instantly captured, it is susceptible to distortions from moving objects. Furthermore, since the image and depth scans were captured independently, is possible to have mismatched Sstatic-butdynnad tionary pedestrrians. We visually check the extracted dataset and remove images containing these artifacts, along with empty/unifors depth maps. The final test set contains 1,175 out of the possible x 92 1, 656 images.
objects, such as parked cars or sta-
Ground-trute depth boundaries are obtained by detecting Canny edges in the depth map. This is sensitive to missing data (duc to infinite depth or highly reflective surfaces), which can cause fake boundaries. We take this into account by masking edges connected to regions with invalid depths, unless these pixels belong to the sky. Accurate sky masks are obtained using a pretrained SotAA semantic segmentation model (Reda) ct al.] 2018). We generate depth edges at three levels of Gaussian smoothing, using either raw, log or inverse depth maps. In practice, we find log-depth and smoothing sigma of to provide the best qualitative results. Example depth boundary predictions for different hyperparameters shown in Figure
As in KEB, use the original image-based metrics defined by Uhrig et al. 20118) complemented by the pointcloud- based metrics from Örnek et al. (2022) To provide more granular results, we optionally compute all metries only at the detected ground-truth depth boundaries. We additionally report. the edge-based metrics Koch et al. 2018);
Acc. Measures the accuracy of the predicted depth edges (pixels) via the distance from each predicted edge to the closest ground-truth edge as
EDDT L EDT Y bin (p)) P (p/Y bin (p) = =1,
(41)
where EDT represents the Euclidean Distance Transform, truncated to maximum distance = 10 and
Yoi & Y bin are the binary maps of depth boundaries for the ground-truth and prediction, respectively. EdgeComp. Measures the completeness of the predicted depth edges (pixels) via the distance from each ground-truth edge to the closest predicted edge as
e =
I EDT(Y bsn (p)) PP {p/Y bin (p) = 13.
(42)
27


Reviews:
[{"review_id": "QSvAaG65Sl", "reviewer": "Reviewer_QF4o", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The authors re-implemented and re-evaluated many representative methods in a unified codebase, which is fair and less biased.\n+ The authors proposed a new test benchmark that contains more diverse scenes than KITTI.\n+ The authors also corrected errors in KITTI benchmarks during evaluation.\n\nWeaknesses\n- The conclusion is somewhat trivial, improving backbone networks \n- The authors observe that the depth regularization decreases the performance, but lacks a deeper analysis.\n- For the monocular sequence, the the pose is estimated from a network, which makes it unclear whether the decreased/increased performance is from other design choices for better depth.\n- In terms of the existing datasets, the authors only investigate KITTI, which has a strong layout bias and may limit the generality of the conclusion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper re-evaluates existing self-supervised monocular depth estimation networks and investigates the significance of some designs. To this end, the authors re-implement several representative methods and evaluate them both on the original benchmarks and the proposed SYNS-Patches test sets. The authors conclude that the backbone network is critical for improved performance, while some other designs, such as the regularization loss, are less significant or even decrease the performance.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Depth regularization: the authors should dive deeper on the depth regulariziation.\n  - First, nonlocal reuglarization term (or at least bileratal term) should be used as the edge-aware regularziaton. The gradient weighted one (Godard et al., 2017) is too sensitive to noises.\n  - Second, if remove depth regularization is better, the author should investigate how the network avoid ill-posed solutions. For example, what are the addtional sources that provide the regularziation, e.g., the network architecture.\n\n- Pose: the authors should use \"ground-truth\" poses for monocular sequences during training. It will exclude inaccurate poses and refocus to the depth network design.\n\n- Datasets: the authors should include less layout biased dataset, both indoor datasets such as ScanNet (training & testing) and outdoor datasets such as DIODE (testing only) into evaluation.", "broader_impact_concerns": "I do not see a potential ethical liability issue in the paper. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BTAjRYZuRN", "reviewer": "Reviewer_WQwG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n(+) This is a unique study on the architecture, component, and loss choices.\n(+) The paper concentrates on methods that use self-supervision signals like the photo-consistency of frames or feature consistency after warping using the predicted depth and  motion.\n(+) The authors re-implemented and reevaluated 16 state of the art methods.\n(+) The evaluation was done on real data in urban and real environments\n(+) The authors corrected previously identified mistakes in the ground-truth of KITTI-Eigen due to the positioning of Lidar vs camera and the sparsity of the Lidar.\n(+) The new dataset contains also two new evaluation measures that consider the distance to depth edges and completeness of depth edges.\n(+) The authors found out that ConvNeXt backbone architectures outperform all others.\n(+) Evaluation is also done in true 3D comparing depth predictions with point cloud positions.\n\n(-) It would be helpful to always clarify whether we talk about training or inference. For example, in the caption of Fig.~1 \"have much higher error as they do not use stereo inputs.\" It obviously means training since the paper is about mono-depth.\n\n\n(-) There is no study on the effect of what pose regression method is used. \n(-) It is unclear to the reader why the authors do not use COLMAP in training and testing.\n(-) This makes it evident that the task of monocular depth reconstruction is ill-defined.  Does monocular mean monocular video or single-frame? If it is self-supervised it has to be video.\n(-) If it means monocular video, why do the authors not evaluate any dense visual SLAM systems that do not use priors? One could use a baseline like plane sweeping stereo using the poses obtained by COLMAP.\n(-) Why do the authors not evaluate the methods on UNSEEN environments, meaning complete scenes like new cities where a robot has never been before? For example in the VO task, DroidSLAM trains only on TartanAir datasets and tests on EuRoC, ETH-3D, and TUM-RGBD.\n(-) There are smaller datasets like NBVC by Mordohai et al. IROS 2020\n \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper is about what matters in the architecture of a self-supervised monocular reconstruction. The main finding of the paper is that updating the backbone makes standard baselines outperform recently proposed systems. Moreover, the authors identify fundamental mistakes in the evaluation protocol. They provide code and a new dataset SYNS-Patches. \n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The results are worth publishing, but if published by themselves, they present a distorted view of reality in monocular depth reconstruction.\n\nThe following adjustments are absolutely necessary for publication:\n\n1. Explain and enumerate seen and unseen scenes in the new dataset.\n2. Devote a paragraph explaining the weaknesses above:\n2a If so many approaches were re-implemented why was no COLMAP used for training/testing?. \n2b Why was no prior-free approach tested that extracts video from successive frames on a monocular video. \n\nThe above is a good step toward publication.\nI would give a strong acceptance if \n1. the authors would evaluate a prior-free approach compared to learning based approaches on performance in an unseen environment (not just different parts of the same city in kitti).\n2. the authors would add a COLMAP option on one of the best performing approaches. Even a limited evaluation on this would be sufficient.", "broader_impact_concerns": "There is no broader impact session needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "V71qArvOHf", "reviewer": "Reviewer_hVhm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- *Writing*: The paper is overall well written and straightforward to follow, with clear tables and visualizations. The included charts help the reader understand the evidence for the claims made in the paper. The draft is in good shape\n- *Identifying the evaluation errors in prior benchmarks (KITTI-Eigen) and determining their impact. Proposing the new KITTI-Eigen-Benchmark*: Sequences of multiple computer vision papers tend to follow the evaluation methods (dataset splits, evaluation scripts) from one popular initial work for the sake of consistency. For practical reasons, comparisons often are done by taking numbers from a previous paper and evaluating the new method using the same evaluation code as before. This paper correctly argues that such an approach leads to errors in evaluation being propagated across many papers. This work follows Uhrig et al. 2019 in identifying the inaccurate ground truth in KITTI , as well as calling out that the ground truth depth is in the LIDAR reference frame rather than the camera reference frame. In Tables 4 and 5 the paper presents evidence of the discrepancy in the ranking of models based on their performance when the KITTI-Eigen is corrected.\n- *SYNS patches dataset*: A new testing dataset of images from diverse settings (indoor, outdoor, urban and rural) with high quality ground truth depths to evaluate methods outside of the automotive domain. The paper presents evidence of significantly lower performance on this testing domain than on the automotive KITTI.\n- *Thorough comparison of 16 prior depth estimation methods*: This paper presents an evaluation of 16 depth estimation methods using a modern backbone (ConvNeXt), the same optimization setting and per-method changes that increase each baseline’s performance. The main contribution of this evaluation is identifying that relative improvements are significantly lower when the comparison is well designed than what is originally reported.\n- *Establishing a new SOTA*: the improved benchmarks through both revising KITTI and the new SYNS-patches dataset as well as the evaluation of previous methods through a unified codebase represents a new picture state of the art in depth estimation. This paper proposes a new method (which appears to be based on the most significant method design decisions identified through the empirical investigation) which ranks highest based on the KITTI-Eigen-Benchmark and SYNS-patches.\n- *Identifying that depth regularization hurts performance* - on the fixed KITTI benchmark, this paper identifies that a common technique of smoothness and occlusion-based depth regularization in fact hurts performance, whereas it was likely useful on the original KITTI due to the inaccurate edge boundaries.\n\n**Weaknesses**\n- *Clarity of presentation in some cases*: please see more detail in “Requested Changes” below\n- *Potential overemphasis on the finding of the effect of different backbones*: While it is valuable to understand the effect of swapping more modern backbones with orders of magnitude more parameters, it is generally understood that doing so results in a \"rising tide lifts all boats\" kind of effect . I suggest the authors consider revising their claims (2nd to last paragraph in the intro) that swapping a backbone for an old method results in better improvement than changes in algorithm design.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper aims to gain a better understanding of recent developments in depth reconstruction methods from two perspectives: (1) by revising evaluation criteria through making changes and corrections datasets and evaluation metric implementations (2) revising the implementations of 16 prior works to fairly include design decisions that improve performance across the board. This paper presents an extensive empirical study to investigate these two directions. The main takeaways are \n- When trained using a similar optimization setup and in a unified codebase, all 16 methods get improved performance, and the relative performance improvement over SotA originally reported in the papers is significantly higher than the improvements obtained in the presented study. This indicates that when methods are made more equal, relative improvements are less significant. \n- There are flaws in the popular evaluation datasets and their accompanying evaluation code, that when corrected, result in a different ranking of prior methods based on their performance.\n\nThis paper also presents a novel dataset (SYNS-patches) of images with accurate ground truth depth from a diverse set of scene types, allowing for . The release of the codebase and the evaluation methodology presented in this paper has potential for high impact in the monocular depth estimation community.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Adjustments that would take a good empirical paper to a great one**\n- Can the authors please provide more details about the design of the “Ours” method? Given that a code release will follow with this paper this isn’t so significant, but it would be good to add to the paper.\n- While the changes made to the previous methods to make them more comparable do result in improved performance, what were the criteria used to pick which design changes to make to which algorithms? This would also be helpful if added to the paper.\n\n**Extremely Minor** \n- Small typo in conclusion: “the change is performance” → “the change in performance”\n- Zeki et al. citation needs to be corrected. Please check the correctness of the citations.\n", "broader_impact_concerns": "I do not have any concerns about the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ae1rZxomOG", "reviewer": "Reviewer_YWtU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- A benchmark with standardized/correct depth metrics is introduced. The proposed SYNS-Patches dataset seems to have high-quality depth ground-truth with fine details. \n- The ablation study is thorough, which includes backbone architecture/pre-training and depth regularization losses. The finding made by the paper that recent improvements in self-supervised depth estimation methods comes mostly from a better backbone network is interesting. This suggests research in this field needs to consider isolating the effect of backbone network when claiming for contributions in self-supervised losses.\n\n**Weaknesses**\n- Given the large number of methods that the submission aims to analyze, the paper didn't do a good job organizing and differentiating them. In its current state, it is difficult to keep track of the contributions each method made. A table of related work would help summarizing/grouping/differentiating their contributions in this case. For instance, what type of loss each method proposed / used and which backbone each method applied. The losses proposed by each method are sparsely discussed in Sec 4.3/Sec 5.1, and can be merged into a single table.\n- The task of self-supervised depth estimation is not well-motivated. For instance, given there are high-quality depth ground-truth (e.g., Omnidata/MiDAS), and supervised depth estimation methods available, why would one care about self-supervised depth estimation task.\n- Training dataset is limited to KITTI and data for indoor scenes (such as NYU/Scannet) are not considered or discussed. \n- Qualitative results in Fig 6 and 7 are not very accurate compared to state-of-the-art supervised depth estimation methods, such as MiDAS/DPT. Is there an explanation of why this happens?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission analyzes recent progress on self-supervised monocular depth estimation task. It aims to decouple the improvements brought by better architecture and better loss functions. To do so, it re-implements 16 recent methods within the same codebase, and performs systematic study on network architecture and loss functions. The experiments are performed on a newly-introduced depth estimation benchmark, which includes the error-corrected KITTI dataset and a new dataset called SYNS-patches (containing 1k+ images of various urban and natural scenes with high-quality ground-truth). One surprising finding is that simply replacing the backbone with an up-to-date model results in a much bigger gain, compared to the improvements brought by loss functions.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "**Besides the points in the weakness**\n- P14 conclusion “Even in the case where they do, the change is performance is drastically” => \"the change of performance is...\"\n- Tab. 6 is out of the page margin. May consider resizing the table or reducing the number of columns.\n- Intro-paragraph “ It is a core component enabling mid-level tasks such as Structure-from-Motion” -- does depth estimation help sfm, which by its definition takes correspondences as input instead of depth?\n\n**Changes that would strengthen the work**\n- In Fig 4, besides accuracy, another angle to look at the backbone ablation is the speed / training iterations. \n- Would be helpful to provide more details on the SYNS-patches dataset in terms of depth range (of the ground-truth) and density. Which depth metric is sensitive to depth range, and which are range-invariant?\n\n- Arguably, if calibrated stereo inputs are used at training time, it is not fair to call them self-supervised depth prediction, since ground-truth depth can be easily computed by triangulation / stereo matching algorithm. In that sense, another baseline for all the self-supervised methods would be “converting” stereo input to depth “pseudo”-ground truth, and then apply an additional supervised losses.\n", "broader_impact_concerns": "No.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 3 out of 4 reviewers are positive about the paper and suggest acceptance. The more negative reviewer would have liked to see more base architectures (e.g. transformer-based) and more analysis into the loss function. I think the paper as is has already enough quality and value for acceptance, but suggest a minor revision where the authors can incorporate the additional feedback provided in reviewer YWtU decision, and if possible also the loss function analysis requested by reviewer QF4o.

Congratulations!
Decision: Accept (poster)