RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions
Authors: Lingdong Kong
Shaoyuan Xie
Hanjiang Hu
Lai Xing Ng
Benoit R Cottereau
Wei Tsang Ooi
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: benchmark
monocular depth estimation
common corruptions
robustness
TL;DR: RoboDepth is a systematically designed robustness evaluation suite for monocular depth estimation under OoD corruptions.

Abstract: Depth estimation from monocular images is pivotal for real-world visual perception systems. While current learning-based depth estimation models train and test on meticulously curated data, they often overlook out-of-distribution (OoD) situations. Yet, in practical settings -- especially safety-critical ones like autonomous driving -- common corruptions can arise. Addressing this oversight, we introduce a comprehensive robustness test suite, RoboDepth, encompassing 18 corruptions spanning three categories: i) weather and lighting conditions; ii) sensor failures and movement; and iii) data processing anomalies. We subsequently benchmark 42 depth estimation models across indoor and outdoor scenes to assess their resilience to these corruptions. Our findings underscore that, in the absence of a dedicated robustness evaluation framework, many leading depth estimation models may be susceptible to typical corruptions. We delve into design considerations for crafting more robust depth estimation models, touching upon pre-training, augmentation, modality, model capacity, and learning paradigms. We anticipate our benchmark will establish a foundational platform for advancing robust OoD depth estimation.

RoboDepth: Robust Out-of-Distribution Depth Estimation under Corruptions
Lingdong Kongl:
Shaoyuan Xie
Hanjiang Hu*
Lai Xing Ng
Benoit R. Cottereau Wei Tsang Ooi
National University of Singapore 2CNRS @ CREATE
Huazhong University of Science and Technology
*Carnegie Mellon University SInstitute for Infocomm Research, A*STAR
IPAL, CNRS IRL 2955, Singapore
"CeCCo, CNRS UMR 5549, Université Toulouse III
Abstract
Depth estimation from monocular images is pivotal for real-world visual perception systems. While current learning-bor hased depth estimation models train and test on meticulously curated data, they often overlook out-of-distribution (OoD) situations. Yet, in practical settings especially safety-critical ones like autonomous driv- ing - common corruptions can arise. Addressing this oversight, we introduce a comprehensive robustness test suite, RoboDepth, encompassing 18 corruptions spanning three categories: i) weather and lighting conditions; ii) sensor failures and movement; and fii) data processing anomalies. We subsequently benchmark 42 depth estimation models across indoor and outdoor scenes to assess their resilience to these corruptions. Our findings underscore that, in the absence of dedicated robustness evaluation framework, many leading depth estimation models may be susceptible typical corruptions. We delve into design considerations for crafting more robust depth estimation models, touching upon pre-training, augmentation, modality, model capacity, and learning paradigms. We anticipate our benchmark will establish a foundational platform for advancing robust OoD depth estimation.
Introduction
Monocular depth estimation (MDE) involves predicting a scene's depth information from monocular images, without relying on data acquired from more sophisticated sensors [11, 28, 25, 37]. These images are predominantly captured using RGB cameras mounted on diverse platforms like drones, mobile robots, and vehicles [65, 41, 9, 30]. As an instrumental facet of visual perception, precise MDE paves the way for a broad array of applications. Bolstered by the rise learning-based paradigms, numerous MDE algorithms have emerged, demonstrating remarkable depth estimation performances on standard benchmark datasets [14, 52, 49, 8, 47].
However, the resilience of existing MDE models to out-of out-of-distriben distribbutioon (OoD) challenges is yet to be thoroughly explored, especially under the lens of real-world corrppiions such as adverse weather [19, 46] and sensor malfunctions [29]. The prevailing
bbased visual perception models often display heightened sensitivity to nuances lighting, noise, texture variations, among other factors, which are compromising the accuracy of depth predictions [18, 23]. The ability to generalize across new scenes, objects, and backgrounds, especially when they have not been part of the training data, is another pivotal challenge [40].
Despite the strides achieved on relatively pristine datasets [14, 49, 8], lacuna exists: a robustness benchmark tailored to foster the evolution of resilient and scalable MDE systems. In light of these
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

challenges, models dedicated to MDE often inadvertently embed systematic errors, stemming from real-world image imperfections like altered lighting, motion blur, shadows, and data compression. which the current MDE solutions rarely address effectively [4, 34].
Seeking bridge this gap, our contribution charts the inaugural path towards robust and reliable MDE, unveiling the K/7TI-C, NYUDepth2-C, and KITTI-S benchmarks. Contrasting prior works that merged datasets for cross-domain MDE [44, 33, 62] or devised adversarial patches to subvert MDE models [7, 4], our benchmarks meticulously simulate commonplace corruptions that are intrinsic to real-world settings. As delineated Fig. 2, we structure eighteen corruption varieties across three cardinal categories: i) weather and lighting conditions, ii) sensor malfunctions and movements, and tit) data processing complications. Further stratified by diverse severity, these corruptions encapsulate a gamut of scenarios fostering image distortions, texture shifts, or degraded visuals [18, 40J.
Given that MDE models intrinsically depend on lucid visual cues for depth inference, the aforementioned corrup- tions naturally impose significant hur- dles. Our preliminary analysis, visu- alized in Fig. showcases a spectrum of responses from distinct MDE model architectures when faced with diverse corruptions. Penetrating into these dy- namics is quintessential to understand- ing the underlying causes of perfor- mance faltering, enabling to archi- tect MDE models that are both robust and reliable. Pursuing this vision, we undertake a meticulous benchmarking of extant MDE models on these new datasets, embarking on an exhaustive study of their robustness vis-a- vis the spectrum of corruptions. We probe queries about the resilience of MDEE models to real-world corruptions, the influence of training input modalities, and learning paradigms- addressed in assess the fi- delity of our simulated corruptions and delve into the impact of texture-shiff corruptions (style alterations) gaug- ing MDE model resilience.
MonaDapch2
CADupth
92, 94
D4
Depthitats
FSRS-Depth Depth
ManyCopphh
MonoVit
RA-Depth
Mono
Bright
Dark
Fog
Frost
Controst
Sec.4.2. Concurrently,
Delocus
Glass
Motion
Zoom
Eastic
Guant
Gaussian
Impua
Shot
ISC
Ploelatte
JPEG
Figure 1: The depth estimation robustness (in terms of depth estimation error (DEE) defined in Sec. 3.3) under 18 corrup- tions in radar charts. Different MDE models exhibit diverse strengths and weaknesses against different corruptions that occur in the real world.
From our benchmark findings, we dis- till several intriguing insights and prof- fer recommendations to amplify robust- ness focusing on strategies like model pre-training, input resolution tuning, model sizing, complexity modulation, and corrupt-image fine-tuning (Sec. 4.3).
To encapsulate, our work offers the following seminal contributions:
We introduce RoboDepth, the first systematically designed robustness evaluation suite for MDE under data corruptions, sensor failure, and style shifts. See our repository at this link for more details.
We benchmark 42 state-of-the-arr MDE models from indoor and outdoor scenes, their robustness against corruptions, via three newly established datasets: KITTI-C, NYUDepth2-C, and K/77I-S. The corruption simulation toolkit has been open-sourced facilitate future development.
Based on our observations, we draw in-depth discussion and analysis on the design considerations of building more robust MDE models for reliable, scalable, and practical applications.
Furthermore, we initiated the RoboDepth Challenge [27], which garnered participation from over one hundred teams, underscoring the community's 's interest and the challenge's Comprehensive details about the competition can be accessed on our competition website at this link.
's relevance.

---

weather lighting
sonsor movement
processing
Defocus
Glass
Caunzian Nollnn
Imoulse No
Fog
Figure 2: Corruption taxonomy. We break down common corruptions in depth estimation scenarios into three categories: i) Weather and lighting conditions, such as sunny, low-light, fog, frost, snow, and contrast conditions. ii) Sensor failure and movement, such as potential blurs (defocus, glass, motion, zoom) caused by motion. iii) Data processing issues, such as noises (Gaussian, impulse, ISO) happen due to hardware malfunctions. Examples shown are from the proposed KITTI-C benchmark.
Related Work
Monocular Depth Estimation (MDE). Since the pioneering works [10, 12, 68, 16] first adopted deep neural networks to perform monocular depth estimation, sigmificant progress has been made in many aspects. Notable innovations include network architectures [31, 43, 66, 63, 21], optimization functions [17, 64, 6], internal constraints [60, 67]. multi-task learning [55, 22], geometry constraint [56, 51], and various sources of supervisions [44, 50, 33]. Based on the learning paradigm, most MDE methods can be split into supervised or self-supervised models. The former mainly focuses on indoor scenes and
ground truth from RGB-D cameras LiDAR sensors to train regression model [1, 35]; while the latter formulates MDE as a novel view synthesis task to minimize the photometric loss between stereo pairs or from monocular video frames [68]. Although promising results have been achieved, the robustness of MDE models under adverse scenarios is still unknown. risk of being vulnerable corruptions. In
Due the lack of relevant datasets, existing models this work, we fill in this gap by establishing comprehensive evaluation benchmarks and testing 42 MDE models from both indoor and outdoor environments to analyze their OoD robustness.
Robust MDE. To the best of our knowl- edge, only few works targeted robust learning of MDE and they focused on dif- ferent aspects. Ranftl al. [44] proposed a unified objective for merging multiple datasets with different depth scales and ranges for training robust models. Simi-
Severity Level
Color Quant Brighteos
lar works [58, 33, 53, 5, 62] resort to web stereo data or 3D movies to train MDE models and adapt them unseen datasets. Kopf et al. [29] estimate stable camera tra- jectories for hand-held cellphone videos. SC-Depth [50] generates pseudo-depth to refine depth details for scenes with dy- namic objects. Li et al. [32] proposed an attention module to choose scene-specific
Defocus Blur
Figure 3: Corruption severity level. We create versatile corruption sets with different levels of severity. Examples shown are from the proposed NYUDepth2- benchmark.
features for MDE on both indoor and outdoor scenes. SeasonDepth [19] contributed dataset with depth maps under sunny, cloudy, and foliage weather. Most recently, there are works [7, 4] design adversarial patches to attack MDE models. Conversely, we aim to test the MDE robustness to corruptions that occur real-world environments. We establish the first benchmark of this kind and incorporate ample number of MDE models for in-depth analysis.
Corruption Robustness. ImageNet-CC [18] the pioneering work in this line of research which bench- marks classical image classification models to common corruptions and perturbations. Follow-up studies extend on the aspect to other visual perception tasks, e.g., object detection [40], segmenta- tion [23, 26, 38], navigation [3], video classification [61], and pose estimation [54]. The essentiality of evaluating model robustness has been repeatedly validated. Since we are targeting a different task, i.e., MDE, most of the well-studied corruption types become less realistic or suitable for such data
3

---

format. This motivates us to explore new taxonomy for defining more proper corruption types for MDE. In this work, we contribute new datasets and benchmarks for probing the MDE robustness.
3 The RoboDepth Benchmark
In this section, we first introduce the taxonomy of corruptions included in our benchmarks (Sec. 3.1). We then elaborate on more details of the proposed datasets (Sec. 3.2) and corresponding robustness evaluation metrics (Sec. 3.3). Examples from our datasets are shown in Fig. 2, Fig. 3, and this page.
3.1 Corruption Definition
Weather & Lighting Condition. The cameras on drones or vehicles operating under different weather and times day capture distribution-shiftee images which are rare or lacking in current MDE datasets [14, 49]. To probe the robustness of MDE models under adverse weather and lighting conditions, we simulate six corruptions, i.e., "brightness', 'dark', 'fog'. 'frost', 'snow', and 'contrast', which commonly occur in the real-world environment. Compared to clean images, these corruptions tend to affect the intensity and color of the light source, leading to hazy, blurry, and noise-contaminater images, which increase the difficulties for the MDE model to make accurate depth predictions.
Sensor Failure Movement. An MDE system must behave robustly against motion perturbation and sensor failure to maintain safety requirements for practical appli- cations. To achieve this pursuit, we mimic four motion-related cor- ruptions, i.e., 'defocus', 'glass', 'motion', and 'zoom' blurs; we also generate images under 'elastic transformation' and "color quanti- zation', which happen during sen- sor malfunction. These corrup- tion types often associated with issues including edge distortions, contrast loss, and pattern shifts.
mCE (%)
mCE
126
178
98
114
82
DEE (96)
50
8.5
9.8
11.1
12.4
13.7
15.0
4.0
10.6
17.2
23.8
30.4
37.0
(%)
mRR (%)
mRR (%%
94
83
90
86
82
65
DEE
78
(9)
(96)
DEE (%))
DEE (%)
Data & Processing Issue. Data collection and transmission are in- evitably associated with various sources of noise and potential loss of information. We include four such random variations, i.e., 'Gaus- sian', 'impulse', 'shot', and 'ISO' noises. In addition, investigate the degradation caused by 'pixelate' and 'JPEG compression" which are common corruptions in handling image data. Compared clean im- ages, the roise-contaminated data introduce errors the intensity val- ues of pixels, leading to a grainy or speckled appearance. The pixelation and lossy compression tend lead to loss of detail and clarity in the image and can result in visible artifacts, such as blockiness or blurring.
50
42
40
34
26
18
Dight
Fog
Frod
Seor
Coreus
Quart
inc de
50
Bightt
Dar
Claen
Modor
Shot
JPSC
(a) KITTI-C
(b) NYUDepth2-C
Figure 4: Benehmarking results of 42 MDE models KITTI- and NYUDepth2-C. Figures from top to bottom: the depth estimation error (DEE) vs. [1st row] mean corruption error (mCE), [2nd row] mean resilience rate (mRR), and [3rd row] sensitivity analysis among different corruption types.
3.2 Benchmark Establishment
KITTI-C. Based on the KITTI Vision Suite [14], we establish robustness benchmark for outdoor MDE. We simulate the defined corruptions using data from the KITTI val set under Eigen's 's split. Similar to [18], design five severity levels for each corruption to further consolidate the evaluation

---

of robustness changes. As result, this robustness probing dataset has a total number of 62, 730 RGB images with resolution of 192 640. We also include the high-resolution version (320 1024) for evaluating the robustness of MDE models which take larger images as the input.
NYUDepth2-C. We construct benchmark for robust indoor MDE based NYU Depth V2 [49]. 15 of the defined corruptions are used, excluding 'fog', 'frost', and 'snow" which rarely occur in the indoor scenes, Since the indoor environments are less variant than outdoor ones, we only include four severity levels for each corruption. To sum up, this dataset contains 39, 240 images of size 480 x 640, which cover 23 different types of indoor scenes, such as basement, bathroom, bedroom, study, etc.
Style changes, consisting mostly of texture shifts, have proven helpful for analyzing model robustness [18]. To further investigate the root cause of MDE robustness degradation, we form another collection based on KITTI [14] with stylized images via the style transfer model AdaIn [20]. This dataset has 8, 364 images from 12 styles, including 'cartoon', 'digital art', 'ink painting', "kids' drawing', 'murals', 'oil painting', 'penciling', "shadow play', 'sketch', 'stained glass', 'relief", and "water color'. Due space limitations, please refer Appendix or this page for additional examples.
KITTI-S.
Simulation Toolkit. To facilitate a similar study on other MDE datasets, we have open-sourced the code at this link for simulating corruptions and style shifts given arbitrary "clean" images.
3.3 Evaluation Metrics
Depth Estimation Error (DEE). We combine Abs Rel (error rate) and 81 (accuracy), the two main measures defined in [10, 36], into a unified metric as DEE = Abs Rel-ór+1 which is constamtly used as 2
the indicator of depth estimation error in our benchmark. See Appendix for more formal definitions.
Corruption Error (CE). We follow [18] and use the mean CE (mCE) as the primary metric in comparing models' robustness. To normalize the severity effects, we choose MonoDepth2 [17] and AdaBins [i] as the baseline models for the KITTI-C and NYUDepth2-C benchmarks, respectively. The CE across L levels of severity and mCE across N corruption types can be calculated as follows:
Ele (D CE;
1 mCE = N 1=1
N
Σ CE
ΣL (DEE
Resilience Rate (RR). We define mean RR (mRR) as the relative robustness indicator for measuring how much accuracy can an MDE model retain when evaluated under the corruption scenarios, i.e.
EL (1- LX - DEE
N
RR,
mRR
RR
where DEEgtean denotes the task-specific accuracy (or error rate) score on the "clean" evaluation set.
4 Experiments
4.1 Benchmark Configuration
Depth Estimation Models. We benchmark 42 depth estimation models and model variants, which cover most of the open-source MDE models so far. 32 of them are for outdoor MDE and the remaining 10 are for indoor MDE. More detailed descriptions of these models are attached in the Appendix.
Datasets. All benchmarked depth estimation models have been trained on the official training splits of the KITTI [14] (for outdoor MDE) or NYU-Depth V2 [49] (for indoor MDE) datasets, and are tested accordingly on the official va! splits and also our proposed KITTI-C, NYUDepth2-C, and KITTI-S datasets. Additionally,
resort to the real-world ACDC [45], nuScenes [2], Cityscapes [8]. and Segyy-Cityscaps [46] datasets for validating the fidelity of our simulated corruptions. Our datasets and model evaluation toolkit can be downloaded from this page.
Evaluation Protocols. To avoid any unfairness in the MDE robustness comparison, we unify the common configurations among different candidate models, such as backbones, data augmentations, and post-processing. We use public checkpoints whenever possible and reproduce the reported results based on official settings. More details on this aspect are included in the Appendix.
5

---

Table I: Self-Supervised MDE Robustness Benchmark consisting of 32 models on KITTI-C. The
mCE and mRR scores are given in percentage (%). Blocks from top to bottom: [1st]: The baseline Mono RIS [17]: [2nd]: Methods w/ monocular inputs; [3rd]: Methods w/ stereo inputs; [4th]: Methods w/ monocular + stereo inputs. Bold: Best in column. Underline: Second best in column.
The best score of each metric across three categories is shaded in color.
Metbud
mCE
Overall Robustness
mRR
mDEE
mCE
Weather & Lightning
mRR
mDEE
mCE
Sensor & Movement
mRR
mDEE
IBC
Data & Processing
mRR
mDEE
MonooDepthaaa Ria [17]
100.00
84.46
0.256
100.00
84.37
0.257
100.00
90.33
0.204
100.00
78.66
0.307
MonoDeptho [17]
119.75
62.50
0.294
146.17
78.58
0.209
109.50 80
76.48
0.945
MonoDepthng [17]
106.06
82.44
0.270
108.40
80:38
0.2288
115.99
85.50
0.242
02.25
$1.36
0.279
MonoDeepth uso [17]
113.43
80.59
0.228
104.33
63.28
0.265
128.
82.45
0.272
107.07
76.05
0. 0.329
MaskOccee ei [45]
$2.97
0.267
100.98
0.257
108.85
87.67
0.228
102.30
77.11 17.12
0.919
DNet kus [59]
104.71
83.34
0.265
103.04
83.56
0.263
115.56
86.07
0.241
95.53
90.39
0.291
CADepth [60]
110.11
80.07
0.286
102.50
82.04
0.268
119.67
83,
0.252
108. LG
74:25
0. 335
HR-Depth [39]
103.73
82.93
0.261
100.41
63.82
0.256
83,32
0.242
91.76
79.61
DIFFNet [67]
94.96
85.11
0.233
79.33
89.53
0.196
121. 09
81.68
0.267
80.86
00
0. 0.257
MunyDepthe: q [56]
105.41
$9.11
0.271
104.97
$4.15
0.262
102.57
90.06
0.210
108. 70
75.10
0.341
PSRE-Dephh [22]
99.05
83.86
0.253
80.12
87-30
0.210
106.66
75.53
0.327
MonoViT [66]
79.33
89.15
0.197
72.92
91.16
0.179
51 62
92.67
0.165
83.47
53. 61
0.217
MonoViT uR [66]
74.95
89.72
0.187
72.24
90.78
0.177
74.60
93.86
0.150
78.02
$1..1
0.234
DynaDepther kis [64]
110.38
81.50
0.280
102.92
83-45
0.263
81.67
0.279
96.95
79.33
0.288
DynalDephess ess [64]
119.99
77.98
0.305
105.81
81.70
0:2276
788 D8
0.307
110.75 7M
74.15
0.342
RA-Depih [2]
112.73
78.79
0.288
$9.39
85.31
0.229
137. 17
78. 78.19
0.293
111.32
72 82
0.312
TriDepthgns [6]
109.22 26
81.56
0.280
115.07
80.79
0.287
104.61
88,
0
108. 10
75.
0.53T
Lite-Moto muy [63]
92.92
86.69
0.253
90.57
0.219
90.87
0.196
92.71
80.90
0.254
Lite gaal [63]
100.34
84.67
0.251
98.99
85.52
0.243
100.8T 87
87.12
0.221
92.16
$1.37
0. 0.250
Lite-Mone [63]
83.16
85,99
0.235
80.20
87-29
0.221
96.71
SKU, 232
0.197
98..9
50:26
0. 282
Lite-Mooor.op [63]
90.75
85.54
0.232
83.91
87.23
0.217
0.188
96.38
75.95
0.291
MonolDepehaaz aus [17]
117.89
70.15
0.307
111.08
81.79
0.2883
121. .92
84.95
0.255
1:20.08
70:41
0.383
MonoDoptliz oue [17]
128. 88
79,20
0.327
145..7
77:96
0.337
110. 41
91.45
0. 0.225
1.31.99
08: 20
0.120
MonoDepths me 1171
111.46
81.65
0.279
111.75
81.00
0.284
127.51
82.96
0.268
95.16
0.285
DepthHinms [57]
111.41
80.08
0.290
98.81
83.22
0.2662
122.00 07
83.78
0.257
112.36
73.22
0.351
DepthHiits wue [57]
141.61
73.18
0.366
155.51
73.32
0.363
121. 71
86,
0.251
147.60
18..5
0.484
DepthfHimts un [57]
112.02
79.53
0,287
63.30
0.254
135.82
79.89
0.284
103.88
75.41
0.521
Monoliepehh kis
[17]
124.31
75.36
0.334
108.43
80.86
0.2285
127. 18
81.75
0.269
1.36.34
62. 14
0. 0..44
MonoDoptlhe ouee [17]
136.25
76.72
0.345
165. 90
72:79
0.378
106.08
92.17
0.213
1336.77
65 20
0. 0.113
MonoDepth, me [171
106.06
$2.44
0.270
109.95
80.98
0.288
115.99
83,59
92.25
$1.36
0.270
CADepih [X0]
118.29
76.68
0.318
1011/98
$1.40
0.277
119.87
83.90
0.253
1.31.33
84.64
0.425
MonoNiT [66]
75.39
90.
0.184
72.54
91.55 83
0,172
78 42
93. 93.
0.159
75.21
84. 35
0,223
DEE
Mono
Stereo
StereO
0.44
0.35
0.26
0.17
0.08
Bright
Dark
Fog
Frost
Snow
Contra
Defocus
Glass
Mation
Zoom
Elastic
Quant
Gauss
Impul
Shot
Is0
Pixel
JPEG
Figure 5: The depth estimation robustness comparisons among different input modalities. Mono: monocular images; Stereo: stereo pairs; and Mono+Stereo; both monocular and stereo images.
4.2 MDE Robustness Probing
In this section, based on our benchmark, we aim to understand the corruption robustness among different MDE models by answering the following representative questions.
Q-1: "Are existing MDE models robust under real-world corruptions?" A: No, a certain extent. Our benchmark results in Fig. 4 reveal that state-of-the-art MDE models are at risk of being vulnerable to corruptions. Existing MDE models, either from indoor or outdoor scenes, show a flattened or even inverse relationship between the DEE scores and robustness metrics. Due to the lack of suitable robustness evaluation suite, current MDE models are over-fitted on "clean" sets while ignoring the OoD scenarios that are likely to occur in the real world. Nevertheless, we observe varying behaviors of different MDE models under different corruption types (see Fig. 1), caused by different design choices on model architecture. The results from Tab. 1 and Tab. 2 further show that the Transformers- based MDE models exhibit better robustness compared to conventional CNNs. Diving deeper, we can observe from the per-severity enror rates in Fig. 9 that the above conclusion holds true for most corruption types under different severity levels. The qualitative results shown in Fig. 10 also validate that models with long-range receptive fields, such as Mono [66] (74.95% mCE) and Lite-Mono

---

Table 2: Supervised MDE Robustness Benchmark consisting of 10 models the NYUDepth2- C dataset. The mCE and mRR scores are given in percentage (%). Bold: Best in column. Underline:
Second best in column. The best score of each metric across three categories is shaded in color.
Method
mCE
Overall Robustness
mRR
mDER
mCR
Weather & Lightning
mRR
mDRF
mC..
Sensor & Movement mRR
mDRF
mCC.
Data & Processing
mRR
mDRE
AdaBins Eas [1]
100.00
85.80
0.208
100.00
92.42
0.179
100.00
87.97
0.219
100.00
80.41
0.286
BTS asu [31]
122.78
0.202
125.51
0.228
84.21
73.30
Adakins zm [1]
134 6B
81-02
0.313
140:30)
58.64
0:254
131.88
85.63
0.277
134.37
74.11
0.376
DPT wna [43]
83.22
95. 25
0.177
11.66
96..57
0.106
61.22
96.
0.168
79.41
93.00
0.191
SimlPU axe [34]
200.17
241.79
12.220
0-421
94.96
SimIPU truerkie [34]
163.06
85.01
0.357
190.62
87.52
01.338
166.90
86.97
0.343
145.
81.79
0.382
SimIPU kmm [34]
173.78
91.64
0.370
210:25
91.81
0.368
170.75
95.37
0.344
158.56
87.81
0.396
SimIPU WasssOppn [34]
169
0.338
159.50
0.380
DepthFormer swant-nk [35]
106.31
87.25
0.287
122 01
89. 10
0.220
107. 22
88.
0.223
97.61
84.78
0.258
DepthFormer Swartizk
[35]
63.47
94.19
0.139
70. 11
95.84
0.124
69.32
90.25
0.148
54.29
94.29
0.138
DEE
Protrain
No Protrain
0.44
0.35
0,26
0.17
0.08
Bright
Dark
Fog
Frost
Snow
Contra
Defacas
Glass
Motion
Zoom
Elastic
Quant
Gauss
Impul
Shot
150
Pixel
JPEG
DEE
Low-fes
High-Res
0.44
0.35
0.26
0.17
0.08
Bright
Dark
Fog
Frost
Snow
Contra
Defacusa
Glass
Mation
Zoom
Elastic
Quant
Gausa
Impul
Shot
Is0
Pixel
JPEG
Figure 6: Depth estimation robustness of MonoDepth2 [17] under different training configurations.
[top]: Different pre-training techniques. [bottom]: Different resolutions of the input images.
[63] (90. 75% mCE), can better maintain accurate depth predictions under corruptions coped with texture shift, edge distortion, and noise contamination.
"Which MDE modality is the most robust against corruptions?" A: MDE models trained with purely monoeular images rather than stereo pairs as input have better robustness, shown in Fig. 5. Here we analyze the robustness of MonoDepth2 [17] variants trained with different input modalities and observe that using stereo or a mix of monocular and stereo images leads to robustness degradation. MDE models trained with stereo pairs may rely more on the scene structural consistency between left-right pairs, which could be destroyed when dealing with corrupted data. Such constraints, however, could be relaxed if the depth estimation model is trained only on monocular sequences.
Q-2:
Q-3: "Which learning paradigm is better in terms of corruption robustness? ? A: We find different sensitivities between the supervised and self-supervised MDE paradigms. From the third row of Fig. 4 can observe that the self- self-supervised MDE models are less sensitive to lighting changes (texture shift) and motion blurs (local distortion), compared to the supervised models. On the other hand, both models suffer from noise perturbation and behave robustly against lossy image compression.
Q-4: "Are simulated corruptions realistic enough to mimic real-world scenarios?" ?" A: Yes, to a certain extent. We verify this statement by augmenting MDE models via fine-tuning with our corruptions. Fig. 7 (right) shows that the generated corruptions are helpful for closing the distribution gap between the clean training set and real-world datasets [2, 8, 46], especially for the weather contaminated data in the Foggy-City Citysappsss dataset [46]. This supports that our datasets are of high fidelity and scalability. Additionally, we show in the Appendix that the simulated corruptions, especially for the types that are related to weather and lighting changes, are of good quality compared to some real-world corrupted data [2, 45, 8].
Q-5: "Are MDE models robust against style changes?" A: We resort to the KITTI-S dataset (see Tab. 3) for probing model characteristics and find that most MDE models perform worse on this stylized dataset than on KITTI-C. We conjecture that the loss of texture cues for objects (e.g. car and building) for KITTI-S data causes the main impediment. We also observe that MDE models with
7

---

mCE (9)
mCE (%)
Abs
MonoDepth2
w/ Ours
136
136
0.420
0.380
88
88
0.300
72
72
0.260
56
Param
56
FLOPs (G)
nuScenes
nuScemes
Cityscapes
0.5
(M)
(Nigh0
(Foggy)
Figure 7: MDE robustness comparisons based on [left]: Model size of parameters) and [middle]:
Model complexity (FLOPs); and [right]: Corruption fidelity verification by fine-tuning MonoDepth2 [17] with our corrupiions on KITTI and testing it on the official val splits of nuScenes [2] (including its validation set and the nighttime split of the validation set) and Foggy- Cityscapes [46].
Table 3: The Depth Estimation Error (DEE) of 18 models KITTI-S, Bold: Best in column.
Underline: 2nd best in column. Blue Best in row. Red 2nd best in row. Green Worst in row.
mDEE
Digit
Ink
MonoDDepthgss [17]
365
-324
.434
.351
.259
.328
-418
.385
.416
.255
.317
MonoDepth2 rp: [17]
.378
.288
MastOceRus kus [48]
.368
.358
.356
.200
.265
.358
.375
.333
.314
.578
.554
.288
.381
DNet [59]
.100
.411
.381
.293
.20
.389
.330
.422
.250
.608
553
290
.516
CADspih [60]
.006
.466
.380
379
250
.121
413
4700
.377
.885
545
2:2
4011
HR-Dapth [39]
DIFFNet [67]
2280
3900
ManyDepth [56]
.323
.251
.373
212
2015
.344
-000
.300
.343
301
.553
310
.000
FSREDspth [22]
.253
.275
271
2554
221
310
220
270
.001
315
.17
261
352
MonoVIT [66]
.388
.179
229
252
196
203
257
208
325
356
205
221
Dyma [64]
3885
DepaDe [64]
.447
.502
408
289
.877
.398
.437
492
.372
.505
.585
.400
.389
RA-Dupth [+]]
.365
301
.355
3544
262
.10
.310
.312
.425
.372
.176
.379
.455
Triopphess
.379
-311
403
2256
2177
.352
480
.318
.517
.573
209
A36
Lite-Monona [63]
285
Lite-Mono seal [68]
.345
374
.307
335
Lite-Mono aae [33]
288
234
.313
262
218
.211
241
291
240
.459
.476
220
.244
Lite Mano [63]
.233
208
.35
.249
306
385
258
363
210
.458
.007
263
.268
global awareness, e.g. Lite-Mono [63] and Min [66], suffer less degradation in this case. Due to the page limit, kindly refer to the Appendix for more detailed findings on this style-shifted dataset.
4.3 MDE Robustness Enhancement
Pertaining strategy tends to help improve MDE robustness. We observe that transferring knowl- edge from other tasks, such as classification and segmentation, brings both strengths and weaknesses to MDE robustness. Fig. 6 (top) highlights that MDE models pre-trained object-centric datasets, e.g. ImageNet, are more robust against weather and lighting changes (except for 'snow") and data processing noises, which are mostly texture-shifted corruptions. Motion and sensor corruptions, however, contain more edge and object distortions and could be eased by models without ImageNet pre-training. More concrete results in Tab. 1 imply that the CNN-based MDE models like Mon- shape-biased biased when pre-trained on object-centric datasets. More evidence from the KITTI-S benchmark (see Tab. 3) further verifies this finding, where the stylized data (texture-biased) are causing severe degradation for models with ImageNet pre-training.
Depth2 [17] could become more shape-
Training on high-resoe higl-resolution images yields more robust MDE models. From Fig. 6 (bottom) we observe that MDE models trained with higher resolution inputs will likely yield more robust feature learning (relative 30% better) on noise-contaminated, corruptions (Gaussian, impulse, shot, ISO noises). Since these noise contamination types mainly affect the global pixel distribution instead of the local ones, the CNN-based MDE models trained with high-resolution images likely to capture more fineggriieed information to suppress the degradation caused by noises. For more details, kindly refer the per-corruption scores of benchmarked models in the Appendix.
Larger model size might not lead better MDE robustness. is often intuitive that larger models are likely to learn more general representations and lead to better performance unseen data. However, we observe from Fig. (left) that MDE models with more trainable parameters are getting less robust, mainly because they deliberately tuned towards clean distribution and thus

---

Clean
.20
.116
137
.217
.144
156
.99
.91
175
.170
.121
.62
.285
.285
.25
.282
.133
.167
+Bright
.188
223
139
.218
.377
164
.222
.218
.90
.172
.127
.667
.269
.285
2661
.271
.136
.660
+Dark
144
.599
193
.226
.356
235
.243
224
210
.191
.133
.166
.274
.236
2449
.256
.144
.15
+Fog
.177
230
122
.211
.389
133
.244
.206
178
.167
.124
.770
.327
.350
309
.348
.137
192
+Snow
130
249
145
.166
.134
213
.198
.175
166
.168
.127
.669
.318
.331
268
.320
.129
.80
+Contr
123
278
126
.228
.117
123
.259
222-
225
.174
.127
.181
.330
.335
308
.343
.137
.811
+Defoc
155
282
153
233
387
173
.132
.158
165
.158
.141
207
.349
3369
337
.363
.136
180
Motio
135
262
147
.221
.313
167
.151
.158
129
1449
.126
168
.321
.318
294
.324
.131
171
+Zoom
178
348
179
347
.366
201
.181
205
153
.140
1.60
220
.407
397
390
.41.
.163
202
Elasti
126
238
145
.216
394
195
.118
.195
203
.176
.124
166
347
3.61
326
.372
.131
162
Gauss
131
184
177
.155
395
246
207
190
183
1.1
.129
159
166
.172
145
141
.132
140
+mpul
133
.97
190
.221
.397
254
.822
.170
169
.68
-130
160
.89
.147
156
165
.130
141
+hott
129
.181
164
.204
.381
217
.922
.173
170
.15
.129
159
.175
1.82
145
.44
.131
142
+SO
136
.181
195
.266
.007
269
271
180
227
198
1:30
157
188
199
161
.151
.133
138
-Pixel
.129
.222
144
2.30
.453
173
.07
193
193
173
24
177
300
316
26.3
.308
.127
1599
+JPEG
.131
201
160
.205
.397
227
217
208
187
30
165
.5
255
224
.238
.133
.144
+All
.25
.555
133
.17
.553
136
148
15
144
.51
22
150
147
144
135
140
.129
138
Bright
Dark
Fog
Frost
Snow
Contr
Defoc
Glass
Motia
Zoom
Elasti
Quan
Gouss
Impul
Shot
IS0
Pixel
JPEG
Figure 8: The Per-Corruption Error scores of Mono [17] fine-tuned with typical corruption as data augmentations and tested on each of the corruption types. For each column (corruption) in this figure: The darker the color, the lower the Abs Rel score (better robustness), and vice versa.
Bright
Dark
Fog
Frost
Snow
Contrast
0.12
0.329
0.85
0.000
0.43
0.004
0.99
0319
0.15
0200
0140
Defocus
Glass
Motion
Zoom
Elastic
Quant
0145
0.60
0415
0.19
0.500
0.93
0.55
0370
0335
0.72
0.5
0325
0200
0255
0162
0117
0.05
0.42
3.00
0.055
0.03
0.12
0.101
2
Geussian
Impulse
Shot
IS0
Pixelate
JPEG
0.14
0410
0415
0.422
0.20
0264
0340
0333
0214
0200
0254
0,21
0.66
0184
0154
100
0093
0.158
3.
03555
5
2
ManaDepth2
HR-
Depth
RA Depth
CADspth
DIFFNet
MonaViT
Lite-Mano
Figure 9: The Per-Severity Error Rate of eight typical MDE models [17, 39, 22, 42, 60, 67, 66, 63] trained on the clean K/777 dataset [14] and tested on each of the corruption set in K/TTI-C. For each subfigure (corruption) in this figure: The horizontal axis denotes severity levels from one to five. The vertical axis denotes the depth estimation error (Abs Rel). Best viewed in color.
losing generalizability. Our results suggest that a modcrate-sizee sized model with suitable corruption suppression modules is likely to yield the best possible trade-off between robustness and efficiency.
Model complexity shows direct correlation with MDE robustness. Based Fig. 7 (middle) we reveal that higher training complexity tends to lead to better robustness. This is mainly because exist- ing MDE models with high FLOPs per parameter are either trained with high-resolution monocular inputs or contain the computationally intensive self-attention mechanism and yield more stable depth predictions (as analyzed in previous discussions) but cause heavier training overhead.
Training with corrupted images does not always enhance model robustness. The robust fine- tuning study in Fig. 8 shows that overfitting the evaluation distribution (e.g. train and test on the same

---

Input
MonoDepth2
DIFFNet
RA-Depth
Ute-Mono
MonoV/
Frozt
Contrast
Delocus
Impulls
Shat
150
Poebts
JPEG
Figure 10: Qualitative results of representative MDE models [17, 67, 42, 63, 66] under defined corruptions in the KITTI-C benchmark. Best viewed in color and zoom-ed in for details.
corruption) may help improve MDE robustness for this specific type of corruption, as shown from the darker-colored cell in the diagonal. However, those corruptions with severe perturbations (weather and movement) will likely lead to sub-par performance on other tested corruptions and even hurt the overall robustness. This indicates that sophisticated considerations, e.g. case study, must be made before applying corruptions as data augmentations to the training stage.
Discussion & Conclusion
Throughout this research, we introduced the RoboDepth benchmark, dedicated tool tailored for probing the OoD robustness of MDE models under range of corruptions. By crafting three distinct datasets (KITTI-C, NYUDepth2-C, and KITTI-S) coupled with two metrics (mCE and mRR), we offered comprehensive landscape for gauging both indoor and outdoor MDE robustness. Our findings underscore the imperative nature of robustness evaluation in the realm depth estimation. Moreover, by dissecting various influential factors encompassing architecture, modality, pre-training approaches, resolution, and model capacity we provided insights to fortify model resilience against corruptions. In essence, our endeavors methodologics. As we navigate forward, our ambition to augment the breadth and granularity of benchmark, aiming encompass a wider spectrum of MDE models and corruption variants.
compass directing the evolution of robust MDE
Potential Limitations. Despite our extensive evaluation of diverse MDE models across expansive corruption sets, and validating the authenticity of our simulated corruptions, certain avenues remain uncharted. Notably, our current framework does not accommodate scenarios wherein multiple corruptions manifest simultaneously. Additionally, evolving from rigidly defined five severity levels to a more nuanced, continuous scale might offer deeper insights into MDE robustness. These unexplored terrains present intriguing prospects for subsequent research endeavors.
10

---

Acknowledgements
This research is part of the programme DesCartes and supported by the National Research Founda- tion, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
References
[1] Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In IEFF/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 4009-4018, 2021. 3, 7, 26, 34, 35, 36
[2] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Veniee Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 1121111661,, 2020. 5, 7, 8, 22
[3] Prithvijit Chattopadhyay, Judy Hoffman, Roozbeh Mottaghi, and Aniruddha Kembhavi. Robustnav: Towards benchmarking robustness in embodied navigation. In /EEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 15691-15700, 2021. 3
[4] Hemann Chawla, Kishaan Jeeveswaran, Elahe Arani, and Bahram Zonooz. Image masking for robust self-supervised monocular depth estimation. arXiv preprint arXiv:2210.03357, 2022. 2,
[5] Weifeng Chen, Shengyi Qian, and Jia Deng. Learning single-image depth from videos using quality assessment networks. JEEE/CVF Conf. Comput. Vis. Puttern Recog, (CVPR), pages 5604-5613, 2019.
[6] Xingyu Chen, Ruonan Zhang, Jiang. Yan Wang, Ge Li, and Thomas Li. Self-supervieed monocular depth estimation; Solving the edge-fattening problem. In JEEE/CVF Eur. Conf. Comput. Vis. (ECCV) Conf. Appli. Comput. Vis. (WACV), pages 5776-57785 5786, 2023. 3, 6, 8, 26, 30, 31, 32, 33, 37
[7] Zhiyuan Cheng. James Liang, Hongjun Choi, Guanhong Tao, Zhiwen Cao, Dongfang Liu, and Xiangyu Zhang. Physical attack monocular depth estimation with optimal adversarial patches. In Eur. Conf. Compur. Vis. (ECCV), pages 514-532, 2022. 2,3
[8] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schicle. The cityscapes dataset for semantic urban scene understanding. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 3213-3223, 2016. 1, 7, 22
[9] Xingshuai Dong, Matthew A. Garratt, Sreenatha G. Anavatti, and Hussein A. Abbass. Towards real- time monocular depth estimation for roboties: A survey. IEEE Trans. Intell. Transport, Syst. (TITS), 23(10): 19900-6699-,,,,,,, 2022.
[10] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Ady. Neural Inform. Process. Syst. (NeurIPS), 2014. 3,5
[11] Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew J. Davison, Jörg Conradt, Kostas Danillidis, and Davide Scaramuzza. Event-based vision: A survey. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), 44(1):154-180, 2022.
[12] Ravi Garg, BG Vijay Kumar, Gustavo Carneiro, and Ian Reid. Unsupervised cnn for single view depth estimation: Geometry the rescue. In Eur. Conf. Comput. Vis. (ECCV), pages 740-756, 2016.
[13] Timnit Gehru, Jamie Morgenstern, Briana Veechione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé lii, and Kate Crawford. Datasheets for datasets. Communications of the ACM, 64(12)886-92. 2021. 15
[14] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In /EEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 3354-3361, 2012. 1,4, 5.9
[15] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In Int. Conf. Learn. Represent. (ICLR), 2019. 22
[16] Clément Godard, Oisin Mac Aodha, and Gabriel Brostow. Unsupervised monocular depth estimation with left-right consistency. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 270-279, 2017.
11

---

[17] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel J. Brostow. Digging into self-supervised monocular depth prediction. In /EEE/CVF Inf. Conf. Comput. Vis, (ICCV), pages 3828-3838, 2019. 3, 5. 6, 7, 8, 9. 10, 24, 28, 29, 30, 31, 32, 33, 37, 38, 41, 42, 43
[18] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common comuutiions and perturbations. In Int. Conf. Learn. Represent. (ICIR), 2019. 1, 2, 3, 4,5, 5,
[19] Hanjiang Hu, Baoquan Yang, Zhijian Qiao, Shiqi Liu, Ding Zhao, and Hesheng Wang. Scasondepth: scason monocular depth prediction datasct and benchmark under multiple environments. In Int. Conf. Mach. Learn. Worksh. (ICMLW), 2022. 3,
[20] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 1501-1510, 2017. 5, 22
[21] Adrian Johnston and Gustavo Carneiro. Self-supervised monocular trained depth estimation using self- attention and discrete disparity volume. In /EEF/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 47.56-4765, 2020.
[22] Hyunyoung Jung, Eunhyeok Park, and Sungjoo Yoo. Fine-grained semantics-aware representation en- hancement for self-supe monocular depth estimation. In /EEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 12642- 12652,, 2021. 3, 6, 8, 9, 25, 30, 31, 32, 33, 37
[23] Christoph Kamann and Carsten Rother. Benchmarking the robustness semantic segmentation models. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 8828-888 2020. 1,3
[24] Oguzhan Fatih Kar, Teresa Yeo, Andrei Atanov, and Amir Zamir. 3d common corruptions and data augmentation. In JEEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), pages 18963-18974, 2022. 21
[25] Lingdong Kong, Youquan Liu, Runnan Chen, Yuexin Ma, Xinge Zhu, Yikang Li, Yuenan Hou, Yu Qiao, and Ziwei Liu. Rethinking range view representation for lidar segmentation. In JEEF/CVF Int. Conf. Comput. Vis. (ICCV), pages 228 228-240, 2023.
[26] Lingdong Kong, Youquan Liu, Xin Li. Runnan Chen, Wenwei Zhang, Jiawei Ren, Liang Pan, Kai Chen, and Ziwei Liu. Robo3d: Towards robust and reliable 3d perception against corruptions. In IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 19994 20006, 2023.
[27] Lingdong Kong, Yaru Niu. Shaoyuan Xie, Hanjiang Hu, Lai Xing Ng. Benoit Cottereau, Ding Zhao, Liangjun Zhang. Hesheng Wang, Wei Tsang Ooi, Ruijie Zhu, Ziyang Song, Li Liu, Tianzhu Zhang, Jun Yu. Mohan Jing, Pengwei Li, Xiaohua Qi, Cheng Jin. Yingfeng Chen, Jie Hou, Jie Zhang, Zhen Kan, Qiang Lin, Liang Peng, Minglei Li, Di Xu, Changpeng Yang. Yuanqi Yao, Gang Wu, Jian Kuai, Xianming Liu, Junjun Jiang, Jiamian Huang, Baojun Li, Jiale Chen, Shuang Zhang, Sun Ao, Zhenyu Li, Runze Chen, Haiyong Luo, Fang Zhao, and Jingze Yu. The robodepth challenge: Methods and advancements towards robust depth estimation. arXiv preprint arXiv:2307. 15061, 2023. 2,
[28] Lingdong Kong, Jiawei Ren. Liang Pan, and Ziwei Liu. Lasermix for semi-supervised lidar semantic segmentation. In JEEE/CVF Conf. Comput. Vis. Puttern Recog. (CVPR), pages 21705-21715, 2023.
[29] Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In JEEF/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 161 1-621,, 2021. 1,3
[30] Hamid Laga, Laurent Valentin Jospin, Farid Boussaid, and Mohammed Bennamoun. A survey on deep learning techniques for sterco-based bascd depth estimation. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), 44(4):1738 1764, 2020.
[31] Jin Han Lee, Myung- Kyu Han, Dong Wook Ko, and Il Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv:1907.10324 2019. 3, 7, 26, 34, 35.
[32] Ruibo Li, Ke Xian, Chunhua Shen, Zhiguo Cao, Hao Lu, and Lingxiao Hang. Deep attention-based classification network for robust depth prediction. Asian Conf. Comput. Vis. (ACCV), pages 663-678, 2019. 3
[33] Zhengqi Li and Noah Snavely, Megadepth: Learning single-view depth prediction from internet photos. In JEEE/CVF Conf. Comput. Vis. Pattern Recog, (CVPR), pages 2041-2050, 2018. 2,
[34] Zhenyu Li, Zehui Chen, Ang Li, Liangji Fang. Qinhong Jiang. Xianming Liu, Junjun Jiang. Bolei Zhou, and Hang Zhao. Simipu: Simple 2d image and 3d point cloud unsupervised pre-training for spatial- aware visual representations. In AAAI Conf. Artifi. Intell. (AAAI), 2022. 2, 7, 26, 34, 35,
12

---

[35] Zhenyu Li. Zehui Chen, Xianming Liu, and Junjun Jiang. Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation. arXiv preprint urXiv:2203.14211, 2022. 3. 7. 26, 29, 34, 35, 36
[36] Fayao Liu, Chunhua Shen, Guosheng Lin, and Ian Reid. Learning depth from single monocular images using deep convolutional neural fields. IEEE Trans. Pattern Anol. Mach. Intell. (PAMI), 38 38(10):2024-20399 2015. 5
[37] Youquan Liu, Runnan Chen, Xin Li, Lingdong Kong, Yuchen Yang, Zhaoyang Xia, Yeqi Bai, Xinge Zhu, Yuexin Ma, Yikang Li, Yu Qiao, and Yuenan Hou. Uniseg: A unified multi-modal lidar segmentation network and the openpeseg codebase. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 21662-21673, 2023.
[38] Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, and Ziwei I. Liu. Segment any point cloud sequences by distilling vision foundation models. arXiv preprint w9X2v72306.09347, 2023. 3
[39] Xiaoyang Lyu, Liang Liu, Mengmeng Wang, Xin Kong, Lina Liu, Yong Liu, Xinxin Chen, and Yi Yuan. Hr-depth: High resolution supe-super monocular depth estimation. AAAI Conf. Artifi. Intell. (AAAI). pages 2294-2301, 2021. 6, 8, 9, 25, 30, 31, 32, 33, 37
[40] Claudio Michaelis, Benjamin Mitzkus, Robert Geirhos, Evgenia Rusak, Oliver Bringmann, Alexander S. Ecker, Matthias Bethge, and Wieland Brendel. Benchmarking robustness in object detection: Autonomous driving when winter is coming. arXiv preprint arXiv: 1907:0907700784 2019. 1, 3, 21
[41] Yue Ming, Xuyang Meng. Chunxiao Fan, and Hui Yu. Deep learning for monocular depth estimation: reviews. Neurocomputing. 438:14-3 14-33, 33, 2021.
[42] He Mu, Hui Le, Bian Yikai, Ren Jian, Xie Jin, and Yang Jian. Ra-depth: Resolution adaptive self- supervuee monocular depth estimation. In Eur. Conf. Comput. Vis. (ECCV), pages 565-581, 2022. 6, 8, 9, 10, 26, 29, 30, 31, 32, 33, 37, 41, 42, 43
[43] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 12179-121888 2021. 3, 7, 26, 34, 35, 36
[44] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), (443)):62311133 2022. 2.3
[45] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acde: The adverse conditions dataset with correspon- dences for semantic driving scene understanding. In IEEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 10765-10775, 2021. 5, 22
[46] Christos Sakaridis, Dengxin Dai, and Lue Van Gool. Semantic foggy scene understanding with synthetic data. Int. J. Comput. Vis., 126(9)973-992, 2018. 1,5, 8, 22
[47] Ashutosh Saxena, Min Sun, and Andrew Y. Ng. Make3d: Learning 3d seene structure from a single still image. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), (1(5):244840, 2008.
[48] Maarten Schellevis. Improving self self- supervised single view depth estimation by masking occlusion. arXiv preprint arXiv: 1908. 1/112, 2019. 6, 8, 24, 30, 31, 32, 33,37 37
[49] Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Eur. Conf. Comput. Vis. (ECCV), pages 746-760, 2012. 1,4, 5
[50] Libo Sun, Jia- Maag Bian, Huangying Zhan, Wei Yin, Ian Reid, and Chunhua Shen. Sc-depthv3: Robust self-supervised monocular depth estimation for dynamic scenes. arXiv preprint wrXiv:2211.03660 2022. 3
[51] Fabio Tosi, Filippo Alcotti, Matteo Poggi, and Stefano Mattoccia. Learning monocular depth estimation infusing traditional stereo knowledge. In IEFF/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 9899, 2019.
[52] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In Int. Conf. 3D Vis. (3DV), pages 11 11-20, 2017. 1
[53] Chaoyang Wang, Simon Lucey, Federico Perazzi, and Oliver Wang. Web stereo video supervision for depth prediction from dynamic scenes. In Inf. Conf. 3D Vis. (3DV), pages 348 348-357, 2019. 3
13

---

[54] Jiahang Wang, Sheng Jin. Wentao Liu, Weizhong Liu, Chen Qian, and Ping Luo. When human pose estimation meets robustness: Adversarial algorithms and benchmarks. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 11855-11864, 2021. 3
[55] Lijun Wang. Jianming Zhang, Oliver Wang. Zhe Lin, and Huchuan Lu. Sde-depth: Semantic divide-and- conquer network for monocular depth estimation. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR). pages 541-550, 2020. 3
[56] Jamie Watson, Oisin Mac Aodha, Victor Prisacariu, Gabriel Brostow, and Michael Firman. The temporal opportunist: Self-supervised multi frame monocular depth. In /EEE/CVF Conf. Comput. Vis. Patern Recog. (CVPR), pages 1 1164 1174, 2021. 3, 6, 8, 25, 30, 31, 32, 33, 37
[57] Jamie Watson, Michael Firman, Gabriel J. Brostow, and Daniyar Turmukhambetov. Self-supervised monocular depth hints. In /EEE/CVF Int. Conf. Comput. Vis. (ICCV), pages 2162-2171, 2019. 6, 24, 30, 31, 32, 33
[58] Ke Xian. Chunhua Shen, Zhiguo Cao, Hao Lu, Yang Xiao, Ruibo Li, and Zhenbo Luo. Monocular relative depth perception with web stereo data supervision. In IEEF/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 311-320, 2018.
[59] Feng Xue, Guirong Zhuo, Ziyuan Huang. Wufei Fu, Zhuoyue Wu, and Marcclo H. Ang. Toward hierarchical self supervised monocular absolute depth estimation for autonomous driving applications. JEEE/RSJ Int. Conf. Intell. Robots Syst. (IROS), pages 2330-2337, 2020. 6, 8, 24, 30, 31, 32, 33, 37
[60] Jaxing Yan, Hong Zhao, Penghui Bu, and YuSheng Jin. Channel-wise attention- based network for self- supervised monocular depth estimation. In Int. Conf. 3D Vis. (3DV), pages (664447,,000 2021. 3, 6, 8, 9, 25, 30, 31, 32, 33,37 37
[61] Chenyu Yi, Siyuan Yang, Haoliang Li, Yap peng Tan, and Alex Kot. Benchmarking the robustness of spatial-temporal models against corruptions. In Ady. Neural Inform. Process. Syst. (NeurIPS), 2021. 3
[62] Chi Zhang, Wei Yin, Billzb Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hicrarchical normalization for robust monocular depth estimation. In Adv. Neural Inform. Process. Syst. (NeurlPS), pages 14128-14139, 2022. 3, 3
[63] Ning Zhang, Francesco Nex, George Vosselman, and Norman Kerle. Lite-mono: A lightweight enn and transformer architecture for self-supervised monocular depth estimation. In JEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), 2023. 3, 6, 7, 10, 26, 29, 30, 31, 32, 33, 39, 41, 42,
[64] Sen Zhang, Jing Zhang, and Dacheng Tao. Towards scale-aware, robust, and generalizahle unsupervised monocular depth estimation by integrating imu motion dynamics. In Fur. Conf. Comput. Vis. (ECCV), pages 143-160, 2022. 3, 6, 8, 26, 30, 31, 32, 37 37
[65] Chaoqiang Zhao, Qiyu Sun, Chongzhen Zhang, Yang Tang, and Feng Qian. Monocular depth estimation based on deep learning: An overview. Science China Technological Sciences, 631))111121167,, 2020.
[66] Chaoqiang Zhao, Youmin Zhang, Mattco Poggi, Fabio Tosi, Xianda Guo, Zheng Zhu, Guan Huang, Yang Tang, and Stefano Mattoccia. Monovit: Self-supervised monocular depth estimation with a vision transformer. In Int. Conf. 3D Vis. (3DV), 2022. 3, 6, 8, 9, 10, 25, 29, 30, 31, 32, 33, 37, 40, 41, 42, 43
[67] Hang Zhou, David Greenwood, and Sarah Taylor. Self-supervised monocular depth estimation with internal feature fusion. In Brit. Mach. Vis. Conf. (BMVC), 2021. 3, 6, 8, 9, 10, 25, 29, 30, 31, 32, 33, 37, 41, 42, 43
[68] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video. In IEEE/CVF Conf. Comput. Vis. Pattern Recog. (CVPR), pages 1851-1858, 2017.
14

---

Appendix
In this appendix, we supplement the following materials to support the findings and conclusions drawn in the main body of this paper:
Sec. documents necessary information about the proposed datasets and benchmarks.
Sec. elaborates on more details in terms of benchmark definitions and implementations,
well as the details of data collection, organization, licensing, and access.
Sec. C introduces The RoboDepth Challenge, an academic competition held based on the datasets and benchmarks constructed in this work.
Sec. provides the complete quantitative results of different monocular depth estimation models in the established KITTI-C, NYUDepth2 C, and KITTI-S benchmarks.
Sec. E contains additional qualitative results of different monocular depth estimation models under out-of-distribution corruption scenarios.
Sec. F acknowledges the public resources used during the course of this work.
A
Datasheets
In this section, we follow Gebru et al. [13] to document necessary information about the proposed datasets and benchmarks.
A.1 Motivation
The questions in this section are primarily intended to encourage dataset creators to clearly articulate their reasons for creating the dataset and to promote transparency about funding interests. The latter may be particularly relevant for datasets created for research purposes.
1. " For what purpose was the dataset created?" A: research in the area of depth estimation robustness under diitr--distribution corruptions.
The dataset was created to facilitate relevant
2. "Who created the dataset (e.g., which team, research group) and on behalf of which entily (e.g. company, institution, organization)?" A: The dataset was created by Lingdong Kong (National University Singapore), Shaoyuan Xie (Huazhong University of Science and Technology), Hanjiang Hu (Carnegie Mellon University), Lai Xing Ng (Institute for Infocomm Research, A STARR, Benoit Cottereau (CNRS), and Wei Tsang Ooi (National University of Singapore).
3. "Who funded the creation of the dataset?" A: The creation of the dataset is funded by related affiliations the authors in this work, as listed in the above item.
4. "Any other comments?" A: N/A.
A.2 Composition
Most of the questions in this section are intended to provide dataset consumers with the information they need to make informed decisions about using the dataset for their chosen tasks. Some of the questions are designed to elicit information about compliance with the EU's General Data Protection Regulation (GDPR) or comparable regulations in other jurisdictions. Questions that apply only to datasets that relate to people are grouped together at the end of the section. We recommend taking broad interpretation of whether a dataset relates to people. For example, any dataset containing text that was written by people relates to people.
1. "What do the instances that comprise our datasets represent (e.g., documents, photos, people, countries)?" A: The instances that comprise the dataset are mainly images captured by camera sensors, providing visual representations of indoor and outdoor scenes observed.
2. "How many inslances are there in total (of each type, if appropriate))" A: The KITTI-C dataset contains a total number of 62, 730 RGB images from 18 corruption types, with resolution of 192 x 640. The NYUDepth2-C dataset contains total number of 39, 240
15

---

RGB images from 15 corruption types, with resolution of 480 x 640. The KITTI-S dataset contains a total number of 364 RGB images from 12 style type, with resolution of 192 x 640. Each image in these three datasets is associated with single-channel depth map of the same size as the image.
3. "Does the dataset contain all possible instances or a sample (not necessarily random) of instances from a larger set?" A: Yes, our datasets contain all possible instances that have been collected so far.
4. "Is there label or target associated with each instance?" A: Yes, each instance in our datasets is associated with a single-channel depth map of the same size as the image.
5. "Is any information missing from individual instances?" A: No.
6. "Aee relationships between individual instances made explicit (e.g, users' movie ratings, social network links)?" A: Yes, the relationship between individual instances is explicit.
7. "Are there recommended data splits (e.g., training, development/validdtte testing)?" A: Yes, we provide detailed data splits for our datasets.
8. "Is the dataset self-contained, or does it link or otherwise rely on external resources (e.g.. websites, tweets, other datasets)?" " A: Yes, our datasets are self-contained.
9. "Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?" A: No, all data are clearly licensed.
10. "Does the dataset contain data that, if viewed directly, might be offensive, insulting, threat- ening, or might otherwise cause anxiety?" A: No.
11. "An other comments?" A: N/A.
A.3 Collection Process
In addition the goals outlined the previous section, the questions this section are designed to elicit information that may help researchers and practitioners create alternative datasets with similar characteristics. Again, questions that apply only to datasets that relate to people are grouped together at the end of the section.
1. " "How was the data associated with each instance acquired?" A: Please refer to the details listed in Section B.
2. " "What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)?" A: Please refer to the details listed in Section B.
3. "If the dataset is a sample from larger set, what was the sampling strategy (e.g., deter- ministic, probabilistic with specific sampling probabiliries)?" A: Please refer to the details listed in Section B.
A.4 Preprocessing, Cleaning, and Labeling
The questions in this section are intended to provide dataset consumers with the information they need determine whether the "raw" data has been processed in ways that are compatible with their chosen tasks. For example, text that has been converted into of-goos-words' is not suitable for tasks involving word order.
1. "Was any prepocesing/cceaingg/bbbiini of the data done (e.g., discretization or buck- eting, tokenization, ofre-ch tagging, SIFT feature extraction, removal of instances, processing of missing values)?" A: Yes, we preprocessed and cleaned data in our datasets.
2. " "Was the 'raw' data saved in addition to the preproceseddllennelthhee support unanticipated future uses)?" A: Yes, raw data is accessible.
data (e.g.. to
3. "Is the software that was used to preprocess/llann/lbel the data available?" A: Yes, the necessary software used to preprocess and clean the data is publicly available.
4. "Any other comments?" A: N/A.
16

---

A.5
Uses
The questions in this section are intended to encourage dataset creators reflect on tasks for which the dataset should and should not be used. By explicitly highlighting these tasks, dataset creators can help dataset consumers make informed decisions, thereby avoiding potential risks or harms.
1. "Has the dataset been used for any tasks already?" A: No.
2. "Is there a repository that links any all papers or systems that use the dataset?" A: Yes, we provide such links in our GitHub repository.
3. "What (other) tasks could the dataset be used for?" A: The dataset could be used for relevant perception, tracking, and planning tasks based on camera sensors.
4. "Is there anything about the composition of the dataset or the way it was collected and that might impact future uses?" A: N/A.
5. "Are there tasks for which the dataset should not be used?" A: N/A.
6. "Any other comments?" " A: N/A.
A.6 Distribution
Dataset creators should provide answers to these questions prior distributing the dataset either internally within the entity on behalf of which the dataset was created or externally to third parties.
1. "Will the dataset be distributed to third parties outside of the entity (e.g. company, institution, organization) on behalf of which the dataset was created?" A: No.
2. "How will the dataset be distributed (e.g., tarball on website, API, GitHub)?" A: Very likely to be distributed by website, API, and GitHub repository.
3. "When will the dataset be distributed?" A: The datasets are publicly accessible.
4. "Will the dataset be distributed under copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?" A: Yes, the dataset is under the Creative Commons SaaittionnoooCCmmmiilal--SSi-ei
4.0 International License.
5. "Have any third parties imposed IP-based or other restrictions on the data associated with the instances?" A: No.
6. "Do any export controls or other regulatory restrictions apply the dataset individual instances?" A: No.
7. "Any other comments?" ?" A: N/A.
A.7 Maintenance
As with the questions in the previous section, dataset creators should provide answers these questions prior to distributing the dataset. The questions this section are intended to encourage dataset creators plan for dataset maintenance and communicate this plan to dataset consumers.
1. "Who will be supportigghhstinnnaaitttiinii the dataset?" A: The authors of this work serve for supporting, hosting, and maintaining the datasets.
2. "How can the owner/curator/manager of the dataset be contacted (e.g., email address)?"
A: The curators be contacted via emails follows:
Lingdong Kong (lingdongOcomp. nus edu. sg);
Shaoyuan Xie (Shaoyuanxie@huss.. edu. end.
Hanjiang Hu (banjianghuucmuu.ddu
Lai Xing Ng Log_llaixinngiir.cc.
edu.sg);
Benoit Cottereau (benoit. www.biitc.ottereeauucnereeer
Wei Tsang Ooi (coiwt@comp. nus .us.edu.
3. "Is there an erratum?" A: There is no explicit erratum; updates and known errors will be specified in future versions.
17

---

weather & lighting
sensor & movement
data & processing
Brightness
Dark
Defocus Blur
Glass Blur
Gaussian Noise
Impulse Naise
Fog
Frost
Motion Blur
Zoom Blur
Shot Noise
ISO Noise
Contrast
Elestic
Color Quant
Pixelate
JPEG
Figure A: The 18 corruption types from three main categories defined in the RoboDepth benchmark. Examples shown are from the proposed KITTI-C benchmark.
4. "Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?" A: No, for the current version. Future updates (if any) will be posted on the dataset website.
5. "Will older versions of the dataset continue to be supprree/hoseedmnnntnnee! A: Yes. This is the first version of the release; future updates will be posted and older versions will replaced.
6. "If others want to extend/augment/huild on/contribute to the dataset, there a mechanism for them to do so?" A: Yes, we provide detailed instructions for future extensions.
7. "Any other comments?" A:N/A.
B Datasets and Benchmarks
In this section, first provide the detailed definitions of different corruption types in the RoboDepth benchmark (Sec. B. )), we then elaborate on the data collection process (Sec. B.2), the license of our datasets (Sec. B.3), and the procedures for downloading these datasets (Sec. B.4). Lastly, we attach the summary of technical contributions and implementation details for the benchmarked monocular depth estimation methods (Sec. B.5). An overview of the 18 corruption types is shown in Fig. A. The histogram of pixel values under each corruption is shown in Fig. B.
B.1 Corruption Definition
Brightness refers to the level of lightness or darkness in an image. affected by the intensity and color of the light source, as well as the reflectivity of the objects in the scene. Images captured under different lighting conditions may have varying levels of brightness, which can affeett the visibility and depth estimation quality.
Dark images can result from low-lighting conditions underexposure. These images can have low contrast and low visibility, making it difficult to distinguish the objects and backgrounds in the scene. Dark images also suffer from increased electronic noise and color distortion.
Fog is type of atmospheric scattering that can reduce the contrast and visibility of and backgrounds in a scene. occurs when water droplets or ice crystals in the air scatter and absorb light, leading to hazy or blurry appearance.
objects
Frost forms when lenses or windows are coated with ice crystals, leading to image distortion and inaccuracies in depth estimation. Frost can be modeled as convolution the image with a non-uniform kernel that depends on the shape and size of the ice crystals. It can affect the accuracy of depth estimation by reducing the contrast of the image and distorting the edges of objects.
18

---

Gright
Dark
Fog
30-
30-
20
20
20
10
10
10
0
1.6
1.8
2.0
2.2
0.3
0.4
0.5
0.6
0.7
1.2
1.3
1.4
1.5
30
30-
30-
20
20
20
0
1.8
2.0
2.2
1.8
2.0
2.2
1.0
1.1
1.1
1.1
Delocas
30
30
30-
20-
0
0
1.0
1.2
1.5
1.8
1.2
1.5
1.8
2.0
1.0
1.2
1.5
1.8
2.0
Zoom
Electic
Quant
30
30-
0
0
1.2
1.4
1.6
1.0
1.2
1.5
1.8
2.0
0.8
1.0
1.4
1.6
Cauissin
Imputse
Shot
20-
10
10
10-
0
1.2
1.4
0
1.2
11
11
0
1.0
1.2
1.8
ISD
Pixalltt
JPEG
30
20
20
20
10
10
10
1.0
1.4
1.8
1.0
1.2
1.5
1.8
2.0
1.0
1.2
1.5
1.8
Figure B: The histogram of pixel values under each of the 18 corruption types in KITTI-C. Note that the horizontal axis of each subfigure is scaled, shifted, and centered to fit the window. The numerical range at the scale of le6. Zoomed-in for more details.
Snow is visually obstructive form of precipitation that can obscure objects and backgrounds in the image and make difficult to estimate accurate depth. can be modeled as a random distribution of white pixels with a high probability of occurrence. Snow can affect the accuracy of depth estimation by introducing errors in the intensity values of pixels and obscuring the edges of objects.
Contrast is the difference luminance or color between different parts of an image. High- contrast images have large differences between light and dark areas, while low-contrast images have more similar levels of brightness. Contrast can be affected by lighting conditions and the color and texture of the objects and backgrounds in the scene.
19

---

Clssn
Severity Lovel
Sowryy Level
Sovenit Level
Severity Level
Severity Level
Figure C: Illustrative examples of each corruption type across the five severity levels (level the lightest and level 5 is the hardest) in the KITTI-C benchmark. Best viewed in color. Zoomed-in for more details.
Defocus blur occurs when image is out of focus, causing objects and backgrounds in the image to appear blurred. It can be modeled as a convolution of the image with a Gaussian kernel, where the blur size depends on the distance between the camera and the object. Defocus blur can affect the accuracy of depth estimation by reducing the contrast of the image and distorting the edges of objects.
Glass blur is type of image distortion that appears with glass windows panels. It can lead inaccuracies depth estimation due to the opaque and irregular nature of the glass. Glass blur be modeled convolution of the image with non-uni uon-uoffomm kernel that depends on the shape and thickness of the glass.
Motion blur appears when a camera moving quickly, causing objects and backgrounds in the image to appear blurred. It can be modeled as convolution of the image with a motion kernel that depends the direction and speed of the camera movement.
Zoom blur occurs when camera moves toward an object rapidly, causing the image to appear blurred. can be modeled convolution of the image with non-uniform kernel that depends the zoom factor and the distance between the camera and the object. Zoom
20

---

blur can affect the accuracy of depth estimation by distorting the edges of objects and reducing the contrast of the image.
Elastic transformations are spatial transformations that deform small regions of image while preserving the overall structure. They are commonly used in data augmentation techniques for deep learning models to increase the robustness of the model to deformations and variations in the input images.
Color quantization is the process of reducing the number of colors an image while preserving its overall visual appearance. This technique is commonly used to reduce the storage and processing requirements for digital images. However, this process can result in loss of detail and color accuracy in the image, particularly in areas with complex color gradients or patterns.
Gaussian noise is type of noise that appears in low-light conditions and can cause random fluctuations in image intensity. It is modeled as additive white Gaussian noise and has a normal distribution with zero mean and standard deviation that represents the noise level. Gaussian noise can affect the accuracy of depth estimation by reducing the contrast of the image and introducing errors in intensity values.
Impulse noise is type of noise that can be caused by bit errors and appears as isolated pixels with incorrect intensity values. It can be modeled as a random distribution of black and white pixels with low probability of occurrence. It can affect the accuracy of depth estimation by introducing errors in the intensity values pixels and distorting the image.
Shot noise, also called Poisson noise, electronic noise caused by the discrete nature of light itself. It occurs when photons hit sensor and is modeled as Poisson distribution. Shot noise can lead to irregularities in image intensity and affect the accuracy of depth estimation, especially in low-light conditions.
ISO noise is a type of noise that can appear in digital images captured with high ISO settings. ISO refers to the sensitivity of the camera's sensor to light, with higher ISO values resulting in brighter images. However, increasing the ISO setting can also introduce additional electronic noise into the image, leading to a grainy or speckled appearance. This type of noise can be particularly challenging to remove without losing important image details.
Pixelation occurs when an image is displayed or printed at a low resolution, resulting in individual pixels becoming visible. This can Iead to loss of detail and clarity in the image, bringing extra challenges for depth estimation models.
JPEG is a lossy image compression format commonly used for storing and sharing digital images. JPEG compression reduces the file size of an image by removing some of the image data that is deemed less important less noticeable to the human eye. Such compression can result in visible artifacts, such as blockiness or blurring, in the compressed image.
B.2 Corruption Simulation
The corruption simulation tools are from resources. We use the imagecorruptions tool from Michaelis el al. [40] to simulate 'brightness', 'fog', 'frost', 'snow', 'contrast', 'defocus blur', 'glass blur', 'motion blur', 'zoom blur', 'elastic transform', 'Gaussian noise', 'impulse noise', 'shot noise', 'pixelate', and 'JPEG compression'. Additionally, We use the 3DCC tool² by Kar et al. [24] to simulate 'dark', "color quantization', and 'ISO noise'.
We follow Hendrycks and Dietterich [18] to split each corruption simulation into five severity levels for KITTI-C and four severity levels for NYUDepth2-C. The split strategy is the same as the ImageNet-C paper [18]. Illustrative examples of different severity levels in KITTI-C are shown in Fig. C. Illustrative examples of different severity levels in NYUDepth2 C are shown in Fig. D, Fig. E, and Fig. F.
We provide a comprehensive corruption simulation toolkit that can be used to generate the defined 18 corruption types on any image dataset. This toolkit publicly accessible at: https:// htap:://githhi thub. com/ldkongi htt///oooDepth/Bl wtwpooeetphhhobo///eeoeboooooeee hatt//llb/maaaaae waab//man///a /ttp///ccc//////////////////cact /CEAATE md. Refer to this page for more details and the code implementations of corruption simulations.
Mooe details at: https: Cttpa://e /ttp//ggttttttt////// htwpittub.coombeeeeeeeeeeeeeeeee com/bbehheellle httperggglab/immggeorrrpiioooooo
More details at: https: http://ggthuh.oooo...o VILAB SAA.PLLLL.LLAAA//OOCooooorrrroa
21

---

Cleon
Sever Levol
Sevrry Lovel
Sovenby Level 3
Sever Levol
T
35
Figure D: Illustrative examples of each of the five corruption types ('brightness', 'dark', 'contrast', 'defocus blur', and "glass blur") across four severity levels (level 1 is the lightest and level 4 is the hardest) in the NYUDepth2-C benchmark. Best viewed in color. Zoomed-in for more details.
To validate the simulated corruptions are of high fidelity, especially for the types that are related to weather and lighting changes, we conduct study on the pixel distributions. Assuming that corruption simulation realistic enough to reflect real-world situations, the distribution of a corrupted "clean" set should be similar to that of the real-world corruption set. We validate this using ACDC [45], nuScenes [2]. Cityscapes [8], and Doggy-Ciryscapes [46], since these datasets contain: i) real- world corruption data; and ti) clean data collected by the same sensor types from the same physical locations.
We simulate corruptions using "clean" images and compare the distribution patterns with their corresponding real-world corrupted data. We do this to ensure that there is no extra distribution shift from aspects like sensor difference (e.g. FOVs and resolutions) and location discrepancy (e.g. environmental and semantic changes). The results are shown in this paper: https: //tpt//tttttt//t com/ Bakoong1255/RobDDeppe
blob/maing docs/ Baaiccs/VAADDDA VALIDITY md.
What is more, follow Geirhos al. [15] and use AdaIn [20] generate stylized images in KITT7-S. AdaIn is classical style transfer model that takes a pair of two images (a reference image and a style image) input and outputs stylized image. The checkpoint of this style transfer model can be down- loaded at: https: / //g//gitta giub....... com/ http://att0////ytrchhhooooooa AdaIN. The detailed procedures for gen- erating stylized images are attached https://githut: com/rg http//gerrhos//eoooo Serrhsss/Syyiiedd ImageNet. Illustrative examples of different styles in KITT7-S are shown in Fig. G.
B.3 License
Our datasets and benchmark toolkit are released under the Attibutioo--onnCommmrraaall ASaree 4.0 International (CC BY-NC-SA 4.0) license", under the following terms:
More details https: https://cceativeccmmee org/ strg/iicenees/by-nee ntwp://bbbnno-so-oooo s.
22

---

Cleon
Sever Levol
Severyy Lovel
Soveniy Lovel 3
Sever Levol
17
Figure Illustrative examples of each of the five corruption types ('motion blur', zzom blur', 'elastic transform', 'color quantization', and 'Gaussian noise') across four severity levels (level 1 the lightest and level 4 is the hardest) in the NYUDepth2-C benchmark. Best viewed in color. Zoomed-in for more details.
Attribution You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not any way that suggests the licensor endorses you your use.
NonCommercial - You may not use the material for commercial purposes.
ShareAlike -
If you remix, transform, or build upon the material, you must distribute your contributions under the same license the original.
B.4 Download
Our datasets and benchmark toolkit are publicly accessible. We provide detailed procedures for accessing and downloading them. Please refer to the following pages for more details:
GitHub
Installation & Environment:
Srán//ocs/INSTALE htTppe///ggtaaaa.oo/o////gg2222p
Data Preparation: DATA DATA_PREPARE md.
Benchmark Details: http:///gggggtaaooo///oooo/////r
Corrpttin Simulation Tool:
httppe///sttaaacooo////eoo/////p main/docs/CREATE.C
23

---

Cleon
Sever Lovol
Sevrry Love
Soveniy Lovel
Sever Level
O
17
W
Figure F: Illustrative examples of each of the five corruption types ('impulse noise', 'shot noise". IISO noise', 'pixelate', and 'JPEG compression') ") across four severity levels (level 1 is the lightest and level 4 is the hardest) in the NYUDepth2-C benchmark. Best viewed color. Zoomed-in for more details.
B.5 Benchmarked Methods
MonoDepth2 [17]: A seminar work in the field of MDE which proposed a series of improvements on top of prior works, including a per-pixel minimum re-projection loss, an auto-masking strategy, and a multi-scale estimation framework. We evaluate three versions of this model, under the Mono, Stereo, and Mono+Stereo settings, respectively. We also test this model with different backbones (ResNet-18 and ResNet-50) and pretraining strategies. The code is accessible at: https: / thub. com ctmmniinnt//eeeeeceeeeeeeccciice miantiiciiiiii clabs www.b//ooooopphh2.....
DepthHints [57]: A stereo model that makes use of depth hints, which are small but informative cues about the scenes and can be extracted from the image itself. These hints then used to guide the network during training, allowing the model to learn a better depth estimation representation without requiring any explicit ground truth depth information. The code is accessible at: https: : //tgttgttttt.tt//// com/ niantiniiiiimimmmianmmmiiimmiiam clabs depph-hpphhie hints.
MaskOce [48]: A monocular model takes into account occlusion in depth estimation. The occlusions occur when objects in the scene block each other from view. The proposed method masks out occluded regions in the input image during training, allowing the model to focus on learning from only the visible parts of the scene. This approach results in more accurate depth estimation, especially in regions with occlusion. The code accessible at: https: //gi thub. www/schellleeccccccccoccccccw.ce monodepth2.
DNet [59]: A monocular model consists of a coarse-to-fine depth estimation pipeline, where deep neural network is trained to predict absolute depth maps from single input image. This model is built upon the MonoDepth2 baseline. The code is accessible at: https:/ / //gi thub. com com/TJ- IPLab/DNet.

---

Example
Exampl 2
Examppe
Examppe
Exarcle
Examplo
Figure G: Illustrative examples of 12 different styles in the KITTI-S benchmark. Best viewed in color. Zoomed-in for more details.
CADepth [60]: new network architecture that employs attention mechanisms to selec- tively attend to certain image regions, which helps to improve the accuracy of the depth esti- mation. The code is accessible at: https: // gi thub. com www.miiggt//ADeeppepeaae
mas ter.
new depth estimation model that can generate high-resolution depth maps from a single input image. The main contribution of the work is the introduction a hierarchical residual pyramid network architecture that leverages multi-scale features to produce accurate depth estimates. The code is accessible at: https: /ttpp:///ii httpt/gttb////////////// thub. com/ shawLyu/ HHaeeu//RRRDDD -Detth.
HR-Depth [39]:
DIFFNet [67]: A monocular model based on a deep neural network architecture that uses internal feature fusion mechanism combine information from different layers of the network. Such a mechanism used in the network helps to capture global and local features of the input images, which improves the accuracy of the depth estimates. The code is accessible at: https: github.. com/ com/brande brandlaaddlldd eeleeeyyihiaaaae
Many Depth [56]: A multi-monocular model that leverages temporal consistency between adjacent frames in video sequence estimate accurate depth maps. The proposed framework consists of a network that takes as input multiple frames of video sequence and outputs a corresponding depth map for each frame. The code is accessible at: https: //github..cmm com ni miantico clabs/ aaabss/manddephh..
FSRE-Depth [22]: A monocular model consists of two-stage network architecture that enhances the representation of the input image by incorporating fine- -eeiggraidd semantic information. The first stage of the network uses semantic segmentation module to extract semantic information from the input image. The second stage of the network utilizes depth refinement module to refine the initial depth prediction and generate the final depth estimation. The code is accessible at: https: /tts:///iitaa //tbttb...ttt///g//gg/// com/hyylue/FSIt FSRt--epE..........
Mono ViT [66]: monocular model that consists of novel training scheme that leverages the spatial and semantic representations learned by the Vision Transformers to predict depth
25

---

from a single RGB image. The code is accessible at: attps://giithn www.w//.oomc/ece/ / MonoViT.
DynaDepth [64]: A monocular model that incorporates the motion dynamics data from an inertial measurement unit (IMU) to improve the accuracy and robustness of the depth estimation in challenging conditions. The proposed method is able to estimate the scale of the scene, which is a major challenge in monocular depth estimation, and is also more robust illumination changes and dynamic scenes. The code accessible at: https: // //github. com/ Sonm/eenZHAGGG. GitHub/eHtt imm-mmmees depth.
RA-Depth [42]: A monocular model that can adaptively adjust the output resolution based the input image resolution. This is achieved through a two-stage architecture that first generates a coarse depth map and then refines it to the desired output resolution using a depth super-resolution network. The code accessible at: https://gi thub. com/ hmhemu/ RA Depth.
TriDepth [6]: A monocular model with a new depth estimation loss function that incorpo- rates scale-invariant gradient term. This allows the model to learn predict sharper edges and finer details in the depth map, which is critical for accurate depth estimation. The code is accessible https: / //ggttbttt.../////////// com/ www.gynnnnggnnnneeececee /triitriiiii trrit//ttiid......eeeeee
Lite-Mono [63]] A monocular model with lightweight convolution backbone and transformer architecture for self-supervised monocular depth estimation. The authors propose hybrid approach that combines the strengths of both CNNs and transform- ers to achieve saae-offtth-aa performance on several benchmarks while using signif- icantly fewer parameters compared to existing methods. The code is accessible at: https://A //gi thub. com hatm//ooazn//Liie LLite-Moo.
BTS [3]]: A monocular model involves detecting and utilizing local planar structures in the image multiple scales improve depth estimation accuracy. Specifically, the authors propose a novel deep neural network architecture that incorporates a local planar guidance module each scale of a feature pyramid. This module predicts of local planar patches in the image and uses them to provide guidance for the depth estimation module. The code is accessible https: / //gi thub. com/c./cc//oooo c/ cleinc/bts.
AdaBins [1]: A monocular model with adaptive binning scheme that dynamically adjusts the bin sizes the network to better capture the distribution of depth values in the scene. In traditional binning schemes, fixed bin sizes used, which can lead to under or over- representation of certain depth values. AdaBins overcomes this limitation by adapting the bin sizes based on the distribution of depth values in the input image. The code is accessible at: https:// //gi///iie github......... com/ wwt.cooooc/seeeeeeeee cahariqfrroog 23/ AdaBins.
DPT [43]: A monocular model that leverages Vision Transformers in place of CNNs the backbone for dense prediction tasks including MDE. Several new techniques are proposed to facilitate dense predictions using self-attention. The code is accessible at: https:// /gi thub. wwt.a...o.......aaaa...a.....ta. www.cooooii.... woom/ssl-ae org/ /DPT.
SimIPU [34]: A multi-modal contrastive learning framework consists of a simple pertaining strategy that leverages the spatial perception module to learn a spatial- representation from images and point clouds. The code is accessible at: https / / thub. com/ zhyever/ SimIPU.
DepthFormer [35]: A monocular model that is tailored facilitate the long-range correla- tion for accurate MDE. The backhone is adopted from Vision Transformers, where a hierar- chical aggregation and heterogeneous interaction module proposed to enhance the features via element- wise interaction. The code is accessible at: https: / gi thub. com/ zhyever/ Monocular- Depth- Dwwth-eettimaeeeee Estimation Toolbox/ ttttttreee//eee wttpe//maannonniieiieeeeeeeeeaee wgg. gs/ depthf ormer.
B.6 Depth Estimation Metrics
In our benchmark, we adopt the conventional reporting of Abs Rel (error rate) and 8n (aceuracy) for measuring the depth estimation performance.
26

---

M DesCartes
The Robo Depth Challenge
ICRA 2023
Figure H: We successfully hosted The RoboDepth Challenge [27] at ICRA 2023, which is an academic competition established based on the datasets and benchmarking toolkit proposed in this work. More details of this competition can found at: https: haeeee//rooooppheee gi githu........ io.
nno V
wwW
(a) Track
(b) Track? 2
Figure The submission and scoring statistics for the two tracks in the RoboDepth competition.
Abs Rel measures the absolute relative difference between the ground-truth (gt) and the prediction
(pred), as calculated via the following equation:
1
lgt pred
Abs Rel
D
gt
predED
The 8 metric the depth estimation accuracy given the threshold:
gl. pred,
5, D If pred E DI max
pred' < 1.25')| x 100%,
(4)
where 81 5 1.25, 82 8 1.252, 8g 1.253 are the three conventionally used accuracy scores prior works.
We combine Abs Rel and 8, the two main measures, into unified metric DEE which is constantly used as the indicator of depth estimation error in our benchmark.
Rel-6
2
C The RoboDepth Challenge
In this section, introduce The RoboDepth Challenge based on the datasets and benchmarks proposed in this work.
academic competition that is established
C.1 Competition Overview
The RoboDepth Challenge [27] aims to facilitate relevant research in the area of robust monocular depth estimation. hosted by the 40th IEEE Conference on Robotics and Automation (ICRA 2023). More materials on this competition provided in the following links;
27

---

Table A: The top-ten performing teams the leaderboard of Track in the RoboDepth competition.
Team Name
Abs Rel
Sq Rel
RMSE
log RMSE L
< 1.25
8 8 1.25"
6 1.25'
OpenSpaceAT USTC-1AT-Unieed
0.121
0.919
4.981
0.200
0.861
0.953
0.980
0.123
0.932
4.873
0.202
0.861
0.954
0.979
YYQ
0.123
0.885
4.983
0.950
0.979
dlut
0.124
0.899
4.938
0.203
0.852
0.950
0.979
UMCV
0.124
0.845
4.883
0.202
0.847
0.950
0,98U
THU ZS
0.124
0.892
4.928
0.203
0.851
0.951
0,980
THU_Chen
THU Chen
0.125
0.865
4.924
0.203
0.846
0.950
0.980
Seesee
0.126
0.990
4.979
0.206
0.857
0.952
0.978
namename
U.126
0.994
4.950
U.204
0.860
U.953
0.979
USseCxNetEaseFuxi
0.129
0.973
5.100
0.208
0.846
0.948
0.978
MenoDe [17]
0.221
1.988
7.117
0.312
0.654
0.859
0.938
Competition Page: https://robodepch..ghbbb..
Competition Report: httpss//aariiv..rr//b///////../
Toolkit: https:://itthub.cmm/// http:://tthtacccccooo//////////h
Workshop Recordings:
Server for Track https://ccoaaaab htwp://w.....aa...../////oott/oa
Server for Track 2: https://codalas listillbbliii...... http.p/.aaaayyy...tottttttttttoa
C.2 Statistics
The RoboDepth Challenge started on January 1, 2023, and ended on May 25, 2023. There are two tracks in this competition: self-supervised learning track for robust outdoor depth estimation and supervised learning track for robust indoor depth estimation.
This competition attracted lot of participants from around the world. Specifically, 226 teams registered our evaluation servers. Among them, 66 teams made total number of 1137 valid submissions. The detailed statistics in terms of submission and scoring are shown in Fig. I. The top-three performing teams Track achieved the Abs Rel scores of 0.121, 0.123, and 0.123, respectively. The top-three performing teams of Track achieved the al scores of 0.940, 0.928, and 0.898. More results are attached to our evaluation servers and competition homepage.
For more details, please refer to our competition page at: https: /ttp::////thbb/ www.wooooom/o000/// RoboDepth/ tree wwwe/maii/eeeeeeee htte///mmi/oooottiiioooo
C.3 Workshop
We hosted The RoboDepth Workshop at ICRA on June 02, 2023. The video recordings of this workshop publicly available at: bttpp://wwwe youtube........... wwt.uu....../.tttteeeeeeeeeeeeee BBaxrIffC--BG22x__eel 21t_22s__e1M22_YYxxiHHXtaaa
The slides of this workshop be downloaded from https://ldkors comttalks/ccra22 8 robodepth. pdf.
C.4 Leaderboards
The results of the top-ten teams from each of the two tracks are shown in Table A and Table B, respectively. For more details, please refer to our competition page at: betps://gittii com/
D Additional Quantitative Result
In this section, we provide the complete results of different monocular depth estimation models the established KITTI-C, NYUDepth2-C, and KITTI-S benchmarks.
28

---

Table B: The top-ten performing teams the leaderboard of Track in the RoboDepth competition.
Team Name
Abs Rel
Sq Rel
RMSE
log RMSE L
< 1.25
K 1.25"
6 1.25'
0.088
0.046
0.347
0.115
0.940
0.985
0.996
OpenSpaceAI
0.095
0.045
0.341
0.117
0.990
GANCV
0.104
0.391
0.131
0.982
shinonomei
0.123
0.080
0.450
0.153
0.861
0.975
0.993
YYQ
0.125
0.085
0,470
0.159
0.851
0.970
0,950
Hyg
0.124
0.089
0.474
0.158
0.851
0.967
0.990
DepthSquad
0.137
0.085
0.462
0.158
0.845
0.976
0.996
kinda
0.146
0.095
0.480
0.165
0.831
0.973
0.993
dx3
0.131
0,095
0,507
U.170
0.825
U.963
0.989
uuht
0.150
0.100
0.492
0.168
0.822
0.973
0.993
DepthFormer [35]
0.190
0.179
0.717
0.248
0.655
0.898
0.970
D.I KITTI-C
The clean performances of 32 MDE models under the standard KITTI Eigen split are shown in Table C.
The per-corruption DEE scores, CE scores, and RR scores of 32 MDE models in the KITTI-C benchmark are shown in Table D, Table E, and Table F, respectively.
D.2 NYUDepth2-C
The per-corruttion DEE scores, CE scores, and RR scores of 10 MDE models in the NYUDepth2- C benchmark are shown in Table G, Table H, and Table I, respectively.
D.3 KITTI-S
The per-corruption DEE scores of 32 MDE models in the KITTI-S benchmark are shown in Table J.
E Additional Qualitative Result
In this section, we provide additional qualitative results of different monocular depth estimation models under
corruption scenarios.
Specifically, the qualitative results of MonoDepth2 [17], Lite-Mono [63], and Mono ViT [66] under each of the 18 corruption types across five severity levels in our benchmark are shown in Fig. J, Fig. K, and Fig. I., respectively.
The qualitative results of MonoDepth2 [17], DIFFNet [67]. RA-Depth [42], Lite-Mono [63], and ViT [66], under each of the 18 corruption types in our benchmark are shown Fig. M and Fig. N.
The qualitative results of MonoDepth2 [17], DIFFNet [67], RA-Depth [42], Lite-Mono [63], and MonoViT [66], under each of the 12 styles in our benchmark are shown in Fig. O.
29

---

Table C: The reproduced scores of 32 monocular depth estimation models under the standard KITTI Bigen split. Blocks from top to bottom: [1st] the baseline
MenoDepth2 RI8 [17]; [2nd] methods monocular inputs; [3rd] methods stereo inputs: [4th] methods monocular stereo inputs. M+S demotes models with both monocular and stereo inputs. The checkpoints for reproduction will be publicly available in our GitHub repository.
1.25
0.981
0.977
0.980
0,981
0.981
0.983
0.983
0.983
0.982
0.983
0.981
0.984
0.981
0.982
0.985
0.983
0.982
0.982
0.983
0.983
0.975
0.968
0.977
0.981
0.976
0.981
0.979
0.974
0,980
0.980
0.984
1
1.25
0.959
0.948
0.959
0.990
0.960
0.963
0.962
0.965
0.959
0.904
0.967
0.968
0.959
0.961
0.968
0.962
0.90
0.961
0.963
0.965
0.948
0.933
0.953
0.959
0.947
0.961
0.957
0.943
0.959
0.958
0..677
<<
1.25
0.877
0.845
0.877
0.878
0.877
0. 1,890
0.884
0. 0.897
0.871
0. 0.880
O. 1,900
0 0.908
0.877
0.882
0.903
0.878
0.880
0.879
0. 0.886
0.897
0.864
0.831
0.873
0.878
0 0.851
0.889
0.874
0.836
0.877
0.881
0.904
<
RMSE
0.193
0.210
0.193
0.191
0.183
0.185
0.179
0.192
0.182
0.175
0.172
0.192
0.188
0.171
0.186
0.187
0.186
0.183
0.178
0.200
0.232
0.202
0.191
0.211
0.187
0.196
0.220
0.193
0.192
0.174
RMSE
4.864
5. 131
4.631
4.788
4.819
4.592
4.632
4.450
4.765
4.540
1.373
4.292
4.848
4.6455
4.216
4.574
4.715
1.677
4.500
4.459
1.967
5.488 6.482
4.776
4.655
5.099
4.454
4.752
6.263
1.31
4.645
4.333
Sq Rel
0.905
1.042
0.50T
0,885
0.567
0.792
0.753
0.894
0.708
0.708
0.713
0.839
0.785
0.632
0.775
0.541
0.509
0.766
0.732
0.877
1.143
0.851
0.791
0.971
0.734
0.821
1.027
0.507
0.788
0
Abs Rel 1
0.1
0.132
0. 0.106
0.113
0.114
0.107
0. 1.109
0.102
0.118
0. 1.105
0.099
0. 0.096
0.112
0.109
0.096
0.112
0.110
0.110
0.101
0.110
0. 130
0.107
0.104
0.097
0.1 10G
0.122 1.127
0.106
1.1
0.098
Pretrain
ImageNNt
None
ImageNNet
Image Net
Image
ImageNet
Cityscapes
Image Net
ImageNNe Net
ImageNNr Net
Image Net
ImageNer
ImageNet
ImageNet
Image Net
ImageNNNr
Image Net
Image Net
ImageNer
Image
ImaggeNe
None
Image
ImageNNet
None
ImageNet
Image Net
None
Image Net
ImageNNr
Image Net
Resolution
640 192
640 192
1024 320
640 192
640 192
640 192
640 192
640 192
640 1.0x/19 192
640 192
640 192
320
x024 320
640 192
640 192
640 x 192
640 192
640 192
640 192
640 192
640 192
640 192
640 192
1024 x 320
640 192
1024 x 320
1024 320
640 192
640 192
640 192
640 192
1024 x
Backbone
ResNet-18
ResNet-18
ResNet- 18
ResNet- 18
ResNett-8
ResNet-
ResNer-18
HRNet
ResNet-18
ResNet-11 18
MPVIT
MPVIT
ResNer-18
RcsNcr-50
ResNett18
ResNet-18
ResNer- 18
ResNet-18
ResNet-18
ResNer-18
ResNet-18
ResNer-15 18
ResNet-15 ResNet-18 18
18
ResNet-18 18
ResNet-18
ResNer-18
ResNet-18 18
ResNet-18 18
ResNet-13 18
MPVIT
Modality
Mono
Mono
Mono
Mono
Mono
Mono
Mono
Mono
Mono
Mono
Mono
Mono
Mono
MoRo
Mono
Mono
Mono
Mono
Mono
Monn
Stereo
Stereo
Stereo
Stereo
Stereo
Sterco
M+S
M+S
M+S
M+S
M+S
Method
[17]
[17]
[17]
Mask RIS [48]
DNet RI8 (59)
HR- Depth [39]
DIFFNet [67]
Many ManyDepth RIR [56]
FSRE-DA FSRE-Depth Depth [22]
MonoViT VIT [66]
Mono VIT HR [66]
DenaDepth RM [64]
DynaDepth vso [64]
RA- Depth (42)
[6]
[63]
[63]
Lite-Monn Mono Lagg [63]
Mepthe RI8 [17]
MonoD Depth noge [17]
[17]
Depth supott [57]
DepthHinis HR [57]
MonoDepthy RIW [17]
MonoDepthynoe nogt [17]
Mono Deptha HR [17]
Min [66]
Mono MonoDepthi Deptht Ri8
MonnDDephia HR
CADepth
TriDepth x
Lite Mono Tay
Lite-h Mono Smill
Lite Lite-Mone Mono Basc
MonoDepthh, Depthy IIR
DeptiHinns
CADepth
MonoDeptha
30

---

Table D: The Depth Estimation Error (DEE) of 32 monocular depth estimation models in the KITTI- C benchhark Blocks from top to bottom: [1st] the buseline
MonoDepth2 RI8 [17]: [2nd] methods w/ monocular inputs; [3rd] methods w/ stereo imputs; [4th] methods w monocular stereo inputs. Bold: Best in col.
JPPG
196
.171
.198
.211
.177
.19
.195
.176
.202
.182
.221
.154
.148
.150
.216
.192
.215
.161
.104
.165
.184
.194
.178
.209
.192
.191
.183
.179
.177
.198
.192
.144
Pixel
.145
153
.138
.55
113
.145
.14
133
.135
.147
.147
.162
.135
.141
.153
.175
.142
.147
.154
.151
160
.144
164
.150
.157
.152
.141
136
.161
.138
.115
.47
ISO
.388
.55
-344
.425
.177
.357
.418
.374
.270
.452
.407
.20
.285
.379
4488
.455
.438
330
3356
.363
.362
.493
.556
-344
.455
.692
424
.629
.606
3344
.582
260
Shot
.310
.402
.317
.368
-364
.326
.383
.33
.255
.387
.370
.271
.259
.336
.382
.402
.380
.366
319
331
329
.463
.521
-322
.412
.599
.380
.561
5549
317
.521
.42
Impul
.389
-452
-352
.403
.410
.366
.43
.373
.206
.50
.144
.23
.277
.379
437
.403
-438
3322
-336
.300
.348
.509
.564
-343
.447
.658
.22
.605
556
352
.559
.257
Gaus
.384
.326
-109
4003
.302
.223
.363
.292
.33
400
310
.302
.399
4118
.423
.418
.358
350
.361
.363
.492
.539
.343
.410
.613
.397
.377
.570
.326
.516
.277
Quant
193
.212
.215
198
.144
189
.99
.187
1655
.189
.179
.146
.145
.186
.17
.178
199
.159
.161
.175
144
.200
.236
.206
.192
.217
.183
.213
.229
-215
.186
.147
Flast
.129
.13
130
129
131
121
.144
.114
.111
119
.188
108
.177
.125
122
126
.127
128
.22
.121
.33
156
130
130
.142
.118
122
.153
123
126
112
Zoom
.201
.207
-232
.008
201
2255
208
.24
197
.194
.185
.199
1844
.218
.214
190
188
.204
196
192
-099
222
.246
196
.231
.262
.004
.230
.232
193
166
Motio
.116
.230
.277
.288
242
.255
.663
280
.211
2144
.163
145
.005
.31
.378
.212
198
.40
.211
.22
.264
.217
.279
.663
.411
320
253
.216
.250
260
161
Glass
.212
.232
.57
.392
.295
.20
.347
.30
.300
.279
.273
.204
.171
.447
.454
.368
.301
.288
-282
.237
.225
.332
.232
312
.400
.289
.363
.466
.221
-257
4122
.184
Defoc
.211
.244
.393
.427
3188
.348
.356
.356
5113
.251
291
J80
1445
501
5532
499
.288
.257
-353
.211
.22
.390
.266
.432
.300
.357
450
447
.228
393
333
179
Underline: Second best in col. Blue : Best in row. Red Worst in row.
Contr
.187
.214
.33
200
-209
193
.213
161
.227
170
119
.177
208
.201
.66
.255
.91
.238
.177
149
.234
-342
-238
164
.393
180
199
440
.244
160
131
Snow
.511
.300
.582
-182
.02
.504
.299
.811
.352
479
.303
.275
.219
.500
.540
.108
.354
.40
.384
433
.515
4435
522
.515
-663
480
.596
428
.582
.556
215
Frost
.277
.271
.277
.283
.317
.32
.299
199
.288
.237
.213
.25
.20
324
.239
.E88
.25
.211
.215
.255
.305
.352
-356
.321
.374
337
295
.421
.271
.111
-113
Fog
.156
331
.155
1555
.144
156
1 12
151
.331
1669
130
116
.114
.166
.152
127
188
157
1102
11
126
.161
.202
166
137
.301
141
.550
-328
155
137
122
Dark
.280
.343
.376
294
.255
2644
.300
.289
222
.274
.261
.243
.238
280
314
.00
.277
-251
.259
.27
.22
.255
.310
.476
.282
404
460
376
.357
245
Bright
130
183
129
127
130
128
121
121
.111
135
128
106
102
.128
.18
.133
131
127
127
119
110
133
1881
132
124
173
.12
127
1993
129
123
104
mDFE
119
.44
.144
117
117
118
.008
.112
102
.13
100
099
.944
.17
.133
.096
.17
115
.155
110
1122
123
150
.17
113
1144
.144
116
146
114
110
098
Method
DepoDe RUS [17]
MonoDeptha moye [17]
HK [17]
[17]
[48]
[59]
CeDept [60]
HR Depth [39)
DIFENer [67]
Many ManyDepthh Depth [56]
FSREDepth [22]
MonoViT [66]
MonoVi ViT II [66]
DynaDepth RUS [64]
[64]
RA-Depth [42]
[6]
Lite-Monu Monu Tay [63]
[63]
Lite- MieeMononnn lox [63]
Lite- Mono Lege [63]
[17]
MepoDepthh nog [17]
[17]
DeptHHints [57]
DepthHints noy [57]
DepthHints [57]
Depot RIB [17]
DonoDeptha nog [17]
MonoDepthz HK [17]
CADepth [60]
Mono VIT [66]
ManoDepthh Depohhp 850
MaskOu DN
DynaDepth a5a
InDDepth RIS
Lite- Mono Small
MepoDeptht RIS
DenoDepth HK
31

---

Table E: The Corruption Error (CE) of 32 monocular depth estimation models in KITTI-C. All scores are given in percentage (%). Blocks from top bottom: [1st]
the baseline MonoDepth2 RIS [17]: [2nd] methods w/ monocular inputs, [3rd] methods w/ stereo inputs; [4th] methods w/ monocular + stereo inputs. Bold: Best in
JJPGG
100.0
87.2
101.0
107. .7
90.3
96.9
99.3
80
103.1
92.9
114.3
78.6
75.5
91.8
110.2
98.0
104.6
82.1
83.7
81.2
93.9
99.0
90.8
100.6
98.99 0
97.5
93.4
91.3
90.3
101.0
98.0
73.5
Pixel
100.0
105.5
95.2
106.9
M6.9
98
100.0
99.3
93.1
93.
101.4
1014
111.7
93.
97.2
105.5
120.7
97.9
101.4
106.2
10
110.3
99.3
113.1
103-5
108.3
104.8
97.2
93.8
111.0
95.2
100.0
101.4
ISO
100.0
116.8
88.7
109.5
107.5
92.0
115.5
96.4
69.6
116.5
104.9
74.7
73.5
97.7
115.3
117.3
112.9
92.8
91.8
93.6
93.3
127.1
143.3
88.7
117.3
178
109.3
162.1
156.2
88.7
160.0
67.0
Shot
100.0
118.2
93.2
108.2
107
95.9
112.7
98.8
75.0
113.8
108
79.7
76.2
98.8
112.4
118.2
111.8
98.8
93.8
98.2
96.8
136.2
153.2
94.7
121
176.2
111.8
165.0
161.5
93.2
151.1
71.2
Impul
100.0
116.2
90.5
103.6
105.4
94.1
111.3
95.9
08
115.7
106.4
75.3
71.2
97.4
112.3
103.6
112.6
87.9
86.4
87.1
89.5
130.9
145.0
88.2
114.9
169.2
108.2
155.5
153.2
90.5
113.7
66.
Gaus
100.0
114.8
84.9
106.5
105.0
94.3
110.2
94.5
76.0
112.0
104.2
80.7
78.7
98.7
108.9
110.2
108.9
93.2
91.2
91.0
94.5
128.
140.4
89.3
114.6
1.5 6
103.5 8.44
150.3
148.4
84.9
142.5
72.1
Quant
100.0
109.5
111.4
102.66 6
95.3
97.9
99.5
96.9
85.5
97.9
92.8
75.7
75.
96.4
102.
92.2
1
82.4
83.4
90.7
76
103. 6
122 3
106.7
99.5
128.8 0
94.8
105. 2
118.7
111.4
965
76.2
Flast
100.0
114.7
95.4
100.8
100.0
101.6
93.8
96.1
88.4
103.9
92.3
91
83.7
98.5
94.6
97.7
98
98.2
100.0
98.8
101.7
120.9
100.8
100.8
110.1
91
90.
118.6
95.4
97.7
86
Zoom
100.0
103.0
115.4
103.5
100.0
107.0
103.5
1114
98.0
96.5
92.0
89
91.5
103.0
108.5
106.5
91.5
92.5
101.5
97.5
95.5
104.0
113.9
122.4
97.5
114.9
1304
101.5
114.4
115.4
96.
83 63.6
Motio
100.0
92.1
106.5
125.2
105.6
112.0
131.9
121.8
129.6
97.7
99.1
75.5
67.1
141.2
147.2
175.0
98.2
91.7
113.9
102.3
101.9
122.2
100.5
129.2
121.8
111.6
148.2
131.0
100.0
106.5
120.4
74.5
Glass
100.0
95 .
106.2
162.0 0
121 0
132.2
143.4
124.0
136.4
115.3
112.8
84.3
70.7
14344
187.6
152.1
124.4
102.5
116.5
97.9
93.0
137.2
95.0
128.9
169. 4
119.4
150.0
143.0
91.3
106.2
170.3
76.0
Defoc
100.0
104.1
161.1
175.0
130.3
142.6
145.9
145.9
210.3
104.1
119.3
73.8
59.4
205
218.0
204.5
108.8
105.3
144.7
91.8
91.0
159.8
109.0
177.1
143.4
146.3
188.1
181.2
93
161.1
138.
73.4
col. Underline: Second best in col. Blue Best in row. Red Worst in row.
Contr
100.0
222.5
114.4
124.6
107.0
111.8
103.2
113.9
86.1
121.4
90.9
63.6
62.6
111.2
107.5
88.2
141.7
102.1
127.3
91.7
79.7
125.1
182.9
127.3
87.7
210.2
96.3
106.4
235.3
114.4
91 4
70.1
Snow
100.0
78.1
113.9
96.3
96.3
98.6 .6
103.5
94.1
68.9
93.7
76.9
53.8
52
90
107.4
80.8
97.5
60.3
84.2
75.2
84.7
100.8
85.1
103.5
100.8
90.6 .
93.9
104.9
83
113.9
108. 8
42.1
Frost
100.0
112.6
97.8
103.6
102.2
114.4
117.0
100.7
71.8
104.0
85.6
76.9
81.2
104.3
117.0
86.3
122.0
81.2
90.6
88.5
92.1
110.1
127.1
128.5
115.9
135.0
114.4
106.5
152.0
97.8
112.3
76.9
Fog
100.0
200.7
100.0
100.0
98.7 7
100.7
91.6
97.4
84.5
109.0
89.7
74.8
73.6
100.7
81.9
121.3
101.3
104.5
92.9
81.3
103.9
188.4
107.7
88.4
194.2
91.0
96.88
211.6
100.0
88.
78.7
Dark
100.0
122.5
134.3
105.0 0
101.8
94.3
107
103.2
79.3
97.9
93.2
86.8
85.0
103.2
106.4
112.1
107_1
91.8
89.6
92.5
81.
124.3
150.7
101.8
110.7
170.0
100.7
1443
164.3
134.3
127.5
87.5
Bright
100.0
140.8
99.2
17.7
100.0
98.5
93.1
93.1
85.4
103.9
98.5
81.5
78.5
08.3
98.5
86.9
100.8
97.7
97.7
91.5
84.6
102.3
130.2
101.5
95.4 4
133.1
93.9
17.7
148.5
99.2 2
91.
80.0
mCF
100.0
119.8
106.1
113.4
104.1
104.7
110.1
103.7
95.0
105.4
99.1
79.3
75.0
110.4
120.0
112.7
100.3
92.9 9
100.3
93.2 2
90.8
117.7
129.0
11..5
111.4
141.6
112.0
124.3
136.3
106.1
118.3
75.4
Method
MonoDepthh RiS [1]]
MonoDeptha moye [17]
HK [17]
[17]
MaskOu DN [59]
[48]
CeDepth [60]
HR Depth (39)
DIFFNet [67]
Many Danny/eephggu RUS [56]
Depth [22]
MonoVij [66]
MonoVi ViT II [66]
DynaDepth RUS [64]
DynaDepth a5a [64]
RA-Depth [42]
[6]
Mon-Monu Tay [63]
[63]
[63]
[63]
MepoDepth, RIS [17]
MepoDepthh nax [17]
DenoDe HK [17]
DepthHints [57]
DepthHints noy [57]
[57]
Depth RIB [17]
MonoDeptha nog [17]
MonoDepthh HK [17]
CADepth [60]
Momo VIT [66]
MonoDepthh Depttt 850
FSRE- (SRE-Depli
TriDepth RIS
Lite- Mono Small
Lite- Mhe-Mono Box
Lite- Mono Lage:
ManoDepthh
DepthHints
32

---

Table F: The Resilience Rate (RR) of 32 monocular depth estimation models in KITTI-C. All scores are given in percentage (%). Blocks from top to bottom: [1st]
the baseline MonoDepth2 RIS [17]: [2nd] methods w/ monocular inputs; [3rd] methods w/ stereo inputs; [4th] methods w/ monocular + stereo inputs. Bold: Best in
JPEG
91.3
96
90.5
59.4
93.2
91.8
90.3
92.8
88.9
93.3
87.1
93.9
94.0
92.0
88.4
89.
00,00
94.8
94.5
93.5
00.9
91.9
96.7
89.6
91.1
91.2
03.4
92.0
96.4
90.5
90.8
04.8
Pixel
97.1
99.0 19.0
97.3
95.7
97.1
96.9
96.0
97.4
96.3
97.3
95.7
93.0
95.5
97.3
95.5
97.2
96. 6.44
95.6
95.
93.5
97.6
98.4
96.3
95.1
95.9 5
97.9
97.7
98.2
97.3
96.1
94.5
ISO
69.5
63.9
74.0
65.
66.0
72.9
61.9
70.5
81.3
62.5
06.0
78.8
78.9
70.3
02.2
60.3
63.7
72.3
72.8
71.6
71.1
57.8
52.2
74.3
61.4
61.3
35.6
42.0
46.1
74.0
47.0
82.0
Shot
74.9
69.8
77.1
71.6
72.0
76.4
69.2
74.8
83.0
69.9
70.7
80.9
81.8
75.2
69.7
66.2
70.2
75.0
77.0
71.8
74.7
61.2
56.4
76.5
66.3
69.2
46.3 3
49.7
52.5 8
77_1
53.5
83.9
Impul
69.1
64.0
73.1
67.6
66.8
71.9
63.6
70.6
817
62.7
65.8
78.5
79.8
70.3
63.5
66.0
63.7
74.4
75.0
74.2
72.6
56.0
51.3
74.4
62.3
61.6
39.5
44.7
47.3
73.1
19.6
82.3
Gaus
69.9
05.3
76.1
66.9
67.6
72.3
64.7
71.7
78.8
65.0
67.3
76.0
77.0
70.3
63.8
65.9
72.5
73.5
71.8
70.9
57.9
54.2
74.4
63.1
67.3
44.7
47.9
50.4
76.1
51.01
80.1
Quant
91.6
88 6
90.8
92.4
92.0
90.6
91.6
93.0
92.5
92.1
94.8
91.4
92.2
90.5
90.9
90.7
95.0
91 8
92.7
94.9
91 91.2
80.9
89.9
91.1
91.2
87.0
90.2
90.3
58.0
91.5
94.5
Elast
98.9
99.5
99.0
98.5
98.6 6
98.5
98.5
98.7
98.7
98.8
98.3
97.9
98.
98.9
98.7
97.1
98.0
98.6 6
98.5
97.9
97.9
98.6
90
98.5
98.
98. d
99.
98.
90.9 0.
99. 0
98. 2
98.3
Zoom
90.7
92. 6
86.7
89.7
00.5
89.0
88.8
87.4
89.4
91.9
91.5
91.1
90.1
59.8
58.2
87.0
91.7
92.0
89.9
90.3
90.0
90.2
90.7
85.4
90.6
82.4
88.8
910.1
90.2
86.7
90.7
92.1
Motio
89.0
93.6
86.9
S1.9
87.4
85.9
80.2
83.0
80.2
90.0
88.2
92.9
94.4
78.7
76.9
68.5
89.2
90.6
85.2
87.5
86.9
83.9
92.1
81.7
83.
75.9
87.6
$1.1
91.8
86
83.2
92.9
Glass
86.0
89.7
83.9
68.5 9
79.4 8
77.1
73.2
78.8
74.6
82.2
81.6
88.4
91.5
74.0
61.6
09.9
79.2
85.0
81.1
85.7
86.3
76.2
90.1 4
77.9
66.5 55
71.1
82.1
74.0
1.
83.9
06 1
90.4
Defoc
85
87.2
08.5
64.9
77.2
73.9
72.2
72.5
54.2
85.1
79.6
91.0
94.4
56.3
52.8
55.4
N2.9
84.0
73.1
87.2
86
09.9
69.6
86.4
04.3
73.3
60.4
74.3
62.6
90.4
08.5
71.1
90.9
col. Underline: Second best in col. Blue Best in row. Red Worst in row.
Contr
92.3
2--2
58.7
N6.5 9
90.6
89.7
90.3
88.6
93.4
88.1
23.2
03.
97.8
97.5
80.7
90.1 1
92.4
N1.2
01.4
86.1
92.5
94.8
87.3
77.4
86.3
94.3
91.5
70.1
90.6
05.6
88.7
M1.4
06 0..2
Snow
55.5
70.2
47.2
57.5
57.5
56.2
52.8
58.5
72.2
59.4
68.1
80.5
80.7
55.8 55.6
50.9
64.9
56.9
73.0
04.4
69.2
63.1
55.3
66.5
53.3
54.7
58.0
62.0
52.5
67.0
47.2
49.9
86 9
Frost
82.1
80.4 60.4
82 3
80.8
81.2
77.4
75.8
81.2
89.2
81.2
85.6
87.4
85.5
80.5
76.2
84.2
75.0
87.6
84.6
81.8
83.0
79.3 79.3
76.2
72.9
76.6
76.2
72.3
79.8
67.8
82.3
77.4
87.2
Fog
95.9
95.4
95.7
95.0
95.7
96.2
95.6
96.8
94.8
96.6
98.1
97.8
95.6
96.6
92.0
95.3
94.7
965.2
97.3
95.7
83
94.3
97.3
95.9
80.7
965
78.7
95.4
97.0
97.2
Dark
81.7
76.8
70.4
S1.0
$1.0
83.5
78.5
80.1
86.6
82.8
82.9
84.0
84.1
80.5
79.1
75.9
79.3
84.0
84.6
83.3
86.1
74.3
68.0
$1.0
77.8
S0.1
60.5
67.4
63.2
70.4
72.3
83.6
Bright
98.8 .8
95.4
98 3
98 J
98.5 5
98.9
98.5
99.0
99.0
98.6
97.9
99.2
99.1
98.3
98.
98 6
98.0 0
99.0
99.1
988.9
96.4
98.3 3
98.8 8
98.0 0
95.5
98. 8
94.5
98.3
98 5
99.2 2
mRR
84.5
82.5
82.4
S0.6
83.0
83.3
80.1
82.9
85.4
83.1
83.9
89.2
89.7
$1.5
78.0
78.8
$1.6
86.7
84.7
86.0
85.5
79.1
79.2
81.6
80.1
79.5
73.2
75.4
76.7
82.4
76.7
90.4
Method
MonoDepthh Depthy Ria [17]
Mono Deptha spp [17]
[17]
[17]
MaskOcc m [48]
DN 218 [59]
CeDephh [60]
HR- Depth [39]
DIFFNer [67]
Depth [56]
EDepth [22]
MonOViT [66]
[66]
[64]
Re-Depth [42]
[6]
[63]
[63]
[63]
[63]
[17]
[17]
[17]
DepthHints [57]
Depthfinnts uppt [57]
[57]
[17]
Meptlg app [17]
[17]
CADephh [60]
[66]
MopoDepphy HR
Depthh 250
Monn VIT HR
DynaDopth R18
DynaDepth R30
TriDepth RUS
Lite Lite-Mon Tayy
Lite: Mono Small
Lite- Mono Buss
Lite Mono Largs
MepoDeptha R18
Depthn atpi
Mepolepplyyet HR
288
Depthyy HK
MoROViT
DepthHints
MenoDepthy
33

---

The Depth Estimation Error (DEE) of 10 monocular depth estimation models in the NYUDepth2 CC benchmark. Bold: Best col. Underline: Second best Blue
JPEG
.175
.175
.211
.171
338
.272
3322
.265
.179
.108
Pixel
.139
139
118
111
.35
.247
.144
.166
160
100
ISO
.474
.474
.481
199
461
.449
.443
.555
-305
.157
Shot
.392
.302
.401
.191
450
.144
.222
.16
280
148
Impul
-512
.512
-519
.213
471
463
4442
-461
321
156
Gaus
.442
442
.488
.00
465
445
432
439
.94
155
Qnant
.180
180
.216
165
.391
.28
333
.287
.192
.119
Elast
.182
.182
.204
133
.381
-306
.366
.55
.15
.122
Zoom
.322
.32
330
.222
433
422
335
405
300
.37
Motio
.231
.231
.245
1555
400
334
535
313
191
119
Glass
.262
.262
.20
166
401
337
333
.327
-200
172
Defoc
-337
.337
-339
154
416
3377
360
.380
-20
127
Contr
-265
265
.289
180
448
336
406
.398
-235
123
Dark
.269
.269
.293
182
427
330
.33
.318
.279
.150
Bright
.149
.49
.79
136
388
.269
326
.269
.147
.099
Best in row. Red Worst row.
mDEE
.22
122
.15
136
.372
.244
.312
.213
125
086
Method
AdaBins [1]
BTS [31]
AdaBins s00 []]
DPT vIT VIT' [4.3]
SimIPU me [34]
[34]
[34]
[34]
DepthFormer SwinT-1k [35]
DepthFormer 22% [35]
SimIPU TinIPUmmeqonn ImsgeNat
SimIPU WaymeODen
SimIPU
Table G:
in col.
34

---

Table H: The Corruption Error (CE) of monocular depth estimation models NYUDeptk2-C benchmark. All scores are given percentage (%). Bold: Best in col. Underline: Second best in
JPEG
100.0
113.6
137.0
113.0
245.5
176.6
209.1
172.1
116.2
70.1
Pixel
100.0
112.1
150.0
137.9
302.4
199.2
253.2
198.
129.8
81.5
ISO
100.0
124.7
126.6
52.4
121.3
118.2
116.6
119.7
80.3
41.3
Shot
100.0
133.3
136.4
65.0
153 1
140.8
143.5
141.5
98.3
50.3
Impul
100.0
136
138.5
57.0
126
123.8
118.2
123
85.8
41.7
Gaus
100.0
113.3
117.4
51.3
119.2
114.1
110.8
112.6
754
40.8
Quant
100.0
110.2
143. 1
109.3
258 9
191 .4
220.5
190.1
127
78.8
Elast
100.0
119.0
133.3
90.9 .9
249.0 0
200.0
206. 5
167.3
1144
73.2
Zoom
100.0
96.9
101.6
60.4
112
109 .
100.5
105.5
78.1
61.7
Motio
100.0
125.5
133.2
81.2
217
776.1
882.1
770.1
103.8
64.7
Worst row:
Glass
100.0
127.2
135.9
80.6
6
194.7
163
161.7
158.7
126
83.5
Defoc
100.0
143.4
144.3
65.5
177.0
160.4
153.2
161.7
93.6
54.0
|
Blue Best in row. Red
Contr
100.0
125.0
136.3
81.9
211.3
177.4
191.5
187.7
110.9
58.0
Dark
100.0
138.7
151.0
93.8
220.1
190.7
192.3
179.1
1438
77.3
Bright
100.0
112.9
135.6
102.3
203
203.8
247.0
203
111.4
75.0
mCE
100.0
122.8
134.7
83.2
200.2
163.
173.8
159.5
106.3
63.5
Method
AdaBins [1]
BTS a5o [31]
[43]
[34]
SimIPU ImageNec [34]
SimIPU SimIPUKIrn KuT [34]
SimlPU Waym-Oppn [34]
DepthFormerswe Swint-1k Swint- [35]
DopehFomerro Swinf-33 [35]
AdaBins 830
DPT wnD vnn
SimlPU mmn
35

---

Table The Resilience Rate (RR) 10 monocular depth estimation models the NYUDepti2-C C benchmark. All scores are given in percentage (%). Bold: Best in col. Underline:
JPEG
95.3
94.0
93.7
95.6
99. 0
96.3
98.6
97.1
93.8
97.6
Pixel
98.7
98.1
96.7
96. 0
U9.5
99.6
99. 99.7
99. 6
95.9
98 4
ISO
69.8
50 90.9
61.6
92 7
85.
72.9
8L.0
72.0
79.4
92.2
Shot
79.5
60.3
71.1
93 6
87.6
77.5
84.0
77.2
81.3
93.2
Impul
70.5
55.6
57.1
91.1
81.2
71.0
81.1
71.2
77
92.3
Gaus
68.7
63.6 6
04.4
92. 6
55.2
73.4
82.6
74.1
80.7
92.0
Quant
95.6
93. 4
93.
96.6 6
97.0
94.1
97.0
91.2
92.3
96.4
Elast
95.4
93.2
94.5
99.
98.6 6..6
91.8
99
98.3
94.3
97.2
Zoom
69.4
71.5
72.5 5
88 9
90.
76.5
89.2
78.6
80.0
83.5
Motio
91.9
87.6
89.7 7
97.6 8
U6.5
89.4
96.7
90.8
92.5
96.4
Glass
89.4
84.1
85. .
96.5
95.4
87.7
97.0
88. 9
84.6
90.6
Second best in col. Blue Best in row. Red Worst row.
Defoc
86.2
75.5
78.5
97.9
93.0
82.4
93.0
81 11.9
80
95.5
|
Contr
88.7
83.7
84.4
91.9
87.0
82.5
86.3
79.5
87.4
96.0
Dark
90.8
83.3 3
84.0
94.7
91.2
83.3
91.1
86.1
82.4
93.0
Bright
97.8
96.9 9
97.5
100.0 0
97.3 .
96.7
98.0 0
96.6
97.5
98.6
mRR
85.8
80.6
81.6
95.3
92.5
85.0
91.6
85.7
87.3
94.2
Method
AdaBins [1]
BTS a50 [31]
AdaBins acu [1]
DPT vIT [4.3]
SumIPU mupt [34]
SimIPUmmaee Ioogehae [34]
SimIPU [34]
SimIPU WoymoOpen [34]
FepthForeer Swint-lk [35]
Former Swart-22 22k [35]
36

---

Table J: The Depth Estimation Error (DEE) 32 monocular depth estimation models in the KITTI-S benchmark. Blocks from top to bottom: [1st] the baseline
MonoDepth2 R18 [17]: [2nd] methods w/ monocular inputs; [3rd] methods w/ stereo inputs; [4th] methods w/ monocular sterco inputs. Bold: Best in col.
Water Color
.317
.464
.384
.516
4001
.331
.360
.300
.352
.221
.585
.589
.475
.436
.252
.336
.264
.206
.486
.6443
.438
443
.336
.250
Relief
.256
.259
.288
.20
.242
.265
.294
.310
.261
.205
.298
.460
.379
.249
.237
.295
.229
.263
.1633
.305
.99
.264
.314
.117
Stained Glass
.666
.596
.554
.553
.543
.531
.502
.553
.417
.356
.534
.505
.476
.573
.446
.435
.476
5507
.770
.774
.809
.743
.552
.338
Sketch
.288
.576
.608
5585
3388
.385
.331
.318
.325
4002
555
.372
.517
.356
.445
.459
.458
.584
.316
.159
3321
.615
331
Shadow Play
.589
.314
.200
3317
.316
.44
.301
.208
.363
.372
.425
.318
-232
.307
.240
.210
:388
.475
.367
-45
.250
.205
Penciling
.188
.290
.333
422
470
.337
.276
.270
.237
.338
.492
.342
4880
.285
.374
.201
.363
-449
.357
-138
.304
.479
.226
Paint
.228
-249
.355
.336
413
3306
.211
.300
-200
2003
.293
.47
.310
3352
.283
.28
.211
.288
4422
:309
402
.38
.423
200
Murals
.326
412
.358
.89
421
.299
.386
344
.240
-288
399
340
440
.254
-221
.231
344
499
440
.311
445
4776
.008
Kids' Draw
.20
-235
.65
.20
.250
.228
206
205
.21
196
-264
-287
.262
.217
.211
.210
.2188
3306
.299
-234
-267
236
-200
192
Ink Paint
-889
-260
-293
.379
-238
.206
.212
.944
.252
-291
-259
.354
.256
-251
.218
.262
.240
.320
-587
-275
.510
.255
-224
Digital Art
4334
-289
.356
.881
3300
-332
3351
373
.277
222
3358
408
333
400
330
.346
.313
435
4883
-344
414
350
444
-266
Cartoon
.321
.279
-358
444
446
318
.227
.211
.275
.779
3349
.502
.3144
.004
-236
.210
.244
.208
.382
.351
3315
-336
-398
173
Underline: Second best in col.
mDEF
.365
.378
.368
400
406
-344
.300
323
.293
2388
.37
4447
3665
.37
2880
-303
.2888
.323
456
-227
411
403
.401
.232
Method
Mepthh RI8 [17]
Depthh moy [17]
MaskOce RIS [48]
DN [59]
CeDepth [60]
HR. Depth [39]
DIFFNet [67]
Many Depth [56]
FSREDepth [22]
MonoVil ViT [66]
DynaDepth [64)
DynaDepth e:n [64]
R Depth [42]
TriDepthni [6]
Lite Mono Toy [63]
[63]
[63]
[63]
Depth RIS [17]
MenoDepthz.oo mays [17]
MonoDepdh2 RIB [17]
DeptDe nog [17]
CADephh [60]
MonoViT [66]
Lite-Monosee Small
Lite Mono Box
Lite-Mi Mono aage
37

---

Lovel
Smonthy
Level
Leval
Figure J: Qualitative results of MonoDepth2 [17] under each corruption type across five severity levels. We show examples from the third level in the first column. The lighter regions correspond to near distances and vice versa. Best viewed in color. Zoomed-in in for more details.
38

---

Loon!
Smontiy Looal
Level
Leval 4
Lovel
Figure K: Qualitative results of Lite-Mono [63] under each corruption type across five severity levels. We show examples from the third level in the first column. The lighter regions correspond near distances and vice versa. Best viewed in color. Zoomed-in for more details.
39

---

Lavel
Lovel
Smontiy
Lovel
1.
1
115
T
L.5
11b
+18
+
+1b
t1b
f
97
Figure L: Qualitative results of Mono [66] under each corruption type across five severity levels. We show examples from the third level in the first column. The lighter regions correspond near distances and vice versa. Best viewed in color. Zoomed-in for more details.
40

---

ManeDa
CIF
Ute Mino
MoroVi
Du5
Figure M: Qualitative results five monocular depth estimation models in the KITTI-C benchmark, including MonoDepth2 [17], DIFFNet [67], RA-Depth [42], Lite-Mono [63], and MonoViT [66]. The lighter regions correspond to near distances and vice versa. Best viewed color. Zoomed-in for more details.
41

---

Irpur
MonoDapp12
DIF-Not
26 Depit
Uhe Mono
MonoVTT
Food
Figure N: Qualitative results of five monocular depth estimation models in the KITTI-C benchmark. including MonoDepth2 [17], DIFFNet [67], RA-Depth [42], Lite-Mono [63], and Mono ViT [66]. The lighter regions correspond to near distances and vice versa. Best viewed color. Zoomed-in for more details.
42

---

MonoL2epth2
Ner
D0-Dapth
Inn-Aooo
Monovi
MaDi Colo
Figure O: Qualitative results of five monocular depth estimation models in the KITTI-S benchmark, including MonoDepth2 [17], DIFFNet [67], RA-Depth [42], Lite-Mono [63], and MonoViT [66]. The lighter regions correspond near distances and vice versa. Best viewed in color. Zoomed-in for more details.
43

---

Public Resources Used
In this section, we acknowledge the use of public resources, during the course of this work:
KITTI Vision Benchmark
CC BY-NC-SA
NYU Depth Dataset V2
Unknown
nuScenes
CC BY-NC-SA 4.0
nuScenes-devkit"
Apache License 2.0
Cityscapes
Custom Cityscapes License
Seegy-Cityscapes"
Custom Foggy- Pragy-Cityscape License
ACDC
Custom ACDC License
SeasonDepth Benchmark Toolkit"
CC BY-NC-SA 4.0
Make3D
Custom Make3D License
MonoDepth2
Custom MonoDepth2 License
DepthHints
Custom DepthHints License
MaskOco
Custom MonoDepth2 License
DNet
Custom MonoDepth2 License
CADepth
MIT License
HR-Depth 18
MIT License
"
Unknown
ManyDepth 20
Custom ManyDepth License
FSRE-Depth"
MIT License
MonoViT
MIT License
DynaDepth
Unknown
RA-Depth
Unknown
TriDepth
GNU General Public License v3.0
Lite-Mono
Unknown
html.
htttpp//ggghabbl.cco/////ao////p 21
19
25
44

---

Monocular-Depth-Estiee Estimation-Toollle
Apache License 2.0
BTS
GNU General Public License v3.0
AdaBins
GNU General Public License v3.0
DP
MIT License
SimIPU3I
Apache License 2.0
DepthFormer
Apache License 2.0
ImageCorruption
Apache License 2.0
3DCC
Attribution 4.0 International
ImageNeetC**
Apache License 2.0
StylizeDatasets
MIT License
PyTorch-AdalN*7
MIT License
28
depthformer.
httppp///....aaooo////eaeeaa/aas 37
45


Reviews:
[{"review_id": "BWuMqN2Lqz", "reviewer": "Reviewer_j7EB", "summary": "", "questions": "", "limitations": "The authors have included a section discussing the limitations of their work. In my opinion, the limitations associated to the use of synthetic data, currently missing, should be discussed. I cannot see a potential negative societal impact of this work, and it is not discussed in the authors' work.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "+ The topic addressed is relevant.\n+ The paper is mostly well structured and written. The evaluation metrics are very clear, also the dataset specifications, and the experimental validation of the data is thorough and solid. \n+ The validation is extense, including results for 42 depth models and interesting conclusions. \n+ The dataset has been used in a challenge at ICRA 2023, with a reasonable success, so the adoption of the data has already started.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "Ic0T2XF8rc", "reviewer": "Reviewer_YXSL", "summary": "", "questions": "", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "Comprehensive experimental results and discussions.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "gJ4963E8V7", "reviewer": "Reviewer_ji4t", "summary": "", "questions": "", "limitations": "The authors addressed the limitations of their work, but not the societal impact.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed RoboDepth benchmark on evaluating the robustness of depth estimation models under corruptions addresses an important gap in current testing procedures, which often do not account for these common situations in practical scenarios.\n\n2. The benchmark consists of 18 corruptions across three categories for evaluating indoor and outdoor depth estimation models, covering a broad range of realistic conditions. And it presents a comprehensive evaluation on 42 existing depth estimation models.\n\n3. The authors have made their well-documented work available on GitHub.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "F5wYs1BF9f", "reviewer": "Reviewer_76NB", "summary": "", "questions": "", "limitations": "The authors adequately addressed the limitations.", "rating": 9, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The paper introduces the RoboDepth OoD cases, which comprise 18 different corruptions covering various aspects of real-world scenarios. This comprehensive test suite allows for a thorough evaluation of depth estimation models. The dataset collections are comprehensive and meaningful\n\nEvaluating depth estimation models under out-of-distribution OoD scenarios, such as weather and lighting conditions, sensor failure, and data processing issues, aligns well with real-world applications, especially for autonomous driving. \n\nThe benchmarking is comprehensive: it  provides a holistic view of the current state-of-the-art 42 existing depth estimation models.\n\nThe paper's detailed description of the RoboDepth test suite and the evaluation process ensures that the experiments can be easily reproduced by other researchers. \n\nThe project is well-documented and maintained.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mCIM9H6Gom", "reviewer": "Reviewer_pgHi", "summary": "", "questions": "", "limitations": "The corruptions are simulated and thus easy to reproduce during training for future methods that will want to use this benchmark. In my opinion this is not a deal-breaker, but still a significant limitation of the proposed method.\n\nReal data might have bee desirable here, even if it implies creating a dedicated dataset, which might not be feasible.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "Robustness is an important topic in critical applications like robotic navigation. The authors propose a systematic approach to test different methods over a large set of possible corruptions.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Four out of five reviewers liked this paper, which offers valuable data for testing depth estimation models. The only reviewer with a rating below the acceptance threshold (5) raises minor concerns about the writing and positioning of the work. The authors addressed them well in the revision (but the reviewer did not seem to check back).
Decision: Accept (Poster)