SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lingdong Kong $^{1,2}$  Shaoyuan Xie $^{3}$  Hanjiang Hu $^{4}$  Lai Xing Ng $^{5,6}$  Benoit R. Cottereau $^{2,6,7}$  Wei Tsang Ooi $^{1,6}$
$^{1}$ National University of Singapore  $^{2}$ CNRS@CREATE
$^{3}$ Huazhong University of Science and Technology
$^{4}$ Carnegie Mellon University  $^{5}$ Institute for Infocomm Research, A*STAR
$^{6}$ IPAL, CNRS IRL 2955, Singapore  $^{7}$ CerCo, CNRS UMR 5549, Université Toulouse III
https://github.com/1dkong1205/RoboDepth
Depth estimation from monocular images is pivotal for real-world visual perception systems. While current learning-based depth estimation models train and test on meticulously curated data, they often overlook out-of-distribution (OoD) situations. Yet, in practical settings - especially safety-critical ones like autonomous driving - common corruptions can arise. Addressing this oversight, we introduce a comprehensive robustness test suite, RoboDepth, encompassing 18 corruptions spanning three categories: i) weather and lighting conditions; ii) sensor failures and movement; and iii) data processing anomalies. We subsequently benchmark 42 depth estimation models across indoor and outdoor scenes to assess their resilience to these corruptions. Our findings underscore that, in the absence of a dedicated robustness evaluation framework, many leading depth estimation models may be susceptible to typical corruptions. We delve into design considerations for crafting more robust depth estimation models, touching upon pre-training, augmentation, modality, model capacity, and learning paradigms. We anticipate our benchmark will establish a foundational platform for advancing robust OoD depth estimation.
Monocular depth estimation (MDE) involves predicting a scene's depth information from monocular images, without relying on data acquired from more sophisticated sensors [11, 28, 25, 37]. These images are predominantly captured using RGB cameras mounted on diverse platforms like drones, mobile robots, and vehicles [65, 41, 9, 30]. As an instrumental facet of visual perception, precise MDE paves the way for a broad array of applications. Bolstered by the rise of learning-based paradigms, numerous MDE algorithms have emerged, demonstrating remarkable depth estimation performances on standard benchmark datasets [14, 52, 49, 8, 47].
However, the resilience of existing MDE models to out-of-distribution (OoD) challenges is yet to be thoroughly explored, especially under the lens of real-world corruptions such as adverse weather [19, 46] and sensor malfunctions [29]. The prevailing learning-based visual perception models often display heightened sensitivity to nuances in lighting, noise, texture variations, among other factors, which are compromising the accuracy of depth predictions [18, 23]. The ability to generalize across new scenes, objects, and backgrounds, especially when they have not been part of the training data, is another pivotal challenge [40].
Despite the strides achieved on relatively pristine datasets [14, 49, 8], a lacuna exists: a robustness benchmark tailored to foster the evolution of resilient and scalable MDE systems. In light of these
challenges, models dedicated to MDE often inadvertently embed systematic errors, stemming from real-world image imperfections like altered lighting, motion blur, shadows, and data compression, which the current MDE solutions rarely address effectively [4, 34].
Seeking to bridge this gap, our contribution charts the inaugural path towards robust and reliable MDE, unveiling the KITTI-C, NYUDepth2-C, and KITTI-S benchmarks. Contrasting prior works that merged datasets for cross-domain MDE [44, 33, 62] or devised adversarial patches to subvert MDE models [7, 4], our benchmarks meticulously simulate commonplace corruptions that are intrinsic to real-world settings. As delineated in Fig. 2, we structure eighteen corruption varieties across three cardinal categories:  $i)$  weather and lighting conditions,  $ii)$  sensor malfunctions and movements, and  $iii)$  data processing complications. Further stratified by diverse severity, these corruptions encapsulate a gamut of scenarios fostering image distortions, texture shifts, or degraded visuals [18, 40].
Given that MDE models intrinsically depend on lucid visual cues for depth inference, the aforementioned corruptions naturally impose significant hurdles. Our preliminary analysis, visualized in Fig.1, showcases a spectrum of responses from distinct MDE model architectures when faced with diverse corruptions. Penetrating into these dynamics is quintessential to understanding the underlying causes of performance faltering, enabling us to architect MDE models that are both robust and reliable. Pursuing this vision, we undertake a meticulous benchmarking of extant MDE models on these new datasets, embarking on an exhaustive study of their robustness vis-a-vis the spectrum of corruptions. We probe queries about the resilience of MDE models to real-world corruptions, the influence of training input modalities, and learning paradigms - addressed in Sec.4.2. Concurrently, we assess the fidelity of our simulated corruptions and delve into the impact of texture-shift corruptions (style alterations) on gauging MDE model resilience.
From our benchmark findings, we distill several intriguing insights and proffer recommendations to amplify robust
ness - focusing on strategies like model pre-training, input resolution tuning, model sizing, complexity modulation, and corrupt-image fine-tuning (Sec. 4.3).
Figure 1: The depth estimation robustness (in terms of depth estimation error (DEE) defined in Sec. 3.3) under 18 corruptions in radar charts. Different MDE models exhibit diverse strengths and weaknesses against different corruptions that occur in the real world.
To encapsulate, our work offers the following seminal contributions:
$\diamond$  We introduce RoboDepth, the first systematically designed robustness evaluation suite for MDE under data corruptions, sensor failure, and style shifts. See our repository at this link for more details.
$\diamond$  We benchmark 42 state-of-the-art MDE models from indoor and outdoor scenes, on their robustness against corruptions, via three newly established datasets: KITTI-C, NYUDepth2-C, and KITTI-S. The corruption simulation toolkit has been open-sourced to facilitate future development.
Based on our observations, we draw in-depth discussion and analysis on the design considerations of building more robust MDE models for reliable, scalable, and practical applications.
$\diamond$  Furthermore, we initiated the RoboDepth Challenge [27], which garnered participation from over one hundred teams, underscoring the community's interest and the challenge's relevance. Comprehensive details about the competition can be accessed on our competition website at this link.
Figure 2: Corruption taxonomy. We break down common corruptions in depth estimation scenarios into three categories: i) Weather and lighting conditions, such as sunny, low-light, fog, frost, snow, and contrast conditions. ii) Sensor failure and movement, such as potential blurs (defocus, glass, motion, zoom) caused by motion. iii) Data processing issues, such as noises (Gaussian, impulse, ISO) happen due to hardware malfunctions. Examples shown are from the proposed KITTI-C benchmark.
Monocular Depth Estimation (MDE). Since the pioneering works [10, 12, 68, 16] first adopted deep neural networks to perform monocular depth estimation, significant progress has been made in many aspects. Notable innovations include network architectures [31, 43, 66, 63, 21], optimization functions [17, 64, 6], internal constraints [60, 67], multi-task learning [55, 22], geometry constraint [56, 51], and various sources of supervisions [44, 50, 33]. Based on the learning paradigm, most MDE methods can be split into supervised or self-supervised models. The former mainly focuses on indoor scenes and uses ground truth from RGB-D cameras or LiDAR sensors to train a regression model [1, 35]; while the latter formulates MDE as a novel view synthesis task to minimize the photometric loss between stereo pairs or from monocular video frames [68]. Although promising results have been achieved, the robustness of MDE models under adverse scenarios is still unknown. Due to the lack of relevant datasets, existing models are at risk of being vulnerable to corruptions. In this work, we fill in this gap by establishing comprehensive evaluation benchmarks and testing 42 MDE models from both indoor and outdoor environments to analyze their OoD robustness.
Robust MDE. To the best of our knowledge, only a few works targeted robust learning of MDE and they focused on different aspects. Ranftl et al. [44] proposed a unified objective for merging multiple datasets with different depth scales and ranges for training robust models. Similar works [58, 33, 53, 5, 62] resort to web stereo data or 3D movies to train MDE models and adapt them to unseen datasets. Kopf et al. [29] estimate stable camera trajectories for hand-held cellphone videos. SC-DepthV3 [50] generates pseudo-depth to refine depth details for scenes with dynamic objects. Li et al. [32] proposed an attention module to choose scene-specific
features for MDE on both indoor and outdoor scenes. SeasonDepth [19] contributed a dataset with depth maps under sunny, cloudy, and foliage weather. Most recently, there are works [7, 4] design adversarial patches to attack MDE models. Conversely, we aim to test the MDE robustness to corruptions that occur in real-world environments. We establish the first benchmark of this kind and incorporate an ample number of MDE models for in-depth analysis.
Corruption Robustness. ImageNet-C [18] is the pioneering work in this line of research which benchmarks classical image classification models to common corruptions and perturbations. Follow-up studies extend on the aspect to other visual perception tasks, e.g., object detection [40], segmentation [23, 26, 38], navigation [3], video classification [61], and pose estimation [54]. The essentiality of evaluating model robustness has been repeatedly validated. Since we are targeting a different task, i.e., MDE, most of the well-studied corruption types become less realistic or suitable for such a data
Severity Level
Figure 3: Corruption severity level. We create versatile corruption sets with different levels of severity. Examples shown are from the proposed NYUDepth2-C benchmark.
format. This motivates us to explore a new taxonomy for defining more proper corruption types for MDE. In this work, we contribute new datasets and benchmarks for probing the MDE robustness.
3 The RoboDepth Benchmark
In this section, we first introduce the taxonomy of corruptions included in our benchmarks (Sec. 3.1). We then elaborate on more details of the proposed datasets (Sec. 3.2) and corresponding robustness evaluation metrics (Sec. 3.3). Examples from our datasets are shown in Fig. 2, Fig. 3, and this page.
3.1 Corruption Definition
Weather & Lighting Condition. The cameras on drones or vehicles operating under different weather and times of day capture distribution-shifted images which are rare or lacking in current MDE datasets [14, 49]. To probe the robustness of MDE models under adverse weather and lighting conditions, we simulate six corruptions, i.e., 'brightness', 'dark', 'fog', 'frost', 'snow', and 'contrast', which commonly occur in the real-world environment. Compared to clean images, these corruptions tend to affect the intensity and color of the light source, leading to hazy, blurry, and noise-contaminated images, which increase the difficulties for the MDE model to make accurate depth predictions.
Sensor Failure & Movement. An MDE system must behave robustly against motion perturbation and sensor failure to maintain safety requirements for practical applications. To achieve this pursuit, we mimic four motion-related corruptions, i.e., 'defocus', 'glass', 'motion', and 'zoom' blurs; we also generate images under 'elastic transformation' and 'color quantization', which happen during sensor malfunction. These corruption types are often associated with issues including edge distortions, contrast loss, and pattern shifts.
Data & Processing Issue. Data collection and transmission are inevitably associated with various sources of noise and potential loss of information. We include four such random variations, i.e., 'Gaussian', 'impulse', 'shot', and 'ISO' noises. In addition, we investigate the degradation caused by 'pixelate' and 'JPEG compression' which are common corruptions in handling image data. Compared to clean images, the noise-contaminated data introduce errors in the intensity values of pixels, leading to a grainy or
(a) KITTI-C
(b) NYUDepth2-C
Figure 4: Benchmarking results of 42 MDE models on KITTI-C and NYUDepth2-C. Figures from top to bottom: the depth estimation error (DEE) vs. [1st row] mean corruption error (mCE), [2nd row] mean resilience rate (mRR), and [3rd row] sensitivity analysis among different corruption types.
speckled appearance. The pixelation and lossy compression tend to lead to a loss of detail and clarity in the image and can result in visible artifacts, such as blockiness or blurring.
3.2 Benchmark Establishment
KITTI-C. Based on the KITTI Vision Suite [14], we establish a robustness benchmark for outdoor MDE. We simulate the defined 18 corruptions using data from the KITTI val set under Eigen's split. Similar to [18], we design five severity levels for each corruption to further consolidate the evaluation
of robustness changes. As a result, this robustness probing dataset has a total number of 62, 730 RGB images with a resolution of  $192 \times 640$ . We also include the high-resolution version ( $320 \times 1024$ ) for evaluating the robustness of MDE models which take larger images as the input.
NYUDepth2-C. We construct a benchmark for robust indoor MDE based on NYU Depth V2 [49]. 15 of the defined corruptions are used, excluding 'fog', 'frost', and 'snow' which rarely occur in the indoor scenes. Since the indoor environments are less variant than outdoor ones, we only include four severity levels for each corruption. To sum up, this dataset contains 39, 240 images of size  $480 \times 640$ , which cover 23 different types of indoor scenes, such as basement, bathroom, bedroom, study, etc.
KITTI-S. Style changes, consisting mostly of texture shifts, have proven helpful for analyzing model robustness [18]. To further investigate the root cause of MDE robustness degradation, we form another collection based on KITTI [14] with stylized images via the style transfer model AdaIn [20]. This dataset has 8,364 images from 12 styles, including 'cartoon', 'digital art', 'ink painting', 'kids' drawing', 'murals', 'oil painting', 'penciling', 'shadow play', 'sketch', 'stained glass', 'relief', and 'water color'. Due to space limitations, please refer to Appendix or this page for additional examples.
Simulation Toolkit. To facilitate a similar study on other MDE datasets, we have open-sourced the code at this link for simulating corruptions and style shifts given arbitrary "clean" images.
Depth Estimation Error (DEE). We combine Abs Rel (error rate) and  $\delta_{1}$  (accuracy), the two main measures defined in [10, 36], into a unified metric as DEE =  $\frac{\text{Abs Rel} - \delta_1 + 1}{2}$ , which is constantly used as the indicator of depth estimation error in our benchmark. See Appendix for more formal definitions.
Corruption Error (CE). We follow [18] and use the mean CE (mCE) as the primary metric in comparing models' robustness. To normalize the severity effects, we choose MonoDepth2 [17] and AdaBins [1] as the baseline models for the KITTI-C and NYUDepth2-C benchmarks, respectively. The CE across  $L$  levels of severity and mCE across  $N$  corruption types can be calculated as follows:
$$
\mathrm {C E} _ {i} = \frac {\sum_ {l = 1} ^ {L} \left(\mathrm {D E E} _ {i , l}\right)}{\sum_ {l = 1} ^ {L} \left(\mathrm {D E E} _ {i , l} ^ {\text {b a s e l i n e}}\right)}, \quad \mathrm {m C E} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathrm {C E} _ {i}. \tag {1}
$$
Resilience Rate (RR). We define mean RR (mRR) as the relative robustness indicator for measuring how much accuracy can an MDE model retain when evaluated under the corruption scenarios, i.e.,
$$
\mathrm {R R} _ {i} = \frac {\sum_ {l = 1} ^ {L} \left(1 - \mathrm {D E E} _ {i , l}\right)}{L \times \left(1 - \mathrm {D E E} _ {\text {c l e a n}}\right)}, \quad \mathrm {m R R} = \frac {1}{N} \sum_ {i = 1} ^ {N} \mathrm {R R} _ {i}, \tag {2}
$$
where  $\mathrm{DEE}_{\mathrm{clean}}$  denotes the task-specific accuracy (or error rate) score on the "clean" evaluation set.
4.1 Benchmark Configuration
Depth Estimation Models. We benchmark 42 depth estimation models and model variants, which cover most of the open-source MDE models so far. 32 of them are for outdoor MDE and the remaining 10 are for indoor MDE. More detailed descriptions of these models are attached in the Appendix.
Datasets. All benchmarked depth estimation models have been trained on the official training splits of the KITTI [14] (for outdoor MDE) or NYU-Depth V2 [49] (for indoor MDE) datasets, and are tested accordingly on the official val splits and also our proposed KITTI-C, NYUDepth2-C, and KITTI-S datasets. Additionally, we resort to the real-world ACDC [45], nuScenes [2], Cityscapes [8], and Foggy-Cityscapes [46] datasets for validating the fidelity of our simulated corruptions. Our datasets and model evaluation toolkit can be downloaded from this page.
Evaluation Protocols. To avoid any unfairness in the MDE robustness comparison, we unify the common configurations among different candidate models, such as backbones, data augmentations, and post-processing. We use public checkpoints whenever possible and reproduce the reported results based on official settings. More details on this aspect are included in the Appendix.
Figure 5: The depth estimation robustness comparisons among different input modalities. Mono: monocular images; Stereo: stereo pairs; and Mono+Stereo: both monocular and stereo images.
4.2 MDE Robustness Probing
In this section, based on our benchmark, we aim to understand the corruption robustness among different MDE models by answering the following representative questions.
Q-1: "Are existing MDE models robust under real-world corruptions?" A: No, to a certain extent. Our benchmark results in Fig. 4 reveal that state-of-the-art MDE models are at risk of being vulnerable to corruptions. Existing MDE models, either from indoor or outdoor scenes, show a flattened or even inverse relationship between the DEE scores and robustness metrics. Due to the lack of a suitable robustness evaluation suite, current MDE models are over-fitted on "clean" sets while ignoring the OoD scenarios that are likely to occur in the real world. Nevertheless, we observe varying behaviors of different MDE models under different corruption types (see Fig. 1), caused by different design choices on model architecture. The results from Tab. 1 and Tab. 2 further show that the Transformers-based MDE models exhibit better robustness compared to conventional CNNs. Diving deeper, we can observe from the per-severity error rates in Fig. 9 that the above conclusion holds true for most corruption types under different severity levels. The qualitative results shown in Fig. 10 also validate that models with long-range receptive fields, such as MonoViT [66] (74.95% mCE) and Lite-Mono
Table 2: Supervised MDE Robustness Benchmark consisting of 10 models on the NYUDepth2-C dataset. The mCE and mRR scores are given in percentage (%). Bold: Best in column. Underline: Second best in column. The best score of each metric across three categories is shaded in color.
Method	Overall Robustness	Weather & Lightning	Sensor & Movement	Data & Processing
mCE	mRR	mDEE	mCE	mRR	mDEE	mCE	mRR	mDEE	mCE	mRR	mDEE
AdaBins_EB5 [1]	100.00	85.83	0.238	100.00	92.42	0.179	100.00	87.97	0.219	100.00	80.41	0.286
BTS_R50 [31]	122.78	80.63	0.292	125.51	87.97	0.228	121.86	84.21	0.261	122.34	73.39	0.356
AdaBins_R50 [1]	134.69	81.62	0.313	140.99	88.64	0.254	131.88	85.63	0.279	134.37	74.11	0.376
DPT_ViT-B [43]	83.22	95.25	0.177	93.66	96.57	0.166	81.82	96.24	0.168	79.41	93.60	0.191
SimIPU_nopt [34]	200.17	92.52	0.419	241.79	92.20	0.421	201.63	94.96	0.404	177.90	90.23	0.433
SimIPU/ImageNet [34]	163.06	85.01	0.357	190.62	87.52	0.338	166.90	86.97	0.343	145.45	81.79	0.382
SimIPU_KITTI [34]	173.78	91.64	0.370	210.25	91.81	0.368	170.75	95.37	0.344	158.56	87.81	0.396
SimIPU_WaymoOpen [34]	159.46	85.73	0.351	190.30	87.41	0.338	158.90	88.77	0.328	144.59	81.86	0.380
DepthFormer_SwinT-1k [35]	106.34	87.25	0.237	122.01	89.10	0.220	107.22	88.80	0.223	97.64	84.78	0.258
DepthFormer_SwinT-22k [35]	63.47	94.19	0.139	70.11	95.84	0.124	69.32	93.25	0.148	54.29	94.29	0.138
Figure 6: Depth estimation robustness of MonoDepth2 [17] under different training configurations. [top]: Different pre-training techniques. [bottom]: Different resolutions of the input images.
[63] (90.75% mCE), can better maintain accurate depth predictions under corruptions coped with texture shift, edge distortion, and noise contamination.
Q-2: "Which MDE modality is the most robust against corruptions?" A: MDE models trained with purely monocular images rather than stereo pairs as input have better robustness, as shown in Fig. 5. Here we analyze the robustness of MonoDepth2 [17] variants trained with different input modalities and observe that using stereo or a mix of monocular and stereo images leads to robustness degradation. MDE models trained with stereo pairs may rely more on the scene structural consistency between left-right pairs, which could be destroyed when dealing with corrupted data. Such constraints, however, could be relaxed if the depth estimation model is trained only on monocular sequences.
Q-3: "Which learning paradigm is better in terms of corruption robustness?" A: We find different sensitivities between the supervised and self-supervised MDE paradigms. From the third row of Fig. 4 we can observe that the self-supervised MDE models are less sensitive to lighting changes (texture shift) and motion blurs (local distortion), compared to the supervised models. On the other hand, both models suffer from noise perturbation and behave robustly against lossy image compression.
Q-4: "Are simulated corruptions realistic enough to mimic real-world scenarios?" A: Yes, to a certain extent. We verify this statement by augmenting MDE models via fine-tuning with our corruptions. Fig. 7 (right) shows that the generated corruptions are helpful for closing the distribution gap between the clean training set and real-world datasets [2, 8, 46], especially for the weather contaminated data in the Foggy-Cityscapes dataset [46]. This supports that our datasets are of high fidelity and scalability. Additionally, we show in the Appendix that the simulated corruptions, especially for the types that are related to weather and lighting changes, are of good quality compared to some real-world corrupted data [2, 45, 8].
Q-5: "Are MDE models robust against style changes?" A: We resort to the KITTI-S dataset (see Tab. 3) for probing model characteristics and find that most MDE models perform worse on this stylized dataset than on KITTI-C. We conjecture that the loss of texture cues for objects (e.g. car and building) for KITTI-S data causes the main impediment. We also observe that MDE models with
Figure 7: MDE robustness comparisons based on [left]: Model size (# of parameters) and [middle]: Model complexity (FLOPs); and [right]: Corruption fidelity verification by fine-tuning MonoDepth2 [17] with our corruptions on KITTI and testing it on the official val splits of nuScenes [2] (including its validation set and the nighttime split of the validation set) and Foggy-Cityscapes [46].