SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: The Depth Estimation Error (DEE) of 18 models on KITTI-S. Bold: Best in column. Underline: 2nd best in column. Blue : Best in row. Red : 2nd best in row. Green : Worst in row.
Method	mDEE	Cart	Digit	Ink	Kids	Mural	Oil	Pencil	Shadow	Sketch	Glass	Relief	Water
MonoDepth2R18[17]	.365	.324	.434	.351	.259	.326	.328	.418	.388	.416	.566	.255	.317
MonoDepth2nopt[17]	.378	.279	.289	.589	.235	.412	.249	.290	.589	.288	.596	.259	.464
MaskOccR18[48]	.368	.358	.356	.260	.265	.358	.375	.333	.314	.576	.554	.288	.384
DNetR18[59]	.400	.444	.381	.293	.280	.389	.336	.422	.290	.608	.553	.290	.516
CADepth[60]	.406	.446	.380	.379	.250	.421	.413	.470	.317	.585	.543	.242	.401
HR-Depth[39]	.324	.318	.332	.238	.228	.299	.306	.337	.316	.388	.531	.265	.331
DIFFNet[67]	.310	.227	.351	.206	.206	.386	.244	.276	.289	.385	.502	.294	.360
ManyDepth[56]	.323	.251	.373	.212	.205	.344	.300	.360	.343	.331	.553	.310	.300
FSREDepth[22]	.293	.275	.277	.294	.221	.310	.220	.270	.301	.318	.417	.261	.352
MonoViT[66]	.238	.179	.229	.252	.196	.240	.203	.237	.208	.325	.356	.205	.221
DynaDepthR18[64]	.371	.349	.358	.291	.264	.288	.293	.338	.363	.492	.534	.298	.585
DynaDepthR50[64]	.447	.502	.408	.259	.287	.398	.437	.492	.372	.595	.565	.460	.589
RA-Depth[42]	.365	.304	.335	.354	.262	.340	.310	.342	.425	.372	.476	.379	.475
TriDepthR18[6]	.379	.304	.403	.256	.217	.440	.352	.480	.318	.517	.573	.249	.436
Lite-MonoTiny[63]	.280	.236	.310	.251	.221	.254	.283	.285	.232	.356	.446	.237	.252
Lite-MonoSmall[63]	.303	.210	.346	.218	.210	.224	.238	.374	.307	.445	.435	.295	.336
Lite-MonoBase[63]	.288	.234	.313	.262	.218	.231	.241	.291	.240	.459	.476	.229	.264
Lite-MonoLarge[63]	.323	.208	.435	.249	.306	.348	.258	.363	.210	.458	.507	.263	.266
global awareness, e.g. Lite-Mono [63] and MonoViT [66], suffer less degradation in this case. Due to the page limit, kindly refer to the Appendix for more detailed findings on this style-shifted dataset.
4.3 MDE Robustness Enhancement
Pertaining strategy tends to help improve MDE robustness. We observe that transferring knowledge from other tasks, such as classification and segmentation, brings both strengths and weaknesses to MDE robustness. Fig. 6 (top) highlights that MDE models pre-trained on object-centric datasets, e.g. ImageNet, are more robust against weather and lighting changes (except for 'snow') and data processing noises, which are mostly texture-shifted corruptions. Motion and sensor corruptions, however, contain more edge and object distortions and could be eased by models without ImageNet pre-training. More concrete results in Tab. 1 imply that the CNN-based MDE models like MonoDepth2 [17] could become more shape-biased when pre-trained on object-centric datasets. More evidence from the KITTI-S benchmark (see Tab. 3) further verifies this finding, where the stylized data (texture-biased) are causing severe degradation for models with ImageNet pre-training.
Training on high-resolution images yields more robust MDE models. From Fig. 6 (bottom) we observe that MDE models trained with higher resolution inputs will likely yield more robust feature learning (relative  $30\%$  better) on noise-contaminated corruptions (Gaussian, impulse, shot, ISO noises). Since these noise contamination types mainly affect the global pixel distribution instead of the local ones, the CNN-based MDE models trained with high-resolution images are likely to capture more fine-grained information to suppress the degradation caused by noises. For more details, kindly refer to the per-corruption scores of benchmarked models in the Appendix.
Larger model size might not lead to better MDE robustness. It is often intuitive that larger models are likely to learn more general representations and lead to better performance on unseen data. However, we observe from Fig. 7 (left) that MDE models with more trainable parameters are getting less robust, mainly because they are deliberately tuned towards clean distribution and thus
Clean	.120	.216	.137	.217	.414	.156	.191	.191	.175	.170	.121	.162	.285	.285	.253	.282	.133	.167
+Bright	.118	.223	.139	.218	.377	.164	.222	.218	.190	.172	.127	.167	.269	.285	.261	.271	.136	.160
+Dark	.144	.159	.193	.226	.356	.235	.243	.224	.210	.191	.135	.156	.274	.288	.249	.258	.144	.155
+Fog	.127	.230	.122	.211	.389	.133	.234	.206	.178	.167	.124	.170	.327	.350	.309	.348	.137	.192
+Snow	.130	.249	.145	.166	.134	.213	.198	.175	.166	.168	.127	.169	.318	.331	.288	.320	.129	.180
+Contr	.123	.278	.126	.228	.417	.123	.259	.224	.225	.174	.127	.181	.330	.335	.308	.343	.137	.181
+Defoc	.155	.282	.153	.233	.387	.173	.132	.158	.165	.158	.141	.207	.349	.369	.337	.363	.136	.180
+Motio	.135	.262	.147	.221	.313	.167	.151	.158	.129	.149	.126	.168	.321	.318	.294	.324	.131	.171
+Zoom	.178	.348	.179	.347	.366	.201	.181	.205	.153	.140	.160	.220	.407	.397	.390	.411	.163	.202
+Elasti	.126	.238	.145	.216	.394	.195	.218	.195	.203	.176	.124	.166	.347	.361	.326	.372	.131	.162
+Gauss	.131	.184	.177	.215	.395	.246	.207	.190	.183	.171	.129	.159	.166	.172	.145	.141	.132	.140
+Impul	.133	.197	.190	.221	.397	.254	.182	.170	.169	.168	.130	.160	.189	.147	.156	.165	.130	.141
+Shot	.129	.181	.164	.204	.381	.217	.192	.173	.170	.175	.128	.159	.175	.182	.145	.148	.131	.142
+ISO	.138	.181	.195	.226	.407	.269	.271	.180	.227	.198	.130	.157	.188	.199	.161	.151	.133	.138
+Pixel	.129	.232	.144	.230	.453	.173	.207	.193	.193	.173	.124	.177	.308	.316	.283	.308	.127	.159
+JPEG	.131	.201	.160	.205	.397	.227	.277	.217	.208	.187	.130	.165	.251	.255	.224	.238	.133	.134
+All	.125	.155	.133	.167	.153	.136	.148	.158	.144	.151	.127	.150	.147	.144	.135	.140	.129	.138
Bright	Dark	Fog	Frost	Snow	Contr	Defoc	Glass	Motto	Zoom	Elasti	Quan	Gauss	Impul	Shot	ISO	Pixel	JPEG
Figure 8: The Per-Corruption Error scores of MonoDepth2 [17] fine-tuned with typical corruption as data augmentations and tested on each of the corruption types. For each column (corruption) in this figure: The darker the color, the lower the Abs Rel score (better robustness), and vice versa.
Figure 9: The Per-Severity Error Rate of eight typical MDE models [17, 39, 22, 42, 60, 67, 66, 63] trained on the clean KITTI dataset [14] and tested on each of the corruption set in KITTI-C. For each subfigure (corruption) in this figure: The horizontal axis denotes severity levels from one to five. The vertical axis denotes the depth estimation error (Abs Rel). Best viewed in color.
losing generalizability. Our results suggest that a moderate-sized model with suitable corruption suppression modules is likely to yield the best possible trade-off between robustness and efficiency.
Model complexity shows a direct correlation with MDE robustness. Based on Fig. 7 (middle) we reveal that higher training complexity tends to lead to better robustness. This is mainly because existing MDE models with high FLOPs per parameter are either trained with high-resolution monocular inputs or contain the computationally intensive self-attention mechanism and yield more stable depth predictions (as analyzed in previous discussions) but cause heavier training overhead.
Training with corrupted images does not always enhance model robustness. The robust fine-tuning study in Fig. 8 shows that overfitting the evaluation distribution (e.g. train and test on the same
Figure 10: Qualitative results of representative MDE models [17, 67, 42, 63, 66] under defined corruptions in the KITTI-C benchmark. Best viewed in color and zoom-ed in for details.
corruption) may help improve MDE robustness for this specific type of corruption, as shown from the darker-colored cell in the diagonal. However, those corruptions with severe perturbations (weather and movement) will likely lead to sub-par performance on other tested corruptions and even hurt the overall robustness. This indicates that sophisticated considerations, e.g. case study, must be made before applying corruptions as data augmentations to the training stage.
Throughout this research, we introduced the RoboDepth benchmark, a dedicated tool tailored for probing the OoD robustness of MDE models under a range of corruptions. By crafting three distinct datasets (KITTI-C, NYUDepth2-C, and KITTI-S) coupled with two metrics (mCE and mRR), we offered a comprehensive landscape for gauging both indoor and outdoor MDE robustness. Our findings underscore the imperative nature of robustness evaluation in the realm of depth estimation. Moreover, by dissecting various influential factors – encompassing architecture, modality, pre-training approaches, resolution, and model capacity – we provided insights to fortify model resilience against corruptions. In essence, our endeavors serve as a compass directing the evolution of robust MDE methodologies. As we navigate forward, our ambition is to augment the breadth and granularity of our benchmark, aiming to encompass a wider spectrum of MDE models and corruption variants.
Potential Limitations. Despite our extensive evaluation of diverse MDE models across expansive corruption sets, and validating the authenticity of our simulated corruptions, certain avenues remain uncharted. Notably, our current framework does not accommodate scenarios wherein multiple corruptions manifest simultaneously. Additionally, evolving from rigidly defined five severity levels to a more nuanced, continuous scale might offer deeper insights into MDE robustness. These unexplored terrains present intriguing prospects for subsequent research endeavors.
This research is part of the programme DesCartes and is supported by the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.
In this appendix, we supplement the following materials to support the findings and conclusions drawn in the main body of this paper:
- Sec. A documents necessary information about the proposed datasets and benchmarks.
- Sec. B elaborates on more details in terms of benchmark definitions and implementations, as well as the details of data collection, organization, licensing, and access.
- Sec. C introduces The RoboDepth Challenge, an academic competition held based on the datasets and benchmarks constructed in this work.
- Sec. D provides the complete quantitative results of different monocular depth estimation models in the established KITTI-C, NYUDepth2-C, and KITTI-S benchmarks.
- Sec. E contains additional qualitative results of different monocular depth estimation models under out-of-distribution corruption scenarios.
- Sec. F acknowledges the public resources used during the course of this work.
A Datasheets
In this section, we follow Gebru et al. [13] to document necessary information about the proposed datasets and benchmarks.
1. "For what purpose was the dataset created?" A: The dataset was created to facilitate relevant research in the area of depth estimation robustness under out-of-distribution corruptions.
2. "Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)?" A: The dataset was created by Lingdong Kong (National University of Singapore), Shaoyuan Xie (Huazhong University of Science and Technology), Hanjiang Hu (Carnegie Mellon University), Lai Xing Ng (Institute for Infocomm Research, A*STAR), Benoit Cottereau (CNRS), and Wei Tsang Ooi (National University of Singapore).
3. "Who funded the creation of the dataset?" A: The creation of the dataset is funded by related affiliations of the authors in this work, as listed in the above item.
4. "Any other comments?" A: N/A.
Most of the questions in this section are intended to provide dataset consumers with the information they need to make informed decisions about using the dataset for their chosen tasks. Some of the questions are designed to elicit information about compliance with the EU's General Data Protection Regulation (GDPR) or comparable regulations in other jurisdictions. Questions that apply only to datasets that relate to people are grouped together at the end of the section. We recommend taking a broad interpretation of whether a dataset relates to people. For example, any dataset containing text that was written by people relates to people.
1. "What do the instances that comprise our datasets represent (e.g., documents, photos, people, countries)?" A: The instances that comprise the dataset are mainly images captured by camera sensors, providing visual representations of indoor and outdoor scenes observed.
2. "How many instances are there in total (of each type, if appropriate)?" A: The KITTI-C dataset contains a total number of 62,730 RGB images from 18 corruption types, with a resolution of  $192 \times 640$ . The NYUDepth2-C dataset contains a total number of 39,240
RGB images from 15 corruption types, with a resolution of  $480 \times 640$ . The KITTI-S dataset contains a total number of 8,364 RGB images from 12 style type, with a resolution of  $192 \times 640$ . Each image in these three datasets is associated with a single-channel depth map of the same size as the image.
3. "Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?" A: Yes, our datasets contain all possible instances that have been collected so far.
4. "Is there a label or target associated with each instance?" A: Yes, each instance in our datasets is associated with a single-channel depth map of the same size as the image.
5. "Is any information missing from individual instances?" A: No.
6. "Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?" A: Yes, the relationship between individual instances is explicit.
7. "Are there recommended data splits (e.g., training, development/validation, testing)?" A: Yes, we provide detailed data splits for our datasets.
8. "Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?" A: Yes, our datasets are self-contained.
9. "Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non-public communications)?" A: No, all data are clearly licensed.
10. "Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?" A: No.
11. "Any other comments?" A: N/A.
In addition to the goals outlined in the previous section, the questions in this section are designed to elicit information that may help researchers and practitioners create alternative datasets with similar characteristics. Again, questions that apply only to datasets that relate to people are grouped together at the end of the section.
1. "How was the data associated with each instance acquired?" A: Please refer to the details listed in Section B.
2. "What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)?" A: Please refer to the details listed in Section B.
3. "If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?" A: Please refer to the details listed in Section B.
A.4 Preprocessing, Cleaning, and Labeling
The questions in this section are intended to provide dataset consumers with the information they need to determine whether the "raw" data has been processed in ways that are compatible with their chosen tasks. For example, text that has been converted into a "bag-of-words" is not suitable for tasks involving word order.
1. "Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)?" A: Yes, we preprocessed and cleaned data in our datasets.
2. "Was the 'raw' data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)?" A: Yes, raw data is accessible.
3. "Is the software that was used to preprocess/clean/label the data available?" A: Yes, the necessary software used to preprocess and clean the data is publicly available.
4. "Any other comments?" A: N/A.
The questions in this section are intended to encourage dataset creators to reflect on tasks for which the dataset should and should not be used. By explicitly highlighting these tasks, dataset creators can help dataset consumers make informed decisions, thereby avoiding potential risks or harms.
1. "Has the dataset been used for any tasks already?" A: No.
2. "Is there a repository that links to any or all papers or systems that use the dataset?" A: Yes, we provide such links in our GitHub repository.
3. "What (other) tasks could the dataset be used for?" A: The dataset could be used for relevant perception, tracking, and planning tasks based on camera sensors.
4. "Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?" A: N/A.
5. "Are there tasks for which the dataset should not be used?" A: N/A.
6. "Any other comments?" A: N/A.
1. "Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created?" A: No.
2. "How will the dataset be distributed (e.g., tarball on website, API, GitHub)?" A: Very likely to be distributed by website, API, and GitHub repository.
3. "When will the dataset be distributed?" A: The datasets are publicly accessible.
4. "Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)?" A: Yes, the dataset is under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
5. "Have any third parties imposed IP-based or other restrictions on the data associated with the instances?" A: No.
6. "Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?" A: No.
7. "Any other comments?" A: N/A.
1. "Who will be supporting/hosting/maintaining the dataset?" A: The authors of this work serve for supporting, hosting, and maintaining the datasets.
2. "How can the owner/curator/manager of the dataset be contacted (e.g., email address)?"
A: The curators can be contacted via emails as follows:
- Lingdong Kong (lingdong@comp.nus.edu.sg);
- Shaoyuan Xie (shaoyuanxie@hust.edu.cn);
- Hanjiang Hu (hanjianghu@cmu.edu);
- Lai Xing Ng (ng_lai_xing@i2r.a-star.edu.sg);
- Benoit Cottereau (benoit.cottereau@cnrs.fr);
- Wei Tsang Ooi (ooiwt@comp.nus.edu.sg).
3. "Is there an erratum?" A: There is no explicit erratum; updates and known errors will be specified in future versions.
Figure A: The 18 corruption types from three main categories defined in the RoboDepth benchmark. Examples shown are from the proposed KITTI-C benchmark.
4. "Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)?" A: No, for the current version. Future updates (if any) will be posted on the dataset website.
5. "Will older versions of the dataset continue to be supported/hosted/maintained?" A: Yes. This is the first version of the release; future updates will be posted and older versions will be replaced.
6. "If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so?" A: Yes, we provide detailed instructions for future extensions.
7. "Any other comments?" A: N/A.
B Datasets and Benchmarks
In this section, we first provide the detailed definitions of different corruption types in the RoboDepth benchmark (Sec. B.1), we then elaborate on the data collection process (Sec. B.2), the license of our datasets (Sec. B.3), and the procedures for downloading these datasets (Sec. B.4). Lastly, we attach the summary of technical contributions and implementation details for the benchmarked monocular depth estimation methods (Sec. B.5). An overview of the 18 corruption types is shown in Fig. A. The histogram of pixel values under each corruption is shown in Fig. B.
B.1 Corruption Definition
- Brightness refers to the level of lightness or darkness in an image. It is affected by the intensity and color of the light source, as well as the reflectivity of the objects in the scene. Images captured under different lighting conditions may have varying levels of brightness, which can affect the visibility and depth estimation quality.
- Dark images can result from low-lighting conditions or underexposure. These images can have low contrast and low visibility, making it difficult to distinguish the objects and backgrounds in the scene. Dark images can also suffer from increased electronic noise and color distortion.
- Fog is a type of atmospheric scattering that can reduce the contrast and visibility of objects and backgrounds in a scene. It occurs when water droplets or ice crystals in the air scatter and absorb light, leading to a hazy or blurry appearance.
- Frost forms when lenses or windows are coated with ice crystals, leading to image distortion and inaccuracies in depth estimation. Frost can be modeled as a convolution of the image with a non-uniform kernel that depends on the shape and size of the ice crystals. It can affect the accuracy of depth estimation by reducing the contrast of the image and distorting the edges of objects.
Figure B: The histogram of pixel values under each of the 18 corruption types in KITTI-C. Note that the horizontal axis of each subfigure is scaled, shifted, and centered to fit the window. The numerical range is at the scale of  $1e6$ . Zoomed-in for more details.
- Snow is a visually obstructive form of precipitation that can obscure objects and backgrounds in the image and make it difficult to estimate accurate depth. It can be modeled as a random distribution of white pixels with a high probability of occurrence. Snow can affect the accuracy of depth estimation by introducing errors in the intensity values of pixels and obscuring the edges of objects.
- Contrast is the difference in luminance or color between different parts of an image. High-contrast images have large differences between light and dark areas, while low-contrast images have more similar levels of brightness. Contrast can be affected by lighting conditions and the color and texture of the objects and backgrounds in the scene.
Figure C: Illustrative examples of each corruption type across the five severity levels (level 1 is the lightest and level 5 is the hardest) in the KITTI-C benchmark. Best viewed in color. Zoomed-in for more details.
- Defocus blur occurs when an image is out of focus, causing objects and backgrounds in the image to appear blurred. It can be modeled as a convolution of the image with a Gaussian kernel, where the blur size depends on the distance between the camera and the object. Defocus blur can affect the accuracy of depth estimation by reducing the contrast of the image and distorting the edges of objects.
- Glass blur is a type of image distortion that appears with glass windows or panels. It can lead to inaccuracies in depth estimation due to the opaque and irregular nature of the glass. Glass blur can be modeled as a convolution of the image with a non-uniform kernel that depends on the shape and thickness of the glass.
- Motion blur appears when a camera is moving quickly, causing objects and backgrounds in the image to appear blurred. It can be modeled as a convolution of the image with a motion kernel that depends on the direction and speed of the camera movement.
- Zoom blur occurs when a camera moves toward an object rapidly, causing the image to appear blurred. It can be modeled as a convolution of the image with a non-uniform kernel that depends on the zoom factor and the distance between the camera and the object. Zoom
blur can affect the accuracy of depth estimation by distorting the edges of objects and reducing the contrast of the image.