SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- Elastic transformations are spatial transformations that deform small regions of an image while preserving the overall structure. They are commonly used in data augmentation techniques for deep learning models to increase the robustness of the model to deformations and variations in the input images.
- Color quantization is the process of reducing the number of colors in an image while preserving its overall visual appearance. This technique is commonly used to reduce the storage and processing requirements for digital images. However, this process can result in a loss of detail and color accuracy in the image, particularly in areas with complex color gradients or patterns.
- Gaussian noise is a type of noise that appears in low-light conditions and can cause random fluctuations in image intensity. It is modeled as additive white Gaussian noise and has a normal distribution with zero mean and a standard deviation that represents the noise level. Gaussian noise can affect the accuracy of depth estimation by reducing the contrast of the image and introducing errors in intensity values.
- Impulse noise is a type of noise that can be caused by bit errors and appears as isolated pixels with incorrect intensity values. It can be modeled as a random distribution of black and white pixels with a low probability of occurrence. It can affect the accuracy of depth estimation by introducing errors in the intensity values of pixels and distorting the image.
- Shot noise, also called Poisson noise, is electronic noise caused by the discrete nature of light itself. It occurs when photons hit a sensor and is modeled as a Poisson distribution. Shot noise can lead to irregularities in image intensity and affect the accuracy of depth estimation, especially in low-light conditions.
- ISO noise is a type of noise that can appear in digital images captured with high ISO settings. ISO refers to the sensitivity of the camera's sensor to light, with higher ISO values resulting in brighter images. However, increasing the ISO setting can also introduce additional electronic noise into the image, leading to a grainy or speckled appearance. This type of noise can be particularly challenging to remove without losing important image details.
- Pixelation occurs when an image is displayed or printed at a low resolution, resulting in individual pixels becoming visible. This can lead to a loss of detail and clarity in the image, bringing extra challenges for depth estimation models.
- JPEG is a lossy image compression format commonly used for storing and sharing digital images. JPEG compression reduces the file size of an image by removing some of the image data that is deemed less important or less noticeable to the human eye. Such compression can result in visible artifacts, such as blockiness or blurring, in the compressed image.
B.2 Corruption Simulation
The corruption simulation tools are from two resources. We use the imagecorruptions tool<sup>1</sup> from Michaelis et al. [40] to simulate 'brightness', 'fog', 'frost', 'snow', 'contrast', 'defocus blur', 'glass blur', 'motion blur', 'zoom blur', 'elastic transform', 'Gaussian noise', 'impulse noise', 'shot noise', 'pixelate', and JPEG compression'. Additionally, We use the 3DCC tool<sup>2</sup> by Kar et al. [24] to simulate 'dark', 'color quantization', and ISO noise'.
We follow Hendrycks and Dietterich [18] to split each corruption simulation into five severity levels for KITTI-C and four severity levels for NYUDepth2-C. The split strategy is the same as the ImageNet-C paper [18]. Illustrative examples of different severity levels in KITTI-C are shown in Fig. C. Illustrative examples of different severity levels in NYUDepth2-C are shown in Fig. D, Fig. E, and Fig. F.
We provide a comprehensive corruption simulation toolkit that can be used to generate the defined 18 corruption types on any image dataset. This toolkit is publicly accessible at: https://github.com/ldkong1205/RoboDepth/blob/main/docs/CREATE.md. Refer to this page for more details and the code implementations of corruption simulations.
1More details at: https://github.com/bethgelab/imagecorruptions.
2More details at: https://github.com/EPFL-VILAB/3DCommonCorruptions.
Figure D: Illustrative examples of each of the five corruption types ('brightness', 'dark', 'contrast', 'defocus blur', and 'glass blur') across four severity levels (level 1 is the lightest and level 4 is the hardest) in the NYUDepth2-C benchmark. Best viewed in color. Zoomed-in for more details.
To validate the simulated corruptions are of high fidelity, especially for the types that are related to weather and lighting changes, we conduct a study on the pixel distributions. Assuming that a corruption simulation is realistic enough to reflect real-world situations, the distribution of a corrupted "clean" set should be similar to that of the real-world corruption set. We validate this using ACDC [45], nuScenes [2], Cityscapes [8], and Foggy-Cityscapes [46], since these datasets contain: i) real-world corruption data; and ii) clean data collected by the same sensor types from the same physical locations.
We simulate corruptions using "clean" images and compare the distribution patterns with their corresponding real-world corrupted data. We do this to ensure that there is no extra distribution shift from aspects like sensor difference (e.g. FOVs and resolutions) and location discrepancy (e.g. environmental and semantic changes). The results are shown in this paper: https://github.com/ldkong1205/RoboDepth/blob/main/docs/VALIDITY.md.
What is more, we follow Geirhos et al. [15] and use AdaIn [20] to generate stylized images in KITTI-S. AdaIn is a classical style transfer model that takes a pair of two images (a reference image and a style image) as input and outputs a stylized image. The checkpoint of this style transfer model can be downloaded at: https://github.com/naoto0804/pytorch-AdaIN. The detailed procedures for generating stylized images are attached at: https://github.com/rgeirhos/Stylized-ImageNet. Illustrative examples of different styles in KITTI-S are shown in Fig. G.
Our datasets and benchmark toolkit are released under the Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license<sup>3</sup>, under the following terms:
3More details at: https://creativecommons.org/licenses/by-nc-sa/4.0.
Figure E: Illustrative examples of each of the five corruption types ('motion blur', 'zoom blur', 'elastic transform', 'color quantization', and 'Gaussian noise') across four severity levels (level 1 is the lightest and level 4 is the hardest) in the NYUDepth2-C benchmark. Best viewed in color. Zoomed-in for more details.
- Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- NonCommercial — You may not use the material for commercial purposes.
- ShareAlike — If you remix, transform, or build upon the material, you must distribute your contributions under the same license as the original.
B.4 Download
Our datasets and benchmark toolkit are publicly accessible. We provide detailed procedures for accessing and downloading them. Please refer to the following pages for more details:
- GitHub Repo: https://github.com/1dkong1205/RoboDepth.
- Installation & Environment: https://github.com/ldkong1205/RoboDepth/blob/main/docs/INSTALL.md.
- Data Preparation: https://github.com/ldkong1205/RoboDepth/blob/main/docs/DATA_PREPARE.md.
- Benchmark Details: https://github.com/1dkong1205/RoboDepth#benchmark.
- Corruption Simulation Tool: https://github.com/ldkong1205/RoboDepth/blob/main/docs/CREATE.md.
Figure F: Illustrative examples of each of the five corruption types ('impulse noise', 'shot noise', 'ISO noise', 'pixelate', and 'JPEG compression') across four severity levels (level 1 is the lightest and level 4 is the hardest) in the NYUDepth2-C benchmark. Best viewed in color. Zoomed-in for more details.
B.5 Benchmarked Methods
- MonoDepth2 [17]: A seminar work in the field of MDE which proposed a series of improvements on top of prior works, including a per-pixel minimum re-projection loss, an auto-masking strategy, and a multi-scale estimation framework. We evaluate three versions of this model, under the Mono, Stereo, and Mono+Stereo settings, respectively. We also test this model with different backbones (ResNet-18 and ResNet-50) and pretraining strategies. The code is accessible at: https://github.com/nianticlabs/monodepth2.
- DepthHints [57]: A stereo model that makes use of depth hints, which are small but informative cues about the scenes and can be extracted from the image itself. These hints are then used to guide the network during training, allowing the model to learn a better depth estimation representation without requiring any explicit ground truth depth information. The code is accessible at: https://github.com/nianticlabs/depth-hints.
- MaskOcc [48]: A monocular model takes into account occlusion in depth estimation. The occlusions occur when objects in the scene block each other from view. The proposed method masks out occluded regions in the input image during training, allowing the model to focus on learning from only the visible parts of the scene. This approach results in more accurate depth estimation, especially in regions with occlusion. The code is accessible at: https://github.com/schelv/monodepth2.
- DNet [59]: A monocular model consists of a coarse-to-fine depth estimation pipeline, where a deep neural network is trained to predict absolute depth maps from a single input image. This model is built upon the MonoDepth2 baseline. The code is accessible at: https://github.com/TJ-IPLab/DNet.
Figure G: Illustrative examples of 12 different styles in the KITTI-S benchmark. Best viewed in color. Zoomed-in for more details.
- CADepth [60]: A new network architecture that employs attention mechanisms to selectively attend to certain image regions, which helps to improve the accuracy of the depth estimation. The code is accessible at: https://github.com/kamiLight/CADepth-master.
- HR-Depth [39]: A new depth estimation model that can generate high-resolution depth maps from a single input image. The main contribution of the work is the introduction of a hierarchical residual pyramid network architecture that leverages multi-scale features to produce accurate depth estimates. The code is accessible at: https://github.com/shawLyu/HR-Depth.
- DIFFNet [67]: A monocular model based on a deep neural network architecture that uses an internal feature fusion mechanism to combine information from different layers of the network. Such a mechanism used in the network helps to capture global and local features of the input images, which improves the accuracy of the depth estimates. The code is accessible at: https://github.com/brandleyzhou/DIFFNet.
- ManyDepth [56]: A multi-monocular model that leverages temporal consistency between adjacent frames in a video sequence to estimate accurate depth maps. The proposed framework consists of a network that takes as input multiple frames of a video sequence and outputs a corresponding depth map for each frame. The code is accessible at: https://github.com/nianticlabs/manydepth.
- FSRE-Depth [22]: A monocular model consists of a two-stage network architecture that enhances the representation of the input image by incorporating fine-grained semantic information. The first stage of the network uses a semantic segmentation module to extract semantic information from the input image. The second stage of the network utilizes a depth refinement module to refine the initial depth prediction and generate the final depth estimation. The code is accessible at: https://github.com/hyBlue/FSRE-Depth.
- MonoViT [66]: A monocular model that consists of a novel training scheme that leverages the spatial and semantic representations learned by the Vision Transformers to predict depth
from a single RGB image. The code is accessible at: https://github.com/zxcqlf/MonoViT.
B.6 Depth Estimation Metrics
In our benchmark, we adopt the conventional reporting of Abs Rel (error rate) and  $\delta_{1}$  (accuracy) for measuring the depth estimation performance.
Figure H: We successfully hosted The RoboDepth Challenge [27] at ICRA 2023, which is an academic competition established based on the datasets and benchmarking toolkit proposed in this work. More details of this competition can be found at: https://robodepth.github.io.
(a) Track 1
(b) Track 2
Figure I: The submission and scoring statistics for the two tracks in the RoboDepth competition.
Abs Rel measures the absolute relative difference between the ground-truth (gt) and the prediction (pred), as calculated via the following equation:
$$
\operatorname {A b s} \operatorname {R e l} = \frac {1}{| D |} \sum_ {\text {p r e d} \in D} \frac {| g t - \text {p r e d} |}{g t}. \tag {3}
$$
The  $\delta$  metric is the depth estimation accuracy given the threshold:
$$
\delta_ {t} = \frac {1}{| D |} \left| \left\{p r e d \in D \mid \max \left(\frac {g t}{p r e d}, \frac {p r e d}{g t}\right) <   1.25 ^ {t} \right\} \right| \times 100 \% , \tag{4}
$$
where  $\delta_1 = \delta < 1.25$ ,  $\delta_2 = \delta < 1.25^2$ ,  $\delta_3 = \delta < 1.25^3$  are the three conventionally used accuracy scores in prior works.
We combine Abs Rel and  $\delta_{1}$ , the two main measures, into a unified metric as  $\mathrm{DEE} = \frac{\mathrm{AbsRel} - \delta_1 + 1}{2}$  which is constantly used as the indicator of depth estimation error in our benchmark.
C The RoboDepth Challenge
In this section, we introduce The RoboDepth Challenge – an academic competition that is established based on the datasets and benchmarks proposed in this work.
C.1 Competition Overview
The RoboDepth Challenge [27] aims to facilitate relevant research in the area of robust monocular depth estimation. It is hosted by the 40th IEEE Conference on Robotics and Automation (ICRA 2023). More materials on this competition are provided in the following links:
Table A: The top-ten performing teams on the leaderboard of Track 1 in the RoboDepth competition.
Team Name	Abs Rel ↓	Sq Rel ↓	RMSE ↓	log RMSE ↓	δ < 1.25 ↑	δ < 1.25² ↑	δ < 1.25³ ↑
OpenSpaceAI	0.121	0.919	4.981	0.200	0.861	0.953	0.980
USTC-IAT-United	0.123	0.932	4.873	0.202	0.861	0.954	0.979
YYQ	0.123	0.885	4.983	0.201	0.848	0.950	0.979
zs_dlut	0.124	0.899	4.938	0.203	0.852	0.950	0.979
UMCV	0.124	0.845	4.883	0.202	0.847	0.950	0.980
THU_ZS	0.124	0.892	4.928	0.203	0.851	0.951	0.980
THU_Chen	0.125	0.865	4.924	0.203	0.846	0.950	0.980
seesee	0.126	0.990	4.979	0.206	0.857	0.952	0.978
namename	0.126	0.994	4.950	0.204	0.860	0.953	0.979
USTCxNetEaseFuxi	0.129	0.973	5.100	0.208	0.846	0.948	0.978
MonoDepth2 [17]	0.221	1.988	7.117	0.312	0.654	0.859	0.938
- Competition Page: https://robodepth.github.io.
- Competition Report: https://arxiv.org/abs/2307.15061.
- Toolkit: https://github.com/ldkong1205/RoboDepth/tree/main/competition.
- Workshop Recordings: https://www.youtube.com/watch?v=mYhdTGGiIGCY&list=PLxxrIfcH-qBGZ6x_e1AT2_YnAxiHIKtkB.
- Server for Track 1: https://codalab.lisn.upsaclay.fr/competitions/9418.
- Server for Track 2: https://codalab.lisn.upsaclay.fr/competitions/9821.
C.2 Statistics
The RoboDepth Challenge started on January 1, 2023, and ended on May 25, 2023. There are two tracks in this competition: a self-supervised learning track for robust outdoor depth estimation and a supervised learning track for robust indoor depth estimation.
This competition attracted a lot of participants from around the world. Specifically, 226 teams registered at our evaluation servers. Among them, 66 teams made a total number of 1137 valid submissions. The detailed statistics in terms of submission and scoring are shown in Fig. I. The top-three performing teams of Track 1 achieved the Abs Rel scores of 0.121, 0.123, and 0.123, respectively. The top-three performing teams of Track 2 achieved the a1 scores of 0.940, 0.928, and 0.898. More results are attached to our evaluation servers and competition homepage.
For more details, please refer to our competition page at: https://github.com/ldkong1205/RoboDepth/tree/main/competition.
C.3 Workshop
We hosted The RoboDepth Workshop at ICRA on June 02, 2023. The video recordings of this workshop are publicly available at: https://www.youtube.com/watch?v=mYhdTGiIGCY&list=PLxxrIfcH-qBGZ6x_e1AT2_YnAxiHIKtkB.
The slides of this workshop can be downloaded from https://ldkong.com/talks/icra23_robodepth.pdf.
C.4 Leaderboards
The results of the top-ten teams from each of the two tracks are shown in Table A and Table B, respectively. For more details, please refer to our competition page at: https://github.com/ldkong1205/RoboDepth/tree/main/competition.
D Additional Quantitative Result
In this section, we provide the complete results of different monocular depth estimation models in the established KITTI-C, NYUDepth2-C, and KITTI-S benchmarks.
Table B: The top-ten performing teams on the leaderboard of Track 2 in the RoboDepth competition.
Team Name	Abs Rel ↓	Sq Rel ↓	RMSE ↓	log RMSE ↓	δ < 1.25 ↑	δ < 1.25² ↑	δ < 1.25³ ↑
USTCxNetEaseFuxi	0.088	0.046	0.347	0.115	0.940	0.985	0.996
OpenSpaceAI	0.095	0.045	0.341	0.117	0.928	0.990	0.998
GANCV	0.104	0.060	0.391	0.131	0.898	0.982	0.995
shinonomei	0.123	0.080	0.450	0.153	0.861	0.975	0.993
YYQ	0.125	0.085	0.470	0.159	0.851	0.970	0.989
Hyq	0.124	0.089	0.474	0.158	0.851	0.967	0.990
DepthSquad	0.137	0.085	0.462	0.158	0.845	0.976	0.996
kinda	0.146	0.095	0.480	0.165	0.831	0.973	0.993
dx3	0.131	0.095	0.507	0.170	0.825	0.963	0.989
uuht	0.150	0.100	0.492	0.168	0.822	0.973	0.993
DepthFormer [35]	0.190	0.179	0.717	0.248	0.655	0.898	0.970
D.1 KITTI-C
The clean performances of 32 MDE models under the standard KITTI Eigen split are shown in Table C.
The per-corruption DEE scores, CE scores, and RR scores of 32 MDE models in the KITTI-C benchmark are shown in Table D, Table E, and Table F, respectively.
D.2 NYUDepth2-C
The per-corruption DEE scores, CE scores, and RR scores of 10 MDE models in the NYUDepth2-C benchmark are shown in Table G, Table H, and Table I, respectively.
D.3 KITTI-S
The per-corruption DEE scores of 32 MDE models in the KITTI-S benchmark are shown in Table J.
E Additional Qualitative Result
In this section, we provide additional qualitative results of different monocular depth estimation models under out-of-distribution corruption scenarios.
Specifically, the qualitative results of MonoDepth2 [17], Lite-Mono [63], and MonoViT [66] under each of the 18 corruption types across five severity levels in our benchmark are shown in Fig. J, Fig. K, and Fig. L, respectively.
The qualitative results of MonoDepth2 [17], DIFFNet [67], RA-Depth [42], Lite-Mono [63], and MonoViT [66], under each of the 18 corruption types in our benchmark are shown in Fig. M and Fig. N.
The qualitative results of MonoDepth2 [17], DIFFNet [67], RA-Depth [42], Lite-Mono [63], and MonoViT [66], under each of the 12 styles in our benchmark are shown in Fig. O.
Table G: The Depth Estimation Error (DEE) of 10 monocular depth estimation models in the NYUDepth2-C benchmark. Bold: Best in col. Underline: Second best in col. Blue : Best in row. Red : Worst in row.
Method	mDEE	Bright	Dark	Contr	Defoc	Glass	Motio	Zoom	Elast	Quant	Gaus	Impul	Shot	ISO	Pixel	JPEG
AdaBins EB5 [1]	.122	.149	.269	.265	.337	.262	.231	.372	.182	.180	.442	.512	.392	.474	.139	.175
BTS R50 [31]	.122	.149	.269	.265	.337	.262	.231	.372	.182	.180	.442	.512	.392	.474	.139	.175
AdaBins R50 [1]	.158	.179	.293	.289	.339	.280	.245	.390	.204	.216	.458	.519	.401	.481	.186	.211
DPT ViT-B [43]	.136	.136	.182	.180	.154	.166	.155	.232	.139	.165	.200	.213	.191	.199	.171	.174
SimIPU nopt [34]	.372	.388	.427	.448	.416	.401	.400	.433	.381	.391	.465	.471	.450	.461	.375	.378
SimIPU ImageNet [34]	.244	.269	.370	.376	.377	.337	.324	.422	.306	.289	.445	.463	.414	.449	.247	.272
SimIPU KITTI [34]	.312	.326	.373	.406	.360	.333	.335	.386	.316	.333	.432	.442	.422	.443	.314	.322
SimIPU WaymoOpen [34]	.243	.269	.348	.398	.380	.327	.313	.405	.256	.287	.439	.461	.416	.455	.246	.265
DepthFormer SwinT-1k [35]	.125	.147	.279	.235	.220	.260	.191	.300	.175	.192	.294	.321	.289	.305	.161	.179
DepthFormer SwinT-22k [35]	.086	.099	.150	.123	.127	.172	.119	.237	.112	.119	.159	.156	.148	.157	.101	.108
Table H: The Corruption Error (CE) of 10 monocular depth estimation models in the NYUDepth2-C benchmark. All scores are given in percentage (%). Bold: Best in col. Underline: Second best in col. Blue : Best in row. Red : Worst in row.
Method	mCE	Bright	Dark	Contr	Defoc	Glass	Motio	Zoom	Elast	Quant	Gaus	Impul	Shot	ISO	Pixel	JPEG
AdaBins EB5 [1]	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0	100.0
BTS R50 [31]	122.8	112.9	138.7	125.0	143.4	127.2	125.5	96.9	119.0	119.2	113.3	136.9	133.3	124.7	112.1	113.6
AdaBins R50 [1]	134.7	135.6	151.0	136.3	144.3	135.9	133.2	101.6	133.3	143.1	117.4	138.8	136.4	126.6	150.0	137.0
DPT ViT-B [43]	83.2	102.3	93.8	84.9	65.5	80.6	84.2	60.4	90.9	109.3	51.3	57.0	65.0	52.4	137.9	113.0
SimIPU nopt [34]	200.2	293.9	220.1	211.3	177.0	194.7	217.4	112.8	249.0	258.9	119.2	125.9	153.1	121.3	302.4	245.5
SimIPU ImageNet [34]	163.1	203.8	190.7	177.4	160.4	163.6	176.1	109.9	200.0	191.4	114.1	123.8	140.8	118.2	199.2	176.6
SimIPU KITTI [34]	173.8	247.0	192.3	191.5	153.2	161.7	182.1	100.5	206.5	220.5	110.8	118.2	143.5	116.6	253.2	209.1
SimIPU WaymoOpen [34]	159.5	203.8	179.4	187.7	161.7	158.7	170.1	105.5	167.3	190.1	112.6	123.3	141.5	119.7	198.4	172.1
DepthFormer SwinT-1k [35]	106.3	111.4	143.8	110.9	93.6	126.2	103.8	78.1	114.4	127.2	75.4	85.8	98.3	80.3	129.8	116.2
DepthFormer SwinT-22k [35]	63.5	75.0	77.3	58.0	54.0	83.5	64.7	61.7	73.2	78.8	40.8	41.7	50.3	41.3	81.5	70.1
Table I: The Resilience Rate (RR) of 10 monocular depth estimation models in the NYUDepth2-C benchmark. All scores are given in percentage (%). Bold: Best in col. Underline: Second best in col. Blue : Best in row. Red : Worst in row.
Method	mRR	Bright	Dark	Contr	Defoc	Glass	Motio	Zoom	Elast	Quant	Gaus	Impul	Shot	ISO	Pixel	JPEG
AdaBins EB5 [1]	85.8	97.8	90.8	88.7	86.2	89.4	91.9	69.4	95.4	95.6	68.7	70.5	79.5	69.8	98.7	95.3
BTS R50 [31]	80.6	96.9	83.3	83.7	75.5	84.1	87.6	71.5	93.2	93.4	63.6	55.6	69.3	59.9	98.1	94.0
AdaBins R50 [1]	81.6	97.5	84.0	84.4	78.5	85.5	89.7	72.5	94.5	93.1	64.4	57.1	71.1	61.6	96.7	93.7
DPT ViT-B [43]	95.3	100.0	94.7	94.9	97.9	96.5	97.8	88.9	99.7	96.6	92.6	91.1	93.6	92.7	96.0	95.6
SimIPU nopt [34]	92.5	97.5	91.2	87.9	93.0	95.4	95.5	90.3	98.6	97.0	85.2	84.2	87.6	85.8	99.5	99.0
SimIPU ImageNet [34]	85.0	96.7	83.3	82.5	82.4	87.7	89.4	76.5	91.8	94.1	73.4	71.0	77.5	72.9	99.6	96.3
SimIPU KITTI [34]	91.6	98.0	91.1	86.3	93.0	97.0	96.7	89.2	99.4	97.0	82.6	81.1	84.0	81.0	99.7	98.6
SimIPU WaymoOpen [34]	85.7	96.6	86.1	79.5	81.9	88.9	90.8	78.6	98.3	94.2	74.1	71.2	77.2	72.0	99.6	97.1
DepthFormer SwinT-1k [35]	87.3	97.5	82.4	87.4	89.1	84.6	92.5	80.0	94.3	92.3	80.7	77.6	81.3	79.4	95.9	93.8
DepthFormer SwinT-22k [35]	94.2	98.6	93.0	96.0	95.5	90.6	96.4	83.5	97.2	96.4	92.0	92.3	93.2	92.2	98.4	97.6