SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Quantitative depth comparison on the NYU-Depth-v2 dataset. The maximum depth is capped at  $10\mathrm{m}$ .  $\ddagger$  stands for that the model is trained using auxiliary scene class information.
Method	Backbone	Abs Rel ↓	RMSE ↓	log10↓	δ < 1.25 ↑	δ < 1.252 ↑	δ < 1.253 ↑
Chen et al. [49]	SENet	0.166	0.494	0.071	0.757	0.943	0.984
VNL [42]	ResNeXt-101	0.183	0.541	0.082	0.696	0.912	0.973
BTS [7]	DenseNet-161	0.172	0.515	0.075	0.740	0.933	0.980
AdaBins [6]	E-B5+Mini-ViT	0.159	0.476	0.068	0.771	0.944	0.983
LocalBins [19]	E-B5	0.156	0.470	0.067	0.777	0.949	0.985
PixelFormer [20]	Swin-Large†	0.144	0.441	0.062	0.802	0.962	0.990
BinsFormer‡ [12]	Swin-Tiny	0.162	0.478	0.069	0.760	0.945	0.985
BinsFormer‡ [12]	Swin-Large†	0.143	0.421	0.061	0.805	0.963	0.990
Ours	Swin-Tiny	0.157	0.476	0.069	0.768	0.950	0.987
Ours	Swin-Large†	0.135	0.405	0.059	0.822	0.971	0.993
Table 4: Generalization to the SUN RGB-D dataset in a zero-shot setting with models trained on the NYU-Depth-v2 dataset.
Our framework is implemented in the PyTorch library [50] and trained on 4 NVIDIA A5000 24GB GPUs. The training process runs a total number of 20 epochs and takes around 24 hours. We utilize the Adam optimizer [51] and a batch size of 8. The learning rate is gradually reduced from 2e-5 to 2e-6 via the polynomial decay strategy.
4.3 Comparison to previous state-of-the-art competitors
KITTI. We report results on the Eigen split and official benchmark split, as summarized in Tables 1 and 2, respectively. For the Eigen split, the proposed method exceeds the leading approaches by a large margin, e.g., compared to PixelFormer, which is also based on the classification-regression. In terms of the official benchmark split, the results are generated by the online server and the proposed method outperforms previous approaches again. It is worth noting that the main ranking metric SILog is reduced considerably.
In Fig. 4, we present qualitative depth comparison on the KITTI dataset. As we can see, the proposed method is capable of correctly predicting the depth of foreground objects, such as poles, even when the background is very complex, e.g., lush foliage and bushes. In such complex scenes, PixelFormer and NeWCRFs are prone to foreground and background depth aliasing. Moreover, we notice that the compared methods, especially PixelFormer, tend to have mosaic-like artifacts at the top of the depth maps, but the proposed method does not.
NYU-Depth-v2. To further present the superiority of the proposed method in the indoor scenario, we evaluate it on the NYU-Depth-v2 dataset. The results are reported in Table 3, indicating that the proposed method surpasses prior competing approaches and achieves consistent improvements on most metrics. In particular, it improves BinsFormer by  $7.4\%$  and  $4.8\%$  on the Abs Rel and RMSE, respectively, which emphasizes the efficacy of our IEBins. In Fig. 5, we demonstrate qualitative depth
Figure 4: Qualitative depth comparison on the KITTI dataset. The red boxes show the regions to focus on.
Figure 5: Qualitative depth comparison on the NYU-Depth-v2 dataset. The white boxes indicate the regions to focus on.
comparison. As can be seen, the proposed method preserves fine-grained details, such as boundaries and generates more continuous depth values in planar regions.
4.4 Zero-shot Generalization
Similar to previous approaches [6, 12], we conduct a cross-dataset evaluation in a zero-shot setting where the models are trained on the NYU-Depth-v2 dataset but evaluated on the SUN RGB-D dataset. As shown in Table 4, the proposed method achieves superior results than the compared approaches. Besides, we notice that the proposed method with Swin-Tiny backbone performs slightly worse than AdaBins on the NYU-Depth-v2 dataset, but on the SUN RGB-D it even surpasses AdaBins on some metrics like Abs Rel, which is an indicator of its excellent generalization ability.
To better indicate the influence of each individual component, we conduct several ablation studies, divided into IEBins, bin types and effect of bin numbers.
IEBins. The importance of IEBins is first evaluated by comparing it with standard regression and IBins. We construct a baseline by removing the iterative optimizer from the whole framework. The standard regression is achieved by using the baseline to directly predict the depth map. The IBins represents that we replace the elastic target bin with the original target bin at each stage. Table 5 shows that the standard regression performs worse than IBins and IEBins. Benefiting from the elastic target bin, the more robust IEBins takes the IBins a step further.
Bin types. We then compare the IEBins against other choices including uniform bins [5], space increasing bins [5], adaptive bins [6] and local bins [19]. To make a fair comparison, we remain all other configurations the same except for bin types. As listed in Table 5, the baseline equipped with IEBins improves the performance markedly and surpasses other variants.
Method	Abs Rel ↓	RMSE ↓	log10 ↓	δ < 1.25 ↑	δ < 1.252 ↑	δ < 1.253 ↑
Baseline + Regression	0.095	0.337	0.041	0.921	0.991	0.998
Baseline + UBins [5]	0.091	0.328	0.040	0.925	0.991	0.998
Baseline + SIBins [5]	0.090	0.326	0.039	0.928	0.992	0.998
Baseline + AdaBins [6]	0.089	0.320	0.038	0.931	0.991	0.998
Baseline + LocalBins [19]	0.090	0.319	0.038	0.932	0.992	0.998
Baseline + IBins	0.090	0.317	0.038	0.932	0.991	0.998
Baseline + IEBins	0.087	0.314	0.038	0.936	0.992	0.998
Table 5: Comparison of different bin types on the NYU-Depth-v2 dataset. The baseline is built by removing the iterative optimizer from our whole framework. UBins: uniform bins; SIBins: space increasing bins; AdaBins: adaptive bins; LocalBins: local bins; IBins: iterative bins, which stands for replacing the elastic target bin with the original target bin at each stage. The number of bins for UBins, SIBins, AdaBins and LocalBins is set to 256 following [5, 6, 19].
Number of Bins	Abs Rel ↓	RMSE ↓	log10 ↓	δ < 1.25 ↑	δ < 1.252 ↑	δ < 1.253 ↑
4	0.105	0.335	0.043	0.900	0.982	0.999
8	0.089	0.317	0.038	0.934	0.992	0.998
16	0.087	0.314	0.038	0.936	0.992	0.998
32	0.088	0.317	0.038	0.932	0.992	0.998
Table 6: Effect of bin numbers on the NYU-Depth-v2 dataset.
Method	Backbone	Abs Rel ↓	Parameters (M) ↓	Inference Time (s) ↓
NeWCRFs [8]	Swin-Large†	0.095	270	0.052
BinsFormer‡[12]	Swin-Large†	0.094	255	0.216
Ours	Swin-Large†	0.087	273	0.085
Table 7: Comparison of model parameters and inference time on the NYU-Depth-v2 dataset.
Effect of bin numbers. To examine how the number of bins affects the performance, we train our model using different values of the bin number. The results are reported in Table 6. The error drops sharply as the number of bins increases, and then this drop disappears when it is greater than 16 bins. Therefore, we use 16 bins in our final model.
4.6 Model Parameters and Inference Time
We conduct a comparison between the proposed method, NeWCRFs and BinsFormer based on their inference time and the number of model parameters in Table 7, with the Swin-Large backbone. We measure the inference time on the NYU-Depth-v2 test set with a batch size of 1. It can be seen that the number of parameters of the proposed method is almost equal to that of NeWCRFs and slightly more than that of BinsFormer. Nevertheless, the proposed method is nearly  $60\%$  faster than BinsFormer in inference time, also as the classification-regression-based method. Meanwhile, the proposed method achieves much better performance than these two counterparts. Hence, the proposed method provides a better balance between performance, number of parameters and inference time.
We acknowledge the following limitations: First, we use the classification-regression to predict depth. Compared with the classification, the depth acquired by classification-regression is smoother and more continuous, but at the same time it may blur the boundaries due to the weighted average of all depth candidates. Second, we use the pixel-wise loss to supervise depth without imposing a direct supervision signal on the probabilistic distribution. This does not guarantee that the depth candidate corresponding to the peak point of the probability distribution is the real optimal depth candidate, thereby affecting the depth estimate.
In this paper, we introduce a novel concept of iterative elastic bins for the classification-regression-based monocular depth estimation. The proposed iterative elastic bins uses multiple small number of bins to progressively search for high-quality depth and can be plugged into other frameworks as a strong baseline. In addition, we present a dedicated framework composed of a feature extractor and an iterative optimizer. The performance is evaluated on two popular datasets from both outdoor and indoor scenarios and the proposed method exceeds previous state-of-the-art competitors. Furthermore, its generalization ability is verified in a zero-shot setting on the SUN RGB-D dataset.
This work was supported in part by the National Natural Science Foundation of China under grant U1909215 and 51975029, in part by the A\*STAR Singapore through Robotics Horizontal Technology Coordinating Office (HTCO) under Project C221518005, in part by the Key Research and Development Program of Zhejiang Province under Grant 2021C03050, in part by the Scientific Research Project of Agriculture and Social Development of Hangzhou under Grant No. 20212013B11, and in part by the National Natural Science Foundation of China under grant 61620106012 and 61573048.
[1] Weibin Jia, Wenjie Zhao, Zhihuan Song, and Zhengguo Li. Object servoing of differential-drive service robots using switched control. Journal of Control and Decision, 10(3):314-325, 2023.
[2] Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 559-568, 2011.
[3] Po-Yi Chen, Alexander H Liu, Yen-Cheng Liu, and Yu-Chiang Frank Wang. Towards scene understanding: Unsupervised monocular depth estimation with semantic-aware representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2624-2632, 2019.
[4] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in Neural Information Processing Systems, pages 2366-2374, 2014.
[5] Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2002-2011, 2018.
[6] Shariq Farooq Bhat, Ibrahim Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4009-4018, 2021.
[7] Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv:1907.10326, 2019.
[8] Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fully-connected crfs for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3916-3925, June 2022.
[9] Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 14475-14485, 2023.
[10] Shuwei Shao, Zhongcai Pei, Weihai Chen, Ran Li, Zhong Liu, and Zhengguo Li. Urcdc-depth: Uncertainty rectified cross-distillation with cutflip for monocular depth estimation. IEEE Transations on Multimedia, 2023.
[44] Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Luc Van Gool. P3depth: Monocular depth estimation with a piecewise planarity prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1610–1621, 2022.
[45] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-6114. PMLR, 2019.
[46] Shubhra Aich, Jean Marie Uwabeza Vianney, Md Amirul Islam, and Mannat Kaur Bingbing Liu. Bidirectional attention network for monocular depth estimation. In 2021 IEEE International Conference on Robotics and Automation, pages 11746-11752. IEEE, 2021.
[47] Siyuan Qiao, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3997-4008, 2021.
[48] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In 2017 International Conference on 3D Vision, pages 11–20. IEEE, 2017.
[49] Xiaotian Chen, Xuejin Chen, and Zheng-Jun Zha. Structure-aware residual pyramid network for monocular depth estimation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 694-700, 2019.
[50] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems Workshop Autodiff, 2017.
[51] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.