Title: IEBins: Iterative Elastic Bins for Monocular Depth Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Monocular depth estimation

Iterative refinement

Deep learning
Abstract: Monocular depth estimation (MDE) is a fundamental topic of geometric computer vision and a core technique for many downstream applications. Recently, several methods reframe the MDE as a classification-regression problem where a linear combination of probabilistic distribution and bin centers is used to predict depth. In this paper, we propose a novel concept of iterative elastic bins (IEBins) for the classification-regression-based MDE. The proposed IEBins aims to search for high-quality depth by progressively optimizing the search range, which involves multiple stages and each stage performs a finer-grained depth search in the target bin on top of its previous stage. To alleviate the possible error accumulation during the iterative process, we utilize a novel elastic target bin to replace the original target bin, the width of which is adjusted elastically based on the depth uncertainty. Furthermore, we develop a dedicated framework composed of a feature extractor and an iterative optimizer that has powerful temporal context modeling capabilities benefiting from the GRU-based architecture. Extensive experiments on the KITTI, NYU-Depth-v2 and SUN RGB-D datasets demonstrate that the proposed method surpasses prior state-of-the-art competitors. The source code is publicly available at https://github.com/ShuweiShao/IEBins.

IEBins: Iterative Elastic Bins for Monocular Depth Estimation

Abstract

Monocular depth estimation (MDE) is a fundamental topic of geometric computer vision and a core technique for many downstream applications. Recently, several methods reframe the MDE as a classification-regression problem where a linear combination of probabilistic distribution and bin centers is used to predict depth. In this paper, we propose a novel concept of iterative elastic bins (IEBins) for the classification-regression-based MDE. The proposed IEBins aims to search for high-quality depth by progressively optimizing the search range, which involves multiple stages and each stage performs a finer-grained depth search in the target bin on top of its previous stage. To alleviate the possible error accumulation during the iterative process, we utilize a novel elastic target bin to replace the original target bin, the width of which is adjusted elastically based on the depth uncertainty. Furthermore, we develop a dedicated framework composed of a feature extractor and an iterative optimizer that has powerful temporal context modeling capabilities benefiting from the GRU-based architecture. Extensive experiments on the KITTI, NYU-Depth-v2 and SUN RGB-D datasets demonstrate that the proposed method surpasses prior state-of-the-art competitors. The source code is publicly available at https://github.com/ShuweiShao/IEBins .

1 Introduction

Monocular depth estimation (MDE) is a long-standing and fundamental topic in geometric computer vision, with many applications in robotics [1], 3D reconstruction [2], scene understanding [3], etc. It consists in inferring the depth map from a single RGB image, which is ill-posed and has the challenge of scale ambiguity, because the same 2D image can be projected from infinitely many 3D scenes. Recently, more and more learning-based approaches have been proposed to promote the development of MDE [4, 5, 6, 7, 8, 9, 10, 11]. Without loss of integrity, these methods can be grouped into three categories: regression , classification and classification-regression [12].

Regression is the most primitive and straightforward formulation [4, 7, 13, 14, 15, 16, 8], which directly generates continuous pixel-wise depth under the supervision of a regression loss. Despite its great success as a universal paradigm, the regression-based model suffers from unsatisfactory results [5]. Classification is proposed in [5] and [17] to formulate the MDE as per-pixel classification and predict the optimal depth interval. More specifically, it discretizes the full depth range into multiple intervals (bins) in uniform/Log-uniform space (Fig. 1 (a) and (b)) and takes the bin center (depth candidate) of classified target bin as the final depth prediction. While the classification makes the MDE easier and significantly improves the model performance, the poor visual quality with discontinuity artifacts tends to appear [6].

(a) Uniform Bins

(b) Space Increasing Bins

(d) Iterative Elastic Bins

In order to overcome the discontinuity artifacts in classification, some methods [6, 18, 12, 19, 20] reframe the MDE as a per-pixel classification-regression problem, learning the probabilistic distribution on each pixel and using the linear combination with depth candidates as the final depth prediction. Theoretically, it can achieve the sub-pixel depth estimation. On top of that, Bhat et al. [6] noticed the extreme fluctuations in depth distribution across different scenes and proposed to derive adaptive bins (Fig. 1 (c)) from the image content. Li et al. [12] and Bhat et al. [19] further improved [6] by disentangling bins generation and probabilistic distribution learning or performing local predictions of depth distributions in a gradual step-wise manner. Moreover, Agarwal et al. [20] developed a bin center predictor that uses pixel queries at the coarsest level to predict bins.

In this paper, we introduce a novel concept termed iterative elastic bins (IEBins, Fig. 1 (d)) , tailored for the classification-regression-based MDE. The IEBins leverages multiple small number of bins, instead of one standard number of bins, to search for high-quality depth by progressively reducing the search range. To specify, the proposed IEBins involves multiple stages, where each stage predicts depth at different granularities and performs a finer-grained depth search in the target bin (the bin in which the predicted depth is located in our case) of its previous stage. Unfortunately, the depth groundtruth may fall outside the target bin due to wrong depth predictions, resulting in unstable optimization and degraded accuracy. To mitigate the error accumulation during iterations, we elastically adjust the width of the target bin according to the depth uncertainty that indicates the potential depth errors. Inspired by [21], we utilize the variance of the probabilistic distribution to quantify the uncertainty. Last but not least, we develop a dedicated framework (Fig. 2) consisting of two main components: a feature extractor that generates strong feature representations and a gated recurrent unit (GRU)-based iterative optimizer with powerful temporal context modeling capabilities that predicts the per-pixel probabilistic distribution from its hidden state for the depth classification-regression.

To summarize, our main contributions are three-fold:

We introduce a novel iterative elastic bins strategy for the classification-regression-based MDE. The IEBins performs an iterative elastic search using multiple small number of bins in light of the depth uncertainty.

We develop a framework to instantiate the proposed IEBins, where a feature extractor attains strong feature representations and a GRU-based iterative optimizer predicts the probabilistic distribution.

Extensive experiments are conducted on the KITTI [22], NYU-Depth-v2 [23] and SUN RGB-D [24] datasets, and the experimental results show that the proposed method exceeds previous state-of-the-art competitors.

2 Related work

Monocular depth estimation. Learning-based MDE has witnessed tremendous progress in recent years. Saxena et al . [27] proposed a pioneering work that uses Markov Random Field to capture critical local- and global-image features for depth estimation. Later, Eigen et al . [4] introduced a convolutional neural network (CNN)-based architecture to attain multi-scale depth predictions. Since then, CNNs have been extensively studied in MDE. For instance, Laina et al . [28] utilized residual CNN [29] for better optimization. Recently, Transformer [30] has attracted widespread attention in the computer vision community [31, 32, 33]. Following the success of visual Transformer in

with

P

GRU

Probabilistic Distribution

Depth

Candidates

P

GRU

Probabilistic Distribution

Depth

Candidates

P

GRU

Linear

Combination

Probabilistic Distribution

Depth

Candidates

P

GRU

Linear

Combination

Probabilistic Distribution

Depth

Candidates

P

GRU

Probabilistic Distribution

Depth

Candidates

P

GRU

Probabilistic Distribution

Depth

Candidates

IEBins

IEBins

Hidden State

Hidden State

Stage 1

Stage 2

Stage 3

U

P

Projection

U

Upsample

P

Projection

U

Upsample

Feature

Extractor

Linear

Combination

Linear

Combination

other tasks, Yang et al. [14], Ranftl et al. [34] and Yuan et al. [8] replaced CNN with Transformer, further improving the performance. However, the above methods suffer from sub-optimal solutions induced by the inherent drawback of regression [5]. Fu et al. [5] and Cao et al. [17] proposed to formulate MDE as a classification problem and discretized the full depth range into multiple bins to predict the optimal depth interval. Diaz et al. [35] softened the classification label in [5] during the training phase. In addition, Bhat et al. [6], Johnston et al. [18], Li et al. [12], Bhat et al. [19] and Agarwal et al. [20] reframed the MDE as per-pixel classification-regression to mitigate the discontinuity artifacts caused by depth discretization. Among them, [6] introduced an adaptive bins strategy to boost the performance. In contrast, we propose an iterative elastic bins paradigm for the classification-regression-based MDE, which uses multiple small number of bins rather than one standard number of bins like [6]. [19] also refines the binning structure in an iterative manner. However, [19] divides all bins from the previous stage at each stage, and when the stage increases, the number of bins increases, while the proposed IEBins locates and divides the target bin only, and the number of bins is not changed at different stages.

Iterative refinement. Recently, Teed et al. [36] proposed to iteratively refine a displacement vector field through the GRU for optical flow estimation, emulating the first-order optimization. This idea gradually receives attention in other tasks, such as stereo [37, 38], structure from motion [39] and scene flow [40]. In this paper, we deploy a GRU-based iterative optimizer to predict the per-pixel probabilistic distribution, where the hidden state is updated at each stage for more accurate estimation.

3 Methodology

In this section, we first introduce the iterative elastic bins tailored for the classification-regressionbased MDE. Then, we demonstrate a detailed description regarding the feature extractor and iterative optimizer in our framework. Finally, we present the training loss function.

3.1 Iterative Elastic Bins

The proposed IEBins embodies the idea of iterative division of bins, and is composed of two parts, initialization and update . In the initialization stage, we perform a coarse and uniform discretization of the full depth range. During each subsequent stage, we follow an iterative process to locate and uniformly discretize the target bin by using the target bin as the new depth range. The details are presented below.

Initialization. To initialize the IEBins, we discretize the full depth range [ d min , d max ] into N bins in the uniform space,

Stage 1

Stage 3

Stage 2

Stage 4

Stage 5

Stage 6

where e n denotes the n -th bin edge, B denotes the bin width, d max and d min are set to 80 and 0.1, and 10 and 0.1 for KITTI [22] and NYU-Depth-v2 [23] datasets, respectively, and N is set to 16 when not otherwise specified, a significantly fewer number of bins than the 256 bins used in AdaBins [6]. As we cannot directly use discrete bins to predict depth, we take the bin centers to represent the depth candidates of bins,

where D n denotes the n -th depth candidate. Once the per-pixel probabilistic distribution associated with depth candidates is predicted, we can acquire a depth prediction via their linear combination

where ̂ D ( p ) denotes the predicted depth, p denotes the pixel coordinate and P n denotes the n -th depth probability.

Update. Each subsequent stage searches at a finer-grained granularity in the target bin of its previous stage. The target bin represents the corresponding bin in which the predicted depth is located, which we obtain by comparing the predicted depth with bin edges

and denote as [ e n, e n +1 ] . However, as discussed in the introduction section, the depth ground-truth may fall outside the target bin due to depth prediction errors. In this case, the errors will gradually accumulate as the stage increases, resulting in unstable optimization and decreased accuracy. To account for such a failure case and make the paradigm more robust, we propose to leverage the elastic target bin to update the depth candidates. The width of the elastic target bin is adjusted flexibly based on the depth uncertainty reflecting the likelihood of depth inaccuracies. Inspired by [21], we capture the uncertainty via the variance of the probabilistic distribution. In particular, the variance ̂ V ( p ) is calculated as

The corresponding standard deviation is ̂ σ ( p ) = √ ̂ V ( p ) . Given the target bin [ e n, e n +1 ] and standard deviation ̂ σ ( p ) , we can acquire the elastic target bin by

where κ is a coefficient that determines the error tolerance and is set to 0.5. For a pixel with severe depth errors, the standard deviation will become larger such that the elastic target bin has higher immunity against errors. On top of that, we renew d min and d max as e n -0 . 5 ̂ σ ( p ) and e n +1 +0 . 5 ̂ σ ( p ) , respectively. Through Eqs.1, 2, 3 and 4, we can achieve the updated depth candidates and prediction. The update step will be repeated until reaching to the final stage. It should be noted that although we do not show p behind D in above formulas, the depth candidates D can actually be spatially-varying due to different elastic target bins in subsequent update steps. In Fig. 3, we display examples of depth maps and associated uncertainty maps at each stage.

3.2 Feature Extractor

The feature extractor adopts an encoder-decoder structure with skip-connections. The encoder uses the recently proposed Swin-Transformer family backbones [33, 41], which take an RGB image of size H × W as input and generate a four-level feature pyramid. Then, the skip-connections propagate the pyramid features into the decoding phase. The decoder uses three neural conditional random field (CRF) modules [8] to capture the vital long-range correlation. We alternate between the neural CRF module and pixel shuffle [25] up to the H 4 × W 4 resolution. Next, the H 4 × W 4 resolution feature maps from the encoder and the decoder are respectively sent to the iterative optimizer as the context feature and the initialization of GRU hidden state.

3.3 Iterative Optimizer

To efficiently predict the probabilistic distribution at each stage, we deploy a GRU-based iterative optimizer taking inspiration from [36], since the GRU is capable of retaining the information from history stages and can fully exploit the temporal context during iterations. The optimizer operates at H 4 × W 4 resolution.

More specifically, we first project the depth candidates D into the feature space using four 3 × 3 convolutional layers and each convolutional layer is followed by a ReLU activation [26]. We then concatenate the projected feature and the context feature to constitute a tensor I k as the input. The structure inside GRU is

where k is the stage index, z is the update gate, r is the reset gate, Conv 5 × 5 is the separable 5 × 5 convolution, glyph[circledot] is the element-wise multiplication, W z , W r and W h stand for learnable parameters. The hidden state h is initialized by the H 4 × W 4 resolution output in the decoder. The probabilistic distribution is finally predicted from the updated hidden state with two 3 × 3 convolutional layers. Meanwhile, the feature maps are regularized by ReLU and Softmax activations, respectively. A linear combination of probabilistic distribution and depth candidates is applied to obtain the depth prediction, which is further upsampled to the original resolution by the bilinear interpolation.

Equipped with this optimizer, initiating at a coarse prediction, the predicted depth is iteratively refined and eventually converges to the final result.

3.4 Training Loss Function

Pixel-wise depth loss . Following [8, 6], we leverage a scaled Scale-Invariant loss for depth supervision [7],

where g ( p ) = log ̂ D ( p ) -log D gt ( p ) , K is the maximum number of stages and is set to 6, T stands for the set of pixels having valid ground-truth values, α and β are set to 10 and 0.85 based on [7].

4 Experiment

We evaluate the proposed method on both outdoor and indoor datasets, which include KITTI [22], NYU-Depth-v2 [23] and SUN RGB-D [24]. In the following, we start by introducing the relevant datasets, evaluation metrics and implementation details. Then, we present quantitative and qualitative comparisons to prior state-of-the-art competitors, generalization and ablation studies, model parameters and inference time comparison.

4.1 Datasets and Evaluation Metrics

KITTI is an outdoor dataset captured by equipment mounted on a moving vehicle, providing stereo images and corresponding 3D laser scans. The images are around 376 × 1241 resolution. Here we adopt two data splits, Eigen training/testing split [4] and official benchmark split [48]. The former uses 23488 left view images for training and 697 images for testing. The latter consists of 85898 training images, 1000 validation images and 500 test images without the depth ground-truth. The evaluation results on the official benchmark split are generated by the online server.

NYU-Depth-v2 is an indoor dataset that has RGB images and ground-truth depth maps at a 480 × 640 resolution. We evaluate the proposed method on the official data split, which involves 36253 images for training and 654 images for testing.

SUN RGB-D is collected from indoor scenes with high diversity using four sensors, containing roughly 10K images. The dataset is only used for zero-shot generalization study and the official 5050 test images are adopted.

Evaluation metrics. Similar to previous work [8], we leverage the standard evaluation protocol to validate the efficacy of the proposed method in experiments, i.e., relative absolute error (Abs Rel), relative squared error (Sq Rel), root mean squared error (RMSE), root mean squared logarithmic error (RMSE log), inverse root mean squared error (iRMSE), log 10 error ( log 10 ), threshold accuracy ( δ < 1 . 25 , δ < 1 . 25 2 and δ < 1 . 25 3 ) and square root of the scale invariant logarithmic error (SILog).

4.2 Implementation Details

Our framework is implemented in the PyTorch library [50] and trained on 4 NVIDIA A5000 24GB GPUs. The training process runs a total number of 20 epochs and takes around 24 hours. We utilize the Adam optimizer [51] and a batch size of 8. The learning rate is gradually reduced from 2e-5 to 2e-6 via the polynomial decay strategy.

4.3 Comparison to previous state-of-the-art competitors

KITTI. We report results on the Eigen split and official benchmark split, as summarized in Tables 1 and 2, respectively. For the Eigen split, the proposed method exceeds the leading approaches by a large margin, e.g. , compared to PixelFormer, which is also based on the classification-regression. In terms of the official benchmark split, the results are generated by the online server and the proposed method outperforms previous approaches again. It is worth noting that the main ranking metric SILog is reduced considerably.

In Fig. 4, we present qualitative depth comparison on the KITTI dataset. As we can see, the proposed method is capable of correctly predicting the depth of foreground objects, such as poles, even when the background is very complex, e.g. , lush foliage and bushes. In such complex scenes, PixelFormer and NeWCRFs are prone to foreground and background depth aliasing. Moreover, we notice that the compared methods, especially PixelFormer, tend to have mosaic-like artifacts at the top of the depth maps, but the proposed method does not.

NYU-Depth-v2. To further present the superiority of the proposed method in the indoor scenario, we evaluate it on the NYU-Depth-v2 dataset. The results are reported in Table 3, indicating that the proposed method surpasses prior competing approaches and achieves consistent improvements on most metrics. In particular, it improves BinsFormer by 7.4 % and 4.8 % on the Abs Rel and RMSE, respectively, which emphasizes the efficacy of our IEBins. In Fig. 5, we demonstrate qualitative depth

Input

PixelFormer

NeWCRFs

Ours

Input

GT

BTS

Ours

PixelFormer

NeWCRFs

comparison. As can be seen, the proposed method preserves fine-grained details, such as boundaries and generates more continuous depth values in planar regions.

4.4 Zero-shot Generalization

Similar to previous approaches [6, 12], we conduct a cross-dataset evaluation in a zero-shot setting where the models are trained on the NYU-Depth-v2 dataset but evaluated on the SUN RGB-D dataset. As shown in Table 4, the proposed method achieves superior results than the compared approaches. Besides, we notice that the proposed method with Swin-Tiny backbone performs slightly worse than AdaBins on the NYU-Depth-v2 dataset, but on the SUN RGB-D it even surpasses AdaBins on some metrics like Abs Rel, which is an indicator of its excellent generalization ability.

4.5 Ablation Study

To better indicate the influence of each individual component, we conduct several ablation studies, divided into IEBins , bin types and effect of bin numbers .

IEBins. The importance of IEBins is first evaluated by comparing it with standard regression and IBins. We construct a baseline by removing the iterative optimizer from the whole framework. The standard regression is achieved by using the baseline to directly predict the depth map. The IBins represents that we replace the elastic target bin with the original target bin at each stage. Table 5 shows that the standard regression performs worse than IBins and IEBins. Benefiting from the elastic target bin, the more robust IEBins takes the IBins a step further.

Bin types. We then compare the IEBins against other choices including uniform bins [5], space increasing bins [5], adaptive bins [6] and local bins [19]. To make a fair comparison, we remain all other configurations the same except for bin types. As listed in Table 5, the baseline equipped with IEBins improves the performance markedly and surpasses other variants.

Effect of bin numbers. To examine how the number of bins affects the performance, we train our model using different values of the bin number. The results are reported in Table 6. The error drops sharply as the number of bins increases, and then this drop disappears when it is greater than 16 bins. Therefore, we use 16 bins in our final model.

4.6 Model Parameters and Inference Time

We conduct a comparison between the proposed method, NeWCRFs and BinsFormer based on their inference time and the number of model parameters in Table 7, with the Swin-Large backbone. We measure the inference time on the NYU-Depth-v2 test set with a batch size of 1. It can be seen that the number of parameters of the proposed method is almost equal to that of NeWCRFs and slightly more than that of BinsFormer. Nevertheless, the proposed method is nearly 60% faster than BinsFormer in inference time, also as the classification-regression-based method. Meanwhile, the proposed method achieves much better performance than these two counterparts. Hence, the proposed method provides a better balance between performance, number of parameters and inference time.

5 Limitations

We acknowledge the following limitations: First, we use the classification-regression to predict depth. Compared with the classification, the depth acquired by classification-regression is smoother and more continuous, but at the same time it may blur the boundaries due to the weighted average of all depth candidates. Second, we use the pixel-wise loss to supervise depth without imposing a direct supervision signal on the probabilistic distribution. This does not guarantee that the depth candidate corresponding to the peak point of the probability distribution is the real optimal depth candidate, thereby affecting the depth estimate.

6 Conclusion

In this paper, we introduce a novel concept of iterative elastic bins for the classification-regressionbased monocular depth estimation. The proposed iterative elastic bins uses multiple small number of bins to progressively search for high-quality depth and can be plugged into other frameworks as a strong baseline. In addition, we present a dedicated framework composed of a feature extractor and an iterative optimizer. The performance is evaluated on two popular datasets from both outdoor and indoor scenarios and the proposed method exceeds previous state-of-the-art competitors. Furthermore, its generalization ability is verified in a zero-shot setting on the SUN RGB-D dataset.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under grant U1909215 and 51975029, in part by the A*STAR Singapore through Robotics Horizontal Technology Coordinating Office (HTCO) under Project C221518005, in part by the Key Research and Development Program of Zhejiang Province under Grant 2021C03050, in part by the Scientific Research Project of Agriculture and Social Development of Hangzhou under Grant No. 20212013B11, and in part by the National Natural Science Foundation of China under grant 61620106012 and 61573048.

References

Weibin Jia, Wenjie Zhao, Zhihuan Song, and Zhengguo Li. Object servoing of differential-drive service robots using switched control. Journal of Control and Decision , 10(3):314-325, 2023.

Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology , pages 559-568, 2011.

Po-Yi Chen, Alexander H Liu, Yen-Cheng Liu, and Yu-Chiang Frank Wang. Towards scene understanding: Unsupervised monocular depth estimation with semantic-aware representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2624-2632, 2019.

David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In Advances in Neural Information Processing Systems , pages 2366-2374, 2014.

Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 2002-2011, 2018.

Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Adabins: Depth estimation using adaptive bins. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4009-4018, 2021.

Jin Han Lee, Myung-Kyu Han, Dong Wook Ko, and Il Hong Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv preprint arXiv:1907.10326 , 2019.

Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window fullyconnected crfs for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3916-3925, June 2022.

Zhenda Xie, Zigang Geng, Jingcheng Hu, Zheng Zhang, Han Hu, and Yue Cao. Revealing the dark secrets of masked image modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 14475-14485, 2023.

Shuwei Shao, Zhongcai Pei, Weihai Chen, Ran Li, Zhong Liu, and Zhengguo Li. Urcdc-depth: Uncertainty rectified cross-distillation with cutflip for monocular depth estimation. IEEE Transations on Multimedia , 2023.

Shuwei Shao, Zhongcai Pei, Weihai Chen, Xingming Wu, and Zhengguo Li. Nddepth: Normaldistance assisted monocular depth estimation. Proceedings of the IEEE International Conference on Computer Vision , 2023.

Zhenyu Li, Xuyang Wang, Xianming Liu, and Junjun Jiang. Binsformer: Revisiting adaptive bins for monocular depth estimation. arXiv preprint arXiv:2204.00987 , 2022.

Lam Huynh, Phong Nguyen-Ha, Jiri Matas, Esa Rahtu, and Janne Heikkilä. Guiding monocular depth estimation using depth-attention volume. In European Conference on Computer Vision , pages 581-597. Springer, 2020.

Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe, and Elisa Ricci. Transformer-based attention networks for continuous pixel-wise prediction. In Proceedings of the IEEE International Conference on Computer Vision , pages 16269-16279, October 2021.

Xiaoxiao Long, Cheng Lin, Lingjie Liu, Wei Li, Christian Theobalt, Ruigang Yang, and Wenping Wang. Adaptive surface normal constraint for depth estimation. In Proceedings of the IEEE International Conference on Computer Vision , pages 12849-12858, October 2021.

Shuwei Shao, Ran Li, Zhongcai Pei, Zhong Liu, Weihai Chen, Wentao Zhu, Xingming Wu, and Baochang Zhang. Towards comprehensive monocular depth estimation: Multiple heads are better than one. IEEE Transactions on Multimedia , 2022.

Yuanzhouhan Cao, Zifeng Wu, and Chunhua Shen. Estimating depth from monocular images as classification using deep fully convolutional residual networks. IEEE Transactions on Circuits and Systems for Video Technology , 28(11):3174-3182, 2017.

Adrian Johnston and Gustavo Carneiro. Self-supervised monocular trained depth estimation using self-attention and discrete disparity volume. In Proceedings of the IEEE International Conference on Computer Vision , pages 4756-4765, 2020.

Shariq Farooq Bhat, Ibraheem Alhashim, and Peter Wonka. Localbins: Improving depth estimation by learning local distributions. In Proceedings of the European Conference on Computer Vision , pages 480-496. Springer, 2022.

Ashutosh Agarwal and Chetan Arora. Attention attention everywhere: Monocular depth prediction with skip attention. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision , pages 5861-5870, 2023.

Dongting Hu, Liuhua Peng, Tingjin Chu, Xiaoxing Zhang, Yinian Mao, Howard Bondell, and Mingming Gong. Uncertainty quantification in depth estimation via constrained ordinal regression. In Proceedings of the European Conference on Computer Vision , pages 237-256. Springer, 2022.

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013.

Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer Vision , pages 746-760. Springer, 2012.

Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 567-576, 2015.

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1874-1883, 2016.

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , pages 315-323. JMLR Workshop and Conference Proceedings, 2011.

Ashutosh Saxena, Sung H Chung, Andrew Y Ng, et al. Learning depth from single monocular images. In Advances in Neural Information Processing Systems , volume 18, pages 1-8, 2005.

Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 2016 Fourth International Conference on 3D Vision , pages 239-248. IEEE, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , June 2016.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998-6008, 2017.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations , 2021.

Zhiliang Peng, Wei Huang, Shanzhi Gu, Lingxi Xie, Yaowei Wang, Jianbin Jiao, and Qixiang Ye. Conformer: local features coupling global representations for visual recognition. In Proceedings of the IEEE International Conference on Computer Vision , pages 367-376, October 2021.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Computer Vision , pages 10012-10022, October 2021.

René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE International Conference on Computer Vision , pages 12179-12188, 2021.

Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4738-4747, 2019.

Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Proceedings of the European Conference on Computer Vision , pages 402-419. Springer, 2020.

Lahav Lipson, Zachary Teed, and Jia Deng. Raft-stereo: Multilevel recurrent field transforms for stereo matching. In 2021 International Conference on 3D Vision (3DV) , pages 218-227. IEEE, 2021.

Fangjinhua Wang, Silvano Galliani, Christoph Vogel, and Marc Pollefeys. Itermvs: iterative probability estimation for efficient multi-view stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8606-8615, 2022.

Xiaodong Gu, Weihao Yuan, Zuozhuo Dai, Siyu Zhu, Chengzhou Tang, Zilong Dong, and Ping Tan. Dro: Deep recurrent optimizer for video to depth. IEEE Robotics and Automation Letters , 8(5):2844-2851, 2023.

Zachary Teed and Jia Deng. Raft-3d: Scene flow using rigid-motion embeddings. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 8375-8384, 2021.

Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 12009-12019, June 2022.

Wei Yin, Yifan Liu, Chunhua Shen, and Youliang Yan. Enforcing geometric constraints of virtual normal for depth prediction. In Proceedings of the IEEE International Conference on Computer Vision , pages 5684-5693, 2019.

Sihaeng Lee, Janghyeon Lee, Byungju Kim, Eojindl Yi, and Junmo Kim. Patch-wise attention network for monocular depth estimation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 1873-1881, 2021.

Vaishakh Patil, Christos Sakaridis, Alexander Liniger, and Luc Van Gool. P3depth: Monocular depth estimation with a piecewise planarity prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 1610-1621, 2022.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning , pages 6105-6114. PMLR, 2019.

Shubhra Aich, Jean Marie Uwabeza Vianney, Md Amirul Islam, and Mannat Kaur Bingbing Liu. Bidirectional attention network for monocular depth estimation. In 2021 IEEE International Conference on Robotics and Automation , pages 11746-11752. IEEE, 2021.

Siyuan Qiao, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Vip-deeplab: Learning visual perception with depth-aware video panoptic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3997-4008, 2021.

Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In 2017 International Conference on 3D Vision , pages 11-20. IEEE, 2017.

Xiaotian Chen, Xuejin Chen, and Zheng-Jun Zha. Structure-aware residual pyramid network for monocular depth estimation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 694-700, 2019.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In Advances in Neural Information Processing Systems Workshop Autodiff , 2017.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , 2015.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces an Iterative Elastic Bins (IEBins) approach for monocular depth estimation. Many conventional monocular depth estimation approach uses a soft-argmax representation (Eq. (4)) that sums the product between depth probability and pre-defined depth bins. However, the large number of pre-defined bins hinders model convergence (@L.43). To alleviate the problem, the paper proposes to use adaptive ranges of depth bins for each pixel based on previous intermediate depth estimates. Along with the number of iteration steps, the depth ranges get smaller for fine-detail estimation (yet, taking uncertainty into account). The paper shows empirically good accuracy over previous methods.\n\nSTRENGTHS:\n+ Good results\n\n  Table 1, 2, 3, and 4 show that the method achieves better accuracy than published methods in the public benchmark datasets (KITTI Eigen, KITTI, NYU-Depth-v2, and SUN RGB-D). \n\n+ Good ablation study\n\n   Table 5 and 6 demonstrates how each design choice affects/improves the accuracy. Table 5 shows the strength of the proposed IEBin idea over previous approaches (UBins, SIBins, AdaBins, and IBins). Table 6 also justifies the design choice of the number of bins.\n\nWEAKNESSES:\n- Question on the number of stages (Fig. 3)\n\n  The method uses 6 stages (iteration steps) in total. I wonder what happens if it uses more iteration steps during the inference (i.e., 6 iterations during training, but 8 or 10 iterations during test time). Will the method keep improving the accuracy with more iteration steps? Or, even during training, what if it uses more iteration steps other than 6? How the number 6 is set?\n\n- Possibly an unfair comparison? Table 5, Comparison with different bin types.\n\n  I wonder if other methods (UBins, SIBins, AdaBins, and IBins) also use the iterative optimizer. For a fair comparison, I think it's also good to prepare a baseline with the iterative optimizer and only change the depth bin types. Then it can differentiate the performance gain of bin types from the iterative optimizer. \n\n\nDespite the good results, I would like the paper Borderline Reject for now, mainly due to the possible unfair comparison in Table 5. The source of gain is not so clear if it's from the iterative optimizer or the depth bin types. \n\n----\n\nI share the same concern with Reviewer 4WPj that the main contributions don't seem to be so strong. The iterative refinement ideas have been proposed in other literature (e.g., optical flow, SfM), and the paper demonstrates empirical accuracy gain. That said, I am raising the score to Borderline Accept as the authors' response resolves my main concerns. I hope all the comments and concerns during the discussion phase are reflected in the updated version.\n\nQUESTIONS:\n- In Fig. 2 the three depth maps on the right look the same. Probably it may need to be changed to the actual experiment results. \n- In the last example (chair) in Fig. 5, the object boundary near the chair is still blurry. (Just out of curiosity) I wonder if the proposed representation can resolve this blurry object boundary problem because it limits the depth range, so I would expect it outputs much clearer depth around object boundaries. \n- In Table 7, what is the source of the faster inference time than BinsFormer?\n\nLIMITATIONS:\nThe paper didn't include its limitations or societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

comments: SUMMARY:
The paper introduces a method for monocular depth. The method uses a recurrent network based on RAFT to predict a probability distribution over a set of bins, which enables the depth at each iteration to be computed as the expectation over all bins and the margins of each bin to be adjusted using the computed variance. In the subsequent iterations, a finer-grained search is performed in the adjusted bins. This new strategy is introduced as "IEBins". The authors motivate this strategy as being robust to uncertain initial predictions.

STRENGTHS:
The method outperforms prior work across multiple standard evaluation settings.

The method is interpretable, producing confidence for different regions and steadily improving depth predictions.

To the best of my knowledge, the iterative design is substantially different than the usual approach for monocular depth.

The introduced method is very fast.

The IEBins are compared against other binning strategies in ablation experiments.

WEAKNESSES:
No obvious weaknesses.

QUESTIONS:
None

LIMITATIONS:
I did not see discussion of limitations
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a classification-regression-based monocular depth estimation pipeline.\nPrevious classification or classification-regression-based MDE approaches often suffer from huge complexity and loss of generalizability issue, due to the nature of requiring more depth hypothesis for better performance.\nIn this paper, it propose iterative elastic bins, ie, IEBins technique, which iteratively adjusts bin ranges as the prediction converges, so that with less number of bins and fast inference time, the method can still perform on the state-of-the-art level.\nIt additionally introduces dedicated transformer-based feature extractor and GRU-based iterative optimizer.\n\nSTRENGTHS:\nThis paper is well-written and easy to follow.\nThe proposed IEBins is an intuitive and interesting idea, and the performance also backs up its effectiveness.\n\nWEAKNESSES:\nMore in-depth analysis on 'iterative' perspective of IEBins will be appreciated.\n\n1) For example, what happens when iteration is less or more than 6? How much does the number of iterations affect the final performance?\n\n2) The authors claim that the proposed GRU-based iterative optimizer is also one of their contributions, saying that it is helpful to capture temporal information during IEBins-based depth estimation. Yet, there is no ablation study regarding this. Comparison between adding/excluding GRU unit, alongside its effectiveness regarding the number of iteration will help strengthening the claimed contribution.\n\nQUESTIONS:\n1. I have some questions regarding updating rules of the proposed IEBins.\n\n1-1. In Eq. 7, it denotes that elastic target bin edges are modified using uncertainty from previous stage's probability distribution. Does it mean that for each pixel, bin edges are set differently? If this is the case, the network should remember separate bin ranges for every pixel, which seems like to require huge memory. Correct me if I'm wrong.\n\n1-2. In L133, the paper says that d_min and d_max values are updated with new bin edges. In the next iteration, are 16 bins additionally set within this new min-max range? Then, does it mean that in 2nd iteration, for example, new bin range are approximately 1/256 from original min-max range?\n\n2. In Fig 5, the range of the proposed method seems significantly off compared to GT. How is the visualization done?\n\n3. In Tab 6, why does the performance drop when using 32 bins instead of 16? Is this aligned with overfitting issue stated in the introduction section? If it's true, than doesn't it mean that the proposed method already suffers from 32 bins while AdaBins, for example, can operate until 256 bins? How can this phenomenon be analyzed?\n\nLIMITATIONS:\nThe authors did not address any limitations. Yet, other than weakness and questions I commented above, I don't see any severe limitation to this paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the task of monocular depth estimation which is of fundamental importance in computer vision and has many downstream applications. Several recent works use bin-based approaches and follow the adaptive binning framework where the distribution of bin centers on the depth interval (that are treated as depth candidates) can vary per image or per-pixel. \n\nThis paper proposes a novel approach to the adaptive binning framework called IEBins. Instead of working with only one (potentially large) set of bins (per image or per pixel), IEBins refines the binning structure in an iterative manner. Starting with a coarse uniform division of the original depth interval (dmin, dmax), the idea is to recursively find and uniformly divide the ‘target’ bin by using the ’target’ bin as the new target depth interval for the next step. In addition, the target bin is made elastic (the new target depth interval’s ends are allowed to change) based on the uncertainty estimate. \n\nThe work achieves state-of-the-art results on popular benchmarks including KITTI and NYU-Depth-v2 and the authors promise to release the code and models publicly.\n\nSTRENGTHS:\n* State-of-the-art results. The work achieves SOTA in a highly competitive space of monocular depth estimation with over 7% improvement in RMSE over the prior SOTA. IEBins idea is also validated and shows about 2% improvement to adaptive bins baseline in fair settings.\n* Interesting idea. The idea of recursive division of bins is obvious in retrospect yet creative. Elastic nature of bins based on uncertainty also makes sense and subtly introduces the ‘adaptive’ nature of bins. \n* Well exploited ideas and good architecture design. Authors have well exploited the ideas from other works to obtain their goal. e.g. Iterative refinement using GRU, developed in separate field of optical flow estimation, has been introduced here in a well designed architecture. \n* Potential to be foundational. Recently, depth estimation has seen foundational ideas like adaptive binning that serve as a starting point for a series of several follow up works. I believe the iterative refinement of bins can prove to be as influential. \n* Good to see ablations on all bin types in fair settings.\n\nWEAKNESSES:\n* W1 **Incomplete literature review**. Authors have missed some important published works that are highly relevant to this work:  \n    * [a] - LocalBins (ECCV’22) also introduces the idea of “splitting” the bins and step-wise refinement of bins in a coarse-to-fine manner. IEBins and LocalBins are highly related and pursue the same goal but follow different approaches. Quantitative comparison with [a] and providing the insights to differences and similarities is highly suggested. \n    * [b] - PixelFormer (WACV’23) is also based on the adaptive binning framework and introduces layer-wise refinement of “pixel queries”. \n* W2 **Absence of quantitative evidence towards the working of the fundamental idea**. According to the proposed idea of iteratively making the ‘target’ depth interval smaller, the bin-width (or its median across an image) should exponentially decrease stage-by-stage, with some flexibility introduced by the elastic nature. At the same time, if the bins are too elastic i.e. final bin-widths are comparable to initial bin-widths (e.g. only 50% smaller than previous or so), then the iterative restrictive nature of depth search is invalidated. Although authors visualise the refinement process in Fig. 3 but these results can still be explained by ‘too elastic’ bins as the elastic-bin-adjustment varies spatially. Also, are the uncertainty 'heat maps' scaled individually or globally? If individually (which seems to be the case), then the uncertainty visualizations only show that uncertainty tends to take much higher values near edges, rather than an overall absolute decrease and therefore tell nothing about the bin widths. The authors are suggested to provide evidence to show that method actually works as described to avoid misleading conclusions. For example, via a simple line plot of evolution of target bin-width (multiple lines for pixels across a row or single line plot of median elastic bin width across image etc) through the course of refinement, or any other form that authors deem suitable that delivers the evidence clearly. \n* W3 **Unsubstantiated claim at L44**. Authors claim that large number of bins are undesirable and use this as the main motivation for the work. Authors state at L43-44 that “Learning ambiguity grows rapidly as the number of bins increases” but fail to provide any explanation, reference or evidence. As this information can be potentially misleading, authors should either remove the claims or provide an explanation, reference or evidence towards the claim.\n\n[a] Bhat, Shariq Farooq, Ibraheem Alhashim, and Peter Wonka. \"Localbins: Improving depth estimation by learning local distributions.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[b] Agarwal, Ashutosh, and Chetan Arora. \"Attention attention everywhere: Monocular depth prediction with skip attention.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n\nQUESTIONS:\n* How do the bin-widths actually evolve during the iterative refinement? What is the mean of elasticity factor (new adjusted width divided by the bin-width at that stage with no elasticity)?\n* What is the maximum bin-width in meters reached at the final stage e.g. for the NYU test set? This can also answer the elasticity to some extent.\n* Why is a large number of bins undesirable? What is the evidence?\n* How does IEBins compare with LocalBins?\n\nLIMITATIONS:\nThe authors do not explicitly reflect upon the limitations or scope of the work. Refer to \"weaknesses\" for major concerns.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}

comments: SUMMARY:
This paper proposes iterative elastic bins (IEBins), a multi-stage coarse-to-fine method for monocular depth estimation (MDE). It progressively searches the target depth bin on top of the previous step. To reduce the error accumulation in the iterations, an elastic bin is proposed whose width is adjusted based on the depth prediction uncertainty. Experiments on the KITTI, NYU-v2, and SUN RGB-D datasets show improved performances on MDE.

STRENGTHS:
- The depth bin is adjusted elastically based on the depth prediction uncertainty.

WEAKNESSES:
- The contributions are not enough to my knowledge. I think only the elastic bins are new. The iterative manner for depth estimation by GRUs has been proposed in RAFT[28] and adopted in many follow-up works, e.g., RAFT-Stereo[29] and Itermvs[30]. The depth bins as shown in Eq 1,2,3&4 have been investigated in Adabins [3]. As for the framework of the feature extractor and GRU-based layers, standard implementations are involved. The multi-stage depth estimation pattern was also seen in previous works, e.g., CasMVS [Gu et al, cvpr2020] and IterMVS[30].
- Some minor ones:
  - line 141: a RGB --> an RGB
  - line 151: during iteration --> during iterations

QUESTIONS:
- Have other depth uncertainty presentations been investigated or compared with the variance of the probabilistic distribution used in Eq 6?

LIMITATIONS:
- N/A for the limitations.
- I suggest the authors provide the failure cases (and the corresponding explanations) of the proposed IEBins, w.r.t. for example, the number of bins and the elastic width of bins.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-14

comments: SUMMARY:
The paper proposes an iterative elastic bins (IEBins) strategy for monocular depth estimation. The IEBins use a small number of bins adaptively at each iteration. It use a GRU to predict the depth distribution at each stage. The authors conduct experiments on 3 commonly used datasets and the proposed method shows better results than previous methods.

STRENGTHS:
The paper is well written and easy to follow. 
The idea of using iterative update is interesting.
The experiments and ablation study are thorough and validate the proposed method's effectiveness.

WEAKNESSES:
1. The motivation of iterative updates is not very clear. RAFT used iterative updates because at each iteration, a new cost volume can be constructed based on current prediction. But I don't see where the new information comes from in MDE. 
2. Based on 1. Maybe the improved performance come from the depth candidates projects since the model knows what the depth is. Would be interesting to see how the model performance by using a positional encoding of depth candidates to other non-iterative methods, such as adabins.

QUESTIONS:
L44. "This learning ambiguity grows rapidly as the number of bins increases, exacerbating the difficulty of model convergence and prone to overfitting." Why the ambiguity grows in this case? And why the proposed iterative method doesn't lead to increased ambiguity?

LIMITATIONS:
1. Currently only the depth maps are shown. It would be great to compare the point cloud with other methods since it's much easier to see the 3D structure in point cloud.
2. How is the improved depth estimation benefits in downstream task (compared to other MDE methods)? For example, could it be used for SLAM or 3D reconstruction from multiview images.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25