Harmonious convergence for confidence estimation in depth estimation and completion

Authors: ZAIWANG GU, Yanyu Xu, Lile Cai, Xulei Yang, Jun Cheng
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: confidence estimation, monocular depth estimation, depth completion

Abstract:
Confidence estimation for monocular  depth estimation and completion  is important for their deployment  in real-world applications. Recent models for confidence estimation in these regression tasks   mainly   rely on the statistical characteristics of training and test data, while ignoring the information from the model training.  We propose a harmonious convergence estimation approach for  confidence estimation in the regression tasks, taking training consistency into consideration. Specifically, we propose an intra-batch convergence estimation algorithm with two sub-iterations to compute the training consistency for confidence estimation. A  harmonious convergence loss is newly designed to encourage the consistency between confidence measure and depth prediction. Our experimental results on the NYU2 and KITTI datasets show improvements ranging from 10.91\% to 43.90\% across different settings in monocular depth estimation, and from 27.91\% to 45.24\% in depth completion, measured by Pearson correlation coefficients, justifying the effectiveness of the  proposed method. We will release all the codes upon the publication of our paper.

Decision: reject
Reviewer scores: 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730254882703

thread_type:
official_review

<thread_object>
creator: reviewer_t3bm

<content>
summary:
This paper proposes a confidence estimation method in depth estimation and depth completion. The existing dense prediction confidence estimation methods may suffer high memory demands, so this paper proposes an "intra-batch convergence estimation algorithm" for consistency computation, and designs a "harmonious convergence loss function" that integrates training consistency
into confidence estimation for monocular depth estimation and completion tasks. Some experiments conducted on NYU and KITTI datasets prove the effectiveness of this method. 

However, this paper fails to adequately summarize existing methods, and the proposed solution also lacks theoretical support, and some related works are missing for comparison.

strengths:
1. This paper proposes an "intra-batch convergence estimation algorithm" for consistency computation, and designs a "harmonious convergence loss function" that integrates training consistency into confidence estimation for monocular depth estimation and completion tasks. 
2. Some experiments conducted on NYU and KITTI datasets prove the effectiveness of this method.

weaknesses:
1. This paper does not provide a good summary of the problems of existing methods. So, I can not figure out the motivation behind the entire design. As mentioned in Line 48 "One challenge is addressing spatial misalignment of training samples caused by random augmentations", but why I should overcome this point? In other words, I do not understand why we have to introduce "training consistency" into confidence estimation networks. I can not see any advantages of doing so over existing methods. The only explanation is in line 130 "Although significant progress has been achieved, these methods fail to take the information from training process into consideration." Through these words, I can not understand the author's motivation.
2. There are some missing related works for comparison (Bae G et al. 2022, Xiang M et al. 2024). But the most important is I need to know the Strengths between this paper and these existing determined confidence estimation methods.

3. The proposed solution in the papers lacks theoretical support. In my opinion, the inter-batch loss is just a constraint that ensures the gradient is sufficiently small during the optimization process of the network. This is a simple modification of (Li et al. 2023) without corresponding explanations. And the ablation experiment (Table 3) also shows that the $L_h$  is the most important part of the whole network, not $L_c$.

[1]  Bae G, Budvytis I, Cipolla R. Irondepth: Iterative refinement of single-view depth using surface normal and its uncertainty[J]. arXiv preprint arXiv:2210.03676, 2022.

[2]  Xiang M, Zhang J, Barnes N, et al. Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2024.

[3] Chen Li, Xiaoling Hu, and Chao Chen. Confidence estimation using unlabeled data. In The Eleventh International Conference on Learning Representations (ICLR), 2023.

questions:
See in the Weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730453454515

thread_type:
official_review

<thread_object>
creator: reviewer_had4

<content>
summary:
This paper addresses the confidence estimation for monocular depth estimation and completion. To this end, the authors propose a intra-batch convergence estimation algorithm and a harmonious convergence loss. The experiments are conducted on the NYU2 and KITTI datasets and the results show improvements ranging from 10.91% to 43.90% across different settings in monocular depth estimation, and from 27.91% to 45.24% in depth completion.

strengths:
The paper is well-written and structured;

The experiments are thorough and conducted on multiple datasets;

The proposed method is effective according to the quantitative and qualitative results.

weaknesses:
Could you provide a more detailed analysis of how intra-batch convergence estimation or harmonious convergence loss contribute to the performance gains compared to previous approaches?

In Fig. 1, D0 and D1 appear to be different, but C0 and C1 look exactly the same；Could you explain this apparent discrepancy and discuss its implications for the method's effectiveness? Additionally, could you provide a more detailed visualization or analysis of how C0 and C1 differ?

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662463672

thread_type:
official_review

<thread_object>
creator: reviewer_ldwo

<content>
summary:
The paper proposes confidence estimation in monocular depth estimation and completion. The authors introduce a new method termed harmonious convergence estimation to integrate confidence estimation and for regression tasks taking training consistency.

- Introducing a harmonious convergence estimation and intra-batch convergence for monocular depth estimation and completion.
- Conducting diverse experiments across multiple datasets

strengths:
S1. Clear Contribution with Confidence Estimation. The paper presents a contribution by incorporating confidence estimation for monocular depth estimation and completion.

S2. Diverse Experiments: The paper shows diverse experiments with two different datasets

S3. Writing and Presentation: The paper is well-written and easy to understand.

weaknesses:
The paper lacks experimental results showing depth performance using commonly employed metrics in the depth estimation and completion research fields, such as δ (delta) or MAE.
This paper claims that the proposed confidence estimation method works well even in the more challenging task of regression compared to classification. 
Therefore, it is necessary to show more clearly that the performance of depth estimation or completion improves according to the confidence estimation even if this paper's main focus is related to confidence estimation.

questions:
I would like to start by thanking the authors for their contribution to this field with their submission.

This relates to Weakness. Is there any reason that those metrics are skipped in the experimental results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732720960021

thread_type:
official_comment

<thread_object>
creator: area_chair_kzfw

<content>
title:
Reminder: Last day for author feedback

comment:
This is a reminder that today is the last day allotted for author feedback. If there are any more last minute comments, please send them by today.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors proposed an confidence estimation (related to uncertainty quantification) approach for monocular depth completion and estimation. The main claim is that the proposed harmonious (intra-batch) convergence improves confidence of predictions. Reviewers T3bM and had4 brought up critical point regarding the motivation of why (theoretic) one would take this approach and what contributes to the gain, to which the author answered by re-describing the approach and stating the reason as being an extension of [1]; while T3bM and had4 did not reply, we do not think these responses adequately address the questions but bring up concerns that the approach may be incremental. Additionally, Reviewer LDwo brought up the need for additional evaluation metrics, we find that this is helpful to improve the paper but not critical. However, the comparisons made are of higher concern. Uncertainty quantification has been widely studied for both monocular depth completion and estimation. The choice of methods for comparisons were intended for broad range of tasks (with the exception of [F] which is for monocular depth estimation) and not necessarily depth completion or estimation. If the goal is to obtain better confidence or uncertainty then the experiments should at least include the tasks considered by the previous work. If the goal is to obtain better confidence or uncertainty specifically for depth completion and estimation then the authors should consider existing literature such as [A, B, C, D]. 

[A] Qu et al. Bayesian Deep Basis Fitting for Depth Completion with Uncertainty. ICCV 2021.

[B] Eldesokey et al. Uncertainty-aware cnns for depth completion: Uncertainty from beginning to end. CVPR 2020.

[C] Poggi et al. On the uncertainty of self-supervised monocular depth estimation. CVPR 2020.

[D] Marsal et al. MonoProb: Self-Supervised Monocular Depth Estimation With Interpretable Uncertainty. WACV 2024.

[E] Li et al. Confidence estimation using unlabeled data. ICLR 2023.

[F] Hornauer and Belagiannis. Gradient-based uncertainty for monocular depth estimation. ECCV 2022.