Self-supervised Monocular Depth Estimation Robust to Reflective Surface Leveraged by Triplet Mining

Authors: Wonhyeok Choi, Kyumin Hwang, Wei Peng, Minwoo Choi, Sunghoon Im
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Self-supervised Monocular Depth Estimation, Deep Metric Learning, Knowledge Distillation

Abstract:
Self-supervised monocular depth estimation (SSMDE) aims to predict the dense depth map of a monocular image, by learning depth from RGB image sequences, eliminating the need for ground-truth depth labels.
Although this approach simplifies data acquisition compared to supervised methods, it struggles with reflective surfaces, as they violate the assumptions of Lambertian reflectance, leading to inaccurate training on such surfaces.
To tackle this problem, we propose a novel training strategy for an SSMDE by leveraging triplet mining to pinpoint reflective regions at the pixel level, guided by the camera geometry between different viewpoints.
The proposed reflection-aware triplet mining loss specifically penalizes the inappropriate photometric error minimization on the localized reflective regions while preserving depth accuracy on non-reflective areas.
We also incorporate a reflection-aware knowledge distillation method that enables a student model to selectively learn the pixel-level knowledge from reflective and non-reflective regions. This results in robust depth estimation across areas.
Evaluation results on multiple datasets demonstrate that our method effectively enhances depth quality on reflective surfaces and outperforms state-of-the-art SSMDE baselines.

Decision: accept (poster)
Reviewer scores: 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730211123454

thread_type:
official_review

<thread_object>
creator: reviewer_tqa3

<content>
summary:
This paper aims to address inaccurate depth estimation on the reflective surfaces in a scene. To this end, this paper proposes a training strategy for self-supervised depth estimation by leveraging triplet mining. The key idea is to penalize the inappropriate photometric error minimization on the localized reflective regions while preserving depth accuracy on non-reflective areas. In addition, this paper introduces a reflection-aware knowledge distillation method that enables a student model to selectively learn the pixel-level knowledge from reflective and non-reflective regions. Finally, the proposed framework is validated on various indoor datasets.

strengths:
1.	This paper is well-written and the proposed method is easy to comprehend.
2.	This method is simple and general, which can be used for most self-supervised depth estimation methods.

weaknesses:
1.Whether Eq. (8) can accurately reflect the reflective surface? Due to the ambiguity of photometric errors, the magnitude of photometric errors is not even a measure of the accuracy of depth [1], let alone reflective surfaces .
2.The overall performance is very marginal. From the results on the test set in Table 4, we can find that, like most end-to-end self-supervised training solutions, all comparison methods have limited improvement or even worse after using the idea proposed by the author. 
3.Experimental verification is inadequate. The selected baselines, Monodepth2(2019), HRDepth(2021), MonoViT(2020) are the early works, not the SOTA framework. I am concerned about the effectiveness of the triplet mining loss when incorporated into recent works. Moreover, there is a lack of comparison on common depth estimation datasets, such as KITTI, NYUv2, etc., where reflective surfaces also exist.

[1] Frequency-aware self-supervised monocular depth estimation.

questions:
1.	Verify effectiveness on some recent SOTA frameworks, such as [1,2,3]
2.	Training and validation on common self-supervised depth estimation datasets like KITTI, NYUv2.
3.	See weaknesses for other issues.

[1] RA-Depth: Resolution Adaptive Self-Supervised Monocular Depth Estimation.
[2] GasMono: Geometry-Aided Self-Supervised Monocular Depth Estimation for Indoor Scenes.
[3] Self-supervised Monocular Depth Estimation with Large Kernel Attention.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730526286543

thread_type:
official_review

<thread_object>
creator: reviewer_jmz7

<content>
summary:
This paper proposes a novel self-supervised monocular depth estimation training strategy, which uses triplet mining loss and reflection-aware knowledge distillation to pinpoint reflective regions at the pixel level, resulting in a robust depth estimation.

strengths:
1.The proposed triplet mining loss can be integrated into a general monocular depth estimation framework, enhancing the accuracy of reflective surfaces.

2.The proposed reflection-aware knowledge distillation enhances the accuracy of reflective surfaces while preserving high-frequency details of non-reflective surfaces.

3.Experiments on multiple datasets demonstrate that the proposed method achieves state-of-the-art performance.

weaknesses:
1.Ablation study on the proposed pixel-level reflection regions Mr are necessary to further analyze its effectiveness in reflection-aware triplet mining loss and reflection-aware knowledge distillation.

2.Will the proposed triplet mining strategy and reflection-aware knowledge distillation introduce additional computational overhead?

3.The description in lines 246-247 is confusing. The relative pose [R|t]s2r is obtained by computing the inverse of [R|t]s2r?

4. Depth completion is an extended task from depth estimation. Thus, it would be better if adding some depth completion methods [1-5] into the related work section.

[1] Bilateral Propagation Network for Depth Completion, CVPR 2024

[2] Tri-Perspective View Decomposition for Geometry-Aware Depth Completion, CVPR 2024

[3] RigNet: Repetitive Image Guided Network for Depth Completion, ECCV 2022

[4] Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera, ICRA 2019

[5] Desnet: Decomposed Scale-Consistent Network for Unsupervised Depth Completion, AAAI 2023

questions:
I would appreciate it If the authors can address my concerns in Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730659522871

thread_type:
official_review

<thread_object>
creator: reviewer_ptjg

<content>
summary:
The paper proposes a reflection-aware training method for self-supervised monocular depth estimation. The proposed method leverages the triplet mining loss to regularize the inappropriate regions for photometric loss.
Key contributions of this work includes:
- Introducing reflection-aware triplet mining loss and knowledge distillation for reflective regions.
- first end-to-end method for self-supervised monocular depth estimation in reflective regions.
- diverse experimetns on various indoor datasets.

strengths:
S1. This is the first end-to-end method for self-supervised monocular depth estimation in reflective regions. The proposed triplet loss is very simple and can be integrated into any framework for SSMDE.

S2. Diverse Experiments: The paper shows diverse experiments with multiple datasets and cross validations.

S3.  Writing and Presentation: The paper is well-written and easy to understand.

weaknesses:
W1. Many self-supervised monocular depth estimation methods are evaluated in outdoor scenes like KITTI. However, this paper shows their results in only indoor scenes using Scannet, 7-Scenes and Booster datasets.

W2. I could not find the proof of the authors' assumption that reflective regions have low disparities (instead of larger disparities).

questions:
I would like to start by thanking the authors for their contribution to this field with their submission. Here are some questions:

Q1: This relates to W1. Are there any reasons for not including experiments on outdoor scenes, such as the KITTI dataset? The proposed method is demonstrated on indoor datasets, where pose prediction networks might be challenging to train due to prevalent rotational camera movements. I am curious whether this method is adaptable when a pose network is involved or if it can be applied to depth estimation tasks in outdoor scenes.

Q2: This relates to W2. I am concerned about whether it is possible to say that the predicted depth maps for reflective regions always have small disparities (instead of randomness errors). Is there any provements or previous works showing this is true?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732067680603

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We sincerely thank the reviewers for their thorough evaluation of our work and for providing insightful and constructive feedback. We appreciate the time and effort spent reviewing our manuscript, and we value the opportunity to address the concerns and questions raised.

We provide detailed responses to each reviewer’s comment, along with additional experiments and clarifications where necessary. Below, we detail additional experiments conducted per reviewer’s requests:

>In response to **Reviewer ptjG** and **Reviewer tqa3**, we included the experiments that evaluate our method on two common datasets: KITTI and NYUv2. All these experiments were conducted with a pose network instead of GT poses to validate that our method consistently performs well even when involving a pose network, similar to the existing SSMDEs. 
>
>Additionally, we have reported experimental results on the pittsStereo-RGBNIR dataset **[6]**, an outdoor scene dataset composed of challenging materials, to analyze the impact of our method on both reflective and non-reflective classes.

>Responding to **Reviewer jMZ7**, we summarized the computational overhead of monodepth2 **[1]** under four scenarios: (1) training with traditional self-supervision, (2) applying our triplet loss, (3) applying our proposed distillation method, and (4) employing the 3D distillation **[2]** approach.

>Addressing the inquiries from **Reviewer jMZ7** and **Reviewer tqa3**, we introduce the experiment results and analysis of the proposed reflective region mask $M_r$. Note that ablation studies of $M_r$ are already included in ```Section D``` of the supplementary material.

>Based on the feedback from **Reviewer tqa3**, we added the experiment results of brand-new methods, including RA-depth **[3]**, GasMono **[4]**, and Lite-mono **[5]**.

We hope that our responses adequately address the reviewers' concerns and further demonstrate the robustness and significance of our contributions. Please do not hesitate to let us know if additional clarifications are needed.

### **References**

_[1] Godard, Clément, et al. "Digging into self-supervised monocular depth estimation." Proceedings of the IEEE/CVF international conference on computer vision. 2019._

_[2] Shi, Xuepeng, et al. "3D Distillation: Improving Self-Supervised Monocular Depth Estimation on Reflective Surfaces." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023._

_[3] He, Mu, et al. "Ra-depth: Resolution adaptive self-supervised monocular depth estimation." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022._

_[4] Zhao, Chaoqiang, et al. "GasMono: Geometry-aided self-supervised monocular depth estimation for indoor scenes." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023._

_[5] Zhang, Ning, et al. "Lite-mono: A lightweight cnn and transformer architecture for self-supervised monocular depth estimation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023._

_[6] Zhi, Tiancheng, et al. "Deep material-aware cross-spectral stereo matching." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018._

PittsStereo Dataset Demo: https://www.cs.cmu.edu/~ILIM/projects/AA/RGBNIRStereo/
</content>
</thread_object>

</thread>
<thread>
creation date:
1732439929646

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Friendly Reminder for Discussion Phase Participation

comment:
First, thank you for your constructive reviews on our paper. Your feedback has been invaluable, and we have put our best effort into addressing the points raised in our responses.

We would like to kindly remind you that there are only 3 days left in the authors-reviewers discussion phase for our paper. If you have any issues or additional questions regarding the response, please feel free to ask. 

We appreciate your time and effort, and we look forward to your comments.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a novel training strategy for an SSMDE, focusing on addressing regions that violate the assumptions of Lambertian reflectance. Initially, the reviewers expressed concerns regarding the need for additional ablation studies and results. However, after the rebuttal, they were satisfied with the additional results provided. All reviewers acknowledged the simplicity and straightforwardness of the method. The AC also reviewed the paper, the feedback, and the rebuttal, and similarly recognized the method as both simple and effective. Therefore, the AC recommends acceptance. The paper would benefit from including additional results and discussions in the final version.