Segmentation-Enhanced Depth Estimation Using Camera Model Based Self-supervised Contrastive Learning

Authors: jinchang zhang, Guoyu Lu
Primary area: applications to robotics, autonomy, planning
Keywords: Contrastive Learning; Depth Estimation

Abstract:
Depth estimation is a key topic in the field of computer vision. Self-supervised monocular depth estimation offers a powerful method to extract 3D scene information from a single camera image, allowing training on arbitrary image sequences without the need for depth labels. However, monocular unsupervised depth estimation still cannot address the issue of scale and often requires ground truth for calibration.
In the deep learning era, existing methods primarily rely on relationships between images to train unsupervised neural networks, often overlooking the fundamental information provided by the camera itself. In fact, the intrinsic and extrinsic parameters of the camera can be used to compute depth information for the ground and its related areas based on physical principles. This information can offer rich supervisory signals at no additional cost. Additionally, by assuming that objects like people, cars, and buildings share the same depth as the corresponding ground, the physical depth of the entire scene can be inferred, and gaps in the depth map can be filled.
Since some areas may have depth estimation errors, to make full use of these regions, we introduce a contrastive learning self-supervised framework. This framework consists of two networks with the same structure: the Anchor network and the Target network. While calculating depth, the network also outputs semantic segmentation results to assist in computing the physics depth, which is then used as the label for the model. Semantic segmentation can identify dynamic objects, reducing photometric reprojection errors caused by moving objects. The predictions from the Anchor network are used as pseudo-labels for training the Target network. Reliability is determined by entropy, dividing the predicted depth into positive and negative samples to maximize the use of physics depth information.

Decision: reject
Reviewer scores: 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730265674302

thread_type:
official_review

<thread_object>
creator: reviewer_gkxx

<content>
summary:
The paper proposes a new training pipeline for depth estimation which consists of two stages, i.e., one for contrastive learning and the other for generating physics-based depth with semantic labels. The authors leverage both intrinsic and extrinsic camera parameters with semantic segmentation task to estimate where the flat area (ground, road) is in the real-world coordinate. With this physics depth, it is used as an auxiliary self-supervised depth labels along with monocular video to improve the performance. Also, similar to MoCo, EMA-based update is adopted in contrastive learning pipeline. The proposed method achieves state-of-the-art results.

strengths:
1. The paper is straightforward and easy to follow.
2. The paper proposes a way to utilize both intrinsic camera parameters with semantic labels to estimate the flat area where the depth estimation model can use this value as additional depth labels.
3. The paper shows the contrastive learning can be also useful in depth estimation pipeline.

weaknesses:
1. As the main contribution is to use physics-based depth as self-supervised depth labels with semantic segmentation labels, the authors need to compare the other self-supervised methods which also leverages semantic labels as additional cues for depth estimation.

questions:
1. How do you train semantic segmentation?
2. Is the power really from physics model? I suggest the authors to decouple semantic segmentation from depth such that the author do additional experiment by using different encoder and decoder for depth estimation and semantic segmentation, then use only semantic labels for physics depth.
3. Compare it using GT, large-scale semantic segmentation network like SAM, and your proposed semantic segmentation network.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730339928082

thread_type:
official_review

<thread_object>
creator: reviewer_4ys9

<content>
summary:
This paper proposes a camera model that combines image semantics with the physical model of the camera to compute the depth information of road surfaces. On this basis, the depth of ground objects is inferred, and a dense depth map is generated by filling in the missing points, which is then used as training data for a pseudo-labeled supervised model. Furthermore, a self-supervised network framework based on contrastive learning is designed to effectively leverage the physics-based depth information. Unreliable predictions are treated as negative samples for the least likely classes, and a positive-negative sample contrastive learning strategy is proposed to maximize the use of depth information computed directly from the camera model.

In addition, the network presented in this paper outputs semantic segmentation, which not only aids in the computation of physical depth but also identifies dynamic objects. The segmentation-guided photometric reprojection loss effectively reduces errors caused by moving objects, further enhancing the accuracy of depth estimation. Tests on the physical depth estimation results show that, compared with LiDAR depth, the error of the physics-based depth is within an acceptable range, demonstrating the strong potential of physical depth to replace the scale factor in self-supervised monocular depth estimation.

Experiments on various depth estimation datasets demonstrate that the proposed method outperforms existing self-supervised approaches, and the effectiveness of each proposed module is verified through ablation experiments.

strengths:
The originality of this paper is highlighted by leveraging the intrinsic and extrinsic parameters of the camera to compute depth information of the ground and surrounding areas based on physical principles. This depth information is then used to generate dense depth maps by filling in missing points, which serve as pseudo-labeled data for supervised model training. Furthermore, a self-supervised network framework based on contrastive learning is designed to effectively utilize the physics-based depth information, enhancing the overall model performance.

weaknesses:
The related work section on depth estimation methods in Section II mainly reviews some relatively early depth estimation approaches and lacks coverage of recent self-supervised depth estimation techniques. This omission makes it difficult to clearly illustrate the advantages of the proposed method compared to more recent advancements. 

Moreover, the titles of Sections 5.1 and 5.2 are not clearly differentiated in terms of meaning and may lead to reader confusion. It is recommended to either revise these titles to make them more indicative of the specific experiments or consider combining them into a single section if appropriate.

Additionally, there are some formatting issues that require attention: Figure 4 is missing in Section 4.2, and there are symbol formatting errors in Equations 16, 21, and 22. It is important to address these details to enhance the overall quality and readability of the paper.

questions:
Overall, the algorithm design and experimental procedures in this paper are presented relatively clearly; however, some details require further clarification and refinement. Specifically, it is recommended to provide a more comprehensive introduction to the related work section, particularly regarding self-supervised depth estimation. Additionally, careful attention should be given to the wording and formatting throughout the paper to ensure consistency and clarity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730805741432

thread_type:
official_review

<thread_object>
creator: reviewer_n8ue

<content>
summary:
This paper presents a physic-driven monocular depth estimation system on outdoor autonomous driving scenes. By assuming the camera height is known, authors derives the grounding depth by leveraging the camera model (intrinsics), and designs supervision losses accordingly to supervise the training framework. To fully utilize every pixel's prediction, authors further design a contrastive learning module to supervise other less-reliable regions. Extensive experiments are conducted on KITTI and Cityscape datasets.

strengths:
1. The use of the camera model is technically sound and I appreciate the effort made from authors on deriving the depth over ground pixels leveraging the camera intrinsic.
2. Experiments are conducted in a comprehensive way to validate the proposed modules.

weaknesses:
1. To me the major concern of this paper is the presentation. The motivation of the method should be presented in a clearer way in both abstract and introduction. The methodology are mixed together in a single paragraph instead of presenting them logically. Besides, there are notable mistakes on Figure 1: decoded segmentation and depth map are evidently swapped. These issues cannot convince me that the paper is finished in a good shape.

2. The advantage over other methods are not significant, especially at Table 2, 4, 6. Most of the vehicles are mounted with Lidars so Lidar points are easy to obtain, alleviating the significance of leveraging ground pixels to predict the scale factor. In addition, this paper leverages groundtruth segmentation to supervise the training, whereas other methods did not explicitly use such information. Thus I assume this is not a fair comparison.

3. The ablation study of Physic Depth is missing. This is the key contribution of the proposed system, thus removing it and keeping other parts are strongly encouraged to verify its significance.

questions:
Please try to address my concerns listed in the weakness part. Due to notable mistakes in the paper's presentation and the unconvincing technical impact, I lean towards reject this paper for now. But I am also open to discussing with other reviewers, and will refer to their valuable comments.
</content>
</thread_object>

</thread>


Metareview: