Depth Pro: Sharp Monocular Metric Depth in Less Than a Second

Authors: Alexey Bochkovskiy, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan Richter, Vladlen Koltun
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: depth estimation, computer vision

Abstract:
We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image. Extensive experiments analyze specific design choices and demonstrate that Depth Pro outperforms prior work along multiple dimensions. We release code & weights at https://github.com/apple/ml-depth-pro

Decision: accept (poster)
Reviewer scores: 6, 10, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1729354938026

thread_type:
official_review

<thread_object>
creator: reviewer_85gd

<content>
summary:
This paper presents a novel framework for high-resolution metric depth estimation, demonstrating both excellent organization and a well-articulated motivation. The qualitative outcomes are impressive, and the quantitative results also demonstrate the state-of-the-art performance. The authors assert four main contributions: (1) an innovative network architecture, (2) a newly proposed evaluation metric for boundary tracing, (3) newly designed loss functions, and (4) the capacity to estimate camera focal length.

strengths:
- This paper is well-written and motivated.
- The proposed method demonstrates impressive performance.
- It addresses the metric monocular depth estimation problem, a crucial task with significant application prospects.
- The method estimates camera focal length, eliminating the need for image metadata—a practical advancement.
- The experiments detailed in the main text and appendix are comprehensive and solid.

weaknesses:
There are some minor issues:
- There are a few unclear statements regarding Figure 3:
  - Along which dimension does the concatenation operation perform?
  - What is the specific implementation of the Merge operation?
- At Line 234, the statement "produces a feature tensor at resolution 24 × 24 per input patch (features 3 – 6 in Fig. 3)" seems to not coincide with the figure label.

questions:
- Marigold and Depth Anything have demonstrated that using synthetic training data can effectively improve details in depth prediction. Depth Pro also uses synthetic data to enhance high-frequency details in the depth map, yet its predicted boundaries are significantly better than those of existing methods. What is the main reason for this?significantly better than the existing methods. What is the main reason for this?
- Comparative methods such as Depth Anything and Marigold are trained with low-resolution images, while Depth Pro is trained with high-resolution images. Is this the reason why Depth Pro achieves better detail in depth prediction?
- According to Table 13, Depth Pro appears to use more training data than other competitors. Can you compare the scale of training data used by Depth Pro with that of other competitors?
- According to Table 1, Depth Pro has a significant gap with the best performing methods on some datasets (such as ETH3D). Can you analyze the reasons for this discrepancy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730534902984

thread_type:
official_review

<thread_object>
creator: reviewer_azew

<content>
summary:
The paper introduces Depth Pro, a model for zero-shot metric monocular depth estimation that generates high-resolution depth maps with precise boundary details at sub-second processing speeds. Key innovations include a multi-scale vision transformer architecture, a mixed training regimen with real and synthetic data for improved accuracy, and a dedicated evaluation approach for boundary sharpness. The model also features state-of-the-art focal length estimation without the need for camera metadata. Depth Pro outperforms prior approaches in accuracy and efficiency, making it suitable for applications like novel view synthesis.

strengths:
- The model demonstrates strong zero-shot performance for metric depth estimation, effectively handling diverse input images without fine-tuning.
- It achieves superior focal length estimation accuracy, which enhances depth map scaling and generalization across different image conditions.
- Comprehensive ablation studies are provided to analyze the impact of key architectural and training choices, showcasing the robustness of the proposed method.
- The experiments are conducted on a wide range of datasets, underscoring the model’s effectiveness and applicability across various scenarios.
- Detailed training procedures are described, allowing for reproducibility and providing insights into the model’s development process.
- The method exhibits higher edge accuracy compared to conventional techniques, maintaining competitive inference times.
- It outperforms state-of-the-art approaches, such as DepthAnything, DepthAnything v2, and Marigold, with clear comparative analysis highlighting these advancements.

weaknesses:
The most critical components that influence performance and boundary accuracy among various technical contributions are not clearly highlighted, leaving ambiguity about which elements contribute most to the model's success.

questions:
1. If the loss function were designed using depth instead of canonical inverse depth, would the results differ? Have you conducted any experiments to compare these approaches?
2. Among the various technical contributions (e.g., curriculum learning, focal length estimation, and the patch encoder with multiple scales), which components most significantly affect the overall performance metrics (e.g., abs_rel, δ_1) and boundary accuracy, respectively? Although multiple technical contributions are mentioned, it is unclear which are the most impactful.
3. Is there any tradeoff between overall performance and boundary accuracy related to certain hyperparameters in the deep neural network or the training loss?
4. For calculating the final depth, does the proposed method use the predicted inverse depth C, the predicted focal length f_px, and the ground-truth width w, as in the equation D = f_px / (wC)?
5. As I understand it, all experiments were conducted with evaluation using metric scale depth, without median-scaling evaluation. Is my understanding correct?
6. Have you conducted experiments on fine-tuning the proposed model (DepthPro)? If so, is there a significant performance improvement between the zero-shot setting and fine-tuning?
7. What is the training time and GPU resources required to train the proposed method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730538173568

thread_type:
official_review

<thread_object>
creator: reviewer_a8yt

<content>
summary:
The proposed Depth Pro model employs a ViT architecture for zero-shot metric monocular depth estimation, targeting applications such as novel view synthesis. By employing patch-based, multi-scale processing, Depth Pro achieves high-resolution depth predictions while preserving real-time performance and edge sharpness. This paper introduces a two-stage training approach that integrates synthetic and real-world datasets, enhancing depth boundary accuracy. Furthermore, the model includes boundary evaluation metrics and a focal length estimation component, improving its robustness in handling images lacking metadata.

strengths:
1. Structured Training Curriculum: The curriculum is specifically tailored to handle the unique strengths and weaknesses of synthetic and real-world datasets, resulting in a model that generalizes effectively while producing fine boundary details in depth maps.

2. Enhanced Boundary Evaluation Metrics: New metrics for evaluating depth boundaries address a gap in existing benchmarks by focusing on boundary precision, which is critical for applications like view synthesis that demand fine details.

weaknesses:
1. Dependence on pretrained ViT with Limited Architectural Innovation: Depth Pro benefits from pretrained ViT backbones, yet its architecture primarily builds on existing elements rather than introducing fundamentally new mechanisms for depth estimation, which limits its architectural novelty.

2. Heavy Reliance on Synthetic Data for Boundary-Sensitive Training: The model’s second-stage training emphasizes synthetic datasets for boundary sharpness. This reliance on synthetic data could impact generalization to real-world environments, particularly in complex or unstructured settings where boundaries are less distinct.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730622207077

thread_type:
official_review

<thread_object>
creator: reviewer_el31

<content>
summary:
This paper introduces a foundational model for single-image depth estimation, trained on a large collection of datasets. The approach results in high-resolution depth maps with sharp object boundaries and enables the estimation of camera intrinsic parameters. The model outputs metric-scale depth predictions and demonstrates its effectiveness across multiple datasets, outperforming previous methods in both quantitative and qualitative evaluations. The contributions of this work are significant, with the pre-trained model holding substantial potential for a range of downstream tasks.

strengths:
1. High-Resolution Predictions: The model produces depth maps at resolutions as high as 1536 x 1536, attributed to its multi-scale vision transformer architecture.

2. Comprehensive Training: The use of extensive depth estimation datasets for training and testing demonstrates substantial effort in data processing. The pre-trained model is valuable to the community and holds potential for multiple downstream tasks.

3. Camera Intrinsic Estimation: The model can estimate camera intrinsic parameters and shows strong zero-shot generalization due to its training on diverse datasets, which is crucial for real-world scenarios where camera calibration is challenging.

4. Visual and Quantitative Superiority: The qualitative results clearly show better object boundary preservation compared to previous methods, and the quantitative evaluations across datasets highlight the model's excellent zero-shot performance. The evaluation on downstream tasks like image synthesis showcases the model’s adaptability.

weaknesses:
1. Limited Novelty: Although effective, the proposed method's novelty is somewhat constrained as it leverages existing approaches. The loss functions and training strategy draw heavily from DPT (Ranftl et al., 2022), and the metric depth estimation follows ideas from Metric3D (Yin et al., 2023). More discussion on the unique aspects would strengthen the paper.

2. Depth Consistency in Videos: While single-image results are impressive, depth consistency across video frames is not explored. Given the model’s metric-scale depth predictions, it would be valuable to assess its utility in video-based tasks such as visual SLAM, camera pose estimation, and 3D reconstruction.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730844829485

thread_type:
official_review

<thread_object>
creator: reviewer_yfuy

<content>
summary:
The paper introduces a new approaches for estimating a depth map from a single image. The paper claims many advantages of the method over competing approaches, in particular the ability to estimate absolute scale under all conditions (i.e., without metadata), high resolution, and speed. The approach includes a transformer-based architecture, hybrid training approach, and focal length estimation to address scale issues. Extensive experimental results are presented.

strengths:
- Important task (monocular depth estimation)
- impressive visualizations (e.g. figure 1)
- Achieves an averaged best performance on multiple datasets and demonstrated great generalization for open-world images.
- Excels  in distinguish boundaries
- Evaluation across a large number of data sets/ablations. Of particular note is the difficulty of avoid cross-data set contamination (test samples appearing in training, etc.) and restricted access.

weaknesses:
- Method section is not clear enough. Missing model details.
- How are features 3-6 obtained since the images are splits into {1x,3x,5x} 384^2 and none of them matches the feature resolution at 48^2 and 96^2?
- what's the architecture of the patch encoder?
- What's the detailed architecture of the focal length head? How does this module fuse both multi-scale features and the features from the extra "image encoder for local length"? The role of the this estimation piece is critical in understanding how it is possible for the method to achieve absolute depth estimation even when camera parameters are absent but the description is lacking.
- Overall, the writing of this paper is poor. It's hard to follow both the proposed method and the experimental evaluation though the results are impressive.

questions:
- the proposed method excels in detecting boundaries. Do the authors have a sense why this happens? Due to the multi-scale derivative objective? Ablations about the reason are encouraged.
- Fig. 4 is only mentioned in l457-458. No details are provided unless in their caption. It's hard to understand how DepthPro helps novel view synthesis.
- How does the proposed model handle images of an arbitrary scale?
- One claim of this paper is the efficient inference. Running time numbers should be reported. Currently they are only demonstrated in Figure 1.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732735966932

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their feedback!

We revised the paper accordingly. As multiple reviewers asked for clarification on the effect of individual contributions on performance, we provide an answer here and respond to other questions after the respective reviews. 

**Effect of contributions on performance**. We thank reviewers yFUY and AZEW for their suggestions. The increase in boundary accuracy over prior work can be attributed predominantly to the high output resolution, the loss functions, and the architecture. To better disentangle these factors, we added boundary metrics computed on the validation set of Hypersim to our controlled experiments and conducted an experiment on different output resolutions.

* Tab. 11 demonstrates that our proposed losses for stage 1 (config 1C) double the accuracy on boundaries in comparison to a standard mean absolute error on metric datasets (config 1A). The losses in stage 2 are chosen to balance metric depth accuracy and boundary accuracy (see Tab. 12). The same rationale applies to the training curriculum in Tab. 13 - training with just a single stage (config 3B) results in even sharper boundaries than our proposed configuration 3A but yields less accurate depth predictions on average.
* In Tab. 9 we compare our network architecture to alternatives that also predict at 1536^2. We find that transformer-based architectures fare better than convolutional architectures and among the transformers, ViT-based architectures perform best. Notably, our patch-based processing increases boundary recall on DIS5K by ~20% over scaling up the DINOv2 architecture, and more than 2.8x in comparison to e.g. SWINv2. 
* Finally, to assess the effect of the output resolution on UnrealStereo4K by downsampling the ground truth to a specific resolution, upsampling again to the full resolution, and evaluating against the unmodified ground truth. This experiment provides an approximate upper bound on metric depth and boundary accuracy a method could achieve if it predicted the perfect ground truth at a specific resolution. We observe that the effect on boundary accuracy is ~3x per doubling of image dimensions. While this result suggests that predicting at high resolution is a necessary condition for high-quality boundaries, it does not present a sufficient condition. This is demonstrated by the comparison of Depth Pro against variable-size approaches like ZeroDepth and PatchFusion, which predict at a higher output resolution but fare worse in boundary metrics.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received mostly positive ratings from reviewers. The concerns raised by reviewers were effectively addressed in the rebuttal, leading to general satisfaction with the authors’ responses. Reviewer yFUY raised issues regarding poor writing, unclear technical details, and insufficient runtime comparison. The authors provided more detailed explanations of technical aspects and clarified runtime comparisons in their response. While these improvements addressed the reviewer’s concerns, the area chair encourages the authors to further enhance the clarity and quality of the writing in the final version. Reviewers eL31 and a8Yt expressed concerns about the limited novelty of the work. The authors clarified the contributions of their work in the rebuttal, particularly emphasizing the innovative aspects of their approach. Reviewer AZEW noted that the critical components influencing performance and boundary accuracy among the technical contributions were not clearly highlighted. The authors included additional ablation studies in their rebuttal, which helped clarify the importance of specific components. The reviewer acknowledged that their concerns were addressed. Reviewer 85gd requested a better explanation for the strong performance of the proposed method. The authors provided a more comprehensive explanation of the factors contributing to the method’s performance, which satisfied the reviewer. Given the authors’ effective rebuttal and the reviewers’ satisfaction with the addressed concerns, this paper makes a valuable contribution to the field and is recommended for acceptance.