Depth Pro: Sharp Monocular Metric Depth in Less Than a Second

## Abstract

Abstract We present a foundation model for zero-shot metric monocular depth estimation. Our model, Depth Pro, synthesizes high-resolution depth maps with unparalleled sharpness and high-frequency details. The predictions are metric, with absolute scale, without relying on the availability of metadata such as camera intrinsics. And the model is fast, producing a 2.25-megapixel depth map in 0.3 seconds on a standard GPU. These characteristics are enabled by a number of technical contributions, including an efficient multi-scale vision transformer for dense prediction, a training protocol that combines real and synthetic datasets to achieve high metric accuracy alongside fine boundary tracing, dedicated evaluation metrics for boundary accuracy in estimated depth maps, and state-of-the-art focal length estimation from a single image. Extensive experiments analyze specific design choices and demonstrate that Depth Pro outperforms prior work along multiple dimensions. We release code & weights at https://github.com/apple/ml-depth-pro

## 1 Introduction

Figure: Figure 1: Results on images from the AM-2k (Li et al., ) (1st & 3rd column) and DIS-5k (Qin et al., ) (2nd column) datasets. Input image on top, estimated depth maps from Depth Pro, Marigold (Ke et al., ), Depth Anything v2 (Yang et al., ), and Metric3D v2 (Hu et al., ) below. Depth Pro produces zero-shot metric depth maps with absolute scale at 2.25-megapixel native resolution in 0.3 seconds on a V100 GPU.
Refer to caption: fig/dis5k/4892828841_7f1bc05682_o.jpg

Zero-shot monocular depth estimation underpins a growing variety of applications, such as advanced image editing, view synthesis, and conditional image generation. Inspired by MiDaS (Ranftl et al., ) and many follow-up works (Ranftl et al., ; Ke et al., ; Yang et al., ; Piccinelli et al., ; Hu et al., ), applications increasingly leverage the ability to derive a dense pixelwise depth map for any image.

Our work is motivated in particular by novel view synthesis from a single image, an exciting application that has been transformed by advances in monocular depth estimation (Hedman et al., ; Shih et al., ; Jampani et al., ; Khan et al., ).
Applications such as view synthesis imply a number of desiderata for monocular depth estimation. First, the depth estimator should work zero-shot on any image, not restricted to a specific domain (Ranftl et al., ; Yang et al., ). Furthermore, the method should ideally produce *metric* depth maps in this zero-shot regime, to accurately reproduce object shapes, scene layouts, and absolute scales (Guizilini et al., ; Hu et al., ). For the broadest applicability ‘in the wild’, the method should produce metric depth maps with absolute scale even if no camera intrinsics (such as focal length) are provided with the image (Piccinelli et al., ). This enables view synthesis scenarios such as “Synthesize a view of this scene from 63 mm away” for essentially arbitrary single images (Dodgson, ).

Second, for the most compelling results, the monocular depth estimator should operate at high resolution and produce fine-grained depth maps that closely adhere to image details such as hair, fur, and other fine structures (Miangoleh et al., ; Ke et al., ; Li et al., ).
One benefit of producing sharp depth maps that accurately trace intricate details is the elimination of “flying pixels”, which can degrade image quality in applications such as view synthesis (Jampani et al., ).

Third, for many interactive application scenarios, the depth estimator should operate at low latency, processing a high-resolution image in less than a second to support interactive view synthesis “queries” on demand. Low latency is a common characteristic of methods that reduce zero-shot monocular depth estimation to a single forward pass through a neural network (Ranftl et al., ; Yang et al., ; Piccinelli et al., ),   but it is not always shared by methods that employ more computationally demanding machinery at test time (Ke et al., ; Li et al., ).

Figure: Figure 2: Boundary recall versus runtime. Depth Pro outperforms prior work by a multiplicative factor in boundary accuracy while being orders of magnitude faster than works focusing on fine-grained predictions (e.g., Marigold, PatchFusion).
Refer to caption: x1.png

In this work, we present a foundation model for zero-shot metric monocular depth estimation that meets all of these desiderata. Our model, Depth Pro, produces metric depth maps with absolute scale on arbitrary images ‘in the wild’ without requiring metadata such as camera intrinsics. It operates at high resolution, producing 2.25-megapixel depth maps (with a native output resolution of $1536\times 1536$ before optional further upsampling) in 0.3 seconds on a V100 GPU. Fig. 1 shows some representative results. Depth Pro dramatically outperforms all prior work in sharp delineation of object boundaries, including fine structures such as hair, fur, and vegetation. As shown in Fig. 2, Depth Pro offers unparalleled boundary tracing, outperforming all prior work by a multiplicative factor in boundary recall. (See Sec. 4 for additional detail.) Compared to the prior state of the art in boundary accuracy (Ke et al., ; Li et al., ), Depth Pro is one to two orders of magnitude faster, yields much more accurate boundaries, and provides metric depth maps with absolute scale.

Depth Pro is enabled by a number of technical contributions.
First, we design an efficient multi-scale ViT-based architecture for capturing the global image context while also adhering to fine structures at high resolution.
Second, we derive a new set of metrics that enable leveraging highly accurate matting datasets for quantifying the accuracy of boundary tracing in evaluating monocular depth maps.
Third, we devise a set of loss functions and a training curriculum that promote sharp depth estimates while training on real-world datasets that provide coarse and inaccurate supervision around boundaries, along with synthetic datasets that offer accurate pixelwise ground truth but limited realism.
Fourth, we contribute zero-shot focal length estimation from a single image that dramatically outperforms the prior state of the art.

## 2 Related work

Early work on monocular depth estimation focused on training on individual datasets recorded with a single camera (Saxena et al., ; Eigen et al., ; Eigen & Fergus, ). Although this setup directly enabled metric depth predictions, it was limited to single datasets and narrow domains.

Zero-shot depth estimation.
MegaDepth (Li & Snavely, ) demonstrated that training on a diverse dataset allows generalizing monocular depth prediction beyond a specific domain. MiDaS (Ranftl et al., ) advanced this idea by training on a large mix of diverse datasets with a scale-and-shift-invariant loss. Follow-up works applied this recipe to transformer-based architectures (Ranftl et al., ; Birkl et al., ) and further expanded the set of feasible datasets through self-supervision (Spencer et al., ; Yang et al., ).
A line of work uses self-supervision to learn from unlabeled image and video data (Petrovai & Nedevschi, ; Yang et al., ). A number of recent approaches (Ke et al., ; Gui et al., ) harness diffusion models to synthesize relative depth maps.
Although some of these methods demonstrated excellent generalization, their predictions are ambiguous in scale and shift, which precludes downstream applications that require accurate shapes, sizes, or distances.

Zero-shot metric depth.
A line of work sought to improve metric depth prediction through a global distribution of depth values (Fu et al., ; Bhat et al., ; ; Li et al., ) and further conditioning on scene type (Bhat et al., ).
A different approach directly takes into account camera intrinsics. Cam-Convs (Fácil et al., ) conditioned convolutions on the camera intrinsics. LeReS (Yin et al., ) trains a separate network for undistorting point clouds to recover scale and shift, Metric3D (Yin et al., ) scales images or depth maps to a canonical space and remaps estimated depth given the focal length, and ZeroDepth (Guizilini et al., ) learns camera-specific embedddings in a variational framework. DMD (Saxena et al., ) conditions a diffusion model on the field of view. Metric3D v2 (Hu et al., ) leverages surface normals as an auxilliary output to improve metric depth. All of these methods require the camera intrinsics to be known and accurate. More recent works attempt to reason about unknown camera intrinsics either through a separate network (Spencer et al., ) or by predicting a camera embedding for conditioning its depth predictions in a spherical space (Piccinelli et al., ).
Akin to these recent approaches, our method does not require the focal length to be provided as input.
We propose to directly estimate the field of view from intermediate features of the depth prediction network, and show that this substantially outperforms the prior state of the art in the task of cross-domain focal length estimation.

Sharp occluding contours.
SharpNet (Ramamonjisoa & Lepetit, ) incorporates normals and occluding contour constraints, but requires additional contour and normal supervision during training.
BoostingDepth (Miangoleh et al., ) obtains detailed predictions from a low-resolution network by applying it independently to image patches. Since the patches lack global context, BoostingDepth fuses them through a sophisticated multi-step pipeline.
PatchFusion (Li et al., ) refines this concept through image-adaptive patch sampling and tailored modules that enable end-to-end training.
A recent line of work leverages diffusion priors to enhance the sharpness of occlusion boundaries (Gui et al., ; Ke et al., ). These approaches predominantly focus on predicting relative (rather than metric) depth.
We propose a simpler architecture without task-specific modules or diffusion priors and demonstrate that even sharper and more accurate results can be obtained while producing metric depth maps and reducing runtime by more than two orders of magnitude.

Guided depth super-resolution uses the input image to upsample low-resolution depth predictions (Metzger et al., ; Zhong et al., ). SMDNet (Tosi et al., ) predicts bimodal mixture densities to sharpen occluding contours. And Ramamonjisoa et al. (Ramamonjisoa et al., ) introduce a module for learning to sharpen depth boundaries of a pretrained network.
These works are orthogonal to ours and could be applied to further upsample our high-resolution predictions.

To evaluate boundary tracing in predicted depth maps, Koch et al. () introduce the iBims dataset with manual annotations of occluding contours and corresponding metrics.
The need for manual annotation and highly accurate depth ground truth constrain the benchmark to a small set of indoor scenes.
We contribute metrics based on segmentation and matting datasets that provide a complementary view by enabling evaluation on complex, dynamic environments or scenes with exceedingly fine detail for which ground-truth depth is impossible to obtain.

Multi-scale vision transformers.
Vision transformers (ViTs) have emerged as the dominant general-purpose architecture for perception tasks, but operate at low resolution (Dosovitskiy et al., ). Naïvely scaling the architecture to higher resolutions is prohibitive due to the computational complexity. Several works identified the attention layers as the main obstacle to scaling up ViT and have proposed alternatives (Zhu et al., ; Liu et al., ; Li et al., ; Chu et al., ; Liu et al., ; ; Cai et al., ; Jaegle et al., ).

Another line of work modified the ViT architecture to produce a hierarchy of features (Fan et al., ; Xie et al., ; Yuan et al., ; Ranftl et al., ; Chen et al., ; Lee et al., ).

Rather than modifying the ViT architecture, which requires computationally expensive retraining, we propose a network architecture that applies a plain ViT backbone at multiple scales and fuses predictions into a single high-resolution output. This architecture benefits from ongoing improvements in ViT pretraining, as new variants can be easily swapped in (Oquab et al., ; Peng et al., ; Sun et al., ).

Pretrained vision transformers have been adapted for semantic segmenation and object detection. ViT-Adapter (Chen et al., ) and ViT-CoMer (Xia et al., ) supplement a pretrained ViT with a convolutional network for dense prediction, whereas
ViT-Det (Li et al., ) builds a feature pyramid on top of a pretrained ViT.
Distinct from these, we fuse features from the ViT applied at multiple scales to learn global context together with local detail.

## 3 Method

### 3.1 Network

The key idea of our architecture is to apply plain ViT encoders (Dosovitskiy et al., ) on patches extracted at multiple scales and fuse the patch predictions into a single high-resolution dense prediction in an end-to-end trainable model. Fig. 3 illustrates the architecture.
For predicting depth, we employ two ViT encoders, a patch encoder and an image encoder. The patch encoder is applied on patches extracted at multiple scales.
Intuitively, this may allow learning representations that are scale-invariant as weights are shared across scales. The image encoder anchors the patch predictions in a global context. It is applied to the whole input image, downsampled to the base input resolution of the chosen encoder backbone (in our case 384$\times$384).

Figure: Figure 3: Overview of the network architecture. An image is downsampled at several scales. At each scale, it is split into patches, which are processed by a ViT-based patch encoder, with weights shared across scales. Patches are merged into feature maps, upsampled, and fused via a DPT decoder. Predictions are anchored by a separate image encoder that provides global context.
Refer to caption: x3.jpeg

The whole network operates at a fixed resolution of 1536$\times$1536, which was chosen as a multiple of the ViT’s 384$\times$384. This guarantees a sufficiently large receptive field and constant runtimes for any image while preventing out-of-memory errors (which we repeatedly observed for variable-resolution approaches on large images). Confirming this design choice, the results we report in Sec. 4 and Tab. 5 demonstrate that Depth Pro is consistently orders of magnitude faster than variable-resolution approaches while being more accurate and producing sharper boundaries.
A key benefit of assembling our architecture from plain ViT encoders over custom encoders is the abundance of pretrained ViT-based backbones that can be harnessed (Oquab et al., ; Peng et al., ; Sun et al., ). We evaluate several pretrained backbones and compare our architecture to other high-resolution architectures in the appendices (Tab. 8 and Sec. B.2).

After downsampling to 1536$\times$1536, the input image is split into patches of $384\times 384$. For the two finest scales, we let patches overlap to avoid seams. At each scale, the patches are fed into the patch encoder, which produces a feature tensor at resolution $24\times 24$ per input patch (features 3 – 6 in Fig. 3).
At the finest scale we further extract intermediate features (features 1 & 2 in Fig. 3) to capture finer-grained details.
We merge the feature patches into maps, which are fed into the decoder module, which resembles the DPT decoder (Ranftl et al., ).

In addition to sharing representations across scales, the patch-based application of the encoder network allows trivial parallelization as patches can be processed independently.
Another source of computational efficiency comes from the lower computational complexity of patch-based processing in comparison to scaling up the ViT to higher resolutions.
The reason is multi-head self-attention (Vaswani et al., ), whose computational complexity scales quadratically with the number of input pixels, and thus quartically in image dimension.

### 3.2 Sharp monocular depth estimation

#### Training objectives.

For each input image $I$, our network $f$ predicts a canonical inverse depth image $C=f(I)$. To obtain a dense metric depth map $D_{m}$, we scale by the horizontal field of view, represented by the focal length $f_{\mathit{px}}$ and the width $w$ (Yin et al., ): $D_{m}=\tfrac{f_{\mathit{px}}}{wC}.$

We train with several objectives, all based on canonical inverse depth, because this prioritizes areas close to the camera over farther areas or the whole scene, and thus supports visual quality in applications such as novel view synthesis.
Let $\hat{C}$ be the ground-truth canonical inverse depth.
For all metric datasets we compute the mean absolute error ($\mathcal{L}_{\mathit{MAE}}$, Eq. 1) per pixel $i$, and discard pixels with an error in the top $20\%$ per image for real-world (as opposed to synthetic) datasets:

$$ $\displaystyle\mathcal{L}_{\mathit{MAE}}(\hat{C},C)=\frac{1}{N}\sum^{N}_{i}|\hat{C}_{i}-C_{i}|.$ (1) $$

For all non-metric datasets (i.e., those without reliable camera intrinsics or inconsistent scale), we normalize predictions and ground truth via the mean absolute deviation from the median (Ranftl et al., ) before applying a loss. We further compute errors on the first and second derivatives of (canoncial) inverse depth maps at multiple scales.
Let $\nabla_{*}$ indicate a spatial derivative operator $*$, such as Scharr (S) (Scharr et al., ) or Laplace (L), and $p$ the error norm. We define the multi-scale derivative loss over $M$ scales as

$$ $\displaystyle\mathcal{L}_{*,p,M}(C,\hat{C})=\frac{1}{M}\sum^{M}_{j}\frac{1}{N_{j}}\sum_{i}^{N_{j}}|\nabla_{*}C^{j}_{i}-\nabla_{*}\hat{C}^{j}_{i}|^{p},$ (2) $$

where the scales $j$ are computed by blurring and downsampling the inverse depth maps by a factor of 2 per scale.
As shorthands we define the Mean Absolute Gradient Error $\mathcal{L}_{\mathit{MAGE}}=\mathcal{L}_{S,1,6}$, the Mean Absolute Laplace Error $\mathcal{L}_{\mathit{MALE}}=\mathcal{L}_{L,1,6}$, and the Mean Squared Gradient Error $\mathcal{L}_{\mathit{MSGE}}=\mathcal{L}_{S,2,6}$.

#### Training curriculum.

We propose a training curriculum motivated by the following observations.
First, training on a large mix of real-world and synthetic datasets improves generalization as measured by zero-shot accuracy (Ranftl et al., ; ; Yang et al., ; Hu et al., ).
Second, synthetic datasets provide pixel-accurate ground truth, whereas real-world datasets often contain missing areas, mismatched depth, or false measurements on object boundaries.
Third, predictions get sharper over the course of training.

Based on these observations, we design a two-stage training curriculum.
In the first stage, we aim to learn robust features that allow the network to generalize across domains.
To that end, we train on a mix of all labeled training sets.
Specifically, we minimize $\mathcal{L}_{\mathit{MAE}}$ on metric datasets and its normalized version on non-metric datasets. $\mathcal{L}_{\mathit{MAE}}$ is chosen for its robustness in handling potentially corrupted real-world ground truth. To steer the network towards sharp boundaries, we aim to also supervise on gradients of the predictions. Done naïvely, however, this can hinder optimization and slow down convergence. We found that a scale-and-shift-invariant loss on gradients, applied only to synthetic datasets, worked best. Controlled experiments are reported in the appendices.

The second stage of training is designed to sharpen boundaries and reveal fine details in the predicted depth maps. To minimize the effect of inaccurate ground truth, at this stage we only train on synthetic datasets that provide high-quality pixel-accurate ground truth. (Note that this inverts the common practice of first training on synthetic data and then fine-tuning on real data (Gaidon et al., ; Gómez et al., ; Sun et al., ).)
Specifically, we again minimize the $\mathcal{L}_{\mathit{MAE}}$ and supplement it with a selection of losses on the first- and second-order derivatives: $\mathcal{L}_{\mathit{MAGE}}$, $\mathcal{L}_{\mathit{MALE}}$, and $\mathcal{L}_{\mathit{MSGE}}$.
We provide a detailed specification of the loss functions that are applied at each stage in the appendices.

#### Evaluation metrics for sharp boundaries.

Applications such as novel view synthesis require depth maps to adhere to object boundaries. This is particularly challenging for thin structures. Misaligned or blurry boundaries can make objects appear distorted or split into parts. Common benchmarks for monocular depth prediction rarely take boundary sharpness into account. This may be attributed in part to the lack of diverse and realistic datasets with precise pixel-accurate ground-truth depth. To address this shortcoming, we propose a new set of metrics specifically for the evaluation of depth boundaries. Our key observation is that we can leverage existing high-quality annotations for matting, saliency, or segmentation as ground truth for depth boundaries. We treat annotations for these tasks as binary maps, which define a foreground/background relationship between an object and its environment. (This relationship may not hold in every case, especially for segmentation masks. However, we can easily discard such problematic cases through manual inspection. It is much easier to filter out a segmentation mask than to annotate it.) To ensure that the relationship holds, we only consider pixels around edges in the binary map.

We first define the metrics for depth maps and later derive the formulation for binary segmentation masks.
Motivated by the ranking loss (Chen et al., ), we use the pairwise depth ratio of neighboring pixels to define a foreground/background relationship.
Let $i,j$ be the locations of two neighboring pixels. We then define an occluding contour $c_{d}$ derived from a depth map $d$ as $c_{d}(i,j)=\left[\tfrac{d(j)}{d(i)}>(1+\tfrac{t}{100})\right]$, where $[\cdot]$ is the Iverson bracket. Intuitively, this indicates the presence of an occluding contour between pixels $i$ and $j$ if their corresponding depth differs by more than $t\%$.
For all pairs of neighboring pixels, we can then compute the precision ($P$) and recall ($R$) as

$$ $\text{P}(t)=\frac{\sum_{i,j\in N(i)}c_{d}(i,j)\wedge c_{\hat{d}}(i,j)}{\sum_{i,j\in N(i)}c_{d}(i,j)}\text{ and }\text{R}(t)=\frac{\sum_{i,j\in N(i)}c_{d}(i,j)\wedge c_{\hat{d}}(i,j)}{\sum_{i,j\in N(i)}c_{\hat{d}}(i,j)}.$ (3) $$

Note that both $P$ and $R$ are scale-invariant. In our experiments, we report the F1 score.
To account for multiple relative depth ratios,
we further perform a weighted averaging of the F1 values with thresholds that range linearly from $t_{min}=5$ to $t_{max}=25$,
with stronger weights towards high threshold values.
Compared to other edge-based metrics (such as the edge accuracy and completion from iBims (Koch et al., )), our metric does not require any manual edge annotation,
but simply pixelwise ground truth, which is easily obtained for synthetic datasets.

Similarly, we can also identify occluding contours from binary label maps that can be derived from real-world segmentation, saliency, and matting datasets.
Given a binary mask $b$ over the image, we define the presence of an occluding contour $c_{b}$ between pixels $i,j$ as $c_{b}(i,j)=b(i)\land\lnot b(j)$.
With this definition at hand, we compute the recall $\text{R}(t)$ by replacing the occluding contours from depth maps in Eq. 3 with those from binary maps. Since the binary maps commonly label whole objects, we cannot obtain ground-truth occluding contours that do not align with object silhouettes. Thus the boundary annotation is incomplete – some but not all occluding contours are identified by this procedure. Therefore we can only compute the recall but not the precision for binary maps.

To penalize blurry edges, we suppress non-maximum values of
$c_{\hat{d}}$ within the valid bounds of $c_{\hat{d}}(i,j)$ connected components. For additional experiments and qualitative results we refer to the appendices.

### 3.3 Focal length estimation

To handle images that may have inaccurate or missing EXIF metadata, we supplement our network with a focal length estimation head. A small convolutional head ingests frozen features from the depth estimation network and task-specific features from a separate ViT image encoder to predict the horizontal angular field-of-view. We use $\mathcal{L}_{2}$ as the training loss. We train the focal length head and the ViT encoder after the depth estimation training. Separating the focal length training has several benefits over joint training with the depth network. It avoids the necessity of balancing the depth and focal length training objectives. It also allows training the focal length head on a different set of datasets, excluding some narrow-domain single-camera datasets that are used in training the depth estimation network, and adding large-scale image datasets that provide focal length supervision but no depth supervision. Further details are provided in the appendices.

## 4 Experiments

This section summarizes the key results. Additional details and experiments are reported in the appendices, including details on datasets, hyperparameters, experimental protocols, and the comparison of runtimes, which is summarized in Fig. 2. The appendices also report controlled experiments, including controlled studies on network architectures, training objectives, and training curricula.

Here we summarize a number of key comparisons of Depth Pro to state-of-the-art metric monocular depth estimation systems.
One challenge in conducting such a comparison is that many leading recent systems are trained on bespoke combinations of datasets. Some systems use proprietary datasets that are not publicly available, and some use datasets that are only available under restrictive licenses. Some recent systems also train on unlabeled datasets or incorporate pretrained models (e.g., diffusion models) that were trained on additional massive datasets.
This rules out the possibility of a comparison that controls for training data (e.g., only comparing to systems that use the same datasets we do). At this stage of this research area, the only feasible comparison to other leading cross-domain monocular depth estimation models is on a full system-to-system basis. Fully trained models (each trained on a large, partially overlapping and partially distinct collection of datasets) are compared to each other zero-shot on datasets that none of the compared systems trained on.

Zero-shot metric depth.
We evaluate our method’s ability to predict zero-shot *metric* depth and compare against the state of the art in Tab. 1.
Our baselines include Depth Anything (Yang et al., ), Metric3D (Yin et al., ), PatchFusion (Li et al., ), UniDepth (Piccinelli et al., ), ZeroDepth (Guizilini et al., ) and ZoeDepth (Bhat et al., ). We also report results for the very recent Depth Anything v2 (Yang et al., ) and Metric3D v2 (Hu et al., ).

As an overall summary measure of metric depth accuracy, Tab. 1 uses the $\delta_{1}$ metric (Ladicky et al., ), which is commonly used for this purpose (Yin et al., ; Yang et al., ; Piccinelli et al., ). It is defined as the percentage of inlier pixels, for which the predicted and ground-truth depths are within 25% of each other. We picked this metric for its robustness, with the strictest threshold found in the literature ($25\%$).

Corresponding tables for additional metrics can be found in Sec. A.2 of the appendices, including $\mathit{AbsRel}$ (Ladicky et al., ), $\mathit{Log}_{10}$, $\delta_{2}$ and $\delta_{3}$ scores, as well as point-cloud metrics (Spencer et al., ).
Tab. 1 also reports the average rank of each method across datasets, a common way to summarize cross-dataset performance (Ranftl et al., ).

We report results on Booster (Ramirez et al., ), Middlebury (Scharstein et al., ), Sun-RGBD (Song et al., ), ETH3D (Schöps et al., ), nuScenes (Caesar et al., ), and Sintel (Butler et al., ), because, to our knowledge, they were never used in training any of the evaluated systems.
Despite our best efforts, we were not able to run ZeroDepth on Booster, Middlebury, or Sun-RGBD as it consistently ran out of memory due to the high image resolutions. More details on our evaluation setup can be found in Sec. C of the appendix.

The results in Tab. 1 confirm the findings of Piccinelli et al. (), who observed considerable domain bias in some of the leading metric depth estimation models.
Notably, Depth Anything v1 & v2 focus on *relative* depth estimation; for metric depth, they provide different models for different domains, fine-tuned either for indoor or for outdoor scenes. Metric3D v1 & v2 provide domain-invariant models, but their performance depends strongly on careful selection of the crop size at test time, which is performed *per domain* in their experiments and thus violates the zero-shot premise. We tried setting the crop size automatically based on the aspect ratio of the image, but this substantially degraded the performance of Metric3D; for this reason, we use the recommended non-zero-shot protocol, with the recommended per-domain crop sizes. Since domain-specific models and crop sizes violate the strict zero-shot premise we (and other baselines) operate under, we mark the Depth Anything and Metric3D results in gray in Tab. 1.

We find that Depth Pro demonstrates the strongest generalization by consistently scoring among the top approaches per dataset and obtaining the best average rank across all datasets.

**Table 1: Zero-shot metric depth accuracy. We report the $\delta_{1}$ score per dataset (higher is better) and aggregate performance across datasets via the average rank (lower is better). Methods in gray are not strictly zero-shot. Results on additional metrics and datasets are presented in the appendices.**
| Method | Booster | ETH3D | Middlebury | NuScenes | Sintel | Sun-RGBD | Avg. Rank$\downarrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DepthAnything (Yang et al., ) | 52.3 | 9.3 | 39.3 | 35.4 | 6.9 | 85.0 | 5.7 |
| DepthAnything v2  (Yang et al., ) | 59.5 | 36.3 | 37.2 | 17.7 | 5.9 | 72.4 | 5.8 |
| Metric3D  (Yin et al., ) | 4.7 | 34.2 | 13.6 | 64.4 | 17.3 | 16.9 | 5.8 |
| Metric3D v2  (Hu et al., ) | 39.4 | 87.7 | 29.9 | 82.6 | 38.3 | 75.6 | 3.7 |
| PatchFusion  (Li et al., ) | $22.6$ | 51.8 | 49.9 | $20.4$ | $14.0$ | $53.6$ | $5.2$ |
| UniDepth  (Piccinelli et al., ) | $27.6$ | $25.3$ | $31.9$ | 83.6 | $16.5$ | 95.8 | 4.2 |
| ZeroDepth  (Guizilini et al., ) | OOM | OOM | $46.5$ | $64.3$ | $12.9$ | OOM | $4.6$ |
| ZoeDepth  (Bhat et al., ) | $21.6$ | $34.2$ | 53.8 | $28.1$ | $7.8$ | 85.7 | $5.3$ |
| Depth Pro (Ours) | 46.6 | 41.5 | 60.5 | $49.1$ | 40.0 | 89.0 | 2.5 |

**Table 2: Zero-shot boundary accuracy. We report the F1 score for dataset with ground-truth depth, and boundary recall ($R$) for matting and segmentation datasets. Qualitative results are shown on a sample from the AM-2k dataset (Li et al., ). Higher is better for all metrics.**
|  | Method | Sintel F1$\uparrow$ | Spring F1$\uparrow$ | iBims F1$\uparrow$ | AM R$\uparrow$ | P3M R$\uparrow$ | DIS R$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Absolute | DPT (Ranftl et al., ) | 0.181 | 0.029 | 0.113 | 0.055 | 0.075 | 0.018 |
| Metric3D (Yin et al., ) | 0.037 | 0.000 | 0.055 | 0.003 | 0.003 | 0.001 |  |
| Metric3D v2 (Hu et al., ) | 0.321 | 0.024 | 0.096 | 0.024 | 0.013 | 0.006 |  |
| ZoeDepth (Bhat et al., ) | 0.027 | 0.001 | 0.035 | 0.008 | 0.004 | 0.002 |  |
| PatchFusion (Li et al., ) | 0.312 | 0.032 | 0.134 | 0.061 | 0.109 | 0.068 |  |
| UniDepth  (Piccinelli et al., ) | 0.316 | 0.000 | 0.039 | 0.001 | 0.003 | 0.000 |  |
| Rel. | DepthAnything (Yang et al., ) | 0.261 | 0.045 | 0.127 | 0.058 | 0.094 | 0.023 |
| DepthAnything v2 (Yang et al., ) | 0.228 | 0.056 | 0.111 | 0.107 | 0.131 | 0.056 |  |
|  | Marigold (Ke et al., ) | 0.068 | 0.032 | 0.149 | 0.064 | 0.101 | 0.049 |
|  | Depth Pro (Ours) | 0.409 | 0.079 | 0.176 | 0.173 | 0.168 | 0.077 |

Zero-shot boundaries.
Tab. 2 summarizes the evaluation of boundary accuracy for Depth Pro and a number of baselines. This evaluation is conducted in a zero-shot setting: models are only evaluated on datasets that were not seen during training.
Since our boundary metrics are scale-invariant, our baselines here also include methods that only predict relative (rather than absolute metric) depth.
Our absolute baselines include Metric3D (Yin et al., ),
Metric3D v2 (‘giant’ model) (Hu et al., ),
PatchFusion (Li et al., ), UniDepth (Piccinelli et al., ), and ZoeDepth (Bhat et al., ).
We also report results for the relative variants of Depth Anything v1 & v2 (Yang et al., ; ) because they yield sharper boundaries than their metric counterparts.
Lastly, we include Marigold (Ke et al., ), a recent diffusion-based relative depth model that became popular due to its high-fidelity predictions.
We use the boundary metrics introduced in Sec. 3.2, and report the average boundary F1 score for datasets with ground-truth depth, and boundary recall ($R$) for datasets with matting or segmentation annotations. For image matting datasets, a pixel is marked as occluding when the value of the alpha matte is above $0.1$.

The datasets include Sintel (Butler et al., ) and Spring (Mehl et al., ), which are synthetic. We also include the iBims dataset (Koch et al., ) which is often used specifically to evaluate depth boundaries, despite having low resolution. We refer to the appendices for a full slate of iBims-specific metrics.
To evaluate high-frequency structures encountered in natural images (such as hair or fur), we use AM-2k (Li et al., ) and P3M-10k (Li et al., ), which are high-resolution image matting datasets that were used to evaluate image matting models (Li et al., ).
Additionally, we further report results on the DIS-5k (Qin et al., ) image segmentation dataset. This is an object segmentation dataset that provides highly accurate binary masks across diverse images. We manually remove samples in which the segmented object is occluded by foreground objects.
Fig. 2 visually summarizes the boundary recall metric on the AM-2k dataset, as a function of runtime.

We find that Depth Pro produces more accurate boundaries than all baselines on all datasets, by a significant margin. As can be observed in Fig. 1, in the images in Tab. 2, and the additional results in Sec. A, the competitive metric accuracy of the recent Metric3D v2 and Depth Anything v2 models does not imply sharp boundaries. Depth Pro has a consistently higher recall for thin structures like hair and fur and yields sharper boundaries. This is also true in comparison to the diffusion-based Marigold, which leverages a prior trained on billions of real-word images, as well as PatchFusion, which operates at variable resolution. Note that the runtime of Depth Pro is orders of magnitude faster than Marigold and PatchFusion (see Fig. 2 & Tab. 5).
Fig. 4 demonstrates the benefits of sharp boundary prediction for novel view synthesis from a single image.

Figure: Figure 4: Impact on novel view synthesis. We plug depth maps produced by Depth Pro, Marigold (Ke et al., ), Depth Anything v2 (Yang et al., ), and Metric3D v2 (Hu et al., ) into a recent publicly available novel view synthesis system (Khan et al., ). We demonstrate results on images from AM-2k (Li et al., ) (1st & 3rd column) and DIS-5k (Qin et al., ) (2nd column). Depth Pro produces sharper and more accurate depth maps, yielding cleaner synthesized views. Depth Anything v2 and Metric3D v2 suffer from misalignment between the input images and estimated depth maps, resulting in foreground pixels bleeding into the background. Marigold is considerably slower than Depth Pro and produces less accurate boundaries, yielding artifacts in synthesized images. Zoom in for detail.
Refer to caption: fig/tmpi/024/024_depthpro_frame0038.jpg

Focal length estimation.
Previous work (Piccinelli et al., ; Kocabas et al., ; Baradad & Torralba, ) does not provide comprehensive systematic evaluations of focal length estimators on in-the-wild images. To address this, we curated a *zero-shot* test dataset. To this end, we selected diverse datasets with intact EXIF data, enabling reliable assessment of focal length estimation accuracy. FiveK (Bychkovsky et al., ), DDDP (Abuolaim & Brown, ), and RAISE (Dang-Nguyen et al., ) contribute professional-grade photographs taken with SLR cameras. SPAQ (Fang et al., ) provides casual photographs from mobile phones. PPR10K (Liang et al., ) provides high-quality portrait images. Finally, ZOOM (Zhang et al., ) includes sets of scenes captured at various optical zoom levels.

**Table 3: Comparison on focal length estimation. We report $\delta_{25\%}$ and $\delta_{50\%}$ for each dataset, i.e., the percentage of images with relative error (focal length in mm) less than 25% and 50%, respectively.**
|  | DDDP | FiveK | PPR10K | RAISE | SPAQ | ZOOM |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | $\delta_{25\%}$ | $\delta_{50\%}$ | $\delta_{25\%}$ | $\delta_{50\%}$ | $\delta_{25\%}$ | $\delta_{50\%}$ | $\delta_{25\%}$ | $\delta_{50\%}$ | $\delta_{25\%}$ | $\delta_{50\%}$ | $\delta_{25\%}$ | $\delta_{50\%}$ |
| UniDepth (Piccinelli et al., ) | 6.8 | 40.3 | 24.8 | 56.2 | 13.8 | 44.2 | 35.4 | 74.8 | 44.2 | 77.4 | 20.4 | 45.4 |
| SPEC (Kocabas et al., ) | 14.6 | 46.3 | 30.2 | 56.6 | 34.6 | 67.0 | 49.2 | 78.6 | 50.0 | 82.2 | 23.2 | 43.6 |
| im2pcl (Baradad & Torralba, ) | 7.3 | 29.6 | 28.0 | 60.0 | 24.2 | 61.4 | 51.8 | 75.2 | 26.6 | 55.0 | 22.4 | 42.8 |
| Depth Pro (Ours) | 66.9 | 85.8 | 74.2 | 92.4 | 64.6 | 88.8 | 84.2 | 96.4 | 68.4 | 85.2 | 69.8 | 91.6 |

Tab. 3 compares Depth Pro against state-of-the-art focal length estimators and shows the percentage of images with relative estimation error under 25% and 50%, respectively. Depth Pro is the most accurate across all datasets. For example, on PPR10K, a dataset of human portraits, our method leads with 64.6% of the images having a focal length error below 25%, while the second-best method, SPEC, only achieves 34.6% on this metric. We attribute this superior performance to our network design and training protocol, which decouple training of the focal length estimator from the depth network, enabling us to use different training sets for these two tasks. Further controlled experiments are reported in the appendices.

## 5 Conclusion & limitations

Depth Pro produces high-resolution metric depth maps with high-frequency detail at sub-second runtimes.
Our model achieves state-of-the-art zero-shot metric depth estimation accuracy without requiring metadata such as camera intrinsics, and traces out occlusion boundaries in unprecedented detail, facilitating applications such as novel view synthesis from single images ‘in the wild’. While Depth Pro outperforms prior work along multiple dimensions, it is not without limitations. For example, the model is limited in dealing with translucent surfaces and volumetric scattering, where the definition of single pixel depth is ill-posed and ambiguous.

## Supplemental material

In Section A, we provide additional results and experiments. Sec. A.1 presents further qualitative comparisons to baselines, Sec.A.2 presents a more detailed zero-shot evaluation, Sec.A.3 lists runtimes for all evaluated methods, and Sec.A.4 presents additional experiments on boundary accuracy.
Section B showcases a selection of controlled experiments on Depth Pro that helped guide architectural choices (Sec.B.1, Sec.B.2, and Sec.B.5), training objective design (Sec.B.3), and curriculum training (Sec.B.4).
In Section C, we provide additional implementation, training and evaluation details, including a complete summary of the datasets that were involved in this paper.
Finally, Section D provides additional material on downstream applications.

## Appendix A Additional Results

### A.1 Qualitative results

Figure: Figure 5: Zero-shot results of Depth Pro, Marigold (Ke et al., ), Metric3D v2 (Hu et al., ), and Depth Anything v2 (Yang et al., ) on images from Unsplash (Li et al., ), AM-2k (Li et al., ), and DIS-5k (Qin et al., ).

Figure: Figure 6: Zero-shot results of Depth Pro, Marigold (Ke et al., ), Metric3D v2 (Hu et al., ), and Depth Anything v2 (Yang et al., ) on images from Unsplash (Li et al., ), AM-2k (Li et al., ), and DIS-5k (Qin et al., ).

Figure: Figure 7: Zero-shot results of Depth Pro, Marigold (Ke et al., ), Metric3D v2 (Hu et al., ), and Depth Anything v2 (Yang et al., ) on images from Unsplash (Li et al., ), AM-2k (Li et al., ), and DIS-5k (Qin et al., ).

We provide additional qualitative results of Depth Pro, Marigold (Ke et al., ), Metric3D v2 (Hu et al., ), and Depth Anything v2 (Yang et al., ) on in-the-wild images from AM-2k (Li et al., ), DIS-5k (Qin et al., ), and Unsplash(^1^11[https://www.unsplash.com](https://www.unsplash.com)) in Fig. 5, Fig. 6, and Fig. 7.
Fine details are repeatedly missed by Metric3D v2 and Depth Anything v2. Marigold reproduces finer details than Metric3D v2 and Depth Anything v2, but commonly yields noisy predictions.

### A.2 Zero-shot metric depth

Expanding on the summary in Tab.1, we provide additional results for zero-shot metric depth estimation in Tab. 4. We report results on Booster (Ramirez et al., ), Middlebury (Scharstein et al., ), Sun-RGBD (Song et al., ), ETH3D (Schöps et al., ), nuScenes (Caesar et al., ), and Sintel (Butler et al., ).
Our baselines include Depth Anything (Yang et al., ) and Depth Anything v2 (Yang et al., ), Metric3D (Yin et al., ) and Metric3D v2 (Hu et al., ), PatchFusion (Li et al., ), UniDepth (Piccinelli et al., ), ZeroDepth (Guizilini et al., ), and ZoeDepth (Bhat et al., ).
To preserve the zero-shot setting, we do not report results for models that were trained on the same dataset as the evaluation dataset.
We report commonly used metrics in the depth estimation literature, namely $\mathit{AbsRel}$, $\mathit{Log}_{10}$ (Saxena et al., ), $\delta_{1}$, $\delta_{2}$ and $\delta_{3}$ scores (Ladicky et al., ), as well as point-cloud metrics (Spencer et al., ).
Due to the high resolution of Booster images, we were not able to obtain point-cloud metrics in reasonable time.

**Table 4: Additional zero-shot metric depth evaluation. We report additional metrics used in the depth estimation literature, namely $\mathit{AbsRel}$ (Ladicky et al., ), $\mathit{Log}_{10}$, $\delta_{2}$ and $\delta_{3}$ scores, as well as point-cloud metrics (Spencer et al., ) on Booster (Ramirez et al., ), Middlebury (Scharstein et al., ), Sun-RGBD (Song et al., ), ETH3D (Schöps et al., ), nuScenes (Caesar et al., ), and Sintel (Butler et al., ). For fair comparison, all reported results were reproduced in our environment.**
| NuScenes | AbsRel$\downarrow$ | Log${}_{10}\downarrow$ | $\delta_{2}\uparrow$ | $\delta_{3}\uparrow$ | SI-Log$\downarrow$ | PC-CD$\downarrow$ | PC-F$\uparrow$ | PC-IoU$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| DepthAnything (Yang et al., ) | 0.453 | 0.151 | 73.876 | 90.301 | 28.153 | 24.146 | 0.007 | 0.004 |
| DepthAnything v2 (Yang et al., ) | 0.614 | 0.326 | 31.837 | 47.265 | 29.737 | 37.516 | 0.008 | 0.004 |
| Metric3D  (Yin et al., ) | 0.422 | 0.132 | 77.220 | 83.605 | 33.827 | 29.284 | 0.007 | 0.004 |
| Metric3D v2  (Hu et al., ) | 0.197 | 0.080 | 93.252 | 95.736 | 27.032 | 14.876 | 0.008 | 0.004 |
| PatchFusion  (Li et al., ) | 0.392 | 0.226 | 48.742 | 76.035 | 31.171 | 20.836 | 0.006 | 0.003 |
| UniDepth  (Piccinelli et al., ) | 0.138 | 0.060 | 93.006 | 96.415 | 21.801 | 11.629 | 0.009 | 0.004 |
| ZeroDepth  (Guizilini et al., ) | 0.237 | 0.121 | 82.596 | 89.908 | 30.703 | 23.348 | 0.007 | 0.004 |
| ZoeDepth  (Bhat et al., ) | 0.498 | 0.182 | 64.947 | 82.704 | 31.501 | 39.183 | 0.006 | 0.003 |
| Depth Pro (Ours) | 0.287 | 0.164 | 73.836 | 84.252 | 29.548 | 22.480 | 0.010 | 0.005 |

### A.3 Runtime

To assess the latency of our approach in comparison to baselines, we test all approaches on images of varying sizes and report results in Tab. 5. We pick common image resolutions (VGA: 640$\times$480, HD: 1920$\times$1080, 4K: 4032$\times$3024) and measure each method’s average runtime for processing an image of the given size.
All reported runtimes are reproduced in our environment and include preprocessing, eventual resizing (for methods operating at a fixed internal resolution), and inference of each model.
We further report the parameter counts and flops (at HD resolution) for each method as measured with the fvcore package.

Among all approaches with a fixed output resolution, Depth Pro has the highest native output resolution, processing more than 3 times as many pixels as the next highest, Metric3D v2 (Hu et al., ). Yet Depth Pro has less than half the parameter count and requires only a third of the runtime compared to Metric3D v2.

The variable-resolution approaches (PatchFusion (Li et al., ) and ZeroDepth (Guizilini et al., )) have considerably larger runtime, with the faster model, ZeroDepth, taking almost 4 times as long as Depth Pro, even for small VGA images.

**Table 5: Model performance, measured on a V100-32G GPU. We report runtimes in milliseconds (ms) on images of multiple sizes, as well as model parameter counts and flops. For fairness, the reported runtimes are reproduced in our environment. Entries are sorted by the native output resolution.**
| Method | Parameter<br>count | FlopsHD$\downarrow$ | Native output<br>resolution<br>$\!\uparrow$ | $t_{\mathit{VGA}}$ (ms) $\downarrow$ | $t_{\mathit{HD}}$ (ms) $\downarrow$ | $t_{\mathit{4K}}$ (ms) $\downarrow$ |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DPT | 123M | - | 384 $\times$ 384 | = 0.15 MP | 33.2 | 30.6 | 27.8 |
| ZoeDepth | 340M | - | 384 $\times$ 512 | = 0.20 MP | 235.7 | 235.1 | 235.4 |
| DepthAnything v2 | 335M | 1827G | 518 $\times$ 518 | = 0.27 MP | 90.9 | 91.1 | 91.2 |
| UniDepth | 347M | 630G | 462 $\times$ 616 | = 0.28 MP | 178.5 | 183.0 | 198.1 |
| Metric3D | 203M | 477G | 480 $\times$ 1216 | = 0.58 MP | 217.9 | 263.8 | 398.1 |
| Marigold | 949M | - | 768 $\times$ 768 | = 0.59 MP | 5174.3 | 4433.6 | 4977.6 |
| Metric3D v2 | 1.378G | 6830G | 616 $\times$ 1064 | = 0.66 MP | 1299.6 | 1299.7 | 1390.2 |
| PatchFusion | 203M | - | Original (tile-based) | 84012.0 | 84029.9 | 84453.9 |  |
| ZeroDepth | 233M | 10862G | Original | 1344.3 | 8795.7 | 34992.2 |  |
| Depth Pro | 504M | 4370G | 1536 $\times$ 1536 | = 2.36 MP | 341.3 | 341.3 | 341.3 |

### A.4 Boundary experiments

Boundary metrics empirical study. To illustrate how our boundary metrics work, we report additional qualitative edge metric results in Fig. 8. In particular, we show the occluding contours derived from the ground-truth and predicted depth, which illustrate how incorrect depth boundary predictions can impact the metric.
Furthermore, to illustrate the behavior of the boundary precision and recall measurements under various image perturbations we also provide an empirical study in Fig. 9. We report both quantitative and qualitative results on samples from the UnrealStereo4K dataset (Tosi et al., ). Our results empirically demonstrate the correlation between erroneous depth
edge predictions and low precision and recall values.

Figure: Figure 8: Evaluation metrics for sharp boundaries. We propose novel metrics to evaluate the sharpness of occlusion boundaries. The metrics can be computed on ground-truth depth maps (first two rows), and binary maps that can be derived from matting or segmentation datasets (subsequent rows). Each row shows a sample image, the ground truth for deriving occlusion boundaries, our prediction, ground-truth occluding contours, and occluding contours from the prediction. For these visualizations we set $t=15$.
Refer to caption: x10.png

Figure: Figure 9: Boundary evaluation metrics empirical study. We demonstrate how various types of image perturbations impact our proposed edge metrics. We report quantitative and qualitative results for multiple ground-truth perturbations, such as simple image shifts, downsampling followed by upsamplings, and Gaussian blurring. We report both ground-truth and perturbed occluding contours, used to derive our $F$1 scores. Our results empirically demonstrate the correlation between erroneous depth edge predictions and low precision and recall values.
Refer to caption: x35.png

Results on the iBims dataset (Koch et al., ). We supplement our boundary evaluation by results on the iBims dataset, which is commonly used for evaluating depth boundaries.
iBims consists of images of indoor scenes that have been laser-scanned. The images are at $640\times 480$ resolution and have been supplemented with manually annotated occlusion boundary maps to facilitate evaluation.
The iBims benchmark uses *Depth Directed Errors* (DDE), which evaluate overall metric depth accuracy, *Depth Boundary Errors* (DBE), which are similar in spirit to our proposed boundary metric but require manual annotation, and *Planar Errors*, which evaluate the accuracy of planes derived from the depth maps.

We find that Depth Pro is on par with the state of the art according to the DDE and PE metrics, and significantly outperforms all prior work according to the boundary metrics.

**Table 6: Zero-shot metric depth evalution on the iBims dataset (Koch et al., ). We report the iBims-specific Depth Directed Errors (DDE), Depth Boundary Errors (DBE) and Planar Errors (PE). For fairness, all reported results were reproduced in our environment. Please see Sec. A.4**
| Method | DDE (in %) | DBE (in px) | PE (in m/^∘) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | $\varepsilon^{0}_{\text{DDE}}$$\uparrow$ | $\varepsilon^{-}_{\text{DDE}}$$\downarrow$ | $\varepsilon^{+}_{\text{DDE}}$$\downarrow$ | $\varepsilon^{\text{acc}}_{\text{DBE}}$$\downarrow$ | $\varepsilon^{\text{comp}}_{\text{DBE}}$$\downarrow$ | $\varepsilon^{\text{plan}}_{\text{PE}}$$\downarrow$ | $\varepsilon^{\text{orie}}_{\text{PE}}$$\downarrow$ |
| DPT (Ranftl et al., ) | 58.744 | 41.255 | 0.000 | 3.580 | 39.372 | 0.138 | 31.837 |
| Metric3D (Yin et al., ) | 88.608 | 1.337 | 10.054 | 2.073 | 19.011 | 0.100 | 22.451 |
| Metric3D v2  (Hu et al., ) | 84.721 | 0.546 | 14.732 | 1.843 | 10.062 | 0.095 | 19.561 |
| ZoeDepth (Bhat et al., ) | 85.600 | 13.874 | 0.525 | 1.960 | 18.166 | 0.103 | 20.108 |
| Depth Anything (Yang et al., ) | 88.951 | 10.741 | 0.308 | 2.081 | 19.172 | 0.106 | 20.680 |
| Depth Anything v2 (Yang et al., ) | 91.773 | 1.619 | 6.607 | 1.959 | 8.350 | 0.095 | 19.406 |
| PatchFusion (Li et al., ) | 85.765 | 12.602 | 1.633 | 1.711 | 20.722 | 0.117 | 23.926 |
| Marigold (Ke et al., ) | 58.738 | 41.261 | 0.000 | 1.855 | 12.742 | 0.168 | 33.734 |
| UniDepth  (Piccinelli et al., ) | 73.020 | 0.041 | 26.939 | 1.999 | 14.234 | 0.098 | 19.114 |
| Depth Pro (Ours) | 89.725 | 1.809 | 8.464 | 1.680 | 10.138 | 0.095 | 18.776 |

## Appendix B Controlled Experiments

We conduct several controlled experiments to investigate the impact of various components and design decisions in Depth Pro. Specifically, we aim to assess the contribution of key components in the network architecture (Sec. B.1), training objective (Sec. B.3), training curriculum (Sec. B.4), and the focal length estimation head (Sec. B.5).

### B.1 Network backbone

We begin by evaluating various candidate image encoder backbones within our network architecture. To assess their performance, we conduct a comparative analysis utilizing off-the-shelf models available from the TIMM library (Wightman, ).
Using the pretrained weights, we train each backbone at 384 $\times$ 384
resolution across five RGB-D datasets (Keystone, HRWSI, RedWeb, TartanAir, and Hypersim) and evaluate their performance in terms of metric depth accuracy across multiple datasets, including Booster, Hypersim, Middlebury, and NYUv2, utilizing metrics such as $\mathit{AbsRel}$ for affine-invariant depth and $\mathit{Log}_{10}$ for metric depth in Tab. 7.
We find that ViT-L DINOv2  (Oquab et al., ) outperforms all other backbones by a significant margin and conclude that the combination of backbone and pretraining strategy considerably affects downstream performance.
Following this analysis, we pick ViT-L DINOv2 for both our encoder backbones.

### B.2 High-resolution alternatives

We further evaluate alternative high-resolution 1536$\times$1536 network structures and different pre-trained weights (Tab. 8). To do this, we test generalization accuracy by training on a train split of some datasets and testing on a val or test split of other datasets, following the Stage 1 protocol for all models in accordance with Tab. 14 and Tab. 15. All ViT models use a patch size of 16$\times$16. For weights pretrained with a patch size of 14$\times$14 we apply bicubic interpolation to the weights of the convolutional patch embedding layer and scale these weights inversely to the number of pixels (i.e., the weights are reduced by a factor of 1.3). All ViT models use resolution 1536$\times$1536, for this we apply bicubic interpolation to positional embeddings prior to training. The Depth Pro approach in all cases uses ViT with resolution 384$\times$384 and patch size 16$\times$16 for both the patch encoder and the image encoder. SWINv2 and convolutional models are pretrained on ImageNet (Deng et al., ). Other models use different pretraining approaches described in their papers: CLIP (Radford et al., ), MAE (He et al., ), BeiTv2 (Peng et al., ), and DINOv2 (Oquab et al., ). For the Segment Anything model we use publicly available pretrained weights, which were initialized using MAE pretraining (He et al., ) and subsequently trained for segmentation as described in their paper (Kirillov et al., ).

We find that the presented Depth Pro approach is faster and more accurate for object boundaries than the plain ViT, with comparable metric depth accuracy.
In comparison to other transformer-based and convolutional models, Depth Pro has comparable latency, several times lower metric depth error, and several times higher recall accuracy for object boundaries.

**Table 7: Comparison of image encoder backbones candidates. We train each backbone at 384$\times$384 resolution across five RGB-D datasets: Keystone, HRWSI, RedWeb, TartanAir, and Hypersim. To ensure fair comparison, we select backbone candidates with comparable computational complexity, measured in Flops using the fvcore library (Facebook Research, ), and an equivalent number of parameters. We identify ViT-L DINOv2 (Oquab et al., ) as the optimal choice for our image encoder backbone, given its superior depth accuracy performance.**
| Backbone | Flops (G) | Params (M) | AbsRel $\downarrow$ | Log${}_{10}\downarrow$ |  |
| --- | --- | --- | --- | --- | --- |
| ViT-L DINOv2-reg4 (Oquab et al., ) | 248 | 345 | 0.039 | 0.138 |  |
| ViT-L DINOv2 (Oquab et al., ) | 247 | 345 | 0.040 | 0.129 |  |
| ViT-L MAE (He et al., ) | 247 | 343 | 0.041 | 0.150 |  |
| ViT-L BeiTv2 (Peng et al., ) | 242 | 336 | 0.042 | 0.134 |  |
| ViT-L BeiT (Bhat et al., ) | 259 | 336 | 0.048 | 0.147 |  |
| ViT-L SO400m-siglip (Zhai et al., ) | 311 | 471 | 0.051 | 0.174 |  |
| ViT-L CLIP-quickgelu (Radford et al., ) | 247 | 344 | 0.053 | 0.166 |  |
| ViT-L CLIP (Radford et al., ) | 247 | 345 | 0.057 | 0.156 |  |
| ViT-L (Dosovitskiy et al., ) | 247 | 345 | 0.061 | 0.163 |  |
| ConvNext-XXL (Liu et al., ) | 514 | 867 | 0.075 | 0.216 |  |
| ViT-L DeiT-3 (Touvron et al., ) | 247 | 345 | 0.078 | 0.176 |  |
| ConvNext-L-mlp (Liu et al., ) | 162 | 214 | 0.081 | 0.222 |  |
| ConvNextv2-H (Woo et al., ) | 405 | 680 | 0.085 | 0.242 |  |
| SegAnything ViT-L (Kirillov et al., ) | 245 | 330 | 0.087 | 0.311 |  |
| SWINv2-L (Liu et al., ) | 177 | 212 | 0.091 | 0.240 |  |
| CAFormer-B36 (Yu et al., ) | 124 | 108 | 0.091 | 0.248 |  |
| EfficientViT-L3 (Liu et al., ) | – | – | 0.109 | 0.303 |  |

**Table 8: High-resolution alternatives. Generalization accuracy of alternative high-resolution 1536$\times$1536 models and different pretrained weights. All models are trained identically using Stage 1 in accordance with Tab. 14 and Tab. 15. Latency measured on a single GPU V100 with FP16 precision using batch=1. All ViT models use a patch size of 16$\times$16. Depth Pro employs a ViT-L DINOv2 (Oquab et al., ) for the image and patch encoders.**
|  |  |  | Metric depth accuracy | Boundary accuracy |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Method | Latency, ms $\downarrow$ | NYUv2 $\delta_{1}\!\uparrow$ | iBims $\delta_{1}\!\uparrow$ | iBims F1$\uparrow$ | DIS R$\uparrow$ |
| Conv. | EfficientNetV2-XL (Tan & Le, ) | 118 | 4.4 | 7.0 | 0.005 | 0.000 |
| ConvNext-XXL (Liu et al., ) | 304 | 68.0 | 38.3 | 0.134 | 0.031 |  |
|  | ConvNextv2-H (Woo et al., ) | 287 | 70.0 | 56.6 | 0.131 | 0.044 |
| Trans. | S. Anything (Kirillov et al., ) (ViT-L) | 349 | 53.2 | 38.9 | 0.140 | 0.051 |
| S. Anything (Kirillov et al., ) (ViT-H) | 365 | 51.7 | 41.1 | 0.146 | 0.050 |  |
|  | SWINv2-L (Liu et al., ) (window=24) | 272 | 58.4 | 33.1 | 0.117 | 0.028 |
| ViT | ViT-L CLIP (Radford et al., ) | 384 | 92.2 | 81.9 | 0.157 | 0.052 |
| ViT-L BeiTv2 (Peng et al., ) | OOM | 90.4 | 86.5 | 0.149 | 0.042 |  |
| ViT-L MAE (He et al., ) | 390 | 92.7 | 84.7 | 0.163 | 0.065 |  |
| ViT-L DINOv2 (Oquab et al., ) | 392 | 96.5 | 90.3 | 0.161 | 0.065 |  |
|  | Depth Pro | 341 | 96.1 | 91.3 | 0.177 | 0.080 |

### B.3 Training objectives

To assess the efficacy of our training curriculum, we compare it to alternative training schedules. We first examine the different stages individually and then compare full curricula.

Stage 1 training objectives.
We first evaluate loss combinations for the first stage (Tab. 9).

**Table 9: Comparison of stage 1 training objectives. 1A only applies the $\mathcal{L}_{\mathit{MAE}}$ to metric, and the $\mathcal{L}_{\mathit{SSI\mhyphen MAE}}$ to non-metric datasets. 1D additionally minimizes gradients on all datasets. 1B minimizes gradients only on synthetic datasets. We use 1C, which minimizes gradients with a scale-and-shift-invariant $\mathcal{L}_{\mathit{SSI\mhyphen MAGE}}$ loss on all synthetic synthetic datasets irrespective of whether the dataset is metric.**
|  | HRWSI | Hypersim | Apolloscape |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Cond. | AbsRel$\downarrow$ | $\delta_{1}\!\uparrow$ | Log${}_{10}\!\downarrow$ | AbsRel$\downarrow$ | $\delta_{1}\!\uparrow$ | Log${}_{10}\!\downarrow$ | AbsRel$\downarrow$ | $\delta_{1}\!\uparrow$ |
| 1A | 0.166 | 82.1 | 0.083 | 0.259 | 75.4 | 0.156 | 0.339 | 45.6 |
| 1D | 0.138 | 85.1 | 0.077 | 0.246 | 78.4 | 0.128 | 0.424 | 60.6 |
| 1B | 0.156 | 83.3 | 0.078 | 0.249 | 77.3 | 0.152 | 0.300 | 47.3 |
| 1C | 0.150 | 83.7 | 0.074 | 0.235 | 79.9 | 0.084 | 0.235 | 75.6 |

Condition 1A only applies a mean absolute error loss to all datasets. For non-metric datasets, we use the scale-and-shift-invariant version. Condition 1B adds gradient losses to all synthetic datasets. We again use the scale-and-shift-invariant version for non-metric datasets.
Following our observations from Sec. 3, we propose to apply an appropriate mean absolute error loss as in other conditions depending on a dataset being metric, but apply a scale-and-shift-invariant gradient loss irrespective of a dataset being metric or not (C).
Although the intuition of applying a gradient loss is to sharpen the boundaries and improve edge-related metrics, it also improves other, non-boundary-related metrics.
The key finding is that the gradient loss must be scale-and-shift-invariant, irrespective of the dataset being metric or not. Applying a regular gradient loss on metric datasets that early in training actually harms convergence.

Stage 2 training objectives.

**Table 10: Comparison of stage 2 training objectives. We evaluate the efficacy of derivative-based losses for sharpening boundaries. Employing first- and second-order derivative losses (2A) yields the best results on balance as indicated by the average rank over metrics. More details in the text.**
|  |  |  |  |  | HRWSI | Hypersim | Apolloscape |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Condition | $\mathcal{L}_{\mathit{MSE}}$ | $\mathcal{L}_{\mathit{MAGE}}$ | $\mathcal{L}_{\mathit{MSGE}}$ | $\mathcal{L}_{\mathit{MALE}}$ | AbsRel$\downarrow$ | $\delta_{1}\!\uparrow$ | Log${}_{10}\!\downarrow$ | AbsRel$\downarrow$ | $\delta_{1}\!\uparrow$ | Log${}_{10}\!\downarrow$ | AbsRel$\downarrow$ | $\delta_{1}\!\uparrow$ |
| 2A | ✓ | ✓ | ✓ | ✓ | 0.149 | 83.6 | 0.072 | 0.235 | 81.3 | 0.092 | 0.303 | 72.9 |
| 2B | ✓ | ✓ | ✓ |  | 0.148 | 83.7 | 0.072 | 0.230 | 81.0 | 0.092 | 0.299 | 73.1 |
| 2C | ✓ | ✓ |  |  | 0.150 | 83.7 | 0.072 | 0.235 | 80.8 | 0.091 | 0.300 | 73.2 |
| 2D | ✓ |  |  |  | 0.150 | 83.4 | 0.074 | 0.239 | 79.8 | 0.096 | 0.349 | 72.8 |
| 2E |  |  |  |  | 0.159 | 82.7 | 0.074 | 0.242 | 80.6 | 0.096 | 0.346 | 73.3 |

The second stage of our training curriculum focuses on sharpening depth boundaries.
To that end, we only employ synthetic datasets due to their high quality ground truth.
The obvious strategy for sharpening predictions is the application of gradient losses. We evaluate our combination of multiscale derivative-based losses in an ablation study. Condition 2A uses all of the losses, namely $\mathcal{L}_{\mathit{MAE}}$, $\mathcal{L}_{\mathit{MSE}}$ ,$\mathcal{L}_{\mathit{MAGE}}$, $\mathcal{L}_{\mathit{MALE}}$, and $\mathcal{L}_{\mathit{MSGE}}$. Tab. 10.
2B removes the second-order loss $\mathcal{L}_{\mathit{MALE}}$. 2C further removes the squared first order losses $\mathcal{L}_{\mathit{MSGE}}$. 2D removes all derivative-based losses.
2E applies the $\mathcal{L}_{\mathit{MAE}}$ to all datasets.
Removing $\mathcal{L}_{\mathit{MALE}}$ improves results on Apolloscape.
Our combination of 0th- to 2nd-order derivative losses (2A) performs best across metrics and datasets in aggregate (e.g., in terms of the average rank across metrics).

### B.4 Full curricula

**Table 11: Comparison of full curricula. We evaluate our curriculum (3A) against single stage training (3B), and pre-training on synthetic, fine-tuning with real data (3C).**
|  | HRWSI | Hypersim | Apolloscape |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Cond. | AbsRel$\downarrow$ | $\delta_{1}\!\uparrow$ | Log${}_{10}\!\downarrow$ | AbsRel$\downarrow$ | $\delta_{1}\!\uparrow$ | Log${}_{10}\!\downarrow$ | AbsRel$\downarrow$ | $\delta_{1}\!\uparrow$ |
| 3A (Ours) | 0.149 | 83.6 | 0.072 | 0.235 | 81.3 | 0.092 | 0.303 | 72.9 |
| 3B | 0.148 | 83.9 | 0.073 | 0.245 | 81.3 | 0.095 | 0.292 | 72.1 |
| 3C | 0.153 | 83.6 | 0.166 | 0.386 | 37.1 | 0.586 | 0.712 | 0.5 |

Finally, we assess the efficacy of our complete training curriculum in comparison to alternatives. Condition 3A represents our two-stage curriculum. Condition 3B trains in a single stage and applies all the second-stage gradient losses throughout the whole training. Condition 3C reverses our two stages and represents the established strategy of pretraining on synthetic data first and fine-tuning with real-world data.

### B.5 Focal length estimation

Additional analysis of zero-shot focal length estimation accuracy.
In Fig. 10, we present a more comprehensive analysis of our focal length predictor’s performance compared to baseline models.
To that end, we plot the percentage of samples below a certain absolute relative error for each method and dataset in our zero-shot evaluation set up.
Depth Pro outperforms all approaches on all datasets.

Figure: Figure 10: Evaluation of focal length estimation. Each plot compares a number of methods on a given dataset. The $x$ axis represents the AbsRel error and the $y$ axis represents the percentage of samples whose error is below that magnitude.
Refer to caption: x56.png

**Table 12: Ablation study on focal length estimation.**
| Architecture | $\delta_{25\%}\uparrow$ |
| --- | --- |
| DPT only | 60.0 |
| ViT only | 74.4 |
| DPT & ViT in series | 63.6 |
| DPT & ViT in parallel | 78.2 |

Controlled evaluation of network structures.
We evaluate a number of choices for the focal length estimation head and report results in Tab. 12. The models are evaluated on 500 images randomly sampled from Flickr (Thomee et al., ).
The first row, “DPT only” shows the performance of a network with the frozen DPT depth feature extractor (Ranftl et al., ) and a convolutional head. The second row, “ViT only”, demonstrates the performance of only using a small ViT network trained from scratch (Dosovitskiy et al., ). The third row, “DPT & ViT in series”, uses the frozen DPT feature extractor followed by a small ViT network. The fourth row, “DPT & ViT in parallel”, represents our chosen architecture depicted in Fig. 3, which utilizes frozen features from the depth network and task-specific features from a separate ViT image encoder in parallel.

We observe that “DPT & ViT in series” exhibits similar performance to “DPT only”, suggesting that adding more computation on top of the frozen DPT feature in addition to our small convolutional head does not provide extra benefits despite the increased computation. When comparing “DPT & ViT in series” with “DPT & ViT in parallel”, we observe an accuracy difference of 14.6 percentage points. This indicates that accurate focal length prediction requires extra task-specific knowledge in addition to depth information. Furthermore, “DPT & ViT in parallel” outperforms “ViT only”, which highlights the importance of features from the pretrained depth network for obtaining a high-performing focal length estimator.

## Appendix C Implementation, Training and Evaluation Details

In this section we provide additional details on the datasets used for training and evaluation, hyperparameter settings for our method, and details on the evaluation setup.

### C.1 Datasets

Tab. 13 provides a comprehensive summary of the datasets utilized in our study, detailing their respective licenses and specifying their roles (e.g., training or testing).

**Table 13: Datasets used in this work.**
| Dataset | URL | License | Usage |
| --- | --- | --- | --- |
| 3D Ken Burns (Niklaus et al., ) | [https://github.com/sniklaus/3d-ken-burns](https://github.com/sniklaus/3d-ken-burns) | CC-BY-NC-SA 4.0 | Train |
| AM-2K (Li et al., ) | [https://github.com/JizhiziLi/GFM](https://github.com/JizhiziLi/GFM) | [Custom](https://jizhizili.github.io/files/gfm_datasets_agreements/AM-2k_Dataset_Release_Agreement.pdf) | Testing |
| Apolloscape (Huang et al., ) | [https://apolloscape.auto/](https://apolloscape.auto/) | [Custom](https://apolloscape.auto/license.html) | Val |
| ARKitScenes (Dehghan et al., ) | [https://github.com/apple/ARKitScenes](https://github.com/apple/ARKitScenes) | [Custom](https://github.com/apple/ARKitScenes?tab=License-1-ov-file#readme) | Train |
| Bedlam (Black et al., ) | [https://bedlam.is.tue.mpg.de/#data](https://bedlam.is.tue.mpg.de/#data) | [Custom](https://bedlam.is.tuebingen.mpg.de/license.html) | Train |
| BlendedMVG (Yao et al., ) | [https://github.com/YoYo000/BlendedMVS](https://github.com/YoYo000/BlendedMVS) | CC BY 4.0 | Train |
| Booster (Ramirez et al., ) | [https://cvlab-unibo.github.io/booster-web/](https://cvlab-unibo.github.io/booster-web/) | CC BY NC 4.0 | Test |
| DDAD (Guizilini et al., ) | [https://github.com/TRI-ML/DDAD](https://github.com/TRI-ML/DDAD) | CC-BY-NC-SA 4.0 | Testing |
| DIML (indoor) (Kim et al., ) | [https://dimlrgbd.github.io/](https://dimlrgbd.github.io/) | [Custom](https://dimlrgbd.github.io/) | Train |
| DIS5K (Qin et al., ) | [https://xuebinqin.github.io/dis/index.html](https://xuebinqin.github.io/dis/index.html) | [Custom](https://github.com/xuebinqin/DIS/blob/main/DIS5K-Dataset-Terms-of-Use.pdf) | Test |
| DPDD (Abuolaim & Brown, ) | [https://github.com/Abdullah-Ab…pixel](https://github.com/Abdullah-Abuolaim/defocus-deblurring-dual-pixel) | MIT | Testing |
| Dynamic Replica (Karaev et al., ) | [https://github.com/facebookres…stereo](https://github.com/facebookresearch/dynamic_stereo) | CC BY-NC 4.0 | Train |
| EDEN (Le et al., ) | [https://lhoangan.github.io/eden/](https://lhoangan.github.io/eden/) | [Custom](https://lhoangan.github.io/eden/) | Train |
| ETH3D (Schöps et al., ) | [https://www.eth3d.net/](https://www.eth3d.net/) | CC-BY-NC-SA 4.0 | Testing |
| FiveK (Bychkovsky et al., ) | [https://data.csail.mit.edu/graphics/fivek/](https://data.csail.mit.edu/graphics/fivek/) | [Custom](https://data.csail.mit.edu/graphics/fivek/legal/LicenseAdobe.txt) | Testing |
| HRWSI (Xian et al., ) | [https://kexianhust.github….Ranking-Loss/](https://kexianhust.github.io/Structure-Guided-Ranking-Loss/) | [Custom](https://github.com/KexianHust/Structure-Guided-Ranking-Loss) | Train, Val |
| Hypersim (Roberts et al., ) | [https://github.com/apple/ml-hypersim](https://github.com/apple/ml-hypersim) | [Custom](https://github.com/apple/ml-hypersim?tab=License-1-ov-file#readme) | Train, Val |
| iBims (Koch et al., ) | [https://www.asg.ed.tum.de/lmf/ibims1/](https://www.asg.ed.tum.de/lmf/ibims1/) | [Custom](https://www.asg.ed.tum.de/lmf/ibims1/) | Test |
| IRS (Wang et al., ) | [https://github.com/HKBU-HPML/IRS](https://github.com/HKBU-HPML/IRS) | [Custom](https://github.com/HKBU-HPML/IRS) | Train |
| KITTI (Geiger et al., ) | [https://www.cvlibs.net/datasets/kitti/](https://www.cvlibs.net/datasets/kitti/) | CC-BY-NC-SA 3.0 | Testing |
| Middlebury (Scharstein et al., ) | [https://vision.middlebury.edu/stereo/data/](https://vision.middlebury.edu/stereo/data/) | [Custom](https://vision.middlebury.edu/stereo/data/) | Testing |
| MVS-Synth (Huang et al., ) | [https://phuang17….mvs-synth.html](https://phuang17.github.io/DeepMVS/mvs-synth.html) | [Custom](https://phuang17.github.io/DeepMVS/mvs-synth.html) | Train |
| NYUv2 (Silberman et al., ) | [https://cs.nyu.edu/…v2.html](https://cs.nyu.edu/~fergus/datasets/nyu_depth_v2.html) | [Custom](https://cs.nyu.edu/~fergus/datasets/nyu_depth_v2.html) | Testing |
| nuScenes (Caesar et al., ) | [https://www.nuscenes.org/](https://www.nuscenes.org/) | [Custom](https://www.nuscenes.org/nuscenes) | Testing |
| P3M-10k (Li et al., ) | [https://github.com/JizhiziLi/P3M](https://github.com/JizhiziLi/P3M) | [Custom](https://jizhizili.github.io/files/p3m_dataset_agreement/P3M-10k_Dataset_Release_Agreement.pdf) | Testing |
| PPR10K (Liang et al., ) | [https://github.com/csjliang/PPR10K](https://github.com/csjliang/PPR10K) | Apache 2.0 | Testing |
| RAISE (Dang-Nguyen et al., ) | [http://loki…download.html](http://loki.disi.unitn.it/RAISE/download.html) | [Custom](http://loki.disi.unitn.it/RAISE/download.html) | Testing |
| ReDWeb (Xian et al., ) | [https://sites.google.com/site/redwebcvpr18/](https://sites.google.com/site/redwebcvpr18/) | [Custom](https://sites.google.com/site/redwebcvpr18/) | Train |
| SAILVOS3D (Hu et al., ) | [https://sailvos.web.illin…index.html](https://sailvos.web.illinois.edu/_site/_site/index.html) | [Custom](https://sailvos.web.illinois.edu/_site/_site/index.html) | Train |
| ScanNet (Dai et al., ) | [http://www.scan-net.org/](http://www.scan-net.org/) | [Custom](https://kaldir.vc.in.tum.de/scannet/ScanNet_TOS.pdf) | Train |
| Sintel (Butler et al., ) | [http://sintel.is.tue.mpg.de/](http://sintel.is.tue.mpg.de/) | [Custom](http://sintel.is.tue.mpg.de/) | Testing |
| SmartPortraits (Kornilova et al., ) | [https://mobile…SmartPortraits/](https://mobileroboticsskoltech.github.io/SmartPortraits/) | [Custom](https://mobileroboticsskoltech.github.io/SmartPortraits/) | Train |
| SPAQ (Fang et al., ) | [https://github.com/h4nwei/SPAQ](https://github.com/h4nwei/SPAQ) | [Custom](https://drive.google.com/drive/folders/1wZ6HOHi5h43oxTe2yLYkFxwHPgJ9MwvT) | Testing |
| Spring (Mehl et al., ) | [https://spring-benchmark.org/](https://spring-benchmark.org/) | CC BY 4.0 | Testing |
| Sun-RGBD (Song et al., ) | [https://rgbd.cs.princeton.edu/](https://rgbd.cs.princeton.edu/) | [Custom](https://rgbd.cs.princeton.edu/) | Testing |
| Synscapes (Wrenninge & Unger, ) | [https://synscapes.on.liu.se/](https://synscapes.on.liu.se/) | [Custom](https://synscapes.on.liu.se/) | Train |
| TartanAir (Wang et al., ) | [https://theairlab.org/tartanair-dataset/](https://theairlab.org/tartanair-dataset/) | CC BY 4.0 | Train |
| UASOL (Bauer et al., ) | [https://osf.io/64532/](https://osf.io/64532/) | CC BY 4.0 | Train |
| UnrealStereo4K (Tosi et al., ) | [https://github.com/fabiotosi92/SMD-Nets](https://github.com/fabiotosi92/SMD-Nets) | [Custom](https://github.com/fabiotosi92/SMD-Nets) | Train |
| Unsplash | [https://unsplash.com/data](https://unsplash.com/data) | [Custom](https://unsplash.com/data) | Testing |
| UrbanSyn (Gómez et al., ) | [https://www.urbansyn.org/](https://www.urbansyn.org/) | CC BY-SA 4.0 | Train |
| VirtualKITTI2 (Gaidon et al., ) | [https://europe.naverlabs.com…-worlds/](https://europe.naverlabs.com/research-old2/computer-vision/proxy-virtual-worlds/) | CC BY-NC-SA 3.0 | Train |
| ZOOM (Zhang et al., ) | [https://github.com/ceciliav…inference](https://github.com/ceciliavision/zoom-learn-zoom?tab=readme-ov-file#quick-inference) | - | Testing |

### C.2 Training hyperparameters.

We specify the training hyperparameters in Tab. 14 and Tab. 15.

**Table 14: Depth Pro model training hyperparameters.**
|  | Stage 1<br>Stage 2 |
| --- | --- |
| Epochs | 250<br>100 |
| Epoch length | 72000 |
| Schedule | $1\text{\,}\mathrm{\char 37\relax}$ warmup, $80\text{\,}\mathrm{\char 37\relax}$ constant LR, $19\text{\,}\mathrm{\char 37\relax}$ $\times$0.1 LR |
| LR for Encoder | 1.28e-5 |
| LR for Decoder | 1.28e-4 |
| Batch size | 128 |
| Optimizer | Adam |
| Weight decay | 0 |
| Clip gradient norm | 0.2 |
| Pretrained LayerNorm | Frozen |
| Random color change probability | $75\text{\,}\mathrm{\char 37\relax}$ |
| Random blur probability | $30\text{\,}\mathrm{\char 37\relax}$ |
| Center crop probability for FOV-augmentation | $50\text{\,}\mathrm{\char 37\relax}$ |
| Metric depth normalization | CSTM-label (Yin et al., ) |
| Number of channels for Decoder | 256 |
| Resolution | 1536$\times$1536 |
| DepthPro model structure: |  |
| Image-Encoder resolution | 384$\times$384 |
| Patch-Encoder resolution | 384$\times$384 |
| Number of 384$\times$384 patches in DepthPro | 35 |
| Intersection of 384$\times$384 patches in DepthPro | $25\text{\,}\mathrm{\char 37\relax}$ |

**Table 15: Training loss functions for different datasets and stages.**
| Loss function | Datasets |
| --- | --- |
| Stage 1 |  |
| MAE<br>SSI-MAGE | Hypersim, Tartanair, Synscapes, Urbansyn, Dynamic Replica, Bedlam, IRS, Virtual Kitti2, Sailvos3d |
| MAE (trimmed = $20\text{\,}\mathrm{\char 37\relax}$) | ARKitScenes, Diml Indoor, Scannet, Smart Portraits |
| SSI-MAE<br>SSI-MAGE | UnrealStereo4k, 3D Ken Burns, Eden, MVS Synth |
| SSI-MAE (trimmed = $20\text{\,}\mathrm{\char 37\relax}$) | HRWSI, BlendedMVG |
| Stage 2 |  |
| MAE, MSE, MAGE, MALE, MSGE | Hypersim, Tartanair, Synscapes, Urbansyn, Dynamic Replica, Bedlam, IRS, Virtual Kitti2, Sailvos3d |

**Table 16: Dataset evaluation setup.For each metric depth dataset in our evaluation, we report the range of valid depth values, number of samples, and resolution of ground truth depth maps. Due to the large size of the validation set (approximately 35K samples), we used a randomly sampled subset of NuScenes.**
| Dataset | Minimum distance (m) | Maximum distance (m) | Number of Samples | Depth Resolution (px) |
| --- | --- | --- | --- | --- |
| Booster | $0.001$ | $10$ | 228 | $3008\times 4112$ |
| ETH3D | $0.1$ | $200$ | 454 | $4032\times 6048$ |
| iBims | $0.1$ | $10$ | 100 | $480\times 640$ |
| Middlebury | $0.001$ | $10$ | 15 | $1988\times 2952$ |
| NuScenes | $0.001$ | $80$ | 881 | $900\times 1600$ |
| Sintel | $0.01$ | $80$ | 1064 | $436\times 1024$ |
| Sun-RGBD | $0.001$ | $10$ | 5050 | $530\times 730$ |

### C.3 Baselines

Below we provide further detail on the setup of the baselines.

DepthAnything. Depth Anything v1 and v2 each released a general model for *relative* depth, but their *metric* depth models are tailored to specific domains (indoor vs. outdoor). For the metric depth evaluation, we match these models to datasets according to their domain, and for datasets containing both indoor and outdoor images, we select the model with the best performance. For qualitative results and the (scale and shift invariant) zero-shot boundary evaluation, we employ the relative depth models, since they yield better qualitative results and sharper boundaries than the metric models.

Metric3D.
For Metric3D v1 and v2, we found that the crop size parameter strongly affects metric scale accuracy. In fact, using a fixed crop size consistently yielded very poor results on at least one metric dataset. In order to obtain acceptable results, we used different crop sizes for indoor (512, 1088) and outdoor (512, 992) datasets. As in the case of Depth Anything, we mark these results in gray to indicate that they are not strictly zero-shot. For Metric 3D v2, we use the largest (‘giant’) model.

UniDepth. For UniDepth, we use the ViT-L version, which performs best on average among the UniDepth variants.

ZoeDepth. We use the model finetuned on both indoor and outdoor data (denoted ZoeD_NK).

### C.4 Evaluation setup

In evaluating our approach and baselines, we found the range of valid depth values, the depth map resolution used for computing metrics, the resizing approach used for matching the resolution of the ground truth depth maps, and the choice of intrinsics to affect results, sometimes strongly. This is why we made an effort to setup and evaluate each baseline in the same, fair evaluation set up, which we detail below.

Tab. 16 lists our evaluation datasets, the range of depth values used for evaluation, the number of samples, and the resolution of the ground truth depth maps. In case a method predicted depth maps at a different resolution, we resized predictions bilinearly to match the ground truth resolution.

Since several factors outlined above can affect the reported accuracy of a method, few baselines report sufficient detail on their evaluation setup, and the exact evaluation setups may differ across baselines, it is generally impossible to exactly reproduce reported results while guaranteeing fairness. We prioritized fair comparison and tried to evaluate all baselines in the same environment. We were able to match most reported results, with the following three notable differences. ZeroDepth reported better results on nuScenes, which we attribute to the use of a different validation set in their evaluation.
UniDepth reported different results on ETH3D, which we attribute to the handling of raw images; specifically, in our setup, we use the raw images without any post-processing, and take the intrinsics from the accompanying EXIF data; we believe this best adheres to the zero-shot premise for single-image depth estimation. Finally, on SUN-RGBD, Depth Anything fairs better in our evaluation setup than in the evaluation reported in the original paper.

Evaluation metric for sharp boundaries. For both our depth-based and mask-based boundary metrics, we apply the same weighted-averaging strategy to account for multiple relative depth ratios. F1 values (depth-based metrics) and Recall values (mask-based metrics) are averaged across thresholds that range linearly from $t_{min}=5$ to $t_{max}=25$. Weights are computed as the normalized range of threshold values between $t_{min}$ and $t_{max}$, such that stronger weights are given towards high threshold values.

## Appendix D Applications

Metric, sharp, and fast monocular depth estimation enables a variety of downstream applications. We showcase the utility of Depth Pro in two additional contexts beyond novel view synthesis: conditional image synthesis with ControlNet (Zhang et al., ) and synthetic depth of field (Peng et al., ).

Depth-conditioned image synthesis.
In this application we stylize an image through a text prompt via ControlNet (Zhang et al., ). To retain the structure of the input image, we predict a depth map from the input image and use it for conditioning the image synthesis through a pre-trained depth-to-image ControlNet SD 1.5 model.
Figure 11 shows the input image, prompt, and predicted depth maps and synthesis results for Depth Pro, Deoth Anything v2, Marigold, and Metric3D v2. We find that only Depth Pro accurately predicts the cables and sky region, resulting in a stylized image that retains the structure of the input image. Baselines either miss cables, causing the cable car to float mid-air (Depth Anything v2), or add a gradient to the sky (Marigold).

Figure: Figure 11: Comparison on conditional image synthesis. We use ControlNet (Zhang et al., ) to synthesize a stylized image given a prompt (top row, right) and a depth map. The depth map is predicted from the input image (Li et al., ) (top row, left) via Depth Pro, and baselines. The left column shows depth maps, the right column the synthesized image. For the baselines, missing cables (Depth Anything v2 & Matric3D v2) or a spurious gradient in the sky (Marigold) alter the scene structure of the synthesized image.
Refer to caption: fig/controlnet/_DqbrkesR2M/_DqbrkesR2M_reference.jpg

Synthetic depth of field.
Synthetic depth of field can be used to highlight the primary subject in a photo by deliberately blurring the surrounding areas. BokehMe (Peng et al., ) introduces a hybrid rendering framework that marries a neural renderer with a classical physically motivated renderer. This framework takes a single image along with a depth map as input. In this context, it is essential for the depth map to delineate objects well, such that the photo’s subject is kept correctly in focus while other content is correctly blurred out. Furthermore, the depth map should correctly trace out the details of the subject, to deep these (and only these) details correctly in focus. Figure 12 shows the advantage afforded by Depth Pro in this application. (We keep the most salient object in focus by setting the refocused disparity (disp_focus) hyperparameter of BokehMe as the disparity of the object.)

Figure: Depth Pro (Ours)
Refer to caption: fig/out-of-focus/366_out/366_reference.jpg

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: - the proposed method excels in detecting boundaries. Do the authors have a sense why this happens? Due to the multi-scale derivative objective? Ablations about the reason are encouraged.
- Fig. 4 is only mentioned in l457-458. No details are provided unless in their caption. It's hard to understand how DepthPro helps novel view synthesis.
- How does the proposed model handle images of an arbitrary scale?
- One claim of this paper is the efficient inference. Running time numbers should be reported. Currently they are only demonstrated in Figure 1.
soundness: 3
strengths: - Important task (monocular depth estimation)
- impressive visualizations (e.g. figure 1)
- Achieves an averaged best performance on multiple datasets and demonstrated great generalization for open-world images.
- Excels  in distinguish boundaries
- Evaluation across a large number of data sets/ablations. Of particular note is the difficulty of avoid cross-data set contamination (test samples appearing in training, etc.) and restricted access.
summary: The paper introduces a new approaches for estimating a depth map from a single image. The paper claims many advantages of the method over competing approaches, in particular the ability to estimate absolute scale under all conditions (i.e., without metadata), high resolution, and speed. The approach includes a transformer-based architecture, hybrid training approach, and focal length estimation to address scale issues. Extensive experimental results are presented.
weaknesses: - Method section is not clear enough. Missing model details.
- How are features 3-6 obtained since the images are splits into {1x,3x,5x} 384^2 and none of them matches the feature resolution at 48^2 and 96^2?
- what's the architecture of the patch encoder?
- What's the detailed architecture of the focal length head? How does this module fuse both multi-scale features and the features from the extra "image encoder for local length"? The role of the this estimation piece is critical in understanding how it is possible for the method to achieve absolute depth estimation even when camera parameters are absent but the description is lacking.
- Overall, the writing of this paper is poor. It's hard to follow both the proposed method and the experimental evaluation though the results are impressive.

---

confidence: 5
contribution: 4
initial_rating: 10
presentation: 4
questions: See weakness
soundness: 4
strengths: 1. High-Resolution Predictions: The model produces depth maps at resolutions as high as 1536 x 1536, attributed to its multi-scale vision transformer architecture.

2. Comprehensive Training: The use of extensive depth estimation datasets for training and testing demonstrates substantial effort in data processing. The pre-trained model is valuable to the community and holds potential for multiple downstream tasks.

3. Camera Intrinsic Estimation: The model can estimate camera intrinsic parameters and shows strong zero-shot generalization due to its training on diverse datasets, which is crucial for real-world scenarios where camera calibration is challenging.

4. Visual and Quantitative Superiority: The qualitative results clearly show better object boundary preservation compared to previous methods, and the quantitative evaluations across datasets highlight the model's excellent zero-shot performance. The evaluation on downstream tasks like image synthesis showcases the model’s adaptability.
summary: This paper introduces a foundational model for single-image depth estimation, trained on a large collection of datasets. The approach results in high-resolution depth maps with sharp object boundaries and enables the estimation of camera intrinsic parameters. The model outputs metric-scale depth predictions and demonstrates its effectiveness across multiple datasets, outperforming previous methods in both quantitative and qualitative evaluations. The contributions of this work are significant, with the pre-trained model holding substantial potential for a range of downstream tasks.
weaknesses: 1. Limited Novelty: Although effective, the proposed method's novelty is somewhat constrained as it leverages existing approaches. The loss functions and training strategy draw heavily from DPT (Ranftl et al., 2022), and the metric depth estimation follows ideas from Metric3D (Yin et al., 2023). More discussion on the unique aspects would strengthen the paper.

2. Depth Consistency in Videos: While single-image results are impressive, depth consistency across video frames is not explored. Given the model’s metric-scale depth predictions, it would be valuable to assess its utility in video-based tasks such as visual SLAM, camera pose estimation, and 3D reconstruction.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1. Structured Training Curriculum: The curriculum is specifically tailored to handle the unique strengths and weaknesses of synthetic and real-world datasets, resulting in a model that generalizes effectively while producing fine boundary details in depth maps.

2. Enhanced Boundary Evaluation Metrics: New metrics for evaluating depth boundaries address a gap in existing benchmarks by focusing on boundary precision, which is critical for applications like view synthesis that demand fine details.
summary: The proposed Depth Pro model employs a ViT architecture for zero-shot metric monocular depth estimation, targeting applications such as novel view synthesis. By employing patch-based, multi-scale processing, Depth Pro achieves high-resolution depth predictions while preserving real-time performance and edge sharpness. This paper introduces a two-stage training approach that integrates synthetic and real-world datasets, enhancing depth boundary accuracy. Furthermore, the model includes boundary evaluation metrics and a focal length estimation component, improving its robustness in handling images lacking metadata.
weaknesses: 1. Dependence on pretrained ViT with Limited Architectural Innovation: Depth Pro benefits from pretrained ViT backbones, yet its architecture primarily builds on existing elements rather than introducing fundamentally new mechanisms for depth estimation, which limits its architectural novelty.

2. Heavy Reliance on Synthetic Data for Boundary-Sensitive Training: The model’s second-stage training emphasizes synthetic datasets for boundary sharpness. This reliance on synthetic data could impact generalization to real-world environments, particularly in complex or unstructured settings where boundaries are less distinct.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. If the loss function were designed using depth instead of canonical inverse depth, would the results differ? Have you conducted any experiments to compare these approaches?
2. Among the various technical contributions (e.g., curriculum learning, focal length estimation, and the patch encoder with multiple scales), which components most significantly affect the overall performance metrics (e.g., abs_rel, δ_1) and boundary accuracy, respectively? Although multiple technical contributions are mentioned, it is unclear which are the most impactful.
3. Is there any tradeoff between overall performance and boundary accuracy related to certain hyperparameters in the deep neural network or the training loss?
4. For calculating the final depth, does the proposed method use the predicted inverse depth C, the predicted focal length f_px, and the ground-truth width w, as in the equation D = f_px / (wC)?
5. As I understand it, all experiments were conducted with evaluation using metric scale depth, without median-scaling evaluation. Is my understanding correct?
6. Have you conducted experiments on fine-tuning the proposed model (DepthPro)? If so, is there a significant performance improvement between the zero-shot setting and fine-tuning?
7. What is the training time and GPU resources required to train the proposed method?
soundness: 4
strengths: - The model demonstrates strong zero-shot performance for metric depth estimation, effectively handling diverse input images without fine-tuning.
- It achieves superior focal length estimation accuracy, which enhances depth map scaling and generalization across different image conditions.
- Comprehensive ablation studies are provided to analyze the impact of key architectural and training choices, showcasing the robustness of the proposed method.
- The experiments are conducted on a wide range of datasets, underscoring the model’s effectiveness and applicability across various scenarios.
- Detailed training procedures are described, allowing for reproducibility and providing insights into the model’s development process.
- The method exhibits higher edge accuracy compared to conventional techniques, maintaining competitive inference times.
- It outperforms state-of-the-art approaches, such as DepthAnything, DepthAnything v2, and Marigold, with clear comparative analysis highlighting these advancements.
summary: The paper introduces Depth Pro, a model for zero-shot metric monocular depth estimation that generates high-resolution depth maps with precise boundary details at sub-second processing speeds. Key innovations include a multi-scale vision transformer architecture, a mixed training regimen with real and synthetic data for improved accuracy, and a dedicated evaluation approach for boundary sharpness. The model also features state-of-the-art focal length estimation without the need for camera metadata. Depth Pro outperforms prior approaches in accuracy and efficiency, making it suitable for applications like novel view synthesis.
weaknesses: The most critical components that influence performance and boundary accuracy among various technical contributions are not clearly highlighted, leaving ambiguity about which elements contribute most to the model's success.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - Marigold and Depth Anything have demonstrated that using synthetic training data can effectively improve details in depth prediction. Depth Pro also uses synthetic data to enhance high-frequency details in the depth map, yet its predicted boundaries are significantly better than those of existing methods. What is the main reason for this?significantly better than the existing methods. What is the main reason for this?
- Comparative methods such as Depth Anything and Marigold are trained with low-resolution images, while Depth Pro is trained with high-resolution images. Is this the reason why Depth Pro achieves better detail in depth prediction?
- According to Table 13, Depth Pro appears to use more training data than other competitors. Can you compare the scale of training data used by Depth Pro with that of other competitors?
- According to Table 1, Depth Pro has a significant gap with the best performing methods on some datasets (such as ETH3D). Can you analyze the reasons for this discrepancy?
soundness: 4
strengths: - This paper is well-written and motivated.
- The proposed method demonstrates impressive performance.
- It addresses the metric monocular depth estimation problem, a crucial task with significant application prospects.
- The method estimates camera focal length, eliminating the need for image metadata—a practical advancement.
- The experiments detailed in the main text and appendix are comprehensive and solid.
summary: This paper presents a novel framework for high-resolution metric depth estimation, demonstrating both excellent organization and a well-articulated motivation. The qualitative outcomes are impressive, and the quantitative results also demonstrate the state-of-the-art performance. The authors assert four main contributions: (1) an innovative network architecture, (2) a newly proposed evaluation metric for boundary tracing, (3) newly designed loss functions, and (4) the capacity to estimate camera focal length.
weaknesses: There are some minor issues:
- There are a few unclear statements regarding Figure 3:
  - Along which dimension does the concatenation operation perform?
  - What is the specific implementation of the Merge operation?
- At Line 234, the statement "produces a feature tensor at resolution 24 × 24 per input patch (features 3 – 6 in Fig. 3)" seems to not coincide with the figure label.

Meta-review: This paper received mostly positive ratings from reviewers. The concerns raised by reviewers were effectively addressed in the rebuttal, leading to general satisfaction with the authors’ responses. Reviewer yFUY raised issues regarding poor writing, unclear technical details, and insufficient runtime comparison. The authors provided more detailed explanations of technical aspects and clarified runtime comparisons in their response. While these improvements addressed the reviewer’s concerns, the area chair encourages the authors to further enhance the clarity and quality of the writing in the final version. Reviewers eL31 and a8Yt expressed concerns about the limited novelty of the work. The authors clarified the contributions of their work in the rebuttal, particularly emphasizing the innovative aspects of their approach. Reviewer AZEW noted that the critical components influencing performance and boundary accuracy among the technical contributions were not clearly highlighted. The authors included additional ablation studies in their rebuttal, which helped clarify the importance of specific components. The reviewer acknowledged that their concerns were addressed. Reviewer 85gd requested a better explanation for the strong performance of the proposed method. The authors provided a more comprehensive explanation of the factors contributing to the method’s performance, which satisfied the reviewer. Given the authors’ effective rebuttal and the reviewers’ satisfaction with the addressed concerns, this paper makes a valuable contribution to the field and is recommended for acceptance.

Decision: Accept (Poster)