BenchDepth: Are We on the Right Way to Evaluate Depth Foundation Models?

## Abstract

Abstract Depth estimation is a fundamental task in computer vision with diverse applications. Recent advancements in deep learning have led to powerful depth foundation models (DFMs), yet their evaluation remains challenging due to inconsistencies in existing protocols. Traditional benchmarks rely on alignment-based metrics that introduce biases, favor certain depth representations, and complicate fair comparisons.
In this work, we propose BenchDepth , a new benchmark that evaluates DFMs through five carefully selected downstream proxy tasks: depth completion, stereo matching, monocular feed-forward 3D scene reconstruction, SLAM, and vision-language spatial understanding. Unlike conventional evaluation protocols, our approach assesses DFMs based on their practical utility in real-world applications, bypassing problematic alignment procedures.
We benchmark eight state-of-the-art DFMs and provide an in-depth analysis of key findings and observations. We hope our work sparks further discussion in the community on best practices for depth model evaluation and paves the way for future research and advancements in depth estimation.

## 1 Introduction

Depth estimation plays a crucial role in various computer vision applications, from 3D scene reconstruction autonomous driving, to robotics  [1, 2, 3, 4]. In recent years, deep learning-based approaches have significantly advanced the field, leading to powerful foundation models capable of generating high-quality depth predictions across diverse input domains [5, 6, 7, 8, 9, 10, 11]. However, despite these advancements, evaluating and comparing depth estimation models remains an open challenge [12]. Existing evaluation protocols often overlook critical factors that impact both the validity and comparability of results.

A major limitation in current depth evaluation lies in its reliance on alignment-based metrics, where predictions are aligned to ground truth before computing metrics. However, this alignment process introduces several biases that can affect fairness. Depth estimation methods adopt different representations—some predict metric depth directly, while others estimate affine-invariant disparity or affine-invariant depth, requiring distinct alignment strategies. Applying the same alignment solver across these representations can be problematic, as depth and disparity are related by a non-linear transformation. Furthermore, the widely used least squares solver is highly sensitive to outliers, favoring smoother depth predictions with lower global error while penalizing sharper estimates that may contain large gradients. These biases raise concerns about the robustness and fairness of existing evaluation protocols, as discussed in Sec. [3](https://arxiv.org/html/2507.15321v1#S3).

Figure: Figure 1: BenchDepth illustration and results. We evaluate different types of depth predictions (highlighted with different colors) with proxy tasks in a bottom-to-top manner, where MoGe [10] adopts various alignment algorithms to compare with different types of depth methods. We show the rank of existing methods for each task on the left and present the average rank in the right table. Note that there are metric depth models, like Metric3DV2 [14], using scale and shift during inference, conflicting with the task definition.
Refer to caption: x1.png

Additionally, inconsistencies arise when evaluating metric depth predictions. While some works directly compare metric depth to ground-truth values [13, 14], others treat these predictions as scale- or affine-invariant and apply alignment before computing metrics [10, 12]. This lack of standardization complicates the interpretation of results and creates unnecessary variation in evaluation procedures.

These challenges not only hinder fair comparisons among existing methods but also discourage the adoption of novel depth representations. For instance, MoGe [10] estimates an affine-invariant point map, where z-coordinates represent affine-invariant depth. To compare MoGe with affine-invariant disparity-based methods, the authors must first recover the shift term via point map optimization, then convert it into scale-invariant depth, and finally invert it to obtain affine-invariant disparity before aligning it with the inverse of the ground-truth depth. This multi-step process complicates model evaluation and introduces additional sources of error due to multiple optimization steps.

Moreover, downstream tasks increasingly rely on depth as a guidance, emphasizing the need for an evaluation framework that can reveal a model’s potential across various applications [16, 4, 3, 17, 18]. Traditional benchmarks primarily focus on numerical accuracy within constrained depth estimation settings, failing to assess how well different models generalize to real-world tasks [12].

To address these challenges, we propose a new approach for benchmarking depth foundation models. Rather than relying solely on traditional depth evaluation metrics, we use downstream tasks as proxy tasks for model evaluation. This approach is inspired by the success of large language model (LLM), vision language model (VLM), and image classification [19, 20, 21, 22], where the evaluation is often based on downstream tasks. To this end, we propose BenchDepth, a benchmark consisting of five downstream proxy tasks: stereo matching [23], depth completion [16], monocular feed-forward 3D scene reconstruction [4], SLAM [3], and 3D-VQA [24]. The tasks are selected in a bottom-to-top manner as shown in Fig. [1](https://arxiv.org/html/2507.15321v1#S1.F1), ranging from applications in low-level to high-level vision.
These tasks allow us to evaluate the practical utility of depth foundation models in a fair manner, without relying on potentially problematic depth alignment procedures.

In this paper, we fairly benchmark eight state-of-the-art 3D foundation models with DepthBench. By examining their performance on a set of proxy tasks, we provide a more robust and holistic understanding of what constitutes a good foundation depth model. Our main findings and conclusions are as follows:

- 1.
Most depth foundation models improve the performance of downstream tasks, highlighting their potential for broader applications in the future.
- 2.
Overall, DAV2 [8] achieves the best results across proxy tasks, demonstrating the benefits of scaling up training data and incorporating synthetic data.
- 3.
Affine-invariant disparity methods consistently outperform other depth estimation approaches, even with MiDaS [9] being the oldest method among them.
- 4.
Despite being fine-tuned on a single dataset (Hypersim [25], synthetic), DAV2-Met significantly outperforms other metric depth models [14, 13] trained on multiple datasets. This aligns with the conclusion of ZoeDepth [6] that fine-tuning a well-pretrained affine-invariant disparity model enhances metric depth estimation. Moreover, the performance gap suggests that incorporating synthetic data for metric depth training is crucial, as it allows models to learn high-frequency details that are often lost in real-world datasets [8, 26].
- 5.
The performance improvement from Marigold [7] to GenPercept [15] underscores the importance of effective fine-tuning strategies for Stable Diffusion [27], a powerful foundation model. Expanding the training data could further unlock their potential, following the success of other methods, as the current fine-tuning process is limited to VKITTI [28] and Hypersim.
- 6.
MoGe [10], as a novel approach for geometry estimation, demonstrates potential on DepthBench, though further research is needed to improve its performance.
- 7.
For the highest-level task, VLM spatial understanding, all methods yield comparable results. This suggests that at this higher level, different depth estimation approaches can be equally effective.

We hope that our work will spark further discussion in the community about the best practices for depth model evaluation and pave a way for the further research and development of depth estimation.

## 2 Related Works

### 2.1 Depth Foundation Model (DFM)

Monocular depth estimation has seen significant advancements with the availability of large-scale public datasets [29, 30, 31], improved architectural designs [5, 32, 33, 34], and enhanced training strategies [35, 36, 37], etc. While earlier works primarily focused on achieving high performance in in-domain inference, the scaling of both models and datasets in deep learning [38] has shifted recent research toward developing foundation models with strong zero-shot generalization across unseen domains (i.e., diverse real-world images). For example, MiDaS [9] introduces a mixture-dataset training approach and adopted an affine-invariant disparity representation to handle cross-dataset inconsistencies. DAV2 [39, 8] follows a similar formulation but scaled training further using a semi-supervised learning paradigm. Other works leverage the prior knowledge of Stable Diffusion [27] and fine-tune pretrained models for affine-invariant depth estimation [7, 15]. Other lines of research such as Metric3DV2 [14] and UniDepth [13] aim to predict metric depth by incorporating explicit camera models. MoGe [10] proposes a novel formulation using affine-invariant point maps [40] to represent monocular geometry. Despite the rapid progress in depth foundation models, a key challenge remains: fairly evaluating and comparing these models across different depth representations and real-world applications.

### 2.2 Evaluations of DFMs

Eigen et al [5] introducs the first deep learning framework for monocular metric depth estimation, along with several standard evaluation metrics that remain widely used today. However, while depth estimation methods have diversified into various depth representations (as summarized in Tab. [1](https://arxiv.org/html/2507.15321v1#S1.F1)), existing works attempt to adopt the same evaluation protocol designed for metric depth estimation [9, 8, 14, 7, 15, 10]. This might lead to seemingly comparable numerical results that may still be biased due to inconsistencies in the alignment process. Inspired by large language models (LLMs) [19], vision-language models (VLMs) [20], and self-supervised learning in image classification [21, 22], where the evaluation is often based on downstream tasks, we propose a proxy-task-based benchmark for DFMs. By assessing DFMs on a diverse set of real-world tasks in a bottom-to-top manner, our approach enables a fairer and more practical comparison, eliminating the need for problematic alignment procedures. Compared with [41], our benchmark focuses on the monocular setting and the practical potential for downstream tasks.

Figure: Algorithm 1 Robustness analysis of the alignment algorithm in depth and disparity spaces.

## 3 Overlooked Alignment for Evaluating DFMs

In this section, we analyze the limitations of the current depth evaluation protocols. We focus on the widely used metric, $\delta$, which measures the proportion of pixels satisfying $\max(a_{i}/d_{i},d_{i}/a_{i})<1.25$, where $a$ and $d$ are the aligned prediction and ground-truth depth, respectively.

### 3.1 Alignment in Different Spaces

As summarized in Tab. [1](https://arxiv.org/html/2507.15321v1#S1.F1), affine-invariant disparity estimation methods align their predictions with the inverse of the ground-truth depth, while other methods align predictions directly in depth space. The commonly used least squares solver for alignment is designed for ordinary linear first-order differential equations, but the inverse operator is inherently non-linear. This discrepancy introduces different behaviors when aligning predictions in these two spaces, leading to potential unfairness in comparisons.

To analyze the robustness of the alignment process in different spaces, we conduct an experiment (Alg. [1](https://arxiv.org/html/2507.15321v1#algorithm1)) where a magnifying disturbance is added to the predicted depth and disparity, both initialized as ground-truth values. The standard protocol is then applied to compute the evaluation metric after alignment. As illustrated in Fig. [2(a)](https://arxiv.org/html/2507.15321v1#S3.F2.sf1), aligning in disparity space exhibits higher robustness to small errors compared to depth space. However, it becomes more sensitive to larger errors. This asymmetric behavior in different alignment spaces introduces inconsistencies in evaluation, revealing issues in current depth evaluation protocols.

### 3.2 Sensitivity of Scale-and-Shift Alignment

Since the least squares solver is sensitive to large outliers [42, 43, 10], we conduct an experiment to investigate its impact on the depth evaluation metric. In Alg. [2](https://arxiv.org/html/2507.15321v1#algorithm2), we initialize a predicted depth map identical to the ground-truth depth and introduce a disturbance with decreasing size. We then compute depth metrics with and without alignment.

Fig. [2(b)](https://arxiv.org/html/2507.15321v1#S3.F2.sf2) reveals that the $\delta$ metric exhibits entirely different monotonicity patterns depending on whether alignment is applied. Without alignment, the metric behaves as expected: as the disturbance size decreases, the proportion of pixels satisfying the accuracy threshold increases. However, with alignment, the presence of outliers significantly disrupts the alignment results, leading to a counterintuitive $\delta$ metric that fails to accurately reflect depth prediction quality. Adopting RANSAC to filter outliers can alleviate the impact, but the issue still exists. This suggests that alignment biases the evaluation protocol in favor of smoother depth predictions, while sharper depth maps, which can introduce stronger outlier gradients, suffer from degraded evaluation scores.

To eliminate biases in alignment, we propose benchmarking depth estimation using proxy tasks. By directly feeding depth predictions into proxy task frameworks without any alignment, we enable a fair comparison among different depth estimation methods, independent of scale and shift variations.

Figure: Algorithm 2 Analysis of the influence of the alignment algorithm’s sensitivity to the depth metric.

Figure: (a) Algo. [1](https://arxiv.org/html/2507.15321v1#algorithm1) plot.
Refer to caption: extracted/6639340/figs/alignment_compare_update_a1.png

## 4 BenchDepth

We introduce BenchDepth, a novel benchmark for depth estimation, designed with carefully selected proxy tasks in a bottom-up manner (Fig. [1](https://arxiv.org/html/2507.15321v1#S1.F1)). As lower-level tasks, we select depth completion [16] and stereo matching [23]. These tasks closely resemble depth estimation, as they belong to the category of metric depth estimation but incorporate additional prompts (e.g., sparse depth from real sensors or stereo image pairs with a fixed baseline). Middle-level tasks feed-forward 3D Gaussian Splatting (3DGS) [4] and SLAM [3], focus on 3D reconstruction but differ in representation (3DGS [44] and neural implicit representations [45]) and the number of input images (single or multiple). At the highest level, we evaluate depth estimation for vision-language models (VLMs) [46], aiming to assess the role of depth in enhancing spatial understanding.

Selected depth foundation estimation methods for benchmarking are summarized in Tab. [1](https://arxiv.org/html/2507.15321v1#S1.F1). We choose the most representative methods from each depth estimation category. Note that though DAV2-Met [8], Metric3DV2 [14], and UniDepth [13] are all metric methods, DAV2-Met is fine-tuned on a single metric dataset (Hypersim [25]), whereas the other two methods are trained with a mixture of many datasets. We use the default camera parameter assumption for Metric3DV2 and UniDepth. Since the original version of Marigold [7] is hard to be adopted to online training due to the large number of inference steps, we use the end-to-end fine-tuned version of Marigold [47] that supports one-step inference as a replacement.

Key features of BenchDepth include:

- 1.
Fair comparisons among various DFMs without reliance on alignment.
- 2.
Evaluation of the broader applicability of DFMs beyond standard benchmarks [12].

Below, present the five proxy tasks in detail and describe the modifications applied to selected methods to support depth evaluation using DepthBench. We use 8 GPUs to conduct the benchmark.

Figure: (a) (a)
Refer to caption: x3.png

**Table 1: Benchmark with metric depth completion. We select DepthPrompting [16] as the baseline method and apply depth predictions from various foundation models as the guidance. We use different amounts of sparse samples (from 100 to 1) in this experiment. Best results are in bold, second best are underlined. imp. (%) indicates the average improvement ratio, and rank is calculated based on it.**
| Method | 100 | 32 | 8 | 4 | 1 | imp. | rank |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE | RMSE | MAE |  |  |  |
| w/o depth [16] | 0.206 | 0.102 | 0.334 | 0.199 | 0.486 | 0.340 | 0.514 | 0.370 | 0.550 | 0.406 | - | - |
| \cellcolorSalmonMidas [9] | 0.204 | 0.114 | 0.294 | 0.182 | 0.449 | 0.311 | 0.493 | 0.355 | 0.556 | 0.414 | +3.09 | 4 |
| \cellcolorSalmonDAV2-Rel [8] | 0.191 | 0.099 | 0.279 | 0.166 | 0.427 | 0.292 | 0.471 | 0.336 | 0.533 | 0.396 | +9.26 | 1 |
| \cellcolorGoldenrodDAV2-Met [8] | 0.202 | 0.112 | 0.287 | 0.178 | 0.431 | 0.297 | 0.472 | 0.338 | 0.529 | 0.392 | +6.48 | 2 |
| \cellcolorGoldenrodMetric3DV2 [14] | 0.216 | 0.128 | 0.306 | 0.195 | 0.454 | 0.317 | 0.497 | 0.359 | 0.557 | 0.415 | -0.38 | 8 |
| \cellcolorGoldenrodUniDepth [13] | 0.210 | 0.122 | 0.296 | 0.187 | 0.438 | 0.308 | 0.480 | 0.349 | 0.540 | 0.404 | +2.97 | 5 |
| \cellcolorApricotMarigold [7] | 0.210 | 0.121 | 0.296 | 0.187 | 0.448 | 0.314 | 0.491 | 0.356 | 0.555 | 0.414 | +1.76 | 6 |
| \cellcolorApricotGenPercept [15] | 0.199 | 0.110 | 0.284 | 0.174 | 0.436 | 0.301 | 0.479 | 0.342 | 0.542 | 0.402 | +6.16 | 3 |
| \cellcolorLimeGreenMoGe [10] | 0.210 | 0.124 | 0.295 | 0.188 | 0.444 | 0.312 | 0.489 | 0.355 | 0.558 | 0.417 | +1.53 | 7 |

Depth Completion: Given sparse metric-scale depth measurements from sensors (e.g.LiDAR, Radar) and corresponding images, depth completion aims to generate dense metric depth predictions. We select DepthPrompting [16] as the baseline method. While DepthPrompting enables the adaptation of foundation depth models for completion, its reliance on feature extractors from these models [32] introduces bias, as the extractor quality may influence performance more than the predicted depth itself. To mitigate this, we standardize feature extractors across models and inject depth predictions using zero convolutions [1] (Fig. [3(a)](https://arxiv.org/html/2507.15321v1#S4.F3.sf1)). Additionally, we omit the alignment module in DepthPrompting to enable direct comparisons across depth methods. We use the NYU Depth V2 dataset [29] for this proxy task, following the official split with about 50k training samples and 654 testing samples.

Stereo Matching: This task estimates disparity from two images with a known baseline. Metric depth can be recovered from disparity using camera parameters. We adopt IGEV [23] as our baseline and incorporate zero convolutions [1] to inject depth predictions as shown in Fig. [3(b)](https://arxiv.org/html/2507.15321v1#S4.F3.sf2). Unlike prior works that develop task-specific strategies to integrate depth into stereo matching models [18, 17], our simple yet general approach allows for a more straightforward assessment of depth prediction quality. We use the SceneFlow dataset [48], which contains 35,454 training pairs and 4,370 test pairs with dense disparity maps. Middlebury 2014 [49] and ETH3D [50] are used for zero-shot evaluation.

**Table 2: Benchmark with stereo matching. We select IGEV [23] as the baseline method and apply depth predictions from various foundation models as the guidance to fine-tune the baseline model. We present rank for each each dataset whereas avg. rank indicates the average rank of all evaluation performances.**
| Method | SceneFlow [48] | Middlebury [49] | ETH3D [50] | imp. | rank |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| EPE$\downarrow$ | $>$3pt$(\%)\downarrow$ | EPE$\downarrow$ | $>$2pt$(\%)\downarrow$ | EPE$\downarrow$ | $>$1pt$(\%)\downarrow$ |  |  |  |
| w/o depth [23] | 0.496 | 2.599 | 0.857 | 6.655 | 0.283 | 3.575 | - | - |
| \cellcolorSalmonMidas [9] | 0.483 | 2.502 | 1.061 | 7.316 | 0.273 | 3.383 | -3.07 | 7 |
| \cellcolorSalmonDAV2-Rel [8] | 0.456 | 2.432 | 0.834 | 6.399 | 0.275 | 3.189 | +5.77 | 1 |
| \cellcolorGoldenrodDAV2-Met [8] | 0.471 | 2.473 | 0.938 | 6.177 | 0.270 | 3.698 | +1.46 | 5 |
| \cellcolorGoldenrodMetric3DV2 [14] | 0.482 | 2.521 | 0.949 | 7.309 | 0.275 | 3.523 | -1.74 | 6 |
| \cellcolorGoldenrodUniDepth [13] | 0.477 | 2.521 | 0.964 | 7.242 | 0.285 | 3.822 | -3.68 | 8 |
| \cellcolorApricotMarigold [7] | 0.475 | 2.499 | 0.899 | 6.519 | 0.273 | 3.485 | +1.87 | 4 |
| \cellcolorApricotGenPercept [15] | 0.473 | 2.485 | 0.935 | 6.649 | 0.265 | 3.374 | +1.99 | 3 |
| \cellcolorLimeGreenMoGe [10] | 0.473 | 2.481 | 0.907 | 5.951 | 0.279 | 3.544 | +2.70 | 2 |

**Table 3: Benchmark with feed-forward monocular 3D scene reconstruction by novel view synthesis. We select Flash3D [4] as the baseline method and apply depth predictions from various foundation models as the model input. Following [4], we present results of small, medium and large baseline ranges separately.**
| Method | 5 frames | 10 frames | $\mathcal{U}\left[-30,30\right]$ frames | imp | rank |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR$\uparrow$ | SSIM$\uparrow$ | LPIP$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIP$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIP$\downarrow$ |  |  |  |
| w/o depth [4] | 24.285 | 0.803 | 0.151 | 21.767 | 0.729 | 0.203 | 21.241 | 0.705 | 0.230 |  |  |
| \cellcolorSalmonMidas [9] | 24.964 | 0.812 | 0.125 | 22.290 | 0.735 | 0.179 | 21.769 | 0.710 | 0.212 | +5.24 | 1 |
| \cellcolorSalmonDAV2-Rel [8] | 24.965 | 0.812 | 0.129 | 22.305 | 0.733 | 0.185 | 21.703 | 0.706 | 0.218 | +4.21 | 3 |
| \cellcolorGoldenrodDAV2-Met [8] | 25.000 | 0.812 | 0.128 | 22.341 | 0.735 | 0.182 | 21.842 | 0.711 | 0.215 | +4.81 | 2 |
| \cellcolorGoldenrodMetric3DV2 [14] | 24.468 | 0.787 | 0.150 | 21.994 | 0.713 | 0.204 | 21.396 | 0.690 | 0.233 | -0.05 | 5 |
| \cellcolorGoldenrodUniDepth [13] | 23.983 | 0.786 | 0.145 | 21.530 | 0.708 | 0.202 | 21.036 | 0.687 | 0.235 | -0.10 | 6 |
| \cellcolorApricotMarigold [7] | 23.974 | 0.779 | 0.162 | 21.515 | 0.701 | 0.219 | 20.952 | 0.676 | 0.248 | -4.19 | 8 |
| \cellcolorApricotGenPercept [15] | 24.119 | 0.787 | 0.140 | 21.489 | 0.705 | 0.197 | 21.029 | 0.682 | 0.230 | -0.14 | 4 |
| \cellcolorLimeGreenMoGe [10] | 23.930 | 0.780 | 0.144 | 21.309 | 0.696 | 0.202 | 20.851 | 0.673 | 0.235 | -1.60 | 7 |

Feed-Forward Monocular 3DGS: This task reconstructs scenes and synthesizes novel views from a single image using 3D Gaussian Splatting [44]. We use Flash3D [4] as the baseline model. Flash3D incorporates a frozen depth foundation model in its first stage to estimate depth from the input image. The predicted depth and image are then processed by a UNet-like [51] network to estimate 3DGS parameters. Since the foundation depth model remains frozen and no features from the foundation model are used in the second stage, we can adopt different foundation models for the first stage and train Flash3D following the default recipe. We use the RealEstate10k dataset [52]. It consists of real estate videos from YouTube, with 67,477 training scenes and 7,289 test scenes. Some outdated samples were removed, causing slight deviations from the results reported in [4].

Simultaneous Localization and Mapping: Simultaneous Localization and Mapping (SLAM) is a fundamental problem in computer vision with broad applications. We employ NICER-SLAM [3] as our baseline, as it integrates dense SLAM with a neural implicit representation for tracking and mapping from monocular RGB videos. Since NICER-SLAM can process RGB-D sequences, we replace the original sensor depth with depth predictions from different foundation models and train the system accordingly. To better assess the impact of depth predictions, we omit pseudo-depth loss during training. We evaluate models on the Replica dataset [53], which provides RGB-(D) images rendered using the official renderer. All 8 scenes are used for benchmarking. For benchmarking, we replace the original input depth with estimated depth from different methods and omit the monocular depth loss (Eq. 13 in [3]), which depends on another depth model. We exclude Metric3DV2 since it was trained on this dataset, though there is no evidence of overfitting.

VLM Spatial Understanding: Vision-Language Models (VLMs) have demonstrated strong performance in 2D image understanding but remain limited in spatial reasoning [46]. Since depth maps contain spatial information, incorporating them as additional inputs may improve VLMs’ 3D understanding. For this proxy task, we adopt SpatialBench [46] to evaluate the impact of different depth models on VLM spatial reasoning. We use two VLMs: ChatGPT-4o and SpatialBot-Phi2-3B [46]. Since ChatGPT-4o is not trained with depth maps, we render depth predictions using the magma colormap and provide corresponding text prompts.

## 5 Benchmark Results

Depth Completion. Tab. [1](https://arxiv.org/html/2507.15321v1#S4.T1) presents the benchmark results. DAV2-Rel [8] is the only method that consistently improves performance across almost all settings, achieving rank 1. Most methods provide a performance boost, except for Metric3DV2 [14], which performs worse that the baseline. Interestingly, depth methods tend to be more beneficial when the available sparse ground-truth (GT) depth is limited. This suggests that foundation models provide useful guidance when GT depth is scarce. However, as GT depth increases, the ambiguity in selecting the appropriate depth source limits further improvements compared to using only sparse GT depth for guidance.

Stereo Matching. Tab. [2](https://arxiv.org/html/2507.15321v1#S4.T2) presents the results for stereo matching. In the in-domain setting, all foundation depth models significantly improve baseline performance, with an average 4.5% EPE gain. However, in zero-shot cross-domain evaluation, not all methods generalize well. DAV2-Rel, GenPercept [15], and Marigold [7] perform best. Metric depth models, such as Metric3DV2 [14] and UniDepth [13], underperform compared to other types of depth estimation methods. Notably, DAV2-Met [8] outperforms other metric depth models, possibly benefiting from fine-tuning DAV2-Rel, despite being trained on only one dataset (Hypersim [25]). The ability of DAV2-Met to predict sharper metric depth may also contribute to its superior performance.

**Table 4: Benchmark with Simultaneous Localization and Mapping (SLAM). We select Nicer-SLAM [3] as the baseline method and apply depth predictions from various foundation models as the model input. acc and com are short for accuracy and completion, respectively. Rendered indicates that the input depth map is rendered by the dataset. We exclude Metric3DV2 and use gray for its results as it is trained with this dataset.**
| Method | rm-0 | rm-1 | rm-2 | off-0 | off-1 | off-2 | off-3 | off-4 | imp. | rank |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| acc$\downarrow$ | com$\downarrow$ | acc$\downarrow$ | com$\downarrow$ | acc$\downarrow$ | com$\downarrow$ | acc$\downarrow$ | com$\downarrow$ | acc$\downarrow$ | com$\downarrow$ | acc$\downarrow$ | com$\downarrow$ | acc$\downarrow$ | com$\downarrow$ | acc$\downarrow$ | com$\downarrow$ |  |  |  |
| w/o depth [16] | 3.37 | 3.93 | 4.01 | 4.61 | 3.58 | 3.97 | 7.26 | 8.25 | 5.82 | 6.52 | 6.98 | 7.72 | 6.98 | 6.92 | 4.26 | 6.09 | - | - |
| \cellcolorSalmonMidas [9] | 3.25 | 3.63 | 3.59 | 4.12 | 3.49 | 3.78 | 8.09 | 9.04 | 6.02 | 7.08 | 4.63 | 6.19 | 4.93 | 5.40 | 3.95 | 5.71 | +2.32 | 5 |
| \cellcolorSalmonDAV2-Rel [8] | 3.30 | 3.92 | 3.52 | 3.85 | 3.28 | 3.59 | 6.16 | 6.94 | 5.78 | 6.62 | 6.55 | 7.09 | 7.00 | 6.43 | 4.26 | 6.09 | +10.00 | 1 |
| \cellcolorGoldenrodDAV2-Met [8] | 3.22 | 3.39 | 3.48 | 3.98 | 3.47 | 3.87 | 8.58 | 9.64 | 4.59 | 5.40 | 6.38 | 7.43 | 6.13 | 5.59 | 3.98 | 6.29 | +1.95 | 6 |
| \cellcolorGoldenrodMetric3DV2 [14] | 3.48 | 3.64 | 3.45 | 3.93 | 3.73 | 4.09 | 9.55 | 10.53 | 5.82 | 6.41 | 5.20 | 6.67 | 6.73 | 6.78 | 4.51 | 6.65 | -4.19 | - |
| \cellcolorGoldenrodUniDepth [13] | 3.11 | 3.49 | 3.73 | 4.38 | 3.80 | 4.06 | 5.96 | 6.91 | 5.05 | 6.05 | 6.48 | 7.41 | 5.83 | 5.95 | 4.60 | 6.76 | +7.08 | 2 |
| \cellcolorApricotMarigold [7] | 3.01 | 3.67 | 3.77 | 4.07 | 3.70 | 4.00 | 7.07 | 7.93 | 6.23 | 7.01 | 4.83 | 6.43 | 6.32 | 6.26 | 4.52 | 6.79 | +4.67 | 4 |
| \cellcolorApricotGenPercept [15] | 3.28 | 3.47 | 3.77 | 4.34 | 3.33 | 3.73 | 7.06 | 7.65 | 4.14 | 5.06 | 4.38 | 6.35 | 5.30 | 5.05 | 4.40 | 6.20 | +6.16 | 3 |
| \cellcolorLimeGreenMoGe [10] | 3.26 | 3.67 | 3.67 | 4.23 | 3.89 | 4.33 | 8.86 | 9.83 | 4.55 | 5.58 | 5.68 | 6.73 | 6.40 | 6.32 | 3.92 | 5.98 | -4.04 | 7 |
| Rendered | 3.00 | 3.29 | 3.69 | 4.41 | 4.14 | 4.47 | 5.57 | 6.85 | 5.95 | 6.75 | 5.91 | 7.91 | 6.64 | 6.65 | 4.01 | 6.05 | - | - |

Figure: Figure 4: Showcases of ChatGPT-4o on SpatialBot positional benchmark. We highlight the text prompt describing rendered depth map in blue and mistakes made by ChatGPT-4o in red, respectively. In the first case, ChatGPT-4o correctly answers the question but misinterprets the depth map despite detailed prompts. As for the second one, despite correctly parsing the depth map, ChatGPT-4o provides an incorrect answer.
Refer to caption: x5.png

**Table 5: Benchmark with spatial understanding of Vision Language Model (VLM). We evaluate the effectiveness of depth predictions from various foundation models on the SpatialBench [46]. The rank column is omitted since all depth models perform similarly.**
| Method | Pos.$\uparrow$ | Exist$\uparrow$ | Count$\uparrow$ | Reach$\uparrow$ | Size$\uparrow$ | Method | Pos.$\uparrow$ | Exist$\uparrow$ | Count$\uparrow$ | Reach$\uparrow$ | Size$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ChatGPT-4o | 64.70 | 95.00 | 80.88 | 54.44 | 31.11 | SpatialBot [46] | 61.76 | 75.00 | 92.41 | 51.67 | 28.33 |
| \cellcolorSalmonMidas [9] | 62.74 | 90.00 | 80.26 | 54.44 | 37.22 | \cellcolorSalmonMidas [9] | 55.88 | 55.00 | 92.41 | 46.67 | 30.00 |
| \cellcolorSalmonDAV2-Rel [8] | 61.76 | 88.33 | 77.11 | 52.22 | 35.55 | \cellcolorSalmonDAV2-Rel [8] | 55.88 | 60.00 | 93.13 | 46.67 | 30.00 |
| \cellcolorGoldenrodDAV2-Met [8] | 61.76 | 86.66 | 80.44 | 59.44 | 38.88 | \cellcolorGoldenrodDAV2-Met [8] | 55.88 | 65.00 | 93.13 | 45.00 | 28.33 |
| \cellcolorGoldenrodMetric3DV2 [14] | 62.74 | 88.33 | 79.45 | 59.44 | 28.88 | \cellcolorGoldenrodMetric3DV2 [14] | 58.82 | 55.00 | 93.13 | 50.00 | 28.33 |
| \cellcolorGoldenrodUniDepth [13] | 64.70 | 93.33 | 80.55 | 62.22 | 37.77 | \cellcolorGoldenrodUniDepth [13] | 58.82 | 60.00 | 92.41 | 53.33 | 28.33 |
| \cellcolorApricotMarigold [7] | 57.84 | 83.33 | 80.68 | 58.88 | 31.66 | \cellcolorApricotMarigold [7] | 55.88 | 60.00 | 93.13 | 46.67 | 30.00 |
| \cellcolorApricotGenPercept [15] | 60.78 | 85.00 | 81.03 | 57.77 | 37.77 | \cellcolorApricotGenPercept [15] | 55.88 | 65.00 | 93.13 | 48.33 | 28.33 |
| \cellcolorLimeGreenMoGe [10] | 60.78 | 85.00 | 79.06 | 56.11 | 33.33 | \cellcolorLimeGreenMoGe [10] | 55.88 | 60.00 | 93.13 | 50.00 | 28.33 |

Feed-Forward Monocular 3DGS. Tab. [3](https://arxiv.org/html/2507.15321v1#S4.T3) shows the benchmark results. DAV2-Met achieves better performance compared with DAV2-Rel, suggesting that metric depth properties are beneficial for novel view synthesis tasks in real 3D environments. MiDaS [9], despite being an older method, performs remarkably well with a rank of 1. DAV2-Rel also achieves strong results but slightly underperforms compared to MiDaS. Most metric depth methods, except for DAV2-Met and affine-invariant depth methods, fail to improve the baseline.

Simultaneous Localization and Mapping. Tab. [4](https://arxiv.org/html/2507.15321v1#S5.T4) presents the SLAM results. DAV2-Rel achieves the best results with a promising gap with other methods, indicating a superior potential for this task. UniDepth achieves the second best results, highlighting the importance of metric depth for this task. GenPercept also obtains good results, possibly due to fine-tuning on Hypersim, a similar synthetic dataset. The performance gap between GenPercept and Marigold highlights the effectiveness of its fine-tuning strategy.

VLM Spatial Understanding. We use SpatialBench [46] for this task. Unlike its original purpose of benchmarking different vision-language models (VLMs), we focus on evaluating the effectiveness of different depth estimations for the same VLM. We select ChatGPT-4o and SpatialBot [46] as baseline VLMs, without and with depth inputs during training, respectively.

Surprisingly, for both VLMs, adding depth as an additional input does not significantly improve performance, even in SpatialBot, which is trained with depth maps. All depth methods yield similar results, indicating similar effectiveness for this high-level spatial reasoning task. Fig. [4](https://arxiv.org/html/2507.15321v1#S5.F4) illustrates two cases from the positional benchmark in SpatialBench. In the first case, ChatGPT-4o correctly answers the question but misinterprets the depth map despite detailed prompts, suggesting that the training-stage with depth signals is crucial for the proper usage of depth maps. In the second case, despite correctly parsing the depth map, ChatGPT-4o provides an incorrect answer, highlighting VLMs’ current limitations in reasoning within 3D space, even when given accurate spatial information.

## 6 Limitations and Future Work

While BenchDepth provides a more practical evaluation framework for depth foundation models (DFMs) by leveraging downstream proxy tasks, it also introduces certain challenges. First, training on downstream tasks is computationally expensive and time-consuming. An evaluation model that can predict downstream score would alleviate this issue. Second, while we carefully selected five diverse proxy tasks to assess different aspects of depth estimation, the current set of tasks may not fully capture all potential applications of DFMs. In future work, we plan to expand BenchDepth by incorporating additional downstream tasks to further explore the capabilities.

## 7 Conclusion

We introduced BenchDepth, a benchmark for evaluating depth foundation models (DFMs) through downstream proxy tasks rather than alignment-based metrics. By benchmarking eight SoTA DFMs across depth completion, stereo matching, 3D scene reconstruction, SLAM, and vision-language spatial understanding, we provide a fairer and more practical assessment of their effectiveness. Our experiments reveal key insights into the performance improvement of DFMs in real-world applications as shown in Sec. [1](https://arxiv.org/html/2507.15321v1#S1). By shifting depth evaluation towards real-world utility, we hope BenchDepth inspires further research, encouraging the community to rethink evaluation strategies for DFMs.

## References

- [1]
L. Zhang, A. Rao, and M. Agrawala, “Adding conditional control to text-to-image diffusion models,” in ICCV, pp. 3836–3847, 2023.
- [2]
Y. Li, Z. Ge, G. Yu, J. Yang, Z. Wang, Y. Shi, J. Sun, and Z. Li, “Bevdepth: Acquisition of reliable depth for multi-view 3d object detection,” in AAAI, vol. 37, pp. 1477–1485, 2023.
- [3]
Z. Zhu, S. Peng, V. Larsson, Z. Cui, M. R. Oswald, A. Geiger, and M. Pollefeys, “Nicer-slam: Neural implicit scene encoding for rgb slam,” in 2024 International Conference on 3D Vision (3DV), pp. 42–52, IEEE, 2024.
- [4]
S. Szymanowicz, E. Insafutdinov, C. Zheng, D. Campbell, J. F. Henriques, C. Rupprecht, and A. Vedaldi, “Flash3d: Feed-forward generalisable 3d scene reconstruction from a single image,” arXiv preprint arXiv:2406.04343, 2024.
- [5]
D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” NeurIPS, vol. 27, 2014.
- [6]
S. F. Bhat, R. Birkl, D. Wofk, P. Wonka, and M. Müller, “Zoedepth: Zero-shot transfer by combining relative and metric depth,” arXiv preprint arXiv:2302.12288, 2023.
- [7]
B. Ke, A. Obukhov, S. Huang, N. Metzger, R. C. Daudt, and K. Schindler, “Repurposing diffusion-based image generators for monocular depth estimation,” in CVPR, pp. 9492–9502, 2024.
- [8]
L. Yang, B. Kang, Z. Huang, Z. Zhao, X. Xu, J. Feng, and H. Zhao, “Depth anything v2,” arXiv preprint arXiv:2406.09414, 2024.
- [9]
R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” IEEE TPAMI, vol. 44, no. 3, 2022.
- [10]
R. Wang, S. Xu, C. Dai, J. Xiang, Y. Deng, X. Tong, and J. Yang, “Moge: Unlocking accurate monocular geometry estimation for open-domain images with optimal training supervision,” arXiv preprint arXiv:2410.19115, 2024.
- [11]
J. Wang, M. Chen, N. Karaev, A. Vedaldi, C. Rupprecht, and D. Novotny, “Vggt: Visual geometry grounded transformer,” in CVPR, pp. 5294–5306, 2025.
- [12]
Y. Ge, G. Xu, Z. Zhao, L. Sun, Z. Huang, Y. Sun, H. Chen, and C. Shen, “Geobench: Benchmarking and analyzing monocular geometry estimation models,” arXiv preprint arXiv:2406.12671, 2024.
- [13]
L. Piccinelli, Y.-H. Yang, C. Sakaridis, M. Segu, S. Li, L. Van Gool, and F. Yu, “Unidepth: Universal monocular metric depth estimation,” in CVPR, pp. 10106–10116, 2024.
- [14]
M. Hu, W. Yin, C. Zhang, Z. Cai, X. Long, H. Chen, K. Wang, G. Yu, C. Shen, and S. Shen, “Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation,” IEEE TPAMI, 2024.
- [15]
G. Xu, Y. Ge, M. Liu, C. Fan, K. Xie, Z. Zhao, H. Chen, and C. Shen, “What matters when repurposing diffusion models for general dense perception tasks?,” arXiv preprint arXiv:2403.06090, 2024.
- [16]
J.-H. Park, C. Jeong, J. Lee, and H.-G. Jeon, “Depth prompting for sensor-agnostic depth estimation,” in CVPR, pp. 9859–9869, 2024.
- [17]
H. Jiang, Z. Lou, L. Ding, R. Xu, M. Tan, W. Jiang, and R. Huang, “Defom-stereo: Depth foundation model based stereo matching,” arXiv preprint arXiv:2501.09466, 2025.
- [18]
J. Cheng, L. Liu, G. Xu, X. Wang, Z. Zhang, Y. Deng, J. Zang, Y. Chen, Z. Cai, and X. Yang, “Monster: Marry monodepth to stereo unleashes power,” arXiv preprint arXiv:2501.08643, 2025.
- [19]
J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
- [20]
J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in International conference on machine learning, pp. 19730–19742, PMLR, 2023.
- [21]
K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in CVPR, pp. 9729–9738, 2020.
- [22]
M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al., “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023.
- [23]
G. Xu, X. Wang, X. Ding, and X. Yang, “Iterative geometry encoding volume for stereo matching,” in CVPR, pp. 21919–21928, 2023.
- [24]
Y. Zuo, K. Kayan, M. Wang, K. Jeon, J. Deng, and T. L. Griffiths, “Towards foundation models for 3d vision: How close are we?,” arXiv preprint arXiv:2410.10799, 2024.
- [25]
M. Roberts, J. Ramapuram, A. Ranjan, A. Kumar, M. A. Bautista, N. Paczan, R. Webb, and J. M. Susskind, “Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding,” in ICCV, pp. 10912–10922, 2021.
- [26]
Z. Li, S. F. Bhat, and P. Wonka, “Patchrefiner: Leveraging synthetic data for real-domain high-resolution monocular metric depth estimation,” arXiv preprint arXiv:2406.06679, 2024.
- [27]
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in CVPR, pp. 10684–10695, 2022.
- [28]
A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The kitti dataset,” The international journal of robotics research, vol. 32, no. 11, pp. 1231–1237, 2013.
- [29]
N. Silberman, D. Hoiem, P. Kohli, and R. Fergus, “Indoor segmentation and support inference from rgbd images,” in ECCV, pp. 746–760, Springer, 2012.
- [30]
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in CVPR, pp. 3354–3361, IEEE, 2012.
- [31]
M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in CVPR, pp. 3213–3223, 2016.
- [32]
Z. Li, Z. Chen, X. Liu, and J. Jiang, “Depthformer: Exploiting long-range correlation and local information for accurate monocular depth estimation,” Machine Intelligence Research, pp. 1–18, 2023.
- [33]
S. F. Bhat, I. Alhashim, and P. Wonka, “Adabins: Depth estimation using adaptive bins,” in CVPR, pp. 4009–4018, 2021.
- [34]
Z. Li, S. F. Bhat, and P. Wonka, “Patchfusion: An end-to-end tile-based framework for high-resolution monocular metric depth estimation,” arXiv preprint arXiv:2312.02284, 2023.
- [35]
W. Chen, Z. Fu, D. Yang, and J. Deng, “Single-image depth perception in the wild,” NeurIPS, vol. 29, 2016.
- [36]
H. Fu, M. Gong, C. Wang, K. Batmanghelich, and D. Tao, “Deep ordinal regression network for monocular depth estimation,” in CVPR, pp. 2002–2011, 2018.
- [37]
Z. Li, X. Wang, X. Liu, and J. Jiang, “Binsformer: Revisiting adaptive bins for monocular depth estimation,” arXiv preprint arXiv:2204.00987, 2022.
- [38]
J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020.
- [39]
L. Yang, B. Kang, Z. Huang, X. Xu, J. Feng, and H. Zhao, “Depth anything: Unleashing the power of large-scale unlabeled data,” arXiv preprint arXiv:2401.10891, 2024.
- [40]
S. Wang, V. Leroy, Y. Cabon, B. Chidlovskii, and J. Revaud, “Dust3r: Geometric 3d vision made easy,” in CVPR, pp. 20697–20709, 2024.
- [41]
W. Cong, Y. Liang, Y. Zhang, Z. Yang, Y. Wang, B. Ivanovic, M. Pavone, C. Chen, Z. Wang, and Z. Fan, “E3d-bench: A benchmark for end-to-end 3d geometric foundation models,” arXiv preprint arXiv:2506.01933, 2025.
- [42]
C. L. Lawson and R. J. Hanson, Solving least squares problems.
SIAM, 1995.
- [43]
M. T. Heath, Scientific computing: an introductory survey, revised second edition.
SIAM, 2018.
- [44]
B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering.,” ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023.
- [45]
Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar, “Neural fields in visual computing and beyond,” in Computer Graphics Forum, vol. 41, pp. 641–676, Wiley Online Library, 2022.
- [46]
W. Cai, I. Ponomarenko, J. Yuan, X. Li, W. Yang, H. Dong, and B. Zhao, “Spatialbot: Precise spatial understanding with vision language models,” arXiv preprint arXiv:2406.13642, 2024.
- [47]
G. M. Garcia, K. A. Zeid, C. Schmidt, D. de Geus, A. Hermans, and B. Leibe, “Fine-tuning image-conditional diffusion models is easier than you think,” arXiv preprint arXiv:2409.11355, 2024.
- [48]
N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox, “A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation,” in CVPR, pp. 4040–4048, 2016.
- [49]
D. Scharstein, H. Hirschmüller, Y. Kitajima, G. Krathwohl, N. Nešić, X. Wang, and P. Westling, “High-resolution stereo datasets with subpixel-accurate ground truth,” in Pattern Recognition: 36th German Conference, GCPR 2014, Münster, Germany, September 2-5, 2014, Proceedings 36, pp. 31–42, Springer, 2014.
- [50]
T. Schops, J. L. Schonberger, S. Galliani, T. Sattler, K. Schindler, M. Pollefeys, and A. Geiger, “A multi-view stereo benchmark with high-resolution images and multi-camera videos,” in CVPR, pp. 3260–3269, 2017.
- [51]
O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” 2015.
- [52]
T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, “Stereo magnification: Learning view synthesis using multiplane images,” arXiv preprint arXiv:1805.09817, 2018.
- [53]
J. Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, et al., “The replica dataset: A digital replica of indoor spaces,” arXiv preprint arXiv:1906.05797, 2019.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 2
presentation: 3
questions: 1. Can you clarify on negative correlations between proposed benchmarks and standard metrics? This is my main concern that needs to be addressed.
2. Can you provide a table explicitly listing training datasets for each DFM and indicating where they overlap with each downstream task’s evaluation data?
3. Did you consider re-running one downstream task using an alternative baseline architecture to check whether performance rankings are stable across pipelines?
soundness: 1
strengths: 1. The work addresses a well-recognized gap: depth metrics (e.g., AbsRel, RMSE, δ thresholds) do not reliably predict whether a depth model is useful when integrated into 3D or geometric pipelines. Positioning evaluation around actual deployment tasks is well justified.

2. Diverse and well-chosen downstream tasks. The inclusion of tasks spanning low-level to high-level reasoning adds credibility to the claim that the benchmark captures general utility rather than performance tuned to a particular application.
summary: The authors presented a benchmark framework evaluating depth foundation models (DFMs) across a diverse set of downstream proxy tasks (depth completion, stereo matching, 3D Gaussian splatting reconstruction, learning-based SLAM, and VLM spatial reasoning). The benchmark aims to test practical utility rather than purely pixelwise accuracy, and the authors report that performance rankings under downstream tasks differ from those captured by standard depth metrics.
weaknesses: 1. Interpretation of the reported negative correlation between traditional metrics and downstream rankings requires clarification. The paper claims that standard depth metrics negatively correlate with downstream task performance. Taken at face value, this would imply that models performing worse under standard metrics could perform better in real applications, which is conceptually difficult to justify. 

2. The benchmark only uses one baseline architecture per task. Performance may depend on model–pipeline interactions (e.g., certain DFMs may integrate more naturally into particular stereo or SLAM architectures). Using a single model per task risks overstating generality of the results.

3. There is a potential dataset overlap between DFMs’ training data and downstream task datasets. Some DFMs could have been trained or fine-tuned on datasets closely related to those used in the downstream evaluations. This may weaken conclusions, as models might benefit from dataset familiarity rather than intrinsic superiority as transferrable depth priors.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 3
questions: 1. Can you provide a principled explanation or analysis (beyond empirical observation) for the decoupling between geometric accuracy and downstream performance?
2. Have you tested, if replacing the depth maps with noisy or random ones yields similar weak correlations, to confirm that the benchmark truly measures depth utility rather than task variance? Many downstream networks may not be sensitive to the input depth maps. If that is the case, then even when provided with random or noisy depth maps, they might still produce similar outputs.
soundness: 2
strengths: 1. The paper shifts the benchmark focus from geometry-based metrics to application-driven performance, which brings a new insight for the community.
2. The benchmark covers both low-level and high-level depth tasks, which provides a multi-layered perspective on depth model utility. 
3. The paper presents detailed quantitative analysis and interesting observations, such as strong internal consistency between proxy tasks but weak correlation with traditional benchmarks.
4. The multiple aspects of this benchmark could serve for evaluating DFMs beyond pure geometric fidelity for other works.
summary: This paper introduces BenchDepth to evaluate depth foundation models based on downstream task performance, rather than traditional geometric accuracy metrics.
To assess how well DFMs perform in downstream tasks, the authors propose five proxy tasks: depth completion, stereo matching, feed-forward 3D reconstruction, SLAM, and vision-language spatial understanding.
Results show weak or even negative correlations between traditional geometric metrics and downstream task performance. The authors claim that geometry-centric evaluation may not reflect real-world effectiveness.
weaknesses: 1. Depth estimation is fundamentally and inherently a metric prediction problem (Unlike LLMs and VLMs mentioned in line 49 – 52, whose evaluation rely on plausibility or relevance rather than quantitative accuracy). Thus, any evaluation suggesting the opposite conclusion must explain why geometric fidelity loses its predictive power. The authors fail to provide a compelling theoretical basis for this discrepancy, leaving open the possibility that the issue comes from their task integration rather than the metric itself.
2. Many downstream tasks such as SLAM and 3DGS contain complex neural components that can overshadow the role of depth inputs. As a result, the reported low correlation might reflect task noise rather than a true limitation of geometric benchmarks. Deeper analysis is still needed for this paper.
3. The benchmark only coversdoes not cover recent DFMs, e.g., CH3Depth [1]
[1] CH3Depth: Efficient and Flexible Depth Foundation Model with Flow Matching.
4. The paper is almost entirely empirical. The conceptual analysis of which geometric properties (such as scale, continuity, edge fidelity) actually matter for different tasks may build a bridge between geometric accuracy and downstream task performance, which is needed in this paper.
5. The authors themselves acknowledge that all DFMs perform similarly in VLM spatial understanding. This reduces the claimed significance of BenchDepth, as high-level reasoning tasks do not meaningfully discriminate model quality.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: The weakness contains a list of the issues I'm concerned with. It would be helpful if these were either addressed or you explained why they didn't matter.

On top of this, the writing needs tightening. 

>Our correlation analysis (Fig. 1) shows stronger consistency among proxy tasks (e.g., depth
completion and SLAM: 0.88), indicating that the selected five tasks collectively form a
representative and coherent benchmark.

The correlation of selected measures, combined with the negative correlation of other measures, indicates that the benchmark is *not* representative of standard tasks in the literature. By no means is this a bad thing. Coherence is probably a better indication of benchmark usefulness. It's better to measure one thing well rather than a couple of things badly.

It's also not clear how meaningful the negative correlation in previous tasks is. Presumably, we could introduce trivial models such as constant depth, or even older models that perform badly on all tasks. This would then result in a positive correlation on all tasks. 

Is this negative correlation just an example of a publication bias resulting in a collider bias? To be published, you need to be tuned to perform well on one of these types of task but not both, and that results in an apparent negative correlation.
soundness: 1
strengths: Improved benchmarking in monocular reconstruction is very welcome. 

The use of correlation to identify a coherent subset of tasks that make a benchmark is sensible.

Generally, I appreciate the design decisions, and I think this will make a valuable contribution to the literature when it is finished.
summary: Paper proposes a novel benchmark for evaluating monocular reconstruction, based on five tasks. They perform correlation analysis over 8 models and 6 tasks, and show that four of them are strongly correlated and represent a coherent benchmark when averaged.

This is an interesting paper, and I like the direction, but it is incomplete.

The paper reports results for five results, one is excluded from the benchmark results in figure 1, and the two negative tasks are not reported.

The average of the positively correlated tasks is not reported, and presumably this is the actual benchmark. As such, no one can compare new methods on the benchmark.
weaknesses: The coherence of the benchmark depends on the correlation analysis highlighted in Figure 1, and this is incomplete.

Task 5 in the text (VLM Spatial Understanding) is excluded from the figure, and the justification for excluding it doesn't really make sense. While the measure "Pos" shows little variation between the models, many of the other metrics show extremely large shifts. 

Tasks 5,6,7 in Figure 1 (average, traditional delta, and traditional absrel) are not reported in the text.

In addition to this, the correlation analysis is performed using only 8 models. This doesn't automatically invalidate the statistics, but it means that the authors need to be very careful that one outlier isn't dominating the analysis. The authors need to generate plots for each pair of benchmarks in the analysis and confirm that the expected monotonic relationship exists. 

One model is excluded from the SLAM benchmark table, Metric3DV2 and it is not clear how this was dealt with in the correlation analysis.

The negatively correlated benchmarks are also only reported on a subset of models, in other papers, and it's not clear how this altered the correlation analysis. The authors really need to bite the bullet and run all models on these benchmarks.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: 1. What specific research question or hypothesis are you addressing?

2. Is BenchDepth publicly released and standardized for reproducible evaluation, or is it a one-time experimental setup?

3. How can future researchers extend or adopt this benchmark. Is there a unified scoring or evaluation pipeline?

4. Can you provide qualitative or conceptual insights explaining why DFMs behave differently across tasks?
soundness: 4
strengths: 1. The paper targets an important and timely question regarding how to evaluate large-scale depth foundation models. Through experimental results, the paper identifies an evaluation gap between geometric accuracy and downstream performance on different tasks, which is an interesting observation.

2. The authors perform a large and comprehensive experimental study, covering multiple downstream tasks and eight modern DFMs. The empirical effort is substantial, and the collected results could serve as a useful reference dataset for the community. I appreciate the authors' effort in delivering the results.
summary: This paper introduces BenchDepth, a benchmark designed to evaluate depth foundation models (DFMs) through downstream tasks rather than traditional geometric metrics. The benchmark comprises five proxy tasks: depth completion, stereo matching, 3D scene reconstruction, SLAM, and vision-language spatial reasoning, aiming to capture the practical utility of DFMs in real-world applications. Eight recent DFMs are evaluated, and the authors report that model rankings differ significantly from conventional geometric benchmarks, highlighting a disconnect between numerical geometry metrics and downstream task effectiveness.
weaknesses: 1. The core research question is posed in the title (“Are we on the right way to evaluate DFMs?”). However, after carefully reading the paper, I find this question never brought up in the paper, never addressed in the paper, never answered in the paper, and never discussed in-depth in the paper. Sure, observations made by the paper (mismatch between current evaluation protocols and downstream performance) are valuable to this question, yet the discussion ends up right there without much high-level or technical insight. The paper reads as an experimental report rather than a proper research study. The paper states, "We hope that our work will spark further discussion (L119)", yet the discussion in the paper is minimal. No strong takeaways or principles explain why some DFMs perform better in certain downstream tasks.

2. Following 1, the technical depth is limited. The work mainly aggregates results from existing models and tasks without proposing new evaluation metrics, theories, or methodological innovations. Thus, the paper fails to clarify the intended contribution: is BenchDepth meant as a benchmark/tool or an analytical study? As a benchmark, details about usability, reproducibility, and release are missing. As an analytical study, it only contains shallow observations.

3. The writing and structure are weak. The introduction reads like related work, and the related work reads like an extension of the introduction. As 2. motivation and positioning are underdeveloped.