Benchmarking and Analyzing Monocular Geometry Estimation Models

## Abstract

Abstract Recent advances in discriminative and generative pretraining have yielded geometry estimation models with strong generalization capabilities. While discriminative monocular geometry estimation methods rely on large-scale fine-tuning data to achieve zero-shot generalization,
several generative-based paradigms show the potential of achieving impressive generalization performance on unseen scenes by leveraging pre-trained diffusion models and fine-tuning on even a small scale of synthetic training data.
Frustratingly, these models are trained with different recipes on different datasets, making it hard to find out the critical factors that determine the evaluation performance.
Besides, the current widely used geometry evaluation benchmarks have two main drawbacks that may prevent the development of the field, i.e ., limited scene diversity and unfavorable label quality .
To resolve the above issues, (1) we build fair and strong baselines in a unified codebase for evaluating and analyzing the state-of-the-art (SOTA) geometry estimation models in terms of both different finetuning paradigms and training recipes;
(2) we evaluate monocular geometry estimators on more challenging benchmarks for geometry estimation task with
diverse scenes and high-quality annotations. Our results reveal that pre-trained using large data, discriminative models such as DINOv2, can outperform generative counterparts with a small amount of high-quality synthetic training data under the same training configuration, which suggests that fine-tuning data quality is a more important factor than the data scale and model architecture.
Our observation also raises a question: if simply fine-tuning a general vision model such as DINOv2 using a small amount of synthetic depth data produces SOTA results, do we really need complex models, e.g., Marigold [ 32 ] and DepthFM [ 18 ] for depth estimation? We believe that this work can propel advancements in geometry estimation tasks
and a wide range of other downstream vision tasks.

## 1 Introduction

Monocular depth and surface normal estimation, also referred to as “monocular geometry estimation”, poses a fundamental yet intricate challenge of inferring distance and surface orientation from a single image. Its significance is underscored by its broad utility across various downstream tasks, including object detection [, , ], visual navigation [, , , ], novel view synthesis [, ], controllable image generation [, , ], and 3D scene reconstruction [, ]. The importance of this task has led to a significant body of research, resulting in numerous models [, , , , ] over the past decade.

Although a large number of monocular geometry estimation models exist, they can be divided into two paradigms, *i.e*., discriminative-based and generative-based.
Most contemporary discriminative monocular geometry estimation models leverage the pre-train priors from fully-supervised image classification backbones, *e.g*., ConvNeXt [], EfficientNet [] and ViT [], or self-supervised backbones. *e.g*., DINOv2 [], previous best discriminative depth estimation models scale-up their performance with pre-trained DINOv2 backbone and a large scale of fine-tuning data. Generative geometry estimation
models [, , , , ] unleash the power of pre-trained text-to-image diffusion models, *e.g*., Stable Diffusion (SD) [].
Several generative geometry estimation models [, , , ] show strong generation capability with even a small-scale high-quality synthetic fine-tuning data.

However, none of the previous works have systematically investigated the performance of these geometry estimation methods with fair and faithful comparison.
The reason is twofold. Firstly, the different selections of datasets and training configurations hinder the fair evaluations of the newly designed methodologies. (1) The performance distinction for different generative-based finetuning paradigms is unclear. It is hard to evaluate whether the actual improvement is from the algorithmic perspective or the data perspective since they are trained on different datasets and different training configurations. (2) The performance distinction between discriminative and generative geometry estimation models when trained on the same scale and quality of data also remains unclear.
Secondly, existing popular geometry estimation benchmarks may not reveal the real performance of the models. NYUv2 [] and ScanNet [] are still popular in the evaluation of indoor monocular depth estimation. However, they are collected by an older Kinect-v1 system with noisy depth measurements and noisy imaging for RGB patterns, with only $640\times 480$ resolution.
DIODE [] and ETH3D [] collect both outdoor and indoor scenes with high-quality data while with low diversity scenes for evaluation.
KITTI [] collects depth maps from the LIDAR sensor and focuses on outdoor driving scenes.
For surface normal evaluation, NYUv2 [], ScanNet [], iBims-1 [], Sintel [] and Virtual KITTI [] are widely used by generating surface normal maps from the ground truth depth maps. However, the depth noises in NYUv2 [], ScanNet [] and iBims-1 [] yield unsatisfactory surface normal ground truth. The limited scene diversity of synthetic datasets, *i.e*., Sintel [] and Virtual KITTI [], cannot evaluate the robustness of the surface normal estimation model for in-the-wild geometry reconstruction. Overall, the existing geometry benchmarks are hindered by two main issues: ground-truth quality and scene diversity. This lack of fair and comprehensive benchmarks can significantly impede the development of geometry estimation research.

To address the aforementioned problems, we perform a comprehensive geometry estimation benchmarking study from two perspectives:

- 1.
Training strategy. We reimplement a comprehensive list of
SOTA algorithms in a unified codebase, including Marigold [], Geowizard [], GenPercept [], DepthFM [], DMP [], Depth-Anything [] and DSINE []. As such, we can fairly evaluate their performance under the same training configuration, and figure out whether the performance improvement is coming from the model architecture or coming from the high-quality training data. Previous generative geometry models are all based on Stable Diffusion 2.1 [] with limited trained data (compared to the data scale of discriminative models), we further explore the potential of generative geometry models by conducting data scale-up and model scale-up ablations.
- 2.
More benchmark datasets. Apart from traditional geometry evaluation benchmarks, we build more diverse scenes with high-quality labels for geometry evaluation. For depth estimation, we introduce three extra benchmark datasets, InSpaceType [], MatrixCity [], and Infinigen []. InSpaceType is an
indoor depth evaluation benchmark, which contains 12 scenes, 1260 images, and $2208\times 1242$ resolution. It is a good complement for indoor benchmarks like NYUv2 and ScanNet. MatrixCity is a rendered dataset with real city-scale scenes, we select 808 street images and 403 aerial images for evaluation. It is suitable for evaluating driving and city scenes. Infinigen is also a high-quality rendered dataset, which contains diverse nature scenes. We use it to verify the generalization capability of depth estimation foundation models in wild scenes. For surface normal estimation, we expand existing benchmark datasets with more high quality and diverse datasets, *e.g*., indoor MuSHRoom dataset [], outdoor Tank and Temples (T&T) dataset [](^1^11The surface normal annotation of MushRoom and T&T are obtained from Gaustudio []), and wild Infinigen [] dataset.

With the unified codebase, training data, and comprehensive benchmark datasets, we surprisingly find that:

- 1.
Without bells and whistles, the
depth estimation model pre-trained on the DINOv2 [] backbone, followed by a simple DPT head [], can already achieve great generalization performance by using only a small amount of high-quality synthetic datasets ($77$K training samples).
With the same training configuration, the discriminative model outperforms the recent generative counterpart, Marigold [].
Besides, the result is a sharp contrast to previous discriminative models using large data to achieve generalization capability, *e.g*., Metric3Dv2 [], which focuses on collecting more diverse training datasets ($16$M training samples), and Depth-Anything [], which focuses on scaling up model performance with large-scale pseudo labels ($63.5$M training samples). Based on the result, we argue that training data quality may be more important than the data scale.
- 2.
Generative-based geometry estimation models can generate high-resolution and detailed depth and surface normal maps, which is an advantage over the existing discriminative depth estimation models. We mainly attribute this to high-quality synthetic fine-tuning data and the design of the VAE [] decoder, which can decode compressed depth latent to the same resolution depth map as the input image.
- 3.
For surface normal estimation, the discriminative model DSINE [] outperforms generative-based fine-tuning protocols with the same training configuration, suggesting that, apart from large-scale pre-training, inductive bias [] can also be an important factor in providing rich
information for surface normal estimation tasks.

We hope that our benchmarking results can
pave the way for designing more powerful geometry estimation algorithms and developing more high-quality geometry estimation training datasets in the future.

## 2 Preliminaries

Task definition.
Given an input image $x\in\mathbb{R}^{H\times W\times 3}$, the goal of monocular geometry estimation is to predict the depth map $d\in\mathbb{R}^{H\times W}$, which can be affine-invariant or metric depth, and surface orientation, which can be represented as either a unit vector $\mathbf{n}\in S^{2}$, or a 3D axis-angle $\mathbf{R}\in SO(3)$.

Discriminative geometry estimation models.
With the widespread application of deep learning [], learning-based methods have demonstrated their ability to estimate geometric information from monocular images [, , , ]. Early works primarily relied on discriminative models using either supervised or unsupervised methods. Eigen et al. [] proposed the first learning-based method for monocular depth estimation, employing two deep network stacks and using ground truth depth for supervision. Zhou et al. proposed an early unsupervised framework, SfMLearner [], in which camera pose and monocular depth are learned together. With the availability of large amounts of data, recent methods [, , , ] have shown a trend toward using large-scale datasets to develop robust geometry estimation models that generalize well to diverse environments. For instance, Ranftl et al. [] introduced a method that demonstrates strong zero-shot testing ability by utilizing mixed training datasets. Yang et al. [] further improved zero-shot monocular depth estimation performance by proposing Depth-Anything, which leverages large-scale data to achieve strong generalization ability. Meanwhile, Yin et al. [, ] proposed Metric3D, which can output accurate metric depth by training models on 11 public RGB-D datasets. Apart from depth estimation, advancements in surface normal information have also been achieved through the use of discriminative models. Surface normal information can not only be calculated directly from depth maps but can also be independently obtained through surface normal estimation techniques [, , , ]. For example, Bae et al. [] proposed a method that demonstrates strong generalization capabilities and produces high-quality surface normal predictions by investigating inductive biases. Overall, the use of discriminative models for both depth and surface normal estimation has shown its significance in improving performance, thereby broadening the applications of monocular geometry estimation.

Generative geometry estimation Models.
Given the impressive results of recent generative models [] in image generation tasks, many studies have endeavored to incorporate generative-based pipelines into geometry estimation. Ji et al. [] proposed a method to extend the denoising diffusion process into the modern perception pipeline, which can be generalized to most dense prediction tasks, such as depth estimation. Saxena et al. [] formulated optical flow and monocular depth estimation as image-to-image translation using generative diffusion models, without specialized loss functions and model architectures. Zhao et al. [] proposed VPD, a framework that exploits the semantic information of a pre-trained text-to-image diffusion model in visual perception tasks. Ke et al. [] introduced a method for affine-invariant monocular depth estimation, where the depth information is derived from retained rich stable diffusion priors. Fu et al. [] proposed a foundation model for jointly estimating depth and surface normal from monocular images, which not only achieves surprisingly robust generalization on various types of real or synthetic images but also faithfully captures intricate geometric details.
In summary, recent generative-based methods have provided new solutions and demonstrated their applications for depth estimation.

Geometric evalutaion metrics.
We use widely adopted evaluation metrics for assessing the performance of depth and surface normal estimation. Specifically, for the depth estimation task, we use mean absolute relative error (AbsRel) and accuracy under thresholds ($\delta_{i}<1.25^{i},i=1,2,3$) for accuracy comparisons. These evaluation metrics for depth estimation are calculated as follows:
(1) mean absolute relative error (AbsRel): $\frac{1}{n}\sum_{i=1}^{n}\frac{\left|z_{i}-z_{i}^{\ast}\right|}{z_{i}^{\ast}}$;
(2) the accuracy under threshold ($\delta_{i}<1.25^{i},i=1,2,3$): $\%$ of $z_{i}$ s.t. max$\left(\frac{z_{i}}{z_{i}^{\ast}},\frac{z_{i}^{\ast}}{z_{i}}\right)<1.25^{i}$;
where $z_{i}$ is the ground truth depth and $z_{i}^{\ast}$ represents the predicted depth. For surface normal estimation, we calculate the angular error for the pixels with ground truth and report both the median and mean values (lower is better). In addition, we measure the percentage of pixels with an error below $t\in[5.0^{\circ},11.25^{\circ},30.0^{\circ}]$ (higher is better). Please refer to [] for calculation details.

## 3 Benchmarking Depth Estimation Foundation Models

### 3.1 A Brief Overview of SOTA Methods

**Table 1: Quantitative comparison on 5 zero-shot affine-invariant depth benchmarks with author released weights. We mark the best results in bold and the second best underlined. Discriminative methods are colored in blue while generative ones in green.**
| Method | Train<br>Samples | Year | NYUv2 [] | KITTI [] | ETH3D [] | ScanNet [] | DIODE [] |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ |  |  |  |
| Metric3Dv2 [] | 16M | arXiv’24 | 3.9 | 97.9 | 5.2 | 97.9 | 4.0 | 98.3 | 2.3 | 98.9 | 14.7 | 89.2 |
| DepthAnything [] | 63.5M | CVPR’24 | 4.3 | 98.0 | 8.0 | 94.6 | 5.8 | 98.4 | 4.3 | 98.1 | 26.1 | 75.9 |
| Marigold [] | 74K | CVPR’24 | 5.5 | 96.4 | 9.9 | 91.6 | 6.5 | 96.0 | 6.4 | 95.1 | 30.8 | 77.3 |
| GeoWizard [] | 280K | arXiv’24 | 5.9 | 95.9 | 12.9 | 85.1 | 7.7 | 94.0 | 6.6 | 95.3 | 32.8 | 75.3 |
| GenPercept [] | 74K | arXiv’24 | 6.3 | 96.0 | 13.3 | 84.1 | 7.2 | 95.5 | 6.6 | 96.0 | 32.3 | 76.0 |
| DepthFM [] | 63K | arXiv’24 | 8.2 | 93.2 | 17.4 | 71.8 | 10.1 | 90.2 | 9.5 | 90.3 | 33.4 | 72.9 |

To demonstrate the performance of the SOTA methods, we consider some latest and representative algorithms, *i.e*., two discriminative models, (Metric3Dv2 [], Depth-Anything []), and four generative models (Marigold [], DepthFM [], Geowizard [] and GenPercept []). We fairly evaluate their performance by using the official released model weights on 5 popular benchmarks, *i.e*., NYU v2 [], KITTI [], ETH3D [], ScanNet [] and DIODE [], in Table 1. Notably, all the methods do not use these benchmarks as training data. We can easily observe that (1) Metric3Dv2 [] achieves the best performance on all evaluation datasets, another discriminative-based method, Depth-Anything [] achieves the second best performance. Both of them are trained on large-scale datasets, with 16M and 63.5M training data separately. (2) Generative methods can achieve impressive results on these evaluation benchmarks with even a small amount of fine-tuning data.

In addition to quantitative results, we further test their generalization capability by qualitative visualization in several challenging scenes.  Fig. 1 demonstrates the results of three algorithms on line drawing images (left), color draft images (middle), and photo-realistic images (right). Surprisingly, Metric3D fails on both line draw images and color draft images, while Marigold [] and Depth-Anything [] show some generalization capability on this kind of non-geometrically consistent hand-drawn images. We conjecture that discriminative-based Metric3D does not see cartoon images in the training stage, which leads to poor performance in this scenario. Contrarily, although Marigold [] also does not see cartoon images in their training set, it leverages the priors stored in the pre-trained Stable Diffusion [] model. Stable Diffusion [] model has seen millions of text-cartoon pairs when performing text-to-image generation training.

Figure: Figure 1: Depth visualization on cartoon images. ‘MG’ indicates Marigold [], ‘DA’ indicates Depth-Anything [], ‘M3D’ indicates Metric3Dv2 [].
Refer to caption: x1.png

Fig. 2 shows the robustness of existing depth estimation models on challenging scenes like rainy, blurry, dark, and foggy environments. Both Metric3D and Depth-Anything fail on the rainy scene; both Marigold and Metric3D fail to estimate the sky in the second blurry scene. None of the algorithms can handle all environments perfectly.
Fig. 3 illustrates the depth estimation results on the Infinigen [] dataset (first two lines) and BEDLAM [] dataset (last line). Infinigen [] is a photo-realistic rendered dataset with diverse nature scenes. BEDLAM [] is a human-centered high-quality rendered dataset with versatile indoor and outdoor scenes. Mainstream depth evaluation metrics overlook the depth accuracy on the edges of the objects. We use these two datasets to demonstrate the fine-grained depth estimation results since both datasets have high-quality annotations. For measuring the accuracy of depth estimation on edges. We use Canny [] edge detector
to extract the edge mask from the image and then calculate the traditional depth metrics. As shown in Table 2, Depth-Anything achieves the highest performance on the Infinigen dataset; Marigold achieves the best AbsRel on the BEDLAM [] dataset.

In a nutshell, discriminative models trained on large data, *i.e*., Depth-Anything [], obtains
the highest performance in most cases, while generative models fine-tuned on small data, *e.g*., Marigold [], show competitive generalization capability on unseen scenes.

Figure: Figure 2: Depth visualization on four challenging scenes, *i.e*., rainy (top-left), blurry (top-right), dark (bottom-left), and foggy (bottom-right) environments.
Refer to caption: x2.png

**Table 2: Benchmarking depth estimation on Infinigen [] and BEDLAM [] datasets. ‘Standard’ indicates using standard evaluation metrics. ‘Canny’ indicates only evaluating the performance on pixels that belong to canny edges. We mark the best results in bold.**
| Method | Train<br>Samples | Infinigen-Standard | Infinigen-Canny | BEDLAM-Standard | BEDLAM-Canny |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ |  |  |
| Marigold [] | 74K | 32.9 | 80.9 | 28.0 | 78.7 | 16.2 | 82.4 | 19.6 | 80.3 |
| Metric3Dv2 [] | 16M | 14.5 | 80.7 | 18.6 | 77.8 | 28.1 | 84.7 | 26.3 | 80.8 |
| Depth-Anything [] | 63.5M | 12.0 | 88.4 | 14.3 | 84.7 | 46.2 | 69.0 | 46.8 | 67.8 |

Figure: Figure 3: Fine-grained depth estimation comparison. We select two scenes (first two rows) from the Infinigen Dataset [] and one scene (last row) from the BEDLAM dataset [].
Refer to caption: x3.png

### 3.2 Benchmarking Different Generative Fine-Tuning Paradigms

Several fine-tuning paradigms have been proposed for diffusion-based depth estimation. Based on
the
network architecture, they can be divided into two categories. The first category of methods (Marigold [] and DepthFM []) concatenate the image latent and depth latent encoded by VAE [] encoder as the input of the UNet latent denoiser. As such, the input channels of the latent denoiser are doubled (8 input channels) to fit the expanded input. The second category methods (DMP [] and GenPercept []) drop the depth latent, so they follow the original latent denoiser’s architecture (4 input channels). Based on fine-tuning paradigms, they can be divided into four categories. (1) Marigold [] treats the initial depth latent as standard Gaussian noise and progressively denoise it with the same scheduler as the original Stable Diffusion pipeline. (2) DepthFM also treats the initial depth latent as standard Gaussian noise, however, the difference is that they finetune the denoiser with Flow Matching [] pipeline, with auxiliary surface normal loss. (3) DMP [] reformulates the task as a blending process, *i.e*., translating the image latent to depth latent with the Stable Diffusion v-prediction [] learning target. (4) GenPercept [] further improve the efficiency of DMP [] by proposing a one-step inference pipeline. Based on the amount of fine-tuned parameters, they can be divided into two categories. The first category methods (Marigold, DepthFM, GenPercept) directly fine-tune the UNet parameters. The second category method (DMP) adds LORA [] layers into the UNet architecture to achieve the goal of depth estimation.

In this section, we fairly benchmark the four fine-tuning protocols by training on the Hypersim dataset (38,387 samples), with
$480\times 640$ resolution, $3\times 10^{-5}$ learning rate, $96$ batch sizes, and $10,000$ iterations. We choose Stable Diffusion 2.1 [] as the base model. As shown in Table 3, (1) Marigold [] outperforms other protocols in outdoor benchmarks, GenPercept [] gets the best performance in indoor benchmarks. Overall, Marigold has better generation capability than GenPercept. (2) Fine-tuning all UNet parameters outperforms using LORA layers. (compare line 1 and line 2 on DMP) (3) One-step GenPercept [] can outperform multi-step DMP []. We conjecture that the RGB blending strategy proposed in DMP makes it hard to decouple image latent and depth latent during the inference stage (see Supp. Mat. for visualization). (4) DepthFM [] uses flow matching as the fine-tuning protocol for efficient inference (two steps). Although its performance is not comparable with Marigold, we speculate this is due to the gap between the fine-tuning flow pipeline and pre-training v-prediction pipeline in Stable Diffusion 2.1. With the rise of flow-based generative models, *e.g*., Stable Diffusion 3 [], it may become a suitable fine-tuning strategy for flow-based models.

**Table 3: Benchmarking different generative finetuing paradigms on 5 zero-shot affine-invariant depth benchmarks. We mark the best results in bold and the second best underlined.**
| Method | Train<br>Samples | FT<br>Strategies | NYUv2 [] | KITTI [] | ETH3D [] | ScanNet [] | DIODE [] |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ |  |  |  |
| DMP [] | 38K | LORA | 13.2 | 85.1 | 19.2 | 74.3 | 16.2 | 83.7 | 14.1 | 84.6 | 45.6 | 62.1 |
| DMP [] | 38K | UNet | 10.1 | 90.6 | 15.4 | 80.0 | 10.0 | 91.0 | 10.9 | 89.0 | 38.2 | 68.7 |
| Marigold [] | 38K | UNet | 6.9 | 95.0 | 13.8 | 80.7 | 7.5 | 93.7 | 7.1 | 94.3 | 28.7 | 74.6 |
| GenPercept [] | 38K | UNet | 6.5 | 96.1 | 18.4 | 69.2 | 8.5 | 92.8 | 6.6 | 96.4 | 32.4 | 74.5 |
| DepthFM [] | 38K | UNet | 10.9 | 89.5 | 19.2 | 68.8 | 12.9 | 86.3 | 11.4 | 87.7 | 33.6 | 72.4 |

### 3.3 Benchmarking the Inference Efficiency of Depth Estimation Foundation Models

**Table 4: Inference latency and speed benchmark for different components and methods. ‘Infer Steps’ indicates the minimum repeat times of the U-Net for achieving optimal results. All models are inference with $512\times 512$ resolution, except CLIP [] ($224\times 224$).**
| Method | Components | Params/M | Macs/GFLOPs | Latency/s | Memory/G | Inference Steps |
| --- | --- | --- | --- | --- | --- | --- |
| Depth-Anything [] | ViT-L [] + Head | 335.3 | 586.0 | 0.19 | 2.24 | 1 |
| Metric3Dv2 [] | ViT-L [] + Head | 411.9 | 1014.0 | 0.60 | 2.67 | 1 |
| DSINE [] | EfficientNet B5 [] + Head | 72.6 | 38.7 | 0.06 | 0.73 | 1 |
| - | VAE-Tiny [] | 2.4 | 131.9 | 0.03 | 0.61 | 1 |
| - | VAE [] | 83.7 | 1781.2 | 0.11 | 0.65 | 1 |
| Geowizard [] | CLIP [] | 304.0 | 77.8 | 0.04 | 1.25 | 1 |
| Marigold-LCM [] | VAE [] + UNet [] | 949.6 | 3138.4 | 0.29 | 5.27 | 4 |
| Geowizard [] | VAE [] + UNet [] + CLIP [] | 861.2 | 9846.1 | 0.85 | 5.24 | 10 |
| DepthFM [] | VAE [] + UNet [] | 949.6 | 2459.8 | 0.21 | 5.40 | 2 |
| GenPercept [] | VAE [] + UNet [] | 949.6 | 2120.4 | 0.18 | 5.40 | 1 |
| GenPercept [] | VAE-Tiny [] + UNet [] | 868.3 | 471.1 | 0.18 | 5.40 | 1 |

Compared to discriminative models, the inference efficiency may become a bottleneck of the generative-based methods. In this section, we give detailed inference efficiency evaluation in Table 4. We can see that discriminative methods have fewer parameters than generative models. The main inference consumption of the generative models happens on VAE [] and multiple inference steps of UNet. The last line of Table 4 shows that GenPercept [] can achieve comparable inference latency with Depth-Anything (ViT-Large) and a tiny VAE encoder []. In Table 5, we found LCM [] can effectively reduce the inference steps of Marigold [] while maintaining the performance. Besides, a pre-trained tiny VAE [] can substitute the standard VAE [] with a minimal performance loss.

**Table 5: Benchmarking the inference efficiency of Marigold. We mark the best results in bold.**
| Method | VAE<br>Version | Infer<br>Steps | NYUv2 [] | KITTI [] | ETH3D [] | ScanNet [] | DIODE [] |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ |  |  |  |
| Marigold [] | base | 50 | 5.5 | 96.4 | 9.9 | 91.6 | 6.5 | 96.0 | 6.4 | 95.1 | 30.8 | 77.3 |
| Marigold-LCM [] | base | 4 | 6.1 | 95.8 | 10.1 | 90.6 | 6.3 | 96.0 | 6.9 | 94.7 | 30.9 | 77.3 |
| Marigold-LCM [] | tiny [] | 4 | 6.9 | 95.0 | 13.8 | 80.7 | 7.5 | 93.7 | 7.1 | 94.3 | 32.8 | 73.8 |
| Marigold-LCM [] | tiny [] | 1 | 6.6 | 95.4 | 13.0 | 83.6 | 7.8 | 93.2 | 7.0 | 94.5 | 33.3 | 73.1 |

### 3.4 Benchmarking Discriminative and Generative Models on the Same Training Data

**Table 6: Benchmarking deterministic and generative depth model with the same training data ($77$K).**
| Network | Pretrain<br>Style | Backbone | NYUv2 [] | KITTI [] | ETH3D [] | ScanNet [] | DIODE [] |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ |  |  |  |
| ViT+DPT Head | Random init | ViT-L | 21.1 | 62.5 | 27.2 | 53.1 | 23.4 | 61.1 | 19.2 | 67.4 | 32.4 | 57.7 |
| ViT+DPT Head | DINOv2 [] | ViT-L | 4.9 | 97.5 | 8.5 | 94.1 | 8.1 | 97.0 | 5.1 | 97.6 | 24.5 | 74.6 |
| Marigold [] | SD21 [] | UNet | 6.9 | 95.8 | 12.2 | 85.7 | 9.2 | 95.5 | 7.1 | 95.4 | 25.2 | 73.0 |
| Marigold [] | SDXL [] | UNet | 6.8 | 95.8 | 11.1 | 89.2 | 8.9 | 96.7 | 6.3 | 96.2 | 24.5 | 73.6 |

Can discriminative depth estimation models achieve competitive results with small-scale high-quality training datasets like generative-based methods? To answer this question, we benchmark discriminative and generative geometry model with the same amount of training data and the same training strategy. Specifically, we use three training datasets, *i.e*., Hypersim (38,387) [], Virtual Kitti (16,790) [] and Tartanair (31,008) [], with total 77,897 samples. Both models are trained with 20,000 iterations, with a total batch size of 96 on 4 GPUs. For the discriminative depth model, we follow the network architecture of Depth-Anything [] (ViT-Large backbone pre-trained with DINOv2 and DPT [] head), supervised with the affine-invariant loss []. For the generative geometry model, we choose Marigold [] as our baseline.

We can see from Table 6 that (1) the discriminative model is largely inferior to generative-based Marigold on all evaluation datasets without DINOv2 pre-train (line 1 *vs*.  line 3). However, the discriminative model beats Marigold by a large margin when initialized with DINOv2 pre-train weight (line 2 *vs*. line 3); (2) scale-up Marigold from SD21 to SDXL brings consistent improvement in all benchmarks. We can see from Table 7 that our discriminative model trained on 77K data outperforms Metric3Dv2 [] in all three datasets, and, is comparable with Depth-Anything [] on
two datasets (InspaceType and Infinigen). This phenomenon suggests that high-quality fine-tuning data, rather than large-scale training data, is indispensable for discriminative models to achieve strong generalizable performance. The qualitative visualization of discriminative models is not as good as generative models, more visualizations are available in the Supp. Mat.

**Table 7: Benchmarking depth estimation foundation models on more diverse benchmarks. We mark the best results in bold.**
| Network | Pretrain<br>Style | Backbone | Train<br>Samples | InspaceType [] | MatrixCity [] | Infinigen [] |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ |  |  |  |  |
| Metric3Dv2 [] | DINOv2 [] | ViT-L | 16M | 10.1 | 89.7 | 9.5 | 89.3 | 14.5 | 80.7 |
| Depth-Anything [] | DINOv2 [] | ViT-L | 63.5M | 8.2 | 92.9 | 16.4 | 89.7 | 12.0 | 88.4 |
| ViT+DPT Head | DINOv2 [] | ViT-L | 77K | 8.4 | 94.0 | 28.0 | 82.4 | 11.4 | 89.5 |
| Marigold [] | SD21 [] | UNet | 77K | 9.2 | 92.7 | 17.0 | 82.9 | 14.1 | 83.9 |

## 4 Benchmarking Surface Normal Estimation Foundation Models

### 4.1 A Brief Overview of SOTA Methods

DSINE [] and Metric3Dv2 [] are two representative discriminative surface estimation models, which leverage the geometry priors from two distinct perspectives. DSINE leverages two forms of inductive bias: (1) per-pixel ray direction, and (2) the relationship between the neighboring surface normal, to learn a generalizable surface normal estimator. Metric3Dv2 [] proposes to optimize the surface normal map by distilling diverse data knowledge from the estimated metric depth. Different from discriminative models, GeoWizard [] is a generative surface normal estimator without using any inductive bias from the geometry priors. It purely relies on pre-trained diffusion priors to estimate the surface normal map. Table 9 summarizes their performance on six benchmarks. The Mushroom [] (indoor), T&T [] (outdoor), and Infinigen [] (wild) datasets are constructed by us to add more diverse scenes with accurate surface normal labels in the evaluation benchmarks. We can see that Metric3Dv2 [] outperform DSINE [] and GeoWizard [] in most datasets. Note it is an unfair comparison since (1) Metric3Dv2 [] is trained on 16M images, while DSINE is trained on 160K images, and GeoWizard is trained on 280K images. (2) DSINE use a much smaller backbone, EfficientNet-B5 [], while Metric3Dv2 [] employs the ViT-Large [] backbone, pretrained using DINOv2 with registers [].

### 4.2 Benchmarking Discriminative and Generative Models on the Same Training Data

In this section, we fairly benchmark discriminative DSINE [] and several representative generative geometry models, *i.e*., Marigold [], DMP [], GenPercept [], and DepthFM [], with 5 training datasets, Hypersim [] ($38,387$), Tartanair [] ($31,008$), Virtual Kitti [] ($16,790$), BlendedMVS [] ($17,819$), ClearGrasp [] ($22,720$), a total of $126,724$ samples. For generatative-based models, we represent the output surface normals as unit vectors. We follow DSINE [] to represent the outputs of discriminative-based model as axis-angles with three degrees of freedom. All models are trained with $20,000$ iterations, $96$ batch sizes, $480\times 640$ resolution on 4 A800 GPUs. All generative-based models use $3\times 10^{-5}$ learning rate. For discriminative model, we follow DSINE [] to use $3\times 10^{-5}$ learning rate for the backbone and $3\times 10^{-4}$ learning rate for the decoder.

We can see from Table 8 that (1) DSINE can scale up the performance by using ViT-Large backbone with DINOv2 pretrain (compared with ImageNet pretrained Efficient-B5 backbone). (2) For generative-based fine-tuning protocols, DepthFM [] outperforms other paradigms in most benchmarks. We attribute this to the decoder supervision during the training. Paradigms that requires multi-step denoising inference steps, *e.g*., Marigold [] and DMP []are not suitable to perform decoder supervision during the training. (3) Discriminative models, equipped with inductive bias, outperform generative-based models with only diffusion priors. It is promising to inject inductive bias into the diffusion-based models, as such, the surface normal estimator can effectively leverage the diffusion priors and inductive bias to boost the performance. (4) DSINE (ViT-Large in Table 8) trained with $120$K samples achieves comparable performance with Metric3Dv2 trained with $16$M samples (Table 9). The results verify the point that data-quality is more important than the data-scale.

**Table 8: Quantitative evaluation of the generalization capabilities possessed by various methods with official released weights. For each metric, the best results are bolded. Discriminative methods are colored in blue while generative ones in green.**
| Method | NYU v2 [] | ScanNet [] | Sintel [] |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ |  |
| Metric3Dv2 [] | 13.5 | 6.7 | 40.1 | 53.5 | 65.9 | 82.6 | 87.7 | 11.8 | 5.5 | 46.6 | 60.7 | 71.6 | 85.4 | 89.7 | 22.8 | 14.2 | 18.4 | 28.5 | 41.6 | 66.7 | 75.8 |
| DINSE [] | 16.4 | 8.4 | 32.8 | 46.3 | 59.6 | 77.7 | 83.5 | 18.3 | 9.3 | 27.1 | 42.0 | 56.3 | 75.0 | 81.2 | 32.0 | 23.9 | 9.0 | 15.0 | 23.8 | 47.5 | 59.4 |
| Geowizard [] | 19.8 | 11.2 | 18.0 | 32.7 | 50.2 | 73.0 | 79.9 | 21.1 | 11.9 | 15.9 | 29.7 | 47.4 | 70.7 | 77.8 | 36.1 | 28.4 | 4.1 | 8.6 | 16.9 | 39.8 | 52.5 |
| Method | MuSHRoom Subset [] (Indoor) | T&T Subset [] (Outdoor) | Infinigen Subset [] (Wild) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ |  |
| Metric3Dv2 [] | 14.3 | 7.9 | 31.9 | 48.1 | 61.8 | 81.7 | 87.2 | 22.3 | 14.1 | 19.2 | 31.4 | 43.0 | 64.8 | 73.5 | 32.6 | 27.3 | 5.1 | 10.1 | 17.8 | 41.3 | 54.4 |
| DINSE [] | 14.8 | 8.6 | 28.1 | 44.6 | 59.7 | 80.4 | 87.0 | 17.3 | 11.0 | 24.2 | 37.3 | 50.6 | 74.1 | 82.4 | 35.9 | 32.6 | 2.1 | 4.6 | 9.8 | 30.5 | 45.1 |
| Geowizard [] | 16.5 | 10.7 | 14.7 | 30.5 | 52.5 | 79.6 | 86.2 | 20.8 | 13.4 | 10.7 | 23.4 | 42.2 | 70.3 | 78.5 | 36.2 | 32.0 | 1.8 | 4.0 | 8.86 | 30.8 | 46.2 |

**Table 9: Quantitative evaluation of the generalization capabilities with the same training data on different benchmarks. The best results are bolded. ‘EB5’ indicates ImageNet [] pre-trained EfficientNet-B5 []. ‘ViT-L’ indicates DINOv2 [] pre-trained ViT-Large. The best results are bolded. The best results of generativate-based models are underlined.**
| Method | Backbone | NYUv2 [] | ScanNet [] | Sintel [] |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ |  |  |
| DSINE [] | EB5 [] | 19.2 | 10.0 | 27.1 | 40.1 | 53.9 | 73.8 | 80.1 | 17.3 | 11.0 | 24.2 | 37.3 | 50.6 | 74.1 | 82.4 | 35.9 | 32.6 | 2.1 | 4.6 | 9.8 | 30.5 | 45.1 |
| DSINE [] | ViT-L [] | 16.2 | 8.2 | 32.8 | 46.9 | 60.6 | 78.5 | 84.1 | 16.1 | 7.4 | 34.5 | 50.6 | 63.8 | 79.4 | 84.3 | 24.6 | 16.1 | 11.1 | 21.1 | 35.1 | 63.8 | 74.1 |
| GenPercept [] | UNet [] | 17.8 | 9.5 | 24.1 | 40.5 | 55.9 | 75.7 | 82.2 | 18.5 | 9.4 | 23.0 | 40.2 | 56.7 | 75.4 | 81.3 | 38.6 | 27.1 | 4.5 | 9.1 | 18.0 | 42.5 | 54.2 |
| Marigold [] | UNet [] | 20.2 | 10.9 | 21.8 | 36.0 | 51.2 | 72.8 | 79.4 | 20.5 | 10.3 | 19.7 | 36.0 | 53.5 | 73.6 | 79.2 | 41.3 | 28.7 | 5.5 | 11.1 | 19.7 | 40.9 | 51.7 |
| DMP [] | UNet [] | 21.9 | 11.3 | 19.7 | 34.2 | 49.7 | 71.1 | 77.6 | 22.5 | 11.2 | 17.6 | 32.5 | 50.3 | 71.2 | 76.9 | 45.0 | 39.3 | 4.2 | 7.9 | 13.8 | 29.9 | 39.0 |
| DepthFM [] | UNet [] | 17.8 | 9.3 | 27.7 | 41.9 | 56.5 | 76.7 | 82.5 | 18.5 | 8.6 | 28.4 | 44.7 | 58.8 | 75.6 | 81.0 | 34.1 | 25.8 | 7.1 | 13.5 | 22.0 | 44.8 | 55.7 |
| Method | Backbone | MuSHRoom Subset [] (Indoor) | T&T Subset [] (Outdoor) | Infinigen Subset [] (Wild) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ | mean | med | $5.0^{\circ}$ | $7.5^{\circ}$ | $11.25^{\circ}$ | $22.5^{\circ}$ | $30^{\circ}$ |  |  |
| DSINE [] | EB5 [] | 17.9 | 10.1 | 23.9 | 38.9 | 53.8 | 75.8 | 82.5 | 21.7 | 15.4 | 13.4 | 25.7 | 39.0 | 64.4 | 75.2 | 36.5 | 32.7 | 2.0 | 4.4 | 9.3 | 29.8 | 44.8 |
| DSINE [] | ViT-L [] | 12.8 | 6.9 | 34.6 | 53.5 | 67.9 | 84.9 | 89.6 | 18.8 | 11.2 | 22.5 | 36.3 | 49.9 | 71.1 | 79.5 | 33.6 | 28.7 | 2.7 | 5.9 | 13.1 | 38.3 | 52.2 |
| GenPercept [] | UNet [] | 15.0 | 7.9 | 32.4 | 48.0 | 62.7 | 80.6 | 86.3 | 27.4 | 14.0 | 17.8 | 30.1 | 43.1 | 63.7 | 71.0 | 38.8 | 33.5 | 2.1 | 4.9 | 10.5 | 31.2 | 44.3 |
| Marigold [] | UNet [] | 17.7 | 9.9 | 19.6 | 36.8 | 55.7 | 77.0 | 83.1 | 29.1 | 14.6 | 14.4 | 26.0 | 40.3 | 63.0 | 70.2 | 39.2 | 34.0 | 2.4 | 5.4 | 11.5 | 31.4 | 43.9 |
| DMP [] | UNet [] | 20.4 | 10.0 | 19.3 | 36.0 | 55.2 | 73.8 | 79.1 | 27.7 | 17.9 | 9.1 | 17.2 | 31.9 | 57.9 | 66.1 | 43.1 | 38.1 | 1.7 | 4.0 | 9.5 | 26.3 | 38.1 |
| DepthFM [] | UNet [] | 17.0 | 9.0 | 25.9 | 42.5 | 58.9 | 77.4 | 82.4 | 22.1 | 13.9 | 14.2 | 26.7 | 42.1 | 65.4 | 74.1 | 31.9 | 27.9 | 2.5 | 5.4 | 11.6 | 38.3 | 54.0 |

## 5 Benchmarking Geometric Correspondence

Can current monocular geometry estimation models improve the 3D awareness of the original representation models, *e.g*., DINOv2 and Stable Diffusion? To answer the question, we follow Probe3D [] by using geometric correspondence estimation, since 3D awareness implies consistency of representations across different views. Specifically, given two views of the same scene, geometric correspondence estimation needs to identify pixels across views that depict the same point in 3D space. We extract feature maps from either trained monocular geometry models or representation models, *e.g*., DINOv2, and directly compute correspondence between the dense feature maps of different views. We use Paired ScanNet [] for scene evaluation and NAVI wild set [] for object evaluation. Following [], we report the correspondence recall, *i.e*., the percentage of correspondence that falls within some defined distance.

Figure: Figure 4: Geometry correspondences evaluation. ‘GeowizardD’, ‘GeowizardN’ indicate depth and normal features from Geowizard.
Refer to caption: x4.png

We can see from Fig. 4 that:

- 1.
Depth-Anything, pretrained with DINOv2 [] and fine-tuned on $77$K training samples, is comparable to the original DINOv2; Metric3Dv2 pretrained with DINOv2-reg [] outperforms original DINOv2-reg on ScanNet and inferior to original DINOv2-reg on NAVI dataset. While generative-based models, *i.e*., Marigold, DepthFM, GenPercept, and Geowizard, are inferior to the original Stable Diffusion [] model on both datasets.
- 2.
All models struggle with larger view changes, while generative-based models see a larger drop. In general, monocular geometry estimation models are not 3D-consistent with large viewpoints and thus not yet good enough to encode the 3D structure of the real-world scenario.

## 6 Conclusion and Discussion

In this work, we present the first large-scale benchmarking of discriminative and generative geometry estimation foundation models with diverse evaluation datasets. We identify that with a strong pre-train model, either Stable Diffusion or DINOv2, the fine-tuning data quality is a more important factor than fine-tuning data scale and model architecture to achieve generalizable geometry estimation. We believe that this benchmarking study can provide strong baselines for unbiased comparisons in geometry estimation studies. Limitations and future works are discussed in the Supp. Mat.

## Appendix A Appendix

Benchmarking: We introduce new benchmarks with diverse scenes and high-quality annotations to overcome limitations in scene diversity and label quality of existing benchmarks.

Findings: The study reveals that discriminative models pre-trained on large data can outperform generative models with less but high-quality synthetic data, suggesting the importance of data quality over quantity and architecture.

### A.1 Depth Estimation Visualization of Different Methods

We visualize the depth estimation results in Fig. 5 and  Fig. 6.

Figure: Figure 5: Visualization of different depth estimation methods.
Refer to caption: x5.png

Figure: Figure 6: Visualization of different depth estimation methods.
Refer to caption: x6.png

### A.2 Surface Normal Estimation Visualization of Different Methods

We visualize the surface normal estimation results in Fig. 7 and  Fig. 8.

Figure: Figure 7: Visualization of different surface normal estimation methods.
Refer to caption: x7.png

Figure: Figure 8: Visualization of different surface normal estimation methods.
Refer to caption: x8.png

### A.3 Correspondence Estimation Results

We give more detailed correspondence estimation results in Table 10 for reference. Note that we find that multi-step inference, e.g., 10 steps, can improve the performance of Stable Diffusion in correspondence estimation tasks. Metric3Dv2 [] employs DINOv2 with registers [] as the backbone, which has higher performance than DINOv2 without registers [].

**Table 10: Correspondence Estimation Results. The results are presented for features extracted at different layers with performance binned based on the viewpoint variation for the image pair. ‘DA’ indicates Depth-Anything. ‘DA (77K)’ indicates Depth-Anything trained with only 77K synthetic data. ‘SD10’ indicates Stable Diffusion model inference 10 steps. ‘MIX’ indicates using a mixture of datasets during the training. The higher the recall in the table, the better the performance.**
|  |  |  |  | Spair-71k | Paired ScanNet | NAVI |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | Architecture | Dataset | Layers | $d{=}0$ | $d{=}1$ | $d{=}2$ | all | $\theta_{0}^{15}$ | $\theta_{15}^{30}$ | $\theta_{30}^{60}$ | $\theta_{60}^{180}$ | $\theta_{0}^{30}$ | $\theta_{30}^{60}$ | $\theta_{60}^{90}$ | $\theta_{90}^{120}$ |
| Pre-train Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| DINOv2 [] | ViT-L14 | LVD | Block0 | 8.5 | 6.2 | 5.3 | 7.5 | 17.2 | 14.1 | 10.1 | 4.7 | 66.2 | 37.2 | 19.6 | 11.5 |
| DINOv2 [] | ViT-L14 | LVD | Block1 | 25.0 | 14.0 | 10.8 | 19.3 | 29.0 | 20.8 | 13.5 | 5.2 | 92.1 | 57.9 | 25.6 | 12.8 |
| DINOv2 [] | ViT-L14 | LVD | Block2 | 53.9 | 34.6 | 31.6 | 44.5 | 35.2 | 24.1 | 16.3 | 6.6 | 95.3 | 70.0 | 35.4 | 18.5 |
| DINOv2 [] | ViT-L14 | LVD | Block3 | 62.8 | 53.3 | 54.2 | 57.2 | 36.5 | 27.0 | 20.8 | 12.2 | 92.2 | 72.3 | 48.9 | 35.0 |
| DINOv2 [] | ViT-L14+reg | LVD | Block0 | 12.2 | 8.8 | 8.1 | 10.4 | 14.0 | 14.2 | 11.4 | 5.0 | 79.9 | 40.8 | 24.5 | 13.6 |
| DINOv2 [] | ViT-L14+reg | LVD | Block1 | 41.2 | 22.8 | 17.1 | 32.0 | 52.0 | 39.4 | 23.7 | 9.1 | 95.4 | 65.6 | 32.7 | 15.8 |
| DINOv2 [] | ViT-L14+reg | LVD | Block2 | 64.2 | 45.9 | 42.4 | 55.0 | 50.6 | 39.3 | 26.2 | 12.0 | 95.2 | 75.0 | 49.1 | 28.6 |
| DINOv2 [] | ViT-L14+reg | LVD | Block3 | 59.3 | 53.2 | 54.9 | 55.0 | 45.0 | 35.4 | 26.1 | 15.4 | 88.6 | 71.2 | 54.3 | 36.1 |
| SAM [] | ViT-L16 | SA-1B | Block0 | 9.9 | 6.1 | 5.4 | 8.0 | 14.5 | 9.9 | 7.5 | 3.5 | 78.0 | 43.3 | 20.4 | 11.4 |
| SAM [] | ViT-L16 | SA-1B | Block1 | 22.6 | 15.8 | 12.5 | 18.3 | 37.2 | 29.7 | 19.7 | 6.2 | 86.4 | 52.0 | 23.8 | 12.5 |
| SAM [] | ViT-L16 | SA-1B | Block2 | 34.8 | 23.1 | 17.0 | 28.2 | 47.6 | 40.4 | 27.3 | 8.7 | 91.2 | 60.1 | 28.2 | 14.2 |
| SAM [] | ViT-L16 | SA-1B | Block3 | 30.2 | 18.1 | 13.0 | 24.1 | 52.6 | 43.9 | 28.7 | 9.6 | 88.5 | 57.6 | 26.9 | 13.5 |
| SD10 [] | UNet | LAION | Block0 | 13.2 | 5.3 | 3.5 | 9.2 | 10.8 | 5.4 | 3.2 | 1.3 | 75.1 | 32.5 | 16.6 | 7.4 |
| SD10 [] | UNet | LAION | Block1 | 58.6 | 36.4 | 28.6 | 47.8 | 67.0 | 56.1 | 32.0 | 8.7 | 93.4 | 59.7 | 26.2 | 11.4 |
| SD10 [] | UNet | LAION | Block2 | 24.0 | 16.8 | 13.4 | 20.2 | 61.4 | 49.5 | 28.4 | 9.4 | 79.0 | 42.5 | 22.3 | 12.2 |
| SD10 [] | UNet | LAION | Block3 | 4.6 | 4.3 | 4.4 | 4.3 | 17.2 | 12.8 | 8.9 | 5.0 | 35.3 | 22.9 | 15.2 | 11.0 |
| Deterministic Geometry Foundation Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| MiDaS [] | ViT-L16 | MIX 6 | Block0 | 15.6 | 10.2 | 8.7 | 13.0 | 50.3 | 39.0 | 24.4 | 11.2 | 79.0 | 49.1 | 25.0 | 14.5 |
| MiDaS [] | ViT-L16 | MIX 6 | Block1 | 27.3 | 22.8 | 23.2 | 24.5 | 56.4 | 47.4 | 31.6 | 13.9 | 83.2 | 56.0 | 32.1 | 21.6 |
| MiDaS [] | ViT-L16 | MIX 6 | Block2 | 28.2 | 23.4 | 25.1 | 25.5 | 55.5 | 46.0 | 30.8 | 14.3 | 82.2 | 56.3 | 33.1 | 22.9 |
| MiDaS [] | ViT-L16 | MIX 6 | Block3 | 25.8 | 21.3 | 23.6 | 23.4 | 52.4 | 42.1 | 27.6 | 13.1 | 79.6 | 53.0 | 31.4 | 21.6 |
| DA [] | ViT-L16 | MIX | Block0 | 8.0 | 6.1 | 5.3 | 6.8 | 21.4 | 17.5 | 12.2 | 5.4 | 66.1 | 35.6 | 20.6 | 12.5 |
| DA [] | ViT-L16 | MIX | Block1 | 24.4 | 13.8 | 11.1 | 19.4 | 34.2 | 26.4 | 17.0 | 6.1 | 92.4 | 55.9 | 27.7 | 14.0 |
| DA [] | ViT-L16 | MIX | Block2 | 51.4 | 31.6 | 28.4 | 42.2 | 30.2 | 23.5 | 16.2 | 6.8 | 95.2 | 68.1 | 35.1 | 17.5 |
| DA [] | ViT-L16 | MIX | Block3 | 58.9 | 48.6 | 49.7 | 53.5 | 29.8 | 21.4 | 16.8 | 9.3 | 90.9 | 67.8 | 47.9 | 30.5 |
| DA(77K) [] | ViT-L16 | MIX | Block0 | 8.0 | 5.8 | 5.2 | 6.7 | 18.3 | 15.1 | 10.7 | 5.0 | 63.6 | 34.9 | 20.4 | 12.4 |
| DA(77K) [] | ViT-L16 | MIX | Block1 | 24.2 | 13.6 | 11.0 | 19.1 | 34.4 | 25.7 | 16.6 | 6.4 | 92.4 | 54.6 | 26.9 | 13.7 |
| DA(77K) [] | ViT-L16 | MIX | Block2 | 50.8 | 31.0 | 28.0 | 41.6 | 43.4 | 32.9 | 23.2 | 8.9 | 94.9 | 67.0 | 34.9 | 17.8 |
| DA(77K) [] | ViT-L16 | MIX | Block3 | 53.6 | 42.2 | 43.4 | 47.7 | 38.4 | 29.8 | 21.7 | 11.8 | 92.8 | 71.9 | 50.7 | 31.0 |
| Metric3Dv2 [] | ViT-L16 | MIX | Block0 | 12.0 | 8.6 | 7.9 | 10.1 | 10.2 | 10.5 | 8.7 | 4.3 | 79.1 | 39.7 | 23.5 | 12.7 |
| Metric3Dv2 [] | ViT-L16 | MIX | Block1 | 39.0 | 22.0 | 16.0 | 30.7 | 55.7 | 42.8 | 25.2 | 8.8 | 94.2 | 61.4 | 29.6 | 14.2 |
| Metric3Dv2 [] | ViT-L16 | MIX | Block2 | 60.2 | 41.5 | 39.8 | 51.6 | 63.1 | 54.7 | 36.8 | 14.8 | 94.1 | 68.1 | 36.9 | 20.9 |
| Metric3Dv2 [] | ViT-L16 | MIX | Block3 | 53.6 | 42.3 | 42.8 | 48.0 | 59.5 | 50.3 | 35.1 | 16.3 | 86.6 | 56.5 | 29.6 | 17.2 |
| Generative Geometry Foundation Models |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Marigold [] | UNet | MIX | Block0 | 14.0 | 4.6 | 3.5 | 9.6 | 8.4 | 5.8 | 3.4 | 1.3 | 81.7 | 37.0 | 17.0 | 8.0 |
| Marigold [] | UNet | MIX | Block1 | 53.8 | 29.5 | 23.7 | 42.5 | 42.2 | 32.4 | 18.7 | 4.4 | 92.8 | 59.1 | 25.6 | 11.7 |
| Marigold [] | UNet | MIX | Block2 | 27.2 | 15.8 | 12.5 | 21.3 | 45.5 | 34.1 | 18.1 | 5.4 | 83.5 | 45.4 | 21.5 | 11.2 |
| Marigold [] | UNet | MIX | Block3 | 8.0 | 6.4 | 6.3 | 7.1 | 18.0 | 12.8 | 7.5 | 3.5 | 43.3 | 25.2 | 15.9 | 9.8 |
| DepthFM [] | UNet | MIX | Block0 | 20.0 | 8.4 | 6.1 | 14.3 | 23.1 | 16.4 | 7.4 | 2.2 | 85.9 | 40.6 | 17.0 | 8.0 |
| DepthFM [] | UNet | MIX | Block1 | 50.8 | 31.4 | 25.2 | 42.1 | 46.4 | 39.1 | 24.0 | 7.2 | 94.1 | 62.4 | 29.2 | 13.0 |
| DepthFM [] | UNet | MIX | Block2 | 22.6 | 13.8 | 10.7 | 18.8 | 46.0 | 36.7 | 20.0 | 6.2 | 80.5 | 41.7 | 20.7 | 11.0 |
| DepthFM [] | UNet | MIX | Block3 | 3.9 | 3.5 | 3.0 | 3.6 | 11.2 | 8.4 | 6.3 | 3.8 | 39.0 | 25.6 | 16.3 | 10.4 |
| GeowizardD [] | UNet | MIX | Block0 | 13.7 | 4.7 | 2.9 | 9.7 | 8.0 | 5.1 | 3.2 | 1.34 | 81.9 | 35.1 | 16.6 | 8.5 |
| GeowizardD [] | UNet | MIX | Block1 | 41.3 | 19.1 | 13.4 | 31.2 | 43.0 | 32.5 | 16.9 | 3.8 | 89.3 | 52.3 | 22.5 | 10.7 |
| GeowizardD [] | UNet | MIX | Block2 | 20.2 | 11.4 | 8.4 | 16.3 | 38.3 | 27.1 | 12.8 | 3.7 | 71.1 | 35.4 | 17.8 | 10.1 |
| GeowizardD [] | UNet | MIX | Block3 | 8.5 | 5.7 | 5.8 | 7.2 | 13.8 | 9.9 | 5.4 | 2.7 | 32.5 | 20.1 | 12.7 | 8.1 |
| GeowizardN [] | UNet | MIX | Block0 | 11.1 | 3.6 | 2.9 | 7.6 | 8.5 | 5.1 | 3.0 | 1.3 | 80.6 | 33.9 | 15.1 | 7.6 |
| GeowizardN [] | UNet | MIX | Block1 | 43.3 | 20.2 | 15.5 | 32.8 | 48.6 | 37.8 | 20.8 | 5.0 | 88.8 | 53.7 | 22.4 | 10.4 |
| GeowizardN [] | UNet | MIX | Block2 | 22.5 | 12.3 | 9.4 | 18.0 | 43.4 | 32.8 | 16.3 | 4.5 | 68.9 | 36.6 | 17.5 | 9.2 |
| GeowizardN [] | UNet | MIX | Block3 | 6.8 | 5.4 | 4.8 | 6.2 | 13.0 | 10.2 | 6.3 | 2.7 | 27.5 | 17.6 | 12.0 | 7.3 |
| GenPercept [] | UNet | MIX | Block0 | 21.5 | 9.6 | 7.0 | 16.0 | 22.7 | 16.1 | 7.1 | 1.8 | 84.4 | 40.8 | 17.3 | 8.0 |
| GenPercept [] | UNet | MIX | Block1 | 62.0 | 41.9 | 34.4 | 52.2 | 55.7 | 46.4 | 27.8 | 6.4 | 94.5 | 64.9 | 29.7 | 13.3 |
| GenPercept [] | UNet | MIX | Block2 | 28.2 | 16.4 | 13.3 | 22.9 | 54.9 | 43.0 | 23.8 | 6.1 | 84.5 | 45.1 | 21.5 | 10.8 |
| GenPercept [] | UNet | MIX | Block3 | 8.0 | 5.9 | 5.9 | 7.0 | 26.6 | 19.0 | 10.6 | 3.8 | 58.3 | 31.4 | 17.4 | 10.0 |

### A.4 Surface Normal Estimation Datasets

NYUv2 [] is an real indoor dataset comprised RGB-D video sequences from a variety of indoor scenes captured from the Microsoft Kinect. We evaluate on the official test (654 images) set with the ground-truth surface normal generated by Ladicky *et al*.  [].

ScanNet [] is a real RGB-D video dataset of indoor scenes. We use the ground-truth surface normal and test split (800 sampled images) provided by FrameNet []. To mitigate the noise, it first computes two (X and Y) tangent principal directions by adopting the 4-RoSY field using QuadriFlow [] as proposed by TextureNet [], and the ground-truth normal can be directly computed as the cross product of them.

DIODE [] 1024$\times$768 collects both outdoor and indoor scenes. It collects high-quality data, but it contains very low diversity with only 2 scenes for evaluation.

Sintel [] is a synthetic dataset derived from an open-source 3D animated short film. We calculate the ground-truth surface normal with the provided ground-truth depth maps and intrinsic parameters following the depth-to-normal procedure of DSINE [].

BEDLAM [] contains synthetic monocular RGB videos with ground-truth 3D bodies with varying numbers of people in realistic scenes with varied lighting and camera motions. We calculate the ground-truth surface normal with the provided ground-truth depth maps and intrinsic parameters following the depth-to-normal procedure of DSINE [].

Infinigen [] generates diverse high-quality 3D synthetic scene data, which offers broad coverage of objects and scenes in the natural world with natural phenomena. The surface normal is rendered based on Blender.

MuSHRoom [] is an indoor real-world multi-sensor hybrid room dataset, which contains 10 rooms captured by Kinect, iPhone, and Faro scanner. We use the ground-truth normal annotations supported by gaustudio [].

Tank and Temples (T&T) [] is a dataset including both outdoor scenes and indoor environments, whose ground-truth data is captured using an industrial laser scanner. We use the ground-truth normal annotations supported by gaustudio [].

Figure: Figure 9: Visualization of the ground-truth surface normal from different datasets.
Refer to caption: x9.png

### A.5 Limitations and Future Works

The discussion of monocular depth estimation in this work is limited to single-image monocular affine-invariant depth estimation. Monocular metric depth estimation and video geometry estimation are also important topics, we leave them for future exploration.

### A.6 Broader Impacts

In this section, we aim to discuss the potential societal impacts. The positive societal impacts encompass two aspects. First, it helps the research community gain in-depth knowledge about monocular geometry estimation, including performance comparisons between different models, technical details of current models, and future approaches. The release of this work also helps researchers perform experiments to evaluate their methods more comprehensively, fairly, and conveniently. Furthermore, it will significantly boost the progress of downstream tasks. As we mentioned in the paper, monocular geometry estimation can be applied to many downstream tasks, thereby accelerating their progress. In summary, we believe this work will have substantial positive effects on the research community, enriching the capacity of current and future applications and products, and ultimately improving people’s lives. We also evaluated the negative societal impacts and found none.

Acknowledgements:
This work was supported by National Key R&D Program of China (No. 2022ZD0118700).

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: - Do you think testing 3D consistency solely with the method proposed by Probe3D is sufficient?
- How can we design a depth estimation model that improves 3D consistency?
- Are you suggesting that overall, data quality is more important than quantity, but for metric depth estimation, both data quantity and quality are essential?
- What kind of networks, designs, and datasets would be good to create and train when constructing new models for relative depth estimation, metric depth estimation, surface normal estimation, and 3D consistency? The summary of suggested insight will be beneficial.
soundness: 3
strengths: - Expensive experiments: The paper conducts a thorough comparison of discriminative and generative geometry models using a wide range of datasets and models, evaluating their performance under different training conditions. This comprehensive approach provides valuable assistance to researchers, allowing them to follow the latest depth estimation models and gain insights from the detailed comparisons.

- Practical Insights: The paper highlights the strengths and weaknesses of specific models and approaches, providing practical implications for future research. The observation that Marigold demonstrates greater robustness and good generalization capabilities on new data is interesting.
summary: The paper integrates the latest depth estimation models, dividing them into discriminative-based and generative-based approaches, and evaluates them using diverse datasets, including new benchmark datasets created by the authors. It conducts extensive experiments on various tasks, such as general depth estimation, metric depth, surface normal estimation, and cross-view consistency, analyzing the strengths and weaknesses of each model. Additionally, the paper explores different fine-tuning methods and experiments with lighter network architectures, providing practical insights for the next depth estimation model development.
weaknesses: - Lack of Depth in Analysis: The paper often bases its conclusions on somewhat limited experimental results. For example, it concludes that complex fine-tuning protocols are not necessary based solely on the strong performance of GenPercept's one-step fine-tuning. However, this could be a risky generalization. This conclusion might easily be challenged if a more effective, complex fine-tuning method is proposed in the future. Rather than relying solely on performance metrics, a more in-depth analysis of GenPercept’s network architecture and training dynamics is needed, along with validation under diverse experimental conditions. For instance, evaluating the robustness of different types of datasets or under significant viewpoint changes could enhance the reliability of this conclusion.

- Novelty: Although the paper provides important insights through extensive experimentation, these insights do not introduce entirely new directions. The study effectively aggregates and compares recent research to showcase the current leading approaches. However, this approach leans more towards a re-evaluation of existing work rather than proposing novel methodologies or technological breakthroughs. Therefore, it is questionable whether such comparisons alone are sufficient as substantial contributions.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: Q1: L404: ‘We can see from Table 7 that our discriminative model trained on 77K data outperforms Metric3Dv2 (Hu et al., 2024) in all three
datasets’. How to get that conclusion? According to Table7, it seems that 'ViT+DPT Head' model performs poorly on MatrixCity dataset, with its performance significantly lower than that of Metric3D v2.

Q2: In Sec. 4.1 and Sec. 4.2 texts, are the Table 9 and Table 10 incorrectly referenced?

Q3: Why some methods in Table 9 are not included in Table 10?
soundness: 3
strengths: 1. Exhaustive experiments have led to a wealth of meaningful conclusions
2. Comparisons based on the same training conditions are helpful in identifying superior structures, loss functions, and training methods.
summary: This paper thoroughly experiments with and compares existing state-of-the-art (SOTA) monocular depth estimation methods, with the experimental comparisons covering both discriminative and generative pretraining methods. The paper not only categorizes and compares the methods in terms of technical details but also contrasts them using the released weights of each method. Additionally, the paper re-trains the models under consistent experimental conditions to compare the performance differences among the methods. It provides a comprehensive summary and comparison of the existing methods. Furthermore, the paper presents some effective conclusions that serve as guidance for subsequent monocular depth estimation research and the authors build a unified code base, which may promote community development.
weaknesses: W1:  L419 'However, the performance of small-scale dataset finetuning is largely behind the official model trained on 16M data samples. Hence, large-scale datasets with diverse scales and cameras is still indispensable for metric depth estimation.' This conclusion is not convincing enough, since the original Metric3D V2 has been trained for 800k iterations with a batch of 192, while for the retrained model, the training iteration is 30k with unknown batch number (maybe 96 mentioned in L397?).

W2: About the evaluation metrics. Since the authors pay attention to the current zero-shot depth estimation and normal estimation methods, which pay much attention to 'robustness', maybe a metric about the error variance is needed to represent the robustness of each model on the benchmark. For example, now the AbsRel and delta_1 are all the mean values, maybe you can present the variance of AbsRel and delta_1 to present the model robustness.

W3: the overall performance is needed. Since most of the Tables are evaluated through several datasets/benchmarks, a overall result in each table is needed to present the model performance.

W4: most of the experiments are conducted under insufficient training (20k iterations), which may affect the validity of the experimental conclusions.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Can you explain this expression in Line 64 to 65: The performance distinction between discriminative and generative geometry estimation models when trained on the same scale and quality of data also remains unclear.
soundness: 3
strengths: The article is well-written, clear, and easy to understand. It presents extensive comparative experiments to analyze and contrast various methods thoroughly. The paper offers several practical and insightful conclusions that contribute to advancing the field. For example, the findings suggest that the diffusion-based protocol may not be optimal for fine-tuning generative models. Instead, a simpler one-step fine-tuning approach is sufficient for tasks such as depth and surface normal estimation. Notably, both discriminative and generative pretraining methods perform effectively with small-scale, high-quality data, though discriminative models like DINOv2 show a slight advantage over generative counterparts. Additionally, the study highlights that metric depth estimation requires significantly more fine-tuning data than scale-invariant depth estimation to accurately capture depth scale. These findings provide valuable guidance for future developments in geometry estimation research.
summary: This paper examines factors affecting geometry estimation model performance, comparing discriminative and generative pretraining methods. While discriminative models need extensive fine-tuning data, generative methods can generalize using minimal synthetic data. To fairly assess models, the authors establish unified baselines. Results suggest that simpler one-step fine-tuning suffices for generative tasks like depth and surface normal estimation, with DINOv2 discriminative models slightly outperforming generative ones. Additionally, metric depth estimation requires more fine-tuning data than scale-invariant tasks. This study aims to guide improvements in geometry estimation models.
weaknesses: 1. The article resembles an experimental report more than a formal academic paper. While it thoroughly documents the comparisons, it lacks the depth and structure typically expected in scholarly research.

2. Lack of novelty. Although the paper offers valuable recommendations, it falls short of proposing specific solutions or implementing them, which limits its contribution to advancing methodology.

3. While the paper appears to aim at establishing a new benchmark, it does not introduce a new dataset but rather relies on existing datasets. This reliance on pre-existing data weakens the contribution toward setting a truly novel benchmark, as it doesn’t address potential limitations or gaps within current datasets that new benchmarks typically seek to fill.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 1
questions: How can I see from Table 7 that "high-quality fine-tuning data" was used or not used? What makes fine-tuning data "high-quality"?

What is, from your point of view, the strongest finding in this paper? Why?
soundness: 2
strengths: I generally like papers that run a deeper analysis on existing methodology. This paper aims a bit in this direction, unfortunately not in a sufficiently rigorous way (see weaknesses). One small positive step in this direction I noticed in conjunction with the decoder supervision when training the surface normal estimation and the intervention by modifying the GenPercept method.
summary: The paper is a survey on some recent monocular depth estimation and surface normal estimation methods, in particular such models that are derived via finetuning from DINOv2 or Stable Diffusion 2.1 foundation models. The paper runs several comparison experiments between these methods with different fine-tuning modes and tries to derive some conclusive insights from these experiments.
weaknesses: 1. The paper lacks clarity and is hard to read. I was wondering actively why after each sentence I read I had problems remembering what I just read. The result of my analysis: the sentences use strange or vague terminology with insufficient context and were not to the point. The same is true for whole paragraphs. Information is spread all over the place without any help given to the reader to make sense of it. This makes it very tedious to read this paper. It was no fun reading it. I did it because I had to. 

2. There are only vague formulations when it comes to the contributions. For example, it is claimed that "we build fair and strong baselines", but when reading on, the baselines are simply existing models that have been finetuned on a different data subset. This indeed allows a more direct comparison, but nothing new is built here. Another example on benchmarks: "we build more diverse scenes with high-quality labels for geometry evaluation. For depth estimation, we introduce three extra benchmark datasets", but in fact these datasets already exist and were simply used for the experiments. Again, nothing new is built here. 

3. It would be okay with nothing new being built, if this paper had a strong scientific side. There was some hope from the abstract, which talks about many insights. However, the insights are anecdotal and only mildly conclusive. Except one attribution to decoder supervision, there is no analysis of the causes of a certain behavior. Therefore, the findings can be due to multiple reasons not controlled in the experiments. As a result, there is nothing conclusive I can take home. The conclusion by the authors "We identify that a strong pre-train
model, either Stable Diffusion or DINOv2, combined with high-quality fine-tuning data, is the key to achieving generalizable monocular geometry estimation." doesn't really tell anything. Only Stable Diffusion and DINOv2 were tested, so it is impossible to identify that they are really important, and that high-quality fine-tuning data (what does high-quality mean exactly?) is important for good results is not a new finding.