BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation
Authors: Xiang Zhang
Bingxin Ke
Hayko Riemenschneider
Nando Metzger
Anton Obukhov
Markus Gross
Konrad Schindler
Christopher Schroers
Venue: NeurIPS 2024 poster (2024)
Keywords: Monocular depth estimation
diffusion model
zero-shot transfer
plug-and-play
Primary area: machine_vision
TL;DR: We present BetterDepth to efficiently achieve robust affine-invariant monocular depth estimation with fine-grained details.

Abstract: By training over large-scale datasets, zero-shot monocular depth estimation (MDE) methods show robust performance in the wild but often suffer from insufficient detail. Although recent diffusion-based MDE approaches exhibit a superior ability to extract details, they struggle in geometrically complex scenes that challenge their geometry prior, trained on less diverse 3D  data. To leverage the complementary merits of both worlds, we propose BetterDepth to achieve geometrically correct affine-invariant MDE while capturing fine details. Specifically, BetterDepth is a conditional diffusion-based refiner that takes the prediction from pre-trained MDE models as depth conditioning, in which the global depth layout is well-captured, and iteratively refines details based on the input image. For the training of such a refiner, we propose global pre-alignment and local patch masking methods to ensure BetterDepth remains faithful to the depth conditioning while learning to add fine-grained scene details. With efficient training on small-scale synthetic datasets, BetterDepth achieves state-of-the-art zero-shot MDE performance on diverse public datasets and on in-the-wild scenes. Moreover, BetterDepth can improve the performance of other MDE models in a plug-and-play manner without further re-training.

BetterDepth: Plug-and-Play Diffusion Refiner for Zero-Shot Monocular Depth Estimation
Xiang Zhangl .2 Bingxin Ke', Hayko Riemenschneider'' Nando Metzger', Anton Obukhov Markus Gross Konrad Schindler', Christopher Schroers EETH Zürich, *DisseyyeseacchStuuiis
Abstract
By training over large-scale datasets, zero-shot monocular depth estimation (MDE) methods show robust performance in the wild but often suffer from insufficient detail. Although recent diffusion-based MDE approaches exhibit a superior ability extract details, they struggle in geometrically complex scenes that challenge their geometry prior, trained on less diverse 3D data. To leverage the complementary merits of both worlds, we propose BetterDepth to achieve geometrically correct invarrint MDE while capturing fine details. Specifically, BetterDepth is a conditional diffusion- based refiner that takes the prediction from pre-trained MDE models as depth conditioning, in which the global depth layout is well-captured, and iteratively refines details based on the input image. For the training of such refiner, propose global pre-alignment and local patch masking methods ensure BetterDepth remains faithful to the depth conditioning while learning to add fine-grained scene details. With efficient training small-scale synthetic datasets, BetterDepth achieves state-of tftte-of-thhaart tht-hhh-rt art zero-shot MDE performance on diverse public datasets and on in-the-wild scenes. Moreover, BetterDepth improve the performance of other MDE models in a plug-and- aade-aaddlllyy manner without further re-training.
1 Introduction
As fundamental task in computer vision, monocular depth estimation (MDE) aims to extract depth information from single-view images, benefitting various real-world applications [46 48 27]
Unlike traditional depth estimation techniques that utilize geometric relationships from stereo 21
or structured light setups [42), MDE highly ill-posed task and relies on the geometric prior knowledge learned from training datasets, where real data plays pivotal role ensure generalization to in-the-wild applications [31 30 49). However, due to the difficulty of collecting fine-grained depth labels in real scenarios, real-world depth labels often coarse, noisy, and incomplete, resulting in a trade-off between the quality and generalization of MDE. Thus, although significant progress in zero-shot MDE has been achieved with techniques like mixing diverse training datasets [31 and unleashing large-scale unlabeled data [4), previous MDE approaches often suffer from over-smoothing of details, indicated by the red arrows in Fig.
Recently, diffusion models have exhibited promising performance in a variety of computer vision tasks 13 44 22
47 32 including MDE [39 17 10). Benefitting from the iterative refinement scheme, diffusion- based MDE methods can produce impressive depth maps with fine granularity depicted in Fig. I However, training diffusion- based MDE generally requires complete depth which is in practice achieved by rendering synthetic datasets. Compared to real data, existing synthetic RGB-I) datasets exhibit lower variety and contain fewer samples which limits the generality the learned prior. Despite several attempts to improve the generalization of diffusion-based MDE, such as label infilling 39 and transferring 2D image priors (17), current
labels 117 10
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Input Image
MiDaS
DPT
Depth Anything
Marigold
BetterDepth (Ours)
Figure 1: Monocular depth estimation (depth map and 3D reconstruction with color-coded normals). Feed-forward methods, like Depth Anything [49], produce robust global 3D shape but suffer from over-smoothed details. Diffusion-basee methods, like Marigold [77] extract fine details but fall short in zero-shot global shape recovery. Our proposed BetterDepth offers the best of both worlds and achieves robust zero-shot depth estimation with fine details.
diffusion-based approaches still have relatively limited a-priori knowledge of global layout. This results in less accurate predictions challenging scenes compared to models trained with diverse datasets, e.g., Depth Anything
49
(Tab. 2
In this work, we aim for robust affine-invariant MDE while also capturing fine-grained details. Motivated by the complementary merits of feed-forward and liffusion-basee MDE methods, we propose BetterDepth boost pre-trained MDE models with diffusion refiner, simultancously leveraging rich geometric priors for zero-shot transfer and diffusion models for detail refinement. Specifically, BetterDepth is designed as a depth--ondiitonnd diffusion model retain the zero-shot generalization power of pre-trained MDE models. Through efficient training on small-scale synthetic datasets, BetterDepth further attains a remarkable ability to extract details (Fig. [ and can directly improve other MDE models, without re-training. To learn detail refinement and simultaneously preserve the prior knowledge from pre-trained MDE models, we introduce global pre-alignment and local patch masking strategies during training, to ensure the faithfulness of BetterDepth to depth conditioning while enabling fine- ane-grained detail extraction. In this way, BetterDepth combines the advantages of zero-shot and diffusion-basse MDE models, exhibiting state-of-the-arr performance and producing visually superior results on diverse datasets. Our main contributions are:
We propose BetterDepth to boost zero-shot MDE methods with a plug soug-and-play diffusion refiner, achieving robust affine-invariant MDE performance with fine- one-grained details.
We design global pre-alignment and local patch masking strategies to enable learning the refinement from small-scale synthetic datasets while preserving the rich prior knowledge in pre-trained zero-shot MDE models.
2
Related Work
Zero-Shot Monocular Depth Estimation. A variety of attempts are devoted to improving the robustness of MDE in the wild, i.e., zero-shot depth estimation, which aims predict depth for any input image taken in unconstrained settings [3] 4 geometrically ill-posed problem, many zero-shot MDE works are designed to estimate affine- intine-ivvrrant depth, i.e., predicting the depth values up to an unknown global scale and shift [31 I7 53 12 49). For example, MegaDepth 23 and DeverseDepth [51] collect internet images for network training, improving adaptation to unseen scenes. Furthermore, MiDaS 31 proposes family of scale- and shift-invariant losses to handle the different depth representations, e.g., metric depth and inverse depth (disparity). across datascts, as to mix diverse training datasets and reach robust zero-shot
53 55
52
15 29 Considering that MDE is

---

transfer. By replacing CNN backbones with powerful vision transformers. DPT [30] and Omnidata g further boost the performance of zero-shot depth estimation. Recently, Depth Anything developed a semi-supervised strategy to unleash the power of large-scale unlabeled images (62M) and acquire a robust representation for in-the-wild prediction [49]. Although the zero-shot generalization of MDE grows with the amount of training data, the lower-quality labels in real-world datasets tend to hinder the reconstruction of fine-grat tnne-griied depth details, resulting in over-smoothing as shown in Fig. |
Diffusion- Basiodd----d Monocular Depth Estimation. The emergence of denoising diffusion probabilistic models (DDPMs) brought up a new paradigm for image generation, producing high-quality images with realistic details [13 44 [4]. Many works have showcased the effectiveness of diffusion models in generating rhoto-realistic results for various computer vision tasks [37 [22 [3] 25 [5]. In the realm of MDE, DDP [6] describes a diffusion-based framework for dense visual prediction tasks, and
DeftusionDepthh further utilizes Swin transformers [24] for image encoding, performing iterative
refinement in the depth latent space. Considering the noisy and sparse depth labels in practice, several techniques are proposed, e.g., depth infilling [40] and sife-suueeiiee pre-training (39, to achieve better MDE performance. A recently emerging trend is to exploit the prior knowledge in foundational
diffusion models for MDE [56] [7 12]. Marigold [7 proposes an efficient fine-tuning protocol to leverage the rich prior in the Stable Diffusion model (34] for depth estimation, producing visually compelling depth results. Following this direction, DepthFM [12] improves inference speed with flow matching, and GeoWizzrt [10] utilizes cross- modal relations for joint depth and normal prediction. However, existing diffusion- based approaches still struggle to outperform the feed-forward MDE models like Depth Anything
[49] (Tab. D). due to the difficulty of learning diverse geometric depth priors from datasets with few or sparse depth labels [39]. By contrast, our BetterDepth efficiently leverages the rich prior knowledge of feed-forward models and improves the extraction of details with diffusion, achieving state-of-the-ar MDE performance (Tab. ( with compelling visual results
(Figs. and 5)
3 Method
We first analyze existing MDE methods and formulate our objective in Sec.
3.1 Based on the analysis, we then propose our DepthrDepth framework in Sec. 3.2 and introduce the training and inference strategies designed specifically for BetterDepth in Sec. 3.3 and 3.4 respectively.
3.1 Problem Formulation
Model architecture and training data are two key factors that determine MDE performance. Given depth dataset [(x,, d:)]; E D with x; and d, corresponding to images and depth labels, previous zero-shot MDE approaches usually employ feed-forward models MFED and learn depth estimation using the following training objective (0, 49
LMDE(d, MFFD(X:)).
where (-DE(.) represents a suitable loss function, e.g., scale- and shift-invariant losses [3 Since d only used to supervise model outputs in Eq. ① feed-forward MDE methods can easily handle invalid pixels in depth labels via techniques like masking, and thus gain robust zero-shot capability by learning from diverse large-scale datasets [3130) 49]. To handle the synthetic-to- gaps caused by synthetic data D syn I. real-world datasets Dreat are often simultancously employed to learn more robust representations for in-the-wild prediction. However, the quality of depth labels in Dreat usually hinders present in the inputs, resulting in over-smoothed details, as depicted in Fig. [
domain
methods from learning to capture high-frequency information
By contrast, diffusion-based MDE approaches generally excel at capturing fine-graines details via iterative refinement 17 [TO. Different from feed-forward methods, diffusion models MDM comprise a T -seep forward process to gradually corrupt samples with Gaussian noise at each timestamp t {I..... T), and learned reverse process to transform random Gaussian noise to a sample from the target data distribution [13 [4]. Instead of directly fitting d, in Eq. ( one typically learns to estimate the added Gaussian noise from x; and d, at each timestamp t, i.e.:
Lma MAM (x;, AddNoise (d,, e,))
where e ~ NO I) denotes Gaussian noise; AddNoise(.) is an operator that corrupts depth labels d; with noise according to t; (DM(() represents loss function for diffusion models, like the velocity
3

---

Table 1: Performance comparison between feed-forward and diffusion- MDE. MFFD and MDM correspond to denote synthetic and real datasets, respectively. ( D) the output distribution with a selected model M and training set D. Our goal is to approach the ideal distribution X (MMieaal Didea)) and
and diffusion-based architectures, respectively. Dayn and Dreal
achieve shot MDE with precise details.
Model
Training Data
Outpui Distribution
Fine Grained Details
Zero Shot Generalizability
MppD
Dayn, Dreat
XM (Den, Draai (MM.. Dayn)
MMM
D Daynt
Madeal
Dateal
(d Dideal)
diffisioo methods are trained on synthetic data, to superior recomstruction details.
prediction loss [38]. Since the depth labels are treated as model inputs in Eq. (2), directly training MDM with sparse depth labels becomes challenging [39], preventing training with diverse real-world data, thus limiting the generalization ability of Liffuusio-based MDE.
Based on the above analysis, we summarize the characteristics of feed-forward and diffusion- based MDE methods in Tab. [ where X' (MM, D) represents the output distribution, as a function of the employed model architecture M and training datasets D. Motivated by the complementary strengths of X(MFrD. (Dayn: Dreat) and X' (MM, Dayn), our goal is to approach the ideal distribution X' (Miaal, Dideal) and achieve robust zero-shot MDE with fine-gran tne-grained details. However, to reach this in a tractable manner, challenges exist from both the model and data perspectives:
Model Limitation. potential solution is to train diffusion models over diverse datasets, i.e., Mideal = MDM and D ideell (Dayu: )reat)- However, how to efficiently train MDM with Dreat while preserving the functionality to extract fine ome-grained details remains an open question. In addition, training over large datasets is required to gain robust zero-shot generalization, which would be extremely time-consuming and resource-intensive.
Data Limitation. Another possible method is to train feed-forward models MFFD with high-quality diverse datasets. However, although high-quality labels are available in Dayn= training solely with Dayn introduces a detrimental synthetic-to-real domain gap []] Mean- while, real depth labels in Dreat must be collected with depth sensors like ToF cameras or LiDAR 11I. which inherently limits the achievable quality of the supervision.
3.2 BetterDepth Framework
To circumvent the aforementioned limitations, we propose BetterDepth to efficiently leverage the strengths of feed-forward and diffusion- based methods, achieving better MDE performance. Specifi- cally, BetterDepth is composed of a conditional latent diffusion model and a pre-trained feed-forward MDE model, illustrated in Fig. 2 Since MFED is known to reach strong zero-shot generalization by training on large and diverse datasets, we first utilize the rich geometric prior from pre-trained MFFD, e.g., DPT BOO Depth Anything [49], to ensure accurate estimation of the global depth context. Based on this, a learnable MDM is employed to locally improve the estimation of details via iterative refinement. To enable the processing of high-resolution images, we follow Marigold [17] and implement MDM with Stable Diffusion [34), which maps from pixel space to lower-dimensional latent space with a variational autoencoder (VAE) [9] and performs denoising with a U-Net in latent space. Because we treat MFED as knowledge reservoir for zero-shot generalization and only need to train MDM for refinement, BetterDepth only requires small synthetic training dataset, e.g., 400 data pairs as shown in Tab, _ Furthermore, the trained MDM in BetterDepth can be directly transferred to improve other MEED models, without re-training.
3.3 Training Strategies
The training pipeline of BetterDepth is illustrated in Fig.2 Although the pre-trained model MEED in BetterDepth provides coarse depth estimates as reliable conditioning, directly training the diffusion- based refiner MDM with synthetic data still tends to overfit the training data distribution, resulting in similar performance as X (M, Dayn) and degarding generalization. enhance the faithfulness of BetterDepth to the depth conditioning while still enabling refinement of details, modify the

---

Macked Truining Objective
Aftine Invorant
Latentt
Latent UNCE
Maro
Sy
Gintel
alvent Enceader
a'
Proviigg
d'
Dowescate
Threshokingg
Smilrty
Nusaa
Messure
Frosen
Figure 2: BetterDepth training pipeline. Given training images and labels d, we first estimate coarse depth maps d with the pre-trained MEED and apply global pre-alignment to d using d as reference. Afterwards, the frozen latent encoder is employed to convert the image x, the depth labels d, and the aligned depth conditioning d' to the latent space. To construct the masked training objective, d' and are split into overlappag patches {d'} } and {d,). J. and dissimilar patches are filter out by thresholding, producing the patch-level similarity mask. Finally, the mask is downscaled to the latent space resolution for diffusion training.
diffusion training pipeline to include global pre-alignment and local patch masking techniques, simultaneously promoting zero-shot MDE capability and fine-grained detail extraction.
Global Pre-Aliinmentt. To alleviate overfitting, we first propose a global pre-alignment method to narrow the gap between the conditioning depth map and the ground truth depth, enforcing BetterDepth to follow depth conditioning at global scale, Given pre-trained affine-invariat depth model MEFD and data pair (x. d) D eyn (subscript is omitted for brevity), we first estimate coarse depth map d via MFFD (x) as depicted in Fig. 2 Although d and d correspond to the same image x, the estimated depth values
generally deviate from due to the unknown scale and shift, which stops BetterDepth from establishing strong dependency between the depth conditioning and the final estimate during training. We resolve this with a global pre-alignment to eliminate the difference caused by the unknown scale and shift. Inspired by the affine- invariant depth evaluation protocol 31. we first estimate the scale and shift b and then align d to the depth labels d, i.c.
d' = sd + b, where (s, b) arg min ssd + bbdl 1|2
Eq. (3) is solved via least squares fitting and d' indicates the aligned depth conditioning. Afterwards, the frozen latent VAE encoder is employed to project x, d', to latent space, corresponding to x², zd' zd. We then follow the DDPM training scheme 13 to generate a noisy sample zi = Vazza + +I- ā,e with Gaussian noise ~ (O, I), where z := zá, ā. == I1 1- and 10,,... 3r is the variance schedule of T-sstep process. Finally, the noisy sample z is concatenated with the latent image and depth conditioning x² zā² as inputs to train the latent U-Net.
Although our pre-alignment strengthens the conditioning by ensuring similar global depth range between d' and the depth label d, misalignment still exists in local regions due to the estimation bias of the pre-trained MDE model. Even though rectifying the coarse depth conditioning d' to the high-quality label d during training might intuitively seem helpful to MDE performance, we find that rectifying significantly different local regions between d' and d also degrades the zero-shot performance. This is because the pre-trained depth model embeds rich prior knowledge of the visual world, which is more important than the dataset-specific knowledge learned in small-scale training sets. Thus, we next propose local patch masking further improve the efficacy of depth conditioning in local regions while learning detail refinement.
Local Patch Masking, As shown in Fig. 2 we first estimate the latent space mask m from depth label d and the aligned depth conditioning d'. and then construct masked diffusion objective for training. In detail, d' and d are first split into non-overlapping local patches {d' {d,), ), where d', e =xx
5

---

and d, E R=x", with w the patch size. For each pair of patches we measure the similarity using the Euclidean distance, i.e.,
(ist d..) =|dd' dxl12
(4)
and then generate the pixel space mask M by
1, if Dist(d,,, d,)) S w-17.
Mn (o.
(5)
otherwise,
where y indicates the average tolerance per pixel in the patch and controls the trade-off between depth conditioning and refinement details. To fit the latent diffusion scheme, the pixel space mask M is then downscaled to latent space mask m via m MM)) Finally, m is applied to the velocity prediction objective 38) for model training.
P [ [0 (2, 1) v(2u: 6, m
where 7 is the number of valid elements in m; (z indicates the velocity estimated from U-Net with z Cat zá ne);; v(z) 6, () denotes the ground-truth velocity defined as v(zg e, = vā,e - V1 - dg [38]. With the masked training objective, BetterDepth not only strengthens the depth conditioning by discarding significantly dissimilar patches but learns to capture fine-grained details from the remaining patch pairs without overfitting the training data.
We further analyze the effectiveness of our training strategies from the perspective of data distribution. As illustrated in Fig. the learned output distribution of BetterDepth (denoted as X) initially covers X' (MM., Dayu) without either pre-alignment or patch masking, as we essentially train a diffusion model with synthetic data in Better- Depth. Thus the resulting model is able to extract fine-grained details but falls short in generalization according to Tab. I By applying global pre-alignment, we bring X' closer to the output distribution of the pre-trained depth model, i.e., X((FFDD. (Dayn; D Fr. which equips BetterDepth with better zero-shot capability by enhancing the conditioning strength at the global scale. Finally, with local patch masking, we filter out significantly different patches and further shrink & toward the intersection of (((MFD,, (Dayn: Dreal)) }) and X'MDM. Dsyu)- Therefore, BetterDepth gains the advantages both worlds and inherits the prior knowledge from the pre-trained depth model while learning to extract fine-gra fine-grained details with diffusion, approximating M Didea) in Tab.
0
X(MYrD, (Dyya Dmw))
X(Mosrr Day))
Pre-Aligmment
Patch Masking
Figure 3: Illustration of output distributions after applying aeegallgnment and patch masking. The output distribution of BetterDepth (λ') is pushed towards the intersection of X (MFFD. (Dyya Dreat }) and A'MMDM, Daya) to achieve detailed zero-shot MDE.
3.4 Inference Strategies
The inference pipeline is depicted in 4 Similar to the training procedure, we first generate a coarse depth map d from the input image x, i.e., d MFFD (x). and then convert into latent codesz*, zd as conditioning. In the latent space, we sample the initial value from standard Gaussian noise, i.e., Cat(z*, zá z
zi_r (O, and concatenate it with z², za as input to the U-Net. where the depth conditioning ensures generalization and the image conditioning provides auxiliary information for refinement. After T-step iterative refinement with the pre-trained U-Net to (z, (), the clean latent z is decoded to a final depth map d via the latent VAE decoder.
Blug-and-Play Once trained, BetterDepth can directly refine the output of previously unseen MDE models, without any additional training. This advantage comes from the different roles of MFFD and MpM in BetterDepth. According to proposed training strategy, BetterDepth treats MFFD

---

Herative Refinement
invariant Myzo
Latent Encoder
Latent UNet
Dencise
Latent Decodar
Sample
Noise A/(0.))
Encoder
Figure 4: BetterDepth inference pipeline. Given an image x and pre-trained depth model, we first estimate the coarse depth map d as conditioning, After converting x and to latent space, we concatenate the latent codes z² zd with the depth latent 2! for denoising. After T-step refinement, random Gaussian noise z has been converted to z and is decoded to the final estimate d.
as the knowledge reservoir to ensure zero-shot MDE performance and utilizes MAM only to refine details. When faced with different MFFD. BetterDepth inherits correspondingly different prior, but maintains the functionality to add fine-grained details to it. Given the increasing trend to train foundational MDE models [49], BetterDepth can be flexibly added to models as refinement module to enhance the extraction of details.
Experiments and Analysis
4.1 Experimental Settings
Implementation. We employ Depth Anything 49 as MIFD and use the Marigold architecture 17
with Stable Diffusion weight initialization 34 MDM in our BetterDepth,, where only fine-tune the denoising U-Net. BetterDepth is trained for 5K iterations with batch size 32. The training takes around 1.5 days
single NVIDIA RTX A6000 GPU. The Adam optimizer [18) is used with the
learning rate set to 3 x 10-5, We set the patch size w = 8 and the masking threshold v) = 0.1 under
the depth range [-1, 1]. For inference, apply the DDIM scheduler with 50-step sampling 4 and
obtain the final result with 10 test-time ensemble members [17].
Datasets and Evaluation. We follow Marigold [17 and use 74K samples from two synthetic datasets Hypersim 333 and Virtual KITTI [2] for training. Additionally, we construct two smaller datasets by randomly selecting 2K and 400 samples, respectively, from the full training dataset to test the performance of BetterDepth with fewer training samples (denoted as BetterDepth-22 and Better 400). For evaluation, we employ five unseen datasets NYUv2 (28 (654 samples), KITTI [I]] (652
samples from the Eigen test split (9), ETH3D 43
(454 samples), ScanNet [6] (800 samples based on the Marigold split (77, and DIODE 45 (325 indoor samples and 446 outdoor ones), and conduct
quantitative comparisons with two metrics, AbsRel (absolute relative error: N Σ [d, dgl/d; with N denoting the number of pixels) and 51 (percentage of max(a, (a,/d, /d d, < 1.25). In-the- wild images are also collected for qualitative evaluation of zero-shot MDE.
4.2 Benchmarking
In this section, compare BetterDepth with state-of-the inttrnnivvariant MDE methods to show its superior zero-shot performance and reconstruction details.
Zero-Shot Performance. Tab. shows the results for BetterDepth compared with both feed- forwadd and diffusion-basse based MDE approaches. Benefitting from the proposed framework and training strategies, BetterDepth successfully combines the geometric prior from the pre-trained depth model with the ability to model fine details. Specifically, BetterDeethh22K already achieves state-of-the- performance and DetherDee still compares favorably to prior art. In addition, different MDE models can directly plugged into the BetterDepth framework, which consistently improves their outputs across most datasets, demonstrated in Tab, BetterDepth also outperforms existing MDE

---

Table Quantitative evaluation of zero-shot performance with state-of-the -f--of--teerrr-- affineiivvariant MDE methods. #Train is the amount of training data. FFD and DM correspond to feed-fore forward and diffusion models. Metrics are shown percentage with best and second-best results marked. The average rank cannot be computed for DepthFM due to missing metrics on ETH3D and ScanNet.
Type DM
NYUv2
ETIED
DIODE AbsRal].
Avg.
Method
#Train
FFD
AhcRell
811
AbsRalj Al1
AhsRell. dlf
AhsRell 811
61
Rank
DeverseDepih 51
320K
11.7
MiDaS 31
2M
9.5
91.5
18.3
71.1
19.0
88.4
9.9
90.7
26.6
71.3
10.3
LeReS
354K
9.0
91.6
14,9
78.4
17.1
77.7
91.7
27.1
76.6
Omnidata
8
12.2M
7.4
94.5
14.9
83.5
16.6
77.8
7.5
93.6
33,9
74.2
8.9
HDN 55
300K
6.9
94.8
11.5
86.7
12.1
83.3
93.9
24.6
78.0
DPT 20
91.9
11.1
88.1
11.5
92.9
93.2
73.0
Depth Anything
49
63.5M
4.3
98.0
8.0
94.6
6.2
98.0
4,3
98.1
26.0
75,9
2.9
Marigold 1
74K
5.5
96.4
9.9
91,6
6.5
96.0
6,4
95.1
30.8
77.3
5.6
DeptiFM
12
63K
6.5
95.6
8.3
93.4
22.5
80.0
GeoWizard h0
280K
5.2
96.6
92.1
96.1
95.3
79.2
RetterDepth-400 (Ours)
97.9
94.5
97.8
97.8
75.3
BetterDepth-2K (Ours)
2K
4.4
97.9
7,4
95.1
4.7
98.1
4.3
98.0
22.0
75.5
2.7
BetterDepth (Ours)
74K
4.2
98.0
7.5
95.2
4.7
98.1
98.1
22.6
75.5
Table 3: Plug-and-play experiments. BetterDepth directly works with CNN-based (MiDaS [11
and transformer-based MDE models (DPT (30), improving their results without re-training.
NYUv2
KITTI
ETH3D
ScanNet AbsRell 61f
DIODE AbsRel] 61f
Method
AbsRel. 611
AbsRel4 51f
AbsRel
811
MiDaS 31
9.5
91.5
18.3
19.0
88.4
90.7
26.6
71.3
MetterDepti-MiDaS
8,4
93.4
15.1
78.4
17.9
91.2
9.3
91.6
26.6
71.9
Improvement
1.1
1.9
3.2
7.3
1.1
2.8
0.6
0.9
0.0
0.3
DPT 30
9.1
91.9
11.1
88.1
11.5
92.9
8.4
93.2
26.9
73.0
BetterDepth+PPT
7.9
93.7
10.0
89.8
10.3
94.5
7.8
93.8
26.5
73.6
Improvement 1.2
1.8
1.1
1.7
1.2
1.6
0.6
0.6
0.4
0.6
Table 4: Quantitative evaluation of detail extraction on Middlebury 2014 41
Edge-based metrics,
i.e., the completeness and accuracy of depth boundaries (DBE_ _emmpccmppp and DBE_ace (20] and the edge precision and recall (EP and ER) [14], are also shown to evaluate performance specifically on high-frequency details. The best and second-best results marked.
Method
AbsRel (%)
81 (%)
DBE_ comp
DBE_ac
EP (%)
ER (%)
Marigold 17
7.57
93.24
5.60
3.09
16.65
23.75
Depth Anything BetterDepth (Ours)
3.14
99.44
6.35
2.66
24.73
16.12
2.95
99.52
3.61
2.09
28.49
50.35
approaches in visual quality as depicted in Fig. andd Compared with previous methods that either suffer from over-smoothing or inaccurate depth layout, BetterDepth correctly recovers the spatial structure of different scenes while capturing small details, leading to visually improved results.
Fine-Grained Detail Extraction. Despite achieving state-of-the-ane performance, Tab, cannot fully represent the performance of BetterDepth (especially w.r.t. details), as the depth labels in commonly used datasets are sparse or noisy, e.g.. Fig. A17 Thus, we further evaluate the ability to reconstruct details on high-resolution RGB-D dataset Middlebury 2014 [41]. Four additional
edge-based metrics are employed to focus on depth discontinuities: the completeness and accuracy of depth boundary errors (20) and the precision and recall for edges [14] As shown in Tab. 4 BetterDepth delivers more accurate estimates in terms of both global and edge-based metrics and succeeds in capturing challenging details, e.g., the fine mesh in Fig. 6
4.3 Ablation Study
we study the effectiveness of each design choice in BetterDepth and draw the following conclusions: (i) Depth Conditioning. Without depth conditioning, model #1 in Tab, performs
In Tab. 5

---

Input Images
MiDaS
DPT
Depth Anything
Marigold
BetterDepth (Ours)
ANNNA
Figure 5: Qualitative comparisons of depth estimation and 3D reconstruction results (colored as normals), where Marigold predicts depth values and the others output disparity.
Image
Marigold
Depth Anything
BetterDepth (Ours)
Figure 6: Visual comparisons Middlebury 2014 41 Details are zoomed in.
Table 5: Ablation study. All variants are trained on the full 74K training pairs for 5K iterations. The best and second-bee best results are marked.
Depth Conditioning
Global PeeAAlimmeet
Local Patch Masking
NYUv2
KITTI
ID
AbsRell
511
AbsRell
811
#1
x
x
6.1
96.1
9.1
90.7
#2
x
X
5.2
97.0
8.6
92.2
X
97.5
94.4
V
4.2
98.0
7.5
95.2
similarly to previous diffusion-baan based methods like Marigold [77, and struggles with generalization only from synthetic training data. By utilizing the geometric prior from the pre-trained depth estimator, model #2 achieves consistent improvements in both indoor and outdoor scenarios, as shown Tab, 5 (ii) Global Pre-Alignment. Despite the improvements gained with depth conditioning, we find the zero-shot performance still remains below the pre-trained Depth Anything model [49]. In other words, even having good depth maps from the pre-trained model as initialization, the naive

---

0
61
(1()))
90
Marigold
Marigold
80
Marigold
BetterDepth
BetterDepth
BetterDepth
60
70
1 3
10
20
Training ieeration
Ensemble size
Denoising step
(a) Convergence comparisons
(b) Impact of ensembling size
(c) Impact of denoising step
Figure 7: Training and inference efficiency compared with Marigold [7 on the KITTI dataset.
conditioning model (#2) struggles to balance the contribution of different priors and does not yield an improvement. This is because model #2 overfits the distribution of training data and under-utilizes the prior knowledge in the pre-trained model. By aligning the depth conditioning to the ground truth during training, model #3 better learns to follow the depth conditioning a global scale and brings further improvements in zero-shot generalization. (iii) Local Patch Masking, Our full model #4, with the masked training objective, exhibits the best performance. By filtering out significantly dissimilar regions with patch masking, ensure that BetterDepth closely adheres to depth conditioning at local scales, thus better exploiting the prior for zero-shot transfer. Meanwhile, operating at patch level fully retains the information in local regions and thus benefits the reconstruction of details, e.g. edges and fine structures, as illustrated in Fig. I and
4.4 Method Analysis
In this section, we further analyze BetterDepth with respect training and inference efficiency.
Training Efficiency. We compare the training efficiency of BetterDepth with the state-of-the-ar diffusion-based method Marigold [17]. Helped by the additional depth conditioning, BetterDepth converges significantly faster than Marigold, as depicted in Fig. 7a With only 200 iterations (*s 1.5 hours of training), BetterDepth achieves comparable performance iterations. Furthermore, since we must only learn to refine details, thanks to the proposed training strategies, BetterDepth outperforms Marigold with fewer training samples, e.g., BetterDepth-400 in Tab. 2 validating the overall strategy.
Marigold trained with 5K
Inference Efficiency. We compare the efficiency at inference time with different ensemble sizes and numbers of denoising steps. Test-time ensembling aims to aggregate information from multiple predictions, and larger ensemble sizes generally bring better and more stable results (17.. As depicted in Fig. 7b on KITTI the 81 difference between a single inference and an ensemble of 10 members is 1.2 percentage points for Marigold but only 0.4 for BetterDepth, confirming its better stability, Meanwhile, BetterDepth produces comparable or even better results than 50-step Marigold with only 2 inference steps, shown in Fig. 7c In terms of inference speed, the 50-step Marigold achieves 91.6% 51 accuracy on KITTI with 10 ensemble members, spending 30.5 seconds per sample on an NVIDIA GeForce RTX 4090 GPU. In contrast, our 2-step BetterDepth achieves 92.5% 81 accuracy in single inference pass with only 0.4 seconds per sample (0.38 seconds for the diffusion denoising and 0.02 seconds for the depth conditioning prediction).
Conclusion
We have presented BetterDepth to achieve robust, detailed, and efficient effine-invarian monocular depth estimates. The proposed method combines the strong prior of massively pre-trained MDE models with the recovery of fine details enabled by diffusion models, and devises training strategies to maximally retain the strengths of both discriminative depth estimation and conditional depth map generation. In this way, BetterDepth achieves state-of-the-art MDE performance and is able to refine different feed-forwate depth estimators without re-training.
10

---

References
[1] Amir Hlapour-Aharghouce and Toby Breckon. Real-time monocular depth estimation using synthetic data with domain adaptation via image style transfer. In CVPR, pages 2800-2810, 2018.
[2] Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual KITTI 2. arXiv preprint arXi:: 2020.
[3] Weifeng Chen, Zhao Fu, Dawei Yang, and Jia Deng. Single-image depth perception the wild. In NIPS, volume 29, 2016.
[4] Weifeng Chen, Shengyi Qian, David Fan, Noriyuki Kojima, Max Hamilton, and Jia Deng. Oasis: large-scale dataset for single image 3d in the wild. In CVPR, pages 679-688, 2020.
[5] Zheng Chen, Yulun Zhang, Ding Liu, Jinjin Gu, Linghe Kong, Xin Yuan, et al. Hierarchical integration diffusion model for realistic image deblurring. In NIPS, volume 36, 2023.
[6] Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
[7] Yiqun Duan, Xianda Guo, and Zheng Zhu, Diffusiondepth: Diffusion denoising approach for monocular depth estimation. arXiv preprint arXiv:2303.05021 2023.
[8] Ainaz Fftekhar, Alexander Sax, Jitendra Malik, and Amir 7amir. Omnidata: scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In ICCV, pages 10786-10796, 2021.
[9] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. In NIPS, 2014.
[10] Xiao Fu, Wei Yin, Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. arXiv preprint arXiv:2403. 12013, 2024.
[11] Andreas Geiger. Philip Lenz. and Raquel Urtasun. Are we ready for autonomous driving? the KITTI vision benchmark suite. In CVPR, 2012.
[12] Ming Gui, Johannes S Fischer, Ulrich Prestel, Pingchuan Ma, Dmytro Kotovenko, Olga Grebenkova, Stefan Andreas Baumann, Vincent Tao Hu, and Björn Ommer. Depthfm: Fast monocular depth estimation with flow matching. arXiv preprint arXiv:2403. 137888 2024.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NIPS, volume 33, pages 6840 6851, 2020.
[14] Junjie Hu, Mete Ozay. Yan Zhang, and Takayuki Okatani. Revisiting single image depth estimation: Toward higher resolution maps with accurate object boundaries. In WACV, pages 1043-1051. IEEE, 2019.
[15] Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometrie foundation model for zero-shot metric depth and surface normal estimation. PAMI, 2024.
[16] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. In ICCV, pages 21741-21735 2023.
[17] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing hiffusiionhased image generators for monocular depth estimation. In CVPR, 2024.
[18] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
[19] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
[20] Tobias Koch, Lukas Liebel, Friedrich Fraundorfer, and Marco Korner. Evaluation of nasdd single- image depth estimation methods. In ECCVW, 2018
[21] Hamid Laga, Laurent Valentin Jospin, Farid Boussaid, and Mohammed Bennamoun. A survey on deep lcarning techniques for sterco-based depth cstimation. PAMI, 44(4): 2020.
[22] Haoying Li, Yifan Yang. Meng Chang, Shiqi Chen, Huajun Feng. Zhihai Xu, Qi Li, and Yueting Chen. Srdiff: Single image super-resolution with diffusion probabilistic models. Neurocomputing, 479:47-55 59, 2022.
11

---

[23] Zhengqi Li and Noah Snavely. Megadepth: Learning single-view depth prediction from internet photos. In CVPR. pages 2041-2050, 2018.
[24] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012-10022) 2021.
[25] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In CVPR, pages 11461-11471, 2022.
[26] Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprini urXiv:2310.04378, 2023.
[27] Lukas Mehl, Andrés Bruhn, Markus Gross, and Christopher Schroers. Stereo conversion with disparity- aware warping, compositing and inpainting. In WACV, pages 4260-4269, 2024.
[28] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from RGBD images. In ECCV, 2012.
[29] Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In CVPR, pages 10106-10116, 2024.
[30] René Ranftl, Alexcy Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In ICCV, pages 12179- 12188, 2021.
[31] René Ranftl, Katrin Lasinger. David Hafner, Konrad Schindler. and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. PAMI, 6433 2020.
[32] Lucas Relic, Roberto Azevedo, Markus Gross, and Christopher Schroers, Lossy image compression with foundation diffusion models. arXiv preprint arXiv:2404.08550 2024.
[33] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In /CCV, 2021.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser. and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[35] Chitwan Saharia, William Chan, Huiwen Chang. Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-iman -o-iaaae diffusion models. In ACM SIGGRAPH, pages 1 1-10, 2022.
[36] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorcalistic text-to- image diffusion models with deep language understanding. In NIPS, volume 35, pages 36479-36494, 2022.
[37] Chitwan Saharia. Jonathan Ho, William Chan, Tim Salimans. David Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. PAMI, 45(4):4713 2022.
[38] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
[39] Saurabh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David Fleet. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. In NIPS, volume 36, 2023.
[40] Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David Fleet. Monocular depth estimation using difffusion models. arXiv preprint arXiv:2302. 14816. 2023.
[41] Daniel Scharstein, Heiko Hirschmüller, York Kitajima, Greg Krathwohl, Nera Nešić, Xi Wang, and Porter Westling. High-rcsolution stereo datasets with subpixcl-accurate ground truth. In PR, pages 31-42. Springer, 2014.
[42] Daniel Scharstein and Richard Szeliski. High-accuracy stereo depth maps using structured light. In CVPR, volume 1, pages 1-I IEEE, 2003.
[43] Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler. Marc Pollefeys, and Andreas Geiger. A multi- view stereo benchmark with high-resolution images and multi- camera videos. In CVPR, pages 2260-3269, 2017.
12

---

[44]
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[45] Igor Vasiljevic, Nick Kolkin, Shanyi Zhang, Ruotian Luo, Haochen Wang, Falcon Z. Dai, Andrea F. Daniele, Mohammadreza Mostajabi, Steven Basart, Matthew R. Walter, and Gregory Shakhnarovich. DIODE: Dense Indoor and Outdoor DEpth Dataset. arXiv preprint arXiv: 2019.
[46] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, pages 8445 8453, 2019.
[47] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar. Deblurring via stochastic refinement. In CVPR, pages 16293-16303, 2022.
[48] Diana Wofk, Fangchang Ma, Tien-Ju Yang, Sertac Karaman, and Vivienne Sze. Fastdepth: Fast monocular depth estimation on embedded systems. In ICRA, pages 6101-6108 IEEE, 2019.
[49] Lihe Yang. Bingyi Kang. Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, 2024.
[50] Lihe Yang. Ringyi Kang, Zilong Huang. Zhen Zhao, Xiangang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. NIPS, 2024.
[51] Wei Yin, Xinlong Wang, Chunhua Shen, Yifan Liu, Zhi Tian, Songcen Xu, Changming Sun, and Dou Renyin. Diversedepth: Affine-invariant depth prediction using diverse data. arXiv preprint a0Xiv:200.000569, 2020.
[52] Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang. Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from a single image. In ICCV, pages 9043-9053, 2023.
[53] Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. learning to recover 3d scene shape from single image. In CVPR, pages 204-213, 2021.
[54] Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar++:: Accurate depth for 3d object detection in autonomous driving. In ICLR, 2020.
[55] Chi Zhang, Wei Yin, Billzh Wang, Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normalization for robust monocular depth estimation. In NIPS, volume 35, pages 14128-14139 2022.
[56] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In ICCV, pages 5729-5739, 2023.
13

---

Appendix
this appendix, we provide more implementation details, experiments, analysis, and discussions for a compre- hensive evaluation and understanding of BetterDepth.. Detailed contents are listed as follows:
Contents of Appendix
Training Procedure
14
Comparison with Depth Anything V2
15
Combination of Prior Knowledge
15
More BetterDepth Variants
16
Noise Suppression with Mean Ensembling
17
F
Hyperparameter Analysis
17
Influence of Patch Size
17
Masking Threshold and Trade-O1
17
Error Bar Analysis
17
More Visual Results
18
Limitation and Future Work
18
Discussion of Societal Impacts
18
Algorithm 1 BetterDepth Training Procedure
1: repeat
2: (x.d) d) ~ Day
Sample image and depth label
3: MxrD(x)
Estimate coarse depth as conditioning
d' sd b with (s,b) arg min *
sd +
d
Global pre-alignment
5: PatchhMaskEstimatedd
Estimate patch mask
6: xx E(x), 2 E(d'), 2d C(d)
Encode with frozen latent encoder & Sample timestamp Gaussian noise Add noise with velocity prediction method P Concatenate latent features as U-Net input Compute ground truth velocity
Uniform( (1,,..,. ,,,, (0,1 8: Vanza vI are
9: Cat(z* ,2" 2
c,t) 11: Take gradient descent step on
10: u = Vāte- VI- 0:
V | (z, 1) @ m v(z", t) @ ml|
Train latent U-Net with masked objective
12: until converged
A Training Procedure
displays the complete training procedure for the proposed BetterDepth method, where the output type of BetterDepth is consistent with that of the employed MFrD. e.g., BetterDepth predicts affine-invariant inverse depth Depth Anything 449.. Compared with the previous diffusion training scheme for MDE models 33 [7 10], we first design a cepth-conditioned framework to efficiently utilize the rich geometric prior from pre-trained depth models. In addition, global pre alignment and local patch masking methods are proposed to enable learning detail refinement while maintaining the faithfulness of BetterDepth to depth conditioning. achieving robust shot MDE performance with tine-grained details.
Algorithm] I
14

---

Input Image
Depth Anything V1
Depth Anything V2
Marigold
BetterDepth
Figure A1: Visual comparisons with Depth Anything V2 50
Table A1: Quantitative evaluation of zero-shot performance on five unseen datasets. The best and second-best results are marked.
NYUv2
KITTI
ETH3D
ScanNet AbsRell, 61f
DIODE
Avg.
Method
AbsRelj 81t
AbsRel 811
AbsRel, ö1f
AbsRelf, 51f
Rank
Marigold 17
5.5
96.4
9.9
91.6
6.5
96.0
6.4
95.1
30.8
77.3
3.7
Depth Anything
49
4.3
98.0
8.0
94.6
6.2
98.0
4.3
98.1
26.0
75.9
1.9
Depth Anything V2 50
4,4
97.8
8.3
93.9
6.2
98.2
4.2
97.8
26.4
75.4
2.6
BetterDepth (Ours)
4.2
98.0
7.5
95.2
4.7
98.1
4.3
98.1
22.6
75.5
1.4
Table A2: Quantitative evaluation of detail extraction performance the high-resolution dataset Middlebury 2014 [41] Edge-based metrics, i.e., the completeness and accuracy of depth boundary errors (denoted DBE_ comm and DBE_ace (20) and the edge precision and edge recall (denoted
as EP and ER) 14), are also employed to evaluate detail extraction performance. The best and
second- best results marked.
Method
AbsRel (%) 1 81 (%)
DBE_ DBE_ acc
EP (%) 1 ER (%)
Avg. Rank
Marigold 17
7.57
93.24
5.60
3.09
16.65
23.75
3.7
Depth Anything
49
3.14
99.44
6.35
2.66
24.73
16.12
3.2
Depth Anything V2 550
3.06
99.38
4.19
2.23
26.74
35.89
2.2
BetterDepth (Ours)
2.95
99.52
3.61
2.09
28.49
50.35
B
Comparison with Depth Anything V2
this section, we compare BetterDepth to the concurrent work Depth Anything [50 By training on high- quality synthetic datasets, Depth Anything V2 achieves significant performance improvements, e,g., fine detail and transparent objects, over Depth Anything [49). However, we found that both the training dataset and the model architecture are crucial for MDE performance. As shown in Tab, A and A2 although Depth Anything V2 achieves promising performance detail extraction, our BetterDepth still exhibits better performance even with much less synthetic training data (595K Depth Anything V2 v.s. 74K in BetterDepth), thanks to iterative refinement of diffusion model. In addition, BetterDepth also captures better details like the cat's hair in Fig. Al validating its overall best performance.
C Combination of Prior Knowledge
Due to the ill- posedness of the MDE task, rich prior knowledge has been proven important in accurate depth estimation from single-view input [31 30
17 49
Unlike previous MDE methods that mainly exploit single- image priors in Marigold [17]. our BetterDepth
sourced knowledge, e.g., geometric priors in MiDaS [1]
15

---

Table A3: Contribution of the geometric prior and the image prior BetterDepth, where geometric and image priors correspond to the knowledge gained from the pre-trained depth model, Le., Depth Anything [4 and the Stable Diffusion model [34] The model without geometric prior uses the same network and fine-tuning method as Marigold 17 but estimates inverse depth (following
Depth Anything
[49 instead of relative depth. For the model without image prior, we follow Stable
Diffusion [34] train the latent UNet from scratch and keep the pre-trained VAE unchanged. Metrics
are shown percentage terms, where the best and second-ber results are marked.
NYUv2
KITTI AbsRel
ETH3D
ScanNet AbsRely, 611
DIODE AbsRel., 51
Geometric Prior Image Prior
AhsRel, 611
61f
AbsRel. 61
6.1
96.1
9.1
90.7
8.5
96.1
6.5
95.0
22.2
73.7
4.3
98.0
8.0
94.4
5.5
97.8
4.4
98.1
22.6
75.1
4.2
98.0
7.5
95.2
4.7
98.1
4.3
98.1
22.6
75.5
Table A4: Performance of the BetterDepth trained with DPT [30
means the previously unseen models, ie., MiDaS (31 and Depth Anything [49] are directly plugged into the BetterDepth framework (pre-trained with DPT) for improved MDE performance. The best and second-best results are marked.
NYUv2
KITTI
ETH3D
ScanNet AbsRel4 81f
DIODE AbsRelf 811
Avg.
Method
AbsRel 81
AbsRel ö1f AbsRel 51f
Rank
MiDaS 3I
18.3
71.1
19.0
88.4
90.7
26.6
71.3
DPT 30
9.1
91.9
11.1
88.1
11.5
92.9
8.4
93.2
26.9
73.0
4.1
Depth Anything
49
4.3
98.0
8.0
94.6
6.2
98.0
4.3
98.1
26.0
75.9
1.5
BetterDDeth++MiDaa"
7.7
94.3
13.5
81.9
17.8
92.5
8.8
92.3
26.9
72.0
4.7
BetterDepth-DPT
7.3
94.5
9.9
90.4
11.9
95.1
7.5
94.3
27.2
73.6
3.4
BetterDepth++Deph Anything"
4.3
98.1
7.9
94.7
5.5
97.9
4.3
98.1
23.0
75.3
1.2
combines knowledge from different domains. Specifically, BetterDepth utilizes the geometric prior from the pre-trained models. which contains task-specific knowledge for robust depth estimation. Furthermore. BetterDepth also exploits the rich image prior via the Stable Diffusion weight initialization (34), benefiting the extraction of fine-graines details. To investigate the contribution of geometric and image priors in BetterDepth, a related ablation experiment is performed in Tab, A3 It is evident that combining prior knowledge from different sources leads the best MDE performance.
D More BetterDepth Variants
additionally train BetterDepth variant in combination with DPT (30 to further verify effectiveness flexibility of our proposed method. As demonstrated in Tab. A4 BetterDepth+DPT achieves 665/1.76% average performance gain over DPT on AbsRel/81 accuracy across all datasets. When directly combined with previously unseen MDE models, iee. MiDaS 31 and Depth Anything 44), BetterDepth also demonstrates general improvements public zero-shot datascts, showing the ficxibility of our proposed method practical usage.
Apart from the BetterDepth model trained with Depth Anything
Input Image
Median ensembling
Mean ensembling
Figure A2: BetterDepth results, where mean ensembling alleviates the wobble effects.
16

---

Table AS: Performance BetterDepth with different test-time ensembling methods. The best
results are marked.
NYUv2
KITTI
ETH3D
ScanNet AbsRel 61f
DIODE AbsRel 611
Method
AbsRell
811
AbsRell 511
AbsRelj 611
Median ensembling
4.2
98.0
7.5
95.2
4.7
98.1
4.3
98.1
22.6
75.5 75.5
Mean ensembling
4.2
98.1
7.4
95.3
4.6
98.1
4.3
98.1
22.5
E
Noise Suppression with Mean Ensembling
Due to random noise in diffusion process, diffusion- MDE methods, e.g., Marigold BetterDepth. tend to introduce subtle variations in the results, like the wobble effects in the surface normals shown Fig. A simple fix to this issue is to replace the default median operation in the test time ensembling [77 with the operation, which smooths the estimated depth with multiple predictions. As shown in Fig. A2 and Tab, A5 the mean ensembling approach alleviates the wohble effects and achieves slightly hetter depth estimation.
F Hyperparameter Analysis
Influence Patch Size
Patch size u is hyper-
parameter used to estimate patch masks for training. To investigate its impact monocular depth estimation (MDE) performance, we conduct experiments with different choices 8 to 128, where is the min- imal patch size as the em- ployed VAE latent encoder performs 8x downscaling for pixel-to-latent conver-
4.5
NYUv2
98.2
-
AbsRel -
-
AbaRel - 81
95.4
AbuRet
4.4
98.1
(16
AbdRel (36)
7.8
95.2
4.3
98
7.6
4.2
97.9
94.8
from
4.1
97.8
7.4
Patch
Figure A3: Influence of patch size on NYUv2 and KITTI.
sion. As depicted in Fig. [33 the overall MDE performance fluctuates with different patch sizes, and we find setting w = 8 leads to the overall best performance, indicating that small patches are sufficient for learning detail refinement.
F.2 Masking Threshold and Trade-Off
The masking threshold de- termines the difference erance level between local patches {d', } and {d,} } to filter significantly dissimi- lar regions during training. Since inputs are all con-
4.6
NVUv2
98.2
KITTI
AbsRal (38)
AbsRtel
98.1
AhsRRll (56)
7.8
AbeRel - 51
95.4
4.4
51 (56)
98
95
4.2
97.9
verted to [-1,1] space be-
94.8
fore feeding into the VAE
0.05 0.1 0.15 0.2
0.3
0.05 0.1 0.15 0.2
0.3
Threshold
Threshold
latent encoder. we conduct experiments with varying from 0.05 to 0.30, as shown in Fig. A4 Lower gener-
Figure A4: Influence of masking threshold on NYUv2 and KITTI.
ally means stricter filtering, i.e., the remaining patch pairs d', and d,, are more similar to each other. thus often leads to stronger conditioning strength. By contrast, higher is more tolerant when selecting patches and leaves more room for learning detail refinement. Thus, the hyperparameter 7) controls the tradc-off between depth conditioning strength and detail refinement performance, and find a sweet spot 0.1, which shows a good balance both aspects and achieves the overall best MDE results.
G Error Bar Analysis
17

---

Due to the stochastic nature of diffusion models, we perform error bar analysis evaluate the performance stability of BetterDepth on the NYUv2 dataset [28]. Instead of employing the test time ensembling tech- nique 177. we directly generate 10 predictions for the same input with 50 denoising steps and then compute the metrics for each estimate. Finally. we obtain the mean and standard deviation on NYUv2 dataset and compare them with the state-of-the-art diffusion- based MDE method Marigold [17] under the same setting. As illustrated in Fig. A5 BepthrDepth shows significantly better results on both AbsRel and 81 accuracy metrics than Marigold. Meanwhile, thanks to informative geometric cues embedded in the depth conditioning, BetterDepth also exhibits more stable MDE performance than Marigold.
55.99
96
-07.94
AbsRel (30)
3
51
96
96
85
4.34
Marigold BetterDopth
Marigold BetterDepth
Figure A5: Error bar analysis on NYUv2.
H More Visual Results
We provide more visual comparisons both in-the-wild (Fig. A6 and A7) and public datasets (Fig. A8 [A17]. In-the-wild images are captured on diverse indoor/outdoor scenes with varying camera perspectives. The 3D reconstruction results colored with surface normals are also provided in Fig. and [7 for better comparison of detail extraction. By contrast. public datasets contain more specific scenarios, e.g. the indoor dataset NYUv2 (23 and the driving-scene dataset KITTI [ Overall, the proposed BetterDepth shows the best performance estimating the accurate layout of target scenes and extracting fine-graing ee-ggraindd seene details.
I Limitation and Future Work
While remarkable performance is achieved by BetterDepth,, limitations still exist: (i) Model Size and Inference Speed. Since BetterDepth comprises pre-trained MDE model and diffusion-based refiner, the model size determined the chosen architectures of both components. Apart from focusing the utilization of large foundation models, we also plan to investigate the possibility of using more lightweight components in BetterDepth framework, e.g., efficient U-Net (36) as the diffusion refiner, in future research to benefit efficient deployment in practice. In addition, the inference speed is also bounded by the chosen depth model and diffusion network, where the diffusion part usually poses the trade-off between speed and quality [ [7]. Although BetterDepth could potentially boost speed using fewer ensemble members and fewer denoising steps, with slight performance drops as depicted in Fig. [7b and techniques like latent consistency models (26 could also be taken into account for further improvements. (ii) Utilization of Training Data. From the perspective of the training strategies, better pre-alignment approaches like outlie--aware methods could have more patches survive during training for better performance. Although the models trained with small datasets, e.g., BetterDepth- 2K in Tab. [ already achieve comparable results to our full model, indicating that limited patches can be sufficient, better alignment methods could potentially improve patch preservation to further boost the training. (iii) Metric Depth. Finally, improving the performance of metric depth estimation
[52 15 29 and transferring affine-invaaran depth to metric depth promising directions but pose several challenges, e,g., scale/shift ambiguity and diverse depth ranges. would interesting to unlock the potential of BetterDepth in metric depth estimation, and we leave as future work.
J Discussion of Societal Impacts
Our work aims to improve the depth estimation performance from single image with similar scope to other MDE methods. BetterDepth represents progress towards zero-shot, highly detailed depth estimation, and thus might amplify any impacts that MDE has in the socictal context. On the one hand, because of the ficxibility extracting depth information from a single image, MDE can potentially benefit variety of real-world applications, including autonomous driving [4]] [54] roboties [48]. and film production [27]. With the improved performance, BetterDepth could bring positive societal impacts such providing more realistic 3D models, enhancing the precision of depth perception in autonomous vehicles, and accelerating the stereo conversion process for 3D movies. On the other hand, MDE could, like many other computer vision techniques. have negative societal impacts when used improperly. For instance, depth estimation in surveillance systems might raise privacy concerns since it can potentially enable more invasive monitoring and tracking of individuals in public spaces.
18

---

Imput Images
MIDeS
DPT
Depth Anything
Marigeld
BetterDepth (Ours)
Figure A6: Qualitative comparisons in-the-wild samples, part 1. Marigold predicts depth while the others output disparity values. Red indicates the close plane and blue means the far plane.
19

---

Inpul Imagee
MiDaS
DPT
Depth Anything
Marigold
BetterDepth (Ours)
Figure A7:
Qualitative comparisons in-the-wild samples, part 2. Marigold predicts depth while the others output disparity values. Red indicates the close plane and blue means the far plane.
20

---

Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold 1ir
BetterDepth (Ours)
Input Image
Depth Anything
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT
Depth Anything
4
Ground Truth
Marigold 17
BetterDepth (Ours)
Figure A8: Qualitative comparisons on the NYUv2 dataset (28 part 1. Predictions are aligned
ground truth. For better visualization, color coding is consistent across all results, where red indicates the close plane and blue means the far plane.
21

---

Input Image
DPT
31
Depth Anything 49
Ground Truth
Marigold 1i
BetterDepth (Ours)
Input Image
Depth Anything 49
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold 17
BetterDepth (Ours)
Figure A9: Qualitative comparisons on the NYUv2 dataset (28) part Predictions are aligned
ground truth. For better visualization, color coding is consistent across all results, where red indicates the close plane and blue means the far plane.
22

---

Input Image
Depth Anything
Ground Truth
Marigold 1i
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 449
Ground Truth
Marigold Lir
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold
1ir
BetterDepth (Ours)
Figure A10: Qualitative comparisons on the KITTI dataset part 1. Predictions are aligned ground truth. For better visualization, color coding is consistent across all results, where red indicates the close plane and blue means the far plane.
23

---

Input Image
Depth Anything
Ground Truth
Marigold 1i
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything [49
Ground Truth
Marigold Dir
BetterDepth (Ours)
Input Image
DPT
Depth Anything 49
Ground Truth
Marigold
1ir
BetterDepth (Ours)
Figure All: Qualitative comparisons on the KITTI dataset part 2. Predictions are aligned ground truth. For better visualization, color coding is consistent across all results, where red indicates the close plane and blue means the far plane.

---

Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold Lir
BetterDepth (Ours)
Input Image
DPT
31
Depth Anything 49
Ground Truth
Marigold Lirl
BetterDepth (Ours)
Input Image
DPT B1
Depth Anything 44
Ground Truth
Marigold
1i
BetterDepth (Ours)
Figure A12: Qualitative comparisons on the ETH3D dataset [43), part 1. Predictions are aligned to ground truth. For better visualization, color coding is consistent across all results, where red indicates the close plane and blue means the far plane.
25

---

Input Image
DPT
Depth Anything 49
Ground Truth
Marigold Lir
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold Li7
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 44
Ground Truth
Marigold
17
BetterDepth (Ours)
Figure A13: Qualitative comparisons on the ETH3D dataset [43), part 2. Predictions are aligned to
ground truth. For better visualization, color coding is consistent across all results, where red indicates close plane and blue means the far plane.
26

---

Input Image
DPT
Depth Anything
40
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT
Depth Anything 44
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT
Depth Anything
49
Ground Truth
Marigold
Tir
BetterDepth (Ours)
Qualitative comparisons on the ScanNet dataset (6), part 1. Predictions are aligned to ground truth. For better visualization, color coding is consistent across all results, where red indicates the close plane and blue means the far plane.
Figure A14:
27

---

Input Image
DPT
Depth Anything
49
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT
Depth Anything 49
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT
Depth Anything 40
Ground Truth
Marigold
1v
BetterDepth (Ours)
Figure A15: Qualitative comparisons on the ScanNet dataset 66, part 2. Predictions are aligned to
ground truth. For better visualization, color coding is consistent across all results, where red indicates the close plane and blue means the far plane.
28

---

Input Image
DPT
Depth Anything
49
Ground Truth
Marigold 1i
BetterDepth (Ours)
Input Image
DPT
Depth Anything 49
Ground Truth
Marigold 1ir
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold 1iri
BetterDepth (Ours)
Figure A16: Qualitative comparisons on the DIODE dataset [45] part 1. Predictions are aligned to
ground truth. For better visualization, color coding is consistent across all results, where red indicates the close plane and blue means the far plane.
29

---

Input Image
Depth Anything
49
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold 17
BetterDepth (Ours)
Input Image
DPT 31
Depth Anything 49
Ground Truth
Marigold 177
BetterDepth (Ours)
Figure A17: Qualitative comparisons on the DIODE dataset [45]. part 2. Predictions are aligned to
ground truth. For better visualization, color coding is consistent across all results, where red indicates the close plane and blue means the far plane.
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect our contribution.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see the Limitation and Future Work section (Sec. ! in the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiscless settings, model well-speeti sell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, authors should discuss possible limitations their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't ack acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve integrity the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Question:
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
31

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core
paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please the Experiment and Analysis scction (Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing model checkpoint, or other mcans that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results way to reproduce the model (c.g.. with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the of closed- source models, may be that access the model is limited some way (e.g.. registered users), but should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, as described supplemental material?
the data and code, with sufficient instructions
Answer: [No]
Justification: This is research done in collaboration with corporate research lab and we haven't t been able to get clearance release the code. However, implementation is described in detail in the Experiments and Analysis section (Sec. 4 and data used for training is publicly accessible (Hyper- Virtual KITTI: https: //tropp naverlabs
sim: httpt:://///// / htt:://ggttttttt com vwswvvs/iiiiii proxy-virtuall virtual worl-oorssssso viiitiiiti---i Mokitti-2/11 2/). We addition
com/ ce o 2t22///mo/errre//////
ally provide the detailed training procedure in Algorithm to facilitate reproducing training models and experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines httpe:://iisss.cppupppee
quuiees/fooeeSttmiisssooooooood. for more details.
32

---

While we encourage the release of code and data, we understand that this might not be possible. "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines (https: httppss//nisscc/ppuutu cc..ppicc//bbli..
to reproduce
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release amonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Please the Experiments and Analysis section (Sec. 4 in the main paper and the Hyperparameter Analysis section (See. F the Appendix.
Guidelincs:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix. supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: Please the Error Bar Analysis section (Sec. G the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error hars, confidence intervals, statistical significance tests, least for the experiments that support the main claims
the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (c.g., Normally distributed errors).
should be clear whether error the standard deviation or standard error of the mean.
is OK to report I-sigma error hars, one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error hars that would yield results that are out of range (e.g. negative emror rates). error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Please the Experiments and Analysis section (experimental settings in Sec.
33

---

Guidelines:
answer means the paper does not include experiments.
paper should indicate the type compute workers CPU or GPU, internal cluster. or cloud
provider, including relevant memory and storage.
The paper should provide the amount compute required for each the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code
Ethics http://///ourrp...co/////////aaa
Answer: [Yes]
Justification:
The research conducted in the paper conforms with NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: Please see the Discussion Societal Impacts section (Sec. the Appendix.
Guidelines:
The answer means there is no societal impact of work performed.
the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address socictal impact.
Examples negative socictal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
34

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines restrictions to access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make hest faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected"'
Answer: [Yes]
Justification: This work properly credits and respects the assets used.
Guidelines:
The answer means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should he provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should be provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Please see the Method (Sec. and Experiments and Analysis section (Sec. Guidelines:
The answer means that paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub-
missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
35

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
36


Reviews:
[{"review_id": "vh0dY34pIn", "reviewer": "Reviewer_LNRC", "summary": "This paper presents a plug-and-play monocular depth estimator with the diffusion model. In the proposed method, the authors first employ the pretrained monocular depth model (MDE) to estimate a coarse depth map as the condition of the diffusion model. Then, a modified diffusion refiner is used to obtain the fine-grained depth map of the scene. Extensive experiments are conducted on benchmark datasets to demonstrate the effectiveness of the proposed method.", "questions": "Please refer to the weakness section. I hope to address the issuses in the rebuttal.", "limitations": "The authors discussed the limitations in the supplementary material.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is well-motivated and easy to follow.  \n2. The proposed method is a plug-and-play module and easy to use in different backbone models.", "weaknesses": "1. The novelty of the proposed method is somewhat limited. The proposed method is basically a conditional diffusion refiner, which combines the zero-shot MDE such as depth anything and diffusion-based MDE such as Marigold. In my opinion, the predicted depth map with depth anything can provide a “good” initialization to the diffusion model. Although the global pre-alignment and local patch masking modules are developed in the proposed method, I still think it is an incremental contribution. \n2. In the proposed local patch masking module, why the patch mask strategy can obtain more refined details of the depth map? How to determine the parameters $\\eta$ to control the mask ratio? In the ablation study of the supplementary material, the range of  the mask ratio is [0.05, 0.3]. Can we set the range to a big interval?\n3. In the experiment section, for the proposed method, different numbers of training samples are used. These samples are real data or synthetic data? For Marigold, the training samples are synthetic data without real data. Therefore, I suggest the authors to highlight the use of the real data or synthetic data. In addition, the authors could provide the reasons why the proposed method is slightly inferior to the existing methods on some datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. We kindly ask the Reviewer to read our **top-level global response** first. Our detailed responses to the comments in the weaknesses (denoted as W) section are listed below.\n> **W-1: Novelty**\n\nOne of our key contributions is the proposed training strategies that combine the merits of feed-forward and diffusion-based MDE methods in an efficient manner. We agree that building a conditional diffusion model to combine two components is straightforward, but **properly leveraging advantages of both** is challenging. Although the depth map estimated by Depth Anything provides good initialization, directly building a diffusion refiner on top of it without our training strategies (denoted as Naive Conditioning) yields inferior results as shown below (metrics are AbsRel $\\downarrow$ / $\\delta1 \\uparrow$).\n|Avg. Rank||Methods||NYUv2||KITTI||ETH3D||ScanNet||DIODE|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|1.8||Depth Anything||4.3/**98.0**||8.0/94.6||6.2/98.0||**4.3**/**98.1**||26.0/**75.9**|\n|2.7||Naive Conditioning||5.2/97.0||8.6/92.2||5.4/96.9||5.6/96.2||**22.4**/74.7|\n|**1.2**||**BetterDepth**||**4.2**/**98.0**||**7.5**/**95.2**||**4.7**/**98.1**||**4.3**/**98.1**||22.6/75.5|\n\nThis is because the naive conditioning overfits the small training datasets and thus underutilizes the prior knowledge in the pre-trained MDE model, resulting in degraded zero-shot performance (Lines 276-279 of the main paper). With the proposed training strategies, BetterDepth learns to utilize the geometric prior in pre-trained models for zero-shot transfer and the image prior in diffusion models for detail refinement, efficiently achieving robust performance with fine-grained details (as discussed in the `contribution` section of the global response).\n\n> **W-2: Local patch masking**\n\nTo achieve better details without disregarding the geometric prior learned in pre-trained MDE models, the key challenge is to ensure the depth conditioning strength and simultaneously enable learning detail refinement. Thus, we design the patch masking mechanism to:\n- improve depth conditioning strength at local regions. By excluding significantly different patches, BetterDepth learns to follow the depth conditioning and better utilizes the geometric prior for robust estimation, which is important for zero-shot performance as the comparison shown in Section W-1 (Naive Conditioning v.s. BetterDepth).\n- learn detail refinement within a reasonable range. With the filtered patches, BetterDepth learns detail refinement without overfitting the training data distribution, achieving better detail extraction while maintaining robust MDE performance. A visual example is provided in Fig. S1 of the attached PDF, where the model can learn to improve the detail of the basket (Fig. S1c) according to the depth label (Fig. S1b).\n\nHowever, higher conditioning strength limits the capacity for detail refinement, and lower conditioning strength results in degraded performance, e.g., the naive conditioning model in W-1. Therefore, the threshold $\\eta$ is used to balance these two contradicting properties. We conduct experiments with larger $\\eta$ (0.5 and 1) and combine the results in Fig. A2 as follows (metrics are AbsRel $\\downarrow$ / $\\delta1 \\uparrow$). Too large $\\eta$ often leads to worse performance as the geometric prior is not well utilized, and we choose the optimal $\\eta=0.1$ in BetterDepth.\n|$\\eta$||NYUv2||KITTI|\n|-|-|-|-|-|\n|0.05||4.21/**98.06**||7.70/95.03|\n|**0.1**||**4.18**/98.04||**7.47/95.22**|\n|0.3||4.43/97.89||7.75/94.83|\n|0.5||4.37/97.91||7.92/94.50|\n|1||4.50/97.72||8.14/94.20|*\n\n> **W-3: Training samples and method performance**\n\nAll the training samples are synthetic data in BetterDepth (Lines 249-253 of the main paper). For the two smaller training sets, we randomly select 400 and 2K samples from the full 74K synthetic dataset (composed of Hypersim and Virtual KITTI, which is the same synthetic dataset used in Marigold), and our experiments (Tab. 2) show promising performance of BetterDepth even with such small-scale training sets. \n\nThe effectiveness of quantitative evaluation heavily relies on the quality of depth labels in the benchmark. However, due to the limitation of depth sensors, the commonly adopted test benchmarks often contain incomplete and noisy depth labels. For example, Fig. A14-A15 illustrates significantly incorrect annotations and noise in the ground-truth of the DIODE dataset, and such label noise makes the reported metrics not fully reliable on a single dataset (similar discussions can be found in Tab. 2 and Sec. 6.1 of Depth Anything V2 [R4]). A more reliable evaluation is to compare the performance across diverse datasets, so we additionally provide the average ranking of the five benchmarks in Tab. 2 and our BetterDepth achieves the overall best performance. In addition, we further validate the superiority of BetterDepth on fine-grained detail extraction, as discussed in the `detail extraction` section of the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Duk52nyl7G", "reviewer": "Reviewer_Qdvk", "summary": "The paper proposes a simple approach to improve and refine current monocular depth estimation (MDE) methods. Leveraging the strong geometric prior from a state-of-the-art discriminative depth estimation method, and the strong image prior from a generative model, the authors set a new state-of-the-art in MDE. They condition a pre-trained latent diffusion model on an image and a corresponding depth map from a pre-trained MDE model and fine-tune the diffusion part to obtain higher-fidelity depth maps. Additional loss masking and alignment of affine-invariant depth to the ground truth are found to be crucial for performance.", "questions": "- Figure 1: I was wondering why the surface normals still show some wobbles for flat surfaces. Since the strong geometric prior from DepthAnything does not show these artefacts, it seems like the diffusion model inserts these. Do you have any explanation for that? Have you evaluated whether this stems from the first stage?\n\n- Line 196: Maybe I am missing something, but is there a reason why you choose max-pooling? As far as I understand that means that if at least one patch in a certain region has a small distance between ground truth depth and scaled and shifted depth, the full region is included for training. Do you have an estimate of how often a full region is rejected? And at which semantic regions in an image that usually is the case (e.g. only for the sky)?\n\n- Furthermore, since you exclude non-matching regions via the local patch masking, wouldnt it be better to directly use an outlier-aware method (e.g. RANSAC)? This would directly better align matching parts and non-matching parts can be more easily filtered out. \n\n- Since depth refinement alone is in my opinion only a moderate contribution, I was wondering whether you could extend your method to transfer affine-invariant depth to metric depth, since this might be a more challenging and thus more interesting task?\n\n- Line 305: Does this time include the MDE model? Maybe I was missing it, but can you give a clear separation of much time goes into which step and how many ensembles you are taking for this inference time measure?\n\n- If possible I would appreciate further clarification on appendix C. Is the model without any geometric prior just Marigold? And did you train the diffusion model without image prior completely from scratch?\n\n- Lastly, one of your claims is to capture more \"fine-grained scene details\" compared to other methods. I was wondering whether you know about some quantitative way to show that?", "limitations": "The authors have adequately addressed and discussed the limitation of their method.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well-organized and well-written, with a clear common thread. The authors present a comprehensive set of experiments, validating the effectiveness of their method, and clearly ablating their contributions. The idea is simple and achieves very good results on a broad range of depth estimation benchmark datasets with minimal additional training effort, utilizing strong image and depth foundation models.", "weaknesses": "- Since the idea is that simple, the general impact or contribution might only be moderate in my opinion. The main contribution is the injection of additional conditioning information into the diffusion model. Since this prior already gives state-of-the-art results, adding a full diffusion model on top with a strong image prior to learn the affine-invariant residual between ground truth and prediction is obvious to improve benchmark metrics. This also explains why there is only few data needed, and why the error bar in appendix D is much lower than Marigold. The task for Marigold is much harder compared to the refinement of a very good depth map.\n\n- Further contribution claims incorporate the global pre-alignment and local patch masking. The first, global pre-alignment, is a simple re-phrasing of scale and shifting the ground truth depth map to the predicted depth map of the MDE method, which is a common approach for testing affine-invariant depth estimation methods. Since the least squares fit cannot satisfy all pixels to match perfectly, discarding some portion of it, which the authors termed local patch-masking, is also just a small trick to further boost performance. Hence, these two contributions lack novelty in my opinion, and the only real novelty is the incoporation of a strong depth prior into the diffusion model.\n\nBesides the concern about contribution impact I think the idea is very clearly and nicely presented, and the paper very nicely polished.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. We kindly ask the Reviewer to read the **top-level global response** first. Our detailed responses to the comments in the weaknesses (denoted as W) and questions (denoted as Q) sections are listed below.\n> **W-1: Contribution**\n\nApart from the depth-conditioned diffusion refiner, a key contribution of BetterDepth is the proposed training strategies to achieve both robust performance and fine details. While Depth Anything gives state-of-the-art results, naively conditioning on it without our training strategies only yields inferior results as shown in Tab. T1 (Naive Conditioning v.s. BetterDepth) and discussed in the `contribution` section of the global response. In addition, the advantages of BetterDepth, e.g., lower error bar, also come from the proposed training strategies. We compare the standard deviation (std) based on the settings in Sec. D, and the results below show that the naive conditioning model is even more unstable than Marigold. This is because the injection of additional conditioning information makes it harder to determine which prior to follow, and the performance of BetterDepth further highlights the importance of our training methods.\n|Methods||AbsRel std $\\downarrow$||$\\delta1$ std $\\downarrow$|\n|-|-|-|-|-|\n|Marigold||0.66||0.99|\n|Naive Conditioning||0.81||1.06|\n|**BetterDepth**||**0.28**||**0.28**|\n> **W-2: Training strategies**\n\nTo achieve robust MDE performance with fine details, the key challenge is how to ensure conditioning strengths while enabling the learning of detail refinement. We agree that our training strategies are not difficult to implement, but the motivation to use them during training is more important.\n- To ensure **conditioning strength**, we propose to narrow the distance between depth conditioning and labels in a global-to-local manner. The pre-alignment first eliminates the global differences caused by unknown scale and shift, and the patch masking further addresses the local estimation bias in depth conditioning.\n- For **detail refinement**, global pre-alignment and local patch masking together contribute to fine-grained detail extraction. Although significantly different regions are excluded to ensure conditioning strength, the combination of pre-alignment and patch masking still enables the learning of detail refinement as shown in Fig. S1 of the attached PDF, e.g., the basket. \n\nThus, the proposed training strategies are critical for better MDE performance and fine-grained details, which is also supported by the results/analyses in W-1.\n> **Q-1: Wobbles**\n\nDiffusion-based MDE methods tend to introduce subtle variation due to the random noise in the diffusion process. This can be fixed by using the mean instead of the median in test-time ensembling (Fig. S2 in the attachment), which also achieves better performance as follows.\n|Method||NYUv2||KITTI||ETH3D||ScanNet||DIODE|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Median||**4.2**/98.0||7.5/95.2||4.7/**98.1**||**4.3/98.1**||22.6/**75.5**|\n|Mean||**4.2/98.1**||**7.4/95.3**||**4.6/98.1**||**4.3/98.1**||**22.5/75.5**|\n\n> **Q-2: Max-Pooling**\n\nMax-pooling is used to convert pixel-level masks to latent space. We follow Marigold to employ 8x8 max-pooling for mask downsampling as the VAE encoder in our diffusion model performs 8x downsampling for pixel-to-latent conversion. Since the patch size is also set to 8x8, max-pooling only happens within each patch without affecting a larger region. We will clarify this in revision.\n> **Q-3: Outlier-aware method**\n\nBetter alignment like outlier-aware methods could indeed have more patches survive for better performance. Although the data efficiency experiments (Tab. 2) show that fewer valid training patches lead to lower model performance, the models trained with small datasets, e.g., BetterDepth-2K, already achieve comparable results to our full model, indicating that limited patches can be sufficient. Nevertheless, better alignment methods could improve patch preservation to further benefit efficient training.\n> **Q-4: Metric depth**\n\nTransferring affine-invariant depth to metric depth is a promising direction to benefit practical use but poses challenges, e.g., scale/shift ambiguity and diverse depth ranges. Nonetheless, BetterDepth shows potential to boost metric depth estimation. We employ the metric Depth Anything model and apply BetterDepth in a plug-and-play manner on zero-shot datasets iBims-1 [R6] and SUN RGB-D [R7]. Due to the unknown scale and shift in the current BetterDepth, we align the outputs of Depth Anything and BetterDepth to depth labels and compare the quality of depth maps. The table below (metrics are AbsRel $\\downarrow$ / $\\delta1 \\uparrow$) and Fig. S3 of the attachment verify the superiority of BetterDepth. One promising next step is to model the scale/shift in an implicit learnable manner (as suggested by Reviewer EiFe), and we will further study this in future works.\n|Method||iBims-1||SUN RGB-D|\n|-|-|-|-|-|\n|Depth Anything||5.1/97.7||13.5/87.7|\n|BetterDepth||**4.5/98.2**||**12.7/88.8**|\n> **Q-5: Inference**\n\nThe inference time includes the pre-trained MDE model, i.e., Depth Anything. To measure the time spent at each step, we reproduce the experiment, and the inference time of the pre-trained model and diffusion model are 0.02 and 0.38 seconds per sample, respectively. The ensemble size is set to 1.\n> **Q-6: Appendix C**\n\nThe model without geometric prior uses the same network and fine-tuning method as Marigold but estimates inverse depth (following Depth Anything) instead of relative depth. For the model without image prior, we follow Stable Diffusion to train the latent UNet from scratch and keep the pre-trained VAE unchanged.\n> **Q-7: Detail evaluation**\n\nThanks for the suggestions. We conduct quantitative evaluation for detail extraction in the `detail evaluation` section of the global response, and the results in Tab. T3 demonstrates the state-of-the-art performance of BetterDepth in detail extraction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "The46MTbNX", "reviewer": "Reviewer_EiFe", "summary": "This paper proposes a plug-and-play diffusion refiner for pre-trained zero-shot feed-forward MDE (monocular depth estimation) models, so that these generalized models can capture fine-grained details. In this paper, the coarse depths output from a pre-trained feed-forward MDE model are used as additional conditions for the proposed diffusion refiner, which is parameterized into any existing diffusion-based MDE model. In addition, a global pre-alignment strategy and a local patch masking strategy are proposed to ensure the faithfulness of the predicted depths.", "questions": "(1) As for the global pre-alignment, since the estimated depth values are derived from their corresponding depth labels, what is the performance of only using them in the local patch masking? And for the local patch masking, the authors simply discard significantly different regions in the estimated depths and real depths, will this lead to the loss of visual world information?\n\n(2) In lines 128-132, the author's conclusion is that large-scale datasets lead to the generalizability of the feed-forward MDE methods. Therefore, in order to improve the  generalizability of diffusion-based MDE methods, why not to simply train them with large-scale datasets following the feed-forward MDE methods? Why is this still a challenge?\n\n(3) In lines 133-137, the author's conclusion is that in the diffusion-based MDE methods, the high-quality labels in the synthetic datasets lead to the ability to capture fine-grained details, but some feed-forward MDE methods also use these synthetic (and real) datasets for training as indicated in Table 1. Why are these feed-forward MDE methods unable to capture fine-grained details?\n\n(4) Is Fig. 3 a schematic diagram or a direct visualization of the output distribution? What is the difference between the output distributions X(M_DM,D_syn) and \\hat{X}? Moreover, for the middle sketch, why does the global pre-alignment not only affect \\hat{X}  but also  X(M_DM,D_syn)? \n\n(5) As indicated in Fig. A2, lower η generally means stricter filtering, but if η is too small, too many local regions would be discarded, which would lead to severe information loss. To this end, I’m wondering how many local regions are discarded in the current version of BetterDepth, and how to balance the strictness of the filter and the loss of information. \n\n(6) Is there any diffusion-based MDE method trained on real datasets?", "limitations": "The proposed method might be improved in two ways: (i) The offsets from the real depths to the estimated depths could be modeled in an implicit learnable manner; (ii) The pairs of significantly different local regions in two depths may not be harmful, but contain useful information for monocular depth estimation.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) The paper is well-organized.\n\n(2) The proposed diffusion refiner is simple and clear, which could be be easily combined with different existing feed-forward MDE methods. \n\n(3) Experimental results on several datasets demonstrate the effectiveness of the proposed method in some cases.", "weaknesses": "(1) The motivation of this work is a bit far-fetched: As stated in the introduction, the low accuracy of the existing feed-forward MDE methods is caused by noisy and incomplete depth labels collected in real-world scenarios, while the poor generalizability of the diffusion-based MDE methods is caused by less diverse synthetic training samples. However, if only training data results in limited MDE performance for these existing methods, it seems that a straightforward motivation for alleviating the above problems is to enrich the training datasets, rather than to modify the model architecture as done in this work.\n\n(2) Some statements in Table 1 are confusing: (i) To the reviewer’s knowledge, many existing feed-forward methods only use real datasets for training, rather than using real and synthetic datasets together. Why do the authors indicate that feed-forward methods use both real and synthetic datasets for training? (ii) Some existing diffusion-based methods (e.g., \"Monocular depth estimation using diffusion models.arXiv preprint arXiv:2302.14816, 2023.\") also use real datasets for training. Why do the authors indicate that diffusion-based  methods use only synthetic datasets for training in Table 1? (iii) The authors state that feed-forward methods have a better generalizability than diffusion-based methods. But to the reviewer’s knowledge, in many zero-shot learning tasks (e.g., zero-shot recognition), generative models (e.g., GANs and diffusion models) generally show their better generalizability than the feed-forward counterparts. Could the authors give some explanations on why to make the above statement?\n\n(3) Comparaive evaluation: As seen from Tables 2 and 3, when the proposed BetterDepth is used together with two early methods MiDaS (published in 2020) and DPT (published in 2021), it could bring some improvements. However, when the proposed BetterDepth is used together with Depth Anything[43] (a SOTA method), it only brings a slight improvement in most of the used datasets (particularly NYUv2 and ScanNet). So it seems that the proposed BetterDepth has a limited effect for boosting the performance of SOTA methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thoughtful comments. We kindly ask the Reviewer to read the **top-level global response** first. Our detailed responses to the comments in the weaknesses (denoted as W) and questions (denoted as Q) sections are listed below.\n\n> **W-1: Motivation**\n\nBoth the training data and the model architecture are important for performance. The very recent method Depth Anything V2 [R4] uses synthetic data for better details. Although promising improvements are achieved in Depth Anything V2, BetterDepth still shows better performance as discussed in the `detail evaluation` section of global response, thanks to the iterative refinement of diffusion models.\n\nEnriching the training dataset helps to boost MDE performance, but (i) obtaining high-quality labels for real datasets is difficult due to depth sensor limitations, (ii) synthetic datasets offer high-quality labels but are costly to generate at scale, and (iii) training on large datasets is both time-consuming and resource-intensive.\n\nBetterDepth efficiently combines the strengths of feed-forward and diffusion-based MDE models, achieving robust performance with fine-grained details with minimal training effort.\n> **W-2: Table 1**\n\n(i) Feed-forward models can easily use large-scale datasets (both synthetic and real ones) to gain robust performance, e.g., Depth Anything V2 [R4], so we include both in Tab. 1 for generality. \n(ii) Recent works [9,11,14] validate the effectiveness of capturing fine details by training diffusion models on synthetic datasets. While techniques like depth infilling [34] enable training on real datasets, the resulting depth maps tend to be smoother due to the sparse and noisy labels. Thus, we focus on the recent diffusion-based MDE methods with synthetic data. We will explicitly indicate this to avoid confusion. \n(iii) Diffusion-based models could generalize better in tasks with diverse, accurately annotated datasets, e.g., recognition, but the sparse/noisy labels of real datasets and the limited diversity of synthetic datasets in the MDE task make it challenging. Tab. 2 also supports the significant performance gap between the state-of-the-art diffusion and feed-forward MDE approaches, e.g., Marigold v.s. Depth Anything. \n> **W-3: SOTA improvements**\n\nBetterDepth aims to achieve robust MDE performance with fine-grained details. Despite achieving state-of-the-art performance, Tab. 2 cannot show the full advantages of BetterDepth (especially detail extraction) due to the sparse and noisy depth labels (Fig. A6-A15), which is also observed in [R4]. Thus, we provide the `detail evaluation` experiment in the global response, and the results in Tab. T3 verifies our significant improvements over the state-of-the-art model.\n> **Q-1: Training strategies**\n\nTo test patch masking with only depth labels, we use a smoothed version of depth label as conditioning to imitate the estimation of pre-trained MDE models. As shown in the table below, patch masking with only depth labels (denoted as Label Only) shows inferior performance due to the significant distribution gap of depth conditioning in the training and inference stages.\n\n|Method||NYUv2||KITTI||ETH3D||ScanNet||DIODE|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Label Only||4.4/**98.0**||8.0/94.0||7.8/97.8||4.4/98.0||23.0/75.4|\n|**BetterDepth**||**4.2**/**98.0**||**7.5**/**95.2**||**4.7**/**98.1**||**4.3**/**98.1**||**22.6/75.5**|\n\nFor patch masking, discarding patches will indeed result in loss of visual information. However, since BetterDepth utilizes the rich geometric prior from the pre-trained model, training on limited visual information already yields promising results, e.g., BetterDepth-2K is comparable with our full model in Tab. 2.\n> **Q-2: Large datasets**\n\nBetterDepth aims to achieve both robust performance and fine details. Although large-scale real datasets can be employed to gain better generalizability, the sparse and noisy labels hinder models from extracting fine details. Synthetic datasets provide high-quality labels for fine detail extraction, but their low diversity limits the learned geometric priors. Similar discussions can be also found in Sec. 2-3 of [R4]. \n\n> **Q-3: Fine details**\n\nTraining with synthetic datasets could help improve detail extraction, but the model architecture is also important. The recent Depth Anything V2 [R4] employs synthetic training data for better details, and we perform comparison in the `detail evaluation` section of the global response. Thanks to the iterative refinement scheme, BetterDepth shows the best performance on detail extraction (Tab. T3).\n> **Q-4: Fig. 3**\n\nFig. 3 is a schematic diagram where X_(M_FFD, {D_syn, D_real}) and X_(M_DM, D_syn) are fixed distributions representing the characteristics of different methods (Tab. 1). By contrast, \\hat{X} indicates the learned output distribution of BetterDepth and we mainly analyze its change under different training strategies. Thus, only the \\hat{X} is affected by global pre-alignment in the middle sketch. We will clarify this to avoid confusion. \n> **Q-5: Filtering**\n\nThe percentage of discarded patches on the training dataset is 36.6% in BetterDepth. Although a small $\\eta$ will lead to fewer valid patches, BetterDepth works well with small-scale training datasets even under the information loss (as discussed in Q-1). We empirically find the optimal value $\\eta=0.1$ to achieve the best performance balance (Fig. A2).\n> **Q-6: Diffusion MDE + real data**\n\nBoth [34] and [R5] employ depth infilling techniques to train diffusion-based MDE models on real datasets and achieve promising results. However, these works primarily show the feasibility of applying diffusion models on MDE without exploring fine detail extraction like recent methods, e.g., Marigold. Besides, they focus on in-domain testing instead of zero-shot evaluation. We will add [R5] to the related work section.\n\n> **Limitation**\n\nThank you. We will explore metric depth and improve information utilization in future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers and area chairs for their valuable time and comments. We will incorporate all suggestions to improve the revised paper. After providing more results/analyses, we would like to give an overall response and re-emphasize the contribution and performance of BetterDepth.
> **Contribution**

BetterDepth aims to efficiently combine the beneficial characteristics of feed-forward and diffusion-based monocular depth estimation (MDE) methods to achieve robust MDE performance with fine-grained details. Although it might seem straightforward to gain better performance by combining two models, naive conditioning without our proposed training strategies only results in inferior performance as shown in Tab. T1 below. Even having good depth maps from the pre-trained model as initialization, the naive conditioning model struggles to balance the contribution of different priors and does not yield an improvement. Thus, to efficiently achieve our goal, challenges still exist:
- **Performance Trade-off.** One solution to better utilize the initial depth map is to improve the conditioning strength. However, stronger depth conditioning limits the capacity for detail refinement, and lower conditioning strength results in degraded performance, e.g., naive conditioning in Tab. T1. Therefore, how to properly utilize the merits of different priors and balance the performance trade-off is non-trivial.
- **Resource Efficiency.** It might be possible to achieve improvements by training on diverse large-scale datasets with high-quality labels, but (i) obtaining high-quality labels for real datasets is difficult due to the imperfection of depth sensors, (ii) synthetic datasets offer high-quality labels but are costly to generate at scale, and (iii) training on large datasets is both time-consuming and resource-intensive.

To this end, we propose global pre-alignment and local patch masking to balance the performance trade-off, ensuring conditioning strength while enabling detail refinement. As a result, BetterDepth efficiently combines the strengths of feed-forward and diffusion-based MDE models, achieving robust performance and fine-grained details with minimal training effort.

**Table T1.**  Comparisons with the naive conditioning model (AbsRel / $\delta1$).
|Avg. Rank||Methods||NYUv2||KITTI||ETH3D||ScanNet||DIODE|
|-|-|-|-|-|-|-|-|-|-|-|-|-|
|1.8||Depth Anything||4.3/**98.0**||8.0/94.6||6.2/98.0||**4.3**/**98.1**||26.0/**75.9**|
|2.7||Naive Conditioning||5.2/97.0||8.6/92.2||5.4/96.9||5.6/96.2||**22.4**/74.7|
|**1.2**||**BetterDepth**||**4.2**/**98.0**||**7.5**/**95.2**||**4.7**/**98.1**||**4.3**/**98.1**||22.6/75.5|
> **Zero-Shot Performance**

With the proposed architecture and training strategies, BetterDepth achieves state-of-the-art performance on the widely adopted zero-shot datasets as shown in Tab. T2 below, where the very recent approach Depth Anything V2 [R4] is also included for comparisons. 

**Table T2.**  Zero-Shot MDE performance (AbsRel / $\delta1$).
|Avg. Rank||Methods||NYUv2||KITTI||ETH3D||ScanNet||DIODE|
|-|-|-|-|-|-|-|-|-|-|-|-|-|
|3.7||Marigold||5.5/96.4||9.9/91.6||6.5/96.0||6.4/95.1||30.8/**77.3**|
|1.9||Depth Anything||4.3/**98.0**||8.0/94.6||6.2/98.0||4.3/**98.1**||26.0/75.9|
|2.6||Depth Anything V2||4.4/97.8||8.3/93.9||6.2/**98.2**||**4.2**/97.8||26.4/75.4|
|**1.4**||**BetterDepth**||**4.2**/**98.0**||**7.5**/**95.2**||**4.7**/98.1||4.3/**98.1**||**22.6**/75.5|

> **Detail Evaluation** 

Following the suggestions of Reviewer Qdvk, we further provide quantitative evaluation of fine-grained detail extraction. Since depth labels in the commonly adopted benchmarks are generally sparse or noisy as discussed in [R4] and shown in Fig. A6-A15, which makes them less reliable for detail evaluation, we conduct experiments on a high-resolution RGB-D dataset Middlebury 2014 [R1]. Four additional edge-based metrics are employed to exclude the influence of non-edge regions: the completeness and accuracy of depth boundary errors (denoted as `DBE_comp` and `DBE_acc`) [R2] and the edge precision and edge recall (denoted as `EP` and `ER`) [R3]. As shown in the Tab. T3 below, although the recent Depth Anything V2 achieves promising improvements in detail extraction by leveraging high-quality synthetic training data, our BetterDepth still exhibits better performance even with much less synthetic training data (595K in Depth Anything V2 v.s. 74K in BetterDepth), thanks to the iterative refinement of diffusion model. In addition, BetterDepth also captures better details like the cat's hair in Fig. S5 of the attached PDF, validating its overall best performance.

**Table T3.**  Evaluation of detail extraction.
|Avg. Rank||Method||AbsRel $\downarrow$||$\delta1$ $\uparrow$||DBE_comp $\downarrow$||DBE_acc $\downarrow$||EP (%) $\uparrow$||ER (%) $\uparrow$|
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
|3.7||Marigold||7.57||93.24||5.60||3.09||16.65||23.75|
|3.2||Depth Anything||3.14||99.44||6.35||2.66||24.73||16.12|
|2.2||Depth Anything V2||3.06||99.38||4.19||2.23||26.74||35.89|
|**1**||**BetterDepth**||**2.95**||**99.52**||**3.61**||**2.09**||**28.49**||**50.35**|

We hope we have addressed your concerns and kindly ask you to consider updating your rating based on our additional explanations and evaluations. Please don’t hesitate to let us know of any additional comments or questions.

**[Reference]**

[R1] High-Resolution Stereo Datasets with Subpixel-Accurate Ground Truth. GCPR 2014.

[R2] Evaluation of CNN-based Single-Image Depth Estimation Methods. ECCVW 2018.

[R3] Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundaries. WACV 2019.

[R4] Depth Anything V2. arXiv preprint arXiv:2406.09414, 2024.

[R5] Monocular depth estimation using diffusion models. arXiv preprint arXiv:2302.14816, 2023.

[R6] Evaluation of CNN-based Single-Image Depth Estimation Methods. ECCVW 2018.

[R7] SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite. CVPR 2015.

Meta-review: The paper received 2 borderline rejects and 1 weak accept. Reviewers approved of the simplicity of the method and the strong quantitative performance. Reviewers questioned the problem setting (is depth refinement or improvement a big enough task?), motivation of the method, and technical novelty.

After reading all the reviews, rebuttal, and the paper myself, I recommend a borderline Accept as Poster. I share Reviewer EiFe's view that the correct, principled approach to the problem would be to directly deal with the data issue or develop new training strategies; however, with that said, I also agree with the authors that there is value in identifying and clearly stating the problem as well as exploring immediate fixes. Given that MDE is a very crucial task that affects a large number of vision subfields (3d, generative, etc.), there's value in having better depth numbers right now, even at the cost of what is essentially an extra network in a diffusion cascade. The quantitative results, particularly the added evaluation in the rebuttal period, are convincing enough to me to suggest that there is value in the work as is.

I would strongly encourage the author's to integrate the rebuttal information into their paper and also to very seriously consider the discussion points mentioned by Reviewer EiFe.
Decision: Accept (poster)