Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues
Authors: Chinmay Talegaonkar
Nikhil Gandudi Suresh
Zachary Novack
Yash Belhe
Priyanka Nagasamudra
Nicholas Antipa
Venue: NeurIPS 2025 spotlight ()
Keywords: Monocular Depth
Defocus Blur
Diffusion Models
Zero Shot Depth
Inference Time Optimization
Primary area: applications
TL;DR: Zero Shot Metric depth from a pre-trained monocular depth diffusion model by incorporating optical defocus cues at Inference time. Outperforms some of the existing Zero Shot Metric depth baselines, while also

Abstract: Recent monocular metric depth estimation (MMDE) methods have made notable progress towards zero-shot generalization. However, they still exhibit a significant performance drop on out-of-distribution datasets. We address this limitation by injecting defocus blur cues at inference time into Marigold, a \textit{pre-trained} diffusion model for zero-shot, scale-invariant monocular depth estimation (MDE). Our method effectively turns Marigold into a metric depth predictor in a training-free manner. To incorporate defocus cues, we capture two images with a small and a large aperture from the same viewpoint. To recover metric depth, we then optimize the metric depth scaling parameters and the noise latents of Marigold at inference time using gradients from a loss function based on the defocus-blur image formation model. We compare our method against existing state-of-the-art zero-shot MMDE methods on a self-collected real dataset, showing quantitative and qualitative improvements.

Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues
Chinmay Talegaonkar, Nikhil Gandudi Suresh, Zachary Novack, Yash Belhe, Priyanka Nagasamudra, Nicholas Antipa University of California San Diego
Abstract
Recent monocular metric depth estimation (MMDE) methods have made no- table progress towards zero-shot generalization. However, they still exhibit a significant performance drop limitation by injecting defocus blur cues inference time into Marigold, a pre- trained diffusion model for zero-shot, scale-invariant monocular depth estimation (MDE). Our method effectively turns Marigold into a metric depth predictor in a training-fiee free manner. To incorporate defocus cues, we capture two images with a small and a large aperture from the same viewpoint. To recover metric depth, then optimize the metric depth scaling parameters and the noise latents of Marigold at inference time using gradients from a loss function based on the defocus-blut image formation model. We compare method against existing state-of-the-art zero-shot MMDE methods on a self-collected real dataset, showing quantitative and qualitative improvements. Our implementation is available https: //gi //ttttb...t..///.///g com/
datasets. We address this
1 Introduction
Estimating metric depth from single camera viewpoint is a central problem in computer vision with numerous downstream applications, including 3D reconstruction (24), autonomous driving [50], and endoscopy [31. This task, known as monocular metric depth estimation (MMDE), is fundamentally ill-posed due to inherent depth-scale ambiguity [46]. Multi-view methods (65) avoid this ambiguity but are often expensive and impractical in settings like endoscopy microscopy. Training data-driven MMDE methods is challenging, as it requires accounting for a diverse set of camera parameters and metric depth scales. As result, existing MMDE models struggle in acro-shot settings, i.c., they generalize poorly to unseen datasets. Recent advances in zero-shot MMDE 775 have demonstrated improved generalization, but there is still considerable performance drop unseen datasets.
In contrast to MMDE, monocular relative depth estimation (MDE) methods recover relative depth map, factoring out the physical depth scale. This enables using large-scale datasets with diverse depth ranges [41 for training data-driven MDE methods. As result, MDE methods achieve better zero-shot generalization at significantly lower training cost than MMDE methods, as shown by recent results 77 71 26). However, despite favorable performance on benchmarks, the absence of metric scale in MDE outputs precludes their applicability in downstream tasks requiring absolute depth.
Existing data-driven MMDE methods commonly suffer from two failure modes: undesirable coupling between image texture and depth predictions (fig.3). and inaccurate estimation of the scene's physical scale (see fig. 4). The first issue of texture coupling also affects MDE methods (see fig. 3 unless
explicitly mitigated through complex training procedures that account for texture variation 22 55
* Corresponding author: ctal ctalegaonkar@ucad
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Comera Parometers (.F.N)
Blumed Image
((0.F)
%
dth i
Delocus (6L.N.
*t|ls,
%,Il
Marigeid
Monocular Depth
Depth
Loss
(.F.NN)
(222
Image
Method overview. We capture two images (same viewpoint) from a camera with focal length f and focused at a distance F: an all-in-focus (AIF) image x (F-stop: N = 22) and blurred image X; (F-stop: N < 22). Using the AIF x and an initial learnable noise vector Z Marigold predicts the relative depth d. We then affine transform d with learnable parameters (a, 8), obtaining the metric depth d Given the AIF depth dm and camera parameters (ʃ, F, N), we synthesize a blurred image &n using the defocus blur forward model. To update the learnable parameters, we compute their gradients w.r.t the 1.2 loss between &, and Xb-
Figure I:
(d)
We demonstrate that incorporating camera physics, particularly defocus blur, with a data-driven MDE model at inference time can effectively address both failure modes without any re-training. We use Marigold [26], a diffusion iuffusion-bbsed MDE model, as it enables backpropagation-bast optimization using defocus cues, while staying its learned manifold of plausible depth maps. To provide these defocus blur cucs, we capture two images from a single viewpoint: a small-aperture all-in-focus (AIF) image, which is given input to Marigold, and large-aperture image that provides physical cues for metric depth. Our approach requires only a variable aperture camera, such as a DSLR, and knowledge of the lens focal length, focus distance, and F-stop, which are readily available from image metadata; this avoids the need for extrinsic calibration required by multiview setups.
inference time
Defocus blur is known to provide coarse metric depth cues in monocular settings 33 59 69), but has rarely been explored alongside recent data-driven advances in MMDE. We show in our work that revisiting this classical cue and integrating it with modern relative depth methods like Marigold improves metric depth estimation performance, without any retraining. While most depth from defocus methods multiple measurements (2D 30 63], we require only two. Prior work has also explored other hardware novelties to encode depth cues, such as dual pixel cameras 19] 68 and coded apertures [28] However, these depth
coarse, and training MMDE models for these systems is limited by the lack of datasets captured with a sufficient camera parameter diversity. This motivates the need for approach that combines strong data-driven pre-trained MDE/MMDE models trained on existing RGB datasets with physics-based cues from non-pinhole cameras.
To this end, our contributions are as follows: We formulate MMDE an inverse problem under the defocus blur image formation model. Using inference-ting optimization, we recover the metric depth scale and correct for texture-depth coupling without retraining Marigold. We devise hardware setup consisting of a rigidly coupled DSLR (RGB camera) and an Intel RealSense depth camera to capture RGB images and the ground truth metric depth. We collect a dataset of 7 diverse real-world indoor scenes, captured at different defocus blur levels. We compare our method on this dataset with current MMDE methods and demonstrate quantitative improvements real data.
2 Related Work
Zero-shot depth models Recent approaches for Monocular (relative) Depth Estimation (MDE) and Monocular Metric Depth Estimation (MMDE) can be classified as discriminative or generative. Discriminative approaches largely rely on vision transformer-ban trained on large-scale datasets. Transformer-based methods have been more successful for MDE 70 71 compared MMDE accuracy-wise, as MMDE a more ill-posed task than MDE. These approaches have very low inference times, but for MMDE, the performance degrades on out-of- distribution test scenes] Most of these methods are purely data-driven and ignore depth- based visual effects in images such as defocus cues. It is also difficult incorporate physics- based depth refinement in these methods at test time without any re- re-training
architectures
74 39
[73]. Since we use a generative MDE, approach can incorporate physical cues at test time in rrining-frre manner.
Refer to 75 for a detailed survey MMDE methods.

---

Current state-of-the-ar gemerative MDE/MMDE methods 47 13] are predominantly diffusion- based.
Several previous methods [T3] 47 32 incorporate diffusion-based depth denoising in their pipelines, achieving highly detailed depth maps - but are not zero-shot. [48] achieves zero-shot MMDE with a
diffusion-basedd approach by incorporating diverse field of view (FOV) augmentations in training, but it is not open source and lags behind cansformer-baset methods in performance. Marigold [26] is trained by fine-tuning Stable Diffusion-v2 on synthetic depth data. It achieves high-quality zero-shot MDE and supports test time refinement, but is not applicable natively for MMDE. Our approach uses defocus cues to refine the relative depth predictions from Marigold (or similar methods [T6), enabling its application to MMDE. Prior work [60] also proposes similar strategy for dense MMDE from a sparse metric depth map using Marigold and test time optimization. In contrast, our method does not require a sparse depth map imput, and solely relies on RGB images and a priori known scene bounds. Using defocus blur cues, our method resolves inaccuracies in monocular depth while also estimating the global scaling parameters for metric depth.
Diffusion model priors for inverse problems We frame MMDE as an inverse problem under the defocus blur image formation model. This framing closely relates to the recent work on solving linear inverse problems using pre- rre-trained diffusion models [22. 9 I 54 [0] incorporate the forward model constraints while sampling pixel-space diffusion models pre-trained on smaller datasets. As result, these methods require many steps while sampling the diffusion model and have limited generalizability as priors. [53 44 use latent variable diffusion models (LDMs) as priors, resulting in better in-the-wild generalizability. We use Marigold the LDM, but instead of incorporating the defocus forward model during sampling, we optimize the latent noise vector based on the error between the observed and predicted image using the forward model. Our approach is inspired by 61 17 which uses noise optimization in conjunction with a differentiable auxiliary guidance loss improve the sampling quality of the diffusion model based on text input. While these methods use trained models as differentiable proxies for guidance, we use physics-based based imaging forward model. (34) also uses a physics-basee baseed forward model with a diffusion prior, but requires re- re-training the diffusion prior from scratch.
recent methods (37 36 77
Depth estimation from passive camera physics cues A substantial body of research on MMDE (DfD),
leverages camera physics, including methods like depth-frrm--effone
164 20
58 2].
phase/aperture masks 28 78, 76 3. 67). and dual pixel sensors [8] 68 T Classical approaches
produce very coarse depth maps. Most DíD methods need a well-aligned (66) multi-image focal stack
(2 (7), to achieve good depth quality. Optical mask-based methods pose a harder inverse problem of jointly estimating both AIF and the depth map. We capture only 2 images at the same focus distance and different apertures, requiring no image alignment and AIF estimation. This simplifies depth refinement with minimal added capture time. Previous work has also used variable apertures for classical [15] and learning- based [56] depth estimation methods, but it is not zero-shot. While learning-basee based approaches [19 8
66 g help improve the depth map quality for these methods, they scenes. Dual pixel-based methods are popular for phone cameras, [18 [8), but are tied the specific camera architecture. While we show results with a standard DSLR sensor, our approach can be adapted to dual- pixel--eel camera architectures as well.
don't ton'tgte generalize well to out-of-distributin
3 Preliminaries
Diffusion-based monocular depth estimation Our approach is built on top of Marigold (26). which is a monocular depth estimator trained by fine-tuning the denoising U-Net of StableDiffusion v2 (SDv2) [43] on synthetic depth data. Marigold allows sampling the conditional distribution of the monocular depth given an input image, p(do x). In particular, Marigold attempts to generate clean monocular depth maps, do given clean input image x by first sampling dr (0, I) from an i.i.d Gaussian distribution, and then iteratively denoising it (where each intermediate step is denoted by d,) according to a fixed noise schedule with parameters Or,, dt As SDv2 is a latent-diffusion model, the entire generative process happens on encoded latent depth maps 2 with latent images xxx) as conditioning. Training M (which we denote as (-)) involves using the standard denoising loss common in image diffusion works, but on depth maps rather than images, and passes the image
Whiie technically SDv2 is trained in e-prediction mode rather than x-prediction, we use x x-prediction given their mathematical equivalence and alignment with Marigold-LCM, which uses x-prediction.
3

---

Depth
Real GT
Camera
Source
Disc
F
Gaussian
RGB
Camera
(a)
(b)
(c)
Figure 2: Comparing simulated PSFs with the PSF captured from our camera setup. (a) A point source placed d distance away from thin lens focused at a focus distance produces a blurred image (PSF) with diameter c, also known as the circle of confusion. The variation of с with source distance is shown in the plot. (b) The Dise approximation the camera PSF lies roughly within the same bounds (dotted red circle) as the PSF captured from the RGB camera (Real GT) in (c). The Gaussian PSF significantly exceeds the bounds. Slight differences between the real and Disc PSF stem from the octagonal aperture and diffraction ignored in our model. (c) We rigidly mount an Intel RealSense on DSLR to capture ground truth depth, and calibrate both cameras to align predicted depth from the RGB image with the ground truth depth for evaluation.
input into the model as well through channel wise concatenation:
E 40 2 (x))) (ttp,,D,,,_p(((,.>oov110.. [w (t)llz)))
(a (d) o,c, ())))
(I)
where wt) is a time-dependent weighting function. At inference time, x is first mapped to a lower dimensional latent vector xxx) = {() through VAE encoder E in SDv2. The inference process starts with sampling noisy latent depth vector z.r'
(d] NO I), which is iteratively refined by applying the (d) is then decoded through the SD decoder D to produce the output depth map D(z)) (())))). Marigold assumes both (xx) .z (dd) e R
denoiser (z, (d) ,z(x), t) to obtain z)) over T sampling steps. Zo
For faster inference, we use the latent consistency model version of Marigold, Marigold-LCM, and obtain (()) with single inference step from
z!(d) Monocular depth d can be mapped metric depth
d! by an affine transform; more details in section
Modeling defocus cues Under the thin lens camera assumption, defocus blur manifests as uniform blur kernel, with a depth-dependent diameter. The blur kernel diameter for a point source placed at a distance d away from the camera is determined by the circle of confusion (CoC) [40] equation
c(d) f² F ddFF-FF
(2)
where the focal length f. focus distance F, and F-stop N are camera parameters that are known from the image EXIF data. s denotes the pixel size in physical units (m). The CoC defines depth-dependent point spread function (PSF) h(i,j u, U, d) that predicts the response at pixel coordinate (i,j) from a point source lateral coordinate (u, v) and depth under defocus blur. Note that (u, v) can represent depth- normalioore pixel coordinates under an ideal pinhole projection, allowing us to write d d [u, u] (valid for general non-vollo -ollmmlumttiic scenes that assume a single depth value per coordinate). We assume that the PSF is shift-
for a given depth d, i.c. h(t,j u, v, d) = h(i J - v 0, [u, e)) h(i - u. --j-u d [u, v)), which is simply the on-axis PSF, modeled as disc (assuming circular aperture) with radius given by CoC equations, translated to be centered at ji, To ensure smooth optimization, we include linear fall-off the boundary of the discontinuous disc kernel, similar to [62. The PSF can then be expressed as
W (i,j u,
Miij u. v.
where Σn W(i.j u,
(3)
1
<(d)-1
W(ij u.
c(d)+1
m,
c(d)
< m <
c(d)+1 2
(4)
<dd)+1 2
m
m v (i u)² + (j v)2.
(5)

---

Marigold-LCMM, which takes in as input the AIF We assume the PSF to be normalized and explicitly account for exposure and energy balancing during image capture and processing (see section While the above PSF can also be approximated as an isotropic 2D Gaussian [20]. we opt for the disc parameterization in eq. (5) similar to [62 [5]] as it better approximates the PSF of the real camera compared to the Gaussian approximation
200 as shown in fig. _ We can see from eq. (2) that an image captured with a very small aperture (high N) would have negligible defocus blur due to very small CoC values. Such an image is referred to as the all-in-focus (AIF) image, Given the AIF the blurred image Xs can be approximated (neglecting occlusion) as a spatially varying convolution between / and x,
lf x(u, v) h(i - u, u; dd vl)dudy
Xb (i.
(6)
For simplicity, we denote the above image formation forward model
g(, d . F,
(7)
The AIF image x, captured high F-stop, serves the blur-free input for both Marigold and the camera blur model eq. 7 while the low F-stop image Xs provides defocus cues for MMDE.
Inference-time optimization Without loss of generality, any generative model (GAN, Diffusion (ow-basedd Φ IR + IN can be construed as a mechanism to map simple probability distribution, such an i.i.d. Gaussian distribution, E (0, I) RM non-linear N-dimensional manifold, such as images or audio, through a differentiable generative process x Φ(E). Inference time optimization 37 refers to manipulating the generation process by updating the initial noise based on gradients from a differentiable loss function (xx) on the generated sample
E 6 V.L(z).
(8)
To further ensure that e still lies close to the Gaussian manifold after the gradient updates, can be rescaled to have L2 norm of VM as in 445. which is a valid approximation for samples drawn from a high dimensional Gaussian distribution as per the Gaussian annulus theorem [4]. This holds
for most generative models, as the initial noise vectors are typically nigh-dimensional (M > 50). In
our case, & corresponds to the noise latent 2((d) in Marigold, which we optimize using loss function
(eq. (T)) governed by the defocus blur forward model eq.
4 Method
We capture two images per scene: with F-stop (Nar 22) and exposure time lait, serves as the blur-free all-in-focus (AIF) image. A second image, Xp, is captured at lower F-stop (N, = 8) with exposure time fa thereby providing strong depth-varying defocus cues. The forward model in eq. ( assumes radiometrically linear images (no gamma correction or non-linear processing) and energy constancy between the AIF and blurred images. We use raw images to satisfy these assumptions. Since total captured energy scales with exposure time (t) and aperture area (x (f/N)²) (25), we
scale X₁ by the factor fat N2 to match the energy in x. Note that we vary the exposure time to
t, Na
ensure well-exposed measurements across F-stop settings, while fixing the camera gain.
We frame metric depth estimation process in eq. (6) as the forward model, and Marigold as the monocular depth prior. To obtain scale-invariant monocular depth d [o, 1],
inverse problem, with the defocus blur image formation
x (encoded to (x)), and learnable depth latent vector 2.f NN0, I). A single inference step
of Marigold- LCM gives us the denoised depth latent 2(1)) = Es ((()) .z 11), which decoded
to monocular depth = )) The predicted monocular depth d jo, 1] then mapped to metric depth by affine transforming d with learnable metric scale (a) and offset (3) per scene, d = a + 3. To ensure that ax 3 remain bounded and differentiable, we parameterize them as = Smax o0 (a) and 3 8min (bb) where a(.) is the sigmoid function; a, b are unconstrained learnable parameters initialized to 0, and 8max and 8min are the upper and lower scene depth bounds, respectively, which we assume are known priori (valid for indoor scenes). To summarize, the metric
5

---

GT
MLPro
Ours (Init)
Ours
UniDepth
Metric3D
1.5
Metric
1.0
Depth (m)
0.6
(a) AIF
(b) Blurred
(c) Depth maps
Figure 3: Correcting texture-depth coupling, We assess MMDE performance on textured fronto-
parallel 2D planes with constant ground truth depths (GT). Using an all-in-focus (a) and blurred image (zoom into insets) (b), our method (RMSE: 0.01) recovers the correct depth maps (c) for the two textured planes. We resolve the texture coupling in the Marigold prediction (Ours Init) and recover the correct metric scale. Competing methods (RMSE: 0.2-0.5) 5) fail to predict both the constant relative depth map (except MLPro and Metrie3D in row and the correct scale.
depth (d"") can be expressed using the optimizable parameters a, b, ZT as:
(d)
8max ((a) D &0 (2) 2 11) + Smin ((b)
dm
y b, (d)
(9)
The optimized metric depth T can then be recovered by solving:
d
arg
|xb g(x, a F,
(10)
d"=y( (a
(d) VM,
subject to
(11)
where g(-) denotes the defocus blur forward model (eq. (7), X₂ and the captured blurred and AIF images, respectively. By optimizing the learnable parameters a, b, by we incorporate defocus blur cues for both correct metric scale recovery (a, b) and refining the initial depth estimate
by Marigold ( See fig. for the overview of our method.
Motivating Synthetic Toy Examples We demonstrate that our approach resolves texture-depth coupling, accurately recovering metric depth in a synthetic scene with a textured plane constant depth. While such plane may actual 3D scene is challenging when viewed from single viewpoint. Data-driven methods are biased towards predicting depths that reflect surface variations in the absence of true depth changes, i.e., their outputs are strongly texture coupled. However, supplementing the AIF image with simulated blurred image provides defocus scene, as demonstrated in fig.3 Jssing toy examples of textured planes with constant depth. While bearning-bssed based methods and initial Marigold outputs suffer from texture coupling and scale errors, our method corrects both, producing accurate, constant-depth maps that outperform all baselines.
simple, distinguishing it from flat 2D image/poster or
that help method disambiguate flat poster from 3D
Accelerating inference We use the distilled latent consistency model version of Marigold (Marigold-LCM) to reduce the number of sampling steps significantly. We observe that a sin- gle sampling step suffices for our case, which significantly speeds up inference-tine optimization [36] [4] relative to the normal 20-50 inference steps (37 61 that Marigold (60] uses. We show an ablation study with more sampling steps in supplement.E. We also implement custom CUDA kernels for the DSFc-PSF forward model. This provides a 2.5x speed up over the Py implementation pro- method to scale higher-resolution images. Using a single sampling step of Marigold-LCM allows us to compute gradients w.r.t zy without gradient checkpointing previously done in 337.
vided by [62 while being more memory efficient, allowing
Optimization details We run the optimization for 200 iterations, which takes roughly 3.5-4 minutes on an NVIDIA A-40 GPU with peak memory usage of 15 GB. We use the Adam optimizer with

---

AIF
Blured
GT
Ours
MrP
UniDepch
Metric 3D
3.65
THORDOG
BOOKS
2.95
STAIRS
PIANE
Metric Dupih
(m)
TOYS
0.85
collected dataset. Our method consistently estimates accurate metric depth across all the scenes. We also observe better relative depth recovery due to leveraging defocus cues (zoom in 4x on blurred) in some regions (blue boxes, STAIRS). While the competing methods perform comparably to ours in some cases (MLPro:PLANE, KITCHEN, UniDepth:STAIRR Metric3D:THORDOG BOOKS), they struggle with the rest of the scenes due incorrect relative depth (TOYS, BOOKS) and metric scale (PLANE) recovery. recovers sharp details but fails at metric scale and relative depth accuracy for many of the scenes. Since the RealSense has wider FOV than the DSLR, we show a roughly aligned crop of the GT depth for comparison.
Figure 4: Comparisons
learning rate 1.5 10-3 3 for (d) x 10-3 for (a, 6). and default values for optimizer parameters. Note that we use the same scene bounds Smin = 1.49, Smax = 3.5 for all the real scenes in our dataset. These values represent conservative upper bound on the potential maximum scale and offset in the real dataset. Please see supplement.A for more details.
Experiments and Results
Dataset details Our method requires 2 images captured with different apertures (but same view- point) to integrate defocus cues. Standard monocular depth datasets 449 35 typically capture in-focus images
single aperture per scene, making them unsuitable for evaluating our method. While one could simulate defocused images with eq. (7) and RGBD data, this does not capture the model mismatch (occlusion, diffraction) in the physical image formation process. Therefore, fairly evaluate our method, we construct a hardware capture setup, shown in fig. 2 comprising an Intel RealSense depth camera rigidly mounted to a DSLR camera (Canon EOS 5D Mark II). We use this system to collect custom real-world dataset of 7 unique scenes, evaluating our method against learning-bast based MMDE baselines. We choose scenes with diverse subjects and depth profiles, placed within the operating depth range of RealSense (0.3-3.8m) to ensure accurate ground truth depth. Note that the CoC changes negligibly with depth beyond these distances, making defocus cues unreliable. For each scene, we capture images at 6 different apertures, f/4 //11, 1/13, ʃ/16, and //22, with the latter serving as the AIF image x. The blurred image, Xb. is selected from lower F-stop images (see fig. 5 for comparison of F-stop setting depth map quality). As described section [ we try to maintain a similar ratio of the exposure time and the camera aperture area for all the measurements taken for scene. The lens focal length (f) and F-stop (N) provided by the camera EXIF data, and the focus distance is read manually from the lens's analog focus scale"
Alternatively. lens with focus motor encoding would allow this be known from EXIF

---

6 different and f initializaatins
0.88
Metric Depth RMSE ws P-Number (IN)
0.7
Initial B
RM050
0.4
0.1
2.0
0.1
0.4
0.0
10 15 20
Initial
Figure 5: Analyzing the effect of different aperture sizes and initializations. Left: We use our forward model to simulate blurred images of a scene from the NYU-v2 dataset. We observe minimum depth error at N = 13, with errors increasing at more extreme aperture values. Right: We plot 81 at the end of optimization for various a, 3 initializations normalized between 0-1 (a _
in the figure for visualization. While the performance degrades for small values of ax 3 (bottom left), it is relatively stable for broad range of initializations.
Please see supplement.B for camera parameters and other dataset details, and supplement.H for more discussion on ground truth depth map quality.
Evaluation Metries We evaluate the predicted metric depth, d'm, from our method against the
RealSense ground truth depth d (with overloaded notation), using the metrics in S (9]. Specifically,
we compute absolute relative error (REL) = M
EM
i=1
d; root mean squared error (RMSE)
V 1/4 Σ dd |= average log error (log10) M7
log d, 1og dr" and accuracy Σ Zisl
a < (1.25)" for n = 1, 3. M denotes the d:
thresholds 6, = fraction of pixels where max
d
number of pixels, and d, a denote the RealSense and predicted depths pixel respectively. We also report point-cloud-based metrics, i.e., Chamfer Distance (CD) and the aggregated F1 score (FA) as defined in [39]. We align the ground truth and predicted depth map, similar to [19], and compute the metrics for pixels with non-zero values across the aligned depth maps. See supplement. C for more details on the depth map alignment/calirration procedure.
Quantitative and qualitative results We evaluate method against some of the recently popular MMDE methods Unith (39], Metric3D [74), and MLPro [7] on our collected dataset (table 2 and the NYUv2 (35) test set. Both Metric3D and UniDepth are provided with the required cam- era parameters as input. We outperform existing methods qualitatively (fig. 4 and quantitatively (table 2 on all the evaluation metrics, averaged over all 7 scenes in our collected dataset. Please see supplement.D for per-scene quantitative metrics on our dataset and supplement.F H for depth error visualizations. On the NYUv2 test set, our method is on par with the MMDE baselines. See supplement.J for more details. For our dataset, the MMDE methods (MLPro, Metric3D) recover sharper details and are on-par with our method on some scenes (see fig. 4. but they lack consistency in their overall performance across all scenes. Our method achieves better consistency across varying scene conditions. Leveraging defocus cues enables our approach recover the correct depth scale while also resolving relative depth errors in some cases (fig. g insets). We also evaluate our method (row 4 table 2) using a Gaussian PSF (00 in the forward model. While previous work (20) uses the Gaussian PSF for training models for unsupervised depth recovery, we find that the model mismatch between the Gaussian PSF and the PSF the real camera (fig. [ leads to severe performance degradation in our case, compared using the Disc PSF, which matches the real camera PSF better. This highlights the value a physically consistent forward model, even with strong learned priors.
Comparing with fine-tuning based methods Our method relies on optimizing noise-latents of a relative depth foundation model (RDFM) such as Marigold or Geowizard at test-time. Our approach (inference-ting optimization) can be valuable in zero-shot settings, where fine-tuning- methods would struggle due to the domain gap. We validate this by comparing method with MMDE
8

---

Method
RMSE 4
REL 4
log10
81 f
82
8x
CD 4
FA
FT Hypersim
0.523
0.314
0.151
0.178
0.825
0.971
0.208
0.616
FT NYUv2
0.407
0.252
0.096
0.654
0.832
0.988
0.183
0.733
Marigold FT NYUv2
0.475
0.231
0.098
0.634
0.878
0.963
0.175
0.808
Ours, No FT
0.273
0.125
0.052
0.879
0.975
0.991
0.103
0.870
Table 1: Comparison with fine-tuning-basee (FT) methods on our dataset. Our method (row 4)
outperforms (FT))based methods across all metrics. NYUv2 (real, depth < 10m) has lesser domain gap to our dataset (real, depth s 3.5m) than HyperSim (synthetic, depth S 20m); as evident by the when FT on NYUv2 (row 2) compared to HyperSim (row 1).
improvement in DepthAnythingy2
models obtained by fine-tuning RDFMs (DepthAnythingge 71 Marigold) on Metric depth datasets
with a restricted depth range, as required by our method (section 4). We evaluate the following fine-tuning- based baselines on our collected dataset in table - Det-thhntttthnggg [71] fine-tuned on the Hypersim [2] dataset (row 1), NYU-v2 train set (row 2). and Marigold Fine-Tuned on NYUv2 (row 3). Our method (row 4, table I outperforms the fine-tuning-basse
baselines. This validates the utility of our inference-time method in zero-shot settings, where fine-tuning- -bandd methods can struggle due to a domain gap. Please see supplement.K K for details on fine-tuning these baselines and additional results on the NYUv2 test set. In addition to tin--tuning-bas based methods, we also outperform [56], which estimates metric depth from multi-aperture inputs via self- self--upervised learning with differentiable forward model. See Supplement.I. for more details on the comparison with [56
Extending our method beyond Marigold Our method is plug-and-play and, in principle, works with any diffusion- model like Marigold that exposes a differentiable mapping from its latent space to depth estimates. This enables test-time latent optimization, which allows for incorporating defocus cues without retraining. We demonstrate this by using Geowizard as the diffusion backbone [T6] instead of Marigold. GeoWizard is stable-diffusion-baeee monoeular depth normal predictor, trained on a more complex data distribution. Our method achieves similar performance with either of the backbones (Rows 5,6 in table 2). but shows minor patch-level artifacts with GezaWr likely due to stronger dexture-depth coupling. See Supplement. for visualizations and hyperparameter details.
Method
RMSE
REL 4
log10 4
61
52
1
53 1
CD
FA f
MLPro
0.468
0.246
0.105
0.597
0.821
0.990
0.205
0.696
UniDepth
0.574
0.358
0.152
0.263
0.757
0.902
0.260
0.612
Metrie3D
0.349
0.195
0.087
0.611
0.958
0.983
0.135
0.814
Ours Gaussian
0.528
0.279
0.142
0.422
0.695
0.928
0.241
0.652
Ours Disc
0.273
0.125
0.052
0.879
0.975
0.991
0.103
0.870
Ours (Disc) with GeoWizard
0.291
0.137
0.061
0.824
0.966
0.990
0.105
0.874
Table 2: Comparison with learning based MMDE methods on our dataset. Our method with the
Disc PSF outperforms all the MMDE baselines averaged over all scenes in our dataset. The disc PSF, being more consistent with the real camera PSF, outperforms the Gaussian PSF UniDepth, Metric3D, and our method are provided with camera intrinsics parameters during inference.
5.1 Ablation Studies
Only single blurred image input: In our method, Marigold takes the all-in-focus (AIF) image (F/22) as imput to predict relative depth, which is combined with Icarnable scalc-offset parameters to synthesize a blurred image. The synthesized image is compared with a captured blurred image (F/8) that provides metric depth cues (52). To assess the contribution of the AIF, we remove the AIF and instead provide a single moderately blurred image (F/16) as input to both Marigold and the loss function (eq. (T)).. This ablation also evaluates whether the diffusion prior alone is strong enough to operate with a single modestly blurred image. Removing the AIF input leads large RMSE increase (1.36 vs. 0.346) and visible artifacts (fig. 6), indicating that while the diffusion prior captures coarse depth cues, the AIF input is essential for accurate metric depth estimation.
Sensitivity to o, initialization: Gradient-based methods are known to be susceptible to local minima for non-convex optimizations. We thus evaluate the sensitivity of our method to the initializa-
tion for a, 3. We run the optimization in (eq. (11) with fixed (d) while grid-searching over initial

---

Socoe
Depth from blureed image Depth from AIF bbuured image
GT Depth: Realsense
Figure 6: Degradation in depth quality on using only a single blurred image. For the TOYS scene, we observe that using single blurred image input results in severely inaccurate relative depth (middle), with all toys, guitar, and the monitor at similar relative depths. Our proposed method (right) recovers the depth ordering between the objects more accurately.
values of a and B. In fig. (right), we observe that performance (measured by 51) drops for small initialization values but remains stable across broad range around a 0.5, B 0.5, supporting the robustness of our chosen initialization. We also ablate on sensitivity to initial ZT in supplement.E.
Improvements in relative depth from defocus cues We quantitatively evaluate the effect of defocus cues in our method on improving the relative depth quality. Optimizing only a, B while holding zig' constant leads performance drop (table 3). This highlights the role of defocus cues in refining the relative depth initially predicted by Marigold. Please see supplement.F for visualizations.
Method
RMSE +.
REL 1
log10
81
f
82
f
83
t
Ours o,, 8 opt
0.297
0.156
0.069
0.743
0.957
0.99
Ours
0.273
0.125
0.052
0.879
0.975
0.991
Table 3: Relative depth quality Optimizing the noise latent along with the affine parameters (ours) performs better than optimizing only the affine parameters (a, opt).
Different aperture sizes We analyze how the aperture (F-stop) used for capturing the blurred image affects performance. To do this, dataset with high-quality ground truth depth annotations. This synthetic setup allows evaluating large F-stops that cannot be captured with our camera, while isolating aperture size from forward model mismatches in a real setup. Using the ground truth depth and our forward model (eq. (7), we simulate the blurred images X,, at varying F-stops (N values) and compute the error metrics between the ground truth and our predicted depth. We observe in fig. that the performance (measured in RMSF) degrades for extreme aperture sizes. This is expected, extreme blur (high low) makes the inverse problem ill-posed, and an optimal blur level is key for accurate depth recovery. While N 13 appears be optimal in simulation (ignoring model mismatch), underperforms N on average for real scenes, likely due to low contrast and insufficient blur cues in some of the scenes (STAIRS, PLANE). Please see supplement. G for results across all apertures captured in the real dataset.
scene from the NYU-v2 [35], an indoor RGBD
Limitations and Future Work
While our method outperforms data-driven MMDE baselines, remains significantly slower at inference time. Our method is best suited to scenes with a small depth range for which defocus blur offers high depth sensitivity. We observe that if the initial Marigold prediction severely incorrect in some regions (visualized fully correct them (ours for SHOER ACK,, THORDOG in fig. 4p. A possible extension of our method is to jointly estimate the AIF and depth map from a single blurred input, as previously explored in non-zero-shot approaches [19 DD Our framework can broaden the utility of pre-trained depth priors to scientific applications involving depth-dependent imaging processes such as hyperspectral imaging [27), endoscopy (31], and microscopy (72). While we avoid discretizing the depth map loses accuracy at occlusion boundaries. We envision further improvements through better PSF engineering (coded aperture masks) and more accurate forward modeling of defocus blur. Another promising direction
supplement.Fi the optimization may not always be able to
(231. our forward
adapt feed-forward methods such as (71) for handling multi-aperture inputs through test-tim eisttimme adaptation and fine-tuning [73]. Large-scale aueti-aperture datasets could make such methods practical for these settings.
Acknowledgements: We thank Alankar Kotwal and Bhargav Ghanekar for helpful discussions, Namrata Mantri for helping collect the real dataset, and Agastya Kalra for proofreading the draft.
10

---

References
[1] Abdullah Abuolaim and Michael S Brown. Defocus deblurring using dual-pixel data. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part 16, pages 111-126. Springer, 2020.
[2] Emma Alexander, Qi Guo, Sanjeev Koppal, Steven Gortler, and Todd Zickler. Focal flow: Measuring distance and velocity with defocus and differential motion. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-1,, 2016, Proceedings, Port III 14, pages 667-682. Springer, 2016.
[3] Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. Diffusercam: lensless einle-exposure 3d imaging. Optica, 5(1):199, 2017.
[4] Imre Bárány, Van Vu, et al. Central limit theorems for gaussian polytopes. The Annals of Probability. 1594)1159-1 2007.
[5] Shariq Farooq Bhat, Reiner BirkI, Diana Wolk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302.12288 2023.
[6] Reiner Birkl, Diana Wofk, and Matthias Müller. Midas v3. 1-a model zoo for robust monocular relative depth estimation. arXiv preprint arXiv:2307. 14460, 2023.
[7] Aleksei Bochkovskii, Amaël Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than second. arXiv preprint arXiv:2410.02073, 2024.
[8] Marcela Carvalho, Bertrand Le Saux, Pauline Trouvé-Peloux, Andrés Almansa, and Frédéric Champagnat. Deep depth from defocus: how can defocus blur improve estimation using dense neural networks? In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[9] Hyungjin Chung, Jeongsol Kim, Michael T. McCann, Mare L. Klasky, and Jong Chul Ye. Diffusion poste- rior sampling for general noisy inverse problems. In International Conference on Learning Representations (ICLR), 2023.
[10] Hyungjin Chung, Byeongsu Sim, Dohoon Ryu, and Jong Chul Yc. Improving diffusion models for inverse problems using manifold constraints. In Advances in Neural Information P Processing Systems (NeurIPS). 2022.
[11] Hyungjin Chung, Byeongsu Sim, and Jong Chul Ye. Come-closer-diffuse faster: Accelerating conditional diffuusion models for inverse problems through stochastic contraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12403-12412, 2022.
[12] Giannis Daras, Hyungjin Chung, Chich-Hsin Lai, Yuki Mitsufuji, Jong Chul Yc, Peyman Milanfar, Alexandros G Dimakis, and Mauricio Delbracio. A survey diffuusio models for inverse problems. arXiv preprint arXiv:2410000083 2024.
[13] Yiquan Duan, Xianda Guo, and Zheng Zhu. Diffusiondepth: Diffusion denoising approach for monocular depth estimation. In European Conference on Computer Vision. pages 432-449 Springer. 2024.
[14] Luca Eyring, Shyamgopal Karthik, Karsten Roth, Alexcy Dosovitskiy, and Zeynep Akata, Reno: Enhancing one-step text-to-image models through reward-basee noise optimization. Advances in Neural Information Processing Systems, 37:125587-125519, 2024.
[15] Hany Farid and Eero P Simoncelli. Range estimation by optical differentiation. Journal of the Optical Society of America A, 15(7):177-17786 1998.
[16] Xiao Fu, Wei Yin. Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In European Conference on Computer Vision, pages 241 258. Springer, 2024.
[17] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Tal Hassner, and Danicl Cohen- Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint 2rXiv:2208.00 01618, 2022.
[18] Rahul Garg, Neal Wadhwa, Sameer Ansari, and Jonathan T Barron. Learning single camera depth estimation using dual- pixels. vision, pages 1628-7637, 2019.
dual-pixels. In Proceedings of the IEEE/CVF international conference on computer

---

[19] Bhargav Ghanekar, Salman Siddique Khan, Pranav Sharma, Shreyas Singh, Vivek Boominathan, Kaushik Mitra, and Ashok Veeraraghavan. Passive snapshot coded aperture dual-pixel rgb-d imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 25348-25355 2024.
ing-ing
[20] Shir Gur and Lior Wolf. Single image depth estimation trained via depth from defocus cues. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7683-7692, 2019.
[21] Caner Hazirbas, Scbastian Georg Soyer, Maximilian Christian Staab, Laura Leal- Laixe, and Daniel Cremers. Deep depth from focus. In Computer Vision ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2 6, 2018, Revised Selected Papers, Part III 14, pages 225-511. Springer, 2019.
[22] Jing He, Haodong Li. Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and Y'ig-CCog Chen. Lotus: Diffusion-based visual foundation model for high-quality dense prediction. arXiv preprint arXiv:2409. /8124, 2024.
[23] Hayato Ikoma, Cindy M Nguyen, Christopher Metzler, Yifan Peng, and Gordon Wetzstein. Depth from defocus with learned opties for imaging and occlusion aware depth estimation. In 2021 IEEE International Conference on Computational Photography (ICCP), pages 1-12. IEEE, 2021.
[24] Kaiwen Jiang, Yang Fu, Mukund Varma T, Yash Belhe, Xiaolong Wang, Hao Su, and Ravi Ramamoorthi. A construct-opimmize approach to sparse view synthesis without camera pose. In ACM SIGGRAPH 2024 Conference Pupers, pages 1-11, 2024.
[25] Loyd A Jones, Emery Huse, and Vincent C Hall. On the relation between time and intensity in photographic exposure. Journal of the Optical Sociery of America, 12(4):321-345 1926.
[26] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 9492 9502, 2024.
[27] Esther Kho, Lisanne L. de Boer, Anouk Post, Koen K Van de Vijver, Katarzyna Jóžwiak, Henricus JCM Sterenborg, and Theo JM Ruers. Imaging depth variations in hyperspectral imaging: development of a method to detect tumor up to the required tumor-free margin width. Journal of biophotonics. 12(11):e0199000S 2019.
[28] Anat Levin, Rob Fergus, Frédo Durand, and William T Freeman, Image and depth from a conventional camera with coded aperture. ACM transactions on graphics (TOG), 26(3):70-es, 2007.
[29] Haiting Lin, Can Chen, Sing Bing Kang, and Jingyi Yu. Depth recovery from light field using focal stack symmetry. In Proceedings of the IEEE International Conference Computer Vision, pages 14599, 2015.
[30] Chao Liu, Srinivasa G Narasimhan, and Artur W Dubrawski. Matting and depth recovery of thin structures using focal stack. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition. pages 6970-6978. 2017.
[31] Yuying Liu and Siyang Zuo Self-supervised monocular depth estimation for gastrointestinal endoscopy. Computer Methods and Programs in Biomedicine, 238:107619, 2023.
[32] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[33] Maxim Maximov, Kevin Galim, and Laura Leal- Taixé. Focus on defocus: bridging the synthetic to real domain gap for depth estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1071-1080. 2020.
[34] Opher Bar Nathan, Deborah Levy, Tali Treibitz, and Dan Rosenbaum. Osmosis: Rgbd diffusion prior for underwater image restoration. In European Conference on Computer Vision, pages 302-319. Springer, 2024.
[35] Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012.
[36] Zachary Novack, Julian McAuley, Taylor Berg-Kirkpatrick, and Nicholas J. Bryan. DITTO-2: Distilled diffusion inference- timennee-imeeteee t-optimization for music generation. In International Society of Music Information Retrieval (ISMIR), 2024.
12

---

[37] Zachary Novack, Julian McAuley, Taylor KirgKKikkaa,iic, and Nicholas J. Bryan. DITTO: Difffusion inference-time t-optimization for music generation. In International Conference on Machine Learning (ICML). 2024.
[38] Liyuan Pan, Shah Chowdhury, Richard Hartley, Miaomiao Liu, Hongguang Zhang, and Hongdong Li. Dual pixel exploration: Simultancous depth estimation and image restoration. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 4340-4349 2021.
[39] Luigi Piccinelli, Yung-Hsu Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Lue Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 10106-10116, 2024.
[40] Michael Potmesil and Indranil Chakravarty. A lens and aperture camera model for synthetic image generation. In Proceedings of the 8th Annual Conference Computer Graphics and Interactive Techniques, SIGGRAPH '81, page 297-305, New York, NY, USA, 1981. Association for Computing Machinery.
[41] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Kolunn. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE transactions on pattern analysis and machine intelligence, 44(3):1623-166 163-)66236666.. 2020.
[42] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10912-10922, 2021.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 10684-10695, 2022.
[44] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[45] Dvir Samuel, Rami Ben-Ari, Nir Darshan, Haggai Maron, and Gal Chechik. Norm-guided latent space exploration for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[46] Ashutosh Saxena, Sung Chung, and Andrew Ng. Learning depth from single monocular images. Advances in neural information processing systems, 18, 2005.
[47] Saurabh Saxena, Charles Hermaan, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David Fleet. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation. In A. Oh, Naumann, Globerson, Saenko. M. Hardt, and Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 39443-39469. Curran Associates, Inc., 2023.
[48] Saurabh Saxena, Junhwa Hur, Charles Herrmann, Deqing Sun, and David J Fleet. Zero-shot metric depth with a field-of-view conditioned diffusion model. arXiv preprint arXiv:2312.13252 2023.
[49] Daniel Scharstein and Richard Szeliski. High-accuracy stereo depth maps using structured light. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pages 295-202. IEEE, 2003.
[50] Markus Schön, Michael Buchholz, and Klaus Dietmayer. Mgnet: Monocular geometric scene understand- ing for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15804-15815, 2021.
[51] Yichen Sheng, Zixun Yu, Lu Ling. Zhiwen Cao, Xuaner Zhang. Xin Lu, Ke Xian, Haiting Lin, and Bedrich Benes. Dr. bokeh: Differentiable occlusion-award bokeh rendering. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4515-4525, 2024.
[52] Haozhe Si, Bin Zhao, Dong Wang. Yunpeng Gao, Mulin Chen, Zhigang Wang, and Xuelong Li. Fully self-supervised depth estimation from defocus clue. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9140-9144009 91099914922.09 2023.
[53] Bowen Song, Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solving inverse problems with latent diffusion models via hard data consistency. arXiv preprint arXiv:2307.08122 2023.
[54] Jiaming Song, Arash Vahdat, Morteza Mardani, and Jan Kautz. Pseudoinverse- guided diffusion models for inverse problems. In International Conference on Learning Representations, 2023,
13

---

[55] Ziyang Song, Zerong Wang, Bo Li, Hao Zhang, Ruijie Zhu, Li Liu, Peng- Tao Jiang, and Tianzhu Zhang. Depthmaster: Taming diffusion models for monocular depth estimation. arXiv preprint urXiv:250.00277. 2025.
[56] Pratul Srinivasan, Rahul Carg. Neal Wadhwa, Ren Ng. and Jonathan T Barron. Aperture supervision for monocular depth estimation. In Proceedings of the I.F.P. Conference Computer Vision and Pottern Recognition, pages 6393-6401, 2018.
[57] Michael Strecke, Anna Alperovich, and Bastian Goldluecke. Accurate depth and normal maps from occlusion-aware focal stack symmetry. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 281 1--222, 2017.
[58] Murali Subbarao and Gopal Surya. Depth from defocus: A spatial domain approach, International Journal of computer vision, 13(3):271-22) 1994.
[59] Huixuan Tang, Scott Cohen, Brian Price, Stephen Schiller, and Kiriakos Kutulakos. Depth from defocus in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2740-2748, 2017.
[60] Massimiliano Viola, Kevin Qu, Nando Metzger, Bingxin Ke, Alexander Becker, Konrad Schindler, and Anton Obukhov. Marigold-dc: Zero-shot monocular depth completion with guided diffusion. arXiv preprint arXiv:2412. 13389, 2024.
[61] Bram Wallace, Akash Gokul, Stefano Ermon, and Nikhil Vijay Naik. End-to-end diffusion latent optimiza- tion improves classifier guidance. 2023 IEEE/CVF International Conference Computer Vision (ICCV), pages 7246-7256, 2023.
[62] Chao Wang. Krzysztof Wolski. Xingang Pan, Thomas Leimkühler, Bin Chen, Christian Theobalt, Karol Myszkowski, Hans- Peter Seidel, and Ana Serrano. An implicit neural representation for the image stack: Depth, all in focus, and high dynamic range. Technical report, 2023.
[63] Ning-Hsu Wang, Ren Wang, Yu-Lun Liu, Yu-Hao Huang, Yu-Lin Chang, Chia-Ping Chen, and Kevin Jou. Bridging unsupervised and supervised depth from focus via all-in-focus supervision. In Proceedings of the IFFF/CVF international conference on computer vision, pages 12621-126311 2021.
[64] Masahiro Watanabe and Shree K Nayar. Rational filters for passive depth from defocus. International Journal of Computer Vision, 27:203-225, 1998.
[65] Bowen Wen, Matthew Trepte, Joseph Aribido, Jan Kautz, Orazio Gallo, and Stan Birchfield. Foundation- stereo: Zero-shot stereo matching, arXiv preprint arXiv:2501. 009898, 2025.
[66] Changyeon Won and Hae- Jeon. Learning depth from focus in the wild. In European Conference on Computer Vision, pages 1-18. Springer, 2022.
[67] Yicheng Wu, Vivek Boominathan, Huaijin Chen, Aswin Sankaranarayanan, and Ashok Veeraraghavan. Phasecam3d learning phase masks for passive single view depth estimation. In 2019 IEEE International Conference on Computational Photography (ICCP), pages 1-12. IEEE, 2019.
[68] Shumian Xin, Neal Wadhwa, Tianfan Xue, Jonathan T Barron, Pratul Srinivasan, Jiawen Chen, Ioannis Gkioulekas, and Rahul Garg. Defocus map estimation and deblurring from single dual-pixel image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2228-2238, 2021.
[69] Yalin Xiong and Steven A Shafer. Depth from focusing and defocusing, Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 68 68-73. IEEE, 1993.
[70] Lihe Yang. Bingyi Kang. Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10371-10381, 2024.
[71] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint 2406:22406009144, 2024.
[72] Kyrollos Yanny, Nick Antipa, William I. Liberti, Sam Dehaeck, Kristina Monakhova, Fanglin I. Linda Liu, Konlin Shen, Ren Ng. and Laura Waller. Miniscope3d: optimized single-shot miniature 3d fluorescence microscopy. Light: Science & Applications, 9()::111. 2020.
[73] Teresa Yeo, Oguzhan Fatih Kar, Zahra Sodagar, and Amir Zamir. Rapid network adaptation: Learning to adapt neural networks using test-time feedback. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4687 2023.
14

---

[74] Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang. Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from single image. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9043-9053, 2023.
[75] Jiuling Zhang. Survey monocular metric depth estimation, 2025.
[76] Yucheng Zheng and M Salman Asif. Joint image and depth estimation with mask-based lensless cameras.
IEEE Transactions on Computational Imaging, 6: 167-1178, 2020.
[77] Yuxuan Zheng. Yifan Li, Yizhuo Zhang. Yiran Zhang, Lin 7hang, and Lei Zhang. TiNO-Fdit: Timestep and noise optimization for robust diffusion-based image editing. arXiv preprint arXiv:2304.06720, 2023.
[78] Changyin Zhou, Stephen Lin, and Shree Nayar. Coded aperture pairs for depth from defocus. In 2009 IEEE 12th international conference computer vision, pages 325-332. IEEE, 2009.
15

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We demonstrate evidence of the claims in the abstract in the section S
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In section 6 we discuss the limitations of our method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We will mention all the hyperparameters in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c)
If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in the supplemental material?
Answer: [Yes]
17

---

Justification: We will release the code after acceptance, But the dataset will be provided as an anonymous link the supplement.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guides.
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips.cc////// httppssccpppulli/guddeeeeeeeeeee Sttp://oooeeSiissseeoeee issimmisssiii wtt.imiissoopoooioooooooiii onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify implementation details in the paper, additional details will be included in the supplement.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We perform thorough ablations on different initialization, and characterize the effects of different aperture sizes method through synthetic study. In the supplement, we characterize the effect of the stochasticity of the initial latent noise vector of the diffusion model by repeating an experiment 10 times with different random initializations for the latent noise vector, and report the variance in the observed final metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention the GPU used, memory usage, and optimization time for our algorithm.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics /ttttp://neurrii http::pp........cpcctcccccceeeee htt.t//ggg//////..o/.o/////o///a
Answer: [Yes]
Justification:
We confirm the ethics guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work has no particularly negative societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:(Yes Yes]
Justification: We have cited the datasets and models used for comparison. These datasets have been widely used in published papers on this topic.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will release the self-collected dataset in the supplementary material with guidelines on how to use it, and all the camera parameters used while capturing it.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
crowdsourcing involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No human subjects involved
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
21

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: This paper is not related LLM research or applications.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttppr////euups..... http///rips..ccoooffeeeeeeee//// /c//Conferrecccs//225//LL for what should or should be described.
22


Reviews:
[{"review_id": "qO35pYdY1q", "reviewer": "Reviewer_W1fb", "summary": "This paper presents a novel method for upgrading a relative depth map to metric via unsupervised test-time optimization that requires the existence of a second, defocused image of the scene besides the primary sharp image. A major contribution of the paper is the introduction of a rigorous, physics-based defocus blur forward model, which depends on the predicted metric depth map and allows to define a self-supervised loss based on consistency of the captured blurry image and the respective blurry image which is synthesized based on the all-in-focus image, the relative depth map, and the learned scale and shift. The authors validate their approach on a small real-world indoor dataset they have collected and demonstrate superior performance compared to state-of-the-art direct learned metric depth estimators on their dataset.", "questions": "Please refer to the questions which are included in the lists of Major Weaknesses and Minor Weaknesses in my response above.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The usage of the central defocus cue for depth in a learned end-to-end framework via test-time optimization is innovative and insightful and it provides an interesting alternative to direct learned models for metric depth estimation.\n\n2. The derivation of the complete model, especially the defocus blur forward model, is mathematically solid, thorough, and well-grounded on optics.\n\n3. The experimental results on a small yet real-world dataset are very satisfactory, as the proposed method outperforms existing state-of-the-art direct learned metric depth estimators substantially on this dataset across all evaluation measures.\n\nMajor Weaknesses:\n\n1. The applicability of the proposed method is quite limited. As the authors already acknowledge, the employed forward model limits the method to indoor scenes with restricted depth ranges. Moreover, the need to capture two different images with different apertures limits the method to specialized camera types such as DSLR, apparently making it inapplicable to e.g. phone cameras or other commodity cameras. At the same time, the method cannot work for dynamic scenes including even slow object motions, as the two images - the all-in-focus and the defocused one - need to be captured in non-overlapping temporal exposure intervals. Thus, the generality and universality of the method is hurt.\n\n2. Metric depth on its own right is not sufficient for metric estimation, as one also needs the directions of the rays corresponding to the pixel grid in order to obtain a metric 3D estimation of the scene structure. Camera intrinsics thus need to be known or estimated. Supposing that the proposed method were applied to an in-the-wild setting with unknown intrinsic calibration, its output is thus insufficient to achieve the aforementioned goal. While the majority of the monocular metric depth methods assume known camera parameters, there are a few that do not [39, A] and that can estimate both camera intrinsics and scene range/depth. Moreover, the quality of metric estimation needs to be evaluated with 3D-wise metrics, such as the Chamfer distance and the F-score. The restriction to mere metric depth maps living in the 2D image space and to traditional depth metrics limits the practical relevance of the method.\n\n3. The authors state in L. 239-240 that they align the predicted depth map with the ground truth depth map before performing the evaluation. What exactly does this alignment entail? Is it applied for competing methods as well? I note here that a major challenge and desideratum in metric depth estimation is to lift the need for aligning / rescaling the model's output in any way and to rather obtain directly a metrically accurate prediction. This is already satisfied in the standard evaluation of recent metric depth estimation works [39, A]. Can the authors present comparisons without the aforementioned alignment?\n\nMinor Weaknesses:\n\n4. It is unclear what the exact optimization algorithm used at inference (cf. L. 212) is. Is it simple gradient descent? Or does it use conjugate gradients? Or is it of another type? What step sizes are used across the 200 iterations? The authors should clarify these points.\n\n5. The exact dependence of the range of the output depth with the scale and shift parameters $\\alpha$ and $\\beta$ is unclear to me. Based on Eq. (9), since the minimal values of the sigmoids are 0, the overall output depth value could be as low as 0. However, the authors state in L. 186 that the lower bound of depth is $s_{\\text{min}}$, not 0. Moreover, in Fig. 5 (right) and L. 266-269, the scale and shift parameters $\\alpha$ and $\\beta$ seem to be defined to range between 0 and 1, while based on Eq. (9) they should range between 0 and $s_{\\text{max}}$ and $s_{\\text{min}}$, respectively. Which of the above is true?\n\n[A] Piccinelli et al. UniK3D: Universal Camera Monocular 3D Estimation. In CVPR, 2025.", "was_revised": true, "final_justification": "The paper presents an innovative, optically and geometrically rigorous avenue towards metric depth estimation and has shown, also taking the rebuttal into account, very convincing results. In the rebuttal, the authors have clearly addressed the initial point on applicability/practicality of the method, the flexibility to multiple diffusion-based frameworks, the accuracy of their 3D predictions, the merit of the method against previous learned multi-shot / multi-aperture methods, and the quality of depth ground truth and alignment of it with the camera coordinate frame. Thus, I have increased my rating to 5: Accept, and I highly recommend acceptance of the paper.", "rebuttal": "We thank the reviewer (W1fb) for appreciating our experiments on real-world data and our insights on incorporating defocus cues with pretrained diffusion models, and have addressed their questions below. \n\n\n**Major Weakness 1: Concerns about limited applicability**.             \n*1a: Restricted depth range, dynamic scenes*.    \nWe agree with W1fb that our method is best applicable to scenes with a restricted depth range, which is predominantly true for indoor scenes, but also possible for outdoor scenes (e.g., STAIRS in our dataset). We will clarify this in L:289.\n\nWhile dynamic scenes are not the focus of our work, we acknowledge that handling dynamic scenes in our framework would be valuable. Our current assumption of static scenes aligns with standard practice in monocular depth estimation, as followed by prior works [A, B, E, D]. Note that a two-exposure imaging method does not preclude application to scenes containing any motion. In principle, short exposures can be used with rapid actuation of a camera aperture to capture our two input images with a delay that is on the order of 2x slower than a single image alone. This would only require the motion to be reduced by 2x vs. a single image method set to capture with minimal motion blur. Admittedly, some inter-exposure alignment would likely be needed, but we leave that to future work. \n\n*1b: Variable aperture cameras are also readily available on commodity devices*.    \nOur method only requires a single variable aperture camera, which is readily available in the form of handheld commodity devices like Canon DSLRs (as used in our work) and even in some smartphones such as the Samsung Galaxy S9, S10, and upcoming S26. While fixed aperture cameras are more common, variable aperture is still widely accessible and avoids the need for custom hardware like phase or aperture masks used in prior methods [D, F].\n\n**Major Weakness 2: 3D metrics and availability of camera Intrinsics**.                \n*2a: Additional 3D metrics*.    \nWe agree that 3D point cloud metrics such as Chamfer Distance and F1 score (derived from Chamfer Distance) [A, B] are also representative metrics for 3D metric depth; we have added them to *Table 1* below for our method and the competitive baselines. We will add these numbers in a revision. Our reported metrics (L:234 in main paper) are also well accepted in the literature [B, E] for depth accuracy – $\\delta_1, \\delta_2, \\delta_3, RMSE, log10$ capture metric 3D depth accuracy. \n\n| Method             | RMSE $\\downarrow$ | REL $\\downarrow$   | log10 $\\downarrow$ | δ1 $\\uparrow$    | δ2 $\\uparrow$    | δ3 $\\uparrow$    | CD $\\downarrow$    | FA $\\uparrow$     |\n|--------------------|-------|-------|-------|-------|-------|-------|-------|--------|\n| Ours                | 0.273 | 0.125 | 0.052 | 0.879 | 0.975 | 0.991 | 0.103 | 0.870  |\n| MLPro [E]             | 0.468 | 0.246 | 0.105 | 0.597 | 0.821 | 0.990 | 0.205 | 0.696  |\n| UniDepth [A]          | 0.644 | 0.376 | 0.157 | 0.259 | 0.684 | 0.954 | 0.244 | 0.633  |\n| Metric3D [B]          | 0.459 | 0.295 | 0.106 | 0.650 | 0.825 | 0.895 | 0.180 | 0.725  |.      \n\n**Table 1: Results with 3D Metrics (CD: Chamfer Distance), (FA: Aggregated F1 score as used in UniDepth [A]) on our dataset**. On the newly added 3D metrics CD (chamfer distance) and FA (aggregated F score), our method outperforms the existing baselines on our real dataset. This is consistent with the trends observed with the other metrics as well. $\\uparrow, \\downarrow$ indicate whether higher or lower is better, respectively.\n\n*2b: Availability of Camera Intrinsics*.          \nOur capture setup involves capturing an in-focus and a blurred image from a real DSLR camera. The metadata in these images contains EXIF data (camera intrinsics) by default. As the reviewer noted, this is standard practice, with most methods similarly relying on EXIF metadata [G, H]. Hence, we do not focus on the case of unknown camera intrinsics in our work, as that information is always going to be available by design in our capture setup. \n\n**Major Weakness 3: Processing the Intel RealSense depthmap to prepare our dataset**.   \nThe input images in our setup are captured with an RGB camera, and the ground truth depth map is captured from a RealSense camera (see Fig. 2c in the paper for the capture setup). These two cameras are physically close but not co-located; as a result, the depth from the RealSense does not geometrically correspond to the RGB camera (due to geometric misalignment). \n\nTo handle this misalignment, the cameras need to be calibrated using standard camera calibration. This is a standard data pre-processing step [C, D], and is also done by real public RGBD datasets like NYUv2 [C], which allows them to provide geometrically aligned RGB images and depth maps. This allows a fair comparison between the predicted depth from the RGB camera and the GT depth in the same coordinate system. In L:239-240 of our work, this is what we refer to as alignment. \n\nAt a high level, first, the pose between the RGB and RealSense cameras is estimated via standard camera calibration (section C of the supplement, L:241 main paper), similar to [D]. The RGB depth map is warped to the GT depth camera’s frame using the estimated pose, enabling comparison between the predicted and GT depth in a shared geometric space.\n\n *This alignment is crucial for a principled and accurate comparison between GT depth and predicted depth, and is the only alignment we use in our work – both for our method and the baselines.*  Therefore, all reported metrics in the rebuttal and Table 1 of the main paper are computed directly on the raw metric depth outputs. \n\n**Minor Weakness 1: Exact Inference Algorithm, Learning Rates**.      \nWe have included these details in section A of the supplement, which we reference in L:214 of the paper. We used the Adam optimizer in PyTorch for our optimization, and section A of the supplement contains learning rates and other optimization details.\n\n**Minor Weakness 2: Dependence of output depth on scale and shift parameters ($\\alpha$/$\\beta$)**.           \n The scale and shift parameters $\\alpha,\\beta$ are parameterized such that $\\alpha\\in[0, s_{max}]$ and $\\beta\\in[0,s_{min}]$, where $s_{min}, s_{max}$ denote the maximum possible values for the lower and upper bounds of the scene depths. We assume $s_{min}, s_{max}$ to be known since we operate in a restricted depth range. (*L 184:186, main paper*).   To ensure both boundedness, and differentiability during optimization, $\\alpha, \\beta$ are parameterized as (L:184, L:185 main paper) – \n$$\n\\alpha = s_{\\text{max}} \\cdot \\sigma(a)~~~\\textrm{and}~~~\n\\beta = s_{\\text{min}} \\cdot \\sigma(b),$$ where $a,b$ are the learnable unbounded variables, and $\\sigma(\\cdot)$ denotes the sigmoid activation. \n\nFor the performance ablation plot with $\\alpha,\\beta$ values (Fig. 5), we mention in the figure caption that the $\\alpha,\\beta$ values are normalized to 0-1 ($\\alpha \\rightarrow \\frac{\\alpha}{s_{max}}, \\beta \\rightarrow \\frac{\\beta}{s_{min}}$) for plotting.  We will update the figure caption to clarify this.     \n[A] UniDepth: Universal Monocular Metric Depth Estimation. CVPR 2024.     \n[B] Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image: ICCV 2024.    \n[C] NYU Depth Dataset V2: ECCV 2012.      \n[D] Passive snapshot coded aperture dual-pixel RGB-D imaging: CVPR 2024.     \n[E] Depth Pro: Sharp Monocular Metric Depth in Less Than a Second: ICLR 2025.     \n[F] Learning Phase Masks for Passive Single View Depth Estimation: ICCP 2019.    \n[G] Depth Estimation via Reconstructing Defocus Image: CVPR 2023.    \n[H] Aperture Supervision for Monocular Depth Estimation: CVPR 2018."}, {"review_id": "YBRJa7804E", "reviewer": "Reviewer_1d2m", "summary": "The paper proposes a novel training-free framework for zero-shot metric depth estimation. It achieves this by intelligently combining a pre-trained monocular depth estimator based on diffusion models (Marigold) with explicit physical cues derived from defocus blur. The system requires two images captured from the same viewpoint but with different aperture settings (an all-in-focus image and a blurred image). Marigold is initially used to estimate a relative depth map from the all-in-focus image. This relative depth is then converted to metric scale through an inference-time optimization process, where the depth parameters are iteratively refined to ensure consistency between the estimated depth, a physically-based defocus model, and the input blurred image. This approach offers a compelling solution to data scarcity in metric depth estimation and demonstrates promising zero-shot generalization capabilities.", "questions": "### Questions for Authors\n\n* **Regarding Weakness 1 (Misleading SOTA Comparison and Input Modality):**\n    * Could the authors provide a more direct and fair quantitative comparison against state-of-the-art **Depth from Defocus (DFD)** methods or other approaches that utilize focal stacks/multi-aperture inputs? This would better contextualize the proposed method's performance and novelty within its true category.\n    * Regarding the ablation study in Ln. 259 and Fig. 6, the comparison between their loss optimization (with AIF + Blurred images) and Marigold taking only the blurred image seems inconsistent. Most depth models are trained on AIF or nearly AIF datasets. Providing Marigold with a blurred image, as opposed to the original AIF setting it was likely optimized for, may not be a fair baseline. Could the authors instead provide a comparison where: 1) Marigold is given the blurred image, and 2) their loss function method also uses *two blurred images* with different aperture sizes? This would offer a more robust demonstration of the method's tolerance and effectiveness when dealing with blurred inputs.\n\n* **Regarding Weakness 2 (Limited Practicality):**\n    * To further demonstrate the practicality and \"plug-and-play\" nature of this inference-time optimization, could the authors showcase its effectiveness when applied to other state-of-the-art foundation relative depth models, such as Depth Anything v2 or GeoWizard?\n\n* **Regarding Weakness 3 (Lack of Clear Justification for Defocus Blur Choice):**\n    * The paper primarily uses defocus blur as the metric depth cue. Could the authors elaborate on the specific advantages of defocus blur in this *training-free, zero-shot* context, especially when compared to other potential metric depth cues like stereo correspondence?\n    * What are the unique challenges and benefits of integrating a diffusion-model-based prior (Marigold) with defocus blur, as opposed to other types of physical cues?\n\n* **Regarding Weakness 4 (Limited Discussion on Marigold's Specific Advantages):**\n    * While Marigold provides a strong relative depth prior, could the authors delve deeper into *why* a diffusion-based model like Marigold is particularly suited for this DFD integration? Are there inherent properties of diffusion models' representations (e.g., robustness to noise, handling of fine details, ability to capture global scene structure) that make them uniquely advantageous for this task compared to other state-of-the-art monocular depth estimators?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* **Innovative Training-Free Metric Depth Estimation:** The paper proposes a novel framework for zero-shot metric depth by uniquely combining a powerful pre-trained diffusion model (Marigold) with explicit defocus blur cues. This approach circumvents the need for large, labeled metric depth datasets, promoting robust generalization.\n* **Effective Repurposing of a Foundation Model:** The work intelligently leverages Marigold, a state-of-the-art diffusion-based monocular depth estimator. Marigold provides a strong initial relative depth prior due to its extensive pre-training, which is crucial for the subsequent physically-informed metric scale refinement.\n* **Principled Integration of Defocus Blur:** The use of defocus blur is a theoretically sound choice for recovering metric scale. Its well-defined physical relationship to depth allows for a robust gradient-based optimization to refine Marigold's relative depth output to metric units.\n\n### Weaknesses\n\n* **Misleading SOTA Comparison and Input Modality:** The comparison to strictly monocular metric depth estimation (MMDE) methods is not entirely fair. The proposed system requires two images (all-in-focus and blurred), placing it in the domain of multi-image depth estimation (Depth from Defocus, DFD), not purely monocular. One of the reasons why monocular doesn't perform as well as multi-view or focal stack is the lack of physical reference from multiple images. More appropriate comparisons would be against other DFD or focal-stack based approaches but restrict them with a stack number of two.\n* **Significant Computational Latency and Limited Practicality:** The reported inference speed of 3.5-4 minutes per image pair on an NVIDIA A40 GPU is a major practical disadvantage due to its per-image, inference-time optimization. This high latency severely restricts applicability in real-time or high-throughput scenarios. Another limited practicality is that since this paper proposed \"training-free\" but and inference time optimization for affine-transform, I believe to demonstrate the effectiveness and practicality, it should be tested on several SOTA foundation relative depth methods as a plug-and-play method, instead of purely tight to Marigold.\n* **Lack of Clear Justification for Defocus Blur Choice:** The paper doesn't thoroughly justify why defocus blur was chosen over other potential metric depth cues (e.g., stereo priors). A brief discussion on the unique advantages of DFD in this training-free context, compared to the challenges of integrating other cues, would strengthen the methodological choice.\n* **Limited Discussion on Marigold's Specific Advantages:** While Marigold provides a strong prior, the paper could elaborate more on *why* a diffusion-based model like Marigold is particularly suited for this DFD integration compared to other state-of-the-art monocular depth estimators.", "was_revised": true, "final_justification": "I have reviewed the authors' responses. Based on the authors' comprehensive and highly effective rebuttal, I am updating my scores and my final recommendation.\n\nMy initial review, which led to a Reject score, was based on the paper's submitted state, which I felt had significant technical and methodological shortcomings, particularly in its evaluation and claims of generality.\n\nHowever, the authors' rebuttal has provided substantial and convincing evidence that addresses these issues directly. Specifically:\n\n* **Weakness 1 (Misleading SOTA Comparison)**: The authors provided new experimental data (Tables 2, 4, and 5) that performs a more appropriate comparison against other multi-aperture and fine-tuning-based baselines. This new evidence effectively contextualizes the method's performance and validates its value in zero-shot scenarios where fine-tuned models suffer from a domain gap.\n\n* **Weakness 2 (Limited Practicality)**: The authors demonstrated the plug-and-play nature of their inference-time optimization solution by applying it to another foundation model (GeoWizard), as shown in Table 3. This addresses the concern that the method was tightly coupled to Marigold and strengthens its claim of generality.\n\n* **Weakness 3 & 4 (Justifications)**: The authors provided clear, technically sound justifications for their design choices, including the use of Marigold's differentiable latent manifold and the unique advantages of defocus blur over other physical cues. Their clarification on \"focus breathing\" was also robust and well-cited.\n\nThe authors' rebuttal has transformed the paper from one with significant flaws to a compelling and well-supported submission. The technical contributions, while initially presented with weak evidence, are now convincingly demonstrated. The paper now offers a clear and valuable trade-off: a slower, per-image optimization process that provides superior zero-shot metric depth over fine-tuning-based approaches in new domains.", "rebuttal": "We thank 1d2m for their detailed critique and for recognizing the novelty of our work. Our work demonstrates that Marigold, a generative model not built to incorporate defocus cues, can be repurposed to include them in a principled training-free manner. This achieves robust generalization without labeled depth datasets or retraining. We have validated our claims and the proposed framework through real-world experiments on a self-collected dataset, which also demonstrates the importance of correctly modeling the defocus blur (Fig. 2, Tab.1, L:253-257 in the main paper).  We have addressed the questions from 1d2m below:    \n\n**Weakness 1: Misleading SOTA comparison and input modality, comparison with multiaperture methods**.         \n*Misleading SOTA Comparison*.     \nWe would like to emphasize that our work aims to establish an existence proof of the proposed framework, and that our intent was not to suggest that we establish a new state-of-the-art for multi-shot depth imaging. If such an impression was conveyed, it was entirely unintentional, and we will clarify this more clearly in the paper. We intended to showcase the effectiveness of our approach within the context of our framework, without implying any misrepresentation of its standing relative to prior work.     \n*Comparison with Multi-Aperture Methods:*.          \nWe show that incorporating defocus cues at test time enables Marigold to achieve zero-shot metric depth, which is competitive in quality with zero-shot metric depth foundation models by simply using an additional blurred image as input along with the all-in-focus (AIF) image. We agree that this does make our method multishot and have presented comparisons with relevant baselines below, as we believe that contextualizing our performance to multi-shot methods is valuable. However, we emphasize that we don't claim to be the best multi-shot/defocus base method. While these additional comparisons are valuable, they do not affect the claims (L:34) and the demonstrated validation in our paper. \n\nWhile our work indeed falls in the category of monocular metric depth estimation with more than 1 image, we are different from DFD-based methods, since we use 2 images with different apertures, but the same focus distance. This is different from the majority of existing DfD methods that require at least 4-5 images captured at different focus distances [H, A]. We attempted to adapt DEReD [A], a SOTA self-supervised learning based DFD baseline, but observed that it performs very poorly and fails to converge when given only an AIF and a blurred image as input, as recovering metric depth from 2 images is a highly ill-posed problem, unless very strong priors or really large datasets are used – neither of which has been used in most DfD works. We also discuss this in more detail in the response to MnCO. \n\nOur setup is more similar to [B], which also relies on two images captured at the same focus distance: an AIF and a blurred image as inputs. They propose a differentiable model that takes an AIF and simulates defocus blur by interpolating pre-blurred AIFs at discrete depth levels, with weights predicted by a CNN that implicitly encodes scene depth. The network is trained to match the synthesized image to the captured (ground truth) blurred image.\n\nWe implemented their method, and also a depth-supervision-based baseline mentioned in their work, which involves directly regressing metric depth from a network with the AIF image stack as input. Following their training-based framework, we trained and validated these baselines on NYU-v2 [E], a popular indoor RGBD benchmark. We observe in Table 2 that the methods show poor performance metrics on the NYU-v2 test set. We observe that the Depth supervision baseline produces blurry results, as observed in [B], and the method proposed in [B] struggles with strong texture-depth coupling, resulting in inaccurate metric depth.\n| Method | REL $\\downarrow$  | $RMSE \\downarrow$   | $\\log10\\downarrow$  | δ1 $\\uparrow$   | δ2  $\\uparrow$ | δ3 $\\uparrow$  |\n|------------------|-------|-------|-------|------|------|------|\n| Ours             | 0.150 | 0.535 | 0.045 | 0.899| 0.949| 0.969 |\n| MultiAperture Depth Supervision| 0.394  | 1.2667| 0.1626 | 0.3815| 0.6668| 0.8401|\n| [B] | 0.5221  | 1.7415| 0.2571 | 0.2411| 0.4592| 0.6369|\n**Table 2: Results on NYUv2 dataset for Our method and Multi-aperture stack (Camera Physics) based deep learning approaches**. We outperform previous multi-aperture methods. All methods use an AIF and blurred image as input. We omit Chamfer Distance and FA for baselines due to a large gap between them and our method on the other 3D metrics.\n\n*Repeating Fig 6*.      \nIn Fig. 6, we used a single blurred image as input to Marigold, not as a baseline but just to check if a single image is sufficient for metric depth estimation (RMSE: 1.36).  However, as suggested, we used 2 blurred images (f/16 and f/8) as input, with f/16 as the proxy to AIF (RMSE: 0.6). While better than using a single blurred image, F/16 is a less accurate proxy for our AIF (f/22 vs f/8), which leads to a higher error (RMSE: 0.341). This highlights the importance of a high-quality AIF and shows that performance degrades gracefully with increased blur in the AIF.\n\n\n**Weakness 2: Limited Practicality**.     \n*Applying our method to GeoWizard*.                     \nWe agree with the reviewer and show results below from applying our method with the Geowizard model (Table 3), demonstrating the plug-and-play nature of our method. GeoWizard is similar to Marigold but trained on a larger and more complex data distribution by fine-tuning the stable diffusion UNet. We observe that after the inference time optimization, Geowizard relative depths exhibit minor patch-level artifacts, potentially due to stronger texture-depth coupling. We will add these numbers and visualizations in a revision.\n\n| Method |  $RMSE \\downarrow$ | $REL \\downarrow$ | $log10 \\downarrow$ | δ1 $\\uparrow$ | δ2 $\\uparrow$ | δ3 $\\uparrow$  | CD $\\downarrow$ | FA $\\uparrow$ |\n|--------------------|-------|-------|-------|-------|-------|-------|-------|--------|\n| Ours  w Marigold | 0.273 | 0.125 | 0.052 | 0.879 | 0.975 | 0.991 | 0.103 | 0.870  |\n| Ours w GeoWizard   | 0.291 | 0.137 | 0.061 | 0.824 | 0.966 | 0.990 | 0.105 | 0.874  |\n**Table 3: Results on our real dataset using our method with Marigold (row 1) and GeoWizard [F] (row 2) as the diffusion prior, showing comparable performance.**\n\n*Computational Latency*.            \nWe agree that our method is slower than feedforward depth models; however, this trade-off does not diminish our core contribution (L:34-39). We are similar in runtime to other diffusion-based inference-time optimization methods [I], and future advances in this space will also benefit our runtime.\n\n**Weakness 3: Justification for defocus blur**.        \nDefocus blur provides metric depth cues and is readily available in images as it's a fundamental property of the camera lens. Compared to other cues, such as stereo (2 cameras), defocus blur allows metric depth estimation with a single camera, and specifically without requiring calibration for the camera extrinsics, which is required for stereo. Moreover, while stereo cameras are widely used, their applicability is limited in compact settings like endoscopy. Other camera physics methods, such as coded apertures, require manufacturing custom hardware. In comparison, defocus cues are more accessible, as they are present in images captured with any standard camera lens. While Metric3D resizes images based on camera focal lengths to ensure stable training over datasets of different depth scales, it doesn’t provide a physically grounded depth cue like defocus blur, and thus cannot be leveraged for per-scene inference time optimization. \n\n*Unique Challenges and benefits of Marigold + defocus blur*.     \nDefocus blur offers a differentiable forward model using an AIF and depth map, making it suitable for integration with Marigold at test time. To address mismatches with real cameras (L234, Fig. 2), we adopt a more accurate disk kernel over a Gaussian. While this model simplifies gradient computation, it ignores occlusions, leading to minor errors at depth discontinuities.\n\n**Weakness 4: Marigold Specific Advantages:**.    \n1. Unlike feedforward methods such as DepthAnythingv2 or UniDepth, diffusion-based methods like Marigold provide access to a low-dimensional latent manifold that maps from a noise vector to a relative depth map differentiably. This enables test-time refinement of Marigold’s predictions by incorporating physical cues like defocus blur through differentiable forward models, as shown in Fig. 3.\n2. Marigold is obtained by fine-tuning StableDiffusion [G] (trained on internet-scale RGB data), allowing it to inherit strong real-world generalization capabilities from StableDiffusion. Most other feedforward models are trained only on RGBD datasets, which are relatively smaller compared to the stable diffusion training dataset.\n3. Marigold also retains the well-known benefits of diffusion-based depth predictors, such as better fine detail recovery [F] compared to feedforward methods. Additionally, Marigold also provides a latent consistency checkpoint, which speeds up inference compared to using Geowizard.\n\n[A] Depth Estimation via Reconstructing Defocus Image: CVPR 2023.      \n[B] Aperture Supervision for Monocular Depth Estimation: CVPR 2018.     \n[C] High-Accuracy Stereo Depth Maps Using Structured Light: CVPR 2003.     \n[D] Passive snapshot coded aperture dual-pixel RGB-D imaging: CVPR 2024.     \n[E] NYU Depth Dataset V2: ECCV 2012.     \n[F] GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image: ECCV 2024.     \n[G] SDXL: ICLR 2024.    \n[H] Bridging Unsupervised and Supervised Depth From Focus via All-in-Focus Supervision: ICCV 2021.     \n[I] DITTO: Diffusion Inference-Time T-Optimization for Music Generation : ICML 2024"}, {"review_id": "zc1hhBnAyd", "reviewer": "Reviewer_QwFg", "summary": "This paper proposes to adapt a pre-trained affine-invariant depth model (Marigold) for metric depth estimation by utilizing defocus blur cues. Based on the defocus blur image formation model, the proposed approach designs a test-time optimization method that simultaneously improves the depth map and estimates the scale and shift for metric depth conversion. Specifically, two images (one all-in-focus and one with defocus blur) are first captured at the same viewpoint. Then, a synthesized blurry image will be generated based on the depth estimated from the all-in-focus image. Finally, the depth estimation result can be optimized by minimizing the distance between the captured defocus image and the synthesized one. Despite some prior information being needed, e.g., focal length, focus distance, and F-stop, the proposed method achieves promising performance without re-training the affine-invariant model. Extensive experiments on self-captured datasets demonstrate the superiority and effectiveness of the proposed method, especially in addressing texture-depth coupling and inaccurate depth scaling.", "questions": "This paper proposes an interesting idea to convert affine-invariant depth to metric depth via defocus blur cues. Although some concerns exist in terms of the limited application and the adaptation capability (please see the above weaknesses section for more details), the strengths of the proposed method outweigh its weaknesses, and thus my initial rating is Accept.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n- This paper proposes a test-time optimization method to achieve relative-to-metric depth conversion via defocus blur cues, which is a novel approach to me. One benefit of such a design is that the learned geometry prior in the pre-trained depth model is well preserved, as no re-training is needed. In addition, it's interesting to see that optimizing the noisy latents of diffusion models could help generate better depth estimation results.\n\n- Since the defocus blur cues provide reliable depth hints, the proposed method better handles the texture-depth decoupling, where high-contrast image textures often mislead the depth model to produce incorrect depth discontinuities. Besides, more accurate depth scaling parameters can be obtained by matching the synthesized defocus image with the captured one.\n \n- Extensive experiments are conducted to validate the effectiveness of the proposed method, and sufficient ablation studies as well as toy experiments are presented to support the design choices, such as the Disc point spread function.\n\n- The paper is well written, with a clear structure and smooth flow. The key ideas are easy to follow, and an appropriate level of detail is provided. For example, the illustrative figures, such as Fig. 3, are beneficial for the readers to understand the current limitations of existing methods and the advancements made by the proposed work.\n\n**Weaknesses**\n\n- As discussed in the limitation section, the proposed method may have a limited range of applications, as the defocus blur cues are more sensitive under small depth ranges. Apart from this perspective, the proposed setup could also limit its application. For instance, it's difficult to apply the proposed method on in-the-wild images (like the images collected from the Internet), as a defocus version is required for optimization. In addition, it might be challenging to handle dynamic scenes, as the captured two images might have different/unaligned contents.\n\n- I am wondering if it's possible to apply the proposed method to feed-forward depth models, such as Depth Anything [1]. Compared with the diffusion-based Marigold, Depth Anything shows better depth estimation accuracy [2], and thus could potentially improve the performance of the proposed method. However, the current approach relies on optimizing the noisy latents, which don't exist in feed-forward methods like Depth Anything. It would be interesting to see a discussion about this point.\n\n- The self-curated dataset is relatively limited in diversity and size, where only 7 indoor scenes are used for evaluation. It would be better to collect more diverse scenes to support the robustness of the proposed method or to show the visual results of some failure cases, which could be helpful for follow-up work.\n\n\n**References**\n\n[1] Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR 2024.\n\n[2] Betterdepth: Plug-and-play diffusion refiner for zero-shot monocular depth estimation. In NeurIPS 2024.", "was_revised": true, "final_justification": "This paper proposes to achieve metric depth estimation via a test-time optimization approach, which provides an alternative solution to popular learning-based methods and could be adopted for more robust depth estimation. Extensive experimental results validate the effectiveness of the proposed method. Besides, the additional results and discussions provided during the rebuttal strengthen the contribution of this manuscript. Therefore, I maintain my original rating and recommend acceptance of this paper.", "rebuttal": "We thank the reviewer for appreciating the novelty of our approach, the clarity of the paper, and our toy experiments and validation. We agree that handling dynamic scenes with the current approach is challenging and would be an interesting avenue for future work. We will expand our collected dataset to add more test scenes to facilitate future research in this area. \n\n**Discussion on adapting the method to feedforward depth models**.     \nOur method requires that the inference process of our depth model works by *differentiably* mapping from some latent embedding space to the output depth, such as the one implicitly provided by diffusion models from noise to data (GANs and VAEs could in principle similarly work). This enables inference time optimization of the latent vector at test time. However, discriminative or autoregressive transformer-based approaches, such as DepthAnything, don’t provide access to such a latent space. Adapting pre-trained autoregressive transformers to new datasets/domains typically involves test-time adaptation [A], which requires collecting a sizable dataset and then adding some extra layers, which are then fine-tuned with this dataset while keeping the rest of the weights frozen. Such an approach can be potentially used to extend methods like DepthAnythingv2 for handling multi-aperture input. But unlike our method, which is training-free, that would require fine-tuning / re-training the adaptation layers in the model. We are unaware of any large-scale datasets with multi-aperture defocus captures, but this approach might become viable in the future if such datasets emerge. We will add this discussion in a revision.\n\n**Real dataset size**.    \nOur motivation to collect this data was to create a validation set to verify our framework on real hardware. We agree that a larger dataset with depth maps, all in focus, and multiple aperture inputs would be useful for the community, and we plan to continue adding more examples to our released dataset. As suggested by other reviewers, we have also evaluated our method through simulated measurements on the NYUv2 dataset (654 test images), which consists of ground truth AIF and depth maps, and demonstrate comparable performance to SOTA metric depth foundation models. (Table 1 in MnCO rebuttal response).\n\n[A] Rapid network adaptation: ICCV 2023."}, {"review_id": "hNnkEEYfvE", "reviewer": "Reviewer_MnCo", "summary": "This paper proposes one method to extend relative depth to metric depth via defocus blur cues. Given an image pair x0 and blur x1, the base model predict relative depth d0 based on x0. Then, by optimizing though defocus blur (reconstructing x1 by x0 and d0 with the help of camera parameters), scale and shift factors are estimated and then used to transfer the relative depth to a metric one. Experiments show that this method achieves better metric depth compared with those metric depth estimators.", "questions": "Note that it is not necessary to reply my Weakness. All my questions are listed below.\n\nMy major concern is about the evaluation part. (1) There is no direct comparison with previous works using camera physics. (2) The authors are not using public datasets and currently the paper cannot demonstrate that this dataset is high-quality and suitable for this evaluation. Given this, my questions are:\n\n- It seems that most of previous works using camera physics cues train a deep-learning based network to digging these camera cues and predict metric depth. To me, it sounds like the authors take one step back and solve the same issue by optimizing per image via gradient descent. Could you explain why this is better than previous works and it would be good if you can support the claim with several experiments.\n\n- Could you please use a standard public dataset to do the evaluation? If not, could you please comment about the concern in weakness 3? If it's inevitable, I think using error map for visualization can be important. If the majority part of error locates around boundaries, it somehow indicates that the data is not that suitable.\n\nExtra questions\n\n- Is that necessary to adopt the Marigold? It seems to be a general method and showing its ability with various baselines can better demonstrate the effectiveness and make better impact.\n\n\nI'm not a researcher with enough experience of depth with camera physics. It's hard for me to judge the novelty of this paper, though the method itself is not technically sound. I'd like to check opinions from other reviewers. Currently, I give a borderline reject. but I'd like to change if my concern about evaluation is solved.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Strengths\n\n    - It's easy for me to follow this paper (good clarity in terms of reading and understanding).\n\n    - Given the fact that current metric depth estimator still fails to recover the metric scale properly in some cases, fully investigating depth cues serves a reasonable way to achieve better metric depth estimation.\n\n    - Experiments clearly demonstrate that the proposed methods are better than current SoTA depth estimators.\n\n- Weakness\n\n    - It seems that most of previous works using camera physics cues train a deep-learning based network to digging these camera cues and predict metric depth. To me, it sounds like the authors take one step back and solve the same issue by optimizing per image via gradient descent. Since there is no direct comparison with related works using camera physics cues, the results are less persuasive.\n\n    - The optimization stage leads to longer inference time. Though the authors already put this point in the limitation section, it is a serious shortcoming that cannot be ignored.\n\n    - The authors are not using some standard public datasets to conduct experiments. From several GT examples shown in the paper, the data quality is a bit low. The details are blur. This can lead to an fair comparison for metric models with good capability for sharp estimation.\n\n    - There is no error map for the visualization. It's hard to figure out where the error is from.", "was_revised": true, "final_justification": "Most of my concerns are addressed by the authors. Based on the rebuttal and discussion, I rise my score to the borderline accept.", "rebuttal": "We thank the reviewer for appreciating the clarity of our work and the motivation to explore other depth cues in conjunction with foundation model-based depth estimators. We want to emphasize that our work aims to demonstrate that incorporating camera physics, particularly defocus blur, into a data-driven Monocular Depth Estimation model at inference time can effectively address both texture coupling and incorrect metric scale estimation modes without requiring any re-training (L:34-39, main paper). We have validated both these claims through synthetic toy examples (Fig. 3) and also on real camera hardware data. We agree that our method is closely related to defocus blur-based multi-shot approaches, so comparing it with them does contextualize things better. To that end, we have added comparisons with relevant previous works below. However, we would like to reiterate that developing the best-performing single / multi-shot method was not our primary goal or claim, and that the results/validations presented in the manuscript adequately validate our claims (L:34-39). \n\n**Comparison with previous camera physics + deep learning based work**.    \nMost of the previous methods (L:98-L:110) train a deep learning network, but on relatively smaller datasets, while also requiring >=4 images as inputs. This severely limits generalizability and performance outside the training data distributions. While these methods [A, B] attempt to use priors such as total variation or task-specific loss functions while training the network, they are not strong enough to overcome the ill-posedness of the depth from defocus blur inverse problem with only aperture supervision and 2 images as input -- which is the regime our method operates in. We demonstrate this quantitatively in Table 1 below. \n\nAs suggested, we trained [B] and the direct depth supervision baseline mentioned in [B] with the same inputs as our method on the NYUv2 dataset [E], which is a well-known RGBD public benchmark for Monocular Depth Estimation. NYUv2 contains AIF images and GT depth pairs (654 test, 795 training). For our method, we synthesize the blurred input image at aperture (F/8) and focus distance (1.5m) using our forward model for a simulated comparison. \n\nThe results in Table 1 below show that our method achieves comparable performance with existing SOTA metric depth foundation model baselines (MLPro, UniDepth, Metric3D) reported in our work, while previous camera physics + deep learning methods (last 2 rows, Table 1) struggle in performance.\n\n| Method | REL $\\downarrow$  | $RMSE \\downarrow$   | $\\log10\\downarrow$  | δ1 $\\uparrow$   | δ2  $\\uparrow$ | δ3 $\\uparrow$  | CD $\\downarrow$    | FA  $\\uparrow$  |\n|------------------|-------|-------|-------|------|------|------|--------|-------|\n| Ours             | 0.150 | 0.535 | 0.045 | 0.899| 0.949| 0.969| 0.271  | 0.867 |\n| MLPro            | 0.115 | 0.507 | 0.054 | 0.880| 0.960| 0.980| 0.246  | 0.848 |\n| UniDepth         | 0.077 | 0.323 | 0.034 | 0.943| 0.988| 0.996| 0.161  | 0.887 |\n| Metric3D         | 0.106 | 0.423 | 0.045 | 0.900| 0.969| 0.987| 0.204  | 0.870 |\n| MultiAperture Depth Supervision| 0.5257  | 1.6789| 1.7322 | 0.3010| 0.5170| 0.6582| —| —|\n| [B] | 0.5221  | 1.7415| 0.2571 | 0.2411| 0.4592| 0.6369| —| —|\n**Table 1: Results on NYUv2 test set for Our method, Foundation Model baselines (MLPro, UniDepth, Metric3D), and Multi-aperture stack (Camera Physics) based deep learning approaches (last 2 rows)**. We achieve comparable performance to SOTA foundation model baselines (UniDepth, Metric3D, MLPro) across all metrics, while previous methods (bottom 2 rows) perform poorly. The metrics are computed by directly comparing each method's metric depth output with the NYUv2 ground truth.    \n\nWe would like to emphasize that the leading methods above in Table 1 are trained on Millions of Image-depth pairs to predict metric depth (Metric3D: 8M, UniDepth: 3M), while our method performs comparably to them *without any re-training* using defocus cues with a scale-invariant monocular depth estimator (Marigold). \n\nWhile our method does use gradient-based optimization, we would like to remind the reviewer that this gradient-based optimization happens in the latent space of Marigold, which lets us leverage internet-data scale priors in Marigold that are strong enough to achieve competitive results for metric depth estimation, without re-training, using only 2 input images. Previous approaches struggle in this scenario, as shown in Table 1( last 2 rows) above. Our work aims to demonstrate (L:34) that incorporating physics cues with a pre-trained foundation model can be a powerful unlock for tasks that it hasn't been explicitly trained on, but lie in the vicinity of the base task. The current evidence presented in our paper validates this claim. \n\n**GT Data Quality**.   \nAcquiring very high-quality depth maps from real hardware is non-trivial, and often requires complex structured light setups [C]. While we acknowledge that RealSense is not as precise as LiDAR or structured-light scanners, it is a much cheaper, hence academically viable alternative. It provides dense, metrically accurate depth within its operating range (0.2–3.8 m) using stereo cues, and has been used as ground truth in prior monocular depth from defocus work [D]. We clarify that our goal is not to achieve sub-millimeter accurate ground truth, but rather to evaluate relative and metric consistency in practical indoor scenarios. RealSense has a much larger field of view compared to the DSLR camera used to capture the all-in-focus (AIF) image. As a result, while plotting in 2D, the details in the depth map look slightly blurry. As suggested by the reviewer, we have visualized the error maps for our method and MLPro against the ground truth. We will add these visualizations to the camera-ready / appendix, but the observations from the visualizations are as follows: The visualizations show that MLPro (which has sharp boundaries in its predictions) struggles to predict correct depth scales for some areas in the scene, resulting in significant errors on planar regions. Our visualizations indicate that there is no particular bias towards high error on sharp boundaries in our collected data, making it suitable as ground truth for comparison. \n\n*Lack of real-world datasets with multiaperture inputs*.      \nOur method requires an AIF and a blurred image as inputs, and a ground truth depth map for evaluation. Most public RGBD datasets only contain the RGB image and ground truth depth map, but not the blurred image. So it's hard to evaluate our method on those. However, as suggested by the reviewer, we have also evaluated our method and other baselines from our work on the test set (654 images) of the well-established NYUv2 test set (see Table 1 above). \n\n**Is Marigold Necessary? Extending to other methods**.   \nOur method is generally applicable to any diffusion-based model like Marigold, since this provides access to a differentiable mapping from the diffusion model latent space to our depth estimates and thus allows for optimizing these latents at test time, which is required to incorporate defocus cues without re-training the base model. Additionally, our choice of Marigold was also motivated by the fact that the Marigold authors released a Latent Consistency Model (LCM) version of Marigold, which allows sampling the diffusion model in a single step, which is beneficial for faster inference. However, our method is plug-and-play and can be, in principle, applied to any diffusion-based estimator, which provides access to the latent space of the model. We demonstrate this by extending our method to Geowizard as well. We are referencing Table 3 here from our response to 1d2m. \n\n| Method          |  $RMSE \\downarrow$ | $REL \\downarrow$   | $\\log10 \\downarrow$ | δ1 $\\uparrow$    | δ2 $\\uparrow$    | δ3 $\\uparrow$    | CD $\\downarrow$    | FA $\\uparrow$     |\n|--------------------|-------|-------|-------|-------|-------|-------|-------|--------|\n| Ours  w Marigold | 0.273 | 0.125 | 0.052 | 0.879 | 0.975 | 0.991 | 0.103 | 0.870  |\n| Ours w GeoWizard   | 0.291 | 0.137 | 0.061 | 0.824 | 0.966 | 0.990 | 0.105 | 0.874  |\n**Results with Our Method (using Marigold), and Our Method applied to the Geowizard model on our self-collected real dataset**. Our method produces similar results on using either Marigold or GeoWizard as the pre-trained diffusion model. \n\nGeoWizard [F] is a diffusion-based monocular depth + normal predictor, trained on a larger and more complex data distribution by fine-tuning the stable diffusion UNet. We observe that the relative depth from geowizard after inference time optimization exhibits minor patch-level artifacts, potentially due to stronger texture-depth coupling. Unlike Marigold, Geowizard does not provide a latent consistency model (LCM) checkpoint; hence, it requires more sampling steps. The metrics for Geowizard obtained in Table 2 are with 50 optimization iterations, and 5 sampling steps for Geowizard per iteration. Geowizard is 4x slower than the Marigold-LCM checkpoint on an A-40 GPU, but still achieves comparable results. This highlights the flexibility of our approach: as stronger or faster diffusion models become available, our method can seamlessly leverage them for improved performance. \n\nWe will add these numbers, visualization, and discussion in a revision. \n\n[A] Depth Estimation via Reconstructing Defocus Image: CVPR 2023.            \n[B] Aperture Supervision for Monocular Depth Estimation: CVPR 2018.      \n[C] High-Accuracy Stereo Depth Maps Using Structured Light: CVPR 2003.      \n[D] Passive snapshot coded aperture dual-pixel RGB-D imaging: CVPR 2024.     \n[E] NYU Depth Dataset V2: ECCV 2012.   \n[F] GeoWizard: Unleashing the Diffusion Priors for 3D Geometry Estimation from a Single Image: ECCV 2024."}]

Author rebuttal:

Meta-review: The paper proposes a training-free way to estimate metric depth from a pair of images taken from the same viewpoint with different apertures. To that end, a diffusion-based monocular depth estimator (Marigold) is combined with a physics-based defocus model. The latter enables an upgrade from relative to metric depth via test-time optimisation. The initial reviews were mixed, in particular there were concerns about generalisation (particularly outdoor settings), about the evaluation protocol (one vs. two input views, alignment to ground truth, known vs. estimated  focal length), and about the defocus mechanism (refocussing vs. aperture). During the extensive discussion, the authors were able to dispel those concerns and convince the reviewers, who explicitly acknowledged the candid and helpful responses and converged to a unanimously positive recommendation (2x accept, 2x borderline accept).The AC agrees.
Decision: Accept (spotlight)