Title: Repurposing Marigold for Zero-Shot Metric Depth Estimation via Defocus Blur Cues
Venue: NeurIPS 2025
Accepted: yes
Keywords: Monocular Depth

Defocus Blur

Diffusion Models

Zero Shot Depth

Inference Time Optimization
Abstract: Recent monocular metric depth estimation (MMDE) methods have made notable progress towards zero-shot generalization. However, they still exhibit a significant performance drop on out-of-distribution datasets. We address this limitation by injecting defocus blur cues at inference time into Marigold, a \textit{pre-trained} diffusion model for zero-shot, scale-invariant monocular depth estimation (MDE). Our method effectively turns Marigold into a metric depth predictor in a training-free manner. To incorporate defocus cues, we capture two images with a small and a large aperture from the same viewpoint. To recover metric depth, we then optimize the metric depth scaling parameters and the noise latents of Marigold at inference time using gradients from a loss function based on the defocus-blur image formation model. We compare our method against existing state-of-the-art zero-shot MMDE methods on a self-collected real dataset, showing quantitative and qualitative improvements.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes one method to extend relative depth to metric depth via defocus blur cues. Given an image pair x0 and blur x1, the base model predict relative depth d0 based on x0. Then, by optimizing though defocus blur (reconstructing x1 by x0 and d0 with the help of camera parameters), scale and shift factors are estimated and then used to transfer the relative depth to a metric one. Experiments show that this method achieves better metric depth compared with those metric depth estimators.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n\n    - It's easy for me to follow this paper (good clarity in terms of reading and understanding).\n\n    - Given the fact that current metric depth estimator still fails to recover the metric scale properly in some cases, fully investigating depth cues serves a reasonable way to achieve better metric depth estimation.\n\n    - Experiments clearly demonstrate that the proposed methods are better than current SoTA depth estimators.\n\n- Weakness\n\n    - It seems that most of previous works using camera physics cues train a deep-learning based network to digging these camera cues and predict metric depth. To me, it sounds like the authors take one step back and solve the same issue by optimizing per image via gradient descent. Since there is no direct comparison with related works using camera physics cues, the results are less persuasive.\n\n    - The optimization stage leads to longer inference time. Though the authors already put this point in the limitation section, it is a serious shortcoming that cannot be ignored.\n\n    - The authors are not using some standard public datasets to conduct experiments. From several GT examples shown in the paper, the data quality is a bit low. The details are blur. This can lead to an fair comparison for metric models with good capability for sharp estimation.\n\n    - There is no error map for the visualization. It's hard to figure out where the error is from.\n\nQUESTIONS:\nNote that it is not necessary to reply my Weakness. All my questions are listed below.\n\nMy major concern is about the evaluation part. (1) There is no direct comparison with previous works using camera physics. (2) The authors are not using public datasets and currently the paper cannot demonstrate that this dataset is high-quality and suitable for this evaluation. Given this, my questions are:\n\n- It seems that most of previous works using camera physics cues train a deep-learning based network to digging these camera cues and predict metric depth. To me, it sounds like the authors take one step back and solve the same issue by optimizing per image via gradient descent. Could you explain why this is better than previous works and it would be good if you can support the claim with several experiments.\n\n- Could you please use a standard public dataset to do the evaluation? If not, could you please comment about the concern in weakness 3? If it's inevitable, I think using error map for visualization can be important. If the majority part of error locates around boundaries, it somehow indicates that the data is not that suitable.\n\nExtra questions\n\n- Is that necessary to adopt the Marigold? It seems to be a general method and showing its ability with various baselines can better demonstrate the effectiveness and make better impact.\n\n\nI'm not a researcher with enough experience of depth with camera physics. It's hard for me to judge the novelty of this paper, though the method itself is not technically sound. I'd like to check opinions from other reviewers. Currently, I give a borderline reject. but I'd like to change if my concern about evaluation is solved.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to adapt a pre-trained affine-invariant depth model (Marigold) for metric depth estimation by utilizing defocus blur cues. Based on the defocus blur image formation model, the proposed approach designs a test-time optimization method that simultaneously improves the depth map and estimates the scale and shift for metric depth conversion. Specifically, two images (one all-in-focus and one with defocus blur) are first captured at the same viewpoint. Then, a synthesized blurry image will be generated based on the depth estimated from the all-in-focus image. Finally, the depth estimation result can be optimized by minimizing the distance between the captured defocus image and the synthesized one. Despite some prior information being needed, e.g., focal length, focus distance, and F-stop, the proposed method achieves promising performance without re-training the affine-invariant model. Extensive experiments on self-captured datasets demonstrate the superiority and effectiveness of the proposed method, especially in addressing texture-depth coupling and inaccurate depth scaling.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- This paper proposes a test-time optimization method to achieve relative-to-metric depth conversion via defocus blur cues, which is a novel approach to me. One benefit of such a design is that the learned geometry prior in the pre-trained depth model is well preserved, as no re-training is needed. In addition, it's interesting to see that optimizing the noisy latents of diffusion models could help generate better depth estimation results.\n\n- Since the defocus blur cues provide reliable depth hints, the proposed method better handles the texture-depth decoupling, where high-contrast image textures often mislead the depth model to produce incorrect depth discontinuities. Besides, more accurate depth scaling parameters can be obtained by matching the synthesized defocus image with the captured one.\n \n- Extensive experiments are conducted to validate the effectiveness of the proposed method, and sufficient ablation studies as well as toy experiments are presented to support the design choices, such as the Disc point spread function.\n\n- The paper is well written, with a clear structure and smooth flow. The key ideas are easy to follow, and an appropriate level of detail is provided. For example, the illustrative figures, such as Fig. 3, are beneficial for the readers to understand the current limitations of existing methods and the advancements made by the proposed work.\n\n**Weaknesses**\n\n- As discussed in the limitation section, the proposed method may have a limited range of applications, as the defocus blur cues are more sensitive under small depth ranges. Apart from this perspective, the proposed setup could also limit its application. For instance, it's difficult to apply the proposed method on in-the-wild images (like the images collected from the Internet), as a defocus version is required for optimization. In addition, it might be challenging to handle dynamic scenes, as the captured two images might have different/unaligned contents.\n\n- I am wondering if it's possible to apply the proposed method to feed-forward depth models, such as Depth Anything [1]. Compared with the diffusion-based Marigold, Depth Anything shows better depth estimation accuracy [2], and thus could potentially improve the performance of the proposed method. However, the current approach relies on optimizing the noisy latents, which don't exist in feed-forward methods like Depth Anything. It would be interesting to see a discussion about this point.\n\n- The self-curated dataset is relatively limited in diversity and size, where only 7 indoor scenes are used for evaluation. It would be better to collect more diverse scenes to support the robustness of the proposed method or to show the visual results of some failure cases, which could be helpful for follow-up work.\n\n\n**References**\n\n[1] Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR 2024.\n\n[2] Betterdepth: Plug-and-play diffusion refiner for zero-shot monocular depth estimation. In NeurIPS 2024.\n\nQUESTIONS:\nThis paper proposes an interesting idea to convert affine-invariant depth to metric depth via defocus blur cues. Although some concerns exist in terms of the limited application and the adaptation capability (please see the above weaknesses section for more details), the strengths of the proposed method outweigh its weaknesses, and thus my initial rating is Accept.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel training-free framework for zero-shot metric depth estimation. It achieves this by intelligently combining a pre-trained monocular depth estimator based on diffusion models (Marigold) with explicit physical cues derived from defocus blur. The system requires two images captured from the same viewpoint but with different aperture settings (an all-in-focus image and a blurred image). Marigold is initially used to estimate a relative depth map from the all-in-focus image. This relative depth is then converted to metric scale through an inference-time optimization process, where the depth parameters are iteratively refined to ensure consistency between the estimated depth, a physically-based defocus model, and the input blurred image. This approach offers a compelling solution to data scarcity in metric depth estimation and demonstrates promising zero-shot generalization capabilities.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* **Innovative Training-Free Metric Depth Estimation:** The paper proposes a novel framework for zero-shot metric depth by uniquely combining a powerful pre-trained diffusion model (Marigold) with explicit defocus blur cues. This approach circumvents the need for large, labeled metric depth datasets, promoting robust generalization.\n* **Effective Repurposing of a Foundation Model:** The work intelligently leverages Marigold, a state-of-the-art diffusion-based monocular depth estimator. Marigold provides a strong initial relative depth prior due to its extensive pre-training, which is crucial for the subsequent physically-informed metric scale refinement.\n* **Principled Integration of Defocus Blur:** The use of defocus blur is a theoretically sound choice for recovering metric scale. Its well-defined physical relationship to depth allows for a robust gradient-based optimization to refine Marigold's relative depth output to metric units.\n\n### Weaknesses\n\n* **Misleading SOTA Comparison and Input Modality:** The comparison to strictly monocular metric depth estimation (MMDE) methods is not entirely fair. The proposed system requires two images (all-in-focus and blurred), placing it in the domain of multi-image depth estimation (Depth from Defocus, DFD), not purely monocular. One of the reasons why monocular doesn't perform as well as multi-view or focal stack is the lack of physical reference from multiple images. More appropriate comparisons would be against other DFD or focal-stack based approaches but restrict them with a stack number of two.\n* **Significant Computational Latency and Limited Practicality:** The reported inference speed of 3.5-4 minutes per image pair on an NVIDIA A40 GPU is a major practical disadvantage due to its per-image, inference-time optimization. This high latency severely restricts applicability in real-time or high-throughput scenarios. Another limited practicality is that since this paper proposed \"training-free\" but and inference time optimization for affine-transform, I believe to demonstrate the effectiveness and practicality, it should be tested on several SOTA foundation relative depth methods as a plug-and-play method, instead of purely tight to Marigold.\n* **Lack of Clear Justification for Defocus Blur Choice:** The paper doesn't thoroughly justify why defocus blur was chosen over other potential metric depth cues (e.g., stereo priors). A brief discussion on the unique advantages of DFD in this training-free context, compared to the challenges of integrating other cues, would strengthen the methodological choice.\n* **Limited Discussion on Marigold's Specific Advantages:** While Marigold provides a strong prior, the paper could elaborate more on *why* a diffusion-based model like Marigold is particularly suited for this DFD integration compared to other state-of-the-art monocular depth estimators.\n\nQUESTIONS:\n### Questions for Authors\n\n* **Regarding Weakness 1 (Misleading SOTA Comparison and Input Modality):**\n    * Could the authors provide a more direct and fair quantitative comparison against state-of-the-art **Depth from Defocus (DFD)** methods or other approaches that utilize focal stacks/multi-aperture inputs? This would better contextualize the proposed method's performance and novelty within its true category.\n    * Regarding the ablation study in Ln. 259 and Fig. 6, the comparison between their loss optimization (with AIF + Blurred images) and Marigold taking only the blurred image seems inconsistent. Most depth models are trained on AIF or nearly AIF datasets. Providing Marigold with a blurred image, as opposed to the original AIF setting it was likely optimized for, may not be a fair baseline. Could the authors instead provide a comparison where: 1) Marigold is given the blurred image, and 2) their loss function method also uses *two blurred images* with different aperture sizes? This would offer a more robust demonstration of the method's tolerance and effectiveness when dealing with blurred inputs.\n\n* **Regarding Weakness 2 (Limited Practicality):**\n    * To further demonstrate the practicality and \"plug-and-play\" nature of this inference-time optimization, could the authors showcase its effectiveness when applied to other state-of-the-art foundation relative depth models, such as Depth Anything v2 or GeoWizard?\n\n* **Regarding Weakness 3 (Lack of Clear Justification for Defocus Blur Choice):**\n    * The paper primarily uses defocus blur as the metric depth cue. Could the authors elaborate on the specific advantages of defocus blur in this *training-free, zero-shot* context, especially when compared to other potential metric depth cues like stereo correspondence?\n    * What are the unique challenges and benefits of integrating a diffusion-model-based prior (Marigold) with defocus blur, as opposed to other types of physical cues?\n\n* **Regarding Weakness 4 (Limited Discussion on Marigold's Specific Advantages):**\n    * While Marigold provides a strong relative depth prior, could the authors delve deeper into *why* a diffusion-based model like Marigold is particularly suited for this DFD integration? Are there inherent properties of diffusion models' representations (e.g., robustness to noise, handling of fine details, ability to capture global scene structure) that make them uniquely advantageous for this task compared to other state-of-the-art monocular depth estimators?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel method for upgrading a relative depth map to metric via unsupervised test-time optimization that requires the existence of a second, defocused image of the scene besides the primary sharp image. A major contribution of the paper is the introduction of a rigorous, physics-based defocus blur forward model, which depends on the predicted metric depth map and allows to define a self-supervised loss based on consistency of the captured blurry image and the respective blurry image which is synthesized based on the all-in-focus image, the relative depth map, and the learned scale and shift. The authors validate their approach on a small real-world indoor dataset they have collected and demonstrate superior performance compared to state-of-the-art direct learned metric depth estimators on their dataset.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The usage of the central defocus cue for depth in a learned end-to-end framework via test-time optimization is innovative and insightful and it provides an interesting alternative to direct learned models for metric depth estimation.\n\n2. The derivation of the complete model, especially the defocus blur forward model, is mathematically solid, thorough, and well-grounded on optics.\n\n3. The experimental results on a small yet real-world dataset are very satisfactory, as the proposed method outperforms existing state-of-the-art direct learned metric depth estimators substantially on this dataset across all evaluation measures.\n\nMajor Weaknesses:\n\n1. The applicability of the proposed method is quite limited. As the authors already acknowledge, the employed forward model limits the method to indoor scenes with restricted depth ranges. Moreover, the need to capture two different images with different apertures limits the method to specialized camera types such as DSLR, apparently making it inapplicable to e.g. phone cameras or other commodity cameras. At the same time, the method cannot work for dynamic scenes including even slow object motions, as the two images - the all-in-focus and the defocused one - need to be captured in non-overlapping temporal exposure intervals. Thus, the generality and universality of the method is hurt.\n\n2. Metric depth on its own right is not sufficient for metric estimation, as one also needs the directions of the rays corresponding to the pixel grid in order to obtain a metric 3D estimation of the scene structure. Camera intrinsics thus need to be known or estimated. Supposing that the proposed method were applied to an in-the-wild setting with unknown intrinsic calibration, its output is thus insufficient to achieve the aforementioned goal. While the majority of the monocular metric depth methods assume known camera parameters, there are a few that do not [39, A] and that can estimate both camera intrinsics and scene range/depth. Moreover, the quality of metric estimation needs to be evaluated with 3D-wise metrics, such as the Chamfer distance and the F-score. The restriction to mere metric depth maps living in the 2D image space and to traditional depth metrics limits the practical relevance of the method.\n\n3. The authors state in L. 239-240 that they align the predicted depth map with the ground truth depth map before performing the evaluation. What exactly does this alignment entail? Is it applied for competing methods as well? I note here that a major challenge and desideratum in metric depth estimation is to lift the need for aligning / rescaling the model's output in any way and to rather obtain directly a metrically accurate prediction. This is already satisfied in the standard evaluation of recent metric depth estimation works [39, A]. Can the authors present comparisons without the aforementioned alignment?\n\nMinor Weaknesses:\n\n4. It is unclear what the exact optimization algorithm used at inference (cf. L. 212) is. Is it simple gradient descent? Or does it use conjugate gradients? Or is it of another type? What step sizes are used across the 200 iterations? The authors should clarify these points.\n\n5. The exact dependence of the range of the output depth with the scale and shift parameters $\\alpha$ and $\\beta$ is unclear to me. Based on Eq. (9), since the minimal values of the sigmoids are 0, the overall output depth value could be as low as 0. However, the authors state in L. 186 that the lower bound of depth is $s_{\\text{min}}$, not 0. Moreover, in Fig. 5 (right) and L. 266-269, the scale and shift parameters $\\alpha$ and $\\beta$ seem to be defined to range between 0 and 1, while based on Eq. (9) they should range between 0 and $s_{\\text{max}}$ and $s_{\\text{min}}$, respectively. Which of the above is true?\n\n[A] Piccinelli et al. UniK3D: Universal Camera Monocular 3D Estimation. In CVPR, 2025.\n\nQUESTIONS:\nPlease refer to the questions which are included in the lists of Major Weaknesses and Minor Weaknesses in my response above.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}