Jasmine: Harnessing Diffusion Prior for Self-supervised Depth Estimation
Authors: JiYuan Wang
Chunyu Lin
cheng guan
Lang Nie
Jing He
Haodong Li
Kang Liao
Yao Zhao
Venue: NeurIPS 2025 poster ()
Keywords: depth estimation
self-supervision
diffusion models
generative model
Primary area: applications

Abstract: In this paper, we propose \textbf{Jasmine}, the first Stable Diffusion (SD)-based self-supervised framework for monocular depth estimation, which effectively harnesses SD’s visual priors to enhance the sharpness and generalization of unsupervised prediction. Previous SD-based methods are all supervised since adapting diffusion models for dense prediction requires high-precision supervision. In contrast, self-supervised reprojection suffers from inherent challenges (\textit{e.g.}, occlusions, texture-less regions, illumination variance), and the predictions exhibit blurs and artifacts that severely compromise SD's latent priors. To resolve this, we construct a novel surrogate task of mix-batch image reconstruction. Without any additional supervision, it preserves the detail priors of SD models by reconstructing the images themselves while preventing depth estimation from degradation. Furthermore, to address the inherent misalignment between SD's scale and shift invariant estimation and self-supervised scale-invariant depth estimation, we build the Scale-Shift GRU. It not only bridges this distribution gap but also isolates the fine-grained texture of SD output against the interference of reprojection loss. Extensive experiments demonstrate that Jasmine achieves SoTA performance on the KITTI benchmark and exhibits superior zero-shot generalization across multiple datasets.

Jasmine: Harnessing Diffusion Prior for Self-Supervised Depth Estimation
Jiyuan Wang!
Chunyu Lin Haodong Li
Cheng Guan
Lang Nie Yao Zhao
Jing He
Kang Liao
BJTUU
NTU
HKUST
*CQUPT
Corresponding Author
Row: Muti-Sceme
21 Row: Mono
VIFT IIT (ECCV2024)
Row: Jasmine (Ours)
Figure I:
Without any high-precision depth supervision, Jasmine achieves remarkably detailed and accurate depth estimation results through zero-shot generalization across diverse scenarios.
Abstract
In this paper, we propose Jasmine, the first Stable Diffusion (SD)-based self- supervised framework for monocular depth estimation, which effectively harnesses SD's visual priors to enhance the sharpness and generalization of unsupervised prediction. Previous SD-based methods are all supervised since adapting diffusion models for dense prediction requires high-precision supervision. In contrast, self- supervised reprojection suffers from inherent challenges (e.g., occlusions, texture- less regions, illumination variance), and the predictions exhibit blurs and artifacts that severely compromise SD's latent priors. To resolve this, we construct a novel surrogate task of mix-batch image reconstruction. Without any additional supervision, it preserves the detail priors SD models by reconstructing the images themselves while preventing depth estimation from degradation. Furthermore, to address the inherent misalignment between SD's scale and shift invariant estimation and self-supervised scale-invariant depth estimation, we build the Scale-Shift GRU. It not only bridges this distribution gap also isolates the fine-grained texture of SD output against the interference of reprojection loss. Extensive experiments demonstrate that Jasmine achieves SoTA performance on the KITTI benchmark and exhibits superior zero-shot generalization across multiple datasets. Project page and code are available at here.
Introduction
Estimating depth from monocular images is fundamental problem in computer vision, which plays an essential role in various downstream applications such as 3D/4D reconstruction162, 61], autonomous driving[ 10], etc. Compared with supervised methods/25, 33, 21], self-supervised monocular depth estimation (SSMDE) mines 3D information solely from video sequences, significantly reducing reliance on expensive ground-truth truth depth annotations. These methods derive supervision from geometric constraints (e.g., scene depth consistency) through cross-frame reprojection loss, and the ubiquitous video data further suggests an unlimited working potential. However, view reconstruction- based losses suffer from occlusions, texture-less regions, and illumination changes[83], which
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

severely restrict the model's capacity to recover fine-graing details and may cause pathological overfitting to specific datasets.
Recent studies(]]] demonstrated that SD possesses powerful visual priors to elevate depth prediction sharpness and generalization, which offers promising potential to address the above limitations. In addition, E2E FT[33] and Lotus[21] further reveal that single-step denoising can achieve better accuracy, which is particularly critical for self-supervised paradigms. It not only accelerates the inference denoising process but also significantly reduces the training costs in sell-reprojection supervision, thereby creating opportunities to integrate SD into the SSMDE framework.
However, linettuning diffusion models for dense prediction requires sesh-precision supervision to preserve their inherent priors[25]. Supervised methods typically employ synthetic RGB-D datasets, where clean depth annotations align with the high-quality SD's training data, thereby keeping its latent space intact. In contrast, directly applying self-supervision introduces critical challenge: reprojection losses or pre-trained depth pseudo-labels propagate perturbed gradients caused by artifacts and blurs into SD's latent space, rapidly corrupting its priors during early training stages. Namely, high-precision "supervision" must exist at the beginning to protect SD's latent space. Such supervision seems impossible in self-supervised learning, but we find handy and valuable alternative: the RGB image. In fact, the image inherently contains complete visual details, avoids external depth dependencies in self- superviuerisi and aligns perfectly with SD's original objective of image generation. Therefore, we construct a surrogate task of mix-batch image reconstruction (MIR) through task switcher, where the same SD model alternately reconstructs synthesized/real images and predicts depth maps within each training batch. This strategy repurposes self-supervised reprojection loss to tolerate color variations while maintaining structural consistency[ [ 4], intentionally decoupling color fidelity from depth accuracy, and finally preserving SD priors successfully.
Another challenge is the output range of SD's VAE[27] that is inherently bounded within a fixed range, i.e., [-1, IJ. Existing methods typically normalize GT depth maps to this range in the training procedure. During inference, these supervised approaches perform least-squares alignment to recover absolute scale and shift, yielding scale- and shift- invariant (SSI) depth predictions. However, self- supervised frameworks rely on coupled depth-pose optimization, which theoretically requires shift invariance to be strictly zero for stable convergence, ultimately producing predictions. To bridge this inherent distribution gap, we propose a gated recurrent unit (GRU)- novel transform module termed Scale-Shift GRU (SSG). It not only iteratively aligns SSI depth to SI depth by refining scale-shift parameters but also acts as a gradient filter, which suppresses anomalous gradients caused by artifact-onttamintted in self-supervised training, thereby preserving the fine-grained texture details of SD's output while enforcing geometric consistency.
(SI) depth
Extensive experiments show that our proposed method, Jasmine, achieves the SoTA performance among all SSMDEs on the competitive KITTI dataset, shows remarkable zero-shot generaliza- tion across multiple datasets (even surpassing models trained with augmented data), @ demonstrates unprecedented detail preservation. As the first work to bridge self-supervised and zero-shot depth estimation paradigms, we also provide an in-depth analysis of the effects of different depth de-normalization strategies employed in their respective domains. To sum up, the main contributions are summarized as follows:
We first introduce SD into a self-supervised depth estimation framework. Our methods eliminate the dependency on high- nigh-precision depth supervision while retaining SD's inherent advantages in detail sharpness and cross-domain generalization.
We proposed surrogate task of MIR that anchors SD's priors via self-supervised gradient sharing to avoid SD's latent-space corruption caused by reprojection artifacts.
We proposed Scale-Shift GRU (SSG) to dynamically align depth scales while filtering noisy gradients to solve SSI versus distribution mismatch problems in self-supervised depth estimation.
2 Relative Work
Self-supervised Depth Estimation Due to the high costs of GT-depth collection from LiDARs
and other sensors, self-supervised depth estimation (SSDE) has gained significant research attention. depth from synchronized image [52, 14, 12, 3, 2] and monocular-based methods (using sequential video frames) 87, 85, 32, 37, 83, 20, 34, 19, 29]. Additionally, when focusing on inference capability, existing methods can further diverge
SSDE can be broadly categorized into stereo-based(leamii

---

surrogate switch s single step
1.g
Mix-bateh Im
T
Im
warp
ph
Scale-Shift GRUx2
Denoiser U-Net
D
Noise
3
Dest
Dst
Dx L.e
Depth Prediction
Lie
Le
normalization
edge extraction
Lec
Training Image
Dic
Figure 2: Finetuning Protocol of Jasmine. The I, and In are each concatenated n[33] and fed into VAE encoder e. Next, the U-Net performs single-step denoising guided by the task switcher s, and subsequently decodes the SSI-depth prediction DSSı and the reconstructed image with the 'D (Sec. 3.2). Afterward, the Dssi is processed by the SSG for distribution refinement, yielding the final depth estimation DSI. The Ltes Le: Lph and L. are supervision loss and they are detailed in Sec. 3.4. The edge extraction module detailed Sec C
into single-frame frame and multi-frame approaches[55, 17, 9. 1]. Comparisons between these paradigms detailed in Sec. E.2. Recently, DepthAnything v1/v2[68. 69] has revealed that can obtain an accurate single image depth prediction model with strong generalization by training on large-scale image depth pairs. However, we argue that such datasets still remain small fraction the ubiquitous video data available. This observation motivates our exploration of the most challenging configuration: training exclusively on video sequences while maintaining mangle-frame inference capability, thereby laying the groundwork for developing genuinely versatile "DepthAnything" models.
Diffusion for Depth Perception As the diffusion paradigm showcases its talents in generative tasks|22, 46, 28, 76, 51, 591, DDP[24] first reformulates depth perception as a depth map denoising task and leads to giant progress. Followers like DDVM[40), MonoDiffusion[44
and D4RD[50] (the latter two are self-supervised methods employ self-designed diffusion) all demonstrate the advan- tages of this paradigm in various MDE sub-tasks. Subsequently. the most renowned diffusion model, Stable Diffusion[39] has demonstrated significant potential for depth perception tasks. VPD[84], TAPD[26]. and Prior-Diffusion(77 use Stable Diffusion as multi-modal feature extractor, leverag- ing textual modality information to improve depth estimation accuracy. Concurrently, Marigold[25] and GeoWizard]] enhanced model generalization and detail preservation by fine-tuning Stable Diffusion, capitalizing on its prior training with large-scale, high-quality datasets. Afterward, E2E FT[33] and Lotus[21] further accelerated inference by optimizing the noise scheduling process. In this work, our Jasmine continues these works and extends SD to the field celf-supervision
3 Methods
In this section, we will introduce the foundational knowledge of the SSMDE and SD-based MDE (Sec. 3.1), the surrogate task of mix-batch image reconstruction (MIR, Sec. 3.2), the scale-shift adaptation with GRU (Sec. 3.3) and the SD finetune protocol specified for self sul--superviioon (SSG, See. 3.4). An overview of the whole framework is shown in Fig. 2 and its training pseudocode is shown in Algorithm
3.1 Preliminaries
Self-Supervised Monocular Depth Estimation makes use of the adjacent frames I, to supervise the output depth with geometric constraints. Given the current frame I,, the MDE model as F I - DER W
can synthesize warped current frame I 1 with:
Pe- = I (proj (D, TE-e,
E (1 (1- 1,1+,,+1),
where Ti- denotes the relative camera poses obtained from the pose network, K denotes the camera intrinsics, and (-) denotes the grid sample process. Then, we can compute the photometric reconstruction loss between I, and I constrain the depth:
Lon (Ie, Ir)) = 7pr (1 SSIM (I. Ir-)))/ /2 T/p | Iy
Stable Diffusion- Monocular Depth Estimation reformulates depth prediction as image- conditioned annotation generation task. Typically, given the image and processed GT depth y, SD first encodes them to the low-dimension latent space through VAE encoder e, (s'z)z) xd,z") e(L

---

Afterward, gaussian noise is gradually added at levels T E [1 into z' to obtain the noisy sample, with z va,z" + +V-a,, then the model learns iteratively reverse by removing the
predicted noise:
é S z²)
and finally decodes the depth prediction with D = D(z)) ). Here ~ (0, 1), fg is the e-prediction is the noise schedule with U-Net, ai. = IT==1 Ba): D is the VAE decoder and {,1, , Brr
T steps.
Equation 1 demonstrates that the self-supervised approach needs the depth prediction D for image warping. However, obtaining zi requires iterative computation of Eq. 3, which becomes computation- ally infeasible considering the enormous size of SD models. Fortunately, Lotus and E2E FT[33, 21] demonstrate that we can obtain comparable results with single-step denoising and directly predict depth, D = D (fā (z%, z²) z²)) T T, which makes it possible to train SD with self-supervision. The step-by-step workflow is detailed in Sec B.
3.2 Surrogate Task: Image Reconstruction
Self-supervision Compromise SD Prior. The photometric reconstruction losses (Eq. 2) inevitably introduce the supervision with noise and artifacts due to occlusions, texture-less regions, and pho- tometric inconsistencies. As shown in Fig. 3 (a), consider a scenario where relative camera poses Ti-ee represents pure horizontal translation. The point p-which should ideally have 5 pixels of disparity (reciprocal of depth) incorrect pixel matches, resulting in erroneous depth alignment with point (a detailed explanation is provided in Sec. F). This phenomenon propagates to neighboring points, collectively eroding structural details (e.g.. the tree's edge) while generating imprecise supervisory signals that rapidly degrade SD's ' fine-grained prior knowledge.
occluded. Instead, it must displace 10 to compensate for
To preserve these details, we notice that both SSMDE and SD inherently rely on image consistency: SSMDE uses photometric constraints (Eq. 2), while SD's training directly minimizes image gen- eration errors. Inspired by this, we propose surrogate task: image reconstruction. Concretely, following [11], we design a switcher s E {sx, 8y) alternate the U-Net fă between the main and surrogate tasks. When activated by $n, we have D D (fø (s,. z'))- In contrast, we have I= (fz (sy, zZ, z²zz")) Notably, the switcher is a processed one-hot vector and it is combined with the time embeddings fed into the fä. This allows the U-Net condition its internal operations on the currently selected task. Therefore, we follow the SD paradigm and initially formulate the surrogate loss as:
L = 1 (N 2/, )12
Mix-batch Images Reconstruction with Photometric Supervision. We show that it is possible to preserve the SD priors by introducing a compact surrogate task. However, our experiments reveal that naively applying Eq. for SSMDE optimization yields suboptimal results (Fig. 3())). Through empirical investigation, we identify three critical insights: 1) Inferior reconstructed images (e.g., KITTI) introduce block artifacts (Fig. 3()). We attribute this to the latent space operating at g resolution: When inputs align with the pre-trained e and D, smooth supervision is achieved. Conversely, each latent pixel supervision manifests as 8x8 block artifacts in prediction. 2) Introducing high-quality synthesized images (maintaining self-super compliance) offers potential solution, but exclusive training on these data causes the model to only excel at reconstructing synthetic images but fails to generalize this capability to reduce depth estimation blurriness(igg. 3 (d)). 3) Mixing these images within a training batch allows synthesized images to anchor the model to latent priors, while real- world data enforces geometric structure alignment, is a possible solution. But this strategy shows notable sensitivity to mix rate X (Fig. 3 (g), (e) is a failure scene).
To address the mismatch between VAE and image, we proposed to replace Eq. 4 with the photometric loss Lih (Eq. 2) in the image domain. Compared to Eq. 4, Lyh emphasizes structural consistency rather than color fidelity, which aligns better with depth estimation objectives. As shown in Fig. 3(f g). this supervision not only makes MIR robust to À but also significantly improves the estimation quality. Therefore, we take the Hypersim dataset(4.2.1) photorcalistic synthetic dataset specifically designed for geometric learning, as the auxiliary image and update Eq. 4 with
In ó(I IH, A), L. Lph (I, D(fā (sy. 2. z))),,
(5)
4

---

Target View
(a) Principle
Source View
20mm
Shift 510 pixels
Oreclde
10m
Shift pixels
r(Correet)
(b) Origin
Shift 10 pixels
CoCrect
( Results
AbsRel
A1
(c) Direct
(d) Hly pepsiim
10.00 0.
1,0
Latent
(1) Ours
Figure 3: The attempts to preserve the SD prior. The meanings of (a)-(f) are detailed in Sec. 3.2. Notably, while (e) demonstrates superior visual quality, erroneously interprets surface textures (e.g., house windows) as depth edges. (g) shows the performance variations under different settings for photometric supervision (Eq. 5) and latent supervision (Eq. 4). The complete metrics and their definitions are provided in Sec. E.4.
where á denotes random choice; I and | are KITTI and Hypersim images, respectively.
In summary, MIR constructs each training batch by randomly selecting images from these two datasets, and supervises the reconstructed image with the photometric loss in Eq. 5.
Analysis of Auxiliary Images The specific synthetic data usage may raise concerns about appli- cability boundaries. To clarify, we conduct additional experiments and present three insights about the relationship between auxiliary data and performance: (1) Synthetic images are not essential; our surrogate task maintains efficacy with real-world imagery. (2) The dataset scale proves non-critical, as competitive performance emerges with samples under 1k. (3) Domain divergence between auxiliary and primary datasets enhances results and even more important than image quality. Please refer to Sec. 4.4 for detailed experimental support.
This analysis and related experiments reveal MIR is highly promising training paradigm. It not only imposes no inherent limitations on any dense prediction tasks but also challenges the notion that fine-tuning SD requires high-quality annotation. Even with legacy datasets like KITTI, we can still leverage readily available images to effectively utilize SD priors and enhance depth estimation sharpness.
3.3 Scale-Shift GRU
The misalignment of SSI-SI depth. We first analyze the training procedure of SSMDE. Denoting the relative camera poses To-re as [R TT), we can further expand the proj process in Eq. I with:
D'D'= = K K + +)),
(6)
where C' denote homogeneous coordinates and D, D' represent depths in I, and Ir, respectively. Afterward, the coordinate ç'[u,v] ) in Ie maps to [u,v] in Ie-se and we can grid sample every pixel to get pe However, this mapping is not unique. We can scale both sides the equation with 8ci
D's K(RK +
where both T and s.T represent valid relative poses. But if we further introduce shift 8h, we can derive the formula (detailed derivations at Sec. A) to obtain:
K ('''''g( RK- '(s + (T),,
8
where (-) and (2( are affine transformations (SSI depth is affine depth). The above equation means that the affine depth of any scene (91 (D')) can appear as a plane from a certain perspective. This plane has the depth s, and the extrinsic transformation of this perspective and the original one is [R|T]. which is undoubtedly impossible (more explanations in Sec. A). Thus, the shift 8h, does not exist under the geometric constraints, and SSMDE predicts Scale-Invar Invariant Depth Ds1-
Additionally, we analyze the training process SD-based MDE. The processed GT depth y mentioned in Sec. 3.1 satisfies the VAE's[27] inherent boundary [-1, 1]. typically obtained by:
((Dor - DDT.e)
DGT,2) 0.5) x 2,
where DCT.: corresponds to the i% percentiles of individual depth maps. Obviously, compared to the raw depth, y is normalized and differs from Der by absolute scale and shift, which can
5

---

(a);
Du (Pan)
10
D2 (Da, final prodictioa)
Distribution (%)
hanb
Do + D
standerd Ds
Sep-Conv GRU
Transformer
iiteration l
standard Des
Sex
Dx
Sigmoid Conv
( Pixel addition
2 Concatenation
Scale Shift Transformer
Diffusion
O Pixel multiply
8.00
0.25
0.50 0.75 1.00
Figure 4: (a): Model Structure of SSG. It correpoonds to the gray rectangle shown in Fig. 2. standing for an iteration within two consecutive ones. The pipeline of SSG is comprehensively described in Sec. 3.3 (DepthHead is omit in (a) for clear). (b): Depth distribution alignment visualization. We statistically analyze each stage of Jasmine's s SSG module on the KITTI test set. The standard SI and SSI depths are obtained by applying Eq. 9 and dividing the maximum value to the depth GT,, respectively.
be recovered by the least squares alignment in evaluation. Consequently, SD-based MDE predicts Seae-SSiftt-nnariiin Depth Dsst-
This inherent distribution gap between SSI and SI depth creates barriers for SD integration, and the SSG is specifically designed to fix
The Design of SSG. Transforming the depth distribution from SSI depth to SI depth requires profound scene understanding, and the scale (s,) and shift (s;,) factors are tightly coupled. Therefore, as shown in Fig. 4(a), compared traditional GRU, SSG introduces a core component Scale-Shift Transformer (SST), and modifies the iterative prediction formula:
From D4+1 = D& + D2 To D2+1 D5 + D3 Sh:
(10)
where D3 = DepthHHead(h&+)) & denotes the iteration step and h denotes the hidden state (prelimi- naries of GRU Sec D). Specifically, the SST employs learnable scale/shift queries (Qsc/Qsı)
(QHH) that
interact with SD's hidden states (keys/values) via cross-attention. The output vector is subsequently split and processed by MLPs to produce se and st. For the hidden state update, to enhance spatial awareness, the current input x* is defined as the concatenation of image features z' and the current depth DR. The hidden state h² evolves via a standard GRU iteration to produce the refined hidden state h and subsequently update D3 to DR+1-
To balance computational efficiency and GRU's iterative benefits, we employ two GRU iterations: starting from the initial depth Do (Dssı) to sequentially produce D1 and D2 (s))). As shown in Fig. 4(b), the distribution of Do tends to align with the standard SSI depth distribution, while D and D2 progressively converge towards the SI depth. This clearly demonstrates that SSG iteratively aligns SSI depth to SI depth by refining the scale-shift parameters. GRU is preferred over other architectures due to its reset gate mechanism. During training. the reset gate r can prevent the back- propagation of anomalous gradients to the former step by selectively resetting parts of the hidden state. Therefore, this mechanism enables the fine-grained Dsst to filter out erroneous supervision signals from reprojection losses and exhibit richer details than Dss (shown Fig. 2). To preserve these one-grained details in the final prediction, we further constrain the edge alignment between Dssı and Dsl with an edge extraction module, detailed in Sec. C.
3.4 Steady SD Finetune with Self-Supervision
As the first framework finetune SD with self-supervision, encountered novel challenge: training instability, which mainly due to the SD's enormous size, joint training across modules, and indirect self- supervuuery mechanisms. To enhance the convergence reliability and reproducibility, explored a straightforward approach by introducing pre-trained self-supervised teacher model (e.g.. MoToViT) estimate Die as pseudo labels. Drc provides direct supervision but has performance upper bound, which can stabilize model training in the early stages while gradually decreasing loss weights throughout the training process:
Lte (LB (norm (Dr.) Dssi) + LD (Dte, D1)-m)))))))))) 7h +LB (filter (Dr.). ,D..),Ds)) Dsr) 1/2) /Teepp (11)
where T.B the Berhu Loss "norm" is the [-1,1] normalization, and "filter" are adaptive strategies to avoid performance bottlenecks. Through extensive experimentation and error bar analysis,

---

demonstrate that this pseudo-label training proves particularly crucial for steady training in complex, multi-module sul-rvuee systems. The implementation details are in Sec. C. Finally, the total training loss of the Jasmine model is:
L = L. Lph Lie + La 7a
(12)
where L, refers to Eq.5, Loh refers Eq. 2, Lie refers to Eq. 11, and Le some auxiliary adjustment losses (e.g., gds loss[34] LGDS, edge loss Le, etc.) with tiny weight. They will be detailed in supplementary material Sec. C.
4 Experiment
4.1 Implement Details
We implement the proposed Jasmine using Accelerate [6] and PyTorch[35] with Stable Diffusion v2[39] as the backbone. Following the pipeline in Fig. 2, we disable text conditioning while maintaining most hyperparameter consistency with E2E FT[33]. The loss weights specified in Sec. are empirically configured as:
1/a Se 3, 7/1 = 0.6, 1/22 = 0.9, 7pp = 0.85, 7/22 = 0.15, Kktp = max (1, 30 (step /stepmoz))
Training uses the Adam W optimizer[30] with a base learning rate of 3e 5. All experiments conducted 8 NVIDIA A800 GPUs with total batch size of 32, training for total of 25k training steps, requiring around day. Following [15], we also employed standard data augmentation techniques (horizontal flips. random brightness, contrast, saturation, and hue jitter).
4.2 Evaluation
4.2.1 Datasets
Unless specified, all datasets are finally resized to 1024x320 resolution for training.
Training Datasets. KITTI[13]: Following the previous work[15], we mainly conduct our experiments on the widely used KITTI dataset. We employ Zhou's split[86] containing 39,810 10 training and 4,424 validation samples after removing static frames. The evaluation uses 697 Eigen raw test images with metrics from [15], applying 80m ground truth clipping and Eigen crop preprocessing[8] Hypersim[38]: This photorealistic synthetic dataset (461 indoor scenes) contributes approximately 28k samples from its official training split for mix-batch image reconstruction. Each iteration uses random crops from the original 1024x 768 to 1024 x 320 resolution.
Zero-shot Evaluation Datasets DrivingSterco(67]] Contains 500 images per weather condition (fog, cloudy, rainy, sunny) for zero-shot testing. CityScape[7]: Evaluated on 1,525 test images with dynamic vehicle-rich urban scenes, using ground truth from [55].
MIR Analysis Datasets ETH3D[41]: We resize this high-resolution (6048 x4032) dataset to 4K resolution, then randomly cropped to 1024x per iteration (898 total samples). Virtual KITTI[4] is a synthetic street scene dataset. We processed this dataset identically to real KITTI data.
4.2.2 Performance Comparison
For all the evaluations, only Jasmine", Marigold, E2E FT. and Lotus adopt the least squares alignment. The other self-supervised methods use median alignment. The definitions and differences of these alignments are detailed discussed in Sec. 4.3. The meaning of each metric is detailed in Sec. G.
KITTI result To fully demonstrate the advantages of our approach, we compare Jasmine against the most efficient SSMDE models and SoTA SD-based methods. As shown in Table 1, Jasmine achieves the best performance across all metrics on the KITTI benchmark. Notably, our method makes significant progress on the a metric, reflecting an overall improvement in depth estimation accuracy. This systematic advancement stems from the rich prior knowledge of SD. As shown in Fig. 1, without any specialized design for reflective surfaces, our method can accurately distinguish scene elements from their reflections. Additionally, the first row of Fig. 5 also demonstrates that our approach preserves structural details better than existing methods. Moreover, when compared to the other SD-based zero-shot models, none of which, including Jasmine", the ground truth depth of KITTI, our method demonstrates a substantial performance advantage, further highlighting the value of self- self-super techniques.
7

---

Table 1: Quantitative results on the KITTI dataset. For the error-based metrics the lower value is better; and for the accuracy-based metrics the higher value is better. The best and second-bes results are marked in bold and underline, Jacho et al* is combined model that applies both Jacho's [[34] (handle dynamic objects) and TriDepth's s[5] (solve edge flatten) approach. Jasmine and Jasmine are the same model but use different alignments (discussed in Sec. 4.3). In the data column, Syn, K. and H represent the synthetic, KITTI, and Hypersim datasets, respectively. The number of images and depth labels usage are in brackets. All experiments are conducted at 1024x320 resolution (Performance of Marigold/F2FE FT/L.otus is robust to this resolution).
Method
Venue
Notes
Data
AbaRcl
SqRel
RMSE
RMSElog
R1
dz
4g
Marigold[25]
CVPR2024
ZeroShot
4y4(74K+74K)
0.120
0.672
4.033
0.184
0.874
0.968
0.985
E2E FT(33]
WACV2025
ZeroShol
Sym(74K+74K)
0.112
0.649
4,099
0.180
0.890
0.969
0.985
I [
ICLR2025
ZeroShot
Syn($9K+559K)
0.110
0.611
3.807
0.175
0.892
0.970
0.986
Jasmine
Mono
KH(68K+0)
0.102
0.540
3.728
0.162
0.907
0.973
0.987
Monodepth2113 15]
ICCV2019
Mono
K(40K++0)
0.115
0.882
0.190
0.879
0.961
0,982
HR [31
AAA12021
Mona
K(40K+0)
0.106
0.755
4.472
0.181
0,892
0.966
0.984
R-MSHM66871
ICCV2021
Mono
K(40K+0)
0.108
0.748
0.185
0.889
0.963
0,982
[85]
ECCV2022
Mona
K(40K+0)
0.100
0.699
4,412
0.174
0,893
0.966
0.985
DepthSogNet
BCCV2022
Mono
K(40K+0)
0.099
0.624
0.171
0.902
0.969
0,985
SD 3D-SSMDE[37]
CVPR2022
Mona
K(40K+0)
0.098
0.674
4,187
0,170
0.902
0.968
0.985
[833
3DV
Mona
K(40K+0)
0.096
0.714
0.172
0.908
0.968
0.984
LiteMono[81]
CVPR2023
Mona
K(40K+0)
0.102
0.746
4,444
0.179
0,896
0.965
0.983
DaCCN[20]
ICCV2023
K(40K+0)
0.094
0.624
0.169
0.909
0.970
0.985
Jachu et all [5. 34]
CVPR2024
Mono
K(40K+0)
0.091
0.604
4.066
0.164
0.913
0.970
0.986
RP:Depth[9]]
ECCV2024
Mona
K(40K+0)
0.091
0.612
0.162
0.910
0.971
0.986
BCCV2024
Mono
K(40K+0)
0.093
0. 58589
4.072
0. 168
0.909
0.969
0.985
Jasmine
Mona
KH(68K+0)
0.090
0.581
3.944
0.161
0.919
0.972
0.986
Table 2: Quantitative zero-shot results on the CityScape and DrivingStereo dataset and its variants (Rainy, Cloudy, Foggy). Alignment protocols and annotation rules follow Table I's specifications. AbsRel, RMSE, and 01 metrics are shown.
Method
DrivingStereo AhsRel
Rainy
CityScape RMSE
Cloudy RMSE
Foggy
RMSE
4:
AbsRel
RMSE
41
AbeRel
AbsRel
43
AbsRel
RMSE
4s
Marigold[5]]
0.178
6.638
0.749
0,148
6.770
01801
0.164
6.632
0.763
0.173
6.88T
0.751
0.146
6.545
01.9%
E2E PT]33]
0.160
5.477
0.795
0.164
6.671
0.793
0.160
6,944
0.792
0.157
5.522
0.797
0.141
6.034
0.836
T [1]
0.173
5.816
0.771
0.167
6.675
0.775
0.147
6.582
0.824
0.159
5.640
0.795
0.150
6.173
0.798
Jasminee
0.134
4.666
0.854
0.159
6.071
0.825
0.107
5,000
0.907
0.134
4,762
0.846
0.113
4
0.897
MonnDephh2 15]
0.191
K3359
0.770
0.200
123377
0.609
0.158
8.185
0.783
0.192
10.07
0.775
0,156
10.425
0.799
MonoV/][83]
0,150
7.657
0.815
0.190
9.407
0.724
0.140
7.913
0.802
0.134
7,280
0.849
0.107
7,899
0.882
0.158
6.723
0.798
0.400
13.960
0.484
0.134
7.372
0.817
0,154
6.883
0,800
0.160
8.494
0.769
WealberDepoh(49]
0.166
6.986
0.796
0.166
8 1.844
0.748
0.137
6.515
0.837
0.167
7.566
0.793
0.132
7.679
0.859
Jasmine
0.136
5,340
0.850
0,160
7.194
0.787
0.123
6.618
0.852
0.133
5.651
0.849
0.098
5.702
0.902
Generalization Results Compared to in-domain evaluation, Jasmine exhibits even more remark- able results in zero-shot generalization. Due to the large number of comparison datasets and the unavailability of some models' open-source code, we only compare with Monodepth2 (the most classic model), MonoViT (renowned for its robustness), Mono (the latest model), and Weath- erDepth (trained with additional weather-augmented data, totaling 278k samples), all of which are self-supervised models. In fact, Mon already surpasses the zero-shot capability for nearly all models in the SSMDE, making it sufficiently strong baseline[42]. Additionally, SD-based methods have excelled in generalization, allowing us to conduct a fair zero-shot comparison here. As shown in Table 2, Jasmine demonstrated state-of-the-arr performance on the datasets of Cityscape
Wentherdepth Figure 5: Qualitative results on KITTI, DrivingStereo, and CityScape datasets. We compare Jasmine with the most generalizable and best berf-perfo SSMDE methods in both in-domain and zero-shot scenarios.
RGB Image (Part) Zoom/Enhance
MonoViI
MoneVIFi
Jasmine
8

---

and four weather scenarios of driving stereo. Remarkably, our method maintains effectiveness in out-of-distribuutin (OOD) rainy conditions even without specialized training on weather-enhancee datasets like WeatherKrTTI
As illustrated in Fig. 1, Jasmine successfully identifies water surface reflections while producing refined depth estimates. The fine-grain estimation details are further demonstrated in Fig. 5, pedestrian chins (second row), tree support structures (third row), and bicycle-rider contours (fourth row) are all predicted delicately and precisely. These sharp results were completely disrupted by the reprojection loss in previous self-supervised methods.
Further Analysis We also deeply compare Jasmine with other SSDE configurations (stereo training and multi-frams inference) and on KITTI improved GT benchmark in Sec. E.2, E.1.
4.3 Analysis of Depth De-normalization
(a)
(b)
The performance gap between Jasmine and Jasmine* in Tables 1 and 2 high- lights the impact of de-normalization median alignment for self- supervised methods and LSQ alignment for zero-shot settings. As the first work bridging these two domains, we provide an in-depth analysis of how these choices affect evaluation.
(c)
(d)
Figure 6: Comparison of different de-normalization
Depth de- normalization refers to the pro- cess of transforming the model's predicted depth values back to the GT distribution for metric evaluation. The typical de- normalization strategies include:
schemes. The blue points are predictions after align- ment and the green line is the ideal GT depth. Sub- figures (a,c) are the results of LSQ alignment, while (b,d) are median alignment.
No operation: Models predict metric depth [23]: Deval = Dpred-
Median Alignment: Models predict SI Depth [15], which can be recovered by scaling the ratio of the median of the GT depth to the median of the predicted depth: Deval = Dpred
median DcT) ht////mimnnn....e (Dee)))
@ Least-Square (LSQ) Aligmment: Models predict SSI Depth [25]. which can be recovered by affine transformation (scaled and shifted) to best fit the GT depth. The evaluated depth is given by: Deval s" pred t*, where the optimal scale 8* and shift t* are determined by minimizing the
sum of squared differences: (8", = arg min C( Dpred,t - t) D
8.6
From Tables and 2, we have following observations:
Metric inconsistency: For the exact same model, the evaluation metrics can vary dramatically depending on the de-normalization method, making direct comparison unfair.
Metrics characteristics: Under LSQ alignment, quadratic metrics (e.g., SqRel, RMSE) are usually better, but first-order metrics (e-g., AbsRel) and overall accuracy (a1) are often worse.
Scenario suitability: For in-domain training, median alignment is generally superior, while in zero-shot scenarios, LSQ alignment usually stronger.
As shown in Fig. 6 (a, c). LSQ alignment tends to accommodate outliers a and a', resulting in a larger overall shift in the aligmment. In contrast, in sub-figures (b, d), outliers have little effect on the median, so the accuracy for the majority of points is preserved. In the context of depth estimation, and a' can represent the model's predictions for regions that are difficult to estimate. Clearly, for quadratic metrics, under median alignment, the large error between a' and the GT in (d) will be further amplified by squaring, leading to drop in the overall metric. However, when computing overall accuracy, a' or are typically outside the threshold for a1 and thus do not affect it, while the originally accurate predictions for other points become less accurate due to the shift introduced by LSQ alignment. Similarly, first-order metrics also degrade due to these shifts. This explains why Jasmine* achieves better RMSE and SqRel but worse úi and AbsRel compared to Jasmine.
For the last observation, in in-domain scenarios (Table 1). Jasmine outperforms Jasmine* because the shift-freee estimation learned by Jasmine is disrupted by LSQ alignment, introducing suboptimal shift and degrading performance. In out-of-domain scenarios (Table 2), Jasmine* * performs better, as the depth distributions of different datasets may differ significantly, and using least-squares to estimate the best fit is clearly better choice.

---

4.4 Ablation Study
As shown in Table 3, we conduct ablation studies to validate our designs. Firstly, in sub-table (a), we gradually tested the effects of our basic components, such as SD prior, MIR, and SSG. The SD prior proves most critical training from scratch (IDO vs. IDI) causes catastrophic failure (AbsRel1473%, RMSE1206%). The other experiments also demon- strate that disabling MIR (ID4) or SSG (ID3) degrades performance by 47%/43% in Ab- distribution alignment and SD detail preservation.
sRel, proving the necessity of depth In sub-table (b), we further melted down our proposed SSG. ID (6) reveals that the naive GRU can initially solve the distribu- tion misalignment by estimating the D& (Eq. 4). However, the scale difference be- tween SSI and SI depth makes it difficult to restore through linear addition. Therefore, after introducing SST, the overall model performance is further enhanced by 10%, ultimately achieving SoTA performance. A comprehensive analysis of MIR was con- ducted in sub-table (c) and we can draw similar conclusions Sec. 3.2. Jasmine significantly outperforms the alternatives, such as KITTT/yynthetic-nnly reconstruc-
Table 3:
Ablation Studies. vK and Hy mean the virtual KITTI and Hypersim datasets. Dataset/n denote we downsam- ple the image to resloution(i.e. Hy/l.6 means downsample to 640x 192, where 640=1024/1.6 6) and resize them back.
(ID) Method
AbsRel
SqRel
RMSE
aj
0g
Ours
(0) Jasmine
0.090
0,581
3.944
0.919
0.972
(a) Basic Component
w/o Prior
0.516
6.019
12.06
0.258
0.501
w/o MIR +SSG
0.175
2.264
7.969
0.790
0.929
(3) w/o SSG
0.129
0.938
4,470
0.872
0,956
(4) w/o MIR
0.132
0.673
4.271
0.852
0.967
(b) Scale-Shift GRU
(5) w/o SSG
0.129
0.9.8
4,470
0.872
0.956
w/o SST
0.098
0.715
4,350
0.909
0.969
(c) Mix hatch Image Reconstruction
w/o MIR
0.132
0.673
4,271
0.852
0,967
tion (IDs (8.9) and Fig. 3 (c,d)) and latent- space supervision (ID (10) and Fig. 3 (e)), which strongly proves the effectiveness of our proposed MIR. The analysis of auxil- iary images is presented in sub-table (d). The experiments in IDs (0, 13, 14) indicate that, compared to real/synthesis datasets, the content of the dataset more impor- tant, and diverse scenes offer greater ben- efits than street views (virtual KITTI images) similar to our primary dataset, KITTI. Furthermore, IDs (0. 15, 16) demonstrate that our surrogate task is robust to image sampling resolutions, as downsampling to IE or even á has minimal impact on the results. Moreover, ID(13) further confirms that MIR remains effective even when trained on small-scale datasets (fewer than 1K samples). These insights provide potential opportunities for applying SD models to other dense estimation tasks and enhancing result sharpness. In summary, these ablation results validate the effectiveness of our proposed adaptation protocol, indicating that each design plays a crucial role in optimizing the diffusion model for superviee depth estimation tasks.
(8) Direct
0.129
0.679
4,385
0.858
0.962
(9) Only Hy
0,106
0.614
4,181
0.901
0,970
(10) Latent space
0.095
0.606
4.138
0.909
0,970
(d) Auxiliary
Image Analysis
(12) KITTI
0.095
0.616
4.040
0,912
0.972
(13) KITTI-ETHRD
0.090
0.586
3.937
0.916
0.972
(14) KITTI-vK
0.094
0.606
4.068
0.911
0,972
(15) KITTI+Hy/4
0.091
0.596
3.943
0.917
0,972
(16) KITTI=Hy/I.6
0.090
0.591
3.971
0.918
0.972
4.5 Inference Latency
As shown in the table below (MACs and Runtime are measured on a image with 1024x resolution on RTX 4090.), while Jasmine is more computationally expensive than prior sule-supervised methods, it follows the trend of models like Marigold in trading cost for superior performance. Notably. our
SSG module adds negligible latency, with Jasmine's s runtime being comparable to Lotus.
Method
Marigold
Lotus
E2F-Mono
Monodepth2
MonoVIT
MonoViFi
Jasmine
MACs
133T
2.65T
2.6ST
21.43G
25.63G
28.79G
2.83T
Runtime
9.88s
157ms
152ms
33ms
29ms
25ms
172ms
5 Conclusion
We propose Jasmine, the first SD-based self sulf-supervised framework for monocular depth estimation, effectively leveraging SD's priors to enhance sharpness and generalization without high-precision supervision. To achieve this objective, we introduce two novel modules: Mix- Image Recon- struction (MIR) for mitigating reprojection artifacts and preserving Stable Diffusion's 's latent priors, alongside Scale-Shift GRU (SSG) to align scale-invariant depth predictions while suppressing noisy gradients. Extensive experiments demonstrate that Jasmine achieves SoTA performance on KITTI and superior zero-shot generalization across datasets. Our approach establishes a new paradigm for unsupervised depth estimation, paving the way for future advancements in self-supervised learning.
10

---

Acknowledgment
This work was supported by the National Natural Science Foundation of China (NSFC) under Grant (U2441222,62112203) and Graduate Research Innovation Project under Grant (KKYJS25001536),
References
Antyanta Bangunhareana, Ahmed Magd, and Kyung-Soo Kim. Dualrefine: Self-supervised depth and pose estimation through iterative epipolar sampling and refinement toward equilibrium. In CVPR, pages 726-738, 2023. 3, 23
[2] Juan Luis Gonzalez Bello and Munchurl Kim. Forget about the lidar: Self-supervised depth estimators with med probability volumes. In NewrIPS, pages 12626-12637, 2020. 2
[3] Juan Luis Gonzalez Bello and Munchurl Kim. Plade-net: Towards pixel-level accuracy for self- suff-super single-view depth estimation with ncural positional encoding and distilled matting loss. In CVPR, pages 0851-6860, 2021. 2
[4] Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual kitti 2. arXiv preprint arXi:2001.10773. 2020. 7
[5] Xingyu Chen, Ruonan Zhang, Ji Jiang, Yan Wang, Ge Li, and Thomas H Li. Self-supervised monocular depth estimation: Solving the edge-fattening problem. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5776-5786, 2023. 8
[6] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 2132-2143, 2022. 27
[7] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bertt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213-3223, 2016. 7
[8] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from single image using multi-scale deep network. Advances in neural information processing systems, 27, 2014.
[9] Ziyue Feng, Liang Yang, Longlong Jing, Haiyan Wang, Yingli Tian, and Bing Li. Disentangling object motion and occlusion for unsupervised multi-frame monocular depth. In ECCV, pages 228-244, 2022.
[10] Michaël Fonder, Damien Ernst, and Marc Van Droogenbroeck. M4depth: Monocular depth estimation for autonomous vehicles in unseen environments, 2021.
[11] Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing diffusion priors for 3d geometry estimation from single image. arXiv preprint 1211 2024. 3,4
[12] Ravi Garg, Vijay Kumar BG, Gustavo Carneiro, and Ian Reid. Unsupervised cnn for single view depth estimation: Geometry to the rescue, 2016. 2
[13] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3354-3361.
IEEE, 2012. 7
[14] Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE conference computer vision and pattern recognition. pages 270-279, 2017. 2
[15] Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3828 3838, 2019. 7, 8, 9, 29
[16] Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https: httppp///ggtauubb.oo//gggggggii http///gggiiggaccc///eeeeeeeeeee
2022. 7
[17] Vitor Guizilini, Rares Ambrus. Dian Chen, Sergey Zakharov, and Adrien Gaidon. Multi-fram self- supervised depth with transformers. In CVPR, pages 160-170 170, 2022.
11

---

[18] S. Mahdi H. Miangoleh, Mahesh Reddy, and Yağız Aksoy. Scale-invariant monocular depth estimation via ssi depth. In Special Interest Group Computer Graphics and Interactive Techniques Conference Conference Pupers '24, SIGGRAPHI '24, page 1-11, ACM, July 2024. 22
[19] Wencheng Han and Jianhing Shen. High-precision self-supervised monocular depth estimation with rich-resource prior. In European Conference on Computer Vision, pages 146-162. Springer, 2024. 2, 8
[20] Wencheng Han, Junbo Yin, and Jianbing Shen. Self-supervised monocular depth estimation by direction- aware cumulative convolution network. In ICCV, 2023. 2,
[21] Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu, and Ying- Cong Chen. Lotus: Diffusion cradusioo--baee visual foundation model for high-quality dense prediction. arXiv preprint Arx: 200244999/24444. /8124, 2024. 1, 2, 3, 4, 8, 28
[22] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, 2020. 3
[23] Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang. Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero- shot metrie depth and surface normal estimation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[24] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu. Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. In 2023 IEEE/CVF International Conference Computer Vision (ICCV). IEEE, October 2023. 3
[25] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Meizger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9492 9502, 2024. 2, 3, 9, 28
[26] Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimarães, and Pietro Perona. Text-image alignment for diffusion-based perception, 2023. 3
[27] Anders Boesen Lindbo Larsen, Spren Kaae Sønderby, and Ole Winther. Autoencoding beyond pixels using learned similarity metric. CoRR, abs/1512.099300, 2015. 5,5
[28] Kang Liao, Zongsheng Yue, Zhouxia Wang, and Chen Change Loy. Denoising as adaptation: Noise-space domain adaptation for image restoration. arXiv preprint arXiv:2406. 185 2024.
[29] Jinfeng Liu, Lingtong Kong, Bo Li, Zerong Wang, Hong Gu, and Jinwei Chen. Mono-vifi: A unified learning framework for self seff-supervised single and multi-frame monocular depth estimation. In European Conference on Computer Vision, pages 90 107. Springer, 2024. 2, 8, 23, 28
[30] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 7
[31] Xiaoyang Lyu. Liang Liu, Mengmeng Wang, Xin Kong, Lina Liu, Yong Liu, Xinxin Chen, and Yi Yuan. Hr-depth: High resolution self-supervised monocular depth estimation. In AAAI, pages 2294-2301, 2021. 2,
[32] Jingyuan Ma, Xiangyu Lei, Nan Liu, Xian Zhao, and Shiliang Pu. Towards comprehensive representation enhancement in semanticx-guided self-supervised monocular depth estimation. In ECCV, pages 304-321,
2022. 2, 8
[33] Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt, Daan de Geus, Alexander Hermans, and Bastian Leibe, Fine-tuning image-conditional diffusion models is easier than you think. Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), 2025. 1, 2, 3, 4, 7, 8, 28
[34] Jacho Moon, Juan Luis Gonzalez Bello, Byeongjun Kwon, and Munchurl Kim. From-ground-to-oe Coarse-to-fire self-supervised monocular depth estimation of dynamic objects with ground contact prior. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10519- 10529, 2024. 2, 7, 8, 25
[35] Adam Paszke et al. Pytorch: An imperative style, high- geghhperformance deep learning library. In NeurlPS, pages 8026-8037, 2019. 7
[36] Rui Peng, Ronggang Wang, Yawen Lai, Luyang Tang, and Yangang Cai. Excavating the potential capacity of selfsupervised monocular depth estimation. In ICCV, pages 15560- 15569, 2021. 23
12

---

[37] Andra Petrovai and Sergiu Nedevschi. Exploiting pseudo labels in self-supervised learning framework for improved monocular depth estimation. In CVPR. pages 1568-1578, 2022. 8
[38] Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10912-10922, 2021.7 7
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022. 3,7
[40] Saurahh Saxena, Charles Herrmann, Junhwa Hur, Abhishek Kar, Mohammad Norouzi, Deqing Sun, and David J. Fleet. The surprising effectiveness of diffusion models for optical flow and monocular depth estimation, 2023. 3
[41] Thomas Schops, Johannes L. Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler. Mare Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi- camera videos. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3260-330 02000-32,,2200 2017.
[42] Lingdong Seo, Liang Weng, Zhiqiang Zhao, Xuan Wang, Jing Qiu, and Yinlong Liu. Robodepth: Robust out-of-distribution depth estimation under corruptions. In Advances in Neural Information Processing Systems, volume 36, 2024.
[43] Shuwei Shao, Zhongeai Pei, Weihai Chen, Peter CY Chen, and Zhengguo Li. Nddepth: Normal-distance assisted monocular depth estimation and completion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. 27
[44] Shuwei Shao, Zhongeai Pei, Weihai Chen, Dingchi Sun, Peter C. Y. Chen, and Zhengguo Li. Monodiffusion: Self-supervised monocular depth estimation using diffusion model. 2023. 3
[45] Shuwei Shao, Zhongeai Pei, Xingming Wu. Zhong Liu, Weihai Chen, and Zhengguo Li. lebins: Iterative elastic bins for monocular depth estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 27
[46] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, 2020. 3
[47] Jonas Uhrig, Nick Schneider, Lukas Schneider, Uwe Franke, Thomas Brox, and Andreas Geiger. Sparsity invariant cnns. In 2017 international conference on 3D vision (3DV), pages 11 11-20. IEEE, 2017. 28
[48] Haochen Wang, Anlin Zheng, Yucheng Zhao, Tiancai Wang, Zheng Ge. Xiangyu Zhang, and Zhaoxiang Zhang. Reconstructive visual instruction tuning. arXiv preprint 2411v:2410009575. 2024. 23
[49] Jiyuan Wang. Chunyu Lin, Lang Nie, Shujun Huang. Yao Zhao, Xing Pan, and Rui Ai. Weatherdepth: Curriculum contrastive learning for self-supervised depth estimation under adverse weather conditions. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 4976-4982, IEEE, 2024.
8, 9
[50] Jiyuan Wang, Chunyu Lin, Lang Nie, Kang Liao, Shuwei Shao, and Yao Zhao. Digging into contrastive learning for robust depth estimation with diffusion models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 4129-4137, 2024. 3, 27
[51] Ji Wang, Chunyu Lin, Lei Sun, Rongying Liu, Lang Nie, Mingxing Li, Kang Liao, Xiangxiang Chu, and Yao Zhao. From editor to dense geometry estimator. arXiv preprint (tl:25509043335 2025. 3
[52] Ruoyu Wang, Zehao Yu, and Shenghua Gao. Planedepth: Self-supervised depth estimation via orthogonal planes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21 21425-21434, 2023. 2, 23
[53] Xiangi Wang, Hao Yang, Gangwei Xu, Junda Cheng. Min Lin, Yong Deng. Jinliang Zang. Yurui Chen, and Xin Yang. Zerostereo: Zero-shot stereo matching from single images. arXiv preprint arXiv:2501.08655 2025. 30
[54] Zhengxue Wang, Zhiqiang Yan, Jinshan Pan, Guangwei Gao, Kai Zhang, and Jian Yang. Dorne: A degradation oriented and regularized network for blind depth super-resolution. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 15813-15822 2025. 30
13

---

[55] Jamie Watson, Oisin Mac Aodha, Victor Adrian Prisacariu, Gabriel J. Brostow. and Michael Firman. The temporal opportunist: Self-supervised nulti-frame monocular depth. In CVPR, pages 1164-1174, 2021. 3.
7, 23
[56] Gangwei Xu, Haotong Lin, Hongcheng Luo, Xianqi Wang, Jingfeng Yao, Lianghui Zhu, Yuechuan Pu, Cheng Chi, Haiyang Sun, Bing Wang, et al. Pixel-perfect depth with semantics prompted diffusion transformers. arXiv preprint arXiv:2510.07316, 2025. 30
semantics-prompted
[57] Gangwei Xu, Jiaxin Liu, Xianqi Wang, Junda Cheng, Yong Deng, Jinliang Zang, Yurui Chen, and Xin Yang. Banet: Bilateral aggregation network for mobile stereo matching. arXiv preprint wrXiv25503.03555 2025. 30
[58] Gangwei Xu, Xianqi Wang, Zhaoxing Zhang, Junda Cheng, Chunyuan Liao, and Xin Yang. Igev++:: Iterative multi-range geometry encoding volumes for stereo matching. IEEE Transactions on Pattern Analysis and Machine Intelligence, (((8):7108-712))) 2025. 2025, 30
[59] Guangkai Xu, Yongtao Ge. Mingyu Liu, Chengxiang Fan, Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. What matters when repurposing diffusion models for general dense perception tasks? arXiv preprint arXiv:2403.006 06090, 2024. 3
[60] Guangkai Xu, Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Simon Chen, and Jia- la-Wang Bian. Towards domain- domain-agnostic depth completion. Machine Intelligence Research, (1(4)652-669) 2024. 30
[61] Guangkai Xu and Feng Zhao. Toward 3d scene reconstruction from locally scale-aligned monocular vidco depth. JUSTC, 5444)040-11-002-111 2024. 1
[62] Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, and Marc Pollefeys. Depthsplat: Connecting gaussian splatting and depth. arXiv preprint arXiv:24 13662, 2024.
[63] Zhiqiang Yan, Yuankai Lin, Kun Wang, Yupeng Zheng, Yufci Wang, Zhenyu Zhang, Jun Li, and Jian Yang. Tri-perspective view decomposition for geometry-aware depth complction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4874-4884, 2024, 30
[64] Zhiqiang Yan, Kun Wang, Xiang Li, Guangwei Gao, Jun Li, and Jian Yang. Tri-perspective view decomposition for geometry aware depth completion and super-resolution. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. 30
[65] Zhiqiang Yan, Kun Wang, Xiang Li, Zhenyu Zhang, Jun Li, and Jian Yang. Rignet: Repetitive image guided network for depth completion. European Conference on Computer Vision, pages 214-230. Springer, 2022. 30
[66] Zhiqiang Yan, Zhengxue Wang. Kun Wang, Jun Li, and Jian Yang. Completion as enhancement: A -egradation-aware selective image guided network for depth completion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 26943-26953, 2025. 30
[67] Guorun Yang, Xiao Song, Chaoqin Huang, Zhidong Deng, Jianping Shi, and Bolei Zhou. Drivingstereo: large-scale datasct for stereo matching in autonomous driving scenarios. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[68] Lihe Yang. Bingyi Kang, Zilong Huang. Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEF//VF Conference on Computer Vision and Pattern Recognition, pages 10371-10381, 2024. 3
[69] Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.044 094144 2024.
[70] Zhu Yu, Zehua Sheng. Zili Zhou, Lun Luo, Si- Yuan Cao, Hong Gu, Huaqi Zhang, and Hui-Liang Shen. Aggregating feature point cloud for depth completion, In Proceedings of the IEEE/CVF international conference on computer vision, pages 8732-8743, 2023. 30
[71] Zhu Yu, Runmin Zhang. Jiacheng Ying. Junchen Yu, Xiaohai Hu, Lun Luo, Si- Yuan Cao, and Hui-Liang Shen. Context and geometry aware voxel transformer for semantic scene completion. Advances Neural Information Processing Systems, 37:1531-1555 2024. 30
[72] Zhenlong Yuan, Cong Liu, Fei Shen, Zhaoxin Li, Jinguo Luo, Tianlu Mao, and Zhaoqi Wang. MSP-MVS: Multi guatulgranularity segmentation prior guided multi view stereo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 9753-9762, 2025. 30
14

---

[73] Zhenlong Yuan, Jinguo Luo, Fei Shen, Zhaoxin Li, Cong Liu, Tianlu Mao, and Zhaoqi Wang. DVP-MVS: Synergize depth-edge and visibility prior for multi-view stereo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 9743-9752, 2025.
[74] Zhenlong Yuan, Zhidong Yang. Yujun Cai, Kuangxin Wu, Mufan Liu, Dapeng Zhang. Hao Jiang. Zhaoxin Li, and Zhaoqi Wang. SED-MVS: Segmentation- Driven and Edge-Aligned Deformation Multi-View Stereo with Depth Restoration and Occlusion Constraint. IEEE Transactions on Circuits and Systems for Video Technology, 2025. 30
[75] Zhenlong Yuan, Dapeng Zhang. Zehao Li, Chengxuan Qian, Jianing Chen, Yinda Chen, Kehua Chen, Tianlu Mao, Zhaoxin Li, Hao Jiang, and Zhaoqi Wang. Dvp-mvs++: Synergize depth-normal-edge and harmonized visibility prior for multi-view stereo, 2025. 30
[76] Zongsheng Yue, Kang Liao, and Chen Change Loy. Arbitrary-steps image super-resolution via diffusion inversion. arXiv preprint arXiv:2412.09013 2024.
[77] Ziyao Zeng, Jingcheng Ni, Daniel Wang. Patrick Rim. Younjoon Chung, Fengyu Yang, Byung-Wo Woo Hong, and Alex Wong. Priordiffusio:: Leverage language prior in diffusion models for monocular depth estimation, 2024.
[78] Ziyao Zeng, Jingeheng Ni, Daniel Wang, Patrick Rim, Younjoon Chung, Fengyu Yang. Byung- Woo Hong, and Alex Wong. Priordiffusion: Leverage language prior in diffusion models for monocular depth estimation. arXiv preprint arXiv:2411.16755 2024. 30
[79] Ziyao Zeng, Daniel Wang, Fengyu Yang. Hyoungseob Park, Stefano Soatto, Dong Lao, and Alex Wong. Wordepth: Variational language prior for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9708 9719, 2024. 30
[80] Ziyao Zeng, Yangchao Wu, Hyoungseob Park, Daniel Wang. Fengyu Yang, Stefano Soatto, Dong Lao, Byung-Wov WWo Hong, and Alex Wong. Rsa: Resolving seale ambiguities in monocular depth estimators through language descriptions. Advances in neural information processing systems, 17:112664-127755 2024. 30
[81] Ning Zhang, Francesco Nex, George Vosselman, and Norman Kerle. Lite-mono: lightweight enn and transformer architecture for self-supervised monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18537-18546. 2023. 8
[82] Renrui Zhang, Ziyao Zeng. Ziyu Guo, and Yafeng Li. Can language understand depth? In Proceedings of the 30th ACM International Conference on Multimedia, pages 6868 68744, 2022. 30
[83] Chaoqiang Zhao, Youmin Zhang, Mattco Poggi, Fabio Tosi, Xianda Guo, Zheng Zhu, Guan Huang, Yang Tang, and Stefano Mattoccia. Monovit: Self-supervised monocular depth estimation with a vision transformer. In 2022 international conference 3D vision (3DV), pages 668 668. IEEE, 2022. 1, 8, 28
[84] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu, Unlcashing text-to-image diffusion models for visual pereeption. ICCV, 2023. 3
[85] Kaichen Zhou, Lanqing Hong, Changhao Chen, Hang Xu, Chaoqiang Ye, Qingyong Hu, and Zhenguo Li. Devnet: Self-supervised monocular depth learning via density volume construction. In ECCV, pages 125-142,
2022. 2,
[86] Tinghui Zhou, Matthew Brown, Noah Snavely, and David G. Lowe. Unsupervised learning of depth and ego-motion from video, 2017.
[87] Zhongkai Zhou, Xinnan Fan, Pengfei Shi, and Yuanxue Xin. R-msfm: Recurrent multi-scale fcature modulation for monocular depth estimating. In ICCV, pages 12757-12766, 2021. 2, 8
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [TODO]
Guidelines:
15

---

The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:[Yer Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch provide intuition.
16

---

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nipe cc/ /wwidee/ooeeeuumiiii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not
17

---

reproducible, they including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss wwwppppwc/gg.p.ieeeeeeeeeooopeee /ewpddddeeeeeeuueaiiiiiiieaae http///ooouuissiiooooo wwwpamisssioooiicyy...oo.. for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.).
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma еrror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
18

---

8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: / httprpps.../ppp..ic/hcccccceded
Answer:[Yes)
Justification:
[TODO]
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[Yes)
Justification: [TODO]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
19

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer[[NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual- dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ com/datsset has curated licenses for some datasets. Their licensing guide can help determine license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:[Ye]]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not release new assets.
20

---

Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer[[NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: [TODO]
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: htt////eenp.p///// cc/Confferecces/2225 for what should or should not be described.
21

---

Appendix
In this appendix, we provide more implementation details, experiments, analysis, and discussions for a comprehensive evaluation and understanding of Jasmine, Detailed contents are listed as follows:
Contents
A Proof of Scale-Invariant Depth
22
Detailed Training Workflow and Pseudocode
23
B.1 Step-by-Step Workflow
23
B.2 Training Pseudocode
24
Supervision Details
24
C.I SGD Loss for Dynamic Object
25
C.2 Edge Loss for Sharp Prediction
25
C.3 Sky Loss for Anti-artifact
25
C.4 Eliminating the Bottleneck Teacher Loss
26
Preliminaries of GRU
27
Addition Experiments Results
28
E.1
Evaluation on KITTI Improved Benchmark.
28
E.2
Comparison with Other Self-Supervised Settings
28
E.3 Qualitative Comparisons ................................
................................
28
E.4 Mix-batch Ratio Ablation Details
28
Analysis of SD Prior Degradation via Self-Supervision
29
G Evaluation Metries
29
H Error Bar Analysis
29
Limitation and Future Work
30
Proof of Scale-Invariant Depth
For SSI depth, we assume that the estimated depth î and the ground truth depth / have the following relationship:
D DD Sh
Meanwhile, the depth from another viewpoint be denoted gı (D'). Here, 91 represents transformation, and it is straightforward to see that this must be a linear transformation since the known identity only contains linear terms. Introducing a nonlinear transformation would not hold for the depth of arbitrary scenes. Thus, we can define:
91 (D())) 0 + bj
Similarly, we define the transition g2 g2(T), which also a linear transformation, given by:
ga (T 02T + by
Note that for transformations within the same scene, the relative pose R remains constant[18] Also, and b here can be arbitrary numbers, so we have:
g(x) ~ g(x) a3 ~ g(x)
(13)
22

---

Table 4: Additional quantitative results the KITTI dataset.
Method
Venue
Notes
AbsKcl
SqKel
RMSE
KMSElog
02
0g
ManyDepth 55]
CVPR2021
(-1,0)
K(40K+0)
0.091
0.694
4.245
0.171
0.911
0.968
0.983
Dual Refine I
CVPR2023
(-1,0)
K(40K +0)
0.087
0.674
4.130
0.167
0.915
0.969
0.984
Mono [9]]
ECCV2024
(-1,0,1)
K(40K+0)
0.556
3,981
0.164
EPCDepth[36
ICCV2021
Stereo
K(45K
0.091
0.646
4.207
0.176
0.901
0.966
0.983
PlaneDepth(S (52]
CVPR2023
Stereo
K(45K+0)
0.085
0.563
4.023
0.171
0.910
0.968
0.984
Jasmine
Monn
KH(68K*0)
0.090
0.581
3,944
0.161
0.919
0.972
0.986
Assuming the SSI depth is work, we have:
S'g1(D') = K (RK" (RK-'C(s.DD 3h)) + 92()))
(14)
Eq. in the paper is:
C's = (RK- (RK- C8.D TTT))
(15)
Subtracting Eq. 15 from Eq. 14, we have:
s' (g') SeD") = K (RK-IÇaA (8h + (32(T) - sTT))
From Eq. 13, it can be simplified as:
('g1 K (RK- Csh. + 92(())
Multiplying both sides by K we get:
K K- (''( = RK- + (2())
This is Eq. 8 the paper.
B Detailed Training Workflow and Pseudocode
B.1 Step-by-Step Workflow
In this section, temporarily omit the specifies of MIR and SSG to clarify the integration of self-supervision with the Stable Diffusion framework in Jasmine:
Input: A sequence of temporally adjacent images (like video frames), e.g., a source image lp and target image 1t-
-Here I is exactly the training image, Te can be the previous frame T or the next frame Tt+1 of T..
Step 1: Depth Prediction (via SD U-Net):
The SD U-Net, fū. takes the latent of the target image, of = VAE.encode(L)) 42), and a pure noise vector, n, input.
performs single-step denoising process to predict the latent representation of a depth map, 20-
This is the key point we mentioned in Sec 3.1: single-step denoising make this step become a fast, feed-forward process rather than computationally prohibitive with slow, iterative denoising.
The VAE decoder then converts z into the final depth map, D = VAE.decode(2"). Crucially, no GT depth is ever used in this step.
Step 2: Pose Prediction:
A separate PoseNet takes the image pair (I,, Ie) input and predicts the relative camera pose (rotation and translation). Tr--.-
Step 3: Self-Supervised Signal Generation (Image Reprojection[48):
Using the predicted depth map D and the predicted pose Tr-ee we perform warping operation to reproject the pixels from the source image Ie onto the target image's 's coordinate system. This creates synthesized target image, i.-
Step 4: Loss Calculation and Optimization:
23

---

A photometric reprojection loss, Lph, is calculated by comparing the synthesized image I with the original target image 4.
This loss, Lp is the core self-supervised signal. If the predicted depth D is incorrect, the reprojected image i, will not match the original 4t, resulting in a high loss.
Step End-to-End Backpropagation:
The gradient from Lph is backpropagated through the entire computational graph. This means the gradient flows back to update the weights of both the PoseNet and, most importantly, the SD U-Net (fă).
Output: The predicted depth map D obtained from Step 1.
B.2 Training Pseudocode
Algorithm outlines the complete training procedure for Jasmine, incorporating all components including MIR, SSG, and the full loss computation (Corresponds to the pipeline in Fig. 2).
Algorithm Jasmine Training Algorithm
1: Initialize: SD U-Net, PoseNet, SSG, VAE, optimizer
2: VAE.eval()
b VAE weights are frozen
3: for each batch of inputs 'I' in dataloader do
4:
b Mix-batch Image Reconstruction (MIR)
5: Sdepth [1, 0]. Syecon [0,
Task switchers
6:
24, J 2m 2 &- "AE.eencoee(I] I[' m'], noise)
7:
Dees f VAE.decode(S)) (SD UNet([ [=] 20 999, s = sdepth))
8:
Lrec VAE.decode (SD_UNet({ ([2) m1 999, s 5recon))
9:
Scale-Shift GRU (SSG)
10:
Drset + SSG(D sait 2
Diist contains [D e D1, Dal Pose Estimation for Self-Supervision
11:
pose PoseNet(I['L ([ss)ee(("Le,,,I""L"")
[t]],[[[_""))
12:
loss +- compute loss(I, Disst pose, Iree)
b Loss Computation and Optimization
13:
optimizer.zero grad(
14:
loss. bosksbackward()
15:
optimizer.step()
16:
end for function NOMPUTE_LOSSSinpEe Drist, pose, Irea)
18:
Dai, Dzsi + Dis, Drise
reproj_imp, < reprojecttinpuuss[["""
pose)
20:
L, photometric_loxx(qqprojj img. inputs('I_t'))
Lph + photometric Low.etrrii_lss(rreeeeee insuutsss inputs("I (mmm)
b Core self-supervised signal Surrogate task signal b As per Eq. 11
22:
Lie & teacher loss inputs("D D tc'l)
La <- compute compute_auxiliary auxillaryyloss(ipuuus Dsi, Dasi)
24: loss <- T.ph Ty + Lue + L., 0.008
return loss
26:
end function
27:
function COMPUTE COMPUTE_AUXILLANNE AUXILIARY _iSSSsooosssssuusssss Dai, Desi)
28: LADS gds losslosssinpusssLLL [_t inputs( seg'], Dsi)
29:
LSKY t- sky_loss( D Dasi, inputs["skt sky_mask"
30: L. f edge loss( D...aaa.
31: return LGDS LSKY + Le
32:
end function
C Supervision Details
Self seff-supervised depth estimation has been researched for a decade, and hundreds of works have proposed numerous progressive ideas. Therefore, to achieve SoTA performance, inevitable that we will reuse some loss constraints from previous works optimize our results. In the following text, except for the edge loss, the other losses are mostly referenced from prior papers and are not the core contributions of this paper. Thus, have not included them in the main paper. The effectiveness of

---

the other papers' losses has been comprehensively proved in their papers, and we did not conduct additional ablation studies for them. For the loss specific in this paper, we present some visualized ablation results in Fig. 7.
C.1 SGD Loss for Dynamic Object
We first introduce the most commonly used smoothness loss Lsn in self-supervised depth estimation, which encourages locally smooth depth maps while preserving edges in the image. Its specific expression is as follows:
Lam = De -10.11 + 1,, j0, 0,11
(16)
where Î the depth normalized by the mean of D.
Building on this, we adopt the GDS loss (only the first stage) from to handle dynamic objects. This approach is based the smoothness loss and introduces ground-contact-pe mask Mgr. defined as:
Mir().)) = M...) + (1 M((i.))
where Mj is the dynamic object segmentation obtained from semantic segmentation network (1 represents dynamic and O for static), and y is the weighting parameter for MM empirically set to 100. Considering the bottom pixels of dynamic regions like the car tire, they impose a high weighting on [ with (i. = y, thereby enforcing its depth consists with its neighboring ground pixels below. So the final loss is:
LGDS 10,
10.1 100011 + 10, 10,11 10,11
(17)
C.2 Edge Loss for Sharp Prediction
We further introduce edge loss to enhance the prediction's details. Since the surrogate and primary tasks are decoupled after the U-Net's final layer, and the Scale-Shift Invariant (SSI) depth is shielded from photometric loss interference through SSG-based isolation, the SSI depth retains significantly richer structural details. To transfer these details to the final result, we introduce a simple edge loss. Specifically, we design GradNet,
GradNet(z) ((conv2d(r,, wz)), /conv2d(z; x. wy)l),
where Wr 11, 0, 2). (1,00, 1]] and wy = [[-1, 2, -1]. 00, 0. 0], 11 2, 1]] are the convolutional kernels for computing gradients the x and y directions. Subsequently, the edge loss is defined as:
Le = (C,,,, Dz) + LC Dy),
(18)
where (Cz, Cyy GradNet (Dsst) represents the normalized gradient of the detached SSI depth, and Dy) GradNet( (itee) represents the gradient of blur depth. We implement this edge loss to the SSG outputs D1 and SI depth Dz. The detached Dssi, while the depth distribution is inaccurate, has sharper edges, making it an excellent teacher. The prediction comparison with and without edge loss are shown Fig. 7 (d-4) and (d-3), respectively.
C.3 Sky Loss for Anti-artifact
Our experiments reveal that (Fig. (b-2)), although the edge loss significantly enhances the model's details, artifacts still appear outside object edges, particularly in the sky. This because the sky is a texture-less region, causing self-supervised models to produce erroneous estimates. However, this typically does not affect performance, as during testing, the sky is either cropped (Eigen Crop) or its ground truth depth is invalid and excluded from evaluation. To address this issue, introduce sky loss:
(19) where D is the predicted depth, Dmux is the maximum depth value, 1p is tensor of ones with the same shape as D, and May is the sky mask derived from the semantic segmentation. May is a weighting factor set to 0.1.
Laky 1/aky Li Dmax 1n, May),
Note that the sky loss does not add extra details. As shown in Fig. 7 (b-2) and (b-3), the original details are already captured by the model; the sky loss merely sets the sky depth to infinity.
25

---

(a-1)
(a-2) Segment
(h-1)
(b>2) wio Luay
(b-3) Ours
(a-3)
(a-4)
(-2)==
(d-1)
(d-2) Dest
(d-3) L.
(d-4) Ours
Figure 7: Qualitative Ablation Study of Adaptive Loss. Notably, while (a-2) demonstrates superior visual quality, it exhibits entirely inaccurate depth distribution. (a-4) is the result of E2E FT, which can serve pseudo-labe here.
RGB Image (Part)
Zuone Enlaance
MonoNiT
Weatherdepth
MonoXIFE
Jasmine
Figure 8: Qualitative results on zero-shot Scale-Invariant depth estimation.
C.4 Eliminating the Bottleneck Teacher Loss
In this section, we elaborate on the details of the implementation of teacher loss. We use MinoViT as the teacher model. The model estimates disparity dp, and thus we obtain the depth Dic through:
clip
dp, 0,
Duc
Dmaz
Dmin
D
where Dmin and Dmar are set to 0.1 and 100, respectivley. We find that the depth range at range [0,3] can already describe all information within the maximum depth.
26

---

For the SSI depth supervision, we perform normalization similar to Eq. 9 in paper:
Dic min(Dr) maxx(Di)) min 0.5
(eem(De))= -
(20)
Furthermore, to avoid the performance cciling imposed by the teacher model, we draw inspiration from D4RD[50], and employ adaptive filtering mechanism:
A
Mic min ph (I, c
T/step
where À a constant set 1.5, - is the warped target image using the teacher disparity Dic. and Khep = max(1, 1, 30 nstepnow// http//temmman moz) is dynamic factor that adjusts with training progress. This adaptive weight initially allows the model to converge across the entire depth map and subsequently filters out less accurate regions, mitigating the adverse effects of inaccurate pseudo-depth labels. Therefore, we have:
filter (Dx) = Die Mic:
In the supervision process, we introduce the BerHu loss LB and get better results:
if S c,
LB( =
(21)
if
where c = 0.2 max([2 - y/) is a threshold that adapts to the error magnitude. This loss imposes a greater penalty pixels with larger errors using an L--like penalty, while retaining the robustness of L1 loss for small errors.
Thus, The Eq. 13 in paper:
Lte = (Ln (norm (Dte), Dssi) + La (Dre, D1) T/
(22)
+ Lg (filter (Dıc) DsI) 7T2) Msepp
is fully detailed in this part. The teacher loss not only stabilizes our model but also avoids limiting its performance potential.
The segmentation model mentioned above is Mask2Former[6] throughout. To avoid the miscon- ception that the details stem from the segmentation network's output, we further expand Fig. 3 (in paper) with segment-basee methods. In fact, to preserve these details, a naive approach is to employ semantic segmentation constraints(34]. However, as shown in Fig. 7 (a-2), using the semantic triplet loss from [34] not only disrupts the depth distribution but also introduces spurious edges, rendering incompatible with SD's s latent priors. Furthermore, the results in paper Table are compared with a segmentation-bait model Jacon et al&, and our method achieves superior performance.
D Preliminaries of GRU
The Gated Recurrent Unit (GRU) is type of recurrent neural network (RNN) that uses gating mechanisms to control the flow of information between the previous hidden state and the current input, making it effective for sequence modeling tasks. Recently, GRUs have been gradually used in depth estimation[45, 43]. A GRU cell updates its hidden state =+1 based on the previous hidden state h and the current input x² This update process is primarily managed by two gates: the reset gate (r) and the update gate (z). The reset gate determines how much of the past information (from the previous hidden state) effectively "forget" "reset" before computing hidden state. The update gate then decides how much of this new candidate hidden state should be incorporated into the final hidden state, versus how much of the previous hidden state to carry over. The mathematical formulation for these operations as follows:
candidate
z A++ = a ((h* x*1, W=),
p++1 = o x*1 W,),
h +++ tanh (Conv(C @ xt, Wi)),
(23)
hi+1 (1 (1-2h+l) o h gttl @ h+1
where & denotes iteration steps, is the sigmoid activation function (outputting values between 0 and ideal for gating), and @ indicates element-wise multiplication. The terms W.. : W,, Wa, represent the learnable parameters (typically weight matrices and biases) for the update gate, reset gate, and
27

---

candidate hidden state computation, respectively. The Cone notation suggests that convolutional layers are used for these transformations, as common when applying GRUs to feature maps in computer vision tasks. In our paper, the refined hidden state ha+1 predicts depth adjustment Da to update DR. yielding DE+1 for subsequent iterations, as utilized in our Scale-Shift GRU (SSG) module (see Sec.3.3). 3.3).
E
Addition Experiments Results
E.1 Evaluation on KITTI Improved Benchmark
The standard self-supervised evaluation on the KITTI dataset is typically conducted using the raw LiDAR GT. However, due to the noisy nature of LiDAR data and known preprocessing issues", we also provide the eigen improved benchmark result. Following the practice adopted in Mono- ViFi [29], as shown in Table 5, this evaluation further confirms Jasmine's ' SoTA performance, where consistently outperforms other leading methods.
Table 5: Quantitative results on the KITTI dataset using the improved GT [47]. The performance of SD-based methods is not fully reported in their original papers.
Method
AbsRell
SqRel
RMSE]
RMSElog.
421
as²
Marigold [25]
0.099
0.916
0.987
E2E-FT [33]
0.096
0,921
0,980
Lotus [21]
0.081
0
0.987
Jasmine
0.064
0.294
2.982
0.097
0,957
0.994
0.998
MonoVi VIT [83]
0.068
0.314
3.125
0.105
0,948
0.992
0.998
MonoViFi [29]
0.071
0.338
3.539
0.113
0.937
0.990
0,998
Jasmine
0.061
0.255
2.765
0.092
0.963
0.995
0,999
F,2
Comparison with Other Supe-Supe Settings
As mentioned in Sec. our single-frame monocular approach more challenged but practical compared to other self-supervised configurations. Compared to stereo-based methods, monocular methods face additional challenges with dynamic objects and pose estimation inaccuracy, but monoc- ular methods can eliminate the need for synchronized binocular cameras and precise calibration. Similarly, single-framt frame models neglect temporal information during inference, while multi -ramm methods leverage consecutive frames to construct cost volumes and even support iterative refinement at test time, yielding improved accuracy. However, the practical deployment of multi-frame methods remains constrained by the availability of multiple frames.
Despite these inherent disadvantages, as shown in Table 4, our method still outperforms the state- of-the-art approaches in both multi-frame and stereo-supervision domains across most metrics. This remarkable achievement, especially considering our more challenging problem setting, further demonstrates the substantial strength and generalizability of our approach.
E.3 Qualitative Comparisons
Table 6: Ablation Studies the mix-batch ratio. Ph
refers to supervision using Eq. 5, while Latent refers to supervision using Eq. 4. The notation +A denotes the proportion of the KITTI dataset used (e.g., +0.3 indicates KITTI:Hypersim ratio of 3:7 in the MIR training data.)
In Fig. 8, we further compare the perfor- mance of our Jasmine with other methods in multi scenes. The quantitative results obviously demonstrate that method can produce much finer and more accurate depth predictions, particularly in complex regions with intricate structures, which sometimes cannot be reflected by the met-
(ID) Method
AbsRel
SqRel
RMSE
02
Ph+0
0.089
0,573
3.973
0.918
0.972
Ph+0.3
0.090
0.581
3.944
0.919
0.972
Ph+0.6
0.092
0.593
3,933
0.915
0.973
rics.
Ph+1
0.093
0.590
3,970
0.915
0.973
Latent+0
0.106
0.614
4.181
0.901
0,970
E.4 Mix-batch Ratio Ablation Details
Latent+0.3
0.095
0.606
4.138
0.909
0.970
Latent+0.6
0.121
0.649
4,322
0.876
0.970
Latent+1
0.129
0.679
4.385
0.858
0.962
In the paper, Fig. 3 (g) illustrates the per-
formance variation versus the mix-batch ratio under two supervision schemes. We further present
Discussed in MonoDepth2 repository: https://githus http:/tttw....ooocaccaaaaaaaaate
28

---

the complete results in Table 6. Note that although "Ph+0" offers better metrics, it predicts blurred results (7 (c-2)).
F Analysis of SD Prior Degradation via Self-Supervision
For clarity, we first revisit the core definition of disparity: when capturing two images of the same scene from different camera positions, the same point will appear at different pixel coordinates in each image. This difference is known as disparity. fact, disparity and depth are inversely proportional and correspond one-to-one.
From this perspective, the loss function in Eq. 1 designed to find, for each point the target view I, the most similar point in the source view Ig.. From the coordinate difference between these points Iy- we obtain the depth supervision (Eq. 2). However, if the corresponding point Iz is occluded in Iv, the optimization process is forced to select the "least bad" alternative, resulting in an incorrect match and, consequently, erroneous depth estimation.
Taking Fig. 3 (a) as an example, the scene consists of a infinite background (depth=oo), an orange rectangle (20m), and light blue tree (10m). Here, the camera only shifts horizontally between the target and source views. For point (tree), the correct disparity is 10 pixels (10m). For point p.r (rectangle), the correct disparity should be pixels (20m).
Due to camera movement, the tree in the source view occludes the correct matching point of p. To minimize photometric loss, the algorithm searches for the most similar region nearby and once again finds the orange area at p', which is 10 pixels (10m).
This error causes the expected depth edge on the right side of point disappear (right side's s depth are all 10m, reason same to p), resulting map is used as supervisory signal to guide the SD model, it effectively introduces "noisy" data. This forces the SD model learn and reproduce these incorrect and blurred boundaries, thereby undermining its strong prior knowledge of clear object boundaries.
blurred boundary the depth map. When such depth
G Evaluation Metrics
Similar to [15], we employ the following evaluation metries in our experiments,
AbsRel: Mail Edexa,. 1d - dgt /dgti
SqRel: Mail EdeMar l|d - dgt 11? /dge:
RMSE: V [M EdEmo l|d dgt lg:l
RMSElog: MMaill EdeM, [log(d)) -
az percentage of d such that max( d. dy1) 1.25' where dge and d denote the GT and estimated dze
pixel depth, Mat is the valid mask set to le - 3 < dgt < 80.
H Error Bar Analysis
As highlighted in Sec 3.4, directly training Jasmine can be unstable due to the SD's enormous size, joint training across mod- ules, and indirect self-supervisory mecha- nisms. To further ensure stability, we also implement module-wise freezing training strategy, which involves 3 key phases: In the beginning, we enable gradient updates for all network components. Subsequently, we freeze the SSG and Posenet modules to decouple depth-por pose optimization while maintaining fixed parameters. After achiev- ing convergence to suboptimal solution, we reintroduce gradient updates to these modules for final optimization. In the training procedure, the first training phase involved 15k steps,
10.54
91.99
$9.56
Abaitl m%
10.5
51
Figure Error Bar Analysis on KITTI Eigen test split. We conduct this analysis through multiple training runs and observe the performance oscillations after 25k steps.
29

---

while the second and third phases automatically transitioned, sharing an additional 10k steps for total of 25k training steps. As illustrated in Fig. 9, we demonstrate that the pseudo-label supervision and module-wise freezing training are particularly crucial for steady training in complex, multi-module self-supervised systems.
I Limitation and Future Work
As noted in Sec. 2, the ubiquitously available videos suggest that the self-supervised methods possess significant data advantages and working potential. In this paper. Jasmine is trained on only tens of thousands of data samples and only on driving dataset (KITTI), leaving room for further exploration in scaling up training data to other domains (industry, indoor, etc). We believe that if there really emerges "GPT" moment for 3D perception in the future, it will more likely involve self-supervised methods trained on videos rather than learning from annotated depth. Furthermore, we believe that the unsupervised Stable Diffusion fine-tuning paradigm proposed in this paper can be applied to other related fields, such as depth completion [65, 63, 66, 70, 71, 60], depth super-resolution [64, 54], Multi-view Stereo [74, 75, 72, 73J, Stereo Matching [56, 58, 57, 53] and Language Aid Depth Estimation [78, 80, 79, 82].
30


Reviews:
[{"review_id": "Ogh4ejnknE", "reviewer": "Reviewer_arKR", "summary": "This paper proposes the first self-supervised monocular depth estimation framework based on Stable Diffusion. It introduces Mix-batch Image Reconstruction to protect Stable Diffusion's visual priors by supervising the model with image reconstruction instead of noisy reprojection losses. Additionally, it designs a Scale-Shift GRU (SSG) to align SD’s scale-shift-invariant depth outputs with the scale-invariant requirements of self-supervised training. It achieves state-of-the-art results on KITTI and demonstrates strong generalization across diverse zero-shot scenarios.", "questions": "While the method performs well on KITTI and shows some cross-domain generalization, could the authors further evaluate its robustness on more diverse domains, such as indoor scenes?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nLeveraging the rich visual priors encoded in SD, such as semantic understanding, structural perception, and fine-grained texture modeling, is a natural and promising direction for enhancing monocular depth estimation. However, directly applying SD to self-supervised depth frameworks introduces significant challenges. This paper successfully addresses these issues by introducing MIR and SSG, making it the first to effectively integrate diffusion priors into self-supervised depth estimation. The approach establishes a new paradigm for the task, supported by thorough experiments, well-designed ablations, and strong performance in both quantitative metrics and qualitative results. Overall, I find the contributions meaningful and the methodology sound, and I would lean toward an Accept rating.\n\nWeaknesses\n1) Some baseline results in Table 1 are reported at 640×192 resolution, while Jasmine uses 1024×320 for both training and evaluation. This discrepancy is not clearly annotated in the table, which may mislead readers when comparing numbers directly.\n\n2) Including a 640×192 variant would facilitate direct comparison with prior work and help future research adopt and benchmark Jasmine more easily, especially in low-resource or real-time settings.\n\n3) The paper does not report inference latency or runtime performance, which is important considering that Jasmine relies on SD components and a recurrent GRU module.", "was_revised": true, "final_justification": "Most of my concerns, including fair comparison at the same resolution, inference latency, and indoor performance, have been satisfactorily addressed, and I have decided to maintain my original score.", "rebuttal": "# Overall\n\nWe sincerely thank you for your time and effort in providing such a constructive and detailed review. We are greatly encouraged that you found our **contributions meaningful**, our **methodology sound**, and that our work **establishes a new paradigm for self-supervised depth estimation.**\n\nYour suggestions regarding the fairness of benchmark comparisons, the inclusion of runtime analysis, and further evaluation of generalization on more diverse domains like indoor scenes are highly insightful. We fully agree that addressing these points will significantly strengthen our paper's clarity and completeness. Below, we provide a point-by-point response to your comments and will incorporate all the valuable changes into our final manuscript.\n\n# On Fairness of Benchmarking and Additional Experiments (W1, W2)\n\nWe have carefully checked Table 1 and ensured that all of the baseline results are reported at the same resolution of 1024x320.\n\nWe have included the performance evaluation of our retrained model at a resolution of 640 × 192 in Table A, alongside comparisons with other latest self-supervised models. The results continue to demonstrate the SoTA performance of our method.\n\n| Model             | Venue     | AbsRel ⬇️     | SqRel ⬇️      | RMSE ⬇️       |  RMSE_log ⬇️  | δ1 ⬆️        | δ2 ⬆️        | δ3 ⬆️        |\n| ----------------- | --------- | --------------- | --------------- | --------------- | :-------------: | --------------- | --------------- | --------------- |\n| Jaeho et al       | CVPR 2024 | 0.096           | 0.696           | 4.327           |      0.174      | 0.904           | 0.968           | 0.985           |\n| RPrDepth          | ECCV 2024 | 0.097           | 0.658           | 4.279           |      0.169      | 0.900           | 0.967           | **0.985** |\n| MonoViFi          | ECCV 2024 | 0.096           | 0.627           | **4.179** |      0.170      | 0.903           | **0.969** | **0.985** |\n| **Jasmine** | --        | **0.092** | **0.622** | 4.180           | **0.168** | **0.908** | **0.969** | **0.985** |\n\n`Table A.` Quantitative results on the KITTI dataset with 640x192 resolution. The best results are marked in **bold**.\n\n# On Inference Latency and Runtime Performance (W3)\n\nWe present the inference latency and computational complexity of the Jasmine model in Table B, alongside comparisons with both SD-based and self-supervised methods. Our results indicate that the inclusion of SSG has minimal impact on inference delay, as Jasmine and Lotus exhibit comparable inference time. Although incorporating SD does lead to a notable increase in computational complexity relative to other self-supervised approaches, Jasmine strikes a trade-off between performance and computational cost.\n\n|   Method:   | Marigold | Lotus | E2E-Mono | Monodepth2 | MonoViT | MonoViFi | Jasmine |\n| :----------: | :------: | :---: | :------: | :--------: | :-----: | :------: | :-----: |\n|  MACs ⬇️  |   133T   | 2.65T |  2.65T  |   21.43G   | 25.63G |  28.79G  |  2.83T  |\n| Runtime ⬇️ |  9.88s  | 157ms |  152ms  |    33ms    |  29ms  |   25ms   |  172ms  |\n\n`Table B.` Computational efficiency comparison. MACs and Runtime are measured under a single image with 1024x320 resolution on one RTX 4090 card. MACs is the number of core multiply-accumulate operations and MACs≈FLOPs/2.\n\n# On Generalization to More Diverse Domains (Q1)\n\nAs the **first work** to introduce SD priors into self-supervised depth estimation, in the paper, our experiments were initially focused on the most established self-supervised setting—the street view dataset. But to address the reviewer’s concerns, we performed a zero-shot evaluation of several **self-supervised** models on indoor datasets, as presented in Table C. Despite the substantial distribution shift between the training and testing data, Jasmine continues to exhibit strong performance, further underscoring the effectiveness and generalizability of our approach.\n\n| Model      | AbsRel ⬇️     | SqRel ⬇️      | RMSE ⬇️       |  RMSE_log ⬇️  | δ1 ⬆️        | δ2 ⬆️        | δ3 ⬆️        |\n| ---------- | --------------- | --------------- | --------------- | :-------------: | --------------- | --------------- | --------------- |\n| Monodepth2 | 0.447           | 1.514           | 1.845           |      0.469      | 0.407           | 0.682           | 0.841           |\n| MonoViT    | 0.249           | 0.324           | 0.896           |      0.287      | 0.607           | 0.870           | 0.961           |\n| MonoViFi   | 0.248           | 0.268           | 0.797           |      0.287      | 0.622           | 0.868           | 0.956           |\n| Jasmine    | **0.198** | **0.257** | **0.701** | **0.269** | **0.689** | **0.899** | **0.963** |\n\n`Table C.` Zero-shot evaluation on the NYUv2 dataset, test on the official test split."}, {"review_id": "4DJ4WY7wjB", "reviewer": "Reviewer_SPog", "summary": "This paper introduces Jasmine, a self-supervised monocular depth estimation framework that leverages Stable Diffusion (SD) priors to improve depth map sharpness and generalization, without requiring high-precision ground-truth supervision. The authors address the challenge of integrating SD priors in the self-supervised setting by introducing a mix-batch image reconstruction (MIR) surrogate task and a Scale-Shift GRU (SSG) module to reconcile scale-shift invariance differences between supervised and self-supervised estimates. Experimental results on KITTI and several zero-shot benchmarks demonstrate that Jasmine sets new state-of-the-art results for self-supervised methods and achieves compelling detail preservation in predictions compared to prior work.", "questions": "1. How exactly the SD is integrated with self-supervision paradigm? Not technically but intuitively, as L29-L34 and L119-L123 didn't provide direct motivation explanation of this point.\n2. Why the introducing of task switcher can address the problem of SD prior perturbation?\n2. As far as i understood, the task switcher let the SD the be trained with either depth map or color image, as the equation stated in L137 and L138. However, for the depth predicted by SD, it requires to be pretrained on z^y (as explained in Sec. 3.1) which is groundtruth depth, and that obviously make the SD becomes fully supervised.\n3. Due to the figure caused huge confusion, how exactly the self-supervision compromise SD prior need clear explanation.\n\nother problems:\n1. At L48-L49, there is no experiment or citation supporting the claim.\n2. At L164-L170, the first point states “synthetic images are not essential” which is totally contradict to the explanation in L155-L160.\n3. The self-reprojection supervision in L33 may should be the reprojection in self-supervision\n\nspecific to the figures:\n1. At L125, the Fig. 3 (a) makes me seriously doubt about how the self-supervision compromise SD prior. The explaination of the figure is useless, there are so many inconsistencies between the text and the figure. \n2. As Fig. 3 (g) is better to be comprehend than Fig. 3 (a), but the axes label should also be explained in main text as the missing of immediate reference makes it so confused to understand the main content.\n3. At L201, I cannot locate the corresponding DepthHead in Fig. 4 (a). And the right bottom part of this figure really confused me, what are those arrows from nowhere means? how the z^I integrate into the computation? Such confusion also appears when the author says in L207 “subsequently update D_k to D_k+1.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n    - Technical Soundness and Contributions: The paper addresses a long-standing challenge in self-supervised depth estimation—loss of fine detail and generalization—by creatively leveraging SD priors without ground-truth depth.\n    - Comprehensive Results: Extensive experiments are presented, including on the KITTI dataset and challenging zero-shot settings. Jasmine achieves state-of-the-art results among SSMDEs, with significant improvements reported over existing methods. The detailed ablation studies carefully disentangle contributions of key components (SD prior, MIR, SSG), clearly showcasing their necessity and complementary effects.\n\nWeaknesses:\n    - Poor Writing Logic & Figure Presentation: The writing logic makes me very confused, which greatly influence my judgment over the following technical motivation. Same to the writing, the figures (especially for Fig. 3 (a) (g) and Fig. 4 (a)) are not self-exlained and makes the comprehension harder.\n    - Unclear Technical Motivation: Through out the presentation, how the self-supervision are integrated into SD is unclear, even though I can make deduction by myself, but it should be very clear for a good submission.\n    - Confused Technical Implementation: There are concerns about the implementation of the techniques as stated in the Questions.", "was_revised": true, "final_justification": "The repeated existence of unclarity greatly hindered the quality of this submission which is critical for a real strong submission especially for NIPS-like top-tier venues. Thus, I prefer to keep the initial rating.", "rebuttal": "# Overview\n\nWe sincerely thank you for your time and for providing this critical feedback. We apologize that some of the preliminaries and figure presentation were unclear. Despite these major issues with clarity, we are encouraged that the reviewer still recognized that **our paper addresses a long-standing challenge in this area and shows strong experimental results.** Following the reviewer's guidance, we try to address every concern as follows.\n\n# Weaknesses: Writing Logic & Figure Presentation\n\nIn the paper, we first present our motivation: SSMDE suffers from poor generalization and limited prediction detail, whereas SD priors offer a promising solution. However, when introducing SD priors, we encountered 2 challenges: (1) self-supervised signals can disrupt SD priors, and (2) there are distributional differences between self-supervised depth and SD outputs (SI-SSI). We addressed them with MIR and SSG, ultimately enabling the effective integration of SD priors, which led to significant improvements in performance, generalization, and prediction detail. In the method section, after introducing the preliminaries on traditional SSMDE and SD protocol, we provide a detailed explanation of MIR and SSG. For MIR, we first discuss the challenges (1) principle, then describe the implementation details, and further expand MIR with diverse auxiliary images usage. For SSG, we also begin with the challenges (2) principle,  then elaborate on the design details. We hope this clarifies the overall writing logical of our paper. Questions regarding the figures will be addressed in 1️⃣,2️⃣,3️⃣ below.\n\n# How self-supervision introduce to SD\n\n> **Weaknesses: Unclear Technical Motivation\n> Questions 1.\n> Questions 3.**\n\nFirst, it is important to clarify that the self-supervision used here is distinct from the unsupervised learning of foundational vision models like DINOv2. In our context, self-supervision means the training process leverages the temporal and spatial relationships between sequential images to train the depth estimator, without GT depth.\n\nIntuitively, integrating SD with self-supervision fundamentally changes the source of supervision for the SD output. In the standard single-step denoising paradigm, the original SD model directly predicts depth, and during training, GT depth is used to supervise the U-Net’s output via an MSE loss. In our approach, however, the supervisory signal for the SD output comes from a reprojection loss. Specifically, we predict the relative pose between adjacent frames and use **SD output + pose** to reproject the adjacent frames onto the current input image. If the SD-predicted depth is accurate, the reprojected/reconstructed image should closely match the original input image. This photometric consistency serves as the self-supervised signal to guide depth learning.\n\nFor **Question 3**, in supervised SD-based depth estimation (e.g., Marigold), $y$ indeed represents the GT depth map. However, in our self-supervised framework, $y$ should be interpreted as **the target output that the model aims to generate, rather than a pre-existing supervised signal.** Specifically, as described in Sec 3.1, in our single-step denoising strategy, the U-Net $f_\\theta^z$ takes pure noise $\\mathbf{n}$ and an image latent variable $z^I$ as input, and the model's task is to learn a mapping that transforms $\\mathbf{n}$ into a meaningful depth map latent $z_0^y$. Thus, **the model never sees and never requires the 'correct answer' $z^y$**.\n\n# How self-supervision compromise SD prior\n> **Questions 4.\n1️⃣Specific to the figures 3 (a)**\n\nFor clarity, we first revisit the core definition of disparity: when capturing 2 images of the same scene from different camera positions, the **same point** will appear at different pixel coordinates in each image. This difference is known as *disparity*. Disparity and depth are inversely proportional and correspond one-to-one.\n\nFrom this perspective, the loss function in Eq. 2 is designed to find the most **similar point** in the source view $I_{t'}$ for each point in the target view $I_t$. From the coordinate difference between these points $I_{t'\\rightarrow t}$, we obtain the depth supervision (Eq.1). However, if the corresponding point in $I_t$ is occluded in $I_{t'}$, the optimization process is forced to select the \"least bad\" alternative, resulting in an incorrect match and, consequently, erroneous depth estimation.\n\nTaking **Fig 3(a)** as an example, the scene consists of an infinite background (depth=$\\infty$), an orange rectangle (20m), and a light blue tree (10m). Here, the camera only shifts horizontally between the target and source views. For point q (tree), the correct disparity is 10 pixels (10m). For point p,r (rectangle), the correct disparity should be 5 pixels (20m).\n\nDue to camera movement, the tree in the source view occludes the correct matching point of p. To minimize photometric loss, the algorithm searches for the most similar region nearby and once again finds the orange area at p', which is 10 pixels (10m).\n\nThis error causes the expected depth edge on the right side of point p to disappear (the right side's depth is all 10m, same reason as p), resulting in a blurred boundary in the depth map. When such a depth map is used as a supervisory signal to guide the SD model, it effectively introduces \"noisy\" data. This forces the SD model to learn and reproduce these incorrect and blurred boundaries, thereby undermining its strong prior knowledge of clear object boundaries.\n\n# Others\n\n> **2️⃣Specific to Fig 4(a)**\n\nSince we cannot include the revised figure in this rebuttal, we will provide a detailed textual explanation of the process below.\n\n**For DepthHead** We want to clarify that the relationship of $(D_k,h^k)$ is same to $(y,z^y)$  in the SD model. The SSG network does not directly process the original image, but the encoded features. The difference is that SD uses VAE for encoding and decoding, while GRU only uses one convolutional layer, so we ignored this structure in the figure. We thank you for pointing this out and will include it in the final version.\n\n**Arrows from nowhere** In fact, Fig 4(a) corresponds to the gray rectangle shown in Fig. 2, standing for an iteration within two consecutive ones. Thus, the origins and destinations of 'nowhere' arrows indicate connections to the preceding and following SSG iterations, respectively.\n\n**$z^I$ integration and $D_k$ update**\nTo better clarify these questions and Fig. 4(a), we detail the first iteration (k=0) below:\n\n1. We first encode the SD output $D_0(D_{SSI})$ to obtain the hidden state $h^0$.\n2. We use learnable $Q_{SC}/Q_{SH}$(as query) to interact with the $h^0$ (as key/value) with cross-attention and generate scale $s_c$ and shift $s_h$.\n3. The input $x^0$ is formed by concatenating the image features $z^I$ and the updated depth $s_c*D_0+s_h$.\n4. The GRU takes $h^0$ and $x^0$ as input and outputs next hidden state $h^1$. After decoding, depth residual $D_{\\delta}$ is obtained. Finally, we use Eq 10 to calculate the refined depth map $D_1$.\n\n> **3️⃣Specific to Fig 3(g)**\n\nWe agree that Figure 3(g) should be more self-contained, and we will revise its caption in the final version, like:\n\nIn sub-figure (g), the x-axis ($\\lambda$) is the proportion of the KITTI dataset used in training(e.g., 0.3 indicates a KITTI: Hypersim ratio of 3:7 in the MIR training data), the y-axes are the metrics of the performance, including AbsRel (Absolute Relative Error, lower is better) and A1 (Accuracy, higher is better), definitions detailed in Sec E. We show the performance variations under different $\\lambda$ settings for photometric supervision (Eq. 5, better) and latent supervision (Eq. 4, worse).\n\n> **Questions 2.**\n\nThe task switcher itself cannot solve the SD prior perturbation problem, but the surrogate task (MIR) enables can. As we explained in \"How self-supervision compromises SD priors,\" self-supervision introduces \"noisy\" supervision signals. The role of the switcher is to introduce \"clean\" ones. Specifically, the training target of the image reconstruction task is the input image itself, which contains the most perfect, sharpest, and structurally correct visual details. Therefore, it guides the model to focus on and preserve realistic edges, textures, and structures. During training, the \"bad gradients\" from the self-supervised loss and the \"good gradients\" from the image reconstruction task are mixed together. These \"good gradients\" act as a normalization or an anchor, constantly pulling the model's parameters back to a \"correct\" region capable of generating sharp, realistic details, effectively counteracting the negative impact of the \"bad gradients\" on the model's priors.\n\n> Other Problems 1.\n\nThis claim originates from the foundation work on self-supervised depth estimation by Godard et al. (2017), titled *\"Unsupervised Monocular Depth Estimation with Left-Right Consistency\"* (CVPR 2017). \n\n> other problems 2.\n\nAt L155-L160, the paper states that we utilize the synthetic Hypersim dataset as auxiliary image. However, using synthetic data does **NOT** mean they must be used. As demonstrated in our experiments (Table 3, (d), (13), L325-l327), using high-quality real images can achieve similarly effective results. So they are not contradictory.\n\n> other problems 3.\n\nWe will fix it in the final version."}, {"review_id": "oBvilTkiiU", "reviewer": "Reviewer_fCDa", "summary": "This paper presents a novel approach to utilize Stable Diffusion (SD)'s visual prior to significantly enhance the accuracy and generalizability of self-supervised monocular depth estimation (SSMDE). Leveraging SD priors in SSMDE faces challenges due to artifacts from reprojection loss and misalignment between Scale-Shift-Invariant (SSI) depth and Scale-Invariant (SI) depth. Experiments on the KITTI dataset achieves best performance compared to other SSMDE methods. Zero-shot evaluation on StereoDriving and Cityscapes datasets shows remarkable generalization ability also in out-of-distribution diverse weather conditions.", "questions": "Q01. SD-based comparison. For fair comparison with the SD-based methods, it is recommended to conduct the training and evaluation following their protocols, e.g. in LOTUS the KITTI images are cropped to 1216 x 352.\n\nQ02. Indoor evaluation. It is interesting to see if the proposed method can also work for indoor SSMDE tasks, which is missing in the paper.\n\nQ03. Evaluation on (more) reliable GT. Extra evaluation on the KITTI improved ground truth is needed for more reliable comparison.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS01. SD priors for SSMDE. The well-written paper introduces a novel approach to harness the SD visual prior for SSMDE by addressing the challenges of preserving the SD latent space and aligning SSI and SI depth, which effectively enhances the accuracy and sharpness of SSMDE results and meaningfully propagates the usage of SD priors in the self-supervised framework.\n\nS02. Motivation and justification. The approach in anchoring the SD priors is intuitively appealing and well-justified by in-depth analysis and visualizations, demonstrating the effectiveness of the proposed surrogate task and stability to mixture ratio and different size of the auxiliary dataset. The motivation and design of the SSG are well-explained with solid derivations and straightforward visualizations, and the method is shown to effectively tackle the misalignment issue and integrate the SD outputs to SSMDE framework.\n\nS03. Strong experimental results. Strong qualitative and quantitative results in evaluations on KITTI dataset and zero-shot evaluations, showing the effectiveness of the proposed method in improving depth estimation accuracy and sharpness, and generalization ability across diverse weather conditions and high-dynamic data.\n\nWeaknesses:\n\nW01. Resolution discrapancies in evalution. The image resolution in KITTI and zero-shot evaluation is following common practice in SSMDE. However, the evaluation resolution used by SD-based methods, namely Marigold, E2E FT, and Lotus, is different, which causes a notable performance gap between those metrics reported in this paper and the original papers. Therefore, the comparison is not entirely fair, since the depth estimation performance can be sensitive to the input image size.\n\nW02. Missing indoor evaluation. Indoor data is often considered challenging for SSMDE due to the change of space scale and complex egomotion. With the proposed method, it would be interesting to see if the strong SD priors can also help in indoor SSMDE tasks, which is missing in the paper.\n\nW03. GT choice on KITTI. The evaluation on the KITTI dataset is only conducted on the raw LiDAR ground truth, however, due to noisy nature of LiDAR data and preprocessing issue reported [e.g. in MonoDepth2 Github Issue 274](https://github.com/nianticlabs/monodepth2/issues/274), it is recommended to also evaluate on the KITTI improved ground truth for more reliable comparison as done in Mono-VIFI.\n\nW04. Missing experiments on perturbation of SD priors. With the approach of protecting the SD latent space from perturbations, the SD is ought to be least influenced and still able to conduct the original task as image generation. However, experiments on perturbation of SD priors are missing, which can be interesting in order to analyze the effect of the proposed method on Stable Diffusion, e.g. by image reconstruction metrics.", "was_revised": true, "final_justification": "The paper provides interesting insights for the use of SD priors in a self-supervised domain backed up by a through motivation and clear experiments. The zero-shot performance on NYUv2 (W02/Q02) is impressive and the reviewer is very happy for adding these to the content of the paper / supplementary material. Although an additional quantitative result would provide additional insights for the SD prior perturbation, the reviewer thinks it is not urgently necessary for presentation. The paper contributions (in particular S01, S02, S03) justify a publication at NeurIPS. The reviewer therefore recommends acceptance.", "rebuttal": "# Overview\n\nWe sincerely thank you for your thorough and highly constructive feedback on our manuscript. We are greatly encouraged by your positive assessment and are pleased that you found our work to be a **\"novel approach\" (S01)** with **appealing motivation and well-justified analysis(S02)**, supported by **\"strong experimental results\" (S03)**.\n\nWe especially appreciate the valuable and actionable suggestions for improvement. In our revision, we have worked to address all the weaknesses and questions raised, including:\n\n# W01, Q01\n\n**Fair comparison with SD-based methods under specific evaluation resolution.**\n\nFirst, the reviewer noted that there is a performance gap between the results reported in our paper and those in the original work. It is primarily **due to differences in the GT used for evaluation, NOT the input resolution.** We will address this issue in detail in the W03 analysis.\n\nSecond, we would like to clarify that for SD-based **supervised** models—particularly those trained on the vKITTI dataset (cropped to $1216 \\times 352$) and the Hypersim dataset (resized to $480 \\times 640$), such as Marigold and Lotus—the impact of input resolution on depth estimation performance is relatively limited. In contrast, self-supervised models are relatively sensitive to resolution changes due to the strong dependence of photometric reprojection losses on image details.\n\nAs illustrated in Table D, the performance of supervised models remains close across resolutions of $1024 \\times 320$ and $1216 \\times 352$, whereas self-supervised models experience a significant drop in accuracy at higher resolutions. (note: the $1216 \\times 352$ resolution follows the Marigold, images cropped from the original ones.)\n\n| Model    | Marigold    |          | Lotus       |          |  | MonoViFi    |          | MonoViT     |          |\n| -------- | ----------- | -------- | ----------- | -------- | - | ----------- | -------- | ----------- | -------- |\n|          | AbsRel ⬇️ | δ1 ⬆️ | AbsRel ⬇️ | δ1 ⬆️ |  | AbsRel ⬇️ | δ1 ⬆️ | AbsRel ⬇️ | δ1 ⬆️ |\n| 1024x320 | 0.120       | 0.874    | 0.110       | 0.892    |  | 0.093       | 0.909    | 0.096       | 0.908    |\n| 1216x352 | 0.122       | 0.871    | 0.111       | 0.891    |  | 0.106       | 0.886    | 0.110       | 0.888    |\n\n`Table D.` Performance comparison at different resolutions. **Marigold/Lotus are SD-based supervised models, and MonoViFi/MonoViT are self-supervised models.**\n\nDespite the inherent disadvantages of self-supervised training under these conditions, we leveraged the strengths of SD priors and evaluated our model at $1216 \\times 352$ (while it was trained at $1024 \\times 320$). As shown in Table E, our approach still outperforms SD-based supervised models, highlighting the effectiveness of our method.\n\n| Model    | AbsRel ⬇️     | SqRel ⬇️      | RMSE ⬇️       |  RMSE_log ⬇️  | δ1 ⬆️        | δ2 ⬆️        | δ3 ⬆️        |\n| -------- | --------------- | --------------- | --------------- | :-------------: | --------------- | --------------- | --------------- |\n| Marigold | 0.122           | 0.813           | 4.404           |      0.208      | 0.871           | 0.961           | 0.982           |\n| Lotus    | 0.111           | 0.615           | 3.802           |      0.174      | 0.891           | **0.971** | 0.986           |\n| E2E-FT   | 0.131           | 0.790           | 4.338           |      0.206      | 0.855           | 0.962           | 0.982           |\n| Jasmine* | **0.107** | **0.605** | **3.751** | **0.170** | **0.897** | 0.969           | **0.986** |\n\n`Table E.` Performance comparison under the Lotus $1216 \\times 352$ evaluation protocols on the KITTI benchmark.\n\n# W02, Q02\n\n**New experiments and analysis on indoor datasets.**\n\nAs the **first work** to introduce SD priors into self-supervised depth estimation, our experiments were initially focused on the most established self-supervised setting—the street view dataset. The primary challenge in indoor self-supervised depth estimation arises from frequent and significant rotational motion (whereas KITTI is predominantly translational), which necessitates improvements to the pose network. However, this aspect is beyond the scope of our core contribution, as our work does not modify the pose estimation network.\n\nImportantly, we argue that **evaluating the benefit of strong SD priors for indoor SSMDE tasks is less about training on indoor data and more about assessing the generalization capability on it.** To this end, we performed a zero-shot evaluation of several self-supervised models on indoor datasets, as presented in Table C. Despite the substantial distribution shift between the training and testing data, Jasmine continues to exhibit strong performance, further underscoring the effectiveness and generalizability of our approach.\n\n| Model      | AbsRel ⬇️     | SqRel ⬇️      | RMSE ⬇️       |  RMSE_log ⬇️  | δ1 ⬆️        | δ2 ⬆️        | δ3 ⬆️        |\n| ---------- | --------------- | --------------- | --------------- | :-------------: | --------------- | --------------- | --------------- |\n| Monodepth2 | 0.447           | 1.514           | 1.845           |      0.469      | 0.407           | 0.682           | 0.841           |\n| MonoViT    | 0.249           | 0.324           | 0.896           |      0.287      | 0.607           | 0.870           | 0.961           |\n| MonoViFi   | 0.248           | 0.268           | 0.797           |      0.287      | 0.622           | 0.868           | 0.956           |\n| Jasmine    | **0.198** | **0.257** | **0.701** | **0.269** | **0.689** | **0.899** | **0.963** |\n\n`Table C.` Zero-shot evaluation on the NYUv2 dataset, test on the official test split.\n\n# W03, Q03\n\n**Evaluation and Comparison at the KITTI improved GT**\n\nThank you for the additional reminder! As shown in Table D, we report a performance comparison between Jasmine, SD-based methods, and self-supervised models on the KITTI improved GT benchmark, and we have consistently achieved the SoTA level. The inconsistency in metrics mentioned in W01 has also been resolved here.\n\n| Model    | AbsRel ⬇️ | SqRel ⬇️ | RMSE ⬇️  | RMSE_log ⬇️ | δ1 ⬆️   | δ2 ⬆️   | δ3 ⬆️   |\n| -------- | ----------- | ---------- | ---------- | :-----------: | ---------- | ---------- | ---------- |\n| Marigold | 0.099       | -          | -          |       -       | 0.916      | 0.987      | -          |\n| E2E-FT   | 0.096       | -          | -          |       -       | 0.921      | 0.980      | -          |\n| Lotus    | 0.081       | -          | -          |       -       | 0.931      | 0.987      | -          |\n| Jasmine* | **0.064**  | **0.294** | **2.982** |  **0.097**  | **0.957** | **0.994** | **0.998** |\n| MonoViT  | 0.068       | 0.314      | 3.125      |     0.105     | 0.948      | 0.992      | 0.998      |\n| MonoViFi | 0.071       | 0.338      | 3.539      |     0.113     | 0.937      | 0.990      | 0.998      |\n| Jasmine  | **0.061**  | **0.255** | **2.765** |  **0.092**  | **0.963** | **0.995** | **0.999** |\n\n`Table F.` Quantitative comparison between Jasmine and SD-based/self-supervised methods on the KITTI dataset using improved GT.\n\n# W04\n\n**Experiment on SD prior perturbation.**\n\nThe motivation for protecting SD priors stems from the observation that self-supervised training can disrupt the integrity of these priors. The supervised approaches, like Marigold, can successfully leverage synthetic data to *keep the SD latent space intact (Original sentence from Marigold paper)*. However, even models like Marigold are **NOT** directly applicable to the original image generation task, as their adaptation for depth estimation does not preserve the generative pathway required for RGB pixel synthesis. Instead, the inheritance of SD priors in Marigold is reflected through improved prediction details and enhanced generalization ability. And both of these points are embodied in and ablated in our paper, undoubtedly proving the protection of SD priors.\n\nHowever, we agree that a more direct evaluation of SD prior perturbation — through image reconstruction metrics — would provide additional insights, and it should be done in image generation fine-tuning. We consider this a valuable direction for future work.\n\n# End\n\nWe believe these additions have significantly strengthened the paper. The above results will be included in the final manuscript."}, {"review_id": "M8yFmWVeGn", "reviewer": "Reviewer_s3Km", "summary": "The manuscript proposes Jasmine, the Stable Diffusion (SD) based self-supervised monocular depth estimation framwork that utilizes SD’s visual priors to enhance the sharpness and generalization of unsupervised prediction. Jasmine leverages a mix-batch image reconstruction task to preserve the detailed latent priors of SD models. Additionally, a Scale-Shift GRU is introduced to for learning the scale misalignment from the pseudo-lables.", "questions": "•\tProvide a detailed literature review to substantiate the novelty of the mix-batch reconstruction and Scale-Shift GRU (how SST works).\n\n•\tInclude ablation studies to validate $L_{GDS}$ and $L_{SKY}$, comparing their impact to existing techniques for dynamic objects and noise mitigation.\n\n•\tPresent a comprehensive comparison across multiple datasets (e.g., Cityscapes, Waymo Open) and resolutions. \n\n•\tEvaluate performance with other teacher models (e.g., monodepth2, LiteMono and Mono-VIFI) and without a teacher model to demonstrate the method’s independent effectiveness.\n\n•\tReport computational requirements and inference latency, comparing Jasmine to other models.", "limitations": "please refer to the weakness", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Paper Strengths:\n\n1.\tThe introduction is clear in describing the addressed problem and describing the proposed method. Differences to existing work are clearly discussed in the related work section.\n\n2.\tThe method description is clear with detailed mathematical descriptions for the proposed loss components.\n \n3.\tThe proposed Jasmine reprensents an interesting and structured approach to levrage Stable Diffusion model’s visual priors to obtain fine-grained depth prediction.\n\n\nPaper Weaknesses:\n\n1.\tThe proposed mix-batch image reconstruction task appears to be a straightforward adaptation of Stable Diffusion’s Img2Img pipeline, with minimal modifications for depth estimation. Similarly, the use of synthetic images as training inputs directly mirrors strategies in Marigold [1] Ke B, Obukhov A, Huang S, et al. Repurposing diffusion-based image generators for monocular depth estimation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 9492-9502, offering little to no novel contribution to the field. \n\n2.\tThe paper employs $L_{GDS}$ from [2] Moon J, Bello J L G, Kwon B, et al. From-Ground-To-Objects: Coarse-to-fine self-supervised monocular depth estimation of dynamic objects with ground contact prior[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 10519-10529. and $L_{SKY}$ as part of the proposed framework but fails to include ablation studies to quantify their impact on performance. While $L_{GDS}$ reportedly enhances performance on KITTI by handling dynamic objects, and $L_{SKY}$ may reduce noise. Their discussion is limited to the appendix, reducing novelty and transparency and the lack of empirical validation obscures the method’s contributions. \n\n3.\tThe training process requires sequential optimization of multiple networks (Stable Diffusion model, Scale Shift Transformer and Sep-Conv GRU and Posenet) with numerous loss functions ($L_s$, $L_{ph}$, $L_{tc}$ and $L_{GDS}$, $L_{SKY}$, $L_{e}$) and multiple switch overs. This complexity may hinder reproducibility (no code available) and deployment. \n\n4.\tWhile Stable Diffusion models offer rich visual priors, their large model size introduces significant computational burdens and high inference latency, which the paper fails to address explicitly. This omission limits the fairness of comparisons with lighter models (e.g., CNN-based self-supervised methods) and obscures practical deployment constraints.", "was_revised": true, "final_justification": "I think the authors have addressed my concerns to some extent, I would like to upgrade my rating as borderline accept", "rebuttal": "# Overview\n\nWe thank you for your feedback. We appreciate that you recognized our work as **\"clearly described\"** and an **\"interesting and structured approach.\"** Regarding the concerns raised, we found the feedback on our ablation studies and computational cost analysis invaluable for strengthening the paper. However, on the core assessment of our novelty/transparency, we must respectfully beg to differ. Due to space constraints, some tables below are included in other reviewers' rebuttals, and the ablation table only highlights essential metrics.\n\n# On Novelty: Differentiating from Prior Art\n\n> Weaknesses 1.\n> Questions 1.\n\nJasmine is the **first** SD-based SSMDE framework, and our contribution can be located at Lines 65-71. **For the MIR:**\n\n1. We have made numerous modifications: within each training batch, the same SD model alternately reconstructs synthesized or real images, as well as predicts depth maps through a switcher; the reconstructed images are supervised using a photometric loss rather than the traditional latent MSE loss.\n2. Our motivations also differ fundamentally from those of SD: Self-supervision tends to erode the prior knowledge embedded in SD, making it unsuitable for direct application as in Marigold. Under this motivation, we introduced MIR to preserve this prior **, with important modifications for SSMDE**.\n3. We conducted extensive ablation experiments on the use of auxiliary images, which yielded valuable insights. It is worth noting that MIR is only one aspect of our overall contribution.\n\n**For synthetic images usage,** Marigold must use synthetic data to keep its latent space intact, as noise-free, dense depth GT is extremely rare. In contrast, our Jasmine model employs real KITTI images for its depth training, and its auxiliary images can be flexibly selected from either real or synthetic sources.  This is because Jasmine's self-supervised process does not require high-quality GT, but supervised Marigold does. Therefore, we actually do **NOT** mirror Marigold, but a pioneering work that effectively integrates diffusion priors into SSMDE.\n\nIn addition to MIR, the improvement of SSG is also the **first** work in self-supervision to analyze the difference between the two types of depth de-normalization. We provide a detailed analysis of these differences in Sec B, which, to our knowledge, is unprecedented in the literature. The SST module uses cross-attention to learn the scale/shift parameter from the last iteration prediction, as referred to the reviewer `SPog` rebuttal for more details.\n\nFinally, we note that the reviewer `arKR` highlighted our work as 'the first to integrate diffusion priors into SSMDE' and as 'establishing a new paradigm for the task.' Reviewer `SPog` remarked that our approach 'addresses a longstanding challenge in self-enhanced depth estimation,' and reviewer `fCDa` stated that we 'introduce a novel approach to harness the SD visual priors for SSMDE.' Collectively, these comments affirm that our work is innovative and makes meaningful contributions to the field.\n\n# Missing Ablation Studies\n\n> Weaknesses 2. Questions 2.\n\nWe would like to clarify that $L_{GDS}$, $L_{e}$, and $L_{sky}$ are **NOT** the core contributions of our work, which is why they are presented in the appendix. SSMDE is a well-established field, and many of its challenges have already been addressed by prior research. The loss functions mentioned in the appendix are **mature, widely adopted solutions,** and our intention is **NOT** to reinvent these established methods.\n\nSecond, we have already visualized the ablation results for $L_{sky}$ and $L_{e}$ in Appendix Fig. 7 (b-2, b-3, d-3, d-4). To further address the reviewer's concerns, we have also provided ablation results in Table G. Without these losses, Jasmine remains at the SoTA level.\n\n|Model|AbsRel⬇️|RMSE⬇️|δ1⬆️|\n|-|-|-|-|\n|w/o$L_{GDS}$|0.092|4.121|0.912|\n|w/o$L_{e}$|0.090|4.055|0.915|\n|w/o$L_{sky}$|0.091|3.990|0.917|\n\n`Table G.` Ablation study on $L_{GDS}$,$L_{sky}$ and $L_{e}$, evaluation rules are same with Table 1 in the paper.\n\nFinally, another experiment is designed to **compare the performance of our improved SD priors against traditional frameworks, both building upon these losses.** In fact, in our paper, Table 1 (Jaeho et al.\\*) shows the performance of the MonoViT framework with these mature Losses. Our method consistently outperforms Jaeho et al.\\*. Here, we further include a zero-shot performance comparison in Table H, which confirms that incorporating the improved SD prior significantly enhances the model's generalization. These results show that the performance depends mainly on our new paradigm, rather than on these mature Losses.\n\n|Model|DrivingStereo||CityScape||\n|-|-|-|-|-|\n||AbsRel⬇️|δ1⬆️|AbsRel⬇️|δ1⬆️|\n|Jaeho et al.\\*|0.143|0.833|0.132|0.821|\n|Jasmine|0.136|0.850|0.123|0.852|\n\n`Table H.`\n\n# Training complexity\n\n> Weaknesses 3.\n\nDue to the rebuttal policy, we regret that we are forbidden to release our full code at this time. However, to enhance transparency, we have provided detailed pseudocode in our rebuttal and commit to releasing our code upon paper acceptance.\n\n```\nSD_UNet, PoseNet, SSG, VAE = init_models()\noptimizer = init_optim(...)\nVAE.eval()\nfor inputs in dataloader:\n #MIR\n s=[[0,1],[1,0]]#switcher\n z^I_t, z^I_m, z^n = VAE.encode(inputs['I_t'],inputs['I_m'],noise)\n D_ssi = VAE.decode(SD_UNet([z^I_t,z^n], t=999,s=s[0]))\n I_rec= VAE.decode(SD_UNet([z^I_m,z^n], t=999,s=s[1]))\n #SSG\n D_list = SSG(D_ssi, z^I_t) #D_list=[D_ssi,D_1,D_si]\n #SSMDE Pose\n pose = PoseNet(inputs['I_t'], inputs['I_(t')'])\n #Loss\n loss = compute_loss(inputs, D_list, pose,I_rec)\n #Optim\n loss.backward(), optimizer.step()\ndef compute_loss(inputs,D_list, pose,I_rec):\n D_si,D_ssi=D_list['D_si'],D_list['D_ssi']\n reproj_img = reproject(inputs['I_(t')'],D_si, pose)\n L_ph = ph_loss(reproj_img, inputs['I_t'])\n L_s = ph_loss(I_rec, inputs['I_m'])\n L_tc= tc_loss(D_list,inputs['D_tc'],type='berhu')#Eq 11\n L_a= compute_L_a(inputs,D_si,D_ssi)\n loss =L_ph + L_s + L_tc+L_a*0.008\n return loss\ndef compute_L_a(inputs,D_si,D_ssi):\n L_GDS = gds_loss(inputs['I_t'],inputs['seg'],D_si)\n L_SKY = sky_loss(D_ssi, inputs['sky_mask']) \n L_e = edge_loss(D_si,D_ssi)\n return L_GDS+L_SKY+L_e\n```\nOur training process is fully **end-to-end**. While multiple network modules are involved, only **3** core components require optimization: the U-Net of Stable Diffusion, PoseNet, and our proposed SSG. All modules are jointly updated using a single optimizer `optimizer.step()`, as illustrated in the provided pseudocode.\nAlthough there are multiple loss functions, they are all based on well-established and open-source prior works in the field(`ph_loss-> Monodepth2, sgd_loss->From Ground to Objects). This design choice ensures that our method remains conceptually straightforward to implement.\nWith the provided pseudocode and clear references to existing methods, we believe our approach is logically transparent. \n\n# Computational costs and actual deployment\n> Weaknesses 4. Question 5.\n\nWe acknowledge that SD-based methods introduce higher computational overhead, inference latency and affect the fairness of comparisons with traditional CNN-based SSMDE. But we argue this has a limited impact on our contribution.\n\nFirst, we report the computational requirements and inference latency in Table B (refer to rebuttal for reviewer `arKR`), comparing Jasmine with both self-supervised and SD-based models.\n\nSecond, we wish to position our work in current MDE research trends. Recently, a series of leading works represented by **Marigold (CVPR 2024 Best Paper Candidate), Lotus(ICLR 2025) and DepthAnything (CVPR/NIPS 2024)** have established a research paradigm within the community: **there is a willingness to accept a certain level of computational cost at this stage in exchange for the unprecedented generalization and quality afforded by Large Vision Models(LVMs).** Our work shares the same philosophy as these studies. Therefore, the main contribution of our paper is not to pursue the ultimate inference speed, but to **explore and successfully introduce the powerful prior of SD into an SSMDE framework for the first time, aiming to solve the long-standing problems** of weak generalization and poor detail recovery in this field.\n\nThird, for **Application Potential**, the rapid adoption of Marigold by 30+ downstream tasks demonstrates that **even computationally intensive models can find widespread application if their performance is significant enough.** Similarly, we believe Jasmine, which does not require depth GT, has strong potential for real-world applications where annotated data is limited but leveraging LVMs is desirable.\n\n# Others\n> Question 3.\n\nTables A and E (see rebuttal for reviewers `arKR` and `fCDa`) provide a detailed performance comparison at 640×192 and 1216×352 resolutions. Results on the Cityscapes dataset are already reported in Table 2 of the main paper. Since we have performed zero-shot evaluations on several street-view datasets, we believe that additional testing on a similar Waymo dataset (less used in this community) would offer limited insights. Instead, to better address your generalization concerns, we conducted evaluations on indoor datasets, as presented in Table C (see rebuttal for reviewers `arKR`).\n\n> Question 4.\n\nTable J reports the results when using monodepth2 as the teacher model, as well as the performance of our method without any teacher model. Due to time constraints, we are unable to perform comprehensive training with additional teacher models and choose the monodepth2 (a relatively worse one that puts our model at a disadvantage). We find that the teacher model has a trivial impact on final results, as it serves to stabilize the initial training phase. However, omitting the teacher model leads to suboptimal performance.\n\n|Model|AbsRel⬇️|RMSE⬇️|δ1⬆️|\n|-|-|-|-|\n|monodepth2 as teacher|0.090|3.955|0.918|\n|w/o teacher|0.093|4.432|0.910|\n\n`Table J`"}]

Author rebuttal:

Meta-review: This paper received mixed but mostly positive reviews (1 borderline reject, 1 borderline accept, 2 accepts).

The paper addresses one of the primary weaknesses of self-supervised depth estimation through the use of the generative prior and adds a method to preserve that prior (cross-task training or more simply co-training) as well as introducing a new architectural element that addresses the issue of scale and shift prediction. The reviewers praised the task as well as results, with the only reservation being the presentation of the paper.

After reading the paper, reviews, rebuttals, and discussions, I advocate for acceptance. I believe this is a very good paper that addresses an interesting task. The only thing that is preventing me from recommending a higher status than acceptance is that it is not clear how important the task of self-supervised depth is anymore due to the emergence of geometric foundation models like Dust3r and VGGT. While it's not this paper's fault that they do not address this, that particular lack of comparison and understanding prevents me from recommending something higher than acceptance.
Decision: Accept (poster)