Depth Any Video with Scalable Synthetic Data
Authors: Honghui Yang
Di Huang
Wei Yin
Chunhua Shen
Haifeng Liu
Xiaofei He
Binbin Lin
Wanli Ouyang
Tong He
Venue: ICLR 2025 Poster (2025)
Keywords: Video Depth Estimation
Synthetic Game Data
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse virtual environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates—even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency. The code and model weights are open-sourced.

Published as a conference paper at ICLR 2025
DEPTH ANY VIDEO WITH SCALABLE SYNTHETIC DATA
Honghui Vangl.2,, Di Huang", Wei Yin², Chunhua Shen', Haifeng Liu't Xiaofei He', Binbin Lin Wanli Ouyang', Tong Hea
1 State Key Lab of CAD&CG, Zhejiang University Shanghai Laboratory
School of Software Technology, Zhejiang University 4The University of Sydney
https://depthhanyvidee gi itubbboo
Figure 1: We present Depth Any Video, versatile foundation model supporting both image (top half) and video (bottom half) depth estimation. Derived from Stable Video Diffusion and fine-tuned with diverse and high-quality synthetic data, our model achieves remarkably robust generalization across various real and synthetic unseen scenarios. Additionally, faithfully captures intricate fine-grained details while ensuring temporal consistency throughout the video.
ABSTRACT
Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, introduce Depth Any Video, model that tackles the challenge through two key innovations. First, we develop scalable synthetic data pipeline, capturing real-time video depth data from diverse virtual environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-

---

Published as a conference paper at ICLR 2025
world videos effectively, integrating advanced techniques such rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-lengte video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates- even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high- high-resolution video depth across sequences of up 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency. The code and model weights are open- ppen-soureed.) |
INTRODUCTION
Depth estimation is a foundational problem in understanding the 3D structure of the real world. The ability to accurately perceive and represent depth in video sequences is crucial for a broad range of applications, including autonomous navigation (Borghi et al. 2017), augmented reality (Holynski &
and advanced video editing (Zhang et 2002, Peng et al. (024). Although recent
advancements in single-image depth estimation (Ke et al. 2024] Yang et al. 2022aa Fu et al. 20224]
Ranfd et al., 2021) have led to significant improvements in spatial accuracy, ensuring temporal consistency across video frames remains a substantial challenge.
A major bottleneck in existing video depth estimation (Wang et al., 2023 Shao et al. 2024) is the lack of diverse and large- scale video depth data that capture the complexity of real-world environments. Existing datasets (Geiger et al. 2012; Li ct al. 2023 Karaev et al. 2023) are often limited in terms of scale, diversity, and scene variation, making it difficult for models to generalize effectively across different scenarios. From a hardware perspective, depth sensors like LiDAR, structured light systems, and time-of-flight cameras can provide accurate depth measurements but are often costly, limited in range or resolution, and struggle under specific lighting conditions or when dealing with reflective surfaces. Another common approach (Wang et al. [023] Hu et al. 2024b) is to rely on unlabeled stereo- matching methods (Jing et al., 2024; Xu et al., 2023); ; however, such methods are complex, computationally intensive, and often fail areas with weak textures. These limitations hinder the development of robust models that can ensure both spatial precision and temporal consistency in dynamic scenes.
stereo video datasets (Alahari et al.) (003)) and state-of-the--
To tackle the challenge, we propose a solution from two complementary perspectives: (1) demon- strating a scalable data collection pipeline to expand data and (2) designing novel framework that leverages powerful visual priors of generative models to effectively handle various real-world videos.
Synthetic Data: Modern virtual environments offer highly realistic graphics and simulate diverse real-world scenarios. For example, driving simulations accurately recreate real real-world road conditions, while open-ended virtual worlds depict various complex scenes. Given that modern graphics rendering pipelines often incorporate depth buffers, real-time rendering engines (e.g., Unreal Engine, Unity) offer functionalities for generating and accessing high-quality video depth data, enabling its extraction during interactive sessions. In light of this, we construct DA-V, synthetic dataset comprising 40,000 video clips collected from diverse interactive virtual environments over two weeks by 10 human players. DA-V captures a wide range of scenarios, covering various lighting conditions, dynamic camera movements, and intricate object interactions in both indoor and outdoor environments, providing the opportunity for models to generalize effectively to real-world environments.
Framework: To complement the dataset, we propose a novel framework for video depth estimation that leverages the rich prior knowledge embedded in video generation models. Drawing from recent advancements (Ke et al., 2024), we build upon SVD (Blattmann al.) (023a) and introduce two key innovations to enhance generalization and efficiency. First, mixed-duration training strategy is introduced to simulate videos with varying frame rates and lengths by randomly dropping frames. To handle videos of different lengths, those with the same duration are grouped into the same batch, and batch sizes are adjusted accordingly, thus optimizing memory usage and improving training efficiency. Second, depth interpolation module is proposed, generating intermediate frames conditioned on globally consistent depth estimates from key frames, allowing for high-resolution and coherent inference of long videos under limited computational constraints. Additionally, we refine the pipeline
"Equal Contribution. This work was done during his internship at Shanghai AT Laboratory.
2

---

Published as a conference paper at ICLR 2025
by introducing flow-matee mewtmhininng approach (Lipman al. 2023) and rotary position encoding (Su et al. 202I) to further improve inference efficiency and flexibility.
Our contributions can be summarized as follows:
We demonstrate feasible path for scaling depth data from diverse virtual environments and systematically validate that high-fidelity synthetic data can improve the generalization of video depth models real-world scenarios.
We propose new training and inference framework that integrates a mixed-duration training strategy and long-video inference module, enabling the model to handle varying video lengths while ensuring spatial accuracy and temporal consistency.
Our method achieves state-of-the-ae performance among generative depth models, setting a new benchmark for accuracy and robustness in video depth estimation.
2 SYNTHETIC DATA WORKFLOW
Rendering Pipeline Overview. Modern rendering pipelines are central to transforming geometric data into visual output, which involves three key stages: 1) Vertex Processing: Geometric data is first processed to determine the position and attributes of each vertex in 3D space. 2) Rasterization: The processed geometric data is then converted into pixels. This stage determines which pixels on the screen correspond parts of the 3D models, establishing depth and color data for each pixel. 3) Shading: Shaders compute the lighting, coloring, and textures. Deferred shading, a popular technique, separates lighting calculations from geometry rendering to optimize processing. Buffers such as depth buffers store intermediate data essential for accurate lighting and shading.
Real-time Data Collection. Following prior works (Richter al. 2016; Krähenbühl 2018),
leverage ReShade [ReShade Contributors, 2024) to access depth information available in the rendering process and OBS [Contributors) 2024g to capture synchronized depth and color data from the screen. The collected dataset encompasses diverse scenes and conditions, including modern virtual environments such as Cities: Skylines II with its expansive urban landscapes and Grand Theft Auto V with its varied open-world settings. Notably, the use and sharing of certain game content are permitted under specific terms and conditions (Rockstar Games 2024; Paradox Interactive 2024), including concemmeercial use and restrictions story spoilers. To ensure compliance, we further remove UI elements and on on-screen text using a ReShade plugir and manually filter specific scenes to prevent unintended storyline disclosures.
Data Filtering, After collecting initial synthetic data, occasional misalignments between the im-
Table 1: Comparisons of synthetic datasets.
Ousloce Indoor
Dynanic
Viãeo
Frante
age and depth are observed, especially during menu switches. To filter these frames, first employ scene cut method to detect scene tran- sitions based on significant color changes. Then, the depth model (detailed in Sec. 3.1). trained on a hand hand-picked subset of collected data, used to filter out splited video sequences with low depth metric scores. However, this straightforward approach can lead to excessive filtering of unseen data. We further use a CLIP (Radford et al.) (02T) model to compute semantic similarity between the actual and predicted depth, both colorized from single channel to three. Finally, we uniformly sample 10 frames from each video segment. both the median semantic and depth metric scores fall below predefined thresholds, the segment is removed. After filtering, we obtain curated synthetic dataset of 40,000 video clips, each recorded at 30 FPS and lasting 5 seconds, as shown in Table I
Hypersim Moberm al. Ioc MVV--yydr Fumme 2an5
68K
12K
VKrI Gammer Paring
25K
Matridul Frerar 2004
519K
Sineel [uuier 20721
LAK
Dynanireyytta Jalre 2 2003
/
169%
DA-V (Ours)
6M
GENERATIVE VIDEO DEPTH MODEL
In this section, we introduce Depth Any Video, a generative model designed for robust and consistent video depth estimation. The model builds upon prior video foundation models, framing vidco depth estimation as a conditional denoising process (Sec 3.I. mixed-duration training strategy is then
3

---

Published as a conference paper at ICLR 2025
Deneise
Concul
Latent Encuder
Diflusion U-Net
Latent Decuder
Ze
Rg
Triining Ouly
(+) Frame Dropout
(b) Video Packing
Inforence Only
(0.1)
1-0
T-2 T-
Rank 0
*
Latent Encoder
Rank
Add Noise
RAPE
Ensdgn Rasgl
Rank K
(ea)
The overall architecture. The input video I, and depth Ea are first encoded into latent space using pretrained latent encoder E. During training, Gaussian noise is added to the latent depth in forward process, while denoising model Dg, conditioned on the latent video Ze. removes the noise in reverse process. After training. the inference flow begins with pure noise. progressively denoises it, and then uses latent decoder D to transform it into the prediction depth & with the original resolution. Besides, the pipeline incorporates a mixed-duratin training strategy: (a) frame dropout and (b) video packing to enhance model generalization and training efficiency.
Figure 2:
presented to improve model generalization and training efficiency (Sec 3.2 Finally, extend the model to estimate high-resolution depth in long videos (Sec. 3.3
3.1 MODEL DESIGN
Our approach builds upon the video foundation model, Stable Video Diffusion (SVD) (Blattmann et al. 2023a), and reformulates monocular video depth estimation as a generative denoising process. The overall framework is illustrated in Figure 2 The training flow consists of forward process that gradually corrupts the ground truth video depth æa by adding Gaussian noise ~> W(O, /) and reverse process that uses denoising model v8, conditioned on the input video re., to remove the noise. Once Ug is trained, the inference flow begins with pure noise and progressively denoises it, moving towards a cleaner result with each step.
Latent Video Condition. Following prior latent diffusion models (Rombach et al. 2022 Esser et al. 2024), the generation process operates within the latent space of a pre-trained variational autoencoder (VAE), allowing the model to handle high-resolution input without sacrificing computational effi- ciency. Specifically, given a video depth Zg. we first apply normalization as in Ke et al. (2024) to
ensure that depth values fall primarily within the VAE's input range of [-1,
de
0.5 2,
dgs d2
where de and dgg represent the 2% and 98% percentiles of aa. respectively. Then, the corresponding latent code is obtained using the encoder E: 2d (āa). From this latent code, the normalized video depth can then be recovered by the decoder D: & = P(aa). Unlike the recent advanced 3D VAE (Yang et al., 2024c OpenAl 2024), which compresses the input across both temporal and spatial dimensions into the latent code, we focus on compressing only the spatial dimension, as in Blattmann al. (2023a). This is because temporal compression potentially causes motion blur artifacts when decoding latent depth codes, especially in videos with fast motion (detailed in Sec. 4.4).
To condition the denoiser
the input video, we first transform the video into latent space as Ze = E(x.). Then, 2e is concatenated with the latent depth code zα frame by frame to form the input for the denoiser. Unlike SVD, we remove the CLIP embedding condition and replace it with zero embedding, as we find it has minimal impact on performance.
Conditional Flow Matching. To accelerate the denoising process, we replace the original EDM framework (Krras et al., (022) in SVD with conditional flow matching (Lipman et al.) 2023). which achieves satisfactory results in just step, compared to the original 25 steps. Concretely, the data corruption in our framework is formulated as a linear interpolation between Gaussian noise (0, 7) and data
p(x) along straight line:
(e(x) fx -1 l)
4

---

Published as a conference paper at ICLR 2025
where (z(x)) represents the corrupted data, with t [0, 1] as the time-dependent interpolation factor. This formulation implies a uniform transformation with constant velocity between data and noise. The corresponding time-dependent velocity field, moving from noise to data, is given by:
Ur vx(x) x
The velocity field i [0, 1] R2 + Rd defines an ordinary differential equation (ODE):
( = ve (x)( dt.
By solving this ODE from = 0 to 1, we can transform noise into a data sample using the approximated velocity field Up. During training, the flow matching objective directly predicts the target velocity to generate the desired probability trajectory:
Lo Eg 1/ (dr (za) Ze: t) - us (za)ll
(5)
where za and &e represent the latent depth code and video code, respectively.
3.2 MIXED-DURATUN TRAINING STRATEGY
Real-world applications often encounter data in various formats, including images and variable-length videos. To enhance the model's generalization across tasks like image and video depth estimation, we implement mniedd-duratin training strategy to ensure robustness across various inputs. This strategy includes frame dropout augmentation, which preserves training efficiency when handling long video sequences, and video packing technique that optimizes memory usage for variable-length videos, enabling our model scale efficiently across different input formats.
Directly training foag-frame videos is computationally expensive, requiring substantial training time and GPU resources. Inspired by context extension techniques (Chen et al.)
Frame Dropout.
2024 2023 Liu et al. 2024) in large language models, we propose frame dropout augmentation with rotary position encoding (RoPE) (Su et al.) 202T) to enhance training efficiency while maintaining adaptability for long videos. Concretely, in each temporal transformer block of the 3D UNet used in SVD, we replace the original sinusoidal absolute position encoding for Exed-frame vidcos with RoPE support variable frames. However, training generalize to longer ones with unlearned frame positions, as shown in Figure a(a). To mitigate this, we retain the original frame position indices =[0.-. [O....
short video with RoPE still struggles to
1] of the long video with T frames, and randomly sample K frames with their original indices for training. This simple strategy helps the temporal layer generalize effectively across variable frame lengths.
Video Packing, To train videos of varying lengths, an intuitive way is to use only one sample per batch, as all data within batch must maintain a consistent shape. However, this leads to inefficient memory usage for shorter videos. To solve this, we first group videos by similar resolution and crop them to a fixed size. For each batch, we then sample examples from the same group and apply the same frame dropout parameter K. The process is illustrated in Figure (b). In particular, we increase the batch size for small-resolution and short-duration videos to improve training efficiency.
3.3 LONG VIDEO INFERENCE
Using the trained model, we can process up to 32 frames at 960 540 resolution in one forward pass on sin- gle 80GB A 100 GPU. To handle longer high-resolution videos. Wang et al. (023) applies a sliding window process short segments independently and concatenate the results. However, this would lead to temporal inconsisten- cies and flickering artifacts between windows. Thus, we first predict consistent key frames, and then each window generates intermediate frames using a frame interpolation network conditioned on these key frames to align the scale and shift of the depth distributions, as shown in Figure 3 Specifically, the interpolation network is finetuned from the video depth model n9 in Sec. 3.1 Instead of conditioning solely the video, the first and last
Sliding Window
Kay
Condition
Figure Illustration of the frame inter- polation network, conditioned key frames to produce coherent predictions.
5

---

Published as a conference paper at ICLR 2025
Input Image
ChronoDepth
Marigold
Depth Anything V2
Ours
NYUV2
KITTI
ScanNet
FTH3D
In the wild
Figure 4: Qualitative comparisons of monocular depth estimation methods across different datasets.
We are able capture fine-grained details and generalize effectively on in-the-wild data.
key frames of each window are also used, with masking map indicating which frames are known. The frame interpolation is formulated as follows:
E = uo (the ( Ed, m.
where & represents the predicted key frames, with non-key frames padded with zeros. The masking map m is used to indicate known key frames, which are set to 1, while other frames are set to 0. The masking map is replicated four times align with the latent feature dimensions. To preserve the pre- pre-traines trained structure and accommodate the expanded input, we duplicate input channels of ug and halve the input layer's weight tensor initialization.
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION METRICS
Training Datasets. In addition to the collected DA-V dataset, we follow Ke et al. (2024) by incorporating single-frame synthetic datasets, Hypersim (Roberts al. (021) and Virtual KITTI 2 (Cabon et al. 2020). Hypersim is photorealistic synthetic dataset featuring 461 indoor scenes, from which we use the official train and val split, totaling approximately 68K samples. Virtual KITTI 2 is synthetic urban dataset comprising 5 scenes with variations in weather and camera configurations, contributing about 25K samples to our training.
Evaluation Datasets. For monocular depth estimation, we conduct series of experiments to evaluate our model's performance on four widely used benchmarks. NYUv2 (Silberman et al.) 2012) and ScanNet (Yeshwanth et al. 2023) provide RGB-D data from indoor environments captured using Kinect cameras. FTH3D (Schops et al., (017)) features both indoor and outdoor scenes, with depth data collected by laser scanner. KITTI (Geiger et al. 2012) comprises outdoor driving scenes captured by cameras and LiDAR sensors. For video depth estimation, we sample 98 video clips from ScanNet++ (Yeshwanth al. 2023) with each clip containing 32 frames. The overlap ratio between adjacent frames in each clip exceeds 40%, ensuring sufficient continuity for video depth estimation.
Evaluation Metrics. All evaluations conducted in the zero-shot setting. Following prior meth- ods (Ke et al. 2024 Fu et al.) 2024), we evaluate invariant depth predictions by optimizing for scale and shift between the predicted depth and the ground truth. The quantitative comparisons

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative comparisons with state-of-the-ari depth estimation methods using single- frame input across four zero-shot affine-invarian depth benchmarks.
Training
NYUv2
Method
Real Synthetic
AbsRel + 61
AbaRel L 51
AbsRel !
61
AhsRel
61
Dicriminative Model:
DeverseDenth (Yin
2020
320K
87.5
19.0
70.4
22.8
69.4
10.9
88.2
MiDaS Lasinger el
2019
2M
9.5
91.5
18.3
71.1
19.0
88.4
9.9
90.7
LeReS Tim
354K
9.0
91.6
14.9
78.4
17.1
77.7
9.1
91.7
Omnidala Elickhir
2021
12.1M
59K
7.4
94.5
14.9
83.5
16.6
77.8
7.5
93.6
HDN Zhang
2022
300K
6.9
94.8
11.5
86.7
12.1
83.3
8.0
93.9
DPT Kamil el A021
1.4M
9.1
91.9
11.1
88.1
11.5
92.9
8.4
93.2
Metrit3U 0l.. 00A5
8M
5.8
96.3
5.8
97.0
6.6
96.0
7.4
94.1
Depth Anydiing Yangg 01
2024g
63.5M
98.0
94.6
6.2
98.0
4.3
98.1
Generative Model:
Marigold et al. 2024
74K
5.5
96.4
9.9
91.6
6.5
96.0
6,4
95.1
DephhFM Gm 2022
63K
6.5
95.6
8.3
93.4
GeaWizard [T al 2014
0.3M
5.2
96.6
9.7
92.1
6.4
96.1
6.1
95.3
Depth Any Video (Ours)
6M
5.1
97.0
7,3
95.1
4.7
97.9
5,3
96.6
conducted with metrics AbsRel (absolute relative error: $ Σ
N-1
[Rg
mal where N denoting the Za
kk=
number of pixels) and and 51 accuracy (percentage of N Σ k= the temporal consistency of video depth, we further introduce the temporal alignment error (TAE):
max fi
< 1.25). To assess
T Σ AbsRel (f($), p"), #, + AbsRel (f(#t-1, ),
TAE 2(T - k=
(7)
where the number of frames, f is the projection function that uses transformation matrix p to map depth # from the k-th frame to the (k + 1)-th frame, and p is the inverse matrix for projection in the reverse direction. The transformation matrix consists of both intrinsic and extrinsic camera parameters, which can be obtained from the dataset.
4.2 IMPLEMENTATION DETAILS
Our implementation is based on SVD (Blattmann et al. 2023a), using the diffusers library (von
Platen et al.) 2022). We employ the AdamW optimizer (Loshchilov & Hutter 2019) with learning
rate of 6.4 10-o, The model is trained various resolutions: 512 x 512, 480 x 640, 707 x 707, 352 x 1216, and 1024 x 1024, with corresponding batch sizes of 384, 256, 192, 128, and 64. The video length is sampled from 1 to with batch size adjusting correspondingly to meet GPU memory requirements. Experiments are conducted on 32 NVIDIA A100 GPUs for 20 epochs, with a total training time of approximately 1 day. For training efficiency, we utilize Fully Sharded Data Parallel (FSDP) with ZeRO Stage 2, gradient checkpointing, and inference, we set the number of denoising steps to 3 and the ensemble size to 20 for benchmark comparison, following Ke et al. (2024), to ensure optimal performance. In contrast, the ablation studies do not utilize the ensemble strategy to focus on the isolated impact of individual components. The runtime evaluation performed on single NVIDIA A100 GPU with resolution of 480 640.
training. During
4.3 ZERO-SHOT DEPTH ESTIMATION
Our model demonstrates exceptional zero-shot generalization in depth estimation across both indoor and outdoor datasets, as well as single-frame and
datasets.
Quantitative Comparisons. Table 2 presents our model's performance in comparison to of the-art depth estimation models using single-frame inputs. Our model significantly surpasses all previous generative models across various datasets and achieves results that are comparable to, and in some cases better than, those top cop-performing discriminative models. For example, compared to
Table 3: Temporal consistency and spatial accuracy comparisons on SeanNet++.
Table 4: Performance and inference efficiency parisons on the ScanNet dataset.
Method
AbsRel
61
TAE
Method
Step
Param
Runtime
51
NVDS Wang ml. M023
22.2
61.9
3.7
Marigold (Ke 2024
50
865 9M
2.06s
94.5
Chroeobeppr
2024
10.4
90.7
2.3
ChronoDepu (Saa
2024
10
1524.6M
1.04s
93.4
DepthCrafier Hu ef 20240
11.5
88.1
2.2
DepthCrrteer
Depth Any Video (Uurs)
9.3
93.4
2.1
Depth Any Video (Ours)
1422.8M
0.37s
96.1
7

---

Published as a conference paper at ICLR 2025
Input Video
Marigold
ChronoDepth
DepthCrafter
Ours
Figure 5: Qualitative comparisons of depth estimation models on in-the-wild videos. Red boxes show changes in color or depth over time vertical red lines in videos. Best viewed by zooming
GeoWizard (Fu al. 2024). model shows improvements of 0.4 in 51 and 0.1 in AbsRel on the NYUv2 dataset, 3.0 in and 2.4 AbsRel on KITTI, 1.8 in and 1.3 in d1 and 0.8 in AbsRel 2024a), we achieve gains of 0.5 in and 0.7 in AbsRel on KITTI, along with a 1.5 improvement in the AbsRel metric on the ETH3D dataset. The impressive results primarily attributed to the large- scale synthetic data we collected. Table presents a comprehensive comparison of our model against previous video depth models. All generative models process multi-frame inputs in single forward pass. Notably, our model demonstrates improved temporal consistency and spatial accuracy on the ScanNet++ dataset, highlighting its effectiveness in video depth estimation. Table presents detailed comparisons with previous generative methods without ensemble techniques. Our model has fewer parameters than ChronoDepth (Shao al.) 2024) because utilize parameter- RoPE instead of learnable absolute positional embeddings. It also reduces complexity compared to DepthCrafter (Hu al. 2024b) by removing the clip embedding condition and classifier-fres guidance. Additionally, we achieve lower inference time and fewer denoising steps while attaining better spatial accuracy on the ScanNet dataset compared to Marigold, ChronoDepth, and DepthCrafter.
AbsRel on ETH3D, and the ScanNet dataset. When compared to Depth Anything (ang et al.)
Qualitative Comparisons. Figure presents qualitative monocular depth estimation results across different datasets. highlights the ability of our model to capture fine-grained details compared to Depth Anything V2 (Yang et al. 2024b). such the cup in the NYUv2 dataset and the ladder in the ETH3D dataset. Moreover, our model handles objects with similar colors more effectively. For example, the KITTI dataset, the person's 's head blends with the background, which Depth Anything V2 fails to predict. Compared to generative methods like Marigold and CeronoDepth, our model generalizes well in-the-wild data, offering a better distinction between the sky and foreground. The DA-V dataset significantly contributes to this, enabling our model to generalize effectively across various environments, particularly in complex, real-world scenarios. Figure further demonstrates qualitative results of depth estimation open-world videos, covering wide range of scenarios such as dust and sand, animals, architecture, and human motion presented in both generated and real-world videos. Following Wang et al. 2023), we visualize the changes in estimated
Table Ablation study of each component. All variants are trained for 10 epochs ensure training efficiency. Memory Uiil. refers to the minimum GPU memory utilization across all GPUs, while Averuge Metric represents the average accuracy across four datasets.
Generative
Conditional
Synthetic
Mixed-duration Training
Runtime
Training Time (hours)
Memory Util. (5)
Average Metric
Prior
Matching
x
X
21.0
65.1
7.5
X
0.37
6.9
94.9
12
63
6,4
95.8
8

---

Published as a conference paper at ICLR 2025
6.4
6.2
96.2
96.6
6.3
AbsRRel (%6)
Abss (%)
AbsRal (%)
96.0
51 (%)
96.4
6.0
51 (%)
8.0
94.0
6.1
96.2
95.8
96.0
92.0
6.0
95.6
3 4
10
5.9
95.8
10
20
5
10
20
40
Denoising Step
Ensemble Size
Training Epoch
(a) Impact of denoising step
(b) Impact ensemble size
(c) Impact of training epoch
Figure 6: Ablation study of hyper- param--rsseeerss on depth estimation performance. Average accuracy across four datasets is reported to provide comprehensive evaluation.
depth values over time at the vertical red lines by slicing along the time axis to better capture temporal consistency. Marigold exhibits zigzag artifacts on a per-frame basis, while ChronoDepth displays similar issues at per-window level. DepthCrafter, with its large overlap between windows and interpolation of overlap space, achieves smoother transitions across windows but still struggles with window- wise flickering. In contrast, our method ensures global consistency by predicting key frames and interpolating intermediate frames, significantly reducing flicker artifacts between windows.
4.4 ABLATION STUDIES
In this section, we evaluate the effectiveness of each component in Depth Any Video. For training efficiency, unless otherwise specified, the model is trained for only 10 epochs during ablation studies.
Generative Visual Prior. We investigate the impact of prior visual knowledge from the stable video diffusion model, shown in T The first two rows clearly demonstrate that incorporating this prior significantly boosts the model's overall performance. Additionally, Figure 6[c) illustrates that this prior provides a strong initialization, leading to fast training convergence and enabling the model to achieve impressive results with as few as five epochs. This suggests that the generative visual prior not only enhances model performance but also improves training efficiency.
Conditional Flow Matching, The second and third rows in Table indicate that flow matching not only reduces inference time, achieving 6.5x acceleration compared to the original EDM scheduler in SVD, but also results in improvements of 0.6 in AbsRel and 1.1 in 81. The faster inference is primarily attributed to the ability to achieve strong performance with fewer denoising steps, as shown in Figure a It demonstrates that even single denoising step yield strong results, with the sweet spot identified at three steps for optimal performance. Figure b).. we further evaluate the effectiveness of ensembling multiple predictions, varying noise initializations produce minor variations in outputs. The results show consistent performance gains as the number of predictions increases, with improvements becoming less pronounced after five predictions.
Synthetic Data. In Table 5 the third and fourth rows show that our collected DA-V dataset brings gains of 0.7 in 81 and 0.4 in AbsRel. The accuracy improvement in outdoor scenes is particularly significant, as shown in Table 2 likely due to the diverse range of outdoor environments in our dataset. Additionally, Figure and demonstaate that models trained on our synthetic data generalize well in-the-win wild scenarios, showcasing both the realism and effectiveness of our synthetic data.
Mixed-duration Training. The fourth and fifth entries in Table show that the mixed mixed-duration training strategy improves training efficiency while simultancously enhancing spatial accuracy. It can save 33% of training time and increase GPU utilization by 40% because different batch sizes can be applied to video sequences of varying lengths, thus optimizing training efficiency. The accuracy
Table 6: Ablation study of the reconstruction quality of different variational autoencoders. We categorize the VAEs into 2D and 3D based on the presence of temporal feature interactions.
(a)
Mothod
Type
AbsRel
61
(b)
SD2 Rombach of
2022
2D
1.2
99.0
SD3 litee 20149
2D
0.6
99.7
XidXi (aag et
2024e
3D
2.2
98.6
SVD Blartnanni
3D
1.5
98.1
Figure 7: Visualization of reconstruction quality.

---

Published as a conference paper at ICLR 2025
improvement is attributed the increased proportion of individual frames during training, achieved by randomly dropping out frames, which enhances the single-fram framm-frame performance.
VAE Variants. Since our depth is generated in the latent space, the quality of the VAE directly impacts the upper bound of the final result. Therefore, we provide a detailed comparison of the depth reconstruction results across different VAEs in Table 6 We find that the 2D VAE surpasses the 3D VAE in reconstruction quality, particularly with the VAE from SD3 (Esser et al. (024). This indicates there is still potential to further enhance our model's performance. In Figure 7 we present visual comparisons of the reconstructions produced by different 3D VAEs. Specifically, (a) shows the input ground truth depth, (b) displays the reconstruction results from Yang et al. (2024c), which incorporates temporal compression, and (c) shows the results from Blattmann et al. (2023a). Although the VAE with temporal compression can reduce the computational complexity of the latent model, struggles to handle fast motion effectively, as indicated by the red box in (b).
5 RELATED WORK
Monocular Depth Estimation. Existing models for monocular depth estimation can be roughly divided into two categories: discriminative and generative. Discriminative models are trained end-to-end to predict depth from images. For example, MiDaS (Lasinger et al. 2019) focuses on relative depth estimation by factoring out scale, enabling robust training on mixed datasets. Depth Anything (Yang et al. 2024a by builds on this concept, leveraging both labeled and unlabeled images to further enhance generalization. ZoeDepth (Bhat et al. 2023) and Metric3D (Yin al. 2023 Hu
et al.) 2024a) aim to directly estimate metric depth. Generative models (Saxena et al. 2023), such as
Marigold (Ke et al., 2024) and GeoWizard (Fu et al. 2024), leverage powerful priors learned from large-scale woaldworld data, allowing them to generate depth estimates in zero-shot manner, even on
unseen datasets. Our work falls into the second category, but focuses on video depth estimation.
Video Depth Estimation. Unlike single-image depth estimation, video depth estimation requires maintaining temporal consistency between frames. To eliminate flickering effects between consecutive frames, some works (Luo et al. 22200; Chen et al., 2019 Zhhng et al. 2021 use optimization procedure to overfit each video during inference. Other approaches (Guizilini et al. [002 Zhang et al.,
2019 Teed & Deng 2020) directly predict depth sequences from videos. For instance, NVDS (Wang) et al. 2023 proposes a refinement network to optimize temporal consistency from off-the-sheil depth predictors. Some concurrent works (Hu al. 2024b Shao al., 2024) have focused leveraging video diffusion models to produce coherent predictions. However, they often face challenges due to a lack of sufficiently high-quality and realistic depth data.
Video Generation. Diffusion models (Ho et al. 22000; Song et al. 2021 have achieved high-fidelity image generation from text descriptions. benenting from large-scale aligned image-text datasets. Building on this success, VDM (Ho et al., 2022b) first introduces unconditional video generation in pixel space. Imagen Video (Ho et al. 2022a) and Make-a- Video (Singer et al. (023) are cascade models designed for text-to-video generation. Align Your Latent (Blattmann et al. 2023b) and SVD (Blattmann et al./ 2023a) extend Rombach et al. (2022) by modeling videos in the latent space of autoencoder. Our model builds upon the generative visual prior of SVD, which is trained on diverse real video data, to maintain robust generalization in real-world scenarios.
6 CONCLUSION
We present Depth Any Video, novel approach for versatile image and video depth estimation. powered by generative video diffusion models. Leveraging diverse and high-quality depth data collected from virtual environments, our model could generate temporally consistent depth sequences with fine-grained details across a broad spectrum of unseen scenarios. Equipped with mixed- duration training strategy and frame interpolation, it generalizes effectively to videos of various lengths and resolutions. Compared previous generative depth estimation models, approach sets a new state-of the in performance while significantly enhancing efficiency.
Limitations. There are still certain issues in our model, such as difficulties in estimating depth for mirror- like reflections on water surfaces and challenges with extremely long videos. Future work will focus on collecting data for these challenging scenarios and improving model efficiency.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by The National Nature Science Foundation of China (Grant Nos.: 62273301, 62273303). in part by Yongjiang Talent Introduction Programme (Grant No.: 2022A-240- G). in part by Ningbo Key R&D Program (Nos.: 2023Z231, 20232229), in part by The National Key R&D Program of China (No. 20222D0160101..
ETHICS STATEMENTS
Our work utilizes game-derived video clips and depth data develop video depth estimation models. We maintain ethical integrity through three foundational commitments:
Legal Data Acquisition: All data was obtained through legally purchased games via authorized platforms (Steam/Epic), with industry-standate capture tools (OBS for video recording). Depth extraction was performed via ReShade's non-invasive post-processing shaders that process decrypted frame buffers provided by the game runtime, with neither cheating methods employed nor reverse engineering of the source code conducted during acquisition.
2) Copyright-Complee Compliant Utilization: Our method complies with research exceptions of international copyright law (such as the Berne Convention for the Protection of Literury and Artistic Works and the WIPO Copyright Treaty) and aligns with the principles of fair use. Fair use by law is not unrestrictedly prohibited by the standard clauses the end user license agreement that have not been negotiated that would be contrary the goal of the intellectual property legal system in promoting technological innovation and development. Focused exclusively on validating synthetic data pipelines and model training strategies for academic advancement, this non-commercial study does not infringe upon the economic interests or legal rights of right holders.
exception permitted
3) Proactive Responsibility: Although claiming fair use protections, recognize data ownership resides with copyright holders. We are actively negotiating permissions for potential dataset sharing and will implement strict access control (c.g., institutional authorization) if granted. Any legal consequences arising from this work will be independently borne.
REFERENCES
Karteek Alahari, Guillaume Seguin, Josef Sivic, and Ivan Laptev. Pose estimation and segmentation of people in 3d movies. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2112-2119, 2013.
Shariq Farooq Bhat, Reiner Birkl, Diana Wolk, Peter Wonka, and Matthias Müller. Zocdepth: Zero-shot transfer by combining relative and metric depth. CoRR, abs/2302. 12288, 2023.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rom- bach. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, abs/2311.15127, 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563 22575,, 2023b.
Guido Borghi, Marco Venturelli, Roberto Vezzani, and Rita Cucchiara. Poseidon: Face-from-depth for driver pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5494-5503, 2017.
Daniel Butler, Jonas Wulff, Garrett B Stanley, and Michael Black. A naturalistic open source movie for optical flow evaluation. In European Conference on Computer Vision, pp. 61 61155,, 2012.
Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual KITTI 2. CoRR, abs/2001.10773, 2020.
11

---

Published as a conference paper at ICLR 2025
Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. CoRR, abs/2306.15595, 2023.
Yuhua Chen, Cordelia Schmid, and Cristian Sminchisescu. Self-supervised learning with geometric constraints in monocular video: Connecting flow, depth, and camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7063-7072, 2019.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. In International Conference on Learning Representations, 2024.
OBS Contributors, Obs studio, 2024. URL
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from 3d scans. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10786-10796 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari. Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference on Machine Learning, 2024.
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. CoRR, abs/2403. 12013, 2024.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3354-3361, 2012.
Ming Gui, Johannes S. Fischer, Ulrich Prestel, Pingchuan Ma, Dmytro Kotovenko, Olga Grebenkova, Stefan Andreas Baumann, Vincent Tao Hu, and Björ Ommer. Depthfm:: Fast monocular depth estimation with flow matching. CoRR, abs/2403.13788 2024.
Vitor Guizilini, Rares Ambrus, Dian Chen, Sergey Zakharov, and Adrien Gaidon. Multi-frame self- supervised depth with transformers. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 160-170, 2022.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi. David Fleet, and Tim Salimans. Imagen video: High definition video generation with diffusion models. CoRR, abs/2210.023003 2022a.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. In Advances in Neural Information Processing Systems, volume 35, 2022b.
pp.
Aleksander Holynski and Johannes Kopf. Fast depth densification for occlusion-aware augmented reality. ACM Transactions on Graphics, 37(6): 2018.
Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for sero metric depth and surface normal estimation. CoRR, abs/2404.15506, 2024a.
Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Deptherafter: Generating consistent long depth sequences for open-worke videos. CoRR, abs/24009020095, 2024b.
Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra Ahuja, and Bin-Bin Huang. Deepmvs: Learning multi-view view stereopsis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2821-2830, 2018.
12

---

Published as a conference paper at ICLR 2025
Junpeng Jing. Ye Mao, and Krystian Mikolajczyk. Match-stereo- videos: Bidirectional alignment for consistent dynamic stereo matching. CoRR, abs/2403.10055, 2024.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Dynamicstereo: Consistent dynamic depth from stereo videos. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp- 13229-13233 13239,, 2023.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. In Advances in Neural Information Processing Systems, volume 35, pp. 26565-26577, 2022.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
9492-9502, 2024.
Philipp Krähenbühl. Free supervision from video games. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2955-2964, 2018.
Katrin Lasinger, René Ranftl, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. CoRR, abs/ abs/1907 007.113111 2019.
Yixuan Li, Lihan Jiang, Linning Xu, Yuanbo Xiangli, Zhenzhi Wang, Dahua Lin, and Bo Dai. Matrixcity: A large-scale city dataset for city-scale neural rendering and beyond. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3182-3192 IEEE, 2023.
Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In International Conference Learning Representations, 2023.
Jiaheng Liu, Zhiqi Bai, Yuanxing Zhang, Chenchen Zhang, Yu Zhang, Ge Zhang, Jiakai Wang, Haoran Que, Yukang Chen, Wenbo Su, Tiezheng Ge, Jie Fu, Wenhu Chen, and Bo Zheng. B'2-llm: Efficient and extreme length extension of large language models. CoRR, abs/2401 06951, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopf. Consistent video depth estimation. ACM Transactions Graphics, 39(4):71-1, 2020.
OpenAl. Sora, 2024. URL
Paradox Interactive.
End User License Agreement, 2024. URL https://stoon
steampowered. com attpa/__9920___aa______
eula www.lalangsenggle....
Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Ming-Chang Yang, and Jiaya Jia. Controlnext: Powerful and efficient control for image and video generation. CoRR, abs/2408. 000070, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12179-12188. 2021.
ReShade Contributors. ReShade, 2024. URL https::/rreshaiee
Stephan R. Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pp. 102-118 2016.
13

---

Published as a conference paper at ICLR 2025
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Ángel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings the IEEE/CVF International Conference Computer Vision, pp. 10892-10902, 2021.
Rockstar Games. Policy on Posting Copyrighted Rockstar Games Material, 2024. URL http:
//tinyura //tinyyrl..com//////. coo.pjfooop
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Saurabh Saxena, Junhwa Hur, Charles Herrmann, Deqing Sun, and David J. Fleet. Zero-shot metric depth with field-of- view conditioned diffusion model. CoRR, 1t.//232..13222 2023.
Thomas Schops, Johannes L. Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. A multi-view stereo benchmark with high-resolution images and multi-camera videos. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2260-3269, 2017.
Jiahao Shao, Yuanbo Yang. Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, and Yiyi Liao. Learning temporally consistent video depth from video diffusion priors. CoRR, abs/2406.011433, 2024.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from RGBD images. Andrew W. Fitzgibbon, Svetlana I Lazebnik, Pietro Perona, Yoichi Sato, and Cordelia Schmid (eds.), European Conference Computer Vision, pp. 746-760, 2012.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-video: Text-to- video generation without text-video data. In International Conference on Learning Representations, 2023.
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, CoRR, abs/2 hass2104409964/ 2021.
Zachary Teed and Jia Deng. Deepv2d: Video to depth with differentiable structure from motion. In International Conference on Learning Representations, 2020.
Patrick Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-ther attt diffusion models. https: https://qqitube com/ hww.gigggggggeg//////oo diffusers] 2022.
Yiran Wang, Min Shi, Jiaqi Li, Zihao Huang, Zhiguo Cao, Jianming Zhang, Ke Xian, and Guosheng Lin. Neural video depth stabilizer. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 9432-9442, 2023.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Paltern Analysis and Machine Intelligence, 2023.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEF/CVF Conference Computer Vision and Patlern Recognition, pp. 10371-10381, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything V2. CoRR, abs/2406.09414, 2024b.
14

---

Published as a conference paper at ICLR 2025
Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang. Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with expert transformer. CoRR. (lss22088006722, 2024c.
Chandan Yeshwanth, Yueh-Cheng Liu, Matthias NieBner, and Angela Dai. Scannet++: A high- fidelity dataset of 3d indoor scenes. In Proceedings of the IEEE/CVE International Conference on Computer Vision. pp. 12-22, 2023.
Wei Yin, Xinlong Wang, Chunhua Shen, Yifan Liu, Zhi Tian, Songeen Xu, Changming Sun, and Dou Renyin. Diversedepth: Affine-invariant depth prediction using diverse data. CoRR, abs/2002.00569, 2020.
Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to recover 3d scene shape from single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 204-213, 2021.
Wei Yin, Chi Zhang, Hao Chen, Zhipeng Cai, Gang Yu, Kaixuan Wang, Xiaozhi Chen, and Chunhua Shen. Metric3d: Towards zero-shot metric 3d prediction from single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9043-004300900 2023.
Chi Zhang, Wei Yin, Billzb Wang. Gang Yu, Bin Fu, and Chunhua Shen. Hierarchical normalization for robust monocular depth estimation. In Advances in Neural Information Processing Systems, volume 35, pp. 1112-11139, 2022.
Haokui Zhang, Ying Li, Yuanzhouhan Cao, Yu Liu, Chunhua Shen, and Youliang Yan. Exploiting temporal consistency for real-time video depth estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1725-1734, 2019.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free free controllable text-to-video generation. In International Conference on Learning Representations, 2024.
Zhoutong Zhang, Forrester Cole, Richard Tucker, William Freeman, and Tali Dekel. Consistent depth of moving objects in video. ACM Transactions Graphics, 40,4):1-12, 2021.
15


Reviews:
[{"review_id": "CQiHIbvqH4", "reviewer": "Reviewer_ATwr", "summary": "The paper introduces a video-depth method that is trained on synthetic datasets only. The results show consistent video depth, generalizing diverse real-world scenes across diverse benchmarks. The paper provides practical engineering strategies, e.g., a batching strategy for better usage of RAM, temporal interpolation module, frame dropout, etc.\n\n---\n\nI rate the paper marginally below the acceptance threshold. The empirical results are very strong, but the paper mostly focuses on engineering solutions without scientifically interesting ideas. Also, there might be a license concern on the synthetic depth dataset. I am happy to improve the rating if the questions in the Weakness and Question sections are properly addressed.", "questions": "* Insights on why 2D VAE variants surpasses 3D VAE variants? I am wondering why 2D VAE variants outperform. Or asking differently, where do the artifacts of the 3D VAE variant come from? A good video depth model may have a good 3D prior, so 3D VAE might be a more natural choice to encode 3D information, but why does it underperform? Is it more difficult to train 3D VAE properly? Providing any insights or discussion is appreciated.\n\n* In Fig. 7 (a) why do more denoising steps hurt accuracy?\n\n* (nit) When looking at the qualitative examples, there are thin depth boundaries along the person's boundaries (the 5th example, woman, in Fig. 6), probably it's a kind of an average of depth of the person and the background. Why it's the case? What would be the source of this error?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* Good empirical results\n\n  The method shows very good empirical results compared with the previous methods. The model outputs consistent depth estimates without temporal flickering, and good accuracy on public benchmarks. The ablation study in Table 3 validates the technical design choices of the method. \n\n* Clarity\n\n   It's easy to follow the paper. The paper provides sufficient technical details on the datasets, training, architecture, etc.", "weaknesses": "* A bit of engineering work\n\n   The paper is mostly about engineering. It adopts conditional flow matching, uses large-scale synthetic datasets to boost accuracy, and introduces mixed-duration training to improve memory usage. All these aspects attribute better accuracy and performance, but it doesn't necessarily provide novel findings. If wanting to emphasize, what would be the most interesting, novel findings of the paper?\n\n* Dataset licence and reproducibility\n\n   It's curious if the collected synthetic dataset can be released or made public. What is the license condition of each game in DA-V? Are there any concerns about using the commercial game engine for research? Is there any plan to release the data? It can affect the reproducibility of the method.\n\n\n* The effect of the DA-V dataset\n\n   The ablations study in Table 5 shows that the synthetic game data improves the depth accuracy but quite marginally although the dataset size is around 6M. Why doesn't it improve the accuracy significantly? Is there any qualitative improvement that the metrics or numbers don't show?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w8owhj85Wj", "reviewer": "Reviewer_vgJ9", "summary": "Depth Any Video is proposed for video depth estimation. It has a scalable synthetic data pipeline from game environments. A novel framework uses generative video diffusion models’ priors. It has a mixed-duration training strategy and a depth interpolation method. The model outperforms previous generative depth models, achieving good spatial accuracy and temporal consistency.", "questions": "See Weaknesses", "limitations": "", "rating": 10, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper constructs a large-scale synthetic dataset of 40,000 video depth clips from 12 diverse modern video games.This dataset provides a scalable and cost-effective way to gather ground-truth video depth data and helps the model generalize to real-world scenarios.\n- A mixed-duration training strategy is proposed. It includes frame dropout augmentation with rotary position encoding and a video packing technique.\n- Effective Model Design\n- The method achieves state-of-the-art performance among generative depth models.", "weaknesses": "The work in the article is very solid, with good model performance and efficiency, and comprehensive evaluation. The only concern for the reviewer is how the author ensures that the dataset, which is a major contribution, will be open-sourced as promised. This is very important for the community, but there are many difficulties regarding copyright and other aspects. In addition, it is necessary to evaluate and compare the diversity of the dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ddFa6ivQP", "reviewer": "Reviewer_gu5t", "summary": "This paper works on video depth estimation with diffusion models. First, considering the lack of video depth data, they collect 40,000 videos with depth annotation from video games. With the collected data, this paper fine-tunes the video diffusion model to achieve depth estimation. In addition, rotary position encoding and flow matching are introduced to further enhance the performance. With the proposed techniques, this paper achieves good spatial and temporal consistency on long video depth estimation.", "questions": "My main concern is the representation of video depth data. I feel it is not hard to use a shared scale and shift following DepthCrafter, which would be more reasonable for video depth estimation. In addition, I am also curious about the shift from EDM to Flow Matching.\n\n**Post Rebuttal:** My previous questions are mainly addressed. However, the copyright issue remains severe after discussing with authors and other reviewers.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation for the framework is clear and reasonable, considering the limited data, inference speed and the long video in the applications.\n\n2. Collecting and annotating high-quality data can improve the model and also inspire the community. It would be more beneficial if the data or collection pipeline can be open-sourced\n\n3. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed method.", "weaknesses": "1. **The representation is more image depth estimation rather than video depth estimation.** If I understand correctly, although the paper focuses on video depth estimation, the predicted relative depth maps are independent for each frame, which is demonstrated in the input normalization and alignment during evaluation. Specifically, each frame is normalized based on the depth range of itself and the scale and shift are also aligned for each frame during inference. In my view, this is incorrect for video depth estimation. To use the accurate relative depth of a video, the scale and shift should be the shared values aligned to the whole video, like DepthCrafter (Hu et al.).\n\n2. **Shift from EDM to Flow Matching.** The SVD model was pre-trained with the EDM denoising scheduler, which has a different optimization objective with flow matching. However, this paper directly fine-tunes the SVD model with conditional flow matching. As far as I know, InstaFlow (Liu et al, ICLR2024) optimized rectified flow for stable diffusion with velocity distillation instead of directly fine-tuning. I hope the authors could provide more explanation about the shift from EDM to Flow Matching. \n\n3.  A follow-up question about the shift. The ablation study of removing the pre-trained SVD model is required to demonstrate if the EDM pre-trained weights benefit or not.\n\n4. **More explanation about the video interpolation model.** As illustrated in Sec. 3.3, video interpolation model takes the key frames as input and interpolate other frames. Do the key frames replace the original noise? Or are they concatenated on the channels of latent? In addition, why are the mask maps replicated 4 times?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)