Depth Any Video with Scalable Synthetic Data

## Abstract

Abstract Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video , a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse synthetic environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations.
Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency.
Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates—even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames.
Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency. † † ∗ Equal contribution. † † † Corresponding author.

## 1 Introduction

Depth estimation is a foundational problem in understanding the 3D structure of the real world.
The ability to accurately perceive and represent depth in video sequences is crucial for a broad range of applications, including autonomous navigation (Borghi et al., ), augmented reality (Holynski & Kopf, ), and advanced video editing (Zhang et al., ; Peng et al., ).
Although recent advancements in single-image depth estimation (Ke et al., ; Yang et al., ; Fu et al., ; Ranftl et al., ) have led to significant improvements in spatial accuracy, ensuring temporal consistency across video frames remains a substantial challenge.

A major bottleneck in existing video depth estimation (Wang et al., ; Shao et al., ) is the lack of diverse and large-scale video depth data that capture the complexity of real-world environments.
Existing datasets (Geiger et al., ; Li et al., ; Karaev et al., ) are often limited in terms of scale, diversity, and scene variation, making it difficult for models to generalize effectively across different scenarios.
From a hardware perspective, depth sensors like LiDAR, structured light systems, and time-of-flight cameras can provide accurate depth measurements but are often costly, limited in range or resolution, and struggle under specific lighting conditions or when dealing with reflective surfaces.
Another common approach (Wang et al., ; Hu et al., ) is to rely on unlabeled stereo video datasets (Alahari et al., ) and state-of-the-art stereo-matching methods (Jing et al., ; Xu et al., ); however, such methods are complex, computationally intensive, and often fail in areas with weak textures. These limitations hinder the development of robust models that can ensure both spatial precision and temporal consistency in dynamic scenes.

To tackle the challenge, in this paper, we propose a solution from two complementary perspectives: (1) constructing a large-scale synthetic video depth dataset and (2) designing a novel framework that leverages powerful visual priors of generative models to effectively handle various real-world videos.

Synthetic Data:
Modern video games offer highly realistic graphics and simulate diverse real-world scenarios. For example, car racing games realistically replicate driving environments, while open-world games simulate various complex scenes.
Given that modern rendering pipelines often include depth buffers, it becomes possible to extract large-scale, highly accurate video depth data from synthetic environments, which is scalable and cost-effective.
In light of this, we construct DA-V, a synthetic dataset comprising 40,000 video clips.
DA-V captures a wide range of scenarios, covering various lighting conditions, dynamic camera movements, and intricate object interactions in both indoor and outdoor environments, providing the opportunity for models to generalize effectively to real-world environments.

Framework:
To complement the dataset, we propose a novel framework for video depth estimation that leverages the rich prior knowledge embedded in video generation models.
Drawing from recent advancements (Ke et al., ), we build upon SVD (Blattmann et al., ) and introduce two key innovations to enhance generalization and efficiency.
First, a mixed-duration training strategy is introduced to simulate videos with varying frame rates and lengths by randomly dropping frames.
To handle videos of different lengths, those with the same duration are grouped into the same batch, and batch sizes are adjusted accordingly, thus optimizing memory usage and improving training efficiency.
Second, a depth interpolation module is proposed, generating intermediate frames conditioned on globally consistent depth estimates from key frames, allowing for high-resolution and coherent inference of long videos under limited computational constraints.
Additionally, we refine the pipeline by introducing a flow-matching approach (Lipman et al., ) and rotary position encoding (Su et al., ) to further improve inference efficiency and flexibility.

Our contributions can be summarized as follows:

- •
We introduce a large-scale synthetic dataset of 40,000 video depth clips collected from diverse rendering engines. This is the first dataset to leverage a variety of high-fidelity synthetic environments and to verify its generalization capability in real-world scenarios.
- •
We propose a new training and inference framework that integrates a mixed-duration training strategy and a long-video inference module, enabling the model to handle varying video lengths while ensuring spatial accuracy and temporal consistency.
- •
Our method achieves state-of-the-art performance among generative depth models, setting a new benchmark for accuracy and robustness in video depth estimation.

## 2 Synthetic Data Workflow

Real-time Data Collection.
To address the challenges of depth data, we collect a large-scale synthetic dataset comprising approximately 40,000 video clips.
A significant portion of this dataset is derived from state-of-the-art synthetic engines, leveraging their ability to generate photorealistic environments with accurate depth information.
We extract depth data from a diverse set of virtual environments, carefully selected to encompass a wide range of scene conditions.
In Table 1, we compare our dataset with previous public synthetic datasets. To the best of our knowledge, ours is the largest synthetic video dataset covering a wide range of realistic scenes.

**Table 1: Comparisons of synthetic datasets.**
| Dataset | Outdoor | Indoor | Dynamic | Video | # Frame |
| --- | --- | --- | --- | --- | --- |
| Hypersim (Roberts et al., ) | ✗ | ✓ | ✗ | ✗ | 68K |
| MVS-Synth (Huang et al., ) | ✓ | ✗ | ✗ | ✓ | 12K |
| VKITTI (Cabon et al., ) | ✓ | ✗ | ✗ | ✓ | 25K |
| MatrixCity (Li et al., ) | ✓ | ✗ | ✗ | ✓ | 519K |
| Sintel (Butler et al., ) | ✓ | ✗ | ✓ | ✓ | 1.6K |
| DynamicReplica (Karaev et al., ) | ✗ | ✓ | ✓ | ✓ | 169K |
| DA-V (Ours) | ✓ | ✓ | ✓ | ✓ | 6M |

Data Filtering.
After collecting initial synthetic video data, occasional misalignments between the image and depth are observed.
To filter these frames, we first employ a scene cut method(^1^11https://github.com/Breakthrough/PySceneDetect) to detect scene transitions based on significant color changes.
Then, the depth model (detailed in Sec. 3.1), trained on a hand-picked subset of the data, is used to filter out splited video sequences with low depth metric scores.
However, this straightforward approach can lead to excessive filtering of unseen data.
Therefore, we further use a CLIP (Radford et al., ) model to compute semantic similarity between the actual and predicted depth, both colorized from a single channel to three.
Finally, we uniformly sample 10 frames from each video segment.
If both the median semantic and depth metric scores fall below predefined thresholds, the segment is removed.

## 3 Generative Video Depth Model

In this section, we introduce Depth Any Video, a generative model designed for robust and consistent video depth estimation.
The model builds upon prior video foundation models, framing video depth estimation as a conditional denoising process (Sec.3.1). A mixed-duration training strategy is then presented to improve model generalization and training efficiency (Sec.3.2).
Finally, we extend the model to estimate high-resolution depth in long videos (Sec. 3.3).

### 3.1 Model Design

Our approach builds upon the video foundation model, Stable Video Diffusion (SVD) (Blattmann et al., ), and reformulates monocular video depth estimation as a generative denoising process.
The overall framework is illustrated in Figure 2.
The training flow consists of a forward process that gradually corrupts the ground truth video depth $x_{d}$ by adding Gaussian noise $\epsilon\sim\mathcal{N}(0,I)$ and a reverse process that uses a denoising model $v_{\theta}$, conditioned on the input video $x_{c}$, to remove the noise.
Once $v_{\theta}$ is trained, the inference flow begins with pure noise $\epsilon$ and progressively denoises it, moving towards a cleaner result with each step.

Latent Video Condition.
Following prior latent diffusion models (Rombach et al., ; Esser et al., ), the generation process operates within the latent space of a pre-trained variational autoencoder (VAE), allowing the model to handle high-resolution input without sacrificing computational efficiency.
Specifically, given a video depth $x_{d}$, we first apply a normalization as in Ke et al. () to ensure that depth values fall primarily within the VAE’s input range of $[-1,1]$:

$$ $\tilde{x}_{d}=\left(\frac{x_{d}-d_{2}}{d_{98}-d_{2}}-0.5\right)\times 2,$ (1) $$

where $d_{2}$ and $d_{98}$ represent the 2% and 98% percentiles of $x_{d}$, respectively.
Then, the corresponding latent code is obtained using the encoder $\mathcal{E}$: $z_{d}=\mathcal{E}(\tilde{x}_{d})$.
From this latent code, the normalized video depth can then be recovered by the decoder $\mathcal{D}$: $\hat{x}_{d}=\mathcal{D}(z_{d})$.
Unlike the recent advanced 3D VAE (Yang et al., ; OpenAI, ), which compresses the input across both temporal and spatial dimensions into the latent code, we focus on compressing only the spatial dimension, as in Blattmann et al. ().
This is because temporal compression potentially causes motion blur artifacts when decoding latent depth codes, especially in videos with fast motion (detailed in Sec. 4.4).

To condition the denoiser $v_{\theta}$ on the input video, we first transform the video $x_{c}$ into latent space as $z_{c}=\mathcal{E}(x_{c})$.
Then, $z_{c}$ is concatenated with the latent depth code $z_{d}$ frame by frame to form the input for the denoiser.
Unlike SVD, we remove the CLIP embedding condition and replace it with a zero embedding, as we find it has minimal impact on performance.

Figure: Figure 2: The overall architecture. The input video $x_{c}$ and depth $x_{d}$ are first encoded into latent space using a pretrained latent encoder $\mathcal{E}$. During training, Gaussian noise $\epsilon$ is added to the latent depth in a forward process, while a denoising model $v_{\theta}$, conditioned on the latent video $z_{c}$, removes the noise in a reverse process. After training, the inference flow begins with pure noise, progressively denoises it, and then uses a latent decoder $\mathcal{D}$ to transform it into the prediction depth $\hat{x}_{d}$ with the original resolution. Besides, the pipeline incorporates a mixed-duration training strategy: (a) frame dropout and (b) video packing to enhance model generalization and training efficiency.
Refer to caption: extracted/5923385/figure/struct.png

Conditional Flow Matching.
To accelerate the denoising process, we replace the original EDM framework (Karras et al., ) in SVD with conditional flow matching (Lipman et al., ), which achieves satisfactory results in just 1 step, compared to the original 25 steps.
Concretely, the data corruption in our framework is formulated as a linear interpolation between Gaussian noise $\epsilon\sim\mathcal{N}(0,I)$ and data $x\sim p(x)$ along a straight line:

$$ $\phi_{t}(x)=tx+\left(1-t\right)\epsilon,$ (2) $$

where $\phi_{t}(x)$ represents the corrupted data, with $t\in[0,1]$ as the time-dependent interpolation factor.
This formulation implies a uniform transformation with constant velocity between data and noise.
The corresponding time-dependent velocity field, moving from noise to data, is given by:

$$ $v_{t}(x)=x-\epsilon.$ (3) $$

The velocity field $v_{t}:[0,1]\times\mathbb{R}^{d}\rightarrow\mathbb{R}^{d}$ defines an ordinary differential equation (ODE):

$$ $d\phi_{t}(x)=v_{t}\left(\phi_{t}(x)\right)dt.$ (4) $$

By solving this ODE from $t=0$ to $t=1$, we can transform noise into a data sample using the approximated velocity field $v_{\theta}$.
During training, the flow matching objective directly predicts the target velocity to generate the desired probability trajectory:

$$ $\mathcal{L}_{\theta}=\mathbb{E}_{t}\left\|v_{\theta}\left(\phi_{t}\left(z_{d} \right),z_{c},t\right)-v_{t}\left(z_{d}\right)\right\|^{2},$ (5) $$

where $z_{d}$ and $z_{c}$ represent the latent depth code and video code, respectively.

### 3.2 Mixed-duration Training Strategy

Real-world applications often encounter data in various formats, including images and variable-length videos.
To enhance the model’s generalization across tasks like image and video depth estimation, we implement a mixed-duration training strategy to ensure robustness across various inputs.
This strategy includes frame dropout augmentation, which preserves training efficiency when handling long video sequences, and a video packing technique that optimizes memory usage for variable-length videos, enabling our model to scale efficiently across different input formats.

Frame Dropout.
Directly training long-frame videos is computationally expensive, requiring substantial training time and GPU resources.
Inspired by context extension techniques (Chen et al., ; ; Liu et al., ) in large language models, we propose frame dropout augmentation with rotary position encoding (RoPE) (Su et al., ) to enhance training efficiency while maintaining adaptability for long videos.
Concretely, in each temporal transformer block of the 3D UNet used in SVD, we replace the original sinusoidal absolute position encoding for fixed-frame videos with RoPE to support variable frames.
However, training on a short video with RoPE still struggles to generalize to longer ones with unlearned frame positions, as shown in Figure 2(a).
To mitigate this, we retain the original frame position indices $i=[0,\cdots,T-1]$ of the long video with $T$ frames, and randomly sample $K$ frames with their original indices for training.
This simple strategy helps the temporal layer generalize effectively across variable frame lengths.

Video Packing.
To train videos of varying lengths, an intuitive way is to use only one sample per batch, as all data within a batch must maintain a consistent shape.
However, this leads to inefficient memory usage for shorter videos.
To solve this, we first group videos by similar resolution and crop them to a fixed size.
For each batch, we then sample examples from the same group and apply the same frame dropout parameter $K$.
The process is illustrated in Figure 2(b).
In particular, we increase the batch size for small-resolution and short-duration videos to improve training efficiency.

### 3.3 Long Video Inference

Figure: Figure 3: Illustration of the frame interpolation network, conditioned on key frames to produce coherent predictions.
Refer to caption: extracted/5923385/figure/interp_net.png

Using the trained model, we can process up to 32 frames at $960\times 540$ resolution in one forward pass on a single 80GB A100 GPU.
To handle longer high-resolution videos, Wang et al. () applies a sliding window to process short segments independently and concatenate the results.
However, this would lead to temporal inconsistencies and flickering artifacts between windows.
Thus, we first predict consistent key frames, and then each window generates intermediate frames using a frame interpolation network conditioned on these key frames to align the scale and shift of the depth distributions, as shown in Figure 3.
Specifically, the interpolation network is finetuned from the video depth model $v_{\theta}$ in Sec. 3.1.
Instead of conditioning solely on the video, the first and last key frames of each window are also used, with a masking map indicating which frames are known.
The frame interpolation is formulated as follows:

$$ $\tilde{z}_{d}=v_{\theta}\left(\phi_{t}\left(z_{d}\right),z_{c},\hat{z}_{d},m,t \right),$ (6) $$

where $\hat{z}_{d}$ represents the predicted key frames, with non-key frames padded with zeros. The masking map $m$ is used to indicate known key frames, which are set to $1$, while other frames are set to 0 0. The masking map is replicated four times to align with the latent feature dimensions.
To preserve the pre-trained structure and accommodate the expanded input, we duplicate input channels of $v_{\theta}$ and halve the input layer’s weight tensor as initialization.

Figure: Figure 4: Qualitative comparisons of monocular depth estimation methods across different datasets. We are able to capture fine-grained details and generalize effectively on in-the-wild data.
Refer to caption: x2.png

## 4 Experiments

### 4.1 Datasets and Evaluation Metrics

Training Datasets.
In addition to the collected DA-V dataset, we follow Ke et al. () by incorporating two single-frame synthetic datasets, Hypersim (Roberts et al., ) and Virtual KITTI 2 (Cabon et al., ).
Hypersim is a photorealistic synthetic dataset featuring 461 indoor scenes, from which we use the official train and val split, totaling approximately 68K samples.
Virtual KITTI 2 is a synthetic urban dataset comprising 5 scenes with variations in weather and camera configurations, contributing about 25K samples to our training.

Evaluation Datasets.
For monocular depth estimation, we conduct a series of experiments to evaluate our model’s performance on four widely used benchmarks.
NYUv2 (Silberman et al., ) and ScanNet (Yeshwanth et al., ) provide RGB-D data from indoor environments captured using Kinect cameras. ETH3D (Schops et al., ) features both indoor and outdoor scenes, with depth data collected by a laser scanner.
KITTI (Geiger et al., ) comprises outdoor driving scenes captured by cameras and LiDAR sensors.
For video depth estimation, we sample 98 video clips from ScanNet++ (Yeshwanth et al., ), with each clip containing 32 frames.
The overlap ratio between adjacent frames in each clip exceeds 40%, ensuring sufficient continuity for video depth estimation.

Evaluation Metrics.
All evaluations are conducted in the zero-shot setting.
Following prior methods (Ke et al., ; Fu et al., ), we evaluate affine-invariant depth predictions by optimizing for scale and shift between the predicted depth and the ground truth.
The quantitative comparisons are conducted with metrics AbsRel (absolute relative error: $\frac{1}{N}\sum_{k=0}^{N-1}\frac{\left|\hat{x}_{d}-x_{d}\right|}{x_{d}}$, where $N$ denoting the number of pixels) and and $\delta 1$ accuracy (percentage of $\frac{1}{N}\sum_{k=0}^{N-1}\max(\frac{\hat{x}_{d}}{x_{d}},\frac{x_{d}}{\hat{x}
_{d}})<1.25$).
To assess the temporal consistency of video depth, we further introduce the temporal alignment error (TAE):

$$ $\text{TAE}=\frac{1}{2(T-2)}\sum_{k=0}^{T-1}\text{AbsRel}\left(f(\hat{x}_{d}^{k },p^{k}),\hat{x}_{d}^{k+1}\right)+\text{AbsRel}\left(f(\hat{x}_{d}^{k+1},p_{-} ^{k+1}),\hat{x}_{d}^{k}\right),$ (7) $$

where $T$ is the number of frames, $f$ is the projection function that uses transformation matrix $p^{k}$ to map depth $\hat{x}_{d}^{k}$ from the $k$-th frame to the $(k+1)$-th frame, and $p_{-}^{k+1}$ is the inverse matrix for projection in the reverse direction.
The transformation matrix consists of both intrinsic and extrinsic camera parameters, which can be obtained from the dataset.

### 4.2 Implementation Details

Our implementation is based on SVD (Blattmann et al., ), using the diffusers library (von Platen et al., ).
We employ the AdamW optimizer (Loshchilov & Hutter, ) with a learning rate of $6.4\times 10^{-5}$. The model is trained at various resolutions: $512\times 512$, $480\times 640$, $707\times 707$, $352\times 1216$, and $1024\times 1024$, with corresponding batch sizes of $384$, $256$, $192$, $128$, and $64$.
The video length is sampled from $1$ to $6$, with the batch size adjusting correspondingly to meet GPU memory requirements.
Experiments are conducted on 32 NVIDIA A100 GPUs for $20$ epochs, with a total training time of approximately $1$ day.
For training efficiency, we utilize Fully Sharded Data Parallel (FSDP) with ZeRO Stage 2, gradient checkpointing, and mixed-precision training.
During inference, we set the number of denoising steps to $3$ and the ensemble size to $20$ for benchmark comparison, following Ke et al. (), to ensure optimal performance.
In contrast, the ablation studies do not utilize the ensemble strategy to focus on the isolated impact of individual components.
The runtime evaluation is performed on a single NVIDIA A100 GPU with a resolution of $480\times 640$.

**Table 2: Quantitative comparisons with state-of-the-art depth estimation methods using single-frame input across four zero-shot affine-invariant depth benchmarks.**
| Method | # Training Data | NYUv2 | KITTI | ETH3D | ScanNet |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Real | Synthetic | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ | AbsRel $\downarrow$ | $\delta 1\uparrow$ |  |
| Discriminative Model: |  |  |  |  |  |  |  |  |  |  |
| DiverseDepth (Yin et al., ) | 320K | - | 11.7 | 87.5 | 19.0 | 70.4 | 22.8 | 69.4 | 10.9 | 88.2 |
| MiDaS (Lasinger et al., ) | 2M | - | 9.5 | 91.5 | 18.3 | 71.1 | 19.0 | 88.4 | 9.9 | 90.7 |
| LeReS (Yin et al., ) | 354K | - | 9.0 | 91.6 | 14.9 | 78.4 | 17.1 | 77.7 | 9.1 | 91.7 |
| Omnidata (Eftekhar et al., ) | 12.1M | 59K | 7.4 | 94.5 | 14.9 | 83.5 | 16.6 | 77.8 | 7.5 | 93.6 |
| HDN (Zhang et al., ) | 300K | - | 6.9 | 94.8 | 11.5 | 86.7 | 12.1 | 83.3 | 8.0 | 93.9 |
| DPT (Ranftl et al., ) | 1.4M | - | 9.1 | 91.9 | 11.1 | 88.1 | 11.5 | 92.9 | 8.4 | 93.2 |
| Metric3D (Yin et al., ) | 8M | - | 5.8 | 96.3 | 5.8 | 97.0 | 6.6 | 96.0 | 7.4 | 94.1 |
| Depth Anything (Yang et al., ) | 63.5M | - | 4.3 | 98.0 | 8.0 | 94.6 | 6.2 | 98.0 | 4.3 | 98.1 |
| Generative Model: |  |  |  |  |  |  |  |  |  |  |
| Marigold (Ke et al., ) | - | 74K | 5.5 | 96.4 | 9.9 | 91.6 | 6.5 | 96.0 | 6.4 | 95.1 |
| DepthFM (Gui et al., ) | - | 63K | 6.5 | 95.6 | 8.3 | 93.4 | - | - | - | - |
| GeoWizard (Fu et al., ) | - | 0.3M | 5.2 | 96.6 | 9.7 | 92.1 | 6.4 | 96.1 | 6.1 | 95.3 |
| Depth Any Video (Ours) | - | 6M | 5.1 | 97.0 | 7.3 | 95.1 | 4.7 | 97.9 | 5.3 | 96.6 |

### 4.3 Zero-shot Depth Estimation

Our model demonstrates exceptional zero-shot generalization in depth estimation across both indoor and outdoor datasets, as well as single-frame and multi-frame datasets.

Quantitative Comparisons.
Table 2 presents our model’s performance in comparison to state-of-the-art depth estimation models using single-frame inputs.
Our model significantly surpasses all previous generative models across various datasets and achieves results that are comparable to, and in some cases better than, those of top-performing discriminative models.
For example, compared to GeoWizard (Fu et al., ), our model shows improvements of 0.4 in $\delta 1$ and 0.1 in AbsRel on the NYUv2 dataset, 3.0 in $\delta 1$ and 2.4 in AbsRel on KITTI, 1.8 in $\delta 1$ and 1.7 in AbsRel on ETH3D, and 1.3 in $\delta_{1}$ and 0.8 in AbsRel on the ScanNet dataset.
When compared to Depth Anything (Yang et al., ), we achieve gains of 0.5 in $\delta 1$ and 0.7 in AbsRel on KITTI, along with a 1.5 improvement in the AbsRel metric on the ETH3D dataset.
The impressive results are primarily attributed to the large-scale synthetic data we collected.
Table 4.2 presents a comprehensive comparison of our model against previous video depth models.
All generative models process multi-frame inputs in a single forward pass.
Notably, our model demonstrates improved temporal consistency and spatial accuracy on the ScanNet++ dataset, highlighting its effectiveness in video depth estimation.
Table 4.2 presents detailed comparisons with previous generative methods without ensemble techniques.
Our model has fewer parameters than ChronoDepth (Shao et al., ) because we utilize a parameter-free RoPE instead of learnable absolute positional embeddings.
It also reduces complexity compared to DepthCrafter (Hu et al., ) by removing the clip embedding condition and classifier-free guidance.
Additionally, we achieve lower inference time and fewer denoising steps while attaining better spatial accuracy on the ScanNet dataset compared to Marigold, ChronoDepth, and DepthCrafter.

Figure: Figure 5: Qualitative comparisons of depth estimation models on in-the-wild videos. Red boxes show changes in color or depth over time at vertical red lines in videos. Best viewed by zooming in.
Refer to caption: x3.png

**Table 3: Ablation study of each component. All variants are trained for $10$ epochs to ensure training efficiency. Memory Util. refers to the minimum GPU memory utilization across all GPUs, while Average Metric represents the average accuracy across four datasets.**
| Generative | Conditional | Synthetic | Mixed-duration | Runtime | Training Time | Memory Util. | Average Metric |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Visual Prior | Flow Matching | Synthetic Data | Training | (s) | (hours) | (%) | AbsRel $\downarrow$ | $\delta 1\uparrow$ |
| ✗ | ✗ | ✗ | ✗ | - | - | - | 21.0 | 65.1 |
| ✓ | ✗ | ✗ | ✗ | 2.4 | - | - | 7.5 | 93.8 |
| ✓ | ✓ | ✗ | ✗ | 0.37 | - | - | 6.9 | 94.9 |
| ✓ | ✓ | ✓ | ✗ | - | 16 | 23 | 6.5 | 95.6 |
| ✓ | ✓ | ✓ | ✓ | - | 12 | 63 | 6.4 | 95.8 |

Qualitative Comparisons.
Figure 4 presents qualitative monocular depth estimation results across different datasets. It highlights the ability of our model to capture fine-grained details compared to Depth Anything V2 (Yang et al., ), such as the cup in the NYUv2 dataset and the ladder in the ETH3D dataset.
Moreover, our model handles objects with similar colors more effectively.
For example, on the KITTI dataset, the person’s head blends with the background, which Depth Anything V2 fails to predict.
Compared to generative methods like Marigold and ChronoDepth, our model generalizes well to in-the-wild data, offering a better distinction between the sky and foreground.
The diversity of DA-V significantly contributes to this, enabling our model to generalize effectively across various environments, particularly in complex, real-world scenarios.
Figure 5 further demonstrates qualitative results of depth estimation on open-world videos, covering a wide range of scenarios such as dust and sand, animals, architecture, and human motion presented in both generated and real-world videos.
Following Wang et al. (), we visualize the changes in estimated depth values over time at the vertical red lines by slicing along the time axis to better capture temporal consistency.
Marigold exhibits zigzag artifacts on a per-frame basis, while ChronoDepth displays similar issues at a per-window level. DepthCrafter, with its large overlap between windows and interpolation of overlap space, achieves smoother transitions across windows but still struggles with window-wise flickering. In contrast, our method ensures global consistency by predicting key frames and interpolating intermediate frames, significantly reducing flicker artifacts between windows.

Figure: (a) Impact of denoising step
Refer to caption: extracted/5923385/figure/denoise_step.png

### 4.4 Ablation Studies

In this section, we evaluate the effectiveness of each component in Depth Any Video. For training efficiency, unless otherwise specified, the model is trained for only $10$ epochs during ablation studies.

Generative Visual Prior.
We investigate the impact of prior visual knowledge from the stable video diffusion model, as shown in Table 3.
The first two rows clearly demonstrate that incorporating this prior significantly boosts the model’s overall performance.
Additionally, Figure 6(c) illustrates that this prior provides a strong initialization, leading to fast training convergence and enabling the model to achieve impressive results with as few as five epochs. This suggests that the generative visual prior not only enhances model performance but also improves training efficiency.

Figure: Figure 7: Visualization of reconstruction quality.
Refer to caption: x4.png

Conditional Flow Matching.
The second and third rows in Table 3 indicate that flow matching not only reduces inference time, achieving a 6.5$\times$ acceleration compared to the original EDM scheduler in SVD, but also results in improvements of 0.6 in AbsRel and 1.1 in $\delta 1$. The faster inference is primarily attributed to the ability to achieve strong performance with fewer denoising steps, as shown in Figure 6(a).
It demonstrates that even a single denoising step can yield strong results, with the sweet spot identified at three steps for optimal performance.
In Figure 6(b), we further evaluate the effectiveness of ensembling multiple predictions, as varying noise initializations produce minor variations in outputs. The results show consistent performance gains as the number of predictions increases, with improvements becoming less pronounced after five predictions.

Synthetic Data.
In Table 3, the third and fourth rows show that our collected synthetic data brings gains of 0.7 in $\delta 1$ and 0.4 in AbsRel. The accuracy improvement in outdoor scenes is particularly significant, as shown in Table 2, likely due to the diverse range of outdoor environments in our dataset.
Additionally, Figure 4 and 5 demonstrate that models trained on our synthetic data generalize well to in-the-wild scenarios, showcasing both the realism and effectiveness of our synthetic data.

Mixed-duration Training.
The fourth and fifth entries in Table 3 show that the mixed-duration training strategy improves training efficiency while simultaneously enhancing spatial accuracy.
It can save 33% of training time and increase GPU utilization by 40% because different batch sizes can be applied to video sequences of varying lengths, thus optimizing training efficiency.
The accuracy improvement is attributed to the increased proportion of individual frames during training, achieved by randomly dropping out frames, which enhances the single-frame performance.

VAE Variants.
Since our depth is generated in the latent space, the quality of the VAE directly impacts the upper bound of the final result. Therefore, we provide a detailed comparison of the depth reconstruction results across different VAEs in Table 7.
We find that the 2D VAE surpasses the 3D VAE in reconstruction quality, particularly with the VAE from SD3 (Esser et al., ). This indicates there is still potential to further enhance our model’s performance.
In Figure 7, we present visual comparisons of the reconstructions produced by different 3D VAEs.
Specifically, (a) shows the input ground truth depth, (b) displays the reconstruction results from Yang et al. (), which incorporates temporal compression, and (c) shows the results from Blattmann et al. ().
Although the VAE with temporal compression can reduce the computational complexity of the latent model, it struggles to handle fast motion effectively, as indicated by the red box in (b).

## 5 Related Work

Monocular Depth Estimation.
Existing models for monocular depth estimation can be roughly divided into two categories: discriminative and generative. Discriminative models are trained end-to-end to predict depth from images. For example, MiDaS (Lasinger et al., ) focuses on relative depth estimation by factoring out scale, enabling robust training on mixed datasets. Depth Anything (Yang et al., ; ) builds on this concept, leveraging both labeled and unlabeled images to further enhance generalization. ZoeDepth (Bhat et al., ) and Metric3D (Yin et al., ; Hu et al., ) aim to directly estimate metric depth.
Generative models (Saxena et al., ), such as Marigold (Ke et al., ) and GeoWizard (Fu et al., ), leverage powerful priors learned from large-scale real-world data, allowing them to generate depth estimates in a zero-shot manner, even on unseen datasets.
Our work falls into the second category, but focuses on video depth estimation.

Video Depth Estimation.
Unlike single-image depth estimation, video depth estimation requires maintaining temporal consistency between frames.
To eliminate flickering effects between consecutive frames, some works (Luo et al., ; Chen et al., ; Zhang et al., ) use an optimization procedure to overfit each video during inference. Other approaches (Guizilini et al., ; Zhang et al., ; Teed & Deng, ) directly predict depth sequences from videos. For instance, NVDS (Wang et al., ) proposes a refinement network to optimize temporal consistency from off-the-shelf depth predictors.
Some concurrent works (Hu et al., ; Shao et al., ) have focused on leveraging video diffusion models to produce coherent predictions. However, they often face challenges due to a lack of sufficiently high-quality and realistic depth data.

Video Generation.
Diffusion models (Ho et al., ; Song et al., ) have achieved high-fidelity image generation from text descriptions, benefiting from large-scale aligned image-text datasets. Building on this success, VDM (Ho et al., ) first introduces unconditional video generation in pixel space. Imagen Video (Ho et al., ) and Make-a-Video (Singer et al., ) are cascade models designed for text-to-video generation. Align Your Latent (Blattmann et al., ) and SVD (Blattmann et al., ) extend Rombach et al. () by modeling videos in the latent space of an autoencoder.
Our model builds upon the generative visual prior of SVD, which is trained on diverse real video data, to maintain robust generalization in real-world scenarios.

## 6 Conclusion

We present Depth Any Video, a novel approach for versatile image and video depth estimation, powered by generative video diffusion models. Leveraging diverse and high-quality depth data collected from diverse synthetic environments, our model could generate temporally consistent depth sequences with fine-grained details across a broad spectrum of unseen scenarios. Equipped with a mixed-duration training strategy and frame interpolation, it generalizes effectively to videos of various lengths and resolutions. Compared to previous generative depth estimation models, our approach sets a new state-of-the-art in performance while significantly enhancing efficiency.

Limitations.
There are still certain issues in our model, such as difficulties in estimating depth for mirror-like reflections on water surfaces and challenges with extremely long videos. Future work will focus on collecting data for these challenging scenarios and improving model efficiency.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * Insights on why 2D VAE variants surpasses 3D VAE variants? I am wondering why 2D VAE variants outperform. Or asking differently, where do the artifacts of the 3D VAE variant come from? A good video depth model may have a good 3D prior, so 3D VAE might be a more natural choice to encode 3D information, but why does it underperform? Is it more difficult to train 3D VAE properly? Providing any insights or discussion is appreciated.

* In Fig. 7 (a) why do more denoising steps hurt accuracy?

* (nit) When looking at the qualitative examples, there are thin depth boundaries along the person's boundaries (the 5th example, woman, in Fig. 6), probably it's a kind of an average of depth of the person and the background. Why it's the case? What would be the source of this error?
soundness: 3
strengths: * Good empirical results

  The method shows very good empirical results compared with the previous methods. The model outputs consistent depth estimates without temporal flickering, and good accuracy on public benchmarks. The ablation study in Table 3 validates the technical design choices of the method. 

* Clarity

   It's easy to follow the paper. The paper provides sufficient technical details on the datasets, training, architecture, etc.
summary: The paper introduces a video-depth method that is trained on synthetic datasets only. The results show consistent video depth, generalizing diverse real-world scenes across diverse benchmarks. The paper provides practical engineering strategies, e.g., a batching strategy for better usage of RAM, temporal interpolation module, frame dropout, etc.

---

I rate the paper marginally below the acceptance threshold. The empirical results are very strong, but the paper mostly focuses on engineering solutions without scientifically interesting ideas. Also, there might be a license concern on the synthetic depth dataset. I am happy to improve the rating if the questions in the Weakness and Question sections are properly addressed.
weaknesses: * A bit of engineering work

   The paper is mostly about engineering. It adopts conditional flow matching, uses large-scale synthetic datasets to boost accuracy, and introduces mixed-duration training to improve memory usage. All these aspects attribute better accuracy and performance, but it doesn't necessarily provide novel findings. If wanting to emphasize, what would be the most interesting, novel findings of the paper?

* Dataset licence and reproducibility

   It's curious if the collected synthetic dataset can be released or made public. What is the license condition of each game in DA-V? Are there any concerns about using the commercial game engine for research? Is there any plan to release the data? It can affect the reproducibility of the method.


* The effect of the DA-V dataset

   The ablations study in Table 5 shows that the synthetic game data improves the depth accuracy but quite marginally although the dataset size is around 6M. Why doesn't it improve the accuracy significantly? Is there any qualitative improvement that the metrics or numbers don't show?

---

confidence: 5
contribution: 4
initial_rating: 10
presentation: 3
questions: See Weaknesses
soundness: 4
strengths: - The paper constructs a large-scale synthetic dataset of 40,000 video depth clips from 12 diverse modern video games.This dataset provides a scalable and cost-effective way to gather ground-truth video depth data and helps the model generalize to real-world scenarios.
- A mixed-duration training strategy is proposed. It includes frame dropout augmentation with rotary position encoding and a video packing technique.
- Effective Model Design
- The method achieves state-of-the-art performance among generative depth models.
summary: Depth Any Video is proposed for video depth estimation. It has a scalable synthetic data pipeline from game environments. A novel framework uses generative video diffusion models’ priors. It has a mixed-duration training strategy and a depth interpolation method. The model outperforms previous generative depth models, achieving good spatial accuracy and temporal consistency.
weaknesses: The work in the article is very solid, with good model performance and efficiency, and comprehensive evaluation. The only concern for the reviewer is how the author ensures that the dataset, which is a major contribution, will be open-sourced as promised. This is very important for the community, but there are many difficulties regarding copyright and other aspects. In addition, it is necessary to evaluate and compare the diversity of the dataset.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: My main concern is the representation of video depth data. I feel it is not hard to use a shared scale and shift following DepthCrafter, which would be more reasonable for video depth estimation. In addition, I am also curious about the shift from EDM to Flow Matching.

**Post Rebuttal:** My previous questions are mainly addressed. However, the copyright issue remains severe after discussing with authors and other reviewers.
soundness: 3
strengths: 1. The motivation for the framework is clear and reasonable, considering the limited data, inference speed and the long video in the applications.

2. Collecting and annotating high-quality data can improve the model and also inspire the community. It would be more beneficial if the data or collection pipeline can be open-sourced

3. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed method.
summary: This paper works on video depth estimation with diffusion models. First, considering the lack of video depth data, they collect 40,000 videos with depth annotation from video games. With the collected data, this paper fine-tunes the video diffusion model to achieve depth estimation. In addition, rotary position encoding and flow matching are introduced to further enhance the performance. With the proposed techniques, this paper achieves good spatial and temporal consistency on long video depth estimation.
weaknesses: 1. **The representation is more image depth estimation rather than video depth estimation.** If I understand correctly, although the paper focuses on video depth estimation, the predicted relative depth maps are independent for each frame, which is demonstrated in the input normalization and alignment during evaluation. Specifically, each frame is normalized based on the depth range of itself and the scale and shift are also aligned for each frame during inference. In my view, this is incorrect for video depth estimation. To use the accurate relative depth of a video, the scale and shift should be the shared values aligned to the whole video, like DepthCrafter (Hu et al.).

2. **Shift from EDM to Flow Matching.** The SVD model was pre-trained with the EDM denoising scheduler, which has a different optimization objective with flow matching. However, this paper directly fine-tunes the SVD model with conditional flow matching. As far as I know, InstaFlow (Liu et al, ICLR2024) optimized rectified flow for stable diffusion with velocity distillation instead of directly fine-tuning. I hope the authors could provide more explanation about the shift from EDM to Flow Matching. 

3.  A follow-up question about the shift. The ablation study of removing the pre-trained SVD model is required to demonstrate if the EDM pre-trained weights benefit or not.

4. **More explanation about the video interpolation model.** As illustrated in Sec. 3.3, video interpolation model takes the key frames as input and interpolate other frames. Do the key frames replace the original noise? Or are they concatenated on the channels of latent? In addition, why are the mask maps replicated 4 times?

Meta-review: The paper proposes an approach to video depth estimation with a generative model, integrating prior RoPE and flow matching, introducing a strategy for training on video clips of varying lengths, and further proposing a video depth interpolation technique for inference at high-resolution. The paper also describes a data synthesis pipeline to capture video depth ground truth from commercial video games.
It leverages the pipeline to generate a dataset with 40K video clips that is further used to train the proposed model.
The method outperforms prior generative models in spatial accuracy and temporal consistency as demonstrated through experimental results.
 
Reviewers appreciated the strong empirical results and found claims sufficiently validated through controlled experiments.
The paper is well written and easy to follow. 

Reviewers requested additional information on the implementation details, and data synthesis pipeline. These were sufficiently met during the discussion phase. Concerns about insufficient improvements in metrics due to the proposed dataset as pointed out by reviewer ATwr are not shared by the AC - the supposedly small improvements in the delta1 metric would correspond to a ~20% reduction in error, which are significant. The main concern of the reviewers is the legality of the data collection as the proposed method
extracts internal representations from commercial games leveraging open source debugging tools.
This type of data extraction, the use and distribution may conflict with end user license agreements and copyrights.
Notably, prior work has set precedence for this kind of data collection and the authors obtained permission from a subset of concerned game publishers to release the data under specific conditions, which were not fully provided during the discussion period.

The AC considers the contribution and demonstrated experimental results as sufficient for acceptance, but nevertheless agrees with the request for an ethics review by the reviewers. As the dataset is a major contribution of the paper, its release and legality for follow-up research will strongly affect the impact of this paper.

Decision: Accept (Poster)