Depth Any Video with Scalable Synthetic Data

Authors: Honghui Yang, Di Huang, Wei Yin, Chunhua Shen, Haifeng Liu, Xiaofei He, Binbin Lin, Wanli Ouyang, Tong He
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video Depth Estimation, Synthetic Game Data

Abstract:
Video depth estimation has long been hindered by the scarcity of consistent and scalable ground truth data, leading to inconsistent and unreliable results. In this paper, we introduce Depth Any Video, a model that tackles the challenge through two key innovations. First, we develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse virtual environments, yielding 40,000 video clips of 5-second duration, each with precise depth annotations. Second, we leverage the powerful priors of generative video diffusion models to handle real-world videos effectively, integrating advanced techniques such as rotary position encoding and flow matching to further enhance flexibility and efficiency. Unlike previous models, which are limited to fixed-length video sequences, our approach introduces a novel mixed-duration training strategy that handles videos of varying lengths and performs robustly across different frame rates—even on single frames. At inference, we propose a depth interpolation method that enables our model to infer high-resolution video depth across sequences of up to 150 frames. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency. The code and model weights are open-sourced.

Decision: accept (poster)
Reviewer scores: 5, 10, 5

Reviews and discussion:
<thread>
creation date:
1730192748006

thread_type:
official_review

<thread_object>
creator: reviewer_gu5t

<content>
summary:
This paper works on video depth estimation with diffusion models. First, considering the lack of video depth data, they collect 40,000 videos with depth annotation from video games. With the collected data, this paper fine-tunes the video diffusion model to achieve depth estimation. In addition, rotary position encoding and flow matching are introduced to further enhance the performance. With the proposed techniques, this paper achieves good spatial and temporal consistency on long video depth estimation.

strengths:
1. The motivation for the framework is clear and reasonable, considering the limited data, inference speed and the long video in the applications.

2. Collecting and annotating high-quality data can improve the model and also inspire the community. It would be more beneficial if the data or collection pipeline can be open-sourced

3. Extensive experiments and ablation studies demonstrate the effectiveness of the proposed method.

weaknesses:
1. **The representation is more image depth estimation rather than video depth estimation.** If I understand correctly, although the paper focuses on video depth estimation, the predicted relative depth maps are independent for each frame, which is demonstrated in the input normalization and alignment during evaluation. Specifically, each frame is normalized based on the depth range of itself and the scale and shift are also aligned for each frame during inference. In my view, this is incorrect for video depth estimation. To use the accurate relative depth of a video, the scale and shift should be the shared values aligned to the whole video, like DepthCrafter (Hu et al.).

2. **Shift from EDM to Flow Matching.** The SVD model was pre-trained with the EDM denoising scheduler, which has a different optimization objective with flow matching. However, this paper directly fine-tunes the SVD model with conditional flow matching. As far as I know, InstaFlow (Liu et al, ICLR2024) optimized rectified flow for stable diffusion with velocity distillation instead of directly fine-tuning. I hope the authors could provide more explanation about the shift from EDM to Flow Matching. 

3.  A follow-up question about the shift. The ablation study of removing the pre-trained SVD model is required to demonstrate if the EDM pre-trained weights benefit or not.

4. **More explanation about the video interpolation model.** As illustrated in Sec. 3.3, video interpolation model takes the key frames as input and interpolate other frames. Do the key frames replace the original noise? Or are they concatenated on the channels of latent? In addition, why are the mask maps replicated 4 times?

questions:
My main concern is the representation of video depth data. I feel it is not hard to use a shared scale and shift following DepthCrafter, which would be more reasonable for video depth estimation. In addition, I am also curious about the shift from EDM to Flow Matching.

**Post Rebuttal:** My previous questions are mainly addressed. However, the copyright issue remains severe after discussing with authors and other reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652539942

thread_type:
official_review

<thread_object>
creator: reviewer_vgj9

<content>
summary:
Depth Any Video is proposed for video depth estimation. It has a scalable synthetic data pipeline from game environments. A novel framework uses generative video diffusion models’ priors. It has a mixed-duration training strategy and a depth interpolation method. The model outperforms previous generative depth models, achieving good spatial accuracy and temporal consistency.

strengths:
- The paper constructs a large-scale synthetic dataset of 40,000 video depth clips from 12 diverse modern video games.This dataset provides a scalable and cost-effective way to gather ground-truth video depth data and helps the model generalize to real-world scenarios.
- A mixed-duration training strategy is proposed. It includes frame dropout augmentation with rotary position encoding and a video packing technique.
- Effective Model Design
- The method achieves state-of-the-art performance among generative depth models.

weaknesses:
The work in the article is very solid, with good model performance and efficiency, and comprehensive evaluation. The only concern for the reviewer is how the author ensures that the dataset, which is a major contribution, will be open-sourced as promised. This is very important for the community, but there are many difficulties regarding copyright and other aspects. In addition, it is necessary to evaluate and compare the diversity of the dataset.

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730680547870

thread_type:
official_review

<thread_object>
creator: reviewer_atwr

<content>
summary:
The paper introduces a video-depth method that is trained on synthetic datasets only. The results show consistent video depth, generalizing diverse real-world scenes across diverse benchmarks. The paper provides practical engineering strategies, e.g., a batching strategy for better usage of RAM, temporal interpolation module, frame dropout, etc.

---

I rate the paper marginally below the acceptance threshold. The empirical results are very strong, but the paper mostly focuses on engineering solutions without scientifically interesting ideas. Also, there might be a license concern on the synthetic depth dataset. I am happy to improve the rating if the questions in the Weakness and Question sections are properly addressed.

strengths:
* Good empirical results

  The method shows very good empirical results compared with the previous methods. The model outputs consistent depth estimates without temporal flickering, and good accuracy on public benchmarks. The ablation study in Table 3 validates the technical design choices of the method. 

* Clarity

   It's easy to follow the paper. The paper provides sufficient technical details on the datasets, training, architecture, etc.

weaknesses:
* A bit of engineering work

   The paper is mostly about engineering. It adopts conditional flow matching, uses large-scale synthetic datasets to boost accuracy, and introduces mixed-duration training to improve memory usage. All these aspects attribute better accuracy and performance, but it doesn't necessarily provide novel findings. If wanting to emphasize, what would be the most interesting, novel findings of the paper?

* Dataset licence and reproducibility

   It's curious if the collected synthetic dataset can be released or made public. What is the license condition of each game in DA-V? Are there any concerns about using the commercial game engine for research? Is there any plan to release the data? It can affect the reproducibility of the method.


* The effect of the DA-V dataset

   The ablations study in Table 5 shows that the synthetic game data improves the depth accuracy but quite marginally although the dataset size is around 6M. Why doesn't it improve the accuracy significantly? Is there any qualitative improvement that the metrics or numbers don't show?

questions:
* Insights on why 2D VAE variants surpasses 3D VAE variants? I am wondering why 2D VAE variants outperform. Or asking differently, where do the artifacts of the 3D VAE variant come from? A good video depth model may have a good 3D prior, so 3D VAE might be a more natural choice to encode 3D information, but why does it underperform? Is it more difficult to train 3D VAE properly? Providing any insights or discussion is appreciated.

* In Fig. 7 (a) why do more denoising steps hurt accuracy?

* (nit) When looking at the qualitative examples, there are thin depth boundaries along the person's boundaries (the 5th example, woman, in Fig. 6), probably it's a kind of an average of depth of the person and the background. Why it's the case? What would be the source of this error?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732159311582

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We extend our gratitude to all the reviewers for their positive evaluations and constructive feedback. Below, we address the general concerns and questions raised:

**Dataset Release**

We plan to release our dataset and model within two months following the acceptance of our manuscript.
The model and code are already available [here](https://anonymous.4open.science/r/iclr2025-0715/DepthAnyVideo/README.md) for use.
The dataset is currently undergoing cleaning and enhancement, with additional labels (e.g., camera pose) being added to make it more comprehensive and suitable for various 3D tasks, such as 3D reconstruction and SLAM.
These efforts require additional time to ensure thorough and accurate processing.

To comply with copyright regulations, we will perform data anonymization by removing sequences that include distinctive game-specific scenes or characters. Furthermore, we will restrict the dataset's use to personal and non-commercial purposes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes an approach to video depth estimation with a generative model, integrating prior RoPE and flow matching, introducing a strategy for training on video clips of varying lengths, and further proposing a video depth interpolation technique for inference at high-resolution. The paper also describes a data synthesis pipeline to capture video depth ground truth from commercial video games.
It leverages the pipeline to generate a dataset with 40K video clips that is further used to train the proposed model.
The method outperforms prior generative models in spatial accuracy and temporal consistency as demonstrated through experimental results.
 
Reviewers appreciated the strong empirical results and found claims sufficiently validated through controlled experiments.
The paper is well written and easy to follow. 

Reviewers requested additional information on the implementation details, and data synthesis pipeline. These were sufficiently met during the discussion phase. Concerns about insufficient improvements in metrics due to the proposed dataset as pointed out by reviewer ATwr are not shared by the AC - the supposedly small improvements in the delta1 metric would correspond to a ~20% reduction in error, which are significant. The main concern of the reviewers is the legality of the data collection as the proposed method
extracts internal representations from commercial games leveraging open source debugging tools.
This type of data extraction, the use and distribution may conflict with end user license agreements and copyrights.
Notably, prior work has set precedence for this kind of data collection and the authors obtained permission from a subset of concerned game publishers to release the data under specific conditions, which were not fully provided during the discussion period.

The AC considers the contribution and demonstrated experimental results as sufficient for acceptance, but nevertheless agrees with the request for an ethics review by the reviewers. As the dataset is a major contribution of the paper, its release and legality for follow-up research will strongly affect the impact of this paper.