Emerging Tracking from Video Diffusion

Authors: Chenshuang Zhang, Kang Zhang, Joon Son Chung, In So Kweon, Junmo Kim, Chengzhi Mao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: pixel-level object tracking, temporal correspondence, diffusion models

Abstract:
We find video diffusion models, renowned for their generative capabilities, surprisingly excel at pixel-level object tracking without any explicit training for this task. We introduce a simple and effective method to extract motion representations from video diffusion models, achieving state-of-the-art tracking results. Our approach enables the tracking of identical objects, overcoming limitations of previous methods reliant on intra-frame appearance correspondence. Visualizations and empirical results show that our approach outperforms recent self-supervised tracking methods, including the state-of-the-art, by up to 6 points. Our work demonstrates video generative models can learn intrinsic temporal dynamics of video, and excel in tracking tasks beyond original video synthesis.

Decision: withdrawn
Reviewer scores: 6, 5, 6, 8, 5

Reviews and discussion:
<thread>
creation date:
1730345278896

thread_type:
official_review

<thread_object>
creator: reviewer_ekws

<content>
summary:
The paper introduces a simple method of extracting diffusion features from small clips to perform the object tracking task. It enables tracking similar objects, overcoming limitations of per-frame processing proposed in previous works. The performances are quite improved on segmentation-, pixel- and pose-level tracking.

strengths:
- The technical statements are quite clear and compelling.
- The PCA analysis seems interesting.
- The performances are quite improved on three input levels.

weaknesses:
- The progression from per-frame processing to batch processing seems natural, with no leap leading the methodology forward.
- The methodology of using generative models in simple understanding tasks was quite interesting when first DIFT was proposed. However, simply making incremental improvements to these models for marginal performance gains may not fully realize their power.
- Data statistics are missing, how is the number of samples/frames/objects? Why is the dataset claimed as a contribution while reusing from Youtube-VOS?

questions:
- I could not get from Fig.5 whether the feature visualization is a heatmap or something else. If it is a heatmap ranging from 0 to 1, how can one object be distinguished from another?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730539158717

thread_type:
official_review

<thread_object>
creator: reviewer_k917

<content>
summary:
This paper introduces a method from video diffusion, TED, to extract motion representations for object-level tracking tasks. By leveraging diffusion representation and existing label propagation methods, the proposed method shows a very robust identification in similar appearance matching. The experiments demonstrate its effectiveness in tracking tasks and achieving SOTA results.

strengths:
- The paper is well written. It is easy to follow the key idea of the paper.
- the experiments are comprehensive and the results are quite good. Especially, the demos, ie. fig 5, on identical appearance object matching.
- the findings along with understanding are interesting. how to deal with the similar appearance of objects in label propagation is very fundamental in the field. 
- helping understand diffusion models and latent representation from another perspective. It can benefit a board of readers in tracking and diffusion groups.

weaknesses:
- Several improvements can be achieved. It would be better if more metrics were provided to understand the proposed method, i.e. the time-consuming. the details of the proposed framework can be provided in supplementary for readers to replicate the method.

questions:
- What about the time cost of the method? Considering the diffusion model often takes time to infer, how much time it will cost to track, for example, 100 frames? 
- And if the video sequence is very long, would the results decline? 
- as the method is built on standard Unet diffusion models, would some inference speed-up methods in the diffusion model help TED faster?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730567663281

thread_type:
official_review

<thread_object>
creator: reviewer_1z6r

<content>
summary:
The authors leverage latent representations from video diffusion models to capture the temporal information for pixel-level tracking. Without additional training, the proposed method improves tracking performance in various video scenarios, even enabling tracking of similar-looking objects where previous methods struggle. The authors also introduce a new benchmark, Youtube-Similar,  to evaluate the complex scenario of tracking multiple similar-looking objects in real-world videos. The proposed method, TED, is evaluated on several benchmarks and achieves better tracking results.

strengths:
1. TED extracts the temporal information of tracking targets from the video diffusion models to assist in tracking.
2. TED introduce a new benchmark, Youtube-Similar, to evaluate the complex scenario of tracking multiple similar-looking objects in real-world videos. 
3. TED achieves good tracking results, especially on the Youtube-Similar benchmark, to show the better tracking ability to tackle the complex scenario of multiple similar-looking objects.

weaknesses:
1. The tracking results, which only train on ImageNet, are suggested to be provided for fare comparison.
2. The tracking speed and computation cost are suggested to be provided.
3. The authors are suggested to discuss the benefits of the temporal information extracted from diffusion compared with other models like transformers.

questions:
Please see the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730662161672

thread_type:
official_review

<thread_object>
creator: reviewer_wmnb

<content>
summary:
Uses video diffusion models to extract exact motion representations for strong appearances for visual tracking.

strengths:
The paper presents an interesting idea in leveraging visual diffusion for strong appearance correspondence and how it can be used for visual tracking with multiple objects.

weaknesses:
I am not exactly sure how this proposed paper differs from the DIFT method?  They appear to be the same except for perhaps different training data.
I understand that you are leveraging the latent representations from visual diffusion however no where in the paper do you explicitly give the visual tracking algorithm in terms of a flow chart or pseudo code, this would be helpful.
The ablation study refers to t which is the tilmestep in diffusion however when it appears, you also use t for time, so this is confusing to the reader.

questions:
How does this method proposed differ from DIFT (Tang 2023)?, this is not clear?
What is the computational cost to compute the video diffusion model if we wanted to use this method for real time tracking?, or can we do tracking in real time with this step included?  How and in what manner do we include windowing for the video diffusion and how does this blend in for a real time algorithm for visual tracking.
Table 2, it would be good if the datasets were the same for a fair comparison.  The results for your approach and DIFT appear the same and sometimes DIFT is better, comment please?  
It appears that the method is dependent on the dataset used?  I am curious on if the same dataset is used, how does DIFT differ from your algorithm in terms of results
</content>
</thread_object>

</thread>
<thread>
creation date:
1730687140241

thread_type:
official_review

<thread_object>
creator: reviewer_5chb

<content>
summary:
The paper investigates using features from a generative video diffusion model for the task of label propagation in videos. The findings appear to point to using a combination of the image diffusion model and video diffusion model to yield good results.

strengths:
(1) The reported results show improvement over prior works

(2) The motivating toy example is a great way to show the lack of temporal information that the paper is investigating.

weaknesses:
#### Presentation
The paper is lacking in key details, specifically:
 1. What is the method? The closest paper that comes to the actual description of how the label is propagated is probably on L312, "recurrently predict labels for subsequent frames". Appendix A.1 is not much more helpful in explaining how tracking is performed; however, it suggests that the actual method is DIFT with just appended video features.
  2. What are the evaluation datasets? The paper mentions DAVIS, Kubric-Similar and Youtube-Similar; however, these datasets are not detailed, especially Kubric-Similar and Youtube-Similar, which seem to be new. Please include details of these datasets and how they were made/curated, including examples. For Davis, please indicate clearly which year and which "task" flavour (semi-supervised, probably) is being used.

#### Contributions
 3. Some of the provided answers do not seem to be well supported. E.g. on L453 (Fig. 7 caption), it is stated that given the lack of appearance at a high noise level, Rv must capture temporal cues useful for tracking. However, temporal cues must be derived from appearance, which is obfuscated with noise. Similarly, it does not explain why it performs worse when there is no noise. Temporal ques are still available. It is also not clear how these experiments were performed, given the randomness involved; how many trials does each point in the graph represent? What are the variance bounds?
 4. It is also not clear what are the results in Fig. 9? Does each point correspond to the video or average over a dataset? Why does this positive correlation imply that training new video diffusion models will improve tracking? Does this hold for both SDV and I2V models?
 5. The methods reported in Table 2 for supervised are rather old and might leave a false impression of the difference in the performance of supervised vs unsupervised. Looking at the DAVIS 2020 challenge [1] (Davis 2017 is a subset) results, it is clear that supervised performance is in 80+ region. 
 6. Given that the contribution of the work is to append video diffusion features to the image of DIFT. There is arguably lack of significant learnings or exploration presented, as only 2 video diffusion models are tested. For example,  why is I2V better than SDV? Would time-tuned self-supervised features work instead [2]? Why only stop at 2 models? Are there better combinations for image features, as shown in [3]? Why are video features so much worse in isolation?
 7. The performance constraints/cost are not mentioned, but they are usually key considerations in tracking settings, as systems need to be real-time due to safety critical (e.g. self-driving) or UX (e.g. film editing) concerns. What are the performance characteristics, e.g. FPS, GPU memory, to run this method?

---

[1] DAVIS 2020 challenge. https://davischallenge.org/challenge2020/leaderboards.html 
[2] Salehi et al. Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations, ICCV 2023
[3] Zhang et al. A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence, NeurIPS 23

questions:
Please see weaknesses section for all questions. It is critical to consider and address the issues with the presentation, and better explain the experimental details.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733274925087

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Acknowledgements to all the reviewers and ACs

comment:
We sincerely thank all reviewers for their valuable feedback. We address the reviewers' comments below and have revised the paper accordingly (with edits highlighted in blue).


The major novelty of the paper is to explore the temporal features from video diffusion models in perception tasks. We are glad that Reviewer k917 finds how to deal with the **similar appearance of objects in label propagation** is **very fundamental** in the field. Our work is **the first to highlight and solve** this task while prior studies fail. Reviewer k917 also finds that our work helps **understand diffusion models and latent representation** from another perspective, and **can benefit a board of readers in tracking and diffusion groups.** We are also grateful for the reviewers' positive comments and recognition of the following strengths:


* **Improved tracking accuracy over prior studies** (Reviewer 5chB, 1z6R, k917, EKWS).  
* **Our method enables tracking similar-looking objects in label propagation task** (Reviewer 1z6R, k917, EKWS).  
* **Our work analyzes the role of temporal information in tracking**, such as the motivating toy example (Reviewer 5chB) and object matching analysis in Figure 5 (Reviewer k917, EKWS).  
* **Comprehensive experiments** (Reviewer k917). 
* **Clear and compelling technical statements** (Reviewer EKWS). 
* **Excellent contribution and soundness** (Reviewer 1z6R, k917).
* **Excellent presentation** (Reviewer 1z6R) and **well-written paper** (Reviewer k917). 

We believe our work can benefit a board of readers in tracking and diffusion groups as recognized by Reviewer k917, and is worth publishing. We also thank the ACs for their time and effort in evaluating our submission.
</content>
</thread_object>

</thread>


Metareview: