Title: Emergent Temporal Correspondences from Video Diffusion Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Generative Models

Video Diffusion Models

Temporal Correspondence
Abstract: Recent advancements in video diffusion models based on Diffusion Transformers (DiTs) have achieved remarkable success in generating temporally coherent videos. Yet, a fundamental question persists: how do these models internally establish and represent temporal correspondences across frames?  We introduce DiffTrack, the first quantitative analysis framework designed to answer this question. DiffTrack constructs a dataset of prompt-generated video with pseudo ground-truth tracking annotations and proposes novel evaluation metrics to systematically analyze how each component within the full 3D attention mechanism of DiTs (e.g., representations, layers, and timesteps) contributes to establishing temporal correspondences. Our analysis reveals that query-key similarities in specific (but not all) layers play a critical role in temporal matching, and that this matching becomes increasingly prominent throughout denoising. We demonstrate practical applications of DiffTrack in zero-shot point tracking, where it achieves state-of-the-art performance compared to existing vision foundation and self-supervised video models. Further, we extend our findings to motion-enhanced video generation with a novel guidance method that improves temporal consistency of generated videos without additional training. We believe our work offers crucial insights into the inner workings of video DiTs and establishes a foundation for further research and applications leveraging their temporal understanding.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a study on emergent temporal correspondences in video generation using Diffusion Transformer (DiT) models. \nThe authors first construct a synthetic dataset by generating videos, annotating the first frame, and using CoTracker [36] to obtain pseudo ground truth trajectories.\nUsing this dataset, they systematically analyze how well temporal correspondences can be extracted from intermediate features and query-key similarities across different layers and denoising steps.\nBased on these findings, they apply their method to zero-shot point tracking on the TAP-Vid benchmark, achieving state-of-the-art performance compared to both vision foundation models and self-supervised video point tracking methods.\nIn addition, they propose a novel guidance strategy that perturbs cross-frame attention to discourage degraded temporal correspondence during generation. \nThis approach enhances the temporal consistency of the generated videos.\nQuantitative metrics and a user study demonstrate that the proposed guidance method significantly outperforms the baseline in terms of motion fidelity and overall video quality.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The paper is very well written. The narrative is clear, smooth, and logically structured, making the problem easy to follow and well motivated.\n- The preliminary section (section 2) is compact, generic, and well-structured. It effectively narrows the focus to cross-frame attention, setting up the reader for the analysis that follows.\n- Section 3.4 offers valuable insights through a thorough and well-executed analysis. I believe these insights will be highly valuable for the community in understanding DiT video diffusion models.\n- The CAG component for temporal refinement is creative, and has a significant impact on the quality of the generated videos.\n\n**Weaknesses**\n1. The paper lacks a dedicated related work section. Given the relevance of semantic correspondence task, it is important to discuss prior work in this area such as [1–3] to clarify what has already been achieved, as well as approaches that analyze temporal information in earlier video diffusion models like SVD [4].\n\n2. Fig. 3(a,b) is hard to interpret. It is unclear which layer or timestep performs best. Clearer visualizations or summaries would make the findings easier to follow.\n\n3. The object dataset was chosen to focus primarily on animals. It would be helpful to clarify the reasoning behind this choice.\n\n4. To strengthen the evaluation, it would be beneficial to include recent diffusion-based semantic correspondence methods such as [1–3]. Additionally, adding CoTracker results to Table 1 would help position the proposed approach relative to state-of-the-art supervised methods.\n\n5. It would be interesting to evaluate CAG on larger video models to assess whether the observed improvements persist.\n\n\n[1] Luo, Grace, et al. \"Diffusion hyperfeatures: Searching through time and space for semantic correspondence.\" Advances in Neural Information Processing Systems 36 (2023): 47500-47510.\n\n[2] Zhang, Junyi, et al. \"Telling left from right: Identifying geometry-aware semantic correspondence.\" CVPR (2024)\n\n[3] Stracke, Nick, et al. \"CleanDIFT: Diffusion Features without Noise.\" CVPR (2025).\n\n[4] Wang, Qian, et al. \"Zero-shot video semantic segmentation based on pre-trained diffusion models.\" CVPR (2025).\n\nQUESTIONS:\n1. Does the proposed approach generalize to object other than animals, which was the main focus of the evaluation dataset?\n\n2. What is the inference time for the zero-shot tracker and the computational overhead from the CAG component?\n\n3. An interesting future direction would be to explore learnable feature aggregation across timesteps and denoising levels, as proposed in [1]. Additionally, handling ambiguous matching, such as switching between different wheels of the car in the first supplementary video, could benefit from techniques like [2].\n\n[1] Luo, Grace, et al. \"Diffusion hyperfeatures: Searching through time and space for semantic correspondence.\" Advances in Neural Information Processing Systems 36 (2023): 47500-47510.\n\n[2] Zhang, Junyi, et al. \"Telling left from right: Identifying geometry-aware semantic correspondence.\" CVPR (2024)\n\nLIMITATIONS:\nYes in the supplementary material, but I suggest moving it to the main paper and showing some failure cases of both point tracking and refined video generation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DiffTrack, a framework designed to analyze the underlying temporal correspondences between frames (thereby point tracking) during the generation process of video diffusion transformers (DiTs). The paper shows results probing the internal mechanisms of DiTs, investigating which representations (e.g. query-key similarities vs. intermediate features), layers, and timesteps are most critical for tracking. Their key findings are that query-key similarities in the attention mechanism are the primary source of correspondence information, and the tracking quality improves as the denoising process progresses. With these insights, the paper presents two practical applications. First, it shows that video DiTs can be used as zero-shot point trackers, achieving results comparable to existing tracking methods. Second, it introduces a training-free guidance method to enhance motion quality in the generated video, by perturbing the cross-frame attention maps in some key layers to steer the model towards generating videos with fewer artifacts.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper successfully translates analytical findings about the inner mechanisms of video DiTs into concrete applications like tracking and video quality enhancement.\n- The paper is exceptionally well-written and easy to follow. The concepts are explained clearly.\n- The analysis is systematic and provides clear and valuable insights into the behavior of video DiTs.\n\nWeaknesses\n- The central discovery that attention maps in diffusion models capture spatial correspondences is not entirely novel. Lots of prior works found this phenomenon in image diffusion models. This paper's contribution is to verify this unsurprising finding in the context of video DiTs.\n- There's a quite significant gap between the tracking performance of CogVideoX and CogVideoX-I2V, which is revealed in the appendix. It would be more appropriate to discuss this in the main paper.\n- Table 1's message is that CogVideoX combined with the proposed DiffTrack \"outperform all vision foundation models and self-supervised video models\". However, when CogVideoX-I2V is combined with the proposed DiffTrack, its performance is worse than all other baseline methods, including SVD which is an image-to-video model. These results seem to indicate that the proposed DiffTrack achieves mixed results.\n\nQUESTIONS:\n- How well will the findings (e.g. layer selection) generalize across different text prompts?\n- Will different content types lead to different layers that provide strong correspondence information?\n- Using the proposed method on CogVideoX-2B-I2V, the performance is actually a lot worse than SVD, which is also an I2V model. Can you comment on this?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work designs a novel way for identifying which intermediate variables are most relevant to temporal correspondences and leveraging them to solve downstream tasks: point tracking and motion-enhanced video generation. Empirical results verify the effectiveness of proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is generally easy to follow. \n2. Detailed experiments on three video generation models (CogVideoX-2B ，HunyuanVideo and CogVideoX-2B-I2V) are presented.\n\nWeakness:\n1. The fairness and integrity of the comparison of experiments need to further verify. For point tracking task, the implementation of baseline models is not stated. For motion-enhanced video generation task, the results were only compared to the vanilla model. \n2. Given that the best timestep is not always the last one, the generality of the proposed point tracking method needs further validation. For example, DiffTrack for zero-shot point tracking  using the last timestep. While according to the appendix, for CogVideoX-2B-I2V, the best timestep is not the last. This can undermine point tracking performance. The author need to either prove the best timestep is the last for most video generation models or prove using the  non-last timesteps can also achieve good point tracking performance.\n\nQUESTIONS:\n1. I am wondering how the author leveraged the existing vision foundation and supervised video models to handle the zero-shot point tracking task. Does the author also construct the track based on the query-key attention scores of specific layers? Please clarify it to prove the comparison is fair.\n2. The author only compares to the vanilla model, but there should be other relevant works for the motion-enhanced video generation task. Please provide their results to further verify the effectiveness of the method.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to systematically analyze transformer-based video diffusion models (DiTs) to uncover the mechanism responsible for preserving the temporal correspondence. The authors propose a way to measure temporal correspondence through key-point tracking, and create a dataset of prompt-videos to do so. This leads to the finding that key-query 3D attention captures the temporal consistency in the generation. The authors propose to use this property to develop a SoTA video keypoint tracking approach and to improve temporal consistency of the generated videos with guidance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is really well written and easy to follow\n- The idea of analyzing both affinity and matching accuracy is grerat. One is indeed not sufficient.\n- The results in both video generation and point tracking are impressive.\n- I really like the pipeline for video generation with negative guidance.\n\nWeaknesses:\n- My biggest concern is the prompt-video dataset. To establish the dataset, the authors generate videos with a text-to-video diffusion model. Then, to get the annotations, apply SAM and Co-Tracker to the generated videos. I have two questions here:\n    - The SAM and Co-tracker models were trained primarily on real videos. How is the accuracy of those models on the generated videos? Does this introduce an error? I find this gap potentially concerning. Given that some of the generated objects’ appearance may drift over time due to imperfect generation, what would it even mean to match the new halluzitated keypoints to the original frame?\n    - By generating videos from a specific generative model (e.g., CogVideo), doesn’t it make the dataset somewhat biased for evaluating other video generative models?\n\nQUESTIONS:\nQuestions/Suggestions:\n- It feels like the zero-shot point tracking is a separate contribution not aligned with the main message of the paper. Perhaps the authors may want to consider removing or rephrasing it, to make the paper more to the point. \n- Long sequence handling (line 249): each chunk contains the first global frame. How does that affect 3D VAE (or is it done after) and how does this affect the 3D attention?\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}