Emergent Temporal Correspondences from Video Diffusion Transformers
Authors: Jisu Nam
Soowon Son
Dahyun Chung
Jiyoung Kim
Siyoon Jin
Junhwa Hur
Seungryong Kim
Venue: NeurIPS 2025 poster ()
Keywords: Video Generative Models
Video Diffusion Models
Temporal Correspondence
Primary area: deep_learning

Abstract: Recent advancements in video diffusion models based on Diffusion Transformers (DiTs) have achieved remarkable success in generating temporally coherent videos. Yet, a fundamental question persists: how do these models internally establish and represent temporal correspondences across frames?  We introduce DiffTrack, the first quantitative analysis framework designed to answer this question. DiffTrack constructs a dataset of prompt-generated video with pseudo ground-truth tracking annotations and proposes novel evaluation metrics to systematically analyze how each component within the full 3D attention mechanism of DiTs (e.g., representations, layers, and timesteps) contributes to establishing temporal correspondences. Our analysis reveals that query-key similarities in specific (but not all) layers play a critical role in temporal matching, and that this matching becomes increasingly prominent throughout denoising. We demonstrate practical applications of DiffTrack in zero-shot point tracking, where it achieves state-of-the-art performance compared to existing vision foundation and self-supervised video models. Further, we extend our findings to motion-enhanced video generation with a novel guidance method that improves temporal consistency of generated videos without additional training. We believe our work offers crucial insights into the inner workings of video DiTs and establishes a foundation for further research and applications leveraging their temporal understanding.

Emergent Temporal Correspondences from Video Diffusion Transformers
Jisu Nam
Soowon Son*
Dahyun Chung²
Jiyoung Kim
Siyoon Jin
Junhwa Hur
Seungryong Kim
KAIST
2 Korea University
Google DeepMind
https:/ httta://cvlab-tee
github........... io/.iffrrooiiiffcckkkkkkkkk.aack
Prompt: "A fox, its russet fur illuminated by
panorama revealing majestic mountain range cascading into tranquil sea, dotted with islets..."
Prompt
golden hues of dawn, stands on edge of meadow..."
Video
Video
Starting Point(s)
Starting Point(s)
Tralectory Attention
Trajectory Attertton
Figure I: Teaser: DiffTrack reveals how video Diffusion Transformers (DiTs) establish temporal correspon- dences during video generation. Given a prompt and starting points, DiffTrack tracks how individual points align across subsequent frames via cross-frame attention in video DiTs (sccond row). This enables the extraction of coherent motion trajectories (third row) from both generated and real-world videos in zero-shot manner.
Abstract
Recent advancements in video diffusion models based Diffusion Transformers (DiTs) have achieved remarkable success in generating temporally coherent videos. Yet, fundamental question persists: how do these models internally establish and represent temporal correspondences across frames? We introduce DiffTrack, the first quantitative analysis framework designed answer this question. DiffTrack constructs dataset of prompt-generater video with pseudo ground-truth tracking annotations and proposes novel evaluation metrics systematically analyze how each component within the full 3D attention mechanism of DiTs (e.g., representa- tions, layers, and timesteps) contributes to establishing temporal correspondences. Our analysis reveals that query-key similarities in specific, but not all, layers play critical role in temporal matching, and that this matching becomes increasingly prominent during the denoising process. We demonstrate practical applications DrffTrack in zero-shot point tracking, where it achieves tfate-ofthh--a perfor- mance compared to existing vision foundation and self-supervised video models. Further, we extend our findings to motion-enhanced video generation with a novel guidance method that improves temporal consistency of generated videos without additional training.
believe our work offers crucial insights into the inner workings of video DiTs and establishes foundation for further research and applications leveraging their temporal understanding.
"Equal contribution. Co-corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
84 88 powered by Diffusion Transformers (DiTs) [2 [T], have achieved remarkable progress in generating realistic videos with high photo- metric fidelity, temporal coherence, and complex motion. DiTs utilize full 3D attention to process all frames and text single sequence, enabling the effective propagation of spatial and temporal information, thus improving temporal coherence in generated videos. Their strong temporal priors have also led to various downstream tasks, such as video depth estimation 87], pose estimation [9] and 3D/4D reconstruction [2 53 74 80).
Recent video diffusion models 25
28
49 50
54
63
69
Despite the success of these models, fundamental question remains unanswered: How do video DiTs establish temporal correspondences between frames during generation, and how can we explicitly 75 86] explore internal representations in U-Net-based image diffusion models, they mainly focus on two-frame correspondences.
extract them? While existing works [31 42 56
57
In this paper, we introduce DiffTrack, the first in-depth quantitative analysis framework designed to pinpoint temporal matching within the video DiT architecture. DiffTrack provides insights into how and where video DiT' establish temporal correspondences during video generation, enabling the direct extraction of motion information from generated videos (cf. Fig. Tp. Notably, our framework is adaptable to any DiT-based video generative models.
Our analysis investigates 3D attention, a core component of video DiTs, to determine which repre- sentations (e.g. query-key similarities vs. intermediate features), layers, and denoising timesteps are most critical for establishing temporal correspondence among frames.
To systematically analyze their intricate interplays during video generation, we construct a dataset of prompt-generated video using video backbone under analysis and obtain pseudo ground-truth motion tracks from an off-the-shelf tracking method 44]. We then propose novel evaluation metrics that jointly assess temporal matching accuracy and confidence. Given the dataset, we extract descriptors at desired layers and timesteps, estimate temporal correspondence, and evaluate these correspondences using our proposed metries.
Through the analysis, we uncover several key findings. 1) Query-key matching in 3D attention blocks provides clear temporal matching information. 2) A few specific layers play dominant role in establishing temporal matching. 3) Temporal matching strengthens throughout the denoising process.
We further demonstrate the practical value of our analysis through experiments in zero-shot point tracking. By using the most significant feature descriptors for temporal matching, extracted at the optimal layer and timestep identified by DiffTrack, our approach achieves state-of-the-art re- sults compared to existing vision foundation models [0 18 60 68 (5) and supe-vuue video models 3 6 40
52 65 [33, demonstrating the effectiveness of our framework.
Additionally, we extend our analysis to motion-enhanced video generation with a novel guidance technique, Ecros-AAttentin Guidance (CAG). CAG works by perturbing cross-frame attention maps in the most dominant layers identified by DiffTrack, and guides the model away from degraded samples during sampling. Notably, CAG achieves significant gains in both human evaluations and automatic metrics, without requiring additional training, auxiliary networks, or supervision.
In summary, our contributions are:
We identify the importance of understanding temporal correspondence in video DiTs and introduce DiffTrack, a novel framework that quantitatively analyzes and identifies temporal matching information within DiT' during video generation.
We provide a detailed analysis of the open-source video DiT models including CogVideoX 84 HunyuanVidee [4)] and CogVideoX-I2V 84] revealing key insights into their internal mechanisms.
We demonstrate the effectiveness of DiffTrack in zero-shot point tracking, achieving state- of-the-art performance among existing vision foundation and self-supervised video models.
We present motion-enhanced video generation with CAG, a novel guidance method that im- proves the motion consistency of generated videos without auxiliary models or supervision.

---

2 Preliminaries
2.1 Video Diffusion Models
Video diffusion models 25 28
63 69 84 88
generate videos from input text prompts
through iterative denoising. To enhance efficiency, latent video diffusion models 30 perform this denoising in a latent space, typically using 3D Variational Autoencoders (VAE) 84 for spatial and temporal compression. A 3D VAE encodes a video E [(++F))XXWW x3, with its frame count (I+F), height (H), width (W), into compressed latent representation. This latent is then patchified and unfolded into a sequence Zvideo of length (1+f) where f. h, and w are derived from spatial compression ratio (i.e., h=H p and w=W/p) and a temporal compression ratio q (i.e.. f=F/q) with often skipping the first frame [84]. To incorporate text input, a text encoder [67] embeds the prompt into the text embedding Ztext with sequence length S. Then the concatenated embeddings (Zvideo and 7eext). of length (1 f'/ + S, guide the denoising process, enabling text-aligned video generation. Given predetermined noise schedule, forward process progressively adds Gaussian noise to z for timestep f, producing noisier latents Zvideo, 1+11- In the denoising process, a neural network eØ (2vido Zrext; () predicts and removes the noise obtain Zvideo,t-1- aligning with the text prompt. After T denoising steps, the fully denoised latent Zyideo,0 is decoded by the 3D VAE to generate the final video X'.
2.2 Video Diffusson Transformers
Following the success of Sora (54), DiT [1] has become a standard architecture for video gener- ative models [25 28 49 50 84 DiT employs multiple layers of full 3D attention, enabling direct interaction between visual and textual information through attention and feed-forward processing.
At each timestep f. the concatenated sequences of Zvideo.t and Ztext are augmented with 3D positional embeddings, e.g.. RoPE (73] or sinusoidal embeddings [79]. Then, each layer full 3D attention transforms the sequences into queries Qr, keys 1, and values Ve, all lying in (()+))+++))xxd with the channel dimension d. The resulting output of full 3D attention is computed as:
Attn Qt,,, K.,,, V..) = A.,
with A Sof tmax Q.,/K,,/VV)
To understand how the video latent Z and text embedding teex interact in full 3D attention, the notations of query Q₁, and key K,, can be rewritten at the token sequence level:
Qt. Concat QQ
Qu,....
Q1+/ Q
K Concat (K) 1, ....... K. K (
Here, Q and Ki, are projections from the i-th frame latent, with i E from the text embeddings. Concat(-) concatenates these across the token sequences.
--el--Fraaeee A ttention a aSeff-Text ttention
1+5], and Q and Ki are
Cross- Frame ttention = Text Frame A ttention
Ki
KE
K
1+f
klest
A
A
A
A,1
1.1
Laext
Fig. 2 iillustrates full 3D attention in video DiTs. It can be categorized into four key interactions: 1) self-frame atten-
23
2:1
A?
A
A 2.1+/
A
tion Ai,, 2) cross-frame attention A, 3) text-frame at- tention A,, text..) A i,text and 4) self-text attention ! where i.j E [1, 1+f] with i * j. Self-frame attention at- tends to its own frame, focusing on nearby spatial patches the pixel location. Text-frame attention injects semantic information from text embeddings into the frame latents. Self-text attention refines the textual context itself.
A31
A33
AC
A,
3.1+f
Ar
3.text
1+7
14fa
18f2
A1
1453
A 1+5 i)
A
Of these, cross-frame attention A E kgwwxhw is of par- ticular interest to our study. Its ability to allow every pixel in one frame to interact with any pixel in another frame is what explicitly enables the model to build and understand temporal relationships across the video sequence.
A
text1
Atext2
A
text3
A
testans
A
lext
Figure 2: Illustration of full 3D attention in video DiTs. where video frame latents and text embeddings interact.
3

---

Starting Points
Starting Points
"A curious sea turtle, its shell dappled with Prompt: "A breathtaking aerial view captures the hues of emerald and bronze, drifts gracefully through the clear blue waters of a vibrant coral reef. Sunlight long shadows across the rugged terrain. The drone filters through the gentle waves, ...
Prompt:
light dawn spills over mountain peaks, casting
glides smoothly, revealing a tapestry of colors..."
(a) Object Dataset
(b) Scene Dataset
Figure 3: Our curated evaluation dataset includes: (a) an object dataset for dynamic object-centric videos and video
(b) scene dataset for static scenes with camera motion. Each dataset comprises 50 pairs per video generative model (e.g. CogVidcoX-2B (84). In the benchmark, we predefine starting points the first frame and obtain pscudo ground-truth trajectories using an off-the-shelf tracking method 44
3 DiffTrack
We introduce DiffTruck, a framework to quantify how video DīTs capture temporal correspondence during video generation. This crucial yet nontrivial due to the inherent complexity of video DiTs, which involve multiple layers, denoising timesteps, and full 3D) attention. To systematically analyze them, DiffTrack provides an evaluation dataset and novel metrics for assessing the accuracy, confidence, and influence of temporal matching. While present an in-depth analysis of the open-source video DiT model, Cog VodeoX-28 884, our framework is broadly applicable to other DiT' architectures, demonstrated by our analyses of VodeoX-555 (84). Hunyuan Video 449 and Viideo--5--B- 84] in Sec.
3.1 Evaluation Dataset
To accurately investigate temporal correspondence during video generation, it's crucial simulate the generation process faithfully. Reconstructing real-world videos often introduces challenges such as inversion errors [7] due to imprecise prompts and distribution shifts from training data. To circumvent this, we curate specialized evaluation prompts and generate corresponding video datasets using the specific backbone under analysis. This approach allows perfect video reconstruction during evaluation, as we use paired prompts with fixed seeds. For an analysis using real-world videos from DAVIS 664. please refer Sec.
To analyze videos with object and camera motion, we curate two distinct datasets: (1) an object dataset for dynamic object-centric videos and (2) scene dataset for static scenes with camera motion. Each dataset includes 50 text prompts with corresponding 50 videos (e.g. 480 720 resolution, 49 frames, generated by Cog X-g [4
To assess temporal matching, the datasets also include pscudo ground truth track annotations. We predefine set of starting points p' E Rg (in latent resolution) in the first frame of each video, where is the number of points. For the object dataset, we use SAM [48] to segment foreground objects and sample grid points with spacing of 1/20 of the video's 's spatial resolution. For the scene dataset, we sample 10 x 10 regular grid of points across the entire frame. Fig. shoows examples of each dataset; further details are provided in Sec. A Since ground- ground-truth truth tracks unavailable for generated videos, we use an off-the-shelf tracking method, CoTracker [44]. to obtain pseudo ground truth T E RR FxNx2 (in original resolution).
3.2 Temporal Correspondence Estimation within Video DiTss
To evaluate matching accuracy, we first extract temporal correspondences across video frames using feature descriptors from video model. We independently establish pairwise correspondences between the first frame and each subsequent frame.
We first construct matching cost between the deseriptors from the first latent and each j-th latent, with j E f 1+f), at each timestep t and layer l:
c Sof tmxx (D. (D,,
(D,,)) /vd)..
(5)

---

where D and Di,, are the feature descriptors corresponding to the first frame latent and the j-th latent, and is the channel dimension of both D and D',- Softmax() is applied over the keys for each pixel in the query. Descriptors Dr, are internal representation candidates from video DiTs, such intermediate features after attention blocks or query-key matrices within attention blocks.
Matched correspondence points Pi, E R at the j-th frame latent, timestep i, and layer I, with the number of points .N, are obtained through Argmax, which identifies the spatial location x of the highest value in C within the spatial domain of the j-th latent Q.
P = Argmax Ct (p',x). xen
(6)
Motion tracks along video frames are obtained by concatenating (Concat) the starting point p' with the estimated matches p across the video latent space. These tracks, in the latent spatial coordinates, are then apatio-temporally upscaled to the original RGB coordinates through linear interpolation (Interp), yielding Ty, E R FxXx2
Tw = Interp(Concat(p' .i.,......P!! pfr .PTT))
3.3 Evaluation Metrics
Given the estimated matched points, we propose three complementary metrics for evaluating temporal matching in video generation: matching accuracy, confidence score, and attention score. Specifically, matching accuracy measures the precision of estimated tracks across frames. The confidence score quantifies the certainty with which the starting point attends its estimated match. The attention score reflects the relative strength of cross- cross-frame frame attention during generation, compared to self-frame and text-frame attention.
Matching Accuracy. We evaluate point accuracy using the percentage of correct keypoints (PCK) with predefined error threshold between the estimated track Tu, and visible points in the ground truth T. The matching accuracy averages PCK over all visible points across cross- lats-ts and videos.
Confidence Score. We use the maximum attention values between the first and j-th frame latent A (cf. Sec. 2.2h to measure how confidently the starting points p attend to their predicted matched points, formulated as:
M
Max
A 1.1 (p' xx), XEi
(8)
which quantifies the attention of p' to the estimated point in the j-th latent at timestep l and layer I. Max takes the maximum attention value over all spatial locations Ω. The confidence score is the average of these maximum values across all cross-latents with j E (2,..., 1+ all visible points in ground truth, and videos.
Attention Score. We use the sum of cross-attention values across all cross-frame latents. This allows us to assess the influence of cross-frame interactions during video generation, in comparison other types of attention, such as text-frame or self-frame attention. This formulated as:
St, = >>
A (p' x))
(9)
JEFxER XEN
which quantifies the sum of attention from p' across cross-frames at timestep t and layer I, over all spatial locations x E Ω and all cross-frame indices j E F. where F = [2, 1+1]. The attention score the average of these summation values across all visible points in the ground truth and all videos.
Three metrics must be considered together (as detailed in Sec. 3.4), as none alone is sufficient to ensure temporal matching. For instance, even with high matching accuracy across frames, it may not influence the generation process when attention scores from others (e.g., self-frame or text-frame) are higher. In another case, high confidence indicates the certainty of the matching score but does not ensure the correctness of the match. We thus compute the harmonic mean of the normalized matching accuracy, confidence score, and attention score to identify instances where all three metrics are high.
5

---

Intermediate Feature Matching
Query-Key M Matzhing
8
matchiig
Hamnonnic Mean
1.0
Object Dotreet
Mlaun HHrnnnii
0.8
Acc Timeston (1
0.6
82
1
Timestep (9)
0.2
Sannse ())
Layer (
10
20
Layer (1)
10
30
Layer ())
10
30
30
0.0
50
30
20
Timestep (7)
Matching
Hamnonic Mean
1.0
68
Mlaun Hirnnni
0.8
Acc. Timestep (1)
8.
0.4
Timestop (1)
Timestep (
20
0.2
1
21
17
Layer
Layer (I)
30
Layer (l)
50
Timestep (f)
30
20
(a) Representatinn Selection
(b) wiser A nalysis
(c) Noise-level Analysis
Figure 4: Analysis of temporal matching in -- (84). (a) Query-key matching outperforms
intermediate feature matching. highlighting the effectiveness harmonic mean of query-key matching shows that temporal matching is primarily driven by a few specific layers. (c) Temporal matching improves progressively during the denoising hut slightly degrades near the final steps.
Fross-frame interactions 3D attention. (b) The
3.4 Analysis
With DiffTrack, we systematically analyze Cog Videoo-22 [8 in the context of temporal correspon- dence. For all analyses, we use full 3D attention in the model, which consists of 30 layers with 50 denoising timesteps.
Our analysis considers the following three perspectives. Representation selection compares intermedi- ate features and query-key representations. Layer-wise analysis explores how well temporal matching encoded at different depths within the attention blocks. Noise-level analysis examines how temporal
relationships evolve throughout the denoising process. Further in-depth analysis is provided in Sec.
E
Additional analyses of Cog VideoX-o555 84), Hunyuan Video 44) and CogVideoX-SBB VideoX-5B--12
[84 are
provided in Sec. B
Representation Selection. Fig. 4 a) compares the accuracy of intermediate feature matching, where features are extracted after each attention layer, and query-key matching, where queries and keys are obtained within each attention layer. Our results indicate that query-key matching consistently outperforms intermediate feature matching. This finding aligns with prior works (57, in which query-key matching captures geometric relationships for correspondence, while values contain visual appearance, potentially diluting geometric cues for accurate matching.
Layer-wise Analysis. Fig. 4() presents the harmonic mean of query-key matching across all timesteps and layers to identify which feature descriptors at which layer and timestep play a predomi- nant role in temporal matching. We observe that the top-20 scores (red) originate from the same layer, indicating that a specific layer predominantly governs temporal correspondence. This behavior is further observed in the top-50 scores (green): limited set of layers drives temporal matching.
Noise-level A Analysis. Fig. cc) presents the har- monic mean of query-key matching across timesteps in the selected layers from Fig. 4Tb), which are iden- tified as leading layers for temporal matching. Tem- poral matching improves during the denoising pro- cess but slightly degrades toward the end. This because earlier timesteps (1) contain noisier latents, which hinder precise temporal matching, and (2) rely heavily on text embeddings and self-frame attention to establish the overall video semanties and layout. Toward the final timesteps, (3) self- and text-frame at-
1.0
1.0
Seli-Prame
Self-Trame
Attention Sanre
Cooa-Frame
Atteattonn Senre
Cros-Framn
Tess- Frane
Teat-Frame
0.6
0.6
0.4
0.4
90
40
30
20
10
90
40
30
30
10
Timestep (0)
Timestep (1)
Object Datuset
Scene Dutaset
Figure 5: Evolution of attention scores across timesteps.

---

1.0
Matshing Cost
Frame
Frame 17
Frame
Soll-Frmme
Top 20
Attention Score
0.3
Timessen (1)
Timestop ()
0.6
Text Frame
25
30
Query
Sanmy
88 B 1
0.4
Top 20
15 20 25
0.0
50
40
30
20
Layer (!)
10
15
20
25
30
Key
Layer (1)
(a)
Timestop
(b)
(a)
Figure 6: Analysis of positional bias temporal matching.
Figure 7: Impact of persistent text-frame attention on attention score.
tention slightly increase again refine the remaining appearance details, once the motion in the synthesized video has been established.
This observation is supported by Fig. 5 which shows the attention scores for text-frame, relf-frame, and cross- fosss-frmmmeeee attention along timesteps. After the carly denoising steps, text-frame and self- frame attention remain low, reducing the influence of textual guidance. Meanwhile, cross- moss-frame attention remains the most influential, enhancing cross-frame coherence, with a slight drop near the end the timesteps.
Additional Analysis on Metrics. We further emphasize the importance of jointly considering
three metrics, matching accuracy, confidence score, and attention score. Fig. ([a) presents the top-20 timesteps and layers where the confidence score high, but matching accuracy low. We find that specific layers exhibit this discrepancy they are everhhelmed by positional information induced by positional embeddings at each timestep. In Fig. Kc). PCA visualization of queries and keys in these layers reveals dominance of positional cues, while Fig. b[b) shows that in these layers, matching cost visualizations indicate that points the first frame tend to match exactly with their initial locations in other frames. This suggests that these points are not correctly matched to their actual counterparts but instead strongly attend to the same spatial location across frames, reflecting the impact of positional bias.
Additionally, Fig. (7 presents the top-20 timesteps and layers where matching accuracy and confidence scores are high, but attention scores are low. We observe that this property is exhibited in certain layers. As shown in Fig. 5 this occurs because text-frame attention remains highly active in these layers, maintaining value around 0.5, unlike in other layers where text-frame attention drops below 0.2 (cf. Fig. 5). This reduces the attention scores, which in turn limits the influence of cross-frame interactions during the generation process.
4 DiffTrack for Zero-Shot Point Tracking
Drff enables the joint extraction of motion trajectories and video generation, selecting the optimal layer and timestep based on matching accuracy. We demonstrate this in zero-shot point tracking (5) on real videos, without training specialized architectures [19] or fine-tuning video
diffusion models [1 To achieve this, we use the inverted noise of real videos obtained through DDIM inversion (71 at the selected timestep and extract features from the chosen layer. Notably, the inversion error is negligible, as we use the final timestep & 1 based on our analysis of matching accuracy in Fig. a). However, this still faces challenges such as temporal context loss from 3D VAE compression and handling of long-term video sequences. We address these challenges below. The overall architecture and its details are provided in Sec. C
Temporal Compression in 3D VAE. As discussed in Sec. 2.1 the 3D VAE temporally compresses video frames into a single-frame frame latent with a compression ratio While linear interpolation can recover motion trajectories from the latent to the RGB video space, often fails to capture per-frame motion details, limiting accuracy. To mitigate this, we set q=1 to establish a direct one-to-one mapping between each video frame and its latent, thereby avoiding temporal compression and enabling precise tracking. In Sec. E we demonstrate that the one-to-one mapped latents can still reconstruct the original videos using the 3D VAE decoder.
Long-term Video Sequences. The fixed temporal resolution pre-trained video models (e.g. 49 frames in Cog VogeeXXX222B (84)) limits their ability to model long-term contexts. Naively splitting
7

---

Starting Points the First Frame
Starting Points in the First Frame
Difllackk Dattrack
(Cog (CogVideos) -B) mayamm
DiffTrcck
Time
Time
Figure 8: Qualitative comparison. Cog VideoX-22
84
combined with Diff produces smoother and which struggle with temporal dynamics and often yield
accurate trajectories than DINOv2 [60] and VFS 83
inconsistent tracks.
Kinetics
DAVIS
Backhone
<
<
<
83
<
DINO (VTT-B/16)
10
2.8
10.9
33.9
58.2
74.8
36.1
2.9
10.7
34,3
59.8
77.0
37.3
DINOv2-Reg Reg (VT-B/114
2.8
10.8
32.8
60.6
78.7
37.1
11.7
36.5
67.5
84.1
40.6
DINOv2 (ViT
B/14)
(1)
3.0
11.4
34.6
63.0
80.3
38.4
3.1
12.1
38.7
70.7
85.8
42.1
DIFT (SD1.5)
3.7
14.6
44.6
69.0
77.5
41.9
3.5
13.0
39.3
63.1
72.2
38.2
DIF2
3.7
14.9
45.4
70.9
79.6
42.9
13.3
40.1
65.8
75.7
39.7
SMTC (ViT-S/16)
65
4.1
15.5
34.2
$4.4
72.1
36.1
2.6
12.1
29.4
$2.5
73.0
33.9
CRW (ResNet-18)
40
5.2
19.4
42.7
62.9
74.3
40.9
3.1
13.9
34.7
57.1
70.5
35.9
Spa-then-Temp (ResNet-50)
52
5.3
19.4
41.6
58.9
69.7
13.8
33.1
53.4
67.5
34.2
VFS (ReaNet-550)
33
5.4
20.1
44.6
65.4
76.6
42.4
3.5
15.2
37.2
60.8
75.2
38.4
SVD [
14.6
71.4
ZeroCo (CroCo)
3
14.5
22.9
35.9
60.4
79.7
42.6
4.6
8.8
19.5
44,9
65.6
28.7
DiffTrack (Hunyuan Video
(9
5.9
22.0
49.1
70.4
80.3
45.5
4.4
18.2
44.8
70.1
82.8
44.1
DifTTrack (CogVideoX-2B
79.9
84.3
DifTrack (Cog V5d
84
6.8
25.9
55.4
74.9
82.7
49.2
5.2
20.5
50.7
73.9
84.3
46.9
Table 1: Quantitative comparison on the TAP-Vid datasets [19].
combined with Track outperform all vision foundation models trained on single images and self-supervised models trained two-view images videos for zero-shot tracking.
Video DiTs
449
84
and processing video chunks separately disrupts direct temporal correspondence with the global first frame. To address this, we construct each chunk to include the global first frame, maintaining a direct temporal connection while interleaving subsequent frames minimize large motion changes.
4.1 Experimental Settings
Implementation details and ablation studies are provided in Sec. and Sec. F.I
We evaluate zero- rero-shot tracking on two real-video datasets with precisely annotated tracks: Vid--Dd-DAV [19] and TAP-Vid- Min-Kinetiss [19]. following [46]. We measure the position accuracy of estimated tracks as the percentage of predicted points within thresholds from visible ground-truth points. We adopt five threshold levels [16 19 46 : 80 81, 52, 83 8*, corresponding to pixel distances of 1, 2, 4, 8,
and 16, respectively,
report the average accuracy across all thresholds as dg. Starting points are sampled from the first frame in [5].

---

CogVideoX-50i
Cross-Ateetionn Guidance on CogVideoX-SH
AMIU
Prompt: .... He takes slow, appreciative sip, his eyes closing momentarily as he savors the complex flavors.
Prompt:
"A determined individual. ascends thick, rugged rope hanging from towering rock face.
Qualitative comparison with baseline [4] and CAG. CAG enhances temporal matching and corrects motion inconsistencies in the synthesized videos.
Figure 10:
4.2
Experimental Results
We compare our method with existing vision foundation models trained on single images 10
18
60
68 75 and self-supervised models trained on two-view images or videos 40
52
65
As shown in Tab. [ our approach achieves superior performance on both the Kinetics and DAVIS datasets, ultimately obtaining the highest average accuracy in 6ag- The results highlight our in dn-tepth analysis of temporal matching within the full 3D attention mechanism of video DiTs.
Fig. 8 shows predicted motion trajectories on the DAVIS dataset, alongside qualitative comparisons
between our method and prior approaches (60, 83 Unlike previous methods, which struggle with
temporal dynamics and often yield inconsistent tracks, DiffTrack on CogVideoX-2B [84] produces
smoother and more accurate trajectories. Additional qualitative and quantitative results are provided
Sec. G and Sec. H
5 DiffTrack for Motion-Enhanced Video Generation
We extend our findings to generate motion- enhanced videos by improving temporal corre- spondence within full 3D attention. As illus- trated in Fig. 9 we propose Cross-Attention Guidance (CAG), a novel diffusion guidance technique applied at specific layers identified Fig. 4b).. steering video generation to- ward motion-enhanced samples. Unlike prior works [1 that require large-scale video- trajectory training pairs, CAG requires addi- tional training, external conditions, or auxiliary modules, and operates entirely within the exist- ing diffusion framework.
Qui
viideo
V Pull 3D A thenton
Eo (Zerein Arsa. Eg T Teman
Q+
Ass
K2
V,
Zeroing Out
Zvideo.f
Cross- A ttention
Inspired by PAG (2). CAG simulates degraded motion by zeroing out selected cross-frame at- tention maps (e.g. | = 13, 17, 21 in Cog VideoX-2B, based on the harmonic mean in Fig. 4()),, then guides the diffusion model away from these degraded samples, promoting temporally coherent video generation. This can be formulated as:
Figure 9: Overall architecture of CAG.
(Z Zeextt 1) Eg (Zvideeo, Zrext 1) +8 (eess(EEggiiaa (Eg(Zvideo,t> Zrexts 'gg Brext: ()), (10)
where (-)-) is the standard noise prediction at timestep i, conditioned on the text. én(-) denotes the (gj in selected
noise prediction from a perturbed forward pass, where cross-frame attention maps A layers are zeroed out to simulate motion degradation, producing Á & the guidance scale, and the final guided prediction éo steers the model to denoise away from motion-degraded samples.
(.)

---

5.1 Experimental Settings
Further implementation and evaluation details are provided in Sec. D and ablation studies are included in Sec. F2 We evaluate CAG against its baselines, Cog and CogVideoX-SB, using both automatic metries and human evaluation.
For automatic metrics, we report temporal quality metrics: Subject Consistency, Background Consis- tency, and Dynamic Degree from Vbench [38]. Subject Consistency and Background Consistency measure the temporal coherence of subject and background appearance, respectively. While static scenes can achieve high scores these metrics, we additionally calculate Dynamic Degree quan- tify motion dynamics. We also report a frame-wise quality metric from Vbench, Imaging Quality, which detects frame-wise distortions.
For human evaluation, we follow the Two-Alternative Forced Choice (2AFC) protocol 6
I 68).
where each rater compares video from the baseline with video from method (baseline + CAG), and selects one based on overall video quality, motion, and text-video alignment. We collect 750 responses from total of 25 participants for each baseline.
5.2 Experimental Results
In Tab. 2 CAG outperforms the baseline across all human evalua- tions. CAG achieves higher scores in Subject Consistency, Back- ground Consistency, Dynamic De- gree, and Imaging Quality on VBench, indicating that our guid- ance improves motion dynamics while enhancing motion consisteney and overall video fidelity.
Human Eval
Metrics
Metbed
Viden
Motion Fddelry
Text Farmtileess
Subject Consstency
Bookground
Dysunic
Inaging
Quality
Consistency
Degree
Quality
VigVideeX 29 CogVidaoX-28- CAG
12:85
14.99
18.56
0.9276
0.5400
0,7917
0,5667
87.16
$5.00
81.44
0L9864
0.4644
CidMideXX
0.686
0,6051
Table 2: Quantitative results of CAG. Human evaluation reports the percentage of votes.
In Fig. 10, the baseline often fails to generate consistent and natural motion, resulting in physically implausible motion (e.g. drinking beer in the first row) or blurry and disjointed appearances (e-g.
blurred and fragmented human body in the second row). In contrast, CAG effectively corrects these artifacts by enhancing temporal matching and motion consistency, ensuring that corresponding More qualitative results are provided in Sec. G
physical points remain coherent across frames.
We further compare CAG with Spatiotempo- ral Skip Guidance (STG) 339, which also aims to enhance motion without additional training. STG degrades the original model by selectively skipping spatiotemporal layers (including self-
Auto. Metries
Method
Suhject Consissency
Rockgrund
Dynamic
Imeging
Consistency
Degree
Quality
CogNideoX-28
0.9276
0.7917
0.5657
CogVidesX-2B
0.9763
0.9507
0.7777
0.6031
CagVilasX-2B 4 CAG
0.9313
0.9564
0,8235
Table 3: Quantitative comparison between STG and CAG.
frame and text-frame attention) and then uses the degraded model as guidance.
As shown in Tab. 3 CAG outperforms STG across motion consistency, motion dynamics, and video quality. While STG modifies both self-frame and text-frame attention and often distorts scene layout or content, CAG only zeroes out cross-frame attention in dominant layers, enhancing motion while preserving the original spatial structure and semanties. Moreover, STG does not analyze how temporal correspondence emerges within video DiTs, thus requiring hcuristic layer selection for each model, whereas CAG leverages DiffTrack-basee analysis to identify optimal layers for motion-enhanced guidance.
6 Conclusion
We introduce DiffTrack, framework for analyzing temporal correspondence in video DiTs, revealing how these models establish temporal correspondences during video generation. Our analysis identifies the crucial role query-key similarities within specific layers of the full 3D attention mechanism and shows that temporal matching strengthens during denoising. We demonstrate the practicality of DiffTrack in zero-shot point tracking and motion-enhanced video generation, paving the way for leveraging temporal understanding in downstream tasks and improving video generation quality.
10

---

Acknowledgments and Disclosure of Funding
This research was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) RRS-2019-11190075, RS-2024- 00509279, 15--005-II121000
20222222222299222. RS-2025- 02214479, RS-2024-00457882, RS- 2025-25441833 1838, DS--2025-255 25441838,, RS-2025-022 20055-22144799 RS-2025-022) (112500221259) and the Culture,
Sports, and Tourism R&D Program through the Korea Creative Content Agency grant funded by the Ministry of Culture, Sports and Tourism (RS-2024-00345025, RS-2024-0033306 RS-2023- 00222280,
and National Research Foundation of Korea (RS-2024-003465977)

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, GPT-4 techmical report. arXiv preprint N.XY::2333.087744 2023.
[2] Donghoon Ahn, Hyoungwon Cho, Jaewon Min, Wooseok Jang, Jungwoo Kim. SeonHwa Kim, Hyun Hee Park, Kyong Hwan Jin, and Seungryong Kim. Self-rectifying diffusion sampling with perturbed-attemm guidance. In ECCV, pages 1-17, 2024.
[3] Honggyu An, Jinhyeon Kim, Sconghoon Park, Jacwoo Jung, Jisang Han, Sunghwan Hong, and Scun- gryong Kim. Cross-view completion models are zero-shot correspondence estimators. arXiv preprint arXiv:2412.0907,, 2024.
[4] Jie An, Songyang Zhang, Harry Yang, Sonal Gupta, Jia-Bin Huang, Jicbo Luo, and Xi Yin. Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv arep 20:4073040088477, 2023.
[5] Görkay Aydemir, Weidi Xie, and Fatma Güney, Can visual foundation models achieve long-term point tracking? arXiv preprint arXiv:2408. 13575, 2024.
[6] Andreas Blattmann, Tim Dockhor, Sumith Kulal, Daniel Mendelevitch, Macicj Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023.
[7] Andreas Blatmannn Robin Rombach, Huan Ling, Tim Dockhor, Scung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, pages 22563-22575 2023.
[8] Minghong Cai, Xiaodong Cun, Xiaoyu Li, Wenze Liu, Zhaoyang Zhang. Yong Zhang. Ying Shan, and Xiangyu Yue. DiTCtrl: Exploring attention control in multi-modal diffusion transformer for tuning-free multi-prompt longer video generation. arXiv preprint arXiv:2412.1 2255:977/2/88997, 2024.
[9] Ruojin Cai, Jason Y. Zhang, Philipp Henzler, Zhengqi Li, Noah Snavely, and Ricardo Martin-Brualla, Can generative video models help pose estimation? arXiv preprint arXiv:2412. 14141.1..55... 2024.
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pages 9650-9660, 2021.
[11] Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak. Yaniv Taigman, Lior Wolf, and Shelly Sheynin. VideoJAM: Joint appearance-motion representations for enhanced motion generation in video models. arXiv preprint arXiv:2552..02492. 2025.
[12] Haoxin Chen, Menghan Xia, Yingqing Hc, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310. 19512, 2023.
[13] Scokju Cho, Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Scungryong Kim. CATs: Cost aggregation transformers for visual correspondence. NeurlPS, 34:901 1-9023, 2021.
[14] Seokju Cho. Sunghwan Hong, and Seungryong Kim. CATs++: Boosting cost aggregation with convolutions and transformers. IEEE TPAMI, 45(6)7174-7194, 2022.
[15] Scokju Cho. Jiahui Huang, Scungryong Kim, and Joon- YoungY Lee. Scurat: From moving points to depth. arXiv preprint arXiv:2504. 14687, 2025.
[16] Seokju Cho, Jiahui Huang, Jisu Nam, Honggyu An, Seungryong Kim, and Joon- Joon-Young Lee. Local all- pair correspondence for point tracking. In ECCV, pages 306-325. 2024.
[17] June Suk Choi, Kyungmin Lee, Sihyun Yu, Yisol Choi, Jinwoo Shin, and Kimin Lee, Enhancing motion dynamics of image-to-viee -oiicooo models via adaptive low-pass guidance. arXiv preprint arXiv:2506.08A. 2025.
[18] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint 2339v: /6588, 2023.
[19] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. TAP-Vid: A benchmark for tracking any point in a video. NeurIPS, 35:13610- 13626, 2022.
12

---

[20] Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. TAPIR: Tracking any point with per-frame initialization and temporal refinement. In ICCV, pages 10061-10072, 2023.
[21] Patrick Esser, Sumith Kulal, Andreas Blattmann. Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, al. Scaling rectified flow transformers for high-resolution image synthesis. In ICMI,, 2024.
[22] Yuming Fang. Hanwei Zhu, Yan Zeng, Kede Ma, and Zhou Wang. Perceptual quality assessment of smartphone photography. In CVPR, pages 3677-3686, 2020.
[23] Ruiqi Gao, Aleksander Holynski, Philipp Henzler, Arthur Brussee, Ricardo Martin- Brualla, Pratul Srini- vasan, Jonathan T. Barron, and Ben Poole. CAT3D: Create anything in 3D with multi-view diffusion models. arxiv preprint arXiv:2405. 10314, 2024.
[24] Daniel Geng, Charles Herrmann, Junhwa Hur, Forrester Cole, Serena Zhang, Tobias Pfaff, Tatiana Lopez- Guevara, Carl Doersch, Yusuf Aytar, Michael Rubinstein, et al. Motion prompting: Controlling vidco generation with motion trajectories. arXiv preprint arXhv:2412.02700, 2024.
[25] Genmo. Mochi I: A new SOTA in open-source video generation models, 2023.
[26] Yuwei Guo, Ceyuan Yang, Anyi Rao, Mancesh Agrawala, Dahua Lin, and Bo Dai. SparseCtrl: Adding sparse controls to text-to-video diffusion models. In ECCV, pages 330-348, 2024.
[27] Yuwei Guo, Ceyuan Yang. Anyi Rao, Zhengyang Liang, Yaohui Wang. Yu Qiao, Mancesh Agrawala, Dahua Lin, and Bo Dai. DiimateDiff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04775. 2023.
[28] Yoay HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. LTX-Video: Realtime video latent diffusion. arXiv preprint arXiv:2501.00103 2024.
[29] Adam W. Harley. Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In ECCV, pages 59-75, 2022.
[30] Yingging He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high- high-fidelity long video generation. arXiv preprint arXiv:2211.13221 2022.
[31] Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. NeurlPS, 36:8266 8279, 2023.
[32] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurlPS, 33:6840- 6851, 2020.
[33] Sunghwan Hong. Seokju Cho, Seungryong Kim, and Stephen Lin. Integrative feature and cost aggregation with transformers for dense correspondence. arXiv preprint arX1v:2209.08742, 2022.
[34] Sunghwan Hong. Scokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4D convolutional swin transformer for few-shot segmentation. In ECCV, pages 108-126 Springer, 2022.
[35] Sunghwan Hong, Jacwoo Jung, Heeseong Shin, Jisolong Yang, Scungryong Kim, and Chong Luo. Unifying correspondence pose and nerf for generalized pose-free novel view synthesis. In CVPR, pages 2024.
[36] Sunghwan Hong and Seungryong Kim. Deep matching prior: Test-time optimization for dense correspon- dence. ICCV, pages 9907-9917, 2021.
[37] Sunghwan Hong, Jisu Nam, Seokju Cho. Susung Hong, Sangryul Jeon, Dongbo Min, and Seungryong Kim. Neural matching fields: Implicit representation matching fields for visual correspondence. NeurIPS, 35:13512-13526, 2022.
[38] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang. Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. VBench: Comprehensive benchmark suite for video generative models. In CVPR, pages 21807-21818, 2024.
[39] Junha Hyung, Kinam Kim, Susung Hong. Min-Jung Kim, and Jaegul Choo. Spatiotemporal skip guidance for enhanced video diffuuion sampling. In Proceedings of the Computer Vision and Puttern Recognition Conference, pages 11006 10006-11015,1 11015, 2025.
13

---

[40] Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as contrastive random walk. NeurIPS, 33:19545-19560, 2020.
[41] Hyeonho Jeong. Chun- Paul Huang. Jong Chul Ye, Niloy Mitra, and Duygu Ceylan. Track4Gen: Teach- ing video diffusion models to track points improves video generation. arXiv preprint arXiv:2412.06016. 2024.
[42] Siyoon Jin, Jisu Nam, Jiyoung Kim, Dahyun Chung, Ycong-Seok Kim, Joonhyung Park, Heonjeong Chu, and Seungryong Kim. Appearance matching adapter for exemplar based semantic image synthesis. arXiv preprint 21X.v:2412.03150, 2024.
[43] Nikita Karaev, Turii Makarov, Jianyuan Wang. Natalia Neverova, Andrea Vedaldi, and Christian Rup- precht, CoTracker3: Simpler and better point tracking by pscudo-labelling real videos. arXiv preprint arXiv:2410. 18831, 2024.
[44] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht, CoTracker: It is better to track together. In ECCV, pages 18-35, 2024.
[45] Junjie Ke, Qifei Wang. Yilin Wang, Peyman Milanfar, and Feng Yang. MUSIQ: Multi-scale image quality transformer. In ICCV, pages 5148-5157, 2021.
[46] Inès Hyconsu Kim, Scokju Cho, Jiahui Huang, Jung Yi, Joon-Joung Young Lee, and Seungryong Kim. Exploring -emporally-aware features for point tracking. arXiv preprint arXiv:2501.12218. 2025.
[47] Seyeon Kim, Siyoon Jin, Jihye Park, Kihong Kim, Jiyoung Kim, Jisu Nam, and Seungryong Kim. MoDiTalker: -itioo-disentangledd diffusion model for high-fidelity talking head generation. In AAAI, pages 13302-3110 2025.
[48] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan Lo, et al. Segment anything. In ICCV, pages 4015-4026. 2023.
[49] Weijie Kong, Qi Tian, Zijian Zhang. Rox Min, Zuozhuo Dai, Jin Zhou. Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, al. Hunyuan A systematic framework for large video generative models. arXiv preprint arXiv:2412.0388 2024.
[50] Kuaishou. Kling: Video generation by kuaishou, 2024. Accessed: 2025-03.
[51] Muyang Li, Ji Lin, Chenlin Meng, Stefano Ermon, Song Han, and Jun- Yan Zhu. Efficient spatially sparse inference for conditional gans and diffusion models. NeurlPS, 35:28858 28873, 2022.
[52] Rui Li and Dong Liu. Spatial- tentialthen-temporal self-supervised learning for video correspondence. In CVPR, 2023.
pages
[53] Fangfu Liu, Wenqiang Sun, Hanyang Wang. Yikai Wang. Haowen Sun, Junliang Ye, Jun Zhang, and Yueqi Duan. ReconX: Reconstruct any scene from sparse views with video diffusion model. arXiv preprint arXiv:2408. 16767, 2024.
[54] Yixin Liu, Kai Zhang, Yuan Li, Zhiling Yan, Chujic Gao, Ruoxi Chen, Zhengqing Yuan, Yue Huang, Hanchi Sun, Jianfeng Gao, et al. Sora: review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.1 17177, 2024.
[55] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell. Diffusion hyperfea- tures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 36:47500-47510 475 2023.
[56] Benyuan Meng. Qianqian Xu, Zitai Wang, Xiaochun Cao, and Qingming Huang. Not all diffusion model activations have been evaluated as discriminative features. NeurIPS, 37:55141-5511 2025
[57] Jisu Nam, Heesu Kim, DongJae Lee, Siyoon Jin, Seungryong Kim, and Seunggyu Chang. DreamMatcher: appearance matching self-attention for semanticaly--coniiteen text-to-image personalization. In CVPR, pages 8100 81 80,0, 2024.
[58] Jisu Nam, Gyuseong Lee, Sunwoo Kim, Hyeonsu Kim, Hyoungwon Cho. Seyeon Kim, and Seungryong Kim. Diffusion model for dense matching, arXiv preprint 105. 2023.
[59] Jisu Nam. Soowon Son. Zhan Xu, Jing Shi, Difan Liu, Feng Liu, Aashish Misraa, Seungryong Kim, and Yang Zhou Visual Persona: Foundation model for full-body human customization. arXiv preprint arXiv:2503.15406 2025.
14

---

[60] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2504.07198 2023.
[61] William Peebles and Saining Xie. Scalable diffusion models with transformers, ICCV, pages 4195-4205, 2023.
[62] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arxiv:2307.01952 2023.
[63] Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yat Ma, Ching Yao Chuang, et al. Movie Gen: A cast of media foundation models, 2025.
Yao
[64] Jordi Pont-Tuset, Federico Perazzi. Sergi Caelles, Pablo Arbeláez. Alex Sorkine- Hornung, and Luc Van Gool. The 2017 DAVIS challenge video object segmentation. arXiv preprint 1rXiv:1704.00675 2017.
[65] Rui Qian, Shuangrui Ding, Xian Liu, and Dahua Lin. Semantics meets temporal correspondence: Self- 2023.
supervised object-centric learning in videos. In ICCV, pages
[66] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, pages 8748-8763, 2021.
[67] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR, 21(140)):166, 2020.
[68] Robin Rombach, Andreas Blattmann. Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684-10695, 2022.
[69] Runway. Introducing Gen-3 Alpha, 2024. Accessed: 2025-03.
[70] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-A-Videss Text-to-video generation without text-video data, arXiv preprint arXiv:2209. 14792, 2022.
[71] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.0240 020502, 2020.
[72] Nick Stracke, Stefan Andreas Baumann, Kolja Bauer, Frank Fundel, and Bjöm Ommer. Cleandift: Diffusion features without noise. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 117-127, 2025.
[73] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:1 127063, 2024.
[74] Wenqiang Sun, Shuo Chen, Fangfu Liu, Zilong Chen, Yueqi Duan, Jun Zhang, and Yikai Wang. Dimen- sionX: Create any 3D and 4D scenes from single image with controllable video diffusion. arXiv preprint axX9v:2411.04928, 2024.
[75] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. NeurlPS, 36:1 1363-1389, 2023.
[76] Zachary Teed and Jia Deng. RAFT: Recurrent all- pairs field transforms for optical flow. In ECCV, pages 402 419, 2020.
[77] Prune Truong, Martin Danelljan, and Radu Timofte. GLU-Net: Global-local universal network for dense flow and correspondences. In CVPR, pages 2258-6268, 2020.
[78] Prune Truong, Martin Danclljan, Lue Van Gool, and Radu Timofte. Learning accurate dense correspon- dences and when to trust them. In CVPR, pages 57 2021.
[79] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Iliia Polosukhin. Attention all you need. NeuriPS, 30, 2017.
15

---

[80] Rundi Wu, Ruiqi Gao. Ben Poole, Alex Trevithick, Changxi Zheng, Jomathan T. Barron, and Aleksander Holyyski. CAT4D: Create anything in 4D with multi-view video diffusion models. arXiv preprint urXiv:441118613. 2024.
[81] Zeqi Xiao, Yifan Zhou, Shuai Yang. and Xingang Pan. Video diffusion models are training-free free motion interpreter and controller. arXiv preprint 2tti:2405..188644 2024.
[82] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Gongye Liu, Xintao Wang. Ying Shan, and Tsin Wong. DynamiCratter: Animating open-domain images with video diffusion priors. In ECCV, pages 199-417, 2024.
[83] Jiarui Xu Xiaolong Wang. Rethinking supe-supe correspondence learning: A video frame-level similarity perspective. In ICCV, pages 10075-10085, 2021.
[84] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang. Guanyu Feng, et al. Cog VideoX: Text-to-video diffusion models with an expert transformer. arXiv preprint ar08v:2408. 06072, 2024.
[85] Junyi Zhang, Charles Herrmann, Junhwa Hur, Eric Chen, Varun Jampani, Deqing Sun, and Ming-Hsuun Yang. Telling left from right: Identifying geometry aware semantic correspondence. 2023.
[86] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. NeurlPS, (6:45533-45544 2023.
[87] Qihang Zhang, Shuangfei Zhai, Miguel Angel Bautista, Kevin Miao, Alexander Toshev, Joshua Susskind, and Jiatao Gu. World-consistent video diffusion with explicit 3D modeling. arXiv preprint arXiv:2412.01821. 2024.
[88] Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-Sora: Democratizing efficient video production for all. arviv preprint arXiv:2412. 20404, 2024.
16

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction (Sec. D clearly present the main claims, contri- butions, and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
17

---

Justification:
The appendix includes the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--pecfiiation. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: All theoretical results and assumptions are clearly stated in the main paper, with detailed proofs presented in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides the necessary details, including architectural details, experimental setups and evaluation protocols, in the main paper and the appendix. We will make our code and evaluation dataset publicly available.
18

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel urchitecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We will release the code to reproduce the main experimental results in public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
hhipss.c//ublic/uuddds wwtpee/e/ooooSeueoooooeeioe
www.:bimiiiiiooooooo.iooo for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
19

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The main paper Sec. 4.1 and appendix provide detailed experimental setups, including hyperparameters, dataset curation, and method implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: To ensure statistical reliability, fixed the random seed for all experiments, including all comparison methods, to maintain consistency and minimize random variability.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include the computer resources in the Appendix.
Guidelines:
20

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: pecttps:/nnurrppee / http://cccc..tiiicciiiieeeeeeeee
Answer: [Yes]
Justification:
The research strictly adheres to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive and negative societal impacts of the works in the appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications,
authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
21

---

Answer: [NA]
Justification: The proposed method does not involve data or models with high risk for misuse this is analysis paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets used in this paper, including datasets and models, properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the bataasess.paprsstttt paperswi theode. com/datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release our code and evaluation dataset in public.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
22

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes)
Justification: The paper includes full instructions and relevant screenshots for human evaluation in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
The study did not receive IRB approval it was conducted in setting where such approval not mandatory. However, potential risks were disclosed to participants before the study began.
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The methods and results presented in this paper do not involve the use of large language models (LLMs) as essential components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: //ppp/r////rrpp Cettpp///neerrp...ccooooeeeeeree ccc/Conferennce//225511/ for what should or should not be described.
23

---

Appendix
A Dataset Curation Details
In this section, we provide further details on the curation of our evaluation dataset. In Fig. A.13 and Fig. A.14 we present 50 prompts per dataset for analysis on CogVideoX-2B. Additionally, provide more dataset examples in Fig. A.15
Curating Prompt-Vided Pairs. To analyze motion trajectories in depth, require high-quality prompt-video pairs to ensure motion consistency and video fidelity, as suboptimal videos could introduce noise into our analysis. To achieve this, we begin with collecting GPT-enhanced prompts from VBench BS a benchmark designed for evaluating video generative models. For the object dataset, we gather prompts from animal categories, while for the scene dataset, collect prompts from architecture and lifestyle categories. We further augment both datasets by generating additional prompts using GPT-4o M.. with VBench prompts as references, ultimately collecting 300 prompts for both the object and scene datasets. We then synthesize videos from each prompt using video generative models under analysis (e.g. CogVideoX
884 To simulate the generation process.
we curate distinct evaluation dataset for each analyzed model (CogVideoX:
(84). Cog VideoX-
5B (84 Hunyuan Video 49]. and Cog VideoX-d2B-BBB-.
(84) For Cog VideoX-2B [84] each video
has a resolution of 480 x 720 and consists of 49 frames. Human annotators carefully select the final
prompt-generant video pairs based on motion consistency and overall video fidelity, resulting the
top-50 pairs for each dataset.
Generating Pseudo Ground-Truth. To evaluate temporal matching, our dataset includes pseudo ground-truth trajectories, as no ground-truth trajectories exist for synthesized videos. We generate these using the off-the-shelf point tracking method CoTracker 44 Specifically, we first define the starting points of the trajectories in the first frame. For the object dataset, which focuses on object dynamics, we segment the centered object using SAM (48) and sample grid points 1/20 of the video's ' spatial resolution. For the scene dataset, where we aim to capture overall camera movement, we uniformly sample 10 10 grid across the entire frame. Using CoTracker, we obtain per-point trajectories along with their visibility. For our analysis, we consider only points that CoTracker estimates visible.
A.1

---

B DiffTrack on Other Video DiTs
DiffTrack compatible with any off-the-shelf video DiT architecture. We extend our analysis
to additional DiT-based video models, including CogVideoX-5B 84) Hunyuan Video - 184.
[49 and
DiffTrack on Vog In Fig. A. we present an analysis of Cog Vog using Diff- Track. The model consists of 42 transformer layers, and
50 sampling timesteps for this study. Our observations reveal that 1) query-key matching achieves higher matching accuracy compared to intermediate feature matching (Fig. (A..(A. 2) small number of layers dominate temporal matching (Fig. b)), and 3) temporal matching improves as noise levels decrease, with a slight drop at the final stages of denoising (Fig. A.I (c).))). These findings are consistent with the analysis presented in
Sec. 3.4 for CogVideoX-2IB
DiffTrack Hunyuan Video. Fig. A.2
presents additional analysis of Hunyuan Video using Difk The model consists 60 layers, and we use 30 sampling timesteps for this experiment. We observe that: query-key matching outperforms intermediate feature matching (Fig. A.2 a)), few specific layers play dominant roles temporal correspondence (Fig. (b), and temporal matching improves diffusion noise decreases (Fig. A.2
DiffTrack on CogVideoX-5B--22. In Fig. A.3 we present further analysis of Cog VideoX-
5B-12V 84). We observe that query-key key matching outperforms intermediate feature matching (Fig. A.3 a)), small number of layers dominate temporal correspondence (Fig. A.3( and temporal matching improves over diffusion timesteps (Fig. A.3[c).
In Tab. Tab. A.I we further evaluate Cog VideoX-SB- 12V on point tracking accuracy using the DAVIS dataset, and observe that outperforms SVD, U-Net-based model. However, CogVideor VoggoXXo-SBBB 12V underperforms compared its T2V coun- terpart, Cog VideoX-5B. We attribute this to the image-to-video fine-tuning objective, which pri- marily focuses on preserving the appearance of the first frame. This tends to produce more static videos and weakens the temporal correspondence required for generating dynamic motion, as also discussed in 17
Backhone
<
<
< 4
K
35,9
DifTrack (CogVideoX-5B-
84]
3.9
38.2
69.0
36.8
DifTTrck (CogVidenXX 55
(44)
5.2
30.7
84.3
46.9
Table A.1: Quantitative comparison on the DAVIS dataset [19].
A.2

---

Intermediate Feature Matching Query-Key Matching
Accc Maahinn
1.0
6
0.8
Object Dataset
40
0.6
30
Meeun Timestep (1)
0.1
20
0.2
10
0.4
1- 17
(1) dassaee
29
0.2
30
Layer (I)
21 28 45-22
30
Layer (I)
21 28 42
Layer (I)
24 28 75 42
0.0
50
40
30
20
(
Timestep (f)
Ace. Meaching
Harmonig
89 38
8
1.0
Salimnann ho0 Timestep (
0.8
0.8
Scene Datase!
0.6
20
Mean Timestep
0.4
0.6
0.2
2 20
0.4
10
1=
17
0.2
50
40
1421 28 35
40
24 28 42
14 28
Layer
Layer (I)
9
Layer (I)
50
40
Timestep (i)
30
20
10
(a) Representation Selection
(b) Layer-wwer wise Analysis
(c) Noise-level Analysis
Figure A.I: Analysis of temporal matching Cog VideoXX5SB [84]. (a) Representation selection: Query-key
matching achieves higher accuracy than intermediate feature matching. (b) Layer-wise analysis: Temporal correspondence is primarily governed by limited set of layers. (c) Noise-level analysis: Temporal matching
improves as noise decreases but slightly degrades near the final steps.
Internediate Feature Matching Query-Koy Key Matching
Meann Hatonui
Matching
1.0
& 9 10
0.8
Objeet Dalzes
0.6
AcC. A
30
0.4
20
0.2
0.4
16
Timestep (7)
10
Timestop (7)
Timesn (7)
10
0.2
[ 33
20
30 40 50
To 30 50 60
30
to 30 40 60
34
to 20
Layer
Layer ())
30
Layer (I)
0.0
30
25
20
15
10
5
Timestep (t)
Matching Acc.
Matching Agg Timestop
1.0
1.0
0.8
tirmmmie
0.88
Sours Dataset
Mean Timestog (1)
0.2
16
Timestop
19
10
0.2
33
20
20
34
0 10 20 30 40 50
lo 20 30 50
30
Layer (I)
30 50 60
0.0
30
20
15 10
Layer
Layer
Timestep
(a) Representation Selection
(b) Layer- A nalysis
(c) Noise-level Analysis
Figure A.2: Analysis of temporal matching Hunyuan Video [49]. (a) Representation selection: Query-key
matching achieves higher accuracy than intermediate feature matching. (b) Liser-wise analysis: Temporal
correspondence is primarily governed by a limited set of layers. (c) Noise-level analysis: Temporal matching improves as noise decreases.
A.3

---

Intermediate Feature Matching
Query-Key Matching
boy Madding
%0
Matching Aec.
8
0.8
0.8
Object Dataset
8
38 8
0.6
Lu3 Timestep
0.2
0.4
Timestep (()
08989 50
22
1=
15
35
14.2 35
14 35
0.0
50 (
Layer
Layer (I)
Layer (I)
Timestep (0)
Aocr Matthnng
Hamm Comoniia
1.0
8
1.0
35
08
0.88
88
066
Scene Dataset
36
0.4
0.6
(- 11
20
10
0.2
/= 15
30
40
50
Layer (I)
14 28 35
10
50 0
Layer ())
35
40
Layer (I)
14 21 35
0.0
Timestep (1)
30 20
(a) Representation Selection
(b) Layer-wise Analysis
(c) Noise-level Analysis
Figure A.3: Analysis of temporal matching in VigVideo X-5VV-X 84 (a) Representation selection: Query-
key matching achieves higher accuracy than intermediate feature matching. (b) Layer-wise analysis: Temporal correspondence primarily governed by a limited set layers. (c) Noise-level analysis: Temporal matching improves during denoising steps.
A.4

---

C Zero-Shot Point Tracking Details
Frame
Frame
Frame
Frame 4
Frame 23 Frame 24 Frame
(c)
(b)
F Frame
Frame 2
F Frame 24
Frame
Frame
Frame 25
VAF
VAE
VAF
VAF
VAE
VAF
(a)
Vidco Diffusion Transformer
Vidco Diffusion Transformer
2
R
R
Q
R
0
R
C/
2
pl
Pat
Pi.
Pij
Pil
P7.
24
PCl
Figure A.4: Overall architecture of Traf for zero-shot point tracking. (a) Removing temporal compres- sion improves point accuracy by avoiding interpolation. (b) A global first-frame inserted into each chunk for direct matching. (c) Interleaved chunk construction reduces the temporal gap the first frame.
C.1 Architectural Details
Figure A.4 illustrates the overall architecture Diff'rrack for zero-shot point tracking on real videos. In this example, 25-frame video is divided into two chunks for visualization. We further provide component analysis of this architecture in Sec. F.T
We find that temporal compression in the 3D VAE limits point accuracy due to the linear interpolation used to reconstruct motion trajectories from latent space. To mitigate this, we set the temporal compression ratio to g 1, establishing a one-to-one mapping between each video frame and its latent (Fig. a)). This is achieved by passing each frame individually to the pre-trained VAE, which compresses + 4ff frames to 1 + latents, where we set f O.
To enable direct temporal matching with the global first frame, we insert into every chunk (Fig. b)). To reduce the temporal gap between the global first frame and other frames within each chunk, construct chunks using interleaved subsequent frames (Fig. A.4 (c). Frames within each chunk are sampled at uniform intervals, determined by dividing the total video length by f - 1. Chunks slide with a stride of 1, and the matching costs for overlapping frames across chunks are averaged.
In full 3D attention, each frame latent is projected into Q and K where denotes the frame index. With slight abuse of notation, we represent the projections the inserted global first-frame latent in the second chunk as Q and K!'- For each chunk, we compute the matching cost between the first-frame query and the key of the j-th frame (cf. Eq. (). To enhance performance, following 3 13 14 37]. we employ bidirectional matching cost that additionally incorporates the transposed inverse cost between the query of the j-th frame and the key the first frame. This
1. for the first chunk and àl' for the second chunk. Next, apply argmax operation results in
A.5

---

(ef. Eq. (6) to obtain the matched correspondence points: the starting point Pt, and its match př,, in the j-th frame. By concatenating p across frames, we obtain the full motion trajectory.
C.2 Implementation Details
Zero-shot Point Tracking. For zero-shot tracking evaluation [5] we used the most significant
layer and timestep, identified by matching accuracy in Sec. 3.4 and Sec. B 1 = 17, t = for Cog VideoX-2B, 16, t 1 for for Cog VideoX-5B, and experiments were conducted on A6000 GPU.
6, t 1 for Hunyuan Video. All
We evaluate zero-shot tracking two real- video datasets with precisely annotated tracks: TAP- Vid- H-P-Vid-Kinetic
DAVIS 19 and TiP-Vid-BB Kinetics [19]. DAVIS includes 30 videos with diverse object motions and appearance variations, while Kinetics comprises 1,189 in-the-wild videos featuring rapid scene transitions and motion blur.
Following prior work [6 43 44 we evaluate point accuracy at 256 x 256 resolution. For Cog VideoX- 2B, we resize videos to 256 256 and then upsample to the training resolution of 480 x 720. The resulting feature descriptors have spatial resolution of 30 x 45, as the 3D VAE decompresses spatial size to 1/16.
Human Evaluation.
We also provide an example of human evaluation in Fig. A.16
C.3
Comparison
We demonstrate the effectiveness of Drak'Track in zero-shot point tracking [5] by comparing it with a diverse set of vision foundation models trained on single images and self-supervised models trained on videos or tio-view images. Below, we detail the models included in our comparison. For fair evaluation, following (5), we resize inputs to produce feature maps of size 30 45, except for ZeroCo [3]
Vision Foundation Models. We evaluate DiffTrack against vision foundation models, including DINO, DINOv2, DINOv2-Reg, and DIFT (SD1.5, SD2.1).
DINO 10 is self-supervised vision transformer that learns localized features of salient objects.
DINOv2 60 improves upon DINO by leveraging larger dataset and optimized training. DINOv2- Reg 18 introduces register tokens to further reduce attention artifacts. We use ViT-B/16 for DINO
and ViT-B/14 for both DINOv2 and DINOv2-Reg.
Stable Diffusion 1.5 (SD1.5) and 2.1 (SD2.1) [8] are U-Net-based text-to-image diffusson mod- els. Following DIFT 75 we extract features from the third upsampling block to compute point correspondence.
Self-Supervised Models. We further evaluate DiffTrack using self-supervised video models, in- cluding SMTC, CRW, Spa-then-Temp. VFS, and SVD, as well as a self-supervised model trained on two-view images, ZeroCo. All of these models are trained solely on videos or two-view images without any labels.
SMTC [65] proposes self-supervised video model that improves semantic and temporal consistency by training the architecture in teacher-student manner. We use SMTC with ViT-S/16 for comparison.
Contrastive Random Walk (CRW) 40 trains 18 to learn temporally consistent feature representations through cycle-consistency, maximizing the likelihood of returning to the initial points when walking through palindromic video sequences.
Spa-then-Temp [52] combines spatial and temporal self-supervised learning by first leveraging contrastive learning for spatial features and then enhancing these features through hierarchical frame reconstruction and local correlation distillation. We Spa-then- based on ResNet-50 for
comparison.
Video Frame-level Similarity (VFS) 3 compares frame-level features from the same video as positive pairs, while frames from different videos serve as negative pairs. We use VES with ResNet- 50 for comparison.
Stable Video Diffusion (SVD) 6 is U-Net-hased text-to-video generative model extended from SD 2.1 by incorporating additional temporal layers. Following 41], we use features from the upsampler
A.6

---

layer of the third decoder block to calculate point accuracy. We empirically observe that query-key matching achieves higher point accuracy than feature matching; therefore, we adopt query-key matching in the third decoder block of SVD.
ZeroCo K demonstrates query-key matching in the cross-attention map within the self-supervised cross-view completion model CroCo [3, capturing geometric correspondence more effectively than other correlation maps from the encoder decoder. For comparison, we use an input size of 224 x 224 with a feature size of 14 x 14, as we empirically found that this trained resolution yields the best point accuracy compared other input resolutions with higher feature sizes.
A.7

---

D Cross-Attention Guidance Details
Architectural Details. In Fig. we present the overall architecture Cross-Attention Guidance (CAG). Inspired by PAG 22). which enhances image fidelity by transforming selected self-attention maps in the diffusion U-Net into identity matrices, we extend this idea the video DiT architecture.
In PAG, the identity matrices are created by multiplying a diagonal mask into the attention map before the softmax operation-wheree diagonal elements set to 0 and off-diagonal elements -oo. After softmax, this yields an identity matrix (diagonal values of 1, others 0), allowing values to pass through unchanged.
naive extension to video assigns -oo to cross-frame positions and elsewhere before the softmax operation. However, this undesirably suppresses the scale of self-frame and text-franee attention values. To address this, we instead zero out only the cross- frame attention values after softmax in Ar, producing modified attention maps Á that preserve all other interactions.
Implementation Details. For CAG, we used the top-3 dominant layers = 13, 17,21 21 for Cog Vid and I 15, 17, for Cog CogVideoX-5B, as identified by harmonic mean in Sec. 3.4 and Fig. A.1 Following [2], applied the guidance at all sampling timesteps.
Evaluation Details. We evaluate CAG against its baselines, CogV VideoX2222 and CogVideoX-SB, on VBench (38). We used the prompt suite provided by VBench for each evaluation dimension.
To assess temporal quality, report three metrics: Subject Consistency, Background Consistency, and Dynamic Degree. Subject Consistency measures whether the appearance of the main subject (e.g. person object) remains consistent across frames, computed using DINO feature similarity. Background Consistency assesses the temporal coherence background scenes, measured by CLIP feature similarity across frames. While completely static video can achieve high scores the aforementioned temporal quality metrics, it is essential to also evaluate the presence and magnitude of motion. To this end, calculate Dynamic Degree, which quantifies motion dynamics using optical flow computed by RAFT 76).
To evaluate frame-wise quality, report Imaging Quality, which detects frame- wise distortions (e.g., blur, noise), computed using the MUSIQ [45] image quality prediction model trained on the SPAQ 22 dataset.
A.8

---

E
Additional Analysis
In this section, we present additional analysis Cog VideoX-22 84 To calculate point accuracy (PCK), predefined error threshold 8 for all analyses presented in the main paper and appendix.
using our framework, DiffTrack.
Intermediate Feature Matching Query-Key Matching
Saaae Matchinks Acc
Matching Acc.
Harmonic Mean Timestep (
1.0
0.4
0.6
0.2
0.4
1-
Timestop (0)
Timestop
10
199 1996
20
0.2
Layer
20
30
30
40
Layer
20
30
S 50
Layer
20
30
0.0
30
40
30
1 21
Timestep (t)
(a) Representation Selection
(b) Lay er wise nalysis
(c) Noise-level A nalysis
Figure A.5: Analysis of temporal matching in Cog V2d (84) with DAVIS [9] dataset. (a) Representa-
tion selection: Query-key matching achieves higher accuracy than intermediate feature matching. (b) Layer-wise analysis: Temporal correspondence is primarily governed by limited set layers. (c) Noise-level analysis:
Temporal matching improves throughout the denoising process and slightly degrades near the end.
Analysis with Real Videos. In Fig. A.5 we analyze Cog CogVideoX-2B using real videos from the DAVIS dataset [19.. As discussed in Sec. Sec.B.4) reconstructing real videos often introduces inversion errors [71] due to challenges in finding accurate text prompts or discrepancies between real videos and the training distribution.
To address this, instead of applying diffusion inversion, we add Gaussian noise to the frame latents at each timestep Specifically, we truncate DAVIS videos to frames, excluding those shorter than 49 frames. At each timestep f, we add Gaussian noise corresponding to f to the frame latent, pass it through the diffusion transformer, and extract feature descriptors for analysis. Notably, this approach is consistent with prior work [75 86), which identifies two-frame correspondences in U-Net-based image diffusion models.
The analysis with DAVIS in Fig. A.5 shows similar trend to that observed in Fig. using curated synthetic dataset. Specifically, que matching consistently outperforms intermediate feature matching, Additionally, a few layers predominantly contribute to temporal matching, and these dominant layers align with those identified Fig. 4 Temporal matching strengthens during the denoising process, with slight drops near the end of the timesteps.
Frame
Frame 8
Frame
Frame
Frame 32
Frame 40
Frame
Query-Key Key Feature
Figure A.6: Attention visualization comparison between intermediate feature matching query-key matching.
Feature Matching Query-Key Matching. Fig. compares attention maps across timesteps, derived from intermediate feature matching and query-key matching. We observe that query-key matching successfully tracks physically matched points, whereas intermediate feature matching often fails. This further supports the observation in Fig. a4 that query-key matching yields better temporal correspondence.
This finding is consistent with prior works 3 57]. which demonstrate that query-key similarities include geometric cues crucial for accurate matching, whereas value warps the visual appearance based on these similarities.
A.9

---

Frame 0
Frame 24
Frume 48
PCA Visualization. Fig. A.7 visualizes the Principal Component Analysis (PCA) of queries, keys, and values across video frames. Queries and keys exhibit stronger structural cues, with similar distributions for nearby pixels, enabling more effective geometric matching, In contrast, values contain potentially diluting structural cues for correspondence.
Query
noisy appearance features,
This analysis aligns with 3 57 which show that query- key similarities capture structural information crucial for accurate geometric matching, while values encode semantic appearance information, further warped by the query-key similarities.
Key
Value
Attention Visualization. Fig. A.17 presents cross-frame attention maps between the first frame and subsequent frames. In the top-ranked layers based on harmonic mean scores (Fig. a.)7 the attention is sharp and accurately localized at the matched point. In contrast, the bottom three layers with the lowest harmonic scores (Fig. (b)7 exhibit diffuse and scattered attention patterns. This supports our analysis in Fig. 4(b),, which shows that only few layers contribute significantly to temporal matching.
Figure A.7: PCA visualization of queries, keys and values.
Fig. A.18 illustrates how cross frame attention evolves throughout the denoising process, revealing that attention becomes progressively sharper at later timesteps. This observation supports our analysis in Fig. 4 c), which shows that temporal matching improves denoising progresses.
Fig. 19 displays the text-to-frame atten-
Quexy- Matching
Key-Key Matching
Quey-Key Malching
tion maps for the query word "shark" across timesteps. As discussed in Fig. []] the atten- tion evolves from coarse to fine as noise lev- els decrease, highlighting that text primarily de- termines the global semantic layout during the early stages of denoising.
Maccing Adc.
A 1 SS9a Timeste (1)
Mitcing
soy Timest (1)
Trmesen (
Query-Query vs. Key-Key vs. Query-Key Matching. Fig. [A.8] shows the matching ac- curacy of query-query, key-key, and query-key interactions on the object dataset. Query-key matching achieves the highest accuracy, indicating that query-key interactions within full 3D attention inherently learn temporal matching.
Layer
Layer
10
50
Layer
Figure A.8: Comparison of query-query, key-key, and
query-key matching.
A.10

---

F Ablation Study
F.1 Zero-Shot Point Tracking
Component Analysis. Tab, A.2 ab- lates the effects of temporal com- pression and long-term sequence han- dling in zero-shot point tracking per- formance DAVIS. We evaluate four configurations: (I) the baseline, which employs temporal compression in the 3D VAE and sequential chunking: (II) the baseline without temporal compression (q 1), ensuring a direct one-to-one mapping between frames and frame latents; (III) an approach that inserts the first frame into every chunk, enabling explicit interaction with the first frame in all chunks: and (IV) our final method, which further applies interleaved frame construction to better handle long-term context. Compared to (I), (II) improves per- formance by eliminating interpolation errors introduced by temporal compression. Compared (II,, (III) shows significant improvement, demonstrating the importance of direct temporal interaction with the global first frame within each chunk. Additionally, (IV) further enhances performance by interleaving frames within each chunk, effectively reducing the frame interval between the first and other frames.
Component
<
<
8
<
<
<
(
Baseline
2.6
21.9
60.2
26.9
(I))
( + wlo temporal compression
2.7
31.6
64.5
32.3
(III)
(II) + w/ first- uu-ffrmm insertion
4.7
47.6
70.5
44.6
(IV)
(III interleaved frames
84.3
46.3
Table A.2: Ablation study: analyzing the impact of temporal
compression and long-term handling.
Impact of Feature Selection. Tab. A.3 highlights the effectiveness of feature selection by comparing matching
Layer (l)
Timestep (t)
< 83
accuracy the object dataset using three strategies: (I) averaging matching costs over all timesteps at the most
(I)
I == 17
66 [1.50]
15.34
(II)
IE [0,29]
t=
16.69
dominant layer (1 17), (II) averaging over all layers at the most dominant timestep based on matching accuracy =I,,,, and (III) using matching costs from the most dominant layer and timestep (I = 17, = 1). Selecting optimal features significantly improves accuracy, emphasizing the value of analysis and the importance of feature selection for reliable temporal matching.
(III) 1= 17
t=1
63.50
Table A.3: Ablation study, analyzing the impact of feature selection.
Multi-Feature Fusion for Zero-Shot Point Tracking. Prior works [1 42 56
Layer ())
Timestep (f)
<
<
5
84
5
86) on intermediate diffusion features suggest that fusing multiple timesteps and layers improves semantic correspondence by leveraging the hi- erarchical structure of diffusion representations. Motivated by this, we investigate whether such fusion also benefits temporal matching.
57 75,,
I 17
1=1 1
4,8
49.2
84,3
46.3
13, 17,18
4.9
48.6
84.2
46.0
1= 1,2,3
85.0
46.6
Table A.4: Ablation study, analyzing fusing features across multiple timesteps and layers.
Tab. summarizes results on the DAVIS dataset 64 In contrast to prior findings in seman- tic correspondence, we observe that fusing across timesteps and layers has negligible impact on point accuracy. This discrepancy arises because temporal matching requires precise, pixel-level ge- ometric alignment to track the same physical point across frames, whereas semantic correspondence benefits from multi-scale contextual cucs to match similar regions.
Frame 0 Frame 24 Frame 48
Frame Frame 24 Frame 48
(a) Query Visualization = 17)
(b) Key Visualization 17)
Frame 0 Frame 24 Frame 48
Frame 0 Frame 24 Frame 48
PCA visualizations in Fig. A.9 support this finding. In Fig. A.Ra)) and (b), query-key features from dif- ferent timesteps exhibit high similarity, indicating limited benefit from temporal fusion. In contrast, Fig. A.9 c) and (d) show greater variation across features from different layers, suggesting that the tracking performance is primarily driven by a single dominant layer and that layer fusion introduces noise rather than informative diversity.
10
(c) Query Visualization (t
(d) Key Visualization
Figure A.9: PCA visualizations of query-key fea- tures across timesteps and layers.
A.11

---

Frame
GT
Starting Point
= 4
(a) Reconstructed Frames by VAE Decoder
(b) Attention Map Visualization
Figure A.10: Analysis of temporal compression. (a) Reconstructed frames from the VAE decoder using different temporal compression ratios q. (b) Visualization of attention maps under different temporal compression ratios.
Temporal Compression. In Fig. A.10, we present decoded frames from the 3D VAE decoder and corresponding attention visualizations for different temporal compression ratios q. (a) For q = the reconstructed frame is relatively blurred due to the limited expressivity of the compressed repre- sentation, while (b) The attention visualization for 4 shows more diffuse attention, while q 1 exhibits sharper, more focused attention. This supports our architectural design Fig. A.4 4a), which contributes to improved point aceuracy.
allows each frame to be reconstructed almost perfectly without compression.
Starting Point
# of Frames 2 of Frames
#Frames
<
82
84
<
lng
4.4
43.9
77.1
41.7
4.5
46.3
82.2
44.3
8
4.7
47.9
83.8
45.5
4.8
48.5
46.1
4.8
49.2
84.3
46.3
Figure A.11: Ablation study on the number of frames per chunk.
Figure A.12: Attention map for different number of frames.
A.I. ablates matching accuracy on DAVIS using different numbers of frames per chunk. We observe that as the number of frames increases, matching accuracy consistently improves. This is further supported by Fig. A212, which visualizes attention maps across different chunk sizes, showing that longer chunks lead to sharper attention. This suggests that multi-frane interaction enhances temporal correspondence, highlighting the role of cross-framt frame attention over multiple frames within video DiT', in contrast to image-based models ( 86).
Number of Frames per Chunk. Fig. A.11
F22 Motion-Enhancen Video Generation
Layer Selection. Tab. A.5 compares the least top-3 and most top-3 dominant lay- ers for temporal matching (1 30.31. 30,31,25 32 vs. / = 13, 17, 18) of V5 motion-enhanced video generation. We ob- serve that guiding with the most dominant
Subject Consinency
Background Consssency
Dynmiic Degree
Tanginn Quality
Method
OrgVilenN-GB
0.158
0.9500
0.5667
0.5531
OigVidenN-5B. CAG (1- 90.811.22
0.147
0.9581
0,7159
0.5683
CugVideuX-SE CAG i 17,18.
0J223
0.9644
0.6863
0.6811
Table A.5: Quantitative comparison of CAG applied at
different layers.
layers (1 = 13, 17, 18) consistently improves all evaluation metries compared to the baseline, whereas guiding with the least dominant layers (I = 30, 31, 32) results in only marginal gains Image Quality and even lower performance in Subject Consistency and Background Consistency, under the same guidance scale. These findings further support our analysis that specific layers play a critical role in enhancing temporal consistency during generation.
Memory and Time Consumption. Tab. A.6 compares inference time and memory usage per denoising timestep across three settings: the baseline, baseline with Classifier-Fres Guidance (CFG). and baseline with CFG and Cross-Attention Guidance (CAG). While inference runtime increases due to the increase of the number of function evaluations, CAG's GPU memory footprint remains very similar to that CFG since CAG simply applics attention mask to a few dominant layers.
Method
Memory (GB)
Time (s)
Baseline (Cog VideoXX2)) Baseline CFG
2.75
1.31
12.75
2.60 (198%)
Baseline CFG CAG
12.84
4.59 (177%)
Table A.6: Memory and Time Consumption.
A. 12

---

This stands in contrast to prior works [ which introduce additional modules and require training for motion enhancement. In comparison, our method is entirely training- free and operates within the model itself via novel guidance mechanism.
A.13

---

Backbone
< 8
<
< 6
<
8
DIFT (SDL.5) 75
3.5
13.0
39.3
63.1
72.2
38.2
DIFT (SD2.1)
25
3.6
13.3
40.1
65.8
75.7
39.7
Diflusion Hyperfeatures (SD1.5) 55
3.0
12.9
33.1
58.4
74.5
36.4
Diffusion Hyperfeatures (SD2.1) 55
3.7
15.5
38.8
64.5
78.9
40.3
CleanDIFT (SD1.5) ?P4
3.1
13.7
36.5
64.9
79.6
39.6
CleanDIFT (SD2.1)
3.7
16.0
41.8
68.6
81.9
42.4
TIR (SD1.5 DINOv2 (VTT-B/14)) 85
4.3
17.7
44.6
69.8
82.0s
43.7
Daff'rack (Cog VoogVide
(84
4.8
19.4
49.2
73.6
84.3
46.3
DiffTrack (Cog VideoX -5B
84
5.2
20.5
50.7
73.9
84.3
46,9
Table A.7: Quantitative comparison on the DAVIS dataset [99]
G
Additional Qualitative Results
Additional qualitative results for zero-shot point tracking and motion-cnhanced video generation are provided in Fig. A.20 and Fig. A.2I
H
Additional Quantitative Results
Additional quantitative results for zero-shot point tracking on the DAVIS dataset are provided in
Tab. A.7
I Related Works
70 82 were primarily based on U-Net [32 62 68), often achieved by inflating pre-trained image diffusion models 33 62 68],
Video Diffusion Models. Early approaches 4
typically with separate spatial and temporal attention mechanisms. Although efficient, this separation restricts direct frame-to- to-frame spatial and temporal interactions, leading to temporal inconsistency or lack of large motions in generated videos. Sora [54] has demonstrated the effectiveness of Diffusion Transformers (DiTs) [1 increasing scalability and improving temporal coherence. Subsequent 88 such as Cog VideoX [84) MovieGen 633. Mochil [25] and
works 25 28 50
69
Videv 28 have adopted the DiT architecture, achieving unprecedented performance. Unlike U-Net-based methods, DiTs employ full 3D attention, enabling cross-frame information sharing between frame latents as well as with text embeddings. This explicit cross-frame attention improves temporal coherence 84 Despite these advances, how video DiTs capture temporal correspondence during video generation remains unexplored.
Representation Analysis in Video Diffusion Models. Recent works [4] 81]
explore internal
representations in vidco diffusion models 26 27 for controlled video generation and improved motion consistency but do not analyze temporal correspondence and are based on U-Nets 32
68),
which are known to struggle with large motion. Another work [8] explores attention control in video DiT 84 for subject consistency in long video generation but focuses solely text-to-video attention, overlooking temporal matching between frames.
Exploring Correspondence in Diffusion Models. Numerous studies 31
56
57
59
75
86 have explored intermediate features from pre-trained image diffusion models 32
62
68 for
correspondence [14] 58 77 788. However, their analyses mainly focus on two-frame correspondence, as image diffusion models are not designed for temporal correspondence in video sequences.
Temporal Correspondence. TAP-Vid Vid [19 formulates temporal correspondence in video sequences as point tracking, aiming estimate the motion physical points across frames. PIPs (29) iteratively refine estimated trajectories within temporal windows, while TAPIR 200 incorporates depthwise
convolutions and enhances initialization. CoTracker 44 jointly tracks near-dense trajectories using spatial correlations. These methods often rely on training with synthetic datasets [19]. as annotating
real-world data is highly challenging. To address this, recent study explored zero-shot tracking with visual foundation models [10
18 48
60 66 68) showing promising results. However, these
analyses are limited single-image models such as Stable Diffusion (SD) [68 and DINOv2 (60]
which process single frame and thus lack temporal awareness.
A.14

---

vibrant honeyhee, its wings shimmering in the sunlight, delicately lands blooming lavender flower, tiny legs brushing against the soft A vihrant forest scene unfolds as the camera gracefally moves through the lush canopy, revealing intricate bird nests nestled among the branches... close-up view reveals smail with glistening, spiraled shell, slowly traversing lush, dow-kissed leaf. The
captures the intricate
close-up view reveals brown csterpillar with intricate patterns slong its segrented body, slowly inching acroes a vibrant green leaf. ... A majestic eagle perches on sturdy tree branch, its sharp eyes scanning the vast landscupe below. The hird's poweriul taloes grip the rough mijestic woll' stands in snowy forest, its thick fur blend of grass and whites, glistening under the soft winter sunlight. The camera majestic white fox, with its pristine für glistening under the soft glow of the light perches gnaefully stop nagged, moss- rock meticulously crafted horse figurine stands majestically on a polished wooden surface. glossy finish reflecting ambient light. camera - A close up revcals a vibrant, shimmering fish caught in woven net, its stales glistening with iridescent huss of silver and blue under the expanse golden wild gass, mane flowing the pentli brouze under warm, goldon glow
majestic lion stands rogaily a serene mendow bithed in the golden light of down, a graceful deer with a sleek, tawny cost grazes pescefullly amidst a sea of iajestic hend of elephants roams the savanna, their massive forms silhouetted against the golden hues of setting sum The leader, wise A detailed close-up captures a fly perched on a vibrant green leaf. its iridescent wings shimmerring with hoes of blue and green under the soft.. sun-dappled sovannah, its sleek, spotted A close-up shot captures a kangaroo in its natural habitat, its fur a rich blend of earthy browns and grays. as it gently scratches its
majestic cheetah reclines gracefully
blending seumlessly with the golden grass.
majesiic great blue beron stands gracefully the edge of's tranquil lakesido, its long nock elogantly curved, and its striking blue-gray solitary seugall, with pristine white feathers and a hint gray on wings, gmoefully strolls along the sandy shore, its slender legs An American crocodile hasks on sunlit riverbank, its ruugh, scaly skin glastening under the warm sunlight, showcasing shodes of olive and gray. curious wild rabbit soft. brown fur and twitching whiskers alenty in lush, green meadow, surounnded by vibrant wildflowers tall A majostic clouded leopard, with its distinctive dusky rosetles and clongatcd tail, gracefully perches on sturdy tree branch hrghh above the
An African penguin waddles gracefally across a sunlit beach, distinetive black white plumage contrasting against the golden A satterjack tead, with its distinctive olive-green skin adomedd with warts and striking yellow stripe down it back. rests on a sunlit rock. vibrant animation, majestic whlle emerges, crafted entirely from disposable objects plastic bottles, straw, bigs each piece... A whimsical scene unfolds with intricately enitted paper outouts, each element meticulously detuiled. Two delicate hands, with visible paper lush green lawn, sways precariously
vibrant pink plastic flamingo. perched A curious monkey sits atop a weathered stone, surrounded by hush greenery, its fur mix of carthy browns and grays, blending scamlesaly with the In dimily lil eave, solitary but lungs upside down from the rocky coiling, its wings wrapped smuply around its small, fanyy body.
gusty wind sweeps across the
cousing its slender...
A sleck barbor seal glides gracefully through the crystal-clear waters near the rocky shoreline, its smooth. speckled gray cool shimmering under majeslic great while shark glides gracefillly throuh the crystal-clear ocean waters, its powerful body cuiling elforllessly through the gentle.
majestic gost with thick, shaggy cost impressive curved homs stands proully stop a nagged rock formation, silhouetic framed against A vihrant hutterfly, with iridescent wings displaying kuleidoscope of blues, purples, and orunges, delicately perches on hudding flower in glussy, iridescent beetle slowly emerges from the golden sand, shell glistening under warm sunlight.
grains of sand cascade off A play ful penguin chick, its flully gray feathers rufled by the Autarctic breeze, waddles clumsily across the icy lerrin. Each step sends tiny mujestic beld eagle perches stop jagged cliff, talons goiping the weathered rock as the wind ruffles its pristine feathers, sharp A curious meerkal, standing tall on its hind legs, peuss actoss the sun-scorched sands of the African desert, its slock fur dusted with fine
lowering antlers A curious red fox, its russet fur illuminated by the golden hues down, stands poised on the edge of hoss--kissed mendow. Delicate puffs majestic moose, massive anters adomedd with strands autumn leaves, stands partially submerged in tranquil forest pond. Ripples spread A curious sea turtle, its shell duppled with hass of emerald and bronze, drifts gravcfully through the clear blue waters of a vibrant voral reef.... ... mujestic gray woll, its thick für dusted with regal stag, its antiers crowned with frout-kissed leaves, stands poised stop a bathed the golden hues dawn. curious red fox, its russel cosl vibeaol against the snowy backdrop a winter forest, poers out from behind frosi-coverel pinc tree. Ils griceful swan, its snowy white feathers shimmering the golden light sunset, glides serenely across a glassy lake. Gentle ripples fan out A majestic hison, its massive frame dusted with frost, stands resolutely amid the now-covered plains of the American wilderness. Warm beeath sleck munta ray. wings spanning wide it glides effertieslly through the azure depths the occan, moves A vibrani green iguana, its lextured scales shimmering in shades of emerald and jode, basks slop a sun-warmel rock beneath the canopy of graceful swan surveys surroundings in the serene lake, with shimmering wings reflecting the golden light of down. The A proed sing stands mationless stop a snow-dusted hill, its antlers adorned with delicate strands of frost that glisten in the pale winter graceful white swan glides serenely across the glassy surface a moonlit like. its reflection shimmering with each rigple of the water... A powerful walf, its thick fur a blend of silver and charcoal, stands poised stop a rocky lodge overlooking a mist-chronded forest. Its amber eyes
majestic
with
of golden moss, stands poised the edge sun-drenched meadow. dark eyes gleam
stands slop rugged cliff the pale light of down breaks across the distant....
eyes gleam
fluid grace. Sunlight-
filled with
Figure 131 Evaluation prompts for the object dataset. Our high-quality text prompts are curated from existing benchmarks and generated by a large language model, followed by human annotation to ensure motion consistency and video fidelity in the generated videos.
A.15

---

drane gracefially glides the bauntingly silent, abandoond school building Pripysc, Ukminn, capturing the cerie beauty of decaylnng
A charming house front door, adoened with festive Christmas decorations, stands as the centerpince of cozy winter seene. The docr, painted
In the hearl of uncient temple, sacred sculpture stands mujestically, bathed the soft of Tickering candlelight. The intricate
stiking low-angle view captures the towering facade of modern spartment building, slack glass windows reflecting the vibrant hues
A breathtaking panorama reveals an ancient Asian temple complex, nestled amidst lush green hilks with intricately carved stone pagodas and ornate breathtaking acrial view cuptures the iconic Stultgari TV Tower, standing tall amidst lush, verdant forest, its slock. modem design....
A breathtaking view unfolds as the camera pans upward, capturing the sky framed by towering skysempers The buildings, with their sleek glass
winding, unpaved road stretches through lush, verdant
flanked by sowering trees with vibrant green leaves, casting dappled shadows
sleck drone glides over a secluded house nesiled amidst lush tropical vegutation, eapturing the vibran greens of lowering pulm trves and dense In dense, misty firest, towering trees surround several controlled slash piles, their flames nickering and crackling, casting warn glow
misty, mounlit garden, carved jack-d lantem with a mischicvous grin prominently on rustic wouden tablc suroounedd by aray In traqqiil meadow at down, the sun's golden nys pierce through delicute spider web, intricately woven berween two tall blades of g'grass. winter forest, delicate snowllakes gentyy blanket the intricate branches of towering trees, creating mesmerizing tnpestry of white selitary palm tree stands tall and majestic, its slender tunk reaching skyward crowned with a lush canopy of vibrant green fronds that sway-.. breathtaking actial view revealss vast, snow-govered landscupe, where breathtaking aesial view captures the light down spills over majestic mountain peaks, casting long shadows across the rugged... A breathtaking panorama unfolds, rovealing majesiic mountain range cascading into a tranquil sea, dotted with charming islets. These islets
snow piles treate mesmerizing puttern
the terriin The
the loumling remsins of abandoned house, vibrant green grass and resilient plants werve through cracked flocebuands An acrial view revesls a stunning Crostian hay, where tirquaise waters gendly lop against the nigee lossttine, dotted with lush greasery and From a breatbsaking acrial perspective, the
crumbling walls,
sweeps cory bedroom bathad in soft morning light features a plush, king-sizo had with a tulled headbeard, adomned with crisp white linens and The grand interior of Jewish synagogue unfolds, showcasing intricate architecturl details and a serene armosphere. The space is adomned with...
builling cityscapo, revealng sturning saa of skyserapers piereing the sky.
An expansive assial view reveals the interior of bustling warchouse, In an opulent hallroom, grand chandeliers hang from the ornate calling, their csystal prisms casting a koleidosoope of light across the palished dimly lit, abandoned indoor swimming pool, the vidrant tiles cracked and faded, echo tales of forgorten laughter and splashes Inside the grand, decaying halls of an abandoned mansion, vibrant graffai ant covers the cracked, peeling walls, transforming the space into a trams/ into
rows of towering shelves meticulously organized
A living
lnven, fealuring plush beige sula adomaed with sull, culorful cushions, and rustic wooden colliee cnry bedroom fentures a uriking exposed brick wall, adding mistic charm to the space. In sleck, contemparary home studio, thate-o digital audio workstation siis at the center, survunded by dual high-reaolution menitors... pristine buthroam buthed soft, natural light features sleek, modem design. A winding forest road, flanked by tomering trees with lush green folinge, stretches into the distance under caropy of dappled sunlight. cozy, sunlit kitchen with rustic wooden cabinets and large famahouse sink, where marning light streams through window adomned with lace grand, indoor library
room is softly lit by a vintage floor lamp
centerpiace is freestanding white bathtub with elegant..............e. ...
towering wonden booleshelves filled with countless books, their spines in various colors textures, stretches up to A cozy nursery bathed in soft, natural light feanures pastel- colored walls adorned with whimsical animal murals. A white crib with mobile of Rising the icy expanse Antarctica, breathtaking palace shimmers a crystalline jewel. Walls translucent ice refract the
With its soaring dome and marble facade, a grand Remsissamc calledrrl dominates the skyline of Florenx. Ornatic carvings and liesooed cailings historic square Sunlight floos through lurg windows into vibrani artisl's 's studio, illuminatngg canvases propped against the walls. An casel stands at the Rich mahogany shelves, filled with lesther- hoond books, line the walls of's vimagg libmry crackling fireplace casts flickering light on the minimalist home gym equipped with sleck
the heart a drenched Spanish
alive with vihrant energy. Encireled by whitewashed buildings with..
dumbbells, a statiorry bike, a yogn loid on the polished wooden floor. Large- a bright, cheerful kitchen, the air is filled with the scent of freshly cut flowers and cituss fruits. A vase of sunflowers sits on... Raindrops patter gently against window, creating soothing hackdrop
capy living A vintage study filled with leather leather-bound bound books and antique fumiiture exudes timeless elegance. A mahogany desk. cluttred with handwriteen notes traquil meditation
soft sofh draped with knirted blankets and
buthed in soft natural light offers space for quiet reflection. Flour cushions in earthy a rustic workshop with exposed wooden beams and stone walls, lhe air is filled with the scent freshly cui wood and sawdust.
arranged
Soft lighting and warm textiles create a haven of corfort in cazy bedroom. A bed, dressed in linen sheets and knirted blanket, rests fuluristi: skyseraper, with its twisting glass facade, pierees the skylinc of's bustling metropolis. Its reflective A colossal Art Nouvenn open house communds attention with its Howingg organic forms and ierriatt ironwork Delicate floral motifs swirl neross majestic Renaissunce cathedma stands proudly the heart of ancient European city, its marble focade adorned with intricate carvings and A grand Byzntine buolliaa, with its mussive central dome and gilded mostics, stands as tosiament centuries a'rehiilccturl mastery....
mirurr clouds.... ...
Evaluation prompts for the scene dataset. Our high-quality text prompts are curated from existing benchmarks and generated by a large language model, followed by human annotation ensure motion consistency and video fidelity in the generated videos.
Figure A.14:
A.16

---

Starting Points
Starting Points
Prompt: "A solitary sesgull, with pristne white fenthers and hint wings, grscefully strolls along the sandy slender
Prompt "A drone gracefully glides over the hocutaglly silout shandaned school huikking Pripyat, Ukraine, copturing cene beauty
Prompt: "An African penguin waddles gracefilly neross a sunlit besch, its distinctive black and white plaragge contrasting against the gulden sand.
Prompt "A charming house front doorr adamed with festive Chrisnea
decorations, stands zu
winter
Prompt: "A curious monkey sits stop weathered stone, survonnded greeneyy fur of curthy grays, blending
Prompt: : "In the haunting remains an abundoned house, vibrant green grass and resilient planis through cracked floorboands and crumbling walls,
Prompt "A sleck harber soal glides gmecfully through the crystal-clour waters near the mick shorcline, smooth, speckled cont shimmering..
Prompt "The grand interior Jewish syungogue unfolds, showeasing intricate architectural detail and a screne stmoophere. The adoomed
Prompt majestic its massive antlers aderned with strands of stands purtially submergged tranquil forest pond.
Prompt "In opulent ballroom grand chandeliers hang from ceiling, crystal prisms casting kalsidoscope light
Prompt: "A majestic elk, its towering anilers crowned with strands of golden stands odge sun-drenched meadow, dark eyes
Prompt "Inside the grand, decaying halls of an abanduned munsion, vibrant graftti the emaked, wails, transfonaing space
Prompt "A graceful swas. its snowy white feathers shimmering the
Prompt: : "Rising from the expanse of Antarctcaa, breathaking ice polace shimmers crystalline translucest refract
golden light serenely glassy lnke, Gentle ripples fan..."
Prompt "A proud sing stands metionless stop a snow-dusted bill, its anilers adorned with delicatt strands frost that glisten the pole winter
Prompé: "A striking low-angle view captures the towering ficado of modem apartmert building, sleck glass windows reflecting the vibrant huzs of
(a) Object Dataset
(b) Scene Dataset
Figure A.15: Additional examples of curated dataset. (a) An object dataset for dynamic object-centric videos and (b) a scene dataset for static scenes with camera motion. The dataset includes predefined starting points in the first frame and their pscudo ground-truth trajectories, obtained using an tff-the-shelf tracking method.
A.17

---

survey comprises sections with total questions take approximately 5-7 minutes to complete.
In the following pages, you will be presented with pairs of videos and text prompt.
Please choose the video you consider superior based on the following criteria:
Text Alignment: The degree which the video reflects the given text.
Video Quality: The overall visual quality of the video.
Motion Fidelity: The naturalness smoothness of motion the video.
Vidco
Vidco
Text Prompt: A man playing the electronic guitar, high electronic guitar.
Text Alignment:
Which video better
reflects the given
text prompt?
Video Quality:
Which video has better overall visual
quality?
Motion Fidelity:
Which vidco has natural and smooth motion?
Figure A. 16: An example of human evaluation. Participants are presented with a pair vidcos and text prompt and are instructed to evaluate text alignment, video quality, and motion fidelity.
A.18

---

Tnme
Frine
Prired
Prim
Frane 17
Prime 21
Trine
Frime
Thine 31
Prime
Franed)
Prmeed
21
24
Figure A.17: Cross-frame attention visualization across layers at timestep { 1. (a) Top-3 layers (1 13, 17, 21) exhibit sharp and precise localization. (b) Bottom-3 3 layers (1 24, 29) display diffuse and scattered attention.
Prane!
Frome 5
Frone
Frane 13
Fazze
Frume 21
Frone 25
Fratie 29
Frome 33
Frane 37
Frome
Frate 45
Flane 4P
Figure A.18: Cross-frame attention visualization across denoising timesteps at layer 17. Attention
progressively sharpens and localizes throughout denoising (t = 50 to 1).
Frazle
Franc
Frane
Figure A919: Text-to-frame attention visualization for the word "shark" across denoising timesteps layer I 17. Attention evolves from coarse to fine throughout denoising 50 1), indicating that the text prompt
primarily guides the global semantic layout early timesteps.
A.19

---

Starting Points the First Frame
Starting Points in the First Frame
VFS
Difflack
(Cog (Cog VideaX-5B1 (Cog VideoX-2BB (Hunyaan)
DifTTack
DiffTacc
Time
Time
Starting Points in the First Frame
Starting Points in the First Frame
VFS
Dit'irack
(Cog VoogVideo (B) (Cog Vidcox- (-X (Hunyuan)
Dillhack Ditllack
Time
Time
Figure A.20: Additional qualitative comparison for zero-shot point tracking. Our method produces smoother
and accurate trajectories compared VFS 83, which struggle with temporal dynamics and often yield inconsistent tracks.
A.20

---

CogVidenX-SB
Cross-Attention Guidance on JogVideoX-55
Prompt: "A panda standing on surfboard in the occan in sunset."
Prompt: ".. its claws scratching the ground to uncover hidden seeds and insects
Prompt: "A is playing the electronic guitar."
Prompt: motorcycle cruising along coastal highway." "
Prompt: 'A person playing guitar"
Figure A.21: Additional qualitative results motion-enhanced generation with CogVideoX-5B. Our sampling method, CAG, produces videos with improved motion consistency.
A.21

---

Broader Impact
DiffTrack advances the understanding of temporal correspondence in video diffusion transformers (DiTs), enabling applications such as zero-shot tracking and motion-enhanced video generation.
These capabilities can benefit diverse downstream tasks, including point tracking 16
44 4D
point tracking [1], and motion iisiion-mappulatedd video generation [24]
However, Difk'rack's ability enhance video quality may raise ethical concerns if misused create misleading or fake content. It is essential to ensure that advancements from DiffTrack are applied responsibly, especially in video synthesis and manipulation.
K
Limitations
Diff'rrack relies pre-trained video diffusion transformers (DiTs), meaning that advancements in video backbones could enhance performance, resulting more accurate tracking and coherent motion generation.
While DiffTrack effectively analyzes temporal correspondences and improves motion consistency, does not currently support motion manipulation-diiee control of video synthesis with user-defined motion trajectories. Extending DiffTrack for motion-conditioned video generation is a potential future direction.
Furthermore, Diff performs zero-shot point tracking without fine-tuning specific datasets. Incorporating fine-tuning could further improve real-world performance, which we plan to explore in future research.
A.22


Reviews:
[{"review_id": "JLRwJgKv56", "reviewer": "Reviewer_EnKm", "summary": "The paper proposes to systematically analyze transformer-based video diffusion models (DiTs) to uncover the mechanism responsible for preserving the temporal correspondence. The authors propose a way to measure temporal correspondence through key-point tracking, and create a dataset of prompt-videos to do so. This leads to the finding that key-query 3D attention captures the temporal consistency in the generation. The authors propose to use this property to develop a SoTA video keypoint tracking approach and to improve temporal consistency of the generated videos with guidance.", "questions": "Questions/Suggestions:\n- It feels like the zero-shot point tracking is a separate contribution not aligned with the main message of the paper. Perhaps the authors may want to consider removing or rephrasing it, to make the paper more to the point. \n- Long sequence handling (line 249): each chunk contains the first global frame. How does that affect 3D VAE (or is it done after) and how does this affect the 3D attention?", "limitations": "none", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is really well written and easy to follow\n- The idea of analyzing both affinity and matching accuracy is grerat. One is indeed not sufficient.\n- The results in both video generation and point tracking are impressive.\n- I really like the pipeline for video generation with negative guidance.\n\nWeaknesses:\n- My biggest concern is the prompt-video dataset. To establish the dataset, the authors generate videos with a text-to-video diffusion model. Then, to get the annotations, apply SAM and Co-Tracker to the generated videos. I have two questions here:\n    - The SAM and Co-tracker models were trained primarily on real videos. How is the accuracy of those models on the generated videos? Does this introduce an error? I find this gap potentially concerning. Given that some of the generated objects’ appearance may drift over time due to imperfect generation, what would it even mean to match the new halluzitated keypoints to the original frame?\n    - By generating videos from a specific generative model (e.g., CogVideo), doesn’t it make the dataset somewhat biased for evaluating other video generative models?", "was_revised": true, "final_justification": "I thank the authors for answering my questions. I keep my score and vote for accepting the paper.", "rebuttal": "### **General Reply**\n\nThank you for your constructive review and valuable suggestions. Below, we address each comment in detail. If any concerns remain, we welcome further clarification and will respond promptly. All discussions will be reflected in the final manuscript.\n\n---\n\n### **Weakness 1. Performance of SAM and Co-Tracker on Synthetic Datasets**\n\n> The SAM and Co-tracker models were trained primarily on real videos. How is the accuracy of those models on the generated videos? Does this introduce an error? I find this gap potentially concerning. Given that some of the generated objects’ appearance may drift over time due to imperfect generation, what would it even mean to match the new halluzitated keypoints to the original frame?\n> \n\nThank you for your question. As noted in Appendix A (line 342), we first collect 300 prompts for each object and scene dataset. Human annotators then carefully select the top 50 video–prompt pairs per dataset based on motion consistency and overall video fidelity. This manual selection process excludes generated samples that exhibit hallucination or significant appearance drift, which may cause errors in SAM or CoTracker.\n\nAdditionally, as shown in Appendix E (Figure A.5), we perform the same analysis on real videos from the DAVIS dataset, which are free from synthetic artifacts that could affect the performance of SAM or CoTracker. The results show consistent trends with those from the curated synthetic datasets, further supporting the robustness of our analysis. We will include this discussion in our final manuscript.\n\n---\n\n### **Weakness 2. Potential Bias from Using a Dataset Generated by a Specific Model**\n\n> By generating videos from a specific generative model (e.g., CogVideo), doesn’t it make the dataset somewhat biased for evaluating other video generative models?\n> \n\nThank you for your question. We would like to clarify that we use a different synthetic dataset for each video generative model to avoid bias in the evaluation (lines 126–128 in the main paper). Specifically, we curate video–prompt pairs separately for each generative backbone and use them only for analyzing that corresponding model. We will revise the paper to make this point clearer and prevent potential confusion.\n\n---\n\n### **Suggestion 1. Clarifying the Role of Zero-Shot Point Tracking**\n\n> It feels like the zero-shot point tracking is a separate contribution not aligned with the main message of the paper. Perhaps the authors may want to consider removing or rephrasing it, to make the paper more to the point.\n> \n\nThank you for your suggestion. The main goal of our paper is to understand how temporal correspondences emerge in video DiTs, and how these correspondences can be extracted (via zero-shot point tracking) and leveraged (for motion-enhanced video generation). In this context, the zero-shot point tracking experiments provide concrete evidence that the extracted correspondences from video DiTs are meaningful and superior to those from existing vision foundation models and self-supervised video models.\n\nIn the revised version, we will clarify the connection between our core message and the zero-shot tracking experiment to avoid confusion about its role in the overall contribution.\n\n---\n### **Question 1. Long Sequence Handling**\n\n> Each chunk contains the first global frame. How does that affect 3D VAE (or is it done after) and how does this affect the 3D attention?\n> \n\nThank you for your question. To clarify, the inclusion of the global first frame in each chunk is done after the 3D VAE encoding.\n\nAs described in Appendix C.1 and Figure A.4, we first pass all video frames individually through the pre-trained 3D VAE with a temporal compression ratio of $q=1$ (Figure A.4(a)). We then construct each chunk by inserting the global first-frame latent (Figure A.4(b)) alongside interleaved subsequent frame latents (Figure A.4(c)).\n\nIn the 3D attention, all frame latents in each chunk interact with each other through cross-frame attention. We then extract queries and keys from the optimal layer and timestep identified by DiffTrack to compute matching costs between the first frame’s query and all other frames’ keys within each chunk (Eq. (5)).\n\nAs demonstrated in Table A.2 of Appendix F, our design choice avoids VAE compression loss, ensures direct matching between the global first frame and subsequent frames, and leverages cross-frame interaction in 3D attention, resulting in significantly improved point accuracy for long-term tracking.\n\nWe will make this workflow clearer in the revised version."}, {"review_id": "n1npQTivSJ", "reviewer": "Reviewer_vZ2J", "summary": "This work designs a novel way for identifying which intermediate variables are most relevant to temporal correspondences and leveraging them to solve downstream tasks: point tracking and motion-enhanced video generation. Empirical results verify the effectiveness of proposed method.", "questions": "1. I am wondering how the author leveraged the existing vision foundation and supervised video models to handle the zero-shot point tracking task. Does the author also construct the track based on the query-key attention scores of specific layers? Please clarify it to prove the comparison is fair.\n2. The author only compares to the vanilla model, but there should be other relevant works for the motion-enhanced video generation task. Please provide their results to further verify the effectiveness of the method.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. The paper is generally easy to follow. \n2. Detailed experiments on three video generation models (CogVideoX-2B ，HunyuanVideo and CogVideoX-2B-I2V) are presented.\n\nWeakness:\n1. The fairness and integrity of the comparison of experiments need to further verify. For point tracking task, the implementation of baseline models is not stated. For motion-enhanced video generation task, the results were only compared to the vanilla model. \n2. Given that the best timestep is not always the last one, the generality of the proposed point tracking method needs further validation. For example, DiffTrack for zero-shot point tracking  using the last timestep. While according to the appendix, for CogVideoX-2B-I2V, the best timestep is not the last. This can undermine point tracking performance. The author need to either prove the best timestep is the last for most video generation models or prove using the  non-last timesteps can also achieve good point tracking performance.", "was_revised": true, "final_justification": "The authors' response have clarified my concerns. I hope the authors can refine their final paper accordingly.", "rebuttal": "### **General Reply**\n\nThank you for your constructive review and valuable suggestions. Below, we address each comment in detail. If any concerns remain, we welcome further clarification and will respond promptly. All discussions will be reflected in the final manuscript.\n\n---\n\n### **Weakness 1 & Question 1. Implementation of Comparison Models**\n\n> The fairness and integrity of the comparison of experiments need to further verify. For point tracking task, the implementation of baseline models is not stated.\n> \n\n> I am wondering how the author leveraged the existing vision foundation and supervised video models to handle the zero-shot point tracking task. Does the author also construct the track based on the query-key attention scores of specific layers? Please clarify it to prove the comparison is fair.\n> \n\nWe kindly note that implementation details for all comparison models are provided in Appendix C.3.\n\nFor vision foundation models such as DINO, DINOv2, DINOv2-Reg, and DIFT (SD 1.5 and SD 2.1), we strictly followed the protocols from [1], which benchmark zero-shot point tracking performance across vision foundation models. Since Stable Diffusion (SD 1.5, SD 2.1) does not provide optimal feature descriptors for correspondence, we followed DIFT [2], consistent with [1], and used features from the third upsampling block. \n\nFor self-supervised video models (SMTC, CRW, SPA-the-Temp, VFS), we also used their official implementations as feature extractors. For Stable Video Diffusion (SVD), since the official implementation does not specify an optimal feature for matching, we followed [3] and extracted features from the third upsampling block. The aforementioned models do not use query-key features for matching. For ZeroCo, we used query-key matching from the cross-attention map, which is their default matching setting.\n\nFollowing [1], since different models produce features at varying spatial resolutions, we first downsampled the input frames to 256 × 256, and then resized them such that all output feature maps have a uniform resolution of 30 × 45 before computing correlations.\n\nWe will expand this explanation in the final manuscript and publicly release the evaluation code including all comparison baselines.\n\n\n**References**\n\n[1] Aydemir, G., Xie, W., & Güney, F. (2024). Can Visual Foundation Models Achieve Long-term Point Tracking?. arXiv preprint arXiv:2408.13575.\n\n[2] Tang, L., Jia, M., Wang, Q., Phoo, C. P., & Hariharan, B. (2023). Emergent correspondence from image diffusion. *Advances in Neural Information Processing Systems*, *36*, 1363-1389.\n\n[3] Jeong, H., Huang, C. H. P., Ye, J. C., Mitra, N. J., & Ceylan, D. (2025). Track4gen: Teaching video diffusion models to track points improves video generation. In *Proceedings of the Computer Vision and Pattern Recognition Conference* (pp. 7276-7287).\n\n---\n\n### **Weakness 2. Best Timestep for Point Tracking**\n\n> Given that the best timestep is not always the last one, the generality of the proposed point tracking method needs further validation. For example, DiffTrack for zero-shot point tracking using the last timestep. While according to the appendix, for CogVideoX-2B-I2V, the best timestep is not the last. This can undermine point tracking performance. The author need to either prove the best timestep is the last for most video generation models or prove using the non-last timesteps can also achieve good point tracking performance.\n> \n\nThank you for your insightful comment. Regarding CogVideoX-2B-I2V, we used unofficial pre-trained weights from a third-party GitHub repository, with unknown training strategy and dataset. Notably, this model concatenates the first condition image to all frame latents, unlike the official I2V setting, which concatenates it only to the first latent while zero-padding the rest. This often causes the model to over-rely on the first image and produce largely static videos, which may explain the highest matching accuracy at timestep $t = 11$. We also think such misalignment stems from the model’s unknown training strategy or data and warrants further investigation.\n\nFor rigorous analysis, we further evaluated the officially released CogVideoX-5B-I2V model. In this case, we consistently observed that the last timestep yields the highest matching accuracy. In the table below, we show its point tracking performance of CogVideoX-5B-I2V ($l=16$, $t=1$) on DAVIS, revealing a significant performance gap compared to the unofficial 2B model. We appreciate your careful observation and will clarify this point in the revised version.\n\n|  | $<\\delta^0$ | $<\\delta^1$ | $<\\delta^2$ | $<\\delta^3$ | $<\\delta^4$ | $<\\delta^x_{avg}$ |\n| --- | --- | --- | --- | --- | --- | --- |\n| CogVideoX-2B-I2V (Unofficial) | 0.8 | 6.0 | 13.3 | 26.5 | 44.7 | 18.3 |\n| CogVideoX-5B-I2V (Official) | **3.9** | **16.0** | **38.2** | **57.0** | **69.0** | **36.8** |\n\n---\n\n### **Weakness 1 & Question 2. Motion-Enhanced Video Generation**\n\n> For motion-enhanced video generation task, the results were only compared to the vanilla model.\n> \n\n> The author only compares to the vanilla model, but there should be other relevant works for the motion-enhanced video generation task. Please provide their results to further verify the effectiveness of the method.\n> \n\nThank you for your valuable suggestion. \n\nWhile prior works on motion-enhanced video generation [1,2] typically rely on additional supervision or auxiliary modules, our method specifically focuses on enhancing motion without any supervision or auxiliary modules, enabling a more general and scalable solution.\n\nTo strengthen our evaluation, we compare our Cross-Attention Guidance (CAG) with Spatiotemporal Skip Guidance (STG) [3], which also aims to enhance motion without training. STG degrades the original model through selective skipping of spatiotemporal layers (including self-frame and text-frame attention), and then uses the degraded model as guidance.\n\nAs shown in the table below, CAG outperforms STG across motion consistency, motion dynamics and video quality. While STG modifies self-frame and text-frame attention, often altering the overall layout, structure, and content of the generated videos, CAG only zeros out cross-frame attention in dominant layers. This allows CAG to enhance motion while preserving the original layout and content of the generated video.\n\nMoreover, STG does not analyze how temporal correspondence emerges within video DiTs, and therefore suffers from the following limitation, as stated in their paper: \n\n*“STG’s performance depends on scale and layer selection, with the optimal configuration varying across models, requiring users to set these through heuristic tuning.”*\n\nIn contrast, CAG benefits from our DiffTrack-based analysis, which identifies optimal layers for motion-enhanced guidance, resulting in consistent performance without any manual tuning by users.\n\nWe will include this comparison and discussion in the final manuscript to better clarify the advantages and positioning of our approach.\n\n|  | Subject Consistency | Background Consistency | Dynamic Degree | Imaging Quality |\n| --- | --- | --- | --- | --- |\n| (1) CogVideoX-2B | 0.9276 | 0.9490 | 0.7917 | 0.5657 |\n| (2) (1) + STG | 0.9263 | 0.9507 | 0.7777 | 0.6031 |\n| (3) (1) + CAG | **0.9313** | **0.9564** | **0.8235** | **0.6054** |\n\n**References**\n\n[1] Chefer, H., Singer, U., Zohar, A., Kirstain, Y., Polyak, A., Taigman, Y., ... & Sheynin, S. (2025). Videojam: Joint appearance-motion representations for enhanced motion generation in video models. *arXiv preprint arXiv:2502.02492.*\n\n[2] Jeong, H., Huang, C. H. P., Ye, J. C., Mitra, N. J., & Ceylan, D. (2025). Track4gen: Teaching video diffusion models to track points improves video generation. In *Proceedings of the Computer Vision and Pattern Recognition Conference* (pp. 7276-7287).\n\n[3] Hyung, J., Kim, K., Hong, S., Kim, M. J., & Choo, J. (2025). Spatiotemporal skip guidance for enhanced video diffusion sampling. In *Proceedings of the Computer Vision and Pattern Recognition Conference* (pp. 11006-11015)."}, {"review_id": "dM487HlqK1", "reviewer": "Reviewer_YjNJ", "summary": "This paper introduces DiffTrack, a framework designed to analyze the underlying temporal correspondences between frames (thereby point tracking) during the generation process of video diffusion transformers (DiTs). The paper shows results probing the internal mechanisms of DiTs, investigating which representations (e.g. query-key similarities vs. intermediate features), layers, and timesteps are most critical for tracking. Their key findings are that query-key similarities in the attention mechanism are the primary source of correspondence information, and the tracking quality improves as the denoising process progresses. With these insights, the paper presents two practical applications. First, it shows that video DiTs can be used as zero-shot point trackers, achieving results comparable to existing tracking methods. Second, it introduces a training-free guidance method to enhance motion quality in the generated video, by perturbing the cross-frame attention maps in some key layers to steer the model towards generating videos with fewer artifacts.", "questions": "- How well will the findings (e.g. layer selection) generalize across different text prompts?\n- Will different content types lead to different layers that provide strong correspondence information?\n- Using the proposed method on CogVideoX-2B-I2V, the performance is actually a lot worse than SVD, which is also an I2V model. Can you comment on this?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The paper successfully translates analytical findings about the inner mechanisms of video DiTs into concrete applications like tracking and video quality enhancement.\n- The paper is exceptionally well-written and easy to follow. The concepts are explained clearly.\n- The analysis is systematic and provides clear and valuable insights into the behavior of video DiTs.\n\nWeaknesses\n- The central discovery that attention maps in diffusion models capture spatial correspondences is not entirely novel. Lots of prior works found this phenomenon in image diffusion models. This paper's contribution is to verify this unsurprising finding in the context of video DiTs.\n- There's a quite significant gap between the tracking performance of CogVideoX and CogVideoX-I2V, which is revealed in the appendix. It would be more appropriate to discuss this in the main paper.\n- Table 1's message is that CogVideoX combined with the proposed DiffTrack \"outperform all vision foundation models and self-supervised video models\". However, when CogVideoX-I2V is combined with the proposed DiffTrack, its performance is worse than all other baseline methods, including SVD which is an image-to-video model. These results seem to indicate that the proposed DiffTrack achieves mixed results.", "was_revised": true, "final_justification": "The author response has addressed my concerns.", "rebuttal": "### **General Reply**\n\nThank you for your constructive review and valuable suggestions. Below, we address each comment in detail. If any concerns remain, we welcome further clarification and will respond promptly. All discussions will be reflected in the final manuscript.\n\n---\n\n### **Weakness 1. Novelty of DiffTrack**\n\n> The central discovery that attention maps in diffusion models capture spatial correspondences is not entirely novel. Lots of prior works found this phenomenon in image diffusion models. This paper's contribution is to verify this unsurprising finding in the context of video DiTs.\n> \n\nThank you for the comment. We would like to highlight that reviewers Enkm, vZ2J, and YjNJ acknowledged the novelty in both our analysis (Enkm, vZ2J) and applications (vZ2J, XJK1).\n\nDiffTrack is the first work to quantify temporal correspondence in full 3D attention within video DiTs. As you mentioned, previous works [1,2,3] have explored intermediate features from pre-trained image diffusion models for correspondence. However, their analyses are primarily limited to two-frame correspondence, as image diffusion models are not designed to capture temporal relationships across video sequences.\n\nRecent works [4,5] have attempted to analyze internal representations of video diffusion models for controlled video generation. However, they do not explicitly study or quantify temporal correspondence, and are typically based on U-Net architectures, which are known to struggle with large motion. Another recent study [6] investigates attention control in video DiTs for subject consistency in long video generation, but it focuses solely on text-to-video attention, overlooking temporal alignment between frames.\n\nIn contrast, our work systematically quantifies how video DiTs build temporal correspondence across frames during video generation, using novel evaluation metrics and systems. We show how these temporal signals can be extracted (zero-shot point tracking) and leveraged for motion-enhanced video generation (Cross-Attention Guidance).\n\nWe believe our work offers crucial insights into the inner workings of video DiTs, and establishes a strong foundation for future research and applications that aim to leverage their temporal understanding.\n\n**References**\n\n[1] Hedlin, E., Sharma, G., Mahajan, S., Isack, H., Kar, A., Tagliasacchi, A., & Yi, K. M. (2023). Unsupervised semantic correspondence using stable diffusion. *Advances in Neural Information Processing Systems*, *36*, 8266-8279.\n\n[2] Meng, B., Xu, Q., Wang, Z., Cao, X., & Huang, Q. (2024). Not all diffusion model activations have been evaluated as discriminative features. *Advances in Neural Information Processing Systems*, *37*, 55141-55177.\n\n[3] Tang, L., Jia, M., Wang, Q., Phoo, C. P., & Hariharan, B. (2023). Emergent correspondence from image diffusion. *Advances in Neural Information Processing Systems*, *36*, 1363-1389.\n\n[4] Jeong, H., Huang, C. H. P., Ye, J. C., Mitra, N. J., & Ceylan, D. (2025). Track4gen: Teaching video diffusion models to track points improves video generation. In Proceedings of the Computer Vision and Pattern Recognition Conference (pp. 7276-7287).\n\n[5] Xiao, Z., Zhou, Y., Yang, S., & Pan, X. (2024). Video diffusion models are training-free motion interpreter and controller. Advances in Neural Information Processing Systems, 37, 76115-76138.\n\n[6] Cai, M., Cun, X., Li, X., Liu, W., Zhang, Z., Zhang, Y., ... & Yue, X. (2025). Ditctrl: Exploring attention control in multi-modal diffusion transformer for tuning-free multi-prompt longer video generation. In *Proceedings of the Computer Vision and Pattern Recognition Conference* (pp. 7763-7772).\n\n---\n### **Weakness 2, 3 & Question 3. Performance of Image-to-Video Models**\n\n> There's a quite significant gap between the tracking performance of CogVideoX and CogVideoX-I2V, which is revealed in the appendix. It would be more appropriate to discuss this in the main paper.\n> \n\n> Table 1's message is that CogVideoX combined with the proposed DiffTrack \"outperform all vision foundation models and self-supervised video models\". However, when CogVideoX-I2V is combined with the proposed DiffTrack, its performance is worse than all other baseline methods, including SVD which is an image-to-video model. These results seem to indicate that the proposed DiffTrack achieves mixed results.\n> \n\n> Using the proposed method on CogVideoX-2B-I2V, the performance is actually a lot worse than SVD, which is also an I2V model. Can you comment on this?\n> \n\nThank you for your valuable comments. \n\nRegarding CogVideoX-2B-I2V, we used unofficial pre-trained weights from a third-party GitHub repository, with unknown training strategy and dataset. Notably, this model concatenates the first condition image to all frame latents, unlike the official I2V setting, which concatenates it only to the first latent while zero-padding the rest. This might lead to a significant drop in performance, as the model over-relies on the first image and ultimately produces largely static videos. We also suspect this low performance stems from the model’s undocumented training procedure or data, which warrants further investigation.\n\nFor rigorous analysis, in the table below, we further evaluate the officially released CogVideoX-I2V-5B on DAVIS and observe that it outperforms SVD in point tracking accuracy.\n\nWe also observe CogVideoX-5B-I2V underperform compared to their T2V counterpart CogVideoX-5B. We believe this is due to the image-to-video finetuning objective, which primarily focuses on preserving the appearance of the first frame, producing more static videos and weakening the temporal correspondence required for generating dynamic motion. \n\nThis phenomenon is also discussed in [1], which states:\n\n*“Adapting a T2V model for I2V suppresses motion dynamics of the generated outputs, resulting in more static videos compared to their T2V counterparts... It stems from the premature exposure to high-frequency details in the input image, which biases the sampling process toward a shortcut trajectory that overfits to the static appearance of the reference image.”*\n\nWe will incorporate this discussion more explicitly into our main paper.\n\n|  | $<\\delta^0$ | $<\\delta^1$ | $<\\delta^2$ | $<\\delta^3$ | $<\\delta^4$ | $<\\delta^x_{avg}$ |\n| --- | --- | --- | --- | --- | --- | --- |\n| SVD | 3.6 | 14.6 | 34.1 | 55.7 | 71.4 | 35.9 |\n| CogVideoX-5B-I2V | 3.9 | 16.0 | 38.2 | 57.0 | 69.0 | 36.8 |\n| CogVideoX-5B | 5.2 | 20.5 | 50.7 | 73.9 | 84.3 | 46.9 |\n\n**References**\n\n[1] Choi, J. S., Lee, K., Yu, S., Choi, Y., Shin, J., & Lee, K. (2025). Enhancing Motion Dynamics of Image-to-Video Models via Adaptive Low-Pass Guidance. *arXiv preprint arXiv:2506.08456*.\n\n---\n\n### **Question 1 & 2. Generalizability across Different Text Prompts and Contents**\n\n> How well will the findings (e.g. layer selection) generalize across different text prompts?\n> \n\n> Will different content types lead to different layers that provide strong correspondence information?\n> \n\nThank you for your valuable question. We would like to clarify that, to find generalizable timesteps and layers for temporal correspondence across different prompts and contents, we intentionally curated two distinct datasets: an object dataset for dynamic object-centric videos and a scene dataset for static scenes with different camera motions (lines 131–134). Each dataset includes 50 diverse prompts, covering a wide range of content types such as animals, objects, architectures, and landscapes.\n\nWe observed that the same timesteps and layers are dominant across both datasets, indicating consistent temporal correspondence behavior regardless of text prompts or content types.\n\nThis observation is further supported by Appendix Figure A.5, where we conducted the same analysis on the DAVIS dataset, which contains real videos with dynamic human motion. The same dominant layers and timesteps emerged, strongly aligning with our findings from synthetic videos."}, {"review_id": "pefkW0mma3", "reviewer": "Reviewer_XJK1", "summary": "The paper presents a study on emergent temporal correspondences in video generation using Diffusion Transformer (DiT) models. \nThe authors first construct a synthetic dataset by generating videos, annotating the first frame, and using CoTracker [36] to obtain pseudo ground truth trajectories.\nUsing this dataset, they systematically analyze how well temporal correspondences can be extracted from intermediate features and query-key similarities across different layers and denoising steps.\nBased on these findings, they apply their method to zero-shot point tracking on the TAP-Vid benchmark, achieving state-of-the-art performance compared to both vision foundation models and self-supervised video point tracking methods.\nIn addition, they propose a novel guidance strategy that perturbs cross-frame attention to discourage degraded temporal correspondence during generation. \nThis approach enhances the temporal consistency of the generated videos.\nQuantitative metrics and a user study demonstrate that the proposed guidance method significantly outperforms the baseline in terms of motion fidelity and overall video quality.", "questions": "1. Does the proposed approach generalize to object other than animals, which was the main focus of the evaluation dataset?\n\n2. What is the inference time for the zero-shot tracker and the computational overhead from the CAG component?\n\n3. An interesting future direction would be to explore learnable feature aggregation across timesteps and denoising levels, as proposed in [1]. Additionally, handling ambiguous matching, such as switching between different wheels of the car in the first supplementary video, could benefit from techniques like [2].\n\n[1] Luo, Grace, et al. \"Diffusion hyperfeatures: Searching through time and space for semantic correspondence.\" Advances in Neural Information Processing Systems 36 (2023): 47500-47510.\n\n[2] Zhang, Junyi, et al. \"Telling left from right: Identifying geometry-aware semantic correspondence.\" CVPR (2024)", "limitations": "Yes in the supplementary material, but I suggest moving it to the main paper and showing some failure cases of both point tracking and refined video generation.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n- The paper is very well written. The narrative is clear, smooth, and logically structured, making the problem easy to follow and well motivated.\n- The preliminary section (section 2) is compact, generic, and well-structured. It effectively narrows the focus to cross-frame attention, setting up the reader for the analysis that follows.\n- Section 3.4 offers valuable insights through a thorough and well-executed analysis. I believe these insights will be highly valuable for the community in understanding DiT video diffusion models.\n- The CAG component for temporal refinement is creative, and has a significant impact on the quality of the generated videos.\n\n**Weaknesses**\n1. The paper lacks a dedicated related work section. Given the relevance of semantic correspondence task, it is important to discuss prior work in this area such as [1–3] to clarify what has already been achieved, as well as approaches that analyze temporal information in earlier video diffusion models like SVD [4].\n\n2. Fig. 3(a,b) is hard to interpret. It is unclear which layer or timestep performs best. Clearer visualizations or summaries would make the findings easier to follow.\n\n3. The object dataset was chosen to focus primarily on animals. It would be helpful to clarify the reasoning behind this choice.\n\n4. To strengthen the evaluation, it would be beneficial to include recent diffusion-based semantic correspondence methods such as [1–3]. Additionally, adding CoTracker results to Table 1 would help position the proposed approach relative to state-of-the-art supervised methods.\n\n5. It would be interesting to evaluate CAG on larger video models to assess whether the observed improvements persist.\n\n\n[1] Luo, Grace, et al. \"Diffusion hyperfeatures: Searching through time and space for semantic correspondence.\" Advances in Neural Information Processing Systems 36 (2023): 47500-47510.\n\n[2] Zhang, Junyi, et al. \"Telling left from right: Identifying geometry-aware semantic correspondence.\" CVPR (2024)\n\n[3] Stracke, Nick, et al. \"CleanDIFT: Diffusion Features without Noise.\" CVPR (2025).\n\n[4] Wang, Qian, et al. \"Zero-shot video semantic segmentation based on pre-trained diffusion models.\" CVPR (2025).", "was_revised": true, "final_justification": "The paper is clearly written and presents technical novelty supported by thorough experiments and analysis. T\nhe authors have addressed all of my questions and concerns during the rebuttal. \nI therefore recommend acceptance.", "rebuttal": "### **General Reply**\n\nThank you for your constructive review and valuable suggestions. Below, we address each comment in detail. If any concerns remain, we welcome further clarification and will respond promptly. All discussions will be reflected in the final manuscript.\n\n---\n\n### **Weakness 1 & 4. Lack of Dedicated Related Works and More Comparisons**\n\n> The paper lacks a dedicated related work section. Given the relevance of semantic correspondence task, it is important to discuss prior work in this area such as [1–3] to clarify what has already been achieved, as well as approaches that analyze temporal information in earlier video diffusion models like SVD [4].\n> \n\n> To strengthen the evaluation, it would be beneficial to include recent diffusion-based semantic correspondence methods such as [1–3]. Additionally, adding CoTracker results to Table 1 would help position the proposed approach relative to state-of-the-art supervised methods.\n> \n\nThank you for your feedback. First, we would like to clarify that we provide related works in Appendix H, covering prior studies on diffusion-based semantic correspondence and feature analysis in earlier video diffusion models. Following your suggestion, we will incorporate references [1, 2, 3, 4] into this section to better position our work.\n\nIn the table below, we present additional comparisons with recent diffusion-based semantic correspondence methods [1–3] and CoTracker3 [5] for point accuracy on DAVIS. We observe that our method outperforms these two-image diffusion-based approaches [1-3]. We will include these results in the final manuscript.\n\n|  | $<\\delta^0$ | $<\\delta^1$ | $<\\delta^2$ | $<\\delta^3$ | $<\\delta^4$ | $<\\delta^x_{avg}$ |\n| --- | --- | --- | --- | --- | --- | --- |\n| CoTracker3 (Online) | 42.9 | 67.3 | 85.2 | 93.2 | 96.8 | 77.1 |\n| CoTracker3 (Offline) | 42.1 | 66.5 | 84.0 | 93.1 | 96.4 | 76.4 |\n| DIFT (SD1.5) | 3.5 | 13.0 | 39.3 | 63.1 | 72.2 | 38.2 |\n| DIFT (SD2.1) | 3.6 | 13.3 | 40.1 | 65.8 | 75.7 | 39.7 |\n| Diffusion Hyperfeatures (SD1.5) | 3.0 | 12.9 | 33.1 | 58.4 | 74.5 | 36.4 |\n| Diffusion Hyperfeatures (SD2.1) | 3.7 | 15.5 | 38.8 | 64.5 | 78.9 | 40.3 |\n| CleanDIFT (SD1.5) | 3.1 | 13.7 | 36.5 | 64.9 | 79.6 | 39.6 |\n| CleanDIFT (SD2.1) | 3.7 | 16.0 | 41.8 | 68.6 | 81.9 | 42.4 |\n| TLR (SD1.5 + DINOv2 (ViT-B/14)) | 4.3 | 17.7 | 44.6 | 69.8 | 82.0 | 43.7 |\n| CogVideoX-2B | 4.8 | 19.4 | 49.2 | 73.6 | 84.3 | 46.3 |\n| CogVideoX-5B | **5.2** | **20.5** | **50.7** | **73.9** | **84.3** | **46.9** |\n\n**References**\n\n[1] Luo, G., Dunlap, L., Park, D. H., Holynski, A., & Darrell, T. (2023). Diffusion hyperfeatures: Searching through time and space for semantic correspondence. *Advances in Neural Information Processing Systems*, *36*, 47500-47510.\n\n[2] Zhang, J., Herrmann, C., Hur, J., Chen, E., Jampani, V., Sun, D., & Yang, M. H. (2024). Telling left from right: Identifying geometry-aware semantic correspondence. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 3076-3085).\n\n[3] Stracke, N., Baumann, S. A., Bauer, K., Fundel, F., & Ommer, B. (2025). Cleandift: Diffusion features without noise. In *Proceedings of the Computer Vision and Pattern Recognition Conference* (pp. 117-127).\n\n[4] Wang, Q., Eldesokey, A., Mendiratta, M., Zhan, F., Kortylewski, A., Theobalt, C., & Wonka, P. (2024). Zero-shot video semantic segmentation based on pre-trained diffusion models. *arXiv preprint arXiv:2405.16947*.\n\n[5] Karaev, N., Makarov, I., Wang, J., Neverova, N., Vedaldi, A., & Rupprecht, C. (2024). Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. *arXiv preprint arXiv:2410.11831*.\n\n---\n\n### **Weakness 2. Unclear Visualization**\n\n> Fig. 3(a,b) is hard to interpret. It is unclear which layer or timestep performs best. Clearer visualizations or summaries would make the findings easier to follow.\n> \n\nThank you for your comment. We will include 2D visualizations of the per-timestep and per-layer analysis corresponding to Fig. 3(a, b) in the final manuscript to more clearly highlight the dominant layers and timesteps and reduce potential confusion. We will also add a concise summary to help readers interpret the findings more easily.\n\n---\n\n### **Weakness 3 & Question 1. Reason to Choose Animal Dataset**\n\n> The object dataset was chosen to focus primarily on animals. It would be helpful to clarify the reasoning behind this choice.\n> \n\n> Does the proposed approach generalize to object other than animals, which was the main focus of the evaluation dataset?\n> \n\nWe chose animals for the object dataset because animal motion is typically non-rigid, dynamic, and deformable, unlike rigid objects. This results in more complex and challenging tracking scenarios, which we believe are valuable for analyzing temporal correspondence.\n\nAdditionally, we include the same analysis on the DAVIS dataset (Figure A.5 in Appendix E), which contains real-world videos with dynamic human motion, further supporting our findings. We will clarify this reasoning and include it in the final version of the manuscript.\n\n--- \n\n### **Weakness 5. Generalizability of Cross-Attention Guidance**\n\n> It would be interesting to evaluate CAG on larger video models to assess whether the observed improvements persist.\n> \n\nIn Table 1 and 2 below, we evaluate the generalizability of Cross-Attention Guidance (CAG) on a larger model, CogVideoX-5B. We observe that CAG consistently outperforms the baseline across all automatic metrics and human evaluation.\n\nThese results suggest that the effectiveness of CAG persists even in larger-scale models, demonstrating its scalability and robustness. We will include these quantitative results, along with additional qualitative examples, in the final version of the manuscript.\n\nTable 1. Quantitative results of CAG on CogVideoX-5B using automatic metrics on VBench.\n|  | Subject Consistency | Background Consistency | Dynamic Degree | Imaging Quality |\n| --- | --- | --- | --- | --- |\n| CogVideoX-5B | 0.9158 | 0.9590 | 0.6667 | 0.5531 |\n| CogVideoX-5B + CAG | **0.9283** | **0.9644** | **0.6863** | **0.6051** |\n\nTable 2. Quantitative results of CAG on CogVideoX-5B using human evaluation. \n|  | Video Quality | Motion Fidelity | Text Faithfulness |\n| --- | --- | --- | --- |\n| CogVideoX-5B | 39.10 | 40.91 | 30.00 |\n| CogVideoX-5B + CAG | **60.90** | **59.09** | **70.00** |\n\n---\n\n### **Question 2. Computational Overhead for CAG**\n\n> What is the inference time for the zero-shot tracker and the computational overhead from the CAG component?\n> \n\nThe table below compares inference time and memory usage per denoising timestep across three settings: the baseline, baseline with Classifier-Free Guidance (CFG), and baseline with CFG and Cross-Attention Guidance (CAG). While inference runtime increases due to the increase of the number of function evaluations, CAG’s GPU memory footprint remains very similar to that of CFG since CAG simply applies an attention mask to a few dominant layers.\n\nThis stands in contrast to prior works [1,2], which introduce additional modules and require training for motion enhancement. In comparison, our method is entirely training-free and operates within the model itself via a novel guidance mechanism.\n\n|  | Memory (GB) | Time (s) |\n| --- | --- | --- |\n| (1)  Baseline (CogVideoX-2B) | 12.75 | 1.31 |\n| (2) (1) + CFG | 12.75  | 2.60 (198% $\\uparrow$) |\n| (3) (2) + CAG | 12.84 | 4.59 (177% $\\uparrow$) |\n\n**References**\n\n[1] Chefer, H., Singer, U., Zohar, A., Kirstain, Y., Polyak, A., Taigman, Y., ... & Sheynin, S. (2025). Videojam: Joint appearance-motion representations for enhanced motion generation in video models. *arXiv preprint arXiv:2502.02492.*\n\n[2] Jeong, H., Huang, C. H. P., Ye, J. C., Mitra, N. J., & Ceylan, D. (2025). Track4gen: Teaching video diffusion models to track points improves video generation. In *Proceedings of the Computer Vision and Pattern Recognition Conference* (pp. 7276-7287).\n\n--- \n\n### **Suggestion 1. Feature Aggregation across Timesteps**\n\n> An interesting future direction would be to explore learnable feature aggregation across timesteps and denoising levels, as proposed in [1]. Additionally, handling ambiguous matching, such as switching between different wheels of the car in the first supplementary video, could benefit from techniques like [2].\n> \n\nWe appreciate your valuable suggestion. We agree that a promising direction for DiffTrack is to explore learnable feature aggregation across different timesteps and layers [1] to further enhance temporal correspondence.\n\nWe also find your point about handling ambiguous or noisy matches insightful, for example, the issue of switching between similar keypoints like car wheels. As you suggested, adopting more advanced matching techniques such as [2] could be an effective solution.\n\nThank you again for your thoughtful feedback. We plan to actively investigate these directions in our future work.\n\n**References**\n\n[1] Luo, G., Dunlap, L., Park, D. H., Holynski, A., & Darrell, T. (2023). Diffusion hyperfeatures: Searching through time and space for semantic correspondence. *Advances in Neural Information Processing Systems*, *36*, 47500-47510.\n\n[2] Zhang, J., Herrmann, C., Hur, J., Chen, E., Jampani, V., Sun, D., & Yang, M. H. (2024). Telling left from right: Identifying geometry-aware semantic correspondence. In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition* (pp. 3076-3085).\n\n--- \n\n### **Suggestion 2. Limitations**\n\n> Yes in the supplementary material, but I suggest moving it to the main paper and showing some failure cases of both point tracking and refined video generation.\n> \n\nThank you for the comment. We will move the limitations section into the main paper in the final version and provide additional failure cases of point tracking and video generation."}]

Author rebuttal:

Meta-review: After the rebuttal, all reviewers confirmed that the issues raised have been effectively addressed and expressed consistent support for this submission. The manuscript is well-written, and the proposed method is both interesting and novel. I recommend its acceptance. I encourage the authors to enhance the camera-ready version based on the feedback provided.
Decision: Accept (poster)