Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision
Authors: Chenshuang Zhang
Kang Zhang
Joon Son Chung
In So Kweon
Junmo Kim
Chengzhi Mao
Venue: NeurIPS 2025 poster ()
Keywords: Pixel-level object tracking
Generative Models
Primary area: applications

Abstract: Distinguishing visually similar objects by their motion remains a critical challenge in computer vision. Although supervised trackers show promise, contemporary self-supervised trackers struggle when visual cues become ambiguous, limiting their scalability and generalization without extensive labeled data. We find that pre-trained video diffusion models inherently learn motion representations suitable for tracking without task-specific training. This ability arises because their denoising process isolates motion in early, high-noise stages, distinct from later appearance refinement. Capitalizing on this discovery, our self-supervised tracker significantly improves performance in distinguishing visually similar objects, an underexplored failure point for existing methods. Our method achieves up to a 6-point improvement over recent self-supervised approaches on established benchmarks and our newly introduced tests focused on tracking visually similar items. Visualizations confirm that these diffusion-derived motion representations enable robust tracking of even identical objects across challenging viewpoint changes and deformations. Project page: \small{\url{https://chenshuang-zhang.github.io/projects/ted}}.

Video Diffusion Models Excel at Tracking Similar-Looking Objects Without Supervision
Chenshuang Zhang'
Kang Zhang' Junmo Kim
Joon Son Chung' Chengzhi Mao
In So Kweon
KAIST Rutgers University²
Abstract
Distinguishing visually similar objects by their motion remains critical challenge in computer vision. Although supervised trackers show promise, contemporary sef--supervised trackers struggle when visual cues become ambiguous, limiting their scalability and generalization without extensive labeled data. We find that pre- trained video diffusion models inherently learn motion representations suitable for tracking without task-specific training. This ability arises because their denoising process isolates motion in early, high-noise stages, distinct from later appearance refinement. Capitalizing on this discovery, our self-supervised tracker significantly improves performance in distinguishing visually similar objects, an underexplored failure point for existing methods. Our method achieves up to a 6-point improve- ment over recent self-supprvised approaches established benchmarks and our newly introduced tests focused on tracking visually similar items. Visualizations confirm that these diffusion-deervved motion representations enable robust tracking of even identical objects across challenging viewpoint changes and deformations. Project page: https: htrpss//ccheshuang- zhang..g-hngg.. github. io/projects/ted
1 Introduction
Imagine tracking one of two similar-looking deer walking in the forest (Figure Ta)). Humans effortlessly resolve such visual ambiguities by relying the distinct motion signatures of objects. This ability to perceive coherent objects through their unique temporal dynamics, even when static appearances are confounding, is fundamental. However, imbuing visual representations with this innate understanding of temporal dynamics, especially for tracking similar-looking objects, remains a
significant challenge in computer vision [3 23 10 51].
Many self-supervised methods 23 45 while good at learning intra-frame appearance features, fail when confronted with visually similar targets (see DIFT [45] in Figure [f)). Their Achilles' heel is the neglect of frame temporal relationships. Even approaches that incorporate temporal signals
(igure[[)).
through training objectives like cycle-consistency [32 52 24 often process frames independently at
inference using 2D image encoders. This inherently limits their ability to model the continuous motion crucial for disambiguating similar objects in dynamic scenes (see CRW (24 and Spa-then-Temp [3] Figure ()).
In this paper, we show that representations for similar-looking object tracking do not need to be learned from scratch with intricate tracking-speeific objectives. Instead, they can be repurposed within the internal workings of pre-trained video diffusion models (58 2]. Unlike methods that view video as sequence of isolated images (20 23 36). video diffusion models, by their very nature of generating coherent and realistic video, must implicitly capture the complex interplay of inter-frame dynamics. We find that the denoising process, particularly as it reconstructs motion from highly noisy states, already encodes rich, motion- motion-aware representation through its feature activations representation ripe for tracking without any explicit tracking supervision.
"Coreesponingg author. Junmo Kim <junmo. kim @ kaist.ac. Chengzhi Mao <cmm I @ww.11388s..ee cutue.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

First Frame
Target Frame
First Frame
Target Frame
(a)
(b)
(c)
DIFT
X
CRW
X
-e-tthen-Temp
Ours
(a) Video Frames
(b) Ground Truth Label
(c) Predictions For Target Frame
Video label propagation similar-looking objects. State-of-the oftte-of-the-att self-supervised trackers, such as DIFT [45.. CRW [24] and weetthheeeeempp BI], often struggle when multiple objects look similar in video. This failure is due to their exclusive reliance appearance features. By dissecting and repurposing pretrained video diffusion models, we construct a feature that captures correctly track similar-looking objects, such the deer highlighted by the green box in (c). In this figure, the green and red masks represent segmentation maps of different objects, while the blue, green, and red boxes highlight the ground truth regions, correctly predicted regions, and incorrectly predicted regions, respectively.
Figure 1:
intra-frame motions in videos, allowing
We introduce the Temporal Enhanced Diffusion tracking framework (TED), simple yet remark- ably effective approach that harnesses these latent diffusion features. TED synergizes the motion intelligence distilled by video diffusion models with conventional appearance features, enabling and robustly track visually indistinct objects
it conquer the limitations of prior art 6 23 45 (Figure (c)..
Experimental results show that our TED method outperforms 17 popular self- self-supervised models, achieving hfateoofthhe-ee performance pixel-level object tracking. On the widely-used DAVIS-2017 benchmark [35]. our TED significantly outperforms recent self- self-supprvised methods (3 45 36 31] by up to 6%. When evaluated on videos that include multiple similar-looking objects, our TED method achieves larger improvement by up to 10%. Visualizations confirm that representations encode differently for similar looking objects with different motion. Our approach also achieves significant improvement in other challenging scenarios, such as appearance-identiee objects, real- world viewpoint changes, and object deformations.
2
Related Work
Learning video representations for temporal correspondence is crucial for visual tracking [46 55 31 Due to limited annotations, recent studies have proposed various pretext tasks to learn representations in self-supervised manner. We discuss related work below.
Self-supervieed representation learning from images. Prior studies learn appearance features in video representations by training models on independent images 20
6
23
45 Some methods adopt
instance discrimination as a pretext task, such as MoCo 20) and SimCLR 6
SFC (3 improves
further by integrating image-level and pixel-level cues for representation learning. DIFT [45]
leverages knowledge from image diffusion models [40]. However, these methods only learn intra- frame appearance features, which fail in tracking visually similar objects (Figure((c)).
Self-supervised representation learning from videos. Some methods introduce temporal signals to model training, using two pretext tasks: sycle-consistency over time and frame reconstruction. Cycle- consistency task tracks a patch backward and forward in time to align its start and end points B2 52 24], while frame reconstruction aims reconstruct pixels from adjacent frames [47
27 Recent
studies integrate temporal and spatial cues for training, such as Spa-then-Temmp 31 and SMTC [36
However, during inference, these models process frames independently using 2D image encoders, neglecting temporal context. Therefore, they fail to track similar-looking objects as in Figure c).

---

16
1 label
16 label
DIFT
X
CRW
X
Spa-Temp
X
Ours
(a) Video frames
(b) Ground truth label
(c) Predictions from SOTA models (d) Video diffusion
Figure 2: Our approach successfully tracks objects with identical appearances. We conduct controlled study, that we perform object label propagation on videos featuring two identical-looking and independently moving balls, with frames and their ground truth labels shown in (a) and (b). State- of-the-art methods (24] 31 455 fail to distinguish these two balls, leading to incorrect predictions (c). In contrast, our approach accurately track both balls despite their identical appearance (d).
Video object segmentation. Supervised methods for video object segmentation 57 3 achieve impressive results but rely on large-scale annotated datasets for model training. For example, SAM2 38 is trained 50.9K videos with 35.5M masks. In contrast, work addresses self- supervised tracking: no segmentation labels are used. Furthermore, models like SAM2 [8 use discriminative training objectives, whereas our work explores the inherent tracking capability of generative models. We find that video diffusion models can effectively track visually similar objects without any tracking-specific supervision, pointing to promising direction for future trackers.
Video diffusion models. Diffusion models 21 have achieved great success in image generation 37 43 34 [2], such Stable Diffusion 440 and ADM [14). Video diffusion models further include
temporal blocks for frame consistency
3 48 with pioneering work Sora 4., 12VGen-XL. [58]. and
Stable Video Diffusion
(2] Diffusion models have also been used in tasks like image classification 29
12). semantic segmentation [ (60 (33 and pose estimation [22] [6]. In contrast to these studies, we the first to show that video diffusion models excel tracking similar-looking objects without any tracking-specific training. Our finding that video diffusion models learn motions at high-noise stages also advances the understanding of video diffusion models.
Track by diffusion models. There has been recent interest in applying diffusion models to track- ing (25 59 50]. Track4Gen [25] tackles point tracking by training video diffusion models on labeled point trajectories, whereas our work explores pretrained video diffusion models for object segmen- tation without any packing-speciingg training.
[59] is built on image diffusion models with additional motion encoders to learn temporal cues. By contrast, our work directly explores the built-in motion of pretrained video diffusion models without extra modules. VidSeg (50) performs instance-agnostic video semantic segmentation that cannot distinguish different objects in the same category. It also requires maintaining and updating additional KNN classifier learn temporal changes during tracking. By contrast, our approach can distinguish even similar-looking objects without extra components.
Challenges for Tracking Visually Similar Objects
Task definition. We focus video label propagation task, which aims to transfer ground truth labels of the first frame (e.g., segmentation map) to subsequent frames [47]. The key is training models to obtain frame representation R. which learns pixel-level correspondence among frames 23 36 B1) Due to limited annotations, prior studies train models in a self-supervised manner [24] 31. with pretext tasks like instance discrimination 20 frame, the label is predicted by aggregating labels of its most similar pixels from previous frames, where the pixel similarity is computed using representation R (see Section 4.4 for details).
During inference, for each pixel in the current
Challenges for tracking -lnikar-lookig objects. Label propagation for visually similar objects demands capturing robust motion signals, as appearance cues can become ambiguous and misleading. While prior studies excel at tracking objects using appearance features 23 36 31] they often fail when tracking multiple, similar- lmilaag-lookig objects. We find this is due to their excessive dependence appearance a shortcut effective for distinct objects but a point of failure when objects are visually alike.
To illustrate this, we begin our investigation with a controlled toy experiment featuring two identical- looking, independently moving balls (Figure
aa)). a)). Given the ground truth segmentation map of each

---

Time
Vidco Sequence
Framz Regresenatione
X, (4-4.--
(R) ,., R13
Netto
Top X:00
N2
Video Dificdon Uitet
Handle long slding windows
Mation representation each
propagation
Figure 3: Framework. Our work tracks objects via video label propagation, which transfers ground truth label of the first frame to subsequent frames. As video diffusson models typically have a maximum input length, we first divide the long video into overlapping video windows (see (a)). For each window, we use video diffusion models to to extract frame representations that capture rich inter-frame motion features(see (b)). Specifically, our method uses the 3D UNet backbone that can process the entire video sequence along the temporal axis. Finally, to predict the label for query pixel in the target frame (R'), we follow prior studies to aggregate the labels of its most similar pixels in previous frames (see (c); details in Section We term our method Temporal Enhanced Diffusion tracking framework (TED). Experiments demonstrate that our TED improves tracking performance across diverse video scenarios, including those with similar-lookee objects.
ball in the first frame (Figure b), left), the task is to predict pixel-level labels of subsequent frames. Since the balls are identical, motion is the only signals that the trackers can rely on to make the right prediction. Figure c) shows that state-of-the arrt methods [45 24 31 struggle with object identity, leading to poor tracking. Moreover, we experiment on real- world videos with similar-looking objects. As shown in Figure [c),, state-of-the-art trackers 45] 24 31 fail to distinguish similar-looking deer
when they swap positions. These findings highlight the difficulty tracking multiple similar-looking objects in video label propagation.
4 Teemporal-Enhanced Diffusion for Tracking
In Section 3
show that state-of-the-art methods fail track visually similar objects, highlighting the difficulty of self-supervised tracking when visual cues are ambiguous. To address this challenge, propose a new Temporal- Vemporal-Enhanced Diffusion tracking framework (TED). We show that video representations for similar-objects tracking do not necessarily to be learned from scratch with tracking- specific objectives. Our TED leverages the motion intelligence from pre-trained video diffusion model, enabling robust tracking similar-looking objects.
In this section, we first introduce the tracking setup and video diffusion models. We then show how we obtain motion-aware representations without any tracking-specific objectives, and how to complement them with appearance features for further improvement. Finally, we show how these representations yield tracking results by label propagation.
4.1 Preliminaries: Tracking Setup and Video Diffusion Models
We focus video label propagation task 47 defined in Section 3 Following prior work (27), aim to learn a frame representation. R', , for each frame I, of a video, such that the similarity between representations reflects the true correspondence of pixels across frames.
Our approach builds upon video diffusion models, which are often obtained by adding a temporal dimension image diffusion models, using 3D architectures to capture temporal context. Video diffusion models are trained to generate realistic video sequences by learning to reverse a diffusion process 21 39 This process involves adding Gaussian noise to clean video X at different noise levels, indicated by step r. The model, 60. is trained to predict the added noise at each step 7, minimizing the following loss:
Ex,e~ [[|e co(X", .T)
(1)
where
X Va,X" -I-virr are, 8~ N~,11
(2)

---

Video Frames
DIFT x
Spa-then-Temp x
Rm (Ours) v
Figure Representation visualization. We show a video with two similar looking deer swapping positions in the first column, and PCA results of model representations in column 2-4 (similar pixel colors indicate similar features). While state-of- the-art Spa-then- Temp [1]] and DIFT [45] ob- tain similar features for both deer, our R. dis- tinguishes each deer by their different motions.
Time
Correct Matching
X Incorrect Matching
Here, a, is noise schedule parameter, with larger indicating higher noise levels. W0, denotes the Gaussian distribution. This training process forces the model to learn not only the visual content of individual frames, but also the coherent motion that connects them. Given noisy video X", the model obtains a cleaner X by removing the predicted noises in X", termed denoising process. In this work, we study video diffusion models using the widely-used 3D UNet cø. which is built by inserting temporal layers, such as temporal attention and 3D convolution, into a 2D UNet [41]. Our work is also model-agnosse which adapts to any pretrained video diffusion model that may not use 3D UNet.
4.2 Space-time Learning for Motion-aware Representations
We begin by investigating the internal feature activations of video diffusion models. We find that high noise levels during the denoising process encode rich representation of motion, while low noise levels primarily capture appearance information (see Figure Section 5.3). This discovery motivates us to a novel approach for object tracking. We leverage the internal feature activations (Ep) at these high noise levels to extract robust motion cues.
Formally, given a video sequence
{11, (2,........ .2......... IN), we first add noise to obtain X (following Equation 2) Then, we perform a single forward pass through the UNet of video diffusion models (UNet,) to obtain features from n, layer, with entire video X as input (Figure (b)):
R R R UNe,,,(" n.))
(3)
where T represents the noise level. Crucially, unlike prior methods that compute RR by independently processing each frame (R' = F(I)) using 2D image encoders, our video diffusion model's 's features R R² R²...., ......,RA R. incorporate temporal information through temporal attention and 3D convolutions. Consequently, each R represents not only the appearance of frame It but also inter-frame motion dynamics captured within the representation from high-noise inputs, enabling effective tracking.
Handling long videos: sliding window approach. Video diffusion models typically have maximum input length, L. To handle videos longer than that, inspired by temporal segment networks [49), we adopt a sliding window method, as shown in Figure (a.) The video X (71, I2,..., IN) is divided into multiple overlapping short video clips {X&} with window size L.
X1 121,22,.... Ltb.
X₂ U1+ L=Lo-ovsapp:- -... J L- ovverapp,
where 0 < overlap < L. By integrating the temporal knowledge from the 3D UNet, for each frame representation R in the clip, it encodes the temporal motions from all frames in the current video clip Xş. Overlapping frames further improve motion consistency among video clips.
Visualization of motion-aware representations. After obtaining representations from video diffu- sion models (R we study if R,, can differentiate similar-looking objects. We visualize both our representations R,, and representations from state-of-the-ann art methods [31] 45 in Figure where two similar-looking deer swapping positions over time. We perform principal component analysis (PCA) 33 on two frames (denoted s and t) for each model (e.g.. R R PCA(R R for our R.,) In Figure 4 similar pixel colors indicate similar representations. Figure shows that prior capture similar features for different deer, indicated by similar colors. contrast, our R,, learns clearly distinct features for each deer, shown as different color. These results highlight the superiority of our method in capturing object motions, enabling tracking similar-looking objects. Note that PCA is used only for visualization, and the original Rm is used for tracking.
methods 33 45
5

---

First Frame
Spa-then-Temp X
DIFT X
TED (Ours)
Ground Truth
(a)
(b)
(c)
(d)
(e)
Figure 5: Predictions for pixel-level object tracking. We evaluate TED on the video label prop- agation task, comparing its predicted segmentation maps with those from state-of-the-art meth- ods [31 45). Our TED consistently outperforms both methods [31] 45 on DAVIS (Figure a-d) and YouTube-Siminn Similar (Figure e-f) datasets. aligning with Table Notably,
TED delivers more accurate predictions in scenarios with complex deformations (a) and viewpoint changes (b), while Spa-then-Temp []] and DIFT 45 struggle with tracking completeness, c.g., the missing arm in (a).
Our TED also achieves superior tracking in multi-object scenarios, such as interacting objects (c-d) and similar-looking objects (e-f). In contrast, Spa-then-Temp [3 and DIFT [45] have mislabeling issues, such as incorrect labels for the gun in (d) and misaligned labels for sheep in the background (f). These results show that our TED significantly improves tracking performance, highlighting the superiority of our motion-aware representations in tracking. (Best viewed when zoomed in.)
4.3
Motion Meets Appearance for Robust Tracking
While the motion-aware representations extracted from the video diffusion model (R,,) are pow- erful, they not the only source of useful information for tracking. Appearance cues remain important, particularly for distinguishing objects that are not identical. Inspired by the Two-Stream ConvNets [44), we combine the orthogonal motion (R,,) features from vidco diffusion models and appearance (R.,) features from a pre-trained image diffusion model 45
Rmm Rm
R. RR, 1|
R, concat A
(1
-
(5)
where denotes L.2 normalization, and X is a weighting factor (between and 1) that controls the relative importance of motion and appearance. Different from R,, which learns inter-frame features as defined in Equation] 3] R., is obtained by feeding each frame independently into the 2D UNet; of an image diffusion model. Specifically, for each frame Ir, we compute its appearance feature as R = UNet, [ m4). where ! is computed by adding noise to Jt, and nı is the block index within image diffusion model for feature representation. We refer to this combined approach using frame representation R, for tracking as Temporal-En Ennancced Diffusion tracking method (TED).

---

4.4 Tracking via Label Propagation
To perform tracking (i.e., label propagation). we follow the standard protocol used in previous work [52 24] [2].. Given the ground truth labels in the first frame I1 we use a recurrent method to propagate the labels to subsequent frames from I2 to IN. based on frame representations (Ry).
Figure 3 c) shows how we predict the label for a query pixel i in the target frame (R'). Define the first frame and the previous m frames reference frames, we first compute the pairwise similarities between pixel i and pixels in the reference frames. In Figure []c), we show the case with one
reference frame, with representation termed R'. Following prior studies [24] 23 31], restrict the similarity computation to a spatially local neighborhood (i)) around i. This yields similarity matrix Aer. where each element (i.j) the dot product of the representations of pixel in R and pixel j in R' (with j E S(i)). To identify the most similar pixels to i, we retain only the top-K values from Aer to form A,,, setting all other values to zero. Finally, the label for pixel is predicted by aggregating the labels from its most similar pixels in the reference frames using weighted sum:
Me (i)
Σ Σ Arr -,())))(ii)) JES(8)
(6)
The pixel-level labels yt is computed in the representation space and then interpolated to match the We show the pseudocode of our TED in Appendix A
size of video frames following [24 31.
4.5 Implementation Details
Motion-aware representations R... Any pretrained video diffusion models can serve as our motion feature backbone. We default to the widely-used 12VGen-XL (58) and also explore Stable Video Diffusion [2]. Since 12VGen-XL supports up to 16 frames, longer videos are split into multiple 16-frame clips. We pass each clip through model's 3D UNet and extract features from the third block as Rm The diffusion step F for computing model input X is chosen empirically.
Appearance-aware representations Rg. Our TED framework supports any frame representations that learn appearance features. Different from Rm that takes the entire video sequence as model input (Section 4.2p we obtain R., for each frame by inputting the image independently to the image encoder (i.e., R = F(I,). We default to image-diffusion model ADM [4] as image encoder, extracting features from its eighth block as R.g- We also test Stable Diffusion (39]
Tracking via label propagation. Our TED method uses R a fusion of Rm, and R., to obtain tracking results. We follow the setups of prior studies (24 23 45 for label propagation, with pseudocode and details in Appendix A and Appendix B.I
Experiments
5.1 Experimental Setups
Baselines. Our TED advances self-supervised tracking without any labeled training data. We evaluate on video label propagation task, and compare against 17 sfate-of-the-ard sulf-supervised methods.
Self-supervised representation learning from images. We evaluate 7 models that learn appearance features by training on independent images. We consider instance-discrinination methods, e.g., MoCo 200.. We also test SFC (23), a strong baseline that integrates image-level and 45). which leverages knowledge from image diffusion models 40
pixel-level cues, and DIFT
Self-supervised representation learning from videos. We benchmark 10 models that incorporate temporal cues to training through diverse pretext tasks. We evaluate on strong basclines trained for frame reconstruction (c.g., UVC 32)) cycle consistency (c.g., CRW (24). and video We also include recent Spa-then-Tema [1 and SMTC (36..
contrastive learning (e.g., VFS 555
Datasets. Our method uses pretrained diffusion models for tracking, without additional training. We benchmark on following test sets, with video examples in Appendix [B.2 We follow prior studies report region similarity (Jn.) and contour accuracy (J m. for evaluation.
24 36 31
36 31 and evaluate on the widely-used DAVIS-2017 validation set (35), which contains 30 videos (2023 frames, 59 objects).
Standard benchmark. We follow previous work
332
27
52
7

---

[I] by 6.4% and Table 1: Results for pixel-level similar-looking object tracking task. Our TED advances self- supervised tracking without any labeled data. We evaluate it the video label propagation task against 17 state-of-the-ar self-supervised methods. "Temporal Train' indicates whether the method uses temporal signals during training. Colored numbers indicate the best results. Our TED achieves significant improvements across all datasets. On the widely-used DAVIS benchmark [35,, our TED outperforms recent methods by up 6%. On Simiilr, featuring real-world similar-looking objects, our TED achieves an even larger gain 10%. On Kubric-Similar, with two identical- looking. independently moving balls, TED reaches high Im of 87.2%, while most methods stay near 50%, equivalent to random guessing due to the objects' identical sizes. These results highlight the effectiveness of our TED in tracking himilar-looking objects.
Temporal
Method
DAVIS Jm(1)
Youtube-S Sinilar Ja(1) Fa(1)
Kubric-Sinilar Jm(1) Fm
Train
F&FF((t)
Fm((1)
JkFFm(1)
J&FFa(t)
InstDis
5
63.9
MoCo
20
65.9
63.4
68.4
48.0
48.5
47.4
56.6
$1.6
61.6
SimCLR 6
66.9
64.4
69.4
37.5
36.9
38.1
55.6
50.3
60.9
X
BYOL 19
66.5
64.0
46.5
54.8
49.2
60.5
SimSiam
67.2
64.8
68.8
47.4
47.9
47.0
58.4
52.6
64.1
71.2
74.0
55.5
55.3
55.7
47.7
43.1
$2.3
DIFT 45
75.7
72.7
78.6
60.7
59.8
61.7
55.1
$2.7
57.6
Colorization
+7
34.0
34.6
32.7
TimeCycle
48.7
46.4
50.0
39.8
41.3
38.2
50.6
44.0
57.2
CorFFlow
28
48.4
39.6
40.0
39.3
32.6
27.0
38.3
UVC 12
60.9
59.3
62.7
49.7
49.8
49.7
56.9
$1.3
62.6
VINCE
44.9
44.3
54.1
48.5
59.7
MAST 2x
65.5
63.3
67.6
CRW 24
67.6
64.8
70.2
$2.0
52.3
51.6
54.9
49.7
60.1
VFS 55
66.5
71.3
57.3
57.1
57.5
44.2
38.5
49.9
SMTC 36
73.0
69.4
76.6
57.5
57.2
57.9
68.6
64.7
72.5
Spa-then-Temp
31
59.6
59.2
60.1
48.9
44.0
53.8
TED (Ours)
80.8
65.1
67.0
90.2
87.2
93.1
similar-looking benchmark. We introduce Coutube-Similas including 28 videos featuring similar-looking objects from Youtube-VOS [56], totally 839 frames and 69 objects.
Real-world
benchmark. In real-world vidcos, visually similar objects can still differ due to factors like gestures. To climinate these variations, we introduce Kubric-Similan including 30 videos (480 frames, 60 objects) in which two identical-lookin balls move independently. The dataset is generated by Kubric simulator [18], with random ball colors, sizes, and motions.
Controlled
5.2 Experimental Results
Quantitative results. We compare our TED method with 17 self-supervised methods in Table Our TED achieves the state-of-the-art tracking performance all datasets. On the standard DAVIS dataset, our TED significantly outperforms recent methods by up to 6%, such as SFC [5] by 6.4%, SMTC 36 by 4.6%, Spa-then- Temp 31 by 3.5% and DIFT 45 by 1. 1.9%. By introducing motion- aware features from video diffusion models, our TED achieves an even greater improvement when tracking similar-looking objects on Youtube-Simil Similar, such as Spa-then-Tems DIFT [45) by 5.3%. On Kubric-Similar that includes identical objects, many methods achieve Jm around 50%, no better than random guessing due to identical sizes of two balls. By contrast, our TED achieves high Jm of 87.2%. These improvements highlight the effectiveness of our method in object tracking, even for challenging settings with multiple similar-looking objects.
Visualizations. Figure Bjouparccompeee our tracking results with state-of-the-arr methods on the DAVIS dataset (a-d) and CouTube--Smiila (e-f). Our TED approach significantly outperforms prior methods, aligning with Table Our TED effectively handles complex deformations (Figure and viewpoint changes (Figure 5(b)), while Spa-then Tem--eemp [II and DIFT (45) struggle with elements like the human arm FFigur(5(a)) Our TED also excels in -ojtti scenarios, such as interacting objects (Figure c-c)) and similar-looking objects (Figure 5 e-f).. By contrast, Spa-then-Temp [31] and DIFT 45 often confuse different objects, leading incorrect tracking results. For example, in mislabels a gun as human and DIFT shows significant contour errors. In Figure 5 f), both Spa-then- Spamthen-Temp and DIFT mistakenly assign the target label to background
Figure 5 d), Spa-then-Tems

---

DAVIS
Soutube-Simila
Lubric-Similar
80
60
(a)
40
+
Rm
%
200
Denoising step (T)
Denoising step (T)
Denoising step (T)
(b)
Clean frame
T=100
T=300
Y-600
7=900
Figure 6: Tracking results under different denoising steps. We evaluate tracking performance using model inputs X various denoising steps т, where larger indicates more noise (see (b)). The performance of appearance features R, degrade significantly as r increases, while our motion feature Rm maintains high tracking accuracy even with a large т. Notably, Rm peaks at 7=600 on Koutube-Similar and 7=900 on Kubric-Simine -imila,, where appearance cues are almost available. These results reveal that video diffusion models can learn object motions from highly noisy inputs, enabling effective, motion- aware tracking.
sheep. These results demonstrate that our TED significantly outperforms prior methods across various scenarios, highlighting the superiority of motion-aware features in work.
5.3 Analysis and Ablation Studies
The impact of diffusion features from different noisy levels on motion-based tracking, We evaluate tracking performance using model inputs X" various denoising steps (see Equation and Equation 3), as shown in Figure 6 We will show that video diffusion models capture object motions even early denoising steps when the imput X highly noisy.
Figure 6 a) shows that the tracking performance of appearance feature R., drops significantly with larger 7 (e.g., 2 600). This is because X is heavily corrupted at high noise levels, shown in Figure b). thus appearance features are almost unavailable. In contrast, our motion feature R., achieves high tracking results high noise levels. Interestingly, R.m even achieves its best performance (marked by a star in the figure) at 7=600 on Youtube-Similn and r=900 on Kubric- crucial for identifying similar-looking
Similar, when R,, almost fails. While motion features objects, appearance features provide fine-grained details for accurate segmentation. This explains why our fused representation R, outperforms motion-only features R. world data, highlighting the benefit of jointly leveraging motion and appearance cues for tracking.
An interesting question is: how do video diffusion models learn object motions with highly noisy in- put X With loss defined in Equation [ diffusion models are trained to reconstruct clean input from its noisy counterparts. To achieve this goal, they solve different tasks at different noise levels (TI. When X is highly corrupted high noise levels, video diffusion models are trained to solve the hard task that learns coarse-grained signals in the video, such as motion (c.g., changes of object positions among frames). Therefore, its representation R,, encodes rich motion information that enables effective track- ing of smilar-looking objects. When input X" is less noisy, diffusion model is trained to denoise appear- ance details, where motion features are also learned but may not be prioritized, leading to performance decrease at low noise levels. Our analysis and results
90
DAVIS
Youtube-Simiarr
80
Kubric-Similar
70
60
0.0(R,)
0.2
0.4
0.6
0.8
1.0(R...
Fusion weight (A)
Figure 7: Fusion weight (A). Our method integrates the advantages of motion and ap- pearance features. For dataset where visual clues are ambiguous, such as Kubric, more video diffusion features are important in im- proving the accuracy.
provide new insights into both tracking and
video diffusion models.

---

The effect of coefficients for combining motion and appearance. Figure 7 shows tracking accuracy with varying fusion weight À (see Equation 5p. where x=1 gives R, = Rm while x=0 gives R = Ra. On Kubric-Similar with visually identical balls, motion features solely are sufficient for successful tracking. Our results align with this expectation by achieving the best result with A=1.0. On real-world DAVIS and Youtube-Similar, our R performs best with moderate À value around 0.5. These results show that our R effectively integrates the advantages of motion and appearance features in complex scenarios, outperforming the case of using either Rm R. alone.
The effect of layers for feature representation. We extract representations from internal layers of video and image diffusion models for tracking, with block indices denoted as n, and as in Section 4 Our framework is agnostic to specific layers. Motion and appearance features can be taken from different layers, and the optimal layers for the two backbones do not need to be the same. Following [45), we use the decoder representations from UNet. We report the tracking results using Rm alone from different
Table 2: Block indexes. Motion represen- tation R., achieves the best tracking results when extracted from the third block of pre- trained 12VGen-Xi XL (58]
Block
FkFm((1)
Jm(1)
Fm
1
24.8
28.2
21.4
2
47.6
52.7
42.5
66.3
63.4
69.1
4
31.5
27.2
35.8
decoder blocks in Table 2 Table 2 shows that the medium block (block 3) yields the best performance among all blocks on the DAVIS dataset.
The impact of overlapping frames on tracking. Motion features are crucial for successful tracking of similar-looking objects. F shows the track- ing accuracy on Youtube-Similas versus overlapping frames (l) among video clips. Compared to the non-overlapping case (I=0), introducing overlapping frames (I>0) achieves higher tracking accuracy due to improved motion consistency among video clips. At the same time, a small I (e.g., (=2) is sufficient for good performance since tracking accuracy improves marginally with higher values of I. These results highlight the importance of accurate inter-frame mo- tions in frame representations when tracking similar- looking objects.
65.5
65.3
04.3
12
Overlapping frames())
Figure 8: Overlapping frames among video clips (l). A small / (e.g., (=2) is sufficient for boosting tracking accuracy, highlighting the importance of using multiple input frames in capturing motion clue for our method.
The effeet of diffusion models. We evaluate our TED using representations from different diffusion models on DAVIS dataset, as shown in T Our TED achieves the best tracking results using motion features from 12VGen-XL. (58) and appearance fea- tures from ADM [14, which are used by default in this work.
Table 3: Pretrained diffusion models.. Our TED achieves the best tracking results using representations from 12VGen-XI. ADM 14.
58 and
R...
R.
J&FF,(tt
Jn(1)
Fi((1)
SVD
39
71.5
74.1
SVD
2
ADM
14
76.6
73.6
79.7
Computation cost analysis. Our method has 20% more computation time than generative model method DIFT [45] DAVIS videos and 89% than the popular self-supervised discriminative model method SFC 23). For memory, our method uses 64% more than DIFT and 705% more than SFC. See more details in Appendix B.3
71.7
74.5
12VGen 58
ADM
14
77.6
74.4
80.8
6 Conclusion
We demonstrate that video diffusion models excel tracking similar-looking objects without any task- specific training. We show that pre-trained video diffusion models possess an inherent, previously unrecognized ability to encode motion information at high noise levels. Rather than designing complex architectures or training objectives for tracking, we simply extract this readily available motion representation, achieving state-of-the-art tracking performance. Our approach achieves significant improvement over prior methods in diverse scenarios, such challenging viewpoint changes and deformations. Our work opens new avenues for leveraging the latent capabilities of diffusion models beyond generation.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea (oVe)) NNo.RS-2022-1122018.4 Develop- ment and Study of Al Technologies to Inexpensively Conform Evolving Policy on Ethies), and by the ITRC (Information Technology Research Center) grant funded by the Korean government (UTP-2025-RG) RS- SE0-5-SS-2223-00220 2023-00259991),
References
[1] Tomer Amit, Eliya Nachmani, Tal Shaharbany, and Lior Wolf. Segdiff: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:212..003%) 2021.
[2] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arX5::211..15127, 2023.
[3] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.
[4] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[5] Mathilde Caron. Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[6] Ting Chen, Simon Komblithh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[7] Xinlci Chen and Kaiming Hc. Exploring simple siamese representation Icarning. Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 15750-1577 15758, 2021.
[8] Ho Kei Cheng, Scoung Wug Oh, Brian Price, Joon- Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 3151-3161, 2024.
[9] Ho Kei Cheng and Alexander Schwing. Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model. In European Conference on Computer Vision, pages 640-655 658. Springer, 2022.
[10] Ho Kci Cheng, Yu- Wing Tai, and Chi- Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in Neural Information Processing Systems, 34:117814 1111111111177 11794, 2021.
[11] Jooyoung Choi. Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11472- 1481, 2022.
[12] Kevin Clark and Priyank Jaini, Text-to-image diffusion models are zero shot classifiers. Advances in Neural Information Processing Systems, 36:58911-58997,, 2023.
[13] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344 16359, 2022.
[14] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
[15] Henghui Ding, Chang Liu, Shuting Hc, Xudong Jiang, Philip HS Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF international conference computer vision, pages 20224-20231 2023.
[16] Junqiao Fan, Jianfei Yang. Yuecong Xu, and Lihua Xie. Diffusson model is good pose estimator from 3d ri-vinon.. In European Conference Computer Vision, pages 1-18. Springer, 2024.
11

---

[17] Daniel Gordon, Kiana Ehsani, Dieter Fox, and Ali Farhadi. Watching the world go by: Representation learning from unlabeled videos. arXiv preprint arXiv:2003.07990, 2020.
[18] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David J Fleet, Dan Gmanapragasam, Florian Golemo, Charles Herraann al. Kubric: A scalable dataset generator. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3749-3761, 2022.
[19] Jean-Base non-BBatien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Docrsch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self supervuee learning. Advances in neural information processing systems, 33:21271-21284, 2020.
[20] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Gitshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 9729-9738, 2020.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information proessing systems, 33:6840-6851, 2020.
[22] Karl Holmquist and Bastian Wandt. Diffpose: Molti-hypothesis human pose estimation using diffussin models. In Proceedings of the IEEE/CVF international conference on computer vision. pages 15977-15987, 2023.
[23] Yingdong Hu, Renhao Wang, Kaifeng Zhang, and Yang Gao. Semantic-aware fine-grained correspondence. In European Conference Computer Vision, pages 97-115. Springer, 2022.
[24] Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence contrastive random walk. Advances neural information processing systems, 33:19545 19560, 2020.
[25] Hyconho Jeong, Chun-Hao P Huang, Jong Chul Ye, Niloy Mitra, and Duygu Ceylan. Track4gen: Teaching video diffusion models to track points improves video generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 7276-7287, 2025.
[26] Hueihan Jhuang. Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael Black. Towards understanding action recognition. In Proceedings of the IEEE international conference on computer vision, pages 3192 3199, 2013.
[27] Zihang Lai, Erika Lu, and Weidi Xie. Mast: A eeemory-augmenterr self-supervised tracker. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479-6488, 2020.
[28] Zihang Lai and Weidi Xie. Self-supervised leamning for video correspondence flow. In BMVC, 2019.
[29] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2206-2217, 2023.
[30] Muyang Li, Yujun Lin. Zhekai Zhang. Tianle Cai, Xiuyu Li, Junxian Guo, Enze Xie, Chenlin Meng. Jun-Yan Zhu, Song Han. Svdqunat: Absorbing outliers hy low-rank components for 4-bit diffusion models. arXiv preprint arXiv:241 1.05007, 2024.
[31] Rui Li and Dong Liu. Spatial-then-temppral self-supervised learning for video correspondence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2279-2288, 2023.
[32] Xueting Li, Sifei I. Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. Joint-task self-supervised learning for temporal correspondence. Advances Neural Information Processing Systems, 32, 2019.
[33] Andrzej Mackiewicz and Waldemar Ratajezak. Principal components analysis (pea). Computers Geosciences, 19(3):303-344 1993.
[34] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffuusin models. arXiv preprint arXiv:2112. 10741, 2021.
[35] Jordi Pont-Tuset Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Hornine- Van Gool. The 2017 davis challenge video object segmentation. arXiv:1704.00665, 2017.
Hornung, and Luc
12

---

[36] Rui Qian, Shuangrui Ding, Xian Liu, and Dahua Lin. Semantics meets temporal correspondence: Self- supervised object-centric learning in videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16675-16687, 2023.
[37] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):, 2022.
[38] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 200,:4440.00714, 2024.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[41] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention MMICAAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[42] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Ktir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 22500-22510 2023.
[43] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, al. Photorealistic text-to- image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479 (144499-66494, 2022.
[44] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
[45] Luming Tang, Menglin Jía, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:1363-1389
2023.
[46] Ran Tao, Efstratios Gavves, and Amold WM Smeulders, Siamese instance search for tracking. In Proceedings the IEEE conference computer vision and pattern recognition, pages 1420 1429, 2016.
[47] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In Proceedings of the European conference computer vision (ECCV). pages 391-408, 2018
[48] Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06:: 2023.
[49] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang. and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20-36.30 Springer, 2016.
[50] Qian Wang, Abdelrahman Eldesokey, Mohit Mendiratta, Fangneng Zhan, Adam Kortylewski, Christian Theobalt, and Peter Wonka. Vidseg: Training free video semantic segmentation based on diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22985 22994, 2025.
[51] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, pages 1328-1338, 2019.
[52] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle-consistency of time. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2566-2576, 2019.
[53] Thaddaus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. Video models are scro-shot learners and reasoners. arXiv preprint Stti:25509.03222 2025.
13

---

[54] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings the IEEE conference computer vision and pattern recognition. pages 3733-3742, 2018.
[55] Jiarui Xu and Xiaolong Wang. Rethinking self-supervised corresoondence learning: A video frame-level similarity perspective. Proceedings of IEEE/CVF International Conference on Computer Vision, pages 10075-10085,,022
[56] Ning Xu, Linjie Yang. Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vor vos: A large-scale vidco object segmentation benchmark. arXiv preprint 1809,.88800033800 2018.
[57] Zongxin Yang. Yunchao Wei, and Yang. Associating objects with transformers video object segmentation. Advances in Neural Information Processing Systems, 34:2491-2500 2021.
[58] Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qing. Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. 2023.
[59] Zhengbo Zhang, Li Xu, Duo Peng, Hossein Rahmani, and Jun Liu. Diff-tracker: text-to-image diffusion models are unsupervised trackers. In European Conference on Computer Vision, pages 319-337. Springer, 2024.
[60] Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3729-5739, 2023.
[61] Qixian Zhou, Xiaodan Liang, Ke Gong, and Liang Lin. Adaptive temporal encoding network for video instance-leve human parsing. In Proceedings of the 26th ACM international conference on Multimedia, pages 1527-1535, 2018.
14

---

A Pseudocode of Our Temporal Enhanced Diffusion Tracking Method (TED)
We provide the pseudocode of our Temporal Enhanced Diffusion tracking method (TED) in Algo- rithm [ For clarity, we denote the process of obtaining noisy imput X for video diffusion model in Equation 2 of Section 4.1 as the function AddNoise We also term the similar process for image diffusion models that adds noise to image inputs as AddNoiselD.
B Experimental Setups and Results
B.1 Tracking Setups for Label Propagation
We follow the experimental setups of prior studies 23
45
3
for video label propagation, as
summarized in Table
Table 4: Experimental setups of TED for video label propagation.
Dataset
Video diffusion Timestep
Image diffusion
Fusion weight
Softmax temp
Propagation radius
for
Model
Block
Model
Timestep
Block
top-k
DAVIS
12VGen-XL
ADM
0.4
0.2
15
Youtube-Similar
12VGen-XL
600
3
ADM
0.6
0.1
15
Muhrii-SSiilla
12VGen-XX.
900
3
ADM
51
8
1.0
0.1
15
10
B.2 Datasets
Our method applies pretrained diffusion models for tracking, without additional training. We introduce the test sets in Section 5.] and show video examples from each dataset in Figure 9
DAVIS-2017
Youtube-Simile
Kubris-Simlir
Figure 9: Video examples from test sets. Follow- ing prior studies 24 36 31 45 evaluate
the standard DAVIS- 2017 benchmark [35] (first column). To evaluate tracking visually simi- lar objects, we introduce Youtube-Similar (second column), real-world test with similar-looking objects, and Kubric- -imiilr (third column), con- trolled set with identical objects.
To ensure the quality of Youtube-Similar, three graduate students were hired to manually select videos from Youtube- VOS [56] according to the following rubrics. Only the videos that all annotators find qualified are included in the final YouTube-Similar dataset. The first rubric is object similarity. To create dataset with similar-looking objects, we first select videos that contain least two objects belonging to the same category from YouTube-Vor VOS. The second rubric is video filtering. We exclude static videos from the pool obtained in the first stage. In such videos, spatial appearance features can serve as a shortcut for tracking, which may influence our evaluation of the motion features learned in model representations.
B.3
Computational Cost Analysis
We compare computation cost with prior methods in Table 5 tested on a single A 100 GPU using DAVIS videos. Our full model achieves the highest accuracy (77.6%) using a time of 682 ms per frame, compared to DIFT (75.7%, 566 ms) and SFC (71.2%, 360 ms).
For each video, technique to boost accuracy in our full method is averaging representations computed from multiple noisy inputs. We also provide an efficient variant by removing this averaging step, which runs 521 ms and achieves 77.2% accuracy, still significantly outperforming prior methods. Our efficient version achieves a tradeoff between tracking accuracy and efficiency.
Our work is the first to show that video diffusion models can track similar-looking objects without vacking-speeifie training. Our finding that motion features are learned at high-noise stages also provides new insight into video diffusion models. Additionally, method is compatible with
15

---

Algorithm I: Temporal Enhanced Diffusion Tracking (TED)
Input:
Video frames 1 I2,...... ......,,,y IN; Ground-truth label Y1 for 1;: Video diffusion model UNet;: Image diffusion model UNet; Denoising steps: Ty (video diffusion) and т, (image diffusion); Block index for features: Ry (video diffusion) and n, (image diffusion); Fusion weight A.
Output: Label predictions Y2, Y3..., Yy for frames I2..... IN- Initialize queue Q / for storing representations and labels of reference frames;
2 Let L be the maximum input length for UNet, and / be the number of overlapping frames between video clips;
3 Divide the entire long video to ClipNummerr = L(NLLL L) L)J + video clips using sliding window approach.
for k = 0 to ClipNumber - 1 do
Define current video clip X (1144L- 0y -
Step 1: Compute Frame Representations
(a) Motion-aware Rm Compute R., in one forward pass of UNet.:
R 1++((1-1) ,Rm R. UNet, (ddN, TT): n.):
(b) Appearance-aware R.: Compute R., in multiple forward pass of UNet,:
For each frame 1, E Xg. compute R UNet, UNet,(AddNoll (AddNoiseID(4a, T.), m4);
11
(c) Fused Ry: For each frame 1 Xx, compute fused representation:
R =
concat A
R |R
(1 - |R
R
12
Step 2: Predict Tracking Labels
if k 0 then Resize label Yi to match the size of Ry, termed as as yı. Add (R) 31) to Q:
14
15
for each frame I E Xx do
16
if (R'.W) we) is already Q then continue;
17
for each query pixel in R2 do for each pixel from each reference frame R, E Q do j j locates the spatial neighborhood of pixel (S(1)) then Compute similarity score A (i. = CooPrroduct(R, RT);
19
20
22
Identify the most similar pixels to by retaining the top- K values Ay, and setting others as zero, obtaining Aby; Predict the label of pixel by: (1) = Ereq Ejes(y) (1, vr(
23
24
Add (Ry, Ub) to Q;
25
if Size(Q) equals the maximum allowed reference frames then remove the oldest entry from Q;
26
27
Interpolate y/ to the original frame size to obtain Y;;
28
return Y2,Y Ya.... .yi
Table 5: Computation cost analysis. We report the tracking accuracy (J&FF) and computation cost per frame on DAVIS. Our full method achieves the highest accuracy, while our efficient version achieves a tradeoff between tracking accuracy and efficiency.
Method
Accuracy
Time (ms)
Memory (GB)
SPC
2X
71.2
360
1.9
DIFT
75.7
566
Ours (Efficient)
77.2
11.8
Ours (Full)
77.6
682
16

---

acceleration techniques such as FlashAttention [13] and quantization [3) for further speedup. We leave further efficiency optimization for future work.
B.4 Results More Tasks and Datasets
We conduct experiments for human pose tracking on JHMDB dataset [26] and human part tracking on VIP dataset [6T] shown in Table We also evaluate our method on YouTube- [56] and MOSE [15] for video object segmentation, as shown in Table 7 Experimental results show that our method consistently improves tracking accuracy across diverse datasets and tracking tasks.
Table Results on JHMDB for human pose tracking and VIP dataset for human part tracking.
Dataset
JHMDB
PCK
PCK@0.2(1)
mloU (t)
SFC 25
61.9
83.0
38.4
Spa- pen-then-temp
84.4
DIFT [45]
63.4
84.3
43.7
(Ours)
Table 7: Results on YouTube- VOST and MOSE datasets for video object segmentation.
YouTube-VOS
MOSE
J&FF((1)
Jm(t)
Fa(1)
(&FF.(t)
Jn(*)
Fm(()
DIFT [45
70.5
68.2
72.7
34.5
28.9
40.1
TED (Ours)
71.1
68.9
73.4
35.6
30.1
41.1
C Discussions
Limitations and future work. Although our approach achieves significant tracking improvement across various scenarios. it comes with certain limitations. As discussed in Section [.3 and Ap- pendix B33] using video and image diffusion models for tracking increases computational cost compared to previous methods. However, our work aim show that video diffusion models can ef- fectively track similar objects without tracking-specific training, suggesting a new direction for future trackers. Our finding that motions are learned high-noise stages also advances the understanding of video diffusion models. One promising research direction distilling motion intelligence from video diffusion models into smaller models for more efficient tracking. We leave further efficiency improvements for future work.
Broader impacts. Our work leverages the motion intelligence from video diffusion models and achieves the state-of-the-an tracking performance without task-specific supervision, reducing reliance on costly labeled data. This greatly benefit various applications, such as robotics and autonomous driving. Since our method leverages pretrained diffusion models for tracking, its performance may reflect biases present in those models, such the underrepresentation of certain video types. We believe that mitigating bias in diffusion models a promising research direction. We hope our work encourages further studies in this field, which benefit not only generative tasks but also perception tasks such as tracking.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our work finds that video diffusion models excel at tracking similar-looking objects without task task-specific training. Building this insight, propose self- supervieed tracking method, Temporal Enhanced Diffusion (TED), which significantly improves track- ing performance across diverse scenarios. Our abstract and introduction reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
18

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the suppiemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
Justification: Our work is reproducible, and introduce all the information needed re- produce the main experimental results in Section 4 Section 5.1 Appendix A Appendix B.I
and Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release the code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cc
public/guides/Coeeee/ http:///oooeeummiiiiiiioeeneeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips. Siiaps.cccpublie wwtppbbiccgguueeeeeooooooccpcped
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We introduce experimental setups and details in Section 5.1 Appendix A Appendix B.I and Appendix B.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Our work uses pre-trained models without additional training. During tracking, our work averages video representations computed by batch of noisy inputs for each video, achieving stable results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer:
[Yes]
Justification: We introduce the computation cost of proposed method in Section 5.3 and Appendix [B.3
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httpp/////.aaeep...cpppppc/hceaa
Answer: [Yes]
Justification: Our work conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in Appendix C
Guidelines:
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not include data or models that appear to have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use public pretrained models and datasets. We also properly cite and introduce them Section 4.5 and Section 5.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduce new datasets, Kubric-Similar and Youtube-Similar, in Section 5.1 and Appendix B.2
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does include crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
This work does not involve LLMs for the core method development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
24


Reviews:
[{"review_id": "zQgd6r7FYv", "reviewer": "Reviewer_HZ3G", "summary": "This paper proposes a simple yet effective method for visual tracking, especially in the scenario of tracking similar-looking objects.The method employs the learned features by pre-trained video diffusion model in the early denoising stage as the motion feature based on the observation that such features excel at capturing motion dynamics. \n\nThe idea is simple yet interesting and the experiments show the effectiveness of the method.", "questions": "Refer to the weaknesses.", "limitations": "Yes.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea is simple yet interesting. \n2. The paper is presented well.\n3. The experiments demonstrate the effectiveness of the method.\n\nWeaknesses:\n1. The paper is based on the observation/assumption that the learned features by the pretrained video diffusion models at early denoising stage have the innate ability of capturing motion dynamics. Is there any theoretical evidence for this apart from the experimental observations?\n2. I don't think it is a fair comparison to compare the proposed method with only self-supervised methods since the proposed method is built upon the pre-trained diffusion models for both videos and images. Is it possible to compare it with supervised trackers or develop a supervised version of the proposed method and then make a comparison with existing supervised trackers?\n3. How about the performance of the proposed method on typical/routine tracking scenarios/datasets?\n4. I am curious about the efficiency of the proposed method. Extracting features during inference requires the inference of both a pretrained video diffusion model and a pretrained image diffusion model, which seems computationally expensive. \n5. I doubt the effectiveness of the weighting factor 'lambda' in Equation 5, since the R_m and R_a are extracted from two individual feature space.", "was_revised": true, "final_justification": "Thanks for the response which has addressed most of my concerns. Nevertheless, I am still not quite satisfied about the lack of a more convincing demonstration of the basic observation/assumption that the learned features by the pretrained video diffusion models at early denoising stage have the innate ability of capturing motion dynamics.\nBased on the response, I am glad to raise my score to 4.", "rebuttal": "Thank you for the valuable feedback. We are glad that the reviewer finds **our proposed tracking method interesting and effective, supported by experiments**. We truly appreciate your time and effort in reviewing our work, and we address your concerns below. Please let us know if any further information is needed.\n\n\n  ### 1. Paper scope and contribution of our empirical findings \n\n Thank you for your comment. As you pointed out, our work empirically demonstrates that features learned by pretrained video diffusion models at early denoising stages capture motion dynamics. We highlight that **one of our contributions is to be the first to report this property and validate its effectiveness for tracking through empirical experiments.** Theoretical derivation is beyond the scope of this work, and rigorous theory remains an open challenge for the deep learning field.\n\n\n### 2. Our method is fully self-supervised\n\nThank you for your comment. **Our work uses diffusion models trained without any tracking-specific labeled data, and is self-supervised in terms of tracking.** \n\nAs you suggested, we ran experiments comparing our approach with state‑of‑the‑art supervised methods, including SAM2[1], Cutie[2], XMem[3], and STCN[4]. Self‑supervised methods often underperform their supervised counterparts. Nevertheless, **our method achieves the best tracking accuracy among self‑supervised approaches and significantly narrows the gap between self‑supervised and supervised methods**. We will include these results in the revised paper.\n\n\n|            |               |      | Davis|      |    | Youtube-Similar  ||\n|-----------------|---------------|:------:|:------:|:------:|:------:|:------:|:------:|\n|                 |               | J&Fm ($\\uparrow$) | Jm  ($\\uparrow$)            | Fm ($\\uparrow$)  | J&Fm ($\\uparrow$) | Jm ($\\uparrow$)   | Fm ($\\uparrow$)  |\n| Self-supervised | SFC           | 71.2 | 68.3            | 74.0   | 55.5 | 55.3 | 55.7 |\n|                 | Spa-Then-Temp | 74.1 | 71.1            | 77.1 | 59.6 | 59.2 | 60.1 |\n|                 | DIFT          | 75.7 | 72.7            | 78.6 | 60.7 | 59.8 | 61.7 |\n|                 | Ours          | 77.6 | 74.4            | 80.8 | 66.0   | 65.1 | 67.0   |\n| Supervised      | STCN          | 85.3 | 82.0              | 88.6 | 63.9 | 62.6 | 65.2 |\n|                 | XMem          | 86.2 | 82.9            | 89.5 | 82.1 | 79.9 | 84.4 |\n|                 | Cutie         | 87.2 | 84.3            | 90.1 | 87.3 | 84.7 | 89.8 |\n|                 | SAM2          | 89.8 | 86.7            | 92.9 | 86.9 | 84.1   | 89.6 |\n\n\n\n[1] Nikhila Ravi,  Valentin Gabeur, et al. Sam 2: Segment anything in images and videos. ICLR 2025.\n\n[2] Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon-Young Lee, and Alexander Schwing. Putting the object back into video object segmentation. CVPR 2024.\n\n[3] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson327 shiffrin memory model. ECCV 2022.\n\n[4] Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. NeurIPS 2021.\n\n### 3. Consistent improvement across three additional standard datasets and tasks\n\nThank you for your suggestions. In the main paper, we have already shown significant performance improvements on three datasets, including standard DAVIS, YouTube-Similar, and Kubric-Similar. \n\nFollowing your suggestions, we conducted experiments on three additional, widely-used typical datasets and tasks, including:\n* Standard datasets for video object segmentation:  YouTube-VOS [5] and MOSE [6].\n* Standard dataset for human pose tracking: JHMDB [7].\n\nExperimental results in the following table demonstrate that **our method consistently improves tracking accuracy across diverse datasets and tracking tasks.** We will include these results in the revised paper.\n\n|  |      | Youtube-VOS     |  |      |  MOSE    | |\n|-------------|:------:|:------:|:------:|:------:|:------:|:------:|\n|             | J&Fm ($\\uparrow$) | Jm ($\\uparrow$)   | Fm ($\\uparrow$)   | J&Fm ($\\uparrow$) | Jm ($\\uparrow$)   | Fm ($\\uparrow$)   |\n| DIFT        | 70.5 | 68.2 | 72.7 | 34.5 | 28.9 | 40.1 |\n| **Ours**        | **71.1** | **68.9** | **73.4** | **35.6** | **30.1** | **41.1** |\n\n\n|          |  JHMDB  |    |   \n|---------------|:---------:|:---------:|\n|               | PCK@0.1 ($\\uparrow$) | PCK@0.2 ($\\uparrow$) | \n| SFC           | 61.9    | 83.0      |      \n| Spa-then-temp | 66.4    | 84.4    |      \n| DIFT          | 63.4    | 84.3    |      \n| **Ours**          | **68.3**    | **85.8**    |      \n\n\n[5] Ning Xu, Linjie Yang, et. al. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.\n\n[6] Henghui Ding, Chang Liu, et.al. MOSE: A New Dataset for Video Object Segmentation in Complex Scenes. ICCV2023.\n\n[7] Hueihan Jhuang, Juergen Gall, et.al. Towards understanding action recognition. ICCV2013.\n\n### 4. Clarification on computation cost\n\nThank you for your comment. We report the computation cost of our method in Section 5.3 (lines 290–293) and provide more details in Appendix B.4. Our method has 20% more computation time than generative model method DIFT on DAVIS videos and 89% than the popular discriminative model method SFC. We also discuss optimizing the computational efficiency of our method in Appendix B.4 and Appendix C.\n\n\n### 5. Discussions on fusion weight factor\n\nThank you for your comment. We conduct experiments to investigate the effect of the weighting factor $\\lambda$ in Section 5.3 (lines 270–281). Experimental results show that, across a range of fusion weight $\\lambda$, our fused representation consistently outperforms using either motion or appearance features alone in real‑world videos. These results demonstrate the effectiveness of integrating features from video and image diffusion models through a simple weighted fusion scheme."}, {"review_id": "TuSZo6lXLZ", "reviewer": "Reviewer_TBTL", "summary": "The paper studies the task of label propagation specifically for the challenging case of objects with similar appearance. They identified that the 3D UNet denoising models learned for generating videos with video diffusion models (pretext task) generate motion representations which can generalize to the task of label propagation (downstream task), highlighting the tracking capabilities of such models. \n\nThey generate frame features for each frame of a video by concatenating frame appearance features and frame motion features. For motion features, they add noise to a video by following the noise schedule of diffusion models. This noisy video is fed jointly (using video clips of fixed length) to a 3D-UNet model from which they extract frame features at the nth level.  For appearance features, they feed each noisy frame separately to a 2D UNet of an image diffusion model.\n\nThey propagate labels from the first frame to the current one by weighting the previous frame's pixel labels based on pixel similarity scores within the local neighborhood of the current frame's pixels.\n\nFor this study, they leverage pretrained diffusion models trained offline, and don’t pretrain them.", "questions": "- Can this work be extended to other video label propagation datasets? Traditionally, the video label propagation models are also evaluated on two other datasets (JHMDB[A] and VIP[B]) where the propagated labels are not only segmentation masks but human body parts or pose keypoints. Could the proposed method be extended to those scenarios?\n- Could the author provide videos the following two videos (can be merged in one):\n  - the segmentation label propagation to emphasize the quality of the tracker and\n  - video extending figure 4, which would emphasize the temporal motion consistency of the feature extracted?\n- Could the authors compare the results of Sam2 and Cutie models on their proposed datasets?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Paper Strengths:\n\n[An important study for the community]\n- The task and the problem is well identified and motivated with a simple toy example. Tracking similar appearance objects is still a challenging task that is worth studying and which is understudied in supervised tasks such as VOS because of lacking datasets highlighting the challenge.\n\n[New datasets contribution]\n- Two datasets are going to be released upon acceptance of the paper (written in the Neurips paper checklist). Highlighting the challenges of tracking similar looking objects is an essential part of this paper, and releasing the two new datasets is a clear contribution that would benefit not only to the self-supervised community but would also benefit to the supervised community. \n\n[Paper Clarity]\n- Well written paper and well organised paper structure, with a clear motivation. Qualitative visualizations support the analysis, which strengthens the message.\n\n[Qualitative analysis demonstrate the quality of the tracker]\n- Figure 4 presents examples with not only similar appearance objects (e-f) but also challenging motion examples including motion blur (d), similar appearance background/foreground (b), body part occlusions (a), zoomed out to zoomed in objects (c ). Such results are impressive, and it would be interesting to confirm them with video results.\n\n## Weaknesses:\n\n[Clarification of the claim about video diffusion models]\n- The study focuses on video diffusion models. What is the definition of a video diffusion model? is it to have a 3D Unet noise model? Because technically, a video diffusion model could be built on top of a 2D UNet noise model, or even have different architecture. The title makes a strong claim, whereas the experimental study (I2VGen-XL and Stable Video Diffusion) is only performed for 2 pretrained diffusion models using 3D-Unet. \n- Specifically, l123, they introduce 2D-UNet, and l139-140, they mention temporal attention and 3D convolution to be essential for temporal information. A clarification paragraph at the end of section 4.1 is needed.\n\n[Importance of the layer from which the representation is extracted]\n- In the pseudo code in the appendix section, two parameters are mentioned: “Block index for features: nv (video diffusion) and ni (image diffusion)”, but I cannot find reference about them in the main paper. Which layers were selected in your model? Should it be the same layer for the motion and image features? \n\n[Missing evaluations]\n- It would be valuable to report performance on additional segmentation label propagation datasets (such as YouTube VOS[51], MOSE[C]) which contain but are not limited to similar appearance objects. This would better demonstrate the proposed model’s general effectiveness on the label propagation task for segmentation masks. Relying solely on the DAVIS-17 validation split, which is a small dataset with limited similar challenging examples, is insufficient for a comprehensive evaluation.\n- Can this work be extended to other video label propagation datasets? Traditionally, the video label propagation models are also evaluated on two other datasets (JHMDB[A] and VIP[B]) where the propagated labels are not only segmentation masks but human body parts or pose keypoints. Could the proposed method be extended to those scenarios?\n\n[Extension to the supervised trackers of VOS]\n- The authors note that supervised trackers require extensive labeled data, yet the paper lacks evaluation of SoTA supervised models (SAM2 [35], Cutie [8], and others [9] [10]). Given that these models also struggle to distinguish between objects that have similar appearances, the proposed datasets would be valuable to quantify the performance of both self-supervised and supervised models on similar appearance object tracking. However, the absence of results for SAM2 and Cutie is a missed opportunity. Comparing supervised and self-supervised models—even if the latter underperform—would strengthen the study, as it addresses different, yet related, tasks.\n\n[Video qualitative analysis]\n- Missing video examples for a video analysis. Such an analysis would demonstrate the quality of the tracker. \n\n\n## Minor weaknesses:\nIn the main method section, the appearance features are not described, but are described partially in the implementation details l191-193. It would benefit from being in the core method.\n\n\n## Additional references:\n[A] H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black. Towards understanding action recognition.\n\n[B] Qixian Zhou, Xiaodan Liang, Ke Gong, and Liang Lin. Adaptive temporal encoding network for video instance-level human parsing.\n\n[C]  Henghui Ding,  Chang Liu,  Shuting He,  Xudong Jiang,  Philip H.S. Torr,  Song Bai MOSE: A New Dataset for Video Object Segmentation in Complex Scenes", "was_revised": true, "final_justification": "This paper claims that pretrained video diffusion models provide good representations that capture motion dynamics. They can be **used as is** to enrich representation only learned from image diffusion models. To highlight the benefits of their new proposed representation, they designed a specific use case where motion dynamics is important to learn: label propagation of similar appearance objects. \n\nThey introduce two new datasets:\n* Kubric similar: a toy dataset where balls with similar appearance are moving to disambiguate the contribution of appearance and motion cues and **quantify if models can capture motion dynamics**. On this dataset, supervised models achieve almost perfection and the authors show that their proposed method is the first self-supervised method to be able to bridge the gap with SoTA supervised models (+ 80% compared with the best self supervised approach). They even surpass a less recent supervised method emphasizing that it relies more on appearance cues than on motion cues.\n* YouTube-Similar: a real world dataset extracted from the YouTube VOS dataset. On this dataset they provide a +20% boost compared with the best performing self supervised method (and with SAM2 as the upper limit). If the gains are more limited than on the Kubric similar dataset, they are still significant in more complex scenarios where other factors (lightning, deformation of the object) that still rely on the appearance cue are also non-negligible. **By concatenating the appearance and motion feature maps they propose a simple integration of appearance and motion cues and emphasize the importance of both in the design of a good video representation**. Simple concatenation is not enough, but it is a valid starting point for this study.\n\nEventually, they also prove the validity of the proposed representation learning scheme on standard datasets with not only similar appearance objects for three label propagation tasks:\n- VOS task with Davis, YTVOS and MOSE dataset\n- Human body part propagation with the VIP dataset\n- Human pose with the J-HMDB dataset\n\nFor all of those cases, the proposed method provides a boost in performance with regard to other self supervised models. However, it doesn’t bridge the gap with supervised models.\n\nRegarding the time consumption of the proposed model, it is more consuming than other self supervised approaches (+20% to the most costly method) which can be balanced with the accuracy boost it provides. Nonetheless, given the exploratory component of this study, I would not take into account a lack of accuracy and time performance for accepting this paper (but this is a personal opinion, and I can completely understand diverging opinions).\n\n\nRegarding my rating:\n- The 5 rating is a pretty strong rating (looking at the text near the rating), and I cannot say that the evaluation conducted is excellent with strong confidence, because probably more could have been done before the rebuttal and still after the rebuttal (specifically on the VIP dataset). However, I believe this is a technically solid paper, and that representation learning of motion dynamics is an important research question, which justifies (in my opinion) the impact on at least one sub-area of AI. As a reader, I was very interested in this paper, and would be happy to discuss it with co-workers or the authors. I believe it deserves to be shared with the community.", "rebuttal": "Thank you for your positive review and insightful suggestions. We are glad that **the reviewer recognizes our work as an important study for the community,  for highlighting the challenging and understudied task of tracking similar-looking objects**. We also appreciate the **positive comments on the contribution of new datasets, the well written paper with clear motivation,  and the impressive qualitative visualizations that demonstrate significant improvements across diverse video scenarios.** We truly appreciate your time and effort in reviewing our work, and we address your concerns below. Please let us know if any further information is needed.\n\n### 1. Clarification on video diffusion models\n\n#### 1.1 Discussions on video diffusion models\n\nThank you for your comment. Video diffusion models are often obtained by adding a temporal dimension to image diffusion models, using 3D architectures to capture temporal context. In this work, we study the widely used 3D UNet, which is built by inserting temporal layers into a 2D UNet. Our tracking method achieves significant improvement in tracking similar-looking objects based on these models. However, our tracking method is model‑agnostic: it relies on intermediate features from any pretrained video diffusion model, which may not be a 3D UNet. We will include these discussions in the revised paper.\n\n\n#### 1.2 Clarification on temporal layers in video diffusion models\n\nThank you for pointing this out. The key to effectively tracking similar-looking objects is learning temporal motions among frames. We adopt the widely-used 3D UNet from video diffusion models, which is typically created by inserting the temporal layers into a 2D UNet. Temporal attention and 3D convolution are two successful and widely used temporal layers in video diffusion models. We will follow your suggestions to clarify this part in Section 4.1 of the revised paper.\n\n\n### 2. Discussions on model layers for feature extraction \n\nThank you for pointing this out. In Section 4.2 (lines 136-137), we define $n$ as the layer index for feature extraction. Following your suggestions, we will revise the paper to explicitly define $n_v$ and $n_i$ as the block indices for video and image diffusion models, respectively.\n\nOur framework is agnostic to specific layers.  Motion and appearance features can be taken from different layers, and the optimal layers for the two backbones do not need to be the same. We empirically extract motion features from the 3rd block of I2VGen‑XL and appearance features from the 8th block of ADM. These choices are noted in Section 4.5 (lines 189 and 194). For your reference, we report the effect of layers from video diffusion models on DAVIS dataset in the following table, which shows that features from 3rd block achieve the best tracking results.  We will include these experiments and discussions in the revised paper.\n\n| Block | J&Fm ($\\uparrow$) | Jm ($\\uparrow$)   | Fm ($\\uparrow$)   |\n|-------|:------:|:------:|:------:|\n| 1     | 24.8 | 28.2 | 21.4 |\n| 2     | 47.6 | 52.7 | 42.5 |\n| 3     | 66.3 | 63.4 | 69.1 |\n| 4     | 31.5 | 27.2 | 35.8 |\n\n\n### 3. Additional evaluations\n\n#### 3.1 Evaluation on additional video object segmentation datasets \n\nThank you for your suggestions. We conducted experiments on Youtube VOS and MOSE datasets as you suggested. Experimental results in the following table show that our method consistently outperforms the state-of-the-art DIFT method in both datasets. **These results demonstrate the general effectiveness of our proposed methods across various video scenarios.** We will include these results in the revised paper.\n\n\n|  |      | Youtube-VOS     |  |      |  MOSE    | |\n|-------------|:------:|:------:|:------:|:------:|:------:|:------:|\n|             | J&Fm($\\uparrow$) | Jm($\\uparrow$)   | Fm($\\uparrow$)   | J&Fm($\\uparrow$) | Jm ($\\uparrow$)  | Fm ($\\uparrow$)  |\n| DIFT        | 70.5 | 68.2 | 72.7 | 34.5 | 28.9 | 40.1 |\n| **Ours**        | **71.1** | **68.9** | **73.4** | **35.6** | **30.1** | **41.1** |\n\n\n#### 3.2 Evaluation on additional tracking tasks\n\nThank you for your comment. Yes, our method can be extended to other video label propagation tasks. Specifically, we conducted experiments on the human pose keypoints tracking task on JHMDB dataset, and report the experimental results in the following table. **Experimental results show that our approach significantly outperforms baseline  methods, demonstrating the effectiveness on tasks beyond video object segmentation.** We will include these results in the revised paper. \n\n|          | JHMDB   |    |  \n|---------------|:-------------:|:---------:|\n|               | PCK@0.1($\\uparrow$) | PCK@0.2($\\uparrow$) | \n| SFC           | 61.9    | 83.0      |      \n| Spa-then-temp | 66.4    | 84.4    |      \n| DIFT          | 63.4    | 84.3    |      \n| **Ours**          | **68.3**    | **85.8**    |      \n\n\n### 4. Comparison with supervised methods\n\nThank you for your suggestions. We conducted experiments to evaluate state-of-the-art supervised models, including SAM2 [35], Cutiie [8],  XMem[9],  and STCN[10]. As the reviewer anticipated, self-supervised methods underperform the supervised ones. We will include these results in the revised paper.\n\n\n|            |               |       | Davis |       |       |  Youtube-Similar     ||      | Kubric-Similar| |\n|-----------------|---------------|:-------:|:-----------------:|:-------:|:-------:|:-------:|:----------------:|:------:|:------:|:------:|\n|                 |               | J&Fm ($\\uparrow$)  | Jm  ($\\uparrow$)              | Fm ($\\uparrow$)    | J&Fm ($\\uparrow$)  | Jm ($\\uparrow$)    | Fm ($\\uparrow$)             | J&Fm ($\\uparrow$) | Jm ($\\uparrow$)   | Fm ($\\uparrow$)   |\n| Self-supervised | SFC           | 71.2  | 68.3            | 74.0    | 55.5  | 55.3  | 55.7           | 47.7 | 43.1 | 52.3 |\n|                 | Spa-Then-Temp | 74.1  | 71.1            | 77.1  | 59.6  | 59.2  | 60.1           | 48.9 | 44.0   | 53.8 |\n|                 | DIFT          | 75.7  | 72.7            | 78.6  | 60.7  | 59.8  | 61.7           | 55.1 | 52.7 | 57.6 |\n|                 | Ours          | 77.6  | 74.4            | 80.8  | 66.0    | 65.1  | 67.0             | 90.2 | 87.2 | 93.1 |\n| Supervised      | STCN           | 85.3  | 82.0              | 88.6  | 63.9  | 62.6  | 65.2           | 60.5 | 56.8 | 64.1 |\n|                 | XMem          | 86.2  | 82.9            | 89.5  | 82.1  | 79.9  | 84.4           | 98.8 | 97.9 | 99.7 |\n|                 | Cutie          | 87.2  | 84.3            | 90.1  | 87.3  | 84.7  | 89.8           | 99.1 | 98.2 | 100    |\n|                 | SAM2     | 89.8 | 86.7           | 92.9 | 86.9 | 84.1 | 89.6          | 99.4 | 98.8 | 100    |\n\n\n### 5. Visualization of video examples\n\nThank you for your suggestions. We have prepared the video examples you suggested, showing that our approach captures consistent motion features and consistently outperforms baseline methods across different frames.  However, due to this year's NeurIPS policy that prohibits external links or appendix materials during the rebuttal stage, we are afraid that we are unable to present video examples at this time. We will include them in the revised version of the paper to better demonstrate the effectiveness of our approach.\n\n\n### 6. Implementation details about appearance features\n\nThank you for your suggestions. We will revise the paper to describe the appearance features in the main method section. Specifically, we will incorporate the explanation currently in lines 191–193 into the method section (Section 4.3) to be more clear."}, {"review_id": "dMCn6M3jS0", "reviewer": "Reviewer_ueSt", "summary": "This paper tackles the problem of video label propagation, which aims to propagate the given pixel labels in the first video frame to the subsequent frames. The authors explore using pre-trained video diffusion model and image diffusion model to extract motion and appearance features, where no additional training is needed. The experimental results show good performance.", "questions": "1. The features extracted from diffusion models are interpolated to match input image size. Without further training, can the proposed method handle objects with irregular or complex shapes?\n\nPlease see weaknesses 1-4. My major concern is with the experiment settings.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is easy to follow.\n2. The provided analysis is thorough and insightful.\n3. The proposed method shows good results on the compared benchmarks.\n\nWeaknesses:\n1. The proposed method shows good performance. Yet, many existing works have used pre-trained diffusion models for tracking and segmentation, e.g., [a,b,c]. Thus, it's not surprising that leveraging powerful video and image diffusion models can improve the results. However, as video label propagation is closely related to segmentation and tracking, it would be better to discuss and compare with similar works.\n2. The compared methods in Tab. 1 seem outdated; the authors did not compare with methods after 2023.\n3. The proposed pipeline involves multiple forward passes of video diffusion model and image diffusion model to perform the task. This would cause heavy overhead on computation and inference time, in my experience. Yet, there is no analysis on computation costs.\n4. The authors emphasized \"similar-looking objects\" in the task and introduced two datasets (Youtube-Similar and Kubric-Similar), where the proposed method shows the most significant improvement over the compared methods. How are the videos in the introduced datasets selected? What's the proposed method's performance on other standard benchmarks (e.g., JHMDB)?\n\n\n\n[a] Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation. CVPR 2025.\n[b] Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers. ECCV 2024.\n[c] VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models. CVPR 2025.", "was_revised": true, "final_justification": "After reading through the authors' response, most of my concerns and questions are answered. I'm inclined to accept this paper.", "rebuttal": "Thank you for the valuable feedback. We are glad that the reviewer finds **our analysis thorough and insightful, while the proposed method achieves good results**. We truly appreciate your time and effort in reviewing our work, and we address your concerns below. Please let us know if any further information is needed.\n\n### 1. Discussions on related work\n\nThank you for suggesting these papers[a,b,c]. We will cite and discuss them in the revised paper.  We highlight that **our work significantly differs from these studies[a,b,c] as follows**.\n\n\n* Track4Gen [a] tackles point tracking by **training video diffusion models on labeled point trajectories**, whereas our work explores pretrained video diffusion models for object segmentation **without any tracking‑specific training**.\n\n* Diff‑Tracker [b] is built on **image diffusion models** with additional motion encoders to learn temporal cues. By contrast, our work directly explores the **built‑in motion of pretrained video diffusion models** without extra modules.\n\n* VidSeg [c] performs **instance‑agnostic** video semantic segmentation that cannot distinguish different objects in the same category. It also requires maintaining and updating an additional KNN classifier to learn temporal changes during tracking. By contrast, our approach **can distinguish even similar‑looking objects** without extra components.\n\nApart from the above differences from [a,b,c], we highlight **our contributions go beyond performance improvements, with unique insights and findings which are not discussed in [a,b,c].**\n\n* Our work is the first to reveal that features from video diffusion models at early denoising stages can capture temporal motions.\n* Our work is the first to reveal that video diffusion models excel at tracking similar-looking objects without supervision.\n\nNotably, the studies [a,c] are published in June 2025 (CVPR 2025), after the NeurIPS 2025 submission due in May.  According to NeurIPS 2025 policy quoted as follows, *\"For the purpose of the reviewing process, papers that appeared online after March 1st, 2025 will generally be considered \"contemporaneous\" in the sense that the submission will not be rejected on the basis of the comparison to contemporaneous work.\"*  these studies[a,c] are contemporaneous work to our submission.\n\n[a] Track4Gen: Teaching Video Diffusion Models to Track Points Improves Video Generation. CVPR 2025. [b] Diff-Tracker: Text-to-Image Diffusion Models are Unsupervised Trackers. ECCV 2024. [c] VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models. CVPR 2025.\n\n\n### 2. Comparison with recent self‑supervised methods\n\nThank you for your comment. To the best of our knowledge, we have compared with the latest self-supervised baselines available at the time of submission. If the reviewer is aware of a relevant self-supervised method published recently, please let us know, and we would like to compare it. \n\n### 3. Clarification on computation cost\n\nThank you for your comment. We report the computation cost of our method in Section 5.3 (lines 290–293) and provide more details in Appendix B.4. Our method has 20% more computation time than generative model method DIFT on DAVIS videos and 89% than the popular discriminative model method SFC. We also discuss optimizing the computational efficiency of our method in Appendix B.4 and Appendix C.\n\n### 4. Implementation details and evaluation results on three more standard datasets and tasks \n\nThank you for your comment. We describe the selection process for the YouTube-Similar and Kubric-Similar datasets in Section 5.1 (lines 215–220). YouTube-Similar is manually created from the YouTube-VOS dataset by selecting videos that contain multiple objects with similar appearances. Kubric-Similar is a new dataset generated using the Kubric simulator, with randomized object colors, sizes, and motions. We include all generated videos as Kubric-Similar.  We will revise the paper for clarity.\n\n\n\nFollowing your suggestions, we conducted experiments for human pose tracking on the standard JHMDB dataset. We also evaluated on two additional standard datasets for video object segmentation: YouTube-VOS [1] and MOSE [2]. Experimental results in the following table demonstrate that **our method consistently improves tracking accuracy across diverse datasets and tracking tasks.** We will include these results in the revised paper.\n\n\n|          |  JHMDB  |    |   \n|---------------|:---------:|:---------:|\n|               | PCK@0.1 ($\\uparrow$) | PCK@0.2 ($\\uparrow$) | \n| SFC           | 61.9    | 83.0      |      \n| Spa-then-temp | 66.4    | 84.4    |      \n| DIFT          | 63.4    | 84.3    |      \n| **Ours**          | **68.3**    | **85.8**    |      \n\n\n\n|  |      | Youtube-VOS     |  |      |  MOSE    | |\n|-------------|:------:|:------:|:------:|:------:|:------:|:------:|\n|             | J&Fm ($\\uparrow$) | Jm ($\\uparrow$)   | Fm ($\\uparrow$)   | J&Fm ($\\uparrow$) | Jm ($\\uparrow$)   | Fm ($\\uparrow$)   |\n| DIFT        | 70.5 | 68.2 | 72.7 | 34.5 | 28.9 | 40.1 |\n| **Ours**        | **71.1** | **68.9** | **73.4** | **35.6** | **30.1** | **41.1** |\n\n\n[1] Ning Xu, Linjie Yang, et. al. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint arXiv:1809.03327, 2018.\n\n[2] Henghui Ding, Chang Liu, et.al. MOSE: A New Dataset for Video Object Segmentation in Complex Scenes. ICCV2023.\n\n### 5. Discussions on the interpolation of segmentation maps\n\nThank you for your comment. Following prior work, we first generate segmentation maps at the feature resolution and then interpolate these segmentation maps to the original image size. While this lower-resolution process may lose details for objects with irregular or complex shapes, it is a common challenge in existing methods. Our experiments on multiple datasets (e.g., Figure 5) empirically show that our approach is effective for tracking objects in real-world videos."}, {"review_id": "eRAzu65ezF", "reviewer": "Reviewer_aXR3", "summary": "This paper utilizes the pre-trained video diffusion model as a feature extractor for video frames, where each frame representation can be encoded with rich motion knowledge and appearance information with various noise levels. By combining the motion feature and appearance feature for each frame, the proposed method consistently improves label propagation for tracking on three benchmarks without supervised fine-tuning.", "questions": "1. Figure 1 is unclear. Please indicate the difference between the blue, green, and red boxes and the green and red masks in Figure 1.c in the caption.\n2. Please also refer to the weakness.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "### Strengths:\n\n1.\tThe proposed approach is novel and conceptually interesting. It leverages video diffusion models in a previously unexplored way for tracking. \n\n2.\tThe method achieves state-of-the-art performance without supervised training, demonstrating strong generalization capabilities.\n\n3.\tExtensive studies on different video and image diffusion models show the generalizability of the proposed method.\n\n### Weaknesses:\n1.\tAccording to the results shown in Figure 6.a, the appearance feature (R_f) plays a negative role in motion tracking, but combining it with the motion feature (R_m) can bring some benefit. Please provide more insights about why this happens. \n2.\tHigh computational cost. Despite the superior performance, the high computational cost of the diffusion model may limit the real-time tracking.", "was_revised": true, "final_justification": "This paper presents a novel and conceptually compelling use of video diffusion models for object tracking, achieving state-of-the-art performance without supervised training and demonstrating strong cross-backbone generalization. The rebuttal addressed my main concerns well. The authors clarified the role of motion versus appearance features: at higher noise levels, motion features dominate and are sufficient for successful tracking, while appearance features remain valuable for fine-grained segmentation. They also clarified Figure 1(c) and will revise the caption accordingly. Although computational cost is non-trivial, the authors provided quantitative evidence and a reasonable discussion. Given the methodological novelty and empirical gains, I view efficiency as an essential yet sufficient condition for future work, rather than a blocker. Overall, the paper makes clear technical contributions and provides insightful analysis that is likely to influence follow-up research on efficient diffusion-based tracking. I recommend acceptance.", "rebuttal": "Thank you for your positive review and insightful feedback. We are glad that you find **our approach novel and conceptually interesting, which also achieves state-of-the-art tracking performance with strong generalization capabilities**. We truly appreciate your time and effort in reviewing our work, and we address your concerns below. Please let us know if any further information is needed.\n\n### 1. Discussions on the appearance and motion features\n\nThank you for your comment. When tracking similar-looking objects, motion features are most useful for determining object identity, while appearance features provide fine-grained details for precise segmentation. Once identity is determined, the remaining task is to leverage those appearance cues to refine the object mask. Therefore, appearance features also bring some benefit after combining with motion features, when tracking similar-looking objects. We will include these discussions in the revised paper.\n\n\n### 2. Discussions on computation cost\n\nThank you for your comment. Our method has 20% more computation time than generative model method DIFT on DAVIS videos and 89% than the popular discriminative model method SFC. The significant improvement in tracking accuracy is worth this additional computational cost.  We also discuss optimizing computational efficiency in Appendix B.4 and Appendix C.\n\n\n### 3. Clarifying the visual elements in Figure 1\n\nThank you for your suggestions. In Figure 1(c), the green and red masks represent segmentation maps of different objects. The boxes are used to draw the reader’s attention to specific regions. The blue, green, and red boxes highlight the ground truth regions, correctly predicted regions, and incorrectly predicted regions, respectively. We will follow your suggestions to revise the caption of Figure 1 to be more clear."}]

Author rebuttal:

Meta-review: This paper addresses the problem of tracking objects with similar appearances. The authors leverage pre-trained video and image diffusion models to extract motion and appearance features without additional training. By combining these features, the method enables effective pixel-level label propagation across frames. Experiments demonstrate consistent improvements over existing approaches on multiple benchmarks.

All reviewers are generally positive about this paper. However, two minor concerns were raised after the rebuttal: The first concern is the lack of stronger theoretical justification for why features learned by pre-trained video diffusion models in the early denoising stage naturally capture motion dynamics. As clarified in the authors’ response, one of the main contributions of this work is to be the first to observe and empirically validate this property for tracking. While a theoretical derivation would further strengthen the paper, it is beyond the intended scope. The AC finds this acceptable. The second concern relates to the higher computational cost compared to other self-supervised approaches. However, as Reviewer TBTL noted, this overhead is offset by the clear accuracy improvements provided by the method. 

Based on the overall positive feedback and the minor nature of the remaining concerns, the AC recommends acceptance.
Decision: Accept (poster)