DINTR: Tracking via Diffusion-based Interpolation
Authors: Pha Nguyen
Ngan Hoang Le
Jackson Cothren
Alper Yilmaz
Khoa Luu
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual Tracking
Diffusion
Unification
Primary area: machine_vision
TL;DR: This work proposes a generative methodology to formulate the object tracking task and an interpolation operation as a faster approach to the diffusion mechanics.

Abstract: Object tracking is a fundamental task in computer vision, requiring the localization of objects of interest across video frames. Diffusion models have shown remarkable capabilities in visual generation, making them well-suited for addressing several requirements of the tracking problem. This work proposes a novel diffusion-based methodology to formulate the tracking task. Firstly, their conditional process allows for injecting indications of the target object into the generation process. Secondly, diffusion mechanics can be developed to inherently model temporal correspondences, enabling the reconstruction of actual frames in video. However, existing diffusion models rely on extensive and unnecessary mapping to a Gaussian noise domain, which can be replaced by a more efficient and stable interpolation process. Our proposed interpolation mechanism draws inspiration from classic image-processing techniques, offering a more interpretable, stable, and faster approach tailored specifically for the object tracking task. By leveraging the strengths of diffusion models while circumventing their limitations, our Diffusion-based INterpolation TrackeR (DINTR) presents a promising new paradigm and achieves a superior multiplicity on seven benchmarks across five indicator representations.

DINTR: Tracking via Diffusion- Interpolation
Pha Nguyen Ngan Le', Jackson Cothren's Alper Yilmaz', Khoa Luu'
Universityy of Arkansas "Ohoo State University
(panguyen, thile, jeothre, khoalmm/@arrk.en edu Pyilmaz. 1ewos50oocco edu
Zi-
26
ZT
Z
Zk -
ZO
(a)
Zi
2k-1
(b)
(c)
(d)
(e)
Point
Box
Segment
Text
Figure Diffusion- based processes. (a) Probabilistic diffusion process [1], where q(-) is noise sampling and po(-) is denoising. (b) Diffusion process in the 2D coordinate space [2, 3, 4]. (c) A purely visual diffusion-based data prediction approach reconstructs the subsequent video frame. (d) Our proposed data interpolation approach DINTR interpolates between two consecutive video frames, indexed by timestamp l, allowing a seamless temporal transition visual content understanding. temporal modeling, and instance extracting for the object tracking task across various indications (e).
Abstract
Object tracking is a fundamental task in computer vision, requiring the localization of objects of interest across video frames. Diffusion models have shown remarkable capabilities in visual generation, making them well-suited for addressing several requirements of the tracking problem. This work proposes a novel diffusion-bar methodology to formulate the tracking task. Firstly, their conditional process allows injecting indications of the target object into the generation process. Secondly, diffusion mechanics can be developed to inherently model temporal correspondences, cnabling the reconstruction actual frames in video. However, existing diffusion models rely extensive and unnecessary mapping to Gaussian noise domain, which can be replaced by a more efficient and stable interpolation process. Our proposed interpolation mechanism draws inspiration from classic image- image-processing techniques, offering a more interpretable, stable, and faster ap- proach tailored specifically for the object tracking task. By leveraging the strengths diffusion models while circumventing their limitations, our Diffusion-bar INterpolation TrackeR (DINTR) presents promising new paradigm and achieves a superior multiplicity on seven benchmarks across five indicator representations.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Object tracking is a long-standing computer vision task with widespread applications in video analysis and instance- based understanding. Over the past decades, numerous tracking paradigms have been explored, including sruking-by-rregee
[5], -eteccton [6], -segmentation [7] and two more recent tracking-by-o by-attention [8. 9], -unification [10] paradigms. Recently, generative modeling has achieved great success, offering several promising new perspectives in instance recognition. These include denoising sampling bounding boxes to final prediction [2, 3, 4], sampling future trajectories [1 1]. Although these studies explore the generative process in instance-based understanding tasks, they perform solely on coordinate refinement rather than performing on the visual domain, as in Fig. lb.
In this work, we propose a novel tracking framework solely based on visual iterative latent variables of diffusion models [12, 13], thereby introducing the novel and true Tracking-by Bracking-by-Difueeee by-Diffusinn paradigm. This paradigm demonstrates versatile applications across various indications, comprising points, bounding boxes, segments, and textual prompts, facilitated by the conditional mechanism (Eqn. (3)).
Moreover, our proposed Diffusion-based INterpolation TrackeR (DINTR) inherently models the temporal correspondences via the diffusion mechanics, i.e., the denoising process. Specifically, by formulating the process to operate temporal modeling online and ruto-regressively (i.e. next-frame reconstruction, as in Eqn. (4)), DINTR enables the capability for iustance-based video understanding tasks, specifically the object tracking. However, existing diffusion mechanics rely on an extensive and unnecessary mapping Gaussian noise domain, which argue can be replaced by more efficient interpolation process (Subsection 4.3). Our proposed interpolation operator draws inspiration from the image processing field, offering more direct, seamless, and stable approach. By leveraging the diffusion mechanics while circumventing their limitations, our DINTR achieves superior multiplicity seven benchmarks across five types of indication, as elaborated in Section 5. Note that Interpolation process does not aim to generate high-fidelity unseen frames [14, 15, 16, 17]. Instead, its objective is seamlessly transfer internal states between frames for visual semantic understanding.
Contributions. Overall, (i) this paper reformulates the Brucking-by-Diffussee paradigm to operate on visual domain (ii) which demonstrates broader tracking applications than existing paradigms. (iii) We reformulate the diffusion mechanics to achieve two goals, including (a) temporal modeling and (b) iterative interpolation 2 faster process. (iv) Our proposed DINTR achieves superior multiplicity and State-of- the-Art (SOTA) performances on seven tracking benchmarks of five representations. (v) Following sections including Appendices A elaborate on its formulations, properties, and evaluations.
Related Work
2.1 Object Tracking Paradigms
Tracking-by-Regree Regreesson methods refine future object positions directly based visual features. Previous approaches [31, 45] rely on the regression branch of object features in nearby regions. CenterTrack [5] represents objects via center points and temporal offsets. It lacks explicit object identity, requiring the appearance [31], motion model [46], and graph matching [47] components.
Tracking-by-Dettcction methods form object trajectories by linking detections over consecutive frames, treating the task as an optimization problem. Graph- based methods formulate the tracking problem as a bipartite matching maximum flow [48]. These methods utilize variety of techniques, such as link prediction [49], trainable graph neural networks [47, 34], edge lifting [50], weighted graph labeling [51], multi-cuts [52, 53], general-purpose solvers [54]. motion information [55], learned models [56], association graphs [S7], and distance-based [58, 59, 60]. Additionally, Ap- pearance- based methods leverage robust image recognition frameworks to track objects. These techniques depend on similarity measures derived from 3D appearance and pose [61], affinity esti- mation [62], detection candidate selection [62], learned re-identification features [63, 64], twin neural networks [65]. On the other hand, Motion modeling is leveraged for camera motion [66], observation-centn manner [67], trajectory forecasting [1]], the social force model [68, 69, 70, 71], based on constant velocity assumptions [72, 73], or location estimation [74, 68, 75] directly from trajectory sequences. Additionally, data-driven motion [76] need to project 3D into 2D motions [77].
leverages detailed pixel information and addresses the challenges of unclear backgrounds and crowded scenes. Methods include cost volumes [7], point cloud representa-

---

Table I: Comparison of paradigms, mechanisms of SOTA tracking methods. Indication Types
defines the representation to indicate targets with their corresponding datasets: TAP-Vid [18],
PraeTrack [19, 20J, MOT [21, 22, 23], VOS [24], VIS [25], MOTS [26], KITTI [27]. LaSOT [28],
GroOT [29]. Methods in color gradient support both types of single- and multi-target benchmarks.
Indication Types
Method
Paradigm
Mechanism
Point
Segment
Text
TAPIR [30]
Iter. Refinement Regression Head
TAP-Vid
Tracktor++ [3]]
MOT
X
Regression
CenterTrack [5]
Offsct Prediction
X
MOT
X
X
GTI [32]
Rgn-Ipl Integ.
X
LaSOT
X
LaSOT
DecpSORT [33]
Cascade Assoc.
X
MOT
X
X
GSDT [34]
Relation Graph
X
MOT
X
X
JDE [35]
Detection
Multi Task
MOT
X
ByteTrack [36]
Two-stage Assoc.
X
MOT
X
X
TrackR-CNN [37]
3D Convolution
X
MOTS
X
MOTSNel [26]
Mask- Powling
X
MOTS
X
Segmentation
CAMOT [38]
Hypothesis Select
X
KITTI
X
PointTrack [39]
Seg. Points
MOTS/KITTI
MixFormer [40]
Mixed Attention
TransVLT [41]
X-Modal Fusion
LaSOT
LaSOT
Attention
McMOTR [42]
Memory Aug.
X
MOT
X
X
MENDER [29]
Tensor Decomp.
MOT
X
GroOT
SiamMask [43]
Variant Head
X
LaSOT
VOS
X
TraDSS [7]
Cost Volume
VIS/MOTS
Unification
UNICORN
Unified Embed.
LaSOT/OT PoseTraak LaSOT/MOT
VOS/MOTS VOS/MOTS
X
UniTrack [44]
Primitive Level
X
DiffusionTrrak [3]
Denoised Coord.
MOT
X
X
DifMOT [4]
Diffusion
Motion Predictor
MOT
X
X
DINTR (Ours)
Visual Interpolat. TAP-Vid PoseTrack LaSOT/MOT VOS/MOTS LaSOT/GroOT
Ite: Iterative. Rgn-Tpl Integ.: Region-Template Integration. Assoc:: Association. X: Cross. Decomp.:
Decomposition. Embed.: Embedding. Coord.: 2D Coordinate. Motion: 2D Motion. Interpolat.: Interpolation.
tions [39], mask pooling layers [26], and mask-based [38] with 3D convolutions [37]. However, its reliance on segmented multiple object tracking data often necessitates bounding box initialization.
Fracking-by-Attention applies the attention mechanism [78] to link detections with tracks at the feature level, represented tokens. TrackFormer [8] approaches tracking unified prediction task using attention, during initiation. MOTR [9] and MOTRv2 [79] advance this concept by integrating motion and appearance models, aiding in managing object entrances/exits and temporal relations. Furthermore, object token representations can be enhanced via memory techniques, such as memory augmentation [42] and memory buffer [80, 81]. Recently, MENDER [29] presents another stride, transformer architecture with tensor decomposition to facilitate object tracking through descriptions.
Tracking-by- -a-ng-bb-Unittee -niffiiiatioon aims to develop unified frameworks that can handle multiple tasks simul- taneously. Pioneering works in this area include TraDeS [7] and SiamMask [43], which combine object tracking (SOT/MOT) and video segmentation (VOS/VIS). UniTrack [44] employs separate task-specific heads, enabling both object propagation and association across frames. Furthermore, UNICORN [10] investigates learning robust representations by consolidating from diverse datasets.
2.2 Diffusion Model in Semantic Understanding
Generative models have recently been found to be capable of performing understanding tasks.
Visual Representation and Correspondence. Hedlin et al. [82] establishes semantic visual corre- spondences by optimizing text embeddings to focus on specific regions. Diffusion Autoencoders [83]
form a diffusion- autoencoder encapsulating high-level semantic information. Similarly, Zhang
al. [84] combine features from Stable Diffusion (SD) and DINOv2 [85] models, effectively merging the high-quality spatial information and capitalizing on both strengths. Diffusion Hyperfeatures [86] uses feature aggregation and transforms intermediate feature maps from the diffusion process into single, coherent descriptor map. Concurrently, DIFT [87] simulates the forward diffusion process, adding noise to input images and extracting features within the U-Net. Asyrp [88] employs the
asymmetric reverse process to explore and manipulate semantic latent space, upholding the original

---

performance, integrity, and consistency. Furthermore, DRL [89] introduces an latent code that offers discretionary control over the granularity of detail.
Generative Perspectives in Object Tracking. A straightforward application of generative models in object tracking is augment and enrich training data [90, 91, 92]. For trajectory refinement, QuoVadis [11] uses the social generative adversarial network (GAN) [93] to sample future trajectories to account for the uncertainty future positions. DiffusionTrack [3] and DiftMOT [4] utilize the
diffusion process in the bounding box decoder. Specifically, they pad prior 2D coordinate bounding boxes with noise, then transform them into tracking results via denoising decoder.
2.3 Discussion
This subsection discusses the key aspects of our proposed paradigm and method, including the mech- anism comparison of our DINTR against alternative diffusion approaches [2, 3, 4], and the properties that enable Tracking-by) Brreking-by-Difeee by-Diffusion on visual domain to stand out from the existing paradigms.
Conditioning Mechanism. As illustrated in Fig. lb, tracking methods performing diffusion on the 2D coordinate space [3.4] utilize generative models to model 2D object motion or refine coordinate predictions. However, they fail to leverage the conditioning mechanism [13] of Latent Diffusion Models, which are principally capable of modeling unified conditional distributions. As a result, these cnfffusion-basct approaches have specified indicator representation limited to the bounding box, that cannot be expanded to other advanced indications, such as point, pose, segment, and text.
In contrast, we formulate the object tracking task as two visual processes, including one for diffusion- based Reconstruction, as illustrated in Fig. Ic, and another 2 faster approach that Interpolation. as shown in Fig. 1d. These two approaches demonstrate their superior versatility due to the controlled injection Po (z(rr) implemented by the attention mechanism [78] (Eqn. (3)) during iterative diffusion.
Current methods under tracking-by-uniffcen byuuuficattiinnn face challenges due to the separation of task-specific heads. This issue arises because single-object and multi-object tracking tasks are trained on distinct branches [7, 44] or stages [35], with results produced through a manually designed decoder for each task. The architectural discrepancies limit the full utilization of network capacity.
Unification.
In contrast, Tracking- Berkkinng----Dee Difffussion operating on the visual domain addresses the limitations of unification. Our method seamlessly handles diverse tracking objectives, including (a) point and pose regression, (b) bounding box and segmentation prediction, and (c) referring initialization, while remaining (d) data- and process-unified through iterative process. This is possible because our approach operates on the base core domain, allowing it to understand contexts and extract predictions.
Application Coverage presented in Table validates the unification advantages of our approach. As highlighted, our proposed model DINTR supports unified tracking across seven benchmarks of eight settings comprising five distinct categories of indication. It can handle both single-targed and multiple-targel benchmarks, setting a new standard in terms of multiplicity, flexibility, and novelty.
Problem Formulation
Given two images L, and I++1 from video sequence V, and an indicator representation L, (e.g.. point, structured points set for pose, bounding box, segment, or text) for an object in I,, our goal is to find the respective region Le+1 in I,+1, The relationship between L, and L,+1 can encode semantic correspondences [87, 86, 94] (i.e., different objects with similar semantic meanings), geometric correspondence [95, 96, 97] (i.e., the same object viewed from different viewpoints) temporal correspondence [98, 99, 100] (i.e., the location of a deforming object over a video sequence).
We define the object-tracking task as temporal correspondence, aiming to establish matches between regions representing the same real-world object as it moves, potentially deforming or occluding across the video sequence over time. Let denote a feature encoder E-) that takes input the frame I, and returns the feature representation z'. Along with the region L; for the initial indication, the online and auto-regressive objective for the tracking task can be written follows:
T arg min dist (E(T)) E (Te+))
where (..)(:;)) semantic distance that can be cosine [33] or distributional softmax [101]. A special case is to give L, textual input and return Le+1 as a bounding box for the referring object

---

tracking [29, 102] task. In addition, the pose is treated as multiple-pon point tracking. The output L++1 is then mapped to point, box, or segment. We explore how diffusion models can learn these temporal dynamics end-to-end to output consistent object representations frame-to-free to-frame in the next section.
4
Methodology
This section first presents the notations and background. Then, we present the deterministic frame reconstruction task for video modeling. Finally, our proposed framework DINTR is introduced.
4.1 Notations and Background
Latent Diffusion Models (LDMs) [1. 13, 103] are introduced to denoise the latent space of an autoencoder. First, the encoder E-) compresses a RGB image I into an initial latent space z = E(I),, which can be reconstructed to new image D D(z))- Let denote two operators Q and Pa c0
are corresponding to the sampling noise process ( (=) and the denoising process pa (2 /
where Pe, is parameterized by U-Net Eg [104] noise prediction model via the objective:
min Exj (1.1), [||e
Peu (Q k), 11
where = To(Le).
(2)
Localization. All types of localization Le, e.g., point, pose (i.e. set of structured points), bounding box, segment, and especially text, are unified as guided indicators. TØ(-) is the respective extractor, such as the Gaussian kernel for point, pooling layer for bounding box and segment, or word embedding model for text. z a noisy sample of zó at step à [1...., T). and = 50 is the maximum step.
The Conditional Process po (z)++1
(T), containing cross-attention Alln(e, 7) to inject the indication T to an autoencoder with U-Net blocks so(;,)) :), is derived after noise sampling z = 2(26, k):
A)
P=o (Q(z): k,) softmax (Ee(v 0g2 + vI and) x We (TX (TX x(r x Wv),
(3)
Vd
Atta(s.r)
where W....v are projection matrices, d is the feature size, and og is a scheduling parameter.
4.2 Deterministic Next-Frame Reconstruction by Data Prediction Model
The noise prediction model, defined in Eqn. (2). can not generate specific desired pixel content while denoising the latent feature to the new image. To effectively model and generate exactly the desired video content, we formulate next-frame reconstruction task, such that D(P:, (z4, T, T)) R I In this formulation, the denoised image obtained from the diffusion process should approximate the next frame in the video sequence. The objective for data prediction model (Fig. 1c) derives that goal
min E gitt RY [|
Pea (Q(zé, k), k 1
(4)
In layman's ' terms, the objective of the data prediction model formulates the task of establishing temporal correspon- dence between frames by effectively capturing the pixel-level changes and reconstructing the real next frame from the current frame. With the pre-trained decoder D(-) in place, the key optimiza- tion target becomes the denoising pro- cess itself. To achieve this, combina- tion of step-wise KL divergences is used to guide the likelihood of current frame latents z toward the desired latent representations for the next frame 2 described in Alg. and derived as:
Algorithm 1 Inplace Reconstruction Finetuning
Input: Network Eg, video sequence V. indication L,= 1: Sample (t, + ~ U(,,, | 22)
2: T + (L(
3: Draw I,,+1 E V and encode 1,1+1 E(T,,++)
4: Sample 1 ~ (1,
5: Optimize minş
2 ++1
P (Q(z). k), k. T 1|
6: Optimize ming [[] DDPPo (Q(z): k), k, 7) T)
1 Baaitt -- [1<] -P., (Q(2). k) J." dou
d Dx (9(2) 1+1 (ett))p. (2) (2k-1 126,)) das.
2
(5)
5

---

C3) interpolating
Mle
Algorithm Temporal Interpolation in DINTR
T' steps
Clean latents
z
20
Input: Network Øg, latent feature znj. T + (L))
1: Initialize Σ7 + z for h: [T.....0) do
denoising.
sampling T steps
T steps
zt11 + Poo (2k
k, (). then break 2 k) + 2( ; 1)
% f- 2
5: end for
Noisy latents
z
6: return
22 E (T.
Figure 2: Illustration of the reconstruction and interpolation processes, where the purple dashed arrow ++1 z)).
g(24// 2)) and the purple solid arrow Pe (2 (2) (2)-1 while the blue arrow illustrates pa p0
where Cg:: = This loss function constructed from the extensive step-wise divergences creates an accumulative path between the visual distributions. Instead, propose to employ the classic interpolation operator used image processing to formulate a new diffusion-basee process that iteratively learns to blend video frames. This interpolation approach ultimately converges towards the same deterministic mapping toward 2 but is simpler to derive and more stable. The proposed process is illustrated in Fig. 2, and interpolation operators are elaborated the next Subsection 4.3,
4.3 DINTR for Tracking via Diffusion-based Interpolation
Denoising Process as Temporal Interpolation. We relax the controlled Gaussian space projection of every step. Specifically, we impose temporal bias by training data interpolation model thg. The data interpolation process is denoted as Peo producing intermediate interpolated features 2 s++1 that Pbo (z). T)
so
Zu
-++1
ss zo [+1
The goal is obtain (z. +++ 25)) by optimizing the objective:
min E c.+ [||=]*
Pa, (zg. T, 1)
(6)
This data interpolation model de (Fig. Id) allows us to derive a straightforward single-step loss as:
L= DKL ++1 Po(z (26))
log Po (%)
The simplicity of the loss function comes from the knowledge that we are directly modeling the frame transition in the latent space, that is, 2t11 as z x+1 where A: E (T.....1) is not required. Therefore, we not use the noise sampling operator Q(-) as in the step-wise reconstruction objective defined in Eqn. (4). Instead, noise is added in the form of an offset, as described in L4 of Alg. 2. Note that the same network structure of En can be used for die without changing layers. Additionally, with the base case z
z/. the transition is accumulative within the inductive data interpolation itself:
te (T 1,...,,};
r++ P 00 2x+1
(2tht) kk, Poo (2)
((4)1 zi), 1,T))
(8)
2f41
Intespolation operatoe
Table 2: Equivalent formulation of interpolative operators, where z
1,+1 kk
Qz 1,1+1
[k 11).
linear blending
(b) learning from z 111
(c) learning from zí
(d) learning offset
k Oik-1
zi
+++ R
+t+1
z
2111 = 2 s++1
(1 - o22-1) 1 z
x
1-06-1
(
z/i)
(ax 02 (z z
I-ure
stable deterministic
unstable, when or 0
unstable, when a&
stable
nondeterm., missing x²
nondeterm. missing x
deterministic
nonaccumulative
accumulative, Eqn. (C.19) accumulative, Eqn. (C.21)
accumulative, Eqn. (8)

---

Interpolation Operator is selected based on the theoretical properties between the equivalent variants [105], presented in Table 2 and derived in Section C. In this table, we define Or 7he then the selected operator (2d), which adds noise in offset form @(2 A 1) Q(ab. k), is derived as:
x111 = 2 (at ax-11) (xi-1 zit) % k (1: - 1) (zi-1 zi).
(9)
x 26 (2k-1 2h)
26 2(zó, k) + Q 2+1 .. 1), as in L4 of Alg. 2.
(10)
Intuitively, the proposed interpolation process to generate the next frame takes the current frame as the starting point of the noisy sample. The internal states and intermediate features of the diffusion model transition from the current frame, resulting in a more stable prediction for video modeling.
Correspondence Extraction via Inter- nal States. From Eqn. (3), we demon- strate that the object of interest can be injected via the indication. From the ob- jectives in Eqn. (4) and Equ. (6). we show that the next frame I,+1 can be recon- structed or interpolated from the current frame I, Subsequently, internal accumu- lative and stable states, such the atten- tion map Attm(.,)) -). which exhibit spatial correlations, can be used to identify the target locations and can be effortlessly extracted. To get into that, the self- and cross cross-ateention maps (Ag, Ax) over N layers and T time steps averaged and performed element- multiplication:
Algorithm 3 Correspondence Extraction
Input: Internal Attn's while processing Poo 1: for k [0, T x 0.8 do
2: As.x += E Attmy,ay(s Attmmal (e. r)
3: end for
requires accumulativeness in Table 2 T0088
4: As,x t NxTx0.8 Σ As,x
5: A As Ax
6: L++1 <- map(A")
D as described in Eqn. (12)
7: return L,+1
As NXT
Att 1-1 Nm
Ax
Attmp,ale, r),
NXT
(11)
I-1 Rum
A As Ax,
A' E 10,1) where (H x W) is the size of I,+1-
Self-attention captures correlations among latent features, propagating the cross-attention to precise locations. Finally, as in Fig. le, different mappings produce desired prediction types:
arg ("""
point
L++1 map((a)) (min; 8, miny 3, max; 8, max, 3). = ((i.)) Ar, > 0}. if box
A' > 0,
if segment (12)
In summary, the entire diffusion- based tracking process involves the following steps. First, the indication of the object of interest at time is injected as condition by ps (2(2)) derived via Eqn. (3), Next, the video modeling process operates through the deterministic next-frame interpolation Pa(z)" 1+1 (z)), as described in Subsection 4.3. Finally, the extraction of the object of interest the next frame is performed via a so-called "reversed conditional process" Po (2 ( (r), outlined in Alg. 3.
Experimental Results
5.1 Benchmarks and Metries
TiP-Vid [18] formalizes the problem of long-term physical Point Tracking. It contains 31,951 points tracked on 1.219 real videos. Three evaluation metrics are Occlusion Accuracy (OA), 8 averaging position accuracy, and Jaccard @ 8 quantifying occlusion and position accuracies.
1 [20] is similar to MOT17 [22]. In addition to estimating Bounding Box for each person, the body Pose needs to be estimated. Both keypoint-basse based and standard MOTA [106], IDF1 [107], and HOTA [108] evaluate the tracking performance for every keypoint visibility and subject identity.
DAVIS [24] and MOTS [26] are included to quantify the Segmentation Tracking performance. For the single-target dataset, evaluation metrics are Jaccard index J. contour accuracy F and an overall J J&F score [24]. For the multiple-target dataset, MOTSA and MOTSP [26] are equivalent to MOTA and MOTP, where the association metric measures the mask IoU instead of the bounding box IoU,
7

---

Finally, LaSOT [28] and GroOT [29] evaluate the Referring Tracking performance. The Precision and Success metrics are measured on LaSOT, while GroOT follows the evaluation protocol of MOT.
5.2 Implementation Details
We fine-tune the Latent Diffusion Models [13] inplace, follow [109, 110]. However, different from offline fixed batch retraining, our fine-tuning is performed online and auto-regressively between consecutive frames when a new frame is received. Our development builds on LDM [13] for settings with textual prompts and ADM [111] for localization settings, initialized by their publicly available pre- rreitrained weights. The model is then tine-tuned using our proposed strategy for 500 steps with a learning rate of 10 10-5 The model is trained on NVIDIA Tesla A100 GPUs with a batch size of 1, comprising pair of frames. We average the attention As and Ax in the interval & E [0,T I x 0.8) For the first frame initialization, we employ YOLOX [112] as the detector, HRNet [113] as the pose estimator, and Mask2Former [114] as the segmentation model. We maintained linear noise scheduler across all experiments, as it is the default in all available implementations and directly dependent the number of diffusion steps, which is analyzed the next subsection. Details for handling multiple objects are in Section D.
of the DDIM steps with the total timestep
5.3 Ablation Study
Diffusion Steps. We systematically varied the number of diffusion steps (50, 100, 150, 200, 250) and analyzed their impact on performance and ef- ficiency. Results show that we can reconstruct an image close the ori- gin with timestep bound T 250 in the reconstruction process of DINTR.
Table 3: The timestep bound I' affects reconstruction quality.
T (steps)
50
100
150
200
250
MSE
20.5
15.4
10.3
5.2
0.04
J&Ft
75.4
75.8
76.0
76.3
76.5
Reconstruction time (s) 4
6.2
12.7
17.5
23.6
28.7
Interpolation time (s) 4 3.2
5.7
8.5
10.6
14.7
Alternative Approaches to the proposed DINTR modeling are discusses this subsection. To substantiate the discussions, we include all ablation studies Table comparing against our base setting. These alternative settings are different interpolation operators as theoretically analyzed in Table 2, and different temporal modeling, including the Reconstruction process as visualized in Fig. lc. Results demonstrate that our offset learning approach, which uses two anchor latents to deterministically guide the start and destination points, yields the best performance. This approach provides superior control over the interpolation process, resulting in more accurate and visually coherent output. For point tracking on TAP-Vid Vid, DINTR achieves the highest scores, with AJ values ranging from 57.8 to 85.5 across different datasets. In pose tracking on Pose'Track, DINTR scores 82.5 mAP, significantly higher than other methods. For bounding box tracking on LaSOT, DINTR achieves the highest 0.74 precision and 0.70 success rate with text versus 0.60 precision and 0.58 success rate without text. In segment tracking on VOS, DINTR scores 75.7 for J &FF, 72.7 for J. and 78.6 for F, consistently outperforming other methods.
Table 4: Ablation studies of different temporal modeling alternatives (the second sub-block) and interpolation operators (the third sub-block on point tracking (A), pose tracking (B), bounding box tracking with and without text (C), and segment tracking (D).
A. TAP-Vid
Kincties Rogg
Kubric K2ng
DAVIS
RGB
Stacking Gis
PoseTrack
mAP
MOTA
IDF1
HOTA
DINTR
82.5
64.9
71.5
55.5
DINTR
15788
72.5
[85.5
90.5
622.3
74.6
|65.2
77,5
(Ie) Recon.
77.8
55.8
65.5
50.5
Recon.
53.6
130.5
662.0
[62.3
(20) Linsor
27.6
34.8
1546
60.1
48.1
51.6
15.6
66.3
(2a) Linear
59.7
39.2
43.6
34.7
(2h) glts
34.1
43.3
64.9
63.9
51.6
34.8
59.7
60.3
(2h) x
69.1
43.6
55.1
40.7
(2ej m
63.3
62.0
51.4
53.9
58.6
(20)
68.5
43.0
C. LaSOT
Precision
Success
Precision
Success
VOS
J&F
J
DINTR
0.74
0.70
0.60
0.58
DINTR
75.7
72.7
78.6
(Ic) Recon.
0.66
0.64
0.52
0.50
(1c) Recon.
73.9
71.8
76.1
(2a) Linear
0.46
0.43
0.42
0.40
(2a) Linear
43.8
46.1
(2b) z
0.52
0.49
0.46
0.45
(2h) z
51.1
51.3
50.9
(2c) 2
0.51
0.48
0.44
0.44
(2c) xh
50.5
51.0
49.9
8

---

Table 5: Point tracking performance against several methods on TAP- Vid [18].
Kinetics [115]
Kubric [116]
DAVIS [24]
RGB Stackit [117]
TAP-Vid
AJ
OA
AJ
OA
AJ
6
QA
AJ
< 6
OA
COTR [118]
19.0
38.8
57.4
40.1
60.7
78.5
35.4
51.3
80.2
6.8
13.5
79.1
Kubric-VFS-Like Vike [116]
40.5
59.0
80.0
51.9
69.8
84.6
33.1
48.5
79.4
57.9
72.6
91.9
RAFT [119]
34.5
52.5
79.7
41.2
58.2
86.4
30.0
46.3
79.6
44.0
58.6
90.4
PIPs [120]
35.1
54.8
77.1
59.1
74.8
88.6
42.0
59.4
82.1
37.3
51.0
91.6
TAP-Net [18]
46.6
60.9
85.0
65.4
77.7
93.0
38.4
53.1
82.3
59.9
72.8
90.4
TAPIR [30]
57.1
70.0
87.6
84.3
91.8
95.8
59.8
72.3
87.6
66.2
77.4
93.3
DINTR
57.8
72.5
89.4
85.5
90.5
95.2
62.3
74.6
88.9
65.2
77.5
91.6
The reconstruction-basee method (Ic) generally ranks second in performance across tasks. The decrease in performance for reconstruction is expected, as does not transfer forward the final prediction to the next step. Instead, it reconstructs everything from raw noise at each step, as visualized in Fig. D.5. Although visual content can be well reconstructed, the lack of seamlessly transferred information between frames results in lower performance and reduced temporal coherence.
The performance difference between (2b) and (2c), which use single anchor either the starting latent point (z)) destination latent point (z) ()) respectively, is minimal. However, observed slightly higher effectiveness when controlling the destination point (2b) compared to the starting point (2b), suggesting that end-point guidance has a marginally stronger impact on overall inter- polation quality. Linear blending (2a) consistently shows the lowest performance. Derivations of
alternative operators blending (2a), learning from 2 (2b), learning from zó (2c), and learning offset (2d) are theoretically proved to be equivalent elaborated in Section C.
5.4 Comparisons to the Statc-of-the-Arts
Point Tracking. As presented in Table 5, our DINTR point model demonstrates competitive performance compared to prior works due to its thorough capture of local pixels and high-quality reconstruction of global context via the diffusion process. This results the best performance on DAVIS and Kinetics datasets (88.9 and 89.4 OA). TAPIR [30] extracts features around the estimations rather than the global context. PIPs [1 [120] and Tap-Net [18] lose flexibility by dividing the video into fixed segments. RAFT [119] cannot easily detect occlusions and makes accumulated errors due to per-frame tracking. COTR [118] struggles with moving objects as it operates on rigid scenes.
Pose Tracking. Table 6 compares DINTR against other pose-tracking methods. Classic tracking methods, such as CorrTrack [121] and Tracktor++ [31], form appearance features with limited descriptiveness on keypoint representation. We also include DiftPose [122], another diffusion- based performer on the specific keypoint estimation task. The primary metric in this setting is the average precision computed for each joint and then averaged over all joints obtain the final mAP. DiftPose [122] employs a similar diffusion based generative process but operates on a different heatmap domain, achieving similar performance the pixel domain of our interpolation process.
Bounding Box Tracking. Table 7 shows the performance of single object tracking using bounding boxes or textual initialization. Similarly, Table 8 presents the performance of MOT using bounding boxes (left), against DiffussionTrack [3] and DiffMOT [4] or textual initialization (right), against MENDER [29] and IDDER+TTrckKormee [129, 8J. Unlike DiffussionTTrack [3] and DiffMOT [4], which are limited to specific initialization types, our approach allows flexible indicative injection from any type, improving unification capability, and achieving comparable performance. Moreover,
Table 6: Pose tracking performance against several methods on PoscTrack21 [20].
Table 7:
Single object tracking without (left) and with (right) textual prompt input.
PoseTracc21
mAP
MOTA
IDF1
HOTA
LaSOT
Precision
Success Precision Success
Cor/Track [121]
72.3
66.5
51.1
SiamRPN+ [125]
0.50
01.55
Tracktor++ [31] w/ poses
71.4
63.3
69.3
52.2
GlobalTrack [126]
0.53
0,52
X
CorrTrack [121] ReID
72.7
63.8
66.5
52.7
OCEAN [127]
0.57
Tracktorr++ [31] w/ com:
73.6
61.6
69.3
54.1
UNICORN [10]
0.74
0.68
DCPose [123]
80.5
X
FAMI- Pose [124]
81.2
X
X
X
GT1
X
0.47
0.47
DifiPose [122]
83.0
X
X
X
AdaSwitcher [128]
X
0.55
0.51
DINTR
82.5
64.9
71.5
55.5
DINTR
0.74
0.70
0.60
0.58

---

Table 8:
Multiple object tracking without (left) and with (right) textual prompt input.
MOT17
MOTA
IDFI
HOTA
MT
MI.
IDs
GroOT
MOTA
IDF1
HOTA
AssA
DetA
MOIR [9]
73.4
68.6
57.8
42.9%
19.1%
2439
MDETR+IFm
62.6
64.7
51.5
50.9
52.2
TOa [130]
76.7
75.1
61.7
51.0%
16.4%
2346
MENDER [29]
65.5
63.4
53.2
52.9
53.7
UNICORN [10]
77.2
75.5
61.7
58.7%
11.2%
5379
DiffusionTraie
77.9
73.8
60.8
DINTR
68.9
68.5
57.5
56.9
58,2
DiNMOT [4]
79.8
79.3
64.5
(Ie) Reconstruct.
63.0
58.6
48.4
48.0
49.1
DINTR
78.0
77.6
63.5
54.2%
14.6%
4878
(2b) x
58.7
58.2
45.2
48.9
capturing global contexts via diffusion mechanics helps our model outperform MENDER and TrackFormer relying solely on spatial contexts formulated via transformer-based learnable queries.
segment tracking performance against unified methods [44, 10], single-targer methods [43, 131], I, and multiple-targe -trrpeeeetrrerteeee methods [37, 7, 8, 132]. Our DINTR achieves the best sMOTSS of 67.4, an accurate object tracking and segmentation. Unified methods perform the task separately, either using different branches [44] or stages [10]. leads to a discrepancy in networks. Our DINTR that is both data- and process-unffied avoids this shortcoming.
Segment Tracking. Finally, Table 9 presents
6 Conclusion
In conclusion, we have introduced Tracking-by-Diin by-Difussionn paradigm that reformulates the tracking framework based solely on visual iterative diffusion models. Unlike the existing denoising process, our DINTR offers a more seamless and faster approach to model temporal correspondences. This work has paved the way for efficient unified instance temporal modeling, especially object tracking.
Limitations. There is still a minor gap in performance to methods that incorporate motion models, e.g., DiffMOT [4] with 2D coordinate diffusion, illustrated in Fig. lb. However, our novel visual generative approach allows us to handle multiple representations in a unified manner rather than waste 5x efforts on designing specialized models. As our approach introduces innovations from feature representation perspective, comparisons with advancements stemming from heuristic optimizations, such as Byte [36], are head-to-heat as these are narrowly tailored increments for a specific type rather than paradigm shifts. However, exploring integrations between core representation and advancements offers promising performance. Specifically, final predictions are extracted by the so-called "reversed conditional process" Pg (z) (7) rather than sophisticated operations [133, 134]. Finally, time and resource consumption limit the practicality of Reconstruction. However, offline trackers continue play vital role in scenarios that demand comprehensive multimodality analysis.
Future Work & Broader Impacts. DINTR is a stepping stone towards more advanced and real-time visual Tracking- Bornkkingbb--iffie by-Diffusion in the future, especially to develop new tracking approach that can manipulate visual contents [135] via the diffusion process or a foundation object tracking model. Specific future directions include formulating diffusion-based tracking approaches for open vocabu- lary [136], geometric constraints [11], camera motion [66, 137, 95], temporal displacement [5], object object representation [61] and management [140]. The proposed video modeling approach can be exploited for unauthorized surveillance and monitoring, or manipulating instance-based video content that could be used to spread misinformation.
state [138], motion modeling [139, 6, 4],
Acknowledgment. This work is partly supported by NSF Data Science and Data Analytics that are Robust and Trusted (DART), USDA National Institute of Food and Agriculture (NIFA), and Arkansas Biosciences Institute (ABI) grants. We also acknowledge Trong-Thuar Nguyen for invaluable discussions and the Arkansas Brigh-Perforance Computing Center (AHPCC) for providing GPUs.
Table 9: Segment tracking performance on DAVIS [24] and MOTS [26].
VOS
J&F
J
F
MOTS
sMOTSA
IDFI
MT
ML
IDSw
SiamMask [43]
56.4
54.3
58.5
Track R-CNN [37]
40.6
42.4
38.7%
21.6%
567
Siam R-CNN [131]
70.6
66.1
75.0
TraDeS [7]
50.8
58.7
49.4%
18.3%
492
TruckFormer [8]
54.9
63.6
278
UniTrack [44]
58.4
PointTrackV2 [132]
62.3
42,9
56.7%
12.5%
541
UNICORN [10]
69.2
65.2
73.2
UNICORN [10]
65.3
65.9
64.9%
10.1%
398
DINTR
75.4
72.5
78.4
DINTR
67.4
66.4
66.5%
8.5%
484
10

---

References
[1] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models, Advances in neural information processing systems, 33:684006851, 2020. 1.5
[2] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19830-19843,
2023. 1, 2,
[3] Run Luo, Zikai Song, Lintao Ma. Jinlin Wei, Wei Yang, and Min Yang. Diffusiontrack:: Diffusion model for multi-object tracking. Proceedings of the AAAI Conference Artificial Intelligence, 2024, 1, 2,
9, 10
[4] Weiyi Lv, Yuhang Huang, Ning Zhang, Ruei-Sung Lin, Mei Han, and Dan Zeng. Diffimot: A real-time diffusion-basee multiple object tracker with non-linear prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024. 1, 3.4. 9, 10
[5] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking ohjects as points. In Proceedings of the European Conference on Computer Vision (ECCV), pages 474-490, 2020. 3, 10
[6] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In 2017 IEEE international conference on image processing (ICIP), pages 3645-3649 IEEE, 2017. 2, 10
[7] Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang. Ming Yang. and Junsong Yuan. Track detect and segment: An online multi-ohject tracker. In Proceedings of the IEEE/CVF conference conferecc on computer vision and pattern recognition, pages 123611, 2021. 2, 3, 4, 10
[8] Tim Meinhardt, Alexander Kirillov, Laura Leal- Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi- object tracking with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8844 8854, 2022, 2, 3,9, 10
[9] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang. Xiangyu Zhang, and Yichen Wei. Motr: End-to-end objett tracking with transformer. In European Conference on Computer Vision, pages 659-675. Springer, 2022. 2, 3,
[10] Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu. Towards grand unification of object tracking. In Luropean Conference on Computer Vision, pages 733-751. Springer, 2022. 3,9, 10
[11] Patrick Dendorfer, Vladimir Yugay, Aljoša OSep, and Laura Leal-Taixe Quo vadis: Is trajectory forecasting the key towards long-term multi-ohject tracking? Advances in Neural Information Processing Systems, 35, 2022. 2, 4, 10
[12] Jascha Sohl- -ick-iick,,,,n, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using noncquilibrium thermodynamics. In International conference on machine learning, pages 2256 2666-2265. PMLR, 2015.
[13] Robin Rombach, Audreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages (0684-10695, 2022. 2, 4. 5, 8, 21
[14] Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. Film: Frame interpolation for large motion. In European Conference on Computer Vision, pages 250-266.
Springer, 2022. 2
[15] Zhewei Huang. Tianyuan Zhang, Wen Heng. Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Computer Vision, pages 624-642. Springer, 2022. 2
[16] Siddhant Jain, Daniel Watson, Eric Tabellion, Aleksander Hotyński, Ben Poole, and Janne Kontkanen. Video interpolation with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2
[17) Zhen Li. Zuo -iann Zhu, Ling- Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All- pairs multi field transforms for efficient frame interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9801 9800-9 2023. 2
11

---

[18] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adrià Recasens, Lucas Smaira, Yusuf Aytar. Joāo Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benchmark for tracking any point in video. Advances Neural Information Processing Systems, 35: 13110016136,,, 2022. 3, 7.9
[19] Mykhaylo Andriluka, Umar lqbal, Eldar Insafutdinov, Leonid Pishchulin, Anton Milan, Juergen Gall, and Bernt Schiele. Posetrack: A benchmark for human pose estimation and tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5167-51766 2018. 3
[20] Andreas Doering, Di Chen, Shanshan Zhang. Bernt Schiele, and Juergen Gall. Posetrack21: A dataset for person search, multi-object tracking and multi-person pose tracking. In Proceedingy the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20963-20972, 2022. 3, 7.9
[21] Leal-TTaxé, A. Milan, I. Reid, S. Roth, and Schindler. MOTChallenge 2015: Towards benchmark for multi- target tracking. arXiv:1504.0144 [cs], April 2015. arXiv: 1504.01942. 3
[22] A. Milan, L. Leal-Taxxé, I. Reid, S. Roth, and K. Schindler. MOT16: A benchmark for multi-object tracking, arXiv: arXiv: 001660000083// [cs]. March 2016. arXiv: 1603.00831. 3,7
[23] Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20993-21002, 2022.
[24] Federico Perazzi, Jordi Pont- Pont-Tuset, Brian McWilliams, Lue Van Gool, Markus Gross, and Alexander Sorkinc-Hormun A benchmark dataset and evaluation methodology for video object segmentation. In Pruceedings of the IEEE conference computer vision and pattern recognition, pages 724-732, 2016. 3, 7,9, 10, 26, 27
[25] Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5188-5197, 2019. 3
[26] Lorenzo Porzi, Markus Hofinger, Idoia Ruiz, Joan Serrat, Samuel Rota Bulo, and Peter Kontschicder. Learning multi-object tracking and segmentation from automatic annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6846-6855, 2020. 3,7, 7,
[27] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEFE conference on computer vision and pattern recognition, pages 3354-3361. IEEF, 2012.
[28] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juchuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129:439-461 2021. 8,8
[29] Pha Nguyen, Kha Gia Quach, Kris Kitani, and Khoa Lun. Type-to-track: Retrieve any object via prompt-based tracking. Advances in Neural Information Processing Systems, 36, 2023. 3, 5, 9, 10
[30] Carl Doersch, Yi Yang, Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. ICCV, 2023. 3,9
[31] Philipp Bergmann. Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 941-951, 2019. 2,
[32] Zhengyuan Yang, Tushar Kumar, Tianlang Chen, Jingsong Su, and Jiebo Luo. trounding-tracking integration. IEEE Transactions on Circuits and Systems for Video Technology, 31(9):3433 3443, 2020. 3,
[33] Nicolai Wojke and Alex Bewley. Deep cosine metric learning for person re- identiicaation. In 2018 IEEE winter conference on applications of computer vision (WACV), pages 748-756. IEFF, 2018. 3,
[34] Yongxin Wang, Kris Kitani, and Xinshuo Weng. Joint object detection and multi-object tracking with graph neural networks. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pages 13708-13715, IEEE, 2021. 2,3
[35] Zhongdao Wang. Liang Zheng, Yixuan Liu, Yali Li, and Shengjin Wang. Towards real- sal-time multi-object tracking. In Proceedings of the European Conference on Computer Vision (ECCV), pages 107- 122. Springer, 2020. 3,4
12

---

[36] Yifu Zhang, Peize Sun. Yi Jiang, Dongdong Yu. Fucheng Weng. Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In Proceedings of the European Conference on Computer Vision (ECCV), 2022. 3, 10
[37] Paul Voigtlaender, Michael Krause, Aljosa Osep, Jonathon Luiten, Berin Balachandar Gnana Sekar, Andreas Geiger, and Bastian Leibe. Mots: Multi-ohject tracking and segmentation. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pages 7942-7951, 2019. 3, 10
[38] Aljoša OSep, Wolfgang Mehner, Paul Voigtlaender, and Bastian Leibe. Track, then decide: Category- agnostic vision- based multi-object tracking. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3494-3501. IEEE, 2018. 3
[39] Zhenbo Xu, Wei Zhang. Xiao Tan. Wei Yang. Huan Huang, Shilei Wen, Errui Ding, and Liusheng Huang, Segment as points for efficient online multi-object tracking and segmentation. In Computer ECcon-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pages (61. Springer, 2020."
[40] Yutao Cui, Tianhui Song, Gangshan Wu, and Limin Wang. Mixformerv2: Efficient fully transformer tracking. Advances in Neural Information Processing Systems, 36, 2024. 3
[41] Haojic Zhao, Xiao Wang, Dong Wang. Huchuan Lu, and Xiang Ruan. Transformer vision- -aaggul tracking via proxy loken guided cross-modal fusion. Pattern Recognition Letiers, 2023. 3
[42] Ruopeng Gao and Limin Wang. Memotr: Long-tern cemnory-augmentee transformer for multi-object tracking. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 9901 9910, 2023. 3
[43] Qiang Wang. Li Zhang. Luca Bertinetto, Weiming Hu, and Philip HS Torr. Fast online object tracking and segmentation: A unifying approach. In Proceedings of the IEEE/CVF conference Computer Vision and Pattern Recognition, pages 1328 1338, 2019. 10
[44] Zhongdao Wang, Hengshuang Zhao, Ya-Li Li, Shengjin Wang, Philip Torr, and Luca Bertinetto. Do different tracking tasks require different appearance models? Advances in Neural Information Processing Systems, 34:726-738, 2021. 3, 4, 10
[45] Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Detect to track and track to detect. In Proceedings of the IEFE international conference on computer vision, pages 3038-3046, 2017. 2
[46] Qiankun Liu, Qi Chu, Bin Liu, and Nenghai Yu. Gsm: Graph similarity model for multi-object tracking. In IJCAI, pages 530-536, 2020. 2
[47] Guillem Brasó and I. Laura I. Teie- Learning a neural solver for multiple object tracking. In Proceedings of the IEFE/CVF conference computer vision and pattern recognition, pages 6247-6257, 2020. 2
[48] Jerome Berelaz, Francois Fleuret, Engin Turetken, and Pascal Fua. Multiple object tracking using k- shortest paths optimization. IEEE transactions on pattern analysis and machine intelligence, 33(9):1806- 1819, 2.11.
[49] Kha Gia Quach, Pha Nguyen, Huu Le. Thanh-Dat Truong, Chi Nhan Duong, Minh-Triet Tran, and Khoa Luu. Dyglip: A dynamic graph model with link prediction for accurate multi-camera multiple object tracking, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13784-13 137933, 2021. 2
[50] Andrea Hornakova, Roberto Henschel, Bodo Rosenhahn, and Paul Swohoda. Lifted disjoint paths with application in multiple object tracking. In International Conference on Machine Learning, pages 4364 4375. PMLR, 2020.
[51] Roberto Henschel, Laura LealeTaixé, Daniel Cremers, and Bodo Rosenhahn. Improvements to frank- wolfe optimization for multi-detector multi-object tracking. arXiv preprint 17X 2017.
2
[52] Siyu Tang, Mykhaylo Andriluka, Bjoern Andres, and Bernt Schiele. Multiple people tracking by lifted multicut and person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3539-3548, 2017. 2
[53] Duy MH Nguyen, Roberto Henschel, Bodo Rosenhahn, Daniel Sonntag, and Paul Swoboda. Lmgp: Lifted multicut meets geometry projections for multi canttr- multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8866 8875, 2022. 2
13

---

[54] Qian Yu, Gérard Medioni. and Isaac Cohen. Multiple target tracking using spatio-temporal markov chain monte carlo data association. In 2007 IEEE Conference Computer Vision and Pattern Recognition, pages 1-8, IEEE, 2007.
[55] Margret Keuper, Siyu Tang. Bjoern Andres, Thomas Brox, and Bernt Schicle. Motion segmentation & multiple object tracking by correlation co-clustering. IEEE transactions on pattern analysis and machine intelligence, (2(1):140115: 2018. 2
[56] Chanho Kim, Fuxin Li, Arridhana Ciptadi, and James M Rehg. Multiple hypothesis tracking revisited. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 4696-4704, 2015.
2
[57] Hao Sheng, Yang Zhang, Jiahui Chen, Zhang Xiong, and Jun Zhang. Heterogeneous association graph fusion for target association in multiple object tracking. IEEE Trunsactions on Circuits and Systems for Video Technology, 29(1 1):3269 3280, 2018. 2
[58] Hao Jiang, Sidney Fels, and James Little. A linear programming approach for multiple object tracking. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pages 1-8. IEEE, 2007. 2
[59] Hamed Pirsiavash, Deva Ramanan, and Charless C Fowlkes. Globally-optime greedy algorithms for tracking a variable number of objects. In CVPR 10 pages 1201-1208. IEFE, 2.11. 2
[60] Li Zhang, Yuan Li, and Ramakant Nevatia. Global data association for multi-object tracking using network flows. In 2008 IEEE conference on computer vision and pattern recognition, pages 1-8. IEEE, 2008. 2
[61] Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, and Jitendra Malik. Tracking people by predicting 3d appearance, location and pose. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2740-2749, 2022. 2, 10
[62] Peng Chu and Haibin Ling. Famnet: Joint learning of feature, affinity and multi-diimnnsonal assignment for online multiple object tracking. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 6172-6181, 2019. 2
[63] Jiangmiao Pang, Linlu Qiu, Xia Li, Haofeng Chen, Li,, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 164-173, 2021. 2
[64] Ergys Ristani and Carlo Tomasi. Features for multi-target multi-camera tracking and re-identification. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 6036-6046, 2018.
2
[65] Laura Leal-Taixt Cristian Canton-Ferrer, and Konrad Schindler. Learning by tracking: Siamese enn for robust target association. In Proceedings of the IEFE conference on computer vision and pattern recognition workshops, pages 33-40, 2016. 2
[66] Nir Aharon, Roy Orfaig, and Ben-Zion Bobrovsky. Bot-sort: Robust associations multi-pedestrian tracking. arXiv preprint arXiv:2206. 14651, 2022. 2, 10
[67] Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, and Kris Kitani. Observation-centiic sort: Rethinking sort for robust multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9686 06666966,6770 2023. 2
[68] Laura -aixé, Gerard Pons-Moll, and Bodo Rosenhahn. Everybody needs somebody: Modeling social and grouping behavior on linear programming multiple people tracker. In 20// IEEE international conference on computer vision workshops (ICCV workshops), pages 120-127. IEEE, 2011. 2
[69] Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll Il never walk alone: Modeling social behavior for multi-target tracking. In 2009 IEEE 12th international conference computer vision pages 261-268. IEEE, 2009. 2
[70] Paul Scovanner and Marshall F Tappen. Learning pedestrian dynamics from the real world. In 2009 IEEE 12th International Conference on Computer Vision, pages 381-388. IEEE, 2009.
[71] Kota Yamaguchi. Alexander C Berg, Luis E Ortiz, and Tamara L Berg. Who are you with and where are you going? In CVPR 2011, pages 1345 1352. IEEE, 2011.
[72] Anton Andriyenko and Konrad Schindler. Multi-target tracking by continuous energy minimization. In CVPR 2011, pages
IEEE, 2011. 2
14

---

[73] Long Chen, Haizhou Ai, Zijie Zhuang, and Chong Shang. Real-time multiple people tracking with deeply learned candidate selection and person re-identification. In 2018 IEEE international conference on multimedia and expo (ICME), pages 1-6, IEEE, 2018.
[74] Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 961-971, 2016.
[75] Alexandre Robicquet, Amir Sadeghian, Alexandre Alahi, and Silvio Savarese. Learning social etiquette: Human trajectory prediction in crowded scenes. European Conference on Computer Vision (ECCV), volume 2, page 5, 2016. 2
[76] Laura Texie Michele Fenzi, Alina Kuznetsova, Bodo Rosenhahn, and Silvio Savarese. Learning an image-bute maage-based motion context for multiple people tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3542-35502 3549, 2014. 2
[77] Pavel Tokmakov, Jie Li, Wolfram Burgard, and Adrien Gaidon. Learning to track with object permanence. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10860-10869, 2021. 2
[78] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need, Advances in neural information processing systems, 30, 2017. 3,4
[79] Yuang Zhang, Tiancai Wang, and Xiangyu Zhang. Motrv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22056 22065, 2023. 3
[80] Jiarui Cai, Mingze Xu, Wei Li, Yuanjun Xiong, Wei Xia, Zhuowen Tu, and Stefano Soatto. Memot: Multi-object tracking with memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8090-8 1190-81011 8100, 2022. 3
[81] Xinyu Zhou, Pinxue Guo, Lingyi Hong, Jinglun Li, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. Reading relevant feature from global representation memory for visual object tracking. Advances in Neural Information Processing Systems, 36, 2024. 3
[82] Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, Andrea Tagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stable diffusion. Advances in Neurul Information Processing Systems, 36, 2023.
[83] Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Diffusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10619-10629, 2022. 3
[84] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances Neural Information Processing Systems, 36, 2023. 3
[85] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Mare Szafranicc, Vasil Khalidow, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaacldin El-Nouby. Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Yao Huang. Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. 3
[86] Grace Luo, Lisa Dunlap, Dong Huk Park, Aleksander Holynski, and Trevor Darrell, Diffusion hyperfea- tures: Scarching through time and space for semantic correspondence. In Advances in Neural Information Processing Systems, volume 36, 2023. 3,
[87] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. In Thirth-seventh Conference on Neural Information Processing Systems, 2023, 3.4 4
[88] Mingi Kwon, Jacseok Jeong, and Youngjung Uh. Diffusion models already have semantic latent space. In The Eleventh International Conference on Learning Representations, 2023.
[89] Sarthak Mittal, Korbinian Abstreiter, Stefan Bauer, Bernhard Schölkopf. and Arash Mehrjou. Diffusson based representation learning. In International Conference on Machine Learning, pages PMLR, 2023. 4
15

---

[90] Charan D Prakash and Lina J Karam. It gan do better: Can-based detection of objects on images with varying quality. IEEE Transactions Image Processing, 30:9220-9230. 2021. 4
[91] Pengxiang Li, Zhili Liu, Kai Chen, Lanqing Hong, Yunzhi Zhuge, Dan Yeung, Huchuan Lu, and Xu Jia. Trackdiffusion: Multi-object tracking data generation via diffusion models. arXiv preprint arXiv:2312. 00651, 2023. 4
[92] Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation. Advances in Neural Information Processing Systems, 36, 2024.
[93] Agrim Gupta, Justin Johnson, Li Fei-Fei, Silvio Savarese, and Alexandre Alahi, Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2255 2264, 2018.
[94] Koichi Namekata, Amirmojtaba Sabour, Sanja Fidler, and Seung Wook Kim. Emerdiff: Emerging pixel-level semantic knowledge in diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
[95] Pha Nguyen, Kha Gia Quach, Chi Nhan Duong, Son Lam Phung, Ngan Le, and Khoa Luu. Multi-camera multi-object tracking on the move via single-stage global association approach. Pattern Recognition, page 110457, 2024. 4, 10
[96] Pha Nguyen, Kha Gia Quach, John Cauch, Samee U Khan, Bhiksha Raj. and Khoa Luu. Utopia: Unconstrained tracking objects without preliminary examination via cross-domain adaptation. arXiv preprint 230600633, 2023, 4
[97] Thanh- Truong, Chi Nhan Duong, Ashley Dowling, Son Lam Phung, Jackson Cothren, and Khoa Luu. Crovia: Seeing drone scenes from car perspective via cross-view adaptation. arXiv preprint arXiv:2304.07119, 2023. 4
[98] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.0700 2023, 4
[99] Guillaume Moing, Jean Ponce, and Cordelia Schmid. Dense optical tracking: Connecting the dots. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2024.
[100] Yuxi Xiao, Qianqian Wang. Shangzhan Zhang, Nan Xue, Sida Peng. Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2024.
[101] Tobias Fischer. Thomas E Huang. Jiangmiao Pang, Linlu Qiu, Haofeng Chen, Trevor Darrell, and Fisher Yu. Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[102] Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, and Jianbing Shen. Referring multi-object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4633-14642, 2023. 5
[103] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[104] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assiste Intervention MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings. Part III 18. pages 234 241. Springer, 2015. 2,
[105] Eric Heitz, Laurent Belcour, and Thomas Chambon. Iterative a-(de) blending: A minimalist deterministic diffusion model. ACM SIGGRAPH 2023 Conference Proceedings, pages 1-8, 2023.
[106] Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metries. EURASIP Journal on Image and Video Processing, 2008:1-10, 2008. 7
[107] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and a data set for multi target, multi-camera tracking, In European conference on computer vision, pages 17-35. Springer, 2016.
[108] Jonathon Luiten, Aljosa Osep. Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal- Taixé, and Bastian Leibe. Hota: A higher order metric for evaluating multi-objed tracking. International journal of computer vision, 129:548- 578, 2021.
16

---

[109] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu. Yufei Shi. Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuiing of image diffusion models for lexl-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7623-7633, 2023. 8, 25
[110] Chenyang Qi. Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang. Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 8, 25
[111] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. Neural Information Processing Systems, 2021. 8
[112] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arxiv:2/07.00665
2021.
[113] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pages 5693-5703, 2019. 8
[114] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel elassification is not all you need for semantic segmentation. Advances Neural Information Processing Systems, 34:17864-117 17875, 2021.
[115] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kineties dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017. 9
[116] Klaus Greff, Francois Belletti, Lucas Beyer, Carl Doersch, Yilun Du, Daniel Duckworth, David Fleet, Dan Gnanapragasam, Florian Golemo, Charles Herrmann, et al. Kubric: A scalable dataset generator, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3749-3761, 2022. 9
[117] Alex X Lee, Coline Manon Devin, Yuxiang Zhou, Thomas Lampe, Konstantinos Bousmalis, Jost Tobias Springenberg, Arunkumar Byravan, Abbas Abdolmaleki, Nimrod Gileadi, David Khosid, et al. Beyond pick-and-place Tackling robotic stacking of diverse shapes. Conference on Robot Learning, pages 1089 1131. PMLR, 2022. 9
[118] Wei Jiang, Eduard Trulls, Jan Hosang, Andrea Tagliasacchi, and Kwang Moo Yi. Cotr: Correspondence transformer for matching across images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6207-6217, 2021. 9
[119] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16. pages 402 419. Springer, 2020. 9
[120] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. In European Conference Computer Vision, pages 59-75. Springer, 2022.
[121] Umer Rafi, Andreas Doering, Bastian Leihe, and Juergen Gall, Self-supervised keypoint correspondences for multi multi-person perttiipprsonnnn pose estimation and tracking in videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XX 16, pages 36-52. Springer, 2020. 9
[122] Runyang Feng, Yixing Gao, Tze Ho Elden Tse, Xueqing Ma, and Hyung Jin Chang. Diffpose: Spa- tiotemporal diffusion model for video-based human pose estimation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 14861-14872, 2023. 9
[123] Zhenguang Liu, Runyang Feng, Haoming Chen, Shuang Wu, Yixing Gao, Yunjun Gao, and Xiang Wang. Temporal feature alignment and mutual information maximization for video-based human posc estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1006-11016, 2022.
[124] Zhenguung Liu, Haoming Chen. Runyang Feng, Shuang Wu, Shouling Ji, Bailin Yang, and Xun Wang. Deep dual consecutive network for human pose estimation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 525-534, 2021. 9
[125] Bo Li. Junjie Yan, Wei Wu. Zheng Zhu, and Xiaolin Hu. High performance visual tracking with siamese region proposal network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8971-888888 8980, 2018. 9
17

---

[126] Lianghua Huang. Xin Zhao, and Kaiqi Huang, Globaltrack: A simple and strong baseline for long- term tracking. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 11037-11044, 2020.
[127] Zhipeng Zhang, Houwen Peng, Jianlong Fu, Bing Li, and Weiming Hu. Ocean: Object-aware anchor-free tracking. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020. Proceedings, Part XXI 16. pages 771-787. Springer, 2020. 9
[128] Xiao Wang, Xiujun Shu, Zhipeng Zhang. Bo Jiang, Yaowei Wang. Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 13763- 13773, 2021. 9
[129] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr modulated detection for end-to-ent to end mudti- understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1788 2021. 9
[130] Peng Chu, Jiang Wang, Quanzeng You, Haibin Ling, and Zicheng Liu. Transmot: Spatial-temporal graph transformer for multiple object tracking. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4870-4880, 2023. 10
[131] Paul Voigtlacnder, Jonathon Luiten, Philip HS Torr, and Bastian Leibe. Siam r-cnn; Visual tracking by re-detection. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 6378-6588, 2020. 10
[132] Zhenbo Xu, Wei Yang. Wei Zhang, Xiao Tan, Huan Huang. and Liusheng Huang. Segment as points for efficient and effective online multi-object tracking and segmentation. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 44(10):6424-644 6437, 2021. 10
[133] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end end object detection with transformers. In European conference on computer vision, pages 213 229. Springer, 2020. 10
[134] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming Hc, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 21 2117-2125, 2017. 10
[135] Uriel Singer. Adam Polyak. Thomas Hayes. Xi Yin. Jie An. Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-a-vides Text- to-video generation without text-vidco data. In The Eleventh International Conference on Learning Representations, 2023. 10
[136] Siyuan Li. Tobias Fischer, Lei Ke, Henghui Ding, Martin Danelljan, and Fisher Yu. Ovtrack: Open- vocabulary multiple object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5567-5577, 2023. 10
[137] Pha Nguyen, Kha Gia Quach, Chi Nhan Duong, Ngan Le, Xuan-Bac Nguyen, and Khoa Luu, Multi- camera multiple 3d object tracking on the move for autonomous vehicles. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 2569-2578, June
2022. 10
[138] ShiJie Sun, Naveed Akhtar, Xiang Yu Song, HuanSheng Song, Ajmal Mian, and Mubarak Shah. Simulta- neous detection and tracking with motion modelling for multiple object tracking. In Proceedings of the European Conference Computer Vision (ECCV), pages 626-643, 2020. 10
[139] Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Uperoft. Simple online and realtime tracking. In 2016 IEEE International Conference on Image Processing (ICIP), pages 2016. 10
IEEE,
[140] Daniel Stadler and Jurgen Beyerer. Improving multiple pedestrian tracking by track management and occlusion handling. In Proceedings of the IEED/CVF conference on computer vision and pattern recognition, pages 10958-1000 10967, 2021. 10
[141] Amir Hertz, Ron Mokady, Jay Tenenbaum, Křr Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to- prompt image editing with cross-attention control. In The Eleventh International Conference on Learning Representations, 2023. 21
[142] Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, and Jun Zhu. Your diffusion model is secretly certifiably robust classifier. arXiv preprint 240.22222222166 2024. 21
18

---

[143] Alexander C Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. In Proceedingy of the IEEE/CVF International Conference Computer Vision, pages 2206-2217, 2023. 21
19

---

Appendices
A Glossary
Table A.10: Notations used throughout the paper.
I
Current processing frame (image), I E "
I41
Next frame (image) in the processing video
Indicator representation in the current processing frame I, (e.g. point, bounding box, segment, texf)
Lt
Location the current processing frame It (e.g, point, bounding box, or segment)
Li+1
EII
Visual encoder extracting visual features
[Ld)
Pooled visual features of the current frame at the indicated location
D(20)
Visual decoder decoding latent feature to image
0
Network parameters
c
noise variable, c N/
r
Indicator representation
E0 (2(2
Denoising autoencoders, i.e., U-Net blocks Interpolation network, having the same structure as Eg
2 (Zx)
11.
L norm
20,
Zk:
2T
Latent variables of the noise sampling process Latent variables of the reconstructive interpolation process
20, ...
2k.
ZT
ank
The scheduling parameter
2(-)
Noise sampling process
(-)
Reconstruction/Demoiiing process, configured by Ee
(-)
Interpolation process, configured by de
To(-)
Indication feature extractor
Ee, I(-)
Expectation of a loss function L(.) with respect to
(kI(()
Kullback-Leibler divergence of and Q Conditional probability of 2 given x Conditional probability of denoising 2 given zi. configured by
(
Pe pe(2k-1121)
Po Po(2/_112h)
Conditional probability of interpolating 2 given Z configured by á
(Pso (-) + Poo (-)
Induction process
As
Average self-attention maps among visual features in U-Net
Ax
Average cross-attention maps among visual features U-Net
Element-wise product of self- and cross-attention
20

---

Q
KV
U-Net
cross aross-attention
skip connection
TØ
Pixel Space
Q
Q KV
Q
E
Segment
I
KV
KV
KV
Bounding
Zo
ZT
box
Inversion
Text
Point
D
0
Diffuser
Reconstruction
prompt
I
Latent Space
Conditions
ZO
2
ZT
Figure B.3: The conditional LDMs utilizes U-Net [104] blocks. First, clean image L is converted to a noisy latent Zy via the noise sampling process 2(-) (top branch). Then, well-structured regions are reconstructed from that extremely noisy input via the denoising/reconstruuctii branch). Additionally, conditions can be added as indicators of the regions interest. While the figure style is adapted from LDMs [13], we made distinct change reflecting the injected sampling process, following Prompt-to-Promp [141].
process Pe. (-) (bottom
B Overall Framework
Salient Representation. The ability of the diffuser to, first, convert clean image noisy latent, having no recognizable pattern from its origin, and then, reconstruct well-structured regions from extremely noisy inpul, indicates that the diffuser produces powerful semantic contexts [142, 143].
TØ
ZT
ZT
TØ
Z
Z
As
Atteico TV
Ax
Atenéso To
Reconstruct
Reconstruct
Altolee.rr
Aeniee
I
Zo
Zo
M
3
D
Figure B.4: Our proposed autoregressive framework constructed via the diffusion mechanics for temporal modeling. The current frame is input the encoder &(T,) to produce an initial latent 2g- The sampling process Q(-) adds noises into the latent in sequence of I' steps. Next, reconstruction process Peo (-) is manipulated through KL divergence optimization w.r.t. ++1 This shapes the reconstructed image I, be more similar to the future frame I++1. Finally, the location of the targets can be extracted by spatial correspondences, exhibited by the attention maps As and Ax.
21

---

In other words, the diffuser can embed semantic alignments, producing coherent predictions between two templates. To leverage this capability, we first consider the generated image I, in the diffusion process. Identifying correspondences on the pixel domain can achieved if::
dist ((T))) 0 is optimal from Eqn. (2), then dist (ELI))L), 0. (B.13)
We extract the latent features of their intermediate U-Net blocks at specific time step k during both processes. This is then utilized to establish injected correspondences between the input image I and the generated image I,-
Injected Condition. By incorporating conditional indicators into the Inversion process, we can guide the model focus on particular object of interest. This conditional input, represented as points, poses (i.e., structured set of points), segments, bounding boxes, or even textual prompts, acts as an indicator to inject the region of interest into the clean latent, which we want the model to recognize in the reconstructed latent.
These two remarks support the visual diffusion process in capturing and semantically manipulating features for representing and distinguishing objects, as illustrated in Fig. B.3. Additionally, Fig. B B.4 presents the autoregressive process that injects and extracts internal states identify the target regions holding the correspondence temporally.
C Derivations of Equivalent Interpolative Operators
This section derives the variant formulations introduced in Subsection 4.3.
C.1 Interpolated Samples
In the field of image processing, interpolated data point is defined weighted combination of known data points through a blending operation controlled by weighted parameter og:
& ag zij + (1 .ak 2 ++1
(C.14)
We can thus rewrite its known samples z,"" and zó in the following way:
20
2
2++
2k
O'' 2
(C.15)
ak
2k
(1
a2. 2
1+1
2
(C.16)
Otk
C.2 Linear Blending (2a)
In the vanilla version of the algorithm, blended sample of parameter og is obtained by blending 2f+1 and zij as similar as Eqn. (C.14):
c++ 1-1
aa-1 2/ + (1 (1-1) 2
(C.17)
To train our interpolation approach using this operator, because the accumulativeness property does not hold, then the step-wise loss as defined in Eqn. (5) has to be employed. As a result, this equivalent to the reconstruction approach Reconstruct. described Eqn. (4) and reported in Subsection 5.3.
22

---

C.3 Learning from z0" 1+1 (2b)
By expanding zi from Eqn. (C.17) using Eqn. (C.16), we obtain:
z 2+11 k-1 = (1 - ak-1) z + 0k-1 zy:
6k
x(+1
(1
(1-00))
20
1+1
(1 0k-1)) 1) % + 0k-1
Okk
Okk
0k-1
xK-1 (1 oxk)
z
1+1
+
01--1 2
akk
ak
Ogk 01--1
Ok-1 - (1rkk))
20
1+1 +
01
1
2k
Ok
O
0k-1
1+1 +
0k-1
ot+1
20
0kk
20
[+1
+
a--1
(2)+1
t+1)
(C.18)
Inductive Process. With the base case 2.7" data interpolation:
z/. the transition is accumulative within the inductive
k E {T - 1, .......,. 17.
Poul (2 +
11+1
1+1) ),,)),,TT) - Pou (20+1 + ax-1 (2(+1 (2t-1 - 2 k 1.r)).
x41
ax++1
ak
(C.19)
C.4 Learning from zi (2c)
By expanding ++1 from Eqn. (C.17) using Eqn. (C.15), we obtain:
*++ - (1 - -xx-1) 1)
20
[+1
+ Oa--- 1 zyy
2 1++1 1
a3 z/1
(1 1)-1)
+ aa-k-1 z/:
1 a3.
1- aakk
0z 1) 10kk
ak
1 0k-1
ak-1
2/ +
2
- Okk
1 Ckk
0k-1
(1 ax, (1 0kk11) akk
1 0kk-1
z/ +
2
- ax
1 - Cck
1 02 (1 ak-1)
- Ok-1 1111
z +
2x
1 Oik
1 0kK
ak-1
1- 0kk-
2 1t+1
z +
1- 0kk
1- akk
20
1 a--1
(11+1
(C.20)
ak
Inductive Process. With the base case 24,11 2/. the transition is accumulative within the inductive data interpolation:
A E (T - 1, 1}.
P (z)
1-
111
k)),k,T) - Por (2) +
1 - 0k-- c0441
(2)
zh), -1,7).
-ak41
1 0*k
(C.21)
23

---

Due to the absence of the deterministic property and the target term 20 2-11 the loss in Eqn. (7) becomes the sole objective guiding the learning process toward the target. Consequently, we prefer to perform the interpolation operator (2b) in Subsection 5.3, which is theoretically equivalent to this operator.
C.5 Learning Offset (2d)
By rewriting 0k-1 0k-1 + CJ Okk in the definition of z
k-1' obtain:
a(+1 R
(1 0; - z + ax- z
(1 a Or ag) v0t (okk- akk O1k) zú:
(1 a1)) % + ar zg + (axx-1 - ax) (z)- z 1+1)
(C.22)
Replace (1 ax)) 4+1 + Ore z by x/=²² from Eqn. (C.14), we obtain:
2 h-1
k + (ax-1 ax) (z)- 2 1
x++
= xxx + (crk - (k-1)
(z) (+1 x0)
(k:
$ +
(2) t-1 26))
(C.23)
T
By multiplying the step (z)+t - z5)) by a larger factor (e.g., T), the scaled step maintain their magnitude and not to become too small when propagated through many layers. Then we obtain:
2 x 21+1 + (z6+1 z
signified
(C.24)
x c + 1+1 - zi.
(C.25)
/ + (e (z ,
2 (z) k)), in L4 of Alg. 2.
(C.26)
Inductive Process. With the base case 2++1 z/. the transition is accumulative within the inductive data interpolation:
k E (T - 1,...,, 1};
Poo (2++1 + (2kt] 1);1); + Poo (+++1 + (2/ - zi), -1,
(C.27)
D Technical Details
Multiple- Margep Handling. Our method processes multiple object tracking by first concatenating all target representations into a joint input tensor during both the Inversion and Reconstruction passes through the diffusion model. Specifically, given M targets, indexed by i, each with indicator representation Li, we form the concatenated input:
T= T.. (T)())))) To (LT
(D.28)
where [. | is the concatenation operation.
This allows encoding interactions and contexts across all targets simultaneously while passing through the same encoder, decoder modules, and processes. After processing the concatenated output Pes (zi, T, T), we split it back into the individual target attention outputs using their original index order:
Ax AQ 1.. Ax E [0, MxHx
(D.29)
So each A' contains the refined cross-attention for target i after joint diffusion with the full set of targets. This approach allows the model to enable target-specific decoding. The indices linking inputs
24

---

to corresponding outputs are crucial for maintaining identity and predictions during the sequence of processing steps.
Textual Prompt Handling. This setting differs from the other four indicator types, where Lo comes from dedicated object detector. Instead, we leverage the unique capability of diffusion models to generate from text prompts [109, 110]. Specifically, we initialize Lo using a textual description as the conditioning input. From this textual Lo, our process generates an initial set of bounding box proposals as L1. These box proposals then propagate through the subsequent iterative processes to tracking outputs.
refine into the next L....., L ,
Pseudo-code for One-shot Training. Alg. D.4 and Alg. D.5 are the pseudo-code for our fine-tuning and operating algorithms in the proposed approach within the Tracking-by- beg---bf--Diiion paradigm, respectively. The pseudo-code provides an overview of the steps involved in our inplace fine-tuning.
Algorithm D.4 The one-shot fine- tne-tuning pipeline of Reconstruction process
Input: I, I+11 T [To(4]]] | T <- 50
1: Zo - E(I))
2: + E(4+1)
3: 2r + Q(zu, T") % injected Inversion
4: LELBO + (T) % lr
5: for .T..... 2) do
6: LELBO += KT. [xx -2, k) h; % 4-
7: end for
8: LELBO log (())) % 6g
9: Take gradient descent step on LELBO
Algorithm D.5 The tracker operation
Input: Video V, set tracklets + {Lg..... ..O MM11, 3 = 4, < 50
1: for trtt E {0, .0,..... [ 2} do
2:
Draw (I, 1,+1
[ ... || (I. MI] % 'T not change if Li is textual prompt
3:
4:
finetuning 1, T) % via Alg. D.4
5:
2T + P
6:
for (T. (T....,] do
7:
i E [1, x 0.8) then
8:
As += Σ E= Attn Eg)
9:
Ax Eis
Allmi,u Tg)
10:
end if
11:
2u + P(z T)
12:
end for
13:
As +
1 I As
NXT
Ax 1 IT Ax
NXI
15:
A + (As) 3 Ax
16:
[T0 [ M-D +- mapping(A") % via Eqn. (12)
T (14+1-- LM-1
18:
end for
Process Visualization. Fig. D.5 and Fig. D.6 are visualizing the two proposed diffusion- based processes that are utilized in our tracker framework.
25

---

T steps
T steps
2 T steps
Figure D.5: The visualization depicts the diffusion-based Reconstruction process on the DAVIS bench- mark [24]. Unlike the interpolation process in Fig. D.6, where internal states efficiently transferred between frames, the reconstruction process samples visual contents from extreme noise (middle column), and attention maps cannot be transferred. Although visual content can be reconstructed, the lack of seamlessly transferred information between frames results in lower performance and reduced temporal coherence as in Tables 6, 7, 8, and 9.
26

---

T steps
Figure D.6: Visualization of the diffussin-bbaed Interpolation process on the DAVIS benchmark [24]] Different from the reconstruction process in Fig. D.5, where each frame is processed independently. visual contents (top), internal states, and attention maps (bottom) are efficiently transferred from the previous frame to the next frame. This seamless transfer of information between frames results in more consistent and stable tracking, as the model can leverage temporal coherence.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Contributions in Section 1. Assumptions in diffusion models are clearly stated, including the conditional mechanism, and diffusion mechanics (i.e., the denoising process).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please see Limitations in Section 6.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
28

---

Answer: [NA]
This paper does not include pure theoretical results, but the equivalences or derivations of formulas are included.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please find the Subsection 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. .., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
29

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: The techniques presented in this work are the intellectual property [Affilia-
tion], and the organization intends to seek patent coverage for the disclosed process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ //nips. wwtpccccppccc.oec.cc..cccc.ccttt htapli//guiieeee gwwges/iee/////e Cooeeuumiis wwabbbbmsssoon sssissssssss Psloooolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please see Subsection 4.3 and Subsection 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
30

---

The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please Subsection 5.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https http://eeeurpppeeeee https//ccccpppttttecccceeeeeeeee
Answer: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., . if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Future Work & Broader Impacts in Section 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
31

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system Iearns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers that produced the code package or dataset cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's 's creators.
32

---

13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
33


Reviews:
[{"review_id": "S8kB9Cz593", "reviewer": "Reviewer_ddR4", "summary": "The paper \"DINTR: Tracking via Diffusion-based Interpolation\" introduces a novel approach for object tracking using diffusion models. The proposed methodology, Diffusion-based INterpolation TrackeR (DINTR), leverages diffusion mechanics to model temporal correspondences and reconstruct actual frames in video sequences. The authors claim that their interpolation mechanism offers a more interpretable, stable, and faster approach tailored specifically for the object tracking task, outperforming existing methods on several benchmarks.", "questions": "In Section 5.2, the model is described as being fine-tuned online. Is this a fair comparison with models that are trained offline?\nThe paper mentions the use of 4 NVIDIA Tesla A100 GPUs for training, which may not be accessible to all researchers. Discussing the computational requirements and potential optimizations for less powerful hardware would be useful.\nInclude more qualitative results, such as side-by-side comparisons of tracked objects using DINTR and other methods. Adding error analysis tables to show where and why DINTR performs better or worse would be useful.\nInclude a table that explicitly states the experimental setup for each method being compared. This should cover the datasets used, hardware specifications, and evaluation metrics.", "limitations": "The limitations have been briefly addressed in the paper", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The use of diffusion models for object tracking is a novel idea that has the potential to advance the field. The authors' proposal to replace the extensive mapping to a Gaussian noise domain with a more efficient interpolation process is compelling.\n2. DINTR supports multiple types of indication representations, including points, bounding boxes, segments, and textual prompts, making it versatile for different tracking tasks.\n3. The experimental results show that DINTR achieves superior performance on seven benchmarks across five different indication types. The method's ability to handle both single-target and multiple-target tracking tasks is impressive.\n4. The paper provides a thorough explanation of the proposed interpolation mechanism and its advantages over existing diffusion-based approaches. The inclusion of algorithm descriptions and detailed equations enhances the reproducibility of the work.\n5. The authors benchmark their method against several state-of-the-art tracking methods, demonstrating the superiority of DINTR in various scenarios. The use of multiple metrics (e.g., MOTA, IDF1, HOTA) provides a comprehensive evaluation of the method's performance.", "weaknesses": "1. In the Ablation Study (Section 5.3), the impact of different configurations is briefly mentioned, but detailed analysis and discussions on parameter sensitivity are lacking.\nRecommendation: Conduct comprehensive ablation studies, systematically varying key parameters such as the number of diffusion steps, noise levels, and interpolation techniques. Discuss how changes in these parameters affect the overall performance and stability of DINTR.\n2. While the authors claim near real-time performance, the actual feasibility of deploying DINTR in real-time applications is not thoroughly explored. More experiments focusing on real-time constraints and efficiency improvements could strengthen the paper.\n3. The paper does not mention the real-time image processing speed (i.e., frame rate) of the proposed tracking model, which is crucial for practical applications of tracking models.\n4. While the authors connect multiple tracking tasks using diffusion models, there is a lack of comprehensive comparison with the latest state-of-the-art models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's feedback regarding our ablation study, practicality, and comparison.\n\n#### 1. Ablation Study\nWe have significantly expanded our ablation studies, as detailed in the global response. This comprehensive analysis provides a deeper understanding of DINTR's behavior under various configurations. Key aspects of our expanded study include:\n- A systematic evaluation of diffusion steps and their impact on image reconstruction quality and computational efficiency.\n- A comparative analysis of different interpolation techniques, including our novel offset learning approach.\n- An exploration of various temporal modeling strategies and their relative merits.\nThis thorough investigation enhances the robustness of our findings and provides a solid foundation for further advancements in the field.\n\n#### 2. Feasibility\nWe have expanded our analysis to include Table I (subtables C and D) and Table II, which provide comprehensive data on processing speed and FPS for DINTR under various configurations. Specifically, our model can be flexibly trained offline (similar to Tune-A-Video [114]) for known video lengths or applied online for variable-length videos (as detailed in Implementation Details in the main paper). We have added a new scenario in Table I.C and I.D where our offline-trained model is distilled to a single-step inferencing diffusion model (T = 1). \n\nPlease note that, **offline trackers**, such as SUSHI [B], continue to play a vital role in scenarios where comprehensive analysis (**multimodality** in our case) is needed. Additionally, as demonstrated in our Supplementary video, our interpolation process is adaptable to any framerate (e.g., sampled 10x skip frame in the video), showcasing the flexibility of the diffusion process.\n\n#### 3. Further Practical Applications\nWhile real-time deployment is achievable as discussed above, we emphasize that DINTR's primary contribution lies in its robust theoretical framework for **autoregressive video modeling**, focusing on flexible instance-based understanding. This method offers significant potential **beyond just tracking**, opening new avenues for video applications based on these aspects:\n- Generative point and pose regression\n- Generative bounding box and segmentation prediction\n- Generative textual referring\n\nAdditionally, in Future Work, we elaborated on potential extensions of our framework to other instance-based understanding tasks, including visual content manipulation, visual motion modeling, temporal displacement analysis. DINTR is groundbreaking in its ability to bridge the gap between generative methods and fine-grained instance-based understanding tasks.\n\n#### 4. Comprehensive Comparison\nWe would like to emphasize the comprehensive nature of our comparisons in the main paper. Table 1 provides a feature and modality comparison with State-of-the-Art models across all paradigm categories, including tracking-by-regression, -detection, -segmentation, -attention, -unification.\n\nFor numerical comparisons, we included a wide range of approaches, extending ***beyond traditional tracking methods*** to encompass recent advancements in related fields, including PoseTrack21: FAMI-Pose, DiffPose (*Pose Estimation*, not tracking - CVPR 2023); LaSOT: methods with and without textual prompt input; MOT: diffusion-based approaches (AAAI 2023 and CVPR 2024) and unification approaches. This comprehensive comparison allows for a thorough evaluation relative to the latest SOTA across various tasks and methodologies. \n\nWe welcome the reviewer's suggestions for additional recent methods to include in our revision, further enhancing the depth of our comparative analysis.\n\n#### 5. Questions:\n5.1. Fairness of Comparison: Our fine-tuning approach is considerably fair to compare with offline-trained models, and may even be at a disadvantage for our model. Unlike existing methods that **explicitly learn to predict object location**, our model learns to reconstruct or interpolate real frames **without learning to predict direct location from training data**. Note that this work should be seen from a novel generative perspective, where apple-to-apple comparison to traditional methods could not be always fit.\n\n5.2. Computational Requirements: The computational resources mentioned in our paper represent the hardware available to us for conducting this research. The minimum requirement for running our model is a GPU with at least 10GB of VRAM.\n\n5.3. As we always strive to improve our paper quality, we will incorporate qualitative comparisons to strengthen the comparative analysis in the revision.\n\nWe believe these clarifications and additions will provide a more comprehensive understanding of our model's performance and requirements relative to existing approaches. We will include these clarifications to the revision.\n\n[B] Cetintas, Orcun, Guillem Brasó, and Laura Leal-Taixé. Unifying short and long-term tracking with graph hierarchies. In CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5cXR2uZS8f", "reviewer": "Reviewer_oiCF", "summary": "The paper introduces DINTR (Diffusion-based INterpolation Tracker), an object-tracking framework that uses diffusion models to perform tracking in the visual domain. It proposes a new \"Tracking-by-Diffusion\" paradigm that reformulates tracking based on visual iterative diffusion models. DINTR uses an interpolation approach instead of the standard denoising process, which is claimed to be faster and more stable. The method can handle multiple types of object representations (points, poses, bounding boxes, segments, text) in a unified manner.\nExperiments show competitive or state-of-the-art performance across several tracking benchmarks.", "questions": "Please refer to the weakness.", "limitations": "NO.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- DINTR can handle multiple types of object representations in a unified framework, which is flexible.\n- The method achieves competitive or state-of-the-art results on several benchmarks across different tracking tasks.", "weaknesses": "-  While faster than standard diffusion approaches, the method may still be computationally expensive compared to some traditional tracking methods. A more detailed description of the model size and speed should be provided.\n- The authors need to provide more experimental details. How is the training? What datasets are used during training? How are multiple tasks unified during training?\n- The paper lacks an overall framework, making it difficult to understand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in reviewing our paper. We hope to address your concerns by directing your attention to relevant sections where these details were already discussed in our original submission.\n\n#### 1. Model Size and Speed\nWe have included speed metrics in the rebuttal PDF for your reference. The model size is similar to LDM [13] and ADM [116] checkpoints as we initialize our weights from their public models. Please refer to our **Feasibility** discussion with Reviewer **ddR4**.\n\n#### 2. Experimental Details\nWe provided comprehensive experimental details in Section F of Appendices. This includes Algorithm F.4, which outlines our online training for the Reconstruction process, and Algorithm F.5, which details our tracker's operation. \n\nOur method diverges from traditional tracking approaches in its training and operation. ***We encourage viewing this work through a novel generative perspective, not similar to traditional trackers*** (ddR4 - 5. Questions). Our network captures and models video content, allowing conditioned instances in different modalities to be extracted seamlessly from this modeling process. Key points include:\n\n- No explicit object location training: Unlike conventional trackers, our approach doesn't require training on specific datasets or modalities to predict object locations.\n- Frame reconstruction focus: The model learns to reconstruct actual frames of the *testing video* through an autoregressive process (essentially, next frame prediction).\n- Flexible training options:\n  - Online fine-tuning: The model can be adjusted as a new frame of the *testing video* is received.\n  - Offline training: Similar to **offline** tracking methods like SUSHI [B], our model can be trained to capture the complete visual distribution of a video.\n  - Moreover, our model can also be distilled to operate single-step diffusion, as mentioned in our global response.\n - Generalization to different modalities (point, pose, box, segment, and text)\n\nUnification of multiple tasks: We utilize a unified diffusion approach similar to ControlNet [C]. Specific operations are used for each representation, such as Gaussian kernel for points, masking operation for bounding boxes and segments, word embedding model for text. Extracted representation will be passed to a unified attention layer to compute feature correspondence. We encourage the Reviewer refer to DIFT [87] or ControlNet [C] implementation for this operation.\n\n#### 3. Overall Framework\nWe illustrated our overall autoregressive framework in Figure B.4 in our original submission, which builds upon the conditional Diffusion process shown in Figure B.3. For your convenience, we have included these figures in the rebuttal PDF with detailed captions that describe the process thoroughly.\n\n#### 4. Limitations\nWe have addressed the limitations of our approach in the paper, as noted by Reviewers **ddR4** and **kgMr**.\n\n[B] Cetintas, Orcun, Guillem Brasó, and Laura Leal-Taixé. Unifying short and long-term tracking with graph hierarchies. In CVPR 2023.\n\n[C] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UYT1jx5TVs", "reviewer": "Reviewer_kgMr", "summary": "This paper applies the diffusion mechanism to the image interpolation process, which can realize object tracking tasks. Five object representation object tracking tasks, such as bbox, point, and text, are applied using diffusion-based interpolation. The benchmark experiments show promising results.", "questions": "See weakness", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1.\tThe diffusion mechanism applied in tracking tasks is novel, which may inspire others to apply generative methods to tracking. \n2.\tFive object representations in tracking tasks are applied, which shows the application ability of the method. \n3.\tExperiments are good and extensive, and results are promising.", "weaknesses": "1.\tThe tracking process, including the training and inference process, needs to be explained more clearly. \n2.\tSome results are still lagging behind the diffusion-based methods. For example, DiffPose in posetrack. DiffMOT in mot benchmark. \n3.\tMore ablation is needed. It lacks enough analysis of the diffusion scheme.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's feedback regarding our details and experimentation.\n\n#### 1. Process Details\nWe encourage the reviewer find our discussion about the Implementation Details with Reviewer **oiCF** and Section F in our Appendices. The training and inference processes are outlined in Algorithms F.4 and F.5. These details will be organized to include in the revision.\n\n#### 2. Generalization vs Specialization\nOur work focus is a **comprehensive generalization** across five representations and seven tracking benchmarks, as highlighted in your Strengths section. This level of grand unification is unprecedented, surpassing not only existing diffusion-based methods (DiffPose, DiffMOT), but also methods in other video understanding tasks. Other than that, we included DiffPose and DiffMOT because of the similarity in approach, but technically DiffPose is for Pose Estimation task (not tracking as ours) and DiffMOT models the motion on coordinate space.\n\n#### 3. Ablation Study\nWe have significantly expanded our ablation studies, as detailed in the global response. This comprehensive analysis provides a deeper understanding of DINTR's behavior under various configurations. Key aspects of our expanded study include:\n\n- A systematic evaluation of diffusion steps and their impact on image reconstruction quality and computational efficiency.\n- A comparative analysis of different interpolation techniques, including our novel offset learning approach.\n- An exploration of various temporal modeling strategies and their relative merits. This thorough investigation enhances the robustness of our findings and provides a solid foundation for further advancements in the field.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the reviewers' insightful comments and suggestions. The feedback highlights our paper's strengths, including its novel generative approach, the method's impressive versatility, thorough explanation for reproducibility, and comprehensive evaluation. Reviewers **ddR4** and **kgMr** lean towards acceptance (Weak Accept and Accept, respectively) due to the novelty and extensive experimentation. Reviewer **9rR3** assigned a Borderline Reject rating, primarily due to a perceived lack of specific details in our original submission. We will first address the common key point (KP) about additional Ablation Study, followed by individual responses to specific comments.

### (KP) Additional Ablation Studies: 
We appreciate the reviewers' feedback on our ablation study. In response, we have conducted more comprehensive ablation studies, as presented in Tables I and II of the rebuttal file:

#### 1. Diffusion Steps
We systematically varied the number of diffusion steps (1, 50, 100, 150, 200, 250) and analyzed their impact on performance and efficiency. Results show that with a timestep bound T = 250 in the reconstruction process, we can reconstruct an image extremely close to the original (per-pixel MSE of 0.04). In Table I.C and I.D, we added a new scenario where our offline-trained model is distilled to a single-step inferencing diffusion model (T = 1).

#### 2. Noise Scheduler
We maintained a linear noise scheduler across all experiments, as it is the default in all available implementations and directly dependent on the number of diffusion steps above.

#### 3. Interpolation Techniques
We compared four interpolation methods: Linear (2a), Two learning methods (2b & 2c), Our proposed offset technique (DINTR). Results demonstrate that our offset learning approach, which uses two anchor latents to deterministically guide the start and destination points, yields the best performance. This method provides superior control over the interpolation process, resulting in more accurate and visually coherent output. The performance difference between methods 2b and 2c, which use a single anchor at either the start or destination point respectively, is minimal. However, we observed slightly higher effectiveness when controlling the destination point (2c) compared to the starting point (2b), suggesting that end-point guidance has a marginally stronger impact on overall interpolation quality.

#### 4. Temporal Modeling
We evaluated three additional diffusion-based temporal modeling approaches: Pseudo-noise Latents (i), Inflated Self-Attention (ii), Semi-online Processing (iii). Their formulation, relative strengths, and weaknesses are discussed as follows:

**(i) Pseudo-noise Latents**

The real image $\\mathbf{I}\_{t}$ itself does not come from the training distribution of the U-Net $\\epsilon\_\\theta$. DIFT[87] proposed a straightforward approximation. Sampled noise respective to time step $k$ is directly ***added*** to the real image latent $\\mathbf{z}\_{k}$. Without temporal modeling, this process approximately moves the image into the noise distribution that the U-Net was learned to reconstruct without fine-tuning, formally presented as follows:

$$
 \\tilde{\\mathbf{z}}\_{k} = \\epsilon\_\\theta(\\mathbf{z}\_{0} + \\epsilon, k), \\text{where } \\epsilon \\sim \\mathcal{N}(0, 1)
$$

It enables extracting latent features even though the real image does not match the training distribution. However, this approach could only partially bridge the distribution shift. As a result, this approach performs the worst performance overall.

**(ii) Inflated Self-Attention**

Instead of the one-shot fine-tuning strategy, another approach can approximate this goal. To maintain the temporal coherence, VDMs [A] proposed to further extend the spatial 2D convolution layers and self-attention to the spatiotemporal domain. Specifically, the ***inflated*** self-attention is derived as:

$$
\\text{from } Attn \\Big(\\epsilon\_\\theta(\\mathbf{z}\_k, k), \\epsilon\_\\theta(\\mathbf{z}\_k, k)\\Big) \\text{ to } Attn \\Big(\\epsilon\_\\theta(\\mathbf{z}\_k, k), \\epsilon\_\\theta\\big([\\mathbf{z}\_k \\| \\mathbf{x}\_k], k\\big)\\Big)
$$

where $[\\cdot \\| \\cdot]$ is the concatenation operation, and the attention parameters only need to be reshaped **without fine-tuning pre-trained weights**. This solution is feasible for generating longer videos due to its flexibility.
However, similar to (i), the actual distribution is not well captured, resulting in lower performance.
Converting to this approach from our DINTR base leads to 5\% - 8\% performance drop as in Table I. This decrease is anticipated as target distributions cannot fully be incorporated into the reconstruction process.

**(iii) Semi-online Processing**

In addition to frame-by-frame operation, we extend (ii) to a clip-by-clip paradigm. Formally, given a video clip $v\_t \in \\mathbb{R}^{I \times H \times W \times 3}$, where $I$ is the ***fixed*** clip length (e.g. $I$ = 16), we pass it into the conditioned diffusion model. This semi-online approach extracts multiple frame features $\\mathbf{z}^{v\_t}\_k$ via the U-Net $\\epsilon\_\\theta$. Here, a sparse causal attention computes matrices between frame $\\mathbf{z}^{v\_t}\_k$ and two previous frames $\\mathbf{z}^{v\_0}\_k$ and $\\mathbf{z}^{v\_{t-1}}\_k$ as:

$$
\text{from } Attn\\Big(\\epsilon\_\\theta(\\mathbf{z}^{v\_t}\_k, k), \\epsilon\_\\theta(\\mathbf{z}^{v\_t}\_k, k)\Big) \text{ to } Attn\Big(\\epsilon\_\\theta(\\mathbf{z}^{v\_t}\_k, k), \\epsilon\_\\theta\\big([\\mathbf{z}^{v\_0}\_k \| \mathbf{z}^{v\_{t-1}}\_k], k\big)\Big)
$$

The outputs constitute $I$ trajectory predictions across the $I$ frames of the clip. This approach achieves mediocre performance, better than (i) but lower than (ii) because of the feature discrepancy between batches.

[A] Ho, Jonathan, et al. Video diffusion models. Advances in Neural Information Processing Systems 35, 2022.

Meta-review: The paper introduces a novel method for object tracking using diffusion models. The proposed method, DINTR, replaces traditional Gaussian noise mapping with a more efficient interpolation process, enhancing tracking accuracy and stability. The method's versatility is evident as it handles various object representations, including points, bounding boxes, segments, and textual prompts. Overall, DINTR is a technically solid paper with innovative contributions and promising results. Despite some areas for improvement, such as detailed ablation studies and real-time performance evaluation, the paper’s strengths outweigh the weaknesses. Given its novel approach, versatile application, and strong experimental results, the ACs recommend accepting this paper.
Decision: Accept (poster)