DeTrack: In-model Latent Denoising Learning for Visual Object Tracking
Authors: Xinyu Zhou
Jinglun Li
Lingyi Hong
Kaixun Jiang
Pinxue Guo
Weifeng Ge
Wenqiang Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: visual object tracking; denoising learning;in-model latent;
Primary area: deep_learning_architectures

Abstract: Previous visual object tracking methods employ image-feature regression models or coordinate autoregression models for bounding box prediction. Image-feature regression methods heavily depend on matching results and do not utilize positional prior, while the autoregressive approach can only be trained using bounding boxes available in the training set, potentially resulting in suboptimal performance during testing with unseen data. Inspired by the diffusion model, denoising learning enhances the model’s robustness to unseen data. Therefore, We introduce noise to bounding boxes, generating noisy boxes for training, thus enhancing model robustness on testing data. We propose a new paradigm to formulate the visual object tracking problem as a denoising learning process. However, tracking algorithms are usually asked to run in real-time, directly applying the diffusion model to object tracking would severely impair tracking speed. Therefore, we decompose the denoising learning process into every denoising block within a model, not by running the model multiple times, and thus we summarize the proposed paradigm as an in-model latent denoising learning process. Specifically, we propose a denoising Vision Transformer (ViT), which is composed of multiple denoising blocks. In the denoising block, template and search embeddings are projected into every denoising block as conditions. A denoising block is responsible for removing the noise in a predicted bounding box, and multiple stacked denoising blocks cooperate to accomplish the whole denoising process. Subsequently, we
utilize image features and trajectory information to refine the denoised bounding box. Besides, we also utilize trajectory memory and visual memory to improve tracking stability. Experimental results validate the effectiveness of our approach, achieving competitive performance on several challenging datasets. The proposed in-model latent denoising tracker achieve real-time speed, rendering denoising learning applicable in the visual object tracking community.

DeTrack: In-model Latent Denoising Learning for Visual Object Tracking
Xinyu Zhou'
Jinglun Li
Lingyi Hong'
Kaixun Jiang?
Pinxue Guo?
Weifeng Ge
Wenqiang Zhang1.2+
'Shanghai Key Lab of Intelligent Information Processing, School of Computer Science, Fudan University, Shanghai, China
Shhnggaai Engineering Research Center of AI & Robotics, Academy for Engineering and Technology. Fudan University, Shanghai, China zhouxinyu20@@ fudan.edu. edu.en jingli960423 @ gmmail.com, lyhong22 kxjiang22 @ m. Sindan.edu.co pxguo21 @ m. m.fudan.ec. edu.cn, f..uua..edu.cc, weifeng.ge ge. gmail.com. wqzhang @ fudan. edu.cn
fdd..n.du.cnn
Abstract
Previous visual object tracking methods employ image-feature regression models or coordinate autoregression models for bounding box prediction. Image-feature regression methods heavily depend matching results and do not utilize positional prior, while the autoregressive approach can only be trained using bounding boxes available in the training set, potentially resulting in suboptimal performance during testing with unseen data. Inspired by the diffusion model, denoising learning enhances the model's robustness to unseen data. Therefore, We introduce noise to bounding boxes, generating noisy boxes for training, thus enhancing model robustness testing data. We propose new paradigm to formulate the visual object tracking problem as a denoising learning process. However, tracking algo- rithms are usually asked to object tracking would severely impair tracking speed. Therefore, we decom- pose the denoising learning process into every denoising block within model, not by running the model multiple times, and thus we summarize the proposed paradigm as an in-model latent denoising learning process. Specifically, we propose a denoising Vision Transformer (ViT), which is composed of multiple denoising blocks. the denoising block, template and search embeddings are projected into every denoising block as conditions. A denoising block is responsible for removing the noise in a predicted bounding box, and multiple stacked denoising blocks cooperate to accomplish the whole denoising process. Subsequently, we utilize image features and trajectory information to refine the denoised bounding box. Besides, we also utilize trajectory memory and visual memory to improve tracking stability. Experimental results validate the effectiveness of our approach, achieving competitive performance several challenging datasets. The proposed in-model latent denoising tracker achieve real-time speed, rendering denoising learning applicable in the visual object tracking community.
in real-time, directly applying the diffusion model
"corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Noisy Imagg
Impur Image
Templates
Sourch Region
Noisy box
Extraction
Enonder
Repeat
Nuisy hux Ai
Denoising
Repeat
Decoder
Repeat
Mapping
In-model L
Denoising
Nuisy Imagee,
box Denolsing Leaminng in Object Detection
Box;
Desolsing Learning image Generation
The Proposed Denoistag Leaminng Tracking Paradigm
Figure I: Difference of denoising learning paradigm. (a) Diffusion model in image generation task. (b) Diffusion model in object detection task. (c) The proposed In-model latent denoising learning paradigm. The pink box indicates the denoising module. x indicates denoising for N times.
Introduction
Visual object tracking is a fundamental task in computer vision, which involves localizing and tracking specific object in video given its initial position. It finds broad applications in video understanding, surveillance, and robot navigation [45] ofate-of-thh-ar approaches can be broadly categorized into two classes. The first class [17 51 41
8 [9 [5] 29 [56] directly predicts the
bounding box of the tracked target based on image features. The second class 6 43 employs the coordinate autoregression framework.
While the mainstream methods have achieved prominent success, there are still certain issues to be addressed. Methods based on the image-feature regression framework rely heavily on the matching results between the template and the search region, which cannot utilize positional prior. Meanwhile, based on the autoregressive approach, it is necessary to utilize the bounding box from the previous frame train the model, which can only utilize the existing bounding boxes in the training set. Therefore, during the testing phase, it may exhibit suboptimal performance for some unseen data.
The Diffusion model has achieved significant success in image generation task, allowing the genera- tion of many images not seen in the training set 3 Inspired by the Diffusion model[66) we add noise bounding boxes during training stage. The noisy box can have arbitrary size and position, thereby enhancing the robustness of the model to unseen data during testing stage. As illustrated in Fig I (a) and (b), the diffusion model in image generation task requires multiple iterations of U-Net, while in object detection task, denoising is accomplished through multiple iterations of the decoder. However, tracking algorithms are usually asked run in real-time, directly applying the diffusion model to object tracking would severely impair tracking speed.
Therefore, motivated by 4AE[40, we propose novel denoising learning paradigm (DeTrack) for visual object tracking that decomposes the denoising learning process in every denoising block with tracking model. We use templates, search region, and noisy boxes as inputs. During the denoising process, we inject the template and search region conditions to predict the noises in previous predictions. We repeatedly conduct the conditional denoising process and finally achieve accurate object location prediction. Specifically, as shown in [c) we propose a novel denoising ViT. We decompose a complete denoising process into several denoising blocks within VIT model and implement every denoising operation with a denoising block. Then the denosing learning process can be implemented in a single forward pass the tracking model, which can reduce the computational
cost drastically. To benefit from the in-context information [21 [6 9 28 20 22). also put the previously predicted bounding boxes into a trajectory memory, and put the templates from previous frame into visual memory. We use them additional conditions to help locate objects more accurately.
Our contributions can be summarized as follows:

---

We propose novel in- model latent denoising learning paradigm for visual object tracking, which provides a new perspective for the research community. decomposes the classical explicit denosing process into several denoising blocks and solves the problem with tracking network in a single forward pass, which valuable for real applications.
We present a tracking model including a denoising ViT, comprised of multiple denoising blocks. The denoising process can be completed by progressively denoising through the denoising blocks within ViT. Furthermore, we construct a compound memory the model that improve the tracking results using visual features and trajectory.
Experimental results on several popular experiments, including AVisT, GOT-1OK, LaSOT, and LaSOT ezts demonstrate that the proposed method achieve competitive results.
2 Related Work
Visual Object Tracking. The existing visual object tracking methods can be broadly categorized into two main classes. The first class [ [1] 48]
46 8 19
41
32
31
51
10
35
44 [5]] involves directly
regressing the bounding box from image features, second class 43 treats the bounding box as
four distinct tokens, employing an autoregressive model to sequentially predict these four tokens.
In the first class, deep neural networks initially used extract visual features, followed by the design of various prediction heads for regressing the bounding box. Since 2016, some prevalent methods have adopted a two-stream framework, employing siamese networks separately extract visual features from the template and the search region. One type of prediction head 46
32] [3
53 uses branch to predict the possible location of the target and other branches to predict the
corresponding bounding box for that location. Another type of prediction head [10] 48 17 consists of two branches that predict the coordinates of the top-left and bottom-right corners. Subsequently, OSTrack [5I) introduces a one-stream tracking paradigm that combines feature extraction and feature fusion into single step, achieving new wate-of-the-an performance. For the second class, SeqTrack 66] proposes transforming the bounding box into four tokens, predicting them sequentially in the order of x, w, and h. When predicting the bounding box, each box requires four passes through the decoder. Another autoregressive method, ARTrack [43] is similar to SeqTrack but differs in that incorporates trajectory information in the input to enhance the model's awareness of trajectories.
Denoising Learning. DDPM [26] introduces denoising diffusion learning, which enhances the quality and diversity of generated images by adding noise to and denoising images. Subsequently, denoising learning has experienced explosive growth, being applied in various domains and achieving significant success. the super-Resolution field, SR3 [37] leverages DDPM for conditional image generation, employing stochastic denoising process for super-resolution Meanwhile, CDM [27 comprises sequence of multiple diffusion models, each responsible for generating images with progressively higher resolutions. In video generation, the Flexible Diffusion Model (FDM) [23
utilizes a generative model designed for sampling arbitrary subsets of video frames, facilitated by specialized architecture tailored for this purpose. The Residual Video Diffusion (RVD) model
(50)
employs an autoregressive, end-to-end optimized video diffusion model. In addition to generative tasks, denoising learning has also found extensive applications in discriminative task. DiffusionDet
4 applics the diffusion model to object detection, utilizing DDIM (38 for denoising. However, this
approach still requires multiple passes through the decoder for denoising, impacting inference speed.
Method
In this section, we start by formulating the proposed tracking paradigm learned through denoising learning (Section[..] Next, we present our overall model architecture (Section 3.2). which includes a proposed denoising ViT, box refining and mapping module, and compound memory.
3.1 How to Formulate the Denoising Learning Tracking Paradigm?
Image and Box Inputs. We utilize both visual memory and the search region as conditional inputs while introducing noisy boxes x, to predict the true position of the target, where represents the I-th state in the denoising process. The visual memory stores templates, which are cropped based on
3

---

previous frames. The search region is cropped based on the current frame and encompasses the area where the target may be present. In training stage, inspired by DDPM[66, we obtain noisy boxes x, by adding Gaussian noise to the ground truth box Xo:
X1 Vaxi evi- a E I0,
where ã = I1 Jij and a; = 1 8z. 83 E (0, is the variance schedule, T' is the time step.
Optimization for Denoising Learning. We take the visual memory and search region as conditional inputs c, and predict the true target position from the noisy box X₁, Px (x), where & represents the neural network parameters. We aim to maximize the probability pe that the neural network predicts Xo. enabling the model to predict the true target position:
maximize (pe (xol x,,x)),,)) c)).
(2)
To maximize po, we need make the predicted Xg by the network fo close the ground truth xy:
X
fo(c,x1),
(3)
minmize|x, -xol-
Decomposes the Denoising Process into Multiple Demoising Block within Model. According to the principle of Markov, we can expand Equation into a Markov chain:
pa x1, c) p(xx) II pa (x--x (xx, p(xr) II po (x (xx,,
(4)
i=1
In the traditional Diffusion model each step
c) is iteratively predicted using a neural network model Sa. However, our denoising paradigm decomposes the iterations of neural network into the iterations of denosing blocks within a neural network, Jo = dd,,d2,- ... di ). where each denoising block di is responsible for predicting state po (x Xi, c), where I denotes the number of model.
blocks. This allows our model to complete denoising with only a single forward pass of the tracking
Discussion on the Differences from the Diffusion Model. The proposed denoising learning tracking paradigm is not a diffusion model. (1) In the reverse denoising process of diffusion model, sampling noise from standard Gaussian distribution introduces randomness, making it more suitable for generating diverse images in image generation tasks. However, bounding boxes for visual object tracking are deterministic. Therefore, our proposed DeTrack does not involve sampling process in reverse denoising process, making it more suitable for visual object tracking. (2) Each step of diffusion model is predicted recursively using a neural network. The proposed DeTrack predicts states using denosing blocks within network (3) The diffusion model requires iterative prediction of neural network, whereas our method only requires a single forward pass through the network. Please refer to the Appendix A. for detailed analysis.
3.2 Model Architecture
Inputs representation. As show in Fig. we use noisy bounding boxes as input and take visual memory and a search region as conditional inputs. Visual memory stores multiple templates. Specif- ically, gaussian noise added to the ground truth bounding box to obtain noisy bounding box {z]", vy"" "v]","" -- v E R where denotes noise addition, while and 2 respectively denote the upper left corner and lower right corner. Subsequently, the noisy box mapped to a high-dimensional space by word embedding, resulting in noisy box embedding x, IR I Additionally, we map tem- plates and the search region to templates embedding E TR. NCC and search embedding s E IR N.C by image embedding, where N₂- n x He W., N, H. W. denotes the number of templates, H and W represent the height and width of the image, respectively. For details, please refer Model implement details in Section 4.1
16 16
16 TØ
Denoising VIT (In-model Latent Denoising).
ViT Transformer Block. The specific transformer block structure is the same as the VIT transformer block [3]. Therefore, we only introduce integrating the features of templates and search region within the ViT block. Specifically, we perform attention image embedding. We first obtain qa (search

---

Inputs Representation
Denoksing VIT (In-mode Latınt Denoksing)
Bux Refining and Mapping
Denoising Block
VT TornConn
WT TreeRhonn
Trjjectcyy Memery
NoIeY Rot
Sparch Benson Eraboniiig
Ls
Mapping
Denociing
A Earetion
Search Rogion
Demikling
Denclsad Box craboddnng
Denoseed Box
FFN
NoisePooi
Nolay
Cagtion
Novis
Search Unppe
Subtract
Mig +
Sumple Gassorih
of of
Selrees Mappon
Depclied Box Firtbeiiing
(8)
(b)
The overview of model architecture. (a) The model architecture comprises the input representation, the proposed Denoising ViT, and Box Refnnnng and Mapping. It also includes Visual Memory and Trajectory Memory. (b) The proposed Denoising Block within Denoising VIT.
Figure 2:
query), q. (templates query), k, (search key), ky (templates key), v, (search value) and 02 (templatee value) through linear layer. The image attention is employed to interact and fuse image embedding:
[ss 4z [2 (i.is, kiz]
Amtention//agec2.. s) Softmax
[e.. Uz]),
Vd
where |-] denotes concatenation. d is the dimensionality of the key.
Denoising Block. As shown Fig. 2 the input to the denoising block comprises the noisy box embedding and the search region embedding. These are passed through linear layers to obtain the 9x. (box query), ki (search key), and u2 (search value). Subsequently, denoising attention mechanism is employed for the first time of denoising:
Attention (8, x.) = Softmax Ix. ki
U.).
(6)
Vd
Then, incorporate Feedforward Neural Network (FFN) layer to enhance x/:
Attention Demoising x,) + x,-
(7)
FFN(X)).
Finally, we use two linear layers to predict noise for the second time of denoising, Subtracting the noise from the box embedding yields the result after denoising through NoisePred module:
NoisePred x LiinearReLLuLinnn
(9)
x
E.
Denoising is performed through Denoising blocks. Ultimately, denoising is accomplished with single forward pass the denosing ViT, resulting denoised box embedding Xo:
Xo = X
Σ 1=1
(10)
Box Refining and Mapping. As shown in Fi) we start by applying self-attention to trajectory and denoised box embedding. We maintain that the current box embedding can only attend to its preceding box embedding by an attention mask in the self-attention, introducing temporal information. Subsequently, the output of self-attention is used query for cross-attention with the

---

Trijettoy Marsey
Sourch Embooding
Box Emboddine
tart Aon ... W.
Replace
Ward Eribeddig
Refined Enbedding
Sinillaity map
louNet
pass
Box Refinnn
Bax Rednning
Regloce?
Bention Xaak
Crsse-ataanion
Mampiea
Embodanng
Image Embosdingg
Templatee
Dynanic Templates
New Template
Box refining and Mapcing
Visual memory
Figure 3: Box refining and mapping and the updating of visual memory. (a) Box refining and mapping introduces the trajectory memory to improve tracking performance, (b) Visual memory updating based on collaboratively decision including (loU score) and (Softmax score).
image features. After undergoing six times box refining, we compute the similarity between the refined box and word embedding, apply Softmax to obtain probabilities for different positions in the word embedding, and use the position with the highest probability as bounding box, which is similar to ARTrack(43]
Compound Memory. We design a compound memory that includes both a visual memory and trajectory memory. The visual memory enhances the model's ability to adapt to changes in the appearance of the target and the environment in the video. Besides, the trajectory memory enables the model to continue tracking the target even in the presence of occlusions or disappearances.
Visual Memory. As shown in Fig b)), visual memory consists of dynamic templates and fixed template. The first template dynamic templates is discarded, and a new template is added. Directly updating the template can lead cumulative errors. Therefore, we propose collaborative updating mechanism. This involves inputting the search embedding extracted after Denoising ViT into loUNet obtain the corresponding loU score 81. Additionally, the Softmax as confidence value A collaborative decision on the quality of the new template frame is made based on two threshold values di and 02, determining whether updating.
from Box Refining serves
Trajectory Memory. The proposed trajectory memory stores the boxes of the previous 7 frames, using first-in-irst-out (FIFO) approach when updated trajectory box used for refining model with prior positional information and target size, allowing accurate prediction of the bounding box even in cases of visual occlusion.
box needs to be stored. This results in continuously the denoised box. The trajectory memory can provide the
4 Experiments
4.1
Implementation Details
Model implement details. We design two variants of DeTrack with different resolutions as shown in
Tab
Table 1: The Floating-Poin Point Operations per Second(FLOPss) and speed of the model variants.
Template Size
Search Region Size
Model
Flops
Speed
Device
DeTrackass
128 x 128
256 x 256
53.0G
42FPS
RTX3090
DeTracks84
192 x 192
384 x 384
117.1G
30FPS
RTX3090
Our denoising ViT adopts ViT-B [3] and utilizes MAE for weight initialization, with a total of denoising blocks. The box refining includes 6 transformer layers for self-attention and cross-attention. Additionally, we trained two models, namely De DeTracke5s and DeTrack The template cropped based on twice the size of the bounding box, while the search region is cropped based on four times (DeTrackose) and five times (DeTrackges) the size of the bounding box. To map

---

Table 2: ofate-of-the-arr comparison AViST 36
GOT-1Ok 330 LaSOT [4 and LaSOT card
15
Where denotes our model only trained on GOT- 10k. The best results are highlighted in bold.
AVisT
GOT 10k*
LaSOT
LaSOT
Method
AUC
OP50
OP75
AO
SRo.5
SRg.70 75
AUC
P Norr
ALC
PNoom
SiamPRN++255
31
39.0
43.5
21.2
51.7
61.6
32.5
49.6
56.9
49.1
34.0
41.6
39.6
DsMP2ss
[
61.1
71.7
49.2
56.9
65.0
56.7
39.2
47.6
45.1
ATOMgss
22.2
51.5
50.5
PrDiMP2ss
P2
43.3
48.0
28.7
63,4
73.8
54.3
59.8
68.8
60.8
Occañ225 53
38.9
43.6
20.5
61.1
72.1
47.3
56.0
65.1
56.6
Alpha 2e
38.2
65.3
TransT256
0
49.0
56.4
37.2
67.1
76.8
60.9
64.9
73.8
69.0
TaMP
34
51.9
59.5
38.9
67.6
78.0
72.2
45.9
DATT250
TATrckkse
24
73.0
83.3
68.5
68.1
77.2
72.2
CTTrcckase
39
56.3
66.1
44.8
71.3
80.7
70.3
67.8
77.8
74.0
TMTa52
33.8
63.9
KeepTrackgog
35
49.4
56.3
37.8
67.1
77.2
70.2
48.2
STARK
+
51.1
59.2
39.1
68.8
78.1
64.1
67.1
77.0
AiATracks2o
Li
69.6
80.0
63.2
55.6
Mixformerya
10
56.5
66.3
45.1
70.7
80.0
67.8
69.2
78.7
74.7
OSTrack2se
S1
54.2
63.2
42.2
71.0
80.4
68.2
69.1
78.7
75.2
47.4
$1.3
53.3
OSTrackss4
[
57.7
67.3
48.3
73.7
70.8
71.1
81.1
77.6
61.3
57.6
SwinTrack224
X3
71.3
64.5
67.2
70.8
47.6
53.9
Swin'Trakkge
72.4
80.5
67.8
71.3
76.5
49.1
$5.6
ROMTrack256
57.8
67.6
48.6
72.9
82.9
70,2
69.3
78.8
75.6
ROMTmckas4
39.1
68.7
50.5
74.2
84.3
72.4
71.4
81.4
78.2
BDMTrackg36
49
72.7
82.0
69.9
69.9
79.4
75.8
47.9
$7.9
$4.0
F-BDMIrackgs1
75.4
84.3
72.9
72.0
81.5
77.7
50.8
61.3
57.8
GRM256
18
54.5
63.1
45.2
73.4
70.4
69.9
79.3
75.8
GRMg20
88
55.2
64.2
46.8
73.4
82,9
70.5
69.9
79.3
75.8
SeqTrackoss
6
56,8
66.8
45.6
74.7
84.7
71.8
69.9
79.7
76.3
49.5
60.8
SegTrackss4
57.8
67.4
48.0
72.2
71.5
81.8
77.8
61.6
ARTrackaoe
3
73.5
82,2
70.9
70.4
79.5
76.6
46.4
56.5
52.3
AKTrackys4
75.5
84.3
74.3
72.6
81.7
79.1
51.9
62.0
58.5
DeTrackgno (ours)
60.1
69.7
50.6
77.1
86.1
73.5
71.3
80.1
76.8
47.9
56.6
52.1
DeTrack3es (ours)
60.2
69.1
50.2
77.9
86.5
74.9
72.9
81.7
79.1
53.6
64.4
60.4
boxes into high- nigh-dimensional space, utilize word embedding, similar to Pix2Seq (5), with the number bins being 800 and 1200 for DeTrack256 and De DeTrack384 respectively.
Training, Our experiments are conducted on Intel( Xeon(R) Gold 6326 CPU @ 2.90GHz with 252GB RAM and NVIDIA GeForce RTX 3090 GPUs with 24GB memory. In the first stage, there is only visual memory, which randomly samples two frames from the video. The model is trained on full dataests (COCO, GOT-1Ok. TrackingNet. and LaSOT). total 240 epochs are trained, with the learning rate set to 8e-5 for the denoising ViT and 8e-6 for the box refining. The learning rate decreases by a factor of 10 the 192-th epoch. In the second stage, trajectory memory is introduced to refine the box, and sequential training is adopted. Consecutive frames are sampled from the video, with each frame's prediction result stored the trajectory memory and updated a fot-in--irst-out manner. The training is conducted three datasets excluding COCO. A total of 60 epochs trained, with the learning rates decreasing to 4e-6 and 4e-7 for the denoising VIT and box refining, respectively. In the third stage, only IoUNet is trained while other parts are frozen. The learning rate set to le-4, and total of epochs are trained, with a 10x learning rate decay at the 30-th epoch. For GOT- 10k, the learning rate remains consistent with training the full dataests. In the first stage, we train for 120 epochs, with a 10x decrease learning rate at the 96-th epoch, followed by training for 25 epochs in the second stage. During the training GOT-10k, IoUNet is not used. The loss functions is cross-entropy and SloU, which is the same as ARTrack 45]
Inference. During the testing phase, we the search region and template image inputs and initialize the box with the previous box (predicted bounding box of t-1 frame). Additionally, the update interval of the visual memory is set to 5 for <= 100, doubled every 100 frames until = 500, and then remains 160. While testing on the GOT- 10k dataset, the visual memory is updated directly. For other datasets. the IoU score and confidence score is applied to filter templates. The trajectory memory stores seven bounding boxes, updating with a frequency every frame. Inference conducted on an NVIDIA GeForee RTX 3090.
7

---

4.2 State-of-the-Art Comparisons
AViST, The AVisT dataset, as described in [36]. covers a broad spectrum diverse and demanding situations, encompassing harsh weather conditions like thick fog, intense rainfall, and sandstorms. Our tracker demonstrates outstanding performance on AViST [36.. a dataset with extreme weather conditions and harsh environments. It outperforms Seq Track3s4 by 2.4% in AUC, substantiating our tracker's excellence in extreme envirommental conditions.
GOT-10k. GOT-10k comprises a training dataset consisting of 10,000 videos and testing dataset with 180 videos. There no overlap between the training and test sets, necessitating trackers to demonstrate robust generalization capabilities towards unseen data. As shown in Tab. 2 our method demonstrates superior performance on the GOT- 10k [30]. Our DeTrack256 achieves a significant improvement in AUC compared to SeqTrack256 [6] with increases of 3.0% and 2.4%, respectively. Our DeTrack3s4 outperforms the state-of-the-ar method ARTrack3s: by 2.4%. This is attributed to the non-overlapping nature of the training and testing sets in the GOT-1OK 10k dataset, indicating our method's s strong performance on unseen data. The denoising learning paradigm has learned powerful denoising capabilities while facing with arbitrary positions and sizes of boxes.
LaSOT. LaSOT is benchmark designed for long-term tracking, featuring test collection consisting of 280 videos. Our DeTrack2556 achieves an AUC of 71.3%, exhibiting performance improvement compared to other methods based on 256 resolution. Additionally, our DeTrack388 also demonstrates state-of-the-an thrtrr-rt performance, validating the strong competitiveness of our approach in long-term dataset. This attributed to our compound memory design, which leverages historical trajectory and appearance information to enhance the model's generalization ability on long-teri dataset.
LaSOT ent. LaSOT ext tracking dataset. It comprises 150 video sequences and encompasses 15 object classes. Our De- Track384 shows significant improvements compared other methods, with 1.7% increase in AUC over BrqTTrak384 and a 2.4% improvement in P Porm- This demonstrates the strong generalization capability our approach even with extended data, particularly manifesting notable advantages in the accuracy of bounding box center point.
an extension of the LaSOT dataset, also categorized as long-tern
4.3
Ablation study Denoising Learning
Table 3: Ablation study of denoising steps on GOT-1Ok. The best results are highlighted in bold.
step1
step2
step3
step4
step5
step6
step7
step8
step9
step10
step1
step12
AO
1.1
1.6
4.8
7.5
12.5
21.4
33,1
52.3
65.7
70.2
74.8
77.1
SRo.5
0.1
0.2
1.2
2.8
8.0
17.9
34.1
57.6
74.7
78.7
83.7
86.1
SRa.75
0.0
0.0
0.2
0.8
2.9
8.0
17.8
39.1
56.9
64.6
70.5
73.5
Influence of denoising steps. We investigate the impact of the number of denoising iterations on the performance of the tracker. Our proposed In-model latent denoising consists of a total of steps based on denosing blocks, requiring only forward pass to complete denosing. As shown in Tab the model's performance is nearly zero the first and second denoising steps because the bounding boxes are still filled with noise. However, there is significant qualitative improvement in model performance at the eighth denoising step, reaching its peak at the twelfth step. As shown in Fig 4 the results improve progressively step by step, consistent with Tab.
Analysis of denoising paradigm. Although our method completes denoising with only a single forward pass through the tracking model, it can also be adapted to perform multiple forward passes, similar traditional Diffusion model 26 Therefore, further analyze and compare the multiple
Table 4: Ablation study of denoising paradigm on GOT-10k. The best results are highlighted in bold.
Denoising paradigm
Steps
AO
SRo.s
SRo 75
FLOPP
Speed
Multiple forward passes
96
75.7
84.8
72.9
424.0G
8FPS
Multiple forward passes
48
75.7
84.6
72.5
212.0G
12FPS
Multiple forward passes
24
75.9
84.9
72.4
106.0G
29FPS
Single forward pass
12
77.1
86.1
73.5
53.0G
42FPS
8

---

Table 5: Ablation study of denoising block on GOT-1Ok.
The best results are highlighted in bold.
Denosing block
Denoising attention
NoisePred
AO
SRo.s
SRp.7o
FLOPS
74.0
84.7
72.5
51.2G
75.1
84.1
72.0
52.7G
77.1
86.1
73.5
53.0G
Step1
Step2
Step3
Slep4
Stap6
Stapp
Step7
Step8
Slepg
Step10
Slpp11
Slep12
Figure 4: Visualization of the denoising step GOT-10k. The first row is the video GOT-1OK- Test-000040, the second row is the video COT-10k-Test-0000511
and the third row is the video
forward passes and single forward pass paradigms, shown in Tab. 4 In DrTrack, the performance of multiple forward passes is not superior to that of single forward pass. Additionally, denoising is performed similarly traditional Diffusion models, the computational cost increases significantly. Single forward pass only requires 53.0G FLOPS and achieves a speed 42 FPS, while multiple forward passes incurs exponentially higher computational costs with a linear decrease in speed.
Analysis of the denoising block. shown in Tab there is no NoisePred module, AO will decrease by 2.0%, and SRp.5 will decrease by 2.0%. This demonstrates that noise prediction and gradually subtracting noise are crucial for the model. Furthermore, removing denoising attention leads to further performance degradation, demonstrating that utilizing image features conditional inputs can also assist in denoising. Moreover, the computational overhead the denoising block increased by only 1.80G, owing the fact that the box comprises merely 4 tokens. Thus, even with the addition of denoising attention and NoisePred, this remains a negligible computational burden.
4.4 Ablation study on Compound Memory
Because the memory mechanism is designed to address the challenge of dynamic changes in video, and considering the greater variety of cnvironmental and appearance changes in long video datasets, chose the LaSOT (long-term tracking dataset, averaging 2448 frames per video) to validate the effectiveness of our memory mechanism.
725
71.
135
66.
AUC
Length
lou
Score
Schmak Soore
mirroa Verrory
mamawy
Visual Memory Length
Trajectory Memory Langth
Threshald
Softmax Threshold
(6) Compound memory
Figure 5: Ablation study of memory on LaSOT. (a) Different visual memory lengths; (b) Different trajectory memory lengths; (c) Different IoU thresholds are applied for template updates; (d) The influence Softmax thresholds. (e) With or without compound memory.
Exploration on the length of the visual memory and the trajectory memory. We firstly explore the impact of different visual memory lengths. As shown in Fig when the length is only the model's AUC is only 70.2. However, with an increase in memory length, performance gradually improves, reaching peak at the 3-rd frame. Subsequently, performance declines. This because when the memory too short, the model cannot adapt changes in the target and the environment.

---

Conversely, when the memory is too long, it stores incorrect information. Unlike visual memory, as shown in Fig. (b), trajectory memory does not exhibit trend of initially rising and then falling with an increase in stored boxes. The performance consistently improves as the number of boxes ranges from 7. As shown in F (e), we also achieved a performance of 70.2 by removing all memories, which further confirms the effectiveness of our memory.
Effects of IoU score and Softmax scorefor visual memory updating. For the update of visual memory, we strive to avoid updating poor templates into our visual memory. This would lead to tracking drift. Therefore, as shown in Fig. 5]d) keeping IoU score fixed, we conduct an ablation study on different Softmax score values. The study found that an accuracy update can be achieved when the Softmax score is set to 0.9, obtaining 71.3% on AUC. As shown in Fig. (5cc) keeping Softmax score fixed, the best IoU score is 0.75. When the IoU score threshold is set to 0.85, it leads to a decrease in AUC. because the overly strict condition reduces the frequency of visual memory updates.
Limitation
Despite achieving real-time speed and competitive performance, DeTrack still has certain limita- tions. Existing tracking methods struggle to recover the target when facing challenges such as object occlusion and out-of-view situations. Although our proposed trajectory memory can assist in target reacquisition after target loss in some cases, further improvements are needed to address challenges like object occlusion and out-of-view scenarios. We will investigate the challenges in these scenarios.
6 Conclusion
Traditional visual object tracking methods using image-feature regression or coordinate autoregression models faced limitations in handling positional priors and unseen data. Inspired by the diffusion model, we introduced denoising learning to enhance model robustness. Our approach, employing noisy bounding boxes for training, introduces a novel paradigm of denoising learning in object tracking. By decomposing the process into individual denoising blocks within our proposed denoising Vision Transformer (ViT), we achieved real-time performance while maintaining effectiveness. Experimental results demonstrate the efficacy of our method, showcasing competitive performance and rendering denoising learning applicable in the visual object tracking community.
Acknowledgement This work was supported by National Natural Science Foundation of China (oo.62072110 12), National Natural Science Foundation of China under Grant Nos. 62106051 and the National Key R&D Program of China 2022YFC3601405.
References
[1] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully-convolutional siamese networks for object tracking. In European conference computer vision, pages 850-865. Springer, 2016.
[2] Goutam Bhat, Martin Danelljan, Lue Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6182-6191, 2019.
[3] Yidong Cai, Jie Liu, Jie Tang, and Gangshan Wu. Robust object modeling for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9589 9600,, 2023.
[4] Shoufa Chen, Peize Sun. Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 19830-19843 2023.
[5] Ting Chen. Saurabh Saxena, Lala Li. David Fleet, and Geoffrey Hinton. Pix2seq: language modeling framework for object detection. arXiv preprini wrXiv:2109.1085: 2021.
[6] Xin Chen, Houwen Peng, Dong Wang, Huchuan Lu, and Han Hu. Seqtrack: Sequence to sequence learning for visual object tracking. In Proceedings of the IEEF/CVP Conference on Computer Vision and Pattern Recognition, pages 14572-14581, 2023.
[7] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang. Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 8126-8135, 2021.
[8] Zedu Chen, Bineng Zhong, Guorong Li, Shengping Zhang, and Rongrong Ji. Siamese box adaptive network for visual tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6668 6677, 2020.
10

---

[9] Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in Neural Information Processing Systems, 34:11781-11791 2021.
[10] Yutao Cui, Cheng Jiang. Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13608-13618, 2022.
[11] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Atom: Accurate tracking by overlap maximization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4660-4669, 2019.
[12] Martin Danelljan, Luc Van Gool, and Radu Timofie. Probabilistic regression for visual tracking. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 7183-7192, 2020.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199: 2020.
[14] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Mingzhen Huang, Juchuan Liu, Yong Xu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129 2021.
[15] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Mingzhen Huang, Juehuan Liu, Yong Xu, et al. Lasot: A high-quality large-scale single object tracking benehmark. International Journal of Computer Vision, 129:439-440 461, 2021.
[16] Zhihong Fu, Qingjie Liu, Zehua Fu, and Yunhong Wang. Stmtrack: Template free visual tracking with space-time memory networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13774 13783, 2021.
[17] Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang. and Junsong Yuan. Aiatrack: Attention in attention for transformer visual tracking. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt XXII, pages 146-164. Springer, 2022.
[18] Shenyuan Gao, Chunluan Zhou, and Jun Zhang. Generalized relation modeling for transformer tracking. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 18686- 18695, 2023.
[19] Dongyan Guo, Jun Wang. Ying Cui, Zhenhua Wang, and Shengyong Chen. Siamear: Siamese fully convolutional classification and regression for visual tracking. Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages 6269-6277, 2020.
[20] Pinxue Guo, Lingyi Hong, Xinyu Zhou, Shuyong Gao, Wanyun Li, Jinglun Li, Zhaoyu Chen, Xiaoqiang Li, Wei Zhang, and Wenqiang Zhang. Clickvos: Click video object segmentation. arXiv preprint arXiv:2403.088 200, 2024.
[21] Pinxue Guo, Wanyun Li, Hao Huang, Lingyi Hong, Xinyu Zhou, Zhanyu Chen, Jinglun Li, Kaixun Jiang, Wei Zhang, and Wenqiang Zhang. X- X-prompt: Multi-modal visual prompt for video object segmentation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 5151-5 160, 2024.
[22] Pinxue Guo, Wei Zhang, Xiaoqiang Li, and Wenqiang Zhang. Adaptive online mutual learning bi- di-ddcssers for video object segmentation. IEEE Transactions on Image Processing, 31:7063-7077, 2022.
[23] William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffuusion modeling long videos. Advances in Neural Information Processing Systems, 35:27953-2799 2022.
[24] Kaijie He, Canlong Zhang, Sheng Xie, Zhixin Li, and Zhiwen Wang. Target-aware tracking with long-term context attention. arXiv preprint 230.:2232211840000 2023.
[25] Kaiming He, Xinlei Chen, Saining Xic, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[26] Jonathan Ho. Ajay Jain, and Pieter Abbeel, Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 13.554068851, 2020.

---

[27] Jonathan Ho. Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. The Journal of Machine Learning Research, 23(1):2249-233 2281, 2022.
[28] Lingyi Hong. Zhongying Liu, Wenchao Chen, Chenzhi Tan, Yuang Feng, Xinyu 7hou, Pinxue Guo, Jinglun Li, Zhaoyu Chen, Shuyong Gao, al. Lvos: henchmark for large-scale long-term video object segmentation. arXiv preprint arXiv:2404. 19326, 2024.
[29] Lingyi Hong. Shilin Yan, Renrui Zhang, Wanyun Li, Xinyu Zhou, Pinxue Guo, Kaixun Jiang. Yiting Chen, Jinglun Li, Zhaoyu Chen, Onetracker: Unifying visual object tracking with foundation models and efficient tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19099,19091, 2024.
[30] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 53(5):1562- 1577, 2019.
[31] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. Siamrpu++: Evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4282-4291, 2019.
[32] Bo Li, Junjie Yan, Wei Wu, Zheng 7hu, and Xiaolin Hu. High performance visual tracking with siamese re- gion proposal network. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8971-8980, 2018
[33] Liting Lin, Heng Fan, Zhipeng Zhang, Yong Xu, and Haibin Ling. Swintrack: A simple and strong baseline for transformer tracking. Advances Neural Information Processing Systems, 35:16743-106
2022.
[34] Christoph Mayer, Martin Danelljan, Goutam Bhat, Matthicu Paul, Danda Pani Paudel, Fisher Yu, and Luc Van Gool. Transforming model prediction for tracking. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8731-8740, 2022.
[35] Christoph Mayer, Martin Danelljan. Danda Pani Paudel, and Luc Van Gool. Learning target candidate association keep track of what not to track. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13444-13454, 2021.
[36] Mubashir Noman, Wafa AI Ghallabi, Daniya Najiha, Christoph Mayer, Akshay Dudhane, Martin Danelljan, Hisham Cholakkal, Salman Khan, Lue Van Gool, and Fahad Shahhaz Khan, Avist: A benchmark for visual object tracking in adverse visibility. arXiv preprint arXiv:2208.06888 2022.
[37] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David Fleet, and Mohammad Norouzi. Image resolutin via iterative refinement. JEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713-4722) 2022.
[38] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
[39] Zikai Song, Run Luo, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang. Compact transformer tracker with correlative masked modeling. arXiv preprint 230.933001.1033. 2023.
[40] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and fierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. Proceedings of the 25th international conference on Machine learning, pages 1096-1 103, 2008.
[41] Paul Voigtlaender, Jonathon Luiten, Philip HS Torr, and Bastian Leibe. Siam r-cnn: Visual tracking by re-detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 6578-6588, 2020.
[42] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li. Transformer meets tracker: Exploiting temporal context for robust visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1571-,580, 2021.
[43] Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, and Yihong Gong. Autoregressive visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9697-9706, 2023.
[44] Qiangqiang Wu, Tianyu Yang, Ziquan Liu, Baoyuan Wu, Ying Shan, and Antoni B Chan. Dropmae: Masked autoencoders with spatial attentio dropout for tracking tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14561-1457 2023.
12

---

[45] Junliang Xing, Haizhou Ai, and Shihong Lao. Multiple human tracking based on multi-view upper-body detection and discriminative learning. In 2010 20th International Conference on Pattern Recognition, pages 1698-1701. IEEE, 2010.
[46] Yinda Xu, Zeyu Wang. Zuoxin Li, Ye Yuan, and Gang Yu. Siamfe++: Towards robust and accurate visual tracking with target estimation guidelines. Proceedings of the AAAI Conference on Artificial Intelligence. volume 34, pages 12549-12556, 2020.
[47] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10448 10457, 2021.
[48] Bin Yan, Xinyu Zhang, Dong Wang. Huchuan Lu, and Xiaoyun Yang. Alpha-refine: Boosting tracking performance by precise bounding box estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5289-5889008 5298, 2021.
[49] Dawei Yang, Jianfeng He, Yinchao Ma, Qianjin Yu, and Tianzhu Zhang. Foreground-background distri- bution modeling transformer for visual object tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10117-10127, 2023.
[50] Ruihan Yang. Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video genera- tion. Entropy, (E((10)1469, 2023.
[51] Botao Ye. Hong Chang, Bingpeng Ma. Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: one-stream framework. In Computer Esion-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 341-357. Springer, 2022.
[52] Qiangian Yu, Keqi Fan, and Yuhui Zheng. Domain adaptive transformer tracking under occlusions. JEEE Transactions on Multimedia, 2023.
[53] Zhipeng Zhang, Houwen Peng, Jianlong Fu, Bing Li, and Weiming Hu. Ocean: Object-award anchor-free tracking. In European Conference on Computer Vision, pages 771-787. Springer. 2020.
[54] Haojie Zhao, Dong Wang, and Huchuan Representation learning for visual object tracking by masked appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1869-187705 2023.
[55] Xinyu Zhou, Pinxue Guo, Lingyi Hong, Jinglun Li, Wei Zhang. Weifeng Ge, and Wenqiang Zhang. Reading relevant feature from global representation memory for visual object tracking, In shirth-seventh Conference on Neural Information Processing Systems, 2023.
[56] Xinyu Zhou, Pinxue Guo, Lingyi Hong, Jinglun Li, Wei Zhang, Weifeng Ge, and Wenqiang Zhang Reading relevant feature from global representation memory for visual object tracking. Advances in Neural Information Processing Systems, 36, 2024.
A Appendix
Table 6: The differences between DDPM 26), DAE[40), and DeTrack 44
DDPM
DAE
DeTrack
Noise Type
Gaussian noise Noisy image(x:) fa(x²)
Gaussian noise Noisy image(x")
Gaussian noise Noisy box(b")
Input Encoding Decoding
fo(x")
512, 211 ... 21 fe(b")
Noise go(=)
image ge(z)
box by = ga(222
Optimization objective
b bo
Inference
1t-1
OFE
Ie go(2)
be gr
A.1
The differences between DDPM, DAE, and DeTrack
According to Tab we compares and analyzes the differences between DDPM, DAE, and DeTrack in denoising learning, highlighting the advantages of the DeTraak model in visual object tracking. All three use Gaussian noise to simulate the noise characteristics of the input; however, they differ in input data, encoding methods, decoding methods, optimization objectives, and inference approaches. DDPM and DAE take noisy images input (z"), aiming to restore generate high-quality images,
13

---

while DeTrack innovatively uses noisy bounding boxes (6*) as input, making it more suitable for visual tracking tasks complex backgrounds and scenarios with fast-moving objects.
In terms of encoding, DDPM and DAE single-layer feature encoding to obtain fo(); ; in contrast, DeTrack employs multi-layer feature encoding with layer-by-layer denoising within the model, resulting in multiple hidden states z; (212, Fll,.... 21 = fa (b)). This layer-wise denoising approach retains and optimizes target feature information, enhancing robustness to bounding box noise. For decoding, DDPM's s decoding target is to restore the noise es = go (z), while DAE directly decodes the image 2e g0(z). De Dearack, the other hand, decodes to denoised bounding box by go (Z12), ensuring
localization the target bounding box.
Regarding the optimization objective, DDPM minimizes the error between generated noise and target noise (<-co) co), DAE minimizes the error between the denoised image and the original image (æ zz), and DeTrack optimizes the eπror between the denoised bounding box and the original bounding box - b9), making more suitable for accurate visual target localization. For inference, DDPM uses a reverse diffusion process to progressively denoise and generate image, while DAE and DeTrrack directly generate denoised results inference: DAE outputs the image Tg (o), and DeTrrack outputs the bounding box by = go (=12).
Overall, DeTrack's multi-layer feature encoding with internal model denoising, specific decoding approach, and optimization objective enable it to exhibit higher robustness in noisy and complex backgrounds, making it well-suited for target tracking tasks dynamic and complex scenarios.
Table 7: Comparison of noise prediction pattern GOT-10k. The best results are highlighted in bold.
AO
SRo.5
SRo.75
Predicting the total noise Predicting noise layer by layer
75.2
84.1
71.4
77.1
86.1
73.5
A.2 Comparison of noise prediction pattern
According 7 Predicting the total noise resulted in a decrease of 1.9 AO, in SRg.5, and 2.1 in . compaedd to multiple noise predictions. We analyze that this is because predicting the total noise directly is more challenging than predicting it layer by layer. Layer-by-laye denoising allows the model to learn to filter out some noise intermediate layers before arriving at the final result, rather than achieving it in one step.
A.3 Analysis of Denoising Paradigm with ViT-Small
Table 8 presents ablation study different denoising paradigms and step settings evaluated on the GOT-10k dataset with a Vit-Small backbone. We compare performance metrics such as Average Overlap (AO) and Success Rates two different overlap thresholds (SRo.s and SRp.75).
The results indicate that multiple forward passes generally yield better performance compared to a single forward pass. Specifically, step count of 48 achieves the best AO, SRp.s. and SR values, with scores of 69.4, 78.5, and 63.4, respectively, highlighted in bold in Table [8] This suggests that while increasing the number of steps from 12 (single forward pass) to 48 improves performance, further increasing to 96 steps does not result in additional gains, possibly due to diminishing returns in iterative refinement or over-smoothing of features.
Table 8: Ablation study of denoising paradigm on GOT-10k (Vit-Small). The best results are highlighted in bold.
Denoising paradigm
Steps
AO
SRo.5
SRo.75
Multiple forward passes
96
68.9
78.2
63.2
Multiple forward passes
48
69.4
78.5
63,4
Multiple forward passes
24
69.4
78.0
63.0
Single forward pass
12
69.1
78.3
62.9
14

---

Notably, the AO metric remains at 69.4 for both 48 and 24 steps, although the success rates (SRo.s i and 75) are slightly lower 24 steps. This finding implies that 48 steps might strike balance between computational efficiency and denoising effectiveness, providing optimal tracking performance without the need for excessive forward passes.
In summary, the experiments demonstrate that while iterative denoising is beneficial, there exists an optimal step count (48 in this case) that maximizes tracking accuracy. This demonstrates that our proposed Deara,,, when using ViT-Small as the backbone, can enhance tracking accuracy through recursive denoising approach, similar to DDPM. However, this recursive denoising introduces significant increase in computational complexity.
A.4 Applying DiffusionDet to Tracking
Table 9: Comparison of Configurations between DiffusionTrack and DeTrack.
Denoising paradigm
Encoder
Decoder
DiffusionTrack
DeTrack Encoder
DiffusionDet Decoder
DeTrrack
DeTrack Encoder
DeTrack Decoder
DiffusionDet cannot be directly applied to object tracking, as it requires interaction between the template and search region in tracking. Therefore, as shown in Tab. 10 we use the Encoder from DeTrack, which enables this interaction, as the encoder for DiffusionDet. The decoder is taken from DiffusionDet. We call this model TuffusiioTTraciinp and it uses a resolution of 384x384. For fairness, the learning rate, number of epochs, weight decay, and other training parameters are kept consistent.
Table 10: Performance Comparison on GOT-10k between Diffusion Track and DeTrack. The best results are highlighted in bold.
Denoising paradigm
Step
AO
SRg.s
SRa.75
FLOPS
DiffusionTrack
71.8
81.0
69.6
120.0G
DiffusionTrack
72.1
81.1
70.0
123.4G
Diffusion Track
71.9
81.1
69.3
133.5G
Diffusion
8
73.5
82.9
71.2
147.2G
DiffusionTrace
12
72.5
81.9
70.2
162.7G
DeTrack
12
77.9
86.5
74.9
119.0G
A.5 Comparison GOT-10k between ConfuusionTrack and DeTrack
The performance comparison on GOT- 10k dataset between Diffusion Track and DeTrack demonstrates notable differences in tracking accuracy and computational efficiency across various denoising steps. For DiffusionTrrack the tracking performance generally improves as the step count increases, reaching its peak with steps, where the Average Overlap (AO) is 73.5%, SRo.5 is 82.9%, and SRa.75 71.2%. However, this improvement comes at the cost increased computational requirements, with the FLOPS reaching 147.2G at 8 steps and 162.7G 12 steps.
In contrast, DeTrack, tested with 12 steps, achieves the highest performance overall, with an AO of 77.9%. SR,.s of 86.5% and SRa.75 of 74.9%, surpassing all Diffusion Track configurations. DeTrack also maintains lower computational complexity with 119.0G FLOPS, suggesting a more optimal balance of tracking accuracy and efficiency. This analysis indicates that while DiffusionTrace benefits from increased steps in tracking performance, Track achieves superior results both in accuracy and computational efficiency.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We accurately describe main contributions and the covered domains in the abstrac[[ant introduction[1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the Li the main text, we describe the limitations of our method.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification: we have provide the full set of assumptions and complete (and correct) proof in section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed descriptions of our tracking paradigm and methods in sections and elaborate on the model's details well as training and inference procedures
in section4.1
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
17

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will provide the code and model after the paper be accepted
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/publicc/uuieeee CoodeSumeeieeeeee ssi iessonPolll mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the training and test details section4.1 and provide the update threshold in ablation s.444.4)
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: We followed previous visual object tracking methods 6 conduct Experiment Statistical Significance.
43 51
and did not
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide sufficient information on the computer resources in section and report the speed in Tabl4 and Tab
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The paper conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our method is visual object tracking, not image generation, and does not pose issues such as generating false information.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper has no relevant risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite all the assets utilized in our paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "CnNqCls1cS", "reviewer": "Reviewer_JcZZ", "summary": "In this paper, the authors reformulate the visual tracking problem using denoising learning process. Given noisy bounding box coordinates, the conditional denoising tracker model's task is to remove the bounding box noise for accurate target state estimation. For the denoising formulation, they introduce an in-model latent denoising process where they claim that it can be executed with a single forward pass of the model. The effectiveness of their proposed method is validated on various visual tracking datasets.", "questions": "Please refer to the questions in the weaknesses section.", "limitations": "The authors included a separate discussions and limitations section in their paper, with adequate explanations and descriptions on the weaknesses and possible future directions.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The authors evaluate their proposed method on multiple challenging visual tracking benchmarks, where they show competitive performance on these datasets to some extent when compared to other state-of-the-art methods.\n\n- The authors show results for multiple ablation experiments on varying hyperparameter settings such as memory length and threshold value, showing some robustness to hyperparameter choices.\n\n- The motivation of the proposed method is plausible, where the authors focus on the timely subject of reformulating visual tracking to solve the persisting issues of previous two-stream feature matching-based visual tracking methods.", "weaknesses": "- The claim that the model only require a \"single forward pass\" seems somewhat misleading since the full latent denoising model contains $l=12$ denoising blocks and require same number of denoising steps for full inference. \n\n- From my understanding, the motivation for generative models that employ iterative denoising process using Markov chains was that modeling the complex data distribution with multiple modes cannot be achieved with a single Gaussian normal distribution. From this perspective, what is the motivation for modeling visual tracking using this iterative multi-step process, since it seems overly cumbersome and intricate.\n\n- Related to the previous question, how does changing the number of denoising blocks $l$ affect the performance of the tracker? Experimental results in Table 3 does not seem enough since the model is trained with $l=12$ as its default value.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt gratitude for your perceptive insights and your recognition of the unique and meaningful contributions made by our research. Your support is highly valued, and we would be honored if you could serve as an advocate for our work.\n\n>***Q1**.The claim that the model only require a \"single forward pass\" seems somewhat misleading since the full latent denoising model contains 𝑙=12 denoising blocks and require same number of denoising steps for full inference.*\n\n- I apologize for the confusion. While it is true that we perform denoising in 12 steps, these 12 denoising steps are included  within the Denoising ViT. Therefore, What we intend to convey is not that the denoising steps require a single forward pass, but rather that our method requires only **a single forward pass through the entire network** to complete the denoising process. In contrast, diffusion models require **multiple forward passes through the entire network** to achieve denoising. The specific differences are as follows:\n| ------------------------------------------------------------------------------------- | Diffusion Model | DeTrack |\n| ------------------------------------------------------------ | --------------- | ------- |\n| **Number of forward passes through the entire network**      | n times         | 1 time  |\nTherefore, we refer to it as a \"single forward pass.\"\n\n>***Q2**. What is the motivation for modeling visual tracking using this iterative multi-step process?*\n\n- **Noisy bounding boxes, which can have arbitrary shapes and positions.** Using training set bounding boxes may not  provide sufficient robustness to unseen data. By introducing Gaussian noise to create noisy boxes with arbitrary shapes and positions, we enhance robustness during testing. On GOT-10k, where training and testing sets have different categories, our method shows superior performance, demonstrating that adding noise effectively improves the tracker's robustness to unknown data.\n\n- **From frame t−1 to frame t, we assume that the object undergoes numerous small movements.** These movements are complexly distribution in nature, with each step's movement being dependent on the previous position. Therefore, we model the object's motion pattern using this approach. We simulate these numerous small movements with Gaussian noise, where the process from frame t to frame t−1 is viewed as the noise-adding process, and from frame t−1 to frame t as the denoising process.\n\n>***Q3**.Related to the previous question, how does changing the number of denoising blocks 𝑙 affect the performance of the tracker? Experimental results in Table 3 does not seem enough since the model is trained with 𝑙=12 as its default value.*\n\n-  Based on your valuble suggestion, we retrained the model according your suggestion. We added supervision signals at different layers and changed the number $l$ of denoising blocks to evaluate their performance impact on the tracker. The latest results are shown in the table below: compared to Table 3 in the original paper, the updated results demonstrate higher accuracy in the first 8 steps, with faster convergence in denoising. However, by the 10th step, the performance matches that in Table 3, and at the 11th step, it is even slightly lower than in Table 3. \n||step1|step2|step3|step4|step5|step6|step7|step8|step9|step10|step11|step12|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|AO|16.9|21.9|25.2|34.8|42.5|48.7|52.4|60.6|65.4|70.5|71.6|77.1|\n|SR$_{0.5}$|12.5|17.2|21.3|34.9|43.1|52.3|57.3|67.1|72.4|78.2|80.8|86.1|\n|SR$_{0.75}$|2.9|6.9|9.7|16.9|23.9|32.1|37.3|49.1|55.5|62.6|65.9|73.5|\nWe greatly appreciate your suggestion, as it has helped us improve our manuscript further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uuh7IjVYtk", "reviewer": "Reviewer_wkvN", "summary": "The paper addresses the task of visual object tracking. The main idea is to pose tracking as a latent denoising problem where the bounding box from the previous frame is denoised to obtain the current frame box. The authors use a ViT to obtain embeddings using the search regions as well as a set of template images as input. The embeddings obtained from each ViT block serves as the conditioning input to a denoiser block that iteratively denoises the noise bounding box estimate. The final box is further refined by a block which takes the trajectory (previous 7 estimated bounding boxes) as input. The authors maintain a dynamic memory of previous templates, based on the reliability of tracking. The method obtains state-of-the-art results on standard tracking benchmarks such as LaSOT, GOT-10k, AVisT. The authors further provide ablation studies and analyze the key components of the method.", "questions": "1. As stated in the weaknesss 1, please provide details on how the intermediate denoising outputs are supervised.\n\n2. Related to weakness 2, what is the performance obtained by a model trained to directly predict the total noise, while keeping the rest of the architecture and training identical?\n\n3. Please provide more details on the model training, as stated in weakness 3.\n\n4. How many templates are stored in the visual memory? Is it 3? Do you always keep the first frame in the memory?\n\n5. In table 4, when running the denoising process multiple times, how was the model trained? Do you just run the model corresponding to \"Single forward pass denoising\" multiple times, or train a new model in a DDPM style manner? Instead of using the same ViT-B architecture to do repeated denoising, what happens if you use a smaller model (ViT-S or say even small with only a couple of Transformer blocks), but apply it multiple times?\n\n6. In table 5, what exactly does not doing\"NoisePred\" correspond to? What is the model architecture in this case?\n\n7. It appears that the tracking performance consistently improves with using larger trajectory memory. Did you experiment beyond using 7 previous frames? Would using a larger trajectory cause significant latency overhead?\n\n8. There seems to be a typo in line 114, in the definition of alpha. \n\n9. Tracking datasets such as LaSOT can contain snippets with long occlusions. How does the model handle such cases? Do you simulate occlusions / out-of-frame scenarios during training to ensure that the denoising process doesn't diverge? In the limitations, the authors state that the trajectory memory can help to recover from occlusions. However since it contains only 7 previous frames, I'm not sure if that's sufficient to recover from such challenging cases since they can last longer. Does the visual memory help the tracker to redetect the object after long occlusions?", "limitations": "The authors briefly discuss the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The paper obtains strong results, outperforming recent trackers on large scale tracking benchmarks such as  LaSOT, GOT-10k, AVisT.\n2. Rather than directly regress the target bounding box, or generate a score map indicating target location, the proposed architecture iteratively denoises the previous frame bounding box for tracking. This is novel and quite interesting. The method is also shown to perform very well.\n3. The authors also employ a trajectory memory to integrate object motion information into the tracking which is shown to help. Most of the current trackers do not utilize any trajectory information, except using some weak prior that object motion is small. So it's interesting to see object trajectory directly utilized in the model prediction, rather than as a ad-hoc post-processing step.", "weaknesses": "1. While the authors present the method as a latent denoising approach motivated by DDPM, it is not fully clear how much of the DDPM theory is applicable to the method in the current form. For instance, how are each of the denoising stages supervised? Do you follow some noise schedule to supervise the output of each block? Unlike in a DDPM where a denoising network is shared and a timestamp is randomly sampled for each training iteration, the proposed approach utilizes a stacked architecture. Thus during training, are inputs to the i’th denoising block the outputs of i+1’th denoising block, or a randomly sampled noise? It would be helpful is the authors provide more details on how the intermediate denoising outputs are supervised.\n\n2. I couldn’t find a clear experimental evidence on the benefit training the model to iteratively denoise the bounding box, instead of say directly predicting the total noise (i.e. the offset from previous to current box), or directly regressing the new box, while keeping the rest of the architecture identical. While the authors provide an ablation study in Table 5 on impact of denoising blocks, it seems to only analyze the impact of architectural choices in the denoising block (please correct me if I’m missing something here). I believe such a comparison is important the judge the impact of the core idea of doing iterative denoising.\n\n3. There are some missing details for the training, especially related to training of visual and trajectory memory blocks. Do you perform a sequence based training to simulate the online tracking scenario? If not, how are the templates for the visual memory, as well as the trajectory obtained? Do you just use previous frame boxes (after adding some noise?), do you take consecutive frames from a video or sample randomly from a video?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the acknowledgement of the main contributions of our work, the useful comments and the relevant feedback provided on the technical side. We have provided detailed responses to your questions. \n\n>***Q1**.While the authors present the method as a latent denoising approach motivated by DDPM, it is not fully clear how much of the DDPM theory is applicable to the method in the current form.*\n\n- **Our method only borrows the noise addition technique from DDPM, but our training approach and denoising process are different from DDPM.** DDPM directly predicts noise during training, while we directly predict the ground truth. DDPM denoising requires sampling from a Gaussian distribution, which introduces randomness, whereas we can directly predict the target bounding box using a stacked denoising method. **Therefore, our method does not need to fully adhere to DDPM's theoretical framework.** \n\n- **Compared to DDPM, our method is more similar to the Denoising Autoencoder (DAE) [1].** A DAE adds noise to the input, passes it through an encoder to obtain the latent vector $z$, and then inputs $z$ into a decoder to reconstruct the output. In our method, we add noise to the bounding box and use a stacked denoising approach to obtain multiple intermediate latent vectors $z_i (i=1-12)$. We then input the latent vector $z_{12}$ into the box refining and mapping to reconstruct the target bounding box. \n\n-  **Below is a table that describes the differences between our method and DDPM, as well as the similarities with DAE:**\n\n   |  |DDPM|DeTrack| DAE|\n   |-|-|-|-|\n   |Noise addition|Gaussian noise|Gaussian noise|Gaussian noise|\n   | Input | Noisy image($x^*$) | Noisy box($b^*$) | Noisy image($x^*$) |\n   | Encoding | $z=f_\\theta(x^*)$ |$z_{12}, z_{11}\\cdots,z_{1}=f_\\theta(b^*)$| $z=f_\\theta(x^*)$ |\n   | Decoding | Noise $\\epsilon_\\theta=g_\\theta(z)$ | box $b=g_\\theta(z_{12})$ |image $x=g_\\theta(z)$ |\n   |Optimization objective| $\\epsilon-\\epsilon_\\theta$ | $b-b_\\theta$|$x-x_\\theta$|\n   |Inference| $x_{t-1} = \\frac{1}{\\sqrt{\\alpha_t}}(x_{t}-\\frac{\\beta}{\\sqrt{1-\\bar{\\alpha_t}}}\\epsilon_\\theta)+ \\sigma_t \\epsilon$ | $b=g(z_{12})$|$x=g(z)$|\n\n> ***Q2**.For instance, how are each of the denoising stages supervised?Do you follow some noise schedule to supervise the output of each block?*\n\n- We do not apply supervision to each block individually; instead, we use an implicit learning process similar to DAE[1]. By minimizing ||$b-b_\\theta$||,the model is able to automatically learn the intermediate latent vectors.\n\n- We also experimented with adding supervision to intermediate layers, applying supervisory signals to each layer individually, but this approach did not yield better performance:\n|| AO       |SR$_{0.5}$|SR$_{0.75}$|\n|-|-|-|-|\n|**Supervise each layer**|72.8|81.5|68.8|\n|**DeTrack**|**77.1**|**86.1**|**73.5**|\n\n- **We hypothesize that early supervision may make denoising more challenging.** For instance, adding supervision to the first layer implies that the model must complete the denoising in a single step. In contrast, applying supervision to the final layer allows the model to adaptively learn the denoising according different layers. As shown in Table 3 of the original paper, the denoising performance is relatively poor in the initial layers, and better denoising results are exhibited in the later layers.\n\n> ***Q3**.Thus during training, are inputs to the i’th denoising block the outputs of i+1’th denoising block, or a randomly sampled noise?* \n- The output of $i$th layer is the input of $i+1$th layer.\n\n> ***Q4**. I couldn’t find a clear experimental evidence on the benefit training the model to iteratively denoise the bounding box, instead of say directly predicting the total noise (i.e. the offset from previous to current box), or directly regressing the new box, while keeping the rest of the architecture identical.*\n\n- **Directly regressing the new box**. I apologize for any confusion. Please allow me to clarify. We have already conducted experiments where the bounding box is directly regressed while keeping the original structure unchanged, as shown in **Table 5 of the original paper**.\n\n   ||Deoising attentiion|NoisePred| AO       | SR$_{0.5}$ | SR$_{0.75}$ |\n   |-|-|-|-|-|-|\n   | **Directly regressing box(image)** ||| 74.0   | 84.7      | 72.5     |\n   | **Directly regressing box(box)** |✓|| 75.1    | 84.1       | 72.0      |\n   | **DeTrack**                    |✓|✓| **77.1** | **86.1**   | **73.5**    |||\n\n- The first row represents the case without any denoising module, directly regressing the bounding box from image features. The second row removes the noise prediction module,  similar to DAE[1], directly reconstructing the box to remove noise. It can be seen that DeTrack performs better, as DeTrack has learned the denoising pattern within the model's latent vector space.\n\n- **Predicting the total noise.** Based on your valuable suggestion, we conducted experiments predicting the total noise. The results are shown in the table below:\n\n- |                      | AO       | SR$_{0.5}$ |SR$_{0.75}$|\n| -------------------- | -------- | ---------- | -------- |\n| **Predicting the total noise** | 75.2  | 84.1     | 71.4  |\n| **DeTrack**          | **77.1** | **86.1**   | **73.5** |\n\n- Predicting the total noise resulted in a decrease of 1.9 in AO, 2 in SR0.5, and  2.1  in  SR0.75,compared to multiple noise predictions. We analyze that this is because predicting the total noise directly is more challenging than predicting it layer by layer. Layer-by-layer denoising allows the model to learn to filter out some noise at intermediate layers before arriving at the final result, rather than achieving it in one step.\n\n[1]. Vincent P, Larochelle H, Bengio Y, et al. Extracting and composing robust features with denoising autoencoders[C]//Proceedings of the 25th international conference on Machine learning. 2008: 1096-1103.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5zPRHl62Zu", "reviewer": "Reviewer_b9Bg", "summary": "The paper introduces bounding box denoising to visual object tracking. To make it more suitable for tracking, the paper devises an in-model denoising pipeline that reformulate the repetitive sampling process as multiple ViT blocks in one forward pass. It also proposes a compound memory strategy to assist long-term tracking with visual feature and trajectory cues. The paper provides extensive results to validate the efficacy of its designs. The proposed method, named DeTrack, achieves competitive performance across multiple datasets.", "questions": "Q1) Have the authors ever applied the pipeline of DiffusionDet directly to tracking? If practical, how is its performance?\n\nQ2) From Fig. 4, it seems that the bounding box is randomly initialized. Will the randomness affect the tracking performance much between different runs? While this is not a limitation, I am wondering if starting from a fixed anchor (whether calculated by statistics or learned as Sparse R-CNN) would help. Is it a choice that the authors have already considered?", "limitations": "The authors have discussed the limitations.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Q1) Different from previous image-feature regression and autoregressive methods, the idea of denoising learning for tracking is novel and well-motivated, and the proposed pipeline is streamlined and promising to lead a new tracking paradigm.\n\nQ2) Convincing ablations and intuitive visualizations are provided to help understanding the mechanism of the proposed method and demonstrate its effectiveness.\n\nQ3) The proposed tracker achieves a satisfactory performance gain on several benchmarks in a real-time running speed.\n\nQ4) Implementation details are specified to ensure the reproducibility of the proposed method.", "weaknesses": "W1) It is unfair to claim that the pipeline of DiffusionDet (Fig. 1(b)) is inefficient for tracking. The advantage compared to the former pipeline should be explained more meticulously due to the following reasons:\n\n- Frist, for Fig. 1(b), the computation efficiency mainly depends on the capacity of each block instead of the pipeline. In fact, the decoder block in DiffusionDet could be more lightweight than the ViT block in this paper.\n\n- Second, the recycling process of DiffusionDet equals to cascading several denosing blocks that share weights. Therefore, it has no difference in terms of architecture compared to the proposed pipeline when the denoising step is the same.\n\n- Third, as indicated in Table 3, the proposed pipeline has the potential to further improve its performance by using more denoising steps. However, while DiffusionDet allows inferring with an arbitrary number of steps, the proposed pipeline requires retraining more blocks and fails to benefit from such flexibility.\n\nW2) The state-of-the-art comparison on TrackingNet, which is a representative benchmark in tracking, is missing. It should be easy to evaluate the proposed tracker on TrackingNet since the training configuration has been already aligned with other baselines. It would be great if the performance can be reported.\n\nW3) Some minor issues in writing:\n\n- The records in Table 2 needs to be double checked. For example, the proposed tracker does not achieve the best performance in $\\text{P}_{\\text{Norm}}$ on LaSOT, but it is marked in **bold**.\n\n- When introducing the motivation, the authors claim that diffusion is more robust to unseen data. Though the claim is generally true, but it is expected to cite some related references accordingly.\n\n- In Line 252, it is a little bit confusing to say \"reaching its peak at the twelfth step\" as there is no step13. Words like \"perform best\" could be more appropriate here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for the reviewer's acknowledgment of our work. We have taken the time to thoroughly address your questions and have included detailed responses to clarify any uncertainties. \n>***Q1**.Have the authors ever applied the pipeline of DiffusionDet directly to tracking? If practical, how is its performance?*\n\n- **DiffusionDet cannot be directly applied to object tracking, as it requires interaction between the template and search region in tracking**. Therefore, we use the Encoder from DeTrack, which enables this interaction, as the encoder for DiffusionDet. The decoder is taken from DiffusionDet. We call this model **DiffusionTracking**, and it uses a resolution of 384x384. For fairness, the learning rate, number of epochs, weight decay, and other training parameters are kept consistent.\n    The specific configuration is shown in the table below:\n\n  | **Denoising paradigm** | Encoder| Decoder|\n   |-|-|-|\n   |DiffusionTracking|DeTrack Encoder| DiffusionDet Decoder |\n   |DeTrack|DeTrack Encoder|DeTrack Decoder|\n - DiffusionDet uses the DiffusionDet method to achieve tracking through iterative denoising. As shown in the table below, with an increase in the number of iterations, the computation significantly increases (120G -> 147G) and the speed significantly decreases (27fps -> 14fps). DeTrack holds a significant advantage over DiffusionTracking in terms of speed and performance, achieving 3-5 points higher on the GOT-10k dataset. It also scores 3 to 7 points higher on AVisT, Lasot, and LaSOT_ext. DeTrack achieves real-time performance (30fps), whereas DiffusionTracking only reaches 14fps. **The primary reason for this is that DeTrack is specifically designed for object tracking, while DiffusionTracking, derived from DiffusionDet, is more suited for object detection.**\n- |-|-|-|GOT-10k|-|-|-|\n|-|-|-|-|-|-|-|\n|**Denoising paradigm**|**Steps**|**AO**|**SR$_{0.5}$**|**SR$_{0.75}$**|**Flops**|**Speed**|\n|DiffusionTracking|1|71.8|81.0|69.6|120.0G|27fps|\n|DiffusionTracking|2|72.1|81.1|70.0|123.4G| 25fps |\n|DiffusionTracking|4|71.9|81.1|69.3|133.5|16fps|\n|DiffusionTracking|8|73.5|82.9|71.2|147.2G|14fps|\n|DiffusionTracking|12|72.5|81.9|70.2|**162.7G**|10fps|\n|**DeTrack**|12|**77.9**|**86.5**|**74.9**|119.0G|**30fps**|\n\n- |-|-|---|AVisT|---|---|TrackingNet|---|---|LaSOT|---|---|LaSOT$_{ext}$|---|Flops|Speed|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|**Denoising paradigm**|Steps|AUC|OP50|OP75|AUC|Pnorm|P|AUC|Pnorm|P|AUC|Pnorm|P|Flops|Speed|\n|DiffusionTracking|8|54.3|63.5|45.8|83.7|88.6|82.7|69.2|78.3|75.3|46.5|57.0|52.0|**147.1G**|14fps|\n|**DeTrack**|12|**60.2**|**69.1**|**50.2**|***84.5**|**89.1**|**83.6**|**72.9**|**81.7**|**79.1**|**53.6**|**64.4**|**50.4**|119.0G|**30fps**|\n\n>***Q2**.the computation efficiency mainly depends on the capacity of each block instead of the pipeline. In fact, the decoder block in DiffusionDet could be more lightweight than the ViT block in this paper.*\n- As shown in the table below, the DiffusionDet decoder has 22.6 times more computation than the DeTrack denoising block (3.4G vs 0.15G). This is because the DiffusionDet decoder contains 6 transformer layers and a series of fully connected layers. \n- |**Denoisingparadigm**|Step|Flops|\n|-|-|-|\n|DiffusionTracking Decoder|1|3.40G|\n|DiffusionTracking Decoder|8|27.20G|\n|DeTrack Denoising Block|1|0.15G|\n|DeTrack Denoising Block|12|1.80G|\n\n- If we make the DiffusionDet decoder more lightweight, it would lead to a performance drop. The decoder is crucial for both object detection and tracking as it helps us locate the target more accurately. To validate this, we reduced the computation of the DiffusionDet decoder, which resulted in a significant performance drop, as shown in the table below:\n- ||||-| GOT-10k|-||||\n|-|-|-|-|-|-|-|-|-|\n|**Denoisingparadigm**|**Steps**|**Decoder Flops**|**Number of Decoder TransformerLayer**|**AO**|**SR$_{0.5}$**|**SR$_{0.75}$**|**Flops**|**Speed**|\n|DiffusionTracking|8|0.56G|1|65.9|74.6|59.9|121.1G|26fps|\n|DiffusionTracking|8|3.4G|6|73.5|82.9|71.2|**147.2G**|14fps|\n|**DeTrack**|12|2.4G|6|**77.9**|**86.5**|**74.9**|119.0G|**30fps**|\n\n- When we reduced the DiffusionTracking decoder from 6 layers to 1 layer, the AO dropped by 7.6 points, and the SR0.75 dropped by 11.3 points.\n\n>***Q4**. It has no difference in terms of architecture compared to the proposed pipeline when the denoising step is the same.*\n- **Although DeTrack and DiffusionDet can be considered equivalent in terms of denoising steps in some form, this is not our primary contribution.** Our contribution lies in integrating the denoising process into the Vision Transformer (ViT), allowing for feature extraction, feature interaction, and denoising to be performed simultaneously. This approach enables DeTrack to complete denoising in a single forward pass, significantly reducing the computational load of the tracker.\n\n- In contrast, DiffusionTrack, derived from DiffusionDet, requires denoising after feature extraction and interaction, utilizing a Decoder. **While the reviewer suggests that the Decoder could be made more lightweight, the performance impact of the Decoder on object tracking is substantial.** A reduction in the Decoder's complexity results in a noticeable performance decline. As described in Table **Q2**, when the Decoder's computational cost is reduced from 3.4G to 0.56G, the AO drops by 7.6 points and SR$_{0.75}$ drops by 11.3 points. If the computational cost were to match the 0.15G of DeTrack's denoising block, performance might degrade further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate the thorough review provided by all the reviewers. The valuable feedback from the reviewers has significantly contributed to enhancing the quality of our manuscript. We extend our gratitude to Reviewer **JcZZ**, Reviewer **wkvN** and Reviewer **b9Bg** for acknowledging the novelty of our work. Their positive recognition of the innovation in our research is greatly appreciated. 
Based on the comments from the reviewers, I have summarized the strengths of our paper as follows:

- Show competitive performance on these datasets to some extent when compared to other state-of-the-art methods.(Reviewer **JcZZ**)
- Rather than directly regress the target bounding box, or generate a score map indicating target location, the proposed architecture iteratively denoises the previous frame bounding box for tracking. This is novel and quite interesting. The method is also shown to perform very well. (Reviewer **b9Bg**)
- The idea of denoising learning for tracking is novel and well-motivated, and the proposed pipeline is streamlined and promising to lead a new tracking paradigm.(Reviewer **wkvN**)

We have summarized our novelty as follows:

- We propose a novel in-model latent denoising learning paradigm for visual object tracking, which provides a new perspective for the research community. It decomposes the classical explicit recycle denosing process into several denoising blocks and solves the problem with a tracking network in a single forward pass, which is valuable for real applications.

- We present a tracking model including a denoising ViT, comprised of multiple denoising blocks. The denoising process can be completed by progressively denoising through the denoising blocks within ViT. Furthermore, we construct a compound memory in the model that improve the tracking results using visual features and trajectory.

- Experimental results on several popular experiments, including AVisT, GOT-10k, LaSOT, and LaSOT$_{ext}$,  demonstrate that the proposed method achieve competitive results.

We hold the belief that these innovative contributions elevate the value and significance of our research in the area of visual object tracking. 

We intend to heed the insightful suggestions from the reviewers by incorporating additional essential experiments. Additionally, a thorough review of the manuscript will be conducted to rectify any typographical and grammatical errors.

Meta-review: After rebuttal and discussion, all reviewers agree that this manuscript should be accepted: two plain accepts and one weak accept. The main strengths of the manuscript is the novelty of the method, the good performance, and the reproducibility.
Decision: Accept (poster)