DeTrack: In-model Latent Denoising Learning for Visual Object Tracking

Authors: xinyu  zhou
jinglun  li
lingyi  hong
kaixun  jiang
pinxue  guo
weifeng  ge
wenqiang  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 12:44:32

Abstract: Previous visual object tracking methods employ image-feature regression models or coordinate autoregression models for bounding box prediction. Image-feature regression methods heavily depend on matching results and do not utilize positional prior, while the autoregressive approach can only be trained using bounding boxes available in the training set, potentially resulting in suboptimal performance during testing with unseen data. Inspired by the diffusion model, denoising learning enhances the model’s robustness to unseen data. Therefore, We introduce noise to bounding boxes, generating noisy boxes for training, thus enhancing model robustness on testing data. We propose a new paradigm to formulate the visual object tracking problem as a denoising learning process. However, tracking algorithms are usually asked to run in real-time, directly applying the diffusion model to object tracking would severely impair tracking speed. Therefore, we decompose the denoising learning process into every denoising block within a model, not by running the model multiple times, and thus we summarize the proposed paradigm as an in-model latent denoising learning process. Specifically, we propose a denoising Vision Transformer (ViT), which is composed of multiple denoising blocks. In the denoising block, template and search embeddings are projected into every denoising block as conditions. A denoising block is responsible for removing the noise in a predicted bounding box, and multiple stacked denoising blocks cooperate to accomplish the whole denoising process. Subsequently, we
utilize image features and trajectory information to refine the denoised bounding box. Besides, we also utilize trajectory memory and visual memory to improve tracking stability. Experimental results validate the effectiveness of our approach, achieving competitive performance on several challenging datasets. The proposed in-model latent denoising tracker achieve real-time speed, rendering denoising learning applicable in the visual object tracking community.

Main paper: 
 introduction: 
 Visual object tracking is a fundamental task in computer vision, which involves localizing and tracking a specific object in a video given its initial position. It finds broad applications in video understanding, surveillance, and robot navigation [45] . State-of-the-art approaches can be broadly categorized into two classes. The first class [17, 51, 41, 8, 19, 55, 29, 56] directly predicts the bounding box of the tracked target based on image features. The second class [6, 43] , employs the coordinate autoregression framework.While the mainstream methods have achieved prominent success, there are still certain issues to be addressed. Methods based on the image-feature regression framework rely heavily on the matching results between the template and the search region, which cannot utilize positional prior. Meanwhile, based on the autoregressive approach, it is necessary to utilize the bounding box from the previous frame to train the model, which can only utilize the existing bounding boxes in the training set. Therefore, during the testing phase, it may exhibit suboptimal performance for some unseen data.The Diffusion model has achieved significant success in image generation task, allowing the generation of many images not seen in the training set [38] . Inspired by the Diffusion model [26] , we add noise to bounding boxes during training stage. The noisy box can have arbitrary size and position, thereby enhancing the robustness of the model to unseen data during testing stage. As illustrated in Fig. 1 (a) and (b), the diffusion model in image generation task requires multiple iterations of U-Net, while in object detection task, denoising is accomplished through multiple iterations of the decoder. However, tracking algorithms are usually asked to run in real-time, directly applying the diffusion model to object tracking would severely impair tracking speed.Therefore, motivated by DAE [40] , we propose a novel denoising learning paradigm (DeTrack) for visual object tracking that decomposes the denoising learning process in every denoising block with a tracking model. We use templates, search region, and noisy boxes as inputs. During the denoising process, we inject the template and search region as conditions to predict the noises in previous predictions. We repeatedly conduct the conditional denoising process and finally achieve accurate object location prediction. Specifically, as shown in Fig. 1 (c), we propose a novel denoising ViT. We decompose a complete denoising process into several denoising blocks within ViT model and implement every denoising operation with a denoising block. Then the denosing learning process can be implemented in a single forward pass of the tracking model, which can reduce the computational cost drastically. To benefit from the in-context information [21, 16, 9, 28, 20, 22] , we also put the previously predicted bounding boxes into a trajectory memory, and put the templates from previous frame into a visual memory. We use them as additional conditions to help locate objects more accurately.Our contributions can be summarized as follows:• We propose a novel in-model latent denoising learning paradigm for visual object tracking, which provides a new perspective for the research community. It decomposes the classical explicit denosing process into several denoising blocks and solves the problem with a tracking network in a single forward pass, which is valuable for real applications.• We present a tracking model including a denoising ViT, comprised of multiple denoising blocks. The denoising process can be completed by progressively denoising through the denoising blocks within ViT. Furthermore, we construct a compound memory in the model that improve the tracking results using visual features and trajectory.• Experimental results on several popular experiments, including AVisT, GOT-10k, LaSOT, and LaSOT ext , demonstrate that the proposed method achieve competitive results.. 
related work: 
 Visual Object Tracking. The existing visual object tracking methods can be broadly categorized into two main classes. The first class [1, 11, 48, 46, 8, 19, 41, 32, 31, 51, 10, 35, 44, 54] involves directly regressing the bounding box from image features, the second class [43, 6] treats the bounding box as four distinct tokens, employing an autoregressive model to sequentially predict these four tokens.In the first class, deep neural networks are initially used to extract visual features, followed by the design of various prediction heads for regressing the bounding box. Since 2016, some prevalent methods have adopted a two-stream framework, employing siamese networks to separately extract visual features from the template and the search region. One type of prediction head [46, 32, 31, 53] uses a branch to predict the possible location of the target and other branches to predict the corresponding bounding box for that location. Another type of prediction head [10, 48, 17] consists of two branches that predict the coordinates of the top-left and bottom-right corners. Subsequently, OSTrack [51] introduces a one-stream tracking paradigm that combines feature extraction and feature fusion into a single step, achieving a new state-of-the-art performance. For the second class, SeqTrack [6] proposes transforming the bounding box into four tokens, predicting them sequentially in the order of x, y, w, and h. When predicting the bounding box, each box requires four passes through the decoder. Another autoregressive method, ARTrack [43] , is similar to SeqTrack but differs in that it incorporates trajectory information in the input to enhance the model's awareness of trajectories.Denoising Learning. DDPM [26] introduces denoising diffusion learning, which enhances the quality and diversity of generated images by adding noise to and denoising images. Subsequently, denoising learning has experienced explosive growth, being applied in various domains and achieving significant success. In the Super-Resolution field, SR3 [37] leverages DDPM for conditional image generation, employing a stochastic denoising process for super-resolution. Meanwhile, CDM [27] comprises a sequence of multiple diffusion models, each responsible for generating images with progressively higher resolutions. In video generation, the Flexible Diffusion Model (FDM) [23] utilizes a generative model designed for sampling arbitrary subsets of video frames, facilitated by a specialized architecture tailored for this purpose. The Residual Video Diffusion (RVD) model [50] employs an autoregressive, end-to-end optimized video diffusion model. In addition to generative tasks, denoising learning has also found extensive applications in discriminative task. DiffusionDet [4] applies the diffusion model to object detection, utilizing DDIM [38] for denoising. However, this approach still requires multiple passes through the decoder for denoising, impacting inference speed.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In the Limitation5 of the main text, we describe the limitations of our method. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 In this section, we start by formulating the proposed tracking paradigm learned through denoising learning (Section 3.1). Next, we present our overall model architecture (Section 3.2), which includes a proposed denoising ViT, a box refining and mapping module, and a compound memory.Image and Box Inputs. We utilize both visual memory and the search region as conditional inputs c, while introducing noisy boxes x I to predict the true position of the target, where I represents the I-th state in the denoising process. The visual memory stores templates, which are cropped based on previous frames. The search region is cropped based on the current frame and encompasses the area where the target may be present. In training stage, inspired by DDPM [26] , we obtain noisy boxes x I by adding Gaussian noise ϵ to the ground truth box x 0 :EQUATIONwhere ᾱ = T j=0 α j and α j = 1 -β j . β j ∈ (0, 1) is the variance schedule, T is the time step. Optimization for Denoising Learning. We take the visual memory and search region as conditional inputs c, and predict the true target position x 0 from the noisy box x I , p θ (x 0 |x I ), where θ represents the neural network parameters. We aim to maximize the probability p θ that the neural network predicts x 0 , enabling the model to predict the true target position: maximize(p θ (x 0 |x I , c)).(EQUATIONTo maximize p θ , we need to make the predicted x ′ 0 by the network f θ close to the ground truth x 0 :EQUATION)Decomposes the Denoising Process into Multiple Denoising Block within a Model. According to the principle of Markov, we can expand Equation 2 into a Markov chain:EQUATIONIn the traditional Diffusion model [26] , each step p θ (x i-1 |x i , c) is iteratively predicted using a neural network model f θ . However, our denoising paradigm decomposes the iterations of neural network into the iterations of denosing blocks within a neural network,f θ = {d 1 , d 2 , • • • , d l },where each denoising block d l is responsible for predicting a state p θ (x i-I l |x i , c), where l denotes the number of blocks. This allows our model to complete denoising with only a single forward pass of the tracking model.Discussion on the Differences from the Diffusion Model. The proposed denoising learning tracking paradigm is not a diffusion model. (1) In the reverse denoising process of diffusion model, sampling a noise from a standard Gaussian distribution introduces randomness, making it more suitable for generating diverse images in image generation tasks. However, bounding boxes for visual object tracking are deterministic. Therefore, our proposed DeTrack does not involve a sampling process in reverse denoising process, making it more suitable for visual object tracking. (2) Each step of diffusion model is predicted recursively using a neural network. The proposed DeTrack predicts states using denosing blocks within a network (3) The diffusion model requires iterative prediction of neural network, whereas our method only requires a single forward pass through the network. Please refer to the Appendix A.1 for detailed analysis.Inputs representation. As show in Fig. 2 , we use noisy bounding boxes as input and take visual memory and a search region as conditional inputs. Visual memory stores multiple templates. Specifically, gaussian noise is added to the ground truth bounding box to obtain a noisy bounding box{x 1 * I , y 1 * I , x 2 * I , y 2 * I } ∈ R 4×1, where * denotes noise addition, while 1 and 2 respectively denote the upper left corner and lower right corner. Subsequently, the noisy box is mapped to a high-dimensional space by word embedding, resulting in noisy box embedding x I ∈ R 4×C . Additionally, we map templates and the search region to templates embedding z ∈ R Nz×C and search embedding s ∈ R Ns×C by a image embedding, where N z = n × Hz 16 × Wz 16 , N s = Hs 16 × Ws 16 , n denotes the number of templates, H and W represent the height and width of the image, respectively. For details, please refer to Model implement details in Section 4.1.ViT Transformer Block. The specific transformer block structure is the same as the ViT transformer block [13] . Therefore, we only introduce integrating the features of templates and search region within the ViT block. Specifically, we perform attention on image embedding. We first obtain q s (search Denoising ViT (In-model Latent Denoising )x 𝑖-1Denoising Block𝑥 𝐼 1 * , 𝑦 𝐼 1 * , 𝑥 𝐼 2 * , 𝑦 𝐼 2 * 𝑥 𝑖 1 , 𝑦 𝑖 1 , 𝑥 𝑖 2 , 𝑦 𝑖 2 𝑥 𝑖-1 1 , 𝑦 𝑖-1 1 , 𝑥 𝑖-1 2 , 𝑦 𝑖-1 2 𝑥 0 1 , 𝑦 0 1 , 𝑥 0 2 , 𝑦 0 2 … …X 𝐼 = 𝛼X 0 + 𝜀 1 -𝛼 𝜀~𝑁(0, 1)Denoised query), q z (templates query), k s (search key), k z (templates key), v s (search value) and v z (templates value) through linear layer. The image attention is employed to interact and fuse image embedding:EQUATIONwhere [•] denotes concatenation. d is the dimensionality of the key. Denoising Block. As shown in Fig. 2 , the input to the denoising block comprises the noisy box embedding and the search region embedding. These are passed through linear layers to obtain the q xi (box query), k s (search key), and v s (search value). Subsequently, a denoising attention mechanism is employed for the first time of denoising:EQUATIONThen, we incorporate a Feedforward Neural Network (FFN) layer to enhance x ′ i :EQUATIONxEQUATIONFinally, we use two linear layers to predict noise for the second time of denoising. Subtracting the noise from the box embedding yields the result after denoising through a NoisePred module:EQUATIONDenoising is performed through l Denoising blocks. Ultimately, denoising is accomplished with a single forward pass of the denosing ViT, resulting in denoised box embedding x 0 :EQUATION)Box Refining and Mapping. As shown in Fig. 3 (a), we start by applying self-attention to the trajectory and denoised box embedding. We maintain that the current box embedding can only attend to its preceding box embedding by an attention mask in the self-attention, introducing temporal information. Subsequently, the output of self-attention is used as a query for cross-attention with the image features. After undergoing six times of box refining, we compute the similarity between the refined box and word embedding, apply Softmax to obtain probabilities for different positions in the word embedding, and use the position with the highest probability as the bounding box, which is similar to ARTrack [43] .Compound Memory. We design a compound memory that includes both a visual memory and a trajectory memory. The visual memory enhances the model's ability to adapt to changes in the appearance of the target and the environment in the video. Besides, the trajectory memory enables the model to continue tracking the target even in the presence of occlusions or disappearances.Visual Memory. As shown in Fig. 3 [36] , GOT-10k [30] , LaSOT [14] and LaSOT ext [15] .Where * denotes our model only trained on GOT-10k. The best results are highlighted in bold. boxes into a high-dimensional space, we utilize word embedding, similar to Pix2Seq [5] , with the number of bins being 800 and 1200 for DeTrack 256 and DeTrack 384 respectively.Training. Our experiments are conducted on Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz with 252GB RAM and 8 NVIDIA GeForce RTX 3090 GPUs with 24GB memory. In the first stage, there is only visual memory, which randomly samples two frames from the video. The model is trained on full dataests (COCO, GOT-10k, TrackingNet, and LaSOT). A total of 240 epochs are trained, with the learning rate set to 8e-5 for the denoising ViT and 8e-6 for the box refining. The learning rate decreases by a factor of 10 at the 192-th epoch. In the second stage, trajectory memory is introduced to refine the box, and sequential training is adopted. Consecutive frames are sampled from the video, with each frame's prediction result stored in the trajectory memory and updated in a first-in-first-out manner. The training is conducted on three datasets excluding COCO. A total of 60 epochs are trained, with the learning rates decreasing to 4e-6 and 4e-7 for the denoising ViT and box refining, respectively. In the third stage, only IoUNet is trained while other parts are frozen. The learning rate is set to 1e-4, and a total of 40 epochs are trained, with a 10× learning rate decay at the 30-th epoch.For GOT-10k, the learning rate remains consistent with training on the full dataests. In the first stage, we train for 120 epochs, with a 10× decrease in learning rate at the 96-th epoch, followed by training for 25 epochs in the second stage. During the training on GOT-10k, IoUNet is not used. The loss functions is cross-entropy and SIoU, which is the same as ARTrack [43] .Inference. During the testing phase, we use the search region and template as image inputs and initialize the box with the previous box (predicted bounding box of t-1 frame). Additionally, the update interval of the visual memory is set to 5 for t <= 100, doubled every 100 frames until t = 500, and then remains 160. While testing on the GOT-10k dataset, the visual memory is updated directly. For other datasets, the IoU score and confidence score is applied to filter templates. The trajectory memory stores seven bounding boxes, updating with a frequency of every frame. Inference is conducted on an NVIDIA GeForce RTX 3090.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: We will provide the code and model after the paper be accepted Guidelines:• The answer NA means that paper does not include experiments requiring code. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: Our method is visual object tracking, not image generation, and does not pose issues such as generating false information. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
state-of-the-art comparisons: 
 AVisT. The AVisT dataset, as described in [36] , covers a broad spectrum of diverse and demanding situations, encompassing harsh weather conditions like thick fog, intense rainfall, and sandstorms.Our tracker demonstrates outstanding performance on AVisT [36] , a dataset with extreme weather conditions and harsh environments. It outperforms SeqTrack 384 by 2.4% in AUC, substantiating our tracker's excellence in extreme environmental conditions.GOT-10k. GOT-10k comprises a training dataset consisting of 10,000 videos and a testing dataset with 180 videos. There is no overlap between the training and test sets, necessitating trackers to demonstrate robust generalization capabilities towards unseen data. As shown in Tab. 2, our method demonstrates superior performance on the GOT-10k [30] . Our DeTrack 256 achieves a significant improvement in AUC compared to SeqTrack 256 [6] , with increases of 3.0% and 2.4%, respectively. Our DeTrack 384 outperforms the state-of-the-art method ARTrack 384 by 2.4%. This is attributed to the non-overlapping nature of the training and testing sets in the GOT-10k dataset, indicating our method's strong performance on unseen data. The denoising learning paradigm has learned powerful denoising capabilities while facing with arbitrary positions and sizes of boxes.LaSOT. LaSOT is benchmark designed for long-term tracking, featuring a test collection consisting of 280 videos. Our DeTrack256 achieves an AUC of 71.3%, exhibiting performance improvement compared to other methods based on 256 resolution. Additionally, our DeTrack384 also demonstrates state-of-the-art performance, validating the strong competitiveness of our approach in long-term dataset. This is attributed to our compound memory design, which leverages historical trajectory and appearance information to enhance the model's generalization ability on long-term dataset.LaSOT ext . LaSOT ext [15] is an extension of the LaSOT dataset, also categorized as a long-term tracking dataset. It comprises 150 video sequences and encompasses 15 object classes. Our De-Track384 shows significant improvements compared to other methods, with a 1.7% increase in AUC over SeqTrack384 and a 2.4% improvement in P norm . This demonstrates the strong generalization capability of our approach even with extended data, particularly manifesting notable advantages in the accuracy of bounding box center point. Influence of denoising steps. We investigate the impact of the number of denoising iterations on the performance of the tracker. Our proposed In-model latent denoising consists of a total of 12 steps based on denosing blocks, requiring only a forward pass to complete denosing. As shown in Tab.3, the model's performance is nearly zero at the first and second denoising steps because the bounding boxes are still filled with noise. However, there is a significant qualitative improvement in model performance at the eighth denoising step, reaching its peak at the twelfth step. As shown in Fig. 4 , the results improve progressively step by step, consistent with Tab. 3.Analysis of denoising paradigm. Although our method completes denoising with only a single forward pass through the tracking model, it can also be adapted to perform multiple forward passes, similar to traditional Diffusion model [26] . Therefore, we further analyze and compare the multiple forward passes and single forward pass paradigms, as shown in Tab. 4. In DeTrack, the performance of multiple forward passes is not superior to that of single forward pass. Additionally, if denoising is performed similarly to traditional Diffusion models, the computational cost increases significantly.Single forward pass only requires 53.0G FLOPS and achieves a speed of 42 FPS, while multiple forward passes incurs exponentially higher computational costs with a linear decrease in speed.Analysis of the denoising block. As shown in Tab.5, if there is no NoisePred module, AO will decrease by 2.0%, and SR 0.5 will decrease by 2.0%. This demonstrates that noise prediction and gradually subtracting noise are crucial for the model. Furthermore, removing denoising attention leads to further performance degradation, demonstrating that utilizing image features as conditional inputs can also assist in denoising. Moreover, the computational overhead of the denoising block increased by only 1.80G, owing to the fact that the box comprises merely 4 tokens. Thus, even with the addition of denoising attention and NoisePred, this remains a negligible computational burden.Because the memory mechanism is designed to address the challenge of dynamic changes in video, and considering the greater variety of environmental and appearance changes in long video datasets, we chose the LaSOT (long-term tracking dataset, averaging 2448 frames per video) to validate the effectiveness of our memory mechanism. Exploration on the length of the visual memory and the trajectory memory. We firstly explore the impact of different visual memory lengths. As shown in Fig. 5 (a), when the length is only 1, the model's AUC is only 70.2. However, with an increase in memory length, performance gradually improves, reaching its peak at the 3-rd frame. Subsequently, performance declines. This is because when the memory is too short, the model cannot adapt to changes in the target and the environment. Conversely, when the memory is too long, it stores incorrect information. Unlike visual memory, as shown in Fig. 5 (b), trajectory memory does not exhibit a trend of initially rising and then falling with an increase in stored boxes. The performance consistently improves as the number of boxes ranges from 1 to 7. As shown in Fig. 5 (e), we also achieved a performance of 70.2 by removing all memories, which further confirms the effectiveness of our memory.Effects of IoU score and Softmax scorefor visual memory updating. For the update of visual memory, we strive to avoid updating poor templates into our visual memory. This would lead to tracking drift. Therefore, as shown in Fig. 5(d ) keeping IoU score fixed, we conduct an ablation study on different Softmax score values. The study found that an accuracy update can be achieved when the Softmax score is set to 0.9, obtaining 71.3% on AUC. As shown in Fig. 5(c ) keeping Softmax score fixed, the best IoU score is 0.75. When the IoU score threshold is set to 0.85, it leads to a decrease in AUC. It is because the overly strict condition reduces the frequency of visual memory updates. 
limitation: 
 Despite achieving real-time speed and competitive performance, our DeTrack still has certain limitations. Existing tracking methods struggle to recover the target when facing challenges such as object occlusion and out-of-view situations. Although our proposed trajectory memory can assist in target reacquisition after target loss in some cases, further improvements are needed to address challenges like object occlusion and out-of-view scenarios. We will investigate the challenges in these scenarios. 
conclusion: 
 Traditional visual object tracking methods using image-feature regression or coordinate autoregression models faced limitations in handling positional priors and unseen data. Inspired by the diffusion model, we introduced denoising learning to enhance model robustness. Our approach, employing noisy bounding boxes for training, introduces a novel paradigm of denoising learning in object tracking. By decomposing the process into individual denoising blocks within our proposed denoising Vision Transformer (ViT), we achieved real-time performance while maintaining effectiveness.Experimental results demonstrate the efficacy of our method, showcasing competitive performance and rendering denoising learning applicable in the visual object tracking community.while DeTrack innovatively uses noisy bounding boxes (b * ) as input, making it more suitable for visual tracking tasks in complex backgrounds and scenarios with fast-moving objects.In terms of encoding, DDPM and DAE use single-layer feature encoding to obtain z = f θ (x ) ; in contrast, DeTrack employs multi-layer feature encoding with layer-by-layer denoising within the model, resulting in multiple hidden states z i (z 12 , z 11 , . . . , z 1 = f θ (b ) ). This layer-wise denoising approach retains and optimizes target feature information, enhancing robustness to bounding box noise. For decoding, DDPM's decoding target is to restore the noise ϵ θ = g θ (z), while DAE directly decodes the image x θ = g θ (z). DeTrack, on the other hand, decodes to a denoised bounding box b θ = g θ (z 12 ), ensuring high-precision localization of the target bounding box.Regarding the optimization objective, DDPM minimizes the error between generated noise and target noise (ϵ -ϵ θ ), DAE minimizes the error between the denoised image and the original image (x -x θ ), and DeTrack optimizes the error between the denoised bounding box and the original bounding box (b -b θ ), making it more suitable for accurate visual target localization. For inference, DDPM uses a reverse diffusion process to progressively denoise and generate an image, while DAE and DeTrack directly generate denoised results in inference: DAE outputs the image x θ = g θ (z), and DeTrack outputs the bounding box b θ = g θ (z 12 ).Overall, DeTrack's multi-layer feature encoding with internal model denoising, specific decoding approach, and optimization objective enable it to exhibit higher robustness in noisy and complex backgrounds, making it well-suited for target tracking tasks in dynamic and complex scenarios. . According to 7, Predicting the total noise resulted in a decrease of 1.9 in AO, 2 in SR 0 .5, and 2.1 in SR 0 .75,compared to multiple noise predictions. We analyze that this is because predicting the total noise directly is more challenging than predicting it layer by layer. Layer-by-layer denoising allows the model to learn to filter out some noise at intermediate layers before arriving at the final result, rather than achieving it in one step. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: Our paper has no relevant risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite all the assets utilized in our paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: .Table 8 presents an ablation study on different denoising paradigms and step settings evaluated on the GOT-10k dataset with a Vit-Small backbone. We compare performance metrics such as Average Overlap (AO) and Success Rates at two different overlap thresholds (SR 0.5 and SR 0.75 ).The results indicate that multiple forward passes generally yield better performance compared to a single forward pass. Specifically, a step count of 48 achieves the best AO, SR 0.5 , and SR 0.75 values, with scores of 69.4, 78.5, and 63.4, respectively, highlighted in bold in Table 8 . This suggests that while increasing the number of steps from 12 (single forward pass) to 48 improves performance, further increasing to 96 steps does not result in additional gains, possibly due to diminishing returns in iterative refinement or over-smoothing of features. Justification: We accurately describe our main contributions and the covered domains in the abstract1 and introduction1 Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: After rebuttal and discussion, all reviewers agree that this manuscript should be accepted: two plain accepts and one weak accept. The main strengths of the manuscript is the novelty of the method, the good performance, and the reproducibility.

Reviews: review: paper_summary: paper_summary: In this paper, the authors reformulate the visual tracking problem using denoising learning process. Given noisy bounding box coordinates, the conditional denoising tracker model's task is to remove the bounding box noise for accurate target state estimation. For the denoising formulation, they introduce an in-model latent denoising process where they claim that it can be executed with a single forward pass of the model. The effectiveness of their proposed method is validated on various visual tracking datasets.
strength_weakness: strength_weakness: - The claim that the model only require a "single forward pass" seems somewhat misleading since the full latent denoising model contains $l=12$ denoising blocks and require same number of denoising steps for full inference. 

- From my understanding, the motivation for generative models that employ iterative denoising process using Markov chains was that modeling the complex data distribution with multiple modes cannot be achieved with a single Gaussian normal distribution. From this perspective, what is the motivation for modeling visual tracking using this iterative multi-step process, since it seems overly cumbersome and intricate.

- Related to the previous question, how does changing the number of denoising blocks $l$ affect the performance of the tracker? Experimental results in Table 3 does not seem enough since the model is trained with $l=12$ as its default value.
questions: questions: Please refer to the questions in the weaknesses section.
limitations: limitations: The authors included a separate discussions and limitations section in their paper, with adequate explanations and descriptions on the weaknesses and possible future directions.
paper_summary: paper_summary: The paper addresses the task of visual object tracking. The main idea is to pose tracking as a latent denoising problem where the bounding box from the previous frame is denoised to obtain the current frame box. The authors use a ViT to obtain embeddings using the search regions as well as a set of template images as input. The embeddings obtained from each ViT block serves as the conditioning input to a denoiser block that iteratively denoises the noise bounding box estimate. The final box is further refined by a block which takes the trajectory (previous 7 estimated bounding boxes) as input. The authors maintain a dynamic memory of previous templates, based on the reliability of tracking. The method obtains state-of-the-art results on standard tracking benchmarks such as LaSOT, GOT-10k, AVisT. The authors further provide ablation studies and analyze the key components of the method.
strength_weakness: strength_weakness: 1. While the authors present the method as a latent denoising approach motivated by DDPM, it is not fully clear how much of the DDPM theory is applicable to the method in the current form. For instance, how are each of the denoising stages supervised? Do you follow some noise schedule to supervise the output of each block? Unlike in a DDPM where a denoising network is shared and a timestamp is randomly sampled for each training iteration, the proposed approach utilizes a stacked architecture. Thus during training, are inputs to the i’th denoising block the outputs of i+1’th denoising block, or a randomly sampled noise? It would be helpful is the authors provide more details on how the intermediate denoising outputs are supervised.

2. I couldn’t find a clear experimental evidence on the benefit training the model to iteratively denoise the bounding box, instead of say directly predicting the total noise (i.e. the offset from previous to current box), or directly regressing the new box, while keeping the rest of the architecture identical. While the authors provide an ablation study in Table 5 on impact of denoising blocks, it seems to only analyze the impact of architectural choices in the denoising block (please correct me if I’m missing something here). I believe such a comparison is important the judge the impact of the core idea of doing iterative denoising.

3. There are some missing details for the training, especially related to training of visual and trajectory memory blocks. Do you perform a sequence based training to simulate the online tracking scenario? If not, how are the templates for the visual memory, as well as the trajectory obtained? Do you just use previous frame boxes (after adding some noise?), do you take consecutive frames from a video or sample randomly from a video?
questions: questions: 1. As stated in the weaknesss 1, please provide details on how the intermediate denoising outputs are supervised.

2. Related to weakness 2, what is the performance obtained by a model trained to directly predict the total noise, while keeping the rest of the architecture and training identical?

3. Please provide more details on the model training, as stated in weakness 3.

4. How many templates are stored in the visual memory? Is it 3? Do you always keep the first frame in the memory?

5. In table 4, when running the denoising process multiple times, how was the model trained? Do you just run the model corresponding to "Single forward pass denoising" multiple times, or train a new model in a DDPM style manner? Instead of using the same ViT-B architecture to do repeated denoising, what happens if you use a smaller model (ViT-S or say even small with only a couple of Transformer blocks), but apply it multiple times?

6. In table 5, what exactly does not doing"NoisePred" correspond to? What is the model architecture in this case?

7. It appears that the tracking performance consistently improves with using larger trajectory memory. Did you experiment beyond using 7 previous frames? Would using a larger trajectory cause significant latency overhead?

8. There seems to be a typo in line 114, in the definition of alpha. 

9. Tracking datasets such as LaSOT can contain snippets with long occlusions. How does the model handle such cases? Do you simulate occlusions / out-of-frame scenarios during training to ensure that the denoising process doesn't diverge? In the limitations, the authors state that the trajectory memory can help to recover from occlusions. However since it contains only 7 previous frames, I'm not sure if that's sufficient to recover from such challenging cases since they can last longer. Does the visual memory help the tracker to redetect the object after long occlusions?
limitations: limitations: The authors briefly discuss the limitations.
paper_summary: paper_summary: The paper introduces bounding box denoising to visual object tracking. To make it more suitable for tracking, the paper devises an in-model denoising pipeline that reformulate the repetitive sampling process as multiple ViT blocks in one forward pass. It also proposes a compound memory strategy to assist long-term tracking with visual feature and trajectory cues. The paper provides extensive results to validate the efficacy of its designs. The proposed method, named DeTrack, achieves competitive performance across multiple datasets.
strength_weakness: strength_weakness: W1) It is unfair to claim that the pipeline of DiffusionDet (Fig. 1(b)) is inefficient for tracking. The advantage compared to the former pipeline should be explained more meticulously due to the following reasons:

- Frist, for Fig. 1(b), the computation efficiency mainly depends on the capacity of each block instead of the pipeline. In fact, the decoder block in DiffusionDet could be more lightweight than the ViT block in this paper.

- Second, the recycling process of DiffusionDet equals to cascading several denosing blocks that share weights. Therefore, it has no difference in terms of architecture compared to the proposed pipeline when the denoising step is the same.

- Third, as indicated in Table 3, the proposed pipeline has the potential to further improve its performance by using more denoising steps. However, while DiffusionDet allows inferring with an arbitrary number of steps, the proposed pipeline requires retraining more blocks and fails to benefit from such flexibility.

W2) The state-of-the-art comparison on TrackingNet, which is a representative benchmark in tracking, is missing. It should be easy to evaluate the proposed tracker on TrackingNet since the training configuration has been already aligned with other baselines. It would be great if the performance can be reported.

W3) Some minor issues in writing:

- The records in Table 2 needs to be double checked. For example, the proposed tracker does not achieve the best performance in $\text{P}_{\text{Norm}}$ on LaSOT, but it is marked in **bold**.

- When introducing the motivation, the authors claim that diffusion is more robust to unseen data. Though the claim is generally true, but it is expected to cite some related references accordingly.

- In Line 252, it is a little bit confusing to say "reaching its peak at the twelfth step" as there is no step13. Words like "perform best" could be more appropriate here.
questions: questions: Q1) Have the authors ever applied the pipeline of DiffusionDet directly to tracking? If practical, how is its performance?

Q2) From Fig. 4, it seems that the bounding box is randomly initialized. Will the randomness affect the tracking performance much between different runs? While this is not a limitation, I am wondering if starting from a fixed anchor (whether calculated by statistics or learned as Sparse R-CNN) would help. Is it a choice that the authors have already considered?
limitations: limitations: The authors have discussed the limitations.

Comments: title: Official comment by reviewer wkvN
Further response serve to adress reiviewer's concerns（Part 2/2）
Further response serve to adress reiviewer's concerns（Part 1/2）
Further response serve to adress reiviewer's concerns
comment: We sincerely appreciate the reviewer's response and are very grateful for the reviewer's recognition of our work. Additionally, we will no longer claim that our model requires a single forward pass. Your suggestion is very helpful and will aid us in improving the manuscript.
I thank the authors for their detailed response to my questions, and the additional details provided by the authors were very helpful.
Also, based on the concerns from the other reviewers and the authors' response to address these issues, I am inclined to raise my rating to "accept".
However, aside from the novel approach proposed in the manuscript, I also suggest the authors to reconsider their claim that the proposed algorithm requires only "single forward pass", since it can confuse the readers.
Thank you very much for your response and for recognizing our work. 
> ***Q1**.  When generating the results in Table 3 or Figure 4, are you just reusing the final "mapping" block to generate the intermediate boxes (i.e. after steps 1, 2, 3 ...) from the denoised latents?*

- Yes, we use the mapping layer to project the denoised boxes into the image coordinate space. I will include these details in the paper to help improve the quality of the manuscript. Thank you very much for your question.

> ***Q2**. I would suggest that the authors try to incorporate the additional details provided in the rebuttal regarding the denoising process, sequences training etc in the manuscript since this will be very helpful.*

- Based on your suggestions, we will incorporate the additional details regarding the denoising process, sequence training, and other relevant aspects into the manuscript, as this will greatly enhance the quality of our paper.

>***Q3**.  I think it would also be better to motivate the method from the point of denoising autoencoders rather than DDPM.*
- We will also describe our method from the perspective of denoising autoencoders rather than DDPM.

> ***Q4**. I would suggest to tone down on the claims of "single forward pass".*
- Additionally, we will reduce the emphasis on the concept of a single forward pass and instead focus on highlighting our denoising model.
Thank you for your response, and we sincerely appreciate your recognition of our work.
I thank the authors for providing a detailed rebuttal. It answers most of my concerns. Hence I'm raising my rating to "Accept".

I would suggest that the authors try to incorporate the additional details provided in the rebuttal regarding the denoising process, sequences training etc in the manuscript since this will be very helpful. I think it would also be better to motivate the method from the point of denoising autoencoders rather than DDPM. Additionally I would suggest to tone down on the claims of "single forward pass". A standard diffusion model can also be viewed as a "single forward pass" model which just stacks multiple blocks with shared weights.

One additional question: The authors state that they do not provide any auxiliary supervision to the intermediate denoising stages. Thus when generating the results in Table 3 or Figure 4, are you just reusing the final "mapping" block to generate the intermediate boxes (i.e. after steps 1, 2, 3 ...) from the denoised latents?
Dear authors,

Thanks for your prompt reply. It is a good paper that should be recommended. I will maintain my initial rating as I think it is high enough.
Thank you for your valuable feedback. Please allow me to clarify further to alleviate your concerns.
> ***Q1**. What is the DeTrack Encoder in the experiment for Q1? How many interaction blocks are used in it?*
- DeTrack uses ViT-B (12 blocks) along with an additional 3 Transformer blocks as our encoder for feature interaction and feature extraction. This is a one-stream architecture, which allows for both feature extraction and feature interaction to be performed simultaneously.

> ***Q2**. From the visualization in Fig. 4, if the denoising starts from a reliable prior, why are the bounding boxes almost random after the first block? It appears that the first block serves as an adverse step in the denoising process that destroys the prior information.*
- We sincerely apologize for the confusion caused. To demonstrate the denoising effect, the bounding boxes in Figure 4 were visualized by sampling from Gaussian noise, which introduced an element of randomness. We will provide a detailed explanation of this in the revised version to enhance the quality of the manuscript.

> ***Q3**.  I recommend clarifying these differences earlier in the paper. In addition, I hope the discussions and experiments in the rebuttal can be incorporated as supplementaries of revision, which would be quite helpful for readers.*
- Thank you very much for your suggestions. Based on your feedback, we  will introduce the table ( Disccusion with reviewer wkvN) discussing the differences and similarities between DDPM, DeTrack, and DAE at an earlier section.
- Additionally, as per your recommendation, we will include the disccusion and experimental analysis from the rebuttal in the supplementary materials to aid readers' understanding.

Once again, thank you for your valuable suggestions.
Dear authors,

Thanks for your detailed responses and adequate experiments, which have addressed most concerns of mine. After reading the rebuttal and comments from other reviewers, I have some follow-up questions:
- What is the DeTrack Encoder in the experiment for Q1? How many interaction blocks are used in it?
- In authors' response to Q6, the previous box is used as the input, typically centered in the search region with a similar bounding box size to the object in the current frame. That raises another question. From the visualization in Fig. 4, if the denoising starts from a reliable prior, why are the bounding boxes almost random after the first block? It appears that the first block serves as an adverse step in the denoising process that destroys the prior information.
- As other reviewers have suggested, there are some confusing expressions in the descriptions of denoising. While the authors have highlighted the differences from the Diffusion Model on Page 4, I recommend clarifying these differences earlier in the paper. In addition, I hope the discussions and experiments in the rebuttal can be incorporated as supplementaries of revision, which would be quite helpful for readers.
> ***Q9**. It appears that the tracking performance consistently improves with using larger trajectory memory. Did you experiment beyond using 7 previous frames? Would using a larger trajectory cause significant latency overhead?*
- Based on your suggestion, we conducted experiments with a longer trajectory memory(On LaSOT dataset). The experimental results are as follows:
- | **Length of trajectory memory**  |  AUC| Pnorm|P|
| -| -------- | ---------- | ----------- |
| **7**   | 71.3| **80.1**| 76.8 |
| **8**   |71.3 | 79.8| 76.9  |
| **9**   |70.7 | 79.0 | 76.0|
| **10** | **71.6** |**80.1** |**77.2**|
- Increasing the length of the trajectory memory indeed helps improve tracking performance. We will explore more details regarding trajectory memory length and include these findings in the paper. Thank you very much for your suggestion; it has been very helpful in enhancing our manuscript.


> ***Q10**. There seems to be a typo in line 114, in the definition of alpha*

- We will correct the typos in our subsequent revisions. Thank you very much for your suggestions, which are very helpful for improving our manuscript.

> ***Q11**.Tracking datasets such as LaSOT can contain snippets with long occlusions. How does the model handle such cases? Do you simulate occlusions / out-of-frame scenarios during training to ensure that the denoising process doesn't diverge?*

- Yes, during the training phase, we simulated out-of-view scenarios. Our bounding box coordinates are first normalized to the range -0.5 to 1.5, where the range 0-1 represents coordinates within the search area, and -0.5 to 0 and 1.0 to 1.5 represent coordinates outside the search area to simulate some out-of-view situations. I

> ***Q12**. In the limitations, the authors state that the trajectory memory can help to recover from occlusions. However since it contains only 7 previous frames, I'm not sure if that's sufficient to recover from such challenging cases since they can last longer. Does the visual memory help the tracker to redetect the object after long occlusions?*

- In the limitations section, we mention that our trajectory memory can recover the target in some cases, alleviating the out-of-view problem, but cannot completely solve it. This is a common issue with local trackers, including OSTrack[2], Mixformer[3], and others, and is also a limitation of our DeTrack. We will further investigate this issue in the future. 
- Visual memory also provides some benefit, as it can offer visual reference information.

[1]. Vincent P, Larochelle H, Bengio Y, et al. Extracting and composing robust features with denoising autoencoders[C]//Proceedings of the 25th international conference on Machine learning. 2008: 1096-1103.

[2] Ye B, Chang H, Ma B, et al. Joint feature learning and relation modeling for tracking: A one-stream framework[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 341-357.

[3] Cui Y, Jiang C, Wang L, et al. Mixformer: End-to-end tracking with iterative mixed attention[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 13608-13618.
> ***Q5**. Do you perform a sequence based training to simulate the online tracking scenario? If not, how are the templates for the visual memory, as well as the trajectory obtained? Do you just use previous frame boxes (after adding some noise?), do you take consecutive frames from a video or sample randomly from a video?*

- The training of our model is divided into three stages. In the first stage, there is no trajectory memory, only visual memory, which randomly samples two frames from the video. This is consistent with previous visual object tracking methods. In the second stage, trajectory memory is introduced, and sequential training is adopted. Consecutive frames are sampled from the video, with each frame's prediction result stored in the trajectory memory and updated in a first-in-first-out manner. The third stage involves training the IOU head. In this stage, the tracker is fixed and not trained; only the IOU head is trained to evaluate the quality of the template. **We greatly appreciate your suggestion, we will include these training details in the paper in the future， as it has helped us improve our manuscript further.**

> ***Q6**. How many templates are stored in the visual memory? Is it 3? Do you always keep the first frame in the memory?*
- The length of the visual memory bank is always 3 frames, which was experimentally verified to achieve the highest performance in Section 4.4. 
- The first frame is always fixed, as described in Section 3.3 on visual memory, where the first frame is fixed and only the dynamic frames are updated.

> ***Q7**. in table 4, when running the denoising process multiple times, how was the model trained? Do you just run the model corresponding to "Single forward pass denoising" multiple times, or train a new model in a DDPM style manner? Instead of using the same ViT-B architecture to do repeated denoising, what happens if you use a smaller model (ViT-S or say even small with only a couple of Transformer blocks), but apply it multiple times?*

- We run the model corresponding to "Single forward pass denoising" multiple times. Based on your suggestion, we retrained the model similarly to DDPM(**DiffusionTracking**). The results are shown in the table below:
- |-|-|-|GOT-10k|-|-|-|
|-|-|-|-|-|-|-|
|**Denoising paradigm**|**Steps**|**AO**|**SR$_{0.5}$**|**SR$_{0.75}$**|**Flops**|**Speed**|
|DiffusionTracking|1|71.8|81.0|69.6|120.0G|27fps|
|DiffusionTracking|2|72.1|81.1|70.0|123.4G| 25fps |
|DiffusionTracking|4|71.9|81.1|69.3|133.5|16fps|
|DiffusionTracking|8|73.5|82.9|71.2|147.2G|14fps|
|DiffusionTracking|12|72.5|81.9|70.2|**162.7G**|10fps|
|**DeTrack**|12|**77.9**|**86.5**|**74.9**|119.0G|**30fps**|

- |-|-|---|AVisT|---|---|TrackingNet|---|---|LaSOT|---|---|LaSOT$_{ext}$|---|Flops|Speed|
|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|
|**Denoising paradigm**|Steps|AUC|OP50|OP75|AUC|Pnorm|P|AUC|Pnorm|P|AUC|Pnorm|P|Flops|Speed|
|DiffusionTracking|8|54.3|63.5|45.8|83.7|88.6|82.7|69.2|78.3|75.3|46.5|57.0|52.0|**147.1G**|14fps|
|**DeTrack**|12|**60.2**|**69.1**|**50.2**|**84.5**|**89.1**|**83.6**|**72.9**|**81.7**|**79.1**|**53.6**|**64.4**|**50.4**|119.0G|**30fps**|

- Based on your suggestion, we retrained the model with ViT-S, the table as show below:

- | **Denoising paradigm**  | Steps | AO       | SR$_{0.5}$ | SR$_{0.75}$ |
| ----------------------- | ----- | -------- | ---------- | ----------- |
| **Recycle denoising**   | 96    | 68.9     | 78.2       | 63.2        |
| **Recycle denoising**   | 48    | **69.4** | **78.5**   | **63.4**    |
| **Recycle denoising**   | 24    | **69.4** | 78.0       | 63.0        |
| **Single forward pass** | 12    | 69.1     | 78.3       | 62.9        |

- It can be observed that there is a performance gap between ViT-S and ViT-B. This is because, in DeTrack, the ViT serves not only for denoising but also plays a crucial role in feature extraction and feature interaction. ViT-S, having a smaller model size, therefore has weaker representational capacity, and ViT-S lacks MAE pre-training, which is critical for tracking, as demonstrated in OSTrack[2]. Performance improves with multiple iterations of denoising on ViT-S, reaching its peak after 48 denoising steps.

> ***Q8**. In table 5, what exactly does not doing"NoisePred" correspond to? What is the model architecture in this case?*
- No NoisePred’ indicates the absence of a noise prediction module, where the model directly predicts the bounding box. The overall process is more similar to DAE [1]. For specific details, please refer to Q4.

[1]. Vincent P, Larochelle H, Bengio Y, et al. Extracting and composing robust features with denoising autoencoders[C]//Proceedings of the 25th international conference on Machine learning. 2008: 1096-1103.
>***Q5**.while DiffusionDet allows inferring with an arbitrary number of steps, the proposed pipeline requires retraining more blocks and fails to benefit from such flexibility.*
- We acknowledge that increasing the denoising steps would require retraining the network. However, our design prioritizes efficiency and targeted performance. By carefully optimizing the existing 12 denoising modules, we ensure that the model is highly efficient with the predetermined number of denoising steps.
- While DiffusionDet offers greater flexibility with its iterative approach, our method provides a computationally efficient alternative, particularly suitable for resource-constrained scenarios. 
- Overall, our approach strikes a balance between computational efficiency and flexibility, making it suitable for specific use cases where reducing the number of forward passes is crucial.

>***Q6**. Will the randomness affect the tracking performance much between different runs? While this is not a limitation, I am wondering if starting from a fixed anchor would help. Is it a choice that the authors have already considered?*

Due to the randomness introduced by Gaussian noise, different runs can indeed result in varying performance. Therefore, during the testing phase, we have used a fixed box (previous box). The previous box provides a prior on the target's position, leading to the highest performance.
- |-|GOT-10k|-|-|
|-|-|-|-|
|**Boxtype**|**AO**|**SR$_{0.5}$**|**SR$_{0.75}$**|
|Noisy box(First run)|74.8|83.3|71.0|
|Noisy box(Second run)|75.6|84.5|73.4|
|Previous box(Fixed anchor)|**77.9**|**86.5**|**74.9**|

>***Q7**. It would be great if the performance can be reported.*
- Based on your suggestion, we tested the TrackingNet dataset, and the results are shown in Table **Q1**. We will report these results in the revised version of the paper. Thank you very much for your input; it has been extremely helpful in improving our manuscript.

>***Q8**. Some minor issues in writing*
- Thank you for your valuable feedback. We appreciate your comments on the minor issues in writing. We have carefully reviewed the manuscript and addressed these issues to improve the clarity and readability of our paper.