Efficient Motion Prompt Learning for Robust Visual Tracking
Authors: Jie Zhao
Xin Chen
Yongsheng Yuan
Michael Felsberg
Dong Wang
Huchuan Lu
Venue: ICML 2025 poster (2025)
Keywords: visual object tracking
temporal encoding
prompt learning
Primary area: applications->computer_vision

Abstract: Due to the challenges of processing temporal information, most trackers depend solely on visual discriminability and overlook the unique temporal coherence of video data. In this paper, we propose a lightweight and plug-and-play motion prompt tracking method. It can be easily integrated into existing vision-based trackers to build a joint tracking framework leveraging both motion and vision cues, thereby achieving robust tracking through efficient prompt learning. A motion encoder with three different positional encodings is proposed to encode the long-term motion trajectory into the visual embedding space, while a fusion decoder and an adaptive weight mechanism are designed to dynamically fuse visual and motion features. We integrate our motion module into three different trackers with five models in total. Experiments on seven challenging tracking benchmarks demonstrate that the proposed motion module significantly improves the robustness of vision-based trackers, with minimal training costs and negligible speed sacrifice. Code is available at https://github.com/zj5559/Motion-Prompt-Tracking.

Efficient Motion Prompt Learning for Robust Visual Tracking
Jie Zhao' Xin Chen?
Yongsheng Yuan Michael Felsberg" 3
Dong Wang'
Huchuan Lu'
Abstract
Due to the challenges of processing temporal in- formation, most trackers depend solely on visual discriminability and overlook the unique tempo- ral coherence of video data. In this paper, we propose a lightweight and plug-and-play motion prompt tracking method. It can be easily inte- grated into existing vision-based trackers to build joint tracking framework leveraging both motion and vision cues, thereby achieving robust track- ing through efficient prompt learning. A motion encoder with three different positional encodings is proposed to encode the long-term motion tra- jectory into the visual embedding space, while fusion decoder and an adaptive weight mech- anism are designed to dynamically fuse visual and motion features. We integrate our motion module into three different trackers with five mod- els in total. Experiments on seven challenging tracking benchmarks demonstrate that the pro- posed motion module significantly improves the robustness of dision-based trackers, with mini- mal training costs and negligible speed sacrifice. Code is available at https: / tuub com/ 2 2
Backkoone Relation (Smilariy leaming)
Hisad
Vidus
Misian
Vision besel Trocking Faanwork
Human Tradking Paodign
Tesphat taae,
Bockbane & Relation (Simibrity Itamnin)
Sena irage
Fasion Deceder
Head
Tajedary
Prampt Eacader
(Motioa Embeddingg
(o) Tracking Framework with Motion Pronpe (Ours)
Figure I. Illustration different tracking paradigms. Our plug- and-play method enables visual trackers to benefit from motion prompts, making them more akin to the human tracking paradigm.
on the discriminative ability of visual models but overlook the crucial temporal coherence in videos. The temporal consistency of these trackers can be only reflected in deter- mining the next local search region by the predicted target position. As shown in Fig. I(a), due to inherent limita- tions of appearance cues, these vision-based trackers are prone to tracking drift when confronted with complex chal- lenges, like distractors, severe occlusion, and so on. Unlike relying solely on visual information, the human tracking paradigm (Ramachandran, 1985; Sokhandan & Monadjemi, 2024) seamlessly integrates both vision and motion cues. As shown in Fig. 1(b), humans can effortlessly identify the specified target among multiple visually similar objects by discerning its motion patterns from trajectories.
wwwcoompt-rraceteeeeee Tracking.
1. Introduction
However, due to the blend of object and camera movement, the irregularity observed motion patterns makes it chal- lenging to incorporate temporal coherence into tracking frameworks. Recent state-of-the-arr (SOTA) trackers ( (Wei et al., 2023; Bai et al., 2024; Zheng al., 2024) start ex- ploring effective temporal mechanisms to enhance track- ing accuracy, but sequential training (Kim et al., 2022) is required for these methods, which considerably increases computational and memory demands. Beyond the substan- tial training costs, constrained computing resources will also limit the ability of these models perceive long temporal information. In this paper, we post question: Is heavy se- quential training necessary for trackers to capture temporal relations? We answer this question by proposing training- efficient Motion Prompt-base Tracking module (MPT), which demonstrates that sequential training is unnecessary for temporal-related tracking.
Given sequence and arbitrary object with interest, visual object tracking (VOT) aims to locate this specified object in each subsequent frame. As one of the fundamental com- puter vision tasks, VOT has been developed rapidly over the past decade. Mainstream trackers (Li et al., 2019; Bhat et al., 2019; Zhao et al., 2022b; Chen et al., 2023) perceive VOT as a visual matching problem between a pair of dis- crete image patches, including template and local search image cropped from the initial and current frames, respec- tively. These vision-based tracking frameworks mainly rely
'Dalian University of Technology *City University of Hong Kong 'Linköping University. Correspondence to: Dong Wang <wdice @ dlut.cdu.cn">
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficient Motion Prompt Learning for Robust Visual Tracking
Specifically, our MPT module can be flexibly integrated into existing vision-based trackers achieve joint tracking based on both visual and motion cues. Within lightweight frame- level fine-tuning, our method can enhance various base- line trackers to achieve comparable performance to those sequential-based SOTA methods. As shown in Fig. I(c), our joint tracking framework contains a fixed iision-based tracker and several additionally proposed motion modules. The prompt encoder (the orange block) independently codes historical trajectories into the visual embedding space by three different positional encodings. A fusion decoder (the green block) is then employed to dynamically fuse vi- sual and motion features by a Transformer network and an adaptive weighting mechanism. Furthermore, we adopt prompt learning to integrate our motion modules into vision- based SOTA trackers more efficiently and flexibly. We develop our MPT ing to learn the ability for dynamic adaptation to baseline visual trackers. All parameters of the baseline are fixed during training, enabling our method to have fast training with minimal memory requirements. By integrating mo- tion prompts from object trajectories, our MPT can sustain robust tracking under challenging scenarios, reflecting tracking paradigm more akin to human behavior.
et al., 2017; Dosovitskiy et al., 2021) to obtain the fused features of the template and search region, which is then used to make the final prediction. Due to global correlation and deep fusion, these trackers lead to improved accuracy compared to similarity- methods. Despite achieving superior performance on existing benchmarks, such solely vision-based matching strategies encounter difficulties in complex visual environments. To alleviate this bottleneck, our work proposes general tracking module based mo- tion prompts. By incorporating temporal motion cues into these based trackers, we enrich the information avail- able to the head network for predictions, thereby signif- icantly improving the tracking robustness in challenging scenarios.
Temporal-related Tracking. Beyond visual information, some works have investigated the significance of temporal information for robust tracking. Some traditional track- ers (Weng et al., 2006; Zhang et al., 2014) primarily utilize kalman filtering to predict the next motion step of the target. However, they assume that the target undergoes regular mo- tion patterns, which usually do not align with the complex tracking scenarios. The lack of deep visual representations also leads substantial lag in their performance.
plug-and-play prompt module, aim-
In the era of deep learning, some methods (Bhat et al., 2019; Zhao et al., 2022a; Yan et al., 2021; Cui et al., 2022; Liu et al., 2024) employ update mechanisms for models or tem- plates to capture the temporal changes of visual information, these methods primarily depend on richer appearance infor- mation to enhance tracking. TCTrack++ (Cao et al., 2023) and ODTrack (Zheng et al., 2024) employ heavy temporal mechanisms, which implicitly integrate temporal informa- tion via propagating consecutive visual features. Different from them, our lightweight module directly encodes and integrates motion trajectories into the visual tracking frame- work, offering a fresh perspective. Our experiments with the template update tracker SeqTrack (Chen et al., 2023) also demonstrate that benefits derived from motion and updates are complementary rather than conflicting.
Our main contributions can be summarized as follows:
We propose a plug-and-play motion prompt tracking module (MPT), which can be flexibly integrated into various tracking models to capture and utilize temporal coherence from historical trajectories.
Efficient prompt learning is employed to fine-tune our method, which unbinds the necessity of heavy sequen- tial training for temporal learning, thereby releasing huge requirements of training resources.
We integrate our MPT into three visual trackers across five models. Experiments on seven benchmarks demon- strate that our MPT enhances the robustness of existing rision-based trackers in challenging scenarios, within minimal training costs and negligible speed sacrifice.
Recently, Alireza et al. (Sokhandan & Monadjemi, 2024) adopt optical flows to represent motion vectors, which raises computational burdens. SLT (Kim et al., 2022) proposes a sequential training method, incorporating motion cues into training by locating short video clips. But it still fol- lows vision- tracking in inference. Implicitly learning temporal correlations among consecutive frames is also a complex and computationally intensive task. ARTrack (Wei et al., 2023) and its extension (Bai et al., 2024) share a simi- lar motivation with ours, integrating temporal information into tracking by individually encoding the motion trajectory. However, its motion encoding is tailored for autoregres- sive models, which limits its applicability other tracking frameworks. Autoregressive model and sequential train-
2. Related Work
In this section, we review mainstream visual trackers, temporal-related trackers, and prompt learning, respectively.
Visual Tracking, Existing trackers commonly adopt vi- sual matching strategy to model the tracking task. Most earlier trackers (Bertinetto et al., 2016; Dai et al., 2019; Li et al., 2019) employ correlation operations to compute the similarity map between visual features of the template and search region. TransT (Chen et al., 2021) and subsequent Transformer- baased works (Ye et al., 2022; Lin et al., 2022; Chen et al., 2023) employ Transformer networks (Vaswani
2

---

Efficient Motion Prompt Learning for Robust Visual Tracking
Visual embeddings
Fusion of Visual
Fused embeddings
Template image
Backhone & Relation
and Motion Cues
Search image
(5. 20F, C)
Head
Fusion Decoder
Adaptive Weight
Historical Triectory
Cross Attn.
Mechanism
WMI
RTw
loU GT,
Rep. Tokens
Head
Motion Embedding
MLP
Kfu Min. MI
Rep. Tokens
RT, RT...
Heady
BM
Motion
Cross Attn.
Motion
(h 2..37. C
AT-
Tokens
Vision Bwton-Moon Motion
Motion flow
1
Encoder
Vision flow
Self Attn.
Weight flow
Spatial
Point
Temporal
+2
Backpropagation
PE embedding
PE
Weighted average
Figure 2. Pipeline of our motion prompt-based tracking method. The historical trajectory is first embedded into the visual embedding space by our motion encoder, and then fused with the visual embedding using the proposed fusion decoder. An adaptive weight mechanism is employed to further dynamically adjust vision and motion cues. Ultimately, the obtained fused embedding is used to make a robust tracking prediction by the tracking head Headre.
ing also increase memory and computational costs. Lim- ited hardware memory will further restrict the model from perceiving long trajectories. In comparison, our method employs a low-cost prompt learning approach to leverage longer motion trajectories for guiding existing trackers, of- fering greater flexibility and versatility.
still treated as a discrete image processing task. In contrast, our MPT leverages the temporal consistency of videos by incorporating continuous object trajectories into the visual tracking framework, offering new perspective on employ- ing prompt learning the tracking field. Our method proves to be more effective in addressing visually challenging narios, such occlusions and distractors.
Prompt I Learning, Prompt learning is an emerging machine learning paradigm, garnering particular attention in the field of natural language processing (Liu et al., 2023b). Its core idea involves interacting with the model via task-related prompts rather than directly adjusting model parameters. In computer vision and multi-modal domains, prompt learning proves to be more efficient and flexible than traditional fine- tuning methods. It shows success in various tasks, including visual recognition (Jia et al., 2022), lasion-language under- standing (Liu et al., 2023a), dense prediction (Chen et al., 2022b), and video understanding (Ju et al., 2022).
3. Method
3.1. Revisit of Vision-based Tracking Framework
Mainstream trackers formulate the tracking problem as B² = (leadTr(ys..22 X*)).
(1)
A visual encoder (-) is commonly employed embed both the template image Z and the search image X pro- ducing the discriminative visual representations, like sim- ilarity maps (Li et al., 2019; Bhat et al., 2019) or fused features (Cui et al., 2022: Chen et al., 2022a) of image pairs. The final tracking result B' for the f-th frame is predicted by a tracking head module, Headm. Some trackers (Ye et al., 2022; Yan et al., 2021) employ a lightweight head to directly predict coordinates, whereas other methods (Chen et al., 2023; Wei al., 2023) implement the head module as a heavy decoder to tackle the sequential coordinate pre- diction problem. For these vision-based tracking methods, powerful visual encoder y is necessary to achieve robust tracking. However, good visual models need to achieve in- variant representations to appearance changes for the same
In the field of visual tracking, in addition to the aforemen- tioned ARTrack and its extension, which adopt trajectories as prompts by employing sequential training and autoregres- sive models, ProTrack (Yang et al., 2022) and ViPT (Zhu et al., 2023) utilize prompt learning to achieve superior per- formance for multi-modal tracking, yet they do not exploit the potential of prompt learning in general visual track- ing. PromptVT (Zhang et al., 2024) can be treated as an appearance- prompt method. It efficiently enhances appearance features using dynamic appearance informa- tion, thereby enabling the model to be robust to appearance changes during tracking. However, the tracking problem is
3

---

Efficient Motion Prompt Learning for Robust Visual Tracking
object, while maintaining distinctive for distractors. Ac- quiring such balanced discriminative visual representations poses a challenging learning task for visual models. In addi- tion, when faced with occlusion, illumination changes, or distractors, visual information often turns out to be unreli- able, leading to tracking drift in these vision-based trackers.
(1) Spatial positional encoding. Inspired by SAM (Kirillov et al., 2023), we represent each bounding box using the normalized coordinates of two corner points, namely, the top-left and bottom-right points. A Gaussian- PE (Tancik et al., 2020) is adopted to map 2-dimensional point coordinates into C-dimensional spatial vectors, align- ing the motion embedding with the visual embedding in terms of dimensions.
spatial
In this paper, we propose a motion prompt-based tracking method, which can integrate motion cues into vision-based baseline trackers by prompt learning. This joint tracking framework incorporating both motion and vision cues can adaptively extract complementary features, thereby enhanc- ing the robustness of tracking predictions, particularly in the face of complex challenges.
(2) Point embedding. After being mapped by the spatial PE, the trajectory of T frames is transformed to a group of motion tokens E 22TxC_ To distinguish between the two corner point types for each token, we introduce two learnable point embeddings and add them to their respective motion tokens. This enables the model to perceive the point type of each motion token.
3.2. Overview of Motion Prompt Tracking
Our tracking method with motion prompts is formulated as
(3) Temporal positional encoding. To establish temporal orders for motion tokens, we incorporate a learnable tempo- ral PE into each token. Tokens from the same frame share identical temporal PE. Since the motion information closer to the current frame is typically more crucial in a long-term trajectory, we initialize this temporal PE in a non-linear manner, formulated as follows:
B" (Dy
mm ( (B--T=-1))).
(2)
In addition to the visual encoder (t). we employ a motion encoder (-) embed the historical trajectory B into the latent embedding space consistent with visual fea- tures. The trajectory is denoted as consecutive bounding boxes of the target from previous T frames. A decoder (-)() is then utilized to fuse motion and visual features, and the resulting fused representation will be forwarded to the downstream head to generate robust tracking predictions.
a lgg + +1 n2/id
PF 26) sin
(3)
a agg(t + 1) 22///d
PE(1, 2i + 1) cos
As shown in Fig. 2, the two input flows, including the visual input and the object motion trajectory, are processed sep- arately through the baseline visual model and our motion encoder to extract corresponding features. Through calcula- tions of several self-attention and cross-attention modules, features from the two branches are complementarily fused. An adaptive weight mechanism is additionally utilized to establish a dynamic residual connection between visual fea- tures and the output of the fusion decoder, thereby achieving stable fused embeddings. By integrating complementary motion prompts, the proposed motion modules empower the vision- based baseline tracker to achieve enhanced per- formance, especially under challenging scenes.
Here, i, d denotes the dimension variable and the total di- mension, L E [o, T 1] represents the temporal position of the point in the trajectory. A larger t corresponds to closer position to the current frame. The constant n is set to 10 and a is a parameter employed to control the frequency range. To maximize spatial resolution without causing aliasing, we set a 7.23 according the Nyquist frequency (Nyquist, 1928). Both theoretical proof and ex- perimental analysis can be found in Appendix A.
In addition to the 2T motion tokens, we also introduce three learnable representative (Rep.) tokens. Among them, RTu and RT br are employed to summarize motion information from all top-left and bottom-right trajectory points, respec- tively. RT represents the confidence of the motion prompt. We set two sub-tasks to guide the learning of three Rep. tokens, namely, bounding box regression and weight re- gression. These sub-tasks encourage our model to develop the ability to extract crucial motion prompts and predict dynamic weights. As the output of the motion encoder, we concatenate 2T motion tokens and the three Rep. tokens to represent the extracted motion feature.
3.3. Model Architecture
Our motion prompt tracking method mainly consists of three modules, i.e. a motion encoder, a fusion decoder, and an adaptive weight mechanism. Detailed architectures of each module are described as follows.
Motion Encoder. Given consecutive object bounding boxes of previous T frames, our motion encoder embeds these sequential coordinates into multiple motion tokens aligned with the visual embedding space. The following three types of positional encodings (PE) are utilized to characterize each token across both spatial and temporal dimensions.
Fusion Decoder. As shown in Fig. 2, we propose a lightweight fusion decoder to fuse the visual feature f. e Hwx and motion feature fm E (2x where denotes the batch size, and H, W represent resolutions of
4

---

Efficient Motion Prompt Learning for Robust Visual Tracking
the visual feature. The proposed fusion decoder consists of N Transformer-based blocks, each including motion self-attention module, two cross-altention modules with dif- ferent directions, and a multi-layer perception (MLP) layer. Specifically, the self-attention module first explores crucial motion information from motion features themselves. The first cross-attention module then fuses visual cues into mo- tion features, in which motion features fm are set as queries (q). After the update by an MLP, we take the three Rep. to- kens as keys (k) and values (t) for the second cross-altention module, and update the visual feature Su: (as queries) with representative motion prompts. For each cross-attention module, we incorporate a dense spatial map as positional en- coding onto visual features. This ensures a correspondence between visual and motion features in spatial coordinates. Regarding temporal positions, we also apply the temporal PE to motion features in the first cross- aetssttttntionn module.
plies less confidence in motion prompts. We thereby adopt the IoU between BM and GT, as the label of the predicted weight WM, and calculate MSE loss to achieve explicit supervision. The adaptive weight loss is formulated as
Lw = LMM, GT,)).
(5)
The overall loss function of our MPT involves the above three terms, represented as
= LTv + AM + Lw),
(6)
where Am = 1 in our experiments.
Inference. During the inference, predicted bounding boxes of previous T frames are first aligned and normalized based on the coordinate system of the current search region. Nor- malized trajectory is sent together with the image pair into our motion prompt-based tracking model for tracking pre- diction. As Headæ primarily serves to facilitate the more effective training for our motion modules, this network is disregarded during inference to save computational costs.
Adaptive Weight Mechanism. Considering the instability of motion trajectories, we calculate the weighted average of the fused feature and original visual feature using an adaptive weight mechanism. A weight head Headw is em- ployed to dynamically predict the weight WM according to the output RTw w from MLP. To explicitly guide the learning of WM. we additionally introduce a motion head Heady to predict the current bounding box Ba according to RT. and RTb We calculate the Intersection-ove-Ummee the groundtruth and BM, and use it to explicitly supervise the learning for WM-
4. Experiment
To demonstrate the effectiveness and training efficiency of our MPT, we integrate our MPT into two different vi- sual trackers, i.e., a one-stream tracker OSTrack', and an autoregressive-based tracker TeqTrack We also apply our MPT to ARTrack to show that our MPT can further comple- ment temporal-related trackers rather than conflicting. We select a total of five baseline models with different back- bones and resolutions. To distinguish them, base large backbones are represented as "B L", while the follow- ing numbers represent image resolutions. We compare methods with their baselines and other SOTA trackers on seven challenging tracking benchmarks. Experimental re- sults demonstrate that our MPT can be seamlessly integrated into various trackers, enhancing their tracking robustness with fast and memory-efficient training.
(IoU) of
3.4. Training and Inference
Training, We adopt prompt learning to achieve efficient training. All parameters in the baseline tracker are fixed, and only our MPT modules are fine-tuned, including the motion encoder, fusion decoder, learnable embeddings, Headw, and HeadM. The backpropagation process is illustrated as dashed red arrows in Fig. 2, The loss function for our MPT comprises the following three terms:
Baseline tracking loss. We follow the same tracking loss as baseline trackers, namely LT, (B, GT,), where B denotes the output of Headr,, and GT, represents the groundtruth of the object bounding box.
4.1. Implementation Details
Our methods are implemented in Python with PyTorch. Models are trained on NVIDIA A 100 GPUs, and tested on a single NVIDIA RTX2080Ti GPU.
Motion regression loss. To enable Rep. tokens RT. and RT,r to extract essential motion patterns, we set bounding box regression task on Headæ- The general- ized IoU loss (Rezatofighi et al., 2019) and & loss are employed to supervise this sub-task, formulated as
Model. The length of the historical trajectory T is set to 30 based on experimental results. Each 2-dimensional coordi- nate is encoded into a motion token with the same dimension as baseline visual features. The lightweight fusion decoder is implemented as a two-layer Transformer network. The weight head Headw and motion head Heady are imple- mented by a two-layer MLP, where the hidden size is 256.
Lm = AtoU LIOU + dr, C1,
where ALoU= = 2 and de. = 5 in our experiments.
Adaptive weight loss. In our assumption, a lower qual- ity of the motion tracking prediction By typically im-
'hhe official model without the candidate elimination module.
5

---

Efficient Motion Prompt Learning for Robust Visual Tracking
and 10.2% EAO on VOT2018.
Table Performance comparison on three VOT datasets. The best results are highlighted with bold.
LaSOT and LaSOTEXT. LaSOT (Fan et al., 2019) con- tains 280 long-term sequences with 70 different categories, while its extension LaSOT (Fan et al.. 2021), consists of 150 extremely challenging sequences with 15 unseen categories. The success rate (AUC), normalized precision (P norm)) and precision (Pre) are adopted to evaluate track- ers. As reported in Tab. 2. our MPT increases baseline trackers by average 1.6% AUC on LaSOT Fur- thermore, our training-efficient method (+MPT) enhances Seq SeqTrack-L384 to achieve an AUC of 73.9% on LaSOT, which is competitive with sequential- training method using multi mulliiteemplte strategy, ODTrack (Zheng et al., 2024).
VOT2018
VOT2020
VOT2022 (STB)
Method
EAO (1)
R (L)
EAO (t) R (t)
EAO (1) R (t)
MixFormer-22K
0.228
1.836
0.317
0.824
0.533
0.844
ROMTrack-B256
0.357
1.010
0.318
0.815
0,554
0.825
OST5
0.287
1.204
0.311
0.803
0,530
0.823
+MPT (Ours)
0.406
0.473
0.334
0.846
0.572
0.859
Brack-
0.339
1.067
0.316
0.806
0.523
0.815
+MPT (Ours)
0.428
0.471
0.315
0.810
0.528
0.825
ARTrack-B2566
0,399
0.399
0.315
0.809
0.532
0.818
+MPT (Ours)
0.469
0.491
0.336
0.828
0.566
0.842
MixForme-T.
0.238
1,596
0.325
0.825
0.549
0.843
ROMTrack-B384
0.297
1.231
0.309
0.794
0,540
0.812
ARTrack-L384
0.391
0.520
0.336
0.834
0.570
0.852
OSTrack-3384
0,271
1,408
0.288
0.767
0.518
0.796
+MPT (Ours)
0.363
0.679
0.314
0.819
0.548
0.826
TNL2K (Wang et al., 2021). TNL2K is a recently released dataset with 700 challenging sequences. As shown in Tab. 2. our MPT yields average 0.6% and 1.0% improvements in terms of AUC and P norm. respectively.
SeqTrack-L334 L384
0.319
1.117
0.337
0.859
0.568
0.869
+MPT (Ours)
0.461
0.414
0.341
0.870
0.579
0.882
Average gain
+10.2%
+57.3%|
+1.5%
+2.6%
+2.4%
+2.3%
TrackingNet (Muller et al., 2018). TrackingNet contains 511 sequences covering diverse categories and scenarios. As presented in Tab. 2, our MPT brings baselines an average performance improvement of around 0.4% on each metric.
Training. We adopt prompt learning to efficiently train our MPT, where the baseline model is frozen. We se- lect the training splits of LaSOT (Fan et al., 2019), GOT- 10K (Huang et al., 2019)², and TrackingNet (Muller et al., 2018) as the training data. For the motion input, we adopt D8MP-18 (Bhat et al., 2019) to generate real tracking pre- dictions for each of training sequences, and employ reverse sampling, sparse sampling and CutMix (Yun et al., 2019) for data augmentations. More implementation details can be found in Appendix B, and extended experimental analysis in terms of data augmentations and trajectory length can be found in Appendix C.
Discussion. We find that the performance improvement of our MPT is slightly biased across different datasets, with more substantial gains on more challenging ones (like VOTs, LaSOT Txx)). To explore the underlying reasons, we divide each of the three datasets (i.e., LaSOT, LaSOTEXT, and TNL2K) into "hard/ / easy" subsets based on the performance of each sequence, and analyze the performance changes of MPT on each subset (shown as Tab. 7). As discussed in Ap- pendix C, our MPT shows great consistency in hard easy sets, achieving average AUC changes of +4.4% / -0.8%, with small standard deviations of 0.6 / 0.4 across five base- lines. Therefore, we reach a consistent conclusion that the advantage of our MPT is more evident in hard scenarios. This is because our MPT primarily aims to reduce tracking failures by motion cues, leading to significant improve- ment in tracking robustness rather than accuracy. This can also be intuitively demonstrated by the performance gains on the three VOT benchmarks (shown as Tab. 1).
4.2. State-of-the-Art Comparison
We compare our methods with baselines and other SOTA trackers on the following seven tracking benchmarks.
VOT. The committee of VOT challenge proposes a series of challenging tracking benchmarks. Trackers are evaluated primarily by the expected average overlap (EAO), which is a principled combination in terms of tracking accuracy and robustness (R). As shown in Tab. 1, we compare our methods with several SOTA trackers on three representative VOT datasets, including VOT2018 (Kristan et al., 2018), VOT2020 (Kristan et al., 2020), and VOT2022 (Kristan et al., 2022). For fair comparison, all of the presented trackers are run with their official models and settings, and evaluated based on bounding box outputs. Compared with SOTA methods, trackers integrating our MPT exhibit supe- rior performance on these challenging sequences, especially in terms of robustness. Our MPT significantly enhances the performance of the five baseline models, averaging an improvement of 2.0% EAO across VOT2020 and VOT2022,
In addition, compared with some trackers like ODTrack and ARTrackV2, which adopt sequential-level training or multi- template strategies enhance their tracking performance, our MPT requires less memory and computing resources during training. As shown in Tab. 3, taking a sequential- MRKTrack-2550 as a reference, the set-
training tracker ting of 8 batch sizes (BS) needs 50G GPU memory, within more than 3 days of training. In contrast, our MPT adopts lightweight architecture and efficient frame-level training, introducing only 13M parameters (Params) and negligible speed (FPS) degradation for baselines. For OSTrack with 128 BS, our MPT only needs 5 hours of fine-tuning using
FFolowing the VOT protocol, 1k sequences are removed.
6

---

Efficient Motion Prompt Learning for Robust Visual Tracking
Table Comparisons on four tracking benchmarks. Top two results (except for sequential-traiiing methods are marked bold and underlined. For fair comparison, list the training strategy (Train) and template amounts (Temp.) of each tracker.
LaSOT
LaSOTEXI
TNL2K
TrackingNet
Method
Train Temp.
AUC
Pueem
Pre
AUC
Puorm
Pre
AUC
Puarm
Pre
AUC
P Paorm
Pre
SiamRCNN (Voigtlaender ct al., 2020)
Frame
64.8
72.2
81.2
85.4
80.0
TransT (Chen et al., 2021)
Frame
64.9
73.8
69.0
50.7
51.7
81.4
86.7
80.3
STARK-STI01 (Yan et al., 2021)
Frame
2
67.1
77.0
82.0
86.9
ToMP-101 (Mayer et al., 2022)
Frame
68.5
79.2
73.5
45.9
81.5
86.4
78.9
AiATrack (Gao et al., 2022)
Frame
2
69.0
79.4
73.8
46.8
54.4
54.2
82.7
87.8
80.4
MixFormer-22k (Cui et al., 2022)
Frame
2
69.2
68.7
74.7
83.1
88,1
81.6
Sim-B16 (Chen al., 2022a)
Frame
69.3
78.5
54.8
53.8
82.3
86.5
ROMTrack- (Chen et al., 2022a)
Frame
69.3
78.8
75.6
48.9
59.3
55.0
83.6
88.4
82.7
MixViT (Con (Cui et al., 2024)
Frame
70.4
80.4
76.7
84.5
89.1
83.7
DSTruck-B250 (Ye et al., 2022)
Frame
68.6
78.0
74.5
46.9
56.9
52.8
55.6
72.1
56.1
83.0
87.7
81.7
4PP
Frame
68.9
78.6
74.8
48.7
59.1
$5.2
55.8
72.7
56.5
83.6
88.4
82.4
SeqTrrck-B2255 B256 (Chen et al., 2023)
Frame
69.4
79.2
75.8
49.8
61.2
56.8
56.5
74.2
58.6
83.2
88.3
82.3
+MPT (Ours)
Frame
2
70.1
80.4
76.7
50.8
62.4
58.0
57.8
75.9
60.3
83.6
88.8 8
82.6
ARTrack-B22 BBr6ack-2256/a (Wei et al., 2023)
Seq
70.4
79.5
76.6
48.4
57.7
53.7
57.9
74.0
59.6
84.2
88.7
83.5
+MPT (Ours)
Frame
70.3
80.1
76.8
50.4
60.7
57.5
58.5
75.2
60.9
84.2
88.9
83.2
ARTrackV2-B256 (Bai et al., 2024)
Seq
2
71.6
80.2
77.2
50.8
61.9
57.7
59.2
84.9
89.3
84.5
MixFormer-L. (Cui et al., 2022)
Frame
2
70.1
79.9
76.3
83.9
88.9
83.1
Sim-L14 (Chen et al., 2022a)
Frame
70.5
79.7
55.6
55.7
83.4
87.4
ROMTrack-B334 (Cai al., 2023)
Frame
2
71.4
81.4
78.2
51.3
62.4
58.6
84.1
89.0
83.7
MixViT-L (ConyMAE) (Cui et al., 2024)
Frame
73.3
82.8
80.3
86.1
90.3
86.0
OSTrack-B384 (Ye et al., 2022)
Frame
71.0
80.9
77.3
50.9
61.6
57.7
57.5
74.2
58.6
83.8
88.4
83.1
+MPT (Ours)
Frame
70.7
80.5
76.9
52.8
64.0
60.7
57.8
74,8
59.5
84.6
89.1
83.8
SeqTrack-L3384 (Chen al., 2023)
Frame
72.5
81.4
79.2
50.4
61.2
57.2
59.6
76.6
63.5
85.6
90.0
85.9
+MPT (Ours)
Frame
73.9
83.2
81.0
51.7
62.7
58.8
60.4
77.5
64.2
85.8
90.1
86.2
ARTrack-L.384 (Wei ct al., 2023)
Seq
73.1
82.0
80.2
52.0
62.4
59.0
60.8
77.5
64.5
85.4
90.2
85.8
ODTrack-B38. (Zheng et al., 2024)
Seq
3
73.2
83.2
80.6
52.4
63.9
60.1
60.9
85.1
90.1
84.9
ARTrackV2-L3384 (Bai al., 2024)
Seq
73.6
82.8
81.1
53.4
63.7
60.2
61.6
86.1
90.4
86.2
ODTrack-1.384 (Zheng et al., 2024)
Seq
74.0
84.2
82.3
53.9
65.4
61.7
61.7
86.1
91.0
86.7
Average gain
+0.4%
+0.8%
+0.6%
+1.6%
+2.1%
+2.4%
+0.6%
+1.0% +1.0%
+0.4%
+0.4%
+0.3%
Table 3. Efficiency of training and inference. Two A100 GPUs are used with 128 batch sizes (BS), except for ARTrack with BS. Timeer and Nr denote training time and frame numbers of temporal information.
Table 4. Impact of positional encodings, including respective role of each PE and comparisons of TPE variants.
i
Ours
w/o SPE w/o TPE w/o P Init Initu
Non-Learn
Tracker (BS) FLOPs Params FPS Mem Timetr Nr
LaSOT 48.7
44.8
47.5
48.1
45.9
47.4
47.4
VOT2022
0.572
0.556
0.547
0.573
0.567
0.561
0.552
OSTrack (128) 31.2G
92M
94
35G
38h
-2.8%
-1.9%
-0.3%
-1.7%
-1.2%
-1.7%
+MPT (128)
35.9G
105M
86
6G
5h
30
ARTrack (8) 55.2G
202M
26
50G
26+53h
7
+MPT (128) 56.8G
215M
21G
12h
three positional encodings, we individually remove one type of PE, shown as Tab. 4. In our motion encoder, the spatial PE (SPE) is mainly used to encode each point coordinate to the visual embedding space, facilitating the alignment of motion and visual features in terms of spatial positions.
6G memory. demonstrates that our MPT is training ef- ficient, and offers incremental benefits to various trackers with minimal training resources.
4.3. Ablation Study and Analysis
When we replace our SPE with a simple MLP (2), the track- ing performance decreases by 2.8% on average. Besides, our temporal PE (TPE) integrates temporal orders to mo- tion tokens in a non-linear way, and the employed point embeddings (P) identify the type of corner points for motion tokens. Removing any of them (3 and ④) results in a certain degree of performance decline. These results indicate that each type of PE plays a crucial role in the performance of
To deeply analyze our MPT, we conduct comprehensive ablation studies on LaSOT and VOT2022, choosing (eTrack-B256+ME as our baseline (denoted as Ours). We present AUC and EAO for two benchmarks, respectively, and also the averaged performance change (denoted as A).
Positional encoding. To explore impacts of the adopted
7

---

Efficient Motion Prompt Learning for Robust Visual Tracking
Success Rate
Table 5. Impact of different losses and components.
1
0.9
Ours Cn + Lx
Liv
|Motion
Vision w/o Weight
0.8
0.8
EXS 48.7
48.6
45.1
48.5
46.9
46.4
0.6
VOT2022 0.572 A
0,558
0,556
0.525
0.530
0,568
0.6
0.4
-0.8%
-2.6%
-2.5%
-3.0%
-1.4%
0.4
0.2
0.3
0.2
(0.0.1)
[0.1.02)
(0..033)
(0.3.04)
(0:40.5)
(0.5.066)
(066..7)
(0.7.0.8)
oo (o.80.9)
(09.1.0)
Jolurr
Prokiolded wdgs
Figure 4. Success rate of among varying trajectory qualities.
Amplyy
Distribution
[Success] loUga
1
[Success] lo
0.8
[Fafture] loD
Figure Illustration adaptive weights, and tracking perfor- mance based on different cues.
0.6
[Failure Joo
0.4
0.2
our MPT, particularly SPE and TPE.
Furthermore, we adopt a non-linear initialization on a learn- able embedding for our TPE (formulated as Eq. 3) to make the model dynamically focus on the significant motion in- formation. To explore the impact of different TPE initializa- tions, we compare random with our non-linear method((), It shows that our non-linear TPE initialization performs superior to others. Besides, to investigate the efficacy of the learnable attribute in our TPE, we also compare our method with a variant with non- learnable TPE (0). Disabling the learning capability of TPE results in a performance degradation of 1.7%
[0.0.1)
(0.1.022
[02.033)
[0.3,044)
[0.4.0.5)
(0.5.066)
(0660.7)
(0.7.08))
10..099)
109,1100
Figure 5. IoU distributions of average trajectory and the last frame in success and failure cases.
(5) and linear (@) initializations
decoder for tracking predictions. As observed, our MPT, compared to the tracker relying solely on visual information, yields more robust tracking results by incorporating mo- tion prompts. Additionally, the adaptive weight mechanism contributes to our method achieving further improvements.
Loss function. We formulate three loss terms for model op- timization. In addition to the baseline tracking loss CT. introduce the other two loss terms to enhance supervision. Among them, the motion regression loss Lm guides Rep. tokens to extract essential motion patterns from the trajec- tory. The adaptive weight loss Lw is employed to explicitly supervise the weight prediction, allowing the model adap- tively adjust its reliance on visual motion cues. Results reported in Tab. 5 demonstrate the significance of both Ls and Lw. The combined utilization of all three losses (Ours) boosts the model to achieve superior performance.
Besides, we visualize frame-level qualities of tracking re- sults predicted by the aforementioned three variant models, namely "Vision", "Motion", and our MPT. As shown in Fig. 3, when encountering plenty of distractors, the predic- tion of the vision model (blue curve) is notably unstable. In contrast, our MPT (red curve) can exhibit superior and robust performance. Furthermore, the values of our dynamic weights (green curve) consistent with the quality of mo- tion predictions (orange curve). This demonstrates that adaptive weight mechanism can achieve dynamic balance between visual and motion cues.
Adaptive weight mechanism. Results in Tab. 5 (right) demonstrate the respective roles of visual and motion in- puts. as well as the effectiveness of the proposed adaptive weight mechanism. "Motion" represents the tracking results directly predicted by the motion head Hcadæı, while "Vi- sion" shows the results of the baseline vision-based tracker. We also implement a variant of MPT without the adaptive weight mechanism, denoted as "w/o Weight". That is, in- stead of calculating the weighted average with the original visual features, we directly use the output of the fusion
Sensitivity to the trajectory quality. To further analyze the sensitivity of our MPT to varying noisy trajectories, we evaluate the success rate (IoU 0.5) under different quality trajectory inputs on LaSOTEXT: as shown in Fig. 4. Here, b))b) represents the range where the average IoU of the tra- jectory is no less than and less than b. We can find that the success rate increases significantly with the improvement of trajectory quality, especially when the trajectory IoU is in the range of [0.6, 1.0). Besides, our method shows a certain level of robustness to noisy trajectory inputs, obtaining a
8

---

Fine-tuning strategies. Our method freezes all of the base- line parameters, and only fine-tunes parameters of MPT during training. We also evaluate two variants, shown in Tab. 6. Finetune fine-tunes both MPT and baseline pa- rameters, while FenetuneHead fine-tunes parameters of MPT and the tracking head of the bascline, i.e., only freezing the backbone of the baseline model. Our method slightly outper- forms the other two variants, and is more efficient in terms of memory footprint and training time. The potential reason is that freezing the baseline parameters enables the model to focus on learning motion cues and the fusion mechanism. Efficient Motion Prompt Learning for Robust Visual Tracking
Rifereee
Seasshlpp.
AmC
Seaellepp
Scamthepare
Allig
Table Impact fine-tuning strategies, in terms of tracking performance and training efficiency.
OSTrack Ours FinetumeAll FinetuneHead
LaSOTExT
46.9
48.7
48.3
48.1
VOT2022
0,530
0.572
0.567
0.548
Mem (G)
35,0
6,1
37,5
7.7
Time:- (h)
38
8
Figure 6. Visualization of cross-attention maps, and discrimi- native comparison score maps.
success rate of 21% even when the trajectory IoU is in the range of 10.1, 0.2).
response to distractors, thereby achieving robust tracking.
In addition, to deeply explore the success and failure cases of our MPT, we separately evaluate the distribution of tra- jectory IoU (IoU and tracking IoU of the last frame (IoU in both conditions on LaSOTExT EXT. Here, we de- fine a success case when the predicted IoU of our MPT exceeds that of the visual baseline by more than 0.3. Con- versely, when the predicted IoU of our MPT is lower than that of the visual baseline by more than 0.3, we define it failure case. As shown in Fig. 5, in success cases, the input trajectory IoU primarily falls within a relatively broader range of [0.6, 1.0), especially in [0.8, 0.9). In failure cases, the input trajectory IoU mainly falls within the range of [0.0, 0.2). particularly [0.0, 0.1). This indicates that addressing misleading tracking issues caused by extremely poor-quality trajectories remains& a challenge for our method. This is limitation of our method that requires future research.
Due to space limitations, more visualizations can be found in Appendix E, including the optimizing process during training (Fig. 11), qualitative comparisons (Fig. 13), and some failure cases (Fig. 12).
Limitations. Although our MPT excels in training effi- ciency and tracking robustness, its tracking accuracy is slightly inferior to those sequential training methods. We will further explore effective methods to capture dynamic relations in efficient prompting. In addition, as shown in Fig. 5, addressing misleading tracking issues caused by ex- tremely poor-quality trajectories remains a challenge for our method. which requires future research.
5. Conclusion
In this paper, we present flexible and efficient motion prompt tracking method. It can complement existing vision- based trackers by learning motion prompts, thereby main- taining robust tracking in challenging scenarios. Our method includes a motion encoder with three different positional en- codings, Transformer- based fusion decoder, and adap- tive weight mechanism. Motion trajectories of the object are independently encoded and then adaptively fused with vi- sual features. We integrate our method into five visual track- ing models. Experiments on seven datasets show that our method efficiently enhances tracking performance within minimal computational costs and memory usage. We hope that our new insights into temporal tracking and prompt tracking will leverage future research on these topics.
Visualization of cross-attention maps. For more straight- forward understanding of how our MPT fuses visual and motion features, we visualize the cross-attention maps (av- eraged over heads) of the last decoder block, shown as Fig. 6. Among them, "Attnn"" and "Attmbe" represent cross-attention maps between visual features and two rep- resentative motion tokens (RT,) and RT,), respectively. In addition, we also visualize the score maps predicted by the uision-based baseline tracker (ScoreMapy), ), and our MPT model (SporeMapMPT). We find that it is difficult to dis- tinguish distractors based on a solely visual model. Due to the complement of motion prompts, our MPT pays more attention to the left and bottom-right corner points of the interested target. This enables the model weaken the
A Acknowledgments
The computational resources were provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at C3SE partially funded by the Swedish Research Council grant 0022-06725. This work was supported in part by the Wallenberg Artificial Intelligence, Autonomous Systems and Software Program (WASP), funded by Knut and Alice Wallenberg Foundation, in part by the strategic research environment ELLIIT funded by the Swedish gov- ernment, and the Swedish Research Council grant 2022-

---

Efficient Motion Prompt Learning for Robust Visual Tracking
04266, in part by National Natural Science Foundations of China (No. 62402084 and No. 621 62176041), in part by Liaoning Province Science and Technology Plan (No.
Cui, Y., Jiang, C., Wang, L.,, and Wu, G. MixFormer: End- to-end tracking with iterative mixed attention. In IEEE Conference Computer Vision and Pattern Recognition, 2022.
2024JH2/1026000000 and in part by China Postdoctoral Science Foundation (No. 2024M750319).
pp.
Cui, Y., Jiang, C., Wu, G., and Wang, L. MixFormer: End-to- end tracking with iterative mixed attention. IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 46 0144129-4444 2024.
Impact Statement
This work advances the field of visual object tracking through efficient motion prompt learning. The technique may benefit real-time applications such as surveillance sys- tems.
Dai, K., Wang, D., Lu, H., Sun, C., and L. Visual track- ing via adaptive regulaiy-e correlation filters. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 4670 2019.
References
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M.. Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representation, 2021.
Bai, Y.. Zhao, Z., Gong, Y., and Wei, X. ARTrack V2: Prompting autoregressive tracker where to look and how to describe. IEEE Conference on Computer Vision and Pattern Recognition, pp. 19048-19057, 2024.
Bertinetto, I.., Valmadre, J., Henriques, J. F., Vedaldi, A., and Torr, P. H. Fully-convolutional siamese networks for object tracking. In European Conference on Computer Vision, pp. 850-865, 2016.
Fan, H., Lin, I.., Yang, F., Chu, P., Deng, G., Yu, S., Bai, H., Xu, Y., Liao, C.,, and Ling, H. LaSOT: A high-quality benchmark for large-scale single object tracking. IEEE Conference on Computer Vision and Pattern Recognition, pp. 5374-5383, 2019.
Bhat, G., Danelljan, M., Gool, L. V., and Timofte, R. Learn- ing discriminative model prediction for tracking. In IEEE International Conference on Computer Vision, pp. 6182- 2019.
2019.
Fan, H., Bai, H., Lin, L.,, Yang, F., Chu, P., Deng, G., Yu, S., Huang, M., Liu, J., Xu, Y., al. LaSOT: A high-quality large-scale single object tracking benchmark. Interna- tional Journal of Computer Vision, 129:439 461, 2021.
Cai, Y., Liu, J., Tang, J., and Wu, G. Robust object modeling for visual tracking. In IEEE International Conference Computer Vision, pp. 9589-9600, 2023,
Gao, S., Zhou, C., Ma, C., Wang, X., and Yuan, J. AiATrack: Attention in attention for transformer visual tracking. In European Conference on Computer Vision, pp. 146-164, 2022.
Cao, Z., Huang, Z., Pan, L., Zhang, S., Liu, Z., and Fu, C. Towards real-world visual tracking with temporal contexts. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Huang, L., Zhao, X., and Huang. K. GOT-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1562-1577 2019.
Chen, B., Li, P., Bai, I.., Qiao, I.., Shen, Q., Li, B., Gan, W., Wu, W., and Ouyang, W. Backbone is all your need: A simplified architecture for visual object tracking. In European Conference on Computer Vision, pp. 375-392, 2022a.
Jia, M., Tang, L., Chen, B.-C., Cardie, C., Belongie, S.. Hariharan, B., and Lim, S.-N. Visual prompt tuning. In European Conference Computer Vision, pp. 709-727, 2022.
Chen, X., Yan, B., Zhu, J.. Wang, D., Yang, X., and Lu, H. Transformer tracking. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 8126 8135, 2021.
Chen, X., Peng, H., Wang, D., Lu, H., and Hu, H. SeqTrack: Sequence to sequence learning for visual object tracking. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 14572 14581, 2023.
Ju, C., Han, T., Zheng, K., Zhang, Y., and Xie, W. Prompting visual- llnguuge models for efficient video understanding. In European Conference on Computer Vision, pp. 105- 124. Springer, 2022.
Chen, Z., Duan, Y., Wang, W., He, J., Lu, T., Dai, J., and Qiao, Y. Vision transformer adapter for dense predictions. arXiv preprint arXiv:2205.n 08534,, 2022b.
Kim, M., Lee, S., Ok, J., Han, B., and Cho, M. Towards sequence- level training for visual tracking. In European Conference on Computer Vision, pp. 534-551 2022.
10

---

Efficient Motion Prompt Learning for Robust Visual Tracking
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A. C.,, Lo, W.-Y., et al. Segment anything. In IEEE International Conference on Computer Vision, pp. 4015-4026 2023.
Ramachandran, V. The neurobiology of perception. Percep- tion, 14(2)97-103,, 1985.
Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S. Generalized intersection over union: metric and loss for bounding box regression. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 658-666,24 2019.
Kristan, M., Leonardis, A., Matas, J., Felsberg. M., Pflugfelder, R., "Cehovin Zaje, L.,, Vojir, T., Bhat, G., Lukezie, A., Eldesokey, A., et al. The sixth visual ob- ject tracking VOT2018 challenge results. In European Conference on Computer Vision, 2018.
Sokhandan, A. and Monadjemi, A. Visual tracking in video sequences based on biologically inspired mechanisms, Computer Vision and Image Understanding, 239:102724, 2024.
Kristan, M. Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Kämäräinen, J..K.. Danelljan, M., Zaje, L. C., Lukežič, A., Drbohlav, O., et al. The eighth visual object tracking VOT2020 challenge results. In European Conference on Computer Vision, pp. 547-601, 2020.
Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil S., Raghavan, N., Singhal, U., Ramamoorthi, R., Bar- ron, J., and Ng. R. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems, volume 33, pp. 7537-7547, 2020.
Kristan, M. Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Kämäräinen, J..K., Chang, H. J., Danell- jan, M., Zaje, L. c., Lukežič, A., et al. The tenth visual object tracking vor2022 challenge results. In European Conference on Computer Vision, pp. 431-460, 2022.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. Atten- tion is all you need. In Advances in Neural Information Processing Systems, pp. 1998-6008 2017.
Li, B., Wu, W., Wang. Q. Zhang, F., Xing, J., and Yan, J. SiamRPN++: Evolution of siamese visual tracking with very deep networks. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 12222-4991 2019.
Voigtlaender, P., Luiten, J., Torr, H., and Leibe, B. Siam R-CNN: Visual tracking by re-detection. In IEEE Con- ference on Computer Vision and Patlern Recognition, pp. 2020.
Lin, L., Fan, H., Xu, Y., and Ling, H. SwinTrack: A simple and strong baseline for transformer tracking. In Advances in Neural Information Processing Systems, 2022.
Wang. X., Shu, X., Zhang, Z., Jiang, B., Wang, Y., Tian, Y., and Wu, F. Towards more flexible and accurate object tracking with natural language: Algorithms and bench- mark. In IEEE Conference on Computer Vision and Pat- tern Recognition, pp. 13763-13773, 2021.
Liu, C.,, Yuan, Y., Chen, X., Lu, H., and Wang. D. Spatial- temporal initialization dilemma: towards realistic visual tracking. Visual Intelligence, 2(1):35, 2024.
Liu, H., Li, C., Wu, Q.. and Lee, J. Visual instruction tuning. In Advances in Neural Information Processing Systems, volume 36, pp. 14892-34916 2023a.
Wei, X., Bai, Y., Zheng, Y., Shi, D., and Gong, Y. Autore- gressive visual tracking. In IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 9697-9706. 2023.
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., and Neubig, G. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023b.
Weng. S.-K., Kuo, C.-M., and Tu, S.-K. Video object track- ing using adaptive kalman filter. Journal of Visual Com- munication and Image Representation, 17(6): (7(6):1190-110 190--128, 2006.
Mayer, C., Danelljan, M., Bhat, G., Paul, M., Paudel, D. P., Yu, F., and Van Gool, L. Transforming model prediction for tracking. In JEEE Conference on Computer Vision and Pattern Recognition, pp. 8731-8740, 2022.
Yan, B., Peng, H., Fu, J., Wang. D., and Lu, H. Learning spatio-temporal transformer for visual tracking. In IEEE International Conference on Computer Vision pp. 10448- 10457, 2021.
Muller, M., Bibi, A., Giancola, S., Alsubaihi, S., and Ghanem, TrackingNet: A large-scale dataset and benchmark for object tracking in the wild. In European Conference on Computer Vision, pp. 300-317, 2018.
Yang, J., Li, Z., Zheng, F., Leonardis, A., and Song, J. Prompting for multi-modal tracking. In ACM Interna- tional Conference Multimedia, pp. 3492-39000000 2022.
Nyquist, H. Certain topics in telegraph transmission the- ory. Transactions of the American Institute of Electrical Engineers, 47(2):617 644, 1928.
Ye, B., Chang, H., Ma, B., Shan, S., and Chen, X. Joint feature learning and relation modeling for tracking:

---

Efficient Motion Prompt Learning for Robust Visual Tracking
one-stream framework. In European Conference on Com- puter Vision, pp- 341-357, 2022.
Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y. CutMix: Regularization strategy to train strong clas- sifiers with localizable features. In JEEE International Conference Computer Vision, pp. 6023-6032, 2019.
Zhang, M., Zhang, Q., Song, W., Huang, D., and He, Q. Promptvt: Prompting for efficient and accurate visual tracking. IEEE Transactions Circuits and Systems for Video Technology, 2024.
Zhang, S., Chan, S.-C., Liao, B., and Tsui, K. M. A new visual object tracking algorithm using bayesian kalman filter. In IEEE International Symposium on Circuits and Systems, pp. 522-525, 2014.
Zhao, J., Dai, K., Zhang, P., Wang, D., and Lu, H. Robust online tracking with meta-updater. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5))6168- 6182, 2022a.
Zhao, J., Zhang, Li, D., and Wang, D. Vision-based anti-uav detection tracking. IEEE Transactions on Intelligent Transportation Systems, 23(12):55323-2533, 2022b.
Zheng, Y., Zhong, B., Liang, Q. Mo, Z.,, Zhang, S., and Li, X. ODTrack: Online dense temporal token learning for visual tracking. In AAAI Conference on Artificial Intelligence, 2024.
Zhu, J., Lai, S., Chen, X., Wang, D., and Lu, H. Vi- sual prompt multi-modaa tracking. In IEEE Conference on Computer Vision and Pattern Recognition, pp. 9516- 9526, 2023.
12

---

Efficient Motion Prompt Learning for Robust Visual Tracking
Appendix
This appendix contains proof of non-linear temporal positional encoding in Section A, additional implementation details in Section B, extended ablation studies in Section C, in-depth performance analysis in Section D, and more visualizations in Section E.
A. Proof of Non-Linear Temporal Positional Encoding
EAO
Default setting
0.575
0.572
0.565
0.559
0.564
0.559
0.553
0.561
0.555
0.545
0.535
Baseline: 0.530
0.525
6.75 7.23 7.75
15
Parameter a
Figure Impacts of different parameter VOT2022.
To allow our model to perceive temporal orders of motion tokens, we adopt a learnable temporal positional encoding and initialize in the following non-linear manner:
Alg((+1))
PE(t, 21) = sin
n2//d
aglg(t +
PE(t, 2i 1) = cos
nii/d
The parameter is employed control the frequency range, and set 7.23 based on the Nyquist frequency (Nyquist, 1928). The proof of this setting is presented as follows.
According to the theory of Nyquist frequency, a signal should be sampled at a rate at least twice its highest frequency to avoid aliasing. Due to the logarithmic mapping of the time parameter t, the frequency w of Eq. changes in non-linear way and needs be determined using the derivative (note that lg(t 1) In(t 1)/ In(1))):
w(t, = atP 21)
a PE(t, 2i 2 ln( + 1)n
(8)
Its maximum is attained (t, i) (0, From (0,00) n, we obtain a In(10) 7.23.
Moreover, also verify its experimental effectiveness by comparing it with other parameter values. As shown in Fig. 7, results VOT2022 (Kristan et al., 2022) show that the best performance is attained cause slight performance decays, but they are still superior to the baseline tracker (marked as the horizontal dashed line).
7.23. Other parameter values
B. Implementation Details
Model. For our motion encoder, we encode each 2-dimensional coordinate into motion token with the same dimension the baseline visual features, which 768 and 256 for OSTrack-B ARTrack-B, and SeqTrack-B, respectively. The lightweight fusion decoder is implemented as two-layer Transformer network. For each layer, the number of attention heads is 8, and the hidden size of MIP is set to 1024 and 256 for OSTrack / ARTrack and SeqTrack, respectively. The weight head Headw and motion head Heady are both implemented by a two-layer MLP, where the hidden size is 256.
Training. The model is trained for 60 epochs with 60k image pairs per epoch. We set the batch size to 128, and the learning rate is decreased by factor of 10 after 40 epochs. The initial learning rate and other training settings are set the same as
13

---

Efficient Motion Prompt Learning for Robust Visual Tracking
EAO
EAO
0.575
0.575
0.572
Default setting
0.572
Default setting
0.565
0.565
0.565
0.557
0.555
0.555
0.553
0.551
0.545
0.545
0.536
0.535
0.535
0.526
Baseline: 0,530
Baseline: 0.530
0.525
0.525
0.5
0.7
Probability of CutMix
Sparse degree
(a)
(b)
Figure 8. Impacts of data augmentations on VOT2022.
corresponding baseline trackers. For the motion input, we sparsely sample T -iramess trajectory either before or after the current search frame. In trajectory, we employ equally spaced sparse sampling to preserve the original motion pattern of the target. Such offline trajectory generation strategy not only simplifies and accelerates the training process, but also allows the model to confront real tracking noises and learn adapt motion trajectories with varying qualities. In addition, we adopt CutMix strategy (Yun et al., 2019) on search images to simulate challenges like background clutter. Specifically, we crop the object area from another frame, and paste it on random position of the search image. This data augmentation offers more chances for our model to learn how to effectively fuse and utilize extra motion information. The following Section will analyze the impacts of the adopted data augmentations.
LaSSOTETi
[VOT2022]
AUC
EAO
49
0.575
0.572
48.7
48.5
48.2
0.562
0.565
0.558
0,564
48.0
0.555
47.5
0:546
0.554
47.8
0.545
47
46.9
47.1
0.535
46.5
0.53
46.5
46
0.525
0
10
20
30
40
50
Figure 9, Impact the trajectory length. Our MPT obtains the best performance when the length of trajectories T 30.
C. Extended Ablation Study
C.I. Analysis of Data Augmentations
Fig. illustrates the individual impacts of CutMix and sparse sampling. For the CutMix, training our method without using CutMix strategy (f...brbbyillity is set to 0) obtains slightly inferior performance than the baseline visual tracker. The potential reason is that most original training samples are easy for the baseline visual model, limiting chances for our motion model to learn the ability to complement and optimize tracking predictions. CutMix, however, can simulate hard scenes by introducing distractors and slight occlusion, which mitigates this training issue. Fig. 8(a) shows that the best performance of our model attained when the probability of CutMix is set to 0.5. Therefore, we set the probability of 0.5 as the default in our experiments.
14

---

Efficient Motion Prompt Learning for Robust Visual Tracking
Table Quantitative analysis performance bias
LaSOT
EXT Num
LaSOT
TNL2K
ALL
Num
Gain
Gain
Num
Gain
Num
Gain
50
+4.8
80
+4.8
284
+2.3
414
+4.0
OSTrack B256
230
-0.7
70
-1.6
416
-1.4
716
-1.2
40
+5.6
71
+4.5
252
+2.1
363
+4.1
OSTrack-Baa -3344
240
-1.3
79
-0.5
448
-0.7
767
-0.8
44
+5.5
70
+2.8
271
+3.8
385
+4.0
Seq/Trck-B226
236
-0.2
80
-0.4
429
-0.4
745
-0.3
38
+9.3
76
+4.4
234
+2.8
348
+5.5
SeqTrrck-L3384
242
+0.2
74
-1.8
466
-0.3
782
-0.6
43
+5.9
78
+4.8
255
+3.2
376
+4.6
ARTrack-B22
237
-1.9
72
-0.9
445
-0.8
754
-1.2
For sparse sampling, increasing the range of sparseness for trajectory sampling enables the model learn from more diverse motion imputs, but it concurrently means that model faces more intricate and challenging motion patterns during the training. As shown in Fig. 8(b), the sparseness of 5 is optimized choice, which is also our default setting.
C.2. Impact of Trajectory Length
Fig. 9 illustrates the impact of the motion trajectory length T the performance of our MPT. Our method exhibits improved performance T increases, and reaches its peak at T = 30. Since the longer trajectory might bring more motion noises, further increasing the trajectory length results in slight performance degradation. Therefore, we set T = 30 in our experiments. This ablation study confirms our conjecture that long-term temporal information can provide richer tracking cues. Different from prior works (Kim et al., 2022; Wei al., 2023) employing heavy temporal processing mechanisms, our lightweight MPT independently encodes motion inputs using negligible computing resources, which allows the model to manage long-term motion trajectories.
D. In-depth Performance Analysis
D.1. In-depth Analysis about Performance Bias
Aiming to reduce tracking failures by motion cues, our MPT can significantly improve tracking robustness over accuracy. Our advantages are more evident in hard scenarios (like VOTs, LaSOTexT). This may cause slight bias the overall performance across different trackers or benchmarks. To give evidence this conjecture, we dynamically split each dataset into two subsets based on performance baselines, namely hard casy sets, As shown in Tab. the hard set (i) includes hard videos with baseline AUCs below 50.0%, while the easy (2) includes the remaining videos. Hard set proportions of LaSOT, TNL2K, LaSOTEXT 0.4%, 0.6%, 1.6%. Moreover, our MPT shows better consistency in hard / easy sets, achieving average AUC changes of +4.4% / -0.8%, with small standard deviations of 0.6 0.4 across five baselines.
around 15%, 30%, 50%. This aligns with the trend in average performance gains:
D.2. Attribute Analysis
For more intuitive observation of the impacts of our MPT on different attributes, we report performance gains achieved by our MPT across different attributes on LaSOTEXT (Fan et al., 2021), shown as Fig. 10. As we can see, our MPT significantly enhances the baseline tracker when faced with the challenge background clutter, obtaining 6.3% AUC improvements. Besides, challenges like viewpoint change, partial occlusion, low resolution, deformation, and full occlusion can also be better addressed by our MPT.
15

---

Efficient Motion Prompt Learning for Robust Visual Tracking
Aspect Ratio Change
Viewpoint Change
Background Clutter
Scale Variation
4.5
5/4
Camera Motion
1,2
Rotation
Deformation
0.3
0.
-1.5
1.
311
1.1
Partial Occlusion
2.7
Fast Motion
1.3
2.1
Out-of-View
2.8
2.5
Full Occlusion
Motion Blur
Illumination Variation
Low Resolution
AUC
Norm Precision
Figure 10. Performance gains of our method on each attribute.
E. Visualizations
E.1. Visualization of the Training Process
As shown in Fig. 11, we present the optimizing process of our model by visualizing dynamic changes of (a) loss values, (b) Intersection-ove--Unine (IoU), and (c) location accuracy of the score map (Acc). Here, we consider the location prediction to be accurate if the position of the maximum value in the score map coincides with the center the target groundtruth. As we can see, due to fine-tuning few parameters, the model exhibits fast convergence during the training process. Moreover, compared with the baseline visual model OSTrack-B22 (Ye al., 2022), our motion prompt-based model yields more accurate score map (see Fig. 11())), consequently leading These curves further substantiate the efficacy of our MPT method in terms of both training efficiency and performance.
precise prediction of the object position (see Fig. 11(b)).
Loss
IoU
Ace
3.3
2.3
0.7
1.3
0.77
0.68
Epoch
Epoch
Epoch
0.75
LOSSMPT (Train)
loUMeT (Train) loUy (Train)
AсcмPт (Train)
Accy (Train)
LosSMPT (Val)
loUMer (Val)
loUy (Val)
АсcиPт (Val)
Accy (Val)
(a)
(b)
Figure 11. Visualization of optimizing process of model during training.
E.2. Qualitative Comparison
To intuitively demonstrate the effectiveness of our MPT, we make qualitative comparison of our methods with corresponding baseline trackers and other ofate-of-the-an art (SOTA) trackers. As shown in Fig. 13, our MPT allows SOTA vision-based baseline trackers to perform more robustly when faced with challenges, like distractors, occlusion, severe appearance changes, fast motion, etc.
16

---

Efficient Motion Prompt Learning for Robust Visual Tracking
1674
1749
2022
# #3
321
# 3744
78
765
# 957
GroundTruth
OSTrack MPT
OSTrack
Figure 12. Several failure cases of our method ("_MP""..
cwm/ i -on.-o---tt--taaccat to show more qualitative comparisons on three challenging sequences. For each sequence, we visualize bounding boxes predicted by our method ("+MPT") and the corresponding baseline tracker OSTrack. We also compare score maps output by these two trackers. The historical trajectories are plotted in our score maps (PPT", Compared with the baseline tracker, our MPT can utilize object motion cues to help the tracker recognize the interested object from multiple distractors. Besides, when faced with severe occlusion, our MPT can predict the next position of the object based mainly the motion prompt, thereby avoiding tracking drift.
Moreover, we make a video (please find it at https::
However, since motion trajectories often include inherent noises like wrong observations, low-quality motion cues sometimes interfere with visual cues. We propose an adaptive weight mechanism mitigate this phenomenon, but some tracking failures still inevitably happen. Some failure cases our method are shown in Fig. 12.
17

---

Efficient Motion Prompt Learning for Robust Visual Tracking
263
1698
# 3810
1643
2135
# 2836
Noc Dolg
0
Humaa
Nox Devil
2-0
B-Hmma
Noc: Dovil
0.0
HLria
123
519
1173
1814
# 2710
347
#481
87
876
1113
GroundTruth
AiATrack
ARTrack
ROMTrack
OSTrack
OSTrack_Mi MPT
SeqTrack
Teq MPT
Figure Qualitative comparisons of our trackers ("LMPT") with SOTA trackers.
18


Reviews:
[{"review_id": "PhKUqCPaE1", "reviewer": "Reviewer_7U4B", "summary": "This paper proposes an Efficient Motion Prompt Learning (EMPL) framework for robust visual tracking. The key idea is to leverage historical motion trajectories as prompts that are encoded via three components: Spatial Encoding (SPE), Point Embedding, and Temporal Positional Encoding (TPE).\n\n## update after rebuttal\nI have reviewed the rebuttal and found that all my concerns were thoroughly addressed with clear analysis and additional experiments. I therefore maintain my original score of 4 (Accept).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback, and address the concerns below.\n1. **Appearance-based prompt trackers:** Taking PromptVT as an example, appearance-based prompt methods efficiently enhance appearance features using dynamic appearance information, thereby enabling the model to be robust to appearance changes during tracking. However, the tracking problem is still treated as a discrete image processing task. In contrast, our MPT leverages the temporal consistency of videos by incorporating continuous object trajectories into the visual tracking framework. Our method proves to be more effective in addressing visually challenging scenarios, such as occlusions and distractors. These two kinds of methods are complementary: while appearance-based prompt methods focus on enhancing visual features, our MPT can integrate historical motion cues into this visual framework, achieving a joint enhancement of both visual and motion information.\n\n   Thanks for your valuable advice. We will add the above comparison and related references to our paper. \n\n2. **Sensitivity to the trajectory quality:** The proposed adaptive weight mechanism in our method can mitigate the impact of noisy trajectories to some extent. As shown in Figure 3 (Page 8 in submitted paper), the predicted weights align with the accuracy of motion predictions which implicitly reflect the quality of input trajectories. \n|Success Rate|[0.0,0.1)|[0.1,0.2)|[0.2,0.3)|[0.3,0.4)|[0.4,0.5)|[0.5,0.6)|[0.6,0.7)|[0.7,0.8)|[0.8,0.9)|[0.9,1.0)|\n|-|-|-|-|-|-|-|-|-|-|-|\n|OSTrack_MPT|1%|21%|26%|32%|42%|63%|82%|94%|99%|100%|\n\n   To further analyze the sensitivity of our MPT to varying noisy trajectories, we evaluate the success rate (IoU>0.5) under different quality trajectory inputs on LaSOTEXT, as shown in the table above. Here, [a,b) represents the range where the average IoU of the trajectory is no less than a and less than b. The following conclusions can be drawn:\n   * The success rate increases significantly with the improvement of trajectory quality, especially when the trajectory IoU is in the range of [0.6, 1.0). \n   * Our method shows a certain level of robustness to noisy trajectory inputs, obtaining a success rate of 21% even when the trajectory IoU is in the range of [0.1,0.2).\n\n   **Due to the character limit, please find additional deep analysis of success/failure cases in our response to Reviewer EZ8N (2)**\n\n3. **Architecture of fusion decoder:** According to your advice, we supplement two sets of ablation studies in terms of the architecture of the fusion decoder, as shown in the table below. \n||OSTrack-B256|+MPT(Ours)|+OnewayCA|D3|D4|\n|-|-|-|-|-|-|\n|VOT22(EAO)|0.530|0.572|0.554|0.555|0.555|\n|LaSOTEXT(AUC)|46.9|48.7|43.8|47.7|49.2|\n\n   * First, our method employs bidirectional Cross-Attention (CA) for better alignment and fusion. Compared with one-way CA, our method obtains an average performance improvement of 3.4% across the two datasets, which demonstrates the effectiveness of our fusion decoder architecture.\n\n   * Second, our method implements a two-layer fusion decoder, improving the baseline by an average performance of 3.0% across the two datasets. Deeper variants (D3/4) exhibit slightly inferior or comparable performance, with average improvements of 1.7% and 2.4%, respectively. Therefore, the two-layer structure excels in both performance and efficiency. In addition to reducing efficiency, there are two potential risks for a deeper-layer fusion decoder. First, since the number of prompt tokens is much smaller than that of visual tokens, a deeper fusion process may lead to the dilution and loss of motion prompts, thereby weakening the effectiveness of the motion information. Second, deeper-layer decoders introduce more trainable parameters, increasing the risk of overfitting.\n\n   Thanks for your valuable advice. We will add the above experimental analysis to our paper.", "claims_and_evidence": "The paper claims that incorporating motion prompts can significantly improve tracking robustness, particularly in the presence of occlusions and distractors. Experimental evidence on several benchmarks (VOT, LaSOT, etc.) supports this claim.", "methods_and_evaluation_criteria": "The proposed method employs SPE, Point Embedding, TPE to map motion trajectories into a high-dimensional vector space aligned with visual features. The evaluation is based on established tracking benchmarks, and ablation studies are performed to validate the key design choices.", "theoretical_claims": "The paper includes a theoretical rationale behind the non-linear initialization of the Temporal Positional Encoding (TPE) using Nyquist frequency principles. The provided intuition is compelling.", "experimental_designs_or_analyses": "The experimental design is comprehensive, testing on multiple datasets and including ablation studies on individual components.", "relation_to_broader_scientific_literature": "The paper builds on established ideas in positional encoding and prompt learning, extending them to the domain of motion cues for visual tracking.", "essential_references_not_discussed": "Can you discuss PromptVT[A] in this paper? \n\n[A] Zhang, Minghua, et al. \"Promptvt: Prompting for efficient and accurate visual tracking.\" IEEE Transactions on Circuits and Systems for Video Technology (2024).", "other_strengths_and_weaknesses": "Strengths:\n1. Innovative use of motion trajectories as prompts, which is relatively unique in the tracking literature.\n2. The design is efficient, both in terms of computational cost and training resources.\n\nWeaknesses:\n1. The method’s reliance on accurate spatial (SPE) and point (Point Embedding) encodings may be vulnerable to errors in detection; any mistakes could potentially lead to cascading errors.\n2.  There is no direct comparison with appearance-based prompt methods (e.g., PromptVT), which would help contextualize the benefits of using motion cues exclusively.\n3. The Fusion Decoder is fixed at two layers without analysis on whether a deeper or alternative architecture could further improve performance.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Have you considered how your motion prompt framework might be integrated with appearance-based prompt methods like PromptVT?\n2. Can you provide more insights or experiments on how sensitive your model is to errors in the initial detection of bounding box corners?\n3. While you have chosen a two-layer Transformer for the Fusion Decoder, did you perform any preliminary experiments with varying depths?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OhhY657mnz", "reviewer": "Reviewer_kbHt", "summary": "This paper presents a flexible and efficient motion prompt tracking method, which complements existing trackers based on motion prompts. It includes a motion encoder with three different positional encodings, a Transformer-based fusion encoder, and an adaptive weight mechanism. The tracker is evaluated on several benchmarks and achieves encouraging results.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback, and address the concerns below.\n1. **More efficiency metrics:** We have provided several efficiency metrics for both baselines and our methods in Table3 (Page 7 in submitted paper), including memory footprint (Params, and Mem), latency (FPS), and training time (Time_tr). Thanks for your advice, we will add the FLOPs metric to our paper to further demonstrate the efficiency, as shown in the table below.\n||OSTrack |+MPT(Ours)|ARTrack|+MPT(Ours)|\n|-|-|-|-|-|\n|FLOPs (G)|29.1|33.7|40.3|41.9|\n\n2. **Impact of fine-tuning strategies:** According to your advice, we supplement the ablation study in terms of fine-tuning strategies. Our method freezes all of the baseline parameters, and only fine-tunes parameters of MPT during training. Here, we also evaluate two variants. FinetuneAll fine-tunes both MPT and baseline parameters, while FinetuneHead fine-tunes parameters of MPT and the tracking head of the baseline, i.e., only freezing the backbone of the baseline model. As shown in the table below, our method slightly outperforms the other two variants, and is more efficient in terms of memory footprint and training time. The potential reason is that freezing the baseline parameters enables the model to focus on learning motion cues and the fusion mechanism.\n||OSTrack-B256|+MPT(Ours)|FinetuneAll|FinetuneHead|\n|-|-|-|-|-|\n|VOT22(EAO)|0.530|0.572|0.567|0.548|\n|LaSOTEXT(AUC)|46.9|48.7|48.3|48.1|\n|Mem(GB)|35.0|6.1|37.5|7.7|\n|Training Time(h)|38|5|8|6|\n\n   Thanks for your valuable advice, we will add the above experimental analysis to our paper.\n\n3. **Comparison with PEFT-based tracking methods, like LoRAT:** PEFT methods primarily include adapter-based and prompt learning-based approaches, both of which demonstrate great efficiency in various downstream tasks. LoRAT successfully applies LoRA, a typical adapter-based PEFT method, to the tracking task. The main contribution of LoRAT lies in designing a LoRA-friendly visual tracking model to achieve efficient training and inference. In contrast, our work aims to build a joint tracking framework that incorporates both visual and motion cues, leveraging the temporal consistency of videos and complementing visual limitations. Additional motion inputs are naturally compatible with prompt learning-based PEFT, which is why we choose prompt learning to ensure efficiency.\n\n   Furthermore, we integrate our MPT into LoRAT, as shown in the table below. Results demonstrate that our MPT improves LoRAT by an average performance of 1.5% across the two datasets, showing good complementarity of our method with such PEFT-based tracking methods. \n   ||LoRAT|+MPT(Ours)|\n   |-|-|-|\n   |VOT22(EAO)|0.541|0.558|\n   |LaSOTEXT(AUC)|49.9|51.2|\n\n\n   Thanks for your advice, we will add the above comparison to our paper.\n4. **Comparison with recent trackers:** Since our method is a plug-and-play module, and can be flexibly integrated into varying visual trackers to achieve performance complementarity, especially in challenging scenarios, the relative performance improvement over baseline methods can fairly demonstrate the effectiveness of our method. Thank you for your advice. We will add the above experimental comparisons with LoRAT to our paper to showcase the effectiveness of our method on recent trackers.\n5. **Minor performance gains for ARTrack and on some datasets:** Compared with ARTrack, we provide new insights into efficient temporal tracking. In addition, the apparent performance bias is primarily due to the varying distributions of easy and hard sequences across different datasets, which is further clarified through the quantitative analysis of performance bias in Table 6 (Page 13 of the submitted paper). Our method consistently exhibits robustness advantages in challenging scenarios, demonstrating its prospects for real-world tracking applications.", "claims_and_evidence": "Comprehensive analysis and support have been provided concerning the significance of motion patterns and the need for greater computational resources for trackers relying on sequential training. The MPT module exhibits outstanding performance on certain datasets, such as the VOT dataset; its enhancement on other datasets is less conspicuous.", "methods_and_evaluation_criteria": "The proposed MPT method makes sense as it addresses the challenges of incorporating temporal coherence in tracking. Its flexible integration and training-efficient design fit the problem well.", "theoretical_claims": "The MPT module can be flexibly integrated into existing vision-based trackers and enhance various baseline trackers, which is demonstrated through a description of its internal components and their interactions. For example, within a lightweight frame level fine-tuning, the MPT achieves comparable performance to those sequential-based SOTA methods. \nAlthough the text provides an overview of the design and anticipated functions of MPT, when it comes to trackers like Artrack, which already incorporate temporal information, the enhancements that MPT makes turn out to be fairly restricted. Likewise, its contributions on datasets such as LaSOT and TrackingNet prove to be rather meager as well.", "experimental_designs_or_analyses": "1. The Efficiency comparison is insufficient. For example, the increase in FLOPS ought to be provided.\n2. The authors are suggested to provide the results of full-finetuning the head instead of training an adaptive weight mechanism to support effectiveness and efficiency or using LoRA.", "relation_to_broader_scientific_literature": "This paper contributes to the broader literature by refining motion-prompt-based tracking through improved encoding, Transformer-based fusion, and adaptive weighting. It aligns with prior research on Transformers in tracking but introduces motion prompt mechanisms to enhance efficiency and flexibility.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths: The paper introduces a motion encoder with three different positional encodings, which appears to be an effective motion-tracking approach.\n\nWeaknesses:\n1.Include experimental results comparing with LoRAT and other recent trackers to ensure fair benchmarking.\n2. Provide efficiency metrics (e.g., FLOPs, latency, memory footprint) to justify the claim of being “efficient.”", "other_comments_or_suggestions": "More comparisons about PEFT-based tracking methods, like LoRAT, are suggested.", "questions_for_authors": "Please refer to the above suggestion.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RZWjGBgK5v", "reviewer": "Reviewer_EZ8N", "summary": "The paper presents a lightweight Motion Prompt Tracking module that integrates motion cues into vision-based trackers. Using a motion encoder with spatial, point, and temporal encodings and a Transformer-based fusion decoder with adaptive weighting, it improves robustness against occlusion and distractors. The method employs prompt learning, freezing the visual model while fine-tuning only the added modules.\n\n## update after rebuttal\n\nThank the author for the rebuttal. I will keep my original rating which was already positive.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive feedback, and address the concerns below.\n1. **Sensitivity to the trajectory quality:** The proposed adaptive weight mechanism in our method can mitigate the impact of noisy trajectories to some extent. As shown in Figure 3 (Page 8 in submitted paper), the predicted weights align with the accuracy of motion predictions which implicitly reflect the quality of input trajectories. \n|Success Rate|[0.0,0.1)|[0.1,0.2)|[0.2,0.3)|[0.3,0.4)|[0.4,0.5)|[0.5,0.6)|[0.6,0.7)|[0.7,0.8)|[0.8,0.9)|[0.9,1.0)|\n|-|-|-|-|-|-|-|-|-|-|-|\n|OSTrack_MPT|1%|21%|26%|32%|42%|63%|82%|94%|99%|100%|\n\n   To further analyze the sensitivity of our MPT to varying noisy trajectories, we evaluate the success rate (IoU>0.5) under different quality trajectory inputs on LaSOTEXT, as shown in the table above. Here, [a,b) represents the range where the average IoU of the trajectory is no less than a and less than b. The following conclusions can be drawn:\n   * The success rate increases significantly with the improvement of trajectory quality, especially when the trajectory IoU is in the range of [0.6, 1.0). \n   * Our method shows a certain level of robustness to noisy trajectory inputs, obtaining a success rate of 21% even when the trajectory IoU is in the range of [0.1, 0.2).\n\n   Thanks for your valuable advice, we will add the above analysis to our paper.\n\n2. **Deep analysis of success/failure cases:** To deeply explore the success/failure cases of our MPT, we evaluate the proportion (Rate) of these two conditions on LaSOTEXT, along with the corresponding average trajectory IoU (Avg_IoU_traj) and average IoU of the previous tracking result (Avg_IoU_pre). \n\n   Here, we define a success case when the predicted IoU of our MPT exceeds that of the visual baseline by more than 0.3. Conversely, when the predicted IoU of our MPT is lower than that of the visual baseline by more than 0.3, we define it as a failure case. As shown in the table below, success cases of our MPT usually occur when the historical trajectory IoU is around 0.72, while failure cases usually happen when the historical trajectory IoU is around 0.18. Additionally, our method is more likely to succeed when the tracking result of the previous frame is better, whereas poorer previous tracking results are more likely to lead to failure. This is mainly because the tracking result of the previous frame affects the current search area and the completeness of the target.\n   ||Rate|Avg_IoU_traj|Avg_IoU_pre|\n   |-|-|-|-|\n   |Success|2%|0.72|0.76|\n   |Failure|1%|0.18|0.1|\n\n   Moreover, we separately evaluate the distribution of trajectory IoU and previous frame IoU in both success and failure cases, shown in the following two tables. In success cases, the input trajectory IoU primarily falls within a relatively broader range of [0.6, 1.0), especially in [0.8, 0.9). In failure cases, the input trajectory IoU mainly falls within the range of [0.0, 0.2), particularly in [0.0, 0.1). This indicates that addressing misleading tracking issues caused by extremely poor-quality trajectories remains a challenge for our method. This is a limitation of our method that requires future research.\n\n   |Distribution_succ|[0.0,0.1)|[0.1,0.2)|[0.2,0.3)|[0.3,0.4)|[0.4,0.5)|[0.5,0.6)|[0.6,0.7)|[0.7,0.8)|[0.8,0.9)|[0.9,1.0)|\n   |-|-|-|-|-|-|-|-|-|-|-|\n   |IoU_traj|1%|2%|3%|4%|6%|6%|10%|20%|34%|15%|\n   |IoU_pre|2%|0%|1%|3%|4%|6%|9%|16%|37%|23%|\n\n   |Distribution_fail|[0.0,0.1)|[0.1,0.2)|[0.2,0.3)|[0.3,0.4)|[0.4,0.5)|[0.5,0.6)|[0.6,0.7)|[0.7,0.8)|[0.8,0.9)|[0.9,1.0)|\n   |-|-|-|-|-|-|-|-|-|-|-|\n   |IoU_traj|55%|12%|8%|6%|6%|5%|4%|3%|1%|0%|\n   |IoU_pre|74%|11%|4%|2%|2%|2%|1%|1%|1%|1%|\n\n   Thanks for your valuable advice, we will add the above analysis to our paper.\n\n3. **Extension to MOT:** Different from single-object tracking (SOT), typical multi-object tracking (MOT) frameworks involve object detection and association. Therefore, integrating our MPT into an MOT framework will differ from that of SOT. However, the core idea of our method remains highly compatible with MOT problem. Specifically, historical trajectories of each object can be utilized as prompts to enhance MOT performance. On the one hand, our MPT can be integrated into the object detection process to enable motion-aware detection, thereby improving detection accuracy. On the other hand, during the association process, our MPT can help maintain the coherence of the object trajectory, achieving more accurate matching.\n   \n   In addition, compared with SOT, MOT usually encounters more complex backgrounds with a higher number of distractors. As an advantage of our MPT, introducing motion prompts can potentially help address distractor challenges. However, complex scenes are more prone to generating noisy trajectories, which may also pose a risk of misleading tracking issues.\n\n   Thanks for your valuable advice, we will add the above discussion to our paper.", "claims_and_evidence": "Main Claims:\n- Motion prompts can effectively complement visual features to improve tracking robustness.\n- A non-linear temporal positional encoding is theoretically justified and empirically optimal.\n- The proposed module improves performance on multiple tracking benchmarks with minimal additional resources.\nEvidence:\n- Extensive quantitative results across benchmarks (VOT2018/2020/2022, LaSOT, LaSOTEXT, TNL2K, TrackingNet) demonstrate significant performance improvements.\n- Ablation studies highlight the contributions of each component (positional encodings, data augmentation strategies, trajectory length).\n- Visualizations and qualitative comparisons illustrate improved robustness over baseline trackers.", "methods_and_evaluation_criteria": "Methods:\n- Motion Encoder: Encodes historical trajectories using spatial, point, and non-linear temporal positional encodings.\n- Fusion Decoder: A two-layer Transformer network that fuses motion and visual features.\n- Adaptive Weight Mechanism: Dynamically balances the contribution of visual and motion cues based on an auxiliary prediction.\n- Training Strategy: Prompt learning with frozen visual backbones, reducing training time and memory usage.\n\nEvaluation Criteria:\n- Performance metrics such as Expected Average Overlap (EAO), AUC, precision, and robustness.\n- Comparison against state-of-the-art trackers and baseline models on standard benchmarks.\n- Analysis of training efficiency, convergence, and ablation of different components.", "theoretical_claims": "- The paper derives a non-linear temporal positional encoding using the Nyquist frequency principle.\n- It demonstrates mathematically that setting $\\alpha \\approx 7.23$ optimally controls the frequency range of the temporal encoding.\n- Experimental results confirm that this setting achieves the best performance compared to other $\\alpha$ values.", "experimental_designs_or_analyses": "Design:\n- Integration of the MPT module into various baseline trackers (OSTrack, SeqTrack, ARTrack).\n- Evaluation on multiple challenging benchmarks to test robustness and accuracy.\n\nAnalyses:\n- Ablation studies that individually assess the impact of each positional encoding, data augmentation strategies (CutMix, sparse sampling), and trajectory lengths.\n- In-depth performance analysis comparing hard versus easy scenarios to validate robustness improvements.\nVisualizations of attention maps and training dynamics to further substantiate the method's benefits.", "relation_to_broader_scientific_literature": "- The work builds on established vision-based tracking methods and recent advancements in prompt learning and Transformer architectures.\n- It positions itself relative to methods like ARTrack, ODTrack, and other sequential training approaches by emphasizing efficiency and robustness.\n- The integration of motion cues into visual tracking is contextualized in traditional motion estimation techniques and modern deep learning frameworks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Weaknesses:\n- The focus on robustness may come at a slight cost to tracking accuracy compared to some sequential training methods.\n- The approach might be sensitive to the quality of motion inputs, which could be further explored.\n- Limited discussion on potential challenges when extending the method to all moving subjects, which is multi-object tracking scenarios. Does it cause any conflict?", "other_comments_or_suggestions": "- A deeper analysis of failure cases and conditions under which the motion cues might mislead the tracker would be beneficial.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a lightweight, plug-and-play motion prompt tracking method to enhance existing vision-based trackers. This method integrates motion and vision cues for robust tracking through efficient prompt learning. Key components include a motion encoder with three positional encodings to encode long-term motion trajectories into visual embedding space, and a fusion decoder with an adaptive weight mechanism to dynamically fuse visual and motion features. Experiments on seven challenging tracking benchmarks show significant improvements in robustness with minimal training costs and negligible speed sacrifice. 

The authors have successfully addressed most of the reviewers' concerns, resulting in an "accept," a "weak accept," and a "weak reject" rating for the paper. The main concern of the reviewer rating "weak reject" is computational load, which the authors have addressed in the rebuttal. After discussion with the Senior AC, we concur with the positive assessment and recommend acceptance of the paper. The authors should address the reviewers' comments in the final version.
Decision: Accept (poster)