Reading Relevant Feature from Global Representation Memory for Visual Object Tracking
Authors: Xinyu Zhou
Pinxue Guo
Lingyi Hong
Jinglun Li
Wei Zhang
Weifeng Ge
Wenqiang Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: object tracking;global representation memory;transformer

Abstract: Reference features from a template or historical frames are crucial for visual object tracking. Prior works utilize all features from a fixed template or memory for visual object tracking. However, due to the dynamic nature of videos, the required reference historical information for different search regions at different time steps is also inconsistent. Therefore, using all features in the template and memory can lead to redundancy and impair tracking performance. To alleviate this issue, we propose a novel tracking paradigm, consisting of a relevance attention mechanism and a global representation memory, which can adaptively assist the search region in selecting the most relevant historical information from reference features. Specifically, the proposed relevance attention mechanism in this work differs from previous approaches in that it can dynamically choose and build the optimal global representation memory for the current frame by accessing cross-
frame information globally. Moreover, it can flexibly read the relevant historical information from the constructed memory to reduce redundancy and counteract the negative effects of harmful information. Extensive experiments validate the effectiveness of the proposed method, achieving competitive performance on five challenging datasets with 71 FPS.

Reading Relevant Feature from Global Representation Memory for Visual Object Tracking
Xinyu Zhou'
Pinxue Guo?
Lingyi Hong'
Jinglun Li²
Wei Zhang'
Weifeng Ge
Wenqiang Zhang1.2
'School of Computer Science, Fudan University, Shanghai, China
2 AAcademy for Engineering and Technology, Fudan University, Shanghai, China
zhouxinyu20@@ fudan. edu.en, pxguo21 @w.ffdam..c.ooooomommmmmm www..c...yoooon22.G. fud.n.e
jingli960423 @ @maaillalloooo gmail.com, weizhang @ fudan.edu.cn, weifeng, seffeng..e.icc. @ww.allgmm.wwwwoeooooooooo wqzzang Sudan.edu.cr
Abstract
Reference features from template or historical frames are crucial for visual object tracking. Prior works utilize all features from fixed template or memory for visual object tracking. However, due to the dynamic nature of videos, the required reference historical information for different search regions at different time steps also inconsistent. Therefore, using all features in the template and memory can lead to redundancy and impair tracking performance. To alleviate this issue, we propose a novel tracking paradigm, consisting relevance attention mechanism and a global representation memory, which can adaptively assist the search region in selecting the most relevant historical information from reference features. Specifically, the proposed relevance attention mechanism in this work differs from previous approaches in that it can dynamically choose and build the optimal global representation memory for the current frame by accessing cross- frame information globally. Moreover, it can flexibly read the relevant historical information from the constructed memory reduce redundancy and counteract the negative effects of harmful information. Extensive experiments validate the effectiveness of the proposed method, achieving competitive performance on five challenging datasets with 71 FPS.
Introduction
Visual object tracking (VOT) is a fundamental task in computer vision, which provides only the location of an object in the first frame and then accomplishes continuous predictions of the object's position in subsequent video frames. present, visual object tracking widely used in autonomous vehicle (25). video surveillance{48) and other scenes. As video frames are dynamic and continuously changing. presents significant challenge in addressing the appearance variation of the target and the changes in the environment. The current mainstream approach to addressing this problem is two perspectives: the first type
as shown in Fig. [ (),, is to perform interactive computations on all patches of a fixed template and the search region. The second 9 14 43), as shown [i) is to perform the interactive computation of template memory and the search region. Despite their great success, they neglect the appearance variation of the target and the changes in the environment. In other words, the relevant reference information for the search region at different time steps should be diverse. Retrieving all information from the
54
32
49
26
[1] 52
corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Relation Modeling
Relation Modeling
Relation Modeling
Tomplate
Memory
Relevant takens
Reglace with
Undate Tokens
Fg Template
-t-1
tokens
Search
Memory
Template
Search
GR Memory
Template
Search
One template with relation model
(b) Memory with relation modal
(c) Memory model
Three different methods of tracking pipeline. The purple dots represent the selected points from the new template that are updated into the memory, while the red dots and blue dots represent the selected points that are fed into the relation model.
Figure I:
template memory the search region is redundant, and some unnecessary historical information that does not match the current time step may potentially degrade tracking performance.
In this paper, we propose novel tracking paradigm (RFGM) to address the aforementioned problems. As shown in [i.. the core insight of our approach is extract relevant historical cues from memory for different search frames at different time steps. The approach dynamically selects historical information from memory that more suitable for the current search region, thereby improving tracking performance. Additionally, to obtain the target appearance features of the entire video sequence as reference features, we utilize the dynamic selection ability of our tracker to construct global representation memory(GR Memory), which dynamically stores the target appearance features of historical frames for adapting to the appearance variation of the target and improving the robustness of the model.
Specifically, as shown [i we design relevance attention specifically for tracking, which is different from previous attention 54] 51 5 based on global information reading. The relevance attention mechanism can read global information across frames on the timeline of the video sequence, and decide which token to use in the attention mechanism based on dynamic ranking. The most suitable historical information for the current frame adaptively selected from the memory through the attention mechanism. Each stage of the relevance attention discards unselected tokens, so the computational complexity of later layers is smaller than that previous layers. Finally, to construct the global representation memory, unlike directly replacing the earliest template with new one in (b),, we utilize the relevance attention to automatically select the target tokens from the new template and the original memory the new memory. The constructed GR memory retains the most important target features of the original memory and updates new target features into the memory, as presented in Fig After multiple updates, it will obtain the most representative target features in the entire video sequence.
Our contribution can be concluded in three-fold: (i)We propose novel tracking framework that adapts to changes in target appearance and background by constructing global representation memory at the token level across frames and reading from this memory to capture the most relevant features at the current time step:(ii) We design a relevance attention mechanism for the search region to selectively extract template features from memory. Simultaneously, it is utilized to update the global representation memory at the token level, reduce memory consumption and enhance tracking speed; (ifi) We conduct systematic experiments and validate the effectiveness of the proposed designs. Our tracker achieves competitive performance on five widely used benchmarks.
Related work
2.1 Visual Object Tracking Paradigms
In the past few years, many trackers have achieved great success in the field of visual object tracking. Typically, the linear matching method with cross- cros-coorrelation is used to locate the target position in the search frame[[ 49 26
K 2 SiamFC[]] is the first to propose the use of Siamese networks for feature extraction and linear correlation operation for visual object tracking.

---

further extends this method by feeding the features after linear correlation into three different prediction heads for classification, quality assessment, and regression, respectively, to decompose the tracking task. After the emergence of the transformer, the cross-attention operation [15
SiamFC++ [49
41 54
5 based on the attention mechanism replace the original cross-correlation as the mainstream paradigm for object tracking. TransT incorporates a CFA module to achieve bidirectional interaction between the template and the search region. AiAtrack [15] utilizes an attention in attention mechanism to achieve matching from the template to the search region, while OSTrack[5A4 [54] and MixFormaer[]] use one-stream paradigm to extract features and interact between the reference features and search region simultaneously.
However, the historical information required for different search regions at different time steps should be different. Therefore, cross-correlation and cross-attention operations introduce redundant information, potentially deleting tracking performance. To address this problem, we propose a novel tracking paradigm, RFGM, which automatically extracts the most suitable information for the search region from the GR memory, improving the adaptability to the appearance variation of the target and the changes in the environment.
2.2 Attention mechanism
The attention mechanism is widely employed in the tracking domain, and it has demonstrated remarkable performance, exemplified by TransT[55, STARK[51], and (54), etc. However, these mechanisms integrate information from all templates with the information from the search region, resulting in redundant template features. In contrast, deformable attention possesses feature selection. Deformable attention, derived from deformable CNN [59], has been applied in various fields such object detection and image classification [19]. It can adaptively select neighboring tokens based on the current query. For instance, Deformable DETR (60 uses deformable attention to regress the offset of the neighborhood based on the query coordinate in order to assist the query in selecting the most appropriate key and value, which accelerates the convergence of the DETR model for object detection. DAT [46] extends deformable attention to the backbone for classification. However, these deformable methods are based on regressing the coordinates of neighboring tokens within the current frame and cannot globally read all tokens across frames. Inspired by Dynamic ViT we design a relevance attention mechanism specifically for visual object tracking. Instead of regressing the coordinate offsets, we utilize ranking to dynamically assist the search region in globally reading useful features across historical frames.
2.3 Memory networks
Information from historical frames enables the network to adapt to the variation of appearance and background. Many approaches are devoted to template updates for improving performance in different
fields, such as video object segmentation 36 17| g 758) video obejct tracking [52 14, [11 [
etc. In video object segmentation, STM (66] and STCN construct memory networks. enabling the model to store extensive historical frame information for robust object tracking and segmentation. However, since memory incurs memory overhead and reduces model computation speed, it becomes challenging to store all historical frames during long-term video object segmentation. Therefore, XMem (7) introduces three distinct types of memory to facilitate long-term object tracking and segmentation. In object tracking, GradNet (27) updates the template by backwarding the gradients. With template controller based on LSTM, MemTrack [52] design memory network for robust object tracking. STMTrack [1] proposes a memory- tracker, which updates templates at fixed interval. Besides, Mixformer [9] and STARK[ST] design a scoring head to select the representative template. We utilize the proposed relevance attention to construct a GR memory. In contrast to previous methods that update the memory with the entire template, the GR memory adaptively selects relevant tokens from both the original memory and new template tokens using relevance attention. After multiple updates, the GR memory contains the most representative tokens of the target in the entire video sequence.
&
3

---

update
Relevance A ttmntion Transformer Block
CR. MeToo
TIP
A Adaptive Token Rank
Decoder
Atkaylive Tokken Rock
Thordoenee Blck
Multhoo--ie Atiitiie
Trendome Book
Milubeec-- Attration
Embodkan
Rank
FTN
PPN
MLP
Trmcking
Enooder
Decoder
Figure 2: The framework of RFGM. It consists of a memory, token filter(TF), an encoder, and decoder. The encoder is composed of Attention and relevance attention, while the decoder consists of a prediction head.
3 Method
3.1 Tracking with GR memory and relevance attention
Pipeline.
Encoder. As shown in 2ig The RFGM utilizes the vanilla V1T the encoder for feature interaction. The initial input of RFGM pair of images, which is referred to as template & E R3xH, and search E xxx,, The template and search in RFGM are first divided into N₂ and N₂ non-overlapping patches z² and x", respectively. The size of these patches is S S, and N, H.W./S² N₂ Hz/2 These patches series of patch embeddings (tokens) by a convolutional layer, T = [TF. . ... TK. } and 7 = "I..2..... [ T' E RIxC where C is the dimension of patch embedding. We concatenate the T and initial T and feed them into the encoder to obtain the tracking results of the first frame. In the subsequent tracking process, the predicted box will be used to crop a new template to adapt to changes in the target's appearance. Besides, we establish GR memory M to store the most representative target features throughout the video. Then, we feed M, new template, and T into Token Filter (TF) module to get the selected tokens from memory and new template. Finally, we feed the selected token and 7 into the encoder for feature interaction. The entire encoder consists of 12 transformer blocks, with relevance attention transformer blocks employed at the 4-th, 7-th, and 10-th layers.
then linearly mapped
Decoder. The prediction head is based on the decoder of [54] and consists of three branches: score branch, offset branch, and size branch. Each branch is composed of three convolutional layers. The score branch is used to predict the position R [0, 1] Ma Hi of the target center, the offset branch is used to compensate for the error E E [0, 1] 22 4t x 48 caused by downsampling, and the size branch predicts the height and width O E [0, 1 He x Hh of the target. The position with the highest score the score map of the score branch is selected, i.c., (Ir, yr) = ar argmar (r,, Ur) Rnya combined with the target size to obtain the final bounding box:
(x, ws h) (Æ, + E0O, Zr, Vr), 9r ((1, Tr, 3r), 0,,, Ir, vr), 01, Br))))
Global Representation Memory. Previous visual object tracking methods 15 9 43 511 generally push a template 2, to memory M,-1 = {20, 21. 21-1,,
1,,, M E x
at fixed time
intervals and discard the earliest template 21 in fisst-in-frst-ouut manner, where is the number of templates, and is the time step in the video. zo is the template cropped with ground truth, it will be stored in memory all the time. The memory updating can formulated
M, {zo,
1 U 2
(2)
However, directly pushing the entire template to memory will introduce distractor information, and discarding the earliest template is prone to losing representative features.

---

Token Filter
Adaptive Token Rank
GRL
Mullihheet-- Alleziiee
Mulliladd
Mallbbzee Albrriion
Crocstendd
FPN
Rank
Figure 3: The token filter consists of three regular transformer blocks and an adaptive token rank, which effectively updates the features in memory. GR-M M represents the global representation memory. stands for the new template, and represents the search region.
To address the aforementioned issues, instead of updating the whole template, we propose a global representation memory (GR memory) updating as token level, which can adaptively select the most representative tokens from both the previous time step memory Mp-1 and new template of 24. We initialize memory Mo (7 } and add the new coming template tokens T of to it when the memory is not full. When memory M = {T T ... T reaches certain maximum number Nmar of tokens, as shown in Fig we feed M,-1 and T into the Token Filter module to obtain relevant scores, and subsequently rank these tokens score in order, selecting the top tokens that most relevant to the current search region and merge them with the initial memory Mo to obtain the GR memory Mt. Because 7 = {Ti T2 ... TN,) ). the GR memory updating can be formulated as:
Topk(Rank' (MonkoM,,,.. (M, UT*))
Rank(M UT*) Rank (T TT?
TW. , 1 (T T ,
T, T T
T T
M, = Mo U 17 T T7
T
(5)
where m (TT.) ........... TN is the selected top l tokens, the size of m is Nmax. The way of memory updating can automatically discard some distractor information and select the most suitable tokens for the search region. Through multiple rounds of updates, the final memory includes the most representative target tokens throughout the entire video sequence.
Relevance attention. Previous feature interaction for the search region typically involves reading all information from regular memory M, {=o, 21. 2t using linear correlation 49 or cross-attention 54 9] which can be represented by the following formula:
Mi*r
Σ (zck2)
(6)
x., Mr
MHA(g,k,e))
where is linear correlation 49 M. the variables q, k, and v are used to denote the query, key, and value that are inputted into the multi-head attention HM AA) block However, utilizing all the information in the memory is redundant and some harmful information may deteriorate the performance.
Therefore, we propose relevance attention that can automatically select the most relevant tokens from memory, which can be represented by the following formula:
m'
T Topk (aan(M
(
where Mt is the memory tokens at time step and serves the input the I-th layer of the transformer block. Then, we take the tokens m' that are most relevant to current search region as the query, key, and value. We also add the search region tokens 1 to the query, key and value, resulting in the
5

---

final query, key, and value. The relevance attention can be formulated as:
q == == m' U T d =m' U. + MHA k,
= + FFN(O)
where 77 represents the search region tokens of 1-th layer transformer block, FFN represents feed-forward network (FFN), and of is the output [-th layer transformer block.
3.2 Adaptive Token Ranking
To enable ranking in relevance attention and GR memory, we design an adaptive token ranking module in the 4-th, 7-th, and 10-th transformer layers. Attention weights can represent the relevance between tokens in memory and the current search area feature. Therefore, we input the attention weights into multi-layer (M)))ptron(MLLP) to enhance the relevance, obtaining the relevant score for the current search area feature. Finally, we select the most relevant tokens by adaptive token ranking,
To simplify, we only use T of one template from the memory to demonstrate the formula. Therefore, the multi-head attention weights of T and T in I-th layer in encoder w E RaxN, where h 12 is the number of multi-head. The score prediction can be represented follows:
N2 w' = 2j=0 Wi
Ws E RhxNN Cw
(10)
N
π = Softmaa(MLL(u ))
RN
(11)
where 7y.0 represents the score of selecting the y-th token, Ty,1 represents the score of discarding the 3-th token, y E [0 ~ N₂)
Training stage. We keep all tokens that Ty.0 is greater than T.. As the token selection based on the score is non-differentiable, we exploit the Gumbel-Softmal function to enable gradient backpropagation. It can be formulated as:
D = Gumbel - Softmaz(T) E {0, 13.
(12)
where D is one-hot tensor with length .N., the differentiability of Humbel-Softmail enables end-to- end training. Finally, multiplying with attention weights w to discard low-scoring tokens:
Wmasked Concat(wo,o o wo,1 D........ Wm.n D). E (0, E N=,
(13)
where Wmasked represents the masked attention weights, and Concat represents recombining all (Wm,n D E RN, to Wmasked E h N, x N..
Inference stage. Instead of simply using D as mask for binary classification (keeping or discarding), we rank the scores of tokens in T (TT. . .............................. Th } and select the top A tokens, which allows selecting the tokens that are most relevant T It can be formulated as:
T TTop([Rank(1>)
(14)
3.3
Loss Fucntion
In the training stage, we generate a Gaussian map using ground truth and use focal loss [oss[29] to supervise the score branch. In addition, we use L1 loss and GIoU[39] loss to supervise the offset and size branches respectively. We also set up a ratio loss to supervise D the adaptive token ranking, which can help us constrain the ratio of kept tokens:
N₂ N
Lratio
3
(15)
BS
where q [0.9, 0.8,0,000 0.7] represents kept target ratio for stages, and stages include 4-th, 7-th, 10-th layer in encoder. B is the batchsize. Finally, the total loss of combination of the above objectives:
Lnctat = Ascore Jooeaat + Xiou Lngion + Au Lu + Aratio
(16)
where Ascore 1, Noo 2, Au 5 and Aratio 1 are the weights to balance the objectives.

---

Table 1: Comparisons with fatte-of-the- art methods with search resolution 300 x 300 three
large-scale benchmarks. The top three metrics are highlighted with red, blue, and green fonts.
TrackingNet (3
GOT-100 10k*
[2
LaSOT [T3
Method
AUC
PNrrm
AO
SRo.5
SRa.75
AUC
PNorm
P
RFGM-B256
84.7
89.6
84.6
70.3
82.0
76.4
Sim'Track 4
83.4
87.4
69.8
78.8
66.0
70.5
79.7
OSTrack-2556 54
83.1
87.8
82.0
71.0
80.4
68,2
69.1
78.7
75,2
Resolution< 300 x 300
SwinTrack
22
81.1
78.4
71.3
81.9
64.5
67.2
70.8
SLT [4]
82.8
87,5
81.4
67.5
76.5
60.3
66.8
75.5
SBT 47
70.4
80.8
64.7
66.7
71.1
AutoMatch
56
76.0
72.6
65.2
76.6
54.3
58.3
59.9
TransT 5
81.4
86.7
80.3
67.1
76.8
60.9
64.9
73.8
69.0
SiamAttn
551
75.2
81.7
56.0
64.8
SiamBAN 6
$1.4
59.8
DSTrpn 40
64.9
58.9
43.4
54.4
Occan 57
61.1
72.1
47.3
56.0
65.1
56.6
SiamPRN+
26
73.3
80.0
69.4
51.7
61.6
32.5
49.6
56.9
49.1
MDNet
35
60.6
70.5
56.5
29.9
30.3
9.9
39.7
46.0
37.3
4 Experiments
4.1
Implementation Details
Model. The proposed RFGM uses VIT-B as the encoder, namely RFGM-B256. VIT-B initialized using the pretrained weights from the MAE 18) RFGM-B256: the search region, which is 4 times the area the target object, resized to 256 256 pixels. The template, which 2 times the area of the target object, is resized to 128 x 128 pixels. The proposed relevance attention consists of three linear layers and the GELU activation function 20 The output dimensions of the three linear layers are 384, 192, and 2, respectively.
Training Our experiments are conducted on Intel(R) Xeon(R) Gold 6326 CPU 2.90GHz with 252GB RAM and 4 NVIDIA GeForce RTX 3090 GPUs with 24GB memory. This model training is divided into two stages. the first stage, number of templates set 3 frames and trained for 300 epochs, with 60k image pairs per epoch. The learning epochs. In the second stage, fine-tuning is performed based on the first stage, with the number of templates increased to 7 frames and trained for 50 epochs. For GOT- 10k, the model is trained for 100 epochs and decreases the learning rate at epoch 80 in the first stage. the second stage, the model finetuned for extra 30 epochs. We set the learning rate of the prediction module in the relevance attention of the encoder and the decoder to 4e-4 and set the learning rate of the remaining parameters in the encoder to 4e-5. Additionally, the weight decay is set to le-4. Our training datasets includes COCO LaSOT 13 GOT-1Ok (2), and TrackingNet 34] Data augmentation techniques include Random horizontal flip and brightness jittering.
decreases by factor of 10 after 240
Inference. The size of GR memory Nmax is set to x N₂ by default. The update interval of the GR memory is set to 5 for t <= 100, doubled every 100 frames until 500, and then remains 160. The gradual increase in the update interval is to reduce the accumulation of model errors caused by the template from inaccurate tracking results. 3 stages relevance attention is set to floor(3 x N, x 0.9), floor(3 x N₂ x 0.8) and floor(3 x N₂ x 0.7) respectively, where floor means rounding down to the nearest integer. We employ single NVIDIA GeForce RTX 3090 for inference.
4.2 State-of-the-Art Comparisons
We compare the proposed RFGM with state-of-the-ar methods on five tracking benchmarks, includ- ing large-scale Senchnarks(TrackingNN (34, GOT-1OK (2]],, and LaSOT and two small-scale benchmarks(OTB 45) and UAV TAAVI23(12 (3)).
TrackingNet. TrackingNet is a dataset containing large-scale testing videos, with total of 511 testing videos. As reported in Tab RFGM-B256 performs better than the recent ofate-of-the-art method SimTrack[[] with 1.4 AACC score improvement. Importantly, we only use ViT-B as our encoder and SimTTrack use ViT-I. as the encoder. addition, our model significantly outperforms other models with search resolution smaller than 300, achieving AUC of 84.7%, P Poorm of 89.6 %

---

Table 2: Comparisons with state-of -fttte--fhhheeeeee art methods with search resolution 300 300 three
large-scale benchmarks. The top three metrics are highlighted with red, blue and green fonts.
TrackingNet
34
GOT-
10k*
22
LaSOT [T3]
Method
AUC
PNorm
AO
SRa.3
SRo.75
AUC
PNorm
RFGM-B256
89.6
83.6
84.6
71.8
70.3
82.0
76.4
Resolution>> 300 x 300
OSTrack-384 [44
83.9
88.5
83.2
73.7
83.2
70.8
71.1
81.1
77.6
SwinTrack-384
28
84.0
82.8
72.4
80.5
67.8
71.3
76.5
Mixformer-L.
83.9
88.9
83.1
70.1
79.9
76.3
Mixformer-22K
9
$8.1
81.6
70.7
80.0
67.8
69.2
78.7
74.7
AiATrack
82.7
87.8
80.4
69.6
80.0
63.2
69.0
79.4
73.8
UTT [31
79.7
77.0
67.2
76.3
60.5
64.6
67.2
CT 41
86.7
78.9
65.4
66.2
75.2
70.9
STARK [1
82.0
86.9
68.8
78.1
64.1
67.1
77.0
RIS B7
81.6
86.0
79.4
69.7
76.2
73.7
Unicorn 50
83.0
86.4
82.2
68.5
Table 3: Comparisons with state-of-the art methods on two small-scale benchmarks, The top three
metrics are highlighted with red, blue and green fonts.
SiamRPN++ PrDiMP TransT
T:DiMP STARK
AiATrack CSWinTT
ToMP
Mixformer-I
26
[1]
5
43
51
15
41
332
ours
ОTB
69.6
69.6
69.4
71.1
68.5
69.6
70.1
70.4
71.5
UAV123
61.3
68.0
69.1
67.5
69.1
70.6
70.5
66.9
68.7
68.5
Speed(fps)
35
30
50
35
42
38
12
24
18
71
and P of 83.6. Notably, as shown in Tab comparisons. our model also surpasses all methods with search resolution greater than 300 in the TrackingNet benchmark.
GOT-10k. GOT-10k consists of a training set of 10,000 videos and validation set of 180 videos. Moreover, the training and test sets do not overlap, requiring trackers to have a strong generalization ability for unseen data. Following the official evaluation rules, we only evaluate models trained on the GOT-101 10k training set. As reported in Tab and Tab similar to the results on TrackingNet, our method surpasses methods with resolutions greater than 300 and smaller than 300 by using only search resolution of 256, achieving 74.1% % of AO, 84.6%of SRO.5, and 71 71.8% of SRO.75, This demonstrates the strong generalization ability of our model to unseen data.
LaSOT. LaSOT is a comprehensive benchmark for long-term tracking tasks, including test set comprising 280 videos. The average length of the videos the test set is 2448 frames. As shown in Tab[ī our model achieves the highest performance in P Pnorm and P, with scores of 82.0% and 76.4% respectively, outperforming other methods. It only lags behind SimTrack by 0.2% in terms of AUC, mainly due to their use of VIT-L while we only use ViT-B. In comparison with methods using search resolutions greater than 300 Tab 2 our model still demonstrates competitive performance on Lasot dataset.
ОТB, As shown in Tab OTB is classic dataset in the field of object tracking, consisting of 100 video sequences. RFGM demonstrates excellent performance on this dataset, This demonstrates that our method also exhibits generalization performance on the classical object tracking dataset.
UAV123. As shown in Tab UAV123 a dataset specifically designed for UAV (Unmanned Aerial Vehicle) target tracking, consisting of 123 video sequences with relatively small objects. RFGM achieves competitive performance on the dataset. Achieving a frame rate of 71 FPS, our approach significantly outperforms other methods in terms of speed.
4.3 Ablation study
Why does GR memory work? We conduct ablation experiments on different memory types, as shown in Tab[4] When using only one template, the performance is low because there no template update to adapt to the appearance variations of the target. When using regular memory updating with a fixed interval, competitive performance achieved on GOT-10k and TrackingNet, but the AUC on LaSOT is only 64.5. This is because LaSOT is a long-term benchmarks, and errors can accumulate in the memory when updating the template, resulting performance degradation. By using the center score from the prediction head, errors in the memory can be reduced, but it cannot store the

---

Template
stagel
stage2
stage3
Template
stagel
stage?
stage3
Template
stage!
stage2
stage3
Figure 4: Visualization of relevance attention. Taking one template example, white areas represent discarded regions, while the remaining areas represent the regions selected by relevance attention. Stages to 3 indicate the progressive application of three relevance attention layers.
Table Ablation study our proposed GR memory. One template represents only the use of the first frame as the template. Memory represents replacing an old template with a new template in fixed interval. Score memory represents replacing the template according the center score of the prediction head. The best metrics are highlighted with red fonts.
Net
34
GOT-1Ok*
22
LaSOT 13
Method
AUC
P PNrm
AO
SRo.5
SRo.75
AUC
PNorm
One template
84.0
88.2
82.5
70.4
80.0
66.3
69
80.1
74.0
Memury
84.5
88.8
82.65
71.9
82.0
69.4
64.5
73.9
68.2
Score memory
84.6
88.9
82.7
74.3
84.0
70.8
81.5
76.2
GR memory
84.7
89.6
83.6
74.1
84.6
71.8
70.3
82.0
76.4
most representative features. In contrast, our proposed GR memory can store the most representative features throughout the entire video sequence and also reduce enror accumulation in the memory, resulting in the highest performance. The visualization of GR memory updates illustrated in Fig5)
Attention comparison. As reported in Tab using only the attention mechanism from VIT cannot adaptively select the most suitable features from the reference for the current search frame. Therefore, its performance is lower than proposed relevance attention. It is worth noting that the baseline here includes GR memory, where all features have already undergone round filtering using relevance attention within the GR memory. In other words, even within the filtered GR memory,
Add
Memory
New template
GR Memory
New template
GR Memory
New templote
t=10
Memory
t=
New template
Memory
t=20
New templiste
GR Memory
b=225
New template
tomplate
Memory
tomplate
t+00
t-85
t+00
Figure Visualization of GR memory updates. Over time t, the GR memory accumulates an increasing number of representative penguin features. In the second and third rows, white tokens represent discards, while the other tokens retained in memory.

---

Table 5: Attention represents the original multi-heas head attention of VIT [2] in encoder. Relevance attention represents using relevance attention in our encoder. The best metrics are highlighted with red fonts.
TrackingNet
34
GOT-10k
22
LaSOT 13
Method
Macs
Parameter
AUC Norm
P
AO
SRo.5
SRa.75
AUC
Norm
P
Attention
84.5
89.3
83.7
73.9
84.1
72.6
69.8
81.3
75.7
40.00G
92.12M
Relevance attention
84.7
89.6
83.6
74.1
84.6
71.8
70.3
82.0
76.4
37.66G
92.35M
Table 6: Ablation experiments on memory size.
The numbers 64, 192, 256, and 448 represent the Table 7: The MACs and parameters measured token number in the memory
with and without deformation attention
64
192
256
448
w/o
stage
stages
stages
AUC
70.3
69.6
69.5
Macs
40.00G
39.63G
38.85G
37.66G
P Poorm
80.1
82.0
81.5
81.3
Parameters
92.12M
92.20M
92.27M
92.35M
74.0
76.4
75.6
75.6
further performance improvement can be achieved by applying relevance attention. Additionally, relevance attention can reduce the number of computational parameters (40.00G VS 37 37.66G) while maintaining negligible parameter overhead(92.12M) VS 92.35M). Tab also presents the macs and parameters with or without relevance attention. The visualization of relevance attention is shown in Fig[4
What is the best size of GR memory? As shown in Tab. we conducted ablation experiments on the size of the memory. We found that the larger memory size does not lead to better performance. In fact, when the memory size exceeds 192 tokens, the performance starts to decline. This is because storing templates in the memory based on incorrect tracking results can lead error accumulation. With a larger memory size, more errors are accumulated as the tracking progresses. Therefore, the optimal memory size for RFGM is 192 tokens.
5 Conlusion and Limitation
In this paper, we present a novel tracking framework tracking(RFGMM) consisting of relevance attention and GR memory. Relevance attention adaptively selects the most suitable features for the current search region, allowing for adaptation to target appearance variations and environmental changes. Additionally, we construct a GR memory that utilizes relevance attention to select features from incoming templates. Through multiple rounds of updates, the GR memory stores the most representative features from the entire video sequence. Comprchensive experiments demonstrate that method achieves fatee-of-the-ar performance, and ablation experiments verify the effectiveness of the proposed GR memory and relevance attention. Due to the template updates, incorrect information will be stored into the memory, resulting in the accumulation of distractors over the process of the object tracking, we plan to further investigate methods to reduce the accumulation of errors in the memory.
Acknovledgement This work was supported by National Key RD Program of China A0000AAA00830000 National Natural Science Foundation of China (No.62072112 and No. No.62106051) Scientifie and Technological innovation action plan of Shanghai Science and Tech- nology Committee NN.2255110222 (No.XM03211178) and the Shanghai Pujiang Program N60221PJ1400600
Fudan University Double First-class Construction Fund
10

---

References
[1] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully-convolttional siamese networks for object tracking. In European conference on computer vision, pages 850-865. Springer, 2016.
[2] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte. Learning discriminative model prediction for tracking. In Proceedings of the IEEF/CVF international conference on computer vision, pages 6182-6191,
2019. 2019.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part I 16, pages 213 229.. Springer, 2020.
[4] Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, and Wanli Ouyang. Backbone is all your need: a simplified architecture for visual object tracking. In Computer Vicion-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 375-392. Springer, 2022.
[5] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang. Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8126-8135, 2021.
[6] 7edu Chen, Bineng Zhong, Guorong Li, Shengping Zhang, and Rongrong Ji. Siamese box adaptive network for visual tracking. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6668-6677 2020.
[7] Ho Kei Cheng and Alexander Schwing. Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model. In European Conference on Computer Vision, pages 640-658. Springer, 2022.
[8] Ho Kei Cheng, Yu- Wing Tai, and Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in Neural Information Processing Systems, 14:117881111 11794, 2021.
[9] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13608-1361 2022.
[10] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 764-773. 2017.
[1I] Martin Danelljan, Luc Van Gool, and Radu Timofte. Probabilistic regression for visual tracking. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7183 7192, 2020.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth Transformers for image recognition scale, arXiv preprint arXiv:2010.1 11999, 2020.
16x16 words:
[13] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Mingzhen Huang, Juehuan Liu, Yong Xu, Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129(2):439 461, 2021.
[14] Zhihong Fu, Qingjie Liu, Zehua Fu, and Yunhong Wang. Stmtrack: Template- free visual tracking with space-time memory networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13774 13783, 2021.
[15] Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang, and Junsong Yuan. Aiatrack: Attention in attention for transformer visual tracking. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Purt XXII, pages 146-164. Springer, 2022.
[16] Dongyan Guo, Jun Wang, Ying Cui, Zhenhua Wang, and Shengyong Chen. Siamcar: Siamese fully convolutional classification and regression for visual tracking. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6269-627777 6677. 2020.
[17] P Guo, W Zhang, and X Li. Adaptive online mutual learning bi-decoders for video object segmentation. IEEE Transactions on Image Processing: Publication of the IEEE Signal Processing Society, 2022.
11

---

[18] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16000-16009, 2022.
[19] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[20] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv: 1606.08415, 2016.
[21] Lingyi Hong, Wei Zhang, Liangyu Chen, Wenqiang Zhang, and Jianping Fan. Adaptive sclection of reference frames for video object segmentation. IEEE Transactions on Image Processing, 31:1057-1071, 2021.
[22] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1562- 1577, 2019.
[23] Eric Jang. Shixiang Gu and Ben Poole. Categorical reparameterizatioo with -ooubel-offtmx.. arXiv preprint arXiv:161..01144 2016.
[24] Minji Kim, Seungkwan Lec, Jungseul Ok, Bohyung Han, and Minsu Cho. Towards sequence-lewe training for visual tracking. In Computer Vision ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 534-551. Springer, 2022.
[25] Kuan-Hui Lee and Jenq-Neng Hwang. On-road pedestrian tracking across multiple driving recorders. IEEE Transactions on Multimedia, 17(9):122-1143, 2015.
[26] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang. Junliang Xing, and Junjie Yan. Siamrpu++: Evolution of siamese visual tracking with very deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4282-4291, 2019.
[27] Peixia Li, Boyu Chen, Wanli Ouyang, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Gradnet: Gradient- guided network for visual object tracking. In Proceedings of the IEEE/CVF International conference on computer vision, pages 61 6162-6171, 2019.
[28] Liting Lin, Heng Fan, Zhipeng Zhang, Yong Xu, and Haibin Ling. Swintrack: A simple and strong baseline for transformer tracking, Advances in Neurul Information Processing Systems, 35:16743-16754, 2022.
[29] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980- 2988, 2017.
[30] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zimicck. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[31] Fan Ma, Mike Zheng Shou, Linchao Zhu, Haoqi Fan, Yilei Xu, Yi Yang, and Zhicheng Yan. Unified transformer tracker for object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8781-8790, 2022.
[32] Christoph Mayer. Martin Danelljan, Goutam Bhat, Matthieu Paul, Danda Pani Paudel, Fisher Yu, and Luc Van Gool. Transforming model prediction for tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 8731-8740, 2022.
[33] Matthias Mueller, Neil Smith, Bernard Ghanem. A benchmark and simulator for uav tracking. In European conference on computer vision, pages 445 461. Springer, 2016.
[34] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. Trackingnet: A large-scale dataset and benchmark for object tracking in the wild. Proceedings of the European conference on computer vision (ECCV), pages 300-317, 2018
[35] Hyeonscob Nam and Bohyung Han. Learning multi-domain convolutional neural networks for visual tracking. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4293-4300 2016.
[36] Scoung Wug Oh, Joon- Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space- time memory networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9226 9235, 2019.
12

---

[37] Matthieu Paul, Martin Danelljan, Christoph Mayer. and Luc Van Gool. Robust visual tracking by segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel. October 23-27, 2022, Proceedings, Purt XXII, pages 571-588. Springer, 2022.
[38] Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamievit: Efficient vision transformers with dynamic token sparsification. Advances neural information processing systems, 34:13937-13949 13949, 2021.
[39] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, lan Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 658-666, 2019.
[40] Jianbing Shen, Yuanpei Liu, Xingping Dong, Xiankai Lu, Fahad Shahbaz Khan, and Steven Hoi. Distilled siamese networks for visual tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(12):886-88009 2021.
[41] Zikai Song, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang. Transformer tracking with eyclic shift- ing window attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8791-8800, 2022.
[42] Paul Voigtlaender, Jonathon Luiten, Philip HS Torr, and Bastian Leibe. Siam r-cnn: Visual tracking by re-detection. In Proceedings of the IEEL/CVF conference on computer vision and pattern recognition, pages 6578-6588, 2020.
[43] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li. Transformer meets tracker: Exploiting temporal context for robust visual tracking. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 1571-15800 2021.
[44] Qiang Wang. Zhu Teng. Junliang Xing. Jin Gao, Weiming Hu, and Stephen Maybank. Learning attentions: residual attentional siamese network for high performance online visual tracking. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4854 4863, 2018.
[45] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Online object tracking: A benchmark. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 241 2411-2418, 2013.
[46] Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 4794-4803, 2022.
[47] Fei Xie, Chunyu Wang, Guangting Wang, Yue Cao, Wankou Yang, and Wenjun Zeng Correlation aware deep tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8751 8760, 2022.
[48] Junliang Xing, Haizhou Ai, and Shihong Lao. Multiple human tracking based multi-view upper-body detection and discriminative learning. 2010 20th International Conference on Pattern Recognition. pages 1701. IEEE, 2010.
[49] Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu. Siamfe++: Towards robust and accurate visual tracking with target estimation guidelines. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12549-12556, 2020.
[50] Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu, Towards grand unification of object tracking. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXI, pages 733-751. Springer, 2022.
[51] Bin Yan, Houwen Peng. Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10448-10457, 2021.
[52] Tianyu Yang and Antoni Chan. Learning dynamic memory networks for object tracking. Proceedings of the European conference on computer vision (ECCV), pages 152 152-167, 2018
[53] Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by foreground- background integration. In European Conference on Computer Vision, pages 332-348. Springer, 2020.
[54] Botao Ye, Hong Chang, Bingpeng Ma. Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In Computer Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXII, pages 341-357. Springer, 2022.
2022: 17th European
13

---

[55] Yuechen Yu, Yilei Xiong, Weilin Huang, and Matthew R Scott. Deformable siamese attention networks for visual object tracking. In Proceedings of the IEEE/CVF conference computer vision and puttern recognition, pages 6728-6737, 2020.
[56] Zhipeng Zhang. Yihao Liu, Xiao Wang, Bing Li, and Weiming Hu, Learn to match: Automatic matching network design visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13339-13348, 2021.
[57] Zhipeng Zhang. Houwen Peng, Jianlong Fu, Bing and Weiming Hu. Ocean: Object-aware awere anchor-free tracking. In European Conference on Computer Vision, pages 771-787. Springer, 2020.
[58] Zechu Zhou, Xinyu Zhou, Zhaoyu Chen, Pinxue Guo, Qian- Yu Liu, and Wenqiang Zhang. Memory network with pixel-level spatio-temporal learning for visual object tracking. IEEE Transactions on Circuils and Systems for Video Technology, 2023.
[59] Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. Deformable convnets v2: More deformable, better results. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 9308-9316, 2019.
[60] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-en end object detection. arXiv preprint arXiv:2010.04159, 2020.
14


Reviews:
[{"review_id": "RuDiNSlOHq", "reviewer": "Reviewer_vhAu", "summary": "This work proposes a deformable memory for visual tracking task to mine historical target features for enhancing the tracking performance. It is achieved by accessing global cross-frame features. Additionally, it possesses the capability to retrieve pertinent historical information from the constructed memory, thereby reducing redundancy and mitigating the detrimental impacts of irrelevant background features. The effectiveness of the proposed method is extensively validated through experiments, demonstrating competitive performance.", "questions": "plz refer to the weakness part. ", "limitations": "nothing.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Strong intuition to make use of historical features to enhance tracking performance. \n\n2. Differentiable learning of the memory ranking with gumble softmax is interesting. \n\n3. The design of an efficient online update in the deformable memory, which involves effective token discarding, is a noteworthy contribution.", "weaknesses": "1. The technical contribution of this work is limited.  The token ranking and memory discarding mechanism in this paper is proposed in previous work like OSTrack and STMTrack. The difference of detail design of this work is analogous to these components in previous works. \n\n2. Unfair performance comparison, the effectiveness of this work is not verified.  Though strong performance gain is achieved in the proposed tracker according to Tab.1 and Tab.2, the compared tracker only utilize first frame template without temporal contexts. Authors use strong temporal context in both training and inference phase, the performance of the tracker is certainly stronger than spatial only trackers. This work should further compare the proposed work with spatial temporal trackers like START-ST and MixFormer-ST, and give ablation stuides about the performance variation about directly using two template concatentation and the deformable memory for mining temporal context to demonstrate the effectiveness of this work.\n\n3. Reported tracking speed of 71 fps is not convincing.  According to tracking performance reported in Tab.3,  the proposed tracker is twice faster than ToMP and STARK. I think it impossible to have such tracking speed. Sec4.1 mentioned this work use ViT-B formulates a one-stream spatial temporal tracker, which has 12 attention layers. But as we known, ToMP has a much efficient network architecuture, why it's much more ineffecient compared to authors work ?\n\n4. Writing of the paper need to be improved. The main figures in the paper is messy, especially Fig.2. and there are minor grammer errors, like line 144, vdeio -> video. \n\n5. Advise on this work: The authors' attempt to address the long-standing and challenging problem of mining temporal context in visual tracking is indeed interesting. However, to further validate the effectiveness of their approach, it is recommended that they conduct a more fair comparison to existing methods. This will help establish the superiority of their proposed method more convincingly. Moreover, it would be beneficial for the authors to focus on developing techniques that effectively mine temporal context while minimizing the increase in the number of tokens in the one-stream tracking framework. This would help alleviate the computational overload brought about by switching from a spatial-only tracker to a spatial-temporal tracker. By addressing this challenge, the proposed approach would become more practical and efficient in visual tracking.\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your constructive and insightful comments. Thanks for your recognition of our work.\nPlease allow me to restate the novelty of our work.\n#### **Q1.The token ranking is proposed in OSTrack**\n**Different Motivations**\nOstrack employs attention weights to discard features by token ranking from the search region, thereby accelerating the model. \nOur method's motivation and contribution revolve around the exploration of the most fundamental challenge of objec appearance variation changes. The challenge has not yet been adequately addressed. Consequently, our approach leverages the capability of deformable attention to select features (ranking) in a global, cross-frame manner, constructing a memory with discriminative features to adapt to the core challenge of object appearance variation.\nMoreover,  we extract features most pertinent to the current search region from the memory as reference information. **To the best of our knowledge, we are the first to propose selecting historically relevant features most closely related to the current frame to address the challenges of object appearance and background changes**.\n\n**Targeting different Entities**\nOstrack discards feature (token ranking) in the search region.\nIn contrast, our approach involves selecting features (token ranking) from historical frame information and the memory.\n\n**Different Implementation Approaches**\nOSTrack employs attention weights for ranking, while we utilize Gumbel-Softmax to select tokens through a classification network, predicting scores implicitly.\n\n#### **Q2.The memory discarding mechanism is proposed in STMTrack**\n**Deformable Memory Superior to STMTrack Memory**\nSTMTrack constructs a memory by a process of adding and discarding entire template to update it, which akin to Mixformer, AIATrack, and STARK. **However, we identified drawbacks in this memory construction method:** it fails to retain the most discriminative features from historical frames across frames. Moreover, methods like STMTrack require a considerable number of frames to adapt to changes in target appearance, significantly impeding tracker speed and demanding substantial gpu memory resources, resulting in a runtime of only 37fps.\n\nBy leveraging the capabilities of deformable attention, we preserve the most discriminative features from historical frames across frames at the token level, creating a robust memory. This memory comprises only 192 tokens, enabling us to retain valuable information from historical frames. The smaller memory size reduces memory consumption, achieves 71fps, nearly twice as fast as STMTrack.\n\n**Template level VS Token level:** STMTrack adopts frame-by-frame updates, whereas our approach updates based on tokens, more efficient than STMTrack. \n\n#### **Q3. Unfair performance comparison...**\nSome high-performance trackers in Table 1 and Table 2 adopt a spatial-temporal approach, such as STARK, Mixformer, Swintrack, and AiATrack. Among these, STARK and Mixformer, as you mentioned, also employ a spatial-temporal strategy involving whole-frame updates. **Based on your valuable comment, for fair comparisons, we augmented the OSTrack with temporal context during both training and testing phases. As shown in the following table, we still achieved the highest performance compared to these spatial-temporal trackers.**\n Because the utilization of simplistic template updates that involve discarding older frames and incorporating new ones results in limited feature storage, preventing the establishment of a holistic video representation.\n|Method|temporal|TrackingNet| GOT-10k|Lasot|\n|:-:|:-:|:-:|:-:|:-:|\n|||AUC, PNorm, P|AO, SR0.5, SR0.75|AUC, PNorm, P| \n|OSTrack-256-temporal|√|83.8, 89.0, 82.8 |73.4, 83.3, 72.1 |68.5, 77.9, 74.8| \n|OSTrack-384-temporal|√|83.9, 89.4, 83.4 |73.2, 83.1, 71.4|71.1 81.2, 77.8|\n|SwinTrack-256|√|81.1, -, 78.4|71.3, 81.9, 64.5|67.2, -, 70.8|\n|SwinTrack-384|√|84.0, -, 82.8|72.4, 80.5, 67.8|71.3, -, 76.5|\n|Mixformer-L |√|83.9, 88.9, 83.1|-, -, -|70.1, 79.9, 76.3|\n|Mixformer-22k |√|83.1, 88.1, 81.6|70.7, 80.0, 67.8|69.2, 78.7, 74.7|\n|AiATrack|√|82.7, 87.8, 80.4|69.6, 80.0, 63.2|69.0, 79.4, 73.8|\n|STARK|√ |82.0, 86.9, -|68.8, 78.1, 64.1|67.1, 77.0, -|\n|**DefTrack-256**|√|**84.7, 89.6, 83.6**|**74.1, 84.6, 71.8**|**70.3, 82.0, 76.4**|\n|**DefTrack-384**|√|**84.9, 89.6, 84.3**|**75.6, 85.6, 72.5**|**72.5, 84.5, 78.7**|\n#### **Q4.Ablation study on using two template concatentation and the deformable memory ...**\nWe show one template, multi-template and our deformbale memory as the following table, which demonstrate the effectiveness of our proposed method.\n|Method|Deformbale training|TrackingNet| GOT-10k|Lasot|\n|:-:|:-:|:-:|:-:|:-:|\n|||AUC, PNorm, P|AO, SR0.5, SR0.75|AUC, PNorm, P| \n|one template|x|83.7, 87.9, 82.2 |71.7, 81.4, 68.9|67.7, 76.7, 73.1| \n|Multi-template|x|83.8, 89.0, 82.8 |73.4, 83.3, 72.1|68.5, 77.9, 74.8|\n|**Deformable memory**|√|**84.7, 89.6, 83.6**|**74.1, 84.6, 71.8**|**70.3, 82.0, 76.4**|\n#### **Q5.Reported tracking speed of 71 fps...**\nThe speed of the model is influenced not only by its size but also by the resolution of the input images. STARK employs a resolution of 320 for input images, while TOMP uses 480. In contrast, our model resolution is 256. And they utilize a two-stream framework, leading to two rounds of feature extraction, which increases computational load. Furthermore, our model is built upon the one-stream framework OSTrack, which achieves a speed of 105.4 FPS using the ViT-B architecture.\n#### **Q6.There are minor grammar errors..**\nWe apologize for any grammatical errors, and we will make careful revisions in subsequent versions.\n\n#### **Q7.Advise on minimizing the increase in the number of tokens in the one-stream tracking framework**\nOur deformable attention reduces the number of tokens in the template tokens every 3 layers, thereby reducing computational demands. In the future, we intend to delve deeper into research aimed at further reducing the token count."}, {"review_id": "tO5hnFVnQt", "reviewer": "Reviewer_jJzj", "summary": "The authors propose an improved memory mechanism user-initialized tracking. Firstly, instead of storing a fixed subset of previous frame encodings in memory they store a subset of tokens that are dynamically selected at every iteration. This makes memory more efficient and increases discriminability. Secondly, they propose a ranking-based attention mechanism that only retrieves the most relevant tokens from memory. The later has only a marginal effect on the performance though. For some reason, they call both modules 'deformable' though what's deformable about them is not quite clear to me. \n\nIn an experimental evaluation on several user-initialized tacking benchmarks the proposed approach performs on par with the state-of-the-art while being significantly faster. ", "questions": "Please provide an overview of the related work in VOS and highlight the relationship between your contributions and latest VOS methods (in particular XMem).\n", "limitations": "Limitations are addressed in the manuscript.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed approach seems sound.\n\nThe advantages of the approach are convincingly demonstrated (sota performance with significantly higher FPS compared to strongest baselines).\n\nThe paper is relatively well written, though there are still quite a few typos and grammatical mistakes.\n\nA minimal ablation study is reported. ", "weaknesses": "Related work overview ignores the works in video objet segmentation (VOS), which is effectively the same problem as the one studied in this paper, just operating on masks, not boxes. As a results, sota approaches for both problems are very similar. In particular, XMem [Cheng and Schwing, ECCV'22] is highly relevant. \n\nPresentation still requires significant improvement. In addition to the typos, sota results are incorrectly bolded for LaSOT large res (the proposed approach is not sota).\n\nThe benefits of the proposed approach are limited (though not insignificant). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We extend our heartfelt gratitude for your perceptive insights and your recognition of the unique and meaningful contributions made by our research. Your support is highly valued, and we would be honored if you could serve as an advocate for our work.\n\n#### **Q1.Related works about VOS**\n**Related work about VOS**:\nWe will include a review of VOS in our related work. VOS (Video Object Segmentation) is indeed closely related to VOT (Visual Object Tracking). I will incorporate VOS methods into the relevant literature later on. Specifically, VideoMatching achieves segmentation through matching, while MaskTrack propagates the mask from the previous frame to accommodate target appearance variations. OSVOS adapts to dynamic changes in target and background through online fine-tuning. FeelVOS and CFBI achieve high performance by matching the first frame and the previous frame. Subsequently, STM and STCN construct memory networks, enabling the model to store extensive historical frame information for robust object tracking and segmentation. This serves as ample evidence of the importance of historical frame information in VOS or VOT tasks. However, since memory incurs memory overhead and reduces model computation speed, it becomes challenging to store all historical frames during long-term object tracking and segmentation.\nTherefore, XMem introduces three distinct types of memory to facilitate long-term object tracking and segmentation. These include working memory, which stores recent frame-level information for immediate utilization; long-term memory, which stores features discarded from the working memory and compresses them to reduce memory overhead while preserving historical features to the greatest extent possible; and Sensor memory, which employs GRU to retain low-level information from the first few frames, compensating for the deficiencies of working memory and long-term memory in this aspect.\n#### **Differences with Xmem**\n**Different Motivations**: XMem is designed to reduce memory consumption for long-term tracking and segmentation while preserving a maximum amount of historical frame information. Over time, features that enter the memory first may be discarded. In contrast, our deformable memory is designed to selectively choose discriminative features across frames in a global manner, independent of time. Importantly, our deformable attention allows us to selectively retrieve features from memory that are most relevant to the current search region for tracking. For instance, certain features present in the first and second frames might no longer appear in the tenth frame, yet they remain useful for the current frame. In this context, we need to selectively read relevant features for tracking while minimizing the impact of irrelevant features on performance. XMem, on the other hand, utilizes all the information within memory. To the best of our knowledge, whether in VOT or VOS, we are the first to propose this selective feature retrieval concept from memory.\n\n**Different Memory Sizes**: XMem employs three memory types that are larger in size compared to our memory. Our memory, however, contains only 192 tokens, equivalent to the size of three templates.\n\n**Different Update Approaches**: XMem's working memory stores frame-level information. When the memory limit is reached, surplus features are transferred to the long-term memory, which employs a least-frequently-used (LFU) mechanism with a fixed threshold for feature discarding. The sensor memory is updated using GRU. In contrast, our approach relies entirely on deformable attention to select discriminative features as the memory from existing memory and new templates, discarding the remaining features without requiring a threshold.\n\n#### **Q2.Typos**\nWe sincerely apologize for any typographical errors. We will make every effort to thoroughly review and rectify them in our subsequent revisions.\n\n#### **Q3.The benefits of the proposed approach are limited (though not insignificant).**\n**Memory ablation study**: The limited performance improvement observed in the previous memory ablation experiments can be attributed to the fact that I applied the deformable training approach to the regular memory methods as well. This led to a relatively constrained enhancement in results. However, upon removing the deformable training aspect, as illustrated in the table below, we showcase the efficacy of our deformable memory. Notably, there is a significant improvement, particularly evident in the LaSOT dataset where Pnorm exhibits a notable enhancement of 4.1%.\n|Method|Deformbale training|TrackingNet| GOT-10k|Lasot|\n|:-:|:-:|:-:|:-:|:-:|\n|||AUC, PNorm, P|AO, SR0.5, SR0.75|AUC, PNorm, P| \n|one template|x|83.7, 87.9, 82.2 |71.7, 81.4, 68.9|67.7, 76.7, 73.1| \n|Multi-template|x|83.8, 89.0, 82.8 |73.4, 83.3, 72.1|68.5, 77.9, 74.8|\n|**Deformable memory**|√|**84.7, 89.6, 83.6**|**74.1, 84.6, 71.8**|**70.3, 82.0, 76.4**|\n\n**Thank you once again for your valuable suggestions. The field of VOS has made significant progress in exploring the construction of memory and historical frames, as evidenced by contributions such as Xmem, STM, and STCN. In contrast, recent advancements in the tracking domain have primarily relied on powerful backbones and pre-trained models, neglecting the essential aspect of historical frame exploration. By incorporating VOS concepts into the related works, researchers in the tracking field can gain insights into the focal points of similar domains, thereby further enhancing the quality of our manuscript.**\n"}, {"review_id": "4QTQ9fgzkX", "reviewer": "Reviewer_nyT7", "summary": "The paper presents a method to dynamically select templates for visual tracking. Based on the vision transformer (ViT) framework, template selection is formulated as token selection. A prediction module is proposed to rank tokens. Top-K tokens from the history memory is selected to interact with tokens from the search region. The proposed method is evaluated on five benchmark datasets and achieves very competitive results compared with state-of-the-art methods.", "questions": "See Weaknesses.", "limitations": "See Weaknesses.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper present a method to dynamically select tokens from template memory which are used to interact with tokens from the search region. It can effectively select most relevant tokens and discard irrelevant tokens, therefore improving tracking performance. \n\nThe proposed method achieves very competitive performance compared with state-of-the-art methods. Also, ablation study is conducted to validate the effectiveness of different components in the proposed method. The proposed method runs fast at 71 FPS.", "weaknesses": "The novelty of the proposed method is limited. The idea of dynamic token selection has explored in previous work, e.g. [55], [A] and [B].\nEspecially, in both [A] and [B], the gumbel softmax is also used to learning the token score prediction module. The idea of token selection presented in this paper is  similar to the ideas in [A] and [B]. \n[A] Generalized Relation Modeling for Transformer Tracking, Gao et al, arxiv 2023\n[B] Dynamicvit: Efficient vision transformers with dynamic token sparsification, Rao et al, Neurips 2021. \n\nThe comparison with state-of-the-art methods is not very fair. Most previous methods listed in Tables 1 and 2 only used a single template from the first frame, while the proposed method uses multiple templates. It is expected that using multiple templates can boost tracking performance. How is the performance of the proposed method with search region resolution of 384 x 384?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly value your constructive and insightful feedback. In terms of novelty, please allow me to reiterate the distinctive aspects of our research\n#### **Q1. The idea of dynamic token selection has explored in previous work...**\n##### **Different Motivations:**\nGRM's motivation is the pollution of the template by the background of the search during cross-attention, thereby affecting the object localization in the search region. Therefore, GRM classifies tokens from the search features to alleviate the pollution of background features from the search for template features. DVIT, aims to enhance model speed by reducing the number of tokens in the search area.\n\nIn contrast, our approach involves the cross-frame selection of the most discriminative features within a video sequence to construct a deformable memory, addressing the challenge of object tracking concerning changes in object appearance and background. Besides, we employ deformable attention to select the most relevant features from the memory with respect to the current search area. This enhances the tracker's robustness to object appearance changes.\n\n##### **Different Target Focus:**\nGRM focuses on the classification of tokens within the search area, DVIT targets the discarding of tokens in the predicted image, whereas our method focus on the selection of tokens from historical frame and tokens within the memory.\n\n#### **Q2. the gumbel softmax is also .... the token score prediction module...**\n##### **Our contribution does not lie in Gumbel-Softmax:**\n As mentioned in our introduction. It serves merely as a technique employed to implement our proposed deformable tracking framework. Similar to using convolutions to construct a neural network or employing transformers for multi-modal interactions.\n##### **Our Contribution:** \nGiven that target deformation poses an inherent and highly challenging obstacle in object tracking, with no satisfactory resolution thus far, recent advancements in tracking performance have primarily stemmed from the utilization of powerful backbones and pretrained models. These advances, however, have largely disregarded an in-depth exploration of the fundamental problem itself.\n\nPrior research has either relied solely on single-template object tracking methods, which fail to address the challenges posed by variations in object appearance or has employed simplistic strategies such as adding or discarding templates for memory updates, as seen in methods like Mixformer, Aiatrack, Swintrack, and Stark. Such approaches lead to either an excessive number of templates stored in memory, significantly impeding tracker speed, or an inability to retain the most discriminative features from historical frames when utilizing a smaller memory. Moreover, certain features in the memory dynamically evolve over time and may no longer appear in the current search area, potentially affecting performance.\n\nIn order to address these issues, we introduce deformable attention, which enables the retention of the most discriminative features (at the token level) across the entire video sequence, while eliminating undesired features to construct a robust memory. Simultaneously, deformable attention is leveraged to access template features most relevant to the current frame, thereby adapting to changes in target appearance. To the best of our knowledge, our work is the first to introduce this novel concept in both the realm of visual object tracking (VOT).\n#### **Q3. The comparison with state-of-the-art methods is not very fair... it is expected that using multiple templates...**\nThe high-performing trackers in Table 1 and Table 2 adopt a multi-template approach, such as Mixformer, Swintrack, Aiatrack, and Stark. In addition, based on your valuable comment, for fair comparisons, we conducted multi-template experiments on the high-performance single-template method OSTrack (baseline for DefTrack). \nWhile we observe performance gains for short video datasets like Got10k (averaging 100-150 frames), we do not witness similar improvements for longer video datasets such as TrackingNet (averaging 441.5 frames) and LaSOT (averaging 2448 frames). This is attributed to the increased challenges posed by longer videos with more complex and challenging variations in target appearance and background. Simply relying on multiple templates is insufficient to address this issue.\n\nFurthermore, we conducted experiments for the **DefTrack 384x384 model**. As shown in the tables, in comparison to multi-template methods, we have still achieved the highest performance. This outcome underscores the effectiveness of our approach in thoroughly harnessing the features from historical frames, rendering it more robust in handling variations in target appearance changes.\n\n| Method |temporal| TrackingNet| GOT-10k|Lasot|\n|:-:|:-:|:-:|:-:|:-:|\n|OSTrack-256|×|83.1, 87.8, 82.0 |71.0, 80.4, 68.2|69.1, 78.7, 75.2|\n|OSTrack-256-temporal |√|83.8, 89.0, 82.8|73.4, 83.3, 72.1 |68.5, 77.9, 74.8| \n|OSTrack-384 |×|83.9, 88.5, 83.2|73.7, 83.2, 70.8|71.1, 81.1, 77.6|\n|OSTrack-384-temporal |√|83.9, 89.4, 83.4|73.2, 83.1, 71.4|71.1 81.25, 77.8|\n\n|Method|temporal|TrackingNet| GOT-10k|Lasot|\n|:-:|:-:|:-:|:-:|:-:|\n|||AUC, PNorm, P|AO, SR0.5, SR0.75|AUC, PNorm, P| \n|OSTrack-256-temporal|√|83.8, 89.0, 82.8 |73.4, 83.3, 72.1 |68.5, 77.9, 74.8| \n|OSTrack-384-temporal|√|83.9, 89.4, 83.4 |73.2, 83.1, 71.4|71.1 81.25, 77.8|\n|SwinTrack-256|√|81.1, -, 78.4|71.3, 81.9, 64.5|67.2, -, 70.8|\n|SwinTrack-384|√|84.0, -, 82.8|72.4, 80.5, 67.8|71.3, -, 76.5|\n|Mixformer-L |√|83.9, 88.9, 83.1|-, -, -|70.1, 79.9, 76.3|\n|Mixformer-22k |√|83.1, 88.1, 81.6|70.7, 80.0, 67.8|69.2, 78.7, 74.7|\n|AiATrack|√|82.7, 87.8, 80.4|69.6, 80.0, 63.2|69.0, 79.4, 73.8|\n|STARK|√ |82.0, 86.9, -|68.8, 78.1, 64.1|67.1, 77.0, -|\n|**DefTrack-256**|√|**84.7, 89.6, 83.6**|**74.1, 84.6, 71.8**|**70.3, 82.0, 76.4**|\n|**DefTrack-384**|√|**84.9, 89.6, 84.3**|**75.6, 85.6, 72.5**|**72.5, 84.5, 78.7**|"}, {"review_id": "5WrQUCbOwr", "reviewer": "Reviewer_UhjK", "summary": "In this paper the authors present a single object tracking algorithm that flexibly explores historical information and spatial information, in the terms of deformable memory and deformable attention, respectively. The proposed algorithm was tested on five standard benchmarks and performs better or similar to state-of-the-arts.", "questions": "* What are the performances on TrackingNet/GOT-10K/LaSOT that do not distinguish the search resolutions (i.e., combination of Table 1 and Table 2)?\n* Since the token selection is across frames, is it possible sometimes all selected tokens will from very few frames? Would that hurt the tracking performance?\n\n\nI have read the author’s rebuttal and I raised my final rating accordingly.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The overall architecture of the proposed solution is solid and well designed.\n* Adaptively selecting templates from past frames is an effective strategy and the proposed deformable memory solution seems a novel solution. \n* Similarly, the deformable attention for selecting token/patches across spatial-temporal dimensions is novel, and nicely addressed the template update issue.\n* The paper in the high level is well organized and presented.", "weaknesses": "* The title “deformable tracking” is misleading since it has been used for “tracking deformable objects”, such as deformable object tracking, deformable surface tracking, etc.\n* The experimental results, compared with state-of-the-arts, show only marginal improvements (e.g., around or less than 1% in many cases) or even worse (e.g., on LaSOT and UAV123). While it is argued that a smaller backbone (ViT-B) is used, it would be more convincing to provide results using larger ones (e.g., ViT-L).\n* The specific technique presentation is very confusing and painful to follow, especially the notations such as z^p, p^z (note that p is a row vector), etc. \n* (Minor issues) Typos: L292, “?.”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to express our gratitude for your support and valuable insights. It's reassuring to know that you recognize the originality and efficacy of our research. We would greatly appreciate your continued support in championing our work.\n#### **Q1.The title “deformble tracking” is misleading...**\nThank you very much for your suggestions. We will proceed to modify the titles to avoid ambiguity in the future.\n#### **Q2.The experimental results, compared with the state-of-the-arts, show only marginal improvements....it would be more convincing to provide results using larger ones (e.g., ViT-L).**\n\n**GOT-10k, TrackingNet, LaSOT** Among all datasets, GOT-10k, LaSOT, and TrackingNet are significant datasets in the field of tracking and are considered large-scale datasets. Using ViT-B16, we achieved a 2.8% performance improvement over Swintrack on AO and a 2.7% improvement on SR0.5 in the GOT-10k dataset. Furthermore, we observed a 3.6% enhancement over Ostrack-256 on SR0.75. On the TrackingNet dataset, our method outperformed SimTrack-large by 1.3% in AUC, and demonstrated gains of 1.8% and 1.6% in Pnorm and P, respectively, when compared to Ostrack-256. In the LaSOT dataset, our approach exhibited a 2.3% improvement in Pnorm over SimTrack, and a 1.2% improvement over Swintrack.\n\n**ViT-base-384**.In response to your suggestion to enhance the persuasiveness of our study, we proceeded to train the  ViT-B-384 model. Our model has achieved further improvements on ViT-B-384, attaining state-of-the-art (SOTA) performance. **Thanks for your advice again, after adding ViT-B-384, we significantly surpass the previous method on GOT-10k, TrackingNet, and LaSOT**.\n|Method|temporal|TrackingNet| GOT-10k|Lasot|\n|:-:|:-:|:-:|:-:|:-:|\n|||AUC, PNorm, P|AO, SR0.5, SR0.75|AUC, PNorm, P| \n|OSTrack-256-temporal|√|83.8, 89.0, 82.8 |73.4, 83.3, 72.1 |68.5, 77.9, 74.8| \n|OSTrack-384-temporal|√|83.9, 89.4, 83.4 |73.2, 83.1, 71.4|71.1 81.25, 77.8|\n|SwinTrack-256|√|81.1, -, 78.4|71.3, 81.9, 64.5|67.2, -, 70.8|\n|SwinTrack-384|√|84.0, -, 82.8|72.4, 80.5, 67.8|71.3, -, 76.5|\n|Mixformer-L |√|83.9, 88.9, 83.1|-, -, -|70.1, 79.9, 76.3|\n|Mixformer-22k |√|83.1, 88.1, 81.6|70.7, 80.0, 67.8|69.2, 78.7, 74.7|\n|AiATrack|√|82.7, 87.8, 80.4|69.6, 80.0, 63.2|69.0, 79.4, 73.8|\n|STARK|√ |82.0, 86.9, -|68.8, 78.1, 64.1|67.1, 77.0, -|\n|SimTrack|√|83.4, 87.4, -|69.8, 78.8, 66.0|70.5, 79.7, -|\n|**DefTrack-256**|√|**84.7, 89.6, 83.6**|**74.1, 84.6, 71.8**|**70.3, 82.0, 76.4**|\n|**DefTrack-384**|√|**84.9, 89.6, 84.3**|**75.6, 85.6, 72.5**|**72.5, 84.5, 78.7**|\n\n**The newly introduced datasets: TNL2k, AvisT and NfS** Additionally, we introduced three new tracking datasets. Observing the experimental results, we significant surpass previous method by 3 to 7 % on AUC. we achieved state-of-the-art (SOTA) performance on seven out of eight datasets, providing substantiated evidence for the effectiveness of our exploration into historical frame information.\n|Method|temporal|Avist| TNL2k|NfS|\n|:-:|:-:|:-:|:-:|:-:|\n|||AUC, OP50, OP75|AUC, P|AUC| \n|SiamRPN++|x|39.0, 43.5, 21.2|41.3, 41.2|5| 50.2|\n|Ocean|x|38.9, 43.6, 20.5|38.4, 37.7|49.4|\n|OSTrack-256|x|54.2, 63.2, 42.2|54.3, 54.3|64.7|\n|OSTrack-384|x|57.7, 67.3, 48.3|55.9, 56.7|66.5\n|Mixformer|√|56.5, 66.3, 45.1|55.3, 55.3|64.9|\n|**DefTrack-256**|√|**59.0, 68.1, 49.2**|**61.3, 62.3**|**66.6**|\n|**DefTrack-384**|√|**62.2, 71.9, 53.5**|**62.4, 63.4**|**68.6**|\n#### **Q3.The specific technique presentation is .....Typos: L292,**\nWe apologize for any confusion. The notation z^p represents the embedding obtained from the ViT after passing through convolutional layers, encompassing various tokens p^z. We will proceed to make adjustments to ensure clearer expression and rectify any typographical errors.\n#### **Q4.What are the performances on TrackingNet/GOT-10K/LaSOT that do not distinguish the search resolutions (i.e., combination of Table 1 and Table 2)?**\nTypically, higher resolutions lead to improved model performance, but larger resolutions also entail greater computational demands. To address this, we introduced results for the ViT-B-384 model, which exhibits enhanced performance, as indicated in the results.\n#### **Q5.Since the token selection is across frames, is it possible sometimes all selected tokens will from very few frames? Would that hurt the tracking performance?**\nWe have visualized the updates to our memory, and visualization results of the deformable memory update process can be found in the supplementary materials submitted with our initial manuscript. From the visualization, interestingly, our deformable memory aggregates discriminative features from various historical frames, rather than relying solely on a small subset of frames.\n"}]

Author rebuttal:
We sincerely appreciate the thorough review provided by all the reviewers. The valuable feedback from the reviewers has significantly contributed to enhancing the quality of our manuscript. We extend our gratitude to Reviewer **jJzj** and Reviewer **UhjK** for acknowledging the novelty of our work. Their positive recognition of the innovation in our research is greatly appreciated. Furthermore, we kindly request Reviewer **vhAu** and Reviewer **nyT7** to reconsider our work after reviewing our response. Your reconsideration will be highly valued.

Based on the comments from the reviewers, I have summarized the strengths of our paper as follows:

1. Strong intuition to make use of historical features to enhance tracking performance.

2. The design of an efficient online update in the deformable memory, which involves effective token discarding, is a noteworthy contribution

3. The advantages of the approach are convincingly demonstrated (sota performance with significantly higher FPS compared to strongest baselines.

4. The paper presents a method to dynamically select tokens from template memory that are used to interact with tokens from the search region. It can effectively select the most relevant tokens and discard irrelevant tokens, therefore improving tracking performance. The proposed method achieves very competitive performance compared with state-of-the-art methods. Also, an ablation study is conducted to validate the effectiveness of different components in the proposed method. The proposed method runs fast at 71 FPS.

We have summarized our novelty as follows:

1. Adapting to variations in target appearance and background has consistently been the most fundamental and challenging aspect of object tracking, yet it has remained inadequately addressed. To tackle this issue, we propose the utilization of deformable attention to establish a robust memory that spans across frames, enabling global context awareness. Through token-level updates, our deformable memory encapsulates the most discriminative features from the entire video sequence, consequently enhancing tracking performance. The maintenance of token-level features contributes to reduced computational load, enabling our tracker to achieve a remarkable speed of 71 FPS.

2. We devised a deformable attention mechanism to filter out features irrelevant to the current temporal search region, enabling the selection of the most pertinent features for tracking at the present time step. To the best of our knowledge, we are the first to introduce this concept in VOT.

3. Our tracker demonstrates high performance across eight datasets, including the newly added Avist, TNL2K, and NFS. These datasets encompass GOT-10k, TrackingNet, LaSOT, AVisT, OTB, NFS, TNL2K, and UAV. Furthermore, our tracker achieves a notable speed of 71 FPS.

We hold the belief that these innovative contributions elevate the value and significance of our research in the realm of visual object tracking. We kindly request the reviewers to reassess our work considering these aspects and extend their support to our endeavor.

We intend to heed the insightful suggestions from the reviewers by incorporating additional essential experiments.  Additionally, a thorough review of the manuscript will be conducted to rectify any typographical and grammatical errors.

We have also provided comprehensive responses to each reviewer, meticulously addressing all specific points raised. We extend our thanks for the valuable feedback provided by all reviewers, as well as the dedication of the program chair and area chair. Your support in our endeavors would be greatly appreciated. We are try our best in our commitment to address the raised concerns and refining our manuscript accordingly.

















Meta-review: The submitted paper introduces a deformable memory for the visual tracking task, leveraging historical target features. Reviewers highlighted the strong intuition, effective token discarding mechanism, and significant FPS advantage as strengths. Concerns raised include possible limited novelty, performance comparisons, and some presentation issues. The authors, in their rebuttal, have addressed these concerns, emphasizing the uniqueness of their approach for visual tracking, and their competitive performance across multiple datasets. Given the positive aspects and potential impact, with a commitment from authors to address the feedback, this paper is recommended for acceptance.
Decision: Accept (poster)