MemVLT: Vision-Language Tracking with Adaptive Memory-based Prompts
Authors: Xiaokun Feng
Xuchen Li
Shiyu Hu
Dailing Zhang
Meiqi Wu
Jing Zhang
Xiaotang Chen
Kaiqi Huang
Venue: NeurIPS 2024 poster (2024)
Keywords: Object Tracking; Visual-Language Multimodality; Adaptive Prompts
Primary area: machine_vision
TL;DR: Drawing from the Complementary Learning Systems theory, we propose a novel vision-language tracker (MemVLT), which can provide adaptive multimodal prompts for tracking guidance, achieving SOTA performance on 4 mainstream datasets..

Abstract: Vision-language tracking (VLT) enhances traditional visual object tracking by integrating language descriptions, requiring the tracker to flexibly understand complex and diverse text in addition to visual information. However, most existing vision-language trackers still overly rely on initial fixed multimodal prompts, which struggle to provide effective guidance for dynamically changing targets. Fortunately, the Complementary Learning Systems (CLS) theory suggests that the human memory system can dynamically store and utilize multimodal perceptual information, thereby adapting to new scenarios. Inspired by this, (i) we propose a Memory-based Vision-Language Tracker (MemVLT). By incorporating memory modeling to adjust static prompts, our approach can provide adaptive prompts for tracking guidance. 
(ii) Specifically, the memory storage and memory interaction modules are designed in accordance with CLS theory. These modules facilitate the storage and flexible interaction between short-term and long-term memories, generating prompts that adapt to target variations.
 (iii) Finally, we conduct extensive experiments on mainstream VLT datasets (e.g., MGIT, TNL2K, LaSOT and LaSOT$_{ext}$). Experimental results show that MemVLT achieves new state-of-the-art performance. Impressively, it achieves 69.4% AUC on the MGIT and 63.3% AUC on the TNL2K, improving the existing best result by 8.4% and 4.7%, respectively.

MemVLI Vision-Language Tracking with Adaptive Memory-based Prompts
Xiaokun Fengl.2 Xuchen Li
Shiyu Hu
Dailing Zhang 1:2
Meiqi Wu
Jing Zhang Xiaotang Chen 1,2,4 Kaiqi Huuagg.2...
School of Artificial Intelligence, University of Chinese Academy of Sciences
Institute of Automation, Chinese Academy of Sciences
Schooo of Computer Science and Technology, University of Chinese Academy of Sciences
CCenter for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences
"School of Physical and Mathematical Sciences, Nanyang Technological University (fengxiaokun2022, lixuchen2024) @ ia, shiyu.hu @ntu. edu.sg, zhangdailing20236 @ ia.ac.cn wumeiqi @ mails.ucas.ac. cn, jing saiee_hangee an, sichen,kaiqi.huun
Abstract
Vision-language tracking (VLT) enhances traditional visual object tracking by inte- grating language descriptions, requiring the tracker to flexibly understand complex and diverse text addition to visual information. However, most existing vision- language trackers still overly rely initial fixed multimodal prompts, which strug- gle to provide effective guidance for dynamically changing targets. Fortunately, the Complementary Learning Systems (CLS) theory suggests that the human mem- ory system can dynamically store and utilize multimodal perceptual information, thereby adapting new scenarios. Inspired by this, (i) propose Memory- based Vision- Lisgun-Language Tracker (MemVLT). By incorporating memory modeling adjust static prompts, approach provide adaptive prompts for tracking guidance. (ii) Specifically, the memory storage and memory interaction modules are designed in accordance with CLS theory. These modules facilitate the storage and flexible interaction between short-term and long-term memories, generating prompts that adapt target variations. (iii) Finally, conduct extensive experi- ments on mainstream VLT datasets (e.g.. MGIT, TNL2K, LaSOT and LaSOTea)) Experimental results show that Mem VLT achieves new state-of-the-ar perfor- mance. Impressively, it achieves 69 4% AUC on the MGIT and 63.3% AUC on the TNL2K, improving the existing best result by 8.4% and 4.7%, respectively. The code and models will be released at: httss//ghhub.com//iiaonnneenn
Introduction
The vision-language tracking (VLT) task (1 aims locate a user-defined object video sequence using multimodal prompts, which comprise a template patch and a language description. As an extension of traditional visual single object tracking (SOT) task [2] 3 4. VLT can harness the complementary advantages of multiple modalities. Therefore, dision-languse trackers (VLTs) have the potential to achieve more promising tracking performance, which has recently attracted widespread attention 6. 7 g
Similar SOT, VLT still adopts the one-shot setting 9.. providing prompts only the initial moment. However, these fixed prompts struggle to provide continuous reference for targets in video sequences due their inherent dynamic variability [T0]. As shown in Fig. (a), (a), the initial prompts depict gun placed on a table. However, when the target is subsequently picked up, there is a significant deviation in the target's state from given prompts. Regarding the target state different frames,
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Intil prompis
Vntal prompts
Nencortec fLong memony
Long Aorn memory
Teap pers
Ling dvarption
Smiartz
Srdarts
Temprane patth:
Search
Search
Stork-tarm
memmory
(hor--een menary
Happacappu
Figure 1: (a) Illustration of a video sequence. Given the initial prompts, we respectively plot consistency curves between prompts of two modalities and the subsequent searched target. (b) Framework of previous vision-language trackers (VLTs). They primarily obtain tracking results by matching the search image with the initial prompts based on similarity. (c) Framework of our proposed MemVLT (left) by modeling Complementary Learning Systems (CLS) Theory (right). McmVLT effectively models the storage and interaction of long-term and short-term memories, resulting in prompts that adapt to the search image.
quantitatively evaluate its consistency with the initial multimodal prompts (implementation details are introduced in Sec. B.I). It is evident that the consistency is poor most of the time, indicating that the initial prompts are ineffective as a reference.
overly rely these static prompts. As illustrated in Fig. (b). they identify the target in each frame that is most similar to the initial prompts as the tracking result. Despite achieving some success, the difficulty lies in the limited generalizability of static prompts to subsequent frames, thereby impacting tracking performance.
Despite this, most existing VLTs 12
1
14
15
16
Unlike existing VLTs, humans can adaptively locate a target, even when the target appears in form vastly different from its initial perception state [T7] Neuroscientists have long been interested in this ability and have conducted studies using the visual search task [88 [9, which is similar to the VLT. Numerous research findings reveal that human adaptive visual search capability can attributed to the sophisticated memory mechanisms 21).
The Complementary Learning Systems (CLS) theory 22 as a well-known memory model, has recently revealed the relationship between human generalization adaptability and memory (23). Generally speaking, humans continuously adjust their memory by integrating perceived information to better adapt the environment. As shown in Fig. (c), the human brain achieves memory through two complementary systems: the hippocampus and the neocortex. The hippocampus plays a primary role in short- memory, which is subsequently consolidated into long-term memory stored in the neocortex [24 (3). The interaction between short-term and long-term memories enables humans to adapt to different environments 23 B. Given the advantages of this memory mechanism, a critical question arises: How can we incorporate into the design of trackers?
To achieve that, we propose a Memory- based Visual-Language Tracker named MemVLT. As depicted in Fig. (c), the core insight of McmVLT lies in efficiently adapting the dynamic changes of the target by emulating the storage and interaction of memory information, thereby enabling effective tracking. Specifically, (i) we first develop a memory storage module to simulate the functioning of short-term and long-tern memory systems. Inspired by the system consolidation process (23], we introduce an efficient long-term memory storage method named section-top. (ii) Drawing upon the stored memory, we incorporate memory interaction module to emulate the interaction between short-term and long-term memories (25] generating adaptive visual and textual prompts. These adaptive prompts then guide subsequent tracking by integrating with the search feature. Through these modules, ML facilitates adaptive tracking by utilizing memory information.
In summary, our contributions are as follows: (i) Inspired by the CLS theory, we introduce a novel tracker named MemVLT. Leveraging memory mechanism modeling, this approach facilitates the generation of adaptive prompts to effectively guide the tracking process. (ii) In the proposed MemVLT, we incorporate memory storage and memory interaction modules. They faithfully model the storage and flexible interaction processes between short-term and long-term memories in the human brain, yielding multimodal prompts that adapt to dynamically evolving targets. (iii) We evaluate the performance of MemVL.I on the MGIT [5) TNL2K (26). LaSOT [27] and LaSOT (8 and achieve sfateoof-the-at tracking results. Notably, Mem MemVLT achieves 69.4%, AUC on MGIT and 63.3% AUC on TNL2K, improving the existing best result by 8.4% and 4.7%, respectively.

---

2 Related Work
2.1 Vision-Language Tracking
Vision-language tracking is emerging multimodal task that aims achieve tracking by utilizing a given language deseription and an initial template patch. Adhering to the principle of similarity-
T6] leverage the static given prompts as reference to identify the most similar target in the search frame. Among them, SNLT [II] introduces universal language region proposal network, which improves tracking performance through dynamic aggregation module. MMTrack [[6] designs an effective tracking pipeline that treats the VLT token generation task. While these VLT' perform well in simple scenes, they overlook the dynamic nature of targets, making it challenging to track when targets undergo significant changes.
matching [.. most existing VLTs [ [2] 13
[4
[5
To address this limitation, certain VLTs attempt utilize temporal information to obtain the dynamic reference. GTI [29] and AdaSwitcher [26) locate the target by integrating tracking and grounding results at each time step. Their reliance pre-defined threshold can lead to error accumulation. JointNLT 14 incorporates temporal information in the form a temporal query during the prediction stage. Since the length of the query tokens is much shorter than that of the static prompt tokens, indicates that static prompts still dominate the tracking process. Differing from them, we depart from human memory mechanisms utilize temporal information. Building upon initial static prompts, we inject dynamic features by incorporating memory information, which not only avoids the error accumulation but also generates prompts adapted to the dynamic changes of the target.
By treating temporal information as memory, some trackers 30 31 32 have incorporated memory mechanism modeling. Extensive previous studies confirm that memory modeling is viable approach, which is the paradigm our work follows. Distinguishing our work from existing research, to our knowledge, MemVLI is the first to apply CLS theory to tracker design. Recent research in CLS theory B3 23 underscores the importance of the interaction between long and short-term memories. Motivated by this insight, we develop the memory interaction module generate adaptive prompts. In contrast, existing approaches, such as DecoupleTNN [32], focus solely on modeling long and short-term memories but overlook the interaction process. In addition to memory interaction, propose the section-top memory storage method, which is inspired by the system consolidation process [23 (4] within the CLS theory. Compared the sliding window storage method commonly used by existing trackers [4 32 it demonstrates more effective tracking performance (see Sec. 4.3)
2.2 Prompt Learning
In the natural language processing field, prompt learning 34 refers to the automatic learning of instructions in the form of sentences, thereby enabling better task understanding. Considering the significant advantages this approach in enhancing model adaptability. recent studies have extended it language tasks [35] (36). CoOp (35] efficiently fine-tunes CLIP [37] for few-shot transfer by constructing language branch inputs using set of learnable vectors. Bahng (36) performs prompt tuning on CLIP by prompting on the vision branch. In this work, we exploit the core idea of prompt learning by introducing learnable vectors to model the memory information. These vectors are used to adjust the initial static prompts, making them adaptive the dynamic changes of the target.
Recently, several vision-only trackers [38 39 40) have attempted to model temporal information using learnable vectors. These vectors, also referred to as temporal queries, are utilized capture global information at each moment and guide the subsequent tracking process. What sets us apart is that, (i) we focus on the vision- lisoua-la multimodal scenario. not only modeling temporal queries but also emphasizing their adjustment to static prompts to comprehensively leverage temporal information.
(ii) We design novel multimodal query storage method called section-top to enhance the existing sliding window storage approach.
3 Methodology
3.1
Overview
The framework of MemVLT is depicted in Fig. 2 (a). Given two static prompts (text description and template patch) and the search image at general time step (t >> 0), the text and vision encoders first
3

---

Larguage
The gan saler
Text Enccder
5W15 Genentoe Love
SAN Genorde sireer
Templise
Tranefornar Deceder
Trandfrner +-1
Vidaa
mi
K
Prodecton
Search
Decesier
Deceder
Tracking Result
Acapthe
Search
Templose Tomeet J Visal
Testun' Plus' Aday textall Adazthe Hauo! Textual Vanaa
Adayciv prarepttooo
Figure 2: (a) Framework of our proposed Mem Given language description and template patch as references, MemVLT tracks the target in search images at time t. The input is first encoded using text and vision encoders. Subsequently, the memory interaction module processes static prompt features based on stored memory, generating adaptive prompts. After incorporating these prompts, the search features are fed into the prediction head to obtain the tracking results. Additionally, the memory storage module utilizes processed data to represent and store memory information. (b) Diagram of memory interaction. To illustrate the process of memory interaction, organize the memory information from the perspectives of long-term and short-term memories. Through the interaction between these memories, adaptive visual and textual prompts are obtained.
embed them into specific feature spaces, denoted as ff fr and fíz respectively. Concurrently, we obtain the global visual and textual semantic representation tokens (i.e., Sye and fg)) Subsequently, these encoded features, along with the memory information M obtained from the memory storage module (MSM), are fed into the memory interaction module (MIM). Through the mutual interaction of long-term and short-term memories, we obtain the latest visual and textual short-term memories (i.e.. m and m²), as well as the adaptive prompts (i.e., fav and fa)) These adaptive prompts then undergo feature fusion with the search feature, yielding the target-related feature f.. Next, ft is fed into the prediction head to obtain the tracking result and its corresponding confidence score p-c- Finally, ve and the short-term memory are used to update the MSM, providing memory information Ms for tracking the next time step (1 + 1). In the following sections, we will provide detailed introductions to each module and the dimensions of the above variables.
3.2 Input Encoder
Vision Encoder. For the visual input of the search image and template patch, we adopt the encoding paradigm of the one-stream network [4I], implementing it with the HiViT (42 39] 40). Specifically, the template patch e 33x H, xW. and search image 2, E R33 H, XW, are first projected into the
feature space and flattened to produce token embeddings. Additionally, we introduce a [CLS] toker"] to capture global semantic feature 43 This token is concatenated with the template-search tokens and then fed into transformer layers for feature modeling. Finally, obtain the search feature f E IR N₂ xD template feature ft E N. and global visual feature f E IR. ID
Text Encoder.
Since being proposed, the BETR [43 series models have been widely used for text representation. Therefore, we utilize RoBERTa 44, a classic pre-trained model, as our text encoder. Specifically, for given input sentence, we tokenize it into sequence of text tokens. The token sequences then are fed into the RoBERTa to extract the text embedding feature fi E N Additionally, we perform pooling on f to obtain the sentence-level global feature f E B1XD
3.3 Memory Interaction Module
Due to the constraints of initial static visual-textual prompts, the ff and f derived from them face challenges in offering consistent references for dynamically changing targets [9] To alleviate this limitation, MIM attempts to leverage the long-term memory M stored in MSM inject dynamic
It is worth noting that the symbols CLS and [CLS] represent two distinct concepts in this paper. The former denotes the Complementary Learning Systems theory, while the latter represents the classification token 43]

---

feature reflecting target variations into the static prompts, thereby better guiding the tracking, As shown in Fig. 2 (a), under the influecc of long-term memories (i.e., M and Mt we first integrate with the current features to acquire the latest short-term memories (i.e., mf and my). Then, through the interaction between the memory information m' and the initial prompts, we obtain adaptive prompts and fuse them with the search features f yielding target-related features f,-
Corresponding to the two modal prompts of given cues, we utilize both modalities to represent memory. Before explaining the specific construction of memory, we reiterate the short-term and long-term memory information that our model focuses on, shown in Fig. 2 (b). For the long-term memory provided by MSM, denoted as M {m²) Im i=1"
it consists of Lm short-term memory units m² = (m= mf). Here, m. E RIxD and mi E R1xD represent the i-th short-term visual and textual memory, respectively. Additionally, we also consider the initial given prompts as a form of long-term memory due to their role throughout the tracking process.
3.3.1 The Acquisition of the Short-Term Memory
To begin with, we introduce the generation of short-term memory under the influence of long-term memory. Initially, we concatenate the long-term visual memory (m²)) and textual memory {mi)/ with the current global semantic features fgo and fgg respectively. This process yields more comprehensive visual and textual memory information, H E Re(l++IxxD and H [l4=+1)xD
H = M : [g0 = [m. m Lm sgol:
H = 1-- fg = imf:...;:0 f. flll-
(2)
where denotes extracting all elements in the set, and [:] indicates concatenation along the first dimension. Following this, we introduce the memory query to associate long-term memory with the current information, enabling the generated short-term memory to consider both historical and current features simultaneously.
Considering the remarkable capability of the transformer network in feature interaction design the short-term memory generation (SMG) layer by modifying In each SMG layer, the memory query sequentially interacts with long-term memory information and corresponding modality features generate short-term memory, Taking the visual branch as an example, the computational process of the visual query d. E R2 glx is as follows:
45 we
d Norm(a: PCAld H!-1)),
(3)
S Norm(g + tcAl f2)),
(4)
4
*+1 = Norm(a FFN(C.)))
(5)
Here, (c)(+:)) denotes the cross-attention operation where the first element serves as Q and the second element serves to obtain K and V [46). Norm represents the layer normalization operation
and F FN denotes the feed-forward network. 9.t denotes the query after being processed by the k-th layer. For brevity, we omit the positional encoding.
After passing through L, stacked SMG layers, we obtain the visual short-term memory m q Similarly, we utilize the textual query of E RIxD obtain the textual short-term memory m/
g.
+ = Norm(at + Poald H;
(6)
t = Norm(q + Pc f2)), k+1 Norm(q + (FN(()))
(7)
(8)
3.3.2 The Generation and Fusion of the Adaptive Prompts
Based on the acquired short-term memory (i.e., m = (m², mf)), on one hand, we feed it into MSM to update the long-term memory (see Sec. (.4). On the other hand, we utilize it to modulate the long-term memory represented by the initial visual-textual prompts, enabling them to adapt to the
5

---

variations of the target. For the textual branch, we first utilize mi to modulate the initial textual feature fi then use the adjusted textual feature to guide the search feature fx.
fal Dre m().
(9)
fx TransDeo fat)-
(10)
Where Trans Dec denotes the vanilla transformer decoder layer [46 (see Sec. A.2 for detailed computation process). far represents the search features fused with the adaptive textual prompts fa-
For the visual branch, we adopt similar approach. As shown in Eq and Eq 12 we ultimately obtain the search feature f21, integrated with adaptive visual and textual prompts, simplified as f,. We treat f as the target-related features fed into the prediction head.
f.v = -Transned (J's m!);
(11)
f. Sa Trans (f2 Sav)
(12)
3.4 Memory Storage Module
In this section, we will introduce the construction of long-term memory M Given video sequence of length T, MIM generates short-term memory m' at each time step. Typically, due to the computational burden of integrating all short-term memories, Lm is much smaller than T'. Therefore, the key to long-term memory storage is to design a short-term memory selection mechanism.
Prior trackers, e.g., JointNLT and AQATrack, commonly use the sliding window approach to store temporal information, which be considered form of long-term memory. Specifically, when t >> I.m. short-term memories are selected from the most recent L,,, frames (i.e., from t- In to t- 1) for storage. While achieving some effectiveness, the redundancy of video frames leads redundant memories being stored, and it can only consider limited temporal range. The CLS theory refers to the process of integrating short-term memory into long-term memory as system consolidation 24 Relevant studies [33 23 indicate the long-term memory system primarily stores short-term memory conducive to generalization. We adhere to this principle propose a novel long-term memory storage method called section-top. It first uniformly divides past time intervals into L., sections and then stores the most representative short-term memory within each section.
To establish short-term memory selection criterion, we design a confidence prediction module to obtain the confidence score pf. E [0, 1] corresponding to the tracking result, which will be introduced in Sec. 3.5 Intuitively, a higher confidence score suggests that the current tracking result is more conducive for subsequent tracking and should thus be stored. The thorough storage process is provided in Sec. A.3] Compared to the sliding window, this method can consider a longer temporal range. Additionally. ly. the stored memories are not adjacent each other, which reduces redundancy.
3.5 Prediction Head and Loss
Based on the target-related search feature f E NDD the prediction head is used to predict the final bounding box b' and its corresponding confidence score pç. We employ a classic CNN-based tracking
head [1 40 First, ff is transformed into 2D spatial feature map. Subsequently, after passing through the stacked Conv-BN-ReLL. layers, we obtain classification score map P E [0, 1x,,
the size of the bounding box B E [0, 1|2XH, and the offset size O E [0, 1) 1)2xH, xW. Based on these features, we predict the centroid position and scale of the target, yielding the predicted bounding box b'. Additionally, we employ additional CNN-based branch to predict vL:
For the b², we employ the focal loss Lete [7]. L1 loss, and the generalized IoU loss Loo [48] for supervision, which are widely used in tracker design. Regarding the confidence score p'.. we first calculate the IoU value between the tracking result and the ground truth, and utilize the L2 loss between loU and Pt for supervision. The overall loss function is formulated as follows:
Tralt = >I.. Trets Vion Tion + dz, In + \ Ioz,
(13)
where \Las 1, Noo 2, de, 5 and dLa = are the regularization parameters.

---

Table 1: Comparison with state-of satte-of-the-aae on four popular benchmarks: MGIT [5], TNL2K [26].
LaSOT (27 and LaSOT nzt (28]. The best two results are highlighted in red and blue, respectively.
MGIT (Action)
TNL2K
LaSOT
LaSOT
Method
AUC
Prem
P
AUC
Prom
Promm
AUC
Prorm
Wang 50
27.7
30.4
Feng 51
25.0
34.0
27.0
50.0
56.0
Feng 52
25.0
33.0
27.0
35.0
35.0
47.8
47.6
TNL2K-11
42.0
50.0
42.0
SNLT
3.6
22.6
0.4
54.0
63.6
57.4
Li 12
44.0
52.0
45.0
53.0
56.0
VLTTT 15
46.8
60.2
31.8
54.7
71.8
55.3
67.3
80.2
71.5
48.4
59.9
54.3
TrTnsVLT 53
56.0
61.7
66.4
70.8
JointN1.T
14
61.0
78.6
44,5
56.9
73.6
58.1
60.4
69.4
63.6
TransNLT 16
57.0
75.0
57.0
60.0
63.0
DecoupleTNL.
82
56.7
56.0
71.2
MMTrack 16
58.6
59.4
70.0
75.7
49.4
59.9
QueryNLT
54
56.9
73.6
58.1
59.9
69.6
63.5
Ours
69.4
81.3
63.7
63.3
80.9
67.4
72.9
85.7
80.5
52.1
63.3
59.8
4 Experiments
4.1
Implementation Details
We RoBERTa- Base 44
vision encoder, with the token dimension D set to 512. The sizes of template patches and search images are 192 x 192 and 384 384, respectively. For the acquisition of short-term memory, both the visual and textual branches consist of three SMG layers. For the generation and fusion adaptive prompts, all Transpec operations conducted using single transformer decoder layer. Additionally, in the memory storage module, the default length for long-term memory is set to eight.
text encoder and HiViT-Base 42
39 40
We use the training splits of LaSOT (27, TNL2K (26), RefCOCOg 49 and OTB99-Lang M to train our model. Each training sample consists of a text description, along with one template patch and eight search frames from the same video sequence. Utilizing the text description and template patch as prompts, we iteratively train the model by selecting one search image at a time. We employ the AdamW to optimize the network parameters and conduct total of 200 training epochs. 20,000 image pairs are randomly sampled in each epoch. The model trained on a server with four A5000 GPUs and tested RTX-3090 GPU. The tracking speed is about 32 FPS.
4.2 Comparison with State-of-the-art
We evaluate MemVIe on four benchmarks, including MGIT 5), TNL2K 26) LaSOT 277 and LaSOT ext (28). Mem VLT is compared with existing late-of-the-art (SOTA) VLTs, which share the similar task setting and training datasets configuration to ensure a fair comparison.
MGIT. MGIT is a novel large-scale benchmark 2 specifically tailored for the VLT task. Each sequence contains complex spatio-temporal causal relationships and annotated with language descriptions at three levels of granularity: action, activity, and story (5). As shown in Tab, [ Me demonstrates superior performance compared other VLTs the action granularity. Particularly, MemVLT excels over the SOTA tracker JointNLT [14), surpassing it by 8.4%, 2.7%. and 19. 19.2% in area under the curve (AUC), normalized precision (P and precision score (P), respectively. Although JointNLT is equipped with temporal module, this temporal information does not interact with the initial prompts. other words, the initial static prompts still dominate the tracking process. These results highlight that the utilization of adaptive prompts plays crucial role complex scenarios. Additionally, our model also achieves optimal performance under the activity and story text settings (see Tab. AT.

---

Table 3: Ablation study on important model com-
Table Comparison of different long- term memory storage methods.
ponents. The best results are shown in red.
#
Visionadap
Textadap
AUC
Noor
P
Method
AUC
Preem
P
59.0
77.6
62.4
sliding window
62.3
79.8
66.6
2
62.3
66.2
top-L
62.5
80.0
66.8
3
62.4
79.9
66.5
section-L
62.8
80.7
66.9
4
/
63.3
80.9
67.4
section-top
63.3
80.9
67.4
TNI.2K. TNL2K designed for the VLT task, and the introduction of attributes such as "adversarial samples", "modality switch" and "cross-camera" significantly adds to the challenges [26]. In Tab. our proposed framework demonstrates superior performance compared to existing VLTs. When compared with SOTA tracker MMTrack [16]. which doesn't utilize the temporal information, approach achieves gains of 4. 4.7%, 5.7%, and 8.0% in AUC, PNom. and respectively. These results underscore the effectiveness of leveraging memory information for providing adaptive prompts.
LaSOT and LaSOT LaSOT and LaSOT est are extensions of traditional long-term visual tracking benchmarks [27 55 by adding lan-
Table 2: Results of efficiency analysis.
guage annotations. In addition to the challenges
Model
Params
Speed
AUC
P
long-term tracking, sequences in LaSOT curt also include many similar distractors. further complicating the tracking task. Nevertheless, our method still achieves outstanding perfor- mance, as demonstrated in Tab. T For instance, model outperforms the SOTA algorithms in LaSOT and eatOTent by 1.7% and 2.7% in terms of AUC, respectively. The strong performance across multiple benchmarks also reflects the generalization capability of Mem to diverse video environments and corresponding linguistic annotation styles.
JointNLT
153M
31FPS
56.9
58.1
MMTrack
177M
37FPS
58.6
58.1
MemVLT
175M
32FPS
63.3
67.4
Efficiency analysis. In Tab. we compare Mem VLT with the two latest VLTI' in terms of efficiency (Params and Speed) and performance (AUC and on TNL2K). Compared JointNLT 14, which only introduces temporal features in the decoder phase, MemVLT fully leverages memory information during the feature fusion stage. Despite the increased number of parameters, the model performance is significantly improved without compromising tracking speed. As for MMTrack [16), the absence of temporal modeling accelerates its forward process but limits tracking performance. These results underscore the trade-off between efficiency and performance achieved by MemVLT.
4.3 Ablation Study
In this section, we conduct comprehensive ablation studies on the TNL2K 26 benchmark.
Study on important model components. The core insight of our proposed model lies in the modeling and utilization of memory information, which adjusts initial static visual-textual prompts to adapt to the dynamic changes of the target. To investigate the effect of different model components, we conduct ablation analyses based whether the incorporation of memory information utilized to generate the adaptive visual or textual prompts, corresponding to the Visionadap and Textadap entries in Tab. 3 Tab. (#1) presents the performance without utilizing the memory information. Comparing with the results in Tab. 3 (#2, #3), we observe that utilizing memory information in any single modality can enhance performance. For example, adaptive visual or textual prompts lead 3.3% and 3.4% increase in AUC, respectively. Additionally, Tab. 3 (#4) further demonstrates that, under the joint influence of adaptive visual-textual prompts, the model achieves optimal performance.
Study on memory interaction. The MIM plays a crucial role in the forward process, and we conduct ablation analysis specific model designs. Tab. (#2) indicates that global visual and textual tokens (i.e.. fg and ft) are excluded from the construction of memory information. Comparing with the baseline (Tab. 1)) it is evident that the inclusion of global tokens enhances tracking performance. Tab. (#3) shows that directly integrating search features with short-temi terr-ttrmm memories significantly

---

Table Ablation study on memory interaction.
Table Generalizability analysis on SOT.
Setting
AUC
PNom
P
Setting
AUC
Prorm
1
baseline
63.3
80.9
67.4
Naive
57.4
74,9
60.6
2
remove global tokens
62.1
79.2
63.9
2
+ MIM
58.3
76.1
61.0
3
remove adaptive prompts
60.9
79.0
65.2
MSM
59.6
76.8
62.3
reduces the model's performance. This indicates the importance of the udaptive prompts generation
process.
Study on long-term memory lengths. we employ a buffer M with a length of Im to represent long- term memory. Here, analyze the model's performance under different buffer lengths. As shown in Tab. 7 we observe that as the buffer length increases, the model's performance initially rises and then stabilizes. Notably, the model achieves optimal performance when the buffer length is 8.
Study long-term memory storage method. For proposed section-top method, we compare it with other storage methods. As shown in Tab, 4 sliding window is commonly adapted by re- cent trackers involving temporal saving [4
Table 7: Ablation study on long- term memory lengths.
31 32]; top-I. prioritizes storing the top L short-term memories with the highest confidence; section-L. divides historical memory into sections, with each section storing the latest short-term memory. It can be observed that the performance of top-L is superior to sliding win- dow, highlighting the significance of storing important short-term memories. Additionally, section-L outperforms lop-L, indicating the benefits of considering longer temporal range. By combin- ing the advantages of section-L and top-L, section-top achieves optimal performance.
30
Length
AUC
NNom
62.1
79.2
65.9
2
62.5
79.9
66.5
4
62.8
80.2
66.9
6
63.1
80.7
67.5
8
63.3
80.9
67.4
10
63.1
80.8
67.5
Generalizability of our memory mechanism. Given that our proposed memory mechanism modeling significantly enhances the performance of vision-language trackers, natural question arises: can this memory mechanism generalize to vision-only taskers? To explore this, is necessary to conduct relevant experimental analysis. Specifically,
remove the text-related components from MemVLT, seamlessly converting it into a vision-only tracker. As shown in Tab, 6 our proposed Memory Interaction Module (MIM) and Memory Storage Module (MSM) progressively improve the performance of the dision-only tracker, demonstrating the strong generalizability of our method.
For more detailed implementation specifics and further comparative analysis, please refer to Sec. and Sec. C
m
1000
0.-
Confidence pr
IoU
(8)
(b)
(c)
Framm Indwa
Figure 3: (a)-(c): Hcatmaps obtained during the forward process of MLT,LT, integrating various adaptive prompts. (d): Heatmap guided solely by the initial fixed prompts. The above process diagrams depict the types and sequence of feature integration. We also illustrate the tracked result bbox and groundtruth bbox. Better viewed with zoom-in.
Figure 4: Comparison between fidence prediction and IoU values (taking the "advSamp INF sequence as an Example).
4.4 Qualitative Analysis.

---

Fig. 3 (c)-(c) depict the variations of
heatmaps over the search region dur- ing the forward process. Through the integration of adaptive prompts, the model ultimately directs its focus to- wards the tracking target. Compared to merely fusing initial fixed multi- modal prompts (Fig. (d)), Mem VLT demonstrates improved capability in
RE61
6470
Language Besodption gei as
larguage
"sho whie skiri woidd
scenarios where the target state devi- ates from the given static prompts.
Taking the "advSamp__ SI INF_ INF_bus6" se- quence from TNL2K as exam-
Tempptat
Figure 5: Qualitative comparison results of our tracker with other two latest VLTS (i.e., JointNIT [44 and MMTrack (б6) on three challenging sequences from TNL2K [26) benchmark. The first column indicates the provided initial template information. Better viewed in color with zoom-in.
ple, we showcase the model's con- fidence prediction scores against the actual IoU values in Fig. 4 can be observed that their variations align closely, indicating the effectiveness of our confidence prediction module.
This facilitates the utilization of the confidence prediction score as criteria for selecting short-term memory.
As shown in Fig. 5 we visualize the tracking results of our model and the previous two SOTA models on three challenging sequences from TNL22 [26]. In these sequences, the scenes contain distractors, and the state of the target undergoes significant changes. is evident that our model exhibits greater robustness (56) compared to others. This validates that our adaptive prompts contribute to addressing these challenges, further demonstrating the efficacy of our proposed model.
5 Conclusion
We propose novel vision-language tracker, MemVLT, which models memory mechanisms to provide adaptive multimodal prompts for tracking guidance. Drawing from the Complementary Learning Systems theory, we emphasize the importance of storing and interacting between short-term and long- term memories for generalized adaptation. Therefore, we incorporate memory storage and memory interaction modules. By comprehensively leveraging memory information to generate adaptive prompts, MemVLT provides consistent references for dynamically changing targets, thus achieving effective tracking performance. Extensive experiments demonstrate that our method achieves new state-of-the-ane performance on four widely used benchmarks, showcasing its generalization ability across various video environments and linguistic annotation styles.
Limitations. Our proposed method leverages learnable queries implicitly construct the relation- ship between historical target information and the current situation, resulting in short-term memory representations of the target. The notable results obtained demonstrate the effectiveness of this memory modeling mechanism. However, these memory representations lack explicit supervision, which diminishes their interpretability. To address this limitation, we believe that incorporating tracking result data (such as target and background information at various time steps) into the memory representation could be beneficial. This integration could enhance the comprehensiveness of the memory representations and facilitate the design of diverse loss functions for explicit supervision. We consider this approach a promising direction for our future research.
Acknowledgments and Disclosure of Funding
This work is jointly supported by the National Science and Technology Major Project (No. 20222020001 (67700011443),, the National Natural Science Foundation of China (No.62176255) and the Strategic Priority Research Program of Chinese Academy of Sciences (No (No.XDA27010201),
10

---

References
[1] Zhenyang Li, Ran Tao, Efstratios Gavves, Cees GM Snoek, and Arnold WM Smeulders. Tracking by natural language specification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6495-6503, 2017.
[2] Shiyu Hu, Xin Zhao, Lianghua Huang, and Kaiqi Huang. Global instance tracking: Locating target more like humans. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1):576-592, 2023.
[3] Shiyu Hu, Xin Zhao, and Kaiqi Huang. Sotverse: A user-defined task space of single object tracking. International Journal of Computer Vision, 132:872-930, 2024.
[4] Xin Zhao, Shiyu Hu, Yipei Wang, Jing Zhang, Yimin Hu, Rongshuai Liu, Haibin Ling, Yin Li, Renshu Li, Kun Liu, and Jiadong Li. Biodrone: A bionic drone-based single object tracking benchmark for robust vision. International Journal of Computer Vision, 132: 13511659-1684, 2024.
[5] Shiyu Hu, Dailing Zhang, Meiqi Wu, Xiaokun Feng, Xuchen Li, Xin Zhao, and Kaiqi Huang. A multi-modal global instance tracking benchmark (mgit): Better locating target in complex spatio-temporal and causal relationship. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
[6] Xuchen Li, Xiaokun Feng, Shiyu Hu, Meiqi Wu, Dailing Zhang, Jing Zhang, and Kaiqi Huang. Dill-vit: Diverse text generation for visual language tracking based on Ilm. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pages 7283-7292, 2024.
[7] Xuchen Li, Shiyu Hu, Xiaokun Feng, Dailing Zhang. Meiqi Wu, Jing Zhang, and Kaiqi Huang. Dtvlt: A multi-modal diverse text benchmark for visual language tracking based on 1lm. arXiv preprint arXiv:2410.02450
2024.
[8] Xuchen Li, Shiyu Hu, Xiaokun Feng. Dailing Zhang, Meiqi Wu, Jing Zhang, and Kaiqi Huang, Visual language tracking with multi-modal interaction: A robust benchmark. arXiv preprint arXiv:2409.0 08887, 2024.
[9] Alper Yilmaz, Omar Javed, and Mubarak Shah. Object tracking: A survey. Acm computing surveys (CSUR), 38(4): 2006.
[10] Pengyu Zhang, Dong Wang, and Huchuan Lu, Multi-modal visual tracking: Review and experimental comparison. Computational Visual Media, 10(2):19321144 2024.
[11) Qi Feng, Vitaly Ablavsky, Qinxun Bai, and Stan Sclaroff. Siamese natural language tracker: Tracking by natural language descriptions with siamese trackers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5847-5856, 2021.
[12] Yihao Li, Jun Yu, Zhongpeng Cai, and Yuwen Pan. Cross-modal target retrieval for tracking by natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages -9400, 2022.
[13] Mingzhe Guo, Zhipeng Zhang, Heng Fan, and Liping Jing. Divert more attention to vision- language tracking. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Proceedings of the Advances in Neural Information Processing Systems, volume 35, pages 4446-4466 2022.
[14] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 23151-23160, 2023.
[15] Rong Wang. Zongheng Tang, Qianli Zhou, Xiaoqian Liu, Tianrui Hui, Quange Tan, and Si Liu. Unified transformer with isomorphic branches for natural language tracking. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
[16] Yaozong Zheng, Bineng Zhong, Qihua Liang, Guorong Li, Rongrong Ji, and Xianxian Li. Towards unified token learning for vision-language tracking. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
11

---

[17] Daniel L Schacter, Donna Rose Addis, and Randy L Buckner. Remembering the past to imagine the future: the prospective brain. Nature reviews neuroscience. 8(9):657-661, 2007.
[18] John Duncan and Glyn Humphreys. Beyond the search surface: visual search and attentional engagement. Journal of Experimental Psychology: Human Perception and Performance, 1992.
[19] Timothy J Vickery, Li-Wer King, and Yuhong Jiang. Setting up the target template in visual search. Journal of vision, 5(1):8 8, 2005.
[20] Christian NL Olivers, Judith Peters, Roos Houtkamp, and Pieter R Roelfsema. Different states in visual working memory: When guides attention and when does not. Trends in cognitive sciences, 15(7):32-333,,22 2011.
[21] Geoffrey F Woodman, Steven J Luck, and Jeffrey D Schall. The role of working memory representations in the control of attention. Cerebral cortex, 17(suppl ():1111-111221 18-14 2007.
[22] James L McClelland, Bruce L McNaughton, and Randall C O'Reillyy Why there comple- mentary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
[23] Weinan Sun, Madhu Advani, Nelson Spruston, Andrew Saxe, and James E Fitzgerald. Orga- nizing memories for generalization in complementary learning systems. Nature neuroscience, 2688):4438-144 2023.
[24] Pablo Alvarez and Larry R Squire. Memory consolidation and the medial temporal lobe: a simple network model. Proceedings of the national academy of sciences, 91(15):7041-7045 1994.
[25] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation. 98)::1335-178)) 1997.
[26] Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang. Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13763-13773, 2021.
[27] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pages 5369-5378, 2019.
[28] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juchuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129:439 461, 2021.
[29] Zhengyuan Yang, Tushar Kumar, Tianlang Chen, Jingsong Su, and Jiebo Luo. Grounding- tracking-integration. IEEE Transactions on Circuits and Systems for Video Technology, 31(9):34333444 2021.
[30] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg, Atom: Accurate tracking by overlap maximization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4660-4669, 2019.
[31] Zhihong Fu, Qingjie Liu, Zehua Fu, and Yunhong Wang. Stmtrack: Template-free visual tracking with space-time memory networks. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 13774-13783, 2021.
[32] Ding Ma and Xiangqian Wu. Tracking by natural language specification with long short-term context decoupling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[33] Dharshan Kumaran, Demis Hassabis, and James L. McClelland. What learning systems do intelligent agents need? complementary learning systems theory updated, Trends cognitive sciences, 20(7): 2016.
12

---

[34] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 2021.
[35] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasion-language models. International Journal of Computer Vision, 130(9):2337-2348, 2022.
[36] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modifying pixel space to adapt pre-trained models. arXiv preprint arXiv:2203.17274. 3:11-12, 2022.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8768 PMLR, 2021.
[38] Yaozong Zheng, Bineng Zhong, Qihua Liang, Zhiyi Mo, Shengping Zhang, and Xianxian Li. Odtrack: Online dense temporal token learning for visual tracking. arXiv preprint arXiv:2401.0166 2024.
[39] Liangtao Shi, Bineng Zhong, Qihua Liang, Ning Li, Shengping Zhang, and Xianxian Li. Explicit visual prompts for visual object tracking. arXiv preprint arXiv:2401.03144 2024.
[40] Jinxia Xie, Bineng Zhong, Zhiyi Mo, Shengping Zhang, Liangtao Shi, Shuxiang Song, and Rongrong Ji. Autoregressive queries for adaptive tracking with spatic-temppraalaanssommee.. arXiv preprint arXiv: 10574, 2024.
[41] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one- framework. In Proceedings of the European Conference on Computer Vision, pages 341-351 2022.
[42] Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang, Qi Dai, Qixiang Ye, and Qi Tian. Hivit: A simpler and more efficient design of hierarchical vision transformer. In The Eleventh International Conference on Learning Representations, 2022.
[43] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv:1810.04 04805, 2018.
[44] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint
2019.
[45] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
[46] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proceedings of the Advances in Neural Information Processing Systems, 30, 2017.
[47] Hei Law and Jia Deng. Comermet: Detecting objects as paired keypoints. In Proceedings of the European conference on computer vision (ECCV), pages 734-750, 2018.
[48] Hamid Rezatofighi, Nathan Tsoi, Jun JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. In Proceedings of the IEEF/CVF conference on computer vision and patlern recognition, pages 2019.
[49] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 11-20, 2016.
13

---

[50] Xiao Wang, Chenglong Li, Rui Yang, Tianzhu Zhang, Jin Tang, and Bin Luo. Describe and attend track: Learning natural language guided structural representation and visual attention for object tracking. arXiv preprint arXiv:
2018.
[51] Qi Feng, Vitaly Ablavsky, Qinxun Bai, and Stan Sclaroff. Robust visual object tracking with natural language region proposal network. arXiv preprint arXiv: ptt.: 02044,, 1()):8, 2019.
[52] Qi Feng, Vitaly Ablavsky, Qinxun Bai, Guorong Li, and Stan Sclaroff. Real-tims visual object tracking with natural language description. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 700-709, 2020.
[53] Haojie Zhao, Xiao Wang, Dong Wang, Huchuan Lu, and Xiang Ruan. Transformer vision- language tracking via proxy token guided cross-modal fusion. Pattern Recognition Letters, 168: 10-16, 2023.
[54] Yanyan Shao, Shuting He, Qi Ye, Yuchao Feng, Wenhan Luo, and Jiming Chen. Context-aware integration of language and visual references for natural language tracking. arXiv preprint arXiv:2403. /9975, 2024.
[55] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang. Object tracking benchmark. IEEE Transactions on Puttern Analysis and Machine Intelligence, 37(09):1834-1845 2015.
[56] Honghao Chen, Yurong Zhang, Xiaokun Feng, Xiangxiang Chu, and Kaiqi Huang. Revealing the dark secrets of extremely large kernel convnets on robustness. arXiv preprint arXiv:2407.0897 087.77 2024.
[57] Yutao Cui, Cheng Jiang, Limin Wang. and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13608-13611 2022.
(58) Mingzhe Guo, Zhipeng Zhang, Heng Fan, Liping Jing, Yilin Lyu, Bing Li, and Weiming Hu. Learning target-aware representation for visual tracking via informative interactions. arXiv preprint arXiv:2201.025520 2022.
[59] Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang, and Junsong Yuan. Aiatrack: Attention in attention for transformer visual tracking. In European Conference on Computer Vision, pages 146 164. Springer, 2022.
[60] Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, and Wanli Ouyang. Backbone all your need: simplified architecture for visual object tracking. In European Conference Computer Vision, pages 375-392. Springer, 2022.
[61] Shenyuan Gao, Chunluan Zhou, and Jun Zhang. Generalized relation modeling for transformer tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18686-18695, 2023.
[62] Xin Chen, Houwen Peng, Dong Wang, Huchuan Lu, and Han Hu. Seqtrack: Sequence to sequence learning for visual object tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14572-14581, 2023.
[63] Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, and Yihong Gong. Autoregressive visual tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 9697-9706, 2023.
[64] Xin Li, Yuqing Huang, Zhenyu He. Yaowei Wang, Huchuan Lu, and Ming- Ming-Hsuan Yang. Citetracker: Correlating image and text for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 9974-9983. 2023.
[65] Qiangqiang Wu, Tianyu Yang, Ziquan Liu, Baoyuan Wu, Ying Shan, and Antoni B Chan. Dropmae: Masked autoencoders with spatial-attention dropout for tracking tasks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14561-14577 2023.
14

---

[66] Yidong Cai, Jie Liu, Jie Tang, and Gangshan Wu. Robust object modeling for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9589-9600, 2023.
[67] Dawei Yang, Jianfeng He, Yinchao Ma, Qianjin Yu, and Tianzhu Zhang. Foreground- background distribution modeling transformer for visual object tracking. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10117-10127, 2023.
[68] Yifan Bai, Zeyang Zhao, Yihong Gong, and Xing Wei. Artrackv2: Prompting autoregressive tracker where to look and how to describe. arXiv preprint arXiv:23/2.171333 2023.
[69] Wenrui Cai, Qingjie Liu, and Yunhong Wang. Learning historical status prompt for accurate and robust visual tracking. arXiv preprint arXiv:2311.02077 2023.
[70] Yuqing Huang, Xin Li, Zikun Zhou, Yaowei Wang, Zhenyu He, and Ming-Hsuan Yang. Rtracker: Recoverable tracking via pn tree structured memory. arXiv preprint arXiv:2403. 19242, 2024.
[71] Lingyi Hong, Shilin Yan, Renrui Zhang, Wanyun Li, Xinyu Zhou, Pinxue Guo, Kaixun Jiang, Yiting Chen, Jinglun Li, Zhaoyu Chen, et al. Onetracker: Unifying visual object tracking with foundation models and efficient tuming. arXiv preprint arXiv:2403.090 0066499344 2024.
[72] Huanlong Zhang, Jingehao Wang, Jianwei Zhang, Tianzhu Zhang, and Bineng Zhong. One- stream vision-languale memory network for object tracking. IEEE Transactions on Multimedia, 16:1720- 2023.
[73] Chunhui Zhang, Xin Sun, Yiqian Yang. Li Liu, Qiong Liu, Xi Zhou, and Yanfeng Wang. All in one: Exploring unified rision-languee tracking with multi- alignment. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5552-5561. 2023.
15

---

Appendices
A
More Details the Mem VLT Model
In the following sections, we will provide comprehensive details on the implementation of Mem VLT.
A.1 Input Encoder
Vision Encoder. As discussed in Sec. 3.2 vision encoder is based HiVit 442 which is constructed by stacking a series of transformer network layers. To obtain global semantic features, introduce the [CI.S] token [X] in the last two layers. Specifically, the initial template and search image tokens are concatenated and jointly modeled, aligning with the existing one-stream network architecture [41]. After obtaining the initial encoded template-search features, we introduce the [CLS] token. The [CLS] token is then concatenated with the obtained template-search features. This concatenated feature is fed into the last two layers for further feature extraction and relationship modeling. Finally, we obtain the search feature fr E R N.X template feature f E R Nxx and the global visual feature fo E RI
Text Encoder. We employ the RoBERTa 44 model as our text encoder. Existing VLTS typically set maximum text truncation length, e-g., JointNIT supports maximum text length of 40. However, for datasets providing detailed descriptions of target states, such as the MGIT dataset the text length often exceeds this threshold. Simply truncating the text would diminish its utility for the tracker. Therefore, we do not set maximum text truncation length and use sinusoidal position encoding accommodate variable text lengths.
A.2 Memory Interaction Module
the Memory Interaction module mainly consists two types of networks: the short-term memory generation (SMG) layer and the transformer decoder layer. To facilitate better understanding of their specific computational processes, we illustrate their diagram in Fig. A1
As shown in Fig. -
For the transformer decoder layer [66 we illustrate the detailed computation process corresponding to Trons (as shown in Eq.
Eq.
10 Eq.
Eq. 12 as follows:
x'
N x + @cc(z,))),
(A1)
Norm(a' + FFF((=)).
(A2)
Here, +c denotes the cross-attention operation where the first element serves as Q and the second element serves to obtain K and V [46]. Norm represents the layer normalization operation and FFN denotes the feed-forward network. For brevity, we omit the positional encoding.
Compared to the standard transformer decoder layer 46 This helps reduce computational overhead.
omit the initial self-attention operation.
A.3 Memory Storage Module
For the section-top long-term memory storage method proposed in this work, the basic approach is to first uniformly divide past time intervals into Ly sections and then select the most representative short-term memory within each section to save.
To achieve this goal, we introduce two buffers, B² and B which are used to store all memory units up to time and their corresponding confidence scores, respectively. We initialize B and B empty sets. For the long-term memory information M that we aim to obtain, we initialize each memory unit with 0 IR IDD, At time t (t 1), the detailed computation process is illustrated in Algorithm
A.4 Prediction Head
The prediction head used to predict the final bounding box b' and its corresponding confidence score v/. We employ CNN-based tracking head [1 40) which is widely adopted in tracker design. Firstly, the target-related search feature f N transformed into a 2D spatial feature map. Subsequently, after passing through LA stacked Conv-BN-Rel layers, obtain classification
16

---

[+M
gg
Av (or A
91)
for (+ -/ rflill
K
X
Add Norm
Cross Cross-Attentes Attention
( [x] for [f]:f11)
K
Q
Add & Norm
Cross Attention
FFN
Add & Norm
Add & Norm
FFN
Add & Norm
TransDec
mf (or
mf)
(a) Short-term Memory Generation Layer
(b) Transformer Decoder Layer
Figure A1: (a) The diagram of the short-term memory generation (SMG) Layer. Depending on the input modality data, this module is capable of generating short-term memories for either visual or textual inputs. (b) The diagram the transformer decoder layer. primarily consists cross- attention operation and fully connected operation.
score map Pe [0. I|IxH,XW, the size of the bounding box B E [0, 1j2xHxxXX, x.,, and the offset size O E 10,1)2xH,XW,
Based on these features, calculate the tracking bounding box b9 and the confidence score pf through two branches. On one hand, the position with the highest classification score considered to be the target position, i.e., (Ia, Va) arg max().)) Pay. The final target bounding box is obtained as:
(I, y, w. h)) = (xa + 0 x.. ya), yd + O za ua), S(0, æd, S(,, Va)).
(A3)
On the other hand, employ additional branch predict p:: We first concatenate P, B, and O along the first dimension. Then, this concatenated feature is processed through stacked Conv-BN- ReLU layers. Finally, the processed features flattened and passed through softmax function produce a regression prediction value within the range [0.1].
B More Implementation Details Experimental Analysis
In the following sections, will present the implementation details of experimental analysis, including the consistency analysis between search images and prompts, as well as the implementation methods for various model variants in the ablation study.
B.1 Consistency Analysis
As shown in Fig. (a), to quantitatively analyze the consistency between multimodal prompts and the search target, we calculate and plot the consistency curves. For determining the consistency between search target and the language description, we manually evaluate whether the target state matches the language description, assigning consistency value of or 0 accordingly. For computing the consistency between the search target and the template patch, we first crop search target from the search image based on the groundtruth bounding box. Then, input both the cropped target
17

---

Algorithm Dection-Top Long-term Memory Storage Algorithm
Input: Bt-1, B M m', P at time
Output: M
I: % save the latest memory unit and its corresponding confidence score.
2: -B ap-__apped(mm') B - .pppendppn)
3: M -M
4: i S. Lm then
5: % select the latest memory to save.
6: [t = m
7:
else
8:
fm = t mod Lm, tu Lm
9:
start (ta + 1) (tm 1) + 1, Lend Istart La
10:
select most representative memory unit in each section for in fm to Lm do
11:
12:
Secm = [taan Ienal. Sece = B3 Ienal
13:
Imxx MaxIndex(Sece)
14:
[ = [mamxxx]
15:
Taart Tend + 1, Tend start l7
16:
end for
end if'
18:
return M
and the template patch into pretrained ResNet backbone network for feature encoding, Finally, the consistency value is obtained by calculating the cosine similarity between the encoded features.
B.2 Effectiveness Analysis of Different Model Components
Corresponding to Tab. we conduct ablation analyses to investigate the effect of different model components, based on whether the incorporation of memory information utilized to generate adaptive visual textual prompts. When adaptive visual prompts are not used (i.e., Vision is not selected), the MIM module neither generates short-term visual memory nor subsequent adaptive visual prompts. Instead, the search feature interacts only with the initial template feature, replacing Eq. 3 Eq. Eq. [ Eq. and Eq. I0 with Eq. A4 Additionally, to ensure a fair comparison, we set the number of network layers used Eq. A4 to be the sum of the network layers in Eq. Eq.
Eq. Eq. and Eq. 10
fat = Trans fi).
(A4)
Additionally, not selecting Visionadap means that short-term visual memory is stored in the MSM
module. Similarly, by not selecting Textadap- we can achieve this by replacing Eq. 6 Eq. Eq. and Eq. II with the following equation:
Eq. g
ft Trans f2))
(A5)
B.3 Variants of Memory Interaction Module
Corresponding to Tab. 5 we conduct ablation analyses on the relevant model designs in MIM. The "remove global tokens" setting means that the global visual and textual tokens (i.e., fū and f are excluded from the construction of memory information. Specifically, the modified expressions for H and H are as follows:
H [' [m. :.; mm Lm].
(A6)
H = (MM [m] m mm. Fmm.
(A7)
The "remove adaptive prompts" setting means directly integrating search features with short-term memories, which removes the generation of adaptive prompts and their subsequent fusion with the
18

---

Algorithm 2 Sliding Window Long-term Memory Storage Algorithm
Input: M m', Pe at time
Output: M
1: M =M-11
2: if I Im then
3: % select the latest memory to save.
4: [
5: else
6:
% sliding window process.
7: .M'1 (Lm= 1)] M2 Lm]
8: .M'Lmm m
9: end if
10: return M
Algorithm 3 Section-L Long-term Memory Storage Algorithm
Input: B--1, M m', pl at time
Output: M
1: % save the latest memory unit.
2: B
3: M =M*M - (pppen(((m))
4: i < Im then
5: % select the latest memory to save.
6:
[t = m
7: else
8:
Lm = l mod Lms La
Lm
9:
Istart (fa + 1) (fm + 1, Tend T start + td
select the latest memory unit in each section
11:
for in f. to I,m do
12: [ 1d [eend]
13: Itaat Lend + 1. Lond Istart + la
14:
end for
15:
end if
16:
return M
search features. For the visual branch, we replace Eq. and Eq. 10 with Eq. A8 Similarly, for the textual branch, we replace Eq. TT and Eq. [2 with Eq.
fa Trans Deo mt),
(A8)
J Tee m.).
(A9)
Additionally, to ensure a fair comparison, we set the number of network layers used in Eq, A8 to be the sum of the network layers in Eq. and Eq. 10 and set the number of network layers used in Eq. A9 to be the sum of the network layers in Eq. and Eq. 12
B.4 Variants of Memory Storage Module
Corresponding to Tab. we conduct ablation analyses different long-tern memory storage methods MSM. In addition to the section-top long-term memory storage method shown in Algo- rithm I we also present the implementation processes of three other storage methods in Algorithm 2 Algorithm 3 and Algorithm 4 For top-l. method, we introduce a new variable Mt which stores confidence store corresponding to each memory unit in M
19

---

Algorithm 4 Top-L Long-term Memory Storage Algorithm
Input: M M _M m P at time t
Output: M
1: M M* MMM M
2: if I Lm then
3: % select the latest memory to save.
4: [t = m
5:
Mi =
6:
else
7:
Imin MinInder(M..) Pmiin = Min(M
8:
if S Lm then
9:
Imaz Maaaaandeee (Sece) M = =m² [2[ = P'c
10:
12:
end for
13:
end if
14:
return M
MGIT Leaderboard (OPE Mooliioioiom Tiny Version Action
MGIT Lenderboard (DRE Mechanaam with Tiny Vensann Activtyy
MGIT Leade board IOPE Mechamtonn with Tiny Wersion Story
(a) Results on action granularity
(b) Results on activity granularity
(c) Results on story granularity
Figure A2: Screenshot of the Latest MGIT Leaderboard for action granularity, activity granularity,
and story granularity. Better viewed with zoom-in.
C Additional Experimental Analysis
C.1 Experimental Results of All Granularities on MGIT
MGIT is the latest large-scale benehmark specifically tailored for the VLT task, focusing on long-term challenges with complex spatio-temporal causal relationships. Each sequence is annotated with language descriptions at three levels of granularity: action, activity, and story (5). Due to space constraints, we only present the experimental results for the action granularity in Tab. We have anonymously submitted results for all three granularitics on the official testing platform. As shown in Fig. A2 our model achieves state-of-the-ar performance across nearly all metrics for all three granularities. Specifically, McmVLT surpasses the existing best results by 9%, 8%, and 3% in area under the curve for the three granularities, respectively. We expect that proposed MemVI VLT can serve as a solid baseline.
C.2 Comparison with Visual-only Trackers
In line with the prevailing paradigm of vasion-language tracking models 13 additional comparisons with visual-only trackers to comprehensively showcase the performance of model. As shown in Tab. Al organize and present the results of visual-only tracking models to supplement Sec. 4.2
14 16
we provide
TNL2K. Despite being proposed for the VLT task, TNL2K has become commonly used bench- mark for the visual tracking task (as shown in Tab. AT). The primary intention of TNI.2K to exploit the complementarity of visual and textual modalities to achieve robust tracking performance,
20

---

Table A1: Comparison with -ftte-of-the-ars three popular benchmarks: 'NL2K [26] LaSOT 227 and LaSOT [28]. The vision-only type of methods are evaluated by bounding box initialization, while the vision-language (VL) type of methods are evaluated by joint bounding box and natural language initialization. The best two results are highlighted in red and blue, respectively.
TNL2K
LaSOT
LaSOT ext
Type
Method
AUC
PNom
P
AUC
Nerm
AUC
Narm
P
Mixformer
57
69.2
78.7
74.7
TransInMo 58
52.0
58.5
52.7
65.7
76.0
70.7
OSTrack-256
54.3
69.1
78.7
75.2
47.4
57.3
53.3
OSTrack-384
55.9
71.1
81.1
77.6
50.5
61.3
57.6
AiATrack (59
69.0
79.4
73.8
47.7
55.6
55.4
SimTrack 600
69.3
78.5
GRM [6]]
69.9
79.3
75.8
ScqTrack-B226 62
54.9
69.9
79.7
76.3
49.5
60.8
56.3
ScqTTrck-B3344 62
56.4
71.5
81.1
77.8
50.5
61.6
57.5
ARTrack-256 63
57.5
70.4
79.5
76.6
46.4
56.5
52.3
ARTrack-384 63
59.8
72.6
81.7
79.1
51.9
62.0
58.5
Vision-related
CiteTracker [64
57.7
59.6
69.7
78.6
75.7
DropTrack 65
56.9
57.9
71.8
81.8
78.1
52.7
63.9
60.2
ROMTrack-256 66
69.3
78.8
75.6
48.9
59.3
55.0
ROMTrack-384
66
71.4
81.4
78.2
51.3
62.4
58.6
F-BDMTrrac-2266
67
56.4
56.5
69.9
79.4
75.8
47.9
57.9
54.0
F-BDMTTrac-384
67
57.8
59.4
72.0
81.5
77.7
50.8
61.3
57.8
EVPTrack-224 39
57.5
58.8
70.4
80.9
48.7
59.5
55.1
EVPTrack-384
59.1
62.0
72.7
82.9
80.3
53.7
65.5
61.9
ODTrack-B
38
60.9
73.2
83.2
80.6
52.4
63.9
60.1
ODTrrackLL
38
61.7
74.0
84.2
82.3
53,9
65.4
61.7
AQATrack-226
57.8
59.4
71.4
81.9
78.6
51.2
62.2
58.9
AQATrack-384
40
59.3
62.3
72.7
82.9
80.2
52.7
64.2
60.8
ARTrackV2-256 668
71.6
80.2
77.2
50.8
61.9
57.7
ARTrackV2-384
668
73.0
82.0
79.6
52.9
63.4
59.1
H1PTrack (69
72.7
82.9
53.0
64.3
60.6
RTracker-L 70
60.6
63.7
74.7
84.5
54.9
65.5
62.7
OneTTracker (71
70.5
79.9
76.5
Wang 50
27.7
30.4
Feng
25.0
34.0
27.0
50.0
56.0
Feng 122
25.0
33.0
27.0
35.0
35.0
GTT 29
47.8
47.6
Vision- Language
TNL.2-11
26
42.0
50.0
42.0
51.3
55.4
SNLT
54.0
63.6
57.4
Li 12
44.0
52.0
45.0
53.0
56.0
VLTTT
[R
54.7
71.8
55.3
67.3
80.2
71.5
48.4
59.9
54.3
TransVLT
56.0
61.7
66.4
70.8
JointNLT
14
56.9
73.6
58.1
60.4
69.4
63.6
TransNLT
151
57.0
75.0
57.0
60.0
63.0
DecoupleTNL.
332
56.7
56.0
71.2
75.3
MMTTrck
58.6
59.4
70.0
82.3
75.7
49.4
59.9
55.3
QucryNLT
56.9
73.6
58.1
59.9
69.6
63.5
Ours
63.3
80.9
67.4
72.9
85.7
80.5
52.1
63.3
59.8
especially in challenging video sequences where the target state changes drastically 266 However, recent vasion-language trackers [14] 16) have shown inferior performance compared vision-only trackers (38 70). In contrast, our model outperforms SOTA vision-language trackers and visual-only trackers. Specifically, we achieve improvements of 4.7 % in AUC over MMTrack [16] and 1.6 %
in AUC over RTracker (38]. These results highlight the importance of the various adaptive prompts (visual and textual) we incorporated in addressing complex scenarios.
LaSOT and LaSOT ext- As the mainstream benchmarks for visual tracking tasks, LaSOT and LaSOT [28 are extended the VLT task by incorporating language annotations. As shown
21

---

in Tab. A1 our proposed MemVLT outperforms existing vision vision-language trackers on these two benchmarks. Compared to SOTA tracker MMTrack [16], our model surpasses it by 2.9 and 2.7 % in area under the curve, respectively. However, we also notice that the performance of MemVLT does not surpass existing SOTA visual-only trackers [8 [0] We speculate that this may related to the quality of some language descriptions on these two benchmarks. For instance, some descriptions may be too vague to effectively locate the target [14 26.. leading to misleading guidance instead of facilitating tracking.
C.3 Comparison with All-in-one and OVLM
As emphasized in Sec. 4.2 Mem is com- pared with existing state-of-the-art VLTS that share similar task settings and training dataset configurations to ensure fair comparison. As
Table A2: Comparison with the OVLM.
Model
TNL2K
LaSOT
Average
shown Tab. | and Tab. [A] the VLTs we com- pare against are primarily trained using the La- SOT [27]. TNL2K 226,, RefCOCOg [49], and OTB99-Lang [1 datasets. In addition, two cent VLTs, namely OVLM [72) and All-in-one 773, utilize training dataset configurations that differ significantly from ours. Therefore, conduct a separate comparison with these two VLTs.
AUC
P
AUC
AUC
P
OVLM McmVLT
64.7
69.3
67.7
74.2
66.4
72.1
63.3
67.4
72.9
80.5
68.8
74.9
The comparison results are shown in Tab. A2 Our Mem significantly outperforms OVLM on LaSOT (AUC +5.2%) but slightly falls short on TNL2K (AUC -1.4 -1.4%). Overall, our model demonstrates superior average performance (AUC +2.4%). Besides, although OVLM involves memory mechanism modeling, there are key differences between our method:
OVLM.
1. Memory Representation: OVLM assumes the text prompt is precise long-term cue, using vision and text features to represent short and long-term memories, respectively. However, it overlooks the misalignment between text information and video targets (see Fig. I (a)), which can mislead the tracker [14]. Our method addresses this by using both text and vision features comprehensively represent short and long-term memories, and modulate them with dynamic information.
2. Memory Interaction: OVLM focuses solely on the unidirectional modulation of short-term memory features by textual long-term memory, whereas our method achieves bidirectional modulation between long-term and short-term memories. The importance of bidirectional interaction has been highlighted by recent research 233
In summary, our model provides more comprehensive representation and interaction of memory information, resulting superior performance.
All-in-one. As shown in Tab. A3 our tracker significantly outperforms All-in-one on all benchmarks except LaSOTezt- Overall, the average per- formance of our model far exceeds that of All-in-one (AUC +4.5%). Al- though All-in-one uses a larger train- ing dataset to align text and vision modalities, it neglects the modeling of dynamic temporal information. In contrast, MemVLT effectively models temporal information through the memory mechanism, resulting in superior tracking performance.
Table A3: Comparison with the All-in-one.
Model
TNL2K
LaSOT
LaSOT AUC
Average
AUC
AUC
P
AUC
P
All-in-one McmVLT
55.3
57.2
67.3
78.5
54.5
66.0
60.4
68.6
63.3
67.4
72.9
80.5
52.1
59.8
64.9
71.4
C.4 Effect of the Confidence Prediction Module
To predict the confidence score pi: we introduce a confidence prediction module and supervise using the L2 loss between Pe and the loU value. As shown in Tab. A4 we analyze the impact of this module on the model's performance using the TNI.2K dataset. Without confidence scores, we lack criterion for short-term memory selection. Therefore, we conduct tests using the sliding
22

---

Table A4: Study on the Effect of the Confidence Prediction Module. Results are tested on TNL2K. The best results are highlighted in red.
#
Confidence Prediction
Storage Method
AUC
PNoms
P
sliding window
62.3
79.7
66.4
2
sliding window
62.3
79.8
66.6
3
section-top
63.3
80.9
67.4
Table A5: Study on the length of visual queries. Results tested TNI.2K.
Table A6: Study on the length of textual queries. Results are tested on TNI.2K.
Length
AUC
Nerm
P
Length
AUC
Noom
1
62.78
79.94
66.79
62.78
79.94
66.79
2
62.79
80.09
66.93
2
62.75
80.67
66.92
4
62.77
80.16
66.96
62.62
80.54
66.73
window storage method (Tab. A4 (#1)). When the confidence prediction module is introduced but the sliding window method is still used (Tab. A4 (#2)), we observe no significant change in performance. However, when the section-top storage method is employed, we significant improvement in model performance (Tab. A4 (#3)). This demonstrates that our proposed long-term memory storage method effectively leverages the confidence prediction information.
C.5
The Impact of Query Length
In the process of generating short-term memories, we draw inspiration from the widely acknowledged paradigm of prompt learning [35]. which introduces Iearnable query tokens understand and represent corresponding features. Specifically, in the generation of visual and textual short-term memories mi, and mi, we incorporate corresponding query tokens Iu and qi, respectively.
As shown Tab. A5jand Tab. Ao we conduct the ablation analysis on the lengths of these query tokens. Under the setting of long-term memory length of 4, we test the influence of different lengths of visual and textual queries on the model performance. The results indicate that there no substantial variation in model performance with changes in query length. This underscores the adequacy of query length of 1. Consequently, we default to a query length of I in the experiments presented in the paper.
C.6 Ablation Study on the Backhone and Image Resolution
In Sec. 4 all our experiments are conducted using the HiViT-Baad backbone and a 384x 384 image resolution. To provide a more comprehensive understanding of our model's performance, we perform ablation studies by evaluating the model under the ViT-Base backbone and a 256 256 resolution.
In Tab. A7 we not only evaluate the model's performance using two different backbones but also analyze the impact of our proposed memory mechanism. Comparing #1 and #2, it is evident that the introduction of our memory mechanism improves the AUC by 4.3%. Comparing #1 and #3, we observe that the HiViT backbone provides an AUC improvement of 1.1 %, which is consistent with previous research findings 40 39.. Furthermore, the performance boost brought by our memory mechanism surpasses that of the HiVIT-basee backbone, indicating that the superior performance of our work primarily originates from the memory mechanism rather than the HiViT-based backbone.
In Tab. A8 evaluate the performance of our model under different search image resolutions. As expected, reducing the image resolution slightly degrades performance. However, Mem 256 still significantly outperforms several recent models with larger resolutions, such as MM'Track [66 338) and QueryNLT [54] (320 x200) (see Tab. 4.2). This demonstrates that our method maintains strong performance even lower resolutions.
C.7 Alation Study the Number of SMG Layers
23

---

Table A7: Ablation study on the backbone and our memory mechanism. Results are tested on TNL.2K.
Table A8: Ablation study on the image
resolution. Results are tested on TNL2K.
#
Setting
AUC
PNorm
P
Backbone
Memory
AUC
P NNmn
P
MemVL.T-256
63.3
80.9
HiVIT
63.3
80.9
67.4
MemVLT-384
59.0
77.6
HIVIT
59.0
77.6
62.4
VIT
62.2
79.8
66.4
In our proposed MemVLT, the SMG module, responsible for generating short-term memory. plays a critical role. To investigate its impact. we conduct ablation experiments on the number of SMG layers. Specifically, we test the performance of SMG with and layers, and compare the results with the existing model using 3 layers.
Table A9: Ablation study on the number of SMG layers. Results are tested on TNL2K.
Layers
AUC
PNrm
P
1
60.9
78.3
64.8
As shown in Tab, A9, is observed that using too few lay- ers (comparing #1 and #2) significantly harms the model's performance, while increasing the number of layers (com- paring #2 and #3) further does not provide noticeable per- formance gains. Therefore, our model effectively demon- strates performance convergence.
2
3
63.3
80.9
67.4
3
5
63.2
81.0
67.5
D Broader Impact
In this paper, we introduce novel vision-language tracker named Mem Inspired by the Comple- mentary Learning Systems (CLS) theory, this model emulates human memory systems to adapt given static prompts to dynamically changing targets, thereby achieving effective tracking. Generic object tracking is one of the fundamental problems in computer vision with numerous applications such as video surveillance, robotics, and autonomous vehicles. The vision-language tracking task extends type of human- machine interaction,
this field by incorporating text modality. Besides providing this task leverages the complementary advantages of visual and language modalities to achieve more promising tracking results. Our research improve tracking performance while maintaining reasonable running speed (32 FPS). However, of particular concern that this tracker could be misused for illegal surveillance and positioning. Despite achieving promising results, applying this technology to real-world scenarios, such as autonomous driving, remains challenging. To mitigate the risks associated with using MemVLT, we encourage researchers thoroughly understand the implications of using trackers in specific real-world scenarios.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately summarize the key contributions and scope of this paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of work are discussed in detail in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
25

---

Justification:
This paper does not include theoretical results. The effectiveness of the model is analyzed and validated through experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: provide detailed description of the proposed model Sec. andd Sec. These sections contain sufficient information to reproduce our model.
A
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The data used in this paper are open-source datasets, which can be ac- cessed through the references provided. The details for constructing the model are thor- oughly described in Sec. 3 and Sec. [ For the code and models, they will be released at: https///gihuu..ooo//// www.//iiooonommmnn//eeemmemmmmm
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.c)
p Cotpe///SSSuuumiiiiiiiic. soteesssubbiisiiooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https / ctltsps.ccpubbicc/// gwwc///gueeeee/eee SonneeSSumissiionPooe
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide a detailed description of the implementation details in Sec. 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: To the best of our knowledge, almost all tracker test results based on models that have been fully trained, typically requiring numerous iterations (e.g., several hundred epochs) to converge. Considering the high cost of single training process, the expense conducting multiple trainings for subsequent statistical analysis is prohibitive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We carefully review the NeurIPS Code of Ethics and ensure that the research aligns with it in all aspects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of this paper in Sec. D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper focuses on the video tracking task, which does not pose such high
risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit the creators original owners of all assets used in the paper, and cite the relevant papers for each asset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have not yet released the model proposed this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
30

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "EDwrtm6QFs", "reviewer": "Reviewer_BESZ", "summary": "This paper proposes a Memory-based Visual-Language Tracker based on the complementary learning system theory. Memory mechanism modeling facilitates the generation of adaptive prompts to effectively guide the tracking process. Extensive experiments demonstrate that MemVLT achieves new state-of-the-art performance.", "questions": "1) The long and short-term memory have been widely used in visual object tracking, so it is not a new technique. What's the difference for vision-language tracking?\n2) Is it general to apply the proposed memory prompts for single-modality visual tracking framework?", "limitations": "In addition to performance, tracking speed (fps) is also need to be compared and discussed for real-time applications.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) Integrating memory mechanisms with adaptive prompts is interesting for vision-language tracking. \n2) This paper is well-writing and well-organized, and includes appropriate citations and references.\n3) The experiments across various benchmarks validate MemVLT's performance compared to other trackers.", "weaknesses": "1) For experiments, recent SOTA trackers are missing in Table 1, such as MMA (CVPR 2024), CAI (CVPR 2024), OVLM (TMM23) ...\n2) OVLM is also a recent vision-language memory network for tracking, but the authors have not dicuss and compare it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer BESZ,**\n\nThanks for taking the time to review our work. We appreciate your recognition of our interesting insights, good writing, and strong performance. We hope the following responses will address your concerns.\n\n---\n\n### **Weakness 1 & 2: comparison with recent SOTA trackers**\n\n(1) **MMA [1]:** We found that MMA is not a tracker. It focuses on vision-language pre-training, with its primary downstream task being image classification. Due to the significant gap, a direct comparison is not feasible.\n\n(2) **CAI [2]:** The tracker proposed in the CAI paper is called **QueryNLT**, which we have already compared in our paper (see **Table 1**). Our model outperforms QueryNLT across all benchmarks. For instance, on the TNL2K, our model achieves a 6.4% higher in AUC. QueryNLT modulates multimodal prompts by filtering tokens based on visual-text similarity. In contrast, our modulation approach embeds dynamic information into the multimodal prompts, avoiding the potential loss of important tokens due to similarity calculation errors. We suspect this accounts for the inferior performance of QueryNLT.\n\n(3) **OVLM [3]:** The comparison results are shown in the table below. Our MemVLT significantly outperforms OVLM on LaSOT (AUC +5.2%) but slightly falls short on TNL2K (AUC -1.4%).  Overall, our model demonstrates superior average performance (AUC +2.4%). Although OVLM  involves memory mechanism modeling, there are key differences between our method (addressing weakness 2):\n\n1). **Memory Representation:** OVLM assumes the text prompt is a precise long-term cue, using vision and text features to represent short and long-term memories, respectively. However, it overlooks the misalignment between text information and video targets (see **Fig 1(a)**), which can mislead the tracker [4]. Our method addresses this by using both text and vision features to comprehensively represent short and long-term memories, and modulate them with dynamic information.\n   \n2). **Memory Interaction:** OVLM focuses solely on the unidirectional modulation of short-term memory features by textual long-term memory, whereas our method achieves bidirectional modulation between long-term and short-term memories. The importance of bidirectional interaction has been highlighted by recent research [5].\n\nIn summary, our model provides a more comprehensive representation and interaction of memory information, resulting in superior performance.\n\n| Model  | TNL2K AUC | TNL2K P | LaSOT AUC | LaSOT P | Average AUC | Average P |\n|--------|-----------|---------|-----------|---------|-------------|-----------|\n| OVLM   | 64.7      | 69.3    | 67.7      | 74.2    | 66.4        | 72.1      |\n| MemVLT | 63.3      | 67.4    | 72.9      | 80.5    | 68.8        | 74.9      |\n\n---\n\n### **Question 1: novelty in long and short-term memory modeling**\n\n(1) To highlight the differences between our method and existing trackers based on long and short-term memory, we have provided direct comparisons in **Sections 2.1** and **2.2**. Additionally, please refer to our summary of distinctions in **global response 5**.\n\n(2) For the VLT task, introducing textual cues brings new challenges. Due to the modality differences between text and vision cues, previous visual memory modeling methods may not directly apply to textual memory. For example, acquiring the memory unit in TrDiMP[6] relies on RoI processing in the search image, which cannot be applied to the text modality. Our tracker proposes a novel approach to represent textual memory features based on the prompt learning method. The table below also demonstrates the necessity of specifically designing textual memory information.\n\n| Our text memory mechanism | AUC  | P_N  | P    |\n|------------------|------|------|------|\n| ×     | 59.0 | 77.6 | 62.4 |\n| √      | 62.4 | 79.9 | 66.5 |\n\n---\n\n### **Question 2: generalizability of our memory mechanism**\n\nAs introduced in the **global response 4**, we conduct experiments under a vision-only tracking setting and find that our method can be generalized to the vision tracker. Specifically, we remove the text-related components from MemVLT, seamlessly transforming it into a vision tracker. As shown in the table below, the proposed Memory Interaction Module (MIM) and Memory Storage Module (MSM) incrementally improve the performance of the vision-only tracker. This validates that our method can generalize to a visual tracking framework.\n\n| Setting | AUC  | p_N  | P    |\n|---------|------|------|------|\n| Naive   | 57.4 | 74.9 | 60.6 |\n| +MIM    | 58.3 | 76.1 | 61.0 |\n| +MSM    | 59.6 | 76.8 | 62.3 |\n\n---\n\n### **Limitations : efficiency of MemVLT**\n\nWe have already included comparisons and analyses of efficiency in our paper (see **Table 2** and **Section 4.2**). Overall, our model is comparable to the latest trackers in terms of parameters and fps, while demonstrating superior performance. Additionally, our model achieves a speed of 32 fps, which meets the requirements for real-time processing.\n\n---\n\nWe hope this detailed response addresses your concerns. We would appreciate it if you could reconsider your rating. If you have additional feedback or require further clarification, please let us know.\n\n---\n\n[1] MMA: Multi-Modal Adapter for Vision-Language Models, Yang et al., in CVPR 2024.\n\n[2] Context-Aware Integration of Language and Visual References for Natural Language Tracking, Shao et al., in CVPR 2024.\n\n[3] One-Stream Vision-Language Memory Network for Object Tracking, Zhang et al., in TMM 2023.\n\n[4]  Joint visual grounding and tracking with\n natural language specification, Zhou et al., in CVPR 2023.\n\n[5] Organizing memories for generalization in complementary learning systems, Sun et al., In Nature 2023.\n\n[6] Transformer meets tracker: Exploiting temporal context for robust visual tracking, Wang et al., in CVPR 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qPiFoCJhOl", "reviewer": "Reviewer_GRLs", "summary": "This paper introduces MemVLT (Memory-based Vision-Language Tracker), an approach to vision-language tracking (VLT) that incorporates memory mechanisms inspired by human cognition. The key innovation is the use of adaptive memory-based prompts to guide tracking, as opposed to relying solely on initial static prompts.\nThe model consists of two main components:\n-A Memory Storage Module (MSM) that efficiently stores and manages long-term memory information.\n-A Memory Interaction Module (MIM) that generates adaptive visual and textual prompts by integrating stored memory with current input.\nThe authors evaluate MemVLT on several VLT benchmarks, demonstrating state-of-the-art performance, particularly on challenging datasets like MGIT and TNL2K.", "questions": "The main limitation of this paper lies in its limited novelty. The authors should provide a more detailed comparison based on this weakness.", "limitations": "no.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper introduces a paradigm for VLT by incorporating memory mechanisms inspired by cognitive science, specifically the Complementary Learning Systems theory.\n- Strong performance: MemVLT achieves state-of-the-art results on multiple benchmarks, showing significant improvements over existing methods.\n- Broader impact discussion: The paper includes a thoughtful discussion of potential societal impacts and limitations.", "weaknesses": "- The insights of using adaptive prompts for tracking guidance have been introduced by CiteTracker[1]. More analysis and comparison between them should be provided.\n- Short-term and long-term memory have been exploited more in the video object segmentation task like Xmem[4]. More analysis and comparison between them should be provided.\n- The authors are suggested to conduct more comparisons with the most recently published VL tracker like[2][3].\n- While the model achieves good performance, it may be more computationally intensive than simpler approaches due to the memory mechanisms.\n\n[1] Li X, Huang Y, He Z, et al. Citetracker: Correlating image and text for visual tracking[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 9974-9983.\n[2] Zhang C, Sun X, Yang Y, et al. All in one: Exploring unified vision-language tracking with multi-modal alignment[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 5552-5561.\n[3] Wu Z, Zheng J, Ren X, et al. Single-model and any-modality for video object tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 19156-19166.\n[4] Cheng H K, Schwing A G. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 640-658.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer GRLs,**\n\nWe sincerely appreciate your time and effort in reviewing our work. We are grateful for your recognition of our cognitive science-inspired modeling approach, strong performance, and broader impact discussion. We notice that you are particularly concerned about the differences between our model and other works. Therefore, we have conducted a detailed comparative analysis to highlight the novelty of our method.\n\n---\n\n### **Weakness 1: Comparison with CiteTracker**\n\n(1) **Differences in adaptive prompts:**\n\n1). CiteTracker is a vision tracker. Its adaptive prompts refer to discrete and closed-set text cues identified based on visual target information. This is primarily achieved through image attribute recognition.\n\n2). Our proposed MemVLT is designed for the vision-language tracking task. It considers textual information that is diverse and open-set in language description, making it a more challenging task. Our adaptive prompts refer to initial multimodal prompts implicitly embedded with dynamic interaction information. Unlike CiteTracker, which only models adaptive textual prompts, our approach models both textual and visual adaptive prompts, aiding in better tracking guidance. Additionally, our adaptive prompts are implicitly obtained through prompt learning, which helps mitigate the impact of classification errors caused by explicit image attribute recognition.\n\n(2) **Performance comparison:** As shown in the table below, our MemVLT significantly outperforms CiteTracker, demonstrating the effectiveness of our adaptive prompts mechanism.\n\n| Model      | TNL2K AUC | TNL2K P | LaSOT AUC | LaSOT P | Average AUC | Average P |\n|------------|-----------|---------|-----------|---------|-------------|-----------|\n| CiteTracker| 57.7      | 59.6    | 69.7      | 75.7    | 64.5        | 68.8      |\n| MemVLT     | 63.3      | 67.4    | 72.9      | 80.5    | 68.8        | 74.9      |\n\n---\n\n### **Weakness 2: Comparison with Xmem**\n\n(1) **Key differences in memory mechanism modeling:**\n\n1). Xmem uses the **Atkinson-Shiffrin memory model**, which emphasizes the representation of short-term, working, and long-term memories. A crucial modeling processing of Xmem is the unidirectional information transfer from short-term to long-term memory. In contrast, our memory model is based on the **Complementary Learning Systems (CLS) theory**, which emphasizes the interaction between short-term and long-term memories. The importance of bidirectional interaction has been highlighted in recent studies [1].\n\n2). For memory storage and updating processes, Xmem adopts a **top-k** updating method based on usage frequency as the importance indicator. In contrast, our proposed **section-top** method not only considers the importance of memory units but also accounts for the temporal representation range(see **Table 4**).\n\n(2) **Different tasks:** Xmem focuses on vision-only video object segmentation, while MemVLT concentrates on vision-language multimodal tracking. Due to the significant differences in input and output, a direct quantitative comparison is not feasible, and existing VLTs have not conducted performance comparisons with it.\n\n---\n\n### **Weakness 3: Comparison with recent SOTA trackers**\n\n(1) **All-in-one:** As shown in the table below, our tracker significantly outperforms All-in-one on all benchmarks except LaSOT_ext. Overall, the average performance of our model far exceeds that of All-in-one (AUC +4.5%). Although All-in-one uses a larger training dataset to align text and vision modalities, it neglects the modeling of dynamic temporal information. In contrast, MemVLT effectively models temporal information through the memory mechanism, resulting in superior tracking performance.\n\n| Model       | TNL2K AUC | TNL2K P | LaSOT AUC | LaSOT P | LaSOT_ext AUC | LaSOT_ext P | Average AUC | Average P |\n|-------------|-----------|---------|-----------|---------|---------------|-------------|-------------|-----------|\n| All-in-one  | 55.3      | 57.2    | 67.3      | 78.5    | 54.5          | 66.0        | 60.4        | 68.6      |\n| MemVLT      | 63.3      | 67.4    | 72.9      | 80.5    | 52.1          | 59.8        | 64.9        | 71.4      |\n\n(2) **Untrack:** In fact, Untrack is not a VL tracker. Although it claims to handle tracking in any modality, it does not include our VLT task, making direct comparison infeasible. Additionally, we found a recent tracker with a similar motivation to Untrack that includes the VLT task, called OneTracker [2], and compared its performance with ours (see table below). Evidently, our model significantly outperforms OneTracker across all metrics. We speculate that the primary reason for this is OneTracker's lack of modeling dynamic temporal information.\n\n\n| Model       | TNL2K AUC | TNL2K P | LaSOT AUC | LaSOT P | Average AUC | Average P |\n|-------------|-----------|---------|-----------|---------|-------------|-----------|\n| OneTracker  | 58.0      | 59.1    | 70.5      | 76.5    | 65.1        | 69.0      |\n| MemVLT      | 63.3      | 67.4    | 72.9      | 80.5    | 68.8        | 74.9      |\n\n---\n\n### **Weakness 4: computational efficiency of MemVLT**\n\nWe understand your concerns regarding the computational efficiency (e.g., fps, parameters). Actually, we have already included comparisons and analyses in our paper (see **Table 2** and **Section 4.2**). Overall, our model is comparable to the latest trackers in terms of parameters and fps, while demonstrating superior performance.\n\n---\n\nWe hope this detailed response addresses your concerns and would be grateful if you could reconsider your rating. Should you have any additional feedback or require further clarification, please do not hesitate to let us know.\n\n---\n\n[1] Organizing memories for generalization in complementary learning systems, Sun et al., in Nature 2023.  \n\n[2] OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuning, Hong et al., in CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbE8q7Y0v0", "reviewer": "Reviewer_unPL", "summary": "This paper presents a new visual-language tracking method, called MemVLT, which incorporates memory modeling to adjust static prompts.The design of MemVLT is inspired by the theory of Complementary Learning Systems (CLS), which adapts to changes in the target by mimicking the storage and interaction mechanisms of the human memory system.", "questions": "See Weakness", "limitations": "Experiments are not sufficient.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. MemVLT can better utilize the memory information for vision-language tracking.\n2. MemVLT achieves promising results in several benchmarks.", "weaknesses": "1. Performance comparison is unfair. MemVLT is built on HiViT-base and the input resolution is 384 instead of 256. HiViT-base can improve the tracking accuracy compared to ViT, which has been proved in previous researches. However, authors did not provide the ablation study on backbone. The input resolution is 384 larger than previous models such as DecoupleTNL, and larger resolution can bring obvious performance improvement. Ablation study on the input resolution is lacked. Author should clarify the input resolution in Table 1 and add more ablation study on resolution and backbone to prove the improvement comes from the memory mechanism instead of stronger backbone and larger resolution.\n2. I do not think the memory mechanism can only be applied for visual tracking without language. In another word, the memory mechanism is also suitable for visual tracking. Authors should add experiments to explore the performance of the memory mechanism in visual tracking.\n3. Writing is confusing. The explanation of method is confusing. Authors should try to clarify this part to make it easy-to-follow.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer unPL,**\n\nThanks for your careful review of our work. Your acknowledgment of our memory mechanism modeling and its promising performance is invaluable to us. We notice that you are particularly concerned about the model's ablation analysis. Therefore, we have conducted the corresponding experiments and hope that the following experimental analysis will address your concerns.\n\n---\n\n### **Weakness 1: more ablation analysis**\n\n（1）**Ablation on backbone:**\n\nAs you mentioned, previous research [1][2] has demonstrated that the HiViT-based backbone offers performance improvements over the ViT-based backbone, which is why we chose it. It is important to note that the primary contribution of our paper is the modeling of memory mechanisms rather than backbone design. Therefore, our ablation experiments focus on the former. To address your concerns, we have conducted the relevant experiments, with results shown in the table below:\n\n| # | Backbone | Our Memory Mechanism | AUC  | P_N  | P    |\n|---|----------|----------------------|------|------|------|\n| 1 | HiViT    | √                    | 63.3 | 80.9 | 67.4 |\n| 2 | HiViT    | ×                   | 59.0 | 77.6 | 62.4 |\n| 3 | ViT      | √                    | 62.2 | 79.8 | 66.4 |\n\nComparing #1 and #2, it is evident that the introduction of our memory mechanism improves AUC by 4.3%. Comparing #1 and #3, we can see that the HiViT backbone offers AUC gains by 1.1%, consistent with previous research conclusions [1][2]. Moreover, the performance improvement brought by our memory mechanism exceeds that of the HiViT-based backbone, demonstrating that the superior performance of our work primarily stems from the memory mechanism rather than the HiViT-based backbone.\n\n（2）**Ablation on image resolution:**\nTo our knowledge, vision-language trackers typically report results at a single resolution setting, as seen in DecoupleTNL [3] (256), QueryNLT [5] (320), and MMTrack [6] (384). This limits our comparison to the results provided by DecoupleTNL at a resolution of 256 and is also the reason why we did not perform resolution-related ablation experiments. We will clarify the input resolution in Table 1. To address your concerns, we supplement our experiments at the 256 resolution setting (tested on TNL2K):\n\n| # | Setting    | AUC  | P_N  | P    |\n|---|------------|------|------|------|\n| 1 | MemVLT-256 | 62.1 | 79.6 | 65.9 | \n| 2 | MemVLT-384 | 63.3 | 80.9 |67.4 |\n\nAs expected, reducing the image resolution somewhat degrades performance. However, MemVLT-256 still significantly outperforms some recent models with larger resolutions, such as MMTrack [6] (384) and QueryNLT [5] (320) (see **Table 1**). This demonstrates that our method maintains strong performance even at lower resolutions.\n\n---\n\n### **Weakness 2: generalization of the model**\n\nThanks for your suggestion. We have conducted experiments under the vision-only tracking setting. The results demonstrate that our method can generalize to the vision-only tracking task. Specifically, we remove the text-related components from MemVLT, seamlessly transforming it into a vision tracker. As shown in the table below, our proposed Memory Interaction Module (MIM) and Memory Storage Module (MSM) incrementally improve the performance of the vision-only tracker, demonstrating the good generalizability of our method.\n\n| Setting  | AUC  | P_N  | P    |\n|----------|------|------|------|\n| Naive    | 57.4 | 74.9 | 60.6 |\n| +MIM     | 58.3 | 76.1 | 61.0 |\n| +MSM     | 59.6 | 76.8 | 62.3 |\n\n---\n\n### **Weakness 3: method description**\n\nWe apologize for any confusion caused by the description of our method. Due to the need for temporal modeling, the connections between various modules are relatively complex. For instance, when introducing the memory interaction module, we need to explain the meaning of the memory features used in the memory storage module. We suspect this might have caused the confusion. In the revised version, we will meticulously review our paper's wording and enhance the introduction of key concepts in method section. We hope this will provide readers with a reading experience similar to that of Reviewers BESZ and jufM, who highly commended the readability of our paper.\n\n---\n\nWe believe this detailed response has clarified your concerns. We would appreciate it if you could reconsider your rating based on this additional information. Please reach out if you have further questions.\n\n---\n\n[1] Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers, Xie et al., in CVPR 2024.\n\n[2] Explicit Visual Prompts for Visual Object Tracking, Shi et al., in AAAI 2024.\n\n[3] Tracking by Natural Language Specification with Long Short-Term Context Decoupling, Ma et al., in ICCV 2023.\n\n[4] Joint Visual Grounding and Tracking with Natural Language Specification, Zhou et al., in CVPR 2023.\n\n[5] Context-Aware Integration of Language and Visual References for Natural Language Tracking, Shao et al., in CVPR 2024.\n\n[6] Towards Unified Token Learning for Vision-Language Tracking, Zheng et al., in TCSVT 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gVVBYFIW4W", "reviewer": "Reviewer_jufM", "summary": "This paper introduces a memory-based vision-language tracker, called MemVLT. Specifically, the memory storage and memory interaction modules are designed to storage and flexible interaction between short-term and long-term memories, achieving vision-language tracking. A large number of experiments demonstrate the effectiveness of the proposed method.", "questions": "- What do MIM and MSM stand for in line 146? The paper doesn't specify.", "limitations": "The limitations of the method have been accounted for in this paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.the paper is well written and can be easily understood by the reader.\n2.The proposed tracker can achieve SOTA performance and adequate ablation experiments are accomplished.", "weaknesses": "- The idea of designing a long and short-term memory network based on transformers is not novel enough. Its main technical solution consists of multiple transformer layers and memory strategies, which have been fully explored by many previous approaches, e.g., AQATrack [32].\n\n- Figure 1(a) shows the consistency curves for search frames and linguistic descriptions (and template), respectively. However, there is a lack of improved consistency curves, which helps to demonstrate the effectiveness of the proposed method. In addition, consistency analysis why ResNet is used as the feature extraction network instead of ViT or HiViT.\n\n- Since HiViT is used as the visual encoder, I suspect that the performance gain of MemVLT comes from the visual backbone and not from the proposed long and short-term memory module. I would like the authors to provide comparison experiments under different trackers, e.g., ostrack based on ViT backbone [35], to demonstrate the generalization of the proposed method.\n\n- The authors are requested to provide ablation experiments on the number of SMG and transformer decoder layers to observe the effect on the proposed tracker.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer jufM,**\n\nThanks for thoroughly reviewing our work. We appreciate your acknowledgment of our model's strong performance and the quality of our writing. Below, we address your concerns and provide detailed responses:\n\n---\n\n### **Weakness 1: novelty of our work**\n\n(1) **Novelty in long and short-term memory modeling:**\nTo highlight the differences between our method and existing work based on long and short-term memory, we have provided direct comparisons in **Sections 2.1** and **2.2**. Additionally, please refer to our summary of distinctions in **global response 5**.\n\n(2) **Contributions of our technical solution:**\n\n1). Considering the strong feature representation capability of transformers, we used vanilla transformer layers [1] and transformer variants with multiple cross-attention layers [2][3] to achieve feature interaction among different elements. This is a commonly adopted approach in mainstream model design. However, it is important to emphasize that our primary contribution lies in the modeling of memory mechanisms using these basic network layers, rather than the design of these transformer layers.\n\n2). Our CLS-based memory strategy shares similarities with the temporal modeling approaches in some contemporary visual trackers (e.g., AQATrack [1]) but also exhibits distinct differences, which we have directly compared in **Section 2.2**. For instance, in terms of memory storage, our proposed section-top storage strategy significantly outperforms the sliding windows approach used in AQATrack (see **Table 4**).\n\n---\n\n### **Weakness 2: regarding Figure 1(a)**\n\n(1) The purpose of the consistency curve in **Figure 1(a)** is to illustrate an inherent fact: static initial prompts (i.e., the template patch and language description) and dynamic targets are difficult to maintain consistently over time. Even without this curve, we can qualitatively observe this fact from **Figure 1(a)**. The curve is a supplementary explanation, and this fact cannot be changed. Therefore, we cannot improve the consistency through model design. This challenge motivates our model design to introduce implicit dynamic memory to better guide tracking. Since these features cannot be directly measured, we cannot demonstrate our method's effectiveness by improving this consistency curve.\n\n(2) As described in **Section B.1**, plotting the consistency curve only requires a pretrained backbone capable of extracting image features, such as ResNet or ViT. It is important to note that the backbone used here differs from the backbone of the tracker, as their purposes differ. To address your concerns, we have plotted the consistency curve based on ViT in the supplementary PDF, and you can see that the curves based on ResNet and ViT are consistent.\n\n---\n\n### **Weakness 3: ablation of backbone**\n\nWe understand your concerns because previous research [3][4] has demonstrated that the HiViT-based backbone offers performance improvements over the ViT-based backbone, which is why we chose it. To address your concerns, we have conducted the relevant experiments and performed the following analysis:\n\n| # | Backbone | Our Memory Mechanism | AUC  | P_N  | P    |\n|---|----------|----------------------|------|------|------|\n| 1 | HiViT    | √                    | 63.3 | 80.9 | 67.4 |\n| 2 | HiViT    | ×                   | 59.0 | 77.6 | 62.4 |\n| 3 | ViT      | √                    | 62.2 | 79.8 | 66.4 |\n\n(1) We find that our proposed memory mechanism brings a 4.3% AUC improvement (comparing #1 and #2), while the HiViT backbone offers AUC gains of 1.1% (comparing #1 and #3). This indicates that the performance improvement brought by our memory mechanism exceeds that of the HiViT-based backbone.\n\n(2) The experimental results in #3 show that with a ViT-based backbone, our model still performs well, surpassing some of the latest VLTs, such as MMTrack [5] and QueryNLT [6] (see **Table 1**). This demonstrates the generalizability of our method.\n\n---\n\n### **Weakness 4: ablation of block layers**\n\nThanks for your suggestion. We have conducted ablation experiments on the number of block layers. Due to limited time, we have temporarily supplemented the performance of SMG and transformer decoder with 1 and 5 layers and compared them with the existing model at 3 layers. It can be observed that too few block layers (comparing #1 and #2) significantly harms the model's performance, while continuously increasing the block layers (comparing #2 and #3) does not yield noticeable performance gains. Our provided model effectively reflects the convergence performance.\n\n| # | Layers | AUC  | P_N  | P    |\n|---|--------|------|------|------|\n| 1 | 1      | 60.9 | 78.3 | 64.8 |\n| 2 | 3      | 63.3 | 80.9 | 67.4 |\n| 3 | 5      | 63.2 | 81.0 | 67.5 |\n\n---\n\n### **Question 1: meaning of symbols**\n\nWe apologize for any confusion caused by the abbreviations. The meanings of **MIM (Memory Interaction Module)** and **MSM (Memory Storage Module)** are provided in the overview of the model (**lines 117-127**). We will make these abbreviations bold in the future to enhance the readability of the paper.\n\n---\n\nWe hope this detailed response addresses your concerns. We would appreciate it if you could reconsider your rating. If you have additional feedback or require further clarification, please let us know.\n\n---\n\n[1] Attention is All You Need, Vaswani et al., in NeurIPS 2017.\n\n[2] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection, et al., in arXiv 2023.\n\n[3] Autoregressive Queries for Adaptive Tracking with Spatio-Temporal Transformers, Xie et al., in CVPR 2024.\n\n[4] Explicit Visual Prompts for Visual Object Tracking, Shi et al., in AAAI 2024.\n\n[5] Towards Unified Token Learning for Vision-Language Tracking, Zheng et al., in TCSVT 2023.\n\n[6] Context-Aware Integration of Language and Visual References for Natural Language Tracking, Shao et al., in CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**Dear Reviewers,**

We would like to thank all the reviewers for their feedback and help in improving our work. We are glad that they acknowledge the **interesting insights** from introducing CLS theory into the VLT task (Reviewers BESZ, GRLs, and unPL), **adequate ablation analysis** (Reviewer jufM), **strong performance** (Reviewers BESZ, GRLs, unPL, and jufM), and **clear writing** (Reviewers BESZ and jufM). Before addressing each reviewer's concerns individually, we provide preliminary responses to some common questions raised:

---

### **1. Model computational efficiency**

Like Reviewers BESZ (Limitations) and GRLs (Weaknesses 4), we also think that computational efficiency (e.g., fps, parameters) is an essential factor. Therefore, we have already analyzed this in our paper (see **Table 2** and **Section 4.2**). The results indicate that our MemVLT is comparable to the latest trackers in terms of parameters and fps, while demonstrating superior performance.

---

### **2. Comparison with recent work**

We thank Reviewers BESZ (Weaknesses 1&2) and GRLs (Weaknesses 1,2&3) for pointing out some recent VL trackers and related models in other tasks. We have provided detailed comparisons and analyses in the individual rebuttals addressed to each reviewer. Our findings are as follows:

(1) For all VL trackers mentioned by the reviewers, our model still shows the **best average performance**, e.g., exceeding OVLM by 2.4%, and All-in-one by 4.5%. Thus, these latest trackers do not alter the conclusion that MemVLT achieves SOTA performance. 

(2) We have also thoroughly compared models from other tasks mentioned by the reviewers. Although similar terminologies like "adaptive prompts" are used in CiteTracker, our model design significantly differs from theirs.

---

### **3. Further ablation analysis on backbone**

Since the core contribution of our work revolves around memory interaction and storage modeling, our ablation experiments are focused on these aspects. We are grateful for the suggestions from Reviewers unPL (Weaknesses 1) and jufM (Weaknesses 3) regarding additional ablation analysis on the backbone (HiViT v.s. ViT).   We have conducted corresponding experiments and will present the detailed results in the rebuttals to each reviewer. Key findings include:

(1) The performance improvement from our memory mechanism surpasses that provided by the HiViT-based backbone.

(2) Even without a HiViT-based backbone, our tracker still achieves SOTA performance.

This confirms the efficacy of our core contribution in memory mechanism modeling.

---

### **4. Generalizability of memory mechanism modeling**

We thank Reviewers BESZ (Weaknesses 2) and unPL (Weaknesses 2) for their suggestion to analyze whether our method can generalize to the vision tracker. We conducted relevant experiments, and the results demonstrate that our proposed memory interaction and storage mechanisms progressively enhance the performance of the vision tracker. This indicates that our memory mechanism modeling can indeed generalize to visiual tracking.

---

### **5. Novelty in long and short-term memory modeling**

Reviewers BESZ (Questions 1), GRLs (Weaknesses 2), and jufM (Weaknesses 1) are particularly interested in how our work differs from existing methods based on long and short-term memory modeling. It is important to note that long and short-term memory is a generalized concept that includes specific memory theories such as Atkinson-Shiffrin and CLS. The CLS theory, which inspires our module design, is based on the latest discoveries in cognitive science from the **past two years** [1][2]. To highlight the novelty and distinction of our method, we have directly compared our approach with existing work in **Sections 2.1** and **2.2**. To summarize, the main distinctions are:

 (1) **Modeling motivation:** Existing works on long and short-term memory modeling, such as DecoupleTNL, typically aim to represent long and short-term memory and guide tracking with the initial prompt. However, they overlook scenarios where significant deviations exist between static prompts and dynamic targets (see **Fig.1(a)**), and using the initial prompts constantly might mislead tracking. In contrast, our work is motivated by adjusting the initial prompts using long and short-term memory to generate prompts that adapt to dynamic target changes. We then use these adaptive prompts to guide tracking.

(2) **Memory theory:**  To our knowledge, our work is the first to introduce CLS theory into the VLT task. This theory emphasizes the process of obtaining adaptive memory through long and short-term memory interaction, aligning with our modeling motivation for adaptive prompts.

(3) **Model Design:** Inspired by the CLS theory, we design a memory interaction module (MIM) and a memory storage module (MSM), which significantly differ from existing trackers. For the MIM, we emphasize adjusting static prompts through memory interaction. For the MSM, our proposed section-top storage method takes into account both the importance of memory units and the length of temporal representation, surpassing existing sliding window storage methods.

(4) **Performance:** As all reviews have emphasized, our model achieves strong performance across all benchmarks, surpassing existing VLTs based on long and short-term memory modeling. This validates the soundness of our modeling motivation and the effectiveness of our model design.

---

Below we have responded to every reviewer individually. In the discussion phase, please enlighten us on anything we can do to further improve the paper.

---

[1] Organizing memories for generalization in complementary learning systems, Sun et al., in Nature 2023.

[2] A generative model of memory construction and consolidation, Eleanor et al., in Nature 2024.

Meta-review: The paper initially received mixed borderline reviews: 3 BR, 1 BA. The major concerns raised by the reviewers were:

1) missing SOTA trackers for comparison  (BESZ, GRLs)
2) What's the novelty of vision-language tracking? (BESZ)
3) the generality of the proposed memory prompts? (BESZ)
4) novelty of adaptive prompts for tracking compared to CiteTracker? (GRLs)
5) novelty of short- and long-term memories compared to Xmem, AQATrack (GRLs, jufM)
6) computational efficiency? (GRLs)
7) unfair comparison due to different backbones, missing corresponding ablation study (unPL, jufM)
8) memory mechanism could also be applied to visual tracking (unPL)
9) consistency curves don't improve (jufM)
10) missing ablation study on the number of SMG and layers. (jufM)

The authors wrote a response. After the response, reviewers had a fruitful discussion with the authors and eventually all reviewers raised their ratings to 2BA, 1WA, and 1A. The authors should update the paper according to the reviews and discussions.
Decision: Accept (poster)