MemVLT: Vision-Language Tracking with Adaptive Memory-based Prompts

Authors: xiaokun  feng
xuchen  li
shiyu  hu
dailing  zhang
meiqi  wu
jing  zhang
xiaotang  chen
kaiqi  huang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 08:21:51

Abstract: Vision-language tracking (VLT) enhances traditional visual object tracking by integrating language descriptions, requiring the tracker to flexibly understand complex and diverse text in addition to visual information. However, most existing vision-language trackers still overly rely on initial fixed multimodal prompts, which struggle to provide effective guidance for dynamically changing targets. Fortunately, the Complementary Learning Systems (CLS) theory suggests that the human memory system can dynamically store and utilize multimodal perceptual information, thereby adapting to new scenarios. Inspired by this, (i) we propose a Memory-based Vision-Language Tracker (MemVLT). By incorporating memory modeling to adjust static prompts, our approach can provide adaptive prompts for tracking guidance. 
(ii) Specifically, the memory storage and memory interaction modules are designed in accordance with CLS theory. These modules facilitate the storage and flexible interaction between short-term and long-term memories, generating prompts that adapt to target variations.
 (iii) Finally, we conduct extensive experiments on mainstream VLT datasets (e.g., MGIT, TNL2K, LaSOT and LaSOT$_{ext}$). Experimental results show that MemVLT achieves new state-of-the-art performance. Impressively, it achieves 69.4% AUC on the MGIT and 63.3% AUC on the TNL2K, improving the existing best result by 8.4% and 4.7%, respectively.

Main paper: 
 introduction: 
 The vision-language tracking (VLT) task [1] aims to locate a user-defined object in a video sequence using multimodal prompts, which comprise a template patch and a language description. As an extension of traditional visual single object tracking (SOT) task [2, 3, 4] , VLT can harness the complementary advantages of multiple modalities. Therefore, vision-language trackers (VLTs) have the potential to achieve more promising tracking performance, which has recently attracted widespread attention [5, 6, 7, 8] .Similar to SOT, VLT still adopts the one-shot setting [9] , providing prompts only at the initial moment. However, these fixed prompts struggle to provide continuous reference for targets in video sequences due to their inherent dynamic variability [10] . As shown in Fig. 1 (a), the initial prompts depict a gun placed on a table. However, when the target is subsequently picked up, there is a significant deviation in the target's state from given prompts. Regarding the target state in different frames, we MemVLT effectively models the storage and interaction of long-term and short-term memories, resulting in prompts that adapt to the search image.quantitatively evaluate its consistency with the initial multimodal prompts (implementation details are introduced in Sec. B.1). It is evident that the consistency is poor most of the time, indicating that the initial prompts are ineffective as a reference.Despite this, most existing VLTs [11, 12, 13, 14, 15, 16] overly rely on these static prompts. As illustrated in Fig. 1 (b), they identify the target in each frame that is most similar to the initial prompts as the tracking result. Despite achieving some success, the difficulty lies in the limited generalizability of static prompts to subsequent frames, thereby impacting tracking performance.Unlike existing VLTs, humans can adaptively locate a target, even when the target appears in a form vastly different from its initial perception state [17] . Neuroscientists have long been interested in this ability and have conducted studies using the visual search task [18, 19] , which is similar to the VLT. Numerous research findings reveal that human adaptive visual search capability can be attributed to the sophisticated memory mechanisms [20, 21] .The Complementary Learning Systems (CLS) theory [22] , as a well-known memory model, has recently revealed the relationship between human generalization adaptability and memory [23] . Generally speaking, humans continuously adjust their memory by integrating perceived information to better adapt to the environment. As shown in Fig. 1 (c), the human brain achieves memory through two complementary systems: the hippocampus and the neocortex. The hippocampus plays a primary role in short-term memory, which is subsequently consolidated into long-term memory stored in the neocortex [24, 23] . The interaction between short-term and long-term memories enables humans to adapt to different environments [23, 8] . Given the advantages of this memory mechanism, a critical question arises: How can we incorporate it into the design of trackers?To achieve that, we propose a Memory-based Visual-Language Tracker named MemVLT. As depicted in Fig. 1 (c), the core insight of MemVLT lies in efficiently adapting to the dynamic changes of the target by emulating the storage and interaction of memory information, thereby enabling effective tracking. Specifically, (i) we first develop a memory storage module to simulate the functioning of short-term and long-term memory systems. Inspired by the system consolidation process [23] , we introduce an efficient long-term memory storage method named section-top. (ii) Drawing upon the stored memory, we incorporate a memory interaction module to emulate the interaction between short-term and long-term memories [25] , generating adaptive visual and textual prompts. These adaptive prompts then guide subsequent tracking by integrating with the search feature. Through these modules, MemVLT facilitates adaptive tracking by utilizing memory information.In summary, our contributions are as follows: (i) Inspired by the CLS theory, we introduce a novel tracker named MemVLT. Leveraging memory mechanism modeling, this approach facilitates the generation of adaptive prompts to effectively guide the tracking process. (ii) In the proposed MemVLT, we incorporate memory storage and memory interaction modules. They faithfully model the storage and flexible interaction processes between short-term and long-term memories in the human brain, yielding multimodal prompts that adapt to dynamically evolving targets. (iii) We evaluate the performance of MemVLT on the MGIT [5] , TNL2K [26] , LaSOT [27] and LaSOT ext [28] and achieve state-of-the-art tracking results. Notably, MemVLT achieves 69.4% AUC on MGIT and 63.3% AUC on TNL2K, improving the existing best result by 8.4% and 4.7%, respectively.2 Related Work 
vision-language tracking: 
 Vision-language tracking is an emerging multimodal task that aims to achieve tracking by utilizing a given language description and an initial template patch. Adhering to the principle of similaritymatching [9] , most existing VLTs [11, 12, 13, 14, 15, 16] leverage the static given prompts as reference to identify the most similar target in the search frame. Among them, SNLT [11] introduces a universal language region proposal network, which improves tracking performance through a dynamic aggregation module. MMTrack [16] designs an effective tracking pipeline that treats the VLT as a token generation task. While these VLTs perform well in simple scenes, they overlook the dynamic nature of targets, making it challenging to track when targets undergo significant changes.To address this limitation, certain VLTs attempt to utilize temporal information to obtain the dynamic reference. GTI [29] and AdaSwitcher [26] locate the target by integrating tracking and grounding results at each time step. Their reliance on pre-defined threshold can lead to error accumulation.JointNLT [14] incorporates temporal information in the form of a temporal query during the prediction stage. Since the length of the query tokens is much shorter than that of the static prompt tokens, it indicates that static prompts still dominate the tracking process. Differing from them, we depart from human memory mechanisms to utilize temporal information. Building upon initial static prompts, we inject dynamic features by incorporating memory information, which not only avoids the error accumulation but also generates prompts adapted to the dynamic changes of the target.By treating temporal information as memory, some trackers [30, 31, 32] have incorporated memory mechanism modeling. Extensive previous studies confirm that memory modeling is a viable approach, which is the paradigm our work follows. Distinguishing our work from existing research, to our knowledge, MemVLT is the first to apply CLS theory to tracker design. Recent research in CLS theory [33, 23] underscores the importance of the interaction between long and short-term memories. Motivated by this insight, we develop the memory interaction module to generate adaptive prompts.In contrast, existing approaches, such as DecoupleTNL [32] , focus solely on modeling long and short-term memories but overlook the interaction process. In addition to memory interaction, we propose the section-top memory storage method, which is inspired by the system consolidation process [23, 24] within the CLS theory. Compared to the sliding window storage method commonly used by existing trackers [14, 32] , it demonstrates more effective tracking performance (see Sec. 4.3).In the natural language processing field, prompt learning [34] refers to the automatic learning of instructions in the form of sentences, thereby enabling better task understanding. Considering the significant advantages of this approach in enhancing model adaptability, recent studies have extended it to vision-language tasks [35, 36] . CoOp [35] efficiently fine-tunes CLIP [37] for few-shot transfer by constructing language branch inputs using a set of learnable vectors. Bahng [36] performs prompt tuning on CLIP by prompting on the vision branch. In this work, we exploit the core idea of prompt learning by introducing learnable vectors to model the memory information. These vectors are used to adjust the initial static prompts, making them adaptive to the dynamic changes of the target.Recently, several vision-only trackers [38, 39, 40] have attempted to model temporal information using learnable vectors. These vectors, also referred to as temporal queries, are utilized to capture global information at each moment and guide the subsequent tracking process. What sets us apart is that, (i) we focus on the vision-language multimodal scenario, not only modeling temporal queries but also emphasizing their adjustment to static prompts to comprehensively leverage temporal information.(ii) We design a novel multimodal query storage method called section-top to enhance the existing sliding window storage approach. 
overview: 
 The framework of MemVLT is depicted in Fig. 2 (a). Given two static prompts (text description and template patch) and the search image at a general time step (t > 0), the text and vision encoders first The input is first encoded using text and vision encoders. Subsequently, the memory interaction module processes static prompt features based on stored memory, generating adaptive prompts. After incorporating these prompts, the search features are fed into the prediction head to obtain the tracking results. Additionally, the memory storage module utilizes processed data to represent and store memory information. (b) Diagram of memory interaction. To illustrate the process of memory interaction, we organize the memory information from the perspectives of long-term and short-term memories. Through the interaction between these memories, adaptive visual and textual prompts are obtained. embed them into specific feature spaces, denoted as f t l , f t z , and f t x , respectively. Concurrently, we obtain the global visual and textual semantic representation tokens (i.e., f t gv and f t gl ). Subsequently, these encoded features, along with the memory information M t-1 obtained from the memory storage module (MSM), are fed into the memory interaction module (MIM). Through the mutual interaction of long-term and short-term memories, we obtain the latest visual and textual short-term memories (i.e., m t v and m t l ), as well as the adaptive prompts (i.e., f t av and f t al ). These adaptive prompts then undergo feature fusion with the search feature, yielding the target-related feature f t r . Next, f t r is fed into the prediction head to obtain the tracking result and its corresponding confidence score p t c . Finally, p t c and the short-term memory are used to update the MSM, providing memory information M t for tracking in the next time step (t + 1). In the following sections, we will provide detailed introductions to each module and the dimensions of the above variables.Vision Encoder. For the visual input of the search image and template patch, we adopt the encoding paradigm of the one-stream network [41] , implementing it with the HiViT [42, 39, 40] . Specifically, the template patch z ∈ R 3×Hz×Wz and search image x t ∈ R 3×Hx×Wx are first projected into the feature space and flattened to produce token embeddings. Additionally, we introduce a [CLS] token 1 to capture global semantic feature [43] . This token is concatenated with the template-search tokens and then fed into transformer layers for feature modeling. Finally, we obtain the search feature f tx ∈ R Nx×D , template feature f t z ∈ R Nz×D , and global visual feature f t gv ∈ R 1×D . Text Encoder. Since being proposed, the BETR [43] series models have been widely used for text representation. Therefore, we utilize RoBERTa [44] , a classic pre-trained model, as our text encoder. Specifically, for a given input sentence, we tokenize it into a sequence of text tokens. The token sequences then are fed into the RoBERTa to extract the text embedding feature f t l ∈ R N l ×D . Additionally, we perform pooling on f t l to obtain the sentence-level global feature f t gl ∈ R 1×D .Due to the constraints of initial static visual-textual prompts, the f t l and f t z derived from them face challenges in offering consistent references for dynamically changing targets [9] . To alleviate this limitation, MIM attempts to leverage the long-term memory M t-1 stored in MSM to inject dynamic feature reflecting target variations into the static prompts, thereby better guiding the tracking. As shown in Fig. 2 (a), under the influence of long-term memories (i.e., M t-1 l and M t-1 v ), we first integrate with the current features to acquire the latest short-term memories (i.e., m t l and m t v ). Then, through the interaction between the memory information m t and the initial prompts, we obtain adaptive prompts and fuse them with the search features f t x , yielding target-related features f t r . Corresponding to the two modal prompts of given cues, we utilize both modalities to represent memory. Before explaining the specific construction of memory, we reiterate the short-term and long-term memory information that our model focuses on, as shown in Fig. 2 (b ). For the long-term memory provided by MSM, denoted as M t = m i Lm i=1 , it consists of L m short-term memory unitsm i = (m i v , m i l ).Here, m i v ∈ R 1×D and m i l ∈ R 1×D represent the i-th short-term visual and textual memory, respectively. Additionally, we also consider the initial given prompts as a form of long-term memory due to their role throughout the tracking process.To begin with, we introduce the generation of short-term memory under the influence of long-term memory. Initially, we concatenate the long-term visual memory m i v Lm i=1 and textual memory m i l Lm i=1 with the current global semantic features f t gv and f t gl respectively. This process yields more comprehensive visual and textual memory information,EQUATIONEQUATIONwhere * denotes extracting all elements in the set, and [;] indicates concatenation along the first dimension. Following this, we introduce the memory query to associate long-term memory with the current information, enabling the generated short-term memory to consider both historical and current features simultaneously.Considering the remarkable capability of the transformer network in feature interaction [45] , we design the short-term memory generation (SMG) layer by modifying it. In each SMG layer, the memory query sequentially interacts with long-term memory information and corresponding modality features to generate short-term memory. Taking the visual branch as an example, the computational process of the visual query q 0 v ∈ R 1×D is as follows:EQUATIONEQUATIONEQUATIONHere, Φ CA (•, •) denotes the cross-attention operation where the first element serves as Q and the second element serves to obtain K and V [46] . N orm represents the layer normalization operation and F F N denotes the feed-forward network. q k v denotes the query after being processed by the k-th layer. For brevity, we omit the positional encoding.After passing through L v stacked SMG layers, we obtain the visual short-term memory m t v = q Lv v . Similarly, we utilize the textual query q 0 l ∈ R 1×D to obtain the textual short-term memory m t l = q L l v .EQUATIONEQUATIONEQUATIONBased on the acquired short-term memory (i.e., m t = (m t v , m t l )), on one hand, we feed it into MSM to update the long-term memory (see Sec. 3.4). On the other hand, we utilize it to modulate the long-term memory represented by the initial visual-textual prompts, enabling them to adapt to the variations of the target. For the textual branch, we first utilize m t l to modulate the initial textual feature f t l , then use the adjusted textual feature to guide the search feature f t x .EQUATIONEQUATIONWhere T rans Dec denotes the vanilla transformer decoder layer [46] (see Sec. A.2 for detailed computation process). f t xl represents the search features fused with the adaptive textual prompts f t al . For the visual branch, we adopt a similar approach. As shown in Eq.11 and Eq.12, we ultimately obtain the search feature f t xlv integrated with adaptive visual and textual prompts, simplified as f t r . We treat f t r as the target-related features fed into the prediction head.EQUATIONEQUATIONIn this section, we will introduce the construction of long-term memory M t . Given a video sequence of length T , MIM generates a short-term memory m t at each time step. Typically, due to the computational burden of integrating all short-term memories, L m is much smaller than T . Therefore, the key to long-term memory storage is to design a short-term memory selection mechanism.Prior trackers, e.g., JointNLT and AQATrack, commonly use the sliding window approach to store temporal information, which can be considered a form of long-term memory. Specifically, when t > L m , short-term memories are selected from the most recent L m frames (i.e., from t-L m to t-1) for storage. While achieving some effectiveness, the redundancy of video frames leads to redundant memories being stored, and it can only consider a limited temporal range. The CLS theory refers to the process of integrating short-term memory into long-term memory as system consolidation [24] .Relevant studies [33, 23] indicate the long-term memory system primarily stores short-term memory conducive to generalization. We adhere to this principle to propose a novel long-term memory storage method called section-top. It first uniformly divides past time intervals into L m sections and then stores the most representative short-term memory within each section.To establish a short-term memory selection criterion, we design a confidence prediction module to obtain the confidence score p t c ∈ [0, 1] corresponding to the tracking result, which will be introduced in Sec. 3.5. Intuitively, a higher confidence score suggests that the current tracking result is more conducive for subsequent tracking and should thus be stored. The thorough storage process is provided in Sec. A.3. Compared to the sliding window, this method can consider a longer temporal range. Additionally, the stored memories are not adjacent to each other, which reduces redundancy.Based on the target-related search feature f t r ∈ R Nx×D , the prediction head is used to predict the final bounding box b t and its corresponding confidence score p t c . We employ a classic CNN-based tracking head [41, 40] . First, f t r is transformed into a 2D spatial feature map. Subsequently, after passing through the stacked Conv-BN-ReLU layers, we obtain a classification score map P ∈ [0, 1] 1×Hs×Ws , the size of the bounding box B ∈ [0, 1] 2×Hs×Ws , and the offset size O ∈ [0, 1) 2×Hs×Ws . Based on these features, we predict the centroid position and scale of the target, yielding the predicted bounding box b t . Additionally, we employ an additional CNN-based branch to predict p t c . For the b t , we employ the focal loss L cls [47] , L 1 loss, and the generalized IoU loss L iou [48] for supervision, which are widely used in tracker design. Regarding the confidence score p t c , we first calculate the IoU value between the tracking result and the ground truth, and utilize the L 2 loss between IoU and p t c for supervision. The overall loss function is formulated as follows:EQUATIONwhere λ L cls = 1, λ iou = 2, λ L1 = 5 and λ L2 = 1 are the regularization parameters.Table 1 : Comparison with state-of-the-arts on four popular benchmarks: MGIT [5] , TNL2K [26] ,LaSOT [27] and LaSOT ext [28] . The best two results are highlighted in red and blue, respectively. 4 Experiments 
implementation details: 
 We use RoBERTa-Base [44] as our text encoder and HiViT-Base [42, 39, 40] as our vision encoder, with the token dimension D set to 512. The sizes of template patches and search images are 192×192 and 384 × 384, respectively. For the acquisition of short-term memory, both the visual and textual branches consist of three SMG layers. For the generation and fusion of adaptive prompts, all T rans Dec operations are conducted using a single transformer decoder layer. Additionally, in the memory storage module, the default length for long-term memory is set to eight.We use the training splits of LaSOT [27] , TNL2K [26] , RefCOCOg [49] , and OTB99-Lang [1] to train our model. Each training sample consists of a text description, along with one template patch and eight search frames from the same video sequence. Utilizing the text description and template patch as prompts, we iteratively train the model by selecting one search image at a time. We employ the AdamW to optimize the network parameters and conduct a total of 200 training epochs. 20,000 image pairs are randomly sampled in each epoch. The model is trained on a server with four A5000 GPUs and tested on an RTX-3090 GPU. The tracking speed is about 32 FPS.We evaluate MemVLT on four benchmarks, including MGIT [5] , TNL2K [26] , LaSOT [27] and LaSOT ext [28] . MemVLT is compared with existing state-of-the-art (SOTA) VLTs, which share the similar task setting and training datasets configuration to ensure a fair comparison.MGIT. MGIT is a novel large-scale benchmark [2, 5] specifically tailored for the VLT task. Each sequence contains complex spatio-temporal causal relationships and is annotated with language descriptions at three levels of granularity: action, activity, and story [5] . As shown in Tab. 1, MemVLT demonstrates superior performance compared to other VLTs at the action granularity. Particularly, MemVLT excels over the SOTA tracker JointNLT [14] , surpassing it by 8.4%, 2.7%, and 19.2% in area under the curve (AUC), normalized precision (P Norm ), and precision score (P), respectively. Although JointNLT is equipped with a temporal module, this temporal information does not interact with the initial prompts. In other words, the initial static prompts still dominate the tracking process. These results highlight that the utilization of adaptive prompts plays a crucial role in complex scenarios. Additionally, our model also achieves optimal performance under the activity and story text settings (see Tab. A1). TNL2K. TNL2K is designed for the VLT task, and the introduction of attributes such as "adversarial samples", "modality switch" and "cross-camera" significantly adds to the challenges [26] . In Tab. 1, our proposed framework demonstrates superior performance compared to existing VLTs. When compared with SOTA tracker MMTrack [16] , which doesn't utilize the temporal information, our approach achieves gains of 4.7%, 5.7%, and 8.0% in AUC, P Norm , and P, respectively. These results underscore the effectiveness of leveraging memory information for providing adaptive prompts. Efficiency analysis. In Tab. 2, we compare MemVLT with the two latest VLTs in terms of efficiency (Params and Speed) and performance (AUC and P on TNL2K). Compared to JointNLT [14] , which only introduces temporal features in the decoder phase, MemVLT fully leverages memory information during the feature fusion stage. Despite the increased number of parameters, the model performance is significantly improved without compromising tracking speed. As for MMTrack [16] , the absence of temporal modeling accelerates its forward process but limits tracking performance. These results underscore the trade-off between efficiency and performance achieved by MemVLT.In this section, we conduct comprehensive ablation studies on the TNL2K [26] benchmark.Study on important model components. The core insight of our proposed model lies in the modeling and utilization of memory information, which adjusts initial static visual-textual prompts to adapt to the dynamic changes of the target. To investigate the effect of different model components, we conduct ablation analyses based on whether the incorporation of memory information is utilized to generate the adaptive visual or textual prompts, corresponding to the Vision adap and Text adap entries in Tab. 3. Tab. 3 (#1) presents the performance without utilizing the memory information. Comparing it with the results in Tab. 3 (#2, #3), we observe that utilizing memory information in any single modality can enhance performance. For example, adaptive visual or textual prompts lead to a 3.3% and 3.4% increase in AUC, respectively. Additionally, Tab. 3 (#4) further demonstrates that, under the joint influence of adaptive visual-textual prompts, the model achieves optimal performance.Study on memory interaction. The MIM plays a crucial role in the forward process, and we conduct ablation analysis on specific model designs. Tab. 5 (#2) indicates that global visual and textual tokens (i.e., f t gv and f t gl ) are excluded from the construction of memory information. Comparing with the baseline (Tab. 5 (#1)), it is evident that the inclusion of global tokens enhances tracking performance. Tab. 5 (#3) shows that directly integrating search features with short-term memories significantly Study on long-term memory storage method. For our proposed section-top method, we compare it with other storage methods.As shown in Tab. 4, sliding window is commonly adapted by recent trackers involving temporal saving [14, 30, 31, 32] ; top-L prioritizes storing the top L short-term memories with the highest confidence; section-L divides historical memory into L sections, with each section storing the latest short-term memory. It can be observed that the performance of top-L is superior to sliding window, highlighting the significance of storing important short-term memories. Additionally, section-L outperforms top-L, indicating the benefits of considering a longer temporal range. By combining the advantages of section-L and top-L, section-top achieves optimal performance. Generalizability of our memory mechanism. Given that our proposed memory mechanism modeling significantly enhances the performance of vision-language trackers, a natural question arises: can this memory mechanism generalize to vision-only taskers? To explore this, it is necessary to conduct relevant experimental analysis. Specifically, we remove the text-related components from MemVLT, seamlessly converting it into a vision-only tracker. As shown in Tab. 6, our proposed Memory Interaction Module (MIM) and Memory Storage Module (MSM) progressively improve the performance of the vision-only tracker, demonstrating the strong generalizability of our method.For more detailed implementation specifics and further comparative analysis, please refer to Sec. B and Sec. C. The above process diagrams depict the types and sequence of feature integration. We also illustrate the tracked result bbox and groundtruth bbox. Better viewed with zoom-in. , JointNLT [14] and MMTrack [16] ) on three challenging sequences from TNL2K [26] benchmark. The first column indicates the provided initial template information. Better viewed in color with zoom-in.𝑓 ! " 𝑚 # " 𝑚 $ " 𝑓 !#$ " 𝑓 !# " 𝑓 % " , 𝑓 # " … 𝑓 % " (a) (b) (c) (d)Fig. 3 (a)-(c) depict the variations of heatmaps over the search region during the forward process. Through the integration of adaptive prompts, the model ultimately directs its focus towards the tracking target. Compared to merely fusing initial fixed multimodal prompts (Fig. 3 (d )), MemVLT demonstrates improved capability in scenarios where the target state deviates from the given static prompts.Taking the "advSamp_INF_bus6" sequence from TNL2K as an example, we showcase the model's confidence prediction scores against the actual IoU values in Fig. 4 . It can be observed that their variations align closely, indicating the effectiveness of our confidence prediction module. This facilitates the utilization of the confidence prediction score as criteria for selecting short-term memory.As shown in Fig. 5 , we visualize the tracking results of our model and the previous two SOTA models on three challenging sequences from TNL2K [26] . In these sequences, the scenes contain distractors, and the state of the target undergoes significant changes. It is evident that our model exhibits greater robustness [56] compared to others. This validates that our adaptive prompts contribute to addressing these challenges, further demonstrating the efficacy of our proposed model. 
conclusion: 
 We propose a novel vision-language tracker, MemVLT, which models memory mechanisms to provide adaptive multimodal prompts for tracking guidance. Drawing from the Complementary Learning Systems theory, we emphasize the importance of storing and interacting between short-term and longterm memories for generalized adaptation. Therefore, we incorporate memory storage and memory interaction modules. By comprehensively leveraging memory information to generate adaptive prompts, MemVLT provides consistent references for dynamically changing targets, thus achieving effective tracking performance. Extensive experiments demonstrate that our method achieves new state-of-the-art performance on four widely used benchmarks, showcasing its generalization ability across various video environments and linguistic annotation styles.Limitations. Our proposed method leverages learnable queries to implicitly construct the relationship between historical target information and the current situation, resulting in short-term memory representations of the target. The notable results obtained demonstrate the effectiveness of this memory modeling mechanism. However, these memory representations lack explicit supervision, which diminishes their interpretability. To address this limitation, we believe that incorporating tracking result data (such as target and background information at various time steps) into the memory representation could be beneficial. This integration could enhance the comprehensiveness of the memory representations and facilitate the design of diverse loss functions for explicit supervision.We consider this approach a promising direction for our future research.(a) Short-term Memory Generation Layer On the other hand, we employ an additional branch to predict p t c . We first concatenate P , B, and O along the first dimension. Then, this concatenated feature is processed through stacked Conv-BN-ReLU layers. Finally, the processed features are flattened and passed through a softmax function to produce a regression prediction value within the range [0,1]. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts of this paper in Sec. D.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper focuses on the video tracking task, which does not pose such high risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We properly credit the creators or original owners of all assets used in the paper, and cite the relevant papers for each asset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In the following sections, we will present the implementation details of our experimental analysis, including the consistency analysis between search images and prompts, as well as the implementation methods for various model variants in the ablation study.As shown in Fig. 1 (a) , to quantitatively analyze the consistency between multimodal prompts and the search target, we calculate and plot the consistency curves. For determining the consistency between the search target and the language description, we manually evaluate whether the target state matches the language description, assigning a consistency value of 1 or 0 accordingly. For computing the consistency between the search target and the template patch, we first crop the search target from the search image based on the groundtruth bounding box. Then, we input both the cropped target Algorithm 1 Section-Top Long-term Memory Storage Algorithm Input: B t-1 , B t-1 c , M t-1 , m t , p t c at time t Output: M t 1: % save the latest memory unit and its corresponding confidence score.2: B t = B t-1 .append(m t ), B t c = B t-1 c .append(p t c ) 3: M t = M t-1 4: if t ≤ L m then 5:% select the latest memory to save. t m = t mod L m , t d = ⌊ t Lm ⌋ 9: I start = (t d + 1) × (t m -1) + 1, I end = I start + t d 10:% select most representative memory unit in each section 11:for i in t m to L m do 12:Sec m = B t [I start : I end ], Sec c = B t c [I start : I end ]13:I max = MaxIndex(Sec c ) 14: M t [i] = B t [I max ]15:I start = I end + 1, I end = I start + t d -1 16:end for 17: end if 18: return M t and the template patch into a pretrained ResNet backbone network for feature encoding. Finally, the consistency value is obtained by calculating the cosine similarity between the encoded features.Corresponding to Tab. 3, we conduct ablation analyses to investigate the effect of different model components, based on whether the incorporation of memory information is utilized to generate adaptive visual or textual prompts. When adaptive visual prompts are not used (i.e., Vision adap is not selected), the MIM module neither generates short-term visual memory nor subsequent adaptive visual prompts. Instead, the search feature interacts only with the initial template feature, replacing Eq. 3, Eq. 4, Eq. 5, Eq. 9 and Eq. 10 with Eq. A4. Additionally, to ensure a fair comparison, we set the number of network layers used in Eq. A4 to be the sum of the network layers in Eq. 3, Eq. 4, Eq. 5, Eq. 9 and Eq. 10.f t xl = T rans Dec (f t x , f t l ). (A4)Additionally, not selecting Vision adap means that no short-term visual memory is stored in the MSM module. Similarly, by not selecting Text adap , we can achieve this by replacing Eq. 6, Eq. 7, Eq. 8, Eq. 11 and Eq. 11 with the following equation:f t r = T rans Dec (f t xl , f t z ). (A5)Corresponding to Tab. 5, we conduct ablation analyses on the relevant model designs in MIM. The "remove global tokens" setting means that the global visual and textual tokens (i.e., f t gv and f t gl ) are excluded from the construction of memory information. Specifically, the modified expressions for H t-1 v and H t-1 l are as follows:H t-1 v = [ * M t-1 v ] = [m 1 v ; . . . ; m Lm v ]. (A6) H t-1 l = [ * M t-1 l ] = [m 1 l ; . . . ; m Lm l ]. (A7)The "remove adaptive prompts" setting means directly integrating search features with short-term memories, which removes the generation of adaptive prompts and their subsequent fusion with the % select the latest memory to save. % sliding window process.7: M t [1 : (L m -1)] = M t [2 : L m ] 8: M t [L m ] = m t 9: end if 10: return M t Algorithm 3 Section-L Long-term Memory Storage Algorithm Input: B t-1 , M t-1 , m t ,B t = B t-1 .append(m t ) 3: M t = M t-1 4: if t ≤ L m then 5:% select the latest memory to save.6:M t [t] = m t 7: else 8: t m = t mod L m , t d = ⌊ t Lm ⌋ 9: I start = (t d + 1) × (t m -1) + 1, I end = I start + t d 10:% select the latest memory unit in each section 11:for i in t m to L m do 12:M t [i] = B t [I end ]13:I start = I end + 1, I end = I start + t d -1 14:end for 15: end if 16: return M t search features. For the visual branch, we replace Eq. 9 and Eq. 10 with Eq. A8. Similarly, for the textual branch, we replace Eq. 11 and Eq. 12 with Eq. A9EQUATION)f t r = T rans Dec (f t xl , m t v ). (A9)Additionally, to ensure a fair comparison, we set the number of network layers used in Eq. A8 to be the sum of the network layers in Eq. 9 and Eq. 10, and set the number of network layers used in Eq. A9 to be the sum of the network layers in Eq. 11 and Eq. 12.Corresponding to Tab. 4, we conduct ablation analyses on different long-term memory storage methods in MSM. In addition to the section-top long-term memory storage method shown in Algorithm 1, we also present the implementation processes of three other storage methods in Algorithm 2, Algorithm 3, and Algorithm 4. For top-L method, we introduce a new variable M t c , which stores the confidence store corresponding to each memory unit in M t .Input: M t-1 c , M t-1 , m t , p t c at time t Output: M t 1: M t = M t-1 , M t c = M t-1 c 2: if t ≤ L m then 3:% select the latest memory to save. in Tab. A1, our proposed MemVLT outperforms existing vision-language trackers on these two benchmarks. Compared to SOTA tracker MMTrack [16] , our model surpasses it by 2.9 % and 2.7 % in area under the curve , respectively. However, we also notice that the performance of MemVLT does not surpass existing SOTA visual-only trackers [38, 70] . We speculate that this may be related to the quality of some language descriptions on these two benchmarks. For instance, some descriptions may be too vague to effectively locate the target [14, 26] , leading to misleading guidance instead of facilitating tracking.All-in-one and OVLM As emphasized in Sec. 4.2, MemVLT is compared with existing state-of-the-art VLTs that share similar task settings and training dataset configurations to ensure a fair comparison. As shown in Tab. 1 and Tab. A1, the VLTs we compare against are primarily trained using the La-SOT [27] , TNL2K [26] , RefCOCOg [49] , and OTB99-Lang [1] datasets. In addition, two recent VLTs, namely OVLM [72] and All-in-one [73] , utilize training dataset configurations that differ significantly from ours. Therefore, we conduct a separate comparison with these two VLTs.OVLM. The comparison results are shown in Tab. A2. Our MemVLT significantly outperforms OVLM on LaSOT (AUC +5.2%) but slightly falls short on TNL2K (AUC -1.4%). Overall, our model demonstrates superior average performance (AUC +2.4%). Besides, although OVLM involves memory mechanism modeling, there are key differences between our method:1. Memory Representation: OVLM assumes the text prompt is a precise long-term cue, using vision and text features to represent short and long-term memories, respectively. However, it overlooks the misalignment between text information and video targets (see Fig. 1 (a) ), which can mislead the tracker [14] . Our method addresses this by using both text and vision features to comprehensively represent short and long-term memories, and modulate them with dynamic information. 2. Memory Interaction: OVLM focuses solely on the unidirectional modulation of short-term memory features by textual long-term memory, whereas our method achieves bidirectional modulation between long-term and short-term memories. The importance of bidirectional interaction has been highlighted by recent research [23] .In summary, our model provides a more comprehensive representation and interaction of memory information, resulting in superior performance. All-in-one. As shown in Tab. A3, our tracker significantly outperforms All-in-one on all benchmarks except LaSOT ext . Overall, the average performance of our model far exceeds that of All-in-one (AUC +4.5%). Although All-in-one uses a larger training dataset to align text and vision modalities, it neglects the modeling of dynamic temporal information. In contrast, MemVLT effectively models temporal information through the memory mechanism, resulting in superior tracking performance.To predict the confidence score p t c , we introduce a confidence prediction module and supervise it using the L 2 loss between p t c and the IoU value. As shown in Tab. A4, we analyze the impact of this module on the model's performance using the TNL2K dataset. Without confidence scores, we lack a criterion for short-term memory selection. Therefore, we conduct tests using the sliding window storage method (Tab. A4 (#1)). When the confidence prediction module is introduced but the sliding window method is still used (Tab. A4 (#2)), we observe no significant change in performance. However, when the section-top storage method is employed, we see a significant improvement in model performance (Tab. A4 (#3)). This demonstrates that our proposed long-term memory storage method effectively leverages the confidence prediction information.In the process of generating short-term memories, we draw inspiration from the widely acknowledged paradigm of prompt learning [35] , which introduces learnable query tokens to understand and represent corresponding features. Specifically, in the generation of visual and textual short-term memories m t v and m t l , we incorporate corresponding query tokens q v and q l , respectively. As shown in Tab. A5 and Tab. A6, we conduct the ablation analysis on the lengths of these query tokens. Under the setting of a long-term memory length of 4, we test the influence of different lengths of visual and textual queries on the model performance. The results indicate that there is no substantial variation in model performance with changes in query length. This underscores the adequacy of a query length of 1. Consequently, we default to a query length of 1 in the experiments presented in the paper.In Sec. 4, all our experiments are conducted using the HiViT-Base backbone and a 384×384 image resolution. To provide a more comprehensive understanding of our model's performance, we perform ablation studies by evaluating the model under the ViT-Base backbone and a 256×256 resolution.In Tab. A7, we not only evaluate the model's performance using two different backbones but also analyze the impact of our proposed memory mechanism. Comparing #1 and #2, it is evident that the introduction of our memory mechanism improves the AUC by 4.3%. Comparing #1 and #3, we observe that the HiViT backbone provides an AUC improvement of 1.1%, which is consistent with previous research findings [40, 39] . Furthermore, the performance boost brought by our memory mechanism surpasses that of the HiViT-based backbone, indicating that the superior performance of our work primarily originates from the memory mechanism rather than the HiViT-based backbone.In Tab. A8, we evaluate the performance of our model under two different search image resolutions. As expected, reducing the image resolution slightly degrades performance. However, MemVLT-256 still significantly outperforms several recent models with larger resolutions, such as MMTrack [16] (384×384) and QueryNLT [54] (320×320) (see Tab. 4.2) . This demonstrates that our method maintains strong performance even at lower resolutions. In our proposed MemVLT, the SMG module, responsible for generating short-term memory, plays a critical role. To investigate its impact, we conduct ablation experiments on the number of SMG layers. Specifically, we test the performance of SMG with 1 and 5 layers, compare the results with the existing model using 3 layers.As shown in Tab. A9, it is observed that using too few layers (comparing #1 and #2) significantly harms the model's performance, while increasing the number of layers (comparing #2 and #3) further does not provide noticeable performance gains. Therefore, our model effectively demonstrates performance convergence.In this paper, we introduce a novel vision-language tracker named MemVLT. Inspired by the Complementary Learning Systems (CLS) theory, this model emulates human memory systems to adapt given static prompts to dynamically changing targets, thereby achieving effective tracking. Generic object tracking is one of the fundamental problems in computer vision with numerous applications such as video surveillance, robotics, and autonomous vehicles. The vision-language tracking task extends this field by incorporating text modality. Besides providing a new type of human-machine interaction, this task leverages the complementary advantages of visual and language modalities to achieve more promising tracking results. Our research can improve tracking performance while maintaining a reasonable running speed (32 FPS). However, it is of particular concern that this tracker could be misused for illegal surveillance and positioning. Despite achieving promising results, applying this technology to real-world scenarios, such as autonomous driving, remains challenging. To mitigate the risks associated with using MemVLT, we encourage researchers to thoroughly understand the implications of using trackers in specific real-world scenarios.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: We provide this information in Sec. 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: We carefully review the NeurIPS Code of Ethics and ensure that the research aligns with it in all aspects.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).It is worth noting that the symbols CLS and [CLS] represent two distinct concepts in this paper. The former denotes the Complementary Learning Systems theory, while the latter represents the classification token[43]. 


Decision: The paper initially received mixed borderline reviews: 3 BR, 1 BA. The major concerns raised by the reviewers were:

1) missing SOTA trackers for comparison  (BESZ, GRLs)
2) What's the novelty of vision-language tracking? (BESZ)
3) the generality of the proposed memory prompts? (BESZ)
4) novelty of adaptive prompts for tracking compared to CiteTracker? (GRLs)
5) novelty of short- and long-term memories compared to Xmem, AQATrack (GRLs, jufM)
6) computational efficiency? (GRLs)
7) unfair comparison due to different backbones, missing corresponding ablation study (unPL, jufM)
8) memory mechanism could also be applied to visual tracking (unPL)
9) consistency curves don't improve (jufM)
10) missing ablation study on the number of SMG and layers. (jufM)

The authors wrote a response. After the response, reviewers had a fruitful discussion with the authors and eventually all reviewers raised their ratings to 2BA, 1WA, and 1A. The authors should update the paper according to the reviews and discussions.

Reviews: review: paper_summary: paper_summary: This paper proposes a Memory-based Visual-Language Tracker based on the complementary learning system theory. Memory mechanism modeling facilitates the generation of adaptive prompts to effectively guide the tracking process. Extensive experiments demonstrate that MemVLT achieves new state-of-the-art performance.
strength_weakness: strength_weakness: 1) For experiments, recent SOTA trackers are missing in Table 1, such as MMA (CVPR 2024), CAI (CVPR 2024), OVLM (TMM23) ...
2) OVLM is also a recent vision-language memory network for tracking, but the authors have not dicuss and compare it.
questions: questions: 1) The long and short-term memory have been widely used in visual object tracking, so it is not a new technique. What's the difference for vision-language tracking?
2) Is it general to apply the proposed memory prompts for single-modality visual tracking framework?
limitations: limitations: In addition to performance, tracking speed (fps) is also need to be compared and discussed for real-time applications.
paper_summary: paper_summary: This paper introduces MemVLT (Memory-based Vision-Language Tracker), an approach to vision-language tracking (VLT) that incorporates memory mechanisms inspired by human cognition. The key innovation is the use of adaptive memory-based prompts to guide tracking, as opposed to relying solely on initial static prompts.
The model consists of two main components:
-A Memory Storage Module (MSM) that efficiently stores and manages long-term memory information.
-A Memory Interaction Module (MIM) that generates adaptive visual and textual prompts by integrating stored memory with current input.
The authors evaluate MemVLT on several VLT benchmarks, demonstrating state-of-the-art performance, particularly on challenging datasets like MGIT and TNL2K.
strength_weakness: strength_weakness: - The insights of using adaptive prompts for tracking guidance have been introduced by CiteTracker[1]. More analysis and comparison between them should be provided.
- Short-term and long-term memory have been exploited more in the video object segmentation task like Xmem[4]. More analysis and comparison between them should be provided.
- The authors are suggested to conduct more comparisons with the most recently published VL tracker like[2][3].
- While the model achieves good performance, it may be more computationally intensive than simpler approaches due to the memory mechanisms.

[1] Li X, Huang Y, He Z, et al. Citetracker: Correlating image and text for visual tracking[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 9974-9983.
[2] Zhang C, Sun X, Yang Y, et al. All in one: Exploring unified vision-language tracking with multi-modal alignment[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 5552-5561.
[3] Wu Z, Zheng J, Ren X, et al. Single-model and any-modality for video object tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 19156-19166.
[4] Cheng H K, Schwing A G. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 640-658.
questions: questions: The main limitation of this paper lies in its limited novelty. The authors should provide a more detailed comparison based on this weakness.
limitations: limitations: no.
paper_summary: paper_summary: This paper presents a new visual-language tracking method, called MemVLT, which incorporates memory modeling to adjust static prompts.The design of MemVLT is inspired by the theory of Complementary Learning Systems (CLS), which adapts to changes in the target by mimicking the storage and interaction mechanisms of the human memory system.
strength_weakness: strength_weakness: 1. Performance comparison is unfair. MemVLT is built on HiViT-base and the input resolution is 384 instead of 256. HiViT-base can improve the tracking accuracy compared to ViT, which has been proved in previous researches. However, authors did not provide the ablation study on backbone. The input resolution is 384 larger than previous models such as DecoupleTNL, and larger resolution can bring obvious performance improvement. Ablation study on the input resolution is lacked. Author should clarify the input resolution in Table 1 and add more ablation study on resolution and backbone to prove the improvement comes from the memory mechanism instead of stronger backbone and larger resolution.
2. I do not think the memory mechanism can only be applied for visual tracking without language. In another word, the memory mechanism is also suitable for visual tracking. Authors should add experiments to explore the performance of the memory mechanism in visual tracking.
3. Writing is confusing. The explanation of method is confusing. Authors should try to clarify this part to make it easy-to-follow.
questions: questions: See Weakness
limitations: limitations: Experiments are not sufficient.
paper_summary: paper_summary: This paper introduces a memory-based vision-language tracker, called MemVLT. Specifically, the memory storage and memory interaction modules are designed to storage and flexible interaction between short-term and long-term memories, achieving vision-language tracking. A large number of experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: - The idea of designing a long and short-term memory network based on transformers is not novel enough. Its main technical solution consists of multiple transformer layers and memory strategies, which have been fully explored by many previous approaches, e.g., AQATrack [32].

- Figure 1(a) shows the consistency curves for search frames and linguistic descriptions (and template), respectively. However, there is a lack of improved consistency curves, which helps to demonstrate the effectiveness of the proposed method. In addition, consistency analysis why ResNet is used as the feature extraction network instead of ViT or HiViT.

- Since HiViT is used as the visual encoder, I suspect that the performance gain of MemVLT comes from the visual backbone and not from the proposed long and short-term memory module. I would like the authors to provide comparison experiments under different trackers, e.g., ostrack based on ViT backbone [35], to demonstrate the generalization of the proposed method.

- The authors are requested to provide ablation experiments on the number of SMG and transformer decoder layers to observe the effect on the proposed tracker.
questions: questions: - What do MIM and MSM stand for in line 146? The paper doesn't specify.
limitations: limitations: The limitations of the method have been accounted for in this paper.

Comments: title: Acknowledgment of Rating Increase and Planned Revisions
Further Clarification and Insights Based on Reviewer Discussions
Acknowledgment of Rating Increase and Planned Revisions
Acknowledgment of Rating Increase and Planned Revisions
Comment
Acknowledgment of Rating Increase and Planned Revisions
Comment
Further Response on Reviewer's Concerns (continue)
Further Response on Reviewer's Concerns
Concerns about performance improvement
Further Response on the Details of the Ablation Study
Details on Ablation Study
comment: **Dear Reviewer BESZ,**

Thank you very much for your further feedback. We are honored that our response has resolved most of your concerns, and we are delighted that you have raised our rating to "Weak Accept."

We will also incorporate your valuable suggestions and include the results from the discussions above in the revised manuscript.

**Wishing you all the best !**
Thanks for your enthusiastic response, which addressed most of my concerns. I decide to raise my rating to the borderline accept.
**Dear Reviewer BESZ,**

Thank you once again for taking the time to review our paper. Your insightful suggestions have provided us with valuable inspiration.

This comment is not intended to rush you, but rather to share with you some discussions we've had with other reviewers who raised similar concerns. We sincerely hope that by sharing these insights, we can further address any remaining concerns you may have.

---

### **Weakness 1 & 2: Comparison with recent SOTA trackers**

Regarding the recent models you mentioned, namely MMA (CVPR 2024), CAI (CVPR 2024), and OVLM (TMM23), we conducted direct comparisons and analyses in our initial rebuttal. These analyses highlighted several distinct differences between our model and existing works, while also demonstrating the superior tracking performance of our approach.

In addition, **Reviewer GRLs** also brought up recent works such as CiteTracker (ICCV 2023), Xmem (ECCV 2022), All-in-one (ACMMM 2023), Untrack (CVPR 2024), and OneTracker (CVPR 2024). After thorough comparison and analysis, **Reviewer GRLs** acknowledged the novelty and effectiveness of our method and subsequently raised our rating.

We greatly appreciate the references to these recent works provided by both you and **Reviewer GRLs**. We will cite them in future versions of our paper to further enhance our work.

---

### **Question 1: Novelty in long and short-term memory modeling**
Similar to your concerns, **Reviewer jufM** also focused on the distinctions between our CLS-based long and short-term memory modeling and existing work. In addition to the responses we provided in our initial reply to you, we conducted further analysis on the technical contributions of our approach as suggested by **Reviewer jufM**. After discussing, **Reviewer jufM** confirmed that our response fully addressed his concerns and subsequently raised our rating. We sincerely hope that our response can also resolve your concerns in this question.

---

### **Question 2: Generalizability of our memory mechanism**

Your insightful suggestion regarding the generalizability of our memory mechanism was echoed by **Reviewer unPL**. In addition to presenting and analyzing the third table from our initial response to you, we conducted additional analyses on the experimental setup in our discussions with **Reviewer unPL**. After these discussions, **Reviewer unPL** agreed that our memory mechanism generalizes well to visual trackers and subsequently raised our rating. We believe that these responses and discussions should effectively address this issue.

---

### **Limitations: Efficiency of MemVLT**

Similarly, we noticed that **Reviewer GRLs** was also concerned about the computational efficiency of our model. After reviewing Table 2 and Section 4.2 of our paper, **Reviewer GRLs**' concerns were alleviated, and our rating was raised.

---

We truly appreciate the time and effort you've invested in reviewing our paper. We believe that the additional information provided above will help clarify your concerns, and We sincerely hope you can provide further guidance on what steps we can take to improve our rating.
**Dear Reviewer jufM,**

We are thrilled to receive your response and deeply honored that our reply has resolved all of your concerns. Your decision to raise the rating is incredibly encouraging for us.

We will incorporate your valuable suggestions and include the aforementioned ablation analyses in the revised manuscript.

**Sincerely wishing you all the best ~~~**
**Dear Reviewer GRLs,**

Thanks for your response. We are honored that our reply has resolved most of your concerns. Your decision to raise the rating is of great significance to us.

We will incorporate your valuable suggestions and include the comparisons mentioned above in the revised manuscript.

**Wishing you all the best !!!**
Thank you for addressing most of my concerns in your response. I recommend that the authors add the above comparisons to the manuscript. This addition would enhance the paper's clarity. I decide to raise my rating to the borderline accept.
Thank you for your response, which addresses all of my concerns. I decided to raise my rating to accept.
**Dear Reviewer unPL,**

We sincerely appreciate your positive response.  Your decision to raise the rating is of great importance to us.

We highly value your constructive suggestions. In the revised manuscript, we will clarify our comparison conditions and include the ablation experiments related to backbone and resolution.

Wishing you all the best ~~~
Thanks for your reply. Considering your contribution, I will raise my rating to Borderline accept. I think authors should clarify the comparison condition in the later version and add the ablation study on backbone and resolution.
### **Question 2: What contributes to our strong performance?**

 **1.Our efforts to analyze what contributes to the performance improvement**

(1) Regarding our core contribution, i.e., memory mechanism modeling, we have conducted extensive ablation studies in the paper (see Sec.4.3, Tab.3–5) to explain the impact of different module designs on model performance. The main experimental conclusions can be summarized as follows:
   
   - Tab.3 shows that applying our memory mechanism modeling to both visual and textual cues results in significant performance improvements. This demonstrates that our memory mechanism can enhance the performance.
   - Tab.4 explores the effectiveness of our memory storage method within our memory mechanism modeling. The superior performance of our storage method compared to others proves its effectiveness.
   - Tab.5 investigates the effectiveness of our memory interaction method within the memory mechanism modeling. By ablating key computational operations, we explain the reasons for the performance improvements brought by our memory interaction method.

(2) We noticed that you are particularly concerned about the impact of the encoder backbone and image resolution on the model's performance, as well as the generalization of the memory mechanism to visual tracking tasks. Therefore, we supplemented the relevant ablation experiments. The main conclusions are as follows:

   - In the first table of our initial response, we have demonstrated that the strong performance of our model primarily stems from our memory mechanism design, rather than the choice of the HiViT-based backbone.
   - In the second table of our initial response, our model maintains SOTA performance even at a 256 resolution, indicating that the strong performance of our model is not due to the choice of a 384 resolution.
   - in the third table of our initial response, by using the memory-related modules as variables, we have proven that our memory mechanism modeling can generalize to visual tracking tasks.

In summary, we have thoroughly explained the reasons for the performance improvements of our model through extensive experiments. The conclusions can be summarized as follows:

   - From an overall architecture perspective, the strong performance of our model primarily comes from our memory mechanism modeling (which is also the core contribution of our work), rather than from the HiViT-based backbone or the 384 resolution.
   - Focusing on memory mechanism modeling, on one hand, we have validated the effectiveness of our proposed memory interaction module and memory storage method; on the other hand, we have also confirmed that the memory mechanism can generalize to visual tracking tasks.

**2. Regarding the generalization experiment of the memory mechanism**

(1) First, we need to reiterate the purpose and design of our experiment. To verify whether our memory mechanism modeling can generalize to visual tracking, we first removed the text-related modules from MemVLT, transforming it into a visual tracker. We then conducted the relevant experiments with the memory-related modules as the only controlled variables. The experimental results (as shown in the third table of our initial response) demonstrate that as we introduce the memory modules (MIM, MSM), the performance of the visual tracker improves significantly. This is sufficient to prove that our memory mechanism modeling can generalize to visual tracking.

(2) Based on the second question you raised, it seems that you may have misunderstood the purpose of this experiment as proving that our model in the visual tracking setting should outperform SOTA visual trackers, and that surpassing ARTrack is a measure of generalization. We have the following two points of clarification:

   - ARTrack and our model have numerous differences, making it impossible to verify the generalization of our memory mechanism based on the principle of controlled variables.
   - We conducted the experiment by removing the text-related components from MemVLT. Compared to recent SOTA models specifically designed for visual tracking, our model is at a significant disadvantage in terms of parameters. For example, ARTrack's decoder has 6 layers, whereas our visual memory-related module has only 3 similar decoder layers. Despite this, our model achieved comparable performance to the SOTA visual trackers, e.g., our model slightly underperforms ARTrack-384 (AUC -0.2%,CVPR 2023) but surpasses AQATrack-384 (AUC+0.2%, CVPR2024) and F-BDMTrack-384 (AUC+1.8%, ICCV 2023).
   - To further address your concerns, we are currently training the model in a visual tracking setting with more decoder layers. We will share the results with you as soon as they are available.

The above content addresses your concerns regarding "unfair comparison" and "unclear reasons for accuracy improvement." We sincerely hope you can provide further guidance on what steps we can take to improve our rating.
**Dear Reviewer unPL,**

Thanks for your further response.

Regarding your comments on "unfair comparison" and "unclear reason for accuracy improvement," we respectfully disagree. First, we highly value your feedback, which is why we promptly conducted additional model training and testing analyses as soon as we saw the concerns you raised. Second, based on the issues you mentioned, we recognize that you have a deep understanding and professional background in tracking tasks.  However, we believe there may be some misunderstanding regarding our experimental settings and motivations, so we feel it is necessary to provide further clarification to address your concerns about the performance.

### **Question 1: Is the performance comparison between our model and others truly unfair?**

**1. Comparison with DecoupleTNL**  

We have summarized the performance of the DecoupleTNL and our model under different backbone and resolution settings in the table below. The conclusions are as follows:

(1) As you mentioned, previous research has demonstrated that a ViT-based backbone and a 256 resolution negatively impact tracker performance. Despite this, even under these challenging settings (#1), our model still achieves strong performance. Compared to DecoupleTNL, we observe a slight performance decrease on LaSOT (AUC -0.7%), but a significant performance improvement on TNL2K (AUC +4.5%). Considering the number of search frames across the two datasets, our model shows a higher average performance (AUC +1.5%).

(2) From a model perspective, the DecoupleTNL includes both a frame encoder and a video encoder as part of its visual backbone, whereas our model only uses a single image encoder. This means our model is at a disadvantage in terms of input information (i.e., no multi-frame video data input). Despite this, we chose to compare under the same resolution (256) in #4 to ensure as much fairness as possible. Clearly, the performance in #4 significantly exceeds that of #1.

| #  | Model               | TNL2K AUC | TNL2K P | LaSOT AUC | LaSOT P | Average AUC | Average P |
|----|---------------------|-----------|---------|-----------|---------|-------------|-----------|
| 1  | DecoupleTNL (256)   | 56.7      | 56.0    | 71.2      | 75.3    | 65.0        | 67.0      |
| 2  | MemVLT (ViT-256)    | 61.2      | 65.0    | 70.5      | 77.3    | 66.5        | 72.0      |
| 3  | MemVLT (ViT-384)    | 62.2      | 66.4    | 71.9      | 79.7    | 67.7        | 74.0      |
| 4  | MemVLT (HiViT-256)  | 62.1      | 65.9    | 71.7      | 79.4    | 67.6        | 73.6      |
| 5  | MemVLT (HiViT-384)  | 63.3      | 67.4    | 72.9      | 80.5    | 68.8        | 74.9      |

**2. What settings represent a model's final performance to demonstrate SOTA performance?**  

To compare models and demonstrate SOTA performance, it is essential to focus on the model's absolute performance. To our knowledge, almost all VLTs consider only one setting for absolute performance when comparing with other models to claim SOTA results. Given that it is common for visual trackers to show results under "additional settings", we have also provided such analyses.

However, it is important to note that when comparing with other models, these "additional settings" are typically limited to resolution. For example, in OSTrack and AQATrack, models' performances at 256 and 384 resolutions are presented to compare with others.

Therefore, the absolute performance of our model can be represented by the results in the table above (#4, #5). Clearly, our model outperforms existing VLTs significantly (see Tab.1), which sufficiently demonstrates that our model achieves SOTA performance.


### **Question 2: What contributes to our strong performance?**  
The explanation for this question will be provided in the next comments.
Thanks for your response. Based on your experiment results and rebuttal, it seems that the main reason of performance improvement comes from stronger backbone and larger resolution. Firstly, for visual tracking with language, MemVLT with 256 and ViT achieves 61.2 AUC on TNL2K and 70.5 AUC on LaSOT. MemVLT  with 256 and ViT outperforms previous models on TNL2K, while achieves the similar performance on LaSOT with DecoupleTNL. Secondly, the performance of MemVLT with HiViT and 384 resolution on visual object tracking task is the same as ARTrack384 and lower than some other models, which demonstrate that the memory mechanism may not be as effective as the authors claim. 
I'm sorry that I tend to maintain my rating because of the unfair comparison and unclear reason of accuracy improvement.
**Dear Reviewer unPL,**

Thanks for your timely response. We are very glad that our previous reply addressed some of your concerns. Regarding the new questions, we have conducted relevant tests and analyses as quickly as possible, and we are eager to share and discuss these new results with you.

---

### **Question 1: Performance of MemVLT with ViT Backbone and 256 Resolution on TNL2K and LaSOT**

The three tables in our previous reply were tested on TNL2K, a recent dataset specifically tailored for the vision-language tracking task. To address your concerns, we have tested the performance of MemVLT on TNL2K and LaSOT with different backbone and resolution settings. The comprehensive results are summarized in the table below:

| Backbone | Resolution | TNL2K AUC | TNL2K P | LaSOT AUC | LaSOT P |
|----------|------------|-----------|---------|-----------|---------|
| ViT      | 256        | 61.2      | 65.0    | 70.5        | 77.3    |
| ViT      | 384        | 62.2      | 66.4    | 71.9      | 79.7    |
| HiViT    | 256        | 62.1      | 65.9    | 71.7      | 79.4    |
| HiViT    | 384        | 63.3      | 67.4    | 72.9      | 80.5    |

---

### **Question 2: Implementation Details of the Memory Mechanism Generalization Experiment**

In this experiment, we use the HiViT-based backbone with a resolution of 384 to construct different visual tracker variants and test the results on TNL2K. 

It is important to note that although we conduct the experiment with only this backbone and resolution setting, it sufficiently provides evidence of the generalization capability of our memory mechanism. This is because our memory mechanisms (i.e., memory interaction and memory storage modules) are decoupled from the backbone (see **Sec.3** and **Fig.2**). According to the basic principle of controlled variables, to verify the generalization of our memory mechanism, only the memory mechanism modules need to be adjusted while keeping the backbone and image resolution fixed.

---

We believe that the above responses address the two new questions you raised, and we sincerely hope that you can reconsider your rating in light of these new experimental results. If you have any further questions, please feel free to let us know.
Thanks for your reply, which has solved part of my concerns. I am still confused about the ablation study. Firstly, what's the performance of MemVLT when evaluated on TNL2K and LaSOT with ViT backbone and 256 resolution? Secondly, in the response to Weakness 2: generalization of the model, on which dataset did you conduct your experiments and what's the backbone and resolution?