ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model

Authors: yiming  sun
fan  yu
shaoxiang  chen
yu  zhang
junwei  huang
yang  li
chenhui  li
changbo  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-30 13:15:48

Abstract: Visual object tracking aims to locate a targeted object in a video sequence based on an initial bounding box. Recently, Vision-Language~(VL) trackers have proposed to utilize additional natural language descriptions to enhance versatility in various applications. However, VL trackers are still inferior to State-of-The-Art (SoTA) visual trackers in terms of tracking performance. We found that this inferiority primarily results from their heavy reliance on manual textual annotations, which include the frequent provision of ambiguous language descriptions. In this paper, we propose ChatTracker to leverage the wealth of world knowledge in the Multimodal Large Language Model (MLLM) to generate high-quality language descriptions and enhance tracking performance. To this end, we propose a novel reflection-based prompt optimization module to iteratively refine the ambiguous and inaccurate descriptions of the target with tracking feedback. To further utilize semantic information produced by MLLM, a simple yet effective VL tracking framework is proposed and can be easily integrated as a plug-and-play module to boost the performance of both VL and visual trackers. Experimental results show that our proposed ChatTracker achieves a performance comparable to existing methods.

Main paper: 
 introduction: 
 Visual object tracking stands as a foundational and challenging task in the computer vision realm. It aims to locate an object in each frame of a video given an initial object box. Recently, Vision-Language (VL) trackers leverage additional natural language descriptions to boost their efficacy. For instance, the shape of a target may change during tracking. However, the semantic information of the target, such as its category or material, remains the same. This makes language text more potential and stable to describe such an appearance-changing object than an image template solely. Despite these advantages, current VL trackers [12, 37, 10] are still inferior to SoTA Visual Trackers [27, 6] on mainstream benchmarks [7, 21] . We identify the following reasons for this: 1) VL trackers heavily rely on manual annotations, which often contain ambiguous language descriptions. 2) Manual textual annotations primarily focus on the tracking target and neglect the semantic information embedded in the text, such as the presence of various background objects and their relations to the target. However, most VL trackers mainly focus on better aligning the vision-language modal features [17, 37, 12] , overlooking how inaccurate textual annotations in the dataset can adversely affect VL trackers performance. In the last few years, Large Language Models (LLMs) [1] and Multimodal Large Language Models (MLLMs) [25, 38, 31] have progressed rapidly. The wealth of world knowledge encoded in the pre-trained LLMs and MLLMs, along with their capabilities in processing and understanding VL information, has attracted immediate attention from the research communities. Inspired by these advancements, we contemplate whether they can be utilized to achieve better language descriptions for visual tracking. However, we find that directly using the language descriptions generated by MLLMs hardly improves tracking performance as shown in Fig. 1 . Two primary causes are identified: 1) The VL tracker is unable to comprehend the language descriptions directly from the MLLM, resulting in the VL trackers identifying incorrect targets. This is because the MLLM and VL trackers are trained on different datasets, leading to a mismatch between the text generated by the MLLM and the visual content in the VL tracker's latent space.2) The inherent limitations of MLLMs in understanding alternate modalities exacerbate the phenomenon of "hallucination" in multi-modal contexts [5] , leading to outputs that are inaccurate or even erroneous.To address the aforementioned issues, we propose a novel framework, ChatTracker, to integrate MLLMs into visual object tracking. By utilizing the capabilities of MLLMs, a Reflection-based Prompt Optimization (RPO) module is introduced to generate accurate language descriptions of both foreground and background objects. The core idea is to provide feedback to the MLLM about inaccuracies or incomprehensible content of initial language outputs with the VL tracker. This feedback mechanism drives the iterative refinement of the MLLM's output, making it more aligned with the image content and more understandable for the VL tracker, thus effectively addressing the above mentioned issues. In addition, a novel semantic tracking module is proposed to effectively utilize the semantic information obtained from the MLLM and yield the final tracking results.Comprehensive experiments on several widely recognized public datasets are conducted, including LaSOT [7] , TrackingNet [21] , TNL2K [26] , and OTB [13] , to demonstrate the effectiveness and efficiency of our proposed method.Our main contributions are summarized as follows:1. We propose ChatTracker, a novel framework that leverages MLLMs for visual object tracking. To the best of our knowledge, this is the first work to incorporate MLLMs into tracking frameworks. It offers a plug-and-play module enhancement for existing visual and VL trackers with limited computational overhead.2. We introduce a Reflection-based Prompt Optimization (RPO) module to narrow the knowledge gap between the VL tracker and the MLLM. By reflecting on the feedbacks from tracking, the RPO module can iteratively optimize the prompt for the MLLM and finally produces accurate and relevant descriptions for tracking targets. These descriptions are superior in both tracking performance and image-text alignment compared to manually annotated texts in datasets.3. Our proposed ChatTracker achieves SoTA performance on several tracking datasets. We conduct extensive experiments including ablation studies to demonstrate the effectiveness of the proposed method and its individual modules.2 Related Work 
vision-language trackers: 
 Vision-Language tracking methods [37, 16, 12, 34, 26] have explored the use of linguistic cues to enhance visual object tracking. These approaches can be categorized based on their text sources: those using manually annotated texts and those generating descriptions from a predefined dictionary. In the first category, manually annotated texts have been prevalently employed in target tracking tasks. Datasets like LaSoT [7] , TNL2K [26] and MGIT [11] datasets provide manual annotated language descriptions for each sequence. Trackers like the SNLT tracker [8] utilize both visual and language descriptions to predict the target state, then dynamically combine these predictions to produce the final results. JointNLT [37] combines visual grounding and tracking guided by natural language, efficiently addressing the distinct requirements of both processes. The second category leverages a predefined dictionary to generate language descriptions. CiteTracker [12] meticulously develops a category vocabulary that includes attributes like color, texture, and material of the target. During tracking, it uses CLIP [23] to compare the similarity between images and text, selecting the text that closely matches the image as the target's description. In contrast to these approaches, our work exclusively employs MLLM to acquire precise text descriptions of targets. This approach effectively eliminates the reliance on manual text annotations or predefined dictionaries.Large Language Models (LLMs) like ChatGPT [1] and Llama [25] are auto-regressive models trained on extensive internet-scale text. They encapsulate a vast range of world knowledge within their weights. To integrate visual information into LLMs, various approaches have been developed [38, 4, 33] . Recently, GPT-4V(ision) was released, attracting immediate attention from the community for its outstanding multimodal perception and reasoning capabilities. Its superiority and generality are highlighted in [31] . This has paved the way for a broader spectrum of vision-centric tasks to be addressed. For instance, recent image classification approaches [22, 28, 20, 2] , first leverage a LLM to transform class names into more descriptive captions. Following this, the CLIP model is used to classify the images, enhancing the precision of classification tasks. These advancements are primarily directed towards fundamental visual recognition, such as classification and detection. In this work, we are dedicated to integrating the rich world knowledge contained in LLMs into the field of visual object tracking. 
preliminaries: 
 Problem Definition. Given a video V consisting of N frames: {I t } N t=1 , where I t represents the t-th frame of the video. A visual object tracker is tasked to predict bounding boxes P t V T that tightly wrap the target in further incoming frames with an initial bounding box G (i.e., the position of the target object in the first frame), P t V T = F V T I t ; I 1 , G . Multimodal Large Language Models. A Multimodal Large Language Model (MLLM) F M LLM takes an image I ∈ R H×W ×3 and a text prompt T p as inputs, and generates a sequence of textual outputs T o . It can be formulated as: T o = F M LLM (I, T p ) . In this paper, we utilize GPT-4V [31] , Gemini-1.0 [24] and LLaVA-7B [14] as the MLLM.Grounded Visual Language Models. A Grounded Visual Language Model (GVLM) F GV LM is designed to accept an image I ∈ R H×W ×3 and a text T with M tokens as inputs, and generates grounded region proposals P and alignment scores S for each token in the text:EQUATIONP ∈ R N ×4 denotes the bounding box coordinates of the regions, and S ∈ R N ×M is the alignment score, which quantifies the confidence of the model in aligning each word with the corresponding region in the image. M is the number of tokens in the input text and N represents the number of region proposals. The proposed ChatTracker consists of three components: a Reflection-based Prompt Optimization (RPO) module, a Semantic Tracking module, and a Foreground Verification module. The RPO module takes the template image as input and generates text descriptions of the foreground T f ore and background T back . Then for each frame I t , the Semantic Tracking module F ST takes the textual descriptions of both the foreground T f ore and background T back as inputs, utilizes a GVLM to obtain foreground region proposals P t f ore and background region proposals P t back : P t f ore , P t back = F ST I t , T f ore , T back .(EQUATIONThe Semantic Tracking module also includes an off-the-shelf single-object visual tracker. We feed it the first frame of the video I 1 marked with initial bounding box G and search area I t and obtain visual tracking results: P t V T = F V T I t ; I 1 , G for each frame. P t V T is incorporated as the supplemental foreground proposals into P t f ore . Finally, the Foreground Verification module selects the foreground proposal with the highest confidence as the tracking result by considering the their relation with foreground proposals, background proposals, and the template. In the following subsections, we will introduce the details of each module.Initialization. We draw a green bounding box on the tracking target in the first frame I 1 , creating a new image input I m . A pre-defined human-provided prompt template T init along with I m are input into the MLLM, resulting in initial descriptions of both the foreground and background:EQUATIONwhere Extract() refers to the function that reads T 0 f ore and T 0 back from the output text of the MLLM according to a predefined output format. However, due to the hallucination issue of current MLLMs, T 0 f ore may contain ambiguous language descriptions. Inspired by the successes of LLM reflection [30, 19, 9] , we propose a reflection-based iterative method to refine the textual descriptions of the foreground target in case LLMs/MLLMs fail to generate ideal responses in a single attempt.Reflection-based Prompt Optimization. At iteration i, the MLLM generates the foreground descriptions T i f ore with M i words. We input T i f ore and the template image I z into the GVLM to obtain grounding results:EQUATIONwhere P z ∈ R N ×4 denotes the grounded regions of each target in the template image, and S z ∈ R N ×M is the alignment score between each pair of word and region in the image. Subsequently, based on P z and S z , we categorize the words {w i 1 , w i 2 , ..., w i M i } in T i f ore into positive words T i pos and negative words T i neg :EQUATIONEQUATIONwhere G is the ground truth box for target in the template image, and IoU(•) computes the Intersectionover-Union for two boxes. We define a positive word to be the word that has at least one semantically matching proposal (S nm z > θ 2 ) that overlaps significantly with the target (IoU > θ 1 ). If all matching proposals of the word can not reach the IoU threshold(θ 3 ), then the word is classified as a negative word. We assess the overall quality of the current set of foreground descriptions T i f ore using the maximum IoU between all proposals and the ground truth G, which is denoted as R i . A high value of R i means that the foreground descriptions can be well-understood by the GVLM to produce accurate grounding results. We set a threshold of ϵ for R i , and if R i > ϵ, we use T i f ore as the final foreground description. Otherwise, it indicates that the current T i f ore is inadequate for the GVLM to locate the target. In this case, we construct a new prompt based on the current positive and negative words for the MLLM to generate refined foreground text description:EQUATIONwhere Update indicates filling T i pos and T i neg into a pre-defined prompt template provided by humans, resulting in a reflection prompt. Subsequently, this reflection prompt is fed into the MLLM to derive T i+1 f ore . Note that the background description is kept the same since initialization, i.e., T i+1 back = T i back . This is because the tracking task lacks background groundtruth, preventing T i back 's iterative optimization. Although it may contain vague language description, T i back still provides strong semantic information in the tracking scenario. We iterate the above process until the foreground description generated by the MLLM is sufficient for the GVLM to locate the target, i.e., R i > ϵ, or a maximum number of iterations is reached.After obtaining accurate descriptions, we derive two novel semantic insights previously absent: 1) the relationship between the target and background in the scene, and 2) language descriptions of both foreground and background objects. To utilize these semantic information, we design the Semantic Tracking Module benefiting from MLLMs and the RPO module. Initially, we input the image I m along with a pre-defined human-provided prompt template into the MLLM. The MLLM then determines whether the target is suitable for tracking using textual information about the relationship between the target and other objects in the scene. If the MLLM deems it unsuitable using textual description, we directly use the visual tracker's prediction P t V T as the P t f ore and set P t back to empty. Otherwise we use language descriptions of the foreground T f ore and background T back to obtain foreground proposals P t f ore and background proposals P t back . We first perform grounding using the concatenated words of T f ore and T back on the template image I z :EQUATIONwhere C(•) denotes the word concatenation operation with each word separated by '.'. Then tokens associated with bounding boxes that exhibit a high IoU score with the ground truth G (exceeding threshold θ 1 ) are classified as foreground tokens V f ore . Conversely, tokens linked to bounding boxes with a low IoU score (below threshold θ 3 ) are categorized as background tokens V back :V f ore = {v m | ∃ n such that S nm z > θ 2 ∧ IoU(P n , G) > θ 1 }, V back = {v m | ∃ n such that S nm z > θ 2 ∧ IoU(P n , G) < θ 3 } \ V f ore .Note that v m is the m-th token (one word may contain multiple tokens) in C(T f ore , T back ). We categorize foreground and background by tokens instead of words because we empirically found it leads to better performance in semantic grounding and tracking. After the foreground and background tokens are divided, they are fixed and used during the tracking of all subsequent frames. For the t-th frame, we obtain region proposals: P t , S t = F GV LM (I t , C(T f ore , T back )). Then, using V f ore and V back , we classify the region proposals P t into foreground proposals P t f ore and background proposals P t back :P t f ore = {P t n | ∃ m such that S t mn > θ 2 ∧ v m ∈ V f ore }, P t back = {P t n | ∃ m such that S t mn > θ 2 ∧ v m ∈ V back }.Because P t f ore may be empty, we additionally incorporate the result of the visual tracker P t V T as the supplemental foreground proposals into P t f ore .To further select result in P t f ore , we compute two types of metrics: W f ore and W back . The W f ore is determined based on the similarity between the proposal and the template, whereas W back assesses the proposal's relationship with the background proposals. The final score is then established through a combination of W f ore and W back .Motivated by [36] , we trained a neural network f (•) with generated foreground and background proposals to map the target template and foreground proposals into a discriminative Euclidean space.Its loss function is as follows:K i=1 ∥f (X a i ) -f (X p i )∥ 2 2 -∥f (X a i ) -f (X n i )∥ 2 2 + α +, where X a i denotes the i-th bounding box of a specific target, X p i a positive sample of identical target in other frames, and X n i is a negative sample of any other target or background. α is a margin value. During inference, we can determine the foreground scores W f ore = {s 1 f ore , s 2 f ore , ..., s N f ore } of the foreground proposals P t f ore = {P t1 f ore , P t2 f ore , ..., P tN f ore } using a cosine similarity metric as s i f ore = max similarity f (z), f ϕ P ti f ore , 0 , where z represents the target template in the first frame and ϕ P ti f ore is the image cropped within the bounding box P ti f ore . Background Scorer. To enhance tracking performance by incorporating background information, we develop a background scorer. This scorer scores a foreground proposal P t f ore by its maximum IoU with all the background proposals P t back . During the inference stage, background scores W back = {s 1 back , s 2 back , ..., s N back } are computed as follows:EQUATIONFinally, the overall score W all = s 1 , s 2 , . . . , s N is determined by combining the foreground and background scores:EQUATIONIn the final step, the foreground proposal b i with the highest score s i is selected as the output of the tracking process.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: Our research does not involve ethical issues or data. We are sure that the research conducted in the paper fully confirm the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experimental settings: 
 Our experiments are conducted on an NVIDIA 3090 GPU. The alignment score threshold θ 2 is set to 0.2, while the IoU thresholds for foreground and background, θ 1 and θ 3 , are set to 0.3 and 0.1, respectively. In the proposed RPO module, ϵ is set to 0.4. We adopt GPT4V-preview1106 [31] as our default MLLM, GroundingDINO-T [15] as the GVLM. We use MixFormer and ARTrack as the visual trackers for ChatTracker-L and ChatTracker-B, respectively. ChatTracker-L is designed for better performance, while ChatTracker-B is designed to achieve a better trade-off between accuracy and speed. We used UVLTrack-B [17] in place of STM and FVM in ChatTracker-B. As shown in Table 1 , we compare ChatTracker against 5 state-of-the-art vision-language trackers and 8 state-of-the-art visual trackers on three popular datasets [7, 21, 26] .LaSOT [7] is a large-scale, long-term single object tracking benchmark with 280 videos, each averaging more than 2,448 frames. And each video includes a phrase simply describing the tracking target. On this dataset, ChatTracker-L achieves the top-tier performance, with an AUC of 74.1%, surpassing JointNLT by a large margin of 13.5%. This again proves the text generated by iterative refinement of the MLLM can enhance the tracker's understanding of both the target and the overall scene, which leads to improvements on long term tracking performance.TrackingNet [21] , a prominent large-scale benchmark for short-term object tracking, comprises an extensive collection of 30,643 video segments. But it does not include textual annotations describing the target. In this challenging dataset, ChatTracker-L has achieved a remarkable AUC of 86.1%, surpassing all previous trackers. This performance demonstrates our tracker's superior capability in handling diverse and dynamic short-term tracking scenarios. It it noteworthy that, the lack of textual annotations in the TrackingNet test set renders most vision-language trackers ineffective. However, our ChatTracker demonstrates its unique ability to adapt to this scenario, thus broadening the scope of vision-language tracking applications.TNL2K [26] is a benchmark designed for evaluating vision-language tracking algorithms. And each video here contains a more detailed phrase describing the tracking target. Compared to the recent vision-language tracker JointNLT, which utilizes both the annotated sentences and bounding box, our approach surpasses it by 12.1% in precision. This indicates that with the proposed RPO module, our tracker is able to utilize optimized textual descriptions for more accurate tracking.Our proposed framework can act as a plug-and-play solution that boosts the performance of both visual trackers and VL trackers, demonstrating superior generalization capabilities.For visual trackers, we have integrated four distinct visual trackers with ChatTracker-B. Table 2 shows that this integration results in significant performance improvements for all trackers. This suggests that our proposed framework can be generally used with other existing visual trackers to boost their performance.For VL trackers, we replaced the manually annotated text inputs from the dataset with the foreground descriptions generated by ChatTracker. These results in Table 3 demonstrate that ChatTracker can generate language descriptions that are more accurate than manually annotated descriptions and can effectively boost tracking performance in general. With different MLLM. We also replace GPT-4V with gemini-1.0-pro-vision-latest [24] and LLaVA-7B [14] in the ChatTracker-B to study whether our method can adapt to different MLLMs (both proprietary and open-source). As shown in Table 5 , these MLLMs generally lead to performance improvements, and surprisingly, the results of adopting LLaVA-7B are comparable with proprietary MLLMs. This shows the effectiveness of our method itself regardless of the choice of the MLLM.To further validate the generation of high-quality language descriptions of our proposed method, we conduct image-text matching experiments. Specifically, we cropped the target from each frame and calculated its text-to-image alignment scores [23] with both manually annotated textual descriptions and ChatTracker-generated descriptions. In Table 4 , we report the maximum text-to-image alignment score during the iteration process for each sequence. As shown in Table 4 , ChatTracker-generated descriptions have better text-to-image correlation compared with manually annotated descriptions across three datasets. Such advancements highlight the potential for enhancing future vision-language trackers by providing more accurate language descriptions.To validate the effectiveness of the proposed modules, we perform ablation studies on three variants of our model. Base Model exclusively employs the visual tracker to perform the tracking task. In the ablation study, we use TransT-N4 [3] as the visual tracker. w/o RPO utilizes manually annotated text from the dataset as input for semantic tracking. Due to the absence of background descriptions generated by the RPO module, we only generate foreground proposals P t f ore , and use W f ore to select the tracking results. w/o ITER solely utilizes the foreground and background text descriptions generated from the first iteration of the RPO module for tracking.First, w/o RPO achieves similar performances with Base Model on all three datasets, which indicates that manually annotated text is sub-optimal for performing vision-language tracking. Then, comparing w/o ITER with Base Model or w/o RPO, we observe a modest enhancement across three datasets. The improvements validate the effectiveness of our proposed Semantic Tracking module and Foreground Verification module. The textual descriptions directly obtained from an MLLM might be inaccurate and noisy. Nevertheless, our approach effectively mitigates this noise by utilizing the semantic matching ability of the GVLM. Furthermore, when we compare our method with Base Model and w/o ITER, there is a notable performance improvement across all datasets. These improvements demonstrate that our proposed RPO module can generate accurate descriptions of the target by utilizing the rich knowledge of the MLLM, and the generated descriptions are even better than the manually annotated ones provided by the datasets. Finally, the performance gain of our method over w/o ITER verifies the effectiveness of iterative optimization of the RPO module.To better understand the effectiveness of the proposed RPO module, we illustrate the process of prompt optimization in a dialogue scenario in Fig. 3 .The example on the right is from the Swing-14 sequence, where a person is depicted mid-air, gripping a horizontal bar, which is the tracking target. However, the dataset's manual annotation describes it as "swing swinging above a man in black pants". Without context, even humans might struggle to identify the target using this description. Initially, the MLLM, drawing from its extensive knowledge, accurately describes the object as a "Trapeze bar". Yet, due to the knowledge gap between the vision-language tracker and the MLLM, the tracker fails to locate the target. After receiving the feedback, the MLLM rephrases its output with simple terms like "person" and "bar". It then uses the semantic context of "hanging" to assist the tracker in target identification. At this stage, the tracker can approximately locate the target. However, since the IoU is below the preset threshold of 0.4, it sends "hanging" back as a positive sample and "person" as a negative sample to the MLLM for further refinement. In the final iteration, the MLLM pinpoints "hands" as the critical term. This term is both easy to understand and consistently visible on the target throughout the sequence.When the tracking target is in low resolution or lacks discernible visual features, the MLLM struggles to provide an accurate language description of the target. Additionally, accessing the MLLM via API necessitates an internet connection, which may pose challenges in edge deployments due to intermittent or unreliable network access.Temporal changes to target and background are one of the challenges in the visual tracking domain. However, we do not update language descriptions for two key reasons. First, the tracker's predictions are not always accurate, and there are no annotations for background objects, making it harder to generate prompts dynamically. Second, calling MLLMs multiple times during tracking to update these prompts adds much computational cost. Achieving a good balance between performance and efficiency requires extensive research.Additionally, our ChatTracker focuses solely on the visual features of the tracking target, such as shape, texture, and color, without addressing the impact of different granularities of text annotations as discussed in MGIT [11] . 
conclusion: 
 In this work, we introduced ChatTracker, the first method that utilizes the Multimodal Large Language Model (MLLM) to enhance the performance of visual tracking. We proposed a Reflection-based Prompt Optimization (RPO) module to iteratively refine the ambiguous and inaccurate language descriptions of the target with tracking feedback. Moreover, a simple yet effective visual-language tracking framework was proposed to boost the performance of existing trackers as a plug-and-play method. Experimental results on multiple datasets demonstrated that our method outperformed state-of-the-art methods. This suggests that incorporating MLLMs into visual tracking had a notable effect on improving tracking performance. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The paper discuss both positive and negative social impacts of this work in A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All the creators or original owners of assets used in this paper are properly credited and cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: LaSOT-giraffe-15LaSOT-monkey-17 Foreground Verification module.ChatTracker-L with Ground Truth does not use the Foreground Verification module but selects the proposal with the highest IoU with Ground Truth as the tracking result. This entry is used to establish the theoretical upper bound.ChatTracker-L with Random Selection does not use the Foreground Verification module and randomly selects a proposal as the tracking result. The results are shown in the Table 7 . The results show that using the Foreground Verification module significantly outperforms the random selection group. This demonstrates that our Foreground Verification module is effective. Although there may still be room for improvement, our ChatTracker-L result is close to the theoretical upper bound.We conduct performance comparison on OTB-Lang dataset. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. All reviewers recommend acceptance after discussion (with two borderline accepts, one weak accept and one accept), and the ACs concur. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.

Reviews: review: paper_summary: paper_summary: The paper proposes a new Visual-Language (VL) tracking framework called ChatTracker, that integrates MLLMs into VL tracking through iterative refinement of text prompts for VL trackers. The text prompts optimized using the proposed Reflection-based Prompt Optimization (RPO) module are more accurate than manual text annotations in the datasets and improve the tracking performance by taking into account both foreground and background objects.
strength_weakness: strength_weakness: - The proposed framework does not take into account the potential temporal changes in the video that might require a more optimal foreground/background text prompt to proceed. For example, in Fig. 3 we see the framework landing on ‘hands, bar’ as the positive prompt, which focuses on hands being attached to the bar, which may not be true for the rest of the video. The background text of the video is also never updated. In fact, neither the GVLM that the framework heavily depends on nor the Foreground Verification module have any temporal components. Therefore, the framework would have to heavily rely on the baseline tracker in this aspect.
- The quality of the region proposals coming out of the Semantic Tracking Module is un-addressed. How often GVLM proposals are preferred to the visual tracker results (unless the prompt optimization fails as mentioned in Appendix A)? How dependent is the final performance on how well the Foreground Verification module is performing (since it’s only been trained on the LaSOT training set)?
questions: questions: - The potential limitations mentioned in the weaknesses should be addressed or discussed in the paper. Temporal changes to target and background are the main challenges in visual tracking and the proposed framework does not add any mechanisms to address these even though it might heavily depend on GVLM results for its answers.
- Related to the above point: Have the authors done any experiments where foreground/background prompts were updated while the video is being processed? What were the insights?
- As mentioned above, how often does the visual tracker proposal get picked over a GVLM proposal and vice versa when both exist in the pool going into the Foreground module? Or does the visual tracker proposal mostly get used when there are no GVLM proposals?
- Related works section can be expanded a bit. An important motivation for this paper is the disadvantages brought by the manual textual annotations in the VOT benchmarks (that have them). For the broader audience, the authors could add a short section on related benchmarks and if they have manual text annotations, explain how those came about (why are they low quality?). Line [300-301] mentions the prompt “swing swinging above a man in black pants” - which almost sounds like a generated text prompt, having this context in the paper itself would be helpful.
- Equation (6) - What’s the purpose of $T^i_{pos}$ in this equation? 
- Lines [152-153] - What happens if a word matches to no proposals? Can we add that to negative words? Why not?
- Lines [204-205] - To confirm, the anchor sample is always the target template, but do you sample positive samples from tracker result or foreground proposal? Visual tracking is mentioned in Line 202, but not in the Appendix.
- Line [221] - The IoU threshold for especially foreground seems a little low, what’s the reason for this? Do you have any statistics for what the accepted IoUs are from the foreground verification? Is it generally lower with the proposals?
- Please use $P_{norm}$ or NP consistently across tables. Also, for a broader audience you should mention what the metrics are even if it's in a couple sentences.
- Some tables report NP and P, others one or the other. It would be better to report consistently even if it’s just across smaller tables. Moreover, baseline tracker results in Table 3 are not matching the same tracker’s results in Table 1. If I’m missing some details, please let me know.
- Fig. 3 - not seeing any blue boxes in the image, even though CiteTracker is in the legend.
limitations: limitations: Authors did address some limitations of their method in the Appendix. Please see my discussions about other potential limitations in the weaknesses and questions sections above.
paper_summary: paper_summary: The paper proposes a novel Multimodal Large Language Model framework to improve the vision-language visual tracking performance.
By introducing the reflection-based prompt optimization module, the tracking prompt can be iteratively refined via tracking feedback.
The proposed method shows state-of-the-art results compared to prior vision-language trackers and visual trackers.
strength_weakness: strength_weakness: The paper should illustrate the number of chat iterations in the reflection-based prompt optimization module and their performance effect. The example in Figure 5 shows that the module needs 2 iterations to get an accepted prompt but the overall analysis should be considered, especially in the images with complex scenes and objects.
questions: questions: Please refer to the weakness.
limitations: limitations: The authors have mentioned the limitations.
paper_summary: paper_summary: This paper proposes ChatTracker, a novel framework that leverages MLLMs for visual object tracking. The Reflection-based Prompt Optimization (RPO) module can narrow the knowledge gap between the VL tracker and the MLLM. ChatTracker also achieves SoTA performance on several tracking datasets.
strength_weakness: strength_weakness: 1. SOTA trackers are missing in Table 1, such as OVLM (TMM23), MMTrack(TCSVT23), etc. There is a discrepancy between the data from UVLTrack in Table 1 and the data provided by the official source.
2. I want to know the model's performance on MGIT (NeurIPS '23) and OTB99_Lang (CVPR '17) to verify the generalizability of the method in different scenarios (short-term tracking and global instance tracking).
3. Why is it necessary to design ChatTracker-L and ChatTracker-B, but use different backbone networks? Also, I believe comparing the L model in Table 1 is unfair, as UVLTrack provides the L model, and annotations are needed for models of different scales.
4. For analysis of language descriptions generated by ChatTracker, if the natural language description includes background information, is it reasonable to only crop out the target for calculating image-text similarity?
5. I believe the phrase "via Chatting with Multimodal Large Language Model" in the title overstates the contribution of the entire framework.
questions: questions: Please refer to weaknesses.
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduces a simple yet effective VL tracking framework based on a multimodal large language model, called ChatTracker. The main idea is to utilize the rich world knowledge in multimodal large language model to generate high quality language descriptions and improve tracking performance. Specifically, a reflection-based prompt optimization module is proposed to iteratively refine the ambiguous and inaccurate descriptions of the target with tracking feedback. A large number of experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: - Line 127 mentions only the initial bounding box and search frame as inputs and does not mention template frames as inputs. However, the equation in line 128 writes template as input.

- In section 4.1, MixFormer-L and ARTrack-256 as visual coders for ChatTracker-L and ChatTracker-B? I have a question, why not use mixformer-B/L or ARTrack-B/L uniformly as visual encoders. Could the authors please provide more results for mixformer-B and ARTrack-L for performance comparison.

- In section 4.3 and Table 2, when applying the proposed module to other visual trackers, the paper does not describe how the linguistic description is fused with the visual features, could the authors please explain this process specifically.

- The LaSOT_ext and OTB_Lang test datasets are also included in the VL tracking task, and the authors are asked to provide performance on these datasets to help enrich the experimental results in this paper.
questions: questions: - It is an interesting idea to apply large language models to the visual tracking field. The general practice would be to do fine-tuning operations on the downstream data so that the large language model is more adapted to the downstream task. I have a question, does the proposed approach require fine-tuning in the downstream tracking dataset?
limitations: limitations: The limitations of the method have been accounted for in this paper.

Comments: title: Thanks to  Reviewer aCWh
Thanks to Reviewer Bwy6
Thanks to Reviewer gPPX
Comment
Thanks and Response to Review z5a5
Official Comment by Reviewer z5a5
Response to Reviewer z5a5（part2）
Response to Reviewer gPPX（part3）
Response to Reviewer  gPPX（part2）
comment: Dear Reviewer,

Thank you very much for recognizing our work.

Best wishes,

The Authors
Thank the authors for their response. I am satisfied with the answer. After carefully reading other reviews, I will maintain the score.
Dear Reviewer,

Thanks a lot for your recognition of our work.

Best wishes,

The Authors
Dear Reviewer,

Thank you once again for your valuable comments. We also agree with reviewer z5a5's advice on this further research direction. It would be very interesting to extend our framework in MGIT's multi-granular annotations and dataset.  

Since analyzing the effect of multi-granular annotations is non-trivial and beyond the scope of this paper, we will investigate this direction in our future work.

Thank you very much!

Best wishes,

The Authors
Thanks to the author's reply, my concerns were largely addressed. I will maintain my original rating score.
I want to thank the authors for all their efforts in responding to my and other reviewers' questions and comments. The analyses the authors provided were really helpful for clarifying the mechanics of their method for me. After reading the other reviews and the authors' responses to them, I'm more comfortable with my assessment of the paper. Thanks to the authors for including all this new information in their revised manuscript. I do agree with reviewer z5a5's comment that it would very valuable to analyze the proposed method on MGIT's multi-granular annotations.
Dear Reviewer:

Thank you again for taking the time to review our work and for providing us with valuable feedback. We now address your additional comments.

***Q1.  MGIT provides flexible and diverse semantic labels. This benchmark is somewhat closer to the "chat" feature than current VLT benchmarks. Therefore, I suggest that the author conduct further experiments on this benchmark to analyze the effects and limitations of the method under different granularities and evaluation mechanisms, and consider adding this part of the analysis to the article to highlight the motivation and theme of the article.***

We appreciate your valuable suggestion. MGIT is a high-quality vision-language benchmark. The MGIT dataset focuses on the impact of text annotations at different granularities on tracking performance. 

However, our ChatTracker only uses the initial bounding box as input and does not utilize additional text descriptions as input.
Although we agree that further applying our proposed method to MGIT is very interesting, employing different text annotation granularities is beyond the scope of this paper. 

In the rebuttal, we have provided ChatTracker's results on the MGIT dataset with the initial bounding box and we will include these results in the revised manuscript.

Thank you again for your valuable suggestion. We will further investigate these text inputs into our framework in future work.

***Q.2 In addition, please note that the author cited the wrong MGIT literature (Reference 5) in the second reply window for me. I am unsure what the relationship is between the work cited by the author and the rebuttal itself.***

Thank you for your careful reading. We have corrected Reference [5] in the comment. 

***Q.3 Finally, NeurIPS officially recommends that the reply to each reviewer during the rebuttal should not exceed 6,000 characters (only one window). I noticed that the author's reply to 2 reviewers had exceeded the limit of one window. In a sense, this is unfair (for other authors of the same period who only use one window for rebuttal). I hope the authors can pay attention to these details.***

Thanks again for your advice. For additional comments, we follow the [NeurIPS 2024] Clarification on author rebuttal email:

> - Comments to paper and reviews will be fine. Comments can be seen in time. Please set the readers correctly when you post them. Reviewers are not required to take comments into consideration.

To better discuss the issues, we responded to the longest review (up to 11 questions) in one additional Comments and provided relevant references in another Comment. 

We strictly follow the policy and instructions given by the NeurIPS 2024 PCs email. This email is sent to all authors and we don't think there is any fairness issue.

We hope our responses have fully addressed your concerns. If you have any other questions or need further clarification, please feel free to let us know.

Thank you very much!

Best wishes,

Authors
I read the author's experiments and replies to all reviewers. I am grateful for the author's efforts during the rebuttal, which largely answered my doubts. There are still some small details that need to be paid attention to. On the one hand, MGIT provides flexible and diverse semantic labels. This benchmark is somewhat closer to the "chat" feature than current VLT benchmarks. Therefore, I suggest that the author conduct further experiments on this benchmark to analyze the effects and limitations of the method under different granularities and evaluation mechanisms, and consider adding this part of the analysis to the article to highlight the motivation and theme of the article. In addition, please note that the author cited the wrong MGIT literature (Reference 5) in the second reply window for me. I am unsure what the relationship is between the work cited by the author and the rebuttal itself. Finally, NeurIPS officially recommends that the reply to each reviewer during the rebuttal should not exceed 6,000 characters (only one window). I noticed that the author's reply to 2 reviewers had exceeded the limit of one window. In a sense, this is unfair (for other authors of the same period who only use one window for rebuttal). I hope the authors can pay attention to these details. In summary, I would like to temporarily maintain my original score and listen to the opinions of other reviewers on the authors' rebuttal.
Due to the word limit in the rebuttal, we provide the remaining citations here.

[1]OVLM: Huanlong Zhang, Jingchao Wang, Jianwei Zhang, Tianzhu Zhang, Bineng Zhong "One-stream Vision-Language Memory Network for Object Tracking" TMM 2023

[2]MMTrack: Yaozong Zheng, Bineng Zhong, Qihua Liang, Guorong Li, Rongrong Ji, Xianxian Li"Towards Unified Token Learning for Vision-Language Tracking" TCSVT 2023

[3]Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen  Huang, Juehuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark.  International Journal of Computer Vision, 129:439–461, 2021.

[4]Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng  Wu. Towards more flexible and accurate object tracking with natural language: Algorithms  and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern  Recognition, pages 13763–13773, 2021.

[5]Shiyu Hu, Dailing Zhang, Meiqi Wu, Xiaokun Feng, Xuchen Li, Xin Zhao, and Kaiqi Huang. 2023. "A Multi-modal Global Instance Tracking Benchmark (MGIT): Better Locating Target in Complex Spatio-temporal and Causal Relationship." Paper presented at the 37th Conference on Neural Information Processing Systems (NeurIPS 2023), Track on Datasets and Benchmarks.

[6]Y. Wu, J. Lim and M. -H. Yang, "Object Tracking Benchmark," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1834-1848, 1 Sept. 2015, doi: 10.1109/TPAMI.2014.2388226.

[7] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with  natural language specification. In 2023 IEEE/CVF Conference on Computer Vision and Pattern  Recognition (CVPR), pages 23151–23160, 2023.

[8]Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, and Yihong Gong. Autoregressive visual  tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern  Recognition, pages 9697–9706, 2023.

[9]Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Jinpeng Zhang, and Mengxue Kang.  Unifying visual and vision-language tracking via contrastive learning, 2024
Due to the word limits, we provide the remaining citations here.

[1]Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen  Huang, Juehuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark.  International Journal of Computer Vision, 129:439–461, 2021.

[2]Y. Wu, J. Lim and M. -H. Yang, "Object Tracking Benchmark," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 9, pp. 1834-1848, 1 Sept. 2015, doi: 10.1109/TPAMI.2014.2388226. 

[3] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with  natural language specification. In 2023 IEEE/CVF Conference on Computer Vision and Pattern  Recognition (CVPR), pages 23151–23160, 2023.

[4]Yinchao Ma, Yuyang Tang, Wenfei Yang, Tianzhu Zhang, Jinpeng Zhang, and Mengxue Kang.  Unifying visual and vision-language tracking via contrastive learning, 2024

[5]Xin Li, Yuqing Huang, Zhenyu He, Yaowei Wang, Huchuan Lu, and Ming-Hsuan Yang.  Citetracker: Correlating image and text for visual tracking. In Proceedings of the IEEE/CVF  International Conference on Computer Vision, pages 9974–9983, 2023.
Due to the word limit in the rebuttal, we provide the remaining responses here.

***Q7  Lines [204-205] - To confirm, the anchor sample is always the target template, but do you sample positive samples from tracker result or foreground proposal? Visual tracking is mentioned in Line 202, but not in the Appendix.***

No, we do not sample positive samples from the visual tracker's results during training. We only randomly choose a target patch from other frames of the same video as the positive sample. In Line 202, we use the trained $ f(.) $​ (which is a neural network) to calculate the foreground score of the visual tracker's result. We will clarify this in the revised manuscript.



***Q8 Line [221] - The IoU threshold for especially foreground seems a little low, what’s the reason for this? Do you have any statistics for what the accepted IoUs are from the foreground verification? Is it generally lower with the proposals?***

-  The IoU threshold for especially foreground seems a little low, what’s the reason for this? Is it generally lower with the proposals?

For the IoU threshold for foreground(i.e. $\theta_1$ in equation (5)  ), we set $\theta_1$  to 0.3. This is to ensure there are enough positive words  $T_{pos}^i$  as feedback during the early iterations, helping the iterative optimization process converge quickly.

For accepted IoUs (i.e., $\epsilon$ in line157) ，we choose the IoU threshold based on the experiments. We tested on the LaSOT dataset, and the results are shown in the table below. The results indicate that an IoU threshold (i.e., $\epsilon$ in line157) of 0.4 leads to the best results. 

**Table.2  The performance of chattracker on Lasot dataset with different IoU thresholds**

| IoU threshold | 0.3   | 0.4   | 0.5   | 0.6   |
| ------------- | ----- | ----- | ----- | ----- |
| AUC           | 69.87 | 71.68 | 71.54 | 70.92 |

- Do you have any statistics for what the accepted IoUs are from the foreground verification? 

Yes, the average accepted IoU is 0.79. We calculated the distribution of accepted IoUs on the LaSOT dataset, and the results are shown in the table below.

**Table.3 Data distribution of IoU values on the Lasot dataset.**

| IoU        | [0,0.4) | [0.4,0.6) | [0.6,0.8) | [0.8, 1.0]|
| ------------------ | ------- | --------- | --------- | ---------- |
| number.of.sequence | 20      | 14        | 52        | 194        |

Thank you for your question. We will add the relevant results and analysis in the revised manuscript.







***Q9  Please use $P_{norm}$ or NP consistently across tables.   Some tables report NP and P, others one or the other.  It would be better to report consistently even if it’s just across smaller tables.***

Thanks for pointing this out. In the revised manuscript, we will use $P_{norm}$ or NP consistently and report metrics consistently.





***Q10  baseline tracker results in Table 3 are not matching the same tracker’s results in Table 1.***

Thanks for pointing this out.

In Table 1, the baseline trackers (i.e., JointNLT[3] and UVLTrack[4] ) are initialized using both natural language and the initial bounding box to compare with the best-performing model.

In Table 3, the baseline trackers (i.e., JointNLT and UVLTrack) are initialized using only natural language. The purpose is to compare the effectiveness of ChatTracker-generated text versus manually annotated text for Vision-Language trackers without the influence of the initial bounding box.

We will clarify this in the revised manuscript.

***Q11  Fig. 3 - not seeing any blue boxes in the image, even though CiteTracker is in the legend.***

Thanks for pointing out this typo. CiteTracker[5] is not included in this comparision and we reuse the legend picture in this figure mistakenly. We revise the manuscript accordingly. For clarification, we have placed the revised figure in the .pdf file.