ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model
Authors: Yiming Sun
Fan Yu
Shaoxiang Chen
Yu Zhang
Junwei Huang
Yang Li
Chenhui Li
Changbo Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Single object tracking
Visual object tracking
Vision-Language trackers
Multimodal learning
Primary area: machine_vision
TL;DR: We utilize Multimodal Large Language Model to enhance visual object tracking performance

Abstract: Visual object tracking aims to locate a targeted object in a video sequence based on an initial bounding box. Recently, Vision-Language~(VL) trackers have proposed to utilize additional natural language descriptions to enhance versatility in various applications. However, VL trackers are still inferior to State-of-The-Art (SoTA) visual trackers in terms of tracking performance. We found that this inferiority primarily results from their heavy reliance on manual textual annotations, which include the frequent provision of ambiguous language descriptions. In this paper, we propose ChatTracker to leverage the wealth of world knowledge in the Multimodal Large Language Model (MLLM) to generate high-quality language descriptions and enhance tracking performance. To this end, we propose a novel reflection-based prompt optimization module to iteratively refine the ambiguous and inaccurate descriptions of the target with tracking feedback. To further utilize semantic information produced by MLLM, a simple yet effective VL tracking framework is proposed and can be easily integrated as a plug-and-play module to boost the performance of both VL and visual trackers. Experimental results show that our proposed ChatTracker achieves a performance comparable to existing methods.

ChatTracker: Enhancing Visual Tracking Performance via Chatting with Multimodal Large Language Model
Yiming Sun 112 Fan Yu" Shaoxiang Chen Yu Zhang', Junwei Huang', Yang Li 1,4 Chenhui Li!, Changbo Wang
School of Computer Science and Technology, East China Normal University, Shanghai, China School of Computer Science, Fudan University, Shanghai, China
Meituan Inc
"Shanghai Frontiers Science Center of Molecule Intelligent Syntheses, Shanghai, China.
Abstract
Visual object tracking aims to locate a targeted object in a video sequence based on initial bounding box. Recently, Vision-Language (VL) trackers have proposed to utilize additional natural language descriptions to enhance versatility in various applications. However, VL trackers are still inferior State-of-The-Ar (SoTA) visual trackers in terms of tracking performance. We found that this inferiority primarily results from their heavy reliance on manual textual annotations, which include the frequent provision of ambiguous language descriptions. In this pa- per, we propose ChatTracker leverage the wealth of world knowledge in the Multimodal Large Language Model (MLLM) to generate high-qualit, language descriptions and enhance tracking performance. To this end, we propose a novel reflection-basee prompt optimization module to iteratively refine the ambiguous and inaccurate descriptions of the target with tracking feedback. To further utilize semantic information produced by MLLM, simple yet effective VL tracking framework is proposed and can be easily integrated as plug-and-pla module boost the performance of both VL and visual trackers. Experimental results show that our proposed Chat achieves a performance comparable to existing methods.
Introduction
Visual object tracking stands as foundational and challenging task in the computer vision realm [27, 3]. It aims to locate an object in each frame of video given an initial object box. Recently, Vision- Language (VL) trackers leverage additional natural language descriptions to boost their efficacy. For instance, the shape a target may change during tracking. However, the semantic information target, such its category or material, remains the same. This makes language text more potential and stable to describe such an oopearance-changing
object than an image template solely. Despite these advantages, current VL trackers [15, 46, 13] are still inferior to SoTA Visual Trackers [31, 8] on mainstream benchmarks [10, 24]. We identify the following reasons for this: 1) VL trackers heavily rely manual annotations, which often contain ambiguous language descriptions. 2) Manual textual annotations primarily focus on the tracking target and neglect the semantic information embedded in the text, such as the presence of various background objects and their relations to
Eqqual contribution. "Corresponding author. Email: @ www.eeeeeeeedu.c
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Manual Anmotation GPT-4V
swing swinging under a man black coal
Prompts generate initial description
positive: negsiive: Human, swing Regenerate Raquest
Viuaal Language Tracker
Human on ssving
"Human on swing"
Iterate times
Chattracker
Chatting
poeriiv: None negalliee: swing Regenerate Request
footwears swing sest
Visual-
footwears on swing seat"
Language
"Human on swing seat"
Great! accept it.
Iruth
Maruel Annotation
Figure I: Comparison of different text generation methods. (a) shows manual descriptions and GPT-4V generated descriptions of the tracking target, which are both sub-optimal for tracking. (b) illustrates the generation method used in Cha(Tracker.
the target. However, most VL. trackers mainly focus on better aligning the vision-language modal features [20, 46, 15], overlooking how inaccurate textual annotations in the dataset can adversely affect VL trackers performance. In the last few years, Large Language Models (LLMs) [1] and Multimodal Large Language Models (MLLMs) [29, 48, 38] have progressed rapidly. The wealth of world knowledge encoded in the pre-trained LLMs and MLLMs, along with their capabilities in processing and understanding VL information, has attracted immediate attention from the research communities. Inspired by these advancements, we contemplate whether they can be utilized to achieve better language descriptions for visual tracking. However, we find that directly using the language descriptions generated by MLLMs hardly improves tracking performance as shown in Fig. 1. Two primary causes are identified: 1) The VL tracker unable to comprehend the language descriptions directly from the MILM,, resulting in the VL trackers identifying incorrect targets. This because the MLLM and VI. trackers are trained on different datasets, leading to mismatch between the text generated by the MLLM and the visual content in the VL tracker's 's latent space. 2) The inherent limitations MLLMs understanding alternate modalities exacerbate the phenomenon of "hallucination" in multi-modal contexts [7], leading to outputs that are inaccurate even erroneous.
To address the aforementioned issues, we propose novel framework, ChatTracker, integrate MLLMs into visual object tracking. By utilizing the capabilities of MLLMs, Reflection- based Prompt Optimization (RPO) module is introduced both foreground and background objects. The core idea is to provide feedback to the MLLM about inaccuracies or incomprehensible content of initial language outputs with the VL tracker. This feedback mechanism drives the iterative refinement of the MLLM's output, making it more aligned with the image content and more understandable for the VL tracker, thus effectively addressing the above mentioned issues. In addition, novel semantic tracking module is proposed to effectively utilize the semantic information obtained from the MLLM and yield the final tracking results. Comprehensive experiments several widely recognized public datasets are conducted, including LaSOT [10], TrackingNet [24], TNL2K [30], and OTB [16], to demonstrate the effectiveness and efficiency of our proposed method.
generate accurate language descriptions of
Our main contributions are summarized as follows:
1. We propose CharTracker, a novel framework that leverages MLLMs for visual object tracking. It offers a plug-and-play module enhancement for existing visual and VL trackers with limited computational overhead.
2. We introduce Reflection-based Prompt Optimization (RPO) module narrow the knowl- edge gap between the VL tracker and the MLLM. By reflecting on the feedbacks from tracking, the RPO module iteratively optimize the prompt for the MILM and finally produces accurate and relevant descriptions for tracking targets. These descriptions are superior in tracking performance compared to manually annotated texts in datasets.
3. Our proposed CharTrrccer achieves comparable performance on several tracking datasets. We conduct extensive experiments including ablation studies to demonstrate the effectiveness of the proposed method and its individual modules.

---

2 Related Work
2.1 Vision-Language Trackers
Vision-Language tracking methods [46, 19, 15, 41, 30] have explored the use of linguistic cues to be categorized based on their text sources: those using manually annotated texts and those generating descriptions from a predefined dictionary. In the first category, manually annotated texts have been prevalently employed in target tracking tasks. Datasets like LaSoT [10], TNI.KK [30] and MGIT [14] datasets provide manual annotated language descriptions for each sequence. Trackers like the SNLT tracker [11] utilize both visual and language descriptions to predict the target state, then dynamically combine these predictions to produce the final results. JointNLT [46] combines visual grounding and tracking guided by natural language, efficiently addressing the distinct requirements of both processes. The second category leverages a predefined dictionary to generate language descriptions. CiteTracker [15] meticulously develops a category vocabulary that includes attributes like color, texture, and material of the target. During tracking, it uses CLIP [26] to compare the similarity between images and text, selecting the text that closely matches the image as the target's is description. In contrast to these approaches, our work exclusively employs MILLM to acquire precise text descriptions of targets. This approach effectively eliminates the reliance on manual text annotations or predefined dictionaries.
enhance visual object tracking. These approaches
2.2 Large Language Model in Vision Tasks
Large Language Models (LLMs) like ChatGPT [1] and Llama [29] are auto-regressive models trained on extensive internet-scale text. They encapsulate a vast range of world knowledge within their weights. To integrate visual information into LLMs, various approaches have been developed [48, 6, 40]. Recently, GPT-4V was released, attracting immediate attention from the community for its outstanding multimodal perception and reasoning capabilities. Its superiority and generality highlighted in [38]. This has paved the way for broader spectrum of vision-centric tasks to be addressed. For instance, recent image classification approaches [25, 34, 23, 2]. first leverage a L.M to transform class names into classify the images, enhancing the precision of classification tasks. These advancements are primarily directed towards fundamental visual recognition, such as classification and detection. In this work, we are dedicated to integrating the rich world knowledge contained in LLMs into the field of visual object tracking.
descriptive captions. Following this, the CLIP model is used to
3 Method
3.1 Preliminaries
Problem Definition. Given video V consisting of N frames: {1') where I' represents the t-th frame of the video. A visual object tracker is tasked to predict bounding boxes Ptr that tightly wrap the target in further incoming frames with an initial bounding box G (i.e., the position of the target object in the first frame), P Fvr (I&I),
Multimodal Large Language Models. A Multimodal Large Language Model (MLLM) FMLLM takes an image 7 TR Hx x3 and text prompt 7P as inputs, and generates sequence of textual outputs 7 It can be formulated as: T = FMLLM (T, TP). In this paper, we utilize GPT-4V[38]. Gemini-1.0 0 [28] and LLaVA-7B [17] as the MILMM
Grounded Visual Language Models. A Grounded Visual Language Model (GVLM) Fav is designed accept an image / E Hxx x3 and text T with M tokens as inputs, and generates grounded region proposals P and alignment scores S for each token in the text:
P.S (I,T)
(1)
P. E Nx denotes the bounding box coordinates of the regions, and E NRN the alignment score, which quantifies the confidence of the model in aligning each word with the corresponding region in the image. M the number of tokens in the input text and N represents the number of region proposals.
3

---

Reflection-bane based Prompt Optimization Module
Tyorei Human on swing seat.
Optimized Text Description
MLLM
P
GVLM
Template Image
iterate N times
Optimized Prompt
P.
1P2,
Furei
footwears swing seat
pos seat.
0.26
0.12
0.01
0.15
0.21
0.11
Taog human. swing.
human. mountain. house
0.14
0.03
0.27
0.01
0.23
Prompt Template
Positie/Negative
0.16
0.04
0.00
0.18
0.29
0.36
Word Selection
swing
Search image
Foreground Proposals
Semantic Tracking Module
Yore
Foreground Verification Module
Background Proposals
P
Place
Figure 2: Overall framework of the proposed algorithm. It primarily consists of three parts: A Reflection-based Prompt Optimization Module designed to generate descriptions of both the fore- ground and background elements track accurately, Semantic Tracking Module tasked with creating region proposals for these areas based on the generated descriptions, and a Foreground Verification Module that utilizes these region proposals to select the most precise tracking results. Note that the values in the figure are for visualization and may not match the actual implementation exactly.
3.2 ChatTracker Framework
The proposed CharTracker consists of three components: a Reflection-based Prompt Optimiza- tion (RPO) module, Semantic Tracking module, and Foreground Verification module. The RPO module takes the template image as input and generates text descriptions of the foreground T fore and background T beek- Then for each frame 1", the Semantic Tracking module Fsr takes the textual descriptions of both the foreground fore and background Thack as inputs, utilizes a GVLM to obtain foreground region proposals Pjore and background region proposals back-
ffore:Phe Fst (I', Ffore,Tbacke
(2)
The Semantic Tracking module also includes an off-the-shelf single-object visual tracker. We feed it the first frame of the video 1 marked with initial bounding box G and search area /' and obtain visual tracking results: Ptr Fvr (T'; T2, T2,00 G) for each frame. Pt. is incorporated as the supplemental foreground proposals into pt fore Finally, the Foreground Verification module selects the foreground proposal with the highest confidence the tracking result by considering the their relation with foreground proposals, background proposals, and the template. In the following subsections, we will introduce the details of each module.
3.3 Reflection-based Prompt Optimization Module
Initialization. We draw green bounding box the tracking target in the first frame 1 creating new image input 1" pre-defined numan-provvdee prompt template Timit along with 1 are input into the MILM, resulting in initial descriptions of both the foreground and background:
Tares Thack Extract (FMLLLM (I" (Tintt)))
where Extract() refers to the function that reads Torre and T Track from the output text of the MILLM according to a predefined output format. However, due to the hallucination issue of current MLLMs, 7 Fore may contain ambiguous language descriptions. Inspired by the successes of LLM reflection [37, 22, 12], we propose a reflection-bas based iterative method to refine the textual descriptions of the foreground target in case L.Ms//MLMM fail generate ideal responses in single attempt.
iteration i, the MILMM generates the foreground descriptions T fore with M words. We input Tore and the template image I into the GVLM to
Reflection-basee Prompt Optimization.

---

obtain grounding results:
S. = FGVLM ( .Tjoe),,
(4)
where Ps= E RR- denotes the grounded regions of each target in the template image, and S e R is the alignment score between each pair of word and region in the image. Subsequently, based on P3 and S., we categorize the words {wi, Wy ..., w in fore into positive words T and negative words T neg"
poss
T = {w- | n such that S.mh > 02
A Io G) > 01}:
(5)
T neg = {w for all that S.m > 0₂
A Alo G) 03] \ Tpos
(6)
where G is the ground truth box for target in the template image, and IoU(.) computes the Intersection- over-Union for two boxes. We define a positive word be the word that has at least one semantically
matching proposal (S > 02) that overlaps significantly with the target (loU > 01). If all matching proposals of the word can not reach the IoU threshold(Og). then the word is classified as a negative word. We assess the overall quality of the current set of foreground descriptions Tore using the
maximum IoU between all proposals and the ground truth G, which is denoted R'. A high value of R' means that the foreground descriptions can be well-understood by the GVLM to produce accurate grounding results. We set threshold of for R', and if R > we use Toore as the final foreground description. Otherwise, indicates that the current T Tý fore is inadequate for the GVLM to locate the target. In this case, we construct new prompt based on the current positive and negative words for the MLLM to generate refined foreground text description:
Tit1 FMLLM (1" Update(TTjos T Teeg))
where Update indicates filling pos and T into pre-defined prompt template provided by humans, resulting in a reflection prompt. Subsequently, this reflection prompt is fed into the MLLM to derive Ti+1 Note that the background description is kept the same since initialization, i.e., baok foree T back- This is because the tracking task lacks background groundtruth, preventing T 's iterative optimization. Although it may contain vague language description, Tiuck still provides strong semantic information in the tracking scenario. We iterate the above process until the foreground description generated by the MLLM is sufficient for the GVLM to locate the target, i.c., R or maximum number of iterations is reached.
3.4 Semantic Tracking Module
After obtaining accurate descriptions, we derive two novel semantic insights previously absent: 1) the relationship between the target and background in the scene, and 2) language descriptions of both foreground and background objects. To utilize these semantic information, Tracking Module benefiting from MLLMS and the RPO module. Initially, we input the image I" along with a pre-defined human--rovidee determines whether the target is suitable for tracking using textual information about the relationship between the target and other objects in the scene. If the MLLM deems it unsuitable using textual description, we directly use the visual tracker's prediction POT as the Pf and set Prack to empty. Otherwise we use language descriptions of the foreground Tore and background Tback to obtain foreground proposals pt fore and background proposals back. We first perform grounding using the concatenated words of T fore and T Thack on the template image I=:
design the Semantic
prompt template into the MILM. The MLLM then
S. = FEVIM ,C(Tforr; Tback)),
(8)
where C(-) denotes the word concatenation operation with each word separated by '.. Then tokens associated with bounding boxes that exhibit a high loU score with the ground truth G (exceeding threshold 01) are classified as foreground tokens Vfore. Conversely, tokens linked
to bounding boxes with a low loU score (below threshold 0z) are categorized as background
tokens Voock: Viore
fum 3 n such that Smam 0₂ A loU(P", G) 01), Voack
5

---

{Um 3 n such that STh > 82 A IoU(P", G) < 85} Vfore. Note that um is the m-th to- ken (one word may contain multiple tokens) background by tokens instead of words because we empirically found it leads to better perfor- mance semantic grounding and tracking. After the foreground and background tokens are divided, they are fixed and used during the tracking of all subsequent frames. For the t-th frame, we obtain region proposals: p St FGVLM
fore: b.cc)). We categorize foreground and
fore: Tback)), )). Then, using Vore and Vback. we classify the region proposals p' into foreground proposals Pt fore. and background
E m such that Sm > O2 A Um Vfore), Back {Pr 3 m such that Smm > 02 A Um E Vhock Because Pfore may be empty, we addition- ally incorporate the result of the visual tracker PVr as the supplemental foreground proposals into fore
proposals Pt Phack Pt fore
(Px
3.5 Foreground Verification Module
To further select result in fore: we compute two types of metrics: Fore and Waeck- The W fore is determined based on the similarity between the proposal and the template, whereas W Waec assesses the proposal's relationship with the background proposals. The final score is then established through a combination of W fore and W Waack
Foreground Scorer. Motivated by [44], trained neural network f()) with gen- erated foreground and background proposals to map the target template and foreground proposals into discriminative Euclidean space. Its loss function is as follows: EK [Ilf (x - (xP))
(XT)2) + where X denotes the i-th bound-
ing box of specific target, X?" positive sample of identical target in other frames, and X, is a negative sample of any other target or background. a is a margin value. During infer- ence, we can determine the foreground scores Woore fore: o7 :ooe: "o,, fore) fore] of the fore- ground proposals Pjore = =PPlore Pl2 Piore..... Pfote using a cosine similarity metric as s/ore = max (similarity (f(=), (o (P%% ()),)) where represents the target template in the first
frame and Φ (Pfore) is the image cropped within the bounding box P
Background Scorer. To enhance tracking performance by incorporating background information, develop background scorer. This scorer scores foreground proposal Po by its maximum IoU with all the background proposals Phack During the inference stage, background scores S. back) are computed as follows:
W Whack = {sback bhakk
bbakk = max (IoU (ffore Pij Phics))))
3
Finally, the overall score Wall = [s], s2 ...., N is determined by combining the foreground and background scores:
jore (1 bbock).
(10)
In the final step, the foreground proposal b; with the highest score s¹ is selected as the output of the tracking process.
4 Experiment
4.1 Experimental Settings
Our experiments are conducted an NVIDIA 3090 GPUs. The alignment score threshold 0g is set to 0.2, while the IoU thresholds for foreground and background. 81 and θ.,, are set to 0.3 and 0. respectively. In the proposed RPO module, e is set to 0.4. We adopt GPT4V-rreview 106 [38] as our default MILM,, GroundingDINO-T [18] 8] as the GVLM. We use MixFormer and ARTrack as the visual trackers for ChatTracker-L. and ChatTraccerr- respectively. CharTracker-L. is designed for better performance, while ChatTracker-B designed to achieve a better trade-off between accuracy and speed. We used UVLTrack-B B [20] in place of STM and FVM in ChatTracker-B.

---

Table I: State-of-the-art comparisons the datasets of TNL2K, LaSOT and TrackingNet. The best two results are shown in red and blue color. Our approach performs favorably against the tate-of-the-art methods all datasets. indicates vision-language trackers. All metrics of performance are in % in tables unless otherwise specified.
LaSOT
TrackingNet
TNL.2K
Method
Source
AUC
Porrm
AUC
PNorm
AUC
Norm
CharTracker-L
Ours
74.1
83.8
86.1
90.3
86.0
65.4
76.5
70.2
ChatTracker-B
71.7
83.6
82.2
59.6
UVL.Trac-BB [21]
AAA12024
69.4
74.9
83.4
82.1
63.1
80.9
66.7
CitcTracker [15]
ICCV2023
69.7
78.6
75.7
84.5
89.0
84.2
57.7
73.6
59.6
DecoupleTNL [19]
ICCV2023
71.2
75.3
56.7
56.0
JointNLT [46]
CVPR2023
60.4
69.4
63.6
56.9
73.5
58.1
RGFM-B256 [47]
NeurIPS2023
70.3
82.0
76.4
84.7
89.6
83.6
MixformerV2-BB [9]
NcurIPS2023
70.6
80.8
76.2
83.4
88.1
81.6
57.4
58.4
F-BDMTrack-384 [36]
ICCV2023
72.0
81.5
77.7
84.5
89.0
84.0
57.8
59.4
MITS [33]
ICCV2023
72.0
80.1
78.5
83.4
88.9
84.6
ARTrack-384 [31]
CVPR2023
72.6
81.7
79.1
85.1
84.8
59.8
SeqTrack-L384 [4]
CVPR2023
72.5
85.5
85.8
57.8
DropTrack [32]
CVPR2023
71.8
81.8
78.1
84.1
88.9
56.9
57.9
MATTracker [43]
CVPR2023
67.8
77.3
81.9
86.8
51.3
MMTrack" [45]
TCSVT2023
70.0
82.3
75.7
58.6
75.2
59.4
4.2 Comparison with Existing Trackers
As shown in Table I, we compare ChatTracker against 5 state-of-the-as vision-language trackers and 8 state-of-the-art visual trackers on three popular datasets [10, 24, 30].
LaSOT [10] is large-scale, long-term single object tracking benchmark with 280 videos, each averaging more than 2,448 frames. And each video includes a phrase simply describing the tracking target. On this dataset, ChatTracker achieves the top-tier performance, with AUC of 74.1% surpassing JointNLT by a large margin of 13.5% %. This again proves the text generated by iterative refinement of the MLLM can enhance the tracker's understanding of both the target and the overall scene, which leads to improvements on long term tracking performance.
TrackingNet [24], prominent large-scale benchmark for short-term object tracking, comprises extensive collection of 30,643 video segments. But it does not include textual annotations describing the target. In this challenging dataset, CratTracker-L has achieved remarkable AUC of 86. 1,, surpassing all previous trackers. This performance demonstrates our tracker's superior capability in handling diverse and dynamic short-tere tracking scenarios. It noteworthy that, the lack of textual annotations the TrackingNet test set renders most vision- Lasion-language trackers ineffective. However, ChatTracker demonstrates its unique ability to adapt to this scenario, thus broadening the scope of langune tracking applications.
TNL2K [30] is benchmark designed for evaluating vision- vision-language tracking algorithms. And each video here contains more detailed phrase describing the tracking target. Compared to the recent vision-language tracker JointNLT, which utilizes both the annotated sentences and bounding box, approach surpasses by 12.1% in precision. This indicates that with the proposed RPO module, our tracker is able utilize optimized textual descriptions for more accurate tracking.
4.3 Generalization and Universality
Our proposed framework can act as a plug-and-play solution that boosts the performance of both visual trackers and VL trackers, demonstrating superior generalization capabilities.
For visual trackers, we have integrated four distinct visual trackers with ChatTracker-B. Table shows that this integration results in significant performance improvements for all trackers. This suggests that our proposed framework can be generally used with other existing visual trackers to boost their performance.
For VL trackers, we replaced the manually annotated text inputs from the dataset with the foreground descriptions generated by ChatTracke These results in Table 3 demonstrate that CharTracker can generate language descriptions that are more accurate than manually annotated descriptions and can effectively boost tracking performance in general.

---

Table Results of Visual Trackers with the integration of CharTracker (marked by +). All results are measured on the same device.
LaSOT
TNL2K
OTB-lang
Methods
AUC
P
P PNorm
AUC
P
PNrrm
AUC
P
Norm
OStrack-256 [39]
69.11
75.22
78.68
54.16
53.12
69.02
69.20
90.39
83.64
OStrack-256
70.23
76.49
80.04
56.51
56.85
72.06
69.69
90.72
84.20
TransT-N4 [5]
64.85
69.02
73.78
53.16
54,26
69.70
69.55
90.61
84.25
TransT-N4+
67.34
72.38
76.73
56.27
58.47
73.09
70.06
90.63
84.51
Stark-S [35]
65.78
69.73
75.15
53.10
51.95
68.90
67.25
86.92
81.70
Stark-S+
66.76
71.29
76.35
55.63
56.12
71.59
67.93
87.81
82.60
Table 3: The comparison of results for Vision- Language trackers using ChatTracker-genenee text (marked by *). We report the AUC value on the datasets.
Table 4: Text-to-image alignment scores for manually annotated and ChatTracker- generated language descriptions. ViT and RN refer to the use ViT-B/32 and RN-50 as CLIP [26] image encoders, respectively.
Methods
LaSOT
OTB-lang
TNL2K
Join [46]
56.74
58.57
54.38
Source
LaSOT
TNL2K
OTB-lang
Manual-ViT
24.74
23.58
23.13
JointNLT*
57.96
60.07
54.82
ChatTracker-ViT
24.87
23.93
23.67
UVLTrack [21]
56.55
59.39
54.78
Manual-RN
18.03
17.57
16.87
UVLTrack*
57.23
59.98
55.61
ChafTracker-RN
18.46
18.13
17.41
With different MILM. We also replace GPT-4V with gemini-1. cemini--.0-pre-viiio---eee pr-r--viiion-aatest [28] and LLaVA- 7B [17] in the ChatTracker-B to study whether our method can adapt to different MLLMs (both proprictary and open-source). As shown in Table 5, these MLLMs generally lead to performance improvements, and surprisingly, the results of adopting VA-7B are comparable with proprietary MLLMs. This shows the effectiveness of our method itself regardless of the choice of the MLLM.
4.4 Analysis Language Descriptions Generated by ChatTracker.
To further validate the generation of high-quality language descriptions of our proposed method, we conduct image-text matching experiments. Specifically, we cropped the target from each frame and calculated its text-to-image alignment scores [26] with both manually annotated textual descriptions and TweTTrrrce--ggeeeaee descriptions. In Table 4, we report the maximum text-to-image alignment score during the iteration process for each sequence. As shown in Table 4, ChatTracker ShaaTracke--geerrtteee generated descriptions have better text-to-image correlation compared with manually annotated descriptions across three datasets. Such advancements highlight the potential for enhancing future vision- langunl trackers by providing more accurate language descriptions.
4.5 Ablation Study
To validate the effectiveness of the proposed modules, perform ablation studies three variants of our model. Base Model exclusively employs the visual tracker to perform the tracking task. In the ablation study,
TransT-N4 [5] the visual tracker. w/o RPO utilizes manually annotated text from the dataset as input for semantic tracking. Due to the absence of background descriptions generated by the RPO module, we only generate foreground proposals Pf Pfore> and use W Wfore to select the tracking results. w/o ITER solely utilizes the foreground and background text descriptions generated from the first iteration of the RPO module for tracking.
First, w/o RPO achieves similar performances with Base Model on all three datasets, which indicates that manually annotated text sub-optimal for performing -isioaelanguage tracking. Then, comparing w/o ITER with Base Model or w/o RPO, we observe modest enhancement across three datasets. The improvements validate the effectiveness our proposed Semantic Tracking module and Foreground Verification module. The textual descriptions directly obtained from MLLM might be inaccurate

---

Table 5: Results of CharTracker-B using various MLLMs. BaseTTracke is ARTracker-2555 [31].
Table 6: Ablation study of the proposed algo- rithm. The best results in each part of the table are marked in bold.
LaSOT
TNL2K
OTB-lang
Methods
AUC
Prowes
AUC
Pywm
AUC
Pytrr
Base Model
wlo RPO
w/o ITER
Ours
Tascera
70.77
79.54
58.09
74.33
69.90
84,10
LaSOT
ALC
64.85
69.02
68.80
64.63
69.12
64.87
73.07
67.89
GPT-4V
71.68
59.63
76.27
70:77
Varrs
73.78
73.75
73.95
77.18
Geminil.0
70.98
80.13
60.23
76.97
70.96
85,61
ADC
53.16
55.70
53.66
56.39
TNL.2K
P
54.26
57.27
54.65
58.76
LLaVA-7B
71.36
80.54
59,90
76.49
70.86
85.57
Pvvrma 69.70
72.60
70.28
73,03
Industraa roboti ortss
Gripper mechanism
cripping mechanism the and effector
Trapeze bar
hanging
hands Farnging
pusiiie Narw industiiin industrial rebotis arms
postiet gropping negulivar atl the
negetloe
posp None
punilis hanging negalive: parson
posiiioe hurdss bur regattine oin
Nonc
Baraton
Torstion
Inturion
Itretion
hrndton
Iurrtion
Figure 3: Illustrations of prompt optimization in a dialogue scenario. Each set shows the initial manual annotation, the subsequent prompts generated by the LLM, and the final optimized prompt that successfully guided the Vision-Language tracker to the target.
and noisy. Nevertheless, our approach effectively mitigates this noise by utilizing the semantic matching ability of the GVLM. Furthermore, when we compare our method with Base Model and w/o ITER, there is notable performance improvement across all datasets. These improvements demonstrate that our proposed RPO module can generate accurate descriptions of the target by utilizing the rich knowledge of the MILM,, and the generated descriptions are even better than the manually annotated ones provided by the datasets. Finally, the performance gain of our method over w/o ITER verifies the effectivvness of iterative optimization of the RPO module.
4.6 Qualitative Study
To better understand the effectiveness of the proposed RPO module, we illustrate the process of prompt optimization in dialogue scenario in Fig. 3.
The example on the right is from the Swing-14 /4 sequence, where a person is depicted mid-air, gripping a horizontal bar, which is the tracking target. However, the dataset's manual annotation describes it as "swing swinging above a man in black pants". Without context, even humans might struggle to identify the target using this description. Initially, the MLLM, drawing from its extensive knowledge, accurately describes the object as a "Trapeze bar". Yet, due to the knowledge gap between the dision-languse tracker and the MLLM, the tracker fails to locate the target. After receiving the feedback. the MLLM rephrases its output with simple terms like "person" " and "bar". It then uses the semantic context of "hanging" to assist the tracker in target identification. At this stage, the tracker can approximately locate the target. However, since the IoU is below the preset threshold of 0.4, it sends "hanging" back as a positive sample and "person" as a negative sample to the MLLM for further refinement. In the final iteration, the MLLM pinpoints "hands" " as the critical term. This term is both easy to understand and consistently visible on the target throughout the sequence.
4.7 Limitations
When the tracking target is in low resolution or lacks discernible visual features, the MLLM struggles to provide accurate language description of the target. Additionally, accessing the MLLM via API necessitates an internet connection, which may pose challenges in edge deployments due to intermittent unreliable network access.
Temporal changes to target and background are one of the challenges in the visual tracking domain. However, we do not update language descriptions for two key reasons. First, the tracker's predictions are not always accurate, and there are no annotations for background objects, making it harder to generate prompts dynamically. Second, calling MILMs multiple times during tracking to update these prompts adds much computational cost, Achieving good balance between performance and efficiency requires extensive research.

---

Additionally, our ChatTracker focuses solely on the visual features of the tracking target, such as shape, texture, and color, without addressing the impact of different granularities of annotations as discussed in MGIT [14].
5 Conclusion
In this work, we introduced ChatTracker, the first method that utilizes the Multimodal Large Language Model (MLLM) to enhance the performance of visual tracking. We proposed Reflection- -bsed Prompt Optimization (RPO) module to iteratively refine the ambiguous and inaccurate language descriptions of the target with tracking feedback. Moreover, a simple yet effective visual-language tracking framework was proposed to boost the performance of existing trackers as a plug-and-play method. Experimental results on multiple datasets demonstrated that method outperformed state-of-the- art methods. This suggests that incorporating MLLMs into visual tracking had notable effect on improving tracking performance.
Acknowledgements This work was supported by the National Natural Science Foundation of China (62102152, 62072183), and the Shanghai Urban Digital Transformation Special Fund Project (202301027). This work was also sponsored by the Shanghai Frontiers Science Center of Molecule Intelligent Syntheses.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[2] Tom Brown and et. al. Language models are few-shot learners, 2020.
[3] Changgu Chen, Junwei Shu, Lianggangxu Chen, Gaoqi He, Changbo Wang, and Yang Li. Motion-zero: Zero-shot moving object control framework for diffusion-based video generation. arXiv preprint arXiv:2401.10154 2024.
[4] Xin Chen, Houwen Peng. Dong Wang, Huchuan Lu, and Han Hu. Seqtrack: Sequence to sequence learning for visual object tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14572-14581, 2023.
[5] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking, 2021.
[6] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[7] Chenhang Cui, Yiyang Zhou, Xinyu Yang. Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination gpt-4v(ision): Bias and interference challenges. ArXiv, abs/231 1. 003287, 2023.
[8] Yutao Cui, Cheng Jiang. Limin Wang, and Gangshan Wu. Mixformer: End-to-end tracking with iterative mixed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13608-13618, 2022.
[9] Yutao Cui, Tianhui Song, Gangshan Wu, and Limin Wang. Mixformerv2: Efficient fully transformer tracking, 2024.
[10] Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen Huang, Juchuan Liu, et Lasot: high-quality large-scale single object tracking benchmark. International Journal of Computer Vision, 129:439-46 461, 2021.
[11] Vitaly Feng Qi and, Ablavsky, Qinxun Bai, and Stan Sclaroff. Siamese natural language tracker: Tracking by natural language descriptions with siamese trackers. 2021.
[12] Zhi Gao, Yuntao Du, Xintong Zhang, Xiaojian Ma, Wenjuan Han, Song-Chun Zhu, and Qing Li. Clova: closed-loop visual assistant with tool usage and update. arXiv preprint a7X1v22312.10008, 2023.
10

---

[13] Mingzhe Guo, Zhipeng Zhang, Heng Fan, and Liping Jing. Divert more attention to vision- language tracking. Advances in Neural Information Processing Systems, 35:4446 4460, 2022.
[14] Shiyu Hu, Dailing Zhang, wu meiqi. Xiaokun Feng, Xuchen Li, Xin Zhao, and Kaiqi Huang. A multi-modal global instance tracking benchmark (mgit): Better locating target in complex spatio- temporal and causal relationship. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 25007-25030. Curran Associates, Inc., 2023.
[15] Xin Li, Yuqing Huang, Zhenyu He, Yaowei Wang, Huchuan Lu, and Ming-Hsuan Yang. Citetracker: Correlating image and text for visual tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9974-9983, 2023.
[16] Zhenyang Li, Ran Tao, Efstratios Gavves, Cees GM Snock, and Amold WM Smeulders. Tracking by natural language specification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6495-6503, 2017.
[17] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
[18] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303. 05499, 2023.
[19] Ding Ma and Xiangqian Wu. Tracking by natural language specification with long short-term context decoupling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14012-14024 2023.
[20] Yinchao Ma, Yuyang Tang, Wenfei Yang. Tianzhu Zhang, Jimpeng Zhang, and Mengxue Kang. Unifying visual and lision-lagguage tracking via contrastive learning. Proceedings of the AAAI Conference Artificial Intelligence, 38(5):4107-411 Mar. 2024.
[21] Yinchao Ma, Yuyang Tang, Wenfei Yang. Tianzhu Zhang, Jinpeng Zhang, and Mengxue Kang. Unifying visual and vision-language tracking via contrastive learning. 2024.
[22] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoyc, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXw::230.116551, 2023.
[23] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprini 1XXi:22100001882 2022.
[24] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. Track- ingnet: A large-scale dataset and benchmark for object tracking in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[25] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15691-15701, 2023.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. PMLR, 2021.
pages
[27] Yiming Sun, Yang Li, and Changbo Wang. Multi-source templates learning for real-time aerial tracking. In ICASSP 2023 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[28] Gemini Team. Gemini: A family of highly capable multimodal models, 2024.
[29] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Ilama: Open and efficient foundation language models, 2023,
11

---

[30] Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13763-13773, 2021.
[31] Xing Wei, Yifan Bai, Yongchao Zheng, Dahu Shi, and Yihong Gong. Autoregressive visual tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pages 9697-9706, 2023.
[32] Qiangqiang Wu, Tianyu Yang, Ziquan Liu, Baoyuan Wu, Ying Shan, and Antoni B Chan. Dropmae: Masked autoencoders with spatial-attention dropout for tracking tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14561-14571, 2023.
[33] Yuanyou Xu, Zongxin Yang, and Yi Yang. Integrating boxes and masks: A multi-object framework for unified visual tracking and segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[34] An Yan, Yu Wang. Yiwu Zhong, Chengyu Dong. Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3090-3100, 2023.
[35] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking, 2021.
[36] Dawei Yang. Jianfeng He, Yinchao Ma, Qianjin Yu, and Tianzhu Zhang. Foreground- background distribution modeling transformer for visual object tracking. In Proceedings the IEEF/CVF International Conference on Computer Vision, pages 101 10117-10127, 2023.
[37] Kevin Yang, Yuandong Tian, Nanyun Peng, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. arXiv preprint arXiv:2210.06774 2022.
[38] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung- Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(), 2023.
[39] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework, 2022.
[40] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi. Ji Zhang. and Fei Huang. mplug-owl: Modularization empowers large language models with multimodality, 2023.
[41] Chunhui Zhang, Xin Sun, Yiqian Yang, Li Liu, Qiong Liu, Xi Zhou, and Yanfeng Wang. All in one: Exploring unified vision- language tracking with multi-modas alignment. Proceedings of the 31st ACM International Conference on Multimedia, pages 5552-5561, 2023.
[42] Huanlong Zhang, Jingchao Wang, Jianwei Zhang, Tianzhu Zhang, and Bineng Zhong. One- stream iision-language memory network for object tracking. IEEE Transactions on Multimedia, 26:1720-177 2024.
[43] Haojie Zhao, Dong Wang, and Huchuan Lu. Representation learning for visual object tracking by masked appearance transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 18705, 2023.
[44] Haojie Zhao, Bin Yan, Dong Wang, Xuesheng Qian, Xiaoyun Yang, and Huchuan Lu. Effective local and global search for fast long-term tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, (551))460-474 2023.
[45] Yaozong Zheng, Bineng Zhong, Qihua Liang, Guorong Li, Rongrong Ji, and Xianxian Li. Towards unified token learning for vision-language tracking. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
12

---

[46] Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23151-23160, 2023.
[47] Xinyu Zhou, Pinxue Guo, Lingyi Hong, Jinglun Li, Wei Zhang, Weifeng Ge, and Wenqiang Zhang. Reading relevant feature from global representation memory for visual object tracking. In Thirty-seet seventh Conference Neural Information Processing Systems, 2023.
[48] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
13

---

Appendix
A Broader Impact
In this paper, we introduced ChacTracker, efficient and precise tracking framework that integrates MLLM. The remarkable efficiency and effectiveness ChatTracker enable its seamless integration into monitoring systems for unauthorized observations. The proposed PRO Module, through the LLM feedback mechanism, bridges the knowledge gap between the vision language trackers and the MLLM, allowing the MLLM to better adapt the downstream vision language tracking task. We believe this offers valuable insights for addressing the knowledge gap between visual and textual modalities.
B Analysis on Language Descriptions Generated by ChatTracker.
In this section, we present the specific methods for calculating image-text alignment. For tracking dataset that contains N video sequences: V {11, .2,...., UN }. each video sequence Uk consists of my frames: E2 = (16, ........................ J. corresponding tracking result is represented by Y Each sequence is accompanied by manually annotated textual description T and a foreground description generated by CharTracker, Tj Based on these tracking results, we extract image regions from I form: Cx {C Ct. ...... C }. The text-to-ine to--tiagigge alignment score for the manually annotated descriptions (S.) and the ChatTracker generated descriptions (Se) are computed as follows:
In the k-th sequence, I refers to the j-th frame, and the
M(C)) (U(T))
S.
(11)
N
k=1
N
me
- ((TD)
S. N
(12)
m
I (M(TT)2
Here, ft and Ji represent CLIP's [26] text and image feature extractors, respectively. S. and S. indicate the degree of alignment between textual and the tracking target. Notably, ChatTracker- generated descriptions consistently outperform manually annotated descriptions across three datasets. Such advancements highlight the potential for enhancing future visual- lisuul-language trackers by providing more accurate language descriptions.
When background information is present in the language description, the image-text similarity is lower compared to language descriptions without background information. Therefore, the higher the image-text similarity between the language descriptions and the cropped target, the less background information is included in the language descriptions. The less background information included in the language descriptions indicates that the text quality higher. This aligns with the observation that higher alignment scores (S,) demonstrate the improved capability of ChatTracker in generating descriptions, further reducing background interference.
more
C Visualized Results
Figure 4 shows the visualized results of CiteTracker [15] and Our ChatTracker on two datasets with total of six sequences. It is clear to see that the accuracy of our method greatly outperforms that of CiteTracker.
D Supplimentary Experiments
D.1 More about Foreground Verification module
The Foreground Verification module is crucial to the whole framework and its final performance. To better prove this, we design an additional ablation experiment using ChatTracker-L. on the OTB-Lang dataset. The experiment involves three versions of our trackers: Complete ChatTracker-L uses the
14

---

4
LaSOT-giraffe-15
video
H435
6796
$1006
TNL2K- TNL2K-Xinyangaan Xinr--ngnannggiueeeea... video 12-Done
2652
#605
TNL2K-test 039 COD video 01 done
FLOSS
PL201
F1350
$1387
ChatTracker
CiteTracker
Ground Truth
Figure Visualized results of the proposed algorithm and the CiteTracke method on six challenging sequences with drastic changes. ChatTracker demonstrates superior performance. In contrast, CiteTracker faces difficulties handling these complex sequences.
Table The performance comparison Foreground Verification module random selection module in our framework on OTB-lan -TB-lang Dataset
Method
AUC
P
Norm
ChatTracker-L
71.78
94.25
86.82
ChatTrackk-L Laaddo-mmmmmmmmmmmm-ammmm Laa_arraaaapplllllla sample
42.98
58.74
52.11
ChatTracke-L. upperbound
73.91
96.17
88.61
Foreground Verification module. module.ChatTraacke.. ChatTracker-L with Ground Truth does not the Foreground Verification module but selects the proposal with the highest loU with Ground Truth as the tracking result. This entry is used establish the thcoretical upper bound. ChatTracker-L with Random Selection does not use the Foreground Verification module and randomly selects a proposal as the tracking result. The results are shown in the Table 7. The results show that using the Foreground Verification module significantly outperforms the random selection group. This demonstrates that our Foreground Verification module is effective. Although there may still be room for improvement, our ChatTracker: result is close to the theoretical upper bound.
D.2
More performance comparison
We conduct performance comparison OTB OTB-Lang dataset.
15

---

Table Results the OTB-Lang dataset
Method AUC
P
Norm
CharTracker-I. 71.78
94.25
86.82
CharTracker-B
70.77
92.00
85.29
JointNLT
65.52
86.23
80.41
ARTrack-256
69.90
91.15
84.10
And we also conduct more SoTA comparison with other trackers.
Table 9: More state-of-the-arr comparisons the datasets of TNL2K and LaSOT. And indicates vision-language trackers.
LaSOT
TNL2K
Method
Source
AUC
P NNorm
P
AUC
PNorm
P
ChatTracker-L
Ours
74.1
83.8
81.2
65.4
76.5
70.2
ChatTracker-B
Ours
71.7
77.5
80.9
59.6
76.3
62.1
VLT 7T [13]
NeurIPS2022
67.3
77.6
72.1
53.1
53.3
OVLM-384" [42]
TMM2023
67.7
77.6
74.2
64.7
82.6
69.3
OVLMM-256* [42]
TMM2023
65.6
75.6
71.1
62.5
80.0
66.5
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper's contributions have been clearly listed in sec. and its scope belongs to visual object tracking.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work performed by the author discussed in 4.7.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All the information necessary to reproduce our main experimental results presented in 3, 4.1 and 5.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
18

---

Answer: [Yes]
Justification: We will provide raw results and release code for readers to reproduce the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
ssi ssionoolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The detailed settings of the experiments can be found in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: This paper follows the convention in the visual object tracking field of research and usually the tracking community does include discussion of error bars in the content.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: 4.1 specifies that all our experiments require one NVIDIA 3090 GPU.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https https://neet /ttpe://neeui..eeeeeeee
Answer: [Yes]
Justification: Our research does not involve ethical issues or data. We are sure that the research conducted in the paper fully confirm the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper discuss both positive and negative social impacts of this work A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All the creators or original owners of assets used this paper are properly credited and cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowd sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "0AmlO8cTby", "reviewer": "Reviewer_gPPX", "summary": "The paper proposes a new Visual-Language (VL) tracking framework called ChatTracker, that integrates MLLMs into VL tracking through iterative refinement of text prompts for VL trackers. The text prompts optimized using the proposed Reflection-based Prompt Optimization (RPO) module are more accurate than manual text annotations in the datasets and improve the tracking performance by taking into account both foreground and background objects.", "questions": "- The potential limitations mentioned in the weaknesses should be addressed or discussed in the paper. Temporal changes to target and background are the main challenges in visual tracking and the proposed framework does not add any mechanisms to address these even though it might heavily depend on GVLM results for its answers.\n- Related to the above point: Have the authors done any experiments where foreground/background prompts were updated while the video is being processed? What were the insights?\n- As mentioned above, how often does the visual tracker proposal get picked over a GVLM proposal and vice versa when both exist in the pool going into the Foreground module? Or does the visual tracker proposal mostly get used when there are no GVLM proposals?\n- Related works section can be expanded a bit. An important motivation for this paper is the disadvantages brought by the manual textual annotations in the VOT benchmarks (that have them). For the broader audience, the authors could add a short section on related benchmarks and if they have manual text annotations, explain how those came about (why are they low quality?). Line [300-301] mentions the prompt “swing swinging above a man in black pants” - which almost sounds like a generated text prompt, having this context in the paper itself would be helpful.\n- Equation (6) - What’s the purpose of $T^i_{pos}$ in this equation? \n- Lines [152-153] - What happens if a word matches to no proposals? Can we add that to negative words? Why not?\n- Lines [204-205] - To confirm, the anchor sample is always the target template, but do you sample positive samples from tracker result or foreground proposal? Visual tracking is mentioned in Line 202, but not in the Appendix.\n- Line [221] - The IoU threshold for especially foreground seems a little low, what’s the reason for this? Do you have any statistics for what the accepted IoUs are from the foreground verification? Is it generally lower with the proposals?\n- Please use $P_{norm}$ or NP consistently across tables. Also, for a broader audience you should mention what the metrics are even if it's in a couple sentences.\n- Some tables report NP and P, others one or the other. It would be better to report consistently even if it’s just across smaller tables. Moreover, baseline tracker results in Table 3 are not matching the same tracker’s results in Table 1. If I’m missing some details, please let me know.\n- Fig. 3 - not seeing any blue boxes in the image, even though CiteTracker is in the legend.", "limitations": "Authors did address some limitations of their method in the Appendix. Please see my discussions about other potential limitations in the weaknesses and questions sections above.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies a clear gap for Visual-Language trackers and how manually annotated or naively generated text prompts can provide sub-optimal results for language-assisted visual tracking.\n- The proposed framework combines the power of a strong MLLM like GPT-4V and a GVLMs like Grounding DINO-T to formulate the iterative refinement procedure for generating better text prompts for tracking tasks. It uses a GVLM powered by the optimized prompts in conjunction with a visual tracker to get a set of proposals. It also incorporates GVLM generated region proposals to train its Foreground Verification module, which not only does foreground classification but scores candidate regions with how they overlap with background proposals. While different parts of the framework are pre-existing and straight-forward, the paper uses them in a new way to improve visual tracking performance in relevant benchmarks.\n- The method is plug-and-play, and any visual tracker (regardless of whether it is a VL tracker or not) can be used as the baseline tracker in the Semantic Tracking Module of the framework.\n- The experimental results and ablations support the claims in the paper.\n- The paper is well-written, well-organized and clear.", "weaknesses": "- The proposed framework does not take into account the potential temporal changes in the video that might require a more optimal foreground/background text prompt to proceed. For example, in Fig. 3 we see the framework landing on ‘hands, bar’ as the positive prompt, which focuses on hands being attached to the bar, which may not be true for the rest of the video. The background text of the video is also never updated. In fact, neither the GVLM that the framework heavily depends on nor the Foreground Verification module have any temporal components. Therefore, the framework would have to heavily rely on the baseline tracker in this aspect.\n- The quality of the region proposals coming out of the Semantic Tracking Module is un-addressed. How often GVLM proposals are preferred to the visual tracker results (unless the prompt optimization fails as mentioned in Appendix A)? How dependent is the final performance on how well the Foreground Verification module is performing (since it’s only been trained on the LaSOT training set)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1.  The potential limitations mentioned in the weaknesses should be addressed or discussed in the paper.  Have the authors done any experiments where foreground/background prompts were updated while the video is being processed? What were the insights?***\n\nThanks for pointing this out. Temporal changes to target and background are indeed one of the main challenges in the visual tracking domain. Our ChatTracker relies on a baseline tracker to consider potential temporal changes in the video. However, compared to traditional visual trackers, which rely only on a single image template to locate the target, **natural language provides more temporal-invariant features**, such as object categories and textures. Our ChatTracker incorporates MLLMs and designs a novel Reflection-based Prompt Optimization (RPO) module to obtain accurate target descriptions, providing better robustness when facing temporal changes.\n\nUpdating foreground and background prompts is not straightforward for two main reasons:\n\n1. **Accuracy of Predictions**: During tracking, the tracker's predictions of the target are not always accurate, and there are no annotations of background objects. This makes it difficult to dynamically generate foreground and background prompts.\n2. **Trade-off Between Performance and Efficiency**: If MLLMs are called multiple times during tracking to update foreground and background prompts, it incurs a certain computational overhead (although this overhead is decreasing as technology advances). Achieving a good balance between performance and efficiency requires extensive research.\n\nWe intend to delve into this challenge as part of our future work, and to address this, we have incorporated a discussion in the limitations section.\n\n\n\n\n\n***Q2.  How often does the visual tracker proposal get picked over a GVLM proposal?  Or does the visual tracker proposal mostly get used when there are no GVLM proposals?*** \n\nA  GVLM proposal gets picked 33.27% of the all time in our experiment. We analyzed 685,360 frames in the LaSOT dataset **[1]**, of which 228,042 used the GVLM proposal as the result.\n\n\n\n\n\n***Q3. How dependent is the final performance on how well the Foreground Verification module is performing (since it’s only been trained on the LaSOT training set)?***\n\nThanks for pointing this out. The Foreground Verification module is crucial to the whole framework and its final performance. To better answer your question, we design an additional ablation experiment using ChatTracker-L on the OTB-Lang dataset[2]. The experiment involves three versions of our trackers:\n\n1. **Complete ChatTracker-L**: Uses the Foreground Verification module.\n2. **ChatTracker-L with Ground Truth**: Does not use the Foreground Verification module but selects the proposal with the highest IoU with Ground Truth as the tracking result. This entry is used to establish the theoretical upper bound.\n3. **ChatTracker-L with Random Selection**: Does not use the Foreground Verification module and randomly selects a proposal as the tracking result.\n\nThe results are shown in the table below. The results show that using the Foreground Verification module significantly outperforms the random selection group. This demonstrates that our Foreground Verification module is effective. Although there may still be room for improvement, our ChatTracker-L result is close to the theoretical upper bound. Thank you for your suggestion. We will discuss and add this in the supplementary.\n\n**Table.1  The performance comparison of Foreground Verification module ,random selection module in our framework on Lasot Dataset** \n\n| Method| $AUC$ | $P$| $P_{Norm}$ |\n|-|-| - |-|\n| ChatTracker-L| 71.78 | 94.25 | 86.82 |\n| ChatTracke-L_random_sample | 42.98 | 58.74 | 52.11 |\n| ChatTracke-L_upperbound | 73.91 | 96.17 | 88.61 |\n\n\n\n\n\n***Q4 Related works section can be expanded a bit.***\n\nThanks for your suggestions. We will add a section in the revised manuscript to introduce related benchmarks for broader audiences to understand the insights. This section will include whether the LaSOT, TrackingNet, TNL2K, and other benchmarks contain text annotations, the methods used for text annotation, and potential reasons for low quality.\n\n\n***Q5 What’s the purpose of  $T_{pos}^i$ in equation  (6) ?***\n\nIn equation (6): \n\n $ T_{neg}^i = \\\\{  w_m^i|for\\ all\\ n\\ that\\ S_z^{nm}>\\theta_2 \\wedge IoU(P_z^n,G)<\\theta_3 \\\\}  \\backslash T_{pos}^i$\n\nThe purpose of    $T_{pos}^i$  is to ensure that a word does not appear in both  $T_{pos}^i$  and $T_{neg}^i$ . \n\n$T_{pos}^i$ and $T_{neg}^i$  are used to construct a reflection prompt during the iterative optimization process to help the MLLM generate better language descriptions. If a word is present in both$T_{pos}^i$  and $T_{neg}^i$ , it may confuse the MLLM and prevent it from generating an effective response.\n\n***Q6 Lines [152-153] - What happens if a word matches to no proposals? Can we add that to negative words? Why not?***\n\nIf a word matches no proposals, it will be ignored and not added to negative words.\n\n $T_{neg}^i$ is used to inform the LLM that the GVLM has associated this word with objects in the background. The LLM will further enable the GVLM to identify the correct target based on this information. \n\nWhen a word matches no proposals, it is considered meaningless (i.e., it doesn't contain any information that helps the LLM distinguish the target), so we do not add it to  $T_{neg}^i$ .\n\nFor example, in the swing-17 example in Fig. 2, when the text description is “Human on a swing seat,” \"Human\" and \"Swing\" are categorized as negative words, and \"seat\" is categorized as positive. They all correspond to image regions in the figure. However, \"on\" and \"a,\" which match no proposals, do not provide effective feedback to the MLLM. Therefore, we do not add them to negative words.\n\nDue to the word limit in the rebuttal, we provide the remaining responses  in the following comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ijbgf3HgX3", "reviewer": "Reviewer_aCWh", "summary": "The paper proposes a novel Multimodal Large Language Model framework to improve the vision-language visual tracking performance.\nBy introducing the reflection-based prompt optimization module, the tracking prompt can be iteratively refined via tracking feedback.\nThe proposed method shows state-of-the-art results compared to prior vision-language trackers and visual trackers.", "questions": "Please refer to the weakness.", "limitations": "The authors have mentioned the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper investigates the multimodal large language models for the visual tracking problem.\n- By showing the limitations of prompts created by MLLM and manual annotation for visual tracking, the paper introduces a novel iterative prompt generation via chatting and semantic prompt verification.\n- The paper shows adequate evaluation comparing the proposed method with both vision-language trackers and visual trackers.", "weaknesses": "The paper should illustrate the number of chat iterations in the reflection-based prompt optimization module and their performance effect. The example in Figure 5 shows that the module needs 2 iterations to get an accepted prompt but the overall analysis should be considered, especially in the images with complex scenes and objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1:   The paper should illustrate the number of chat iterations in the reflection-based prompt optimization module and their performance effect.  The example in Figure 5 shows that the module needs 2 iterations to get an accepted prompt but the overall analysis should be considered, especially in the images with complex scenes and objects.***\n\n-  The paper should illustrate the number of chat iterations in the reflection-based prompt optimization module.\n\nThanks for your advice. The iterative refinement process averages **2.3** rounds. We illustrated the number of chat iterations in Section 4.1 (Line 229 - 230).\n\n-  The example in Figure 5 shows that the module needs 2 iterations to get an accepted prompt but the overall analysis should be considered, especially in the images with complex scenes and objects.\n\nWe conducted experiments using ChatTracker-B to study the performance effect of the maximum number of iterations. The results show that when the maximum number of iterations is less than 10, the tracker performance improves with an increase in iterations. When the number of iterations exceeds 10, the improvement in tracker performance becomes less significant.  In our experiments, the default maximum number of iterations is set to 20.\n\nFor complex scenes and objects, we will add more visual examples in the revised manuscript to help our readers understand the impact of iteration numbers on tracker performance.\n\n**Table.1  Impact of the number of iteration rounds on tracker performance on Lasot[1] and TNL2k[2] datasets**\n\n| Number of Maximum Iterations | 1     | 10    | 20    | 30    |\n| ---------------------------- | ----- | ----- | ----- | ----- |\n| LaSOT AUC                    | 64.87 | 67.24 | 67.89 | 67.91 |\n| TNL2K AUC                    | 53.66 | 55.63 | 56.39 | 56.42 |\n\n[1]Heng Fan, Hexin Bai, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Harshit, Mingzhen  Huang, Juehuan Liu, et al. Lasot: A high-quality large-scale single object tracking benchmark.  International Journal of Computer Vision, 129:439–461, 2021.\n\n[2]Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng  Wu. Towards more flexible and accurate object tracking with natural language: Algorithms  and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern  Recognition, pages 13763–13773, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wnBMlgAv8E", "reviewer": "Reviewer_z5a5", "summary": "This paper proposes ChatTracker, a novel framework that leverages MLLMs for visual object tracking. The Reflection-based Prompt Optimization (RPO) module can narrow the knowledge gap between the VL tracker and the MLLM. ChatTracker also achieves SoTA performance on several tracking datasets.", "questions": "Please refer to weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper designs a new framework that leverages MLLMs for visual object tracking. The entire framework's process is clear and easy to follow.\n2. The approach performs favorably against the state-of-the-art methods on several datasets.", "weaknesses": "1. SOTA trackers are missing in Table 1, such as OVLM (TMM23), MMTrack(TCSVT23), etc. There is a discrepancy between the data from UVLTrack in Table 1 and the data provided by the official source.\n2. I want to know the model's performance on MGIT (NeurIPS '23) and OTB99_Lang (CVPR '17) to verify the generalizability of the method in different scenarios (short-term tracking and global instance tracking).\n3. Why is it necessary to design ChatTracker-L and ChatTracker-B, but use different backbone networks? Also, I believe comparing the L model in Table 1 is unfair, as UVLTrack provides the L model, and annotations are needed for models of different scales.\n4. For analysis of language descriptions generated by ChatTracker, if the natural language description includes background information, is it reasonable to only crop out the target for calculating image-text similarity?\n5. I believe the phrase \"via Chatting with Multimodal Large Language Model\" in the title overstates the contribution of the entire framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: SOTA trackers are missing in Table 1, such as OVLM[1], MMTrack[2].***\n\nThank you for your valuable suggestion. We will include comparisons with OVLM and MMTrack in the revised manuscript. Additionally, we have found that compared to their variants with highest performance, our ChatTracker demonstrates better tracking performance. Furthermore, our method does not require manually annotated target text for initialization.\n\n**Table.1** Complete comparison of OVLM and MMTrack with ChatTracker's performance on LaSOT[3] and TNL2K[4] Datasets.\n\n| Method| LaSOT | | TNL2K | |\n|-|-|-|-|-|\n| | $AUC$ | $P$    | $AUC$   | $P$ |\n| MMTrack | 70.0  | 75.7 | 58.6  | 59.4 |\n| OVLM-256 | 65.6  | 71.1 | 62.5  | 66.5 |\n| OVLM-384 | 67.7  | 74.2 | 64.7  | 69.3 |\n| ChatTracker-B | 71.7  | 80.9 | 59.6  | 62.1 |\n| ChatTracker-L | 74.1  | 81.2 | 65.4  | 70.2 |\n\n***Q2:  I want to know the model's performance on MGIT[5] and OTB99_Lang[6].***\n\nThank you for your valuable suggestion. We evaluated ChatTracker-B on MGIT and OTB99_Lang.\n\n- **For the MGIT dataset**, we used only the first frame's bounding box (BBOX) for initialization and used ChatTracker's self-generated language descriptions for tracking. For the comparison method JointNLT[7], we selected Mechanism A (i.e., the first frame's BBOX and the annotated text in the dataset) for comparison. For ARTrack-256[8], we used only the first frame's BBOX for initialization.\n\n**Table.2**. Results on the MGIT dataset\n\n| Tracker | **Normalized Precision (N-PRE)** | **Precision (PRE)** | S**uccess Rate (SR_IoU)**. |\n|-|-|-|-|\n| ChatTracker-B | 0.833 | 0.691  | 0.801 |\n| JointNLT      | 0.786 | 0.445  | 0.610 |\n| ARTrack-256   | 0.711 | 0.516   | 0.599 |\n\n- **For the OTB99_Lang dataset,** our ChatTracker used only the first frame's bounding box (BBOX) for initialization and utilized ChatTracker's self-generated language descriptions for tracking.  The settings for JointNLT and ARTrack-256 are the same as those for MGIT.\n\n**Table.3**: Results on the OTB99_Lang dataset\n| Tracker    | $AUC$   | $P$     | $P_{Norm}$    |\n|-|-|-|-|\n| ChatTracker-B | 70.77 | 92.00 | 85.29 |\n| JointNLT      | 65.52 | 86.23 | 80.41 |\n| ARTrack-256   | 69.90 | 91.15 | 84.10 |\n\nThe results show that, thanks to more accurate text descriptions and the proposed vision-language tracking framework, our ChatTracker outperforms existing methods. We will include these results in the revised manuscript.\n\n***Q3 Why is it necessary to design ChatTracker-L and ChatTracker-B, but use different backbone networks?*** \n\n- For the design of ChatTracker-L and ChatTracker-B:\n\nChatTracker-L is designed for better performance, while ChatTracker-B is designed to achieve a better trade-off between accuracy and speed. Here, L stands for Large, and B stands for Base.\n\n- For the  use of different backbone networks:\n\nAt the time of submission, ARTrack[1] had not released the training code and model weights for ARTrack-L384, so we use the Mixformer-L for ChatTracker-L. \n\n***Q4: There is a discrepancy between the data from UVLTrack[9] in Table 1 and the data provided by the official source. Also, I believe comparing the L model in Table 1 is unfair, as UVLTrack provides the L model, and annotations are needed for models of different scales.***\n\nThank you for pointing this out.  In Table 1, we reported the results for UVLTrack-B. As shown in the table below, our AUC scores on LaSOT, TrackingNet, and TNL2K are higher than UVLTrack-L. For other trackers in Table 1, we selected the variant with the highest performance for a fair comparison. We will replace the comparison results of UVLTrack-B in Table 1 with the UVLTrack-L version and annotate the scales of the compared methods in the revised manuscript. We have included the revised Table.1 in the uploaded PDF file.\n\n**Table.4 Comparison results of UVLTrack-L and ChatTracker on the LaSOT, TrackingNet, and TNL2K datasets**\n\n| | LaSOT |      |  | TrackingNet | |  | TNL2K |      |\n|-|-|-|-|-|-|-|-|-|\n|   | $AUC$ | $P$  |  | $AUC$ | $P$  |  | $AUC$ | $P$  |\n| UVLTrack-L    | 71.3  | 78.3 |  | 84.1  | 82.9 |  | 64.8  | 68.8 |\n| ChatTracker-L | 74.1  | 81.2 |  | 86.1 | 86.0 |  | 65.4  | 70.2 |\n\n\n***Q5:  if the natural language description includes background information, is it reasonable to only crop out the target for calculating image-text similarity?***\n\nIt is reasonable to only crop out the target for calculating image-text similarity. \n\nWhen background information is present in the language description, the image-text similarity is lower compared to language descriptions without background information. Therefore, the higher the image-text similarity between the language descriptions and the cropped target, the less background information is included in the language descriptions. The less background information included in the language descriptions indicates that the text quality is higher.\n\nTherefore, in our experiments, cropping the target from each frame and calculating its text-to-image similarity is reasonable. We will clarify this in the revised manuscript.\n\n***Q6:  I believe the phrase \"via Chatting with Multimodal Large Language Model\" in the title overstates the contribution of the entire framework.***\n\n \"via Chatting with Multimodal Large Language Model\"  refers to the iterative process where the MLLM generates language descriptions, and the GVLM provides feedback on the quality of these descriptions to the MLLM, helping it generate better language descriptions. This process is akin to chatting with the MLLM. We think this phrasing helps our readers better understand our method intuitively. \n\nWe are open to changing the title if there is a more suitable one. We would appreciate it if you could give more detailed suggestion or more explanation about how does the current one overstate the contribution.\n\nDue to the word limit in the rebuttal, we provide the remaining citations in the following comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4hfovKf9nT", "reviewer": "Reviewer_Bwy6", "summary": "This paper introduces a simple yet effective VL tracking framework based on a multimodal large language model, called ChatTracker. The main idea is to utilize the rich world knowledge in multimodal large language model to generate high quality language descriptions and improve tracking performance. Specifically, a reflection-based prompt optimization module is proposed to iteratively refine the ambiguous and inaccurate descriptions of the target with tracking feedback. A large number of experiments demonstrate the effectiveness of the proposed method.", "questions": "- It is an interesting idea to apply large language models to the visual tracking field. The general practice would be to do fine-tuning operations on the downstream data so that the large language model is more adapted to the downstream task. I have a question, does the proposed approach require fine-tuning in the downstream tracking dataset?", "limitations": "The limitations of the method have been accounted for in this paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.the paper is well written and can be easily understood by the reader.\n\n2.The proposed tracker can achieve SOTA performance and adequate ablation experiments are accomplished.", "weaknesses": "- Line 127 mentions only the initial bounding box and search frame as inputs and does not mention template frames as inputs. However, the equation in line 128 writes template as input.\n\n- In section 4.1, MixFormer-L and ARTrack-256 as visual coders for ChatTracker-L and ChatTracker-B? I have a question, why not use mixformer-B/L or ARTrack-B/L uniformly as visual encoders. Could the authors please provide more results for mixformer-B and ARTrack-L for performance comparison.\n\n- In section 4.3 and Table 2, when applying the proposed module to other visual trackers, the paper does not describe how the linguistic description is fused with the visual features, could the authors please explain this process specifically.\n\n- The LaSOT_ext and OTB_Lang test datasets are also included in the VL tracking task, and the authors are asked to provide performance on these datasets to help enrich the experimental results in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Q1: Line 127 mentions only the initial bounding box and search frame as inputs and does not mention template frames as inputs. However, the equation in line 128 writes template as input.***\n\nThanks for your careful reading.\n\nIn the equation  $P^{t}_{VT} = \\mathcal{F}_V$$_T$$(I^{t};I^{1}, G) $ ,  $I^t$  represents the image of the t-th frame, $G$ represents the initial bounding box, and $I^1$ is the first frame of the video with  $G$ as the initial bounding box.\nWe will clarify this in the revised manuscript.\n\n***Q2: In section 4.1, MixFormer-L and ARTrack-256 as visual coders for ChatTracker-L and ChatTracker-B? I have a question, why not use mixformer-B/L or ARTrack-B/L uniformly as visual encoders. Could the authors please provide more results for mixformer-B and ARTrack-L for performance comparison.***\n\nThanks for your question.\n \nAt the time of submission, ARTrack[1] had not released the code and model weights for ARTrack-L384, so we did not use ARTrack-L as the visual encoder for ChatTracker-L.The choice of MixFormer-L[2] for ChatTracker-L leads to high accuracy, and ARTrack-B for ChatTracker-B provides a better trade-off between accuracy and speed.\n\nWe provide more results for MixFormer-B and ARTrack-L for performance comparison. In the table below, results of visual trackers with the integration of ChatTracker are marked by *.\n\n**Table.1  The comparison of results forVision Language trackers using ChatTracker-generated text  (marked by\\*). The speed was tested on the same device.**\n\n|                           | LaSOT[3] |       |       | TNL2K[4] |       |       | Speed (fps) |\n| ------------------------- | ----------- | ----- | ----- | ------------ | ----- | ----- | ----------- |\n| Choice for visual tracker | $AUC$ | $P$  | $P_{Norm}$    | $AUC$  | $P$     | $P_{Norm}$    |             |\n| Mixformer-B               | 69.15        | 74.70 | 78.70 | 55.12        | 55.17 | 71.21 | 57          |\n| Mixformer-B*              | 70.43        | 76.27 | 80.28 | 57.38        | 57.93 | 73.94 | 44          |\n| Mixformer-L               | 70.1         | 76.3  | 79.9  | 60.55        | 63.01 | 76.16 | 27          |\n| Mixformer-L*              | 74.1         | 83.8  | 81.2  | 65.41        | 70.20 | 83.25 | 20          |\n| ARTrack-256               | 70.77        | 76.23 | 79.54 | 58.09        | 59.90 | 74.33 | 51          |\n| ARTrack-256*              | 71.68        | 77.50 | 80.92 | 59.63        | 62.06 | 76.27 | 48          |\n| ARTrack-L384              | 73.49        | 80.56 | 82.34 | 60.58        | 64.36 | 77.25 | 22          |\n| ARTrack-L384*             | 74.65        | 82.07 | 84.71 | 62.01        | 65.65 | 78.93 | 18          |\n\n\n***Q3: In section 4.3 and Table 2, when applying the proposed module to other visual trackers, the paper does not describe how the linguistic description is fused with the visual features, could the authors please explain this process specifically.***\n\nIn fact, we do not fuse the visual tracker into our ChatTracker at the feature level. When replacing visual trackers, we incorporate the different tracking results of the corresponding visual tracker, $P^t_{VT}$, into $P^t_{fore}$ as supplemental foreground proposals. At the same time, the foreground proposals generated by the GVLM using linguistic descriptions do not change.\n\nWe will provide a detailed description of how we replace the other visual tracking methods in the revised manuscript and we will release our code to help our readers better understand this process.\n\n\n***Q4: The LaSOT_ext and OTB_Lang test datasets are also included in the VL tracking task, and the authors are asked to provide performance on these datasets to help enrich the experimental results in this paper.***\n\nThank you for your suggestion. Our performance on these two datasets is as follows, demonstrating that our method also achieves SOTA results on these datasets. \n\n**Table.2  Performance comparison of ChatTracker, UVLTrack, and ARTrack on the LaSOT_ext and OTB_Lang datasets.**\n\n|               | LaSOT_ext |       |       | OTB_Lang[5] |       |       |\n| ------------- | --------- | ----- | ----- | --------------- | ----- | ----- |\n|               | $AUC$       | $P$     | $P_{Norm}$    | $AUC$             | $P$     | $P_{Norm}$    |\n| ChatTracker-L | 56.06     | 64.84 | 68.18 | 71.78           | 94.25 | 86.82 |\n| ChatTracker-B | 53.72     | 62.09 | 65.13 | 70.77           | 92.00 | 85.29 |\n| UVLTrack-L    | 51.21     | 59.00 | 62.30 | 71.89           | 93.21 | 87.77 |\n| ARTrack-L384  | 52.8      | 59.7  | 62.9  | 71.66           | 92.80 | 85.78 |\n| ARTrack-256   | 48.36     | 53.73 | 57.69 | 69.90           | 91.15 | 84.10 |\n\n***Q5: Does the proposed approach require fine-tuning in the downstream tracking dataset?***\n\nNo, we don't need to fine-tune MLLM in the downstream tracking dataset. We use the MLLM reflection to narrow the knowledge gap between the VL tracker and the MLLM. This allows ChatTracker to achieve favorable performance without fine-tuning. \n\nThanks for your suggestion. We will investigate the fine-tuning version of our proposed framework in future work.\n\n\n[1]Xing Wei, et al.  Autoregressive visual  tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern  Recognition,2023.\n\n[2] Yutao Cui, et al.  Mixformer: End-to-end tracking with  iterative mixed attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and  Pattern Recognition,2022.\n\n[3]Heng Fan, et al.  Lasot: A high-quality large-scale single object tracking benchmark.  International Journal of Computer Vision, 2021.\n\n[4]Xiao Wang, et al.  Towards more flexible and accurate object tracking with natural language: Algorithms  and benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern  Recognition, 2021.\n\n[5]Zhenyang Li, et al.   Tracking by natural language specification. In Proceedings of the IEEE conference on computer  vision and pattern recognition, 2017", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear All Reviewers,

Thank you again for taking the time to review our work and for providing us with valuable feedback.

We are excited that you found our results impressive (gPPX, z5a5, Bwy6) and our experiments well-designed (gPPX, aCWh, Bwy6), appreciate the innovative uses of multimodal large language models for visual tracking (gPPX, aCWh, z5a5), and found our paper well-written (gPPX, Bwy6). We will open-source the code to facilitate a better understanding of our method. The revised Table 1 and Figure 3 are included in the PDF submitted with this response.

We have carefully considered your comments and have provided our responses. If you have any further questions or require additional clarification, please kindly let us know.

Thank you again for your valuable input.

Best wishes,
Authors

Meta-review: The rebuttal provided clarifications about the proposed method and its analysis that were useful for assessing the paper's contribution and responded adequately to most reviewer concerns. All reviewers recommend acceptance after discussion (with two borderline accepts, one weak accept and one accept), and the ACs concur. The final version should include all reviewer comments, suggestions, and additional clarifications from the rebuttal.
Decision: Accept (poster)