DTVLT: A Multi-modal Diverse Text Benchmark for Visual Language Tracking Based on LLM

## Abstract

Abstract Visual language tracking (VLT) has emerged as a cutting-edge research area, harnessing linguistic data to enhance algorithms with multi-modal inputs and broadening the scope of traditional single object tracking (SOT) to encompass video understanding applications. Despite this, most VLT benchmarks still depend on succinct, human-annotated text descriptions for each video. These descriptions often fall short in capturing the nuances of video content dynamics and lack stylistic variety in language, constrained by their uniform level of detail and a fixed annotation frequency. As a result, algorithms tend to default to a “memorize the answer” strategy, diverging from the core objective of achieving a deeper understanding of video content. Fortunately, the emergence of large language models (LLMs) has enabled the generation of diverse text. This work utilizes LLMs to generate varied semantic annotations (in terms of text lengths and granularities) for representative SOT benchmarks, thereby establishing a novel multi-modal benchmark. Specifically, we (1) propose a new v isual l anguage t racking benchmark with d iverse t exts, named DTVLT , based on five prominent VLT and SOT benchmarks, including three sub-tasks: short-term tracking, long-term tracking, and global instance tracking. (2) We offer four granularity texts in our benchmark, considering the extent and density of semantic information. This is achieved through DTLLM-VLT, a method for generating high-quality, diverse text by leveraging the extensive knowledge base of LLMs to produce descriptions rich in world knowledge. We expect this multi-granular generation strategy to foster a favorable environment for VLT and video understanding research. (3) We conduct comprehensive experimental analyses on DTVLT, evaluating the impact of diverse text on tracking performance and hope the identified performance bottlenecks of existing algorithms can support further research in VLT and video understanding. The proposed benchmark, experimental results, and toolkit will be released gradually on http://videocube.aitestunion.com/.

## 1 Introduction

Single object tracking (SOT) is a pivotal task in computer vision, designed to follow a single moving object throughout a video sequence. Researchers have observed that the effectiveness of most trackers often diminishes when tracking objects in lengthy videos with intricate content. Moreover, relying solely on visual cues significantly hinders the flexibility of these trackers.

Consequently, there has been a pronounced trend in recent studies to integrate semantic annotations into SOT, leading to the development of the visual language tracking (VLT) task. This new task is advantageous as it extends the potential applications of SOT, including advancements in video understanding.
Utilizing natural language in place of bounding boxes (BBox) provides a more user-friendly and intuitive alternative. This method facilitates more precise representations of objects, encompassing both their spatial positioning and intricate semantic attributes, thereby enhancing the efficacy of the tracking process.

Figure: Figure 1: Comparison of DTVLT and other VLT benchmarks. (a-c) Examples of video content and semantic descriptions on OTB99_Lang (Li et al. ()), LaSOT (Fan et al. ()), and MGIT (Hu et al. ()). The green bounding box (BBox) indicates ground truth, while the red dashed BBox indicates other objects that satisfy the semantic description. (a) and (b) are sequences with simple narrative content. And their semantic annotations mainly describe the first frame, which may misguide the algorithm bacause of misleading text and multiple qualified target. In the VLT task, if the error caused by incorrect text accumulates for the tracker, it will have an irreversible impact on the tracking results. (c) in MGIT has such complex text that they are not conducive to algorithmic learning. (d) An example of the multi-granular generation strategy used by DTVLT. We provide more diverse concise and detailed descriptions for each hundred frames of the object to be tracked, covering five representative datasets across three mainstream tracking tasks. The term “# xx” represents the frame ID. Compared to existing benchmarks, the generated text provides more prosperous and flexible information to portray long videos.
Refer to caption: extracted/5913655/imgs/motivation-v2.png

When defining the VLT task, researchers integrate text annotations from two primary perspectives:
(1) Short text annotations. Representative VLT benchmarks such as OTB99_Lang (Li et al. ()), TNL2K (Wang et al. ()), and LaSOT (Fan et al. (; )) primarily employ short text.
This straightforward approach is clear and easy to understand, aiding in the learning and comprehension by VLT trackers. However, these methods are susceptible to vague semantic descriptions and potential ambiguities.
For instance, as depicted in Fig. 1 (a) and (b), the description captures only the object’s initial state. As the object moves, the positional constraint in the semantic information can become misleading, making the semantic descriptions restrictive over time.
(2) Long text annotation. MGIT (Hu et al. ()) adopts a multi-granular semantic annotation strategy aimed at providing more precise semantic descriptions.
This method stands out from other benchmarks with two key features: extended text lengths and periodic updates, transitioning from simple to dense and detailed descriptions.
Nonetheless, this approach encounters challenges such as the time-consuming nature of text annotation and the necessity for algorithms capable of robust text processing and multi-modal alignment. As shown in Fig. 1 (c), the text in MGIT can be excessively lengthy and complex.

Clearly, while the intent of these studies is to extend the SOT task into a multi-modal one to improve tracking performance, the singular granularity used in most research not only impedes algorithms from achieving the desired results but also complicates VLT research.
Therefore, a superior approach to constructing a VLT benchmark would be to move beyond offering a mere natural language description for short videos. Instead, it would involve devising a systematic method to supply multi-granular texts that support trackers in understanding various video contents.

Offering a variety of environmental texts—including short, long, sparse, and dense formats—and evaluating algorithm performance across these descriptions allows us to effectively identify the strengths and weaknesses of methods under various semantic granularities. This insight can guide the improvement of multi-modal algorithms.
What excites us is the potential of large language models (LLMs) to aid in reaching this objective. By integrating the LLM seamlessly into the process of text generation, we can create a diverse multi-modal environment that is favorable for VLT research.

Our work is motivated by the aforementioned considerations and aims to construct a new VLT benchmark named DTVLT. This benchmark leverages the DTLLM-VLT (Li et al. ()) method, which employs LLM to generate a wide variety of texts for tracking datasets.
Specifically, we integrate text length and generation density to create four distinct levels of granularity. With this framework, we have selected a range of VLT trackers for experimental analysis to assess how diverse texts affect algorithmic performance. The experimental results not only illustrate that this diversified setting can support detailed evaluation and analysis of algorithmic capabilities but also indicate the potential for future improvements in the multi-modal learning capabilities of trackers by using generated data.

Contributions.
(1) We propose a new VLT benchmark named DTVLT based on five prominent VLT and SOT benchmarks including three tracking tasks: short-term tracking, long-term tracking, and global instance tracking.
(2) We offer four granularity combinations for our benchmark, considering the extent and density of semantic information using DTLLM-VLT, which leverages LLM to generate diverse high-quality language descriptions. We expect this multi-granular generation strategy can provide a favorable environment for VLT and video understanding research.
(3) We conduct comprehensive experimental analyses, evaluating the impact of diverse text on tracking performance and hope the explored performance bottlenecks of existing algorithms can support further VLT research.

## 2 Related Work

Single Object Tracking Benchmark.
The SOT task involves initializing and tracking a specific object within a video sequence. It starts by identifying the object through its bounding box (BBox) in the first frame and then continues to track and locate the object in subsequent frames.
Since 2013, several benchmarks, such as OTB (Wu et al. (; )) and VOT (Kristan et al. (); Bibliographie Goecke et al. (); Kristan et al. (; ; )), have been developed to provide standardized datasets and scientific evaluation mechanisms for SOT research. However, with the progress in deep learning techniques, these short-term and small-scale benchmarks have faced difficulties in sufficiently supporting data-driven trackers. This has led researchers to create larger-scale datasets like GOT-10k (Huang et al. ()) and TrackingNet (Muller et al. ()). Some work has also focused on SOT in drone scenarios, such as BioDrone (Zhao et al. ()), a vision benchmark for SOT based on bionic drones and WebUAV-3M (Zhang et al. ()).
More recently, researchers introduced the global instance tracking task along with a new benchmark called VideoCube (Hu et al. ()), allowing the tracking of arbitrary moving objects in various types of videos. To scientifically assess tracker performance under different challenging conditions, researchers have also introduced SOTVerse (Hu et al. ()), a user-defined space for the SOT task.

Visual Language Tracking Benchmark.
Over the past few decades, visual benchmarks have seen considerable development, yet benchmarks that incorporate semantic information, known as VLT benchmarks, have only recently become prominent. OTB99_Lang (Li et al. ()) is notable for being the first VLT benchmark, augmenting the sequences from the OTB100 (Wu et al. ()) benchmark with natural language descriptions. However, the limited scale of the dataset has hindered the broader acceptance of the VLT task. Following this, the introduction of LaSOT (Fan et al. (; )), a multi-modal benchmark for long-term tracking, represented a major advancement. In the same year, researchers launched the TNL2K (Wang et al. ()) benchmark, which aimed to improve the flexibility and precision of object tracking through text descriptions. Recently, researchers have proposed a novel multi-modal benchmark called MGIT (Hu et al. ()), which introduces a multi-granular annotation approach (Li et al. ()). VastTrack (Peng et al. ()) facilitates the development of more general visual tracking via encompassing abundant classes and videos.

Algorithms for Visual Language Tracking.
VLT is a burgeoning multi-modal task that seeks to enhance tracking by utilizing both linguistic descriptions and initial template. Most current VLT trackers (Guo et al. (); Wang et al. (); Zhao et al. (); Li et al. (); Feng et al. (); Wang et al. (); Feng et al. (); Zhang et al. (; )) operate on the principle of similarity-matching, using language descriptions and template patches to pinpoint the most analogous object within the search frame. Among these, SNLT (Feng et al. ()) stands out with its adaptable language-based region proposal network, which boosts tracking precision by using a dynamic aggregation mechanism. On the other hand, MMTrack (Zheng et al. ()) offers a streamlined and potent approach to tracking, viewing the VLT task as a series of token generation. Certain VLT trackers have started to incorporate temporal data to build a more dynamic reference. For example, GTI (Yang et al. ()) and AdaSwitcher (Wang et al. ()) recognize objects by combining tracking and localization results at each time step. JointNLT (Zhou et al. ()) also moves in this direction by incorporating temporal information as queries during the prediction phase. UVLTrack (Ma et al. ()) design a modality unified feature extractor and propose a multi-modal contrastive loss. QueryNLT (Shao et al. ()) ensures spatio-temporal consistency by leveraging historical visual information to improve tracking performance.

Most benchmarks for VLT offer a single natural language description per video, with text annotations that are either overly simplistic or excessively complex. These inconsistencies impede the evaluation of algorithms and the understanding of video content for VLT trackers. Additionally, all these studies provide semantic information through manually annotated data, which is a lengthy and resource-intensive process. Fig. 1 suggests that a more scientific approach is also necessary for delivering high-quality semantic information. These limitations have led us to propose DTVLT, the first comprehensive VLT benchmark using LLM to provide multi-granularity diverse semantic information, with the aim of creating a more flexible and comprehensive environment for VLT and video understanding research.

**Table 1: Comparison of current datasets for object tracking. DTVLT is the first comprehensive VLT benchmark using LLM to provide multi-granularity diverse semantic information, covering five mainstream tracking datasets across three tracking tasks. “STT”, “LTT” and “GIT” refer to Short-term Tracking, Long-term Tracking and Global Instance Tracking.**
| Dataset | Video<br>number | Min<br>frame | Mean<br>frame | Max<br>frame | Total<br>frames | Tracking<br>task | Text Annotation |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Granularity | Sentence Number | Word Number | Tool |  |  |  |  |  |  |  |
| OTB99_Lang | 99 | 71 | 590 | 3,872 | 59K | STT | 1 | 99 | 358 | Human |
| GOT-10k | 10,000 | 29 | 149 | 1,418 | 1.5M | STT | 0 | 0 | 0 | - |
| LaSOT | 1,400 | 1,000 | 2,506 | 11,397 | 3.52M | LTT | 1 | 1,400 | 9,842 | Human |
| TNL2K | 2,000 | 21 | 622 | 18,488 | 1.24M | STT | 1 | 2,000 | 10,098 | Human |
| MGIT | 150 | 4,008 | 14,920 | 29,834 | 2.03M | GIT | 3 | 1,753 | 77,652 | Human |
| DTVLT (Ours) | 13,134 | 21 | 611 | 29,834 | 8.17M | STT & LTT & GIT | 4 | 240.8K | 5.2M | LLM |

## 3 Construction of DTVLT

Figure: Figure 2: The pipeline of text generation for DTVLT based on DTLLM-VLT, which can provide dense concise/detailed text generation based on given video frames and BBox of object.
Refer to caption: x1.png

### 3.1 Data Collection

We have chosen five notable datasets—OTB99_Lang (Li et al. ()), GOT-10k (Huang et al. ()), LaSOT (Fan et al. ()), TNL2k (Wang et al. ()), and MGIT (Hu et al. ())—to build DTVLT. (For further details, please refer Appendix A.2.)
GOT-10k stands as a traditional SOT benchmark.
OTB99_Lang and LaSOT are enhancements of traditional SOT benchmarks, incorporating additional language annotations. TNL2k is a benchmark created specifically for the VLT task.
It is worth noting that OTB99_Lang, GOT-10k, and TNL2k are considered representative datasets for short-term tracking, primarily offering a text for the first frame of each sequence. LaSOT, on the other hand, represents long-term tracking. Its textual annotations focus solely on describing the appearance of the target object, without including information about relative positions.
MGIT introduces a new, large-scale benchmark for global instance tracking. The text annotations for each sequence employ a multi-granular annotation strategy.

**Table 2: Summary of sentence number (word number) of four granularity generated language description in DTVLT. We using LLM and provide far more diverse semantic information based on representative environments to form our DTVLT benchmark. “Dense” indicates that provides text for every 100 frames, “initial” indicates that only the first frame of text is provided, and “concise” and “detailed” indicate the richness of information, respectively. We illustrate the diversity of text by analyzing the sentence number of texts at different granularities and the number of words.**
| Sentence Number (Word Number) of four granularity language description in DTVLT |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| DTVLT | Data Source | Dense Concise | Dense Detailed | Initial Concise | Initial Detailed |
| OTB99_Lang | 0.6K (3.2K) | 0.6K (21.9K) | 0.1K (0.5K) | 0.1K (3.6K) |  |
| GOT-10k^1 | 43.0K (253.6K) | 43.0K (1.8M) | 9.5K (49.3K) | 9.5K (346.5K) |  |
| LaSOT | 35.2K (182.6K) | 35.2K (1.2M) | 1.4K (7.1K) | 1.4K (47.4K) |  |
| TNL2K | 12.4K (71.6K) | 12.4K (476.0K) | 2.0K (10.7K) | 2.0K (73.0K) |  |
| MGIT^2 | 16.1K (83.2K) | 16.1K (553.7K) | 0.1K (0.6K) | 0.1K (4.5K) |  |

### 3.2 Generation Tool

Traditional datasets for VLT are dependent on manual annotations. This process is costly, operates at a single annotation granularity, and is not suitable for annotating large volumes of data. To overcome these challenges, we have developed DTLLM-VLT (Li et al. ()), a method capable of producing extensive and diverse text generation based on LLM. The pipeline of text generation for DTVLT is illustrated in Fig. 2. By taking video frames and the BBox of objects as inputs, DTLLM-VLT generates concise and detailed descriptions for the relevant objects. This methodology enables us to generate large-scale, diverse granularities text at low costs. The detailed workflow and ablation study has been outlined in Appendix A.3.1 and A.3.2.

### 3.3 Generation Strategy

The volume and linguistic annotations of the VLT datasets determine the tracking performance.
Table 1 shows that the current tracking dataset for VLT is equipped with just 5,252 (99 + 1,400 + 2,000 + 1,753) official textual descriptions. This limited data is considered inadequate for algorithms to learn effectively. These official annotations are only sufficient to describe the short-term changes of the object. The lack of textual descriptions impedes the trackers’ ability to acquire a comprehensive understanding of visual contents, leading to a substantial decline in performance when generalizing to new videos. Furthermore, inaccurate textual descriptions can obstruct object tracking, turning natural language annotations into a hindrance rather than a support.

Figure: Figure 3: Examples of the text generation in DTVLT. We provide four different natural language descriptions for each video. Diverse multi-granularity text can support fine-grained evaluation of trackers, providing guidance for the development of tracking.
Refer to caption: x2.png

In this work, we design a multi-granular generation strategy to provide scientific natural language information. To enhance the accuracy and generality, we generate texts for five datasets to construct DTVLT, as shown in Table 2, establishing a robust foundation for VLT. This generation strategy can be expanded to more VLT and SOT datasets.

Initial and dense text descriptions.
Inspired by the text annotations approach in OTB99_Lang (Li et al. ()) and TNL2K (Wang et al. ()), we generate text for the first frame of each video. Additionally, recognizing that 4 seconds is the threshold between human instant memory and short-term memory (Radvansky (); Strous et al. (); Atkinson & Shiffrin ()), we prepare for the most challenging scenario where the algorithm may not have an efficient memory mechanism. Therefore, at a frame rate of 25 FPS, equating to every 100 frames in 4 seconds, we provide the algorithm with generated text. We believe that this frequency of updates optimally maintains the algorithm’s memory state and improves tracking capabilities.

Concise and detailed text descriptions.
For the algorithm, if the BBox already adequately captures the temporal and spatial dynamics of the object, the texts should concentrate on delivering key semantic elements such as the object’s category and location. When the BBox does not provide enough information for the tracker’s efficient learning, more comprehensive texts are required to make up for the absent temporal and spatial connections. As a result, we generate two types of textual descriptions: concise and detailed. As depicted in Fig. 2, the concise text conveys essential information about the object, like its category (bear) and position (in the water), whereas the detailed text encompasses further spatio-temporal specifics such as color, relative position, and activities.

### 3.4 Generation Analysis

We provide four granularities of natural language descriptions for each video, which are the initial concise description, initial detailed description, dense concise description, and dense detailed description. This is depicted in Fig. 3. For more examples of different tracking tasks in DTVLT, please refer to Appendix A.3.5. Our goal is to use diverse textual information to enhance the learning and evaluation environment for the algorithm, as well as to offer direction for algorithmic development and model optimization.

We generate text descriptions for the DTVLT using the DTLLM-VLT (Li et al. ()), which includes 26.2K initial descriptions (divided equally between 13.1K concise and 13.1K detailed texts) and 214.6K dense descriptions (also equally divided into 107.3K concise and 107.3K detailed texts). The quantity of our dense texts is 45.9 times larger than the official annotations. Additional information on the number of semantic descriptions is available in Table 1. These semantic descriptions consist of 5.2M words, featuring 17.6K non-repetitive words. Text descriptions for each frame only takes 2 seconds, and the entire method can be directly run on an RTX-3090 GPU. The vocabulary is rich, allowing for a comprehensive description of changes in the object during the tracking process. In summary, compared to previous tracking datasets, the DTVLT we constructed is a multi-task-oriented, multi-granular, large-scale dataset that utilizes LLM for automatic text annotation. Word cloud have been illustrated in Fig. 4. More detailed analysis has been outlined in Appendix A.3.3 and A.3.4.

Figure: Figure 4: The word cloud of semantic descriptions.
Refer to caption: extracted/5913655/imgs/wordcloud.png

## 4 Experimental Results

### 4.1 Datasets and Evaluation Methods

As shown in Fig. 3, we follow generation granularities to design various mechanisms.
We select several VLT trackers, MMTrack (Zheng et al. ()), JointNLT (Zhou et al. ()) and UVLTrack (Ma et al. ()) as baseline models and evaluate them on DTVLT (as shown in Table 3 and Fig. 5). The experimental section including both iid (independent and identically distributed) and ood (out of distribution) settings, such as LaSOT and GOT-10k being evaluated under iid and ood settings, respectively. It can verify the model’s generalization ability. With the diverse environment, we can analyze the strengths and weaknesses of various trackers from the experimental results, offering insights into the development of VLT trackers. Compared with other algorithms, MMTrack is designed to be flexible with text length, avoiding the truncation of lengthy text segments. Moreover, it treats the VLT task as a token generation process, facilitating more effective learning of visual-linguistic data. While JointNLT and UVLTrack sets 50 as a maximum limit and truncates the excess information. (See Appendix B.3 for more details.)

To fairly compare the tracking performance on five datasets, we use two evaluation mechanisms. (A) We directly use the officially provided weights to test with the official annotations and on the DTVLT. (B) We retrain these models for 50 epochs on the basis of the official weights using DTVLT and test under the corresponding settings to evaluate Area Under the Curve (AUC), and tracking precision (P). For more details on the evaluation metrics, please refer to Appendix B.1.

When retraining the tracker, we selected LaSOT (Fan et al. ()), OTB99_Lang (Li et al. ()), TNL2K (Wang et al. ()), and RefCOCOg Mao et al. () as the training data, with a ratio of 1:1:1:1. The template image and search image sizes are 128x128 and 256x256, respectively. We used AdamW (Mao et al. ()) as the optimizer and continued training for 50 epochs on the basis of the official weights, randomly sampling 30,000 image pairs per epoch. All trackers are trained on a server with four A5000 GPUs and tested on an RTX-3090 GPU. For dense text, the text is dynamically updated every hundred frames, and for initial text, only the first frame provides text information.

**Table 3: Comparison with testing directly on DTVLT. The best two results are highlighted in red and blue, respectively.**
| MMTrack | OTB99_Lang | MGIT (Activity) | LaSOT | TNL2K | GOT-10k |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| AUC | P | AUC | P | AUC | P | AUC | P | AUC | P |  |
| Official | 69.0 | 89.5 | 73.5 | 54.3 | 69.9 | 75.7 | 58.6 | 59.3 | - | - |
| Initial Concise | 70.6 | 91.1 | 73.9 | 54.9 | 69.0 | 74.7 | 56.6 | 56.9 | 82.9 | 79.5 |
| Initial Detailed | 68.0 | 88.4 | 72.7 | 53.4 | 68.7 | 74.4 | 55.9 | 55.4 | 82.7 | 79.0 |
| Dense Concise | 70.2 | 90.8 | 74.2 | 55.0 | 69.1 | 74.8 | 56.5 | 56.7 | 82.8 | 79.3 |
| Dense Detailed | 68.6 | 89.4 | 72.9 | 53.5 | 69.0 | 74.7 | 56.1 | 55.6 | 82.8 | 79.2 |

### 4.2 Testing Directly on DTVLT (Mechanism A)

We directly use the models provided by the official for testing. From Table 3, we can draw the following conclusions:
(1) Most trackers perform poorly when faced with the diverse text in DTVLT, such as JointNLT (Zhou et al. ()) and UVLTrack (Ma et al. ()), with this phenomenon being particularly prominent in JointNLT. When faced with texts not seen in the training data, JointNLT experiences a significant performance drop across various datasets. The lack of diverse VLT datasets makes it difficult for researchers to comprehensively evaluate algorithm performance when designing and testing algorithms, leading to a phenomenon similar to “memorizing the answer” observed with JointNLT and UVLTrack. (2) The approach of sequence generation is more conducive to learning unified visual-language features. It can be observed that MMTrack (Zheng et al. ()) has achieved further performance improvements on some datasets by incorporating the diverse texts from DTVLT, showing a stronger adaptability to text. (3) We think that the current algorithm’s handling of long texts and the alignment of multiple modalities needs refinement, as it does not make the most of temporal and spatial relationships. Such temporal and spatial data are essential for enhancing tracking capabilities. In instances where the BBox’s temporal-spatial details are insufficient to reliably pinpoint the object, detailed textual information is required to supply extra high-level semantic insights necessary for object
tracking.

Through direct testing and comparison of tracking performance under different texts, it has been observed that the variation in texts has a significant impact on tracking performance. DTVLT compensates for the shortcomings of existing VLT datasets that cannot provide a flexible and comprehensive assessment.

### 4.3 Retraining and Testing on DTVLT (Mechanism B)

As previously discussed, when the dataset text information becomes denser and more accurate, it can make up for the deficiencies in the BBox annotations. The algorithm can acquire supplementary knowledge through these textual updates, which may lead to an enhancement in its performance. Consequently, we have retrained and evaluated models using a range of differently generated textual inputs.
As shown in Fig. 5, we plot the performance differences between the model after retraining and direct testing, where the red line represents the mean of these differences. For the absolute performance values of the model, please refer to the detailed data in the Appendix B.2.2. By comparing with other trackers, MMTrack (Zheng et al. ()) and UVLTrack (Ma et al. ()) have seen further improvements in performance, but the performance of JointNLT (Zhou et al. ()) has continued to decline instead. While our hope is that the inclusion of additional modalities will enhance the tracking performance, the current VLT trackers struggle to effectively integrate various types of information, leading to an underutilization of the multi-modal data. This outcome—where contemporary VLT trackers perform more poorly than those relying solely on visual cues—is also documented in other studies, highlighting the significant room for improvement in multi-modal tracking.

Figure: Figure 5: Comparison with retraining for 50 epochs and testing on DTVLT. We plot the performance differences between the model after retraining and direct testing, where the red line represents the mean of these differences.
Refer to caption: extracted/5913655/imgs/retrain-diff.png

By comparing results under mechanisms A and B, it is evident that in this flexible and comprehensive setting, trackers that are thoughtfully crafted (that is, those equipped with the capacity for extended input processing and the ability to align multi-modal data) can achieve superior results through diverse descriptions, rather than relying solely on brief descriptions. The experiments conducted highlight two pivotal insights: (1) Richer semantic data can enhance tracking capabilities compared to a simple sentence, which also substantiates the precision and relevance of the proposed multi-scale semantic generation strategy. (2) Providing only a basic description to VLT trackers is not practical. As a result, initiating the tracking procedure with longer and detailed sentences, or regularly updating the semantic data throughout the sequence, has proven to be more efficacious in precisely locating targets amidst intricate scenes.

### 4.4 Summary

Among the three algorithms, MMTrack (Zheng et al. ()) demonstrated the best performance, showing some content worth further research and exploration in Mechanism A and Mechanism B. For instance, on the MGIT (Hu et al. ()) dataset, dense concise text achieved optimal performance under direct testing conditions, which is somewhat different from the motivation proposed by the MGIT dataset. We believe that the current algorithms lack in long text processing and multi-modal alignment capabilities, so when facing long videos and high-difficulty sequences like MGIT, they cannot make good use of the official long text annotations. Additionally, on the OTB99_Lang (Li et al. ()) dataset, using initial concise text for direct testing yielded the best performance. The early datasets represented by OTB99_Lang have provided sufficient information for tracking in the BBox, and in this case, the text only needs to provide the most basic information to assist in enhancing tracking performance. This trend was further reflected after retraining and testing. We believe that MMTrack’s good performance lies in its modeling approach to the VLT task, learning visual-linguistic features through sequence generation, which can enhance the generalization of the tracker.

JointNLT (Zhou et al. ()), as a representative of the recent SOTA algorithms, has shown surprisingly disappointing results. Both in direct testing and after retraining and testing, JointNLT’s performance has declined significantly, which also confirms our analysis of the current VLT benchmark. That is, the existing text is not sufficient to support tracking like JointNLT to learn strong visual-linguistic tracking capabilities. They still adopt a strategy of “memorizing the answer” to complete the VLT task. UVLTrack’s (Ma et al. ()) performance falls between the two, but it also exhibits phenomena similar to JointNLT.

In summary, the emergence of DTVLT can provide a high-quality flexible experimental environment for research, and help the algorithms quickly identify bottleneck issues under various evaluation mechanisms, thereby accelerating the development of VLT algorithms.

### 4.5 Visualization and Bad Case Analysis

Figure: Figure 6: Visualization of tracking results on dense concise text annotations retrained algorithm. (A-B) In the LaSOT dataset, VLT trackers tend to identify and follow similar objects. (C) VLT trackers struggle to adjust to the intricacies of complex scenes, as observed in the MGIT dataset.
Refer to caption: extracted/5913655/imgs/demo.png

We delve deeper into the limitations of the VLT algorithms through the bad cases shown in Fig. 6.
The first two cases are sourced from LaSOT (Fan et al. ()), while the final case is taken from MGIT (Hu et al. ()).
MGIT and LaSOT face similar challenges in VLT task, such as interference between the object and the background, as well as significant variations in the object’s appearance from the initial frame to subsequent frames. These challenges not only increase the difficulty of tracking but also affect the overall performance of existing trackers. In this context, the introduction of a diverse text generation method has become a more viable solution. By providing multi-granularity text, it enables trackers to better handle changes in the object’s appearance and interference from complex backgrounds.
To further enhance performance, current trackers require a more sophisticated and intelligent semantic information processing module. This module should be able to precisely extract relevant details indicated by semantic tags, helping the tracker locate and follow the object more accurately in complex scenarios. However, the current design of trackers has not yet been specifically optimized to meet this requirement, lacking mechanisms to fully utilize semantic information, which leaves room for improvement in handling complex scenarios. For more bad cases, please refer to Appendix B.4.

## 5 Conclusions

Summary.
Object tracking forms the foundation for advanced tasks such as video understanding, and VLT may offer a promising approach to enhancing tracking capabilities. Unlike existing VLT benchmarks that primarily feature ambiguous descriptions, we (1) introduce a new VLT benchmark named DTVLT based on five benchmarks, and (2) develop a multi-granular text generation strategy to create diverse semantic information. DTVLT is the first comprehensive VLT benchmark using LLM to provide multi-granularity diverse semantic information.
In conclusion, we hope this work aids researchers in advancing their studies in VLT and video understanding.

Limitations.
Future work can address some current limitations.
First, DTVLT can be expanded with additional SOT and VLT benchmarks, creating a more complex and challenging environment for tracking algorithms.
Additionally, a more comprehensive evaluation system can be designed to better assess VLT and video understanding capabilities.

## Appendix A Dataset Information

### A.1 Basic Information

In this work, we propose a new visual language tracking benchmark with diverse texts, named DTVLT, based on five prominent VLT and SOT benchmarks, including three sub-tasks: short-term tracking, long-term tracking, and global instance tracking, aiming to support further research in VLT and video understanding.

Currently, the vast majority of VLT benchmarks are annotated with a single granularity in natural language, and there is an issue of only describing the changes in the target of the first frame, which hinders the algorithm’s understanding of the video content. The algorithms tend to adopt a ‘memorizing the answer’ approach to accomplish the task of object tracking.
This phenomenon highlights the constraints imposed by single granularity text descriptions on the comprehension of long videos. Consequently, our research endeavors to incorporate diverse semantic cues, with the goal of equipping algorithms to more effectively navigate the intricate narrative dynamics inherent to target tracking and the understanding of video contents.

### A.2 Data Selection

We have selected five representative benchmarks, covering short-term tracking (OTB99_Lang (Li et al. ()), GOT-10k (Huang et al. ()) and TNL2K (Wang et al. ())), long-term tracking (LaSOT (Fan et al. ())), and global instance tracking (MGIT (Hu et al. ())) tasks. The number of videos in each dataset and the number of texts officially annotated are shown in Table A1.

**Table A1: Summary of selected datasets.**
| Dataset | Number of Videos | Official Annotations |  |
| --- | --- | --- | --- |
| Train | Evaluation |  |  |
| OTB99_Lang (Li et al. ()) | 51 | 48 | 99 |
| GOT-10k (Huang et al. ()) | 9,335 | 360 | 0 |
| LaSOT (Fan et al. ()) | 1,120 | 280 | 1,400 |
| TNL2K (Wang et al. ()) | 1,300 | 700 | 2,000 |
| MGIT (Hu et al. ()) | 105 | 45 | 1,753 |

### A.3 Diverse Text Generation

#### A.3.1 Generation Pipeline

The pipeline of text generation in DTVLT with DTLLM-VLT (Li et al. ()) is depicted in Fig. A1. An input video frame accompanied by the respective object BBox is processed by SAM (Kirillov et al. ()), which employs an image encoder, a prompt encoder, and a mask decoder to extract the masks of the object in question. These masks, along with the video frame, are then fed into Osprey (Yuan et al. ()). Within Osprey, the images and masks undergo encoding, are merged with pre-defined prompts, and subsequently, the system leverages a LLM (Chiang et al. (); Touvron et al. ()) to produce succinct and comprehensive descriptions for the objects. This methodology allows for the generation of large-scale, diverse granularities textual data for SOT and VLT datasets at minimal expense.

Figure: Figure A1: The pipeline of text generation for DTVLT based on DTLLM-VLT (Li et al. ()), which can provide dense categories on MGIToncise/detailed text generation based on given video frames and BBox of object.
Refer to caption: x3.png

#### A.3.2 Reason for selecting LLM and Ablation Study

We will introduce the reason for selecting LLM and corresponding ablation study.

Purpose of using LLM for DTVLT: Existing benchmarks provide video-level text that struggles to effectively capture the dynamic changes in video content, which also hinders the development of efficient video-language trackers. Therefore, providing more diverse semantic descriptions that align with the dynamic characteristics of videos for existing VLT benchmarks can offer a rich data environment for further algorithm optimization and holds significant research value. To achieve this goal, we need to using LLM and construct a pipeline to understand the dynamic changes in the video process, especially to effectively perceive fine-grained target variations.

Reason for choosing SAM and Osprey: Currently, the BBox in tracking benchmarks only provides relatively coarse information. Thus, we first obtain masks through SAM (Kirillov et al. ()) and acquire pixel-level information of the object, laying a solid foundation for fine-grained perception. On this basis, Osprey’s (Yuan et al. ()) goal is to achieve pixel-level understanding, which coincides with our needs for dynamic changes in the tracking process. In addition to inputting complete images for perceiving foreground and background information, it also provides a fine-grained encoder for masks, which will enhance the understanding of dynamic environments. Moreover, Osprey’s training data is regenerated through GPT Achiam et al. () and is currently the only model that can provide detailed text descriptions for tracking objects in region-level caption field. The entire method is plug-and-play, and each module can be replaced at any time.

**Table A2: Ablation Study of SAM (Kirillov et al. ()) and LLM Backbone with Text Similarity Comparison on OTB99_Lang (Li et al. ()). For SAM, we replaced the SAM-B with SAM-H and SAM-L, and for Osprey (Yuan et al. ()), we replaced the Osprey with Osprey-chat and Controlcap (Zhao et al. ()). We report BLEU (Papineni et al. ()), GLEU (Mutton et al. ()), METEOR (Banerjee & Lavie ()), Recall, Precision and F1 score (Goutte & Gaussier ()). We compared the similarity between the texts generated after replacing the backbone and the DTVLT texts, so the scores of ours (combining SAM-B with Osprey) are all 1.00.**
| Text Similarity Comparison on OTB99_Lang |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SAM Backbone | LLM Backbone | Text Granularity | BLEU | GLEU | METEOR | Recall | Precision | F1 |
| SAM-H | Osprey | Dense Concise | 0.76 | 0.79 | 0.84 | 0.87 | 0.87 | 0.87 |
| Dense Detailed | 0.56 | 0.57 | 0.68 | 0.72 | 0.71 | 0.71 |  |  |
| SAM-L | Dense Concise | 0.68 | 0.72 | 0.79 | 0.85 | 0.87 | 0.86 |  |
| Dense Detailed | 0.56 | 0.58 | 0.67 | 0.74 | 0.73 | 0.72 |  |  |
| SAM-B | Osprey-Chat | Dense Concise | 0.25 | 0.34 | 0.48 | 0.59 | 0.60 | 0.60 |
| Dense Detailed | 0.17 | 0.23 | 0.41 | 0.48 | 0.51 | 0.49 |  |  |
| ControlCap^1 | Dense Concise | 0.09 | 0.14 | 0.28 | 0.30 | 0.37 | 0.34 |  |

**Table A3: Ablation Study of SAM (Kirillov et al. ()) and LLM Backbone with Visual Language Tracking Performance on OTB99_Lang (Li et al. ()). For SAM, we replaced the SAM-B with SAM-H and SAM-L, and for Osprey (Yuan et al. ()), we replaced the Osprey with Osprey-chat and Controlcap (Zhao et al. ()). We test directly on DTVLT (Mechanism A) and report AUC and Precision score.**
| Visual Language Tracking Performance on OTB99_Lang |  |  |  |  |
| --- | --- | --- | --- | --- |
| SAM Backbone | LLM Backbone | Text Granularity | Precision | AUC |
| SAM-B (Ours) | Osprey (Ours) | Official | 89.5 | 69.0 |
| Dense Concise | 90.8 | 70.2 |  |  |
| Dense Detailed | 89.4 | 68.6 |  |  |
| SAM-H | Osprey | Dense Concise | 90.5 | 69.9 |
| Dense Detailed | 90.6 | 69.6 |  |  |
| SAM-L | Dense Concise | 90.0 | 69.7 |  |
| Dense Detailed | 90.4 | 69.4 |  |  |
| SAM-B | Osprey-Chat | Dense Concise | 88.8 | 68.5 |
| Dense Detailed | 89.0 | 68.2 |  |  |
| ControlCap^1 | Dense Concise | 90.7 | 69.9 |  |

Details of ablation study: We conduct a detailed ablation analysis of SAM (Kirillov et al. ()) and Osprey (Yuan et al. ()). We employed two types of metrics to compare the results of the ablation study, including Precision and AUC for tracking, and Recall, Precision, F1 (Goutte & Gaussier ()), BLEU Papineni et al. (), GLEU Mutton et al. () and METEOR Banerjee & Lavie () for text similarity comparison, to evaluate the choice of the backbone from multiple perspectives. For SAM, we replaced the SAM-B with SAM-H and SAM-L, and for Osprey, we replaced the Osprey with Osprey-chat and Controlcap (Zhao et al. ()). Since ControlCap cannot generate detailed text, we only analyzed its concise text. When comparing text similarity, we compared the similarity between the texts generated after replacing the backbone and the DTVLT texts.

Conclusion of ablation study: We present the results of the ablation study. From Table A2, it can be observed that replacing SAM (Kirillov et al. ()) does not significantly affect text generation; the text similarity metrics are all very high. Considering both the time cost and performance, we have chosen SAM-B. However, after replacing the LLM backbone, there is a noticeable decrease in the metrics. This is because different LLMs have different training data and strategies, and the descriptions for the same object may vary in style. Our goal is to use diverse texts for tracking tasks, so we further conducted ablation analysis for tracking experiments based on the generated texts. In Table A3, we conducted tracking experiments on the OTB99_Lang (Li et al. ()) dataset for dense texts. The backbone model we used achieved the best performance on OTB99_Lang, which further verifies the importance and rationality of the texts generated by DTVLT. And it can be seen that our proposed framework still enhances tracking performance with most dense texts after replacing different modules.

#### A.3.3 Generation Analysis

Figure: Figure A2: word cloud of four granularities on DTVLT.
Refer to caption: x4.png

We calculated the visual-textual similarity between the official annotations and the text generated by DTVLT with respect to the tracking targets. Specifically, we cropped the target from the video frames according to the BBox, and then calculated the similarity between the text and the target using CLIP (Radford et al. ()). The visual-textual similarity between the official text annotations and the target is 0.187. In the DTVLT, the visual-textual similarity for the concise text and detailed text generated with respect to the target are 0.185 and 0.195, respectively. We think the visual-textual similarity of the text generated in DTVLT is comparable to that of the manually annotated text. The word cloud of various texts is shown in Fig. A2. Our generated texts are more diverse than official annotations.

#### A.3.4 Generation File

We propose a multi-granular generation strategy to generate the semantic description, and use txt format to save the natural language annotation for each video sequence.
Here we illustrate an example to show the txt file structure for video sequence bear-17 with dense detailed texts in the LaSOT benchmark, as shown in Fig. A3.
Due to the limited space, we only illustrate some representative information, with additional information of a similar structure represented by ellipses.
Please download and check the dataset for more detailed annotation about each video sequence.

Figure: Figure A3: Example of texts generation for DTVLT.
Refer to caption: extracted/5913655/imgs/supple_full.png

#### A.3.5 More Example in DTVLT

DTVLT encompasses three tracking tasks designed to assess the capabilities of tracking systems under varying conditions and durations. The three tasks are:

Short-term Tracking (STT). This task focuses on tracking objects over short sequences where the object remains visible throughout. It tests a tracker’s ability to handle rapid movements, occlusions, and changing environments over brief periods.

Long-term Tracking (LTT). LTT challenges trackers to maintain the identity of objects over extended sequences, where the object might disappear and reappear. It evaluates the endurance of trackers in maintaining tracking consistency over long durations and their ability to re-identify the object after loss of track.

Global Instance Tracking (GIT). This involves tracking an object across different scenes and conditions, possibly even when the object changes its appearance significantly. GIT tests a tracker’s ability to generalize the object’s identity across various scenarios and to handle large-scale variations in appearance.

The DTVLT framework provides a multi-granular textual description for these tasks, enabling a detailed evaluation of each tracker’s performance on specific challenges posed by different tracking environments. Such a setup not only pinpoints the strengths and weaknesses of each tracking algorithm but also offers insights that can drive innovations in tracker design. Examples of each tracking task in DTVLT are illustrated in Fig. A7 (STT), Fig. A5 (LTT) and Fig. A6 (GIT).

Figure: Figure A4: Example for Short-term Tracking (STT) in DTVLT.
Refer to caption: extracted/5913655/imgs/STT.png

Figure: Figure A5: Example for Long-term Tracking (LTT) in DTVLT.
Refer to caption: extracted/5913655/imgs/LTT.png

Figure: Figure A6: Example for Global Instance Tracking (GIT) in DTVLT.
Refer to caption: extracted/5913655/imgs/GIT.png

## Appendix B Experiment Information

### B.1 Evaluation Metrics

Precision P typically refers to the proportion of frames where the distance between the tracker’s predicted bounding box and the ground-truth bounding box is less than or equal to a given threshold $\theta_{d}$. The specific definition is as follows:

$$ $\mathrm{P}(E)=\frac{1}{\lvert E\rvert}\sum_{l=1}^{\lvert E\rvert}\frac{1}{ \lvert L\rvert}\lvert\left\{t:d_{t}\leq\theta_{d}\right\}\rvert$ $$

Here, $\lvert E\rvert$ represents the total number of sequences in the dataset, $\lvert L\rvert$ represents the number of frames in sequence $l$, $d_{t}$ is the distance between the predicted position $p_{t}$ and the ground-truth position $g_{t}$ in frame $t$, and $\theta_{d}$ is a preset threshold.

AUC is the area under the cumulative distribution function (CDF) of Precision P calculated at different $\theta_{d}$ values. It provides a comprehensive measure to evaluate the performance of the tracker at various distance thresholds. The value of AUC ranges between 0 and 1, with higher values indicating better tracker performance. The definition of AUC is as follows:

$$ $\text{AUC}=\int_{0}^{d_{\text{max}}}\mathrm{P}(\theta_{d})\,d\theta_{d}$ $$

Here, $d_{\text{max}}$ is the maximum possible distance value, and $\mathrm{P}(\theta_{d})$ is the precision at a specific threshold $\theta_{d}$. The calculation of AUC is typically done by plotting the Precision-Recall Curve at a range of $\theta_{d}$ values, then approximating the integral using numerical integration methods (such as the trapezoidal rule).

In practice, AUC is obtained by plotting the Precision-Recall Curve, where Precision P is the point on the curve, and Recall is the proportion of frames correctly predicted by the tracker to the total number of frames. The AUC is the area under this curve.

### B.2 Evaluation Mechanism

#### B.2.1 Mechanism A

To evaluate the performance of existing algorithms on DTVLT, we implement Mechanism A. Utilizing the official weight files provided (URLs as shown in Table A4), we keep all parameters unchanged. During the evaluation process, we replace the official texts with texts from DTVLT to test the performance of various VLT algorithms under the initialization conditions of Natural Language (NL) and Bounding Box (BBox).

#### B.2.2 Mechanism B

Furthermore, we retrain three algorithms and then retest them on DTVLT, establishing Mechanism B. Specifically, we continue training for an additional 50 epochs based on the official weights, using datasets such as OTB99_Lang (Li et al. ()), LaSOT (Fan et al. ()), TNL2K (Wang et al. ()), and RefCOCOg (Mao et al. ()). During the training process, we replace the official texts with different texts. After the training was completed, we reassess the performance of the algorithms under the corresponding text settings.

Figure: Figure A7: Comparison with retraining for 50 epochs and testing on DTVLT.
Refer to caption: extracted/5913655/imgs/retrain.png

### B.3 Baseline Information

Detailed information about the baselines are illustrated in Table A4, we use the parameters provided by the original authors.

**Table A4: Table: The model architectures and URLs of open-sourced algorithms used in this work.**
| Tracker | Architecture | Initializa | URL |
| --- | --- | --- | --- |
| JointNLT (Zhou et al. ()) | Transformer | NL & BBox | https://github.com/lizhou-cs/JointNLT |
| MMTrack (Zheng et al. ()) | Transformer | NL & BBox | https://github.com/Azong-HQU/MMTrack |
| UVLTrack (Zheng et al. ()) | Transformer | NL & BBox | https://github.com/OpenSpaceAI/UVLTrack |

JointNLT (Zhou et al. ()) operates as a combined visual grounding and tracking framework, utilizing natural language specifications for tracking. This framework integrates the tasks of tracking and grounding, allowing it to handle various references within these processes. Moreover, it introduces a semantics-guided temporal modeling module, which offers temporal cues derived from historical predictions to the joint model, thereby enhancing its ability to adapt to changes in the appearance of the object.

MMTrack (Zheng et al. ()) redefines vision-language tracking by conceptualizing it as a token generation task. It develops an innovative pipeline that taps into the capabilities of VL multi-modal learning from a holistic modeling standpoint. The method is both simple and adaptable, combining language and bounding boxes into multi-cue token inputs. It simplifies the process by discarding unnecessary sub-task learning and optimization goals, using cross-entropy solely as its single training objective.

UVLTrack (Ma et al. ()) presents a groundbreaking unified tracker for both visual and vision-language tracking, which is adept at managing three distinct types of target references (BBOX, NL, NL+BBOX) simultaneously. It has engineered a modality-unified feature extractor that facilitates the concurrent learning of visual and language features and implements a multi-modal contrastive loss to integrate these modal features into a cohesive semantic framework. It introduces a modality-adaptive box head that effectively extracts scenario-specific features from various modal references and precisely localizes the target through a contrastive approach, thereby boosting its robust performance in all reference scenarios.

### B.4 More Bad Cases

In order to demonstrate how different algorithms perform in diverse environments, We specifically selected several cases where they performed poorly as shown in Fig. A8, A9 and A10. These examples show that only in a diverse environment can we observe the strengths and weaknesses of algorithms in finer detail. Therefore, providing such an environment is crucial as it enables us to more accurately assess and compare the performance of various algorithms.

Figure: Figure A8: Bad Case for UVLTrack in DTVLT.
Refer to caption: extracted/5913655/imgs/uvltrack.png

Figure: Figure A9: Bad Case for MMTrack in DTVLT.
Refer to caption: extracted/5913655/imgs/mmtrack.png

Figure: Figure A10: Bad Case for JointNLT in DTVLT.
Refer to caption: extracted/5913655/imgs/jointnlt.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: (1) What are the specific scientific challenges this dataset aims to address, beyond what existing benchmarks like Dtllm-vlt have already solved?
(2) What quality control measures were implemented to ensure that LLM-generated texts accurately describe the visual content, and how were issues like semantic bias or redundancy managed?
(3) For additional questions, please refer to the Weaknesses section.
soundness: 2
strengths: (1) Enhanced Text Diversity: The benchmark introduces a multi-granularity approach to text descriptions, which offers different semantic densities. This allows for a finer evaluation of VLT algorithms' ability to handle both concise and detailed language input.
(2) Integration with LLMs: By leveraging LLMs, the paper automates text generation, potentially reducing the need for costly manual annotations and allowing for large-scale text generation across various scenarios.
(3) Comprehensive Experimental Setup: The paper evaluates multiple VLT models on DTVLT, providing a detailed analysis of how different text granularities impact tracking performance. This analysis can offer insights into the limitations of current VLT models.
summary: This paper presents DTVLT, a new multi-modal benchmark for Visual Language Tracking (VLT), aiming to enrich existing VLT datasets by introducing diverse textual annotations generated by large language models (LLMs). DTVLT is built upon five existing VLT and SOT benchmarks, and it employs a multi-granularity text generation strategy to provide descriptions of varying lengths and detail levels. The benchmark is designed to support three main tasks—short-term tracking, long-term tracking, and global instance tracking—providing a more comprehensive environment for evaluating VLT algorithms.
weaknesses: (1) This paper appears more like a pipeline proposal, merely leveraging existing models to enhance an existing dataset, without focusing on the scientific questions that should be addressed when establishing this dataset.
(2) The motivation is unclear, as Dtllm-vlt[1] seems to have already addressed the short-term tracking, long-term tracking, and global instance tracking challenges mentioned in this paper.
(3) DTVLT heavily relies on LLM-generated text descriptions but lacks a quality control method for these generated texts. The automatically generated text may contain semantic biases, inconsistencies, overly long or redundant information, and may even be disconnected from the visual content, particularly in multi-modal tasks. Analyzing the quality of generated texts and investigating how to produce better descriptions would be more worthwhile.

[1] DTLLM-VLT: Diverse Text Generation for Visual Language Tracking Based on LLM.

---

confidence: 5
contribution: 3
initial_rating: 3
presentation: 3
questions: Please refer to the concerns and issues raised in the "Weaknesses".
soundness: 3
strengths: 1. The manuscript is well-structured, with clear, fluent writing that makes the content easy to understand.

2. The authors introduce a new benchmark, DTVLT, based on five prominent VLT and SOT benchmarks, addressing three tracking tasks: short-term tracking, long-term tracking, and global instance tracking. DTVLT provides four granularity combinations, capturing the extent and density of semantic information, and incorporates DTLLM-VLT, which uses LLM to generate diverse, high-quality language descriptions.

3. Comprehensive experimental analyses evaluate the effect of diverse textual descriptions on tracking performance, with insights into performance bottlenecks in existing algorithms that may aid future VLT research.
summary: The paper introduces a new benchmark, DTVLT, for Visual Language Tracking (VLT) that enhances traditional single-object tracking by integrating more diverse linguistic data through large language models (LLMs). Current VLT benchmarks rely on brief, standardized annotations, which limit the depth of video content understanding, leading algorithms to rely on memorization rather than genuine comprehension. By leveraging LLMs, DTVLT provides varied semantic annotations at multiple levels of detail for existing VLT and single-object tracking datasets, supporting tasks like short-term, long-term, and global instance tracking. This benchmark includes four text granularity levels, enabling richer semantic understanding by capturing broader context and subtle video content dynamics. Experimental analyses show that diverse text annotations improve tracking performance and reveal limitations in current algorithms, encouraging further VLT research.
weaknesses: 1. In Table 3, did the author use the original implementations and pretrained weights for these trackers (MMTrack, JointNLT, and UVLTrack)? If so, it would be helpful for the author to discuss any discrepancies between their results and those reported in the original papers, as the performance results for these trackers in Table 3 do not align with the performance reported in their respective publications.

2. As this paper proposes a benchmark, additional trackers should be tested to fully evaluate it (i.e., VLT ("Divert More Attention to Vision-Language Tracking")).

3. I am skeptical about the necessity of the "Initial Concise" annotation by the LLM. This annotation seems less effective than the official language, as the performance decreases when directly testing the trackers, as shown in Table 3. Additionally, Figure A7 indicates that even with retraining, performance still declines with the "Initial Concise" annotation. Could the author provide a more detailed analysis of why they included the "Initial Concise" annotation despite its apparent limitations, and what specific value it adds to the benchmark.

4. When directly testing tracking performance with "Dense Concise" or "Dense Detailed" annotations—such as for the N-th test frame in a video, which language annotation does the author use? How their choice of annotation for each frame might impact the tracking results, and whether they considered alternative approaches.

5. In the training process using "Dense Concise" or "Dense Detailed' annotation, the model needs sample the N-th training frame from the given video. Which language annotation does the author use in this case. how the choice of annotation during training might affect the model's ability to generalize to different text granularities during testing.

6. In Figure A7, the performance with "Dense Detailed" annotation appears worse for the AUC metric. Could the author provide a hypothesis for why the "Dense Detailed" annotation leads to worse AUC performance, and to discuss the implications of this finding for the design of VLT systems.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: See the Weaknesses.
soundness: 3
strengths: - The proposed DTVLT is the first comprehensive VLT benchmark using LLM to provide multi-granularity and diverse semantic information.

- The quantity of the dense texts in DTVLT is 45.9 times larger than the official annotations of previous datasets, which will highly support further research in VLT and video understanding.

- The paper is easy to follow and clearly states its contributions, providing a comprehensive presentation of the content, including bad cases.
summary: The paper proposes a new visual language tracking benchmark featuring diverse texts, built upon five prominent VLT and SOT benchmarks. It is the first to leverage the extensive knowledge base of large language models (LLMs) to deliver multi-granularity and diverse semantic information. Experiments initially demonstrate the impact of diverse texts on tracking performance.
weaknesses: - Tracking by natural language specification enhances human-machine interaction and has the potential to improve tracking outcomes. Although this paper introduces four levels of granularity in text annotations within the benchmark, the individual contributions of each text type to model performance remain ambiguous. As shown in Figure 5, models retrained on the DTVLT dataset exhibit improved performance on DTVLT; however, it is difficult to determine whether these models have merely memorized the patterns within DTVLT. Therefore, could you evaluate the models trained on DTVLT on an entirely different VLT dataset to assess model performance improvements? Additionally, could you train separate models on each of the four text granularity datasets and compare their performance to isolate the impact of each type?

- DTVLT provides significantly more diverse semantic information than previous datasets, with texts generated by large language models (LLMs), which differ from human language. Therefore, conducting real-world experiments to compare models trained with and without DTVLT is valuable for evaluating performance and robustness in practical applications. Have you considered conducting user studies or real-world deployment tests in specific domains like autonomous driving or surveillance to compare the performance of models trained with and without DTVLT?

- Considering the word distribution in DTVLT, as shown in Figure 4, terms like 'center', 'positioned', and 'black' constitute a large proportion, which may lead to bias in models trained on this dataset. It would be beneficial to analyze how this distribution impacts model performance. Could you conduct an ablation study to measure how removing or reducing the frequency of common terms like 'center', 'positioned', and 'black' affects model performance? 

- In terms of bad cases, complex scenarios with multiple similar objects often pose difficulties. As shown in Figure A9, many objects match the detailed description, such as 'A zebra is standing in the middle of the field', which may make it harder for the model to accurately identify the target. Therefore, filtering out redundant information may enhance the quality of the dataset. Have you considered assessing the frequency of such ambiguous cases in the dataset and implementing a post-processing step to refine the generated descriptions?

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Given the paper’s use of LLM-based information, a question arises: why not expand the scope from single-object to multi-object tracking, a more semantically challenging task that could further elevate the benchmark’s utility? Multi-object tracking would offer a richer context for assessing model capabilities in complex, real-world scenarios and create additional opportunities to explore and refine model understanding in multi-modal video tracking.
soundness: 3
strengths: 1.   This work presents a highly promising and comprehensive research direction for the community by providing annotations at different granularities across multiple scales, addressing a significant gap in current Visual Language Tracking (VLT) and Single Object Tracking (SOT) benchmarks. The addition of annotations at various levels of detail and frequency enriches the benchmark and enhances its adaptability to different levels of semantic complexity, which is crucial for future VLT research. By covering short-term, long-term, and global instance tracking sub-tasks, this benchmark enables a broader application range and contributes to a nuanced understanding of tracking performance across varying degrees of difficulty, making it a valuable resource for the community.
2.    The paper is very well-written, with a clear and concise structure that effectively conveys the research objectives, methodology, and findings. The coherent organization and compact style allow readers to easily follow the research narrative and understand the underlying significance of the proposed benchmark. The structured approach also highlights the value of including multiple granularity levels in annotations, underscoring the need for such diversity to achieve a deeper understanding of video content.
3.    The authors perform a thorough analysis of the dataset structure, providing the research community with a clear picture of how this dataset challenges existing models. This comprehensive evaluation allows researchers to recognize the distinct aspects of the dataset that may stress-test model capabilities. By offering insights into the specific challenges posed by the dataset’s diverse annotations, the authors present a strong case for the benchmark’s relevance in advancing multi-modal video understanding, especially given its focus on varied text descriptions and granularities.
 4.    The study maximizes the potential of large language models (LLMs) and pre-trained models, creating robust annotations that are contextually rich and diverse. Leveraging these models has allowed the authors to generate a wide range of annotations that are both high-quality and informed by extensive world knowledge. This methodological approach is particularly commendable, as it goes beyond simple label generation and instead creates an annotation set that reflects nuanced information, adding substantial depth to the benchmark.
summary: This paper addresses the limitations of current Visual Language Tracking (VLT) benchmarks, which often rely on simple, human-annotated text descriptions that lack nuance, stylistic variety, and granularity, leading algorithms to adopt a memorization approach rather than achieving genuine video content understanding. To overcome this, the authors leverage large language models (LLMs) to generate diverse semantic annotations for existing VLT and single object tracking (SOT) benchmarks, creating a new benchmark called DTVLT. DTVLT includes varied text descriptions across multiple levels of detail and frequency, supporting three sub-tasks: short-term tracking, long-term tracking, and global instance tracking. Through the method DTLLM-VLT, they produce high-quality, world-knowledge-rich descriptions in four levels of granularity. Experimental analyses on DTVLT reveal the effects of text diversity on tracking performance, aiming to uncover current algorithm limitations and drive advancements in VLT and video understanding research.
weaknesses: 1.    While the dataset is a valuable addition to the field, offering some level of novelty, it falls short of significant innovation. The work’s main strength lies in diversifying existing benchmark annotations rather than introducing a groundbreaking methodology or framework. Although this level of novelty is acknowledged, the lack of an entirely new approach may limit the perceived impact of the benchmark within the research community. The work could benefit from a clearer distinction between its novel contributions and those of prior benchmarks to emphasize its unique value.
2.    The dense captioning provided by the dataset primarily centers on spatial relationships, which, while useful, presents a relatively narrow challenge to models. By focusing mainly on spatial relations, the annotations risk being overly uniform in terms of complexity, potentially limiting the depth of the benchmark’s impact on model evaluation. A more varied approach, incorporating a broader spectrum of semantic relationships beyond spatial ones, could introduce more diverse challenges for model training and testing, offering a more robust assessment of model capabilities.
3.    The paper does not introduce new baseline models, especially for dense caption-based tasks, to validate the proposed benchmark’s utility. The absence of baseline models that are specifically designed to adapt to the dense captioning structure leaves an open question about whether existing models are sufficient or if structural changes are required to adapt models without compromising previous task integrity. Designing or adapting baseline models could serve as a valuable proof of concept, demonstrating the benchmark’s effectiveness and providing the community with a clear starting point for future research on model adaptations for dense captioning.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: See weaknesses
soundness: 3
strengths: 1. The presentation is quite clear and the motivation is sound. Figures and tables are well-illustrated.
2. The introduced DTVLT is a comprehensive benchmark that incorporates diverse text annotations, addressing the limitations of existing VLT benchmarks which often rely on uniform descriptions. 
3. The authors claim that the uniformity of texts may lead to "memorizing the answers”, and they conduct relevant experiments on the new benchmark to testify that variation in texts has a significant impact on tracking performance, which further bolsters the significance of DTVLT.
summary: This paper proposes DTVLT, a novel benchmark for Visual Language Tracking that integrates diverse text annotations across several prominent VLT and SOT benchmarks, which is generated via the usage of LLMs. DTVLT includes four levels of text granularity, allowing for a richer representation of video content, which is beneficial for VLT and video understanding research. Authors also conduct well-rounded experiments evaluating the impact of diverse text on tracking performance.
weaknesses: 1. The data generation pipeline as well as the data collection procedure are lack of novelty, by simply using previous DTLLM-VLT and VLT and SOT benchmarks.
2. The quality of the benchmark relies heavily on the performance of DTLLM-VLT, and there seems to be no correction/filtering strategy to compensate for this.
3. The evaluation of VLT methods should be more comprehensive, including more NLT approaches to further validate the effectiveness of DTVLT