Consistencies are All You Need for Semi-supervised Vision-Language Tracking

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-31 12:06:37

Abstract: Vision-Language Tracking (VLT) requires locating a specific target in video sequences, given a natural language prompt and an initial object box. Despite recent advancements, existing approaches heavily rely on expensive and time-consuming human annotations. To mitigate this limitation, directly generating pseudo labels from raw videos seems to be a straightforward solution; however, it inevitably introduces undesirable noise during the training process. Moreover, we insist that an efficient tracker should excel in tracking the target, regardless of the temporal direction. Building upon these insights, we propose the pioneering semi-supervised learning scheme for VLT task, representing a crucial step towards reducing the dependency on high-quality yet costly labeled data. Specifically, drawing inspiration from the natural attributes of a video (i.e., space, time, and semantics), our approach progressively leverages inherent consistencies from these aspects: (1) Spatially, each frame and any object cropped from it naturally form an image-bbox (bounding box) pair for self-training; (2) Temporally, bidirectional tracking trajectories should exhibit minimal differences; (3) Semantically, the correlation between visual and textual features is expected to remain consistent. Furthermore, the framework is validated with a simple yet effective tracker we devised, named ATTracker (Asymmetrical Transformer Tracker). It modifies the self-attention operation in an asymmetrical way, striving to enhance target-related features while suppressing noise. Extensive experiments confirm that our ATTracker serves as a robust baseline, outperforming fully supervised base trackers. By unveiling the potential of learning with limited annotations, this study aims to attract attention and pave the way for Semi-supervised Vision-Language Tracking (SS-VLT).

Main paper: 
 introduction: 
 Vision-Language Tracking (VLT), a fundamental and challenging task in multi-modal video understanding, has drawn increasing attention due to its vast applications in intelligent surveillance [40] , autonomous driving [37] , human-computer interaction [19] and other fields. The primary objective of this task is to localize target within video frames (termed search images), guided by semantic provided by the language prompt and visual cues dubbed template.To date, all previously proposed methods for VLT are trained under a fully-supervised setting. Despite the remarkable success achieved by these deep learning-based trackers, their data-hungry nature demands a substantial amount of fully annotated data for effective training. Moreover, VLT distinguishes itself significantly from other tasks (e.g., object detection, semantic segmentation), In contrast, our approach identifies the object with the most relevant semantic to the language prompt as the pseudo target. By further leveraging multi-consistences errors during tracking in raw videos, we train the tracker without heavyweight annotations.as categories of the target are not restricted to a pre-defined set, naturally introducing an open-world setting [2, 17] . Hence, training a robust tracker requires more than one large dataset to enhance performance in open-world scenarios. For instance, the widely used TNL2k [53] dataset comprises 1,240,000 video frames along with corresponding bounding boxes, and 2,000 matched natural language descriptions. The manual annotation of such vast amounts of data is time-consuming and labor-intensive.To tap into the wealth of unlabeled videos available on the Internet, several attempts have been made to explore visual tracking without direct supervision [50, 58] . Generally, these efforts strive to extract self-supervision signals from the temporal dimension of videos (shown in Fig. 1(a) ). They assume that a robust tracker can accurately predict both forward and backward tracking, even when the target is a randomly cropped region from the search image. Consequently, differences between the bi-directional predictions of the target in the initial frame can serve as a source of supervision.However, we argue that such a learning framework is unsuitable for training a VL tracker. In contrast to vision-only tracking, the crux of VLT lies in the integration of an additional natural language prompt. This linguistic input contributes a more precise expression to the target by incorporating high-level semantic information (e.g., attributes, categories, and structural relationships with other objects). Yet, these advantages are valid only when the visual information and linguistic prompt are properly aligned and fused [13] . A randomly selected region in the search image is unlikely to contain an object with meaning, let alone provide multi-modal semantics to facilitate VL representation learning for tracking. On the other hand, random samples lack clear object edges, hindering the regression learning for coordinates prediction. Moreover, unsupervised methods solely rely on visual cues and treat tracking as a template-matching problem in each frame. Without an understanding of the language instruction, these methods struggle in situations such as the ambiguity of bounding box (bbox) and the extensive appearance variation of the target. Therefore, challenges persist in discovering supervision signals from unlabeled data that are beneficial for cross-modal learning in VLT.In this paper, we concentrate on training a Vision-Language tracker with minimal manual intervention, aiming to bridge the research gap between fully-supervised and unsupervised tracking. We insist that an efficient VL tracker should first understand what to track, and then follow closely to the target regardless of its changes or the direction of the time. To achieve this, the tracker must perceive the semantic from both modalities and align multi-modal information to estimate trajectories in subsequent frames, overcoming challenges such as occlusions and appearance variations. Therefore, we propose a learning scheme (shown in Fig. 1(b) ) for VLT that generates high-quality pseudo labels and explores multiple consistencies inherent in raw videos, achieving competitive performances with limited annotations. Specifically, (1) To equip the tracker with the ability to perceive multi-modal semantic and accurately locate the target, we leverage spatial consistency in each frame, since an image and any object cropped within it naturally form an image-bbox pair for self-training. To ensure the cropped object contains rich semantic and is highly related to the linguistic description, we fine-tune a Vision-Language Pre-trained Model (VLPM) to discover an object with clear edges as pseudo label. Ideally, the identified object is the target of interest. Hence, valuable Vision-Language knowledge are transferred from VLPM via the generated image-bbox-text triplets. (2) Leveraging the reversibility inherent in the video, we exploit temporal consistency by conducting tracking in both forward and backward directions. Given the initial state of the target, we first track it forwardly across frames. Subsequently, we treat the predicted object in the last frame as the new target and proceed to trace it backward to the initial frame. Ideally, predictions should be identical regardless of the temporal direction. Hence, the differences between the forward and backward trajectories serve as valuable supervision signals.(3) From a semantic standpoint, the correlation of visual and linguistic features represents the highlights of the tracker's attention, which should remain consistent in bidirectional tracking, mainly focusing on the target.We integrate the consistencies-based learning framework into a baseline method we devised, named Asymmetrical Transformer Tracker (ATTracker). ATTracker is a simple yet efficient real-time tracker, only consisting of an Asymmetrical Multi-source Encoder (AME) and an MLP-based Decoder. It first tokenizes Multi-source information (i.e., template, search, and language prompt) and sends them to the Asymmetrical Multi-source Encoder, which modifies the conventional Multi-Head Attention [48] in an asymmetrical way. Through this adjustment, AME reinforces the representation of target-related features while effectively suppressing irrelevant background information. Additionally, we introduce a special learnable prediction token, denoted as [REG] , and concatenate it with the original tokens from the template, search, and language prompt. It aims to capture the complex relation between multi-modal features for subsequent regression in MLP-based Decoder, functioning similarly to the [CLS] token in standard ViT. Extensive experimental results indicate that without bells and whistles, the proposed semi-supervised tracker achieves comparable performance with the baseline fully supervised VL trackers [31, 53] .To sum up, our main contributions are as follows:• To the best of our knowledge, we propose the first semisupervised learning scheme for Vision-Language Tracking, which provides robust supervision signals via mining consistencies inherent in unlabeled videos.Tracker to verify the effectiveness of our framework, serving as a strong baseline for SS-VLT. • Extensive experiments demonstrate the favorable performance of the proposed method and highlight the potential of learning a VL tracker with limited annotations. 
related work 2.1 unsupervised visual tracking: 
 The pursuit of training a tracker without annotated data has been a longstanding objective since the inception of this task. As a videocentric task, exploring the temporal reversibility and leveraging the consistencies in bidirectional trajectories become a natural avenue for obtaining supervision signals. Kalal et al. [18] explicitly decompose the long-term tracking task into sub-tasks, and then detect tracking failures via forward-backward matching. Lee et al. [24] analyze geometric consistencies between a pair of forwardbackward trajectories to measure the robustness of a base tracker.To harness the capabilities of deep learning, Wang et al. [50] revisit this scheme and train UDT, a Discriminative Correlation Filters (DCF)-based tracker, with the supervision of consistency loss. Subsequently, Zheng et al. [58] propose an unsupervised training approach by initially training from a single frame in the first stage and then adopting cycle training to learn temporal correspondences. Similar ideas can also be found in data annotation and tracker evaluation for TrackingNet [39] . In summary, the success achieved by the aforementioned works confirms the effectiveness of utilizing bidirectional tracking to train robust trackers. However, these approaches merely utilize visual cues and treat tracking as a template-search matching task, relying heavily on the localization ability brought by the correlation filters or CNNs within. Without understanding the semantic of the target, their performances degrade significantly when encountering vast appearance variations of the target. Different from these works, our method focuses on multi-modal perception with limited labels, leveraging multiple consistencies inherent in videos to achieve superior performance in long-term tracking.The integration of vision and language presents promising opportunities for enhancing the robustness and addressing inherent limitations in vision tasks [42] , including tracking [10, 55] . Compared to initializing a tracker only with a bounding box (bbox), employing a language prompt emerges as a more user-friendly and straightforward mode of communication for humans. This approach not only allows for a precise description of the target but also provides comprehensive semantic details such as attributes and category. With the aid of such abundance information, the tracker can effectively tackle the inherent ambiguity caused by bbox and properly handle the extensive appearance variations of the target. As a pioneering work, Li et al. [31] define the task of Vision-Language Tracking and lay the foundation for subsequent researches. Ma et al. [38] introduce the CapsuleTNL, which utilizes the Capsule Network and two interaction routing modules to effectively integrate image and text features. Wang et al. [53] introduce TNL2K, a new large-scale benchmark for VLT, which stands out due to several desirable features: high quality, adversarial samples and significant appearance variation. SNLT [11] improve tracking performance with their Dynamic Aggregation Module to combine both modalities. JointNLT [60] propose a novel framework that reformulates grounding and tracking as a unified task, focusing on the alignment of multi-modal semantic. Nonetheless, these works rely on separate visual and textual encoders to extract features, without direct interactions between modalities during representation learning. In contrast to these approaches, Guo et al. [13] introduce ModaMixer, a novel module aiming to learn a unified visual-language representation for tracking. SATracker [12] further validates the importance of unified feature extraction and interaction. Despite these advancements, all of them require a large amount of human-annotated data for training. Consequently, we propose a pioneering semi-supervised method for the VLT task, aiming to mitigate the dependency on high-quality yet costly labeled data.Vision-Language Pre-Trained Models (VLPMs) have established a dominant presence across vast multi-modal tasks [6, 15, 25, 26] , e.g., visual question answering, cross-modal reasoning, and visual grounding. These VLPMs can be generally divided into two categories according to their training frameworks: coarse-grained and fine-grained. Coarse-grained approaches concentrate on extracting and encoding overall image features with convolutional network [16] or vision transformer [20, 35] . While effective, these methods may not perform as well as fine-grained approaches on objectlevel tasks due to the lack of region-centric features. In contrast, the fine-grained methods [6, 29, 36, 46] utilize a pre-trained object detector [1, 44] as the image encoder. Trained on annotations of common objects [22, 32] , these detectors can identify regions likely to contain objects and perform object classification on each region. However, these methods can only recognize objects within the given categories, such as the 80 object categories in the COCO dataset. There are also some methods [34, 54, 56] attempting to learn both object-level and image-level alignments. In this work, we choose to fine-tune 𝑋 2 -VLM [56] for generating pseudo-labels of objects in each frame based on the language prompt. Notably, 𝑋 2 -VLM distinguishes itself by not relying on a closed-set object detector and learning vision-language alignments in a unified manner, which caters well to our demands of obtaining supervision signals beneficial for multi-modal perception in VLT. 
the proposed method 3.1 overview: 
 As illustrated in Fig. 2 , our semi-supervised learning scheme framework can be generally divided into two stages: data generation and consistences-based training. The first stage in section 3.2 aims to discover the object with semantic that is highly related to the given language prompt. To accomplish this, we fine-tune a Vision-Language Pre-trained Model (VLPM) 𝑋 2 -VLM to visually ground the desire object in each frame, effectively transferring the offthe-shelf Vision-Language knowledge from VLPM to our tracker. The details of the training framework in the second stage are presented in section 3.3, where multi-consistencies from raw videos are exploited to serve as supervision signals. Finally, the baseline Asymmetrical Transformer Tracker (ATTracker) and its learning objective are introduced in section 3.4. Notably, the semi-supervised training setting [47] in Vision-Language Tracking falls conceptually between supervised and unsupervised learning. This setting allows the utilization of large amounts of unlabeled data, commonly available in many scenarios, along with typically smaller sets of labeled data. We adopt the partition setting employed in similar tasks [5, 49] , where the proportion of labeled data remains below 50%. In addition, we introduce a novel partition setting of labeled data for VLT, considering the initialization of the VL tracker. In this setting, the tracker can only leverage the initial information-comprising visual cues and the language prompt in the first frame-apart from the raw video data.When the precise location of the target in each frame is unavailable, the most straightforward and efficient approach for training a Vision-Language (VL) tracker is to generate pseudo bounding boxes that, if possible, closely align with the ground truth [23] . Unlike unsupervised tracking methods that treat randomly cropped regions as pseudo labels [50, 58] , we propose to discover the object of interest in each frame of unlabeled videos. To achieve this, we employ 𝑋 2 -VLM [56] to identify the object most relevant to the given language prompt. It is an open-source Vision-Language Pre-trained Model that is designed to image-level and region-level vision-language alignments in a unified manner. Ideally, it can directly predict the location of the target. 𝑋 2 -VLM mainly consists of a visual encoder built on BEiT2 [41] , a textual encoder adopted from BERT [8] , and a multi-modal fusion module. Trained on vast amounts of datasets and pre-tasks, 𝑋 2 -VLM possesses abundant VL knowledge that can be harnessed for pseudo-label generation. The detailed pipeline of VLPM-driven data generation is illustrated in the left part in Fig. 2 . During fine-tuning, we leverage the static information for initializing the VL tracker, i.e., the bbox 𝐵 𝑔𝑡 0 of the template 𝑇 0 , the search image 𝑆 0 , and the language prompt 𝐿 in the first frame. To prevent high-quality vision-language knowledge of 𝑋 2 -VLM from being disturbed by limited data, we only fine-tune the fusion module for task adaptation while keeping the encoders frozen. Subsequently, the output [CLS] token 𝑋 𝑐𝑙𝑠 from 𝑋 2 -VLM is processd by an MLP to predict the bbox B0 of the object. For fine-tuning the entire model, we use a linear combination of the classic 𝐿 1 loss and the scale-invariant Generalized IoU loss [28] :EQUATIONAfter fine-tuning, the model predicts the bbox 𝐵 𝑝𝑔𝑡 of the most relevant object as pseudo target 𝑇 𝑝 frame by frame, according to the language prompt of the video. Therefore, we are able to effortlessly collect a substantial quantity of noisy yet valuable image-bboxtext triplets that is beneficial for the subsequent training of the VL tracker.Despite the abundance of pseudo labels generated in section 3.2, the presence of noise within these training triplets cannot be ignored.Therefore, directly employing the normal training procedure used by fully-supervised trackers is harmful. To mitigate the negative influence brought by noisy labels, we introduce a training framework that progressively exploit multiple consistencies inherent in raw videos. As depicted in Fig. 2 (b), our framework primarily consists of three phases, each focusing on different aspects based on the inherent attributes of a video: space, time, and semantics. By conducting progressive training under this framework, the differences measured in each phase will altogether formulate a multiple consistencies loss. Optimized by this loss, the asymmetrical transformer tracker can make robust predictions of the target's location with limited ground-truth annotations.Inspired by the strategy of curriculum learning [52] , we decouple the tracking task into subtasks from easy to hard: image-level perception and video-level trajectory prediction. To grasp image-level semantics, the tracker must align multi-modal information to understand what to track in the first place, then regresses the coordinates of the desired object in each frame based on the multi-modal instructions. In an ideal scenario where the target remains stable, exhibiting minimal appearance variations and no occlusions from the initial state, a VL tracker should be able to locate the target without relying on previous contextual information.To extract supervision signals from each image, we propose leveraging spatial consistency by instructing the VL tracker to self-track in the current frame 𝑆 𝑖 . This self-supervised task is based on the observation that an image and any of its sub-regions naturally form a training pair for visual tracking [45] . Specifically, the VL tracker 𝐴𝑇𝑇 𝑟𝑎𝑐𝑘𝑒𝑟 (illustrated in section 3.4) can only use the language prompt 𝐿 𝑖 and the pseudo target 𝑇 𝑝 𝑖 in frame 𝑖 to perform tracking. Subsequently, the differences between the prediction B𝑖 and the bbox 𝐵 𝑝𝑔𝑡 𝑖 of pseudo target 𝑇 𝑝 𝑖 can be measured:EQUATIONOptimized with a loss function emphasizing spatial consistency during Phase I training, the ATTracker acquires the fundamental ability to align and perceive information from both visual and textual modalities for Vision-Language Tracking.To enable video-level trajectory predictions, a robust VL tracker needs to address challenges like large motion and appearance variations. Relying solely on spatial consistency for training is insufficient, as it may result in failures to track objects over long temporal spans or in complex scenes. Therefore, we propose to further train the ATTracker by leveraging temporal consistency inherent in the video.The overall process of Phase II training is illustrated in Fig. 2(b ). Given that the direction of time flow in a video is reversible, a robust tracker should be capable of tracking the target efficiently in both forward and backward directions. Therefore, supervision signals are derived from the discrepancies between the trajectories obtained from forward and backward tracking. Specifically, we initiate forward tracking from a template 𝑇 0 in the initial frame up to frame 𝑘, with a frame interval of 𝑑. At each tracking step, we simultaneously crop the predicted object as an online target 𝑇 𝑜 and concatenate it with the template 𝑇 0 to capture long-term variations of the target. Here, we illustrate the forward tracking process in frame 𝑖 as an example:EQUATIONThen the predicted bbox of forward tracking in frame 𝑘 are regarded as the new target 𝑇 𝑘 . Likewise, we further conduct backward tracking to the original search area 𝑆 0 in the first frame:EQUATIONFinally, the distinctions between the trajectories obtained from forward and backward tracking in each frame are accumulated, forming the loss function across the temporal dimension:EQUATIONBy incorporating temporal consistency and the simple online updating strategy into the training process, the ATTracker can effectively follow the target across multiple frames in both directions, thereby enhancing its ability to handle challenges brought by long-term tracking.Recent studies [33, 43, 59] have revealed that the correlation between visual and linguistic features plays a crucial role in directing attention within their models and improving the performance of dense prediction tasks. Motivated by this, we believe that the region of the target should also be strongly correlated with the textual description in VLT. Moreover, such correlation should remain consistent in bidirectional tracking. Thus, we compute the semantical response map using the search feature 𝑓 𝑆 and the language prompt feature 𝑓 𝐿 in the final layer:𝑚𝑎𝑝 = 𝑢𝑝𝑠𝑎𝑚𝑝𝑙𝑖𝑛𝑔(𝑓 𝑆 ′ 𝑓 𝐿 ′ ⊤).(EQUATIONHere 𝑓 𝑆 ′ and 𝑓 𝐿 ′ are L2 Normalized along the channel, 𝑚𝑎𝑝 ∈ 𝑅 𝐻 𝑢𝑝 ×𝑊 𝑢𝑝 , 𝐻 𝑢𝑝 and 𝑊 𝑢𝑝 are hyperparameters.To prevent a significant shift of the attention in backward tracking, we treat the response map 𝑚𝑎𝑝 𝑓 0 in the initial frame as the pseudo ground truth and compute the consistency of semantic with a binary cross-entropy (BCE) loss:EQUATION)Here 𝜏 represents the temperature coefficient.Previous unsupervised trackers are constructed either with discriminative correlation filters (DCFs) or CNNs, where the matching operation between the template and search areas plays a critical role in fusing visual features. However, the linear nature of this matching operation often limits the tracker's ability to capture the complex nonlinear interactions among features [4] . Moreover, advancements with the transformer architecture has demonstrated its remarkable efficiency in fusing multi-modal representations, particularly for integrating information from both vision and language modalities [42] . Therefore, we propose the first fully transformer tacker for semi-supervised vision-language tracking (SS-VLT), named Asymmetrical Transformer Tracker (ATTracker). The overall architecture is depicted in Fig. 3 . ATTracker takes four types of information as input: static and dynamic templates, search area, natural language, and the learnable prediction token [REG] . The static template represent the state of the target in the initial frame while the dynamic one is periodically updated online at fixed intervals. Its Asymmetrical Multi-source Encoder (AME) primarily comprises 𝑁 layers of asymmetrical attention modules (AAMs), where template tokens are restricted to self-attention (𝑆𝐴) to mitigate noise disruption. Meanwhile, it facilitates the interaction of other tokens for multi-modal feature learning through asymmetrical crossattention (𝐶𝐴). To elaborate, we begin by applying a linear projection to generate queries, keys, and values following the standard transformer procedure, which are subsequently concatenated for the asymmetrical attention operation. Here we denote the static template as 𝑄 𝑡 𝑠 , 𝐾 𝑡 𝑠 , 𝑉 𝑡 𝑠 , the dynamic online template as 𝑄 𝑡 𝑜 , 𝐾 𝑡 𝑜 , 𝑉 𝑡 𝑜 , search areas as 𝑄 𝑠 , 𝐾 𝑠 , 𝑉 𝑠 , the language prompt as 𝑄 𝑙 , 𝐾 𝑙 , 𝑉 𝑙 , and the [REG] token as 𝑄 𝑟 , 𝐾 𝑟 , 𝑉 𝑟 . The queries are concatenated as follows: 𝑄 𝑡 = Concat(𝑄 𝑡 𝑠 , 𝑄 𝑡 𝑜 ), 𝑄 𝑜𝑡ℎ𝑒𝑟𝑠 = Concat(𝑄 𝑠 , 𝑄 𝑙 , 𝑄 𝑟 ), and 𝑄 𝑎𝑙𝑙 = Concat(𝑄 𝑡 , 𝑄 𝑜𝑡ℎ𝑒𝑟𝑠 ). Same for the formulation of keys and values (𝐾 𝑡 , 𝐾 𝑜𝑡ℎ𝑒𝑟𝑠 , 𝐾 𝑎𝑙𝑙 , 𝑉 𝑡 , 𝑉 𝑜𝑡ℎ𝑒𝑟𝑠 , 𝑉 𝑎𝑙𝑙 ). In each layer of AAM: EQUATIONwhereEQUATIONHere 𝐶 denotes the dimension of the 𝑘𝑒𝑦, and 𝐴𝑡𝑡𝑒𝑛 𝑡 and 𝐴𝑡𝑡𝑒𝑛 𝑜𝑡ℎ𝑒𝑟𝑠 represent the attention maps for the targets and others, respectively. The attention maps are then passed through a linear layer and added to their respective original tokens using a residual connection. Afterwards, similar to the [CLS] token used in standard ViT, the outputted [REG] token is sent to a simple MLP to decode the box coordinates. Additionally, the visual and linguistic features of the last layer in AME are leveraged in learning semantical consistency.Overall, the ATTracker is trained progressively with the combination of multiple consistencies-based losses:EQUATIONwhere 𝜆 𝑠 is a hyperparameter. 
experiments 4.1 experiment setup: 
 Implementation Details. The proposed semi-supervised approach is implemented using PyTorch 1.9, running on two NVIDIA RTX 3090 GPUs. We only use the training split of TNL2K [53] , OTB99 [31] , and LaSOT [9] datasets through the experiments. For optimization, we use ADAM [21] with weight decay of 0.1. Initially, we train the ATTracker following the training procedure outlined in Phase I, which spans 200 epochs. The learning rate for the backbone encoder is set to 4 × 10 -5 , while for the MLP-based decoder, it is 4 × 10 -4 . The learning rates are reduced to one-tenth after the 160th epoch. Subsequently, we fine-tune the model during Phases II and III for an additional 100 epochs each. The learning rate for the backbone encoder is adjusted to 2 × 10 -6 , and for the MLP-based decoder, it is 2×10 -5 . Similarly, the learning rates decrease to one-tenth after the 60th epoch. For hyperparameters, the batch size is set to 64 during Phase I and reduced to 16 for Phases II and III. The 𝜆 𝑔𝑖𝑜𝑢 , 𝜆 𝐿 1 and 𝜆 𝑆 are set to 2,5,0.1. The length of search frames is 4 with an interval of 10. in Phase II. The upsample size of the semantical response map is 40 × 40. We set the temperature parameter 𝜏 = 0.07. Datasets and metrics. We assess the effectiveness of our approach on mainstream benchmarks specifically tailored for Vision-Language (VL) tracking: TNL2K and OTB99. Additionally, we evaluate our method on LaSOT, a long-term visual tracking benchmark that includes natural language descriptions. These datasets employ success (SUC), precision (PRE), and normalized precision (Norm. PRE) metrics to measure tracking performance.We conduct a thorough comparison of our semi-supervised learning approach with baseline methods and state-of-the-art (SOTA) trackers under various initialization and supervision settings, as summarized in Table 1 .TNL2K. TNL2K, a dataset comprising 2,000 video sequences, presents significant challenges for VLT due to its high-quality content, presence of adversarial samples, and substantial appearance variations.As shown in Table 1 , we evaluate the proposed ATTracker with SOTA trackers including JointNLT [60] , CTRNLT [30] , SNLT [11] , TNL2K-2 [53] , AutoMatch [57] , USOT [58] , and LUDT [51] . Despite only utilizing labels from the first frame in each video (0.07% of labeled data), our proposed method achieves competitive performance with fully-supervised baselines (i.e., SiamFC [3] , TNLS-III [31] , and TNL2K-2 [53] ). Moreover, when the proportion of labeled data reaches 40%, the proposed ATTracker outperforms all other algorithms except JointNLT by a significant margin (at least 5% in SUC and 11.1% in PRE), irrespective of whether they are supervised or not. These results demonstrate the effectiveness of our approach for SS-VLT, establishing ATTracker as a strong baseline for future comparisons. OTB99. The size of the OTB99 dataset is relatively small, with only 51 videos for training and 48 videos for testing. Compared with unsupervised visual trackers, our method ranks the top among them, surpassing LUDT by 7.4% in SUC and 11.7% in PRE. Meanwhile, the performance of our ATTracke is comparable to that of fully-supervised trackers such as TNL2K-2, SNLT, and JointNLT.LaSOT. LaSOT provides a comprehensive benchmark for visual tracking, focusing on maintaining tracking accuracy over long sequences. With access to 40% labeled data, our semi-supervised tracker performs favorably against fully-supervised SOTA trackers including JointNLT, AutoMatch and SNLT. Notably, among unsupervised trackers, our ATTracker stands out as the top-performing method with only the label of the initial frame (0.07%), surpassing USOT by 15.2% and 15.0% in terms of the SUC and PRE. Qualitative Evaluation. We conduct a visual comparison between the proposed ATTracker and several supervised trackers (such as JointNLT and SNLT), as well as the leading unsupervised method USOT, across four challenging sequences. These videos involve difficulties such as vast appearance variations, view-point changing, semantic understanding, and occlusions. As demonstrated in Fig. 4, our approach performs competitively compared to supervised trackers, even with limited labels.We first perform ablation studies to comprehensively analyze the impact of main phases within the consistencies-based training framework. Then, explorations of key designs within our approach for semi-supervised vision-language tracking are provided. Ablation of Main Phases. To verify the contribution of each phase during training, we conduct ablation studies on six variants of our approach and fix the proportion of labels to 40%. The experimental results of these variants on all datasets are presented in Tab. 2.• In Setting ①, we mainly rely on the pseudo label generated by the 𝑋 2 -VLM for learning spatial consistency in Phase I. In this variant, the ATTracker is trained via selftracking within the same frame, aiming to acquire fundamental image-level perception ability.• In contrast, Setting ② focuses solely on mining temporal consistency via bi-directional tracking in Phase II. The performance significantly degrades across all datasets compared to that of Setting ①, indicating that spatial modeling ability is more crucial for a transformer-based tracker. • Setting ③ incorporates Phase III training to enhance semantical consistency of Setting ①. With proper optimization, the tracker can better perceive target from both modalities. • Similarly, Setting ④ further utilizes the semantical consistency loss on the base of Setting ②, unleashing the power of VL representation learning for VLT. • Setting ⑤ concentrates on utilizing supervision signals from both spatial and temporal aspects. However, without semantic guidance to redirect the tracker's attention, its performance is inferior compared to that of Setting ⑥. • Our approach (Setting ⑥) leverages multi-consistencies from raw videos, achieving outstanding results compared to fully-supervised baselines and unsupervised methods.Impact of Portion of Labeled Data. We follow the partition setting utilized in similar tasks [5, 49] , where the proportion varies from 10% to 50%. Additionally, we introduce a novel partition setting for labeled data, considering how the VL tracker is initialized. In this setting, only the information from the initial frame is provided (0.07% of labeled data). As depicted in Fig. 5 , with the increase of annotation proportion, the performance of the tracker gradually improves. When the proportion of labeled data reaches 40%, the tracker achieves its best performance, which is comparable to that under full supervision. Even with the initial information, our approach achieves competitive results compared with supervised base trackers and significantly surpassing the unsupervised methods. Impact of Pseudo Label Generation. To ensure that the generated pseudo bbox in each frame contains meaningful content (i.e., the object highly related to the given language prompt), we finetune a large VLPM with the initial information in each video. The performance of the fine-tuned 𝑋 2 -VLM shown in Tab. 3 indicates that it indeed possesses abundant VL knowledge that is beneficial for VLT task. Additionally, we employ the original 𝑋 2 -VLM to generate pseudo labels in the Setting without fine-tuning, thereby evaluating the necessity of fine-tuning the VLPM. Impact of Designs in the ATTracker. In the Setting plain outlined in Tab. 3, our asymmetrical attention layers within AT-Tracker are replaced with vanilla transformer layers, allowing other information to interrupt the template tokens. Consequently, this modification results in inferior performances across all datasets. 
conclusion: 
 In this paper, we present a pioneering semi-supervised learning approach for vision-language tracking, which harnesses inherent consistencies in spatial, temporal, and semantical aspects of raw videos. We then propose the ATTracker, a simple yet effective model, to validate the effectiveness of our training pipeline. Extensive experiments confirm that the devised ATTracker serves as a solid baseline, outperforming both fully-supervised base trackers and unsupervised methods. By showcasing the potential of training a tracker with limited labels, we seek to draw interest and lay the groundwork for further exploration into semi-supervised visionlanguage tracking. 
 
 Appendix:  


Decision: This paper was reviewed by four experts in the field. This work Introducing the semi-supervised visual-language tracking task, the proposed temporal consistency alignment is reasonable, the extensive experiments  well validate the effectiveness of the proposed method. 

The reviewers raised concerns regarding experiments, comparison with previous methods, some details of the method, etc. After rebuttal and discussion period, all the reviewers gave positive ratings. The AC checked the rebuttal and was convinced that the authors well addressed this concern. The AC recommend this paper for acceptance.

Reviews: review: paper_summary: paper_summary: This work proposes to perform semi-supervised learning for vision-language tracking. Under the semi-supervised setting, only up to 50% of annotations are available during training. This work also proposes an Asymmetrical Transformer Tracker for the proposed semi-supervised task.
strength_weakness: strength_weakness: 1. Introducing the semi-supervised visual-language tracking task.
2. The temporal consistency alignment is reasonable.
limitations: limitations: 1. Comparing the semi-supervised setting with the unsupervised setting is unfair as shown in Tab.1, as the unsupervised setting doesn't provide any annotation except raw videos and the semi-supervised setting can utilize up to 50% BBoXs and language annotations. Comparing the ``init info'' setting with the unsupervised setting is also unfair, as ATTracker is empowered with a vison-language pretrained model giving clues of the object description and its visual appearance.
2. The ATTracker is claimed to harness inherent consistencies of raw videos while the consistencies are directly measured with the output of the data generated from the pseudo annotator fine-tuned on the vision-language pretrained model. The generated data are considered as pseudo-ground-truth, regarding less potential false positive annotations.


Minimal:
The red point highlights 50% of the annotations for LaSOT in Fig. 5.
Lacking \hat in Eq. 5 for the backward box.
paper_summary: paper_summary: The paper proposes a semi-supervised learning approach for Vision-Language Tracking by leveraging multiple consistencies inherent in raw video data. In particular, the method introduces a progressive training framework on their Asymmetrical Transformer Tracker with three phases focusing on spatial, temporal, and semantic attributes of videos.
strength_weakness: strength_weakness: * The design on consistency learning seems reasonable and interesting.
* The authors provide detailed discussions in the related work about the development of Vision-Language Tracking, which offer a clear view of this field.
limitations: limitations: * Why combine static and dynamic as "template", but search area, language prompt, and special token [REG] as "others" in Asymmetrical Attention Module? Having more meaningful discussions and conducting detailed experiments about the separation of these could enhance the readers' understanding.
* The authors have cited [13], but what is the reason for not comparing to it? Since [13] discussed the asymmetrical design extensively, it is crucial to compare it with the Asymmetrical Attention Module in this work for ablation study.
  - [13] Divert More Attention to Vision-Language Tracking - NeurIPS 2022
* The comparisons to the recent baselines and the experiment on semi-supervised learning are insufficient. Since the main focus of this work is semi-supervised learning, but the comparing baselines are outdated - ICCV 2021 (not even using Natural Language as an additional source), which is not convincing enough to claim the benefits. 
  - Also, here are some recent baselines on supervised learning that the authors are not comparing with:
    - Towards Unified Token Learning for Vision-Language Tracking - TCSVT 2023 
      - TNL2K (SUC): $\textbf{58.6}$, OTB99 (SUC): 70.5, LaSOT (SUC): 70.0
    - All in One: Exploring Unified Vision-Language Tracking with Multi-Modal Alignment - MM 2023
      - TNL2K (SUC): 55.3, OTB99 (SUC): $\textbf{71.0}$, LaSOT (SUC): $\textbf{71.7}$
    - The performance in this work
      - TNL2K (SUC): 56.9, OTB99 (SUC): 69.3, LaSOT (SUC): 63.7
paper_summary: paper_summary: The paper presents a semi-supervised learning approach for Vision-Language Tracking (VLT). The authors argue that existing methods are overly reliant on extensive manual annotations, which are costly and time-consuming. To address this, they propose a novel scheme that leverages inherent consistencies in spatial, temporal, and semantic aspects of videos to generate high-quality pseudo labels and train the tracker with limited annotations. The method is validated using an Asymmetrical Transformer Tracker (ATTracker).
strength_weakness: strength_weakness: (1) The paper introduces a pioneering semi-supervised scheme that significantly reduces the dependency on labeled data.
(2) The paper includes extensive experiments that validate the effectiveness of the proposed framework.
(3) It provides a strong baseline for future research in semi-supervised vision-language tracking.
limitations: limitations: (1) The authors claim that their semi-supervised method outperforms fully supervised method, but they compare their approach under conditions of full supervision with other fully supervised approaches. And there are many fully supervised methods that are superior to the authors' method, such as UVLTrack(AAAI2024).
(2) On OTB99, the authors’ method enhanced with “Init. Info. (0.07%)” and natural language is not better than USOT(ICCV21) and LUDT(IJCV21).
(3) The authors have not conduct experiments to verify the effect of the data generation module on tracking performance.
paper_summary: paper_summary: This paper employs various consistencies to achieve semi-supervised learning in V-L Tracking, particularly the introduction of semantics, which enables the method to surpass previous approaches, including supervised ones. The writing is clear and accessible, and the experiments are thorough.
strength_weakness: strength_weakness: 1. Consistency is a crucial aspect of tracking. Identifying and leveraging these consistencies is vital. This paper is insightful in its use of multiple consistencies, effectively enhancing performance by correlating them.
2. The method leverages the semantic strengths of the V-L model to obtain more tracking results, which can inspire subsequent work.
limitations: limitations: 1. The method adopts approaches from the BERT series instead of using newer LLM methods like Llama or CLIP. Appropriate comparisons or discussions are necessary.
2. There is a lack of discussion on the issue of consistency in semi-supervised related work. It is necessary to supplement the relevant literature.