GLATrack: Global and Local Awareness for Open-Vocabulary Multiple Object Tracking

Authors: guangyao  li
yajun  jian
yan  yan
hanzi  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 13:51:04

Abstract: Open-vocabulary multi-object tracking (MOT) aims to track arbitrary objects encountered in the real world beyond the training set. However, recent methods rely solely on instance-level detection and association of novel objects, which may not consider the valuable fine-grained semantic representations of the targets within key and reference frames. In this paper, we propose a Global and Local Awareness open-vocabulary MOT method (GLATrack), which learns to tackle the task of real-world MOT from both global and instance-level perspectives. Specifically, we introduce a region-aware feature enhancement module to refine global knowledge for complementing local target information, which enhances semantic representation and bridges the distribution gap between the image feature map and the pooled regional features. We propose a bidirectional semantic complementarity strategy to mitigate semantic misalignment arising from missing target information in key frames, which dynamically selects valuable information within reference frames to enrich object representation during the knowledge distillation process. Furthermore, we introduce an appearance richness measurement module to provide appropriate representations for targets with different appearances. The proposed method gains an improvement of 6.9% in TETA and 5.6% in mAP on the large-scale TAO benchmark.

Main paper: 
 introduction: 
 Multiple Object Tracking (MOT) is recognized as a fundamental task in computer vision, primarily aimed at detecting and tracking objects in a video sequence. This task plays a crucial role in various applications, such as autonomous driving [20, 39] and video analysis [29, 48] . While MOT methods [6, 15, 34, 54, 63] have made significant progress in recent years, they are typically trained on predefined data distributions. Additional annotations are required to track unseen targets, which not only restricts the scope of target perception but also hampers the ability to track novel objects in the visual world. Consequently, a disparity emerges between performance evaluation and real-world applications. Several works [24, 30, 31] attempt to address such limitations by leveraging an open-world context. Ošep et al. [30] integrate a classifier following the tracking of detections to enhance general tracking capability in scenarios where no semantic information is accessible. Liu et al. [24] establish the concept of open-world tracking and perform both open and closed-world tracking. Recently, visual language models (VLMs) pre-trained on large-scale imagetext pairs, such as CLIP [33] , have shown remarkable classification capability in various visual tasks. Li et al. [23] first introduces CLIP as an open-vocabulary classifier to detect unseen categories and propose a data hallucination strategy employing static images to achieve data association. However, there are three inherent difficulties to this setup. First, using CLIP directly in MOT is suboptimal, as shown in Fig. 1 . This is because the region features acquired by RoI Align [18] suffer from a significant loss of contextual information, exacerbating the fine-grained differences between image regions and text spans. Second, the semantic information in the appearance of targets exhibits diversity. Therefore, using the same dimension to represent objects with different appearance semantic richness is unreasonable. Third, the adjacent frames over a video contain coherent trajectories and semantic information about the targets. However, only the target's information from the key frames is utilized during the classification phase, neglecting the semantic information available in the adjacent frames. This underutilization hinders the potential of CLIP in the task of open-vocabulary MOT.In this paper, we introduce a novel open-vocabulary MOT method, termed GLATrack, which aims to track unseen objects beyond the pre-defined training categories from both global and instance-level perspectives. Specifically, we devise a plug-and-play Region-aware Feature Enhancement (RFE) module that can be seamlessly integrated into the tracking pipeline, capturing global features from the FPN to enhance the region features. In detail, the RoI Align with the max pooling suffers from the loss of details, while Transformer [40] has considerable potential in capturing contextual information. To address this issue, we leverage a Transformer encoder to establish global relationships, with region features serving as the queries of the decoder. This enables the region features to perceive and comprehend the semantic characteristics based on global features. As a result, we can obtain more discriminative semantic representations for the objects to strengthen the detection and tracking ability.To further improve the capability of GLATrack in tracking generic unknown objects without annotation information, we introduce an Appearance Richness Measurement (ARM) module. This module facilitates fine-grained transformations from small to large dimensions, providing scalable representations for diverse target appearances based on their semantic richness and basic attributes. The ARM module is highly adaptable, avoiding assumptions about fixed representation dimension, allowing the model to acquire essential clues through self-learning for accurate judgment autonomously.In addition to the localization and association branches, the classification pipeline also plays a crucial role in open-vocabulary MOT, which requires identifying targets in scenes with complex camera and object motion patterns. To effectively address these challenges, we propose a Bidirectional Semantic Complementarity (BSC) strategy to optimize semantic representations in key frames. Specifically, the reference frame contains the appearances of objects at different time points, which could provide valuable cues for classification. Consequently, we utilize CLIP to extract semantic information about the matching targets from the reference frame and employ it to filter out erroneous information in key frames that may interfere with object recognition. Subsequently, the refined representations are used to determine the categories of the targets.The main contributions are summarized as follows:• We propose a new open-vocabulary MOT method, GLATrack, which detects and tracks objects of arbitrary categories in the real world from both global and instance-level perspectives.• We propose a flexible plug-and-play RFE module, which effectively leverages global information from the feature pyramid network to complement local targets by leveraging the context modeling ability of the Transformer.• We propose a novel ARM module to generate adaptive appearance representations for objects with diverse semantic richness. Additionally, we introduce a BSC strategy, leveraging valuable information within reference frames to address classification biases arising from occluded object appearances in key frames.• Experimental results demonstrate that the proposed method outperforms the state-of-the-art methods on the large-vocabulary MOT benchmark TAO, showing significant improvement on both TETA and mAP metrics. 
related work: 
 Multi-object Tracking. Some MOT methods [5, 32, 37, 38, 41, 44, 51] typically adopt the tracking-by-detection paradigm [1] , which involve two main stages: detection and association. SORT [4] utilizes Kalman filters and the Hungarian algorithm for motion prediction and frame-to-frame data association. ByteTrack [60] leverages the correlation between detection boxes and tracking trajectories to maintain high-confidence detection results and restore low-confidence detections. Recently, numerous studies have concentrated on utilizing Transformer to improve motion estimation. For example, MOTR [57] and Trackformer [27] framework. Most methods mentioned above work well at detecting and tracking a closed set of categories present in training datasets, while performing blindly in unseen categories in inference. To address this limitation, Dave et al. [10] introduce a large-scale and diverse benchmark TAO [10] . Based on this benchmark, GTR [68] takes a brief sequence of frames as input and computes the global trajectories for all objects. TET [22] computes and compares class exemplars for each localized object to identify potential matching candidates, then conducts class-agnostic association based solely on instance appearance features. However, these methods still suffer from limitations due to pre-defined object categories. Instead, the proposed method aims to achieve effective tracking of novel objects within an open-vocabulary context. Open-Vocabulary Detection and Tracking. Open-vocabulary learning is widely applied in the field of computer vision [21, 42, 49, 50, 53, 58, 65, 66] , which enables the recognition of unseen classes by large vocabulary knowledge. OVR-CNN [56] proposes the concept of open-vocabulary object detection, where image captions are utilized to pre-train the text encoder. Recently, based on the TAO dataset [10] , TAO-OW [24] is proposed as a new benchmark for open-world tracking. OWL-ViT [19] utilizes the Transformer decoder to propagate representations through time, enabling the open-vocabulary detection model [28] to adapt to videos. To transcend pre-defined categories and achieve open-vocabulary MOT, OVTrack [23] leverages CLIP to locate and classify unseen targets. However, that method still suffers from the limitation that using static image classification neglects valuable semantic information in a given video sequence. In contrast, the proposed GLATrack tries to leverage the semantic information contained in adjacent frames to improve the accuracy of object classification.Learning Tracking from Static Images. Recently, several methods [14, 46, 59, 61, 67] employ static images directly to train the track head in MOT and achieve promising performance. For example, KDMOT [59] utilizes a pre-trained teacher model to distill general recognition capabilities into the Re-ID head during the training while maintaining an efficient architecture during inference. QDTrack [14] proposes a quasi-dense similarity learning strategy, achieving competitive tracking performance at the static image level through contrastive learning for the association process. Li et al. [23] introduce a data hallucination strategy based on denoising diffusion probabilistic models [36] to generate pseudo-LVIS videos and apply a fixed Re-ID [62, 64] feature dimension for similarity learning. However, it is less reasonable as the semantic richness of the objects in LVIS that contains 1203 categories varies greatly. Therefore, we propose an adaptive encoding network that encodes the objects with different appearance richness into corresponding Re-ID features for data association. Knowledge Distillation. Knowledge distillation techniques aim to distill the knowledge of Vision-and-Language Models (VLMs) into specific tasks [2, 7, 13, 26] . ViLD [17] employs instance-level visual-to-visual knowledge distillation to integrate open-vocabulary knowledge into a two-stage detector. HierKD [26] introduces a global-level knowledge distillation module, which aligns globallevel image representations with caption semantics. OADP [43] employs instance-level distillation along with global and block distillation methods to construct relationship information among objects during distillation. PCL [8] adopts an image captioning model to generate detailed labels that describe object instances from multiple perspectives. Instead of directly applying image knowledge distillation to open-vocabulary MOT, our GLATrack exploits the auxiliary role of the reference frames to aggregate the features of the same target across various spatiotemporal contexts, thereby obtaining precise and comprehensive object information. 
methodology 3.1 overview: 
 The overview architecture of the proposed method is shown in Fig. 2 .GLATrack is designed to track objects beyond predefined categories. Initially, global features 𝐹 𝑔 are extracted by a pre-trained backbone, and then the RPN generates region proposals, which are fed into the RoI Align to produce fixed-size features denoted as 𝐹 𝑟 . Based on 𝐹 𝑔 , the RFE module enables the region features 𝐹 𝑟 to retrieve and restore their prominent characteristics that may have been lost during the pooling process. Utilizing the enhanced features 𝐹 𝑠 as input, the ARM module and BSC strategy are employed to improve tracking and classification performance, respectively.In The input of the RFE module is the region proposals generated by RPN for each frame in the video sequences. Specifically, given the key frame t, RPN generates the object candidates R (𝑛) = {r 1 , r 2 , ..., r 𝑛 }, represented by the 2D bounding box coordinate b = [𝑥, 𝑦, 𝑤, ℎ] and the corresponding object confidence score p ∈ [0, 1]. Aligning the variable number of detected targets in each frame to the same dimensions becomes challenging. To address this, we introduce a learnable query M of size 𝑁 × 𝐷, 𝑁 is the max number of objects in all batches and 𝐷 equals 𝑤 × ℎ.EQUATIONwhere 𝑛 indicates the number of objects within a batch, q 𝑖 denotes the region query corresponding to r 𝑖 , m 𝑗 is the masked query. For the global feature 𝐹 𝑔 , we employ a Transformer encoder to model the global relationship further, obtaining refined features denoted as 𝐹 𝑒 , which are used to enhance the region features. Technically, we extend region candidates R (𝑛) to region queries within the Transformer, and the representations of these queries are updated in the Transformer decoder. Specifically, region queries are updated through cross-attention with 𝐹 𝑒 . where the masked queries generate values during the update process. These masked queries will be re-masked after each decoder layer to prevent them from influencing region queries. The updated region queries are further translated to enhance region features f 𝑠 𝑖 . Such a way ensures that region features capture the global image context comprehensively, significantly reducing the disparity between the entire image and region distributions.EQUATIONThe existing open-vocabulary MOT method, OVTrack [23] , connects the Faster R-CNN with CLIP directly, which ignores the rich target information that adjacent frames may contain. To alleviate this limitation, the BSC strategy is proposed to incorporate feature information from the reference frame into the CLIP classification process, as shown in Fig. 3 .To accelerate inference speed, we employ the CLIP image encoder to supervise the training of the image head, which generates the embeddings I = {i 𝑠 𝑘 , i 𝑠 𝑟 } ∈ R 𝐻 ×𝑊 ×𝑑 for each enhanced feature 𝑓 𝑠 . To ensure the image embedding fusion at a more granular level, we separately apply 3×3 filters to i 𝑘 and i 𝑟 to derive deeper-level embeddings e 𝑘 and e 𝑟 . Subsequently, three independent full-connected layers transform the visual features as 𝑸, 𝑲 , and 𝑽 :EQUATIONwhere 𝑾 𝑞 , 𝑾 𝑘 , 𝑾 𝑣 are the learnable weight, 𝑝 𝑘 and 𝑝 𝑟 are the position embeddings, respectively. We then perform matrix multiplication with 𝑄, 𝐾, and 𝑉 , followed by weighting the language features using the resulting similarity matrix. Subsequently, the e 𝑘 are combined with the key frame features influenced by the reference frame to obtain the fused embeddings e 𝑠 𝑘 ∈ R 𝐻𝑊 ×𝑑 :EQUATIONwhere 𝑑 is the feature dimension. Similarly, the features of the reference frame undergo the process mentioned above to produce the e 𝑠 𝑟 ∈ R 𝐻𝑊 ×𝑑 . ê = 𝐷𝑒𝑐𝑜𝑑𝑒𝑟 (𝑐𝑜𝑛𝑐𝑎𝑡 (e 𝑠 𝑘 , e 𝑠 𝑟 )).By concatenating e 𝑠 𝑘 and e 𝑠 𝑟 along the channel dimension and decoding them, restored embeddings ê ∈ R 𝐻𝑊 ×𝑑 are aligned with the counterpart embeddings generated by the CLIP image encoder.Current MOT methods in open-vocabulary environments heavily rely on appearance cues for data association where motion cues are delicate due to various camera and object motion patterns. However, restricting target appearance to uniform dimensions hampers the capability of the model to grasp the intrinsic semantic information in these appearances, making learning robust target appearances challenging. To address this issue, the proposed ARM module characterizes the target appearance using specific feature dimensions corresponding to their appearance semantic richness. Specifically, given the input image pairs (𝑥 𝑘𝑒𝑦 , 𝑥 𝑟𝑒 𝑓 ), we extract corresponding enhanced RoIs (𝐹 𝑠 𝑘 , 𝐹 𝑠 𝑟 ). As each 𝑓 𝑠 ∈ 𝐹 𝑠 encapsulates semantic features of the target, we convert it into a semantic assessment score 𝑠𝑐𝑜𝑟𝑒 𝑡 .EQUATIONwhere 𝐷 is the predefined maximum representation scale. However, as different 𝑓 𝑠 are represented with equal information content, a single 𝑠𝑐𝑜𝑟𝑒 𝑡 cannot fully capture its informational significance in the input image. Therefore, we introduce 𝑠𝑐𝑜𝑟𝑒𝑠 𝑜 , derived from the fundamental information 𝑁 (𝑤, ℎ, 𝑢, 𝑟 ) of 𝑓 𝑠 in the original image, where 𝑤 and ℎ represent the width and height, 𝑢 indicates the area and 𝑟 denotes the proportion to the entire image. Subsequently, 𝑠𝑐𝑜𝑟𝑒 𝑡 is adaptively combined with the computed 𝑠𝑐𝑜𝑟𝑒𝑠 𝑜 .EQUATIONwhere 𝜆 serves as a trainable parameter. Finally, 𝑠𝑐𝑜𝑟𝑒 𝑑 is utilized to determine the representation scale, with the position associated with the highest score indicating the scale of 𝑓 𝑠 .During the process of transferring CLIP knowledge to our task, we utilize L1 loss to supervise the training of the image head. The goal is to minimize the distance between ê𝑖 and 𝛽 𝑖 .EQUATIONwhere 𝛽 𝑖 is the corresponding image embedding encoded by the CLIP image encoder. For the class label of 𝑓 𝑠 , we generate text prompts A (ĉ), which consists of 𝐿 context vectors c 𝑣 and a class name embedding c 𝑛 . Then the CLIP text encoder E takes the input of the prompt and outputs the corresponding text embedding t 𝑐 , ∀c ∈ C base . We calculate the similarity between the predicted embeddings t 𝑝 and their corresponding CLIP embeddings t 𝑐 to synchronize them.EQUATIONwhere 𝑐𝑜𝑠 (•, •) indicates the cosine similarity, 𝜆 denotes a temperature parameter and t 𝑏 represents a learned background prompt.EQUATIONThe cross-entropy loss L CE is calculated using 𝑐𝑟 as the class label of 𝑟 . We employ contrast learning to achieve robust tracking based on appearance cues. For the matched RoIs in the reference frame 𝑥 𝑟𝑒 𝑓 , we group objects with the same identity into 𝐾 + and separate objects with different identities into 𝐾 -.EQUATIONEQUATIONwhere 𝛼 equals1 |𝐾 + (k) | .𝑘 is feature embedding of the training sample in the key frame, 𝑘 + and 𝑘 -denote its positive and negative targets, respectively. During the inference phase, given the enhanced RoIs, existing trajectories T are directly associated with targets 𝑟 ∈ R using appearance feature similarity. The similarity between targets 𝜏 within trajectories and candidate targets 𝑟 is measured using both bidirectional softmax and cosine similarity.f (𝜏, 𝑟 ) = 1 2 exp(u 𝑟 • v 𝜏 ) 𝑟 ′ ∈ R exp(u 𝑟 ′ • v 𝜏 ) + exp(u 𝑟 • v 𝜏 ) 𝜏 ′ ∈ T exp(u 𝑟 • v 𝜏 ′ ).The higher bi-softmax value indicates that the distance between the two matched objects in the feature space is reduced. If f (𝜏, 𝑟 ) > 𝛼, the target with the maximum similarity is added to the current trajectory. If objects 𝑟 fail to match any existing tracks but maintain a confidence score higher than 𝜎, a new track will be initiated. In the classification pipeline, the enhanced features 𝑓 𝑠 serve as inputs to the distilled image head, generating corresponding image embeddings ê. These image embeddings, along with generated text embeddings, are used to compute the similarity for determining the category of the targets. 
experiments 4.1 datasets: 
 We conduct experiments on the publicly available large-vocabulary MOT benchmark TAO [10] . GLATrack is trained on the hallucinated LVIS dataset [23] within the base categories. We evaluate GLATrack on both the validation and test sets of TAO. LVIS Dataset. Unlike MOT datasets [9, 11, 55] with limited object classes, LVIS offers annotations for a large vocabulary of object categories, covering a wide spectrum of real-world objects. Following the OVTrack [23] , we use their hallucinated LVIS dataset. Compared with TAO, this dataset simulates object motion in an open-world context with a smaller dataset size, making it suitable for open-vocabulary MOT tasks, especially in scenarios with limited computational resources. In this dataset, we employ the 886 frequent and common categories as the base classes C 𝑏𝑎𝑠𝑒 , while reserving the 337 rare categories as the novel categories C 𝑛𝑜𝑣𝑒𝑙 . TAO Dataset. The TAO dataset is a comprehensive video dataset designed to provide rich visual and semantic data for diverse video analysis tasks. A notable feature of the TAO dataset is its openness, allowing tracking of any object rather than being limited to specific TETA. TETA [22] is built upon the HOTA [25] , extending its functionality to address challenges posed by long-tail scenarios. TETA is composed of three parts: localization score (LocA), classification score (ClsA), and association score (AssocA). The TETA score is obtained by the arithmetic mean of these three scores. Track mAP. Track mAP [52] offers a comprehensive evaluation by considering both the detection accuracy and the ability to associate targets. This assessment involves measuring the overlap between the predicted bounding boxes and the ground truth.We use Faster R-CNN [35] with ResNet-50 [16] as the open-vocabulary detector. The model is trained on 4 GPUs, with 2 images per GPU.Our training process consists of two stages. Initially, we train the open-vocabulary object detector following [13, 17] . Then we finetune the model for tracking, assigning loss weights of 0.25 for L track and 1.0 for L aux based on [14, 23] . During training, non-maximum suppression (NMS) is applied to filter boxes 𝑅 with an IoU threshold of 0.7, and we randomly select |𝑅| boxes per image (|𝑅| = 256). For inference, boxes with an IoU threshold of 0.5 are selected. We employ the SGD optimizer with an initial learning rate of 0.02 and set the weight decay and momentum to 0.0001 and 0.9, respectively. In terms of track management, each track maintains a history of 10 frames, with parameters 𝛼 = 0.5 and 𝜎 = 0.0001. Open-vocabulary MOT. We compare GLATrack with other methods on the challenging MOT benchmark TAO [10] . Closed-set trackers, such as QDTrack [14] and TETer [22] , are trained on the entire LVIS dataset, while DeepSORT [45] and Trackor++ [10] are integrated with the open-vocabulary detector ViLD [17] for openvocabulary MOT. OVTrack [23] , along with the aforementioned methods, are trained only on the base categories of LVIS. The evaluation results on the validation and test sets of TAO are presented in Tab. 1, which are divided into two parts: base categories and novel categories. The proposed method achieves significant performance on the benchmark, outperforming comparison methods across the base and novel categories on all metrics. Specifically, GLATrack achieves the highest score of 37.9 TETA in base classes and 31.0 TETA in novel classes on the validation set. Although QDTrack [14] and TETer [22] are trained on both C 𝑏𝑎𝑠𝑒 and C 𝑛𝑜𝑣𝑒𝑙 , their performance is unsatisfactory compared with GLATrack, which is trained only on base classes yet achieves superior detection results on both base and classes. Additionally, GLATrack outperforms the first open-vocabulary tracker, OVTrack [23] , by a substantial margin, achieving +2.4 TETA on base classes and +3.2 TETA on novel classes, respectively. Similarly, GLATrack also achieves excellent performance on the TAO test set. The competitive results demonstrate that GLATrack can effectively tackle object tracking in complex scenarios.Closed-set MOT. We evaluate GLATrack against existing methods on the TAO validation set using TETA and Track mAP, as shown in Tab. 2 and 3. GLATrack and OVTrack [23] are both trained on the base categories using static images, while other closed-set trackers are trained on video datasets, which include these unseen categories. As AOA [12] integrates multiple few-shot detection and Re-ID models trained on other datasets, it achieves a higher classification score than other methods. However, GLATrack outperforms previous methods in almost all evaluation metrics. Specifically, GLATrack achieves the highest TETA score of 37.1 and AssocA score of 38.5, surpassing OVTrack by 2.4 points in TETA and 1.8 points in AssocA. For Track mAP, GLATrack achieves 22.2 Track mAP50 and 11.5 Track mAP75 on the validation set, which are 1.0 points and 0.9 points higher than the baseline, respectively, resulting in an overall performance improvement of 0.9 points. These improvements show that GLATrack exhibits excellent performance We conduct ablation studies on the validation set of TAO to assess the effectiveness of the modules in the proposed GLATrack, using TETA as the main evaluation metrics for the ablation experiments. The qualitative comparison results illustrated in Fig. 4 demonstrate the capabilities of both OVTrack and GLATrack in tracking novel targets. However, in the left side results, OVTrack encounters an ID switch and target loss when the squirrels are occluded and blend with the background color in the second and third frames, respectively. In contrast, GLATrack maintains a correct ID and successfully locates the targets. Additionally, the right side results depict challenges of fine-grained classification in target occlusion scenarios. OVTrack exhibits an unclear position of the elephant in the first frame and loses its position in the third frame, whereas GLATrack accurately localizes the target. We present the failure cases of the proposed method: throughout the tracking process, GLATrack struggles to differentiate the tree trunk next to the pool due to limited valuable information, yet it maintains stable tracking. 
conclusion: 
 In this paper, we propose GLATrack, a novel open-vocabulary multiobject tracking framework. Recognizing the importance of robust semantic representation, we designed an RFE module that utilizes the powerful contextual modeling capability of Transformer to enhance pooled region features. To efficiently and accurately detect novel objects, we propose a BSC module to distill knowledge from large-scale vision-language models, leveraging information from both key and reference frames. Additionally, we introduce an ARM module to determine the representation scale of the target based on fundamental attributes and semantic richness, facilitating robust data association relying on appearance cues. Extensive experiments on the large-scale, large-vocabulary TAO benchmark demonstrate that the proposed method outperforms existing trackers in detecting and tracking objects across both base and unseen novel classes, achieving state-of-the-art performance. 
 
 Appendix:  


Decision: The paper was reviewed by four experts in the field and received three borderline accept and one weak reject in the final ratings. The paper focuses on the challenging open-vocabulary multi-object tracking (MOT) by integrating global and local awareness, semantic complementarity, and appearance richness measurement. Its pros including reasonable idea, good results and sufficient ablation study, significant improvements than previous sota methods like TETA, etc. Reviewer KrCU questioned the unclear motivation, reliance on high-quality annotations, and vague description of Region-aware Feature Enhancement module in Section 2.3. The AC checked the rebuttal and was convinced the authors well addressed these concerns. Given the recommendations from three reviewers, the AC recommend this paper for acceptance.

The authors are required to include the comparison with OVTrack, ablation studies, motivation, and so on in the rebuttal to the final version. A through proofreading/polishing is suggested to improve clarity and avoid typos.

Reviews: review: paper_summary: paper_summary: This paper proposed a Global and Local Awareness open-vocabulary MOT method (GLATrack), which learns to tackle the task of real-world MOT from both global and instance-level perspectives. Experiments on TAO shows its effectiveness.
strength_weakness: strength_weakness: 1. This paper is well-organized, and the idea is reasonable, which is a incremental work indeed.
2. We propose a flexible RFE module to leverage global information and local targets, and ARM module to generate adaptive appearance representation, 
3. Experiments are sufficient to validate its effectiveness.
limitations: limitations: 1. This paper just improves the OVTrack from global and local awareness features, but OVTrack combines OV Object Detection and MOT.  GLATrack seems just a improvement of MOT, which lacks of novelty. 
2. The RFE, BSC and ARM module are designed for the association of MOT, so are they benefical for MOT methods, such as ByteTrack, OCSORT ....
paper_summary: paper_summary: The paper proposes a method called GLATrack (Global and Local Awareness open-vocabulary Multi-Object Tracking) to address challenges in multi-object tracking (MOT) where arbitrary objects may be encountered beyond the training set. Traditional methods primarily focus on instance-level association and identification of novel objects, often neglecting the fine-grained semantic representations of targets within key and reference frames. GLATrack incorporates both global and instance-level perspectives to improve tracking performance.
strength_weakness: strength_weakness: This paper presents several strengths:
This paper introduces a novel approach to open-vocabulary multi-object tracking (MOT) by integrating global and local awareness, semantic complementarity, and appearance richness measurement. These components address the limitations of existing methods that primarily focus on instance-level association and identification of novel objects. The method achieves significant improvements in tracking accuracy compared to state-of-the-art methods, demonstrating its effectiveness in open-world scenarios where tracking arbitrary objects is essential. Specifically, it shows a 6.9% increase in TETA and a 5.6% increase in mean Average Precision (mAP) on the large-scale TAO benchmark.
limitations: limitations: While the paper addresses an important challenge in open-vocabulary multi-object tracking and introduces novel approaches, I have raised concerns regarding several aspects of the work:

1. The metrics for base+RFE in Table 4 are not significantly different from those for base+RFE+ARM, making it difficult to assess ARM's advantages. The authors should conduct additional experiments to provide further evidence.

2. In Figure 5, the authors did not explain in the text that OVTrack focuses more on objects compared to GLATrack, contrary to the description in the caption.

3. The authors need to review some spelling errors, such as "$\mathcal{L}CE$" -> "$\mathcal{L}_{CE}$" on line 528.
paper_summary: paper_summary: This paper introduces GLATrack, a novel Global and Local Awareness open-vocabulary Multi-Object Tracking (MOT) method. GLATrack integrates fine-grained semantic representations of targets from both global and instance-level perspectives. Specifically, the method includes a region-aware feature enhancement module that refines global knowledge to complement local target information, thereby bridging the distribution gap between image feature maps and pooled regional features. Additionally, this paper proposes a bidirectional semantic complementarity strategy to address semantic misalignments caused by missing target information in key frames, by dynamically selecting valuable information from reference frames to enrich object representation during the knowledge distillation process. An appearance richness measurement module is also introduced, providing appropriate representations for targets with varying appearances.
strength_weakness: strength_weakness: 1.GLATrack introduces innovative components like the Region-aware Feature Enhancement (RFE) Module and Appearance Richness Measurement (ARM) Module that advance open-vocabulary MOT by integrating global and local semantic information to improve the tracking accuracy of diverse object types in complex scenes.
2.By leveraging transformers to refine region features and integrating bidirectional semantic complementarity, GLATrack effectively bridges the theoretical aspects of semantic understanding with practical multi-object tracking challenges. This enhances the system's ability to handle semantic misalignments and appearance variations among objects.
3.The method might inherit biases from pre-trained models used in the system, particularly in scenarios not well represented in training data. This could affect the system's performance in diverse or less common tracking scenarios.
limitations: limitations: 1.In the paper, Figure 1 does not effectively illustrate the motivation behind the study, which is crucial for understanding the context and significance of the research. Additionally, the abstract could provide a clearer description of the motivation. Enhancing these aspects would not only help clarify the initial impetus for the study but also enable readers to grasp the importance and potential impact of the research more intuitively. It is recommended that the authors revise Figure 1 to better reflect the foundational concepts and motivations of the study, and refine the abstract to explicitly highlight the key drivers and objectives of the proposed method.
2.The performance enhancements reported are heavily reliant on the availability of high-quality annotations and extensive pre-training on large datasets, which might not be feasible in all real-world applications where such detailed annotations are unavailable.
3.The description of the Region-aware Feature Enhancement module in Section 2.3 is somewhat vague. Providing specific examples of how this module interacts with other components of the system or including pseudocode could enhance understanding.
paper_summary: paper_summary: This paper proposes a new open-vocabulary MOT method, GLATrack, which detects and tracks arbitrary classes of objects in the real world from a global and instance-level perspective. GLATrack proposes the RFE module to solve the problem of aligning image regions with region-level descriptions when using CLIP. Then, ARM is proposed for generating adaptive appearance representations for objects with different semantic richness. BSC is proposed to utilize valuable information in reference frames to resolve classification bias caused by the presence of occluding objects in key frames.
strength_weakness: strength_weakness: (1) For objects with different semantic richness, ARM novelly and adaptively generates different appearance representations, effectively improving the model's correlation capabilities.
(2) BSC alleviates the problem of misclassification caused by occlusion by fusing information in reference frames.
(3) GLATrack has achieved good results, and the ablation experiments of related modules are relatively sufficient, which proves the effectiveness of the method used in this article.
limitations: limitations: (1) It is known from the experimental part that among the three proposed modules, RFE plays a major role. But it seems to work similarly to the method in RegionCLIP. The author needs to explain the difference between the two.
(2) As mentioned in line 405, contrastive learning-based methods for association are deployed in RFE. The author should perform ablation experiments on the role played by this part.