###### Abstract
The increasing scale of data and demand for real-time processing highlight the need for more efficient algorithms, but the high computational complexity limits their deployment in real-time systems. Additionally, many MOT frameworks under-use rich contextual information like keypoints and descriptors, leading to suboptimal performance and challenges with track consistency and false positives. Addressing these limitations requires advanced embedding techniques,  
Figure 1: Diverse and challenging environments captured in the MOT15, MOT16, MOT17, MOT20, DanceTrack, and WildTrack datasets. improved use of contextual data, and more scalable algorithms to enhance performance across diverse applications. Figure 2 illustrates these issues, showing tracking failures due to IDSws, occlusions, and inefficiencies in current systems. In the first frame (\(t\)), three individuals (ID1, ID2, and ID3) are correctly detected. In the next frame (\(t+1\)), ID1 and ID3 are still tracked (TP), but ID2 is occluded or exits, causing a false negative (FN). A new individual (ID4) enters but is mistakenly identified as ID2, leading to an IDSw. By the third frame (\(t+2\)), ID1 is missed (FN), ID3 remains correctly tracked (TP), and ID2 reappears but is confused with ID5, causing another IDSw. Additionally, ID5 is falsely introduced (FP), while ID4 is correctly tracked, reflecting a true negative (TN). This highlights how IDSws, FNs, and FPs degrade tracking, especially in dynamic, crowded scenes. Traditional MOT methods often suffer from IDSws and FM, which degrade tracking accuracy and continuity, particularly in crowded and complex environments. ReTrack-VLM aims to tackle these challenges by leveraging the advanced detection capabilities of YOLOv8, the robust feature representation of VLMs, and the zero-shot ReID capability for associating tracks across frames. By integrating these components into a unified pipeline, the proposed method reduces IDSws and FM, offering a robust and scalable solution for MOT. The embedding of ReTrackVLM, which is generated using XFeat [6] and LightGlue [7], encapsulates spatial and appearance features essential for accurate tracking. Integrated with fine-tuned VLMs like Detectron2 (DETR2) [8], Florence-2 [9], and CLIP [10], ReTrackVLM evaluates whether a query bbx belongs to an existing track, improving accuracy by reducing IDSw. The system's track management module, combined with confident track storage and a zero-shot re-identification module, ensures consistent long-term tracking, even when objects reappear after occlusion. ReTrackVLM delivers a robust, scalable tracking system for real-world applications. We can summarize the main contributions of this work as follows:  
* **Novel Embedding Structure:** We propose a novel embedding structure that combines bbxes, keypoints, descriptors, overlap ratios, and confidence scores, providing a comprehensive representation of detected objects in MOT tasks. Figure 2: Key challenges in MOT systems, including IDSws, occlusions, and failures in re-id, as well as issues related to computational complexity and data integration across consecutive frames, which highlights the difficulties in maintaining consistent object identities due to occlusions, object interactions, and changes in the scene. * **Integration with VLMs:** ReTrackVLM leverages fine-tuned VLMs, such as DETR2, Florence-2, CLIP, and OpenCLIP, to improve the accuracy and robustness of object tracking by determining the association between bbxes and existing tracks. * **Enhanced Track Management:** We introduce a track management module that utilizes a confident track storage system, ensuring consistent track maintenance and reducing the occurrence of IDSws in challenging scenarios. * **Zero-Shot ReID:** A zero-shot ReID module is incorporated, allowing the system to match newly detected tracks with previously established ones without requiring extensive retraining, thereby enhancing the adaptability of the system to varying environments.