## 3 Methodology
### Cross-Modal Embeddings and Fine-Tuning of VLMs  
To create a robust cross-modal embedding for multi-object tracking (MOT), our approach integrates visual and textual representations derived from image data. These embeddings are generated from different datasets, including MOT15, MOT16, MOT17, DanceTrack, CrowdHuman, and WildTrack, and are designed to be compatible with several VLM architectures, such as Florence-2, CLIP, OpenCLIP, and DETR2. The embeddings for each VLM are constructed by associating the extracted visual features with corresponding textual or categorical descriptions. This process is repeated for each image in the dataset, resulting in structured data stored in various formats suitable for the target VLMs:  
* Florence-2 Embedding incorporates visual features with bbxes, keypoints, descriptors, overlap ratios, and confidence scores, labeling each entity as a \"pedestrian\". * CLIP and OpenCLIP Embeddings convert visual features into textual descriptions, detailing the position, keypoints, descriptors, and confidence scores of each human figure within the frame. * DETR2 Embedding organizes features in a format that includes bbxes, keypoints, descriptors, and overlap ratios, compatible with the DETR2 framework. These data are stored in a DETR2-specific JSON format, allowing for seamless integration with downstream tasks. To create effective cross-modal embeddings for MOT tasks, we first define a multi-dimensional vector representation for each tracklet. The embedding vector \(e_{i}\) for tracklet \(i\) is composed of various features as given in Equation (7):  
\[e_{i}=[v_{i},k_{i},d_{i},b_{i},c_{i},r_{i}]\]where  
* \(v_{i}=[f_{n},h,w]\) represents the visual features, including the image file name \(fn\), image height \(h\), and image width \(w\). * \(k_{i}\) is the keypoint vector extracted using XFeat, representing the spatial configuration of the detected human's keypoints. * \(d_{i}\) is the 64-dimensional descriptor vector derived from LightGlue, encoding the appearance characteristics. * \(b_{i}\) represents the bbx \([x,y,w,h]\), capturing the spatial extent of the detected object. * \(c_{i}\) is the confidence score associated with the detection. * \(r_{i}\) is the overlap ratio of bboxes across frames, used for temporal consistency. It measures the degree of spatial overlap between different bboxes. This information is crucial for managing occlusions and ensuring accurate tracking across frames. These features are concatenated to form a unified embedding \(e_{i}\) that captures both visual and spatial-temporal information. To bridge the gap between different modalities (e.g., image and textual descriptions), we project the embeddings into a shared latent space \(L\). Let \(W_{v}\) and \(W_{t}\) be the projection matrices for visual and textual features, respectively. The projection can be formulated as given in Equation (8):  
\[z_{i}^{v}=W_{v}\cdot e_{i}^{v},z_{j}^{t}=W_{t}\cdot e_{j}^{t} \tag{8}\]  
where  
* \(z_{i}^{v}\) is the projected visual embedding for tracklet \(i\). * \(z_{j}^{t}\) is the projected textual embedding for the corresponding textual description \(j\). The loss function for aligning the modalities is typically based on contrastive learning as given in Equation (9):  
\[L_{contrast}=-log\frac{exp(cos(z_{i}^{v},z_{j}^{t})/\tau)}{\sum_{k}exp(cos(z_{ i}^{v},z_{j}^{t})/\tau)} \tag{9}\]  
where  
* \(cos(\cdot)\) represents the cosine similarity between the projected embeddings. * \(\tau\) is a temperature parameter that controls the sharpness of the similarity distribution. For fine-tuning VLMs, we leverage the cross-modal embeddings to refine the decision boundaries in the joint embedding space.