DOVTrack: Data-Efficient Open-Vocabulary Tracking
Authors: Zekun Qian
Ruize Han
Zhixiang Wang
Junhui Hou
Wei Feng
Venue: NeurIPS 2025 poster ()
Keywords: Open Vocabulary
Multi-Object Tracking
Primary area: applications
TL;DR: Efficient Data Utilization in OVMOT Training.

Abstract: Open-Vocabulary Multi-Object Tracking (OVMOT) aims to detect and track multi-category objects including both seen and unseen categories during training.
		Currently, a significant challenge in this domain is the lack of large-scale annotated video data for training.
		To address this challenge, this work aims to effectively train the OV tracker using only the existing limited and sparsely annotated video data.
		We propose a comprehensive training sample space expansion strategy that addresses the fundamental limitation of sparse annotations in OVMOT training. Specifically, for the association task, we develop a diffusion-based feature generation framework that synthesizes intermediate object features between sparsely annotated frames, effectively expanding the training sample space by approximately 3× and enabling robust association learning from temporally continuous features. For the detection task, we introduce a dynamic group contrastive learning approach that generates diverse sample groups through affinity, dispersion, and adversarial grouping strategies, tripling the effective training samples for classification while maintaining sample quality. Additionally, we propose an adaptive localization loss that expands positive sample coverage by lowering IoU thresholds while mitigating noise through confidence-based weighting. Extensive experiments demonstrate that our method achieves state-of-the-art performance on the OVMOT benchmark, surpassing existing methods by 3.8\% in TETA metric, without requiring additional data or annotations. The code will be available at https://github.com/zekunqian/DOVTrack.

DOVTrack: Data-Efficient Open-Vocabulary Tracking
Zekun Qian",, Ruize Han"; Zhixiang Wang', Junhui Hou', Wei Feng'
'College of Intelligence and Computing, Tianjin University,
Shenzhen University of Advanced Technology. *City University of Hong Kong {clarkqian, zhixiang_ wang wfeng)@tju. edu. en, hanruizedsuat-aa wtt.s/uutttssee......... edu.ed...c. en, jo.houuciyyu.ere edu.
Abstract
Open- Vocabulary Multi-Object Tracking (OVMOT) aims to detect and track multi- category objects including both seen and unseen categories during training. Cur- rently, significant challenge in this domain is the lack of large-scale annotated video data for training. To address this challenge, this work aims to effectively train the OV tracker using only the existing limited and sparsely annotated video data. We propose a comprehensive training sample space expansion strategy that addresses the fundamental limitation of sparse annotations in OVMOT training. Specifically, for the association task, we develop a diffusiin-bbsed feature gener- ation framework that synthesizes intermediate object features between sparsely annotated frames, effectively expanding the training sample space by approxi- mately 3x and enabling robust association learning from temporally continuous features. For the detection task, we introduce dynamic group contrastive learn- ing approach that generates diverse sample groups through affinity, dispersion, and adversarial grouping strategies, tripling the effective training samples for classification while maintaining sample quality. Additionally, we propose an adaptive localization loss that expands positive sample coverage by lowering IoU thresholds while mitigating noise through confidence-bart based weighting. Extensive experiments demonstrate that our method achieves sate-of-theear performance on the OVMOT benchmark, surpassing existing methods by 3.8% in TETA metric, without requiring additional data or annotations. The code will be available at https: / //gg//gttttt/ttt com http//euiniia/DVTrrc....
Introduction
nren-Vocabulary Multi-Object Tracking (OVMOT) aims to track objects of any given category within a scene, including unseen classes during training [1]. Unlike traditional Multi- Multi-OOject Tracking (MOT) tasks, OVMOT is not limited to specific categories of objects, such as pedestrians and vehicles; instead, can handle a broader range of object categories. This capability enhancement significantly improves the applicability of tracking problems but introduces greater challenges. One major challenge is building datasets that are both large-scale and diverse in object categories. However, most existing large MOT datasets [2-6] often suffer from limited categories, making it difficult to train effective OVMOT algorithms.
To obtain effective training data, most current approaches [1, 7, 8] primarily rely on large-scale image datasets like LVIS [9], which facilitates OVMOT training by augmenting single images into image pairs. While such image data can supplement target categories and increase annotation scale, they are fundamentally limited by their statie nature and lack of temporal information. This limitation restricts the model's is ability to learn essential dynamic features of videos, such as object deformations, viewpoint changes, and gradual occlusion. Therefore, while modern OVMOT research relies on these image datasets, training robust OV tracker using continuous video data appears more promising.
CCoresponning author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recently, the TAO dataset [10], with its 833 categories, is the only suitable video training set that meets the diversity requirements for current OVMOT research. Although TAO offers a rich variety of categories, its small size and sparse annotations pose significant challenges for model training. Specifically, although TAO consists of 500 videos, one annotated frame provided every 30 frames, resulting in an average of only 37 annotated frames per video. Additionally, each frame has limited number of annotated targets, with an average of only 2 annotated objects per frame. This spatiotemporal sparse annotation format and limited data scale make it difficult for the model to effectively capture temporal dynamic changes of the targets and the spatially dense objects in each frame, thus leading to challenges in OVMOT training. As validated by experiments in OVTrack [1] and SLAck [11], using such a small-scale dataset with sparse annotations cannot be used to train a usable OV tracker. This leads us to a key question: Is it possible to train an effective and robust OV tracker directly using the TAO dataset with only limited and sparsely annotated data? This problem is challenging for two key reasons. First, it is essential effectively utilize the temporally sparse annotated data to obtain more continuous target features at intervals (un-annotated frames), thereby enhancing the continuity of the tracking training. Second, the training is constrained by the limited size of data, necessitating the learning of sufficient meaningful information from the small-scale data for efficient training.
For the temporal sparse annotation issue, the sole work, SLAck [11], has attempted to train an OVMOT using the TAO dataset. It utilizes annotations from previous frames and the targets detected at intervals to generate pscudo labels based on Intersection over Union (loU) for training. This method aims convert the originally sparse annotations in the videos into a denser format, enhancing the continuity of the video annotations and increasing the number of learnable samples. However, this approach has notable limitations: First, pseudo labels built on previous frames are only effective for adjacent frames. Those generated at longer intervals often exhibit significant deviations, especially for rapidly moving targets. Second, this method relies on frame-by-frame processing of un-annotated images and on the quality of the detector, resulting in time-consuming preprocessing and unstable performance. Third, this approach computes IoU only with single annotated frame to predict annotations, without considering relationships with neighboring annotated frames. It focuses solely on unidirectional continuity and fails to align pseudo labels generated within intervals with annotations on both sides.
To address these challenges, we aim to develop an implicit ap- proach directly obtain the tar-
No Real Interval Frame Needed
Denoising
get features at un-annotated in-
tervals. Inspired by the recent success of diffusion models [12] in pixel-level generation, we ex-
Initial Anmolated Frame
Un-ammllated Interval Frumes
Next Annotated Frume
(a) Diffusion-bused Continuous Data Generation
plore the potential of applying generative models to feature-level generation. Instead of relying solely on the final denoised result at the starting point, we consider using the intermediate results ob- tained during the denoising pro- cess as usable outputs. This leads to novel diffusion-based ap- preach for generating object fea- tures at the intervals from sparse annotations. As illustrated in Fig- ure ((a). For the first time, we model the denoising process of diffusion as the temporal evolution of target features, simulating the changes in target features between the two sparse annotated frames to acquire the object features in the intervals. Specifically, we construct the starting point of the diffusion model's denoising process by adding Gaussian noise to the object features at the initial annotated frame т. We then perform denoising at various time steps to learn the target features at the interval frames. The output of the denoising process is the next annotated frame -'. To ensure the reliability of the object features during interval frames, we develop 1) a reconstruction loss that utilizes interpolation information as supervision to ensure the feature reliability; 2) an endpoint loss to guarantee that the endpoint of the
Original Samples (high IoU)
Dynamic Group
New Samples (low lol)
Original Samples Ix (b) Dynamic Group Data Generation
=3x New Samples
(c) More Localization Samples
Figure 1: Illustration of our strategies for effectively enriching trainable samples. (a) Generating unlabeled object features at intervals using diffusion model. (b) Generating approximately three times more samples via a dynamic grouping strategy. (c) Harvesting additional low-confidence samples by lowering the IoU threshold between detections and ground truth.
2

---

denoising path accurately aligns with the object from r'; 3) smoothness loss that balances feature smoothness between un annotated frames and neighboring annotated frames achieve smooth object feature transitions. By training the diffusion model, we simulate the object feature variations and obtain the object features at intervals between two annotated frames. These features will subsequently be used for training, addressing the challenges of effective training with sparse annotations.
Furthermore, most existing detection-based OVMOT methods directly use LVIS pre-trained Open- Vocabulary Detector (OVD) and do not consider OVD during the tracker training process. To investigate thoroughly, we first find that fine-tuning the pre- pre-trained OVD on small-scale datasets, like TAO with tailee category distributions, leads to a significant decline in performance. The primary reason for this issue that models pre-trained on large-scale datasets, like LVIS (164K samples), struggle to effectively adapt to the smaller and unevenly distributed samples in datasets like TAO. Since detection is very important for OVMOT, we explore how to effectively fine-tune the OVD using the TAO with small-scale data and annotations. For this purpose, we propose a couple of new strategies to address the issue of insufficient learnable samples in TAO, thereby improving object classification and localization performance. First, as shown in Figure 1(b), we design dynamic group contrastive learning strategy that categorizes object features into affinity groups, dispersion groups, and adversarial groups, respectively, to enhance the object category diversity. With this strategy, the available samples have increased approximately threefold, effectively improving the contrastive learning for object classification under sparse and limited data conditions. It also effectively increases category diversity by adding the implicit new samples. Additionally, we introduce an adaptive localization loss strategy. Specifically, as shown in Figure I(c), by deliberately lowering the sampler IoU threshold between detection boxes and ground truth (GT), we first incorporate more low-confidence positive samples. Then, for the localization task, we dynamically adjust loss weights to make full use of the low-confidence samples, effectively mining the information of valid samples. These two strategies significantly increase the quantity of learnable samples, resulting in higher accuracy and robustness in the model's localization and classification tasks. Our contributions can be summarized as follows:
1. Addressing the OVMOT association training problem under sparse annotations: We propose a novel irffuusion-basedd model to simulate the features of targets at (un-annotated) interval moments, effectively enriching the feature space by enhancing the continuity of association features, thereby overcoming key challenges posed by temporal sparse annotations.
2. Achieving efficient the-ttning of OV detectors under small-scale data: We propose new dynamic group contrastive learning strategy to improve the classification by implicitly constructing new training samples. We also propose an adaptive localization loss that not only significantly expands the sample size but also alleviates the influence of low-confidence samples for localization learning.
3. Achieving state-of-the-ar (SOTA) performance on the OV-TAO benchmark: Extensive results show that the proposed method achieves SOTA results, and verify the effectiveness and superiority of cach component in our method.
2 Related Work
Open-woold/yocaabllrr object detection. Open-world object detection diverges from traditional
approaches by discovering salient objects without relying on a predefined label set [13-15]. Rather than assigning instances known categories, it formulates recognition clustering around learned class prototypes, enabling the discovery of novel objects [14]. Open vocabulary detection (OVD) takes this a step further by requiring explicit prediction of unseen class names [16]. commonly achieved by integrating text embeddings into the detector's 's training pipeline [17, 18]. The emergence of vision-language models like CLIP [19], which align visual features with textual descriptions, has markedly enhanced open- vocabulayy classification. Building on CLIP, recent studies [20-22] adapt these pre- pre-trained models for both open- apen-vocabulary and few-shot object detection. Furthermore, prompt-learmn leappt-lerrnng techniques that refine class-description embeddings [23-25) have been shown to boost detection aceuracy in open-vocabulary scenarios. Unlike existing OVMOT approaches that simply deploy pretrained OVD in downstream tasks to avoid performance loss, our method explicitly addresses the degradation in detection performance that occurs when fine-tuning OVD on small-scale datasets, substantially enhancing both detection and classification accuracy.
3

---

openn-orrd//ocaaulary object tracking, Research on open-worid tracking remains limited. Early methods either segment or follow every moving object in a video [26, 27] or employ class-agnostic detectors for generic object tracking [28-30]. The TAO-OW benchmark [31] is introduced to drive progress in this area, but evaluates only class-agnostic metrics and overlooks class-specific performance. To address these gaps, OVTraack [1] brings open-vocabulary capabilities to tracking by integrating OVD into a framework to recognize a wide variety of scene objects. It also establishes new baseline and benchmark built upon the TAO dataset. However, most existing OVMOT methods [1, 32, 8] rely on a frozen OVD and train the association head exclusively on large sets of static image pairs, thereby ignoring the temporal continuity present in video data. In contrast, our approach achieves effective OVD fine-tuning with only a few samples and introduces a diffusion- based surget-feature association learning algorithm that attains SOTA performance using merely sparsely annotated frames. Moreover, when compared to other TAQ-trained OVMOT algorithms [1 ]], Le., SLAck, our method surpasses its tracking accuracy despite not using any interval frames. While SLAck augments appearance features with motion and classification cues, we still exceed its performance by relying solely appearance information.
Diffusion model. The diffusion model is a recently popular class of deep-learning-basee generative models that recover valid samples from random distribution via iterative denoising process. It has achieved remarkable success in image generation, e.g., DALL-E 2 [33], Stable Diffusion [34] and VQ- VQ-Diffusion [35], etc. Following the trajectory of the success of image generation. video generation methods [36-40] have also made significant progress, using text prompts to guide diffusion models in producing coherent frame sequences. However, existing pixel-level video diffusion models demand large size of data to generate intermediate frames, incurring substantial computational costs and training time. They also cannot generate accurate annotations for those synthesized frames, leaving the problem of learning from sparsely labeled datasets unaddressed. To address this problem, we propose a feature-level diffusion framework that models object state transformation between sparsely annotated frames. Our method uses only few sparse annotations to directly simulate each target's feature evolution between labeled frames, efficiently generating intermediate object representations and thus enabling high-performang OV tracker training.
3 Proposed Method
This work primarily focuses on addressing the challenges of training the OVMOT model with limited sparsely annotated data, rather than on the design of the model structure itself. Therefore, we utilize the baseline architecture from OVTrrckk [1], which is built upon the ResNet-50 backbone and includes the localization, classification and association heads as below:
Localization: It employs a class-agnostic object proposal approach from Faster R-CNN [41] to localize objects for both base and novel categories, generating bounding boxes and confidence scores.
The classification head enhances the framework's 's open-vocabulary capabilities through feature distillation with CLIP [19]. It obtains classification features Fass of objects and aligns them with the text features Fiex generated by CLIP's 's text encoder. By calculating the cosine similarity between these features and predefined novel class names, the framework determines if the similarity exceeds a threshold, thereby identifying the corresponding novel classes
Classification:
Association:
The association head links detected objects across frames by learning object appearance features Fasso for similarity measurements. If the appearance similarity between objects exceeds a certain threshold, they are considered the same target and assigned to the same tracking trajectory.
3.1 Diffusion-based Data Generation for Association Training
Inspired by the success of diffusion models in various generative tasks, aim to leverage this frame- work for feature association in OVMOT. Classical diffusion methods typically focus on obtaining a final denoised feature representation. However, our approach takes novel perspective by utilizing intermediate results throughout the diffusion process instead of solely relying on the final output.
As shown in Figure 2, given the object features from one annotated frame as the key feature Fiey E Bxd We denote the features of the same object from the next annotated frame 7' as the reference feature Fier E TR. Bxd, where B is the batch size and d the dimension of the features. We model the denoising process learning the mapping from Faey to Fist The aim is to utilize the

---

Annotated Key Frame
Un-annotated Interval Frames (only for illustraion)
Annotated Reference Frame
reconn
P Pkey +
asso
Fasso
Fref
fend
smooth
Po Po (F 1 F Fret)
F
F
p
Denoising Pracess
Figure 2: Illustration of the denoising process in our diffusion model, which simulates the transforma- tion of object features from Fkey to Fier obtain the un-annotated object features interval frames.
diffusion model to simulate the intermediate states of the object and extract associated features at different timesteps.
Forward Process. The forward noise perturbing process at time is defined as
go [F = NNFF; :I= 3, F, ,,.1).
(1)
which gradually adds Gaussian noise to the data according a variance schedule 31,..., .3r. The construction of features at time / can be expressed as F Var + v1 o4é, where Oi IT' B.) and (O, I). Here, or represents the cumulative product of the variance schedule up to time t.
Reversed Process. The reverse process De(F*=1[F" |F,, FieE) then simulates the transition from Flay to Fies systematically obtaining Fie from Ft through the denoising process, as
1
Ft-1
F"
fo Fres.)
Var
where fo is neural network that estimates the noise present in Ft, incorporating Fes as a crucial input for refining its predictions.
Loss Functions. To ensure effective training of the diffusion model, we define the following loss functions that encapsulate the mapping process from Faey to Fis- In the following, denote the intermediate features Ft as Fi corresponding to the association features in OVMOT at timestep t.
1) Reconstruction loss. To ensure that the features at frame accurately represent the changes between Fee and Fier we construct a reconstruction loss through interpolation. We use time to balance the interpolation ratio, resulting in the following loss
Creson
Σ
[t'Feey + (le
N
(E(t..a.....A))
where t; are the timesteps, N is the number of sampled timesteps, and t' [0, 1] is the normalized timesteps. This loss encourages the model to reconstruct Fi... as a linear interpolation between Fkey and Fiet based on the current normalized timestep t', reinforcing the model to approximate the state between the key and reference features.
2) Endpoint loss. In addition to ensuring the accuracy of the reconstructed features at intermediate timesteps, is also essential that the final denoised features align with Fer to establish a complete transformation chain from Fey Pres, resulting in the following loss
Lend
F
usso Frrll
This loss ensures that the final associated features F process) exactly match the reference features Fret.
(resulting from the complete denoising
3) Smoothness loss. Although the two losses above ensure the accuracy of the results at each timestep, they do not account for the smoothness of the feature changes at each timestep, while the
5

---

objects always change smoothly in continuous video. To address this, we consider the consisteney relationship between features during the transformation process and construct the following loss
F F
Fkey
Fref
F
smoohh
2550
t e (21,tt.........
(5)
Fkey|
FFief
F assol
This loss encourages smooth transitions between the associated features across timesteps, preventing abrupt changes in feature representation, which can occur due to the inherent noise in observed data.
Overall, these losses assist the diffusion model in constructing accurate and smooth feature represen- tations at intermediate timesteps. facilitating the overall mapping from Fkey to Fiet. The intermediate features F obtained during the denoising process of the diffusion model will be utilized as the object association features at un un-annotated interval frames for association training, addressing the issue of sparse annotations.
Enhanced Association Training with Generated Features. Our diffusion model fundamentally transforms sparse association training by generating intermediate features F at multiple timesteps between annotated frames. This enhancement operates through two complementary mechanisms:
1) Quantitative Sample Space Expansion. For each object trajectory originally containing only Iwo annotated features (Ficy and Fer separated by 30 frames), our method generates N intermediate features where N corresponds to sampling steps (typically 3). This directly expands the positive features, while simultaneously enriching the
sample set Q+ for each object identity from negative sample set Q with generated features from different object trajectories. Consequently, the total training samples increase by approximately 3x, providing substantially more positive and negative pairs for robust contrastive learning.
2) Qualitative Feature Continuity Enhancement. Beyond quantity expansion, our generated intermediate features F exhibit superior temporal continuity compared to sparse annotations. The diffusion-based generation ensures smooth feature transitions through: (1) reconstruction loss enforcing accurate interpolations between keyframes; (2) smoothness loss guaranteeing consistent feature evolution; (3) endpoint loss ensuring precise alignment with reference features. This results in temporally coherent feature representations that better capture object state transitions, providing higher-quality training samples for association learning.
Building upon these enhancements, we employ the same association loss framework as OVTrack [1] but with significantly enhanced effectiveness due to enriched sample space. The complete tracking loss Lunck is formulated as:
exp(q q /T) Σ log PosD(q) + Eq EQ-(a) explq ( m (+)
Lrack =- - qEQ
-E
Q+(a)
(6)
where PosD(q) = प Eq- exp(q /T) and is the temperature parameter. The enriched Q+ (q) contains both original keyframe features and our generated intermediate features, while Q (q) includes negative samples from different trajectories.
3.2 Training Sample Extending for Detection
Besides association, we next consider how to train (fine-tune) the detection model using the limited data in TAO, which includes both the classification and localization heads. Given that TAO is a long-tailed dataset with limited annotations, we aim to address two main challenges. First, for classification, we need to overcome the limitations of sample quantity and diversity to effectively learn class representations from a small number of samples. This way, we propose dynamic group contrastive learning approach, which increases the number of samples used in contrastive learning by approximately 3 times, thereby significantly enhancing classification training. Second, for localization, we seek to increase the number of localization samples (bounding boxes) by lowering the IoU threshold for positive samples, and apply adaptive confidence weights to reduce the noise from cow-confieence samples. We will elaborate on the methods in detail below.
Dynamic Group Contrastive Learning. We present the proposed dynamic group contrastive learning from 1) how to construct dynamic groups and why dynamic groups effectively enhance sample quality, and 2) how to utilize the generated dynamic groups for contrastive learning.

---

css
F Fa
ds F
F di
F dsss
Construct Group
Affinity Groups (closest)
Dispersion Groups (farthest)
Fss
P text
Group Size= 4, 4....
Fds F
cle
Adversarial Groups (random with noise)
No Original Samples
3 x O(N.) New Samples
Figure 3: Illustration of the construction of our dynamic groups, which generates diverse and robust group samples, effectively tripling the training samples. For clarity, the diagram depicts only the construction process for a group size of two; other group sizes follow the same principles.
1) Dynamic group construction. As shown in Figure 3, given the classification features (obtained from the classification head discussed above) of all N: samples in each class, for each sample i, we first calculate the Euclidean distance d' to its corresponding text center features Fiest of this class. Then, we sort the samples in ascending order of distance, resulting in F . Fals F-N.-1 Foils This sorting process ranks the samples according their similarity to the CLIP text embedding of this class, which reflects the difficulty of each sample. We then group these samples in each class by dividing all samples into groups, where g - 2, 4, 8, 16. up to 2 N. with Ne as the total number of samples. The number of groups follows a geometric progression, where each number is double the previous one. The total count of groups formed can be derived from the geometric sum formula, ie., 2 21oog, - 2, with an order of magnitudes of O(N.). After determining the groups, each group aggregates the features of its respective samples, which is achieved by the averaging process. To enhance the diversity and robustness of the newly constructed groups, we have designed three distinct types of grouping manners:
Affinity group: The samples are sorted according to their distances to the corresponding text center. After that, each group consists of several samples that are closest to each other. We achieve this by minimizing the sum of variances of all g groups. The aggregated features from each similarity group serves to reinforce the shared features among similar samples, thereby improving the model's 's ability to distinguish fine-grained differences.
Dispersion group: Similar to the above operation, instead of clustering the closest samples, this manner selects the mutually dispersed samples within each class as much as possible. This achieved by maximizing the sum of variances of all groups. By doing each dispersion group ensures a mix of representative samples that capture varying aspects the feature space. This manner enhances the model's is exposure to broader range of features, ultimately improving its generalization capabilities. The detailed implementation and theoretical proof of grouping process are provided in Appendix A.
@ Adversarial group: In constructing the adversarial group, samples are randomly selected from the pool without considering their similarity-basee ordering. Following the selection, Gaussian noise is added to the features of the chosen samples to create adversarial conditions. This randomness and the addition of noise help to simulate challenging scenarios for the model, forcing it to learn more robust feature representations that can withstand variations and perturbations during testing.
2) Contrastive learning with augmented samples. With the enhanced dataset containing No + O(N)) samples, which includes the No original samples and the 3 O(N.) newly generated samples from 3 types of groups, we can perform contrastive learning improve model robustness and feature representation. The overall training objective for this augmented dataset is to minimize the InfoNCE loss [42] formulated as follows:
LInfONCE
Σ log K exp(cos(F) Fb ///)/p) + exp(cos(Fi. F N
1
exp(cos(Fii. (/Pp
where N is the total number of samples, i.e., N₂ + O(N.). Fi is the feature representation of the i-th sample, Fi+ is the positive sample paired with F K is the total number of negative samples in the batch, cos(-.) denotes the cosine similarity function, / is a temperature parameter that controls the distribution sharpness. The InfoNCE loss encourages the model to bring the same class samples closer together the feature space while pushing different class samples further apart. By leveraging the increased sample size from the augmented groups, the effectiveness of this loss
7

---

function is enhanced, improving the model's ability to discern differences between classes. For scalability analysis of this approach with large category sets, please refer to Appendix B.4.
Adaptive Localization Loss. In object localization subtasks of OVD, it is a common operation to compute losses only for the positive samples that exceed a certain IoU threshold with the ground truth (GT). This is acceptable for training on large-scale datasets, For limited data, to enlarge the sample size, we propose to increase the number of positive training samples by lowering the IoU threshold, but this also introduces additional disturbance into the training process. To address this issue, we implement an adaptive localization loss that considers the confidence of each sample.
We define the bounding box for the i-th sample as B, = (2,, Yi, w/, h,). with o; and yi as the coordinates of the top-left comner and w; and h2 as the width and height, respectively. The corre- sponding ground truth bounding box is denoted as B To manage the disturbance created by the low-IoU bounding boxes, employ a smoothing weight strategy defined for the i-th object as w; = log(ect1)+1 E (0, 1], where 6; = |B; " || denotes the absolute error between the predicted and ground truth bounding box. This smoothing weight strategy assigns lower weights to the boxes with larger errors. The intuition behind this approach is that predictions that significantly deviate from the ground truth are often less reliable. By reducing their impact on the overall loss, we extend the sample variety while emphasizing more accurate predictions, which positively contributes the training process. The overall adaptive localization loss can then be formulated as
Np W/ Lsmootht. I (Bi, B7),
Addaptiv
(8)
where Np is the total number of positive samples and we use the Smooth 11 loss in Faster R-CNN [41]
[0.5|| B, 11
if B; <
Lsmosht.1 (B,, B,)
(9)
| - Bill 0.5
otherwise.
This formulation of the adaptive localization loss effectively balances the inclusion of more training samples and the necessity of disturbance management. By applying adaptive weights based on the confidence of each prediction, we enhance the model's 's robustness and accuracy during training, ultimately improving its performance on the object localization subtask.
3.3 Implementation Details
We adopt the OVTrack [1] architecture and replicate its original pre-training protocol. In the proposed TAO training stage, we jointly optimize the association, localization and classification branches on the base set of the TAO training set. Training is performed for 10 epochs on only 2 RTX 3090 GPUs. In the association training, we employ a MP-baad diffusion model denoise samples drawn from standard normal distribution. fe is implemented as a threc-layer fully-comnected network, with each layer followed by ReLU activation and layer normalization. During the first five epochs, the diffusion model is trained on features produced by the association head (without generating new samples), and the subsequent epochs, we freeze the diffusion model and use to generate augmented data for further association training. The association loss, consisting of a contrastive loss and an auxiliary loss, identical that used in OVTrack. In the classification training, we apply an InfoNCE loss with p 0.1 in Eq. and also employ the standard cross-entropy loss used in OVTrack. In the localization training, the IoU threshold lowered to 0.3. In the inference stage, we retain all OVTrrack settings except that we change the maximum detections per frame to 80, the matching threshold to 0.38 and the memory length to 30.
4 Experimental Results
4.1 Datasets and Metrics
Following other OVMOT methods [1, 11, 7, 8], we perform our evaluation with standard OV settings on the TAO dataset, which categorizes rare classes as novel and the others as base classes, similar to LVIS 19]. Comparative experiments are carried out on both the validation and test sets of TAO. For performance assessment, we implement the standard OVMOT metric tracking- everything accuracy (TETA) [43], which encompasses evaluation of localization accuracy (LocA), classification accuracy
8

---

Table I: Comparison of tracking performance validation and test sets of the open- vocn-uobabllayy TAO benchmark [1]. All methods use ResNet-50 as the backbone. represents using the same detector.
Method
Base
TETA
LocA
AssocA
CisA
TETA
LocA
AssoCA
ClsA
Validation set
QDTrack [49]
0.4
45.6
11.0
TETer 143]
25.7
45.9
31.1
0.2
30.3
47.4
31.6
12.1
DeepSORT (MILD) [45]
21.1
46.4
14.7
2.3
26.9
47.1
15.8
17.7
Tracktorss (VILD)
22.7
46.7
19.3
2.2
28.3
47.4
20.5
17.0
ByteTrack [47]
22.0
48.2
16.6
1.0
28.2
30.4
18.1
16.0
OC-SORT' [48]
23.7
49,6
20.4
1.1
28.9
51.4
19.8
15.4
OVTrack" [1]
27.8
48.8
33,6
1.5
35.5
49.3
36.9
20.2
(R50) [32]
30.0
1.0
19.3
OVTR
314
54.4
34.5
5.4
36.6
52.2
37.6
20.1
OVSORT' [8]
30.8
53.0
37.6
1.9
38.2
55.3
79.9
19.4
SLAck' [I]]
31.1
54.3
37.8
1.3
37.2
55.0
37.6
19.1
Ours
35.2
59.2
6.2
40.4
58.6
20.5
QDTrcc [49]
20.2
39.7
20.9
0.2
25.8
43.2
23.5
10.6
TETer [43]
21.7
39.1
0.0
29.2
44.0
30.4
10.7
DeepSORT (ViLD) [45]
17.2
38.4
11.6
1.7
24.5
43.3
14.6
15.2
Tracktore+ (VILD) [46)
18.0
39.0
13.4
1.7
26.0
44.1
19.0
14.8
OVTrack
1.8
32.6
16,9
OVTR
27.1
47.1
32.1
2.1
34.5
51.1
37.5
14,9
OVSORT" [8]
28.1
48.0
33.4
2.7
35.1
51.6
38.3
15.4
SLAck" [I]]
27.1
49.1
30.0
2.0
34.7
52.5
35.6
16.1
Ours
30.1
3.9
38.7
56.6
17.7
(CIsA), and association accuracy (AssocA), For clear evaluation, we evaluate the performance of base and novel classes separately.
Comparison with State-of-the-Arts
We compare our method with recent tracking methods on both the validation and test sets of TAO. For a fair comparison, methods utilize ResNet-50 as the backbone. We include closed-set baselines trained on all categories, established off-the-shelf trackers such Byte [47], OC-SORT [48], and MASA [32], as well as specialized OVMOT methods like OVTrack [1], the state-of-the-art methods, SLAck [11], OVSORT [8], and transformer-based OVTR [7].
As shown in Table 1, our method demonstrates significant performance improvement over all competing methods in terms of the TETA metric on both the validation and test sets. Notably, achieve TETA score of 35.2% on the validation set and 38.7% on the test set, surpassing the second-best method by 3.8% and 3.6%, respectively. In terms of individual metrics, our method achieves particularly strong results in both novel and base AssocA. Specifically, we report novel AssocA of 40.3% 3 and a base AssocA of 42.1 %, marking significant increases of 4.9% and 4.5% over SLAck, which also trained on the TAO dataset. Note that we only use sparse annotation data, without the interval frames used in SLAck. These improvements highlight the efficacy of our approach for improving tracking performance. Additionally, we observe substantial improvements in both the ClsA and LocA metrics. These gains particularly notable under limited data, where effective fine-tuning plays a crucial role in maximizing performance, which indicates that our proposed strategies for sample augmentation for OVD are highly effective. Importantly, although approach the same architecture as OVTrack and trained on sparse data, our results significantly outperform those of OVTrack. This demonstrates the high efficiency of our training strategy with limited, sparse data.
4.3 Ablation Study
Effectiveness diffusion-basse data generation. As shown the Association module section of Table 2, the results indicate that the three proposed loss functions significantly enhance the association performance. Furthermore, consistent with the findings of SLAck [11], J, we observe that training solely on the original sparse TAO dataset, without supplementing intermediate frame object features, leads to poor association results. Additionally, we compared our approach with a direct linear interpolation data generation method. Although this method showed some performance improvement, the overall results remained inadequate. These ablation experiments demonstrate that the intermediate target features generated by our diffusion-based method substantially improve the association in OVMOT.
Effectiveness of dynamic group contrastive learning. shown the Classification module of Table 2, first, by removing some additional groups in the top three rows, we observe decline in

---

Table 2: Ablation study results on the validation set. We compare the results of different ablation
methods. Each module corresponds to different aspects of association, classification and localization.
Novel
Base
Module
Ablation Method
IETA
LocA
AssocA
ClsA
TETA
LocA
AssocA
ClsA
reconstruction loss w/a smoothness loss
34.3
59,4
38.9
4.7
39.3
58.6
39.7
19.7
34.7
59.1
39.3
5.8
39.9
58.5
41.2
20.1
Association
w/a endpoint loss
34.8
59.2
39.6
5.6
39.8
58.6
40.7
20.1
w/a diffusion- hased data generation
31.1
58.1
31.7
3.6
36.7
58.3
33.5
18.2
lineur interpolation data generation
31.3
57.1
32.7
4.2
37.2
57.9
34.3
19.5
w/o aftinity group
34.3
58.9
40.0
4.1
39.9
58.8
41.3
19.4
dispersion group
34.7
59.3
39.8 8
39.8
58.5
41.8
19.2
Classifictiion
w/o adversarial group
34.2
58.8
39.9
4.0
39.6
58.3
41.6
19.0
extra group data contrastive learming
33.2
58.7
39.6
39.0
58.1
41.2
17.6
w/o dynamic group contrastive learning
33.5
58.8
39.9
1.7
39.4
58.5
41.5
18.2
Localization
w/o lower sampler IOU
udaptive weight
32.3
54.7
37.6
4.7
38. 1
51.3
55.8
38.3
39.9
18.7
18.5
Ours
35.2
59.2
40.3
40.4
58.6
42.1
20.5
the final classification results. the fourth row, we can see a significant impact when we perform contrastive learning using only the limited original samples, without providing the additional training samples derived from our groups. The results in this case are even worse than those in the fifth row, where eliminate contrastive learning altogether and rely solely on cross-entropy loss. Additionally, the experiments in the fifth row further prove the effectiveness of our contrastive learning approach.
Effectiveness of adaptive localization loss. In the Localization section of Table 2, we proposed adaptive weights effectively reduce the impact of noise during localization training. Also, utilizing a lower IoU threshold allows us to obtain greater number of available training samples.
that the
Impact of different sampling steps. In Table 3, we examine the impact of different sampling steps on the results. Notably, even with a sampling step of just 1, there is significant improvement compared to the approach without data generation. Although the results fluctuate with an increase in sampling steps, the differences are relatively small. This indicates that our proposed diffusion-based data generation method quite stable when training with sparse data.
Table 3: Ablation study results of different sampling steps on the validation set.
Sampling Steps
Novel
Base
TETA
LocA
AssocA
CIAA
TETA
LocA
AssucA
ClSA
33.2
58.0
37.4
4.3
38.7
57.8
38.4
19.9
34.8
4.9
40.2
35.2
59.2
40.3
6.2
40.4
58.6
42.1
20.5
35.1
58.7
41.5
5.2
40.1
58.4
42.2
19.8
35.1
59.8
40.5
5.1
40.2
58.6
42.0
20.0
Additional analysis on detection fine-tuning necessity, challenging scenarios, scalability, and association-centricc metries including IDF1 and MOTA are provided in Appendix B. More visu- alization results are provided in Appendix C.
5 Conclusion
In this work, we have proposed a data-efficient OVMOT method. Based on the existing OV video dataset with sparse and limited annotations, we develop series of methods to explore higher data utilization. We consider three sub-tasks in OVMOT: for the association task, we develop a diffusion- based method for temporal object feature construction generate intermediate features between sparsely annotated frames; for classification, we design a dynamic group construction method to increase the data diversity of each object class; for localization, excavate and adaptively use them. Experimental results verify the effectiveness of our method. Through this work, we hope promote the efficient training of OVMOT and provide some insights for effective learning of other detection and tracking problems on limited data.
candidate boxes
Acknowledgment
This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant 62402490, 624221 18, 62476196, 62572349; by the Emerging Frontiers Cultivation Program of Tianjin University Interdisciplinary Center; by the Hong Kong Research Grants Council under Grants 11219324, 12023200, and 11219422.
10

---

References
[1] Siyuan Li, Tobias Fischer, Lei Ke, Henghui Ding, Martin Danelljan, and Fisher Yu. Ovtrack: Open- vocabulary multiple object tracking. In Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5567-5577, 2023.
[2] Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura LealeTaixé, Mor20: A benchmark for multi object tracking crowded scenes. arXiv preprint arXiv:2003.090m 2020.
[3] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The international journal of robotics research, 32(1 12:111112337 2013.
[4] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 11621- 11631, 2020.
[5] Anton Milan, Laura Laix- Ian Reid, Stefan Roth, and Konrad Schindler. Mot16: A benchmark for objei- tracking. arXiv preprint urXiv:1003.00833 2016.
[6] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, al. Scalability in perception for autonomous driving: Waymo open dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2446-2454, 2020.
[7] Jinyang Li, En Yu, Sijia Chen, and Wenbing Tao. Ovtr: End-to-end open- vocabulayyy multiple object tracking with transformer. In The Thirteenth International Conference on Learning Representations, 2025.
Sang-Woon [8] Run Li, Dawei Zhang, Yanchao Wang, Yunliang Jiang, Zhonglong Zheng, Sang- Woon Jcon, and Hua Wang. Open-vocabulle multi-ohject tracking with domain generalized and temporally adaptive features. IEEE Transactions on Multimedia, 2025.
[9] Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
[10] Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: large-scale benchmark for tracking any object. In European Conference on Computer Vision (ECCV), pages 436-454, 2020.
[11] Siyuan Li, Lei Ke, Yung- Hsu Yang. Luigi Piccinelli, Mattia Segù, Martin Danelljan, and Lue Van Gool. Slack: Semantic, location, and appearance aware open-vocabulag tracking. In Proceedings of the European conference computer vision (ECCV), 2024.
[12] Zhen Xing, Qijun Feng, Haoran Chen, Qi Dai, Han Hu, Hang Xu, Zuxuan Wu, and Yu- Yu-Gang Jiang. A survey video diffusion models. ACM Computing Surveys, 57(2): 42, 2024.
[13] Akshay Dhamija, Manuel Gunther, Jonathan Ventura, and Terrance Boult. The overlooked elephant of object detection: Open set. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1030, 2020.
[14] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 5830-5884 5840, 2021.
[15] Thang Doan, Xin Li, Sima Behpour, Wenbin He, Liang Gou, and Liu Ren. Hyp-ow: Exploiting hicrarchical structure learning with hyperbolic distance enhances open world ohject detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1555- 1563,, 2024.
[16] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14393-14402. 2021.
[17] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In Proceedings of the European conference computer vision (ECCV). pages 384-400, 2018.
[18] Shafin Rahman, Salman Khan, and Nick Baress. Improved visual-semantic alignment for zero-shot object detection. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11932-11939 2020.
11

---

[19] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[20] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Opca-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2022.
[21] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. In European Conference on Computer Vision, pages 350-368. Springer. 2022.
[22] Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open- apen-vocabulary detection with region prompting and anchor pre-matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7031-7040, 2023.
[23] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vocabulary object detection with vision-language model. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14084-14093, 2022.
[24] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages
2022.
[25] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 1300))2337-2338, 2022.
[26] Dennis Mitzel and Bastian Leibe. Taking mobile multi-object tracking the next level: People, unknown objects, and carried items. Computer Vision-ECCV 2012: 12th Eumpean Conference on Computer Vision, Florence, Itly, October 7-13, 2012, Proceedings. Part V 12. pages 566-579. Springer, 2012.
(27] Achal Dave, Pavel Tokmakov, and Deva Ramanan. Towards segmenting anything that moves. In Proceed- ings of the IEEE/CVF International Conference Computer Vision Workshops (ICCVW), 2019.
[28] Aljoša Ošep, Alexander Hermans, Francis Engelmann, Dirk Klostermann, Markus Mathias, and Bastian Leibe. Multi-scale object candidates for generic object tracking in street scenes. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3180-3187. IEFF, 2016.
[29] Aljoša Ošep, Wolfgang Mehner, Paul Voigtlaender, and Bastian Leibe. Track, then decide: Category- agnostic bassnn-based multi-object tracking. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3494-3501. IEEE, 2018
[30] Aljoša OKep, Paul Voigtlaender, Mark Weber, Jonathon Luiten, and Bastian Leibe. 4d generic video object proposals. In 2020 IEFE International Conference on Robotics and Automation (ICRA), pages 10031-10037, 2020.
[31] Yang Liu, Idil Esen Zulfikar, Jonathon Luiten, Achal Dave, Deva Ramanan, Bastian Leibe, Aljoša Ošep, and Laura Taixé. Opening up open world tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 19045-19055 2022.
[32] Siyuan Li, Lei Ke, Martin Danelljan, Luigi Piccinelli, Mattia Segu, Lue Van Gool, and Fisher Yu. Matching anything by segmenting anything. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 18963-18973, 2024.
[33] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 220.0065,,, 1(2):3, 2022.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEFF/CVF Conference Computer Vision and Pottern Recognition (CVPR), pages 10684-10695, 2022.
[35] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang. Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10696- 10706, 2022.
[36] Yingqing He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high- fideliyyy long video generation. arXiv preprint 2222 2022.
12

---

[37] Muyang Li, Ji Lin, Chenlin Meng. Stefano Ermon, Song Han, and Jun-Yan Zhu. Efficient spatially sparse inference for conditional gans and diffusion models. Advances neural information processing systems, 09:28858-28873, 2022.
[38] Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018. 2022.
[39] Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. LIm-grounded video diffusion models. arXiv preprint arXiv:2309. 17444, 2023.
[40] Pengxiang Li, Kai Chen, Zhili Liu, Ruiyuan Gao, Langing Hong. Dit- Van Yeung, Huchuan Lu, and Xu Jia. Trackdiffusion: cradtioe- video generation via diffusion models. 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3539-3548, IEEE, 2025.
[41] Shaoqing Ren, Kaiming He, Ross Girshick, and Sun. Faster r-cmn: Towards real time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[42] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint vrXiv:1807.03745 2018.
[43] Siyuan Li. Martin Danelljan, Henghui Ding, Thomas Huang, and Fisher Tracking every thing in the wild. In European Conference on Computer Vision, pages 498-515. Springer, 2022.
[44] Tobias Fischer, Thomas Huang, Jiangmiao Pang. Linlu Qiu, Haofeng Chen, Trevor Darrell, Fisher Yu. Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2023.
[45] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In 2017 IEEE international conference image processing (ICIP), pages 3645-3649, 2017.
[46] Philipp Bergmann, Tim Meinhardt, and Laura Taixe Tracking without bells and whistles. In Proceedings of the IEEE/CVF international conference on computer vision, pages 941-951 2019.
[47] Yifu Zhang, Peize Sun, Yi Jiang. Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[48] Jinkun Cao, Jiangmiao Pang. Xinshuo Weng. Rawal Khirodkar, and Kris Kitani. Observation-centric sort: Rethinking sort for robust multi-object tracking. Proceedingy of IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9686 066669966,600 2023.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract briefly outlines our main contributions and the focus of our work in the field of OVMOT, while the introduction provides detailed overview of the relevant topics.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [No]
14

---

Justification:
This paper does not discuss limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: The supplementary materials include proof of the variance for different group designs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our experimental results can be easily replicated.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code will be made publicly available after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
stt.:esubbiiiioooooooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //ni cttppssscc/ppbbicguuuuc.o
/CodeSubmisst ssisnsoiicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See Section 2.4.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Work in the OVMOT field typically does not involve this type of error analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Section 2.4.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https:/ /tttp::///nurip.... cc (ppppli//tticsuuiiiiieeee
Answer: [Yes]
Justification: Our paper fully complies with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The paper does not include discussion related the design.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
18

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper includes references for all the codes and datasets used.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. aattattsess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: This paper includes the code for the relevant new algorithms.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
19

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: This paper uses the CLIP model.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https http://uuurpp..ceoeeeeeeeeeeeeee hatp:///eeeeeeeee/////////////// for what should or should not be described.
20

---

A Variance Implications of the Grouping Strategy
A.1 Minimizing Variance Sum through Affinity Group Construction
Problem Statement. Given set of numbers I1, .2..... we want to construct & groups, denoted as G1, G2...., Gk. such that the sum of each group is Tg- The mean of each group is defined as jok m.
Is, where my is the number of elements in group We aim minimize the overall variance
5 of the sample set can be expressed as:
S
1=)²
==1
Strategy for Minimizing S To minimize 82, adopt sequential grouping strategy as follows: First, sort the data values I1, 22...., In in ascending order by the similarity from the Fien- Then divide the sorted data into & contiguous groups. The first group contains the smallest values, while the last group contains the largest values.
Proof. Next, we will provide detailed proof of why this affinity group construction strategy can effectively achieve the minimization of S as follows:
where my is the number Definitions and Setup: Let the mean of each group be μ4: mi of elements in each group.
Tz.
Exchange Argument: Suppose there
groups i and with 14 and Pj their respective means, Let there exist an element r < y such that E Gj and E G, Based on the grouping method, we can conclude that T; Tj, meaning that the sum of group is less than the sum of group j. We consider swapping these two elements:
Ty Ti - Tj = T, 1 I.
The new group means after the swap will be:
T
T:
x +
14
171
T"
T
y
+ 20
p4
my
Next, we need calculate the updated variances for groups and j:
m4
S
(x) p4)2
m.
j=l
my yy - 12)
5?
m.
Calculating the Change in Variance: The change in the overall variance AS due to the swap given by:
A 57. (S + 1t).
Calculating AS2 directly can be complex; however, by swapping the larger value y from group Gj with the smaller value x from group G; we significantly alter the internal distribution of the elements within each group. The introduction of the larger value in group G, increases its variance because it increases the deviation from the new mean. Similarly, swapping the smaller value x into group G, will also increase the variance of that group. Therefore, as result of these changes, it can be inferred that:
21

---

AS² >0
This indicates that the overall variance increases after any swap, which suggests that the process of selective grouping can minimize the spread of variance across the groups.
Conclusion: Therefore, using a sequential grouping method (i.e., partitioning the data into contiguous segments) will minimize the overall variance $2.
A.2 Maximizing Variance Sum through Dispersion Group Construction
Problem Statement. Given a set of numbers T1, .......... Zn we want to construct groups, denoted as G1, Gg...., Gk. such that the sum of each group is Tk. The mean of each group is defined as 14k my
T. where my is the number of elements in group k. The overall variance S of the sample set
can be expressed as:
k
S
(c) - 14)2
==1
Expanding this yields:
1
mk
52
Σ
mz
k
m&
This can be rearranged to show a fixed term:
S
772
YIT
3=1
The term Σj= 23 a fixed quantity determined by the sample set. Therefore, to maximize S we need minimize the term: > 14.
-i=1:
Strategy for Minimizing Σx= p... To minimize Σl= H. we adopt a two-end grouping strategy as follows:
First, sort the data values 21, T2...... Tn ascending order by the similarity from the Fext- Then, each group should be formed by selecting elements such that each group contains the largest and smallest values available. Specifically, can construct each group G, by taking the maximum and minimum values from the remaining elements.
Proof. To prove that these strategies effectively minimize Σ uC, consider the following:
Assuming Constant Total: Let's assume the overall sum of group means constant, i.c. M1 + #22 +... + Pa = The goal to minimize Σ 14² under this constraint.
Applying Cauchy-Schwarz Inequality: By applying the Cauchy-Schwarz inequality the context of these means:
k(p + 2 +.....4 m2) 2 (M) + 12 + +x22 T
This implies that:
2
3 12
2
Therefore, minimizing E 14 occurs under the condition that the means are as equal as possible.
22

---

Our group construction method ensures that all N4 values are as equal as possible because we are taking elements from both ends of the distribution. This approach ensures that the means converge to the overall mean of the sample set, thereby fulfilling the necessary condition for minimizing E
Validating the Construction Method:
By focusing on strategies that leverage the largest and smallest available values for group construction and maintaining the overall sum of means as constant, we can effectively minimize the sum of squares of the means μf: u1z ......... --.-; ( thus maximizing the overall variance.
Conclusion:
B Additional Experimental Analysis
B.1 Defection Model Fine-tuning Necessity
To better demonstrate the necessity of the detection fine-tuning stage, we conduct comprehensive ablation experiments that demonstrate the effectiveness of our approach even without detection fine- tuning. As shown in Table 4, we evaluate our method without detection fine-tuning on TAO, using only our diffusion-ber based association enhancement. Our method still achieves competitive performance, particularly in association accuracy, indicating the core effectiveness of our diffusion- -basedd approach.
Table 4: Performance comparison without detection fine-tuning on TAO validation set.
Method
Novel
Base
TETA
LocA
AssoA
ClsA
TETA
LocA
AssoA
ClsA
Track
27.8
48.8
33.6
1.5
35.5
49.3
36,9
20.2
SLAck
31.1
54.3
37.8
1.3
37.2
55.0
37.6
19.1
OVTR
31.4
54.4
34.5
5.4
36.6
52.2
37.6
20.1
Ours (w/o det. fine-tune)
32.4
55.3
39.9
2.1
38.7
55.3
41.1
19.8
Overall, our proposed method benefits from but is not dependent on detection fine-tuning. The complete version with fine-tuning constructs comprehensive data-cfficient training paradigm that maximizes the utility of limited sparse annotations, creating holistic approach that leverages both association improvements (via diffusion) and detection improvements (via fine-tuning) achieve optimal performance under constrrnndt conditions.
B.2 Analysis Challenging Scenarios
To comprehensively assess our method's robustness, we systematically selected the most challenging videos from the TAO validation set (93 videos out of 988 total videos) using multiple quantitative criteria. Videos satisfying any of the following conditions were included in our challenging subset:
Selection Criteria:
Heavy Occlusion Scenarios: Videos with substantial mutual occlusions where object bounding boxes exhibit IoU 0.4 with other objects for more than 20% of the total frames.
Rapid Motion Patterns: Videos with fast-moving objects where the average displacement ratio (bounding box center displacement box diagonal length) 0.3 across consecutive frames.
Frequent Entry/Exit Dynamies: Videos where objects exhibit frequent entry/exit behaviors (more than entry/exit cycles per trajectory on average).
High Object Density: Videos containing crowded scenes with more than 8 concurrent objects per frame on average.
Using these systematic criteria, we identified videos presenting challenging tracking conditions, which we refer to as TAO-Hard. The evaluation results are shown in Table 5.
The results show different levels performance degradation in these challenging scenarios, which is natural given the unpredictable object states, significant appearance changes, and objects leaving and re-entering the scene. Nevertheless, our method maintains superior performance compared to existing approaches, demonstrating strong robustness across diverse scenarios.
23

---

Table 5: Performance comparison on challenging scenarios (TAO-Hard),
MMethod
Novel
Base
TETA
LocA
AssoA
ClsA
TETA
LocA
AssoA
ClsA
TVTrack
27.8
48.8
33.6
1.5
35,5
49.3
36,9
20.2
OVTrack TAQ-Hard
25.9(-1.9)
45.3
31.3
1.1
33.9(-1.6)
49.1
33.4
19.3
OVTR on TAO
31.4
54.4
34.5
36.6
52.2
37.6
20.1
OVTR on TAO-Hard
28.4(-2.0)
51.2
32.1
1.9
35.2(-1.44
51.2
35.2
19.1
Ours on TAO
35.2
59.2
40.3
6.2
40.4
58.6
42.1
20.5
Ours TAO-Hard
33.4(-1.8)
57.6
39.2
3.4
39.2(-1.2)
57.1
40.3
20.1
B.3
Additional Association-Centiis Metrics
To provide more comprehensive evaluation of association performance, we include additional association-centric metrics as suggested by reviewers. Table shows our experimental results on IDFI and ID switches (IDSW) metrics.
Table 6: Association-centinn metrics comparison on TAO validation set.
Methods
IDFI (%)
IDSW
MOTA (%)
OVTrack
18,962
44.4
OVTR
72.6
12.128
48.2
Ours
76.2
10,393
57.3
Our method demonstrates strong association capability with the highest IDFI (76.2%) and lowest ID switches (10,393), confirming the effectiveness of our diffusion-basee -based approach for object association.
B.4 Scalability Analysis
To analyze the computational efficiency with larger category sets, we conduct comprehensive exper- iments with class counts ranging from 10 to 1200 in Table 7. We measure the average time taken for dynamic group contrastive learning per batch, which includes both the construction of dynamic groups and the calculation of contrastive learning loss.
Table 7: Scalability analysis of dynamic group contrastive learning.
Class counts
10
100
200
400
800
1200
Sec/batch
0.11 0.13
0.15
0.20
0.24
0.27
The results demonstrate good scalability of the proposed method, Even at 1200 classes, the average time consumed per batch is only 0.27 seconds, with the majority of this time spent on dynamic group construction. The time required for contrastive learning loss calculation remains below 0. seconds across all class counts, demonstrating that our dynamic group contrastive learning approach scales efficiently and is not computationally prohibitive for very large category sets.
C Qualitative Analysis
We compare our method with the baseline method OVTrack across several challenging scenarios involving novel object classes. As shown in Figure 4, in the first construction site scene, our approach effectively and accurately tracks fast-moving drones, whereas OVTrrack fails to detect the drones at all. Moreover, our method precisely classifies the bulldozer as a novel object category, while truck and initially fails detect the object entirely. Our method also
OVTrack misclassifies demonstrates superior detection and tracking capabilities for base object classes (persons).
In the second racing scenario, characterized by high-speed vehicles and occlusion, successfully tracks and correctly classifies the race cars. In contrast, OVTrack struggles to detect occluded targets and exhibits incorrect ID switching. Its classification is also less precise, categorizing the vehicles under the broader "car" class instead of the specific "race car" category.
method
Figure 5 depicts scene from the African savanna, featuring novel category hippopotamus and two lions chasing Our method successfully classifies the hippopotamus correctly, whereas OVTrack misidentifies it as "clephant". Additionally, OVTrack incorrectly labels the lion "horse" and

---

"cow". Moreover, our approach demonstrates superior detection and tracking accuracy compared to OVTrack.
Figure iillustrate the tracking performance in a field scenario, featuring a fast-moving dragonfly belonging novel category. Compared to OVTrack, proposed method demonstrates superior detection and tracking capabilities, successfully identifying the dragonfly. In contrast, OVTrack fails detect the dragonfly in most frames and cannot accurately classify it.
These results demonstrate that our method, through efficient training, significantly enhances localiza- tion, classification, and association capabilities across diverse and challenging tracking scenarios.
NGD
OVTrack
668
k3i
A45
OVTrack
Figure 4: Visualization results from construction sites and race tracks with novel object categories, including drones and bulldozers the construction scene, along with race cars in the racing track.
#80
OVTrack
#40
#80
#110
Ours
Figure 5; Visualization results in the African savanna with the novel object category, hippopotamus.
25

---

#20
#40
#60
OV Track
#20
#40
#60
Ours
M20
#60
OVTrack
Ours
Figure 6: Visualization results the field with the novel object category, dragonfly.
26


Reviews:
[{"review_id": "XFaZ67cBQL", "reviewer": "Reviewer_NEJd", "summary": "This paper aims to solve the problem of limited annotated data of the Open-Vocabulary Multi-Object Tracking task.   The diffusion modles are introduced to complement the missing objects at the gaps between annotated frame. And the contrastive learning strategy is designed for object detection task. The proposed method with several elaborate-designed modules achieves advanced results on TAO dataset.", "questions": "Please refer to the weakness.", "limitations": "This paper does not discuss limitations. The authors could further discuss the limitations of diffusion models on tracking tasks.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The designed method achieves promising results on the TAO dataset.\n2. Appling the diffusion-based model to solve the problem of parse annotations is novel in the fieled of Open-Vocabulary Multi-Object Tracking.\n3. Sufficient experiments have verified the effectiveness of the proposed method.\n\nWeaknesses:\n1. What unique theoretical contribution does this work make in the field of diffusion model research?\n2. The deisgned Diffusion-based Data Generation module does not seem to be specially designed for tracking tasks. Can it be used for other sparsely annotated video tasks?\n3. The author argued that the pseudo labels generated by SLAck are only effective for adjacent frames. How to prove that this paer solves the problem better?\n4. For the  detection model，is it necessary to use TAO data for fintune? Will this approach reduce the generalization performance of the model in real scenarios?\n5. Considering the complexity of the proposed method, it would be better to provide code to improve the reproducibility of the paper.", "was_revised": true, "final_justification": "My initial concerns were focused on the unique theoretical contribution and insufficient experiments.\nThe authors have provided more experimental results and claerly illustrate the contribution of this paper.\nTherefore, I keep my score for borderline acceptance.", "rebuttal": "We thank Reviewer NEJd for the positive evaluation and constructive feedback. We address each concern below:\n\n> ### 1. Theoretical Contributions\n\nOur work mainly focuses on and takes the first step for the application of diffusion models on OVMOT, which also makes several key theoretical contributions to diffusion models:\n\n**(a) Novel diffusion paradigm:** We pioneer the application of diffusion models to feature-level temporal interpolation in tracking, fundamentally departing from traditional pixel-level generation approaches. Unlike existing diffusion methods that focus on final denoised outputs, we innovatively utilize intermediate denoising results as meaningful feature representations, modeling the denoising process as temporal evolution of object features between sparse annotations.\n\n**(b) Sparse-to-continuous mapping rationale:** We demonstrate theoretically and empirically that diffusion models can effectively bridge temporal gaps in sparse video annotations by simulating object state transitions. It provides the first systematic approach to transform sparse temporal annotations into continuous feature representations, enabling robust tracking learning.\n\n**(c) Multi-objective optimization for temporal sequence completion:** We establish a theoretically grounded training framework that combines three complementary losses:\n- **Reconstruction loss:** ensuring feature reliability through interpolation supervision\n- **Endpoint loss:** guaranteeing accurate feature alignment  \n- **Smoothness loss:** enforcing temporal consistency\n\nThis loss combination also provides a reference for generating reliable intermediate object features with diffusion models.\n\n> ### 2. Generalizability to Other Video Tasks\n\nFrom the perspective of the original intention, our diffusion-based data generation is specifically designed for open-vocabulary tracking tasks, as it effectively simulates object motion changes and state transitions between sparse annotations. The diffusion model captures the temporal dynamics essential for tracking by modeling feature evolution across time gaps.\n\nWhile primarily designed for tracking, the core principle of generating intermediate features between sparse annotations is task-agnostic and can be extended to other sparsely annotated video tasks, such as video action localization and motion trajectory prediction.\n\nThanks for the reviewer's comments, we are willing to explore these insights for other tasks in future work.\n\n> ### 3. Comparison with SLAck\n\nWe provide quantitative evidence that our method outperforms SLAck's pseudo-labeling approach:\n\n**(a) Method and principle:**\nSLAck generates pseudo labels by computing the IoU matching with each annotated frame (one for every 30 frames). We conducted a statistical analysis of SLAck's pseudo-labeling method and found that for each annotated frame, only the adjacent (previous and next) **7.8 frames on average** are used for generating pseudo labels. In contrast, our diffusion-based approach can work across the **entire temporal gaps (30 frames)** between annotated frames.\n\n**(b) Results and comparison:**\nWe achieve **4.1% higher TETA score** than SLAck on validation set for novel classes (35.2% vs 31.1%) and **3.2%** for base classes (40.4% vs 37.2%). Our association accuracy (AssocA) significantly outperforms SLAck by **2.5%** on novel classes (40.3% vs 37.8%) and **4.5%** on base classes (42.1% vs 37.6%), demonstrating the effectiveness of our approach.\n\n**(c) Less demand and more cues:**\nWe achieve these improvements **without using any intermediate un-annotated frames**, which are necessary in SLAck. Additionally, SLAck's IoU-based pseudo-labeling approach cannot consider motion continuity, while our method fully considers motion continuity through reconstruction loss and smoothness loss, ensuring smooth feature transitions.\n\n> ### 4. Detection Model Fine-tuning Necessity\n\nThe reviewer asks about the necessity of the fine-tuning stage and generalization performance.\n\n**(1) Effectiveness without detection fine-tuning:**\nWe conduct extra ablation experiments that demonstrate that even without detection fine-tuning (using only our diffusion-based association enhancement), our method still achieves state-of-the-art results. As shown in the table below, removing detection fine-tuning (on TAO), our method still yields competitive performance, especially for the association performance, indicating the effectiveness of our approach.\n\n| Method | Novel TETA | Novel LocA | Novel AssoA | Novel ClsA | Base TETA | Base LocA | Base AssoA | Base ClsA |\n|--------|------------|------------|-------------|------------|-----------|-----------|------------|-----------|\n| OVTrack | 27.8 | 48.8 | 33.6 | 1.5 | 35.5 | 49.3 | 36.9 | 20.2 |\n| SLAck | 31.1 | 54.3 | 37.8 | 1.3 | 37.2 | 55.0 | 37.6 | 19.1 |\n| OVTR | 31.4 | 54.4 | 34.5 | 5.4 | 36.6 | 52.2 | 37.6 | 20.1 |\n| **Ours (w/o det. fine-tune)** | **32.4** | 55.3 | **39.9** | 2.1 | **38.7** | 55.3 | **41.1** | 19.8 | \n||\n\n**(2) Zero-shot generalization capability:**\n\nTo demonstrate zero-shot generalization capability, we conducted experiments by applying our method on BDD100K validation dataset *without any fine-tuning on BDD100K*, showing superior cross-dataset transfer performance. We can see that our method achieves significant improvements across all metrics (TETA: +2.7%, LocA: +2.1%, AssocA: +2.3%, ClsA: +3.8% over the second-best method OVTR), demonstrating enhanced generalization capability.\n\n| Methods | TETA | LocA | AssocA | ClsA |\n|---------|------|------|--------|------|\n| QDTrack | 32.0 | 25.9 | 27.8 | 42.4 |\n| TETer | 33.2 | 24.5 | 31.8 | 43.4 |\n| OVTrack | 42.5 | 41.0 | 36.7 | 49.7 |\n| OVTR | 43.1 | 42.0 | 37.1 | 50.1 |\n| **Ours** | **45.8** | **44.1** | **39.4** | **53.9** |\n||\n\n**(3) Comprehensive training paradigm rationale:**\n\nOverall, the proposed method benefits from but is not dependent on the detection fine-tuning (on specific dataset, e.g., TAO). The fine-tuning strategy of our method (complete version) constructs a comprehensive data-efficient training paradigm that maximizes the utility of limited sparse annotations. Fine-tuning with our proposed dynamic group contrastive learning and adaptive localization loss specifically addresses small dataset adaptation challenges while maintaining generalization, creating a holistic approach that leverages both association improvements (via diffusion) and detection improvements (via fine-tuning) to achieve optimal performance under data-constrained conditions.\n\n> ### 5. Code Availability\n\nWe commit to releasing all the code to ensure reproducibility upon acceptance of this paper."}, {"review_id": "kqT12gEFtZ", "reviewer": "Reviewer_8GVG", "summary": "This paper tackles Open-Vocabulary Multiple Object Tracking (OV-MOT) with limited and sparsely annotated video data, using the TAO dataset as a case study. The authors propose DOVTrack, a data-efficient training strategy with two main contributions. First, for the association task, they introduce a novel diffusion model that generates object features for the unannotated frames between sparse annotations, effectively creating a continuous stream of training data in the feature space. Second, to improve the open-vocabulary detector, they propose a dynamic group contrastive learning strategy to augment the number and diversity of classification samples, and an adaptive localization loss to incorporate more, lower-confidence bounding boxes into training without destabilizing it. The method achieves state-of-the-art results on the TAO benchmark without using any external data.", "questions": "1.  Could you please elaborate on the training details for the association head? Specifically, once the intermediate features Fassot​ are generated by the diffusion model, how are they used? Are they treated as positive samples for the key feature Fkey​ and reference feature Fref​ in a contrastive loss? \n2.  The primary weakness is the lack of a limitations section. Could you discuss the potential failure cases of the diffusion-based feature generation? For instance, how does the model perform when an object undergoes significant appearance changes, leaves and re-enters the frame, or is occluded for a long duration between two annotated keyframes?", "limitations": "No, the authors explicitly state in the checklist that the paper does not discuss limitations. This is a significant omission.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The key idea of using a diffusion model to interpolate object features between sparsely annotated frames, is novel for tracking and a clever departure from typical pixel-level generation. \n\n2. The work tackles a very important and practical problem. The lack of large, densely annotated video datasets is a major bottleneck for OVMOT. By demonstrating that a high-performing tracker can be trained using only the sparse TAO dataset, this work provides a valuable data-efficient training paradigm for the community.\n\n3. The results are strong, showing a clear state-of-the-art performance on the TAO benchmark. The ablation demonstrates the positive impact of each individual component, from the diffusion model's specific loss functions to the classification and localization enhancement strategies.\n\nWeakness:\n\n1. The most significant weakness is the complete absence of a \"Limitations\" section, which the authors acknowledge in their checklist. \n2. The method section, particularly the diffusion model part, could be explained more clearly. The paper would benefit from a more detailed description of how the generated intermediate features (Fassot​) are practically incorporated into the association head's contrastive training loss. \n3.  The diffusion model's reconstruction loss (Eq. 3) enforces a linear interpolation of features between keyframes. This is a strong simplification, as true object dynamics (e.g., turns, acceleration, deformation) are inherently non-linear. Forcing features to follow a linear path is a potentially brittle assumption that may not capture the complexity of real-world object transformations.", "was_revised": true, "final_justification": "My initial \"borderline\" rating was due to concerns about the unsubstantiated linear interpolation assumption and the lack of a specific failure analysis for the core feature generation module.\n\nThey conducted new, targeted experiments that provided the exact empirical evidence I requested. They successfully validated their assumption and provided a detailed breakdown of the feature generator's specific failure modes and operational boundaries.\n\nI have raised my score and now recommend acceptance.", "rebuttal": "We thank Reviewer 8GVG for the thorough evaluation and insightful comments. We address each point below:\n\n> ### 1. Association Head Training Details\n\nWe provide a comprehensive explanation of how generated intermediate features enhance association training from both quantitative and qualitative perspectives:\n\n**(1) Quantitative sample space expansion:** Our diffusion model fundamentally transforms sparse association training by generating intermediate features $F\\_{\\text{asso}}^t$ at multiple timesteps between annotated frames. For each object trajectory originally containing only two annotated features ($F\\_{\\text{key}}$ and $F\\_{\\text{ref}}$ separated by 30 frames), our method generates $N$ intermediate features where $N$ corresponds to sampling steps (typically 3). This directly expands the positive sample set $Q^+(\\mathbf{q})$ for each object identity from 2 to 5 features. Simultaneously, the generated features from different object trajectories enrich the negative sample set $Q^-$, creating a much denser and more comprehensive sample space $Q = Q^+ \\cup Q^-$. Consequently, the total training samples increase by approximately 3×, providing substantially more positive and negative pairs for robust contrastive learning.\n\n**(2) Qualitative feature continuity enhancement:** Beyond quantity expansion, our generated intermediate features $F\\_{\\text{asso}}^t$ exhibit superior temporal continuity compared to sparse annotations. The diffusion-based generation ensures smooth feature transitions through: 1) reconstruction loss enforcing accurate interpolations between keyframes; 2) smoothness loss guaranteeing consistent feature evolution; 3) endpoint loss ensuring precise alignment with reference features. This results in temporally coherent feature representations that better capture object state transitions, providing higher-quality training samples for association learning.\n\n**(3) Enhanced loss construction:** Building upon the quantitative expansion from (1) and qualitative enhancement from (2), as mentioned in the implementation details of the paper, we employ the same association loss framework as OVTrack but with significantly enhanced effectiveness due to our enriched sample space. The complete tracking loss $\\mathcal{L}\\_{\\text{track}}$ is formulated as:\n\n$$\\mathcal{L}\\_{\\text{track}} = -\\sum\\_{\\mathbf{q} \\in Q} \\frac{1}{|Q^+(\\mathbf{q})|} \\sum\\_{\\mathbf{q}^+ \\in Q^+(\\mathbf{q})} \\log\\left(\\frac{\\exp(\\mathbf{q} \\cdot \\mathbf{q}^+/\\tau)}{\\text{PosD}(\\mathbf{q}) + \\sum\\_{\\mathbf{q}^- \\in Q^-(\\mathbf{q})} \\exp(\\mathbf{q} \\cdot \\mathbf{q}^-/\\tau)}\\right)$$\n\nwhere $\\text{PosD}(\\mathbf{q}) = \\frac{1}{|Q^+(\\mathbf{q})|} \\sum\\_{\\mathbf{q}^+ \\in Q^+(\\mathbf{q})} \\exp(\\mathbf{q} \\cdot \\mathbf{q}^+/\\tau)$ and $\\tau$ is the temperature parameter. The enriched $Q^+(\\mathbf{q})$ contains both original keyframe features and our generated intermediate features (from point 1), while $Q^-(\\mathbf{q})$ includes negative samples from different trajectories. This formulation maximizes positive sample similarity (numerator) while minimizing negative sample similarity (denominator), leveraging both the expanded sample quantity from (1) and improved feature quality from (2) for more robust association learning.\n\n> ### 2. Linear Interpolation Assumption\n\nOur method uses linear interpolation for feature reconstruction, but this is only applied over very small temporal intervals (30 frames between consecutive annotations). It is well-justified by the calculus principles that any smooth function can be approximated linearly over a sufficiently small interval. For such short temporal intervals in video sequences, we use the linear interpolation assumption in this work.\n\nFor longer time spans, object motion patterns are inherently non-linear, involving complex transformations like turns, acceleration and deformation. Within the short intervals where we apply linear interpolation, such dramatic changes rarely occur. Moreover, the smoothness loss in Eq. (5) (in the main paper) further ensures realistic feature transitions.\n\n> ### 3. Challenging Scenarios Analysis\n\nTo comprehensively assess our method's robustness, we systematically selected the most challenging videos from the TAO validation set using multiple quantitative criteria. Videos satisfying any of the following conditions were included in our challenging subset (TAO-Hard):\n\n**(a) Heavy Occlusion Scenarios:** We identified videos with substantial mutual occlusions by computing inter-trajectory IoU overlaps across consecutive frames. Videos were selected if object bounding boxes exhibit IoU > 0.4 with other objects for more than 20% of the total frames, indicating persistent occlusion challenges.\n\n**(b) Rapid Motion Patterns:** We detected fast-moving objects by analyzing trajectory displacement patterns. Specifically, we calculated the ratio of bounding box center displacement to box diagonal length across consecutive frames. Videos with average displacement ratios > 0.3 were selected as they present significant temporal association challenges due to rapid object movement.\n\n**(c) Frequent Entry/Exit Dynamics:** We monitored object appearance and disappearance patterns by tracking trajectory start/end points within video sequences. Videos where objects exhibit frequent entry/exit behaviors (more than 3 entry/exit cycles per trajectory on average) were prioritized to evaluate our method's robustness to discontinuous tracking scenarios.\n\n**(d) High Object Density:** We selected videos containing crowded scenes with high object density, specifically those maintaining more than 8 concurrent objects per frame on average, which creates complex multi-object association challenges.\n\nUsing these systematic criteria, we identified videos presenting the challenging tracking conditions from the TAO validation set, which we refer to as TAO-Hard. The evaluation results are shown below:\n\n| Method | Novel TETA | Novel LocA | Novel AssocA | Novel ClsA | Base TETA | Base LocA | Base AssocA | Base ClsA |\n|--------|------------|------------|--------------|------------|-----------|-----------|-------------|-----------|\n| OVTrack on TAO | 27.8 | 48.8 | 33.6 | 1.5 | 35.5 | 49.3 | 36.9 | 20.2 |\n| OVTrack on TAO-Hard | 25.9(-1.9) | 45.3 | 31.3 | 1.1 | 33.9(-1.6) | 49.1 | 33.4 | 19.3 |\n| OVTR on TAO | 31.4 | 54.4 | 34.5 | 5.4 | 36.6 | 52.2 | 37.6 | 20.1 |\n| OVTR on TAO-Hard | 28.4(-2.0) | 51.2 | 32.1 | 1.9 | 35.2(-1.4) | 51.2 | 35.2 | 19.1 |\n| Ours on TAO | 35.2 | 59.2 | 40.3 | 6.2 | 40.4 | 58.6 | 42.1 | 20.5 |\n| **Ours on TAO-Hard** | **33.4(-1.8)** | **57.6** | **39.2** | **3.4** | **39.2(-1.2)** | **57.1** | **40.3** | **20.1** |\n||\n\nWe can see that the results on all metrics show different levels of degradation on these challenging scenarios, which is natural. Take the association accuracy for example: these challenging scenarios with unpredictable object states, e.g., significant appearance changes, objects leaving and re-entering, make the tracking inherently more difficult. Nevertheless, our method still maintains superior performance compared to existing approaches, demonstrating its strong robustness across diverse scenarios."}, {"review_id": "c43vN1OvQ8", "reviewer": "Reviewer_Eere", "summary": "This paper addresses the characteristics of lacking data annotation or having sparse annotations in videos for the open-vocabulary multi-object tracking task, and proposes using diffusion models to generate object features for intermediate frames to achieve better matching. Additionally, several improvements are proposed for the object detector.", "questions": "- If other methods also adopt the diffusion-model-based enhancement method proposed in this paper, can the method in this paper still achieve an advantage? If other methods use the same detector improvements as this paper, can this paper still maintain the advantage?\n-  I believe the arguments regarding data scarcity in this paper still warrant further discussion. Can state-of-the-art open-vocabulary object detectors address such issues? Can we first train the open-vocabulary detection capability on object detection datasets and then transfer it to multi-object tracking?\n- Please provide more clear depiction on how generated feature used to enhance association training.", "limitations": "The limitations is the same as I mentioned in \"weakness\":\n- The comparison between this paper and other methods lacks rigor. Since this paper has made improvements in both the detector approach and the associated data features, when comparing with other methods, the feature association should be kept constant for comparison, that is, other methods also need to use the same association method. Similarly, for the improvements in the detector part, it is necessary to keep the training data consistent to demonstrate superiority.\n- I believe the arguments regarding data scarcity in this paper still warrant further discussion. Can state-of-the-art open-vocabulary object detectors address such issues? Can we first train the open-vocabulary detection capability on object detection datasets and then transfer it to multi-object tracking?\n- This paper lacks a detailed introduction and comparison of the model, such as the number of parameters, computational complexity, inference speed, etc.\n- This paper provides an unclear description of the method. It proposes generating intermediate features based on diffusion models, but fails to describe how these generated features are used to strengthen the association.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strength\n- The open-vocabulary tracking task focused on in this paper has been less explored so far.\n- The approach of using diffusion models to solely generate features for multi-object tracking is quite interesting.\n\n### Weakness\n- The comparison between this paper and other methods lacks rigor. Since this paper has made improvements in both the detector approach and the associated data features, when comparing with other methods, the feature association should be kept constant for comparison, that is, other methods also need to use the same association method. Similarly, for the improvements in the detector part, it is necessary to keep the training data consistent to demonstrate superiority.\n- I believe the arguments regarding data scarcity in this paper still warrant further discussion. Can state-of-the-art open-vocabulary object detectors address such issues? Can we first train the open-vocabulary detection capability on object detection datasets and then transfer it to multi-object tracking?\n- This paper lacks a detailed introduction and comparison of the model, such as the number of parameters, computational complexity, inference speed, etc.\n- This paper provides an unclear description of the method. It proposes generating intermediate features based on diffusion models, but fails to describe the details that how these generated features are used to strengthen the association.", "was_revised": true, "final_justification": "My initial concerns were primarily focused on the insufficient experiments and unclear descriptions. However, the authors have addressed the issues I raised in their subsequent rebuttal. I now believe that this paper proposes an effective method and also provides a good solution to the problem of data scarcity in the association stage of multi-object tracking. After the authors further revise the main text to improve the descriptions and incorporate the new experimental results, I think this paper meets the borderline acceptance criteria.", "rebuttal": "We thank Reviewer Eere for the detailed feedback. We address each concern systematically in the following.\n\n> ### 1. Controlled ablation studies on component effectiveness:\n\nThe reviewer asked about that since this paper has made improvements in both the detector approach and the associated data features, the evaluation on them should be conducted independently. To address this concern, we conduct systematic ablation studies that evaluate each component respectively to control the experiment conditions.\n\n**(1) Effectiveness of diffusion-based enhancement for different baselines:** The reviewer asks: *\"If other methods also adopt the diffusion-model-based enhancement method proposed in this paper, can the method in this paper still achieve an advantage?\"* To answer this, we integrate our diffusion-based association enhancement (Sec.3.1) into existing OVMOT baselines while keeping other components unchanged.\n\n| Method | Novel TETA | Novel LocA | Novel AssocA | Novel ClsA | Base TETA | Base LocA | Base AssocA | Base ClsA |\n|--------|------------|------------|--------------|------------|-----------|-----------|-------------|-----------|\n| OVTrack | 27.8 | 48.8 | 33.6 | 1.5 | 35.5 | 49.3 | 36.9 | 20.2 |\n| OVTrack + association | 29.7 | 49.7 | 37.7 (+4.1) | 1.8 | 36.2 | 50.3 | 39.4 (+2.5) | 19.8 |\n| MASA(R50) | 30.0 | 54.2 | 34.6 | 1.0 | 36.9 | 55.1 | 36.4 | 19.3 |\n| MASA(R50) + association | 31.7 | 55.3 | 38.6 (+4.0) | 1.3 | 38.3 | 55.4 | 40.1 (+3.7) | 19.5 |\n| **Ours** | **35.2** | **59.2** | **40.3** | **6.2** | **40.4** | **58.6** | **42.1** | **20.5** |\n||\n\nResults demonstrate that diffusion-based association enhancement consistently improves association accuracy (AssocA) across different baselines by approximately 4.0% for novel classes and 2.5/3.7% for base classes, validating the universal effectiveness of this component for association. From the last row, we can see that our method (last row) still achieves the highest TETA scores. This is because that **when using the same association modules as the baseline methods**, our method also shows superior performance on the detection results (LocA and ClsA), which **demonstrates that our method still has the advantages**.\n\n**(2) Effectiveness of detection approach for different baselines:** The reviewer further asks: *\"If other methods use the same detector improvements as this paper, can this paper still maintain the advantage?\"* To verify this, we apply our detection approach (Sec.3.2) to existing baselines.\n\n| Method | Novel TETA | Novel LocA | Novel AssocA | Novel ClsA | Base TETA | Base LocA | Base AssocA | Base ClsA |\n|--------|------------|------------|--------------|------------|-----------|-----------|-------------|-----------|\n| OVTrack | 27.8 | 48.8 | 33.6 | 1.5 | 35.5 | 49.3 | 36.9 | 20.2 |\n| OVTrack + detection | 30.5 | 53.7 (+4.9) | 34.5 | 3.4 (+1.9) | 37.0 | 53.7 (+4.4) | 37.1 | 20.4 (+0.2) |\n| MASA(R50) | 30.0 | 54.2 | 34.6 | 1.0 | 36.9 | 55.1 | 36.4 | 19.3 |\n| MASA(R50) + detection | 32.1 | 57.8 (+3.6) | 35.4 | 3.2 (+2.2) | 38.2 | 57.4 (+2.3) | 37.1 | 20.1 (+0.8) |\n| **Ours** | **35.2** | **59.2** | **40.3** | **6.2** | **40.4** | **58.6** | **42.1** | **20.5** |\n||\n\nThe detection enhancement strategy consistently improves localization (LocA) and classification (ClsA) accuracies on different baselines, demonstrating its general applicability. Similarly, our method consistently outperforms the baselines with the same detector as ours, especially in association-related metric AssoA and the overall metric TETA. This clearly verifies that our method **maintains the advantage compared to the baselines equipped with the same detection enhancement**.\n\n> ### 2. Discussion about the data scarcity\n\nThe reviewer suggests using open-vocabulary detectors or detection datasets. We clarify that existing OVMOT methods already follow a two-stage paradigm: **detection training** (which leverages open-vocabulary detectors/datasets) and **association training** (which requires video data). The challenge lies in the association training stage, where TAO is the only open-vocabulary video dataset but its sparse annotations are insufficient. OVTrack and SLAck have demonstrated that **attempting to train robust OV trackers directly on TAO's sparse annotations fails to produce effective results**.\n\nTo address this fundamental problem, OVTrack and subsequent methods (OVTR, MASA, etc.) use LVIS image pairs to simulate videos for association training. However, **these image pairs lack motion continuity and appearance consistency**, so the video data scarcity problem persists. SLAck is currently the only method that attempts to directly tackle this sparse annotation problem by using pseudo-labeling on TAO's intermediate unannotated frames. While it achieved improvements, it suffers from noise contamination and computational overhead.\n\nOur diffusion-based approach provides a cleaner solution by generating reliable intermediate features in the feature space without requiring any intermediate unannotated frames. We achieve significant improvements over SLAck: **4.1% TETA improvement** on novel classes (35.2% vs 31.1%) and **3.2%** on base classes (40.4% vs 37.2%), with particularly substantial gains in association accuracy of **2.5%** on novel classes and **4.5%** on base classes.\n\n> ### 3. Model complexity analysis\n\nThe reviewer asks about model complexity, parameters, and inference speed. Our method maintains comparable computational complexity to existing methods while achieving the **highest FPS (15.3)** among all compared approaches on a single 3090 GPU. Compared to OVTrack, we add minimal training overhead and no inference cost since the diffusion model only generates features during training.\n\n| Method | Input shape | Test FLOPs | Train FLOPs | Parameters | Model Size | FPS |\n|--------|-------------|------------|-------------|------------|------------|-----|\n| QDTrack | (3,800,1334) | 398.93G | 401.17G | 15.47M | 298.6M | 13.8 |\n| OVTrack | (3,800,1334) | 423.60G | 410.72G | 16.52M | 283.77M | 1.8 |\n| OVTR | (3,800,1334) | 535.35G | 487.58G | 59.36M | 227.29M | 3.4 |\n| **Ours** | (3,800,1334) | 423.60G | 408.66G | 17.57M | 287.79M | **15.3** |\n||\n\n\n> ### 4. Method clarity about association training\n\nThe reviewer requests clearer explanation of how generated features enhance association training. We provide a comprehensive explanation from both quantitative and qualitative perspectives:\n\n**(1) Quantitative sample space expansion:** Our diffusion model fundamentally transforms sparse association training by generating intermediate features $F\\_{\\text{asso}}^t$ at multiple timesteps between annotated frames. For each object trajectory originally containing only two annotated features ($F\\_{\\text{key}}$ and $F\\_{\\text{ref}}$ separated by 30 frames), our method generates $N$ intermediate features where $N$ corresponds to sampling steps (typically 3). This directly expands the positive sample set $Q^+(\\mathbf{q})$ for each object identity from 2 to 5 features. Simultaneously, the generated features from different object trajectories enrich the negative sample set $Q^-$, creating a much denser and more comprehensive sample space $Q = Q^+ \\cup Q^-$. Consequently, the total training samples increase by approximately 3×, providing substantially more positive and negative pairs for robust contrastive learning.\n\n**(2) Qualitative feature continuity enhancement:** Beyond quantity expansion, our generated intermediate features $F\\_{\\text{asso}}^t$ exhibit superior temporal continuity compared to sparse annotations. The diffusion-based generation ensures smooth feature transitions through: 1) reconstruction loss enforcing accurate interpolations between keyframes; 2) smoothness loss guaranteeing consistent feature evolution; 3) endpoint loss ensuring precise alignment with reference features. This results in temporally coherent feature representations that better capture object state transitions, providing higher-quality training samples for association learning.\n\n**(3) Enhanced loss construction:** Building upon the quantitative expansion from (1) and qualitative enhancement from (2), as mentioned in the implementation details of the paper, we employ the same association loss framework as OVTrack but with significantly enhanced effectiveness due to our enriched sample space. The complete tracking loss $\\mathcal{L}\\_{\\text{track}}$ is formulated as:\n\n$$\\mathcal{L}\\_{\\text{track}} = -\\sum\\_{\\mathbf{q} \\in Q} \\frac{1}{|Q^+(\\mathbf{q})|} \\sum\\_{\\mathbf{q}^+ \\in Q^+(\\mathbf{q})} \\log\\left(\\frac{\\exp(\\mathbf{q} \\cdot \\mathbf{q}^+/\\tau)}{\\text{PosD}(\\mathbf{q}) + \\sum\\_{\\mathbf{q}^- \\in Q^-(\\mathbf{q})} \\exp(\\mathbf{q} \\cdot \\mathbf{q}^-/\\tau)}\\right)$$\n\nwhere $\\text{PosD}(\\mathbf{q}) = \\frac{1}{|Q^+(\\mathbf{q})|} \\sum\\_{\\mathbf{q}^+ \\in Q^+(\\mathbf{q})} \\exp(\\mathbf{q} \\cdot \\mathbf{q}^+/\\tau)$ and $\\tau$ is the temperature parameter. The enriched $Q^+(\\mathbf{q})$ contains both original keyframe features and our generated intermediate features (from point 1), while $Q^-(\\mathbf{q})$ includes negative samples from different trajectories. This formulation maximizes positive sample similarity (numerator) while minimizing negative sample similarity (denominator), leveraging both the expanded sample quantity from (1) and improved feature quality from (2) for more robust association learning.\n\nWe appreciate the reviewer's valuable comments and suggestions, and commit to addressing these concerns in our final version."}, {"review_id": "OzJukcrdsN", "reviewer": "Reviewer_8J7k", "summary": "This paper proposes DOVTrack, a data-efficient open-vocabulary multi-object tracking method, addressing sparse-annotated video data challenges. It uses diffusion models for associating interval features, dynamic group contrastive learning for classification, and adaptive localization loss for detection. Experiments on TAO show state-of-the-art TETA scores (35.2% validation, 38.7% test) without extra data.", "questions": "1) The paper demonstrates strong performance on TAO, but it remains unclear whether the proposed methods (diffusion-based feature generation, dynamic grouping) generalize to other OVMOT scenarios with sparse annotations (e.g., custom datasets with different category distributions, longer temporal gaps between annotations, or more extreme object dynamics). Please supplement experiments on at least one additional dataset (e.g., a subset of KITTI-MOT with sparse annotations or a synthetic sparse-annotated video dataset) and analyze performance trends. If results show consistent improvements over baselines across datasets, this would significantly strengthen the claim of robustness, potentially increasing the evaluation score. Conversely, poor generalization would raise concerns about overfitting to TAO’s specific properties.\n2) Tracking performance in challenging cases (fast-moving objects, heavy occlusion, abrupt viewpoint changes) is critical for real-world applicability, yet the paper does not explicitly evaluate these. Please conduct ablation studies on TAO subsets filtered by such scenarios and compare with baselines.\n3) The paper focuses on TAO (833 categories), but real-world OVMOT may require scaling to datasets with larger categories. Please discuss the computational efficiency of key components:\nHow does the diffusion model’s training/inference time scale with the number of categories or temporal gaps between annotations?\nDoes dynamic group contrastive learning become computationally prohibitive for very large category sets?", "limitations": "Please refer to the above weakness and questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\nThe paper demonstrates strong empirical rigor. Experiments are conducted on the TAO dataset (a standard OVMOT benchmark), with comprehensive comparisons against SOTA methods (e.g., OVTrack, SLAck, OVSORT) and ablation studies verifying the effectiveness of key components (diffusion-based feature generation, dynamic group contrastive learning, adaptive localization loss). The results are statistically consistent, with clear improvements in TETA (35.2% on validation, 38.7% on test) and breakdown metrics (LocA, AssocA, ClsA).\n\nWeakness:\n1) While results on TAO are strong, generalizability is underexplored. The paper does not test performance on other OVMOT datasets (e.g., extended splits or custom sparse-annotated videos), leaving uncertainty about whether the methods scale beyond TAO’s specific distribution. For example KITTI has been used in compared works, only one dataset is investigated in this submission.\nAdditionally, ablation studies on \"extreme cases\" (e.g., fast-moving objects, heavy occlusion) are lacking, despite these being critical for tracking robustness.\n2) The work focuses narrowly on TAO, a dataset with 833 categories but small size. It does not discuss how the methods would adapt to larger sparse-annotated datasets, where computational efficiency of diffusion-based feature generation or dynamic grouping might become a bottleneck. This limits discussion of real-world scalability.", "was_revised": true, "final_justification": "The authors have addressed my main concerns regarding the model generalization ability, I'd like to raise my score to 4.", "rebuttal": "We thank Reviewer 8J7k for acknowledging our empirical rigor and providing constructive feedback on generalizability:\n\n> ### 1. Evaluation on Additional Dataset\n\nThe reviewer suggests to supplement experiments on one additional dataset with sparse annotations and analyze performance trends. Since KITTI-MOT only conducts the evaluation on two categories (person and vehicle), we use BDD100K dataset containing more categories. We conduct comprehensive experiments on BDD100K containing 8 categories to address this concern from two perspectives as below.\n\n**① Sparse annotation interval analysis:** Following the reviewer's suggestion, to validate our method's effectiveness under different sparse annotation conditions, we conduct training experiments on BDD100K with different temporal-interval annotations. Since BDD100K is annotated at 5 fps, the minimum interval is 5 frames. We evaluate our method and baselines using the annotation intervals of 5, 15, and 30 frames.\n\n| Methods | TETA | LocA | AssocA | ClsA |\n|---------|------|------|--------|------|\n| TETer + 5-frame interval | 50.8 | 47.2 | 52.9 | 52.4 |\n| OVTrack + 5-frame interval | 50.9 | 46.1 | 53.4 | 53.1 |\n| OVTR + 5-frame interval | 52.5 | 48.9 | 53.9 | 54.7 |\n| |\n| Ours + 30-frame interval | 52.1 | 47.9 | 53.8 | 54.6 |\n| Ours + 15-frame interval | **52.6** | 48.3 | 54.3 | 55.2 |\n| **Ours + 5-frame interval** | **53.4** | 49.1 | 55.7 | 55.4 |\n||\n\nThese results demonstrate three key findings: \n1) Our method with 30 frame intervals achieves comparable performance to baseline methods trained with dense 5 frame intervals, highlighting the effectiveness of our diffusion-based approach for handling sparse annotations. \n2) With 15 and 5 frame intervals, our method significantly outperforms all existing approaches, demonstrating superior performance across different annotation densities. \n3) The consistently good performance across different intervals validates the robustness and general applicability of our data-efficient training paradigm.\n\n**② Cross-dataset generalization evaluation:** To further verify the generalization capability of our method, we conduct zero-shot experiments on BDD100K (training on TAO and testing on BDD100K) validation set without any training on this dataset.\n\n| Methods | TETA | LocA | AssocA | ClsA |\n|---------|------|------|--------|------|\n| QDTrack | 32.0 | 25.9 | 27.8 | 42.4 |\n| TETer | 33.2 | 24.5 | 31.8 | 43.4 |\n| OVTrack | 42.5 | 41.0 | 36.7 | 49.7 |\n| OVTR | 43.1 | 42.0 | 37.1 | 50.1 |\n| **Ours** | **45.8** | **44.1** | **39.4** | **53.9** |\n||\n\nWe can see that our method achieves significant improvements across all metrics (TETA: +2.7%, LocA: +2.1%, AssocA: +2.3%, ClsA: +3.8% over the second-best method OVTR), demonstrating strong zero-shot cross-dataset generalization capability.\n\n> ### 2. Challenging Scenarios Analysis\n\nTo comprehensively assess our method's robustness, we systematically selected the most challenging videos from the TAO validation set (93 videos of 988 videos) using multiple quantitative criteria. Videos satisfying any of the following conditions were included in our challenging subset (TAO-Hard):\n\n**(a) Heavy occlusion scenarios:** We identified videos with substantial mutual occlusions by computing inter-trajectory IoU overlaps across consecutive frames. Videos were selected if object bounding boxes exhibit IoU > 0.4 with other objects for more than 20% of the total frames, indicating persistent occlusion challenges.\n\n**(b) Rapid motion patterns:** We detected fast-moving objects by analyzing trajectory displacement patterns. Specifically, we calculated the ratio of bounding box center displacement to box diagonal length across consecutive frames. Videos with average displacement ratios > 0.3 were selected as they present significant temporal association challenges due to rapid object movement.\n\n**(c) Frequent entry/exit dynamics:** We monitored object appearance and disappearance patterns by tracking trajectory start/end points within video sequences. Videos where objects exhibit frequent entry/exit behaviors (more than 3 entry/exit cycles per trajectory on average) were prioritized to evaluate our method's robustness to discontinuous tracking scenarios.\n\n**(d) High object density:** We selected videos containing crowded scenes with high object density, specifically those maintaining an average of more than 8 concurrent objects per frame, which creates complex multi-object association challenges.\n\nUsing these systematic criteria, we identified videos presenting the challenging tracking conditions from the TAO validation set, which we refer to as TAO-Hard. The evaluation results are shown below:\n\n| Method | Novel TETA | Novel LocA | Novel AssocA | Novel ClsA | Base TETA | Base LocA | Base AssocA | Base ClsA |\n|--------|------------|------------|--------------|------------|-----------|-----------|-------------|-----------|\n| OVTrack | 27.8 | 48.8 | 33.6 | 1.5 | 35.5 | 49.3 | 36.9 | 20.2 |\n| OVTrack on TAO-Hard | 25.9(-1.9) | 45.3 | 31.3 | 1.1 | 33.9(-1.6) | 49.1 | 33.4 | 19.3 |\n| MASA(R50) | 30.0 | 54.2 | 34.6 | 1.0 | 36.9 | 55.1 | 36.4 | 19.3 |\n| MASA(R50) on TAO-Hard | 27.6(-2.4) | 50.3 | 31.2 | 1.2 | 35.0(-1.9) | 52.3 | 34.1 | 18.7 |\n| OVTR | 31.4 | 54.4 | 34.5 | 5.4 | 36.6 | 52.2 | 37.6 | 20.1 |\n| OVTR on TAO-Hard | 28.4(-2.0) | 51.2 | 32.1 | 1.9 | 35.2(-1.4) | 51.2 | 35.2 | 19.1 |\n| Ours | 35.2 | 59.2 | 40.3 | 6.2 | 40.4 | 58.6 | 42.1 | 20.5 |\n| **Ours on TAO-Hard** | **33.4(-1.8)** | **57.6** | **39.2** | **3.4** | **39.2(-1.2)** | **57.1** | **40.3** | **20.1** |\n||\n\nFrom the results we can first see that, on the subset of challenging scenarios, our method still obtains the best performance with significant margins compared with other methods. The results also show that our method exhibits the smallest performance degradation across challenging scenarios. While all methods experience some decline in performance under extreme conditions, our approach shows the most minimal degradation (1.8% for novel and 1.2% for base classes), compared to baseline methods that suffer degradations of 1.9% -- 2.4%. This superior robustness stems from the proposed association and detection strategies that provide more coherent object representations, enabling better handling of challenging tracking scenarios.\n\n> ### 3. Computational Scalability\n\nRegarding the scalability of our method for larger category datasets, we discuss about the computational efficiency of key components in detail as below.\n\n**(a) Scalability of the diffusion model:** The training and inference time of our diffusion model are essentially independent of the number of categories, meaning that an increase in category count does not incur additional computational overhead. The diffusion model operates in the feature space and is primarily influenced by the sampling steps, which determine the number of interpolated frames between adjacent frames. As indicated by our ablation results (Table 3 in the paper), optimal performance can be achieved with just three sampling steps, demonstrating that the associated overhead is minimal regardless of temporal gaps.\n\n**(b) Scalability of the dynamic group contrastive learning:** To directly address the reviewer's concern about computational efficiency with larger category sets, we conducted comprehensive experiments with class counts ranging from 10 to 1200. We measured the average time taken for dynamic group contrastive learning per batch, which includes both the construction of dynamic groups and the calculation of contrastive learning loss.\n\n| Class counts | 10 | 100 | 200 | 400 | 800 | 1200 |\n|--------------|----|----|-----|-----|-----|------|\n| Sec/batch | 0.11 | 0.13 | 0.15 | 0.20 | 0.24 | 0.27 |\n||\n\n\nThe results demonstrate the good scalability of the proposed method. Specifically, even at 1200 classes, the average time consumed per batch is only 0.27 seconds, with the majority of this time spent on dynamic group construction. The time required for contrastive learning loss calculation remains below 0.1 seconds across all class counts. This demonstrates that our dynamic group contrastive learning approach scales efficiently and is not computationally prohibitive for very large category sets.\n\n**(c) Real-world applicability:** Our approach is well-suited for real-world deployment as: 1) the computational overhead does not scale significantly with dataset size, 2) it reduces annotation requirements while maintaining performance, and 3) the method achieves the highest inference speed (15.3 FPS on a single 3090 GPU) among all comparison methods, making it promising for real-time applications."}]

Author rebuttal:

Meta-review: The manuscript addresses the problem of open-vocabulary multi-object tracking. There two technical contributions, a diffusion-based model to simulate the features of targets and a dynamic group contrastive learning strategy to improve the classification. The proposed methods are evaluated in SOTA experiments. The main strengths are the importance of the problem, the diffusion model, and the good results. There are several minor weaknesses regarding the evaluation, reflections, and rigor. As the weaknesses are minor, the paper is accepted.
The manuscript received originally four reviews and after rebuttal and discussion, all reviewers are rating accept or BA. While most reviewers agree on the strengths listed above, they mention different weaknesses and do not support each others statements during the discussion. This is the main reason for considering the weaknesses as minor.
Decision: Accept (poster)