VOVTrack: Exploring the Potentiality in Videos for Open-Vocabulary Object Tracking

Authors: Zekun Qian, Ruize Han, Junhui Hou, Linqi Song, Wei Feng
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Object Tracking, Open-Vocabulary

Abstract:
Open-vocabulary multi-object tracking (OVMOT) represents a critical new challenge involving the detection and tracking of diverse object categories in videos, encompassing both seen categories (base classes) and unseen categories (novel classes). This issue amalgamates the complexities of open-vocabulary object detection (OVD) and multi-object tracking (MOT). Existing approaches to OVMOT often merge OVD and MOT methodologies as separate modules, predominantly focusing on the problem through an image-centric lens. In this paper, we propose OVTracker, a novel method that integrates object states relevant to MOT and video-centric training to address this challenge from a video object tracking standpoint. First, we consider the tracking-related state of the objects during tracking and propose a new prompt-guided attention mechanism for more accurate localization and classification (detection) of the time-varying objects. Subsequently,
we leverage raw video data without annotations by formulating a self-supervised object similarity learning technique to facilitate temporal object association (tracking). Experimental results underscore that OVTracker outperforms existing methods, establishing itself as a state-of-the-art solution for open-vocabulary tracking tasks.

Decision: reject
Reviewer scores: 5, 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730010053726

thread_type:
official_review

<thread_object>
creator: reviewer_h6oh

<content>
summary:
This paper proposed a new tracking-related prompt-guided attention for the localization and classification, and develops a self-supervised object similarity learning strategy for the temporal association in OVMOT. Experiments demonstrate that the proposed method achieves state-of-the-art tracking performance compared to previous methods.

strengths:
1. This paper propose a new tracking-related prompt-guided attention for the localization and classification in OVMOT​.
2. The self-supervised learning strategy leverages unlabeled video data for the temporal association, addresses the challenge of training data shortage.
3. Extensive experimental results, demonstrating that VOVTracker outperforms existing methods on TAO benchmark.

weaknesses:
1. Lack of novelty. The self-supervision is common and there are some considerable work in vision tasks. However, this paper just applies it to OVMOT, and does not compare the proposed method with previous methods. What's the main difference and contributions for OVMOT.
2. While VOVtrack leverages unlabeled video data, its performance seems to depend on the quality and representativeness of the videos, which could affect its robustness across different real-world conditions​.
2. The authors should conduct on the ablation sduty of different parameter settings. Such as parameters related to training and inference.

questions:
1. VOVTrack lacks of a clear definition, what's the full name?
2. For Table 1, the propsed VOVtrack performs worse than OVTrack in terms of ClsA metric. What is the reason behind it?
3. For Table 2, it demonstrate the effectiveness of Prompt-guided attention. But the baseline with the self-supervised association module fail to obtain obvious improvements, so that the contribution can not be effectively convinced.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730150877320

thread_type:
official_review

<thread_object>
creator: reviewer_n3po

<content>
summary:
This work proposes a series of improvements to the existing OV MOT framework (OVTrack), including a novel prompt-guided attention mechanism and a self-supervised object similarity learning method. With the support of additional video data, these enhancements achieve superior performance on the TAO dataset compared to OVTrack.

strengths:
1. Competitive OV tracking performance on TAO.
2. Self-supervised object similarity learning is compatible with unlabeled video data.

weaknesses:
1. Novelty is limited: The overall architecture of the proposed OVOTrack is based on OVTrack. The specific implementation within the tracking-state aware prompt approach does not exhibit a clear association with the tracking state. Moreover, this method does not assess object quality specifically for tracking scenarios, as it remains overly limited to factors such as occlusion and blurriness. On the other hand, the self-supervised object similarity learning proposed in this work closely resembles QDTrack, with no significant differences observed.
2. Writing requires further improvement: The overall writing is somewhat scattered and lacks coherence, with inconsistencies between the motivation of the paper and the proposed approach, raising concerns of possible over-claim. There is considerable redundancy in language, with some sections being overly simplistic while others are unnecessarily verbose. Additionally, certain assumptions and notations in the methodology are not rigorously standardized (e.g., line 218). Therefore, I believe there is substantial room for improvement in the writing.  
3. Experimental setting: The training process described in the paper involves four stages, which makes it overly complex and cumbersome. Compared to OVTrack, this work utilizes a substantial amount of additional TAO video data for training; however, the classification performance on the base categories in the TAO benchmark has declined. Moreover, the ablation study results indicate that the OVOTrack model consistently underperforms in classification metrics, yet no reasonable explanation is provided for this issue. The examples provided in the visualization results are overly simplistic, failing to showcase the model’s true capabilities.

questions:
1. The primary assertion of the paper is that existing OV tracking methodologies do not take into account the states of objects. However, the proposed approaches appear to be largely unrelated to tracking object states, and the introduced prompt, .e.g., occuded, and complete, seems overly naive in my view.
2. The proposed classification method only focuses on the high-quality object. However, exclusively training the classifier on high-quality targets may result in the neglect of low-quality targets that are blurred or occluded. This seems inconsistent with the paper’s claim of addressing issues related to blurring and occlusion in object tracking. Furthermore, could this be the reason for the model’s lower classification performance on the base?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730260144198

thread_type:
official_review

<thread_object>
creator: reviewer_a19j

<content>
summary:
This work presents a novel open-vocabulary multi-object tracking method that integrates object states based on prompt learning. Different from existing works, it combine OVD and MOT in a unified framework. Some self-supervised losses are designed to learning better object associations. Experiments on public dataset demonstrate the effectiveness of the proposed method.

strengths:
A novel method that integrates object states based on prompt learning is proposed to combine OVD and MOT for open-vocabulary multi-object tracking.

Some self-supervised losses are designed to learning better object associations.

Experiments are conducted on public dataset.

weaknesses:
The following details are unclear. Are the designed prompts only used in training procedure? It would be better if visualized results are provided to validate the effectiveness of these prompts in handling challenging frames with occlusions or motion blur.

There is no comparison with works published in 2024, and the effectiveness of the proposed method is thus not fully validated. The relevent and recent trackers including both closed-set and open-vocabulary ones should be included in comparison.

The results suggest that the proposed method performs abnormal under ClsA metric in both result comparison and ablation study. Though it is better than all methods in most metrics, but it performs worse than OVTrack and OVTrack+RegionCLIP in a clear margin, and also worse than other methods in some cases. The reasons behind these results are not well explained.

Some errors:

Line 256:  into ore framework model

Line 456: Our method

questions:
Some important details and explanations should be provided for clarity.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730586283361

thread_type:
official_review

<thread_object>
creator: reviewer_avgc

<content>
summary:
The paper introduces Open-vocabulary multi-object tracking (OVMOT), a significant challenge that involves detecting and tracking various object categories in videos, including both known (base classes) and unknown (novel classes) categories. The authors critique existing OVMOT methods for treating open-vocabulary object detection (OVD) and multi-object tracking (MOT) as separate modules, primarily focusing on image-based approaches. To address this, they present VOVTrack, which integrates object states relevant to MOT with video-centric training, approaching the challenge from a video object tracking perspective. VOVTrack features a prompt-guided attention mechanism that enhances the localization and classification of dynamic objects, and it employs a self-supervised object similarity learning technique for tracking using raw, unlabeled video data. Experimental results demonstrate that VOVTrack outperforms current methods, establishing it as a leading solution for open-vocabulary tracking tasks.

strengths:
1. This manuscript is standardized and the writing is fluent, and the content is easy to understand.

2. This manuscript proposes a new tracking-related prompt-guided attention for the localization and classification (detection) in the open vocabulary tracking problem. This method takes notice of the states of the time-varying objects during tracking, which is different from the open-
vocabulary object detection from a single image.

3. This manuscript develops a self-supervised object similarity learning strategy for the temporal association (tracking) in the OVMOT problem. This strategy, for the first time, makes full use of the raw video data without annotation for OVMOT training, thus addressing the problem of training data shortage and eliminating the heavy burden of annotation of OVMOT.

4. Experimental results on the public benchmark demonstrate that the proposed VOVTrack achieves the best performance with the same training dataset (annotations), and comparable performance with the methods using a large dataset (CC3M) for training.

weaknesses:
1. The experimental tables lack details on model complexity. It would be helpful to include a table or section comparing FLOPs, parameters, model size, and FPS across the different methods evaluated, including the baseline OVTrack method and other state-of-the-art approaches.

questions:
Please refer to the concerns and issues raised in the "Weaknesses".
</content>
</thread_object>

</thread>
<thread>
creation date:
1731100122229

thread_type:
official_review

<thread_object>
creator: reviewer_6vns

<content>
summary:
The paper introduces Tracking-state-aware prompt guided attention, enabling the network to learn the detection of objects in different tracking states. A self-supervised approach is adopted to train tracking, leveraging large-scale, unlabeled video data across various categories. The experimental results on TAO datasets indicate that the proposed method achieves advanced performance.

strengths:
1. The paper offers a detailed explanation of the proposed method. Utilizing unlabeled data for self-supervision serves, to some extent, as an alternative to address the current scarcity of large vocabulary tracking data. 
2. The results show good performance in comparisons on the OVMOT benchmark.

weaknesses:
1. The CIsA for basic categories in your method is lower than that of the baseline method OVTrack. Given that basic categories account for the majority of targets, the decrease on CIsA appears to reflect more than just typical fluctuation effects. Could this imply that your model's approach introduces a degree of conflict between tracking and classification based on the baseline? 
2. There is a lack of visualization and analysis for the prompt guided attention, which does not adequately demonstrate its direct impact.

questions:
1. Considering CLIP's training process, is there a significant distinction in the representations output by CLIP when tracking-state-aware prompts, such as 'unoccluded' and 'occluded', are provided?
2. Why does your method improve the ClA for novel categories compared to the baseline OVTrack but decrease scores for base categories? Does this suggest that your model introduces a conflict between tracking and classification? 
3. Could you include a visual representation and analysis of the prompt guided attention? This would more intuitively demonstrate its role and effect.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733123420136

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of rebuttal

comment:
Dear all reviewers and ACs,

With the approximate ending of the rebuttal period, we would like to make a summary of this work.

This work received the initial ratings of 5 (6vnS), 6 (aVgC), 5 (a19J), 5 (n3Po), 5 (h6oh). After the rebuttal, Reviewers a19J and h6oh have replied and both raised the rating to 6, making the current rating as 5 (6vnS), 6 (aVgC), 6 (a19J), 5 (n3Po), 6 (h6oh).

For Reviewer 6vnS, we have thoroughly addressed all the comments in the rebuttal. In particular, including the explanations for the lower base ClsA (also asked by reviewers a19J and h6oh and have been addressed), the positive interplay between classification and tracking (verified by the new experiments), the effectiveness of CLIP in tracking state awareness (explained and supported by public works), and improved visualizations of prompt-guided attention (in the revised paper).

For Reviewer n3Po, the comments have also been thoroughly addressed. 1) About the novelty. For the proposed tracking-specific state prompt, most other reviewers have acknowledged its novelty, including ‘Novel tracking-related prompt-guided attention’ (Reviewer aVgC), ‘Novel integration of object states with prompt learning’ (Reviewer a19J) and ‘Novel tracking-related prompt-guided attention' (Reviewer h6oh). For the self-supervised learning strategy, we have clarified that it is a fully self-supervised method totally different from QDTrack (actually a supervised one) pointed out by Reviewer n3Po. Its novelty has also been approved by Reviewer h6oh after rebuttal. 2) For the writing, this paper received a commendation from Reviewer 6vnS ('Detailed explanation of proposed method'), and Reviewer aVgC ('Standardized manuscript with fluent writing'). We also promise to further improve it as per Reviewer n3Po's suggestions. 3) For the experiments, we have also added the results and analysis in the detailed response, which will be included in our camera-ready version.

At last, please allow us to highlight again the novelty and contribution of this work. First, based on the recently popular prompt learning, this work estimates the **object states specifically for tracking**, such as occlusion/out-of-view/blurriness, etc, through prompt learning, which is **non-trivial and highly helpful for the open-vocabulary (OV) tracking, especially the (more important) novel class** (see the response to Reviewer h6oh [4]). Moreover, this is the first work to apply the self-supervision for OV tracking task, which **makes the model training on videos possible**. This is **urgently needed for the OV tracking** task since there are still no available (labeled) training videos. Benefitting from these contributions, the proposed method outperforms the homochronous works published in 2024 (see the response to Reviewer a19J [1]).

We sincerely hope the reviewers and ACs can take the above points into consideration in the final decision.

Many thanks.

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary:
This paper proposes OVTracker that integrates object states relevant to MOT and video-centric training for the open vocabulary tracking task. A prompt-guided attention mechanism is developed for more accurate localization and classification and a self-supervised object similarity learning technique is formulated for enhancing object association. The experimental results on the OVMOT dataset are given.

The main strengths are 1）strong performance, and 2) well-structured and easy to follow.  
The main weaknesses are 1) limited novelty (the proposed method is an incremental work of VTrack) and 2) More experiments and visualization are needed.  

After the discussion phase, the main issue of lack of novelty still remains, as recognized by Reviewers 6vnS and n3Po. As these issues significantly degrade the contribution of this paper, the AC does not recommend it be accepted at this conference.
The authors are encouraged to take the comments into consideration for their future submissions.