OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer

Authors: Jinyang Li, En Yu, Sijia Chen, Wenbing Tao
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Open-Vocabulary, Multiple-Object Tracking, Multimodal, Transformer, End-to-End

Abstract:
Open-vocabulary multiple object tracking aims to generalize trackers to unseen categories during training, enabling their application across a variety of real-world scenarios. However, the existing open-vocabulary tracker is constrained by its framework structure, isolated frame-level perception, and insufficient modal interactions, which hinder its performance in open-vocabulary classification and tracking. In this paper, we propose OVTR (End-to-End Open-Vocabulary Multiple Object Tracking with TRansformer), the first end-to-end open-vocabulary tracker that models motion, appearance, and category simultaneously. To achieve stable classification and continuous tracking, we design the CIP (Category Information Propagation) strategy, which establishes multiple high-level category information priors for subsequent frames. Additionally, we introduce a dual-branch structure for generalization capability and deep multimodal interaction, and incorporate protective strategies in the decoder to enhance performance. Experimental results show that our method surpasses previous trackers on the open-vocabulary MOT benchmark while also achieving faster inference speeds and significantly reducing preprocessing requirements. Moreover, the experiment transferring the model to another dataset demonstrates its strong adaptability.

Decision: accept (poster)
Reviewer scores: 8, 6, 3, 6

Reviews and discussion:
<thread>
creation date:
1730533824217

thread_type:
official_review

<thread_object>
creator: reviewer_xftv

<content>
summary:
This paper introduces an End-to-End Open-Vocabulary Multiple Object Tracking with Transformer, OVTR, a novel approach to multi-object tracking (MOT) that extends to handle unseen categories without prior training on those specific categories. The experimental results demonstrate that OVTR outperforms existing state-of-the-art methods in tracking novel categories on standard benchmarks like TAO and KITTI, showcasing its effectiveness and adaptability.

strengths:
1. End-to-End Architecture: OVTR is the first to propose an end-to-end trainable model for open-vocabulary MOT, which integrates motion, appearance, and category information simultaneously.
2. The introduction of the CIP strategy and the dual-branch structure represent creative combinations of existing ideas in a novel application domain.
3. The technical depth and rigorous experimentation by comparing OVTR with state-of-the-art methods provide a strong empirical basis for the claims made.

weaknesses:
1. For the experimental results, although OVTR proposed for handling open-vocabulary multi-object tracking tasks demonstrates innovation and advancement, the performance did not meet the high expectations, particularly in terms of localization accuracy.
2. The authors should introduce more implementation details and release the source code for reproducibility. 
3. The introduction of a dual-branch structure and multiple attention mechanisms increases the model complexity significantly. This might limit the practical deployment on resource-constrained platforms.

questions:
1. The end-to-end MOT framework has been explored in MOT task, so what's the main difference with previous methods (such as TransTrack, TrackFormer, MOTR ...)
2. Could the authors provide visual evidence to support the effectiveness of OVTR?  Specifically, it would be beneficial to show comparative visualizations of the tracking process, including scenarios where the model successfully handles rapid motion changes, partial occlusions, and diverse object appearances across different environments.
2. What about the computational efficiency and real-time applicability of the OVTR model? Discussing its performance in terms of tracking speed and resource usage would greatly enhance the understanding of its practical value.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730619993957

thread_type:
official_review

<thread_object>
creator: reviewer_3y3k

<content>
summary:
The authors present the first application of the Transformer architecture in Open-Vocabulary Multiple Object Tracking. This work simultaneously considers the motion, appearance, and category features of the targets. A novel approach for constructing inter-frame category information flow is proposed, along with a Dual-Branch structure. As a result, an improvement of approximately one point is achieved on the Novel TETA metric of the TAO dataset.

strengths:
The advantages of this work include the following:

1. It effectively leverages the strengths of the Transformer architecture and strives to apply it to the Open-Vocabulary Multiple Object Tracking (OVMOT) task.
2. The motivation throughout the paper is well-founded, as it integrates motion, appearance, and category features, utilizing category information flow to guide subsequent tracking.
3. The use of a dual-branch structure addresses the issue of increased training data arising from proposals with unknown categories during training. Instead, it employs ground truth to construct supervision, thereby reducing burdensome in training process.

weaknesses:
Firstly, possibly due to time constraints, I feel that **this paper is not complete**. I read the article twice carefully. During the first read, I thought I had missed many details. However, upon a second, more thorough review, I found that the paper contains **only fragmented descriptions of the training process** and **lacks any explanation of how inference is conducted**.
I believe the potential issues are as follows:

### 1. Unclear Architecture Representation
Figure 2 lacks clarity in its representation. Several concerns need to be addressed:
- The distinction between Position Part and Content Part is not well-defined
- In Figure 3, it's unclear why the Position Part from the Encoder is directly fed as input - is this representing the feature blocks from Image Features or fused features?
- The nature and purpose of the Content Part input need clarification
- There's an unexplained discrepancy between Figure 2 (5 outputs) and Figure 3 (3 outputs)

### 2. Attention Mechanism Implementation
Section 3.3 presents several implementation issues:
- The matrix $I$ is described as a filter for large detection queries in self-attention, but the implementation details are unclear
- The relationship between $O_{txt}$ (final output) and its application in previous self-attention layers needs elaboration
- The specific self-attention layer where the filtering is applied is not specified
- The Content Isolation mechanism needs clarification, particularly regarding:
  - Its operation between detection and track queries
  - The implementation details
  - Whether it uses category features for query filtering

### 3. Dual-Branch Decoder Components
In Section 3.4:
- The origins of matrices $P$ and $O$ are not shown in the Dual-Branch Decoder diagram
- The purpose and utilization of $C^{(t+1)}$ obtained through propagation are not explained in the paper

### 4. Sequential Loss Computation
Section 3.5 presents logical inconsistencies:
- The sudden introduction of $P$ in the Sequential loss computation lacks context
- The relationship between this $P$ and the position part query from Section 3.3 needs clarification
- The connection between $L$ in Equation 6 and $L$ in Equation 5 is ambiguous

### 5. Inference Process
The paper lacks a comprehensive explanation of:
- The complete inference pipeline
- The feature fusion methodology for matching
- The integration of multiple features during inference

### 6. Claims of Mutual Benefits
The authors claim that "Our approach jointly models localization, appearance, and classification, enabling tracking and classification to mutually benefit from each other." However, this mutual benefit is not convincingly demonstrated:
- On the Validation Set:
  - Only ClsA shows improvement for Novel categories
  - Other metrics merely maintain similar performance levels
  - LocA actually decreases by one point
- On the Test Set:
  - Novel category classification improvement is negligible
  - Shows significant asymmetry compared to Validation Set results
  - The claimed mutual benefits are neither stable nor consistently demonstrated

### 7. Weak Experimental Results
The experimental results raise several concerns:
- Minimal improvement of only ~1 point in TETA for Novel categories
- Base TETA results are lower than OVTrack by one point on Validation Set
- The authors' claim of using only Base proposals is questionable given:
  - No improvement in Base category performance
  - Previous methods using unlabeled proposals achieve superior results without requiring additional annotations
  - The only apparent advantage seems to be computational efficiency

### 8. Methodology Clarification Needed
- The specific methodology for generating image sequences through data augmentation requires explanation

### 9. Inconsistency in Training Description
- Section 4.3 states "All methods are trained are trained solely on novel categories"
- This appears to be a typo and should likely read "solely on base categories"
- This inconsistency needs clarification

### 10. Ablation Study Concerns
The ablation experiments only using 10,000 images show:
- Generally low performance metrics compared with the results in Table 1
- Particularly low ClsA for base categories
- Limited discriminative power across different ablation settings

### 11. Visualization Issues
The supplementary materials' visualizations are problematic:
- The pole case example lacks representativeness
- The comparison with OVTrack is unclear (the right-side detection in the first row appears superior)
- The elephant case visualization is ambiguous
- No guidance is provided to help readers interpret the differences
- The dense bounding box visualizations make quality comparison difficult

questions:
I have included all the questions in the 'Weaknesses section', outlining approximately twelve areas of questions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730713387953

thread_type:
official_review

<thread_object>
creator: reviewer_tywh

<content>
summary:
In this paper, the authors propose OVTR (End-to-End Open-Vocabulary Multiple Object Tracking with TRansformer), the first
end-to-end open-vocabulary tracker.

Specifically, to achieve stable classification and continuous tracking, they designed the CIP (Category Information Propagation) strategy, which establishes multiple high-level category information priors for subsequent frames. 

They also introduce a dual-branch structure for generalization capability and deep multimodal interaction and incorporate protective strategies in the decoder to enhance performance.

strengths:
+ The motivation for this work, the flow of category information, is reasonable.

+ This is the first end-to-end framework for open-vocabulary tracking.

+ Experimental results have verified the effectiveness of the proposed method, to some extent.

weaknesses:
- The writing of the method part is not easy to follow.

- From the results, we can see that the performance of the proposed method is not very excellent, compared to the basic baseline method OVTrack, not to mention the most recent methods.

- The authors are suggested to present the advantages of the end-to-end Transformer methods compared to the tracking-by-OVD methods, based on the experimental, including the performance, speed, model size, etc.

- The authors claim that due to resource constraints, the ablation studies are conducted on a subset of 10,000 images. This makes the evaluation not very convincing. The overall framework can be implemented on the whole dataset. The ablation study without some components can be also conducted.

- The object category in the KITTI dataset is not various, which is not very appropriate for the OVTrack task. The comparison of OVTrack and the proposed method on KITTI using Car Category is not very convective, for the OVTrack problem.

questions:
See the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730983996284

thread_type:
official_review

<thread_object>
creator: reviewer_dqgj

<content>
summary:
This paper proposes the first solution for end-to-end open-vocabulary multiple object tracking called OVTR. The model builds on close-set end-to-end multiple object tracker MOTR while extending it to open-vocabulary by adding the category Information Propagation strategy and dual-branch vision-language alignment training. The method achieves good results on open-vocabulary tracking benchmarks and also demonstrates good zero-shot transfer ability to unseen domains.

strengths:
1. The paper presents the first end-to-end Transformer solution for open-vocabulary tracking that performs joint detection and tracking. This is known to be a hard problem for two main reasons:
   - There is a lack of sufficiently diverse video data to train end-to-end models effectively. Training on a small set of videos for tracking often hurts detection performance. The paper addresses this by augmenting static images for training, demonstrating that this approach can still yield good performance in an end-to-end setup. Although it is not new to use static images for tracking it is rare in end-to-end solutions, especially in open-vocabulary scenarios. 
   - In end-to-end multi-object tracking solutions, queries often perform multiple tasks—tracking, localization, and classification—which is really difficult to balance in open-vocabulary scenarios. The paper’s proposed category and content isolation strategies help balance these tasks, which is a sensible approach.

2. The zero-shot transfer experiments on the KITTI dataset indicate strong performance in association, particularly in terms of the IDF1 metric, showing significant improvement over previous methods like OVTrack.

weaknesses:
1. The paper is not clearly written and lacks important details. For instance, understanding the overall framework and its training process is challenging. The paper mentions using only static images to train the framework; however, using static images implies there is no separation of new and old objects, which are typically handled differently in MOTR training. This critical aspect is not clearly explained in the paper.

2. The ablation results in Table 3 and Table 4 are confusing and show only marginal improvements over the baseline, especially in association. For example, in Table 4, adding category isolation only improves ClsA_b by 0.6 and ClsA_n by 0.1, while adding content isolation alone only improves 0.1 in AssocA performance. A more detailed explanation is needed here to clarify these results.​

3. Missing latest related works. The paper only compares OVTrack. However, In open-vocabulary tracking, there are some new advancements eg. MASA[1] at CVPR2024. Also, SLAck at ECCV2024, which is also a transformer-based end-to-end approach. Maybe SLAck can be a concurrent work but MASA seems to be fair to discuss.

[1] Li, Siyuan, et al. "Matching Anything by Segmenting Anything." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

questions:
Will the code be public? Since this is the first work that performs end-to-end open-vocabulary object tracking, making the implementation details public is of great importance to the research community.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732696169829

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response

comment:
**Common Response**

First of all, we thank all Reviewers for their insightful and valuable comments. We are encouraged that the Reviewers all give the positive feedback to our work including the **first End-to-End Architecture** (Reviewer dQGj, tyWH, xFtV), **well-founded motivation** (Reviewer tyWH
, 3Y3k, xFtV), and **competitive OV-tracking performance** (Reviewer dQGj, tyWH, xFtV).

And then, we sincerely thank all Reviewers for their valuable suggestions and questions. Before formally addressing each reviewer’s comments individually, we would like to first make the following clarifications and statements.

- **OVTR is a novel paradigm of open-vocabulary MOT that achieves OV tracking in a fully end-to-end manner.** The motivation behind this work is to establish a novel tracking paradigm distinct from the mainstream OVD-based tracking architectures. This stems from the fact that, while existing OVD-based architectures demonstrate decent tracking performance, they also have several inherent limitations. These include the need for an additional OV-Detector, complex post-tracking processing, and a lack of scalability to larger datasets, which collectively hinder further improvements and real-world applications.

  In contrast, our proposed OVTR exhibits several advantageous characteristics: a single model, fully end-to-end processing, no post-processing, and a data-friendly nature that facilitates scalability. Thus, although our current approach does not significantly outperform existing OVD-based solutions at this stage, it has already demonstrated immense potential and offers the OV-MOT community a novel research direction to explore.
- **Updates to the Manuscript.** Taking into account the reviewers’ comments and suggestions for this work, we have made the following updates to the manuscript.

  1. Add more details in the Overview of the methods.
  2. Refine and correct some figures, statements and equations of methods (Sec 3.2, 3.3, 3.4, and 3.5).
  3. Update experiment parts, including more experimental setting details, SOTA performance comparisons, and ablation studies.
  4. Provide additional ablation and speed test experiments (Appendix B, beginning from line 732).
  5. Provide more details about data augumentation (Appendix C, beginning from line 805).
  6. Provide model and training hyperparameters (Appendix D, beginning from line 896).
  7. Provide more visualizations analysis (Appendix E, beginning from line 912).
  8. Provide additional preliminaries of OVTR (Sec 3.1, beginning from line 158).

  All changes in the manuscript are highlighted in red.

Finally, we have provided responses to each comment and question from the each reviewer, hoping to address your concerns adequately.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733313221316

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Discussion

comment:
# Summary of the Discussion

We would like to express our sincere gratitude to all four reviewers for their thorough review and valuable comments.

Our method has been consistently recognized by all reviewers in the Official Review for its **novelty** or **well-founded motivation**. During the discussions with the reviewers, our responses were well-received by dQGj, tyWH, and xFtV. Consequently, **our ratings improved** to **8** (dQGj), **6** (tyWH), **6** (xFtV), and **3** (3Y3k), compared to the initial scores of **8** (dQGj), **5** (tyWH), **5** (xFtV), and **3** (3Y3k).

We have carefully addressed all the reviewers' questions with detailed explanations and revisions, which have **been acknowledged and accepted by reviewers dQGj, tyWH, and xFtV**, all of whom increased their ratings during the rebuttal stage. Especially for the issues in the method section that were partially difficult to understand, we have provided detailed explanations and corresponding revisions.

Regarding Reviewer 3Y3k, **we have thoroughly, comprehensively, and meticulously addressed all 12 (11+1) questions** raised by Reviewer 3Y3k, covering both the detailed aspects of our method and the experimental results and analyses, **as evidenced by our detailed 9-page response**. Moreover, many shared **critical and core explanations and revisions** provided in our rebuttal **have been acknowledged and accepted by the other three reviewers**. And many of these questions were related to the underlying prior knowledge of our approach, which we clarified by adding OVTR-related background in the overview and providing comprehensive responses.

Please allow us to once again summarize the advantages of our method as follows:

We propose **the first end-to-end open-vocabulary multi-object tracking (OVMOT) algorithm**, introducing **a fresh perspective** on end-to-end processing and modality interaction fusion in the OVMOT field. This approach **offers** the OVMOT community **a new research direction** to explore. We believe that **many new methods and ideas could enhance our work**, offering **substantial potential for improvement**. Furthermore, we are confident that there is **considerable room for performance enhancement**, as demonstrated by the improvements brought by our adjusted training strategy.

Specifically, we propose the CIP strategy, which enables category information to propagate across multiple frames, enhancing both tracking and classification stability. We also introduced a dual-branch decoder to improve open-vocabulary performance, **eliminating the need for generating hundreds of gigabytes of embeddings** containing category information. Additionally, the inclusion of a decoder protection strategy further boosts performance. Due to the end-to-end nature of our approach, we **eliminate the need for complex post-processing and explicit similarity associations** between frames, leading to **faster inference speeds** compared to OVTrack. Furthermore, our method does not require an additional pre-existing OV-detector. It also reduces dataset requirements and, as a transformer-based tracker with a data-friendly nature, facilitates scalability while **leaving room for further improvement**.


Finally, we would like to clarify that, in line with ICLR's policy on paper revisions, we have highlighted changes in red in the rebuttal revision. The main modifications include **adding prior knowledge** in the overview, **which may give the impression of substantial changes**. Other adjustments were made based on reviewer suggestions, and the experimental results were updated by increasing the training epochs. These revisions have been acknowledged and received by all responding reviewers, including dQGj, tyWH, and xFtV.

We genuinely hope that the reviewers and ACs will take the above points into account in the further discussion and decision-making process.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper received comments from 4 reviewers and three of them have a positive comment on this paper, but the rest (i.e., reviewer 3Y3k) have many concerns. The authors have provided a comprehensive response to the reviewer, seemingly addressing all the issues raised. The reviewer has not engaged in further in-depth communication or discussion; therefore, based on the positive feedback from all reviewers, the paper appears to meet the standards for publication.