OVTR: End-to-End Open-Vocabulary Multiple Object Tracking with Transformer
Authors: Jinyang Li
En Yu
Sijia Chen
Wenbing Tao
Venue: ICLR 2025 Poster (2025)
Keywords: Open-Vocabulary
Multiple-Object Tracking
Multimodal
Transformer
End-to-End
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Open-vocabulary multiple object tracking aims to generalize trackers to unseen categories during training, enabling their application across a variety of real-world scenarios. However, the existing open-vocabulary tracker is constrained by its framework structure, isolated frame-level perception, and insufficient modal interactions, which hinder its performance in open-vocabulary classification and tracking. In this paper, we propose OVTR (End-to-End Open-Vocabulary Multiple Object Tracking with TRansformer), the first end-to-end open-vocabulary tracker that models motion, appearance, and category simultaneously. To achieve stable classification and continuous tracking, we design the CIP (Category Information Propagation) strategy, which establishes multiple high-level category information priors for subsequent frames. Additionally, we introduce a dual-branch structure for generalization capability and deep multimodal interaction, and incorporate protective strategies in the decoder to enhance performance. Experimental results show that our method surpasses previous trackers on the open-vocabulary MOT benchmark while also achieving faster inference speeds and significantly reducing preprocessing requirements. Moreover, the experiment transferring the model to another dataset demonstrates its strong adaptability.

Published as a conference paper at ICLR 2025
OVTR: END-TO-END OPEN- VOCABULARY MULTIPLE OВJЕCT TRACKING WITH TRANSFORMER
Jinyang Li En Yu? Sijia Chen Wenbing Tao"
Huazhong University of Science and Technology (jinyangli, yuen, si sijacchen, weiiaingtaaoJeuust.eeb..o
ABSTRACT
Open-vocabulary multiple object tracking aims to generalize trackers unseen categories during training, enabling their application across a variety real-world scenarios. However, the existing open-vocabulan tracker is constrained by its framework structure, isolated frame-level perception, and insufficient modal inter- actions, which hinder its performance in open-vocabulary classification and track- ing. In this paper, we propose OVTR (End-to-Hae Open- Vocabulary Multiple Object Tracking with TRansformer), the first end-to-end open-vocabulary tracker that models motion, appearance, and category simultancously. To achieve stable classification and continuous tracking, we design the CIP (Category Information Propagation) strategy, which establishes multiple high-level category information priors for subsequent frames. Additionally, we introduce a dual-branch struc- ture for generalization capability and deep multimodal interaction, and incorpo- rate protective strategies the decoder to enhance performance. Experimental results show that our method surpasses previous trackers on the open-vocabulary MOT benchmark while also achieving faster inference speeds and significantly reducing preprocessing requirements. Moreover, the experiment transferring the model to another dataset demonstrates its strong adaptability. Models and code are released at http:::///ttttbb com/ comm//nnynglli//O//
INTRODUCTION
Critical video perception, multiple object tracking (MOT) can currently be applied various downstream tasks such as autonomous driving and video analysis (Bashar et al. 2022). Dominant MOT methods are primarily trained track closed-vocabulary categories, limiting their ability to generalize to unseen categories and broader range of scenarios. Clearly, such approaches do not offer the ultimate solution for human-like video perception intelligence, as humans can perceive and track unseen dynamic objects in open-world context. To address this, the open-vocabulary multiple object tracking (OVMOT) task et. 2023) was proposed, where models are expected to identify and track novel categories in a zero-shot manner, aligning better with weal-world demands, such as more comprehensive video understanding, smart cities, and autonomous driving.
Recently, numerous open-vocabulary detection (OVD) (Gu et al.) 2021 Du et al., 2022 Lin al. 2022 Wu et al., 2023b) Zang et al. 2022) methods have emerged, researchers (Li et al. 2023) ) have extended OVD into the tracking domain by integrating open- -ocabulayy detectors with appearance- based associations, as illustrated in Fig. [ This approach utilizes OVD and data augmentation to improve appearance-based association learning. However, it encounters three problems: (1) Clas- sification and tracking are merely cobbled together, rather than collaborating effectively. Treat- ing classification independently each frame causes instability in category perception and hin- ders the reuse of previous predictions in subsequent frames. (2) From the framework perspective, appearance-based association struggles adapt to the diverse environments typical of OVMOT task. Moreover, the tracking-by-OW framework inevitably relies on complex post-pree post-processing and anchor generation, which reduce inference speed and necessitate hand- -dsignedd operations based on scene-specific prior knowledge, making it difficult to adapt in an open-world context. (3) Achieving
Equal Contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
text embeddings
queries
object representations
category information propegation
T
Tz
T
"bowl"
"Parus"
"bow!
"bow!!
(a) Tracking- by-OVD
T
T
T
"Paras"
"bowl"
(b) Tracking- by-Query (Ours)
Figure 1: Comparison of tracking-by-OVD and our method. Bracking-by-OVi predicts each frame independently, making classification and association susceptible to changes appearance. In contrast, our method, OVTR, propagates location, appearance, and category information from the current frame to subsequent frames, creating a stable, continuously updated information flow. This flow serves as a prior, aiding in capturing the corresponding target in future frames.
open-vocabulary capabilities requires significant time to use a pre-trained image encoder to extract numerous object embeddings that implicitly contain unseen categories from datasets, which pro- longs the model development cycle. Despite this, the performance of open-vocabulary classification remains insufficient.
Regarding problem (I), an iterative tracking mechanism (Zeng et al. 2022) can be leveraged to propagate category information subsequent frames. For problem (2). existing closed-set, end-to- end transformer based tracking methods (Zeng et al. [202 Zhang et al. 2023 Gao & Wang 2023
Yu al. 2023b) focus on achieving sustained tracking
complex scenarios. With their elegant framework, these methods eliminate the need for complex post-processing and explicit tracking as- sociations, showing potential for cross-temporal modeling of targets in specific scenarios. Adopting such structure effectively mitigates these challenges. Taken together, an iterative, transformer- based, end-to-end tracker such as MOTR (Zeng et al. 2022) can serve as a suitable foundation.
Based on the iterative tracking mechanism, frames, which we refer to as the category information propagation (CIP) strategy. By convert- ing predicted category information into priors for subsequent tracking, it establishes higher- flow that enables the continuous tracking corresponding targets. Additionally, due to the intro- duction of attention mechanism, potential interference among differing information must also be considered. To address this, we designed two attention protection strategies. These strategies con- struet masks based on category prediction distributions and the arrangement of queries, effectively intervening to ensure harmonious cooperation between classification and tracking.
propagate category information across multiple
To address problem (3), design our method from the perspective of knowledge distillation and deep modality interaction. Considering that knowledge distillation between two models processing the same modality may be more effective, we use the CLIP (Radford al., [02T] image encoder, source of vocabbocabul,ry to guide our model in acquiring open-vocabulary generalization capabil- ities in image perception, aligning intermediate outputs (termed aligned queries) with CLIP image embeddings. Meanwhile, given the shortcomings in prior work regarding multimodal fusion, we de- sign attention-base fusion in both the encoder and decoder. Particularly, the decoder, we propose dual-branch structure. One branch serves the medium through which the CLIP image encoder guides our model, while the other branch allows the aligned queries to interact with text embeddings from the CLIP text encoder to focus category information, yielding features rich in category in- formation for apen-vocabulary classification. Thus, through the dual-branch decoder design, we avoid cumbersome preprocessing while achieving improved open-vocabulyy performance.
2

---

Published as a conference paper at ICLR 2025
Incorporating all the aforementioned designs, propose OVTR (End-to-End Open- Vocabulary Multiple Object Tracking with TRansformer), the first end-to-end vocabbuayy tracker that in- tegrates motion, appearance, and category modeling. Experimental results the TAO (Dave al. 2020) dataset demonstrate that OVTR outperforms state-of-the-art methods on the TETA (Li et al,, the novel TETA, while on the test set, it surpasses OVTrack by 12.4%. Additionally, in the KITTI (Geiger al. 2012) transfer experiment, OVTR outperforms OVTrack by 2.9% on the MOTA metric.
2022b) metric. Notably, on the validation set, OVTR exceeds OVTrack by 12.9%
To summarize, our main contributions are listed as below:
We propose the first end-to-end open-vocabulary multi-object tracking algorithm, intro- ducing novel perspective to the OVMOT field, achieving faster inference speeds, and possessing strong scalability with potential for further improvement.
We propose the category information propagation (CIP) strategy to enhance the stability of tracking and classification, along with the attention isolation strategies that ensure open- vocabulary perception and tracking operate in harmony.
We propose dual-branch decoder guided by an alignment mechanism, empowering the model with strong open-vocabulary perception and multimodal interaction capabiliiies while eliminating the need for time-consuming preprocessing.
2 RELATED WORK
Open-Vone Vocabulary MOT. Multi-object tracking (MOT) is the task of identifying and following mul- tiple objects across consecutive frames in video. Many trackers exhibit powerful capabilities and performance (Wojke et al. 2017 Zhang et al. 2022b at. 2022 Chen et al. 2024a b). and MOT has been scaled up landle broader range of categories, enabling it operate in more diverse introduced the TAO benchmark, which includes more than 800 categories and emphasizes MOT performance the long-tail distribution of large number of cate- gories. TETA ([i et al. 2022h) decomposes tracking evaluation into three sub-factors: localization, association, and classification, allowing for comprehensive benchmarking of tracking performance even under inaccurate classification, with TETer (Li et al. (022b) achieving strong results on this evaluation. OVTrack (Li at.
environments. Dave et al. (2020)
2023) leverages the open- eecen-ocabulary detector ViLD (Gu al., 2021]
combining appearance-basee association for ooca-vocayulary tracking, and uses diffusion models to generate LVIS (Gupta et al.) 2019) image pairs for training associations. MASA (Li et al. 2024)
learns general appearance matching model using a large number of unlabeled images. SLAck LLi
ct al. 2025) integrates semantic and location information to enhance tracking performance.
Transformer-based MOT. Association-basee trackers have not yet fully realized end-to-end track- ing, as they still depend on post-processing and explicit matching strategies. In contrast, the Trans- former model holds advantages a framework for implementing end-to-end MOT. TransTrack (Sun)
jt al.] 2020) adopts a decoupled network and IoU matching to achieve query-based detection and tracking. TrackFormer (Meinhardt et al.) (022) and MOTR (Zeng et al.] 2022) reformu- late tracking as a sequence prediction task, where each trajectory is represented by a track query. MOTRv2 (Zhang et al. 2023) incorporates an additional object detector to generate proposals serv- ing as anchors, providing detections for MOTR. MOTRv3 (Yu et al., (002b) refines the label as- signment process by employing a release-fetch supervision method. Additionally, the attention mechanism of the Transformer has already been proven effective in handling multi-modal infor-
mation (Nguyen al. 2024 Zhou et al., 2023
Yu al. 2023a Wu et 2023a).
METHOD
3.1 OVERVIEW
Revisiting MOTR. The overall pipeline is illustrated in Fig. 2 We follow the general structure and basic tracking mechanism of MOTR (Zeng et al. 2022). treating MOT as an iterative sequence pre- diction problem. In MOTR, each trajectory is represented by track query. Following DETR- -ikee structure (Carion et al. 2020), detect queries Q for the first frame fr are fed into the Trans- former decoder, where they interact with the image features F extracted by the Transformer
3

---

Published as a conference paper at ICLR 2025
Erg
Aigh
Backboou
P
OFA
gg
Dual-Branch Decoder
Encoder
tintelling
CTT
Oat
Updater
P
L
[larthettl (penor). fark T99t Jerbesly
Shandd Phorept Kgpoasteiiiea
proition part
leserablle contien part
/ content part (uaak))
deteect quarry
query
1 Q,1
inagge Statures Eest TeSCureS
Seature for aligument
prodiction box
OFA beanch output
P pupatodd position part OC beanch output
Figure 2: Overview of OVTR. OVTR processes two modalities of input, with modality interaction structures in both the encoder and decoder. The dual-branch decoder has the OFA branch, which serves as the medium through which the CLIP image encoder guides our model to achieve visual generalization capabilities, and the CTI branch, which handles open-vocabulary interaction and clas- sification. The updated outputs are used as prior queries for the next frame's 's predictions.
encoder. This process yields updated detect queries 1 that contain object information. Detec- tion predictions, including bounding boxes _ and object representations O are subsequently extracted from Q In contrast to DETR, for the query-based iterative tracker, Q are only needed to detect newly appeared objects in the current frame. Consequently, one-to-one assign- ment is performed through bipartite matching exclusively between Q and the ground truth of the newly appeared objects, rather than matching with the ground truth of all objects.
The matched Q will be used to update and generate the track queries Q which, for the second frame fr=2, are fed once again into the Transformer decoder and interact with the image features 2 to extract the representations and locations the objects targeted by Or thereby enabling tracking predictions. Subsequently, the Qt=² maintain their object associations and are updated to generate the Q1=3 for the third frame fr-3. Parallel to Qr,,, and similar to the process for fr-1, Q are fed into the decoder to detect newly appeared objects. After binary matching, the matched Q are transformed into new track queries and added to Q!=3 for f1=3- The entire tracking process can be extended to subsequent frames. Regarding optimization, MOTR (Zeng al., 2022) employs multi-frame optimization, where the loss is computed by considering both ground truth and matching results. The matching results for each frame include both the maintained track associations and the binary matching results between Q' and newly appeared objects.
Tracking Mechanism During Inference. Similar to MOTR, the network forward process during inference in OVTR follows the same procedure as during training. The key difference lies in the conversion of track queries. In detection predictions, if the category confidence score exceeds Taet, the corresponding updated detect query is transformed into new track query, initiating new track. Conversely, if tracked object is lost in the current frame (confidence < Tir), marked as an inactive track. If an inactive track is lost for Tmiss consecutive frames, it is completely removed.
Empowering Open Vocabulary Tracking. Leveraging the iterative nature of the query-based framework, OVTR transfers information about tracked objects across frames, aggregating category information throughout continuous image sequences to achieve robust classification performance.
In the encoder, preliminary image features from the backbone and text embeddings from the CLIP model (Radford et al. (02T)) are processed through fre-fusion to generate fused image features Fang and text features Eixi- We propose dual-branch decoder comprising the OFA branch and the CTT branch. Upon input of Q = [Qaer; Qt.], the two branches respectively enable the model to gain open- vocabulary generalization capabilities in image perception and facilitate deep modality interaction with Ext. leading to the outputs Oung- O Oimg serve as the input for the category information propagation (CIP) strategy, injecting category information into the category information flow. This process is an extension of the aforementioned mechanism where Q' generates Q1++1 Meanwhile, Out are utilized for computing category logits and for contrastive learning.
4

---

Published as a conference paper at ICLR 2025
Encoder
Dual-Branch Decoder
Reference Painsa Genernoor
Self- Attention
OFA branch
Box Head
B
Aligned Quarioo
Image
p
FFN
lnng
Image Cross AttentAt
Alignment Hend
Attuntion
align
Fusion
Emg
Layer
Text
PPN
-000000
Crass-lllnttio
FFN
ttt
CTI branch
Tee Zebwediage
encoding
Figure 3: Architectures of the dual-branch decoder and the encoder. After modality fusion in the encoder, the resulting image and text features are separately fed into the decoder's Image Cross- Attention and Text Cross-Attention for interactions. Aligned queries processed by the OFA and CTI branches to generate bounding boxes B, alignment features Falign- and branch outputs Oxxt-
3.2 LEVERAGING ALIGNED QUERIES FOR SEARCH IN CROSS-ATTENTE
The perception part of OVTR builds on Zhang et al. (2022a), incorporating visual- visual-language modality fusion in both the encoder and decoder. To efficiently conduct multimodal interaction and learn generalization ability, the decoder adopts a dual-branch structure, consisting of the object feature alignment (OFA) branch and the category text interaction (CTI) branch.
Generating Image and Text Embeddings. To obtain the text modality input, we feed the text and prompts (Du al. 2022) into the CLIP (Radford al.) 2021] text encoder to generate text embeddings. Simultancously. we use ground truth boxes to generate image embeddings via the CLIP image encoder and combine embeddings of the same category into single representation.
Unlike the time- consuming preprocessing (Gu et al. 2021 Du et al. 2022) that generates numerous image embeddings from proposals, which are produced by an additional RPN-based detector and partially contain novel categories (unseen during training), our approach is simpler. We only need image embeddings for the closed-set categories. The embeddings are prepared offline.
Feature Pre-fusion and Enhancement. In the encoder, inspired by multi-modal detectors such as GLIP eLi al. 2022a) and Grounding DINO (Liu et al. (2023), we integrated image-to-text and text- to-image cross-attention modules for feature fusion, which enhance image and text representations, preparing them for interaction in the decoder. Since the encoder outputs preliminary content features that may introduce misguidance for the decoder, follow the approach of DINO-DETR (Zhang et al. 2022a) by generating the content parts our queries through learnable initialization, while the position parts are derived from the reference points produced by Faing the outputs of the encoder, through sin-cos positional encoding.
Dual-Branch Structure. As shown in Fig. in the dual-branch structure, the OFA branch consists of a feedforward network followed by a box head and an alignment head, while the category text interaction (CTI) branch comprises text cross-attention followed by feedforward network.
To achieve zero-shot capabilities, we utilize the OFA branch for alignment, distilling visual general- ization from the CLIP image encoder and guiding the queries produced by the image cross-attention layer, referred to as aligned queries, to align with CLIP image embeddings. Since the CLIP image embeddings are aligned with the CLIP text embeddings and the text features E originate from the CLIP text embeddings, aligned queries that correspond to certain categories are able to align implic- itly with Exs of the same categories. This alignment enables the aligned queries to effectively focus on the category information conveyed in Era during the text cross-attention in the CTI branch. Consequently, when images and text inputs containing novel categories are provided, the generalized network produces aligned queries that implicitly capture the features of novel categories. This en- ables the aligned queries to interact effectively with the E of novel categorics, yielding On rich in novel category information to enhance open- vocabulary classification. The box head is incorporated into the OFA branch to ensure that the features of aligned queries are instance-level.
Specifically, We distill knowledge from the CLIP image encoder by aligning the outputs Fangan Roxd from the alignment head with the ground truth CLIP image embeddings Vy Raxd Faigon
5

---

Published as a conference paper at ICLR 2025
corresponds to the matching results mentioned in Sec. 3.1 Each feature is d-dimensional vector, where n represents the number ground truth objects. The alignment loss Latign is formulated as:
d
Calign d
Fagaaiign - Vi.j.g));
(1)
Additionally, the dual-branch structure also aims to prevent category text information from affecting the localization ability.
3.3 ATTENTION ISOLATION FOR DECODER PROTECTION
For the decoder, interference may arise from both multiple category information and the content of track queries. Specifically, interactions between queries in the self-attention layers can entangle information corresponding to multiple categories, hurting classification performance. Moreover, as a tracking-by-quce
framework, the decoder processes both track and detect queries in parallel. Track queries contain content about tracked objects, leading to content gap between them and the initial detect queries. This gap may cause confficts within the decoder layers due to the interactions in self-attention. To address these, we propose attention isolation strategies for decoder protection.
Category Isolation Strategy. The output features of the CTI branch Otxt undergo dot products with the text features Exa. followed by softmax operation produce the category score matrix S E N where N denotes the number of queries Q. and M represents the number of selected categories. We calculate the KI. Skulback--Leibles divergence (Kullback) 1951) between the cate- gory score distributions (vectors in S) of each predictions in Oux form matrix, called the difference matrix D R N The specific formula is as follows:
Si,k
SS.kk S
Drj- (S. ((.,)) DKL (S,.1l
Σ St,e In
+
Σ k=1
SgN In
(2)
S
where Di.j is the sum of the forward and the reverse KL divergences between the category score vectors in S corresponding respectively to the i-th and j-th queries. DKL. represents the calculation of KI. divergence.
We compute the difference matrix D based on S of the current decoder layer to generate the category isolation mask (I e RR-XNN,, which is then added to the attention weights of self-attention layer of generated as follows:
next decoder layer. The category isolation mask
True, if Ds.j > Taol
False, if Di.j S Tsol
where Tis is the threshold for the difference matrix D.
When J is "True", it means the category information carried by these two queries is substantially different. This difference may cause interference between the queries during self-attention. In addi- tion, interactions among Qir which have inherited robust category information, will be maintained to ensure that the self-attention mechanism inhibits duplicate predictions derived from Qu.
Content Isolation Strategy. To mitigate the impact of the content gap between track and detect queries as they jointly enter the decoder, we introduced content isolation mask. The content distri- bution differs between the first frame detection and subsequent tracking, and in the latter case, track queries may interfere with detect queries. Using a vanilla decoder for both could lead to conflicts. To ensure consistent decoder operations across the two processes, we propose the content isolation mask to prevent track queries from interfering with the content of detect queries. Specifically, this mask added to the attention weights of self-attention in the first decoder layer, with the mask positions for detect queries and track queries that attend each other set to True to suppress their interaction. For further details on these two strategics, please refer to appendix A.4
3.4 TRACKING WITH CATEGORY INFORMATION PROPAGATION ACROSS FRAMES
To enable continuous category perception and localization, leverage the iterative nature of the query-based method and propose the category information propagation (CIP) strategy to aggregate tracked object information, thereby reinforcing category priors throughout multi-frame predictions.

---

Published as a conference paper at ICLR 2025
the modified Transformer decoder layer for the CIP strategy. We use the outputs of the OFA branch corresponding to the matched updated queries Q¹'² = [P C. for the t-th frame fi, denoted as Oing to update Q" into the track queries Q [PL C+ for the frame f1+1. C++1 is the content part of Qu which is the core of propagating category information between frames. is performed in the updater of Fig. 2 and formulated as:
Inspired by MOTR (Zng al., 2022).
O2 = MHA(O + P" O + P.... ing O, LayerNorm(Oing Dropout(Os)),
G++1 FFN(FEN(O, Orng) Ct),
ir
where pu and C respectively represent the updated position parts and the content parts of Q MHA and FFN denote multi-head attention and feedforward network, and Osa and O are the inter- mediate attention outputs and residual outputs. We use the sum of O and ps the queries and keys, while using Oing alone as the values for the MHA.
This network aggregates the category information from O with historical content C₁ providing category priors in C++1 for the next frame predictions. In this way, category information is prop- agated to the next frame, enabling multi-frame propagation during the iterations of track queries. Meanwhile, the bounding boxes B corresponding to Q transformed via sin-cos positional encoding into Ps+1 of - We exclusively use the output representations from the OFA branch, instead of the CTI branch, as the OFA branch contains less direct textual information. This can reduce the content gap between detect and track queries. Additionally, since Oing is derived from the image cross-attention layer, it may more readily align with similar information within that layer when re-entering, thereby facilitating the continuous capture of the target.
3.5 OPTIMIZATION
When an image sequence of N frames input, the multi-frame predictions are denoted as 7 { J and the corresponding ground truth y {yr J.y We compute the loss Eseq across mul- tiple frames, including both tracking loss and detection loss, which share the same components. The difference is that tracking loss focuses localizing previously recognized targets, while detection loss handles newly detected ones. Laey can be formulated as follows:
Laey ΣN (C(Vir le.: vir) + L(Toe IQm. Vda))
(5)
where The IQ.. V Vaet QQu. Viee denote the association predictions, association labels, detection predictions, and unassociated labels, respectively. is the total number of the targets in fe=-
The loss function L includes not only the conventional classification loss and bounding box loss but also the alignment loss Calign for the OFA branch, which was mentioned Sec. 3.2 Regarding the classification loss Lels, we perform dot products between each Out and the text Teatures Exx to predict logits, followed by calculating the focal loss. The single-frame loss L can be formulated as:
Aathhess ALIGLI + >giiouCgiou + >atign Calign
where LL denotes the L1 loss, Lyiu is the generalized IoU loss, and Xets ÀLI Aginus and >atign the weighting parameters. We apply auxiliary losses (Al-Rfou et al. 2019) after each decoder layer.
4 EXPERIMENTS
4.1 DATASETS AND EVALUATION METRICS
Datasets. We conduct comparative experiments on the TAO and KITTI datasets. The TAO dataset is diverse video tracking benchmark with 833 categories spanning various scenarios. We utilized the Open- Vecan-Vocaullary MOT benchmark, based on TAO, to assess opca-voccaulary tracking performance. This benchmark adopts the category division setup from prior work, treating rare classes in LVIS as
7

---

Published as a conference paper at ICLR 2025
Table 1: Open-vocabulary MOT performance comparison on TAO dataset. All methods use ResNet50 (He et al., 2016) as the backbone. G-LVIS: Data generated in OVTrack (Li et al. 2023) for training association, with the same number of images as LVIS. Proposalsawai The use of image embedding generated from proposals that partially contain novel categories for distillation. Embeds: The required number of CLIP image embeddings. t: Distills segmentation knowledge from SAM.
Elements
Novel
Validation
Fumbods Proposalannd TETA?
LocAf
AssocAt
C&AT
TETA!
LocAf
AssocAf C&AA
DepSORT (ViLD
Wojke al. 2017
LVIS
99.1M
21.1
46.4
14.7
2.3
26,9
47.1
15.8
17.7
Tracktor++ (VID)
Bergmann et 2019
LVIS,TAO
99.4M
22.7
46.7
19.3
2.2
28.3
47.4
20.5
17.0
OVTrcck
G-LVIS.LVIS
99.4M
27.8
48.8
33.6
1.5
35.5
49.3
36.9
20.2
MASA
IVIS
99.4M
34.6
369
55.1
19.3
OVIR
LVIS
1,732
54,4
34,5
36.6
52.2
37.6
OVTrack)
(+3.6)
(45.6)
(40.9)
(63.9)
(+1.1)
(+2.9)
(+0.7)
Test set
Data
Embeds Proposals,..
TETA
LocAf AssocAT CKAA
TETAI
LocA] AssocAT CI&AT
DeepSORT (VILD)) Wojke al. 2017
IVIS,TAO
99.4M
17.2
38.4
11.6
1.7
24.5
43.8
14.6
15.2
Tracktores (VIL D1 Bergmann et
2019
LVIS
99.4M
18.0
39.9
13.4
1.7
26.0
44.1
19.0
14.8
OVTrack al 2023
G-LVIS,LVIS
99.1M
28.7
32.6
45.6
16.9
OVTR
LVIS
L,732
27.1
47.1
32,1
2.1
34.5
51,1
37,5
14.9
[vs. OVTrack
(+3.0)
(+5.3)
(+3.4)
(+0.3)
(+1.))
(+5.5)
(+2.1)
novel categories and frequent/common classes as base categories. It evaluates a tracker's ' ability to handle novel categories unseen during training, simulating real-world scenarios of identifying and tracking rare objects. The KITTI dataset, comprising 21 training and 29 test sequences, focuses on autonomous driving scenarios with diverse objects, reflecting realistic driving conditions. serves as a benchmark for evaluating trackers' generalization across datasets. For training, we leveraged the LVIS dataset, which includes 1,203 categories, offering rich diversity of objects for open- vocabulary learning. Under the open-vocabulary setting, these categories are divided into 866 base and 337 novel categories, enabling comprehensive model training for rare object tracking.
Evaluation metries. We use TETA as the metric for evaluating vvecbvocayllary performance. TETA separates classification from localization and association, providing an effective measure of model's classification ability in open-vocabulary setting. The key metrics include localization accuracy (LocA), association accuracy (AssocA), and classification accuracy (CIsA). For general- ization experiments on the KITTI dataset, we use the CLEAR-MOT metrics, such as multiple object tracking accuracy (MOTA), ID F1 score (IDF1), mostly tracked rate (MT), mostly lost rate (ML), and identity switches (IDs). Among these, MOTA and IDF1 serve the primary metrics.
4.2 IMPLEMENTATION DETAILS
We use image data from LVIS (Gupta et al.] 2019) for training and augment it to create image sequences. To better train this query-hased tracker, we go beyond basic strategies like random flipping and cropping, incorporating advanced techniques we designed such as random occlusion and dynamic mosaic augmentation. (see appendix for details)
To accelerate convergence, we build OVTR with a ResNer50 (He et al., 2016) backbone and apply a weight-freezing strategy. Training begins with the detection components, using batch size of 2 for 33 epochs, a learning rate of 4e-5 that decays by a factor of 10 at the 20th epoch. Next, the dual-branch decoders and the updater are trained with batch size of for 16 epochs, starting with a learning rate of 4c-5, which decays at the 13th epoch. Multi-frama training is employed, progressively increasing the number of frames from to 4, and 5 at the 4th, 7th, and 14th epochs, respectively. The hyperparameter Tisol, the threshold for the matrix D. is set to a multiple of its mean value due to its variability. Training is conducted on 4 NVIDIA GeForce RTX 3090 GPUs.
4.3 PERFORMANCE COMPARISON ON TAO DATASET
We compare OVTR with state-of-the-art methods on the TAO validation and test sets. We evaluate OVTrack (Li et al. 2023), along with existing methods like DeepSORT (Wojke et al.] 2017) and
Tracktor++ (Bergmann et al. 2019) using off-the-shelf OVD (results from Li et al. (2023) ). against model. All methods are trained solely on base categories, with ResNet50 (He et al. 2016) the backbone. Notably, all methods except OVTR utilize image embeddings that contain implicit novel categories. Additionally, OVTrack leverages DDPM (Ho al. 2020) for data generation.
8

---

Published as a conference paper at ICLR 2025
Table 2: Zero-shot domain transfer to KITTI dataset. We compare OVTR with OVTrack and Cen- ter Track. Our method and OVTrack are both trained using only images and undergo zero-shot cross-domain transfer evaluation, whereas CenterTrack is trained with in-domain videos.
Method
Data
MOTAT
IDFI
MTT
MLL
IDs)
Frag
Zero-shot:
OVTrack Li at. 2023
G-LVIS.LVIS LVIS
69.8
75.6
62.9
5.8
594
3U7
OVTR
71.8
78.3
378
Supervised:
CenterTrack
Zhou al. 2020
KITTI
88.7
85.5
90.3
2.2
403
68
According to the results in Tab. I OVTR outperforms OVTrack on TETA of both novel and base categories across the validation and test sets. Specificallyy, on novel ClsA, OVTR is more than three times that of OVTrack. On the test set, OVTR outperforms OVTrack by 11.8% on AssocA for novel categories. This demonstrates OVTR has better generalization in novel category classification and tracking. Furthermore, OVTR significantly surpasses OVTrack on LocA across both the validation and test sets. These results, achieved without the use of proposals containing novel category infor- mation and with less data, confirm the effectiveness of our approach. They validate the contributions of the CIP strategy and the dual-branch structure in improving open-vocabulary tracking. Addition- ally. we use MASA (Li et al. (024) results from Li et al. (2025) for comparison and find that our method performs better on novel categories, even without distilling segmentation knowledge from SAM Kirillov et al. 2023).
4.4 ZERO-SHOP DOMAIN TRANSFER TO KITTI DATASET
We evaluated our method and the state-of-the-art OVTrack on the KITTI validation set, as divided by CenterTrack (Zhou al. 2020). in the zero-shot domain transfer scenario. As KITTI only evaluates the Car and Pedestran categories, we set the inference category range to include these two categories along with nine randomly selected additional categories to increase the challenge of category diversity. Tab. 2 shows that OVTR demonstrates better results in the Car category, surpassing OVTrack by 2.9% on MOTA and 3.6% on IDF1, while reducing IDs by 36.3%. This indicates that our model generalizes well when transferred to another dataset, further validating OVTR's adaptability to diverse autonomous driving scenarios. Due to the inconsistency between the categorization of pedestrians in the KITTI dataset and the open-vocabulary task, we present additional comparison methods for the Pedestrian category in the supplementary materials.
4.5 ABLATION STUDY
In this subsection, we verify the effectiveness of the model architecture and strategies through abla- tion studies. All models are trained on the LVIS dataset, undergoing 24 epochs of detection training followed by 15 epochs of tracking training. Due to resource constraints, the ablation studies are conducted subset of 40,000 images. For evaluation, we use the TAO validation dataset and designate certain base categories that were not learned during training as novel categories. We use ClsA, and ClsA, represent ClsA of base and novel categories respectively.
Components of the Dual-Branch Decoder. In this part, we verify the effectiveness of the various components of the dual-branch decoder. We decoupled the OFA branch for analysis, leaving only the box head in the output section of the OFA branch. The CTI branch is essential for open-vocabulary interaction and is not analyzed separately. As reported in Tab. 3 the model with the complete dual-branch structure (row 4) outperforms the model with only the CTI branch (row 1) by 6.4% on TETA, and improves by 20.6% on ClsAg. In detail, each component contributes to the performance improvement. The OFA branch (row 2) improves the model slightly on AssocA, while having an effective improvement of 10.3% on ClsA,- When the alignment head and alignment loss are incorporated with the OFA branch (row 4), the model performance is further improved. These findings highlight the effectiveness the alignment-enabled OFA branch design. In addition, when alignment is used for the CTI branch (row 3), the model achieves significant improvement on AssocA, but the classification performance remains inferior to that of the model with a complete dual- structure (row 4). We analyze that aligning with image embeddings introduces conflicts in the text cross- attention,, which weakens the classification performance. Therefore, is essential to utilize both the dual-branch structure and the alignment mechanism simultaneously.

---

Published as a conference paper at ICLR 2025
Ablation study on decoder compo- nents. Align: the alignment head and alignment loss, CTI: the category-text interaction branch, OFA: the object feature alignment branch.
Table 3:
Table 4: Ablation study on the protection strate- gies for the decoders. Category: the category isolation strategy, Content: the content isolation strategy.
CTI OFA Align
TETA AssocA ClsA, ClsAn
Category Content TETA AssocA ClsAB ClsA
1
V
x
x
31.2
31.8
12.6
1.9
X
x
32.1
32.8
14.6
2.3
2
/
V
x
31.9
32,0
13.9
2.1
2
/
x
32.2
33.0
15.6
2.5
3
/
J
32.5
33.9
13.8
2.0
3
X
v
32.4
33.6
14.3
2.5
33.2
34.5
15.2
2.7
33.2
34.5
15.2
2.7
Table 5: Ablation study on alignment methods. We evaluate three methods: using text embed- dings, image embeddings, and the average of both embeddings for alignment.
Table 6: Ablation study on inputs for CIP Icw.
Osxt denotes output of the category text interac- tion branch, while Oimg represents output of the object feature alignment branch.
Alignment
TETA
AssocA
ClsA,
ClsAA
ICP
TETA
AssocA
ClsAn
ClsAn
Text
31.6
32,6
14.0
2.0
Image
33.2
34.5
15.2
2.7
Ost
32.5
33.8
14.7
1.9
Avg
32.3
33.2
14.1
3.2
Oumg
33.2
34.5
15.2
2.7
Decoder protection Strategies. As shown Tab, 4 when the category isolation strategy was ap- plied alone (row 2), it primarily enhanced classification performance, indicating that it effectively prevents interference between different category information. The content isolation strategy applied alone (row 3) resulted in an improvement on AssocA. When both strategies were applied together (row 4), they produced a synergistic effect, resulting in a 3.4% improvement on TETA and a 5.2% increase AssocA compared to the model without either strategy, although the classification per- formance was slightly suppressed compared to using only the category isolation strategy (row 2).
Modality-Specifft Embeddings for Alignment. To explore whether aligning with image embed- dings yields optimal results, we conducted an ablation comparison with three settings: alignment with text embeddings, alignment with image embeddings, and alignment with the average of both embeddings. As shown in Tab. [5] aligning with image embeddings outperformed alignment with text embeddings by 5. 11 on TETA. It is evident that aligning with the average of text and image embeddings leads improved ClsA, We believe this is because the average embeddings retain the generalization ability provided by image embeddings and enable more direct interactions due to the inclusion of text embeddings. However, using average embeddings leads to suboptimal association performance, indirectly resulting in a lower ClsA,, In conclusion, aligning with image embeddings yields improved results in open- open-vocabulary tracking.
Inputs for Category Information Propagation. In this part, we aim to compare the performance of the model when using the output of OFA. Oime versus the output from CTL. Oza. input for the CIP strategy. As shown in Tab. (6 using Oimg demonstrates better open-vocabulary tracking performance compared to O This suggests that the category information flow, carrying Ouny aligned with CLII image embeddings, facilitates improved target capture and continuous classification.
CONCLUSION
In this paper, propose OVTR, the first end-to-end open-vocabulary multiple object tracker that jointly models motion, appearance, and category. By leveraging the category information propaga- tion strategy, we establish a higher-level category information flow, enabling the model classify and track in a stable and continuous manner. The introduction of a tive strategies the decoder enhances the model's generalization capability, fosters deep modality interaction, and allows for harmonious operation of classification and tracking. Our method elimi- nates the need for explicit track associations, complex post-processing, and time-consuming prepro- cessing, resulting in faster inference speeds and a simplified workflow. Despite being RPN-free and not relying on proposals containing novel categories, our model demonstrates strong generalization capabilities, achieving robust open- voccauuauyl tracking. As Transformer-basee based framework, it is data- fainnfriendly, scalable, and has potential for further optimization. We hope this end-to-end model provides more promising solution for open-vocabulary tracking in diverse scenarios.
structure and protec-
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China under Grant 62176096.
REFERENCES
Rami AI-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level lan- guage modeling with deeper self-attention. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 3159-3166, 2019.
Mk Bashar. Samia Islam, Kashifa Kawaakib Hussain, Md Bakhtiar Hasan, ABM Rahman, and Md Hasanul Kabir. Multiple object tracking in recent times: a literature review. arXiv preprint arXiv:2209.04799 2022.
Philipp Bergmann, Tim Meinhardt, and Laura Laixe Tracking without bells and whistles. In Proceedings of the JEEE/CVF international conference on computer vision, pp. 941-951,
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-221199911111222 Springer, 2020.
Sijia Chen, En Yu, Jinyang Li, and Wenbing Tao. Delving into the trajectory toig distribution for muti-object tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19341-19351, 2024a.
Sijia Chen, En Yu, and Wenbing Tao. Cross-view referring multi-object tracking. arXiv preprint arXiv:2412.17807, 2024b.
Achal Dave, Tarasha Khurana, Pavel Tokmakov, Cordelia Schmid, and Deva Ramanan. Tao: A large-scale benchmark for tracking any object. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16. pp. 436-454. Springer, 2020.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open- vecenvocabylary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14084-14093, 2022.
Ruopeng Gao and Limin Wang. Memotr: Long-term memory-augmented transformer for multi- object tracking. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9901-9910, 2023.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and pattern recognition, pp. 3354-3361. IERE, 2012.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.139211 2021.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan- Yen Lo, et al. Segment anything. In Proceed- ings of the IEEF/CVF International Conference Computer Vision, pp. 4015-4026, 2023.

---

Published as a conference paper at ICLR 2025
Solomon Kullback. Kullback-leible divergence, 1951.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Li- juan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 10965-10975, 2022a.
Siyuan Li, Martin Danelljan, Henghui Ding, Thomas E. Huang, and Fisher Yu. Tracking every thing in the wild. In European Conference on Computer Vision, pp. 498-515. Springer, 2022b.
Siyuan Li, Tobias Fischer, Lei Ke, Henghui Ding, Martin Danelljan, and Fisher Yu. Ovtrack: Open- vocabulary multiple object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5567-5577, 2023.
Siyuan Li, Lei Ke, Martin Danelljan, Luigi Piccinelli, Mattia Segu, Luc Van Gool, and Fisher Yu. Matching anything by segmenting anything. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18963-18973, 2024.
Siyuan Li, Lei Ke, Yung-Hsu Yang, Luigi Piccinelli,, Mattia Segů, Martin Danelljan, and Luc Van Gool. Slack: Semantic, location, and appearance aware open-vocabulary tracking. In European Conference on Computer Vision, pp. 1-18. Springer. 2025.
Haiji Liang and Ruize Han. Ovt-b: A new large-scale benchmark for open-vocabulary multi-object tracking. In The Thirly-eighl Conference on Neural Information Processing Systems Datasets and Benchmarks Truck.
Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection. arXiv preprint arXiv:2211.14843 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.055: 034.09393.. 2023.
Tim Meinhardt, Alexander Kirillov, Laura Taixe and Christoph Feichtenhofer. Trackformer: Multi-object tracking with transformers. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 8844-8854, 2022.
Pha Nguyen, Kha Gia Quach, Kris Kitani, and Khoa Luu. Type-to-track: Retrieve any object via prompt-based tracking. Advances in Neural Information Processing Systems, 36, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 0748-8763. PMLR, 2021.
Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple object tracking with transformer. arXiv preprint arXiv:2012. 15460, 2020.
Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In 2017 IEEE international conference image processing (ICIP), pp. 3645- 3649. IEEE, 2017.
Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, and Jianbing Shen. Referring multi-objec tracking. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 14633-14642, 2023a.
Xiaoshi Wu, Feng Zhu, Rui Zhao, and Hongsheng Li. Cora: Adapting clip for open-vocabulary detection with region prompting and anchor pre-mat matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7031-7040, 2023b.
12

---

Published as a conference paper at ICLR 2025
En Yu. Zhuoling Li, and Shoudong Han. Towards discriminative representation: Multi-view tra- jectory contrastive learning for online multi-object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8834-8844 8843, 2022.
En Yu, Songtao Liu, Zhuoling Li, Jinrong Yang. Zeming Li, Shoudong Han, and Wenbing Tao. Generalizing multiple object tracking to unseen domains by introducing natural language rep- resentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3304-3312,
En Yu, Tiancai Wang, Zhuoling Li, Yuang Zhang, Xiangyu Zhang, and Wenbing Tao. Motrv3: Release-fetch supervision for end-to-end multi-object tracking. arXiv preprint arXiv:2305. 14298, 2023b.
Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European Conference on Computer Vision, pp. 106-122. Springer, 2022.
Fangao Zeng. Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End- to-end multip tracking with transformer. In European Conference on Computer Vision, pp. 639-675. Springer, 2022.
Hao Zhang. Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint ar arXiv:2203.03600, 2022a.
Yifu Zhang, Peize Sun. Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In European conference on computer vision, pp. 1-21. Springer, 2022b.
Yuang Zhang, Tiancai Wang, and Xiangyu Zhang. Motrv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22056-22066 2023.
Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 23151-23160, 2023.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. European conference on computer vision, pp. 474-490. Springer, 2020.
13

---

Published as a conference paper at ICLR 2025
A
MORE IMPLEMENTATION DETAILS
A. CATEGORY SELECTION BASED ON DISTRIBUTION.
During training, for each frame input, we randomly sample 250 categories based on the dataset's class distribution, including the ground truth categories of the current frame. These categories are represented as text embeddings, defining the scope of classes for classification learning in the current frame. Specifically, in addition the ground truth of the current frame, the probability of selecting a certain category as a negative class is positively correlated with the 0. 0.7th power of the number of objects in that category. This approach helps mitigate the impact of the long-tailed tailed distribution in the dataset.
A.2 GENERATION OF IMAGE EMBEDDINGS
We use the CLIP (Rafford al. 2021) model to generate image and text embeddings. For the generation of image embeddings, specifically, rather than using large number of proposals detected by an additional RPN-based detector, which may include novel categories, we simply extract specific number of cropped images for each basc category from the dataset. Using the ground truth bounding boxes, we crop the images with a factor of 1.2 to create these cropped images. The cropped images for each category are then fed into the CLIP image encoder. The resulting representations are normalized and averaged to generate the image embedding for that category.
A.3 LEARNABLE SCALING PARAMETERS
The category score matrix obtained through dot products is scaled to an appropriate magnitude before applying softmax. Learnable scaling parameters are used for scaling, with each layer having independent scaling parameters to maintain the operational independence across decoder layers.
A.4 DETAILS OF ATTENTION ISOLATION STRATEGIES
KL divergerettte value
conneponding quary
Da KL. diverggenc
small
large
valid mark (Trme)
Ox
Qg
"dog"
0.1
0.3
0
0,2
"pason""
0.
0.3
0.
0.
"ow"
0.6
0.1
>>
"dccchoord"
0.2
0.
0.1
0.2
Matdi
Matris
Category Isolation Mlask
Content
Figure 4: Attention isolation masks. In the difference matrix, The darker areas indicate smaller KL divergence, meaning the category prediction distributions of the corresponding queries in the current layer are more similar. This suggests that the category information of the corresponding input queries passed to the next layer is similar. The darker areas of the masks represent masked positions, while the red dashed box shows that interactions among track queries will be maintained.
The specific masks of the category isolation strategy and the content isolation strategy are illustrated in Fig. 4 As mentioned in Sec. B.3] the output Out of the current decoder layer predicts the Category Score Matrix S, which is used compute the KL divergence between category predictions for each pair of queries, resulting in the Difference Matrix D. D captures the similarity of category information among queries. Tiol is then applied to D to generate an isolation mask 7, which sets the positions with excessively large KL divergence values to True, thereby masking these positions. This mask I is subsequently utilized in the self-attention layer of the next decoder layer. Since queries updated and propagated between decoder layers, the updated queries ([P', Otal) output by the current layer serve as the input queries for the next layer. Therefore, the similarity captured in D from the current layer well-suited for application to the input queries of the next layer.
Specifically, is added to the attention weights obtained from the batch matrix multiplication be- tween q and & (where both g and l are input queries) during the self-attention in decoder layers
14

---

Published as a conference paper at ICLR 2025
Table 7: Verifying the effectiveness com- ponents in OVTR. Isol: isolation (protection) strategies, Dual: sual-branch structure.
Table 8: Different image sequence lengths for multi-frame optimization. The maximum image sequence lengths set for the four experiments are 2, 3, 4 and 5 respectively.
CIP Isol Dual TETA AssocA ClsA, ClsA
Length
TETA
AssocA
ClsAb
ClsA,
X
28.9
29.1
10.7
1.7
2
/
X
x
30.3
30.2
12.1
1.8
2
27.9
24.9
13.5
2.2
3
/
V
X
31.2
31.8
12.6
1.9
30.5
32.2
14.5
2.5
4
V
X
32.1
32.8
14.6
2.3
31.6
33.8
14.7
2.4
V
V
33.2
34.5
15.2
2.7
5
33.2
34.5
15.2
2.7
to 6. Because the coordinates (i, of I corresponding to queries i and j with significant category information differences are set to True (in practice, the value is set to -oo), the attention weight at this position becomes O after the softmax operation. This enforces attention isolation, preventing category information from interfering with each other. It is worth noting that this mask is not used in the first decoder layer, as the content part C' of the queries fed into the self-attention in the first decoder layer is initialized through learnable parameters and does not contain category information.
The implementation of the content isolation mask is straightforward, as it simply sets the positions for detect queries and track queries that attend to each other to True. This mask is applied solely to the self-attention layer of the first decoder layer for two reasons: first, the content gap between Qan and Qu diminishes after the first decoder layer interaction, as detection queries carry enough object semantic content; and second, to ensure that track queries can capture global information. Apart from being applied in different decoder layers, the content isolation mask performs the same oper- ation in self-attention as the previously mentioned category isolation mask, applied to the attention weights restrict the interaction between queries.
B ADDITIONAL EXPERIMENTS
B. VERIFYING THE EFFECTIVENESS OF COMPONENTS IN OVTR
In this part, we verify the effectiveness of three core components in OVTR: the category information propagation (CIP) strategy, the dual-branch structure, and the decoder isolation protection strategies. The row represents the baseline model we constructed, which does not incorporate category infor- mation when iterating track queries, essentially lacking a category information flow. Specifically, in the single-branch structure (CTI only), we designed auxiliary network structure identical to the standard MOTR (Zeng et al. 2022) decoder, but it operates entirely independently of the CTI branch. As a result, the queries passed through this structure lack category information. When using its output as input to CIP, the category information flow lacks direct category information.
The results are reported in Tab. According to the results, all the components have boosted the open-vocabulary tracking performance effectively. OVTR (row 5) achieves a 14.9% improvement on TETA compared to the constructed baseline (row 1). The adoption of the CIP strategy increases AssocA by 3.8% and ClsAp by 13.1%, significantly improving both association and classification. This suggests that the inclusion of CIP enables better collaboration between classification and track- ing. Specifically, with the addition of the CIP strategy, the dual-branch structure (row 4) provides the greatest performance gain. Compared using CIP alone (row 2). TETA increases by 5.9%, and both AssocA and ClsA see significant improvements, particularly with ClsA, increasing by 20.7%. This suggests that the designed dual-branch structure enhances generalization performance and modal- ity interaction, leading to powerful open vocabulary capability. This also demonstrates that the dual-branch structure strengthens tracking effectively by feeding the aligned representations into the category information flow, helping capture objects and enhance tracking. In contrast, the isolation strategies (row 3) provide a smaller gain compared to the mual-branch structure, but still improve AssocA by 5.3% compared to using CIP alone. This indicates that the protection provided to the decoder effective, helping maintain continuity between the initial frame detection and subsequent tracking, thus improving tracking performance.
B.2 ANALYSIS OF IMAGE SEQUENCES LENGTH FOR OPTIMIZATION
To investigate whether multi frame optimization helps the model learn more stable category infor- mation transfer and tracking, set the maximum number of optimization frames in the ablation
15

---

Published as a conference paper at ICLR 2025
Table 9: Open-vocabulary MOT inference speed test on TAO dataset. OVTR-Lite excludes the category isolation strategy and tensor KL divergence computation.
Method
Speed
Novel
Basc
Validation
FPS
TETA?
LocAt
AssocA
ClsAt
TETAT
LocAt AssocAt
ClsA
OVTrack eLi a 2023
3.1
27.8
48.8
33.6
1.5
35.5
49.3
36.9
20.2
OVTR (Ours)
3,4
31.4
54.4
34.5
5,4
36.6
52.2
37.6
20.1
OViR-Lite (Ours)
12.4
30.1
52.7
34,4
3.1
35.6
51.3
37.0
18.6
experiments to 2, 3, 4 and 5. The first experiment maintains frame count of 2 throughout, In the second experiment, we start with frames and increase the number of frames by at the 4th epoch. The third experiment starts with 2 frames and increases the frame count by 1 at both the 4th and 7th epochs. The fourth experiment also starts with 2 frames but increases the frame count by at the 4th, 7th, and 14th epochs. As shown in Tab. 8 when the length of the video clip gradually increases from 2 to the TETA, AssocA, and ClsA, metries improve by 19.09 19.0%, 38.6% and 12.6%, respectively. This indicates that multi-frame joint optimization contributes to improved tracking performance and classification stability.
B.3 INFERENCE SPEED EVALUATION
We evaluated the inference speed of OVTrrack and OVTR single NVIDIA GeForce RTX 3090 GPU. The results reported in Tab. | indicate that OVTR achieves faster inference compared to OVTrack. Additionally, isolation strategy and tensor KL divergence computation. Despite some performance trade-offs, OVTR-Lite still outperforms OVTrack in overall performance. It achieves 4 times faster inference speed compared to OVTrack, while keeping memory usage below 4GB during inference. The speed test is conducted on the TAO validation set.
tested lightweight version, OVTR-Lite, which excludes the category
B.4 EVALUATION OF PEDESTRIAN CATEGORY ON THE KITTI DATASET
Table 10: Zero-shot domain transfer to KITTI dataset (Pedestrian category).
Method
Data
MOTAT
IDF1
IDs..
MOTP
IDRT
FN]
OVTrack G-LVIS,LVIS
11.2
113
6.2
4083
OVTR
LVIS
40.5
56.1
176
74.1
56.1
1332
Since KITTI differentiates between Pedestrian and Cyclist, which the open- open-vocabulary task does not, this differentiation results in higher false positive rate when evaluating the Pedestrian category, making MOTA metrics less indicative of actual performance for pedestrian tracking. To provide more accurate assessment of pedestrian tracking, we selected IDR, FN, and MOTP to compare the model's performance on pedestrians. Although there may be numerous false positives for cyclists, comparing the recall rate for pedestrians is reasonable. The results in Tab. IO indicate that our method significantly outperforms OVTrack in capturing and tracking pedestrian targets when trans- ferred to the KITTI dataset. This finding is consistent with our tests on TAO, where OVTrack often fails to identify pedestrians. In addition to the model's 's design and performance, this may also be related to the fact that OVTrack includes pedestrians in the classification learning scope during every frame of training, while the annotations for pedestrians in LVIS are not fully comprehensive.
B.5 EVALUATION DATASET WITH HIGHER ANNOTATION FRAME RATE
Table 11: Open-vocabulary MOT performance comparison on OVT-B dataset.
Method
Elements
Novel
Base
OVT-B
Data
Embeds Proposals
TETA
LocAf AssoAA ClsA
TETAT LocAf AssocA ClsAf
OVTrack Liet
2023
G-IVIS.IVIS
99.4M
45.5
61.1
65.5
9.6
46.8
60.5
66.7
13.4
OVTrack+ Liang & Han
G-LVIS.LVIS
99.4M
46.4
62.5
67.3
9.4
47.6
61.6
68.2
13.2
OVTR
LVIS
1,732
45.5
59.7
67.6
9.3
47.6
60.9
68.9
12.9
We compare open-vocabulary MOT performance OVT-B dataset (Liang & Han). Tab. I1 shows that OVTR outperforms OVTrack on base TETA and matches OVTrack+ performance. Without using proposals containing novel categories, OVTR still achieves comparable novel TETA results
16

---

Published as a conference paper at ICLR 2025
(a) Dala ugmentations for OVTrack
Groun Toutk Otpec
Sunglee
Mossic
Aegrontation
T.
on Conoma
(b) Data A ugmentations for OV (Ous)
Dynamic Mosale
Random Occhasion
Sumplee
Inage
Augrectatiee
F
Figure 5: OVTR data augmentations. Unlike OVTrack, which is based on appearance matching, our method does not utilize diffusion models for data augmentation. Instead, we propose Dynamic Mosaic and Random Occlusion data augmentation to simulate object appearance and disappcarance, tracking continuity after occlusion, and maintaining correct associations when relative motion oc- curs between tracked objects and others.
to OVTrack. Our method also achieves the highest AssocA for both base and novel categories, demonstrating superior tracking performance in high high-frame-rate videos.
C STATIC IMAGE AUGMENTATIONS FOR QUERY-BASED TRACKING
Our multi-frame training data augmentations are summarized in Fig. 5 As a MOTR-like query- based tracker, OVTR requires a different approach to data augmentation and has higher demands for training data. Our data augmentation includes conventional techniques, such as applying random resizing, horizontal flipping, color jittering, and random affine transformations to single images to create distinguishable multi-frame data. This part aligns with OVTrack (Li al. (023). Addition- ally, we propose Dynamic Mosaic and Random Occlusion augmentations, specifically designed for MOTR-like trackers.
Unlike appearance- appearance-matcingg-be matching- methods, our approach does not rely on diffusion models for ad- ditional data augmentation. Instead, it focuses enhancing the motion realism of static images dur- ing data augmentation, making them more representative of the physical world. Specifically, while query-based trackers excel maintaining associations over extended periods, they place higher de- mands on the realism of object motion patterns in training data. For OVTR, track queries must not only learn capture the same object moves to new position in the next frame, but also han- dle scenarios such as object appearance and disappearance, tracking continuity after occlusion, and maintaining correct associations when relative motion occurs between tracked objects and others.
To address these challenges, we propose Dynamic Mosaic augmentation, improvement over the Mosaic augmentation OVTrack. In addition to stitching four different images into single com- posite, Dynamic Mosaic generates images with varying relative spatial relationships among objects across different training frames. This simulates scenarios such as objects approaching or reced- ing from each other, crossing paths, and exhibiting relative size changes. The Random Occlusion augmentation is employed to simulate situations where objects disappear due to occlusion and then reappear suddenly emerge in the scene.
17

---

Published as a conference paper at ICLR 2025
T
T
(a) Without Mosaic
(b) With Dynamic Mosaic
Figure 6: OVTR data augmentations visualization. From the images, it can be observed that Dynamic Mosaic augmentation introduces relative motion and relative size changes between targets, while the positional swapping of images simulates target crossing paths. Additionally, the bottom- center image illustrates a random occlusion applied to giraffe.
Here, we detail the specific operations during training. Dynamic Mosaic applied every sam- pled image. To avoid neglecting learning for simple scenes or larger targets, we set the probability of applying Dynamic Mosaic augmentation a sampled training image 0.5. contrast, Random Occlusion is applied to every sampled image. For Dynamic Mosaic augmentation, four sampled im- ages are processed with relative size adjustments, vertical or horizontal translations, random swaps of positions between two images, and possible horizontal flipping of one or more images. Random Occlusion requires a simple preprocessing step, where a script is used to mark targets that rarely overlap with others, based on the ground truth bounding boxes in the annotations. During Random Occlusion augmentation, only these marked targets are randomly processed, preventing unintended occlusion of other targets and avoiding negative impacts model learning. The output images after data augmentation used for training OVTR, including cases with and without Dynamic Mosaic, are shown in Fig. 6
D MODEL AND TRAINING HYPERPARAMETERS
Tab. 12 and Tab. preseent the hyperparameters used in the detection training phase and the tracking training phase, respectively. The hyperparameters used in the tracking training phase are listed in Tab. 13 while other unmentioned parameters, such structural parameters and loss weights, the same as those in the detection phase. Our model follows the standard 6-encoder, 6-decoder structure in DETRs. For the update and propagation of queries between decoder layers, we specifically use the updated position part P' as the input position part of the queries for the next decoder layer, while the representation On output by the CTT branch is used as the content part for the next decoder layer. This is because the CTI branch includes an extra gross-attention layer compared to the OFA branch, allowing Out to contain more refined category information, leading to more accurate priors for classification in subsequent layers. The shuffle ratio, dislocation ratio, single ratio range, and occlusion ratio range are hyperparameters the Dynamic Mosaic and Random Occlusion augmen- tations used to control the extent of augmentation. Sampler lengths specifies the number of frames for multi- frame training during each of the four phases. Sampler steps indicates the epochs where these transitions different multi-frame training lengths occur.
18

---

Published as a conference paper at ICLR 2025
Table 12: Hyper-parameters used in the detection training phase.
Item
Value
optimizer
AdamW
Ir
4e-5
weight decay
lc-4
clip max norm
0.1
number of encoder layers
number of decoder layers
6
dim feedforward
2048
hidden dim
256
dropout
0.0
nheads
8
number of queries
900
set cost class set cost bbox set giou
3.0
5.0
2.0
ce loss coef
2.0
bbox loss coef
5.0
giou loss coef
2.0
alignment loss coef
2.0
Table 13: Additional hyper-parameters used in the tracking training phase.
Item
Value
4e-5
Ir of backbone
4e-6
sampler steps sampler lengths shuffle ratio dislocation ratio single ratio range occlusion ratio range
4, 7, 14
2, 3, 4,
0.1
0.25
0.7, 1.2
0.1, 0.13
E VISUALIZATION
As shown in the figures, the results on the left represent the tracking outcomes of OVTR, while the results on the right depict those of OVTrack. Overall, it can be observed from the four sets of images that our method experiences fewer ID switches and results fewer false positives.
In Fig. it can be seen that our tracker, OVTR (on the left), maintains stable tracking of the three sheep without any switches, whereas OVTrack (on the right) experiences ID switches and loses many previously detected targets. In Fig. 8 OVTrrcck (on the right) generates significantly higher number of false positives compared to our method. In Fig. 9 on the left, our OVTR tracks both the person and the components of the clothing and skateboard, where the person (ID 5) and the jacket (ID 2) remain consistently tracked over 15 frames (after sampling every 30 frames) in a high- speed scenario. The tracking remains stable, and the correct categories preserved, demonstrating the effectiveness
our dual-branch structure and CIP strategy. In contrast, OVTrrack reaches ID 110 Frame 17, indicating large number of ID switches, suggesting some challenges in tracking stability. In Fig. 10 in the second frame, the squirrel on the left partially occludes the squirrel on the right. After the occlusion ends in the third frame, tracker (on the left) maintains the same ID for the squirrel as in the first frame, while OVTrack (on the right) experiences another ID switch and significant number of classification errors.
Overall, from the tracking results, it is evident that tracking diverse targets in scenes with a variety of categories presents challenges. However, our method, combining CIP strategy with a dual- branch structure and decoder protection strategies, achieves relatively robust tracking. The results demonstrated by OVTR suggest that it holds potential for effective tracking such scenarios.
19

---

Published as a conference paper at ICLR 2025
T
T2
T
(a) OVTR (Ours)
(b) OVTrack
Figure 7: Qualitative comparison of OVTR and OVTrack in multi-object motion scenario 1.
T
Tz
T
(a) LOVTR(Our)
(b) OVTrack
Figure 8: Qualitative comparison of OVTR and OVTrack in multi-object motion scenario 2.
20

---

Published as a conference paper at ICLR 2025
Frame
Frame
T
Boanne 16
Prame
T2
T 3
OVTR (Ours)
(b) OVTrack
Figure 9: Qualitative comparison in a concentrated fast-moving multi-object scenario.
T
Tz
T
(a) OVTT(Ours
(b) OVTrack
Figure 10: Qualitative comparison on rare category objects in an occlusion scenario.
21


Reviews:
[{"review_id": "JI9v8os6WK", "reviewer": "Reviewer_dQGj", "summary": "This paper proposes the first solution for end-to-end open-vocabulary multiple object tracking called OVTR. The model builds on close-set end-to-end multiple object tracker MOTR while extending it to open-vocabulary by adding the category Information Propagation strategy and dual-branch vision-language alignment training. The method achieves good results on open-vocabulary tracking benchmarks and also demonstrates good zero-shot transfer ability to unseen domains.", "questions": "Will the code be public? Since this is the first work that performs end-to-end open-vocabulary object tracking, making the implementation details public is of great importance to the research community.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "1. The paper presents the first end-to-end Transformer solution for open-vocabulary tracking that performs joint detection and tracking. This is known to be a hard problem for two main reasons:\n   - There is a lack of sufficiently diverse video data to train end-to-end models effectively. Training on a small set of videos for tracking often hurts detection performance. The paper addresses this by augmenting static images for training, demonstrating that this approach can still yield good performance in an end-to-end setup. Although it is not new to use static images for tracking it is rare in end-to-end solutions, especially in open-vocabulary scenarios. \n   - In end-to-end multi-object tracking solutions, queries often perform multiple tasks—tracking, localization, and classification—which is really difficult to balance in open-vocabulary scenarios. The paper’s proposed category and content isolation strategies help balance these tasks, which is a sensible approach.\n\n2. The zero-shot transfer experiments on the KITTI dataset indicate strong performance in association, particularly in terms of the IDF1 metric, showing significant improvement over previous methods like OVTrack.", "weaknesses": "1. The paper is not clearly written and lacks important details. For instance, understanding the overall framework and its training process is challenging. The paper mentions using only static images to train the framework; however, using static images implies there is no separation of new and old objects, which are typically handled differently in MOTR training. This critical aspect is not clearly explained in the paper.\n\n2. The ablation results in Table 3 and Table 4 are confusing and show only marginal improvements over the baseline, especially in association. For example, in Table 4, adding category isolation only improves ClsA_b by 0.6 and ClsA_n by 0.1, while adding content isolation alone only improves 0.1 in AssocA performance. A more detailed explanation is needed here to clarify these results.​\n\n3. Missing latest related works. The paper only compares OVTrack. However, In open-vocabulary tracking, there are some new advancements eg. MASA[1] at CVPR2024. Also, SLAck at ECCV2024, which is also a transformer-based end-to-end approach. Maybe SLAck can be a concurrent work but MASA seems to be fair to discuss.\n\n[1] Li, Siyuan, et al. \"Matching Anything by Segmenting Anything.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SllZcsuUde", "reviewer": "Reviewer_tyWH", "summary": "In this paper, the authors propose OVTR (End-to-End Open-Vocabulary Multiple Object Tracking with TRansformer), the first\nend-to-end open-vocabulary tracker.\n\nSpecifically, to achieve stable classification and continuous tracking, they designed the CIP (Category Information Propagation) strategy, which establishes multiple high-level category information priors for subsequent frames. \n\nThey also introduce a dual-branch structure for generalization capability and deep multimodal interaction and incorporate protective strategies in the decoder to enhance performance.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ The motivation for this work, the flow of category information, is reasonable.\n\n+ This is the first end-to-end framework for open-vocabulary tracking.\n\n+ Experimental results have verified the effectiveness of the proposed method, to some extent.", "weaknesses": "- The writing of the method part is not easy to follow.\n\n- From the results, we can see that the performance of the proposed method is not very excellent, compared to the basic baseline method OVTrack, not to mention the most recent methods.\n\n- The authors are suggested to present the advantages of the end-to-end Transformer methods compared to the tracking-by-OVD methods, based on the experimental, including the performance, speed, model size, etc.\n\n- The authors claim that due to resource constraints, the ablation studies are conducted on a subset of 10,000 images. This makes the evaluation not very convincing. The overall framework can be implemented on the whole dataset. The ablation study without some components can be also conducted.\n\n- The object category in the KITTI dataset is not various, which is not very appropriate for the OVTrack task. The comparison of OVTrack and the proposed method on KITTI using Car Category is not very convective, for the OVTrack problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ub7r5eUQrB", "reviewer": "Reviewer_3Y3k", "summary": "The authors present the first application of the Transformer architecture in Open-Vocabulary Multiple Object Tracking. This work simultaneously considers the motion, appearance, and category features of the targets. A novel approach for constructing inter-frame category information flow is proposed, along with a Dual-Branch structure. As a result, an improvement of approximately one point is achieved on the Novel TETA metric of the TAO dataset.", "questions": "I have included all the questions in the 'Weaknesses section', outlining approximately twelve areas of questions.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "The advantages of this work include the following:\n\n1. It effectively leverages the strengths of the Transformer architecture and strives to apply it to the Open-Vocabulary Multiple Object Tracking (OVMOT) task.\n2. The motivation throughout the paper is well-founded, as it integrates motion, appearance, and category features, utilizing category information flow to guide subsequent tracking.\n3. The use of a dual-branch structure addresses the issue of increased training data arising from proposals with unknown categories during training. Instead, it employs ground truth to construct supervision, thereby reducing burdensome in training process.", "weaknesses": "Firstly, possibly due to time constraints, I feel that **this paper is not complete**. I read the article twice carefully. During the first read, I thought I had missed many details. However, upon a second, more thorough review, I found that the paper contains **only fragmented descriptions of the training process** and **lacks any explanation of how inference is conducted**.\nI believe the potential issues are as follows:\n\n### 1. Unclear Architecture Representation\nFigure 2 lacks clarity in its representation. Several concerns need to be addressed:\n- The distinction between Position Part and Content Part is not well-defined\n- In Figure 3, it's unclear why the Position Part from the Encoder is directly fed as input - is this representing the feature blocks from Image Features or fused features?\n- The nature and purpose of the Content Part input need clarification\n- There's an unexplained discrepancy between Figure 2 (5 outputs) and Figure 3 (3 outputs)\n\n### 2. Attention Mechanism Implementation\nSection 3.3 presents several implementation issues:\n- The matrix $I$ is described as a filter for large detection queries in self-attention, but the implementation details are unclear\n- The relationship between $O_{txt}$ (final output) and its application in previous self-attention layers needs elaboration\n- The specific self-attention layer where the filtering is applied is not specified\n- The Content Isolation mechanism needs clarification, particularly regarding:\n  - Its operation between detection and track queries\n  - The implementation details\n  - Whether it uses category features for query filtering\n\n### 3. Dual-Branch Decoder Components\nIn Section 3.4:\n- The origins of matrices $P$ and $O$ are not shown in the Dual-Branch Decoder diagram\n- The purpose and utilization of $C^{(t+1)}$ obtained through propagation are not explained in the paper\n\n### 4. Sequential Loss Computation\nSection 3.5 presents logical inconsistencies:\n- The sudden introduction of $P$ in the Sequential loss computation lacks context\n- The relationship between this $P$ and the position part query from Section 3.3 needs clarification\n- The connection between $L$ in Equation 6 and $L$ in Equation 5 is ambiguous\n\n### 5. Inference Process\nThe paper lacks a comprehensive explanation of:\n- The complete inference pipeline\n- The feature fusion methodology for matching\n- The integration of multiple features during inference\n\n### 6. Claims of Mutual Benefits\nThe authors claim that \"Our approach jointly models localization, appearance, and classification, enabling tracking and classification to mutually benefit from each other.\" However, this mutual benefit is not convincingly demonstrated:\n- On the Validation Set:\n  - Only ClsA shows improvement for Novel categories\n  - Other metrics merely maintain similar performance levels\n  - LocA actually decreases by one point\n- On the Test Set:\n  - Novel category classification improvement is negligible\n  - Shows significant asymmetry compared to Validation Set results\n  - The claimed mutual benefits are neither stable nor consistently demonstrated\n\n### 7. Weak Experimental Results\nThe experimental results raise several concerns:\n- Minimal improvement of only ~1 point in TETA for Novel categories\n- Base TETA results are lower than OVTrack by one point on Validation Set\n- The authors' claim of using only Base proposals is questionable given:\n  - No improvement in Base category performance\n  - Previous methods using unlabeled proposals achieve superior results without requiring additional annotations\n  - The only apparent advantage seems to be computational efficiency\n\n### 8. Methodology Clarification Needed\n- The specific methodology for generating image sequences through data augmentation requires explanation\n\n### 9. Inconsistency in Training Description\n- Section 4.3 states \"All methods are trained are trained solely on novel categories\"\n- This appears to be a typo and should likely read \"solely on base categories\"\n- This inconsistency needs clarification\n\n### 10. Ablation Study Concerns\nThe ablation experiments only using 10,000 images show:\n- Generally low performance metrics compared with the results in Table 1\n- Particularly low ClsA for base categories\n- Limited discriminative power across different ablation settings\n\n### 11. Visualization Issues\nThe supplementary materials' visualizations are problematic:\n- The pole case example lacks representativeness\n- The comparison with OVTrack is unclear (the right-side detection in the first row appears superior)\n- The elephant case visualization is ambiguous\n- No guidance is provided to help readers interpret the differences\n- The dense bounding box visualizations make quality comparison difficult", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oprgAQMMVY", "reviewer": "Reviewer_xFtV", "summary": "This paper introduces an End-to-End Open-Vocabulary Multiple Object Tracking with Transformer, OVTR, a novel approach to multi-object tracking (MOT) that extends to handle unseen categories without prior training on those specific categories. The experimental results demonstrate that OVTR outperforms existing state-of-the-art methods in tracking novel categories on standard benchmarks like TAO and KITTI, showcasing its effectiveness and adaptability.", "questions": "1. The end-to-end MOT framework has been explored in MOT task, so what's the main difference with previous methods (such as TransTrack, TrackFormer, MOTR ...)\n2. Could the authors provide visual evidence to support the effectiveness of OVTR?  Specifically, it would be beneficial to show comparative visualizations of the tracking process, including scenarios where the model successfully handles rapid motion changes, partial occlusions, and diverse object appearances across different environments.\n2. What about the computational efficiency and real-time applicability of the OVTR model? Discussing its performance in terms of tracking speed and resource usage would greatly enhance the understanding of its practical value.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. End-to-End Architecture: OVTR is the first to propose an end-to-end trainable model for open-vocabulary MOT, which integrates motion, appearance, and category information simultaneously.\n2. The introduction of the CIP strategy and the dual-branch structure represent creative combinations of existing ideas in a novel application domain.\n3. The technical depth and rigorous experimentation by comparing OVTR with state-of-the-art methods provide a strong empirical basis for the claims made.", "weaknesses": "1. For the experimental results, although OVTR proposed for handling open-vocabulary multi-object tracking tasks demonstrates innovation and advancement, the performance did not meet the high expectations, particularly in terms of localization accuracy.\n2. The authors should introduce more implementation details and release the source code for reproducibility. \n3. The introduction of a dual-branch structure and multiple attention mechanisms increases the model complexity significantly. This might limit the practical deployment on resource-constrained platforms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)