CO-MOT: Boosting End-to-end Transformer-based Multi-Object Tracking via Coopetition Label Assignment and Shadow Sets
Authors: Feng yan
Weixin Luo
Yujie Zhong
Yiyang Gan
Lin Ma
Venue: ICLR 2025 Poster (2025)
Keywords: End-to-End Tracking
Transformer
Multi-object Tracking
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Existing end-to-end Multi-Object Tracking (e2e-MOT) methods have not surpassed non-end-to-end tracking-by-detection methods. One possible reason lies in the training label assignment strategy that consistently binds the tracked objects with tracking queries and assigns few newborns to detection queries. Such an assignment, with one-to-one bipartite matching, yields an unbalanced training, _i.e._, scarce positive samples for detection queries, especially for an enclosed scene with the majority of the newborns at the beginning of videos. As such, e2e-MOT will incline to generate a tracking terminal without renewal or re-initialization, compared to other tracking-by-detection methods.
To alleviate this problem, we propose **Co-MOT**, a simple yet effective method to facilitate e2e-MOT by a novel coopetition label assignment with a shadow concept. Specifically, we add tracked objects to the matching targets for detection queries when performing the label assignment for training the intermediate decoders. For query initialization, we expand each query by a set of shadow counterparts with limited disturbance to itself.
With extensive ablation studies, Co-MOT achieves superior performances without extra costs, _e.g._, 69.4% HOTA on DanceTrack and 52.8% TETA on BDD100K. Impressively, Co-MOT only requires 38% FLOPs of MOTRv2 with comparable performances, resulting in the 1.4× faster inference speed. Source code is publicly available at [GitHub](https://github.com/BingfengYan/CO-MOT).

Published as a conference paper at ICLR 2025
CO-MOT: BOOSTING END-TO-END TRANSFORMER- BASED MULTI-OBIECT TRACKING VIA COOPETITION LABEL ASSIGNMENT AND SHADOW SETS
Feng Yan; Weixin Luo," Yujie Zhong, Yiyang Gan, Lin Ma
Meituan Inc., China. {yanfeng05, Luoweixin, zhongyujie, ganyiyang) @mei tuantuunn.uuuuueeem uan. com forest. 1 linma@gma ccom
ABSTRACT
Existing end-to-end Multi- Aulti-Object Tracking (e2e-MOT) methods have not sur- passed non-end-to-end beekinngby-detection methods. One possible reason lies in the training label assignment strategy that consistently binds the tracked objects with tracking queries and assigns few newborns to detection queries. Such as- signment, with one-to-one bipartite matching, yields an unbalanced training, i.e. scarce positive samples for detection queries, especially for enclosed scenew with the majority the newborns at the beginning of videos. As such, e2e-MOT will incline to generate tracking terminal without renewal re- re-initialization, compared to other wrackingby-ddtecciii propose Co-MOT, simple yet effective method to facilitate e2e-MOT by novel coopetition label assignment with a shadow concept. Specifically, we add tracked objects the matching targets for detection queries when performing the label assignment for training the intermediate decoders. For query initialization, we ex- pand each query by set of shadow counterparts with limited disturbance itself. With extensive ablation studies, Co-MOT achieves superior performances without extra costs, e.g., 69.4% HOTA on DanceTrack and 52.8% TETA on BDD 10K. Impressively, Co-MOT only requires 38% FLOPs of MOTRv2 with comparable performances, resulting in the 1.4x faster inference speed. Source code is publicly available at https: http//gtttut httpa///www....ognnnnneeannnnn/a
methods. To alleviate this problem, we
INTRODUCTION
Multi-Objeet Tracking (MOT) is traditionally tackled by series of tasks, e.g., object detection (Zhao
et al. 2024 Zou et al. 2023). appearance Re-ID (Zheng et al. 2016 Li et al. 2018 Bertinetto et al.,
2016 Ye al.] 2024).
motion prediction (Lefèvre al.) 2014 Welch al. 1995), and temporal
association (Kuhn 1955). The sparkling advantage of this paradigm is task decomposition, leading an optimal solution for each task. However, it lacks global optimization for the whole pipeline.
Recently, there has been a rise in end-to-end Multi-Object Tracking (e2e-MOT) models using trans- formers. These models input consecutive video frames and directly output bounding boxes and association information, climinating the need for pre- or post-processing steps such as separate detectors, Re-ID feature extraction, or IoU matching. Notable contributions in this field include MOTR Zeng et al., 2022) and TrackFormer (Meinhardt et al. 2022), which perform detection and tracking simultaneously in unified transformer decoders. Specifically, tracking queries achieve iden- tity tracking through recurrent attention over time. Meanwhile, detection queries discover newborns in each newly arriving frame, excluding previously tracked objects, due to a Tracking Aware Label Assignment (TALA) during training. However, the TALA matching mechanism often leads to an im- balance between detection queries and tracking queries. This mechanism first matches the tracking queries and then assigns the remaining ground truth objects (newborns) to the detection queries. In many scenarios, especially in closed environments, there are very few newborn objects in the video
'These authors contributed equally to this work.
1 "ooresponning author.

---

Published as a conference paper at ICLR 2025
frames after the initial frame. To illustrate this, conduct an analysis the DanceTrack dataset and found that the ratio of newborn objects to tracked targets is 213:25483. Moreover, we observe that e2e-MOT tends to underperform due to suboptimal detection capabilities. In Figure e2c-MOT consistently results in tracking termination. MOTRv2 /Zang et al. 2023 supports this observation and addresses it by leveraging a pre-trained YOLOX detector to boost the performances. However, introducing an additional detector introduces extra overhead during deployment and undermines the advantages of the e2e-MOT approach.
In this paper, we present a novel perspective for addressing the above limitations of e2e-MOT: de- tection queries are exclusive but also beneficial to tracking queries. To this end, we develop a COopetition Label Assignment (COLA) for training tracking and detection queries. Except for the last Transformer decoder remaining the competition strategy to avoid trajectory redundancy, we allow the previously tracked objects to be reassigned to the detection queries in the intermediate decoders. Due to the self-attention mechanism among all queries, detection queries will be com- plementary to tracking queries with the same identity, resulting in feature augmentation for tracking objects with significant appearance variance. Thus, the tracking terminal problem will be alleviated.
Besides TALA, another drawback in transformer-based detection and tracking is one-to-one bipar- tite matching used, which cannot produce sufficient positive samples, as stated by Co-DETR Zong et al. ) and HDETR (Jia al. 2023 that introduce one-to-mane assignment to overcome this limitation. Different from these remedies with one-to-many auxiliary training, we develop a one-to- set matching strategy with novel shadow concept, where each individual query is augmented with multiple shadow queries by adding limited disturbance to itself, so as to ease the one-to-set optimization. The set of shadow queries endows CO-MOT with discriminative training by optimiz- ing the most challenging query in the set with the maximal cost, which can thereby enhance the generalization ability.
We evaluate our proposed method on the public MOT benchmarks, including DanceTrack (Sun al. 2022). BDD100K (Yu et all, 2020) and MOT17 Milan al.] 2016), and achieve superior performances. The contributions of this work lie in threefold. i) We introduce coopetition la- bel assignment for training tracking and detection queries for e2e-MOT with high efficiency. ii) We develop one-to-set matching strategy with novel shadow concept to address the hunger for positive training samples and enhance generalization ability. iii) Our approach achieves superior performances public benchmarks, while functioning an efficient tool to boost the performance of end-to-end transformer- based MOT.
METHOD
2.1 MOTIVATION
To explore the shortcomings of current end-to-end methods in tracking, we conduct an in-depth study of the effectiveness on Dance TrackTrack validation and MOT17 test dataset by analyzing MOTR, which is one of representative e2e-MOT methods. In Figure D we show the tracking results of MOTR in some video frames, e.g., Dance'rrack0073 and MOTT7-09. In the left three columns of the first row, the 3rd person (in the yellow box) is tracked normally in frame #237. However, in frame #238, due to an inaccurate detection, the bounding box is not accurately placed around that person (the box is too large to include a person on the left side). In frame #239, the tracking is completely wrong and associated with the 2nd person instead. In the right three columns of the first row, the 2nd person (in the yellow box) is successfully detected and tracked in frame #302. However, in frame #312, this person is occluded by other people. When the person appears again in frame #322, she is not successfully tracked or even detected. To determine whether the tracking failure is caused by the detection or association of MOTR, we visualized MOTR's s detection results in the second row. We remove the tracking queries during inference, and the visualization shows that all persons are accurately detected, This demonstrates that the detection will deteriorate due to the nearby tracked objects, though TALA used in training ensures that the detection with the same identity of tracked objects will be suppressed.
We further provide quantitative results of how the queries affect each other in Table All the decoded boxes of both tracking and detection queries are treated as detection boxes, allowing eval- uation by the mAP metric commonly used for object detection. We from the table that the
2

---

Published as a conference paper at ICLR 2025
#237
#238
#239
#302
#812 #322
Figure 1: Visualization of tracking results in Track773 and MOT17-09 videos. The first row displays the tracking results from MOTR, where all individuals are correctly initialized at the beginning (#237 and #302). However, heavy occlusion appears in the middle frames (#238 and #312), resulting in inaccurate detection (indicated by the yellow boxes). The tracking of the yellow targets finally terminates frames #239 and #322. The second row shows MOTR's detection results, where tracking queries are removed during the inference process. Targets in different frames are accurately detected without interference from tracking queries.
Table 1: Detection performance (mAP) of MOTR (v2) on DanceTrack validation dataset. /mans whether the tracking queries are used in the training or inference phase. All decoded boxes of both tracking if applicable and detection queries are treated as detection boxes for evaluation on mAP. We separately evaluate the detection performance for six decoders. For analysis, please refer to the motivation section.
model
Training
Inference
2
6
(a)
MOTR
41.4
42.4
42.5
42.5
42.5
42.5
(b)
MOTR
56.8
60.1
60.5
60.5
60.6
60.6
(c)
MOTR
57.3
62.2
62.9
63.0
63.0
63.0
MOTRv2
67.9
70.2
70.6
70.7
70.7
70.7
(e)
MOTRv2
71.9
72.1
72.1
72.1
72.1
72.1
CO-MOT(ours)
69.1
vanilla MOTR (a) has low mAP of 42.5%, but it increases by 18.1 (42.5% vs 60.6%) when removing tracking queries during inference (b). Then we retrain MOTR as a sole detection task by removing tracking queries (c), and the mAP further increases 66.1 1% (+5.5%). This means the DETR-style MOT model has sparking capability of detection but still struggles with the temporal association of varied appearances, which is the crucial factor of MOT.
We also observe excellent detection performance (70.7%) for MOTRv2, which introduces a pre- trained YOLOX detector. Removing tracking queries during inference brings slight improvement (1.4%) mAP, which means MOTRv2 has almost addressed the poor detection issue with high- quality detection priors from YOLOX. However, the introduction of YOLOX brings extra computa- tional burden, unfriendly for deployment. In contrast, we aim to endow the end-to-end MOT model with its powerful detection capability, rather than introducing any extra pretrained detector.
2.2 TRACKING AWARE LABEL ASSIGNMENT
Here we revisit the Tracking Aware Label Assignment (TALA) used to train end-to-ent Transform- ers such as MOTR and TrackFormer for MOT. At time
N queries are categorized into two types: Ny tracking queries Q = {at ..., 41 NT and N, detection queries Qa {4) ..., 4a Np ). where N = Ny + Np. All the queries will self-attend each other and then cross-attend the im- age feature tokens via decoders, and the output embeddings of the (-th decoder are denoted as E' (ef.... e } and FL {f fr ). At time t, there are Mc ground truth boxes. Among them, My are previously tracked objects, denoted as É = ( ē ). which are assigned to Ny tracking queries, where Mr < Nr as some objects disappear. Formally, j-th tracking em-
3

---

Published as a conference paper at ICLR 2025
Pox
Embrd
Pos. Emibed
Pox. Imored
O
2
64
4-/2
Decoder 1
12
Decoder 2
Decader 3
Decoder L
Backbone
A
92
Groundrruth
S-COLA
5-COUA
5: COLA
SAAALA
mocetin
for
The CO-MOT framework includes CNN-based backbone network for extracting image features, a deformable encoder for encoding image features, and deformable decoder that uses self- attention and cross-attention mechanisms to generate output embeddings with bounding box and class information. The queries in the framework use set queries as units, with each set containing multiple shadows that jointly predict the target. Detection queries and tracking queries used for detecting new targets and tracking existing ones, respectively. To train CO-MOT, S-COLA and S-TALA are proposed for training only.
Figure 2:
bedding e' will be assigned to the same identity as in the previous timestamp if still alive at this zero (disappearing). Besides, Mp newborn objects, denoted
moment; otherwise, will be as F (fi..., fM ), are assigned ND detection queries. Specifically, the Hungarian matching algorithm is used to find the optimal pairing between F and F for each decoder, using a cost func- tion (Tm (c) Ia (b) + (b) E RR that takes into account the class scores and box overlapping. Where (c)(c) represents the focal loss for classification, L1 (b) represents the L cost of the bounding box, and (6) represents the Generalized Intersection over Union cost.
2.3 OVERALL ARCHITECTURE
The entire CO-MOT framework is illustrated in Figure During the forward process, the features of an image in video extracted by the backbone and fed into the deformable encoder to aggregate information. Finally, together with the detection and tracking queries, they are used as the inputs of the L layer decoders (L 6 in this paper by default) detect new targets track the already tracked targets. It is worth noting that queries contain (Nr + ND) x Ns positions (P E R+) and embeddings (E E 2256) as we use deformable attention. Here Ns the number of shadow queries for each set, and we will introduce the shadow set concept in the following section. All the queries predict (Nr- + ND) Ns target boxes, where Nss queries set jointly predict the same target. To train CO-MOT, employ the COLA and TALA on the different decoders, along with the one-to-set label assignment strategy.
2.4 COOPETITION LABEL ASSIGNMENT
Unlike TALA, which only assigns newborn objects to detection queries, we propose a novel COope- tition Label Assignment (COLA). Specifically, we assign My tracked objects to detection queries in the intermediate decoders, i.e., < I,, as illustrated in Figure 2 As shown in the output of the first decoder, the track queries continue to track the 3rd and 4th person. The detection queries not only detect the 1st and 2nd newborns but also detect the 3rd and 4th people. Note that we remain the com- petition assignment for the L-th decoder to avoid trajectory redundancy during inference. Thanks to the self-attentio mechanism used between tracking and detection queries, detection queries with the same identity can enhance the representation of the corresponding tracking queries (e.g., grey 3rd helps blue 3rd).
2.5 SHADOW SET
In densely crowded scenes, objects can be lost or mistakenly tracked to other objects due to minor bounding box fluctuations. We conjecture that one query for one object sensitive to prediction
4

---

Published as a conference paper at ICLR 2025
noise. Inspired by previous works such Group-DETR and H-DETR, propose the one-to- set label assignment strategy for multi-bjject tracking, which is sigmificantly different from the object is no longer tracked by single query but by set of queries, where each member of the set acts as a shadow of each other. Tracking queries are rewritten as QT = =1177 N Jj= (4T Nr Jam and detection queries are rewritten as QD = Hab") "i Jj=1,... {4p " J.=1}- J. The total number of queries is N x Ns. When a particular query in the set tracks the object incorrectly, the other shadows in the same set help it continue tracking the object. In experiments, this strategy prove effective in improving tracking accuracy and reducing tracking failures in dense and complex scenes.
one-to-many manner. During tracking,
Initialization. pij E I and X R which represent the position and embedding of the j-th shadow query in the i-th set, are initialized, significantly affecting convergence and final per- formance. In this paper, we explore three initialization approaches: i) Trand' random initialization; ii Toopy: initializing all shadows in the same set with one learnable vector, i.e., pij fe and X =X, where P and X are learnable embeddings with random initialization; iii) Inoise: adding Gaussian noise (0, op) and (V a.) ) to P and X*J, respectively, in the previous ap- proach. In the experiment, we op and ar to le-6. Although the variance between each shadow in the same set is subtle after initialization, it expands to le-2 at the end of training. The last approach provides similarity for helping optimization and diversity to improve tracking performance.
Training. We propose a shadow-based label assignment method (S-COLA or S-TALA) to ensure that all queries within set are matched to the same ground truth object. Take S-COLA as an example: we treat the set as a whole and select one query as a representative based on certain criteria to participate in subsequent matching. Specifically, for tracking queries Qt, the tracked target in the previous frame is selected to match with the whole set; For detection queries Qa. we first calculate the cost function (m.mm E NoNoNN x MM)
( of all detection queries with respect to all ground truth. We then select the representative query using strategy À (e.g.. Mean, Min, and Max) for each set, resulting in Lm = am) E IR NRND XM6 Lm is then used as an input for Hungarian matching to obtain the matching results between the sets and newborns. Finally, the other shadows within the same set share the representative's 's matching result.
Inference. We determine whether the i-th shadow tracks object by the confidence score of the selected representative. Here we adopt a different strategy ø (e.g.. Mean, Min, and Max) for representative sampling. When the score of the representative is higher than certain threshold т, we select the box and score predictions of the shadow with the highest score as the tracking outputs and feed the entire set to the next frame for subsequent tracking. Sets that do not capture any object will be discarded.
3 EXPERIMENT
3.1 DATASETS AND METRICS
Datasets. We validate the effectiveness of our approach on different datasets, including DanceTrack, MOT17, and BDD100K.
The DanceTran dataset is used for multi-object tracking of dancers and provides high-quality an- notations of dancer motion trajectories. This dataset is known for its significant challenges, such as fast object motion and similar object appearances.
The BDD100K dataset is large-scale autonomous driving scene recognition dataset used for scene understanding in autonomous driving systems. This dataset provides multiple object categories, such as cars, pedestrians, etc. It can be used to evaluate model's 's performance in multi-object tracking across different object categories. The challenges this dataset include rapidly changing traffic and road conditions, diverse weather conditions, and lighting changes.
The MOT17 dataset is a commonly used multi-object tracking dataset, with each video containing a large number of objects. The challenges of this dataset include high object density, long occlusions, varied object sizes, dynamic camera poses, and so on. Additionally, this dataset provides various scenes, such indoor, outdoor, and city centers.
5

---

Published as conference paper at ICLR 2025
the Higher Order Tracking Accuracy (HOTA) metric (et al. 2020 which is a higher-order metric for multi-object tracking. Meantime We analyze the contributions of Detection Accuracy (DetA), Association Accuracy (AssA), Mwltiple- Subttppe-OObjet Track- ing Accuracy (MOTA), Identity Switches (IDS), and Identity Score (IDF1). For BDD100K, to better evaluate the performance of multi-class and multi-object tracking, we the Tracking Every Thing Accuracy (TETA) (Li et al., 20226). Localization Accuracy (LocA), Association Accuracy (AssocA), and Classification Accuracy(CIsA) metrics. The best results of end-to-end methods are marked in bold. Please pay more attention to the metrics with blue.
Metrics. To evaluate method,
Table 2: Comparison to existing methods the DanceTrack test set. and respectively represent the use of DAB-I BAB-Deformable backbone and joint training with CrowdHuman. For static images in CrowdHuman dataset, apply random shifts in CenterTrack to generate video vlips with pseudo tracks.
Source
HOTA
DetA
AssA
MOTA
IDFL
Non-End-to-ent
CenterTrack Zhou et al. 2020
ECCV'20
41.8
78.1
22.6
86.8
35.7
TransTrack (Sun al. 2020)
arXiv'20 20
45.5
75.9
27.5
88.4
45.2
FairMOT Zhang ct al. 2021
JCCV'21
39.7
66.7
23.8
82.2
40.8
QDTrack (Fischer al.
2022
CVPR'21
54.2
80.1
36.8
87.7
50.4
TraDeS (Wu et al. 2002
CVPR'21
43.3
74.5
25.4
86.2
41.2
ByteTrack (Zhngg et al. 2022b
ECCV'22
47.7
71.0
32.1
89.6
53.9
GTR &Zhou et al. [2022
CVPR'22
48.0
72.5
31.9
84.7
50.3
MT-lol Yan ct al. (2022)
arXiv'
66.7
84.1
53.0
94.0
70.6
OC-SORT (Cao et al.) 22)2
CVPR'23
55.1
80.3
38.3
92.0
54.6
C-BloU Yang el al. 2023)
WACV'23
60.6
81.3
45.4
91.6
61.6
MOTRv2 Zhang et al. (023)
CVPR 23
69.9
83.0
59.0
91.9
71.7
Fine'Track (Ren et al. (2023)
CVPR '23
52.7
72.4
38.5
89.9
59.8
GHOST (Seddenschwarz
2023
CVPR 23
56.7
81.1
39.8
91.3
57.7
Walker Segu et al.] (0244)
ECCV'24
52.4
36.1
76.5
89.7
55.7
General' (Qin al. 2024)
CVPR'24
59.2
82.0
42.8
91.8
59.7
MotionTrack (Xiao al.
(2024b)
arXiv'24
58.2
81.4
41.7
91.3
58.6
ConfTrack (Jung ct al. 2024)
WACV'24
56.1
89.6
56.2
MambaTrack (Xtao et al. 2024a
arXiv'24
56.8
80.1
39.8
90.1
57.8
Hybrid-SORT Yang et al.
2024
AAAI'24
62.2
91.6
63.0
UCMCTrack Yt
AAAI'24
63.6
51.3
88.8
65.0
DiffusionTrann (Luo et al. 2024)
AAAI'24
52.4
82.2
33.5
89.3
47.5
End-to-end
MOTR Zeng et al.
2022)
ECCV'22
54.2
73.5
40.2
79.7
51.5
DNMOT (uu et al. 20230
arXiv'23
53.5
89.1
49.7
McMOTR Gao & Wang 2023
ICCV'23
63.4
77.0
52.3
85.4
65.5
McMOTR (Gao & Wang
2023
ICCV'23
68.5
80.5
58.4
89.9
71.2
3 Yu et al. 2023
arXiv'23
68.3
91.7
70.1
SUSHI Cetintas et al. 2023
CVPR'23
63.3
80.
50.1
88.7
63.4
MambaTrack= rack+ (Huang
2024
arXiv'24
56.1
80.8
39.0
90.3
54.9
OuTR Liu et a 2024)
arXiv'24
54.5
88.3
55.7
DifIMOT (Lv et 2024
CVPR'24
62.3
82.5
47.2
92.8
63.0
ByteSSM (Hu et al. 2024
arXiv'24
57.7
81.5
41.0
92.2
57.5
CO-MOT
65.3
80.7
53.5
89.3
66.5
CO-MOT+
69.4
82.1
58.9
91.2
71.9
3.2
IMPLEMENTATION DETAILS
Our proposed label assignment and shadow concept can be applied to any e2e-MOT method. For simplicity, we conduct all experiments on MOTR. It uses ResNet50 the backbone to extract image features and employs a Deformable encoder and Deformable decoder to aggregate features and predict object boxes and categories. We also the data augmentation methods employed in MOTR, including randomly clipping and temporally flipping video segments. To sample video segment for training,
fixed sampling length of 5 frames and sampling interval of 10

---

In com- Published as a conference paper at ICLR 2025
frames. The dropout ratio in attention is set to zero. We train all experiments 8 V100-16G GPUs, with a batch size per GPU. For Dance'Track and BDD100K, we train the model for 20 epochs with initial learning rate of 2c-4, reducing the learning rate by a factor of 10 every eight epochs. We use 60 initial queries for a fair comparison with previous work. For MOT17, we train the model for 200 epochs, with the learning rate reduced by factor of 10 every 80 epochs. We use 300 initial queries due to the large number of targets to be tracked.
3.3 COMPARISON WITH STATE-OF-THE-AR METHODS
DanceTrack. Our method presents promising results on the DanceTrack test set, as evidenced by Table Without bells and whistles, method achieves impressive HOTA score of 65.3%. Compared to other e2c-MOT methods with the ResNet50 backbone, CO-MOT achieves remark- able performance improvements(e..g. 11.1% improvement HOTA compared MOTR, 11.8% compared to DNMOT, and 1.9% compared to MeMOTR). Although falls short of MeMOTR*, it is worth noting that MeMOTR* utilizes the more powerful DAB-Deformable-DDe parison with Non-e2e-MOT methods, our approach demonstrates significant improvements across various tracking metrics. For instance, when compared the state-of-the-art UCMCTrack, CO- MOT achieves a 1.7% improvement in HOTA and 1.5% improvement in AssA. Our approach can avoid tedious parameter adjustments and ad hoc fusion of independent detection and tracking modules. It realizes automatic learning of data distribution and global optimization objectives.
With joint training on CrowdHuman dataset, our method CO-MOT achieves even higher perfor- mance with 69.4% HOTA. This is 1.1% improvement over MOTRv3* with the ResNet50 backbone. Compared to CO-MOT(65.36 vs 69.4% HOTA), we can conclude that increasing the dataset size can lead to further improvements in tracking performance. Additionally, it performs par with the state-of-the-at Non-e2e-Mo MOT method MOTRv2*, which incorporates an additional pre-trained YOLOX detector into MOTR.
As shown in Table CO-MOT achieved the mAP of 69.1, significantly higher than MOTR's 42.5, and slightly lower than MOTRv2's 70.7. As research in this field continues, models like CO-MOT will likely play a crucial role advancing the state-of-the-art in multiple-object tracking, offering more reliable and efficient solutions for variety of applications.
Table 3: Comparison to existing methods on various datasets.
(a) MOT17 Dataset
(b) BDD100K Validation Set
HOTA
AssA
MOTA
IDFI
TETA LocA
AssocA
ClsA
Non-End-He to-e
Von-End-to-crt
CenterTrack
52.2
51.0
67.8
64.7
DeepSORT
48.0
46.4
46.7
51.0
TransTrack
54.1
47.9
74.5
63.9
QDTrack
47.8
45.8
48.5
49.2
FairMOT
59.3
58.0
73.7
72.3
TETer
50.8
47.2
52.9
52.4
QDTrack
63.5
64.5
77.5
78.7
MOTRv2
54.9
49.5
51.9
63.1
ByteTrack
63.1
62.0
80.3
77.3
End-to-end
OC-SORT
63.2
63.2
78.0
77.5
MOTR
50.7
35.8
51.0
DiffusionTrackk
60.8
58.8
77.9
73.8
CO-MOT"
52.8
38.7
56.2
63.6
MOTRv2
62.0
60.6
78.6
75.0
End-to-end
(c) MOT20 Test Dataset
TrackFormer
65.0
63.9
HOTA AssA End-to-end
MOTA
IDF1
MOTR
57.8
55.7
73.4
68.6
McMOT
56.9
55.2
72.5
69.0
MeMOT
54.1
55.0
63.7
66.1
MeMOTR
58.8
58.4
72.8
71.5
TrackFormer
54.7
68.6
65.7
DNMOT
58.0
75.6
68.1
CO-MOT 57.5 65.7
60.1
70.5
CO-MOT
60.1
60.6
72.6
72.7
BDD100K. Table 3b shows the results different tracking methods on the BDD100K validation set. To better evaluate the multi-category tracking performance, adopt TETA. which combines multiple factors such as localization, association, and classification. Compared with other meth- ods, although the LocA was considerably lower, achieve superior performance TETA with

---

Published as a conference paper at ICLR 2025
improvement of 2% (52.8% 50.8%), which is benefited from the strong tracking association per- formance revealed by the AssocA (56.2% vs 52.9%). Compared with MOTRv2, CO-MOT slightly falls behind on TETA, but its AssocA is much better than MOTRv2.
MOT17. Table Ba shows the results of the MOT17 test set. Due to the overemphasis on detection performance in MOT17, Non-e2e-MOT methods, starting from Byte'Track, excel at leveraging pow- erful detectors like YOLOX, achieving excellent detection performance (up to 64.5% DetA) along with other impressive metrics. In this regard, Transformer-based methods, especially e2e-MOT, still have a significant gap in detection performance due to the excessive predictions of dense and small objects MOT17. On the other hand, e2e-MOT suffers from severe overfitting issues because the MOT17 training set is very small, consisting of only about 5K frames. "Transformers lack some of the inductive biases inherent to CNNs, such as translation equivariance and locality, and there- fore do not generalize well when trained on insufficient amounts of data," as mentioned in the ViT paper. MOT17 provides insufficient data to train a Transformer model. Additionally, in MOT17 and DrnceTrack, bounding boxes that are less than 0.005 of the image area account for 60.26% and 1.17%, respectively, while bounding boxes that are greater than 0.02 of the image area account for 12.94% and 54.97%, respectively. This highlights that MOT17 primarily comprises smaller targets, which poses a significant challenge for enhancing detection performance with Transformer-base -based models.
However, despite these challenges, we still achieved a considerable improvement compared to other e2e-MOT methods, reaching HOTA score of 60.1%. %. Specifically, improved the performance of object association, which can be reflected by the AssA and IDFI metrics. These experimental results further validate the effectiveness of our approach.
MOT20. As the End-to-End solution has just emerged in the past year, there are not many methods evaluated on MOT20 that we could find. Here are the ones Table Bc Notably, our approach achieves 57.5% HOTA, which is the state-of-the-ee in End-to-end tracking methods.
3.4 ABLATION STUDY
Table 4: Ablation study on individual CO-MOT components on the DanceTrack Validation Set and MOT17 Test Set. As components are added, the tracking performance improves gradually.
Dance
Validation
MOT17 Test
COLA Shadow
HOTA
DetA
AssA
MOTA
IDF1
HOTA
DetA
AssA
MOTA
IDFI
(a)
56.4
71.8
44.6
79.8
57.5
57.8
60.3
55.7
73.4
68.6
X
60.2
73.2
49.7
81.8
62.4
58.5
58.0
59.2
70.3
70.7
59.0
72.6
48.2
80.9
59.6
/
61.8
73.5
52.2
81.7
63.3
60.1
59.5
60.6
72.6
72.7
Component Evaluation of CO-MOT. Based on the results shown in Table 4 we examine the im- pact of different components of the CO-MOT framework tracking performance. Through exper- imental analysis by combining various components on the DanceTrack (Sun et al. 2022) validation set, we achieve significant improvements over the baseline (61.8% vs 56.4%). By introducing the COLA strategy to the baseline (a), we observe an improvement of 3.8% on HOTA and 5.1% on AssA without any additional computational cost. By incorporating the concept of shadow into the baseline (a), HOTA is improved by 2.6% and AssA is improved by 3.6%. Also, to further illustrate the effectiveness of each component of our method, we also conducte ablation experiments on the MOT17 test dataset, as shown in the Table 4 As components added, the tracking performance improves gradually.
COLA. It is also evident from Table 4 that both COLA and Shadow have minimal impact on DetA (71.8% vs 73.5%), which is detection-relat
However, they have a significant impact on AssA (44.6% 52.2%) and HOTA (56.4% vs 61.8%). which
strongly related to tracking. On the surface, our method seems to help detection as it introduces more matching objects for detection, but it actually helps tracking.
To answer this question, we demonstrate the attention weights between detection and tracking queries in Figure The horizontal and vertical axes denote the attention weights after self-attention
8

---

Published as a conference paper at ICLR 2025
50
75
Attention weight Sc)
10
(2T(MMOR))
70
139MA4SG
HOTA(SA (%)
30
60
20
40MIT7G
MOTR(19
10
50
MOTR~2/14
CO-MCITU19 FFS)
45
0
100
200
300
400
500
5
6
#decoder
FLOPs(G)
Figure 3: The attention weights between different types of queries on different decoders.
Figure 4:
Efficiency comparison for CO- MOT and other end-to-end methods.
between different types of queries on different decoder layers. These weights roughly indicate the contribution one query to another. In our model, there are total of 6 decoder layers. T2T rep- resents the contribution of tracking query to itself. D2T represents the contribution of detection query predicting the same object to a tracking query. Two bounding boxes with an IOU greater than 0.7 are treated the same object. MD2T represents the average contribution of all detection queries to a specific tracking query. which serves as reference metric. D2T(MOTR) and T2T(MOTR) refer D2T and T2T in the MOTR model. Note that normalized attention weights are with sum of 1.
From Figure S it is evident that detection queries make a significant contribution (more than 15%) to their corresponding tracking queries in decoder layers where 7>> even greater than the T2T for #4 and #6 decoders and much higher than the MD2T for all the decoders. This indicates that detection queries pass on the rich semantic information they represent their corresponding tracking queries, which in turn can be utilized by the tracking queries to improve their tracking accuracy. Compared to CO-MOT, there is almost less information transfer (3.74% 30.74%) 44) between detection queries and tracking queries within MOTR. Because in MOTR, detection and tracking queries target different objects, resulting minimal information exchange, which straightforward to understand.
Table 5: Performance metrics of COLA inserting dif- ferent numbers of decoder layers on the DanceTrack validation set.
Table 6: Effect of initialization methods Shadow queries Im on the Dance'Trach validation set.
HOTA
DetA
AssA
MOTA
IDF1
Im
HOTA
DetA
AssA
== 0
59.0
72.9
48.1
81.2
59.8
Im Icopy 60.6
73.9
50.0
3
59.6
74.3
48.0
82.8
60.5
Im Inoise
61.5
73.1
51.9
1= 5
59.9
73.2
49.3
81.3
60.9
Im Trand
59.6
73.2
48.9
Furthermore, Table studies the performance impact of COLA inserting different decoder layers the DanceTrack Validation Set for 5 epochs without Shadow Set. 7 - 0 or mean that the first layer of the 6-layer decoder the first three layers COLA, and the other layers use TALA. It can be seen that deploying COLA in more decoder layers leads to better HOTA.
Table 7: Effeet of different À and ø combinations the DanceTrack validation set.
Table 8: Effect of initialization meth- ods for number of Shadows Nss on the DanceTrack validation set.
max
min min mean max
mean
min mean max
min mean max
Ns
HOTA
DetA
AssA
HOTA
57.6 56.4 55.1
56.7
55.2
52.0
57.5
55.9
51.5
Ns 2
61.5
73.1
51.9
DetA
70.7
69.3 65.4
70.6
66.5
59.0
70.8
66.4
59.3
Ns = 3
61.8
73.5
52.2
AssA
47.3 46.1 46.7
45.9
46.1
46.1
46.8
47.2
45.0
Nş =3
60.8
73.8
50.3

---

Published as a conference paper at ICLR 2025
(b)
Figure 5: Failed cases are often due to the failure to detect the target.
Shadow Set. Tables and list ablation experiments related to three hyperparameters of shadow, which are the number of shadows, initialization method of shadows, and representative sampling strategies and σ. To choose the appropriate option for À and Φ, we first set Ng and train the model only on the DanceTrack training set for 5 epochs using Laand without COLA. Then we try different combinations of and Φ. It can be seen from Table 7 that the combination of X = maz and é min yields the best results. That means we use the most challenging query in the set to train the model, leading to discriminative representation learning. To determine the initialization method, we also fix Ns 2 with COLA and find that the best results achieved using Lnosse- For Trand, there is a considerable variation between different shadows within the same set due to random initialization, making convergence difficult and resulting in inferior results. Finally, we try different values of Ns and find that the best results are achieved when Ns 3. When Ns is too large, we observe that convergence becomes more difficult, and the results deteriorate.
3.5 EFFICIENCY COMPARISON
In Figure efficiency comparisons on DanceTrack test dataset are made between CO-MOT and MOTR(v2). The horizontal axis represents FLOPs (G) and the vertical axis represents the HOTA metric. The size of the circles represents the number of parameters (M). It can be observed that our model achieves comparable HOTA (69.4% vs 69.9%) with MOTRv2 while maintaining similar FLOPs (173G) and number of (arA)eters(40M)) with MOTR. The runtime speed of CO-MOT is much faster (1.4x) than MOTRv2's. Thus, our approach is effective and efficient, which is friendly for deployment does not need an extra detector.
3.6 LIMITATIONS
Despite the introduction of COLA and Shadow, which improve the tracking effect of MOTR, the inherent data-hungry nature of the Transformer model means that there not a significant improve- ment in smaller datasets like MOT17. As shown in Figure 5a prominently visible target has not been detected, but this issue has only been observed in the small MOT17 dataset. And due to the scale problem, the detection and tracking performance is poor for small and difficult targets in Figure 5b In order to further improve the effect, is necessary to increase the amount of training data or use more powerful baseline such DINO.
CONCLUSION
This paper proposes method called CO-MOT to boost the performance of end-to-end Transformer- based MOT. We investigate the issues in the existing end-to-end MOT using Transformer and find that the label assignment can not fully explore the detection queries as detection and tracking queries are exclusive to each other. Thus, we introduce a coopetition alternative for training the interme- diate decoders. Also, we develop a shadow set as units augment the queries, mitigating the unbalanced training caused by the one-to-one matching strategy. Experimental results show that CO-MOT achieves significant performance gains on multiple datasets in an efficient manner. We believe that our method as a plugin significantly facilitates the research of end-to-end MOT using Transformer.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr. Fully- convolutional siamese networks for object tracking. ECCV, pp. 550-865. Springer, 2016.
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Upcroft. Simple online and realtime tracking. In ICIP. IEEE, 2016.
Jinkun Cao, Jiangmiao Pang, Xinshuo Weng, Rawal Khirodkar, and Kris Kitani. Observation-centric sort: Rethinking sort for robust multi-object tracking. In CVPR, pp. 9686-9696, 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, and Alexander Kirillov. End- to-end object detection with transformers. In ECCV, pp. 213-229, Springer, 2020.
Orcun Cetintas, Guillem Brasó, and Laura Leal-Taixé. Unifying short and long-term tracking with graph hierarchies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22877-22887, 2023.
Qiang Chen, Xiaokang Chen, Jian Wang. Shan Zhang, et al. Group detr: Fast detr training with group-wise one-to-many assignment. In ICCV, 2023.
Yunhao Du, Zhicheng Zhao, Yang Song, Yanyun Zhao, Fei Su, Tao Gong, and Hongying Meng. Strongsort: Make deepsort great again. TMM, 2023.
Luiten et al. Hota: A higher order metric for evaluating multi-object tracking. IJC, 129(2):
548-578, 2020.
Tobias Fischer, Jiangmiao Pang, Thomas E Huang, Linlu Qiu, Haofeng Chen, Trevor Darrell, and Fisher Yu. Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking. arXiv preprint arXiv:2210.0690 2022.
Teng Fu, Xiaocong Wang, Haiyang Yu, Ke Niu, Bin Li, and Xiangyang Xue. Denoising- mot:
Towards multiple object tracking with severe occlusions. In MM, pp. 2734-2743, 2023.
Ruopeng Gao and Limin Wang. Memotr: Long-term memory-augmented transformer for multi-
object tracking. In ICCV, pp. 9901-9910, 2023.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430, 2021.
Bin Hu, Run Luo, Zelin Liu, Cheng Wang, and Wenyu Liu. Trackssm: general motion predictor by state-space model. arXiv preprint arXhv:2409.00487 2024.
Hsiang-Wei Huang, Cheng-Yer Yen Yang, Wenhao Chai, Zhongyu Jiang, and Jenq-Neng Hwang. Ex- ploring learning-basee motion models in multi-object tracking. arXiv preprint arXiv:2403.10826 2024.
Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. arXiv preprint 2rX.v:2207. 13080, 2023.
Hyeonchul Jung, Seokjun Kang, Takgen Kim, and HyeongKi Kim. Conftrack: Kalman filter- based multi-person tracking by utilizing confidence score of detection box. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6583-6592, 2024.
Harold W Kuhn. The hungarian method for the assignment problem. NRL, 2(1 1955.
Stéphanie Lefevre, Dizan Vasquez, and Christian Laugier. A survey motion prediction and risk assessment for intelligent vehicles. ROBOMECH Journal, 1(1):1-14, 2014.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M. Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. arXiv preprint m:Xiv:2203.03300 2022a.
Siyuan Li, Martin Danelljan, Henghui Ding, Thomas E Huang, and Fisher Yu. Tracking every thing in the wild. In ECCV, pp. 498-515, Springer, 2022b.

---

Published as a conference paper at ICLR 2025
Wei Li. Xiatian Zhu, and Shaogang Gong. identification. In CVPR, pp. 2285-2294, 2018.
Harmonious attention network for person
Chongwei Liu, Haojie Li. Zhihui Wang, and Rui Xu. Putr: A pure transformer for decoupled and online multi-object tracking. arXiv preprint arXiv:2405. 2024.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi. Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In ICLR, 2022.
Run Luo, Zikai Song, Lintao Ma, Jinlin Wei, Wei Yang, and Min Yang. Diffusiontrack: Diffusion
model for multi- tracking. In Proceedings of the AAAI Conference on Artificial Intelligence, 38(5), pp. 3991-3999, 2024.
Weiyi Lv, Yuhang Huang, Ning Zhang, Ruei-Sung Lin, Mei Han, and Dan Zeng. Diffimot: real- time diffusion-based multiple object tracker with non-linear prediction. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19321-19330, 2024.
Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi-object tracking with transformers. In CVPR, June 2022.
Anton Milan, Laura Leal- Taixe. Ian Reid, Stefan Roth, and Konrad Schindler. Mot16; A benchmark for multi-object tracking. arXiv preprint 160i 00831, 2016.
Zheng Qin, Le Wang, Sanping Zhou, Panpan Fu, Gang Hua, and Wei Tang. Towards generalizable multi-object tracking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18995-19004, 2024.
Hao Ren, Shoudong Han, Huilin Ding, Ziwen Zhang, Hongwei Wang, and Faquan Wang. Focus on details: Online multi-object tracking with diverse fine-grained representation. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 11289-11 11298, 2023.
Mattia Segu, Luigi Piccinelli, Siyuan Li, Lue Van Gool, Fisher Yu, and Bernt Schicle. Walker: Self- supervised multiple object tracking by walking on temporal appearance graphs. arXiv preprint arXiv:2409.17221. 2024.
Jenny Seidenschwarz, Guillem Brasó, Victor Castro Serrano, Ismail Elezi, and Laura Leal-Taixé. Simple cues lead strong multi-ohject tracker. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13813-13823, 2023.
Peize Sun, Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple object tracking with transformer. arXiv preprint arXiv:2012. 15460 2020.
Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In CVPR, pp. 20993-21002, 2022.
Zhongdao Wang, Liang Zheng, Yixuan Liu, Yali Li, and Shengjin Wang. Towards real-time multi- object tracking. In ECCV, 2020.
Greg Welch, Gary Bishop, al. An introduction the kalman filter. Chapel Hill, NC, USA, 1995.
Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with deep association metric. In ICIP, 2017.
Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang, Ming Yang, and Junsong Yuan. Track to detect and segment: An online multi-object tracker. In CVPR, pp. 12352-12361. 2021.
Changcheng Xiao, Qiong Cao, Zhigang Luo, and Long Lan. Mambatrack: simple baseline for multiple object tracking with state space model. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 4082 4091, 2024a.
Changcheng Xiao, Qiong Cao, Yujie Zhong, Long Lan, Xiang Zhang, Zhigang Luo, and Dacheng Tao. Motiontrack: Learning motion predictor for multiple object tracking, 2024b. URL hetps:
//arxi wttp://gggabbb..aeeee 02585
12

---

Published as a conference paper at ICLR 2025
Feng Yan, Zhiheng Li, Weixin Luo, Fan Liang, Xiaolin Wei, and Lin Ma. Multiple object tracking challenge technical report for team mtt. arXiv preprint arXiv:2212.03586, 2022.
Fan Yang, Shigeyuki Odashima, Shoichi Masui, and Shan Jiang. Hard to track objects with irregular motions and similar appearances? make it easier by buffering the matching space. In WACV, pp. 4799-4808, 2023.
Mingzhan Yang. Guangxin Han, Bin Yan, Wenhua Zhang, Jinqing Qi, Huchuan Lu, and Dong Wang. Hybrid-sort: Weak cues matter for online multi-object tracking. Proceedings of the AAAI Conference Artificial Intelligence, 38(7), pp. 6504-6512, 2024.
Mang Ye, Shuoyi Chen, Chenyue Li, Wei-Shi Zheng, David Crandall, and Bo Du Transformer for object re-identification: A survey, arXiv preprint arXiv:240..0696) 2024.
Kefu Yi, Kai Luo, Xiaolei Luo, Jiangui Huang, Hao Wu, Rongdong Hu, and Wei Hao. Uemetrack: Multi-object tracking with uniform camera motion compensation. In Proceedings of the AAAI Conference on Artificial Intelligence, 38(7). pp. 3702-6710, 2024.
En Yu, Tiancai Wang, Zhuoling Li, Yuang Zhang, Xiangyu Zhang, and Wenbing Tao. Motrv3: Release-fetch supervision for end-to-end multi-object tracking. arXiv preprint arXiv:2305. 14298, 2023.
Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, al. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In CVPR, pp. 2636-2645, 2020.
Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-objec tracking with transformer. ECCV, 2022.
Hao Zhang, Feng Li, Shilong Liu, and Lei Zhang. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2233.03003. 2022a.
Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairness of detection and re-identification in multiple object tracking. IJV,, (9311))30693087, 2021. ISSN 1573-1405.
Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, al. Bytetrack: Multi-object tracking by associat- ing every detection box. arXiv preprint arXiv:2110.06864 2022b.
Yuang Zhang, Tiancai Wang, and Xiangyu Zhang. Motry2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors. CVPR, 2023.
Yian Zhao, Wenyu Lv, Shangliang Xu, Jinman Wei, Guanzhong Wang, Qingqing Dang, Yi Liu, and Jie Chen. Detrs beat yolos real-time object detection. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16965-16974, 2024.
Liang Zheng, Yi Yang, and Alexander G Hauptmann. Person re-identification: Past, present and future. arXiv preprint orXiv:/610.022
2016.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. ECCV, 2020.
Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähenbühl. Global tracking transformers. In CVPR, pp. 8771-8780, 2022.
Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. arXiv preprint arXiv:2211.12866 2023.
Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: survey, Proc. IEEE, 2023.
13


Reviews:
[{"review_id": "R3OvbA2XMI", "reviewer": "Reviewer_rzmb", "summary": "The paper tackles end-to-end Transformer-based multiple-object tracking. Previous methods, such as TrackFormer and MOTR, face issues with imbalanced distribution in detection and tracking label assignments, where most objects are assigned to track queries, leaving only a few “newborns” for detection queries. This joint training approach results in weaker detection performance compared to tracking-by-detection methods. To resolve this, the paper proposes a coopetition label assignment strategy to re-balance assignments between track and detection queries. Additionally, it introduces a shadow set that changes the original one-to-one mapping in DETR to a one-to-set mapping, further enhancing tracking performance. Results on various benchmarks demonstrate the effectiveness of this method.​", "questions": "The main experiments are still concentrated on small-scale pedestrian tracking datasets. As mentioned on weakness, for other scenarios, we may face different difficulties. Are there any plans to test the model also on large-scale MOT datasets such as TAO [3]?\n\n[3] Dave, Achal, et al. \"Tao: A large-scale benchmark for tracking any object.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written, with a clear and logical structure that makes it very easy to follow.\n\n2. The observation on the disproportional assignment of track and detection queries is insightful, highlighting an important yet often overlooked issue in transformer-based MOT. This analysis provides valuable context for the community.\n\n3. The proposed coopetition label assignment strategy is simple and effective. The paper also demonstrates its effectiveness on multiple Transformer-based MOT frameworks, including TrackFormer and MOTR.\n\n4. The experiments are thorough, covering multiple benchmarks, including more large-scale autonomous driving scenes such as BDD100K, and demonstrating the method’s robustness and practical impact.", "weaknesses": "1. To solve the issue of disproportional assignment of track and detection queries, there are also other simpler alternatives. A straightforward option would be to train detection queries jointly on image detection datasets alongside video tracking datasets. For example, detection queries could be trained exclusively on image datasets, treating every object as a new object. An ablation study comparing proposed methods to this simple joint training alternative is appreciated.\n\n2. The paper uses the first 5 decoders to train with all queries, while the last one trains separately on detection and tracking queries. An ablation study could clarify whether a different arrangement, such as using the first decoder for track queries and the last five for all queries, would impact performance. An ablation study regarding this would be helpful for readers to understand the optimal configuration.\n\n3. The applicability of the coopetition label assignment strategy is mostly limited to cases where there is more video data than image data for training, leading to an imbalance in track and detection query assignments. However, in many practical settings, the opposite is true—large-scale [1] and open-vocabulary MOT tasks [2] often have substantially more image detection data than video tracking data. In these cases, common practice in MOT is to use joint training with both image and tracking data, which provides sufficient supervision for detection queries. This is contrary to the paper’s analysis, and it would be beneficial for the authors to also at least discuss these more common scenarios.\n\n[1] Li, Siyuan, et al. \"Matching Anything by Segmenting Anything.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Li, Siyuan, et al. \"Ovtrack: Open-vocabulary multiple object tracking.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WxhCpMIhZG", "reviewer": "Reviewer_d8sy", "summary": "This paper addresses the limitations of existing e2e-MOT methods, particularly the unbalanced training issue caused by the label assignment strategy. And it introduces a Coopetition Label Assignment (COLA) strategy and a Shadow Set concept. Through extensive experiments on multiple datasets, it demonstrates superior performance compared to state-of-the-art methods while being more efficient.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The analysis about the drawbacks of exist e2e-trackers is very interesting, it reveals the negative impact of tracking queries on detection queries.\n\n2. The proposed COLA strategy allows tracked objects to be reassigned to detection queries in decoders, resulting in a significant improvement in tracking performance.", "weaknesses": "1. From the public records of OpenReview, it can be seen that this paper was submitted to ICLR2024. The reviewers and AC pointed out many weaknesses last year, while authors have made almost no improvements in the latest version. \n\n2. Many SOTA trackers developed in this year have been overlooked by authors, such as DiffMOT, MambaTrack, TrackSSM, et al. These new methods have made many improvements, and it would be best for the author to provide a comparison with the latest methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "69qjtyrQxX", "reviewer": "Reviewer_mbwq", "summary": "The paper presents an innovative end-to-end Multi-Object Tracking (MOT) framework that aims to enhance transformer-based MOT models. The authors introduce two key contributions: 1. Coopetition Label Assignment (COLA) revises label assignment by allowing detection queries to utilize tracked objects during training in intermediate decoders. This approach boosts feature augmentation for tracking objects with diverse appearances and alleviates the issue of tracking termination. Shadow Set Strategy aims to address training imbalance in one-to-one matching, CO-MOT introduces \"shadow sets,\" which add slight disturbances to each query, thus allowing one-to-set matching. This enhances the discriminative training process and the model's generalization.The proposed method outperforms existing e2e-MOT models on benchmarks like DanceTrack and BDD100K with improved tracking metrics such as HOTA and TETA, demonstrating higher efficiency and inference speed.", "questions": "I have listed my concerns and questions in Weakness part and hope the response from authors.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper's strengths lie in its originality, quality, clarity, and significance. It introduces a novel coopetition-based label assignment (COLA) and shadow sets for one-to-set matching, enhancing the robustness of e2e-MOT without requiring additional detectors. The evaluation across multiple datasets, including ablation studies, demonstrates the effectiveness of CO-MOT, establishing its advantages over state-of-the-art models. The paper is well-structured, with clear explanations and helpful visualizations, although further clarification on certain technical aspects could enhance understanding. Overall, CO-MOT significantly improves the efficiency and performance of transformer-based multi-object tracking, making it a valuable contribution to the field.", "weaknesses": "While the paper presents valuable contributions, several weaknesses could be addressed to strengthen its impact and clarity:\n1. The authors should provide a detailed discussion of the differences between COLA and TALA in Section 2.4, as well as their design in the loss function, to facilitate reader understanding. \n\n2. In the experiments section, the authors need to include comparisons with more methods on the MOT20 and BDD100K datasets.\n\n3. Since the authors analyze the impact of tracking queries on detection performance in transformer-based trackers, if this point serves as one of the motivations, they should compare whether the proposed framework shows improvement in mAP in the experiments.\n\n4. The authors should also analyze the effects of different values of $\\lambda$ and $\\Phi$ in Section 2.5 on the experimental outcomes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lUtSB1xEbI", "reviewer": "Reviewer_Akzq", "summary": "This paper introduces CO-MOT, a novel method aimed at improving end-to-end Transformer-based multi-object tracking (e2e-MOT) through a new coopetition label assignment strategy (COLA) and the introduction of shadow sets. The authors address the issue of unbalanced training in existing e2e-MOT methods, where detection queries often lack positive samples, particularly for newborn objects.", "questions": "- Could the authors provide a detailed analysis of the cost brought by shadow sets?\n- Could the authors provide the evaluation and discussion on MOT20.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- The motivation of the paper is interesting and clearly demonstrated by the experiments.\n\n- The introduction of COLA and shadow sets do mitigate the biased label assignment issue in e2e-MOT. This proposed approach provides a balanced training process, leading to improved performance.\n\n- The authors conduct experiments on three datasets, demonstrating the effectiveness of CO-MOT across different scenarios. The results are robust and consistent, showing improvements over the baseline.", "weaknesses": "- The introduction of shadow sets and the coopetition label assignment strategy may increase the computational complexity and training time. The authors should provide a detailed analysis of the computational overhead and discuss potential optimizations. Notably, Fig.4 in the manuscript only presents the Flops, which is not the actual training and inference time. Intuitively, more object queries would bring larger computation costs. Why do shadow sets not?\n\n\n- Although the proposed method demonstrates strong performance on the tested datasets, it would be advantageous to evaluate CO-MOT on MOT20. The authors assert that the proposed approach enhances detection recall. Thus, the more densely populated nature of MOT20 provides a more suitable context for assessing the effectiveness of the model.\n\n- The authors should investigate the sensitivity of the proposed method to hyperparameters, such as the number of shadow sets and the parameters of the coopetition label assignment strategy. Understanding how these hyperparameters affect performance would provide valuable insights for practical implementation.\n\n- The writing should be improved. For instance, in Fig.3 and Fig.4, the axis titles are overlapped with the axis. The readability of Figures 3 and 4 could be improved by adjusting the axis labels to avoid overlap. This would enhance the overall presentation of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)