SynCL: A Synergistic Training Strategy with Instance-Aware Contrastive Learning for End-to-End Multi-Camera 3D Tracking
Authors: Shubo Lin
Yutong Kou
Zirui Wu
Shaoru Wang
Bing Li
Weiming Hu
Jin Gao
Venue: NeurIPS 2025 poster ()
Keywords: Autonomous Driving
Multi-Camera 3D perception
Multi-Camera 3D Tracking
Primary area: applications

Abstract: While existing query-based 3D end-to-end visual trackers integrate detection and tracking via the *tracking-by-attention* paradigm, these two chicken-and-egg tasks encounter optimization difficulties when sharing the same parameters. Our findings reveal that these difficulties arise due to two inherent constraints on the self-attention mechanism, i.e., over-deduplication for object queries and self-centric attention for track queries. In contrast, removing self-attention mechanism not only minimally impacts regression predictions of the tracker, but also tends to generate more latent candidate boxes. Based on these analyses, we present SynCL, a novel plug-and-play synergistic training strategy designed to co-facilitate multi-task learning for detection and tracking. Specifically, we propose a Task-specific Hybrid Matching module for a weight-shared cross-attention-based decoder that matches the targets of track queries with multiple object queries to exploit promising candidates overlooked by the self-attention mechanism and the bipartite matching. To flexibly select optimal candidates for the one-to-many matching, we also design a Dynamic Query Filtering module controlled by model training status. Moreover, we introduce Instance-aware Contrastive Learning to break through the barrier of self-centric attention for track queries, effectively bridging the gap between detection and tracking. Without additional inference costs, SynCL consistently delivers improvements in various benchmarks and achieves state-of-the-art performance with $58.9\%$ AMOTA on the nuScenes dataset. Code and raw results are available at <https://github.com/shubolin028/SynCL>.

SynCL: A Synergistic Training Strategy with Instance-Aware Contrastive Learning for End-to-End Multi-Camera 3D Tracking
Shubo Lin 1.233 Yutong Kou 1.233 Zirui Wu Shaoru Wang' Bing Lia.3,5, Weiming 222.2,3,4 Jin 1..3. State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA
School of Artificial Intelligence, University of Chinese Academy of Sciences
3 Beijing Key Laboratory of Super Intelligent Security of Multi-Modal Information School of Information Science and Technology, Shanghai Tech University 'People AI, Inc, Harbin Institute of Technology (1inshubo2023) ac en, 2anng ia.iaa..a.. ac. cn
Abstract
While existing query-based 3D end-to-end visual trackers integrate detection and tracking via the tracking-by-atie by-attention paradigm, these two chicken-and-ege tasks encounter optimization difficulties when sharing the same parameters. Our findings reveal that these difficulties arise due to two inherent constraints on the self- attention mechanism, i.e., over-deduplication for object queries and self-centric attention for track queries. In contrast, removing the self-attention mechanism not only minimally impacts regression predictions of the tracker, but also tends to generate more latent candidate boxes. Based on these analyses, we present SynCL, a novel plug-and-play synergistic training strategy designed to co-facilitate multi-task learning for detection and tracking. Specifically, propose Task- specific Hybrid Matching module for a weight-shared pross-attentiin-beee based decoder that matches the targets of track queries with multiple object queries exploit promising candidates overlooked by the self-attention mechanism and the bipartite matching. To flexibly select optimal candidates for the one-to-man -tatmmyyyy matching, we also design a Dynamic Query Filtering module controlled by model training status. Moreover, we introduce Instance-aware Contrastive Learning to break through the barrier of self-centric attention for track queries, effectively bridging the gap between detection and tracking. Without additional inference costs, SynCI. consistently delivers improvements in various benchmarks and achieves state-of- the-art performance with 58.9% AMOTA on the nuScenes dataset, Code and raw results are available at htttt:::/ thub. com/ chubolin028//55 /SynCL
Introduction
The perception system is indispensable component in autonomous driving. Within the system, accurate 3D multi-object tracking (MOT) provides reliable observations for planning. Camera-based tracking algorithms have raised significant attention due their cost-effectiveness [I]. Traditional trackers following the tracking- by-detection paradigm often suffer from tedious and unstable post- processing in new scenarios 3] To achieve cross-domain adaptability, the prevailing methods [4/ adopt the tracking by encoding new-born and persistent targets as object queries and track queries respectively
paradigm, which integrates single-frame detection and inter-temporal
"Corresoondig author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Object Queties
Quadies
Updat
Muli-Va
$ & 0
Kals-Vin View BEY Features
Staddard Decader
BEV Fostures
1844 db
New-ar Tradiie
Stare
Cross-ataelior
Parallal 6
Stawfard cocader
Decoder
ContarTin (ETEN))
Tracking-by-atteeline paradigm (T3A)
(b) proposed training strategy (SyaCL)
Figure I: Comparisons between tracking- attentioo paradigm and our proposed plug-and -nd--aad training strategy, SynCL. SynCL consists of Task speciffe Hybrid Matching, Instance-aware Contrastive Learning powered by Dynamic Query Filtering module.
Dention score
objject query
trakk query
Grunn Objeect query
Track query
Inference stage
Inference stage
ID Bi
ba
Self altention
(a) ttention
Standard decader Self and Cross A ttention
(c) Remove Self-attemtion mechanism in Standard decoder
We compare results from the standard decoder in Fig. 2 with those from the decoder without self-attention Fig. 2 in inference stage. The self-attention mechanism exhibits over-deduplication for object queries and self-centric attention for track queries. Results are from the model trained without utilizing SynCI.
Figure 2:
(Fig. Ta). However, the optimization of detection and tracking, two interdependent yet distinctly characterized tasks, remains an open question due to the sharing of the same model parameters within the end-to-end training pipeline (9 [0]. Previous works attribute the optimization difficulties to conflicts representation (6] and gradient flows across distinct queries [ DQTrack [I] attempts to train detection and barning-based association separately, but this two-stage training pipeline fails to enable synergistic representation learning. ADATrack [6] introduces an additional attention- based association module to strengthen inter-query connections, yet consequently increases model complexity. One OneTTrack [8] groups object queries and blocks conflicting gradient flows, but its design is confined to specific detector, thus lacking generalizability. A question is: Is there universal training strategy that can effectively address the optimization difficulties of detection and tracking under shared model parameters through a joint training paradigm without affecting inference speed?
To seek the answer, we first conduct a comprehensive analysis of attention mechanisms in the paradigm. We find that the self-attention mechanism exhibits two inconspicuous characteristics for object queries and track queries, respectively:
Over-deduplication for object queries: Referring to the purple dashed box in Fig. 2h. the self- attention mechanism in the decoder assigns high attention scores from object queries to similar track queries. Compared to the prediction results before and after removing the self-attention mechanism (see purple circles in Fig. 2p and Fig. 2 we observe that these high attention scores effectively suppress duplicate predictions (e.g.. .. ID O. ID 1 and ID 4). Although this de- de-duplication plays similar role of -ox-mmmmmum Suppression (NMS) some extent, inadvertently results in information loss, potentially eliminating some high-quality candidates (e.g.. ID 2).
Self-centric attention for track queries: Referring to the green dashed box in Fig. 2.. track queries exhibit high attention scores towards themselves, lacking interaction with object queries. As illustrated in the green circles of Fig. 2 and Fig. P., the imprecise predictions of track queries (e.g., ID 6, ID 8, etc.) change little before and after the removal of the self-attention mechanism.
Despite the above issues, directly replacing the self-attention mechanism with NMS will lead to the collapse of joint detection and tracking training. In another way, optimization difficulties may be mitigated by a weight-shared cross-atte www---atettioon-aaeeeee parallel decoder, following [12]. However, simultaneously addressing the over-deduplication of object queries and the self-centric issues of track queries within a parallel decoder remains non-trivial (see Appendix B.3 for more details).
To thoroughly tackle these challenges, propose a novel synergistic training strategy for the two tasks based on the macking-by-attentie paradigm, which we term SynCI.. As presented in Fig. Пp. SynCL introduces weight-shared parallel decoder during training, which removes the

---

paradigm multi-view 3D tracker. self-attention mechanism. Concretely, we propose Task-specific Hybrid Matching module that additionally employs one-to-many assignment for object queries to uncover and preserve promising candidates overlooked by the self-attention mechanism in the parallel decoder. With more diverse and refined candidate representations from one-to-many label supervision, predictions from object queries are robust to over-deduplication. Additionally, we design a Dynamic Query Filtering module based on the Gaussian Mixture Model (GMM) that flexibly selects reliable object queries for the one-to-many assignment according to the prediction quality and optimization state of the model. Moreover, to get rid of constraints from self-centric attention on track queries and establish cross-task connections, introduce Instance-aware Contrastive Learning that aligns joint object and track queries corresponding to identical ground truth targets while separating irrelevant queries in the latent space and thus provides synergistic learning with richer and quality features.
In summary, our main contributions are as follows: (1) We reveal that the optimization difficulties of joint detection and tracking training lie in the imperceptible effect of the self-attention mechanism across different queries. (2) We propose SynCI,, a synergistic training strategy compatible with any tracking-byy cestiing-b-atteention paradigm tracker to address multi-task learning challenges. SynCL implements dynamic Eltering-based hybrid matching and instance-aware contrastive learning to improve the performance of both detection and tracking. (3) We empirically show that SynCL brings remarkable improvements over various tracking- torkkingbb-aateen by-attention baselines with acceptable extra training cost and establishes new state-of- offte-ofttute the-art performance on the camera-based nuScenes MOT benchmark.
2
Related Work
Tracking--y-ddcecctonn Traditional tracking methods in 2D or 3D 22 3 13 14] follow the bauckin-by-detecttoon paradigm, which decouples detection and tracking, regarding tracking as a post-processing step. Early works associate targets with existing trajectories by handcrafted features such as appearance similarity M. geometric distance [1] maucking-by-detcction paradigm achieves further advancements when combined with learning-basce based association methods. Specif- ically, learning modules such as GNN [19] 20) and Transformers [1] are employed to merge multifaceted information. However, detection and tracking are complementary yet non-independent tasks. This partitioned paradigm design does not effectively optimize both tasks simultaneously.
Multi-task learning in MOT. Building upon DETR [22] of encoding detection targets as object queries, MOTR [23] utilizes the track queries, enriched with prior features of object queries, to
propagate across frames, enabling an end-to-end tracking framework. MUTR3D [4] and PF-Track (5]
extend the end-to-end framework multi-camera 3D MOT. However, sharing the same model param- eters for detection and tracking within an end-to-end framework introduces optimization difficulties in multi-task learning. Subsequent works [IIII attempt to address these difficulties. STAR-Trace [24] introduces a Latent Motion Model (LMM) to account for changes in query appearance features across frames and ADA-Track 6 integrates the appearance and geometric clues of the track queries to refine object queries. Both of them increase inference time. TQDTrack [25] introduces temporal query denoising enhance the modeling of track queries. Trackrac// X utilizes attention mask to
block conflicting gradient flows, but its design is confined to [26). Despite these efforts, a general strategy for optimization difficulties in joint detection and tracking training remains unsolved.
3 Methodology
This section elaborates on our proposed method, SynCL, for multi-view 3D object tracking. We start with an overview of the constraints the self-attene attentie mechanism in Sec. 3.2 Then, provide detailed construction of hybrid matching for the parallel decoder in Sec. 3.3 and introduce the filtering module design for one-to-mas Many assignment in Sec. 3.4 To achieve synergistic end-to-end learning for detection and tracking, we further propose instance-aware learning in Sec. 3.5
paradigm trackers in Sec..3.1 and an analysis to validate
3.1 Preliminaries
SynCL is designed to be integrated into any These trackers typically consist of convolutional network-based feature encoder (e.g., ResNet (27]. Vo VNet (28)), transformer- decoder, and predictor head for object classes and boxes. The
3

---

goal of 3D MOT is to generate detection boxes of the instances in the perceptual scene with consistent IDs. Given N images
{I, i 1.2,... .......... N captured by surrounding cameras timestamp the encoder first extracts image features F, { i 2,2,... N with 3D point position embeddings [29] 3D] crucial aspect of realizing end-to-end tracking to allow the prior latent encoding from the previous frame, track queries Q to be updated iteratively by the decoder
interacting with the current frame image features. To detect new-born targets, object queries Q with a fixed number of Nobj are merged together and fed into the decoder:
Decoder Q )
(I)
Each query g E {Qtrk, Qobi includes an embedding vector assigned with a unique 3D reference i.e., y = {e, e). the center-point anchor-free predictor, the reference points are actively involved in the parsing of the bounding box:
B, Box(ei, P. Classification(e))
(2)
where B, = (b", D2, 19,,.. ....; bn) } is the of prediction 3D boxes and P, = {p² xp. ..... ;..,,p')) pd] is the set of classification scores for all categories (N,). Each box (b) consists of 3D center point (c E R9), 3D box size (s R²), yaw angle (0 1, and BEV velocity (u IRE
Considering the heterogeneity the two tasks, object queries and track queries are supervised with different label assignment strategies. Object queries are precisely matched with new-born targets via the Hungarian algorithm and assigned unique IDs, while track queries are directly mapped to persistent targets with consistent IDs.
However, matching object queries to new-born objects contradicts the detection pre-training, substan- tially weakening the supervision signals for object queries. Consequently, when track queries are initialized with object queries from the previous frame, these low-quality priors inevitably exacerbate the optimization difficulties in inter-temporal tracking. We find that this suboptimal assignment stems from the inconspicuous nature of the self-attention mechanism and the bipartite matching, preventing synergistic training for both identity-agnosti detection and identity- tracking.
3.2 Analysis
To further validate our findings in Fig. we vi- sualize the self-attention distributions from shal- low to deep decoder layers. As shown in Fig. [3] over-deduplication stems from the attention of object queries to track queries, which evolves from global non-differentiated responses to lo- cal similarity responses. For track queries, self- centric attention prevents them from collabora- tive learning with object queries. To this end, we separate the self-attention mechanism and re- solve these issues through a wwvide-nnd-conquee method in the reighttssaree parallel decoder.
Sballow
Deep
0.0
1035
9 1031
30
Dendinn
Deadton
Tinck
Figure 3: Analysis of self-attention heatmap the stan- dard decoder. The annotation numbers of the heatmap are aligned with the ID numbers in Fig.
3.3 Task-specific Hybrid Matching
Based on the regression characteristics of cross-attention, we first construct weight-shared parallel decoder with the removal of the self-attention mechanism. To further enhance the inter-temporal modeling capability and exploit promising candidates, we implement one-to-one and one-to-many label assignment for track queries and object queries, respectively.
Parallel Decoders. We design two parallel decoders: standard decoder (S-decoder) and cross- attention-bate based decoder (C (C-decoder). A block in the standard decoder consists of a self-attention layer, cross-attention layer and beed-forward network (FFN). The embeddings of both object and track queries are concatenated and fed into the two decoders, denoted as el, and e/, where el = e' U e The superscript = {0, 1,..,, 5) indicates the index of the decoder block and we omit the notation of the frame index for simplicity. The initial inputs to both decoders are identical, i.e., el = eg. In the 4h block of the S-decoder, the state of both queries is first updated through the self-attention layer

---

Lx
Lunal
Quecina
Qyaei
Trxd Quario
Oljad
G D
Frann Liaetian
BEY Syace
Ettoxt
Parallel Decoder
Standard Decoder
Trpedian
Soptessoo
flox and C beafivation
Cod Basel Quary
Trak-geeffic Hyhrit Matching
Indontteeteaee Contradice curning
Filteinng
Aggregation
Meiling for Doladias
Maccsay for Tankay
FFN
Hadamard product
Disniartee pasking Oraea Soolald
Fannatra
B2
Allisty
68
pit
Doe
Guzsdan Mitraro Modd
Object Quaties
Track Quetes
Lane
Figure 4: Overview of SynCL. SynCL is based on maicking-byaatteniine paradigm trackers, with two weight- shared parallel decoders: a S-decoder (standard decoder) and a C-decoder (devoid of self-attention layers). In C-decoder, hybrid matching with onc-to-many and one-to-one assignment is applicd for object queries and track queries, respectively. Besides, dynamic filtering module is designed to flexibly select reliable object queries for one-to-many assignment. With identical ground-trute matching, contrastive learning unifies representations between object and track queries, co-facilitating multi-task learning for detection and tracking.
bidirectionally, which is formulated as:
= + Atlention = Key = Value e
where Attention (-) indicates the self-attention layer. Next, the two types of queries are interacted with multi-view image features, where object queries generate potential candidates through the positional priors, and track queries locate persistent targets utilizing representation prior:
J+1 FFN(E + Attention (Query = é Key Value
F))
(4)
where Attentioner indicates the cross- tross-attention layer. The C-decoder retains the coss-attention layers and FFNs, sharing parameters with the S-decoder. In this setup. the cross-attentic layers take image features as the sole source (key and value), dynamically aggregating and updating both queries in the stacked decoder blocks:
++1 FFN + Attentioneroas Prrenttioeoos(uueryy e/y, Key Value F))
(5)
Due to the absence of mutual interaction between object and track queries, it is challenging for the C-decoder to remove duplicate candidates with proximal occluded targets. Thus, we design distinct label assignment tailored to the characteristics of the two parallel decoders and the specific requirements their respective tasks.
Hybrid Matching, End-to-end tracking-by- by-tttentonnn paradigm trackers rely on one-to-one assign- ment for both object and track queries. In the S-decoder, assignment of object queries aghi can be obtained by performing the Hungarian algorithm between predictions and new-born targets, while assignment of track queries ofr is consistent with the assigned IDs, which is formulated
obj arg min
Σ
Cmatch (Bot) obj (bot)) Po Shew)
(6)
2m
trk fo IDirk ID,, ID gpst, i = 1,2,.....N!
(7)
where the Inew and 9pst are M new-born and N persistent ground truths (GTs). Cmatch is the matching cost between predictions and GTs. (-)) is the optimal permutation of M N indices, where kιs, ground truth target is assigned to okk) prediction. Each query corresponds to a ground- truth target and vice versa. The track queries corresponding to disappearing targets are not assigned labels. Following [3 we utilize a combination of the classification score and the 3D box as the matching cost metrie, which is formulated
Cmatch (b.p. p. FocalLoss(p,)) L1(b, 6)
(8)
5

---

In the C-decoder, we maintain the Slentity-guidee principle for track queries to avoid ambiguity. However, lacking the NMS effect of self-attention object queries in C-decoder tend to output multiple similarity predictions. To improve the quality of candidate boxes and boost the detection of lost targets, we perform the one-to-many assignment for object queries in C-decoder with persistent GTs, i.e,, one ground-truth target corresponds to multiple object queries:
K
obj
arg min
Cmatch 3=1
obs (ool (o. Por Spst
i=1
where the K is the maximum assignment constraint per GT for object queries. In order to obtain more positive sample pairs for contrastive learning, we do not match object queries with all ground-truth targets. In Sec. 4.3
we explore the impact of different matching variants in detail.
3.4 Dynamic Query Filtering
To effectively explore more high-quality candidate samples for the ddentity-agnossie matching in Hybrid Matching, we propose a cost-based filtering module with a dynamic threshold which filters out unreliable matching pairs whose cost exceeds ò. Specifically, we use Gaussian Mixture Model (GMM) to unsupervisedly cluster matching sample pairs into reliable/unreliabee clusters based on their matching cost 2, which is formulated as:
π,, pr, Σ arg min
- log
NeN(,, E,)
(10)
where ((= (us, Σ.) is the Gaussian probability density function with mean μi and covariance E., and n; is the weight for the i-th component, satisfying Σ म; = 1. The cluster with lower is considered as the reliable cluster and we use its mean Mreliable min; μ4 as the dynamic threshold 8. Notably, this dynamic design intelligently constructs progressive learning schedule ranging from simple to complex depending on the optimization status of the model.
3.5 Instance-aware Contrastive Learning
To further transfer object queries' knowledge persistent targets learned by one-to-many assignment in Hybrid Matching to track queries, we employ contrastive learning for object queries and their corresponding track queries with identical ground-truth matching. As shown in Fig. [4] contrastive learning pulls representations of object queries closer to those of track queries assigned to the same instance, while pushing them further apart from representations of distinct instances in the latent space. Unlike self-supervised learning methods B2 33)
which involve large number of sample pairs (e.g., 4096), there are typically fewer than 100 track queries propagated from the previous frame to samples pairs. To alleviate this issue of insufficient sample pairs, we draw 34 and utilize kernel-basse contrastive learning, which is formulated as:
(KI(K;T) = - log exp(K]ee)t obj e
(11)
I exp(KTTe mbj e 1/T)
where en and ctrl are the embeddings of object and track queries in the C-decoder. T temperature syper-parameter [35] K[|| refers to the kernel function mapping with Φ(-) and ge(-):
_ ( (c d(gor.. (c**k))
(12)
where ( is the inner product. When ob-)) is linear mapping function, KC[-] can be converted to:
Σ (gn (e ), (e*rk)))
(13)
where (s-)x is the Hadamard product. In practice we employ o6.) and g0.s, (-) as a multi-layer perceptron (MLP) and FFN. We do not utilize any projection for track queries for simplicity:
Cbe frk MLP ((F)N((e0)))
(14)
We apply Instance-aware Contrastive Learning only in the training stage. During inference, the additional model parameters used for contrastive learning from Eq. [4 are discarded.

---

Table 1: Comparison results adopting SynCL with
(TBA) baselines on nuScenes val set. We divide the experiment into four groups based on the detector settings. denotes results from 66
Tracking
Detection
Method
Backbone
Detector Resolution
AMOTA AMOTP!
Recall MOTA
IDS)
FPJ
FN1
NDS mAP
MUTR3D* g
R101
DETRAD 900 x 1600
32.1%
1.448
45.2%
28. 28.35
474
15269
43828
SynCI (ours)
R101
DETR3D 900 1600
35.8%
1,391
49.2%
32.9%
588
14311
40740
PF-Track (5
V2-99
PETR
320 x 800
40.8%
1,343
50.7%
37.6%
166
15288
40398
47.7%
37.8%
SynCL (ours)
V2-99
PETR
320 800
44.7%
1.262
56.5%
40.8%
203
15344
36801
49.7%
39.6%
Baselinel! X
V2-99
PETRv2
320 800
43.2%
1.272
55.0%
40.6%
173
14106
37065
50.4%
41.0%
SynCL (ours)
V2-99
PETRv2
320 800
45.7%
1.260
56.8%
43.0%
170
13411
36756
51.1% 42.0%
Baseline#2 4
V2-99
Stream
320 800
49.6%
1.164
57.3%
42.9%
411
13962
33526
57.6% 48.5%
SynCL (ours)
V2-99
Stream
320 x 800
51.8%
1.149
58.8%
45.2%
540
13639
33368
58.7%
49.2%
4 Experiments
4.1
Experimental Setup
Datasets and Evaluation Metrics. We evaluate the proposed SynCL on nuScenes [36). large- scale benchmark for autonomous driving, which contains 700, 150, and 150 scenes for training, validation, and testing, respectively. The dataset covers 10 types of common objects on the road. For the tracking task, nuScenes selects a subset of 7 movable categories, excluding static objects such as traffic cones. Consistent with official evaluation metrics for 3D MOT, we primarily report AMOTA (average multi-object tracking accuracy) and AMOTP (average multi-object tracking precision) [37] We also report metrics of NDS (nuScenes detection score) and mAP (mean Average Precision) for detection to fully evaluate multi-task performance.
Baseline and Implementation Details. To assess the generality of our proposed synergistic training strategy, we integrate SynCL within PF-Track [5] and MUTR3D [4] frameworks using different de- tectors. Specifically, our experiments involve four query-based detectors: DETR3D (30 PETR (2)]. PETRv2 31 and StreamPETR (26). For DETR3D and PETR, we follow the experimental setups in ADA-Track 6 and PF-Track, respectively. Regarding PETRv2, we establish the baseline by substituting only the detector in the PF-Track framework referred to as Baseline#1. Unlike One- track R our experiments with the StreamPETR detector MUTR3D framework are initialized from pre-trained detector weights and we then train the tracker in the sliding window mode to ensure the temporal gradient flow, denoted as Baseline#2. Following common practice, we set the training epoch to 24 for experiments with resolution of 900/640 x 1600, while we set it to 12 for experiments with resolution of 320 x 800. All methods are trained using the Adam (38) optimizer with a weight decay of 1.0 x 10-2 The learning rate is initially set to 2.0 10 and following a cosine annealing schedule. We conduct all experiments on eight 100 GPUs with a batch size of and each training batch consists of clip of three consecutive frames from different timestamps. For the ab ablation studies, configurations of PF-Track with a small resolution 320 x 800 are employed as the default.
Inference. We claim that SynCL does not employ the weight-shared parallel decoder during the inference stage and does not modify the inference procedures of any baseline, thus having no impact on inference speed. For more details on the runtime analysis, please refer to Appendix B.
4.2 Main results
Generality of SynCL. We first evaluate SynCL on the nuScenes validation set. Tab. I presents the comparative results after integrating SynCL into various trackers, with each baseline shown at the top of each detector group. The experimental results lead four key observations: 1) SynCL can be adapted to various
by--ateentioon paradigm methods. Specifically, SynCL improves MUTR3D and Pr-Track by +3.7% and +3.9% AMOTA, respectively. 2) SynCL exhibits compatibility with various detectors utilizing different attention variants, indicating that the issues associated with the self-attention mechanism are widespread. Building upon our Baseline#1 and Baseline#2, SynCL achieves additional +2.% and 2.2% boosts in AMOTA. 3) SynCI. does not significantly increase the number of false positives while consistently improving the rate of recall. This indicates that our dynamic query filtering can select more high-quality candidates. 4) As synergistic training strategy,
7

---

Table Comparison with leading camera- based methods on the nuScenes val set. The FPS is measured on a single NVIDIA A100 GPU, from the input images to the final tracking results. denotes results from (25
Method
Backbone
Detector
Resolution
AMOTA
AMOTP!
Recall MOTA IDS}
FP4
FN4
FPS
CC-3DT 39
R101
BEVFormer
900 x 1600
42.9%
1.257
53,4%
38.5%
2219
DQTruck 1
V2-99
PETRv2
320 800
44.6%
1.251
1193
8.6
MUTR3D* 4
V2-99
PETR
640 x 1600
44.3%
1,299
55.2%
41.6%
175
11943
36861
6.1
PF-Track S
V2-99
PETR
640 x 1600
47.9%
1.227
59.0%
43.5%
181
16149
32778
5.2
ADATrack++
V2-99
PETR
640 x 1600
50.4%
1.197
60.8%
44.5%
613
14839
30616
3.2
OneTrack g
V2-99
Stream
640 x 1600
54.8%
1.088
61.8%
47.9%
389
SynCL (ours)
V2-99
PETR
640 x 1600
50.7%
1.183
61.3%
46.2%
248
14506
30577
5.2
SynCL (ours)
V2-99
Stream
640 x 1600
58.9%
1.016
64.0%
51.5%
652
13946
27330
5.7
SynCL uniquely enhances both detection and tracking performance, providing a novel solution for end-to-end tracking. Notably, SynCL delivers 2% NDS improvements compared to the PF-Track.
State-of-the-art Comparison. In Tab. 2 first compare SynCL with leading camera- based 3D MOT methods on the nuScenes validation set. With the PETR detector configuration, SynCL achieves the highest performance in both AMOTA and AMOTP. Remarkably, SynCI. does not use the additional association module, resulting in lower inference latency compared to the current leader, - 77. With the StreamPETR detector configuration, SynCI sig- nificantly outperforms the previous ofate-of-the-
Table 3: Comparison on nuScenes test set. 'E2E' stands for the end-to-end detection and tracking model.
Method
E2E AMOTA AMOTP, Recall MOTA
CC-3DT
X
41.0%
1.274
53.4%
38.5%
PF-Track 3
/
43.4%
1.252
53.8%
37.8%
STAR-Track 24
/
43.9%
1.256
56.2%
40.6%
ADATrack++ X
/
50.0%
1.144
59.5%
45.6%
DQTrrcck 11
/
52.3%
1.096
62.2%
44.4%
OneTrruk g
/ 55.4%
1.021
60.8%
46.1%
DORT 18
x
57.6%
0.951
63.4%
48.4%
art One Track by a margin of 4.11% % in AMOTA.
SynCL (ours)
58.8%
0.976
67.1%
50.4%
Moreover, we present comparison results on nuScenes test set in Tab. 3 SynCL achieves leading performance with 58.8% AMOTA and excels over the state-of-the-art camera-baser methods. Notably, SynCL surpasses the dominant two-stage tracker [18] by 1.2% in AMOTA.
Computation and memory complexity. We evaluate the training costs of the models in Tab. 1 We implement the parallel decoder in an efficient way following [40]. As result, SynCL only takes an acceptable increase in training cost including GPU memory and training time as shown in Fig. 5 For example, with Baselinet1 1 and MUTR3D, training time is increased by 0.3 and 0.4 hours per epoch. The memory increases are just 4G and IG, respectively. Thus, the weight-shared decoder design slightly increases the computational cache.
HR-Took
TF-Twnkk
Holbelee Raom
Univllatin Homes
Read4:
Deadft
Desi
MUTRED
MUTROD
Troining Time (b)
GPU Memory (G)
Restie
Figure 5: Analysis of training time (h) and GPU
Figure 6: Illustration of our Dynamic Query Filtering
memory (G). The inference speed remains unchanged.
with Gaussian Mixture Modeling.
4.3 Ablations and Analysis
Component effectiveness. In Tab. 4a we analyze the importance of each proposed component in SynCL for the model performance. According to the specific task, we decompose hybrid matching into one-to-many learning for detection and one-to-one learning for tracking. In general, our pro- posed components enjoy consistent performance improvements. Specifically, by introducing hybrid matching, it outperforms the baseline by 1.8% AMOTA. Further integrating the instance-aware con- trastive learning brings an extra 2.1% improvement in AMOTA. Together, our proposed components complement each other to effectively address multi-task learning challenges.
Variants of query filtering. To our knowledge, one-to-many matching has received limited attention in 3D perception. Therefore, we compare our (GMd-based filtering module with methods 41 from 2D tasks. Concretely, divide these methods into Iosed group and cost-based group. As
8

---

Table 4: Ablation study on components and query filtering methods. The terms "Det" and "Trk" denote the matching for detection and tracking in Task-specific Hybrid Matching module, respectively. "CL" denotes the Instance-aware Contrastive Learning for object and track queries in C-decoder.
(a) Evaluation of component effectiveness.
(b) Comparison of query filtering methods.
Tracking
Detection
Tracking
Detection
#
Det Trk CL
Method
AMOTA AMOTP/ Recall
NDS
mAP
AMOTA
AMOTP
NDS
mAP
loU-based
40.8%
1.343
47.7%
37.8%
ATSS 4
39.3% 42.8%
1.367
49.9%
47.0%
37.3% 49.2% 38.8%
2 /
42.2%
1.327
49.8%
38.9%
SimOTA 22
1,323
52.5%
(
40.9%
1.339
48.1%
37.9%
Cost-based
④
/
/
42.6%
1,286
49.9%
39.9%
DETA 43
43.1% 44.7%
1,320
54.2%
49.4% 38.6%
1
/
44.7%
1.262
49.7%
39.6%
GMM (ours)
1.262
56.5%
49.7% 39.6%
Table 5: Comparision results of matching variants for object queries in Task-specific Hybrid Matching module.
Tracking
Detection
#
Decoder Type
Matching Runge
Strategy
AMOTA
AMOTP/
Recall
FP1
FNI
NDS
mAP
S-decoder
New-born GTs
one-to-one
40.8%
1.343
50.7%
15288
40398
47.7%
37.8%
S-decoder
New-horn GTs
one-to-many
38.8%
1.357
52.4%
15461
39930
46.5%
37.1%
C-decoder
New-burn GTs
one-to-many
41.1%
1.339
52.4%
14265
40231
47.8%
38.3%
C-decoder
All GTss
one-to-many
42.6%
1.300
53.3%
14683
37666
49.1%
38.6%
C-decoder
Persistent GT's
one-to-many
44.7%
1.262
56.5%
15344
36801
49.7%
39.6%
Table 6: Ablation of maximum assignment constraint K.
Table 7: Ablation of training sample length T.
Tracking
Detection
Tracking
Detection
K
AMOTA
AMOTP
Recall
NDS
mAP
AMOTA
AMOTP,
Recall
NDS
mAP
4
44.3%
1,286
54.9%
50.1%
39.5%
2
43.1%
1.320
54.2%
49.4%
38.6%
44.7%
1.262
56.5%
49.7%
39.6%
44.7%
1.262
56.5%
49.7%
39.6%
6
42.8%
1.298
53.4%
49.7%
39.2%
4
44,2%
1.282
54.1%
50.1%
39.3%
shown in Tab. 4b IoU-based methods fail to surpass even the
method, DETA 44
We infer that the object queries cannot ensure sufficient IoU overlap with GT boxes. Compared to our dynamic method, DETA proves inadequate for mining potential high-quality candidates. We further demonstrate the candidate selection effect of our GMM--bsed filtering module in Fig. 6
Variants of matching. In Tab. 5] we analyze the impact of different matching variants for object queries in hybrid matching module. We consider from the perspectives of decoder type, matching range, and matching strategy, compared to our baseline (highlighted in gray). Based on the ex- perimental results we draw the following two conclusions: 1) The one-to-many assignment is not applicable the standard decoder due to the NMS effect of the self-attention mechanism (Dvs..) 2) Object queries in the C-decoder cannot benefit from a broader matching range with all ground-truth targets (@vs. We attribute this to the reduction of positive sample proportion in contrastive learning, which may diminish the effectiveness of synergistic learning. Moreover, matching solely with New-born GT's offers limited improvement due to the reduction in positive supervision (3vs.
Hyperparameters. We study two types of hyperparameters, the maximum assignment constraint K per GT in Eq. and the training sample length T. For K, according to Tab. 6 the best performance is achieved when K 5. Possibly, a smaller matching number could lack supervision signals, while a larger matching number could introduce noisy and unreliable candidates. For T in Tab. 7 we observe that when the frame number is increased from 2 to 3, SynCL obtains a boost of +1.6% AMOTA. However, when T > 3. SynCL cannot improve the model further. We infer that more consecutive frames lead to more optimization targets and temporal gradients, affecting the training stability. We thus use K 5 and T 3 as the default setting for all our main results and other ablation studies.
Comparison to One While both Trackack [8] and SynCI. aim to optimize the joint training of detection and tracking, we summarize the core differences between SynCL and Dreckack as follows:

---

Table 8: Comparison adopting SynCL with Onetrack on nuScenes vol set. * denotes our reproduced results.
Tracking
Detection
Methods
Backbone
Detector
#Epochs
AMOTA
AMOTP
NDS
mAP
Baseline
V2-99
PETR
36.5%
1.411
46.2%
36.9%
24
OneTrack*
V2-99
PETR
37.8%
1,380
47.3%
37.7%
SynCL
V2-99
PETR
42.4%
1.347
48.9%
38.4%
Baseline
V2-99
StreamPETR
43.8%
1.265
53.6%
45.1%
24
OneTrack* 8
V2-99
StreamPETR
45.3%
1.237
55.9%
46.4%
24
SynCL
V2-99
StreamPETR
48.1%
1.204
56.8%
47.0%
24
(1) Research perspective. OneTTrack mainly attributes the joint optimization problem to the conflicting gradient flow within the classification heads, whereas we analyze this problem from the perspective of the attention mechanism and the bipartite matching strategy.
(2) Orthogonality. We speculate that the multi-task learning conflict studied by both OneTrack and ourselves is orthogonal. To validate this, we reproduce the one-stage training based on PF-Track-S and our Baseline#2 (@ and (). We then reproduce OneTrrack (@ and ()) with two classification heads for gradient coordination and we do not use dynamic mask and high-cost assignment suppression. Finally, we apply our method the reproduced OneTrack framework. As shown in Tab. [ the consistent improvements (Brs, @,, Tys. @) prove our speculation and thus highlight our work's s novelty.
(3) Training cost and performance. The training epochs of the one-stage training equal the total epochs of the two-stage training. Therefore, with comparable training cost, the models integrated with SynCL outperform Treck under the same configurations as shown in Tab. 2
5 Conclusion
In this paper, we have delved into the optimization difficulties of joint detection and tracking training in end-to-end 3D trackers. We reveal that the self-attention mechanism the transformer-based decoder hinders multi-task learning through over-deduplicating object queries and introducing self- centric attention to track queries. Based on these analyses, we have proposed to synergistically optimize multi-task learning through hybrid matching for both queries with dynamic filtering and contrastive learning within a weight-shared parallel decoder without self-attention. Both quantitative and qualitative results the nuScenes dataset demonstrate the superiority of SynCL with consistent gains, achieving new ofate-of-the-ar performance for the multi-camera 3D MOT task.
Limitations and broader impacts. The parallel decoder's 's construction with SynCL components leads to higher training costs, including increased training time and memory usage. Future work could investigate decoupled incremental training methods, e.g.. LoRA, to further reduce these additional training expenses. Moreover, we have preliminarily validated SynCL's effectiveness under the one-stage training setting. However, the multi-task learning conflicts may not be fully resolved, as evidenced by the decreased performance compared to our two-stage training. Future work could explore more fine-grained training strategies designed for this long-term one-stage training.
Acknowledgments and Disclosure of Funding
This work was supported in part by the Beijing Natural Science Foundation (Grant No. JQ22014, L223003,, L223020), the Natural Science Foundation of China (Grant No. 62422317, U22B2056. 62036011, 62192782, U24A20331, U2441241).
10

---

References
[1] Hou-Ning Hu, Yung-Hsu Yang. Tobias Fischer, Trevor Darrlll, Fisher Yu. and Min Sun. Monocular quasi-dense 3d object tracking. IEEE Trans. Pattern Anal. Mach. Intell., pages 1992-2008, 2023.
Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani, 3d multi-object tracking: A haseline and new evaluation metrics. In IROS, pages 10359-10366, 2020.
[3] Ziqi Pang, Zhichao Li, and Naiyan Wang. Simpletrack: Understanding and rethinking 3d multi-object tracking. In ECCVW, volume 13801, pages 680-696,200 2022.
[4] Tianyuan Zhang. Xuanyao Chen, Yue Wang. Yilun Wang. and Hang Zhao. MUTR3D: A multi-camera tracking framework via 3d-to-2d queries. In CVPRW, pages 4536-4545, 2022.
[5] Ziqi Pang. Jie Li, Pavel Tokmakov, Dian Chen, Sergey Zagoruyko, and Yu-Xiong Wang. Standing between past and future: Spatio-tempora modeling for multi multi-camera 3d multi-object tracking. In CVPR, pages 17928 17938, 2023.
[6] Shuxiao Ding, Lukas Schneider, Marius Cordts, and Juergen Gall. Ada-track: End-to-end multi-camera 3d multi-object tracking with alternating detection and association. In CVPR, 2024.
[7] Shuxiao Ding, Lukas Schneider, Marius Cordis, and Juergen Gall. Ada-track++:: End-to-end multi-camera 3d multi-object tracking with altemating detection and association. CoRR, abs/2405.08900 2024.
[8] Qitai Wang, Jiawei He, Yuntao Chen, and Zhaoxiang Zhang. OneTrack: demystifying the conflict between detection and tracking in end-to-end 3d trackers. In ECCV, volume 15065, pages 387-404, 2024.
Yuang Zhang, Tiancai Wang, and Xiangyu Zhang. MOTRv2: bootstrapping end-to-end multi-object tracking by pretrained object detectors. CVPR, pages 22056-22065, 2023.
[10] En Yu, Tiancai Wang, Zhuoling Li, Yuang Zhang, Xiangyu Zhang, and Wenbing Tao. Motrv3: Release- fetch supervision for end-to-end multi-object tracking. CoRR, abs/2305. 142998 2023.
[11] Yanwei Li, Zhiding Yu, Jonah Philion, Anima Anandkumar, Sanja Fidler, Jiaya Jia, and Jose Alvarez. End-to End-to-end 3d tracking with decoupled queries. In ICCV, pages 18256-18266 2023.
[12] Zhengdong Hu, Yifan Sun, Jingdong Wang, and Yi Yang. DAC-DETR: divide the attention layers and conquer. In NeurIPS, 2023.
[13] Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi oujeet tracking by associating every detection box. In ECCV, pages 1-21, 2022.
[14] Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with deep association metric. In ICIP, pages 3645-3649, 2017.
[15] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3d object detection and tracking. In CVPR, pages 11784-11795 2021.
[16] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfu- sion: Robust lidar-camera fusion for object detection with transformers. CVPR, pages 1080-1089, 2022.
[17] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. CoRR, abs/1900..00 004807800 2019.
[18] Qing Lian, Tai Wang, Dahua Lin, and Jiangmiao Pang. DORT: modeling dynamic objects in recurrent for multi-camera 3d object detection and tracking. In CoRL, volume 229, pages 3749-37 3765, 2023.
[19] Xinshuo Weng, Yongxin Wang, Yunze Man, and Kris M. Kitani. GNN3DMOT: graph neural network for 3d multi-object tracking with 2d-3d multi-feature learning. In CVPR, pages 6498-6507, 2020.
[20] Xinshuo Weng, Ye Yuan, and Kris Kitani. PTP: parallelized tracking and prediction with graph neural networks and diversity sampling. IEEE Robotics Autom. Lett., pages 4640-4641 2021.
[21] Yihan Zeng, Chao Ma, Ming Zhu, Zhiming Fan, and Xiaokang Yang. Cross-modal 3d object detection and tracking for auto-driving. In IROS, pages 3850-3857, 2021.
[22] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229, 2020.
11

---

[23] Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. MOTR: end-to-end multiple-object tracking with transformer. In ECCV. volume 13687, pages 659-675, 2022.
[24] Simon Doll, Niklas Hanselmann, Lukas Schneider, Richard Schulz, Markus Enzweiler, and Hendrik P. A. Lensch. S.t.a.ttrack: Latent motion models for end-to-end 3d object tracking with adaptive spatio-temporal appearance representations. IEEE Robotics Autom. Lett.. pages 1326-1333, 2024.
[25] Shuxiao Ding, Yutong Yang. Julian Wiederer, Markus Braun, Peizheng Li, Juergen Gall, and Bin Yang. Tqd-track: Temporal query denoising for multi-object tracking. CoRR, abs/2504.0322 003258, 2025.
[26] Shihao Wang. Yingfei Liu,, Tiancai Wang. Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view object detection. In ICCV, pages 3598-3608 2023.
[27] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778. 2016.
[28] Youngwan Lee, Joong-Wo Won Hwang, Sangrok Lee, Yuseok Bae, and Jongyoul Park. An energy and gpu-computation efficient backbone network for real-time object detection. In CVPRW, pages 752-760, 2019.
[29] Yingfei Liu. Tiancai Wang, Xiangyu Zhang, and Jian Sun. PETR: position embedding transformation for view 3d object detection. In ECCV, pages 531-548, 2022.
[30] Yue Wang, Vitor Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, Justin Solomon. DETR3D: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180-1911 PMLR, 2021.
[31] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi-camera images. In ICCV, pages 3239-3249, 2023.
[32] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In ICCV, pages 9620-9629, 2021.
[33] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pages 9630-9640, 2021.
[34] Yao-Hung Hubert Tsai, Tianqin Li, Martin Q. Ma, Han Zhao, Kun Zhang, Louis- Philippe Morency, and Ruslan Salakhutdinov. Conditional contrastive learning with kemel. In ICLR, 2022.
[35] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In CVPR, pages 3733-3742, 2018.
[36] Holger Cacsar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, pages I 1618-11628, 2020.
[37] Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani. 3d multi-object tracking: A haseline and new evaluation metrics. In IROS, pages 10359-10366. 2020.
[38] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[39] Tobias Fischer, Hsug-Hsu Yang, Suryansh Kumar, Min Sun, and Fisher Yu. CC-3DT: panoramic 3d object tracking via cross-camera fusion. In CoRL, pages 2294-2303 2022.
[40] Qiang Chen, Xiaokang Chen, Jian Wang. Shan Zhang, Kun Yao, Haocheng Feng. Junyu Han, Errui Ding, Gang 7eng, and Jingdong Wang. Group DETR: fast DETR training with group-wise one-to-many assignment. In ICCV, pages 6610-6619, 2023.
[41] Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. Bridging the gap between anchor- based and anchor-free detection via adaptive training sample selection. In CVPR, pages 9756-9765. 2020.
[42] Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. YOLOX: exceeding YOLO series in 2021. CoRR, 0tt/21077.843000 2021.
[43] Jeffrey Zhang, Jang Hyun Cho, Xingyi Zhou, and Philipp Krähenbühl. NMS strikes back. CoRR, abx/2212.01137, 2022.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
Please refer to Sec. 1 and Sec. 3.2
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Sec. 4. and Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
Please refer to abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpippi.ccppulli//uuuia wwwiiicgguie//eeeee CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Sec. 4.1 and Appendix A. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: It would be too computationally expensive to report Error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
15

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Sec. 4. and Sec.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httpt:htttt:ttte http:/rwww...pppaaaacccccccceaaa
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and we confirm that the research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer Sec. and Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Please refer to Appendix C.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Please refer Sec. 4.1 and reference.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Code and raw results will be publicly available upon acceptance and we will include details about training, license, limitations, etc.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research in this paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research in this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
18

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We have reviewed the LLM policy and confirm that the method development in this research does not involve LLMs as any important, original, or non- standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy bubtps://neur/e /ttrrripp...c/cooofeeeeeee////2 for what should or should not be described.
19


Reviews:
[{"review_id": "DKoyqiP6XL", "reviewer": "Reviewer_rfd7", "summary": "This paper points out two issues with the self-attention mechanism in tracking-by-detection model: over-deduplication for object queries and self-centric attention for existing track queries. To resolve this, a parallel weight-shared decoder that operates without self-attention is introduced during training, and a series of training strategies are applied, including Task-specific Hybrid Matching, Dynamic Query Filtering and Instance-aware Contrastive Learning. Experiments on the nuScenes dataset show that the proposed method improves performance on multiple baseline trackers.", "questions": "Implementation details about dynamic query filtering are missing. For example\n- In cost-based filtering, what is the cost function between prediction and ground-truth ? what is the threshold to distinguish reliable and unreliable clusters ?\n- In IoU-based filtering, is it normal IoU between 3D boxes or other variants, like gIoU?", "limitations": "The authors didn't discuss the limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- This paper provides a very clear analysis of how existing tracking-by-attention models suffer from over-deduplication and self-centric attention issues. Aiming at solving it, a parallel decoder without self-attention is used during training in task-specific hybrid matching module, along with dynamic query filtering module to select reliable candidates, and instance-aware contrastive loss to force interaction between object and track queries corresponding to the same instance.\n\n- Comprehensive experimental results are provided. The proposed method is successfully applied to various baseline models and brings convincing benefits, for example, MUTR3D, PF-Track, PETRv2, StreamPETR, demonstrating its generalization and effectiveness. Finally, 58.9% AMOTA on the nuScenes validation set and 58.8% on the test set are achieved.\n\n\nWeaknesses:\n- The major concern about this paper is limited novelty. Although the analysis of self-attention mechanism and a parallel decoder without self-attention during training are proved very useful, this is not the first time I see this technology. For example, DAC-DETR[1] is published two years ago.  \n- Most discussions in this paper are well-studied in 2D object detection and tracking filed[1][2][3][4]. Adapting these technologies from 2D to 3D seems like an incremental work, so my initial rating is borderline reject. But I will re-evaluate my rating after discussing with other reviewers.\n\n[1] DAC-DETR: Divide the Attention Layers and Conquer\n\n[2] MOTR: End-to-End Multiple-Object Tracking with Transformer\n\n[3] DETA: NMS Strikes Back\n\n[4] YOLOX: Exceeding YOLO Series in 2021:", "was_revised": true, "final_justification": "Thank authors for the detailed response. My concerns about technical details have been addressed. Although I still think adapting 2D technology to 3D is not novel enough, I will increase the score given the good performance number.", "rebuttal": "Thank you for your insightful review and valuable comments! We are pleased that you acknowledge our paper's *clear analysis* for existing tracking-by-attention models and *convincing benefits* across various baseline models, demonstrating the *generalization* and *effectiveness* of our method. We will respond to your concerns point by point.\n\n> Q1: The major concern about this paper's novelty is that DAC-DETR for 2D object detection also investigates the attention mechanisms and most discussions in this paper are well-studied in 2D object detection and tracking field.\n\nWe are sorry for not highlighting our novelty in comparison to the previous 2D detection and tracking technologies thoroughly in our main paper. We propose SynCL as the first novel synergistic training strategy compatible with any tracking-by-attention paradigm tracker, achieving a new state-of-the-art performance of 58.9% AMOTA on the nuScenes dataset. Our novelty and advancements in the tracking-by-attention paradigm have been acknowledged by Reviewer `FdMH` and Reviewer `A2ku`. We are more than willing to further clarify our novelty and contributions as follows:\n\n* Although DAC-DETR also draws on the investigation of the attention mechanisms, we summarize the core differences between SynCL and DAC-DETR.\n    * The multi-camera 3D tracking task, addressed by SynCL, constitutes the significant **expansion in research scope and complexity** compared to 2D object detection, inherently requiring processing and integrating information across additional dimensions, e.g., temporal cues, multi-view inputs, and 3D spatial perception.\n    * We have reached **more comprehensive conclusions** specific to the 3D MOT task through our qualitative (prediction visualization in Figure 2) and quantitative (self-attention heatmap in Figure 3) analyses, e.g., the over-deduplication of object queries and the self-centric issues of track queries.\n    * As we claim in line 55-58 that DAC-DETR cannot address the issues we have identified simultaneously, we supplementarily validate this by directly applying DAC-DETR’s parallel decoder technology to the tracking-by-attention paradigm as shown in ③ of the table below. \n    * Compared to the static matching strategy (DETA) in DAC-DETR, the Dynamic Query Filtering in SynCL uncovers more high-quality candidates with enhanced 3D geometric knowledge and depth-aware information, as shown in Figure 6 of our main paper and Appendix Figure A1.\n    * SynCL further introduces a novel Instance-aware Contrastive Learning to align these high-quality 3D query features with track queries, effectively breaking the constraints of self-centric attention. Importantly, this contrastive learning is built upon our overall designation rather than an independent component, please refer to the **response to Q3 of Reviewer `A2ku` and Q1 of Reviewer `VTpN`**.\n\n* To further substantiate our contributions, we compare SynCL with other 2D object detection and tracking works [2,3,4]. \n    * MOTR [2] is the first work to propose the tracking-by-attention paradigm in 2D MOT, but it does not systematically explore the challenges of joint detection and tracking training.\n    * Although DETA [3] and YOLOX [4] have designed one-to-many matching strategies (DETA and SimOTA) specifically for 2D detectors, adapting these technologies directly from 2D to 3D MOT yields limited improvements, as shown in ② and ③ of the table below.\n    * We would like to emphasize that the ablation results in Table 4(b) of our main paper **incorporate both Hybrid Matching and Instance-aware Contrastive Learning**. We combine these results below (④ and ⑤) for a thorough comparison.\n    * In contrast to the incremental adaptation of these 2D techniques to 3D MOT (② *vs.* ④, ③ *vs.* ⑤), the consistent improvements by SynCL highlight the novelty of our work and its contribution to 3D MOT.\n\n    | # | Methods | Backbone | Detector | AMOTA | AMOTP↓ | NDS | mAP |\n    | - | - | - | - | - | - | - | - |\n    | ① | PF-Track-S | V2-99 | PETR | 40.8% | 1.343 | 47.7% | 37.8% |\n    | ② | DAC-DETR (SimOTA) | V2-99 | PETR | 40.7% | 1.345 | 48.0% | 37.8% |\n    | ③ | DAC-DETR (DETA) | V2-99 | PETR | 41.3% | 1.337 | 48.7% | 38.3% |\n    | ④ | SynCL (SimOTA) | V2-99 | PETR | 42.8% | 1.323 | 49.2% | 38.8% |\n    | ⑤ | SynCL (DETA) | V2-99 | PETR | 43.1% | 1.320 | 49.4% | 38.6% |\n    | ⑥ | SynCL (GMM) | V2-99 | PETR | **44.7**% | **1.262** | **49.7**% | **39.6**% |\n\n* We have additionally integrated SynCL into the one-stage training method, i.e., OneTrack, to further demonstrate the generality and novelty of SynCL. Please refer to the Table in **response to Q1 of Reviewer `A2ku`**.\n\n* We will include these discussions in the revision (e.g., in lines 55-58 and Table 4) to highlight our novelty and contributions compared to previous 2D works.\n\n> Q2: In cost-based filtering, what is the cost function between prediction and ground-truth? what is the threshold to distinguish reliable and unreliable clusters?\n\nThank you for pointing this out. In cost-based filtering, the cost function combines focal loss for classification and L1 loss for 3D box regression, consistent with the cost metrics in Eq. (8). As shown in the yellow box of Figure 4 in our main paper, the filtering threshold in SynCL is adaptively calculated using our GMM-based query filtering method, based on the cost matrix.\n\n> Q3: In IoU-based filtering, is it normal IoU between 3D boxes or other variants, like gIoU?\n\nThank you for pointing this out. As shown in Table 4 (b) of our main paper, we have only replaced the query filtering component in SynCL with other variants to validate the effectiveness of our GMM-based method. All IoU-based filtering variants in Table 4 (b) utilize normal IoU to compute the IoU between prediction 3D boxes and ground-truth 3D boxes, which determines the allocation of the top-k object queries for each ground-truth. Besides, we find that other IoU types, e.g., gIoU, have a **minimal impact** on model performance, as shown below.\n\n| Query Filtering | IoU Type | AMOTA | AMOTP↓ | NDS | mAP |\n| - | - | - | - | - | - |\n| SimOTA | Normal IoU | 42.8% | 1.323 | 49.2% | 38.8% |\n| SimOTA | gIoU | 43.0% | 1.318 | 49.5% | 38.9% |\n\n> Q4: The authors didn't discuss the limitations.\n\nAlthough we have analyzed the computation and memory complexity in Section 4.2 as the limitations we mentioned in checklist, demonstrating that our method incurs additional training costs, we summarize the limitations and broader impact of SynCL in detail as follows:\n\n* The construction of the parallel decoder combined with SynCL's components incurs additional training costs, including increased training time and memory usage. Future work could consider exploring decoupled incremental training methods, e.g., LoRA, to further reduce these additional training expenses.\n* In response to Q1 of Reviewer `A2ku`, we have preliminarily validated SynCL's effectiveness under the one-stage training setting. However, the multi-task learning conflicts may not be fully resolved, as evidenced by the decreased performance compared to our two-stage training. Future work could explore more fine-grained training strategies designed for this long-term one-stage training.\n\nFollowing your advice, we will include these discussions in the revision.\n\n**References**:\n\n[1] Hu Z, Sun Y, Wang J, et al. DAC-DETR: Divide the Attention Layers and Conquer. NeurIPS, 2023.\n\n[2] Zeng F, Dong B, Zhang Y, et al. MOTR: End-to-End Multiple-Object Tracking with Transformer. ECCV, 2022.\n\n[3] Jeffrey Ouyang-Zhang, Jang Hyun Cho, Xingyi Zhou and Philipp Krähenbühl. DETA: NMS Strikes Back. arXiv: 2212.06137 (2022).\n\n[4] Ge Z, Liu S, Wang F, Li Z, and Sun J. YOLOX: Exceeding YOLO Series in 2021. arXiv:2107.08430 (2021)."}, {"review_id": "Iwucd3imoS", "reviewer": "Reviewer_FdMH", "summary": "This paper handles the over-deduplication for object queries and self-centric attention for track queries caused by the self-attention mechanism for robust multi-view 3D object tracking. The authors present SynCL, a novel plug-and-play synergistic training strategy designed to co-facilitate multi-task learning for detection and tracking. A Task-specific Hybrid Matching module is designed to exploit promising candidates overlooked by the self-attention mechanism, as well as a Dynamic Query Filtering module to select optimal candidates for the one-to-many matching. Besides, the Instance-aware Contrastive Learning is applied to bridge the gap between detection and tracking. Following the tracking-by-attention paradigm, experimental results on nuScenes demonstrate the effectiveness of the proposed method to improve the baselines with the state-of-the-art performances.", "questions": "I do have some questions, see those questions in the weakness part.", "limitations": "I don’t see the discussion related to the limitations and broader impact. I hope the authors can provide a brief discussion on this part in the response and are encouraged to detail this part in the revision.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "*Strengths:\n1. Enough Novelty: Observing the impedance of the self-attention mechanism across different queries to joint detection and tracking training, this paper proposes a synergistic training strategy compatible with any tracking-by-attention paradigm tracker to address multi-task learning challenges.\n2. Rational Designation: The proposed SynCL implements dynamic filtering-based hybrid matching and instance-aware contrastive learning to recover over-deduplicated object queries and improve the classification capability of matching object with tracking queries, which is intuitively beneficial for tracking.\n3. Rich Experiments and Analysis. The authors provide extensive experiments and analysis for the proposed method. I appreciate this. The experimental analysis with various ablation studies allows a better understanding of each module and overall performance.\n4. Good writing and organization. This paper is well-written and organized. Each section has a clear motivation. It’s easy to follow the ideas.\n\n*Weaknesses:\n1. Imbalanced Performance Metrics: While achieving considerable performance gains in the main metrics (e.g., AMOTA and AMOTP), the IDS degrades compared with the baselines. What are the possible reasons?\n2. Insufficient Preliminaries: The authors should first express the self-attention mechanism in the tracking-by-attention paradigm (better in a pipeline visualization), instead of directly casting the motivation that may distort the authors.\n3. More Comprehensive Ablation: It’s better to show the generality of the proposed method in the tracking-by-detection paradigm, e.g., applying the Dynamic Query Filtering to the matching process.", "was_revised": true, "final_justification": "After considering the authors' rebuttal and the discussion with other reviewers, I maintain my initial Borderline accept rating.", "rebuttal": "Thank you for your insightful review and valuable comments! We are happy that you acknowledge the *novelty* and *rational designation* of our paper in addressing multi-task learning challenges within the tracking-by-attention paradigm. We also value your appreciation of our *rich experiments and analysis*. We will respond to your concerns point by point.\n\n> Q1: The IDS degrades compared with the baselines. What are the possible reasons?\n\n* We appreciate your highlighting of this concern. We would like to first summarize and analyze the changes in ID switches before and after integrating our SynCL as follows: \n    * In Table 1 of our main paper, PF-Track and Baseline#1 show fewer ID switches compared to MUTR3D and Baseline#2 because the PF-Track framework includes a trajectory prediction module, which prolongs the lifespan of tracklets when occlusions or motion blur result in low-confidence predictions for track queries. \n    * With this trajectory prediction module, if the detector performs better, integrating our SynCL can actually reduce ID switches (Baseline#1 *vs.* SynCL: IDS 173 *vs.* IDS 170) and significantly improve overall metrics, including AMOTA which offers a holistic evaluation of tracking performance by accounting for IDS, FP, FN, *etc.*. \n    * Conversely, without this trajectory prediction module, even when using a stronger detector (MUTR3D *vs.* Baseline#2), integrating SynCL will still lead to an increase in ID switches. \n\n* We summarize the possible reasons for this degradation of the IDS metric as follows:\n\n    * Firstly, our Hybrid Matching module has uncovered more high-quality candidates, leading to an **increase in the model's total positive predictions**, as shown in the table below. From a probabilistic perspective, this has heightened the likelihood of ID switches occurring between tracklets.\n\n    | Methods | Backbone | Detector | IDS↓ | TP | FP↓ | Total Positive Predictions |\n    | - | - | - | - | - | - | - |\n    | MUTR3D | R101 | DETR3D | **474** | 57595 | 15269 | 72864 |\n    | SynCL | R101 | DETR3D | 588 | **60569** | **14311** | **74880** |\n    | PF-Track-S | V2-99 | PETR | **166** | 60879 | 16331 | 77210 |\n    | SynCL | V2-99 | PETR | 203 | **64893** | **15344** | **80237** |\n    | Baseline#1 | V2-99 | PETRv2 | 173 | 64659 | 14106 | 78765 |\n    | SynCL | V2-99 | PETRv2 | **170** | **65923** | **13411** | **79334** |\n    | Baseline#2 | V2-99 | Stream | **411** | 66257 | 13962 | 80219 |\n    | SynCL | V2-99 | Stream | 540 | **67689** | **13639** | **81328** |\n\n    * Secondly, track queries may generate low-quality predictions during the inter-frame propagation due to factors, e.g., occlusion and motion blur. With the integration of SynCL, object queries can discover and produce high-quality predictions for the same target while suppressing the confidence of the corresponding track query predictions, creating new tracklets. Although this substitution may lead to ID switches, it allows for **robust re-detection**. After incorporating PF-Track’s trajectory prediction module, the model can retain track queries for a longer duration before re-detecting them, facilitating the identification of truly lost targets outside the field of view. Therefore, the combination of robust re-detection with the trajectory prediction module is essential for addressing the issue of ID switches.\n\n* We will consider incorporating a similar module in our future work to further enhance the IDS performance of other MOT methods and will include these new analyses in the revision.\n\n> Q2: The authors should first express the self-attention mechanism in the tracking-by-attention paradigm (better in a pipeline visualization), instead of directly casting the motivation that may distort the authors.\n\nThank you for your suggestions. To address this concern and enhance the paper's readability, we will make the following adjustments in the revision:\n\n* We will modify the pipeline visualization in Figure 1 to better illustrate how the self-attention mechanism operates within both the standard decoder and the parallel decoder, along with the corresponding algorithmic workflow.\n\n* We will swap the order of Section 3.1 (Analysis) and Section 3.2 (Preliminaries) to first establish the foundational knowledge of the tracking-by-attention paradigm before analyzing and introducing our work's motivation.\n\n> Q3: It’s better to show the generality of the proposed method in the tracking-by-detection paradigm, e.g., applying the Dynamic Query Filtering to the matching process.\n\nWe appreciate your constructive advice. We choose the state-of-the-art tracking-by-detection paradigm method, DQTrack, as our baseline. We apply Dynamic Query Filtering to the affinity matrix in the association module to dynamically adjust the matching threshold in inference, as shown in the table below. The consistent improvements demonstrate the effectiveness and generality of our GMM-based query filtering module. We will add this new ablation comparison in the revision and include more tracking-by-detection paradigm methods in future work.\n\n| Methods | Backbone | Detector | AMOTA | AMOTP↓ | Recall | MOTA | IDS↓ |\n| - | - | - | - | - | - | - | - |\n| DQTrack | R101 | UVTR-C | 39.6% | 1.310 | 48.8% | 32.9% | 1267 |\n| +Dynamic Query Filtering | R101 | UVTR-C | **40.0**% | **1.307** | **49.1**% | **33.4**% | **1219** |\n| DQTrack | V2-99 | PETRv2 | 44.6% | 1.251 | **54.7**% | 38.1% | 1193 |\n| +Dynamic Query Filtering | V2-99 | PETRv2 | **45.1**% | **1.247** | 54.4% | **38.5**% | **1128** |\n\n> Q4: I don’t see the discussion related to the limitations and broader impact.\n\nAlthough we have analyzed the computation and memory complexity in Section 4.2 as the limitations we mentioned in checklist, demonstrating that our method incurs additional training costs, we summarize the limitations and broader impact of SynCL in detail as follows:\n\n* The construction of the parallel decoder combined with SynCL's components incurs additional training costs, including increased training time and memory usage. Future work could consider exploring decoupled incremental training methods, e.g., LoRA, to further reduce these additional training expenses.\n* In response to Q1 of Reviewer `A2ku`, we have preliminarily validated SynCL's effectiveness under the one-stage training setting. However, the multi-task learning conflicts may not be fully resolved, as evidenced by the decreased performance compared to our two-stage training. Future work could explore more fine-grained training strategies designed for this long-term one-stage training.\n\nFollowing your advice, we will include these discussions in the revision."}, {"review_id": "EEtezkKSyM", "reviewer": "Reviewer_VTpN", "summary": "This paper points out that self-attention causes issues like over-deduplication for object queries and self-centric attention for track queries. To solve this, this paper introduces SynCL, a new training strategy that improves the interaction between detection and tracking. The approach includes a task-specific hybrid matching module, a dynamic query filtering module, and an instance-aware contrastive learning method. These methods help enhance the representation learning for both detection and tracking tasks by refining candidate selection, reducing redundant predictions, and improving the cross-task connections.", "questions": "Please refer to the Weaknesses section for my main concerns. \nAdditionally, as I am not deeply familiar with the 3D tracking field, I may take into account the opinions of other reviewers when adjusting the final score.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1.\tThe paper presents an interesting observation regarding how self-attention contributes to the optimization difficulty in joint detection and tracking tasks.\n2.\tThe proposed method achieves good performance.\n3.\tThe paper is well-written, and the experiments are thorough and comprehensive.\n\nWeaknesses:\nAlthough the paper offers a valuable insight into the limitations of self-attention, the proposed solution mainly involves removing the self-attention module. Other components (Task-specific Hybrid Matching, Dynamic Query Filtering, and Instance-aware Contrastive Learning) appear to be loosely related to the identified problem.", "was_revised": true, "final_justification": "After considering the authors' rebuttal and the discussion with other reviewers, I maintain my original weak accept rating.", "rebuttal": "Thank you for your insightful review and valuable comments! We are pleased that you find our observation in joint detection and tracking learning *interesting*. We also value your appreciation of our *thorough and comprehensive experiments* and *good performance*. We will respond to your concerns point by point.\n\n> Q1: The proposed solution mainly involves removing the self-attention module. Other components appear to be loosely related to the identified problem.\n\n* We are sorry for not highlighting the relationships among the components in SynCL. Firstly, we would like to emphasize that our core findings are not limited to the self-attention mechanism as follows:\n    * The self-attention mechanism serves as our **entry point** for exploring the challenges of multi-task learning within the tracking-by-attention paradigm. From this perspective, we analyze the role of different attention mechanisms under the original bipartite matching strategy in facilitating end-to-end detection and tracking. \n    * We find that while the self-attention mechanism coordinates the functions of different types of queries, it also introduces issues, e.g., the over-deduplication of object queries and self-centric attention of track queries. \n    * These issues stem directly from the self-attention mechanism and the bipartite matching strategy. However, the deeper underlying causes are inherent challenges within multi-camera 3D tracking, e.g., a lack of 3D spatial geometric knowledge, depth estimation capability and difficulties in associating the same target across multiple views and frames. \n    * Our proposed SynCL provides a comprehensive and synergistic method to address not only the limitations of the self-attention mechanism but also these broader challenges.\n\n* Secondly, we are more than willing to further clarify the role of each component within SynCL and the specific problems they aim to resolve: \n    * **Task-specific Hybrid Matching.** To recover over-deduplicated object queries, we enhance the supervision of object query predictions through a one-to-many matching strategy in the parallel decoder. Simultaneously, to improve the model's inter-temporal association capability, we employ identity-guided one-to-one matching for track queries.\n    * **Dynamic Query Filtering.** To uncover more high-quality candidates with enhanced 3D geometric knowledge and depth-aware information, we propose a GMM-based filtering method, dynamically selecting reliable 3D predictions for the one-to-many matching, as shown in Figure 6 of our main paper and Appendix Figure A1.\n    * **Instance-aware Contrastive Learning.** To break the constraints of self-centric attention on track queries and improve inter-frame and cross-view association capability, SynCL introduces a novel Instance-aware Contrastive Learning component. This contrastive learning aligns track queries with the high-quality 3D object query features, addressing the issue of low-quality predictions for track queries caused by occlusions or motion blur during temporal propagation.\n\n* Lastly, we summarize the relationships and underlying design logic among the components of SynCL as follows:\n    * The tight coupling between the self-attention mechanism and bipartite matching within the standard decoder makes it impractical to directly apply our Task-specific Hybrid Matching without the parallel decoder, as shown in the table in **response to Q2 of Reviewer `A2ku`**.\n    * The Task-specific Hybrid Matching in the parallel decoder ensures that the matching ranges of object queries and track queries have an intersection, enabling the partitioning of positive and negative sample pairs in Instance-aware Contrastive Learning.\n    * The benefits brought by Instance-aware Contrastive Learning are mainly determined by the effectiveness of query filtering. The higher the quality of the selected object queries, the more correction information can be obtained for track queries. As shown in Table 4(b) of our main paper, other query filtering methods are significantly less effective in comparison to our GMM-based method.\n\n* These well-targeted and interdependent components highlight the coherent nature of our designation, which has also been acknowledged by Reviewer `FdMH`. We will emphasize the goals of each component within SynCL and the inner relationships among them in the revision. We will also swap the order of Section 3.1 (Analysis) and Section 3.2 (Preliminaries) to first establish the foundational knowledge of the tracking-by-attention paradigm before analyzing and introducing our work's motivation, following the advice of Reviewer `FdMH`."}, {"review_id": "mBYgD9bdd7", "reviewer": "Reviewer_A2ku", "summary": "This paper first analyzed two major issues of the popular tracking-by-attention paradigm in 3D Multi-Object Tracking (MOT), i.e. the over duplication for object queries and the self-centric attention for track queries. Based on the investigations, the authors proposed a plug-and-play training strategy to overcome the above-mentioned issues. The proposed training strategy is targeted on the multi-task learning for detection and tracking, where a task-specific hybrid matching module is applied which uses an additional weight-shared decoder combined with an one-to-many matching for object queries for target-assignment during training. The resulting duplications are filtered using a proposed Dynamic Query Filtering module. An Instance-aware Contrastive Learning is proposed to improve the query feature discriminability. The proposed method consumes reasonable additional training cost while maintaining the same inference scheme and achieving significant performance gain.", "questions": "1. Line 86 the authors claimed that tracking-by-detection can be combined with query-based detectors. However, some of the following listed works [19, 20] are learned data association works, not query-based detectors.\n2. Line 98, \"increase inference speed\" seems to be wrong. Maybe increase inference time?\n3. It is unclear whether the Dynamic Query Filtering is applied only during training or also during inference. \n4. In line 198, it is unclear why $$ \\sum_{i=1}^2 \\pi_i = 1$$.\n5. What is the linear mapping function in Equ. (12) and (13)?", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The proposed method is well-targeted, where advancing the tracking-by-attention paradigm is an interesting but less investigated research problem.\n2. The paper is well presented - It first reveals and analyzes the existing problems, and then the proposed method is designed to solve the problems from the analysis, and finally validates the claims with extensive experiments.\n3. While achieving state-of-the-art performance, the authors demonstrated acceptable computational overhead only during training process.\n4. The paper is well written and easy to follow.\n\n**Weaknesses**\n\n1. The proposed parallel decoder and hybrid matching design shares some similarities to OneTrack [8], which uses two classification heads and a matching with gradient coordination. The purpose of the proposed components in this work also aligns with OneTrack, where OneTrack allows a one-stage end-to-end training, i.e. the pre-training of object detection is unnecessary. It is worth to conduct a more in-depth comparison to OneTrack from both methodological and experimental perspectives. Without such convincing comparison, the novelty of this paper will be limited.\n2. The authors claimed that the over-deduplication of object queries and the self-centric attention of track queries are caused by the inconspicuous nature of the self-attention mechanism (line 147, line 13). However, this conclusion is not precise enough since the self-attention combined with the bipartite matching together cased the problem, not the self-attention itself. In fact, self-attention allow information exchange between queries which could benefit the learning process. Therefore, I have the doubts about whether the self-attention layer in the hybrid matching module should be removed. Also, the claims of \"self-attention is the main cause of the problems in tracking-by-attention\" should be rephrased in the whole paper.\n3. The proposed Instance-aware Contrastive Learning seems like a plug-and-play component in parallel to the main contributions. It could also benefit other MOT methods and this should be validated by experiments.", "was_revised": true, "final_justification": "I would like to appreciate the authors to provide a detailed rebuttal and conduct comprehensive experiments, which addresses all of my concerns. The main points that I found very well in the rebuttal are: \n- I believe the reproduction of OneTrack in the first table requires much effort, while a better interpretation is somehow enough to set SynCL apart from OneTrack.\n- The experiment results in the second table can well support the proposed idea and enhances the interpretability of the idea.\n- All the minor concerns are also very addressed.\n- Concerns of other reviewers seem to be also well addressed.\nI hope the authors could add the insights including experiments and discussions, as well as improvements for writing in the revised version, if the paper is accepted.", "rebuttal": "Thank you for your insightful review and valuable comments! We are pleased that you acknowledge our paper's *good* quality, *excellent* clarity, and contribution in *advancing the tracking-by-attention paradigm* from an *interesting and less investigated research perspective*. We will respond to your concerns point by point.\n\n> Q1: It is worth to conduct a more in-depth comparison to OneTrack from both methodological and experimental perspectives.\n\n* We appreciate your highlighting of this concern. While both OneTrack and our proposed SynCL aim to optimize the joint training of detection and tracking, we summarize the core differences between SynCL and OneTrack as follows:\n    * **Research perspective.** OneTrack mainly attributes the joint optimization problem to the conflicting gradient flow within the classification heads, whereas we analyze this problem from the perspective of the attention mechanism and the bipartite matching strategy. \n    * **Orthogonality.** We speculate that the multi-task learning conflict studied by both OneTrack and ourselves is orthogonal. However, **OneTrack has not yet released its open-source code**. To validate this, we reproduce the one-stage training (① and ④) based on PF-Track-S and our Baseline#2. We then reproduce OneTrack (② and ⑤) with two classification heads for gradient coordination as one of the core contributions in their paper and we do not use *dynamic mask* and *high-cost assignment suppression* due to limited rebuttal time. Finally, we apply our method to the reproduced OneTrack framework (③ and ⑥). As shown in the table below, the consistent improvements (② *vs.* ③, ⑤ *vs.* ⑥) prove our speculation and thus highlight our work's novelty.\n    * **Generalization.** The solution in OneTrack has only been validated within the StreamPETR detector, whereas SynCL can be integrated into any tracking-by-attention paradigm tracker and has been validated within multiple detectors. This has also been acknowledged by Reviewer `FdMH` and Reviewer `rfd7`.\n    * **Training cost and performance.** The training epochs of the one-stage training equal the total epochs of the two-stage training. Therefore, with comparable training cost, the models integrated with SynCL outperform OneTrack under the same detector and resolution configurations as shown in Table 1 and 2 of our main paper.\n\n    | # | Methods | Backbone | Detector | AMOTA | AMOTP↓ | NDS | mAP | #Epochs |\n    | - | - | - | - | - | - | - | - | - |\n    | ① | Baseline | V2-99 | PETR | 36.5% | 1.411 | 46.2% | 36.9% | 24 |\n    | ② | OneTrack (reproduced) | V2-99 | PETR | 37.8% | 1.380 | 47.3% | 37.7% | 24 |\n    | ③ | SynCL | V2-99 | PETR | **42.4**% | **1.347** | **48.9**% | **38.4**% | 24 |\n    | ④ | Baseline | V2-99 | StreamPETR | 43.8% | 1.265 | 53.6% | 45.1% | 24 |\n    | ⑤ | OneTrack (reproduced) | V2-99 | StreamPETR | 45.3% | 1.237 | 55.9% | 46.4% | 24 |\n    | ⑥ | SynCL | V2-99 | StreamPETR | **48.1**% | **1.204** | **56.8**% | **47.0**% | 24 |\n\n* We will follow your advice to include these more in-depth and convincing analyses in the final revision.\n\n> Q2: I have the doubts about whether the self-attention layer in the hybrid matching module should be removed. Also, the claims should be rephrased in the whole paper.\n\n* We appreciate your constructive advice and agree with your key point. We also believe that the bipartite matching determines the optimization objective, which in turn leads to the nature of the self-attention mechanism. However, we find that the self-attention mechanism and bipartite matching are **tightly coupled in the standard decoder**. Directly modifying the matching strategy without changing the decoder architecture is not feasible. We demonstrate the findings with the following key experiments:\n\n    * As shown in row 2 of Table 5 in our main paper, we have applied one-to-many matching to object queries in the standard decoder (S-decoder) **without constructing the parallel decoder**, but this results in a decrease in model performance (① *vs.* ②).\n    * We further try to expand the matching range of object queries to include all GTs while still keeping the self-attention mechanism in the only standard decoder. However, both the detection and tracking performance of the model suffered significantly (② *vs.* ③). This shows that allowing object queries and track queries to match the same ground truth directly in the standard decoder worsens the query competition rather than improving it.\n\n    | # | Decoder Type | Matching Range | Strategy | AMOTA | AMOTP↓ | NDS | mAP |\n    | - | - | - | - | - | - | - | - |\n    | ① | S-decoder | New-born GTs | one-to-one | **40.8**% | **1.343** | **47.7**% | **37.8**% |\n    | ② | S-decoder | New-born GTs | one-to-many | 38.8% | 1.357 | 46.5% | 37.1% |\n    | ③ | S-decoder | All GTs | one-to-many | 26.7% | 1.470 | 38.5% | 17.3% |\n\n    * Finally, we retain the self-attention mechanism in the parallel decoder. Compared to the baseline (④ *vs.* ⑤), this also leads to a severe decrease in model performance, demonstrating that the query competition issues remain with the existence of the self-attention mechanism.\n\n    | # | Parallel Decoder Attention Mechanisim | AMOTA | AMOTP↓ | NDS | mAP |\n    | - | - | - | - | - | - | \n    | ④ | cross | **44.7**% | **1.262** | **49.7**% | **39.6**% |\n    | ⑤ | self+cross | 28.1% | 1.458 | 39.9% | 17.8% |\n\n* These experimental results indicate that it is challenging to address the problems caused by the self-attention mechanism and bipartite matching together without removing the self-attention mechanism. We also agree that the self-attention mechanism can benefit the learning process apart from the over-deduplication and self-centric issues. Based on these observations, we propose SynCL **in a decoupled manner**:\n\n    * We have **retained the self-attention mechanism** in the standard decoder to continue leveraging its advantages.\n    * We further introduce a parallel decoder combined with a hybrid matching strategy to indirectly but effectively mitigate the identified issues (as illustrated in Figure 1 and 4). This rational designation has also been acknowledged by Reviewer `FdMH`. \n\n* Following your advice, we will rephrase all the misleading claims with regard to \"self-attention is the main cause\" and emphasize the effect of bipartite matching (e.g., line 13, line 147) in the revision.\n\n>Q3: Instance-aware Contrastive Learning seems like a plug-and-play component in parallel to the main contributions. It could also benefit other MOT methods and this should be validated by experiments.\n\n* Thank you for your suggestions. We would like to clarify that our proposed Instance-aware Contrastive Learning is **not an independent component** from the following aspects. \n    * Firstly, the partition of positive and negative sample pairs in Instance-aware Contrastive Learning relies on our proposed hybrid matching strategy, which ensures that a ground-truth can be simultaneously assigned to both an object query and a track query.\n    * Secondly, without our Dynamic Query Filtering to identify high-quality candidates, Instance-aware Contrastive Learning cannot effectively facilitate information transfer between track queries and object queries. As shown in Table 4(b) of our main paper, other variants of query filtering are significantly less effective than our GMM-based method.\n    * These interdependent components highlights the coherent nature of our designation. Together, these three components of SynCL can serve as a plug-and-play training strategy that optimizes any tracking-by-attention paradigm method, as shown in Table 1 of our main paper.\n\n* Following your advice, we have successfully integrated SynCL into an additional MOT method, i.e., OneTrack (**in Q1**). These experimental results demonstrate the generality and novelty of our work.\n\n>Q4: Line 86, some of the following listed works [19, 20] are learned data association works, not query-based detectors. Line 98, \"increase inference speed\" seems to be wrong.\n\nWe are sorry for the confusion caused in lines 86 and 98 and we will revise them as follows:\n* Line 86: We will rephrase it to \"*Tracking-by-detection* paradigm achieves further advancements when combined with learning-based association methods.\"\n* Line 98: Thank you for your feedback. The correct statement should be \"increase inference time\". \n\n> Q5: It is unclear whether the Dynamic Query Filtering is applied only during training or also during inference.\n\nWe appreciate your attention to this concern. Dynamic Query Filtering is applied only during training based on the cost matrix between ground-truths and predictions, without affecting the inference process. Following your advice, we will emphasize this in the revision.\n\n> Q6: About the weight normalization in line 198.\n\nWe appreciate your rigorous consideration. In probability theory, a valid probability distribution function must integrate to 1. The weights in the mixture model represent the relative importance that each Gaussian component contributes to the entire model. Therefore, their total must equal 1 to ensure that the mixture model is a legitimate probability distribution.\n\n> Q7: What is the linear mapping function in Equ. (12) and (13)?\n\nThank you for pointing this out. The linear mapping function is described in lines 215-216 of our main paper and we will make this more clear in the revision."}]

Author rebuttal:

Meta-review: Paper Summary:  
This paper identifies that the self-attention mechanism in tracking-by-attention models causes issues like over-deduplication for new object queries and self-centric attention for existing track queries. To address this, the authors propose SynCL, a plug-and-play training strategy that uses a parallel weight-shared decoder without self-attention, alongside several other learning components, to improve the joint learning of detection and tracking tasks.

Main Strengths and Weaknesses:  
The primary strength is its strong empirical results; the proposed method is a well-engineered, plug-and-play solution that demonstrates significant performance improvements when applied to multiple state-of-the-art 3D tracking baselines. The main weakness is the limited technique novelty, as key components of the solution (e.g., a parallel decoder without self-attention) share strong similarities with prior work in 2D object detection and tracking.

Rebuttal Analysis:  
The rebuttal addressed all technical questions and clarity concerns well, leading one reviewer (rfd7) to raise their score and another (A2ku) to affirm their high confidence after receiving detailed responses and new experiments. While some reviewers (FdMH, VTpN) maintained their scores, still harboring concerns about novelty, the rebuttal was strong enough to solidify a positive consensus.

Final Justification:  
All four reviewers converged on a positive recommendation, ranging from "Borderline Accept" to "Accept." While the concerns about novelty remain, the practical contribution of this paper is significant. Considering the identification of a relevant problem in 3D multi-object tracking and the given effective solution, the AC agrees with the reviewers and recommends accepting this paper. Additionally, it is strongly suggested that the authors incorporate the details discussed in the rebuttal to make the paper more comprehensive.
Decision: Accept (poster)