Title: SynCL: A Synergistic Training Strategy with Instance-Aware Contrastive Learning for End-to-End Multi-Camera 3D Tracking
Venue: NeurIPS 2025
Accepted: yes
Keywords: Autonomous Driving

Multi-Camera 3D perception

Multi-Camera 3D Tracking
Abstract: While existing query-based 3D end-to-end visual trackers integrate detection and tracking via the *tracking-by-attention* paradigm, these two chicken-and-egg tasks encounter optimization difficulties when sharing the same parameters. Our findings reveal that these difficulties arise due to two inherent constraints on the self-attention mechanism, i.e., over-deduplication for object queries and self-centric attention for track queries. In contrast, removing self-attention mechanism not only minimally impacts regression predictions of the tracker, but also tends to generate more latent candidate boxes. Based on these analyses, we present SynCL, a novel plug-and-play synergistic training strategy designed to co-facilitate multi-task learning for detection and tracking. Specifically, we propose a Task-specific Hybrid Matching module for a weight-shared cross-attention-based decoder that matches the targets of track queries with multiple object queries to exploit promising candidates overlooked by the self-attention mechanism and the bipartite matching. To flexibly select optimal candidates for the one-to-many matching, we also design a Dynamic Query Filtering module controlled by model training status. Moreover, we introduce Instance-aware Contrastive Learning to break through the barrier of self-centric attention for track queries, effectively bridging the gap between detection and tracking. Without additional inference costs, SynCL consistently delivers improvements in various benchmarks and achieves state-of-the-art performance with $58.9\%$ AMOTA on the nuScenes dataset. Code and raw results are available at <https://github.com/shubolin028/SynCL>.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper first analyzed two major issues of the popular tracking-by-attention paradigm in 3D Multi-Object Tracking (MOT), i.e. the over duplication for object queries and the self-centric attention for track queries. Based on the investigations, the authors proposed a plug-and-play training strategy to overcome the above-mentioned issues. The proposed training strategy is targeted on the multi-task learning for detection and tracking, where a task-specific hybrid matching module is applied which uses an additional weight-shared decoder combined with an one-to-many matching for object queries for target-assignment during training. The resulting duplications are filtered using a proposed Dynamic Query Filtering module. An Instance-aware Contrastive Learning is proposed to improve the query feature discriminability. The proposed method consumes reasonable additional training cost while maintaining the same inference scheme and achieving significant performance gain.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The proposed method is well-targeted, where advancing the tracking-by-attention paradigm is an interesting but less investigated research problem.\n2. The paper is well presented - It first reveals and analyzes the existing problems, and then the proposed method is designed to solve the problems from the analysis, and finally validates the claims with extensive experiments.\n3. While achieving state-of-the-art performance, the authors demonstrated acceptable computational overhead only during training process.\n4. The paper is well written and easy to follow.\n\n**Weaknesses**\n\n1. The proposed parallel decoder and hybrid matching design shares some similarities to OneTrack [8], which uses two classification heads and a matching with gradient coordination. The purpose of the proposed components in this work also aligns with OneTrack, where OneTrack allows a one-stage end-to-end training, i.e. the pre-training of object detection is unnecessary. It is worth to conduct a more in-depth comparison to OneTrack from both methodological and experimental perspectives. Without such convincing comparison, the novelty of this paper will be limited.\n2. The authors claimed that the over-deduplication of object queries and the self-centric attention of track queries are caused by the inconspicuous nature of the self-attention mechanism (line 147, line 13). However, this conclusion is not precise enough since the self-attention combined with the bipartite matching together cased the problem, not the self-attention itself. In fact, self-attention allow information exchange between queries which could benefit the learning process. Therefore, I have the doubts about whether the self-attention layer in the hybrid matching module should be removed. Also, the claims of \"self-attention is the main cause of the problems in tracking-by-attention\" should be rephrased in the whole paper.\n3. The proposed Instance-aware Contrastive Learning seems like a plug-and-play component in parallel to the main contributions. It could also benefit other MOT methods and this should be validated by experiments.\n\nQUESTIONS:\n1. Line 86 the authors claimed that tracking-by-detection can be combined with query-based detectors. However, some of the following listed works [19, 20] are learned data association works, not query-based detectors.\n2. Line 98, \"increase inference speed\" seems to be wrong. Maybe increase inference time?\n3. It is unclear whether the Dynamic Query Filtering is applied only during training or also during inference. \n4. In line 198, it is unclear why $$ \\sum_{i=1}^2 \\pi_i = 1$$.\n5. What is the linear mapping function in Equ. (12) and (13)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

comments: SUMMARY:
This paper points out that self-attention causes issues like over-deduplication for object queries and self-centric attention for track queries. To solve this, this paper introduces SynCL, a new training strategy that improves the interaction between detection and tracking. The approach includes a task-specific hybrid matching module, a dynamic query filtering module, and an instance-aware contrastive learning method. These methods help enhance the representation learning for both detection and tracking tasks by refining candidate selection, reducing redundant predictions, and improving the cross-task connections.

STRENGTHS AND WEAKNESSES:
Strengths:
1.	The paper presents an interesting observation regarding how self-attention contributes to the optimization difficulty in joint detection and tracking tasks.
2.	The proposed method achieves good performance.
3.	The paper is well-written, and the experiments are thorough and comprehensive.

Weaknesses:
Although the paper offers a valuable insight into the limitations of self-attention, the proposed solution mainly involves removing the self-attention module. Other components (Task-specific Hybrid Matching, Dynamic Query Filtering, and Instance-aware Contrastive Learning) appear to be loosely related to the identified problem.

QUESTIONS:
Please refer to the Weaknesses section for my main concerns. 
Additionally, as I am not deeply familiar with the 3D tracking field, I may take into account the opinions of other reviewers when adjusting the final score.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper handles the over-deduplication for object queries and self-centric attention for track queries caused by the self-attention mechanism for robust multi-view 3D object tracking. The authors present SynCL, a novel plug-and-play synergistic training strategy designed to co-facilitate multi-task learning for detection and tracking. A Task-specific Hybrid Matching module is designed to exploit promising candidates overlooked by the self-attention mechanism, as well as a Dynamic Query Filtering module to select optimal candidates for the one-to-many matching. Besides, the Instance-aware Contrastive Learning is applied to bridge the gap between detection and tracking. Following the tracking-by-attention paradigm, experimental results on nuScenes demonstrate the effectiveness of the proposed method to improve the baselines with the state-of-the-art performances.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths:\n1. Enough Novelty: Observing the impedance of the self-attention mechanism across different queries to joint detection and tracking training, this paper proposes a synergistic training strategy compatible with any tracking-by-attention paradigm tracker to address multi-task learning challenges.\n2. Rational Designation: The proposed SynCL implements dynamic filtering-based hybrid matching and instance-aware contrastive learning to recover over-deduplicated object queries and improve the classification capability of matching object with tracking queries, which is intuitively beneficial for tracking.\n3. Rich Experiments and Analysis. The authors provide extensive experiments and analysis for the proposed method. I appreciate this. The experimental analysis with various ablation studies allows a better understanding of each module and overall performance.\n4. Good writing and organization. This paper is well-written and organized. Each section has a clear motivation. It’s easy to follow the ideas.\n\n*Weaknesses:\n1. Imbalanced Performance Metrics: While achieving considerable performance gains in the main metrics (e.g., AMOTA and AMOTP), the IDS degrades compared with the baselines. What are the possible reasons?\n2. Insufficient Preliminaries: The authors should first express the self-attention mechanism in the tracking-by-attention paradigm (better in a pipeline visualization), instead of directly casting the motivation that may distort the authors.\n3. More Comprehensive Ablation: It’s better to show the generality of the proposed method in the tracking-by-detection paradigm, e.g., applying the Dynamic Query Filtering to the matching process.\n\nQUESTIONS:\nI do have some questions, see those questions in the weakness part.\n\nLIMITATIONS:\nI don’t see the discussion related to the limitations and broader impact. I hope the authors can provide a brief discussion on this part in the response and are encouraged to detail this part in the revision.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper points out two issues with the self-attention mechanism in tracking-by-detection model: over-deduplication for object queries and self-centric attention for existing track queries. To resolve this, a parallel weight-shared decoder that operates without self-attention is introduced during training, and a series of training strategies are applied, including Task-specific Hybrid Matching, Dynamic Query Filtering and Instance-aware Contrastive Learning. Experiments on the nuScenes dataset show that the proposed method improves performance on multiple baseline trackers.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- This paper provides a very clear analysis of how existing tracking-by-attention models suffer from over-deduplication and self-centric attention issues. Aiming at solving it, a parallel decoder without self-attention is used during training in task-specific hybrid matching module, along with dynamic query filtering module to select reliable candidates, and instance-aware contrastive loss to force interaction between object and track queries corresponding to the same instance.\n\n- Comprehensive experimental results are provided. The proposed method is successfully applied to various baseline models and brings convincing benefits, for example, MUTR3D, PF-Track, PETRv2, StreamPETR, demonstrating its generalization and effectiveness. Finally, 58.9% AMOTA on the nuScenes validation set and 58.8% on the test set are achieved.\n\n\nWeaknesses:\n- The major concern about this paper is limited novelty. Although the analysis of self-attention mechanism and a parallel decoder without self-attention during training are proved very useful, this is not the first time I see this technology. For example, DAC-DETR[1] is published two years ago.  \n- Most discussions in this paper are well-studied in 2D object detection and tracking filed[1][2][3][4]. Adapting these technologies from 2D to 3D seems like an incremental work, so my initial rating is borderline reject. But I will re-evaluate my rating after discussing with other reviewers.\n\n[1] DAC-DETR: Divide the Attention Layers and Conquer\n\n[2] MOTR: End-to-End Multiple-Object Tracking with Transformer\n\n[3] DETA: NMS Strikes Back\n\n[4] YOLOX: Exceeding YOLO Series in 2021:\n\nQUESTIONS:\nImplementation details about dynamic query filtering are missing. For example\n- In cost-based filtering, what is the cost function between prediction and ground-truth ? what is the threshold to distinguish reliable and unreliable clusters ?\n- In IoU-based filtering, is it normal IoU between 3D boxes or other variants, like gIoU?\n\nLIMITATIONS:\nThe authors didn't discuss the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-06"}