Title: Dual-Path Temporal Decoder for End-to-End Multi-Object Tracking
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Object Tracking

Transformer

Temporal Modeling

Attention Mechanism

Query Update
Abstract: We present a novel end-to-end transformer-based framework for Multiple Object Tracking (MOT) that advances temporal modeling and identity preservation. Despite recent progress in transformer-based MOT, existing methods still struggle to maintain consistent object identities across frames, especially under occlusions, appearance changes, or detection failures. We propose a dual-path temporal decoder that explicitly separates appearance adaptation and identity preservation. The appearance-adaptive decoder dynamically updates query features using current frame information, while the identity-preserving decoder freezes query features and reuses historical sampling offsets to maintain long-term temporal consistency. To further enhance stability, we introduce a confidence-guided update suppression strategy that retains previously reliable features when predictions are unreliable. Extensive experiments on MOT benchmarks demonstrate that our approach achieves state-of-the-art performance across major tracking metrics, with significant gains in association accuracy and identity consistency. Our results demonstrate the importance of decoupling dynamic appearance modeling from static identity cues, and provide a scalable foundation for robust tracking in complex scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a transformer-based Multi-Object Tracking (MOT) framework with a dual-path temporal decoder that separates appearance adaptation and identity preservation. The key contribution is decomposing the temporal modeling into two parallel branches: an appearance-adaptive decoder that dynamically updates query features using current frame information, and an identity-preserving decoder that maintains temporal consistency by freezing track query features and reusing historical sampling offsets. Additionally, the paper introduces a confidence-guided update suppression strategy that prevents unreliable feature updates during inference. Experimental results demonstrate state-of-the-art performance on DanceTrack and SportsMOT benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Clear Technical Approach: The dual-path temporal decoder design is well-motivated and addresses a key challenge in transformer-based MOT, which maintains temporal consistency while adapting to appearance changes.\n\n2. Strong Experimental Results: The method achieves state-of-the-art performance on challenging benchmarks (DanceTrack HOTA: 74.1, SportsMOT HOTA: 73.9).\n\n3. Comprehensive Ablation Studies: The paper provides thorough ablation studies validating both the identity-preserving decoder layer and confidence-guided update suppression components.\n\nWeaknesses\n1. Limited Novelty in Individual Components: While the combination is effective, the individual components (deformable attention, confidence thresholding) are relatively standard. The core novelty lies primarily in the dual-path architecture design.\n2. Questionable Identity-Preserving Strategy: The identity-preserving approach using $t-1$ frame offsets $\\Delta^{(t-1)}_T$ seems overly simplistic for dynamic tracking scenarios. This only considers adjacent frame consistency rather than long-term temporal stability, which limits its effectiveness for long-range tracking.\n3. Insufficient Confidence-Guided Update Suppression Justification: The strategy of completely discarding features below threshold $\\beta$ may be too aggressive. In scenarios with sudden scene changes, objects might have consistently low confidence, leading to permanent feature freezing. The paper lacks a discussion of failure cases and recovery mechanisms.\n4. Limited Long-term Tracking Analysis: The approach primarily addresses frame-to-frame stability but doesn't convincingly demonstrate improvements in long-term tracking scenarios with extended occlusions or re-identification.\n\nQUESTIONS:\n1. Identity-Preserving Mechanism Validity: The reuse of sampling offsets $\\Delta^{(t-1)}_T$ assumes similar attention patterns across consecutive frames. How does this approach handle scenarios with rapid object motion or camera movement where the optimal sampling locations should change significantly? Have you considered learning a dynamic offset adjustment rather than direct reuse?\n2. Confidence-Guided Update Recovery: When an object has low confidence due to temporary occlusion or scene changes, the current mechanism freezes its features indefinitely until confidence recovers. What happens if the confidence threshold is never met again? Do you have mechanisms to prevent permanent feature degradation, and how do you handle re-emerging objects after extended periods?\n3. Long-term vs. Short-term Trade-offs: The identity-preserving decoder only maintains consistency with the immediate previous frame ($t-1$). How does this limited temporal window perform in scenarios requiring longer-term memory? Have you experimented with incorporating features from multiple previous frames or implementing a more sophisticated temporal memory mechanism?\n4. Notation Clarity and Experimental Fairness: The notation $Q_T$ and $Q_T^{(l)}$ can be confusing. Additionally, are the detector backbones consistent across all compared methods?\n\nLIMITATIONS:\n1. Temporal Window Limitations: The paper should acknowledge that the identity-preserving mechanism only considers $t-1$ frame information, which may be insufficient for complex long-term tracking scenarios.\n2. Recovery Mechanisms: Lack of discussion about failure recovery when confidence-guided suppression leads to permanent feature freezing.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a new multi-object tracking (MOT) architecture. The method is built upon DINO-DETR, a variant of the detection transformer (DETR). \n\nThe authors design **a new *dual-path* transformer decoder**. This decoder takes as input:\n1. *track queries*: which come from objects detected at the previous frame\n2. *candidate query features*: which are object candidates from the transformer encoder module (top-k scoring objects).\n\nEvery decoder layer is composed of two parallel decoder layers (i.e., the *appearance-adaptive* and the *identity-preserving* decoder layers) followed by a *feature fusion* module. \n- The *appearance-adaptative* module iteratively refines the object queries using the image features, following DINO-DETR/Deformable-DETR.\n- The *identity-preserving* module follows a similar design, but keeps track queries fixed (from one layer to the other) and reuses historical sampling offsets. This helps safeguard identity-specific information from the past. \n\nThe authors **additionally introduce a selective update mechanism** which only updates queries from previous frames if the object is predicted with high (classification) confidence, to help preserve clean identity embeddings and thus improve long-term tracking.\n\nExperimentally, **the method outperforms previous works on the test sets from DanceTrack and SportsMOT**. It also achieves relatively high accuracy on MOT17's validation split. \nFurthermore, the **ablation studies validate** the utility of the new dual path decoder and the confidence-guided update rule.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The method outperforms state-of-the-art methods on both DanceTrack and SportsMOT.\n- The adaptive update rule is novel and could be beneficial in other methods as well.\n\n### Weaknesses\n\n- The Method section is not easy to read. It would benefit from being improved (e.g., by starting with an overview/intuition instead of notations, etc.). \n- The related work section does not discuss the most similar work, TransTrack [1]. Discussing it would help readers understand how it is similar and how it differs from similar DETR-like approaches: The TransTrack paper, from 2020, also uses two types of queries, a dual transformer decoder, and confidence scores. However, this submission brings a new decoder design and introduces an interesting selective update mechanism.\n- The paper is evaluated on the validation split from MOT17, rather than the official test set. This practice should be avoided, especially when the validation split comes from the same videos as the training split. \n\n\n[1] Sun, Peize, et al. \"Transtrack: Multiple object tracking with transformer.\" arXiv preprint arXiv:2012.15460 (2020).\n\nQUESTIONS:\n1. Could the authors evaluate their method on the official test set from MOT17 instead of the validation set?\n2. Could the authors discuss more in-depth the DETR-like related works (most importantly TransTrack, as it shares many similarities with this method)?\n3. Do you observe any tracking performance degradation under fast motion? Indeed, one would expect that re-using historical offsets in the identity-preserving decoder layers may be detrimental in this case. \n\nThe reviewer is willing to raise the rating if the requests and questions are addressed; however, they will lower the rating if the two main concerns (1 and 2) remain unaddressed. \n\n\n*[Remark: the reviewer is familiar with DETR-based MOT methods but is NOT an author of TransTrack]*\n\nLIMITATIONS:\nNo, the authors only briefly discuss limitations. The reviewer suggests discussing further the method's limitations and future research directions. \n\nFurthermore, the paper does not discuss the potential negative societal impact of MOT (e.g., for autonomous killing robots, surveillance, etc.).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper introduces a dual-path temporal decoder in transformer-based MOT, which separates appearance adaptation and identity preservation using two sets of decoders. The evaluation on the Dancetrack and SportMOT datasets demonstrates the effectiveness of the proposed dual-path temporal decoder.

STRENGTHS AND WEAKNESSES:
Strength
1. The concept of separating appearance and motion is commonly employed in tracking by detection methods. It is interesting to propose such thoughts in the transformer-based methods.
2. The presentation is good. The writing is easy to follow, and the figures are clear.

Weakness
1. The comparisons are not fair.  Other transformer-based methods adopt standard DTER as the object detector, while the proposed tracker adopts a DINO-based detector.
2. The loss functions are missing, which makes the learning objective of each decoder unclear.
3. The benchmark results on the MOT17 and MOT20 datasets are based on the validation sets, which share the same video sequences as the training sets. Such settings make the results not convincing enough.
4. Since the paper introduces two decoders compared to one decoder in existing methods, the computational complexity should be included and compared with existing methods.

QUESTIONS:
1. The performance of the tracker adopting standard DETR should be ablated.
2. Introduce the loss functions for each decoder.
3. Provide results on the test sets of the MOT17 and MOT20 datasets.
4. Provide the computational complexity analysis.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel dual-path temporal decoder to enhance temporal object consistency in transformer-based track-by-query MOT frameworks (e.g., MOTR, TrackFormer). The decoder is integrated into the DINO detector, differing from the Deformable DETR detector used in MOTR/TrackFormer. Specifically, the proposed decoder comprises two parallel branches: an appearance-adaptive branch that refines query features using current frame information, and an identity-preserving branch that maintains temporal consistency by reusing fixed query features and historical sampling offsets (from the deformable attention in DINO) from the previous frame. Additionally, a query feature update strategy is introduced where features are not updated in frames with low confidence scores.\nQuantitative experiments demonstrate SOTA performance of 76.2 HOTA on the DanceTrack dataset, and competitive performance on the SportsMOT dataset and MOT17 validation set. Ablation studies on the DanceTrack validation set validate the effectiveness of the main designs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is clearly-written.\n2. The designed dual-path temporal decoder presents a well-motivated and novel approach to enhancing temporal consistency.\n3. The proposed method achieves excellent performance on the DanceTrack and SportsMOT datasets.  Specifically, it achieves 76.2 HOTA on DanceTrack, establishing a new SOTA result on this benchmark. Furthermore, it demonstrates excellent association performance, highlighting its effectiveness in improving temporal consistency and reducing ID switches.\n\nWeaknesses\n1. The paper reports results for MOT17 on the validation set but omits test set results. The reason for not reporting test set results is not provided.\n2. While MOTRv2 identifies limited detection performance as a key weakness in MOTR and addresses it with an additional detector, and MOTRv3 attributes this to optimization imbalance between detection and track queries, the proposed method achieves better detection performance than MOTRv2. It remains unclear whether this improvement stems primarily from DINO's stronger detection capability and the query denoising technique. Analyzing DINO's contribution in the ablation study would offer valuable insights.\n3. The main text lacks qualitative results (e.g., visualizations of tracking outputs, failure cases) demonstrating the method's effectiveness in improving association performance and reducing identity switches, particularly on challenging sequences. Additionally, the method section is overly detailed. Streamlining this section could save space to incorporate visualizations currently in the supplementary material \n4. The computational overhead introduced by the dual-path decoder and the overall inference speed of the framework are not reported in the main text.\n5. The modification to attention weight computation in the deformable attention (L137-139) is not validated in the ablation study.\n\nQUESTIONS:\nSee the weaknesses section. Each point corresponds to a suggestion.\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}