Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking

Authors: Mattia Segu, Luigi Piccinelli, Siyuan Li, Yung-Hsu Yang, Luc Van Gool, Bernt Schiele
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Synchronized sequence modeling; multiple object tracking

Abstract:
Multiple object tracking in complex scenarios - such as coordinated dance performances, team sports, or dynamic animal groups - presents unique challenges. In these settings, objects frequently move in coordinated patterns, occlude each other, and exhibit long-term dependencies in their trajectories. However, it remains a key open research question on how to model long-range dependencies within tracklets, interdependencies among tracklets, and the associated temporal occlusions. To this end, we introduce Samba, a novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet. Samba autoregressively predicts the future track query for each sequence while maintaining synchronized long-term memory representations across tracklets. By integrating Samba into a tracking-by-propagation framework, we propose SambaMOTR, the first tracker effectively addressing the aforementioned issues, including long-range dependencies, tracklet interdependencies, and temporal occlusions. Additionally, we introduce an effective technique for dealing with uncertain observations (MaskObs) and an efficient training recipe to scale SambaMOTR to longer sequences. By modeling long-range dependencies and interactions among tracked objects, SambaMOTR implicitly learns to track objects accurately through occlusions without any hand-crafted heuristics. Our approach significantly surpasses prior state-of-the-art on the DanceTrack, BFT, and SportsMOT datasets.

Decision: accept (spotlight)
Reviewer scores: 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730388989144

thread_type:
official_review

<thread_object>
creator: reviewer_whqw

<content>
summary:
The paper introduces a tracking-by-propagation framework, SambaMOTR, which wanna tackles the question of how to model long-range dependencies within tracklets, interdependencies among tracklets, and the associated temporal occlusions. The core function of the  SambaMOTR is the Samba module, a novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet.  SambaMOTR is evaluated on DanceTrack, BFT, and SportsMOT datasets and achieves the state-of-the-art performance.

strengths:
1. SambaMOTR introduces the novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet.
2. SambaMOTR achieves good tracking results on DanceTrack, BFT, and SportsMOT datasets.

weaknesses:
1. The authors are advised to analyze the computational complexity.
2. The authors are advised to analyze why Samba is suitable for trajectory modeling and compare it with xLSTM, other SSMs, and RRNs.
3. The authors are advised to provide some tracking cases of SambaMOTR to demonstrate its superiority in trajectory modeling.

questions:
see Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608252336

thread_type:
official_review

<thread_object>
creator: reviewer_zen6

<content>
summary:
The paper introduces the novel SAMBA model architecture for multi-object tracking. It takes a unique approach to tracking in a tracking-by-propagation framework, building on top of selective state-space models captured by Mamba models. Additional self-attention is used to model dependency between tracks. The method is evaluated on three tracking datasets, and it demonstrates a new state-of-the-art performance.

strengths:
(1) The presented method achieves strong performance.

(2) the reason to integrate SSMs into a tracking-by-propagation framework is well motivated.

(3) The ablation studies conducted verified the introduced modifications.

(4) The paper is well written; the provided illustrative examples showcase strong results.

weaknesses:
(1) There is a lack of qualitative comparison between prior works and the proposed method. It is helpful to see qualitative differences between methods' outputs to highlight what improvements in the models' output are contributing to the improvement in the performance. 

(2) On L137, it is stated that  SambaMOTR has "the same GPU memory requirements". However, the paper lacks any measurements to back up this claim. Overall, there is emphasis placed on efficiency in the paper, so it would be good to include some measurements and comparisons to show these points. For example, including peak memory requirements and FLOPS would illustrate this point. 

(3) While the method shows strong results, chiefly due to architectural modifications, it is worth asking whether efficiency gains enable larger (higher capacity) models, which lead to improvements or whether the SSMs formulation provides better biases for the tracking problem. While it might be difficult to disentangle such factors, the paper could help guide future research in this area by reporting some (albeit not great) proxies, such as learnable parameter counts, etc.

questions:
(1) Would it be possible to include a comparison of SambaMOTR on top of YOLOX-X to have an apple-to-apples comparison with prior tracking-by-detections works?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730750640461

thread_type:
official_review

<thread_object>
creator: reviewer_deh5

<content>
summary:
The paper introduces SambaMOTR for multiple object tracking in videos. This method aims to improve tracking by utilizing temporal information about object movements to model historical object data, and occlusions, thereby modeling long-range dependencies, implicit motion, and appearance between tracklets.

strengths:
- Overall, the experiments are extensive with different alterations in modeling provided in the main paper and appendix.

weaknesses:
- Visual comparisons show that the Samba autoregressive model performs better than memory/motion models should be provided to substantiate the claims.
- The interpretations of system $\mathbf{A}$ and control $\mathbf{B}$ matrices in the context of object tracking are unclear. Providing clearer definitions of notations would be helpful.
- I understand the intention to separate two paradigms tracking-by-detection (TBD) and tracking-by-propagation (TBP), but the difference is subtle as the TBP still needs bounding boxes to initiate the states, and needs a detector to recognize newly appearing objects. Then, propagation alone is simply not practical in this problem of multiple object tracking.
- Then it should be fair to compare with TBD methods, and the performance still falls short behind classic motion models as reported in Table A.
- While I value the effort in development, the innovation in the method appears to be limited, as it essentially represents a straightforward combination of MOTR and Mamba and the motivation is not really compelling.

questions:
- Just to confirm, does the state space model operate on the coordinate domain (bounding boxes) or the visual domain (pixel values)? If it does operate on the coordinate domain, how can the auto-regressed output boxes be refined to fit the subjects without looking at visual features?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732152619242

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Comment

comment:
We thank the reviewers for their constructive feedback and for recognizing the strengths of our work, including its novelty and uniqueness [Zen6, WHQW], strong performance [Zen6, WHQW], extensive experiments [dEH5, Zen6], and clarity of presentation [Zen6, WHQW].

We appreciate the suggestions regarding efficiency analysis [Zen6, WHQW] and comparisons with other sequence models [WHQW]. In response, we have conducted a detailed efficiency analysis and provided the requested comparisons, which are included in this General Comment. Additionally, we commit to incorporating further qualitative comparisons in the camera-ready version. Reviewer-specific questions and concerns are addressed in the individual responses.

---

**Efficiency Analysis and Comparison with Other Sequence Models**

We thank Zen6 and WHQW for emphasizing the importance of conducting an efficiency analysis to validate our claims regarding the minimal computational overhead introduced by Samba. Moreover, we thank WHQW for recommending a comparison of SambaMOTR’s performance using different sequence models.

For clarity and conciseness, we combine the two studies. Specifically, we evaluated SambaMOTR with various sequence models, including SSM-based approaches (S4, Mamba) and the recent RNN-based xLSTM, against the baseline MeMOTR. All SambaMOTR models use our additional contributions, such as MaskObs and the efficient training recipe for longer sequences. We trained the models on DanceTrack and report their tracking performance (HOTA) on the DanceTrack test set. The evaluation also includes the following metrics:
- Peak memory consumption (GB) after 100 iterations of multi-GPU training without activation checkpointing.
- Model throughput (images per second) at inference time, measured on a single NVIDIA RTX 3090 for a randomly selected DanceTrack video (0054).
- Model latency (ms) at inference time, measured on a single NVIDIA RTX 3090 for the same video.
- Number of model parameters (millions, M).
- The tracking performance (HOTA).

| **Method** | **Type** | **Sequence Model** | **Sync** | **Peak Memory (GB)** | **Throughput (img/s)** | **Latency (ms)** | **Parameters (M)** | **HOTA** |
|------------|----------|---------------------|----------|-----------------------|------------------------|------------------|---------------------|----------|
| MeMOTR     | -        | -                   | -        | 15.9                 | 18.5                  | 54.0            | 42.9               | 63.4     |
| **SambaMOTR**   | RNN      | xLSTM               | -        | 17.0                 | 18.7                  | 53.4            | 49.5              | 65.2     |
|            | RNN  | xLSTM               | ✓        | 17.5                 | 18.6                  | 54.0            | 55.8               | 66.7     |
|            | SSM      | S4                  | -        | 15.9                 | 18.7                  | 53.3             | 42.4               | 64.7     |
|            |  SSM        | S4                  | ✓        | 16.1                 | 18.6                  | 53.4             | 43.8               | 65.5     |
|            |  SSM        | Mamba               | -        | 15.9                 | 18.6                  | 53.7            | 42.0              | 66.1     |
|            |  SSM        | Mamba               | ✓        | 16.4                 | 18.6                  | 54.1             | 43.6               | **67.2** |

Our results demonstrate that Samba’s long-range set-of-sequences model introduces only a negligible increase in peak memory consumption, throughput, latency, and parameter count compared to MeMOTR’s heuristics-based memory model. All sequence models use a hidden state dimension of 1; increasing the state dimension leads to a proportional increase in synchronization parameters. Among the sequence models, Mamba proves to be the most parameter-efficient while achieving the best performance. Even without synchronization, Mamba outperforms MeMOTR with fewer parameters. 

Crucially, these findings confirm that the observed performance improvements arise from the principled design of our propagation module, which effectively leverages long-range trajectory information, rather than from increased model capacity.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary  
This paper proposes SambaMOTR based on a linear-time set-of-sequences model for multiple object tracking. It processes multiple tracklets simultaneously by synchronizing multiple selective state spaces. The method leverages temporal information about object movements to model historical data and handle occlusions, enabling the modeling of long-range dependencies, implicit motion, and appearance relationships between tracklets. SambaMOTR achieves state-of-the-art performance on the DanceTrack, BFT, and SportsMOT datasets.

Strengths & weaknesses  
All reviewers recognized the strengths of the paper are strong experimental results and innovative ideas, and the weaknesses are lacking some technique details and more experimental results (visual comparisons, efficiency evaluations). After rebuttal, the weaknesses are well addressed by the authors. The final ratings are 8,8,6 and all reviewers recommend accepting this paper.

The AC concurs with the reviewers and recommends accepting this paper. In addition, the authors are encouraged to incorporate the suggested technical details and efficiency evaluations to further enhance the work.