Samba: Synchronized Set-of-Sequences Modeling for Multiple Object Tracking
Authors: Mattia Segu
Luigi Piccinelli
Siyuan Li
Yung-Hsu Yang
Luc Van Gool
Bernt Schiele
Venue: ICLR 2025 Spotlight (2025)
Keywords: Synchronized sequence modeling; multiple object tracking
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: SambaMOTR is a tracking-by-propagation method that (i) leverages long-range dependencies and (ii) synchronizes memory across object trajectories to better estimate future track queries and handle occlusions.

Abstract: Multiple object tracking in complex scenarios - such as coordinated dance performances, team sports, or dynamic animal groups - presents unique challenges. In these settings, objects frequently move in coordinated patterns, occlude each other, and exhibit long-term dependencies in their trajectories. However, it remains a key open research question on how to model long-range dependencies within tracklets, interdependencies among tracklets, and the associated temporal occlusions. To this end, we introduce Samba, a novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet. Samba autoregressively predicts the future track query for each sequence while maintaining synchronized long-term memory representations across tracklets. By integrating Samba into a tracking-by-propagation framework, we propose SambaMOTR, the first tracker effectively addressing the aforementioned issues, including long-range dependencies, tracklet interdependencies, and temporal occlusions. Additionally, we introduce an effective technique for dealing with uncertain observations (MaskObs) and an efficient training recipe to scale SambaMOTR to longer sequences. By modeling long-range dependencies and interactions among tracked objects, SambaMOTR implicitly learns to track objects accurately through occlusions without any hand-crafted heuristics. Our approach significantly surpasses prior state-of-the-art on the DanceTrack, BFT, and SportsMOT datasets.

Published as a conference paper at ICLR 2025
SAMBA: SYNCHRONIZED SET-OF-SEQUENCHE MODELING FOR MULTIPLE OBJECT TRACKING
Mattia Segu 1.3 Luigi Piccinelli', Siyuan Li', Yung-Hsu Yang' Luc Van Gool Bernt Schiele
1 ETH Zurich, 2 INSAIT, Sofia University, St. Kliment Ohridski,
Max Planck Institute for Informatics, Saarland Informatics Campus petps://sambamoott www.rr.githubibi.o.
ABSTRACT
Multiple object tracking in complex scenarios such as coordinated dance perfor- mances, team sports, dynamic animal groups presents unique challenges. In these settings, objects frequently move in coordinated patterns, occlude each other, and exhibit long-term dependencies in their trajectories. However, it remains key open research question on how to model long-range dependencies within track- lets, interdependencies among tracklets, and the associated temporal occlusions. To this end, we introduce Samba, a novel linear-time set-of-sequences model de- signed to jointly process multiple tracklets by synchronizing the multiple selec- tive state-spaces used to model each tracklet. Samba autoregressively predicts the future track query for each sequence while maintaining synchronized long-term memory representations across tracklets. By integrating Samba into a tracking- hy-propagation framework, we propose SambaMOTR, the first tracker effectively addressing the aforementioned issues, including long-range dependencies, track- let interdependencies, and temporal occlusions. Additionally, introduce ef- fective technique for dealing with uncertain observations (MaskObs) and an ef- ficient training recipe scale SambaMOTR longer sequences. By modeling long-range dependencies and interactions among tracked objects, SambaMOTR implicitly learns to track objects accurately through occlusions without any hand- crafted heuristics. Our approach significantly surpasses prior state-of-the-art on the DanceTraae BFT, and SportsMOT datasets.
INTRODUCTION
Multiple object tracking (MOT) involves detecting multiple objects while keeping track of individual instances throughout video stream. It is critical for multiple downstream tasks such as sports analysis, autonomous navigation, and media production (Luo et al., 2021). Traditionally, MOT methods are validated relatively simple settings such surveillance datasets (Milan al. 2016) where pedestrians exhibit largely linear motion and diverse appearance, and rarely interact with each other in complex ways. However, in dynamic environments like team sports, dance performances, or animal groups, objects frequently move in coordinated patterns, occlude each other, and exhibit non-linear motion with long-term dependencies in their trajectories (Fig. [ Modeling the long- term interdependenncies between objects in these settings, where their movements are often synchronized or influenced by one another, remains an open problem that current methods fail to address.
Current tracking-by- -eeeeeccionn methods (Bewley et al. 2016 Wojke et al. 2017 Zhang et al. 2022 Cao al. 2023) often rely on heuristics-based models like the Kalman filter independently model the trajectory of objects and predict their future location. However, these methods struggle with the non-linear nature of object dynamics such motion, appearance, and pose changes. Tracking-by propagation (Sun et al.) 2020 Meinhardt et al. 2022 Zeng et al., 2022) offers an alternative by modeling tracking
end-to-end autoregressive object detection problem, leveraging detection transformers (Carion et al.) 2020) Zhu et al. 2020) to propagate track queries over time. Their flex- ible design fostered promising performance in settings with complex motion, pose, and appearance patterns, such as dance (Sun et al.] 2022), sports (Cui et al., 2023), and bird (Zheng et al.) (024) tracking datasets. However, such methods only propagate the temporal information across adjacent frames, failing to account for long-range dependencies. MeMOTR (Gao & Wang 2023) attempts a preliminary solution this problem by storing temporal information through external heuristics-

---

Published as a conference paper at ICLR 2025
(a) DanceTrack (Sun et al. 2022 (b) BFT (Zheng et al. 2024
(c) SportsMOT CCui et al. 2023
Figure/ 1: Tracking multiple objects in challenging scenarios such as coordinated dance perfor- mances (a); dynamic animal groups (b), and team sports (c) - requires handling complex interac- tions, occlusions, and fast movements. As shown in the tracklets above, objects may move in co- ordinated patterns and occlude each other. By leveraging the joint long-range dependencies in their trajectories, SambaMOTR accurately tracks objects through time and occlusions.
based memory. However, its use of an exponential moving average (EMA) compress past history results in a suboptimal temporal memory representation, as discards fine- lne-grained long-range de- pendencies that are crucial for accurate tracking over time. Moreover, by processing each tracklet independently and overlooking tracklets interaction, current methods cannot accurately model ob- jects' behavior through occlusions, resorting to naive heuristics to handle such cases: some (Zhang et al. 2023 Gao & Wang 2023) freeze the track queries during occlusions and only rely on their last observed state during prolonged occlusions; others (Zeng et al., 2022) delegate occlusion manage- ment to the propagation module, which fails to estimate accurate track trajectories as it only propa- gates information across adjacent frames and does not account for historical information. We argue that effective long-term memory and interaction modeling allow for more accurate inference of oc- cluded objects' behavior in complex environments, such as team sports or dance performances, by leveraging past information and understanding joint motion patterns.
To address these shortcomings, we propose Samba a novel linear-time set-of-sequences model that processes a set of sequences (e.g. multiple tracklets) simultaneously and compresses their his- tories into synchronized long-term memory representations, capturing interdependencies within the set. Samba adopts selective SSMs from Mamba (Gu & Dao 2023) to independently model all tracklets, compressing their range histories into hidden states. We then propose to synchro- nize these memory representations across tracklets at each time step to account for interdependen- cies (e.g. interactions among tracklets). We implement synchronization via self-attention mecha- nism (Vaswani et al. 2017) across the hidden states of all sequences, allowing tracklets to exchange information. This approach proves beneficial in datasets where objects move in coordinated patterns (Tabs. D to 3p. The resulting cet-of-sequences model, Samba, retains the linear-time complexity of SSMs while modeling the joint dynamies of the of tracklets.
By integrating Samba into a brekkig-by--roppgeee SambaMOTR, an end-to-end multiple object tracker that models long-range dependencies and inter- actions between tracklets to handle complex motion patterns and occlusions in a principled manner. munsformer-basee object detector with novel set-of-queries propa-
framework Zeng et al. 2022), we present
SambaMOTR complements gation module based on Samba, which accounts for both individual tracklct histories and their inter- actions when autoregressively predicting the next track queries.
Additionally, some queries result in uncertain detections due to occlusions or challenging scenarios (see Fig. 2 Occlusion). To prevent these detections from compromising the memory representation and accumulating errors during query propagation with Samba, we propose MaskObs. MaskObs blocks unreliable observations from entering the set-of-queries propagation module while updating the corresponding hidden states and track queries using only the long-term memory of their track- lets and interactions with confidently tracked objects. Unlike previous methods that freeze track queries during occlusions, MaskObs leverages both temporal and spatial context- - i.e. past behavior and interdependencies with other tracklets more accurately predict an object's future state. Con- sequently, SambaMOTR tracks objects through occlusions more effectively (Tab, line d).
Samba is named for its foundation on the Synchronization of Mamba's 's selective state-spaces (Gu & Dao 2023). Its name reflects the coordinated motion of tracklets (Fig. D. much like the synchronized movements in the samba dance. By synchronizing the hidden states across sequences, our approach is disentangled from Mamba's 's selective state-space models (SSMs) and can, in principle, he applied to any sequence model that includes intermediate memory representation, such as other SSMs recurrent neural networks.
2

---

Published as a conference paper at ICLR 2025
Long-range Propagation
Sambb
Set-af -t---sseseeces
Set of of-sequences Model
Model
Queries
Occlusion
Decoder
Decoder
Decoder
Dipoctoo OoF
Samba List
Encoder
Synchroizaatin
Query
Track
Track Memoor
Enpty Marnory
Underacted
Input Video Stream
Figure 2: Overview of SambaMOTR. SambaMOTR combines a transformer- object detector with a cet-of-sequencess Samba model. The object detector's encoder extracts image features from each frame, which are fed into its decoder together with detect and track queries to detect newborn objects or re-detect tracked ones. The Samba sequu--sequencess model is composed of multiple syn- chromized Samba units that simultaneously process the past memory and currently observed output queries for all tracklets to predict the next track queries and update the track memory. The hidden states of newborn objects are initialized from zero values (barred squares). In case of occlusions or uncertain detections, the corresponding query is masked (red cross) during the Samba update.
Finally, introduce efficient training recipe to scale SambaMOTR longer sequences by sam- pling arbitrarily long sequences, computing tracking results, and applying gradients only on the last five frames. This simple strategy enables to learn longer-range dependencies for query propaga- tion, improving the tracking performance while maintaining the same GPU memory requirements
previous methods (Zeng et al. 2022 Gao & Wang 2023).
validate SambaMOTR on the challenging DanceTrack (Sun el. 2022), SportsMOT (Cui et al, 2023). and BFT (Zheng et al.] (024) datasets. Owing our contributions, we establish new state of the art on all datasets. We summarize them as follows: (a) we introduce Samba, our novel linear- time set-of-sequences model based synchronized SSMs; (b) introduce SambaMOTR, the first tracking-by backkng--bypropea propaggtiion method that leverages past tracklet history principled manner to learn long-rat range dependencies, tracklets interaction, and occlusion handling; (c) introduce MaskObs, a simple technique for dealing with uncertain observations in SSMs and an efficient training recipe that enables learning stronger sequence models with limited compute.
RELATED WORK
Tracking-h baucking-by-detectiinn bettetciion. Tracking-byy beetteig is popular paradigm in MOT, consisting of an ob- ject detection stage followed by data association to yield object trajectories throughout video. Mo- tion and appearance cues typically utilized to match detections tracklets through hand-crafted heuristics. The motion- based tracker SORT (Bewley et al.) 2016) relies on Intersection over Union (IoU) to assign the tracklet locations predicted with Kalman filter object detections. Byte- Track (Zhang et al. 2022) introduces a two-stage matching scheme to associate low-confidence de- tections. OC-SORT (Cao et al.) 2023) models non-linear motion by taking care of noise accumula- tion under occlusion. Alternatively, appearance descriptors can be used alone (Pang et al. 2021]
al.
2022
2024a) in combination with motion (Wojke al.] 2017 Wang et al.) [2200] Zhhng
et al.
2021 Segu et al. 2024 [i et al. (024b) to match detections to tracklets according to a similar-
ity metric. Due the disentangled nature of the two stages, tracking-by-ded by-eettcto methods histori-
cally leveraged state-of-the-
object detectors to top the MOT challenge (Dendorfer et al.] 2021) However, by relying hand-crafted heuristics, such methods struggle with non-linear motion and
appearance patterns (Sun et al. 2022; Cui et al. 2023 Zheng et al. 2024 Li al.) 2023). and re-
quire domain-specifie hyperparameters (Segu al. 2023 Liu
2023)
Recent transformer-

---

Published as a conference paper at ICLR 2025
based methods have eased the burden of heuristics. TransTrack (Sun et al. 2020) decodes track and detect queries with siamese transformer decoders and associates them with simple IoU matching. McMOT (Cai et al., 2022) fuses large memory bank into tracklet descriptor with transformer- based memory aggregator. However, it requires storing and processing with quadratic complexity the historical information from up 27 past frames. GTR (Zhou et al. 2022) matches static trajec- tory queries to detections to generate tracklets, but fails to model object motion and tracklet interac- tion. In contrast, SambaMOTR implicitly learns motion, appearance, and tracklet interaction mod- els by autoregressively predicting the future track queries with our set-of-sequences model Samba.
Tracking-by-pree byppropgttioon Recent work (Meinhardt et al., 2022 Zeng al. 2022) introduced more flexible and end-to-end trainable trucking-by buacking-b-prrppggttin design that treats MOT as an autore- gressive problem where object detection and query propagation are tightly coupled. Leveraging the transformer- basee Deformable DETR Zhuu et al., 2020) object detector, TrackFormer (Meinhardt et al., 2022) and MOTR (Zeng et al. 2022) autoregressively propagate the detection queries through time to re-detect (track) the same object in subsequent frames. MOTRv2 (Zhang et al. 2023) lever- ages a pre-trained YOLOX object detector to provide anchors for Deformable DETR and boost its detection performance. However, these approaches only propagate queries across adjacent frames, failing to fully leverage the historical information. McMOTR (Gao & Wang 2023) first attempts to utilize the temporal information in tracking- by oy-ropaggttion by aggregating long- (EMA of a track- let's queries through time) and short-term memory (fusion of the output detect queries across the last two observed frames) in a temporal interaction module. By collapsing the tracklet history with an EMA and by freezing the last observed state of track queries and memory under occlusions, MeMOTR cannot accurately estimate track query trajectories through occlusions. Finally, by mod- cling query propagation independently for each tracklet, it does not model tracklet interaction. In contrast, our proposed Samba set-of-sequences model relies on individual SSMs to independently model each tracklet as sequence and it synchronizes the memory representations across the set of tracklets to enable tracklet interaction. Equipped with Samba, SambaMOTR autoregressively pre- diets future queries aware of long-range dynamics and of other tracklets' motion and appearance.
3 PRELIMINARIES
Before introducing SambaMOTR (Sec. we present the necessary background and notation on selective state-space models (Sec. 3.I and tracking-by proppagattiion (Sec. (.2)..
3.1 SELECTIVE STATE- MODELS
Inspired by classical state-space models (SSMs) (Kalman 1960), structured SSMs (S4) (Gu et al, 2021 introduce sequence model whose computational complexity scales linearly. rather than quadratically, with the sequence length. This makes S4 principled and efficient alternative to transformers (Vaswani et al. (017). By further introducing selection mechanism i.e. rendering the SSM parameters dependen- Mamba (Gu & Dao 2023) can model time-variant systems, bridging the performance gap with transformers (Vaswani et al. 2017).
We here formally define selective SSMs (S6) (Gu & Dao 2023). Let s(t) be the in- put signal at time f, h(t)) the hidden state, and y(t) the output signal. Given the system A, control B, and output C matrices, we define the continuous linear time-variant SSM in Eq. ① The discrete-time equivalent system (Eq. (2) of the defined SSM is obtained through a discretization rule. The chosen discretization rule is typically the zero-order hold (ZOH)
model: (() = (t) B(t) = (t))) (exp( (tt) I) (t))
A'(1)= (L))(() B())(())
(I)
h. A(t)h,,1 + B(t),,
(2)
y(1 = C(())())
yi = C()),,
Tt. he Yt the observations sampled at time of the input signal xxt), hidden state h(tt), and
output signal y(t.. While S4 learns linear iime-invariant (LTT) system with Att)= A. A(t) = A, (t)) B and C(t)= C. S6 introduces selectivity to learn a time-variant system by making A (t),
(()) and C(t) dependent on the input z(!), i.e. A ( + & (x(()). B(I) = sn (=))((),, C(I)= sc (I)),, where TA- = softplus, and 84. sB, se are learnable linear mappings.
In this paper, we propose to treat tracking-by-poopaane proppagatio as sequence modeling problem. Given the discrete sequence of historical track queries for certain tracklet, our query propagation mod-
4

---

Published as a conference paper at ICLR 2025
leverages SSMs to account for the historical tracklet information in prin- cipled manner. By recursively compressing all tracklet history into a long-term memory, Samba's complexity scales linearly with the number of frames, enabling efficient training long sequences while processing indefinitely long tracklets at inference time.
ule Samba (Sec. 4.2)
3.2 BAICKING-BY-ROPPACEE
methods alternate between a detection stage and propagation stage, rely- ing on a DETR-like (Carion et al. 2020) transformer object detector and a query propagation mod- ule. At time step I, the backbone and transformer encoder extract image features for a frame 4 The detection stage involves feeding both a fixed-length set of learnable detect queries Quet to the transformer decoder detect newborn objects and variable-length set of propagated track queries Qi to re-detect tracked ones. At time & = O, the set of track queries is empty, i.e. Qo = E = O. Detect and track queries [Q)eet _Q[ck] interact in the decoder with image features to generate the cor- responding output embeddings [Ede,, E and bounding box predictions D D4ckj. We denote the set of embeddings corresponding to newborn objects as Ex and E = [E E as the set of embeddings corresponding to the tracklets St active time t. During the propagation stage,
query propagation module (--) typically takes as input the set of embeddings Eick and outputs fined tracked queries Q O (Etck) to re-detect the corresponding objects in the next frame.
Although prior work failed to properly model long-range history and tracklet interactions Zeng al. 2022 Gao & Wang, 2023] Meinhardt et al., 2022). and given that multiple objects often move synchronously (Fig. we argue that the future state of objects in a scene can be better predicted by (i) considering both their historical positions and appearances, and (ii) estimating their interactions. In this work, we cast query propagation as a set-of-sequences modeling problem. Given a set of multiple tracklets, we encode the history of each tracklet in a memory representation using a state- space model and propose memory synchronization to account for their joint dynamics.
METHOD
In this section, we introduce SambaMOTR, an end-to-end multiple object tracker that combines transformer-based object detection with our set-of-sequences model Samba to jointly model the long-range history of each tracklet and the interaction across tracklets to propagate queries. First, in framework and motivate the need
Sec. we provide background on the maicking-y--roopgatte for better modeling of both temporal information and tracklets interaction. Then, we describe the SambaMOTR architecture (Sec. 4.1 and introduce Samba (Sec. 4.2), our novel set-of-sequences model based on synchronized state spaces that jointly models the temporal dynamics of a set of se- quences and their interdependencies. Finally, in Sec. 4.3 we describe SambaMOTR's query propa- gation strategy based on Samba, our effective technique MaskObs to deal with occlusions in SSMs, a recipe to learn longerange sequence models with limited compute, and our simple inference pipeline.
4.1 ARCHITECTURE
-yropggatttoooattiiinn methods (Meinhardt et al. 2022, Zeng et al. 2022, Gao & Wang 2023), the proposed SambaMOTR architecture (Fig. is composed of a DETR- like (Carion ct al. 2020) object detector and a query propagation module. As object detector, we use Deformable- DETR (Zhu et al. 2020) with ResNet-50 (He et al.) 2016) backbone followed by transformer encoder to extract image features and a transformer decoder to detect bounding boxes from a set of detect and track queries. As query propagation module, we use our set-of-sequences model Samba. Each sequence is processed by a Samba unit synchronized with all others. Samba unit consists of two Samba blocks (Sec. 4.2) interleaved with LayerNorm (Ba et al. 2016 and residual connection.
Similar to other
4.2 SAMBA: SYNCHRONIZED STATE-SPACK MODELS FOR SET-OF-SEQUENCES MODELING
Set-of-sequences modeling involves simultaneously modeling a set of temporal sequences and the interdependencies among them. In MOT, set-of- se-oof-squences models can capture long-range temporal relationships within each tracklet as well as complex interactions across tracklets. To this end, we introduce Samba, linear-time set-of-sequences model based on the synchronization of multiple state-space models. In this paper, we leverage Samba as a set-of-queries propagation network to jointly model multiple tracklets and their interactions in tracking- macking-by-propant bypropaggtiion framework.
5

---

Published as a conference paper at ICLR 2025
Long-Term Memory Update (LTMU)
Outputs
y
ył
y
Synchronized SSM
A
A
4
Selccton
Updateed Memory
x
Memory Synchronization (MS)
Rg-1
ng
Crilasr
Past Memoory
LTMU
Synchronized Memory
Memory
MNCS
LTMU
LTMU
Output Update (OU)
y!
Observations
Sc
x
Figure 3: Synchronized State-Space Models. We illustrate set of k synchronized SSMs. A Long- Term Memory Update block updates each hidden state hi-1 based on the current observation æt, resulting in the updated memory hi The Memory Synchronization block then derives the synchro- nized hidden state hi, which is fed into the Output Update module to predict the output y/-
Synchronized Selective State-Space Models. Let z be the discrete observation at time of the i- th input sequence from a set of sequences S. We choose selective SSMs (Gu & Dao 222)2023) to model each sequence through hidden state h& (Eq. (3a)) encoding long-term memory, but approach applies any other SSM. Given the memory hi define long-term memory update (LTMU) function that updates RE_ based on the current observation æf. resulting in the updated memory hf. We propose memory synchronization (MS) function I-)s(-) that produces set of synchronized hidden states fig Vi S, modeling interactions across the set of sequences S (Eq. (3b)). Finally, derive the output yt for each sequence through the output update (OU) function (E4. (3cb).
LTMU:
h A (t) (t))
(3a)
MS:
h Ties (A1)
(3b)
OU:
v C(t)
(3c)
An ideal memory synchronization function should be flexible regarding the number of its inputs (hidden states) and equivariant to their order. Thus. we propose to define the memory synchro-
nization function I(.)) [FF x NNN as a set of Nayne stacked blocks with multi-
head self-attention (MHSA) (Vaswani et al. 2017) followed by a feed-forward network (FFN), A
schematic illustration of the proposed synchronized state-space model layer is in Fig.
Set-of-sequences Model. Refer to App. B.I for a detailed description of how our synchronized SSM is used in the Samba units that model each sequence in the set-of-sequences Samba model.
4.3 SAMBAMOTR: END-TO-END TRACKING-BY-PROPAAGE
WITH SAMBA
Query Propagation with Samba. As described in Sec. 3.2 the query propagation module O(-) takes as input the decoder output embeddings Eyck and outputs refined track queries Qtck (Étck). SambaMOTR extends this paradigm by accounting for the temporal information and tracklets interaction. In particular, tracklet into a hidden state hú and synchronize it across tracklets to derive the synchronized mem- ory ht. Notice that hj = 0 for a newborn object i. At time t, we first enrich the detector out- put embeddings Etck with position information by summing to them sine-cosine positional encod- ings P (F(-) of the corresponding bounding boxes coordinates Dick to implicitly model object mo- tion and appearance, obtaining the set of input observations X Êtck + (E(D)))). Given the set of input observations Xick and past synchronized hidden states Å.-1 for all tracklets in the set
our Samba module (--) to compress the history of each

---

Published as a conference paper at ICLR 2025
St of tracklets time feed them into Samba (Y, H2) (( HH-1) to obtain the out- put emheddings Y; and updated synchronized hidden states H, Finally, we the output embed- dings Y2 and a learnable mapping 8y to predict a residual AQ = sy (Y,) to the past track queries Qt and generate the new ones, i.e. Qi = Qf AQ! By recursively unfolding this process over time, SambaMOTR can track multiple objects while compressing indefinitely long tracklet his- tories into their long-term memory representations, effectively modeling object motion and appear- ance changes, and tracklets interactions.
may occasionally deal with occluded objects or uncertain detections. Given function c-) :) to estimate the predic- tive confidence of an input observation zij we propose MaskObs, a strategy to handle uncertain ob- servations. MaskObs masks uncertain observations from the state update (Eq. (4), thus defining the system dynamics solely based on its history and the interdependencies with other sequences:
MaskObs:
Dealing with Uncertain Observations.
beokking-b--roppaggeee
hi A (t)h B (t)x) [[con (x)) > Tmaas]]
(4)
1||] is the indicator function, and Tmakk is the confidence threshold, e.g. Taask = 0.5. We implement conf((r)) as the predictive confidence con (rit) of the corresponding bounding box di. In our work, this design choice allows us to better model query propagation through occlusions (Tab. 4 line b).
Efficiently Learning Long-range Sequence models. Previous MOTR-like approaches are trained end-to-end on a sequence of consecutive frames sampled at random intervals. While Sam- baMOTR's s set-of- model Samba already shows impressive generalization performance to long sequences at inference time (Tab. 4 line c). we propose to train on longer sequences (i.e. 10 frames) and only apply gradients to the last 5 frames (Tab. 4 line d). We hypothesize that this strat- egy allows us to learn better history compression for late observations in a sequence, resulting in even better tracking performance while being trained with similar GPU memory requirements. A schematic illustration of our training scheme proposal is in Fig. B
Inference Pipeline. At given time step t, we jointly input the learnable detect queries Qdet and track queries Qf (Qi = 0) into the transformer decoder to produce detection embeddings Edet and tracking embeddings Etck and the corresponding bounding boxes. Each detection bounding box with a confidence score higher than a threshold Tdet will initialize a newborn track Edet, We then propagate the embeddings of newborn Edet and tracked Etck objects together with the track memory H to generate the updated track queries Qita and synchronized memory H141. To deal with occlusions and lost objects,
consider individual track query 4. i.track inactive if its corresponding bounding box confidence cd) at time t is lower than Ttrack- If a track query is inactive for more than Nmiss frames, it is deemed lost and dropped.
Unlike McMOTR (Gao & Wang 2023), which does not update the track embedding and long- term memory for object with low detection confidence
time step t, approach employs a principled query propagation scheme that can hallucinate likely track query trajectories under occlusions by relying its past history attending to other trajectories. Thus, we always update the memory and track query for any tracklet even when occluded as long as it is not deemed lost.
5 EXPERIMENTS
In this section, we present experimental results to validate SambaMOTR. We describe our evaluation protocol (Sec. 5.1 and report implementation details (Sec. 5.2). We then compare SambaMOTR to the previous state-of-the-art methods (Sec. 5.3) and conduct an ablation study (Sec. 5.4) on the method components. We provide more ablations in the appendix. Qualitative results can be found in Fig. aad at the anonymous project page https: http::///onnnmuuuuuu sattinouuussmaaaaaaaaaaaaaaaaaae wig.tu..bb.....
5.1 EVALUATION PROTOCOL
Datasets. To evaluate SambaMOTR, we select variety of challenging datasets exhibiting highly non-linear motion in crowded scenarios, with frequent occlusions and uniform appearances. All datasets present scenes with objects moving synchronously. Thus, they represent a suitable bench- mark for assessing the importance of modeling tracklet interaction. DanceTrack (Sun et al.] 2022) multi-human tracking dataset composed of 100 group dancing videos. The Bird Flock Track- ing (BFT) (Zheng et al. 2024) dataset includes 106 clips from the BBC documentary series Earth- flight (Downer & Tennant] 2011). 1). SportsMOT (Cui et al. 2023) consists of 240 video sequences

---

Published as a conference paper at ICLR 2025
from basketball, volleyball, and soccer scenes. Due to the highly linear motion in MOT17 (Milan ct al., 2016). its small size (only videos), and the subsequent need for training on additional de- tection datasets, end-to-end tracking methods do not provide additional advantages over more naive Kalman- filter--it----eeedd methods. We report its results in the Appendix.
Metrics. Following prior work, we measure the overall tracking performance with the HOTA (Luiten et al., 2021) metric and disentangle detection accuracy (DetA) and association accu- racy (AssA), We report the MOTA (Bernardin & Stiefelhagen 2008) and IDFI (Ristani et al.) 2016) metries for completeness. Since our objective is improving association performance and the overall tracking quality, HOTA and AssA are the most representative metrics.
5.2 IMPLEMENTATION DETAILS
Following prior works (Gao & Wang 2023 Zhang al.. 2023), apply random resize, random crop, and photometric augmentations as data augmentation. The shorter side of the input image is resized to 800 preserving the aspect ratio, and the maximum size is restricted to 1536. For a fair
comparison with prior work (Sun et al.] 2020 Zeng et al. 2022 Gao & Wang (023), we use the Deformablc-DETR (Zhu et al. (020)) object detector with Re (He et al. (016)) and initialize
it from COCO (Lin et al. 2014) pre-trained weights. Similar to MeMOTR (Gao & Wang 2023), inject track queries after one decoder layer. We our experiments on 8 NVIDIA RTX 4090 GPUs, with batch size per GPU. Each batch element contains a video clip with 10 frames, and we compute and backpropagate the gradients only over the last 5. We sample uniformly spaced frames at random intervals from 1 to 10 within each clip. We utilize the Adam optimizer (Loshchilov & Hutter 2017) with initial learning rate of 2.0 x 10 For simplicity, Tdet Ttrack Tmask 0.5. Nmiss 18 35, 20, and 50 on Dance Track, BFT, and SportsMOT, respectively, due to different dataset dynamics. On DanceTTrack (Sun al. 2022).
train SambaMOTR for 15 epochs on the training set and drop the learning rate by a factor of 10 at the 10th epoch. On BFT (Sun et al. 2022). we
train for 20 epochs and drop the learning rate after 10 epochs. On SportsMOT (Cui et al.
2023
train for 18 epochs and drop the learning rate after 8 and 12 epochs. SambaMOTR's inference runs at 16 FPS single NVIDIA RTX 4090 GPUs.
5.3 COMPARISON WITH THE STATE OF THE ART
We compare SambaMOTR with multiple bockiig-by-deteettinn and booking-b--rropggation ap- proaches on the DanceTrrack (Tab. []. BFT (Tab. 2 and SportsMOT (Tab. 3 datasets. All meth- ods are trained without using additional datasets. Since trackers use various object detectors with different baseline performance, we report the detector used for each method. For fair comparison. report the performance of 2020), marking the best in bold. We underle the overall best result.
methods with Deformable DETR (Zhu al.) methods often use the stronger YOLOX-X (Ge et al. 202T). but tracking by propagattion consistently outper- forms them, with SambaMOTR achieving the highest HOTA and AssA across all datasets.
PanceTracks The combination of highly irregular motion and crowded scenes with frequent occlu- sions and uniform appearance historically made DanceTrack challenging for cayking--y-ddetcctoon methods. Despite their higher DetA when using the strong object detector YOLOX-X (Ge et al. 2021), baacking-y--poopgglle and SambaMOTR). SambaMOTR sets a new state of the art, with +3.8 HOTA and +5.2 AssA on the strongest competitor MeMOTR. Our method owes this performance improvement to its bet- ter modeling of the historical information, our effective strategy learn accurate sequence models through occlusions, and our modeling of tracklets interaction (group dancers move synchronously).
significantly outperforms them (see MeMOTR (Gao & Wang 2023
BFT. Bird flocks present similar appearance and non-linear motion. For this reason, OC-SORT works best among tracking-by-des deettectionn methods. Nevertheless, bird flocks move synchronously, and interaction among tracklets is essential cue for modeling joint object motion. Thanks to proposed sequence models synchronization, SambaMOTR achieves +2.8 HOTA and +4.9 AssA over the best competitor overall (OC-SORT), and an impressive +6.3 HOTA and 12.5 improve- ment over the previous best tracking-by-propagati method TrackFormer (Meinhardt et al. 2022).
SportsMOT. Sports scenes typically present non-linear motion patterns that the Kalman filter strug- gles to model, hence the underwhelming performance of ByteTrack (Zhang et al. 2022). For this reason, trackers that model non-linear motion either explicitly (OC-SORT (Cao al. 2023) or im-
8

---

Published as a conference paper at ICLR 2025
Table State-of-the-art comparison on DanceTrrack (Sun al. 2022) method in bold; best overall underlined.
without additional training
data. Best
Methods
Detector
HOTA
AssA
DetA
MOTA
by-delection:
FairMOT (Zhag et al
2021
39.7
23.8
66.7
40.8
82.2
CenterTraa Track (Zhou al
2020
CenterNet DDan et
2019
41.8
22.6
78.1
35.7
86.8
TraDeS ct al 2021
43.3
25.4
74.5
41.2
86.2
Trans Sun ct
2020
Deformable DETR Zhu al
2020
45.5
27.5
75.9
45.2
88.4
GTR Zhou et
2022
CenterNet2 Zhou et
2021
48.0
31.9
72.5
50.3
84.7
ByteTrack ZZhang el al. 2022
47.7
32.1
71.0
53.9
89.6
QDTrack Pang ct ai. 202T
54.2
36.8
80.1
50.4
87.7
YOLOX-X Ge el.
2021
OC-SORT (Cao et a 2025
55.1
38.3
80.3
54.6
92.0
C-BloU (Yng et 2023
60.6
45.4
81.3
61.6
91.6
Trapking-by-roopccetionn MOTR (Zeng et al. 2022
54.2
40.2
73.5
51.5
79.7
McMOIR (Gão & Wang
2023
Deformable DETR
Zhu ct al. 2020
63.4
52.3
77.0
65.5
85.4
SambaMOrR (ours)
67.2
57.5
78.8
70.5
88.1
Table State-of-the-art comparison on BFT Zheng al. 2024 without additional training data. Best
method in bold; best overall underlined.
Method
Detector
HOTA
AssA
DetA
IDFI
MOTA
Trucking-by-den by-delectio:
FairMOT (Zhang et al.
2021
40.2
28.2
53.3
41.8
56.0
CenterNet Duan
2019
CenterTrack Zhou et al.
2120
65.0
54.0
58.5
61.0
60.2
SORT WWojke al al.
2017)
61.2
62.3
60.6
77.2
75.5
ByteTrrack Zhang al 1022
YOLOX-X Ge al.
2021
62.5
64.1
61.2
82.3
77.2
OC-SORT a0 et al
2023
66.8
68.7
65.4
79.3
77.1
TransCenter Xu
2022
60.0
61.1
66.0
72.4
74.1
Deformable DETR
Zhu ct al.
2020
TransTrack Sun
2020
62.1
60.3
64.2
71.4
71.4
Tracking-by-propeqqatio
TrackForner (Mcinhardt et al.
2022
Deformable DETR 7hu al.
2020
61.1
66.0
72.4
74.1
SambaMOTR (ours)
73.6
66.0
81.9
plicitly (TransTrack (Sun al., 2020)) perform well. Notably, our tracking-br Backing--by-peee by-roppagation Sam- baMOTR enables implicit joint modeling of motion, appearance, and tracklet interaction, obtaining the best HOTA overall (69.8) despite the lower DetA of our Deformable-Dei DETR detector compared to OC-SORT's YOLOX-X. Moreover, SambaMOTR exhibits significant +1.6 AssA over the best tracking-by- -proopggattoooaaatitiio method and an impressive +4.6 AssA over OC-SORT.
5.4 ABLATION STUDIES
We ablate the effect of each component of our method in Tab, as detailed in Sec. 4 and illustrated Fig. Additional ablation studies are presented in App-
SSM. Line (a) shows the benefits of sequential representation for tracking. We use a vanilla se- quence model, such as Mamba, as the baseline for query propagation, establishing a robust founda- tion that outperforms McMOTR's 's EMA-base history and temporal attention module.
MaskObs. Handling track queries during occlusions (line b) with MaskObs which masks uncertain observations from the state update and relies on long-term memory and interactions with visible tracklets leads significant overall improvements (+1.3 HOTA), highlighting the effectiveness of managing occluded objects.
Sync. Making tracklets aware of each other through synchronization mechanism (line sults in over 1% improvement across all metrics, demonstrating how modeling interactions between tracklets enhances tracking accuracy by capturing joint dynamics and coordinated movements.

---

Published as a conference paper at ICLR 2025
Table State-of-the-art comparison on SportsMOT (Cui al. 2023
without additional training
data. Best
method in bold; best overall underlined.
Methods
Detector
HOTA
AssA
DetA
MOTA
Tracking-by-den by-delection:
FairMOT Zhang et ul
2021
CenterNet Duan et al.
2019
49.3
34.7
70.2
53.5
86.4
QDTrack Pang
2021
60.4
47.2
77.5
62.3
90.1
ByteTrack Zhang ef
2022
YOLOX-X (Ge al.
2021
62.1
50.5
76.5
69.1
93.4
OC-SORT
CCa al
2023
68.1
54,8
84.8
68.0
93.4
TracsTrack (Snn et al.
2020
Deformable DETR Zhu al.
2020)
57.5
82.7
71.5
92.6
Tranking-b--propuuadiin:
McMOTR (Gao & Wang
2023
Deformable DETR Zhu
2020
68.8
57.8
82.0
69.9
90.2
SambaMOTR (ours)
69.8
59.4
82.2
71.9
90.3
Table Ablation on method components on the DanceTrack test set. Compared to prior work (in gray), we introduce a long-range query propagation module based on state-space models (SSM), we mask uncertain queries during the state update (MaskObs), we synchronize memory representations
across tracklets (Sync), and we learn from longer sequences (Longer).
Method
SSM
MaskObs
Sync
Longer
HOTA
AssA
DetA
IDFI
MOTA
63.5
53.8
75.1
67.0
81.7
(b)
v
64.8
54.3
77.7
68.1
85.7
SambaMOTR (Ours)
(c)
v
65.9
55.6
78.4
68.7
87.4
(d)
/
/
v
67.2
57.5
78.8
70.5
88.1
MOIR Zeng 2022
54.2
40.2
73.5
51.5
79.7
McMOIE K Gao & Wang 2023
63.4
52.3
77.0
65.5
85.4
Long-sequence training. Efficiently incorporating longer sequences during training (line d) helps the model to properly utilize its long-term memory, enabling generalization to indefinitely long sequences and leading notable +1.9 improvement in AssA.
Our final query propagation method (line d) improves MeMOTR's association accuracy by +5.2 (line f), and MOTR's by an impressive +17.3 (line e).
LIMITATIONS
Following the tracking-byr backing-by-proppe bropoapagtiion paradigm, our model drops tracklets that are inactive for more than Nmiss frames to decrease the risk of ID switches. However, in some datasets like SPortsMOT (Cui et al. (203) football players may disappear from the camera view for multiple sec- onds, outliving the Nmiss threshold. We argue that future work should complement tracking-by propagation with long-term re-identificatin tackle this issue. Furthermore, in this paper, in- troduced Samba, a -fuuuenceeecess model. Our ablation study (Tab. (4) shows that Samba signifi- cantly outperforms the already strong SSM baseline. However, this comes with the trade-off of in- creased computational complexity. In particular, SSMs have linear complexity in time and linear complexity in the number of sequences (tracklets) independently modeled. Samba retains linear- time complexity, which enables it to track for indefinitely long-time horizons, but quadratic com- plexity in the number of sequences due to the use of self-attention in memory synchronization. Our ablations show that this trade-off worth the performance improvement.
7 CONCLUSION
The proposed SambaMOTR fully leverages the sequential nature of the tracking task by using our set-of-sequences model, Samba, query propagation module to jointly model the temporal history of each tracklet and their interactions. The resulting tracker runs with linear-time complexity and can track objects across indefinitely long sequences. SambaMOTR surpasses the mate-of-the-ar all benchmarks, reporting significant improvements in association accuracy compared to prior work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the Max Plank ETH Center for Learning Systems.
REFERENCES
Magda Amiridi, Gregory Damell, and Sean Jewell. Latent temporal flows for multivariate analysis of wearables data. arXiv preprint arXiv:2210.07475, 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Keni Bernardin and Rainer Stiefelhagen. Evaluating multiple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing, 2008:1-10, 2008.
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Uperoft. Simple online and realtime tracking. 2016 IEEE international conference on image processing (ICIP), pp. 3464-3468. IEEE, 2016.
Jiarui Cai, Mingze Xu, Wei Li, Yuanjun Xiong. Wei Xia, Zhuowen Tu, and Stefano Soatto. Memot: Multi-object tracking with memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8090-8100 -1990, 2022.
Jinkun Cao, Jiangmiao Pang, Xinshuo Weng. Rawal Khirodkar, and Kris Kitani. Observation- centric sort: Rethinking sort for robust multi-object tracking. In Proceedings of the IEEE/CVF Conference Compuler Vision and Puttern Recognition (CVPR), pp. 9686-9696, 2023.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and
Sergey Zagoruyko. End-to-end object detection with transformers. Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pp. 213-229. Springer, 2020.
Yutao Cui, Chenkai Zeng, Xiaoyu Zhao, Yichun Yang, Gangshan Wu, and Limin Wang. Sportsmot:
A large multi-object tracking dataset in multiple sports scenes. arXiv preprint arXiv:2304.05170, 2023.
Patrick Dendorfer, Aljosa Osep, Anton Milan, Konrad Schindler, Daniel Cremers, Ian Reid, Stefan Roth, and Laura Leal-Taixé. Motchallenge: A benchmark for single-camera multiple target track- ing. International Journal of Computer Vision (IJCV), 129:845 29:845-881, 2021.
John Downer and David Tennant. BBC.
Earthflight, 2011.
URL
https: www. bbc. co. uk/
srogrammes/bblBxxm
Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6569-6578, 2019.
Ruopeng Gao and Limin Wang. Memotr: Long-term memory-augmented transformer for multi- object tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9901-9910, 2023.
Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint 107003200..08430, 2021.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00330 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv: 0ee.1666.88411 2016.
11

---

Published as a conference paper at ICLR 2025
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 1960.
Siyuan Li, Martin Danelljan, Henghui Ding, Thomas E Huang, and Fisher Yu. Tracking every thing in the wild. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 498- 515. Springer, 2022.
Siyuan Li, Tobias Fischer, Lei Ke, Henghui Ding, Martin Danelljan, and Fisher Yu. Ovtrack: Open- vocabulary multiple object tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5567-5577, 2023.
Siyuan Li, Lei Ke, Martin Danelljan, Luigi Piccinelli, Mattia Segu, Luc Van Gool, and Fisher Yu. Matching anything by segmenting anything. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
Siyuan Li, Lei Ke, Yung-Hsu Yang, Luigi Piccinelli, Mattia Segu, Martin Danelljan, and Luc Van Gool. Slack: Semantic, location and appearance aware open-vocabulary tracking, In Com- puter Vision-ECCV 2024: 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings. Springer, 2024b.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab- detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201. 123229, 2022..
Zhizheng Liu, Mattia Segu, and Fisher Yu. Cooler: Class-incrementee learning for appearance-base based multiple object tracking. arXiv preprint arXiv:2310.03006, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled
weight decay regularization. arXiv preprint
1000 2017.
Jonathon Luiten, Aljosa Osep, Patrick Dendorfer, Philip Torr, Andreas Geiger, Laura Leal-Taixé, and Bastian Leibe. Hota: A higher order metric for evaluating multi-object tracking. International Journal of Computer Vision (IJCV), 129:548-578. 2021.
Wenhan Luo, Junliang Xing, Anton Milan, Xiaoqin Zhang, Wei Liu, and Tae-Kyun Kim. Multiple object tracking: A literature review. Artificial intelligence, 293:103448; 2021.
Tim Meinhardt, Alexander Kirillov, Laura Leal-Taixe, and Christoph Feichtenhofer. Trackformer: Multi-object tracking with transformers. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition (CVPR), pp. 8844-8854, 2022.
Anton Milan, Laura Texie Ian Reid, Stefan Roth, and Konrad Schindler. Mot16; A benchmark for multi-object tracking. arXiv preprint arXiv:1603 1e..1600300,,.., 2016.
Jiangmiao Pang. Linlu Qiu, Xia Li, Haofeng Chen, Qi Li, Trevor Darrell, and Fisher Yu. Quasi-dense similarity learning for multiple object tracking. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 164 173, 2021.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, Pmlr, 2013.
Zheng Qin, Sanping Zhou, Le Wang, Jinghai Duan, Gang Hua, and Wei Tang. Motiontrack: Learn- ing robust short-term and long-term motions for multi-object tracking. In Proceedings of the JEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17939-17948, 2023.
12

---

Published as a conference paper at ICLR 2025
Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance mea- sures and a data set for multi-target, multi-camera tracking. In Proceedings of the European Con- ference Computer Vision (ECCV). pp. 17-35. Springer, 2016.
Mattia Segu, Bernt Schiele, and Fisher Yu. Darth: Holistic test-time adaptation for multiple object tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9717-9777, 2023.
Mattia Segu, Luigi Piccinelli, Siyuan Li, Luc Van Gool, Fisher Yu, and Bernt Schiele. Walker: Self- supervised multiple object tracking by walking on temporal appearance graphs. In Computer Vision-ECCV 2024: 18th European Conference, Milan, Italy, September 29 Octoober 4, 2024, Proceedings. Springer, 2024.
Peize Jinkun Cao, Yi Jiang, Rufeng Zhang, Enze Xie, Zehuan Yuan, Changhu Wang, and Ping Luo. Transtrack: Multiple object tracking with transformer. arXiv preprint arXiv:2012 154600 2020.
Peize Sun, Jinkun Cao, Yi Jiang, Zehuan Yuan, Song Bai, Kris Kitani, and Ping Luo. Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In Proceedings the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 20993-21002, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence to sequence-video to text. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), pp. 4534-4542, 2015.
Zhongdao Wang, Liang Zheng, Yixuan Liu, Yali Li, and Shengjin Wang. Towards real-time multi- object tracking. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 107-122. Springer, 2020.
Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun. Transformers in time series: A survey. arXiv preprint arXiv:2202.07/25, 2022.
Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association metric. In 2017 IEEE international conference image processing (ICIP), pp. 3645- 3649. IEEE, 2017.
Jialian Wu, Jiale Cao, Liangchen Song, Yu Wang, Ming Yang, and Junsong Yuan. Track to detect and segment: An online multi-object tracker. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 12352-12361, 2021.
Yuxia Wu, Yuan Fang, and Lizi Liao. On the feasibility of simple transformer for dynamic graph modeling, arXiv preprint arXiv:2401. 2024.
Yihong Xu, Yutong Ban, Guillaume Delorme, Chuang Gan, Daniela Rus, and Xavier Alameda- Pineda. Transcenter: Transformers with dense representations for multiple-object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 45(6):7800-7835 2022.
Fan Yang, Shigeyuki Odashima, Shoichi Masui, and Shan Jiang. Hard to track objects with irregular motions and similar appearances? make easier by buffering the matching space. In Proceedings of the IEEE/CVF Winter conference Applications of Computer Vision (WACV), pp. 4799-4808. 2023.
Xitong Yang, Palghat Ramesh, Radha Chitta, Sriganesh Madhvanath, Edgar A Bernal, and Jiebo Luo. Deep multimodal representation learning from temporal data. In Proceedings of the IEEF/CVF' Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5447-5455, 2017.
Fangao Zeng, Bin Dong, Yuang Zhang, Tiancai Wang, Xiangyu Zhang, and Yichen Wei. Motr: End- to-end multiple-object tracking with transformer. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 659-675. Springer, 2022.
13

---

Published as a conference paper at ICLR 2025
Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Infor- mation Processing Systems, 32, 2019.
Yifu Zhang, Chunyu Wang, Xinggang Wang. Wenjun Zeng, and Wenyu Liu. Fairmot: On the fairesss of detection and re-identification in multiple object tracking. International Journal of Computer Vision (IJCV), 129:3069-308 -3669 2021.
Yifu Zhang, Peize Sun, Yi Jiang, Dongdong Yu, Fucheng Weng, Zehuan Yuan, Ping Luo, Wenyu Liu, and Xinggang Wang. Bytetrack: Multi-object tracking by associating every detection box. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 1-21. Springer, 2022.
Yuang Zhang, Tiancai Wang, and Xiangyu Zhang. Motrv2: Bootstrapping end-to-end multi-object tracking by pretrained object detectors. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 22056-22065, 2023.
Guangze Zheng, Shijie Lin, Haobo Zuo, Changhong Fu, and Jia Pan. Nettrack: Tracking highly dynamic objects with a net. arXiv preprint arXiv:2403. 2024.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Tracking objects as points. In Proceedings of the Europeun Conference Computer Vision (ECCV), pp. 474-490. Springer, 2020.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Probabilistic two-stage detection. arXiv preprint 10Xx:22003.0761,,, 2021.
Xingyi Zhou, Tianwei Yin, Vladlen Koltun, and Philipp Krähenbühl. Global tracking transform- In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 8771-8786 2022.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 4rXiv:2010.04159, 2020.
14


Reviews:
[{"review_id": "5XNbDkNZ9y", "reviewer": "Reviewer_dEH5", "summary": "The paper introduces SambaMOTR for multiple object tracking in videos. This method aims to improve tracking by utilizing temporal information about object movements to model historical object data, and occlusions, thereby modeling long-range dependencies, implicit motion, and appearance between tracklets.", "questions": "- Just to confirm, does the state space model operate on the coordinate domain (bounding boxes) or the visual domain (pixel values)? If it does operate on the coordinate domain, how can the auto-regressed output boxes be refined to fit the subjects without looking at visual features?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Overall, the experiments are extensive with different alterations in modeling provided in the main paper and appendix.", "weaknesses": "- Visual comparisons show that the Samba autoregressive model performs better than memory/motion models should be provided to substantiate the claims.\n- The interpretations of system $\\mathbf{A}$ and control $\\mathbf{B}$ matrices in the context of object tracking are unclear. Providing clearer definitions of notations would be helpful.\n- I understand the intention to separate two paradigms tracking-by-detection (TBD) and tracking-by-propagation (TBP), but the difference is subtle as the TBP still needs bounding boxes to initiate the states, and needs a detector to recognize newly appearing objects. Then, propagation alone is simply not practical in this problem of multiple object tracking.\n- Then it should be fair to compare with TBD methods, and the performance still falls short behind classic motion models as reported in Table A.\n- While I value the effort in development, the innovation in the method appears to be limited, as it essentially represents a straightforward combination of MOTR and Mamba and the motivation is not really compelling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oSMJj7DGmT", "reviewer": "Reviewer_Zen6", "summary": "The paper introduces the novel SAMBA model architecture for multi-object tracking. It takes a unique approach to tracking in a tracking-by-propagation framework, building on top of selective state-space models captured by Mamba models. Additional self-attention is used to model dependency between tracks. The method is evaluated on three tracking datasets, and it demonstrates a new state-of-the-art performance.", "questions": "(1) Would it be possible to include a comparison of SambaMOTR on top of YOLOX-X to have an apple-to-apples comparison with prior tracking-by-detections works?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) The presented method achieves strong performance.\n\n(2) the reason to integrate SSMs into a tracking-by-propagation framework is well motivated.\n\n(3) The ablation studies conducted verified the introduced modifications.\n\n(4) The paper is well written; the provided illustrative examples showcase strong results.", "weaknesses": "(1) There is a lack of qualitative comparison between prior works and the proposed method. It is helpful to see qualitative differences between methods' outputs to highlight what improvements in the models' output are contributing to the improvement in the performance. \n\n(2) On L137, it is stated that  SambaMOTR has \"the same GPU memory requirements\". However, the paper lacks any measurements to back up this claim. Overall, there is emphasis placed on efficiency in the paper, so it would be good to include some measurements and comparisons to show these points. For example, including peak memory requirements and FLOPS would illustrate this point. \n\n(3) While the method shows strong results, chiefly due to architectural modifications, it is worth asking whether efficiency gains enable larger (higher capacity) models, which lead to improvements or whether the SSMs formulation provides better biases for the tracking problem. While it might be difficult to disentangle such factors, the paper could help guide future research in this area by reporting some (albeit not great) proxies, such as learnable parameter counts, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UWR5fTNROP", "reviewer": "Reviewer_WHQW", "summary": "The paper introduces a tracking-by-propagation framework, SambaMOTR, which wanna tackles the question of how to model long-range dependencies within tracklets, interdependencies among tracklets, and the associated temporal occlusions. The core function of the  SambaMOTR is the Samba module, a novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet.  SambaMOTR is evaluated on DanceTrack, BFT, and SportsMOT datasets and achieves the state-of-the-art performance.", "questions": "see Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. SambaMOTR introduces the novel linear-time set-of-sequences model designed to jointly process multiple tracklets by synchronizing the multiple selective state-spaces used to model each tracklet.\n2. SambaMOTR achieves good tracking results on DanceTrack, BFT, and SportsMOT datasets.", "weaknesses": "1. The authors are advised to analyze the computational complexity.\n2. The authors are advised to analyze why Samba is suitable for trajectory modeling and compare it with xLSTM, other SSMs, and RRNs.\n3. The authors are advised to provide some tracking cases of SambaMOTR to demonstrate its superiority in trajectory modeling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)