MambaTrack: a simple baseline for multiple object tracking with State Space Model

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-07 03:16:28

Abstract: Tracking by detection has been the prevailing paradigm in the field of Multi-object Tracking (MOT). These methods typically rely on the Kalman Filter to estimate the future locations of objects, assuming linear object motion. However, they fall short when tracking objects exhibiting nonlinear and diverse motion in scenarios like dancing and sports.  In addition, there has been limited focus on utilizing learning-based motion predictors in MOT. To address these challenges, we resort to exploring data-driven motion prediction methods. Inspired by the great expectation of state space models (SSMs), such as Mamba, in long-term sequence modeling with near-linear complexity, we introduce a Mamba-based motion model named Mamba moTion Predictor (MTP). MTP is designed to model the complex motion patterns of objects like dancers and athletes. Specifically, MTP takes the spatial-temporal location dynamics of objects as input, captures the motion pattern using a bi-Mamba encoding layer, and predicts the next motion. In real-world scenarios, objects may be missed due to occlusion or motion blur, leading to premature termination of their trajectories. To tackle this challenge, we further expand the application of MTP. We employ it in an autoregressive way to compensate for missing observations by utilizing its own predictions as inputs, thereby contributing to more consistent trajectories. Our proposed tracker, MambaTrack, demonstrates advanced performance on benchmarks such as Dancetrack and SportsMOT, which are characterized by complex motion and severe occlusion.

Main paper: 
 introduction: 
 Multi-object tracking (MOT) is a fundamental computer vision task aimed at locating objects of interest and associating them across video frames to form trajectories. It has extensive applications in various domains, including autonomous driving [5, 12, 43] , human behavior analysis [9, 40, 49] , and robotics [28] . Tracking-bydetection [3, 6, 26, 46, 52] has been the dominant paradigm due to its succinct design, which involves two main steps: 1) obtaining the bounding boxes of objects using an off-the-shelf detector, and 2) associating these detections into trajectories based on appearance or motion cues. This paradigm has seen significant progress over the past decade, particularly in scenarios [10, 30] characterized by distinguishable appearance and simple motion patterns.Despite the commendable performance of these trackers on pedestrian tracking benchmarks [10, 30] , their efficacy diminishes notably in intricate scenarios [9, 40] , typified by various and rapid movements, as well as less discriminative appearances. The primary challenge encountered in DanceTrack and SportsMOT resides in the data association phase. Specifically, the limitations stem from the inefficacy of object appearance cues in distinguishing between distinct objects and the insufficiency of conventional motion predictors, Kalman filter, in accurately forecasting object positions in scenes characterized by nonlinear motion patterns and frequent occlusions.To address the challenges posed by these complex scenarios, we turn our attention to leveraging motion information for object association. Given the unreliability of appearance cues, our emphasis is on designing a learnable motion predictor capable of capturing object motion patterns solely from object trajectory sequences. While Long Short-Term Memory (LSTM) [17] and Transformer [42] architectures are both prominent in sequence modeling, they face distinct challenges. LSTM is criticized for its inefficient training and limited capacity for long-term modeling, whereas Transformer suffers from quadratic computational complexity relative to sequence length during inference. In recent years, state space models (SSMs) have shown promise in optimizing performance and computational complexity concurrently. These models capture sequence information through convolutional computing and achieve near-linear complexity during inference. A recent advancement, Mamba [13] , integrates a selective mechanism into SSMs to attend to important parts of sequence data, akin to attention mechanisms [42] . Inspired by Mamba's success in sequence data modeling, we are motivated to incorporate it into Multi-Object Tracking to capture complex object motion patterns. Therefore, we propose a learnable motion predictor, Mamba moTion Predictor (MTP), which takes the historical motion information of object trajectories as input, employs a bi-Mamba encoding layer to encode movement information and predicts the next movement of objects. Subsequently, data association is performed based on the Intersection-over-Union (IoU) similarities between the predicted bounding boxes of tracklets and the detections of the current frame. Experimental results validate the effectiveness of MTP, particularly its significant performance dominates over the classical Kalman filter.Despite exploiting MTP for object association between adjacent frames, we extend its usage to achieve long-term association. Specifically, to re-establish lost tracklets caused by occlusions or detector failures, we introduce a tracklet patching module. This module compensates for missing observation points by employing MTP in an auto-regressive manner, wherein it takes its own predictions as input to continue predicting the next motion of the lost tracklets. With the assistance of tracklet patching, our proposed tracker, MambaTrack, generates more consistent trajectories.In conclusion, the major contributions of this work are as follows:• We propose a data-driven motion predictor, Mamba moTion Predictor (MTP), designed to model diverse motion patterns in complex scenarios. • We introduce a tracklet patching module that employs MTP in an auto-regressive manner to re-establish the lost tracklets.• Equipped with the designed MTP and the tracklet patching module, the proposed online tracker, MambaTrack, addresses the challenging data association problem in complex dancing and sports scenarios. As a motion-based online tracker, MambaTrack achieves state-of-the-art performance on the two merging benchmarks, DanceTrack [40] and SportsMOT [9] . 
related work 2.1 tracking-by-detection methods: 
 With the rapid advancement of detection and re-identification techniques [7, 11, 34, 35, 48] , tracking-by-detection (TBD) methods [3, 6, 32, 44-46, 52, 53] have made significant progress. These methods utilize existing detectors to obtain detection results from video frames, which are then associated with forming object trajectories. Some TBD methods generate object trajectories using complex optimization algorithms [4, 23] in an offline manner, while others operate in an online manner, associating detections with tracklets frame-by-frame. Given the practicality of online methods, researchers have focused on enhancing them from various perspectives. For instance, methods like JDE [45] and FairMOT [53] extract object spatial locations and appearance embeddings from a shared network, thereby improving accuracy and inference efficiency. Additionally, QDTrack [32] employs a contrastive learning strategy to acquire reliable appearance cues. Moreover, ByteTrack [52] employs cascading matching strategies to handle detections with varying confidence levels obtained from a modern detector, resulting in impressive performance. However, the conventional benchmarks [10, 30] primarily feature distinct appearances and regular motion patterns, leading to a heavy reliance on appearance cues and limited utilization of motion information for data association.The changes in the spatial locations of objects serve as crucial cues for tracking objects across frames. Motion models utilized in multi-object tracking can be broadly categorized as filter-based or learning-based. The classical work, SORT [3] , employs the Kalman Filter (KF) [20] to estimate the motion state of objects. Although subsequent works [45, 46, 52, 53] inherit this motion model, they are primarily designed for tracking objects with regular motion patterns and struggle in more complex motion scenarios. OC_SORT [6] addresses the inherent limitations of KF and enhances its capability to handle nonlinear motion and occlusion scenarios. Learning-based methods predict object inter-frame offsets from video frames or rely solely on trajectory information. For example, Tracktor [2] incorporates a regression branch to predict object displacements using information from two consecutive frames. CenterTrack [55] predicts the center offsets of objects using information from two consecutive frames and the last heatmap as input. ArTIST [38] treats object motion as a probability distribution and employs an MA-Net to model interactions among objects. However, these methods tend to be computationally intensive or require complex training procedures. In this work, our proposed tracker relies solely on the historical bounding box sequences of objects to predict their future locations. By adopting this approach, we aim to propose a simple motion-based tracker in diverse motion scenarios while maintaining high accuracy.Inspired by control theory, the integration of linear state space equations with deep learning has been explored to enhance the modeling of sequential data. This fusion was initially catalyzed by the introduction of the HiPPO matrix [14] , which laid the groundwork for subsequent developments. LSSL [16] represents a pioneering effort in this domain, utilizing linear state space equations to model sequence data. [15] introduces Structured State-Space Sequence S4 to model long-range dependency, which advanced the field by employing linear state space representations for contextualization, demonstrating robust performance across a spectrum of sequence modeling tasks. The inherent characteristic of linear scalability in the sequence length of SSMs attracts more attention. Furthermore, SGConv [25] offers an innovative perspective by recasting the S4 model as a global conventional framework. In pursuit of enhanced computational efficiency, GSS [29] incorporates a gating mechanism within the attention unit, thereby reducing the dimension of the state space module. A seminal contribution to the field is the introduction of the S5 layer [39] , which encompasses the parallel scan and the MIMO SSM. This layer significantly streamlines the utilization and implementation of state space models, paving the way for widespread adoption. The state space model has been successfully applied in the domain of computer vision by various research initiatives, such as ViS4mer [18] , S4ND [31] and TranS4mer [19] . Recently, Gu et al. [13] introduced a data-dependent SSM layer in their work, which establishes a generic language model backbone termed Mamba. Mamba exhibits superior performance compared to Transformers across various scales on extensive datasets while also benefiting from linear-time inference and efficient training procedures. Building on the success of Mamba, Mamba attracted the attention of a lot of researchers. MoE-Mamba [33] integrates a Mixture of Expert approach with Mamba, thereby unleashing the scalability potential of SSMs and achieving performance comparable to Transformers. The VideoMamba [24] effectively employs Mamba's linear complexity operator to facilitate efficient long-term modeling, demonstrating notable advantages in tasks related to understanding lengthy videos.𝓑 ••• Frame 𝒕 -𝟑 ••• Frame 𝒕 -𝟐 Frame 𝒕 -𝟏 Frame 𝒕 -𝒏 𝕋 𝕋To the best of our knowledge, there has been no prior research that explores the Mamba architecture to multi-object tracking. It is intuitive to speculate that the Mamba architecture could serve as an effective and efficient solution for the MOT problem, particularly in scenarios involving occlusion and non-linear motion. Therefore, we try to incorporate the Mamba model in multi-object tracking task. 
preliminaries: 
 State Space Models. State Space Models (SSMs) are inspired by linear time-invariant (LTR) system that map the input sequence 𝑥 (𝑡) ∈ R to a response 𝑦 (𝑡) ∈ R through the hidden state vector ℎ(𝑡) ∈ R 𝑁 . Mathematically, the dynamics of the system can be modeled by a set of first-order differential equations:EQUATIONwhere matrices 𝐴 ∈ R 𝑁 ×𝑁 represents the evolution parameters and 𝐵 ∈ R 𝑁 ×1 , 𝐶 ∈ R 𝑁 ×1 𝐷 ∈ R 𝑁 ×1 are the projection parameters.Discretization. Since SSMs are designed to operate on continuous signal 𝑥 (𝑡), they inherently cannot process discrete sequences like time series and natural language {𝑥 0 , 𝑥 1 , . . .}. Therefore, it becomes imperative to adapt SSMs to a discretized version to handle such input data effectively:ℎ 𝑘 = Āℎ 𝑘 -1 + B𝑥 𝑘 , 𝑦 𝑘 = Cℎ 𝑘 + D𝑥 𝑘 . (2)The discretized form of SSM utilizes a time-scale parameter Δ to transform continuous parameter 𝐴, 𝐵, 𝐶 and 𝐷 to discrete parameters Ā, B, C and D. Especially, D, which conventionally serves as a residual connection, is frequently simplified or omitted in certain contexts. The transition often uses the zero-order hold (ZOH) EQUATIONSelective SSMs. The inherent Linear Time-Invariant (LTI) characteristic of SSMs, which relies on the consistent utilization of matrices Ā, B, C, and Δ across various inputs, imposes limitations on their ability to filter and comprehend contextual nuances within diverse input sequences. Mamba [13] address this limitation by treating B, C, and Δ as dynamic, input-dependent parameters, thereby transforming the SSM into a time-variant model. This modification enables the model to adapt more effectively to different input contexts, enhancing its capability to capture relevant temporal dynamics. Consequently, it obtains a more precise and efficient representation of the input sequence. 
the proposed method 4.1 notation: 
 As depicted in Figure 1 , our proposed MambaTrack adheres to the tracking-by-detection paradigm [3, 6, 52] in an online manner. To this end, we employ an off-the-shelf detector, YOLOX [11] , to acquire 𝑀 detections B 𝑡 = {b 𝑡 𝑖 } 𝑀 𝑖=1 for the current frame 𝑡. Each detection b ∈ R 4 is represented by its 2D coordinates (𝑥, 𝑦) denoting the top-left bounding box corner in the image plane, alongside its width 𝑤 and height ℎ. We denote the set of 𝑁 tracklets as T = {T 𝑗 } 𝑁 𝑗=1 , whereT 𝑗 = {b 𝑠 𝑗 , b 𝑠+1 𝑗 , • • • , b 𝑡 𝑗 } denotes the tracklet of object 𝑗.Here, b 𝑡 𝑗 signifies its bounding box in frame 𝑡, and 𝑠 denotes the frame of its initial appearance.At the first frame of the video, we directly initialize the set of T with the detections B 1 . In subsequent frames, the goal is to assign the detection results provided by the detector to the appropriate tracklets. Over time, objects may exit the scene, leading to the termination of their trajectories and their subsequent removal from T. Conversely, new objects may appear, and their trajectories will be added to T. During tracking, trajectories are often interrupted due to occlusion and detector failure. Consequently, we further partition T into T active and T lost , representing trajectories that have just been assigned new observations in the previous frame and trajectories that are temporarily interrupted but not yet removed, respectively. A tracklet 𝑙 in T active is denoted asT 𝑙 = {b 𝑠 𝑙 , b 𝑠+1 𝑙 , • • • , b 𝑡 𝑙 }, while a tracklet 𝑚 in T lost is represented as T 𝑚 = {b 𝑠 𝑚 , b 𝑠+1 𝑚 , • • • , p 𝑡 -2 𝑚 , p 𝑡 -1 𝑚 , p 𝑡 𝑚 }, where b denotes the bounding box provided by the detector, and p denotes the infilling bounding boxes used to fill the missing points caused by occlusion or detector failure.The complexity of multiple object tracking in scenarios such as DanceTrack [40] and SportsMOT [50] arises from the intricate motion patterns of the objects and the substantial occlusion between them. To tackle this challenge, we adopt a divide-and-conquer framework to handle T active and T lost separately. First, we predict the spatial position of the active trajectories in the current frame based on their historical observations, utilizing the motion predictor, Mamba Motion Predictor, proposed in this paper. Second, for the lost trajectories with missing observations, we employ autoregression to fill in the gaps before making predictions. We provide detailed explanations for each of these processes in the subsequent subsections.An overview of the proposed Mamba Motion Prediction (MTP) is depicted in Figure 2 , comprising three main components. The first component comprises an input embedding layer, which takes the historical dynamics of the object trajectory as input and linearly transforms it to obtain a sequence of input temporal tokens. The second component consists of an encoding layer composed of L bi-Mamba blocks with Mamba modules at its core. Finally, the last layer is the prediction head, responsible for predicting the interframe bounding box offsets of the object trajectory.Temporal Tokenization Layer. For a tracklet 𝑖 in T, we first construct the input trajectory feature:EQUATION)where 𝑞 is the size of the look-back temporal window and o = [𝛿𝑐 𝑥 , 𝛿𝑐 𝑦 , 𝛿𝑤, 𝛿ℎ], with 𝛿𝑐 𝑥 , 𝛿𝑐 𝑦 , 𝛿𝑤, and 𝛿ℎ representing the normalized changes of the corresponding bounding box center, width, and height between two observation time steps. We utilize a single linear layer to obtain the input token sequence as follows:EQUATIONwhere X ∈ R 𝑞×𝑑 𝑚 and 𝑑 𝑚 is the dimension of the temporal token. Bi-Mamba Encoding Layer. After obtaining the temporal tokens X of tracklets, we feed them into the designed bi-Mamba encoding layer to explore the motion patterns from the object's dynamic history. The bi-Mamba encoding layer comprises 𝐿 bi-Mamba blocks. Specifically, to fully utilize the information from the object trajectory and address the unidirectional limitation of Mamba, each bi-Mamba block contains bidirectional Mamba modules: one forward and one backward. For the 𝑙-th bi-Mamba block, the forward process can be formulated as follows:EQUATIONwhere X 𝑙 -1 is the output of the (𝑙 -1)-th bi-Mamba block, LN is the layer normalization function [1] , and MLP is a two-layer multilayer perceptron. The selective SSM is the core of the Mamba [13] module which is described in Sec. 3.Prediction head and training. After being processed by the bi-Mamba encoding layer, an average pooling layer is utilized to aggregate the information from X 𝑙 . Subsequently, a prediction head comprising two fully connected layers is employed to predict the offsets Ô. We utilize the smooth L1 loss to supervise the training process:EQUATIONwhere O * = {𝛿 𝑐 𝑥 , 𝛿 𝑐 𝑦 , 𝛿 𝑤 , 𝛿 ℎ } represents the ground truth. In real-world scenarios, objects may go undetected at certain time points due to severe occlusion or motion blur. Consequently, the corresponding tracklets may not receive new updates for several frames during the matching process, leading to early termination of the tracklets and fragmented trajectories. In this subsection, our goal is to extend the tracklets that do not receive new bounding boxes in order to enhance the consistency of the tracklets. For example, if a lost tracklet T 𝑖 in lost tracklets T 𝑙𝑜𝑠𝑡 receives no new update at the last time step 𝑡 -1 and remains unmatched at the current frame 𝑡, we compensate for this missing observation in an autoregressive manner by considering the predicted bounding box b𝑡 -1 𝑖 as the actual observation of frame 𝑡 -1. We then continue to predict its spatial location p𝑡 𝑖 at the current frame. As shown in Figure 3 , if it still fails to match with a new detection in the current frame, we persist in predicting its future bounding boxes frame by frame utilizing the motion predictor MTP, leveraging the historical trajectory sequenceT 𝑝𝑎𝑠𝑡 = {• • • , b 𝑠 𝑖 , b 𝑠+1 𝑖 , • • • , b𝑡 -1 𝑖 }𝑖 in an autoregressive manner:EQUATIONSince the bounding boxes obtained through autoregression for lost tracklets are typically less reliable compared to those of active tracklets, we prioritize the association of active tracklets with the detection results B𝑡 in the current frame. Therefore, the active tracklets are given precedence in being associated with the detection results in the current frame. The remaining detection results are then associated with P𝑡 , the predicted bounding boxes of the lost tracklets. The detailed inference process is described below.During inference, we utilize the proposed Mamba motion predictor to model object motion patterns and predict their future movement.Algorithm 1: Inference of MambaTrack at frame 𝑡.Input: Detections: Following the common practice of SORT-like methods [3, 6] , we implement the tracking process using bipartite matching, as depicted in Algorithm 1. We first associate the active tracks T 𝑎𝑙𝑖𝑣𝑒 based on the Intersection-over-Union (IoU) similarities C 𝑡 between the predicted bounding boxes B𝑡 and detections B𝑡 in the current frame via the Hungarian algorithm [22] . Then, to find the lost tracklets, the remaining detections B 𝑢 will be matched with the predicted bounding boxes P of them at the second matching based on C 𝑙𝑜𝑠𝑡 . Then, to find the lost tracklets, the remaining detections B 𝑢 will be matched with the patched bounding boxes P in the second matching step.B 𝑡 = {b 𝑡 𝑖 } 𝑀 𝑖=1 , tracklets T = { T 𝑗 } 𝑁 𝑗 =1 at frame 𝑡 -1, Motion Predictor: MTP. Output: Active tracklets T 𝑎𝑐𝑡𝑖𝑣𝑒 at current frame 𝑡 . /* First Matching */ 1 T 𝑎𝑐𝑡𝑖𝑣𝑒 , T 𝑙𝑜𝑠𝑡 ← T 2 B 𝑡 ← [b 1 𝑡 , • • • , b 𝑀𝑡 𝑡 ] // Detection set of current frame 3 B𝑡 ← [ b1 𝑡 , • • • , b𝑁 𝑡 ] from T 𝑎𝑐𝑡𝑖𝑣𝑒 //For simplicity, we omit the initialization of new tracklets from the final remaining detection results B 𝑢 and the termination of lost tracks that have not received updates for consecutive 𝑡 terminate=30 frames. We initialize the unmatched detections whose confidence scores are higher than 𝑡 𝑡ℎ𝑟𝑒𝑠ℎ = 0.6 as new tracklets. 
experiments 5.1 datasets and metrics: 
 Datasets. To assess the effectiveness of our proposed method, we conduct evaluations on two emerging datasets known for their diverse and rapid movements and indistinguishable appearances. These datasets are DanceTrack [40] and SportsMOT [9] . DanceTrack consists of 40 training videos, 25 validation videos, and 35 test videos. Objects in the dancing scenarios are easy to detect, but they are similar in appearance, difficult to distinguish, and exhibit complex and varied movement patterns. Additionally, the newly introduced SportsMOT dataset focuses on sports scenarios such as basketball, football, and volleyball. It contains 45 training videos, 45 validation videos, and 150 test video sequences collected from high-level sports events. Due to the fast and diverse motion of athletes, SportsMOT demands robust tracking approaches. The unique characteristics of the SportsMOT and DanceTrack datasets make them ideal benchmarks for evaluating motion-based trackers, avoiding shortcuts taken by appearance-based methods.Metrics. To comprehensively evaluate the proposed algorithm, we employ a range of evaluation metrics, including the Higher Order Tracking Accuracy (HOTA), which encompasses Association Accuracy (AssA) and Detection Accuracy (DetA), as well as the IDF1 metric and metrics from the CLEAR family (MOTA, FP, FN, IDs, etc.) [27, 36, 37] . MOTA is computed from false negatives (FN), false positives (FP), and identity switches (IDs), and its calculation is primarily influenced by the quality of detection results. IDF1 primarily assesses the consistency of object trajectories. HOTA is specifically designed to provide a balanced assessment of both detection and association performance, making it the primary metric for evaluating tracker performance.This study focuses on developing a robust motion-based tracker, we utilize pre-trained weights of the YOLOX detector provided by the DanceTrack [40] and SportsMOT [9] benchmarks for fair comparisons. The bi-Mamba encoding layer comprises 𝐿 = 3 bi-Mamba blocks with the input token dimension 𝑑 𝑚 set to 512. The maximum look-back temporal window q is set to 10, and the batch size is 64. We employ the Adam optimizer [21] with 𝛽 1 = 0.9, 𝛽 2 = 0.98, and 𝜖 = 10 -8 . During the training process, we adjust the learning rate according to the following formula to linearly increase the learning rate after 𝑤 warmup training steps: lr = (𝑑 𝑚 ) -0.5 × min(𝑤 -0.5 , 𝑤 × (𝑤 warmup ) -1.5 ),where 𝑤 is the training step number, and we set the 𝑤 warmup = 4000.DanceTrack. As presented in Table 1 , our proposed MambaTrack outperforms state-of-the-art methods in the key metric HOTA, demonstrating a lead of 2.2 percentage points over OC_SORT without any postprocessing. OC_SORT addresses the limitations of the Kalman Filter in handling nonlinear motion and heavily occluded environments. Our tracker is designed to model the diverse motion patterns of objects and enhance robustness against shortterm missing observations. This is corroborated by achieving the highest IDF1 score of 57.8, surpassing the second-best method by 3.2 percentage points. Furthermore, our method demonstrates improved accuracy in predicting future spatial locations of objects compared to motion-based trackers [3, 6, 52] employing the same detector. Furthermore, our approach outperforms methods that exploit appearance information. This underscores the significance of utilizing motion information, particularly in complex scenarios like DanceTrack, characterized by intricate object motion patterns and homogeneous appearances.SportsMOT. As shown in Table 2 , our proposed tracker, Mam-baTrack, outperforms comparable tracking algorithms that rely solely on motion information across all metrics. Notably, our method exhibits a substantial lead over ByteTrack, which utilizes Kalman Filter, by nearly 10 percentage points in the HOTA metric, and by 3 percentage points and 9.1 percentage points in the IDF1 and AssA metrics, respectively, which assess trajectory consistency. Additionally, our method surpasses OCSORT, an enhanced Kalman Filter-based approach, demonstrating superior performance. These results underscore the advanced capabilities of our method, even in challenging scenarios characterized by the fast and diverse movements of athletes, further validating its effectiveness.In this section, we perform ablation experiments to validate the effectiveness of our proposed Mamba Motion Predictor (MTP) and the tracklet patching module (TPM). All models are trained on the DanceTrack [40] dataset and evaluated on the DanceTrack validation set. We implement a baseline utilizing the Kalman Filter as the motion predictor.Effectiveness of the proposed MTP and TPM. As depicted in Table 3 , we evaluate the contributions of the proposed modules. It is evident from the table that our proposed motion predictor has led to significant improvements across all metrics compared to the baseline method. Specifically, HOTA demonstrates a 9 percentage point improvement, while IDF1 and AssA show enhancements of 3.6 and 7.8 percentage points, respectively. This underscores the effectiveness of the MTP in efficiently modeling the nonlinear motion of objects and accurately predicting their positions in adjacent frames compared to the Kalman Filter. Furthermore, the introduction of the TPM module results in additional enhancements in metrics related to trajectory consistency, with IDF1 and AssA improving by 1.6 and 0.7 percentage points, respectively. Different motion modeling. We conduct a comparative analysis to assess the impact of temporal dynamic information provided by different motion models on data association, as summarized in Table 4 . Across all motion models, significant improvements are observed compared to the most basic approach, which solely relies on IoU matching without incorporating motion information. Specifically, all the data-driven motion models, which leverage identical trajectory features, demonstrate superior performance compared to the Kalman Filter [20] . Notably, in terms of the HOTA metric, LSTM [8] exhibits a 5.4 percentage point improvement, Transformer (TF) [42] leads by 6.6 percentage points, while our proposed MTP achieves the highest improvement of 9 percentage points. These findings emphasize the substantial potential of data-driven motionbased models and affirm the efficacy of our proposed SSM-based MTP module. Ablation on the design of bi-Mamba encoding layer. We conduct further analysis on the design of the bi-Mamba encoding layer. As depicted in Figure 4 , Bi-Mamba exhibits superior performance across various metrics including HOTA, IDF1 and AssA. Specifically, it leads by 2.5 percentage points in the HOTA metric and demonstrates improvements of 3.3 and 2.3 percentage points in the AssA and IDF1 metrics, respectively. These results confirm the effectiveness of utilizing Bi-Mamba in capturing the object's motion patterns and enhancing the accuracy of object motion prediction compared to using the vanilla Mamba [13] only. Furthermore, we evaluate the impact of different numbers of bi-Mamba blocks in Table 5 . Setting 𝐿 to 3 yields the optimal performance for MTP, achieving the highest values for HOTA, IDF1, AssA, and MOTA metrics.Qualitative Analysis. As depicted in Figure 5 , our proposed MambaTrack accurately tracks objects in randomly selected videos featuring crowded group dancing and basketball games, where objects move rapidly and change direction irregularly. 
conclusion: 
 This paper introduces an online motion-based tracker comprising a motion predictor and a tracklets patching module. The Mamba Motion Predictor, grounded in the State Space Model, Mamba, effectively models the temporal dynamics of objects, facilitating accurate association between objects in consecutive frames. Besides, to enhance trajectory consistency, we leverage the motion predictor as an autoregressor to predict bounding boxes for lost trajectories, thereby re-establishing them. Despite its simplicity and intuitiveness, experimental results on complex motion datasets validate the effectiveness of our approach. We aim for our proposed method to serve as a baseline, fostering further exploration and development of motion-based tracking algorithms.  
 
 Appendix:  


Decision: This paper proposes a MambaTrack integrating Mamba-based motion prediction to enhance Multi-object Tracking capabilities. The paper is well-written and easy to follow, presenting a clear description of the proposed method and its experimental validation. The proposed Mamba moTion Predictor (MTP) utilizes bi-Mamba encoding layers to predict complex object motions, demonstrating promising results on benchmarks such as DanceTrack and SportsMOT. The integration of Mamba into MOT via MambaTrack, particularly through MTP, represents a advancement in handling nonlinear and diverse motion patterns. The use of bi-Mamba encoding layers for motion prediction shows promising results, enhancing tracking accuracy in difficult scenarios. While the paper has some novelty, several key issues need to be addressed to improve its technical strength: 1) theoretical grounding of MambaTRack's advantages when compared with SOTA architectures, 2) more ablation studies and a more thorough comparison with other tracking methods using alternative motion predictors, 3)  analysis of computational cost and real-time feasibility.

Reviews: review: paper_summary: paper_summary: This paper integrates the famous Mamba to object tracking. To address the nonlinear and diverse motion in special scenarios, the authors resort to exploring data-driven motion prediction methods and propose Mamba moTion Predictor (MTP). MTP takes the spatial-temporal location dynamics of objects as input, captures the motion pattern using a bi-Mamba encoding layer, and predicts the next motion. The proposed tracker, MambaTrack, demonstrates advanced performance on benchmarks such as Dancetrack and SportsMOT
strength_weakness: strength_weakness: This paper integrates Mamba to object tracking and achieves good results on DanceTrack and SportsMOT.
1. This paper is easy to follow.
2. The writing of this paper is good.
3. The experimental results are relatively outstanding.
limitations: limitations: The biggest drawback of this paper is that MambaTrack is only an integration not an innovative idea.
Of course, Mamba has been a hot topic in recent months. But for object tracking, an attractive paper is not just an integration of some new methods. The most important problems of object tracking such as occlusion and data association are still solved in traditional ways or not mentioned. The proposed MTP is designed to predict the complex motion of objects. However, I have studied many studies that claim they try to solve nonlinear motion patterns, but no one has solved this problem actually. 
For this paper, MTP only takes sequential motion information as input, and predicts the final offsets. This is a comment way in many papers [1][2] and so on. Maybe others use LSTM or Transformer, but this paper takes account of Mamba. However, the motion information still follows the same processing way.
The TPM is similar to the recurrent predicting pattern in the Kalman filter or other RNN-based methods.
Finally, if the authors want to demonstrate the advantages of MambaTrack. At least, they should analyze or compare mamba-based motion prediction with RNN/Transformer-based prediction. Then the reviewers can figure out why the authors select Mamba.
[1] Mamba-FETrack: Frame-Event Tracking via State Space Model, 2024.
[2]  Exploring Learning-based Motion Models in Multi-Object Tracking, 2024.
paper_summary: paper_summary: To handle the non-linear motion modeling problem in MOT, this paper proposes to use a Mamba-based learnable motion predictor named MTP instead of the traditional way using KF. Experiments show that MTP has better motion modeling capability than KF in complex motion scenarios. By combining MTP and a simple TBD tracking pipeline, MambaTrack achieves good results on DanceTrack and SPORTSMOT.

Generally, the idea of using Mamba in MOT is novel and meaningful, and the writing of this manuscript is clear.
strength_weakness: strength_weakness: 1. The authors propose the Mamba-based MPT module to replace the KF for motion prediction and demonstrate its effectiveness through experiments. Such an attempt demonstrates the feasibility of using Mamba for improving MOT tracking performance.
2. A TPM is also proposed to continuously predict motion states of lost tracklets using previous observations and predictions, which also brings performance gains.
3. This work also inspires more work to explore more effective and learnable motion models in MOT instead of KF to deal with complex motions.
limitations: limitations: 1. For the TPM module, a concern is that will the continuous prediction of lost trajectories based on historical observations and predictions lead to error accumulation similar to the KF prediction without update? 
2. In the first paragraph of Section 4.5, the author describes the association between remaining detections and lost tracklets twice. The authors should proofread the manuscript before submission to reduce such mistakes.
3. In Table 3, is the baseline with the same tracking flow as MambaTrack but using KF as the motion predictor? The authors can also try to replace the KF in other trackers with MTP to further demonstrate its effectiveness.
paper_summary: paper_summary: This paper introduces MambaTrack, an advanced tracking system utilizing a novel Mamba-based motion model named Mamba moTion Predictor (MTP) to enhance Multi-object Tracking (MOT) capabilities. It employs bi-Mamba encoding layers to capture complex motion patterns from the spatial-temporal dynamics of objects, enhancing prediction accuracy. Furthermore, MTP is applied in an autoregressive manner to handle instances of occlusions or motion blur by using its predictions as inputs for subsequent predictions, thus ensuring more consistent trajectories.
strength_weakness: strength_weakness: 1.The integration of the Mamba-based State Space Model (Mamba moTion Predictor, MTP) in multi-object tracking (MOT) presents a novel approach, particularly in handling complex motion scenarios like dancing and sports. The utilization of a bi-Mamba encoding layer to capture the dynamic motion patterns is an innovative technical advancement that improves tracking accuracy in challenging environments.
2.While the paper discusses the near-linear complexity of the Mamba model during inference, detailed computational costs, including time and resource requirements for real-time applications, are not thoroughly addressed. This oversight might concern potential adopters of the technology in resource-constrained environments.
3.The paper leverages the strengths of state space models, known for their efficiency in handling sequential data, to address the specific challenges in MOT, offering a theoretical justification for their use in predicting object motion over time, which is both innovative and grounded in solid mathematical frameworks.
limitations: limitations: 1.The performance of MambaTrack heavily relies on the accuracy of the initial detections provided by detectors like YOLOX. This dependence could limit its effectiveness in scenarios where detector performance is suboptimal due to poor lighting conditions, extreme weather, or other adverse environmental factors.
2.The association between remaining detections and lost tracklets is described twice in the first paragraph of Section 4.5. This repetition could potentially confuse readers and detract from the manuscript's overall clarity. It is recommended that the authors carefully review and edit the manuscript to eliminate such redundancies before submission.
3.By substituting the conventional motion predictors with the Mamba Motion Predictor (MTP) in a variety of existing tracker and conducting a series of experiments, the effectiveness and superiority of MTP can be more comprehensively validated.
paper_summary: paper_summary: This paper proposes an online motion-based multi-object tracking method by employing a Mamba motion predictor. The motivation is to improve tracking performance in complex scenarios when the objects exhibit nonlinear and diverse motion patterns. The proposed method is evaluated on two challenging datasets: DanceTrack and SportsMOT. Experimental results show that the proposed approach achieves robust tracking performance compared with the baseline models. Ablation studies has been conducted to help understand the effectiveness of different modules of the MambaTrack model.
strength_weakness: strength_weakness: - The paper is well-written and easy to follow.

- The proposed approach is technically sound and reasonable.

- Experimental results provide evidence that the proposed approach exhibits superior performance compared to other baseline MOT methods.

- The ablation studies are sufficient to demonstrate the effectiveness of the different modules in the proposed model.
limitations: limitations: Several expressions in the paper are inaccurate and discussable.

- Section 3, line 286. "State Space Models (SSMs) are inspired by linear time-invariant (LTR) system..." In fact, the SSM is a general term and not limited to linear systems. It would be better to clarify that SSMs are general mathematical frameworks to describe a dynamical system and LTI systems are a specific case of SSMs. By the way, there is a typo for "LTR" -> "LTI"

- Section 3, line 334. "Since SSMs are designed to operate on continuous signal x(t), they inherently cannot process discrete sequences..." Again, the SSMs are not limited to continuous-time signals, they can have both continuous-time formulation and discret-time formulation. It is inaccurate to say, "they inherently cannot process discrete sequences".

Several technical details are missing.

- If I understood correctly, the size of the so-called "look-back temporal window" is fixed. Please clarify in Section 5.1 how the video sequences are processed. Are they split into fixed-size sequences for training? Besides, during inference, when t<q (i.e. for time frames shorter than the window size), how the input sequence are processed? Are they padded with specific values?

- Please explain why the best scores reported in Table 3 does not correspond to that in Table 1.

Typos

- Section 4.5, line 620, there is a typo for the notation of detections.

- Section 4.5, line 624. The sentence is redundant with the previous one. 

Further improvements

- It would be better to point out that the scores reported in Table 1 and Table 2 for the baseline models are reproduced or directly taken from the corresponding papers.

- Section 5.4, for the qualitative analysis. It would be more convincing to show examples of how the proposed model can address challenging tracking scenarios with occlusions, accompanied by detailed explanations for specific objects.