Object-Level Pseudo-3D Lifting for Distance-Aware Tracking

Authors: haoyuan  jin
xuesong  xie
yunfeng  yan
zhihang  zhu
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-01 04:40:42

Abstract: Multi-object tracking (MOT) is a pivotal task for media interpretation, where reliable motion and appearance cues are essential for cross-frame identity preservation. However, limited by the inherent perspective properties of 2D space, the crowd density and frequent occlusions in real-world scenes expose the fragility of these cues. We observe the natural advantage of objects being well-separated in high-dimensional space and propose a novel 2D MOT framework, ``Detecting-Lifting-Tracking'' (DLT). Initially, a pre-trained detector is employed to capture 2D object information. Secondly, we introduce a Mamba Distance Estimator to obtain the distances of objects to a monocular camera with temporal consistency, achieving object-level pseudo-3D lifting. Finally, we thoroughly explore distance-aware tracking via pseudo-3D information. Specifically, we introduce a Score-Distance Hierarchical Matching and Short-Long Terms Association to enhance accurate and robust association capability. Even without appearance cues, our DLT achieves state-of-the-art performance on MOT17, MOT20, and DanceTrack, demonstrating its potential to address occlusion challenges.

Main paper: 
 introduction: 
 Multi-object tracking (MOT) is a critical task for media interpretation, involving the spatial localization of objects and the temporal association to maintain consistent identities across consecutive video frames. This task supports a wide range of applications such as autonomous driving [33, 39] and intelligent surveillance [9, 36] . However, due to the complexity of real-world scenarios, including dense crowds and frequent occlusions, MOT with 2D images from a monocular camera remains a highly challenging topic.Previous works [16, 38, 43] explore MOT extensively and gradually formulate an effective paradigm named tracking-by-detection (TBD). This paradigm divides the task into two sequential sub-tasks: detecting objects in each frame, followed by associating these objects across frames based on motion and appearance cues which are crucial for tracking performance. However, when faced with the occlusion problem, the confusion caused by the mutual influence of objects in 2D space makes the reliability of these clues significantly degraded. Several works [26] [27] [28] 43] address this issue by improving motion prediction, enhancing appearance representation, and designing matching strategies, yet they still suffer from the inherent limitations of the 2D perspective. In contrast, we recognize the natural advantages of matching in high-dimensional space, where objects that occlude each other in 2D space are well-separated, shown in Figure 1 . Therefore, we would like to ask: How to achieve multi-object tracking in 3D space based solely on 2D images?In our work, we introduce an innovative 2D MOT framework, "Detecting-Lifting-Tracking" (DLT), which decomposes the discussed problem into two questions: (i) How to achieve the transformation from 2D space to 3D space? (ii) How to fully leverage 3D information for a comprehensive tracking process update?(i) Object-level pseudo-3D lifting. For spatial transformations without any sensor or various views, the most straightforward 3D lifting method is the use of depth estimators. DIP [17] and Quo Vadis [6] employ this method for reconstructing 3D scenes and providing inference bases for invisible objects, respectively. Despite the high computational cost for scene-level depth, depth estimators still suffer from low accuracy and are prone to background noise. In contrast, estimating the distances between objects and the camera serves as a pseudo-3D lifting method, realized through distance estimators such as DisNet [14] , EM [45] , and DistSynth [23] . These distance estimators focus on capturing object-level distance, disregarding the redundancy of background. This is consistent with the object-specific perception of MOT tasks because it is more crucial to know the object distance instead of the dense depth map for the entire scene [2] . Thus, we introduce a Mamba Distance Estimator (MDE) based on the Space-Time Mamba Model, integrating historical information to mitigate the impact of temporary occlusions.(ii) Distance-aware tracking. DLT aims to efficiently utilize pseudo-3D distance information for tracking. Bearing such objectives, we revisit the conventional tracking process, dividing it into two main parts: a set-level matching strategy and an instance-level association schema. On one hand, the matching strategy is the focus of MOT tasks. ByteTrack [43] enhances it by decomposing objects into high-and low-score subsets based on confidence scores, which shows surprisingly effective outcomes. For discrete and sequential matching, we propose the Score-Distance Hierarchical Matching (SDHM) strategy. Specifically, to tackle the issue of confidence scores failing to robustly evaluate object existence in occluded scenes, we introduce the Score-based Hierarchizing incorporating an Occlusion Compensation Score based on distance-aware visibility ratios. Additionally, a noticeable density of objects still persists within high-score subset, we propose Distance-based Hierarchizing for a secondary hierarchizing on the distance dimension. On the other hand, the association schema emphasizes object-track correlations to verify identities. We present the Short-Long Terms Association (SLTA) schema, distinguishing the Short-term Association in adjacent frames impacted by dense crowds, from the Long-term Association addressing re-association challenges due to severe occlusion. For the Short-term Association, the Kalman filter [37] is standard for motion prediction and track update, leveraging IoU for correlation metrics [3, 38, 44] . Our DLT introduces a Pseudo-3D Adaptive Kalman Filter that expands the 2D Kalman filter into pseudo-3D space and adaptively modifies prediction and update noise with object-level distance. We also introduce a new Distanceweighted IoU, incorporating distance variation as an explicit clue. For the Long-term Association, prior works primarily utilize discriminative appearance features which are occlusion-sensitive clues [28, 34] . Additionally, Quo Vadis [6] attempts to incorporate trajectory forecasting [4] for non-linear prediction with multiple complex sub-modules. In contrast, we introduce a Probabilistic Autoregressive Motion Model for modeling natural trajectory distributions in pseudo-3D space, facilitating long-term motion prediction.By answering the two specific questions mentioned earlier, DLT achieves accurate and efficient tracking in pseudo-3D space, setting a new state-of-the-art performance on the MOT17, MOT20, and DanceTrack test sets. Our contributions are summarized as follows:• We propose a novel "Detecting-Lifting-Tracking" (DLT) framework for 2D MOT that leverages the natural advantages of pseudo-3D space without any sensors or various views.  
related work: 
 Tracking-by-Detection. As the reliability of object detection increases [10] , the tracking-by-detection (TBD) paradigm gradually becomes the mainstream method in MOT tasks [3, 43] . TBD typically utilizes pre-trained detectors to obtain object bboxes, focusing on the matching process. SORT [2] and DeepSORT [38] are two of the most classic methods, employing Kalman filters for motion prediction and IoU for correlation measurement as the basis for the Hungarian algorithm. Despite the emergence of numerous advanced methods, challenges such as crowd density and occlusions in real-world scenarios still limit tracking performance. To address this issue, various solutions have been proposed, such as some methods employing carefully designed appearance modules to provide multimodal cues [27, 28] , and others introducing more reasonable motion prediction modules [6, 21, 26] . Differently, we consider a more direct and effective method, namely conducting pseudo-3D lifting and combining distance clues to optimize the matching process, thereby realizing a novel "Detecting-Lifting-Tracking" framework.Distance estimation. Distance estimation and depth estimation are two distinct tasks, although both are based on 2D images without additional sensors or multiple viewpoints. Distance estimation solely aims to obtain the distance information of an object from the camera [19] . In contrast, depth estimation seeks to acquire the depth information for all pixels in a scene [42] . Classical methods of distance estimation achieve this by regressing the relationship between an object's geometric shape and its distance [22, 29] . The first attempt at a deep learning method estimates an object's distance based on the width and height of a given bbox using a Support Vector Machine Regressor (SVR) [12] . Similarly, DistNet [14] , also a bbox-based method, does not employ image feature learning, resulting in significantly noticeable error noise. EM [45] combines ResNet and RoI pooling to reconstruct object-level representations for distance estimation. Building on EM, DistSynth [23] adds a real-time model and an FPN branch. Considering that distance estimation is critical for object-level pseudo-3D lifting, we design a Mamba Distance Estimator (MDE) based on the State Space Model [13] , incorporating historical spatiotemporal information to mitigate the effects of occlusion and enhance estimation accuracy.Tracking with 3D information. Our work focuses on utilizing 3D information for 2D MOT tasks, differing from 3D MOT [25] , in that it inputs and outputs 2D bboxes, with the tracking process executed in 3D space. There are few such works, among which DIP [17] stands as a pioneering effort. It employs depth maps obtained from a depth estimator for inferring the presence of objects, thereby enhancing the capability to handle occlusions. Another notable work is Quo Vadis [6] , which merges depth maps and segmentation results using homography transformations to generate bird's-eye view scene maps, thereby introducing trajectory forecasting to alleviate the influence of long-term object occlusion. While these works somewhat improve the handling of occlusions, they are very limited. The underlying reason is the insufficient utilization of 3D information. Hence, we comprehensively revisit the tracking process, implementing depth-aware enhancements in both matching strategy and association schema to considerably mitigate the inherent sensitivity of 2D MOT methods to occlusions. 
methodology: 
 We propose an innovative "Detecting-Lifting-Tracking" (DLT) framework, as illustrated in Figure 2 , which has three main stages: i) detecting to obtain 2D object bboxes, ii) lifting to pseudo-3D space with Object-Level Distance Estimation (OLDE) by Mamba Distance Estimator, iii) tracking through Score-Distance Hierarchical Matching (SDHM) with Short-Long Terms Association (SLTA).In 2D MOT tasks, pixel-wise depth estimators provide an overly redundant amount of information for the entire scene. Our focus shifts toward Object-Level Distance Estimation (OLDE), which obtains the distances between specific objects and the camera. The primary challenges include: (i) previous estimators rely on single-frame images without temporal consistency, making them susceptible to occlusions, and (ii) it becomes even more complex when dealing with small objects, necessitating higher precision. To address these challenges, we introduce a Mamba Distance Estimator (MDE).Space-Time Mamba. We introduce the Space-Time Mamba (ST-Mamba) to estimate distances from multi-frames, leveraging the inter-frame context instead of just 2D spatial information to achieve a smoother estimation. The Mamba consists of State Space Models (SSMs) [13] , which map a 1D function or sequence 𝑥 (𝑡) ∈ R 𝐿 ↦ → 𝑦 (𝑡) ∈ R 𝐿 through a hidden state ℎ(𝑡) ∈ R 𝑁 . SSMs are often represented in a discretized version as follows:EQUATIONEQUATIONEQUATIONwhere a timescale parameter ∆ to transform the continuous parameters A, B to discrete parameters A, B. To adapt the vanilla Mamba for multi-frame input, we introduce the spatiotemporal selective scan, as illustrated in Figure 3 . Specifically, we unfold patches of each frame along rows and columns into sequences and then concatenate the frame sequences to constitute the long sequences 𝑋 ∈ R 𝐶 ×𝑇 𝐻𝑊 . Then, we parallelly proceed with scanning along the forward and backward directions to explore inter-frame dependency. To balance estimation speed and accuracy, we insert Space-Time Mamba layers at the end of each stage in pre-trained MobileNet v3 [15] , which extracts intra-frame features and fuses inter-frame information.Bidirectional Unified Feature Fusion. To enhance accuracy for small objects, we introduce the Bidirectional Unified Feature Fusion (BUFF) module ensuring seamless feature fusion across scales. The BUFF module consists of Fusion and Connection components. The Fusion block is composed of several convolutional layers. We concatenate two input feature maps 𝑋 1 and 𝑋 2 , which are fed into both branches. One branch compresses channels through a 1 × 1 Conv, while the other branch extracts features using a 1 × 1 Conv and Dilated Reparam Blocks [7] (DRepBlocks), ultimately the fused features 𝑌 are obtained through an element-wise addition:EQUATIONEQUATIONThe Connection module comprises upsampling and downsampling layers. In practice, the BUFF starts with top-down encoding for global features and then proceeds with bottom-up encoding to retain detailed features of small objects. The final fused features are sent to an ROI align layer to extract object-level vector features 𝑋 𝑣 ∈ R 𝑁 ×𝐶 ′ 𝑀𝑀 , where 𝑁 and 𝑀 denotes the number of objects and window dimension of the RoI align, respectively. These vectors are then transformed by an MLP layer to generate Gaussian distribution parameters μ and σ2 for distance, ensuring a degree of uncertainty. Given ground truths 𝜇, the proposed MDE is optimized via Gaussian Negative Log Likelihood (GNLL) loss, denoted as follows: EQUATION)The concept of hierarchical matching, due to its discrete and orderly attributes, has been proven to enhance tracking performance, yet there remains room for improvement: i) alleviating the hierarchical strategy sensitivity to occlusions, and ii) introducing new hierarchical methods to address occlusions further. Accordingly, we propose Score-Distance Hierarchical Matching (SDHM), which comprises Score-based Hierarchizing and Distance-based Hierarchizing.Score-based Hierarchizing. As a vanilla method, ByteTrack [43] decomposes objects into two subsets based on confidence scores: high-score subset O 𝑡 high and low-score subset O 𝑡 low . The O 𝑡 high is prioritized for matching. However, the confidence scores of some positive objects are reduced due to occlusions, resulting in putting into the O 𝑡 low and losing the privilege of priority matching. Thus, we propose an Occlusion Compensation Score (𝑠 𝑜𝑐 ) to reflect the existence of objects enabling more robust Score-based Hierarchizing.The visibility ratio of an object can, to a certain extent, reflect the impact of occlusions on the confidence score. However, this metric is not readily obtainable. For overlapping areas, complex methods such as comparing feature similarities are required to determine their ownership. Fortunately, based on the distance information, the overlapping area belongs to the nearer object. Therefore, for a specific object 𝑜 𝑡 𝑎 in O 𝑡 , its visibility ratio can be defined as:EQUATIONwhere 𝐼 (𝑜 𝑡 𝑛 ,𝑜 𝑡 𝑎 ) is the overlapping area. 𝑆 represents the area of a polygon. 𝐻 (𝑥) is the Heaviside function.The 𝑠 𝑜𝑐 compensates the confidence score based on the visibility ratio. O 𝑡 can be divided into two occlusion-insensitive subsets by setting 𝑠 𝑜𝑐 threshold. O 𝑡 high containing most of the positive objects, is prioritized for matching.The 𝑠 𝑜𝑐 is expressed as follows:EQUATION)where 𝑠 is the confidence score. 𝛼 is the compensation degree.Distance-based Hierarchizing. In 2D images, the positional overlap due to occlusion leads to mutual interference. Even with Scorebased Hierarchizing, a noticeable density of objects persists within the O 𝑡 high . However, by introducing distance information, their positions can be easily separated. Consequently, within the O 𝑡 high , we further introduce Distance-based Hierarchizing.For the 𝑡 𝑡ℎ frame, tracking involves bipartite graph matching between the objects set O 𝑡 and the tracklet set T 𝑡 -1 of the former frame. The O 𝑡 high can be further divided into three subsets based on the distance: the near subset O 𝑡 near , the mid subset O 𝑡 mid , and the far subset O 𝑡 far . Likewise, the T 𝑡 -1 is also divided into three corresponding subsets T 𝑡 -1 near , T 𝑡 -1 mid , and T 𝑡 -1 far . Considering that the nearer objects present less difficulty in matching, O 𝑡 near is prioritized for matching with T 𝑡 -1 near of the same distance level, and similarly, this process is followed for the other two levels. It is important to note that the unmatched objects and tracks within each level are transferred to the next level sets for subsequent matching.The associations can be categorized based on different timescales and degrees of occlusion: (i) the short-term association exists between adjacent frames, affected by partial occlusions, and (ii) the long-term association faces the challenge of re-associating objects that have been completely occluded and lost over some time. Thus, we propose Short-Long Terms Association (SLTA), which comprises Short-term Association and Long-term Association.Short-term Association. The traditional association methods utilize the Kalman filter for motion prediction and track updates, employing IoU to calculate the correlation between predicted tracks and objects. We introduce Pseudo-3D Adaptive Kalman Filter (AKF) and Distance-weighted IoU (DIoU) to incorporate distance information as an auxiliary cue, enhancing the robustness of the association.The track's state of the 𝑡-1 𝑡ℎ frame 𝜏 𝑡 -1 𝑖 ∈ R 10 is represented as (𝑥, 𝑦, 𝑎, ℎ, 𝑑, 𝛿), where 𝛿 is the motion velocity ( 𝑥, 𝑦, 𝑎, ℎ, 𝑑) between the 𝑡-1 𝑡ℎ and 𝑡-2 𝑡ℎ frame. Our AKF assumes a constant velocity model with Gaussian noise. Firstly, we scale the prediction noise by the inverse distance [17] , which leads to a smoother tracklet for tracks that are far away. We define the motion prediction step as:EQUATIONEQUATIONwhere 𝜇 𝑡 -1and μ𝑡 𝑖 are the state means of 𝜏 𝑡 -1and predicted track τ𝑡 𝑖 , respectively. The matrix F is the state transition matrix. The matrix Σ 𝑡 -1and Σ𝑡 𝑖 are the state covariance at the 𝑡-1 𝑡ℎ frame and the 𝑡 𝑡ℎ frame. The matrix Q is the prediction noise covariance.Besides, utilizing our 𝑠 𝑜𝑐 to adjust the measurement noise covariance allows some low-quality objects to use the predicted tracks more for state correction. We define the track update step as:EQUATIONEQUATIONwhere 𝜇 𝑡 𝑖 is the state mean of the updated track 𝜏 𝑡 𝑖 , 𝜃 𝑡 𝑖 is the state mean of the object. The matrix H is the state transition matrix. 𝐾 𝑡 𝑖 is Kalman gain, reflecting the proximity of the updated track to the object. The matrix R is the measurement noise covariance.Due to the strikingly similar IoU metrics between the predicted track and densely distributed objects, our DIoU incorporates distance to the vanilla IoU, thus providing highly reliable and discriminative correlations for association. Our DIoU is based on Generalized IoU (GIoU) [30] which is adjusted to [0,2]. We normalize the absolute difference in distance Δ𝑑 between the predicted track and the object as a weight. When Δ𝑑 = 0, meaning they have the same distance, DIoU degenerates into GIoU. As Δ𝑑 gradually increases, GIoU can progressively be penalized, based on the prior that the same object should be proximate between adjacent frames in the distance dimension. The DIoU is expressed as follows:EQUATIONwhere Δ𝑑 𝑡ℎ𝑟 is the distance association threshold, and when Δ𝑑 exceeds Δ𝑑 𝑡ℎ𝑟 , it is considered that association will not occur. 𝛽 is used to adjust the penalty strength. 𝐻 (𝑥) is the Heaviside function.Long-term Association. Previous works utilize either state prediction by heuristic motion models (e.g. Kalman filters) with appearance features extracted by ReID models as the basis for reassociation following long-term object loss. However, the former struggles to predict complex motions, and the latter faces challenges in learning stable features across different postures and viewpoints. In our Long-term Association, inspired by [32] , we introduce a Probabilistic Autoregressive Motion Model (PAM) to learn reliable long-term motion prediction from the historical tracklets.For a tracklet T 𝑖 {𝑠:𝑙 } that is lost at 𝑙 𝑡ℎ frames, the long-term motion model essentially iterates to generate the predicted track for the current frame. Given the stochastic nature of object motion, generating a single deterministic predicted track is unreasonable. Therefore, multimodal stochastic modeling is required to compute the probability distribution of the next plausible predicted track, from which samples can be drawn. PAM also learns the interaction representation 𝜓 𝑡 -1 𝑖 between the object and 𝑁 𝜓 neighboring tracklets with MA-Net. The T 𝑖 {𝑠:𝑙 } and 𝜓 𝑡 -1 𝑖 are taken as inputs. Notably, we lift the spatial dimensionality to preserve the role of distance information in re-association. For each track motion velocity in ( 𝑥, 𝑦, 𝑎, ℎ, 𝑑), we utilize non-parametric k-means clustering to obtain K clusters and treat each cluster centroid as a discrete motion class. Through the PAM, the probability distribution of the predicted track can be obtained, as shown in the following equation:EQUATIONwhere 𝑧 𝑡 -2is the hidden state which carries all previous tracklet information until the 𝑡-2 𝑡ℎ frame. 𝛿 𝑡 -1 𝑖 is the track motion velocity in the 𝑡-1 𝑡ℎ frame. The predicted probability distribution is obtained by iteration starting from the lost 𝑙 𝑡ℎ frame. By polynomial sampling, we can get 𝑁 𝑠 possible predicted tracks. Additionally, due to the uncertainty in long-term motion prediction, we also incorporate an independent threshold constraint on the GIoU term in DIoU.  
experiments: 
 Our novel DLT framework is evaluated on three key datasets: MOT17 [24] , MOT20 [5] , and DanceTrack [35] . Moreover, extensive ablation studies are conducted on MOT17 [24] and MOTSynth [8] .Datasets. MOT17 [24] and MOT20 [5] , encompassing diverse real-world challenges like dense crowds and occlusions, require results to be uploaded to the motchallenge website for evaluation metrics. For ablation studies, we split MOT17 [24] into train and val sets. DanceTrack, a large-scale dataset known for body occlusions and random motion, is ideal for evaluating our motion-only DLT method. MOTSynth [8] is a large synthetic dataset providing authentic 3D coordinates for MDE training. We perform appropriate pre-processing to better tailor it to distance estimation tasks.Metrics. Adhering to established MOT evaluation protocols, we use CLEAR metrics [1] including Multiple-Object Tracking Accuracy (MOTA), False Positive (FP), False Negative (FN), ID Switch (IDs), and Tracker Speed (FPS) to evaluate different aspects of tracking performance. In addition, we combine High Order Tracking Accuracy (HOTA) metrics with Detection Accuracy (DetA) and Association Accuracy (AssA) [20] . MOTA focuses on detection performance, IDF1 [31] emphasizes association performance, and HOTA offers a balanced measure of detection and association.Implementation details. We train YOLOX [10] and our MDE on 4 NVIDIA Tesla A100 GPUs. The training approach for YOLOX is kept consistent with ByteTrack [43] to ensure a fair comparison. Our MDE is trained on MOTSynth using 6-frame video clips at 1280 × 720, with clips sampled with a uniform stride of 6 frames. We utilize a batch size of 4, Adam optimizer with an initial learning rate of 2 × 10 -5 for 50 epochs. For SDHM, 𝛼 is 0.5. For SLTA, Δ𝑑 𝑡ℎ𝑟 is set at 5.0 with a 𝛽 of 2.0, and 𝑁 𝑠 is set at 7.For a fair comparison, we classify methods into two types: those combining motion and appearance, and those only using motion.Despite not leveraging any appearance information, our DLT ranks first in most metrics on MOT17, indicating superior performance as shown in Table 1 . By integrating distance cues, DLT achieves discrete and ordered matching with the SDHM strategy. It substantially surpasses the second-ranked MotionTrack [26] in both comprehensive metrics (+1.3 HOTA, +1.1 MOTA) and association metrics (+1.5 IDF1, +1.6 AssA). SDHM also notably reduces the FN metric by incorporating more objects into matching. MOT20. DLT achieves advanced performance in most metrics on MOT20, noted for denser crowds and longer occlusions, as indicated in Table 3 . It demonstrates robust generalization capabilities, markedly outperforming MotionTrack [26] , which performs sub-optimally on MOT17 (+1.5 HOTA, +2.2 IDF1, +2.5 AssA). It also surpasses the second-ranked FineTrack [28] in comprehensive metrics (+0.7 HOTA, +0.4 MOTA). LSTA enhances the association between adjacent frames and re-association after long-term losses. Unlike FineTrack [28] , DLT exhibits comparable performance in association metrics without complex appearance modules.DanceTrack. Tracking in pseudo-3D space with distance cue, DLT demonstrates exceptionally strong performance on Dance-Track with more reliable matching and association, as shown in Table 2 . Focusing on motion, especially with similarly dressed individuals, DLT significantly outperforms FineTrack [28] (+13.8 HOTA, +8.7 IDF1, +15.6 AssA). Additionally, DLT notably surpasses the second-ranked DiffMOT [21] (+4.2 HOTA, +5.5 IDF1, +6.9 AssA), showcasing its effectiveness in motion-based tracking.Table 3 : Comparison of the state-of-the-art methods under the "private detector" protocol on MOT20 test set. For each metric, the best is bolded and the second best is underlined. ↑ / ↓ indicates the higher/lower values denote the better performance. We conduct comprehensive ablation experiments on MOT17 and MOTSynth val sets.We conduct an ablation study on the MOT17 val set to assess the contribution of each proposed component on our distance-aware tracking, as indicated in Table 4 . SDHM is the overall matching strategy comprising Score-based Hierarchizing (SH) and Distance-based Hierarchizing (DH). SH notably enhances comprehensive metrics (+1.3 HOTA, +0.9 MOTA), while DH further improves performance (+0.5 HOTA, +0.6 MOTA). SLTA is a specific association scheme that includes Short-term Association (SA) and Long-term Association (LA). SA considerably boosts association capabilities (+0.7 IDF1, +0.6 AssA), while LA effectively resolves re-association problems after extended occlusions (-20 IDs).Analysis of distance estimators. We report Root Mean Squared Error (RMSE) [18] , 𝜏-Accuracy (𝛿) [40] , and Average Localization Precision (ALP) [45] as distance estimation evaluation metrics, as shown in Table 5 . The ST-Mamba has different bidirectional 3D scans: a) Time-First, organizing spatial tokens by location then stacking them frame by frame; b) Space-First, arranging temporal tokens based on the frame then stacks along the spatial dimension; c) Space-Time, a hybrid of both Space-First and Time-First. The Space-First bidirectional scan is the most effective method which is employed in MDE. The BUFF employs bidirectional fusion to achieve multi-granularity extraction of spatiotemporal information. The experimental results demonstrate that bidirectional fusion offers advantages over both top-down and bottom-up fusion. (i) Effect of SH. SH aims to separate negative and positive objects based on object scores, avoiding the generation of erroneous associations. We introduce an Occlusion Compensation Score (𝑠 𝑜𝑐 ) to counter the impact of occlusions on scoring, adjusted by approximate visibility ratios obtained from the distance order. By randomly sampling 10 4 object samples, we statistically analyze the distribution of both negative and positive object scores. The SH threshold divides objects into low-and high-score subsets. The error rate is the ratio of negative objects in the high-score subset to the total negatives, or the ratio of positive objects in the high-score subset to the total positives. Compared to the confidence score (𝑠) directly provided by the detector, our 𝑠 𝑜𝑐 plays an important role in improving hierarchizing accuracy, as illustrated in Figure 5 . (ii) Effect of DH. Integrating DH into the high-score subset of SH aims for discrete and ordered matching in distance dimension. As shown in Table 6 , we explore several various DH settings. Switching from equal-distance decomposition (ED) to equal-object-number decomposition (EN) decreases performance slightly (-0.1 HOTA, -0.3 MOTA). Matching from far to near (F→N) is obviously less effective than near to far (N→F) (-0.3 HOTA, -0.4 MOTA). Using DH in both high-score (H) and low-score (L) subsets shows essentially equivalent performance (+0.1 HOTA, -0,2 MOTA), unlike its negative impact when applied only in L (-0.4 HOTA, -0.3 MOTA).Analysis of SLTA. We perform experimental analyses of Shortterm Association (SA) including Pseudo-3D Adaptive Kalman Filter (AKF) and Distance-weighted IoU (DIoU), and Long-term Association (LA) including Probabilistic Autoregressive Motion Model (PAM).(i) Effect of SA. Our SA integrates AKF and DIoU for accurate state estimation and track updates. AKF evolves from 2DKF [2] to P3DKF, and utilizes distance for scaling prediction noise (SPN) and update noise (SUN) with our 𝑠 𝑜𝑐 , enabling adaptive noise adjustments. As shown in Figure 6 (a), compared to 2DKF, AKF introduces a distance dimension to state estimation, with SPN and SUN further enhancing association performance (+0.4 HOTA, +0.7 AssA). Furthermore, we develop a DIoU, integrating object-level distance as a potential cue into the correlation calculation, outperforms IoU, GIoU [30] , SIoU [11] , and HMIoU [41] , as indicated in Figure 6 (b) .(ii) Effect of LA. Our LA tackles extreme occlusion, thereby allowing up to 120 frames for object loss to demonstrate the capability of handling long-term tracking, as shown in Table 7 . We evaluate a no-motion model (assuming the lost object remains static), a linear model (i.e., AKF), and a nonlinear multimodal model (i.e., our PAM) for long-term identity retention. Compared to Static and AKF, PAM remarkably addresses long-term occlusions by capturing the randomness and multimodality of human motion. Additionally, MA-Net's consideration of neighboring object interactions boosts Extra evaluation of occlusion degree. MOT17 provides the visibility ratio annotations of objects, and we consider those with a visibility ratio below 0.3 as occluded. The occlusion degree can be represented by the number of frames in which the occlusion state persists. As the occlusion degree increases, DLT shows slower declines (-7.1 HOTA) compared to ByteTrack [43] (-11.2 HOTA) and Hybrid [41] (-10.7 HOTA), as indicated in Figure 7 (a) .Extra evaluation of low frame rates. At low frame rates, the impact of occlusion is further magnified. We use a frame sample operation for simulation as shown in Figure 7 (b). The performance of ByteTrack [43] and Hybrid [41] declines rapidly with decreasing frame rates (-11.6 HOTA and -7.7 HOTA), whereas DLT demonstrates superior low-frame-rate stability (-4.3 HOTA). In addition, SDHM and SLTA play obvious roles in performance retention. 
conclusion: 
 In this paper, we propose an innovative "Detecting-Lifting-Tracking" (DLT) framework for 2D MOT. We introduce a Mamba Distance Estimator, incorporating historical information to mitigate temporary occlusions, achieving object-level pseudo-3D lifting. To realize distance-aware tracking, we propose a Score-Distance Hierarchical Matching strategy for discrete and sequential matching, along with a Short-Long Terms Association scheme to address issues of dense crowds in short-term associations between adjacent frames and severe occlusions in long-term re-associations. Extensive benchmark validations and ablation studies demonstrate the superior performance of our DLT, showcasing the enormous potential to overcome occlusion challenges. We hope to inspire future research to further explore enhancing 2D MOT tasks with 3D information. 
 
 Appendix:  


Decision: The paper is well written, and an interesting idea - a Mamba distance estimator that combines historical information for time-consistent distance estimation enabling object-level pseudo-3D lifting - is proposed. All the three reviewers recommend acceptance. AC agrees with the reviewers.

Reviews: review: paper_summary: paper_summary: The "Detecting-Lifting-Tracking" (DLT) framework for 2D multi-object tracking utilizes pseudo-3D lifting and distance-aware tracking, achieving state-of-the-art performance on MOT17, MOT20, and DanceTrack without relying on appearance cues.
strength_weakness: strength_weakness: 1.. Lifting: Employing Object-Level Distance Estimation (OLDE) with the Mamba Distance Estimator to create pseudo-3D information for each object.
2.. Tracking: Implementing distance-aware tracking through Score-Distance Hierarchical Matching (SDHM) combined with Short-Long Terms Association (SLTA).
3. This paper is well-written.
limitations: limitations: 1.	Is it suitable for real-time? Score-Distance Hierarchical Matching (SDHM) combined with Short-Long Terms Association (SLTA) are post-processing.
2.	The complexity of Detecting-Lifting-Tracking” (DLT) framework can be discussed since MDE can reduce some computation effort.
3.	Can you show the ablation studies for the Mamba module?
4.	The performance of DIoU should be testified.
paper_summary: paper_summary: This paper presents Detecting-Lifting-Tracking (DLT) framework for Multi-Object Tracking (MOT). DLT firstly performs Object-Level Distance Estimation (OLDE) that estimates the distance between each object in a frame and the camera using Mamba Distance Estimator (MDE). The estimated object-level distances are then used in the distance-aware tracking process consisting of two subprocesses, Score-Distance Hierarchical Matching (SDHM) and Short-Long Terms Association (SLTA). The experimental results on the three datasets show the effectiveness of the proposed DLT.
strength_weakness: strength_weakness: - DLT seems to apply (or extend) very new methods to each of its component processes like Mamba in OLDE, Score-/Distance-based Hierarchizing, GIoU in the short-term association and Probabilistic Autoregressive Motion (PAM) in the long-term association.
- It seems that very extensive experiments have been conducted to validate the effectiveness of each component process in DLT.
limitations: limitations: - I think the biggest problem is that this paper contains too many contents, so each component process in DLT is described very briefly. By referring to the supplemental material, I could manage to get a rough idea about each of the component processes and their relations. But, it seems very difficult to get reasonable understanding of DLT only from the main paper.
- Related to the above point, many things are described without enough explanation. For example, why is the distance $d$ of an object not directly estimated, that is, why are $\hat{\mu}$ and $\hat{\sigma}^2$ of a Gaussian distribution for distances are estimated? Why is GNLL is used (what is the advantage of GNLL)? What is meant by "hierarchical strategy sensitivity to occlusions"? What is the rationale behind Eq. (8) to compute $s_{oc}$? How are $\boldsymbol{F}$, $\boldsymbol{Q}$, $\boldsymbol{H}$, $\boldsymbol{R}$, $\boldsymbol{GIoU}$ computed? What is the rationale behind Eq. (13)? What is MA-Net? and so on. Despite these many unknown things, I could manage to imagine how DLT works by referring to the supplemental material.
- DLT seems to be a combination of the component processes each of which is a slight extension of an existing method. For instance, Mamba is used to extend OLDE to the time dimension, Hierarchizing, Adaptive Kalman Filter (AKF) and GIoU are extended by adopting object-level distances. So, someone may think the contributions of this paper are relatively small.
- Is it possible to devise a baseline method in order to examine how important object-level distances?
- How can I understand Figure 5?
paper_summary: paper_summary: This paper introduces an innovative 2D MOT framework, “Detecting-Lifting-Tracking” (DLT). Specifically, a mamba distance estimator is proposed to obtain the distances from objects to a monocular camera, achieving object-level pseudo-3D lifting. Then, a score-distance hierarchical matching technique is introduced to achieve discrete and sequential matching. Finally, a short-long term association strategy is employed to optimize short-term associations between adjacent frames and to enhance the long-term re-association capability. A large number of experiments demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The paper is well written and can be easily understood by the reader.
2. The idea of the paper is interesting, presenting a novel Mamba distance estimator that combines historical information for time-consistent distance estimation, enabling object-level pseudo-3D lifting.
limitations: limitations: - The necessity of mamba. Whether the performance of the proposed algorithm will be improved if the mamba operator is replaced by using transformer.