S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
Authors: Tao Tang
Lijun Zhou
Pengkun Hao
Zihang He
Kalok Ho
Shuo Gu
Zhihui Hao
Haiyang Sun
Kun Zhan
Peng Jia
XianPeng Lang
Xiaodan Liang
Venue: ICML 2025 poster (2025)
Keywords: 3D Multi-Object Tracking
S2-Track
Primary area: applications->computer_vision
TL;DR: A simple yet strong approach for end-to-end 3D multi-object tracking

Abstract: 3D multiple object tracking (MOT) plays a crucial role in autonomous driving perception. Recent end-to-end query-based trackers simultaneously detect and track objects, which have shown promising potential for the 3D MOT task. However, existing methods are still in the early stages of development and lack systematic improvements, failing to track objects in certain complex scenarios, like occlusions and the small size of target object’s situations. In this paper, we first summarize the current end-to-end 3D MOT framework by decomposing it into three constituent parts: query initialization, query propagation, and query matching. Then we propose corresponding improvements, which lead to a strong yet simple tracker: S2-Track. Specifically, for query initialization, we present 2D-Prompted Query Initialization, which leverages predicted 2D object and depth information to prompt an initial estimate of the object’s 3D location. For query propagation, we introduce an Uncertainty-aware Probabilistic Decoder to capture the uncertainty of complex environment in object prediction with probabilistic attention. For query matching, we propose a Hierarchical Query Denoising strategy to enhance training robustness and convergence. As a result, our S2-Track achieves state-of-the-art performance on nuScenes benchmark, i.e., 66.3% AMOTA on test split, surpassing the previous best end-to-end solution by a significant margin of 8.9% AMOTA. We achieve 1st place on the nuScenes tracking task leaderboard.

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
Tao Tang
Lijun Zhou *2 Pengkun Hao
Zihang He Kalok Ho Shuo Gu Zhihui Hao Haiyang Sun
Kun Zhan
Peng Jia? Xianpeng Lang² Xiaodan Liang?
Abstract
1. Introduction
3D multiple object tracking (MOT) (Li et al., 2023d; Doll et al., 2023; Pang et al., 2023; Qing et al., 2023; Yang et al., 2022; Li et al., 2023a; Wang et al., 2023a; Ding et al., 2024) is an essential component for the perception of autonomous driving systems. The ability to accurately and robustly track objects dynamic environments crucial for ensuring smooth and safe navigation and reasonable decision-making Traditional 3D MOT methods (Yang et al., 2022; Chaabane et al., 2021; Zhou et al., 2020; Yin al., 2021; Pang et al., 2022; Weng et al., 2020; Wojke et al., 7017; Guo et al., 2024) rely on detector outcomes followed by a post-processing module like data association and trajectory filtering, leading to a complex pipeline. To avoid human-crafted heuristic design in detection- trackers, recent advancements in end-to-end query-based approaches have shown impressive potential in addressing the 3D MOT task by simultancously detecting and tracking objects (Zeng et al., 2022; Zhang et al., 2022a; Doll et al., 2023; Li et al., 2023d; Pang et al., 2023; Ding et al., 2024). These methods have demonstrated promising results terms of tracking performance and ef- ficiency. However, current end-to-end trackers are still in the early stages of development and can not effectively han- dle the various complex driving scenarios with their naive solution.
3D multiple object tracking (MOT) plays cru- cial role in autonomous driving perception. Re- cent end-to-end query-based trackers simultane- ously detect and track objects, which have shown promising potential for the 3D MOT task. How- ever, existing methods are still in the early stages of development and lack systematic improve- ments, failing to track objects in certain complex scenarios, like occlusions and the small size of target object's situations. In this paper, we first summarize the current end-to-end 3D MOT frame- work by decomposing it into three constituent parts: query initialization, query propagation, and query matching. Then we propose corresponding improvements, which lead to a strong yet simple tracker: S2-Track. Specifically, for query initial- ization, we present 3D-Prompted Query Initial- ization, which leverages predicted 2D object and depth information to prompt an initial estimate of the object's 3D location. For query propagation, introduce an Uncertainty-awer aware Probabilistic Decoder to capture the uncertainty of complex environment in object prediction with probabilis- tic attention. For query matching, we propose a Hierarchical Query Denoising strategy to en- hance training robustness and convergence. As result, our S2-Track achieves state-of-the-are performance nuScenes benchmark, i.e., 66.3% AMOTA test split, surpassing the previous best end-to-end solution by significant margin of 8.9% AMOTA. We achieve 1st place on the nuScenes tracking task leaderboard.
In driving scenarios, the environment could be highly com- plex, often when driving in cities, with numerous objects such as vehicles and pedestrians interleaving across the scene, and exhibiting substantial variations in their motion patterns. Furthermore, tracked objects often cover wide spatial tracking range and long temporal tracking sequence. As a result, occlusion situations and the small size of tar- get objects. frequently occur. which usually leads to some undetected or occluded objects losing track. These factors present significant challenges to current vanilla end-to-end query-based approaches for achieving accurate and robust 3D MOT. As shown Figure 1 (a), the previous state-of- the-art end-to-end tracker, PF-Track (Pang et al., 2023), fails to track objects in challenging scenarios.
Equal contribution, Work done during internship Li Auto Inc. 'Shenzhen Campus Sun Yat-sen University Li Auto Inc. Corre- spondence to: Xiaodan Liang <liangxd9 @ mai..sysu.e....... sysu.edu.en>..
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we aim to comprehensively enhance the exist- ing end-to-end 3D MOT framework to achieve robust and

---

S2- S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Objeet Tracking
Smalll Soze Objoots
We delve into the current end-to-end 3D MOT frame- work and decompose it into three constitute modules, and propose a stronger yet simple framework, S2- Track, which enhances each module comprehensively.
Tricking Fallre Sccnutios
We propose three well-designed modules, 2D- Prompted Query Initialization, Uncertainty-aware Probabilistic Decoder, and Hierarchical Query Denois- ing to enhance the previous pipeline from multiple aspects, leading to improved tracking performance.
Ous Pipeline e
10 Ofject Buaaz
Query hriuiizaiin
Query Proagaiion
Quary Mashing
Ofject Queries
Updand sperie
2D Prompted Query
Uncertuinty- aware Prolualilitic Decusler
Elerarchical Query Denalling
We demonstrate the effectiveness of our S2-Track framework quantitatively and qualitatively through extensive experiments nuScenes benchmark and achieve leading performance with a remarkable 66.3% AMOTA.
Inieakatiion
+2.9
Figure 1: (a) complex driving scenarios, there are various challenge factors, e.g., the occlusions and small size of target objects, which present significant challenges to achieving accurate tracking. The previous state-of ahrtt and-to-end tracker, PF-Track (Pang et al., 2023), fails to track objects in certain complex scenarios. (b) Our S2-Track proposes three simple yet strong modules to enhance baseline com- prehensively, leading to improved tracking performance.
2. Related Work
Tracking by Detection, Multi-object tracking (MOT) in 3D scenes takes multi-view images from surrounding cam- eras or LiDAR point clouds to track multiple objects across frames (Marinello et al., 2022; Wang et al., 2023b; Qing et al., 2023; Fischer et al., 2022; Yang et al., 2022; Li et al., 2023a; Wang et al., 2023a; Sadjadpour et al., 2023; Guo al., 2024). Taking advances in 3D object detection (Huang et al., 2021; Li al., 2023c; 2022b; Liang et al., 2022; Liu et al., 2023b;a; Lin et al., 2023; Wang al., 2022), most 3D MOT methods follow the tracking by de- tection paradigm (Yang et al., 2022; Chaabane et al., 2021; Zhou et al., 2020; Zhang et al., 2022b; Yin et al., 2021; Pang et al., 2022), where tracking is treated as a post-processing step after object detection. Take the detected objects at each frame, traditional 3D MOT usually uses motion models, e.g., Kalman filter (Weng et al., 2020; Wojke et al., 2017), to pre- dict the status of corresponding trajectory and associate the candidate detections using 3D loU (Pang et al., 2022; Weng et al., 2020) or L2 distance (Zhou et al., 2020; Yin et al., 2021).
accurate tracking results in complex driving environments. As illustrated in Figure (b), we first delve into the current query-based framework and decompose it into three con- stituent parts: query initialization, query propagation, and query matching. Then we propose corresponding improve- ments, which lead to a Strong yet Simple tracker: S2-Track. Firstly, for query initialization, we present the 2D-Prompted Query Initialization module, which leverages predicted 2D object location and depth information to enhance the accu- racy of initial object localization, leading more reliable tracking results. Secondly, for query propagation, we intro- Probabilistic Decoder to capture
duce an Uncertainty- and model the uncertainty of complex environments during object prediction. Specifically, we model attention scores as Gaussian distributions instead of deterministic outputs, to quantify the predictive uncertainty. Moreover, for query matching, we propose an Hierarchical Query Denoising strategy to further improve the training process. During the training stage, we add noises to ground-truth bounding boxes to form noised queries and selectively denoise queries based on their noised levels, enhancing robustness and con- vergence. Experimental results on the nuScenes benchmark demonstrate the effectiveness of our S2-Track framework. achieves -tate-of-the-art performance with an impressive 66.3% AMOTA on the test split, surpassing the previous best end-to-end solution by a significant margin of 8.9% AMOTA. These results highlight our simple yet non-trivial improvements and showcase the potential of our framework in advancing the field of autonomous driving perception.
Tracking with Query. To overcome the independent na- ture of the detection and tracking and implicitly solve the association between frames, the recent tracking with query paradigm models the tracking process with trans- former queries (Zeng et al., 2022: Zhang et al., 2022a; Doll et al., 2023; Li et al., 2023d; Pang et al., 2023; Ding et al., 2024). MUTR3D (Zhang et al., 2022a) extends the object detection method DETR3D (Wang et al., 2022) for tracking by utilizing a 3D track query to jointly model object features across timestamps and multi-view. STAR-TRACE (Doll et al., 2023) proposes a latent motion model to account for the effects of ego and object motion on the latent ap- pearance representation. DQTrack (Li et al., 2023d) sepa- rates object and trajectory representation using decoupled queries, allowing more accurate end-to-end 3D tracking.
To summarize, our contributions are as follows:
2

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
Track Queries
Uncertainty-aware Probabilistic Decoder
MOT
Oulputs
Image Features
Backbone FPN
Multi-view Images (frume !)
Detector
DepthNer
Denoised Roxes
GT Boxes
Query
Box
Neg
Ign
Track Queries from
Prompted Init Queries
Random Init Queries
Noised Queries Multi-level Noised Boxes
Hierarchical Query Demoising
Figure 2: S2-Track framework. The proposed 2D-Prompted Query Initialization (PQI), Uncertainty- awwae Probabilistic Decoder (UPD), and Hierarchical Query Denoising (HQD) are incorporated together to improve tracking performance. Neg: negative, Ign: ignore, Pos: positive, Mask: separate the normal queries and the denoising part to prevent information leakage.
PF-Track (Pang et al., 2023) extends the temporal horizon to provide a strong spatio-temporal object representation. Although these methods achieved impressive performance, when applied complex scenarios, especially the occlu- sions and the small target objects, the tracking performance becomes unsatisfactory. In this work, delve into the cur- rent end-to-end 3D MOT framework and decompose it into three modules, and further propose corresponding improve- ments to enhance each module comprehensively. As a result, these improvements lead to a strong yet simple framework, 52-Track, and consequently achieve leading performance.
Queries initialization. Specifically, following previous studies (Pang et al., 2023; Ding et al., 2024), our S2-Track utilizes set object queries to tackle multi-object tracking from multi-view images. Each query q Q: represents a unique 3D object with feature vector f and a 3D location c/. i.e., qi (f. c.}. The queries randomly initialized as learnable embeddings, denoted Qinit- Here, we pro- pose 2D-Prompted Query Initialization (orange module) to improve the query initialization with predicted 2D object location and depth information.
Queries propagation. During training, the object tracked by updating its unique query. Specifically, the object queries Q, { are propagated from the previous frame 1 (colored squares) and numerous initial queries (gray squares in Figure 2):
3. S2-Track
In this section, we introduce our S2-Track in detail. We first give a brief problem definition and an overview of the framework in Figure 2. Then, we clarify our key contribu- tions: DD-Prompted Query Initialization, Uncertainty-aware Probabilistic Decoder, and Hierarchical Query Denoising.
Q + (.rro(Q.--1 Q
(I)
The queries from the previous frame represent tracked in- stances, while numerous initial queries aim to discover new objects. Here, we introduce an Uncertainty- bilistic Decoder (blue module) to model and capture the uncertainty of complex environments in object prediction.
3.1. Preliminaries
Proba-
Overview, At each timestamp f, given images from sur- rounding cameras, the tracking objective is estimate set of bounding boxes bid B, with consistent id across frames. Under the trucking with query paradigm, an overview of proposed S2-Track is presented in Figure 2, which is conceptually simple: encoder and transformer de- coder are adopted to encode input images and decode 3D MOT outputs with queries. From the perspective of the query, it can be divided into three stages: query initial- ization, query propagation with image features, and query matching with ground truth.
Queries matching. Then, to predict 3D bounding boxes, decoder-only transformer architectures such as DETR3D (Wang et al., 2022) and PETR (Liu et al., 2023a), are utilized to decode image features F, with object queries:
B, Q, Decoder (F,, Q.).
where B, and Q2 are the detected 3D bounding boxes and updated query features respectively. Here, we present an Hierarchical Query Denoising strategy (green module) to enhance the model robustness and convergence.
3

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Objeet Tracking
In the following sections, we give detailed elaboration.
Image Features
3.2. 2D-Prompted Query Initialization
Salf Anrenion
In tracking with query frameworks, high-quality initial queries are crucial for achieving rapid convergence and improving tracking precision. This becomes particularly im- portant when dealing with complex driving scenarios, such as occlusion and small-sized target objects. In previous methods that solely rely on random query initialization, the lack of reliable initial queries often results in some unde- tected or occluded objects losing track. To address this, we propose Prompted Query Initialization (PQI) module which enhances the initialization of queries using learned certain priors obtained from network training. Specifically, after utilizing the shared image backbone and the feature pyramid network (FPN) layers to extract image features from each camera, we introduce additional auxiliary tasks, i.e. the 2D detection and the depth prediction, as follows:
Neno
Probabilistic Atteattion
Nerm
FFN
Nerm
Object
Updated Queries
Queries
Sampling
N(
Figure 3: Uncertainty-aware Probabilistic Decoder (UPD) architecture. The traditional cross-attention is up- graded with probabilistic attention to quantifying the un- certainty. The probabilistic attention utilizes a multi-layer perception that takes the query q and key k as input to gener- ate the mean and standard deviation, which are used to form a Gaussian distribution. Subsequently, the attention value is sampled from the constructed Gaussian distribution.
B D NetworkSansitfany! (Fr),
(3)
where B2dd D, denotes the 2D bounding boxes and the depth respectively. The 2D detection head follows YOLOX (Ge al., 2021). The depth network combines multiple residual blocks and is supervised with the projected LiDAR points. The optimization objectives of the two auxiliary tasks are:
end-to-end 3D MOT methods, leading to uncertainties. The uncertainty issue refers neural networks that do not de- liver certainty estimates suffer from underconfidence,, as current methods struggle to capture the noise and variations inherent in the input data. Although the uncertainty issue has been recognized in certain fields (Gawlikowski et al., 2021; Subedar et al., 2019; Wang et al., 2021), e-g-, action recognition (Guo et al., 2022) and camouflaged object detec- tion (Yang et al., 2021), it has not been discussed explored in the context of 3D MOT. Moreover, due to the complex driving scenarios and the unique challenge of tracking tasks, previous solutions for specific domains cannot be directly applied here.
CPQ = Bet + Depth.
(4)
Then, estimate 3D location C, = {c(} through the coor- dinate transformation: C. Tridar DD 1|T,, where cam and K-1 represent the transformation matrix from the camera coordinate system to the lidar coordinate sys- tem and the camera's 's intrinsic parameters respectively, and (u, v) denotes the center location of the 2D boxes. Then we initialize our object queries with the preliminary 3D loca- tion, denoted as Qpor-inir (dark gray squares in Figure 2). We also retain the random initialization Qiii to explore missing objects. To summarize, the query initialization and propaga- tion process in Equation (1) can be improved as follows:
To address this limitation, inspired by previous uncertainty quantify works (Guo et al., 2022; Pei al., 2022; Blundell et al., 2015), we introduce Uncertainty-aware Probabilistic Decoder (UPD) for 3D MOT. Current end-to-end meth- ods employ decoders with conventional transformers us- ing deterministic attention mechanisms, which compute determinate attention a between queries (Q) and keys (K) as a = Q-K where is the dimension of queries and keys. The deterministic attention limits the ability to quantify uncertainty in predictions effectively. Instead, our UPD utilizes the probabilistic attention computation, which assumes attention a follows Gaussian distribu- tion: ai₃ ~ "Muuj otz). Through the reparameterization trick (Kingma et al., 2015): aij = May + oije, /N 1). As illustrated in Figure 3, a multi-layer perception is adopted to fit the mean and standard deviation with qi, kr as in- put: Mij. o² = MLP(ge k,). Thus, we introduce the uncertainty aware probabilistic parameters Psy and oi into the decoder, allowing uncertainty adaptation in the training
Q + Prop(Q.-11 Qpor-inics
Qinit).
We also provide qualitative results in Fig. of Appendix, which shows the initial queries generated by our PQI module are accurately positioned within the regions of interest for the objects, resulting in more accurate tracking results.
Vd
3.3. Uncertainty-aware Probabilistic Decoder
In complex driving scenarios, the trajectories of multiple tar- gets exhibit substantial variations in their temporal duration and sequence. Furthermore, the target objects themselves can vary in size, ranging from large trucks to small children. These diversities present significant challenges for current
4

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
process. Practically, utilize the negative log-likelihood loss constrain the probabilistic attention as:
where L and boz are respectively focal loss (Lin et al., 2017) and LI loss for the classification and box loss of B,". while Cels is focal loss to distinguish the background.
k: (ag
LUPD- = Σ log( 3
exp(
vi
(6)
By incorporating the query denoising and handling the noised queries based on their noise/challenging levels, our HQD module enhances the training robustness and conver- gence, leading to more stable and accurate results.
2TO
20 tj
As a summary, our UPD module captures and models the uncertainty of complex driving environments by represent- ing attention scores as Gaussian distributions, which are more robust and capable of handling variations and noise in the 3D MOT, and the Equation (2) can be improved as:
3.5. Overall Optimization
To summarize, the overall optimization target of our S2- Track is formulated as:
B, Q f Decoderuro(FF, Q.).
L -Ntracking + APQICrQI
(10)
+ +AUDDLUPD + λHQDLHOD.
3.4. Hierarchical Query Denoising
where Ltracking = -ele+ Lets and Lsor are classifica- tion loss and box loss for the tracked objects, Lroi. Lu and LHQD are defined in Equation (4), Equation (6), and Equation (9), and À indicate the weight balance coefficients that are all set to 1.0 by default.
In complex 3D MOT scenarios, challenges such as occlu- sions and varying object sizes can hinder the learning and convergence of query-based methods. The slow conver- gence and suboptimal results from the instability of bipartite graph matching. To address this challenge, we draw in- spiration from DN-DETR (Li et al., 2022a) and propose Hierarchical Query Denoising (HQD) training strategy, which incorporates query denoising to enhance training process for stable optimization. We perturb GT bounding boxes with noises into the decoder and train the model to reconstruct the original boxes, which effectively reduces graph matching difficulty and leads to faster convergence.
4. Experiment
4.1. Experimental Setup
We conduct experiments on the large-scale nuScenes bench- mark (Caesar et al., 2020) and follow the official evaluation metries from nuScenes. Detailed Metric and Implementa- tion are present in the Appendix.
Specifically, we start by perturbing the ground truth boxes to generate noised queries. To enhance the model's ability to handle various complex driving scenarios, define hi- erarchical challenging levels for the perturbed queries. We set lower and upper bound thresholds, denoted as Slower and Supper respectively, to help categorize the noised queries into three classes based on their challenging levels. We identified positive samples ("Pos" in Figure 2), i.e., low-challenging samples, when the 3D Intersection over Union (loU) be- tween noised query and its corresponding ground truth ceeds the Supper threshold, i.e., LoU,; > Supper. Conversely, negative samples ("Neg" in Figure 2), i.e., high-challenging samples, are defined when the 3D loU falls below the Slwwer threshold, i.c., LoU/ 4;: < Blwerr Intermediate loU values are disregarded ("Ign" in Figure 2), as they do not provide clear indications of any challenging factors, and can disrupt the normal query learning process as demonstrated in Ta- ble 5. The resulting set of noised queries that meet these requirements is denoted as Q noised: and then the decoder process can be expanded as:
4.2. State-of-the-art Comparison
Tracking on nuScenes val set. In Table 1, we compare our S2-Track with state-of-the-art methods on nuScenes val set. First, method significantly outperforms existing algorithms across all tracking metrics, whether they end- to-end or non-end-to-ent methods. Specifically, S2- 2r-Track achieves impressive performance with 65.2% AMOTA and 0.924 AMOTP. When compared with the previous query- based tracker Sparse4D-v33, the performance gap is further enlarged to 8.5% AMOTA. Second, we validate the gener- ality of S2-Track by applying different encoder backbones, i.c., V2-99 and ViT. Equipped with V2-99, the proposed framework achieves consistent gains with 8.7% AMOTA (S2-Track-F 0.566 vs PF-Track-F 0.479 vs ADA-Track 0.479). Moreover, when employing the larger ViT-L. as our backbone, we further achieve leading performance.
Tracking on nuScenes test set. In Table 2, we compare our S2-Track with sfate-of-ther camera- based methods on nuScenes test set. Our proposed S2-Track maintains an end- to-end tracking pipeline without heuristic proc-psogs and achieves leading performance with 66.3% AMOTA, surpassing the previous best solution Sparse4D-v3 by a significant margin of 8.9% AMOTA.
B HQD HQD +- Decoder(F.. QuoDD-nisedd (8)
For optimization, the loss for positive samples and negative samples are calculated to form the target:
LHQD = - + CPos + fase
5

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
Table 1: Comparisons with previous methods on the nuScenes val set. Our S2-Track outperforms all existing camera-
based 3D MOT methods in all metrics.
Method
Backbone
AMOTAT
AMOTP4
RECALL1
MOTAF MOTP 4 IDS
DEFTT [arxiv21]
(Chaabane et al., 2021)
DLA-34
0.201
0.171
QD-3DT [TPAI12022) (Hu al., 2022)
DLA-34
0.242
1.518
39.9%
0.218
5646
MSGLMB [CCCI1244 (Ma et al., 2024)
0.382
1.235
55.6%
0.399
2929
CC-3DT (CoRL2022] (Fischer al., 2022)
R101
0.429
1.257
53.4%
0.385
2219
Cyclic [UCV24] (Guo et al., 2024)
R101
0.469
1.002
45.7%
0.354
3613
QTrack [arxiv23] (Yang et al., 2022)
V2-99
0.511
1.090
58.5%
0.465
1144
RockTrack [arxiv24] (Li et al., 2024)
0.514
1.144
Tracking with Query
MUTR3D [CVPR2022] (Zhang et al., 2022a)
R101
0.294
1.498
42.7%
0.267
0.709
3822
STAR-TRACK TRACK [RA-L2023] (Doll et al., 2023)
R101
0.379
1.358
50.1%
0.360
372
Sparse4D-v3 [anxiv23) (Lin et al., 2023)
R101
0.567
1.027
65.8%
0.515
0.621
557
DQTrack [1oCV23] (Li al., 2023d)
V2-99
0.446
1.251
62.2%
1193
PF-Track-S [CVPR2023] (Pang al., 2023)
V2-99
0.408
1.343
50.7%
0.376
166
HSTrack [arxiv24] (Lin et al., 2024)
V2-99
0.464
1.262
56.9%
0,423
204
PF-Track-F [CVPR2023] (Pang et al., 2023)
V2-99
0.479
1.227
59.0%
0.435
181
ADA-Track [CVPR2024] (Ding et al., 2024)
V2-99
0.479
1.246
60.2%
0.430
767
S2-Track-IF (Ours)
V2-99
0.566
1.090
62.2%
0.498
0.657
174
S2-Track-S (Ours)
VIT-L
0.600
1.020
68.8%
0.538
0.614
167
S2-Track-F (Ours)
ViT-L
0.652
0.924
72.2%
0.574
0.577
134
"S" and "F"" represent the settings of small-resolution and full-resolution respectively.
4.3. Analysis Study
i.e., different visibilities, different object sizes, and different distances. Our S2-Track achieves consistent improvements over PF-Track (Pang et al., 2023) under all visibility, object size, and distance settings. Furthermore, S2-Track brings larger improvements in more challenging situations, such as lower visibilities (+ 6.3% AMOTA), smaller objects (+ 5.5% AMOTA), and more distant objects (+ 4.1% AMOTA), both of which demonstrate our effectiveness addressing diverse challenges MOT.
Analysis and ablations the proposed modules of S2- Track. In Table we validate our proposed modules for model's performance nuScenes val set. It is clear that incorporating each module leads to performance gain in tracking. Specifically, the Uncertainty-awan Probabilistic Decoder (UPD) module significantly improves the baseline with 4.4% AMOTA, and the 2D-Prompted Query Initializa- tion (PQI) and Hierarchical Query Denoising (HQD) mod- ules obtained consist boost with 2.4% and 2.9% AMOTA respectively. Moreover, combining the three modules leads to further improvements.
4.4. Ablation Study
4.4.1. 1. ABLATIONS ON THRESHOLDS OF HQD MODULE.
Analysis of uncertainty. Quantifying uncertainty can be challenging for transformer-based models which do not in- herently provide uncertainty estimates like Bayesian meth- ods. Thus, we can only employ computationally intensive strategies to approximate uncertainty, such as the Monte Carlo Dropout (MC dropout) (Gal & Ghahramani, 2016) and Ensemble strategy (Ens.) (Lakshminarayanaa et al., 2017). As shown in Table 3, the proposed UPD module suc- cessfully reduces uncertainty. Surprisingly, other modules also effectively reduce uncertainty, even though they were not designed to aim at uncertainty.
In Table 5, we validate various settings the threshold of HQD module. (I) In the first setting, we do not con- sider the noise thresholds, meaning that all noised queries undergo box optimization. (2) In the second setting, do not selectively denoise queries based on hierarchical challenging levels; instead, single threshold is used to clas- sify all noised queries into positive and negative samples for optimization. (3) In our HQD module, we selectively denoise queries based on hierarchical challenging levels. The improved performance demonstrates our HQD, which optimizes samples based on different challenging levels, effectively handles various complex driving environments, and achieves superior results. We also investigate varying lower and upper bound thresholds, i.c., Srower and Pupper.
Analysis of complex situations. In Table 4, analyze the performance of S2- Track under different driving conditions,
6

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Objeet Tracking
Table 2: Comparisons with state-of-the-at camera-based methods on the nuScenes test set.
Method
Backbone
AMOTA
AMOTP,
RECALLA
MOTAf
MOTP 4
IDSI
Perma'rrak [CCC2202] (Tokmakov et al., 2021)
DLA-34
0.066
1.491
18.9%
0.060
0.724
3598
DEFT [anxi221] (Chaabane et al., 2021)
DLA-34
0.177
1.564
33.8%
0.156
0.770
6901
QD-3DT [XPAM12022) (Hu al., 2022)
DLA-34
0.217
1.550
37.5%
0.198
0.773
6856
CC-3DT (CoRL2022) (Fischer al., 2022)
R101
0.410
1.274
53.8%
0.357
0.676
3334
Cyclic (UCV24) (Guo al., 2024)
R101
0.433
1.055
49.2%
0.334
6621
QTrack [arxiv23] (Yang et al., 2022)
V2-99
0.480
1.100
58.3%
0.431
0.597
1484
Tracking with Query
MUTR3D [CVPR2022] (Zhang et al., 2022a)
R101
0.270
1.494
41.1%
0.245
0.709
6018
PF-Track-F [CVPR2023] (Pang et al., 2023)
V2-99
0.434
1.252
53.8%
0.378
0.674
249
STAR-TRACK [RA-122025] (Doll et al., 2023)
V2-99
0.439
1.256
56.2%
0.406
0.664
607
DQTack (ICCV23) (Li et al., 2023d)
V2-99
0.523
1.096
62.2%
0.444
0.649
1204
Sparse4D-v3 [anxv233] (Lin et al., 2023)
V2-99
0.574
0.970
66.9%
0.521
0.525
669
ADA-Track [CVPR2024] (Ding et al., 2024)
V2-99
0.456
1.237
55.9%
0.406
834
S2-Track-F (Ours)
V2-99
0.608
0.925
75.8%
0.547
0.559
963
S2-Track-F (Ours)
ViT-L
0.663
0.815
72.3%
0.554
0.530
844
"F" represent on the full-resolution settings.
Table 3: Analysis and ablations on the proposed modules of S2-Track.
Method
Tracking
Uncertainty
UPD
PQI
HQD
AMOTAt
AMOTP4
RECALLA
MOTAT
МОTP
IDS
s.
0.
X
X
0.394
1,363
51.4%
0.372
0.753
178
1.99
0.108
MC Dropout
X
X
0.438
1,261
55.9%
0.419
0.681
175
1.86
0.085
X
/
0.418
1,251
54.6%
0.394
0.667
177
1.91
0.097
X
X
0.423
1.264
55.6%
0.397
0.671
183
1.88
0.093
V
0.458
1.230
56.6%
0.433
0.664
172
1.81
0.078
PF-Track-S
0.408
1.343
50.7%
0.376
166
Ens.
1.96
0.100
S2-Track-S
0.458
1.230
56.6 %
0.433
0.664
172
1.75
0.072
and donate entropy and standard deviation of uncertainty quantification, respectively.
The model achieves the best result when Srower is set 0.30
4.4.3. ABLATIONS ON DIFFERENT DECODERS.
and Supper is set to 0.70.
In Table 7, validate the generality of S2-Track by apply- ing different decoders. Specifically, employ two popular decoders: PETR (Liu et al., 2022) and DETR3D (Wang et al., 2022), which are widely adopted by query-based tracking methods. Our approach achieves excellent results with both decoders, yielding comparable performance. The results obtained with DETR3D are slightly higher, leading us to select DETR3D our default decoder.
4.4.2. ABLATIONS ON NETWORK STRIDES OF PQI MODULE.
In Table 6, we conduct an ablation study to analyze the effects of the network stride PQI module. As described in Equation (3), our PQI module incorporates two additional auxiliary tasks based the extracted feature F,. To reduce computational overhead, we directly reuse features from different layers with varying strides in the feature pyramid network, thereby alleviating the burden on the auxiliary task heads. We present the performance achieved with different strides for the image features. The experimental results reveal that the model performs optimally when using a stride of 16.
4.5. Qualitative Comparison
In Figure 4, we provide the qualitative results on nuScenes dataset. We compare our 92-Track with the previous state- of-the-art end-to-end tracker, PF-Track (Pang et al., 2023). In Figure 4 (a), PF-Track exhibits commendable tracking accuracy when the line of sight is clear at time t,. However, as the occlusion gradually intensifies, the accumulated er-

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
Table 4: Analysis of complex situations.
Table 6: Ablations on network strides of PQI module.
Visibilities
0-40%
40%-60%
60%-
Stride
AMOTAT
AMOTP4
RECALL1 MOTA
IDSI
PF-Track
38.3
38.6
39.5
0,456
1.245
56.1%
0.422
178
S2-Track
44.6 (+6.3)
44.8 (+6.2)
45.2 (+5.7)
16
0.458
1.230
56.6 %
0.433
172
32
0.446
1.268
56.2%
0.419
214
Size
0-2m
2-3.5m
>3.5m
PF-Track
22.8
20.4
19.1
Table 7: Ablations on different decoders.
S2-Track
28.3 (45.5)
25.5 (+5.1)
24.0 (44.9)
Decoder
AMOTM AMOTPL RECALL1 MOTA IDSI
Distance
>30m
20-30m
0-20m
PETR
0.452
1.246
57.1 %
0.429
196
PF-Track
7.5
38,9
64.6
DETR3D
0.458
1.230
56.6%
0.433
172
S2-Track
11.6 (+4.1)
42.8 (+3.9)
67.0 (+2.4)
The metric is AMOTA
PF-Track
Table 5: Ablations thresholds of HQD. (1) Without
thresholds, (2) without selecting by levels, (3) Our HQD.
Threshold AMOTA1 AMOTPL RECALL1 MOTA IDS.
Ouri
(1) 0.440
1.249
55.9%
0.419
181
(2)
0.445
1.239
56.5%
0.425
174
fp-s2
(3)
0.458
1.230
56.6 %
0.433
172
PF-Tmck
Lower Bound Thresholds
0.20
0.450
1.236
56.3%
0.429
179
0.30
0.458
1.230
56.6 %
0.433
172
0.40
0.448
1.241
56.4%
0.427
173
Ours
Upper Bound Thresholds
0.60
0.436
1.261
56.1%
0.419
204
0.65
0.455
1.243
57.5%
0.434
198
Figure 4: Qualitative results on the nuScenes dataset. (a) The tracking results for an occlusion scenario of two pedes- trians of consecutive frames (t, f2) (b) The tracking results on several challenging tracking scenes. Moreover, plot the attention scores of object queries, which indicate how strongly the model focuses on the target objects. A higher concentration of color represents higher attention score and stronger confidence in the corresponding object.
0.70
0.458
1.230
56.6%
0.433
172
0.75
0.444
1.253
55.9%
0.416
214
arising significantly escalates. Especially f4+12, the predicted bounding boxes for two pedestrians completely overlap. In contrast, our S2- 52-Track consistently maintains high level of tracking precision throughout the entire dura- tion of continuous tracking.
5. Conclusion
In complex scenarios of Figure (b), which are charac- terized by multiple complex factors such occlusions in crowded and spacious environments, well as the small size of vehicles and pedestrians, $2-Track achieves more precise tracking bounding boxes and successfully recog- nizes more tracked objects compared to PF-Track (Pang et al., 2023). Furthermore, the visualization of our attention scores demonstrates higher concentration on the object center, indicating that our model pays more attention to the target objects. In contrast, PF-Track (Pang et al., 2023) exhibits low attention scores for challenging samples, fail- ing to capture and track objects. The outstanding 3D MOT results of $2-Track demonstrate the effectiveness of our framework, which addresses the challenges across various complex tracking scenarios.
In this paper, improve the current end-to-end 3D MOT framework from multiple aspects. Specifically, for query initialization, we propose 2D-Prompted Query Initialization to improve object localization accuracy by incorporating predicted 2D location and depth cues. For query propaga- tion, the Uncertainty-awar Probabilistic Decoder models the uncertainty of complex driving situations using prob- abilistic attention, providing comprehensive understand- ing of predictive uncertainty. Then, for query matching, the Hierarchical Query Denoising strategy enhances train- ing robustness and convergence. Experimental results on nuScenes benchmark demonstrate that S2- Track achieves fatte-ofthhe-ee performance, i.e., 66.3% AMOTA on the test split, with significant improvement of 8.9% AMOTA.
8

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Objeet Tracking
Acknowledgments
tive spatio-temporal appearance representations, IEEE Robotics and Automation Letters, 2023.
This work is supported by Scientific Research Inno- vation Capability Support Project for Young Faculty National Natu- ral Science Foundation of China (NSFC) under Grants No.62476293, Shenzhen Science and Technology Program W,.GJH222022011314222000000
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. An image worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
DOOZYYXXNNSKYCCN.ICX..EE
Nansha Key R&D Program under Grant No.2022ZD014, and General Embodied Al Center of Sun Yat-sen University.
Feng, C., Jie, Z., Zhong, Y., Chu, X., and Ma, L.. Aedet: Azimuth-invariann multi-view 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21580-21588, 2023.
Impact Statement
This paper presents strong yet simple tracker, S2-Track, for 3D multiple object tracking. Since the tracking explored in this paper is for generic objects and does not pertain to specific human recognition, so we do not see potential privacy-related issues. At present, we primarily focus on the image representation of scenes. However, our framework can be expanded incorporate other sensors, such as fusion of LiDAR and cameras. We leave the extension of our method towards building such systems for future work. We hope that our work can inspire more future research in this field.
Fischer, T., Yang, Y.-H., Kumar, S., Sun, M., and Yu, F. Cc-3dt: Panoramic 3d object tracking via cross-camera fusion. In 6th Annual Conference on Robot Learning, 2022.
Gal, Y. and Ghahramani, Z. Dropout bayesian approx- imation: Representing model uncertainty in deep learn- ing. In international conference on machine learning, pp. 1050-1059. PMLR, 2016.
Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee. J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al. A survey of uncertainty in deep neural networks. 2021.
References
arXiv preprini arXiv:2107.03332
Bernardin, K. and Stiefelhagen, R. Evaluating multi- ple object tracking performance: the clear mot metrics. EURASIP Journal on Image and Video Processing, 2008: 1-10, 2008
Ge, Z., Liu, S., Wang, F,, Li, Z., and Sun, J. Yolox: Exceed- ing yolo series in 2021. arXiv preprint arXiv:2107.084400 2021.
Guo, H., Wang, H., and Ji, Q. Uncertainty-guider proba- bilistic transformer for complex action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10052-200611, 2022.
Blundell, C., Comebise,, J., Kavukcuoglu, K., and Wierstra, D. Weight uncertainty in neural network. In International conference on machine learning, pp. 1613-1622. PMLR, 2015.
Guo, M., Zhang, Z., Jing, I.., He, Y., Wang, K., and Fan, H. Cyclic refiner: Object-aware temporal representation learning for multi-view 3d detection and tracking. Inter- national Journal of Computer Vision, pp. 1-23, 2024.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q. Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving- In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Hu, H.-N., Yang, Y.-H., Fischer, T., Darrell, T., Yu, F., and Sun, M. Monocular quasi-dense 3d object tracking. IEEE Transactions on Pattern Analysis and Machine Intelli- 2022.
Chaabane, M., Zhang, P., Beveridge, J. R., and O' O'Hara, S. Deft: Detection embeddings for tracking, arXiv preprint arXiv:2102.02267, 2021.
gence, 45(2):1992-22
Huang, J. and Huang, G. Bevdet4d: Exploit temporal cues in multi multi-camera 3d object detection. arXiv preprint arXiv:2203. /7054, 2022.
Ding, S., Schneider, I.., Cordts, M., and Gall, Ada-track: End-to-end multi-camera 3d multi-object tracking with alternating detection and association. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Huang, J., Huang, G., Zhu, Z.. Ye, Y., and Du, D. Bevdet: pergoreeffomeece multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv: 12. 1790, 2021.
DOll, S. Hanselmann, N., Schneider, L., Schulz, R., En- zweiler, M., and Lensch, H. P. Star-track: Latent mo- tion models for end-to-end 3d object tracking with adap-
Jiang, X., Li, S., Liu, Y., Wang, S., Jia, F., Wang, T., Han, I.., and Zhang, X. Far3d: Expanding the horizon

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
for surround-view 3d object detection. arXiv preprini arXiv:2308. 2023.
Liang, T., Xie, H., Yu, K., Xia, Z.,, Lin, Z.., Wang, Y., Tang, T., Wang, B., and Tang, Z. Bevfusion: A simple and ro- bust lidar-camera fusion framework. Advances in Neural Information Processing Systems, 35: 35:10421-1044 2022.
Kingma, D. P., Salimans, T., and Welling, M. Variational dropout and the local reparameterization trick. Advances in neural information processing systems, 28, 2015.
Lin, S., Kou, Y., Li, B., Hu, W. and Gao, J. Hstrack: Bootstrap end-to-end multi-camera 3d multi-object tracking with hybrid supervision. arXiv preprint arXiv:241 2024.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp. 2980-22022 2988, 2017.
Lee, Y., Hwang, J.-w., Lee, S., Bae, Y., and Park, J. An energy and gpu-computation efficient backbone network for real-time object detection. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition workshops, pp. 0-0, 2019.
Lin, X., Pei, Z., Lin, T., Huang. L.,, and Su, Z. Sparse4d v3: Advancing end-to-end 3d detection and tracking, arXiv preprint arXiv:2311.1172 2023.
Li, F., Zhang, H., Liu, S., Guo, J., Ni, I.. M., and Zhang, L.. Dn-detr: Accelerate detr training by introducing query denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13619- 13627, 2022a.
Liu, Y., Wang, T., Zhang, X., and Sun, J. Petr: Position embedding transformation for multi-view 3d object de- tection. In European Conference on Computer Vision, pp. 531 Springer, 2022.
Li, X., Xie, T., Liu, D., Gao, J., Dai, K., Jiang, 7., Zhao, L.., and Wang, K. Poly-mot: A polyhedral framework for 3d multi-object tracking. In 2023 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp- 1991-9398.. IEEE, 2023a.
Liu, Y., Yan, J., Jia, F., Li, S., Gao, A., Wang, T., and Zhang, X. Petrv2: A unified framework for 3d perception from multi-camera images. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3262- 3272, 2023a.
Li, x., Li, P., Zhao, I.., Liu, D., Gao, J., Wu, X., Wu, Y., and Cui, D. Rocktrack: A 3d robust multi-camera- ken multi-object tracking framework. arXiv preprint arXiv:2409. 11749, 2024.
Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D. L., and Han, S. Bevfusion: Multi-task multi-sensor fusion with unified bird's s-eye view representation. In 2023 IEEE international conference robotics and au- lomation (ICRA), pp. 2774-2781. IEEE, 2023b.
Li, Y., Bao, H., Ge, Z., Yang, J., Sun, J., and Li, Z. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1486-1494, 2023b.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. arXiv preprint 1rXiv: 2017.
Ma, L. V., Hussain, M. I., Yow, K.-C,, and Jeon, M. 3d multi- object tracking employing ms-glmb filter for autonomous driving. 2024 International Conference on Control, Automation and Information Sciences (ICCAIS), pp. ? IEFE, 2024.
Li, Y., Ge, Z., Yu, G., Yang, J., Wang, Z., Shi, Y., Sun, J., and Li, Z. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1477-1485, 2023c.
Marinello, N., Proesmans, M., and Van Gool, L. Triplet- track: 3d object tracking using triplet embeddings and Istm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4500- 4510, 2022.
Li, Y., Yu, Z., Philion, J., Anandkumar, A., Fidler, S., Jia, J. and Alvarez, J. End-to-end 3d tracking with decou- pled queries. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 18300..1111110111 18302-18311, 2023d.
Pang, 7.,, Li, 7.,, and Wang, N. Simpletrack: Understanding and rethinking 3d multi-object tracking. In European Conference on Computer Vision, pp. 680-696. Springer, 2022.
Li, 7., Wang, W. H., Xie, F., Sima, C., Lu, T., Qiao, Y., and Dai. J. Bevformer: Learning bird's s-yeeviiew repre- sentation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pp. 1-18. Springer, 2022b.
Pang, Z., Li, J., Tokmakov, P., Chen, D., Zagoruyko, S., and Wang, Y.-X. Standing between past and future:
10

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Objeet Tracking
Spatio-temporal modeling for multi-camera 3d multi- object tracking. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 17928-17938, 2023.
Wang, Z., Li, Y., Guo, Y., Fang, L.,, and Wang, S. Data-uncertainty guided multi-phase learning for semi- supervised object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4568 3568-4577, 2021.
Park, J., Xu, C., Yang, S., Keutzer, K., Kitani, K., Tomizuka, M.. and Zhan, W. Time will tell: New outlooks and baseline for temporal multi-view 3d object detection. arXiv preprint arXiv:2210.0244/ 2022.
Weng. X. and Kitani, K. A baseline for 3d multi-object tracking. arXiv preprint M9Kivv9907.039000 1(2):6, 2019.
Weng, X., Wang, J., Held, D., and Kitani, K. 3d multi-object tracking: A baseline and new evaluation metrics. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 10359 10366. IEEE, 2020.
Pei, J., Wang, C., and Szarvas, G. Transformer uncertainty estimation with hierarchical stochastic attention. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 36, pp. 11147-11155, 2022.
Wojke, N., Bewley, A., and Paulus, D. Simple online and realtime tracking with a deep association metric. In 2017 IEEE international conference on image processing (ICIP), pp. 3645-3649, IEEE, 2017.
Qing, L., Wang. T., Lin, D., and Pang. J. Dort: Modeling dynamic objects in recurrent for multi-camera 3d object detection and tracking. In Conference on Robot Learning, pp. 3765-3765. PMLR, 2023.
Yang. F., Zhai, Q.. Li, X., Huang, R., Luo, A., Cheng, H., and Fan, D.-P. Uncertainty guidee transformer reasoning for camouflaged object detection. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 4146-4155, 2021.
Sadjadpour, T., Li, J., Ambrus, R., and Bohg. J. Shasta: Modeling shape and spatio-temporal affinities for 3d multi-object tracking. IEEE Robotics and Automation Letters, 2023.
Shu, C., Deng, J., Yu, F., and Liu, Y. 3dppe: 3d point positional encoding for transformer-based multi-camera 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3580 3589, 2023.
Yang, J.. Yu, E., Li, Z., Li, X., and Tao, W. Quality mat- ters: Embracing quality clues for robust 3d multi-object tracking. arXiv preprint arXiv:2208. 1000: 2022.
Yin, T., Zhou, X., and Krahenbuhl, P. Center-basel 3d object detection and tracking. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition. pp. 11784-11799 11793, 2021.
Subedar, M., Krishnan, R., Meyer, P. L., Tickoo, O., and Huang. J. Uncertainty-awant audiovisual activity recog- nition using deep bayesian variational inference. In Pro- ceedings of the IEEE/CVF international conference computer vision, pp. 6301-6310 2019.
Zeng. F., Dong, B., Zhang, Y., Wang. T., Zhang, X., and Wei, Y. Motr: End-to-end multiple-object tracking with trans- former. In European Conference on Computer Vision, pp. 659-675. Springer, 2022.
Tokmakov, P., Li, J., Burgard, W., and Gaidon, A. Learning to track with object permanence. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10860-10866) 2021.
Zhang, T., Chen, X., Wang, Y., Wang, Y., and Zhao, H. Mutr3d: A multi-camera tracking framework via 3d-to- 2d queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4537- 4546, 2022a.
Wang, L., Zhang, X., Qin, W., Li, X., Gao, J., Yang, L., Li, Z., J., J., Zhu, L., Wang, H., al. Camo-mot: Combined sopparaacc-motionn optimization for 3d multi-object track- ing with camera- -idar fusion. IEEE Transactions on In- telligent Transportation Systems, 2023a.
Zhang, Y., Sun, P., Jiang, Y., Yu, D., Weng, F. Yuan, Z., Luo, P., Liu, W. and Wang, X. Bytetrack: Multi-object tracking by associating every detection box. In Euro- pean Conference on Computer Vision, pp. 1-21. Springer, 2022b.
Wang. S., Liu, Y., Wang, T., Li, Y., and Zhang, X. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3621-3631, October 2023b.
Zhou, X., Koltun, V., and Krähenbühl, P. Tracking objects as points. In European conference on computer vision, pp. 4744900. Springer, 2020.
Wang. Y., Guizilini, V. C., Zhang, T., Wang, Y., Zhao, H., and Solomon, J. Detr3d: 3d object detection from multi- view images via 3d-to-20 2d queries. In Conference on Robot Learning, pp. 191. PMLR, 2022.
Zong. Z., Jiang, D.,, Song, G., Xue, Z., Su, J., Li, H., and Liu, Y. Temporal enhanced training of multi-view 3d object detector via historical object prediction. arXiv preprint arXiv:2304 000007,00400066, 2023.

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Objeet Tracking
Appendix
A. Additional Details
A.1. Depthnet Details
x3
Residual Block
Conv
Conv
Image Features
Depth
Figure 5: Details of the depth network in PQI.
The depth network of the proposed PQI module is composed of multiple residual blocks, and we provide an illustration in Figure 5.
A.2. Experimental Setup
1.2.1. DATASET.
We conduct experiments on the popular nuScenes benchmark (Caesar al., 2020), which is large-scale autonomous--drivng dataset for 3D detection and tracking, consisting of 700, 150, and 150 scenes for training, validation, and testing, respectively. Each frame contains one point cloud and six calibrated images from the surrounding cameras with full 360-degree field of view. It provides 3D tracking bounding boxes from 7 categories for the tracking task.
A.2.2. METRICS.
We follow the official evaluation metrics from nuScenes. For the 3D tracking task, we report Average Multi-object Tracking Accuracy (AMOTA) (Weng& Kitani, 2019), Average Multi-object Tracking Precision (AMOTP), and the modified CLEAR MOT metries (Bernardin & Stiefelhagen, 2008), e.g., MOTA, MOTP, and IDS. For detailed understanding, please refer to (Caesar et al., 2020; Bernardin & Stiefelhagen, 2008).
A.2.3. IMPLEMENTATION DETAILS.
In this paper, we assess the generalization capability of our S2-Track through experiments using different encoders, e.g.. V2-99 (Lee et al., 2019) and ViT (Dosovitskiy et al., 2020), and diffcrent decoders c.g., PETR (Liu et al., 2022) and DETR3D (Wang et al., 2022). All experiments are conducted on NVIDIA A A100-80GB GPUs. For each training sample, contains three consecutive adjacent frames each with contains six surrounding images, and we use fixed number of 500 initial queries for each sample. We adopt the Adam optimizer (Loshchilov & Hutter, 2017) for network training, with the initial learning rate setting of 0.01 and the cosine weight decay set 0.001. By default, the thresholds Stower and Supper are set to 0.3 and 0.7, and the weight coefficients À that are all set to 1.0, respectively.
Due to the limited computation resources, we follow PF-Track (Pang et al., 2023) to apply two resolution settings, full- resolution and small-resolution. For full-resolution ("-F"), we crop the origin 1600 x 900 image to 1600 x 640. For small-resolution ("-S"), we scale down the cropped image to 800 320 in a further step. We pre-train the image backbone with single-frame detection task for 12 epochs rrsolutrssolution setting) and 24 epochs (full-resolution setting) respectively, and further train the end-to-ena tracker with consecutive frames (set be 3 frames) for another 12 epochs (small-resolution) and 24 epochs (full-resolution). All the ablation studies are conducted on the small-resolution setting with V2-99 backbone.
We ulitze the 3D location to initialize queries by following steps: 1) normalize input coordinates to the [0, 2T range; 2) generate frequency bands using exponential temperature scaling; 3) compute sine/cosine components for each dimension (X, Y. Z); 4) concatenate the encoded dimensions; 5) project the concatenated features through two linear layers with ReLU activation. We will include these implementations in the revision.
12

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
A.2.4. ANALYSIS STUDY DETAILS.
The analysis studies in Tab. are conducted on the small-resolution setting with V2-99 backbone. The experiments are performed the nuScenes val set, in which we focus specific challenging conditions select clips for evaluation. We utilize the attributes of the bounding boxes provided by the nuScenes dataset, e.g., the visibility labels, and then calculate the average for each clip, finally group the results according to different ranges of the attributes. The categorization process involved the following criteria: 1. Different visibilities: the dataset is divided based on the visibility attribute of the objects. Visibility ranges are considered as 0-40%, 40-60%, and 60- 100%. 2. Different object sizes: the dataset divided into three groups based on the average object size: objects with a size greater than or equal to 3.5 meters, objects smaller than 3.5 meters and greater than or equal to 2 meters, and objects smaller than meters. dataset is split based on different distance ranges, namely 0-20 meters, 20-30 meters, and 30 meters and above. By applying these categorizing and calculations, subsets of data were selected from the clips in the validation dataset to evaluate their performance based on the specified challenging conditions.
Different object distances: the
B. Additional Discussion
B.1. PQI vs. 3DPPE
Previous 3DPPE (Shu et al., 2023) also involves depth priors in a query-based framework, it differs from S2-Track in several aspects. First, 3DPPE focuses on 3D object detection, whereas we tackle 3D MOT. Second, 3DPPE introduces 3D point positional encoding, while PQI is designed for query initialization. Moreover, we also retain randomly initialized queries to explore missing objects. We will add this discussion into the revision.
B.2. Analysis of other modules also effectively reduce the uncertainty
Our PQI module leverages learned certain priors, i.e., 2D object location and depth information, enhance the initialization of queries, thus effectively reducing the uncertainty in query initialization and resulting in more accurate object localization and tracking. The HQD strategy introduces different levels of noise to the queries and then applies a denoising process, allowing the model to encounter varying magnitudes of noise (i.e., uncertainty) during training. This effectively helps the model reduce uncertainty during query matching, leading to more stable and accurate tracking performance. Although the motivation these two modules is not uncertainty, they both help the model reduce uncertainty during query initialization and matching. Moreover, they are incorporated together with the UPD module, which aims to reduce uncertainty during query propagation.
C. Additional Results
C.1. Inference Latency
We present the inference latency measure in Table 8. Our proposed S2-Track, demonstrates greater efficiency compared to previous end-to-end methods, i.c., MUTR3D (Zhang et al., 2022a) and DQTrack (Li et al., 2023d). In comparison to the sfte-oof-hhearr PF-Track (Pang et al., 2023), our S2-Track introduces little additional latency, and yet this trade-off results in 5.0% improvement in AMOTA over PF-Track. Further efficiency enhancement in tracking is promising direction for future research.
Table 8: Inference latency. Frame Per Second (FPS) is evaluated on a single NVIDIA A100 GPU from input images to tracking results.
Method
FPS
Pr-Track-S (Pang et al., 2023)
9.2
DQTrack (Li et al., 2023d)
6.0
MUTR3D (Zhang al., 2022a)
6.0
S2-Track-S (Ours)
7.5
13

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Object Tracking
C.2. Detection Results
C.2.1. METRICS.
For the 3D detection task, we follow the official evaluation metrics from nuScenes (Caesar al., 2020), and report nuScenes Detection Score (NDS), mean Average Prediction (mAP), and five True Positive (TP) metrics including mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), mean Average Attribute Error (mAAE).
Table Results of 3D detection on nuScenes val dataset.
Method
Backbone
mAP
NDS1
mATE
mASEL
mAOE + mAVEL mAAE 4
Detection-only
PETR v2 (Liu et al., 2023a)
R101
0.421
0.524
0.681
0.267
0.357
0.377
0.186
BEVDet4D (Huang Huang, 2022)
Swin-B
0.426
0.552
0.560
0.254
0.317
0.289
0.186
Cyclic (Guo al., 2024)
R101
0.433
0.532
0.639
0.270
0.318
0.416
0.201
BEVDepth (Li et al., 2023c)
CNX-B
0.462
0.558
0.540
0.254
0.353
0.379
0.200
AeDet (Feng et al., 2023)
CNX-B
0.483
0.581
0.494
0.261
0.324
0.337
0.195
SOLOFusion (Park et al., 2022)
R101
0.483
0.582
0.503
0.264
0,381
0.246
0.207
StreamPETR (Wang et al., 2023b)
R101
0.504
0.592
0.569
0.262
0.315
0.257
0.199
HoP (Zong et al., 2023)
R101
0.454
0.558
0.565
0.265
0.327
0.337
0.194
Far3D (Jiang et al., 2023)
R101
0.510
0.594
0.551
0.258
0.372
0.238
0.195
Join Tracking and Detection
MUTR3D (Zhang et al., 2022a)
R101
0.349
0.434
DQTrack (Li et al., 2023d)
V2-99
0.410
0.503
PF-Track-F (Pang et al., 2023)
V2-99
0.399
0.390
0.727
0.268
1.722
0.887
0.211
HSTrack (Lin et al., 2024)
V2-99
0.418
0.510
Sparse4D-v3 (Lin et al., 2023)
R101
0.537
0.623
0.511
0.255
0.306
0.194
0.192
S2-Track-F (Ours)
VIT-L
0.589
0.655
0.495
0.250
0.249
0.2100
0.1883
"F" represent the full-resolution settings.
Detection on nuScenes benchmark. As our S2-Track can jointly optimize tracking and detection, we present the detection results nuScenes test set and val set in Table 10 and Table 9 respectively, which demonstrate consistent improvements of our method in the detection task. As framework design for tracking, our model achieves comparable results (62.7% mAP and 68.0% NDS in the test set) with the concurrent leading detection methods, e.g., HoP and Far3D, and outperforms previous end-to-end detection and tracking model Sparse4D-v3 by significant margin of 5.7% mAP and 2.4% NDS in the test set.
D. Additional Visualizations
D.1. Qualitative Results of PQI
We present qualitative results of our PQI module in Figure 6. Our PQI module leverages learned certain priors, i.e., the predicted 2D object location and depth information to formulate initial queries. As shown in Figure 6, the initial queries generated by our PQI module are accurately positioned within the regions of interest for the objects. This indicates that our module effectively bootstraps latent query states of the objects, leading to improved object localization and tracking results.
D.2. More Qualitative Results
We provide additional qualitative results in Figure 7. We compare our S2-Track with the previous state-of-the-art end-to-ent tracker, Tr-Track (Pang et al., 2023), on various complex scenarios. In challenging 3D MOT scenarios, characterized by multiple challenging factors such as occlusions and small target objects, S2-Track successfully predicts a greater number of tracked bounding boxes with higher localization precision compared to PF-Track (Pang al., 2023). Furthermore, the
14

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Objeet Tracking
Table 10: Results of 3D detection on nuScenes test dataset. f provided official model.
indicates the results obtained from our testing using the
Method
Backbone
mAP 1
NDS1
mATEJ
MASEJ
mAOE + mAVEJ mAAE J
Detection-only
BEVDet4D (Huang & Huang, 2022)
Swin-B
0.451
0.569
0.511
0.241
0,386
0.301
0.121
Cyclic (Guo et al., 2024)
R101
0.452
0.549
0.575
0.255
0.405
0.407
0.131
PETR v2 (Liu et al., 2023a)
V2-99
0,506
0.592
0.536
0.243
0.359
0.349
0.120
BEVDepth (Li et al., 2023c)
CNX-B
0.520
0.609
0.445
0.243
0.352
0.347
0.127
BEVSStereo (Li al., 2023b)
V2-99
0,525
0.610
0.431
0.246
0.358
0.357
0.138
SOLOFusion (Park et al., 2022)
CNX-B
0.540
0.619
0.453
0.257
0.376
0.276
0.148
AeDet (Feng al., 2023)
CNX-B
0.531
0.620
0.439
0.247
0.344
0.292
0.130
StreamPETR (Wang et al., 2023b)
VIT-L
0.620
0.676
0.470
0.241
0.258
0.236
0.134
HoP (Zong et al., 2023)
VIT-L
0.624
0.685
0.367
0.249
0.353
0.171
0.131
Far3D (Jiang et al., 2023)
VIT-L
0.635
0.687
0.432
0.237
0.278
0.227
0.130
Join Tracking and Detection
PF-Track-Fi (Pang al., 2023)
V2-99
0.397
0.387
0.688
0.262
1,800
1.079
0.165
Sparse4D-v3 (Lin et al., 2023)
V2-99
0.570
0.656
0.412
0.236
0.312
0.210
0.117
S2-Track-FF (Ours)
VIT-L
0.627
0.680
0.434
0.237
0.311
0.216
0.130
"F" represent on the full-resolution scttings.
visualization of our attention scores demonstrates a higher concentration on the center the objects, indicating that our model pays more attention the target objects. This observation also highlights the effectiveness of our proposed UPD module with probabilistic attention in modeling the uncertainty associated with the object prediction.
D.3. Video Demo
In addition to the figures, we have also attached a video demo the supplementary materials, which consists hundreds of tracking frames that provide more comprehensive evaluation of our proposed approach.
15

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Objeet Tracking
2D-Prompted Init Queries
S2-Track Results
Ground Truth Results
Figure 6: Qualitative results of PQL. The initial queries generated by our PQI module accurately locate the regions of interest for the objects, resulting in more accurate tracking results.
16

---

S2-Track: A Simple yet Strong Approach for End-to-End 3D Multi-Objeet Tracking
PF-Track
Ours
PF- Track
Ours
Figure 7: More qualitative results the nuScenes dataset. The tracking results several challenging tracking scenarios, including the small size of the target objects and the occlusions. Moreover, we plot the attention scores of object queries, which indicate how strongly the model focuses on the target objects. A higher concentration of color represents higher attention score and a stronger confidence in the corresponding object.
17


Reviews:
[{"review_id": "rIgKm6IXG4", "reviewer": "Reviewer_WzGu", "summary": "This paper proposes an end-to-end stronger yet simple 3D multi-object tracking framework named S2-Track, which decomposes the tracking pipeline into three core modules: query initialization, propagation, and matching. Experiments show the effectiveness of each module in complex scenarios, including 2D-Prompted Query Initialization (PQI), Uncertainty-aware Probabilistic Decoder (UPD), and Hierarchical Query Denoising (HQD).  The proposed framework is simple yet strong. It achieves excellent tracking performance when dealing with occlusions and small objects.\n\n## update after rebuttal\nThe authors have addressed most of my concerns, I decide to keep the positive rating.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful feedback. We especially appreciate your recognition of our simple and strong framework with effective modules and SOTA tracking performance. We responded in detail below and will add them to the revision.\n\n> Q1: More analysis of other extreme scenarios\n\nThanks for your suggestion! In Table 4, we have already analyzed different occlusion situations, i.e., different visibilities. Here, we provide additional experiments of different weather and lighting conditions, and the results show that S2-Track is robust under different lighting and weather conditions, significantly boosting the performance under challenging rainy and nighttime scenes.  The metric is AMOTA.\n\n| Method       | Day        | Night      | Sunny      | Rainy      |\n| ------------ | ---------- | ---------- | ---------- | ---------- |\n| PF-Track     | 41.3       | 12.6       | 41.8       | 36.1       |\n| **S2-Track** | 46.5(+5.2) | 19.7(+7.1) | 46.7(+4.9) | 42.6(+6.5) |\n\n> Q2: Results of category-aware AMOTA (e.g., pedestrian vs. vehicle) \n\nThanks for your valuable comment! We provide detailed category-aware AMOTA results on both the val and test set for better comparisons, as the previous methods or leaderboard reports comprehensive test set results. The results show that S2-Track achieves larger improvements in more challenging categories, e.g., pedestrian.\n\n| AMOTA                 | car               | pedestrian        | bicycle | bus  | motorcycle | trailer | truck |\n| --------------------- | ----------------- | ----------------- | ------- | ---- | ---------- | ------- | ----- |\n| Val-PF-Track-CVPR23   | 57.9              | 41.5              | -       | -    | -          | -       | 40.3  |\n| Val-**S2-Track**      | 62.0(+4.1)        | 47.0(+5.5)        | 38.6    | 55.3 | 40.6       | 32.5    | 44.6  |\n| Test-PF-Track-CVPR23  | 62.2              | 45.1              | 32.2    | 40.8 | 44.8       | 38.0    | 40.5  |\n| Test-ADA-Track-CVPR24 | 66.4              | 53.4              | 33.4    | 38.2 | 48.4       | 43.7    | 35.9  |\n| Test-**S2-Track**     | 77.4 (+15.2/11.0) | 70.1 (+25.0/16.7) | 57.6    | 65.8 | 67.5       | 64.3    | 61.0  |\n\n> Q3: Other distributions for UPD\n\nThanks for your suggestion! We conducted additional experiments to explore different distributions. The results show that other distributions did not achieve satisfactory performance, which may be attributed to the natural statistical properties following the central limit theorem, i.e., many natural phenomena (e.g., lighting variations, sensor noise) arise from the superposition of numerous small and independent effects, leading to normal distribution.\n\n| Distribution | Gaussian | Uniform | Exponential |\n| ------------ | -------- | ------- | ----------- |\n| AMOTA        | 45.8     | 22.3    | 4.7         |\n\n> Q4: Generalized to query-based  3D object detection.\n\nThanks for your suggestion! We have already presented our detection results on the nuScenes test and val sets in Table 9 and 10. As a framework designed for tracking, our model also achieves leading detection performance (62.7% mAP and 68.0% NDS on the test set), which clearly demonstrates our strong generalizability.  While integrating our powerful modules into existing SOTA detection methods could lead to further improvements, due to computational resource constraints and the scope of this tracking paper, we leave this exploration to future work.\n\n> Q5: Missed reference: 3DPPE\n\nThanks for your comment! While 3DPPE [1] also involves depth priors in a query-based framework, it differs from S2-Track in several aspects. First, 3DPPE focuses on 3D object detection, whereas we tackle 3D MOT. Second, 3DPPE introduces 3D point positional encoding, while our PQI is designed for query initialization. Moreover, we also retain randomly initialized queries to explore missing objects. We will add this discussion into the revision.\n\n[1] 3DPPE: 3D Point Positional Encoding for Transformer-based Multi-Camera 3D Object Detection\n\n> Q6: One very nearing lady is missed by the baseline PF-Track. I wonder if it is due to the difference in the detection part.\n\nThanks for your comment! In Table 7, S2-Track achieves comparable results with both detection heads, PETR (the default head of PF-Track) and DETR3D (the default head of S2-Track), indicating that the missed lady in the demo is not caused by the detection component. In fact, if you carefully review the video, you will notice that the nearby lady is detected in the first frame. However, due to interference from other vehicles and pedestrians, her bounding box is lost in subsequent frames. This is precisely where our carefully designed modules enhance performance, enabling stable and robust tracking in challenging scenarios.\n\nThanks again for your thoughtful feedback. We believe that your suggestions, along with our revision, have greatly enhanced the persuasiveness and completeness of our work. We hope our rebuttal can address your concerns.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes. It shows that the proposed method works excellent in small object tracking.", "theoretical_claims": "No theoretical claim in this paper.", "experimental_designs_or_analyses": "Yes. \nFor the performance comparison:\nOn nuScenes dataset, S2-Track achieves state-of-the-art performance with an AMOTA of 66.3%, outperforming previous methods by 8.9%.", "relation_to_broader_scientific_literature": "NA.", "essential_references_not_discussed": "One of the 3D object detection methods that also follows a query-based paradigm and exploits depth net is missed: 3DPPE, published in ICCV 2023. The authors are supposed to discuss it.", "other_strengths_and_weaknesses": "Strengths:\n\n1.\tThe three modules (PQI, UPD, HQD) address the tasks in different stages of the query tracking lifecycle. These designs improve the tracking performance in complex scenarios. Meanwhile, the design of UPD is novel, as it integrates uncertainty perception through a probabilistic attention mechanism. This allows the model to maintain stable predictions even in challenging scenarios such as occlusion, small targets, and distant objects.\n\n2.\tThe framework is both simple and strong, avoiding complex designs such as multi-stage tracking pipelines. It also achieves outperforming performance with the refined transformer query mechanisms.\n\n3.\tExtensive ablation studies and visualization results show the effectiveness of each module. The results (66.3% AMOTA) on the nuScenes test set achieve SOTA tracking performance in query-based methods.\n\nWeaknesses:\n\n1.\tThe paper lacks quantitative analysis of performance degradation in extreme scenarios (e.g., heavy occlusion, low-light nighttime conditions). The paper also lacks the results of category-aware AMOTA (e.g., pedestrian vs. vehicle) on the nuScenes validation set.\n\n2.\tIs the assumption of a Gaussian distribution reasonable, or are there other distributions that might be more suitable for UPD?", "other_comments_or_suggestions": "I suppose that the proposed method can be also generalized to some query-based multi-modal 3D object detection methods. And a few of them also report the tracking results. The authors are suggested to add an experiment to show its generalizability.", "questions_for_authors": "Please refer to the above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4DaozVyXwX", "reviewer": "Reviewer_A6kt", "summary": "The paper aim to improve the existing end-to-end 3D multi-object tracking framework. Specifically, the authors propose 2D-prompted query initialization, uncertainty-aware probabilistic decoder, and hierarchical query denoising. Experimental results on nuScenes benchmark show the effectiveness of the proposed framework.\n\n## update after rebuttal\nPlease see the rebuttal comment below.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful feedback. We especially appreciate your recognition of our effective framework with newly-designed modules and well-written paper. We responded in detail below and will add them to the revision.\n\n> Q1: The proposed method is only evaluated on nuScenes dataset\n\nThanks for your suggestion! First, since the nuScenes dataset provides comprehensive detection and tracking tasks, along with evaluation leaderboards, most previous detection and tracking methods (e.g., DQTrack [ICCV23], PF-Track [CVPR23], and ADA-Track [CVPR24]) have been only evaluated on nuScenes. Therefore, following them, we also perform a fair comparison on this dataset. Second, we agree with the reviewer’s point that methods evaluated on only one dataset have not demonstrated robustness across multiple datasets. To address this concern, we present additional evaluations on an in-house autonomous driving dataset, which is collected from real-world scenarios. The results show that our method effectively tracks objects in challenging environments, demonstrating the generalization and robustness of our S2-Track. Additionally, we have provided a video showcasing the real-world results on an anonymous GitHub Page: https://anonymous-github-8ab1cv.github.io/s2-track/.\n\n| Method       | MOTA  | MOTP  | RECALL |\n| ------------ | ----- | ----- | ------ |\n| PF-Track     | 0.549 | 0.476 | 54.8%  |\n| **S2-Track** | 0.712 | 0.334 | 77.3%  |\n\n> Q2: The general framework is still based on existing methods. The insights to this task and the community is limited.\n\nThank you for your comments! However, we respectfully disagree with the \"limited insights\". While S2-Track employs the query-based end-to-end framework as previous works, as mentioned in the Introduction, current end-to-end trackers are still in the early stages of development and are unable to effectively handle the various complex driving scenarios and achieve satisfactory tracking results. Therefore, S2-Track comprehensively enhances the existing end-to-end 3D MOT framework, delivering impressive robust and accurate tracking results, and demonstrating the potential of end-to-end frameworks.\n\nAs **Reviewer 4TdZ** acknowledged, \"The paper builds on recent advances in end-to-end query-based trackers and ... to address limitations in 3D MOT. **Therefore, I believe one of the core contributions of this paper is that the performance of its proposed end-to-end multi-object tracking method surpasses both end-to-end and non-end-to-end methods. I think end-to-end approaches represent the future trend for both autonomous driving and multi-object tracking, and this paper validates the potential of end-to-end methods.**\"\n\n> Q3: Analysis of other modules also effectively reduce the uncertainty. \n\nThanks for your comment! Our **PQI module** leverages learned certain priors, i.e., 2D object location and depth information, to enhance the initialization of queries, thus effectively **reducing the uncertainty in query initialization** and resulting in more accurate object localization and tracking. The **HQD strategy** introduces different levels of noise to the queries and then applies a denoising process, allowing the model to encounter varying magnitudes of noise (i.e., uncertainty) during training. This effectively helps the model **reduce uncertainty during query matching**, leading to more stable and accurate tracking performance. Although the motivation of these two modules is not uncertainty, they both help the model reduce uncertainty during query initialization and matching. Moreover, they are incorporated together with the UPD module, which aims to **reduce uncertainty during query propagation**. We will incorporate this discussion into the revision.\n\n> Q4: The parameters of HQD require searching.\n\nThanks for your comment! The HQD module indeed has two thresholds that need to be set. As shown in Tab 5, the impact of these two thresholds on the results remains relatively stable within a certain optimal range, meaning that we do not need to perform an extremely fine-grained search for their values. Simply identifying the approximate optimal range is sufficient. Furthermore, on our in-house autonomous driving dataset, we used the same values as those in nuScenes and also achieved satisfactory results. While a more detailed search might yield further improvements, the current settings already provide stable and satisfactory performance gains in most cases.\n\n> Q5: Add the GT in Fig 4.\n\nThanks for your suggestion!  We have carefully included the ground truth as separate images in Fig4 in the revision, making it easier for readers to understand.\n\nThanks again for your thoughtful feedback. We believe that your suggestions, along with our revision, have greatly enhanced the persuasiveness and completeness of our work.  We hope our rebuttal can address your concerns and sincerely appreciate that you can rejudge our efforts and potentially update your scores.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "The proposed methods and evaluations are reasonable.", "theoretical_claims": "No theoretical claims and proofs involved.", "experimental_designs_or_analyses": "The proposed method is only evaluated on nuScenes dataset, which may not be enough to demonstrate the effectiveness of the proposed method on the MOT task.", "relation_to_broader_scientific_literature": "The paper further improve the existing multi-object tracking framework by incorporating three novel modules.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strength:\n1. The paper is well-written, with clear structure and illustrations.\n\nWeakness:\n1. The proposed method is only evaluated on one dataset. Not enough to show the robustness in the multi-object tracking task.\n2. The reviewer acknowledge the improvement brought by the newly-designed modules, but the general framework is still based on existing methods. The insights to this task and the community is limited.", "other_comments_or_suggestions": "The qualitative results in Figure 4(a) is hard to see. The reader could not easily tell the ground truth. Maybe show the ground truth as separate image, or a pure image without any annotation.", "questions_for_authors": "1. In the analysis of uncertainty (line 323-325-left column), the authors do not provide an analysis on the observation that other modules also effectively reduce the uncertainty. This weakens the motivation of the designed UPD module, as other modules could also achieve that. Do the authors have insights of why it is the case?\n2. In section 4.4.1, the authors conduct ablation study on upper bound and lower bound thresholds. Does it require searching on parameter pairs? It is very time-consuming. Also, these parameters may need to be re-search for different data distributions.\n3. The major concerns of the reviewer are listed in Other Strengths And Weaknesses section. Although the limitations might not be easily addressed during rebuttal, the reviewer would appreciate any explanation or interpretation on these limitations.\n\nThe reviewer may adjust the final rating after rebuttal based on the clarifications from the authors.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bF3xsDRros", "reviewer": "Reviewer_TUre", "summary": "This paper presents a new method called S2-Track for 3D multiple object tracking (MOT), an essential component for the perception of autonomous driving systems. Existing methods adopt end-to-end query-based trackers to simultaneously detect and track objects, but they fail to track objects in complex scenarios like occlusions and the small size of target object. To address these issues, the authors first summarize current end-to-end 3D MOT framework by decomposing it into three parts and propose corresponding improvements to every part. Experiments on the nuScenes dataset show that the method achieves 0.663 AMOTA on test split, surpassing the previous best end-to-end solution by 8.9%. Overall, S2-Track decomposes current end-to-end 3D MOT framework into three parts and proposes corresponding improvements to every part, which improves AMOTA by 8.9% on test split of the nuScenes dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful feedback. We especially appreciate your recognition of our three useful improvements and superior tracking performance. We responded in detail below and will add them to the revision.\n\n> Q1: How to initialize object queries with 3D location.\n\nThanks for your comment! We ulitze the 3D location to initialize queries by following steps: 1) normalize input coordinates to the [0, 2π] range; 2) generate frequency bands using exponential temperature scaling; 3) compute sine/cosine components for each dimension (X,Y,Z); 4) concatenate the encoded dimensions; 5) project the concatenated features through two linear layers with ReLU activation.  We will include these implementations in the revision.\n\n> Q2: More explanations of core ideas in HQD. \n\nThanks for your suggestion! In complex 3D MOT scenarios, challenges such as occlusions and varying object sizes can hinder the learning and convergence of query-based methods. The slow convergence and suboptimal results from the instability of bipartite graph matching. To this end, we perturb GT bounding boxes with noises into the decoder and train the model to reconstruct the original boxes, which effectively reduces graph matching difficulty and leads to faster convergence. Moreover, we define hierarchical challenging levels for the perturbed queries to enhance the model’s ability to handle diverse driving scenarios. We have included these explanations in the revision.\n\n> Q3: How do you come up with these improvements? Are there connections among these ideas? \n\nThanks for your comment! As mentioned in the Introduction L71–101, with the goal of enhancing existing end-to-end tracker in complex driving environments, we first decompose current query-based framework into three constituent parts: query initialization, propagation, and matching (**Fig. 1(b)**). Then we propose corresponding improvements for each part: PQI, UPD, and HQD. These modules are connected by their shared foundation—the query-based framework, with all improvements targeting challenges posed by complex environments. As **Reviewer WzGu** acknowledged,  \"The three modules (PQI, UPD, HQD) address the tasks **in different stages of the query tracking lifecycle.**\"\n\n> Q4: all these improvements are just other papers’ ideas\n\nThanks for your comments! We, with respect, did not agree. Current end-to-end trackers are still in the early stages and struggle with complex driving scenarios. In response, S2Track comprehensively improves existing framework. For **PQI module**, we leverage predicted 2D locations and depth information to enhance the queries initialization. While previous works in detection have explored the use of depth information, none have leveraged it for query initialization. For **UPD module**, the uncertainty issue has never been explored in the 3D MOT, let alone propose an Uncertainty-aware Probabilistic Decoder for tracking. For **HQD module**, although it draws inspiration from DN-DETR, we have improved it by introducing Hierarchical Query Denoising. As demonstrated in the ablations (Tab 5), our improvements outperform the original DN-DETR. **None of these proposed modules are just other papers’ ideas.** Moreover, S2Track, is not a simple combination of these ideas; it delivers impressive tracking results, showcasing the potential of end-to-end framework, which are acknowledged by all other reviewers:\n\n- **Reviewer WzGu**: \"The design of UPD is **novel**... The framework is both simple and **strong**, .... It also achieves outperforming performance with the **refined transformer query mechanisms**.\"\n- **Reviewer WzGu**: \"... incorporating **three novel modules**,  ... brought by the **newly-designed modules**.\"\n- **Reviewer 4TdZ**: \" (PQI&UPD) **represent innovative improvements**, (PQI) **sounds intresting**, (HQD) **is a novel contribution .. providing a new solution**, (framework) **the core contributions...validates the potential of end-to-end methods**.\"\n\nFinally, we try to understand the reviewer's perspective. However, without S2Track decomposing the current framework into constituent parts, and proposing targeted improvements, it would be difficult for the community to grasp current limitations clearly. **If these problems were so easily solvable in tracking, how could S2Track achieve such a significant improvement (+8.9% AMOTA) over previous SOTA?**   As **Reviewer 4TdZ** stated: \"Effectively enhances the performance of the end-to-end multi-object tracking paradigm, **which sounds quite impressive.**\" We sincerely appreciate that the reviewer can recognize the substantial efforts behind our simple and strong framework.\n\nThanks again for your thoughtful feedback and time. We believe that your suggestions, along with our revision, have greatly enhanced the persuasiveness and completeness of our work. We hope our rebuttal can address your concerns.", "claims_and_evidence": "1.\tAs mentioned in the Queries initialization part, each query consists of a feature vector and a 3D location. I find that it is not explained clear enough about how to initialize object queries with just 3D location in the 2D-Prompted Query Initialization section. Is the feature vector  initialized randomly?\n2.\tI find Hierarchical Query Denoising section hard to follow. Without referring to the DN-DETR paper, it may be difficult to understand. Maybe more explanation would help and let the readers understand the core ideas.", "methods_and_evaluation_criteria": "I think the proposed three improvements are indeed useful for the 3D multiple object tracking task.\nBut I think all these improvements are just other papers’ ideas and the authors just simply combine these ideas together. I can’t figure out the connections between these improvements and think they are just separate tricks to improve the performance.", "theoretical_claims": "I think most technical concepts are explained with appropriate detail and context except the Hierarchical Query Denoising section which I find it hard to understand.", "experimental_designs_or_analyses": "This paper follows the experimental setup from previous works.\nI believe the thorough ablation studies revealing the specific contributions of each proposed improvement and the overall superior performance compared to the recent state-of-the-art methods demonstrates its efficiency.", "relation_to_broader_scientific_literature": "Prior work has shown that using depth information can significantly improve the localization of objects in 3D space. The proposed 2D-prompted query initialization leverages this by using predicted 2D object locations and depth information to guide the object detection process more effectively, thereby addressing the challenge of correctly initializing queries, which is a known limitation of previous transformer-based object detection methods.\nBy using a probabilistic decoder that models and captures uncertainty, the approach is aligned with the idea that complex environments and real-world data require models that can quantify and deal with uncertainty in predictions. This helps improve robustness and performance, especially in tasks where the true object locations or classes are not easily discernible due to noise or occlusions.", "essential_references_not_discussed": "There are no essential references that appear to be missing from the paper.", "other_strengths_and_weaknesses": "The paper presents three efficient improvements over existing methods but I think these proposed methods have already been broadly implemented in other computer vision fields.", "other_comments_or_suggestions": "None", "questions_for_authors": "1.\tHow do you come up with these improvements? Are there some connections among these ideas?\n2.\tIn Hierarchical Query Denoising section, there are few explanations. Can you explain the core ideas there more clearly?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NROkCEWBTp", "reviewer": "Reviewer_4TdZ", "summary": "This paper proposes a novel end-to-end 3D multi-object tracking method, aimed at addressing complex scenarios in autonomous driving perception, such as occlusions and small object tracking. The authors decompose the existing end-to-end 3D MOT framework into three core components: query initialization, query propagation, and query matching, and introduce corresponding improvements for each part. Experimental results demonstrate that the proposed method achieves state-of-the-art performance on the several datasets. Although the optimizations proposed by the authors in the three parts are combinations of existing techniques, they still effectively enhance the performance of the end-to-end multi-object tracking paradigm, which sounds quite impressive.\n\n## Update after rebuttal\nThank you for the authors' response, which has addressed most of my concerns. Although the authors have adopted some existing techniques, the performance improvement in their end-to-end tracking algorithm remains commendable in my assessment. Particularly noteworthy is its successful application in real-world factory settings. Additionally, I hope the code could be made open-access to advance research in end-to-end MOT field. I keep my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and insightful feedback. We especially appreciate your recognition of our well-designed modules with innovative improvements, and found our impressive performance validates the potential of end-to-end methods. We responded in detail below and will add them to the revision.\n\n> Q1: Discuss the computational complexity, in order to more clearly highlight the advantages and disadvantages of their method.\n\nThanks for your valuable suggestion! We provide the additional computational complexity of the proposed method. S2-Track only adds about 7.0% parameters, 2.4% FLOPs over PF-Track, while this trade-off results in a 12.3% improvement in AMOTA. Further improvements in tracking efficiency are a promising direction for future research. To highlight the disadvantages of our method, we have added a \"Limitations and Future Work\" section in the revision, hoping to inspire further exploration in this field.\n\n| Method   | FLOPs | Parameters |\n| -------- | ----- | ---------- |\n| PF-Track | 534G  | 91.8 M     |\n| S2-Track | 547G  | 98.3 M     |\n\n> Q2: Specify whether other methods use full resolution or reduced resolution\n\nThank you for your valuable suggestion! The numbers in our tables are directly taken from the corresponding methods' papers or the official nuScenes leaderboard, and we report the highest values for each method. Based on your suggestion, we have added a column to the tables in the revision to include the corresponding resolutions, allowing readers to make a more comprehensive assessment. Below, we provide some of the resolution settings.\n\n| S2-Track-F | PF-Track-F | S2-Track-S | PF-Track-S | ADA-Track  | Sparse4Dv3 | DQTack     |\n| ---------- | ---------- | ---------- | ---------- | ---------- | ---------- | ---------- |\n| 1600 × 640 | 1600 × 640 | 800 x 320  | 800 x 320  | 1600 × 900 | 1408 × 512 | 1408 × 512 |\n\n> Q3: If deployed on an in-vehicle GPU, would it meet the requirements for real-time inference? If not, how far is this end-to-end multi-object tracking approach from practical application\n\nThank you for your comment! We have successfully implemented our S2-Track on the real-world autonomous vehicle equipped with the NVIDIA Drive AGX Orin platform. By incorporating various engineering optimizations, such as TensorRT quantization, we have achieved real-time performance exceeding 20 Hz on the real-world vehicle. Additionally, we have provided a video showcasing the real-world results on an anonymous GitHub Page: https://anonymous-github-8ab1cv.github.io/s2-track/. However, to achieve large-scale production and deployment for high-level autonomous driving (e.g., L3/L4), further improvements in efficiency or platform (e.g., NVIDIA Drive AGX Thor) will be required.\n\nThanks again for your thoughtful feedback. We believe that your suggestions, along with our revision, have greatly enhanced the persuasiveness and completeness of our work. We hope our rebuttal can address your concerns.", "claims_and_evidence": "Yes, the claims made in the submission are well-supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria are well-suited for the problem of 3D multi-object tracking in autonomous driving.", "theoretical_claims": "Yes, i have checked the theoretical Claims of the proposed method.", "experimental_designs_or_analyses": "Yes, i have checked the experimental designs and analyses.", "relation_to_broader_scientific_literature": "The paper builds on recent advances in end-to-end query-based trackers and leverages ideas from 2D-to-3D perception and probabilistic modeling to address limitations in 3D MOT. Therefore, i believe one of the core contributions of this paper is that the performance of its proposed end-to-end multi-object tracking method surpasses both end-to-end and non-end-to-end methods. I think end-to-end approaches represent the future trend for both autonomous driving and multi-object tracking, and this paper validates the potential of end-to-end methods.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "Strengths: \n\n1.The proposed 2D-Prompted Query Initialization and Uncertainty-aware Probabilistic Decoder represent innovative improvements to existing end-to-end multi-object tracking methods. The integration of 2D information with 3D localization for Query Initialization sounds intresting.\n\n2.The Hierarchical Query Denoising strategy is a novel contribution that addresses noise issues in training, providing a new solution for enhancing the robustness of end-to-end frameworks. Although the core idea originates from DN-DETR.\n\n3. Achieving state-of-the-art (SOTA) performance is highly significant for end-to-end multi-object tracking.\n\nWeaknesses:\n\n1.I believe the authors should discuss the computational complexity (not only inference speed) of the proposed method in comparison to previous approaches, in order to more clearly highlight the advantages and disadvantages of their method.\n\n2.The authors should specify whether other methods use full resolution or reduced resolution, in order to make the comparison more fair.", "other_comments_or_suggestions": "See strengths And weaknesses", "questions_for_authors": "I noticed that the authors mentioned the inference speed of their proposed method is only 7.5 FPS on an NVIDIA A100. If deployed on an in-vehicle GPU, would it meet the requirements for real-time inference? If not, how far is this end-to-end multi-object tracking approach from practical application?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}]

Author rebuttal:

Meta-review: The paper addresses the challenges faced by current end-to-end 3D MOT frameworks in complex scenarios such as occlusions and small target sizes. The authors decompose the framework into three core components—query initialization, propagation, and matching—and introduce targeted improvements for each. Their solution, S2-Track, leverages 2D information for query initialization, employs probabilistic decoders for propagation, and uses hierarchical denoising for matching. This approach not only enhances performance but also maintains simplicity. The results demonstrate significant improvement on the nuScenes benchmark. The authors have effectively addressed the reviewers' concerns, leading all reviewers to agree on a "weakly accept" rating for the paper. ACs concur with this positive assessment and recommend acceptance of the paper.
Decision: Accept (poster)