CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion
Authors: Tobias Fischer
Yung-Hsu Yang
Suryansh Kumar
Min Sun
Fisher Yu
Venue: CoRL 2022 Poster (2022)
Keywords: Autonomous driving
3D multi-object tracking
cross-camera fusion

Abstract: To track the 3D locations and trajectories of the other traffic participants at any given time, modern autonomous vehicles are equipped with multiple cameras that cover the vehicle's full surroundings. Yet, camera-based 3D object tracking methods prioritize optimizing the single-camera setup and resort to post-hoc fusion in a multi-camera setup. In this paper, we propose a method for panoramic 3D object tracking, called CC-3DT, that associates and models object trajectories both temporally and across views, and improves the overall tracking consistency. In particular, our method fuses 3D detections from multiple cameras before association, reducing identity switches significantly and improving motion modeling. Our experiments on large-scale driving datasets show that fusion before association leads to a large margin of improvement over post-hoc fusion. We set a new state-of-the-art with 12.6% improvement in average multi-object tracking accuracy (AMOTA) among all camera-based methods on the competitive NuScenes 3D tracking benchmark, outperforming previously published methods by 6.5% in AMOTA with the same 3D detector. 

CC-3DT: Panoramic 3D Object Tracking via Cross-Camera Fusion
Tobias Fischer
Yung-Hsu Yang National Tsing Hua University royyangggapp nthu. edu. tw
ETH Zürich
tobias fiwcher@viieerini...mmmmmmm. et.eth.ch
Suryansh Kumar
Min Sun
Fisher Yu
ETH Zürich
National Tsing Hua University
ETH Zürich 1@yf 1o
sukumardvision. ethz.ch
sunminêee.c atthueed..e
Abstract: To track the 3D locations and trajectories of the other traffic partici- pants at any given time, modern autonomous vehicles are equipped with multiple cameras that cover the vehicle's full surroundings. Yet, camera-based 3D ob- ject tracking methods prioritize optimizing the single-camera setup and resort to post-hoc fusion multi-camers setup. this paper, we propose a method for panoramic 3D object tracking, called CC-3DT, that associates and models object trajectories both temporally and across views, and improves the overall tracking consistency. In particular, our method fuses 3D detections from multiple cameras before association, reducing identity switches significantly and improving motion modeling. Our experiments large-scale driving datasets show that fusion before association leads to large margin of improvement over post-hoc fusion. We set a new state-of-the-arn with 12.6% improvement in average multi-object tracking ac- curacy (AMOTA) among all camera-based methods on the competitive NuScenes 3D tracking benchmark, outperforming previously published methods by 6.5% in AMOTA with the same 3D detector. Project page: vis.xyz/pub/cc-333
Keywords:
Autonomous driving, 3D multi-ohject tracking, Cross-camera fusion
Introduction
Locating and tracking dynamic objects reliably in 3D is vital to safe autonomous driving systems. Advanced techniques with complex pipelines generally use multiple 3D sensor modalities ranging from stereo cameras to LiDAR and radar. Conversely, high-quality surround-view camera system provides low-cost alternative expensive 3D active sensors to capture vehicle's 's surroundings with a panoramic view. Many modern consumer vehicles are equipped with such multi-camera systems. Yet, it is still challenging effectively utilize temporal and cross-camera association of moving objects in 3D. Such an approach can help to track objects in the long term as cameras can locate them from different viewpoints.
Existing casera-based 3D object tracking methods are designed for single-camera setups [1, 2, 3]. Since those camera-based based approaches follow the difference between them lies in the association of detections over time [5, 6, 7, 8]. For instance, Zhou al. [2] relies on 2D motion associate objects over time, while Chaabane al. [3] utilizes object appearance. In contrast, Hu et al. [1] combines appearance and 3D motion information. Even though these recent methods perform well on the temporal association of object tracks, they fail to associate object tracks in 3D across multiple cameras installed on the vehicle.
paradigm [4], the major
Therefore, we propose effective cross-camera fusion method to the multi-camera 3D tracking problem. We merge 3D detections and their appearance information from multiple cameras into single 3DD tracking component that performs both temporal and cross-camera association. Given
Equal contribution
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Time
t+1
3D Estimation
Cross-Camera Merging
Cross-Camera Association
Motion
Figure I: Method overview. We first perform 3D object detection and appearance feature extraction for each camera view independently. Then, we lift the camera-space detections into world-space and merge them using mon-maximum suppression [11] in 3D. Next, we associate the detections with existing tracks across all cameras. Finally, we refine the detections given the trajectories of the associated tracks across cameras.
the cross-camera association, leverage cross-camera motion model to improve the quality of single-frame 3D detections even when an object has just appeared in particular camera's field of view (FOV), leading to highly accurate tracking with much smoother trajectory of objects in 3D. We call the resulting panoramic 3D object tracking method CC-3DT. The flexible model can work with various 3D object detection methods.
We test CC-3DT on multiple challenging datasets, namely NuScenes [9] and Waymo Open [10]. CC-3DT can associate objects both temporally and across cameras, leading 17% relative im- provement over the single-camera baseline on NuScenes. CC-3DT can exploit the synergy between cross-camera association, Icading to longer cross-camera trajectories, and improved object motion modeling, leading to further improved object association. Consequently, we observe significant improvements in both AMOTP and AMOTA scores. Leveraging the cross-camera information not only provides smoother object trajectories but also offers a new state-of-the-an for camera- based 3D object tracking on the competitive NuScenes [9] benchmark, outperforming existing camera-based methods by a large margin.
2 Related Work
Camera-based 3D detection. Driven by the success of convolutional neural networks (CNNs) in
object detection [11, 12, 13, 14, 15], recent works on camera-based 3D object detection have been using CNNs to regress the 3D parameters of objects from an image [16, 17, 18, 19, 20, 21]. In contrast, another line of works [22, 23, 24] lifts the image to a 3D point cloud using single im- age depth prediction and detects objects in the 3D point cloud, similar LiDAR-based 3D object detectors [25, 26, 27]. Lately, transformer networks have been explored for end-to-end object de- tection [28]. Inspired by this, Wang et al. [29] propose to use transformer network for 3D object detection from multiple cameras. Our work extends beyond single image 3D detection by perform- ing temporal refinement of the 3D bounding box parameters, improving upon the initial detection result.
2D multi-object tracking. Recent works on MOT follow the tracking bargg--by-aa deteccton paradigm [4], where per-frame object detections are associated using different cues, such as motion [30, 5, 31, 32, 33, 2], visual appearance [34, 35, 6, 8, 36, 3, 37, 38] or 3D motion [39, 40, 41, 42, 43]. Recently, researchers have explored the use of transformers in MOT [44, 45, 46). These works take inspiration from Carion et al. [28] and use the query-based detection process for temporal query propagation to track objects across time with a transformer. addition, there broad range of literature on multi-camera MOT [47, 48, 49, 50, 51]. Those works focus on scenarios with highly

---

overlapping, stationary surveillance cameras in e.g. smart city applications [52]. Their aim is to overeome occlusion problems in monocular MOT systems. On the other hand, our work focuses on multi-camera tracking in the context of autonomous driving, where the cameras usually have little overlap and are mounted on a moving platform.
Camera-based 3D multi-object tracking. In the realm of 3D MOT, there are many methods that rely on LiDAR sensors since they provide both accurate 3D localization and 360-degree coverage of the environment [39, 53, 54, 55, 56, 57J. On the contrary, camera-based 3D MOT has been approached by only handful of works [1, 3]. Those methods are conceptually similar to 2D MOT systems, employing tracking- -aougg-b-der be-detcttoon using motion or appearance cues for association. While sharing the properties of 2D MOT systems. the camera-based approaches cannot benefit from the 360-degree coverage of the LiDAR-based 3D MOT methods. Nevertheless, none of the meth- ods above consider cross-camera association explicitly. Concurrent our work, Zhang et al. [58] propose MUTR3D, method that combines Wang et al. [29] with the ideas in [44, 45, 46) into transformer- based multi-camera 3D tracking method. However, our work differs significantly from MUTR3D in that we do not rely a transformer for joint detection and association, but fuse the detections from multiple cameras into a single tracking component. Moreover, our method outper- forms MUTR3D significantly in experimental evaluation, even with weaker 3D detector.
Method
We first introduce our panoramic 3D tracking pipeline inspired by Hu et al. [1, 59]. Afterward, we detail our cross-camera association and motion modeling. Our method is illustrated in Figure 1.
3.1
Overview
Assume we are given set of input images I, from cameras M and the corresponding camera locations in world-frame & in sequence of time steps T. We aim to find the set of tracks T
[1, .N] consists of a set of states (Ti, ...). depending on the time steps that the object is visible in. Each state Tf (f.') b, ce²) at a specific time bounding box by" E IR? and a confidence value of E 1]. The 3D bounding box is defined as by" [x. y, 2, 0,1, w. h) where (I, z, 8) define the 3D center of the object and its orientation around the vertical axis in 3D world-frame, and (1, w, h) define the object dimensions.
(T',...,7*) 7 corresponding to the objects in the scene. Each track
consists of an appearance embedding fü" E R226 3D
3D estimation and merging. We use 3D estimation network to extract detections D from an input image Irm of camera m at time t. We lift all detections D.' from all cameras m M to world space with 5Thh and merge them with non non-maximum suppression (NMS) in 3D, obtaining multi-camera detections D.. The detections E D; share the same state space with tracks T, i.e. de (I,), ct).
Data association. We associate the detections D, with active tracks T, time by greedy assign- ment given affinity matrix A D,) IR || that combines appearance. location, and velocity correlation. We compute the appearance correlation based on the appearance embedding similarity in [37, 38]
exp(f fT) adeap (Te, de) EEED exp(f ff)
exp(f f7)
Erer exp(f. F f
The location correlation is based on the absolute difference between the predicted and observed 3D bounding box parameters as
bj
by
aio (Te: d4) exp(
(2)
where r is constant scalar. The motion correlation considers the absolute difference between the predicted motion Vr, Pr, Pr, and observed pseudo motion Va, Pr. Pd, and the centroid distance Pr. - Pa. as
Anotion (Tx, de) = Weos Acentroid + (1 - Wean) Apseudo
PPP Pa.
apsando exp( -VV,, -Vael) Woon = L(Vr,, Va.).
(3)
where, acantroid = exp(
r
r
3

---

Weos is defined as cosine similarity of two motion vectors to favor centroid distance if the motion vectors are in the same hemisphere and pseudo motion otherwise, so that we do not match a pair with conflicting motions. Note that we transform object center P into world-frame using & to account for ego-motion. Given the individual affinities, (Tr, Dr) is computed as
(Ti, Dr) = Wldeop-Adeeg (T. D:) + (1 wasep) W Apoiion (Te, D:) A1ee (Te, Dz),
(4)
where Wdeap is a constant that weighs appearance and a hybrid of motion and location correlation.
Motion model. We employ two LSTM networks Ppred and & updaate to handle the extrapolation and update of trajectories given track histories and new observations, respectively. Before data associa- tion, we use pred to propagate the previous object state b one time step into the future to com- pute the location and motion correlation. We input the previous - 1 5 object velocities v -i-na-l and the previous LSTM hidden state h into the network, where an object velocity defined as "V h )
vi by- b The network outputs the predicted velocity V7 .. After data association, we use Pupdata to estimate a velocity refinement given the predicted velocity Vi. observed velocity v²T, confidence of the associated detection of and previous hidden state hp-1 The velocity refinement v,, h, h. Pupdara ([V v² cf], h hediate) in turn refines the object state by = by + v. Note that for both networks, we first use a linear projection to encode each input into a 64-dimensional vector which omit for ease of notation.
Track management. In order to handle newly appearing, disappearing, and reappearing objects during inference, we keep an online track memory where each track T E T is assigned one of three possible states {active, inactive, dead]. At each time step t, all tracks currently marked as active or inactive are used calculate their affinity with the current detections Du. If detection di E Du matches with track T E T, dz, it will be assigned to т. If a track T has no matching detection, it will transition into state inactive. If T did not have a matching detection for L time steps, transitions into state dead and will no longer be considered. Note that we propagate the position of inactive tracks forward time with Ppred properly match tracks in cases of occlusion. Each detection without matching track spawns a new track in active state.
3.2 Cross-camera association
Hu et al. [1] employs separate tracker for each camera m E M. particular, the data association of each camera happens independently, as well as the track management and motion modeling, Since each camera has only a limited FOV, there are many disappearing and re-appearing tracks, making the association prone to error and the trajectories less stable. In addition, object identity will not be preserved across camera views. We explore how a panoramic multi-camera tracker can overcome those limitations with a few simple modifications to the tracking pipeline.
First, we investigate how performance is affected if we use NMS in 3D to naively combine the isting tracks from single-camera trackers. We call this approach detect + track - merge. Next, we introduce the detect + merge - track paradigm, i.e. instead of aggregating the tracking result of camera-dependent trackers, we first aggregate the detection results from all cameras m E M and subsequently apply global data association, motion modeling, and track management. The ad- vantages of this are threefold. (i) In the regions where the cameras have an overlapping FOV, the multi-camers tracker has access to more detection candidates, meaning that an object that is highly truncated in one camera can be better recognized in neighboring camera. (ii) Cross-camera camera data association results in long trajectories that span across multiple cameras, which allows for a richer track history and thus better motion modeling. (iii) Our data association scheme leverages appear- ance, location and velocity information to follow tracks more effectively across cameras compared to 3D NMS based merging of single-camera tracking results.
3.3 Cross-camera motion modeling
crucial advantage of our panoramic tracker is that we can exploit longer trajectories across camera views for motion modeling. This leads to an improved multi-frame refinement via T. Pupdata and more accurate location prediction via ppred. which turn benefits the data association since we can expect better estimates for the affinities a10e de) and Amotioa (Tc, de). In addition, we propose to

---

integrate cross camera trajectories into the training process of the motion model. The intuition is that due to object truncation. single-view 3D localization image borders is usually brittle. Thus, if we train the motion model on these trajectories, it can handle those situations better. In contrast to Hu et al. [1], who create the trajectory training data per camera and train on the union of the single-camera trajectories, we create cross-camera trajectory dataset that matches ground truth trajectories spanning multiple cameras to the aggregated set of detections from all cameras. To match detections to ground truth trajectory, we use birds-eye-view (BEV) distance instead of Intersection-over-Uuni (loU) of 2D bounding boxes in Hu et al. [1].
To train Ppred and P uupatee we randomly sample ground truth trajectory and extract the matched ground truth and detection pairs in a random temporal window At of fixed size, possibly spanning across multiple camera views. We use the set of detections to simulate inference, i.e. we obtain the predicted velocities Vaz from Ppred given the track history and the refined velocities Var from Papdata given the predicted velocity, the observed velocity, and the detection confidence. We use the ground truth trajectory to calculate the ground truth velocities Var and the trajectory loss
Leras (Val, VAG Vai)
1
8(V1, v.) 5(v,, Vz), Σ tEAr
(5)
|
where -(:-) is the Huber loss [60]. Moreover, we add a regularization term that encourages o prred to predict smooth trajectory over At:
1
Liteear (Dai) = ||A/||
11(8+11 - - -
(6)
LEAL
Finally, we calculate the overall loss Lmotion weighted combination of the above terms
Lastion EVar, Vac, Vai)= = Liraj (Var, Val. Vat) + WWhinerrrineer( (Vat).
4 Experiments
Datasets. We evaluate our method on two popular 3D MOT benchmarks. First, we experiment on the NuScenes benchmark [9] which consists of street scenes captured in varying scenery and geographic areas in Boston and Singapore. The scenes are captured from moving vehicle with multiple sensors attached. It provides six different camera views with a 360-degree coverage of the environment. In total, the dataset contains 1000 sequences sampled at 2 Hz. Second, we evaluate our method on the Waymo Open benchmark [10], which has similar characteristics to the NuScenes benchmark. However, the sampling rate is 10 Hz, and the five cameras mounted the vehicle cover only about a 5/8 azimuth angle of the horizon.
Evaluation metrics. We follow the evaluation protocol in NuScenes using the metrics in Weng et al. [53]. The main metrics are average multi object tracking accuracy (AMOTA) and average multi object tracking precision (AMOTP) which are integrals over MOTA MOTP [61] scores computed via n-point interpolation. AMOTA is defined as AMOTA
MOTA,, Specifically, MOTA, is recall normal- ized MOTA and IDS,, FP,, FN, are number of ID switches, false positives and false negatives at recall r. P is the total number of ground truth positives. AMOTP is defined AMOTP Σ., di.
-
where MOTA, max (0, 1 - +FS
R Er. 1) E. TP. where di,, is the BEV distance between track i and its ground truth track at time l and TPP denotes the number matched ground truth tracks at time 1.
Implementation details. We train the 3D estimation network using ResNet- 101 [62] as backbone with batch size of 32 for 24 epochs with random image flipping as data augmentation on V100 GPUs. The initial learning rate is set to 0.01 with a linear learning rate warm-up in the first 1000 iterations. We decay the learning rate by factor of 10 at epochs 16 and 22, respectively. We use SGD [63] with a momentum of 0.9 and a weight decay of 0.0001. For NuScenes, we use 1600 x 900 resolution and train on the training set only. On Waymo Open, we scale the images to 1280 854 resolution and use batch size of 16 during training. We decay the learning rate by factor of 5 at epochs 8, 12, 16, 20, 22, respectively. To merge 3D detections across cameras, we use NMS with a 3D IoU threshold of 0.1. To generate the cross-camera trajectory dataset, we use matching threshold of meters BEV distance. We train Preed and Papdate with batch size of 128 for 100 epochs. We use Winnerr = 0.001 and Adam [64] with AMSGrad and weight decay of 0.0001.
5

---

Table I: Pipeline and motion modeling ablation study. We compare the tracking performance of our method with the single-camera baseline [1] and the detect -> track - merge paradigm on the NuScenes validation split. We further analyze the benefit of motion modeling across all pipelines, both with LSTM motion model and a Kalman Filter baseline (KF3D).
Pipeline
Motion Model
AMOTA
AMOTP 4
RECALL f
MOTA t
IDS +
0.221
1.529
0.371
0.193
5256
QD-3DT
KF3D LSTM
0.232
1.530
0.393
0,206
5574
0.247
1,507
0.399
0.218
5919
0.248
1.500
0.386
0.212
4451
Detect -> Track Merge
KF3D
0.256
1.506
0.408
0.225
4256
LSTM
0.264
1,490
0.413
0.235
4470
0.249
1.490
0.405
0.212
2399
Detect + Merge + Track
KF3D LSTM
0.279
1.501
0.415
0.252
1982
0.283
1.490
0.444
0.262
2131
Table 2: Motion model training ablation study. We compare modified training scheme for the LSTM motion model with cross-camera trajectories to the single-camera baseline on the NuScenes validation split. f denotes using new training scheme for 3D estimation network.
Method
Trajectory data
matching
Total Trajectories
AMOTA t
AMOTP +
IDS +
CC-3DT
Single-camera
2D loU
29527
0,283
1.490
2131
CC-3DT
0,289
1,483
2129
Cross-camera
BEV Distance
36112
CC-3DT
0.311
1.433
2536
4.1
Ablation studies
Cross-camera association. In Table 1, we compare the pipeline in Hu et al. [1], naive merging of single camera results (detect track merge) and our pipeline descrihed in section 3.2 using the LSTT motion model, a Kalman Filter baseline (KF3D) and no motion model. Note that all individual components are consistent while only changing the high-level pipeline provide fair comparison. We observe that naive merging helps the overall performance and results in gain of 1.7% in AMOTA using the LSTM motion model. With our cross-camera association (detect + merge- + track), we observe another 1.9% gain in AMOTA while the IDS are cut in half, high- lighting the effectiveness of cross-camera association. Note that while KF3D has slightly less IDS than LSTM. its recall level is also lower, i.e. IDS are lower due to lower recall not better asso- ciation. Interestingly, the gain of cross-camera association is very small when not using motion model (0.15 1)), supporting our hypothesis that the motion model benefits greatly from cross-camera association. Moreover, the influence of the motion model on the final tracking performance the highest when using cross-camert association, improving from 24.9% to 28.3% AMOTA. Our results underline the importance of cross-camera association to achieve consistent tracking results.
Cross-camera motion modeling. Table also shows that motion modeling improves through the use of cross-camera trajectories, i.e. we observe higher gain in AMOTA of 3.4% (LSTM) and 2.0% (KF3D) versus using no motion model compared to only 2.6% (LSTM) and 1.1% (KF3D) when using the pipeline in Hu et al. [1]. The effect of training the motion model with cross-camera trajectories (see section 3.3) is illustrated in Table We observe another improvement of 0.6% in AMOTA compared to utilizing cross-camera trajectories only during inference. Moreover, Table 2 shows that the total number of trajectories for training increases, which can be attributed to the fact that for trajectory to be considered for training, it needs to be at least || time steps long. Cross- camera trajectories help to increase training data diversity. In sum, we achieve a gain of 4.2% in AMOTA, equating 17% relative improvement compared the single-camera baseline.
Qualitative comparison. Figure 2 illustrates CC-3DT results compared to the baselines in Table 1. While our proposed CC-3DT preserves object identity and exhibits smooth, accurate trajectory for the object moving across camera views, the baselines struggle with object localization and object as- sociation. Specifically, QD-3DT [1] cannot preserve the object identity while naive merging (detect - track - merge) results in poor localization, showing the benefit of our method.

---

Table 3: Comparison to MUTR3D [58] using different motion models. We report tracking per- formance on the NuScenes validation split. We use DETR3D [29] detector for fair comparison.
Method
Motion model
AMOTA
AMOTP
RECALL f
MOTA t
IDS І
SimpleTrack FFN
0.293
1,307
0.418
0.263
1695
MUTR3D [58]
0.294
1.498
0.427
0.266
3822
0.307
1.394
0.434
0.263
2419
CC-3DT
KF3D LSTM
0.353
1.379
0.478
0.316
2050
0.359
1.361
0.498
0.326
2152
QD-3DT
Detect >Trakk >eegee
CC-3DT
Back t
Back Left (E 2) Back Left ( +
3D View
Figure 2: Qualitative comparison. Different types of cross-camera association on NuScenes val- idation split. We plot the 3D car states in camera and 3D views and depict car identity as color. Clearly, CC-3DT provides smooth and precise trajectory
cameras (green). Detect Track + Merge shows awry car localization (purple), while QD-3DT lacks car identity (violet).
3D estimation network training scheme. Following [29] and different from [1], optimize the 3D estimation network training scheme by increasing the learning rate the 3D detection head by factor of 10. As shown in Table 2, this improves the overall performance by 2.2% in AMOTA. Note that IDS increase due to increased recall our detector.
4.2 Comparison to state-of-the-art
Comparison on the NuScenes benchmark. We evaluate our method the NuScenes benchmark in Table 4. On the validation set, we outperform the state-of tfate-of-the-ar method MUTR3D by 1.7% with weaker detector, by 6.5% in AMOTA with the same detector (DETR3D [29]), and by 13.5% with stronger detector (BEVFormer [66]). Our AMOTPP is significantly lower (1.361 vs. 1.498) with the same detector, meaning that our trajectories are more precise. We also have the lowest IDS, meaning that our association works best. The testing set results corroborate our findings, i.e. we outperform MUTR3D by as much as 12.4% AMOTA and achieve the lowest AMOTP and IDS. We set new Mate-offthhe-rt of 42.9% and 41.0% AMOTA the validation and testing sets, respectively.
7

---

Table 4: Comparison to state-of-the-art. We compare the tracking performance of our method with existing camera-based methods on the NuScenes validation and testing sets.
Method
Detector
AMOTA
AMOTP
RECALL
MOTA f
IDS
DEFT [3]
CenterNet [65]
0.201
N/A
N/A
0.171
N/A
Validation
QD-3DT
Faster-RCNN [11]
0.247
1.507
0.405
0.221
5919
MUTR3D [58]
DETR3D [29]
0.294
1.498
0.427
0.267
3822
Faster-RCNN [11]
0.311
1.433
0.472
0.278
2536
CC-3DT
DETR3D [29]
0.359
1.361
0.498
0.326
2152
BEVFormer [66]
0.429
1.257
0.534
0.385
2219
CenterTrack [2]
CenterNet [65]
0.046
1.543
0.233
0.043
3807
DEFT [3]
CenterNet [65]
0.177
1.564
0.338
0.156
6901
Testing
QD-3DT [1]
Faster-RCNN [11]
0.217
1.550
0.375
0.198
6856
MUTR3D [58]
DETR3D [29]
0.270
1,494
0.411
0.245
6018
Faster-RCNN [11]
0.284
1.470
0.427
0.251
2945
CC-3DT
DETR3D [29]
0.357
1.343
0.472
0.317
3070
BEVFormer [66]
0.410
1.274
0.538
0.357
3334
Comparison to MUTR3D. Table provides additional comparison MUTR3D using different motion models. MUTR3D reports results with a Kalman Filter baseline [67] and a feed-forward network (FFN). While the FFN can only improve marginally upon the Kalman Filter baseline, our motion model significantly outperforms the Kalman Filter baseline (KF3D). Also, our cross-camera association is more effective than the transformer-based while MUTR3D with Kalman Filter has the lowest IDS, the detection recall is also much lower.
in MUTR3D (lower IDS). Note that
Comparison the Waymo Open benchmark. We compare with [1] on the validation split of Waymo Open. The comparison in Table 5 shows that our method performs better in MOTA than QD-3DT, while also having lower Mismatch ra- tio, meaning that we have both better overall and association performance. We outperform QD- 3DT significantly on each of the 3D IoU thresh- olds. Note that do not evaluate with an IoU threshold of 0.7, as is too restrictive for camera- based methods achieve a meaningful score.
Table 5: Comparison on Waymo Open. We compare our 3D Vehicle Tracking results with QD-3DT the validation set of the Waymo Open benchmark (Level difficulty).
Method
3D loU
MOTA
Mismatch $
0.3
0.1867
0.0134
QD-3DT
0.5
0.0308
0.0055
0.3
0.2032
0.0069
CC-3DT
0.5
0.0480
0.0018
4.3 Limitations
Our method could fail the predicted 3D detection is amiss. Specifically, in cases of high occlusion truncation, the object's 3D properties are often ambiguous to the observer, e-g. object's 3D center could even be out of the camera FOV. This is especially pronounced for big or close objects that are often only partially visible. While combining information from multiple time steps in our motion model alleviates those issues, integrating this information during the detection stage could further improve the results. These challenges can also influence the classification performance the 3D detector. If an object is highly occluded or truncated, it is often challenging to infer its category from a single image, especially for semantically similar classes (e.g. bus and truck). However, if the category is estimated incorrectly, object identity may not be preserved.
Conclusion
This paper presents an effective 3D object tracking method in multi setting. The key idea we present is to merge 3D detections of multiple cameras into single 3D tracking component that handles both temporal and cross-camera association. By leveraging cross-camera association, we observe improved motion model quality, leading to enhanced object association and smoother trajectories across camera views. As result, the suggested approach outperforms existing methods by a large margin achieving tate-of-theear results on the widely-used NuScenes benchmark.

---

Acknowledgments
This work is supported by Ministry of Science and Technology of Taiwan (MOST 110-2634-F- 002-051). We gratefully thank National Center for High-performance (NCHC) for the storage and computational resource.
References
[1] H.-N. Hu, Y.-H. Yang, T. Fischer, T. Darrell, F. Yu, and M. Sun. Monoeular quasi-dense 3d object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[2] X. Zhou, V. Koltun, and P. Krähenbühl. Tracking objects as points. In European Conference on Computer Vision, 2020.
[3] M. Chaabane, P. Zhang, J. R. Beveridge, and S. O'Hara. Deft: Detection embeddings for tracking. arXiv preprint arXiv:2102.02267, 2021.
[4] D. Ramanan and D. A. Forsyth. Finding and tracking people from the bottom up. In IEEE Conference Computer Vision and Pattern Recognition, 2003.
[5] F. Yu, W. Li, Q Li, Y. Liu, X. Shi, and Yan. POI: multiple object tracking with high performance detection and appearance feature. In European Conference on Computer Vision Workshop, 2016.
[6] N. Wojke, A. Bewley, and D. Paulus. Simple online and realtime tracking with a deep associ- ation metric. In International Conference on Image Processing, 2017.
[7] P. Bergmann, T. Meinhardt, and L. Leal-Taixé. Tracking without bells and whistles. arXiv preprint arXiv; arXiv: 11990305625,, 2019.
[8] Z. Lu, V. Rathod, R. Votel, and J. Huang. Retinatrack: Online single stage joint detection and tracking. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[9] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. Liong, Q Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[10] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al. Scalability perception for autonomous driving: Waymo open dataset. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[11] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks, In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
[12] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. Ssd: Single shot multibox detector. European Conference on Computer Vision, 2016.
[13] Redmon, Divvala. R. Girshick, and Farhadi. You only look once: Unified, real-time object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[14] J. Redmon and A. Farhadi. Yolo9000: Better, faster, stronger. In JEEE Conference on Com- puter Vision and Pattern Recognition, 2017.
[15] T.-Y. Lin, Goyal, Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In IEEE International Conference on Computer Vision, 2017.
[16) X. Chen, K. Kundu, Y. Zhu, A. G. Berneshawi, H. Ma, S. Fidler, and R. Urtasun. 3D object proposals for accurate object class detection. In Advances in Neural Information Processing Systems, 2015.
[17] X. Chen, K. Kundu, 7. Zhang, H. Ma, S. Fidler, and R. Urtasun. Monocular 3D object detection for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.

---

[18] A. Mousavian, D. Anguelov, J. Flynn, and J. Košecká. 3D bounding box estimation using deep learning and geometry. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[19] G. Brazil and X. Liu. M3D-RPN: Monocular 3D region proposal network for object detection. In IEEE International Conference on Computer Vision, 2019.
[20] A. Simonelli, S. R. Bulo, L. Porzi, M. López-Antequera, and P. Kontschieder. Disentangling monocular 3D object detection. In JEEE International Conference on Computer Vision, 2019.
[21] D. Park, R. Ambrus, V. Guizilini, J. Li, and A. Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In IEEE International Conference on Computer Vision, 2021.
[22] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger. Pseudo- LiDAR from visual depth estimation: Bridging the gap in 3D object detection for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[23] X. Ma, Z. Wang, Li, Zhang, W. Ouyang, and Fan. Accurate monocular 3D object de- tection via color-embedded 3D reconstruction for autonomous driving. In IEEE International Conference on Computer Vision, 2019.
[24] C. Reading, A. Harakeh, J. Chae, and S. L. Waslander. Categorical depth distribution net- work for monocular 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[25] Y. Zhou and O. Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In IEEE Conference on Computer Vision and Puttern Recognition, 2018.
[26] S. Shi, X. Wang, and H. Li. PointRCNN: 3D object proposal generation and detection from point cloud. In JEEE Conference on Computer Vision and Pattern Recognition, 2019.
[27] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. PointPillars: Fast encoders for object detection from point clouds. IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[28] N. Carion, Massa, G. Synnaeve, N. Usunier, A. Kirillov, and Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, 2020.
[29] Y. Wang, V. Guizilini, T. Zhang, Y. Wang, H. Zhao, and M. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In The Conference on Robot Learning (CoRL), 2021.
[30] A. Bewley, Z. Ge, L. Ott, F. T. Ramos, and B. Upcroft. Simple online and realtime tracking. In International Conference Image Processing, 2016.
[31] B. Xiao, H. Wu, and Y. Wei. Simple baselines for human pose estimation and tracking. In European Conference on Computer Vision, 2018.
[32] D. Held, S. Thrun, and S. Savarese. Learning to track at 100 FPS with deep regression net- works. In European Conference on Computer Vision, 2016.
[33] C. Feichtenhofer, A. Pinz, and Zisserman. Detect to track and track detect. In IEEE International Conference on Computer Vision, 2017.
[34] C. Kim, F. Li, A. Ciptadi, and J. M. Rehg. Multiple hypothesis tracking revisited. In IEEE International Conference on Computer Vision, 2015.
[35] L. Leal--Tixé, C. Canton-Ferrer, and K. Schindler. Learning by tracking: Siamese CNN for robust target association. In IEEE Conference on Computer Vision and Pattern Recognition Workshop, 2016.
[36] Y. Zhang, C. Wang, X. Wang, W. Zeng, and W. Liu. Fairmot: On the faimesss of detection and re-identification in multiple object tracking. arXiv preprint ar arXiv:2004.0/ 018888, 2020.
10

---

[37] J. Pang, L. Qiu, X. Li, H. Chen, Li T. Darrell, and F. Yu. Quasi-dense similarity learning for multiple object tracking. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[38] T. Fischer, J. Pang, T. E. Huang, L. Qiu, H. Chen, T. Darrell, and F. Yu. Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking. arXiv preprint arXiv:2210.06984 2022.
[39] D. Held, J. Levinson, and S. Thrun. Precision tracking with sparse 3d and dense color 2d data. In IEEE International Conference on Robotics and Automation, 2013.
[40] D. Mitzel and B. Leibe. Taking mobile multi-object tracking to the next level: People, unknown objects, and carried items. In European Conference on Computer Vision, 2012.
[41] A. Osep, W. Mehner, M. Mathias, and B. Leibe. Combined image-and world-space tracking in traffic scenes. In IEEE International Conference on Robotics and Automation, 2017.
[42] S. Sharma. J. A. Ansari, K. Murthy, and K. M. Krishna. Beyond pixels: Leveraging geometry and shape cues for online multi-object tracking. In IEEE International Conference on Robotics and Automation, 2018.
[43] J. Luiten, T. Fischer, and B. Leibe. Track to reconstruct and reconstruct to track. IEEE Robotics and Automation Letters, 2020.
[44] T. Meinhardt, A. Kirillov, L. Leal-Taixe, and C. Feichtenhofer. Trackformer: Multi-object tracking with transformers. arXiv preprint arXiv:2101.02702 2021.
[45] P. Sun, J. Cao, Y. Jiang, R. Zhang, E. Xie, Z. Yuan, C. Wang, and P. Luo. Transtrack: Multiple- object tracking with transformer. arXiv preprint arXiv: 2542.0 2020.
[46] F. Zeng, B. Dong, T. Wang, X. Zhang, and Y. Wei. Motr: End-to-end multiple-object tracking with transformer. arXiv preprint arXiv:2 005.03247 2021.
[47] Z. Tang, M. Naphade, M.-Y. -Y. Liu, X. Yang. S. Birchfield, S. Wang, R. Kumar, D. Anastasiu, and J.-N. Hwang. Cityflow: A city-scale benchmark for multi-target multi-camera vehicle tracking and re-identification. IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[48] X. Han, Q. You, C. Wang, Z. Zhang, P. Chu, H. Hu, J. Wang, and Z. Liu. Mmptrack: Large- scale densely annotated multi-camera multiple people tracking benchmark. arXiv preprint arXiv:2111.15157, 2021.
[49] E. Ristani and C. Tomasi. Features for multi-target multi-camera tracking and re-identification. In IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[50] Y. He, X. Wei, X. Hong, W. Shi, and Y. Gong. Multi-target multi-camera tracking by tracklet- to-target assignment. IEEE Transactions on Image Processing, 2020.
[51] K. Shim, S. Yoon, K. Ko, and C. Kim. Multi-target multi-camera vehicle tracking for city- scale traffic management. In IEEE Conference on Computer Vision and Pattern Recognition Workshop, 2021.
[52] M. Naphade, S. Wang. D. C. Anastasiu, Z. Tang, M.-C. Chang, X. Yang, Y. Yao, L. Zheng, P. Chakraborty, C. E. Lopez, A. Sharma, Q. Feng, V. Ablavsky, and S. Sclaroff. The 5th ai city challenge. In JEEE Conference on Computer Vision and Pattern Recognition Workshop, 2021.
[53] X. Weng, J. Wang, D. Held, and K. Kitani. 3d multi-object tracking: A baseline and new evaluation metrics. In JEEE/RSJ International Conference Intelligent Robots and Systems, 2020.
[54] T. Yin, X. Zhou, and P. Krähenbühl. Center-based 3d object detection and tracking. 1EEE Conference on Computer Vision and Pattern Recognition, 2021.
[55] C. Luo, X. Yang, and A. Yuille. Exploring simple 3d multi-object tracking for autonomous driving, In IEEE International Conference on Computer Vision, 2021.
11

---

[56] Y. Wang, K. Kitani, and X. Weng. Joint object detection and multi-object tracking with graph
neural networks. In IEEE International Conference on Robotics and Automation, 2021.
[57] A. Kim, A. Ošep, and L. Laxl-Taixé. Eagermot: 3d multi-object tracking via sensor fusion. In IEEE International Conference on Robotics and Automation, 2021.
[58] T. Zhang, X. Chen, Y. Wang, Y. Wang, and H. Zhao. Mutr3d: A multi-camera tracking frame- work via 3d-to-2d queries. arXiv preprint ar arXiv:2205.000 00061,,,,,,, 2022.
[59] H.-N. Hu, Q.Z Cai, D. Wang, J. Lin, M. Sun, P. Krähenbühl, T. Darrell, and E, Yu. Joint monocular 3d vehicle detection and tracking. 2019.
[60] Huber. Robust estimation location parameter. The Annals of Mathematical Statisties, 1964.
[61] K. Bernardin and R. Stiefelhagen. Evaluating multiple object tracking performance: The clear mot metrics. EURASIP Journal on Image and Video Processing, 2008, 01 2008. doi:10. 155/ 2008/246309.
[62] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[63] H. Robbins and S. Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400-407, 1951.
[64] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv: 141 mtt../112... 2014.
arXiv preprint
[65] X. Zhou, D. Wang, and P. Krähenbühl. Objects as points. arXiv preprint 1rXiv:1904.07855 2019.
[66] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai. Bevformer: Learning bird's- eye-view representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203.172700 2022.
[67] Z. Pang, Z. Li, and N. Wang. Simpletrack: Understanding and rethinking 3d multi-object tracking. arXiv preprint 11..92211,.09621, 2021.
12


Reviews:
[{"review_id": "mOVFAVPLlZL", "reviewer": "Reviewer_qyz1", "summary": "This paper proposes MV-3DT, a multi-camera 3D object tracking pipeline. The major novelty of MV-3DT is how it handles detections results from multiple cameras, where unlike existing works, it aggerates all cameras' detections together and performs temporal track matching across all cameras. MV-3DT performances on large-scale real-world datasets are comparable with base models and is STOA on NuScenes. Also, the paper includes limited ablation study to demonstrate the performance contribution of their proposed multi-camera association pipeline. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper demonstrates MV-3DT performances across multiple large scale datasets and compare it with many base models. \n2.  The idea of using two LSTM to handle loss-tracked objects and tracking objects separately is an interesting approach for disappearing and reappearing objects.\nWeakness\n1. For 3D estimation network, the actual model architecture detail as well as the training loss for its three heads are not mentioned in the paper. \n2. For Cross-camera association, the merge state detail is missing, the paper doesn't discuss how tracks or detections are aggregated. For example for the later detect -> merge -> track pipeline, how are detections of the same object appears in two different cameras be aggregated?\n3. For section 3.2, the paper claims \"The intuition is that due to object truncation, the 3D localization at image borders is usually brittle\", and uses this as support for the idea of cross-camera trajectories. However, my argument is that merging multi-cameras images into a single panoramic image could handle the image border problem even better.  As a follow-up question about this paper's comparison result is that, there is no comparison with methods using bird eye view based detection methods, and I think that is an important part to justify the proposed model is better.\n4. The novelty of this paper is weak, merge detection before track doesn't seem to be enough to justify the novelty ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes MV-3DT, a multi-camera 3D object tracking pipeline. The major novelty of MV-3DT is how it handles detections results from multiple cameras, where unlike existing works, it aggerates all cameras' detections together and performs temporal track matching across all cameras. MV-3DT performances on large-scale real-world datasets are comparable with base models and is STOA on NuScenes. \nThe novelty of this paper is weak, merge detection before track doesn't seem to be enough to justify the novelty. Some model details and pipeline processing methods are not clearly stated in the paper. Also, the paper includes limited ablation study to demonstrate the performance contribution of their proposed multi-camera association pipeline, yet the claim of \"reduces identity switches significantly\" is not supported by ablation study. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. For 3D estimation network, the actual model architecture detail as well as the training loss for its three heads are not mentioned in the paper. \n2. For Cross-camera association, the merge state detail is missing, the paper doesn't discuss how tracks or detections are aggregated. For example for the later detect -> merge -> track pipeline, how is detections of the same object appears in two difference cameras be aggregated?\n3. For section 3.2, the paper claims \"The intuition is that due to object truncation, the 3D localization at image borders is usually brittle\", and uses this as support for the idea of cross-camera trajectories. However, my argument is that merging multi-cameras images into a single panoramic image could handle the image border problem even better.  As a follow-up question about this paper's comparison result is that, there is no comparison with methods using bird eye view based detection methods, and I think that is an important part to justify the proposed model is better."}}, {"review_id": "SeR2rfcQ_1M", "reviewer": "Reviewer_tUyR", "summary": "Accurate and consistent object tracking across multiple cameras is a critical necessity for autonomous vehicles to safely navigate in the real world. The authors introduce a cross-camera 3d fusion method, called MV-3DT, that fuses 3D object detection results from multiple cameras before object identity association while tracking objects in a multi-camera setup. In contrast, existing camera-based 3D object tracking methods first detect objects using a single-camera setup and later post-hoc fuse detections to track objects in a multi-camera setup. Experiments conducted by the authors on large-scale driving datasets namely NuScenes 3D tracking benchmark and Waymo open benchmark empirically demonstrate that by fusing 3D detection results before object association, MV-3DT results in better average multi-object tracking accuracy compared to the existing methods with a significant reduction in object identity switches temporally and across cameras.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors present an intuitive and effective idea of how information across multiple cameras should be combined to have accurate and consistent object tracking.\n- The manuscript is well written and captures the main ideas of the work comprehensively. Showing tracked objects across cameras in figure 1 is quite informative.\n- Videos showcasing object tracking accuracy across multiple cameras were quite insightful in appreciating the effectiveness of the work.\n\nWeaknesses:\n- Bird-eye view figures, especially figure 2, can be improved to make it easier for the reader to follow tracked objects across time steps. Discussed further in the “Issues” section.\n- Further quantitative results on the Waymo open benchmark should also be included.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The ideas introduced in the work are novel and exciting. The manuscript is well-written. It presents the core ideas comprehensively, warranting an acceptance of the work at the conference. However, the authors must address a few issues before I would be confident in recommending the work for publication. If the authors address the raised concerns in their rebuttal satisfactorily, I would be happy to recommend the manuscript for publication.\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- The symbol $I^m_t$ has been used twice (in lines 89 and 98) to denote two different entities, first given input images from a set of cameras and second, a single input image to the 3D estimation network. Different symbols should be used to denote different entities.\n- The affinity matrix A is computed as a weighted sum of the appearance affinity and a hybrid affinity of motion and location correlation. Please clarify why a hybrid affinity of motion and location correlation was considered instead of adding them as individual affinities. \n- In Table 1, the baseline KF3D leads to fewer identity switches compared to the proposed VeloLSTM method (see the last couple of rows). Similarly, in Table 2, MV-3DT without the new training scheme for the 3D estimation network reports fewer object identity switches compared to when the new training scheme is used. Please discuss potential reasons for these in the text and how to mitigate them.\n- In Figure 2, it is difficult to follow what the major implications of the plots are. For example, it is not clear what the green, yellow and red circles represent. If red circles represent smooth and precise object tracking, then why are there no red circles around the blue and green objects in the first row plots?\n- Briefly describe what are the major implications of the data presented in the tables in the caption. This helps the reader to quickly read and appreciate the experimental results.\n\nMinor comments:\n- In line 80, please say that “... Zhang et al. [60] propose MUTR3D, a method ….” as it is an important baseline and mentioning its name here helps the reader to follow the discussion more clearly.\n- Rephrase statement “... i.e. 2.6% vs. 1.6% vs. 3.4% difference 223 of using VeloLSTM versus using no motion model.” in lines 223-224 for better readability.\n"}}, {"review_id": "OePXoj2j1vs", "reviewer": "Reviewer_qceo", "summary": "This paper presents a new way targeting at 3D object tracking in a multi-camera setup. Compared to previous post-hoc fusion approaches, this paper presents presents two main contributions:\n\n1. cross-camera ssociation;\n2. cross-camera motion modelling.\n\nThe idea of fusing 3D detections from individual camera inputs before data association and then predicting and tracking objects is throughly verified in two different benchmark datasets, showing the tracking accuracy improvement brought by the contributions in the paper\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe idea of fusing the detections before performing tracking and predicting object trajectories is good and has been throughly verified in the experiment section.  \n\nWeakness:\nThis paper presents a quite large system, with many components and hyparameters. To understand the core contribution of this paper better, it will be better to address the issues that I have raised below. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper shows an advancement in the multi-camera 3D MOT field. The idea is not difficult but seems to work well. There are a few points that could be further explained better.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "First of all, I would like to admit that I do not have a lot of experience in the field of 3D MOT. From my perspective, the idea of fusing information across camera views before tracking is a good idea. However, after reading this paper, I have a few questions in my head.\n1. The whole paper presents a large system, with different components using different network components. For better reproduction of this paper, it will be better to explain the architecture of each component, for example, in the supplementary materials. \n2. There are some unexplained hyparameters within each model, such as the number of object velocities (L119), embedding dimension (L93), deep weight $w_deep$ (L115), and track management window $L$ (L134).Those parameters probably affect the performance of the system. It is not explained in the main paper or the supplementary document what values are used for these parameters in the experiments and how they are tuned. \n3. Why does Eq (4) choose to directly multiply the motion and velocity correnlation matrix? \n4. When referring to the main contribution in section 3.1 and 3.2, this paper frequently refers to [6], leaving an impression that [6] serves as a baseline of its system. However, system-wise, it is not clearly explained which components are inspired or borrowed from [6].\n5. As a submission to a robotics conference, I would expect to see some analysis (ideally a break-down of the components) of the computation time of the proposed method. For example, compared to the Kalman filter, how much of a computation time trade off does it make to gain the accuracy improvement? \n6. The experiments were conducted on two benchmarks. In the implementation details, it seems that different hyperparameters are needed to tune for different datasets. From my perspective, it is fine, but an explanation may be required. I assume the same training setting does not work well. Is it because of the sampling rate and the FOV difference or something else? And does it mean that the trained model may not generalise well to different sensor setups?"}}, {"review_id": "ApDE8ncD04-", "reviewer": "Reviewer_ANzg", "summary": "The paper “MV-3DT: Multi-View 3D Object Tracking via Cross-Camera 3D Fusion” is proposing a pipeline for tracking objects across multiple cameras and time for autonomous driving applications. The pipeline is relying on a monocular 3D object detection network that predicts bounding boxes for each image in a multi camera system. Afterwards, the object detections are associated across all cameras and merged using 3D non-maximum suppression. The merged 3D objects are tracked over time using a learned LSTM-based motion model. Furthermore, a track management is proposed, where each object track can have the following states: active, inactive, dead. This way objects can be tracked even if they are not detected over a longer time frame.\nThe proposed pipeline is evaluated against recent single-camera (QD-3DT) and multi-camera (MURT3D) on the NuScenes and Waymo Open dataset. On both datasets the proposed method is outperforming the baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is very well structured and easy to read. Therefore, one can easily get the idea of the proposed method and how it differentiates from the existing state of the art\n* All design choice are well explain and a validated in the presented ablation studies\n* Implementation details are given which can be helpful to reproduce the results presented in the paper\n* Extensive experiments are perform to evaluate the proposed pipeline against existing state-of-the-art approaches\n\nWeaknesses:\n* The presented paper shows only minor weaknesses, which are as follows:\nIt would be beneficial to also report runtimes in the evaluations section. E.g. it is shown that the LSTM-based motion model is outperforming a classical EKF. However, it would be beneficial to understand how both model differ in runtime.\n* To improve the presentation of the paper it would be beneficial to move some of the qualitative results from the supplementary to the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is nicely written and easy to follow. Furthermore, it is proposing a novel approach and clearly explaining the improvements to the current state-of-the-art. The performance is evaluated based on extensive experiments.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "* Add runtime measurements for the pipeline\n* Move qualitative results from supplementary to main paper", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)