DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera 3D Object Detection and Tracking
Authors: Qing LIAN
Tai Wang
Dahua Lin
Jiangmiao Pang
Venue: CoRL 2023 Poster (2023)
Keywords: Temporal Modeling
3D Object Detection

Abstract: Recent multi-camera 3D object detectors usually leverage temporal information to construct multi-view stereo that alleviates the ill-posed depth estimation. However, they typically assume all the objects are static and directly aggregate features across frames. This work begins with a theoretical and empirical analysis to reveal that ignoring the motion of moving objects can result in serious localization bias. Therefore, we propose to model Dynamic Objects in RecurrenT (DORT) to tackle this problem. In contrast to previous global BirdEye-View (BEV) methods, DORT extracts object-wise local volumes for motion estimation that also alleviates the heavy computational burden. By iteratively refining the estimated object motion and location, the preceding features can be precisely aggregated to the current frame to mitigate the aforementioned adverse effects. The simple framework has two significant appealing properties. It is flexible and practical that can be plugged into most camera-based 3D object detectors. As there are predictions of object motion in the loop, it can easily track objects across frames according to their nearest center distances. Without bells and whistles, DORT outperforms all the previous methods on the nuScenes detection and tracking benchmarks with 62.8% NDS and 57.6% AMOTA, respectively. The source code will be available at https://github.com/OpenRobotLab/DORT.

DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera 3D Object Detection and Tracking
Qing I Lian
Tai Wang Dahua Lin
Jiangmiao Pang 1
Shanghai AI Laboratory
Thhe Hong Kong University of Science and Technology
'The Chinese University Hong Kong
ęl glLanabeconnect.co hkk {wt019, dhlin]eie cuhk. edu.hk, pangjiangmiao@gmailll
Recent multi-camera 3D object detectors usually leverage temporal information to construct multi-view stereo that alleviates the ill-posed depth es- timation. However, they typically assume all the objects are static and directly aggregate features across frames. This work begins with theoretical and em- pirical analysis reveal that ignoring the motion of moving objects can result in serious localization bias. Therefore, we propose to model Dynamic Objects in RecurrenTT (DORT) to tackle this problem. In contrast to previous global Bird- Eye-View (BEV) methods, DORT extracts object-wise local volumes for motion estimation that also alleviates the heavy computational burden. By iteratively re- fining the estimated object motion and location, preceding features can be precisely aggregated to the current frame mitigate the aforementioned adverse effects. The simple framework has two significant appealing properties. is flexi- ble and practical that can be plugged into most camera- based 3D object detectors. As there are predictions of object motion in the loop, it can easily track objects across frames according to their nearest center distances. Without bells and whis- tles, DORT outperforms all the previous methods on the nuScenes detection and tracking benchmarks with 62.8% NDS and 57. 57.6% AMOTA, respectively. Codes are available https: httpb//////tt.uuueeee//e thub wwtpc///pwooooo.aaaboooooooopp
Abstract:
Keywords: Temporal Modeling, 3D Object Detection
Introduction
Multi-camera 3D object detection is critical robotic systems such autonomous vehicles, hu- manoid robots, and etc. As object depth estimation from single image is naturally ill- ill-posed, recent works use large-scale depth pre-trained models [I] and leverage geometric relationships [2, 3, 4, 5] to alleviate the problem. Because stereo correspondence exists in consecutive frames, some works resort to temporal information for accurate depth predictions. For example, BEVDet4D [6] and BEVFormer [7] warp preceding features to the current frame to enrich the single-frame BEV rep- resentations. DfM [5] constructs temporal cost volumes that explicitly establish the stereo corre- spondence. However, these cross-frame feature aggregations do not consider the motion of moving objects and assume all the objects static, which results in serious 3D localization bias.
In this paper, we first provide a theoretical and empirical analysis to reveal the negative effects of inaccurate object motion to object depth (Fig 1). In particular, the object is moving, the incorrect temporal correspondence would derive biased depth. In the driving scenarios, it is critical that misleading depth is estimated, which might reduce the reaction time of the decision system, leading catastrophic collision accidents. This motivates to devise explicit mechanism involve object motion estimation in the temporal-based 3D detection pipeline.
X Corresponding author.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Modeling dynamic objects in this context has several challenges: (1) We need a flexible object-wise representation for potential object-wise operations hased on motion modeling. (2) Jointly estimating object location and motion is an inherent chicken and egg problem [8]: The temporal correspondence can derive accurate object location only when accurate object motion is given and vice versa. (3) Simultaneously predicting object location and motion from only two frames is also an iol-posed problem theoretically, and thus it is desired to involve right-body assumption and more frames to pose reasonable constraints.
To address these problems, we model Dynamic Objects in RecurrenT (DORT) that simultane- ously estimates object motion and location, and then progressively refines them for accurate 3D object detection. It benefits from local 3D vol- ume representation that not only extracts object- wise 3D features but also alleviates the heavy computational costs of global BEV in previous methods [5, 7, 9]. Based on the object-wise vol- ume, temporal volumes are constructed by warp- ing the volumes from the preceding frame the current frame according to the object motion. Then the obtained cost volumes act as the features for updating the candidate location and motion. We model this estimation and update pipeline as a recurrent process to alleviate the aforementioned chicken and egg problem. In addition, our frame- work can take into more than two frames and pose constraints to the object motion. It inher- ently provides feasible solution to avoid the ill- posed dilemma of estimating object location and motion from only a single pair of correspondence. As there is object motion prediction in the loop, the framework is naturally capable of joint object detection and tracking by utilizing object motion to align the detection results into the same timestamp. also can be plugged into most camera- based 3D object detectors for flexible and practical use.
Inacourate Location
(Previous)
& Motion (staric)
Motion Estimation
Progressively
Refinement
(Ours)
Accurate Location & Motion
Frame T-
Frame T T
Figure 1: Visualization of object localization from temporal correspondence. Previous works ignore the motion of moving objects, leading to imprecise local- ization. Our work progressively refines the object's s location and motion so that the preceding features can be precisely aggregated.
We validate the effectiveness of our framework on the nuScenes detection and tracking benchmarks. Benefiting from the dynamic objects modeling, DORT outperforms all the previous methods with a large margin, leading to 62.8% nuScenes detection metric (NDS) and 57.6% and average multi- object tracking accuracy (AMOTA), respectively.
Related work
Monocular 3D Object Detection Monocular-based 3D object detection was first approached from the framee-frame scenario and evolved into multi-frame to alleviate the ill posed depth estimation.
(a) Methods with A Single Frame The single-frame-base methods [10, 11, 3. 12, 13] first extend 2D object detectors and insert several 3D attribute regression heads to predict 3D bounding boxes. To alleviate the ill-posed depth recovery, several methods improve the model from the perspectives of loss function [11], network architecture [14, 15], regression objective [3, 4], etc. Besides directly regressing depth, later approaches [2, 16, 17] further design 2D-3D geometry constraints to better extract visual cues for depth estimation. To align detection features with the output space, another line of methods [18] designs several transformation modules to lift 2D inputs into 3D space. Pseudo- based methods [19, 20, 21, 21] first predict the per-pixel depth and convert the raw pixel into point cloud for 3D detection. BEV-based methods [18, 22, 9, 23] propose orthographic feature transformation (OFT) to transform the 2D features into 3D voxels and then adopt ADAR-bbsce

---

head to localize objects, Later works improve the OFT by explicit depth distribution modeling [22, 9, 23), incorporating deformable attention module [7] or designing 3D position encoding [24, 25].
(b) Methods with Multiple Frames Although many techniques are designed methods, they still suffer from ill-posed depth recovery, leading to unsatisfactory performance for deployment. To augment the single-view observation, recent works [26, 14, 5, 6, 7, 27] leverage previous frames as additional observations for features augmentation. Kinematic3D [14] leverages 3D Kalman Filter to associate objects across frames and refines 3D boxes. Later studies [5, 6, 7] construct cross-frame cost volumes as another visual cue for 3D detection. The cost volumes are based on the multi-view stereo, which assumes objects are static across frames. However, this assumption does not align with the driving scenario, where the objects can move.
mungle-rrame-bart
Monocular 3D Object Tracking 3D object tracking associates objects across frames and gener- ates set of trajectories for motion prediction. Traditional methods adopt a paradigm that first detects objects in each frame and then associates them by appearance features [26] or objects' displacement with Kalman filter [28, 29, 30, 31, 32, 33]. Besides the above paradigm, several methods [34, 35] design two-stage paradigm that first associates objects and then utilizes the temporal motion to improve the detection performance.
this work, we utilize temporal cost volumes bridge the spatial location and temporal motion and derive recurrent paradigm that iteratively updates them obtain tightly coupled results for joint 3D detection and tracking.
3 Object Motion in Temporal Modeling
In this section, we first conduct analysis to demystify the adverse effects of neglecting object motion in temporal modeling, then discuss the challenges of modeling 3D motion the monocular setting.
Localization Bias from the Static Assumption In previous methods [5, 6, 7J. the object motion ignored by assuming objects are static and the features are directly aggregated after converting the past frames to the current frame. We first show that the static assumption would derive biased depth. Without loss of generality, we consider the two-view case, and it can be naturally extended to more than two views. We denote the camera intrinsic as K with focal length and center offset (f: Cas C.), and the ego motion and object motion from frame to to frame & as T:0go +4
and 7,
f Cu
0 0 z000
0 ohj
K
0 0
Tego 10
0 1 0
Tob
0 1 0
(1)
0
0
200
0
obj
For simplicity, we assume the ego and object mo- tion only contain the translation (xr 0, 2) on the horizontal plane. The analysis can be easily tended to the case that the motion contains rotation. Given the temporal images, we can utilize photo- metric or featuremetric similarity to find the corre- spondence of pixel Pt, = (Mtp 1%2) in the frame to and pixel Pt, = (u,, in the frame l1. Then, the depth zt, can recovered by:
Prrte Fame di
home 41.5
51% ar moring okjects
Cbject velocit (m/h)
Oeject velechy imu
(a) Average depth error vs object velocity
(b) Histogram of abject velocities
Figure Empirical analysis of the depth bias the nuScenes dataset if objects are assumed static.
z090 = Cu) fr
Trgo (Ptos K) = (Pe. K),
Zt,
t4o
Us
where æ denotes the 2D to 3D projection. This derivation assumes that the object is static. However, objects move with corresponding motion Tobj to Then, the object depth is revised as follows:
& (zeg0 + zobj - c..) f0 + xbb)
tt π (P K) = π T(p,, : K),
(3)
Uto 24s
Based on Eq (2) and (3), we can obtain the depth gap:
goy (uto - C.) - -fa ob
Az
(4)
Uso 4ei
3

---

Past Fommes
Terpprra Volumes
Shared RefioeNet
Retined Motion
Past Detection Results
Coav
Associate by Motien
Fred Metion
Warp by TigeMottin
Booled Festarcs
Cony
Prol. Beace Moire
Local Volumoi
Update
Current Frame
Desection Resulss
Vićio Sequace
Objectwwss Volumes Courzuction
Modeling
Joint Decction Trucking
Figure 3: Pipeline overview. Given a vidco sequence, we first extract the 2D features and generate the candidate boxes and motion by a single-frame detector. Then the boxes and motion are progressively refined from the concurrently updated 3D volume features. A fusion process in the recurrent module combines the estimation from each pair of frames. Based the tightly coupled modeling of object location and motion, the framework can achieve joint detection and tracking during inference.
From Eq (4), we can observe that the depth bias linearly correlated with the object motion. In Fig 2, we also display the empirical statisties of object motion and the corresponding depth bias from the nuScenes dataset. We can observe that the empirical depth error is also correlated with the object velocity and increases the time interval enlarges. Besides, the right part in Fig, also shows that almost 51% % of objects are moving across frames, demonstrating the necessity of modeling object motion in the temporal-based framework.
IlI-Posed Problem in Motion Modeling Except for demonstrating the necessity of modeling object motion, we also want mention that simultaneously estimating object location and motion is nontrivial, especially in the two-frame case. As shown in Fig the correspondence of two points can come from infinite combinations of object location and motion. This illustrates that joint location and motion estimation from only one correspondence is ill-posed. To alleviate this issue, we first simplify the object motion as a right-body movement so that multiple correspondences from the points in the object can be used to solve a shared motion. Furthermore, we leverage more than two frames to constrain the flexibility of object motion. More details can be referred Sec 4.3.
Methodology
This section describes the details of DORT. DORT is general joint detection and motion prediction module that can estimate coupled object location and motion results across frames. Based on the tightly coupled location and motion results, DORT is also capable of simultaneously 3D object detection and tracking. Basically, it can be based on most temporal 3D detectors [6, 36, 7]. In this work, we select the popular temporal detector BEVDepth [23] as the base detector and extend to handle both static and moving objects temporal modeling. We first present an overview of temporal-bris basoe frameworks in Sec 4.1 and then introduce our modifications: the local volume for object-wise representation Sec 4.2, the key recurrent dynamic objects modeling in Sec, 4.3, and the object association for monocular 4D object detection in Sec 4.4.
4.1 Overview of Temporal- -Base Frameworks
Current temporal-basee based methods contain three stages: (1) The 2D features extraction stage extracts the features from the 2D images. (2) The view transformation and stereo matching process that first lift the 2D features to 3D volume and then warp the features in each frame to aligned canonical space for matching. Depending on the model design, the order of view transformation and stereo matching may reverse. (3) The detection stage takes the 3D features to estimate 3D bounding boxes.
In this work, we follow previous methods [5, 9, 23] and adopt the widely-used backbone (e.g. ResNet [37]) do the features extraction. For the view transformation stage, we design an object- wise local volume that leverages the candidate 3D boxes to obtain the potential foreground regions and only models them with local wbsect-wise 3D volumes. For the stereo matching and detection

---

stages, we propose recurrent dynamic objects modeling module to progressively refine the de- tection and motion results for accurate 3D temporal features.
4.2 Object-wise Local Volume
In previous works, the 2D-3D transformation con- siders each candidate 3D grid point and constructs global volume for detection. However, there are sev- eral limitations: (1) the global volume contains lots of background regions, which is not vital for detection but increases the computation burden. (2) Modeling global volume needs to pre-define detection range during training, making the detectors fail detect ob- jects with arbitrary depth. (3) It is inconvenient to ma- nipulate global volume with object-wise operations.
Current 3D Volumes
View Transformation
Local Volumes
Currena Framo
Pred Boxes
Warp by Ego-Mottio
View Transformation
Object Motion
Post Frame
Past 3D Velumes
Figure 4: The process of extracting local vol- umes
current and past frames according to predicted bounding boxes and object motion.
Hence, we replace the global volume with an object-wise local volume. Specifically, we leverage the candidate boxes to determine the 3D region of interest (Rol) and set the local volume center as the bounding box center. To keep the object ratio and achieve eross-viis view warping, we assign each 3D Rol volume V R with the same 3D dimension (W, H, L) and channel size C. Different from 2D detection, the objects' dimension in 3D space has less variance and empirically relies less on the Rol-Align [38] operation. We display the construction of object-wise local volumes in Fig 4. For the 2D to 3D transformation, we first follow LiftSplat [22] and lift the images to 2.5D frustum by weighting with depth probability. Then we utilize the grid sample operation to warp the features from the 2.5D) frustum each 3D local volume. Benefiting from the accurate 2D detection performance, the local volume features sampled from the 2.5D frustum would have a large overlap with the foreground objects. Hence, if the proposal 3D location is inaccurate, the later refinement module still can use the features for refinement.
4.3 Recurrent Dynamic Objects Modeling
The pipeline of the recurrent framework is illustrated in Fig. Given the candidate 3D bounding boxes and motion as input, each iteration first constructs the temporal cost volumes thereon, and aggregates these cues to refine the proposal boxes and motion. In particular, we adopt perspective- view based 3D detector (i.e. PGD) to generate the initialized candidate 3D boxes and motion and only predict their residuals for refinement in the subsequent recurrent updates.
Cross-Frand Frame Cost Volumes Construction Given the initial predictions of 3D boxes and their motion, we first obtain object-wise volume features following Sec. 4.2. Then we can construct the temporal cost volumes by warping features from past frames the current frame coordinates based on ego-motion. In contrast previous works [5, 6] assuming objects are static, the object motion into the warping procedure. Specifically, for each point p R3 in the object-wiss, local volume V. we query the corresponding features in previous frame t- At with the consideration of ego-motion 7e0o and the object motion Too and construct the cost as [p)), Note that we simplify the point motion as the object motion with a rigid-body assumption, which can approximate most of the cases in driving scenarios, especially for vehicles [26, 39].
further involve
(T
3D Boxes and Motion Residual Estimation Given the object-wiss wise temporal features built from input 3D boxes and motion, we leverage refinement network to estimate the residual between the input 3D boxes and motion with the ground truth. The refinement network contains several 2D/3D residual-baseed convolutional layers to extract the 3D volumes and 2D BEV features. The detailed architecttre is presented in Supplementary. Formally, the refinement is formulated as the regres- sion of 3D attribute residuals B, including the object's 3D center x y/ 2, 3D size w, h, I, rotation 0 and velocity V,, vy. Since we use the object velocity in the current frame to represent the object motion and assume constant velocity across frame, the supervision for some frames may contain noise(e.g. inaccurate labels, violation of rigid-body assumption, etc). Hence, we model the residual
5

---

Table 1: Experimental results of monocular 3D object detection and tracking the nuScenes test set. The
input resolution 1600 900 with using ConyNeXtt [40] the backbone.
(a) 3D detection results.
(b) 3D tracking results.
Method
mAPEmATE. MASE) mADE! mAAE, NDST
Method
|AMUTA]
AMUTPI MOTAR
Ego3RT [41]
42.3
0.55
0.26
0.43
1.01
0.14
47.3
QD-3DT [34]
21.7
1.50
UVTR (42)
47.2
0.57
0.225
0.39
0.51
0.12
55.1
Time3D [35]
21.4
1.360
BEVFormer [7]
48.1
0.58
0.25
0.37
0.37
0.12
56,9
PolarDETR [30]
27.3
1.185
60.7
PETRv2 [43]
51.2
0.55
0.40
0.13
58.6
MUTR3D [44]
27.0
1.494
BEVDepth [23]
52.0
0.45
0.24
0.35
0.35
0.13
60.9
SRCN3D [3]]
39.8
1.317
70.2
BEVSterco [27]
52.5
0.43
0.35
0.14
61.0
OTTack
S0s.OFusion [36]
54.0
0.45
0.26
0.37
0.27
0.14
61.9
UVTR [42]
51.9
1.125
76.4
DORT (Ours)
55.3
0.43
0.26
0.42
0.24
0.14
62.8
DORT(Ours)
57.6
77.1
as Laplacian distribution and design the loss function as:
Lrefine
Abll
log logob).
(5)
O
BEB
Here, Ab, Ab, and σb are all the network outputs, and represent the ground truth residual, the esti- mated residual, and the estimated standard deviation of residual for each 3D attribute, respectively.
Multiple Estimation Fusion Given n frame as inputs, we can obtain n. 3D volumes (1 for the local volumes from the current view and n 1 for the paired cross-view cost volumes) and obtain estimated residuals from the residual estimation module. Then we weigh the importance of each residual by the estimated deviation and fuse them to obtain ensemble result:
e
bjused
(6)
where i denotes the volume index. For simplicity, here we only estimate the velocity measurement for the referenced frame, Le., the fluctuation of object velocity across different frames would not be considered explicitly. The mechanism for multi-frame fusion is expected to handle this problem adaptively. At the same time, this constraint also provides additional cues when simultaneously estimating object location and motion from more than two frames.
Recurrent Location and Motion Update After each iteration, we can obtain the refined bounding boxes with their motion and thus derive the updated bounding boxes in different frames. With these updated locations and Rols, next-round refinement. Note that any complex or learnable motion modeling can be integrated into this procedure. Here, to be consistent with the multiple estimation fusion designs, we still keep the constant velocity prediction for simplicity to derive the bounding boxes of previous frames.
further update the volume features and proceed to the
In the training stage, we follow the recurrent methods [45, 39] other tasks and set the loss weight for each iteration the same. The overall loss is represented
L= Cpv +
Crefine:
where Epn is the loss in the perspective-view detector [4), fie the refinement loss each iter- ation and k 3 is the number of the iterations. the inference stage, we first take the perspective- view detector (i.e. PGD [4]) generate the initial 3D boxes and their motion and then progressively refine them. In each iteration, we first construct the volume features discussed in Sec 4.3 and feed them into the refinement module to estimate the 3D boxes and motion residuals for each paired frame input. Then we utilize the multiple estimation fusion module in Sec 4.3 to fuse the estimated results and obtain the refined 3D boxes and motion as the next stage input.
4.4 Monocular 4D Object Detection
So far, have introduced our recurrent framework for 3D detection from monocular videos. Based on progressive refinement, our model can estimate tightly coupled object location and motion results and thus can casily associate the object detection results across frames, leading to joint 3D detection

---

Table 2: Experimental results the nuScenes validation set. The input resolution is 704 x 256 using ResNet- 50 the hackbone. * denotes the re-implementation based on the provided code.
Methods
# frame
mAP'
mATE,
mASE.
mAOE,
mAVE.
mAAE.
NDS
PGD [4]
28.8
0.75
0.27
0.52
1.13
0.18
37.0
BEVDeL
29.8
0,73
0.28
0.59
0.86
0.24
37.9
PETR [25]
31.3
0.77
0.28
0.56
0.92
0.23
38.1
DETR3D [24]
34,9
0.72
0.27
0.38
0.84
0.20
43.4
REVDesDD
32.2
0.70
0.28
0.50
0.35
0.21
DEVDepth [23)
2
35.1
0.64
0.27
0.48
0.43
0.20
47.5
DORT (Ours)
0.62
0.45
BEVDepth" [6]
39.8
0.57
0.27
0.49
0.27
0.18
52.3
DORT (Ours)
41.8
0.57
0.26
0.43
0.25
0.19
53.4
SOLOFasion [36]
16
42.7
0.57
0.27
0.41
0.25
0.18
53.4
DORT (Ours)
43.6
0.56
0.26
0.41
0.24
0.18
54.0
and tracking. Specifically, we follow [28, 46, 29, 47] and associate the detection results by warping
current detection to the past frames with object motion. Based on the ego-motion, we first convert the predicted object location to the past frame coordinate and then warp with the estimated object velocity. Then we follow the popular distance-based tracker [28, 29] and associate the objects by the closest distance matching. We provide more details of the tracking pipeline in the Appendix.
5 Experiments
5.1 Experimental Setup
In this section, we describe the used dataset, the evaluation metrics, and the implementation details.
Dataset NuScenes [48] is large-scale driving dataset, which contains 1,000 video sequences. The official protocol splits the video sequences into 700 for training, 150 for validation, and 1550 for testing. Each sequence is annotated with the objects' 3D bounding box, velocity, and tracking id.
Network Details As discussed Sec 4.3, the recurrent module requires proposal detector to generate candidate foreground regions as the 1st stage input. We adopt the popular monocular 3D detector PGD [4] due to its high 2D object detection recall. Following [9, 6, 23], we adopt the ResNet-50 [37] with FPN the 2D feature extractor and mainly conduct experiments on this setting. The 2D feature extractors in PGD and the recurrent module are shared to save computation time. The grid size of the 3D volume is set as 0.8m with the range of [-5m, 5m] in the X and Z (depth) axis and [-4m, 2m] in the Y (height) axis. During 2D to 3D features transformation, we follow [23] and adopt the depth distribution guided 2D to 3D features lifting. Regarding the test set submission, we follow [23, 27] and adopt the ConvNeXt-Baan [40] the image backbone. The image backbone is initialized with ImageNet pre-trained weights, and no other external data is used. We provide more details about the network architecture of the recurrent module and the training configuration in the supplementary materials.
Training Configurations The model is optimized by AdamW optimizer with weight decay 10 We first follow [4] to train the proposal detector and refine the recurrent module with 24 epochs, where the initial learning rate is set x 10 4 and decreases to 2 10-5 and 2 10 at the 18th and 22th epochs. Following [9, 23], use the class balance sampling strategy (CBGS) alleviate the class imbalance problem. We adopt the commonly used 2D data augmentation that randomly flips the image, resizes the image with the range of [0.36, 0.55], and crops the image to the resolution of 704 x 256. Regarding the input video sequences, we follow [6. 23] and sample the preceding keyframes to obtain the past video sequences. Regarding the test set submission, we enlarge the input resolution to 1600 640 and reduce the volume size to 0.4m.
5.2 Main Results
In Table I, we first provide the comparison of our framework with existing state-of-the-art methods on the nuScenes test benchmarks. We draw the following observations: (i) Benefiting from dynamic objects modeling, our method displays a significant improvement in both object detection (mAP) and motion estimation (mAVE), and 1.3% and relatively 11.1% better than the previous best method. These localization and motion estimation improvements also contribute to state-of tfte--of-the-at results in
7

---

Moonilng ectmaator
Piuumne stati
Setting
mAP NDS mAVE
Assumed static
35.0 47.1 0.37
GT motion
39.3
Pred motion
37.9 52.1 0.31
6.-
Acsumes ctare
Figure 5: Left and Middle: mAP mAVE each recurrent iteration step (1 past frame used). Right: Experimental results of different motion modeling strategies.
terms of the nuScenes detection metric (NDS). (ii) With strong localization and motion estimation results, our tracking module can better associate the detected objects in different timestamps, re- sulting in superior performance over all the other trackers with different metrics. Specifically, we improve the second best tracker [42], another distance- based tracker with % and 7.5% relative im- provements on the AMOTA and AMOTP metrics. Compared with the joint detection and tracking methods QDTrack3D [34] and Time3D [35], the performance gain of our method demonstrates the effectiveness of our dynamic objects modeling framework in jointly modeling object motion and location. (iii) In Table 2, we also report our method on the nuScenes validation set with different settings. For the detection performance, we can draw the same observation as in the test set that our method can outperform previous temporal-based methods [6, 23, 36] in terms of mAP and mAVE. Note that our method is also compatible with the components designed in the current BEV-based frameworks, such as the training techniques in BEVDepth [23] and the depth estimation module in SOLOFusion [36]. Furthermore, the local 3D volume is more friendly to practical applications, which can handle objects with arbitrary depth in the image.
5.3 Ablation Study
Ablation Study of Object Motion We further validate the influence of different dynamic object modeling strategies on the detection performance The first experiment compares the assumed static case with that of using the ground truth object motion. As shown in Figure 5, the model with ground truth object motion outperforms the assumed static with 4.3% mAP, demonstrating the necessity of object motion for obtaining accurate temporal correspondence features. When we replace the ground truth object motion with an estimated one, it still can bring 2.9% mAP improvements, illustrating the usefulness of our dynamic objects modeling module.
Experiments with Different Iterations In Figure 5, we provide the comparison of modeling object motion and assumed static with different recurrent iterations. Benefiting from the BEV features modeling, the two configurations display almost 2% mAP improvements in the first iterations. In the later iterations, the improvement in assumed static stops was mainly due to the lack of accurate temporal features. With more and more accurate temporal features, the model with modeling object motion can progressively improve the detection and motion estimation results.
Conclusion and Limitations
This work proposes novel framework to better leverage temporal information for camera-only 3D detection by modeling dynamic objects. We first design an object-wise local volume to save com- putation time and maintain object-wise representation for motion and detection modeling, Then we propose recurrent module to tackle the challenging motion and location modeling problem. Specifically, we progressively update the motion and location results from the concurrently updated 3D volume features. As the object motion and location results are tightly coupled in the recurrent stage, we also demonstrate the framework can naturally achieve 3D tracking.
Although DORT can better handle dynamic objects, we follow previous methods and simplify the motion with a constant velocity assumption. To handle various kinds of scenarios, this assumption could be relaxed with modeling acceleration or explicit trajectory prediction. Additionally, similar to current object-wise methods [24, 25, 49] (i.e. DaTR- baa-ss.ge-baseed, the computation is linearly correlated with the number of instances. To overcome this limitation, point-wise motion modeling and feature extraction with object-wise grouping will be considered future work.
Acknowledgement This project is supported by Shanghai Artificial Intelligence Laboratory.
8

---

References
[1] D. Park, R. Ambrus, V. Guizilini, J. Li, and A. Gaidon. Is pseudo-lidar needed for monocular
3d object detection? In ICCV, 2021.
[2] Li, H. Zhao, Liu, and Cao. Rtm3d: Real-time monocular detection from object keypoints for autonomous driving. In ECCV, 2020.
[3] Y. Zhang, J. Lu, and J. Zhou. Objects are different: Flexible monocular 3d object detection. In CVPR, 2021.
[4] Wang. X. Zhu, Pang, and Lin. Probabilistic and geometric depth: Detecting objects in perspective. In CoRL, 2021.
[5] T. Wang, J. Pang, and D. Lin. Monocular 3d object detection with depth from motion. In ECCV, 2022.
[6] Huang and G. Huang. Bevdet4d: Exploit temporal cues in multi- 3d object detection. arXiv preprint arXiv:2203. 17054, 2022.
[7] Z. Li, W. Wang. H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and J. Dai. Bevformer: Learning bird's eeye--ee representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022.
[8] R. Hartley and A. Zisserman. Multiple View Geometry in Computer Vision. Cambridge Uni- versity Press, New York, NY, USA, 2 edition, 2003. ISBN 0521540514
[9] Huang, G. Huang. Z. Zhu, Y. Ye, and D. Du. Bevdet: high-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint 12. 1790, 2021.
[10] X. Zhou, D. Wang, and P. Krähenbühl. Objects points. In arXiv preprint arXiv:1904.07850 2019.
[11] X. Ma, Y. Zhang, D. Xu, D. Zhou, S. Yi, H. Li, and W. Ouyang. Delving into localization errors for monocular object detection. CVPR, 2021.
[12] Y. Zhou, Y. He, H. Zhu, C. Wang, H. Li, and Q Jiang. Monocular 3d object detection: An extrinsic parameter free approach. In CVPR, 2021.
[13] G. Brazil and Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In ICCV, 2019.
[14] G. Brazil, G. Pons-Moll, X. Liu, and B. Schiele. Kinematic 3d object detection in monocular video. In ECCV, 2020.
[15] Wang. X. Zhu, Pang, and Lin. FCOS3D: Fully convolutional one-stage monocular 3d object detection. In ICCVW, 2021.
[16] Z. Liu, D. Zhou, F. Lu, J. Fang, and L. Zhang. Autoshape: Real-time aware monocular 3d object detection. In ICCV, 2021.
[17] Q. Lian, Li, and Chen. Monojsg: Joint semantic and geometric cost volume for monocular 3d object detection. In CVPR, 2022.
[18] T. Roddick, A. Kendall, and R. Cipolla. Orthographic feature transform for monocular 3d object detection. arXiv preprint 1.Xiv:1811.08//8 2018.
[19] Y. Wang, W.LL. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger. Pseudo- lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019.

---

[20] Y. You, Y. Wang. W.-L. Chao, D. Garg, G. Pleiss, B. Hariharan, M. Campbell, and K. Q. Weinberger. Pseudo-lidar++: Accurate depth for 3d object detection autonomous driving. arXiv preprint arXiv: 1906.06611 2019.
[21] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Q. Weinberger. Pseudo- lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019.
[22] Philion and S. Fidler. Lift, splat. shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020.
[23] Y. Li, Z. Ge, G. Yu, J. Yang, Z. Wang, Y. Shi, J. Sun, and Z. Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint vrXiv:2206.10022 2022.
[24] Y. Wang, V. Guizilini, T. Zhang, Y. Wang, H. Zhao, and M. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In CoRL, 2021.
[25] Y. Liu, T. Wang, X. Zhang, and Sun. Petr: Position embedding transformation for multi-view 3d object detection. In ECCV, 2022.
[26] P. Li, J. Shi, and S. Shen. Joint spatial-temporal optimization for stereo 3d object tracking. In CVPR, 2020.
[27] Y. Li, Н. Bao, Z. Ge, Yang, J. Sun, and Z. Li. Bevstereo: Enhancing depth esti- mation in multi-view 3d object detection with dynamic temporal stereo. arXiv preprint arXiv:2209. 10248, 2022.
[28] A. Bewley, Z. Ge, L. Ott, Ramos, and B. Upcroft. Simple online and realtime tracking. In ICIP, 2016.
[29] T. Yin, X. Zhou, and P. Krähenbühl. Center-based 3d object detection and tracking. CVPR, 2021.
[30] S. Chen, X. Wang, T. Cheng, Q. Zhang, C. Huang, and W. Liu. Polar parametrization for vision-based surround-view view 3d detection. wxXiv2220.110653 2022.
[31] Y. Shi, J. Shen, Sun, Y. Wang. J. Li, S. Sun, K. Jiang, and D. Yang. Sren3d: Sparse r-cnn 3d surround-view camera object detection and tracking for autonomous driving. arXiv:2206. 14451, 2022.
[32] J. Yang, E. Yu, Z. Li, X. Li, and W. Tao. Quality matters: Embracing quality clues for robust 3d multi-object tracking. arXiv:2208, 10976, 2022.
[33] T. Fischer, Y.-H. Yang. S. Kumar, M. Sun, and F. Yu. CC-3DT: Panoramic 3d object tracking via cross-camera fusion. In CORL, 2022.
[34] H.NN Hu, Y.-H. Yang, T. Fischer, T. Darrell, F. Yu, and M. Sun. Monocular quasi-dens: 3d object tracking. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[35] P. Li and J. Jin. Time3d: End-to-end joint monocular 3d object detection and tracking for autonomous driving. In CVPR, June 2022.
[36] Park, C. Xu, S. Yang. Keutzer, K. M. Kitani, M. Tomizuka, and W. Zhan. Time will tell: New outlooks and baseline for temporal multi-view 3d object detection. In The Eleventh International Conference Learning Representations, 2023.
[37] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[38] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In ICCV, 2017.
10

---

[39] Z. Teed and J. Deng. Raft-3d: Scene flow using rigid- eegidmmotion embeddings. In CVPR, 2021.
[40] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. CVPR, 2022.
[41] Lu, Z. Zhou, X. Zhu, H. Xu, and L. Zhang. Learning ego 3d representation as ray tracing. In ECCV, 2022.
[42] Y. Li, Y. Chen, Qi, Z. Li, Sun, and J. Jia. Unifying voxel-based representation with transformer for 3d object detection. In NearIPS, 2022.
[43] Y. Liu, J. Yan, Jia, S. Li, A. Gao, T. Wang, X. Zhang, and J. Sun. Petrv2: A unified framework for perception from multi-camera images. arXiv:2206.-125: 2022.
[44] T. Zhang, X. Chen, Y. Wang, Y. Wang, and H. Zhao. Mutr3d: multi-camera tracking frame- work via 3d-to-2d queries. arXiv preprint arXiv:2205.000 100613,,, 2022.
[45] Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
[46] X. Zhou, V. Koltun, and P. Krähenbühl. Tracking objects as points. In ECCV, 2020.
[47] Z. Pang, Z. Li, and N. Wang. Simpletrack: Understanding and rethinking 3d multi-object tracking. In CVPR, 2021.
[48] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[49] S. Ren, K. He, R. Girshick, and Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[50] Y. Yan, Y. Mao, and B. Li. Second: Sparsely embedded convolutional detection. Sensors, 18 (10):3337, 2018.
[51] Wang, Q. Lian, C. Zhu, X. Zhu, and W. Zhang. Mv-fcos3d++: Multi-view camera-only 4d object detection with pretrained monocular backbones. arXiv preprint arXiv:2207.12716. 2022.
[52] Pang, I.. Qiu, X. Li, H. Chen, Q. Li, T. Darrell, and Yu. Quasi-dense similarity learning for multiple object tracking. In CVPR, 2021.
[53] N. Marinello, M. Proesmans, and L. Van Gool. Triplettrack: 3d object tracking using triplet embeddings and Istm. In CVPRW, 2022.
[54] K. Wang and S. Shen. MVDepthNet: real-time multiview depth estimation neural network. In 3DV, 2018
[55] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan. Mvsnet: Depth inference for unstructured multi- view stereo. In ECCV,2015
[56] T. Taketomi, H. Uchiyama, and S. Ikeda. Visual slam algorithms: a survey from 2010 to 2016. IPSJ Transactions on Computer Vision and Applications, 9, 12 2017. doi:10. 186/ s41074-017-0027--.
[57] Y. Yao, Z. Luo, S. Li, T. Shen, T. Fang, and L. Quan. Recurrent mvsnet for high-resolution multi-view stereo depth inference, In CVPR, 2019.
[58] D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In CVPR, 2018.
[59] J. Sun. L. Chen, Y. Xie, S. Zhang, Q. Jiang, X. Zhou, and H. Bao. Disp r-cnn: Stereo 3d object detection via shape prior guided instance disparity estimation. In CVPR, 2020.

---

[60] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas. Normalized object
coordinate space for category-level 6d object pose and size estimation. In CVPR, 2019.
[61] C. Tang and P. Tan. BA-net: Dense bundle adjustment networks. In ICLR, 2019.
[62] Li, S. Liu, and S. Shen. Multi-sensor 3d object box refinement for autonomous driving. arXiv preprint arXiv: 1909.04942, 2019.
12

---

Supplementary Material for DORT: Modeling Dynamic Objects in Recurrent for Multi-Camera 3D Object Detection and Tracking
A Evaluation Metries
Detection Metrics We adopt the official evaluation protocol provided by nuScenes benchmark [48]. The official protocol evaluates 3D detection performance by the metrics of average translation eгтоr (ATE), average scale error (ASE), average orientation error (AOE), average velocity error (AVE), and average attribute error (AAE). Besides, also measures the mean average precision (mAP) with considering different recall thresholds. Instead of using 3D Intersection over Union (IoU) as the criterion, nuScenes defines the match by 2D center distance d on the ground plane with thresholds 1, m.. The above metrics are finally combined into nuScenes Detection Score (NDS).
Tracking Metrics Regarding the tracking metrics, the nuScenes benchmark mainly measures the average multi-object tracking accuracy (AMOTA), average multi-object tracking precision (AMOTP), and tracking recall. particular, AMOTA and AMOTP are the averages of multi- object tracking accuracy (MOTA) and multi-object tracking precision (MOTP) under different recall thresholds.
B Implementation Details
In the main paper, we have introduced our overall multi-camera object detection and tracking framework and the details of the proposed components, In this supplemental section, we present the details of the other basic modules.
B.1 Network Architecture
Our framework is built based on BEVDet and BEVDepth, and we follow them to design the basic modules.
2D Feature Extraction Given N multi-view images E RNXWxHx3i each frame, we use shared 2D backbone extract the corresponding features. We adopt the standard ResNet-50 [37] as the backbone and initialize with ImageNet pre-trained weights. Then we adopt a modified Feature Pyramid Network (FPN) [50] to extract the multiple-level features and the output 2D features downsampled with the ratio of I6 with channel size 256: Fpo R W. x256
View Transformation Our work is the same as BEVDet and BEVDepth which contains a 2D to 3D view transformation module. Specifically, we first leverage depth prediction head to predict the depth probability for each pixel. Then we lift the 2D features a 2.5D frustum space via out-product it with the depth probability. The depth probability range is set as [0m, 60m] with grid size 0.5m. With the 2.5D frustum features, the 3D features for each local volume camera intrinsic to project the 3D grid back to the frustum and bi-linear sample the corresponding features. As mentioned in the main paper, we aggregate the 3D volume features along the height dimension and obtain the corresponding object-wise BEV features bee E N x Hhhh x256 where Wobj and Hobj are the object features dimension and set as 28 in the main setting.
obtained via utilizing the
RefineNet Given the object-wise features extracted based on the proposal 3D box and motion, RefineNet takes several convolutional neural networks extract the object-wise features and esti- mate the bounding box and motion residual. Specifically, we first adopt an average pooling layer to aggregate the 3D features along the height dimension and obtain the BEV features. Then filter each wije BEV features with 6 basic 2D residual blocks, where each residual block consists
13

---

of two 2D convolution layers and a skip connection module as in ResNet. The channel size of the residual blocks the first three layers is 256 and decreases to 64 the last three layers. Then we aggregate the features along the spatial dimension via average pooling and take 4 layers MLP network to estimate the bounding box and motion residuals.
B.2 The Tracking Module
In this section, we provide the details of the tracking module that omit in the paper. Since DORT can estimate tightly coupled object location and motion, object tracking can easily achieved via nearest center distances association [28, 29, 47]. Hence, our tracking module is mainly adapted from the previous distance-based object tracker [28, 29, 47]. Specifically, the tracking module contains four parts: Pre pre-processing, Association, Status Update and Life-cycle Management.
Pre-processing Given the detection results, the pre- pre-processing stages mainly focus on filtering false negative objects. In our work, we first adopt non-maximum Suppression to remove the duplicated bounding boxes with the threshold of 1. in terms of 3D loU. Then we filter out the bounding boxes that the confidence threshold lower than 0.25.
The association stage associates the tracked objects in frame t-1 and the detection results in frame t. We don't oo use the Kalman filter to predict the location of the trackers from frame t-1. Instead, we utilize the predicted velocity to propagate the detection results in frame back to 1-1. Then, we utilize the L2 distances of object centers to compute the similarity between the detected objects and the tracklets. Finally, the linear greedy matching strategy adopted achieve multi-object matching. Status Update After associating the detection results frame we update the tracklets from frame t-1 into frame t. For the tracklets in frame t-1 that match with bounding boxes in frame t, we replace their object center location with the corresponding detection results to frame t. For the unmatched objects, we utilize the estimated object velocity to update its object center location to frame t. Life-cycle Management The life-cycle management module controls the "birth" and "depth" of the tracklets (i.e. birth, depth). Specifically, for the unmatched bounding boxes, they will be initialized as new tracklets. For the unmatched tracklets, we remove them when they are consecutive unmatched more than 2 times.
Association
Details of depth error calculation in Fig 2. The depth error in Fig2 is calculated as follows. The depth error is the Il distance between the ground truth depth and the depth obtained by assuming objects are static across frames. Specifically, the object depth that assumes objects are static is calculated as follows. We first utilize the ego-motion (camera extrinsic) and the ground-truth 2D location of 3D box centers in the past and current frames to obtain the cross framme correspondence. Then, utilize Eq2 obtain the corresponding object depth.
C More results on the Waymo dataset
In this section, we further provide the experimental results on the Waymo dataset for reference. We adopt ResNet-101 as the image backbone and train the model with 1/3 training data. Regarding initialization methods other than ImageNet pre-trained weights, we present additional results using the commonly used FCOS3D+ pre-trained weights on the Waymo dataset.
D Ablation Studies
In this section, we provide the additional ablation studies that are omitted in the main paper. DORT with Different Proposal Detector We first show that DORT is agnostic with different proposal detectors (e.g. PGD [4], BEVDepth [23]). In Table 4, we display the experimental results of DORT with using PGD and BEVDepth as the proposal detectors. We can observe that the DORT insen- sitive the proposal detector and can consistently improve BEVDepth. We Benefiting from the low computation overhead of BEVDepth in the perspective part and the designed local volume, DORT also can achieve a more lightweight pipeline for dynamic object modeling.
14

---

Table 3: Experimental results the Waymo validation ImageNet pre-trained denotes the supervised classification pre-training for the network backbone. FCOS3D++ on Waymo denotes the further pre-training of the model backbone via FCOS3D++ on the Waymo training dataset.
Method
Pre-traiedd
mAPL
mAP
FCOS3D++ [15]
20.4
28.6
DETR3D [24]
26.1
39.0
BEVDepth [23]
ImageNet
28.2
39.9
MV-FCOS3D++ [51]
28.7
39.9
Ours
30.1
42.3
MV-FCOS3D+ [51]
FCOS3D++
33.8
46.7
Ours
Waymo
35.0
48.9
Table Experimental results on nuScenes validation set. past frame is adopted in the temporal modeling. denotes the BEV FLOPS from the proposal detector.
Flops
Method
mAP NDS
PV
BEV
BeVDepth DORT with PGD
35.1
47.5
120.4
94.5
37.9
238.2
40.2
DORT with BEVDepth
38.1
52.1 120.4
74.4"+40.2
Tracking with Semantic Embedding or Geometry Distance In this work, DORT achieves 3D object tracking via the nearest centerness association. To have a more comprehensive comparison of the tracking pipeline designed, we further provide the comparison of DORT by using semantic embedding associate objects. Specifically, we follow previous methods [34] and adopt the widely- used quasi-dense similarity learning [52] learn the tracking embedding. We extract two kinds is from the perspective-view (PV) and another is from the bird-eye-
of embedding features, view (BEV). In Table 5 and 6, we display the tracking results the nuScenes tracking set. We can observe that DORT with geometry distance association can outperform the embedding-basee methods by a large margin. Furthermore, it is also much simpler and more efficient and does not need to maintain an extra object embedding. Besides, the PV embedding is worse than the BEV- based embedding, which may be due to the view change in different cameras.
Table Experimental results nuScenes validation set. 1 past frame adopted in the temporal modeling. AMOTP, MOTAR!
Method
AMOTA1
PV-Embedding
36.8
1.412
44.2
DEV-Embeddie
40.1
1.356
DORT (Gcometry Distance)
42.4
1.264
49,2
E Theoretical Analysis of Ignoring Object Motion
In the main paper, we have shown that when ignoring object motion, the temporal correspondence would derive biased depth. In this supplementary, we provide the full details of how ignoring object motion introduces a biased depth. We denote the camera intrinsic as K and the ego-motion from frame lo frame I as Teoo
f 0 Cu
1 0 800
K =
0 f Co
Teeo tg
010
(8)
0 0
1011 200
Here, f is the camera's focal length, and (cu, C.)) is the camera center coordinates the image. For 0, ",09,,0,2 290) on the horizontal plane. The analysis also can be easily extended to a more complicated case that the motion con- tains rotation. Given the multiple-view images, temporal-based methods can utilize photometric or feature-mett metric similarity to find the correspondence of pixel Pt, (ut, Vt,) in the past frame to and the pixel Pt, = (uey Vt,) in the current frame f
simplicity, we assume the ego-motion only contains the translation (æ*.0
15

---

Table 6: 3D object tracking results the nuScenes validation set. set the input resolution as 704 x 256.
We adopt ResNet-50 the backbone and
Method
AMOTAT
AMOTPL
Recall
QD-Track3D [34]
24.2
1,518
39.9
Time3D [35]
21.4
1,360
N/A
TriplerTrack [53]
28.5
1,485
N/A
MUTR3D [44]
29.4
1,498
42.7
Q'rackk [32]
34.7
1.347
46.2
DORT
42.4
1.264
49.2
When we ignore object motion, the depth Et, pixel Pt, can be recovered as:
Treoo
TT(to
K) = (Pt,, K),
ue, + Cu
Uso
Cu (ze. 2 200),
Zt1
x000
f
2200 Ca)
fxe
2tg
(9)
Wto
Uti
where æ denotes the projection from 2D image coordinate to 3D camera coordinate.
But showed in the main paper, the moving objects occupy large ratios in the driving scenarios. For example, when the object contains the translation (I. 0, zbi) in the horizontal plane, the object's is motion can be represented as
0 0
ob
0100 0 0
(10)
oby
With the object motion, the depth Fr, pixel pr, recovered as:
Toobj
tt
10-11;
(PPrs K) = K), (2t, 200 200))
2t f -00 ohi Utoo + Cu
ut Cu
2t (zeg0
z0b)) (uto cu) sw + zob))
(11)
Uto
Uti
From Eq (9) and Eq (11), we obtain the depth gap for the temporal correspondence with and without considering object motion:
obj (Uto c.) f0
A=
(12)
Uto
Ut.
In Figure 6, we also provide a toy example to illustrate that one temporal correspondence can come from multiple combinations of object depth and motion (i.e. inaccurate depth with zero motion and accurate depth and GT motion). This means that if inaccurately assume that objects static across frames, the temporal correspondence would derive misleading depth.
E.1 Ill-posed Problem of Simultaneously Estimating 3D Location and Motion
Although object motion plays a critical role in temporal correspondence, however, it is non-trivial to estimate it from the monocular images. As shown in Figure 6, the one correspondence can come from infinite combinations of location and motion (the location can be the point in the ray Pr an ill ill-posed
and O2, Pr and the motion can be the line that connects the points.) Hence, problem that simultaneously estimates the 3D location and motion from the monocular images. To alleviate this issue, we leverage the rigid-body assumption for the objects in the driving scenarios and elaborate more temporal frames with constant velocity regularization further constrain the motion.
16

---

Static Assumption
Candidate Depth
Inaccurate Motion
Motion
G1 Motion
Pi (Us. vi>
Current Frame
(as
Origin O
Past Frame
Origin O
Figure Different object motions make the temporal correspondence derive different depth.
More Related Work
Leveraging multi-view images to recover 3D information is a fun- damental topic, such as structure from motion [54], multi-view stereo [55], simultaneous calization and mapping [56], One line of methods develop neural-network-beee
Multi-View 3D Perception
cost vol- umes [55, 57, 58, 45, 39, 59] construct cross-frame visual cues for 31D perception. Another line of methods [60, 61, 62] constructs geometry constraints and leverages optimization techniques to obtain a tight-coupled 3D structure. However, most of the work assumes the scene and objects are static, making them fail to handle the moving objects in driving scenarios.
17


Reviews:
[{"review_id": "oCvePyy2Iu", "reviewer": "Reviewer_wd6g", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a multi-view object detection and tracking framework that is able to deal with dynamic objects. To this end it introduces per-object cost volumes and parameterizes their location as a function of time. A constant velocity model is assumed and the corresponding location and velocity parameters are refined iteratively. The multi-view object tracking results are competitive.", "summary_of_recommendation": "The paper is technically sound but the contribution is rather incremental.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* Clarify above questions\n* Remove/shorten chapter 3 and relax formatting\n* Discuss what happens if velocity constraints are relaxed", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, the paper is well structured. Some typos here and there could be improved. There are parts that are a bit unclear and this could be improved as well:\n* Multiple Estimation Fusion: looking at Fig.3 it looks like fusion across time happens on a feature level. But then it is stated that fusion happens on a residual level. Could you clarify maybe?\n* Further details on how tracking is done would be helpful (4.4). There is mention of a Kalman filter but details on prediction and update are missing. Also, in the tracking case, would one be able to get rid of the initial candidates and directly use the bounding boxes predicted by the filter?\n\nI am not sure how much value the analysis in chapter 3 adds. It's fairly clear that when the static assumption is violated that there will be an estimation error. And that this estimation error is linear in speed and time is quite intuitive. I would recommend leaving this away and rather improving formatting (it looks very squeezed) and provide more details w.r.t. method and results.\n\nDo you think the constant velocity assumption could be dropped and rather the location residual for all timesteps could be predicted? This would still require proper alignment of the bounding boxes across time frames. The geometric constraint across time would be loosened up, but features would still be shared across time and this would avoid errors caused when the velocity is not constant?\n\nThe employed method seems adequate and the results are competitive. My only concern is that there is no big novelty. Introducing velocity estimation for moving targets has been done before in 3D vision, e.g. even with moving per-object volumes (Xu 2019 - Mid-Fusion). There is still value in putting everything together though. Another small shortcoming is the limitation to planar motion and I am not even sure that a rotational component is included in the velocity. If, as the introduction claims, this is interesting for other domains than autonomous driving, then it would be interesting in knowing what happens if the velocity constraints are relaxed and estimation is done on the full 6 DOF.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "fkYR_z0RIE", "reviewer": "Reviewer_fWSU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a multi-camera 3d object detection and tracking method. The SOTA methods use global feature volumes for this task with an added assumption of the scene of interest being static. This assumption fails for the scenes with dynamic objects. The authors propose to address this issue with object-wise local feature volumes and further estimate the motion and location of the objects using a recurrent module. The experiments were conducted on Nuscenes detection and tracking benchmarks where they were able to achieve 57.6% and 62.8% in AMOTA, and NDS metrics respectively.", "summary_of_recommendation": "This paper proposes to address the problem of dynamic objects in 3D objection detection and tracking using an approach that uses local-feature volumes and iterative refinement of object motion and bounding boxes using a recurrent module. The improvements of the detection module seem marginal but the tracking results are good in all the metrics. However, the current version of the paper needs some more experiments to validate their claims. \n\n\nPost rebuttal, my concerns are addressed and I am happy to change the recommendation to weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Please address the above-mentioned comments.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The problem statement is well articulated with a good explanation in Section 3 with a nice visualization from Figure 2. The proposed method is technically sound and has proven to show better results in some of the metrics of detection and in all the metrics of tracking compared to SOTA. \n\nPlease find my comments for the authors to address in the rebuttal. \n\n1. The proposed method is trained and tested on only the nuScenes dataset. I would like to see more results on the other dataset in the main paper. \n\n2. The authors argue that the proposed method alleviates the heavy computational burden that comes with global BEV methods [lines 8 - 9 ]. I would like to see some experiments or discussions regarding this. \n\n3. Also the authors suggest that the proposed method can be plugged into various existing methods [lines 12-13]. I am keen to see some experiments or discussion regarding that.\n\n4. The authors attribute a 0.5% improvement in mAP as significant. Also, I am not able to get this relative improvement number (both 0.5% and 7.4% from line 275) from Table 1. Please explain the calculations.\n\n5. What do the authors mean by \"right-body movement\" in Line 41, 135? Is it supposed to be rigid-body?\n\n6. I think Figure 5 is mentioned as Table 5 in Line 294.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "Ae5S_z-7v7S", "reviewer": "Reviewer_RSf9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose DORT, a method that estimates both object motion and location which are recurrently refined for better 3D object detection and tracking. They demonstrate empirically and theoretically that ignoring objects, especially when the motion varies between objects, leads to a biased depth estimation, harming the performance detectors.", "summary_of_recommendation": "I’m fairly convinced with the claims in this paper of the necessity of object motion estimation for better detection and tracking. While this seems applicable in robotics, the main application in this paper is for driving scenarios and the evaluation setting is limited to a single dataset. Overall the paper is clearly written and well-motivated and I believe that extending the evaluation setting will benefit this paper, in addition to answering my questions under “Questions for Rebuttal” and additional discussion on limitations (even if it ends up in the Appendix).\n\nPost Rebuttal: I feel that most of them were answered. \n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "* In Figure 2, how is the empirical depth error calculated?\n* In addition to the graphs in Figure 5, what is the performance-computation complexity trade-off w.r.t the number of refinement iterations $k$?\n* While initializing with pre-trained ImageNet weights is a legitimate approach, have you experimented with training from scratch or using pre-trained self-supervised weights?\n* Regarding the limitation I raised in “Weaknesses” above regarding occlusions, do you have an experiment/intuition how your model will handle occlusions? Are there other limitations inherited from previous methods?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n* \\+ the paper provides a better understanding of the necessity of working locally by estimating objects motion and location instead of globally.\n* \\+ the authors promise an open-source code.\n* \\+ the idea is rather simple and leads to better performance.\n* \\+ computationally more efficient than global-features methods.\n* \\+ convincing ablation study (+appendix) on the NuScenes dataset.\n\nWeaknesses/Limitations:\n* \\- limited evaluation setting, the model is only evaluated on a single dataset.\n* \\- no demonstration in a robotic domain.\n* \\- an additional limitation that is not addressed is handling occlusions.\n\nMinor Comments:\n* I think Figure 1 should be refined, as it is hard to understand from it the difference between previous methods and the proposed one. Also, in lines 132-134, the authors mention that this figure illustrates that “the correspondence of two points can come from infinite combinations of object location and motion” - I’m not confident this is indeed demonstrated in figure in its current form.\n* Equation 4: I think $fx^{\\text{ego}}$ should be $fx^{\\text{obj}}$ in the resultant $\\Delta z$.\n\nPost Rebuttal:\n\nI acknowldege that I have read the authors' rebuttal and the other reviewes. Thank you for the additional details and for addressing my concerns, I feel that most of them were answered. I'll keep my score, but as the other reviewers also raised important concerns, I will not argue my case if the other reviewers reach a different recommendation.\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "5xIaRK0SXa", "reviewer": "Reviewer_VMsm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a 3D object detection and tracking algorithm with multiview videos in the scenarios of autonomous driving. The key improvement is the algorithm's ability to aggregate multiview features for each candidate object while accounting for object motions. The author compares the proposed method with multiple current SOTA methods like BEVDepth, Pseudo-LiDAR R-V2, and UV-RCNN on the nuScenes dataset, demonstrating impressive performance improvements of 1.1 NDS and 5.7 AMOTA.", "summary_of_recommendation": "This paper presents an interesting algorithm for multiview, multi-frame 3D object detection and tracking. It improves upon the current state-of-the-art (SOTA) on the nuScenes dataset. The paper is also well-written. Though I think there are a couple of improvements that can be made to this paper (as mentioned in the question part), but overall, I feel it is acceptable.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Most of my questions are mentioned in the weakness section, but let me summarize them here:\n\n* What is the performance of the proposed methods with larger backbones or on the Waymo dataset?\n* Can the author provide more metrics about parameter count, memory cost, and computational cost (FLOPS might be sufficient if the author cannot measure the inference speed)?\n* Can the author provide more analysis on how the first-stage detector affects the final results?\n* I am also curious about whether the proposed method can be easily extended to an online setting. (It's okay if this question isn't answered; I'm just curious.)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strength\n\n* The paper is well written and easily conveys its contributions. The descriptions of the object-wise local volume and recurrent dynamic object modeling are clear and intuitive.The algorithm elegantly combines two-stage detectors with recurrent dynamic modeling. Also, the tracking variant is quite seamlessly built from the detectors. This algorithm design leads me to wonder how it performs in an online setting? Also, can it be easily changed into an autoregressive way?\n\n* Some experimental designs are quite interesting. For example, the experiment shown in Figure-5 validates both the effectiveness of the feature aggregation strategy and the motion prediction.\n\n* The experimental results appear strong. While browsing over the leaderboard of nuScenes detection and tracking, it is evident that the current algorithm achieves SOTA results under the ConvNeXt-B or ResNet setting.\n\nWeakness\n\n* Although the experimental results look good, I believe the author might need to add more experiments to solidify the findings. This could include extending to larger backbones or adding the Waymo dataset. Larger backbones might be essential as the current leaderboard results are all based on them. Tests with large backbones can validate the scalability (including computational cost, memory cost, and performance growth) of the proposed algorithm.\n\n* The paper does not list any metrics on parameters, computational cost, or memory cost, yet it claims efficiency using object-centric features rather than constructing an entire BEV. The author needs numerical evidence to support this claim.\n\n* Another minor concern is about the first stage detector used in this paper. The author seems to use BEVDepth only; can the author provide more analysis on the first-stage detectors and their impact on the final performance?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a a multi-camera 3D object detection and tracking method which leverages local volumes for motion estimation while accounting for object motions. The proposed method introduces some novelty in multi-view dynamic object tracking. The reviewers have mainly raised questions about the limited experimentation, handling of occlusions, list of metrics, etc. Most concerns have been addressed by the authors. The camera ready method should definitely include the additional experiments provided in the rebuttal with sufficient explanations, a description of experimental protocol and metrics, as well as, a broader limitations discussion based on the comments of the reviewers.
Decision: Accept (Poster)