DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
Authors: Yue Wang
Vitor Campagnolo Guizilini
Tianyuan Zhang
Yilun Wang
Hang Zhao
Justin Solomon
Venue: CoRL2021 Poster (2021)
Keywords: multi-camera
3D object detection
autonomous driving

Abstract: We introduce a framework for multi-camera 3D object detection. In contrast to existing works, which estimate 3D bounding boxes directly from monocular images or use depth prediction networks to generate input for 3D object detection from 2D information, our method manipulates predictions directly in 3D space. Our architecture extracts 2D  features from multiple camera images and then uses a sparse set of 3D object queries to index into these 2D features, linking 3D positions to multi-view images using camera transformation matrices. Finally, our model makes a bounding box prediction per object query, using a set-to-set loss to measure the discrepancy between the ground-truth and the prediction. This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per-pixel depth estimation, since it does not suffer from the compounding error introduced by a depth prediction model. Moreover, our method does not require  post-processing such as non-maximum suppression, dramatically improving inference speed. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.

DETR3D: 3D Object Detection from Multi-view Images via 3D-to-2D Queries
Yue Wang
Vitor Guizilini Toyota Research Institute vitor, guizilini@tri.eee
Massachusetts Institute Technology yuewangecsail .mt.meite
Tianyuan Zhang
Yilun Wang
Carnegie Mellon University
Auto yi yilunwêcs stanford.edu
cianyuaz@andrev. cmu. edu
Hang Zhao Tsinghua University hangzhao0mail. tsinghua. edu. en
Justin Solomon Massachusetts Institute of Technology jsolomon@mit.ed
Abstract: We introduce framework for multi-camera 3D object detection. In contrast to existing works, which estimate 3D bounding boxes directly from monocular images or use depth prediction networks generate input for 3D ob- ject detection from 2D information, our method manipulates predictions directly 3D space. Our architecture extracts 2D features from multiple camera images and then uses sparse set of 3D object queries to index into these 2D features, linking 3D positions multi-view images using camera transformation matrices. Finally, our model makes a bounding box prediction per object query, using set-to-set loss to measure the discrepancy between the ground-truth and the pre- diction. This top-down approach outperforms its bottom-up counterpart in which object bounding box prediction follows per- per-pixel depth estimation, since it does not suffer from the compounding error introduced by depth prediction model. Moreover, our method does not require suppression, dramatically improving inference speed. We achieve state-of-the-art performance on the nuScenes autonomous driving benchmark.
such as non-maximum
Introduction
3D object detection from visual information is long-standing challenge for low-cost autonomous driving systems. While object detection from point clouds collected using modalities like LiDAR benefits from information about the 3D structure of visible objects, the camera-based setting even more ill-posed, since must generate 3D bounding box predictions solely from the 2D information contained in RGB images.
Existing methods [1, 2] typically build their detection pipelines purely from 2D computations. That is, they predict 3D information like object pose and velocity using an object detection pipeline de- signed for 2D tasks (e.g., CenterNet FCOS [3]), without considering 3D scene structure or sensor configuration. These methods require several post-processing steps fuse predictions across cam- eras and to remove redundant boxes, yielding a steep trade-off between efficiency and effectiveness. As an alternative these 2D-based methods, some methods incorporate more 3D computations into object detection pipeline by applying 3D reconstruction method like [4, 5, 6] to create pseudo-LiDAR or range input the scene from camera images. Then, they could apply 3D object detection methods to this data if it were collected directly from a 3D sensor. This strategy, how- ever, subject to compounding errors [7]: poorly-estimated depth values have strongly negative effect on the performance of 3D object detection, which also exhibit errors of its own.
*: Equal contribution. 9: Co-advise the project.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

In this paper, we propose a more graceful transition between 2D observations and 3D predictions for autonomous driving, which does not rely on module for dense depth prediction. Our framework, termed DETR3D (Multi- View 3D Detection), addresses this problem in a top-down fashion. We link 2D feature extraction and 3D object prediction via geometric projjeroettion with camera transfor- mation matrices. Our method starts from sparse set of object priors, shared across the dataset and learned end-to-end. To gather scene-specific information, we back-projec a set of reference points decoded from these object priors to each camera and fetch the corresponding image features extracted by a ResNet backbone [8]. The features collected from the image features of the reference points then interact with each other through multi-head self-attention layer [9]. After series of self-attention layers, we read off bounding box parameters from every layer and use a set-to-set loss inspired by DETR [10] to evaluate performance.
Our architecture does not perform point cloud reconstruction or explicit depth prediction from im- ages, making it robust to errors in depth estimation. Moreover, our method does not require any post-processing, such as non-maximum suppression (NMS), improving efficiency and reducing re- liance on hand-designed methods for cleaning its output. On the nuScenes dataset, our method (without NMS) is comparable with prior art (with NMS). In the camera overlap regions, our method significantly outperforms others.
Contributions. We summarize our key contributions as follows:
We present a streamlined 3D object detection model from RGB images. Different from existing works that combine object predictions from the different camera views in final stage, our method fuses information from all the camera views in each layer of computation. To the best of our knowledge, this is the first attempt cast multi-camera detection as 3D set-to-set prediction. We introduce a module that connects 2D) feature extraction and 3D bounding box prediction via backward geometric projection. It does not suffer from inaccurate depth predictions from sec- ondary network, and seamlessly uses information from multiple cameras by back projeproje 3D information onto all available frames.
Similarly to Object DGCNN [1 I., our method does not require post-processing such as per-image or global NMS, and it is on par with existing NMS-based methods. In the camera overlap regions, our method outperforms others by substantial margin.
We release our code to facilitate reprodueibility and future research.
2 Related Work
2D object detection. RCNN [12] pioneered object detection using deep learning. It feeds a set of pre-selected object proposals into convolutional neural network (CNN) and predicts bounding box parameters accordingly. Although this method exhibits surprising performance, it is an order of magnitude slower than others because it performs a ConvNet forward pass for each object proposal. To fix this issue, Fast RCNN [13] introduces shared learnable CNN to process the entire image at single forward pass. To further improve performance and speed, Faster RCNN [13] includes a region proposal network (RPN) that shares full-image convolutional features with the detection network, thus enabling nearly cost-free region proposals. Mask RCNN [14] incorporates a mask prediction branch to enable instance segmentation in parallel. These methods typically involve multi-stage refinements and can be slow in practice. Different from these multi-stage methods, SSD [15] and YOLO [16] perform dense predictions in single shot. Although they are significantly faster than the alternatives above, they still rely on NMS to remove redundant box predictions. These methods predict bounding boxes w.r.t. pre-defined anchors. CenterNet [1] and FCOS [3] change the paradigm by shifting from per-an per-anchor prediction to per- per-pixel prediction, significantly simplifying the common object detection pipeline.
Set-based object detection. DETR [10] casts object detection as a set-to-set problem. employs a Transformer [9] to capture feature and object interactions. DETR learns to assign predictions to set of ground-truth boxes; thus, it does not require post post-processing to filter out redundant boxes. One critical drawback of DETR, however, is that it requires a significant amount of training time. De- formable DETR [17] analyzes DETR's slow convergence and proposes a deformable self-attention module to localize features and accelerate training. Concurrently, [18] attributes the slow conver- gence of DETR to the set- base loss and the Transformer cross attention mechanism. They pro-
https: httpaa//.ceeeaaage//eeeeteeeee
2

---

sing
Use more mage herures
Co
Prrn
Canoro Pustiais
lrage
Dieet Machhng
Figure 1: Overview of our method. The inputs the model are set of multi-view images, which are encoded by a ResNet and a FPN. Then, our model operates on a set of sparse object queries 3D reference point. 2D features transformed refine
in which each query is decoded the object queries by projecting the 3D reference point into the image space. Our model makes per-query predictions and uses set-to-set loss.
pose two variants, TSP-FCOS and TSP-RCNN, to overcome these problematic aspects. SparseR- CNN [19] incorporates set prediction into RCNN-style pipeline; it outperforms multi-stage object detection without NMS. OneNet [20] studies an interesting phenomenon: dense-based object detec- tors can be made NMS-free after they are equipped with minimum-coss set loss. For 3D domains, Object DGCNN [11] studies 3D object detection from point clouds. It models 3D object detection message passing a dynamic graph, generalizing the DGCNN framework to predict set of objects. Similar to DETR, Object DGCNN is also NMS-free.
Monocular 3D object detection. An early method for 3D detection from RGB images is Mono3D [21], which uses semantic and shape cues select from collection of 3D proposals, using scene constraints and additional priors at training time. [22] uses the birds-eye-view (BEV) for monocular 3D detection, and [23] leverages 2D detections for 3D bounding box regression via the minimization of 2D-3D projection error. The use of 2D detectors as starting point for 3D computation recently has become standard approach [24, 25). Other works also explore advances in differentiable rendering [26] 3D keypoint detection [27, 28, 1] to enable state-of- the-art 3D object detection performance. All these methods operate in a monocular setting, and extensions to multiple cameras done by independently processing each frame before merging the outputs in post-processing stage.
3
Multi-view 3D Object Detection
3.1
Overview
Our architecture inputs RGB images collected from a set cameras whose projection matrices (the combination of intrinsics and relative extrinsics) are known, and it outputs set of 3D bounding box parameters for the objects in the scene. In contrast to past approaches, we build our architecture based few high-level desiderata:
We incorporate 3D information into intermediate computations within our architecture, rather than performing purely computations the image plane.
We do not estimate dense 3D scene geometry, avoiding associated reconstruction errors.
We avoid post- procesrocessing steps such as NMS.
We address these desiderata using a new prediction module, which links 2D feature extrac- tion and 3D box prediction by alternating between 2D and 3D computations. Our model contains three critical components, illustrated Figure First, following common practice in 2D vision, it extracts features from the camera images using a shared ResNet [8] backbone. Optionally, these features are enhanced by a feature pyramid network (FPN) [29] ($3.2). Second, detection head ($3.3) our main contribution-linee the computed 2D features to a set of 3D bounding box predic- tions in a geometry-aware manner ($3.3). Each layer of the detection head starts from a sparse set of object queries, which are learned from the data. Each object query encodes 3D location, which

---

is projected to the camera planes and used to collect image features via bilinear interpolation. Simi- larly to DETR [10], we then use multi-head attention [9] to refine the object queries by incorporating object interactions. This layer is repeated multiple times, alternating between feature sampling and object query refinement. Finally, we evaluate a set-to-set loss [30, 10] to train the network ($3.4).
3.2 Feature Learning
Our model starts with a set of images I = fim....... immg } C Him x Win x3 (captured by sur- rounding cameras), camera matrices T (Ti...., Tk] 33x4 ground-truth bounding boxes B = (b)..... bj.... ... bM} } C RR, and categorical labels C = fj.............................. cM C Z. Each by contains position, size, heading angle, and velocity in the birds-eye view (BEV); our model aims to predict these boxes and their labels from the these images. We do not use point clouds, which are usually captured by high-end LiDAR.
These images are encoded with ResNet [8] and a FPN [29] into four sets of features F1. F3. 3. F4. Each set F& = {f f1...... f C "x XC corresponds to a level of features of the 6 images. These multi-scale features provide rich information to recognize objects of different sizes. Next, we detail approach to transform these 2D features into 3D using a novel set prediction module.
3.3 Detection Head
Existing methods for detecting objects from camera input typically employ bottom-up approach, which predicts dense set of bounding boxes per image, filters redundant boxes between the images, and aggregates predictions across cameras in a post-processing step. This paradigm has two crucial drawbacks: dense bounding box prediction requires accurate depth perception, which itself is a challenging problem; and MMSe-baed redundancy removal and aggregation are non- non-parallelizable operations that introduce significant inference overhead. We address these issues using top-down object detection head described below.
Analogously to [11, 17]. DETR3D is iterative; it uses L layers with set-based computations to produce bounding box estimates from 2D feature maps. Each layer includes the following steps:
1. predict a set of bounding box centers associated with object queries;
2. project these centers into all the feature maps using the camera transformation matrices; sample features via bilinear interpolation and incorporate them into object queries; and
4. describe object interactions using multi-head attention.
b- 16-1) 1} operates on a set of object queries Q Motivated by DETR [10], each layer & E Lo,
(qn..... gess- C RC, producing a new set 2++1. A reference point cu E IR3 is decoded from a object query qe follows:
+ rel (q::))
Ct
where &rof is a neural network. ct can be thought of hypothesis for the center of the i-th box. Next, we acquire image features corresponding to ca to refine and predict the final bounding box. Then, Ce (or more accurately its homogeneous counterpart ct,)) projected into each one of the images using the camera transformation matrices:
c₁ Ces ④
Cêmi = Ci:
(2)
where @ denotes concatenation, and Crmi is the projection of the reference point onto the m-th camera. To remove the effects of the feature map size and gather features across different levels, we normalize Ctmi to [-1, 1]. Next, the images features collected by
I f = billinar (Fim, Ctmi),
(3)
where f Jekms is the feature for i-th point from k-th level m-th camera at f-th layer.
A given reference point is not necessarily visible in all the camera images, so we need some heuris- ties to filter invalid points. To that end, we define a binary value Jtmms which is determined based on whether a reference point is projected outside an image plane. The final feature f and object query in next layer q(++1)s are given by
fn = Es Jeems + E ΣΣ Fleme Otems
1
and
q = fts + 9
(4)

---

where e is a small number to avoid division by zero. Finally, for each object query qx we predict bounding box by: and its categorical label Ees with two neural networks te and wrh:
bes
11 (q:)
and
č = 0 (qn)).
(5)
We compute the loss for the predictions Br (ba...... bey.... beM- C R and é; {ūrI,..... Ctj:.... C C Z from every layer during training. During inference, we only use the outputs from the last layer.
3.4 Loss
Following [30, 10], we use set-to-set loss to measure the discrepancy between the prediction set (B,,Ce) and the ground-truth set (B5 C). This loss consists of two parts: a focal loss [31] for the class labels and a I,- loss for the bounding box parameters. For notational convenience, we drop the subscript in B, and Č₁. The number ground-truth boxes M is typically smaller than the number of predictions M so we pad the set of ground-truie truth boxes with Øs (no object) up to M for ease of computation. We establish a correspondence between the ground-trun -ruthh and the prediction via a bi- partite matching problem: o arg mingeP Σ5=1 (e, Pa(s)) (cx) + (c, Sbox (bj, bo()),
where P denotes the set of permutations, (o())((g)) is the probability of class cj for the prediction with index oj)), and Lbox is the L1 loss for bounding box parameters. We use the Hungarian algorithm [32] to solve this assignment problem, as in [30, 10], yielding the set-to-set loss Laup Σj= log Po (2) ())()) (cg) + 1 (e,#0) hhox b bo-()).
4
Experiments
We present our results as follows: first, we detail the dataset, metrics, and implementation in $4.1; then we compare our method to existing works in $4.2; we benchmark the performance of different models in camera overlap regions in $4.3; we compare to a forward prediction model in $4.4; and provide additional analysis and ablations $4.5.
4.1 Implementation Details
Dataset. We test method the nuScenes dataset [33]. nuScenes consists of 1,000 quences; each sequence is roughly 20s long, with a sampling rate of 20 frames/second. Each sam- ple contains images from 6 cameras [front_lefe bach_right]. Camera parameters including intrinsics and extrinsics are available. nuScenes pro- vides annotations every 0.5s; in total there are 28k, 6k, and 6k annotated samples for training, vali- dation, and testing, respectively. 10 from the total 23 classes are available to compute the metrics.
front, front right,
back,
Metrics. We follow the official evaluation protocol provided by nuScenes. We evaluate average translation error (ATE), average scale error (ASE), average orientation error (AOE), average velocity error (AVE), and average attribute error (AAE). These metrics true positive metries (TP metrics) and computed in the physical unit. In addition, we measure mean average precision (mAP). To capture all aspects of the detection task, consolidated scalar metric-the nuScenes Detection Score (NDS) [33]-is defined as NDS 1 [5mAP + EmtreTr( (1 min(1, (1, mT))).
TO
Model. Our model consists of ResNet [8] feature extractor, FPN, and DETR3D detection head. We use ResNet101 with deformable convolutions [34] in the 3rd stage and 4th stage. The FPN [29] takes features output by the ResNet and produces 4 feature maps whose sizes are 1/s, 1/16, 1/32, and 1/61 of the input image sizes. The DETR3D detection head consists of 6 layers, where each layer is a combination of feature refinement step and multi-head attention layer. The hidden dimension of the DETR3D detection head 256. Finally, two sub-networks predict bounding box parameters and class label per object query; each sub-network consists of two fully connected layers with hidden dimensions 256. We use LayerNorm [35] in the detection head.
Training& & inference. We use AdamW [36] to train the whole pipeline. The weight decay is 10-1, We use an initial learning rate 10-4 which decreased to 10-5 and 10-6 8th and 11th epochs.
The model is trained for 24 epochs in total on 8 RTX 3090 GPUs and the Per-GPU batch size is 1.
5

---

Table 1: Comparisons to recent works on the validation set. Our method is robust to the usage of NMS. *: CenterNet uses a customized backbone DLA [38]. f: this model is trained with depth weight 1.0 and initialized from FCOS3D checkpoint; the checkpoint is trained on the same dataset with depth weight 0.2. 3: with test-time augmentation. 4: with test-time augmentation, more epochs, and model ensemble. For details, see [2]. t: our model is also initialized from a FCOS3D backbone;
detection head is initialized randomly. #: trained with CBGS [39]
Method
NDS 1
MAP1 1
MATE
MASE
mAOE
mAVE
mAAE
NMS
CenterNet
0.328
0.306
0.716
0.264
0.609
1,426
0,658
FCOS3D
0.373
0.299
0.785
0.268
0.557
1.396
0.154
FCOS3D fo
0.393
0.321
0.746
0.265
0.503
1.351
0.160
FCOS3D g
0.402
0.326
0.743
0.259
0.441
1.341
0.163
FCOS3D
0,415
0.343
0.725
0.263
0.422
1.292
0.153
DETRAD (Ours)
0.374
0.303
0.860
0.278
0.437
0.967
0.235
DETR3D (Ours)
0.425
0.346
0.773
0.268
0,383
0.842
0.216
DETR3D (Ours) #
0.434
0.349
0.716
0.268
0.379
0.842
0.200
Table 2: Comparisons to top-performing works on the test set from the leaderboard. #: initialized from a DD3D checkpoint. f: initialized from a backbone pre-trained on extra data.
Method
NDS 1
mAP
MATE
mASE
mAOE +
mAVE
MAAE
NMS
Mono3D
0.429
0.366
0.642
0.252
0.523
1.591
0.119
N/A
DHNet
0.437
0.363
0.667
0.259
0.402
1.589
0.120
N/A
PGD [40]
0.448
0.386
0.626
0.245
0.451
1.509
0.127
DD3D [37] f
0.477
0.418
0.572
0.249
0.368
1.014
0.124
DETR3D (Ours)
0.479
0.412
0.641
0.255
0.394
0.845
0.133
The training procedure takes roughly 18 hours. We do not use any post-processing such as NMS during inference. For evaluation, we use the nuScenes evalutation toolkit.
4.2 Comparison to Existing Works
We compare to previous state-of-the-ard methods CenterNet [1] and FCOS3D [2]. CenterNet is an anchor-free 2D) detection method that makes dense predictions in high resolution feature map. FCOS3D employs a FCOS [3] pipeline to make per-pixel predictions. These methods both turn 3D) object detection into 2D problem, and in doing ignore scene geometry and sensor configuration. To perform multi-view object detection, these methods have to process each image independently, and use both per-image and global NMS remove redundant boxes in each view and in the overlap regions respectively. As shown in Table 1, our method outperforms these methods even though we do not use any post-processing. Our method performs worse than FCOS3D in terms of mATE. We suspect this is because FCOS3D directly predicts bounding box depth, which leads to strong supervision on object translation. Also, FCOS3D uses disentangled heads for different bounding box parameters, which can increase performance.
On the test set (Table 2), method outperforms all existing methods as of 10/13/2021; our method uses the same backbone as DD3D [37] for fair comparison.
4.3 Comparison in Overlap Regions
A great challenge lies in the overlap regions where objects are more likely to be cut off. Our method considers all cameras simultaneously, while FCOS3D predicts bounding boxes per camera individ- ually. To further demonstrate the advantages of fused inference, we calculate the metrics for boxes falling into the camera overlaps. To compute the metrics, we select boxes whose 3D center is visible to multiple cameras. On the validation set, there are 18,147 such boxes, 9.7% of the total. Table 3 shows the results; our method outperforms FCOS3D remarkably terms of NDS scores in this setting. This confirms that our integrated prediction approach is more effective.

---

Table 3: Comparisons in Overlap Region. f: this model is trained with depth weight 1.0 and initial- ized from a FCOS3D checkpoint; the checkpoint trained on the same dataset with depth weight 0.2. For details, see [2]. t: our model is also initialized from FCOS3D backbone; the detection head is initialized randomly.
Method
NDS T
mAP
MATE
MASE.
MAOE...
mAVE +
mAAE +
NMS
FCOS3D
0.317
0.213
0.841
0.276
0.604
1.122
0.173
FCOS3D
0.329
0.229
0.816
0.272
0.571
1.084
0.195
DETR3D (Ours)
0.356
0.231
0.825
0.280
0,400
0.863
0.223
DETR3D (Ours) |
0.384
0.268
0.807
0.273
0.453
0.788
0.184
Table 4: Comparisons to pscudo-LiDAR Methods.
Method pseudo-LiDAR
NDS 1
mAP
mATE
mASE
mAOE
mAVE |
mAAE
NMS
0.160
0.048
DETR3D (Ours)
0.374
0.303
0,860
0.278
0.437
0.967
0.235
4.4
Comparison to pseudo-LiDAR Methods
Another way to perform 3D object detection is by generating pscudo-LiDAR point clouds from multi-view images using a depth prediction model. On the nuScenes dataset, there are no publicly available pseudo-LiDAK works for to make direct comparison. Hence, implement a base- line ourselves to verify that our approach more effective than explicit depth prediction. We use pre- pre-trained PackNet [6] network to predict dense depth maps from all six cameras and then vert these depth maps into point clouds using the camera transformations. We also experimented with a self-supervised PackNet model with velocity supervision (as in the original paper), but found that ground-truth depth supervision yielded more realistic point clouds and therefore used a supervised model baseline. For 3D detection, employ the Sicently-proposee CenterPoint chitecture [41]. Conceptually, this pipeline is variant of pseudo-LiDAR [42]. Table 4 shows the results; we conclude that this pseudo-LiDAR method underperforms ours significantly even when depth estimates are generated by state-of-the-ee model. One possible explanation is that pseudo- LiDAR object detectors suffer from compounding errors introduced by inaccurate depth prediction, that in turn is known to overfit to training data and generalizes poorly to other distributions [7].
4.5 Ablation & Analysis
We provide visualization of object query refinement in Figure 2. We visualize bounding boxes decoded from the object queries in each layer. The predicted bounding boxes get closer to the ground-truth as we go into deeper layers in the model.Also, the lefimost figure shows the learned object query priors shared by all data. We also provide quantitative results in Table 5, which shows that iterative refinement indeed improves performance significantly. This suggests that iterative refinement is both beneficial and necessary to fully leverage our proposed architecture. Furthermore, we provide ablations on the number of object queries in Table 6; increasing the number queries consistently improves the performance until it gets saturated at 900. Finally, Table 7 shows the results with different backbones.
5 Conclusion
We propose new paradigm to address the ill-posed inverse problem of recovering 3D information from 2D images. In this setting, the input signal lacks essential information for models to make effective predictions without priors learned from data. While other methods either operate solely on 2D computations or use additional depth networks to reconstruct the scene, ours operates in 3D space and uses backward projection to retrieve image features as needed. The benefits of our approach are two-fold: (1) it eliminates the need for middle-level representations (e.g., predicted depth maps or point clouds), which can be source of compounding errors; and (2) uses information from multiple cameras by projecting the same 3D point onto all available frames.
7

---

Table 5: Evaluation detection results from different layers.
Layer
NDS 1
mAP 1
mATE
MASE
mAOL
mAVE
mAAL
0
0.380
0.302
0.855
0.280
0.435
0,910
0.231
0.410
0.335
0.791
0.275
0,408
0.887
0.217
2 0.420
0.343
0.782
0.271
0.395
0.851
0.214
3
0.420
0.346
0.778
0.268
0.390
0.874
0.218
4
0.424
0,346
0.777
0.268
0.389
0.855
0.217
5
0.425
0.346
0.773
0.268
0.383
0.842
0.216
Layer
Layer 2
Layer 3
IOA TCP
usir m0
Layer 4
Layer 5
Ground-truth
Figure 2: Detection results from layer I to layer 5 in the DETR3D head. We visualize the bounding boxes in the BEV and overlay the point clouds from lidar lip. The predictions get closer to the ground-truth in the deeper layers.
Table 6: Results with different number of queries.
# queries
30
100
300
600
900
1200
1500
mAP f
0.201
0.313
0.338
0.347
0.346
0.340
0.346
NDS T
0.331
0.408
0.415
0.420
0.425
0.415
0.420
Table 7: Results with different backbones.
Backbone
NDS
mATE
mASE
mAOE
mAVE
mAAE
ResNet50
0.373
0.302
0.811
0.282
0.493
0.979
0.212
ResNet101
0.425
0.346
0.773
0.268
0.383
0.842
0.216
DLA34
0.394
0.312
0,829
0.276
0.450
0.844
0.221
Beyond the direct application of our work to 3D object detection for autonomous driving, there are several venues that warrant future investigation. For example, single point projection creates limited receptive field in the retrieved image feature maps, and sampling multiple points for each object query would incorporate more information for object refinement. Furthermore, the new detec- tion head is anput-agnostic, and including other modalities such as LiDAR/RADAR would enhance performance and robustness. Finally, generalizing our pipeline to other domains such indoor nav- igation and object manipulation would increase scope of application and reveal additional ways for further improvement.
8

---

Acknowledgement
The MIT Geometric Data Processing group acknowledges the generous support of Army Research Office grants W911NF2010168 and 10293, of Air Force Office of Scientific Research award FA9550-19 18-00-19---111, of National Science Foundation grants IIS- 1838071 and CHS-1955697, from the CSAIL Systems that Learn program, from the MIT-IBM Watson AI Laboratory. from the Toyota-CSAIL Joint Research Center, from a gift from Adobe Systems, from an MIT.nano Immersion Lab/NCSOFT Gaming Program seed grant, and from the Skoitech-MIT Next Generation Program.

---

References
[1] X. Zhou, D. Wang, and P. Krähenbühl. Objects as points. ArXiv, abs/1904.07850 2019.
[2] T. Wang, X. Zhu, J. Pang, and D. Lin. FCOS3D: Fully convolutional one-stage monocular 3d object detection. arXiv: 104.1044.000 10956, 2021.
[3] Z. Tian, C. Shen, H. Chen, and T. He. FCOS: Fully convolutional one-stage object detection. In Proc. Int. Conf. Computer Vision (ICCV), 2019.
[4] C. Godard, O. Mac Aodha, M. Firman, and G. Brostow. Digging into self-supervised monoc- ular depth prediction. ICCV, 2019.
[5] H. Lee, M.-K. Han, D. W. Ko, and I. H. Suh. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv:1907.10326 2019.
[6] V. Guizilini, R. Ambrus, S. Pillai, A. Raventos, and A. Gaidon. 3d packing for self- self-supervised monocular depth estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[7] A. Simonelli, Rota Bulô, Porzi. P. Kontschieder, and E. Ricci. Demystifying pseudo lidar for monocular 3d object detection, 12 2020.
[8] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Puttern Recognition Recognition (CVPR), 2016.
[9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and Polo- sukhin. Attention is all you need. In Advances in Neural Information Processing Systems (NeurIPS), 2017.
[10] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[11] Anonymous. Object dgenn: 3d object detection using dynamic graphs. 2021.
[12] R. Girshick, J. Donahue, Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[13] S. Ren, K. He, R. Girshick, and J. Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Neural Information Processing Systems (NeurIPS), 2015.
[14] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask R-CNN. In The International Conference on Computer Vision (ICCV), 2017.
[15] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, SSD: Single shot multibox detector. The European Conference on Computer Vision (ECCV), 2016.
[16] Redmon, S. K. Divvala, B. Girshick, and Farhadi. You only look once: Unified, real- time object detection. In The IEEE Conference Computer Vision and Pallern Recognition (CVPR), 2016
[17] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable (detr): Deformable transformers for end-to-end object detection. In International Conference Learning Representations, 2021. URL https: //pprr/vieep//eee///
[18] Z. Sun, S. Cao, Y. Yang, and K. Kitani. Rethinking transformer-based set prediction for object detection. ArXiv, abs/201 108811, 2020.
[19] Sun, R. Zhang. Y. Jiang. T. Kong, C. Xu, W. Zhan, M. Tomizuka. L. Li, Z. Yuan, C. Wang, and P. Luo. SparseR-CNN: End-to-end object detection with learnable proposals. 1rXiv:2011.122 12450, 2020.
[20] Sun, Jiang, F. Xie, Z. Yuan, Wang, and P. Luo. OneNet: Towards end-to-end one-stage object detection. arXiv: 2012.05780 2020.
10

---

[21] X. Chen, K. Kundu, Z. Zhang, H. Ma, S. Fidler, and R. Urtasun. Monocular 3d object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2147-2156, 2016.
[22] T. Roddick, A. Kendall, and R. Cipolla. Orthographic feature transform for monocular 3d object detection. arXiv: /8/ 180888, 2018.
[23] A. Mousavian, D. Anguelov, J. Flymn, and J. Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7074-7082, 2017.
[24] W. Kehl, F. Manhardt, F. Tombari, S. Ilie, and N. Navab. Ssd-6d: Making rgb-based 3d detec- tion and 6d pose estimation great again. In Proceedings of the IEEE international conference on computer vision, pages 1521-1529, 2017.
[25] J. Ku, A. D. Pon, and S. L. Waslander. Monocular 3d object detection leveraging accurate proposals and shape reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11867-11876, 2019.
[26] D. Beker, H. Kato, M. A. Morariu, T. Ando, T. Matsuoka, W. Kehl, and A. Gaidon. Monocular differentiable rendering for self-supervised 3d object detection. arXiv:2009. 14524, 2020.
[27] I. Barabannu, A. Artemov, E. Burnaev, and V. Murashkin. Monocular 3d object detection via geometric reasoning on keypoints. 1.XXvv:905.006111 2019.
[28] Z. Liu, Z. Wu, and R. Tóth. Smoke: single-stage monocular 3d object detection via keypoint estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 996-997, 2020.
[29] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid net- works for object detection. In The IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2017.
[30] R. Stewart, M. Andriluka, and A. Y. Ng. End-to-end people detection in crowded scenes. In 2016 IEEE Conference Computer Vision and Pattern Recognition CVPR, pages 2325-2333 IEEE Computer Society, 2016.
URL https: http//ii.oooogg///// or./10.
1109/CVPR. 2016.202.
[31] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and Dollár. Focal Loss for Dense Object Detection. In The International Conference Computer Vision (ICCV), 2017.
[32] H. W. Kuhn and B. Yaw. The hungarian method for the assignment problem. Naval Res. Logist. Quart, pages 83-97, 1955.
[33] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, E. Liong, Q Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. urXiv:1903.1 11027, 2019.
[34] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformable convolutional networks. In The International Conference Computer Vision (ICCV), 2017.
[35] L. J. Ba, J. R. Kiros, and G. F. Hinton. Layer normalization. CORR, 2016.
[36] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Con- ference on Learning Representationss 2019. URL https://opennevvevwwie wwtp:/iwwwwwwwwwwnieeieiee Bkg6RiCqY7.
[37] D. Park. R. Ambrus, V. Guizilini, J. Li, and A. Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In IEEE/CVE International Conference on Computer Vision (ICCV), 2021.
[38] Yu, Wang, Shelhamer, and T. Darrell. Deep layer aggregation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
11

---

[39] B. Zhu, Z. Jiang, X. Zhou, Z. Li, and G. Yu. Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection. arXiv e-prints, art. 1EXiv:198.009492 Aug 2019.
[40] Wang, X. ZHU, Pang, and D. Lin. Probabilistic and geometric depth: Detecting ob-
jects in perspective. In 5th Annual Conference Robot Learning, 2021. URL https: //openreviev.
[41] T. Yin, X. Zhou, and P. Krähenbühl. Center-based 3d object detection and tracking. CVPR, 2021.
[42] Y. Wang, W.-L. Chao, D. Garg, B. Hariharan, M. Campbell, and K. Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019.
12


Reviews:
[{"review_id": "QpPagal8MBc", "reviewer": "Reviewer_tKqW", "summary": "The paper proposes a novel approach for multi-camera 3D object detection. The proposed model is based on DETR, a transformer-based object detection network, by extending it to multi-camera 3D detection. The main contributions compared to DETR are:\n- projecting the learned object queries into each camera using the calibration matrices,\n- sampling features from the projected queries and feed them to the transformed decoder,\n- regressing bounding boxes in 3D instead of 2D space.\n\nThe method is evaluated on the nuScenes dataset and compared to two existing approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper is well written and well organized.\n+ The proposed approach is very interesting, especially the idea of learning object queries in the 3D space and projecting them into each camera to sample the associated features.\n+ The proposed model does not require post-processing techniques, such as NMS, which is usually required by existing object detection methods.\n+ Differently from existing methods, which perform multi-camera inference by processing each camera independently, the proposed approach directly fuses the features from different cameras.\n\nWeaknesses:\n- My main concern regards the experimental evaluation. In Table 1 the authors state that they \"compare to the FCOS3D without fine-tuning and test-time augmentation\". Since the proposed method does not use test-time augmentation (or at least is not mentioned in the paper), I can understand the comparison against FCOS3D without it. However, I do not understand why they compared against FCOS3D without fine-tuning. Moreover, the results reported for CenterNet are different from the ones reported in the nuScenes leaderboard. Both FCOS3D with fine-tuning and CenterNet leaderboard results are better than the results achieved by the proposed approach.\n- The ablation study is a bit shallow, only the number of decoder layers is analyzed. More ablation studies would be appreciated, such as on the number of object queries and different feature extractors.\n- For readers who are not familiar with the DETR model, the network architecture is not very clear. For example, the set of learned object queries is not explained properly, what do they represent? In general, the understanding of this paper requires a good understanding of the DETR architecture. Moreover, Fig. 1 is not very helpful in understanding the architecture.\n- The paper claims that the proposed approach \"does not require any post-processing, such as non-maximum suppression (NMS), improving efficiency\" (L.46). Therefore, I would expect an analysis of the inference time to be reported.\n\nTypo:\n(L.132): \"from the these images\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Weak Accept: the proposed approach is interesting, but the paper should provide a better description of the architecture, and the results are not entirely convincing.\n\nPost rebuttal: the authors' addressed all my comments and added several comparisons and experiments that strengthen the claim of the paper. I suggest that the paper should be accepted.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- The paper should better discuss the reason why they compared with specific versions of existing approaches, and why the results of some of them differ from the ones available on the leaderboard.\n- The paper should better explain the network architecture, without relying on the readers' knowledge of DETR.\n- The paper should include a runtime analysis to support the claim that the proposed approach improves efficiency."}}, {"review_id": "NPyIpkVtz6c", "reviewer": "Reviewer_KAPs", "summary": "This paper introduces a detection head for 3D bounding box object in multi-view images. The approach is based on a DETR-like architecture, where object queries are iteratively passed through transformer layers to generate predictions and trained with a set-to-set loss. The key technique proposed in this work is to predict 3D bounding box centers in the object queries that are then backprojected to the images to gather multi-view features before further refinening them with the same process iteratively. Experiments are conducted on the nuScenes dataset where 360 degree cameras are available.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n\n- The multi-view feature aggregation method proposed is sensible. Specifically, performing flexible multi-view aggregation with attention seems like a good direction that can suppress spurious errors.\n\n- The approach is comparatively fast (although not real-time) because it doesn't require NMS. \n\n- The paper is clear, self-contained, and overall well-written.\n\nWeaknesses:\n\n- The approach constrains itself to showing results in the setting of 360 degree camera coverage where there is a lack of appropriate baselines, specifically Pseudo-LiDAR. The authors resort to re-implementing a baseline in their setting, but the results indicate that the baseline works so bad that it isn't clear if this issue is due to a problem in the method itself or in the implementation. However, constraining to such a restricted setting is not necessary as far as far as I can tell. For example, when one would only use the two-view stereo setup, it would be possible to evaluate on KITTI where code for the original pseudo-LiDaR work is available. I would strongly encourage such an evaluation.\n\n- The main evaluation compares to CenterNet and FCOS3D, both of which are intrinsically single-image methods. They require a ad-hoc postprocessing heuristic to bring them into the multi-view setting. As such these methods are fundamentally at a disadvantage. It is not clear to me if this comparison is truly fair. As such a proper comparison to the Pseudo-LiDAR line of work seems even more necessary.  Nonetheless, is seems that the baselines, and specifically FCOS3D, perform rather competitive when looking at Table 1. Table 2 suggests a strong advantage of the proposed method in the overlap regions, which seems reasonable given the setting, but also suggest together with Table 1 that the approach is at a strong disadvantage in non-overlap regions.\n\n- The proposed approach foregoes explicit depth estimation and directly regresses 3D bounding box centers which are used for multi-view feature aggregations. The authors state that this is beneficial, as imperfect depth estimates can negatively influence results. However, the task of direct regression seems harder than it needs to be. Some form of depth is always available in this the multi-view setting, even if it might be imperfect. I believe the claims about compounding error in the depth maps could and should be considerably strengthened using more fine-grained ablations: a) use the estimated depths to perform warping and feature aggregation instead of using regressed bounding box centers. b) use the estimated depths as an additional input modality in the existing pipeline. c) potentially use \"groundtruth\" LiDAR depths in both a) and b) as an oracle in order to quantify the performance loss due to imperfect depth maps and imperfect estimates of 3D bounding boxes.\n\n- The baselines in Table 1 are not clear to me. Is the DLA backbone in CenterNet and the overall architecture comparable in terms of capacity? What does it mean that FCOS3D is used without fine-tuning? On what dataset was FCOS3D trained? \n\nMinor comments:\n\n- The captions in Figure 1 are tiny and very hard to read.\n- Figure 2: clarity could be improved by showing groundtruth bounding boxes directly in the layer visualization using a different color, so that the reader can easier see the actual differenc to the groundtruth. One thing to consider could also be color coding all the layer predictions and merging into a single image to better see the evolution of predictions.\n- l. 257: should this be \"upper left\" instead of \"leftmost\"?\n\n\nTypo:\n- l.220: \"ignore\" -> \"ignores\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Technically, the proposed approach is a relatively straight-forward, but sensible, application of the DETR detection paradigm to the task of 3D bounding detection. I believe the approach has potential, but the current evaluation has multiple issues that do not allow to position this work appropriately with respect to existing works.\n\n-- Post rebuttal --\n\nThe authors revised the paper with additional experiments and ablations, which I believe greatly strenghtens the work. I thus raise my score to weak accept.\n", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I'm willing to improve my rating based on the response to my comments in \"Weaknesses\" above.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "Hk9lEALVZi", "reviewer": "Reviewer_Gg7X", "summary": "The paper presents a multi-view (multi-image) 3D object detector based on transformers. The backbone architecture follows DETR, but queries are shared among multiple views. The resulting detector performs on-par with the baseline on nuScenes.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Interesting architecture, clever use of DETR (and its object queries)\n+ Easy to read and follow\n- Limited ablations\n- Performance not SOTA (despite claim)\n\nThe paper is well written and easy to follow. The main idea of using object queries shared between multiple images and a transformer that can exchange information between those is clever and novel.\n\nUnfortunately, the main technical novelty is not really evaluated or ablated. The 360 degree cameras on a car offer only limited parallax, and an even smaller overlap in field of view to take advantage of. It is thus unclear if a 3D lifting is even needed, or if a simple shift and shear operations given the calibrated cameras would suffice?\n\nFinally, the authors claim to be SOTA, but do not outperform FCOS3D. In fact, the presented method only matches the performance of the baseline FCOS3D (without fine-tuning). It seems odd to take prior work, and simply not compare to their best model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The basic idea is cute and interesting, the evaluation and comparison fall short.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Wee above.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: There is a clear agreement between the reviewers that the main idea of the paper, namely the extension of the transformer-based object detector DETR to the multi-view setting, is reasonable and nice. However, there are also a number of important shortcomings. These mainly include a somewhat unbalanced comparison to the state of the art (e.g. skipping fine-tuning in FCOS3D or the two-view pseudo-LIDAR), a number of missing ablation studies (e.g. re number of queries, or estimated depth), and the not obvious motivation to use 3D lifting over simpler 2D techniques. Out of these, the first one weighs most, given that the claim of obtaining SOTA results with the proposed approach becomes questionable. A detailed discussion on this would be very helpful, maybe also including some further reasoning on the choice of experimental comparisons, or else on some potential additional benefits of the proposed approach over the existing techniques. 

Post-rebuttal:
The authors made a significant effort to show that their results are an imprvement over existing apporaches, namely FCOS3D. The results seem to be of interest for the community. The authors are adviced to thoroughly describe and analyse these additional results in the final version of the paper. 
Decision: Accept (Poster)