Uni3DETR: Unified 3D Detection Transformer
Authors: Zhenyu Wang
Ya-Li Li
Xi Chen
Hengshuang Zhao
Shengjin Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: 3d object detection
unified object detection
point clouds

Abstract: Existing point cloud based 3D detectors are designed for the particular scene, either indoor or outdoor ones. Because of the substantial differences in object distribution and point density within point clouds collected from various environments, coupled with the intricate nature of 3D metrics, there is still a lack of a unified network architecture that can accommodate diverse scenes. In this paper, we propose Uni3DETR, a unified 3D detector that addresses indoor and outdoor 3D detection within the same framework. Specifically, we employ the detection transformer with point-voxel interaction for object prediction, which leverages voxel features and points for cross-attention and behaves resistant to the discrepancies from data. We then propose the mixture of query points, which sufficiently exploits global information for dense small-range indoor scenes and local information for large-range sparse outdoor ones. Furthermore, our proposed decoupled IoU provides an easy-to-optimize training target for localization by disentangling the $xy$ and $z$ space. Extensive experiments validate that Uni3DETR exhibits excellent performance consistently on both indoor and outdoor 3D detection. In contrast to previous specialized detectors, which may perform well on some particular datasets but suffer a substantial degradation  on different scenes, Uni3DETR demonstrates the strong generalization ability under heterogeneous conditions (Fig. 1).

Uni3DETR: Unified 3D Detection Transformer
Zhenyu Wang'
Yali Li
Xi Chen
Hengshuang Zhao
Shengjin Wang'
Department of Electronic Engincering, Tsinghua University, BNRist
2 The University of Hong Kong
(wangzy20cmails. liyali13e, wgagj tainghua edu. en, {xchen2, hazhao)@cs. .kuu .hk
VoteMet
#3DWet
Bet
Groupfree
FCAF3D
CAGroup
UniSDETR (ours)
SEC,OND
PointPillar
PoiNtRCAN SUN
30SSD
Centerpoint
UVTR
SUM
S:N
SUN
SUM
Figure 1: The high-level overview comparing the performance of 12 existing 3D detectors and our Uni3DETR on a broad range of 3D object detection datasets: indoor datasets SUN RGB-D (SUN), ScanNet, S3DIS, and outdoor datasets KITTI, nuScenes (nuS). The metrics are AP25 for indoor datasets, AP70 on moderate difficulty car for KITTI, and NDS for nuScenes. The center the circle means that the corresponding metric is less than 10%, and the outermost means 90%. Existing indoor detectors are plotted in red and outdoor detectors are green. Our model has the remarkable capacity to generalize across wide range of diverse 3D scenes larger polygon area).
Abstract
Existing point cloud based 3D detectors designed for the particular scene, either indoor or outdoor ones. Because of the substantial differences in object distribution and point density within point clouds collected from various environments, coupled with the intricate nature of 3D metrics, there is still a lack of unified network architecture that can accommodate diverse scenes. Uni3DETR, a unified 3D detector that addresses indoor and outdoor 3D detection within the same framework. Specifically, we employ the detection transformer with voxet interaction for object prediction, which leverages voxel features and points for cross- aotseatttntion and behaves resistant the discrepancies from data. We then propose the mixture of query points, which sufficiently exploits global information for dense small-range indoor scenes and local information for large- range sparse outdoor ones. Furthermore, our proposed decoupled loU provides an easy-to-optimize training target for localization by disentangling the zy and space. Extensive experiments validate that Uni3DETR exhibits excellent performance consistently on both indoor and outdoor 3D detection. In contrast to previous specialized detectors, which may perform well on some particular datasets but suffer a substantial degradation on different scenes, Uni3DETR demonstrates the strong generalization ability under heterogeneous conditions (Fig. 1). Codes are available at https://ithub..cmm wttp////weewwwww///iiiieeee
this paper, we propose
"Correspondigg author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
3D object detection from point clouds aims to predict the oriented 3D bound- ing boxes and the semantic labels for the real scenes given a point set. Unlike mature 2D detectors [41, 19, 56, 4] on RGB images, which have demonstrated the ability to effectively address diverse conditions, the prob- lem of 3D object detection has been considered under different scenarios, leading to the development of distinct benchmarks and methodologies for each. Specifically, 3D detection ap- proaches are currently addressed sep- arately for indoor and outdoor scenes.
indoor point clouds
outdoor point clouds
point clouds
PointNet
3D sparse conv
3D sparse conv
3D sparse conv
+ grouping & clustering
2D dense conv in BEV
3D dense conv
detection transformer
classification
classification
(a)
(b)
(c)
Figure 2: Illustration of the structures of previous indoor 3D detectors (a), outdoor detectors (b) and our model. Our model has the capacity for 3D detection in both indoor and outdoor scenes.
Indoor 3D detectors [38, 70, 62, 58] usually adopt the "grouping, clustering and classification" man- ner (Fig. 2a), while outdoor detectors [64, 24, 52, 49] typically convert the features into the 2D wird's-eye-view (BEV) space (Fig. 2b). Although these two tasks only differ in their respective application contexts, the optimal approaches addressing them exhibit significant differences.
The key challenge in developing a unified 3D detector lies in the substantial disparities of point cloud data collected in indoor and outdoor environments. In general, indoor scenarios are cluttered, where various objects are close and dense, occupying the majority of the scene. In comparison, objects in outdoor scenes are small and sparse, where background points dominate the collected point clouds. Such disparities result in the lack of a unified 3D detector: 1) For the detection head, because of the severe distraction of excessive background points in outdoor point clouds, together with the hyperparameter sensitivity associated with grouping, the grouping-based indoor detectors are infeasible for outdoor 3D detection. Besides, outdoor objects are separated clearly and not overlapped in the BEV space, which does not apply to indoor objects. The height overlap among indoor objects makes the manner of detecting in the BEV space inappropriate for indoor detection. 2) For the feature extractor, existing backbones in 3D detectors are similarly designed for a singular scene. Indoor detectors are usually equipped with point-based [39, 40] or 3D sparse convolution based [10, 43] backbone, where point-based models are usually susceptible to the diverse structures of points under different scenes, and sparse convolution models are deficient in representing the features of object centers. In contrast, the 2D convolutions in outdoor detectors for extracting BEV features easily lead to information loss for indoor detection.
In this paper, we propose a Unified 3D DEtection TRansformer (Uni3DETR) based on only point clouds for detecting diverse environments (Fig. 2c). Two attributes of our Uni3DETR contribute to its universality for both indoor and outdoor scenes. First, we employ hybrid structure that combines 3D sparse convolution and dense convolution for feature extraction. The pure 3D architecture avoids excessive height compression for indoor point clouds. Simultancously, the sparse convolution prevents the huge memory consumption for large-range outdoor data, and the dense convolution alleviates the center feature missing problem for sparse outdoor points. Second, we utilize transformer [57] for 3D object detection. The set-to-set prediction way in transformer- based detectors [4, 77] directly considers predicted objects and ground-truth labels, thus tends to be resistant to the distinction from data themselves. The transformer decoder is built on the extracted voxel feature and we formulate queries as 3D points from the scene. The points and voxels interact through cross-attention, which well adapts to the characteristics of 3D data.
Based on our 3D detection transformer, we further propose two necessary components for universality under various scenes. One is the mixture of query points. Specifically, besides the learnable query, we introduce the non-learnable query initialized by sampling the original points and the voxelized points, and integrate the learnable and non- non-learnable queries for feeding the transformer. We observe that the learnable query points mostly contain local information and fit the outdoor detection well, while non-learnable query points emphasize global information thus are more effective for dense indoor scenes. The other one is the decoupled 3D loU. Compared to the usual 3D IoU, we decouple the v, y and z axis in the decoupled IoU to provide stronger positional regularization, which not only involves all directions in the 3D space but also is beneficial for optimization in transformer decoders.

---

detection transformer
IoUde
mixture of query points
(loUsy + loU,)/2
non-learnable query points
learnable random
query points
(test only)
input point cloud
3D feature extractor
Attentionn
SelfAAtention
gt
boxes
3D sparse
3D dense
query
Hungarian
voxelize
vord features
Cross-Attention
matching
Figure 3: The overall architecture of our Uni3DETR. We use the hybrid combination of 3D sparse convolution and dense convolution for 3D feature extraction, and our detection transformer for set prediction. The mixture of query points contributes to the sufficient usage of global and local information, and the decoupled IoU provides more effective supervision about localization.
Our main contributions can be summarized as follows:
We propose Uni3DETR, unified 3D detection transformer that is designed to operate effectively in both indoor and outdoor scenes. To the best of our knowledge, this the first point cloud based 3D detector that demonstrates robust generalization across diverse environments.
We propose mixture query points that collaboratively leverages the learnable and non- learnable query. By aggregating local and global information, our mixture of queries fully explores the multi-scene detection capacity.
We decouple the multiple directions in the 3D space and present the decoupled 3D IoU to bring more effective positional supervision for the transformer decoder.
Our Uni3DETR achieves the state-of-the-art results on both indoor [53, 12, 1] and outdoor [16, 3] datasets. It obtains the 67.0% and 71.7% AP25 the challenging SUN RGB-D and ScanNet V2 indoor datasets, and the 86.7% AP in the moderate car category on KITTI validation. On the challenging nuScenes dataset, Uni3DETR also achieves the 61.7% mAP and 68.5% NDS.
2
Uni3DETR
2.1
Overview
We present the overall architecture of our Uni3DETR Fig. 3. It consists of 3D feature extractor and the detection transformer for detecting in various 3D scenes. The mixture of query points is fed into transformer decoders to predict 3D boxes, under the supervision of decoupled 3D IoU.
3D feature extractor. Existing 3D detectors usually adopt point-based [40] or basel [64, 43] backbones for feature extraction. Considering that point-based backbones are vulnerable to the specific structure of point clouds themselves and less efficient in point set abstraction, we utilize the
model for extracting 3D features. After voxelization, we utilize a series of 3D) sparse convolution layers to encode and downsample 3D) features, to avoid the overload memory consumption for large-range outdoor scenes. Then, we convert the extracted sparse features into the dense ones and apply 3D dense convolutions for further feature processing. The dense convolution alleviates the feature missing problem center points.
2.2 Unified 3D Detection Transformer for Diverse Scenes
Detection transformer with point-voxel interaction. We employ the transformer structure based on the extracted voxel features and the set prediction manner for 3D detection. Motivated by recent 2D transformer-base detectors [29, 25, 69] that formulate queries as anchor boxes, we regard 3D points in the 3D space queries. Its structure is illustrated in Fig. 4.
Specifically, we denote Pg = (Iigs Mg: =n) as the q-th point to represent the 4-th object, C, E PRP its content query, where D is the dimension of decoder embeddings. We introduce the deformable

---

attention [77] for the cross-attention module and view Pg as the reference point for point-voxel interaction. Unlike the deformable DETR decoder, here we directly learn the reference point Pg. Suppose the voxel feature as V, the process of the cross-attention is modeled as:
CrossAtt(Cg: V)= = (eformAtt(C. + (L)( Pg)
(I)
where PE denotes the sinusoidal positional encoding. The transformer decoder predicts the relative positions for the query points: (Azy, Alg. Azz). Based on the relative predictions, the query points are refined lyyye-
Compared to 2D detection transformers. because of the stronger positional information in the 3D space, our detection transformer requires less layers for query refinement. To further utilize the information across multiple layers, we average predictions from all transformer decoders except for the first one.
FFN
Mixture of query points. In the learning process, the above query points gradually turn around the object instances for better 3D detection. As they are updated during training, refer them as learnable query points. Since such learnable query points finally concentrate on the points near the objects, they primarily capture the local information of the scene. For outdoor point clouds, where objects are sparse and small within the large-range scene, the learn- able query points help the detector avoid the disturbance of the dominated background points.
Deformable Attention
Self- Afff-Attentinn
In comparison, for the indoor scene, the range of 3D points is sig- nificantly less and the objects in relation to the entire scene are of comparable size. In this situation, besides the local information, the transformer decoder should also utilize the global information to consider the whole scene. We thus introduce the non-learnable query points for global information. Specifically, we use the Far- thest Point Sampling (FPS) [40] for the input point cloud data and take the sampled points as the query points. These query points are frozen during training. Because of the FPS, the sampled points tend to cover the whole scene and can well compensate for objects that learnable query points ignore.
y point query
voxel feature
Figure 4: The structure of our detection transformer. The decoder is built extracted voxel features and we intro- duce 3D points the query. The points and voxels interact through tross-attention
As utilize the moxel-based backbone for feature extraction, vOx- elization is a necessary step for processing the point cloud data. After voxelization, the coordinates and numbers of the raw point clouds may change slightly. To consider the structure of both the raw data and points after voxelization, what we use is two kinds of non-learnable query points Par and Putv- Pai denotes the non-learnable query points from FPS the original point clouds, and Pate comes from sampling the points after voxelization. Together with the learnalle query points Pt. the mixture of query points ultimately consists of three ones: = (Pi, Pat. Juiv.. We follow [5] to perform group-wise self-attention in the transformer decoders for the mixture of query points, where these three kinds of query points do not interact with each other. After transformer decoders, three sets of predicted 3D boxes (bby, bbnz, bbute } are predicted respectively. We apply one-to-one assignment to each set of 3D boxes independently using the Hungarian matching algorithm [23] and calculate the corresponding training loss to supervise the learning of the detection transformer.
At the test time, besides the above three kinds of query points, we further generate series of additional points uniformly and randomly in the 3D voxel space as another set of query points Pra. These randomly generated points evenly fill the whole scene, thus well compensating for some potentially missed objects, especially for the indoor scenes. As a result, four groups of query points P {P,, Pnt, Paiv: Pra) are utilized for inference. Ultimately, four sets of predicted 3D boxes [bb,, bb,r, bb,tt. bb,d) are produced at the test time by Uni3DETR. Because of the one-to-one assignment strategy, the individual box sets do not require additional post- poss-processing methods for removing duplicated boxes. However, there are still overlapped predictions among these four sets. We thus conduct box merging finally among the box sets to fully utilize the global and local information and eliminate duplicated predictions. Specifically, we cluster the 3D boxes based on the 3D loU among them and take the median of them for the final prediction. The confidence score of the final box is the maximum one of the clustered boxes.

---

2.3 Decoupled IoU
Existing transformer-base 2D detectors usually adopt the L1 loss and the generalized IoU loss [42] for predicting the bounding box. The generalized IoU here mitigates the issue of L1 loss for different scales thus provides further positional supervision. However, for 3D bounding boxes, the calculation of ToU becomes more challenging problem. One 3D box is usually denoted by: bb (x. y, 2, w,1, I, h, 0), where (x, y, 2) is the center, (w,1,h) the size and the rotation. The usual 3D loU [72] between two 3D boxes bbj and bb2 is calculated by:
Areaoverlapped *overlapped
LoU3D= Area 21 + Areaz 22 - A Areaoverlapped Zoverlapped
where Area denotes the area of rotated boxes in the zy space.
From Equ. 2, we notice that the calculation of 3D IoU generally consists of two spaces: the zy space for the bounding box area, involving the 2; y, w, variables, and the space for the box height. The two spaces are multiplied together in the 3D IoU. When supervising the network, the gradients of these two spaces are coupled together - optimizing in one space will interfere with another, making the training process unstable. For 3D GloU [72], besides the negative coupling effect, the calculation of the smallest convex hull area is even non-optimizable. As a result, the usual 3D IoU is hard to optimize for detectors. For our detection transformer, where LI loss already exists for bounding box regression, the optimization problem of 3D IoU severely diminishes its effect training, thus being hard to mitigate the scale problem of LI loss.
Therefore, an ideal metric for transformer decoders in 3D should at least meet the following demands: First, it should be easy to optimize. Especially for the coupling effect for different directions, its negative effect should be alleviated. Second, all shape properties of the 3D box need to be considered, so that accurate supervision signals can be provided for all variables. Third, the metric should be scale invariant to alleviate the issue of L. loss. Motivated by these, forward the decoupled IoU:
Areaoverlapped
Zoverlapped
IoUde de
( Areaj + Areag Areanverlapped
)/2
Poverlapped
Specifically, our decoupled IoU is the average of the IoU in the xy space and the z space. Under the summation operation, the gradients from the two items remain separate and independent, avoiding any coupling effect. According to previous research [47], the coupling issue does not exist in 2D or 1D IoU. As result, the decoupled IoU effectively circumvents the negative effects of coupling. The scale-invariant property of 3D IoU is also well-reserved for the decoupled IoU. These make our decoupled IoU well suit the training of the transformer decoder.
The decoupled IoU is introduced in both the training loss and the matching cost. We also introduce it into classification and adopt the variant of quality focal loss [26]. Denote the binary target class label as c, the predicted class probability as p. the classification loss for Uni3DETR is:
Lets - |c- IoU -77 log([1 c Pl)
where a; = a--c c. IoU + (1- a) (1 c. IoUae). The above classification loss can be viewed as using the soft target loUde in focal loss [28]. The decoupled IoU is easy optimize, allowing us to avoid the need for performing the stop-gradient strategy on ToUde. Therefore, the learning of classification and localization is not separated, assisting Uni3DETR predicting more accurate confidence scores, Equ. 4 will force the predicted class probability P towards IoU which might be inconsistent with loUaD used in evaluation. We thus follow [22] and introduce an IoU-branch in our network to predict the 3D IOU. The normal IoUan supervises the learning of the loU-branch and the binary cross- enssopy loss is adopted for supervision. The weighted geometric mean of the predicted 3D IoU and the classification score is utilized for the final confidence score. The final loss function for training Uni3DETR is thus the classification loss (Equ. 4), the LI loss and the IoU loss with IoUne. for bounding box regression, and the binary cross-entropy loss for IoU prediction.
3 Experiments
To demonstrate the universality of our Uni3DETR under various scenes, we conduct extensive experiments in this section. We evaluate Uni3DETR in the indoor and outdoor scenes separately.
5

---

Table I: The performance of Uni3DETR for indoor 3D object detection. The main comparison is based on the best results of multiple experiments. We re-implement VoteNet, H3,Net, and GroupFree on the S3DIS dataset. * indicates the modti-modal method with both point clouds and RGB images.
SUN RGB-D
ScanNet
S3DIS
Method
AP25
AP50
AP25
AP50
AP25
AP30
Im VoteNet* [37]
63.4
TokenFusion [60]
64.9
48.3
70.8
54.2
VoteNet [38]
57.7
35.7
58.6
33.5
58.0
25.3
GSDN [17]
62.8
34.8
47.8
25.1
H3DNet [70]
60.1
39.0
67.2
48.1
50.9
22.0
BRNet [9]
61.1
43.7
66.1
50.9
3DETR [35]
59.1
32.7
65.0
47.0
VENet [62]
62.5
39.2
67.7
GroupFree [30]
63.0
45.2
69.1
52.8
42.8
19.3
RBGNet [59]
64.1
47.2
70.6
55.2
HyperDet3D [71]
63.5
47.3
70.9
57.2
FCAF3D [43]
64.2
48.9
71.5
57.3
66.7
45.9
Uni3DETR (ours)
67.0
50.3
71.7
58.3
70.1
48.0
Datasets. For indoor 3D detection, we evaluate Uni3DETR on three indoor 3D scene datasets: SUN RGB-D [53], ScanNet V2 [12] and S3DIS [1]. SUN RGB-D is single-view indoor dataset with 5,285 training and 5,050 validation scenes, annotated with 10 classes and oriented 3D bounding boxes. ScanNet V2 contains 1,201 reconstructed training scans and 312 validation scans, with 18 object categories for axis-aligned bounding boxes. S3DIS consists of 3D scans from buildings, object classes annotated with axis-aligned bounding boxes. We use the official split, evaluate method on 68 rooms from Area 5 and use the rest 204 samples for training. We use the mean average precision (mAP) under IoU thresholds of 0.25 and 0.5 for evaluating on these three datasets.
For outdoor 3D detection, we conduct experiments on two popular outdoor benchmarks: KITTI [16] and nuScenes [3]. The KITTI dataset consists of 7,481 LiDAR samples for its official training set, and we split it into 3,712 training samples and 3,769 validation samples for training and evaluation. The nuScenes dataset is large-scale benchmark for autonomous driving. using the 32 lanes LiDAR for data collection. We train on the 28,130 frames of samples in the training set and evaluate on the 6,010 validation samples. We use mAP and nuScenes detection score (NDS), a weighted average of mAP and other box attributes like translation, scale, orientation, velocity.
Implementation details. We implement Uni3DETR with mmdetection3D [11]. and train it with the AdamW [32] optimizer. We set the number of learnable query points to 300 for datasets except for nuScenes, where we set to 900. For indoor datasets, we choose the 0.02m grid size. For the KITTI dataset, we use a (0.05m, 0.05m, 0. Im) voxel size and for the nuScenes, we use the (0.075m, 0.075m, 0.2m) voxel size. The nuScenes model is trained with 20 epochs, with the CBGS [75] strategy. For outdoor datasets, we also conduct the ground-truth sampling augmentation [64] and we remove the fround-truth sampling at the last 4 epochs. Dynamic voxelization [73] and ground-truth repeating [21] are also adopted during training. Besides these data-related parameters, other architecture-relate hyper-parameters are all the same for different datasets.
3.1 Indoor 3D Object Detection
We first train and evaluate Uni3DETR on indoor 3D detection datasets and list the comparison with existing state-of-the-arr indoor 3D detectors in Tab. 1. Here we omit some grouping methods like [58]. which relies on mask annotations for better grouping and clustering. Our method obtains the 67.0% AP25 and 50.3% AP5o SUN RGB-D, which surpasses FCAF3D, the state-of-the-a art indoor detector based on the CNN architecture, by almost 3%. On the ScanNet dataset, Uni3DETR surpasses FCAF3D by 1% on AP It is also noticeable that with only point clouds participating in training, our method even obtains better performance than existing multi-modal approaches that require both point clouds and RGB images. On the SUN RGB-D dataset, our model is 2.1 % higher on AP25 than TokenFusion. This strongly demonstrates the effectiveness of Uni3DETR. The visualized results of Uni3DETR SUN RGB-D be seen in the left two of Fig. 5.
Our method also significantly outperforms existing transformer-based indoor detectors, 3DETR and GroupFree. The superiority of our method is more significant, especially in localization: Uni3DETR

---

Table 2: The performance of Uni3DETR for outdoor 3D object detection the KITTI validation set with 11 recall positions. M: means training on three classes and the blank means training only on the car class. *: AP on the moderate car is the most important metric.
Car-3D (IoU=0.7)
Ped.-3D (155=0.5)
Cyc.-3D (loU=0.5)
Method
M
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
SECOND [64]
V
88.61
78.62
77.22
56.55
52.98
47.73
80.59
67.16
63.11
PointPillar [24]
V
86.46
77.28
74.65
57.75
52.29
47.91
80.06
62.69
59.71
PointRCNN [5]]
V
89.06
78.74
78.09
67.69
60.74
55.83
86.16
71.16
67.92
Part-A4 [52]
89.56
79.41
78.84
65.69
60.05
55.45
85.50
69.90
65.49
PV-RCNN [49]
89.35
83.69
78.70
63.12
54.84
51.78
86.06
69.48
64.50
CT3D [46]
89.11
85.04
78.76
64.23
59.84
55.76
85.04
71.71
68.05
RDIoU [47]
89.16
85.24
78.41
63.26
57.47
52.53
83.32
68.39
63.63
UniDDETR (ours)
89.61
86.57
78.96
70.18
62.49
58.32
87.18
72.90
68.86
3DSSD [65]
88.82
78.58
77.47
STD [66]
89.70
79.80
79.30
VoxeI-RCNN [13]
89.41
84.52
78.93
VoTr-TSD [33]
89.04
84.04
78.68
CT3D [46]
89.54
86.06
78.99
BteDet [63]
86.57
RDIoU [47]
89.76
79.04
Uni3DETR (ours)
90.23
86.74
79.31
Table The performance of Uni3DETR for outdoor 3D object detection on the nuScenes valida-
tion set. We compare with previous methods without test-time augmentation. *: the implementation
from OpenPCDet [55].
Method
NDS(%)
mAP(%)
MATET
MASET
MAOEI
mAVEL
mAAE
PointPillar [24]
49.1
34.3
0.424
0,284
0.529
0.377
0.194
CBGS [75]
61.5
51.9
CenterPoint [67]
64.9
56.6
0.291
0.252
0.324
0.284
0.189
VoxeINeXt* [8]
66.7
60.5
0.301
0.252
0.406
0.217
0.186
PillarNet [48]
67.4
59.8
0.277
0.252
0.289
0.247
0.191
UVTR [27]
67.7
60.9
0.334
0,257
0.300
0.204
0.182
Uni3DETR (ours)
68.5
61.7
0.288
0.249
0.303
0.216
0.181
outperforms them by 5.1% SUN RGB-D and 5.5% on ScanNet in AP50- Such results validate that compared with existing transformers in 3D detection, our detection transformer on voxel features with the mixture of query points is more appropriate for 3D detection.
3.2 Outdoor 3D Object Detection
KITTL. We then conduct experiments on the outdoor KITTI dataset. We report the detection results on the KITTI validation set three difficulty levels casy, moderate, and hard Tab, 2. We notice that Uni3DETR also achieves the satisfying performance with the same structure as that for indoor scenes, For the most important KITTI metric, AP on the moderate level of car, we obtain the 86.57% AP, which is more than 1.5% higher than CT3D and 1.3% higher than RDIoU. With only the car class in training, the car moderate AP is 86. 74%, which is also higher than existing methods like BteDet. Its ability in outdoor scenes is thus demonstrated. The superiority of Uni3DETR is also consistent for the pedestrian and cyclist class. This illustrates that our model can also distinguish small and scarce objects well, which is one main aspect that hinders existing indoor detectors in the outdoor environments. The visualized results are shown in the right two of Fig. 5.
nuScenes. We further conduct experiments on the nuScenes dataset. Compared to the KITTI dataset, the range of scenes in nuScenes is larger, with 360 degrees around the LiDAR instead of only the front view. The point cloud in nuScenes is also more sparse (with 32-beam LiDAR compared to the KITTI 64 beams). These make the nuScenes more challenging and even some existing outdoor detectors fail to address the detection problem on nuScenes. We list the comparative results in Tab. We obtain the 68.5% NDS, which surpasses recent methods like PillarNet, UVTR, Xtx Compared to the most recent method VoxeINeXt, Uni3DETR is 1.8% higher in NDS and 1.4% higher in mAP. Besides the detection metric mAP, Uni3DETR also achieves promising results for predicting other attributes of boxes. The ability of our model in the outdoor scenes is further validated.
7

---

Figure 5: The visualized results of Uni3DETR the indoor SUN RGB-D dataset (the left two) and the outdoor KITTI dataset (the right two). Better zoom with colors.
Table 5: Effect of the mixture of query points on 3D indoor and outdoor detection. We compare with different combi- nations of queries. *: AP on the moderate car is the most
Table 4: The 3D performance of our detection transformer the SUN RGB-D dataset compared with some existing transformer de- coder structures.
important metric for KITTI.
SUN RGB-D
KITTI car
query
AP25
AP5o
Easy
Mod.*
Hard
transtormer
25
APs0
[
62.6
46.4
90.20
85.59
78.89
3DETR [35]
59.1
32.7
(Pal)
54.0
39.9
89.42
79.24
78.29
Deform [77]
61.3
45.4
(Pnto)
57.4
43.1
89.70
79.34
78.14
DAB-Deform [29]
62.0
44.3
Pat)
65.1
46.9
90.06
85.94
78.64
DN-Deform [25]
62.0
44.6
(Pt, Pntv)
64.5
46.9
90.04
85.86
78.75
ours ((PP)
62.6
46.4
{Pl, Pui, Paint
66.4
49.6
90.12
86.26
79.01
ours (mixed)
66.4
49.6
(P, Pat, Prte
67.0
50.3
90.23
86.74
79.31
3.3
Ablation Study
3D detection transformer. We first compare the detection AP of our detection transformer with existing transformer structures and list the results in Tab. As 3DETR built on point-based features, its performance is restricted by the less effectiveness of point representations. We then implement other methods based on voxel features. As anchors are negatively affected by the center point missing in 3D point clouds, formulating queries as 3D anchors like DAB- or DN-DETR is less effective than our query points. In comparison, our detection transformer better adapts the 3D points, thus achieves the highest performance among existing methods. Besides, it accommodates the mixture of query points, which brings further detection improvement.
Mixture of query points. Tab. 5 analyzes the effect of the mixture of query points on 3D object detection. For indoor detection on SUN RGB-D, the simple usage of the learnable query points {Pi} obtains the 62.6% AP25- Substituting the learnable query with the non-learnable query hurts performance. This is because non-learnable query points are fixed during training and cannot adaptively attend to information around objects. After we introduce (Pi, Pat we find such mixed usage notably boosts AP25. from 62.6% to 65.1 %: the global information in Ps, overcomes the lack of local information in P1 and contributes to more comprehensive detection results. As result, the object-related metric is enhanced greatly. Utilizing (Pi, Paiv] similarly improves AP25, but less than the effect of Pat- This is because voxelization may remove some points, making the global information not that exhaustive. With (P,, Put, Pute }. the three different query points further enhance the performance. Pato here provides more location-accurall global information, thus APso is improved by almost 3%. Random points at the test time further brings a 0.6% improvement.
For outdoor detection on KITTI, we notice that the effect of non-learnable query points is less significant than that indoor scenes. Using { (Pr, Put) is just 0.35% higher than (Pt). Since outdoor point clouds are large-range and sparse, the effect of global information in non-learnable query will be counteracted by excessive background information. Therefore, local information matters more in the outdoor task. The collaboration of the two-level information in the mixture of query makes it suitable for 3D detection in various scenes.
We further combine multiple groups of learnable query points and list the performance in Tab. 6 to compare with the mixture of query points. Such a combination is actually analogous to Group DETR [5, 6] in 2D detection. We observe that unlike 2D detection, multiple groups of learnable queries do not bring an improvement. Even though the number of groups has been raised to 7, the AP on SUN RGB-D still keeps almost the same. The reason is that multiple groups in 2D mainly speed up the convergence, which does not matter much in 3D. In this situation, by providing multi-level information about the scene, the mixture of query points works better for 3D detection.
8

---

Table 7: Effect of the decoupled IoU on 3D indoor and outdoor detection. IoUna aa-31 37 denotes the 3D IoU in axis- aligned way, loUzy by is the 2D loU in the xy space, IoU, is the 1D ToU in the space.
Table 6: Comparison with multiple groups of learnable query points on the SUN RGB-D dataset.
query [P]]
AP25
AP
SUN
RGB-D
KITTI car
62.6
46.4
loU
25
A0
Easy
Mod.*
Hard
[P] X
62.2
46.2
w/o ToU
20.7
3.7
69.81
61.97
64.85
[P.] x3
62.7
46.6
3oU
48.3
29.4
88.70
78.52
77.60
[P] 5
62.3
46.7
IoUpa-3D
50.4
31.6
89.57
79.44
76.81
(Pt) 7
62.7
46.6
RD-IoU [47]
29.1
14.0
82.08
73.50
73.98
Pnt
65.1
46.9
loU
65.8
49.0
89.91
79.58
77.93
Pr, Pnt, >
66.4
49.6
IoU:
64.7
44.1
90.09
79.74
78.57
IoU
67.0
50.3
90.23
86.74
79.31
Table Comparison of performance and computational complexity against existing methods on the indoor SUN RGB-D and the outdoor nuScenes dataset. The metrics are AP25 for SUN RGB-D and mAP for nuScenes. The computational cost is measured on single RTX 3090 GPU.
performance
efficiency
Method
avg.
indoor
outdoor
latency
params
FLOPS
CenterPoint [67]
37.75
18.9
56.6
0.32
9.17 M
121.10 G
VoxeINeXi [8]
39.30
18.1
60.5
0.29
7.12 M
42.57 G
PillarNet [48]
44.00
28.2
59.8
0.31
12.55 M
100.10 G
UVTR [27]
55.55
50.2
60.9
0.51
26.12 M
451.12 G
Uni3DETR (ours)
64.35
67.0
61.7
0.52
26.71 M
458.74 G
Decoupled IoU. In Tab. we compare our decoupled IoU with different types of IoU in the 3D space. On the SUN RGB-D dataset, optimizing with no loU loss, only classification and L.1 regression loss, just obtains the 20.7% AP2s, indicating that 3D) loU necessary to address the scale problem of L1 loss. Introducing the normal ToUsD alleviates the limitation of L.1 loss to some extent, boosting the AP25 to 48.3%. However, it restricted by the negative coupling effect, thus is even inferior to axis-aligned 3D IoU, which does not take the rotation angle into consideration. In comparison, since this problem does not exist in 2D IoU, even the IoU without considering the z axis, can be 17.5% higher than IoU Our decoupled IoU, which can be viewed as (loUzy + loU,)/ /22 takes all directions into account, thus further improves the AP25 to 67.0%. On the outdoor KITTI dataset, the decoupled IoU is equally critical, 24.77% higher than w/o IoU and 8.22% higher than IoUsD. Its necessity for transformer based 3D detection is further validated.
3.4 Comparison of Computational Cost
Here further list the comparison of both performance and efficiency in the Tab. 8. We observe that the computational budget compared with these methods is not significant: the inference time (latency) is almost the same UVTR and the FLOPS is only 1.16% more than UVTR. In addition, we obtain significantly better detection performance on both indoor and outdoor datasets. Compared with VoxeINeXt, one model that mainly focuses on reducing the FLOPS of 3D detectors, we achieve more than 40% indoor AP and more than 25% average AP improvement. In this paper, we mainly target unified structure. To ensure that the detector can accommodate both indoor and outdoor detection, we have, to a certain extent, made sacrifices in terms of efficiency, in order to prioritize its unified ability, A more efficient and unified structure can be left one of the future works.
3.5 Discussion
A unified voxel size. The word "unified" in our paper specifically refers to the architecture aspect. Since point clouds are collected with different sensors, their ranges and distributions vary significantly (about 3m for indoor but more than 70m for outdoor datasets). For the experiments above, we adopt different values for the voxel size, one data-related parameter. We further conduct the experiment with the same KITTI voxel size (0.05m, 0.05m, m)) and list the results in Tab. 9, Compared with other outdoor detectors, our superiority is still obvious. Therefore, even standardizing such data-related parameter, our model can still obtain higher AP.

---

Table 9: Comparison with existing methods on the indoor SUN RGB-D dataset and outdoor KITTI car dataset with the same voxel size. *;
Table 10: The cross-dataset performance the indoor SUN RGB-D dataset compared with
existing methods. We compare with the RGB
results from training with different voxel sizes.
image based Cude RNN with its metric APaD-
Method
indoor
outdoor
Method
Trained
AP3D
3DSSD [65]
9.5
78.6
Cuhe RCNN [2]
SUN RGB-D
34.7
CenterPoint [67]
18.9
74.4
Cube RCNN
OM
35.4
UVTR [27]
35.9
72.0
Uni3DETR (ours)
SUN RGB-D
64.3
Uni3DETR (ours)
47.3
86.7
Uni3DETR (ours)
ScanNet 50.9
Uni3DETR (ours)
67.0
86.7
Cross-dataset experiments. We further conduct the cross-dataset evaluation with the Omni3D metric [2]. From the results in Tab. 10, it is worth noticing that Uni3DETR has a good cross dataset generalization ability. The cross-dataset AP (ScanNet SUN RGB-D) is 16.2% higher than Cube RCNN trained on the SUN RGB-D dataset. The reason is that our Uni3DETR takes point clouds as input for 3D detection, while Cube RCNN takes RGB images for detection. By introducing 3D space information from point clouds, the superiority of unified architecture for point clouds over Cube RCNN can be demonstrated. We further emphasize that cross-dataset evaluation a more difficult problem for point cloud based 3D object detection, the lataset-interferemn issue is more serious. We believe our Uni3DETR can become the basic platform and facilitate related research.
Limitation. One limitation is that we still require separate models for different datasets. Currently, some methods [2, 68] have tried one single model for multiple datasets for 3D object detection. We hope our Uni3DETR can become the foundation for indoor and outdoor 3D dataset joint training.
4 Related Work
Existing research on 3D Object Detection has been separated into indoor and outdoor categories.
Indoor 3D detectors [38, 70, 9, 58] usually cluster points first based on extracted point-wise features, then conduct classification and detection. Recently, 3D sparse convolution based detectors [43, 45] also adopt the anchor-fir free manner [56] in 2D for 3D indoor detection. In comparison, outdoor 3D detectors [64, 52, 49, 13, 46, 7, 50] usually convert 3D features into the 2D BEV space and adopt 2D convolutions for predicting 3D boxes. However, because of the significant difference in point clouds between indoor and outdoor scenes, a unified 3D detector in various environments is still absent. Some recent methods [20, 31, 60, 44] have performed experiments on both indoor and outdoor datasets. However, they rely on RGB images to bridge the difference of point clouds.
Transformer-based object detectors have been widely used in 2D object detection [4. 77, 34, 29, 25, 69], predicting object instances in the set-to-set manner. Recently, the transformer structure has been introduced in 3D object detection. Methods like [33, 18, 15. 54, 14] introduce transformers into the backbone for 3D feature extraction, while still adopting CNNs for predicting boxcs. In comparison, approaches like [35, 61, 36, 74, 76] leverage transformers in the detection head for box prediction. However, these methods still rely on point-wise features or BEV features for either indoor or outdoor 3D detection, thus restricted by the singular scene.
5 Conclusion
In this paper, we propose Uni3DETR, a unified transformer-basee 3D detection framework that addresses indoor and outdoor 3D object detection within the same network structure. By feeding the mixture of query points into the detection transformer for point-voxe interaction and supervising the transformer decoder with the decoupled IoU, our Uni3DETR fills the gap of existing research in unified 3D detection under various scenes. Extensive experiments demonstrate that our model can address both indoor and outdoor 3D detection with a unified structure. We believe our work will stimulate following research along the unified and universal 3D object detection direction, and Uni3DETR will become basic tool and platform.
A Acknowledgment. This work is supported by the National Key Research and Development Program of China in the 14th Five-Year with Nos. 2021 YFF0602 and 2021 YFF0602 12, National Natural Science Foundation of China (No. 62201484), HKU Startup Fund and Seed Fund for Basic Research.
10

---

References
[1] Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016.
[2] Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild, In CVPR, 2023.
[3] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan. and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[5] Qiang Chen, Xiaokang Chen, Gang Zeng, and Jingdong Wang. Group detr: Fast training convergence with decoupled one-to one-to-many label assignment. In ICCV, 2023.
[6] Qiang Chen, Jian Wang, Chuchu Han, Shan Zhang. Zexian Li, Xiaokang Chen, Jiahui Chen, Xiaodi Wang, Shuming Han, Gang Zhang, al. Group detr v2: Strong object detector with encoder-decoder pretraining, arXiv:2211.03500 2022.
[7] Yukang Chen, Yanwei Li, Xiangyu Zhang. Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. In CVPR, 2022.
[8] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In CVPR, 2023.
[9] Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points for voting-based 3d object detection point clouds. In CVPR, 2021.
[10] Christopher Choy, Joung Gwak, and Silvio Savarese. 4d spatio-temporal convnets:
Minkowski convolutional neural networks. In CVPR, 2019.
[11] MMDetection3D Contributors. Mmdetection3d: Openmmlab next-generation platform for general 3d object detection, 2020.
[12] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nieβner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. CVPR, 2017.
[13] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In AAAI, 2021.
[14] Shaocong Dong, Haiyang Wang, Tingfa Xu, Xinli Xu, Jie Wang, Ziyang Bian, Ying Wang, Jianan Li, et al. Mssvt: Mixed-scale sparse voxel transformer for 3d object detection on point clouds. NeurIPS, 2022.
[15] Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In CVPR, 2022.
[16] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013.
[17] Jun Young Gwak, Christopher Choy, and Silvio Savarese. Generative sparse detection networks for 3d single-shot object detection. In ECCV, 2020.
[18] Chenhang He, Ruihuang Li, Shuai Li, and Lei Zhang. Voxel set transformer: A set-to-set approach 3d object detection from point clouds. In CVPR, 2022.
[19] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
[20] Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In ECCV, 2020.

---

[21] Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In CVPR, 2023.
[22] Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localiza- tion confidence for accurate object detection. In ECCV, 2018.
[23] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955.
[24] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[25] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In CVPR, 2022.
[26] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection, NeurIPS, 2020.
[27] Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3d object detection. NeurlIPS, 2022.
[28] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[29] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. ICLR, 2022.
[30] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In ICCV, 2021.
[31] Zhe Liu, Tengteng Huang, Bingling Li, Xiwu Chen, Xi Wang, and Xiang Bai. Epnet++: Cascade bi-directional fusion for multi-modal 3d object detection. TPAMI, 2022.
[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
[33] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng. Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In /CCV, 2021.
[34] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In ICCV, 2021.
[35] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In ICCV, 2021.
[36] Duy-Kien Nguyen. Jihong Ju, Olaf Booij, Martin R Oswald, and Cees GM Snock. Boxer: Box-attention for 2d and 3d transformers. In CVPR, 2022.
[37] Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas. Invotenet: Boosting 3d object detection in point clouds with image votes. In CVPR, 2020.
[38] Charles R Qi. Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, 2019.
[39] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[40] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS, 2017.
[41] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
[42] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. In CVPR, 2019.
12

---

[43] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Fcaf3d: fully convolutional anchor- free 3d object detection. In ECCV, 2022.
[44] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view
purpose 3d object detection. In WACV, 2022.
[45] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Tr3d: Towards real-time indoor 3d object detection. arXiv:2302.02888, 2023.
[46] Hualian Sheng, Sijia Cai, Yuan Liu, Bing Deng. Jianqiang Huang, Xian-Sheng Hua, and Min-Jian Zhao. Improving 3d object detection with channel-wise transformer. In ICCV, 2021.
[47] Hualian Sheng, Sijia Cai, Na Zhao, Bing Deng, Jianqiang Huang, Xian-Sheng Sheng Hua, Min-Jian Zhao, and Gim Hee Lee. Rethinking iou-based optimization for single-stage 3d object detection. In ECCV, 2022.
[48] Guangsheng Shi. Ruifeng Li, and Chao Ma. Pillamett high-performance pillar-based 3d object detection. ECCV, 2022.
[49] Shaoshuai Shi. Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In CVPR, 2020.
[50] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn++: Point-voxel feature set abstraction with local vector representa- tion for 3d object detection. IJCV, 2023.
[51] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, 2019.
[52] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. TPAMI, 2020.
[53] Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In CVPR, 2015.
[54] Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV, 2022.
[55] OD Team et al. Openpedet: An open-source toolbox for 3d object detection from point clouds, 2020.
[56] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In ICCV, 2019.
[57] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
[58] Haiyang Wang, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang, et al. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. NeurIPS, 2022.
[59] Haiyang Wang, Shaoshuai Shi, Ze Yang. Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. In CVPR, 2022.
[60] Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multi- modal token fusion for vision transformers. In CVPR, 2022.
[61] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In CoRL. 2022.
[62] Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Dening Lu, Mingqiang Wei, and Jun Wang. Venet: Voting enhancement network for 3d object detection. In ICCV, 2021.
13

---

[63] Qiangeng Xu, Yiqi Zhong, and UIrich Neumann. Behind the curtain: Learning occluded shapes for 3d object detection. In AAAI, 2022.
[64] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018.
[65] Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR, 2020.
[66] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In ICCV, 2019.
[67] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, 2021.
[68] Bo Zhang, Jiakang Yuan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Uni3d: A unified baseline for multi-dataset 3d object detection. In CVPR, 2023.
[69] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. ICLR, 2023.
[70] Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometrie primitives. ECCV, 2020.
[71] Yu Zheng, Yueqi Duan, Jiwen Lu, Jie Zhou, and Qi Tian. Hyperdet3d: Learning a scene- conditioned 3d object detector. In CVPR, 2022.
[72] Dingfu Zhou, Jin Fang, Xibin Song, Chenye Guan, Junbo Yin, Yuchao Dai, and Ruigang Yang. lou loss for 2d/3d object detection. In 3DV, 2019.
[73] Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3d object detection in lidar point clouds. In CoRL, 2020.
[74] Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. In ECCV, 2022.
[75] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. urXiv:/908 1908.0942, 2019.
[76] Benjin Zhu, Zhe Wang, Shaoshuai Shi, Hang Xu, Lanqing Hong, and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection. CVPR, 2023.
[77] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021.
14

---

A Datasets and Implementation Details
SUN RGB-D [53]. When training on the SUN RGB-D dataset, the input point clouds are filtered in the range [-3 1-3.2m,33 3.2m] for the x axis, [-0.2m, 6.2m] for the y axis and [-2m, 0.56m] for the z axis. The grid size is set 0.02m. We randomly flip the input data along the x axis and randomly sample 20,000 points for data augmentation. The common global translation, rotation and scaling strategies are also adopted here. We train Uni3DETR with the initial learning rate of 1.67e-4 and the batch size of 32 for 90 epochs, and the learning rate is decayed by 10x on the 70th and 80th epoch. The ratio of the classification score and the predicted IoU is 0.8:0.2.
SeanNet [12]. For the ScanNet dataset, here we adopt the range of [-6.4m, 6.4m] for the x and axis and [-0. 1m. 2.46m] for the z axis after global alignment, with the 0.02m grid size. Here we do not adopt the RGB information of the dataset for training. The input data are randomly flipped along both the x and y axis. We utilize dynamic voxelization [73] on the ScanNet dataset. The initial learning rate is set to 7.5e-5, with the batch size of 24. We train Uni3DETR for 240 epochs, where the learning rate is decayed on the 192nd and the 228th epoch. Other peper-parameters and operations are the same as the SUN RGB-D D dataset.
S3DIS [1]. As the S3DIS dataset point clouds are distributed only on the positive planes, which is adverse to the random flipping augmentation, we first translate the input data to make centered at the original point. Besides the coordinate information,
also utilize the RGB information of
the point clouds. We train the Uni3DETR for 520 epochs, with the learning rate decaying at the 416th and 494th epoch. The batch size is set to 8, with the initial learning rate of 3.33e-5. Other hyper-parameters and operations are the same as the ScanNet dataset.
KITTI [16]. For the KITTI dataset, the data augmentation operations are basically the same as previous methods like [13]. For the ground-truth sampling augmentation, we sample at most 20 cars, 10 pedestrians and 10 cyclists from the database. 18000 points are randomly sampled at the training time. During training, we also adopt van class objects as car objects. The ground-truth sampling augmentation and the object-level noise strategy are removed at the last epochs. We train UniSDETR for 70 epochs, with the learning rate decaying at the 64th epoch. The initial learning rate is set to 9e-5, with the batch size of 8. As the KITTI dataset suffers from the sparse objects seriously, we follow [21] to repeat the ground truth labels 5 times during training. The predicted car objects are filtered at the threshold of 0.5 after inference. The ratio of the classification score and the predicted IOU is 0. 0.2:0.8 on the KITTI dataset.
nuScenes [3]. Compared to the KITTI dataset, the nuScenes dataset covers larger range, with 360 degrees around the LiDAR instead of only the front view. The point cloud in nuScenes is also more sparse (with 32-beam LiDAR compared to the KITTI 64 beams). Besides, the nuScenes dataset contains 10 classes, with the severe long-tail problem. The initial learning rate is set to le-5, with the batch size of 16 and the eyclic schedule. We train the Uni3DETR for 20 epochs.
B Test Set Results
We further conduct the experiment and evaluate our method on the test set of KITTI and nuScenes dataset. The comparison is listed in the Tab. 11 and Tab. 12 separately. For the most important KITTI metric, AP on the moderate level of car, we obtain the 82.26% AP, which is 0.83 points higher than PV-RCNN, 0.49 points higher than CT3D, and 0.38 points higher than PV-RCNN++ On the test set of the nuScenes dataset, we obtain the 65. 2% mAP and 70.8% NDS. The consistent superiority further demonstrates the ability of Uni3DETR on outdoor 3D detection.
C Visualized Results
Comparison results about the mixture of query points. We first provide comparative visualized results in Fig. 6 illustrate the effectiveness of the mixture of query points. For the first case, it can be seen that training with only the learnable query points concentrates on the right region of the bed and the nightstand, and ignores the left sofa. This similarly applies to the rest two cases. For the second case, the right nightstand is not detected and for the third case, three chairs are ignored. The common point among these three cases that these ignored objects are partly occluded, thus with insufficient points. The limited quantities of point clouds therefore restrict the performance of the 3D
15

---

Table 11: The performance Uni3DETR for outdoor 3D object detection on the KITTI test set with 40 recall positions. We train the models on the car category only. *: AP on the moderate car is the most important metric.
Table 12: The performance of Uni3DETR for outdoor 3D object detection on the nuScenes test set. We compare with previous methods with double-flip testing.
Method
mAP(%)
NDS(%)
Method
Easy
Mod s
Hard
PointPillar [24]
30.5
45.3
SECOND [64]
88.61
78.62
77.22
CBGS [75]
52.8
63.3
PointPillar [24]
82.58
74.31
68.99
CenterPoint [67]
58.0
65.5
Part- [52]
87.81
78.49
73.51
Focals Conv [7]
63.8
70.0
PV-RCNN [49]
90.25
81.43
76.82
UVTR [27]
63.9
69.7
CT3D [46]
87.83
81.77
77.16
PV-RCNN+ [50]
81.88
PillarNet [48]
65.0
70.8
Uni3DETR (ours)
91.14
82.26
77.58
Uni3DETR (ours)
65.2
70.8
ground-truth
learnable
mixture
Figure 6: The comparative visualized results of our mixture of query points, compared to the result obtained from learnable query points only. The examples are from the SUN RGB-D dataset.
detector. After we introduce the mixture query points, global information better considered with the help of non-learnable query points. As a result, even under the circumstance of insufficient point cloud information, our Uni3DETR can still recognize these objects and detect them out relying on the knowledge about the whole scene. More visualized examples are also provided in Fig. 7.
Comparison results about the decoupled IoU. We then compare the visualized results of Uni3DETR with the normal 3D IoU and plot the results in Fig. 8. It can be seen that when supervising the detector with the normal 3D IoU, although the 3D detector has the ability to detect the foreground objects out, the localization precision remains significantly low. For example, for the second case, two chairs are indeed detected, but the overlaps between the detected instances with corresponding objects are minimal. Furthermore, the low degree of the overlapped area also results in many duplicated boxes, especially for the above one. The same thing also occurs at the third case. For the first case, besides the localization error, only one chair is detected out. This is because 3D loU is hard to optimize
16

---

ground-truth
learnable
mixture
Figure 7: More comparative visualized results of our mixture of query points, compared to the result obtained from learnable query points only. The left two examples are from the ScanNet dataset and the right is from the KITTI dataset.
thus fails to alleviate the scale problem of L1 loss. In comparison, our decoupled IoU addresses the coupling problem 3D IoU, thus contributing to better localization accuracy.
More visualized results obtained by Uni3DETR. We further provide more visualized results obtained by Uni3DETR different datasets. Uni3DETR obtains satisfying detection results all five datasets, which further demonstrate its effectiveness and universality.
D Per-category Results
Table 13: Per-category AP25 for the classes on the SUN RGB-D dataset.
bathtub
bed
bookshelf
chair
desk
dresser
nightstand
sofa
table
toilet
mAP
H3DNet [70]
73.8
85.6
31.0
76.7
29.6
33.4
65.5
66.5
50.8
88.2
60.1
BRNet [9]
76.2
86.9
29.7
77.4
29.6
35,9
65.9
66.4
51.8
91.3
61.1
GroupFree [30]
80.0
87.8
32.5
79.4
32.6
36.0
66.7
70.0
53.8
91.1
63.0
FCAF3D [43]
79.0
33,0
81.1
34.0
40.1
71.9
69.7
91.3
64.2
UnIDDETR (ours)
80.7
30.7
85.6
38.6
42.7
74.7
59.2
93.9
67.0
Table 14: Per-category AP50 for the 10 classes on the SUN RGB-D dataset.
bathtub
bookshelf
chair
desk
dresser
nightstand
sofa
table
toilet
mAP
HB3Nee [70]
8.6
20.6
45.6
BRNet
55.5
63.8
9.3
61.6
10.0
27.3
53.2
56.7
28.6
70.9
43.7
GroupFree [30]
64.0
67.1
12.4
62.6
14.5
21.9
49.8
58.2
29.2
72.2
45.2
FCAF3D [43]
66.2
69.8
11.6
68.8
14.8
30.1
59.8
58.2
35.5
74.5
48.9
Uni3DETR (ours)
67.4
66.2
10.7
71.7
14.8
33.5
60.3
63.0
36.7
78.6
50.3
We first list the per-category results for the 10 classes on the SUN RGB-D dataset in Tab. 13 and Tab.
14. For the AP25 metrie, Uni3DETR achieves the best for 9 classes out of the total 10 classes. The
17

---

ground-truth
3D lou
decoupled loU
Figure 8: The comparative visualized results of our decoupled IoU, compared to the result obtained from the normal 3D IoU.
ground-truth truth
Uni3DETR
Figure 9: The visualized results of Uni3DETR on the SUN RGB-D dataset.
18

---

ground-truth
Uni3DETR
Figure 10: The visualized results of Uni3DETR on the ScanNet dataset.
ground-truth
Uni3DETR
Figure 11: The visualized results of Uni3DETR on the S3DIS dataset.
Table 15: Per-category AP25
25 for the 18 classes on the ScanNet dataset.
Indg
VatgNer [R]
GSDN [17]
41.6
82.5
61.1
64.2
40.0
75.5
62.8
H3DNer 170]
38.6
64.9
62.0
57.3
67.4
67.2
GroupFreef.00
52.1
92.9
93.5
38.0
70.7
53.7
62.4
16.1
38.5
808
67.9
47.0
76.3
99.6
72.0
95.3
36.4
69.1
PCA [:3]
57.2
87.0
95.0
92.3
70.3
61.1
60.2
64.5
29.9
64.3
71.5
60.1
52.4
83.9
99.9
84.7
36.5
65.4
71.5
Un3DETRR lours)
38.1
37.0
94.9
91.2
71.7
66.9
58.5
59.6
34.6
732
61.
55.5
52.7
31.2
99.6
78.2
83.5
63.7
71.7
Table 16: Per-category APso for the 18 classes on the ScanNet dataset.
cab
bed
chair
wnf
doer
wind
Behhf
unt
deck
Indg
shows
Noth
mAP
VoigNet [38]
76.1
67.2
42.4
1.3
27.3
33.5
GSDN
30.5
8.5
25.9
H5DNer
4.2
GrapFree[30
(onrs)
most significant improvement comes from the sofa and table class, 5.4% and 6.2% respectively. For the AP50 metric, UniDETR also achieves the best for classes. For the sofa class, the improvement up to 4.8%. The effectiveness of UniDETR is thus further demonstrated.
19

---

ground-truth
Uni3DETR
Figure 12: The visualized results of Uni3DETR on the KITTI dataset.
ground-truth
Uni3DETR
Figure 13: The visualized results of Uni3DETR on the nuScenes dataset.
20

---

Table 17: Per-category AP25 for the 5 classes the S3DIS dataset.
table
chair
sota
bkease
board
mAP
GSDN
73.7
98.1
20.8
334
12.9
47.8
FCAF3D [43]
69.7
97.4
92.4
36.7
37.3
66.7
URIDDETR (ours)
98.7
70.1
Table 18: Per-category APso for the 5 classes on the S3DIS dataset.
table
chair
sola
bkease
board
mAP
GSDN [17]
36.6
75.3
6.1
6.5
1.2
25.1
FCAF3D [43]
88.3
UniSDETR (ours)
45,4
87.9
64.1
19.0
23.7
48.0
Table 19:
Per-category AP for the 10 classes on the nuScenes dataset.
NDS
mAP
Car
Truck
Bus
Trailer
C.V.
Ped
Mot
Byc
T.C.
Bar
UVIR [27]
67.7
60.9
85.3
53.0
69.1
41.4
24.0
82.6
70.4
52.9
67.1
63.4
Uni3DETR (ours)
68.5
61.7
87.0
59.0
70.8
41.7
23,9
86.1
66.4
46.0
67.8
68.0
The per-category results for the 18 classes the ScanNet dataset are listed in Tab. 15 and Tab. 16. We achieve the best for 7 classes for the AP25 metric and for classes for the AP5o metric. For the S3DIS dataset, the per-category results listed in Tab. 17 and Tab. 18. For the AP25 metric, achieve the best for 4 classes out of the total 5 ones. For the APso metric, we are 18.1% 1% higher than FCAF3D the board class. These per-category results further demonstrate the ability of Uni3DETR on indoor scenes.
We also list the per-category AP on the nuScenes dataset in Tab. 19. Our Uni3DETR achieves the best for out of 10 classes. The ability of Uni3DETR further demonstrated on outdoor scenes.
21


Reviews:
[{"review_id": "C5b9xxpgzU", "reviewer": "Reviewer_TGtP", "summary": "This paper proposes Uni3DETR, a unified 3D detection transformer that addresses indoor and outdoor 3D detection within the same framework. The paper provides a specific analysis of the inconsistency in the structure of current indoor and outdoor scene detection models. Due to the differences in data distribution between indoor and outdoor environments, indoor datasets have a smaller range of point clouds, with objects being closer together and denser, occupying a larger portion of the point cloud. On the other hand, outdoor scenes have smaller and sparser objects, with background point clouds dominating the overall point cloud. Utilizing the mixture of query points which consist of  the learnable and non-learnable query, the detector is able to exploit the global information for dense small-range indoor scenes and local information for large-range sparse outdoor scenes. Besides, decoupled IoU is proposed for easier and faster training target localization optimization  by disentangling the xy and z space. ", "questions": "1. Decoupled IoU is only used in Uni3DETR, and it can be seen from the ablation study that this loss has a significant performance improvement for the detector. However, Decoupled IoU loss was not used in other mothods when compared to other methods. Can Decoupled IoU improve other detectors, such as GroupFree3D, and also bring significant performance improvements?\n2. The learnable query mentioned in the paper lacks its initialization process, such as how to obtain $P_q$ and $C_q$ mentioned in line 107.\n3. How about the generalization ability of feature extraction based on this structure for indoor scenes? For example, how it performs zero-shot testing by training on SUN RGBD and directly testing the trained model on ScanNet?\n5. The author uses a mixture of query points to obtain detection results through different categories and a sufficient number of queries. Will this result in noticeable time consumption during inference?\n6. It would be better to illustrate more visualization results to support the claim that the learnable and non-learnable queries have different capabilities.\n7. According to the paper: ConQueR: Query Contrast Voxel-DETR for 3D Object Detection: DETRs usually adopt a larger number of queries than GTs (e.g., 300 queries v.s. ∼40 objects in Waymo) in a scene, which inevitably incur many false positives during inference. Could this problem occur in Uni3DETR since many more queries are used in inference?", "limitations": "It seems that the paper now provides a unified paradigm for solving 3D object detection tasks of indoor and out door scenes and do not use one model for testing on all the datasets. Furthermore, the model is separately trained with different hyper-parameters on different datasets.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a detailed analysis of the inconsistent models of indoor and outdoor 3D detectors, with the difference in data distribution being the most important reason. The author explains how to design a unified 3D detector and solves the problem of inconsistent architectures between indoor and outdoor scenes.\n2. The overall logic of the paper is clear, and it is well written.\n3. Uni3DETR demonstrates the strong generalization ability under heterogeneous conditions while maintaining caomparable performance.", "weaknesses": "1. The experiments of model architectures are a bit insufficient. In line 99, the paper mentions, \"Then, we convert the extracted sparse features into dense ones and apply 3D dense convolutions for further feature processing.\" However, no experiments show the performance gain brought by 3D dense convolution to support the claim. Furthermore, will the model architectures vary from datasets? For example, are the numbers of layers different for models trained on KITTI and S3DIS?\n2. The author claims that learnable queries primarily capture the local information of the scene, and non-learnable queries usually concentrate more on global information. Although experiments show that combining different queries could bring better performance, there is no evidence in the paper (e.g., visualization or other analysis) to support the claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The experiments of model architectures are a bit insufficient.**\n\nA1:\n* We conduct the experiments on the SUN RGB-D dataset in the below Tab. 1-1. Dense 3D convolutions contribute to the 6.3% AP25 improvement and 10.1% AP50 improvement. Its effectiveness to the performance can be demonstrated. As sparse convolutions are only conducted to the point positions in the 3D space, it will lead to the feature missing problem of center points. Dense convolutions help alleviate this problem, thus improving the detection accuracy.\n* The architecture of our model is totally the same for all different datasets, including indoor datasets SUN RGB-D, ScanNet, S3DIS and outdoor datasets KITTI, nuScenes. We are the first to build a unified 3D detector, which can perform both indoor and outdoor 3D object detection with the same architecture.\n\nTable 1-1: The performance on the SUN RGB-D dataset with or without 3D dense convolution layers.\n|      |  AP25  | AP50 |\n| :--: | :--: | :--: |\n|w/o dense conv | 60.7 | 40.2|\n|w/ dense conv | 67.0 | 50.3 |\n\n**Q2: There is no evidence in the paper (e.g., visualization or other analysis) to support the claim (the mixture of query points).**\n\nA2:\n* We provide visualized results about the mixture of query points in the section 2 of our supplementary material. With only learnable query points, the detector mainly leverages local information to detect objects in the scene. However, some objects, especially partly occluded and with insufficient points, are easily ignored and missed. This is because without enough points, the local information will be insufficient for these objects to be detected. By providing global information, learning with non-learnable query points can better consider the background. As a result, the detector can obtain an overall understanding of the whole scene, and such global understanding can help the model find and detect these objects.  Therefore, with the mixture of query points, more comprehensive detection results can be obtained.\n* We also provide three more visualized results in Fig. 1 of the rebuttal PDF file to better illustrate the effectiveness of our mixture of query points. We will put such a visualized analysis in the final version of our paper. \n\n**Q3: Can Decoupled IoU improve other detectors and also bring significant performance improvements?**\n\nA3: We involve our decoupled IoU into VoteNet and GroupFree3D, and conduct the experiments on the ScanNet dataset. As we can see, Decoupled IoU still helps improve the detection AP: 7.2% for VoteNet and 2% for GroupFree3D, which demonstrates the easy-to-optimize property of decoupled IoU also helps positional information learning in previous detectors. However, the improvement is less than that in our Uni3DETR. This is because our detector is based on the transformer structure, which is seriously affected by the issue of L1 loss for different scales. Therefore, decoupled IoU is more urgently demanded in our transformer-based detector.\n\nTable 1-2: The performance on the ScanNet dataset. We apply our decoupled IoU loss on the VoteNet and GroupFree3D.\n|      |  Decoupled IoU | AP25  | AP50 |\n| :--: | :--: | :--: | :--: |\n|VoteNet |  | 58.6 | 33.5 |\n|   |  √ | 65.8 | 46.2 | \n|GroupFree3D | | 69.1 |52.8|\n|  | √ |71.1 |53.8|\n\n**Q4. The learnable query mentioned in the paper lacks its initialization process, such as how to obtain $P_q$ and $C_q$ mentioned in line 107.**\n\nA4:\nWe initialize both the content query $C_q$ and learnable query points $P_q$ with the standard normal distribution, i.e. randomly initializing them using a commonly utilized standard Gaussian distribution with zero mean and unit standard deviation. As these queries will be learned during training, the initialization approach does not affect the ultimate performance too much.\n\n**Q5. How about the generalization ability for indoor scenes? For example, how it performs zero-shot testing by training on SUN RGBD and directly testing the trained model on ScanNet?**\n\nA5:\nWe perform inference on the ScanNet dataset with SUN RGB-D trained model, and compare it with two previous methods VoteNet and GroupFree3D. The results show that Uni3DETR can also obtain a satisfying result for this zero-shot testing experiment, obtaining a 57.4% AP25 and 38.6% AP50. Its zero-shot AP on ScanNet surpasses VoteNet by 4.9% and GroupFree by 6.2%. \n\nTable 1-3: The performance on the ScanNet dataset of a SUN RGB-D trained Uni3DETR.\n|      |  AP25  | AP50 |\n| :--: | :--: | :--: |\n| VoteNet | 52.5 | 31.7 |\n|GroupFree3D|51.2|23.3|\n|ours|57.4|38.6|\n\n**Q6. The author uses a mixture of query points. Will this result in noticeable time consumption during inference?**\n\nA6: As the 3D detection transformer is based on the 3D voxel feature after downsampling, the computational budget of the whole detection transformer is actually not large. As a result, the mixture of query points does not affect the inference time consumption too much. We conduct the complexity analysis on the mixture of query points in the below Tab. 1-4. Both the time consumption during inference (latency) and FLOPS retain nearly the same with the mixture of query points. This demonstrates that our mixture of query points does not introduce too much extra computation.\n\nTable 1-4: The complexity analysis about the latency and FLOPS about the mixture of query points.\n|  | latency (s) | FLOPS (G)|\n| :--: | :--: | :--: |\n|w/o mix query | 0.51 |452.23|\n|w/ mix query |0.52 (+ 1.9%) |458.74 (+1.4%)|\n\n**Q7: Could the large number query problem occur in Uni3DETR since many more queries are used in inference?**\n\nA7: During training, queries regarding negative samples will be assigned as the background category through Hungarian matching, and will be learned towards a low objectness score in the classification layer. Then, in the inference process, the confidence score of most false negatives will be low. Therefore, such a false negative problem does not quite matter in our problem. "}, {"review_id": "6MqXBmFYs1", "reviewer": "Reviewer_bu7C", "summary": "The paper proposes Uni3DETR, a unified architecture suitable for indoor and outdoor scenes. Uni3DETR employs the DETR with point-voxel interaction for object prediction. It uses a mixture of query points to exploit global and local information. Finally, Uni3DETR uses a decoupled IoU loss by disentangling the depth and other dimensions to ease out training. Training a per-dataset model and testing on the same dataset improves over the baselines.", "questions": "Please see the weakness.", "limitations": "The authors do not list out the limitations.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The problem of using a unified architecture for indoor and outdoor point clouds is super relevant.\n+ The idea of employing DETR, a mixture of query points, and decoupling 3D IoU is nice.\n+ The per-dataset results are good.", "weaknesses": "- The first claim that the architecture is uniform is only partially correct. L228 says that the authors chose a 0.02m grid size for indoor datasets, while they chose a different voxel size for the KITTI and the nuscenes datasets. The authors should pick a single voxel size and then run it on all the datasets. (It is OK to change the range for nuScenes since it is a multi-camera dataset, but the authors should choose a single voxel size)\n\n- The claim of \"strong universal ability of our model (L336)\" is an overstatement, in my opinion, since the paper only carries out Per-dataset model training and testing. A related paper that trains a single RGB 3D detector is Omni3D + Cube R-CNN [A]. I would urge the authors to follow a similar protocol where the train and test sets are\n   - KITTI + nuScenes front and then test on indoor indoor/outdoor\n   - Train on indoor scenes and then test on outdoor indoor/outdoor\n   - Finally, train on all indoor/outdoor scenes, then test on indoor/outdoor scenes.\n\nBy training Uni3DETR on KITTI + nuScenes front and testing outdoors, one can compare against the Cube R-CNN model of [A].\n\n- How well does the method generalize in zero-shot settings? E.g., How does the Uni3DETR model trained on outdoor scenes work on an unknown lidar scan from an unknown say, Waymo dataset.\n\n- A sad part of 3D detection papers is to use a bigger backbone and show improvements on nuScenes. Please quantitatively compare the flops, model size, training time, and inference time for Uni3DETR against the baselines in Table 3.\n\n- The authors say the learnable query points mostly contain local information and fit the outdoor detection well. In contrast, non-learnable query points emphasize global information and are more effective for dense indoor scenes. Another way to enforce local and global information is to use a dual-path architecture where one branch is attention-based for global information while the other is convolution-based for local information. One can later fuse their outputs by concatenation or predicting a coefficient to add them. Why did the authors not consider this design choice?\n\n- An alternative to using the decoupled IoU loss is the disentangled loss [B], where one constructs a 3D box from GT parameters except the one in consideration. That way, one can avoid explicitly weighing the losses for respective parameters (e.g., the authors use weighing xy and z terms by 0.5). The authors should, therefore, quantitatively compare the decoupled IoU loss against the disentangled loss.\n\n- The details of the 3D feature extractor (Section 2.1) need to be included. \n\n- The authors use the mean average precision (mAP) under IoU thresholds of 0.25 and 0.5 for evaluating datasets  (L216). Using one or two thresholds is quite sensitive. A better alternative is to use exact IoU3D over a series of thresholds of 0.05:0.5:0.05 and then average them as in [A].\n\nMinor:\n- Do the authors plan to release the code?\n\n- Typo: L203: final\n\nReferences:\n\nA. Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild, Brazil, Kumar, Ravi, et al., CVPR 2023.\n\nB. MonoDIS: Disentangling monocular 3D object detection: From single to multi-class recognition, Simonelli, Bulo, Porzi, et al., PAMI 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1. The claim that the architecture is uniform is partially correct.**\n\nA1:\n* The word “unified” in our paper specifically refers to the architecture aspect. The voxel size is a data-related parameter, not architecture-related. Since point clouds are collected with different sensors, their ranges and distributions vary significantly (about 3m for indoor but more than 70m for outdoor datasets). Here we follow the settings of previous works and utilize the same grid size. Then 3D voxels can be processed with a unified architecture. We are the first to build a unified 3D detector for both indoor and outdoor point clouds with the same architecture.\n* We also conduct the experiment with the same KITTI voxel size. The resolution of indoor voxels deteriorates, which hurts the detection. However, compared with other outdoor detectors, our superiority is still obvious. Therefore, even if standardizing such a data-related parameter, our model can still make a higher AP.\n\nTable 2-1: 3D detection with the same voxel size of (0.05m, 0.05m, 0.1m).\n| |AP-indoor (SUN RGB-D)|AP-outdoor (KITTI car)|\n| :--: | :--: | :--: |\n|3DSSD|9.5|78.6|\n|CenterPoint|18.9|74.4|\n|UVTR |35.9|72.0|\n|ours (a single voxel size)|47.3|86.7|\n|ours (different voxel sizes)|67.0|86.7|\n\n**Q2: The claim of \"strong universal ability of our model\" is an overstatement**\n\nA2:\n* The “universality ability” here refers specifically to our construction of a unified architecture for both indoor and outdoor detection. In the current point cloud based 3D detection research, indoor and outdoor detectors are still accomplished with totally distinct structures, and a unified architecture for per-dataset experiments is still absent. From this perspective, the universality ability of our detector is at least better than existing 3D detectors. We will revise it to “Extensive experiments demonstrate that our model can address both indoor and outdoor 3D detection with a unified structure” in the final version. \n* Cube RCNN only takes RGB images for 3D detection. Compared with 2D images, the dataset-interference issue is extremely serious for point clouds. More cross-dataset discrepancies, including sensor type differences (the RGB-D sensor v.s. LiDAR), scene changes (small-range and dense indoor v.s. large-range and sparse outdoor), object distribution (more sparse and small-size in outdoor scenes), make the joint combination of multiple point cloud datasets extremely challenging. As a result, there is still no work for joint training on multiple point cloud datasets. Actually, if we want one model like Cube RCNN but taking point clouds as input, a unified structure should inevitably serve as its prerequisite and foundation. Here we propose a unified architecture for different point clouds, as the foundation towards this. How to bridge the cross-dataset difference for point clouds is not our focus. \n* We also follow a similar protocol to compare with Cube RCNN - training on KITTI and nuScenes then evaluating on KITTI. We observe that even if we do not work on the cross-dataset difference, the performance is still better.\n\nTable 2-2: Comparison by training on KITTI and nuScenes front, then evaluating on KITTI.\n|  |AP-car |\n| :--: | :--: |\n|Cube RCNN|15.0|\n|ours|65.3|\n\n**Q3: How well does the method generalize in zero-shot settings**\n\nA3: We adopt two protocols: training on SUN RGB-D then evaluating on ScanNet; training on KITTI then evaluating on waymo. Its zero-shot AP on ScanNet surpasses GroupFree by 6.2%. Its outdoor zero-shot AP is also 5.2% better than existing outdoor detectors. This well demonstrates its generalization ability. \n\nTable 2-3: 3D detection on ScanNet of SUN RGB-D trained detectors.\n| |AP25|AP50|\n| :--: | :--: | :--: |\n|VoteNet|52.5|31.7|\n|GroupFree3D|51.2|23.3|\n|ours|57.4|38.6|\n\nTable 2-4: 3D detection on waymo of KITTI trained detectors.\n| |AP-car|\n| :--: | :--: |\n|SECOND|5.8|\n|PointPillar|12.1|\n|Part-A2|14.9|\n|ours|20.1|\n\n**Q4: Why did the authors not consider the dual path?**\n\nA4:\n* Convolution-based structures perform 3D box generation directly on the extracted features, thus are sensitive to data distinctions. Such structures thus cannot extract local information well for both indoor and outdoor scenes. We conduct the experiment, where the convolution branch is from Part-A2. Our model outperforms it by more than 10% for both indoor and outdoor datasets. This explains why we do not adopt the dual-path structure. \n* A dual-path architecture will also inevitably bring in more computation budget. \n\nTable 2-5: Comparison with the dual-path structure.\n| |AP-indoor (SUN RGB-D) |AP-outdoor (KITTI car)|\n| :--: | :--: | :--: |\n|dual-path |54.2 |75.6|\n|ours |67.0|86.7|\n\n**Q5: The authors should compare against the disentangled loss**\n\nA5: Our decoupled IoU is 12.5% higher than disentangled loss. The reason is that the original disentangled loss takes RGB images as input, and point clouds only will hurt its performance.\n \nTable 2-6: Comparison on KITTI between our decoupled IoU and the disentangled loss.\n| |AP-car|\n| :--: | :--: |\n|disentangled loss |74.24 |\n|decoupled IoU loss |86.74|\n\n**Q6: The 3D feature extractor details need to be included.**\n\nA6: We add the visualization of our 3D feature extractor in Fig. 2 of our rebuttal PDF file. \n\n**Q7: A better alternative is to use IoU3D over a series of thresholds of 0.05:0.5:0.05.**\n\nA7:\n\n* Here for indoor 3D detection, most previous detectors adopt the AP25 and AP50 metrics. For a fair comparison, we adopt the same metrics.\n* We adopt the same metric as Omni3D for evaluation. We observe that the improvement is consistent. This demonstrates that our model achieves better performance for both recognition and localization.\n\nTable 2-7: Comparison on SUN RGB-D with the Omni3D metric.\n| |AP[0.05:0.5:0.05] |\n| :--: | :--: |\n|VoteNet | 53.4|\n|GroupFree3D |56.8|\n|FCAF3D |60.6|\n|ours|64.3|\n\n**Q8: code release and typos**\n\nA8: We will correct typos in the final version and release the code.\n"}, {"review_id": "b76SUaQVEA", "reviewer": "Reviewer_iCdw", "summary": "Uni3DETR is a unified 3D object detector that is capable of handling both indoor and outdoor scenes within the same framework. This is significant as many existing detectors are specialized for either indoor or outdoor environments, but not both.\n\nThe method employs a detection transformer with a point-voxel interaction mechanism and a mixture of query points. This allows it to exploit both global and local information effectively.\n\nUni3DETR introduces a decoupled Intersection over Union (IoU), which serves as an easy-to-optimize training target for localization. This is aimed at improving the accuracy of the detector.\n\nThe paper demonstrates that Uni3DETR exhibits excellent performance consistently on both indoor and outdoor 3D detection tasks. Moreover, it shows strong generalization ability under heterogeneous conditions, which is beneficial for real-world applications.", "questions": "As evidenced in Table 5, the non-learnable query makes a small contribution to outdoor detection. Can you provide a comparable analysis of the impact of the mixed query points on nuScenes? I'm curious to determine if the performance on nuScenes is primarily driven by your learnable query.\n\nCould you supply a comparison of computational complexity and inference latency between your approach and alternative methods?", "limitations": "Integrating two distinct 3D detection algorithms in this method might potentially increase computational complexity. Nonetheless, computation overhead is a key factor for outdoor detection tasks, such as autonomous driving. Given this, the inclusion of a non-learnable query design seems to be superfluous in such outdoor environments.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Versatility Across Environments: One of the major strengths of Uni3DETR is its ability to handle both indoor and outdoor environments within the same framework. This is a significant advancement as it eliminates the need for different models for different environments, which is common in existing approaches.\n\nOptimization of Localization: The introduction of a decoupled Intersection over Union (IoU) as a training target for localization is a notable strength. This provides an easy-to-optimize objective that can potentially lead to more accurate localization of objects, which is often a challenging aspect of object detection.\n\nEmpirical Validation: The paper provides empirical evidence showing that Uni3DETR consistently performs well on both indoor and outdoor 3D detection tasks. This is important for establishing the credibility and effectiveness of the proposed approach.", "weaknesses": "Innovative Element. The core pipeline appears to be a Deformable DETR complemented by non-learnable queries (voxel features derived from FPS). It essentially resembles a simple combination of two model architectures.\n\nEfficiency. The model's efficiency is not entirely at the cutting-edge level. For instance, in nuScenes, several competitive 3D detection techniques are not listed in your tables. (https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Lidar).\n\nIncrease in Precision. According to Table 5, the non-learnable query appears to have a marginal impact on outdoor detection. This aligns with our prior understanding that FPS is not well-suited for outdoor detection. The mixed query's design might be superfluous for outdoor scenarios.\n\nComputational Complexity. It is widely recognized that greater computation often results in superior performance. The authors should include a breakdown of the mixed query's complexity, such as inference latency and FLOPS, in comparison to other techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: It essentially resembles a simple combination of two model architectures.**\n\nA1:\n* Our main contribution is that we propose a unified architecture to address both indoor and outdoor 3D detection within the same framework. Unlike previous methods, which consider indoor and outdoor 3D detection as two problems and address them with a separate set of benchmarks, we are the first to build a unified architecture and eliminate the need for different models in different environments. This is the main novelty of our work. Through Uni3DETR, it is demonstrated that although significant distinction exists for indoor and outdoor point clouds, a unified structure is still possible. Therefore, our framework can be heuristic and become a basic platform or foundation for future works.\n* A simple combination of two model architectures cannot address the problem of for both indoor and outdoor 3D detection. The deformable DETR is built on learned reference points, thus cannot take non-learnable points as the input. We thus modify its structure to directly take 3D points as the input, and perform cross-attention with voxel features. In addition, previous indoor detectors usually utilize FPS for learning point-wise features. Different from them, we utilize FPS points to remedy global information for learning voxel-wise features, which are more resistant to the data distinction. The learnable and non-learnable query points will interact with each other through attention, for a comprehensive understanding of both global and local information. By focusing on information of different levels, the mixture of query becomes a necessary condition for the success of both indoor and outdoor detection.\n* We also propose a decoupled IoU to provide an easy-to-optimize objective for localization. As can be seen from the ablation study, the decoupled IoU improves a lot for both indoor and outdoor detection, which is thus also a necessary element of our framework and a main contribution to the community for unifying indoor and outdoor detection with the same model.\n\n**Q2: In nuScenes, several competitive 3D detection techniques are not listed in your tables.**\n\nA2: Actually many methods in the nuScenes leaderboard apply many techniques, like larger and more complicated backbones or multiple model ensembling. To evaluate different methods, we should put them into the same setting. Under the same setting and comparing in a fair way, our method is actually better. Take the Real-Aug++ method (top 1 in the leaderboard) for example. It obtains the 81.7% AP on the car category of KITTI validation. In comparison, we obtain the 86.6% AP. The effectiveness of our method can thus be validated.\n\n**Q3. The mixed query's design might be superfluous for outdoor scenarios.**\n\nA3:\n* Indeed, introducing non-learnable query points is more effective for indoor 3D detection, as global information is more important for dense small-range indoor scenes. However, what we want is a unified architecture for both indoor and outdoor detection. Without the non-learnable query, the performance of 3D indoor detection will be limited because of the insufficiency of global information. Therefore, although the mixture of query points improves less for outdoor scenarios, it is still necessary.\n* In some certain situations of outdoor scenes, like small, unclear, or occluded objects, local information only will be insufficient, and the non-learnable query will become more effective. To demonstrate this, we conduct the 3D detection experiment on the 3 classes of KITTI (Tab. 3-1). The mixture of query points improves more for the pedestrian and cyclist class, 2.05% for pedestrian and 3.19% for cyclist. Therefore, although the mixed query is less effective for KITTI car, it is still effective for those small and occluded objects.\n* We also provide one visualized example from KITTI in the rebuttal PDF file. As can be seen from the third example of Fig. 1, the available points for the left car are insufficient, which makes it difficult to be detected. In this situation, by leveraging a global understanding of the scene, the non-learnable query helps detect it. Therefore, the non-learnable query is still effective for certain outdoor situations.\n\nTable 3-1: Effect of the mixture of query points on the 3 classes of the KITTI dataset.\n| |AP-car|AP-ped|AP-cyc|\n| :--: | :--: | :--: | :--: |\n|learnable query only|85.24|60.44|69.71|\n|mixture of query points|86.57|62.49|72.90|\n\n**Q4: The authors should include a breakdown of the mixed query's complexity**\n\nA4: The time consumption during inference (latency) retains nearly the same, and the FLOPS also does not increase too much. This demonstrates that the mixture of query points does not bring too much complexity, thus will not bring extra computation overhead.\n\nTable 3-2: The complexity analysis about the mixture of query points.\n| |latency (s)|FLOPS (G)|\n| :--: | :--: | :--: |\n|w/o mix query|0.51|452.23|\n|w/ mix query|0.52 (+ 1.9%)|458.74 (+1.4%)|\n\n**Q5: Can you provide a comparable analysis of the impact of the mixed query points on nuScenes?**\n\nA5: As can be seen, the mixture of query points brings a 1.9% mAP improvement. The improvement is a little more than KITTI, because nuScenes consists of more small objects. We further select four classes and list their per-category AP. For the car category, the AP improvement is 1%. For other categories with usually small or occluded objects, the improvement is more significant: the mixture of query points boost the pedestrian class by 2.1%, the motorcycle class by 3.6%, and 4% for the traffic cone class. In a word, the non-learnable query is equally effective for these small or occluded objects in outdoor scenes.\n\nTable 3-3: Effect of the mixture of query points on the nuScenes dataset. \n| |mAP|car|pedestrian|motorcycle|traffic cone|\n| :--: | :--: | :--: | :--: | :--: | :--: |\n|learnable query only|59.8|85.6|84.3|64.0|66.0|\n|mixture of query points|61.7|86.6|86.4|67.6|70.0|\n"}, {"review_id": "cS2H3SNvUt", "reviewer": "Reviewer_r5Yb", "summary": "The authors propose a  unified detr-style detector for both the indoor and outdoor 3D object detection. Besides common learnable query, they also adopt unlearnanle query sampled from raw point cloud.", "questions": "1. Uni3DETR follows the design of DETR, which is a ''sparse'' and end-to-end detector. However, Uni3DETR  uses 4 groups of query points for indepedent prediction and needs extra pos-procedure to filer redundant boxes.  The authors claim that the learnable/unlearnable queries capture the local and global information of the scene.  If so many different queries are neccesary to cover the whole scene, why not use a dense detector like pointpillar or centerpoints instead?\n2. I am surprised that the results in Tab. 6. I hope the authors could provide more explanation about why multiple groups of queries brings no performance gain?  Why this ablation is not conducted on other datasets (e.g. KITTI)  like Tab.5 and Tab. 7?\n3. Using 4 groups of query points (learnable  & unlearnable) as proposed in the paper, or simply use more learnable queries, which one is better?\n4. The authors are suggestd to provide results on the KITTI and nuscenes test split. Besides,  KITTI provides a more fair and relibale evaluation setting with 40 recall positions.\n5. What is the diffence between Uni3DETR and DAB-Deform in Tab. 4.\n6. With differenent backbone, model size, inference speed, ect., it is not fair for the comparison with other sota methods in Tab.2 & Tab.3. \n", "limitations": "The novelty is limited, and more comprehensive analysis is required.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method is simple and easy to follow.", "weaknesses": "The novelty is limted, and the experimental results are not fair/convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Why not use a dense detector like pointpillar or centerpoints instead?**\n\nA1:\n* The main reason is that these dense detectors are usually based on the 3D convolution structure and 3D anchors. They perform 3D box generation directly on the extracted features, thus are sensitive to the distinction from the point cloud data. As a result, they cannot perform 3D indoor and outdoor detection with the same structure. In comparison, our 3D detection transformer directly matches 3D predicted boxes and ground-truth boxes, thus tends to be resistant to the difference from data. Therefore, our architecture is more suitable for 3D detection in various environments.\n* We further list the indoor performance on SUN RGB-D and outdoor performance on KITTI of these methods in the following Tab. 4-1. As these dense detectors are designed for outdoor 3D detection, their performance deteriorates seriously for indoor detection. In comparison, our method obtains a satisfying detection performance for both indoor and outdoor detection.\n\nTable 4-1: The performance comparison on the indoor SUN RGB-D and outdoor KITTI dataset with dense detectors PointPillar and CenterPoint. The metrics are AP25 for SUN RGB-D and AP70 for the KITTI car class.\n| |AP-indoor (SUN RGB-D)|AP-outdoor (KITTI)|\n| :--: | :--: | :--: |\n|PointPillar|N/A|77.6|\n|CenterPoint|18.9|74.4|\n|ours|67.0|86.7|\n\n**Q2: I hope the authors could provide more explanation about why multiple groups of queries brings no performance gain? Why this ablation is not conducted on other datasets (e.g. KITTI) like Tab.5 and Tab. 7?**\n\nA2:\n* The reason is that multiple groups of queries simply utilize local information multiple times, and does not provide complementary information. Therefore, it cannot bring further performance improvement. The reason why multiple groups are effective for 2D detection is mainly that it speeds up convergence, which is one of the main problems of transformer-based models for 2D detection. However, in 3D detection, since 3D voxel features contain stronger spatial information, convergence has not been a major problem for transformer-based 3D detectors. Therefore, multiple groups of queries are no longer effective for 3D detection.\n* To validate this, we further conduct such an ablation study on the KITTI dataset and list the comparison in the Tab. 4-2 below. It can be seen that our mixture of query points helps improve AP-car by 0.67%. However, multiple groups of queries fail to improve the 3D detection performance. Such comparison on the outdoor dataset validates that multiple groups of queries cannot boost the performance of 3D detection.\n\nTable 4-2: Comparison with multiple groups of learnable query points on the KITTI dataset. \n|query|AP-car|\n| :--: | :--: |\n|{$P_l$}|85.59|\n|{$P_l$}x2|85.32|\n|{$P_l$}x3|85.18|\n|{$P_l$}x5|85.43|\n|{$P_l$, $P_{nl}$}|85.94|\n|{$P_l$, $P_{nl}$, $P_{nlv}$}|86.26|\n\n**Q3: Using 4 groups of query points (learnable & unlearnable) as proposed in the paper, or simply use more learnable queries, which one is better?**\n\nA3: We conduct the experiment on the SUN RGB-D dataset by simply using 4 times of learnable queries, and list the comparison of learnable query only and our mixture of query points in the following Tab. 4-3. It can be seen that by using more learnable queries, both AP25 and AP50 decrease.  Using more learnable queries cannot provide complementary information, and will even make it more difficult for Hungarian matching. More false positive examples may thus appear, which hurt the performance. In comparison, by using the mixture of query points, we comprehensively utilize the global and local information, thus is better for 3D object detection.\n\nTable 4-3: Comparison with more learnable queries on the SUN RGB-D dataset.\n||AP25|AP50|\n| :--: | :--: | :--: |\n|learnable only|62.6|46.4|\n|query number x 4|61.7|43.6|\n|learnable + non-learnable|67.0|50.3|\n\n**Q4: The authors are suggested to provide results on the KITTI and nuscenes test split. Besides, KITTI provides a more fair and reliable evaluation setting with 40 recall positions.**\n\nA4:\n\nWe conduct the experiment and evaluate our method on the KITTI test set with 40 recall positions. The comparison is listed in the below Tab. 4-4. For the most important KITTI metric, AP on the moderate level of car, we obtain the 82.26% AP, which is 0.83 points higher than PV-RCNN and 0.49 points higher than CT3D.  The consistent superiority further demonstrates the ability of Uni3DETR on outdoor 3D detection.\n\nDue to time constraints, we have not yet been able to provide results on the NuScenes test set. We will include such experimental comparisons in the final version.\n\nTable 4-4: The performance of Uni3DETR for outdoor 3D object detection on the KITTI test set with 40 recall positions. *: AP on the moderate car is the most important metric for KITTI.\n| |easy|moderate*|hard|\n| :--: | :--: | :--: | :--: |\n|SECOND|88.61|78.62|77.22|\n|PointPillar|82.58 |74.31|68.99|\n|Part-A2|87.81|78.49 |73.51|\n|PV-RCNN|90.25 |81.43 |76.82|\n|CT3D|87.83 |81.77 |77.16|\n|ours|91.14|82.26|77.58|\n\n**Q5: What is the difference between Uni3DETR and DAB-Deform in Tab. 4.**\n\nA5: The detection transformer in Uni3DETR takes 3D points as the input of the deformable attention, while DAB-Deform takes 3D anchor boxes (7 dims) as the input. The superiority of taking 3D points as input is mainly twofold. The first is that 3D anchors are negatively affected by the center point missing in 3D point clouds, which makes formulating queries as 3D anchors less effective. In comparison, formulating queries as 3D points better adapts the structure of 3D data, thus achieving better performance. The second is that this accommodates our designed mixture of query points, since FPS points can directly forward into the detection transformer. The mixture of query points can bring further improvement. Ultimately, our Uni3DETR achieves the 66.4% AP (v.s. the 62.0% AP from DAB-Deform).\n"}, {"review_id": "ZfXjDE7MhY", "reviewer": "Reviewer_Q3d3", "summary": "The authors explore and analyze the existing LiDAR-based 3D object detection framework, and propose to adopt transformer-based method to perform detection from differnt LiDAR inputs. The experimental results on a number of datasets are better compared to the existing methods.", "questions": "Please refer to the questions that I describe in the Weakness part. I would also consider the rebuttal and other reviews.", "limitations": "NA.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The task of 3D object detection is very important in the 3D community. Interestingly, the authors propose to adopt transformers to generalize 3D detection across diverse scene inputs.\n\n2. The paper is easy to follow.\n\n3. The authors conduct the experiments on widely-used indoor and outdoor datasets, including  SUN RGB-D, ScanNet, S3DIS,m nuScenes, and KITTI.", "weaknesses": "1. Missing important SOTA works on KITTI. The manuscript does report validation results on KITTI, however, the authors forgot to compare with LiDAR-only SOTAs 'PV-RCNN++' (PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection). However, it would be much convincing if the authors can compare with this SOTA.\n\n2. Results on KITTI and nuScenes test set. In table 2 & 3, the proposed Uni3DETR is comparable/better with the cited methods. However, it would be much convincing if the authors can report the test set results on KITTI and nuscenes.\n\n3. Computation/memory footprint comparison. The authors didn't make a comparison of their work in terms of memory/speed with the existing 3D detection methods. The time consumption might be large since the memory/time are espeically heavy for the transformer-based methods from my point. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: It would be much convincing if the authors can compare with this SOTA (PV-RCNN++)**\n\nA1: To compare with PV-RCNN++, we follow the same setting to train our model on the training and validation set of KITTI, and evaluate on the KITTI test set. The comparison is listed in the below Tab. 5-1. Our method obtains the 82.26% AP on the car category of the KITTI test set, surpassing PV-RCNN++ by 0.38 points. This validates the effectiveness of our method.\n\nTable 5-1: The comparison of Uni3DETR against PV-RCNN++ on the KITTI test set with 40 recall positions.\n|  |AP-car|\n| :--: | :--: |\n|PV-RCNN|81.43|\n|PV-RCNN++|81.88| \n|ours|82.26|\n\n**Q2: It would be much convincing if the authors can report the test set results on KITTI and nuscenes.**\n\nA2:\n\nWe conduct the experiment and evaluate our method on the KITTI test set with 40 recall positions. The comparison is listed in the below Tab. 5-2. For the most important KITTI metric, AP on the moderate level of car, we obtain the 82.26% AP, which is 0.83 points higher than PV-RCNN and 0.49 points higher than CT3D.  The consistent superiority further demonstrates the ability of Uni3DETR on outdoor 3D detection.\n\nDue to time constraints, we have not yet been able to provide results on the NuScenes test set. We will include such experimental comparisons in the final version.\n\nTable 5-2: The performance of Uni3DETR for outdoor 3D object detection on the KITTI test set with 40 recall positions. *: AP on the moderate car is the most important metric for KITTI.\n| |easy|moderate*|hard|\n| :--: | :--: | :--: | :--: |\n|SECOND|88.61|78.62|77.22|\n|PointPillar|82.58 |74.31|68.99|\n|Part-A2|87.81|78.49 |73.51|\n|PV-RCNN|90.25 |81.43 |76.82|\n|CT3D|87.83 |81.77 |77.16|\n|ours|91.14|82.26|77.58|\n\n"}]

Author rebuttal:
Dear all reviewers

We sincerely thanks for valuable comments and suggestions. We first address the common concerns, followed by detailed responses to each reviewer separately. We hope our responses clarify existing concerns and make these points clear. 

**Q: Comparison of computational complexity against the baselines in Table 3.**

A:
* In this paper, we mainly target a unified structure. To ensure that the detector can accommodate both indoor and outdoor detection, we have, to a certain extent, made sacrifices in terms of efficiency, in order to prioritize its unified ability. We list the comparison of both performance and efficiency in the following Table. We can observe that the computational budget compared with these baselines is not significant: the inference time (latency) is almost the same as UVTR and the FLOPS is only 1.16% more than UVTR. In addition, we obtain significantly better detection performance on both indoor and outdoor datasets. The indoor AP is 16.8% higher than UVTR, and the average of indoor and outdoor AP is also 8.8% higher than it. Compared with VoxelNeXt, one model that mainly focuses on reducing the FLOPS of 3D detectors, we achieve more than 40% indoor AP improvement and more than 25% average AP improvement. Meanwhile, the increase in training time is also negligible:  2d 10h v.s. 2d 7h of UVTR. This demonstrates that our model can obtain better detection performance on both indoor and outdoor datasets, with a relatively subtle increase in computational complexity. 
* As the first work to build a unified architecture for both indoor and outdoor detection, we mainly focus on the performance level. From now on, there is still no work that can achieve a good performance on both indoor and outdoor datasets. Therefore, here our Uni3DETR mainly addresses the problem of achieving a good detection performance on both indoor and outdoor datasets with a unified architecture. For model efficiency, there are many other general approaches to address it. For example, we can adjust the number and ratio of sparse and dense convolution layers. We can leave this as one of our future work.

Table: The comparison of both performance and efficiency of our Uni3DETR against previous works on the indoor SUN RGB-D dataset and the outdoor nuScenes dataset. The metrics are AP25 for SUN RGB-D and mAP for nuScenes.
|method |performance | | | efficiency| | |
| :--: | :--: | :--: | :--: | :--: | :--: | :--: |
| |avg. |AP-indoor |AP-outdoor |latency (s) |params (M) |FLOPS (G)
|Centerpoint (CVPR 2021) |37.75 |18.9 |56.6 |0.32 |9.17 |121.10 |
|PillarNet (ECCV 2022) |44.00 |28.2 |59.8 |0.31 |12.55 |100.10 |
|VoxelNeXt (CVPR 2023) |39.30 |18.1 |60.5 |0.29 |7.12 |42.57 |
|UVTR (NeurIPS 2022) |55.55 |50.2 |60.9 |0.51 |26.12 |451.12 |
|ours |64.35 |67.0 |61.7 |0.52 |26.71 |458.74|



Meta-review: This work proposes a unified 3D detection framework that can be used for both outdoor and indoor environments.  

Overall, reviewers are positive on this work.  The AC agrees that this is a important step forward as most work in current 3D detection focuses on just either outdoor or indoor, and commonly even just a single dataset.  Reviewers also point out that the work is not as "unified" as it can be as separate models, with different parameters such as voxel-size need to be trained for indoor vs outdoor.  Still by presenting an unified framework with an initial set of experiments,  this work can encourage researchers to further investigate this direction. 

Authors are encouraged to incorporate reviewer suggestions and clarifications provided during the author response period into their paper (or supplemental materials) including:
1. Incorporate additional experiments provided for the rebuttal (comparison of computational cost, cross-dataset performance, affect of voxel size and other model design choices, additional comparisons against prior work, etc).   
2. Incorporate clarifications, and improve writing to make sure claims about unified architecture is precise and tone down claims of "strong universal ability". 
3. Adding discussion of limitations of the experiments and clarify key differences when training models for indoor vs outdoors.
4. Adding discussion of recent work that investigates having single model for multiple datasets for 3D object detection: 
- Uni3D: A Unified Baseline for Multi-dataset 3D Object Detection, Zhang et al. CVPR 2023
Decision: Accept (poster)