Title: Uni3DETR: Unified 3D Detection Transformer
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3d object detection

unified object detection

point clouds
Abstract: Existing point cloud based 3D detectors are designed for the particular scene, either indoor or outdoor ones. Because of the substantial differences in object distribution and point density within point clouds collected from various environments, coupled with the intricate nature of 3D metrics, there is still a lack of a unified network architecture that can accommodate diverse scenes. In this paper, we propose Uni3DETR, a unified 3D detector that addresses indoor and outdoor 3D detection within the same framework. Specifically, we employ the detection transformer with point-voxel interaction for object prediction, which leverages voxel features and points for cross-attention and behaves resistant to the discrepancies from data. We then propose the mixture of query points, which sufficiently exploits global information for dense small-range indoor scenes and local information for large-range sparse outdoor ones. Furthermore, our proposed decoupled IoU provides an easy-to-optimize training target for localization by disentangling the $xy$ and $z$ space. Extensive experiments validate that Uni3DETR exhibits excellent performance consistently on both indoor and outdoor 3D detection. In contrast to previous specialized detectors, which may perform well on some particular datasets but suffer a substantial degradation  on different scenes, Uni3DETR demonstrates the strong generalization ability under heterogeneous conditions (Fig. 1).

Uni3DETR: Unified 3D Detection Transformer

Abstract

Existing point cloud based 3D detectors are designed for the particular scene, either indoor or outdoor ones. Because of the substantial differences in object distribution and point density within point clouds collected from various environments, coupled with the intricate nature of 3D metrics, there is still a lack of a unified network architecture that can accommodate diverse scenes. In this paper, we propose Uni3DETR , a unified 3D detector that addresses indoor and outdoor 3D detection within the same framework. Specifically, we employ the detection transformer with point-voxel interaction for object prediction, which leverages voxel features and points for cross-attention and behaves resistant to the discrepancies from data. We then propose the mixture of query points, which sufficiently exploits global information for dense small-range indoor scenes and local information for largerange sparse outdoor ones. Furthermore, our proposed decoupled IoU provides an easy-to-optimize training target for localization by disentangling the xy and z space. Extensive experiments validate that Uni3DETR exhibits excellent performance consistently on both indoor and outdoor 3D detection. In contrast to previous specialized detectors, which may perform well on some particular datasets but suffer a substantial degradation on different scenes, Uni3DETR demonstrates the strong generalization ability under heterogeneous conditions (Fig. 1). Codes are available at https://github.com/zhenyuw16/Uni3DETR.

clustering clustering

clustering classification

classification classification

to bev to bev

to bev

1 Introduction classification & detection classification & detection classification & detection

3D object detection from point clouds aims to predict the oriented 3D bounding boxes and the semantic labels for the real scenes given a point set. Unlike mature 2D detectors [41, 19, 56, 4] on RGB images, which have demonstrated the ability to effectively address diverse conditions, the problem of 3D object detection has been considered under different scenarios, leading to the development of distinct benchmarks and methodologies for each. Specifically, 3D detection approaches are currently addressed separately for indoor and outdoor scenes.

detection detection

classification transformer

PointNet or

PointNet or

PointNet or

3D sparse conv

3D sparse conv

3D sparse conv grouping &

grouping &

grouping &

clustering clustering

clustering classification

3D sparse conv

3D sparse conv

3D sparse conv

2D dense conv

2D dense conv

2D dense conv in BEV

in BEV

in BEV

classification transformer

classification transformer

classification classification

PointNet or

3D sparse conv

indoor point clouds

outdoor point clouds

3D sparse conv

2D dense conv

in BEV

point clouds

3D sparse conv

3D dense conv

detection

transformer

grouping &

clustering

classification

classification

(a)

PointNet or

3D sparse conv

indoor point clouds

outdoor point clouds

3D sparse conv

2D dense conv

in BEV

point clouds

3D sparse conv

3D dense conv

detection

transformer

grouping &

clustering

classification

classification

(b)

PointNet or

3D sparse conv

indoor point clouds

outdoor point clouds

3D sparse conv

2D dense conv

in BEV

point clouds

3D sparse conv

3D dense conv

detection

transformer

grouping &

clustering

classification

classification

(c)

Indoor 3D detectors [38, 70, 62, 58] usually adopt the 'grouping, clustering and classification' manner (Fig. 2a), while outdoor detectors [64, 24, 52, 49] typically convert the features into the 2D bird's-eye-view (BEV) space (Fig. 2b). Although these two tasks only differ in their respective application contexts, the optimal approaches addressing them exhibit significant differences.

The key challenge in developing a unified 3D detector lies in the substantial disparities of point cloud data collected in indoor and outdoor environments. In general, indoor scenarios are cluttered, where various objects are close and dense , occupying the majority of the scene. In comparison, objects in outdoor scenes are small and sparse , where background points dominate the collected point clouds. Such disparities result in the lack of a unified 3D detector: 1) For the detection head, because of the severe distraction of excessive background points in outdoor point clouds, together with the hyperparameter sensitivity associated with grouping, the grouping-based indoor detectors are infeasible for outdoor 3D detection. Besides, outdoor objects are separated clearly and not overlapped in the BEV space, which does not apply to indoor objects. The height overlap among indoor objects makes the manner of detecting in the BEV space inappropriate for indoor detection. 2) For the feature extractor, existing backbones in 3D detectors are similarly designed for a singular scene. Indoor detectors are usually equipped with point-based [39, 40] or 3D sparse convolution based [10, 43] backbone, where point-based models are usually susceptible to the diverse structures of points under different scenes, and sparse convolution models are deficient in representing the features of object centers. In contrast, the 2D convolutions in outdoor detectors for extracting BEV features easily lead to information loss for indoor detection.

In this paper, we propose a Unified 3D DEtection TRansformer ( Uni3DETR ) based on only point clouds for detecting in diverse environments (Fig. 2c). Two attributes of our Uni3DETR contribute to its universality for both indoor and outdoor scenes. First, we employ a hybrid structure that combines 3D sparse convolution and dense convolution for feature extraction. The pure 3D architecture avoids excessive height compression for indoor point clouds. Simultaneously, the sparse convolution prevents the huge memory consumption for large-range outdoor data, and the dense convolution alleviates the center feature missing problem for sparse outdoor points. Second, we utilize transformer [57] for 3D object detection. The set-to-set prediction way in transformer-based detectors [4, 77] directly considers predicted objects and ground-truth labels, thus tends to be resistant to the distinction from data themselves. The transformer decoder is built on the extracted voxel feature and we formulate queries as 3D points from the scene. The points and voxels interact through cross-attention, which well adapts to the characteristics of 3D data.

Based on our 3D detection transformer, we further propose two necessary components for universality under various scenes. One is the mixture of query points. Specifically, besides the learnable query, we introduce the non-learnable query initialized by sampling the original points and the voxelized points, and integrate the learnable and non-learnable queries for feeding the transformer. We observe that the learnable query points mostly contain local information and fit the outdoor detection well, while non-learnable query points emphasize global information thus are more effective for dense indoor scenes. The other one is the decoupled 3D IoU. Compared to the usual 3D IoU, we decouple the x, y and z axis in the decoupled IoU to provide stronger positional regularization, which not only involves all directions in the 3D space but also is beneficial for optimization in transformer decoders.

3D feature extractor

input point cloud

voxelize

3D sparse

conv

3D dense

conv

3D detection transformer

non-learnable

query points

learnable

query points

random points

(test only)

gt

boxes

mixture of query points

Self-Attention

Self-Attention

Self-Attention

Cross-Attention

voxel features

query points

Hungarian

matching

IoUde =

(IoUxy +IoUz)/2

Self-Attention indoor Our main contributions can be summarized as follows:

voxel feature FPS non-learnable query box set 1 · We propose Uni3DETR, a unified 3D detection transformer that is designed to operate effectively in both indoor and outdoor scenes. To the best of our knowledge, this is the first point cloud based 3D detector that demonstrates robust generalization across diverse environments.

We propose the mixture of query points that collaboratively leverages the learnable and nonlearnable query. By aggregating local and global information, our mixture of queries fully explores the multi-scene detection capacity.

We decouple the multiple directions in the 3D space and present the decoupled 3D IoU to bring more effective positional supervision for the transformer decoder.

Our Uni3DETR achieves the state-of-the-art results on both indoor [53, 12, 1] and outdoor [16, 3] datasets. It obtains the 67.0% and 71.7% AP 25 on the challenging SUN RGB-D and ScanNet V2 indoor datasets, and the 86.7% AP in the moderate car category on KITTI validation. On the challenging nuScenes dataset, Uni3DETR also achieves the 61.7% mAP and 68.5% NDS.

2 Uni3DETR

2.1 Overview

We present the overall architecture of our Uni3DETR in Fig. 3. It consists of a 3D feature extractor and the detection transformer for detecting in various 3D scenes. The mixture of query points is fed into transformer decoders to predict 3D boxes, under the supervision of decoupled 3D IoU.

3D feature extractor. Existing 3D detectors usually adopt point-based [40] or voxel-based [64, 43] backbones for feature extraction. Considering that point-based backbones are vulnerable to the specific structure of point clouds themselves and less efficient in point set abstraction, we utilize the voxel-based model for extracting 3D features. After voxelization, we utilize a series of 3D sparse convolution layers to encode and downsample 3D features, to avoid the overload memory consumption for large-range outdoor scenes. Then, we convert the extracted sparse features into the dense ones and apply 3D dense convolutions for further feature processing. The dense convolution alleviates the feature missing problem of center points.

2.2 Unified 3D Detection Transformer for Diverse Scenes

Detection transformer with point-voxel interaction. We employ the transformer structure based on the extracted voxel features and the set prediction manner for 3D detection. Motivated by recent 2D transformer-based detectors [29, 25, 69] that formulate queries as anchor boxes, we regard 3D points in the 3D space as queries. Its structure is illustrated in Fig. 4.

Specifically, we denote P q = ( x q , y q , z q ) as the q -th point to represent the q -th object, C q ∈ R D is its content query, where D is the dimension of decoder embeddings. We introduce the deformable box set 2

box set 3

attention [77] for the cross-attention module and view P q as the reference point for point-voxel interaction. Unlike the deformable DETR decoder, here we directly learn the reference point P q . Suppose the voxel feature as V , the process of the cross-attention is modeled as:

where PE denotes the sinusoidal positional encoding. The transformer decoder predicts the relative positions for the query points: (∆ x q , ∆ y q , ∆ z q ) . Based on the relative predictions, the query points are refined layer-by-layer.

Compared to 2D detection transformers, because of the stronger positional information in the 3D space, our detection transformer requires less layers for query refinement. To further utilize the information across multiple layers, we average predictions from all transformer decoders except for the first one.

Mixture of query points. In the learning process, the above query points gradually turn around the object instances for better 3D detection. As they are updated during training, we refer them as learnable query points . Since such learnable query points finally concentrate on the points near the objects, they primarily capture the local information of the scene. For outdoor point clouds, where objects are sparse and small within the large-range scene, the learnable query points help the detector avoid the disturbance of the dominated background points.

In comparison, for the indoor scene, the range of 3D points is significantly less and the objects in relation to the entire scene are of comparable size. In this situation, besides the local information, the transformer decoder should also utilize the global information to consider the whole scene. We thus introduce the non-learnable query points for global information. Specifically, we use the Farthest Point Sampling (FPS) [40] for the input point cloud data and take the sampled points as the query points. These query points are frozen during training. Because of the FPS, the sampled points tend to cover the whole scene and can well compensate for objects that learnable query points ignore.

As we utilize the voxel-based backbone for feature extraction, vox- elization is a necessary step for processing the point cloud data. After voxelization, the coordinates and numbers of the raw point clouds may change slightly. To consider the structure of both the raw data and points after voxelization, what we use is two kinds of non-learnable query points P nl and P nlv . P nl denotes the non-learnable query points from FPS the original point clouds, and P nlv comes from sampling the points after voxelization. Together with the learnable query points P l , the mixture of query points ultimately consists of three ones: P = { P l , P nl , P nlv } . We follow [5] to perform group-wise self-attention in the transformer decoders for the mixture of query points, where these three kinds of query points do not interact with each other. After transformer decoders, three sets of predicted 3D boxes { bb l , bb nl , bb nlv } are predicted respectively. We apply one-to-one assignment to each set of 3D boxes independently using the Hungarian matching algorithm [23] and calculate the corresponding training loss to supervise the learning of the detection transformer.

At the test time, besides the above three kinds of query points, we further generate a series of additional points uniformly and randomly in the 3D voxel space as another set of query points P rd . These randomly generated points evenly fill the whole scene, thus well compensating for some potentially missed objects, especially for the indoor scenes. As a result, four groups of query points P = { P l , P nl , P nlv , P rd } are utilized for inference. Ultimately, four sets of predicted 3D boxes { bb l , bb nl , bb nlv , bb rd } are produced at the test time by Uni3DETR. Because of the one-to-one assignment strategy, the individual box sets do not require additional post-processing methods for removing duplicated boxes. However, there are still overlapped predictions among these four sets. We thus conduct box merging finally among the box sets to fully utilize the global and local information and eliminate duplicated predictions. Specifically, we cluster the 3D boxes based on the 3D IoU among them and take the median of them for the final prediction. The confidence score of the final box is the maximum one of the clustered boxes.

Deformable

Attention

V

Self-Attention

point query

x y

z

voxel feature

K

FFN

2.3 Decoupled IoU

Existing transformer-based 2D detectors usually adopt the L1 loss and the generalized IoU loss [42] for predicting the bounding box. The generalized IoU here mitigates the issue of L1 loss for different scales thus provides further positional supervision. However, for 3D bounding boxes, the calculation of IoU becomes a more challenging problem. One 3D box is usually denoted by: bb = ( x, y, z, w, l, h, θ ) , where ( x, y, z ) is the center, ( w,l, h ) is the size and θ is the rotation. The usual 3D IoU [72] between two 3D boxes bb 1 and bb 2 is calculated by:

where Area denotes the area of rotated boxes in the xy space.

From Equ. 2, we notice that the calculation of 3D IoU generally consists of two spaces: the xy space for the bounding box area, involving the x, y, w, l, θ variables, and the z space for the box height. The two spaces are multiplied together in the 3D IoU. When supervising the network, the gradients of these two spaces are coupled together - optimizing in one space will interfere with another, making the training process unstable. For 3D GIoU [72], besides the negative coupling effect, the calculation of the smallest convex hull area is even non-optimizable. As a result, the usual 3D IoU is hard to optimize for detectors. For our detection transformer, where L1 loss already exists for bounding box regression, the optimization problem of 3D IoU severely diminishes its effect in training, thus being hard to mitigate the scale problem of L1 loss.

Therefore, an ideal metric for transformer decoders in 3D should at least meet the following demands: First, it should be easy to optimize. Especially for the coupling effect for different directions, its negative effect should be alleviated. Second, all shape properties of the 3D box need to be considered, so that accurate supervision signals can be provided for all variables. Third, the metric should be scale invariant to alleviate the issue of L1 loss. Motivated by these, we forward the decoupled IoU:

Specifically, our decoupled IoU is the average of the IoU in the xy space and the z space. Under the summation operation, the gradients from the two items remain separate and independent, avoiding any coupling effect. According to previous research [47], the coupling issue does not exist in 2D or 1D IoU. As a result, the decoupled IoU effectively circumvents the negative effects of coupling. The scale-invariant property of 3D IoU is also well-reserved for the decoupled IoU. These make our decoupled IoU well suit the training of the transformer decoder.

The decoupled IoU is introduced in both the training loss and the matching cost. We also introduce it into classification and adopt the variant of quality focal loss [26]. Denote the binary target class label as c , the predicted class probability as ˆ p , the classification loss for Uni3DETR is:

where ˆ α t = α · c · IoU de +(1 -α ) · (1 -c · IoU de ) . The above classification loss can be viewed as using the soft target IoU de in focal loss [28]. The decoupled IoU is easy to optimize, allowing us to avoid the need for performing the stop-gradient strategy on IoU de . Therefore, the learning of classification and localization is not separated, assisting Uni3DETR predicting more accurate confidence scores. Equ. 4 will force the predicted class probability ˆ p towards IoU de , which might be inconsistent with IoU 3 D used in evaluation. We thus follow [22] and introduce an IoU-branch in our network to predict the 3D IoU. The normal IoU 3 D supervises the learning of the IoU-branch and the binary cross-entropy loss is adopted for supervision. The weighted geometric mean of the predicted 3D IoU and the classification score is utilized for the final confidence score. The final loss function for training Uni3DETR is thus the classification loss (Equ. 4), the L1 loss and the IoU loss with IoU de for bounding box regression, and the binary cross-entropy loss for IoU prediction.

3 Experiments

To demonstrate the universality of our Uni3DETR under various scenes, we conduct extensive experiments in this section. We evaluate Uni3DETR in the indoor and outdoor scenes separately.

Datasets. For indoor 3D detection, we evaluate Uni3DETR on three indoor 3D scene datasets: SUN RGB-D [53], ScanNet V2 [12] and S3DIS [1]. SUN RGB-D is a single-view indoor dataset with 5,285 training and 5,050 validation scenes, annotated with 10 classes and oriented 3D bounding boxes. ScanNet V2 contains 1,201 reconstructed training scans and 312 validation scans, with 18 object categories for axis-aligned bounding boxes. S3DIS consists of 3D scans from 6 buildings, 5 object classes annotated with axis-aligned bounding boxes. We use the official split, evaluate our method on 68 rooms from Area 5 and use the rest 204 samples for training. We use the mean average precision (mAP) under IoU thresholds of 0.25 and 0.5 for evaluating on these three datasets.

For outdoor 3D detection, we conduct experiments on two popular outdoor benchmarks: KITTI [16] and nuScenes [3]. The KITTI dataset consists of 7,481 LiDAR samples for its official training set, and we split it into 3,712 training samples and 3,769 validation samples for training and evaluation. The nuScenes dataset is a large-scale benchmark for autonomous driving, using the 32 lanes LiDAR for data collection. We train on the 28,130 frames of samples in the training set and evaluate on the 6,010 validation samples. We use mAP and nuScenes detection score (NDS), a weighted average of mAP and other box attributes like translation, scale, orientation, velocity.

Implementation details. We implement Uni3DETR with mmdetection3D [11], and train it with the AdamW [32] optimizer. We set the number of learnable query points to 300 for datasets except for nuScenes, where we set to 900. For indoor datasets, we choose the 0.02m grid size. For the KITTI dataset, we use a (0.05m, 0.05m, 0.1m) voxel size and for the nuScenes, we use the (0.075m, 0.075m, 0.2m) voxel size. The nuScenes model is trained with 20 epochs, with the CBGS [75] strategy. For outdoor datasets, we also conduct the ground-truth sampling augmentation [64] and we remove the ground-truth sampling at the last 4 epochs. Dynamic voxelization [73] and ground-truth repeating [21] are also adopted during training. Besides these data-related parameters, other architecture-related hyper-parameters are all the same for different datasets.

3.1 Indoor 3D Object Detection

We first train and evaluate Uni3DETR on indoor 3D detection datasets and list the comparison with existing state-of-the-art indoor 3D detectors in Tab. 1. Here we omit some grouping methods like [58], which relies on mask annotations for better grouping and clustering. Our method obtains the 67.0% AP 25 and 50.3% AP 50 on SUN RGB-D, which surpasses FCAF3D, the state-of-the-art indoor detector based on the CNN architecture, by almost 3%. On the ScanNet dataset, Uni3DETR surpasses FCAF3D by 1% on AP 50 . It is also noticeable that with only point clouds participating in training, our method even obtains better performance than existing multi-modal approaches that require both point clouds and RGB images. On the SUN RGB-D dataset, our model is 2.1% higher on AP 25 than TokenFusion. This strongly demonstrates the effectiveness of Uni3DETR. The visualized results of Uni3DETR on SUN RGB-D can be seen in the left two of Fig. 5.

Our method also significantly outperforms existing transformer-based indoor detectors, 3DETR and GroupFree. The superiority of our method is more significant, especially in localization: Uni3DETR

outperforms them by 5.1% on SUN RGB-D and 5.5% on ScanNet in AP 50 . Such results validate that compared with existing transformers in 3D detection, our detection transformer on voxel features with the mixture of query points is more appropriate for 3D detection.

3.2 Outdoor 3D Object Detection

KITTI. We then conduct experiments on the outdoor KITTI dataset. We report the detection results on the KITTI validation set in three difficulty levels - easy, moderate, and hard in Tab. 2. We notice that Uni3DETR also achieves the satisfying performance with the same structure as that for indoor scenes. For the most important KITTI metric, AP on the moderate level of car, we obtain the 86.57% AP, which is more than 1.5% higher than CT3D and 1.3% higher than RDIoU. With only the car class in training, the car moderate AP is 86.74%, which is also higher than existing methods like BtcDet. Its ability in outdoor scenes is thus demonstrated. The superiority of Uni3DETR is also consistent for the pedestrian and cyclist class. This illustrates that our model can also distinguish small and scarce objects well, which is one main aspect that hinders existing indoor detectors in the outdoor environments. The visualized results are shown in the right two of Fig. 5.

nuScenes. We further conduct experiments on the nuScenes dataset. Compared to the KITTI dataset, the range of scenes in nuScenes is larger, with 360 degrees around the LiDAR instead of only the front view. The point cloud in nuScenes is also more sparse (with 32-beam LiDAR compared to the KITTI 64 beams). These make the nuScenes more challenging and even some existing outdoor detectors fail to address the detection problem on nuScenes. We list the comparative results in Tab. 3. We obtain the 68.5% NDS, which surpasses recent methods like PillarNet, UVTR, VoxelNeXt. Compared to the most recent method VoxelNeXt, Uni3DETR is 1.8% higher in NDS and 1.4% higher in mAP. Besides the detection metric mAP, Uni3DETR also achieves promising results for predicting other attributes of boxes. The ability of our model in the outdoor scenes is further validated.

CH

3.3 Ablation Study

3D detection transformer. We first compare the detection AP of our detection transformer with existing transformer structures and list the results in Tab. 4. As 3DETR is built on point-based features, its performance is restricted by the less effectiveness of point representations. We then implement other methods based on voxel features. As anchors are negatively affected by the center point missing in 3D point clouds, formulating queries as 3D anchors like DAB- or DN-DETR is less effective than our query points. In comparison, our detection transformer better adapts the 3D points, thus achieves the highest performance among existing methods. Besides, it accommodates the mixture of query points, which brings further detection improvement.

Mixture of query points. Tab. 5 analyzes the effect of the mixture of query points on 3D object detection. For indoor detection on SUN RGB-D, the simple usage of the learnable query points { P l } obtains the 62.6% AP 25 . Substituting the learnable query with the non-learnable query hurts the performance. This is because non-learnable query points are fixed during training and cannot adaptively attend to information around objects. After we introduce { P l , P nl } , we find such mixed usage notably boosts AP 25 , from 62.6% to 65.1%: the global information in P nl overcomes the lack of local information in P l and contributes to more comprehensive detection results. As a result, the object-related metric is enhanced greatly. Utilizing { P l , P nlv } similarly improves AP 25 , but less than the effect of P nl . This is because voxelization may remove some points, making the global information not that exhaustive. With { P l , P nl , P nlv } , the three different query points further enhance the performance. P nlv here provides more location-accurate global information, thus AP 50 is improved by almost 3%. Random points at the test time further brings a 0.6% improvement.

For outdoor detection on KITTI, we notice that the effect of non-learnable query points is less significant than that in indoor scenes. Using { P l , P nl } is just 0.35% higher than { P l } . Since outdoor point clouds are large-range and sparse, the effect of global information in non-learnable query will be counteracted by excessive background information. Therefore, local information matters more in the outdoor task. The collaboration of the two-level information in the mixture of query makes it suitable for 3D detection in various scenes.

We further combine multiple groups of learnable query points and list the performance in Tab. 6 to compare with the mixture of query points. Such a combination is actually analogous to Group DETR [5, 6] in 2D detection. We observe that unlike 2D detection, multiple groups of learnable queries do not bring an improvement. Even though the number of groups has been raised to 7, the AP on SUN RGB-D still keeps almost the same. The reason is that multiple groups in 2D mainly speed up the convergence, which does not matter much in 3D. In this situation, by providing multi-level information about the scene, the mixture of query points works better for 3D detection.

Decoupled IoU. In Tab. 7, we compare our decoupled IoU with different types of IoU in the 3D space. On the SUN RGB-D dataset, optimizing with no IoU loss, only classification and L1 regression loss, just obtains the 20.7% AP 25 , indicating that 3D IoU is necessary to address the scale problem of L1 loss. Introducing the normal IoU 3 D alleviates the limitation of L1 loss to some extent, boosting the AP 25 to 48.3%. However, it is restricted by the negative coupling effect, thus is even inferior to axis-aligned 3D IoU, which does not take the rotation angle into consideration. In comparison, since this problem does not exist in 2D IoU, even the IoU xy , without considering the z axis, can be 17.5% higher than IoU 3 D . Our decoupled IoU, which can be viewed as (IoU xy +IoU z ) / 2 , takes all directions into account, thus further improves the AP 25 to 67.0%. On the outdoor KITTI dataset, the decoupled IoU is equally critical, 24.77% higher than w/o IoU and 8.22% higher than IoU 3 D . Its necessity for transformer-based 3D detection is further validated.

3.4 Comparison of Computational Cost

Here we further list the comparison of both performance and efficiency in the Tab. 8. We can observe that the computational budget compared with these methods is not significant: the inference time (latency) is almost the same as UVTR and the FLOPS is only 1.16% more than UVTR. In addition, we obtain significantly better detection performance on both indoor and outdoor datasets. Compared with VoxelNeXt, one model that mainly focuses on reducing the FLOPS of 3D detectors, we achieve more than 40% indoor AP and more than 25% average AP improvement. In this paper, we mainly target a unified structure. To ensure that the detector can accommodate both indoor and outdoor detection, we have, to a certain extent, made sacrifices in terms of efficiency, in order to prioritize its unified ability. A more efficient and unified structure can be left as one of the future works.

3.5 Discussion

A unified voxel size. The word 'unified' in our paper specifically refers to the architecture aspect. Since point clouds are collected with different sensors, their ranges and distributions vary significantly (about 3m for indoor but more than 70m for outdoor datasets). For the experiments above, we adopt different values for the voxel size, one data-related parameter. We further conduct the experiment with the same KITTI voxel size (0.05m, 0.05m, 0.1m) and list the results in Tab. 9. Compared with other outdoor detectors, our superiority is still obvious. Therefore, even if standardizing such a data-related parameter, our model can still obtain a higher AP.

Cross-dataset experiments. We further conduct the cross-dataset evaluation with the Omni3D metric [2]. From the results in Tab. 10, it is worth noticing that Uni3DETR has a good cross-dataset generalization ability. The cross-dataset AP (ScanNet to SUN RGB-D) is 16.2% higher than Cube RCNN trained on the SUN RGB-D dataset. The reason is that our Uni3DETR takes point clouds as input for 3D detection, while Cube RCNN takes RGB images for detection. By introducing 3D space information from point clouds, the superiority of a unified architecture for point clouds over Cube RCNN can be demonstrated. We further emphasize that cross-dataset evaluation is a more difficult problem for point cloud based 3D object detection, as the dataset-interference issue is more serious. We believe our Uni3DETR can become the basic platform and facilitate related research.

Limitation. One limitation is that we still require separate models for different datasets. Currently, some methods [2, 68] have tried one single model for multiple datasets for 3D object detection. We hope our Uni3DETR can become the foundation for indoor and outdoor 3D dataset joint training.

4 Related Work

Existing research on 3D Object Detection has been separated into indoor and outdoor categories. Indoor 3D detectors [38, 70, 9, 58] usually cluster points first based on extracted point-wise features, then conduct classification and detection. Recently, 3D sparse convolution based detectors [43, 45] also adopt the anchor-free manner [56] in 2D for 3D indoor detection. In comparison, outdoor 3D detectors [64, 52, 49, 13, 46, 7, 50] usually convert 3D features into the 2D BEV space and adopt 2D convolutions for predicting 3D boxes. However, because of the significant difference in point clouds between indoor and outdoor scenes, a unified 3D detector in various environments is still absent. Some recent methods [20, 31, 60, 44] have performed experiments on both indoor and outdoor datasets. However, they rely on RGB images to bridge the difference of point clouds.

Transformer-based object detectors have been widely used in 2D object detection [4, 77, 34, 29, 25, 69], predicting object instances in the set-to-set manner. Recently, the transformer structure has been introduced in 3D object detection. Methods like [33, 18, 15, 54, 14] introduce transformers into the backbone for 3D feature extraction, while still adopting CNNs for predicting boxes. In comparison, approaches like [35, 61, 36, 74, 76] leverage transformers in the detection head for box prediction. However, these methods still rely on point-wise features or BEV features for either indoor or outdoor 3D detection, thus are restricted by the singular scene.

5 Conclusion

In this paper, we propose Uni3DETR, a unified transformer-based 3D detection framework that addresses indoor and outdoor 3D object detection within the same network structure. By feeding the mixture of query points into the detection transformer for point-voxel interaction and supervising the transformer decoder with the decoupled IoU, our Uni3DETR fills the gap of existing research in unified 3D detection under various scenes. Extensive experiments demonstrate that our model can address both indoor and outdoor 3D detection with a unified structure. We believe our work will stimulate following research along the unified and universal 3D object detection direction, and Uni3DETR will become a basic tool and platform.

Acknowledgment. This work is supported by the National Key Research and Development Program of China in the 14th Five-Year with Nos. 2021YFF0602103 and 2021YFF0602102, National Natural Science Foundation of China (No. 62201484), HKU Startup Fund and Seed Fund for Basic Research.

References

Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR , 2016.

Garrick Brazil, Abhinav Kumar, Julian Straub, Nikhila Ravi, Justin Johnson, and Georgia Gkioxari. Omni3d: A large benchmark and model for 3d object detection in the wild. In CVPR , 2023.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR , 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020.

Qiang Chen, Xiaokang Chen, Gang Zeng, and Jingdong Wang. Group detr: Fast training convergence with decoupled one-to-many label assignment. In ICCV , 2023.

Qiang Chen, Jian Wang, Chuchu Han, Shan Zhang, Zexian Li, Xiaokang Chen, Jiahui Chen, Xiaodi Wang, Shuming Han, Gang Zhang, et al. Group detr v2: Strong object detector with encoder-decoder pretraining. arXiv:2211.03594 , 2022.

Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. In CVPR , 2022.

Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In CVPR , 2023.

Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points for voting-based 3d object detection in point clouds. In CVPR , 2021.

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR , 2019.

MMDetection3D Contributors. Mmdetection3d: Openmmlab next-generation platform for general 3d object detection, 2020.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR , 2017.

Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In AAAI , 2021.

Shaocong Dong, Haiyang Wang, Tingfa Xu, Xinli Xu, Jie Wang, Ziyang Bian, Ying Wang, Jianan Li, et al. Mssvt: Mixed-scale sparse voxel transformer for 3d object detection on point clouds. NeurIPS , 2022.

Lue Fan, Ziqi Pang, Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In CVPR , 2022.

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR , 2013.

JunYoung Gwak, Christopher Choy, and Silvio Savarese. Generative sparse detection networks for 3d single-shot object detection. In ECCV , 2020.

Chenhang He, Ruihuang Li, Shuai Li, and Lei Zhang. Voxel set transformer: A set-to-set approach to 3d object detection from point clouds. In CVPR , 2022.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , 2017.

Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In ECCV , 2020.

Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In CVPR , 2023.

Borui Jiang, Ruixuan Luo, Jiayuan Mao, Tete Xiao, and Yuning Jiang. Acquisition of localization confidence for accurate object detection. In ECCV , 2018.

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly , 1955.

Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR , 2019.

Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In CVPR , 2022.

Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. NeurIPS , 2020.

Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3d object detection. NeurIIPS , 2022.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV , 2017.

Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. ICLR , 2022.

Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In ICCV , 2021.

Zhe Liu, Tengteng Huang, Bingling Li, Xiwu Chen, Xi Wang, and Xiang Bai. Epnet++: Cascade bi-directional fusion for multi-modal 3d object detection. TPAMI , 2022.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR , 2019.

Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In ICCV , 2021.

Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In ICCV , 2021.

Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In ICCV , 2021.

Duy-Kien Nguyen, Jihong Ju, Olaf Booij, Martin R Oswald, and Cees GM Snoek. Boxer: Box-attention for 2d and 3d transformers. In CVPR , 2022.

Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas. Imvotenet: Boosting 3d object detection in point clouds with image votes. In CVPR , 2020.

Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV , 2019.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR , 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. NeurIPS , 2017.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS , 2015.

Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR , 2019.

Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Fcaf3d: fully convolutional anchorfree 3d object detection. In ECCV , 2022.

Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view general-purpose 3d object detection. In WACV , 2022.

Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Tr3d: Towards real-time indoor 3d object detection. arXiv:2302.02858 , 2023.

Hualian Sheng, Sijia Cai, Yuan Liu, Bing Deng, Jianqiang Huang, Xian-Sheng Hua, and Min-Jian Zhao. Improving 3d object detection with channel-wise transformer. In ICCV , 2021.

Hualian Sheng, Sijia Cai, Na Zhao, Bing Deng, Jianqiang Huang, Xian-Sheng Hua, Min-Jian Zhao, and Gim Hee Lee. Rethinking iou-based optimization for single-stage 3d object detection. In ECCV , 2022.

Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: High-performance pillar-based 3d object detection. In ECCV , 2022.

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In CVPR , 2020.

Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn++: Point-voxel feature set abstraction with local vector representation for 3d object detection. IJCV , 2023.

Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In CVPR , 2019.

Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. TPAMI , 2020.

Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In CVPR , 2015.

Pei Sun, Mingxing Tan, Weiyue Wang, Chenxi Liu, Fei Xia, Zhaoqi Leng, and Dragomir Anguelov. Swformer: Sparse window transformer for 3d object detection in point clouds. In ECCV , 2022.

OD Team et al. Openpcdet: An open-source toolbox for 3d object detection from point clouds, 2020.

Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In ICCV , 2019.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS , 2017.

Haiyang Wang, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang, et al. Cagroup3d: Class-aware grouping for 3d object detection on point clouds. NeurIPS , 2022.

Haiyang Wang, Shaoshuai Shi, Ze Yang, Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. In CVPR , 2022.

Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang. Multimodal token fusion for vision transformers. In CVPR , 2022.

Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In CoRL , 2022.

Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Dening Lu, Mingqiang Wei, and Jun Wang. Venet: Voting enhancement network for 3d object detection. In ICCV , 2021.

Qiangeng Xu, Yiqi Zhong, and Ulrich Neumann. Behind the curtain: Learning occluded shapes for 3d object detection. In AAAI , 2022.

Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors , 2018.

Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR , 2020.

Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In ICCV , 2019.

Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR , 2021.

Bo Zhang, Jiakang Yuan, Botian Shi, Tao Chen, Yikang Li, and Yu Qiao. Uni3d: A unified baseline for multi-dataset 3d object detection. In CVPR , 2023.

Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and HeungYeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. ICLR , 2023.

Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In ECCV , 2020.

Yu Zheng, Yueqi Duan, Jiwen Lu, Jie Zhou, and Qi Tian. Hyperdet3d: Learning a sceneconditioned 3d object detector. In CVPR , 2022.

Dingfu Zhou, Jin Fang, Xibin Song, Chenye Guan, Junbo Yin, Yuchao Dai, and Ruigang Yang. Iou loss for 2d/3d object detection. In 3DV , 2019.

Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3d object detection in lidar point clouds. In CoRL , 2020.

Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. In ECCV , 2022.

Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv:1908.09492 , 2019.

Benjin Zhu, Zhe Wang, Shaoshuai Shi, Hang Xu, Lanqing Hong, and Hongsheng Li. Conquer: Query contrast voxel-detr for 3d object detection. In CVPR , 2023.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR , 2021.

A Datasets and Implementation Details

SUN RGB-D [53]. When training on the SUN RGB-D dataset, the input point clouds are filtered in the range [-3.2m, 3.2m] for the x axis, [-0.2m, 6.2m] for the y axis and [-2m, 0.56m] for the z axis. The grid size is set to 0.02m. We randomly flip the input data along the x axis and randomly sample 20,000 points for data augmentation. The common global translation, rotation and scaling strategies are also adopted here. We train Uni3DETR with the initial learning rate of 1.67e-4 and the batch size of 32 for 90 epochs, and the learning rate is decayed by 10x on the 70th and 80th epoch. The ratio of the classification score and the predicted IoU is 0.8:0.2.

ScanNet [12]. For the ScanNet dataset, here we adopt the range of [-6.4m, 6.4m] for the x and y axis and [-0.1m, 2.46m] for the z axis after global alignment, with the 0.02m grid size. Here we do not adopt the RGB information of the dataset for training. The input data are randomly flipped along both the x and y axis. We utilize dynamic voxelization [73] on the ScanNet dataset. The initial learning rate is set to 7.5e-5, with the batch size of 24. We train Uni3DETR for 240 epochs, where the learning rate is decayed on the 192nd and the 228th epoch. Other hyper-parameters and operations are the same as the SUN RGB-D dataset.

S3DIS [1]. As the S3DIS dataset point clouds are distributed only on the positive planes, which is adverse to the random flipping augmentation, we first translate the input data to make it centered at the original point. Besides the coordinate information, we also utilize the RGB information of the point clouds. We train the Uni3DETR for 520 epochs, with the learning rate decaying at the 416th and 494th epoch. The batch size is set to 8, with the initial learning rate of 3.33e-5. Other hyper-parameters and operations are the same as the ScanNet dataset.

KITTI [16]. For the KITTI dataset, the data augmentation operations are basically the same as previous methods like [13]. For the ground-truth sampling augmentation, we sample at most 20 cars, 10 pedestrians and 10 cyclists from the database. 18000 points are randomly sampled at the training time. During training, we also adopt van class objects as car objects. The ground-truth sampling augmentation and the object-level noise strategy are removed at the last 2 epochs. We train Uni3DETR for 70 epochs, with the learning rate decaying at the 64th epoch. The initial learning rate is set to 9e-5, with the batch size of 8. As the KITTI dataset suffers from the sparse objects seriously, we follow [21] to repeat the ground truth labels 5 times during training. The predicted car objects are filtered at the threshold of 0.5 after inference. The ratio of the classification score and the predicted IoU is 0.2:0.8 on the KITTI dataset.

nuScenes [3]. Compared to the KITTI dataset, the nuScenes dataset covers a larger range, with 360 degrees around the LiDAR instead of only the front view. The point cloud in nuScenes is also more sparse (with 32-beam LiDAR compared to the KITTI 64 beams). Besides, the nuScenes dataset contains 10 classes, with the severe long-tail problem. The initial learning rate is set to 1e-5, with the batch size of 16 and the cyclic schedule. We train the Uni3DETR for 20 epochs.

B Test Set Results

We further conduct the experiment and evaluate our method on the test set of KITTI and nuScenes dataset. The comparison is listed in the Tab. 11 and Tab. 12 separately. For the most important KITTI metric, AP on the moderate level of car, we obtain the 82.26% AP, which is 0.83 points higher than PV-RCNN, 0.49 points higher than CT3D, and 0.38 points higher than PV-RCNN++. On the test set of the nuScenes dataset, we obtain the 65.2% mAP and 70.8% NDS. The consistent superiority further demonstrates the ability of Uni3DETR on outdoor 3D detection.

C Visualized Results

Comparison results about the mixture of query points. We first provide comparative visualized results in Fig. 6 to illustrate the effectiveness of the mixture of query points. For the first case, it can be seen that training with only the learnable query points concentrates on the right region of the bed and the nightstand, and ignores the left sofa. This similarly applies to the rest two cases. For the second case, the right nightstand is not detected and for the third case, three chairs are ignored. The common point among these three cases is that these ignored objects are partly occluded, thus with insufficient points. The limited quantities of point clouds therefore restrict the performance of the 3D

ground-truth

learnable

mixture

detector. After we introduce the mixture of query points, global information is better considered with the help of non-learnable query points. As a result, even under the circumstance of insufficient point cloud information, our Uni3DETR can still recognize these objects and detect them out relying on the knowledge about the whole scene. More visualized examples are also provided in Fig. 7.

Comparison results about the decoupled IoU. Wethen compare the visualized results of Uni3DETR with the normal 3D IoU and plot the results in Fig. 8. It can be seen that when supervising the detector with the normal 3D IoU, although the 3D detector has the ability to detect the foreground objects out, the localization precision remains significantly low. For example, for the second case, two chairs are indeed detected, but the overlaps between the detected instances with the corresponding objects are minimal. Furthermore, the low degree of the overlapped area also results in many duplicated boxes, especially for the above one. The same thing also occurs at the third case. For the first case, besides the localization error, only one chair is detected out. This is because 3D IoU is hard to optimize

ground-truth

中

learnable

中

mixture

1

thus fails to alleviate the scale problem of L1 loss. In comparison, our decoupled IoU addresses the coupling problem of 3D IoU, thus contributing to better localization accuracy.

More visualized results obtained by Uni3DETR. We further provide more visualized results obtained by our Uni3DETR on different datasets. Uni3DETR obtains satisfying detection results on all five datasets, which further demonstrate its effectiveness and universality.

D Per-category Results

We first list the per-category results for the 10 classes on the SUN RGB-D dataset in Tab. 13 and Tab. 14. For the AP 25 metric, Uni3DETR achieves the best for 9 classes out of the total 10 classes. The

ground-truth

AWIMH

3D loU

KAA

decoupled loU

ground-truth

Uni3DETR

ground-truth

Uni3DETR

ground-truth

Uni3DETR

most significant improvement comes from the sofa and table class, 5.4% and 6.2% respectively. For the AP 50 metric, UniDETR also achieves the best for 8 classes. For the sofa class, the improvement is up to 4.8%. The effectiveness of UniDETR is thus further demonstrated.

ground-truth

6143

乐

Uni3DETR

ground-truth

Uni3DETR

D'

The per-category results for the 18 classes on the ScanNet dataset are listed in Tab. 15 and Tab. 16. We achieve the best for 7 classes for the AP 25 metric and for 9 classes for the AP 50 metric. For the S3DIS dataset, the per-category results are listed in Tab. 17 and Tab. 18. For the AP 25 metric, we achieve the best for 4 classes out of the total 5 ones. For the AP 50 metric, we are 18.1% higher than FCAF3D on the board class. These per-category results further demonstrate the ability of Uni3DETR on indoor scenes.

We also list the per-category AP on the nuScenes dataset in Tab. 19. Our Uni3DETR achieves the best for 7 out of 10 classes. The ability of Uni3DETR is further demonstrated on outdoor scenes.

Peer reviews:
comments: SUMMARY:
The authors explore and analyze the existing LiDAR-based 3D object detection framework, and propose to adopt transformer-based method to perform detection from differnt LiDAR inputs. The experimental results on a number of datasets are better compared to the existing methods.

STRENGTHS:
1. The task of 3D object detection is very important in the 3D community. Interestingly, the authors propose to adopt transformers to generalize 3D detection across diverse scene inputs.

2. The paper is easy to follow.

3. The authors conduct the experiments on widely-used indoor and outdoor datasets, including  SUN RGB-D, ScanNet, S3DIS,m nuScenes, and KITTI.

WEAKNESSES:
1. Missing important SOTA works on KITTI. The manuscript does report validation results on KITTI, however, the authors forgot to compare with LiDAR-only SOTAs 'PV-RCNN++' (PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection). However, it would be much convincing if the authors can compare with this SOTA.

2. Results on KITTI and nuScenes test set. In table 2 & 3, the proposed Uni3DETR is comparable/better with the cited methods. However, it would be much convincing if the authors can report the test set results on KITTI and nuscenes.

3. Computation/memory footprint comparison. The authors didn't make a comparison of their work in terms of memory/speed with the existing 3D detection methods. The time consumption might be large since the memory/time are espeically heavy for the transformer-based methods from my point.

QUESTIONS:
Please refer to the questions that I describe in the Weakness part. I would also consider the rebuttal and other reviews.

LIMITATIONS:
NA.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
The authors propose a  unified detr-style detector for both the indoor and outdoor 3D object detection. Besides common learnable query, they also adopt unlearnanle query sampled from raw point cloud.

STRENGTHS:
The proposed method is simple and easy to follow.

WEAKNESSES:
The novelty is limted, and the experimental results are not fair/convincing.

QUESTIONS:
1. Uni3DETR follows the design of DETR, which is a ''sparse'' and end-to-end detector. However, Uni3DETR  uses 4 groups of query points for indepedent prediction and needs extra pos-procedure to filer redundant boxes.  The authors claim that the learnable/unlearnable queries capture the local and global information of the scene.  If so many different queries are neccesary to cover the whole scene, why not use a dense detector like pointpillar or centerpoints instead?
2. I am surprised that the results in Tab. 6. I hope the authors could provide more explanation about why multiple groups of queries brings no performance gain?  Why this ablation is not conducted on other datasets (e.g. KITTI)  like Tab.5 and Tab. 7?
3. Using 4 groups of query points (learnable  & unlearnable) as proposed in the paper, or simply use more learnable queries, which one is better?
4. The authors are suggestd to provide results on the KITTI and nuscenes test split. Besides,  KITTI provides a more fair and relibale evaluation setting with 40 recall positions.
5. What is the diffence between Uni3DETR and DAB-Deform in Tab. 4.
6. With differenent backbone, model size, inference speed, ect., it is not fair for the comparison with other sota methods in Tab.2 & Tab.3.

LIMITATIONS:
The novelty is limited, and more comprehensive analysis is required.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUni3DETR is a unified 3D object detector that is capable of handling both indoor and outdoor scenes within the same framework. This is significant as many existing detectors are specialized for either indoor or outdoor environments, but not both.\n\nThe method employs a detection transformer with a point-voxel interaction mechanism and a mixture of query points. This allows it to exploit both global and local information effectively.\n\nUni3DETR introduces a decoupled Intersection over Union (IoU), which serves as an easy-to-optimize training target for localization. This is aimed at improving the accuracy of the detector.\n\nThe paper demonstrates that Uni3DETR exhibits excellent performance consistently on both indoor and outdoor 3D detection tasks. Moreover, it shows strong generalization ability under heterogeneous conditions, which is beneficial for real-world applications.\n\nSTRENGTHS:\nVersatility Across Environments: One of the major strengths of Uni3DETR is its ability to handle both indoor and outdoor environments within the same framework. This is a significant advancement as it eliminates the need for different models for different environments, which is common in existing approaches.\n\nOptimization of Localization: The introduction of a decoupled Intersection over Union (IoU) as a training target for localization is a notable strength. This provides an easy-to-optimize objective that can potentially lead to more accurate localization of objects, which is often a challenging aspect of object detection.\n\nEmpirical Validation: The paper provides empirical evidence showing that Uni3DETR consistently performs well on both indoor and outdoor 3D detection tasks. This is important for establishing the credibility and effectiveness of the proposed approach.\n\nWEAKNESSES:\nInnovative Element. The core pipeline appears to be a Deformable DETR complemented by non-learnable queries (voxel features derived from FPS). It essentially resembles a simple combination of two model architectures.\n\nEfficiency. The model's efficiency is not entirely at the cutting-edge level. For instance, in nuScenes, several competitive 3D detection techniques are not listed in your tables. (https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Lidar).\n\nIncrease in Precision. According to Table 5, the non-learnable query appears to have a marginal impact on outdoor detection. This aligns with our prior understanding that FPS is not well-suited for outdoor detection. The mixed query's design might be superfluous for outdoor scenarios.\n\nComputational Complexity. It is widely recognized that greater computation often results in superior performance. The authors should include a breakdown of the mixed query's complexity, such as inference latency and FLOPS, in comparison to other techniques.\n\nQUESTIONS:\nAs evidenced in Table 5, the non-learnable query makes a small contribution to outdoor detection. Can you provide a comparable analysis of the impact of the mixed query points on nuScenes? I'm curious to determine if the performance on nuScenes is primarily driven by your learnable query.\n\nCould you supply a comparison of computational complexity and inference latency between your approach and alternative methods?\n\nLIMITATIONS:\nIntegrating two distinct 3D detection algorithms in this method might potentially increase computational complexity. Nonetheless, computation overhead is a key factor for outdoor detection tasks, such as autonomous driving. Given this, the inclusion of a non-learnable query design seems to be superfluous in such outdoor environments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Uni3DETR, a unified architecture suitable for indoor and outdoor scenes. Uni3DETR employs the DETR with point-voxel interaction for object prediction. It uses a mixture of query points to exploit global and local information. Finally, Uni3DETR uses a decoupled IoU loss by disentangling the depth and other dimensions to ease out training. Training a per-dataset model and testing on the same dataset improves over the baselines.\n\nSTRENGTHS:\n+ The problem of using a unified architecture for indoor and outdoor point clouds is super relevant.\n+ The idea of employing DETR, a mixture of query points, and decoupling 3D IoU is nice.\n+ The per-dataset results are good.\n\nWEAKNESSES:\n- The first claim that the architecture is uniform is only partially correct. L228 says that the authors chose a 0.02m grid size for indoor datasets, while they chose a different voxel size for the KITTI and the nuscenes datasets. The authors should pick a single voxel size and then run it on all the datasets. (It is OK to change the range for nuScenes since it is a multi-camera dataset, but the authors should choose a single voxel size)\n\n- The claim of \"strong universal ability of our model (L336)\" is an overstatement, in my opinion, since the paper only carries out Per-dataset model training and testing. A related paper that trains a single RGB 3D detector is Omni3D + Cube R-CNN [A]. I would urge the authors to follow a similar protocol where the train and test sets are\n   - KITTI + nuScenes front and then test on indoor indoor/outdoor\n   - Train on indoor scenes and then test on outdoor indoor/outdoor\n   - Finally, train on all indoor/outdoor scenes, then test on indoor/outdoor scenes.\n\nBy training Uni3DETR on KITTI + nuScenes front and testing outdoors, one can compare against the Cube R-CNN model of [A].\n\n- How well does the method generalize in zero-shot settings? E.g., How does the Uni3DETR model trained on outdoor scenes work on an unknown lidar scan from an unknown say, Waymo dataset.\n\n- A sad part of 3D detection papers is to use a bigger backbone and show improvements on nuScenes. Please quantitatively compare the flops, model size, training time, and inference time for Uni3DETR against the baselines in Table 3.\n\n- The authors say the learnable query points mostly contain local information and fit the outdoor detection well. In contrast, non-learnable query points emphasize global information and are more effective for dense indoor scenes. Another way to enforce local and global information is to use a dual-path architecture where one branch is attention-based for global information while the other is convolution-based for local information. One can later fuse their outputs by concatenation or predicting a coefficient to add them. Why did the authors not consider this design choice?\n\n- An alternative to using the decoupled IoU loss is the disentangled loss [B], where one constructs a 3D box from GT parameters except the one in consideration. That way, one can avoid explicitly weighing the losses for respective parameters (e.g., the authors use weighing xy and z terms by 0.5). The authors should, therefore, quantitatively compare the decoupled IoU loss against the disentangled loss.\n\n- The details of the 3D feature extractor (Section 2.1) need to be included. \n\n- The authors use the mean average precision (mAP) under IoU thresholds of 0.25 and 0.5 for evaluating datasets  (L216). Using one or two thresholds is quite sensitive. A better alternative is to use exact IoU3D over a series of thresholds of 0.05:0.5:0.05 and then average them as in [A].\n\nMinor:\n- Do the authors plan to release the code?\n\n- Typo: L203: final\n\nReferences:\n\nA. Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild, Brazil, Kumar, Ravi, et al., CVPR 2023.\n\nB. MonoDIS: Disentangling monocular 3D object detection: From single to multi-class recognition, Simonelli, Bulo, Porzi, et al., PAMI 2020.\n\nQUESTIONS:\nPlease see the weakness.\n\nLIMITATIONS:\nThe authors do not list out the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Uni3DETR, a unified 3D detection transformer that addresses indoor and outdoor 3D detection within the same framework. The paper provides a specific analysis of the inconsistency in the structure of current indoor and outdoor scene detection models. Due to the differences in data distribution between indoor and outdoor environments, indoor datasets have a smaller range of point clouds, with objects being closer together and denser, occupying a larger portion of the point cloud. On the other hand, outdoor scenes have smaller and sparser objects, with background point clouds dominating the overall point cloud. Utilizing the mixture of query points which consist of  the learnable and non-learnable query, the detector is able to exploit the global information for dense small-range indoor scenes and local information for large-range sparse outdoor scenes. Besides, decoupled IoU is proposed for easier and faster training target localization optimization  by disentangling the xy and z space.\n\nSTRENGTHS:\n1. The paper provides a detailed analysis of the inconsistent models of indoor and outdoor 3D detectors, with the difference in data distribution being the most important reason. The author explains how to design a unified 3D detector and solves the problem of inconsistent architectures between indoor and outdoor scenes.\n2. The overall logic of the paper is clear, and it is well written.\n3. Uni3DETR demonstrates the strong generalization ability under heterogeneous conditions while maintaining caomparable performance.\n\nWEAKNESSES:\n1. The experiments of model architectures are a bit insufficient. In line 99, the paper mentions, \"Then, we convert the extracted sparse features into dense ones and apply 3D dense convolutions for further feature processing.\" However, no experiments show the performance gain brought by 3D dense convolution to support the claim. Furthermore, will the model architectures vary from datasets? For example, are the numbers of layers different for models trained on KITTI and S3DIS?\n2. The author claims that learnable queries primarily capture the local information of the scene, and non-learnable queries usually concentrate more on global information. Although experiments show that combining different queries could bring better performance, there is no evidence in the paper (e.g., visualization or other analysis) to support the claim.\n\nQUESTIONS:\n1. Decoupled IoU is only used in Uni3DETR, and it can be seen from the ablation study that this loss has a significant performance improvement for the detector. However, Decoupled IoU loss was not used in other mothods when compared to other methods. Can Decoupled IoU improve other detectors, such as GroupFree3D, and also bring significant performance improvements?\n2. The learnable query mentioned in the paper lacks its initialization process, such as how to obtain $P_q$ and $C_q$ mentioned in line 107.\n3. How about the generalization ability of feature extraction based on this structure for indoor scenes? For example, how it performs zero-shot testing by training on SUN RGBD and directly testing the trained model on ScanNet?\n5. The author uses a mixture of query points to obtain detection results through different categories and a sufficient number of queries. Will this result in noticeable time consumption during inference?\n6. It would be better to illustrate more visualization results to support the claim that the learnable and non-learnable queries have different capabilities.\n7. According to the paper: ConQueR: Query Contrast Voxel-DETR for 3D Object Detection: DETRs usually adopt a larger number of queries than GTs (e.g., 300 queries v.s. ∼40 objects in Waymo) in a scene, which inevitably incur many false positives during inference. Could this problem occur in Uni3DETR since many more queries are used in inference?\n\nLIMITATIONS:\nIt seems that the paper now provides a unified paradigm for solving 3D object detection tasks of indoor and out door scenes and do not use one model for testing on all the datasets. Furthermore, the model is separately trained with different hyper-parameters on different datasets.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}