SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenyu Wang1 Yali Li1\* Xi Chen2 Hengshuang Zhao2\* Shengjin Wang1
$^{1}$  Department of Electronic Engineering, Tsinghua University, BNRist
2 The University of Hong Kong
{wangzy20@mails., liyali13@, wgsgj@}tsinghua.edu.cn, {xchen2, hszhao}@cs.hku.hk
Figure 1: The high-level overview comparing the performance of 12 existing 3D detectors and our Uni3DETR on a broad range of 3D object detection datasets: indoor datasets SUN RGB-D (SUN), ScanNet, S3DIS, and outdoor datasets KITTI, nuScenes (nuS). The metrics are  $\mathrm{AP}_{25}$  for indoor datasets,  $\mathrm{AP}_{70}$  on moderate difficulty car for KITTI, and NDS for nuScenes. The center of the circle means that the corresponding metric is less than  $10\%$ , and the outermost means  $90\%$ . Existing indoor detectors are plotted in red and outdoor detectors are in green. Our model has the remarkable capacity to generalize across a wide range of diverse 3D scenes (a larger polygon area).
Existing point cloud based 3D detectors are designed for the particular scene, either indoor or outdoor ones. Because of the substantial differences in object distribution and point density within point clouds collected from various environments, coupled with the intricate nature of 3D metrics, there is still a lack of a unified network architecture that can accommodate diverse scenes. In this paper, we propose Uni3DETR, a unified 3D detector that addresses indoor and outdoor 3D detection within the same framework. Specifically, we employ the detection transformer with point-voxel interaction for object prediction, which leverages voxel features and points for cross-attention and behaves resistant to the discrepancies from data. We then propose the mixture of query points, which sufficiently exploits global information for dense small-range indoor scenes and local information for large-range sparse outdoor ones. Furthermore, our proposed decoupled IoU provides an easy-to-optimize training target for localization by disentangling the  $xy$  and  $z$  space. Extensive experiments validate that Uni3DETR exhibits excellent performance consistently on both indoor and outdoor 3D detection. In contrast to previous specialized detectors, which may perform well on some particular datasets but suffer a substantial degradation on different scenes, Uni3DETR demonstrates the strong generalization ability under heterogeneous conditions (Fig. 1). Codes are available at https://github.com/zhenyuw16/Uni3DETR.
3D object detection from point clouds aims to predict the oriented 3D bounding boxes and the semantic labels for the real scenes given a point set. Unlike mature 2D detectors [41, 19, 56, 4] on RGB images, which have demonstrated the ability to effectively address diverse conditions, the problem of 3D object detection has been considered under different scenarios, leading to the development of distinct benchmarks and methodologies for each. Specifically, 3D detection approaches are currently addressed separately for indoor and outdoor scenes.
Indoor 3D detectors [38, 70, 62, 58] usually adopt the "grouping, clustering and classification" manner (Fig. 2a), while outdoor detectors [64, 24, 52, 49] typically convert the features into the 2D bird's-eye-view (BEV) space (Fig. 2b). Although these two tasks only differ in their respective application contexts, the optimal approaches addressing them exhibit significant differences.
(c)
Figure 2: Illustration of the structures of previous indoor 3D detectors (a), outdoor detectors (b) and our model. Our model has the capacity for 3D detection in both indoor and outdoor scenes.
The key challenge in developing a unified 3D detector lies in the substantial disparities of point cloud data collected in indoor and outdoor environments. In general, indoor scenarios are cluttered, where various objects are close and dense, occupying the majority of the scene. In comparison, objects in outdoor scenes are small and sparse, where background points dominate the collected point clouds. Such disparities result in the lack of a unified 3D detector: 1) For the detection head, because of the severe distraction of excessive background points in outdoor point clouds, together with the hyperparameter sensitivity associated with grouping, the grouping-based indoor detectors are infeasible for outdoor 3D detection. Besides, outdoor objects are separated clearly and not overlapped in the BEV space, which does not apply to indoor objects. The height overlap among indoor objects makes the manner of detecting in the BEV space inappropriate for indoor detection. 2) For the feature extractor, existing backbones in 3D detectors are similarly designed for a singular scene. Indoor detectors are usually equipped with point-based [39, 40] or 3D sparse convolution based [10, 43] backbone, where point-based models are usually susceptible to the diverse structures of points under different scenes, and sparse convolution models are deficient in representing the features of object centers. In contrast, the 2D convolutions in outdoor detectors for extracting BEV features easily lead to information loss for indoor detection.
In this paper, we propose a Unified 3D DEetection TRansformer (Uni3DETR) based on only point clouds for detecting in diverse environments (Fig. 2c). Two attributes of our Uni3DETR contribute to its universality for both indoor and outdoor scenes. First, we employ a hybrid structure that combines 3D sparse convolution and dense convolution for feature extraction. The pure 3D architecture avoids excessive height compression for indoor point clouds. Simultaneously, the sparse convolution prevents the huge memory consumption for large-range outdoor data, and the dense convolution alleviates the center feature missing problem for sparse outdoor points. Second, we utilize transformer [57] for 3D object detection. The set-to-set prediction way in transformer-based detectors [4, 77] directly considers predicted objects and ground-truth labels, thus tends to be resistant to the distinction from data themselves. The transformer decoder is built on the extracted voxel feature and we formulate queries as 3D points from the scene. The points and voxels interact through cross-attention, which well adapts to the characteristics of 3D data.
Based on our 3D detection transformer, we further propose two necessary components for universality under various scenes. One is the mixture of query points. Specifically, besides the learnable query, we introduce the non-learnable query initialized by sampling the original points and the voxelized points, and integrate the learnable and non-learnable queries for feeding the transformer. We observe that the learnable query points mostly contain local information and fit the outdoor detection well, while non-learnable query points emphasize global information thus are more effective for dense indoor scenes. The other one is the decoupled 3D IoU. Compared to the usual 3D IoU, we decouple the  $x$ ,  $y$  and  $z$  axis in the decoupled IoU to provide stronger positional regularization, which not only involves all directions in the 3D space but also is beneficial for optimization in transformer decoders.
Figure 3: The overall architecture of our Uni3DETR. We use the hybrid combination of 3D sparse convolution and dense convolution for 3D feature extraction, and our detection transformer for set prediction. The mixture of query points contributes to the sufficient usage of global and local information, and the decoupled IoU provides more effective supervision about localization.
- We propose Uni3DETR, a unified 3D detection transformer that is designed to operate effectively in both indoor and outdoor scenes. To the best of our knowledge, this is the first point cloud based 3D detector that demonstrates robust generalization across diverse environments.
- We propose the mixture of query points that collaboratively leverages the learnable and nonlearnable query. By aggregating local and global information, our mixture of queries fully explores the multi-scene detection capacity.
- We decouple the multiple directions in the 3D space and present the decoupled 3D IoU to bring more effective positional supervision for the transformer decoder.
Our Uni3DETR achieves the state-of-the-art results on both indoor [53, 12, 1] and outdoor [16, 3] datasets. It obtains the  $67.0\%$  and  $71.7\%$ $\mathrm{AP}_{25}$  on the challenging SUN RGB-D and ScanNet V2 indoor datasets, and the  $86.7\%$  AP in the moderate car category on KITTI validation. On the challenging nuScenes dataset, Uni3DETR also achieves the  $61.7\%$  mAP and  $68.5\%$  NDS.
2 Uni3DETR
We present the overall architecture of our Uni3DETR in Fig. 3. It consists of a 3D feature extractor and the detection transformer for detecting in various 3D scenes. The mixture of query points is fed into transformer decoders to predict 3D boxes, under the supervision of decoupled 3D IoU.
3D feature extractor. Existing 3D detectors usually adopt point-based [40] or voxel-based [64, 43] backbones for feature extraction. Considering that point-based backbones are vulnerable to the specific structure of point clouds themselves and less efficient in point set abstraction, we utilize the voxel-based model for extracting 3D features. After voxelization, we utilize a series of 3D sparse convolution layers to encode and downsample 3D features, to avoid the overload memory consumption for large-range outdoor scenes. Then, we convert the extracted sparse features into the dense ones and apply 3D dense convolutions for further feature processing. The dense convolution alleviates the feature missing problem of center points.
2.2 Unified 3D Detection Transformer for Diverse Scenes
Detection transformer with point-voxel interaction. We employ the transformer structure based on the extracted voxel features and the set prediction manner for 3D detection. Motivated by recent 2D transformer-based detectors [29, 25, 69] that formulate queries as anchor boxes, we regard 3D points in the 3D space as queries. Its structure is illustrated in Fig. 4.
Specifically, we denote  $P_{q} = (x_{q},y_{q},z_{q})$  as the  $q$ -th point to represent the  $q$ -th object,  $C_q\in \mathbb{R}^D$  is its content query, where  $D$  is the dimension of decoder embeddings. We introduce the deformable
attention [77] for the cross-attention module and view  $P_{q}$  as the reference point for point-voxel interaction. Unlike the deformable DETR decoder, here we directly learn the reference point  $P_{q}$ . Suppose the voxel feature as  $V$ , the process of the cross-attention is modeled as:
$$
\operatorname {C r o s s A t t} \left(C _ {q}, V\right) = \operatorname {D e f o r m A t t} \left(C _ {q} + \operatorname {M L P} \left(\operatorname {P E} \left(P _ {q}\right)\right), V, P _ {q}\right) \tag {1}
$$
where PE denotes the sinusoidal positional encoding. The transformer decoder predicts the relative positions for the query points:  $(\Delta x_{q},\Delta y_{q},\Delta z_{q})$  . Based on the relative predictions, the query points are refined layer-by-layer.
Compared to 2D detection transformers, because of the stronger positional information in the 3D space, our detection transformer requires less layers for query refinement. To further utilize the information across multiple layers, we average predictions from all transformer decoders except for the first one.
Mixture of query points. In the learning process, the above query points gradually turn around the object instances for better 3D detection. As they are updated during training, we refer them as learnable query points. Since such learnable query points finally concentrate on the points near the objects, they primarily capture the local information of the scene. For outdoor point clouds, where objects are sparse and small within the large-range scene, the learnable query points help the detector avoid the disturbance of the dominated background points.
In comparison, for the indoor scene, the range of 3D points is significantly less and the objects in relation to the entire scene are of comparable size. In this situation, besides the local information, the transformer decoder should also utilize the global information to consider the whole scene. We thus introduce the non-learnable query points for global information. Specifically, we use the Farthest Point Sampling (FPS) [40] for the input point cloud data and take the sampled points as the query points. These query points are frozen during training. Because of the FPS, the sampled points tend to cover the whole scene and can well compensate for objects that learnable query points ignore.
Figure 4: The structure of our detection transformer. The decoder is built on extracted voxel features and we introduce 3D points as the query. The points and voxels interact through cross-attention.
As we utilize the voxel-based backbone for feature extraction, vox-
elization is a necessary step for processing the point cloud data. After voxelization, the coordinates and numbers of the raw point clouds may change slightly. To consider the structure of both the raw data and points after voxelization, what we use is two kinds of non-learnable query points  $P_{nl}$  and  $P_{nlv}$ .  $P_{nl}$  denotes the non-learnable query points from FPS the original point clouds, and  $P_{nlv}$  comes from sampling the points after voxelization. Together with the learnable query points  $P_l$ , the mixture of query points ultimately consists of three ones:  $P = \{P_l, P_{nl}, P_{nlv}\}$ . We follow [5] to perform group-wise self-attention in the transformer decoders for the mixture of query points, where these three kinds of query points do not interact with each other. After transformer decoders, three sets of predicted 3D boxes  $\{bb_l, bb_{nl}, bb_{nlv}\}$  are predicted respectively. We apply one-to-one assignment to each set of 3D boxes independently using the Hungarian matching algorithm [23] and calculate the corresponding training loss to supervise the learning of the detection transformer.
At the test time, besides the above three kinds of query points, we further generate a series of additional points uniformly and randomly in the 3D voxel space as another set of query points  $P_{rd}$ . These randomly generated points evenly fill the whole scene, thus well compensating for some potentially missed objects, especially for the indoor scenes. As a result, four groups of query points  $P = \{P_l, P_{nl}, P_{nlv}, P_{rd}\}$  are utilized for inference. Ultimately, four sets of predicted 3D boxes  $\{bb_l, bb_{nl}, bb_{nlv}, bb_{rd}\}$  are produced at the test time by Uni3DETR. Because of the one-to-one assignment strategy, the individual box sets do not require additional post-processing methods for removing duplicated boxes. However, there are still overlapped predictions among these four sets. We thus conduct box merging finally among the box sets to fully utilize the global and local information and eliminate duplicated predictions. Specifically, we cluster the 3D boxes based on the 3D IoU among them and take the median of them for the final prediction. The confidence score of the final box is the maximum one of the clustered boxes.
2.3 Decoupled IoU
Existing transformer-based 2D detectors usually adopt the L1 loss and the generalized IoU loss [42] for predicting the bounding box. The generalized IoU here mitigates the issue of L1 loss for different scales thus provides further positional supervision. However, for 3D bounding boxes, the calculation of IoU becomes a more challenging problem. One 3D box is usually denoted by:  $bb = (x,y,z,w,l,h,\theta)$ , where  $(x,y,z)$  is the center,  $(w,l,h)$  is the size and  $\theta$  is the rotation. The usual 3D IoU [72] between two 3D boxes  $bb_{1}$  and  $bb_{2}$  is calculated by:
$$
\mathrm {I o U} _ {3 D} = \frac {\text {A r e a} _ {\text {o v e r l a p p e d}} \cdot z _ {\text {o v e r l a p p e d}}}{\text {A r e a} _ {1} \cdot z _ {1} + \text {A r e a} _ {2} \cdot z _ {2} - \text {A r e a} _ {\text {o v e r l a p p e d}} \cdot z _ {\text {o v e r l a p p e d}}} \tag {2}
$$
where Area denotes the area of rotated boxes in the  $xy$  space.
From Equ. 2, we notice that the calculation of 3D IoU generally consists of two spaces: the  $xy$  space for the bounding box area, involving the  $x, y, w, l, \theta$  variables, and the  $z$  space for the box height. The two spaces are multiplied together in the 3D IoU. When supervising the network, the gradients of these two spaces are coupled together - optimizing in one space will interfere with another, making the training process unstable. For 3D GIoU [72], besides the negative coupling effect, the calculation of the smallest convex hull area is even non-optimizable. As a result, the usual 3D IoU is hard to optimize for detectors. For our detection transformer, where L1 loss already exists for bounding box regression, the optimization problem of 3D IoU severely diminishes its effect in training, thus being hard to mitigate the scale problem of L1 loss.
Therefore, an ideal metric for transformer decoders in 3D should at least meet the following demands: First, it should be easy to optimize. Especially for the coupling effect for different directions, its negative effect should be alleviated. Second, all shape properties of the 3D box need to be considered, so that accurate supervision signals can be provided for all variables. Third, the metric should be scale invariant to alleviate the issue of L1 loss. Motivated by these, we forward the decoupled IoU:
$$
\mathrm {I o U} _ {d e} = \left(\frac {\text {A r e a} _ {\text {o v e r l a p p e d}}}{\text {A r e a} _ {1} + \text {A r e a} _ {2} - \text {A r e a} _ {\text {o v e r l a p p e d}}} + \frac {z _ {\text {o v e r l a p p e d}}}{z _ {1} + z _ {2} - z _ {\text {o v e r l a p p e d}}}\right) / 2 \tag {3}
$$
Specifically, our decoupled IoU is the average of the IoU in the  $xy$  space and the  $z$  space. Under the summation operation, the gradients from the two items remain separate and independent, avoiding any coupling effect. According to previous research [47], the coupling issue does not exist in 2D or 1D IoU. As a result, the decoupled IoU effectively circumvents the negative effects of coupling. The scale-invariant property of 3D IoU is also well-reserved for the decoupled IoU. These make our decoupled IoU well suit the training of the transformer decoder.
The decoupled IoU is introduced in both the training loss and the matching cost. We also introduce it into classification and adopt the variant of quality focal loss [26]. Denote the binary target class label as  $c$ , the predicted class probability as  $\hat{p}$ , the classification loss for Uni3DETR is:
$$
L _ {c l s} = - \hat {\alpha} _ {t} \cdot | c \cdot \operatorname {I o U} _ {d e} - \hat {p} | ^ {\gamma} \cdot \log (| 1 - c - \hat {p} |) \tag {4}
$$
where  $\hat{\alpha}_{t} = \alpha \cdot c \cdot \mathrm{IoU}_{de} + (1 - \alpha) \cdot (1 - c \cdot \mathrm{IoU}_{de})$ . The above classification loss can be viewed as using the soft target  $\mathrm{IoU}_{de}$  in focal loss [28]. The decoupled IoU is easy to optimize, allowing us to avoid the need for performing the stop-gradient strategy on  $\mathrm{IoU}_{de}$ . Therefore, the learning of classification and localization is not separated, assisting Uni3DETR predicting more accurate confidence scores. Equ. 4 will force the predicted class probability  $\hat{p}$  towards  $\mathrm{IoU}_{de}$ , which might be inconsistent with  $\mathrm{IoU}_{3D}$  used in evaluation. We thus follow [22] and introduce an IoU-branch in our network to predict the 3D IoU. The normal  $\mathrm{IoU}_{3D}$  supervises the learning of the IoU-branch and the binary cross-entropy loss is adopted for supervision. The weighted geometric mean of the predicted 3D IoU and the classification score is utilized for the final confidence score. The final loss function for training Uni3DETR is thus the classification loss (Equ. 4), the L1 loss and the IoU loss with  $\mathrm{IoU}_{de}$  for bounding box regression, and the binary cross-entropy loss for IoU prediction.
To demonstrate the universality of our Uni3DETR under various scenes, we conduct extensive experiments in this section. We evaluate Uni3DETR in the indoor and outdoor scenes separately.
Table 1: The performance of Uni3DETR for indoor 3D object detection. The main comparison is based on the best results of multiple experiments. We re-implement VoteNet, H3DNet, and GroupFree on the S3DIS dataset. * indicates the multi-modal method with both point clouds and RGB images.
Method	SUN RGB-D	ScanNet	S3DIS
AP25	AP50	AP25	AP50	AP25	AP50
ImVoteNet* [37]	63.4	-	-	-	-	-
TokenFusion* [60]	64.9	48.3	70.8	54.2	-	-
VoteNet [38]	57.7	35.7	58.6	33.5	58.0	25.3
GSDN [17]	-	-	62.8	34.8	47.8	25.1
H3DNet [70]	60.1	39.0	67.2	48.1	50.9	22.0
BRNet [9]	61.1	43.7	66.1	50.9	-	-
3DETR [35]	59.1	32.7	65.0	47.0	-	-
VENet [62]	62.5	39.2	67.7	-	-	-
GroupFree [30]	63.0	45.2	69.1	52.8	42.8	19.3
RBGNet [59]	64.1	47.2	70.6	55.2	-	-
HyperDet3D [71]	63.5	47.3	70.9	57.2	-	-
FCAF3D [43]	64.2	48.9	71.5	57.3	66.7	45.9
Uni3DETR (ours)	67.0	50.3	71.7	58.3	70.1	48.0
Datasets. For indoor 3D detection, we evaluate Uni3DETR on three indoor 3D scene datasets: SUN RGB-D [53], ScanNet V2 [12] and S3DIS [1]. SUN RGB-D is a single-view indoor dataset with 5,285 training and 5,050 validation scenes, annotated with 10 classes and oriented 3D bounding boxes. ScanNet V2 contains 1,201 reconstructed training scans and 312 validation scans, with 18 object categories for axis-aligned bounding boxes. S3DIS consists of 3D scans from 6 buildings, 5 object classes annotated with axis-aligned bounding boxes. We use the official split, evaluate our method on 68 rooms from Area 5 and use the rest 204 samples for training. We use the mean average precision (mAP) under IoU thresholds of 0.25 and 0.5 for evaluating on these three datasets.
For outdoor 3D detection, we conduct experiments on two popular outdoor benchmarks: KITTI [16] and nuScenes [3]. The KITTI dataset consists of 7,481 LiDAR samples for its official training set, and we split it into 3,712 training samples and 3,769 validation samples for training and evaluation. The nuScenes dataset is a large-scale benchmark for autonomous driving, using the 32 lanes LiDAR for data collection. We train on the 28,130 frames of samples in the training set and evaluate on the 6,010 validation samples. We use mAP and nuScenes detection score (NDS), a weighted average of mAP and other box attributes like translation, scale, orientation, velocity.
Implementation details. We implement Uni3DETR with mmdetection3D [11], and train it with the AdamW [32] optimizer. We set the number of learnable query points to 300 for datasets except for nuScenes, where we set to 900. For indoor datasets, we choose the  $0.02\mathrm{m}$  grid size. For the KITTI dataset, we use a  $(0.05\mathrm{m}, 0.05\mathrm{m}, 0.1\mathrm{m})$  voxel size and for the nuScenes, we use the  $(0.075\mathrm{m}, 0.075\mathrm{m}, 0.2\mathrm{m})$  voxel size. The nuScenes model is trained with 20 epochs, with the CBGS [75] strategy. For outdoor datasets, we also conduct the ground-truth sampling augmentation [64] and we remove the ground-truth sampling at the last 4 epochs. Dynamic voxelization [73] and ground-truth repeating [21] are also adopted during training. Besides these data-related parameters, other architecture-related hyper-parameters are all the same for different datasets.
3.1 Indoor 3D Object Detection