Unifying Voxel-based Representation with Transformer for 3D Object Detection
Authors: Yanwei Li
Yilun Chen
XIAOJUAN QI
Zeming Li
Jian Sun
Jiaya Jia
Venue: NeurIPS 2022 Accept (2022)
Keywords: Unified representation
Multi-modality input
3D object detection

Abstract: In this work, we present a unified framework for multi-modality 3D object detection, named UVTR. The proposed method aims to unify multi-modality representations in the voxel space for accurate and robust single- or cross-modality 3D detection. To this end, the modality-specific space is first designed to represent different inputs in the voxel feature space. Different from previous work, our approach preserves the voxel space without height compression to alleviate semantic ambiguity and enable spatial connections. To make full use of the inputs from different sensors, the cross-modality interaction is then proposed, including knowledge transfer and modality fusion. In this way, geometry-aware expressions in point clouds and context-rich features in images are well utilized for better performance and robustness. The transformer decoder is applied to efficiently sample features from the unified space with learnable positions, which facilitates object-level interactions. In general, UVTR presents an early attempt to represent different modalities in a unified framework. It surpasses previous work in single- or multi-modality entries. The proposed method achieves leading performance in the nuScenes test set for both object detection and the following object tracking task. Code is made publicly available at https://github.com/dvlab-research/UVTR.

Unifying Voxel-based Representation with Transformer for 3D Object Detection
Yanwei Li
Yilun Chen
Xiaojuan Qi
Zeming Li
Jian Sun Jiaya Jia
The Chinese University of Hong Kong', MEGVII Technology
The University of Hong Kong² SmartMore*
Abstract
In this work, we present a unified framework for multi-modality 3D object detection, named UVTR. The proposed method aims to unify multi-modality representations in the voxel space for accurate and robust single- or cross cooss-mddality 3D detection. To this end, the modality-specine space is first designed represent different inputs in the voxel feature space. Different from previous work, our approach preserves the voxel space without height compression to alleviate semantic ambiguity and enable spatial connections. To make full use of the inputs from different sensors, the cross-man coosstmod interaction is then proposed, including knowledge transfer and modality fusion. In this way, geometry-awe -aware expressions in point clouds and context-rich features in images are well utilized for better performance and robustness. The transformer decoder is applied to efficiently sample features from the unified space with learnable positions, which facilitates object-level interactions. In general, UVTR presents early attempt to represent different modalities in unified framework. It surpasses previous work in single- or multi-modality entries.
The proposed method achieves leading performance in the nuScenes test for both object detection and the following object tracking task. Code is made publicly available https://gituub.cmm//vi www.ddllwaareear//UUVTTTT
Introduction
Detecting 3D objects with multi-modality sensors (i.e., LiDAR and camera) is regarded as fun- damental task in real-world scenes. For accurate object detection, data from different modalities utilized provide complementary knowledge, like accurate positions from point clouds and rich context from images. Toward this purpose, a unified representation is essential to facilitate knowledge transfer and feature fusion across modalities. However, due the lack of accurate depth from cameras, images can not be naturally represented in voxel space like that of point clouds.
In the unified progress, several representations have been studied that can be roughly separated into input- and feature-level streams. For the first one, multi-modality data is aligned at the beginning of network. In particular, pseudo point clouds in Figure la transformed from image aided by predicted depth [1, 2]. while the range-view image in Figure lb is projected from point clouds [3, 4]. Because of inaccurate depth in pseudo point clouds and collapsed 3D geometry in range-view images, data is damaged, which brings inferior results. For feature-leve method, a typical approach is to transform image features as frustum and then compress BEV space [5, 6], like that in Figure lc. However, due to ray-like like trajectories the frustum [7]. height compression each position aggregates features from various objects and thus introduces semantic ambiguity. Meanwhile, other implicit manners in contemporary work [8, 9, 10] can hardly support explicit feature interactions in 3D space and restrict further knowledge transfer. Therefore, more unified representation desired to bridge modality gap and facilitate interactions from multiple aspects.
the spatial structure
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Y
X
predicted depth
3D Conv
project
20 Conv
Image
Pseudo Point Cloud
Backbone
Paint Cloud
Ronge View
Backbone
(a) Pseudo point cloud transformed from image
(b) Range view projected from point cloud
Y
X
frustrum
somple
feature
2D Conv
with prob
value
position
(x,y,
Image
CNN-based
Imoge
Transformer
BEV Feoture Space
Image Voxel Space
Feature
Head
Feature
Decoder
(c) BEV feature space transformed from image feature
(d) Voxel feature space sampled from image feature
Figure 1: Toy example of methods for unified representation. Compared with others, the proposed
manner in 1d constructs the voxel space by sampling features from the image plane and represents multi-modalities uniformly without height-level compression in I that brings semantic ambiguity.
In this paper, present simple yet effective framework unify the voxel-based representation with transformer, called UVTR. In particular, features from images and point clouds are represented and interacted in the explicit voxel-based space. For images, we construct the voxel space by sampling features from the image plane according to predicted depth scores and geometric constraints, as briefly depicted in Figure 1d. For point clouds, the accurate position naturally allows to associate features with voxels. Then, voxel encoder is introduced for spatial interaction that establishes the relationship among adjacent features. In this way, cross coos-moo interaction is naturally conducted with features each voxel space. For object-level interaction, deformable transformer [11] is adopted the decoder that samples specific feature for each object query with position (ar, y: in the unified voxel space, as illustrated in Figure 1d. Meanwhile, the introduction of 3D query position efficiently alleviates the semantic ambiguity brought by height compression in BEV space as analysed before.
Compared with previous and even concurrent studies [8, 9], more key advances can achieved with the proposed framework. First, the explicit voxel-based representation supports spatial interaction in 3D space and multi-frame scenes that bring significant improvements. Second, the proposed unified manner facilitates coos-mto learning and can be naturally applied for knowledge transfer and feature fusion, which further boosts the performance. Finally, data augmentation for both modalities can be directly synchronized in the voxel space without the complex aligning process [12, 7J.
The overall framework, called UVTR, can be easily instantiated and improved with various image- voxel-based backbones for single- and multi-modalle 3D object detection. Extensive empirical studies are conducted in Section to reveal the effect of each component. The proposed UVTR attains leading performance in various settings. For detection, it achieves 69.7%, 55.1%, and 71.1% % NDS on nuScenes test set with point clouds, images, and multi-modality inputs, respectively. Given naive association strategy, UVTR also achieves strong tracking results with 67.0%, 51.9%, and 70.1% AMOTA on L.DAR-basedl camera- based, and molti-moddlity setting, respectively.
2 Related Work
LiDAR-based 3D Detection. With point clouds captured from LiDAR, traditional methods process
the irregular input and generate 3D boxes with different representations, e.g., point, voxel, and range view. Point-base based detectors usually aggregate features from raw point clouds with set abstraction [13] and then predict box proposals [14, 15, 16]. For voxel- methods, point clouds are transformed into regular grids and processed with 3D sparse convolutions [17, 18] or 2D convolutions [19, 20, 21] top of the bird-eye view (BEV) space with the flatted height axis [22, 23, 24]. There are also studies [3, 4] that project point clouds to range view and process them like images. However, due to the collapsed 3D geometry in range-view images,
directly. Final predictions are usually generated

---

View Tronsform
X
Objectt Detection
Voxel Encoder
XM
X
Vi
X
Image
Image Bockbone
Image Vosel Space
Knowledge Transfer
value
Alteetion
posittin
(X,y,))
X
Vi
Attention
V/
Xy
Fr
Encoder
Unified Space
Object
Cloud
Backbone
Voxel Space
Mosiality Switch
Transformer
Figure 2: The framework of UVTR with multi-modality input. Given single- or multi-frame images and point clouds, we first process them in individual backbone and convert to modality-speciff space V, and Vp, where view transform is utilized for that of image. voxel encoder, features are spatially interacted, and knowledge transfer is easily supported during training. Single- multi-modality features are selected via modality switch according to different settings. Finally, transformer decoder utilized for prediction by sampling features from the unified space Vy with learnable positions.
the relationship in point clouds cannot be fully explored. this work, we follow the voxel-based pipeline but keep the fine- fine-grained voxel space without height compression, as shown in Figure 2.
Camera-based 3D Detection. Camera-based methods perform 3D detection single- multi-view images. With monocular image, previous approaches try to predict 3D boxes based on image features directly [25, 26, 27] utilize the middle representation [1, 2, 5]. For multi-view input, image features are usually optimized in the constructed 3D geometry volume [28, 29]. Most recently, multi-view features are projected and merged in the frustum feature space with the aid of predicted depth [6]. Following the LiDAR-base Rbased paradigm, the frustum feature is collapsed to the BEV space, as briefly introduced in Figure lc. However, LiDAR, which brings semantic ambiguity to BEV space. Other recent studies try to capture geometry clues from multi-view images in an implicit manner [8, 10], which losses the chance for direct spatial interactions. In this paper, we represent image features in an explicit voxel space to alleviate the semantic ambiguity and facilitate further feature interactions, as depicted Figure ld.
accuracy of the predicted depth map is much inferior to that of
Cross-modality Interaction. With input data from various sensors, cross-modality interaction is conducted benefit from different inputs, e.g., modality fusion and knowledge transfer. For modality fusion. the model takes data from different sensors and conducts fusion at point- and instance-leve.. Specifically, point-level fusion [30, 31, 32, combines features from different modalities the early stage of the network, which enables sufficient interaction. And instance-level fusion [33, 34, 35] usually applied at the later stage to combine object-level features. Cross-modas mods--yy knowledge transfer aims to distill specific knowledge [36] across modalities in the training phase. Compared with cross-modality fusion, knowledge transfer is seldom studied for 3D object detection. A prior work is LIGA-Stereo [37] that transfers geometry-aware representations from LiDAR to stereo images via distillation. Different from [37], UVTR represents each modality in a unified manner and supports cross-modality fusion and knowledge transfer simultaneously, which further enables distillation from multi-modality or consecutive frames to the single input.
3 UVTR Framework
The overall framework of UVTR is relatively simple: modality-specife space is constructed to unify the representation of inputs; cross-modality interaction is designed for feature learning across spaces; and transformer decoder is introduced for object-level interaction and final prediction.
3.1 Modality-speciffe Space
Given images X, captured from cameras and point cloud Xp from LiDAR, different branches are utilized to respectively generate and enhance voxel space for each modality, as presented in Figure 2.

---

Vownl Encoder
H
Conv
RelU
sample prob
S,
mitli Egg
Knowledge Transfer
Tr(R,g
W
F,
V1
Conv
-....
BN
ReLU
Vowel Encoder
Depth Distribution
Image Vaxel Space
Figure 3: Details in the view transform.
Figure 4: Details in the knowledge transfer.
Image Voxel Space. For image voxel space, shared backbone is adopted extract features from multi-view or multi-frame images. In this process, FPN [38] is utilized for multi-scale context aggregation that is summed to formulate the feature F, #x where H and W vary with FPN stages. To construct the voxel feature for images, we then transform the image feature of each view the predefined space with the designed view transform in Figure Motivated by [39, 5J, we first generate the depth distribution D, E IR DxHH of each image with a single convolution as
Dy(u, v)
CeltmaxxConvve (F (u, u)).
Here, (u, v) indicates coordinate in the image plane, and is set to 64 to represent the perception limit 64m. It is noted that Dy is predicted without supervision. With the predicted D, in D depth bins, we can easily get the depth distribution each pixel Fy. Let (I,, y, 2) indicates a sampling point that is generated at the center of each bin from the voxel space Vy. The point (u, d) in the image plane is calculated from (æ, y, 2) with the calibration matrix P. where d denotes the reference depth along axis D of Dj. Thus, the corresponding feature in voxel space V1 is easily captured by
(2) where (u(,, v, d) represents the occupancy probability of feature Fy v) in voxel (x., z). For the fralte setting with n sweeps, we use the shared network for all of them and formulate n voxel spaces in total. In this process, each calibration matrix P is aligned to the ego vehicle in the initial frame. To gather temporal cues in each voxel space, relative time offsets from the initial frame attached along the channel axis and merged using single convolution. Then, voxel spaces are concatenated together, and the space-level fusion conducted with a convolutional layer. this way, features along the temporal dimension are integrated into unified space V,, which is proved to bring significant gain in Table 3. Different from methods [5, 6] for BEV space, we preserve the 3D voxel space without collapsing in Z axis to avoid the aforementioned semantic ambiguity and enable further interactions. The effectiveness of the 3D voxel space is empirically studied in Table 1.
(z y, = Dy(u, d.d) X F7 v),
Point Voxel Space. With the accurate position, naturally split point cloud Xp into several regular voxels. Then, the voxel backbone in Figure 2 is utilized to process input voxels with sparse convolution [17]. To enhance multi-scale features in the generated voxel space, parallel heads with various strides are designed to extract feature Fp from the output. In particular, several 2D convolutions applied in each head to aggregate the spatial cues each height. Then, multi-scale features are upsampled to a same resolution and summed together to formulate the voxel space Vp E R X For multi- frame setting with n sweeps, follow previous work [24] and attach all point clouds together with relative time offsets to formulate the input Xp-
Due to the accurate position of point cloud, the semantic ambiguity in 2 axis is much reduced compared with that of images. But we still preserve the 3D space Vp without height compression for convenient cross-modality interaction in Section 3.2 and fine-grained object interaction in Section 3.3. This is also proved to bring superior experimental results in Table 1.
Voxel Encoder. In the above gewerreed space V,, features of adjacent voxels projected from different views have no connection with each other. To solve this issue and facilitate local feature interaction, the voxel encoder is proposed in each voxel space, presented in Figure 2. Specifically, we keep the simplicity of UVTR, and only three basic convolutional blocks are applied in each voxel encoder of Figure 4. In this process, features in each space V, or Vp aggregated in both coplanar and vertical dimensions. The spatial interaction in voxel space establishes connections among adjacent features, which is proved to be essential in Table 2, especially for Vj.

---

3.2 Cross-modality Interaction
With the unified representation in space V and Vp, interactions across modalities can be easily conducted. Given the prior that LiDAR is advanced in localization and cameras provide context for classification, the moos-tt interaction is proposed from two separate aspects, i.e., transferring geometry- knowledge to images in a single-modality setting and fusing context-aware features with point clouds in multi-modality setting. In particular, knowledge transfer aims to optimize the features of the student with guidance from the teacher in the single-modality setting. Meanwhile, modality fusion is designed to better utilize all modalities in both training and inference stages.
Knowledge Transfer. is first designed to optimize features of the student with guidance from the teacher during training, which is important in an environment that lacks multi- moolii-mddlltyy data. Due to inherent properties, the geometry structure contained in images can be further exploited with the aid of point clouds, while the rich context in images can hardly be transferred to sparse point clouds. Therefore, we mainly foeus on transferring knowledge from the geometry-rich modality to the poor one in this work. Benefiting from unified feature spaces, the cross- modosltyy transfer can be easily supported, as illustrated in Figure In particular, we take features before the last ReLU layer in the voxel encoder of V p as the geometry-rich teacher, marked as Tp. Meanwhile, the feature in the same position of Vy is taken as the geometry-poor student, denoted as S, If we take one object query position (zr, y, 2) from Section 3.3, the feature distance for knowledge transfer is formulated as
Considering single modality input in the inference stage, knowledge transfer
dxr = PL2(T P(, U, S,,S,( y, v,2)),
(3)
where PL2 represents the partial L2 distance [40]. Without bells-and-whistee the optimization objective for knowledge transfer averaged from N object queries of transformer decoder in Section 3.3, namely LKT = N (dxx). It should be noted that the whole network is optimized in an end-to-end manner, with no need for extra procedures. Given the object position in each query, we can directly minimize the object-level distance with no need to exclude background features like [37]. In similar pipeline, the knowledge transfer is further extended to support more input streams, like mullti-ramme images. The proposed cross- cross-modality knowledge transfer is flexible with input modalities and brings consistent gains over various baselines in Tables 5 and
Modality Fusion. Different from the knowledge transfer, modality fusion aims to better utilize all modalities in both training and inference stages, which utilizes the complementary knowledge of point cloud and images to improve the performance and robustness. Thanks to the unified representation of each modality, feature fusion can be naturally applied. To be specific, given the processed feature space V and V', we first select candidate modality for final prediction via modality switch, as depicted in Figure 2. That means we support single- or multi-modalle
input for prediction according to different settings. If both modalities are taken, V and Vpp are added together to formulate the unified voxel space Vu E RX XxX x In this way, both modalities are well expressed in a unified manner, which can be further fused with single convolution. The space VU unifies modalities with the explicit representation, which provides an expressive space for object interactions in Section 3.3.
3.3 Transformer Decoder
To obtain accurate and robust predictions, the transformer decoder is utilized for further object-level interaction in the unified voxel space Vy. We draw inspirations from deformable DETR [11] and apply reference positions to efficiently sample representative features, regardless of the spatial size of 3D voxel spaces. In particular, we first initialize N object queries Q E NxC and generate N reference points from object query embedding. Then, object queries other in the self-attention module and summed via skip-connection, as shown in Figure 2. Let represents specific query in Q with corresponding reference point p (I. y, 2). The process of the cross-attention module in Figure 2 is modeled as
interacted with each
(4) where Vy denotes the sampled feature at (a. y. of Vu, and DeformAttn indicates the de- formable attention in [11]. With the
CrossAttanq, Vp(()) = DeformAttu(g, P. Vu),
network and normalization, each object query can easily interact with unified features from Vu inside each block. There are total M blocks in the transformer decoder, where M is respectively set to 3 and 6 for LiDAR-based and camera-based settings. Finally, a shared MI. head utilized for prediction according to the output of each block. And iterative box refinement [1 [11, 8] is applied to refine 3D bounding boxes based on the predictions.
5

---

3,4 Optimization Objectives
Following a general paradigm in recent transformers [41, 11], Hungarian algorithm [42] is adopted for one-to-one target assignment in the training phase. Thus, set-to-set loss Let is computed to optimize detection results, including box regression loss and classification loss. knowledge transfer is applied, the loss LKT is contained to reduce cross-modality feature distance with weight 0.01.
4 Experiments
In this section, we first introduce our detailed experimental setup. Then, analyses of each compo- nent are conducted on different modalities. Comparisons with several leading benchmarks on the nuScenes [43] dataset are presented in the end. More results are attached in supplementary material.
4.1 Experimental Setup
Dataset. nuScenes [43] dataset is a large-scale benchmark for autonomous driving, which is widely adopted for single- multi-modality 3D object detection. It contains 700, 150, 150 scenes in the train, val, and test set, respectively. We use the synced data with 10 object categories that are captured from 32-beam LiDAR at 20Hz and six cameras in 360-degree field of view at 12Hz. Only annotations of keyframes are given at 2Hz. Here, ablation studies are optimized on a mini 1/4 train split by default, and final models are optimized on the whole train set.
Implementation Details. In this work, we conduct experiments on different modalities with 900 object queries Q. Constructed voxel spaces V,, Vp, and Vu share the same shape 128 x 128 x Z, where Z indicates the height of voxel space and is further investigated in Table 1. The channel number C' in voxel spaces and transformer decoder is set 256. And the amount of block M in the decoder is set to 3, 6, and 6 for LiDAR-based, camera-based, and fusion settings, respectively. In particular, for the LiDARRbasedd setting, only the branch with voxel space Vp is kept. With grid size 0.1m, the input point clouds are filtered in range 11.mmm 51.2m for X and axis with [-5.0m, 3.0ml for Z axis. While for grid size 0.075m, the range in X and Y axis is modified to 5m. 54.0m)] The framework is trained with Adam optimizer with an initial learning rate 2e - for 20 epochs. For a camera-based setting, the network is optimized with an initial learning rate 1 for 24 epochs. As for fusion, we initialize two moddlittysspecin branches with corresponding pretrained models and optimize the model with an initial learning rate 4e 5 for 20 epochs. The whole framework trained in an end-to-end manner with different modalities. More details are given in supplementary material.
4.2 Component-wis wise Analysis
In this subsection, we use a randomly sampled 1/4 split of nuScenes train set for efficient validation.
Effect of Height in Voxel Space. As elaborated in Section 3.1, the height axis Z plays a vital role in voxel space, especially for camera-based V, In Table 1, we validate this with different heights on both modalities. Compared with the BEV space with height 1, the increase in height contributes significantly for camera- based V,, which respectively improves 3.1 % and 4.2% NDS with height and 11. Consistent with our analysis, the gain brought by increase of height contributes less to that of LiDAR because of the accurate position, which is up 1% NDS and 1.9% mAP.
Operations in Voxel Encoder. The voxel encoder in Section 3.1 aims to facilitate spatial feature interactions that are essential, especially in the camera-based manner. As presented in Table camera-based network cannot converge if given no spatial interaction. For LiDAR network, it performs satisfactorily without the voxel encoder, which can be attributed to the established relations at the early stage. Overall, for both of them, 3D spatial interaction still plays a vital role and improves 2.6% and 0.6% NDS over 2D convolution for the camera- and LiDAR-base methods, respectively.
Effect of Multi-frame Input. To further release the potential of the designed paradigm, we input multi-frame sweeps and represent them in each voxel space, as shown in Table 3. With more input frames, networks for both modalities achieve consistent gains. The performance gap reaches 5% and 18.1 5% NDS for the camera- and LiDAR-based manner with 5 and 10 sweeps, respectively.
Networks for Voxel Space. In Table 4, we exploit the network for voxel space generation. is clear that the deeper network with larger voxel space contributes more to the final result. With Res

---

Table 1: Effect of different heights Z in voxel space on nuScenes val set.
Table Effect of different operations in voxel encoder on nuScenes val set.
modality
height
NDS(%)
mAP(%%
modality
rype
NDS(%)
mAP(%) (6)
1
31.4
24.9
None
12.0
2.5
Camera
5
34.5
27.0
Camera
Conv2D
31.9
24.8
11
35.6
28.7
Conv3D
34.5
27.0
1
62.8
54.4
None
63.1
54.3
LiDAR
5 63.8
55.5
LiDAR
Conv2D
63.2
54.6
11
63.8
56.3
Conv3D
63.8
55.5
Table 3: Effect different number frames on nuScenes val set. sweep denotes the sweep number multi-frame input.
Table Effect of different models for voxel space construction nuScenes val set. H and denote space height and grid size.
modality
sweep
NDS(%)
mAP(%)
modality voxel net
NDS(%)
mAP(%)
1
34.5
27.0
R50-H5
34.5
27.0
Camera
3
38.0
28.7
Camera
R50-H11
35.6
28.7
5
39.5
29.4
R101-H11
39.4
32.0
1
45.7
42.8
V0.1
63.8
55.5
LiDAR
LiDAR
10
63.8
55.5
VO.075
64.3
56.3
Table 5: Effect of different knowledge transfer settings on nuScenes val set. CS denotes multi- frame camera sweeps.
Table 6: Effect of different network settings for
mross-modality fusion on nuScenes val set. V denotes the split voxel grid size.
student
teacher
NDS(%)
mAP(%)
camera
lidar
NDS(%)
mAP(%)
34.5
27.0
R50
34.5
27.0
CS
36.3
28.1
V0.1
63.8
55.5
Camera
LiDAR
36.4
28.2
VO.1
65.1
59.0
Multi-mod
37.1
28.8
R50
VO.075
65.6
60.1
63.8
55.5
VO.1
65.4
59.4
LiDAR
R101
Multi-mod
64.4
56.1
V0.075
66.3
61.0
and height 11 for space V,, the camera-based method attains nearly 5% gains over the baseline in both NDS and mAP. For LiDAR-base manner, the model performs slightly better with finer grid size that brings higher resolution voxel space. It means the image-based voxel space requires strong backbones to extract expressive features, while the LiDAR- based one less dependent on that.
Knowledge Transfer. Benefiting from the unified representation, knowledge can be easily transferred across modalities. In Table we compare combinations with different students and teachers. In particular, the camera-based student captures geometry geometry-aware cues from camera sweeps or the LiDAR-based teacher, which brings up to 1.9% NDS gain. coupled with context features in multi-modality setting, the gap is enlarged to 2.6% NDS and 1.8% mAP. For the LiDAR- Rbased student, the increase brought by knowledge transfer saturated with 0.6% NDS. This can be attributed to that rich context in images can not be well expressed only with sparse points during inference. Therefore, images are required as input if rich context supplemented, like the following fusion part.
Cross Cross-modality Fusion. In Table 6, we validate capability of UVTR with cross-modality fusion. As presented the table, feature fusion brings significant gains over a single modality. And the LiDAR- based representation dominates the final results, while the camera-based one provides supplementary context. reasonable because point clouds are more accurate position and more representative in geometry expression. Cameras still provide sufficient context for better classification, which yields up to I. 1.6% NDS and 3.9% mAP gain compared with the LiDAR-basee baseline. With finer voxel grids, the performance gap is enlarged to 2.5% NDS and 5.5% mAP.

---

Table 7: Comparisons of different methods with single model the nuScenes val set. We compare with classic methods on different modalities without test-time augmentation. denotes the implementation from MMDetection33 [44]. L, C, CS, and M indicate the LiDAR, Camera, Camera Sweep, and Multi-modality input, respectively. L2 represents knowledge transfer from LiDAR.
Method
Backbone
NDS(%)
mAP
mATE mASEJ mAOEL mAVEJ mAAE|
LiDAR-based
CenterPoint [24]
V0.1
64.9
56.6
0.291
0.252
0.324
0.284
0.189
UVTR-L
V0.1
66.4
59.3
0.345
0.259
0.313
0.218
0.185
UVTR-L
V0.075
67.7
60.9
0.334
0.257
0.300
0.204
0.182
Camera-based
DETR3D [8]
R101
42.5
34.6
0.773
0.268
0.383
0.842
0.216
UVTR-C
R50
41.9
33.3
0.793
0.276
0.454
0.760
0.196
UVTR-C
R101
44.1
36.2
0.758
0.272
0.410
0.758
0.203
UVTR-CS
R50
47.2
36.2
0.756
0.276
0.399
0.467
0.189
UVTR-CS
R101
48.3
37.9
0.731
0.267
0.350
0,510
0.200
UVTR-L2C
R101
45.0
37.2
0.735
0.269
0.397
0.761
0.193
UVTR-L2CS
R101
48.8
39.2
0.720
0.268
0.354
0.534
0.206
+IDAR+Camera
FUTR3D [9] UVTR-M
V0.075-R101
68.3
64.5
V0.075-R100
70.2
65.4
0.332
0.258
0.268
0.212
0.177
Table 8: Comparisons of different distances, weather, and lighting conditions on nuScenes val set.
Distance: NDS(%)
Weather: NDS(%)
Lighting: NDS(%) Duy
Method
Modality
<20m
20-30m
>30m
Sunny
Rainy
Night
CenterPoint [24]
LiDAR
74.1
62.1
34.6
64.6
64.4
65.1
40.1
UVTR-L
LiDAR
75.9
64.9
37.3
67.4
67.9
67.8
41.4
UVTR-C
Camera
52.8
39.7
20.4
43.1
48.3
44.5
23.5
UVTR-M
Multi-mod
77.2
68.2
38.9
69.7
72.0
70.3
42.6
4.3
Main Results
In this section, first report results with various modalities that are optimized on the whole train set. Then, we give analyses of the framework robustness, including camera view drop and translational noise. Comparisons with leading methods on the nuScenes test set presented in the end.
Results with Different Modalities. In Table 7, we carry experiments with different modalities on the nuScenes val set. Compared with classic methods, UVTR achieves significant improvement. In particular, for LiDAR-based method, it attains 1.5% NDS and 2.7% mAP gain over CenterPoint [24]. And a finer resolution contributes better results with 67.7% 37.7% NDS. For camera- based manner, UVTR-C performs better in single frame setting with 1 1.6% NDS gain over DETR3D [8]. If applied knowledge transfer in UVTR-L2C, the gap is enlarged to 2.5% NDS. The performance improves with more frames and attains up to 48.8% NDS. For multi-modality input, UVTR-M achieves 4.5% mAP gain over UVTR-L and outperforms the contemporary FUTR3D [9] with 1.9% NDS in same setting.
Results in Different Conditions. Table 8, we report the performance with different distances. weather conditions, and light situations. (1) Distance: For LiDAR- -bssedd approaches, the proposed UVTR-L achieves better performance in all situations compared with CenterPoint [24]. Equipped with both LiDAR and camera inputs in UVTR-M, the framework attains significant gains, especially in relatively far distance (3.3% NDS gain 20-30m ). If the object is too far (>30m), the performance gain decreases to 1.6% NDS, but still much better than CenterPoin and UVTR-L. (2) Weather condition: is clear that the proposed UVTR-L achieves significant gain compared with CenterPoint in both conditions. And additional camera input brings much better results, especially in rainy weather (4.1% % NDS gain). (3) Light situation: Compared with that in the daylight situation, both LiDAR-based and camera-based approaches perform inferior in the dark night. Compared with

---

70
70
70.2
69.0
68.4
61.7
70.2
70.1
69.9
69.3
68.9
60
60
NDS (%)
Camera
NDS (%)
50
45.0
R2Dan2Ca
LaDAR2Camera
40
44.1
41.3
42.3
39,4
Multi-modality
50
45.0
44.6
43.7
42.3
Multi-modality
30.6
40
43.7
42.7
39.1
30
32.1
29.8
41.3
39.8
38.2
0
0.0
0.2
0.4
0.6
0.8
1.0
Dropped View
Translational Noise (m)
(a) Robustness of dropped camera view
(b) Robustness of sensor calibration noise
Figure 5: We validate the robustness of UVTR by adding two typical errors during inference. For dropped view in 5a, we randomly drop fixed number of camera views to simulate the camera failure. For sensor noise in 5b, we randomly add translational noises LiDAR to camera calibration matrix.
Table 9: Comparisons of leading methods with a single model on the nuScenes test set. L, C, CS, and M indicate the LiDAR, Camera, Camera Sweep, and Multi-modality input, respectively, L2 represents knowledge transfer from LiDAR. Flipping augmentation is adopted for LiDAR. It should be noted that the performance of UVTR-L2CS3 can be further improved with more than sweeps.
Method
Backbone
NDS(%) mAP(%) MATEJ MASE, MAOE/ MAVEJ mAAE/
LiDAR Rbased
3DSSD [45]
Point-based
56.4
42.6
CenterPoint [24]
V0.075
65.5
58.0
HotSpotNet [46]
VO.1
66.0
59.3
0.274
0.239
0.384
0.333
0.133
AFDetV2 [47]
V0.075
68.5
62.4
0.257
0.234
0.341
0.299
0.137
UVTR-L
V0.075
69.7
63.9
0.302
0.246
0.350
0.207
0.123
Camera-
FCOS3D [27]
R101
42.8
35.8
0.690
0.249
0.452
1.434
0.124
DD3D [48]
V2-99
47.7
41.8
0.572
0.249
0.368
1.014
0.124
DETR3D [8]
V2-99
41.2
0.641
0.255
0.394
0.845
0.133
BEVDet [6]
V2-99
48.8
42.4
0.524
0.242
0.373
0.950
0.148
PETR [10]
V2-99
50.4
44.1
0.593
0,249
0.383
0.808
0.132
UVTR-L.2C
V2-99
52.2
45.2
0.612
0.256
0.385
0.664
0.125
UVTR-L2CS3
V2-99
55.1
47.2
0.577
0.253
0.391
0.508
0.123
LiDAR+ Camera
FusionPainting [49]
V0.075-R50
70.4
66.3
MVP [32]
V0.075-DLA34
70.5
66.4
PointAugmenting [50]
V0.075-DLA33
71.0
UVTR-M
V0.075-R101
71.1
67.1
0.306
0.245
0.351
0.225
0.124
CenterPoint, the proposed UVTR-L still performs better. And the camera inputs still bring significant gains in both situations, especially in daylight environment (2.5% NDS gain).
Robustness of the Framework. To validate the robustness of UVTR, we simulate two typical sensor emrors during inference in Figure 5. For loss of view, the multi-modality manner achieves well robustness in Figure 5a. Because LiDAR can still capture surrounding scenes if cameras are offline. As for the camera-based manner, the model losses inputs from dropped scenes, but the network still works well and outputs predictions within the field view. For sensor jitter in Figure 5b, the model performs stable especially in the multi-modalin setting because of the accurate perception from LiDAR. Meanwhile, knowledge transfer consistently improves performance in both settings.
Comparison with Leading Methods. In Table 9, we present comparisons with leading methods on the nuScenes test set. For LiDAR- R-ased method, UVTR-L surpasses AFDetV2 [47] with 1.2% NDS and attains 69.7% NDS. For camera-basee based manner, with single frame, UVTR-L2C outperforms

---

Table 10: Comparisons of different leading tracking methods nuScenes test set. method at the leaderboard with no released publication.
indicates the
Method
Tracker
AMOTA %6)
AMOTP
Recall
LIDAR-based
CenterPoint [24]
Greedy
63.8
0.555 0.656
0.675
UVTR-L
Greedy
67.0
0.703
Camera-
PolarDETR [51]
Transformer
27.3
1.185
0.404
BEVTrack*
Private
34.1
1.107
0.463
UVTR-L2CS3
Greedy
51.9
1.125
0.599
LiDAR+Camera
EagerMOT [52]
Two-stage
67.7
0.550
0.727
AlphaTrack [53] UVTR-M
Dosition+Appeerancee
69.3
0.585
0.723
Greedy
70.1
0.686
0.750
PETR [10] with 1.8% NDS and 1.1% mAP. With 3 camera sweeps, UVTR-L2CS3 obtains significant gain and attains 55.1% 1% NDS, which can be further improved with more frames. For the multi-modality setting, we directly adopt pretrained models from LiDAR- and camera-based manner with simple fine-tuning without bells-and--histles. Compared with similar approaches without special module, UVTR-M achieves 71.1% % NDS and 67. 1% mAP, which is on par with leading approaches.
Tracking Extension. To better illustrate the capability and generality of the proposed UVTR. we further conduct experiments on the downstream tracking task. In particular, we follow the classic macking-by-detectte
paradigm and apply the simple greedy tracker in CenterPoint. The only difference lies in that we adopt threshold 0.2 and NMS to remove low quality results. As presented in Table 10, the proposed UVTR achieves leading tracking performance with the greedy tracker in different settings. Specifically, in a camera-based setting, the proposed UVTR-L2CS3 surpasses previous SOTA at the leaderboard (BEVTrack) with 17.8% AMOTA. It further proves the effectiveness and generality of the proposed cross-modality interaction UVTR.
Discussion and Conclusion
We presented the UVTR, a conceptually simple yet effective framework for multi-modality 3D object detection. The key innovation lies in that it unifies the voxel-based representation for different modalities and facilitates multi-level interactions. In particular, it uniformly encodes inputs from different sensors in the modality space to reduce the semantic ambiguity and enable spatial interaction. With the unified representation, the cross-modality interaction can be casily conducted for knowledge transfer and modality fusion. Moreover, object-level interactions in the unified space are further supported by the transformer decoder for accurate and robust detection. Experiments on the nuScenes dataset prove the effectiveness of UVTR, which attains consistent improvements over various benchmarks and surpasses previous methods with leading performance.
There still exist certain limitations in the current method. First, to construct voxel space for multi-view images, need to process all of them in the shared image backbone, which brings computational cost, especially for multi-frame setting. In the future, we plan to explore a new manner for voxel space construction at the early stage of the network, like that of point clouds. Moreover, we construct the voxel space with a spatial resolution 128 x 128. We believe higher resolution and more image frames could bring stronger voxel space with better results, which remains be explored.
Societal Impacts. The proposed method focuses on 3D object detection that can be used au- tonomous driving. Theoretically, better 3D detector leads safer autonomous vehicles. However, in the short term, the current technique could not solve all the corner cases and extreme situations. may bring potential risks the decision process in real-world autonomous systems.
Acknowledgement. This work supported by Shenzhen Science and Technology Program KQTD2021081 1090049995.
10

---

References
[1] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Camphell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[2] Yurong You, Yan Wang. Wei-Lan Chao, Divyansh Garg. Geoff Pleiss, Bharath Hariharan, Mark Camphell, and Kilian Q Weinberger. Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. In International Conference on Learning Representations. 2020.
[3] Bo Li, Tianlei Zhang, and Tian Xia. Vehicle detection from 3d lidar using fully convolutional network. arXiv:I608, 07916, 2016.
[4] Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Rangedet: In defense of range view for lidar-based 3d object detection. In IEEE/CVF International Conference on Computer Vision, 2021.
[5] Cody Reading, Ali Harakeh, Julia Chae, and Steven Waslander. Categorical depth distribution network for monocular 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[6] Junjie Huang, Guan Huang. Zheng Zhu, and Dalong Du Bevdet: Hiigh-performance multi-camera 3d object detection in bird-eye-view. arXiv:2 17790, 2021.
[7] Yanwei Li, Xiaojuan Qi. Yukang Chen, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Voxel field fusion for 3d object detection. In IEEE Conference on Computer Vision and Puttern Recognition, 2022.
[8] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, 2022.
[9] Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: A unified sensor fusion framework for 3d detection. arXiv:2203. 10644 2022.
[10] Yingfei Liu, Tiancai Wang. Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. M.Xi::2203.06225 2022.
[11] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. International Conference on Learning Representations, 2021.
[12] Wenwei Zhang, Zhe Wang, and Chen Change Loy. Exploring data augmentation for multi- multi-modality 3d object detection, arXiv:20/2. 12741, 2020.
[13] Charles R Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++:: Deep hierarchical feature learning on point sets a metric space. In Advances in Neural Information Processing Systems, 2017.
[14] Zetong Yang. Yanan Sun. Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dent 3d object detector for point cloud. In IEEE/CVF International Conference on Computer Vision, 2019.
[15] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrean: 3d object proposal generation and detection from point cloud. In IEEF Conference Computer Vision and Pattern Recognition, 2019.
[16] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas, Deep hough voting for 3d object detection in point clouds. In IEEE/CVF International Conference on Computer Vision, 2019.
[17] Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks. 1rXiv:1706.01320 2017.
[18] Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. In IEEE Conference Computer Vision and Patter Recognition, 2022.
[19] Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In IEEE Conference on Computer Vision and Patfern Recognition, 2018.
[20] Bin Yang, Ming Liang. and Raquel Urtasun. Hdnet: Conference on Robot Learning. 2018
Exploiting hd maps for 3d object detection. In
[21] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Osear Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In IEEE Conference on Computer Vision and Puttern Recognition, 2019.
11

---

[22] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018.
[23] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnm: Towards high performance voxel-based 3d object detection. In AAAI Conference Artificial Intelligence, 2021.
[24] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In IEEE Conference on Computer Vision and Pottern Recognition, 2021.
[25] Garrick Brazil and Xiaoming I Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In IEEE/CVF International Conference on Computer Vision, 2019.
[26] Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Manuel López-Aniequera, and Peter Kontschieder. Disentangling monocular 3d object detection. In IEEE/CVF International Conference on Computer Vision, 2019.
[27] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Feos3d: Fully convolutional one-stage monocular object detection. In IEEE/CVF International Conference on Computer Vision, 2021.
[28] Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point-based multi-view stereo network. In IEEE/CVF International Conference on Computer Vision, 2019.
[29] Yilun Chen, Shu Liu, Xiaoyong Shen, and Jiaya Jía. Dsgn: Deep stereo geometry network for 3d object detection, JEEE Conference on Computer Vision and Pattern Recognition, 2020.
[30] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In European Conference on Computer Vision, 2018.
[31] Tengteng Huang. Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In European Conference on Computer Vision, 2020.
[32] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3d detection. Advances Neural Information Processing Systems, 2021.
[33] Xiaozhi Chen, Huimin Ma. Wan, Bo Li, and Tian Xia. Multi-view object detection network for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[34] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander. Joint 3d proposal generation and object detection from view aggregation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018
[35] Jin Hyeok Yoo, Yecheol Kim, Jisong Kim, and Jun Won Choi. 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. In European Conference on Computer Vision, 2020.
[36] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv:1503.0233/ 2015.
[37] Xiaoyang Guo, Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Liga-stereo: Learning lidar geometry aware representations for stereo- hereo-based 3d detector. In IEEE/CVF International Conference Computer Vision, 2021.
[38] Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He. Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2017.
[39] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European Conference on Computer Vision, 2020.
[40] Bycongho Heo, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A com- prehensive overhaul of feature distillation. In IEEE/CVF International Conference on Computer Vision, 2019.
[41] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020.
[42] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly. 1955.
12

---

[43] Holger Caesar, Varun Bankiti. Alex H Lang. Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[44] MMDetectionSD Contributors. MMDetection Open Lab next-generation platform for general 3D object detection, batps://githen com/ wttpeeammmmmmmmmeetetttooo
2020.
[45] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[46] Qi Chen, Lin Sun, Zhixin Wang, Kui Jia, and Alan Yuille. Object as hotspots: An anchor- frree 3d object detection approach via firing of hotspots. In European Conference on Compuder Vision, 2020.
[47] Yihan Hu, Zhuangzhuang Ding, Runzhou Ge, Wenxin Shao, Li Huang, Kun Li, and Qiang Liu, Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds. AAAI Conference on Artificial Intelligence. 2022.
[48] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In IEEE/CVP International Conference Computer Vision, 2021.
[49] Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Zhou Bin, and Liangjun Zhang. Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection. In IEEE International Conference on Intelligent Transportation Systems, 2021.
[50] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmentation for object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[51] Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Chang Huang, and Wenyu Liu. Polar parametrization vision-based surround -iew 3d detection. arXiv:2206. 10965, 2022.
[52] Aleksandr Kim, Aljoša OSep, and Laura Leal-Taix6 Eagermot: 3d multi-object tracking via sensor fusion. In IEEE International Conference on Robotics and Automation, 2021.
[53] Yihan Zeng, Chao Ma, Ming Zhu, Zhiming Fan, Xiaokang Yang. Cross-modal object detection and tracking for auto-driving. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section 5.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all thcoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URLY? [Yes] We give detailed experimental setup in Section 4.1 and publicly release the code.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1 and supplementary material.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of
GPUs, internal cluster, or cloud provider)? [Yes] We report them in the supplementary
material. Our experiments in this paper require about 8 NVIDIA Tesla V100 GPUs.
13

---

4. If you are using existing assets (e.g., code, data, models) or curating/releasigg new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.1.
(b) Did you mention the license of the assets? [Yes] We conduct experiments on nuScenes [43] dataset with custom non-commercial license.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] We conduct experiments only on public datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "T_1IOSu7Qy3", "reviewer": "Reviewer_eNGY", "summary": "The paper proposes a new multimodal architecture that receives images and a three-dimensional point cloud and detects 3D objects. The new architecture is based on fusing feature vectors both extracted from voxel spaces and feeding a transformer decoder with an MLP classifier. Besides being able to combine image and point cloud, experiments showed that the proposed strategy can also work in the presence of noise in the point cloud or lack of image data. Furthermore, the experimental results indicate that the proposed architecture is superior to several baselines in terms of NDS and mAP metrics.", "questions": "1.  It is unclear how the “sample with prob” (Figure 3) is being done. According to the text, it seems that the features are being weighted by the occupancy grid store in the tensor D. Why is this a sample and with probability?\n2.  Why is FPN being used? Why partial L2 distance? Why apply deformable attention and not a vanilla attention layer? Is there any property in the task that suggests those are the better choice?\n3.  The voxel encoder is shallow. Please provide more details and motivation related to the design choices.\n4.  The training is not clear. Is the network training in an end-to-end manner? Are the weights of the image and voxel backbone frozen? \n5.  Regarding the knowledge transfer and the training, it is not clear whether there is stage-wise training being applied or all the losses are used to train the networks in an end-to-end manner. Please kindly consider improving the description of the network training.\n6.  Why is not used a linear layer or an MLP to learn how the weigh the features from the modalities before combining them?", "limitations": "Yes. The limitations were adequately addressed in the Discussion and Conclusion section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths: \n    - While every single contribution taken separately seem limited, the integration of all idea appears to be an interesting contribution to field object detection. As far as robust multimodality fusion is concerned, the paper has shown a coherent and wise approach to provide good performance in the presence of noise and lack of data.\n\n    - The experiments are done thoroughly, and the discussion section clearly interprets the observations from the results.\n\n- Weakness:\n    - Many important details for design choices and experiments are either missing or not properly addressed. These points are summarized and listed in the “Questions”.\n \n    - Although the experimental results showed that the proposed approach is superior to the baselines, in several cases (multimodal approaches), the superiority is marginal, e.g., 71.1 versus 70.0, 67.1 versus 66.8. \n \n    - Many technical components (particularly the simple ones) are adopted without strong motivation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PapyNMDTWHx", "reviewer": "Reviewer_Bb8v", "summary": "In this paper, the authors proposed a 3D detection framework by projecting various sensor data, which are acquired at the same time, onto a voxel feature space. They first point out the limitation of conventional methods where input- or feature-level data are represented as 2D data with reduced dimensions (range image, predicted depth map, etc.). To overcome these types of issues, they introduce a unified voxel space without loss of spatial information. Additionally, the authors make the neural network more effective in extracting sensor fusion features by applying for knowledge transfer / Data augmentation without the complex aligning process.", "questions": "I think the authors' cross-modality interaction module should be validated in a more extreme environment. Rather than simply comparing whether features extracted from individual sensors are used, it is necessary to test results in an environment where one of the two sensors does not work at all. (For example, experiments on whether detection is possible only with camera data at a long distance where lidar sensor data does not exist, and experiments on cases where the camera does not work well at night or when passing through a tunnel entrance)\n\nIntroduction of deformable attention for cross attention: Unlike the image space, which contains various scales, the size of the object is determined in the actual 3D space. Therefore, it is not necessary to obtain information of various scales using deformable convolution. Please explain the reason for using deformable attention in unified voxel space. Additionally, can the authors attach a comparison with a network using local attention (basic conv.)?\n", "limitations": "The authors mentioned the shortcomings of their detector and the direction to be taken in the future. The disappointing point is that they restate known limitations, where voxel-based methods have a large increase in computation voxel-based methods have a large increase in the amount of computation compared to other representation methods (e.g., range view / bird’s eye view), in this field.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "(+) They improved the performance of the 3D detector by devising a sensor fusion method on the voxel representation that is richer than the bird-eye's view representation. Through sections 3 and 4 and the supplementary, the authors provide sufficient information to reconstruct and train the proposed network.\n\n(-) The relationship between modality transfer learning techniques and cross-modality interaction feels awkward. As mentioned by the authors, Image and Lidar points are already complementary to each other, so just using the two features should improve performance. However, the knowledge transfer method seems to improve performance by making the features extracted from images similar to the features extracted from the lidar sensor data.\n\n(-) The authors need to conduct performance evaluation in various environments to reveal the advantages of sensor fusion well.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HbpSkyoYtaQ", "reviewer": "Reviewer_Fn2u", "summary": "This paper focuses on 3D object detection in driving scenarios. It proposes to fuse image features with LiDAR point cloud features following a late-fusion pipeline. The fusion leverages a learned unified 3D voxel feature space. Moreover, it adopts a transformer decoder for object relationships\tmodeling. A feature mimicking based method is also proposed for knowledge transfer between two input modalities. This paper demonstrates 3D detection performance improvement over single-modality and other joint-modality competing methods on the nuScenes benchmark.", "questions": "Please see the bullets mentioned in the weakness and limitation sections.", "limitations": "L-1) I’m wondering if the authors have also done experiments on comparing deformable transformer decoder with the widely adopted CenterPoint [1] detection head. This can help justify the importance and advantages of modeling object-level interactions.\n\nL-2) Can the authors provide comparisons on the model inference speed and computation cost? We all know that latency control is critical for 3D object detection in autonomous driving scenarios.\n\nL-3) Can the authors also provide results on other more widely adopted benchmarks such as Waymo Open Dataset and KITTI Dataset?\n\n[1] Yin, Tianwei, Xingyi Zhou, and Philipp Krahenbuhl. \"Center-based 3d object detection and tracking.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "S-1) Image and LiDAR point cloud fusion is a challenging research problem with high real-world application values.\n\nS-2) The cross-modality interaction is simple yet effective. It could be an easily applied method for knowledge transfer between different sensors. \n\nS-3) It is a good choice of using the transformer decoder as detection head because it can capture object-level relationships. Currently, most 3D object detection heads are based on center heat-map estimation, which lacks the ability to model interactions among different objects.\n\nS-4) For the multi-modality fusion model training, it is a good strategy to first separately pre-train the branch of each modality and then conduct fine-tuning on the fusion model. Such a staged training pipeline can improve the training robustness and this is enabled by the proposed unified voxel feature space.\n\nS-5) The ablation studies are extensive, which demonstrate the effectiveness of each proposed module and network architecture design. For example, Table 1 justifies the necessity of using 3D voxels instead of 2D pillars when encoding lifted image features.\n\nS-6) Figure 5 is inspiring. It clearly demonstrates the robustness advantages of using multi-modality inputs.\n\nW-1) In Eq. (1), how is the model trained to estimate the depth distribution $D_{I}$? For example, what losses are used? Moreover, it’s non-trivial to acquire dense depth annotation for each camera view.\n\nW-2) For camera branch, the voxel resolution is $128 \\times 128$ but $D$ is only set to 64. Thus, how do you lift camera features into the voxel space when determining the occupancy probability for voxel centers? Do you do some type of interpolation along the estimated occupancy rays?\n\nW-3) For Table 2, how is 2D convolution used as encoders in a 3D voxel space?\n\nW-4) In Table 5, for camera student and multi-modality teacher, where are the target/teacher features extracted from? Are they from the camera branch or the LiDAR branch, or a mixture of both?\n\nW-5) Also in Table 5, for the best performing multi-modality fusion model, how is knowledge transfer used in its training recipe? It’s not quite clear how does knowledge transfer benefit the fused model. For example do you first train the LiDAR model, and then distill and train the camera model, which followed by the joint fine-tuning?\n\nW-6) Does the transformer decoder jointly detect different types of objects (e.g. pedestrians, vehicles)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept