Multi-Stage Fusion for Multi-Class 3D Lidar Detection

1

Zejie Wang1 *, Zhen Zhao2 , Zhao Jin2 , Zhengping Che2 , Jian Tang3 , Chaomin Shen1 , Yaxin Peng4
School of Computer Science and Shanghai Key Laboratory of Multidimensional Information Processing,East China Normal University
2
Didi Chuxing
3
Syracuse University
4
Department of Mathematics, School of Science, Shanghai University
51194506037@stu.ecnu.edu.cn, alexzhaozhen@163.com{jinzhao, chezhengping}@didiglobal.com, jtang02@syr.edu
cmshen@cs.ecnu.edu.cn, yaxin.peng@shu.edu.cn

Abstract
In autonomous driving, the robust and accurate perceptions of the environment is a fundamental and challenging
task. Resorting to the advancing of different sensors suchas LiDAR and Camera, the autonomous systems are able tocapture and process complementary perceptual information
for better detection and classifying objects. In this paper, we
propose a LiDAR-Camera fusion method for multi-class 3D
object detection. The proposed method makes the utmost
use of data from the two sensors by multiple fusion stages,
and can be learned in an end-to-end manner. First, we apply a multi-level gated adaptive fusion mechanism with thefeature extraction backbone. This point-wise fusion stage
assiduously exploits the image and point cloud inputs, andobtains joint semantic representations of the scene. Next,
given the regions of interest (RoIs) proposed based on the
LiDAR features, the corresponding Camera features are selected by RoI-based feature pooling. These features areused to enrich the LiDAR features in local regions and enhance the proposal refinement. Moreover, we introduce a
multi-label classification task as an auxiliary regularization to the object detection network. Without relying on extra labels, it helps the model better mine the extracted features and discover hard object instances. The experiments
conducted on the KITTI dataset have proved all our fusion
strategies are effective.

* Work done primarily while Zejie Wang was an intern at Didi AI Labs,
Didi Chuxing.

1. Introduction
The 3D object detection tasks [2, 29, 24, 11, 17, 20,
21, 23, 27] create both an opportunity and a challenge forthe intelligent transportation industry as a whole. Demands
for multi-class 3D object detection are increasing in complex traffic situations, particularly in large, metropolitan areas. As the foundational components, LiDAR and Camera
are two most common sensory inputs in autonomous driving. LiDAR points provide 3D structure information, butsuffer from uneven and sparse points distribution. Especially small distant objects are hardly to be recognized by
a LiDAR-only model due to extremely low density points.
Cameras can capture images with rich semantic features
while inevitably lack depth information. The demonstrationis shown in Figure 1. To fully utilize the advantage of eachsensor modality, LiDAR points and image features are combined to enhance detection accuracy. However, many existing works [2, 12, 11, 20, 5] tend to focus on fusing vehicle mixture perceptual information rather than other classes,
such as pedestrian and cyclist. Other fusion works [24, 21]
requires additional complex networks for a priori reasoning.
To fill this gap, we propose an end-to-end learnable architecture for fusing Camera and LiDAR sensors in a featurewise manner for multi-class 3D object detection.
The structure of the fusion model can be divided into
multi-view projection fusion [2, 12, 11, 20] and feature
mapping fusion [24, 21, 23]. In the multi-perspective projection fusion method, most of the works adopt via perspective projection and voxelization to quantify point cloud topseudo-image by using BEV (birds-eye-view) map. The
BEV format typically comes up with a CNN detection headto predict bounding box, which can be processed by the existing mature convolutional neural network. Unfortunately,Pedestrian

CyclistFigure 1. The problems faced by LiDAR-based multi-class 3D detection. The sparse 3D structure captured by LiDAR cannot effectively express the object. The small goals of ‘Pedestrian’ and‘Cyclist’ are very sparse, such as the red areas. For relatively distant target points, it is almost missing, such as the blue area.

the BEV format progressively breaks down the spatial features and inevitably downscales resolution, and the conversion process is inefficient. As a result, the detection
head fails to regress localization precisely due to insufficient point cloud features.
The feature map fusion method is usually recommended
to employ an image segmentation auxiliary network [1, 16,30] to extract semantic information before fusing with LiDAR features to ensure the effective spatial structure of thepoint cloud. The image segmentation auxiliary network is
off-the-shelf and not learnable. Specifically, it is separately
pre-trained and only performs inference in the fusion network’s training phase, which is efficient. [21] has shownthat the precision of the whole fusion network mainly relieson the performance of the segmentation auxiliary network.
Meanwhile, the recognition situation of the auxiliary segmentation network and the point cloud detection networkis split, and mutual optimization cannot be realized. Moreover, obtaining segmentation mask annotations is less costeffective than bounding box labeling in terms of human resources. It is rarely possible to apply for access to open
source dataset for both image segmentation and LiDAR object annotations at same frames.
To address these challenges, we propose a novel endto-end LiDAR-Camera fusion method, multistage fusion
for multi-class 3D object detection (MSF-MC). We design
a classification-aware auxiliary model which can generate
image features for multi-label prediction to guide LiDAR
backbone to learn more discriminative features. Unlike
popular LiDAR-Camera fusion network [21, 23], insteadof image segmentation annotations, the object classification labels are sufficient to provide multi-label supervision.
Importantly, we apply VGG Network as the segmentation
backbone to emphasize that the performance of image backbone is not critical for the LiDAR stream prediction accuracy. The fusion model leverages the attention mechanism to adaptively inject high-dimensional semantic image features into the LiDAR encoder in point-wise manner
through a lightweight gated network module. As a result,the LiDAR encoder-decoder module could maintain the
point cloud structural feature losslessly and efficiently andreceive supplemental image semantic information. We further add image global semantic information to point cloud
ROIs (regions of interest) to amplify the refinement of the
proposal regression. In order to achieve the correlation between LiDAR predicted objects and corresponding image
object classification, we adopt classification-based regularization mechanisms to efficiently assist point cloud detector
to recognize small objects.
In summary, our key contributions are as follows:
1. This is a novel work for multi-class LiDAR-Camera
fusion that can be used for end-to-end training, which does
not require additional segmentation annotations.
2. We propose a multi-level gated fusion method to retain the original point cloud structure information, whileprovide feature supplement in the ROI.
3. We develop a prediction consistency regularization
mechanism to align the feature gap between LiDAR detection and image classification.
4. We conducted extensive experiments on the authoritative dataset KITTI, which proved the effectiveness of our
proposed method.

2. Related Work
LiDAR-based 3D object detection Like the developmentof image object detection, there are two categories in LiDAR object detection, one stage approach and two stage approach. A typical one stage model consists of a point feature
extraction module and a detection module. Point feature extraction module generally produces bird eye’s view or voxelization grids. The joint detection head is either 2D or 3D
CNN to learn the features for 3D box prediction. Complexyolo [19] projected point cloud to BEV format and used
2D detector. PointPillars [10] encoded point cloud with 6
statistical quantities and stacked voxel features as ‘pillar’.
VoxelNet [29] exploited the PointNet as the backbone to
extract features for each voxel. While Yang et al. [26] applied a 3D CNN to a grouped voxel grid. Though one stage
approach efficiently saves computing resources, as a transformed compact presentation, voxelization inevitably loseoriginal spatial information and result in relatively low precision.
The two-stage approach lifts a 3D region of interests inSet Abstraction(SA)Point-wise
FeaturePoint-RCNN
Fusion Gated adaptive

Fusion network

SA2

SA3

𝑆!%

𝑆!&

SA4

𝑆!"

𝑆!#

FeaturePropagation

RPN

…

SA1

SA
Encoder

ROI
Pooling

Consistency regularization

Point-wise semantic
information fusion

FusionFusionFusionFusion

𝐶!

𝐹#(

𝐹#%

𝑓#
Multi

𝑝&
# Label𝐹#$
𝐶"

Block2Block3Block4

Confidence
Prediction

Regularizatio
n

…

𝐹#'

3D bboxRefinement

Local
regionfusionPrediction

MaxPooling

Block5

VGG-NetFigure 2. The structure of the proposed multistage fusion for multi-class 3D object detection model(MSF-MC). The proposed model uses
PointRCNN as the 3D object detection network, and the VGG-16 structure as the backbone network for image feature extraction to achieve
3-stage fusion. The point-wise semantic information fusion structure is used to realize the point-wise fusion of multi-level image depth
semantic information and point cloud features. The object region fusion structure is used to achieve the feature fusion of the object region.
At the same time, the consistency regularization structure realizes the multi-label prediction results to implement the point cloud prediction
consistency regularization mechanism.

stage one and refine the regression in stage two. Part-a2
net [18] generated ROIs from voxel-based feature extraction layer and adopted intra-object part-aware analysis toenrich ROI features. Votenet [13] utilized PointNet++ [15]as the backbone and introduced the Hough voting principle
to group deep features. Yang et al.[28] used PointNet++[15] as well to keep spatial information and divided ROIs
into voxel grid to fit regular CNN. Generally, the two-stage
approach utilizes PointNet++[15] as the backbone, especially SA layers and FP layers, to generate proposal for therefinement module. It achieves better accuracy by learning
more fine-grained features from the proposal.
Multi-sensor 3D object detection
In the past years,
there has been a rapid rise in the use of multi-sensors. Someworks [14, 22] proposed to produce ROIs from image andapplied the PointNet backbone to extract corresponding LiDAR features. One obvious limitation was 3D frustum viewwas extruded from 2D region, the image backbone dominated the performance of 3D detection. Chen et al. [2]gathered image, front view point cloud and bird view point
cloud as the threefold input branches to generate 3D proposal. The variant research [9] proposed image features in
the proposal generation stage. These works had a cumbersome structure that involved different backbones for eachview though the optimization were end-to-end.
There is a trend that researchers are paying more attention on image semantic information in fusion work rather
than relying on image detection. Huang et al. [5] merged
point cloud features with image semantic features in a pointwise way. Xie et al. [23] proposed an attention fusion module to solely merge 3D proposals and image segmentation
masks. Vora et al. [21] applied a semantic segmentation
network obtaining pixel-wise image segmentation scores, todecorate point cloud for fine-grained semantic understanding. The above works inspire us to take advantages of global
semantic features from images.

3. Method
In this section, we propose a multi-stage fusion model
for multi-class 3D object detection (MSF-MC) that canbe used for end-to-end training. We obtain two different
from LiDAR and Camera. Given the sensory inputs
nx[i] , b[i] , c[i] i=1 donate the LiDAR data set containing nframes, where x[i] presents the point cloud at i-th frame, b[i]and c[i] are the bounding box location and object classification
Meanwhile, the calibrated Camera dataset
 relatively.
nz[i] , c[i] i=1 can be obtained, where z[i] represents image
data corresponding to x[i] . Because the z[i] are synchronized with x[i] , it is possible to use c[i] for supervised learn-ing of the image content without introducing additional annotations. For the convenience of the following presentation, we use x and z to represent the current i-th point cloudand image respectively. We aim to thoroughly use two different sensory inputs to gain the understanding of complementary features, and make the 3D object detection more
precise.
The principle of MSF-MC utilizes the gated adaptive
network to accomplish the effective fusion of multi-level
image semantic information and point cloud features. To
additionally refine the ROIs local feature, image global semantic information is concatenated with ROI module in the
LiDAR stream. we also adopt multi-task training strategy
to perform the consistency regularization of cross-modality
object class prediction. As mentioned in the related work,
we select a widely used point-based 3D object detector,
PointRCNN, as the detection model in the LiDAR stream.
In terms of image stream, to simplify the training andto emphasize the effectiveness of our fusion method, thelightweight VGG-16 model is exploited as the backbone formulti-label learning and semantic feature acquisition. Unlike previous work [23, 21], the efficiency of image backbone won’t dominate our fusion performance. Its structureis shown in Figure 2.

3.1. Point-wise Semantic Information Fusion
In multi-class object detection scenarios, it is crucial
that the learned image and point cloud features include
effective local regions. So we propose to employ pointwise regional self-attention fusion principle in the first stage
to realize sensitivity to effective regions. We exploit the
mapping strategy to leverage the correspondence relationship between LiDAR points and image pixels in a feature
wise manner. LiDAR detection branch consists of four
set abstraction (SA) layers as the feature encoding module.
Given the point cloud x, we gain the point cloud feature
set S l (l ∈ [1, 2, 3, 4]) from these set abstraction layers.
Concurrently we obtain the deep semantic features F l(l ∈ [2, 3, 4, 5]) for the the last four layers of the VGG backbone. In order to establish the mapping correspondence ofS l → F l+1 , where l ∈ [1, 2, 3, 4].
In the same way as in [6, 5], we leverage bilinear interpolation and calibration matrix M to project 3D point
to corresponding image feature where we can sample
 2D
feature map and accordingly obtain the feature set V l(l = 2, 3, 4, 5), an example of which is shown in Figure 3.
Specifically, for a specific point e in a given point cloudspace, according to the mapping matrix and bilinear interpolation method, we can obtain its corresponding positionê in the feature map output by the Camera image branch.
Therefore, we can further obtain point-wise feature V l bysampling the size area of the image feature F l . The number of points in V l+1 is exactly the same as the number

Image Feature 𝐹 !

𝑒̂Sampling

project

Point-wise Image Feature𝑉!

Mapping Matrix M

LIDAR Point 𝑆 !

Figure 3. Project 3D point to corresponding image feature. Using
the mapping matrix M , the point cloud is projected into the image, and the image features are sampled at the corresponding pixel
positions.

of points in S l . With this strategy, the original space ofthe point cloud can be preserved and depth semantic information can be introduced at the same time. As la consequence,
we
could
apply
multi-level
fusionon
S and l
V , which is proved more effective in the following experiment section.
On account of illumination, occlusion and truncation,
discrepant representation might occur during cross modal
perception fusion [6, 5]. Inspired by [4, 7], we propose a gated adaptive fusion mechanism which is able toevaluate the relevance of point cloud feature and image
feature accordingly. Detailed demonstration is shown in
Figure 4. Specifically, by concatenating point-wise image feature V l+1 and point cloud feature S l , we get acompressed feature vector SV l , and obtain weighted feature map from W = σ(M LP (SV l ), where M LP denotes shared weighted feature extraction network and σ issigmoid activation function. To balance the complementary feature, we design two boosting attention equationsl
Sw= S l W and Vwl+1 = V l+1 (1 − W ), where
presents element wise production. We further merge above
two weighted feature to update LiDAR feature S l :l
S l = Sw
⊕ Vwl+1(1)

where ⊕ stands for merge operation. Self-attention featurefusion learning is expected to enhance the relevance of S land F l+1 , in order to achieve effective fusion in original
spatial structure.

3.2. Local Region Fusion
In the previous section we were mainly concerned withpoint-wise fusion of corresponding features in the same
space. Since the global semantic information of the image can directly and effectively express the scene, we suggest that in the second stage, the target area feature of
ROI-Pooling extracted by the RCN N module in PointRCNN should be fused with the global depth semantic in-

1-w

𝑉 !#$

cross-entropy loss:𝑉%!#$

Lml i = y c T log(pm ) + (1 − y c )T log(1 − pm )Updated

𝜎
𝑆𝑉

𝑆!

!

𝑊

MLP

w

!

𝑆 𝑆"!

!
𝑆%

Figure 4. Gated adaptive fusion network. The point-wise features
obtained from two different sensors are fused using relevant attention features to update the point cloud features.

formation of the image to enhance target information andachieve more precise regression prediction. In the image
feature extraction stage, z can obtain the feature F 5 ∈RH×W ×C through the last layer of the feature extractor to generate the global depth semantic feature vector
f = M axP ooling(F 5 ) ∈ R1×C . Similarly, in the current point cloud x, the ROI-Pooling region features R ={r1 , r2 , . . . , rt } can be obtained, where t represents thenumber of features obtained. At this time, there is no conflict between the global semantic information and the characteristics of the target area, so we suggest that the global
semantic information of the image can be directly supplemented with the object region information:tR∗ = {M LP (ri ⊕ f )}i=0(2)

where ⊕ represents a cascading operation on two features.
We use the fused object region features R∗ as the input of
the final refined regression task for training update.

3.3. Multi-Label Prediction Auxiliary Regularization
In this section, we introduce the multi-label training
method and the accomplishment of the consistency regularization on the third stage.

The output prediction vector pm ∈ R1×k consists of kobject classification possibilities. Through multi-label classification loss training, we expect to be able to achieve effective extraction of image features and obtain predictions
containing objects.
Cross-modal multi-objective category regularization
In the RP N module of PointRCNN, we can obtain the
N point-wise features of the current point cloud x. Thesefeatures use the classifier to obtain the corresponding categorical variables to construct the segmentation part of the
background before and after for precise adjustment. The
prediction probability matrix Q ∈ RN ×K of K categories
can be obtained on the N point cloud features. Further,
we can get the multi-category prediction probability vector q m = M ax(Q) ∈ R1×K of the current point cloudx, where M ax is max operation, which takes N × K matrix as the input and return the vector of each column’s max
value. We adopt the distribution-wise asymmetric measure- KL divergence (Dkl ) to accomplish the prediction consistency between two different sensor modalities. We prospect
the usage of KL divergence to make prediction robust andavoid incorrect prediction. In order to maintain the consistency between the prediction generated by the 3D detector
and the prediction generated by the image multi-label target
recognition, we suggest to use Sof tmax function to normalize the vector pm and q m . The loss function of crossmodal regularization is defined as following:
Lkl i = Dkl (pm ||q m ) + Dkl (q m ||pm )(4)

3.4. Overall Learning
We introduce three-stage fusion method to promote theperformance of 3D object detection. We group 3D object
detection loss L3d i , image multi-label classification loss
Lml i and cross-modal consistency regularization loss Lkl i
to achieve multi-task training. The total loss function on alldata is interpreted as:
Lall =

Multi-label learning To make the image feature extractor learn fine-coarse semantic information, we send z toit and obtain F 5 as the input of multi-label classification.
The multi-label classifier is composed of K separate binary classification sub-module, C1 , C2 , . . . , Ck , to predict
K different classes. Each sub-module is made up of 3 × 3
kernel, 512 channel convolution layer and a sigmoid activation function. The labels c of image z could be converted toy c ∈ Rk×1 via a fixed [0, 1] encoding transformation. Thenthe multi-label classifier can be learned by minimizing the

(3)

n
X

(L3d i + λLml i + µLkl i )

(5)

i=0

Among them, λ and µ are trade-off parameters forweighing different loss conditions. We use Adaptive Moment Estimation (Adam) [8] optimization algorithm fortraining.

4. Experiments
We evaluated our proposed fusion method (MSF-MC) on
the KITTI 3D object detection dataset [3]. In the following,we introduce the dataset and model implementation details
in Sec.4.1. In Sec.4.2, we provide the evaluation resultsof our model on KITTI. Extensive ablation experiments are
performed in Sec.4.3. Finally, we provide qualitative visualization results in Sec.4.4.

4.1. Dataset and Implementation Details
The KITTI dataset We evaluate our work on KITTI, thewidely used 3D object detection dataset, which has 7,481
training images and 7518 testing images. The training set
was annotated on both point cloud data and image data, andtesting result must be uploaded to official test server. Following the same splitting protocol as [17], we further divide
the training set to 3712 training samples and 3769 validation
samples. We train our model on three commonest classes,‘Car’, ‘Pedestrian’ and ‘Cyclist’. We use average precision(AP) metric as the comparison method, which is calculating recall at 40. In our experiment, we provide the result foreach official difficulty standard, easy, moderate and hard, ofboth validation samples and testing set. The difficulty standard is defined by size, occlusion and truncation. We also
follow the official IoU threshold protocol, 0.7 for ‘Car’, 0.5
for ‘Pedestrian’ and ‘Cyclist’.

Implementation details We adopt the common settingsin [17]. We use the Adam optimizer to train the network
for 80 epochs. The initial learning rate, momentum parameter and weight decay are set to 0.005, 0.01, and 0.9, respectively. Set the λ=0.01, and µ=0.05. In order to alignthe input of the network, 16384 points are obtained in theviewable area of the Camera as the input of PointRCNN.
For scenes with less than 16384 points, we randomly repeat
these points to obtain 16384 points. We select the point
cloud located between the ranges of (0m, 70.4m),(-40m,40m), and (-3m, 1m) along the x,y and z axes, and delete theremaining areas of invisible points. The four set abstraction
layers subsample and encode the point cloud with sizes of4096, 1024, 256, and 64, respectively. We resize the image
size to [1280, 384] as the input of the VGG Network.

Data augment Many LiDAR-based object detection
methods adopt the data augment [25, 10, 17] to increase theperformance. This data augment strategy sampled all theannotated object from point cloud, and pasted some of themrandomly to the scene which has relatively few amount ofobjects. However, we abandon this effective augment strategy during the whole training phase. The correct matching
between the original point cloud and image pixel must be
guaranteed. Any point-cloud-based data augment method
will dilapidate the matching relationship because it won’t
generate corresponding images.

4.2. Results on the KITTI Dataset
We adopt the 3D/BEV indicator task to show our performance. We first compare with our own replicated object
detection detector that can give multi-category confidence
at the same time, and the results on the validation sets areshown in Table 1. Our proposed MSF-MC method is better
than PointRCNN in all three categories of AP indicators,
and the improvement is obvious for ‘Pedestrian’ and ‘Cyclist’. The average performance of ‘Pedestrian’ and ‘Cyclist’ on the three difficulty levels increased by 2.29% and6.85%, respectively. We can also find that our model still
has excellent performance even in the face of the difficultylevel of ‘hard’. This shows that our proposed method is indeed effective and can significantly improve the categories,
which are inherently small and difficult to detect.
We also provide a performance demonstration of BEV
metrics, as shown in Table 2. Our proposed MSF-MC also
showed excellent performance. Significant improvementfor ‘Car’, ‘Pedestrian’ and ‘Cyclist’. It further illustrates
the effectiveness of our method.

4.3. Ablation StudyIn this section, we conduct the comprehensive analysis
to evaluate different proposed methodologies on the ‘Car’,‘Pedestrian’ and ‘Cyclist’ classes.
Investigations on different fusion stages To figure out
the effectiveness of different methodologies, we demonstrate the ablation study in Table 3. When we only use
the point-wise semantic information fusion module (SF),the average performance of ‘Pedestrian’ and ‘Cyclist’ at
the three difficulty levels is increased by 2.94% and 2.42%,
respectively, compared to the baseline. This shows thatthe use of the SF can effectively supplement the semantic features of the point cloud, making the model more
conducive to detecting sparse object. The module of local region fusion (RF) has a more obvious contribution
to ‘Pedestrian’, with an average performance increase of
6.19%. When multi-objective category regularization (MR)module is adopted, the average performance of ‘Pedestrian’and ‘Cyclist’ is increased by 3.88% and 2.23% respectively,
further proving that this mutually influencing regularization
mechanism can effectively promote Camera recognition results to help the 3D detection network to be sensitive tosparse small objects. When we integrate all of these modules, we get the best overall performance at the moment.
However, it is worth mentioning that the category accuracyof ‘Pedestrian’ has a corresponding decline, but it is stillmuch higher than the baseline. At the same time, there isa huge improvement in ‘Cyclist’. It shows that the model
is more inclined to detect ‘Cyclist’ under the condition of
ensuring that the other two categories are at a higher level.

Table 1. The comparison of 3D mAP results of our proposed method with Point-RCNN.
Car
Pedestrian
CyclistEasy
Moderate
Hard
Easy
Moderate
Hard
Easy
Moderate

Hard

PointRCNN[17]

88.78

77.67

75.15

66.26

55.59

48.45

75.38

52.25

48.24

MSF-MC (Ours)

89.63

80.06

75.83

66.69

58.18

56.21

82.36

59.17

58.71

CyclistModerate

Hard

MethodMethodTable 2. The comparison of BEV results of our proposed method with Point-RCNN.
Car
Pedestrian
Easy
Moderate
Hard
Easy
Moderate
Hard
Easy

PointRCNN[17]

92.97

86.28

82.05

72.16

83.82

55.79

75.44

54.69

52.04

MSF-MC (Ours)

93.42

86.97

84.54

74.51

82.75

55.74

86.36

66.66

59.60

Ablation study of SF The SF module uses the idea ofmulti-scale to integrate the features of LiDAR point cloudand image. Semantic content information is more prevalent
at higher levels of the extraction network. Therefore, we begin to merge the SF module from the higher-level featuresand gradually transition to the lower-level feature fusion.
We conduct ablation study on the SF module, and the experimental results are reported in Table 4. We can see thatwhen the third and fourth layers are fused simultaneously,
the overall performance of the model is optimal. But when
added to the lower level (l = 2, 3), it actually degrades its
overall performance, which has a significant negative impact on the ‘Pedestrian’ category. This indicates that thehigher semantic features of the image are more suitable forfusion operation. It is worth noting that, in order to ensure the integrity of the model, the SF module in the above
experiment adopts the method of 4-layers simultaneous fusion.
Parameter sensitivity on λ and µ The multi-objective
category regularization mechanism is based on the multilabel prediction output through the regularization loss Lkl .
The label information output by the image classification
network and the category information from the LiDAR
point cloud are mutually influencing. In order to study
the influence of these loss components, we conduct a morecomprehensive ablation study.
In Figures 5 and 6, we conducted sensitivity analysis on
the two trade-off parameters λ and µ. µ controls the weightof multi-label feature fusion, while λ controls the degreeof focusing on hard-to-classify examples. Other parameters
are set to their default values.
Figure 6 shows the sensitivity of our proposed modelto the changes of λ parameters. We adjust λ by fixingµ = 0.05. We can see that when the parameter λ changes
from 0.001 to 0.01, the basic trend of detection performanceof the model for the three categories is constantly improving. When the value of λ is further increased, it can befound that the model performance is in a downhill stage.
It reveals that in the fusion mechanism of LiDAR-Camera,

Figure 5. Parameter sensitivity analysis on µ.

Figure 6. Parameter sensitivity analysis on λ.

the semantic features learned by multi-label learning havean obvious correlation to the detection performance of 3D
point cloud targets, which can also further explain why
many workers now adopt a very powerful prior model tofuse point cloud. Figure 5 shows the change range of theparameter µ when λ = 0.01 is fixed. We can find that

Ablation
SF

RF

MR

Easy
88.78

X

88.95
X

X

88.50

X

X

89.23

X

89.63

Table 3. The ablation study on SF, RF, MR on val set.
Car
Pedestrian
Moderate
Hard
Easy
Moderate
Hard
77.67
75.15
66.26
55.59
48.45
77.70
75.28
69.79
60.63
51.72
77.60
75.16
72.17
62.85
53.88
77.71
75.18
69.09
60.15
52.73
80.06
75.83
66.69
58.18
56.21

Table 4. Detection results of SF with deployment on different layers. Use l to represent the point-wise semantic information fusionof the point cloud feature S l of the l-th layer and the image semantic feature F l+1 of the l+1-th layer.
SF
mAP
l=1

X

l=2

l=3

l=4

Car

Pedestrian

Cyclist

X

79.65

58.95

65.63

X

X

80.72

58.41

66.23

X

X

X

81.18

54.91

65.02

X

X

X

81.84

60.36

66.75

the multi-objective category regularization mechanism controlled by µ is more sensitive to the category of ‘Cyclist’.

4.4. Compare with other fusion method
In Table 5, it can be found that our proposed method
has no obvious difference in the category of ‘Car’ compared with the latest methods,but it is worth noting that our
method is a multi-class end-to-end training method, mainly
focusing on the other two classes of small target sparse point
clouds. Our focus is on fusion methods, rather than optimizing the basic model of 3D detection.
Table 5. Performance comparison of 3D AP(Car) with previous
methods on KITTI validation sets

MethodMV3D [2]ContFuse [12]AVOD-FPN [9]F-PointNet [14]PI-RCNN [23]PointPainting [21]MSF-MC (Ours)

easy
71.29
82.54
84.41
83.76
88.27
88.38
88.14

3D AP(Car)moderate hard
62.68
56.56
66.22
64.04
74.44
68.65
70.92
63.85
78.53
77.75
77.74
76.76
77.48
75.92

5. Conclusion
In this paper, we study the limitations of current LiDARCamera fusion works and propose a multistage fusion for
multi-label 3D object detection task. We have implemented
end-to-end training by adopting a three-stage fusion method
without the need to introduce additional annotations. In the

CyclistEasy

Moderate

Hard

75.38

52.25

48.24

76.78

55.26

51.10

76.03

54.22

51.51

77.42

53.91

51.22

82.36

59.17

58.71

first stage, the multi-level gated adaptive fusion mechanism
is adopted to achieve point-wise fusion of the features ofpoint cloud and image, so as to ensure the spatial structure of point cloud and introduce effective image semantic
information. In the second stage, the camera features are
fused with point cloud ROI-pooling to enhance the integrityof proposals. We also introduce the third stage, using themulti-label classification task as the auxiliary regularizationof the object detection network to achieve the consistencyof category recognition. The experimental analysis showsthat our proposed method is very effective.

Acknowledgements
This work was supported in part by National Key Research and Development Program of China (grant number 2018YFF01013402), Shanghai Science and Technology Innovation Action Plan (20511100200), and Scienceand Technology Commission of Shanghai Municipality(14DZ2260800).

References[1] Liang-Chieh Chen, Yukun Zhu, George Papandreou, FlorianSchroff, and Hartwig Adam. Encoder-decoder with atrous
separable convolution for semantic image segmentation. In
Proceedings of the European Conference on Computer Vision (ECCV), pages 801–818, 2018.
[2] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia.
Multi-view 3D object detection network for autonomous
driving. In Proceedings of the IEEE conference on Computer
Vision and Pattern Recognition, pages 1907–1915, 2017.
[3] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel
Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231–1237,
2013.
[4] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term
memory. Neural computation, 9(8):1735–1780, 1997.
[5] Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3D
object detection. In European Conference on Computer Vision, pages 35–52. Springer, 2020.
[6] Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Emilie Wirbel, and Patrick Pérez. xmuda: Cross-modal unsupervised domain adaptation for 3d semantic segmentation.

In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 12605–12614, 2020.
[7] Jaekyum Kim, Junho Koh, Yecheol Kim, Jaehyung Choi,
Youngbae Hwang, and Jun Won Choi. Robust deep multimodal learning based on gated information fusion network.
In Asian Conference on Computer Vision, pages 90–106.
Springer, 2018.
[8] Diederik P Kingma and Jimmy Ba. Adam: A method forstochastic optimization. arXiv preprint arXiv:1412.6980,
2014.
[9] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh,
and Steven L Waslander. Joint 3d proposal generation andobject detection from view aggregation. In 2018 IEEE/RSJ
International Conference on Intelligent Robots and Systems(IROS), pages 1–8. IEEE, 2018.
[10] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou,
Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders
for object detection from point clouds. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pages 12697–12705, 2019.
[11] Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-sensor fusion for 3d object detection.
In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pages 7345–7353, 2019.
[12] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun.
Deep continuous fusion for multi-sensor 3d object detection.
In Proceedings of the European Conference on Computer Vision (ECCV), pages 641–656, 2018.
[13] Charles R Qi, Or Litany, Kaiming He, and Leonidas J
Guibas. Deep hough voting for 3d object detection in point
clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9277–9286, 2019.
[14] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J
Guibas. Frustum pointnets for 3d object detection from
RGB-D data. In Proceedings of the IEEE Conference on
Computer Vision and Pattern Recognition, pages 918–927,
2018.
[15] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a
metric space. arXiv preprint arXiv:1706.02413, 2017.
[16] Fitsum A Reda, Guilin Liu, Kevin J Shih, Robert Kirby, Jon
Barker, David Tarjan, Andrew Tao, and Bryan Catanzaro.
Sdc-net: Video prediction using spatially-displaced convolution. In Proceedings of the European Conference on Computer Vision (ECCV), pages 718–733, 2018.
[17] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point
cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 770–779, 2019.
[18] Shaoshuai Shi, Zhe Wang, Xiaogang Wang, and Hongsheng
Li. Part-aˆ 2 net: 3D part-aware and aggregation neural network for object detection from point cloud. arXiv preprint
arXiv:1907.03670, 2(3), 2019.
[19] Martin Simony, Stefan Milzy, Karl Amendey, and HorstMichael Gross. Complex-yolo: An Euler-region-proposal
for real-time 3d object detection on point clouds. In Proceedings of the European Conference on Computer Vision(ECCV) Workshops, pages 0–0, 2018.

[20] Vishwanath A Sindagi, Yin Zhou, and Oncel Tuzel. Mvxnet: Multimodal voxelnet for 3d object detection. In 2019 International Conference on Robotics and Automation (ICRA),
pages 7276–7282. IEEE, 2019.
[21] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4604–4612,
2020.
[22] Zhixin Wang and Kui Jia. Frustum convnet: Sliding frustums
to aggregate local point-wise features for amodal 3d object
detection. arXiv preprint arXiv:1903.01864, 2019.
[23] Liang Xie, Chao Xiang, Zhengxu Yu, Guodong Xu, Zheng
Yang, Deng Cai, and Xiaofei He. Pi-rcnn: An efficient multisensor 3d object detector with point-based attentive contconv fusion module. In Proceedings of the AAAI Conferenceon Artificial Intelligence, volume 34, pages 12460–12467,
2020.
[24] Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation.
In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pages 244–253, 2018.
[25] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10), 2018.
[26] Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Realtime 3D object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern
Recognition, pages 7652–7660, 2018.
[27] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd:
Point-based 3d single stage object detector. In Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11040–11048, 2020.
[28] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya
Jia. Std: Sparse-to-dense 3D object detector for point cloud.
In Proceedings of the IEEE/CVF International Conference
on Computer Vision, pages 1951–1960, 2019.
[29] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learningfor point cloud based 3d object detection. In CVPR, pages
4490–4499, 2018.
[30] Yi Zhu, Karan Sapra, Fitsum A Reda, Kevin J Shih, Shawn
Newsam, Andrew Tao, and Bryan Catanzaro. Improving semantic segmentation via video propagation and label relaxation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8856–8865,
2019.