Study of 3D Object Detection with Normal-map on Point Clouds
Supervisor: Takayoshi Yamashita
TP18051 Jishu Miao
1.Introduction
Object detection is one of the most crucial tasks inautonomous driving. In this task, both accuracy andspeed are important. There are various methods havebeen studied to improve accuracy while accelerating thedetection speed. Although RGB images captured bycameras are used for object detection, point clouds captured by LiDAR are also used for this task. Because
it is insensitive to visible light and can capture objects
day or night. However, point clouds are sparse and different from images in that the order is irregular, leading
to a slow processing speed as 2D convolution cannot
be performed directly. Surface normals extracted fromthe points have a better ability to represent the shapefeatures of the object than 3D coordinates, which we believe will improve the performance of object detection.
In this study, we propose a novel point clouds-based 3D
object detection method for achieving higher-accuracy.
The proposed method employs You Only Look Once
v4 (YOLOv4) as a feature extractor and gives Normalmap as additional input. Our Normal-map is a threechannels Bird-eye view (BEV) map, retaining detailed
surface normal vectors. It makes the input information
have more enhanced spatial shape information and canbe associated with other hand-crafted features easily.

2.Bird-eye View Representation
The BEV map represents the point clouds as a 2D
pseudo-image from the bird-eye view as shown in Figure
1. This approach converts the unordered point clouds
into a sequence ordered image. Conventional methods are to generate three maps by a mapping function
PΩi→j , representing normalized point cloud density (R),maximum height (G), and maximum reflection intensity(B), aszr (Sj ) = min(1.0,! log(N + 1)/64)" N = |PΩi→j | ,zg (Sj ) = max PΩi→j · [0, 0, 1]T ,zb (Sj ) = max (I (PΩi→j )) .

(1)

Since 2D convolution can be applied to the BEV map,the detection task can be accelerated by using a fast
object detection network such as YOLO[1]. However,diﬀerent points may be arranged to a same pixel in theBEV map, which is less expressive than original data.
Besides, the object shape will be lost due to the 3D datais compressed to 2D.

Figure 2 : Overview of Proposed Method.
normals. The mapping function fPS shown in Eq. (2)is used for creating the Normal-map, which allows us touse the normal vector ⃗xɼ⃗y ɼ⃗z of the highest point inPΩi→j when mapping each point into 2D space.
#$PΩi→j = PΩi = [x, y, z]T |Sj = fPS (PΩi , g)
(2)

As shown in Eq. (3), the normal vectors of each point
are extracted as normal⃗x ɼnormaly⃗ ɼnormal⃗z from thePΩi→h for each axis. The normal vectors are represented
by a 3-channel 2D pseudo image, asnormal⃗x (Sj ) = ⃗x (PΩj→h ) ,normaly⃗ (Sj ) = ⃗y (PΩj→h ) ,normal⃗z (Sj ) = ⃗z (PΩj→h ) .

(3)

Since the search range of the normal estimation is wider
than the pixel representation range of the BEV map,
it can include a wider range of information. Moreover,since the normal-map calculated from the normal vectors is also a BEV map, it can be freely combined withother BEV maps to be used as the input data.

3.2 Input Details
Our network uses RGB-map and Normal-map as input. The RGB-map is similar to the BirdNet[2], andconsists of the height map, the density map, and theintensity map. The Normal-map is the normal vectors
in the x, y, and z axes. Thus, the input is a 6-channel
BEV map consisting of these maps concatenated in thechannel axis.

3.3 Object Detection Network
The network predicts the class and size of an object
with the Euler-Region Proposal Network (E-RPN) for3D object detection as shown in Figure 3. We employ
YOLOv4 as the basis network for 3D object detection.
E-RPN predicts the height, width, angle, objectness,and class probability of the bounding box coordinates.
In this study, the number of object classes is 6: Car,Van, Truck, Person, Cyclist, and Tram. The loss function is based on Mean-Square Error.

Figure 1 : Bird-eye View Representation.

3.Proposed Method
We propose a method for 3D object detection usingBEV map with additional normal information. Figure
2 shows an overview of the proposed method.

3.1 Normal Feature Extraction
The normal vector is estimated from the pre-processed
point clouds by Principal Component Analysis (PCA)
with the search radius of 30cm and the maximum searchnumber of 50 points. To make all normals point in thesame direction, the normals opposing the LiDAR arereversed by an orienting system. Normal-map is generated for enabling the 2D convolution of each point’sFigure 3 : Normal-YOLO Network Architecture.

4.Evaluation Experiments
In order to examine the eﬀectiveness of the proposed
method, some comparison experiments are conductedusing the KITTI dataset. We also evaluate the accuracyof the object angle detection by adding a function tocalculate the yaw angle.

Table 1 : Evaluation Results for Bird-eye View Performance on the KITTI Benchmark.
MethodBirdNetComplexer-YOLO
Ours

FPS
9.1
16.7
5.5

Easy
84.17
77.24
72.84

Car
Mod.
59.83
68.96
71.52

Hard
57.35
64.95
67.50

Easy
28.20
21.42
26.71

PedestrianMod.
Hard
23.06
21.65
18.26
17.06
21.19
20.17Easy
58.64
32.00
42.50

Cyclist
Mod.
41.56
25.40
36.06

Hard36.94
22.88
31.18

Average
45.93
38.68
43.30

Figure 5ɿ3D Object Detection Visualization in Camera View.

4.1 KITTI Benchmark Evaluation Results
Table 1 shows the evaluation results. Compared withthe conventional method, the proposed method achieves
the detection accuracy of 72.84% in Car under the Easy
mode, and the highest accuracy of 67.50% at the Hard
mode. It achieves almost the same accuracy as BirdNet[2],which is also a BEV-based method. Even for the sameclass objects with diﬀerent detection modes, our method
shows a more robust performance by adding normal information. In addition, we achieve a higher Average
Precision (AP) than Complexer-YOLO[3], which uses
the same YOLO-based network. Although the input
is BEV map, the accuracy for non-planar objects (e.g.
pedestrians and cyclists) is better for each mode.

4.2 Evaluation of Angle Prediction
Since the normal is the object shape information, weassume that object angle accuracy can be improved byadding the Normal-map. Table 2 shows the result ofcalculating the average included angle θk from the estimated object angles and ground truth for 6 classes.
!
#−1n1"
scoreclass (θk ) =arccos θk(4)nk=1

Normal-map improves the angle accuracy. In particular,
the accuracy of objects with large and flat shapes likecars is further improved.
Table 2 : Yaw Angle Prediction of 6 Classes.
Input Map
RGB
NormalNormal+RG
Normal+RGB

Car
10.18
8.76
9.27
10.34

Score of Angle Accuracy
Van
Truck
Person Cyclist
7.49
5.78
2.22
4.44
5.37
5.35
1.69
3.10
6.43
10.09
2.14
3.34
6.57
8.89
2.06
3.86

Tram
3.24
3.28
3.55
5.25

Figure 4 : Average Precision over distance.

4.4 Visualization Results
As shown in Figure 5, due to traﬃc lights, and traﬃcsigns are cylindrical and have a height similar to a human, the no-normal group could easily make false positive prediction. In contrast, proposed method reduce
the number of such mistakes with the addition of normal information. This indicates that the Normal-map isuseful in avoiding the false detection of objects similar
to human features.

5.Conclusion
We proposed a novel 3D object detection method with
Normal-map on point clouds. We have confirmed thatthe accuracy of BEV map-based object detection is further improved when we introduced normal information
to the BEV map. Since the Normal-map can keep highdetection accuracy without intensity information, it ispossible to use synthesized datasets by simulator withthe virtual environment. In the future, to improve theaccuracy of the method, we would like to explore deeplearning methods for object normal estimation.

References[1] A. Bochkovskiy, et al., “YOLOv4: Optimal Speed andAccuracy of Object Detection”, arXiv, 2020.
[2] J. Beltran, et al., “BirdNet: a 3D Object Detection
Framework from Lidar Information”, ITSC, 2018.

4.3 Evaluation by Distance
Since the density of the point clouds changes depending on the distance, we evaluate the detection accuracy
by distance. Figure 4 shows average precision over distance. From Figure 4, the detection accuracy of thegroup with added Normal-map does not decrease up to30m, and the decrease is smaller than no-normal groupeven at 40m or more.

[3] M. Simon, et al., “Complexer-YOLO: Real-time 3D Object Detection and Tracking on Semantic Point Clouds”,CVPR, 2019.

Research Achievements[1] J. Miao, et al., “3D Object Detection with Normal-map
on Point Clouds”, VISAPP, 2021.
(Other: 1 conference presentation)