Sparsely Embedded Convolutional Detection
By: Nabil Madali
Tags: Computer Vision, Object Detection, Point Cloud, Machine Learning

Object detection based on LiDAR or RGB-D has been widely used, such as autonomous driving and machine vision. The 3D convolutional network based on voxel division has been around for a period of time, and it has enhanced the more complete retention of information when processing point cloud LiDAR data. However, some problems still exist, including slow inference speed and low performance of orientation estimation. Therefore, [1] studies an improved sparse convolution method for this type of network, which significantly improves the speed of training and inference.[1] also introduced a new form of angle loss regression to improve the performance of orientation estimation, and a new data enhancement method that can improve the convergence speed and performance. The proposed network has the most advanced performance on the KITTI 3D object detection benchmark test, while maintaining a faster inference speed.

Main contribution of this article :

The application of sparse convolution in LiDAR-based target detection greatly improves the speed of training and inference.

A method to improve sparse convolution is proposed to make it run faster.

A new angle loss regression method is proposed, which has better performance than other methods.

New data enhancement methods are introduced for learning problems based only on LiDAR, which greatly improves the convergence speed and performance.

The architecture of the proposed SECOND detector is depicted in Figure below,

The structure of proposed SECOND detector. The detector takes a raw point cloud as input, converts it to voxel features and coordinates, and applies two VFE (voxel feature encoding) layers and a linear layer. Then, a sparse CNN is applied. Finally, an RPN generates the detection.Source[1]

The proposed SECOND detector, consists of three components:

A voxelwise feature extractor

A sparse convolutional middle layer

And an RPN.

Voxel feature extraction: [1] performs a similar operation to VoxelNet , that is, the original point cloud is first meshed with voxels, and then the VFE voxel feature extraction network is used to extract the features of each voxel.

The structure of the VFE layer is shown in Figure above.

Voxel feature encoding layer.

First preallocate buffers based on the specified limit on the number of voxels; then, iterate over the point cloud and assign the points to their associated voxels, and save the voxel coordinates and the number of points per voxel. The iterative process will stop once the number of voxels reaches the specified limit.

A VFE layer takes all points in the same voxel as input and uses a fully connected network (FCN) consisting of a linear layer, a batch normalization (BatchNorm) layer and a rectified linear unit (ReLU) layer to extract pointwise features. Then, it uses elementwise max pooling to obtain the locally aggregated features for each voxel. Finally, it tiles the obtained features and concatenates these tiled features and the pointwise features together.

Sparse Convolutional Middle Extractor:In the previous step, after dividing voxels from the point cloud collected by lidar, about 5k ~ 8k voxels and a sparsity of about 0.005 will be generated. Directly using 3D convolution will consume huge computing time and memory, and this is can be avoided by using sparse convolution. [1] uses a convolution structure called submanifold convolution to limit the sparsity of the output through the sparsity of the input data, thereby greatly reducing the amount of calculation for subsequent convolution operations.

The structure of proposed sparse middle feature extractor. The yellow boxes represent sparse convolution, the white boxes represent submanifold convolution, and the red box represents the sparse-to-dense layer. The upper part of the figure shows the spatial dimensions of the sparse data.Source[1]

As shown in Figure above, the sparse convolution feature extraction network used in this article includes a sparse convolution layer (indicated by yellow), a submanifold convolution (white) and a sparse to dense conversion layer (red).

Regional Proposal Network (RPN):[1] uses an RPN similar to the SSD architecture, and the input is a feature map. RPN consists of three stages, each stage contains a down-sampled convolutional layer and a series of convolutional layers; then the feature map of each stage is up-sampled and concatenated to form a feature map, and finally three 1x1 convolution are used for category prediction, position regression and angle regression.

In addition, [1] proposed a data enhancement method to accelerate the convergence of training. The ground truths of the training set are sampled to form a database. During the training process, several samples in the database are randomly selected and introduced into the current point cloud. In order to avoid conflicts, collision detection is also required. The author also introduced random angle noise to the sample, and finally rotated and zoomed the global point cloud in a small range.

The proposed network produces state-of-the-art results on the KITTI 3D object detection benchmarks while maintaining a fast inference speed.

References

1.Yan Yan , Yuxing Mao and Bo Li ,SECOND: Sparsely Embedded Convolutional Detection.