Point-Voxel Feature Set Abstraction for 3D Object Detection
By: Nabil Madali
Tags: Computer Vision, Object Oriented, Machine Learning, Cnn, Point Cloud

3D object detection has been receiving increasing attention from both industry and academia thanks to its wide applications in various fields such as autonomous driving and robotics. LiDAR sensors are widely adopted in autonomous driving vehicles and robots for capturing 3D scene information as sparse and irregular point clouds, which provide vital cues for 3D scene perception and understanding.

Most existing 3D detection methods could be classified into two categories in terms of point cloud representations, i.e., the grid-based methods and the point-based methods.

In order to make comprehensive use of the respective advantages of the above two feature extraction operations,[1] are considering how to deeply integrate these two point cloud feature extraction algorithms into a network to improve the network’s structural diversity and characterization capabilities.

A novel high-performance 3D object detection framework named Point Voxel-RCNN (PV-RCNN) is proposed for accurate 3D object detection from point clouds. The proposed method deeply integrates 3D voxel convolutional neural network (CNN) and PointNet-based ensemble abstraction to learn more discriminative point cloud features. It takes advantage of the high-quality recommendations of 3D voxel CNN and the flexible receptive field of PointNet-based networks.

The overall architecture of our proposed PV-RCNN. The raw point clouds are first voxelized to feed into the 3D sparse convolution based encoder to learn multi-scale semantic features and generate 3D object proposals. Then the learned voxel-wise feature volumes at multiple neural layers are summarized into a small set of key points via the novel voxel set abstraction module. Finally the keypoint features are aggregated to the RoI-grid points to learn proposal specific features for fine-grained proposal refinement and confidence prediction.Source[1]

PV-RCNN consists of a 3D voxel CNN with sparse convolution as the backbone for efficient feature encoding and proposal generation. Given each 3D object proposal, to effectively pool its corresponding features from the scene, we propose two novel operations:

Voxel-to-key-point scene coding : which sums up all the voxels of the entire scene feature amount into A small number of feature key points, point-to-grid.

: which sums up all the voxels of the entire scene feature amount into A small number of feature key points, point-to-grid. RoI feature abstraction : which effectively summarize the scene key point features into the RoI grid for proposal confidence prediction and position adjustment.

First, the input point cloud P is divided into small voxels with a spatial resolution of L×W×H, where the features of non-empty voxels will be directly calculated as the average of the point features of all internal points. Commonly used features are 3D coordinates and reflection intensity. The network uses a series of 3×3×3 3D sparse convolutions to gradually convert the point cloud into feature quantities with 1×, 2×, 4×, and 8× down-sampling sizes.

By converting the encoded 8×down-sampled 3D feature into a 2D bird’s-eye view feature, following the anchor-based method will generate high-quality 3D suggestions.

Specifically, they stack 3D feature quantities along the Z axis (height becomes channel) to obtain a L/8×W/8 bird’s-eye feature map. Each category has 2×L/8×W/8 3D anchor frames (each cell has two anchors for each category), these frames use the average 3D object size of this type, and each pixel of the bird’s-eye feature map evaluate two anchors with 0° and 90° orientations.

Compared with the PointNet-based method, the 3D voxel CNN backbone using the anchor-based scheme achieves higher recall performance.

The author first uses voxel (grid) method for feature extraction of the entire scene, and at the same time adopts a branch to use point FPS for the scene Sampling, and then retrieve the multi-scale voxel features. In fact, it only uses the characteristics of voxel, but it is expressed on the key-point.

First, they use FurthestPoint-Sampling (FPS) algorithm to sample a small number of n keypoints K = {p1, · · · , pn} from the point clouds P.

Denote

as the set of voxel-wise feature vectors in the k-th level of 3D voxel CNN,and

as their 3D coordinates calculated by the voxel indices and actual voxel sizes of the k-th level, where N_k is the number of non-empty voxels in the k-th level. For each keypoint pi , they first identify its neighboring non-empty voxels at the k-th level within a radius r_k to retrieve the set of voxel-wise feature vectors as

The semantic features of the voxels are connected with the local relative coordinates.Then, pass PointNet transforms the voxel features of the adjacent voxel set of the key point to generate the feature of the key point:

where M(·) denotes randomly sampling at most T_k voxels from the neighboring set S ^{(lk)}_i for saving computations, G(·) denotes a multi-layer perceptron network to encode the voxel-wise features and relative locations and max(·) is max-pooling operation.

The above voxel set abstraction is performed at different levels of the 3D voxel CNN, and the aggregated features from different levels can be concatenated to generate the multi-scale semantic feature for the key point pi:

They extend the VSA module by further enriching the keypoint features from the raw point clouds P and the 8× downsampled 2D bird-view feature maps.

For the bird view feature maps, they project the keypoint pi to the 2D bird-view coordinate system, and utilize bilinear interpolation to obtain the features f^{(bev)}_i from the bird-view feature:

Some of the key points are foreground and some are background points. The weight of the foreground should be large and the background points should be small. Therefore, weight prediction is performed on the features of each key point,

where A(·) is a three-layer MLP network with a sigmoid function to predict foreground confidence between [0, 1].

Given each 3D proposal (RoI) generated by the 3D voxel CNN, the features of each RoI need to be aggregated from the keypoint features

They uniformly sample 6 × 6 × 6 grid points within each 3D proposal, which are denoted as G = {g1, · · · , g216}.

They identify the neighboring keypoints of grid point gi within a radius r˜ as

Then a PointNetblock is adopted to aggregate the neighboring keypoint feature set Ψ˜ to generate the feature for grid point gi as

After obtaining each grid’s aggregated features from its surrounding keypoints, all RoI-grid features of the same RoI can be vectorized and transformed by a two-layer MLP with 256 feature dimensions to represent the overall proposal.

Given the RoI features of each proposal, the proposal optimization network will learn to predict the size and position (ie center, size and direction) residuals of the 3D proposal. The refinement network uses a 2-layer MLP, each with two branches, which are used for confidence prediction and proposal refinement.

Conclusion

Compared with conventional pooling operations, the RoI-grid feature points encode much richer context information for accurately estimating object confidences and locations. Extensive experiments on both the KITTI dataset and the Waymo Open dataset show that our proposed PV-RCNN surpasses state-of-the-art 3D detection methods with remarkable margins by using only point clouds.

References

1.Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, Hongsheng Li.PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection,arXiv:1912.13192.