# VirtualFilter: A High-Performance Multimodal 3D Object Detection Method with Semantic Filtering
## 1 Introduction  
Three-dimensional object detection is one of the core technologies in the field of autonomous driving, which provides basic sensory information for subsequent decision-making and control tasks [1; 2; 3]. Recent 3D detection approaches usually use point cloud data as the primary input data due to the inherent three-dimensional superiority of point cloud data. For instance, PointPillars [4] divides the 3D space into vertical columns (pillars) and encodes point cloud features within each pillar using a lightweight PointNet module. PV-RCNN [5] uses a point convolution module to extract local features for each point, while leveraging voxel convolution to capture global contextual information. However, the low scanning resolution of LIDAR for long-distance objects leads to limitations of this kind of approaches. In general, better detection results can be achieved by using 2D camera data as supplementary input, i.e., the multimodal 3D object detection approach [6; 7]. Most of the early work extended the image data in a deterministic manner to the point cloud data in order to fuse 2D features. MV3D [8] converts the LiDAR point cloud into two 2D projections, a bird's eye view (BEV) and a front view, and aligns the camera image with these projections to create a multi-view representation of the scene. PointPainting [9] projects the LiDAR point cloud onto the camera image plane and then \"maps\" each LiDAR point with the corresponding pixel feature in the image. Although such methods utilize the 2D information from the camera, they do not fundamentally solve the problem of lowaccuracy of long-distance object detection. Therefore, multimodal detection methods utilizing 2D images to generate virtual point clouds have attracted much attention. For instance, SFD [6] utilizes a deep complementary network to generate virtual point cloud data, and an RoI fusion strategy called 3D-GAF is used to merge the original point cloud and the generated virtual point clouds. VirConvNet [7] also uses a deep complementary network to generate the virtual point cloud and, based on this, adopts the VirConv operator which can reduce the density of virtual points. Despite the network structure reducing the density of the virtual point cloud, a considerable amount of unnecessary or redundant information still persists within the point cloud. The purpose of using a virtual point cloud is to compensate for the undetectability of using LiDAR-based data on long-range objects; then, in reality, we only need to supplement the point cloud that makes up the objects. That is to say, retaining those virtual point cloud data describing the background is not necessary. Using too much virtual point cloud data that describes the background not only increases the amount of computation but may also cause a decrease in detection accuracy. Another problem is that many approaches use a simplistic RoI feature fusion technique for the original and virtual point clouds, often just concatenating the features without further refinement. Although the virtual point cloud and the original point cloud have the same dimensions, they are fundamentally different in terms of how they are generated. Simple concatenation fusion strategies do not make good use of the RoI feature of the original point cloud and the virtual point cloud. To tackle these two issues introduced above, this study proposed a novel multimodal 3D object detection method named VirtualFilter, which utilizes the semantic filtering strategy and the better RoI feature fusion strategy. We observed that virtual point clouds generated using deep completion contain too much background information. But, in reality, only virtual point clouds containing the object semantics are useful for subsequent detection. Based on this observation, we used a semantic segmentation model to obtain the semantic information of the 2D image and project the virtual point cloud generated using depth completion onto the 2D image for filtering.