# VirtualFilter: A High-Performance Multimodal 3D Object Detection Method with Semantic Filtering
## 3 Method
### Semantic Filtering of Virtual Points  
The 2D image data have the advantage of rich semantic and color information. Previous methods either did not fully utilize the semantic information of 2D images or incorporated semantic information into the feature extraction process, increasing the computational complexity of the model. The use of filtering virtual points can effectively avoid the above two problems, utilizing semantic information without adding more burden to the subsequent feature processing flow. We represent the virtual point set generated by the PENet model as \(\mathcal{V}\), and the 2D semantic features generated by the semantic segmentation model as \(\mathcal{S}\). Given that the set of categories as a result of semantic segmentation is denoted as \(C\), we can build a projection \(T_{\textit{rectlassify}}\) according to the semantics of the categories and reclassify the set of categories into \"background\" and \"non-background\" categories. So, virtual points belonging to the \"background\" category can be deleted during the filtering process. Given the virtual point cloud data \(\mathcal{V}\) and semantically labeled data \(\mathcal{S}\) in a scene, our semantic filtering approach can be divided into three steps. In the first step, the semantic labeling data \(\mathcal{S}\) of an image are converted into a background discriminant matrix \(B\) containing only 0 and 1 by the projection \(T_{\textit{rectlassify}}\), where 0 represents \"background\" and 1 represents \"non-background\". In the second step, the virtual point cloud is projected onto the background discriminant matrix using the projection formula. The projection formula is shown as follows:  
\[[u,v,z^{\prime}]^{T}=P_{2}\odot R_{\textit{rect}}\odot T_{\textit{prod}oTocam }\odot[x,y,z,1]^{T} \tag{1}\]  
where u, v represent the 2D coordinates of the virtual point projected onto the image, and z' is the camera depth. \(P_{2}\), \(R_{\textit{rect}}\), \(T_{\textit{rectlocTocam}}\) are the camera projection matrix, the camera rotation matrix, and the laser-to-camera RT matrix, respectively. For convenience, we rewrite Equation (1) as the function _projection_ as follows:  
\[\textit{projection}(v_{i})=P_{2}\odot R_{\textit{rect}}\odot T_{\textit{prod}oTocam }\odot v_{i} \tag{2}\]  
In the last step, the projected coordinates of the virtual point \(v_{i}\) are used to view the label value of the corresponding background discriminant matrix \(B_{i}\), and if the value is 0, i.e., the virtual point is projected to the background region, the point is removed from the virtual point set. Moreover, it can be represented as follows:  
\[\alpha_{v_{i}}=\beta^{T}(W(\textit{projection}(v_{i}))\wedge B_{i})\gamma \tag{3}\]  
where \(W(x,y)\) denotes a matrix with the same dimensions as \(B_{i}\), except that the coordinate \((x,y)\) position is 1, and all other positions are 0. \(\beta,\gamma\) denotes the column vector of dimension \(W\), \(H\) with all elements in 1, respectively. Algorithm 1 is specifically represented as follows:  
Figure 2 shows some comparative image results during the semantic filtering process. (a) shows the original image sample in the KITTI dataset, and (b) shows the generation of the corresponding 2D image from virtual points according to the projection Formula (1), where the camera depth is used to distinguish the color of the virtual points. (c) shows the semantic category map generated by the semantic segmentation model, with different colors representing different categories, and (d) shows the image of semantically filtered virtual points projected on 2D. Figure 2b,d visually demonstrate the comparison before and after semantic filtering. As can be seen in Figure 2, the number of semantically filtered virtual points is significantly reduced, leaving only virtual point data that contain valid semantics. After semantic filtering, virtual points are significantly reduced, which not only improves computational efficiency but also provides more accurate 3D data for subsequent feature extraction.