# PRNet: 3D Object Detection Network-Based on Point-Region Fusion
## 3 PRNet Framework
### PRF Module  
To enhance the integration of point cloud and image data, the study introduces the point-region fusion (PRF) module, a critical component within our network. This module encompasses four primary functions: projection, point searching, sampling, and fusion, as shown in Figure 2. The functionalities are delineated as follows:  
The first step is projection. In this process, we project a point \(p\) from the point cloud onto the plane of the camera image. Each point \(p\) is expressed in homogeneous coordinates denoted by \(p=\left(x,y,z,1\right)^{T}\). By applying the mapping matrix \(M\), we can determine the corresponding point \(p^{\prime}\) on the image plane:  
\[p^{\prime}=M\times p \tag{1}\]  
Among them, \(M\) represents a \(3\times 4\) mapping matrix. The point \(p\) in the point cloud is represented as a four-dimensional vector within a homogeneous coordinate system. Meanwhile, the point \(p^{\prime}\) on the image plane is expressed as a three-dimensional vector in homogeneous coordinates. The second step is point searching. Following the feature extraction through the set abstraction (SA) module in the point cloud, the study acquires a set of points with feature representations, denoted within \(p^{\prime}=\left\{p_{1},p_{2},\ldots,p_{n}\right\}\). For each point \(p_{i}\), the study employs the K-Nearest Neighbors (KNN) algorithm to identify its \(K\) closest neighboring points, which are represented as \(N_{k}(p_{i})\):  
\[N_{k}(P_{i})=\text{KNN}(p_{i,k}) \tag{2}\]  
Among them, \(N_{k}(p_{i})\) represents \(K\) closest neighboring points of the point \(p_{i}\). KNN diagram is shown in Figure 3.  
The third step is sampling. In this phase, the image features that have been mapped through the sampling process are employed. For each point \(p_{i}\) within the point cloud and its associated set of neighboring points \(N_{k}(p_{i})\), the study defines a sampling function \(S\) to extract the corresponding features from the image:  
\[R_{i}=S(Image,M,N_{k}(p_{i})) \tag{3}\]  
Among them, \(Image\) denotes the input image data. \(R_{i}\) refers to the region of interest or feature area associated with point \(p_{i}\) and its neighboring point \(N_{k}(p_{i})\), which is derived through the process of sampling on the image. Figure 2: PRF module structure. The fourth step is fusion. In the final step, the image features are fused with the point cloud features. Employing the convolution operation \(C\) and the pooling operation \(P\) to extract features \(F_{i}\) from the source \(R_{i}\). Ultimately, these extracted features are integrated with the point cloud features \(F_{i}\), which are acquired from the SA (Self-Attention) module. This integration process incorporates an attention mechanism \(A\), enabling a weighted fusion of these features:  
\[F_{i^{\prime}}=P(A(C(R_{i}))) \tag{4}\]  
\[F_{final}=Concat(F_{i^{\prime}},L_{i}) \tag{5}\]  
Among them, the feature \(F_{i^{\prime}}\) is derived following the convolution operation \(C\) and the pooling operation \(P\), subsequently undergoing a weighting process by the attention mechanism \(A\). \(L_{i}\) represents the intrinsic point cloud features of the point \(p_{i}\). The ultimate fusion feature \(F_{final}\) is prepared for application in the ensuing tasks of object detection and segmentation. The structure of this step is shown in Figure 4.  
Figure 4: Structure of the fourth step: fusion. \[p^{\prime}=M\times p \tag{1}\]0 Diagram of KNN. The PRF module's strength is its capacity to effectively integrate data from point clouds and images, capitalizing on their distinct characteristics across various modalities. This integration is enhanced by the incorporation of the k-nearest neighbors (KNN) algorithm and an attention mechanism, enabling the module to more astutely discern the contextual nuances surrounding the object. This enhances the accuracy and robustness of object detection.