# MPVF: Multi-Modal 3D Object Detection Algorithm with Pointwise and Voxelwise Fusion
## 3 Methodology
### PVWF Module  
To enhance the interaction between multi-modal local and global features, we propose a pointwise and voxelwise fusion module, PVWF, as illustrated in Figure 3. This module comprises the pointwise fusion (PWF) module and the voxelwise fusion (VWF) module, with Stacked VFE as its core component. Initially, the LiDAR point cloud data are projected onto the RGB image, and the pointwise and voxelwise image features are obtained through feature indexing and RoI align operations. The point features are derived from the original point cloud after voxelization. The VFE network serves as a feature learning module designed to encode the raw point cloud within each voxel, extracting geometric information and local features in 3D space. For a given point cloud, let us assume it covers the 3D space along the Z, Y, and X axes, with ranges \(D\), \(H\), and \(W\), respectively. To effectively represent this space, we partition the 3D space into evenly spaced voxel grids, defining the size of each voxel as \(v_{D}\times v_{H}\times v_{W}\). The resulting 3D voxel grid sizes are \(D^{\prime}=D/v_{D}\), \(H^{\prime}=H/v_{H}\), and \(W^{\prime}=W/v_{W}\). Subsequently, points in the point cloud are grouped into their corresponding voxels. From voxels containing more than \(T\) points, a fixed number \(T\) of points are randomly sampled to ensure computational stability and reduce redundant data. Equation (1) describes a non-empty  
Figure 3: Specific Structure of the PVWF Module. voxel \(V\) containing \(t\ \leq\ T\) LiDAR points, where each point includes its XYZ coordinates and reflectance value [33]. \[V=\left\{p_{i}=\left[x_{i},y_{i},z_{i},r_{i}\right]^{T}\in R^{4}\right\}_{i=1..t} \tag{1}\]  
To further encode the local features of these points, we first calculate the local mean of all points as the centroid, denoted as \((v_{x},v_{y},v_{z})\). Next, Equation (2) describes the input feature set formed by expanding each point's relative offset to the centroid [33]. This expanded point feature, combined with the original coordinates and reflectance values of the points, constitutes the voxelized features used as input to the PVWF module. \[V_{in}=\left\{p_{i}^{\prime}=\left[x_{i},y_{i},z_{i},r_{i},x_{i}-v_{x},y_{i}-v_ {y},z_{i}-v_{z}\right]^{T}\in R^{7}\right\}_{i=1..t} \tag{2}\]  
In the main body of the VFE, each point is transformed through a VFE layer that includes fully connected networks (FCN), entering a high-dimensional feature space. During this process, the VFE aggregates the features of each point within the voxel, better encoding the local geometric shape inside it. The pointwise features after FCN processing are denoted as \(f_{i}\) (where \(f_{i}\in R^{m}\)). For a non-empty voxel V containing \(t\ \leq\ T\) points, \(f_{i}\) is aggregated through element-wise max pooling to form the voxelwise local aggregated feature \(f_{agg}\) (where \(f_{agg}\in R^{m}\)). Finally, these pooled local aggregated features \(f_{agg}\) are extended and concatenated with the original features \(f_{i}\) of each point to form a new pointwise concatenated feature \(f_{concat}\), which represents the spatial relationships and local geometric information of all points within the voxel, constituting the output feature set. By stacking the VFE layers, the input point cloud is progressively transformed into high-dimensional feature representations. We denote the \(i_{th}\) VFE layer as \(VFE_{i}(n_{in},n_{out})\), converting the input feature dimension \(n_{in}\) to the output feature dimension \(n_{out}\). The linear layer learns a matrix of size \(n_{in}\times(n_{out}/2)\), while the pointwise concatenation operation yields a final output of dimension \(n_{out}\). For the PWF module, the configuration of Stacked VFE is \(VFE_{1}\) (\(7+16\), \(32\)) and \(VFE_{2}\) (\(32\), \(128\)). The input to the first VFE layer is the concatenated result of the 7-dimensional point features and the 16-dimensional pointwise image features.