# Multi-Layer Fusion 3D Object Detection via Lidar Point Cloud and Camera Image
## 3 The Proposed Approach
### Feature Extractor  
The multi-layer fusion 3D object detection network comprises two feature extraction networks, each dedicated to processing either the image or BEV input data. The structure of both feature extraction networks is the same. We designed the adaptive fusion module (AFM) and combined it with the attention mechanism to design the overall structure of the feature extractor. The network structure is shown in Figure 2.  
**Adaptive Fusion Module. ** The features of each layer of the decoder are fused in the spatial dimension. Different from the previous methods of integrating multi-layer features using elements-wise sum or concatenation, the key idea of this method is to learn the spatial weights of the feature map fusion on each scale adaptively. It consists of two steps: feature map uniformity and adaptive fusion. Feature map uniformity: As shown in Figure 2, feature maps with three different resolutions are represented as \(F_{1}\), \(F_{2}\) and \(F_{3}\). We need to pre-process three feature maps with different resolutions into feature maps with the same dimensions. Since the dimension of \(F_{3}\) was directly used in the final prediction, we adopted different upsampling strategies for the \(F_{1}\) and \(F_{2}\) feature maps, respectively, so that each dimension of \(F_{1}\) and \(F_{2}\) was consistent with that of \(F_{3}\). For \(F_{1}\) upsampling, we first use a \(1\times 1\) convolution layer to compress the number of channels of the feature to the number of channels of \(F_{3}\), and then raise the resolution to be consistent with \(F_{3}\) by nearest neighbor interpolation. Adaptive fusion: we define \(R_{13}\) and \(R_{23}\), respectively, to represent the feature maps obtained after the consistency of the feature maps for \(F_{1}\) and \(F_{2}\), and \(R_{33}\) and \(F_{3}\) are the same feature. \(\alpha(i,j)\), \(\beta(i,j)\) and \(\gamma(i,j)\) represent the weights of the \((i,j)\) vectors corresponding to \(R_{13}\), \(R_{23}\) and \(R_{33}\), respectively. Note that \(\alpha(i,j)\), \(\beta(i,j)\) and \(\gamma(i,j)\) are shared across all channels. Then, the fused feature can be expressed as:  
\[Y(i,j)\ =\ \alpha(i,j)\ \times\ R_{13}\ +\ \beta(i,j)\ \times\ R_{23}\ +\ \gamma(i,j)\times R_{33} \tag{1}\]  
The weight parameters \(\alpha_{ij}\), \(\beta_{ij}\) and \(\gamma_{ij}\) are obtained by a \(1\times 1\) convolution of the three feature maps after uniformization. And the parameters \(\alpha_{ij}\), \(\beta_{ij}\) and \(\gamma_{ij}\) are concatenated through the softmax function so that their range is in \([0,1]\) and the sum is \(1\). Therefore, \(\alpha(i,j)\), \(\beta(i,j)\) and \(\gamma(i,j)\) are obtained by the following calculation:  
\[\left\{\begin{aligned} \alpha(i,j)&=\ \frac{\varepsilon^{\alpha_{ij}}}{ \varepsilon^{\alpha_{ij}}+\varepsilon^{\beta_{ij}}+\varepsilon^{\gamma_{ij}}} \\ \beta(i,j)&=\ \frac{\varepsilon^{\alpha_{ij}}}{ \  
In this way, we obtain the \(Y(i,j)\) for the image and point cloud; they are combined with the RPN network by the fusion network and fulfill classification and regression. **Attention Mechanism. ** Through the attention mechanism, the network can learn to selectively emphasize the informative features using global information and suppress the less useful features. In this network, we adopt SENet [35], a type of channel attention mechanism designed to enhance the network's representation ability by enabling it to perform dynamic channel feature recalibration. We choose to add the SENet module to the decoder of the feature extraction network. The SENet module consists of two operations, squeeze and excitation, and is composed of a pooling layer, a convolutional layer and an activation layer. As shown in Figure 3, the original feature map \(X\) is first globally average-pooled to obtain \(S\), with the dimension changing from \(H\times W\times C\) to \(1\times 1\times C\), corresponding to the squeeze operation.