# Multi-Layer Fusion 3D Object Detection via Lidar Point Cloud and Camera Image
## 3 The Proposed Approach
### D-Fusion  
We designed a fusion approach--dense fusion (D-fusion). Compared with the previous early fusion, late fusion and deep fusion, it can not only combine the features from multiple views but also effectively combine the semantic information of each layer in the network to carry out three-dimensional box regression. The network structure is shown in Figure 4d.  
Since features of different views often have different resolutions, we use RoI pooling for each view to obtain feature vectors of the same length. For the generated 3D suggestions, we are able to project them into any view in the 3D space. In our example, we project them onto two views, the BEV and the RGB Image.  
In order to combine information on different features, the previous work usually adopted early fusion, late fusion or deep fusion. Inspired by DenseNet, we adopted the dense fusion method to fuse every layer of the network densely. A comparison of the architecture of our D-fusion network and the early/late/deep fusion network is shown in Figure 4. For networks with P-layers, early fusion combines \(F_{BEV}\) and \(F_{RGB}\) from multiple views at the input stage:  
\[F_{P}=H_{P}(H_{P-1}(\cdots H_{1}(F_{BEV}\oplus F_{RGB}))) \tag{3}\]  
\(\left\{H_{p},\ p=1,\ldots,P\right\}\) is the feature transformation function. \(H_{p}\) refers to the feature map obtained at the P-layer. \(\oplus\) is a join operation (such as concatenation and sum).  
In contrast, late fusion uses separate subnetworks to independently learn feature transforms and combine their outputs in the prediction stage:  
\[F_{P}=(H_{P}(H_{P-1}(\cdots H_{1}(F_{BEV}))))\oplus(H_{P}(H_{P-1}(\cdots H_{1 }(F_{RGB})))) \tag{4}\]  
In early and late fusion, the operation \(\oplus\) is implemented using the concatenation method. Deep fusion enables more interaction between the middle features from different aspects:  
\[F_{0}=F_{BEV}\oplus F_{RGB} \tag{5}\] \[F_{P}=\left(H_{P}^{BEV}(F_{P-1})\oplus\left(H_{P}^{RGB}(F_{P-1}), \forall p=1,2,\cdots P\right.\]  
The operation \(\oplus\) in deep fusion uses the method of element-wise mean.  
Figure 4: Different types of fusion: (**a**) represents the early fusion method, (**b**) represents the late fusion method, (**c**) represents the deep fusion method, and (**d**) represents the dense fusion (D-fusion) method.  
To further improve the flow of information between layers, we propose a different fusion mode. It works by connecting directly from any layer to all subsequent layers:  
\[\begin{array}{c}F_{0}=F_{BEV}\oplus F_{RGB}\\ F_{1}=H_{1}(F_{BEV}\oplus F_{RGB})\\ F_{P}=a_{p-1}\big{(}a_{p-2}(\cdots a_{2}(a_{1}F_{1}\oplus)\cdots\oplus F_{p-2}) \oplus F_{P-1}\big{)},\\ \qquad\qquad\qquad\qquad\qquad\qquad\qquad\forall p=2,3,\cdots,p\end{array} \tag{6}\]  
The operation \(\oplus\) in the D-fusion uses the weighted summation method between concatenation and element-wise mean. Where \(a_{1},......a_{p-2},a_{p-1}\), respectively, represent the weights given after the feature fusion of each layer. The network adopts a three-layer network structure with the default settings \(a_{1}=1/2\), \(a_{2}=1/2\) and \(a_{3}=1\). We also use the dropout mechanism to mitigate the occurrence of overfitting and achieve a regularization effect to some extent. It also saves computing overhead.