# PIFNet: 3D Object Detection Using Joint Image and Point Cloud Features for Autonomous Driving
## 3 Proposed Point-Image Fusion Network (PIFNet)
### Training Loss  
A multi-task loss function is used to train our neural network, PIFNet. Following EPNet [27], the total loss function is defined as the summation of training objectives for the two-stream Region Proposal Network (RPN) and the refinement network. The loss function can be formulated as follows,  
\[L_{total}=L_{rpn}r+L_{rcnn} \tag{1}\]  
where \(L_{rpn}\) is the loss of RPN network, and \(L_{rcnn}\) is the loss of refinement network.  
Both of them adopt a similar optimization goal, to minimize classification loss, regression loss and Consistency Enforcing (CE) loss, i.e.,  
\[L_{rpn}=L_{cls}+L_{rcg}+\lambda L_{cc} \tag{2}\]  
where \(L_{cls}\) is the classification loss, \(L_{rcg}\) is the regression loss, and \(L_{cc}\) is the CE loss.  
The focal loss is used as the classification loss to balance the positive and negative samples. We use the same parameters, \(\alpha\) = 0.25 and \(\gamma\) = 2.0 as in [27] in the following equation. For a bounding box, the network needs to regress its center point (x, y, z), size (l, h, w), and orientation \(\theta\). We directly calculate the Y-axis offset to the ground truth and the size of the bounding box (h, w, l) with a smooth L1 loss [33]. As for the X-axis, the Z-axis,  
Figure 3: The fusion module architecture.  
and the orientation \(\theta\), we adopt the bin-based regression loss. The loss functions can be written as follows,  
\[L_{cls}=-\alpha(1-c_{t})^{\gamma}logc_{t} \tag{3}\]  
\[L_{total}=L_{rpn}+L_{rcmt}\ L_{rgs}=\sum_{u\in\times z,\theta}E\Big{(}b_{u}, \widehat{b_{u}}\Big{)}+\sum_{u\in\times x,y,z,h,w,l,\theta}S(r_{u},\widehat{r_ {u}}) \tag{4}\]  
where \(E\) and \(S\) denote the cross-entropy loss and the smooth L1 loss, respectively. The parameter \(c_{t}\) is the probability of the point, in consideration, belonging to the ground truth category. The parameters \(\widehat{b_{u}}\) and \(\widehat{r_{u}}\) represent the ground truth of the bins and the residual offsets.  
Semi-supervised learning requires full mining of the value of unlabeled data. As a kind of semi-supervised learning, the core idea of the Consistency-based method is to reduce the consistency loss. For unlabeled data, the model should make consistent predictions.  
We use a Consistency Enforcing (CE) loss [27] to ensure the consistency between the localization confidence and the classification confidence, so that boxes with high localization confidence possess high classification confidence. The consistency enforcing loss can be written as follows,  
\[L_{ce}=-\log\bigg{(}C*\frac{Area(D\cap G)}{Area(D\cup G)}\bigg{)} \tag{5}\]  
where \(D\) and \(G\) represent the predicted bounding box and the ground truth bounding box. \(C\) denotes the classification confidence for the predicted bounding box. To optimize this loss function, the classification confidence and location confidence (i.e., IoU) should be maximized. Therefore, boxes with large overlaps have high classification possibilities and are kept in the Non-Maximum Suppression (NMS) procedure. The CE loss aims at ensuring the consistency between the localization and the classification to assist the NMS procedure, keeping more accurate bounding boxes.