# Multispectral Fusion Approach for Traffic Target Detection in Bad Weather
## 3 Methods
### Convolutional Base and RPN  
Different from the traditional faster RCNN network, our base convolutional network consists of two parts, the infrared image feature extract convolutional network (base network 1), and the visible image feature extract convolutional network (base network 2). In the base network, we use Inception ResNet [22] to generate the initial feature map. In detail, we use fine-tuned ResNet-34 and ResNet-50 as base network 1 and base network 2, respectively. In order to reduce the number of weights, we adjusted the dimensions of ResNet-34 and ResNet-50, as shown in Table 1. In addition, base network 2 is pretrained on the ImageNet dataset, and base network 1 uses Xavier for weight initialization. After the convolution processing of the two images, the corresponding feature maps of the two images are obtained, and then the two feature maps are connected along the channel axis and computed with a \(1\times 1\) convolution to obtain the fusion feature map, as shown in Figure 3a. The region proposal network is used to generate proposals from the input 512-dimensional fusion feature maps extracted by convolutional base, as shown in Figure 3b.  
Figure 2: The architecture of dual input faster RCNN (D-F Net). It is a middle level feature fusion model.  
A series of anchor boxes are output by a shared small network that slides over the fusion feature map of the RPN. The shared small network consists of an intermediate layer and a full connection layer. The convolution kernel size is \(3\times 3\) in the RPN. In traffic surveillance, the size and shape of the objects vary greatly. Thus, the aspect ratios are set as 1:1, 0.4:1 and 2.2:1, while the scales are set as \(128^{2}\), \(256^{2}\) and \(512^{2}\) pixels.  
The RPN generates high-quality region proposals, and the fast RCNN learns those features and performs classification. The loss function for an image is defined as:  
\[L(\{p_{i}\},\{t_{i}\})=\frac{1}{N_{cls}}\sum_{i}L_{cls}(p_{i},p_{i}^{*})+\lambda \frac{1}{N_{reg}}\sum_{i}p_{i}^{*}L_{reg}(t_{i},t_{i}^{*}) \tag{1}\]  
where \(i\) is the anchor's index in a mini batch, \(P_{i}^{*}\) is the ground truth label (when a proposal is an object, \(P_{i}^{*}=1\), otherwise \(P_{i}^{*}=0\)) and \(P_{i}\) is the anchor \(i\)'s predicted probability of being an object. \(t_{i}^{*}\) is the ground truth box of a positive anchor and \(t_{i}\) is a vector indicating four parameterized coordinates of the predicted bounding box. \(N_{cls}\) and \(N_{reg}\) are two normalization parameters, \(L_{cls}\) is classification loss, which is log over two classes (object versus not object). \(L_{reg}\) is regression loss, the term \(p_{i}^{*}L_{reg}\) means the regression loss that is active only for a positive anchor.  
For bounding box regression, the parameterization of the four coordinates can be written as:  
\[t_{x}=\frac{(x-x_{a})}{w_{a}}\quad t_{y}=\frac{(y-y_{a})}{h_{a}} \tag{2}\]  
\begin{table}
\begin{tabular}{c|c|c} \hline \hline
**Layer Name** & **Base Network 1** & **Base Network 2** \\ \hline Conv\_1 & \multicolumn{2}{c}{\(7\times 7\), 64, Stride 2} \\  & \multicolumn{2}{c}{\(3\times 3\) max pool, Stride 2} \\ Conv\_2 & \(\left|\begin{array}{c}3\times 3\\ 3\times 3\end{array}\right.\) 32 & \(\times 3\) & \(\left|\begin{array}{c}1\  
\[t_{w}=\log\!\left(\frac{w}{w_{a}}\right) t_{h}=\log\!\left(\frac{h}{h_{a}}\right) \tag{3}\] \[t_{x}^{\star}=\frac{(x^{\star}-x_{a})}{w_{a}} t_{y}^{\star}=\frac{(y^{\star}-y_{a})}{h_{a}}\] (4) \[t_{w}^{\star}=\log\!\left(\frac{w^{\star}}{w_{a}}\right) t_{h}^{\star}=\log\!\left(\frac{h^{\star}}{h_{a}}\right) \tag{5}\]  
where \(x\) and \(y\) represent the coordinates of the box center, \(w\) and \(h\) represent the weight and height of the bounding box, respectively, and \(x,x_{a}\) and \(x^{\star}\) are the predicated box, anchor box and ground truth box, respectively (likewise for \(y,w,h\)\(y,w,h\)). This can be thought of as bounding box regression from an anchor box to a nearby ground truth box.