# Multispectral Fusion Approach for Traffic Target Detection in Bad Weather
## 4 Procedure and Results  
### Dataset and Data Augmentation  
The dataset is traffic surveillance in [7], collected by [PERSON] et al. This dataset is focused on collecting traffic surveillance video in rainfall and snowfall, with 22 videos from seven different traffic intersections, and each monitor video is five minutes long. The illumination of the scenes varies from broad daylight to dusk and night. The features of scenes are obscured by the glare of the headlights of cars and streetlamps, reflected in puddles and blurred by raindrops on the camera lens.  
This dataset uses optical and thermal infrared cameras to capture video sequences of road users. We have selected 600 frames randomly from each five-minute sequence and any road user in these frames is annotated on an instance level with a corresponding category label. In total, 15,600 frames are annotated, containing 33,297 objects in six categories (car, truck, bike, motorbike, bus, person). We divide 15,600 pairs of infrared and visible images into a training set and a testing set according to the ratio of 8:2. Moreover, all the images in this dataset are normalized to the size of \(640\times 480\) pixels and input to the convolutional neural networks. Some examples of the dataset are shown in Figure 4.  
There are 15,600 pairs of photos in the dataset, but some categories of images are insufficient, such as pedestrian, motorbike and bicycle, which account for a small proportion. In order to reduce the risk of overfitting, we use data augmentation to enrich the original dataset. The spatial transformation of coordinates is used to enlarge our dataset, and the transform of coordinates can be expressed by the following formula:  
\[\left[\begin{array}{c}x^{\prime}\\ y^{\prime}\end{array}\right]=\mathbf{T}\left[\begin{array}{c}x\\ y\end{array}\right]=\left[\begin{array}{cc}t_{11}&t_{12}\\ t_{21}&t_{22}\end{array}\right]\left[\begin{array}{c}x\\ y\end{array}\right] \tag{8}\]  
Figure 4: Some examples of images of traffic surveillance, the visible image is on the left and the infrared image is on the right. In (**a-c**), there are scenes ranging from sunny to rainy in the daytime; in (**d–f**), there are bad lighting conditions and bad visibility of the scenes; (**c**) snow; (**d**) rain and reflections; (**f**) raindrops on the camera lens.  
where \((x,y)\) are pixel coordinates in the original image and \((x^{\prime}x^{\prime},y^{\prime}y^{\prime})\) are the corresponding pixel coordinates of the transformed images. The most common type of coordinate transformation is affine transformation, the general form is as follows:  
\[\left[\begin{array}{c}x^{\prime}\\ y^{\prime}\\ 1\end{array}\right]=\mathbf{A}\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=\left[\begin{array}{ccc}a_{11}&a_{12}&a_{13}\\ a_{21}&a_{22}&a_{23}\\ 0&0&1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right] \tag{9}\]  
This transformation can scale, rotate, translate or shear an image by changing the value of the elements of matrix A. We apply multiple transformations to the image by multiplying different matrices A. The matrix A that we use for affine transformation is shown in Appendix A.