MWNet: object detection network applicable for different weather conditions

IET Intelligent Transport SystemsVolume 13, Issue 9 p. 1394-1400 Research ArticleFree Access MWNet: object detection network applicable for different weather conditions Liu Pei, Liu Pei Electronic and Information Engineering, Beijing Jiaotong University, No. 3 Shangyuancun Haidian District, Beijing, People's Republic of ChinaSearch for more papers by this authorXue Yuan, Corresponding Author Xue Yuan xyuan@bjtu.edu.cn Electronic and Information Engineering, Beijing Jiaotong University, No. 3 Shangyuancun Haidian District, Beijing, People's Republic of ChinaSearch for more papers by this authorXueRui Dai, XueRui Dai Electronic and Information Engineering, Beijing Jiaotong University, No. 3 Shangyuancun Haidian District, Beijing, People's Republic of ChinaSearch for more papers by this author Liu Pei, Liu Pei Electronic and Information Engineering, Beijing Jiaotong University, No. 3 Shangyuancun Haidian District, Beijing, People's Republic of ChinaSearch for more papers by this authorXue Yuan, Corresponding Author Xue Yuan xyuan@bjtu.edu.cn Electronic and Information Engineering, Beijing Jiaotong University, No. 3 Shangyuancun Haidian District, Beijing, People's Republic of ChinaSearch for more papers by this authorXueRui Dai, XueRui Dai Electronic and Information Engineering, Beijing Jiaotong University, No. 3 Shangyuancun Haidian District, Beijing, People's Republic of ChinaSearch for more papers by this author First published: 10 June 2019 https://doi.org/10.1049/iet-its.2019.0023Citations: 1AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract The existing vehicle environment perception systems remain limited with regard to the ability to detect objects under complex weather conditions. This study proposes a novel network architecture named multi-weather network (MWNet), which can improve the performance of the on-board object detection system under extreme weather conditions. It consists of an encoder and a decoder. The encoder is comprised of shared convolutional layers used to extract features, while the decoder consists of three subnets, namely weather classification subnet, bad weather detection subnet, and fair weather detection subnet. Moreover, the results are satisfactory even for images photographed under different weather and illumination conditions. 1 Introduction Developing autonomous driving systems that can assist drivers in making decisions is one of the most prospective and challenging research areas [1]. Its ultimate goal is to develop fully autonomous cars and intelligent transportation systems. However, previous research efforts mainly focused on LIDAR systems, such as Velodyne [2] and hand-annotated maps of the environment. With the development of hardware crafts and software algorithms, more emphasis has been put on cheap on-board cameras which can collect a large amount of traffic information. Despite the research attention to cameras, the robustness of the autonomous driving system is always a focus, which indicates the risk that weather changes can affect the system reliability. Therefore, this paper focuses on object detection in autonomous driving under various weather conditions. Most recent object detection pipelines in the regression method can be trained end to end, which is fast in both training and inference and is compatible with real-time detection application. However, almost all state-of-the-art on-board object detection systems have been tested under good weather conditions, partly because there are few data sets related to bad weather conditions and even some famous traffic data sets, such as the KITTI data set and GTSDB, focused solely on the fair weather traffic scene [3]. In addition, it is difficult to maintain high detection precision under varied weather conditions using the existing object detection systems. Sample images captured under bad weather conditions are shown in Fig. 1. Fig. 1Open in figure viewerPowerPoint Extreme weather samples Present research and investigation propose the following two steps to achieve better detection: (i) Building a network to categorize the current frame into different weather types. (ii) Building specific networks for different weather conditions. This method is referred to as the 'multi-model method' because it is made up of at least three separate models, namely a classification model, a good weather detection model, and a bad weather detection model. However, it demands complicated and repetitive calculations. For example, the feature extraction process in classification and target detection is calculated twice. Moreover, the storage space requirement is high due to the need to store at least three networks. Taking these into consideration, this paper designs MWNet to solve object detection problems under different weather conditions, which consists of an encoder made up of several convolutional layers to extract features and a decoder containing three subnets (classification, bad weather object detection and fair weather object detection subnets) to detect objects. In addition, three subnets share same convolutional layers. The structure of the MWNet is shown in Fig. 2. The weather classification subnet is used to discriminate weather conditions of the current frame as fair or bad, and its result is used to determine the subnet in which the next frame goes. Fig. 2Open in figure viewerPowerPoint MWNet: for the input real-time video sequence, every frame will first go through the shared encoder and then go into the corresponding detection subnet guided by the previous frame's classification result Compared with the multi-model method, MWNet has better properties, like (i) Higher precision : it will generate better results to train multiple tasks in one network than to train multiple networks individually. (ii) Higher unity : establishing a single neural network helps not only accomplish object detection under all kinds of weather conditions but also achieve end-to-end training. (iii) Higher effectiveness : multiple tasks sharing some convolutional layers make model conversion unnecessary, thus saving storage space and running time. 2 Related work This section mainly reviews the two commonly used object detection methods under different weather conditions. The first method begins with the assessment made about the weather through real-time vehicle video sequences, then carries out image preprocessing or chooses a suitable object detection model according to the assessment, and finally detects what we call multi-model methods. For example, an object detection method under foggy weather requires the real-time recognition of the weather condition, and, if it is classified as foggy [4], an atmospheric scattering model will be used. There are some similar methods [5, 6], which, however, not only increase the amount of computation but also lengthen the time for model conversion. The other method, referred to as the single-model method, applies one model to detect objects regardless of the weather. Examples of this method include [7-9]. However, this method is less accurate and has a small range of application. For example, preliminary tests on vehicle detection in the case of blizzard were performed using a region-based Gaussian mixture model with no specific accuracy [10]. It is worth pondering that object detection under bad weather conditions falls into traditional machine learning instead of deep learning. Although most well-known neural networks are trained based on the images captured in the fair weather, the accuracy of these networks, in both classification and detection, is much higher than that of traditional machine-learning methods. Therefore, a neural network structure that can detect objects in different weather conditions is proposed. As for deep-learning object detection, there are two main research orientations. The first is based on region proposals. First, multiple region proposals [11, 12] are generated, then every proposal is classified using CNN, and finally, the RPN net based on CNN is proposed to generate region proposals [13]. Recently, the combination of features in the last few resolutions of CNN has been used to further improve accuracy [14]. The other orientation is the regression method, where end-to-end training can be achieved through a network [15-18]. The training and inference speed of this method is higher than the region proposals method, thus making it viable for practical application. Until now, YOLOv2 [19], YOLOv3 [20], SSD [18], DSSD [21], and RetinaNet [22] are the state-of-the-art mainstream detection algorithms based on the regression method. In order to verify the versatility of the MWNet, the five methods are used in the detection subnet. 3 MWNet Multi-weather network (MWNet) proposed in this study utilizes encoder–decoder architecture to perform object detection under various weather conditions. During the encoding process, some shared convolutional layers are used mainly to extract features. In addition the decoder consists of three subnets indicating three tasks, all of which require a classifier that discriminates weather types and recommends the two detection subsets, respectively under good and bad weather conditions. The structure is shown in Fig. 2. This network can be trained end-to-end. Since the network structure in this paper is for video sequences in vision-based autonomous driving systems, the video should be continuously in time. And classification result of the previous frames can be used to guide the detection branch of this frame. If weather condition in the previous frame is classified as bad, the next frame is directly switched to the detection branch under bad weather. The following section introduces MWNet's encoder and three subnets. 3.1 Encoder Encoder is a feature extraction process which is made up of some convolutional layers. Most of the features needed in the three decoders are extracted from it [23]. Since this paper uses five mainstream detection methods to prove the reliability of the proposed network structure, the encoder part is the convolution layers for feature extraction for different detection methods. Regardless of the feature extraction layers of the five detection methods, the resolution of last feature maps was 1/32 of source image. 3.2 Classification subnet The classification task in this study aimed to classify weather types in the vehicle video in real time which was easily completed. First, a 1*1 convolution was used after the encoder network structure, then a full connection layer was followed for classification, and finally, a softmax was used to output final class probabilities. The structure is shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint Structure of the encoder and the classification subnet decoder in the MWNet for weather condition classification: a 1*1convolution and a softmax layer are following the encoder 3.3 Detection subnet For the two kinds of weather detection subnets, the structure is the same. This paper used the regression method to complete the detection task. In order to verify the versatility of the MWNet, almost all existing state-of-the-art object detection methods were applied. Methods include SSD, DSSD, YOLOv2, YOLOv3, and RetinaNet. RetinaNet, SSD, and DSSD used ResNet101 as the encoder. 4 Training The loss function was defined as the sum of both the classification and detection losses, and cross-entropy was used to classify branch, as is defined in the following formula: (1) where b is batch size, c is the number of classes (there are two classes: fair weather and bad weather), q is the ground truth, and p is the prediction. Since this paper directly used five detection methods as the detection subnet of MWNet, its loss was calculated using the original algorithm. Therefore, the total loss of MWNet was obtained through the weighted sum of two losses, as shown below (2) Training MWNet used to be a big challenge, which has been overcome in this study through the following method. To begin with, training samples were classified into the bad weather images and the fair weather images. Each image was placed in the classification and corresponding detection subnet based on the label, followed by calculating classification and detection losses. Thereafter, the sum of detection and classification losses was used to update the weights of the classification subnet, corresponding detection subnet, and shared convolutional layers. It was found through these experiments that the test result would be the best when bad weather samples were only used to update the bad weather detection subnet and classification subnet instead of the shared convolutional layers, while fair weather samples were adopted to update all of them. The analysis showed that fair weather samples include more basic features, such as location, shape, and color. Therefore, when it comes to bad weather training, these basic features can be used directly, which also helps increase the training speed. 5 Experiment So far, there has been no mature algorithm that can achieve object detection under various weather conditions. Therefore, two structures were built to compare with MWNet. One is the multi-model shown in Fig. 4. The structure of its detection and classification models is the same as that of the MWNet. However, the three tasks of the multi-model network do not share convolutional layers, with each task having its own convolutional layer to extract the features in the contract. The other structure is shown in Fig. 5, which is referred to as the single-model structure. It is made up of several convolutional layers and a detection subnet identical to MWNet. To evaluate the effectiveness of the proposed MWNet, a series of comparative experiments were carried out on the author's data set, to prove that the MWNet, compared with the multi-model and single-model structures, can achieve better results. Thus, more detailed experimental results relating to this stage are provided. Fig. 4Open in figure viewerPowerPoint Multi-model: it is built to compare with the MWNet. The difference between them is that its three models have their own convolutional layers to extract the features Fig. 5Open in figure viewerPowerPoint Single-model: it is built to compare with the MWNet. The difference between them is that it need not classify the samples, but just perform object detection regardless of the kind of the input, enabling it to be trained using all kind of samples 5.1 Data set Until now, there has been no open data set that includes both fair and bad weather images, so the data set was collected first-hand using the camera on the car, which covered different streets. Each video sequence contained thousands of images, from which dozens were extracted and manually annotated. There were shelter and truncation in these images like KITTI data set. When conducting the manual annotation, Yuan et al. 's article [24] was referenced. The labelled objects should be bigger than 30*30, and shelter is bigger than two-thirds of the car body. In addition, image size is 1248*384*3 which is similar to KITTI. Another very important issue is how to define training samples, e.g. which samples are under the fair weather and which others are under the bad weather? To achieve this goal, two detection networks with the same structure and different training samples were used to detect objects in edge images, which were classified by means of detection accuracy. There are three columns of sample images in Fig. 6 that are under fair, bad, and edge weather conditions, respectively. Fig. 6Open in figure viewerPowerPoint Samples in the author's data sets Fig. 1 images captured under bad weather conditions (a) Fair weather samples, (b) Bad weather samples, (c) Edge weather samples The following table shows the detailed information of the author's data set. In total, 34,126 fair weather samples and 9298 bad weather samples were used for training. Among them, 8000 fair weather samples were made up of about 100 consecutive videos and 2000 bad weather samples were made up of about 20 consecutive videos. In addition, bad weather mainly consists of rain and snow. These samples were manually annotated, and a bounding box was drawn around an object in the samples. The objects include pedestrians, vehicles and non-motor vehicles, which cover all operational objects in traffic scenarios. Moreover, there are currently many tricycles used in China for express delivery, which are also classified as non-motorized vehicles. 5.2 Experiment for MWNet on data set Although object detection methods are different, they are all divided into five stages, namely conv-1–conv-5, as the pool is reduced to 1/32 of the original image size. Based on MWNet structure, this paper proposes three networks, among which the difference lies in the number of shared convolutional layers. The shared convolutional layers of Net1 are from conv-1–conv-3 whose resolution is 1/8 that of the original image, Net2 is from conv-1–conv-4 and Net3 is from conv-1–conv-5. However, their detection method is the same, which involves an encoder and a detection subnet. Their network is shown in Fig. 7. Fig. 7Open in figure viewerPowerPoint Proposed networks using the MWNet structure. According to the MWNet structure, this paper proposes three networks. Their difference lies in the number of shared convolutional layers Tables 1 and 2 show object detection results of the five object detection networks on fair and bad weather testing sets. It can be seen that regardless of which object detection method, Net2 performs best, so its structure is recognized as MWNet. Table 1. Data sets Data set Fair weather Bad weather Total samples object Pedestrian Non-motor Car Pedestrian Non-motor Car Fair weather Bad weather training 62,546 16,045 209,508 9571 2340 59,632 32,126 9298 test 12,300 1828 42,308 1344 434 12,088 8000 2000 Table 2. Object detection result (map) using the MWNet structure Network Testing sets SSD (%) YOLOv3 (%) YOLOv2 (%) DSSD (%) RetinaNet (%) Net1 Fair 82.1 89.2 77.2 85.2 85.7 Bad 79.3 86.1 75.3 81.9 82.3 Net2 Fair 82.1 89.2 77.2 85.2 85.7 Bad 81.3 88.1 79.3 83.3 82.6 Net3 Fair 82.1 89.2 77.2 85.2 85.7 Bad 80.9 87.4 78.6 82.1 81.4 Bold values indicate the best performance. The MWNet can promote the performance of current object detection methods under various weather conditions. In order to prove this, the multi-model and single-model using the five object detection methods are compared in Table 3. Table 3. Object detection results (mAP) using the multi-model and the single-model Net Testing sets SSD (%) YOLOv3 (%) YOLOv2 (%) DSSD (%) RetinaNet (%) Multi-model Fair 81.2 88.4 76.9 84.1 83.2 Bad 77.6 83.7 75.7 80.6 81.1 Singlemodel Fair 76.6 84.8 73.5 79.9 79.8 Bad 72.5 80.1 71.2 76.2 77.5 MWNet Fair 82.1 89.2 77.2 85.2 85.7 Bad 81.3 88.1 79.3 83.3 82.6 Bold values indicate the best performance. Tables 4–6 show the results of the comparison between multi-model, single-model, and MWNet using YOLOv3, DSSD, and RetinaNet which requires the storage space, inference running time, object detection accuracy evaluation index, and mean average precision (mAP). Table 4. Comparison results using YOLOv3 Net Multi-model Single-model MWNet storage space 570.1 M 246.1 M 254.4 M bad weather 83.7% 80.1% 88.1% fair weather 88.4% 84.8% 89.2% running time 150.24 ms 76.59 ms 78.13 ms Bold values indicate the best performance. Table 5. Comparison results using RetinaNet Net Multi-model Single-model MWNet storage space 486.7M 162.9M 167.1M bad weather 81.1% 77.5% 82.6% fair weather 83.2% 79.8% 85.7% running time 521.32ms 136.20ms 142.05ms Bold values indicate the best performance. Table 6. Comparison results using DSSD net Multi-model Single-model MWNet storage space 489.0M 163.8M 169.4M bad weather 80.6% 76.2% 83.3% fair weather 84.1% 79.9% 85.2% running time 720.12ms 251.22 ms 268.31ms Bold values indicate the best performance. 5.3 Experiment results The experiment in Section 5.2 has generated the following conclusions: (i) It can be learned from Table 3 that multi-model performs better than single-model, which indicates that it is necessary to match different weather conditions with different models. (ii) Table 2 shows that Net2 performs the best on bad weather test samples, because, compared to Net1, it can obtain more high-level features from good-weather-trained encoders, and, compared with Net3, use more parameters to present features unique to the bad weather. Therefore, Net2 was defined as the MWNet. (iii) It can be learned from the comparison between Tables 2 and 3 that MWNet performs better than both multi-model and single-model. It can be concluded that MWNet structure is effective under all kinds of weather conditions. (iv) It can be learned from Tables 2 and 3 that compared with only one task (detection), two tasks (classification and detection) can generate better results. (v) It can be concluded from Tables 4–6 that MWNet is particularly compatible with real-time applications, suggesting that computational complexity improves significantly due to the shared architecture. 5.4 Discussion It can be seen from Tables 2–4 that, compared with multi-model and single-model, MWNet structure performs better not only on fair weather data sets but also on bad weather ones. The comparison results of fair weather data sets using YOLOv3 are shown in Fig. 8, where the bounding box in blue color indicates a detected car, red a person, and green a non-motor vehicle. Fig. 9 shows the detection results on bad weather sets. Fig. 8Open in figure viewerPowerPoint Detection results comparison on fair weather data sets (a) Is the source image, (b) Is obtained using the single model, (c) Using the multi-model, (d) Using the MWNet Fig. 9Open in figure viewerPowerPoint Detection results comparison on bad weather data sets (a) Is the source image, (b) Is obtained using the single model, (c) Uses the multi-model, (d) Uses the MWNet It can be learned from Figs. 8 and 9 that MWNet structure proposed in this paper has generated good results in detection under both fair and bad weather conditions. The main reasons are as follows: (i) through sharing an encoder, different detection subnets were built for different weather conditions, making the object detection more targeted. In addition, another classification subnet was used to guide the next frame detection subnet, making this proposed network structure more practical. (ii) Encoder trained using good weather samples was directly used in object detection under bad weather conditions, which promoted its performance. In addition to the high accuracy of detection, the speed of MWNet is acceptable for practical application. Moreover, it can save the time spent on model transformation and weather type classification, which is very important for the continuity during the application process. 6 Conclusions The MWNet is proposed in this paper for video sequence-based object detection in vision-based intelligent vehicle systems. To meet real-time and robustness requirements, a unified network structure is established to perform object detection under various weather conditions in real time. The results show that this simple approach can realize end-to-end training, that the universal network structure can be applied to different basic networks, and this approach is effective because the time to implement one detection and classification does not exceed 79 ms on GTX1080, ensuring the accuracy of object detection under both fair and bad weather conditions, and a large number of experimental results have proven that this framework can improve accuracy when dealing with bad weather conditions by 4%, and the processing time is half that of the multi-model method. Future research will further study object detection in the night and unify it into this network structure. 7 Acknowledgments This study was supported by the National Natural Science Foundation of China No. 61673047 and No. 61871024. 8 References 1Geiger A. Lenz P. Urtasun R.: ' Are we ready for autonomous driving? The KITTI vision benchmark suite'. Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit. (CVPR), Providence, RI, USA, 2012, pp. 3354– 3361, http://www.cvlibs.net/datasets/kitti/eval_object.php 2Chen X. Kundu K. Zhang Z. et al.: ' Monocular 3D object detection for autonomous driving'. Computer Vision & Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 2147– 2156 3Everingham M. Van Gool L. Williams C. K. et al.: 'The pascal visual object classes (voc) challenge', Int. J. Comput. Vis., 2010, 2, p. 5 4Dong N. Jia Z. Shao J. et al.: 'Adaptive object detection and visibility improvement in foggy image', J. Multimed., 2011, 6, (1), pp. 14– 21 5Chen D.-Y. Peng Y.-J.: 'Frequency-tuned taillight-based nighttime vehicle braking warning system', IEEE Sens. J., 2012, 12, (11), pp. 3285– 3292 6López-Rubio F. López-Rubio E.: 'Local color transformation analysis for sudden illumination change detection', Image Vis. Comput., 2015, 37, pp. 31– 47 7Tsai L. Hsieh J. Fan K.: 'Vehicle detection using normalized color and edge map', IEEE Trans. Image Process., 2007, 16, (3), pp. 850– 864 8Mu K. Hui F. Zhao X. et al.: 'Multiscale edge fusion for vehicle detection based on difference of Gaussian', Opt. – Int. J. Light Electron Opt., 2016, 127, (11), pp. 4794– 4798 9Shen X.: ' Vision detection of lanes and vehicles for advanced safety vehicles', PhD thesis, National Central University, 2007 10Varadarajan S. Miller P. Zhou H.: 'Region-based mixture of Gaussians modelling for foreground detection in dynamic scenes', Pattern Recognit., 2015, 48, (11), pp. 3488– 3503 11' Under Varying Environments: A review'. Proceedings of the 2nd world congress on civil, structural, and environmental engineering (CSEE'17), Barcelona, Spain, 2017 12Hosang J.H. Benenson R. Dolĺar P. et al.: 'What makes for effective detection proposals?', IEEE Trans. Pattern Anal. Mach. Intell., 2015, 38, (4), pp. 814– 830 13Girshick R.: ' 'Fast R-CNN''. IEEE Int. Conf. on Computer Vision (ICCV), Santiago, Chile, 2015, pp. 814– 830 14Ren S. He K. Girshick R.B. et al.: 'Faster R-CNN: towards real-time object detection with region proposal networks', IEEE Trans. Pattern Anal. Mach. Intell., 2017, 39, (6), pp. 1137– 1149 15Lin T.-Y. Dollár P. Girshick R. et al.: ' Feature pyramid networks for object detection'. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017 16Sermanet P. Eigen D. Zhang X. et al.: ' Overfeat: integrated recognition, localization and detection using convolutional networks', 2013, 2, p. 3, https://arxiv.org/abs/1312.6229 17Redmon J. Divvala S.K. Girshick R.B. et al.: 'You only look once: unified, real-time object detection'. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016 18Liu W. Anguelov D. Erhan D. et al.: ' SSD: single shot multibox detector'. European Conference on Computer Vision, Amsterdam, The Netherlands, 2015, 2 19Redmon J. Farhadi A.: ' YOLO9000: better, faster, stronger'. Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017 20Redmon J. Farhadi A.: 'YOLOv3: an incremental improvement'. 2018 21Fu C.Y. Liu W. Ranga A. et al.: 'DSSD: Deconvolutional Single Shot Detector', 2017 22Lin T.Y. Goyal P. Girshick R. et al.: 'Focal loss for dense object detection', IEEE Trans. Pattern Anal. Mach. Intell., 2017, 1, (99), pp. 2999– 3007 23Szegedy C. Liu W. Jia Y. et al.: ' Going deeper with convolutions'. Computer Vision and Pattern Recognition, Boston, MA, USA, 2015 24Yuan X. Hao X. Chen H. et al.: 'Traffic sign recognition based on a context-aware scale-invariant featuretransform approach', J. Electron. Imaging, 2013, 22, (4), p. 041105 Citing Literature Volume13, Issue9September 2019Pages 1394-1400 FiguresReferencesRelatedInformation