Deep learning‐based vehicle detection with synthetic image data

IET Intelligent Transport SystemsVolume 13, Issue 7 p. 1097-1105 Research ArticleFree Access Deep learning-based vehicle detection with synthetic image data Ye Wang, Ye Wang State Key Laboratory of Automotive Simulation and Control, Jilin University, Renmin Street No.5988, Changchun, People's Republic of ChinaSearch for more papers by this authorWeiwen Deng, Corresponding Author Weiwen Deng wdeng@buaa.edu.cn State Key Laboratory of Automotive Simulation and Control, Jilin University, Renmin Street No.5988, Changchun, People's Republic of China Beijing Advanced Innovation Center for Big Data and Brain Computing, Beihang University, Beijing, People's Republic of ChinaSearch for more papers by this authorZhenyi Liu, Zhenyi Liu State Key Laboratory of Automotive Simulation and Control, Jilin University, Renmin Street No.5988, Changchun, People's Republic of ChinaSearch for more papers by this authorJinsong Wang, Jinsong Wang General Motors Corporation, Detroit, MI, USASearch for more papers by this author Ye Wang, Ye Wang State Key Laboratory of Automotive Simulation and Control, Jilin University, Renmin Street No.5988, Changchun, People's Republic of ChinaSearch for more papers by this authorWeiwen Deng, Corresponding Author Weiwen Deng wdeng@buaa.edu.cn State Key Laboratory of Automotive Simulation and Control, Jilin University, Renmin Street No.5988, Changchun, People's Republic of China Beijing Advanced Innovation Center for Big Data and Brain Computing, Beihang University, Beijing, People's Republic of ChinaSearch for more papers by this authorZhenyi Liu, Zhenyi Liu State Key Laboratory of Automotive Simulation and Control, Jilin University, Renmin Street No.5988, Changchun, People's Republic of ChinaSearch for more papers by this authorJinsong Wang, Jinsong Wang General Motors Corporation, Detroit, MI, USASearch for more papers by this author First published: 27 March 2019 https://doi.org/10.1049/iet-its.2018.5365Citations: 5AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Deep convolutional neural network (CNN)-based object detectors perform better than other kinds of object detectors in recent development. Training CNNs needs large amounts of annotated data. The acquisition and annotation of real images are arduous and often lead to inaccuracy. In order to solve this problem, the authors try to use synthetic images as substitute to train a vehicle detector. Annotation on synthetic images can be performed automatically, and can obtain better uniformity. Furthermore, it is also easy to get more variations in synthetic images. The authors present a pipeline to generate synthetic images for training vehicle detector. In this pipeline, many factors are considered to add more variations and extend the domain of training dataset. Thus, the detector trained with synthetic images is expected to perform well. The extent to which these factors influence detection performance is illustrated. However, the performance of a vehicle detector trained with synthetic images is not as good as that with real images because of domain gap. In this study, the authors develop a transfer learning approach to improve the performance of authors' vehicle detector with only a few manually annotated real images. 1 Introduction Safe, green, and smart vehicles are the trend for future vehicle development. An advanced driver assistance system (ADAS) is considered as an effective way to achieve this target. Vision-based ADAS (V-ADAS), which uses a camera to acquire information from the driving environment, is an important part of the ADAS. The advantages of using a camera as sensor are apparent. First, a camera is cheaper than other sensors, such as radar and lidar. Second, the information contained in images is abundant, including lanes, vehicles, pedestrians, and traffic lights. Third, as a passive sensor, a camera has no influence on the environment or other sensors. Given its advantages, V-ADAS has attracted the attention of researchers and manufacturers. Many kinds of V-ADAS [1-4] have been developed and researched. From the viewpoint of internet of vehicles (IOV), information extracted from camera sensor can be shared through vehicle to vehicle (V2 V) and vehicle to infrastructure (V2I) methods [5]. Some vision-based methods could detect traffic accident [6] and traffic density [7]. When this kind of information could be shared among vehicles, transportation safety and efficiency will be improved significantly. Furthermore, the combination of vehicular vision and IOV can also be used for vehicle positioning [8]. However, object recognition from images is more difficult than that with other sensors because of the complexity of image signals, this feature may be the biggest drawback of using a camera to acquire information. As an important traffic participant, vehicle detection is a useful application in V-ADAS. Most vehicle detection methods consist of two procedures: hypothesis generation (HG) and hypothesis verification (HV). HG usually produces many vehicle candidates roughly for fast computing speed. Then, HV tests the candidates with an accurate classifier. Traditional methods may utilise some knowledge-based information, such as shadows [9], symmetry [10], and vertical/horizontal edges [11], to generate the possible location of vehicles in an image. Then, predefined feature vectors such as Harr, HOG, and Gabor are used together with classifiers such as SVM and AdaBoost for HV [12, 13]. However, due to the complexity of environment, this kind of methods obtained limited performance in both HG and HV procedure. For HG procedure, symmetry-based methods are only valid in rear and front view of vehicle, shadow, and edge-based methods suffer from the selection of threshold. For HV procedure, the representational ability of man-designed feature vectors such as Harr and HOG is also limited, and disadvantageous for final classification. Due to the great representational ability of convolutional neural networks (CNN), recently, CNN-based methods have outperformed other methods and gained considerable attention in research on object detection. The popularity of CNN also benefit by the large amount of annotated image dataset and the progress of hardware computing speed. An important feature of CNN that is different from traditional classifiers is that it does not need a man-designed feature vector extractor. CNN can learn what feature is proper for a specific task through supervised learning. The R-CNN series (R-CNN [14], Fast R-CNN [15], and Faster R-CNN [16]) are typical examples which utilise CNN for object detection. The main idea of these object detectors is to generate candidate regions in the images by using methods such as selective search or region proposal network, this can be seen as HG procedure. Then, classification is performed on these regions, this can be seen as HV procedure. The R-CNN series are general purpose object detectors, not specially designed for vehicle detection. Inspired by them, many variants [17, 18] of them are designed for vehicle detection. All of them are called data-driven methods because large amounts of annotated image data is needed for supervised learning. If we want to obtain a higher performance, we need a more complex network, and more annotated image data. Real image datasets, such as Kitti [19], Pascal [20], and Cityscapes [21], which are aimed at object detection, are available on the Internet. However, manual real image data annotation is an arduous task and may lead to inaccuracy and inconsistency. Furthermore, training image capture is a difficult problem. To cover the target domain as widely as possible, images should be captured under different environments and light conditions. For a specific camera, intrinsic and extrinsic matrices considerably influence the appearance of target objects. Such factors make the utilisation of real images to train a vehicle detector inconvenient and ineffective. All these disadvantages of real image dataset mentioned above can be easily solved when utilise synthetic images to train vehicle detectors. Synthetic images are easier to acquire and annotate because we know all accurate information of target objects. Furthermore, imaging factors such as light condition and background are easy to change which leads to large training data variation. The development in the field of computer graphics and game rendering engine has resulted in synthetic images becoming more similar to real images to the point where human eyes have difficulty in distinguishing between these images. The lighting model shifted from experience-based lighting model, such as that presented by Phong [22], to a physically based lighting model [23] and from a local lighting model to a global lighting model. These lighting models make synthetic images more lifelike. Engines such as Unreal Engine 4 (UE4) [24], Unity [25], and CryEngine [26], which produce lifelike synthetic images, are available on the Internet. Reference [27] built a Virtual Kitti dataset with Unity and generate synthetic images used for object detection and multi-object tracking. It is almost a virtual clone of real Kitti, and building such a dataset is really a difficult task that requires a lot of work and professional skills. Although the synthetic images are really similar to real Kitti dataset, the detection result is still not satisfactory. Some research [28, 29] focused on training CNN with non-photorealistic images, which are easier to acquire and have a large range of variation. Moreover, professional knowledge and skills in 3D scene modelling and rendering are not required in such kind of methods. In this study, we train vehicle detectors with synthetic images with both photorealistic and non-photorealistic synthetic images, due to their different advantages. Synthetic images are easy to acquire, and annotation can be done automatically and accurately. Furthermore, crucial factors such as light condition, camera parameter, and background can be changed easily. Modern game engines can ensure the high fidelity of synthetic images, and randomly changed factors can ensure the large range of variation. However, a domain gap still exists between real image and synthetic image dataset. Generally, detectors trained with only synthetic images deliver lower performance than those trained with real image dataset. Therefore, we develop a transfer learning method to fill the domain gap between real and synthetic images with minimal real image dataset. The contributions of this research are as follows: (1) We present an effective synthetic image dataset generation method for photorealistic and non-photorealistic images, which are combined when training vehicle detectors. Detectors trained solely with a synthetic image dataset can also deliver considerable performance. We discuss the influence of the factors in detail. (2) We provide a detailed insight into how much the transfer learning method can improve the performance of detectors trained on synthetic images. The relationship between the number of added real images and the improvement of performance is also studied and illustrated. This paper is organised as follows: First, previously related work is introduced. Second, the synthetic image generation and annotation procedures are presented, the employed object detector (Faster R-CNN) and the transfer learning method are introduced. Third, the experimental setup is demonstrated, evaluation results are presented, and influential factors are discussed comprehensively. Finally, the conclusion and future work are provided. 2 Related works As an important issue in computer vision, object detection has always been a hot research topic. For CNN-based object detection, CNN can play a role in both HG procedure to generate candidates and HV procedure to perform classification, and obtain a better performance than other methods. In the following paragraphs, different kinds of CNN-based object detectors are introduced. Works on training CNN with synthetic images are also reviewed. 2.1 CNN-based object detection R-CNN utilises selective research method to generate proposal bounding boxes, which are then transmitted to a CNN for classification. Proposal bounding boxes may reach up to the thousands, and most of them overlap. Features in overlapped regions are computed repeatedly, resulting in unsatisfactory computing time. Inspired by SPP-NET [30], the Fast R-CNN method projects the proposal region to the output feature map of the last CNN convolutional layer. The feature map needs to be computed only once for the whole image. Fast R-CNN has better computing speed and detection accuracy than R-CNN. Faster R-CNN utilises a CNN-based method called Region Proposal Network (RPN) for region proposal and further improves computing speed and detection accuracy. Following the main idea of Faster R-CNN, many improving methods based on it were developed. Feature extractor such as Inception [31], DenseNet [32], and FPN [33] are developed to induce multi-scale property. MS-CNN [34] performed RPN and classification on feature maps with different sizes, which also induce multi-scale property. Reference [35] utilised contextual information to further improve the detection performance. Some other detectors use end-to-end method, such as YOLO [36] and SSD [37]. YOLO and SSD use only one CNN network for object detection and need to be trained only once. Without the RPN stage, the computing speed of YOLO and SSD is faster than that of Faster R-CNN. Some researches focused on employing deep learning method for vehicle detection. Reference [38] investigated the performance of R-CNN and Faster R-CNN on vehicle detection. Reference [39] used DNN to improve the performance of vehicle detection and classification. All these methods are supervised learning methods, which means that large amount of annotated data are needed. Original version of them is trained with annotated real image. Although there are many datasets available on the Internet, the amount and variation of them are still limited. Furthermore, manual annotation of them is a burdensome work. This motivates us trying to utilise synthetic images which are easy to obtain and annotate to train vehicle detector. 2.2 Synthetic image training The use of synthetic images to train deep neural networks has been explored in recent years. Reference [40] utilised synthetic images to train a pedestrian detector. A deformable part-based model with mixture-of-parts approach was employed as classifier, and the half-life 2 game engine was used to generate synthetic images. Synthetic images generated from this engine is not lifelike enough. They are utilised just as an auxiliary role for part labelling, and large amount of real annotated data are still needed. Three-dimensional CAD models are easy to acquire online; thus, [41] used these models to generate different types of synthetic images to train a deep learning-based detector, rendering the target object into a background image. The target object was then rendered under two different levels of detail. The background image was varied among three different levels of detail. A total of six levels of detail for synthetic images was acquired. The study investigated different detector performances trained with different levels of detailed synthetic images. However, such a method cannot generate synthetic images with right context information such as shadow. The context information is important for object detection. [42] used Blender, an open-source 3D graphics software, and its APIs to load 3D models and automated the scene rendering. The generated images were used to train an object detector which detected objects in a refrigerator. It took several aspects into consideration when synthetic images such as number, shape, texture, materials of objects, and refrigerators were generated. The 3D CAD model database ShapeNet [43] ensured the large variety of objects. Comparatively speaking, the application scenario of [42] is small and simple. It is easier to cover the domain compared with outdoor scenario for vehicle detection. Reference [27] build a Virtual Kitti dataset with Unity. It is a clone of real Kitti, each virtual image is corresponding to a real image in Kitti. Although it is built with a lot of work, detector trained with it still do not have satisfactory performance because of the domain gap and lack of variation. Here, we try to present a synthetic training image pipeline for vehicle detection and discuss the influence factors in detail. Our synthetic images are generated not only without too much heavy manual works like Virtual Kitti, but also get a better performance compared with it. 3 Synthetic image generation and annotation In this study, two main ideas are presented when generating synthetic images. One is generating photorealistic images. In such images, we attempt to make synthetic images as lifelike as possible. They have realistic context such as shadow and road surface, but they can hardly achieve sufficient variation because of the arduous task of scene modelling. To achieve such images, we need experts on scene modelling and rendering. The other idea is generating non-photorealistic images. We render vehicles into randomly selected background images with different rendering parameters, and dataset augmentation algorithms can be easily implemented in this procedure. Although they have no realistic context, they can induce many variations in the training dataset, which is also important to the final detection performance. Here, these two synthetic datasets are combined to complement each other and improve the performance of a vehicle detector. 3.1 Photorealistic image generation Our synthetic image dataset is generated with the UE4 game engine [24]. This game engine utilises physically based rendering methods to ensure the fidelity of synthetic images. Essentially, rendering algorithm is a simulation of the interaction between the incident light and the object surface. Two main factors in this interaction are object surface optic properties and incident light. The incident light comes from every direction and hits the object surface and is either reflected and absorbed or passes through depending on the surface's optic properties. Bidirectional reflectance distribution function (BRDF) is a characterisation method of object surface properties. As shown in Fig. 1, BRDF approximates how much each individual incident light ray contributes to the final reflected light of a surface given its material properties. In the UE4 rendering system, BRDF consists of two parts: diffuse and microfacet. (1) where l is the incident light direction vector, n is the normal vector of the surface, v is the view vector towards the eye point, h is the halfway vector between l and v, and represents the diffuse coefficient. represents the normal distribution function. It approximates the amount of the surface's microfacets that are aligned to the halfway vector influenced by the roughness of the surface. (2) Fig. 1Open in figure viewerPowerPoint Bidirectional reflectance distribution function is the geometry function. It describes the self-shadowing property of the microfacets. When a surface is relatively rough, the surface's microfacets can overshadow other microfacets, thereby reducing the light reflected by the surface. (3) (4) (5) is the Fresnel equation, which represents the Fresnel reflection when incident light hits the surface point. The Fresnel equation describes the ratio of surface reflection at different surface angles. (6) where is the specular reflectance at normal incidence. In UE4 material editor, three important parameters are considered: base colour, roughness, and metallic, which are relevant to the c_diff, Roughness, and in these equations, respectively. Finally, the rendering Equation [44] is solved to acquire the final image colour. (7) where Li (l) is the incident light. Incident light consists of direct light and indirect light. Direct light is given by users, whereas indirect light needs to be computed. Specific methods such as environmental map, ambient light occlusion, and soft shadow are employed to simulate indirect light propagation in UE4. With the representation of BDRF, the rendering equation, and the specific methods that simulate light propagation, UE4 can generate lifelike synthetic images with a fast computing speed. Aside from fidelity, the variety of synthetic images is also important for training detectors. If the variety of synthetic images is not enough to cover as much of the domain as possible, then the detector will overfit on some features that are not robust. Several things are considered to expand the variation of synthetic images. First, as the target object, the variety of vehicle appearance is important. We collected >100 different vehicle 3D models from the Internet to form a 3D vehicle shape library. The library includes bus, hatchback, minicar, MPV, sedan, sports car, SUV, and truck. Each type has an obviously different appearance from another. The type range covers most of the common vehicles. We also change the material parameters of vehicle paint (base colour, roughness, metallic) randomly to obtain more different vehicle appearances. As shown in Fig. 2, the above-mentioned vehicles ensure the large variety of positive samples. Fig. 2Open in figure viewerPowerPoint Different types and materials of vehicles Aside from the vehicles, the background also has a considerable influence on the training procedure because the background is used as a negative sample. As shown in Fig. 3, we use three different types of traffic scene: one represents downtown, another represents the suburb, and the other represents the highway. The background changes considerably because of different scene layouts. Vehicles are randomly located on the road and beside the road of these three scenes in different orientations. Different relative positions and different orientations of vehicles also lead to different appearances in the final synthetic images. Fig. 3Open in figure viewerPowerPoint Different types of background scene (a) Downtown, (b) Suburb, (c) Highway Light condition greatly influences the appearance of objects in synthetic images regardless if they are target objects or background. We use directional light combined with an HDR environmental map to simulate different light conditions. We employ two main weather conditions: sunny and cloudy days. Two different HDR environmental maps are used to present these two main weather conditions. We also use a simple strategy to simulate different light directions and intensities at different time points of a day. The illumination trait is different between sunny and cloudy days. For sunny days, the directional light takes the main light source and its intensity changes greatly at different time points. We use an HDR environmental map to present the non-directional light source. Compared with directional light, the intensity of non-directional light source in sunny days is lower and changes minimally at different time points. For sunny days, we use a constant non-directional light source intensity and change the direction and intensity of directional light. For cloudy days, the intensity of directional light is relatively low because the clouds scatter most of the sunshine into different directions. The non-directional light takes the main light source. We change the direction of directional light source, but we do not change the intensity. Instead, we change the intensity of non-directional light. This strategy allows us to simulate different light conditions realistically at different time points and weather conditions. Results are shown in Fig. 4. Fig. 4Open in figure viewerPowerPoint Different light conditions at different time points and weather conditions More than 100 sample cameras that are randomly distributed along the road in each scene are utilised. Eight different sample directions are employed at each sample point. Along with different vehicle positions and orientations in the world coordinate system, the vehicle positions and orientations also change considerably in image coordinates, thereby increasing the variety of vehicle appearance in the synthetic images. 3.2 Non-photorealistic image generation Photorealistic images have realistic context, such as under-vehicle shadow, road surface, and roadside infrastructure. However, due to the professional and complex work of building and rendering lifelike 3D scenes, we have difficulty acquiring sufficient variations in photorealistic images. Thus, we add non-photorealistic images to solve this problem. We render target vehicles into real background images to generate non-photorealistic images. As shown in Fig. 5, for each image, we render three vehicles into it. We denote them as Veh0, Veh1, and Veh2 according to their distances to the camera. The azimuth angle of Veh0 (Azimuth0) is randomly determined in a range from −40° to 40° to ensure it can appear in the field of view. The distance of Veh0 (Dist0) to the camera is also randomly determined in a pre-defined range from 5 m to 50 m. The distance of Veh1 reference point (Reference Point1) is equal to Dist0 plus a random shift value (DistShift1) in a range from 3 m to 15 m, and the azimuth angle of Veh1 reference point is equal to Azimuth0. Then, a random shift along Y-axis (YShift1) in a range from −5 m to 5 m is added to Veh1. The distance of Veh2 is equal to the distance of Veh1 plus a random shift value in a range from 3 m to 15 m, and the azimuth angle of Veh2 is randomly determined in a range from −40° to 40°. The yaw angles of the three vehicles are all randomly determined in a range from 0 to 360. In order to present the non-flatness of ground, some small random values are added to height, pitch and roll angle of the three vehicles. With such a strategy, Veh0 could never be occluded, Veh1 could always be occluded, and the occlusion state of Veh2 is uncertain. Three vehicles in one image have different occlusion states, different distances and different poses. Such a large range of variation is good as a training dataset. Vehicles are rendered with the same rendering equation in the previous section. Material (base colour, roughness, metallic) and light condition parameters (intensity, direction) are also randomly determined in a pre-defined range. The colour flag image and foreground image are generated with UE4. The colour flag image determines which pixels are foreground and which pixels are background. Real background images are selected from the PASCAL image dataset which does not contain vehicles. Fig. 5Open in figure viewerPowerPoint Non-photorealistic image generation The main procedure in generating non-photorealistic images is shown in Fig. 5. 3.3 Post-processing The rendered images in previous sections ignore the imaging phenomena caused by the camera. In this study, we generate three camera-related effects (intensity response, blur, noise) on the training dataset. For intensity response, we use linear mapping, which can rep resent different camera response curves, to transform the image intensity from the original range to a randomly selected target range. The edge between the vehicle and the background is sharp in these synthetic images. However, in real images, the edge is more or less smooth because of aberration and depth of field. Edge is an important bottom layer feature in CNN networks. Therefore, inducing this effect to the training dataset is necessary. For the blur effect, we use Gaussian kernel with randomly varied standard deviation to perform convolution on the source image. When the material of a surface is constant (such as car paint), the appearance of the surface in the final image becomes uniform, without any noise. In real images, noise will appear in the image due to sensor response. For the noise effect, two types of noise (Gaussian white noise and salt-and-pepper noise) with randomly varied variances are added to the image. The main procedure of post-processing is shown in Fig. 6. Fig. 6Open in figure viewerPowerPoint Post-processing 3.4 Annotation Annotation is also performed when generating synthetic images. We know all the information of every object that appears in the synthetic images from the interior of the UE4 engine. The annotation method is shown in Fig. 7. For every vehicle, we create a unique RGB colour flag. If the object is not a vehicle, then we set the colour flag to white. Then, we render the scene with the colour flags and call the rendered image ‘colour flag image.’ Each non-white colour in the colour image represents a vehicle. Vehicles that are totally occluded or not in the field of view will not appear in the colour flag image, thus eliminating the need to search all vehicles. Then, we select the vehicles that are too small in the colour flag image. The bounding boxes should not contain only the visible parts of vehicles. As shown in Fig. 7, they should also contain the occluded parts. Therefore, we should not just take the minimum x, minimum y, maximum x, and maximum y of visible colour flag pixel positions of a vehicle in image coordinate as the 2D bounding box label. Throu