Vehicle detection in intelligent transport system under a hazy environment: a survey

IET Image ProcessingVolume 14, Issue 1 p. 1-10 Review ArticleFree Access Vehicle detection in intelligent transport system under a hazy environment: a survey Agha Asim Husain, Corresponding Author Agha Asim Husain aghahusain@gmail.com Department of Mining Machinery Engineering, Indian Institute of Technology (ISM), Dhanbad, IndiaSearch for more papers by this authorTanmoy Maity, Tanmoy Maity Department of Mining Machinery Engineering, Indian Institute of Technology (ISM), Dhanbad, IndiaSearch for more papers by this authorRavindra Kumar Yadav, Ravindra Kumar Yadav Department of Electronics and Communication Engineering, Skyline Institute of Engineering and Technology, Greater Noida, IndiaSearch for more papers by this author Agha Asim Husain, Corresponding Author Agha Asim Husain aghahusain@gmail.com Department of Mining Machinery Engineering, Indian Institute of Technology (ISM), Dhanbad, IndiaSearch for more papers by this authorTanmoy Maity, Tanmoy Maity Department of Mining Machinery Engineering, Indian Institute of Technology (ISM), Dhanbad, IndiaSearch for more papers by this authorRavindra Kumar Yadav, Ravindra Kumar Yadav Department of Electronics and Communication Engineering, Skyline Institute of Engineering and Technology, Greater Noida, IndiaSearch for more papers by this author First published: 01 January 2020 https://doi.org/10.1049/iet-ipr.2018.5351Citations: 9AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Developing an intelligent transportation system has attracted a lot of attention in the recent past. Moreover, with the growing number of vehicles on the road most nations are adopting an intelligent transport system (ITS) for handling issues like traffic flow density, queue length, the average speed of the traffic, and total vehicles passing through a point in a specific time interval and so on. ITS by capturing traffic images and videos through cameras, helps the traffic control centres in monitoring and managing the traffic. Efficient and unfailing vehicle detection is a crucial step for the ITS. This study reviews different techniques and applications used around the world for vehicle detection under various environmental conditions based on video processing systems. This study also discusses the types of cameras used for vehicle detections, and the classification of vehicles for traffic monitoring and controlling. This study finally highlights the problems encountered during surveillance under extreme weather conditions. 1 Introduction Nowadays, traffic detection is attracting a lot of researchers’ attention in computer vision and intelligent transportation systems [1, 2]. The rapid upsurge in several vehicles in the last few years has posed serious problems in the management of the transportation system vividly. Conditions such as accidents, blockages on roads, vehicle robberies, and traffic congestion are a common problem faced in the management of traffic. The intelligent transport system (ITS) has become a dynamic area of research since the last decade. It plays an important role in handling many real-time situations like avoiding traffic congestions and accidents, in the surveillance of traffic, in the collection of toll tax, terrorist monitoring, and in traffic flow under hazy environments [3], and many more. That is why ITS has become a significant field of study. Vehicle detection is the key task of the transport monitoring system, as after the detection of the vehicle, other applications can be used in a better way. Various technologies [4, 5] are being undertaken for detecting and categorising vehicles automatically. Different algorithms (based on shape, size, texture, colour, etc.) are used to categorise different vehicles from videos, using the image processing [6], this information is then transferred to the centralised controlling and monitoring system [7]. Fig. 1 shows the process that is followed in a traffic monitoring system based on videos, which has a camera mounted on the roadside, watching over the traffic scene. The camera works as a sensor device used to capture traffic videos. The captured video images are converted into a digital form using a processor. The images in their digital form are processed and examined using image processing techniques. The extracted information can then be used by any external user, such as a traffic control centre, for traffic monitoring and controlling, etc. Fig. 1Open in figure viewerPowerPoint Block diagram of vehicle detection and categorisation Image processing techniques are applied for a video-based traffic monitoring system in five stages: (i) image acquisition (CCTV) and digitisation, (ii) vehicle detection through background subtraction/foreground extraction, (iii) image segmentation, (iv) vehicle identification under different environmental conditions and classification, and (v) tracking. Fig. 2 shows that in the process of vehicle detection, the video image is processed, and the region where the vehicles are present is determined. This region of interest can be an alone pixel, a line of pixels, or a group of pixels [9]. Fig. 2Open in figure viewerPowerPoint Filtering acquisition, detection, and identification of vehicles [8] Throughout the traffic parameter extraction, they are obtained by comparing the vehicle detection status on the region of interest at the different instant of time frames. In a traffic surveillance system, background and illumination changes make vehicle detection a difficult task. The key problems are changing backgrounds (camera jitter, water rippling, and waving trees, etc.), illumination changes (gradual or sudden light changes), and climatic changes [10]. In these issues, the foreground mask by a Gaussian mixture model (GMM) is used, but there is a big chance of fault detection. To overcome this problem, various projects are being taken up to find better vehicle surveillance ways through which the chances of false detections can be reduced. This paper reviews the vehicle detection process used for surveillance and includes Section 2, which describes traffic surveillance cameras, Section 3 explains about moving vehicle detection techniques and the problems encountered during surveillance, Section 4 describes the classification of vehicles for effective traffic monitoring and controlling. Section 5 discusses the approach used to remove the haziness from image sequences. 2 Traffic surveillance cameras Traffic cameras are modern equipment that makes functional usage of video surveillance technology. Their footage can be seen on TV news during traffic reports. These cameras are implanted at the top of traffic signals or are mounted along the busy streets, or at the busy crossings of the highways. The cameras also helped in recording traffic patterns for future study and helped in monitoring and controlling the traffic [11]. Therefore, traffic cameras are an important device of video surveillance [12], which aims to monitor a specified area for safety and security reasons. Smart traffic cameras are commonly used in many road transportation systems, like surveillance, traffic management, driver assistance, and control access systems, security and law enforcement, etc. [13]. The research-based on vehicle detection methods through videos taken by camera started in the late 1970s [14]. In 1984, the University of Minnesota initiated a wide-area multiple video imaging detection system known as ‘Autoscope’ [15, 16]. It helps in reducing the installation and maintenance costs and led to the development of ITS [17, 18]. The earlier main emphasis was given to the detection of licensed plates only [19], but because of the rising demand, other ways of vehicle classification like colour, size, environment, etc. are being taken as a criterion of vehicle classification [20, 21]. Traffic cameras are generally placed at some common congestion point to monitor vehicle flow on roads, highways, major routes. The monitored information then passes to the traffic control centre for taking necessary action for any mishappening. Traffic cameras are also important in decisions concerning surveillance, future road construction, and development, apart from monitoring the accidents or major closures, jams on roads, etc. Generally, day-to-day traffic flows do not differ much, but in case of a road mishap or a closure, a traffic alert can be very helpful for a time-crunched commuter. Real-time videos from specified locations are collected through surveillance cameras using Pan Tilt Zoom, Gatso, Vector camera system, etc. and the images are then transmitted to the enforcement traffic control centre through the Internet [22-24]. Figs. 3a–d show some of the good cameras used in traffic surveillance. Fig. 3Open in figure viewerPowerPoint Traffic surveillance camera (a) Gatso, (b) Highway agency CCTV, (c) Specs, (d) Trafficmaster [25] One of the cameras named Gatso is based on a radar signal to record a vehicle's speed. The camera machine sends a radio signal to the vehicle, the signal is then returned by the vehicle. The period taken between the signal sent and received back from the vehicle is noted. Similarly, another signal is sent and received back, and the timing is also noted. The difference in the timings enables the ‘Gatso’ camera to know how quickly the vehicle is travelling between the two points. This camera takes several pictures if the vehicle crosses the specified speed limit. There is another camera named ‘Specs’ which can monitor four lanes simultaneously and is generally mounted on gantries. This type of camera is enabled with the Automatic Number Plate recognition system. The camera takes the photograph of each vehicle that passes it and the vehicle's data is then sent to another set of cameras, which is mounted at around 200–250 m on the road. The average time taken by the vehicle in travelling between these two set points can be calculated and helps in estimating the speed of the vehicle. These cameras can work night and day, and in all weather conditions-heat, wind, hazy, ice, and snow, as they are fitted with infra-red sensors that capture electromagnetic radiations [26, 27] of objects. The precautions that are taken while installing these traffic cameras are Cameras are mounted in such a way that the whole intersection area gets covered. The flashlight is installed for recordings at night. Cameras are positioned and calibrated to easily record the data of the license plate of the violating vehicles. Local law enforcement agencies are consulted to know the most bothersome areas/intersections. Cameras are placed in environment-controlled housings to protect from bad weather. Cameras are placed such that they have enough visibility and a decent view of all streets/lanes. 3 Vehicle detection The concept of vehicle detection is, to select the area of interest as to where a vehicle exists on the road in real-time. The system has very high complexity because of real-time acquiring of images, as the traffic scene's surroundings are continuously changing. The complexity of vehicle detection also depends upon light change, weather conditions, shadow, complex background interference, camera parameters, drift and occlusion of vehicles, etc. So, it is very difficult to detect vehicles under these changing conditions. For this, vehicles must be extracted from an image sequence. The ‘moving’ characteristic is used to differentiate a moving vehicle from a stationary background. The precision or accuracy of vehicle detection is of great significance for vehicle tracking movement expression and behaviour understanding and is the basis for subsequent processing. Generally, the algorithm used for vehicle detection is divided into four stages: problem domain, image or video acquisition, frame segmentation, and vehicle recognition [28, 29], as shown in Fig. 4. Fig. 4Open in figure viewerPowerPoint Generalised framework of vehicle detection and recognition Fig. 4 shows the framework of the vehicle detection based on video processing. By using the problem domain information in the video, the vehicle is identified from the image sequence. Then, with the use of embedded technology and smart camera, image acquisition takes place. Since the vehicle is in motion, acquiring images can be a bit complex. Therefore, researchers prepare an algorithm with great robustness to improve the adaptability and precision of detection, which helps in recognising the image [30]. In a broad sense, vehicle detection can be categorised based on [31]: (i) prior knowledge, (ii) motion, (iii) Wavelets (iv) machine learning (ML), (v) deep learning (DL). 3.1 Prior knowledge-based methods Various approaches to detect a vehicle have been made based on prior knowledge. They are symmetry, colour, vertical, or horizontal edge, shadow, wheels, and 3D model, etc. [31, 32]. 3.1.1 Symmetry For both horizontal and vertical directions, there must be a symmetry while taking the observation from the stationary camera. The method has not been considered as the finest method of detection due to noise. 3.1.2 Colour Colour segmentation, along with motion has been introduced to detect the moving vehicle. Colour segmentation generally uses split and merge algorithm, whereas vehicle segmentation uses frame differentiation. 3.1.3 Vertical and horizontal edges Conventional gradient-based and morphological edge detectors are the two kinds of edge detection methods. The conventional gradient-based uses a Sobel operator and generalised Hough transform. Morphological edge detection uses dilation and erosion operation to spot edges, and they are better than conventional edge detectors based on computational cost. Apart from the above-mentioned method, separable morphological edge detector was proposed by Siyal which depends less on edge directionality and increases the robustness of vehicle detection [33]. 3.1.4 Shadow As the region under any vehicle is darker as compared to other regions on the road, this fact becomes the basis of vehicle detection [34]. A threshold method centred on the characteristics of the Hue saturation value colour space is suggested by Cucchiara et al. [35]. Xua et al. [36] suggested another algorithm which suppresses the moving cast shadow of vehicle detection. 3.1.5 Wheel The methods proposed for vehicle detection in the daytime are not valid for nighttime. This fact was discussed by Yoneyama et al. [37] stated that methods applied to daytime do not work properly if it is applied for detection at nighttime. While considering the problem faced by Yoneyama et al., Iwasaki and Kurogi [38] defined a new unique method specifically for nighttime. The method calculates the distance between its wheels (front and rear), which helps in finding the size of the vehicle. Also, this method helped in finding the lane where the vehicle exists. However, for this method, the camera needs to be installed on the roadside rather than on top of any traffic signal. 3.1.6 3D model The vehicles’ 3D pose is mapped to a corresponding model description. Dealing with obstruction is another benefit of the 3D model [39]. Various approaches of 3D modelling have been proposed like graph matching [40], viewpoint consistency check [41], alignment-based 2D method, indexing and invariants [42], gradient-based method [42, 43], neural network, self-similarity [44], etc. In all the above-mentioned methods, 2D image features like line segments, points, and conic sections, are matched with corresponding 3D features of the position [39]. Karsten et al. suggested a method that works on the 3D restoration of a dynamic atmosphere with a fully regulated background for traffic movements [45]. Lou et al. [39] proposed another method that disintegrated the two motions, i.e. translation and rotation. Besides that, Ghosh and Bhanu [46] suggested a 3D model mapping arrangement where he classified the vehicles into various types like hatchback, sedan, and wagons. 3.2 Motion-based methods 3.2.1 Optical flow This method of vehicle detection is based on optical flow calculations, whereas the above-mentioned methods have used spatial features. Motion detection is one of the vital parameters in an ITS. There is a relative motion between the traffic scene and the sensor because of which the pixels on the image appear to be moving. This motion because of the vector field is called optical flow. By using optical flow autonomously any vehicle can be detected from the camera. However, its computation is very complex and subtle to noise. Meyer et al. [47] estimated the concept of a displacement vector field, which initialises a contour-based tracking process. Real-time analysis of video streams without any specialised hardware is not that easy. Barron's et al. [48] have mentioned a detailed discussion on optical flow in his work. 3.2.2 Background subtraction Background subtraction, also named as foreground detection, is one of the vital jobs by which many other applications like vehicle tracking, recognition, and irregularity detection can be realised. It is used for motion segmentation in a static scene [49]. For detection of motion, pixel by pixel image is subtracted from a background image, which is considered as a reference image. A certain threshold level is defined if the difference is above the threshold level, then the image considered as foreground. The formation of the background image is recognised as background modelling. Some morphological post-processing is done after the creation of foreground pixel maps like erosion, dilation, and closing to reduce noise and improve the detected region. The reference background image is updated from time-to-time and the image where the section of the road is shown with no vehicle serves as [49, 50]. There are various techniques by which background subtraction can be applied in terms of foreground detection. Heikkila and Silven [51] gave a very simple concept of background technique in which a pixel at a location (x, y) in the present image It is marked as foreground if is fulfilled; where a pre-defined threshold is denoted as Th. An impulse response filter updates the background image, as shown in the equation below: (1)where α is a constant that ranges between 0 and 1. The foreground pixel map construction is followed by morphological closing and the elimination of small-sized areas [51, 52]. With the help of this easy background model, pixels corresponding to foreground moving vehicles can be detected by thresholding any of the distance functions [53] (2) (3) (4)where R stands for red, G stands for green, and B stands for blue channels, and d0 is a measure operating on greyscale images. Another method is a GMM, which was given by Stauffer and Grimson [54] for real-time tracking. The algorithm depends upon the assumption that the background is made of animated textures like waves or wind-shaken trees. It represents every pixel with a mixture of K Gaussians. Here, the probability of occurrence of colour at a given pixel is (5) is the ith Gaussian model, and weight [53]. Zivkovic [55] suggested a better GMM using a recursive computation to continuously update the parameters of the model. This model is widely adopted by many researchers for the analysis of traffic [56-58] using a fixed number of Gaussians. Rapid changes in illumination pose difficulty in applying background subtraction in a real-world setting. Amali and co-authors [59] suggested a method using sample-based background subtraction called rapid background subtraction. The whole algorithm basically uses three different techniques. In the first technique, the first two consecutive frames are taken as a background model and this model can be updated after the threshold period. The next technique includes the classification of pixels in correspondence to the background pixel model and the shadow detection method. The last technique involves updating of the background pixel model by random pixel locations. By using this technique, accuracy and efficiency can be greatly increased. For online data, the subspace learning technique has been used for the background model. Oliver et al. [60] suggested a method using the concept of principal component analysis (PCA) depending upon probability distribution function. In recent times, Bouwmans et al. [61] introduced a detailed review of robust PCA-PCP based methods for testing and ranking the existing algorithms for foreground detection. The study here is driven by the need for robust vehicle detection and a classification algorithm that can be used in a traffic monitoring system to deal with such changes. Because of critical situations in video surveillance, some researchers introduced the concept of fuzzy logic for background subtraction. 3.3 Wavelet-based methods Since the last decade, the wavelet transform has become one of the vital tools for signal processing applications. The vehicle motion is taken in 3D spatial–temporal data from the image scene [62-64], and this is possible by wavelet transform. The benefit is very low computational complexity and an easy application in both spatial and temporal directions. Though, these methods depend upon noise and on variations of timings of movements. Wang and Chen [63] proposed a new algorithm using the spatial–temporal wavelet transform on videos to obtain motion information of vehicles. A fusion technique of wavelet transformation and colour information features for automatic vehicle identification was proposed by Yin et al. [65]. 3.4 Machine learning ML [66] is an interesting topic that helps in understanding how a machine updates the code on its own. ML is a subset of artificial intelligence (AI) as shown in Fig. 5, which provides computers with the ability to learn without being absolutely programmed. The meaning of this is that once the machine is programmed, it will not change its own code even when a new problem is encountered. The objective of ML is to focus on the development of computer programs that change when exposed to new data. It changes according to a situation where the machine does not change its code. This is a self-learning concept; it learns whatever it has acquired from the past scenario, situations, experiences, etc. So, the machine comes up with new solutions by using the concept of AI. Fig. 5Open in figure viewerPowerPoint DL is a subset of ML which is again the subset of AI There are different approaches to solve any problem, and so there are different ways a machine can learn. Depending on these approaches, there are three ways a machine can learn [67, 68]. The types of ML ways or algorithms are (a) Supervised learning (task driven-regression/classification). (b) Unsupervised learning (data driven-clustering/association) (c) Reinforcement learning (learns to react to an environment). 3.4.1 Supervised learning In this algorithm, continuous monitoring is done to make it understandable and it practices a known dataset called the training dataset to make estimates. The training dataset contains input data and response values. As of it, the supervised learning algorithm seeks out to build a prototype that can make estimations of the response values for a new dataset. 3.4.2 Unsupervised learning In the case of unsupervised learning, the algorithm draws interferences from datasets consisting of input data without categorised reactions. This is simply a self-learning algorithm where a machine takes its own decisions. 3.4.3 Reinforcement learning In this algorithm, the machine can take its own decision depending upon its past experiences. It is totally relying on behaviourist psychology; whose outcomes are to expand the numerical reward. The learner is not advised which move to make; however, rather should find which activity will yield the greatest reward. A major limitation of ML is that it cannot handle multiple dimensional data. Also, complex AI problems cannot be handled by ML. So, DL concept came into existence, which handled high-dimensional data and focused on the right features by themselves, requiring slight supervision from the system analyst. Fig. 6 shows a graph which shows the relation between accuracy and size of training data, and it is found that DL shows better performance with an increase in the size of training data. Fig. 6Open in figure viewerPowerPoint Graph plotted between accuracy (performance) and size of training data 3.5 Deep learning DL [69] is processed through neural networks, which is a multi-layered neural network learning algorithm. Every deep neural network (DNN) consists of three layers, the input layer, the hidden layer, and the output layer. In between the input and output layers, there can be n-multiple hidden layers, and processing of ‘n’ hidden layers is possible because of high resources available these days. Fig. 7 represents a generalised model of DNN where two hidden layers are taken in between input and output layers. Fig. 7Open in figure viewerPowerPoint Deep network with two hidden layers in between input and output layers [68] Since the last decade, DNNs have prompted an arrangement of hops forward on an assortment of assignments, e.g. computer vision, machine interpretation, and voice acknowledgment, and so forth. One of the basic parts achieving these achievements comes about is convolutional neural networks (CNN) [70]. Krizhevsky et al. [71] proposed a concept called AlexNet after which, CNNs have exhibited prevalent execution for picture classification differentiated and conventional ‘shallow learning’ strategies, and have similarly been adequately associated for object detection [72], video classification [73] and segmentation [24], etc. These triumphs motivated a new line of research that focused on formed higher execution CNNs, and the execution of framework models has been improved by utilising further and more broad structures. Simonyan and Zisserman [74] proposed VGGNet, empowering the examination on the use of significant outline in PC vision. Szegedy et al. [75] showed GoogLeNet which contains Inception modules, setting the new best in class for the ImageNet Challenge. To deal with this deprivation issue, He et al. [76] showed a waiting learning framework named ResNet that can get ready liberally more significant frameworks than those used in advance. DL models are quite very interesting and proved to be an outstanding solution for the detection and classification of vehicles, so we briefly overview some of the models. 3.5.1 Convolutional neural networks It is a deep discriminative network, usually looks like that of the ordinary artificial neural network, as mentioned in Fig. 7, where transforms have been done through hidden layers. The deep architecture of CNN is explained in [66] (refer to Fig. 3c) that discussed the arrangements with a multi-channelled image as the participation input [77]. To manage this complicated input, the CNN comprises of a few layers of convolutions with non-linear actuation capacities to figure the output. As an outcome, the CNN contains limited associations whereby every area of the input is associated with a neuron in the output. Each layer applies a distinctive channel and joins their outcomes. What is more, the CNN comprises of the pooling layers for subsampling. During the preparation stage, a CNN consequently takes in the estimations of its channels considering the given assignment. In its first layer, CNN may figure out how to distinguish the edges from the raw pixels. At that point, in the second layer, the CNN utilises the edges to distinguish straightforward shapes. In this form, in the consequent higher layers, by utilising these shapes, CNN might have the capacity to learn more elevated amount highlights like facial shapes of the animal present in the image. In the last layer, a classifier is utilised to exploit these abnormal state highlights. Profound CNN's have ended up being very fruitful in learning task highlights, which have given much-enhanced outcomes, especially on computer vision undertakings, conversely with contemporary ML procedures. Normally, CNN's are prepared by utilising supervised learning strategies whereby an expansive number of input–output sets is important. Notwithstanding, acquiring a considerably huge data set has been a key test in applying CNN's for settling another assignment. Likewise, CNN's have been accounted for to be widely utilised as a part of unsupervised techniques [71, 78, 79]. 3.5.2 Deep Boltzmann machine (DBM) To understand the DBMs, a basic knowledge of basic Boltzmann machines and the restricted Boltzmann machines [80] is necessary. A network of binary stochastic units with a defined ‘energy’ for the network is a Boltzmann machine. A representation of the BM architecture is shown in Fig. 8a. Although learning is ineffective in a shallow BM, it can be made relatively efficient through a restricted Boltzmann machine (RBM), as it does not allow the networks among units in the same layer, as mentioned in Fig. 8b. Fig. 8Open in figure viewerPowerPoint Machine neural architecture (a) Graphical representation of each undirected edge showingdependency for Boltzmann machine architecture [66],(b) Restricted Boltzmannmachine architecture with stochastic neurons [66] After train