Deep learning serves traffic safety analysis: A forward‐looking review

IET Intelligent Transport SystemsVolume 17, Issue 1 p. 22-71 REVIEWOpen Access Deep learning serves traffic safety analysis: A forward-looking review Abolfazl Razi, Corresponding Author Abolfazl Razi arazi@clemson.edu School of Computing, Clemson University, Clemson, SC, U. S. A. Correspondence Abolfazl Razi, School of Computing, Clemson University, Clemson, SC 29634, U. S. A. Email: arazi@clemson.eduSearch for more papers by this authorXiwen Chen, Xiwen Chen orcid.org/0000-0002-8006-4383 School of Computing, Clemson University, Clemson, SC, U. S. A.Search for more papers by this authorHuayu Li, Huayu Li Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ, U. S. A.Search for more papers by this authorHao Wang, Hao Wang School of Computing, Clemson University, Clemson, SC, U. S. A.Search for more papers by this authorBrendan Russo, Brendan Russo Department of Civil Engineering, Northern Arizona University, Flagstaff, AZ, U. S. A.Search for more papers by this authorYan Chen, Yan Chen The Polytechnic School, Arizona State University, Mesa, AZ, U. S. A.Search for more papers by this authorHongbin Yu, Hongbin Yu School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, U. S. A.Search for more papers by this author Abolfazl Razi, Corresponding Author Abolfazl Razi arazi@clemson.edu School of Computing, Clemson University, Clemson, SC, U. S. A. Correspondence Abolfazl Razi, School of Computing, Clemson University, Clemson, SC 29634, U. S. A. Email: arazi@clemson.eduSearch for more papers by this authorXiwen Chen, Xiwen Chen orcid.org/0000-0002-8006-4383 School of Computing, Clemson University, Clemson, SC, U. S. A.Search for more papers by this authorHuayu Li, Huayu Li Department of Electrical and Computer Engineering, University of Arizona, Tucson, AZ, U. S. A.Search for more papers by this authorHao Wang, Hao Wang School of Computing, Clemson University, Clemson, SC, U. S. A.Search for more papers by this authorBrendan Russo, Brendan Russo Department of Civil Engineering, Northern Arizona University, Flagstaff, AZ, U. S. A.Search for more papers by this authorYan Chen, Yan Chen The Polytechnic School, Arizona State University, Mesa, AZ, U. S. A.Search for more papers by this authorHongbin Yu, Hongbin Yu School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ, U. S. A.Search for more papers by this author First published: 27 August 2022 https://doi.org/10.1049/itr2.12257 Abolfazl Razi and Xiwen Chen contributed equally to this work. AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract This paper explores deep learning (DL) methods that are used or have the potential to be used for traffic video analysis, emphasising driving safety for both autonomous vehicles and human-operated vehicles. A typical processing pipeline is presented, which can be used to understand and interpret traffic videos by extracting operational safety metrics and providing general hints and guidelines to improve traffic safety. This processing framework includes several steps, including video enhancement, video stabilisation, semantic and incident segmentation, object detection and classification, trajectory extraction, speed estimation, event analysis, modelling, and anomaly detection. The main goal is to guide traffic analysts to develop their own custom-built processing frameworks by selecting the best choices for each step and offering new designs for the lacking modules by providing a comparative analysis of the most successful conventional and DL-based algorithms proposed for each step. Existing open-source tools and public datasets that can help train DL models are also reviewed. To be more specific, exemplary traffic problems are reviewed and required steps are mentioned for each problem. Besides, connections to the closely related research areas of drivers' cognition evaluation, crowd-sourcing-based monitoring systems, edge computing in roadside infrastructures, automated driving systems-equipped vehicles are investigated, and the missing gaps are highlighted. Finally, commercial implementations of traffic monitoring systems, their future outlook, and open problems and remaining challenges for widespread use of such systems are reviewed. 1 INTRODUCTION Despite recent advances in vehicles' operational safety features, computer-assisted control, and technology-based traffic management systems, traffic safety remains one of the main challenges in today's life. Every year, traffic crashes account for 20–50 million causalities and 1.35 million fatalities worldwide, making it one of the top-10 causes of death. Indeed, traffic crashes is the leading cause of death for people aged 5 to 29 [1]. Using computational intelligence and computer tools for enhancing traffic safety has gained a lot of attention in recent years. Mainstream technological trends include (i) implementing vehicle safety features such as forward collision warning, blind-spot detection, lane departure warning, backup camera, and autonomous emergency braking [2], (ii) simulation-based road infrastructure design such as Site3D, RoadEng [3], and OpenRoads Designer [4], and (iii) intelligent traffic flow management systems such as global positioning system (GPS)-based navigation tools. An example of the last category is Google's road-user interpretive software that can infer the common road behaviour of other drivers that allows engine control units (ECUs) to make better route decisions [5]. Many car manufacturers have developed artificial intelligence (AI)-based safety features [5]. Tesla, Audi, and BWM have already developed sensor-based and vision-based perception systems to help the drivers judge road conditions more accurately and use partial autonomy [6, 7]. Particularly, the recent advances in deep learning (DL) methods for video processing backed by low-cost and high-speed computational platforms such as graphics/tensor processing units (GPUs/TPUs) have accelerated the pace of developing AI-based features both at vehicle and infrastructure levels [8]. Commercial transportation has witnessed unprecedented growth in utilising autonomous driving systems, in recent years. Waymo launched a self-driving taxi service in Phoenix, Arizona in 2020 [9]. Almost the same time, General Motors (GM) presented their prototype for autonomous buses [10]. Meanwhile, Amazon revealed their future public transportation plan, which offers passengers-only compact autonomous shuttles [11]. Starship robots [12] and Nuro's autonomous delivery bots [13] are other examples of AI-based commercial traffic platforms. The autonomous semi-truck concept, independently presented by Tesla and Waymo [14, 15], brings tons of benefits to logistic and cold chain monitoring, building a solid fundamental for industry traffic. In the industry domain, camera-based traffic monitoring systems have been broadly used to detect incidents and apply congestion control [16, 17]. Sensor-based adaptive traffic light control systems are used to reduce traffic jams [18-20]. Modern systems use vehicle-To-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications to enhance the overall driving safety. For instance, roadside unit (RSU)-based radar systems with real-time data analysis are used to monitor pedestrians on the road and minimise crash rates by sending alerts to connected vehicles [21]. These technologies have optimised existing infrastructures and greatly increased traffic safety. 1.1 Architectural view Traffic safety analysis can be viewed as a modular and multi-faceted problem that involves many aspects. As shown in Figures 1 and 2, the overall analysis platform can be viewed as a software pipeline where the collected information undergoes different processing steps until it is translated to navigation commands, advisory messages, or overall guidelines for improving traffic safety. FIGURE 1Open in figure viewerPowerPoint A framework of computer vision (CV)-based traffic safety analysis pipeline FIGURE 2Open in figure viewerPowerPoint The data sources and computing resources used in video-based traffic analysis This paper provides a comprehensive review of the popular methods, tools, software packages, and datasets developed by the scientific community for each vision-based sub-problem. We also highlight the open problems and future challenges on each frontier. Our main emphasis is on the role of vision-based DL methods in enhancing traffic flow (e.g. improving efficiency) and mitigating traffic safety risks. In contrast to the primary trend of processing individual events from an involved vehicle's perspective, we consider traffic safety at both the vehicle-level and network-level by processing videos captured by an external observer. The main information source is captured video by vehicle-mounted cameras and roadside cameras, but we also will review other sensor information that can be used for traffic management. We believe that enabling advanced traffic safety analysis and monitoring platforms will play a crucial role in future smart cities. Figure 3 compares video-analysis from two different perspectives, from an external point of view as well as an internal node's perspective. FIGURE 3Open in figure viewerPowerPoint The example of perspectives: (a) internal perspective and (b) external perspective It is noteworthy that several review papers have been published to review methods and tools used for video-based traffic analysis. A summary of these papers is provided in Table 1. However, most review papers have limitations in certain aspects. TABLE 1. Summary of related review papers. The paper with "*" means although this paper is related but outdated. L √ $L\surd$ denotes a topic is covered in fewer details Paper [22] Hu et al. [23] Mozaffari et al. [24] Grigorescu et al. [25] Yurtsever et al. [26] Janai et al. [27] Badue et al. [28] kumaran et al. [29] Wang et al. [30] Nguyen et al. *[31] shirazi et al. *[32] Mukhtar et al. *[33] Morris et al. Ours Year 2020 2020 2020 2020 2020 2020 2019 2019 2018 2016 2015 2013 Human-driven vehicle √ √ √ √ √ √ √ √ √ √ AVs √ √ √ √ √ √ √ L √ $L\surd$ √ Safety assessment analysis √ √ √ √ √ √ √ √ CV-based method √ L √ $L\surd$ √ √ √ √ √ √ L √ $L\surd$ √ √ √ √ Deep learning method L √ $L\surd$ √ √ √ √ L √ $L\surd$ √ √ L √ $L\surd$ √ Sensors √ √ √ √ √ √ √ √ Datasets √ √ √ √ √ √ √ Network analysis √ √ √ √ √ √ √ Vehicular edge computing √ Behavioural & driver cognition √ Some survey papers (e.g. refs. [25-27, 29, 30]) focused merely on solving traffic-related tasks (such as perception) while not covering safety assessment methods. Some other papers (e.g. refs. [22, 27]) do not provide a comprehensive summary of DL methods, which recently has become the dominant approach in both industry and academia research. Other papers (e.g. refs. [25-27]), which review driving techniques for vehicles equipped with automated driving systems (ADS), keep their attention solely on the DL methods developed for AVs while not investigating the practicality of these methods on human-driven vehicles, which still are the most widely used vehicles. A few papers (e.g. refs. [22, 28]) put their primary focus on DL methods but without special emphasis on vision-related tasks particularly useful for traffic analysis. A different set of papers (e.g. ref. [23]) limit their analysis to the perspective of internal nodes, also known as the first-person perspective. Although computer vision (CV)-based tasks are covered by several papers [29, 30], they do not provide sufficient details on this subject from different perspectives. Most of the papers, including refs. [24, 26-28, 30], although very informative, only review the DL algorithms that are used or can be used for traffic analysis and do not provide any sort of comparative analysis, which does not help choosing the right method for different real-world traffic problems. The relevant surveys we include are from 2013 and after; however, we want to mention some papers, including refs. [31-33] are outdated, but highly related to our survey with the topics of CV-based traffic safety analysis. Table 2 provides more specific details on the main focus and the shortcoming of the recently published survey papers. TABLE 2. Specific details on the shortcomings of the most recent survey papers in CV-based traffic analysis Survey Content Drawback Perspective Application [22](2020) Review existing research works on traffic conflicts based on perception technologies and communication technologies. Discusses very limited DL methods (covers less than 10 DL methods); covers in fewer details the CV tasks (including detection, trajectory extraction, and video analysis). Internal AVs [23](2020) Reviews DL approaches for Trajectory prediction based on the input representation, output type, and prediction method. Because of the limited works focusing on CV data (e.g. BEV), it only discusses 6 (CNN)+4 (CNN+RNN) methods. External Avs; traffic analysis [24](2020) Reviews DL approaches (mainly as CNN, RNN, deep reinforcement learning) for AVs; partially discusses safety assessment. shallowly covers sensor information; less details are provided for performance comparison (only covers 10 detection methods on a general dataset and 4 semantic segmentation methods on CityScapes). Internal AVs [25](2020) Reviews practical challenges and solutions for Avs; the main tasks involve localisation, mapping, perception, planning, and human-machine interfaces; emphasises the whole driving system architecture. Excludes video pre-processing methods (e.g. video stabilisation); provides little details on the performance analysis (9 methods on ImageNet and 7 methods on KITTI 3D). Internal AVs [26](2020) Reviews CV-based approaches for AVs; discusses open problems and current challenges. Lacks the safety assessment analysis; does not give detailed performance comparisons. Internal AVs [27](2020) Reviews technologies for the perception and decision-making system of Avs; discusses the AV industry milestones. Covers limited DL methods (sign and light detection, pavement marking detection, MOT, excluding segmentation methods); does not give detailed performance comparisons. Internal AVs [28](2019) Reviews approaches for anomaly detection for surveillance videos. Only includes tasks related to anomaly detection; does not include other CV tasks; does not provide comprehensive performance analysis. External Traffic analysis [29](2019) Reviews DL approaches for multiple traffic applications, including time series prediction, classification, and optimisation. Covers a few CV-based problems (including traffic sign recognition, vehicle detection, and pedestrian detection, excluding MOT and segmentation); offers a limited comparative analysis (including only the traffic sign and traffic flow prediction). External AVs; traffic analysis [30](2018) Reviews DL approaches for processing traffic data. Covers a few CV-based problems (including vehicle detection, perception on Avs etc,.) without sufficient detail. External Traffic analysis In addition to covering newly published CV-based methods, our paper covers the shortcomings of previous surveys and considers the video-based driving safety problem from different perspectives. More specifically, we list exemplary problems in video-based driving safety analysis; we review requirements and challenges from an external observer's perspective; we review datasets and important industrial developments; we make connections to closely related areas of utilising crowd-sourcing, and edge and cloud computing for bulk processing; we highlight connections to behavioural science, insurance industry, and other policy maker entities. The rest of this paper is organised as follows. Section 2 reviews mainstream lines of DL methods used for vision-based driving safety analysis. In Section 3, data acquisition equipment and methods are reviewed. Section 4 includes discussions about different stages of video pre-processing for safety analysis by highlighting historical milestones, successful methods, current trends, and remaining challenges for each category. Section 5 reviews DL methods for video processing with application to traffic safety analysis. A short list of sample problems in driving safety analysis is provided in Section 6. Section 7 reviews recent trends in deep learning that can influence the field if video-based driving safety analysis. A list of commonly used datasets with applications to traffic monitoring and traffic safety analysis is provided in Section 8. Section 9 list a set of key safety metrics used for assessing the potential for crash occurrence and crash severity. Section 10 provides different key points such as connection to other fields and potential applications of safety analysis methods. A roadmap of this technology is offered in Section 11. Remaining challenges and issues are discussed in Section 12. 2 DEEP LEARNING METHODS The common core of vision-based driving safety analyses is using deep learning methods for image/video processing. We skip the details of DL methods here for the sake of brevity and refer the interested reader to previous reviews [34, 35]. It is noteworthy that most of the recent developments in DL have been driven by two applications, computer vision (CV), and natural language processing (NLP), as two key representatives of visual and sequential processing problems. Most elegantly designed DL platforms utilise multilayer perceptron (MLP), convolutional neural networks (CNN), recurrent neural networks (RNN), and transformers as their building blocks. Table 3 provides a compact and informative comparison of these methods. We will provide a more detailed analysis of custom-built DL methods used in the context of driving video analysis and safety control. TABLE 3. A brief comparison among typical DL architectures Architecture MLP CNN RNN/LSTM Transformer Pros Straightforward to design Appropriate for high-dimensional data Learns features with locality Shift-invariance Fewer parameters Powerful on CV tasks Appropriate for sequential data Can learn long-term dependencies Processes sequences in parallel Appropriate for long sequences Captures long-term dependencies Enables self-attention mechanism Powerful on both CV and NLP tasks Cons Many parameters (dense connections) limited capability Do not encode the position of object Gradient vanishing in deep architectures (solved by ResNet and auxiliary output) Weak on long sequential data Hard to train for gradient issues Handles only serially fed data Powerless on extreme long-term dependencies Many parameters (long training time) Does not consider locality Over complicated for short sequences Inductive bias Weak Locality/spatial invariance Sequentiality/time invariance Weak Training a network is not always straightforward and can be impacted by many factors, such as hyper-parameters (mini-batch size and learning rate). Due to the time sensitivity of safety-related applications, fast training is a key requirement. Table 4 lists some common tricks (e.g. dropout, regularisation, easy-derivative rectified linear unit (ReLU) and leaky ReLU activation function, data augmentation, and soft labelling) to accelerate the training process and boost the recognition performance. For example, word embedding uses low-dimensional space to present high-dimensional data allowing words with similar meanings to be closer in the low-dimensional space. This often outperforms the one-hot encoding. Such tricks can be translated from the NLP context to driving behaviour analysis. An exemplary scenario would be the personalised driving anomaly detection. Suppose there are 20 collected features for each driver. If there are 100 drivers, by one-hot encoding, the input dimensions of data should be 100 + 20 = 120 $100+20=120$ per time point, which leads to the curse of dimensionality. In fact, difference among different drivers (or driving styles) can be well represented by as few as 20 dimensions. TABLE 4. Common tricks to accelerate DL training Name Description Data normalisation Stabilise the training Xavier/Kaiming Initi. Initialise based on the magnitudes of signals Batch/layer/instance normalisation Local response normalisation regularise avoid gradient vanishing Dropout Reduce number of parameters avoid overfitting L1/L2 regularisation Avoid overfitting Weight decay Similar to L2 regularisation but different in some optimisers (e.g. Adam) Leaky Relu Avoid dead neurons (as for Relu) Word embedding Manage high-space categorical data Soft labelling State the relations between different classes; Improve generalisation Data augmentation Improve the variety of data Improve generalisation Residual connection Avoid gradient vanishing Allow building very deep networks 1 × 1 Conv Combine channels Gradient clipping Avoid gradient explosion 3 DATA ACQUISITION In this section, we investigate the role of data acquisition in developing safety-related algorithms. First, we review data modalities and hardware used for information acquisition. Next, we list the most commonly used datasets for testing the developed algorithms. Most traffic analysis platforms rely on data collected by different types of sensors, including cameras, global positioning systems (GPS), radio detection and ranging (Radar), and light detection and ranging (LiDAR). These sensors (except GPS) can be used in vehicles or on external observer systems such as roadside infrastructures, drone-based aerial platforms etc. The following is a short description of sensors followed by a summary provided in Table 5. TABLE 5. Comparison of sensors. Here "√" means this kind of measure has been implemented or available to complete Measures type Sensor type Vehicle count Speed/distance Estimation Vehicle classification Pedestrian detection Road feature Cost Shape modelling Robust to bad environment Camera (visible) √ Hard √ √ √ Low 2D Low Camera (nfrared) √ Hard √ √ √ Low 2D High LiDAR √ Easy √ √ √ High 3D Middle Radar √ Easy √ Moderate weak High UAV camera √ Moderate √ √ √ Low 2D Low 3.1 Video cameras Video cameras are the most widely used means of information collection. Modern cars are heavily equipped with cameras in various parts to capture imagery for processing. Using thermographic cameras for night vision has become more common than ever. Camera feeds can be used by the driver (e.g. backup camera) to minimise safety risks, or by the control computer in an AV for automated driving. The imagery can also be used by more advanced AI platforms for driver's cognition assessment in real-time mode (e.g. driver drowsiness assessment [36], and distraction awareness [37]). Volume collections of imagery can be stored for further analysis for transportation infrastructure revisions. 3.2 GPS GPS systems provide an accurate position of the vehicle by communicating to GPS satellites. The accuracy of GPS depends on different factors, including satellite geometry, signal blockage, atmospheric conditions, and receiver design features/quality. The global average user range error (URE) of GPS can be as low as 2.3 ft.[38]. With recent advances in 5G wireless networking, even more precise positioning is available for vehicles. For instance, Verizon launched its hyper precise location (HPL) using real-time kinematics (RTK), with an unprecedented accuracy of 1–2 cm[39]. 3.3 Radar Radar operates based on Doppler frequency shift in the reflected wave [40]. Radar units are often installed on modern cars, roadside poles, police vehicles, and portable speedometer guns to measure the absolute or the relative speed of other vehicles. Radars are low-cost and relatively robust devices appropriate for different weather conditions and illumination intensities. Some radar units may have a narrow field of view (FoV) but are capable of long-range detection [31]. A traditional radar is a single source-detector, which does not have the spatial resolution required for precise scene/environment description; however, in recent years, imaging radars have been developed by adopting multiple-input, multiple-output (MIMO), and radar-on-chip technologies. Although such advanced technology is still expensive, attempts are made to reduce the cost of broader adoption[41]. 3.4 LiDAR LiDAR uses laser beam reflection to enable accurate positioning down to centimetre's scale [42]. The 3D scanning of multiple laser beams provides a 3D point cloud image (3D map) of the surrounding obstacles with accuracy much higher than regular radars [43]. LiDARs send out a near-infrared laser beam and detect reflections from the object; thus, it can still operate in dark conditions, in contrast to visual sensors. Its use is less common than radars for a few reasons, including its higher cost, relatively sparse spatial resolution, especially for a limited number of scanning laser beams, extremely narrow FoV, and computational complexity of 360° scanning, noting that low-complexity point cloud methods are still under development. Solid-state LiDARs have been developed to reduce the cost, while enhancing the spatial resolution[44]. 3.5 Drones Drones are commonly used nowadays to implement aerial monitoring systems. Most external observer systems utilise sensors in roadside infrastructures. However, the use of drones, also known as unmanned aerial vehicles (UAVs), is gaining more attention in different applications to enable fast, low-cost, and on-demand monitoring [45], and traffic analysis is not an exception. Particularly, drones can provide top-view and clearer occlusion-free images of the traffic flow when needed [46]. A network of cooperative drones can collectively cover relatively large areas [47]. Modern drones may be equipped with advanced sensing platforms, high-resolution cameras, and more importantly advanced features such as learning-based image processing, AI-based autonomous control, collision avoidance and auto lading, auto-calibration, real-time transmission, object tracking, and image restoration. The key challenges of drones are their limited payload, flight time, and communication range under study by several research teams. Aerial images pose new challenges to the research community, such as tackling image stabilisation, small object recognition, and developing lightweight ML algorithms customised for top-view images. 3.6 Other sensors In addition to the aforementioned broadly-used sensors, there exist some custom-built advanced sensors that can be used for traffic monitoring and driving safety analysis. Passive infrared sensors[48], inductive loops[49], and piezoelectric sensors[50] can enable measuring basic flow parameters such as vehicle count, speed, and flow volume, noting that piezoelectric sensors are still used for weigh-in-motion measurement. Environmental sensor stations (ESS)[51] on the roadway can collect atmospheric data, including air temperature and humidity, visibility distance, wind speed, wind direction etc., as side parameters to be used for traffic analysis. We recognise that the properties of sensors are varied. For example, visual cameras require clean and high-visible environments and offer a richer set of information in terms of colour space and visual geometry, hence are more appropriate for detection and classification applications. However, interpreting imagery may require more computation powers. LiDARs, on the other hand, provide more accurate precision for object detection and depth and speed estimation but are expensive and less energy efficient. Some manufacturers like Tesla prefer pure vision-based perception [52]. However, fusing multiple sensors can be advantageous from the safety perspective and is adopted by more vendors, as discussed in ref. [53]. 4 VIDEO AND IMAGE PRE-PROCESSING In this section, we review different stages of a typical video-based traffic analysis framework and highlight key developments, historical milestones, current trends, and existing challenges. 4.1 Super resolution The video and image super-resolution aims to reconstruct a higher resolution (HR) result from a low resolution (LR) observation. Super-resolution is a typical stage in image pre-processing and can be applied to traffic imagery to enhance the performance of the subsequent learning tasks, such as vehicle classification and license plate detection. One popular supervised learning method is the DL-based single image super-resolution (SISR) method which creates a mapping between the low and high-resolution images by training a deep CNN. Most of the existing learning-based SISR methods are trained and evaluated using simulated datasets [54-57], where the LR images are generated by applying a hand-crafted degradation process into the HR samples. For instance, one may apply bi-cubic down-sampling to the original HR samples to obtain LR results. Recently, more advanced SISR [58] methods are developed for real-world applications with unknown and more complicated degradation processes, which can be used as a benchmark method for traffic image analys