# Risk Assessment and Enhancement Suggestions for Automated Driving Systems through Examining Testing Collision and Disengagement Reports

Authors: Kuo-Wei Wu; Wen-Fang Wu; Chung-Chih Liao; Wei-Ann Lin
Journal: Journal of Advanced Transportation
 (2023)

---

## Abstract

The California Department of Motor Vehicles (DMV) reports, including disengagement and collision reports, provide information on each accident or disengagement activity for on-road testing of autonomous driving systems (ADSs) and autonomous vehicles (AVs). Unfortunately, current DMV reports have been misleading in relation to many key details, making it challenging for readers of those reports to discern the events’ root causes and interrelationships. Therefore, appropriate systematic classification methods and principles need to be adopted. We follow an identification method similar to fault tree analysis (FTA) with the help of the driving reliability and error analysis method (DREAM 3.0) and the Haddon matrix to find the potential key accident factors from all disengagement data. We also conduct ADS risk assessments of potential disengagements and genuine accidents classified by traditional accident types. In addition, the automated driving system is composed of various software modules, and a classification method that is suitable from the standpoint of ADS software developers is developed in this paper. Next, we sort out the characteristics of the most frequent accidents based on the risk assessment results. Finally, we propose a workable risk reduction solution according to the characteristics of accidents.


---

## Body

## 1. Introduction

Autonomous driving systems (ADSs) and autonomous vehicles (AVs) have flourished. One reason for their rapid growth is that ADSs may enhance driving safety. The National Highway Traffic Safety Administration (NHTSA) is the official department responsible for driving safety in the US. Its official report has stated that 94% of serious car accidents in the US involve human-driver-related factors [1, 2]. These factors include dangerous driving, distraction, speeding, and illegal driving. After all, such resulting traffic accidents may consume significant medical resources and repair costs. Advanced ADS safety technology could help avoid up to 79% of traffic accidents, including distracted driving, insufficient reaction time, the inability to maintain a safe distance, drunkenness, physical discomfort, and reckless driving. In recent years, there has been an increasing expectation that a well-designed ADS/AV could reduce or entirely avoid these accidents. To achieve a well-designed AV or ADS product, ADS safety is critical and deserves in-depth study.So far, most current research related to ADS safety has focused on essential driverless functions, such as lane departure suppression, collision prevention between forwarding and lane changes, autonomous cruise driving, and driverless parking. Other related studies have focused on driving safety concerns under normal and abnormal conditions, such as driving in bad weather or special road conditions, sudden obstacles, hazard avoidance, blind-spot monitoring, fatigued driving prevention, and abnormal behavior of other road users [3–7]. However, for the public, the results of a fully government-implemented ADS safety test would be more impartial and credible than individual announcements by ADS developers or media reports that may be subjective and misleading. ADS developers must also ensure that their products can gain public trust before entering the market.For these reasons, in 2015, the California Department of Motor Vehicles (DMV) became the first official agency in the world to develop empirical ADS road testing regulations. As of April 4, 2022, there were three different types of permit holders in its Autonomous Vehicle Testing (AVT) program: 48 permit holders with a driver, seven permit holders without drivers, and three deployment permit holders. These permit holders included nearly all major automakers and ADS developers. AVT permit holders are required to submit mandatory collision and disengagement reports each year in this program; otherwise, they may have their permits suspended by the DMV. As a result, the official California DMV AVT collision and disengagement data, being that of the largest and most credible ADS regulator in the world, are able to provide the latest ADS safety test results as well as a glimpse into state-of-the-art ADS technology.Most of the existing safety research literature based on the California DMV AVT program has derived the ADS safety factors or their root causes through proportions by grouping and calculating their disengagement or collision data directly according to the authors’ subjective induction [8–18]. However, these practices may contain some flaws. First, these studies did not examine the characteristics and significance that the data represented or verify whether they were suitable for pooling. For example, these ADS disengagement events are near-miss incidents without real crashes. We can interpret them as drivers’ risk averse reactions or ADS-detecting hazards early. The risk factors for disengagement events may represent “potential” ADS risk factors. Whenever active disengagement (ADE) or passive disengagement (PDE) occurs, a human driver can immediately take over. They may avoid authentic collisions as long as there is sufficient reaction time and relevant skills for human drivers that allow the driver to perform correct reactions. Because of the rapid development of ADS technology, incorporating outdated disengagement data that have never occurred may seriously distort the proportion of existing potential fault data. The results may also not reflect the latest progress in ADS technology that the disengagement data represent. However, the ADS collision data signify that crashes have indeed occurred and can be combined.Second, the California DMV AVT reports, including collision and disengagement reports, often contain subjective perceptions and overly succinct descriptions from human drivers. In some detailed text descriptions, critical analytical judgment factors are often lacking. When writing event descriptions, human drivers sometimes oversimplify, deliberately omit, or inadvertently ignore specific points. Moreover, human drivers may not have sufficient knowledge to determine all root causes of faults because they do not comprehend how the ADS system works in the background. Therefore, it is essential to adopt an impartial, systematic classification method to infer ADS risk factors by referring to all information about events. The process may reference relevant factors such as vehicle damage locations, environment and weather circumstances, and other related factors. Then, the results obtained may become more meaningful.Third, we have summarized eight potential ADS risk factors based on the current research literature on California DMV AVT safety. They are (1) hardware/advanced driver assistance system (ADAS) issues, (2) system and software discrepancies, (3) planning issues, (4) perception issues, (5) environmental conditions, (6) undesired behavior of other road users, (7) control discrepancies, and (8) others [8–18]. Such classification results are difficult for ADS software developers to judge and find the issue locations of their software program modules. After all, the ADS system is an integrated system dominated by software program modules and combined with related hardware for interconnection and operation. It would be of great help if the results could be sorted into software modules.This study thus attempts to improve the abovementioned shortcomings, conduct a risk analysis of California DMV disengagement and collision events, and determine critical risk factors. Finally, based on the results of the risk analysis, probing the characteristics of these risk factors provides a novel and workable solution to reduce ADS risks.The remainder of this paper is organized as follows: In Section2, we review the literature related to ADS safety. Section 3 assesses the potential ADS risks through the California DMV AVT disengagement data classified according to accident types and software modules. In Section 4, we establish genuine ADS risks through the California DMV AVT collision reports. We then provide some solutions based on the characteristics of the most frequent accidents. In Section 5, we summarize this study and discuss its limitations.


## 2. ADS Safety-Related Works

According to current studies, there are some main factors that affect ADS safety, such as hardware/ADAS, systems/software, AI-based highly automated driving (HAD) quality, communication ability, and environment/resilience. The related research is summarized and shown in Table1. The key factors related to the hardware and ADAS include sensors, GPS positioning hardware, ADAS, and hardware discrepancies. ADAS firmware is included. For example, the key factors for the communication type include cybersecurity, vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), intelligent transport systems (ITSs), and roadway geometric layouts. Scenario simulations and empirical road tests are the current ways of verifying their reliability and quality [18, 32, 48].Table 1
Source types and example factors for ADS risks.
Source typesKey factorsReferencesHardware/ADASSensors, GPS positioning hardware, ADAS, hardware discrepanciesBazzi et al. [19], Amersbach and Winner [20], Cicchino [21], Cicchino [22], Deflorio and Carboni [23], Shah et al. [24]Systems/softwareSensing/detection systems, computation, perception systems, planning systems, controlling systems, software and system discrepancies, calibration issuesBanerjee et al.[25], Brell et al. [26], Dixit et al. [27], Jiang [28], Mimura et al. [29], Shladover [5], Sun et al. [6], Vagale et al. [7], Vourgidis et al. [30], Wang et al. [31]AI-based HADDriving decision-making, HAD simulations, other road user behaviorSchnelle et al. [32], Abu Znaid et al. [33], Dai [34], Althoff and Mergel [35], Brell et al. [26], Biever et al. [36], Blanco et al. [37], Cui et al. [38], Fu and Sayed [39], Habibovic et al. [40], Mozaffari et al. [4], Osman et al. [41]Communications/resilienceCybersecurity, V2V, V2I, ITS, roadway geometric layoutsArena and Pau [42], Blanco et al. [37], Scala et al. [43], Khan et al. [44], Jiang [28], Javed et al. [45], Vourgidis et al. [30]EnvironmentWeather, roadway conditionsYaacob et al. [46], Hassan et al. [47], Wu et al. [18]The empirical road tests therefore play a vital role in validating the results of ADS safety. Research by Kalra and Paddock [49] noted that empirical testing with sufficient mileage is crucial for validating ADS safety. Under contemporaneous regulations, it may take decades for ADS developers to bring their SAE Class 3–5 ADS products to the market if they do not ramp up their testing efforts. In addition, Favarò et al. [50] made regulatory recommendations by analyzing the flaws in the draft regulations at the time of testing and deploying ADS on public roads. These studies have led to several ADS safety-related research initiatives based on California DMV AVT test data, and they can be divided into two categories, namely, disengagement and accident pertinent.In studies on disengagement classifications, Dixit et al. [27] classified disengagement origins into the following six causes: weather, construction zones, road infrastructure, driver initiation, system failure, and other road users. Boggs et al. [8] allocated disengagement to six clusters: “environmental and other road users,” “hardware and software discrepancy,” “planning discrepancy,” “control discrepancy,” “perception discrepancy,” and “operator takeover” and found that the first three clusters had higher occurrence probabilities than the others. This is likely to be because drivers could not observe the ADS data processing system running in the background. Wang and Li [51] separated the ADS disengagement causes into three dimensions, namely, perception issues (33%), planning issues (60%), and control issues (7%). They argued that the number of sensors, planning and computing capabilities, and the driver’s trust level in the ADS might influence the occurrence of disengagement. Boggs et al. [8] observed and quantified safety-critical events with 5 Ws, including 124 collisions and 159,840 disengagements. The 5 Ws comprised who (disengagement initiator), when (ADS maturity), where (disengagement location), and what or why (facts causing the disengagement).Therefore, we can summarize these studies into eight potential ADS risk factors. They are (1) hardware/ADAS issues, (2) system and software differences, (3) planning issues, (4) perception issues, (5) environmental conditions, (6) unexpected behavior of other road users, (7) control variance, and (8) others [8–18]. Regrettably, these studies did not cover the impact of technological advances on the data’s variability, nor did they detail the systematic classification methods used. Incorporating a large quantity of legacy ADS disengagement data that never occurred and were outdated into the calculation may have significantly distorted the proportion of existing potential failure data. The results also did not reflect the latest developments in ADS technology that the disengagement data represent. Therefore, this study has conducted a chi-square homogeneity test for the dissociated data in recent years.In the ADS risk assessment-related literature, Wang et al. [52] pointed out that 93.7% of ADS accidents are caused by other parties, including pedestrians, cyclists, motorcycles, and conventional vehicles. Therefore, an excellent passive accident prevention design may dramatically improve AV safety. Xu et al. [53] stated that the predominant accident type for connected and autonomous vehicles (CAVs) was the rear-end crash, accounting for 57.5% of all accidents. By examining the California DMV collision reports, Biever et al. [36] found that ADS frequently encountered no-fault rear-end collisions. The vehicles behind triggered these rear-end crashes which occurred, while ADS vehicles were braking, cornering, or maintaining a safe driving distance. However, the responsibility for negligence was entirely that of other parties. Song et al. [54] and similar studies investigated the California DMV AVT collision reports and briefed that the rear-end collision is the most common type of ADS collision, with 60–80% of ADS crashes occurring at relatively low speeds (i.e., below 10 mph) and involving the ADS vehicle and a second-party vehicle situated behind [25, 54, 55]. Unfortunately, from the perspective of ADS developers, discovering programs or rule errors and identifying potential safety hazards from the risk assessment reports are the approaches that should be adopted to perfect the system. Traditional accident classifications only list a wide range of accident types, and applying them to the software module structure makes it challenging to identify the location for enhancement. In addition, these studies rarely analyze the features of high-risk factors or provide workable solutions.There are various techniques used for assessing accidents involving conventional vehicles, such as the Haddon matrix [46], fault tree analysis (FTA) [56, 57], event tree analysis (ETA) [58], and the driving reliability and error analysis method (DREAM 3.0) [40, 41, 59]. They are also available for SAE levels 3–5 ADS. Unfortunately, the California DMV data lack appropriate and complete accident assessment investigation forms and often lack essential information. Therefore, it is often necessary to combine multiple techniques with accident assessment.Rear-end collisions are the most frequent cause of ADS accidents, including front and rear-end collisions [54]. Existing ADASs help drivers prevent rear-end collisions. The most commonly used rear-end collision-related ADASs are forward collision warning (FCW), automatic emergency braking (AEB), obstacle detection (OD), and backward collision warning (BCW). Many studies have shown that advanced ADAS-equipped vehicles can significantly reduce accident rates. However, some studies have found that multiple ADASs of similar types or their improper use by the driver may increase the front and rear crash rates [21–23, 29, 54]. Song et al. [54] showed that 60–80% of ADS crashes occurred at relatively low speeds involving ADSs and second-party vehicles. The most common crash pattern is a “collision after the AV comes to a stop.” 24% of these collisions disengaged before the collision, and then, 68% of these disengagement events occurred; immediate collisions came afterwards. These disengagements were mainly because of operator precautions, or because the ADS detected reckless behavior by other road users but did not provide sufficient reaction time for drivers to respond. In addition, some studies have designed linear path control and emergency steering assist (ESA) control to avoid rear-end collisions [24, 38]. Raju et al. [60] proposed a safety measure of instantaneous attention time (IHT) that quantifies the follower driver’s attentiveness.Collision-avoidance metrics, such as temporal and spatial proximity metrics, have advanced in recent years to prevent vehicle accidents. For example, the deceleration rate for collision avoidance (DRAC) has become one of the alternative safety measures [61]. The higher the DRAC value, the higher the risk of collision. A crash occurs when the DRAC value exceeds the maximum available deceleration (MADR). Each vehicle has a different MADR value. Factors such as road conditions, vehicle weight, tires, and braking systems also play a role. We can derive the crash risk by calculating the probability that MADR is less than DRAC. Earlier studies used different MADR-specific values for DRAC-related traffic safety assessments. These studies have assumed a conservative MADR-specific value for all vehicles: 8 m/s2 [62]. Wang et al. [31] estimated collisions from a bivariate extreme value model, using MADR values of 8 m/s2 and 12 m/s2 as the extremes at both ends. Fu and Sayed [39] proposed a method to estimate the MADR to avoid crashes. They grounded this method on DRAC and used Bayesian hierarchical models. Unfortunately, there is little research on applying DRAC and MADR to the ADS domain to provide ADS accident risk reduction. We hypothesize that MADR will help prevent ADS accidents, such as rear-end collisions and the unexpected behavior of other road users, which we discuss at the end of this paper.


## 3. ADS Risk Classification

This study has complied with the following ADS risk study procedures shown in Figure1. It contains four stages, namely, the data inspection stage, accident grouping stage, ADS risk assessment stage, and ADS risk reducing stage. First, the purpose of the data inspection stage is to perform the chi-square homogeneity test on the DMV AVT data to confirm that the data are suitable for combined processing procedures. Moreover, the results are also meaningful. Second, the accident grouping phase is intended to identify the root causes of the DMV AVT data and group them into different ADS accident groups. Third, in the ADS risk assessment stage, both the traditional and software module accident classification methods are applied to conduct an ADS risk assessment. Finally, in the ADS risk reduction stage, we attempt to find solutions for relatively high ADS risk factors.Figure 1
ADS risk study approach diagram.In the first stage, the data inspection stage, in order to address the lack of examination of the disengagement data, this study performed the following three chi-square homogeneity tests using a total of 18,557 records for the California DMV AVT data from the last three years (2019–2021), including 11,482 records for ADEs and 3,965 records for PDEs:(1)
The chi-square homogeneity test for ADE (2019, 2020, and 2021): This test confirms whether the number of occurrences for each factor of ADEs and their distributions among these three years are consistent (i.e., the null hypothesis). In this test, thep value is <2.2e − 16 (almost equal to 0). We conclude that the frequency distribution for ADE over the three years is significantly different.(2)
The chi-square homogeneity test for PDE (2019, 2020, and 2021): This confirms whether the number of occurrences for each factor of PDEs and their distributions among these three years are consistent (i.e., the null hypothesis). In this test, thep value is <2.2e − 16 (almost equal to 0). We conclude that the frequency distribution of PDE over the three years is significantly different.(3)
The chi-square homogeneity test for aggregated ADE and PDE (2019, 2020, and 2021): The numbers of occurrences for each factor for ADE and PDE are aggregated. Then, we examine whether the sums of the numbers of occurrences for each factor for ADE plus PDE with their distributions for these three years are consistent (i.e., the null hypothesis). In this test, thep value is <2.2e − 16 (almost equal to 0). We conclude that the frequency distribution of the aggregated ADE differs significantly from the aggregated PDE.Thus, we can confirm that the California DMV disengagement data have varied in the most recent three years. The aggregation of ADE and PDE has also varied. It is inappropriate to combine them for the analysis, and they should thus be processed separately. When many of the same faults repeatedly occur, especially during the phase of official test drives, ADS developers may quickly correct their systems. Accumulating many old faults that no longer exist may distort the proportion of existing potential faults. However, the collision data represent genuine crash accidents, and the more complete they need to be, the better. Therefore, we have chosen disengagement data from the most recent three years instead of all seven years for the analysis in this study.We then apply FTA in this study to identify and evaluate the interrelationships between events leading to faults, unexpected events or states, and unintentional events or conditions. Subsequently, we use DREAM 3.0 to distinguish the accident groups or the Haddon matrix to analyze the root fault causes or identify critical contributing factors when it is difficult to judge descriptions accurately. We then group all similar faults in the disengagement events into the same accident group. Finally, we complete 40 accident groups, including the “other” group. The results are shown in Table2.Table 2
Definitions and descriptions of ADS accident groups.
SQCodeADS accident groupsDescription1H01ADAS issuesADAS component failed to detect, identify, raise alerts, or take specified actions2H02Sensor issuesThe sensor hardware failed to detect the object or could not transmit data to the sensor system of the ADS3H03GPS, localization issuesGPS hardware could not obtain the correct position of the vehicle or could not locate it correctly on the map4H04Communication hardware issuesCommunication components or wires failed to transmit complete and correct data to target recipients5H05Hardware discrepancy and requirement issuesADS hardware components were defective, the quality was not consistent with the design, or the ADS hardware component design did not meet the basic requirements of the ADS hardware6H06Other hardware/ADAS issuesOther hardware or ADAS accidents and incidents7S1-1Invalid object or traffic light detectionInvalid object or invalid traffic light detection caused system or software accidents and incidents8S1-2Computation issues of perceptionPerceptual computing problems caused system or software accidents and incidents9S1-3Delayed perception detectionDelayed sensing or perception detection caused system or software accidents and incidents10S1-4Perception gaps between ADS and driversIssues consisted of the perceptual discrepancies between the ADS system and the driver11S1-5Failed to detect an object correctlyUnable to detect an object, incorrectly identify it, or specify an entity that does not exist12S1-6Other perception issuesPerceptual issues were different from the above13S2-1Improper localization and planningSystem or software accidents and incidents occurred because of inaccurate ADS positioning, map-related issues, or route planning discrepancies14S2-2Computation issues of planningSystem or software accidents and incidents occurred because the performance of planning computation was below expectations15S2-3Motion planning issuesSystem or software accidents and incidents occurred because object motion prediction capabilities or results were inaccurate or could not meet expectations16S2-4Other planning issuesPlanning issues other than the above17S3-1Improper acceleration/deceleration/cruiseSystem or software control discrepancies due to the timing, speed, and scope of the acceleration and deceleration, or cruise control activities did not meet expectations18S3-2Improper steering wheel/lane changeSystem or software control discrepancies occurred due to imprecise steering wheel control or inaccurate lane change activities19S3-3Unwanted maneuver/control irregularitySystem or software control discrepancies occurred due to unnecessary control activities, violations, and irregularities20S3-4Computational issues of controllingSystem or software control discrepancies occurred due to the performance of controlling computation being below expectations21S3-5Improper gap or other control discrepanciesSystem or software control discrepancies occurred for reasons other than the above issues22S4-1Software discrepancySoftware discrepancies caused software programs to run abnormally due to errors in software codes or missing programs23S4-2System discrepancySystem discrepancies caused system programs to run abnormally due to errors in system codes or missing programs24S4-3System tuning and calibration issuesThe inability to accept, process, modify, or fine-tune system performance for higher loads or multiple tasks25S4-4System health and readiness issuesThe system health check or reliability-related examination issues caused accidents and incidents26E1-1Bad weatherAccidents and incidents occurred in bad weather27E1-2Insufficient lightingAccidents and incidents occurred due to insufficient lighting28E1-3Roadway surface and conditionsAccidents and incidents occurred on hazardous roadway surfaces or due to other hazardous conditions29E1-4ConstructionAccidents and incidents occurred due to construction conditions30E1-5Poor lane markingsPoor lane markings caused accidents and incidents31E1-6Blocked laneAccidents and incidents were caused by blocked lanes32E1-7Road debris or rough pavementAccidents and incidents were caused by road debris or rough pavement33E1-8Other environment-associated factorsAccidents and incidents occurred due to other environment-associated factors34E2-1Malbehavior of other road usersInadequate behavior of other road users caused accidents and incidents35E2-2Undesired behavior of emergency vehicleAccidents and incidents were due to the behavior of emergency vehicles36E2-3Forward collisionAccidents and incidents caused forward collisions and hitting other road users37E2-4Rear-endAccidents and incidents caused rear-end collisions and hitting by other road users38E2-5Overly conservative behavior of other road usersAccidents and incidents occurred due to the overly conservative behavior of other road users39E2-6Other road users’ associated factorsAccidents and incidents caused rear-end and hitting by other road users40OOther factorsAccidents and incidents factors were different from the aboveThe eight potential risk factors for ADS accident types of the traditional classification method are shown in Table3. The adopted classification relationships are presented in an FTA-like manner, as shown in Figure 2. The traditional accident-type classification results for the official DMV AVT disengagement data are provided in Table 4. We can see that the PDEs were central to planning issues in 2019 and 2020, accounting for 59.37% and 66.63%, respectively, whereas the ADS cannot handle the issues. In 2021, PDEs that occurred were central to system and software discrepancies (53.47%). By contrast, ADEs that were intervened by human drivers in 2019 were centralized based on the unexpected behavior of other road users (44.26%) and in 2020 on two issues, namely, perception issues (35.40%) and control discrepancies (32.45%). However, in 2021, ADEs were centralized based on planning issues (66.63%) and were followed by perception issues (32.42%). Therefore, to sum up, in the cases of both ADEs and PDEs, planning issues have always occupied a significant proportion. Second, the ADE values of the perception issues differ significantly from the PDE values from year to year, which shows that human drivers lack trust in the ADS perception system. Furthermore, the proportion of ADEs in 2021 for system and software discrepancy issues increased substantially, which deserves close observation. However, their numbers are low with regard to the hardware/ADAS issues and environmental conditions that concern society. Finally, the unexpected behavior of other road users is insignificant except for a higher proportion of ADE in 2019. This part is significantly different from the results of the risk assessment carried out later.Table 3
Classification, definitions, and descriptions of ADS risk factors and accident groups.
SQCodeADS risk factorsDescription1HHardware/ADAS issuesAccidents and incidents are purely related to hardware components or ADAS2S1Perception issuesSystem or software accidents and incidents occur due to the sensing or perception system3S2Planning issuesSystem or software accidents and incidents occur due to the planning and localization system4S3Control discrepancyThe system or software discrepancies occur because of the controlling discrepancy5S4System and software discrepancySystem or software accidents and incidents occur due to the imperfect system or software errors or lack of program codes6E1Environmental conditionsAccidents and incidents occur due to driving environment or conditions7E2Unexpected behavior of other road usersAccidents and incidents are related to the unexpected behavior of other road users8OOther factorsAccidents and incidents factors are different from the aboveFigure 2
Classification diagram for ADS risk factors and accident groups.Table 4
Potential risk assessment results through 2019–2021 DMV AVT disengagement reports (grouped by accident types).
CodeAV failuresADEPDE202120202019202120202019HHardware/ADAS issues381.70%401.37%2273.58%6113.65%698.44%411.52%H01ADAS failure00.00%110.38%40.06%00.00%20.24%00.00%H02Sensors failure351.57%110.38%2113.33%20.45%151.83%100.37%H03GPS, localization issues10.04%00.00%00.00%00.00%202.44%60.22%H04Communication hardware failures00.00%00.00%20.03%00.00%121.47%90.33%H05Hardware discrepancy and requirement issues20.09%80.27%100.16%5913.20%30.37%160.59%H06Other hardware/ADAS issues00.00%100.34%00.00%00.00%172.08%00.00%S4System and software discrepancy431.93%351.20%4867.67%23953.47%172.08%52019.26%S4-1Software discrepancy150.67%20.07%4837.62%16937.81%161.96%1997.37%S4-2System discrepancy261.17%290.99%20.03%102.24%10.12%1776.56%S4-3System tuning and calibration issues20.09%00.00%00.00%00.00%00.00%00.00%S4-4System health and readiness issues00.00%40.14%10.02%6013.42%00.00%1445.33%S2Planning issues77434.71%48816.74%126920.03%5913.20%54566.63%160359.37%S2-1Improper localization and planning57725.87%852.92%100015.78%71.57%637.70%742.74%S2-2Computation issues of planning683.05%933.19%981.55%4810.74%47157.58%78028.89%S2-3Motion planning issues1295.78%1685.76%1712.70%40.89%80.98%74927.74%S2-4Other planning issues00.00%1424.87%00.00%00.00%30.37%00.00%S1Perception issues72332.42%103235.40%2604.10%7316.33%435.26%28310.48%S1-1Invalid object or traffic light detection1667.44%180.62%1332.10%6113.65%10.12%60.22%S1-2Computation issues of perception512.29%47816.40%831.31%122.68%253.06%10.04%S1-3Delayed perception detection1647.35%551.89%440.69%00.00%00.00%27610.22%S1-4Perception gaps between ADS and drivers793.54%46115.81%00.00%00.00%00.00%00.00%S1-5Failed to detect an object correctly26311.79%190.65%00.00%00.00%30.37%00.00%S1-6Other perception issues00.00%10.03%00.00%00.00%141.71%00.00%E1Environmental conditions371.66%280.96%731.15%00.00%303.67%70.26%E1-1Bad weather30.13%10.03%410.65%00.00%50.61%20.07%E1-2Insufficient lighting00.00%20.07%00.00%00.00%20.24%00.00%E1-3Roadway surface and conditions110.49%00.00%110.17%00.00%212.57%40.15%E1-4Construction130.58%10.03%80.13%00.00%10.12%10.04%E1-5Poor lane markings90.40%00.00%00.00%00.00%00.00%00.00%E1-6Blocked lane10.04%160.55%120.19%00.00%10.12%00.00%E1-7Road debris or rough pavement00.00%00.00%10.02%00.00%00.00%00.00%E1-8Other environment-associated factors00.00%80.27%00.00%00.00%00.00%00.00%E2Unexpected behavior of other road users1818.12%34611.87%280544.26%00.00%151.83%10.04%E2-1Malbehavior of other road users1466.55%1966.72%1903.00%00.00%151.83%00.00%E2-2Behavior of emergency vehicles20.09%10.03%30.05%00.00%00.00%00.00%E2-3Forward collision00.00%00.00%00.00%00.00%00.00%00.00%E2-4Rear-end00.00%20.07%00.00%00.00%00.00%00.00%E2-5Overly conservative behavior of other road users331.48%1324.53%261241.22%00.00%00.00%10.04%E2-6Associated factors of other road users00.00%150.51%00.00%00.00%00.00%00.00%S3Control discrepancy42919.24%94632.45%121519.17%153.36%8710.64%1505.56%S3-1Improper acceleration/deceleration/cruise1004.48%51517.67%80312.67%51.12%222.69%331.22%S3-2Improper steering wheel/lane change24510.99%39113.41%1292.04%00.00%182.20%00.00%S3-3Unwanted maneuver/control irregularity833.72%250.86%630.99%81.79%10.12%110.41%S3-4Computation issues of controlling10.04%90.31%2203.47%20.45%00.00%1063.93%S3-5Improper gap or other control discrepancies00.00%60.21%00.00%00.00%465.62%00.00%OOthers50.22%00.00%20.03%00.00%121.47%953.52%Total2230100.00%2915100.00%6337100.00%447100.00%818100.00%2700100.00%As mentioned earlier, it is difficult for ADS software developers to view the traditional accident-type classification method intuitively. Therefore, this study provides a new way for classification from the perspective of five software modules, as shown in Table5. The adopted classification relationships are presented in an FTA-like manner, as shown in Figure 3. The software module classification results for the official DMV AVT disengagement data are shown in Table 6. We can see that PDEs were centralized in the navigation software modules in 2019 and 2020, and their percentages are 59.37% and 66.63%, respectively, whereas ADS cannot handle the issues. However, in 2021, PDEs were centralized in the condition diagnostic software module (67.11%). By contrast, ADEs that were intervened by human drivers were centralized in AI-driven decision-making software modules (45.42%) in 2019. In 2020, ADEs were consolidated into two software modules, namely, the sensing software module (35.40%) and the control software module (32.45%). However, in 2021, ADEs were centralized in navigation software modules (34.71%), followed by sensing software modules (32.42%). Therefore, to sum up, for both ADEs and PDEs, the navigation software module has always occupied a significant proportion. Second, the ADE values of the AI driving decision software module differ significantly from the PDE values from year to year, which shows that human drivers lack trust in the ADS AI driving decisions.Table 5
Definitions and descriptions of ADS software module risk factors.
CodeADS risk factorsDescriptionM1Condition diagnostic software moduleAccidents and incidents happen during the condition diagnostic moduleM2Navigation software moduleAccidents and incidents happen during the diagnostic navigation module due to planning and localization issuesM3Sensing software moduleDuring the sensing module, accidents and incidents happen due to sensing or perception issuesM4AI driving decision software moduleAccidents and incidents happen during the AI driving decision module due to the sensing or perceiving of the driving environment or other road users’ issuesM5Control software moduleAccidents and incidents occur during the control module due to the controlling discrepancyOOther factorsAccidents and incidents factors are different from the aboveFigure 3
Relationship diagram for ADS accident groups and risk factors of software modules.Table 6
Potential risk assessment results through 2019–2021 DMV AVT disengagement reports (grouped by software modules).
CodeAV failuresModuleADEPDE202120202019202120202019M1Condition diagnostic software module issues1813.63%752.57%71311.25%30067.11%8610.51%56120.78%M1H1ADAS issues100.00%110.38%40.06%00.00%20.24%00.00%M1H2Sensor issues1351.57%110.38%2113.33%20.45%151.83%100.37%M1H3GPS, localization issues110.04%00.00%00.00%00.00%202.44%60.22%M1H4Communication hardware issues100.00%00.00%20.03%00.00%121.47%90.33%M1H5Hardware discrepancy and requirement issues120.09%80.27%100.16%5913.20%30.37%160.59%M1H6Other hardware/ADAS issues100.00%100.34%00.00%00.00%172.08%00.00%M1S1Software discrepancy1150.67%20.07%4837.62%16937.81%161.96%1997.37%M1S2System discrepancy1261.17%290.99%20.03%102.24%10.12%1776.56%M1S3System tuning and calibration issues120.09%00.00%00.00%00.00%00.00%00.00%M1S4System health and readiness issues100.00%40.14%10.02%6013.42%00.00%1445.33%M2Navigation module software issues277434.71%48816.74%126920.03%5913.20%54566.63%160359.37%M2P1Improper localization and planning257725.87%852.92%100015.78%71.57%637.70%742.74%M2P2Computation issues of planning2683.05%933.19%981.55%4810.74%47157.58%78028.89%M2P3Motion planning issues21295.78%1685.76%1712.70%40.89%80.98%74927.74%M2P4Other planning issues200.00%1424.87%00.00%00.00%30.37%00.00%M3Sensing module software issues372332.42%103235.40%2604.10%7316.33%435.26%28310.48%M3D1Invalid object or traffic light detection31667.44%180.62%1332.10%6113.65%10.12%60.22%M3D2Computation issues of perception3512.29%47816.40%831.31%122.68%253.06%10.04%M3D3Delayed perception detection31647.35%551.89%440.69%00.00%00.00%27610.22%M3D4Perception gaps between ADS and drivers3793.54%46115.81%00.00%00.00%00.00%00.00%M3D5Failed to detect an object correctly326311.79%190.65%00.00%00.00%30.37%00.00%M3D6Other perception issues300.00%10.03%00.00%00.00%141.71%00.00%M4AI-HAD driving decision module issues42189.78%37412.83%287845.42%00.00%455.50%80.30%M4E1Bad weather430.13%10.03%410.65%00.00%50.61%20.07%M4E2Insufficient lighting400.00%20.07%00.00%00.00%20.24%00.00%M4E3Roadway surface and conditions4110.49%00.00%110.17%00.00%212.57%40.15%M4E4Construction4130.58%10.03%80.13%00.00%10.12%10.04%M4E5Poor lane markings490.40%00.00%00.00%00.00%00.00%00.00%M4E6Blocked lane410.04%160.55%120.19%00.00%10.12%00.00%M4E7Road debris or rough pavement400.00%00.00%10.02%00.00%00.00%00.00%M4E8Other environment-associated factors400.00%80.27%00.00%00.00%00.00%00.00%M4B1Malbehavior of other road users41466.55%1966.72%1903.00%00.00%151.83%00.00%M4B2Emergency vehicles420.09%10.03%30.05%00.00%00.00%00.00%M4B3Forward collision400.00%00.00%00.00%00.00%00.00%00.00%M4B4Rear-end400.00%20.07%00.00%00.00%00.00%00.00%M4B5Overly conservative behavior of other road users4331.48%1324.53%261241.22%00.00%00.00%10.04%M4B6Other road users’ associated factors400.00%150.51%00.00%00.00%00.00%00.00%M5Control software module issues542919.24%94632.45%121519.17%153.36%8710.64%1505.56%M5C1Improper acceleration/deceleration/cruise51004.48%51517.67%80312.67%51.12%222.69%331.22%M5C2Improper steering wheel/lane change524510.99%39113.41%1292.04%00.00%182.20%00.00%M5C3Unwanted maneuver/control irregularity5833.72%250.86%630.99%81.79%10.12%110.41%M5C4Computation issues of controlling510.04%90.31%2203.47%20.45%00.00%1063.93%M5C5Improper gap or other control discrepancies500.00%60.21%00.00%00.00%465.62%00.00%OOthersO50.22%00.00%20.03%00.00%121.47%953.52%Total2230100.00%2915100.00%6337100.00%447100.00%818100.00%2700100.00%Furthermore, the proportion of ADEs in 2021 for the status diagnosis software module increased substantially, which deserves close observation. Finally, the value of the AI-driven decision software module is insignificant except for a higher proportion of ADE in 2019. This part differs significantly from the results of the risk assessment result carried out later.


## 4. ADS Actual Risk Assessment and Suggested Solutions

After evaluating the potential risk of ADS, we used the same method to assess the second set of data: the accident (collision) report of the California DMV AVT. Incident reports included 0 fatalities, 86 injuries, and 358 PDO incidents. Unlike the disengagement profiles of potential risks, accident reports are actual accidents that occurred during the same testing period. The results using the traditional accident-type classification method are shown in Table7, while the results using the software module classification method are shown in Table 8.Table 7
The ADS actual risk assessment results through 2015–2021 DMV AVT collision reports (grouped by accident types).
CodeAV failures2021202020192018201720162015Total∗Mean = times/million kmDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesAll (%)HHardware/ADAS issues002200000000000000000000000000220.51H01ADAS failure001100000000000000000000000000110.25H02Sensor failure001100000000000000000000000000110.25H03GPS, localization issues000000000000000000000000000000000.00H04Communication hardware failures000000000000000000000000000000000.00H05Hardware discrepancy and requirement issues000000000000000000000000000000000.00H06Other hardware/ADAS issues000000000000000000000000000000000.00S4System and software discrepancy000000000011000001110000000001220.51S4-1Software discrepancy000000000000000000000000000000000.00S4-2System discrepancy000000000000000000000000000000000.00S4-3System tuning and calibration issues000000000011000001110000000001220.51S4-4System health and readiness issues000000000000000000000000000000000.00S2Planning issues003300000000000000000000000000330.76S2-1Improper localization and planning001100000000000000000000000000110.25S2-2Computation issues of planning001100000000000000000000000000110.25S2-3Motion planning issues001100000000000000000000000000110.25S2-4Other planning issues000000000000000000000000000000000.00S1Perception issues00550267000000110011001100000214153.81S1-1Invalid object or traffic light detection000000000000000000000000000000000.00S1-2Computation issues of perception00550267000000110000000000000212133.30S1-3Delayed perception detection000000000000000000110000000000110.25S1-4Perception gaps between ADS and drivers000000000000000000000000000000000.00S1-5Failed to detect an object correctly000000000000000000000011000000110.25S1-6Other perception issues000000000000000000000000000000000.00E1Environmental conditions004400000123002200000000000001892.28E1-1Bad weather000000000000000000000000000000000.00E1-2Insufficient lighting000000000000000000000000000000000.00E1-3Roadway surface and conditions002200000101001100000000000001341.02E1-4Construction000000000000000000000000000000000.00E1-5Poor lane markings000000000000000000000000000000000.00E1-6Blocked lane000000000000000000000000000000000.00E1-7Road debris or rough pavement002200000000000000000000000000220.51E1-8Other environment-associated factors000000000022001100000000000000330.76E2Unexpected behavior of other road users03090960828350258910109666905272701911027908031634888.32E2-1Malbehavior of other road users02056600315180727320628300010100123002203714015539.34E2-2Behavior of emergency vehicles000000000000004400000000000000441.02E2-3Forward collision01220011008800110011000100000113143.55E2-4Rear-end09313305121601854610331320516160077025704215617243.65E2-5Overly conservative behavior of other road users001100000000002200000000000000330.76E2-6Associated factors of other road users000000000000000000000000000000000.00S3Control discrepancy01770022000000110000003300000113133.30S3-1Improper acceleration/deceleration/cruise000000000000000000000000000000000.00S3-2Improper steering wheel/lane change014400220000001100000022000001992.28S3-3Unwanted maneuver/control irregularity000000000000000000000011000000110.25S3-4Computation issues of controlling003300000000000000000000000000330.76S3-5Improper gap or other control discrepancies000000000000000000000000000000000.00OOthers000000000000002200000000000000220.51Total0311111170103644026921050972750629290113150279085360394100.00Table 8
The ADS actual risk assessment results through 2015–2021 DMV AVT collision reports (grouped by software modules).
CodeAV failuresModule2021202020192018201720162015TotalDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesDeathInjuryPDOCasesAll (%)M1Condition diagnostic software module issues1002200000011000001110000000001441.02M1H1ADAS issues1001100000000000000000000000000110.25M1H2Sensor issues1001100000000000000000000000000110.25M1H3GPS, localization issues1000000000000000000000000000000000.00M1H4Communication hardware issues1000000000000000000000000000000000.00M1H5Hardware discrepancy and requirement issues1000000000000000000000000000000000.00M1H6Other hardware/ADAS issues1000000000000000000000000000000000.00M1S1Software discrepancy1000000000000000000000000000000000.00M1S2System discrepancy1000000000000000000000000000000000.00M1S3System tuning and calibration issues1000000000010000001100000000001200.00M1S4System health and readiness issues1000000000001000000010000000000020.51M2Navigation module software issues2003300000000000000000000000000330.76M2P1Improper localization and planning2001100000000000000000000000000110.25M2P2Computation issues of planning2001100000000000000000000000000110.25M2P3Motion planning issues2001100000000000000000000000000110.25M2P4Other planning issues2000000000000000000000000000000000.00M3Sensing module software issues300550267000000110011001100000214153.81M3D1Invalid object or traffic light detection3000000000000000000000000000000000.00M3D2Computation issues of perception300550267000000110000000000000212133.30M3D3Delayed perception detection3000000000000000000110000000000110.25M3D4Perception gaps between ADS and drivers3000000000000000000000000000000000.00M3D5Failed to detect an object correctly3000000000000000000000011000000110.25M3D6Other perception issues3000000000000000000000000000000000.00M4AI-HAD driving decision module issues4030941000828350269110409687105272701911027908132435790.61M4E1Bad weather4000000000000000000000000000000000.00M4E2Insufficient lighting4000000000000000000000000000000000.00M4E3Roadway surface and conditions4002200000101001100000000000001341.02M4E4Construction4000000000000000000000000000000000.00M4E5Poor lane markings4000000000000000000000000000000000.00M4E6Blocked lane4000000000000000000000000000000000.00M4E7Road debris or rough pavement4002200000000000000000000000000220.51M4E8Other environment-associated factors4000000000022001100000000000000330.76M4B1Malbehavior of other road users402056600315180727320628300010100123002203714015539.34M4B2Emergency vehicle4000000000000004400000000000000441.02M4B3Forward collision401220011008800110011000100000113143.55M4B4Rear-end409313305121601854610331320516160077025704215617243.65M4B5Overly conservative behavior of other road users4001100000000002200000000000000330.76M4B6Other road users’ associated factors4000000000000000000000000000000000.00M5Control software module issues501770022000000110000003300000113133.30M5C1Improper acceleration/deceleration/cruise5000000000000000000000000000000000.00M5C2Improper steering wheel/lane change5014400220000001100000022000001992.28M5C3Unwanted maneuver/control irregularity5000000000000000000000011000000110.25M5C4Computation issues of controlling5003300000000000000000000000000330.76M5C5Improper gap or other control discrepancies5000000000000000000000000000000000.00OOthersO000000000000002200000000000000220.51Total0311111170103644026921050972750629290113150279085360394100.00As the results show, actual ADS accidents are highly concentrated in the unintended behavior of other road users based on the traditional accident-type classification method (88.32%) or the AI driving decision software module when using the software module classification method (90.61%). This result is significantly different from potential ADS risk assessment using disengagement data. When we take a closer look, the accident group with the most significant risk is the rear-end collision accident group, accounting for 43.65%, and the bad behavior accident group of other road users follows in second place, accounting for 39.34%. However, the advanced ADS safety design has successfully decreased front collision accidents (3.55%), but the reduction in rear-end collisions (44.70%) is not apparent. Therefore, this study first tries to analyze some of the characteristics of rear-end collisions and then attempts to find solutions to reduce the occurrence of these accidents.Having gone through the 394 accident reports provided to the California DMV, 198 were found to be rear-end-related collision reports. After an in-depth analysis of these 198 reports, we found one valuable feature in rear-end collisions. In Figure4, more than half (59.30%) of rear-end collisions occurred when the AV stopped, and 34.30% of rear-end collisions took place at extremely slow speeds (moving at ≤ 10 mph). By contrast, nearly half (47.67%) of other vehicles involved in rear-end collisions were moving at ≤ 10 mph. After excluding the records where the speeds of other vehicles were unknown, the proportion of other vehicles moving at ≤ 10 mph was as high as 85.42%. The fact that, in most cases, two vehicles collided at a slower speed was another reason for the lower property damage. Of particular note, the results differed slightly from the range (60% ∼ 80%) of Song et al. [54]. We assume that the gap existed due to the “unknown speed of other vehicles.” The contents of these records did not include details of their speed.Figure 4
Moving status for the 2015–2021 California DMV AVT rear-end ADS collisions.Although relevant applied research is currently rare, in accordance with this feature, we assume that the MADR theory discussed in the literature review in Section2 [31, 39, 62] may apply to ADSs to reduce the rear-end collision risk and the unexpected behavior of other road users. This is because traditional vehicles are not equipped with components and associated software modules required for the application. The equipment consists of a set of camera groups that surround the body, short-range radars with a monitoring distance of about 8 meters, and a system chip that can perform high-speed calculations and immediately make driving decisions.First, in order to prevent rear-end collisions, rear sensors can monitor the vehicles behind and continuously calculate their speed, DRAC, MADR, and the distance from the AV. According to the study by Fu and Sayed [39], when the DRAC of the rear vehicle exceeds the MADR bivariate extreme value of Wang et al. [31], the ADS can issue a warning and then calculate the distance between the AV and obstacles in front. When sufficient space is available, the ADS can immediately speed up to free up more space to avoid collision accidents or reduce the severity of injuries and PDOs.For example, if we suppose that the ADS stops at a stop sign, the AV can advance by 1.5 m (the distance from the ADS to the vehicle in front) within 0.8 seconds. The AV can also estimate the difference in speed through the rear sensor and HAD when the vehicle behind hits the AV (e.g., 5 mph). According to MADR, the ADS may avoid this crash entirely. Based on research by scholars [25, 54, 55], the AV could have avoided about 60% ∼ 80% of existing collisions altogether. Indeed, after re-examining the California DMV AVT collision reports, over 48 of the 198 rear-end collisions could have been avoided entirely in this way (24.24%), which is also 10.84% of all 394 official ADS collision reports over a period of 7 years.A similar risk prevention approach to rear-end collisions can also be applied to the prevention approach that seeks to reduce the risks of unexpected behavior of other road users. The only difference between these two is that the former only needs to apply the approach to the data on rear sensors (which may include data from the rear camera and short-range radar). However, the latter approach requires that the method involving the data of all surrounding sensors be used. It may also be helpful to include other related studies, such as linear path control design and ESA control [24, 38], to assist the ADS in taking corresponding actions in response to driving decisions.


## 5. Concluding Remarks

ADS safety is one of the critical factors for ADS products. This study aims to provide a solution to reduce the currently high ADS risk according to the California DMV AVT reports. We first found the official California DMV AVT data to be the most trusted data source. Then, we validated the California DMV AVT disengagement data and found that the data for each year varied. It is also inappropriate to combine such data for the analysis, and the different types of data should be processed separately. Therefore, we classified 15,447 records for California DMV AVT disengagement events into 40 accident groups and 6 risk factors using the traditional accident-type classification for potential ADS risk assessment. In addition, we provided another kind of software module classification for prospective software developers to assess their potential ADS risks and compare the two types. Next, we evaluated the ADS risk based on the 394 California DMV AVT collision reports using two classifications. Rear-end collisions and the unexpected behavior of other road users were corroborated as the most significant ADS risk factors with the highest occurrences.After that, we explored the features of the rear-end collisions. One of the features was that most of these accidents occurred at speeds below 10 mph, which is a cause for concern. Therefore, we provided an approach to applying MADR theory in ADS to reduce the rear-end collisions and the unexpected behavior of other road users as a suggested solution to minimize the ADS risk. This approach utilizes the components and related software modules required for applications not incorporated into traditional vehicles and can give full play to the advantages of ADS. After re-examining the California DMV AVT collision reports, more than 48 collisions (24.24%) among the 198 rear-end collisions could have been avoided entirely, which is also 10.84% of all 394 official ADS collision reports over 7 years.This study has a number of limitations, such as multiple interrelationships between accident groups and risk factors, which are because human drivers may not discover all causes of the faults or comprehend the ADS system processes running in the background. Another limitation is that if we want to evaluate the actual performance of the solution, we may need to implement the solutions in AVs and proceed with some crash tests, which are costly to our research.



---

*Source: 3215817-2023-02-01.xml*