Proceedings of the ASME 2015 Dynamic Systems and Control Conference
DSCC2015
October 28-30, 2015, Columbus, Ohio, USA

DSCC2015-9718

ACCELERATED EVALUATION OF AUTOMATED VEHICLES IN LANE CHANGESCENARIOS
Ding Zhao
Huei Peng
Dept. of Mechanical Engineering
University of Michigan
Ann Arbor, MI, USA

Henry Lam
Dept. of Industrial andOperations Engineering
University of Michigan
Ann Arbor, MI, USA

Shan Bao
Kazutoshi Nobukawa
David J. LeBlanc
University of Michigan
Transportation ResearchInstitute
Ann Arbor, MI, USA

Christopher S. Pan
Division of Safety Research, NIOSH, CDC
Morgantown, WV, USA
ABSTRACT
It is important to rigorously and comprehensively evaluate
the safety of Automated Vehicles (AVs) before their production
and deployment. A popular AV evaluation approach is
Naturalistic-Field Operational Test (N-FOT) which tests
prototype vehicles directly on public roads. Due to the low
exposure to safety-critical scenarios, N-FOTs is time-consuming
and expensive to conduct. Computer simulations can be used as
an alternative to N-FOTs, especially in terms of generating
motions of the surrounding traffic. In this paper, we propose an
accelerated evaluation approach for AVs. Human-controlled
vehicles (HVs) were modeled as disturbance to AVs based ondata extracted from the Safety Pilot Model Deployment
Program. The cut-in scenarios are generated based on skewed
statistics of collected human driver behavior, which amplifies
riskier testing scenarios while reserves its statistical information
so that the safety benefits of AV in non-accelerated cases can be
accurately estimated. An AV model based on a production
vehicle was tested. Results show that the proposed method canaccelerate the evaluation process by at least 100 times.

AVs will penetrate the market gradually and will co-exist
with non-AVs for decades. During this transition period, AVs
will interact mostly with HVs. It is estimated that 70-90% of the
motor vehicle crashes are due to human errors [3], [4]. To reduce
crashes, AVs must deal with the imperfect maneuvers initiatedby human drivers. A practical and effective evaluation approachthat accounts for imperfect human driven traffic is essential forthe development and evaluation of AVs.
One approach to study the interactions between AVs and
HVs is through Naturalistic Field Operational Tests (N-FOT) [5].
In an N-FOT, data is collected from a number of equipped
vehicles driven under naturalistic conditions over an extended
period of time [6]. Several N-FOT projects [7]–[15] have been
conducted in the U.S. and Europe. Conducting an N-FOT toevaluate an AV function typically involves non-intrusive
conditions, i.e., the test drivers were told to drive as theynormally would on public roads. This test approach suffers
several limitations. An obvious problem is the time needed.
Under naturalistic conditions, the level of exposure to dangerous
events is very low. In the U.S., there were 5,615,000 policereported motor vehicle crashes and 30,800 fatal crashes in 2012,
while the vehicles traveled a total of 2,968 billion miles [16].
This translates to approx. 0.53 million miles for each policereported crash and 96 million miles for a fatal crash. Since theaverage annual mileage driven by licensed drivers is 14,012
miles, one needs to drive on average 38 years to be involved in apolice-reported crash and 6,877 years for a fatal crash. Becauseof this low exposure rate, the N-FOT projects need a large
number of vehicles, long test duration, and a large budget.

INTRODUCTION
AV technologies are actively studied by many automotive
companies because of their potential to save fuel, reduce crashes,ease traffic congestion, and provide better mobility to the
population that could not operate a car [1]. Currently, almost all
major automakers have research and development programs on
AVs. In 2030, it is estimated that the sales of AV technologies
may reach $87 billion [2].

1

Copyright © 2015 by ASMEThis work is in part a work of the U.S. Government. ASME disclaims all interest in the U.S. Government’s contributions.

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/ on 03/10/2017 Terms of Use: http://www.asme.org/abo

According to [17], N-FOT projects “cannot be conducted withless than $10,000,000”. It should be noted that some researchers
took advantage of big data from N-FOTs and applied N-FOT
data in a simulation environment [18], [19]. Reusing the N-FOT
data in simulations is a good approach to testing various designs
to avoid the large budget for FOT. However, even for computer
simulations, low exposure to safety critical scenarios is still an
issue.
The test matrix approach has been the basis of many test
standards, such as AEB (Autonomous Emergency Braking) test
protocol [20] of the Euro NCAP (New Car Assessment
Program). Much development work was done to advance thisevaluation approach including CAMP [21], HASTE [22], AIDE[23], TRACE [24], APROSYS [25], ASSESS [26], etc. The test
scenarios are frequently selected based on national crashdatabases [27], such as GES (General Estimates System) [28],
NMVCCS (National Motor Vehicle Crash Causation Survey)[29] and EDR (Event Data Recorder databases) [30]. The main
benefits of these test methods are that they are clearly defined,
repeatable, reliable, and can be finished in a reasonable amountof time. However, it is not trivial to choose and reconstruct test
scenarios to represent real-world conditions [6], [31], especiallywhen human interaction is involved. Moreover, because all thetest scenarios are fixed and predefined, control systems can be
adjusted to achieve good performance on these tests, but their
behaviors under broader conditions are not adequately assessed[32].
Another approach, the Worst-Case Scenario Evaluation(WCSE) methodology has been studied by Ma [33], Ungoren[34] and Kou [35] to identify the most challenging scenarios
based on the analysis of vehicle dynamics and control. While theworst-case evaluation method can identify the weakness of avehicle and vehicle control systems, it did not consider the
probability of such worst-case scenarios. Therefore, the worst
case evaluation results do not provide sufficient information
about the risk in real world [36], [37], and may not be the fairest
way to compare different designs.
Zhao [38] proposed the concept of accelerated evaluation of
AV and applied this approach to car-following scenarios. The
dynamics of the lead vehicle was exemplified to generate ariskier conditions for the following AVs. The acceleration rate
was estimated by comparing the crash rate under accelerated
conditions and naturalistic conditions for human driven vehicles.
The proposed approach is useful to compare between AV
designs, but not yet able to rigorously estimate the crash rate ofa new AV under naturalistic conditions and evaluate its safety
benefit.
In this paper, we propose a new approach of accelerated
evaluation that can be used to estimate real world safety benefits
by using the importance sampling techniques. First, HVs aremodeled based on data extract from N-FOT database withstochastic variables to represent the human driving behaviors.
Second, the HV models are modified using importance sampling,which makes the drivers act more aggressively and thus could
generate safety critical scenarios at a higher frequency. The‘exemplified’ results are then corrected to produce estimations

statistically equivalent to the original naturalistic driving
conditions. A concept AV algorithm is modeled and evaluatedusing the proposed approach in lane change scenarios.
MODEL OF THE LANE CHANGE SCENARIOSThe lane change (cut-in) scenario is used as an example toshow the benefits of the proposed accelerated evaluation
approach. In the US, there are between 240,000 and 610,000
reported lane-change crashes, resulting in 60,000 injuries
annually [39]. Few protocols have been published regarding theevaluation of AVs (e.g., Autonomous emergency braking
systems) under lane change scenarios.
Human drivers’ lane change behaviors have been analyzed
and modeled for more than half a century. Early studies based oncontrolled experiments usually have short test horizons andlimited control settings [40]. More recently, researchers started
to use large scale N-FOT databases to model the lane change
behaviors. Lee [40] examined steering, turn signal and brake
pedal usage, eye glance patterns, and safety envelope of 500 lanechanges. The 100-Car Naturalistic Driving Study analyzed lanechange events leading to rear-end crashes and near-crashes [39].
Zhao [41] analyzed the safety critical variables in mandatory anddiscretionary lane changes for heavy trucks [10]. Most of thesestudies are based on hundreds of lane changes. We use the datacollected in the Safety Pilot Model Deployment project, whichcontains hundreds of thousands of lane changes.
Naturalistic lane change events
In this research, we developed a lane change statisticalmodel and demonstrated its use for accelerated evaluation of afrontal collision avoidance algorithm. The database used is the
Safety Pilot Model Deployment (SPMD) database [42]. The
SPMD program aims to demonstrate connected vehicle
technologies in a real-world environment. It recorded naturalistic
driving of 2,842 equipped vehicles in Ann Arbor, Michigan for
more than two years. As of April 2015, 34.9 million miles were
logged, making SPMD one of the largest public N-FOT
databases ever.

Fig. 1. Lane change scenarios that may cause frontal crashes

As shown in Fig. 1, a lane change was detected and recorded
by a SPMD vehicle when the Lane Change Vehicle (LCV)
crosses the lane markers. In the SPMD program, 98 sedans are2

Copyright © 2015 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/ on 03/10/2017 Terms of Use: http://www.asme.org/aboequipped with Data Acquisition System with MobilEye® [43],which provides a) the relative position (range), and b) lane
tracking measures pertaining to the lane delineation both fromthe painted boundary lines and road edge characteristics. The
accuracy of MobilEye® for range and speed measurement was
examined by comparing with a 77 GHz radar over 660 km (412
miles) of driving on a variety of road types, weather, and ambient
light conditions. It was found that the MobilEye® provides
measures similar to the radar when the range is shorter than 75m.
The following criteria were applied to ensure consistency of
the used dataset:𝑣(𝑡𝐿𝐶 ) ∈ (2 m/s, 40 m/s)𝑣𝐿 (𝑡𝐿𝐶 ) ∈ (2 m/s, 40 m/s)

the histogram. 𝑣𝐿 is assumed to remain constant during theexecution of lane change. Only the events with negative range
rate is used to build the lane change model. Out of 403,581 lanechange events, 173,692 are with negative range rate.

 𝑅𝐿 (𝑡𝐿𝐶 ) ∈ (0.1 m, 75 m)
where 𝑡𝐿𝐶 is the time when the center line of the LCV crosses thelane markers; 𝑣𝐿 and 𝑣 are the velocities of the LCV and SPMD
vehicle; 𝑅𝐿 is the range, defined as the distance between the rearedge of the LCV and the front edge of the SPMD vehicle; 𝑇𝑇𝐶(Time To Collision) is defined as
𝑅𝐿
𝑇𝑇𝐶 = −
(1)𝑅̇𝐿
where 𝑅̇𝐿 is the derivative of 𝑅𝐿 . 403,581 lane changes weredetected. Fig. 2 shows the locations of the identified lanechanges.

Fig. 3. Distributions of 𝑣𝐿 (𝑡𝐿𝐶 ) of the lane change events used the our model

To capture the influence of vehicle speed on the range andTTC, we divided lane change events into low, medium and highvelocity conditions. Fig. 4 shows that 𝑣𝐿 has little influence on
the distribution of 𝑅𝐿−1 . Fig. 5 illustrates the fitting of 𝑅𝐿−1 with
Pareto distribution
𝑓𝑅𝐿−1 (𝑥|𝑘𝑅𝐿−1 , 𝜎𝑅𝐿−1 , 𝜃𝑅𝐿−1 )
=

1
𝜎𝑅−1
𝐿

(1 + 𝑘𝑅𝐿−1

𝑥−𝜃 −1𝑅𝐿

𝜎𝑅−1−1−1/𝑘 −1

)

𝑅𝐿

(2)

𝐿

where the shape parameter 𝑘𝑅𝐿−1 , the scale parameter 𝜎𝑅𝐿−1 , andthreshold parameter 𝜃𝑅𝐿−1 are all positive.

Fig. 2. Recorded lane change events queried from SPMD database

Lane change models
Generally, a lane change can be categorized into threephases: decision to initiate a lane change, gap (range)
acceptance, and lane change execution [40]. In this research, we
focused on the effects of gap acceptance, which is mainly
captured by three variables: 𝑣𝐿 (𝑡𝐿𝐶 ), 𝑅𝐿 (𝑡𝐿𝐶 ) and 𝑇𝑇𝐶𝐿 (𝑡𝐿𝐶 ). In
the following contents, unless mentioned specifically, 𝑣𝐿 , 𝑅𝐿
and 𝑇𝑇𝐶𝐿 are the values at 𝑡𝐿𝐶 .
The distribution of 𝑣𝐿 is shown in Fig. 3. The division of
highways and local roads is embodied by the bimodal shape of

Fig. 4. Distributions of 𝑅𝐿−1 (𝑡𝐿𝐶 ) at different vehicle forward speeds

3

Copyright © 2015 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/ on 03/10/2017 Terms of Use: http://www.asme.org/abo

𝑣(𝑡𝐿𝐶 ) = 𝑣𝐿 (𝑡𝐿𝐶 ) − 𝑅̇𝐿 (𝑡𝐿𝐶 )(5)

Fig. 5. Fitting results of 𝑅𝐿−1 (𝑡𝐿𝐶 ) using the Pareto distribution

The histograms of 𝑇𝑇𝐶𝐿−1 for different velocity intervals areshown in Fig. 6. As the vehicle speed increases, the mean of
𝑇𝑇𝐶𝐿−1 decreases. 𝑇𝑇𝐶𝐿−1 can be approximated by an exponential
distributions
𝑓 𝑇𝑇𝐶𝐿−1 (𝑥|𝜆 𝑇𝑇𝐶𝐿−1 ) =1
𝜆 𝑇𝑇𝐶𝐿−1

𝑒

−𝑥/𝜆

−1
𝑇𝑇𝐶𝐿

Fig. 7. Model parameters for 𝑇𝑇𝐶𝐿 (𝑡𝐿𝐶 )(3)

where the scaling factor 𝜆 𝑇𝑇𝐶𝐿−1 varies with the speed of the LCV.

Fig. 8. Distribution of 𝑇𝑇𝐶𝐿 (𝑡𝐿𝐶 ) with variant 𝑅𝐿 (𝑡𝐿𝐶 ) intervals

In summary, the lane change events are generated in thefollowing order: a) generate 𝑣𝐿 based on the empirical
distributions shown in Fig. 3; b) generate 𝑅−1
𝐿 using Fig. 5; c)generate 𝑇𝑇𝐶𝐿−1 using the Exponential distribution withparameters shown in Fig. 7; d) calculate 𝑣(𝑡𝐿𝐶 ) using Eqs.(4) and(5).

Fig. 6. Distribution of 𝑇𝑇𝐶𝐿−1 (𝑡𝐿𝐶 ) at different lane change vehicle speeds

ACCELERATED EVALUATION
Monte Carlo techniques can be used to simulate the driving
conditions using the generated stochastic model, but a naïve
implementation will take a long time to execute. The concept ofaccelerated evaluation is proposed to shorten the simulation
time. In this section, we introduce the Importance Sampling (IS)concept and show how to apply IS to accelerate the evaluation
process.

The dependence of 𝜆 𝑇𝑇𝐶𝐿−1 on vehicle speed is shown in Fig.
7. As vehicle speed increases, 𝜆 𝑇𝑇𝐶𝐿−1 decreases. The blue circles
represent 𝜆 𝑇𝑇𝐶𝐿−1 at the center points of 𝑣𝐿 intervals. We use
linear interpolation and extrapolation to create smooth 𝜆 𝑇𝑇𝐶𝐿−1
for all vehicle speeds.
The effect of range on TTC is shown in Fig. 8. It can be seenthat the influence of range on TTC is very limited. This indicatesthat 𝑅𝐿 and 𝑇𝑇𝐶𝐿 can be modeled independently given thesame 𝑣𝐿 . 𝑅̇𝐿 can then be calculated from

Importance Sampling
IS is based on the concept of variance reduction, which iseffective in handling rare events. IS has been successfully
applied to evaluate critical events in reliability [44], finance [45],insurance [46], and telecommunication networks [47]. General
overviews can be found in [48]–[50].

−1𝑇𝑇𝐶𝑅̇𝐿 = − −1𝐿
𝑅𝐿(4)Finally, velocity of the host vehicle 𝑣 at 𝑡𝐿𝐶 can be calculatedfrom4

Copyright © 2015 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/ on 03/10/2017 Terms of Use: http://www.asme.org/aboand 𝑧𝛼 is defined as

To explain the concept of IS, let ℰ be the rare events ofinterest, and in this paper, frontal crashes with a vehicle cutting
in. The indicator function of the event ℰ is defined as
1, 𝑖𝑓 𝒙 ∈ ℰ
𝐼ℰ (𝒙) = {0, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒𝑧𝛼 = Φ −1 (1 − 𝛼/2)

where Φ−1 is the inverse cumulative distribution functionof 𝒩(0,1). Given that the requirement for estimation accuracy
is to make 𝑙𝑟 smaller than a constant 𝛽, under accelerated
evaluation, it can be derived that(6)

The random vector 𝒙 represents the motions of the lane change
vehicle. Our task is to estimate the probability of ℰ happening,i.e.
γ ≔ P(ℰ) = E(𝐼ℰ (𝒙))(7)𝑧𝛼 √𝐸𝑓∗ ( 𝐼ℰ2 (𝒙) 𝐿2 (𝒙)) − 𝛾 2
=

𝑛≥

(8)

2
2
𝑧𝛼2 𝐸𝑓∗ ( 𝐼𝐸 (𝒙) 𝐿 (𝒙))
(
− 1)
𝛽2
𝛾2

close to 𝛾 2 , resulting in a smaller number of tests (i.e., the test isaccelerated).
Accelerated evaluation in lane change scenarios
When a slower lane changing vehicle cut-in in front of theAV, the events of interest are defined as(9)= ∫[𝐼ℰ (𝒙)𝐿(𝒙)]𝑓 ∗ (𝒙)d𝒙ℰ = {min(𝑅𝐿 (𝑡)|𝑡𝐿𝐶 ≤ 𝑡 ≤ 𝑡𝐿𝐶 + 𝑇𝐿𝐶 ) ≤ R ℰ }= E𝑓∗ (𝐼ℰ (𝒙)𝐿(𝒙))(10)which guarantees the existence of L in (8). The IS sample is
𝐼ℰ (𝒙𝑖 )𝐿(𝒙𝑖 ) where 𝒙𝑖 is generated under 𝑓 ∗ (𝒙), which is anunbiased estimator for 𝛾. The overall IS estimator for test
number 𝑛 is then

𝐿

𝐿

𝑛

𝐿( 𝑅𝐿−1 = 𝑥, 𝑇𝑇𝐶𝐿−1 = 𝑦) =

(11)

𝑖=0

∗
𝑓𝑅∗ −1 (𝑥)𝑓𝑇𝑇𝐶−1 (𝑦)(18)

𝐿

From (9), the probability of ℰ can be estimated as
P(ℰ) = E𝑓 (𝐼ℰ (𝒙)) = E𝑓∗ (𝐼ℰ (𝒙)𝐿(𝒙))(19)There are many possible choices for the family of alteredprobability density function. Here we use a class of family
named the Exponential Change of Measure (ECM) for 𝑇𝑇𝐶𝐿−1 .
Recall that 𝑇𝑇𝐶𝐿−1 ~exp(𝜆 𝑇𝑇𝐶𝐿−1 (𝑣𝐿 )), i.e.

(12)

where 𝑙𝛼 is the half-width given by𝑙𝛼 = 𝑧𝛼 𝜎(𝛾̂𝑛 )𝑓 𝑅𝐿−1 (𝑥)𝑓 𝑇𝑇𝐶𝐿−1 (𝑦)𝐿

The accuracy of the estimation is represented by the relativehalf-width. With the Confidence level at 100(1 − 𝛼) %, therelative half-width of 𝛾̂𝑛 is defined as
𝑙𝛼
𝛾

(17)

where 𝑇𝐿𝐶 represents duration of the lane change event; R ℰ is thecritical range. Eq. (17) means that if the minimum range issmaller than R ℰ anytime during the lane change event, it isdeclared as a event of interest.
The random vector 𝒙 consists of three variables[𝑣𝐿 , 𝑇𝑇𝐶𝐿−1 , 𝑅𝐿−1 ]. 𝑣𝐿 is generated using empirical distributions
shown in Fig. 3 directly. The IS approach considers the modified
probability density functions of 𝑇𝑇𝐶𝐿−1 and 𝑅𝐿−1 denoted by∗∗𝑓𝑇𝑇𝐶−1 (𝒙) and 𝑓 −1 (𝒙). The likelihood ratio is then𝑅

One required condition for (9) to hold is that 𝑓 ∗ (𝒙) must be
absolutely continuous with respect to 𝑓(𝒙) within ℰ, i.e.

𝑙𝑟 =

(16)

When 𝑓 ∗ (𝑥) is properly chosen, 𝐸𝑓∗ ( 𝐼ℰ2 (𝒙) 𝐿2 (𝒙)) can be= ∫ 𝐼ℰ (𝒙)𝑓(𝒙)𝑑𝒙1𝛾̂𝑛 = ∑ 𝐼ℰ (𝒙𝑖 )𝐿(𝒙𝑖 )𝑛2𝑧𝛼 √𝐸𝑓∗ ( 𝐼ℰ (𝒙) 𝐿 (𝒙))
−1≤𝛽
𝛾2
√𝑛

The required minimum test number is thenP(ℰ) = E𝑓 (𝐼ℰ (𝒙))⇒ 𝑓(𝒙) = 0

(15)

𝛾 √𝑛
2

=

where 𝑓(𝒙) is the original probability distribution.
The probability of ℰ satisfies∀𝑥 ∈ ℰ: 𝑓 ∗ (𝒙) = 022𝑙𝛼 𝑧𝛼 𝜎(𝛾̂𝑛 ) 𝑧𝛼 √𝐸𝑓∗ ( 𝛾̂𝑛 ) − 𝐸𝑓∗ ( 𝛾̂𝑛 )𝑙𝑟 = ==𝛾𝛾
𝛾 √𝑛Denote 𝑓(𝒙) as the joint density function of 𝒙. The core ideaof IS is to replace 𝑓(𝒙) with a new IS density 𝑓 ∗ (𝒙) that has ahigher likelihood of the rare events. Obviously, using a different
distribution like 𝑓 ∗ (𝒙) leads to biased samples, and the key of IS
is to provide a mechanism to compensate for this bias andcomputes correct crash rate at the end.
We describe this mechanism as follows. First, we define theso-called likelihood ratio L (Radon-Nikodym derivative [51]) as𝑓(𝒙)𝐿(𝑥) = ∗
𝑓 (𝒙)

(14)

(13)

5

Copyright © 2015 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/ on 03/10/2017 Terms of Use: http://www.asme.org/abo

𝑡𝑓 𝑇𝑇𝐶𝐿−1 (𝑥) = 𝜆 𝑇𝑇𝐶𝐿−1 exp(−𝜆 𝑇𝑇𝐶𝐿−1 ⋅ 𝑥)

𝐸𝑟𝑟
𝐸𝑟𝑟
𝑎𝑑 (𝑡 + 𝜏) = 𝐾𝑝𝐴𝐶𝐶 𝑡𝐻𝑊(𝑡) + 𝐾𝑖𝐴𝐶𝐶 ∫ 𝑡𝐻𝑊(𝜏) 𝑑𝜏

(20)

0

(28)

ECM considers the family
where 𝑡𝐻𝑊 is the time headway, defined as∗
𝑓𝑇𝑇𝐶−1 (𝑥)
𝐿(21)

𝑡𝐻𝑊 = 𝑅𝐿 /𝑣= exp (𝜗 𝑇𝑇𝐶𝐿−1 𝑥 − 𝛹 (𝜗 𝑇𝑇𝐶𝐿−1 )) 𝑓 𝑇𝑇𝐶𝐿−1 (𝑥)𝑎𝑑 is the acceleration command; gains 𝐾𝑝𝐴𝐶𝐶 and 𝐾𝑖𝐴𝐶𝐶 are
calculated using the Matlab Control Toolbox using the following
requirements: a) Loop bandwidth = 10 rad/s, and b) Phase
margin = 60 degree. The ACC control is saturated at |𝑎𝑑 | <
𝑀𝑎𝑥
𝑎𝐴𝐶𝐶
.
The AEB model was extracted from a 2011 Volvo V60,
based on a test conducted by ADAC (Allgemeiner Deutscher
Automobil-Club e.V.) [52] and its analysis from [53] using test
track data, data found in owner’s manuals, European New Car
Assessment Program (Euro NCAP) information, and videos
during vehicle operation. The AEB algorithm becomes active
when 𝑇𝑇𝐶 < 𝑇𝑇𝐶𝐴𝐸𝐵 , where TTCAEB depends on vehicle speed
as shown in Fig. 9. Once triggered, AEB aims to achieve
acceleration aAEB. In [53] aAEB was assumed to be -10 m/s2
on high friction roads. The build-up of deceleration is subject toa rate limit rAEB = -16 m/s3 as shown in Fig. 10. It should be
noted that the AEB modeled here is an approximation but notnecessarily a good representation of the actual system on
production vehicles.

parametrized by 𝜗 𝑇𝑇𝐶𝐿−1 , where 𝛹 (𝜗 𝑇𝑇𝐶𝐿−1 ) is the logarithmic
moment generation function of 𝑇𝑇𝐶𝐿−1 , i.e.,𝛹 (𝜗 𝑇𝑇𝐶𝐿−1 ) = log E (exp (𝜗 𝑇𝑇𝐶𝐿−1 𝑇𝑇𝐶𝐿−1 ))(22)

It can be further shown that∗𝑓𝑇𝑇𝐶
−1 (𝑥)𝐿(23)= (𝜆 𝑇𝑇𝐶𝐿−1 − 𝜗 𝑇𝑇𝐶𝐿−1 ) exp (− (𝜆 𝑇𝑇𝐶𝐿−1 − 𝜗 𝑇𝑇𝐶𝐿−1 ) 𝑥)
Nominally 𝑅𝐿−1 follows a Pareto distribution, i.e.
𝑓𝑅𝐿−1 (𝑥) = 𝑃𝑎𝑟𝑒𝑡𝑜 (𝑥|𝑘𝑅𝐿−1 , 𝜎𝑅𝐿−1 , 𝜃𝑅𝐿−1 )(24)

The ECM cannot be directly applied to a Pareto distribution
directly. We first construct an exponential distribution

̃ 𝑅−1 (𝑥) = 𝜆 𝑅−1𝑓exp(−𝜆 𝑅−1
𝑥)
𝐿
𝐿
𝐿

(29)

(25)

with 𝜆 𝑅𝐿−1 , which was optimized to make (25) to have thesmallest least square error to (24). Then we apply ECM to (25)𝑓𝑅∗ −1 (𝑥) = (𝜆 𝑅𝐿−1 − 𝜗 𝑅𝐿−1 ) exp (− (𝜆 𝑅𝐿−1 − 𝜗 𝑅𝐿−1 ) 𝑥)𝐿(26)

𝜗 𝑅𝐿−1 and 𝜗 𝑇𝑇𝐶𝐿−1 were chosen to make 𝑃(ℰ) converge fast.
Different parameters were applied in low, medium and highvelocity conditions. The parameters we used were shown in
Table I.
TABLE I
𝜗 𝑅𝐿−1 AND 𝜗 𝑇𝑇𝐶𝐿−1 VALUES𝑣𝐿𝜗 𝑅𝐿−1𝜗 𝑇𝑇𝐶𝐿−1

5 − 15 [𝑚/𝑠]
-0.1
-0.5

15 − 25 [𝑚/𝑠]-0.1-1Fig. 9. 𝑇𝑇𝐶𝐴𝐸𝐵 as a function of vehicle speed
25 − 35 [𝑚/𝑠]
-0.1
-1.5

Now the probability of ℰ can be estimated by using (11) with 𝐿in (18) and modified distributions in (23) and (25).
SIMULATION ANALYSISThe AV is assumed to be equipped with both Adaptive
Cruise Control (ACC) and Autonomous Emergency Braking(AEB). When the driving is perceived to be safe (𝑇𝑇𝐶 ≥𝑇𝑇𝐶𝐴𝐸𝐵 ), it is controlled by ACC. The ACC is approximated by
𝐴𝐶𝐶
a PI controller to achieve a desired time headway 𝑇𝐻𝑊
𝑑
𝐴𝐶𝐶
𝐸𝑟𝑟
𝑡𝐻𝑊
=𝑡𝐻𝑊 − 𝑇𝐻𝑊𝑑Fig. 10. The modeled AEB algorithm

To accurately model the dynamics of AV, many subsystems
need to be considered, such as the engine, transmission, braking,aerodynamics, ABS (Anti-lock Braking System), etc. For
simplicity, a first order lag with a time constant 𝜏𝐴𝑉 is used tomodel the transfer function from the commanded acceleration to
the actual acceleration. The accelerated evaluation process can

(27)

6

Copyright © 2015 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/ on 03/10/2017 Terms of Use: http://www.asme.org/abo

be applied on other vehicle models such as CarSim if moreaccurate simulations are desired.
The crash event ℰ𝑐 is occurred as 𝑅𝐿 becomes negative
within 𝑇𝐿𝐶 time after 𝑡𝐿𝐶 , from (17),ℰ𝑐 = {ℰ|R ℰ = 0}

Relative half-width is used as the criteria for convergence.
The evaluation stops when the relative half-width of theestimation is below 𝛽 = 0.25. Fig. 12 shows that the accelerated
evaluation achieves this threshold level after 24,101 simulations,
145 times faster than the naturalistic simulations.

(30)

CONCLUSION
This paper proposes an approach to evaluate theperformance of AVs in an accelerated fashion. By modifying thestochastic behaviors of the primary other vehicle (POV), in thiscase a lane changing vehicle driving at a slower speed than thehost vehicle (the AV), higher-risk driving behaviors are sampled
more frequently. This method is based on the importancesampling theories and have been developed to ensure unbiased
and accurate evaluation under accelerated conditions. Simulation
results show that the proposed method accelerates the evaluation
procedure by about 145 times. Possible future work includes
achieve high acceleration ratio, extending the methodology todynamic driving conditions, and a more systematic way to selectthe modified probability density function for unbiased results.

To estimate 𝑃(ℰ𝑐 ), both accelerated and naturalistic
simulations were conducted. Simulation parameters are shownin Table II.
TABLE IIPARAMETERS FOR THE LANE CHANGE SIMULATIONS
Variable

𝐴𝐶𝐶
𝑇𝐻𝑊
𝑑

𝑀𝑎𝑥
𝑎𝐴𝐶𝐶

𝜏𝐴𝑉

𝑎𝐿 (𝑡𝐿𝐶 )𝑎(𝑡𝐿𝐶 )

𝑇𝐿𝐶

Unit
Value

s
2

m/s2
5

s
0.0796

m/s2
0

m/s2
0

s
8

Road
friction1

The crash rate is calculated after each simulation and theresults are shown in Fig. 11. The naturalistic tests and accelerated
tests were plotted on different scales. It can be seen that theaccelerated evaluation successfully estimates the crash rate of
the naturalistic conditions.

ACKNOWLEDGMENTS
The authors would like to acknowledge the generous
sponsorship of the National Institute for Occupational Safety and
Health (NIOSH).
DISCLAIMERS
The findings and conclusions in the report are those of theauthor(s) and do not necessarily represent the views of the
National Institute for Occupational Safety and Health (NIOSH).
Mention of company names or products does not imply
endorsement by NIOSH.
REFERENCES[1] C. Lewis, “The Economic Benefits of Driverless Cars,”MorganStanley,
2014.
[Online].
Available:
http://robotenomics.com/2014/02/26/morgan-stanley-theeconomic-benefits-of-driverless-cars/. [Accessed: 22-Aug2014].
[2] C. Laslau, “Set Autopilot for Profits: Capitalizing on the$87 Billion Self-Driving Car Opportunity,” Lux Research,
2014.
[3] J. Treat, N. Tumbas, and S. McDonald, “Tri-level study ofthe causes of traffic accidents: final report,” DOT HS 034
3 535, 1979.
[4] L. Blincoe, T. R. Miller, E. Zaloshnja, and B. A. Lawrence,“The Economic and Societal Impact Of Motor Vehicle
Crashes, 2010 (Revised),” NHTSA, DOT HS 812 013,
2015.
[5] FESTA-Consortium, “FESTA Handbook Version 2
Deliverable T6.4 of the Field opErational teSt supporT
Action,” Brussels: European Commission, 2008.
[6] M. Aust, “Evaluation Process for Active Safety Functions:
Addressing Key Challenges in Functional, Formative

Fig. 11. Convergence of the estimation of lane change crash rate

Fig. 12. Relative half width of crash events

7

Copyright © 2015 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/ on 03/10/2017 Terms of Use: http://www.asme.org/abo

[21] K. Deering, Richard, “Crash Avoidance Metrics
Partnership Annual Report, April 2001 - March 2002,”
DOT HS 809 531, 2002.
[22] O. Carsten, N. Merat, W. H. Janssen, E. Johansson, M.
Fowkes, and K. A. Brookhuis, “Human MachineInteraction and the Safety of Traffic in Europe Final
Report,” 2005.
[23] H. Kussmann, H. Modler, J. Engstrom, A. Agnvall, P.
Piamonte, G. Markkula’s, A. Amditis, A. Bolovinou, L.
Andreone, E. Deregibus, and P. Kompfner, “Requirements
for AIDE HMI and safety functions,” D3.2.1, 2004.
[24] V. Karabatsou, M. Pappas, P. van Elslande, K. Fouquet,and M. Stanzel, “A-priori evaluation of safety functions
effectiveness - Methodologies Table of Contents,” D4.1.3,
2007.
[25] T. Wohllebe, J. Vetter, C. Mayer, M. McCarthy, and R. de
Lange, “Integrated Project on Advanced Protection
Systems,” AP-SP13-0035 Project, 2004.
[26] J.-A. Bühne, A. Lüdeke, S. Schönebeck, J. Dobberstein, H.
Fagerlind, A. Bálint, and M. McCarthy, “Assessment ofIntegrated Vehicle Safety Systems for improved vehicle,”
ASSESS D2.2 (2/2) Socio-economic, 2012.
[27] W. G. Najm and J. D. Smith, “Development of Crash
Imminent Test Scenarios for Integrated Vehicle-Based
Safety Systems,” DOT HS 810 757, 2007.
[28] “NASS General Estimates System | National Highway
Traffic Safety Administration (NHTSA).” [Online].
Available:
http://www.nhtsa.gov/Data/National+Automotive+Sampli
ng+System+(NASS)/NASS+General+Estimates+System.
[Accessed: 04-Sep-2014].
[29] “NCSA Publications & Data Requests.” [Online].
Available:
http://wwwnrd.nhtsa.dot.gov/Cats/listpublications.aspx?Id=227&Sho
wBy=Category. [Accessed: 04-Sep-2014].
[30] “Event Data Recorder (EDR) | National Highway Traffic
Safety Administration (NHTSA).” [Online]. Available:
http://www.nhtsa.gov/EDR. [Accessed: 04-Sep-2014].
[31] D. LeBlanc, M. Gilbert, S. Stachowski, and R. Sherony,“Tradeoffs in the Evaluation of Light Vehicle Pre-Collision
Systems,” in SAE 2014 World Congress & Exhibition,
2014.
[32] H. Peng and D. Leblanc, “Evaluation of the performanceand safety of automated vehicles,” White Pap. NSF Transp.
CPS Work., 2012.
[33] W.-H. Ma and H. Peng, “A Worst-Case Evaluation Method
for Dynamic Systems,” J. Dyn. Syst. Meas. Control, vol.
121, no. 2, p. 191, 1999.
[34] A. Ungoren and H. Peng, “An adaptive lateral preview
driver model,” Veh. Syst. Dyn., vol. 43, no. 4, pp. 245–259,
Apr. 2005.
[35] Y. Kou, “Development and evaluation of integrated chassis
control systems,” The University of Michigan, 2010.
[36] Y. Fujii and R. Shiobara, “The Analysis of Traffic
Accidents,” J. Navig., vol. 24, no. 04, p. 534, Jan. 2010.

Evaluation of Advanced Driver Assistance Systems,”
Chalmers University of Technology, 2012.
[7] D. LeBlanc, J. Sayer, C. Winkler, R. Ervin, S. Bogard, M.
Devonshire, J. Mefford, M. Hagan, Z. Bareket, R.
Goodsell, and T. Gordon, “Road Departure Crash Warning
System Field Operational Test : Methodology and
Results,” UMTRI-2006-9-2, 2006.
[8] M. Perez, L. S. Angell, J. Hankey, R. K. Deering, R. E.
Llaneras, C. A. Green, M. L. Neurauter, and J. F. Antin,“Advanced Crash Avoidance Technologies ( ACAT )Program – Final Report of the GM-VTTI Backing Crash
Countermeasures Project,” DOT HS 811 452, 2011.
[9] T. Victor, J. Bärgman, M. Hjälmdahl, and K. Kircher,“Sweden-Michigan Naturalistic Field Operational Test
( SeMiFOT ) Phase 1 : Final Report,” 2010.
[10] J. Sayer, D. LeBlanc, S. Bogard, D. Funkhouser, F.
Operational, T. Final, and P. Report, “Integrated VehicleBased Safety Systems Field Operational Test Final
Program Report,” NHTSA, DOT HS 811 482, 2011.
[11] T. Gordon, H. Sardar, D. Blower, M. Ljung Aust, Z.
Bareket, M. Barnes, A. Blankespoor, I. Isaksson‐ Hellman,
J. Ivarsson, B. Juhas, K. Nobukawa, and H. Theander,“Advanced Crash Avoidance Technologies ( ACAT )Program – Final Report of the Volvo-Ford- UMTRI
Project : Safety Impact Methodology for Lane Departure
Warning – Method Development And Estimation ofBenefits,” DOT HS 811 405, 2010.
[12] Wikipedia, “Google driverless car.” [Online]. Available:
http://en.wikipedia.org/wiki/Google_driverless_car.
[13] C. Kessler and A. Etemad, “Deliverable D11.3 Final
Report.”[14] J. S. Hickman and R. J. Hanowski, “An assessment ofcommercial motor vehicle driver distraction using
naturalistic driving data.,” Traffic Inj. Prev., vol. 13, no. 6,
pp. 612–9, 2012.
[15] T. A. Dingus, S. G. Klauer, V. L. Neale, A. Petersen, S. E.
Lee, J. Sudweeks, M. A. Perez, J. Hankey, D. Ramsey, S.
Gupta, C. Bucher, Z. R. Doerzaph, J. Jermeland, and R. .
Knipling, “The 100-Car Naturalistic Driving Study Phase
II – Results of the 100-Car Field Experiment,” NHTSA,DOT HS 810 593, 2006.
[16] NHTSA, “Traffic Safety Facts 2012,” 2012.
[17] M. Akamatsu, P. Green, and K. Bengler, “Automotive
technology and human factors research: Past, present, andfuture,” Int. J. Veh. Technol., vol. 2013, pp. 1–27, 2013.
[18] H.-H. Yang and H. Peng, “Development and evaluation ofcollision warning/collision avoidance algorithms using anerrable driver model,” Veh. Syst. Dyn., vol. 48, no. sup1,
pp. 525–535, Dec. 2010.
[19] K. Lee, “Longitudinal driver model and collision warningand avoidance algorithms based on human driving
databases,” University of Michigan., 2004.
[20] Euro NCAP, “Euro NCAP test protocol – AEB systems,”
2013.

8

Copyright © 2015 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/ on 03/10/2017 Terms of Use: http://www.asme.org/abo[37] F. Goerlandt and P. Kujala, “Traffic simulation based ship
collision probability modeling,” Reliab. Eng. Syst. Saf.,
vol. 96, no. 1, pp. 91–107, Jan. 2011.
[38] D. Zhao, H. Peng, S. Bao, K. Nobukawa, D. J. LeBlanc,
and C. S. Pan, “Accelerated evaluation of automatedvehicles using extracted naturalistic driving data,” inProceeding for 24th International Symposium of Vehicles
on Road and Tracks, 2015.
[39] G. M. Fitch, S. E. Lee, S. Klauer, J. Hankey, J. Sudweeks,
and T. Dingus, “Analysis of Lane-Change Crashes andnear-Crashes,” US Dep. Transp. Natl. Highw. Traffic Saf.
Adm., 2009.
[40] S. E. Lee, C. B. E. Olsen, W. W. Wierwille, NHTSA, and
D. O. T. Hs, “A Comprehensive Examination of
Naturalistic Lane-Changes,” no. March, 2004.
[41] D. Zhao, H. Peng, K. Nobukawa, S. Bao, D. J. LeBlanc,
and C. S. Pan, “Analysis of Mandatory and Discretionary
Lane Change Behaviors for Heavy Trucks,” in 12th
International Symposium on Advanced Vehciel Control,
2014.
[42] D. LeBlanc, “Data from the Safety Pilot Model
Deployment.”[Online].
Available:
http://assets.conferencespot.org/fileserver/file/46778/filen
ame/39ftvo.pdf. [Accessed: 22-Aug-2014].
[43] J. Harding, G. Powell, R. Yoon, and J. Fikentscher,“Vehicle-to-vehicle communications: Readiness of v2v
technology for application,” 2014.
[44] P. Heidelberger, “Fast simulation of rare events inqueueing and reliability models,” ACM Trans. Model.
Comput. Simul., vol. 5, no. 1, pp. 43–85, Jan. 1995.
[45] P. Glasserman and J. Li, “Importance Sampling forPortfolio Credit Risk,” Manage. Sci., Nov. 2005.
[46] S. Asmussen and H. Albrecher, Ruin probabilities, 2nd ed.
World Scientific, 2010.
[47] C.-S. Chang, P. Heidelberger, S. Juneja, and P.
Shahabuddin, “Effective bandwidth and fast simulation of
ATM intree networks,” Perform. Eval., vol. 20, no. 1–3,
pp. 45–65, May 1994.
[48] J. Bucklew, Introduction to rare event simulation. Springer
Science & Business Media, 2013.
[49] J. Blanchet and H. Lam, “State-dependent importancesampling for rare-event simulation: An overview andrecent advances,” Surv. Oper. Res. Manag. Sci., vol. 17, pp.
38–59, 2012.
[50] S. Juneja and P. Shahabuddin, “Chapter 11 Rare-Event
Simulation Techniques: An Introduction and Recent
Advances,” Handbooks Oper. Res. Manag. Sci., vol. 13,
no. 06, pp. 291–350, 2006.
[51] H. L. Royden, Real Analysis. New York: Prentice Hall,1948.
[52] ADAC, “Comparative test of advanced emergency braking
systems,” 2011.
[53] T. I. Gorman, “Prospects for the collision-free car : The
effectiveness of five competing forward collision
avoidance systems,” Virginia Polytechnic Institute andState University, 2013.

9

Copyright © 2015 by ASME

Downloaded From: http://proceedings.asmedigitalcollection.asme.org/pdfaccess.ashx?url=/data/conferences/asmep/86561/ on 03/10/2017 Terms of Use: http://www.asme.org/abo