Workshop on Consensus Safety Measurement Methodologies

for ADS Equipped Vehicles

June 25, 2019

Development of Safety Testing for 

Automated Driving Systems 

(ADS) Equipped Vehicles

Michelle Chaka, VTTI Program Director
The question of “when” ADS technology will 

be deployed has changed to “where”

• Technically, most companies are just testing, but...

• This testing involves more than engineers monitoring ADS performance and 

tracking disengagements:

• Waymo launched a driverless service in the Phoenix area and is testing in 

Kirkland, Atlanta, Detroit, Austin and multiple areas near Mountain View.

• Ford is testing pizza delivery in Miami with additional testing in Dearborn, 

Miami, Pittsburgh, DC and plans for testing in Austin.

• APTIV and Lyft have a partnership “driving” riders in Las Vegas

• Nuro is delivering groceries in Scottsdale.

• GM plans to pilot a food delivery service in San Francisco this year with testing 

also taking place in Warren and Scottsdale.
Some Milestones in the U.S. Traffic 

Environment

FOOTER CHANGED UNDER INSERT>HEADER & FOOTER

1885 First seat belt patent is awarded

1901 First statewide traffic laws enacted in Connecticut

1910 First drunk driving laws enacted in New York

1953 Motorcycle Helmet developed

1978 First consumer information program: Frontal Impact Testing

2003 National implementation of the

Click It or Ticket seat belt enforcement program 2005 all US States, the District of Columbia,

and Puerto Rico have enacted

.08 BAC laws

2007 Electronic Stability Control (ESC) 

mandated on passenger and light-duty vehicles

2011 Updates to Child Safety recommendations

to be categorized by age rather than type of child seat.

2016 Seat belt use achieved an all-time high rate of 90%

1934 First motor vehicle crash test

1995 49 of the 50 states have mandatory seatbelt laws

1911 First documented use of a painted center line used for road surface marking

1907 First national

road inventory published

(for 1904)

1966 First Botts’ Dots were installed in California

1952 First rumble strips

implemented in New Jersey 

1920 Four-way, three-color traffic

signal tower, installed at an intersection

1968 First seat belt law

1938-1940 Pedestrian Walk/Don't Walk Signal 

1967 FMVSS No. 209, Seat Belt Assemblies, 

was the first FMVSS

2015 Automated Emergency Braking MOU Signed
Development of Safety Testing for ADSs

The development of safety testing for ADSs is needed and will not happen 

overnight

...establishing a thoughtful methodology that allows testing to evolve with the 

technology will be critical to addressing the complexity of ADSs safety testing.
Development of Safety Testing for ADSs

Testing 

Framework

Data and Analysis for 

Framework Implementation

Real-world 

Deployment 

Considerations

Video Removed
Development of Safety Testing for ADSs: 

Testing Framework

NHTSA’s Testable Cases and Scenarios for Automated Driving Systems created a framework 

for describing an ADS test scenario; however, more research is needed to identify testable 

cases and associated test architecture. 

ADS Test 

Scenario

Tactical / 

Operational 

Maneuver

OEDR 

Behavior

Fault / 

Failure 

Behavior

ODD 

Elements

Reference: 

Thorn, E., Kimmel, S., and Chaka, M. (2018, September). A framework for automated driving system testable cases and scenarios (Report No. DOT HS 812 623). Washington, DC: National Highway Traffic Safety 

Administration. URL: https://www.nhtsa.gov/sites/nhtsa.dot.gov/files/documents/13882-automateddrivingsystems_092618_v1a_tag.pdf
Development of Safety Testing for ADSs: Data 

and Analysis for Framework Implementation

Naturalistic Driving Data

U.S. National Accident Data

Field Operational Data

Rules of the Road

Safety testing (specifically identifying test cases) starts with the data and knowledge 

we have today.

Case Type

(e.g., Maneuver Based) 

Operational Design Domain
An industry partnership promoting the development 

of tools, techniques, and data resources to support 

the rapid advancement of automated-vehicle deployment 

for its members.
Development of Safety Testing for ADSs: 

Real-world Cases, Analytics and Tools

VTTI’s AMP Program is working to support the rapid advancement of ADS deployment. Speed at reference

Library of Cases

Interactive 

Analytics 

Tool for the 

Development and 

Evaluation of ADSs

Case 

Reconstruction
AMP Near Crash Left Turn Across Path From 

Opposite Directions Example Case
AMP Near Crash Left Turn Across Path From 

Opposite Directions Example Case
SAE INTERNATIONAL

Copyright © SAE International. Further use or distribution is not permitted without permission from SAE

FOOTER CHANGED UNDER INSERT>HEADER & FOOTER

12

Development of safety testing for ADSs: Data and tool elements
Development of Safety Testing for ADSs: Data 

and Analysis for Framework Implementation

The complexity of the ADS and potential test cases may require a multifaceted testing 

architecture.

simulation

open road

test track
Coping with Variability and Uncertainty

No two work zones are 

the same.

Human drivers, even safe 

ones, are highly variable.

Vulnerable road user 

interactions add to 

unpredictability and error 

severity. 

Operational design domains can be limited, but most ADSs will still need to operate in 

highly dynamic domains (e.g., environmental, situational).
Real World Deployment Considerations

An innovative approach is needed for the deployment of ADSs. 

Safety testing alone may not fully ensure safe, robust, and reliable ADS technologies. 

Some examples for manufacturers to consider include:

Real World Deployment 

Best Practices

Safety Testing 

Methodologies for ADSs 
Development of Safety Testing for ADSs

All elements needed for the development of ADS safety testing are in place, but they 

must evolve rapidly and be universally agreed upon. 

Progress will occur 

one element at a time
Thank you

Advancing Transportation Through Innovation
References for Milestones in the U.S. Traffic 

Environment

1885 The First Seat Belt: The first seat belt patent is secured by Edward J Claghorn of 

New York.

1901 Statewide Traffic Laws: Connecticut creates the first statewide traffic laws. The 

new laws limit motor vehicle speed to 12 mph in cities and 15 mph on country roads.

1910 Drunk Driving: New York introduces the first drunk driving laws, penalizing drivers 

for operating a vehicle while under the influence of alcohol.

1953: Motorcycle Helmet developed by University of Southern California Professor 

C.F. Lombard. Designed to absorb the shock of impact. Patent awarded (later?)

1978 Frontal Impact Testing: The first consumer information program on vehicle safety 

is established and a 35 mph frontal crash test is administered.

2003: The Click It or Ticket seat belt enforcement program goes nantional, 

working to increase seat belt use in all 50 states.

2005 .08 BAC Laws: By 2005, all US States, the District of Columbia, and 

Puerto Rico have enacted .08 per se laws.

2007 Electronic Stability Control (ESC) is mandated on all passenger and 

light-duty vehicles.

2011 Updates to Child Safety recommendations to be categorized by age 

rather than type of child seat.

2016 Seat belt use achieved an all-time high rate of 90 percent (DOT HS 812 662)

1934 General Motors conducts the first vehicle crash test in Milford, Michigan. (barrier 

test) https://www.titlemax.com/resources/a-chronology-of-car-safety/ and other

1939 Buick introduces electric turn signals, replacing hand signals.

https://www.titlemax.com/resources/a-chronology-of-car-safety/

1995 49 of the 50 states have mandatory seatbelt laws

https://en.wikipedia.org/wiki/Seat_belt_laws_in_the_United_States

1911 A painted line along Trenton's River Road in Wayne County, Michigan is the first 

documented use of a painted center line used for road surface marking

https://en.wikipedia.org/wiki/Edward_N._Hines

1907 First national road inventory published (for 1904).

https://www.fhwa.dot.gov/publications/publicroads/96spring/p96sp44.cfm

1966 the first Botts’ Dots were installed on Interstate 80 around Fairfield and on 

Highway 99 near Fresno (CalTrans)

https://www.snopes.com/fact-check/botts-dots/

1952 Rumble strips first implemented on the Garden State Parkway in New Jersey

https://en.wikipedia.org/wiki/Rumble_strip#History 

In 1920, William Potts designed the first four-way, three-color traffic signal tower, 

which was installed at the intersection of Woodward and Michigan Avenues in Detroit 

in October 1920

https://en.wikipedia.org/wiki/William_Potts_(inventor)

1968 First seat belt law required all vehicles (except buses) to be fitted with seat belts 

in all designated seating positions.

https://en.wikipedia.org/wiki/Seat_belt_laws_in_the_United_States

1938-1940: Pedestrian Walk/Don't Walk Signal installed at intersection

https://www.fhwa.dot.gov/infrastructure/walk.cfm

Ref for blue font items: https://one.nhtsa.gov/nhtsa/timeline/index.html