Instantaneous Safety Metric

06. 25. 2019

Frank Barickman – NHTSA

Josh Every – TRC

Bowen Weng – TRC

Scott Schnelle – TRC

Sughosh Rao - TR
Overview Definitions

Challenges in ADS testing

Introduction of concept.

Explanation of current implementation.

1

2

3

4

5

Conclusions from current

development and ongoing research.
Definition

Instantaneous Safety Metric (ISM)

• A measurement of performance not only related to the 

outcome of a situation, but also related to the probably of 

a collision occurring had actions varied.

3
4

Traditional Test Procedure Development

Identify a safety 

problem

Identify a 

technology that 

can mitigate or 

reduce the 

safety problem

Develop a set of 

scenarios, in 

which the 

system is 

desired to 

improve safety

Test systems in 

these scenarios

Develop criteria 

to classify 

system 

performance
Challenges

5

Automated Driving Systems (ADS) have a goal of proceeding 

through an environment safely. There are two main challenges 

in determining if an ADS will consistently accomplish this 

goal.

1. Environments are infinitely variable

–This can be overcome by either performing a large 

amount of real world testing, or through simulation.

2. Safety is not simply a result.

–Just because a collision didn’t occur doesn’t mean an 

ADS actions were safe. 
Driving Thoughts

6

• Suppose you are driving down the road and you ask yourself, “If 

something were to happen right now and I needed to avoid it what 

could I do?”

• Upon reviewing your situation you arrive at the conclusion that there 

are four basic options you can choose from.

–Brake up to full vehicle capability.

–Accelerate up to full vehicle capability.

–Steer up to full capability to the left.

–Steer up to full capability to the right.

• After realizing these options it is next natural to ask yourself, “If I do 

this where will I end up?”
7

Simplifying Assumptions in this prediction

• We are only looking a short distance into the future.

–The same accelerations are maintained for the complete interval.

• Vehicle reaction time and jerk limits are ignored.

–The selected acceleration is achieved from the Time=0.

• The vehicle behaves according to the defined model.

–This is assumed to be true even for very high accelerations.
Possible Outcomes – Full Acceleration

8

TToo TToo + ∆TT
Possible Outcomes – Full Deceleration

9

TToo TToo + ∆TT
Possible Outcomes – Full Left

10

TToo TToo + ∆TT
Possible Outcomes – Full Right

11

TToo TToo + ∆TT
Interesting Regions

12

TToo TToo + ∆TT
Extension of Result to Reality

14

•In real situations the set of all extreme actions is not limited to these four 

cases.

•Applying any combination of these actions will produce a result which is 

related to the proportion by which these actions are pursued.

•If we look at the profiles defined by all possible combinations of actions, 

the facts of the previous example remain.

•There exists a set of possible locations which the vehicle can occupy at a 

certain point in the future.

•If there exists an object which intersects all of these profiles at some 

point in the future a collision cannot be avoided.
Can We Calculate this Region of Possibility?

13

AAXXMMMMMM

AAYYMMMMMM

(AAXXnn, AAyynn)

(AAXXnn, −AAyynn)

•The set of possible accelerations (Ax and Ay) can be represented by and ellipse.

•Given any value of Ax there exist two corresponding values of Ay.
15

•If we use a model that can be adapted to each Ay, Ax pair we can find the 

position and orientation given these inputs at some point in the future 

(Reachable Set).

•The set of profiles is constructed by placing the vehicle’s outline according to 

the Reachable Set.

•The union of all profiles is the region of space that contains all points that may

interact with a part of the vehicle.

•The intersection of all profiles is the region of space that contains all points that 

must interact with a part of the vehicle.

Can We Calculate this Region of Possibility?
Basic Definitions Related to this Method

16

•Possible Space

•Region of space where some part of the vehicle may exist, 

at a specific point in the future.
17

•Unavoidable Space

•Region of space where some part of the vehicle must 

exist, at a specific point in the future.

Basic Definitions Related to this Method
18

Summary of Implementation

Compute all possible locations at 

the desired future point in time.

•A bicycle model is used to calculate 

positions.

•Motion is governed by either lateral 

acceleration or road wheel angle limits.

•Acceleration limits calculated using an 

ellipse defined by AAXXMMMMMM and AAYYMMMMMM

Place a rectangular 

representation of the vehicle 

at all possible points.

•Using geometric operations the 

possible and unavoidable space can 

be found.

•The possible space is the union of 

all rectangles

•The unavoidable space is the 

intersection of all rectangles.
19

Interactions between possible and unavoidable

spaces for multiple vehicles

•These spaces on their own are interesting but not particularly useful.

•By analyzing the overlap of these spaces with those from other vehicles 

complex traffic interactions can be analyzed.

•There are four possible combinations resulting from interaction between 

the possible and unavoidable spaces of two vehicles (Vehicles A & B in this 

case).

1.The possible space of both vehicles overlap. (Possible Interaction)

2.The unavoidable spaces of both vehicles overlap. (Imminent 

Interaction)

3.The unavoidable space of Vehicle A overlaps the possible space of 

Vehicle B. (Critical interaction for Vehicle A)

4.The possible space of Vehicle A overlaps the unavoidable space of 

Vehicle B. (Critical Interaction for Vehicle B)
20

Possible Interactions

•All cases detected by finding overlap between two vehicles’ 

possible spaces.

•Indicates that a collision between these two vehicles is 

possible at some point in time.

•As time → ∞ the possible space becomes infinitely large. 

Therefore a possible interaction occurs for all vehicles for a 

long enough look ahead time.
21

Imminent Interaction

•All vehicle profiles associated with the subject vehicles 

reachable set intersect all members of the traffic vehicle’s(s’) 

reachable set. 

•Indicates that the situation has progressed to the point where 

a collision is imminent. 

•Assuming that the vehicles behave according to their models 

a collision will occur at some point in the future.
22

Critical Interactions

•Risk

•“A situation involving exposure to danger.” (Google)

•In this work we look to detect risk by finding situations where the occurrence 

(or lack) or a collision, is not longer controlled by the subject vehicle.

•Critical Interactions

•An interaction between the subject vehicle and a POV, or group of POVs, in 

which the POV(s) can perform an action that the subject cannot avoid.

•This type of scenario aligns with our general definition of risk in that we have 

limited influence on the outcome of this situation.

•Given knowledge of POV behavior the level of risk the vehicle has been 

exposed to can be calculated
Steps in Detection of Critical Interactions

23

1.If there is no interaction between the possible space of two 

vehicles then an interaction cannot be critical.

•Allow certain cases to be ruled out and saves time in detection.

2.If a POV’s possible space intersects the subjects unavoidable space 

the interaction is critical.

•This is a sufficient but not necessary condition for non-point objects. 

Therefore it only detects a subset of the desired cases but may be more 

computationally efficient.
Detection of Critical Interactions (cont.)

24

•Detect if there exists a POV profile (or set of profiles for multiple 

POVs) that intersects profiles for all subject vehicle actions.

•An efficient algorithm is currently being developed to address 

these cases.
Method Flow Diagram

25

Acceleration Map

• Provides a set of (Ax, Ay) pairs achievable by the vehicle which 

will be used for vehicle position calculations.

Vehicle Modeling

• Utilizes the desired acceleration pairs along with vehicle size 

parameters to calculate possible vehicle trajectories.

Interaction Classification

• Tests the set of profiles from the prior step to determine the 

current interaction classification.

Severity Analysis

• Reviews possible, critical and imminent interactions to quantify 

their severity.
Concluding Remarks

26

•A framework for evaluating vehicle ADS Performance, proving ground and real 

world environments has been created.

•This method focuses on identifying situations in which the automation system 

has placed the vehicle in a position where the actions of other drivers are 

governing the outcome.

•The approach to constructing this framework allows various pieces to be 

changed without the complete method needing revised.

•Going forward this method can be utilized to compare various automation 

systems as they become available in a wide variety of situations and 

environments.