Room C Breakout: Blanco/Weast

• Moderators: Myra Blanco and Jack Weast

• Note Takers: Simone Wilson and Ryan Wee

• Rapporteur: Kelley Coyner
Question 1: What are appropriate definitions of ‘safety’ 

in a measurement context, including whether it may be a 

system measure, a component measure (hardware, 

software, etc.), a behavior/performance measure, or 

some combination of these?

• Scope: Level 4+ 

• Avoidance of unreasonable risk in a predictable manner

• Use ISO 26262 definition as a starting point, but the standard needs to be 

expanded to ADS

• ODD based

• The measurement should be at the system and behavior level
Question 2: Is there a need for widely‐adoptable 

measurement methods for ADS‐equipped vehicle safety? 

Are there risks in not pursuing such methods? If so, what 

are some examples?

• Yes, but it should be ODD based

•National level framework

• Consider regional needs
Question 3: What are possible safety measurement 

methods (simulation, test track, on‐road, etc.)? What are 

possible safety metrics (miles driven, pass/fail vs. formal 

model, etc.)?

• Methods and metrics should be ODD-based

• Methods

• Public Assurance: Visible and measurable tests (test tracks and on-road)

• Simulation: Development Tool vs Safety-Assessment

• Metrics

• Develop a high level safety goal

• Establish a criteria

• Criteria should have a set of valid metrics

• Metrics will be connected to a method
Question 3

• Step 1 - Binary Test: Basic Driving Metrics (Use only portions based on the 

ODD)

• Use similar metrics to the state regulated testing (e.g., Stoplight, Stop sign)

• The procedure should be clearly specify (e.g., if vehicle ODD includes 

nighttime it should be able to perform the given task at night and day)

• Step 2 - Exercise the ODD Safety Operator

• Workout Routines: Setup by ODD and ensure that tasks the ADS will 

encounter could be responded to (consider OEDR, Failure, etc. see Testable 

Cases framework)
Question 4: Are there emerging best‐practices around 

pre‐deployment safety measurement methods? Around 

post‐deployment measurement methods? (including the 

methods and metrics described above).

• Pre-Deployment Methods Best Practice

• Include information for the public, this could be done with a tool similar 

to the VSSA

• The information needed is housed by the organization that publishes the VSSA-type 

document

• Collaboration on Testable Case Framework (NHTSA) and scenarios as a 

starting point

• On-road testing with a Safety Operator

• Post-Deployment

• ODD-Based crash and infractions database

• “How’s my Driving?” Consumer incident-based database
Question 5: Should measurement of human response to 

ADS‐equipped vehicle safety be a part of

the calculation and, if so, in what way?

• Outside of the scope of our discussion 

• Fallback Ready User (L3)

• This is not needed for non-passenger vehicles (freight, goods only 

vehicles)

• Occupant request to terminate ride (L4+ Features Engaged)

• When occupant requests control it engages into a Minimal Risk Condition 

process

• Human Outside of the Vehicle of Interest (VRU)

• This will be based on the safe definition of predictable avoidance of 

minimal risk design
Question 6: What are possible next steps?

• Other topics that might need further discussion

• What is the role of the infrastructure?

• Data/Performance

• What type of data is relevant for pre/post-deployments?

• Wants vs Needs

• Who owns this data?

• Who has access to the data (e.g., law enforcement)?

• Metrics

• Identify the metrics appropriate for a given set of criteria

• How to validate those metrics?

• Delineate guidance of what is the scope of this next set of metrics

• Producer/Manufacturer: Self-certification type assessments

• Government: Would FMVSS-type compliance verification happen in a similar?

• One-time vs Continuous: Which metrics will fall under each of these categories. The safety 

performance could potentially be continuously measured. Define what falls under each metric
Question 6: What are possible next steps?

• Other topics that might need further discussion (cont.)

• Map of other activities/standards to avoid duplication

• Map existing metrics from VSSA and research

• How to define a near-miss/crash?

• Look at pre/post-crash assessment landscape

• Look at what are the simulation state of the art and benefits/limitations

• How we get the framework developed

• Reasonable time for this development?