NIST Workshop –

Perception Keynote

Rajeev Thakur, P.E
Agenda

• Overview

• Vehicle Safety Requirements

• Operational Domain & Perception requirements 

• Sensor characterization

• Evolving sensor technology

• ADAS to AV perception

• Evolution timeline threat

• Summary

9/22/2023 NIST – Perception Keynote 2
Overview

Complex System

Autonomous vehicle HW , 

SW and UX are not difficult 

under ideal test conditions.

Uncertainty with human 

drivers on the road, 

weather , sensor 

performance and latencies 

add an order of magnitude 

to complexity

Public expectations

Evolution of AV is 

happening on the public 

stage

Public expectations for AV 

performance is set very 

high by not discussing the 

probability of failure 

transparently (ASIL D 

means nothing to public – 

zero failure shown as goal) 

Need for standards

NIST , SAE , ISO , IIHS and 

other such bodies are 

needed to establish a 

common standard for 

communicating testing 

and performance (OEMs 

are too busy developing 

and competing !)

9/22/2023 NIST – Perception Keynote 3
Vehicle Safety Requirements

Safety Goal

ASIL D : 

• 1 dangerous failure in 

more 108 hrs operation 

• ~ 0.02 deaths per 100M 

miles @60 mph

• 2021 USA : 1.5 deaths 

per 100M miles

• 1.5/0.02 = ~ 75X

Consumer View

• The adoption success of 

Tesla autopilot despite 

736 crashes since 2019 

with 17 fatalities –

indicates that public is OK 

with a vehicle that is ~ 3 x 

to 10X better per Tesla 

calculations 1(2022 Q4 : 

5M miles per accident vs 

0.5M US average ; Vehicle 

Safety Score calc ~ 3X 

better ; No deaths/100m 

unfortunately)

Reset for AV Adoption

• True ASIL rating for 

deployed AV are likely 

lower and OK (No ASIL D 

certifications for vehicle 

are public)

• Aiming for 75x 

improvement makes cost 

of perception higher and 

delays adoption

• Need open discussion to 

set realistic goals for next 

5 & 10 yr timeline - NIST

9/22/2023 NIST – Perception Keynote 4

1- https://www.tesla.com/VehicleSafetyReport
Operational Domain & Perception

Roads & Speed

• Paved highway

• Example: See brick 

and large pothole > 

150m, FoV 45 x 45

• Paved urban

• Ex: kids / soccer ball , 

50m , FoV 360 x 45

• Geofenced...

• Offroad..

Ambient Conditions

• Day / Night

• Sun / Rain / Snow / Fog

• Safe human drivers will 

reduce speed or 

pullover if perception is 

bad 

• Sensor perception is 

acceptable when well 

characterized

Rating Evaluation

• L4 rating over defined 

operational domain 

needs to be certified by 

a trusted agency 

• L4 test will need random 

edge cases along with 

testing on public road 

with safety driver

• A digital model of the 

vehicle could also be 

added for simulation 

testing

9/22/2023 NIST – Perception Keynote 5

NIST to consider:

1. Sensor requirements and characterization standards

2. Testing standards for vehicle L4 rating based on ODD
Sensor Characterization 

Terminology

• Detection (X pts over Y 

frames)

• Accuracy (over X frames)

• Range 

• Resolution

• Region of Interest

• Frame rate

Ambient Conditions

• Sunlight (0 to ~ 100k lux)

• Rain (2.5 to 50 mm/hr)

• Snow (< 1 to > 5 mm/hr)

• Fog ( 0.25 to > 1 mile 

visibility)

Target definition

• Size

• Reflectivity

• Color

• Material

• Surrounding targets 

(Example: 5% Black with 

retro surrounding)

9/22/2023 NIST – Perception Keynote 6

NIST to consider :

• Define target, ambient condition, frame rate, detection – then 

check range, resolution , accuracy over limited frames

• Characterize performance with all possible critical targets and 

ambient conditions in operational domain
Sensor technology

Mature

• Lidar

• Radar

• Visible Camera

• Ultrasonic

• IMU

• GPS

Knocking on Door

• Thermal Camera

• Imaging Radar

• Ground Penetrating 

Radar

• V2I / V2V 

• 2D GPS barcodes in 

urban canyons

Unmet / Challenges

• Potholes

• Road Surface Condition

• Sensor blockage and 

cleaning

• Calibration with drift 

and service - after a 

fender bender

• Diagnostics for vehicle 

mechanical parts –

motors / wheels / 

structure / etc.

9/22/2023 NIST – Perception Keynote 7

NIST to consider :

1- Role of V2X in AV (Perception at intersections for cross traffic)

2- Potholes , cinder blocks , etc on roads – perception needs
ADAS to AV – Perception

ADAS (L1 – L2)

• Higher Volumes in near 

future

• Driver is backup to 

perception malfunction

• Well defined operational 

domain and edge cases

• Perception blind spots 

around vehicle 

acceptable

AV (L3 – L5)

• Lower volumes in near 

future

• No backup to take 

steering wheel control

• Operational domain and 

perception needs cannot 

be completely defined

• 360 perception around 

vehicle needed 

Comments

• AV adoption needs ADAS 

volume to reduce costs

• Perception and compute 

costs of AV are still very 

high – indicating a long 

development ramp

• AV stack is orders of 

magnitude more 

complex than ADAS

• Perception technology 

will mature from ADAS 

for AV adoption

9/22/2023 NIST – Perception Keynote 8

NIST to consider :

• Data sharing from AV accidents – what perception / 

actuation /HW /SW / object on road caused the failure : Like a 

ML model – add to testing requirements
Evolution Timeline Threat

9/22/2023 NIST – Perception Keynote 9

AV defined by requirements

• Operational domain

• HW architecture

• SW stack

Test & Validation Deploy & Maintain

Redesign

Improve in next gen

New improved sensor 

/ compute HW

Development 

Start

SOP

Years

New improved sensor 

/ compute HW

New improved sensor 

/ compute HW

Gen2 Gen3 Gen4

New improved sensor 

/ compute HW

Gen1

• Unless perception HW & SW are developed in a modular interface 

fashion , many new sensor generations will miss adoption

(for ADAS ..)
Summary 

• AV Safety goal : ASIL D (~ 75X better) Vs Tesla (~ 3X to 10X ) Vs Public goal (0 failure)

• Need certification tests for AV with random scenarios based on ODD + simulations

• Characterize sensors over ODD with edge targets and ambient conditions

• 10 yr AV development cycle could strand Newer sensors adoption

• ADAS sensors help AV adoption by bringing costs down – but have different specs

• NIST and other policy bodies have a significant role to play for bringing the safety and 

convenience benefits of autonomous vehicles to society !

9/22/2023 NIST – Perception Keynote 10
Thank you!

Rajeev Thakur

Golfer_Thakur@yahoo.com