NIST Strategic and Emerging Research Initiatives (SERI) –

AV Sensor Perception Project

Prem Rachakonda

Date: 9/6/2023

Standards and Performance Metrics 

for On-Road Automated Vehicles

September 5-8, 2023 (Virtual Event)
Overview

• Problem statement

• Background and feedback from 2022 workshop.

• Focus on AV Lidars

• Technical approach

• Testbed development

• Initial results.

• Future directions
Problem statement

• There are several parameters that affect Automated vehicle (AV) 

sensor performance, however there are very few standard test 

procedures to evaluate many of these parameters.

Automated 

Vehicle 

Performance
What makes a vehicle Autonomous

Sense

Act Think

❑ For sensing the environment, 

objects, people, ego 

localization and communicate

▪ Lidar, Radar, Camera, 

Thermal, Ultrasonic

▪ Wheel encoders/

odometry, GNSS/IMU 

❑ Use of computing hardware/software to 

interpret the data communicated by the 

sensors and plan a strategy

▪ Object recognition, localization etc.

▪ Path planning ..

❑ Use of actuation

hardware to respond.

▪ Acceleration, 

braking, steering ..

Aided by 

• in-vehicle and ex-vehicle 

communication h/w and s/w, 

• communication protocols, 

• HD maps

• Infrastructure (road signs etc.)

• Computing hardware/software

• Operating systems, middleware, 

hypervisors

Note: There are several other sensors that are not listed here that ensure the safety of a vehicle regardless of the car’s type (autonomous or 

otherwise). Sensors such as O2 sensors, Torque sensors, TPMS etc. Regulations also play a huge part in enabling AVs. 
What makes a vehicle Autonomous

Sense

Act Think

❑ For sensing the environment, 

objects, people, ego 

localization and communicate

▪ Lidar, Radar, Camera, 

Thermal, Ultrasonic

▪ Odometry, Wheel 

encoders, GNSS/IMU 

❑ Use of computing hardware/software to 

interpret the data communicated by the 

sensors

▪ Object recognition, localization etc.

▪ Path planning ..

❑ Use of actuation

hardware to respond.

▪ Acceleration, 

braking, steering ..

Aided by 

• in-vehicle and ex-vehicle 

communication h/w and s/w, 

• communication protocols, 

• HD maps

• Infrastructure (road signs etc.)

• Computing hardware/software

• Operating systems, middleware, 

hypervisors

Note: There are several other sensors that are not listed here that ensure the safety of a vehicle regardless of the car’s type (autonomous or 

otherwise). Sensors such as O2 sensors, Torque sensors, TPMS etc. Regulations also play a huge part in enabling AVs. 

Measuring performance in each of these 

components is critical to improve the quality 

and safety of AVs
Industry Voices: What did stakeholders request 

from NIST?

Develop novel individual and fused sensor measurement 

science solutions for vehicles

Help define testing guidance for stakeholders to meet 

regulatory agency requirements

Develop mitigation standards for adversarial AI

Develop AV simulation-based measurement science

Advance standards with SAE, 3GPP, and Teleoperation 

Consortium

Develop measurement science for traffic infrastructure 

that can support AVs

Develop metrics to identify what aspects of AVs should be 

measured to ensure safety

Create test models and measurement science for AV 

communications

Foster a community of stakeholders to agree on common 

taxonomies and standards

Be a one-stop-shop for pointers to relevant autonomous 

vehicle standards

Measure how different parts of an AV work together

“Do you know that NIST cybersecurity framework? Just do 

that for autonomous vehicles."

Define the data that should be measured before, during, and 

after operation of automated vehicles

Provide reference materials for what infrastructure investment 

state and local governments should invest in

Collect standardized data from the DoT from accidents to 

develop representative testing environments

Provide classification and levels for AV components

Create and enforce a baseline for AV safety systems testing

Enforce sensor specs that should be used in Avs

Create regulation on periodic testing and updating

* Within NIST scope and expertise/infrastructure is available *Within NIST scope and expertise/infrastructure is lacking (NIST can support agencies) *Not within NIST scope
Industry Voices: What did stakeholders request 

from NIST?

Develop novel individual and fused sensor measurement 

science solutions for vehicles

Help define testing guidance for stakeholders to meet 

regulatory agency requirements

Develop mitigation standards for adversarial AI

Develop AV simulation-based measurement science

Advance standards with SAE, 3GPP, and Teleoperation 

Consortium

Develop measurement science for traffic infrastructure 

that can support AVs

Develop metrics to identify what aspects of AVs should be 

measured to ensure safety

Create test models and measurement science for AV 

communications

Foster a community of stakeholders to agree on common 

taxonomies and standards

Be a one-stop-shop for pointers to relevant autonomous 

vehicle standards

Measure how different parts of an AV work together

“Do you know that NIST cybersecurity framework? Just do 

that for autonomous vehicles."

Define the data that should be measured before, during, and 

after operation of automated vehicles

Provide reference materials for what infrastructure investment 

state and local governments should invest in

Collect standardized data from the DoT from accidents to 

develop representative testing environments

Provide classification and levels for AV components

Create and enforce a baseline for AV safety systems testing

Enforce sensor specs that should be used in Avs

Create regulation on periodic testing and updating

* Within NIST scope and expertise/infrastructure is available *Within NIST scope and expertise/infrastructure is lacking (NIST can support agencies) *Not within NIST scope

Develop measurement science to 

make automated vehicles safer.
Industry Voices: What did stakeholders request 

from NIST?

Develop novel individual and fused sensor measurement 

science solutions for vehicles

Help define testing guidance for stakeholders to meet 

regulatory agency requirements

Develop mitigation standards for adversarial AI

Develop AV simulation-based measurement science

Advance standards with SAE, 3GPP, and Teleoperation 

Consortium

Develop measurement science for traffic infrastructure 

that can support AVs

Develop metrics to identify what aspects of AVs should be 

measured to ensure safety

Create test models and measurement science for AV 

communications

Foster a community of stakeholders to agree on common 

taxonomies and standards

Be a one-stop-shop for pointers to relevant autonomous 

vehicle standards

Measure how different parts of an AV work together

“Do you know that NIST cybersecurity framework? Just do 

that for autonomous vehicles."

Define the data that should be measured before, during, and 

after operation of automated vehicles

Provide reference materials for what infrastructure investment 

state and local governments should invest in

Collect standardized data from the DoT from accidents to 

develop representative testing environments

Provide classification and levels for AV components

Create and enforce a baseline for AV safety systems testing

Enforce sensor specs that should be used in Avs

Create regulation on periodic testing and updating

* Within NIST scope and expertise/infrastructure is available *Within NIST scope and expertise/infrastructure is lacking (NIST can support agencies) *Not within NIST scope

NIST Mission: To promote U.S. innovation and industrial 

competitiveness by advancing measurement science, 

standards, and technology in ways that enhance economic 

security and improve our quality of life.
Industry Voices: What did stakeholders request 

from NIST?

Develop novel individual and fused sensor measurement 

science solutions for vehicles

Help define testing guidance for stakeholders to meet 

regulatory agency requirements

Develop mitigation standards for adversarial AI

Develop AV simulation-based measurement science

Advance standards with SAE, 3GPP, and Teleoperation 

Consortium

Develop measurement science for traffic infrastructure 

that can support AVs

Develop metrics to identify what aspects of AVs should be 

measured to ensure safety

Create test models and measurement science for AV 

communications

Foster a community of stakeholders to agree on common 

taxonomies and standards

Be a one-stop-shop for pointers to relevant autonomous 

vehicle standards

Measure how different parts of an AV work together

“Do you know that NIST cybersecurity framework? Just do 

that for autonomous vehicles."

Define the data that should be measured before, during, and 

after operation of automated vehicles

Provide reference materials for what infrastructure investment 

state and local governments should invest in

Collect standardized data from the DoT from accidents to 

develop representative testing environments

Provide classification and levels for AV components

Create and enforce a baseline for AV safety systems testing

Enforce sensor specs that should be used in Avs

Create regulation on periodic testing and updating

* Within NIST scope and expertise/infrastructure is available *Within NIST scope and expertise/infrastructure is lacking (NIST can support agencies) *Not within NIST scope
Industry Voices: What did stakeholders request 

from NIST?

Develop novel individual and fused sensor measurement 

science solutions for vehicles

Help define testing guidance for stakeholders to meet 

regulatory agency requirements

Develop mitigation standards for adversarial AI

Develop AV simulation-based measurement science

Advance standards with SAE, 3GPP, and Teleoperation 

Consortium

Develop measurement science for traffic infrastructure 

that can support AVs

Develop metrics to identify what aspects of AVs should be 

measured to ensure safety

Create test models and measurement science for AV 

communications

Foster a community of stakeholders to agree on common 

taxonomies and standards

Be a one-stop-shop for pointers to relevant autonomous 

vehicle standards

Measure how different parts of an AV work together

“Do you know that NIST cybersecurity framework? Just do 

that for autonomous vehicles."

Define the data that should be measured before, during, and 

after operation of automated vehicles

Provide reference materials for what infrastructure investment 

state and local governments should invest in

Collect standardized data from the DoT from accidents to 

develop representative testing environments

Provide classification and levels for AV components

Create and enforce a baseline for AV safety systems testing

Enforce sensor specs that should be used in Avs

Create regulation on periodic testing and updating

* Within NIST scope and expertise/infrastructure is available *Within NIST scope and expertise/infrastructure is lacking (NIST can support agencies) *Not within NIST scope

Potential Impact

• In the first half of 2022*:

• Total SAE Level 2 car sales in the US’ increased 

to 46.5% and ADAS penetration crosses 70%

NIST expertise

• National Metrology Institute of USA

• Multi-disciplinary experts

• Led the development of several standards related 

to sensors for manufacturing automation

Source: https://www.counterpointresearch.com/adas-penetration-crosses-70-us-h1-2022-level-2-share-46-5/

Industry

Need

NIST 

Expertise

Potential 

Impact
Industry Voices: What did stakeholders request 

from NIST?

Develop novel individual and fused sensor measurement 

science solutions for vehicles

Help define testing guidance for stakeholders to meet 

regulatory agency requirements

Develop mitigation standards for adversarial AI

Develop AV simulation-based measurement science

Advance standards with SAE, 3GPP, and Teleoperation 

Consortium

Develop measurement science for traffic infrastructure 

that can support AVs

Develop metrics to identify what aspects of AVs should be 

measured to ensure safety

Create test models and measurement science for AV 

communications

Foster a community of stakeholders to agree on common 

taxonomies and standards

Be a one-stop-shop for pointers to relevant autonomous 

vehicle standards

Measure how different parts of an AV work together

“Do you know that NIST cybersecurity framework? Just do 

that for autonomous vehicles."

Define the data that should be measured before, during, and 

after operation of automated vehicles

Provide reference materials for what infrastructure investment 

state and local governments should invest in

Collect standardized data from the DoT from accidents to 

develop representative testing environments

Provide classification and levels for AV components

Create and enforce a baseline for AV safety systems testing

Enforce sensor specs that should be used in Avs

Create regulation on periodic testing and updating

* Within NIST scope and expertise/infrastructure is available *Within NIST scope and expertise/infrastructure is lacking (NIST can support agencies) *Not within NIST scope

• NIST developed measurement science, test methods 

and standards can be used:

• by the manufacturer to spec’ their sensors

• by the end users/integrators to evaluate sensors

• to perform an apples-to-apples comparison

Source: https://www.counterpointresearch.com/adas-penetration-crosses-70-us-h1-2022-level-2-share-46-5/

Industry

Need

NIST 

Expertise

Potential 

Impact
Overview

• Problem statement

• Background and feedback from 2022 workshop.

• Focus on AV Lidars

• Technical approach

• Testbed development

• Initial results.

• Next steps 
Focus on Lidars (NIST expertise)

• There are several sensor 

technologies that are used on AVs –

our initial focus is on Lidars.

• Experience in evaluating Terrestrial 

Lidars for over a decade.

• Led the development of multiple 

standards related to Terrestrial Lidars 

and other 3D optical sensors 

through ASTM E57.

• NIST is the National Metrology 

Institute of USA with state-of-the art 

equipment and staff whose 

expertise can be leveraged for this 

effort. 
AV Lidar technology

❑ Based on laser wavelength: 

o Near infrared: 850 nm, 905 nm, 940 nm; Shortwave infrared: 1350 nm, 

1550 nm

❑ Based on measurement principle: 

o Time-of-flight (TOF), Frequency Modulated Continuous Wave (FMCW)

❑ Based on laser diode type (for TOF sensors): 

o Edge-emitting laser (EEL) and Vertical Cavity Surface Emitting Laser 

(VCSEL)

❑ Based on beam steering: 

o Mechanical, MEMS, Solid-state and hybrid

❑ Based on receiver: 

o PIN photodiode, Avalanche photodiode (APD), Single-photon avalanche 

diode (SPAD), Silicon Photomultiplier (SiPM)
AV Lidar technology

• Choice of the technology depends on:

• Considerations for eye safety, power consumption, features, component supply chains, 

manufacturability, packaging complexity, cost (For ex: Silicon for < 1000 nm vs InGaAs for > 

1000 nm) 

• Atmospheric absorption

• Lower photons = higher signal-to noise ratio
• Direct normal spectral irradiance is within a 5.8° field of view centered on the sun. 

• ** Source: ASTM-G173 › Standard Tables for Reference Solar Spectral Irradiances: Direct Normal and Hemispherical on 37° Tilted Surface

• Choice of the technology depends on:

• Considerations for eye safety, power consumption, features, component supply chains, 

manufacturability, packaging complexity, cost (For ex: Silicon for < 1000 nm vs InGaAs for > 

1000 nm) 

• Atmospheric absorption

• Lower photons = higher signal-to noise ratio 

• enables higher power, long range, eye-safe lasers

AV Lidar technology

Solar 

spectral 

Irradiance**
Solar spectral irradiance and Human skin 

reflectance

• *Source: NIST data (Cooksey et al): https://files.cie.co.at/x046_2019/x046-PO065.pdf & Ian Blasch (Jabil.com)

• **Source: ASTM-G173 › Standard Tables for Reference Solar Spectral Irradiances: Direct Normal and Hemispherical on 37° Tilted Surface

Solar 

spectral 

Irradiance**

Human Skin 

Reflectance*
Solar 

spectral 

Irradiance**

Human Skin 

Reflectance*

Solar spectral irradiance and Human skin 

reflectance

• *Source: NIST data (Cooksey et al): https://files.cie.co.at/x046_2019/x046-PO065.pdf & Ian Blasch (Jabil.com)

• **Source: ASTM-G173 › Standard Tables for Reference Solar Spectral Irradiances: Direct Normal and Hemispherical on 37° Tilted Surface
Lidar specifications

Lidar Lidar A1 Lidar A2 Lidar A3 Lidar A4 Lidar B1

Beam-steering Spinning Spinning Spinning Galvo. Mirror Spinning

Laser emitter EEL EEL EEL EEL VCSEL

Photo detector APD APD APD APD SPAD

Vertical lines 16 32 64 32-80* 32

Wavelength 903 nm 903 905 nm 905 nm 850 nm

FOV-Horizontal 360 360 360 120 360

FOV-Vertical 30 40 26.8 16 45

Rotation rate 5 Hz - 20 Hz* 5 Hz - 20 Hz* 5 Hz – 15 Hz* 10-Hz – 25 Hz* 10 Hz - 20 Hz*

Points/sec 300K - 600K 600K - 1200K N/A N/A 655 K

Min. Range N/A N/A N/A 0.1 m* 0.8 m*

Max. Range 100 m 200 m 120 m* 170 m* 120 m*

Range Resolution N/A N/A N/A 4 mm 12 mm

Range Accuracy ±30 mm ±30 mm < 20 mm ±50 mm zero - slight bias

Range Repeatability N/A N/A N/A N/A 15 mm – 100 mm*

False positive rate N/A N/A N/A N/A 1/10000

Beam Size 9.5 mm/12.7 mm* 9.5 mm/12.7 mm* N/A N/A 10 mm

Beam Divergence 0.07/0.18 deg* 0.09/0.18 deg* N/A N/A 0.13 deg

*Depends on target, or sensor setting, or sensor type
Lidar specifications

Lidar Lidar A1 Lidar A2 Lidar A3 Lidar A4 Lidar B1

Beam-steering Spinning Spinning Spinning Galvo. Mirror Spinning

Laser emitter EEL EEL EEL EEL VCSEL

Photo detector APD APD APD APD SPAD

Vertical lines 16 32 64 32-80* 32

Wavelength 903 nm 903 905 nm 905 nm 850 nm

FOV-Horizontal 360 360 360 120 360

FOV-Vertical 30 40 26.8 16 45

Rotation rate 5 Hz - 20 Hz* 5 Hz - 20 Hz* 5 Hz – 15 Hz* 10-Hz – 25 Hz* 10 Hz - 20 Hz*

Points/sec 300K - 600K 600K - 1200K N/A N/A 655 K

Min. Range N/A N/A N/A 0.1 m* 0.8 m*

Max. Range 100 m 200 m 120 m* 170 m* 120 m*

Range Resolution N/A N/A N/A 4 mm 12 mm

Range Accuracy ±30 mm ±30 mm < 20 mm ±50 mm zero - slight bias

Range Repeatability N/A N/A N/A N/A 15 mm – 100 mm*

False positive rate N/A N/A N/A N/A 1/10000

Beam Size 9.5 mm/12.7 mm* 9.5 mm/12.7 mm* N/A N/A 10 mm

Beam Divergence 0.07/0.18 deg* 0.09/0.18 deg* N/A N/A 0.13 deg

*Depends on target, or sensor setting, or sensor type

Many common specifications are missing from 

the specification sheets and limited information 

on how these parameters are being measured.
Lidar specifications

Lidar Lidar A1 Lidar A2 Lidar A3 Lidar A4 Lidar B1

Beam-steering Spinning Spinning Spinning Galvo. Mirror Spinning

Laser emitter EEL EEL EEL EEL VCSEL

Photo detector APD APD APD APD SPAD

Vertical lines 16 32 64 32-80* 32

Wavelength 903 nm 903 905 nm 905 nm 850 nm

FOV-Horizontal 360 360 360 120 360

FOV-Vertical 30 40 26.8 16 45

Rotation rate 5 Hz - 20 Hz* 5 Hz - 20 Hz* 5 Hz – 15 Hz* 10-Hz – 25 Hz* 10 Hz - 20 Hz*

Points/sec 300K - 600K 600K - 1200K N/A N/A 655 K

Min. Range N/A N/A N/A 0.1 m* 0.8 m*

Max. Range 100 m 200 m 120 m* 170 m* 120 m*

Range Resolution N/A N/A N/A 4 mm 12 mm

Range Accuracy ±30 mm ±30 mm < 20 mm ±50 mm zero - slight bias

Range Repeatability N/A N/A N/A N/A 15 mm – 100 mm*

False positive rate N/A N/A N/A N/A 1/10000

Beam Size 9.5 mm/12.7 mm* 9.5 mm/12.7 mm* N/A N/A 10 mm

Beam Divergence 0.07/0.18 deg* 0.09/0.18 deg* N/A N/A 0.13 deg

*Depends on target, or sensor setting, or sensor type

Motivation for this work: Develop a testbed and 

test methods to evaluate AV Perception sensors
Overview

• Problem statement

• Background and feedback from 2022 workshop.

• Focus on AV Lidars

• Technical approach

• Testbed development

• Initial results.

• Next steps 
Technical approach for testbed development

Disclaimer: Commercial equipment and materials may be identified to specify certain 

procedures. In no case does such identification imply recommendation or endorsement 

by the NIST, nor does it imply that the materials or equipment identified are necessarily 

the best available for the purpose.

• Develop a testbed and develop methods to evaluate sensors 

using SI traceable artifacts and instrumentation*.

• Sensor integration with ROS2

• Preliminary tests to understand data quality.

• Internal/external length error tests using calibrated artifacts.

• Lidar-Lidar calibration, registration, evaluation.

• SLAM testing using ICP algorithms

*ROS2: Robot Operating System 2; SLAM: Simultaneous localization and mapping; ICP: Iterative closest point
Testbed and apparatus: Sensors

Ubuntu laptop 

with ROS2

Velodyne

HDL64e

Ouster

OS1-32

Velodyne

VLP-16

Velodyne

VLP-32C

Network switch

Disclaimer: Commercial equipment and materials may be 

identified to specify certain procedures. In no case does such 

identification imply recommendation or endorsement by the NIST, 

nor does it imply that the materials or equipment identified are 

necessarily the best available for the purpose.

Spinning 

lidars 

(850/903 

nm, 

16/32/64 

lines)

Galvo. 

Mirror 

based 

(903 nm)
Testbed and apparatus: Planar targets

Target size: 1.2 m x 2.4 m and dimensional flatness: < 0.2 mm
Testbed and apparatus: Spherical Targets

• Sphere targets:

• Vapor blasted aluminum, with a 

matte finish for near Lambertian 

reflectance.

• ~15” or 381 mm nominal diameter

• Diameter uncertainty of ~0.1 mm

• Worst case roundness of 1.13 mm
Testbed and apparatus: Reference systems

API T3 Laser tracker OptiTrak Motion capture cameras

Source: https://www.youtube.com/watch?v=FEjXfUrkueg

Disclaimer: Commercial equipment and materials may be identified to specify certain procedures. In 

no case does such identification imply recommendation or endorsement by the NIST, nor does it 

imply that the materials or equipment identified are necessarily the best available for the purpose.
Overview

• Problem statement

• Background and feedback from 2022 workshop.

• Focus on AV Lidars

• Technical approach

• Testbed development (Retroreflection facility)

• Initial results.

• Next steps 
Retroreflection Facility

o NIST established 

retroreflection - calibration 

and characterization facility 

in the early 2000s. 

o It was limited to visible 

wavelengths 

supporting ASTM 

activities in E12 – Color 

& Appearance and SAE 

activities.

o In support of automated vehicles, 

the retroreflection facility has been 

upgraded to work with infrared 

wavelengths (800 nm – 1650 nm).

o Infrared wavelengths support 

infrastructure assisting LIDAR and 

infrared imaging systems.

o Calibrated samples will allow in-situ 

measurements and calibrations of 

mounted devices 

Ratio = Radiance/Irradiance
Extending an Automated Vehicles Vision

o The retroreflection facility can be 

extended to characterizing and calibrating 

device through imaging capabilities. 

Demonstration in the visible range are 

shown in the panel to the right.

o Some companies sell materials that are 

retroreflective only in the infrared. 

o Visible to the infrared imaging systems 

while not adding to the confusing 

human signaling environment.

MIT CSAIL

Image credit: Cyalume Technologies 

o Coupling infrared QR codes shown to the right can digitally communicate 

information quickly and with high confidence.
Overview

• Problem statement

• Background and feedback from 2022 workshop.

• Focus on AV Lidars

• Technical approach

• Testbed development

• Initial results.

• Next steps 
Frame 1 Frame 2

Data inconsistencies due to software version/platform

• Change in point density along the azimuth direction – an artifact of the 3rd party ROS2 driver

• (X and Y axes were also swapped – code modification was needed to fix the issue)

Link: velodyne-decoder · PyPI

Data from LidarA1
Data on the target using different software

LidarA1 S/W
• Some sensor/hardware combination can degrade data quality. 

• In this instance, adding a network switch between the 2 improves the 

quality of signal/point cloud.

Lidar ➔ Network Switch ➔ Laptop➔ROS2

Degradation in data quality due to communication issues

Lidar ➔ Laptop➔ROS2
Noise and intensity profiles

LidarA1

LidarA1

Lidar B1

Lidar B1

LidarA1 LidarB1
Retroreflective

road signs

Effect of retroreflective surfaces on lidar data

LidarA1 LidarB1

Region where data is expected 

on the target
Target Angle vs # of Points

LidarA1 LidarB1

Lidar Target
Target Angle vs Plane fit RMSE

LidarA1 LidarB1

Lidar Target
Test procedures: Length error tests

Error in mm (Reference length between spheres = 2.3 m)

Exterior length error (mm) Interior length error

Mean (mm) Std. (mm) Mean (mm) Std. (mm)

LidarA1 10.23 1.16 52.1 0.42

LidarA2 5.27 0.58 13.5 0.38

LidarA3 0.42 0.89 26.9 0.87

LidarB1 21.9 1.78 -96.2 1.27

Exterior length error test

Interior length error test
• LidarA* exhibited +ve length 

error. 

• It could be attributed to the –ve 

offset of the laser origin with respect 

to lidar’s rotation axis.

• Similarly, LidarB exhibited -ve 

length error

• It could be attributed to the +ve 

offset of the laser origin with respect 

to lidar’s rotation axis

Possible explanation of the length error in interior tests
Sphere segmentation:

Scans from Lidar A1 with spheres at 8m away from the lidar (1182 frames combined)

X, m Y, m Y, m X, m

Z, mZ, m
Sphere segmentation: NIST/ASTM E57 Algorithm

X, m X, m Y, m Y, m

Z, m

Z, m

Points on the sphere

Excluded points
• Minimum distance: Minimum 

distance at which Lidar scans 

the target surface.

• Distances are approximate 

and as reported by lidar.

• Targets were positioned 

manually away from the 

lidars until the points appear. 

Minimum Distance (m)

Sensor/

Target ***

White Kynar 

Board

Black Kynar 

Board

15” Aluminum 

Sphere

LidarA1 0.92 0.94 0.99

LidarA2 0.91 0.98 0.80

LidarA3 0.80 0.91 0.87

LidarB1 0.50 0.40** 0.45

* To the outer surface, not lidar-center distance

** Excluding anomaly in black (Missing points on the target surface at certain locations)

*** Kynar Black has a reflectance of 5.9%, Kynar white is at 66.7% and Aluminum sphere is approximately at 55%

Minimum distance test
Multi-lidar calibration

Flat 

Target 1

Sphere 2

Sphere 1

Flat 

Target 2 

Lidars 

Sphere2 Sphere 1

Flat Target 1 

Flat Target2
Multi-lidar calibration

Length error between the spheres in the merged data

Sensors Mean Error (mm) Std (mm)

LidarA1-LidarB1 32.48 2.94

LidarA1-LidarA2 32.96 3.23

LidarA1-LidarA3 46.84 6.65

LidarA1 only* 23.59 1.74

ROS2: Point-to-point ICP, with parameter tuning

MATLAB: Plane-to-plane ICP. 
SLAM using Lidar (KISS-ICP)

Lidar
SLAM using Lidar (KISS-ICP)

• ~52 m/170ft offset between the starting and ending location

• Highlights the need for sensor fusion and improved SLAM algorithms
Overview

• Problem statement

• Background and feedback from 2022 workshop.

• Focus on AV Lidars

• Technical approach

• Testbed development

• Initial results.

• Next steps 
Next steps: AV Development mule (9/15/2023)

Long range 

Radar

Spinning Lidar (850 nm) RGB Camera GNSS/IMU

Industrial grade 

computer system

+ AV software based 

on ROS2

Drive-ByWire System

Shore power 

system

Ford Fusion 2020 Hybrid (Image Source: DataspeedInc.com)

Disclaimer: Commercial equipment and materials may be identified to specify certain procedures. In no case does such 

identification imply recommendation or endorsement by the NIST, nor does it imply that the materials or equipment identified are

necessarily the best available for the purpose.
Future Directions

• Static testing: 

• Expand sensor suite (solid state sensors, cameras) and perform sensor fusion

• Evaluate range performance of sensors to understand the effect of sensor 

degradation, distance, color, reflectance

• Dynamic testing*: 

• Evaluate the localization performance of the development mule in cases of

o Sensor degradation

o GPS denial

o GPS without RTK adjustments

o Alternative SLAM algorithms

o Understanding the sensor and software integration challenges 

(Kinematic Autonomy Software vs Autoware vs CARMA) 

o Initiate and/or participate in benchmarking and standards 

development

* Contingent upon vehicle delivery timeline and funding
❑ Abhilash Mane (University of Maryland; NIST/EL Associate)

❑ Cameron Miller (NIST/PML) – Retroreflection Facility

❑ Heather Patrick, David Allen & Susana Deustua (NIST/PML)

❑ Kamel Saidi, Marek Franaszek, Omar Aboul-Enein (NIST/EL)

❑ Adam Pintar (NIST/ITL)

Acknowledgements
Questions?

❑ Prem.Rachakonda@nist.gov