Summary Analysis of Fields of Interest Regarding Standards and 

Performance Metrics for On-Road Autonomous Vehicles

National Institute of Standards and Technology (NIST)

March 8th and 9th, 2022

Introduction

On-road autonomous vehicles are projected to influence key aspects of everyday life including 

transportation, goods delivery, manufacturing, public safety, and security. In addition, the on-road 

autonomous vehicle sector is a vastly interdisciplinary field with subject areas including 

perception, planning, control, safety, cybersecurity, and communications. While these fields are 

rapidly evolving, they must still overcome steep technological limitations. Therefore, NIST can 

leverage its research experience in fields including measurement science and cyber-physical 

systems to create novel solutions to advance the development of on-road autonomous vehicles.

In addition, on-road autonomous vehicles can pose a risk in the event of unexpected system 

performance. Therefore, industry and government agencies are looking to NIST to characterize 

the performance of these complex systems to mitigate risk to both the manufacturers as well as the 

consumers. NIST is suitably equipped to develop test methods, metrics, and standards to 

characterize the performance of on-road autonomous vehicles to mitigate these risks. Therefore,

NIST is in the process of assessing potential approaches to help build confidence that on-road 

autonomous vehicles will behave safely, securely, and appropriately.

To accomplish its mission, NIST has engaged stakeholders to develop a clear understanding of the 

technological challenges and opportunities in on-road autonomous vehicles. In addition, NIST has 

conducted extensive literature review while participating in standards meetings regarding 

automated vehicle technology. This is intended to foster a collaborative community to advance 

this emerging technology and build confidence in vehicle manufacturers and users towards the 

safe, secure, and appropriate operation of on-road autonomous vehicles.

The potential technological opportunities in performance metrics, test methods, and standards 

arising from NIST engagement with stakeholders are expressed in the following sections.

Primary Focus Areas

Based on stakeholder engagement, NIST has identified five primary research focus areas with the 

highest potential impact for NIST efforts in on-road autonomous vehicles. The focus areas are 

safety, communication, cybersecurity, artificial intelligence, and perception. The following 

subsections describe each of the focus areas in addition to important questions for NIST and 

stakeholders to address to advance the state of on-road autonomous vehicles.
Focus Area: Safety

Safety is arguably the most important attribute of any on-road autonomous vehicle. If the vehicle 

cannot perform safely, it will not be used and adopted. While it is impossible to guarantee that an 

on-road autonomous vehicle will perform safely in any situation it encounters, there are steps that 

can be taken to help mitigate risk. One such step is to develop standards and performance metrics

for quantifying the safety of on-road autonomous vehicles for manufacturers and end-consumers. 

Standards can play many roles, such as providing a common language to enable a shared 

understanding of on-road autonomous vehicle safety challenges. They can also specify minimum 

safety and performance requirements. A companion to safety standards is test methods and 

performance metrics, which provide repeatable and reproducible procedures and frameworks for 

measuring the safety of on-road autonomous vehicles. Standards, test methods, and performance 

metrics for on-road autonomous vehicles exist, but the field is large and there is no guarantee that 

they address all aspects of safety. The primary discussion questions regarding safety in on-road 

autonomous vehicles follow.

1. Autonomous vehicle manufacturers are concerned about safety and liability. How can we 

determine how safe a vehicle can be to be considered as safe?

• Can NIST develop technology-independent test methods to scientifically assess 

autonomous vehicles’ safety?

• Are there performance metrics or levels of safety that can be determined for autonomous

vehicles?

2. Would it be helpful if NIST worked with other agencies, such as the National Highway Traffic 

Safety Administration (NHTSA), to develop appropriate metrics to measure safety, apart from 

crashes per mile? 

• What are the proper attributes to measure safety? 

• How can these attributes be quantifiably measured?

3. How can we define an autonomous vehicle’s operating environment?

• What are the appropriate attributes (e.g., weather, lighting, vehicle density, road 

conditions) to capture about an autonomous vehicle’s operating environment? 

• What is the best way of describing or quantifying these attributes?

• Are taxonomies/ontologies necessary? 

4. The number of scenarios that an autonomous vehicle can encounter is infinite. Can a finite set 

of testing scenarios be developed that provide adequate coverage of an autonomous vehicle’s 

operational environment? 

• How can one extrapolate the performance of an autonomous vehicle after it has been tested 

in a similar, but different scenario?

• Would it be useful for NIST to develop a finite set of testing scenarios that can adequately 

cover an autonomous vehicle’s operating environment?
5. If simulation is used for safety testing, how can we quantitatively measure the accuracy of the 

simulated testing environments as compared to real world scenarios?

• There are multiple aspects, including how accurately the characteristics of the world are 

represented in the simulated environment as well as how well the performance of the 

simulated autonomous vehicle aligns with its behavior in the real world.

6. Is measuring the performance of autonomous vehicles from an engineering perspective 

sufficient, or do we need to look at the social side as well?

• Current efforts predominantly look at autonomous vehicles through an engineering lens. 

Should we also measure their performance from a social perspective (e.g. communicating 

with a pedestrian, human-machine interfaces, or associated ethical decisions in the event 

of a crash)? People use intuition to predict objects’ future movements. How do we measure 

how well an autonomous vehicle does the same?

• How could NIST help with incorporating social elements into the performance of 

autonomous vehicles?

7. There are a variety of autonomous vehicle safety standards. Can NIST help to provide a 

framework/guidance document to help navigate all the standards?

Focus Area: Communication

Communications can make on-road autonomous vehicles safer, enable necessary interactions with 

other roadway actors to make on-road autonomous vehicles practical, support and complement 

sensor-based automated driving capabilities, and create new automated driving capabilities. 5G 

networks have emerged as a major means for supporting vehicular applications, including 

communication services specifically designed for vehicle-to-vehicle and vehicle-to-infrastructure 

communications (referred to as C-V2X). Industry has been studying how 5G can support timesensitive vehicle communication applications such as teleoperation, cooperative sensing, and 

cooperative driving, which extend automated driving capabilities beyond those enabled by 

onboard sensors alone. In addition, on-road autonomous vehicles need a wide range of information 

beyond what their onboard sensors can provide. Such information includes road conditions, 

weather conditions, traffic conditions, and emergencies. Communication is necessary for 

providing such information to the on-road autonomous vehicles in addition to performing safe 

operation. At the same time, overwhelming on-road autonomous vehicles with information could 

unnecessarily increase their onboard processing burden and degrade their automated driving 

performance. Thus, challenges exist in realizing the benefits of communication in vehicles. Our 

engagement with industry experts has resulted in the following important areas and questions.

1. What on-road autonomous vehicle communication needs can 5G and the upcoming 6G meet 

and under what conditions?

• What are important test metrics and methods to assess how well and under what conditions 

5G and the upcoming 6G can support time-sensitive on-road autonomous vehicle

communication needs? 

• How can NIST quantify reliability and timeliness of communicated information (e.g., does 

the vehicle rely on the information to be safe)?
2. For on-road autonomous vehicles to function on public roads, they must communicate with a 

wide range of roadway actors such as vehicles owners, users, and repair technicians. What 

protocols and control messages will be needed for supporting such communications?

• Which communication protocols should be standardized?

• What are the security and privacy risks associated with such communications, and how can 

NIST help address them? For example, the abilities for law enforcement personnel to stop 

an on-road autonomous vehicle or for on-road autonomous vehicle technicians to direct 

on-road autonomous vehicles to specific positions, if compromised, can have severe 

consequences. 

3. Communications can enable new automated driving capabilities beyond those achievable with 

in-vehicle sensors alone. These new capabilities include vehicle teleoperation, cooperative 

sensing where input from sensors and communication devices on different vehicles and the 

roadside can be fused to achieve broader and better situational awareness than each vehicle 

can achieve with its onboard sensors alone, and cooperative driving where vehicles 

communicate with each other to coordinate their maneuvering to avoid crashes and make 

traffic smoother. 

• For each major communication-enabled on-road autonomous vehicle capability and 

system:

i. How can we characterize its performance and safety?

ii. What are the conditions (requirements) for smooth and safe operation?

iii. What measurements will be needed and how can they be obtained? 

iv. What should be standardized?

v. Does teleoperation require interoperability between vendors, so you could have, for 

example, one control station talking to many different types of cars?

4. What information should be provided to on-road autonomous vehicles, in what form, and how 

frequently?

• Should NIST determine (e.g., metrics and models) how relevant specific information will 

be to an individual vehicle or a group of vehicles?

• Information for navigation (e.g., traffic) needs to be distinguished from information for 

safe operation (e.g., response to emergency services). How can we manage processing 

between these two operations?

5. To make on-road autonomous vehicles practical, a range of communication-enabled services 

will be necessary throughout their lifecycles. These services include offboard services to 

complement vehicles’ onboard automated driving functions, monitoring on-road autonomous 

vehicle health, detecting system failures and security compromises, enabling remote real-time 

and non-real-time risk mitigation, and managing onboard software and security systems. How 

should onboard functions and offboard services communicate with each other?

• Which onboard-offboard interfaces and messages should be standardized?

• How can NIST characterize, test, and validate performance and safety of offboard services 

to on-road autonomous vehicles?

• Since wireless connectivity can fail, the onboard system must be capable of the Dynamic 

Driving Task without connectivity, even if only to execute a Minimum Risk Maneuver.
6. Trustworthiness and privacy have been among the top concerns with vehicle communications. 

Can an on-road autonomous vehicle trust the messages it receives? How can we ensure that 

vehicle communications will not be abused to breach user privacy (e.g., vehicle tracking)?

• What levels of trust will be sufficient for vehicle communications?

• How can ensure, measure, and validate trust levels of on-road autonomous vehicle 

communications?

• How can we ensure, measure, and validate security and privacy levels of on-road 

autonomous vehicles’ communication with other roadway actors?

7. Electric vehicles have additional communications requirements than internal combustion 

engine (ICE) powered vehicles. Various temporal and spatial demand patterns of electric 

vehicles complicate power grid operations. The growth of electric vehicles will require close 

integrations of power grid and the charging infrastructure.

• Can NIST help determine which information needs to be communicated from the electric 

vehicles to the Charger Scheduler and the power grid, and vice versa?

• What inter-control center communications are needed when a customer from another 

service territory requests service?

• What standards govern this area and what are missing?

• Which communication protocols best satisfy communications while acting as a probe 

vehicle in-transit, charging, confirming charging needs or other special circumstances?

8. What’s missing in the discussions and ongoing efforts on connected vehicle standards?

• Teleoperation

i. Levels of teleoperation automation?

ii. Characterization of teleoperation safety?

iii. Teleoperation operational design domains?

iv. Minimal requirements for safe teleoperations?

v. Training, testing, and certification of remote operators and their stations?

• 5G/6G 

i. Requirements for 5G/6G to support vehicle communication needs

• Trustworthiness of on-road autonomous vehicles communication

i. Trustworthiness levels and metrics of vehicles communications?

• AV-to-X communication interfaces and messages

i. Procedures, interfaces, and messages for vehicles to interact with other roadway 

actors?

ii. Procedures, interfaces, and messages for providing information to vehicles?

Focus Area: Cybersecurity and Privacy

One of the great unknowns of on-road autonomous vehicles are how malicious actors will attempt 

to leverage vulnerabilities. There are assumptions that can be made based on past activities but 

this level of automation, supported by electronics, has never been seen in the transportation sector. 

While risk approaches and cybersecurity controls exist in information technology broadly, it is 

unknown whether these will be sufficient for the defense of on-road autonomous vehicles. NIST 

has held discussions with stakeholders to help determine research areas in on-road autonomous 

vehicles.
1. What types of privacy-enhancing technologies or solutions would be most helpful for 

addressing cybersecurity risks for on-road autonomous vehicles?

• How can NIST identify the non-cybersecurity related privacy challenges/risks for on-road 

autonomous vehicles?

• How can we mitigate cybersecurity risks for on-road autonomous vehicles for which there 

appears to be no available solutions?

• Can NIST develop measurements or decomposition techniques to allow unknown 

cybersecurity risks to be articulated?

2. Considering the current cybersecurity controls and tools that are available for vehicles, what 

are the challenges in applying them to on-road autonomous vehicles?

• Should NIST advance standards or test methods that quantify the applicability of existing 

cybersecurity controls and tools in vehicles towards on-road autonomous vehicles?

• Can NIST develop techniques or platforms that facilitate the transition between existing 

cybersecurity controls towards on-road autonomous vehicles.

3. AI/machine learning is generally considered as indispensable for the functioning of on-road 

autonomous vehicles. Are there mechanisms and measures to help secure these functions?

• Can NIST develop best practices for securing AI/machine learning stacks in on-road 

autonomous vehicles?

• Should NIST develop methods to quantify cybersecurity and privacy risks associated with 

AI/machine learning for on-road autonomous vehicles?

4. What are the changes in supply chain cybersecurity management that are needed for on-road 

autonomous vehicles? 

• What areas will be added to current supply chain risk management?

• Can NIST identify areas of supply chain risk management for on-road autonomous 

vehicles?

5. What types of privacy-enhancing technologies or solutions would be most helpful for 

addressing these challenges? For example, specific privacy-enhancing technologies (PETs)

lack implementation guidance, are not commercialized or scalable, lack standards, etc.

• Is there a technical or standards area for privacy and on-road autonomous vehicles that 

would benefit most from NIST involvement?

• Should NIST help in integrating privacy requirements into autonomous vehicle standards 

or should privacy standards be standalone?

Focus Area: Artificial Intelligence and Decision Making

Artificial Intelligence (AI) is the term used to describe complex processing within a computer. 

Note that AI technology not only includes neural network-based models, but also programmatic 

decision-making and path planning. These technologies link the vehicle sensor data to the 

command and control of the vehicle, making AI the critical lynchpin in automated vehicle 

technology. Thus, advances in AI are the driving factors in on-road autonomous vehicles while 

also its key limitations. The field of AI has yet to solve its fundamental limitations in encountering 

edge cases and unforeseen conditions. While the same limitations in AI exist in other fields 
including manufacturing, medicine, and sociology, on-road autonomous vehicles are also subject 

to a variety of conditions while adhering to strict safety constraints. In addition, the nature of AI 

technology is complex, leading to many barriers to entry. This complexity leads to a lack of 

consumer understanding, and therefore trust, in AI for on-road autonomous vehicles. Standards, 

test methods, and performance metrics are viewed as a tool to address the limitations and lack of 

trust in AI by increasing transparency and repeatability. 

1. How can we determine if the test methods for an AI algorithm deployed in autonomous vehicles 

are sufficient for the entire scope of conditions an autonomous vehicle will encounter? Also, 

how do we prevent stakeholders from overfitting their AI algorithms to specific testing 

conditions?

• Can NIST play a role in determining a range of test conditions that can be extrapolated to 

real-world autonomous vehicle test conditions?

• Can NIST create or collate a collection of tests for evaluating AI algorithms in autonomous 

vehicles?

2. How can we evaluate AI algorithms that will perform differently in various domains (e.g., selfdriving trucks, robotaxis)?

• Can NIST help identify the various domains that require different AI algorithms for 

autonomous vehicles?

• Can NIST create test methods for AI algorithms dependent on the domains?

3. To test AI algorithms, they must be modified for laboratory or simulation environments. How 

can we ensure that the AI algorithms that are tested are the same as the ones deployed in realworld environments?

• Can NIST create tests methods to minimize the modification to AI for testing purposes?

• If an AI algorithm must be modified for testing, can NIST create methods to record 

differences between experimental AI algorithms and real-world environments?

4. How can we separate validation of an autonomous vehicle from what the AI algorithm 

perceives? For instance, it could be possible that an AI algorithm fails to recognize a 

pedestrian wearing a Halloween costume, but the autonomous vehicle operates safely.

• Can NIST create methods to divide test records into multiple subcomponents for reporting?

• Can NIST create test metrics for AI algorithms to become more transparent to increase 

consumer trust in autonomous vehicles?

5. To deploy AI algorithms in autonomous vehicles, a baseline must be achieved before the 

vehicle is ready to deploy. What are those baseline metrics that should be met for an AI 

algorithm to be ready to be deployed? For instance, should we look at crash rate, carbon 

emissions, bystander harm, etc.? 

• Can NIST create a list of metrics for AI systems to achieve?

• How can a list of metrics for AI systems in on-road autonomous vehicles be maintained?
Focus Area: Sensor Perception

Perception sensors are critical for the operation of any autonomous vehicle. These sensors could 

be a part of the vehicle or a part of the infrastructure in which it is being operated. Perception 

sensors can be based on various underlying technologies like 2D cameras, 3D stereoscopic 

cameras, 3D Lidars, radars, ultrasonic sensors, or a fusion of these technologies. These also may 

include inertial or global navigation satellite systems (GNSS) sensors used to augment the 

performance of the perception sensors. These sensors and systems provide the positional 

awareness for a vehicle to plan and perform vehicle maneuvers and operations. Safety of vehicle 

operations is heavily dependent on such systems and technologies. For this reason, it is important 

that these systems provide high fidelity data with low uncertainties, low latencies that are robust 

and tamper proof. The performance of these systems is also dependent on external factors like 

infrastructure markers, weather, and lighting conditions. Evaluating such systems is still fraught 

with ambiguity. This is due to the lack of standards, test methods, terminology and competition 

that creates confusion among auto manufacturers, integrators, consumers and impedes commerce. 

Addressing these issues will improve the confidence in these systems, their adoption rate and 

safety. Research and stakeholder engagement conducted by NIST highlighted specific focus areas 

and raised questions listed below.

1. How can we address the challenges for on-road autonomous vehicles’ perception and sensors 

metrics and standards?

• How can NIST address the technical gap between the sensor specifications, original 

equipment manufacturer (OEM) specifications, self-certification, and their performance?

• How can we address the interchangeability of sensors (i.e., using one set of Lidar sensors 

from one OEM in place of ones from another OEM)?

• How can we create a standard architecture for using sensors on an on-road autonomous 

vehicle?

2. How can standards development help adoption and improve perception technology in on-road 

autonomous vehicles?

• How can NIST address the lack of standards and references in benchmarking perception 

systems (particularly long-range Lidars) and HD maps?

• Can NIST provide guidance on how Lidar systems are to be tested and/or validated to work 

with on-road autonomous vehicles?

• Should NIST address standards are that may change based on technology or application 

(e.g., robotaxi vs. trucks, etc.)?

• Can NIST help advance standards for solar intensity measurement, calibrated reflectivity, 

false positive rate, and dynamic range of sensors?

• Should NIST advance common objects/targets to be perceived by perception systems?

• How could access to laboratories and facilities for ground truth measurements be 

improved?
3. Computing hardware and software forms an integral part of any on-road autonomous vehicles 

that makes use of the perception data. Do present central processing units (CPUs), graphics 

processing units (GPUs), hardware, and software latencies present a barrier for effective use 

of sensors/sensor fusion?

• Can NIST help to address the lack of metrics and convey confidence in sensor fusion?

• Is computing power due to the choice of vehicle perception technology a concern? 

• Would it be helpful for NIST to benchmark vehicle perception sensors separately from AI 

technology that is used with them?

4. Infrastructure plays a huge role in the performance of on-road autonomous vehicles 

perception and in improving the situational awareness for safe on-road autonomous vehicles 

operation. What are the needs and challenges for infrastructure?

• Should NIST identify the basic sensing required for infrastructure to support on-road 

autonomous vehicles, including camera, lidar, and map specifications?

• Can NIST help in advancing standards for communicating infrastructural changes?

• Infrastructural markers, material reflectivity, wear properties, etc. may introduce 

“confusors” for on-road autonomous vehicles perception and AI algorithms. How can 

NIST address such issues and sub-optimal conditions (e.g., snow, rain, fog, lighting 

conditions)? 

5. Are there any existing standards related to sensors (lidars/radars and other sensors) that could 

be useful for autonomous vehicle perception?

• SPIE Lidar benchmarking – ongoing activity

• ASTM E2938 for evaluating the relative-range performance of long-range 3D imaging 

systems including Lidars

• SAE Active Safety Test Target Validation-Correlation Task Force

Next Steps

Engagement with stakeholders has highlighted significant fundamental issues for which NIST is 

well-suited to develop novel measurement solutions to advance the state of on-road autonomous 

vehicles. While the problems were decomposed into five focus areas, all the areas are intertwined 

in their limitations and potential solutions. Thus, NIST will take a holistic approach towards 

addressing technological limitations by leveraging its strengths in performance metrics, test 

methods, standards, and measurement science to support the on-road autonomous vehicles 

community. In addition, stakeholders have expressed the need for communal collaboration to solve 

problems in the fields of security and safety of on-road autonomous vehicles. Hence, NIST will 

host a workshop to provide an open dialogue as a method to engage stakeholders throughout the 

development of on-road autonomous vehicles. Following the workshop, a report will be providing 

summarizing significant findings and conclusions with respect to the aforementioned discussion 

topics.