Progress on Systems Interaction in Automated

Vehicles

Zeid Kootbally (zeid.kootbally@nist.gov)
Contributions

■ Vinh Nguyen

■ Mac McCall – Kevin Kefauver

■ Dillon Alderson – Alex Hatchett – Steven Huggins – Zachary Saunders

■ Craig Schlenoff – Omar Aboul-Enein – Harold Booth – Edward Griffor – Chumei Liu – Prem

Rachakonda – Thomas Roth – Yishen Sun – Apostol Vassilev – David Wollman

Systems Interaction 1 ʢ 37
Outline

1 Problem Statement

1 Current Types of Testing

2 Proposed Testing

2 Systems Interaction

1 Why Should We Study Systems Interaction?

2 How to Think About Interactions?

3 How is the Study of Systems Interaction Performed?

3 Physical Testbed

1 Roadmap Development

2 Development Mule

4 Summary

Systems Interaction 2 ʢ 37
Systems

Title: Progress on Systems Interaction in Automated Vehicles.

■ In 2022, NIST held the Standards and Performance Metrics for On-Road Autonomous

Vehicles Workshop.

■ The workshop identified several key areas (systems) in which NIST could have an

impact.

■ Artificial Intelligence (AI).

■ Communication.

■ Cybersecurity.

■ Perception.

Systems Interaction 3 ʢ 37
Problem Statement

Testbeds for Automated Vehicles (AVs) can be categorized into two main types.

1. Individual system testing.

■ Testing specific components or

systems of the vehicle in isolation.

■ e.g., evaluating sensor suite.

2. Full vehicle testing.

Evaluation of AVs is performed in the

environment.

Comms AI Cybersecurity Perception

credit: Irina Cheremisinova

Systems Interaction 4 ʢ 37
Problem Statement

■ In this scenario, it would be useful to study how the AV systems interact with each

other to be able to answer crucial questions such as:

■ What happens if the communications between the front and the back vehicles are

delayed?

■ What happens if a cyber event disrupts perception?

■ How would the interactions between some of these AV systems impact the overall

performance of both vehicles?

Systems Interaction 5 ʢ 37
Problem Statement  ̈ Proposed Testing

NIST aims to investigate a systems interaction testbed and evaluation framework that bridges

the gap between individual system testing and full vehicle testing.

Systems Interaction 6 ʢ 37
Problem Statement  ̈ Proposed Testing

1. Individual system testing.

■ Testing specific components or

systems of the vehicle in isolation.

■ Example: Evaluating sensor suite and

communication layers.

2. Systems interaction testbed and

evaluation framework.

■ The interactions between AV systems

are tested.

3. Full vehicle testing.

■ Evaluation of AVs is performed in the

environment.

Comms AI Cybersecurity Perception

credit: Irina Cheremisinova

Systems Interaction 7 ʢ 37
Systems Interaction  ̈ Why Should We Study Systems Interaction?

Standards and Performance Metrics for On-Road Autonomous Vehicles – March 8–9 2022.

Figure: Stakeholders have identified the importance of measuring interactions between systems

that affect overall vehicle performance.

Systems Interaction 8 ʢ 37
Systems Interaction  ̈ Why Should We Study Systems Interaction?

̗ Impact on Quality Attributes

■ Enhanced Safety – Identifying potential conflicts or weaknesses in the interactions can

help implement robust safety mechanisms.

■ Improved Performance – e.g., studying the interactions between AI and perception

systems can enhance object recognition.

■ Reliability – Understanding the interactions between these systems can help identify

potential failure points and vulnerabilities.

■ Scalability – Studying these interactions helps ensure they can operate safely and

reliably in a variety of environments and conditions.

Systems Interaction 9 ʢ 37
Systems Interaction  ̈ How to Think About Interactions?

Systems Interaction 10 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?

■ Testbed – Useful to stakeholders to transition individual system-level testing to

overall vehicle performance.

1. Design – Develop an AV systems interaction architecture which provides a

structured approach to managing and facilitating communication and

collaboration among different AV systems.

2. Implementation – Use the architecture to implement on-road scenarios.

3. Evaluation – Develop and use the Automated Driving Systems Interaction

Evaluation (ADSIE) framework to evaluate the performance of AVs in

on-road scenarios.

Systems Interaction 11 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Design

̗ 1. Design

Systems Interaction 12 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Design

̗ Sense-Think-Act Paradigm

■ Sense – This step involves perceiving and collecting data from the environment using

various sensors.

■ Think – The collected data is processed and interpreted to make informed decisions

based on predefined rules, machine learning models, or other algorithms.

■ Act – Actions (steering, accelerating, or braking) are taken to interact with the

environment.

environment

credit: Irina Cheremisinova

Sense Think Act

Systems Interaction 13 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Design

Systems Interaction 14 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Implementation

̗ 2. Implementation

Systems Interaction 15 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Implementation

̗ Simulation and Physical Testbeds

The testbed is expected to be implemented in simulation and on a physical vehicle.

1. Simulation-based testbed – Translate the structural concepts of the AV systems

interaction architecture into simulation.

■ Requirements (29 were identified):

■ Real time to support hardware-in-the-loop.

■ Ability to introduce perturbations/injections at different levels in the architecture.

■ Create scenarios with minimum time and effort.

■ Physics based with realistic environments (infrastructures, vehicles, and pedestrians).

■ Has some ADS features.

■ ...

2. Physical testbed – Will be put into action following the simulation testbed (next step).

■ Adapt simulation models.

■ Calibration and sensor fusion.

■ Testing procedures.

■ Test execution.

■ Data analysis and comparison.

■ ...

Systems Interaction 16 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Implementation

Tools∗for the Simulation Testbed

Autoware - https://autoware.org

■ Open-source software stack used for research and development of autonomous vehicle

systems. Designed to run in simulation and on actual vehicles.

CARLA (Car Learning to Act) – https://carla.org

■ Physics-based and open-source simulator for autonomous driving research.

■ Not intended for running on physical vehicles but is used for simulating and testing

autonomous driving algorithms and vehicles in a virtual environment.

ns-3 – https://www.nsnam.org/

■ Open-source discrete-event network simulator used for research and development in

computer networks and communication systems.

ROS (Robot Operating System) – https://www.ros.org

■ Open-source framework based on a publish-subscribe messaging system (mainly used

for robotics software).

∗Certain commercial products or company names are identified here to describe the effort. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards

and Technology, nor is it intended to imply that the products or names identified are necessarily the best

available for the purpose.

Systems Interaction 17 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Implementation

Systems Interaction 18 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Implementation

̗ Reveal Driving Scenario

1. Front, Center, and Rear (Ego) vehicles driving in the far left lane on the highway.

2. Front vehicle suddenly stops in its lane (malfunction).

3. Center vehicle passes Front vehicle.

4. Rear vehicle initiates Automatic Emergency Braking (AEB).

1 2 3 4

Systems Interaction 19 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Implementation

̗ Reveal Driving Scenario

1. Front, Center, and Rear (Ego) vehicles driving in the far left lane on the highway.

2. Front vehicle suddenly stops in its lane (malfunction).

3. Center vehicle passes Front vehicle.

4. Rear vehicle initiates Automatic Emergency Braking (AEB).

1 2 3 4

Systems Interaction 20 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Implementation

̗ Reveal Driving Scenario

1. Front, Center, and Rear (Ego) vehicles driving in the far left lane on the highway.

2. Front vehicle suddenly stops in its lane (malfunction).

3. Center vehicle passes Front vehicle.

4. Rear vehicle initiates Automatic Emergency Braking (AEB).

1 2 3 4

Systems Interaction 21 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Implementation

̗ Reveal Driving Scenario

1. Front, Center, and Rear (Ego) vehicles driving in the far left lane on the highway.

2. Front vehicle suddenly stops in its lane (malfunction).

3. Center vehicle passes Front vehicle.

4. Rear vehicle initiates Automatic Emergency Braking (AEB).

1 2 3 4

Systems Interaction 22 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Implementation

̗ Simulated Reveal Driving Scenario

Figure: Reveal scenario with Carla, Autoware, and ROS.

Systems Interaction 23 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ 3. Evaluation

Systems Interaction 24 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ ADSIE Framework

■ NIST proposes the Automated Driving Systems Interaction Evaluation (ADSIE)

framework.

■ Enables stakeholders to create, evaluate, and implement testing scenarios aimed at

capturing the system interaction performance of automated driving features.

■ Aims to accelerate the manufacturing and adoption of automated driving technologies.

■ The ADSIE framework consists of 6 pillars: Systems

Which AV systems

are currently being

examined?

Scenario

What are we

exposing theAV

to?

Metrics

What are we

measuring?

Behavior

Assessment

Which behavior(s)

are we assessing in

the scenario?

Perturbations

What are we

stressing?

Uncertainties

What uncertainties

are propagated?

Systems Interaction 25 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ Approach for Systems Interaction Evaluation

1. Select an interaction pathway.

2. Select a use case.

3. Identify one of the outcomes for the use case.

4. Implement the use case.

5. Evaluate the performance of the AV.

Systems Interaction 26 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ 1. Select an Interaction Pathway

■ Example: Cybersecurity–Communication pathway.

Systems Interaction 27 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ 2. Select a Use Case

■ Example: Denial-of-Service (DoS) Attacks.

■ An attacker floods the network with excessive traffic.

■ Such attacks could disrupt communication between vehicles or between a vehicle and

infrastructure.

■ Result: Affect the vehicle’s ability to operate safely.

Systems Interaction 28 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ 3. Identify One of the Outcomes for the Use Case

■ Example: Latency

■ DoS attacks overwhelm a target system with a flood of traffic.

■ The target system becomes slow or unresponsive.

■ The increased traffic and the effort to mitigate the attack can lead to latency.

Systems Interaction 29 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ 4. Implement the Use Case

■ Example: Use ns-3 to introduce

additional delays in offboard

communications.

■ Front vehicle continuously sends its

current position, acceleration, and a

time stamp to ns-3 server.

■ ns-3 server sends this information to

Rear vehicle with an additional delay

(ms) for the position and the

acceleration.

■ Rear vehicle initiates Electronic

Emergency Brake Lights (EEBL) as

soon as Front vehicle decelerates.

■ Rear vehicle comes to a full stop in its

lane.

Systems Interaction 30 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ 5. Evaluate the AV Performance

Apply the ADSIE framework to the AV performance evaluation.

1. Systems

■ Cybersecurity and Communication.

2. Scenario

■ In this scenario, three vehicles (Front, Center, and Rear) are driving in the far left lane on

a highway. Front vehicle stops in its lane due to a malfunction. Center vehicle swerves to

the right lane to avoid Front vehicle. Rear vehicle initiates EEBL as soon as Front vehicle

decelerates.

3. Metrics

■ Distance between Rear (ego) and Front vehicles.

4. Behavior Assessment

■ Obstacle avoidance from EEBL.

5. Perturbations

■ Additional delays through ns-3 for V2V.

6. Uncertainties

■ V2V.

Systems Interaction 31 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ Final distance between ego and front vehicles

Systems Interaction 32 ʢ 37
Systems Interaction  ̈ How is the Study of Systems Interaction Performed?  ̈ Evaluation

̗ Final distance between ego and front vehicles

Systems Interaction 33 ʢ 37
Physical Testbed  ̈ Roadmap Development

■ NIST is working with VTTI to develop a roadmap to a measurement process for AV and

ADS performance as it relates to safety from simulation-based to physical testbeds.

Figure: Roadmap development (credit: Virginia Tech Transportation Institute (VTTI)).

Systems Interaction 34 ʢ 37
Physical Testbed  ̈ Development Mule

■ Perception – Static testing (range performance of sensors due to sensor degradation)

and dynamic testing (evaluate the localization performance in cases of sensor

degradation and GPS denial).

■ AI – Validate the initial AI test methods.

Figure: Ford Fusion 2020 Hybrid†(credit: https://www.dataspeedinc.com).

†Commercial equipment and materials may be identified to specify certain procedures. In no case does such

identification imply recommendation or endorsement by the NIST, nor does it imply that the materials or

equipment identified are necessarily the best available for the purpose.

Systems Interaction 35 ʢ 37
Summary

■ Problem Statement

■ Current testbeds for AVs focus on individual system testing and full vehicle testing.

■ NIST aims to investigate a systems interaction testbed and evaluation framework that

bridges the gap between individual system testing and full vehicle testing.

■ Current Progress

■ Identified interaction pathways and use cases between AI, cybersecurity,

communications, and perception.

■ Designed and AV systems interaction architecture to manage and facilitate

communication and collaboration among the AV systems.

■ Implemented on-road driving scenarios in simulation.

■ Evaluated the performance of AVs with the ADSIE framework.

■ Next Steps

■ Revise the interaction use cases.

■ Evaluate the performance of AVs with different interaction use cases.

■ Allow stakeholders to leverage NIST resources for testing and evaluation.

■ Start work on physical testbeds: Work with VTTI on the roadmap and perform various

testings with the development mule.

Systems Interaction 36 ʢ 37
Questions?

̗ Questions?

Systems Interaction 37 ʢ 37