National Aeronautics and Space Administration

www.nasa.gov

Airspace Operations and Safety Program (AOSP)

V&V Research Overview 

For Human Factors REDAC Committee

September 2, 2015

Jessica Nowinski,

Safety Integration Manager

1

1

AOSP Organizational Chart

PROGRAM STAFF

AOSP Portfolio and Systems Analysis Team

Lead: Yuri Gawdiak

Director: John Cavolowsky

Deputy Director: Akbar Sultan

(TC 1) ATD-1

Jane Thipphavong

PROJECTS

Associate Director: Yuri Gawdiak

Technical Integration Manager for NextGen: Barry Sullivan

Safety Integration Manager: Dr. Jessica Nowinski (Detailee)

Technical Integration Manager for ARC: Tom Davis

Technical Integration Manager for LaRC: Mark Ballin

Program Resource Analyst: Michele Dodson (RMO)

Program Management Analyst: Kenny McCombs

SMART-NAS Project

Dr. Shon Grabbe

SASO Project

Dr. Parimal Kopardekar

ATD Project

Leighton Quon

AOSP Risk Management Board

Chair: John Cavolowsky

Executive Secretary: Yuri Gawdiak

2

2

SMART-NAS Project

Explore and develop concepts, technologies and a test bed for safe, global, gate-to-gate trajectory based operations.

3

Overview

PROJECT LEVEL 

SMART-NAS Test Bed

Leads: Dr. Kee Palopo and Mike Guminsky

Regional Trajectory Based Operations (TBO) Lead: Dr. Paul Lee

Safe Avionics and ATM Future Evolution (SAAFE)

Lead: Dr. Paul Miner

PM: Dr. S. Grabbe

DPM: R. Aquilina

APM: J. Koelling

APM for Technology: Dr. R. Mah

PP&C (shared):

Host Center Analyst: Julie Nguyen

Center Analyst: Paula Chambers, LaRC

NRA manager: N. Galeon

Systems Engineering Office (shared): 

Lead: R. Johnson

Scheduler: J. Cronauer

Risk manager: S. Kurasaki

Technical Challenges

Emerging Technical Challenge

3

Safe Autonomous Systems Operation Project

Define and safely enable all future airspace operations (ATM+3) using justifiable and optimal autonomy for advanced air and ground capabilities 

PM:  Dr. Parimal Kopardekar

DPM:  Sharon Graves

APM: Craig Hange

Resources:  Julie Huynh   

Scheduling:  Joseph Cronauer

Risk Management:  Seth Kurasaki

NRA Manager:  Naz Galeon

Technical Challenges

Foundational, Cross-Cutting or Exploratory R&D

SASO Technical Advisory Committee: Dr. Danette Allen, Tom Davis, Davis Hackenberg, Dr. Sanjay Garg, Dr. John Ryan, and Don Simon

Autonomous aircraft operations 

Autonomous traffic flow management 

Autonomous localization, sensing, and conformance

Autonomous Airport 

Autonomous AOC 

Additional items from ARC, AFRC, GRC, and LaRC to be considered as part of current planning

PROJECT 

Emerging Opportunities (Candidates for TC in out years)

V&V and Certification Methods for Autonomous Operations (Guillaume Brat and Paul Miner)

Analysis Studies (Stakeholder survey, benefits, affordable air mobility) (Jeremy Smith)

Human-autonomy Teaming (TBD)

Communication, Navigation, and Surveillance (TBD)

 TC1: Enabling Low-Altitude UAS Operations

Lead: Dr. Joey Rios  

TC3: Ab Initio Architecture 

Lead: Dr. Natalia Alexandrov 

TC2: Enable Reduced Crew Operations

Leads: 

Randy Bailey and 

John Kaneshige

4

4

Safe Avionics and ATM Future Evolution 

Overview

Legacy System-wide Safety and Assurance (SSAT) 

Research Activities 

Assurance of Flight Critical Systems TC (SSAT)

National Needs and Stakeholder Interests

Inter-Center Planning Team

FAA V&V Tech Summit

ARC, GRC and LaRC Team Members 

SAAFE TC

FAA/ANG/AVS

Numerous Systems Assurance NRAs

NASA Strategic Thrust 1: Safe, Efficient Growth in Global Operations

NASA Strategic Thrust 6:  Assured Autonomy for Aviation Transformation

Decadal Survey of Civil Aeronautics

National Aeronautics R&D Policy, 2010

NASA Strategic Thrust 5: Real-time System Wide Safety Assurance 

5 

5

5

Goals, State-of-the-Art and Benefits

Overview

Goal: 

Develop methods, tools and techniques for reducing the cost of certification, verification and validation of complex safety critical systems and ATM systems, to include assessing the impact of design decisions throughout the systems evolution. (FY19, TRL 6)

Benefits: 

Development of industry-focused tools, techniques and architectural patterns to enable efficient verification and validation of complex safety-critical systems

Tools will enable the reduction of industry verification and validation costs at all stages of the systems lifecycle

Determination of what evidence is sufficient for establishing and preserving safety with the primary customer being the FAA and industry a secondary beneficiary

6 

Products: 

Document new approaches to software assurance (FY15Q5) [FAA/NASA RTR]

Complete study of regulatory compliance with multitier supplier networks (FY15Q4) [FAA/NASA RTR]

Document representative examples illustrating RTCA DO-330 (FY16Q4)

Publish assurance case applicability for digital system findings (FY17Q4) [FAA/NASA RTR]

Demonstrate semi-automatic reconstruction of safety arguments (FY18Q4)

Transfer tools to FAA/AVS/ANG and at least one avionics manufacturer (FY19Q4)

6

6

Sample Activities

Direct collaboration with FAA

ACAS-X assurance software

Direct delivery to FAA

Explicate ’78

Human Factors (Authority and Autonomy) Research

Brahms modeling efforts

7

Safety Assurance for ACAS-X

Objectives and Approach

Assist the FAA in the certification of ACAS-X. Tasks have been defined in collaboration with Neal Suchy’s verification team

Participation in the SC-147 standard committee meeting for ACAS-X

Provided V&V perspective and support to Neal Suchy’s team

The project applies  advanced  techniques for addressing assurance issues both in the design and verification of the ACAS-X controller

FY’14 internal work is focusing on validation of the MDP model used for creating table that picks advisories given some state estimation

ACAS-X (replacement for TCAS)

Markov Decision Process (MDP) models effect of advisories and uncertainty in pilot response and other airplane movement

MDP solved to generate state/action table

Logic of decisions hidden in table entries, not in the code

8

Argument Based Safety Research Map

Improving

Risk/Hazard

Analysis

Covering

All Life

Cycle

Integrating

Formal

Evidences

Automating

Creation

Managing

Complexity

STPA (MIT)

the System-Theoretic Process Analysis (STPA) approach to hazard analysis 

 Initiated transition to FAA

AdvoCATE

Formal specification of safety case patterns

Automated instantiation to create well-formed safety case instances

AdvoCATE

Extended safety cases with hierarchical abstraction

Specification of queries to produce safety case fragments, reflecting stakeholder-specific views

New NRA with Dependable Computing started June 2013

Extending CertWare with Hazard analysis and Design Info

AdvoCATE: Safety case patterns as the basis for integrating formal methods into safety cases

Extending CertWare with integration of AADL specifications 

9

RTR Activity: Explicate ‘78

“Document in an argument-based form the implicit arguments contained in DO-178C that assert that compliance with the guidance provides sufficient confidence that software systems are safe enough for their purpose.”

Explicate ‘78 report delivered to FAA last year

What are the safety arguments implicit in DO-178C?

Final report documents a complete initial articulation of an explicit, comprehensive assurance case for the guidance contained in DO–178C. 

Also contains information about 

DO–178C and its associated documents, 

assurance cases, 

the Goal Structuring Notation, and 

approaches to evaluating arguments.

Work jointly funded by FAA and NASA under interagency agreement IAI-1073

Work continuing, in conjunction with NRA partners and Phase III SBIR, in support of FY17 RTR Deliverable to the FAA

10

Overview

Developed an approach based on model checking for evaluating the roles and responsibilities of humans and automation. 

Benefits

Can better assist in identifying authority and autonomy issues.

Description

Provides a formal framework for assessing safety in models of human/machine roles and responsibilities in organizations. The approach may involve extending existing modeling languages with formal semantics to support formal analysis.

Formal Analysis of Human-Machine Systems

11

11

Example: Air France 447

Air France flight 447 from Rio de Janeiro to Paris:

Failure of a sensor (a set of pitot tubes) resulted in incorrect speed readings and through a sequence of events lead to a stall which an inexperienced pilot failed to diagnose

Agents Modeled

1 Human agent (pilot)

2 Automation agents

Developed a framework to formally model and 

      verify degrees of belief in a system of agents 

Verified a property encoding the question 

Does a state exist, that is reachable from the initial state, in which the pilot believes that the stall is actually occurring with a degree of less than 5%?

We are able to show in less than 1 hour that there are 5 stalling scenarios in which the pilot would be very unlikely (less than 1.5% chance) to detect a stall. 

“Model Checking Degrees of Belief in a System of Agents,“ by Giuseppe Primiero, Franco Raimondi, and Neha Rungta. In Proceedings of the 13th International Conference on Autonomous Agents and Multi-agent Systems (AAMAS 2014)

12

12

Example: Laguardia Simulation Data

The human agents in the Brahms model include 3 center controllers, 3 TRACON controllers, and one tower controller.  

The Center and TRACON controllers perform handoffs as planes are crossing the relevant parts of the sector.  

They have now successfully simulated 1.5 hours of real traffic in the Brahms model — approximately 150 planes in the airspace with planes landing at a rate of 36-40 planes per hour.  

Departures take off in the gaps between arrivals.

Goal is to use simulation data and use model checking to explore off nominal events as with 447 example. 

Created working models that encode deterministic scenarios for the LaGuardia airspace based on the simulation data. 

These models use the same high-fidelity flight traffic information as the DSAS sims.  

13

14

14

Data mining efforts at NASA

Through individual agreements with major airlines we are using flight data (discrete, continuous, text reports) to develop algorithms

Anomaly Detection

Compare each flight to every other flight in the database (matching for factors such as considering phase of flight, airport)

Algorithm rank orders most anomalous flight

Subject matter experts determine operational relevance

Detected unknown anomalous events, such as low speed and steep bank angle shortly after take off

Can search data for similar (less extreme) events

Validated use of algorithm on surveillance (radar track) data at LAX, DEN and NY Metroplex (e.g. arriving flight over a parallel approach path)

15

Data mining efforts at NASA

Precursor Analysis

Given a particular undesirable state, defined by multiple variables, such as unstable approach, what are the common precursor states?

Prediction 

Given the state at point x, what is the probability of arriving at an undesirable state, such as an unstable approach

16

# image2.jpg

# image1.png

# oleObject1.bin

# image3.png

# oleObject2.bin

# image4.png

# image5.png

# image6.tiff

# image7.png

# image8.tiff

# image9.png

# image10.png

# image11.png

# image12.png

# image13.png

# image14.png