# Presentation: 205168

## System Certification (Safety Assurance) of WAAS 

**Deane Bunce**
**SBAS Approval Workshop****	**
**21-22 June 2005**

## System Certification

**Verify Performance Requirements Met**
- Accuracy
**Integrity **
- Continuity
- Availability
**GAO Report of 2000**
- FAA Underestimated Complexity of Proving the Integrity Requirement Satisfied
- FAA Did Not Closely Monitor the Contractor’s Efforts to Demonstrate Integrity
**Recommendations**
- Establish Checkpoints To Determine Contractor’s Approach For Meeting Performance Requirements
- Have External Organization Evaluate Agency Progress At These Checkpoints

**Notes:**
WAAS is a Complex System whose Safety Cannot be Demonstrated Solely  by Test
Failures in WAAS can Result in Hazards to Users
Must Verify System Design Mitigates Hazards or Reduces to Acceptable Risk 

## Approach for Assuring WAAS Integrity 

**Elements to Consider**
- WAAS Hazard Assessment
- WAAS Architecture
- Need for Technical Oversight (Independent Verification)

## WAAS Hazard Assessment

| Category | Severity | Causes | Probability Limit |
| --- | --- | --- | --- |
| HMI | Hazardous/ Severe-Major (Cat 1 PA) 
Major (ER/NPA) | Failure to Detect Broadcast of Misleading Information (MI)
Failure to Alarm Detected Broadcast of MI Within the Specified TTA | 1E-07 |
| LOC | Minor | Hardware/Software Failure Resulting in Loss of Signal in Space
Corrections or Monitor Trip Resulting in Transition or IGP to NM or DU State
Detection of MI Resulting in User Alarm
 | 1 - 5.5 x 10-5 per approach |
| Personnel | Minor to Catastrophic (per MIL-STD-882D) | Miscellaneous (Electrical, Hazardous Materials, Physical Hazards) | N/A |

## WAAS Architecture

**Correct – Then – Monitor Architecture**
- Trusted Processing for HMI Risk Limited to Safety Processors
- Data Verification and Integrity Monitoring Algorithms Designed to Detect and/or Bound Sources of Data Errors
**Must Verify System Architecture/Design Mitigates HMI Hazards or Reduces to Acceptable Risk**
**Safe Behavior Cannot Be Verified Completely Through Testing for Complex/Non-Deterministic Algorithms**
**System Integrity Performance Is Not Completely Quantifiable**
**Process-Based Assurance Method Required to Assure all Threats to Integrity were Addressed**

**Corrections Processor**

**Safety Processor**

**Uplink/GEO**

**User**

**Monitors Data**
**(Level B)**

**CRC protects data**

**Generates data (Level D)**

**Satellite Signals**

**Receiver**

## Process Based Assurance Method

**Safety Assurance:  Assure that WAAS Design is capable of Meeting System Safety Requirements (Integrity and Continuity) **
**Method of Assurance Based on Aircraft Certification Standards / Prior Navaid Applications**
- SAE ARP 4754/4761, RTCA DO-178B
- Consists of Audits/Reviews of Contractor System Development and Safety Assessment Activities
**Process Captured in Safety Assurance Process Requirements Document [AVR/ATS]**

## Safety Assurance Process

**Assurance Process:  The Planned and Systematic Actions Necessary to Provide Adequate Confidence That a Product or Process Satisfies Given Requirements **

**Safety**
**Assurance**

**System Safety Program Plan**

**MIL-STD-882**

**DO-178B**

**AVR**

**ARP 4761/4754**

*Safety*
*Standards*

*Assurance*
*Guidance*

**Safety**
**Assessment**

**System**
**Development**

**System**
**Development**
**Plan**

**Mission Need Statement**

**Design**
**Implementation**
**and**
**Integration**

**System**
**Design**

**Requirements**
**Definition and**
**Maintenance**

**Program**
**Requirements**

**Verification**

**Requirements,**
**Design, and**
**Code**
**Review**

**Hazard**
**Analysis**
**and**
**Identification**

**Verification**
**Review**

*Integrity Threats*

*Requirements, Design , and Implementation*

*Design*

*Verification Methods and Results*

*Process Input*

**Safety**
**Assurance**
**Requirements**

**ARP 4761**

**ARP 4754**

## System Development Activities

**System and Component Development Activities to Assure Flow-down of Safety Requirements into the WAAS Implementation**
- System Engineering, Specification, and Architecture Design
- Component Design Specification
- Implementation and/or Procurement
**Engineering Processes of Quality Assurance (QA), Configuration Management (CM) and Verification for All Development Activities**

## Safety Assessment Activities

**System Hazard Assessment**
- Fault Tree Analysis
- Algorithm P(HMI) Analysis
- Safety Processor Input Analysis
- Safety Directed Analysis
- Exposure Time/Latency Analysis
- Failure Modes and Effect Analysis (FMEA)
- Common Cause Analysis
**Safety Risk Assessment – HMI Hazards Sufficiently Controlled**

## Safety Assurance Process Requirements

**Audits/**
**Reviews**

**Artifacts**

**Reports/**
**Deliverables**

**Vendor**

**Vendor/FAA**

**Baseline**

## Assurance Approval Process

**WIPP**
**Approval**

**Safety **
**Assurance Complete**

**Monitor Development **
**& Validation Complete**

**Safety Assessments **
**& Requirements **
**Verification**

**Establish Baseline **
**of **
**Safety Artifacts**
** **

**Safety Design **
**Approval **

## Assurance Approval

**WAAS Program Office Establishes a WAAS Integrity Performance Panel (WIPP)**
- Role is to Establish Technical Approach and Solution for Meeting Integrity
**WIPP Comprised of WAAS Experts ... (FAA, Academia, Industry)**
**WIPP Meetings with Contractor Held Monthly to Review and Evaluate Contractor’s Progress**

## WIPP Tasks

**WIPP Tasks**

**SV 19**

**Iono**

**Tropo**

** ****WRS RFI,**
**Multipath**
**Cycle Slips**

**GPS**
**Ephemeris**

**L1/L2**
**Biases**

**WRS Clocks**

**A Great Stew of Feared Events Each Requiring:**
** ****Threat Model**
** ****Detection Algorithm**
** ****P(HMI) Analysis!**

**Target Level**
**of Safety**

**Coding**
**Operating System**
**Computer HW**
**etc.**

***Algorithm ***
***Contribution to HMI***

**Level D SW**

## WIPP Role

*Ready for *
*Test*

*WIPP*
*Approval*

*WIPP*
*Approval*

*WIPP*
*Approval*

*WIPP**Guidance*

***HMI Analysis***

***Algorithm Development***

*Provability*
*Assessment*

*WIPP*
*Approval*

**WIPP Role**

- DefineAnalysisMethodology

- Threat
- Model
- Monitor
- Design &
- Trades

- Evaluate
- with Field
- Data +
- Threat

- Algorithm
- Definition
- Document
- (ADD)
- Coding Spec
- HMI Plan &
- Prelim Results

- Prototype
- & Validate

- Operational
- Software

- Sys.
- Integ.

- Prelim.
- Eval.

- Collect
- Data/Develop
- Tools

- Data-
- Driven
- Analysis

- Theoretical
- Analysis

- HMI Analysis
- Report

## WAAS Design Assurance

**WAAS Design Assurance can be Broken down into Three Major Areas**
- Software Assurance
- Hardware Assurance
- Algorithm (Monitor) Assurance
**Assurance that Hazards were Sufficiently Mitigated was Evaluated by Means of a Fault-Tree Analysis (FTA)**
**Events in Fault Tree Broken Down by Hardware, Software and Algorithm (Monitor) Contributions**

## Software Assurance Approach

**Assurance Level Requirement Assigned According to RTCA/DO-178B Guidelines**
**Assigned According to the Hazard Severity Caused by Software Failure Based on the WAAS FTA**
- HMI – Assurance Level B (Where Software Failure is a Single Point Cause of HMI Without Mitigation or Control; or Software Function Provides Mitigation/Control of an HMI Failure Condition)
- LOC – Minimum Assurance Level D (Where Software Failure is a Single Point Cause of LOC Without Mitigation or Control)
**Software PHMI Contribution is Assigned Based on Assurance Level Achieved**
- HMI – Assurance Level B Software is Assigned a Failure Probability of Zero (0); Assurance Level D Software is Assigned a Failure Probability of One (1)
- LOC – Assurance Level B or D Software is Assigned a Failure Probability of Zero (0)
**Monitor Algorithms Failure to Detect Probability is Assessed Based on Probabilistic Analysis**

## Software Assurance – Alternate Means

**Some Level D Functions Required Assurance **
**Developed a Process called Safety Directed Analysis (SDA)**
- Software Tested and Analyzed to Assure Output Always What is Expected
- Result was Assignment of a Zero Probability for Failure in Fault Tree
- Process and Results were Coordinated with FAA National Resource Specialist (NRS) for Software

## Hardware Assurance

**Assurance Completed at a Functional Level by means of a Failure Modes and Effects Analysis**
**Failure Rates of Hardware Utilized were Based on Several Years of Empirical Data**
**Exposure Times for Failures Captured as Part of Latency Analysis and Rolled into the Fault-Tree**
**Un-trusted Communications Assured by Means of Cyclical Redundancy Checks (CRCs) **

## Algorithm Assurance

**Algorithms Developed to Detect Both Internal and External Faults**
- Internal Faults:
- Data Provided to Monitors from Un-trusted Components was Assumed to be Corrupted
- Required Analysis of all Inputs to the Safety Processor
- Analysis Approach called Safety Processor Input Analysis (SPIA) Considered Corrupted Data, Absent Data and Data Timeliness and the Algorithms (Monitors) ability to Detect Such Conditions
- External Faults or Environmental Conditions Evaluated by Means of an HMI Analysis Methodology

## COTS Use on WAAS

**Applied to Level D (or Lower) WAAS Functions**
- IBM AIX OS
- Network Management Software
- Compilers, CM Tools
**Required to Meet All Applicable RTCA/DO-178B Objectives**
**Assured in Accordance with Allocated Functional Requirements Within the WAAS Application**

## WAAS Network FTA

## WAAS Network FTA

**LNAV-100**

**LNAV-200**

**LNAV-300**

**LNAV-400**

**LNAV-500**

**LNAV-700**

## WAAS Network FTA – Data Transmission

## WAAS Network FTA – SW Failure

**Bias error exists leading to MI**

**Bias error exists and Range Domain Monitor (RDM) FTD**

**RDM FTD  bias errors**

**LNAV-192**

**LNAV-114**

**LNAV-193**

**G= 4.50E-08**

**G= 1.00E+00**

**G= 4.50E-08**

- Note:  LNAV-192 in this example is the result of any one of satellite L1-L2 bias error, receiver L1-L2 bias error, or receiver clock bias error.  Since the error can result from untrusted software or hardware in any of these sources, the worst-case probability is used.

## Safety Assurance Process Compliance Verification

**FAA and Contractor Assure System Safety Program Plans (SSPPs) Address Safety Assurance Process Requirements**
- Contractor SSPP
- Captures When and How Requirements are Satisfied
- FAA SSPP
- Schedule of Reviews Contingent on Contractor Plan
- Establish Audit/Review Teams and Define Roles/Responsibilities
- Seek Plan Approval
**Safety Assurance Audit and Review Activity**
- Contractor Coordinates Artifact Schedule and Availability
- FAA Verifies Artifact Compliance with SAPR
- Completion of Activity Results in Safety Artifact Baseline

## Safety Assurance Documentation Baseline

**Established to Provide Evidence of Compliance with the WAAS Safety Assurance Process Requirements**
**Includes:**
- Program Planning and Standards Definition
- System Specification, Requirements Definition, and Design
- Component Specification, Requirements Definition and Design
- System Safety Assessment
- Component Implementation
- System Integration
- Configuration Management
- Quality Assurance
- Safety Assurance Process Compliance

## Program Plans and Standards

- System Engineering:  System Engineering Management Plan, Reliability Program Plan
- Software Engineering:  Software Development Plan, PSAC, SAS and Supporting Process and Standards Notebooks
- System Test:  Master Test Plan, Development Test Plan, Production Acceptance Test Plan
- System Safety:  System Safety Program Plan, Supporting Process/Procedure Notebooks, Audit Records

## System Specification and Design

**System Specification**
**System Safety Architecture Description**
**Safety Requirements Traceability Matrix**

## Component Specification and Design

**Prime Item Development Specifications**
**Software Requirement Specifications**
**Software Design Documents**
**Software Product Specifications**
**Algorithm Description Documents**
**WIPP Minutes**

## System Safety Assessment

**System Hazard Analysis Report**
**Fault Tree Analysis**
**Algorithm Contribution to HMI**
**Safety Processor Input Analysis**
**Safety Directed Analyses**
**Qualitative Assessments**
**Failure Modes and Effects Analyses/Summaries (FMEAs/FMESs)**
**Failure Rate Source Documentation**
**Common Cause Analysis**
**Hazard Closure Records**

## Component Implementation

**Hardware Test Plans, Test Procedures, and Test Reports**
**Software Test Plans, Test Procedures, and Test Reports**
**Software Test Folders/Regression Test Folders**

## System Integration

**Design Qualification Test Procedures and Reports**
**Reliability Test Procedures and Report **

## Configuration Management

- CM Plan
- Configuration Audit Plan
- Change Control Records
- System Configuration Index

## Quality Assurance

- Quality Control Program Plan (or Quality System Plan)
- Quality Audit/Review Logs

## Safety Assurance Process Compliance

**System Safety Program Plan**
**Safety Assurance Audit Process**
**Safety Audit Records**
**WAAS Safety Assurance Configuration Item List (SACIL)**
- Identifies List of Artifacts Required for Proof of SAPR Compliance
- Defines Artifact Association to SAPR Requirements
- Asserts Configuration Control over Artifacts Under Review
- Creates and Maintains an Artifact Baseline Required for Formal Approval

## SACIL Excerpt

## Maintenance of WAAS Safety Baseline

**Rationale for Safety Requirements a Must**
**Requirements Traceability **
**Baseline Definition and Control **

## Questions?

## Backup Slides

## System Development and Safety Assurance Process Model

## Overview of Safety Assurance Activities

-        VV/CC/QA

- Verif / Valid/CC/QA

-         VV/CC/QA

-            VV/CC/QA

-    VV/CC/QA

- Verif / Valid/CC/QA

- Verif/Valid/CC/QA

- Algorithm failure

- rates

- CRC failure rates

- Quantitative HMI FTA

- HMI FT base event

- exposure times

- HW HMI failure rates

- from FMEA

- System architecture & functional descriptions

- Algorithm data

- CRC data

- HW component

- data

- Other monitoring

- check data

- Qualitative

- HMI FT structure

- Safety directed analysis & test

- of critical level D SW functions

- Quantitative HMI FTA

- assurance review

- Qualitative

- HMI FT structure

- assurance review

- Safety directed analysis & test

- assurance review

- Alg failure rates assurance review

- CRC failure rates

- assurance review

- HW HMI FMEA

- assurance review

- HMI FT base event

- probability calculations

- HMI FT base event

- exposure times

- assurance review

- SW component

- data

- Design

- assurance

- reviews

- HMI FT base event

- probability calculations

- assurance reviews

- = Data flow - intersecting pathes

**are**

-  connected.

-  = Data flow - Intersecting pathes are

**not**

-  connected.

- Legend:

- = Safety team safety assurance activites.

- = Raytheon activites.

- = Raytheon verification, validation, config control, & QA  activities

-    (placed behind corresponding Raytheon development activities)

- Level B SW

- DO-178B

- compliance

- review

- Procedural

- mitigation

- assessment

- Procedural

- mitigation

- cert review

- Start

- End

## PHMI Analysis - What Is It?

**Probability of Monitor Missing an Error that would cause HMI has to be less than the assurance level (on order of 10****-7****)**
**DO-178B Level D inputs can NOT cause HMI when integrity monitor is operating as designed**
**Test 30 Days of WAAS Data (UDREI & GIVEI) for Overbounding**
**Evaluate Worst Case Convolutions of UDREI and GIVEI Histograms**

## PHMI Analysis - How Does it Support FTA 

## PHMI Analysis - How Does it Support FTA 

- 10-7  P(HMIE) + P(HMIEC)

**Output of Algorithm PHMI Analysis Provides the Probability of Missed Detection based on a Fault free WAAS**
**SPIA, SDA, FMECA and Latency Analysis Provide Probability of Missed Detection based on Faulted WAAS (SDA and SPIA qualitative)**

## SPIA - What is it?

**Analysis that Determines the Erroneous Data Inputs that Could Result in HMI being Output from the ****Safety Processor (SP)**
- Define Inputs to each Monitor
- Identify any Input or Combination of Inputs that Could result in HMI
- Input data Conditions include: corrupted data, absent data or data timeliness
- Determine if the fault condition(s) is covered by existing monitors
- Determine the Dependency or Relation to Other Monitors
- Backtracks the contributing data to the input source
- Assess the likelihood for the contributing data being corrupted

## SPIA - How does it support FTA? 

**SPIA Complements and Validates the FT**
- Identifies Events that Cause HMI
- Verifies Events are Captured by FT; or
- Contributes to FT Completeness
- New Failure Event Identification
- Qualitative Event Probability Assessment
- Contributes to Resolution
- No Action for Extremely Low Event Probability; or
- Specific Identification of Level D Software Functions Which may be Assured by SDA; or
- Need for Design Change to Mitigate Failure Event

## SDA - What is it? 

**Qualitative Analysis of DO-178B Level D critical software functions identified in the WAAS fault tree **
- Critical Level D software functions are defined as those that prevent satisfaction of WAAS safety performance requirements
- For Fault Tree Analysis, Level D software has a failure probability of 1
- Safety Directed Analysis is applied to the Level D software that performs these critical functions
- The SDA demonstrates through analysis and test that software threats can be assigned a failure probability of 0
- Ensures that the Level D software is reliable enough to support the WAAS System safety performance

## SDA - How Does it Support FTA? 

**Successful SDA Eliminates Software Failure Events from the Fault Tree**
**Failed SDA assessments will identify Level D functions that must be mitigated**
- Software design change will be implemented to monitor the Critical Level D function or rehost it to Level B
**Fault Tree will be modified to incorporate the updated design and reduced PHMI**

## FMEA/FMES - What is it? 

**Failure Modes and Effects Analysis/Summary**
**Systematic, bottom-up method of identifying:**
- Failure modes of a system, component, or function and effects on next higher architecture level
- Failure rate or probability of occurrence of failure modes at each level (limited to hardware on WAAS)
- Also contains ability to detect severity of failure
**May be performed at any architectural level**
- For WAAS most done at LRU level

## FMEA/FMES - How Does  it Support FTA?

- Data extracted from ENB 1-6-30C
- Figure 3.1-1

## Exposure Time/Latency Analysis - What is it? 

**Analysis of Exposure Time:**
- e.g. 150 secs, PA; 1 Hr NPA
- System element under analysis known good at beginning of exposure time.
**Latency accounts for components not continuously tested**
- The estimate of how long a failure mode can be present in an equipment or function before it is mitigated (or removed).
- Requires failure detection, reporting, and mitigation
- Ground rules: Monitors must be in Level B software and not already included in FTA at higher level.

## Exposure Time/Latency Analysis - How Does  it Support FTA?

- Reliability (Probability of Success) = e-FR*t

- Where 	FR = Failure Rate = 1/MTBF
- 	t = Exposure Time or Latency
- 	Exposure time for PA = 150 seconds, other phases 1 hour

- Probability of Failure (Pf)  = 1- Probability of Success

- Safety Computer Comparator Example:
- Safety Computer (SC) failure rate = 5.08 x10-5 Failures/HourIf testing were continuous, t = 150 seconds (during PA), 			      Pf = 2.11 x10-6 Accounting for SC MTBF (19,704 Hours) Latency, Pf = 0.632Accounting for C&V MTBF (872 hours), Pf = 0.043

- Lowest FTA element requires Probability of Failure

- Source of failure rates (ENB 1-13-107)

## SSAD

**Provide System Architectural and Design Details **
**Describes the Safety Architecture from System to Component to Capability Level**
**Provides Requirements Rationale and Tracing**
**Identifies Analysis to Validate and Verify Requirements at System and Component Level**
- System Satisfies Integrity Requirements
- Failure Modes and Effects Mitigated

## System Safety Architecture Description

***SSAD***

**A121**

**Capability**
**Requirements**
**(CIs, SRSs)**

**Component**
**Capabilities**

**System Safety**
**Requirement**
**(P****HMI****, Cont’y)**

**System Safety Architecture**
**Components**

**System Safety Analyses**

**FTA**

**CCA**

**HEA**

*Hazards*

**Algorithm Design Documents**

**Verification Methods and Results**

**SDFs**

**System Test Procedures**

**System Test Results**

**Operator**
**Safety Procedures**

**IETM/TIB**
**Safety**
**Features**

**IETM/TIB Verification**

**Operational Sequence**
**Diagrams**

**Software**
**Safety**
**Features**

**Feature**
**Requirements**
**(SRSs)**

**SAC Analyses**

**Alg PHMI**

**SPIA**