# Presentation: 943085

## Workforce Transition Strategy and Implementation

- SMA Open House
- December 12, 2005

## System Safety, Reliability and Quality Division  Mission Statement

- The GRC System Safety, Quality and Reliability Division strives to assure NASA’s mission of exploration, advancement of scientific knowledge and technology transfer by defining, evaluating, and reducing program risk.  State-of-the-art technology is utilized to manage and reduce risk resulting in program safety and mission success.

## Safety and Mission Assurance

**PB Decommissioning**
**Program Office (QD)**
*Frank J. Greco*
*Chief*

**Safety, Health, and Environmental Division**
**(QS)**
*Manuel B. Dominguez*
*Chief*

**System Safety, Quality, and Reliability Division**
**(QE)**
*Frank Robinson, Jr.*
*Chief*

**Safety and Mission Assurance**
**Directorate (Q)**
*Vernon W. Wessel, Director*
*Rafael Sanabria, Deputy Director*

**Assurance Technology**
** ****Center (QA)**
*Rafael Sanabria*
*Manager*

##                            System Safety, Quality and Reliability Division (SSQRD)

- QE
- System Safety, Quality and Reliability Division (SSQRD)

- QEA
- Assurance and Risk Management (ARM)

- QEE
- System Safety and Quality Engineering (SSQE)

- Program Assurance Managers

- Risk Mgt.

- Quality

- Software Assurance

- Reliability /EEE Parts

- Materials & Process

- System Safety

- Science

-  Aeronautics Research

- Exploration Systems

- Space Operations

-                            System Safety, Quality and Reliability Division (SSQRD)
-                              Organizational Chart

## Work Package Opportunities

## Background

- This is an effort to plan for the best use of the Agency’s total workforce in the near term (FY 2006-2007) to help maintain core competencies for the NASA centers.
- Who are the principals?
- Agency lead – Charles Stegemoller
- GRC Center POC for Agency – Rich Christiansen
- GRC JSC Team lead – Olga Gonzalez-Sanabria
- GRC KSC Team lead - Bill Wessel
- JSC POC – Robert Kelso
- How is this being implemented?
- Used Internal Task Agreements (ITAs) as the beginning basis for negotiation.

## Response Flow Diagram

- Receiving Center Develop Responses

- ITA Posted by Performing Center

- Receiving Center post Response in  ITA Drop Box,

- * Performing Center (PC) – Currently performing work (e.g. JSC, KSC, GSFC)
- + Receiving Center (RC) – Requesting to perform the work (e.g. ARC, GRC, LaRC)

- Has RC
- Demonstrated
-  That  they  Can perform
- The work
- If >1 RC Acceptable
- Rand order

- Cost analysis+

- No

- Notify RC

- Clarification
- Required
- By  RC

- Yes

- RC contact **ITA Owner** at Performing Center and discuss if ITA’s can be grouped

- No

- Does RC
- Intend to
- Group
- ITA’s

- Yes

- Create a consolidated ITA referencing original ITA’s

- ITA Owner
- Determine if  the
-  response
- Complete and
- Clear

- Yes

- No

- ITA Owner Evaluate all Responses

- ITA Owner contact RC for additional information

- No

- Does RC
- Agree

- Yes

- ITA Owner Document and close Response

- Yes

- ITA Owner Notify Selected RC and Notify RC’s that were not selected

- Does RC
- Not selected
- Agree

- Notify Program

- Yes

- ITA Owner Document and close Response

- No

- No

- Contact
- Stegemoeller to Reclama

- Does RC
- Agree

- Yes

- Contact Cabana to Reclama

- ITA CM Manager move ITA to appropriate folder and notifies ITA owner and Center POC

- Cabana Contact Geveden for Reclama if required

- ITA Owner notify Stegemoeller of recommendation

- A

## Proposals from GRC

- Proposals against existing ITAs

- Proposals for new work

## Johnson Space Center Proposal Detail

## Kennedy Space Center Proposal Detail

## Reliability and Risk Assessment Process and Evolution

**Ed Zampino**
**NASA Glenn Research Center**
**December, 2005**

## Related System Analysis Definitions

- _System Effectiveness:_  The probability that a system can safely and successfully meet an operational demand within a given time when operated under specified conditions.
- _System Reliability:_  The probability of mission success given the mission parameters such as environment, application, and duration.
- _Operational Availability_: The probability of a system being operational at a particular time  based on the  system reliability, maintainability, and the logistics of sparing.

## Setting Reliability Goals
RAM Allocations
Trade Studies and Predictions
Failure Modes and Effects
PRA: Includes Mission Event Tree and Fault Trees
Human Error Risk Assessment 
Worst Case Circuit Analysis
Parts Stress Analysis
Performance Trending
Limited Life Items
Corrective and Preventative Maintenance
Fault Detection & Isolation
Sparing and Logistics
Operational Availability

**RAM Allocations**
**Trade Studies and Predictions**
**Failure Modes and Effects**
**PRA: Includes Mission Event Tree and Fault Trees**
**Human Error Risk Assessment **
**Worst Case Circuit Analysis**
**Parts Stress Analysis**
**Performance Trending**
**Limited Life Items**
**Corrective and Preventative Maintenance**
**Fault Detection & Isolation**
**Sparing and Logistics**
**Operational Availability**

**Supplier Screening, Supplier Quality Assurance**
**Testing: Thermal cycle,  Vibration, and Burn-in**
**Minimize EMI emissions and susceptibility**
**Selection of parts for high quality level and radiation hardness to meet or exceed reliability allocations**
**Thermo-graphic Mapping of circuit cards**
**Problem/Failure Reporting and Corrective Action System (PRACA/FRACAS)**

**Reliability, Availability, and **
**Maintainability (RAM) **
**Analyses for System Design**

**Reliability Practices**
**(Examples)**

## Reliability Uses the Concept of a Probability Density Function

## Probability of Failure and Reliability 

## Fundamental Relationships in the Mathematics of Reliability

## Complex "bridge" configuration 

## RBDs and Fault Trees

- System
- Failure

- The most fundamental difference between Fault Tree Diagrams and RBDs is
-  that you work  in the "success space" in an RBD
-  while you work in the "failure space" in a fault tree.

- IE1

- IE2

- IE3

- IE4

## POTENTIAL RAPTOR MODEL OF HYBRID NTP and NEP SYSTEM CONCEPT

## Example of Fault Tree Constructed with SAPHIRE

## Partial Power Loss Mission Optionsfor Demonstration NEP Mission to Neptune/Triton

## Safety and Mission Assurance 

**System Effectiveness for a **
**Mission Architecture  **

**Safety Requirements**

**Operational Readiness**

**System Design Rules**

**Human Reliability**

**Very Low Probability  for**
**Catastrophic Failures:**
**Loss of Crew**
**Loss of vehicle**
**Inadvertent Re-entry to earth **
**Failure at Launch or assent**

**High Reliability**
**without Repair:**
**Very Long operating Life**

**Maintainability:**
**Mission Elements**

**Optimized**
**Sparing**
**Strategy**

**1. Readiness at the start  of the mission**
**2. Operational Availability**

**System design: **
**Fail Safe**
**Fail Operational**
**Failure tolerance**
**Separation of Redundant Paths**
**Design for Long Life**

**Survivability in **
**Planetary**
** ****Surface, Orbital,**
** ****and deep**
**Space Environments**

**Reduce Error Rate in:**
**Design, build, and Test**
**Maintenance **
**Operations**

**Software Reliability:**
**High Probability**
** ****of Success**
**  ****when hardware operates **
**within specifications or **
**encounters anomalies**

## Analysis Team Approach 

**Timeline**
**Environment**
**Preliminary Design**

**Design for**
**Reliability**
**at Component**
**Level**

**Set Reliability**
**or Availability**
**Requirements**

**Review**
** ****Hazard**
**Analysis**

**Conduct**
** ****FMEA**

**Software**
**FHA or FMEA**

- Construct
- System
- RBD

- Construct
- Fault
- Trees

- Develop
- CIL

- Reliability
- Allocation

- Perform
- Reliability
- Analysis

- Probabilistic
- Risk
- Assessment

**Maintainability**
**Analysis**

- Operational
- Availability
-  Analysis

- Reliability
- Analysis
- Interface

- Construct
- Event
- Diagrams

- Retention
- Rationale

- Key:
- → =
- Flow of
- Information.
- Red: Nothing Starts
- without this info.
- Green= Tasks that
- Can be started
-  in parallel.
**Orange**= Analysis that
- Requires input
-  from earlier analysis
- Tasks.

- R&M
- Analysis Team Approach

## Design for

**Design for**
**Reliability**
**at Component**
**Level**

**Selection**
**of Parts that **
**are Low Risk **
**and **
**Rad Tolerant**

**Review of **
**Derating Analysis**
** ****and Parts**
**List **

**Review  of**
**Thermal**
**Analysis &**
**Test Results**

**Development**
**of Radiation **
**Test & Evaluation**
**Plan**

**Development **
**Of **
**EEE and Mech.**
**Parts**
**Control Plan**

**Review  of**
**Radiation **
**Testing**

**Reliability **
**Analysis **
**Interface**

**Design for **
**Radiation **
**Work **
**Instruction**

**Feedback to**
**Design Team**

**GIDEP/ALERT**
**Searches**

## R&M Interfaces

**with System**
**Safety**

**Interface**
**with**
**SPA**

**Ed Zampino**
**R&M Engineer**
**Stirling and Brayton**

**Robert Makovec**
**Lead R&M Engineer**
**Electronics Technology**

**Jose Cruz**
**Lead R&M Engineer**
**LOX-Methane Development**

**R&M Engineer **
**EEE Parts**

**R&M Engineer**
**Test Methods**

**R&M**
**Lead R&M Engineer**
**Cryogenics Technology**

**Interface with **
**Design**
**Engineers**

**Ed Zampino **
**Lead**

**Project**
**POCs & R&M**
**Planning**

**R&M**
**Lead R&M Engineer**
**Nuclear Technology**

- Chart

- R&M
- Lead Engineer
- Space Station

- R&M
- Lead Engineer
- Space Shuttle

- Project Groups
- Risk Mgmt.
- Facilitators

- Ed Zampino
- R&M Engineer
- CLV

- Lead Maintainability
- Engineer

## *Includes Operational Availability and sparing Plan

**Maintainability **
**Analysis**
**CIR-ANA-1260**

**FCF Product**
** ****Assurance Plan**
**FCF-PLN-0042,Sec.7**

**FCF**
** ****R&M Plan**
**FCF-PLN-0045**

***FIR **
**Maintainability**
**Analysis**
**FIR-ANA-1001**

**CIR **
**FMEA/CILs**

**FIR**
**FMEA/CILs**

**CIR Limited Life**
**Items List**
**CIR-LST-0048**

***Includes Operational Availability and sparing Plan**

**FIR Limited Life**
**Items List**
**FIR-LST-0139**

**CIR Reliability **
**Summary Report**
**CIR-RPT-0063**

**FIR Reliability**
**Summary Report**
**FIR-RPT-0157**

**Package Reliability **
**Analysis**
**Reports**

**Logistics**
**Estimates**
**For Ground Spares**

**Maintenance**
**Task Analysis**

**FCF**
**System **
**Specification**

**TOP LEVEL TREE FOR R&M PROJECT DOCUMENTS**
**Example from  FCF Project **

## Flow down of Requirements for RAM and EEE Parts- MRDOC

**Start**

**SARG**

**Exhibit 1**
** ****and **
**Exhibit 3**
**SOW**

**CDRL**
** ****Data**
**Item **
**Descriptions**

**Product **
**Assurance **
**Plan**
** ****FCF**** **

**MRD-PLN-0007**
**EEE Parts**
**Control**

**FCF**
**System **
**Specs**

**SSP **
**50431**

**Generated Plans To**
**Support Contract Requirements**

**R&M Plan**
**For **
**FCF**

**Reliability**
**Allocations for **
**Design **
**Packages**

**Design**
** ****Package**
**Specs**

**Design Process**

## Explanation on “Physics-Based” 

- Traditional Reliability Analysis and Risk Assessment models are often constructed as block diagrams, event trees, or fault trees.
- Event frequencies, failure rates, or Probability of occurrence/failure, are often obtained from comparison to similar equipment.
- Probabilities of failure that are Physics-Based are derived from the stress and strength distributions and include consideration of design variable uncertainties.
- Improves credibility, confidence, and is derived from physical factors and design variables

## Physics Based Reliability Assessment

## Physics-Based Multi-disciplinary PRA Design System for Reliability

**Physics-Based Multi-disciplinary PRA Design System for Reliability**

## Tentative Reliability & Maintainability Training

- Given: We will have skilled engineering and analytical people from GRC with experience on NASA space or aeronautics programs.
- Will have Instructors from R&M and PRA tool manufacturers come to GRC- coordinated through ATC.
- Classroom Training
- Practitioner’s classes on PRISM (RAC), SAPHIRE (INEEL), and RAPTOR (ARINC)
- NSTC 017- Design for Availability
- NSTC 008- System Safety Fundamentals ( Teaches the interrelationship between Qualitative tools used for both Reliability and System Safety: L. Gregg )
- Probabilistic Risk Assessment (INEEL)
- On-Line Training
- SMA-017-01 Failure Modes and Effects Analysis (SOLAR)
- SMA-080-02 Fault Tree Analysis (SOLAR)
- SMA-051-03 Reliability and Maintainability Overview (SOLAR)
- SMA-016-01  EEE Parts Assurance (SOLAR)
- Best Practices for Maintainability (PBMA)
- On the Job Training: Quality one-on-one Mentorship
- Review of helpful  NASA R&M Guideline and Training Documents (examples:  NASA-STD-8729, NASA/TP-2000-207428 Reliability Training Manual)
- Review of Example Study Reports for some Previous Projects
- NASA TM 4322 NASA Preferred Practices for Design and Test
- Familiarization with Data Handbooks NPRD, EPRD, MIL-STD-217
- Helpful Text Books we use in QEE

## EXAMPLEApproach for Risk-Based Comparison: System Architectures for CEV Project

- Jose Cruz and Ed Zampino
- Reliability Engineering
- NASA GRC
- April 6, 2005

## Description of Risk and Definition of Risk 

**Risk involves three important components:**
**    ****- *****What can go wrong? -*****The Event sequence leading**
**       ****to the Undesired Failure event.**
**    ****- *****How likely is it? - *****The probability of occurrence.**
**    ****- *****What are the consequences? - *****A measure or**
**      ****description of the severity of the possible outcome.**
**In general, **RISK = Probability of occurrence x Severity.
** **
**Recommend that a Risk matrix be used to display the results of risk analysis for CEV options- Severity vs. Probability.**
**Recommend that we do not multiply Severity x Probability as it may blur the significance of the two factors in relation to a specific architecture or failure scenario.**

## How we think of Risk Assessment in R&M as applied to Spacecraft Systems:  Risk Assessment: The application of analytical methodologies in order to identify and evaluate the risks associated with undesired system level End-State events. (This may pertain to the safety and/or the reliability of a system and thus, its impact on people and objectives.)  Probabilistic Risk Assessment (PRA): Goes beyond merely finding the severity and probability for undesired End-States. We identify single failure points. We evaluate the uncertainties and measures of  importance to conduct comparisons of system design alternatives. 

## Proposed Risk Assessment ApproachQualitative Analysis  

- Mission/Vehicle
- Profile and
- Architectures
- 1st Concept

- Mission/Vehicle
- Profile and
- Architectures
- 2nd Concept

**Mission Event**
**Sequence**
**Diagram**

**Fault Tree**
**Analysis of**
**Mission Events**

**Mission Event **
**Sequence**
**Diagram**

**Fault Tree**
**Analysis of**
**Mission Events**

- Assess Fault Trees
- Causal Structure- Cut Sets
- What can go wrong?

**Scenario Development**
**Scenario Qualitative**
** ****Assessment**

## Establish Frequency or Probabilistic Estimates for Basic Events 

- Consistent sources
- Consider Launch vehicle experience based data from similar missions
- Similar equipment data
- Expert Elicitation Technique
-     - Knowledge of technology and application by domain
-       experts
- Using Probabilistic Design Tools- NESSUS, ANSYS, NESTEM, CALCE

## Quantification 

- Determine Probabilistic Output Parameters:
-     - Examples: Probability of Top-Level Events
-                         Importance Measures
-                         Uncertainty
- Interpret and Compare Probabilistic Estimates for the two mission/vehicle architectures
- Construct Risk Matrix for Options
-     - Probability vs. Severity
-     - Labeled by Key Associated Metrics- Vehicle Hardware Mass,
-       Vehicle Fuel Mass, Payload Mass

## References

- Diagrams and Pictures taken from previous presentations for the Reliability and Maintainability Symposium by Ed Zampino, NASA GRC.
- Some graphs and equations were taken from http://www.weibull.com
- For more information see http://www.weibull.com

## Electrical, Electromechanical, 

**and Electronic (EEE)**
**Parts Assurance**

**By Ed Zampino**
**and Robert Makovec**
**NASA Glenn Research Center**
**December, 2005**

## Definitions

- EEE parts engineering tasks include:
-     EEE Parts Control Plan, Parts Selection for Reliability and Tolerance to Radiation, Radiation and Burn-in Testing, Environmental Stress Screening, Parts Stress Analysis, Consideration of Part Grades, Derating, and GIDEP Searches.
- EEE Parts Assurance is the systematic review and evaluation of parts engineering tasks to determine their effectiveness and recommend changes for Reliability.
- “Effective Parts Engineering tasks” implies the real capability to reduce the probability of failure for EEE components and to assure that EEE components will meet or exceed reliability requirements.

## EEE Parts Assurance and Engineering works to Remove Infant Mortality, Reduce the Constant Failure Rate, and Minimize Performance Drift (Age) in EEE Parts. 

## Additional Objective of EEE Parts Assurance: 

**Design for Reliability and Radiation Up Front**** **
**Develop EEE parts and Mechanical Control Plan and Stick-to-it.**
**Select Radiation Tolerant EEE parts-> Must survive Ionizing Dose and Single Event Effects expected during the mission.**
**Select High Reliability (Low Risk) parts: both EEE and Mechanical.  Aim to Exceed Reliability Targets with a Margin.**
**Parts Stress Analysis: Verifies that the EEE parts selected for the circuit design are compatible with the circuit operation and can operate at the expected levels of voltage, current, and temperature.**
**Thermo-graphic mapping- Infrared pictures of actual circuit boards when energized to determine hot spots that could cause premature failure of EEE components.  **
**Radiation test to verify Electronics operates in expected radiation environment.**
**Make sure electronics is derated with respect to the circuit operation.**
**Thermal cycle and vibration test to verify survival in operating environment.**
**Burn-in testing to catch infant mortalities.**
**Verify design has protection from Corona and Arcing.****  **

## EEE Parts Assurance

**Design for**
**Reliability**
**at Component**
**Level**

**Selection**
**of Parts that **
**are Low Risk **
**and **
**Rad Tolerant**

**Review of **
**Derating Analysis**
** ****and Parts**
**List **

**Review  of**
**Thermal**
**Analysis &**
**Test Results**

**Development**
**of Radiation **
**Test & Evaluation**
**Plan**

**Development **
**Of **
**EEE and Mech.**
**Parts**
**Control Plan**

**Review  of**
**Radiation **
**Testing**

**Reliability **
**Analysis **
**Interface**

**Design for **
**Radiation **
**Work **
**Instruction**

**Feedback to**
**Design Team**

**GIDEP/ALERT**
**Searches**

-  and Engineering
- is conducted concurrently
- with Reliability Engineering

## How effective can it be to have a robust EEE Parts Engineering and Assurance Program?

**We learned from JPL’s Success with**
** ****Designing in Long Life **
**for Deep Space Mission Probes**

## JPL SECRETS TO LONG-LIFE

**Wide performance margins**
**Strong environmental test program- Thermal, Vibe, Radiation Tolerance.**
**All missions had some workmanship failures in test that would have been mission limiting. Understand Test Failures and fix them. This may mean re-selection of EEE parts.**
**Maximize operating hours -> 1500 hours burn-in prior to launch.**
**Use Block redundancy when needed**
**Grade B parts minimum. (note: Saves DPA money) **
**Software design flexibility**

***JPL INTERPLANETARY MISSION EXPERIENCE***

**Courtesy T. Gavin, JPL**

## System Safety Quality & Reliability Division (SSQRD) Open House 

**System Safety**
- William Schoren
- SSQRD Flight Safety Technical Lead
- William.R.Schoren@nasa.gov
- x-2356

## System Safety, Quality & Reliability Division (SSQRD) Open House 

**Agenda**
- Introduction
- Why Do System Safety?
- What Is System Safety?
- Definitions
- Hazard Analysis
- Defining the System
- Identifying Hazards
- Assessing Hazards
- Eliminating/Mitigating Hazards
- System Safety Engineering at SSQRD

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Introduction**
**Why Do System Safety?**
- Cost of Accident (Mishap) substantially exceeds the cost of accident prevention
- Human Cost – life/injury
- Fiscal Cost – Hardware damage/repair/schedule delays
- Political/Public Support/Reputation

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Introduction (cont.)**
**What is System Safety?**
- The systematic application of engineering and management principles, criteria, and techniques to achieve acceptable mishap risk, within the constraints of operational effectiveness, time, and cost throughout all phases of the system life cycle.
- System safety efforts for a Program/Project are most effective if they start in the conceptual design phase and follow in step with the design development process.

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Introduction (cont.)**
**What is System Safety?**
- System Safety Engineering – An _engineering discipline_ that employs specialized professional knowledge and skills in applying scientific and engineering principles, criteria, and techniques to identify and eliminate or mitigate _hazards_, in order to reduce the associated _mishap_ risk.

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Definitions**
- Mishap – An unplanned event or series of events resulting in death, injury, occupational illness, damage to or loss of equipment or property, or damage to the environment.
- Hazard – Any real or potential condition that can result in injury, illness, or death to personnel, damage to or loss of a system, equipment or property, or damage to the environment.

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Hazard Analysis**
- Defining the System
- Identify Hazards
- Assessing Hazards
- Eliminating/Mitigating Hazards

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Defining the System**
- Hardware/Software/Operations Design
- System Environment – Structural/Thermal/Pressure
- Mission Phases
- System Environment may change
- Entire system life cycle must be considered
- Interfaces
- System being analyzed may be part of larger overall system

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Identifying Hazards**
- Energy Sources
- Fuels, batteries etc.
- Disruption of Essential Life Processes
- Asphyxiation, toxins etc.
- Historical Hazard Data/Lessons Learned
- Hazard Source/Target
- Shuttle/ISS Payload
- Payload is Hazard Source
- Crew, Shuttle/ISS, other Payloads are Targets
- “How can Hazard Source harm Hazard Targets”

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Assessing Hazards**
- Need method to classify/prioritize hazards and track progress on hazard mitigation
- Hazard Risk Matrix
- Hazard Severity (Consequence)
- Worst case outcome
- Hazard Probability (Likelihood)
- Severity and Probability are Independent
- Reducing probability does not reduce severity

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Assessing Hazards (cont.)**
- Hazard Risk Matrix

| Hazard
Severity | Hazard Probability |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  | Improbable | Remote | Occasional | Probable | Frequent |
| Catastrophic |  |  |  |  |  |
| Critical |  |  |  |  |  |
| Marginal |  |  |  |  |  |
| Negligible |  |  |  |  |  |

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Eliminating/Mitigating Hazards**
- Hazard Reduction Precedence
- Eliminate Hazard
- Toxicity Hazard – Replace Hypergolics with LOx/Methane in Propulsion System
- Incorporate Safety Devices
- Fluid Combustion Facility Racks have door interlocks to interrupt power if doors are open
- Provide Warning Devices
- Smoke detectors
- Develop Procedures and Training
- Building Evacuation Procedures

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Design Strategies for Hazard Mitigation**
- Failure (Fault) Tolerance – the ability of a system to sustain a failure without a mishap occurring.  Safety requirements often impose levels of failure tolerance (single, dual) depending on the hazard severity.  Need to insure levels of failure tolerance are truly independent.
- Design for Minimum Risk (Design with Margin) – subsystems/components that have design margins over the worst case expected environment.  Methods for determining and verifying these margins are well understood e.g. structures, pressure vessels/ components, electrical wire/circuit protection sizes.

## System Safety Quality & Reliability Division (SSQRD) Open House 

**System Safety Engineering at SSQRD**
- Primary work is with Spaceflight Projects with some work in Aeronautics/Aviation Safety
**In-Line System Safety Tasks**
- Developing Hazard Analyses
- Generating Safety Assessments/Safety Data Packages
- Advising Projects on Design Options
**Independent Assessment Safety Tasks**
- Review and comment on In-Line Safety Documentation

## Tentative System Safety Engineering Training

- Classroom Training
- NSTC 008 System Safety Workshop
- NSTC 020 Basic System Safety Practice
- NSTC 021 Advanced System Safety Practice
- NSTC 011 Payload Safety Review and Analysis
- NSTC 016 Payload Safety Review Process and Requirements
- On-Line Training
- SMA-066-01 System Safety (Solar)
- NSTS/ISS 13830C Payload Safety Review & Data Submittal Rqrts. (JSC)
- NSTS 1700.7B Safety Policy and Requirements (JSC)
- NSTS 1700.7B Safety Policy and Requirements ISS Addendum (JSC)
- NSTS 18798B Interpretation of NSTS/ISS Payload Safety Rqrts. (JSC)
- On-the-Job Training
- Mentoring and task ramp-up during FY06

## Workforce Transition Strategy and ImplementationPART II (Afternoon Session)

- SMA Open House
- December 12, 2005

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Quality Management**
- Bipin Patel and Tim Gaydos

## Quality Management

- Quality Management support to aeronautics, space research, and technology projects.
- The QD also provides Safety and Mission Assurance (SMA) support to GRC institutional activities. The key activities and processes for the QD are:
- Quality Engineering 	Quality Engineering is provided as an integral part of the
- 					SMA support to GRC aeronautics, spaceflight, and
- 					technology projects. The QD provides general quality
- 		engineering expertise such as contract review, specification
- 		development, requirement tailoring, and processing and
- 		fabrication guidance. Specific engineering expertise also
- 		exists in areas such as electronic component design and
- 		fabrication. The QD uses the majority of the office’s human 		resources by matrixing them, to aeronautics and 					spaceflight projects as Project Assurance Team leads or 			members

## Quality Management

- 		The QD provides support to Independent Assessment Center 	Director, Headquarters, or other Management Oversight efforts. It 	also supports GRC’s Internal Audit Activities Failure Analysis 	(Electronic and Mechanical). Expertise exists for developing 	failure analysis plans and for arranging to have the planned 	analysis activities performed.  Specific engineering expertise 	exists in specialized fields such as scanning electron microscopy 	analysis of both electronic and mechanical failures

## Quality Management

- 		The M&P provides guidance on materials selection and 	processing, including specialized processes such as heat  	treatment, welding, brazing, and non-destructive evaluation 	(NDE). It also provides guidance for fabrication activities,  	including specialized areas such as composite fabrication in 	Kevlar and graphite/graphite. M&P reviews and approves 	Materials Usage Agreements (MUAs), Materials Identification and 	Usage Lists (MIULs), and Materials Certification Letters as 	required in support of intercenter agreements with JSC, MSFC, 	and GSFC

## Quality Management

- 		The QD provides support to aeronautics and spaceflight projects 	by assisting in conducting audits of suppliers. The QD has 	expertise in various types of audits, such as pre-award surveys, 	periodic requirements compliance verification reviews, and 	special purpose audits
- Critical Test and Inspection Validation
- 	The QD assists spaceflight projects in verifying that product 	design requirements have been met. This is accomplished 	through activities such as conducting critical inspections of 	hardware, approving test plans, and witnessing acceptance tests

## Quality Management

- 		The QD routinely surveys the Center for adequate process 	management and product assurance controls. It also supports  	the BMS/ISO Project Office’s internal audits of GRC by supplying 	lead and internal auditors to  the BMS Project Office when 	requested
- Government-Industry Data Exchange Program (GIDEP)
- GIDEP is a NASA/DOD-sponsored activity that facilitates the voluntary exchange of technical data related to parts, components, and materials. The Center Director has delegated the responsibility to QD to administer this program at GRC and provide a GRC GIDEP representative
- Lessons Learned Information System (LLIS)
- 	The LLIS is a system that collects the lessons learned from a	almost 40 years in the aeronautics and space business

## Quality Management

- The QASAR Award recognizes individual government and contractor employees for significant quality improvements to products or services for NASA. SAAD, through the QD administers this program at GRC
- Corrective and Preventive Action Reporting System (CPARS)
- When requested by Projects, the QD utilizes the Center-wide CPARS system by inputting relevant information in support of microgravity and spaceflight Projects
- Maintenance of Quality Management Documents.
- The QD maintains the GRC documents for the GIDEP, LLIS, QASAR, and the intercenter agreements with JSC, MSFC, and GSFC. Intercenter agreements with JSC, MSFC, and GSFC are maintained by the Chief of the Quality Management Office. Details on Quality Records and Forms for GIDEP, LLIS, and QASAR are stated in the individual Office Procedures for each of these topics

## Tentative Quality Assurance Training

- Classroom Training
- Workmanship Standards - JSC/GSFC/JPL
- QA Integration/Testing  - GSFC
- Online Training
- Quality Engineering  - GSFC Video
- Inspection Planning  - Solar
- EEE Parts/ESD
- Process Control
- GIDEP Participation
- On Job Training
- FAR Part 46  Quality Assurance Documents
- Workmanship Documents (JSC,JPL,GSFC, ICPL etc.)
- JSC Quality Plans (Project Specific)
- JSC ISO System
- Military Inspection Documents

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Software Assurance**
- Cynthia Calhoun

## NASA Software Assurance Challenges

- Performance Based Contracting
**Lack of software assurance requirements in proposals**
**Limited insight into interim products**
- Increased code size and complexity
- Resources
**Constrained budgets**
**Compressed schedules ****	**
**Limited availability of trained personnel**
- Increased visibility
**Loss of mission**
**Loss of credibility**
- Achieving “high quality” software
**	****(and knowing what that means!)**

***MARS***

**Notes:**
Before we review Software Assurance, let’s take a look at some of the challenges we’re facing in the software arena.
NASA has undergone change and numerous challenges in the last decade, including the concept and implementation of Performance Based Contracting and its’ impact to our level of insight into software development.
Software code has increased in both size and complexity as NASA continues to infuse software in autonomous operations and other critical system applications.
Limitations on budget and schedule are certainly not new to NASA and we continue to confront limited resources when it comes to acquiring and retaining trained software personnel.
Increased visibility within the public eye, as well as on Capitol Hill, has also intensified scrutiny and criticism beyond our comfort level.  Our highly  publicized failures and internal/external investigations have precipitated both a loss of credibility and a lack of confidence.  
Finally, achieving “high quality” software is subjective and widely misunderstood across the Agency.  We as “One NASA” need to know what it means to be “high quality” and how to achieve this.

## Agency Response and Actions...

- Renewed emphasis on process improvement initiatives (e.g., CMM, CMMI)
- Improved and accepted tools and techniques
- Early Software Assurance involvement during Software Acquisition activities
- Redefinition of NASA Software Assurance and updates to standard
- Approved NASA Software Engineering Requirements

***NASA Software Assurance Standard was updated in July 2004!***

**Notes:**
Let’s look at how NASA has responded to recent software challenges:  

As an Agency, NASA has recognized that successful software products have mature and effective development and maintenance processes to support them.  Process improvement initiatives are now commonplace within NASA, along with tools,  techniques, and methodologies (e.g., CMM, CMMI, and Six Sigma) to assist practitioners.

Experience has taught us that early involvement in the development life cycle increases our chances of success.  In other words, applying software assurance resources and defining requirements early in the process leads to a common framework of requirements, deliverables, and expectations.

NASA policies, standards, and requirements for software have been revisited and updated to reflect changes in our Agency and the software environment.  NASA approved NASA NPR 7150.2, NASA Software Engineering Requirements and in July 2004, NASA approved extensive updates to the Software Assurance Standard.  An overview of this Standard, its requirements and benefits, are provided next.

## What is Software Assurance?

- A planned and systematic set of activities that ensures conformance of software life cycle processes and products conform to requirements, standards, and procedures.
- “Processes” include all of the activities involved in managing, designing, developing, enhancing, and maintaining software.
- “Products” include the software, associated data, its documentation, and all supporting and reporting materials.
- Focuses on problem prevention though early detection.
- Performed by organizational entities that are separate from the ones doing engineering activities.
- Performed at project conception, through operations and maintenance, until retirement.

## Value of Software Assurance

- Earlier detection is cheaper to fix with lesser impact to schedules
- Increases chances of delivering full functionality within cost and schedule
- Safety issues are identified and addressed early as part of overall system safety
- Software assessments can be used to determine milestone completion
- Metrics provide indicators of the quality and maturity of the software/system

- Relative Cost of Finding, Fixing Errors
- Adapted from Software V&V, Steven Rakitin

- 0

- 10

- 20

- 30

- 40

- 50

- 60

- ...

- 100

- Requirements

- Design

- Code

- Test

- Maintenance

**Phase Error Found & Fixed**

** ****Cost to Fix**

- $1

- $5

- $20

- $50

- $100+

**Notes:**
Let’s take a closer look at some of the potential benefits. 
It’s a well known fact that early error detection is cheaper to fix than later in the development life cycle.  And it can also have a lesser impact to overall development schedules.  That’s not to imply that there’s no disruption in schedule, but you may have more schedule slack (or time to recover) earlier in the development life cycle, rather than during acceptance testing.

With full life cycle coverage, you also increase the probability of delivering full functionality within cost and schedule.  Continuous support in all of the SA disciplines will provide a consistent level of insight into processes and products.  There should be no unknowns or hidden issues.

Given that software safety is part of system safety, safety issues are also identified early and tracked (with time to mitigate any risks or safety hazards).

By conducting software assessments, audits, inspections, etc., you have more information from which to predict and assess milestone completion.  The more information, the better!

And metrics provide another avenue for assessing quality and software maturity.

## Who are the Practitioners?

- Software Assurance practitioners INCLUDE a wide range of personnel, employed throughout the software development life cycle

- Software Engineers

- Safety Engineers

- Systems Engineers

- IV&V Personnel

- Software Managers

***Software Assurance personnel aren’t just those Software Quality folks!!***

- Software Quality Personnel

**Notes:**
So who are the Software Assurance practitioners?

One common misconception is that Software Assurance personnel are just those Software Quality folks!  

In fact, Software Assurance personnel include most everyone in this class today.  They are the Software Managers, the Safety, Software, and Systems Engineers, as well as the Software Quality Engineers and IV&V personnel from the NASA West Virginia Facility.  They include Software Reliability and V&V personnel and perhaps other folks on your development team.  A Center’s organizational structure and governing documents for each program/project further define who’s who is software assurance.

We all have important roles and responsibilities and it’s imperative that we understand the players and their contributions to Software Assurance.  Only through a common framework and knowledge base can we ensure consistency in implementation and performance and realize the benefits of Software Assurance.   

## Software Assurance Disciplines

**SQ**

**V&V**

**Software**
**Safety**

**Software**
**Reliability**

**IV&V**

**Software Assurance is an umbrella risk identification and mitigation strategy for safety and mission assurance of all NASA’s software**

- SQ = Software Quality
- V&V = Verification and Validation
- IV&V = Independent Verification and Validation

**Notes:**
So what is Software Assurance?  Per the NASA Software Assurance Standard, it’s defined as an umbrella risk identification and mitigation strategy for safety and mission assurance of all NASA’s software.

The Software Assurance process is the planned and systematic set of activities that ensure conformance of software life cycle processes and products to requirements, standards, and procedures.  It applies to software developed or acquired for NASA, either internally or externally, including the incorporation of open source, COTS, GOTS, or MOTS into NASA systems.  

For NASA, this includes the disciplines of Software Quality (with the functions of Software Quality Engineering, Software Quality Assurance, and Software Quality Control), Software Safety, Software Reliability, Software Verification and Validation, and IV&V.  Each discipline brings its own perspective to the process; the collective effect of all these efforts provides assurance of mission safety, reliability, and quality.

## Software Quality

- Software Quality assures that quality is built into the software through the functions of
**Software quality assurance**
**Software quality engineering**
**Software quality control**
- Ensures conformance of software life cycle processes and products to requirements, standards, and procedures
- Performs process and product activities throughout the life cycle to provide objective insight into the maturity and quality of the software processes and products
- Promotes continuous process improvement

**Notes:**
Let’s begin with a discussion of Software Quality (perhaps the best known of the 5 software assurance disciplines).  

Software Quality is the planned and systematic set of activities to ensure quality is built into the software.  This is accomplished through the functions of Software quality assurance, Software quality engineering, and Software quality control.  

The software quality discipline consists of product assurance and process assurance activities with the emphasis on conformance to requirements, standards, and procedures.  Activities are conducted throughout the life cycle to provide objective insight into the maturity and quality of the software processes and procedures.  

Strengths and weaknesses identified during assessments can then be used to promote continuous process improvement on current and future software systems.

## Sample Process and Product Activities

- All required plans are developed in accordance with specified requirements, standards, or procedures
- All software requirements are documented and traceable from system requirements to design, code, and test
- Software development records are maintained and up-to-date
- Configuration baselines are managed, maintained, and accurate
- Software quality metrics are in place and used to manage the software development effort

- All plans (e.g., Configuration Management Plan, Software Management Plan) and procedures are implemented according to specified standards and procedures.
- Engineering peer reviews and management reviews are conducted and action items are tracked to closure
- Tests are planned, documented, and conducted using approved test procedures and tools
- Project risks are documented and addressed in accordance with the risk management plan

**Process**

**Product**

*SQ identifies strengths, weaknesses, and areas for improvement!*

**Notes:**
The following examples are provided to enhance your understanding of the various types of process and product activities that may be conducted.

PRESENTER:  Choose several bullets to highlight.  This chart is not intended to be exhaustive.  No need to spend a lot of time on this chart.

## Software Safety

**Software Safety is a systematic approach to identifying, analyzing, tracking, mitigating and controlling software hazards and hazardous functions (data and commands) to ensure safer software operation within a system.**** **

***Software Safety entails...***

- Ensuring that software safety requirements are clearly identified, documented, traced and controlled throughout the software lifecycle

- Analysis of the consistency, completeness, correctness and testability of software safety requirements

- Testing of software safety critical components on actual hardware to ensure that the safety requirements were sufficiently implemented and that applicable controls are in place to verify all safety conditions

- Continuous analysis of proposed changes on system safety** **

***Software Safety is a function of System Safety!***

**Notes:**
Software Safety has grown in importance as more and more NASA systems have critical functions either controlled by software or adversely impacted by a software failure.

The discipline of Software Safety functions as part of the overall System Safety program and coordination is essential  between system safety personnel and software safety personnel.  At some Centers, these personnel may be one in the same.  

It’s also important to note that Software Safety assessments begin early in the development life cycle and continue as proposed changes are made.

PRESENTER:  Be prepared to discuss/provide examples of what is considered software safety-critical.  For example, flight software that detects and reports a specific hazardous state (e.g., instrument overheating) and attempts to take corrective action...

## What is Safety-Critical Software?

- Reside in a safety-critical system (as determined by a hazard analysis) AND at least one of the following:
**Cause or contribute to a hazard**
**Provide control or mitigation for hazards**
**Control safety-critical functions**
**Process safety-critical commands or data**
**Detect and report, or take corrective action, if the system reaches a specific hazardous state**
**Mitigate damage if a hazard occurs**
**Reside on the same system (processor) as safety-critical software**

***Does the software meet any of the following criteria:***

**Notes:**
Well, how do I know if my software is safety critical??

The determination of safety criticality is based on NASA’s Software Safety Litmus Test which was derived from the definition of Hazard Severity in NPG 8715.3, the NASA Safety Manual.

To determine if the software is safety critical, and to what extent, review the following criteria...

## What is Safety-Critical Software? – cont.

- Process data or analyze trends that lead directly to safety decisions (e.g., determining when to turn power off to a wind tunnel to prevent system destruction)
- Provide full or partial verification and validation of safety-critical systems, including hardware and software subsystems

- If the software is determined to be safety-critical, then the project must adhere to the NASA-STD-8719.13, NASA Software Safety Standard

**Notes:**
If the software meets any of the 3 criteria, the software is considered safety-critical.  Refer to the NASA Software Safety Standard and Guidebook for more information.

## Software Reliability

- Software Reliability incorporates reliability into the products starting with requirements, and measures the reliability of the products produced by each process of the life cycle.

**Defines the requirements for software controlled system fault/failure detection, isolation, and recovery**
**Reviews the software development processes and products for software error prevention and/or reduced functionality states**
**Defines the process for measuring and analyzing defects and defines/derives the reliability and maintainability factors**.

**Notes:**
Software Reliability is a discipline exercised by Systems Engineering, Software Engineering, and Software Quality with the emphasis on incorporating and measuring reliability in the products produced by each process of the life cycle.  

To assure reliability...

For measures, including those for software reliability modeling, reference the IEEE Standard Dictionary of Measures to Produce Reliable Software (IEEE Std. 982.1)

## Verification and Validation (V&V)

- Software Verification and Validation
**Ensures that software being developed or maintained satisfies functional and performance requirements**
**Ensures that each phase of the development process yields the right products**
- Every participant in the software life cycle process plays a role in some aspect of V&V!
*V&V activities include, but are not limited to:*
**Analysis of system and software requirements**
**Engineering peer reviews (e.g., code walkthroughs)**
**Test planning and test execution**
**Audits/assessments (e.g., baseline management)**

**Notes:**
Software Verification and Validation is another discipline that engages participants across a variety of functional areas and/or groups.  

First and foremost, verification ensures that “you built it right”, while validation ensures that “you built the right thing”.  

When regard  to software, this means ensuring that the software satisfies functional, performance, and other requirements and that each phase of the development process yields the right product.

Software V&V entails a variety of activities, including engineering peer reviews, formal reviews, analyses of requirements and design, test planning, test execution, and formal audits and assessments.

PRESENTER:  See the V&V Activity Matrix in the Backup Charts!

## Independent V&V – IV&V

- IV&V is Verification and Validation performed by an organization that is technically, managerially, and financially independent of the development organization
- IV&V focuses on mission critical software, provides additional reviews and analyses, and provides in-depth evaluations of life cycle products that have the highest level of risk

**Validation of design to meet system needs/requirements**
**Code analysis of mission-critical software components**
**Design analysis of selected critical algorithms**

***Examples:***

**Notes:**
This brings us to the discipline of Independent Verification and Validation (better known as IV&V).  

Within NASA, IV&V is performed by an organization that is technically, managerially, and financially independent of the development organization.  For NASA, IV&V is performed and managed by the NASA IV&V Facility.  

IV&V focuses on safety and mission critical software by conducting additional reviews and in-depth evaluations of the life cycle products that carry the highest level of risk.  

All software projects that are identified as safety-critical or Class A software will be candidates for IV&V support and funding via Headquarters.  Software projects that are neither Class A nor safety-critical may request Headquarters funding, but funding is reserved for special circumstances.  Software intense projects not selected as candidates for IV&V may negotiate with the IV&V Facility for direct support, but must fund the support themselves.

Examples of representative IV&V tasks or activities are as follows...

## Recommended Reference Documents

**GRC Software@Glenn -**** ****http://software.grc.nasa.gov/**

**Notes:**
The following recommended reference documents are available via the Agency Software Assurance Web site or the NASA Technical Standards Web site.  

The NASA policies, directives, standards, and guidebooks provide a common framework for consistent practices across NASA programs.  Software Assurance practitioners are encouraged to reference these documents often throughout the entire software life cycle!

## Tentative Software Assurance Training

- Classroom Training
- System Requirements (4-days)
- Software Testing and Inspection Methods (4-days)
- Intro to CMMI (3-days)
- GRC Software Process Rollout Training (2-hours)
- On-Line Training
- Software Assurance
- System Safety
- Software System Safety
- On-the-Job Training
- Reading Assignments: GRC Software Assurance Process Definition, GRC ISO Software Development and Software Assurance procedures and checklists; NASA Software Engineering and Software Assurance Requirements and Standards.
- Review Software Assurance Deliverables and Products: Software Assurance Plan, RFA, PRACA, Review Comment Form, Audit Report
- Review Project Documentation: Development Plans, Requirements, Design, Test Plans, Maintenance Plan
- Shadowing
- Assign Software Assurance tasks

## System Safety Quality & Reliability Division (SSQRD) Open House 

**Risk Management**
- Cynthia Calhoun

## Risk Management

- SSQRD is responsible for providing ongoing risk management consultation, facilitation, and training to GRC Program and Project organizations.

*** ******Identify*** - Search for and locate risks before they become problems
*** ******Analyze*** - Convert risk data into useable information for determining priorities and making decisions
*** ******Plan*** - Translate risk information into planning decisions and mitigating actions (both present and future), and implement those actions
*** ******Track*** - Monitor risk indicators and mitigation actions
*** ******Control*** - Correct risk mitigation plan deviations and decide on future actions
*** ******Communicate and Document*** - Provide information to project on risk activities and current/future risks

_***Continuous Risk Management (CRM) Paradigm***_

## Continuous Risk Management Approach

- SSQRD risk facilitators ensure Programs/Projects utilize risk-based decision-making to continuously manage the acquisition, safety, technical, and programmatic risks by assuring they:
**Comply with NPR 8000.4 and NPR 7120.5C.**
**Develop Risk Management Plan during the formulation phase.**
**Continually assess risks and develop effective risk mitigation strategies that get integrated into the project to buy down risk.**
**Build Contingency Plans into project budgets and schedules where risks are high.**
**Require data from risk assessments, augmented with earned value reporting, for monthly management reviews.**

## Facilitator Requirements

- SSQRD personnel who complete the CRM Train-the-Trainer course serve as a Facilitator and:
- Conduct Risks Identification Workshops with their projects.
- Assist projects with developing and updating their Risk Management Plans.
- Review the project’s risk mitigations to ensure the mitigation will actually reduce the likelihood and consequence of occurring.
- All SSQRD personnel report risk management efforts being performed in the projects they support.
- Provide significant input in the “RM Facilitator Reporting” spreadsheet located on PBMA CRM Working Group under the ‘Discipline Mtgs’ folder.
- Facilitators ensure projects use the 5 by 5 risk matrix appropriately (see 5x5 risk matrix on next slide).

## 5 x 5 Risk Matrix

**5 x 5 Risk Matrix**

## CRM Trainer Certification Requirements

- Must be appointed by the Center’s S&MA Director.
- Complete the “Practitioners of Risk Management” course.
- Be observed and certified by Headquarters in teaching the “Foundations of CRM” course.

## Tentative Risk Management Training

- Classroom Training
- NASA Risk Management Practitioner’s Workshop (5-days)
- Risk Identification Workshop (4-hours)
- On-Line Training
- Risk Management Plan Workshop
- Risk As A Resource
- On-the-Job Training
- Reading Assignments: GRC ISO Risk Management Procedure, Tailoring Risk Management for Projects, Facilitated Risk Management Procedure, NASA Risk Management Requirements, NASA Program/Project Management Requirements
- Review Project Documentation: Risk Management Plan and Process, Risks List and Mitigations
- Shadowing
- Assign Risk Facilitator tasks