# Presentation: 712515

## Defect Detection and Prevention (DDP): A Tool for Life Cycle Risk ManagementExplanations, Demonstrations and Applications

**Steve Cornford, Ph. D.**
**Strategic Systems Technology Program Office/**
**Safety and Mission Assurance Directorate**
**Jet Propulsion Laboratory, **
**California Institute of Technology**
**Phone:(818)354-1701, Email: steven.cornford@jpl.nasa.gov**

**GSFC**
**January 30, 2001**

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## BACKGROUND

- NASA’s missions are challenging and “pushing the envelope”
- They may contain significant amounts of advanced technologies or existing technologies in advanced applications
- Risk Management
- FBC + S! (Faster, Better, Cheaper and Safer)
- “Risk as a resource” - Dr. Michael Greenfield, Code Q
- NASA 7120.5, SMO, IPAO
- Team environment
- Fast moving, implementation teams - need to integrate more extensive modeling/simulation results, need more accurate answers
- Faster moving, formulation teams - need to integrate intuition and rapidly evolving designs, need 80% answer quickly
- Various resources are available
- Advanced Design Environments/Tools
- PRA, FMECA, DOORS, etc.
**Challenge: Get the job done effectively and efficiently.  We need a process/tool to enable life-cycle risk management.**

## Approach
Code Q has funded the development of “tools which address residual risk as a function of various risk control options.  Options exist at the planned activity level and in the degree to which potential failure modes are addressed.”
DDP tool has module containing data from ongoing Code Q Failure Detection and Prevention Program (joint GRC/GSFC/JPL RTOP)
DDP Version 2.0 VB has been released, Version 2.5 VB/1.5 Java due in early summer
Have formed partnerships/pilot studies with technologists and mission designers within NASA and JPL, other teaming outside NASA being explored.

- Code Q has funded the development of “tools which address residual risk as a function of various risk control options.  Options exist at the planned activity level and in the degree to which potential failure modes are addressed.”
- DDP tool has module containing data from ongoing Code Q Failure Detection and Prevention Program (joint GRC/GSFC/JPL RTOP)
- DDP Version 2.0 VB has been released, Version 2.5 VB/1.5 Java due in early summer
- Have formed partnerships/pilot studies with technologists and mission designers within NASA and JPL, other teaming outside NASA being explored.

**Parameters in the Problem**

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## “Screening Out” the Defects

**“****Screening Out” the Defects**

**Notes: **
**1) Each box is a collection of PACTs**
**2)  Dotted lines represent “escapes” - Undetected or un-prevented failure modes**
**3)  Illustrative diagram only - nothing is “to scale”**

***PACT***s - Are everything that could be done (e.g. “toolbox” of prevention/detection options)
**P**reventative measures (Redundancy, Design Rules, Materials Selection, Software Architecture, etc.)
**A**nalyses (Reliability (Fault Tree Analyses, Failure Mode and Effects Criticality Analysis (FMECA), Worst Case Analysis), Fatigue, Structural, Performance, Electrical SPICE models, etc.)
- process **C**ontrols (Inspections, Materials purity, QML vendors, Documentation, etc.)
**T**ests (Environmental, Life, Simulations, Performance, etc.)
- Failure Modes (**FM**s)/Defects/Risk Elements
- Failure is used in its broadest sense:  Failure to meet goals/requirements
- “Hard” - Cracks, Explosions, Open Circuits, etc.; “Soft” - Resets, Performance Degradations, etc.

**Notes:**
Have a simple and clear example
  EOL Simulation, Parts Quality and radiation tolerance knowledge and WCA analysis
  Kite??

Mention the 3 things to avoid
   Missed
  Only 1 way and didn’t know it
  Redundant

## Simplified DDP Summary

**Simplified DDP Summary**

- DDP utilizes two matrices: the Requirements matrix (***R***) and the Effectiveness matrix (***E***)

**Impact of a given FM on a particular requirement**

***R***

- Failure Modes/Risk Elements

- Mission
- Requirements

- 

- 

## Overview of the DDP process

- What does the DDP process/tool do?
- Allows users to perform a variety of risk management activities
- What inputs does the DDP process/tool require?
- Information can be pre-existing
- FDPP PACT Effectiveness ‘pre-canned’ information or previous DDP evaluations
- Existing schedules, preliminary risk elements and mitigation options
- Requirements trees, fault trees, etc. at various levels of importability
- Information can be entered prior to sessions or in ‘real time’
- Project Requirements and their relative weights
- Article Trees (breakdown of system into subsystems into assemblies, etc.)
- Failure Modes and Risk Elements (from high-level categories to low-level mechanisms)
- PACT options (from high-level types to specific activities)
- What are the outputs of the DDP process/tool?
**Identify areas requiring additional work or more detailed analysis**
**Driving requirements** (requirements which are producing the most risk)
**Risk Balance** (Can sort by risk type, articles affected, etc.)
- Under-covered risk elements (‘tall poles’)
- Over-covered risk elements (move the resources elsewhere)
**PACT selection** (Can sort by risk type addressed, articles requiring PACTs, etc.)
- PACTs agreed upon to achieve desired risk balance (incl. Costs)
- Value of remaining un-selected PACTs

## Using DDP to Tailor and Optimize

**Using DDP to Tailor and Optimize**

- Risk Balance
- The residual risk is the ‘expected value’ of the failure mode, i.e, the product of it’s likelihood, severity and chance of escaping
- Measures product of how much we care and chance we will miss it
- Risk balancing trades off PACT options against residual risks
- Versus constraints (mass, power, $, etc.)
- Can shift priorities
- Select different PACT combinations
- Capture design and PACT decisions
- Modified/refined with project life cycle

**Risk Balance (before)**

- Failure Modes (same
-  sequence as FM Impact pareto)

**Relative Residual Risk**

**Risk Balance (after)**

- Failure Modes (same
-  sequence as FM Impact pareto)

**Relative Residual Risk**

***Residual Risk****** = r = i x e =***

***The extent of it’s impact x How likely it will occur***

- For each failure mode:

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## DDP integrates intuitive and analytical approaches

**ANALYTICAL**

**100**

**N**

**F**

**HORIZON**

**DESIGN**
**CREDIBILITY**

**INTUITIVE**

## DDP usage in the NASA Mission timeline

- The concept of “What are we trying to accomplish, what could get in our way and what can we do about it” is very broad
- Level of fidelity grows with project/program design maturity
- Can be applied in a number of places in the NASA Mission timeline
- Have done a wide variety of “alpha”, “beta” and more, pilot applications
- Real power is in getting the right team together and quickly, systematically integrating quantitative and qualitative information

- Advanced
- Mission
- Planning

- Specific
- Mission
- Planning

- Focused Technology
- Programs
- (e.g. NMP, X2000)

- Technology
- Development
- (e.g. NASA 632 Program)

- Mission/Project
- Design and
- Implementation

**UNDER DEVELOPMENT**

**IN “BETA”**

**PAST “BETA”**

**IN “BETA”**

**PAST “BETA”/IN “BETA”**

## Applications of DDP to date

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## DDP integrates intuitive and analytical approaches

**Application to Advanced Technology “Roadmapping”**

**ANALYTICAL**

**100**

**N**

**F**

**HORIZON**

**DESIGN**
**CREDIBILITY**

**INTUITIVE**

## Roadmap for DDP sessions

- Perform over 4 (or 3) half-days

## DDP applied to technologies(Technology Infusion and Maturity Assessment (TIMA))

- Hybrid Imaging Technology (HIT) - Cost: 10k$
- Saved $600k radiation fabrication effort and $300k ground test program
- HIT product delivery to customer in ‘00 versus ‘02-’03
- Task alignment with flight implementation expertise
- Compact Holographic Data Storage (CHDS) - Cost: 12k$
- Focused on SNR and BER issues (major show stoppers) **not **memory volume
- Increased focus on  breadboard development (migrate technology off the optical bench)
- Identified required analysis and proof tests
- Alignment with other ongoing R&D (NEPP) and Sandia
- Variety of Others
- National Instruments’ LabView software - Cost: about 10k$
- Active Pixel Sensor (APS) program - Cost: about 10k$
- Micro-gyro program - Cost: 9k$
- ITP/SIM - Cost: varied
- Commercial Industry (disk drives, avionics)

## Successes on technology evaluations

- Have resulted in an “institutionalization” of the process at JPL within the technology community
- Will continue applying to “Proof-of-concept” and earlier technologies
- Will begin to quantitatively validate the process in the lab
- Will begin applying to more far-horizon mission studies
- I have a joint appointment between the Safety and Mission Assurance and Technology Applications Directorates at JPL to help make this happen

## Technology Infusion Process(JPL process in draft)

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## DDP integrates intuitive and analytical approaches

**Application to Mission and System Design**

**ANALYTICAL**

**100**

**N**

**F**

**HORIZON**

**DESIGN**
**CREDIBILITY**

**INTUITIVE**

## Information and Influence by Project Phase (Formulation)

- Science Goals
- Project Teaming
- Subsystem Types and Requirements
- Launch Vehicle
- Preliminary Trajectory
- Technology Requirements
- Risk Posture
- Schedule
- Etc.

**Architectural Options**
**Mission Design Options**
**System Design Options**
- Heritage Applicability
- Environmental Concerns
- Verification and Validation Approaches
- Redundancy and SPF Policies
**Schedule and Cost feasibility**
**Risk Management Policy**
- Margin Philosophy
- Etc.

- FDPP Guidebook
- -Introduction
- -Risk as a Resource
- -Anomaly Trends
- RBP Tool
- DDP Tool (higher level evaluations)

- Medium-level Information

- Medium-level questions/answers

- FDPP Guidebook
- DDP Tool

- Detailed-level Information

- Detailed-level Information

_**Project Phase**_

_**Available Information**_

_**Questions to be answered**_

_**FDPP Applicable Products**_

**Formulation**

**Implementation:**
**Prelim Design**

**Implementation:**
**Detailed **
**Design/ATLO**

- FDPP Guidebook
- DDP Tool

## SUMMARY OF RECENT APPLICATIONTO ARCHITECTURAL ASSESSMENT

- Primary Areas of Assessment
- **Sensors
- **Heat Rejection
- *Avionics Architecture
- **Signal Processing
- *Processor
- *Upset Immunity
- *Thermal Control
- **FPGAs
- Structure
- **Operational Modes
- *Materials and Parts
- Software
- Results of three 1/2 day sessions (Total cost: <14k$):
- Savings of at least 2.5 M$, 154 W (and reduced radiators), and 22 kg.
- Project action items:
- Ripple effects not entirely included (will make it better)
- Some decisions require further analysis (potential savings of 5-8M$, etc.)

- ** =  Significant pay-off
- * = Moderate pay-off

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## DDP integrates intuitive and analytical approaches

**Application to Project Implementation **

**ANALYTICAL**

**100**

**N**

**F**

**HORIZON**

**DESIGN**
**CREDIBILITY**

**INTUITIVE**

## Information and Influence by Project Phase (Preliminary Design)

- High-level information

- High-level questions/answers

- FDPP Guidebook
- RBP Tool
- DDP Tool

- Unit-level requirements
- Environmental exposures and estimates
- Functional Block Diagrams
- Engineering Resource     Allocations
- Parts/Material/Process Candidates
- Heritage Reviews
- Etc.

- Long-lead item requirements
- Environmental Levels
- Reliability Estimates
- Verification and Validation Plans
- Part-type/material/process selection
- Mission Assurance Support Distribution
- Developmental and Engineering Model scope
- Detailed cost profiles/reserves
- Detailed schedules/reserves
- Current risk landscape
- Margin approach
- Etc.

- FDPP Guidebook
- - Failure Mode Types
- -PACT Effectiveness Evaluations
- -PACT Tailoring
- DDP Tool (medium level evaluations)

- Low-level information

- Low-level questions/answers

- FDPP Guidebook
- DDP Tool (lower level evaluations)

_**Project Phase**_

_**Available Information**_

_**Questions to be answered**_

_**FDPP Applicable Products**_

**Formulation**

**Implementation:**
**Prelim Design**

**Implementation:**
**Detailed **
**Design/ATLO**

## Information and Influence by Project Phase (Detailed Design/ATLO)

- Medium-level information

- Medium-level questions/answers

- FDPP Guidebook
- DDP Tool (medium level evaluations)

- Detailed Functional Requirements
- Circuit Diagrams and Detailed Drawings
- Part/Material/Process selections
- Layouts and CAD models
- Analyses and Evaluation Results
- Developmental Test Results
- Etc.

- Test Levels and other details
- Analysis Applicability
- Acceptance criteria
- Rework/retest decisions
- Anomaly resolution and close-out
- Specific risk evaluations
- Inspections
- Management processes
- Margin status/reserve
- Other project implementation details

- FDPP Guidebook
- - Failure Mechanism Information
- -PACT Effectiveness Evaluations
- -PACT Tailoring
- DDP Tool (lower level evaluations)

_**Project Phase**_

_**Available Information**_

_**Questions to be answered**_

_**FDPP Applicable Products**_

**Implementation:**
**Prelim Design**

**Implementation:**
**Detailed **
**Design/ATLO**

- High-level information

- High-level questions/answers

- FDPP Guidebook
- RBP Tool
- DDP Tool

**Formulation**

## DDP Implementation in the Project Implementation phase

- Have performed at all levels of assembly
- System, sub-system, assembly, sub-assembly, device, die
- Have performed on a variety of subsets
- Specific “root causes” (FMECA-type)
- Various risk element types (FTA-type)
- Specific exposure environments
- Have FY01-03 budget to begin piloting several “cradle-to-grave” implementations on NASA flight projects
- IPAO is beta-testing DDP in upcoming assessment of JPL flight project
- A number of project options exist
- Various characteristics
- Various design maturity levels

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## DDP Process Implementation

- Initial brainstorming
- Understand the technology, architecture, mission, etc.
- Requires ‘critical mass’ of relevant expertise
- Use tool in ‘Design Center mode’ - real or virtual
- Use disagreements to guide the depth of evaluation
- Go into detail required to ensure adequacy of the evaluation
- Take from religious discussions into engineering discussions
- Converge on baseline
- Identify areas which could still benefit from additional information
- Evaluate resource costs of baseline PACTs and select baseline
- Identify ‘tall pole’ residual risks (Significant Risk Lists)
- Iterate with project life cycle
- The fidelity evolves with the project life cycle
- Incorporate changes as they occur
- Make real-time adjustments in PACT
- 	implementation

**Notes:**
Initial brainstorming
Understand the technology, architecture, mission, etc.
Requires ‘critical mass’ of relevant expertise
E.g. Optical, thermal, packaging, mechanical, electrical, software, and devices
Generate FMs, PACTs, impacts, effectiveness values
Identify areas requiring additional information and go get it
Use tool in ‘Design Center mode’ - real or virtual
Project onto screen and make real-time inputs and changes 
Have a facilitator to guide the brainstorming process using checklists as reminders
Use disagreements to guide the depth of evaluation (take from religious discussions into engineering discussions)
Converge on baseline
Identify areas which could still benefit from additional information (research, library, test results, etc.)
Evaluate resource costs of baseline PACTs and select baseline
Identify ‘tall pole’ residual risks (Significant Risk Lists)
Iterate with project life cycle
Can be inputs by individuals and use ‘merge tool’
Can do at both high and low-levels: My example today is a specific technology

## Flow chart for DDP implementation

- Significant
- System Impacts?
- OR
- Major milestone?

## DDP Process Summary

- Facilitator/
- Integrator

- Tracking

- Iteration
- with project
- design evolution

- Available information:
- - Guidebook
- - Project R&D
- - Other RTOPs

**DDP Process Summary**

- Organize Session

## Detailed DDP Summary

**Detailed DDP Summary**

**Note**: Including requirement criticalities, ***C, ***and FM
- likelihood, ***L, ***yields weighted Requirements Matrix:  ***R’***=[ ***C*** ]***R***[*** L ***]

***Residual Risk****** = r = i x e =***

- * PACTs=**P**reventative measures, **A**nalyses, process **C**ontrols and **T**ests
- Note:       is the product symbol (a1*a2*...),     is the summation symbol (a1+a2+...)

***Extent of it’s impact x Probability it will still occur ***

- Product of elements within each Column
- yields ***e***, the PACT* coverage for each
- failure mode  (“Escape” chance)

***E***

- Failure Modes

- PACTs*

- Product of elements
- within a Row  yields
***f***, figure of merit
- for each PACT*

- Each column sum yields ***i***, the extent
- to which each FM impacts success

- Sum on each Row yields ***d***, the extent to which each mission requirement is impacted by the FMs

***R***

- Failure Modes

- Mission
- Requirements

-  

- 

- 

- 

-  

-  

- For each failure mode:

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## Proposed process for DDP implementation by IPAO

- Could help IPAO personnel incorporate risk into their assessments
- Could help IPAO assessments remain independent but operate from a position of ‘being up to speed’
- We are trying this out on a JPL project in the near future
- Notes: If project already using DDP, box at upper left may just be a walk-through of their existing information

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## DDP integrates intuitive and analytical approaches

**Application to Technology Portfolio Development**

**ANALYTICAL**

**100**

**N**

**F**

**HORIZON**

**DESIGN**
**CREDIBILITY**

**INTUITIVE**

**“****Wild-eyed” mission concepts**
**Almost exclusively Engineering Judgement**
**(e.g. future directions of biological computing, avionics packaging)**

## High-level RxFM matrix

## High-level investment decision

## Optimizing the high-level decision

- No overlap

- Minimal Risk

## Refined RxFM matrix

## Deeper penetration provides additional insight

## AGENDA

- BACKGROUND
- INTRODUCTION TO THE DDP PROCESS
- APPLICABILITY OF THE DDP PROCESS
- TOOL DEMONSTRATION
- APPLICATION TO:
- ADVANCED TECHNOLOGY ROADMAPPING
- MISSION AND SYSTEM DESIGN
- PROJECT IMPLEMENTATION/OPERATION
- IMPLEMENTING THE DDP PROCESS
- APPLICATION TO:
- INDEPENDENT PROGRAM ASSESSMENTS
- TECHNOLOGY TRADES/PORTFOLIOS
- SUMMARY AND CONCLUSIONS

**Notes:**
NEED TO UPDATE THIS CHART

## Using DDP to do Risk Management

- Risk Identification
- Initial Brainstorming
- Complete Evaluation
- Risk Analysis
- Initial Brainstorming
- Tall Pole Risks
- Driving Requirements
- Risk Planning
- PACT Options and PACT Adoption/Selection
- What-if scenarios
- Generate Baseline
- Risk Tracking
- Assess adequacy and implementation status of planned PACTs, Identify new risk elements
- Risk Control
- Refine Requirements, PACTs, and Risk Elements with project/program evolution

## Navigating the risk landscape

## Summary

- The DDP process has been described:
- A process for achieving clear and continuous insight into the evolving risk landscape
- Level of detail as required for application and project life cycle
- Usage ranges from mission theme planning, to project planning and implementation to detailed technology evaluations
- Fidelity grows with design maturity
- Provides a vehicle for staying abreast of risk balance as the implementation encounters (the inevitable) obstacles and surprises
- Incorporates range of information: from educated guesses to detailed probabilistic assessments
- Helps achieve ‘optimally balanced’ risk consistent with project resource constraints
- Utilizes an underlying database which keeps growing
- FMs, PACTs, and effectiveness: Part of ongoing FDPP Program
- Previous evaluations
- Provides explicit, traceable rationale for the inclusion (or exclusion) of various PACTs and risk elements

## Current work and future plans

- Applications:
- Technology road-mapping:
- Ongoing at JPL, NEPP pilot at GSFC upcoming
- Project Implementation:
- Code Q budget for pilot applications
- NASA Design for Safety Program (DfS)?
- Mission and System Design:
- Code Q budget for pilot applications
- JPL CSMAD teaming, NASA DfS?
- Technology Portfolios:
- Teaming arrangements in development (NASA Code S, NASA DfS, DoD, JPL/TAP)
- Tool Availability:
- Tool “official” releases every 6 months
- Readily available to personnel for performing NASA work

## DDP Tool Development

- [1] Currently available only in the java version of DDP

## What you can do next

- Ignore all of this (I really hope not!)
- Get additional information/education
- Schedule a tutorial, synchronize with a visit out this way
- Get a copy of the tool (**Contact Steve Botzum@GSFC**)
- Watch for upcoming website
- Try it on your project
- We can help facilitate initial usage on a few projects over the next several years
- Tutorials and/or detailed discussions
- Provide facilitator and/or team members
- Contact Information:
- Dr. Steven Cornford: (818)354-1701, steven.cornford@jpl.nasa.gov
- OR
- Mr. Timothy Larson: (818)354-0100, timothy.larson@jpl.nasa.gov

## BACK-UP SLIDES

## Step 1: Develop the Requirements Matrix

**Step 1: Develop the Requirements Matrix**

- Where are we going, what are we doing there, and for how long are we doing it? - Prioritize issues and concerns

- Each column sum yields the extent
- to which each FM impact success

- Sum on each Row yields
- the extent to which each
- mission requirement
- is impacted by the FMs

***R***

- Failure Modes

- Mission
- Requirements

**Impact of a given FM on a particular requirement**
**(e.g. % of requirement lost if FM occurs)**

- 

- 

**Identify requirements**
- Weight by importance to project
- Will result in an indentured list
- Can get information from project personnel or requirements documents
**Identify failure modes**
- May have non-certain likelihood of occurring if we do nothing
- Will result in an indentured list
- From FMECA, brainstorming, FTA, experience, etc.
**Evaluate impacts of FMs (if occurs) on requirements**
- Use percentage of requirement lost
- Start with: 0, 0.1, 0.3, 0.9 and 1.0, refine with better numbers as get more detailed

## Identifying the Failure Modes/Risk Elements

**First step: Understand the system or technology**
- Drawings/schematics, block diagrams, functional requirements, WBS elements, etc.
**Failure Mode Identification Methods**
- Brainstorming with “critical mass” of expertise of designers and specialists
- CogE/expert interviews
- Use requirements to help ID failure modes
- What could keep requirement from being met?
- Integrate Top-down and bottom-up evaluations
- Integrate results/information from other tools and processes
- Fault Trees, Risk Models, Requirement trees, etc.
- Produces a failure mode/risk element tree

**Notes:**
First step: Understand the system or technology
Drawings, schematics, block diagrams, functional requirements, etc.
Failure Mode Identification Methods
Brainstorming with “critical mass” of expertise of designers and specialists
Materials, Processes, Design, Thermal, Mechanical, Optical, Electrical, System, etc.
CogE/expert interviews
Begin with failure modes already considered and PACTs already done
Include possible, but unlikely FMs - application/requirements may change, constraints may emerge
Use requirements to help ID failure modes (What keeps requirement from being met??)
5 year life: wear-out, loss of consumables, fatigue, degradation, etc.
Optical throughput: occlusions, mirror surface, coatings, etc.
Top-down meets bottom-up in the middle
Check for completeness and only go down to detailed level necessary
Use disagreements between experts to identify underlying (lower-level) issues
Fault Trees, Failure Modes and Effects Criticality Analyses (FMECA) 
Will use if available, can generate as part of the DDP process
Have begun discussions with JSC to port FMECAs directly into the tool

## Step 2: Develop the Effectiveness Matrix

**Step 2: Develop the Effectiveness Matrix**

- How do we adequately ensure success in the presence of potentially activated failure modes and defects?

- Product of elements within each Column
- yields ***e***, the net PACT* coverage for each failure mode  (“Escape” chance)

***E***

- Failure Modes

- PACTs

- Product of elements
- within a Row  yields, ***f***,
- figure of merit
- for each PACT*

**Effectiveness of a given PACT* on a particular FM**
**(% chance of detecting or preventing)**

-  

-  

**Utilize failure modes identified in previous step**
**Identify PACT* options**
- We will have a ‘pre-canned’ set
- Include efforts designers have put into clever designs which prevent problems from occurring
**Evaluate effectiveness of PACTs on detecting/preventing failure modes**
- Start with: 0, 0.1, 0.3, 0.9 and 1.0, refine with better numbers as get more detailed
***PACT**s = **P**reventative measures, **A**nalyses, process **C**ontrols, and **T**ests
- 	(i.e. everything we can do to detect/prevent failure modes)

**Notes:**
Tie back tot he waterfall chart

## Step3: Using DDP to Tailor and Optimize

**Step3: Using DDP to Tailor and Optimize**

- Risk Balance
- The residual risk is the ‘expected value’ of the failure mode, i.e, the product of it’s likelihood, severity and chance of escaping
- Measures product of how much we care and chance we will miss it
- Risk balancing trades off PACT options against residual risks
- Versus constraints (mass, power, $, etc.)
- Can shift priorities
- Select different PACT combinations
- Capture design and PACT decisions
- Modified/refined with project life cycle

**Risk Balance (before)**

- Failure Modes (same
-  sequence as FM Impact pareto)

**Residual Risk**

**Risk Balance (after)**

- Failure Modes (same
-  sequence as FM Impact pareto)

**Residual Risk**

***Residual Risk****** = r = i x e =***

***The extent of it’s impact x How likely it will occur***

- For each failure mode:

## Begin with high level
Mission requirements, failure mode and PACT categories
Matrix entries may represent mostly engineering judgement

- Mission requirements, failure mode and PACT categories
- Matrix entries may represent mostly engineering judgement

***E***

- Weighted Failure Modes

- PACTs*

***R***

- Failure Modes

- Requirements

**Reqts, FMs and PACTs are iteratively refined**

## Some Computational Details

**Some Computational Details**

- Use best available information in filling out the matrix
- Use applicable historical data, modeling, simulation or test results, or focused evaluation efforts
- Begin 1, 3, 9 “engineering judgement scale” from Quality Functional Deployment - More typical at higher levels of evaluation
- 0, 0.1, 0.3 and 0.9 are fractions of requirement not met
- or 0, 0.1, 0.3, 0.9 are chance of detection/prevention by a PACT
- Use more detail as knowledge or need warrants - Typically at lower levels
- Advantage of Physics of Failure approach is that we can leverage the volumes of data in industry and universities
- May know particular requirements response or specific PACT effectiveness
- FM likelihoods may be available from statistical models, vendor data, historical data, focused R&D efforts including technology development
- Areas of uncertainty can be flagged as liens which may go away if other PACTs are found effective or impact is evaluated in detail
- Risk Balance
- Can be simple product I just described or more sophisticated functional relationships

## Simplified DDP Summary

**Simplified DDP Summary**

- DDP utilizes two matrices: the Requirements matrix (***R***) and the Effectiveness matrix (***E***)

**Impact of a given FM on a particular requirement**

***R***

- Failure Modes

- Mission
- Requirements

- 

- 

## Process chart for Infuse Technology (IT)

**DT**

- Technology
- Assessments

- Build
- Infusion
- Roadmap

- Evaluate
- Status

- Transfer
- Technology

- Maturation
- Readiness?

**XXX**

- = Other DNT Processes

- XXX

- = IT sub-processes

- Transfer
- Readiness?

**DNP Processes**

- External
- Events

**DNT Processes**

**IT Processes**

- Reports [1]

- Roadmaps [2]

- Reports [3]

- [1] These reports include the results of the various assessments including risk and maturity evaluations, and the information necessary to build infusion roadmaps
- [2] These roadmaps include technical milestones, optimal risk reduction paths, success criteria and critical documents/records
- [3] These reports include the results of element execution and measurements of progress against the roadmaps
- [4] Pedigrees include results and recommendations, but may also include hardware and software components

- Pedigrees [4]

## Tools for Managing Infusion Risk

- Determine the relative importance of various risk elements
- Input trees of requirements (and relative importance)
- Input trees of risk elements
- Evaluate consequence (and likelihood) of risk elements on each requirements

- Select PACT combinations to reduce risk (_**P**_reventative measures, _**A**_nalyses, process _**C**_ontrols and _**T**_ests)
- Use existing database or add new ones
- Each has an effectiveness at detecting (or preventing) the occurrence of some collection of risk elements
- Each has resource costs associated with it ($, schedule, mass, etc.)
- Choose a combination of PACTs

- Results: Requirements drivers (extent to which requirement is/was at risk)
- Total height indicates extent to which requirement was at risk (really needed?)
- Red indicates extent to which requirement is still at risk (need to do more?)
- Blue are requirements not at risk (do they belong?)
- Results: Residual Risk (extent to which a risk element is still present)
- Total height indicates relative criticality of each risk element
- Green indicates extent to which each element which has been eliminated
- Red indicates extent of residual risk of each element
- Results: PACT combination selected for implementation
- Begin detailed WPA development
- Each now has specific, traceable reasons for implementation
- Enables improved tailoring
- Enables decisions regarding consequences of not doing

- Have developed and applied a tool for assessing the maturity of technologies and roadmapping the path to infusion

## Backup