Managing Risk as a Resource using the Defect Detection and Prevention
process

Steven L. Cornford, Ph.D

Jet Propulsion Laboratory

California Institute of Technology

Pasadena, CA 91109

NASA Code QE is funding a number of programs to develop and implement
improvements in the efficacy and efficiency of Verification and
Validation and Risk Management processes for space-flight projects. A
physics-of-failure-based Defect Detection and Prevention (DDP)
methodology has been developed and is currently being implemented on
various JPL projects and as part of JPL's new model-based development
environment. The DDP process weights the criticality of the various
relevant failure modes by including the likelihood and impact on mission
requirements. The net effectiveness of various subsets of **PACT**s
(**P**reventative measures, **A**nalyses, process **C**ontrols and
**T**ests) can then be evaluated against these weighted failure modes to
obtain the residual risk for each of these failure modes and the
associated resource costs to achieve these risk levels. This identifies
the project-relevant "tall pole" failure modes and design drivers and
allows real time tailoring with the evolution of the design and
technology content. The Defect Detection and Prevention methodology
allows risk management in its truest sense: identifies and assesses
risk, provides options and tools for risk decision making and mitigation
and allows for real-time tracking of current risk status.

# Introduction

JPL continues to make progress in its challenge from NASA to fabricate
and operate spacecraft Faster, Better and Cheaper (FBC)\[[^1]\]. No
longer is the posture one of risk avoidance but rather one of risk
management. A key element of NASA/JPL's Risk Management approach is to
consider "risk as a resource"\[[^2]\]. Like schedule, mass and power,
risk is now a resource to be traded against other resources and
optimized subject to constraints. This process has been partially
enabled by the NASA Code Q focus on developing better tools and methods
for managing risk [^3].

The typical NASA/JPL project development cycle is evolving to an 18
month development cycle in which decisions will need to be made almost
"real-time" in a model-based development environment[^4]. Like any
decision, there is a chance that it will be wrong. This leads to the
chance that something fails[^5]. This is where risk is introduced - What
is the risk of failure due to a given decision or action? Since risk can
be reduced by expending resources, and risk itself is a resource, it
would be valuable to have an integrated methodology for trading these
resources.

Early decisions usually have the most influence on the project risk, but
given the realities of the fast-track spacecraft development cycle, it
is often necessary to make decision based on incomplete data.
Furthermore, each of these decisions may result in different, or
additional, derived requirements. The DDP process allows the
requirements to be captured, the current risks to be estimated, and
allows tradeoffs to be made with the available data. This data regarding
risks and consequences ranges from engineering judgement to actual
flight article test data depending on the stage of the project
development process. The DDP process integrates this variety of data in
a "top down" approach which is synchronized to the project development
cycle.

# Approaches to Measuring Risk

In order to fully utilize risk as a resource, and make quantitative
comparisons and decisions, it is necessary to quantify risk (before and
after various mitigation activities). However, some risks are difficult
to quantify, especially in ultra-low volume spacecraft fabrication.
Historically, experienced engineers used their vast experience to
minimize project risk. However, this technique was neither quantitative
nor systematic. Other techniques were introduced to resolve these
shortcomings, but each has their own pros and cons, as summarized in
Error: Reference source not found.

+-----------------------------------------------------------------------+
| #### Engineering Judgement                                            |
|                                                                       |
| Pros: Quick and dirty, Tailored to specific project under             |
| evaluations, Utilizes the experience of the best and brightest,       |
| Widely used                                                           |
|                                                                       |
| Cons: Not systematic or quantitative, Only as good as the expert      |
| opinions provided                                                     |
+-----------------------------------------------------------------------+
| #### MIL-STD-217-type Calculations                                    |
|                                                                       |
| Pros: Systematic approach, Provides detailed failure rate numbers,    |
| Includes approach for modifying failure rates due to hardware         |
| characteristics, Widely used                                          |
|                                                                       |
| Cons: Only valid for random failures, Does not address design,        |
| workmanship or manufacturing defects                                  |
+-----------------------------------------------------------------------+
| #### Physics of Failure *(For example: \[*[^6]*\])*                   |
|                                                                       |
| Pros: Validated models using underlying physics (no guesswork),       |
| Identifies specific parameters and points the way to additional       |
| experiments or evaluations, Can incorporate stochastic or systematic  |
| sources of parametric error, Widely used                              |
|                                                                       |
| Cons: Not really applicable for functional performance anomalies (or  |
| engineering of failure), Can get extremely complex and difficult to   |
| validate if expanded to include all varieties of workmanship or       |
| manufacturing defects, Bottoms-up approach which can get very large   |
| if extended to spacecraft systems                                     |
+-----------------------------------------------------------------------+

**Table 1 Summary of the pros and cons of various risk measurement
techniques**

It would be desirable to have an approach that integrates the best of
the various current approaches. At the highest level, the approach
should be able to begin with historical corporate data and expert
judgement, gaining accuracy and precision, as required, at the lower
levels \[[^7]\]. This can come from failure rates, physics of failure
modes, "root cause" data from internal or external sources, focused
experiments, etc. The DDP process provides a means of integrating this
variety of data.

# DDP Methodology

In addition to integrating the best features of the various means of
risk measurement, the process for implementing DDP is systematic and
synchronized with the project life cycle. It can capture the evolving
requirements, document the rationale for decisions (so they could be
revisited later with better information) and provide a current ranked
risk list to provide focus for the project design team. It is also
integrable into JPL's future model-based development environment.

## 

## DDP is a Four-step Process

The application of the DDP process involves four steps: 1) Develop the
Requirements Matrix, 2) Develop the Effectiveness Matrix, 3) Optimize
the Residual Risk (subject to constraints) and 4) Iterate with the
Project Life Cycle. The key constituents of these steps of the DDP
process are the Failure Modes (or Mechanisms at lower levels),
Mission/Project Requirements (may be derived requirements at lower
levels) and the suite of available PACTs[^8]. The Failure Modes,
Requirements and PACTs occur at various levels that range from mission
level down to the device or semiconductor level. The DDP process is
tailored to evolve with the project development cycle to allow risk
elements to be identified as early as possible and remain consistent
with the necessary initial allocation of resources and facility
scheduling.

The Failure Modes may be found by a variety of techniques: Fault Trees,
FMECAs, Brainstorming with "critical mass" of experts, etc. As lower
levels of failure modes are identified, they will eventually reach the
"root cause" or mechanism level and standard physics of failure
evaluations may be performed as required. However, one of the key goals
of the DDP process is to allow "good enough" decisions to be made at as
high a level as possible. These identified failure modes are critical to
the DDP process and show up in both the Requirements Matrix and in the
Effectiveness Matrix which are described below.

## 

## Developing the Requirements Matrix (RM)

The RM (see Figure 1) consists of mapping failure modes against the
requirements where the entries in the matrix are an evaluation of the
impact on the requirement should the failure mode occur. Note that the
requirements may be individually weighted relative to their importance
and the failure modes may be weighted by their *a priori* probability of
occurrence (i.e. with no mitigation activities) - in most cases this
weighting factor is chosen to be 1.0.

The primary utility of the RM is to identify the "tall pole" failure
modes; that is, the sum of the impacts (and likelihood) of a particular
failure mode on all of the weighted requirements. Repeating this for all
of the failure modes, and sorting by magnitude of residual risk provides
a list of the "tall pole" failure modes. This list is essentially
equivalent to the well-known Significant Risk List.

The requirements may also be evaluated to identify "design drivers" and
irrelevant requirements. This is accomplished by summing across all
failure mode impacts, for a given requirement, to accumulate a total
representing the extent to which various requirements are impacted by
the potential failure modes.

## Developing the Effectiveness Matrix (EM)

The EM consists of evaluating the effectiveness of the various PACT
options effectiveness (i.e. probability of NOT detecting or preventing)
against the weighted failure modes (see Figure 1). PACT options for
detecting and preventing failure modes range from design rules to system
level tests and have varying effectiveness against different failure
modes. The goal is to select the minimum set that achieves the desired
risk level within constraints.

The entries in the EM represent the chance for "escape" and the product
of these entries, for a given failure mode, yield the residual risks.
The EM is thus the means by which to evaluate the overall effectiveness
of a given PACT collection. The determination of the effectiveness of a
given PACT on a given failure mode can be accomplished a number of ways.
One can use historical data, results of experiments or measurements, at
the "root cause" level one can also use data from the literature.

![](media/image1.wmf){width="4.393055555555556in"
height="2.1041666666666665in"}

**Figure 1 The Requirements and Effectiveness matrices.**

## ![](media/image2.wmf){width="4.299305555555556in" height="3.2159722222222222in"}

## 

## 

## Optimizing the Residual Risk

One can now form the product for each failure mode, of the impact should
it occur, with the likelihood that it will escape into the mission, and
the result is the residual risk. Since each PACT has varying
effectiveness against different failure modes, one can (and must) "mix
and match" different PACT combinations each of which has associated
resource expenditures. For example, a design decision like additional
shielding or redundancy may require more mass, while a decision to
perform a test means allocation of schedule and money. Figure 2
illustrates the Risk Balance for a given scenario. Note that the line in
the risk balance is the location where each failure mode is receiving
risk reduction proportional to its importance.

### Iterate versus Project Life Cycle

As discussed previously, the requirements and hardware maturity evolve
with the project life cycle. Their refinement to lower levels may result
in modification of various levels of requirements. Similarly, the
evolution of the design results in evolved failure modes. The DDP
process allows one to capture these changes, evaluate their impact on
the "tall pole failure modes", identify design drivers, update the
Effectiveness matrix and re-evaluate the Residual Risk.

# Summary

The Defect Detection and Prevention process for risk management has been
described. This process evaluates the impact of the project relevant
failure modes on the project requirements. It then evaluates the
effectiveness of a given collection of PACTs and allows optimization of
this collection subject to resource constraints. It identifies the
current "tall pole failure modes" (residual risk) and design drivers,
documents decision rationale, and provides a current list of the
project-required PACTs. It also can identify areas where focused studies
are required.

The DDP is currently in use at JPL and has seen a variety of pilot
applications from the spacecraft level to Multi-chip module level. The
tool for implementation is currently spreadsheet-based, but a relational
data-based tool which can accommodate Monte Carlo evaluations using
statistical distributions of effectiveness and impact entries. After a
number of pilot implementations, we have accumulated a number of lessons
learned that will be the subject of another paper. Briefly, there must
be concurrent usage of a "critical mass" of expertise and the process is
more quickly performed at lower levels of assembly (e.g. a given
technology or device).

# Acknowledgements

I would like to thank Philip Barela, Dr. Gary Blackwood, Mark Gibbel,
Tom Gindorf, Dr. Michael Greenfield and Tim Larson for many valuable
discussions. I would also like to thank the members of the Reliability
Technology group for their help in implementing and refining the DDP
methodology.

The research described in this paper was carried out by the Jet
Propulsion Laboratory, California Institute of Technology under a
contract with the National Aeronautics and Space Administration through
Code Q.

#### References

[^1]: . National Aeronautics and Space Administration*, NASA Strategic
    Management Handbook,* Washington D.C., April 1996b

[^2]: . M. A. Greenfield and T. E. Gindorf, *Risk as a Resource -- A New
    Paradigm,* Proceedings of the ESREL 96 -- PSAM III Conference,
    Create, Greece, Vol. 3, pp. 1597, Springer-Verlag, Berlin, June 24
    -- 28, 1996

[^3]: . Liam Sarsfield, "*The Cosmos on a Shoe String"*, RAND Critical
    Technologies Institute, Washington D. C., MR-864-OSTP, 1998

[^4]: . For example, M. Sander, "Develop New Products Project", JPL
    Internal document, 1997

[^5]: I am using the word "failure" here in its broadest sense: failure
    to meet all of the mission/project requirements.

[^6]: . P.Lall, M.Pecht, "An Integrated Physics-of-Failure Approach to
    Reliability Assessment", *Advances in Electronic Packaging*, ASME
    EEP-Vol.4-1, 1993.

[^7]: . S. Cornford and M. Gibbel*, Methodology for Physics &
    Engineering of Reliable Products*, WESCON 96 Proceedings, Anaheim,
    CA, October 22-24, 1996

[^8]: PACTs are everything which can be done to prevent or detect
    failures; PACT= [P]{.underline}reventative measures,
    [A]{.underline}nalyses, process [C]{.underline}ontrols and
    [T]{.underline}ests.