**Quantitative Risk Analysis: Challenges and Opportunities at NASA**

**Dr. William Vesely**

**Manager, Risk Assessment**

**NASA Headquarters**

**300 E Street SW**

**Washington, DC 20546**

**Phone: 202-358-1556**

[**wvesely@hq.nasa.gov**](mailto:wvesely@hq.nasa.gov)

**Abstract**

NASA carries out a variety of quantitative risk assessments in
conducting its work. These assessments range from specialized studies,
such as the assessment of the safety risk involved in transfer of the
DC-8 aircraft, to large scale probabilistic risk assessments (PRAs),
such as the PRAs conducted on the Space Station and Space Shuttle.
Because of the range of problems addressed, there are a various
challenges in carrying out quantitative risk assessments involving both
modeling and implementation. There are also opportunities for using
quantitative risk analysis to improve decision making and risk
management. This paper presents a spectrum of examples of quantitative
risk analyses, from large scale to specialized. The examples also
illustrate the range of approaches being utilized or being developed.
The examples are the Space Shuttle PRA, a decision application
accounting for risk uncertainties, a project risk assessment of
cumulative risks, and an assessment of software development risks.

**Introduction**

NASA conducts a variety of quantitative risk assessments to support
project management and decision making. "Quantitative risk assessments"
means here assessments that estimate numerical values for probabilities
and consequences, along with associated uncertainties. Many assessments
focus only on estimating probabilities of failure or probabilities of
other undesirable events occurring. These are also considered as being
quantitative risk assessments here. The quantitative risk assessments
that are carried out range from narrow, specialized reliability or
probabilistic assessments to large scale Probabilistic Risk Assessments
(PRAs). Project managers utilize these quantitative risk assessments, as
deemed appropriate, in assessing and managing risk.

Even though a variety of quantitative risk assessments are performed,
there is presently no standard framework for carrying out and for
utilizing quantitative risk results. Risk matrices are standardly used
in project risk management, such as the traditional 5x5 risk matrix,
which categorizes a contributor's probability and consequence into a
given category. However, these are generally viewed as being qualitative
risk assessments, with sometimes minimal underlying, structured
quantitative risk assessments carried out. There is thus a challenge and
opportunity for upgrading the role of quantitative risk assessments to
better assist project management. This paper presents a spectrum of
examples that span different approaches and applications. The examples
are the Space Shuttle PRA, a decision application accounting for risk
uncertainties, a project risk assessment of cumulative risks, and an
assessment of software development risks.

**PRA at NASA**

At NASA, Probabilistic Risk Assessments, or PRAs, are conducted in
various situations to quantify the associated risk. The most extensive
PRA conducted is a full scope PRA, which models a comprehensive set of
scenarios leading to all undesired end states , such as loss of crew and
potential injury to the public. A full scope PRA models the
comprehensive set of initiating events and contributors that can lead to
the undesired end states. The basic causes that are modeled include
hardware failures, human errors, process errors, and phenomenological
events. The next lower level PRA is a limited scope PRA, which focuses
on one or more specific end states such as a specific mission failure.
Credible initiating events and contributors leading to the particular
end states are included. The lowest level PRA is a simplified scope PRA
focusing on a particular end state and including only particular
contributors.

NASA's Procedural Requirement NPR 8705.5 (1) defines when a PRA is
required and the scope of PRA required. In general a full scope PRA is
required for new programs when the consequences can involve human safety
or health or can entail high schedule criticality. Limited scope PRAs or
simplified scope PRAs are required or recommended in other situations. A
limited scope PRA or simplified scope PRA can be waived at the direction
of the program or project with sufficient basis. The table on the next
page summarizes the criteria for the scope of PRA required in new NASA
projects. More details are contained in NASA's NPR 8705.5.

**The Space Shuttle PRA**

The Space Shuttle PRA, has recently been completed and is being revised
and being updated. This is actually the most recent Space Shuttle PRA
since several earlier PRAs have been carried out on the Space Shuttle.
This most recent PRA is most notable for its comprehensiveness, detail,
and involvement by the NASA centers The Space Shuttle PRA is like any
other traditional, full scale PRA in that it constructs accident
scenarios that identify events and failures that can lead to a defined
end state or end states. For the Space Shuttle PRA, the end state that
is the focus is the loss of crew and vehicle (LOCV) during a mission.
The accident scenarios are modeled using event trees. Each accident
scenario that contains one or more system failures generally uses fault
trees to resolve the basic events leading to system failure. Many
accident sequences involve only one event where that event is a loss of
critical tile, a critical debris hit, an explosion or fire, or other
phenomena. The basic events, which in certain scenarios also involve
crew errors or component functional failures, are then quantified to
determine the probabilities of the accident scenarios occurring. Figure
1 illustrates the event tree and fault tree modeling in a PRA. NASA's
PRA Procedure Guide (2) describes the modeling and quantification
approaches that are to applied in developing a PRA.

**Table 1. Criteria for Selecting the Scope of a Probabilistic Risk
Assessment (1)**

![](media/image1.wmf){width="8.990972222222222in"
height="4.064583333333333in"}

![](media/image2.png){width="5.759722222222222in"
height="4.045833333333333in"}

**Figure 1. Event Tree and Fault Tree Models in a PRA**

As was indicated, the Space Shuttle PRA is the first PRA performed on
the Space Shuttle that had extensive NASA Center involvement in the
modeling and quantification. It is the most comprehensive and detailed
PRA that NASA has undertaken, involving extensive fault models, human
error models, and phenomenological models. Figure 2 summarizes general
features of the current Space Shuttle PRA.

![](media/image3.wmf){width="4.895833333333333in"
height="2.8958333333333335in"}

A spectrum of results is obtained from the Shuttle PRA. These range from
the probability distribution for the number of missions to failure to a
listing of the detailed accident contributors. Illustrations of these
results are shown below. The numbers are not shown since the focus here
is on the diverse range of results obtained.
![](media/image4.png){width="4.999305555555556in"
height="3.7493055555555554in"}

**Figure 3. Probability Density Function for the Number of Missions to
LOCV**

![](media/image5.wmf){width="7.020833333333333in" height="3.46875in"}

**Figure 4. Listing of Detailed Contributors to LOCV**

The Space Shuttle PRA is different from more standard nuclear power
plant PRAs in a number of ways. A significant portion of the data
collected by NASA for the Space Shuttle consists of faults, which are
not yet failures, but have the potential of becoming failures if not
corrected. This fault data consists of leaks, cracks, material
anomalies, and insulation debonding faults. Consequently, detailed
quantitative fault models were developed for the Space Shuttle PRA which
involved assessing the severity of each fault and its potential for
becoming a failure, assessing the effectiveness of processes for
detecting the fault, and assessing the effectiveness of processes for
correcting detected faults (3). The dominant types of failures that
originated from faults involved phenomenological events such as
explosions, ruptures, and material anomalies. Thus, phenomenological
probabilities and consequences were a focus in the modeling.

The human error probabilities that were significant contributors were
generally not the more routing errors of omission (e.g, slips) but
involved cognitive errors by the crew, such as the crew engaging the
landing gear of the Orbiter either too early or too late. Thus, detailed
cognitive models were developed which took into account the specific
experience, feedback, communication, and stress. An astronaut was
involved in identifying the human responses and errors, which were then
reviewed by the astronaut office. Selected human error probabilities
were also validated using collected NASA simulation data. Certain
processing faults involving human errors were also important
contributors, such as errors involved in processing the fuel for the
Solid Rocket Boosters (SRBs). These processing faults were quantified
using experience data, as well as expert judgment. Reference 5 describes
PRA modeling with Shuttle applications.

The completed Shuttle PRA presents both opportunities and challenges to
NASA. There exists no standard framework for incorporating PRA in
project decision-making. The particular way probabilistic results are
used is largely up to the project manager. Consequently, work is
underway to develop guidelines for using PRA results, and probabilistic
results in general, to assist decision-making and risk management. One
of the issues with a PRA is the large uncertainties that can be
associated with the quantitative results. This causes managers and
decision-makers to be reluctant to undertake a PRA because of the
difficulty with dealing with the uncertainties in the results. The
subsequent section describes one approach that is being applied that
accounts for uncertainties from a probabilistic analysis, while still
yielding practical decision rules. It also has a decision-theoretic
underpinning. Other approaches are also being pursued, such as defining
action rules and interpretations that account for uncertainties.

For a new manned spaceflight mission or a new highly critical mission,
there can be defined numerical risk criteria or numerical risk goals.
When there is a numerical criterion or goal then the appropriate PRA
result can be compared with the criteria or goal. The next section is
relevant to this type of usage. Often risk results represent one type of
information that is used in decision-making. Consequently, risk results
need to be combined with other types of information such as cost and
performance information. A multi-attribute, risk-informed
decision-making framework is being developed and being reviewed to
assist project decision-making. The section after the next illustrates
one particular approach using the analytic hierarchy principle (AHP).

Because of the dominance of single failures and phenomenological events
in the Space Shuttle PRA, time dependencies and dynamic, functional
failures were not important contributors. However, these time
dependencies and dynamic interactions can be important for certain new
missions, such as the more extended missions. Time dependent and dynamic
methodologies, and tools, are consequently being assembled. The updated
fault tree handbook developed under NASA sponsorship describes some of
these approaches (4). External events such as micrometeoids and orbital
debris (MMOD) are also important and sometimes dominant contributors to
mission failure and loss of crew. Significant effort is being expended
in upgrading the present models to be fully probabilistic with
comprehensive treatments of uncertainties. More generally, Monte Carlo
simulation techniques are being developed and being refined to account
for variabilities and uncertainties. An example is the Monte Carlo
simulation model being refined for foam debris generation from the
external tank of the Space Shuttle.

NASA is in continually in the process of reviewing its risk management
practices to improve their effectiveness. The use of PRA results in
particular, and the use of probabilistic analysis more generally, is
currently being given particular attention. The benefits of a
risk-informed decision-making framework is also being given special
attention. As has been indicated, NASA has developed a PRA procedures
guide and has issued requirements for conducting PRAs. These are
presently being revised. To assist in implementations, a NASA advisory
group has been formed on PRA and probabilistic applications. More
detailed procedures and applications guides are being developed, and a
broad-application component failure rate data base is being assembled.
In these ways, it is the aim that PRAs, and probabilistic analyses in
general, will be more effectively utilized in NASA's risk management and
decision-making..

**Utilization of Probabilistic Results Accounting for Uncertainties**

Quantitative risk assessments often have larger uncertainty bounds
associated with an uncertainty distribution for a quantified result. For
example, the Shuttle PRA has large associated uncertainty bounds (e.g,
5% and 95% bounds) for certain failure probabilities where there is
little data. These uncertainties often pose a quandary for a decision
maker or risk manager who would like to a have a more well-defined risk
estimate to use. Uncertainties in a risk assessment result are
quantified by determining a probability distribution for the result. The
probability distribution gives the probability for the result having
specific values. Figure 3, shown previously, illustrates the uncertainty
distribution obtained for the probability of loss of crew and vehicle
(LOCV) for the Shuttle PRA.

Decision theory provides a systematic framework for considering the
impacts from selecting a specific estimate from an uncertainty
distribution to use in decision making. It does this by defining a loss
function that shows the impacts of using a specific value from the
uncertainty distribution. The decision theory framework is overviewed
here since it provides a basis for specific NASA applications. The
discussion focuses on a failure probability with a quantified
uncertainty, such as obtained in the Shuttle PRA. However, the
discussion applies to any quantified risk result with uncertainty
distribution..

Based on decision theory, the *estimation risk*$R(p)$from selecting a
failure probability estimate $p$ from its uncertainty distribution is
defined to be (6,7,8):

$R(p) = \int_{}^{}{L(p,p)f(p)\text{dp}}$, (1)

where

$L(p,p) =$ the loss when the selected value is $p$ (2)

and the true probability value is $p$,

and

$f(p) =$ the uncertainty distribution (density function) (3)

for the true failure probability value.

According to decision theory principles, the optimal estimation value
$p_{\text{opt}}$ that should be selected for the failure probability to
use in decision making is the one that minimizes the estimation risk:

$R(p_{\text{opt}}) = \underset{p}{\text{min}}\int_{}^{}{L(p,p)f(p)\text{dp}}$.
(4)

The optimal estimation value $p_{\text{opt}}$ accounts for all the
possible values for the actual failure probability through the
uncertainty distribution and minimizes the loss associated with the
selected value. There is a sound decision theoretical basis for this
approach as the references describe.

The standard quadratic function is the form used for $L(p,p)$to provide
guidelines for particular NASA applications. This choice of loss
function also gives results that are consistent with those most often
used in risk decision recommendations, as will be described. The
quadratic loss function is defined as:

$L(p,p) = K(p - p)^{2}$ ; symmetric, quadratic losses when (5)

using an estimate $p$.

K can be any scaling constant. The quadratic form represents equal size
losses in either underestimating or overestimating the true value of the
failure probability. The losses grow quadratically as the
underestimation or overestimation increases. The quadratic form
penalizes large departures of the selected value from the true value.

When the loss function is quadratic then the optimal selected value that
minimizes the risk is the *mean value* of the uncertainty distribution:

$p_{\text{opt}}$= the *mean value* of the uncertainty distribution for
(6)

a quadratic loss function form.

This result can be shown by minimizing Equation (4). The optimal value
does not depend on any scaling constant, but only on the squared form of
the loss function. The rationale for using the squared loss function is
that underestimation or overestimation of the true failure probability
is equally undesirable. Overestimation of the true failure probability
can cause overly conservative actions to be taken that can include not
undertaking a action under the false impression that the failure
probability is too high. Underestimation of the true failure probability
can result in an action erroneously taken with the false impression that
the failure probability is lower than it actually is.

The mean value is the value recommended or standardly used in
implementations other than at NASA. The U.S. Nuclear Regulatory
Commission (USNRC) uses the mean value of estimated failure
probabilities and accident probabilities to compare with its safety
criteria (9,10 ). The USNRC has used the mean value for almost 20 years
to assess the compliance of nuclear power plants in satisfying criterion
on the probability of core damage and the probability of causing early
and latent cancers to the public from a possible nuclear accident. The
experience of the USNRC has shown that the mean value sufficiently
accounts for uncertainties in comparing to a criterion even when the
uncertainties are large, as they are for low probability, high
consequence events.

The American Society of Mechanical Engineers (ASME) also recommends
using the mean value of estimated failure and accident probabilities to
compare with criteria (11, 12). Furthermore, the mean value of the
estimated failure probability or accident probability is standardly used
in Probabilistic Risk Assessments (PRAs), both in the USA and
internationally, to compare with criteria (13, 14). These applications
have shown the mean value to be an effective estimate to compare to a
criterion.

In applications, an acceptable failure criterion is separately
determined and the mean value is then obtained from the quantitative
risk model to compare with the criterion. An example application
involves a risk assessment for the 24 Composite Overwrap Pressure
Vessels (COPVs) on the Space Shuttle. If any one of the vessels
catastrophically fails then there can be loss of life and vehicle,
either at the launch pad or in flight. The consequences of failure can
thus be catastrophic. To define a failure probability criterion, the
number of opportunities for failure needs to be also considered. The
number of vessels can be incorporated by defining an acceptable failure
probability criterion for a launch, which includes the contribution from
all the vessels. The number of launches that are planned or that can be
made need also to considered since each represents a separate
opportunity for failure.

The total, or cumulative, failure probability over the number of
launches needs to be small to control the possibility of a failure
occurring. An example of such a small probability is the criterion that
the cumulative failure probability be less than 1/100 over planned
launches. If there are than 10 launches foreseeable then the associated
individual launch failure probability criterion would be 1/1000 per
launch. The second step is then to determine the mean failure
probability value to compare with the criterion. This involves using
agreed-upon models, data, and uncertainty characterizations. The
uncertainty characterizations are used to determine the uncertainty
distribution for the estimated COPV failure probability per mission from
the 24 vessels. Since the sum of mean values is mean of the sum, the
mean value for each vessel is determined and the sum taken . If the sum
of the means is less than the criterion then the launch failure
probability due to COPV failure is assessed to be acceptable. If the
mean value is above the criterion then action needs to be taken to
reduce the estimated failure probability. This action can involve
removing conservatisms in the analysis, making operational changes, or
making design changes.

**Assessing Cumulative Risks to Assist Project Risk Management**

As a routine part of project risk management at NASA, risk contributors
are portrayed on a 5x5 risk matrix, or on a similar matrix, which is
also a common practice in general project risk management. The x-axis of
the matrix identifies different ranges of probabilities or frequencies.
The y-axis identifies different ranges of consequences. Alternatively,
these axes can be interchanged. The probability and consequence ranges
are ordered in terms of increasing value. The corresponding cells of the
risk matrix are then categorized into different risk levels, such as
High, Medium, and Low, which also generally have different associated
colors such as red, yellow, and green. The risk matrix is a useful tool
for characterizing the risk categories of the individual contributors.

One limitation of the present risk matrix approach is that the
*cumulative risk* *contribution* from a set of contributors is not
obtainable in any accurate manner. This does not allow the total risk to
be determined from a set of contributors. It also does not allow the
*relative cumulative risk contribution* to be determined for a set of
contributors. Thus, for example, a project manager cannot determine that
90% of the total risk can be addressed by focusing on a given set of
contributors. The *total residual risk* that remains after actions are
taken is consequently also not identifiable. The project manager cannot
determine that a set of medium risk contributors provides more
cumulative risk than a smaller set of high risk contributors. This
limits the degree of risk control and limits the implementation of
cost-benefit analysis to identify the most cost-effective options for
risk reduction and risk control.

To determined relative cumulative contributions, relative comparisons
need to be made or the individual contributors. The comparisons need to
be made on a ratio scale to provide valid relative contributions which
can be accumulated. The relative assessments can then be used to sum the
total, relative risk contribution from any set of contributors. These
relative comparisons can be carried out by cross-comparing individual
contributors or by comparing the contributors to a reference
contributor. Since relative comparisons are made, and not absolute
assessments, they can often be implemented in a straightforward manner.

An example of cumulative risk evaluations involves assessing the risk
contributors identified in a project to transfer NASA's DC-8 aircraft to
the University of North Dakota (UND). The example is only illustrative,
but it shows the significant additional information provided to a
project manager from assessing cumulative risks in addition to
individual risks. The following activities in transitioning the DC-8
from NASA to UND are identified to be risk contributors if they failed
(15):

1\. *Cooperative Agreement Establishment-*establishing an acceptable
cooperative agreement between NASA and UND

2\. *Aircraft Transition*-physically transferring the aircraft to the
UND facility

3\. *Pilot* *Transition*-establishing trained pilots and providing NASA
pilots as needed

4\. *Maintenance Personnel Transition*-establishing trained maintenance
personnel at UND

5\. *Maintenance Program Transition-* establishing an acceptable
maintenance program at UND

6\. *Science Equipment Transition-*transferring the airborne science
equipment to UND

7\. *Aircraft Facility* *Acquisition-*acquiring an acceptable facility
for the aircraft

8\. *Fire Response Establishment-*establishing acceptable fire detection
and suppression

9\. *Security Services Establishment-*establishing acceptable security
services

10\. *Safety Program Establishment-*establishing an acceptable safety
program at UND

To assess these risk contributors, the probability and consequences of
each activity failing are assessed. Safety consequences are the focus
here involving injury or death to personnel. For a complete risk
assessment, other consequences would need to be considered, including
damage to the aircraft and environmental impacts. As a standard step in
project risk assessment, the size of the probability and consequence is
assessed for each of the contributors and is assigned to the relevant
cell in the risk matrix. The ways the probabilities and consequence are
assessed are not the focus here, other than to note that they are often
subjective and under the purview of the project. The risk matrix below
shows an example assignment of the probability and consequence of each
contributor.

**Figure 5. Risk Matrix Showing Project Safety Risk Contributors**

The risk matrix provides useful information in categorizing the risk
contributions from the items. However, it is difficult to identify the
total risk that is addressed by reducing particular contributors, such
as the highest risk contributor. The question also arises as to whether
the set of medium risk contributors pose higher total risk than the
highest risk contributor. Furthermore, when risk contributions from
items are reduced, the project manager does not have a clear measure of
the total residual risk that remains.

One of the most straightforward ways to assess the relative cumulative
risk contributions is to compare the individual contributors with regard
to their probabilities and consequences. These comparisons are made in
terms of ratios. The comparisons can be carried out in various ways and
one way is to use a reference contributor to compare with the others.
The table below illustrates an assessment of the relative failure
probabilities and relative failure consequences of the individual
contributors. The ratings are made by comparing the probabilities and
consequences of contributors using a standard Analytical Hierarchy
Principle (AHP) scale for ratio comparisons (16). These scales are
adequate provided the comparison ratios are less than a factor of 10.
Since only a restricted number of contributors need to be compared (that
are connected) to obtain all the comparison ratios, the comparisons can
usually be selected to utilize this scale. If this is not possible, then
an expanded scale can be used.

**Table 1. Comparisons of Contributor Probabilities and Consequences**

![](media/image6.wmf){width="5.995138888888889in" height="2.06875in"}

Software packages exist that allow these relative evaluations to be made
in an efficient manner. Using these packages, contributors can be
pairwise-compared to estimate their ratios of probability and of
consequence. Figure 6 below shows an example of a hierarchical tree that
can be constructed. The hierarchy tree shown below has a top objective
to assess project risk contributors. This is subdivided into two
attributes, the probability of the contributor and the consequence of
the contributor. For more complex assessments, the probability criterion
or the consequence criterion would be subdivided into more specific
sub-criteria. This would be the case, for example, if multiple
consequences were being rated for each contributor instead of the single
safety consequence of human fatalities that was rated here. Uncertainty
distributions can also be assigned to the ratings where there is high
uncertainty or differences among the raters.

![](media/image7.png){width="6.379861111111111in"
height="3.6756944444444444in"}

The table below gives the resulting relative contributions to the total
failure probability, to total consequence and to total risk from the set
of contributors. The normalized relative contributions are obtained by
dividing the ratio comparison by the sum of the values. The relative
risk contributions are obtained by multiplying the relative probability
and consequence then normalizing.

**Table 2. Resulting Relative Probability, Consequence and Risk
Contributions**

![](media/image8.wmf){width="5.995138888888889in"
height="2.3305555555555557in"}

Because the contributors are compared in terms of ratios, valid relative
contributions are obtained. The relative contributions identify the
dominant contributors analogous to the risk matrix, but in addition give
the cumulative contribution for any set of contributors. For example,
what the relative risk contributions show in the last column is that
approximately 78% of the total risk is caused by 3 contributors and that
an 2 additional contributors causes an additional 17% of the risk. Thus,
approximately 95% of the risk is contributed by 5 contributors. Resource
allocations can consequently be based on these cumulative results.

The relative risk contributions show the benefits of supplementing the
project risk matrix assessments with relative, cumulative assessments.
For effective risk management, it is necessary to assess and control
cumulative risk. Also, for effective resource allocation, it is
necessary to prioritize resources according to relative risk
contributions and cumulative contributions. Supplementing the project
risk matrix with cumulative risk evaluations thus is a means of carrying
out more effective risk management. Project risk management methods and
tools are being reviewed at NASA to see how these cumulative risk
assessment approaches can be implemented in the most effective manner.

**Quantification of Software Risk**

Quantification of software risk and reliability is presently missing in
many present quantitative risk assessment. As one step to address this
gap, particularly for project risk assessments, Bayesian networks have
been developed to assess the information they provide. The Bayesian
networks constructed utilize project attributes, quality control
metrics, audit findings, and software performance results to estimate
the predicted software reliability. The prediction is updated as further
information is obtained on the project. Both qualitative and
quantitative information can be used. The predicted software reliability
measure can be qualitative or quantitative. Bayesian networks are
described in various references and are tools used to probabilistically
aggregate and update information for a variety of applications
(17,18,19).

The Bayesian network below is an example of a format that is applicable
to software risk assessment in a project:

![](media/image9.png){width="5.194444444444445in"
height="2.7777777777777777in"}

**Figure 7. Network of Factors Affecting Software Failure Probability**

In the above network, the top node is the prior information, which is an
initial estimate of the resulting software failure probability (or
equivalently the software reliability) that will be produced. For this
application, the software failure is defined to be a failure of the
software that will result in loss of mission. The prior estimate is
based on past history as well as expert judgment. The additional nodes
represent specific information about the software or information that is
obtained from audits or testing as the software is developed. The arrows
to these additional nodes represent the influence of the actual software
failure probability on the results for these nodes. The possible levels
(values) and their assessed probabilities for a given software project
are shown in the table below

**Table 3. Probability of Observing Attributes for a Given Failure
Probability Level**

![](media/image10.wmf){width="5.436805555555556in"
height="6.198611111111111in"}

The first row in the table shows the identified possible levels for the
software failure probability along with the prior assessment of their
probability based on past experience with this type of project. These
qualitative levels can also be assigned approximate numeric probability
ranges for further quantitative risk assessments such as in a PRA:

The remaining probabilities in a column in Table 3 are the assigned
probabilities for different possible findings when the software failure
probability is at a given level as identified at the top of the column.
They are thus diagnostic indicators.

As evidence is gained on the observed levels (values) of the factors,
the estimate of the software failure probability is updated. This allows
the value of the software failure probability to be tracked in
essentially real time. Given below is an example updated assessment of
the software failure probability after given audit findings and test
results.

**Table 4. Updated Probabilities for Different Possible Software
Levels**

![](media/image11.wmf){width="5.96875in" height="4.291666666666667in"}

The first row of the table shows the prior assessment that the software
failure probability as High, Medium-High, Medium-Low, or Low, before any
findings or results. Each subsequent row shows the updated assessments
for these possible levels after a specific finding or result. Each
subsequent row thus revises and updates the assessment based on the new
information obtained, as well as the previous information. The final row
shows the final assessment (final confidence) that the software failure
probability has achieved a given level. As observed there is very high
confidence that the software failure probability is Low or at least
Medium-Low. Using the numeric ranges for the levels , the probabilities
for the different levels can also be translated to a failure probability
distribution (histogram) for further quantitative risk assessment
applications if desired.

The results from the Bayesian net on this project application were
consistent with other assessments made using different tools. They also
served to provide additional, important information.. For example, there
is still a reasonable probability (27%) that the software failure
probability is Medium-Low instead of the desired Low. This is due to the
fact that the software testing criteria did not clearly differentiate
between these two categories but instead focused on the failure
probability not being Medium-High or High, i.e., did not have the
required power. Subsequently, additional tests were identified that
would provide this required differentiation. The results can furthermore
be used in a risk model such as a PRA to quantify the risk of the
mission failing. Additional work is being done to expand this approach
including producing user guidelines and data bases. As a final
postscript, it is interesting to note that the Bayesian framework
requiring probability assignments for the relationships was readily
accepted by project managers as being a natural approach for encoding
their knowledge and information

**Summary and Future Perspectives**

A spectrum of quantitative risk assessments are carried out at NASA.
This paper attempts to present samples of the different approaches and
applications that are involved. The samples presented are by no means
complete, but are chosen to represent the diverse challenges and
opportunities that exit. There are challenges in developing a more
structured framework for conducting the assessments and for implementing
the results. There are presently gaps between the risk assessment
community and the risk management community, not only in NASA, but in
the risk and management community at large. These need to be addressed.
The Space Shuttle PRA is an example of a large scale risk assessment on
an existing spacecraft. Approaches need to be developed to streamline
and focus PRAs for design evaluations for new spacecraft designs. To
assist in these applications, failure rate databases need to be
assembled, not only at the component level but at the subsystem and
system level. Risk-informed decision guidelines and frameworks need also
to be developed to assist in the use of quantitative risk results as
illustrated by the mean-value-based approach described in this paper. To
enhance project risk assessments, current qualitative risk assessments
need to be expanded to be more quantitative. The cumulative risk
assessments described in this paper is one step in this direction.
Finally, software quantitative risk assessments need to be developed and
be expanded. The Bayesian net example in this paper is one step in
quantifying the probability of software failure. There are thus many
challenges, and also many opportunities, for carrying out and
implementing effective quantitative risk assessments.

**References**

1\. NASA. July 12, 2004. *NASA Procedural Requirements*, *Probabilistic
Risk Assessment (PRA) Procedures for NASA Programs and Projects.* NPR
8705.5

2\. Stamatelatos, Michael, et al. April, 2003. *Probabilistic Risk
Assessment Procedures Guide for NASA Managers and Practitioners.*
Version 1.1.

3\. Vesely, W.E. 2004. "Extended Fault Modeling Used in the Space
Shuttle PRA" in *Proceedings of the Reliability and Maintainability
Symposium (RAMS).* IEEE Press.

4\. Vesely, W.E., et al. August 2002. *Fault Tree Handbook with
Aerospace Applications.* NASA Publication. Version 1.1.

5.Guarro, Sergio and Vesely W.E. 2004. "Risk Modeling" in Larry B.
Rainey, editor, *Space Modeling and Simulation.* Aerospace Press. El
Segundo, CA.

6\. Berger, James O. 1986. *Statistical Decision Theory and Bayesian
Analysis,* Second Edition. Springer-Verlag. New York.

7\. Bernardo, Jose M. and Smith, Adrian FM. 1994. *Bayesian Theory*,
Wiley and Sons. New York.

8\. Press, S. James. 2003. *Subjective and Objective Bayesian Modeling.*
Wiley Series in Probability and Statistics. New York.

9\. USRNC. August 1986. *Safety Goal Policy Statement*. Federal Register
51 30028.

11\. USNRC. February 2004. *Approach for Determining the Technical
Adequacy of Probabilistic Risk Assessment Results for Risk-Informed
Activities.* Regulatory Guide 1.2000.

12\. Simard, R. L. et al. April 2002. *Standard for Probabilistic Risk
Assessment.* ASME RA-S-2002..

13\. ASME. 2005. *Guidance for Risk Analysis and Management of Critical
Assets Protection*.

14\. Melchers, R.E., and Soares, C.Guedes, eds. December 2001.
*Reliability Engineering and System Safety: Special Issue on Risk
Assessment of Engineering Facilities*. Vol 74, No. 3.

15\. Moskios, Tom. March 3, 2005. *Risk Assessment Report for the NASA
DC-8 Transition from Dryden to the University of North Dakota*.

16\. Koller, Glenn. 2000. *Risk Modeling for Determining Value and
Decision Making.* Chapman and Hall/CRC Press. Boca Raton, FL.

17\. Jensen, Finn. 2001. *Bayesian Networks and Decision Graphs*.
Springer-Verlag. New York.

18\. Cowell, Robert, Dawid, A. Phillip, Lauritzen, Steffen, and
Spiegelhalter, David. 1999. *Probabilistic Networks and Expert Systems*.
Springer-Verlag. New York.

19\. Van Gurp, Jilles, and Bosch, Jan. December 1998. "Using Bayesian
Belief Networks in Assessing Software Architectures" in *Proceedings of
the 1999 IEEE Conference on Engineering of Computer Based Systems.* IEEE
Press.