**The Risk Analysis Correspondence Course**

This information is presented to Federal Aviation Administration (FAA)
engineers and other interested individuals for aid in the understanding
of the risk assessment process used by the FAA's Engine and Propeller
Directorate. It also has applicability to risk analysis in general.

Contents:

1.  Probabilistic failure modes

2.  Weibulls and other distributions

3.  Risk analysis techniques - Inputs and outputs

4.  Risk analysis techniques - How the simulation works

5.  Risk analysis techniques - Calibration

6.  Assumptions used in the risk analysis

7.  Prioritization of multiple safety problems

8.  Risk factor guidelines

9.  Miscellaneous topics

```{=html}
<!-- -->
```
1.  **Probabilistic failure modes**

In this first installment:

Introduction to failure modes

Hours vs. cycles for tracking failure probability

**Introduction to failure modes**

The probabilistic failure mode of a given piece of engine hardware is a
reflection of its relative probability of failure versus its age. There
are three categories of failure mode - wearout, infant mortality and
random. Wearout is the most common of the three - parts are increasingly
likely to fail as they age. Conversely, infant mortality describes those
situations where parts are more likely to fail early in their life.
Parts are usually considered safe from this mode once they pass a
certain age. Maintenance errors typically result in infant mortality
failures; for example, if a bearing was installed incorrectly, it would
be expected to fail within its first 100 hours of operation. After that
point, it is assumed that the bearing was correctly installed and will
not fail in normal operation.

In a random failure mode, parts or engines are equally likely to fail
whatever their age. For example, fan blade failures due to birdstrikes
are a random event, as birds don't selectively aim for older or newer
blades (and blade leading edge damage and polishing occurs fairly
continuously). Having a random failure situation allows for a simplified
risk assessment through the use of the Mean Time Between Failure (MTBF).
To calculate MTBF, divide the total cycles (or hours) by the number of
events. The future risk (the expected number of additional failures) is
estimated by multiplying the average number of cycles each part will be
expected to run until repair or replacement by the total number of
parts, and dividing by the MTBF. Using the birdstrike example, if we had
4 fan blade fractures due to birds over 2,000,000 cycles of operation,
we would have a 500,000-cycle MTBF (2,000,000 / 4). If we were to then
introduce a bird-proof blade, the future risk until that new blade was
completely incorporated can be calculated: for example, let's say we
have 1,000 engines which will run an additional 500 cycles, on average,
with the old blades. The expected number of additional failures until
all engines have been retrofit with the new blades is

1,000 x 500 / 500,000 = 1.0 additional failures

When multiple failure causes are combined (for example, all causes of
inflight shutdowns), the resultant distribution tends to be random due
to the mixture of wearout, infant mortality, and random modes.

The Weibull distribution (probability of failure versus age) is often
used to model failure data; for wearout problems, the Weibull
distribution will have a slope (beta) greater than 1. Typical values for
the slope are in the area of 3 to 5, but may vary. A Weibull
distribution with a slope equal or close to 1 reflects a random failure
mode, and a slope of less than 1 is indicative of infant mortality. A
more complete discussion of the Weibull and other distributions follows
in the next installment.

**Hours vs. cycles for tracking failure probability**

Parts installed in turbine engines are typically only highly stressed
during takeoff. The hours spent at cruise have little impact on their
life. Each flight, no matter how short or long, basically puts the same
amount of stress on the part. Therefore, the likelihood of failure is
best addressed as a function of the number of cycles on each part. There
are exceptions to this general rule:

Bearings

Oil system components

Fuel nozzles

Tubes, etc.

(In general, any parts that operate in a continuous and consistent
manner throughout the flight.)

In addition, failure causes such as high-cycle (high-frequency) fatigue
are typically a function of the time spent in the excitation area. If
this area occurs at a particular airspeed during descent, for example,
it may still be best to look at part cycles, since each part will spend
the same amount of time passing through that region during descent.
However, if the excitation occurs during cruise, hours would be
considered a better method to track the likelihood of failure.

**2. Weibulls and Other Distributions**

The **Weibull** distribution is commonly used in industry to model
failure data. The main advantages of the Weibull distribution are:

It can model a variety of data.

It handles suspensions (non-failure points) easily.

It provides a simple graphical solution and description of the data.

The Weibull distribution plots time or other such measure on the X axis
versus cumulative percent failing on the Y axis. The Weibull is defined
by two parameters: the shape parameter, called the slope or Beta, and
the characteristic life, Theta or Eta (it goes by either Greek letter,
depending on who you talk to). These parameters define the specific
distribution, as the mean and standard deviation define a Normal
distribution. Therefore, note that "-3 sigma" has no meaning against a
Weibull distribution. The correct term is Bx.x life, where x.x is the
percent of the distribution failing. A commonly used value for minimum
life from a Weibull is the B0.1, which is the 1/1000 failure life (0.1%
of the population).

As in any distribution, confidence limits can be calculated around the
parameters. In general, confidence limits are a measure of how different
the true distribution may reasonably be from the sample of data being
plotted. The true distribution is what the result would be if we knew
the failure time for every part in the population. In reality, we have
only a few failures (most of the parts have not failed), and less than
every part (whether failed or not).

The true distribution is just as likely to be better than the sample as
it is to be worse. However, it is usually the latter case that is of
interest - how bad might the situation reasonably be? Note that the
sample distribution as plotted (not the confidence limit) is usually
used to estimate future risk, for the reason that it represents the
average expectation and calibrates with the experience to date.

Confidence limits are a function of the amount of the data (and the
amount of variability in the population). The more data, the tighter the
confidence limits will be. Also, confidence limits tail off - i.e., the
true distribution is more likely to be closer to the sample plot line
than out at the limit.

Weibulls are often performed with limited failure information (after
all, we don't want to wait until there are lots of failures!) In fact,
Weibulls can be performed without any failures by assuming a slope and
assuming that one failure has occurred. This provides a picture of how
good the experience has proven so far. This type of Weibull is called a
Weibest or Weibayes. Weibests can also be used with multiple failures;
the difference between a Weibull and a Weibest is that the Weibull plots
the best fit through the actual failure data, whereas the Weibest plots
what the distribution should look like given the number of failures and
the times on all the parts (simplistically, it picks the most likely
failures rather than the actual failures). How close the Weibull plots
to the Weibest is a measure of whether the actual failures are outliers
or indicative of multiple populations in the data.

Take with a grain of salt any slope derived from a Weibull with just a
few failure points. In these cases, sometimes it's better to use an
assumed slope (Weibest) based on the knowledge of similar failure
situations.

The failure points should plot in a straight line, or near to it, on
Weibull probability paper. If they do not, either there are multiple
populations, or a Weibull distribution is not appropriate to describe
the data. An exception to this is the use of curved Weibulls (Curbulls)
to fit data which requires a minimum incubation time t(0).

For Weibulls with crack rather than failure data, the Weibull represents
a crack discovery distribution rather than a pure initiation
distribution. In other words, the part is inspected at some age and
found to be cracked. The actual crack initiation occurred at some point
prior to the inspection. Sometimes fracture mechanics is used to back up
the crack to estimated initiation.

Instantaneous failure probability is the probability that the part will
fail in the next cycle given that it hasn't failed yet. The equation is
the probability of failure in the next cycle minus the probability of
failure in the current cycle divided by 1 minus the probability of
failure in the current cycle.

\[P(t+1) - P(t)\] / \[1 - P(t)\]

Other distributions are sometimes used to model failure data. For
example, the log-normal distribution is used extensively by Airbus -
they prefer it to the Weibull. A log-normal distribution converts
failure data to a normal distribution by taking the log of each point
and plotting the transformed data. No particular distribution is
inherently better than the other - it simply a question of which best
fits the data. Often, adequate results can be achieved with several
different distributions, especially those in the same general family
(such as Weibulls, log-normals, exponentials, etc. Note that a Weibull
collapses to an exponential distribution when the Weibull slope is equal
to 1 - a random failure mode, you'll remember from the earlier
discussion.) A non-transformed normal distribution is usually not a good
descriptor of failure data. Normal data is evenly distributed about the
average value, whereas failure data tends to extend much higher above
the average value than it does below. A Weibull can be used to model
normally-distributed data; the slope comes out to be 3.44.

Other distributions of note include the binomial and the Poisson. A
binomial distribution models an either/or situation - for example, a
coin toss (heads/tails). It has applicability in manufacturing
situations as well (defective/not defective). It is important to
remember that either/or does not necessarily imply a 50-50 chance of the
two different outcomes!

The Poisson distribution is used to model such types of data as the
number of defects or inclusions in a part, the number of accidents in a
year, the number of cars passing through a tollbooth every minute, etc.
The two important factors are: the possible number of occurrences is a
great deal larger than the average, and the events are independent of
each other. The Poisson distribution has applicability to the risk
analysis process in that it is used to calculate the probability of 0,
1, 2, or more additional events for a predicted average outcome (called
the **risk factor** in the risk analysis process). For example, a risk
factor of 0.5 predicted additional events translates to a 61%
probability of no events, a 30% chance of 1 (and only 1) event, an 8%
chance of 2 events, a 1% chance of 3 events, etc.

For that matter, I realize that I should define "risk analysis".
Therefore, a couple of definitions:

Risk analysis is the qualitative or quantitative analysis of information
to establish an expected loss from events based on their estimated
probabilities of occurrence.

The risk analysis procedure used within the Engine and Propeller
Directorate is a numeric method based on computer simulations which
quantify the expected number of future events (risk factor) for specific
problems as a function of specific operational and maintenance
constraints.

**3. Risk analysis techniques**

In this installment:

> An overview of risk analysis
>
> Inputs and outputs

**Overview**

As was defined last week, the risk analysis procedure used within the
Engine and Propeller Directorate is a numeric method based on computer
simulations which quantify the expected number of future events (risk
factor) for specific problems as a function of specific operational and
maintenance constraints.

Risk analyses are performed to control risk while optimizing the
inspection and modification programs aimed at addressing unsafe
conditions which are uncovered during service. Their aim is to achieve
the desired risk factor goal (more about that in future installments)
while minimizing operational impact. The analysis will also output the
number of spare parts, shop visits and inspections which will be
required, allowing for timely provisioning.

The centerpiece of the risk analysis is the simulation model. This
simulation builds an computerized model of the field. Once the
simulation has been developed, various future inspection, repair, and
replacement scenarios can be analyzed to determine their effects on the
simulated fleet. Essentially, the program can run the simulated fleet
through many possible "futures". The different "futures" evaluated often
include the one in which no action is taken, to establish a standard for
comparison.

**Inputs and Outputs**

Obviously, we desire to have the best possible correlation between the
actual fleet and the simulation. Various input distributions are
required to help define the simulation model. These include:

-   Part time/cycles - the current age of the affected fleet

-   Crack initiation and propagation - Weibull or similar distributions

-   Shop visit distribution - based on recent experience with the
    applicable fleet

-   Inspection reliability - crack POD (probability of detection)

-   Any changes to the above based on new operational or maintenance
    procedures

These inputs should represent the sum of knowledge about the safety
problem:

-   Realistic assessment of known conditions (based on service
    experience, test results, etc.)

-   Conservative assessment of unknowns

-   Engineering judgment

The main question that the risk analysis answers is this: based on the
current age of my parts, and the intervals at which I plan to inspect,
repair, and/or replace them, how many more events are expected until the
program is complete? Note that I used the term "events" rather than
"failures." One of the other inputs to the risk analysis is the
definition of an "event." For example, consider the case of a fan blade
fracture. The "event" could be defined as:

-   blade fracture

-   uncontained failure (a subset of all fractures)

-   uncontained failure which causes damage to the aircraft (a subset of
    uncontainments)

Actually, the risk analysis is usually set up to output the base "event"
(in this case, fractures), which is then corrected for the probability
of uncontainment, etc. So, the outputs can include the expected number
of fractures (the risk factor for fractures), uncontainments (the risk
factor for uncontainments), and so on.

The risk factor(s) can be calculated versus calendar time. This enables
us to graphically plot the cumulative risk against time. Most of the
risk of additional events during any correction program usually occurs
early on, before many parts have been inspected; the plot of cumulative
risk vs. time thus shows an asymptotic approach to the total risk factor
for the program. It is often of interest to plot the risk factor if no
action is taken along with the planned or possible
inspection/modification scenario(s).

Additional outputs from the risk analysis are possible, including:

-   the number of cracks found

-   the number of replacement parts required

-   the number of inspections

These can also be listed on a month-by-month basis; thus, the risk
analysis can establish how many parts will be needed and when they will
be needed. The predicted number of cracks found vs. time is especially
important, as it allows us to track whether the results in the actual
future corresponded to the predicted results of the simulated future.

**4. Risk analysis techniques (continued)**

In this installment:

> How the simulation works

**How the simulation works**

Basically, there are two techniques to use in arriving at the risk
factor - calculating it directly, or using Monte Carlo techniques. The
first involves calculating the instantaneous probability of failure on a
cycle-by-cycle basis for each part until it is inspected or replaced,
and adding up all the answers to get the total expected number of
failures. (Remember that the instantaneous failure probability was
defined in installment 2 as \[P(t+1) - P(t)\] / \[1 - P(t)\]. ) The
probability of missing a crack must also be factored into the equation,
along with inspection and replacement intervals and all the other input
variables. You can see how this might get a little complicated,
especially if you want the results on a month-by-month basis.

The other method uses many iterations (repeat analyses within the
computer program) to simulate the behavior of the different parts and
sums the results of each of those iterations. The behavior of a
particular part during a particular iteration is randomly selected from
the distributions associated with the different inputs. This process is
called Monte Carlo simulation (after the gambling resort). Monte Carlo
simulation assigns random numbers to the outcomes of a statistical
distribution. The probability of a given value is determined by the
random numbers assigned to that value. The random numbers are assigned
based on the percent of the population failing vs. time. Let me provide
an example from a Weibull distribution of crack initiation, which we
want to be able to simulate using the Monte Carlo technique. Remember
that the Bx.x life is the age at which x.x% of the population will fail
(or, in this case, initiate a crack).

[Weibull distribution]{.underline} [Random number]{.underline}

B0.01 initiation life 2,000 cycles 0.000

B0.1 3,000 cycles 0.001

B1.0 7,000 cycles 0.010

B6.0 11,000 cycles 0.060

etc.

B99.9 35,000 cycles 0.999

B99.99 50,000 cycles 1.000

Now, let's see what this means. The program uses a random number
generator to randomly select a number between 0 and 1. Let's say our
random number for part #1 in iteration #1 is 0.045. The program looks at
the distribution we've input, as above, sees that 0.045 falls between
7,000 and 11,000 cycles initiation life, and interpolates between those
two lives with the same ratio as between their respective random
numbers - in this case,

\[(0.045 - 0.010) / (0.060 - 0.010)\] = \[( X - 7000) / (11000 - 7000)\]

Solving for X, we get an initiation life of 9,800 cycles.

The set of corresponding lives and random number probabilities is
sometimes called pairwise points.

This process is repeated to calculate an initiation life for each part
in the fleet. What we end up with is a distribution of initiation lives.
A similar process is performed for crack propagation, age at inspection,
etc., all based on their respective distributions. For each individual
part, we end up with an assumed initiation life, propagation life,
inspection interval(s), and replacement age. The question of whether
that particular part fails is then simply a matter of looking at whether
it is replaced, or its crack is found, prior to the sum of its
initiation and propagation. For example:

Part #1, currently 7,000 cycles old

Crack initiation life 9,800 cycles

Propagation life 2,500 cycles (so, part fractures at 12,300 cycles)

On-wing inspections: 8,000 cycles - no crack found (none there yet)

10,115 cycles - crack not found (based on inspection

reliability distribution)

Shop visit at 12,000 cycles; part replaced - therefore, part does not
fail, as it is scrapped prior to the point at which it fractures.

Part #2, currently 11,254 cycles old

Crack initiation life 7,714 cycles

Propagation life 4,400 cycles (part fractures at 12,114 cycles)

On-wing inspections: 13,000 cycles

Part fails prior to first inspection.

Part #3, currently 6,050 cycles old

Crack initiation life 5,507 cycles

Propagation life 4,216 cycles (part fractures at 9,723 cycles)

On-wing inspections: 9,013 cycles - crack found

Engine removed - part does not fail.

The number of failures is summed for each iteration. In the case above,
we had only 3 parts, of which 1 failed. In reality, of course, we have
many more parts. The process is the same - simulate the experience for
each part, and count how many fail. The process is iterated many times
(100 or more) to ensure that the simulated distributions correspond to
the actual distributions (remember how we discussed sample size vs. the
true distribution back in the confidence limits topic?) The average
number of failures is then obtained by dividing the total number of
failures for all iterations by the number of iterations. This gives us
the risk factor.

For example: 1000 parts - 100 iterations

40 iterations had 0 failures

53 iterations had 1 failure

6 iterations had 2 failures

1 iteration had 3 failures

total number of failures = (40 x 0) + (53 x 1) + (6 x 2) + (1 x 3) = 68

average number of failures = 68 / 100 iterations = 0.68

The risk factor for the inspection/modification scenario used in this
simulation is thus 0.68

You can see how we can also arrive at the number of cracks, inspections,
etc.

**5. Risk analysis techniques (continued)**

In this installment:

Calibration

Summary: Steps in the risk analysis process

**Calibration**

Before a risk analysis can be used to predict future behavior, it must
be calibrated against the experience to date. This means, if we back the
simulated fleet back up in time, and let the program run forward to the
present, does it give the same results as have occurred in real life?
This should hold true for both failures (fractures, or whatever the
relevant event) as well as cracks (if applicable). If the simulated
results do not match reality within a reasonable amount, the assumptions
and distributions which have been input into the program must be
examined and revised until calibration occurs. Remember that the
simulation may predict that, say, 2.4 failures have occurred to date.
Since we cannot have .4 of a failure, this result is reasonable for
either 2 or 3 actual failures. Depending on the degree of confidence in
the assumptions, we may wish to revise them until the simulation
predicts exactly 2.0 (or 3.0, whichever applies) failures.

Another important aspect of calibration looks at the predicted and
actual results versus calendar time. If the simulation predicts 2
failures, and we have had 2 in reality, does the timing match? In other
words, if the reality is 6 years of successful operation and 2 failures
within the past year, the simulation program's 2 failures should be
within the same time frame.

As was stated at the beginning of this section, calibration of the risk
analysis simulation is vitally important. Unless we have a program that
predicts the experience to date, we cannot have any confidence in its
ability to predict fleet behavior in the future.

**Summary: Steps in the risk analysis process**

Define event of interest

Establish upper limit for risk factor (more on this in later
installments)

Define simulation model characteristics

Define relevant distributions

-   Part life (initiation/propagation)

-   Shop visits

Define operational and maintenance constraints (what can and can't be
done)

Create computer model

Define inspection/modification scenarios

Calibrate model

-   Should predict experience to date

Output information

-   Risk factor

-   Sensitivity analyses

-   Parts requirements

**6. Assumptions used in the risk analysis**

In this installment:

Assumptions and validations

Sensitivity analyses

**Assumptions and validations**

Many assumptions are made in the process of performing a risk analysis.
It is important that all concerned parties buy in to those assumptions.
Therefore, any presentation of results from a risk analysis should begin
with a thorough review of the assumptions made, and what supporting
evidence exists to support them. Each of the assumptions may be based on
data, engineering judgment, or both. Where data are lacking, some
element of conservatism is called for.

A word here about engineering judgment and data. First and foremost, it
is not a case of either/or. Engineering judgment is informed opinion. It
is based on a knowledge of the parts, processes and physics involved, as
well as the ability to relate new situations to experience. Engineering
judgment should reflect the consensus view among the knowledgeable
experts. Data is the engine's judgment - and the engine (or aircraft) is
usually the best expert of all! The engineer should always be interested
in what the engine has to say, and develop his or her informed opinion
in light of that evidence. Conversely, engineering judgment is often
required to interpret the data.

That being said, recognize that some assumptions in a risk analysis will
by definition rely more heavily on judgment than others. Other
assumptions are relatively straightforward. For example, the OEMs
(Original Equipment Manufacturers) have very good data on shop visit and
utilization rates. The only things to be concerned about when reviewing
these distributions are whether they're based on recent data for the
proper models and whether there are any major operational or maintenance
changes forecast that would affect either usage or shop visit
scheduling.

Likewise, initiation and propagation distributions are usually very
heavily data-based, relying on a combination of test data and field
experience. The question always arises as to whether there are "enough
data" to perform an analysis. The answer to this is to recognize that
the available data are never perfect, but represent the best estimate of
the situation as it is currently known. The presence of incomplete data
should not be seen as justification to ignore the data (see paragraph 2,
above). Furthermore, the necessity of calibrating the risk analysis with
the past experience helps to ensure that the future prediction is
reasonable, assuming that operational parameters (derate schedules,
etc.) remain constant. Remember that an important consideration is
whether the failure mode is a function of cycles or hours.

The assumed availability of spare or replacement parts can be validated
by the raw material and finished part production schedules. Future fleet
growth, if involved, is determined by orders and projections.

An important assumption that is often not readily apparent is the extent
of the affected population. What is the justification for excluding
certain models? Has an analysis of the failures versus production date
been performed to identify possible manufacturing-related problems? Are
the failures limited to a particular segment of the fleet or type of
operation? Bear in mind that a statistically-significant difference in
the rate of occurrence between two different populations does not
necessarily mean that the better population will never have the problem,
only that it is not as prevalent. This is a prime example of the use of
engineering judgment to interpret results.

**Inspection reliability**

Inspection reliability is an important assumption. Usually, the only
data available on the probability of crack or defect detection (POD)
come from the laboratory. Judgment must be applied to translate those
carefully-controlled results to what is achievable in practice. An
on-wing inspection is probably inherently not as reliable as an in-shop
inspection due to accessibility and environmental factors. (Of course,
the on-wing inspection has the benefits of convenience and
availability.) For situations where the POD does not vary significantly
versus crack size, a single value for POD may be used. In other cases,
the POD curve (probability of detection vs. crack length or defect size)
is input as a series of pairwise points, as in the initiation and
propagation distributions. Another factor in the assumed inspection
reliability is whether multiple cracking or defect sites exist.
Everything else being equal, a part with two cracks is more likely to
fail the inspection than a part with only one crack.

There may also be human factor considerations. For example, if many
parts are expected to be cracked, the inspection reliability is probably
better than if only one or two parts in the entire fleet are cracked.
The reason for this (and it can be considered a personal opinion) is one
of expectations on the part of the inspector. If cracks are regularly
found, the inspector may feel an increased level of vigilance. On the
other hand, if many inspections are performed without rejects, the
expectation may develop that the next part won't be cracked either.
Needless to say, this is nearly impossible to quantify, but is something
to factor into the judgment aspect of the situation.

The use of dual inspections to increase overall inspection reliability
is a source of contention. If two inspections are truly independent, a
single 90% inspection reliability translates to a dual 99% reliability
(.10 x .10 chance of missing the crack twice). Once again, however,
practice does not quite match theory. It is generally agreed that there
is some improvement in the POD rate, but not the amount predicted by
true independence. The justification of some improvement is the idea
that whatever caused the crack to be missed the first time will not
happen precisely the same way the second time. The justification for not
taking the entire amount of predicted improvement is twofold: endemic
problems associated with the inspection technique or process (the setup
is inadequate, the part misses the inspection process entirely, etc.)
and the possibility that some cracks, by nature of their geometry, may
be inherently less detectable than others. Whenever a dual inspection is
called for, it is important that the second inspector (note the
requirement for two different inspectors) does not have knowledge of the
results of the first inspection. This lack of foreknowledge helps to
improve the independence of the inspections.

**Sensitivity analyses**

A sensitivity analysis involves running the risk analysis with a range
of possible inputs for a given parameter to determine the sensitivity of
the results to that parameter. An obvious use is to evaluate inspection
reliability. If there is disagreement as to the expected reliability,
the analysis can be performed with different PODs to determine the
effect of reduced inspection reliability on the projected risk factor
(remember that risk factor is the expected number of future events).

**Monitoring**

On-going field experience should be monitored to continue validation of
the assumptions. For example, if cracks are projected to be found, but
are not, the problem may be with the inspection reliability (i.e., it's
worse than assumed) or with the initiation distribution. The finding of
any significant shortfalls in the assumptions calls for revision to the
risk analysis.

**7. Prioritization of multiple safety problems (the Continued
Airworthiness Assessment Methodologies process)**

In this installment:

The need for prioritization

CAAM hazard levels

Calculation of the hazard ratio

**The need for prioritization**

Prioritization is the recognition that we cannot solve multiple problems
simultaneously due to the lack of infinite resources (of people,
materials, shop capacity, etc.) Therefore, it is necessary to have a
risk management system which appropriately assigns prioritizes to the
variety of problems we face. The goal of establishing these priorities
is to ensure that the available resources of all concerned parties - the
FAA, the OEMs, and the operators - are applied to the areas of greatest
safety threat.

This prioritization process by definition requires an assessment of an
individual engine problem or event against the concept of "safety." At
the FAA, we usually address safety by managing its converse, the unsafe
condition. An unsafe condition may be defined with various specific
words, but, in general, it refers to any problem which has the potential
to cause harm. A specific definition for the engine world might be: Any
engine, propeller or APU malfunction, defect or failure which can
directly or indirectly hazard the aircraft, its passengers and crew, or
both.

**CAAM hazard levels**

We see, then, that we need an understanding of whether and how a
particular problem might cause an unsafe condition, and how extensive
this potential for harm. To this end, a standardized list of aircraft
situations was developed to relate a particular result to a quantified
level of harm. This list was developed by the joint FAA-industry
committee known as the Continued Airworthiness Assessment Methodologies
(CAAM) committee; the Engine and Propeller Directorate adopted the list
as part of its risk assessment and prioritization process (also known as
CAAM). The quantified level of harm is called the Hazard Level, and
ranges from 1 through 4 (since expanded to 5 from the TAD-EPD-ARAC
coordination activity, though only levels 3 and higher have been
coordinated with TAD). The list of aircraft situations comprising those
hazard levels is as follows:

**CAAM Hazard Levels**

LEVEL 1 - MINOR CONSEQUENCES

a\. Uncontained nacelle damage confined to affected nacelle/APU area.

b\. Uncommanded power increase, or decrease, at an airspeed above V1 and
occurring at an altitude below 3,000 feet (includes inflight shutdowns
(IFSD) below 3,000 feet).

c\. Multiple propulsion system malfunctions or related events, temporary
in nature, where normal functioning is restored on all propulsion
systems and the propulsion systems function normally for the rest of the
flight.

d\. Separation of propeller/components which causes no other damage.

e\. Uncommanded propeller feather.

f\. Excess loads.

LEVEL 2 - SIGNIFICANT CONSEQUENCES

a\. Nicks, dents and small penetrations in aircraft primary structure.

b\. Slow depressurization.

c\. Controlled fires (i.e., extinguished by on-board aircraft systems).

d\. Fuel leaks in a fire zone or in the presence of an ignition source,
which exceed the drainage capability of the compartment.

e\. Minor injuries.

f\. Multiple propulsion system/APU malfunctions, or related events,
where one engine remains shutdown but continued safe flight at an
altitude 1,000 feet above terrain along the intended route is possible.

g\. High speed takeoff abort (usually 100 knots or greater).

h\. Separation of propulsion system, inlet, reverser blocker door,
translating sleeve inflight without significant aircraft damage.

i\. Partial inflight reverser deployment or propeller pitch change
malfunction(s) which does not result in loss of aircraft control or
damage to aircraft primary structure.

j\. Malfunctions or failures that result in smoke or toxic fumes,
delivered through the ECS system, that cause minor impairment or minor
injuries to crew and/or passengers.

LEVEL 3 - SERIOUS CONSEQUENCES

a\. Substantial damage to the aircraft or second unrelated system.

b\. Uncontrolled fires.

c\. Rapid depressurization of the cabin.

d\. Permanent loss of thrust or power greater than one propulsion
system.

e\. Temporary or permanent inability to climb and fly 1000 feet above
terrain.

f\. Any temporary or permanent impairment of aircraft controllability.

g\. Malfunctions or failures that result in smoke or other fumes,
delivered through the ECS system, that result in a serious impairment.

LEVEL 4 - SEVERE CONSEQUENCES

a\. Forced landing.

b\. Loss of aircraft (hull loss).

c\. Serious injuries or fatalities.

LEVEL 5 - CATASTROPHIC CONSEQUENCES

Catastrophic outcome (reference Catastrophe as defined by draft AC
25.1309-B) - an occurrence resulting in multiple fatalities, usually
with the loss of the airplane.

Levels 3, 4 and 5 represent the greatest area of safety concern. CAAM
addresses this special concern by establishing the hazard ratio, or the
conditional probability of serious or severe consequences (i.e., hazard
level 3, 4 or 5) given that a particular engine, propeller, or APU event
with at least some safety significance (at least level 1) occurs. The
paragraphs below describe how to calculate the hazard ratio. First, let
us return to the issue of prioritization.

Prioritization assumes that when we are faced with multiple problems, we
devote the most attention and resources to those problems most likely to
cause serious harm. Within the CAAM process, the prioritization is
therefore undertaken against the risk of level 3 and higher events
(assume from now on that "level 3" means "level 3 and higher"). We
previously discussed how the risk analysis could output a risk factor
for the basic event (fracture, etc.), for uncontainments, and for other
categories of outcomes. We can see now that we are probably most
interested in the risk factor for level 3s. A problem with a high
uncorrected level 3 risk factor ("uncorrected" being the risk factor if
we were to let the situation continue without any corrective action)
should receive a much higher portion of resources than a problem with
low risk of progressing to a level 3 event. This allocation of resources
means that we allow more time for the low-risk problem to be corrected
compared to the high-risk problem. It does not mean we ignore the
low-risk problem, only that we recognize that if we force a quick
response to a problem with minor consequences, we may not have adequate
resources available to react quickly to the next, more serious problem.

**Calculation of the hazard ratio**

Calculating the hazard ratio will require considerable engineering
judgment. Since the hazard ratio is a strong influence on the
prioritization process, it should either be based on validated data or
be assessed conservatively. To this end, we can consider it another
assumption within the risk analysis process. Specific data are often not
available; historical hazard ratios, developed from past events on
similar engine types, should be used cautiously. The hazard ratio is
dependent on the installation (e.g., wing-mounted vs. tail-mounted
engines), and the historical data may be skewed by the amount of
information available for the affected aircraft installation. The
following three methods of calculating the hazard ratio are offered,
depending on whether specific data exist or not:

[1. At least one level 3 event has occurred]{.underline}. Use the value
obtained by dividing the number of level 3 events by the total number of
safety events (i.e., at least a hazard level 1). For example, two level
3 events out of four total would be 2 / 4 = 0.50 hazard ratio. If the
latest event used in the calculation was not level 3, add one additional
level 3 event, and one additional safety event, to the totals to cover
the possibility that the next event would be level 3. E.g., one level 3
out of four total events (1 / 4) becomes two out of five (2 / 5 = 0.40).

[2. No level 3 or higher event has occurred]{.underline}. When no level
3 event has occurred, historical hazard ratios may be used, with care
taken to ensure proper methods of comparison, including: similar
aircraft installation, engine bypass ratio, and any other factors of
possible significance.

[3. No level 3 or higher event has occurred and no historical data are
suitable]{.underline}. Where no level 3 event has occurred and no
historical data are available or suitable, use the method in (1.) above
by assuming the next event would be level 3 (e.g., 0 / 4 becomes 1 / 5 =
0.20). There may be cases where this method is overly conservative. In
those instances, engineering analysis and/or coordination with the
installer and operator(s) may be used to establish a more realistic
hazard ratio.

It may also be desirable to establish the conditional probability of a
level 4 event given that a level 3 event has occurred. Typically,
coordination with the installer and operator(s) is necessary to
establish a level 4 hazard ratio.

The information in this installment is included in the CAAM Advisory
Circular, AC39-8. This AC also contains data for use in calculating
historical hazard ratios for various types of engine, propeller, and APU
events for the period 1992-2000; earlier data are contained in the FAA's
Technical Report on Propulsion System and Auxiliary Power Unit (APU)
Related Aircraft Safety Hazards, dated October 25, 1999.

**8. Risk factor guidelines**

In this installment:

Guidelines for allowable risk factors

Per-flight risk

Cumulative risk

**Risk factor guidelines**

The guidelines described below represent the short-term risk that can be
reasonably allowed during the time period required to correct an unsafe
condition. These guidelines are not targets or typical values; the risk
factor should normally be lower than these guidelines unless a lower
value would result in extreme resource difficulties. The goal of risk
analysis is not to find the most lenient program that still squeaks
under the risk factor upper limit. Any reasonable action which reduces
the risk should be included as part of the correction program (keeping
in mind the principles of prioritization discussed in the previous
installment).

On the other hand, zero risk is unattainable without grounding the
fleet. Also, the plot of risk factor versus impact on resources reveals
an asymptotic relationship; in other words, at some point, any
additional reduction in risk factor comes only at great increase in the
required resources. That particular point varies from situation to
situation. The engineer must decide if the additional burden on the
fleet is worth the reduction in risk.

The risk factor for level 3 events during the correction action period
should not exceed 1.0 except in rare cases. Remember that the level 3
events means those at least level 3 (i.e., level 3, 4 and 5 events). The
risk factor for level 4 events (i.e., level 4 and 5 events) should not
exceed 0.1 for the correction action period. In most instances, however,
level 4 events should be managed to a much lower risk factor to minimize
the cumulative effects of multiple unsafe conditions on overall risk.
There is no guideline as yet for level 5 events.

**Per-flight risk**

Risk factor guidelines apply to the total number of expected events.
However, we also need guidelines for per flight risk. This requirement
recognizes that we must limit the additional risk posed by any one
unsafe condition to any one plane during any one flight. The average
risk per flight is calculated by dividing the risk factor by the total
engine cycles (or hours, as the case may be), corrected for the number
of engines per plane. For example, a risk factor of 0.5 for a correction
period totaling 300,000 engine cycles for a twin-engine aircraft fleet
equates to an average per-flight risk of \[(0.5 / 300,000) x 2\] =
3.3x10-6.

The per-flight guidelines for short-term risk are less than one level 3
event in 25,000 flights (4x10-5) and one level 4 event in 250,000
flights (4x10-6).

Once the problem has been corrected, the long-term average per-flight
risk should not exceed one level 3 event per 100 million flights
(1x10-8). The long-term level 4 guideline is less than one level 4 event
per billion flights (1x10-9).

**Cumulative risk**

Back in installment 2, we discussed the Poisson distribution and how we
use it to relate risk factor to the probability that more events will
occur. Looking at the risk factor guidelines above, we find that a level
3 risk factor of 1.0 equates to a 70% probability that at least one
level 3 event will occur during the correction program. A level 4 risk
factor of 0.1 equates to a 10% probability that at least one level 4
event would occur. You can see why we want these guidelines to be
treated as upper limits rather than as typical values; the cumulative
effects of multiple unsafe conditions over the life of the fleet becomes
a consideration. Allowing all problems to reach the upper limit could
result in an undesirably high cumulative risk. If, for example, during
the life of a fleet, there are seven different problems that are all
allowed to reach a level 4 risk factor of 0.1 events, we would have a
cumulative risk factor of 0.7 level 4 events. This equates to a 50%
probability of at least one level 4 event, and a 16 percent probability
of two or more over the life of the fleet. While there are currently no
guidelines for cumulative risk, expecting to have a level 4 event at
some point is not a desirable situation.

**9. Miscellaneous risk analysis topics**

In this installment:

The requirement for a consistent set of ground rules

Dual-event risk

Why limiting conditions aren't enough

Specimen tests versus part tests

**The requirement for a consistent set of ground rules**

Prioritization requires comparing risk analysis results from multiple
problems to ensure that they are being managed appropriately. A
consistent set of ground rules for constructing these quantitative
assessments is necessary to ensure valid comparisons, and that the risk
factor guidelines are being properly applied. Examples of areas where
consistent ground rules are necessary include: the determination of
flight exposures; event hazard levels; hazard ratios; and per-flight
risk. There may be differences in the ground rules used by different
manufacturers in performing quantitative assessments; it is therefore
important to refrain from directly comparing results from different
manufacturers unless it can be verified that the analyses were performed
using the same ground rules.

**Dual-event risk**

Most of the unsafe conditions we've discussed have arisen from a serious
event, or sequence of events, on a single engine. However, we sometimes
are faced with the situation of having benign single-engine failures
(not in and of themselves unsafe) occur at a rate that raises concern
over the risk of multiple engine failures within the same flight. The
single-engine rate is used to estimate the multiple-engine risk. For all
practical purposes, the risk does not go beyond dual-engine events for
non-common cause problems. And, of course, common-cause problems - fuel
contamination, runway ice ingestion, incorrect maintenance on multiple
engines, etc. - are really single-event unsafe conditions affecting
multiple engines.

The seriousness of the dual-engine event should be evaluated against the
CAAM hazard levels. CAAM hazard level 3 includes "permanent loss of
thrust or power greater than one propulsion system." In addition, the
loss of enough engine thrust to force a landing is an automatic CAAM
level 4 event. For twin-engine aircraft, dual inflight shutdowns (if the
engines cannot be restarted) are thus level 4 situations. Engine
failures which result in the loss of the ability to produce sufficient
thrust, even if the engines were not actually shutdown, must be
considered in the analysis. For example, suppose we had blade fractures
occurring in the high-pressure compressor. Four fractures resulted in
inflight shutdowns (IFSDs), and, in two other cases, the engines were
retarded to idle but not shutdown. The factor to consider when
calculating the appropriate dual-engine risk is whether the failure
condition (HPC blade fracture) renders an engine incapable of producing
any significant amount of thrust. If so, not only are all the IFSDs to
be counted, but the non-IFSD events (operational discrepancies) must be
counted as well, since the engines could not produce thrust if called
upon.

For random failure modes, or for fleets of aircraft with engines of
differing ages (it is a fairly common practice for airlines to stagger
their engines so that, on a given airplane, all engines are not of the
same total hours or cycles), the dual-engine (d.e.) risk can be simply
calculated from the single-engine (s.e.) rate, corrected for the number
of total engines:

for a two-engine plane d.e. risk = (s.e. rate) x (s.e. rate)

for example, s.e. rate = 1x10^-4^ per engine cycle

> d.e. risk = (1x10^-4^ x 1X10^-4^) = 1X10^-8^ per airplane flight

for a three-engine plane d.e. risk = 3 x \[(s.e. rate) x (s.e. rate)\]

the factor of 3 is needed because there are 3 ways to fail 2 engines

on a three-engine plane (engines 1 and 2, 1 and 3,

> or 2 and 3)

for example, d.e. risk = 3 x (1x10^-4^ x 1X10^-4^) = 3X10^-8^

for a four-engine plane d.e. risk = 6 x \[(s.e. rate) x (s.e. rate)\]

the factor of 6 is needed because there are 6 ways to fail 2 engines

on a four-engine plane (1 and 2, 1 and 3, 1 and 4, 2 and 3,

2 and 4, or 3 and 4)

for example, d.e. risk = 6 x (1x10^-4^ x 1X10^-4^) = 6X10^-8^

Note how the risk is automatically higher on planes with more engines;
however, permanently losing two engines is not automatically a level 4
(depending on the phase of flight) for a three- or four-engine aircraft.

For wearout or infant mortality probabilistic failure modes, with
unstaggered engines, the risk must be calculated from the instantaneous
failure probability for the given age of the engines or parts. In these
cases, a summary table is usually produced to show the dual-engine risk
versus the age (cycles or hours, as appropriate) of the engines.
Resources can then be diverted to addressing the population of aircraft
with engines at the greatest risk.

**Why limiting conditions aren't enough**

Occasionally, we are faced with unsafe conditions where a variety of
different parts may fail from the same basic event. For example, an
engine overspeed can cause any of the disks to burst, or excessive loads
can cause the engine mounts, pylon, or other structures to break. In
these situations, an analysis may be presented against the "limiting
condition" - the disk under highest stress, or the structure most likely
to break. However, the aim of a risk analysis is to evaluate total risk.
The limiting condition may represent the part most likely to fail, but
it isn't the only part that *could* fail. Remember in the risk analysis
process that we discussed how material properties (such as crack
initiation and propagation) were simulated for each part. For any given
engine, the limiting condition part may be actually be stronger than
another part because of the difference in their respective material
properties. The additional failure probabilities for each of the other
parts, even though they amount to lower risk than the limiting condition
part, must be considered to properly estimate the total risk exposure
from the basic event (overspeed, etc.) The exception to this would be if
the risk analysis assumed minimum material properties (rather than
distributions) for the limiting condition.

**Specimen tests versus part tests**

To help define part failure distributions, material tests in a
laboratory environment are often performed. Due to the costs involved,
these tests are usually run with specimens rather than actual parts. The
aim is to make these specimen tests as close to actual engine operation
as is possible. Any available field data on cracking should be used to
validate the testing.

In addition, if multiple parts are at risk in the engine (for example,
46 blades in a rotor) but only single blades or specimens are tested in
the laboratory, a correction must be made to the test failure
distribution to account for the multiple parts in an engine. For
example, testing might show that 1 out of 100 blades fails within 5,000
cycles. However, roughly one out of every two engines will have a 1/100
blade somewhere in its rotor of 46 blades. If we have field experience,
this effect sorts itself out - the engine fails when the weakest blade
fails.

This concludes the discussion of the FAA Engine and Propeller
Directorate's risk analysis process. I hope the material has been
useful. Please address any comments or questions to:

Ann Azevedo

Risk Analysis Specialist

781-238-7117

ann.azevedo@faa.gov