# Presentation: 409452

## An HFACS Analysis of General Aviation Data 1990-98: Implications for Training

**An HFACS Analysis of General Aviation Data 1990-98: Implications for Training**

**Scott Shappell, Ph.D.**
- Civil Aeromedical Institute

**Douglas Wiegmann, Ph.D.**
- University of Illinois

**Notes:**
AN HFACS ANALYSIS OF GENERAL AVIATION DATA 1990-98: IMPLICATIONS FOR TRAINING
S. Shappell1* and D. Wiegmann2*
Civil Aerospace Medical Institute1, University of Illinois at Urbana-Champaign2
Purpose. A large effort has been expended during the last several decades to lower the military and commercial aviation accident rates. However, the total number of accidents is considerably fewer that those incurred within general aviation (GA). In an effort to better understand the causal genesis of these accidents and as part of the Safer Skies effort within the FAA and NASA, we previously analyzed nine years (1990-98) of fatal GA accidents using the Human Factors Analysis and Classification System (HFACS). The findings, though significant, reflected only about 20% of the total GA accidents that occurred during the time frame of the study. Therefore, an analysis of the remaining non-fatal GA accidents was conducted to provide a more complete picture of the human factors associated with GA accidents. Methods. Using HFACS, five pilots independently coded the cause factors associated with 15,744 GA accidents occurring between 1990-98.  Of these, 12,033 accidents were associated with some form of aircrew unsafe act (i.e., decision error, skill-based error, perceptual error, and/or violation). Our findings with regard to these aircrew-related accidents will be reported here. Results. With the exception of violations that were more common during fatal (35%) than non-fatal (10%) accidents, the pattern of unsafe acts committed by aircrew was similar between fatal and non-fatal accidents. Furthermore, skill-based errors (primarily technique errors) have been associated with nearly 4 out of every 5 accidents since 1990, followed by decision  (38%) and perceptual errors (less than 10%). Discussion. These analyses have provided unique insight into the genesis of GA accidents. Implications for GA ab initio and recurrent training will be discussed.

## “Swiss Cheese” Model of Accident Causation

**“****Swiss Cheese” Model of Accident Causation**

**Notes:**
Introduction
As many of you may recall from presentations at earlier meetings of the Aerospace Medical Association, much of our work evolved from Jim Reason’s “Swiss Cheese” model of human error.  Working within the nuclear power industry, Dr. Reason described the breakdown of a productive system and the subsequent analyses of failed or absent defenses.  Using an aviation example, this slide briefly describes a series of active and latent failures associated with a hypothetical controlled flight into terrain accident.  However, while Reason’s model is useful in describing the cascade of events leading up to a particular accident, he did not set out to identify all the “holes” in the cheese (those active and latent failures at each level of failure), choosing instead to focus on a general description of each level at which errors might occur.  Consequently, for the ardent accident investigator or scientist, the “Swiss Cheese” model of human error was fascinating but not easily applied.  What was required was a taxonomic framework that systematically defined the various failures (holes in the cheese). Only then would this seemingly nebulous world of human error be sufficiently described. But wait a minute, aren’t there simply too many “holes” to identify?

## Slide 3

**Notes:**
HFACS
Not really. As described elsewhere (see below), the HFACS framework was scientifically and systematically derived to explain the holes in Reason’s “Swiss Cheese” model of accident caustion. This figure provides an overview of the HFACS framework. Briefly, HFACS describes four levels of failure within the system which include: 
 Organizational factors - resource management, climate, and process
 Unsafe supervisory practices - inadequate supervision, planned inappropriate operations, failed to correct known problem, and supervisory violations
 Unsafe conditions of operators - adverse mental states, adverse physiological states, physical mental limitations, crew resource mismanagement, and personal readiness
 The unsafe acts of operators - decision errors, skill-based errors, perceptual errors, violations (routine & exceptional)
So using the HFACS framework we set out to examine the role human error plays in general aviation accidents.

A more complete description of HFACS can be found in:
Shappell, S. and Wiegmann, D. (2001). Beyond Reason: Defining the holes in the Swiss Cheese.  Human Factors in Aviation Safety, 1(1), 59-86.
Shappell, S. and Wiegmann, D. (2000). The Human Factors Analysis and Classification System – HFACS.  Office of Aviation Medicine Technical Report No. DOT/FAA/AM-00/7.  Civil Aeromedical Institute, Oklahoma City, OK 73125, 2000.

## Methods

**Analyzed **_**all**_** FAR Part 91 - GA accidents occurring between 1990 and 1998 associated with aircrew error (n= 15,744).**
**The 15,744 accidents were associated with over 32,000 human causal factors, as reported by the National Transportation Safety Board (NTSB).**
**The NTSB human causal factors were classified into HFACS causal categories independently by five GA pilots.**
**	****	****- All were certified flight instructors**
**	****	****- Mean flight hours = 3,530**

**Notes:**
Methods
Before we begin however, I would like to point out that the original abstract in your programs describes the analysis of all fatal 14 CFR Part 91 (General Aviation) accidents occurring between 1990 and 1998 that were associated with aircrew error and a sample of non-fatal accidents occurring during that same time frame. However, we have since finished our analyses of all General Aviation accidents occurring during that same time frame.1  For comparison purposes we will report both here.
In total, over 32,000 human causal factors2 associated with 15,744 GA accidents were analyzed by five independent raters (all were certified flight instructors and GA pilots) using the HFACS framework. This analysis represents the largest human error analysis of General Aviation data to date and the culmination of two years of work.  To accomplish this, we used narrative and tabular data managed by both the National Transportation Safety Board and the FAA Office of System Safety.  
Five General Aviation pilots were recruited as subject matter experts and trained on the HFACS framework.  All five were certified flight instructors with a minimum of 1000 flight hours in General Aviation aircraft. The mean number of flight hours for the group was 3,530 at the beginning of the study nearly 2 years ago.

1 Not included in the analysis reported here were homebuilt aircraft, gliders, balloons and those commercial aircraft flown under 14 CFR Part 91 as non-revenue flights.
2 Those causal factors not related to human error were noted and archived; but are not explicitly classified within the HFACS framework. 

## Methods

**Each pilot was then randomly paired with a second pilot who coded the same set of accidents to compare codes and achieve consensus.**
**Pilots were then assigned another 1/3 of the accidents for a particular year and randomly paired with another pilot.**
**This process continued until all the accidents had been coded.**

**Methods**

**Notes:**
Methods (cont.)
The five GA pilot-raters were randomly assigned accidents in such a way that each accident was reviewed independently by two pilot-raters. It is important to note that all pilot-raters were instructed to only code those causal factors identified by the NTSB. In other words, no new casual factors were added based upon our analyses (i.e., we did not re-investigate the accident). After the pilot-raters made their initial classifications of the NTSB causal factors using HFACS (i.e., skill-based error, decision-error, etc.) the two ratings were compared.  Where differences existed between the ratings, the two pilots were asked to reconcile their differences and an agreed-upon “consensus” classification was included in the database for further analysis.
Overall, the independent pilot-raters agreed on the classification of human causal factors within the HFACS framework more than 85% of the time, for a Kappa in excess of .75, an excellent value for decision making tasks. It should be noted however, that where differences did exist a consensus agreement was always reached and ultimately reported within the database.  Furthermore, all human causal factors were accommodated using the HFACS framework.

## In general aviation, only the bottom two tiers of HFACS will likely be populated.

**Notes:**
Results
With few exceptions (e.g., flight instructors and flight training institutions), the top two tiers of HFACS (Unsafe Supervision and Organizational Influences) remained sparsely populated when examining the General Aviation accidents leaving the majority of causal factors within the bottom two tiers of HFACS. However, in the interest of time (note that the time allotted for the presentation is 10 minutes) and for the purposes of this initial presentation, we have restricted our analyses to the unsafe acts of aircrew - the bottom tier of the framework. A more complete description of the data will be available at a later date.

## FAR Part 91 - General Aviation:

**Fatal Aircrew-Related Accidents**

**Notes:**
Results (cont.)
The percentage of accidents associated with each form of unsafe act (skill-based errors, decision errors, perceptual errors, and violations) is presented for each year of the study (1990-1998). Note however, that because each accident can, and often does, have multiple causal factors the fact that the percentage of one error form goes up does not necessarily mean that another must go down.  In effect, they are independent of each other.
With that in mind, an inspection of the figure illustrates several heretofore, unknown facts regarding fatal GA accidents.  First, safety efforts over the last several years appears to have had little impact on specific types of human error associated with fatal GA accidents. The notable exceptions are a small dip in the percentage of decision errors in 1994 and 1995 and a gradual decline in violations observed between 1991-94. Other than that, the lines are essentially flat. What these data appear to suggest is that any effect of safety and training efforts during this time period likely had a ubiquitous impact, if any.  This is in direct contrast to what has been reported previously in military and commercial aviation using HFACS.1  Second, skill-based errors have been associated with 4 out of every 5 accidents (roughly 80%) since 1990.  These skill-based errors are primarily technique (stick-and-rudder) type errors indicating failures associated with training and currency/proficiency.  Third, about 35% of all fatal GA accidents are associated with violations of the rules (e.g., visual flight rules [VFR] flight continued into instrument meteorological conditions [IMC]).  It is important to point out that these violations are “willful” departures from the rules and not simply inadvertent flight into the weather (classified as a decision error).
Like violations, decision errors were also associated with about 35% of all fatal accidents, but perceptual errors (often due to visual illusions and spatial disorientation) were associated with less than 15% of all fatal accidents.  Notably, many of our current intervention strategies and research efforts have been aimed at these last two error forms – perhaps lending some support to these efforts.

1 Previous presentations by the authors at AsMA and elsewhere have demonstrated dramatic effects following data-driven interventions using HFACS. 

## FAR Part 91 - General Aviation:

**Non-Fatal Aircrew-Related Accidents**

**Notes:**
Results (cont.)
The pattern of results was similar for non-fatal GA accidents with the notable exception of violations of the rules that accounted were associated with only 10% of the accidents.
Again, the trends across the years were relatively flat. Nevertheless, as with fatal accidents, skill-based errors were associated with more accidents than any other error form, averaging roughly 80% of the accidents examined.  Skill-based errors were followed by decision errors, which were associated with a little over 38% of the accidents, violations with 10%, and perceptual errors about 7%.

## FAR Part 91 - General Aviation:

**Fatal vs. Non-Fatal Accident Comparison**

**Notes:**
Results (cont.)
Perhaps more interesting, however, were the direct comparisons between fatal and non-fatal GA accidents more easily visualized when depicted separately for each error type.  As can be seen from the four graphs, essentially no differences existed between fatal and non-fatal General Aviation accidents for skill-based, decision or perceptual errors.  In contrast, the proportion of accidents associated with violations were considerably lower for non-fatal than fatal GA accidents.  In fact, using a common estimate of risk known as the odds ratio, fatal accidents were more than four times more likely to be associated with violations than non-fatal accidents (odds ratio = 4.588; 95% confidence interval = 4.130 to 5.096, Mantzel-Haenszel test for homogeneity = 900.387, p<.001).1

1 The relative risk estimate is a measure of association between the presence or absence of a factor and the occurrence of an event.  The odds ratio represents the relative risk ratio. Because the confidence interval does not include 1, it can be concluded that the ratio is significant.  The Mantzel-Haenszel is a essentially a t-test for the vale of the common odds ratio.  A significant relationship, as above, suggests that the common odds ratio is likely correct.

## FAA Regional Comparison

**FAA Regional Comparison**

**Notes:**
Results (cont.)
Of particular interest to those involved with GA safety and training programs was the possibility that differences exist in the types of errors committed by GA pilots depending upon which geographic region of the country one examined.  In other words, do GA pilots crash aircraft for different reasons in different regions of the U.S., or is it a “one size fits all” sort of deal? For instance, one might assume that given the often harsh terrain and weather conditions experienced by pilots in Alaska that differences might exist when they were compared with their counterparts in the rest of the U.S. Indeed, some have made that very argument for years – albeit based upon anecdotes and conjecture rather than the accident record. Unfortunately, until now opinion and anecdotes was about all we had to work with. But with the development of HFACS and the completion of the GA analysis we now have a systematic and scientific means to address this issue.
So, with this in mind we parsed the data set by the region where the accident occurred using the existing FAA regional breakout.1 The FAA is divided into nine regions as presented above. While one can certainly question whether putting Hawaii in with California, Nevada and Arizona makes sense or question why one state was considered part of Region X but not Region Y or Z, we chose to work within existing FAA regions as a first pass. That being said, what did we find?

1 Note, this may, or may not, have been where the pilot’s home of record was. However, it is likely that many of the accidents also occurred in the region where the pilot made his home.

## FAR Part 91 - General Aviation:

**FAA Regional Comparison**

**Notes:**
Results (cont.)
Much to the surprise of some, we saw no differences between FAA regions in the relative distribution of errors and violations committed by GA pilots involved in accidents. Even Alaska appears no different than the rest of the U.S. when the data are examined systematically.
OK, but what if we looked at just fatal accidents?

## FAR Part 91 - General Aviation:

**FAA Regional Comparison (Fatal Only)**

**Notes:**
Results (cont.)
Even then, no real differences jump out at you.  Perhaps New England region is associated with more skill-based and decision errors and maybe Central region is associated with slightly more violations of the rules – but are these operationally significant? Perhaps not.
What this tells us is that whether your accident occurred in Alaska or Florida, California or New York, the relative distribution of unsafe acts (errors and violations) committed by aircrew were strikingly similar. Indeed, even those that espouse the “bush pilot” theory of flying in Alaska seem to be off base. While Alaska my witness more accidents, which in turn may be more a function of the fact that folks in Alaska fly aircraft like we take taxis in the continental U.S., the types of errors committed do not appear to vary.
So what does this all mean where training is concerned? 

## Summary of Findings

**Regardless of severity, skill-based errors are the most prevalent causal factors associated with GA accidents – more so with fatal accidents.**
**Roughly forty percent of fatal accidents are associated with violations of the rules.**
**	****- Typically, those violations are continuing into IMC when ****	****VFR only.**
**Fatal GA accidents are more than four-times more likely to be associated with violations of the rules and regulations.**
**There do not appear to be any regional differences in the distribution of errors and violations committed by aircrew.**

**Notes:**
Summary of Findings
For the first time ever, we can now talk about more than the simple fact that nearly 80% of all general aviation accidents are attributable to human error.  After all, that doesn’t tell us much.  How can you fix human error if you don’t know what they are?  It’s like saying that 20% of the accidents are attributable to mechanical failure and not telling you what type of mechanical failure occurred.  You can’t fix what you don’t know.
What we do know now is that the vast majority of GA accidents, regardless of severity, are due to skill-based errors.  Also evident, is that nearly 40% of all fatal accidents are due to violations of the rules and that these unsafe acts are much less common in non-fatal accidents.  What this means is that if you violate the rules and are involved in an accident, you are much more likely to perish or fatally injure someone else.  Not surprisingly, many of these violations are the direct result of continuing into IMC when authorized for VFR flight only.  These are not however, inadvertent flights into IMC. 
As a point of interest, marked differences were observed between fatal and non-fatal human error patterns.  This should dispel the myth that non-fatal accidents are merely a heartbeat away from becoming a fatal accident.  In fact, the intrinsic human causal factors associated do appear to be different – at least in the case of skill-based errors and violations.  They are not simply a manifestation of the same causes separated by luck.
Finally, the human causes of accidents do not seem to differ depending on where in the U.S. the accident occurred. This would seem to lend some credence to the old adage that “there are no new ways to crash aircraft, only new pilots.”

## Implications For Training

**Emphasis should be placed on basic flight skills and GA training (both ab initio and recurrent).**
**If a reduction in fatal accidents is the goal (safer skies), then emphasis should be placed on skill-based errors and violations of the rules and regulations.**
**	****- Improved flight training**
**	****- Safety awareness**
**	****- Rule enforcement**
**Tracking interventions**

**Implications For Training**

**Notes:**
Implications for Training
So what can we do now that the face of human error has been exposed using HFACS?  One thing that can be done to address the large percentage of skill-based errors in the accident data is to place increased emphasis on training and refining basic flight skills during ab initio and recurrent flight training.  However, if the goal is to reduce fatal accidents, as outlined by the Administrator of the FAA as she launched the Safer Skies Initiative, then greater emphasis must also be placed on reducing the number of violations perhaps through improved flight training, safety awareness, and rules enforcement.  All three have been proven effective in military aviation.
Equally important given the preceding data is that the training need not be tailored to any particular region.  That is, the accident data do not appear to support special programs to address human error in Alaska separate from those in Florida or Texas. This is not to say that programs like Alaska Capstone that aim to improve terrain and weather awareness are not a good idea given the climate and terrain in Alaska. It simply means that if the aim is to reduce the overall number of GA accidents appreciably, more needs to be done to address the specific types of human error most often associated with accident causation. HFACS is the logical first step in the risk management process – but only the first step. The arguably more difficult next steps are to design and implement “data-driven” intervention strategies aimed at the specific types of human error identified though the HFACS analysis.
Regardless, we now have baseline human error data covering 9 years of GA accidents and a tool to monitor our intervention strategies and evaluate their effectiveness.  HFACS has proven particularly useful in this regard in the military.  There is no reason to believe that it would not be equally successful in civilian aviation.

## Cirrus Design Corporation

**Cirrus Design Corporation**
**Cirrus Aircraft Parachute System (CAPS) **

**Notes:**
Thank you.  Questions?