Developing a Methodology 

for Assessing Safety 

Programs Targeting 

Human Error in Aviation

Scott Shappell

Clemson University

Clemson, SC 29634

Douglas Wiegmann

Mayo Clinic

Rochester, MN 55905

November 2006

Final Report

DOT/FAA/AM-06/24

Office of Aerospace Medicine

Washington, DC 20591
NOTICE

This document is disseminated under the sponsorship 

of the U.S. Department of Transportation in the interest 

of information exchange. The United States Government 

assumes no liability for the contents thereof.

___________

This publication and all Office of Aerospace Medicine technical reports are 

available in full-text from the Civil Aerospace Medical Institute’s publications 

Web site: 

www.faa.gov/library/reports/medical/oamtechreports/index.cfm


Technical Report Documentation Page 

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No. 

DOT/FAA/AM-06/24 

4. Title and Subtitle 5. Report Date 

Developing A Methodology For Assessing Safety Programs Targeting November 2006 

Human Error In Aviation 6. Performing Organization Code 

 

7. Author(s) 8. Performing Organization Report No. 

Shappell S,1 Wiegmann D2 

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) 

 

11. Contract or Grant No. 

1

Clemson University 

Clemson, SC 29634 

2

Mayo Clinic 

Rochester, MN55905 

 12. Sponsoring Agency name and Address 13. Type of Report and Period Covered 

Office of Aerospace Medicine 

Federal Aviation Administration 

800 Independence Ave., S.W. 

Washington, DC 20591 14. Sponsoring Agency Code 

15. Supplemental Notes 

Work was accomplished under approved task AM-B-05-HRR-521. 

16. Abstract 

There is a need to develop an effective methodology for generating comprehensive intervention strategies that 

map current and proposed safety programs onto well-established types of human error. Two separate studies were 

conducted using recommendations from NTSB accident investigations and several joint FAA and industry 

working groups. The goal of the studies was to validate a proposed framework for developing and examining 

safety initiatives that target human error in aviation. The results suggest five approaches to reducing human 

factors associated with aviation accidents. When combined with the Human Factors Analysis and Classification 

System, the resulting Human Factors Intervention Matrix will provide a useful tool for evaluating current and 

proposed aviation safety programs. 

 

17. Key Words 18. Distribution Statement 

 

 

Human error, Accident Analysis, Intervention, HFACS, 

HFIX, Safety Program Assessment

Document is available to the public through the 

Defense Technical Information Center, Ft. Belvior, VA 

22060; and the National Technical Information 

Service, Springfield, VA 22161 

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price 

Unclassified Unclassified 13 

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized 



Developing a Methodology for Assessing Safety Programs

Targeting Human Error in Aviation

“I believe that the past is prologue.....In our recommendations we try to take what we have 

learned and correct situations so it shouldn’t happen again.” 

—Former National Transportation Safety Board Chairman James Hall (1996)

INTRODUCTION

Indeed, the National Transportation Safety Board 

(NTSB), Federal Aviation Administration (FAA), and 

other safety organizations have committed extraordinary 

resources to prevent civilian aviation accidents. As a result, 

aviation in the U.S., particularly commercial aviation, has 

become one of the safest modes of transportation. Still, 

accidents can happen, often repeating the same sequence 

of events played out many times before. As a result, we 

are often left with the regrettable truth that there are 

really very few “new” accidents, just different players. 

Perhaps that is why Chairman Hall chose the operative 

term “shouldn’t” rather than “won’t” in 1996.

So if there really are few “new” accidents, why has the 

aviation accident rate remained relatively stable over the 

last several years? After all, if we already know what the 

problem is, why have we been unable to fix it? Perhaps 

it has something to do with the current state of aviation 

safety. Truth be told, the industry is extremely safe, and 

the easy fixes have been identified and remedied. What 

remains to be addressed is the small fraction of accidents 

attributable to perhaps the most complex problem facing 

aviation today – human error.

A closer examination of the current aviation accident 

record has revealed that anywhere between 70-80% of 

all aviation accidents are at least partially attributable to 

human error (Wiegmann & Shappell, 2003). Therefore, 

it stands to reason that, if quantifiable improvements 

in aviation safety are to be realized, the primary focus 

should be on the human operator (i.e., aircrew) and those 

involved with the safe conduct of flight (e.g., mechan-

ics, supervisors, air traffic controllers) rather than more 

traditional areas like the aircraft itself.

With this in mind, the FAA has employed the 

Human Factors Analysis and Classification System 

(HFACS; Wiegmann & Shappell, 2001, 2003; Shappell 

& Wiegmann, 2003, 2004) to identify the human factors underlying both commercial and general aviation. 

Principal among the FAA’s findings using HFACS was 

the observation that, while previous safety programs may 

have impacted other areas of aviation, there has been 

little evidence that they have had a significant impact on 

any specific type of human error (Figure 1). That is to 

say, the percentage of accidents associated with aircrew 

error (i.e., skill-based errors, decision errors, perceptual 

errors, and violations) has remained relatively stable 

since 1990.

What this implies is that intervention strategies implemented in the 1990s have had, at best, ubiquitous effects 

on the errors and violations committed by aircrew. More 

likely, however, there has been no sustained impact of 

any particular intervention program (Shappell, Detwiler, 

Holcomb, Hackworth, Boquet, & Wiegmann, 2006). 

The latter should come as no surprise, given that prior 

to these FAA studies, no comprehensive analysis of aircrew and supervisory error had been conducted using a 

human factors approach to accident causation. Not to 

mention that there has been no systematic human factors 

examination of the current or proposed safety programs 

aimed at addressing human error.

But in some ways, that is putting the proverbial cart 

before the horse. After all, while HFACS provided a 

theoretically derived and validated framework for accident/incident investigation and analysis, a similar 

framework did not exist that would allow the FAA and 

other organizations to evaluate the potential benefits of 

current and proposed human error intervention strategies. 

So the better question may be whether a “human factors” 

analysis of safety programs is even possible.

NASA Intervention Strategies

At least one study (Wiegmann & Rantanen, 2003) 

suggests that such an analysis can be performed using a 

set of standards derived from the same body of literature 

used to develop HFACS. In their book, A Human Factors 

Approach to Aviation Accidents, Wiegmann and Shappell 

(2003) described an intervention taxonomy clustered 

around four broad categories:

1. Environment (e.g., the control of temperature, 

noise, vibration, lighting)

2. Human (personnel selection, incentives, training, 

teamwork, communication, etc.)

3. Machine (engineering design, capacity, etc.)

4. Task (ordering/timing of events, procedures, standardization, etc.)
0

10

20

30

40

50

60

70

80

90

100

Environment Machine Human Task Unclassifiable

Intervention Strategy

Percentage of AvSP Products

Figure 2. Percentage of NASA safety programs 

within each intervention category. 

0

10

20

30

40

50

60

70

80

90

100

'90 '91 '92 '93 '94 '95 '96 '97 '98 '99 '00 '01 '02

Skill-based

Errors

Perceptual

Errors

Decision

Errors

Violations

Percentage of Accidents

Year

A.

0

10

20

30

40

50

60

70

80

90

100

90 91 92 93 94 95 96 97 98 99 '00 '01 '02

Skill-based

Errors

Perceptual

Errors

Decision

Errors

Violations

Percentage of Accidents

Year

B.

0

10

20

30

40

50

60

70

80

90

100

1990-1992 1993-1995 1996-1998 1999-2002

Skill-based

Errors

Perceptual

Errors

Decision

Errors

Violations

Percentage of Accidents

Year

C.

Figure 1. Percentage of unsafe acts committed by 

aircrew during general aviation (Panel A) commuter (Panel B) and air carrier (Panel C) operations 

by year. Note that percentages will not add up to 

100% due to multiple causal factors associated 

with most accidents.

Using this framework, Wiegmann and Rantanen 

(2003) examined a variety of technologies developed by 

NASA’s aviation safety program (AvSP). From energy 

absorbing seats, restraints, and structures to synthetic 

vision, each safety program was classified within one 

of the four intervention categories. As shown in Figure 

2, they concluded that NASA’s primary intervention 

strategies targeted the machine rather than the human, 

environment, or task. Two programs, Incident Report-

ing Enhancement Tools and Fast-time Simulation of 

System-wide Risks, were considered unclassifiable by 

the raters using these categories.

In a separate part of their study, Wiegmann & Rantanen (2003) examined the NASA technologies using 

the HFACS framework. Surprisingly, it was determined 

that nearly half of the technologies that NASA was 

developing were rated as having no impact on aircrew 

error. What’s more, those that might have an impact 

primarily targeted decision errors, by providing better 

information, automation, and training. An even smaller 

percentage of the technologies targeted aircrew error, in 

general, and only one of the products primarily targeted 

skill-based errors – the most frequent human error fac-

ing both commercial and general aviation. None of the 

products primarily targeted violations, another area of 

concern within civilian aviation operations.

Purpose

Clearly, if improvements in safety are to be real-

ized, a more systematic methodology is needed for 

generating intervention/prevention strategies that can 

tie into human error frameworks like HFACS. Such a 

methodology would help ensure that factors affecting 

human performance are addressed at multiple levels 

and from multiple directions, thereby facilitating the 

development of effective intervention strategies rather 

than a single, narrowly focused design fix. 

This report describes two studies that build upon 

the methodology originally described by Wiegmann 

and Shappell (2003) and used by Wiegmann and 

Rantanen (2003) with NASA safety programs. The 

first study describes an independent validation of the 

four intervention methodologies using safety recommendations from the NTSB. The second describes the 

examination of proposed FAA aviation safety programs 

using a prototype intervention matrix that maps the 

unsafe acts of operators (i.e., skill-based errors, decision 

errors, perceptual errors, and violations) onto several 

intervention approaches.
STUDY 1: Analysis of NTSB 

Recommendations

Investigating accidents, identifying potential interventions, and issuing safety recommendations are central to 

any safety program and as such are a major function of 

local, state, and federal safety boards. Indeed, one such 

national entity, the NTSB, cites safety recommendations 

as “... the most important part of [their] mandate...” 

(NTSB, 2002). 

Ideally, safety recommendations, when adopted by 

cognizant organizations, will positively influence future 

operations in the field and thereby improve overall system safety. However, recommendations are just that ... 

recommendations and, as such, are not always adopted. 

Moreover, they are often based solely on isolated events 

or at best a few events over a very short period of time 

rather than more global analyses of the system as a whole. 

While these interventions may solve a local or single-point 

problem, they often do not have far-reaching impact.

Further complicating matters, many domains such 

as aviation and their corresponding safety boards have 

traditionally strong relations with quantitative disciplines 

like engineering and physics. Consequently, while these 

organizations may be especially adept at dealing with 

mechanical issues, they tend to be less robust when 

dealing with organizational or human-centered aspects 

of accidents like human error, organizational failure, 

communication, and risk assessment (Stoop, 2002).

Recognizing this, the NTSB, like many safety entities, 

has integrated human factors experts into their organization, presumably leading to recommendations that address the entire system rather than a single engineering or 

mechanical aspect, per se. However, employing human 

factors experts alone does not necessarily translate into 

a breadth of interventions. The question remains, what 

specific intervention approaches does the NTSB employ? 

In other words, does the NTSB tend to be uni-dimensional (like NASA) or multi-dimensional with regard to 

specific intervention approaches?

Method

NTSB Safety Recommendations

To examine this question, aviation safety recommendations associated with commercial (14 CFR Part 121

– air carrier and Part 135 – commuter) aviation accidents 

occurring between 1998 and 2004 were obtained from 

the NTSB’s official Website (www.ntsb.gov). Of the 

147 commercial aviation accidents reports that were 

completed at the time of this study, 622 unique safety 

recommendations were identified. However, several of the 

recommendations consisted of compound solutions. In 

those cases, the original recommendation was separated 

into sub-recommendations while preserving the intent of 

the NTSB. This resulted in a revised list of 872 unique 

recommendations for further analysis.

Clustering Process

The recommendations were independently clustered 

into categories by two analysts (one with a doctoral-level 

background in psychology, the other with a graduate 

background in engineering) based on their similarities. 

The analysts were not instructed to use any predefined 

taxonomy or classification scheme. They were simply 

instructed to independently assign each recommendation 

to categories of their choosing, based upon the nature of 

the recommendation. 

Not surprising, given the vagueness of the instructions, 

there were some differences in the terms used by the two 

analysts, but there were also strong similarities. Wherever 

disagreements occurred, the analysts were asked to discuss 

their clustering heuristic and to agree on a single classification scheme. In the end, all 872 recommendations 

were classified based on their underlying similarities by 

two independent analysts, who later came to a consensus 

on the number and labels for each of these clusters.

Results

Ultimately, the analysts generated nine unique categories of recommendations, which included the design 

of parts/displays, procedures, communication, training, 

requests to conduct focused studies, rules, manuals, 

inspection, and human resources. These nine categories 

were then further clustered into four larger categories 

based on their similarities: 1) administrative/organizational; 2) mechanical/ engineering; 3) human/crew; and 

4) task/mission. Each category and their accompanying 

subcategories are briefly described in Table 1.

Distribution of recommendations

On average, roughly six recommendations spread 

across just under three (2.8) intervention subcategories 

were observed per accident. The actual distribution of 

recommendations across the intervention categories and 

subcategories is presented in Table 2.

From a global perspective, it appears that roughly 

two-thirds of the recommendations were either admin-

istrative/organizational or mechanical/ engineering fixes. 

However, nearly a quarter of the recommendations were 

aimed at either the task or mission. 

Surprisingly few interventions directly targeted operators (aircrew), even though previous studies repeatedly 

show that more major accidents have been attributed to 

human error than to any other single cause (Wiegmann 

& Shappell, 2003; Boquet et al., in review; Detwiler et al., 
Table 1. Proposed categories and subcategories of NTSB recommendations.

Administrative/Organizational 

Rules/Regulations/Policies: Issuing, modifying, 

establishing, amending, and/or reviewing 

policies, rules, or regulations. 

Information Management/ Communication:

Improvements in disseminating, storing, 

archiving and publishing information. Also 

included are recommendations regarding 

collecting data; issuing information bulletins, 

advisory circulars, and reporting activity. 

Research/ Special Study: Conducting research to 

determine the impact of recent technological 

advances or call for special studies to review 

processes, develop/validate methodologies, 

evaluate the feasibility of safety equipment, 

and/or conduct surveys. 

Human Resource Management: Adequacy of 

staff in specific situations, the need for additional 

personnel, and the evaluation of individual skills 

of employees. 

Mechanical/Engineering 

Design/Repair: Specific manufacturing changes 

including the design of parts. Also included is 

the modification, replacement, removal and/or 

installation, or repair of parts and equipment. 

Inspection: Maintenance inspections, 

overhauling, detecting damage, including day-today operations such as inspecting fuel, oil level, 

and recommended safety checks. 

Human / Crew 

Training: Reviewing, developing, and 

implementing training programs. Also included 

is the training of personnel in handling 

emergencies. 

Task/Mission 

Procedures: Amending, reviewing, modifying, 

revising, establishing, developing, and validating 

procedures. 

Manuals: Reviewing, revising, issuing, 

amending, and modifying manuals, bulletins, 

checklists, and other instructions or guidance. 

Table 2. Percentage of recommendations 

associated with each intervention category. 

Intervention Category % n 

Administrative/Organizational 34.18 

 Rules/Regulations/Policies 9.29 81 

 Information management/ Comm. 13.76 120 

 Research/Special study 10.44 91 

 Human resource management 0.69 6 

Mechanical/Engineering 31.20 

 Design/Repair 23.17 202 

 Inspection 8.03 70 

Human/Crew 11.47 

Training 11.47 100 

Task/Mission 23.16 

 Procedures 14.56 127 

 Manuals 8.60 75 

(23.17%) - nearly twice as many as any other category. 

Considerably fewer were aimed at procedures, training, 

information management/ communication, and the 

other subcategories. 

Summary

To date, there have been few attempts to systematically study recommendations generated by investigative 

organizations like the NTSB. This is unfortunate, because 

the results of such studies may help in understanding 

why accident rates have stabilized over the last several 

decades and could lead to the development of more effective intervention strategies. For example, in this study 

alone there were four broad categories of interventions 

identified, comprising nine unique categories of recommendations. 

When examining the breadth and scope of NTSB recommendations, even at this level, it appears that current 

safety recommendations in aviation tend to focus more 

on improving the design of systems or some manner of 

organizational change rather than focusing on operational 

personnel. While these recommendations are obviously 

well-intentioned and often specific to a particular accident, 

they may be misplaced or too narrow in scope. This may 

help explain why the percentage of accidents associated 

with human error has not changed over the last 15 years 

(Wiegmann & Shappell, 2003; Wiegmann et al., 2005; 

Shappell et al., in press).

This is not to say that the design of new technology 

will not have a significant impact on how people perform. 

After all, advances in aviation technology and engineering 

have accounted for marked reductions in the aviation 

accident rate since the late 1950s. On the other hand, 

some of these advances have led to new, occasionally 

2006; Shappell et al., in press). It has also been observed 

that wider systemic issues, including the managerial 

and regulatory context of aviation operations, were also 

mentioned in a large number of reports (Holloway & 

Johnson, 2004; Johnson, in review), even though this 

does not seem to be reflected in the accident record.

A closer examination revealed that, similar to Wiegmann and Rantanen’s study of NASA safety programs, 

design fixes constituted the largest percentage of any 

individual type of recommendation made by the NTSB 
catastrophic, errors (e.g., autopilot-mode errors, Sarter 

& Woods, 1992; 1994). What appears to be required is 

a broader, systematic approach to accident intervention, 

particularly if we are to effectively address human error 

within aviation operations. But how can this be done?

To ensure that safety professionals generate effective 

intervention strategies, rather than a single “knee jerk” 

fix to a problem, knowledge of all viable interventions is 

required. Towards these ends, the present study suggests 

that there are at least four broad categories of interventions that appear tenable within the aviation industry. 

These are Administrative/Organizational, Human/Crew, 

Mechanical/Engineering, and Task/Procedure. 

These four approaches differed slightly from those 

previously proposed by Wiegmann and Shappell (2003) 

and utilized by Wiegmann and Rantanen (2003) to analyze NASA safety programs. One category that naturally 

surfaced from the present analysis, but was missing from 

the Wiegmann and Rantanen study, was Administrative/Organizational interventions. In contrast, “environmental” interventions did not appear in the current 

study but were present in the NASA study (Wiegmann 

& Rantanen, 2003).

In the end, the question is not whether or not there 

are three, four, five, or more approaches to identifying 

potential accident interventions as much as there is definitively more than one. Exactly what those approaches 

are remains to be fully explored. However, the five approaches identified between the present study and the 

investigation conducted by Wiegmann and Rantanen 

(2003) is a reasonable first start.

STUDY 2: HFIX Analysis of JSAT/

JSIT Recommendations

Identifying viable approaches for intervening, however, 

is only the first step. The ability to map interventions onto 

specific types of human error is also vitally important. 

In other words, simply generating a variety of interventions across several domains, whether they are human, 

mechanical, environmental, and so on, is likely to be 

ineffective unless such interventions directly target the 

problem area.

Given that human error continues to be the largest 

contributor to commercial and general aviation accidents, 

it makes sense to map different interventions against 

specific error forms. What is needed is a theoretical framework that captures the underlying causal mechanisms 

of human error along with the intervention approaches 

identified in Study 1.

Human Factors Analysis and Classification System

Such an error framework already exists and is widely 

used within the aviation industry. This framework, 

the Human Factors Analysis and Classification System 

(HFACS), describes two general categories of unsafe 

acts that operators commit: errors – the honest mistakes 

individuals make every day, and violations – the willful 

disregard for the rules and regulations of safety.

 Within 

those two overarching categories, HFACS describes three 

types of errors (decision, skill-based, and perceptual) and 

two types of violations (routine and exceptional). Each 

is briefly described below.

Errors

One of the more common error forms, decision errors, represents conscious, goal-intended behavior that 

proceeds as designed, yet the plan proves inadequate or 

inappropriate for the situation. Often referred to as “honest 

mistakes,” these unsafe acts typically manifest as poorly 

executed procedures, improper choices, or simply the 

misinterpretation or misuse of relevant information.

In contrast to decision errors, the second error form, 

skill-based errors, occurs with little or no conscious thought. 

Just as little thought goes into turning one’s steering wheel 

or shifting gears in an automobile, basic flight skills such 

as stick and rudder movements and visual scanning often 

occur without thinking. The difficulty with these highly 

practiced and seemingly automatic behaviors is that they 

are particularly susceptible to attention and/or memory 

failures. As a result, skill-based errors such as the breakdown in visual scan patterns, inadvertent activation/deactivation of switches, forgotten intentions, and omitted 

items in checklists often appear. Even the manner in (or 

skill) which one flies an aircraft (aggressive, tentative, or 

controlled) can affect safety.

While decision and skill-based errors have dominated most accident databases and have, therefore, been 

included in most error frameworks, the third and final 

error form, perceptual errors, has received comparatively 

less attention. No less important, perceptual errors occur 

when sensory input is degraded, or “unusual,” as is often 

the case when flying at night, in the weather, or in other 

visually impoverished environments. Faced with acting 

on imperfect or incomplete information, aircrews run 

the risk of misjudging distances, altitude, and decent 

rates, as well as responding incorrectly to a variety of 

visual/vestibular illusions.

 A complete description of the entire HFACS framework, including 

all 4 tiers and 19 causal categories, can be found in Wiegmann and 

Shappell, 2003.
Violations

Although there are many ways to distinguish among 

types of violations, two distinct types have been identified based on their etiology. Routine violations tend to be 

habitual by nature and are often enabled by a system of 

supervision and management that tolerates such departures from the rules (Reason, 1990). Often referred to 

as “bending the rules,” the classic example is that of the 

individual who drives his/her automobile consistently 5-

10 mph faster than allowed by law. While clearly against 

the law, the behavior is, in effect, sanctioned by local 

authorities (police) who often will not enforce the law 

until speeds in excess of 10 mph over the posted limit 

are observed.

Exceptional violations, on the other hand, are isolated 

departures from authority, neither typical of the individual 

nor condoned by management. For example, while driving 

65 in a 55 mph zone might be condoned by authorities, 

driving 105 mph in a 55 mph zone certainly would not. 

It is important to note that while most exceptional violations are appalling, they are not considered “exceptional” 

because of their extreme nature. Rather, they are regarded 

as exceptional because they are neither typical of the 

individual nor condoned by authority.

Human Factors Intervention Matrix (HFIX)

A prototype matrix, called the Human Factors Intervention Matrix (HFIX), pits the unsafe acts individuals 

commit against the five different intervention approaches 

presented in Figure 3. The utility of such a framework 

seems intuitive. For example, if one were interested in 

developing interventions to address decision errors, the 

goal would be to identify prospective interventions within 

each approach (i.e., organizational/administrative, human/crew, etc.), thereby ensuring that the widest array 

of interventions were considered. By mapping prospective interventions onto the matrix, it would be readily 

apparent if the scope of a proposed program was uni- or 

multi-dimensional. 

Alternatively, a framework like HFIX could be used 

proactively to determine which areas an organization has 

“covered” and where gaps exist in the current safety program given current trends in the error data. For instance, 

if you knew that the largest threat to safety within your 

organization was skill-based errors, followed by decision 

errors, violations, and perceptual errors (as is the case 

with general and commercial aviation in the U.S.), HFIX 

could be used to determine if your proposed and future 

interventions have the potential to address those needs 

and which areas are currently being targeted.

Hence, the purpose of Study 2 was to determine if 

such an approach could be used within the FAA and 

which types of human error might be affected by current 

and future interventions. In a sense, this analysis would 

provide a “benchmark” of current FAA intervention efforts. When combined with existing HFACS data (e.g., 

Shappell et al., in press; Detwiler et al., 2006; Wiegmann 

et al., 2005) possible gaps, if any, may be identified. 

FAA Safer Skies Initiative

As part of the FAA’s Safer Skies initiative, three teams 

of experts from government, employee advocacy groups 

(e.g., the National Air Traffic Controllers Association), 

the aviation industry, and academia were formed to address civilian aviation accidents. Two of those teams, the 

Commercial Aviation Safety Team (CAST) and General 

Aviation Joint Steering Committee (GA JSC), were formed 

to address specific threats to commercial and general 

aviation, respectively.

With the CAST and the GA JSC providing oversight, 

three working groups were formed: 1) Joint Safety Analysis 

Teams (JSATs), 2) Joint Safety Implementation Teams 

(JSITs), and 3) Joint Implementation Monitoring Teams 

(JIMTs). Particularly germane to this study were outcomes 

derived from the JSAT and JSIT working groups since 

they represented current and future interventions necessary to address human error associated with commercial 

and general aviation accidents. In particular, this study 

was interested in the recommendations from JSAT/JSIT 

teams examining accidents associated with:

•	 Controlled flight into terrain

•	 Approach and landing

•	 Loss of control

•	 Runway incursions

•	 Weather

•	 Pilot decision-making

Decision

Errors

Skill-based

Errors

Perceptual

Errors

Violations

Organizational/

Administrative

Human/

Crew

Technology/ 

Engineering

Task/

Mission

Operational/

Physical 

Environment

Figure 3. The “Human Factors Intervention 

matriX” (HFIX). 
Method

JSAT and JSIT Recommendations

JSAT and JSIT reports were collected from each 

CAST and GA JSC committee by researchers at the 

Civil Aerospace Medical Institute. After eliminating 

duplicate recommendations, a comprehensive list was 

compiled electronically for classification. The final list 

of 614 unique recommendations was then randomized 

to reduce bias.

Categorization of the Data

Eighteen Master of Aeronautical Science candidates 

were recruited from Embry-Riddle Aeronautical University for Study 2. Each had experience in the aviation community as either a pilot, maintainer, or at an administrative 

level, and all had successfully completed a minimum of 

one graduate-level human factors course.

After a roughly 4-hour training session on the HFACS 

and HFIX frameworks, subjects were randomly assigned 

to one of six groups. Each 3-person team was then randomly assigned roughly one-sixth of the recommendations to classify.

Each team member was instructed to independently 

classify each recommendation into only one of the five 

intervention approaches (i.e., organizational/ administrative, human/crew, mechanical/engineering, task/mission, or physical environment). In addition, they were 

instructed to identify any HFACS Unsafe Acts categories 

they felt the intervention would impact.

After the initial rating, team members were permitted to discuss their classification within their group to 

resolve any differences. A final, consensus classification 

for each recommendation was then provided for further 

analysis. 

Results

The results of both classification tasks are presented 

in Figure 4. Several observations can be made from the 

data. First, as with the NTSB recommendations, a large 

percentage (36.6%) of the JSAT/JSIT recommendations 

were directed at organizational/administrative levels. Likewise, several (22.2%) of the recommendations involved 

technological/engineering approaches. However, unlike 

the NTSB where relatively few recommendations targeted 

the human, nearly one-third of those obtained from the 

JSAT/JSITs did so. This may be because, unlike the NTSB 

recommendations, we selectively chose those JSAT/JSIT 

reports that addressed human error issues like pilot decision-making and runway incursions. In that sense, the 

JSAT/JSIT data were much more homogenous (i.e., they 

did not contain non-human related accidents) and the proposed interventions may simply reflect that inherent bias. 

However, if that were true, one might actually expect that 

an even larger percentage of the recommendations would 

target the human/crew than was actually observed.

When examining the HFACS classifications, remember 

that, unlike the specific approaches to accident interventions where subjects were instructed to select only one 

approach, they were permitted to select all of the HFACS 

Unsafe Act categories that they felt would be impacted by 

a given recommendation. Therefore, unlike the intervention approaches whose percentages added up to 100%, 

the total percentages associated with each Unsafe Act 

category did not.

Perhaps not unexpected, interventions aimed at decision errors were associated with nearly three out of every 

four JSAT/JSIT recommendations examined. In contrast, 

skill-based errors were associated with roughly 50% of 

the recommendations followed by perceptual errors 

(37.6%) and violations (26.9%). Of note, these numbers 

are slightly different than the percentage of accidents associated with each type of error where skill-based errors 

account for between 45-80% of the accidents, depending 

on whether one is talking about commercial or general 

aviation, respectively (see Figure 1). Likewise, roughly 1/3 

of the accidents were associated with decision errors, yet 

72.6% of the interventions have some component that 

will potentially affect pilot decision-making. 

This is not to say that there should be a one-to-one 

relationship between the percentage of accidents associated with a given error category and the percentage 

of recommendations aimed at addressing these errors. 

After all, it may take more effort to address one error 

form than another, or more interventions may naturally 

address pilot decision-making. In either case, the global 

analysis presented here suggests that additional review of 

this apparent incongruity is necessary.

Decision

Errors

Skill-based

Errors

Perceptual

Errors Violations

Organizational/ 

Administrative

Human/ 

Crew

Technology/ 

Engineering

Task/ 

Mission

Operational/

Physical 

Environmen

t

25.9%

14.3%

9.0%

13.8%

26.9%

7.8%

12.7%

20.7%

13.4%

2.8%

12.4%

12.5%

5.7%

1.3%

2.4%

2.4%

0.8%

0.7%

1.1%

0.2%

72.6%

26.9%

37.6%

49.7%

36.6% 32.6% 22.2% 7.3% 1.3%

INTERVENTIONS

HFACS UNSAFE ACTS

Figure 4. Percentage of JSAT/JSIT recommendations classified by intervention approach 

and specific HFACS Unsafe Act addressed. 
Perhaps more important, however, was the mapping of 

each intervention within both the intervention approach 

and the HFACS Unsafe Acts category (Figure 4). As can 

be seen (white boxes), three of the 20 possible boxes (organizational/ administrative by decision error, human/crew 

by decision error, and human/crew by skill-based error) 

contained 20% or more of the JSAT/JSIT interventions. 

On the surface, this appears to reflect a narrow rather 

that a broad approach to accident intervention/mitigation by these committees. It is not that the interventions 

contained within these categories will not be effective, 

just that other, potentially equally viable, interventions 

may have been overlooked.

It is interesting to note, however, that if one examines 

those boxes that contained between 10-20% of the possible 

interventions, nearly all of the remaining boxes among 

the organizational/ administrative, human/crew, and 

technology/ engineering approaches are included. What 

was not accounted for were human/crew and technology/engineering approaches dealing with violations of the 

rules and regulations. Obviously, these approaches might 

prove beneficial if an organization wanted to modify or 

curtail a particular unsafe pattern of behavior (e.g., flight 

into instrument conditions while on a visual flight rules 

flight plan) through training or technological means.

More notable was the general lack of interventions 

targeting the specific task/mission of the aircrews or the 

environment they are faced with. Perhaps a closer examination of the operations these aircrews are engaged in or the 

environments they are expected to operate in is warranted. 

In any event, there may have been options along these lines 

that were not considered by these select committees.

Summary

Ideally, tools such as HFIX provide a Gestalt of the 

safety program as a whole rather than an item-by-item 

accounting of each intervention in an organization. After 

all, it is hard to know if pieces are missing in a puzzle until 

you put them together. HFIX allows administrators and 

safety managers to put the intervention pieces together 

in such a way that they can get a “quick look” at the 

strengths and weaknesses of their programs. Additionally, 

it provides decision makers within an organization the 

ability to ensure that a broad spectrum of interventions 

has been considered. After all, only the most elementary 

of puzzles is comprised of just a few pieces; obviously, 

something as complex as human error in aviation will 

consist of a number of pieces. 

That being said, the results from Study 2 using 

JSAT/JSIT interventions, although clearly more multidimensional than NASA’s safety programs, still did not 

appear to fully address the current accident trends in 

commercial and general aviation. At least on the surface, 

it appears that there are gaps in the safety program that 

should be addressed. 

For example, there was an apparent bias toward interventions aimed at pilot decision-making, particularly those 

utilizing organizational and human approaches. While this 

is not inherently bad, previous HFACS analyses suggest 

that additional effort should be placed on skill-based errors 

and violations, two areas that appear underrepresented, 

given current trends in the accident data. 

Also noteworthy, few interventions attempted to 

modify/change the task itself or the environment. A 

closer examination of the actual types of errors may 

suggest changes in routes people fly or the actual type of 

flights being flown. 

However, while HFIX may prove useful when generat-

ing comprehensive intervention strategies, organizations 

simply cannot implement every recommendation. Other 

factors may need to be considered before employing 

a given intervention. Factors such as effectiveness (i.e., 

what is the likelihood that it will work?), cost (i.e., can 

the organization afford the intervention?), feasibility

(i.e., how easy will the intervention be to implement 

or does it actually exist?), and acceptability (i.e., will the 

workforce accept the proposed intervention?) all must 

be considered.

As such, HFIX may actually be HFIX3 mapping human 

error against the intervention approaches and evaluations 

criteria (Figure 5). Although it may appear complex, in 

reality organizational decision makers utilize this third 

dimension all the time. To apply it to the two-dimensional HFIX framework is really not that great a leap. 

However, even without this third dimension, the mapping 

of specific interventions onto a matrix that combines the 

five intervention approaches with general categories of 

human error can provide a broader perspective of the 

FAA’s safety programs.

Feasibility

Cost

Effectiveness

Organizational/ 

Administrative

Human/ 

Crew

Technology/ 

Engineering

Decision

Errors

Skill-based

Errors

Perceptual

Errors

Task/ 

Mission

Violations

Operational/

Physical 

Environment

Acceptability

Figure 5. The HFIX3 framework. 
GENERAL DISCUSSION

Historically, most safety professionals have been 

heavily acculturated by their own academic disciplines. 

While such indoctrination and training can facilitate the 

development of highly specialized interventions, it can 

also lead to “mitigation myopia,” in which provincial 

prevention measures prevail. This is not to say that such 

dogmatism is intentional. Rather, these biases or “cognitive 

constraints” placed on our creativity are quite simply the 

natural byproducts of the acculturation process associated with each academic discipline or society in which 

one lives. It should come as no surprise then that, while 

engineers have traditionally blamed the operator for errors and behaviorists have wanted to fault system design 

for inducing errors, the fixes have been predictable. That 

is, engineers tend to recommend engineering solutions, 

and psychologists tend to recommend behavioral/human-centered fixes. In a broader sense, even societies 

that emphasize individual responsibility for one’s own 

actions tend to emphasize punitive fixes.

In essence, safety recommendations are not simply 

based on empirical findings surrounding an accident. 

Rather, they are based on one’s philosophical view of 

what actually constitutes a “cause” of an event, coupled 

with one’s own biased view of how changes in human or 

system behavior can even be accomplished. Therefore, 

thinking “outside the box” when it comes to generating 

intervention strategies is extremely difficult to do; yet 

failure to do so can leave other potentially viable and 

effective alternatives unexplored.

What shall we say then, that we are forever helpless 

victims of our own acculturation and training? Absolutely 

not! Just like other cognitive biases (e.g., confirmation 

bias and hindsight bias), we must first acknowledge and 

recognize the potential impact that our own “mitigation bias” has on constraining our judgment and then 

generate tools and techniques for circumventing these 

constraints.

In the end, perhaps Reason (2005) put it best when 

he said, “[Human errors] are like mosquitoes. They can 

be swatted one by one, but they still keep coming. The 

best remedies are to create more effective defenses and 

to drain the swamps in which they breed.” Where the 

HFACS framework provides a view of the swamp, HFIX 

makes certain that we are draining the right swamp in 

the most efficient and thorough manner.

REFERENCES

Boquet, A., Detwiler, C., Holcomb, K., Hackworth, 

C., Wiegmann, D., & Shappell, S. (in review). 

Human error associated with air medical transport accidents in the United States. FAA Office of 

Aerospace Medicine Technical Report No. DOT/

FAA/AM-XX/XX. Washington, DC: FAA, Office 

of Aerospace Medicine.

Detwiler, C., Hackworth, C., Holcomb, K., Boquet, 

A., Pfleiderer, E., Wiegmann, D., & Shappell, S. 

(2006). Beneath the tip of the iceberg: A human 

factors analysis of general aviation accidents in 

Alaska verses the rest of the United States. FAA 

Office of Aerospace Medicine Technical Report 

No. DOT/FAA/AM-06/7. Washington, DC: FAA 

Office of Aerospace Medicine.

Holloway, C., & Johnson, C. (2004). Distribution of 

causes in selected U.S. aviation accident reports between 1996 and 2003. 22ndInternational System 

Safety Conference, Providence, Rhode Island, Aug 

2-6. 2004.

Johnson, C. (in review). On the over emphasis of human 

error as a cause of aviation accidents: Systemic failures and human error in U.S. NTSB and Canadian 

TSB aviation reports 1996-2003. Submitted to

Ergonomics.

Hall, J. (1996). As cited at Great Aviation Quotes. www.

skygod.com/quotes/safety.html.

National Transportation Safety Board. (2002). Retrieved 

Feb.19, 2002, from the Internet: www.ntsb.gov/

abt_ntsb/invest.htm.

Reason, J. (1990). Human Error. New York, NY: 

Cambridge.

Reason, J. (2005). Human error: Models and management. British Medical Journal, 320, 768-70.

Sarter, N. & Woods, D. (1992). Pilot interaction with 

cockpit automation: Operational experiences with 

the flight management system. International Journal 

of Aviation Psychology, 2(4), 303-21.

Sarter, N. & Woods, D. (1994). Pilot interaction with 

cockpit automation II: An experimental study of 

pilots’ model and awareness of the flight management and guidance system. International Journal of 

Aviation Psychology, 4(1), 1-28.
10

Shappell, S. & Wiegmann, D. (2003). Reshaping the 

way we look at general aviation accidents using the 

Human Factors Analysis and Classification System. 

Proceedings of the Twelfth International Symposium 

on Aviation Psychology, Dayton, Ohio.

Shappell, S. & Wiegmann, D. (2004). HFACS analysis 

of military and civilian aviation accidents: A North 

American comparison. Proceedings of the Annual 

Meeting of the International Society of Air Safety 

Investigators, Gold Coast, Australia.

Shappell, S., Detwiler, C., Holcomb, K., Hackworth, C., 

Boquet, A., & Wiegmann, D. Human error and 

commercial aviation accidents: A comprehensive, 

fine-grained analysis using HFACS. FAA Office of 

Aerospace Medicine Technical Report No. DOT/

FAA/AM-06/18. Washington, DC: FAA Office of 

Aerospace Medicine.

Stoop, J. (2002). Accident investigations: Trends, Paradoxes and opportunities. International Journal of 

Emergency management. 1(2), 170-82.

Wiegmann, D., Boquet, A., Detwiler, C., Holcomb, K., 

Faaborg, T., & Shappell, S. (2005). Human error 

and general aviation accidents: A comprehensive, 

fine-grained analysis using HFACS. FAA Office of 

Aerospace Medicine Technical Report No. DOT/

FAA/AM-05/24. Washington, DC: FAA Office of 

Aerospace Medicine. 

Wiegmann, D., & Rantanen, E. (2003). Defining the 

relationship between human error classes and 

technology intervention strategies. Final Technical 

Report AHFD-03-15/NASA-03-5.

Wiegmann, D. & Shappell, S. (2001). Human error 

analysis of commercial aviation accidents: Application of the Human Factors Analysis and Classification System (HFACS). Aviation, Space and 

Environmental Medicine, 72, 1006-16.

Wiegmann, D. & Shappell, S. (2003). A human error 

approach to aviation accident analysis: The human 

factors analysis and classification system. Burlington, 

VT: Ashgate.