A Human Factors 

Review of the 

Operational Error 

Literature

David Schroeder

Larry Bailey

Julia Pounds

Carol Manning

Civil Aerospace Medical Institute

Federal Aviation Administration

Oklahoma City, OK 73125

August 2006

Final Report

DOT/FAA/AM-06/21

Office of Aerospace Medicine

Washington, DC 20591
NOTICE

This document is disseminated under the sponsorship 

of the U.S. Department of Transportation in the interest 

of information exchange. The United States Government 

assumes no liability for the contents thereof.

___________

This publication and all Office of Aerospace Medicine technical reports are 

available in full-text from the Civil Aerospace Medical Institute’s publications 

Web site: 

www.faa.gov/library/reports/medical/oamtechreports/index.cfm
i

Technical Report Documentation Page 

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No. 

DOT/FAA/AM-06/21 

4. Title and Subtitle 5. Report Date 

A Human Factors Review of the Operational Error Literature August 2006 

6. Performing Organization Code 

 

7. Author(s) 8. Performing Organization Report No. 

Schroeder D, Bailey L, Pounds J, & Manning C 

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS) 

FAA Civil Aerospace Medical Institute 

P.O. Box 25082 11. Contract or Grant No. 

Oklahoma City, OK 73125 

12. Sponsoring Agency name and Address 13. Type of Report and Period Covered 

Office of Aerospace Medicine 

Federal Aviation Administration 

800 Independence Ave., S.W. 

Washington, DC 20591 14. Sponsoring Agency Code 

15. Supplemental Notes 

Work was accomplished under approved task AM-HRR-524. 

16. Abstract 

This report reviews available documents concerning research and initiatives to reduce operational errors (OEs). It

provides a brief history of OE investigation and reporting. It describes 154 documents published from 1960-2005 

and 222 OE reduction initiatives implemented from 1986 to 2005. Materials are classified by 1) type of study and 2) 

human and other contributing factors (using the JANUS taxonomy). An analysis of the literature identified several 

consistent findings. OEs were related to the amount of traffic (measured nationally rather than by position, early time 

on position, and pilot/controller miscommunications (especially hearback/readback errors). Initiatives included 

developing national and local QA activities, providing resources to supervisors to help them perform their jobs, and 

skills training to address controller mental processes. Many ATO initiatives involved controller training, teamwork, 

and communications. Research and operations seemed to focus on the same 6 areas: a) training and experience, b) 

teamwork, c) pilot-ATC communications, d) Human Machine Interaction (HMI) and equipment, e) 

airspace/surface, and f) traffic. This review concluded that, historically, much (sometimes redundant) research was 

conducted that generated little new information about why OEs occurred. Similarly, many initiatives were 

implemented, but the lack of a systematic follow-up prevented us from learning which were effective. This cycle will 

continue unless relevant data are obtained that can address underlying causal dimensions typically associated with 

human errors. Better data will allow conducting more informative, theory-based analyses. ATO must also continually 

assess the effectiveness of OE mitigation strategies. Research efforts, operational initiatives, and program outcomes 

must be monitored to avoid wasting resources by repeatedly conducting the same analyses, re-discovering the same 

intervention strategies, and addressing only the easy problems. Development of a safety culture requires obtaining 

better data about circumstances surrounding OEs; identifying individual, supervisory, and organizational 

contributions; and measuring the effectiveness of interventions.

 

17. Key Words 18. Distribution Statement 

 

 

ATC, Air Traffic Control, Operational Errors, OEs, Causal 

Factors, Incidents, Human Factors, Contextual Factors, 

Contributing Factors, Quality Assurance, Human Error, 

Error Mitigation, Controller Performance 

Document is available to the public through the 

Defense Technical Information Center, Ft. Belvior, VA 

22060; and the National Technical Information 

Service, Springfield, VA 22161 

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price 

Unclassified Unclassified 66 

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized 

iii

A Human Factors Review of the Operational Error Literature

EXECUTIVE SUMMARY

Purpose.To support the Federal Aviation Administration’s 

(FAA’s) Air Traffic Organization (ATO) goal to reduce 

operational errors, we were asked to review existing 

research literature on operational errors (OEs) so that 

candidate mitigation strategies could be identified for 

immediate deployment.

Method. This report covered the following areas: 

A brief history of OE investigation, classification, 

and reporting.

Collection of 154 OE-related documents covering 

a 45-year span from 1960-2005.

Documentation, review, and classification of 222 OE 

prevention and reduction initiatives implemented 

from 1986 to 2005. 

Classification of all materials using two methods

Type of study.

HF categorization using JANUS taxonomy.

Identification of knowledge gaps in research studies 

and initiatives.

Development of recommendations.

Research Results. We identified several findings by 

analyzing the literature. Some consistent research findings included:

The amount of traffic measured on a national basis is the 

single most important determinant of the frequency of 

OEs. However, while the amount of traffic increased at 

overall in the NAS, the amount of traffic in a sector when 

an OE occurs has generally remained unchanged.

A relatively high percentage of OEs occurred during 

the first 20 minutes on position. That relationship was 

consistent across options and during much of the day. 

While this result was often linked with position relief 

briefings, only a small percentage of OEs were attributed 

to the causal factor in the OE reporting form related to 

position relief briefings. 

Pilot/controller miscommunications were historically 

identified as a primary causal factor associated with OEs 

and hearback/readback errors were studied most often. 

Although analysis of recorded communications revealed 

that few hearback/readback errors resulted in an OE, 

a sizeable proportion of OEs were attributed to hearback/readback errors. One set of studies found a strong 

relationship between the complexity of the controller’s 

transmission and the probability of a readback error in 

the en route and TRACON environments.

•

•

•

•





•

•

•

•

•

Less-frequent, but also important findings include:

One study found a statistically significant relationship between the decline in supervisory staffing and 

the overall increase in OEs.

Another study found that higher per­centages of OEs 

occurred in moder­ate to heavy traffic complexity 

con­ditions, suggesting that the interaction of the 

amount of traffic with traffic complexity may affect 

OEs more than the amount of traffic alone. Other 

studies also identified sector complexity as a factor 

contributing to OEs.

Several studies have found that controllers with five 

years or less experience (after certification) were more 

susceptible to OEs than were more experienced 

controllers. 

Several studies related OEs to problems with perception and vigilance. However, a lack of information 

about these factors on the OE reporting form prevented an in-depth analysis of these causal factors. 

Although memory failures were associated with OEs 

in all ATC environments, much of the memory 

literature relevant to OEs was in the context of 

runway incursions. Several studies found memory 

failures to be the most frequently cited causal factor 

of tower OEs.

Results of Initiatives. Initiatives related to organizational 

and management issues primarily dealt with development and review of national and local QA activities. 

Initiatives described concerns about resources available 

to supervisors to accomplish their jobs and recommended 

additional supervisory training. Some initiatives focused 

on mental processes, especially those efforts addressing 

skills training.

Most initiatives introduced by the air traffic organization 

that related to contextual conditions involved controller 

training, teamwork, and communications. Both the research reports and OE reduction initiatives emphasized the 

same six contextual conditions (although not necessarily 

in the same order): a) training and experience, b) teamwork, c) pilot-ATC communications, d) Human Machine 

Interaction (HMI) and equipment, e) airspace/surface, 

and f) traffic. Although the specific topics addressed by 

research and the OE reduction interventions appeared 

to differ, trend comparisons suggested that research and 

operations focused on the same general areas.

•

•

•

•

•
iv

We also identified several roadblocks to OE reduction. 

These are briefly noted here and an expanded discussion 

is presented in the report.

Initiatives: The legacy Air Traffic Service (ATS) identified and implemented national, regional, and facility 

level OE reduction initiatives. These initiatives addressed error conditions identified through analysis of 

OE reports. While the reports indicated that several 

of these initiatives were successful, we found little 

documentation to support those claims. Without a 

method to track the effectiveness of initiatives, it is not 

possible to learn from past successes and failures.

Historical OE Database: The legacy ATS collected an 

extensive amount of information about OEs during 

their investigation process that was useful for guiding 

previous OE mitigation strategies. 

Incomplete data. The information historically 

gathered during the OE investigation process did not 

address many of the underlying causal dimensions 

typically associated with human error. 

Narrow research focus . Despite the complexity and 

dynamic nature of a controller’s tasks, most research 

focused on only one or two factors. For example, the 

conditions of the complex air traffic environment 

that make a controller more vulnerable to making 

mistakes versus remaining “error free” are unclear. 

Furthermore, objective measures of controller performance besides OEs are not available.

Similar analyses. Similar analyses have been 

repeatedly conducted on OE data that resulted in 

the same findings as previous analyses of those data. 

Analysts must be encouraged to review previous 

work and conduct analyses to examine how multiple 

factors interact to produce OEs. Cognitive models 

of controller performance should be used to guide 

these analyses.

•

•

•

Recommendations. A workgroup consisting of members 

of both the operational and research communities should 

be convened to address these 3 recommendations:

1. Expand OE data available for analysis

 a. Improve the OE data collected to better identify 

human factors associated with OEs. 

 b. Collect baseline data about normal air traffic 

operations to better evaluate conditions surrounding OEs.

 c. Develop objective measures of controller performance.

2. Improve research and analyses 

 a. Account for the interactive nature of factors 

contributing to OEs.

 b. Identify high impact causal factors where the 

payoff for mitigation is high. 

 c. Examine OEs relative to a safety culture.

3. Monitor interventions 

 a. Develop and execute a method for tracking 

and evaluating OE intervention strategies.

Conclusions. Identification of OE causal factors is difficult because they occur in a very small percentage of 

ATC operations. This review has shown that, historically, 

much (sometimes redundant) research was conducted 

that generated little new information about why OEs 

occurred. Similarly, many initiatives were implemented, 

but the lack of a systematic follow-up prevented us from 

learning which were effective. This cycle will continue 

unless steps are taken to 1) obtain relevant data to conduct 

more informative, theory-based analyses and 2) continually assess the effectiveness of OE mitigation strategies. 

The ATO must monitor research efforts and operational 

initiatives to avoid “reinventing the wheel” (repeatedly 

conducting the same analyses and initiatives) and “picking 

the low hanging fruit” (continuing to address apparently 

easy problems) and, thus, wasting increasingly scarce 

resources. The ATO must develop a safety culture by 

obtaining better data about the circumstances surrounding 

OEs and identifying the relative importance of individual, 

supervisory, and organizational contributions.
1

A Human Factors Review of the Operational Error Literature

INTRODUCTION

The Federal Aviation Administration’s (FAA’s) Air Traffic 

Organization (ATO) has a five year performance goal of 

reducing the annual number of serious operational errors 

(OEs) from a total of 637 in 2005 to no more than 563 by 

the end of 2009, equivalent to a rate of 3.18 per million 

activities (ATO, 2005). An OE occurs whenever there is 

a violation of aircraft separation minima that is the result 

of an element within the air traffic system (e.g., a facility 

procedure or an air traffic control specialist). A violation 

of separation minima may involve a) two or more aircraft, 

b) an aircraft and terrain or obstacles, or c) an aircraft 

landing or departing on a closed runway after receiving 

air traffic authorization to do so (FAA, 2006R). 

The ATO seeks to implement initiatives to reduce OEs. 

To best develop effective OE reduction strategies, it is important to look at past efforts to understand and mitigate 

OEs. In this report, we seek to a) review and classify past 

research reports according to type of study and human 

factors (HF) categories, b) identify HF categories not 

addressed by the literature, and c) review previous OE 

reduction initiatives.

We first describe the development of the FAA OE investigation process and the resulting data archive, because 

the data collected during the investigation dictate much 

of what is known about OEs. We then review the scientific literature regarding OEs and describe selected OE 

reduction initiatives. Finally, the discussion will integrate 

information from the review of the scientific literature 

and the OE reduction initiatives to identify targets of 

opportunity for further research and mitigation efforts. 

History of OE Investigation, 

Classification, and Reporting

This section describes the history of the processes used 

to identify, investigate, classify, and report OEs. After 

describing the OE investigation and classification processes, including several indices used to rate OE severity, 

we will discuss a process for analyzing the causal factors 

of OEs, called JANUS, which will be used to classify the 

OE literature. 

OE Identification

For a number of years, OEs (then called system errors) 

were identified through pilot reports, supervisor observations, reports from another controller, and self-reports 

from an involved controller. This led to considerable 

speculation about how many errors actually occurred 

and how severe they were.

Following the initial computerization of the nation’s en 

route air traffic control system, the FAA embarked on 

the development of conflict alert software. The resulting 

conflict alert system was designed to notify controllers of 

an eminent loss of separation between aircraft. On January 9, 1976, the conflict alert system was implemented 

at the 20 en route air traffic control centers (ARTCCs) 

for all aircraft flying above 18,000 feet. In December 

1978, the system was modified to cover all aircraft in 

en route airspace. Meanwhile, in early 1978, the first 

Terminal Radar Approach Control (TRACON) conflict 

alert system was established at the Houston terminal. By 

April 1978, the system was expanded to cover 62 facilities (FAA, 1998b).

During the mid 1980s, the Operational Error Detection 

Patch (OEDP, also known as the “snitch patch”) was 

introduced in en route facilities to alert Area Managers 

when a loss of standard separation occurred. The manager then determined who was responsible for the loss of 

separation. Thus, at en route facilities today, operational 

error detection and reporting is automatic. 

In TRACONs and towers, error detection is based on 

either self-reports or a report from another controller, 

supervisor, or pilot. The Department of Transportation 

Office of the Inspector General (OIG) issued a report 

(DOT, 2003) expressing concern over the potential 

for underreporting of errors at facilities where OEs are 

self-reported. ATO shares this concern and is working 

toward an automated system for the TRACONs that 

will help detect OEs. However, for now, the variability 

in OE reporting (and presumed underreporting in the 

terminal environments) needs to be taken into account 

when interpreting available OE data. 
2

OE Investigation Process

An approach for documenting and classifying air traffic 

control errors was developed in 1965 and described in 

FAA Order 8020. The Air Traffic Service (ATS) System 

Error Reporting Program established a formal process 

to investigate and classify what were then called system 

errors. As O’Connor & Pearson (1965) indicated, a reporting program for system errors should recognize the 

dynamic and complex nature of the air traffic system. It 

should take into account issues associated with equipment, 

personnel, procedures, and aspects of decision-making 

that influence performance. Since that time, periodic 

attempts have been made to update the OE reporting 

form to ensure that sufficient information is gathered 

to assess system safety and identify error-prone events, 

conditions, and procedures that could be modified to 

mitigate OEs. 

Currently, when an OE occurs, a two-stage investigatory 

process is initiated consisting of a preliminary and final 

investigation. As specified in FAA Order 7210.56 (2002), 

an employee who is aware that an OE may have occurred 

must immediately report the occurrence to any available 

supervisor or Controller-in-Charge (CIC). After the OE 

has been reported, the area supervisor will conduct a 

preliminary fact-finding exercise to determine the validity of the suspected OE. If validated, the area supervisor 

follows the procedures listed in FAA Order 7210.56 and 

records the information on the preliminary investigation 

report (Form 7210-2), which includes a limited checklist 

of OE causal factors. After the preliminary investigation 

report is submitted, air traffic management (ATM) at the 

facility determined to be responsible for the OE designates an Investigator-In-Charge (IIC) to conduct a more 

thorough analysis of the factors that contributed to the 

OE. The results of the IIC’s investigation are detailed in 

a final investigation report, which includes a checklist of 

OE causal factors comprising data posting, radar display, 

aircraft observation (at towers), communication, coordination, and position relief briefings. In addition to the 

information provided by the IIC, the manager may also 

offer comments about what may have led to the error. 

An OE may be associated with more than one causal factor. 

Based on data from the final investigation report, an OE 

Severity Index (SI) score (described below) is calculated 

by the ATO-Safety Office of Investigations using a standardized methodology and included in the final report. 

OEs in categories A (high severity) and B (high moderate 

severity) are subjected to greater organizational scrutiny 

than those in categories C (low-moderate severity) and 

D (low severity).

OE Classification

Simpson (1982) was one of the first individuals to attempt 

to classify the risk associated with OEs. He defined three 

collision risk zones based on the amount of vertical and 

horizontal separation between involved aircraft. A “near 

miss” was considered to occur if two aircraft passed within 

500 feet in both directions. The near miss had subzones 

of “critical near miss” (within 100 feet) and “potential 

near miss” (between 100 and 500 feet). From the edge 

of the near miss zone to 1,000 feet vertical and 3,000 

feet horizontal separation was designated as the “near 

approach zone.” The last zone, called the “separation 

infringement” zone, extended to the limits of vertical and 

horizontal separation that apply to the airspace. 

As part of a subsequent OE investigation report (FAA, 

1985), the Air Traffic Quality Assurance Staff introduced 

a system that classified errors as major, moderate, or minor, based on the amount of separation loss. Points were 

assigned for the degree of loss of horizontal (10 points) 

and vertical separation (10 points). A separate rating was 

provided for nonradar/Oceanic positions. A major error 

(20 points) involved less than 1⁄2 mile horizontal separation 

and less than 500 feet vertical separation. Moderate errors involved several different combinations of horizontal 

and vertical separation loss (14-19 points). An example 

would be 2 to less than 2 1⁄2 miles of horizontal separation 

(5 points) and 500 feet to less than 600 feet of vertical 

separation (9 points), for traffic under flight level (FL) 

290. Minor errors were assigned between 1-13 points. 

The total points used to classify severity for operations 

requiring less than 3-mile separation (i.e., in TRACON 

airspace) were slightly different. This process was only 

used until November of 1988.

In their initial report, the FAA Office of Aviation Safety 

(ASF) described a measure of potential risk using proximity (also called Root Mean Square, RMS, distance; 

Rodgers, Mogford, & Mogford, 1998). The proximity 

metric was calculated by computing the square root of 

the sum of the squares of the minimum horizontal and 

vertical separations between the aircraft involved. This 

is equivalent to the Euclidean distance between the two 

involved aircraft. Analysis of this measure suggested that 

the vast majority of OEs occurred at 5,000 feet of proximity or greater. This was true for ARTCCs, TRACONs, 

and control towers (FAA, 1988). 

OE Severity Index 

In 2001, an assessment of the safety risk associated with 

OEs was added to the OE investigation process. The 100-
3

point SI is based on four categories: Category A - high 

severity, Category B - high moderate severity, Category 

C - low moderate severity, and Category D - low severity. 

Classification of severity is based on the following factors: vertical and horizontal separation distances, relative 

flight paths, cumulative closure rates, and the level of 

ATC control. Because most of these factors are based on 

operationally relevant criteria associated with providing 

ATC services, the IIC is typically able to calculate the 

severity of ATC separation losses during the OE investigation. However, personnel assigned to the ATO Safety 

Office at FAA headquarters are responsible for making 

a final decision regarding the severity of each OE. As 

noted earlier, the ATO uses Category A and B events as 

organizational performance metrics. 

RISC Model 

The Runway Incursion Severity Categorization (RISC) 

model is an automated approach to categorizing the 

severity of the outcome of runway incursions (RIs) 

whether they are classified as OEs, pilot deviations, or 

vehicle/pedestrian deviations. The RISC model (Cardosi, 

Hannon, Sheridan, & Davis, 2005) reflects the likelihood 

that an RI could have resulted in a collision. The paths 

of the aircraft or vehicles involved in an RI are reviewed, 

and the closest horizontal and vertical proximity is determined. The model also takes into account avoidance 

maneuvers, visibility at the time of the RI, the types of 

errors made by the pilot and controller, and the time 

available for the pilot to respond. Cardosi pointed out 

that the RISC model should not be considered as a rating 

of “the adequacy of the ATC services or the severity of 

the operational (OE) that resulted in the incursion (pg. 

2).” The levels of risk of collision range from Category 

D, which have little or no chance of collision but met 

the definition of a runway incursion, to Category A, in 

which a collision is narrowly avoided. Events are assigned 

to these categories based on five operational definitions: 

a) available reaction time, b) the need for evasive or correction action, c) environmental conditions, d) aircraft 

or vehicle speed, and e) proximity. 

The goal of the automated rating model was to analyze 

recorded events and identify OE severity in a way that 

resembled the expertise of a group of Subject Matter Experts (SMEs). Cardosi et al. (2005) conducted an initial 

validation study that compared the ratings generated 

by the model with ratings provided by SMEs from the 

Office of Runway Safety and Operational Services. The 

ratings of the model matched the group’s ratings in 67% 

of the cases. However, when the C & D categories were 

combined, there was only a 5% disagreement between the 

model and the SMEs. The model has since been refined 

and FAA continues the validation process.

OE Reporting

Recording of OE investigational data began when the 

investigation process was initiated in 1965. In 1984, the 

Office of Aviation Safety (ASF) designed the National 

Airspace Information Monitoring System (NAIMS) and 

implemented it in 1985. One element of the larger automated NAIMS database is the Operational Error System 

(OES), which includes information derived from the OE 

reports, including possible causal factors. Since 1985, as 

part of their safety oversight, ASF has prepared annual 

reports that analyze the OES. The reports describe many 

variables of interest with regard to OEs: overall error rate, 

error rates by facilities, causal factors, and characteristics 

of errors/error rates by facility (FAA, 1988). 

The form used to obtain information about the background and causes of OEs has changed over the years, 

and the database containing that information has been 

maintained by different organizations. Changes in the 

form necessarily result in changes to variables included 

in the OES database. 

OE Research Considerations

An issue that must be considered when conducting research on OES data is that some data fields recorded on 

the forms have changed over the years as AT investigators, analysts, and managers changed their information 

requirements. For this reason, any analyses of the database 

should include checks to ensure that findings are based 

on characteristics of the underlying data rather than 

reflecting changes made to the form. Reports resulting 

from these analyses should indicate the authors’ actions 

to minimize errors resulting from changes to information in data fields. An assessment is currently underway 

to document when changes to the form (and thus to the 

database) occurred. Similarly, checks should be implemented to identify data entry errors. 

Another issue associated with research in this area is that, 

although several previous attempts have been made to 

quantify the level of risk associated with an OE, most of 

the reports included in this review (especially studies that 

examined the OES database) involved OEs that were studied without regard to the associated level of severity. Thus, 

the nature of the OE review and classification process limits 

data analysis and constrains interpretation of the results. 

Certain questions about HF issues cannot be addressed 

within the limitations of the existing database. 
4

METHOD

This section describes our classification of the OE reports 

reviewed for this project. First, the documents we acquired 

are described, followed by a discussion of two dimensions 

used to classify the documents: a) type of study conducted 

and b) an HF OE classification method. Categories within 

each dimension are described below.

Source Documents

One hundred fifty-nine OE-related documents were 

collected, covering a 45-year span from 1960-2005. We 

searched six aviation technical report databases to identify 

the relevant literature, using key words included in the 

phrase “air traffic control operational errors.” The six 

databases included: a) the European Organization for 

the Safety of Air Navigation Report Database, b) the 

University of Illinois Institute of Aviation’s Aviation Research Laboratory Report Database, c) the W.J. Hughes 

Technical Center Technical Reports Database, d) the 

Civil Aerospace Medical Institute (CAMI) Technical 

Report Database, e) the National Aeronautics and Space 

Administration (NASA) Technical Report Database, and 

f) the National Transportation Safety Board (NTSB) Report Database. Ovid software was also used to search the 

scientific literature contained in the American Psychological Association’s PsychINFO® database. PsychINFO® 

is a comprehensive international bibliographic database 

of psychology that contains citations and summaries of 

peer-reviewed journal articles, book chapters, books, 

dissertations, and technical reports written in the field of 

psychology and other reports addressing the psychological 

aspects of related disciplines. Proceedings from scientific 

sessions were also examined, as were unpublished internal 

FAA documents acquired through a network of professional contacts within the ATO. (To date, there is no 

centralized repository of internal FAA reports describing 

the implementation or evaluation of OE interventions.) 

We included internal FAA documents in this report if 

they addressed causal factors of OEs and not those solely 

focused on ATC or aviation safety. 

Figure 1 shows the distribution over time of documents 

included in the literature review. The increase in the 

number of research reports published after 1985 may be 

due, in part, to an interest in assessing the recovery and 

safety of the NAS following the ATC strike and resulting 

loss of 11,500 controllers in August 1981.

Although the graph suggests that most studies were 

conducted since 1980, it should also be noted that it was 

difficult to identify and obtain information about studies 

conducted before that time. The difficulty stemmed from 

the lack of a central repository for OE research reports. It 

is possible that early studies were conducted but were not 

archived for future reference. Thus, we may find ourselves 

initiating OE studies to rediscover what once was known 

but has been lost from institutional knowledge. Without 

an ongoing systematic means for archiving OE research 

reports, as they are made available, the institutional 

memory of past work is lost.

Figure 1. Distribution of OE-related documents by year of publication. 
5

Classification Procedures

Each document was reviewed and classified along two 

dimensions. The first was the type of study conducted. 

The second involved a comparison of the content with 

a set of human factors-related categories included in a 

recently developed OE analysis method called JANUS. 

Definitions of all classification categories are provided 

below. The results of the classification processes appear 

in Appendices A and B. 

Type of study

Studies were classified into one of four types: Descriptive 

studies (database analysis), theory-based studies, field 

studies, and experiments. This classification system was 

based on the way scientific inquiries are conducted—describe the phenomenon of interest, then develop a theory 

to understand what has been observed, then conduct 

field studies and experiments to better understand the 

mechanisms that affect the phenomenon. Each approach 

has advantages and disadvantages, depending on the goal 

of the study. Each report was classified into only one 

category based on the major thrust of the article. These 

were defined as follows:

Descriptive study (database analysis) 

A report was classified in this group if it described information contained in the OE database and did not necessarily 

test hypotheses. This category also includes exploratory 

studies (i.e., database analysis) performed on OE data to 

try to discover underlying trends and associations between 

variables of interest (e.g., association between the amount 

of traffic activity and the number of OEs).

Theory-based study 

A report was classified in this group if it presented and 

interpreted information about OEs in the context of a 

particular way of thinking (e.g., interpreted OE causal 

factors from an information processing perspective, 

from a perspective based on previous experience, or according to a predetermined organizing rationale [i.e., a 

taxonomy]). Using this definition, a theory-based report 

can range from a “thought piece” in which the author(s) 

speculates about cause and effect mechanisms based on 

his/her knowledge of the issue to a more formal model 

based on the results of empirical research. Examples of 

the former would be a “lessons learned” report written 

by a former certified professional controller (CPC) or the 

reflections of an HF researcher who has spent a career 

studying OEs. An example of the latter would be using a 

human error taxonomy (e.g., JANUS) to organize data so 

that additional insight can be gained about the OE.

Field Study 

A report was classified in this group if it described a study 

conducted under actual job conditions, although the 

nature of the tasks being observed and data being collected may vary. An example would be a time and motion 

study of actual ATC equipment usage conducted while 

controllers are actively controlling traffic, rather than 

participating in ATC simulations. Another example would 

be analyzing routinely recorded ATC data to establish a 

baseline of operations before a new piece of equipment 

is introduced. Because the data collections are conducted 

in actual working conditions, it is often difficult to manipulate the variables of interest to make cause and effect 

determinations. Thus, most ATC field studies fall under 

the category of observational research or baseline/ trend 

analyses and are often used to assess the acceptability of 

changes in equipment and/or procedures.

Experiment 

A report was classified in this group if it presented a study 

conducted in an experimental setting, such as an ATC 

simulation, and used the scientific method to investigate the effects of independent variables (i.e. predictor 

variables such as situation awareness, often called SA; or 

working memory) on OE occurrence (the criterion of 

interest). When the scientific method is strictly applied, 

experiments enable researchers to control the experimental setting, allowing measurement of the direct effects 

of predictor variables on the criteria. In other words, 

experiments enable researchers to test for cause-and-effect relationships.

JANUS as an Organizing Framework

The current OE reporting process produces enormous 

amounts of data about each OE. The resulting OE reports 

are largely descriptive of the circumstances involved in 

the events and do not provide a systematic approach to 

understanding the underlying causal human factors. When 

we began our literature review, we needed a method to 

organize the large number of OE reports identified in 

the literature search. Many of these reports are based on 

analyses of the OES database. 

To organize the OE reports, we borrowed the taxonomy 

from JANUS (Pounds & Isaac, 2002, 2003), a technique 

that integrated two aviation error classification systems: 

a) the Human Factor Analysis and Classification System, 
6

(HFACS; Shappell & Wiegmann, 2000b), and b) Human 

Error Reduction in Air Traffic Management, (HERA; 

EATMP, 2000) to enhance identification of OE causal 

factors. JANUS provides an inclusive HF framework 

that captures an extensive array of potential OE causal 

factors (see Figure 2). Using the JANUS taxonomy, the 

human factors associated with an OE can be classified 

in terms of the person performing a specific task with a 

particular piece of equipment in a specific environment, 

extending to supervisory and organizational influences 

that may affect the person’s performance. Using the categories and factors from this technique provided us with 

a comprehensive framework for classifying the reports 

included in the literature review.

Although originally developed to analyze ATC OEs, the 

JANUS framework has two useful properties that we 

exploited to organize the OE literature. First, the JANUS 

taxonomy enabled the literature to be classified into 

several HF-related areas. Thus, by simply classifying the 

literature according to the JANUS categories, one can see 

which human factors-related areas (e.g., mental processing, contextual conditions, supervision, or organizational 

influences) have received the most attention and which 

have been neglected. Second, when the type of study 

conducted (i.e., experiment, field study, descriptive, or 

theoretical) is used to classify the literature, it is possible 

to identify the amount of scientific rigor that has been 

dedicated to a given human factors-related area.

When classifying the literature using the JANUS taxonomy (see Appendix B), topical categories (defined by 

factors) were used instead of formal definitions. When 

a report addressed several important research topics, 

multiple JANUS categories were used to classify it. The 

JANUS categories were defined as follows:

Organizational Influences 

A report was classified in this group if it described organizational factors contributing to OEs (e.g., organizational 

climate, structure, national or regional policies and procedures, and/or resource management). Organizational 

issues involve FAA Headquarters, Service Areas (formerly 

called FAA Regions), etc. 

Management 

A report was classified in this group if it described 

management factors contributing to OEs (e.g., facility 

policies and procedures, general management planning, 

correction of known problems). Management issues 

involve facility managers, facility Operations Managers 

(OMs), and staff.

Supervision 

A report was classified in this group if it described factors related to controllers’ operational supervisors and 

discussed how they might contribute to OEs (e.g.,general 

O R G A N IZ A T IO N

C L IM A T E S T R U C T U R E P O L IC IE S & 

P R O C E D U R E S R E S O U R C E M G M T .

S U P E R V IS IO N

G E N E R A L 

S U P E R V IS IO N P L A N N IN G C O R R E C T IO N O F K N O W N P R O B L E M S T E A M W O R K N O N - C O M P L I A N C E I S S U E S

C O N T E X T U A L C O N D IT IO N S

M A N A G E M E N T

G E N E R A L M G M T P L A N N IN G C O R R E C T IO N O F K N O W N 

P R O B L E M S T E A M W O R K N O N - C O M P L I A N C E I S S U E S

C L IM A T E S T R U C T U R E P O L IC IE S & P R O C E D U R E S R E S O U R C E M G M T .

T R A F F I C A M B I E N T E N V I R O N M E N T & D I S T R A C T I O N S

A I R S P A C E / S U R F A C E T E A M W O R K

W E A T H E R I N D I V ID U A L / P E R S O N A L FA C T O R S

P I L OT -C O N T R O L LE R C O M M U N I C A T I O N S I N T E R P E R S O N A L / S O C IA L R E LA T I O N S

P I L OT A C T I O N S T R A I N I N G & E X P E R I E N C E 

P R O C E D U R E S H U M A N -M A C H I N E I N T E R F A C E & E Q U I P M E N T

D O C U M E N T S & M A T E R I A L S 

J A N U S J A N U S –– A J o i n t A n a l y s i s S y s te m fo r A T C A J o i n t A n a l y s i s S y s te m fo r A T C

N O N - C O M P L I A N C E I S S U E S

N O N -C O M P L I A N C E IS S U E S

TA S K D E S C R I P TI O N S

C O N T R I B U T IN G F A C T O R S C O N T R I B U T IN G F A C T O R S C O N T R O L L E R P E R F O R M A N C E C O N T R O L L E R P E R F O R M A N C E

P H IL O S O P H Y

R E S P O N S E E X E C U TI O N & A C T D E S C R IP TI O N

M E N TA L P R O C E S S E S

P E R C E P T I O N & V I G IL A N C E M E M O R Y P L A N N I N G & D E C IS I O N M A K I N G 

Figure 2. JANUS, a comprehensive HF framework for analyzing OEs.
7

supervision, planning, correcting known problems, teamwork, and supervisory non-compliance). 

Contextual Conditions 

A report was classified in this group if it described the 

context in which OEs occurred. The 12 categories of 

contextual conditions used were traffic/airspace, weather, 

pilot actions, pilot/ATC communications, procedures, 

documents/materials, ambient environment, teamwork, 

individual/personal conditions, interpersonal and social 

conditions, training and experience, and Human-Machine 

Interaction (HMI)/Equipment. 

Non-compliance Issues 

A report was classified in this group if it addressed purposeful disregard of rules or procedures. 

Task Descriptions 

A report was classified in this group if it described activities in which the controller was primarily engaged at 

the time of an OE, such as communicating with a pilot, 

control room communication, radar monitoring, tower 

observation, giving a position relief briefing.

Response Execution

Response execution refers to the outputs or actions based 

on the earlier processes, that is, how choices and plans 

were enacted as actions (either verbally or manually).

Mental Processes 

A report was classified in this group if it described 

controllers’ thought processes associated with OEs. 

These processes included: a) perception and vigilance, 

b) memory, c) planning and decision-making, and d) 

response execution. 

RESULTS

The results are organized into two sections. The first 

describes the classification of OE reports according to 

type of study. We follow with the coding of the literature 

according to the HF categories contained in JANUS. A 

description is provided for some of the studies contained 

1 In the initial work with EUROCONTROL, the term “contravention” 

was used instead of “non-compliance.” When FAA air traffic controllers 

reviewed the JANUS materials, they preferred to use the term “noncompliance” for this concept.

in the JANUS categories that were most frequently 

addressed by research.

The second section describes available information about 

AT initiatives that have been developed nationally, regionally, and at individual facilities to reduce operational errors. 

Once again, we review these interventions as they relate 

to the various JANUS categories.

Literature Review

Classifying the Literature by Type of Study

Appendix A lists the research documents classified according to the type of study conducted. The distribution 

showing the number of each type of study is shown in 

Figure 3. The majority of documents collected were 

classified as descriptive studies (database analysis; 44%), 

followed by theoretical studies (36%). Looking at the 

distribution in Figure 3, it is apparent that the more 

scientifically rigorous HF methods (experiments, 9%, 

and field studies, 11%) have seldom been employed to 

study OEs and their causes.

Descriptive Studies (Database Analysis)

Given that most of the empirical information available 

about OEs and their causes has come from descriptive 

analyses, it is imperative to examine that literature with 

an eye toward the quality of the information contained 

within the respective OE databases.

Eight databases that contain OE-related information were 

referenced in the descriptive literature. These were the: 

a) National Airspace Information Monitoring System 

(NAIMS), b) NASA’s Aviation Safety Reporting System 

(ASRS), c) NTSB’s Aviation Accident and Incident Data 

System, d) AT Quality Assurance Analysis and Reporting 

(QUASAR) database, e) the FAA’s System Effectiveness 

Information System (SEIS) database, f) FAA’s Operational 

Errors System (OES) database developed from preliminary and final OE reports, g) OE databases at individual 

FAA facilities, and h) the non-specific archival FAA OE 

database. Several of these databases are based on earlier 

versions of the current OES database. Although it is 

beyond the scope of this report to describe each of these 

databases in detail, it is important to note that there 

is no centralized quality assurance (QA) process that 

governs the quality of the data in each of the respective 

databases. This becomes especially problematic when 

trying to compare the results of OE studies that extracted 

data from different databases. For example, how might 
8

one equate studies that used information from the ASRS 

(which includes only self-reports of incidents) or from 

a generic FAA OE database at a field facility (which 

contains only data specific to that facility) with studies 

that used information from the NAIMS (which contains 

information about OEs from all FAA facilities)? 

Theory-based Reports

The authors of theory-based reports apply human factor 

principles and models of human error to speculate about 

the underlying causes of OEs. For example, Reason’s 

(1990) model and Rasmussen’s (1986) information 

processing model both propose generic models of human error. The merit of using generic models of human 

error to understand the underlying HF causes of OEs 

was discussed as early as 1977. Kinney (1977c) noted 

that to some degree, all OEs involve the basic human 

failings of memory, attention, and judgment, regardless 

of the situation. Thus, by developing interventions to 

address common human error vulnerabilities in general, 

the ATC system should experience an overall decline in 

the number of OEs. However, one of the shortcomings 

of this line of thinking is that generic models assume 

that humans are interchangeable. The problem comes 

when attempts are made to apply generic models to address specific events surrounding an OE. Generic models 

produce generic solutions and must be customized to 

the individual controllers/facilities/sectors to solve a 

particular problem.

Most of the theory-based reports we reviewed served as 

heuristics for designing and conducting HF experiments 

and/or field studies, rather than addressing a particular 

operational problem or recommending strategies for error mitigation. However, the recommended experiments 

and/or field studies were seldom completed.

Field Studies

In the field, it is neither possible nor desirable to manipulate working conditions to assess their influence on the 

likelihood of committing an OE. For that reason, none of 

the field studies identified for this literature review directly 

examined the causes of OEs but, instead, addressed OE 

precursors such as traffic volume (Hurst & Rose, 1978a), 

sector complexity (Hurst and Rose, 1978b; Grossberg, 

1989), problematic communications (Burki-Cohen, 

1995; Cardosi, 1993; Cardosi, 1994; Cardosi, Brett, 

& Han, 1996; Prinzo, 1996), SA (Endsley & Rodgers, 

1994), and the ATC culture (Jones, 1997). There is a 

large body of literature that addresses many of these and 

other HF issues where the emphasis is more about the 

effects of these factors on performance rather than on the 

occurrence of OEs. While these investigations provided 

us with greater insight into some of the complexities 

associated with the operational ATC environment, they 

provided little, if any, information about the linkage of 

these factors to OEs.

Figure 3. Distribution of OE-related documents by type of study. 
9

An exception to the typical limitations associated with 

field studies was the joint FAA-EUROCONTOL project 

conducted by Pounds and Isaac (2003) to test the JANUS 

technique. The FAA’s test of the technique encompassed 

twenty-nine ATC facilities which volunteered to collect 

JANUS data in parallel with the FAA OE investigation 

process. This represented a concurrent test of the JANUS 

OE analysis process because researchers interviewed ATC 

personnel to collect JANUS data while the facility’s staff 

conducted their investigation per FAA Order 7210.56. 

The technique was evaluated first based on the feasibility 

of the process and then for the information value of the 

data collected. 

When asked for their feedback, participants in the data 

collection indicated that, after going through the process, 

they had an overall positive opinion about the JANUS 

technique and thought that the questions asked were 

relevant to OE causal factors. In a separate activity, FAA 

managers and NATCA facility representatives from the 

participating facilities indicated that the method could 

be a useful tool if implemented operationally. A separate 

group of ATC management and staff personnel who were 

not associated with the beta test rated the data output from 

JANUS as more comprehensive, informative, practical, 

specific, and useful compared with the output from the 

current process used to identify OE causal factors (as 

described in FAA Order 7210.56).

To assess information value of the JANUS technique, 79 

OEs (64 from ARTCCS and 15 from terminals) obtained 

during the beta test (Pounds & Isaac, 2003) were used 

to compare the causal factors identified by JANUS with 

those identified by the current OE investigation process. 

One difference between the techniques is that the current FAA OE investigation process analyzes the OE as 

a unitary event whereas the JANUS technique permits 

splitting each OE into several “links in the chain” and 

analyzing each link separately. 

In the sample of 79 OEs, 133 causal factor items were 

reported (an average of 1.7 causal factors per OE) and 

were distributed in the following categories found on the 

current OE reporting form: Data Posting (9.8%), Radar 

Display (58.7%), Aircraft Observation (Towers Only; 

1.5%), Communication Error (25.6%), Coordination 

(4.5%), and Position Relief Briefing (0%). In this same 

sample, an average of 3.6 factors per OE was related to the 

mental processes of the controller working the traffic. The 

factors were distributed in the following categories: Perception and Vigilance (41%), Memory (15%), Planning 

and Decision Making (49%), and Response Execution 

(10%). (Because the JANUS technique permitted more 

than one category to be selected, the percentage summed 

across categories was greater than 100%.) Contextual 

factors that influenced the OEs were also identified for 

each “link” resulting in an average of 9.6 factors per OE. 

The factors were distributed in the following categories: 

Traffic & Airspace (49%), Weather (28%), Teamwork 

(26%), Pilot Actions (21%), Individual/Personal Factors 

(21%), Pilot-Controller Communications (20%), Ambient Environment/Distractions (18%), Workplace/Equipment/Human-Machine Interactions (13%), Procedures 

and Orders (11%), Training and Experience (10%), 

Supervision and Management (10%), Organizational 

Factors (10%), Interpersonal/Social Factors (5%), and 

Documents and Materials (0.3%).

Experiments

None of the experiments identified in the literature search 

studied OEs as the criterion variable of interest. Although 

it is possible to create an experimental manipulation in 

which the likelihood of committing an OE is high (but 

would not necessarily be guaranteed to occur), none of 

the experiments was designed in that manner. Instead, 

the focus was on assessing the impact of experimental 

manipulations on typical OE predictors, especially those 

related to SA, memory, and workload. The most common experimental manipulation involved varying traffic 

volume to assess aspects of controller performance. 

The primary hurdle involved in using simulations in 

experiments to assess OE occurrence is the infrequency 

with which OEs occur. Given the vast numbers of communications and control actions issued correctly during 

the course of a single day, we would expect to see few OEs 

occur during the course of a two-hour (or even longer) 

simulation. When we conduct an experiment to assess the 

effects of the amount of traffic on controller performance, 

we can clearly measure how it might influence SA and 

radar scanning. However, are these performance changes 

predictive of OE occurrence? For example, suppose an 

SA experiment reveals that controllers’ radar scanning 

and SA are degraded under conditions of high aircraft 

volume. Can we then extrapolate to say that reductions 

in the amount of traffic, which resulted in improved radar 

scanning and SA, are likely to reduce OEs? This conclusion would likely be invalid, since we already know that 

a significant number of OEs occur under lower traffic 

loads. Thus, organizational interventions designed to 

improve SA may be very successful at improving SA but 

appear to be ineffective in reducing OEs. Until we can 

identify aspects of controller performance that are predictive of OE occurrence, it is highly unlikely that we can 

utilize simulations and experimental methods to define 
10

effective OE intervention strategies. Some would argue 

that, given the dynamic nature of the air traffic environment, we are not likely to identify discrete performance 

predictors of OEs. 

We do not argue that conducting experiments is unimportant. On the contrary, continued simulation-based 

studies are needed to improve our understanding of 

the relationships among OE predictor variables. As we 

enhance our understanding of the role of experience, 

techniques, and amount of traffic on controller performance, we will gain a better understanding of predictors 

and prevention methods for OEs.

Classifying the Literature by JANUS Categories

Figure 4 shows the number of reports that fell within 

each of the high-level JANUS HF categories. Because a 

document could address several JANUS categories, the 

total number of citations included in Figure 4 exceeds 

the number of source documents. 

The OE literature fell primarily into two of the high-level 

JANUS categories: contextual conditions associated with 

an OE (44% of the reports) and mental processing of 

involved controllers (30% of the reports). The literature 

also addressed: organizational influences (7%), managerial influences (4%), supervisory influences (6%), controller non-compliance (1%), the controller task (1%), 

and controller’s response execution to choice and plans 

(7%). Rather than serving as an indication of the relative 

importance of a given JANUS category in the investigation of OEs, these percentages instead reflect the general 

interest of the researchers, as well as the availability of 

information about the topic areas.

While a short description of the literature in each category will be provided, the high percentage of documents 

classified as belonging to the Contextual Conditions and 

Mental Processes categories allows more discussion about 

the literature in those categories. For a complete list of 

reports classified in each of the JANUS categories, refer 

to Appendix B.

Organizational and Management Factors

Using a structured interview process based on the HFACS 

aircraft accident classification system (Shappell and Wiegmann, 2000b), the National Aviation Research Institute 

(Conner & Corker, 2001) interviewed 109 controllers 

at AT facilities that had high or low error rates. They 

found that, while interviewees rated controller performance as the most important factor in error causation, 

organizational factors were also viewed as being of high 

importance. Their list of most important factors also 

included organizational climate.

In an unpublished work group report about the causes of 

surface incidents, Kaulia et al. (1987) stated that, in addition to factors related to controllers, various organizational 

factors needed to be addressed. These included: a) the 

lack of updated Standard Operating Procedures (SOPs), 

b) an insufficient number of trained supervisors, and c) 

supervisors overburdened with administrative duties. 

Figure 4. Distribution of OE-related documents by JANUS category.
11

Supervision 

Going back as far as Kinney (1977b, 1977c), supervisory influences on ATC OEs were noted and echoed by 

various ATC work group reports that emerged in the 

1980s following the PATCO strike. Kaulia et al. (1987) 

identified these supervisory concerns: The inability to 

a) correct the actions of unsafe controllers, b) monitor 

controller performance, and c) ensure that controllers 

were properly trained. 

Another aspect of supervision that has received limited 

research attention, yet appears to have some potential 

for being a factor in reducing OEs, involves aspects 

of the supervisor’s duties and responsibilities. As part 

of a survey of supervisors, Kirk, Mayberry, and Lesko 

(1996), asked for an indication of their time allocation 

across eight functional areas (supervisor operations, manage resources, monitor performance, provide training, 

maintain expertise, foster teamwork, collateral duties, 

and personnel issues) and then asked them to rate the 

importance of each of the above functions on a 5-point 

scale. Supervisor time was primarily allocated to one of 

four functions: enhancing operational effectiveness, managing resources, interpersonal skills, and miscellaneous 

assignments. Supervising the operation was viewed as the 

most important function across all facilities. 

For en route centers, Kirk et al. examined the relationship between the importance ratings provided by the 

supervisors and OEs. Team supervisors who placed greater 

emphasis on the importance of managing resources and 

supervising the operation had fewer OEs. The emphasis 

the area supervisor in charge (ASIC) placed on several 

other functions (promotion of teamwork, monitor performance, provide training, and address personnel issues) was 

also significantly related to fewer OEs. The implication 

is that ASICs who rated the importance of these duties 

and responsibilities highly were more likely to have fewer 

OEs in their role as an ASIC. For example, 75% of the 

ASICs who said that managing resources was among the 

most important supervisory function had one OE or less 

in their role as a supervisor. Those who viewed managing 

resources as the least important supervisory function had 

a median number of two OEs. There was no evidence 

of a significant relationship between allocation of time 

between the supervisory functions and the occurrence 

of OEs. Additional research is needed to understand 

how these attitudes are translated into behaviors in the 

workplace. Using the OE database, Kirk, Mayberry, and 

Lesko (1996) suggested that there was no direct relationship between supervisory activity at the time of the error 

and the error’s severity. 

Kirk et al. (1996) also examined other aspects of the effects 

of supervisory, managerial, and organizational influences 

on OEs at en route centers. They based their assessment 

on: a) visits to selected en route centers, b) interviews 

with controllers, managers, and supervisors, and c) reviews of previous studies. Their extensive investigation 

of the role of supervisors involved several dimensions 

including supervisory duties, training and development, 

staffing changes, teamwork, supervisors’ background, and 

empowerment. Their assessment of changes in staffing 

took into account the potential influence of a government-wide effort (NPR) to increase the span of control 

for supervisors. Within the FAA, this meant that efforts 

were focused on increasing the employee/supervisor ratios from around 7:1 to the proposed government-wide 

target of 15:1. Subsequent reorganization efforts at the 

larger en route facilities led to a significant reduction in 

first-line supervisor staffing. For example, from 1992 to 

1995, supervisory staffing levels declined from around 

11% at ZTL to nearly 40% at ZLC. Over this same time 

period, there was a gradual increase in the number of OEs. 

Kirk, Mayberry, & Lesko (1996) found that there was a 

statistically significant relationship between the decline 

in supervisory staffing and the overall increase in OEs 

(explaining 14% of the overall variance). The relationship was slightly stronger when they took into account 

the ratio of supervisors to controllers in each of the en 

route centers. A similar relationship was not found for 

the TRACON or Tower comparisons. As a result of their 

observations, interviews, and analyses, the authors made 

the following recommendations.

1. Develop a standardized way of measuring sector complexity so that sector staffing can be adjusted based 

on this metric instead of the amount of traffic.

2. Develop action plans for high-OE sectors, which 

included the need for more managerial attention to 

be directed to these sectors and an increase in the 

level of supervision.

3. Monitor the performance of less experienced controllers more closely.

4. Match controller work assignments to the amount 

of expertise required.

5. Identify the supervisor-controller factors that affect 

controller performance during an OE.

6. Minimize the amount of managerial turnover to 

stabilize the culture.

7. Improve the evaluation of controller training and 

performance.

8. Focus more attention on team training.

NARI (Conner & Corker, 2001) found that controllers 

tended to view supervisory practices as a significant factor 
12

that both contribute to OEs and serve as a positive force 

to avoid OEs. Broach and Dollar (2002) provided an 

expanded look at the role of the supervisor-to-controller 

ratio (SCR) in OEs by also taking into account several 

organizational factors derived from the 1997 and 2000 

FAA employee attitude surveys. Their analysis focused 

on en route OEs that occurred between 1997 and 2000. 

They reported that two organizational factors (employee 

perceptions of equipment/facilities and performance 

management) along with the SCR accounted for 50% of 

the overall variance in OE rates. Outcomes from Broach 

and Dollar (2002), Kirk, Mayberry, and Lesko (1996), 

and other studies clearly demonstrate the importance 

of a number of supervisory/ managerial/organizational 

factors in OEs. Additional efforts are needed to further 

identify and clarify the manner in which these factors 

influence OE occurrence.

Contextual Conditions

Forty-four percent of the OE-related documents fell under 

the JANUS Contextual Conditions category. Figure 5 

shows the distribution of documents classified into the 

Contextual Conditions sub-categories. Five sub-categories 

accounted for 66% of the Contextual Conditions: a) the 

combination of the Traffic and Airspace2 sub-categories 

(15%), b) Pilot-ATC Communications (10%), c) Teamwork (12%), d) Training and Experience, (14%), and 

2 While discussed separately, we have combined the Amount of 

Traffic and Airspace sub-categories here because they are interactive 

in their effects.

e) Human Machine Interface (HMI) and Equipment 

(15%). The literature from each of these sub-categories 

will be discussed in some detail. For a listing of the 

reports classified as belonging to the remaining JANUS 

contextual conditions subcategories, the reader is referred 

to Appendix B.

Traffic/Airspace. Two contextual factors under traffic/airspace that have received the greatest attention since the 

1960s are the amount of traffic and airspace complexity. 

Traffic is a dynamic factor, while airspace (including airport surface characteristics) is static, so these two factors 

will be discussed separately. 

1) Amount of Traffic. Kershner (1968) conducted an initial 

study on the amount of traffic and its relationship with 

age, workload, and time-on-shift in 1965-66 OEs. Spahn 

(1977) looked at changes in air traffic levels from 1974 

to 1976. Spahn pointed out that a higher percentage of 

OEs occurred under light and moderate traffic rather than 

during heavy traffic. From the 1970s through the early 

1980s, the trend was toward an increased percentage of 

OEs under light and moderate versus heavy traffic. Using information from previous studies, Schroeder (1982) 

provided information regarding changes in the percentage of OEs occurring under the three traffic levels (see 

Figure 6). From 1965 through 1980, there was a gradual 

increase in OEs occurring under light traffic levels with 

a corresponding decline in the percentage occurring 

under heavy traffic levels. However, the categorization of 

air traffic required the investigator to make a subjective 

Figure 5. Distribution of OE-related reports by sub-categories of JANUS Contextual Conditions.
13

rating. There was little assurance that investigators within 

and across facilities used the same criteria for judging the 

amount of traffic at the time of the OE. 

As part of a MITRE study to assess the validity of common statements regarding the relationship between traffic 

and OEs, Lowry, MacWilliams, Still, & Walker (2005) 

conducted several analyses of the NAIMS 1999-2003 OE 

database. The percentage of OEs that involved only two 

aircraft was small (1% in en route, 4% in TRACONs, 

and 7% in towers). In en route facilities, 21% of the OEs

involved 7-8 aircraft. Our analysis of the database suggests 

that this percentage has remained relatively consistent 

across time. While not discussed in their report, the 

figures used to relate OEs to traffic suggest that slightly 

more than 50% of the OEs in TRACONS and towers 

involved between 4-6 aircraft.

A second question addressed by Lowry et al. (2005) 

was whether OEs occur just prior to the traffic peak. To 

answer this question, the authors analyzed 52 OEs that 

occurred in FY02 and determined the traffic count 30 

minutes prior to and 30 minutes following the OE. In 

61% of the OEs, traffic had already peaked when the OE 

occurred, in another 30% of the cases the OE occurred 

prior to the peak in traffic.

As evident in reports that described aspects of the 1986 

and 1987 OEs (FAA, 1988), trends in OE occurrence 

associated with the day of the week and time of day appeared to be related to overall changes in the amount of 

traffic. OEs tended to peak on Wednesdays and Thursdays, with the lowest OE rates evident on Saturdays and 

Sundays. Throughout the day, OEs increased with a peak 

around 9 to 10 a.m., leveled off, followed by a secondary and somewhat higher peak around 4 to 6 pm. With 

respect to the controller’s 8-hour workday, the lowest 

number of OEs occurred during the seventh and eighth 

hour of the shift. These trends are generally consistent 

across years and are evident in the current OE database 

(Lowry et al., 2005). 

2) Airspace complexity. Besides considering the role of the 

amount of traffic in OEs, Spahn (1977) also looked at the 

influence of traffic complexity. At the time, information 

included in the OE database about both the amount of 

traffic and traffic complexity required subjective judgments to be made by the investigator. These two measures, 

the amount of traffic and traffic complexity, are not 

independent. As the amount of traffic increases, there 

is generally a corresponding increase in traffic complexity. Spahn’s (1977) data suggest an interaction between 

Amount of Traffic

Figure 6. Distribution of amount of traffic associated with OEs by year. 

0

10

20

30

40

50

60

'65 '66 '69 '71 '74 '75 '76 '77 '78 '79 '80 '65 '66 '69 '71 '74 '75 '76 '77 '78 '79 '80 '65 '66 '69 '71 '74 '75 '76 '77 '78 '79 '80

Percent of OEs

Light Moderate Heavy
14

traffic complexity and the amount of traffic, with higher 

percentages of OEs occurring in moderate and heavy 

traffic complexity conditions. This suggests that the 

interaction of amount of traffic with traffic complexity 

may affect OEs more than the amount of traffic alone. 

However, there are significant questions concerning the 

definition and measurement of both traffic and airspace 

complexity.

Most of the research on airspace complexity has been 

targeted at developing measures of dynamic density, 

an indicator of future sector activity, rather than OEs. 

However, several reports have examined the relationship 

between sector complexity and OEs at en route facilities. 

For example, Grossberg (1989) found a correlation of .44 

between an index of sector complexity based on the top 

four complexity factors (complex control adjustments, 

climbing and descending flight paths, large airspace, and 

mix of aircraft types) and the number of operational errors 

at sectors in the Chicago Center. Rodgers, Mogford, and 

Mogford (1998) related Mogford’s (Mogford et. al., 1994) 

16 complexity factors to the incidence of operational errors 

at Atlanta Center (ZTL) sectors. They found that a variable that characterized sectors as having a low, medium, 

or high incidence of OEs predicted several of the sector 

complexity factors. They also found that several sector 

complexity factors (notably radio frequency congestion 

and special use airspace activity) predicted the number 

of errors per sector in an analysis of ZTL data. Thus, 

they concluded that sector characteristics appeared to 

be related to OEs.

NARI (Conner & Corker, 2001) defined Operational 

Complexity as “the characteristics of air traffic control 

processes, environment, and the users who operate in 

it, including the airspace, equipment, traffic, weather, 

aircraft performance, military, information, frequencies, 

etc.” Interviews conducted during the NARI OE study 

(2001) identified Operational Complexity as the second 

highest-rated OE causal factor. Moreover, Pounds and 

Isaac (2003) found that 49% of the errors identified in 

their JANUS validation study were related to either sector 

or traffic characteristics. 

Lowry et al. (2005) conducted several analyses of the 

1999-2003 database that focused on issues often included 

in assessments of airspace complexity (sector size, sector 

configuration, and traffic mix). With respect to sector 

size at en route centers, their analysis supported previous 

research by Rodgers et al. (1998) indicating that smaller 

sectors, with a volume of 50,000 or less volume units, 

comprised 50% of the sectors, handled 52% of the traffic, 

and experienced 64% of the OEs. With respect to traffic 

mix, neither regional jets nor business jets were involved 

in a disproportionate number of OEs. However, in terminal facilities the authors found that turboprops were 

disproportionately involved in OEs in three of the four 

facilities included in the analysis. An additional analysis 

was focused on the impact of a National Choke Point 

Initiative in which two sectors at the ZID and ZOB en 

route centers were redesigned to improve efficiency. While 

Massimini, Nene, Gormley, & Stevens, (2001) found 

that the redesign reduced demand on adjacent sectors 

and improved efficiency, Lowry et al. (2005) found that 

the OE rates two years prior to and two years following 

the redesign were the same.

Pilot-ATC Communications. Most empirical ATC studies 

have been conducted to assess the phraseology associated 

with Pilot-ATC communications. Initial studies involving 

analyses of the FAA/NASA Aviation Safety Reporting 

System (ASRS) database were conducted about 20 years 

ago (Grayson & Billings, 1981; Monan, 1980; Monan, 

1983). Initial efforts were focused on factors associated 

with miscommunications (Grayson & Billings, 1981). 

They included pilot expectations, similar call signs, transposing numbers, garbled phraseology, and inaccurate or 

incomplete messages. Monan (1980, 1983) investigated 

the role of similar call signs as well as readback/hearback 

errors. Cardosi, Falzarano, and Han (1998) conducted 

a more recent review of the communications error information contained in the ASRS reports as a means of 

identifying factors that contributed to the errors. The 

most common factors involved similar call signs, pilot 

expectations, and high controller workload. While not 

focused solely on OEs, these studies provide an overall 

perspective for understanding some of the communications issues associated with OEs. 

Cardosi and her colleagues performed much of the research focused on the role of voice communications as 

a critical safety link in the NAS. A recent report on the 

metrics of communication performance by Cardosi and 

DiFiore (2004) provides an excellent summary of their 

work, some of which is now a decade old and, thus, may 

represent a conservative estimate of communications in 

today’s environments. The metrics incorporated in their 

studies included: (i) number of controller transmissions 

per minute, (ii) number of clearances issued per minute, 

(iii) characteristics of pilot responses to clearances, (iv) 

percentage of clearances that result in a pilot readback 

error, (v) percentage of readback errors that are not 

corrected by the controller, (vi) percentage of controller transmissions that need to be repeated due to pilot 

requests, (vii) factors associated with miscommunications, 

and (viii) the time required to successfully transmit an 
15

instruction to maneuver for traffic avoidance in the en 

route environment. Each of the studies involved an 

analysis of approximately 50 hours of voice tapes from 

tower cabs, TRACON, and en route air traffic control 

facilities (Cardosi, 1993; Burki-Cohen, 1995; Cardosi, 

1994; and Cardosi, Brett, & Han, 1996).

We have elected to summarize the overall results described 

in Cardosi and DiFiore (2004) rather than describe more 

detailed outcomes from each of the studies. Evidence 

suggests that the number of controller transmissions 

varies by environment, from a low of just under 2 to 4.5 

communications per minute in TRACONs and about 

8 communications per minute for the tower Ground 

Control position. Of course the average number of communications for Ground Control may vary considerably 

from larger to smaller towers. Pilot responses to controller 

communications varied considerably; at en route centers 

some 70% contained a full readback, while at the tower 

Local Control position, less than 30% involved a full 

readback.

A consistent finding in Cardosi’s work was that the rate 

of communication errors, defined as readback errors and 

requests for repeated communications, was low across 

facility types. The readback error rate was consistently less 

than 1%; requests for repeated transmissions was similarly 

low, with the exception of the en route environment, 

where the frequency was slightly higher, at 1.4%. While 

a readback error occurred once every 2.5 hours for local 

controllers, at TRACONs the rate was around one every 

half-hour. The vast majority of these errors were caught 

and corrected by the involved controllers.

Causal factors associated with communication errors 

(few of which lead to OEs) included attempting to 

convey too much information, frequency congestion, 

workload, speech rate, poor grasp of the English language 

by international pilots, and similar-sounding aircraft call 

signs. Analyses of the data by Cardosi and her colleagues 

suggest that there is a strong relationship between the 

complexity of the controller’s transmission and the probability of a readback error in the en route and TRACON 

environments. For en route transmissions, the readback 

error rate doubled as complexity increased from three 

to four elements. While 4% of controller transmissions 

involved five or more elements, they contributed 26% 

of the errors in en route facilities. A similar finding was 

observed in TRACONs.

Although nearly all communications are free of errors, 

communication errors play a significant role in the overall 

OE process. Rodgers and Nye (1993) found that 36% 

of en route OEs involved communication errors. In the 

tower environment, Cardosi and Yost (2001) found that 

41% of OEs involved communication errors, with 19% 

involving pilot-controller miscommunications.

Baseline field studies such as those conducted by Cardosi 

provide a necessary perspective on the infrequency of 

communication errors in the course of normal operations. 

For example, imagine an intervention designed to reduce 

hearback/readback errors. Using a pre- vs. post-intervention evaluation design, we could determine the effectiveness of the intervention at reducing hearback/readback 

errors. The effectiveness of the intervention is not based 

on the reduction of hearback/readback errors that occur 

during OEs. Rather, the intervention is being evaluated 

against hearback/readback errors that occur during all 

communication exchanges. Given that less than 0.5% 

of all communication exchanges between controllers and 

pilots result in uncorrected hearback/readback errors, and 

that only a fraction of this 0.5% of communications will 

actually involve an OE, it would be difficult in a field 

study to evaluate the effectiveness of such an intervention 

on the reduction of OEs, per se. This is an important 

factor to keep in mind as we review the literature. Causal 

factors commonly associated with OEs (i.e., data posting errors, radar display errors, tower observation errors, 

communication errors, coordination errors, and position 

relief briefing errors) occur at some base rate within normal operations. Even if an intervention reduces the base 

rate with which these errors occur, the extent to which 

the intervention will have a significant impact on OEs 

remains to be seen.

The safe and efficient flow of traffic in the NAS relies 

heavily on voice communications between pilots and 

controllers. As evident from the results presented earlier, 

OEs are frequently associated with readback errors. In their 

summary report, Cardosi and DiFiore (2004) indicated 

that communication errors appear to be closely linked 

with the complexity of the controller’s transmission. While 

this research has been focused on the controller side of the 

communication link, less information is available about 

factors that contribute to a pilot’s failure to understand 

and respond properly to a communication. The exception involves research focused on age differences in pilot 

responses to air traffic control instructions. While these 

studies were not focused on OEs, they provide insight 

into factors that can lead to miscommunications and 

possible OEs.

Since the complexity of a controller’s transmission is 

linked with communication failures, we can assume that 

working memory capacity plays an important role in 
16

pilot controller communications. Research has demonstrated an age-related decline in working memory capacity (Baddeley & Hitch, 1974) and speed of processing 

(Salthouse, 1980) as well as an increased susceptibility 

to interference or distraction (Engle, Kane, & Tuholski, 

1999). Research has focused on determining the extent to 

which pilot expertise may reduce the typical age-related 

decline in the recall of ATC communications. Previous 

research demonstrated that older pilots exhibited a slightly 

lower ability to recall ATC communications (Morrow, 

Leirer, & Yesavage, 1990; Morrow, Yesavage, Leirer, & 

Tinklenberg, 1993; Taylor et al., 1994; Yesavage et al., 

1999). However, Morrow & Leirer (1997) and Morrow, 

Leirer, Altieri, & Fitzsimmons (1994) found that pilot 

expertise may moderate age differences in recalling ATC 

communications.

In a recent study (Taylor et al., 2005) the authors set out 

to assess the role of cognitive ability and domain-specific 

expertise as they influence age-related differences in pilot 

responses to ATC messages. In addition to assessing the 

role of pilot expertise, as determined by pilot ratings (VFR, 

IFR, and CFII/ATP), the authors varied the length and 

speech rate of the ATC instructions for pilots who were 

not allowed to write down the communications while 

flying a simulator. The 97 pilots in the study were split 

into two groups for the age comparisons (45 to 57 and 57 

to 69). The results demonstrated that pilots recalled 2.05 

of 3 instructions when the instructions were presented at 

a normal speech rate, but performance declined to 1.88 

instructions recalled for the fast speech rate. Also, when 

3 instructions were presented, an average of 2.05 was 

recalled, but longer ATC messages led to lower accuracy 

(only an average of 1.87 instructions was recalled). It 

would seem likely that pilots would recall ATC communications better if they were allowed to write down 

elements of the communications as is commonly done 

while flying.

While this study showed that expertise led to improved 

recall of ATC communications, there was little evidence 

that it reduced the age-related decline in performance. 

They found that the effect of age on recall accuracy was 

largely explained as an “age-associated decrease in working 

memory span, which in turn was explainable as decreases 

in both speed and interference control (pg. 117).” These 

results, along with those of Cardosi and her colleagues, 

can be used to support interventions designed to improve 

communications. The emphasis of these interventions 

should be on a standard speech rate and shorter instructional messages. The results also emphasize the importance 

of involving pilots in efforts to reduce hearback/readback 

and other communication failures.

Teamwork. ATC teamwork is generally discussed in the 

literature relative to either (a) controller task coordination, or (b) interpersonal skills. Discussions of task 

coordination primarily involve controller team members 

functioning within dyads, such as a radar controller and 

radar associate, a ground and local controller, or two 

controllers involved in inter-sector coordination. One 

of the more visible aspects of task coordination occurs 

during position changes. 

Position Relief Briefings. One aspect of teamwork is the 

transfer of position responsibility that takes place when 

one controller relieves a second controller. This process 

is referred to as a position relief briefing. The current 

controller handbook (FAA, 2006) contains rather specific 

instructions regarding the process for accomplishing 

a position relief briefing. During this time, a relieving 

controller observes the traffic situation that he/she will be 

taking over. The controller being relieved then follows a 

checklist to ensure that all relevant information has been 

transferred to the relieving controller.

The current NATCA contract (FAA, 1998a) states, “Unless operational requirements do not permit, employees 

shall not be required to spend more than two (2) consecutive hours performing operational duties without 

a break away from operational areas.” This means that 

during the course of a 24-hour day, several position relief 

briefings are conducted as one controller leaves a position 

and another takes over. 

The identification of position relief briefings as a causal 

factor has been a part of the OE report form for many 

years. Kinney (1977b) indicated that inadequate position 

relief briefings were cited as primary and contributing 

causes of what were then called system errors (now called 

OEs). Their observations suggested that many position 

relief briefings were too short. At that time, he indicated 

that specific rules for position relief briefings “... are not 

in the training curriculum, not in Handbook 7110.65, 

and not on proficiency evaluation checklists. (p. 6-1).” 

He also noted that facility managers and supervisors have 

taken actions to improve relief briefings. Recommendations were provided to improve performance during 

position relief briefings. 

Despite the concern surrounding position relief briefings, data extracted from the OE reports suggest that the 

frequency with which a position relief briefing is cited as 

a contributing factor to an OE is relatively small when 

compared with several other factors. Schroeder (1982), 

in his review of the 1979 and 1980 version of the OE 

database, found that position relief briefings were a direct 

cause in 0.89% (1979) and 0% (1980) of en route OEs. 
17

For terminal operations, the percentages were only slightly 

higher (1.07% and 1.06%, respectively). 

The later analysis of OEs conducted by the Office of 

Aviation Medicine in 1988 cited incomplete position 

relief briefings as a common, error-prone procedure. The 

most recent annual report from the NAIMS (FAA, 2003) 

indicated that position relief briefings were a factor in 

12% of OEs. While an emphasis on position relief briefings has been a frequent intervention strategy, a survey 

of controllers, supervisors, and managers conducted by 

Mayberry et al (1995) suggested that less than 10% of 

controllers and supervisors viewed position relief briefings 

as one of the top reasons for an OE. Analyses by Lowry et 

al. (2005) also found that position relief briefings played a 

limited role in most OEs. While analyses of the database 

indicate that position relief briefings are causal factors in 

a small percentage of OEs, the consistent emphasis on 

initiatives related to this factor (which will be discussed 

later) suggest that operational personnel believe this is 

an important factor in OEs. 

When teamwork addresses interpersonal skills, the notion 

of the team extends beyond a given dyad to include surrounding facility personnel. This extension is addressed 

through documents about team culture and plays a 

prominent role in the Air Traffic Team Enhancement 

program (ATTE). Although ATTE was not designed 

as an OE mitigation effort, per se, it has been used in 

some cases to address teamwork as an OE causal factor. Anecdotal information from ten facilities suggests 

that OEs were reduced when ATTE was successfully 

implemented—although other variables confounded 

these results and made clear interpretation difficult. The 

ATTE program is currently undergoing further refinement and testing.

Efforts to determine the relationship between time-onposition and the occurrence of an OE are often considered a part of the position relief briefing issue. However, 

there may be factors that contribute to the occurrence 

of OEs shortly after assuming a position other than the 

nature and length of the position relief briefing. In the 

first quarter report of the Air Traffic Quality Assurance 

Staff (FAA, 1986a), the analysis of 1985 OE data revealed 

that more than 20% of the OEs occurred in the first 15 

minutes on position. Pounds and Ferrante (2003), in 

their review of the 1997 to 2000 OE database, indicated 

that 9% of OEs occurred within the first 5 minutes, 

18% in the first 10 minutes, and 35% within the first 

20 minutes on position. Lowry et al. (2005), in their 

effort to debunk myths associated with OEs, found in 

an analysis of the NAIMS OE database that about 15 

to 18% of OEs occurred in each of the three 10-minute 

time periods during the first 30 minutes on position. 

The results were similar for en route, TRACON, and 

tower facilities. They correctly pointed out that it is difficult to fully interpret the implications of these findings 

without knowing the average time on position. They also 

pointed out that, in their opinions, there were two possible explanations—that the position relief briefings were 

inadequate or that the controller assumed the position 

during a busy traffic period. A subsequent review of OE 

narratives conducted in support of their study tended to 

refute the importance of the position relief briefings, as 

does the previous literature. 

Our analysis of the OE database suggested that traffic 

volume remains relatively consistent across time on position. A possible alternative explanation to the Lowry 

conclusion is that a portion of these OEs could be attributed to the complexity associated with a controller 

“getting the picture” or “getting up to speed” when 

taking a position. Another possibility is that we do not 

adequately understand whether additional information 

needs to be communicated when a controller assumes a 

position. In either case, the data clearly demonstrate that 

the tendency for a high percentage of OEs to occur early 

on position has been consistent for the past two decades. 

However, there is little evidence to document the extent 

to which the position relief briefing is a prominent factor 

in their occurrence. 

Training and Experience. Historically, controllers learned 

basic air traffic control procedures at the FAA Academy 

using a generic airspace. More recently, new facility trainees 

have taken other types of training such as that provided 

by military facilities or College Training Initiative (CTI) 

schools. When a new trainee arrived at a facility, they were 

called “developmentals” whose skills are further refined 

during each facility’s developmental training program. 

Developmental training consisted of classroom activities, 

laboratory simulation problems, and on-the job training 

(OJT) in which the developmental developed his/her skills 

while under the supervision of a formal OJT instructor. 

Developmentals eventually learned to control traffic 

independently on multiple pieces of airspace (called “positions” in a TRACON and “sectors” in en route facilities). 

TRACON and en route developmental controllers also 

learned to perform increasingly difficult duties, starting 

with entering flight data through radar associate duties 

and ending with radar duties. Tower controller training 

was similar in that it started with the flight data/clearance 

delivery position but later split into two equally difficult 

positions, the ground and local controllers, that had to be 

learned separately. Upon the completion of field training, 

a developmental became a certified professional controller 
18

(CPC) by becoming certified on each position required at 

the facility or area to which he/she was assigned. Because 

of the requirement to become certified, CPCs can only 

control traffic on sectors (in that airspace) on which they 

have been certified.

Two reports, separated by nearly two decades, presented 

similar findings about the relationship of controller 

training and experience to OEs. Spahn (1977) reported 

that controllers with five years or less experience (after 

certification) were more susceptible to OEs than more 

experienced controllers. Mayberry et al. (1995) qualified 

this susceptibility by stating that controllers with six or 

fewer years of experience were about 30% more likely to 

have an OE compared to more experienced controllers. 

Broach and Schroeder (2005), in their analysis of en 

route OE data from 1997 to 2003, found that younger 

controllers with less experience had the highest OE rates. 

However, Lowry et al. (2005) concluded that insufficient 

information was available to draw a conclusion about the 

relationship between experience and OEs. 

Findings such as these suggest that greater attention 

should be spent monitoring the performance of less 

experienced controllers. Though not a research report, 

Boone’s (2004) memo on OEs at Indianapolis Center 

(ZID) suggested that a) a significant percentage of ZID 

OEs were attributed to controllers either during OJT or 

within two years of certification, and b) when the interval 

between certification on all Radar Associate positions and 

the commencement of Radar training was increased, that 

percentage was reduced. 

Although there is considerable literature that describes and 

tracks controller training across time, there is an absence 

of literature on controller performance. For example, all 

controllers certified to control traffic at a given sector are 

considered to be equivalent. Furthermore, there appear 

to be no studies that have reported how controller skills 

improve or degrade with the passage of time. 

Broach and Schroeder (2005) assessed error occurrence 

across time for a group of en route controllers. In a follow-on analysis, the authors identified 5,559 controllers 

employed at an en route center throughout the seven-year 

time period. Of that group, 3,852 (about 69%) were 

OE-free, and 21% had a single OE. This means that 9% 

of the controllers had between two and six OEs in seven 

years. We do not know how these data extrapolate across 

the remaining years of a controller’s career, except that the 

overall OE rate tends to decline across age. Additional 

information is needed to understand what helped 69% 

of the en route workforce to remain OE-free during this 

time period. What distinguishes their performances from 

those controllers who had one or more errors?

There are indications that controllers who made OEs may 

perceive the error process differently than those who did 

not. NARI (Conner & Corker, 2001) found some differences in the perceptions of interviewed controllers who 

previously had zero, one, or multiple OEs. For example, 

controllers with no OEs considered ATC procedure-following to be a significant OE causal factor, but those with 

multiple errors gave it the lowest weight. On the other 

hand, controllers with no OEs rated ATC conditions as 

noncontributory, while controllers with one or multiple 

errors rated it higher. 

HMI and Equipment. Given the considerable expense 

associated with equipment acquisition, it is not surprising that the largest number of documents related to 

Contextual Conditions fell within this category. For the 

most part, the documents that addressed OEs from an 

HMI perspective were theory-based because the authors 

theorized about or described the impact that new and/or 

existing technologies have on controller performance (c.f., 

Hopkin, 1989). However, the effects of new technologies 

on OEs were not specifically studied. Instead, much of 

the focus was directed at problems controllers may have 

after the new technologies are implemented in the field. 

Thus, the focus was more on controller performance, in 

general, rather than on OEs.

Because most of the documents in this sub-category were 

associated with new technologies, it was disappointing not 

to find documents on field studies that tracked the impact 

that the technologies had on OEs. While it is known that 

during the early phase of field implementation, some 

controllers incur an OE due to HMI problems (typically, 

when this happens the equipment and not the controller 

is charged with the OE); once the problems are addressed, 

it appears that further evaluation is not conducted of 

the impact of the technologies on OEs. One exception 

was a concern raised by Corker (2004) about potentially 

negative effects resulting from changes in procedures and 

utilization of the new URET display. 

If a new technology is expected to have a marked impact 

on OE reduction, then once implemented, there should 

be a discontinuity in the OE error rates between preand post- implementation. Despite the relatively simple 

analyses needed to conduct a pre- and post- implementation study, few studies have analyzed the impact that 

new technologies have had on the incidence of OEs. As 

the ATO continues with its modernization program, it is 

critical that that the appropriate baseline data are being 
19

collected to ensure a proper assessment of a technology’s 

positive or negative effect on controller performance.

Non-Compliance Issues

Non-compliance is a purposeful disregard of rules or 

procedures, existing rules, regulations, instructions, or 

standard operating procedures by choosing not to respond 

or selecting an action not in compliance with rules or 

operating procedures. Factors related to non-compliance 

with ATC policies and procedures were not the focus of 

any of the OE literature that we reviewed, and consequently will not be discussed in this paper.

Task Descriptions

A task is an activity that a controller is primarily engaged 

in at a specific time. Controllers often multitask, i.e., 

perform several tasks at a time. The controller task being 

performed at the time of an OE was not the focus of any of 

the OE literature we reviewed, and consequently will not 

be discussed in this paper. However, the OE report forms 

categorize aspects of controller activities, for example, 

monitoring the radar display or observing traffic. 

Response Execution

Response execution refers to the outputs or actions based 

on the earlier processes, that is, how choices and plans 

were enacted as actions (either verbally or manually). 

Response execution errors are often characterized by 

their timing (such as too soon, too late) or information 

quality (such as unclear or incorrect). They can also be 

classified as errors of omission (absent) or commission 

(performed).

Response execution appears closely associated with controller planning and decision making. That is, response 

execution is the result of either planned or unplanned 

actions resulting from controller decisions based on 

factors that affect those decisions, including perception and vigilance, memory, contextual conditions, and 

organizational, managerial, and supervisory influences. 

Consequently, instead of discussing reports about controller actions that resulted in OEs, in this section, we 

discussed those actions as they were relevant to the other 

JANUS categories.

Mental Processes

Thirty-two percent of the OE-related documents fell in 

the JANUS Mental Processes category. Figure 7 shows 

the distribution of documents classified into the JANUS 

sub-categories for Mental Processes. Two sub-categories 

accounted for over three-fourths of the reports: Perception and Vigilance (47%), and Memory (32%). The 

literature from these sub-categories will be discussed in 

some detail. For a list of the reports classified as belonging 

to the remaining JANUS mental processes subcategories, 

the reader is referred to Appendix B.

Figure 7 shows the distribution of the OE literature based 

on the mental processes addressed. Most of the mental 

processing OE literature focused on aspects of perception 

and vigilance, followed by controller memory, planning 

and decision-making, then response execution. Although 

executing a response is not typically thought of as a mental 

process, responses are included here as consequences of 

mental processes.

Perception and Vigilance. Perception involves acquiring 

and processing sensory information to hear, see, touch, 

taste, and smell objects in the environment. Vigilance is 

the maintenance of attention required to obtain information that may be presented infrequently or monitor 

a dynamic and ever-changing environment.

In the OE literature, perception and vigilance have often 

been studied under the name of controller attention, 

(Kinney, 1977b; Schroeder, 1983), controller awareness 

and attention (FAA, 1986a; Rodgers & Nye, 1993), and 

later SA (Endsley & Rodgers, 1994; Rodgers, Mogford, & 

Strauch, 2000; Kelly, Krantz & Spelman, 2001). Studies 

designed to determine the effectiveness of OE interventions tend to rely on surveys, interviews, and self-reports 

from controllers and are lacking in the use of variables 

that measure controller perception and vigilance. 

Estimates of the percentage of OEs related to perception and vigilance vary from a low of 27% (Kelly et 

al., 2001) to a high of 72% (Rodgers, Mogford, & 

Strauch, 2000). The variation in statistics is due to the 

type of data studied (OE narratives vs. causal factors 

identified on the OE final report) and the time period 

over which the evaluation was conducted. Although 

controller perception and vigilance have been often 

studied, a coordinated effort over time to track OEs 

specifically related to a loss or lack of perception and 

vigilance has been impeded because the OE reporting 

form did not include these causal factors. To address 

this gap, Rodgers & Nye (1993) recommended the 

addition of three causal factors to the OE final report 

to track the amount of controller SA: a) failure to 

detect displayed information, b) failure to comprehend displayed information, and c) failure to project 

future status of displayed data. Similarly, in the tower 
20

environment, failure to observe aircraft might also 

be included as an indicator of lack of perception and 

vigilance (Kelly et al., 2001). 

Memory. Memory consists of Working Memory (WM) 

and Long-term Memory (LTM). WM is a “blackboard” 

or “scratchpad” where one creates and maintains an understanding of a situation. WM is defined as a processing 

resource of limited capacity involved in the preservation 

of information while simultaneously processing the 

same or other information. LTM is the “library” of one’s 

learned information and accumulated experiences. The 

LTM system is responsible for storing information on a 

relatively permanent basis.

Although memory failures have been associated with OEs 

in the en route, TRACON, and tower environments (for 

a review of the literature see Vingelis et al., 1990), much 

of the memory literature relevant to OEs has been in 

the context of runway incursions. In 1986, the NTSB 

published a special report on runway incursions that concluded that the primary controller-related factors associated with runway incursions were forgetting aircraft and 

absent or incomplete coordination between controllers. 

In 2001, Cardosi and Yost reviewed the literature and 

analyzed safety data involving controller errors in airport 

tower operations. Their review and analyses confirmed 

previous findings that identified memory failures as the 

most frequently cited causal factor of tower OEs. The 

second most common factor was communication errors, 

followed by coordination errors. 

Memory aids are installed in all FAA facilities, although 

there appears to be no standardization in their use. 

Each facility is left to their own resources to design and 

implement effective memory aids. While the lack of 

standardization leaves room for error, the added flexibility enables facilities to create memory aids unique to 

their environment (e.g. addressing a particular runway 

configuration problem). 

In 2003, the Department of Transportation OIG acknowledged that the FAA’s numerous initiatives directed 

at reducing runway incursions, including the effective 

use of memory aids, have been successful. As the amount 

of traffic in the NAS is returning to pre- September 11, 

2001 levels, the number of runway incursions has steadily 

been declining. However, it is unclear whether any of 

the many FAA initiatives have been responsible for the 

decline. Nevertheless, given the predominant role that 

memory failures have played in tower OEs, it is likely 

that memory-related OEs have also declined.

Planning and Decision Making. Planning and Decision 

Making consists of four types of mental processes: (1) 

Planning refers to the development of a stepwise process 

to reach an outcome. (2) Decision making occurs when 

more than one option is considered and then a choice 

between them is made. (3) When one solution emerges 

from one’s understanding of the situation, this is called 

a judgment. (4) Problem solving occurs when a solution 

is needed to resolve a situation or reach a goal, but no 

ready options are available.

Figure 7. Distribution of OE-related reports by JANUS Mental Processes sub-categories.
21

Earlier, when discussing the type of studies, we noted 

that the experiments reviewed did not use OEs as the 

criteria of interest. Instead, the intent of the studies 

was to investigate factors that affected OE precursors, 

such SA, memory, workload, and communications. As 

discussed, the problem with conducting experiments, in 

general, is that OE precursors do not operate in isolation. Instead, they interact in complex ways to increase 

the vulnerability of a controller to committing an OE. 

That very same complexity is also true with planning 

and decision-making, which is the result of the complex 

interaction of memory, perception, and vigilance (D’Arcy 

& Della Rocco, 2001).

The focus of the FAA’s modernization efforts has been 

directed at providing controllers with improved decisionsupport tools; however, there appear to be no studies 

that examine the effectiveness of operational decisions. 

Herein lies the problem. Although various cognitive 

models have been proposed to elucidate controller decision-making (Wickens, Mavor, & McGee, 1997), these 

models have not been used to validate the effectiveness 

of any decision support tool. For example, in the model 

proposed by Wickens et al., controller memory serves as 

the foundation for SA, which in turn affects the quality 

of the controller’s decision. If the underlying foundation 

is faulty (due to memory failure or incomplete SA), the 

decision must be faulty. 

When assessing the effectiveness of a particular decision 

support tool, it is necessary to determine how the tool 

is integrated into the overall model of the controller’s 

mental process. Moreover, once a decision support tool is 

developed, it is important to track its effect on reducing 

OEs that were the result of faulty decision-making.

FAA Initiatives to Reduce Operational Errors

The FAA has a history of implementing OE reduction 

initiatives developed at various levels within the legacy 

AAT and current ATO organizations. Ongoing analyses 

of OEs by the FAA revealed recurring causal factors, and 

initiatives were periodically fielded to address them. Topics have focused on communicating about OE trends to 

increase awareness of OE causal factors by all personnel 

in the organization, developing clear performance standards, ensuring compliance to standards, and developing 

long-term strategies. 

Examination of initiatives helps to show where the ATO 

has focused its energies. This section is organized in a 

similar manner to the previous section that illustrated 

where the research community focused its energies. If 

there is a disconnect between the two groups, we can 

speculate about why it might have resulted; for example, 

because the organization was not aware of research activities, the researchers were not aware of operational needs, 

or perhaps the research results were not considered useful for operational application. Whatever the reasons, 

if there are gaps between the efforts of the research and 

operational communities, then steps should be taken to 

bridge them.

To examine OE-reduction initiatives, we used any available information that described a program intended to 

reduce OEs. Materials included such items as FAA orders 

and memoranda, briefing materials, and meeting notes. 

Because we were unable to locate a historical archive of 

these initiatives, most included here are fairly recent. 

Nevertheless, we feel confident that similar initiatives 

were previously undertaken, due to the consistent role 

of various factors in OE causation. 

We identified 222 initiatives to classify. Because many 

of the documents were not dated, it was not always 

possible to place a given initiative within a historical 

context. Furthermore, it was not possible to trace the 

interconnections among source materials (i.e., whether 

one document was related to another); thus, some of the 

222 initiatives probably represent duplicate activities. 

When a given document contained multiple initiatives, 

each initiative was classified into its respective JANUS 

category. Thus, the number of initiatives does not reflect 

the number of documents examined. Furthermore, when 

a given initiative addressed several JANUS categories, 

then the initiative was included in multiple categories. 

For example, one facility’s “effective hearback” competition to highlight the frequency and significance of 

hearback/readback issues was classified in the following 

JANUS sub-categories: Perception and Vigilance, PilotController Communications, and Training and Experience. Finally, the classification of OE reduction initiatives 

was based on the topics addressed and not the quality 

of the implementation. Concerning the latter, in most 

cases, due to a lack of documentation, it was not possible to differentiate between an initiative that was only 

planned and one that was actually implemented. Thus, 

initiatives that were either planned and implemented 

or planned but not implemented are included in the 

JANUS classification.
22

With the above caveats in mind, Figure 8 shows the frequencies of initiatives targeting each JANUS category. 

Most initiatives in this sample targeted contextual conditions that contribute to human performance and result 

in OEs. This category was further broken down into its 

subcategories (see Figure 9). Initiatives classified in the 

Organization and Management categories reflect activities 

intended to be accomplished by headquarters or regional 

personnel, such as establishing a method for distributing 

information about OEs to facilities. 

Figure 9 shows that most of the initiatives introduced by 

the Air Traffic Organization involved controller training, 

teamwork, and communications. Other initiatives not 

captured in these categories were related to Performance 

Management/ Performance Standards (n=12) and Controller-Ground Communications (n=1). 

Figure 10 shows the number of initiatives targeting the 

JANUS Mental Processes subcategories. The highest 

percentage of initiatives targeted Perception and Vigilance 

(59%), and about half that many targeted Planning and 

Decision Making (29%). About 17% of the Mental 

Processes initiatives targeted Memory, and only about 

7% targeted Response Execution.

Figure 11 compares the Contextual Conditions categorizations of both the research documents and initiatives. 

Given the possibility that some OE reduction initiatives 

are duplications, comparisons made between OE research 

and OE initiatives should focus on the trends and not 

the number of initiatives in a given JANUS category. It 

is apparent that the same six contextual conditions were 

emphasized by both the research and OE reduction 

initiatives (although not necessarily in the same order): 

a) training and experience, b) teamwork, c) pilot-ATC 

communications, d) HMI and equipment, e) airspace/

surface, and f) traffic. Although the specific topics addressed by research and the OE reduction interventions 

may differ, the trend comparisons suggest that research 

and operations focused on the same general areas.

Figure 12 compares the distributions of initiatives and 

research articles targeting the OE Mental Processes subcategories. Again, when the trends are compared, we see 

that, with the exception of memory, both research and 

OE reduction initiatives had a similar focus on the mental 

processes. While research focused on memory more often 

than did the OE reduction initiatives, this should not 

be misconstrued as a lack of operational concern with 

regard to memory. On the contrary, addressing controller memory issues continues to be of major importance 

in ATC operations and is evident in the wide range of 

memory aids employed at ATC facilities. 

A brief overview of the initiatives is given here. Appendix 

C contains more detailed descriptions of several of the 

initiatives, including air traffic re-creation tools, video 

briefing materials, the National Air Traffic Professionalism program, facility initiatives, activities detailed in 

the Air Traffic Quality Assurance Order, 7210.56 (FAA, 

2002b), initiatives in the FAA/NATCA 3-Year Plan for 

OE Reduction (FAA, 2002c), and activities in the Short 

Initiatives

0

50

100

150

200

250

Organization Management Supervision Contextual Task NoncomplianceMental

Processes

JANUS Categories

Frequency 

Figure 8. Distribution of OE-related initiatives by JANUS category.
23

0

10

20

30

40

50

60

70

Traffic

Airspace/Surface

Weather

Pilot Actions

Pilot-ATC-Com

Procedures

Doc & Materials

Ambient Environment

Teamwork

Individual/Personal

Interpersonal & Social

Training & Experience

HMI & Equipment

Contextual Conditions

Frequency

Figure 9. Initiatives targeting JANUS OE Contextual Conditions subcategories.

0

10

20

30

40

50

60

Perception &

Vigilance

Memory Planning & Decision

Making

Response Execution

Mental Processes

Fre

quenc

y

Initiatives Research

Figure 10. Initiatives targeting JANUS OE Mental Processes subcategories.
24

0

10

20

30

40

50

60

70

Traffic

Airspace/Surface

Weather

Pilot Actions

Pilot-ATC-Com

Procedures

Doc & Materials

Ambient Environment

Teamwork

Individual/Personal

Interpersonal & Social

Training & Experience

HMI & Equipment

Contextual Conditions

Frequency

Initiatives Research

Figure 11. Comparison of results from the categorization of research articles and initiatives related to 

Contextual Conditions.

0

10

20

30

40

50

60

Perception &

Vigilance

Memory Planning & Decision

Making

Response Execution

Mental Processes

Fre

quenc

y

Initiatives Research

Figure 12. Initiatives and research articles targeting subcategories of mental processes.
25

Term OE Reduction Initiative (FAA, 2002a). Examples 

of recommendations from workgroups are also included, 

such as those from the Operational Error Prevention 

and Reduction Workgroup, which met in 1996, and the 

Operational Errors in the U.S. Air Traffic System – A 

Task Group Review (FAA, 1987). 

Initiatives Classified by JANUS Categories

Organization and Management

Recall that organizational and management initiatives 

should be incorporated into national/ facility level policies and procedures. However, the resources dedicated to 

the implementation of national/facility level policies and 

procedures are equally important. For example, the QA 

programs detailed in FAA Order 7210.56 (FAA, 2002b) 

correspond to initiatives that fall within the JANUS 

Organization and Management tiers. 

At the national level, The Air Traffic Evaluations and 

Investigation Manager and Staff are expected to provide 

the necessary guidance to ensure that the QA program 

succeeds. Among other things, this guidance includes 

evaluating facilities’ ATC QA programs and recognizing 

QA programs that have achieved significant accomplishments in OE reduction. For example, when a facility has 

achieved 1,000,000 error-free operations, it is awarded 

a national certificate of recognition as a member of the 

“None in the Million” Club. 

At the regional level, Air Traffic Division (ATD) Managers 

are expected to develop a regional QA program, which 

includes a regional OE/OD prevention plan. The plan 

is included in a Regional QA Order that is then used to 

develop corresponding Facility QA programs and associated facility OE/OD prevention plans. Paralleling the 

national evaluation process, ATD Managers are required 

to conduct annual reviews of existing regional QA orders 

and programs. They also ensure that approved facility 

OE/OD prevention plans are in effect that emphasize the 

use of the basic ATC procedures to prevent and reduce 

OE/ODs and address the items pertinent to a particular 

facility’s past deficiencies. 

However, none of the documents we reviewed indicated 

whether adequate resources were provided to accomplish the initiatives. And, in spite of the existence of a 

mechanism that should provide a detailed assessment of 

OE/OD reduction initiatives, we were unable to find 

documentation about the success or failure of most regional or facility-level OE/OD reduction initiatives. This 

does not mean that the information was not collected. 

To the contrary, it is likely that the information resides 

somewhere within the ATO at the national, regional, 

and/or facility level. For example, Appendix C includes 

descriptions of Houston ARTCC’s Quality Evaluation 

Program (QEP), Kansas City ARTCC’s Performance 

Management Program, and the analysis of the Great Lakes 

Region’s OE Reduction Efforts. Each of these initiatives 

included an evaluation. Overall, however, there appears 

to be no central repository for this kind of information, 

which makes it difficult to conduct a national audit to 

determine the success of the various OE reduction initiatives implemented at different sites.

Supervision

Operational supervisors (first-line supervisors) are important to the success of a facility’s OE reduction program. 

Supervisors are tasked to conduct both primary operational duties and administrative duties. They monitor the 

operations and manage controller performance. However, 

few initiatives indicated whether there was organizational 

support for supervisors to accomplish these tasks. One 

task group that reviewed the air traffic system (FAA, 

1987) raised a similar question. Noting that the quality 

of supervision varied both within and between facilities, 

the task group recommended that greater attention should 

be placed on supervisory training. Although first-line 

supervisors come from the ranks of controllers, this does 

not necessarily mean that their technical skills prepare 

them to manage controller performance and training. 

Good controller skills do not automatically translate into 

good supervisory skills.

Contextual Factors

Most OE reduction initiatives that addressed contextual 

factors focused on three areas: (1) training and experience, 

(2) teamwork, and (3) pilot-ATC communications. 

Training and Experience. Skill development acquired 

through training and experience has been an ongoing 

focus of OE reduction initiatives. Initiatives include 

such things as designing suitable training platforms 

(e.g. computer-based instruction and ATC simulators), 

developing course content to address skill deficiencies 

associated with specific kinds of OEs (i.e., those associated with errors in data posting, radar display, aircraft 

observation, communication, coordination, and position relief briefings), and identifying the types of work 

experiences necessary to certify controllers and ensure 

that they maintain currency. 
26

Although numerous types of ATC simulators have been 

developed, most have been used for the training of novice 

controllers. However, the need to use ATC simulators 

for proficiency training of experienced controllers was 

raised during the 1987 task group that reviewed the air 

traffic system (FAA, 1987), again in the Short Term OE 

Reduction Initiative (FAA, 2002a), and more recently in 

the FAA’s 2005-2009 Flight plan. To date, few initiatives 

have been developed to address these issues for experienced controllers.

Teamwork. OE-reduction initiatives related to teamwork 

primarily focused on two topics: 1) the exchange of 

information among team members and 2) developing a 

positive approach to teamwork. Over a 14-year span, from 

the 1987 task group that reviewed the air traffic system 

(1987) to the 2002 Short Term OE Reduction Initiative 

(FAA, 2002a), the focus of information exchange initiatives was on the position relief briefing and the use of 

flight strips. Initiatives designed to improve the concept 

of teamwork among controllers followed the air traffic 

team enhancement model and its emphasis on professionalism and respect for each other (FAA, 1987; FAA, 

2002a). It should be noted that, although components of 

teamwork (i.e., position relief briefing and coordination) 

have been identified as OE causal factors, deficiencies in 

teamwork have not. Thus, if the emphasis on teamwork 

as an OE mitigation strategy continues, greater attention 

needs to be placed on identifying the specific aspects of 

teamwork that affect OE occurrence, developing specific 

interventions, and tracking their effectiveness.

Pilot-ATC Communications. Initiatives designed to reduce hearback/readback errors are ever-present across 

time (Cardosi, 2001; FAA, 1987, 2002a and 2002b). 

Past initiatives have included developing training videos and materials based on OE case histories, auditing 

pilot-controller communications, and designing pilotcontroller training. Given all the attention directed 

at reducing OEs related to hearback/readback errors, 

one wonders whether any of these initiatives reduced 

OEs in either the short-term (when awareness was 

high) or the long-term (when the topic faded into 

the background and another awareness program was 

fielded). Without clear evaluations of initiatives related 

to hearback/readback errors, it is unclear what can be 

done to further reduce this type of human error—except 

perhaps through technological advancement in digital 

communications. However, the inability to further 

reduce hearback/readback errors is merely an assumption until more information is known about these types 

of performance errors, including information about 

how they occur during normal operations and what 

circumstances contribute to some of these performance 

errors becoming OEs. 

Task Descriptions

Some initiatives addressed the type of task the controller was performing if it was noted as being performed at 

the time of the OE. For example, improving processes 

associated with the transfer of position responsibility received repeated attention because of the increase in OEs 

reported within 20 minutes of a position relief briefing. 

Topics related to the task of communicating with pilots 

also received attention under topics such as improving 

phraseology and reducing readback/hearback errors.

Non-Compliance Issues

National, regional, and local SOPs were established and 

audits were conducted to ensure that facility directives were 

consistent with national Orders. Controller non-compliance with SOPs was not addressed by major initiatives, 

other than those that emphasized “back to basics.”

Response Execution

No major OE reduction initiatives specifically targeted 

factors related to how responses were executed (e.g., the 

action taken was not timely or the wrong entry was made 

on the keyboard). Response execution errors were often 

left for specific training programs related to new systems 

(e.g., STARS, URET) to address.

Mental Processes

As was stated earlier in this report, the controller’s 

job is primarily a mental one and, thus, one would 

expect a large number of initiatives to be developed 

to address controller mental processes such as perception and vigilance, memory, and planning and decision-making. However, this has not been the case. 

Instead, most of the mental processing initiatives we 

reviewed were associated with the training of novice 

controllers. When mental processes were addressed 

for experienced controllers, the initiatives tended to 

be more of the awareness-related refresher training. 

What appear to be lacking are initiatives that allow 

experienced controllers to test their mental processing 

skills and participate in activities designed to improve 

those skills. One initiative that best illustrates the latter 

approach is the National Air Traffic Professionalism 

(NATPRO) program. 
27

NATPRO is a personal performance enhancement program intended to maintain skilled performance rather 

than provide remedial training for OE reduction. Instead 

of relying solely on knowledge-based awareness training, 

this approach integrates the concept of “performance 

coaching” by facility personnel certified as coaches. The 

program combines knowledge with practice so controllers can experience their own strengths and limitations 

and can gain insight into their mental-processing skills, 

especially those used in air traffic control. These skills 

include perception and vigilance, memory, and planning and decision-making, all of which are practiced in 

a dynamic environment. 

NATPRO is designed as a series of installments that 

continue to challenge participants toward enhancement 

of skills associated with mental processing. Series 1 of the 

current NATPRO installment emphasizes perception and 

vigilance through the visual acquisition of information. 

The skills developed include multi-tasking, paying attention, scanning, detecting information, focusing on relevant 

information, and staying “in your zone.” Although the 

skills are generic, it is expected that by improving these 

generic skills, individuals will demonstrate a corresponding 

improvement in job performance. By testing themselves 

against the computer and experiencing how their performance can vary in relation to such factors as distraction, fatigue, and boredom, participants gain increased 

understanding of their own performance and can develop 

personal improvement strategies. It will be necessary to 

investigate whether such personal improvement strategies 

will lead to improvements in controller performance under 

simulated and actual job conditions. 

To the extent that we can measure differences in controller 

performance, we have a means of measuring the effectiveness of these types of skill enhancement programs. 

Developing controller performance measures would be a 

much better alternative than using the number and severity 

of OEs as a performance metric for individuals. 

Discussion of Initiatives

Available information about past and current initiatives 

shows that the FAA has a history of deploying programs 

to reduce OEs that appear to revisit many of the same 

general causal factors. One reason that the FAA continues 

to implement the same strategies over and over may be 

that the factor of interest was not affected by previous 

initiatives (or was only temporarily affected until the 

initiative was concluded) and continues to be identified 

as an OE causal factor. Furthermore, air traffic control 

is a complex domain, and perhaps simplistic strategies 

for reducing OEs are not effective. A third reason why 

the FAA continues to repeat the same strategies might 

be that those who develop new initiatives “go with what 

they know” and hope it will work “this time.” A fourth 

reason might be that personnel who develop new OE 

initiatives are new to the domain and have no institutional 

recollection of programs that have been tried previously. 

Naiveté and a lack of a historical record can lead one to 

select the “low hanging fruit.” A fifth reason might be 

that personnel who develop OE initiatives think their 

program will be more effective at addressing the factor 

than previous programs. It is not productive to repeat the 

same action while expecting a different outcome.

Whatever the reason, the content and outcomes of OE 

reduction initiatives tend to be forgotten because they 

are not sufficiently documented. Repeating unsuccessful 

OE reduction initiatives should be avoided as a waste 

of resources. One way to decrease the likelihood of repeating previously unsuccessful initiatives would be to 

maintain an archive of all OE reduction initiatives (at 

the national, regional, and facility levels) and document 

their outcomes so that future developers would not implement unsuccessful programs. However, it is also unclear 

how to identify successful initiatives. Opinions about an 

initiative’s effectiveness, for example, although relatively 

easy to obtain (e.g., surveys, anecdotes, focus groups) have 

all the disadvantages of subjective data. On the other hand, 

using OE counts or rates, which are infrequent outcomes 

describing very complex situations, may not be sensitive 

enough to evaluate whether the improvement in any one 

factor resulted in a decrease in OEs.

SUMMARY AND CONCLUSIONS

Our literature review was part of a larger effort to identify more immediate short-term strategies that could be 

undertaken to reduce the recent increase in OEs. We also 

anticipated that the effort would identify potential new 

intervention strategies as well as gaps in our OE knowledge 

base that require additional research. We gathered and 

reviewed documents that described previous OE reduction 

initiatives in an effort to identify strategies that may have 

been successful in the past and identify dimensions that 

could benefit from interventions, based on the research 

outcomes. To structure our approach to identifying gaps in 

the literature, we used the JANUS taxonomy to understand 

the extent to which HF issues typically associated with human errors had been addressed by research on OEs. This 

assessment revealed that a majority of the literature was 

focused on contextual conditions and mental processes. 

We then described the findings about contextual conditions and mental processes that figured most prominently 

in the literature.
28

As we begin our discussion of the gaps in the literature, 

it is important to emphasize that these gaps are based on 

the literature we reviewed and do not necessarily mean 

that the information is not known elsewhere within the 

ATO. However, if the information does exist, then it is also 

important to note that, in our search of internal documents 

via the FAA intranet and personal contacts within the ATO, 

we did not uncover it. Our discussion is organized around 

five central themes derived from the literature: 

1. Consistency of OE data collected across time 

2. Consistency of OE research 

3. Evaluation of OE interventions

4. Multiple levels of OE analyses 

5. Multifaceted nature of controller performance

Within each theme, we will describe what we know and 

don’t know based on our review of the literature.

Consistency of OE Data Collected

Across Time

What We Know

During the last two decades, the methods used to conduct 

OE investigations have been relatively consistent. FAA 

Order 7210.56 (2002b) describes a systematic approach 

to assessing factors contributing to an OE. The major 

causal factor categories in the OE reporting form have 

also remained relatively consistent. For example, the major 

categories in the Causal Factor section of Form 7210-3 

(data posting, radar display, aircraft observation, communications, coordination, and position relief briefing) 

are the same as those referenced in Order 7210-47 (FAA, 

1982). These categories do not describe HF issues such as 

controller mental processes but, instead, describe events 

that occurred at the same time as the OE.

This has led to a long history of conducting research 

and implementing initiatives that address the role of 

factors such as the amount of traffic, miscommunications (hearback/readback), coordination, and position 

relief briefings on OEs. Other variables of interest from 

a research perspective have typically not been collected 

or have only been collected on a superficial level: traffic 

complexity, situation awareness, and some temporal and 

demographic factors. 

Efforts have been initiated over the years to improve the 

quality of some of the information obtained during the 

OE investigation process. For example, instead of using 

a subjective measure of traffic volume, the measure is 

now focused on the number of aircraft handled by the 

controller at the time of the error. Three questions about 

SA (failure to detect, comprehend, or project the future 

status of displayed data) were added to the form in the 

late 1990s. Other measures, such as complexity, remain 

subjective and difficult to define while others have not 

been addressed because insufficient data are collected 

during the investigations. 

What We Don’t Know

Because the same kinds of data have been collected over 

time, we tend to conduct many of the same analyses, 

get the same findings, and implement the same kinds 

of initiatives over and over. We know less about how 

other variables (such as organizational, managerial, and 

supervisory factors) that we have not historically collected 

may be related to OEs.

The kind of information that has historically been gathered 

during the OE investigation process tends to be descriptive and does not address many of the underlying causal 

dimensions typically associated with human error. For 

example, while job task analyses and personal reports emphasize the importance of memory and decision making 

in air traffic control, this factor has not been adequately 

addressed as part of the investigation process. This can 

be attributed, in part, to the fact that we cannot directly 

observe how memory influences the controller’s decisions 

and overall performance. 

Another problem with OE data collection involves 

the way OEs are identified. While potential OEs are 

systematically identified (and later confirmed) in en 

route facilities due to the presence of the “snitch patch” 

in the en route HOST software, the current approach 

to identifying OEs that occur at TRACONs and towers comes through verbal reports made by the involved 

controller, a supervisor, fellow controllers, or a pilot. As 

a result of differences in how OEs are identified, we do 

not know the extent to which OEs in TRACONs and 

towers are underreported. We are also unable to ensure 

that intervention strategies found to be effective in one 

area will be of equal benefit in others.

Consistency of OE Research

What We Know

Several variables have received repeated attention over 

the last 40 years; the three areas that have received most 

of the attention are: amount of traffic, time on position, 

and pilot/controller communications, specifically hearback/readback errors.
29

Amount of Traffic 

The amount of traffic counted on a national basis is perhaps 

the single most important determinant of the frequency 

of OEs. As the overall volume of traffic increases, there 

has historically been a corresponding increase in OEs. 

No other single variable has accounted for more variance 

in OEs than traffic volume. However, the relationship 

becomes more complex when considered at the sector/position level. Despite the steady growth in air traffic during 

the 1970s and 1980s, the percentage of OEs that occurred 

under conditions of light and moderate traffic increased 

and was accompanied by a corresponding decline in the 

percentage of OEs attributed to heavy traffic. During the 

past seven years, our review of the OES database reveals 

that the number of aircraft handled by the controller at 

the time of an OE has remained relatively consistent, despite changes in the overall amount of traffic. In general, 

while the amount of traffic has increased at the level of 

the NAS, on a sector level, the amount of traffic associated with OEs has remained unchanged. 

Time on Position 

Research has consistently shown that a relatively high 

percentage of OEs occur during the first 20 minutes on 

position, and that relationship is consistent across options and during much of the day. While the tendency 

for OEs to occur early after taking over a position is 

often linked with position relief briefings, a summary 

of the NAIMS data from 1999 to 2003 revealed that 

only a small percentage of OEs were related to position 

relief briefings. 

Communications 

Miscommunication has historically been identified as 

a primary causal factor associated with OEs. Of all the 

miscommunication errors, hearback/readback errors 

were most commonly studied. Research has shown that, 

under normal operations, only a small percentage of all 

controller communications involve a hearback/readback 

error, and most of those errors are caught. Thus, few 

hearback/readback errors result in an OE. However, this 

does not negate the need to continue to conduct research 

on this issue because it still is an important factor related 

to OE occurrence.

What We Don’t Know

Amount of Traffic 

We know little about the interaction between the amount 

of traffic and other factors such as airspace complexity. 

Most of the research on airspace complexity has addressed 

variables that predict sector congestion, not OEs. The 

limited amount of research that has related complexity 

to OEs has had inconsistent results. The exception is 

that we know that smaller sectors tend to have higher 

error rates. Thus, more research is needed that specifically relates airspace complexity to OEs. We also need 

to develop a better understanding of how much time 

controllers spend working traffic under various workload/complexity conditions. 

Time on Position 

Most research on the relationship between time on position and OEs has focused on position relief briefings, 

even though they were infrequently identified as a causal 

factor in OEs. Does the relatively higher number of OEs 

that occur soon after taking over a position occur because 

most controllers only work on a sector/position for short 

periods of time? Baseline information about how long 

controllers actually work a sector or position is necessary 

to understand this phenomenon. We also don’t know 

how factors other than position relief briefings, such as 

readiness to perform, might contribute to OEs that occur 

soon after taking over a position. 

Communications 

Because most hearback/readback errors are caught, 

we don’t know the conditions that make a controller 

vulnerable to failing to catch this kind of error during 

a communications exchange. For example, workload or 

traffic complexity might occasionally interfere with a 

controller’s ability to catch a hearback/readback error. 

To identify these factors, research involving multiple 

variables is required. 

Introduction of equipment that reduces the requirement 

to communicate verbally might reduce the incidence of 

some kinds of communications errors but might introduce 

other errors in cognitive processing, especially if the new 

equipment places a greater load on visual processes (e.g., 

transmission of text data link messages between the pilot 

and controller). 
30

Evaluation of OE Interventions

What We Know

The two most common areas addressed by national, 

regional, and facility level interventions include communications (specifically hearback/readback errors) and 

position relief briefings.

What We Don’t Know

Despite the continued interest in reducing OEs, we found 

little evidence of any systematic efforts to evaluate the effectiveness of OE reduction initiatives. While we received 

anecdotal reports about the success of certain individual 

facility interventions, we did not find any quantitative 

data to support those claims. This does not mean that 

facility-level data are unavailable; it merely means that 

these data were not included in any available OE data 

sources used for this study. Even at the national level 

where a number of OE reduction initiatives have been 

implemented, we could not find any documentation of 

the effectiveness of those initiatives. Without efforts to 

establish a method to record and track the effectiveness 

of OE mitigation initiatives, we will be unable to learn 

from past successes and failures. We were also unable 

to examine strategies that were claimed to be effective 

at one facility to determine if they could be effective at 

another.

Multiple Levels of OE Analyses

What We Know

Analysis of OEs is done by many parts of the organization at many levels for many purposes. For example, 

senior management has often focused their attention on 

national OE rates and reduced the percentage of OEs that 

fall in the highest severity category. Data analyses and 

initiatives focused at this level on information averaged 

over individual controllers and sectors or positions. On 

the other hand, facility management generally focused 

their efforts on particular controllers and sectors that had 

more OEs than others and on techniques and procedures 

often thought to be closely associated with OEs. Thus, 

national initiatives are often directed towards a more 

generic approach to reducing OEs while facility initiatives may address problems associated with a local issue, 

such as airspace. 

A number of tools are available that allow us to conduct 

OE analyses to support questions asked at different 

levels of the organization. For example, the JANUS 

technique illustrates the many levels of analysis that are 

possible if sufficient information is available from the 

OE investigation. OE re-creation tools (e.g., SATORI 

and RAPTOR) provide an excellent opportunity to gain 

additional understanding about observable conditions 

surrounding OEs. 

What We Don’t Know

We need to understand how various factors interact to 

produce OE conditions by recognizing that multiple 

individual, situational, and organizational factors both 

contribute to OEs and serve as a deterrent to them. 

We should follow recommendations first made three 

decades ago to expand the gathering of human factors 

information during the OE investigation process. Once 

this is accomplished, we can develop analytic strategies 

to identify how various factors interact to contribute to 

the OE process. 

To date, most analyses have focused on one or two variables. The future lies in efforts to fully address the dynamic 

multi-factor and multi-level conditions associated with 

controller performance. For example, we know from 

some studies that supervisors can play an important role 

in reducing the likelihood of OEs. However, research has 

not addressed several relevant questions: What aspects of 

their roles and responsibilities are most critical? To what 

extent can supervisors affect OEs by identifying poor performance, overseeing position relief briefings, providing 

support during high traffic loads, or taking other actions? 

Until we gain a better understanding of the dynamic 

nature of the OE process and the role of interactive 

factors, we are likely to continue to experience difficulties in developing truly effective intervention strategies. 

Finally, we have to consider broader organizational issues. 

Are initiatives designed to address conditions within an 

individual facility suitable for national deployment? We 

know little about these issues other than certain facilities have higher error rates than others. Further research 

is needed at higher organizational levels to understand 

which HF causes of OEs are common across facilities and 

which are specific to a given facility, area, sector, and/or 

individual or team. 

The Multifaceted Nature of Controller Performance

What We Know

Over the course of their careers, many controllers will not 

have an OE, a small percentage will have one, and a still 

smaller percentage will have multiple OEs. All controllers 

are trained and certified to control traffic in a particular 
31

role (i.e., position) and on specific sections of airspace (e.g., 

sectors). Certification is required because each section of 

airspace is unique due to a variety of factors including 

size, airway configuration, traffic patterns, traffic mix, etc. 

These factors interact with the techniques controllers use 

to control traffic to create a unique environment. Despite 

the complex and dynamic nature of a controller’s tasks, 

most investigations have focused on only one or two 

factors in the study of controller performance. 

What We Don’t Know

We do not understand the importance of the techniques 

and procedures some controllers adopt to remain relatively 

error-free. Additional information is needed to identify 

aspects of controller performance that allow some to remain “error free” or relatively error free throughout much 

of their careers. Are these skills developed during training, 

or are some individuals, because of their innate abilities, 

better controllers than others? We suggest that efforts 

to identify variables that affect controller performance 

should be based on a cognitive model of ATCS tasks such 

as that provided in Wickens et al. (1997, p. 93). Using a 

cognitive model to guide the choice of variables and data 

analysis should help us understand where breakdowns 

occur in relation to certain types of errors. 

We have not adequately measured controller performance 

during training and certification. Such measures would 

allow us to evaluate the relative importance of identifiable weaknesses on the occurrence of future OEs. We 

also have limited knowledge about the extent to which 

OEs are influenced by airspace or traffic complexity 

beyond knowing that smaller sectors have higher error 

rates. Because errors in controller performance are likely 

to occur a number of times and not lead to an OE, what 

combination of factors or conditions prevents an OE 

from occurring? Is it related to attention, memory, decision-making, or other factors? 

RECOMMENDATIONS

Based on our review of the OE research literature and the 

discussion above, we recommend the following: 

Expand the OES database to better identify the human factors 

associated with OEs. There are two ways to improve the 

existing database. The first concerns the accuracy of the 

information. Continued efforts are needed to ensure that 

investigators’ judgments about subjective categorizations, 

both within and across facilities, are made consistently. 

For example, criteria about the complexity of traffic conditions need to be clearly laid out so that judgments can 

be made consistently. Even on more objective measures 

(e.g., amount of traffic) we need to ensure that the time 

period during which traffic is counted is consistent across 

OEs. As part of this effort, there is a need to ensure that 

individuals responsible for gathering information during the OE investigative process have been adequately 

trained. Unless data are gathered consistently, we will be 

unable to adequately evaluate differences across facilities. 

Another element about the accuracy of the OE database 

concerns data entry. During our analyses of the database, 

we encountered numerous instances where data were incomplete or inaccurate. Additional safeguards are needed 

to ensure that values for variables in the database do not 

exceed specified limits. This is one way of improving 

data quality.

The second way to improve the database involves adopting 

the repeated recommendations made over the past 20 years 

to include additional categories that address HF dimensions traditionally linked to controller performance and 

human error. This information would more adequately 

describe the individual, situational, and organizational 

events that contributed to each OE. We feel that the current effort to utilize JANUS to analyze OEs at selected 

facilities is a step in the right direction. By improving 

the investigation process, we will have a richer database 

to assess the human factors most closely associated with 

OEs, one that will facilitate our understanding of potential 

linkages between factors and will provide greater support 

for efforts to assess the effectiveness of OE intervention 

strategies. Another way to improve OE investigations 

would be to adopt the DOT Inspector General’s recommendation (DOT, 2003) to develop a method to better 

detect and re-create OEs in terminal facilities. 

Obtain baseline information about air traffic operations to 

better understand conditions surrounding OEs. Baseline 

information about normal AT operations is needed to 

differentiate between conditions in which OEs occur and 

those in which OEs do not occur. For example, how does 

the level of traffic typically associated with OEs compare 

with the amount of traffic controllers typically experience? 

It would also be useful to obtain baseline information 

about the frequency with which supervisors monitor 

controller performance and the traffic situation.

We must also be able to identify and track the introduction of new NAS procedures/ technologies and assess their 

short- and long-term effects on controller performance 

and OE occurrence. Questions have been raised regarding the effects of URET (Corker, 2004), as well as efforts 

to use color to differentiate “own” from “other” aircraft 

with the introduction of STARS at the Philadelphia 
32

TRACON. We also need to ensure that the information 

collected during OE investigations is sufficiently sensitive to allow us to determine how a new technology or 

procedure affected the incidence of OEs. For example, if 

shifting entries on URET’s Aircraft List (ACL) distracted 

the controller so that he/she committed an OE, under 

today’s system the error might be linked to the result of 

the distraction, rather than its cause. Continued efforts 

are needed to document these and other NAS procedural 

and technological changes so that scientists can adequately 

assess their potential influence on the OE process. 

Some of these changes may dictate different ways in 

which we should aggregate and analyze the OE data. 

For example, increases in direct routing of aircraft may 

have shifted conflict points and required controllers to 

adjust some of their procedures and practices. When OE 

data are only aggregated by facility, we may miss subtle 

changes in the OE process. Greater recognition should 

be given to the dynamic nature of air traffic control and 

adjust the baseline and OE data analysis processes to 

more adequately address the new conditions.

Conduct analyses that account for the interactive nature of 

variables associated with OEs. Given that there have been 

few systematic attempts to understand the interaction of 

the multiple factors involved in the OE process, future 

research should be directed toward investigating their 

interaction. Without this emphasis, we are not likely 

to benefit from more sophisticated data collection and 

analysis strategies such as those suggested by JANUS. 

Develop method of tracking and evaluating OE intervention 

strategies. Few facilities have tracked the effects of their 

OE reduction initiatives using measurable outcomes. 

Furthermore, there are few reliable indications that 

these initiatives have caused OE rates to decline. While 

some information about OE reduction initiatives may be 

available to the ATO, we were unable to document any 

systematic efforts to identify or assess the effectiveness of 

initiatives undertaken to reduce OEs. This may be partly 

because there is no central repository of information about 

the implementation of local or national initiatives or a 

record of their outcomes. To develop such an information 

repository, facilities must first provide a description of 

their OE reduction initiatives along with observed results, 

and national offices must compile information from the 

individual reports to track the success of different types of 

initiatives. Once identified, initiatives that are successful 

at one facility may be adapted to others. 

Develop measures of controller performance. Even though 

controllers often feel that poor techniques and procedures 

on the part of their colleagues are causal factors in OEs, 

we could find little scientific literature that adequately 

described the role of techniques and procedures on OEs. 

We also did not find any literature that described the 

techniques and procedures used by controllers who have 

remained error-free over much of their careers. Efforts 

are needed to identify those “good” techniques and procedures that allow some controllers to remain error free. 

By developing a more complete understanding of the 

performance of controllers who do not commit OEs, we 

can design improved selection and training programs to 

ensure that those knowledge, skills, and abilities are present in certified controllers. An improved set of methods 

for periodically assessing operational performance will 

also allow supervisors to more readily identify aspects of 

controller performance that need improvement. Some of 

the initiatives have focused on these activities. However, 

to our knowledge, these efforts have not developed standardized procedures to assess controller performance.

Over the next decade, the number of retiring controllers will lead to a continuous flow of new trainees into 

AT facilities. Efforts are needed to establish a database 

about aspects of controller training performance that 

can be linked with the OE database. There should be 

sufficient granularity of performance assessments made 

during OJT to allow scientists to determine if specific 

weaknesses in training performance are associated with 

subsequent OEs. Without such a database, there will be 

no way of knowing whether training is effective and if 

adjustments in training would result in improved controller performance. Special attention needs to be focused on 

the first few years following certification because evidence 

suggests that younger, more inexperienced controllers 

have higher error rates.

Identify new approaches to reduce high impact causal factors. 

Our review identified several factors (e.g., hearback/readback, amount of traffic, time vulnerability) that have 

historically been associated with OEs. Although previous 

error mitigation strategies attempted to address these 

factors, there is no evidence that their influence has been 

significantly reduced. Thus, we continue to consider them 

to be fruitful areas of research. However, new approaches 

are needed to identify strategies that can reduce the role 

of miscommunication, time vulnerability, and other 

factors on OEs. New investigations should focus on the 

interplay of several factors in OE causation. 

Given the current emphasis on OE severity, we need to 

more clearly understand the relationship between OE 

severity and various causal factors. Do the events that lead 

up to more severe OEs differ from those where severity 
33

remains relatively low? If so, can interventions be developed to interrupt the chain of events that lead up to the 

more severe OEs and thereby ensure that either severity 

remains low or that the OE likelihood is reduced?

From our review of the literature, it is clear that if the 

human factors research community continues to address 

issues concerning OEs in isolation of one another and of 

operations, it is unlikely that we will discover more than 

what we already know. Instead, this review suggests that it is 

time for a more comprehensive and coordinated approach 

to conducting research about HF causes of OEs. A national 

plan coordinated by the ATO’s Safety Service Office is 

needed to identify the respective roles of members of the 

research community in investigating these issues and work 

with the operational community to translate the results 

into a set of operational OE reduction initiatives. 

Examine OEs as a part of a safety culture. It would be 

useful to consider how OEs could be examined from 

the perspective of a “safety culture.” By this, we mean 

the social norms and expectations present in an air traffic facility that influence the behavior of operational 

personnel. For example, a perceived punitive approach 

(norm or expectation) for dealing with OEs can result 

in underreporting (behavior) at facilities where OEs are 

not automatically identified. No data are available about 

how safety-related norms are developed in ATC facilities 

or how they affect behavior. 

Other characteristics of a safety culture include a demonstration in all levels of the organization of values that 

support the goal of continually identifying and resolving 

safety-related problems, proactivity (rather than reactivity) 

in searching for and resolving safety hazards (especially 

through open discussion of problems), and development 

of metrics and data collection methods to assess safety 

levels and identify causal factors associated with errors. 

It is not possible to study a safety culture unless you can 

measure safety. Currently, the FAA Flight Plan (FAA, 

2005b) uses OEs as the primary measure of ATC safety. 

However, OEs may be inappropriate indicators of safety, 

if used alone, because they only occur infrequently and do 

not describe safety conditions in normal operations. Other, 

more appropriate measures might be better indicators of 

air traffic safety, such as the number of miscommunications between pilots and controllers or the number of 

data entry errors that occur at a sector when it gets busy. 

However, such measures are often difficult to gather and 

may be influenced by factors other than safety. There is 

also the problem of whether to measure the process or 

the outcome, or both.

If the goal of a safety culture is to collect all available 

information related to the safety of an operation, it is 

necessary to collect information relevant to all aspects 

of safety, not just those related to when a safety criterion 

is violated. It is necessary to be aware of what happens 

during normal operations and how other safety-related 

measures change before and after a criterion is violated. 

It is also necessary to obtain as much information as possible about a safety violation, which requires providing 

positive reinforcement to individuals to encourage them 

to provide information to help the organization increase 

the safety of its operations.

We believe that the investigation of OEs from the perspective of a safety culture must be based on a systematic 

collection and analysis of all data that could be relevant 

to identifying meaningful causal factors. JANUS is an 

example of a taxonomy that provides a structure for collecting OE causal factors data from a different perspective 

than has been used before. Instead of looking at only 

event-related errors such as those associated with communications, data posting, or radar displays, it might also be 

useful to obtain information about contextual conditions, 

mental processing errors, and supervision issues present 

at the time of the OE. If that kind of information were 

analyzed in the context of normative data about length 

of shifts, time on position, communications, amount 

and complexity of traffic typically present in the airspace, 

and so on, it might be possible to better understand 

how and why OEs occurred. If information from an expanded OE investigation database were used to develop 

OE reduction initiatives, then organizational tracking 

of their effectiveness could lead to an understanding of 

which initiatives would be useful in addressing particular 

or combinations of causal factors associated with OEs. 

This approach would support a “safety culture” concept 

because the organization could learn from the data how 

to continue to improve.

The research community can help the AT operational 

community by identifying data that would be useful to 

include in expanded, safety-related OE databases and by 

conducting analyses using those data. The operational 

community must collect the data, develop OE reduction 

initiatives, and track their effectiveness. The two communities can work together to interpret the data and decide 

how to identify initiatives that might be useful to reduce 

OEs. We know that the amount of traffic will increase over 

time. If OEs are not examined in a different way, then OEs 

will continue to increase along with the traffic. Only by 

working together can the two communities build a safety 

culture, better understand the causes of OEs, and work to 

reduce them and the processes leading to them. 
34

In summary, this review points to the following recommendations:

1) Expand the OES database to better identify the 

human factors associated with OEs, 

2) Obtain baseline information about air traffic operations to better understand conditions surrounding 

OEs, 

3) Conduct analyses that account for the interactive 

nature of variables associated with OEs, 

4) Develop a method for tracking and evaluating OE 

intervention strategies, 

5) Develop measures of controller performance, 

6) Identify new approaches to reduce high-impact 

causal factors, and

7) Examine OEs as a part of a safety culture. 

REFERENCES

Air Traffic Organization. (2005). Performance 

and Metrics. Retrieved April 29, 2005 from http://ato.

faa.gov/

Arjunan, M., Longstreet, E., & Woo, J. (1982). 

An analysis of reports of operational errors (Report No. 

DOT-FAA-ASF-200-83-1). Washington, DC: Federal 

Aviation Administration.

ATC Operational Error Task Group (1987). 

Operational Errors in the U.S. Air Traffic System - A Task 

Group Review. Unpublished manuscript, FAA Office of 

Aviation Safety.

Atlanta Tower (1988). Atlanta tower operational 

error action plan. Unpublished manuscript, FAA Airport 

Traffic Control Tower Atlanta, GA.

Aviation Safety - Air Traffic. (1987). Operational 

error analysis. Unpublished manuscript.

Baddley, A., and Hitch, G. (1974). Working 

memory. In G.H. Bower (Ed.). The psychology of learning 

and motivation (Vol. 8, pp. 47-89). New York: Academic 

Press.

Belanger, R. (1978, June). System Errors. Crosscheck, 41-4.

Bellantoni, J. & Kodis, R. (1981). An analysis of 

runway-taxiway transgressions at controlled airports (Report 

No. FAA-EM-81-5). Washington, DC: Federal Aviation 

Administration.

Billings, C. & Cheaney, E. (1981). Information 

transfer problems in the aviation system (NASA Technical Paper 1875). Moffett Field, CA: Ames Research 

Center.

Billings, C. & Reynard, W. (1981). Dimensions 

of the Information Transfer Problem. In C. E. Billings 

& E. S. Cheaney (Eds.), Information Transfer Problems in 

the Aviation System (NASA Technical Paper 1875 ed., pp. 

9-14). Moffett Field, CA: Ames Research Center.

Boone, D. (2004). Indianapolis ARTCC Training 

Results 1999-2004, Reference OE Performance. Version 

3. Briefing to Air Traffic Organization Manager June 

18, 2004, FAA Indianapolis Air Route Traffic Control 

Center.

Bowen, K. & Winokur, D. (2003). Evolution of 

en route sector operational capabilities (Report No. MTR 

03W0000007). McLean, VA: MITRE Center for Advanced Aviation System Development.

Breedlove, R. (2004). NATPRO Miami Center 

Limited Deployment. Briefing presented to Director of 

Air Traffic Services July 29, 2004. Washington, DC.

Broach, D. (1999). An examination of the relationship between air traffic controller age and en route 

operational errors. Proceedings of the Tenth International 

Symposium on Aviation Psychology. Columbus, OH: The 

Ohio State University.

Broach, D. & Dollar, C. (2002). Relationship of 

employee attitudes and supervisor-controller ration to en route 

operational error rates (Report No. DOT/FAA/AM-02/9). 

Washington, DC: FAA Office of Aerospace Medicine.

Broach, D. & Schroeder, D. (2005). Relationship 

of air traffic control specialist age to en route operational 

errors (Report No: DOT/FAA/AM-5/22). Washington, 

DC: FAA Office of Aerospace Medicine. 

Burki-Cohen, J. (1995). An analysis of tower 

(ground) controller-pilot voice communications (Report 

No. DOT/FAA/AR-94/15). Washington, DC: Federal 

Aviation Administration.

Cardosi, K. (1993). An analysis of en route controller-pilot voice communications (Report No. DOT/FAA/

RD-93/11). Washington, DC: FAA Office of Research 

and Development.

Cardosi, K. (1994). An analysis of tower (local) 

controller-pilot voice communications (Report No. DOT/

FAA/RD-94/15). Washington, DC: FAA Office of Research and Development.

Cardosi, K. (2001). Runway safety. It’s everybody’s 

business. (DOT/FAA/AR-01/66). Washington, DC: FAA 

Office of Research and Development.

Cardosi, K. (2004). Classification of the severity of 

ATC errors that result in runway incursions. Unpublished 

manuscript.
35

Cardosi, K. (n.d.). Cardosi review of CRM draft 2 

with comments - human factors assessment of the CRM 1

course for controllers. Unpublished manuscript.

Cardosi, K., Brett, B., & Han, S. (1996). An analysis 

of TRACON (terminal radar approach control) controller 

- pilot voice communications (Report No. DOT/FAA/AR96/66). Washington, DC: FAA Office of Research and 

Development.

Cardosi, K., and DiFiore, A. (2004) Metrics of communication performance. Unpublished manuscript.

Cardosi, K., Falzarano, P., & Han, S. (1998). 

Pilot-controller communication errors: an analysis of aviation safety reporting system (ASRS) reports (Report No. 

DOT/FAA/AR-98/17). Washington, DC: FAA Office 

of Research and Development.

Cardosi, K., Hannon, D., & Sheridan, T., & Davis, 

W. (2005). A method for rating the severity of runway 

incursions. Proceedings of the 6th USA/Europe Seminar 

on Air Traffic Management Research and Development. 

Baltimore, MD.

Cardosi, K. & Yost, A. (2001). Controller and pilot 

error in airport operations: A review of previous research and 

analysis of safety data (Report No. DOT/FAA/AR-00/51). 

Washington, DC: FAA Office of Aviation Research.

Castaño, D. & Parasuraman, R. (1999). Manipulation of pilot intent under free flight: A prelude to 

not-so-free flight. Proceedings of the Tenth International 

Symposium on Aviation Psychology. Columbus, OH: The 

Ohio State University.

Clough, D. (1986). Preliminary analysis of characteristics of ATCS who are involved in operational errors. 

Unpublished manuscript, Civil Aeromedical Institute 

Aviation Psychology Laboratory, Oklahoma City, OK.

Conner, M. & Corker, K. (2001). Engineering 

Support to the Federal Aviation Administration Evaluations 

and Accident Investigations Staff for a Field Assessment of 

Controller Operational Errors. National Aviation Research 

Institute.

Corker, K. (2004). User Request Evaluation Tool: 

Implementation & Fielding Lessons Learned. Atlantic City, 

NJ. NASA/FAA IIAIPT, William J. Hughes Technical 

Center.

Cox, M. (1992). The cognitive aspects of the air 

traffic control task: A literature review (Report No. IAM 

Report No. 718). Farnborough Hampshire: Royal Air 

Force Institute of Aviation Medicine.

Danaher, J. (1980). Human Error in ATC System 

Operations. Human Factors, 22, 535-45.

D’Arcy, J. & Della Rocco, P. (2001). Air traffic 

control specialist decision making and strategic planning 

- a field survey (Report No. DOT/FAA/CT-TN01/05). 

Washington, DC: Federal Aviation Administration.

Dekker, S. (2001). The Disembodiment of Data 

in the Analysis of Human Factors Accidents. Linköping, 

Sweden. Centre for Human Factors in Aviation, IKP.

Della Rocco, P. (1999). The role of shift work and 

fatigue in air traffic control operational errors and incidents

(Report No. DOT/FAA/AM-99/02). Washington, DC: 

FAA Office of Aviation Medicine. 

Department of Transportation, Office of Inspector 

General (2000). Actions to reduce operational errors and 

deviations have not been effective (Report No. AV-2001-

011). Washington, DC.

Department of Transportation, Office of Inspector 

General (2003). Operational errors and runway incursions: 

Progress made, but the number of incidents is still high and 

presents serious safety risks (Report No. AV-2003-040). 

Washington, DC.

De Valle, E. (2000, December). Radar Trouble 

Caused a Scramble. Miami Herald.

EATMP. (2000). Validation of the human error in 

ATM (HERA) classification system. Draft report. Brussels: 

EUROCONTROL.

Empson, J. (1987). Error Auditing in Air Traffic 

Control Information Systems: Failure Analysis. NATO 

ASI Series, Vol. F32 (pp. 191-8). New York: Springer 

Verlag.

Endsley, M. & Rodgers, M. (1994). Situation 

awareness information requirements for en route air traffic 

control (Report No. DOT/FAA/AM-94/27). Washington, 

DC: FAA Office of Aviation Medicine. 

Endsley, M. & Rodgers, M. (1997). Distribution 

of attention, situation awareness, and workload in a passive 

air traffic control task: Implications for operational errors 

and automation (Report No. DOT/FAA/AM-97/13). 

Washington, DC: FAA Office of Aviation Medicine. 

Endsley, M., Sollenberger, R., Nakata, A., & 

Stein, E. (2000). Situation awareness in air traffic control: 

Enhanced displays for advanced operations (Report No. 

DOT/FAA/CT-TN00/01). Washington, DC: Federal 

Aviation Administration.

Engle, R., Kane, M., & Tuholski, S. (1999). Individual differences in working memory capacity and 

what they tell us about controlled attention, general fluid 

intelligence, and functions of the prefrontal cortex. In 

A. Miyake & P. Shah (Eds.), Models of working memory: 

Mechanisms of active maintenance and executive control

(pp. 102-34). New York: Cambridge.
36

Evans, A., Slamen, A., & Shorrock, S. (1999). 

Use of Human Factors Guidelines and Human Error 

Identification in the Design Lifecycle of NATS Future 

Systems. Proceedings of the FAA/EUROCONTROL Technical Interchange Meeting, Toulouse, France.

FAR Committee. (1987). National operational error 

task group procedures. Unpublished manuscript.

Federal Aviation Administration (1960). FAA air 

traffic management study: A study of cause factors related 

to control errors (Report No. ATM Directive No. 26). 

Washington, DC: Author.

Federal Aviation Administration (1982). Air traffic 

(AT) service operational error/deviation investigation and 

reporting program. (Report No. FAA Order 7210.41A). 

Washington, DC: Author.

Federal Aviation Administration (1985). Operational error analysis. Washington, DC: FAA Air Traffic 

Quality Assurance Staff (AAT-60).

Federal Aviation Administration. (1985, September). A statistical distribution of operational errors and 

operational deviations occurring during the period July 1

- September 30, 1985. Unpublished manuscript, FAA 

Office of Aviation Safety and the Transportation Systems 

Center, Washington, DC.

Federal Aviation Administration. (1986a). Air 

traffic operational error analysis: First quarter 1986. Unpublished manuscript, FAA Air Traffic Quality Assurance 

Staff (AAT-60), Washington, DC.

Federal Aviation Administration. (1986b). Staff 

study: ATC operational errors. Unpublished manuscript, 

FAA Office of Aviation Medicine, Washington, DC.

Federal Aviation Administration (1986c). Profile 

of operational errors in the national airspace system calendar year 1985. Washington, DC: DOT, FAA Office 

of Aviation Safety – Safety Information and Analysis 

Division.

Federal Aviation Administration. (1987). Operational errors in the U.S. air traffic system – a task group 

review prepared by ATC operational error task group. 

Unpublished manuscript, DOT, FAA Office of Aviation 

Safety, Washington, DC.

Federal Aviation Administration (1988). Profile 

of operational errors in the national airspace system calendar year 1987. Washington, DC: DOT, FAA Office 

of Aviation Safety – Safety Information and Analysis 

Division.

Federal Aviation Administration. (1998a). Agreement between the federal aviation administration and 

the national air traffic controllers association. Retrieved 

September 1, 2005 from www.faa.gov/ahr/Policy/agree/

agrees/term/natca-at/Natca98.cfm

Federal Aviation Administration (1998b). FAA 

historical chronology: Civil aviation and the federal government 1926-1996. Washington, DC: DOT, FAA Office 

of Public Affairs.

Federal Aviation Administration. (2002a). Shortterm operational error reduction initiatives. (Report No. 

FAA Notice 7210.52). Washington, DC: Author.

Federal Aviation Administration (2002b). Air traffic quality assurance (FAA Order 7210.56). Washington, 

DC: Author. 

Federal Aviation Administration. (2002c). FAA/

NATCA operational error reduction plan. Washington, 

DC: Author. 

Federal Aviation Administration (2003). Aviation 

safety statistical handbook, national airspace information 

monitoring system (NAIMS). Washington, DC: DOT, 

FAA Air Traffic Organization.

Federal Aviation Administration (2004) Audit of 

controls over the reporting of operational errors. (Report No. 

AV-2004-085). Washington, DC: DOT/FAA.

Federal Aviation Administration (2005a). Facility operation and administration (FAA Order 7210.3). 

Washington, DC: Author.

Federal Aviation Administration (2005b). Flight 

plan 2005-2009. Retrieved January 27, 2006 from www.

faa.gov/about/plans_reports/

Federal Aviation Administration (2006). Air traffic control (FAA Order 7110.65R). Washington, DC: 

Author.

Finkelman, J. M. & Kirschner, C. (1980). An information-processing interpretation of air traffic control 

stress. Human Factors, 22, 561-7.

Flight Safety Foundation (1976a, November). The 

art of listening. Crosscheck, 21-3.

Flight Safety Foundation (1976b, November). To 

err is human. Crosscheck, 18-20.

Fowler, F. D. (1980). Air traffic control problems: 

A pilot’s view. Human Factors, 22, 645-53.

Galster, S. M., Duley, J. A., Masalonis, A. J., & 

Parasuraman, R. (2001). Air traffic controller performance and workload under mature free flight: Conflict 

detection and resolution of aircraft self-separation. The 

International Journal of Aviation Psychology, 11, 71-93.
37

Goldman, S., Manning, C., & Pfleiderer, E. (2006) 

Static sector characteristics and operational errors. (Report 

No. DOT/FAA/AM-06/04). Washington, DC: FAA 

Office of Aviation Medicine.

Gosling, G. D. (n.d.). Analysis of factors affecting 

the occurrence and severity of air traffic control operational 

errors. Unpublished doctoral dissertation, University of 

California.

Grayson, R. L. (1981a). Information transfer in 

the surface component of the system: Problems associated with briefing of relief controllers. In C. E. Billings 

& E. S. Cheaney (Eds.), Information transfer problems in 

the aviation system (NASA Technical Paper 1875 ed., pp. 

15-24). Moffett, CA: Ames Research Center.

Grayson, R. L. (1981b). Information transfer in the 

surface component of the system: Coordination problems 

in air traffic control. In C. E. Billings & E. S. Cheaney 

(Eds.), Information transfer problems in the aviation system

(NASA Technical Paper 1875 ed., pp. 25-46). Moffett 

Field, CA: Ames Research Center.

Grayson, R. L. & Billings, C. E. (1981c). Information transfer between air traffic control and aircraft: Communication problems in flight operations. In C. E. Billings 

& E. S. Cheaney (Eds.), Information transfer problems in 

the aviation system (NASA Technical Paper 1875 ed., pp. 

47-61). Moffett, CA: Ames Research Center.

Gregory, K., Oyung, R., & Rosekind, M. (1999). 

Managing alertness and performance in air traffic control 

operations. In The role of shift work and fatigue in air traffic 

control operational errors and incidents (DOT/FAA/AM99/02 ed., pp. 1-10). Washington, DC: FAA Office of 

Aviation Medicine. 

Grossberg, M. (1987). Development of memory aids 

for air traffic control (draft). Unpublished manuscript.

Grossberg, M. (1989). Relation of sector complexity to operational errors. In Quarterly report of theFAA 

Office of Air Traffic evaluations and analysis. Washington, 

DC: Federal Aviation Administration.

Hansen, M. & Zhang, Y. (2005). The link between 

operational performance and operational errors in the 

national airspace system. Proceedings of the 6th USA/Europe Seminar on Air Traffic Management Research and 

Development, Baltimore, MD. 

Hartman, B. & Fitts, P. M. (1950). The development of techniques and procedures for the study of alertness 

in aviation personnel. Columbus, Ohio. The Ohio State 

University Research Foundation.

Hebert, H. J. (1987). Control tower errors show 

steep increase: FAA official expresses concern. The Washington Post.

Hilburn, B. (2004). Head down time in aerodrome 

operations: A scope study. The Netherlands: Center for 

Human Performance Research (CHPR).

Histon, J. (2003). Analysis of complexity factors in loss 

of separation incidents in Canada (2001-2002). Unpublished manuscript, Massachusetts Institute of Technology 

International Center for Air Transportation.

Hopkin, V. D. (1971). Conflicting criteria in 

evaluating air traffic control systems. Ergonomics, 14,

557-64.

Hopkin, V. D. (1973). Human factors problems in 

conflict detection and resolution (Report No. AGARDCP-105). London: Technical Editing and Reproduction 

Ltd.

Hopkin, V. D. (1989). Implications of automation 

on air traffic control. In Jensen, R.S. (Ed.), Aviation Psychology, 96-108, Aldershot, Hants: Gower Technical.

Hurst, M. W. & Rose, R. M. (1978a). Objective 

workload and behavioural response in airport radar 

control rooms. Ergonomics, 21, 559-65.

Hurst, M. W. & Rose, R. M. (1978b). Objective 

job difficulty, behavioural response, and sector characteristics in air route traffic control centres. Ergonomics, 

21, 697-708.

Hutton, R., Olszewski, R., Thordsen, M. L., & 

Kaempf, G. L. (n.d.). En route air traffic controller decision making model and decision maker vulnerabilities.

Unpublished manuscript.

Isaac, A., Shorrock, S. T., Kennedy, R., Kirwan, 

B., Andersen, H., & Bove, T. (2002). Technical review 

of human performance models and taxonomies of human 

error in ATM (HERA) (Report No. HRS/HSP-002-REP01). Brussels: European Organisation for the Safety of 

Air Navigation.

Isaac, A., Shorrock, S. T., & Kirwan, B. (2002). 

Human error in European air traffic management: The 

HERA project. Reliability Engineering & System Safety, 

75, 257-72.

Jones, D. G. & Endsley, M. R. (2000). Overcoming 

representational errors in complex environments. Human 

Factors, 42, 367-78.

Jones, S. G. (1996). Optimizing ATC performance

(Report No. Technical Report 96-5). Austin, Texas. The 

University of Texas Aerospace Crew Research Project. 

Jones, S. G. (1997). Human error: The role of 

group dynamics in error tolerant systems. Dissertation 

Abstracts International: Section B: the Sciences and Engineering, 58.
38

Jones, S. G. & Tesmer, B. (n.d.). A New tool for 

investigating and tracking human factors issues in incidents. 

Unpublished manuscript.

Kalsbeek, J. W. H. (1971). Standards of acceptable 

load in ATC tasks. Ergonomics, 14, 641-50.

Kanki, B. G. & Prinzo, O. V. (1996). Methods and 

metrics of voice communications (Report No. DOT/FAA/

AM-96/10). Washington, DC: FAA Office of Aviation 

Medicine. 

Kaulia, S., Gonzalez, R., Conner, J., Klasinksi, K., 

& Rainey, H. (1987). Operational error work group surface 

incidents report. Unpublished manuscript.

Kelly, D. R., Krantz, J. W., & Spelman, J. (2001). 

An analysis of the causes and prevention of runway incursions: Key findings and recommendations (Report No. 

MP01W0000173). McLean, VA: MITRE.

Kershner, A. M. (1968). Air traffic control system error data for 1965 and 1966 as related to age, workload, and 

time-on-shift of involved controller personnel. (Report No. 

NA-68-32) Atlantic City, New Jersey: FAA, NAFED 

Kinney, G. C. (1977a). The human element in air 

traffic control: Observations and analyses of the performance 

of controllers and supervisors in providing ATC separation 

services. Supplement 1: Definitions and basic concepts

(Report No. MTR-7655 Supplement 1). McLean, VA: 

METREK Division of The MITRE Corporation.

Kinney, G. C. (1977b). The human element in air 

traffic control: Observations and analyses of the performance 

of controllers and supervisors in providing ATC separation 

services. Supplement 3: Investigating and reporting system 

errors. (Report No. MTR-7655). McLean, VA: METREK 

Division of The MITRE Corporation.

Kinney, G. C. (1977c). The human element in air 

traffic control: Observations and analyses of the performance 

of controllers and supervisors in providing ATC separation 

services. Supplement 4: Effects of mental attitudes on the 

job performance of controllers and supervisors. (Report No. 

MTR-7655). McLean, VA: METREK Division of The 

MITRE Corporation.

Kinney, G. C. (1986). Letter to Dr. Frank Austin 

discussing brief summaries of work done at MITRE. Unpublished manuscript.

Kirchner, J. H. & Laurig, W. (1971). The human 

operator in air traffic control systems. Ergonomics, 14,

549-56.

Kirk, K., Mayberry, P., & Lesko, M. (1996). Roles 

and impact of supervisors in air traffic control (Report No. 

IPR-96-0056). Alexandria, VA: The CNA Corp.

Krol, J. P. (1971). Variations in ATC-work load as 

a function of variations in cockpit workload. Ergonomics, 

14, 585-90.

Lancaster, F. R. (1987). Operational error review 

& action plan - ZKC-1 & ASW-501. Unpublished 

manuscript.

Langan-Fox, C. P. & Empson, J. A. C. (1985). 

‘Actions not as planned’ in military air-traffic control. 

Ergonomics, 28, 1509-21.

Lee, P.U. & Mercer, J. (2005). A Non-linear relationship between controller workload, task load, and 

traffic density: The straw that broke the camel’s back. 

Proceedings of the International Symposium on Aviation Psychology. Columbus, OH: The Ohio State University.

Loftus, G. R. (1979). Short-term memory factors 

in ground controller/pilot communication. Human Factors, 21, 169-81.

Lowry, N.M., MacWilliams, K.J., Still, R.J., and 

Walker, M.G. (2005). Analysis of operational errors. 

(Report No. MP05W0000025, Rev. 1). McLean, VA: 

MITRE.

Lyman, E. G. (1981). ATC contingency operations in 

the en route flight regime (Report No. NASA-CR-166231). 

Moffett Field, CA: Ames Research Center.

Majumdar, A. & Ochieng, W. Y. (2000). The factors affecting air traffic controller workload: A multivariate analysis based upon simulation modelling of controller 

workload. Centre for Transport Studies Department of 

Civil and Environmental Engineering Imperial College 

of Science, Technology and Medicine. London, United 

Kingdom.

Masalonis, A. J. & Parasuraman, R. (2003). Fuzzy 

signal detection theory: Analysis of human and machine 

performance in air traffic control and analytic considerations. Ergonomics, 46, 1045-74.

Massimini, P., Nene, V., Gormley, K., Stevens, 

M. (2001). National choke point initiative. (Report No. 

F063-L-029). McLean, VA: MITRE.

Mayberry, P. W., Kropp, K. V., Kirk, K. M., Breitler, 

A. L., & Wei, M. (1995). Analysis of operational errors 

for air route traffic control centers: Volume 1: Final Report.

(Report No. IPR-95-5108). Alexandria, VA: The CNA 

Corporation.

Maynard, P.W. & Rantanen, E.M. (2005). Teams, 

teamwork, and automation in air traffic control. Proceedings of the International Symposium on Aviation Psychology. 

Columbus, OH: The Ohio State University.
39

McCoy, W. E. & Funk, K. H. (1991). Taxonomy 

of ATC operator errors based on a model of human information processing. In R. S. Jensen (Ed.) Proceedings 

of the 6th International Symposium on Aviation Psychology, 

(pp. 532-7). Columbus, OH.

McFarland, A. & Maroney, D. (2001). Eliminating 

the altitude-for-direction rule and implementing reduced 

vertical separation minimum in the U.S. (Report No. 

MP01W0000167). McLean, VA: MITRE.

McIntyre, H. (1988). Letters from Atlanta Tower on 

readback error action plan. Unpublished manuscript, FAA 

Airport Traffic Control Tower, Atlanta, GA.

McKinley, J. B. & Jago, R. J. (1985). Potential controller error modes in automated en route ATC (AERA). 

In R. S. Jensen & J. Adrion (Eds.) Proceedings of the 3rd 

International Symposium on Aviation Psychology, (pp. 189-

197). Columbus, OH: The Ohio State University.

Metzger, U., Galster, S.M., & Parasuraman, R 

(1999). Effects of a conflict detection aid and traffic 

density on controller mental workload and performance 

in an ATC task simulating free flight. Proceedings of the 

Tenth International Symposium on Aviation Psychology. 

Columbus, OH: The Ohio State University.

MITRE. (2004). Operational error analysis status 

- preliminary data - part 1. Unpublished manuscript.

Mogford, R.H. (2001). Development of metrics 

for measuring complexity and dynamic density in air 

traffic control. Proceedings of the Eleventh International 

Symposium on Aviation Psychology. Columbus, OH: The 

Ohio State University.

Mogford, R. H., Allendoerfer, K. R., Snyder, M. 

D., & Hutton, R. (1996). Evaluation of the recognitionprimed decision (RPD) model of expert decision making.

Unpublished manuscript.

Mogford, R.H., Murphy, E.D., Roske-Hofstrand, 

R.J., Yastrop, G. & Guttman, J.A. (1994). Application of 

research techniques for documenting cognitive processes in 

air traffic control: Sector complexity and decision making. 

(Report No. DOT/FAA/CD-TN94/3). Atlantic City, 

NJ: Federal Aviation Administration.

Mohleji, S. C. & MacWilliams, K. (2004). 

Operational error analysis for TRACONs and towers preliminary results - preliminary data - part 2. Unpublished 

manuscript.

Monan, W. P. (1980). Human factors in air carrier operations: Knowledge of the limitations of the 

ATC system in conflict avoidance capabilities. In NASA 

Aviation Safety Reporting System: Tenth Quarterly Report, 

July 1 - September 30, 1978 (NASA-TM-81197 ed., pp. 

12-26).

Monan, W. P. (1983). Addressee errors in ATC communications: The call sign problem (Report No. NASA-CR166462). Moffett Field, CA: Ames Research Center.

Morrow, D.G., & Leirer, V. (1997). Aging, pilot 

performance, and expertise. In A.D. Fisk & W.A. Rogers 

(Eds.). Handbook of Human Factors And the Older Adult

(pp. 199-230). San Diego, CA: Academic Press.

Morrow, D.G., Leirer, V., Altieri, P., & Fitzsimmons, C. (1994). When expertise reduces age differences 

in performance. Psychology and Aging, Vol. 9:134-48.

Morrow, D.G., Leirer, V., & Yesavage, J. (1990). 

The influence of alcohol and aging on radio communication during flight. Aviation, Space, and Environmental 

Medicine, Vol. 61:12-20.

Morrow, D.G., Yesavage, J., Leirer, V., & Tinklenberg, J. (1993). Influence of aging and practice on piloting 

tasks. Experimental Aging Research, Vol. 19:53-70.

Murphy, L. L., & Smith, K. (2001, March). Task 

demand, workload, and performance. In R. S. Jensen 

(Ed.) Proceedings of the Eleventh International Symposium 

on Aviation Psychology. Columbus, OH: The Ohio State 

University.

Murphy, T. P. & Levendoski, R. J. (1989). Accident/incident data analysis database summaries (Vol. I)

(Report No. DOT/FAA/DS-89/17, I). Washington, DC: 

Federal Aviation Administration.

National Transportation Safety Board (1981). 

Special investigation report- aircraft separation incidents at 

Hartsfield Atlanta International Airport, Atlanta, Georgia, 

October 7, 1980 (Report No. NTSB-SIR-81-6). 

National Transportation Safety Board (1983). Special investigation report- follow-up study of the United States 

air traffic control system (Report No. NTSB-SIR-83-1). 

National Transportation Safety Board (1986a). 

Special investigation report- runway incursions at controlled 

airports in the United States (Report No. NTSB-SIR86-1). 

National Transportation Safety Board. (1986b). 

Safety board report urges steps to curb runway collision 

danger. Unpublished manuscript.

National Transportation Safety Board (1991a). 

Aircraft accident report - Northwest Airlines, Inc. flights 

1482 and 299 runway incursion and collision, Detroit 

Metropolitan/Wayne County Airport, Romulus, Michigan, 

December 3, 1990 (Report No. NTSB/AAR-91/05).
40

National Transportation Safety Board (1991b). 

Aircraft accident report - runway collision of Eastern Airlines 

Boeing 727, flight 111 and EPPS Air Service Beechcraft 

King Air A100, Atlanta Hartsfield International Airport 

Atlanta, Georgia, January 18, 1990 (Report No. NTSB/

AAR-91/03).

National Transportation Safety Board (1991c). Aircraft accident report – runway collision of US Air flight 1493 

Boeing 737 and Skywest flight 5569 Fairchild Metroliner, 

Los Angeles International Airport, Los Angeles, California, 

February 1, 1991 (Report No. NTSB/AAR-91/08).

National Transportation Safety Board (1994). 

Aircraft accident report – runway collision involving Transworld Airlines flight 427 and Superior Aviation Cessna 

441 Bridgeton, Missouri, November 22, 1994 (Report 

No. NTSB/AAR-95/05).

O’Connor, W. F. & Pearson, R. G. (1965). ATC 

system error and appraisal of controller proficiency (Report 

No. AM-65-10). Washington, DC: FAA Office of Aviation Medicine. 

Office of Safety Analysis (1989). Report of the interagency near midair collision working group. Unpublished 

manuscript.

Pape, A.M., Wiegmann, D.A., & Shappell, S. 

(2001). Air traffic control (ATC) related accidents and 

incidents: A human factors analysis. Proceedings of the 

Eleventh International Symposium on Aviation Psychology. 

Columbus, OH: The Ohio State University.

Philipp, U., Reiche, D., & Kirchner, J. H. (1971). 

The use of subjective rating. Ergonomics, 14, 611-6.

Pounds, J. (1999). Pounds notes on REF ID#94 

controller and pilot error in airport operations: A review of 

previous research and analysis of safety data DOT/FAA/AR00/51. Unpublished manuscript.

Pounds, J. & Ferrante, A. S. (2003). FAA strategies 

for reducing operational error causal factors (Report No. 

DOT/FAA/AM-03/19). Washington, DC: FAA Office 

of Aerospace Medicine. 

Pounds, J. & Isaac, A. (2002). Development of an 

FAA-EUROCONTROL technique for the analysis of human error in ATM (Report No. DOT/FAA/AM-02/12). 

Washington, DC: FAA Office of Aerospace Medicine. 

Pounds, J. & Isaac, A. (2003). Validation of JANUS 

technique: Causal factors of human error in operational errors (Report No. DOT/FAA/AM-03/21). Washington, 

DC: FAA Office of Aerospace Medicine. 

Press, J. (2004). A measurement framework for air 

traffic safety metrics. Unpublished manuscript.

Prince, L.C. (1999). Situational awareness for 

expert ATS decision making. Proceedings of the Tenth International Symposium on Aviation Psychology. Columbus, 

OH: The Ohio State University.

Prinzo, O. V. (1996). An analysis of approach control/pilot voice communications (Report No. DOT/FAA/

AM-96/26). Washington, DC: FAA Office of Aviation 

Medicine. 

Prinzo, O. V., Britton, T. W., & Hendrix, A. M. 

(1995). Development of a coding form for approach control/pilot voice communications (Report No. DOT/FAA/

AM-95/15). Washington, DC: FAA Office of Aviation 

Medicine. 

Racine, N. & Sierra, E. A. (2004). Advanced 

controller training in a virtual environment (Report No. 

DOT/FAA/CT-TN05/4). Washington, DC: Federal 

Aviation Administration.

Rasmussen, J. (1986). Information processing and 

human-machine interaction: An approach to cognitive 

engineering. New York: Elsevier Science Publishing Co., 

Inc.

Reason, J. (1990). Human error. New York: Cambridge University Press.

Redding, R. E. (1992). Analysis of operational errors 

and workload in air traffic control. McLean, VA: Human 

Technology, Incorporated.

Remington, R. W., Johnston, J. C., Ruthruff, E., 

Gold, M., & Romera, M. (2000). Visual search in complex displays: Factors affecting conflict detection by air 

traffic controllers. Human Factors, 42, 349-66.

Rodgers, M. D., & Duke, D. A. (1993). SATORI: 

Situation assessment through re-creation of incidents. The 

Journal of Air Traffic Control, 35(4), 10-14.

Rodgers, M. D. & Manning, C. A. (1995). Baseline 

assessment of the national airspace system: An approach 

toward modeling operational errors. Proceedings of the 

Eighth Symposium on Aviation Psychology (pp. 422-5). 

Columbus, Ohio.

Rodgers, M. D., Mogford, R. H., & Mogford, 

L. S. (1998). The relationship of sector characteristics to 

operational errors (Report No. DOT/FAA/AM-98/14). 

Washington, DC: FAA Office of Aviation Medicine. 

Rodgers, M. D., Mogford, R. H., & Strauch, B. 

(2000). Post hoc assessment of situation awareness in air 

traffic control incidents and major aircraft accidents. In 

M. R. Endsley & D. J. Garland (Eds.), Situation Awareness Analysis and Measurement (pp. 73-112). Mahwah, 

NJ: Erlbaum, Inc.
41

Rodgers, M. D. & Nye, L. G. (1993). Factors associated with the severity of operational errors at air route 

traffic control centers. In M. D. Rodgers (Ed.), An Analysis 

of the Operational Error Database for Air Route Traffic 

Control Centers (DOT/FAA/AM-93/22 ed., pp. 11-25). 

Washington, DC: FAA Office of Aviation Medicine. 

Rohmert, W. (1971). An international symposium 

on objective assessment of work load in air traffic control 

tasks. Held at the Institute of Arbeitwissenschaft, The 

University of Technology, Darmstadt, German Federal 

Republic. Ergonomics, 14, 545-7.

Salthouse, T.A. (1980). Age and memory: Strategies 

for localizing the loss. In L.W. Poon, J.L. Fozard, L.S. 

Cermak, D. Arenberg, & L.W. Thompson (Eds.). New 

directions in memory and aging, (pp. 47-65). Hillsdale, 

NJ: Erlbaum.

Scarborough, A. & Pounds, J. (2001). Retrospective human factors analysis of ATC operational errors. 

Proceedings of the Eleventh International Symposium on 

Aviation Psychology. Columbus, OH: The Ohio State 

University.

Schroeder, D.J. (1982, October). The loss of prescribed separation between aircraft: How does it occur? 

Proceedings (P-114), Behavioral Objectives in Aviation 

Automated Systems Symposium, Society of Automotive 

Engineers (SAE) (pp. 257-69). 

Schroeder, D. J. (1983). The loss of prescribed 

separation between aircraft: How does it occur? (Report 

No. 0096-736X/83/9104-4426). Society of Automotive 

Engineers, Inc.

Schroeder, D. J. & Nye, L. G. (1993). An examination of the workload conditions associated with 

operational errors/Deviations at air route traffic control 

centers. In M. D. Rodgers (Ed.), An Analysis of the Operational Error Database for Air Route Traffic Control Centers

(DOT/FAA/AM-93/22 ed., pp. 1-10). Washington, DC: 

FAA Office of Aviation Medicine. 

Shappell, S. & Wiegmann, D. (2000a). A human 

error analysis of commercial aviation accidents using the 

human factors analysis and classification system (HFACS).

(Report No. DOT/FAA/AM-01/03). Washington, DC: 

Federal Aviation Administration. 

Shappell, S. A., & Wiegmann, D. A. (2000b). The 

human factors analysis and classification system – HFACS.

(Report No. DOT/FAA/AM-00/07). Washington, DC: 

Federal Aviation Administration. 

Sheridan, P. J. (1997). Operational errors: An 

evaluation of reduction efforts for FY96 in the Great Lakes 

region and a proposed alternative approach to error analysis

(Report No. SPEA-V590-03). Bloomington, Indiana: 

Indiana University School of Public and Environmental 

Affairs.

Shorrock, S. T. & Kirwan, B. (1997). The development of TRACEr: A technique for the retrospective 

analysis of cognitive errors in ATC. Engineering Psychology 

and Cognitive Ergonomics, 3, 163-71.

Shorrock, S. T. & Kirwan, B. (2002). Development 

and application of a human error identification tool for 

air traffic control. Applied Ergonomics, 33, 319-36.

Simpson, R. W. (1982). A risk classification scheme 

for operational errors. Cambridge, Massachusetts. Flight 

Transportation Laboratory, MIT.

Smoker, A. (10-15-2003). Mitigating culture: Human error in the operational ATC world. Unpublished 

manuscript.

Smolensky, M. W. (1992). The effect of work 

load history on operational errors in air traffic control 

simulation: The hysteresis effect- expectancy perseverance 

or short-term memory overload? Dissertation Abstracts 

International, 52, 553.

Spahn, M. J. (1977). The human element in air 

traffic control: Observations and analyses of the performance 

of controllers and supervisors in providing ATC separation 

services. Supplement 2: Analysis of the system effectiveness 

information system (SEIS) data base (Report No. MTR7655 Supplement 2). McLean, VA: METREK Division 

of The MITRE Corporation.

Sperandio, J. C. (1971). Variation of operators’ 

strategies and regulating effects on workload. Ergonomics, 14, 571-7.

Stager, P. & Hameluck, D. (1990). Ergonomics in 

air traffic control. Ergonomics, 33, 493-9.

Stager, P., Hameluck, D., & Jubis, R. (1989). 

Underlying factors in air traffic control incidents. 

Proceedings of the Human Factors Society 33rd Annual 

Meeting, 43-6.

Stammers, R. B. (1978). Human factors in airfield 

air traffic control. Ergonomics, 21, 483-8.

Stein, E. S. (1989a). Air traffic control scanning and 

eye movements in search of information- a literature review

(Report No. DOT/FAA/CT-TN89/9). Washington, DC: 

Federal Aviation Administration. 
42

Stein, E. S. (1989b). The controller memory handbook (Report No. DOT/FAA/CT-TN89/58). Washington, DC: Federal Aviation Administration.

Stein, E. S. & Bailey, J. (1994). The controller 

memory guide: Concepts from the field (Report No. DOT/

FAA/CT-TN94/28). Washington, DC: Federal Aviation 

Administration.

Taylor, J.L., O’Hara, R., Mumenthaler, M.S., 

Rosen, A.C., and Yesavage, J.A. (2005). Cognitive ability, 

expertise, and age differences in following air-traffic control instructions. Psychology and Aging, Vol. 20:117-33.

Taylor, J.L., Yesavage, J.A., Morrow, D.G., Dolhert, 

N., Brooks, J.O., III, & Poon, L.W. (1994). The effects 

of information load and speech rate on younger and 

older aircraft pilots’ ability to execute air-traffic controller 

instructions. Journal of gerontology, series B: Psychological 

sciences and social sciences, Vol. 49:P191-P200.

The Office of Air Traffic Evaluations and Analysis 

(1987). Air traffic operational errors analysis first quarter 

FY 87. Unpublished manuscript.

Training Committee. (1987). Human performance. 

Unpublished manuscript.

Truelove, A., Hatem, C., & VanDyne, T. (2005). 

[Operational error statistics from Kansas City Air Route 

Traffic Control Center]. Unpublished raw data.

Unknown. (1-9-2002). Memphis ARTC center six 

area project historical summary and justification. Unpublished manuscript.

Unknown. (n.d.). Development of a methodology for 

identifying operational errors correctable through training.

Unpublished manuscript.

Vingelis, P. J., Schaeffer, E., Stringer, P., Gromelski, 

S., & Ahmed, B. (1990). Air traffic controller memory 

enhancement: Literature review and proposed memory aids. 

Rep. (No. DOT/FAA/CT-TN90/38) Washington, DC: 

Federal Aviation Administration.

Wiegmann, D. A., Rich, A. M., & Shappell, S. 

A. (2000). Human error and accident causation theories, 

frameworks and analytical techniques: An annotated bibliography Savoy, Illinois: Aviation Research Lab, Institute 

of Aviation.

Whitfield, D. & Ord, G. (1980). Some human 

factors aspects of computer-aiding concepts for air traffic 

controllers. Human Factors, 22, 569-80.

Wickens, C. D., Mayor, A. S., & McGee, J. P. (Eds.) 

(1997). Flight to the future: Human factors in air traffic 

control. Washington, DC: National Academy Press.

Willems, B. & Truitt, T. R. (1999). Implications 

of reduced involvement in en route air traffic control (Report No. DOT/FAA/CT-TN99/22). Washington, DC: 

Federal Aviation Administration.

Wilson-Hill Associates, I. & McDonald, C. (1980). 

A literature review of human factors studies applicable to 

the operational error/deviation investigations of the Federal 

Aviation Administration’s Air Traffic Service Wilson-Hill 

Associates, Inc.

Xing, J. & Bailey, L. (2005). ATC operational errors: Exceeding the limits of cognitive capacities. Proceedings of the International Symposium on Aviation Psychology. 

Columbus, OH: The Ohio State University.

Yesavage, J.A., Taylor, J.L., Mumenthaler, M.S., 

Noda, A., & O’Hara, R. (1999). Relationship of age and 

simulated flight performance. Journal of the American 

geriatrics society, Vol. 47:819-23.
A-1

APPENDIX A

LIST OF RESEARCH DOCUMENTS CLASSIFIED BY TYPE OF STUDY

DESCRIPTIVE STUDIES (DATABASE 

ANALYSIS; N=70)

Arjunan, M., Longstreet, E. J., & Woo, J. (1982)

ATC Operational Error Task Group (1987)

Atlanta Tower. (1988)

Aviation Safety - Air Traffic (1987)

Belanger, R. G. (1978, June)

Bellantoni, J. & Kodis, R. (1981)

Billings, C. E. & Cheaney, E. S. (1981)

Bowen, K. C. & Winokur, D. J. (2003)

Broach, D. (1999)

Broach, D. & Dollar, C. S. (2002)

Clough, D. L. (1986)

Conner, M. & Corker, K. (2001)

D’Arcy, J.-F. & Della Rocco, P. S. (2001)

De Valle, E. (2000, December)

Della Rocco, P. S. (1999)

Evans, A., Slamen, A., & Shorrock, S. T. (1999)

FAR Committee. (1987)

Federal Aviation Administration (1985)

Federal Aviation Administration (1985, September)

Federal Aviation Administration (1986a)

Federal Aviation Administration (1986b)

Federal Aviation Administration (1988)

Flight Safety Foundation (1976, November)

Fowler, F. D. (1980)

Gregory, K., Oyung, R., & Rosekind, M. (1999)

Histon, J. (2003)

DOT (2003)

Isaac, A., Shorrock, S. T., & Kirwan, B. (2002)

Jones, S. G. & Tesmer, B. (n.d.)

Kelly, D. R., Krantz, J. W., & Spelman, J. (2001)

Lancaster, F. R. (1987).

Lowry, N., MacWilliams, K., & Still, R.J., and Walker, 

M.G. (2005)

Majumdar, A. & Ochieng, W. Y. (2000)

Mayberry, P. W., Kropp, K. V., Kirk, K. M., Breitler, 

A. L., & Wei, M. (1995)

McCoy, W. E. & Funk, K. H. (1991)

Memphis ARTC Center Six Area Project Historical 

Summary and Justification (2002)

MITRE. (2004)

Mogford, R.H. (2001)

Mogford, R. H., Allendoerfer, K. R., Snyder, M. D., 

& Hutton, R. (1996)

Mohleji, S. C. & MacWilliams, K. (2004)

Monan, W. P. (1980)

National Transportation Safety Board (1981)

National Transportation Safety Board (1983)

National Transportation Safety Board (1986a)

National Transportation Safety Board (1986b)

National Transportation Safety Board (1991a)

National Transportation Safety Board (1991b)

National Transportation Safety Board (1991c)

National Transportation Safety Board (1994)

Office of Air Traffic Evaluations & Analysis (1987)

Pape, A.M., Wiegmann, D.A., & Shappell, S. (2001)

Pounds, J. & Ferrante, A. S. (2003)

Pounds, J. & Isaac, A. (2002)

Pounds, J. (1999)

Press, J. (2004)

Prince, L.C. (1999)

Redding, R. E. (1992)

Rodgers, M. D., Mogford, R. H., & Mogford, L. S. 

(1998)

Scarborough, A. & Pounds, J. (2001)

Schroeder, D. J. & Nye, L. G. (1993)

Schroeder, D. J., Nye, L. G., & Dollar, C. S. (1991)

Sheridan, P. J. (1997)

Shorrock, S. T. & Kirwan, B. (1997)

Shorrock, S. T. & Kirwan, B. (2002)

Smoker, A. (10-15-2003)

Spahn, M. J. (1977)

The Office of Air Traffic Evaluations and Analysis 

(1987)

Wiegmann, D. A., Rich, A. M., & Shappell, S. A. 

(2000)

Wilson-Hill Associates, I. & McDonald, C. (1980)

Xing, J. & Bailey, L. (2005)

EXPERIMENTAL (N=15)

Castaño, D. & Parasuraman, R. (1999)

Endsley, M. R. & Rodgers, M. D. (1997)

Galster, S. M., Duley, J. A., Masalonis, A. J., & Parasuraman, R. (2001)

Hartman, B. & Fitts, P. M. (1950)

Jones, D. G. & Endsley, M. R. (2000)

Krol, J. P. (1971)

Lee, P.U. & Mercer, J. (2005)

Loftus, G. R. (1979)

Metzger, U., Galster, S.M., & Parasuraman, R (1999)

Murphy, L. L., & Smith, K. (2001, March)
A-2

Philipp, U., Reiche, D., & Kirchner, J. H. (1971)

Remington, R. W., Johnston, J. C., Ruthruff, E., 

Gold, M., & Romera, M. (2000)

Smolensky, M. W. (1992)

Whitfield, D. & Ord, G. (1980)

Willems, B. & Truitt, T. R. (1999)

FIELD STUDIES (N=16)

Burki-Cohen, J. (1995)

Cardosi, K. (1993)

Cardosi, K. (1994)

Cardosi, K., Brett, B., & Han, S. (1996)

Empson, J. (1987)

Endsley, M. R. & Rodgers, M. D. (1994)

Grossberg, M. (1989)

Hurst, M. W. & Rose, R. M. (1978a)

Hurst, M. W. & Rose, R. M. (1978b)

Jones, S. G. (1997)

Kanki, B. G. & Prinzo, O. V. (1996)

Langan-Fox, C. P. & Empson, J. A. C. (1985)

Masalonis, A. J. & Parasuraman, R. (2003)

Pounds, J. & Isaac, A. (2003)

Prinzo, O. V. (1996)

Sperandio, J. C. (1971)

THEORIZING (N=58)

Billings, C. E. & Reynard, W. D. (1981)

Cardosi, K. & Yost, A. (2001)

Cardosi, K. (n.d.)

Cardosi, K., Falzarano, P., & Han, S. (1998)

Cardosi, K., Hannon, D., & Sheridan, T., & Davis, W. 

(2005)

Cox, M. (1992)

Danaher, J. W. (1980)

Dekker, S. W. A. (2001)

Della Rocco, P. S. (1999)

Development of a methodology for identifying operational errors correctable through training (n.d.)

Endsley, M. R., Sollenberger, R. L., Nakata, A., & 

Stein, E. S. (2000)

Federal Aviation Administration (1960)

Federal Aviation Administration. (1986)

Federal Aviation Administration (1987)

Finkelman, J. M. & Kirschner, C. (1980)

Flight Safety Foundation (1976, November)

Gosling, G. D. (n.d.)

Grayson, R. L. & Billings, C. E. (1981)

Grayson, R. L. (1981a)

Grayson, R. L. (1981b)

Grossberg, M. (1987)

Hansen, M. & Zhang, Y. (2005)

Hebert, H. J. (n.d.)

Hilburn, B. (2004)

Hopkin, V. D. (1971)

Hopkin, V. D. (1973)

Hopkin, V. D. (1989)

Hutton, R., Olszewski, R., Thordsen, M. L., & 

Kaempf, G. L. (n.d.)

Isaac, A., Shorrock, S. T., Kennedy, R., Kirwan, B., 

Andersen, H., & Bove, T. (2002)

Jones, S. G. (1996)

Kalsbeek, J. W. H. (1971)

Kaulia, S., Gonzalez, R., Conner, J., Klasinksi, K., & 

Rainey, H. (1987)

Kinney, G. C. (1977)

Kinney, G. C. (1986)

Kirchner, J. H. & Laurig, W. (1971)

Kirk, K., Mayberry, P., & Lesko, M. (1996)

Letters From Atlanta Tower on Readback Error Action 

Plan (1988)

Lyman, E. G. (1981)

Maynard, P.W. & Rantanen, E.M. (2005)

McFarland, A. & Maroney, D. (2001)

McKinley, J. B. & Jago, R. J. (1985)

Monan, W. P. (1983)

Murphy, T. P. & Levendoski, R. J. (1989)

O’Connor, W. F. & Pearson, R. G. (1965)

Office of Safety Analysis (1989)

Prinzo, O. V., Britton, T. W., & Hendrix, A. M. 

(1995)

Rodgers, M. D. & Manning, C. A. (1995)

Rodgers, M. D. & Nye, L. G. (1993)

Rodgers, M. D., Mogford, R. H., & Strauch, B. 

(2000)

Rohmert, W. (1971)

Schroeder, D. J. (1983)

Stager, P. & Hameluck, D. (1990)

Stager, P., Hameluck, D., & Jubis, R. (1989)

Stammers, R. B. (1978)

Stein, E. S. & Bailey, J. (1994)

Stein, E. S. (1989a)

Training Committee. (1987)

Vingelis, P. J., Schaeffer, E., Stringer, P., Gromelski, S., 

& Ahmed, B. (1990)
B-1

APPENDIX B

LIST OF RESEARCH DOCUMENTS CLASSIFIED 

BY JANUS CATEGORIES CONTRIBUTING FACTORS

Organization (N=19)

Arjunan, M., Longstreet, E. J., & Woo, J. (1982)

ATC Operational Error Task Group (1987)

Belanger, R. G. (1978, June)

Conner, M. & Corker, K. (2001)

FAR Committee. (1987)

Federal Aviation Administration (1987)

Fowler, F. D. (1980)

Hopkin, V. D. (1971)

Isaac, A., Shorrock, S. T., Kennedy, R., Kirwan, B., 

Andersen, H., & Bove, T. (2002)

Majumdar, A. & Ochieng, W. Y. (2000)

Memphis ARTC Center Six Area Project Historical 

Summary and Justification (2002)

Murphy, T. P. & Levendoski, R. J. (1989)

National Transportation Safety Board (1983)

National Transportation Safety Board (1991a)

Office of Safety Analysis (1989)

Pape, A.M., Wiegmann, D.A., & Shappell, S. (2001)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Smoker, A. (10-15-2003)

Management (N=11)

Atlanta Tower. (1988)

Belanger, R. G. (1978, June)

Conner, M. & Corker, K. (2001)

Federal Aviation Administration (1960)

DOT (2003)

Kinney, G. C. (1986)

Lancaster, F.R. (1987)

National Transportation Safety Board (1981)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Wiegmann, D. A., Rich, A. M., & Shappell, S. A. 

(2000)

Supervision (N=16)

Broach, D. & Dollar, C. S. (2002)

Federal Aviation Administration (1960)

Federal Aviation Administration (1987)

Kaulia, S., Gonzalez, R., Conner, J., Klasinksi, K., & 

Rainey, H. (1987)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Kinney, G. C. (1986)

Kirk, K., Mayberry, P., & Lesko, M. (1996)

Lancaster, F.R. (1987)

Mayberry, P. W., Kropp, K. V., Kirk, K. M., Breitler, 

A. L., & Wei, M. (1995)

National Transportation Safety Board (1986a)

National Transportation Safety Board (1986b)

National Transportation Safety Board (1991c)

Pape, A.M., Wiegmann, D.A., & Shappell, S. (2001)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Shorrock, S. T. & Kirwan, B. (2002)

Contextual Conditions

Traffic/Airspace (N=40)

Cardosi, K., Hannon, D., & Sheridan, T., & Davis, W. 

(2005)

Castaño, D. & Parasuraman, R. (1999)

Conner, M. & Corker, K. (2001)

Danaher, J. W. (1980)

Empson, J. (1987)

Federal Aviation Administration (1988)

Federal Aviation Administration (1986a)

Galster, S. M., Duley, J. A., Masalonis, A. J., & Parasuraman, R. (2001)

Gosling, G. D. (n.d.)

Grossberg, M. (1989)

Hansen, M. & Zhang, Y. (2005)

Histon, J. (2003)

Hurst, M. W. & Rose, R. M. (1978a)

Hurst, M. W. & Rose, R. M. (1978b)

Jones, S. G. (1996)

Kelly, D. R., Krantz, J. W., & Spelman, J. (2001)

Kinney, G. C. (1977)

Lancaster, F. R. (1987)

Lee, P.U. & Mercer, J. (2005)

Lowry, N., MacWilliams, K., & Still, R.J., and Walker, 

M.G. (2005)

Mayberry, P. W., Kropp, K. V., Kirk, K. M., Breitler, 

A. L., & Wei, M. (1995)

McFarland, A. & Maroney, D. (2001)

Memphis ARTC Center Six Area Project Historical 

Summary and Justification (2002)

Metzger, U., Galster, S.M., & Parasuraman, R (1999)

Mogford, R.H. (2001)
B-2

Murphy, L. L., & Smith, K. (2001, March)

O’Connor, W. F. & Pearson, R. G. (1965)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Rodgers, M. D. & Manning, C. A. (1995)

Rodgers, M. D. & Nye, L. G. (1993)

Rodgers, M. D., Mogford, R. H., & Mogford, L. S. 

(1998)

Scarborough, A. & Pounds, J. (2001)

Schroeder, D. J. & Nye, L. G. (1993)

Schroeder, D. J., Nye, L. G., & Dollar, C. S. (1991)

Sheridan, P. J. (1997)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Stager, P. & Hameluck, D. (1990)

Stammers, R. B. (1978)

Weather (N=14)

Bowen, K. C. & Winokur, D. J. (2003)

Cardosi, K., Hannon, D., & Sheridan, T., & Davis, W. 

(2005)

D’Arcy, J.-F. & Della Rocco, P. S. (2001)

Hansen, M. & Zhang, Y. (2005)

Histon, J. (2003)

Jones, S. G. (1996)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Lancaster, F. R. (1987)

National Transportation Safety Board (1991a)

O’Connor, W. F. & Pearson, R. G. (1965)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Pilot Actions (N=18)

Cardosi, K., Hannon, D., & Sheridan, T., & Davis, W. 

(2005)

Cardosi, K. & Yost, A. (2001)

Castaño, D. & Parasuraman, R. (1999)

FAR Committee. (1987)

Histon, J. (2003)

Jones, S. G. & Tesmer, B. (n.d.)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Lancaster, F. R. (1987)

McIntyre, H. (1988)

National Transportation Safety Board (1994)

National Transportation Safety Board (1986b)

O’Connor, W. F. & Pearson, R. G. (1965)

Pounds, J. & Ferrante, A. S. (2003)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Stager, P. & Hameluck, D. (1990)

Pilot/ATC Communications (N=26)

Burki-Cohen, J. (1995)

Cardosi, K. (1993)

Cardosi, K. (1994)

Cardosi, K., Brett, B., & Han, S. (1996)

Cardosi, K., & DiFiore, A. (2004) 

Cardosi, K., Falzarano, P., & Han, S. (1998)

Cardosi, K., Hannon, D., & Sheridan, T., & Davis, W. 

(2005)

Cardosi, K. & Yost, A. (2001)

Histon, J. (2003)

Kanki, B. G. & Prinzo, O. V. (1996)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Lancaster, F. R. (1987)

National Transportation Safety Board (1991a)

O’Connor, W. F. & Pearson, R. G. (1965)

Office of Safety Analysis (1989)

Pounds, J. & Ferrante, A. S. (2003)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Prinzo, O. V. (1996)

Prinzo, O. V., Britton, T. W., & Hendrix, A. M. 

(1995)

Rodgers, M. D. & Nye, L. G. (1993)

Scarborough, A. & Pounds, J. (2001)

Schroeder, D. J., Nye, L. G., & Dollar, C. S. (1991)

Shorrock, S. T. & Kirwan, B. (1997)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Procedures (N=12)

Cardosi, K., Hannon, D., & Sheridan, T., & Davis, W. 

(2005)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Kinney, G. C. (1977)

Lancaster, F. R. (1987)

Memphis ARTC Center Six Area Project Historical 

Summary and Justification (2002)

National Transportation Safety Board (1991b)

O’Connor, W. F. & Pearson, R. G. (1965)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Rodgers, M. D., Mogford, R. H., & Strauch, B. 

(2000)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)
B-3

Documents & Materials (N=7)

Controller Error Working Group. (n.d.)

Lancaster, F. R. (1987)

O’Connor, W. F. & Pearson, R. G. (1965)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Ambient Environment & Distractions (N=20)

Cardosi, K., Hannon, D., & Sheridan, T., & Davis, W. 

(2005)

Conner, M. & Corker, K. (2001)

Della Rocco, P. S. (1999)

Federal Aviation Administration (1985)

Federal Aviation Administration (1985, September)

Gregory, K., Oyung, R., & Rosekind, M. (1999)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Lancaster, F. R. (1987)

McIntyre, H. (1988)

National Transportation Safety Board (1981)

National Transportation Safety Board (1991a)

National Transportation Safety Board (1991b)

O’Connor, W. F. & Pearson, R. G. (1965)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Scarborough, A. & Pounds, J. (2001)

Schroeder, D. J., Nye, L. G., & Dollar, C. S. (1991)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Wiegmann, D. A., Rich, A. M., & Shappell, S. A. 

(2000)

Teamwork (N=33)

Aviation Safety - Air Traffic (1987)

Cardosi, K. & Yost, A. (2001)

Cardosi, K. (n.d.)

Cox, M. (1992)

FAR Committee. (1987)

Federal Aviation Administration (1986c)

Federal Aviation Administration (1987)

Grayson, R. L. (1981)

Grayson, R. L. (1981)

Hebert, H. J. (n.d.)

Histon, J. (2003)

Jones, S. G. (1996)

Jones, S. G. (1997)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Lancaster, F. R. (1987)

Maynard, P.W. & Rantanen, E.M. (2005)

McCoy, W. E. & Funk, K. H. (1991)

McIntyre, H. (1988)

Mohleji, S. C. & MacWilliams, K. (2004)

National Transportation Safety Board (1981)

National Transportation Safety Board (1986b)

O’Connor, W. F. & Pearson, R. G. (1965)

Office of Air Traffic Evaluations & Analysis (1987)

Pounds, J. & Ferrante, A. S. (2003)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Pounds, J. (1999)

Rodgers, M. D. & Nye, L. G. (1993)

Smoker, A. (10-15-2003)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Stager, P., Hameluck, D., & Jubis, R. (1989)

Training Committee (1987)

Personal Factors (N=17)

Broach, D. (1999)

D’Arcy, J. F. & Della Rocco, P. S. (2001)

Della Rocco, P. S. (1999)

Finkelman, J. M. & Kirschner, C. (1980)

Gregory, K., Oyung, R., & Rosekind, M. (1999)

Hopkin, V. D. (1971)

Kinney, G. C. (1986)

Lancaster, F. R. (1987)

National Transportation Safety Board (1983)

National Transportation Safety Board (1991b)

O’Connor, W. F. & Pearson, R. G. (1965)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Press, J. (2004)

Schroeder, D. J., Nye, L. G., & Dollar, C. S. (1991)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Interpersonal & Social Relations (N=7)

Jones, S. G. (1996)

Lancaster, F. R. (1987)

O’Connor, W. F. & Pearson, R. G. (1965)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Training & Experience (N=38)

ATC Operational Error Task Group (1987)

Broach, D. (1999)

Clough, D. L. (1986)
B-4

Conner, M. & Corker, K. (2001)

Controller Error Working Group. (n.d.)

Cox, M. (1992)

Development of a methodology for identifying operational errors correctable through training (unknown) 

FAR Committee. (1987)

Federal Aviation Administration (1960)

Federal Aviation Administration (1985)

Federal Aviation Administration (1985, September)

Federal Aviation Administration (1986a)

Federal Aviation Administration (1987)

DOT (2003)

Isaac, A., Shorrock, S. T., Kennedy, R., Kirwan, B., 

Andersen, H., & Bove, T. (2002)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Lancaster, F. R. (1987)

Mayberry, P. W., Kropp, K. V., Kirk, K. M., Breitler, 

A. L., & Wei, M. (1995)

Mogford, R.H. (2001)

Monan, W. P. (1980)

National Transportation Safety Board (1981)

National Transportation Safety Board (1983)

National Transportation Safety Board (1986b)

O’Connor, W. F. & Pearson, R. G. (1965)

Office of Safety Analysis (1989)

Pape, A.M., Wiegmann, D.A., & Shappell, S. (2001)

Pounds, J. & Ferrante, A. S. (2003)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Prince, L.C. (1999)

Redding, R. E. (1992)

Rodgers, M. D., Mogford, R. H., & Strauch, B. 

(2000)

Scarborough, A. & Pounds, J. (2001)

Schroeder, D. J., Nye, L. G., & Dollar, C. S. (1991)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Training Committee. (1987)

Wilson-Hill Associates, I. & McDonald, C. (1980)

Human-Machine Interface (N=40)

ATC Operational Error Task Group (1987)

Bellantoni, J. & Kodis, R. (1981)

Cox, M. (1992)

De Valle, E. (2000, December)

Dekker, S. W. A. (2001)

Endsley, M. R., Sollenberger, R. L., Nakata, A., & 

Stein, E. S. (2000)

Evans, A., Slamen, A., & Shorrock, S. T. (1999)

FAR Committee (1987)

Federal Aviation Administration (1986b)

Federal Aviation Administration (1986c)

Federal Aviation Administration (1987)

Federal Aviation Administration (1988)

Fowler, F. D. (1980)

Hilburn, B. (2004)

Histon, J. (2003)

Hopkin, V. D. (1971)

Hopkin, V. D. (1989)

Isaac, A., Shorrock, S. T., Kennedy, R., Kirwan, B., 

Andersen, H., & Bove, T. (2002)

Kinney, G. C. (1986)

Kirchner, J. H. & Laurig, W. (1971)

Lancaster, F. R. (1987)

Langan-Fox, C. P. & Empson, J. A. C. (1985)

Maynard, P.W. & Rantanen, E.M. (2005)

MITRE. (2004)

Monan, W. P. (1980)

National Transportation Safety Board (1991b)

O’Connor, W. F. & Pearson, R. G. (1965)

Philipp, U., Reiche, D., & Kirchner, J. H. (1971)

Pounds, J. & Ferrante, A. S. (2003)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Rodgers, M. D. & Nye, L. G. (1993)

Rodgers, M. D., Mogford, R. H., & Strauch, B. 

(2000)

Spahn, M. J. (1977)

Sperandio, J. C. (1971)

Stager, P. & Hameluck, D. (1990)

Stammers, R. B. (1978)

Training Committee. (1987)

Whitfield, D. & Ord, G. (1980)

Wilson-Hill Associates, I. & McDonald, C. (1980)

CONTROLLER PERFORMANCE

Task Descriptions (N=2)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Non-Compliance Issues (N=4)

Federal Aviation Administration (1988)

Kaulia, S., Gonzalez, R., Conner, J., Klasinksi, K., & 

Rainey, H. (1987)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)
B-5

Mental Processes

Perception & Vigilance (N=51)

Cox, M. (1992)

Danaher, J. W. (1980)

D’Arcy, J. F. & Della Rocco, P. S. (2001)

Della Rocco, P. S. (1999)

Development of a methodology for identifying operational errors correctable through training (unknown)

Endsley, M. R. & Rodgers, M. D. (1994)

Endsley, M. R. & Rodgers, M. D. (1997)

Endsley, M. R., Sollenberger, R. L., Nakata, A., & 

Stein, E. S. (2000)

Evans, A., Slamen, A., & Shorrock, S. T. (1999)

Federal Aviation Administration (1986a)

Flight Safety Foundation (1976a)

Flight Safety Foundation (1976b)

Galster, S. M., Duley, J. A., Masalonis, A. J., & Parasuraman, R. (2001)

Hartman, B. & Fitts, P. M. (1950)

Hebert, H. J. (n.d.)

Hopkin, V. D. (1973)

Isaac, A., Shorrock, S. T., & Kirwan, B. (2002)

Jones, D. G. & Endsley, M. R. (2000)

Jones, S. G. (1997)

Kalsbeek, J. W. H. (1971)

Kelly, D. R., Krantz, J. W., & Spelman, J. (2001)

Krol, J. P. (1971)

Langan-Fox, C. P. & Empson, J. A. C. (1985)

Lyman, E. G. (1981)

Masalonis, A. J. & Parasuraman, R. (2003)

Maynard, P.W. & Rantanen, E.M. (2005)

McCoy, W. E. & Funk, K. H. (1991)

McKinley, J. B. & Jago, R. J. (1985)

Mogford, R.H. (2001)

National Transportation Safety Board (1986a)

Office of Air Traffic Evaluations & Analysis (1987)

Office of Safety Analysis (1989)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Prince, L.C. (1999)

Redding, R. E. (1992)

Remington, R. W., Johnston, J. C., Ruthruff, E., 

Gold, M., & Romera, M. (2000)

Rodgers, M. D. & Nye, L. G. (1993)

Rodgers, M. D., Mogford, R. H., & Strauch, B. 

(2000)

Rohmert, W. (1971)

Scarborough, A. & Pounds, J. (2001)

Schroeder, D. J. (1983)

Shorrock, S. T. & Kirwan, B. (1997)

Shorrock, S. T. & Kirwan, B. (2002)

Spahn, M. J. (1977)

Stager, P. & Hameluck, D. (1990)

Stager, P., Hameluck, D., & Jubis, R. (1989)

Stein, E. S. (1989a)

The Office of Air Traffic Evaluations and Analysis 

(1987)

Wiegmann, D. A., Rich, A. M., & Shappell, S. A. 

(2000)

Willems, B. & Truitt, T. R. (1999)

Memory (N=35)

Cardosi, K., Hannon, D., & Sheridan, T., & Davis, W. 

(2005)

Cardosi, K. & Yost, A. (2001)

Cox, M. (1992)

D’Arcy, J. F. & Della Rocco, P. S. (2001)

DOT (2003)

Federal Aviation Administration (1987)

Finkelman, J. M. & Kirschner, C. (1980)

Flight Safety Foundation (1976b)

Grossberg, M. (1987)

Isaac, A., Shorrock, S. T., & Kirwan, B. (2002)

Jones, S. G. (1996)

Kaulia, S., Gonzalez, R., Conner, J., Klasinksi, K., & 

Rainey, H. (1987)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Kinney, G. C. (1986)

Langan-Fox, C. P. & Empson, J. A. C. (1985)

Loftus, G. R. (1979)

McCoy, W. E. & Funk, K. H. (1991)

McKinley, J. B. & Jago, R. J. (1985)

Memphis ARTC Center Six Area Project Historical 

Summary and Justification (2002)

Mogford, R.H. (2001)

National Transportation Safety Board (1986a)

Office of Safety Analysis (1989)

Pape, A.M., Wiegmann, D.A., & Shappell, S. (2001)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Pounds, J. (1999)

Shorrock, S. T. & Kirwan, B. (2002)

Spahn, M. J. (1977)

Stager, P. & Hameluck, D. (1990)

Stein, E. S. & Bailey, J. (1994)

Training Committee (1987)

Vingelis, P. J., Schaeffer, E., Stringer, P., Gromelski, S., 

& Ahmed, B. (1990)

Wiegmann, D. A., Rich, A. M., & Shappell, S. A. 

(2000)

Willems, B. & Truitt, T. R. (1999)

Xing, J. & Bailey, L. (2005)
B-6

Planning & Decision Making (N=23)

Conner, M. & Corker, K. (2001)

D’Arcy, J. F. & Della Rocco, P. S. (2001)

Development of a methodology for identifying operational errors correctable through training (1990)

Flight Safety Foundation (1976, November)

Hutton, R., Olszewski, R., Thordsen, M. L., & 

Kaempf, G. L. (n.d.)

Isaac, A., Shorrock, S. T., & Kirwan, B. (2002)

Kelly, D., Krantz, J. W., & Spelman, J. (2001)

Kinney, G. C. (1986)

Lyman, E. G. (1981)

McKinley, J. B. & Jago, R. J. (1985)

Mogford, R. H., Allendoerfer, K. R., Snyder, M. D., 

& Hutton, R. (1996)

Mohleji, S. C. & MacWilliams, K. (2004)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Prince, L.C. (1999)

Schroeder, D. J. (1983)

Shorrock, S. T. & Kirwan, B. (1997)

Shorrock, S. T. & Kirwan, B. (2002)

Spahn, M. J. (1977)

Stager, P. & Hameluck, D. (1990)

Stager, P., Hameluck, D., & Jubis, R. (1989)

Training Committee. (1987)

Wiegmann, D. A., Rich, A. M., & Shappell, S. A. 

(2000)

Response Execution/ACT Description (N=20)

Belanger, R. G. (1978, June)

Billings, C. E. & Cheaney, E. S. (1981)

Billings, C. E. & Reynard, W. D. (1981)

Development of a methodology for identifying operational errors correctable through training (1990)

Federal Aviation Administration (1986a)

Flight Safety Foundation (1976, November)

Grayson, R. L. & Billings, C. E. (1981)

Isaac, A., Shorrock, S. T., & Kirwan, B. (2002)

Kelly, D. R., Krantz, J. W., & Spelman, J. (2001)

Kinney, G. C. (1986)

McCoy, W. E. & Funk, K. H. (1991)

McKinley, J. B. & Jago, R. J. (1985)

MITRE. (2004)

Monan, W. P. (1983)

Pounds, J. & Isaac, A. (2002)

Pounds, J. & Isaac, A. (2003)

Smolensky, M. W. (1992)

Spahn, M. J. (1977)

Stager, P. & Hameluck, D. (1990)

Wiegmann, D. A., Rich, A. M., & Shappell, S. A. 

(2000)
C-1

APPENDIX C

REVIEW OF OE INTERVENTIONS

The legacy ATS organization conducted several broad 

activities over a number of years that focused on multiple 

areas of OE causal factors. The discussion below illustrates the ongoing effort to identify strategies designed 

to reduce OEs. Note that attention kept returning to 

particular causal factors as these factors were repeatedly 

identified in OEs. Although the formal activities overlapped in some ways based on this recurring interest, we 

treat them separately in this document for the purpose 

of discussing the programs. Following these descriptions, 

several other initiatives are described; some were national 

initiatives, some were facility-specific. Some are ongoing, 

while others have been concluded. The initiatives are not 

necessarily presented in chronological order.

OE Working Groups

1986 Task Group Review. In 1986 the FAA Administrator directed the Federal Air Surgeon to form a working 

group to address issues associated with ATC OEs. The 

group, comprised of representatives from the Air Traffic Service, the Office of Aviation Medicine, the FAA 

Technical Center, the Office of Aviation Safety, and 

MITRE focused their investigation on the human factors 

associated with OEs. Following their review of historical 

information and the existing OE database, they made 

several recommendations. These included support for 

the recently implemented Operational Position Standards 

program, an emphasis on the use of “positive air traffic 

control techniques,” implementation of a nationwide 

flow control program, the addition of HF data fields to 

accident/incident reports, improved approaches to using 

simulation to evaluate ATCS proficiency, and a recommendation to periodically sample ATCSs’ operational 

proficiencies. In addition to several other factors associated with advanced displays, the study group identified 

areas in which additional research was needed. Scientists 

at the FAA Technical Center were given responsibility for 

addressing issues involving memory and visual scanning. 

As a result of this effort, three reports were prepared (Stein, 

1989a; Stein, 1989b; and Stein and Bailey, 1994). Stein 

(1989a) reviewed issues related to controller scanning. 

Stein (1989b) and Stein and Bailey (1994) were developed 

as memory guides for controllers. 

Operational Errors in the U.S. Air Traffic System – A Task 

Group Review FAA. (1987). In 1987, the FAA Administrator initiated a study of OEs following a sudden 

increase after a two-year declining trend. A task group 

was directed to study system deficiencies related to OEs 

as well as other factors that impacted safe movement 

of aircraft in the system, such as pilot–controller communications, procedures, and human performance. The 

group was also tasked to make recommendations for 

corrective measures. The group consisted of management 

personnel from FAA Headquarters, operational personnel 

from ATC towers, TRACONs, and ARTCCs, as well as 

representatives from the U.S. Air Force, Navy, and Army, 

NASA, and CAMI. The topics covered were: Procedures 

and Regulations; Operations, Communications, and 

Equipment; Human Performance and Training; and HF 

Special Emphasis Items. 

Problems identified included: existing procedures were 

not being followed, ARTCC separation standards were 

restrictive, Federal Aviation Regulations (FARs) placed 

inadequate responsibility on the pilot, lack of precision 

of ATC equipment (i.e., the J-ring), introduction of 

automated aids reduced controllers’ memory skills, lack 

of organizational coordination before implementing new 

procedures, non-standardized and non-aligned procedures 

and QA practices across facilities, high rates of OEs for 

aircraft in transition (climbing/descending), the OE 

report did not have information necessary to pinpoint 

causal factors, lack of opportunity for developmentals to 

acquire experience on the position, lack of experienced 

OJT instructors, ineffective supervision, controllers didn’t 

recognize when they begin to exceed their capabilities, 

lack of teamwork, lack of knowledge of aircraft performance in different conditions, inadequate position relief 

briefings, poor controller memory and decision making, 

failures in controllers’ memory and decision making, and 

failure in information exchanges.

Recommendations were developed to address each problem identified under these topics. These included: 

increase adherence to procedures by instilling pride 

and motivation, 

use the performance management rating system to 

establish accountability for OEs, 

use a monetary award program for OE-free performance, 

test feasibility of visual separation for the ARTCC 

environment, 

further study reduced separation above FL290 and 

in oceanic environments, 

•

•

•

•

•
C-2

examine diverging course separation minimums, 

revise FAR requirements to specify pilot’s responsibilities in communications with ATC, 

improve coordination processes between controllers 

and use of flight strips, 

improve organizational coordination prior to implementing procedural changes, 

improve QA staffing at facilities, 

improve controller training and instruction by better 

screening of instructors, 

use the FAA Headquarters’ Quality Assurance program to assess the Academy’s training program, 

provide error identification and tracking capabilities 

for terminals, 

increase continuity and standardization of facility 

procedures with national orders, 

track new procedures to identify which contribute 

to OEs versus reducing them, 

make the OE report meaningful in determining 

causal factors, 

streamline the FAA incident reporting system, 

make proficiency training mandatory, 

improve status indicators for equipment systems, 

use simulators for controller training, 

train supervisors, 

ensure that any disciplinary actions related to OEs 

are taken based on FAA policy, 

establish national training for operational supervisors, 

establish an HF newsletter for controllers, 

train all employees in teamwork, 

validate use of position relief checklists, 

conduct training for controllers on decision making 

and positive control, 

develop training for controllers on task organization 

and prioritization, 

develop memory aids and memory training for 

controllers, 

make changes to phraseology and readback procedures, 

institute an educational program for pilots on various 

elements of pilot-controller communications contributing to OEs, 

provide training for controllers emphasizing recognition of reduced alertness and mitigation strategies, 

ensure uniform application of accepted human 

performance principles in designing future systems, 

and

develop a method to assess decrements in human 

performance resulting from operational demands. 

Although several themes emerged across the group’s 

recommendations (e.g., improve the incident reporting 

system, and develop training programs for both controllers 

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

•

and pilots to mitigate certain factors), no information 

is available about what actions, if any, were taken as a 

result of this activity.

Operational Error Prevention and Reduction Workgroup 

(September, 1996). A joint NATCA/FAA OE prevention 

workgroup met at the Center for Management Development (CMD) to review former OE mitigation strategies/research and to recommend changes to related FAA 

Orders. Their recommendations included: Discontinue 

mandatory decertification of controllers subsequent to 

OEs, focus on the system rather than the individual when 

reviewing OE causal factors, create mandatory readback/

hearback training, engage the first-level supervisor to a 

higher degree when reviewing OEs, and develop a new 

method to remove subjectivity from the OE identification process, e.g., find a better technique than using a 

“rubber ruler” method that allowed marked variability in 

the assessment of whether an OE had occurred.

Short-Term Operational Error Reduction Initiative

The Short-Term OE Reduction Initiative focused on 

programs to “improve personnel readiness, awareness, and 

communications skills that should lead to a reduction in 

operational errors and deviations caused by incomplete 

position relief briefings, hearback/readback factors, as well 

as other misunderstandings and miscommunications” 

(FAA, 2002a). Emphasis was placed on maintaining good 

operating practices. This included ensuring that facilities 

had an OE prevention plan that incorporated a “back to 

basics” program. Three focal areas were included in this 

initiative: communications, position relief, and distractions. No archival data are available at this time that assessed the effects of these programs on OE reduction.

Communications. Errors in communications between 

controllers and pilots consistently rank within the top 

five causal factors identified in all operational errors. 

This initiative focused on developing good communication skills to reduce hearback/readback errors as well as 

other misunderstandings in communications. Awareness programs targeting readback/hearback errors were 

developed by headquarters staff as well as by regional 

and facility groups.

Air traffic facilities initiated a regular emphasis on good 

communication skills using random tape reviews that 

highlighted readback/hearback errors and examples of 

correct phraseology. During “tape talks,” the supervisor 

and/or facility staff specialists reviewed voice recordings 

of a controller on a position to assess communication 

performance. Personnel were encouraged to assist others 
C-3

to improve their communications skills through positive 

coaching.

A thirty-seven minute training video titled “Preventing 

Read Back Errors” presented strategies to reduce the complexity of each communication to facilitate understanding. The video was a mandatory briefing item shown to 

all controllers. 

Under this initiative, the first National Hearback-Readback Awareness Month (January, 2002) was established 

to focus on catching and mitigating this type of communication error. Based on its perceived success, this 

awareness program became an annual event with January 

inaugurated as Hearback – Readback Awareness Month, 

which included an ongoing emphasis on good communications skills, random tape monitoring to highlight 

examples of correct phraseology, and positive coaching. 

Anecdotal evidence suggests that some facilities may 

still occasionally conduct a program focusing on hearback/readback awareness. However, because no tracking 

mechanism exists, it is unclear which facilities are doing 

this or what the impact is on OE reduction.

Position Relief. The position relief briefing is a standard 

operating procedure designed to optimize transfer of 

position responsibility while at the same time minimizing 

the additional workload associated with the task of transferring the position. Many times, the relieving controller 

will accept position responsibility before being fully ready 

to do so. In some cases, the controller will have been 

uncomfortable with current or planned actions but will 

still accept the position instead of asking the controller 

being relieved to help work out the situation. The goal 

of the position relief initiative was to reduce operational 

errors caused by incomplete position relief briefings by 

improving personnel readiness and awareness in taking 

position responsibilities. “Most errors occur early in a shift 

or after a break. Take time with your relief briefing. Use 

the checklist. Make sure that the relieving controller has 

data he/she needs in working memory. This means that 

they are actively aware of it” (Stein, & Bailey, 1994). 

As part of their facility OE reduction plan, all managers 

were required to emphasize the use of checklists during 

position relief briefing as part of their “back to basics” 

program. They also had to validate their facility’s position relief briefing checklist, ensuring that it conformed 

to requirements of FAA Orders 7110.65R (FAA, 2006) 

and 7210.3 (FAA, 2005a), as well as provide for a capability to record the briefing and take necessary actions 

to ensure that audio recording of position relief briefings 

occurred. Operations supervisors/CICs were required to 

ensure that position relief procedures were followed, that 

the checklist was used, and where available, ensure that 

position relief briefings were recorded. 

All personnel received mandatory training on use of the 

position relief briefing checklist and the audio recording 

requirement. They were trained and encouraged to “have 

the picture” before taking over a position and to accept 

position responsibility only after they were fully aware 

of the traffic. Both the relieving and relieved controllers 

were to ensure that a complete transfer of responsibilities 

was conducted. The relieved controller should ensure that 

the relieving controller had “the picture” before leaving 

the position. In the past, mandatory overlap periods had 

been established i.e., 5 minute pre/post relief briefings. 

The relieving controller and the controller being relieved 

were to establish an appropriate overlap period as part of 

the position relief briefing that would ensure complete 

transfer of position responsibilities. Some facilities mandated a specific amount of time that the relieved controller 

would remain on position. Other facilities adopted this 

as a good practice and allowed the overlap period to vary, 

depending upon traffic demands. The intent of the latter 

practice was to base the overlap period on traffic demand, 

reducing the likelihood of potential distractions associated with multiple personnel unnecessarily remaining in 

operational areas. 

Although not part of this particular initiative, local adaptations to the position relief briefing were sometimes 

made to meet system needs. For example, “each URET 

CCLD facility was asked to ensure that pertinent URET 

CCLD information was integrated into any position 

relief briefing list, whether manual or electronic” (FAA 

Order 7210.3; FAA, 2005a). Not all en route centers 

were “URET CCLD” facilities, so not all were required 

to alter their briefing procedures in this way, but anecdotal 

evidence suggests that some facilities made other local 

adaptations to their position relief processes.

Distractions. The goal of this initiative was to reduce 

control room distractions so that controllers could increase their focus on the operations. For example, during 

periods of light traffic, controllers can be distracted from 

their primary duties by conversations unrelated to the 

control room operations. Under this initiative, during 

light traffic, operational supervisors/CICs were to reduce 

distractions in the control room and assure control room 

focus on operations. 

Control Room Management. Effective control room 

management requires operations supervisors and managers to ensure that combining and decombining of sectors 
C-4

and positions are appropriate for current and forecasted 

traffic conditions. The distractions initiative included 

an effort to encourage the appropriate combining/decombining of sectors and positions. Air traffic managers 

were instructed to brief all operational personnel on 

their expectations that sectors and positions would be 

combined/de-combined based on current and forecasted 

traffic conditions. Operational Supervisors and CICs were 

instructed to actively ensure operational focus by exercising good judgment and ensuring that sector/ position 

openings and closings were appropriate to the current 

and forecasted level of traffic.

FAA/NATCA 3-Year Plan for OE Reduction

The FAA/NATCA Operational Error Reduction Plan 

was a three-year effort (2002-2004) to prevent and 

reduce OEs. It was collaboratively developed between 

the Agency and NATCA to reverse the upward trend in 

OEs and focus attention on those scenarios identified 

as having the potential for the greatest risk to safety. No 

archival data were available that assessed the effects of 

these programs on OE reduction.

The Plan was based on principles of continuous communication at all levels about performance expectations 

and feedback about performance so that OE reduction 

strategies could be proactive instead of reactive. Initiatives 

in the 3-year plan included:

a) Establishing an FAA/NATCA coordinating structure made up of a National Safety Board, Steering 

Committee, and local facility Safety Boards. These 

groups were to provide periodic communications 

to employees about the program. For example, the 

Steering Committee planned to publish an Annual 

QA Program Oversight Report. A review of the 3-

year plan was to be conducted annually to ensure 

the program met its goals.

b) Cataloging national, regional and local initiatives, 

including those from areas/facilities that had low 

numbers of OEs, those that reduced the number 

and/or severity of OEs, and those that identified/

corrected problems before an OE occurred. 

c) Empowering local safety boards to develop more extensive in-depth training programs using tools such as 

URET, DYSIM, JANUS, ETG, and ATTE. Memory 

enhancement training and other programs of performance enhancement (e.g., the CLT training prototype) 

and self-development were also encouraged. 

d) Publishing and distributing a weekly Quality Assurance Digest of OEs to all facilities as a mandatory read 

and initial binder item to heighten OE awareness.

e) Changing the Operational Error Final Report to 

develop a new format useful and practical for identifying the root cause(s) of OEs (possibly including 

Janus, once development is completed).

f) Preparing recommendations for a comprehensive 

system to provide a realistic view of safety and performance of the NAS as an alternative to the current 

measures and separation standards.

g) Establishing a Professional Standards Program based 

on peer assessment and intervention.

Teamwork. This initiative facilitated teamwork by advocating adoption of methods to clearly communicate 

expectations, such as holding team meetings and all-hands 

meetings, having briefings on current trends and issues, 

and discussing OE trends, evaluations, and customer 

inputs. 

A format to emphasize lessons learned was established. It 

included the “In The Zone” program, which distributed 

information about noteworthy air traffic sessions, e.g., 

excellence in individual/team/facility performance, and 

the “In The Ozone” program, which distributed tapes 

of operational errors and near-OEs for the purpose of 

refresher training and lessons learned. It was strongly 

recommended that training conducted using the “In 

The Zone” and “In the Ozone” should involve groups 

of controllers to promote teamwork. 

The 3-Year Plan also specified that ATTE training 

would be included in the curriculum for new hires. It 

was recognized that the addition of new hires provided 

an opportunity to instill cultural change at the onset of 

training, to emphasize that team responsibility is highly 

valued. Training was to be developed jointly by both 

parties and instituted at the Academy and CTI schools. 

Some elements of ATTE were incorporated into the 

Academy’s courses for the terminal options. Discussions 

between the Academy and NATCA regarding what elements to include in courses for the en route option have 

not been completed.

This initiative emphasized information that could be used as 

refresher training for all controllers and targeted to specific 

individuals. Use of both refresher and skill enhancement 

training was encouraged. Techniques included using previous OE scenarios and re-creations as part of training, 

sharing personal accounts by controllers as lessons learned, 

and increasing awareness of QA focus items (both errors 

and good examples) through facility and standup briefings, 

employee work groups, and bulletin board information. 
C-5

The Year of Quality Assurance. To facilitate the 3-Year Plan’s 

OE reduction goals, “The Year of Quality Assurance” 

was launched in 2003. Representatives from air traffic 

management, the National Air Traffic Supervisors’ Committee (SUPCOM), and the National Air Traffic Controllers’ Association (NATCA) embarked on the “National 

Road Show,” sending small teams to air traffic facilities 

to conduct interactive sessions about the 3-Year Plan’s 

goals, technical training discussions, and performance 

management – as an ongoing process of observation and 

feedback. This effort was designed to develop a shared 

understanding of the goal from both a national perspective 

and incorporating the viewpoints of all employees. The 

traveling team explained the objectives of the National 

Safety Board, the Steering Committee, and local facility 

Safety Boards. The team also illustrated how employees 

could support the OE reduction initiatives. 

Identifying Causal Factors. As part of the 3-Year Plan, the 

FAA and NATCA proposed that the current OE Final 

Report Form was time-intensive, required significant 

resources, and did not adequately address a full range of 

causal factors. In support of data-driven decisions about 

OE intervention strategies, the FAA Air Traffic Office 

of Investigations determined that an improved method 

was needed to identify causal factors related to human 

performance. This effort was responsive to goals in the 

FAA’s 1999 Strategic Plan and the 1999 National Aviation Research Plan. The result was a technique, JANUS, 

to identify OE factors related to human performance 

along with contextual and contributing conditions. 

JANUS was developed and tested by both the FAA and 

EUROCONTROL (Pounds & Isaac, 2001). 

The JANUS structure was described earlier as part of the 

method for organizing this report. However, when the 

technique is used to analyze an OE, it can identify very 

specific factors for each JANUS category. For example, 

factors of mental processes (i.e., expectation bias) or of 

teamwork (i.e., the lack of inter-facility coordination) 

can be identified. These factors can then be targeted 

for interventions or used in more complex analyses to 

prioritize factors. The Steering Committee reviewed 

the current OE Form and JANUS beta test results to 

determine if a new format for the OE Form could be 

implemented that would be more useful and practical 

as an aid in determining the root cause(s) of operational 

errors. As part of this effort, the JANUS technique was 

programmed as a web-based tool (eJANUS) and began 

field-testing at en route centers. 

Based on the findings from the JANUS beta test and 

a field survey of operational personnel commissioned 

by AAT-200 (Conner & Corker, 2001), several studies 

were initiated. One study, using data from Indianapolis 

ARTCC, is being conducted by CAMI to examine the 

relationship between both static and dynamic sector 

characteristics and OE occurrence. A series of studies 

is examining the role of Operational Supervisors in OE 

prevention to identify requirements, practice, and “best 

practices” so that strategies can be developed to bridge 

these if needed.

Traffic Re-creations

During the 1990s, the FAA Air Traffic Evaluations and 

Investigations staff and Atlanta air route traffic control 

center (ARTCC) collaborated with the CAMI to develop 

the Systematic Air Traffic Operations Research Initiative 

(SATORI; Rodgers & Duke, 1993). SATORI graphically 

re-creates radar data that are routinely recorded by en route 

air traffic control facilities. SATORI enables its users to 

re-create traffic samples in a format similar to what was 

displayed to the controller, for example, showing relative 

location and separation, speeds, and headings of aircraft 

involved in an OE. SATORI is a tool that can be used 

to expand proficiency training with the capability of 

reviewing significant traffic samples, such as emergency 

situations, distracting events, peak periods of traffic flow 

(“pushes”), the effects of weather on traffic flow, and situational distracters, to promote learning from them. The 

Radar Audio Playback Terminal Operations Recording 

(RAPTOR) was developed by Air Traffic Services in the 

New England Region and is a similar re-creation tool for 

terminal operations. SATORI, RAPTOR, and other recreation tools are currently being used in diverse ways for 

training, e.g., OJT, skill enhancement, refresher training, 

and review of OEs during investigations.

Severity Index 

Every violation of separation standards provides an 

important opportunity for lessons learned and system 

improvement, although not all operational errors share 

the same characteristics. Separation standards and procedures differ depending upon, for example, the type 

of airspace, weather conditions, type of aircraft, and 

altitude. The U.S. Department of Transportation, Office 

of Inspector General (DOT, 2000) recommended that 

the FAA Air Traffic Investigations Division tackle the 

problem of modeling and defining the severity of OEs 

that occurred above the runway surface to describe the 

extent to which applicable separation standards were 

violated. The purpose was to group airborne OEs and 

thus be able to both focus resources on the most severe 

events and identify factors related to specific categories of 
C-6

events. Data about systemic causes of OEs could then be 

used to more explicitly direct action towards the prevention of future occurrences. 

In 2001, the FAA Air Traffic Evaluations and Investigations staff developed a classification system to distinguish 

between airborne OEs based on characteristics of the 

operational environment. The SI categories give the FAA 

a new method to distinguish high severity OEs from 

those less severe. This approach allows the FAA to focus 

energy on OEs having common characteristics. Targeting 

efforts towards the more severe OEs is a more effective 

use of available resources.

Training Initiatives

The FAA Air Traffic Evaluations and Investigations Staff 

initiated several programs that focused attention on

proactive skill building through performance maintenance and enhancement activities rather than remedial 

training programs.. It was anticipated that programs that 

provide regular ongoing support of skill sets routinely 

used by controllers to control traffic would sustain an 

already highly skilled workforce and would also decrease 

OEs when incorporated in the controller’s day-to-day 

performance. 

Video Briefing Materials. Analysis of OE data by the Air 

Traffic Investigations Division Staff revealed several complex system vulnerabilities. Consequently, they produced 

a series of videos to focus awareness on recurring factors 

found to occur during OEs. Periodically, videos were 

also produced to address factors related to incidents and 

accidents. Copies of these were sent to all FAA air traffic 

facilities for use as briefing materials.

The “Break the Chain” video illustrates how events, if 

uninterrupted, can culminate in an accident or incident 

and how attending to details can help break that chain of 

events. The video “Collision Course: What are the Odds?”

depicts an actual incident, demonstrating how rare and 

improbable events can occur. “Consequences of Simple 

Omissions” is a compilation of actual events illustrating 

how small omitted actions, lack of attention, failure to 

follow operational practices, compounded by poor facility 

practices, and lack of self discipline or professionalism 

resulted in incidents ranging from operational errors to 

fatal accidents. “Preventing Readback Errors,” describes 

strategies to reduce the complexity of communications 

and chunk information in meaningful units to facilitate 

understanding.

Skills Training – Keeping the Mental Edge. The National 

Air Traffic Professionalism (NATPRO) program was 

a new training approach that was initially developed 

under the sponsorship of the FAA Air Traffic Investigations Division. The program is a personal performance 

enhancement program intended to maintain skilled 

performance, rather than provide remedial training for 

OE reduction. Instead of relying solely on knowledgebased training, this approach integrates the concept of 

“performance coaching” by facility personnel who are 

certified as coaches (NATPRO Cadre Instructor, Course 

No. 55095) to deliver the training (NATPRO, Course 

No. 55096) at their facility. The program combines 

knowledge with practice so that controllers can experience 

their own strengths and limitations and can gain insight 

into their mental processing skills, especially those used 

in air traffic control. These skills include perception and 

vigilance, memory, and planning and decision-making, 

all of which are practiced in a dynamic environment. In a 

pilot field study at an FAA en route center, this program 

was demonstrated to be a more cost-effective, personally 

rewarding method of delivering training than traditional 

proficiency testing or remedial training programs (Breedlove, 2004). NATPRO Series 1 was deployed at en route 

centers in 2004.

The initial NATPRO program (Series 1) emphasized 

visual acquisition of information. The course content 

included multi-tasking, paying attention, scanning, detecting information, focusing on relevant information, 

being ready to take position, working “ahead of the traffic,” and staying “in your zone.” Although the skills are 

generic, it is expected that by improving general skills, 

individuals will demonstrate a corresponding improvement in job performance. By testing themselves against 

the computer and experiencing how their performance 

can vary in relation to factors such as distraction, fatigue, 

boredom, etc., participants gain increased understanding of their own performance and can develop personal 

improvement strategies. It will eventually be necessary to 

determine empirically whether those personal improvement strategies will lead to corresponding improvements 

in controller performance under simulated and actual 

job conditions.

Initiatives by Air Traffic Managers

In 2004, air traffic managers recommitted themselves to 

a program of immediate and long-term OE reduction 

through facility-level actions and consistent performance 

management at all organizational levels. The following 
C-7

are examples of initiatives designed by air traffic managers and their staffs to address facility-specific problems. 

No archival data are available to identify which of these 

were executed by what facilities or to assess the effects of 

these programs on OE reduction.

Communications. All operational personnel were briefed 

on performance management initiatives. Managers, OMs, 

and Operations Supervisors (OSs) held face-to-face briefings and focused on individual employees’ performance 

management history, CIC/OJTI status, and disciplinary 

history. Facility air traffic management briefed their facilities on expectations about performance management and 

the link between FAA performance, facility performance, 

and supervisory performance. OMs and OSs were required 

to provide feedback to their employees. 

Processes were examined to ensure that employees received 

the instruction and help they required when training 

needs were identified. These efforts guaranteed that special emphasis items would be assessed and/or addressed. 

Activities would be communicated and coordinated and 

appropriate follow-up actions would be initiated. 

Operational Standards. Each facility was required to 

develop clear operational standards of conduct and performance. These standards were tailored to meet specific 

facility needs and management-ensured 100% compliance. In the operational areas, for example, supervisors 

actively engaged in directing the operation (e.g., observing 

position relief briefings, interacting with TMU, staffing to traffic, making position and break assignments), 

performance management (e.g., on-the-spot corrections, 

teamwork), and managing the operational environment 

(e.g., reducing noise levels, limiting distractions). OMs 

and OSs ensured 100% compliance with requirements 

for technical training discussions, operational error follow-up, timely response, and thorough documentation of 

all identified performance deficiencies. All performance 

or conduct deficiencies were discussed with affected 

employees and documented for follow-up. 

Long-Term Strategies. Air traffic managers of en route 

and large TRACON facilities used the existing monthly 

telecon to discuss issues related to developmental training, performance management, refresher training, performance and conduct actions related to multiple OEs, 

and ongoing development and sharing of automation to 

support performance management.

Training strategies were implemented including a redesign of qualification training to include topics of traffic 

management, operational error prevention, teamwork, 

and changes to the air traffic environment (e.g., URET). 

Refresher training addressed causal factors trends related 

to operational errors. Developmental training was changed 

to include true simulation. The NATPRO program was 

deployed to all en route centers.

Teamwork. For a period of time, different incentive programs were popular. For example, “None in a Million” 

HQ awards were handed out to facilities having no OEs 

during a million facility operations. Competitions at 

the facility level also took place, with various incentives 

given for error-free performance, such as by one area of 

specialization within a center.

“Best Practices.” In 2004, Air Traffic developed and distributed a list of “best practices” for towers, centers, and 

flight service stations to emphasize a focus on “back to 

basics” performance. The list was distributed as a “pocket 

card” reminder. For example, “best practices” included: 

1) Give and receive a complete and thorough position 

relief briefing (record & overlap); 

2) Listen to ensure that readbacks are correct;

3) Ensure that coordination and communication are 

clear, complete, and correct; and

4) Avoid distractions. 

Performance Management Emphasis

The intent of this initiative was to make training proactive 

(i.e., administered before an OE occurs) and not reactive (i.e., administered only after an OE has occurred). 

All facility personnel were asked to plan and support 

performance management together. To accomplish this, 

a formal process was instituted for conducting Technical 

Training Discussions (TTDs). 

Technical Training Discussions (TTDs) and an example 

of implementation. A TTD is a discussion between the 

employee and his/her first level supervisor about the 

employee’s performance. To assess a controller’s performance, the first-level supervisor is required to conduct a 

TTD for each employee certified on at least one operational position. These discussions are neither “pass/fail” 

nor “satisfactory/ unsatisfactory;” nor do they include 

discussions of conduct.

TTDs are not based on a single “snapshot” of performance but instead on a continuous assessment of the 

employee’s technical performance using both direct and 

indirect methods and on the appropriate job functions 

and indicators as described in FAA Order 3120.4. Indirect 

methods may include remote monitoring, tape reviews, 
C-8

and SATORI re-creations of traffic samples that he/she 

controlled.

In a TTD, all training administered during an OE/D is 

reviewed to develop and direct individualized technical 

training that appropriately addresses identified technical 

performance issues. An employee may demonstrate overall 

acceptable technical performance, but might benefit from 

technical training on a particular skill or task. If no new 

issues are identified, this too is discussed and documented. 

Reports are designed to allow the facility to identify recurring and significant proficiency training needs so that 

effective training plans can be developed.

A TTD is not conducted only following an OE. They can 

be held whenever the first-level supervisor identifies an area 

in an employee’s technical performance that might benefit 

from individualized technical training, no earlier than 6 

months following the employee’s previously documented 

technical training discussion, no later than 60 days after 

the first-level supervisor assumes responsibility for an 

employee who has not had a technical training discussion 

documented during the previous 6 months.

Houston Center’s Quality Evaluation Program (QEP). 

In October 2003, Houston Center developed and 

implemented a Quality Evaluation Program (QEP) to 

improve the assessment of controller performance using Technical Training Assessments (TTAs) and TTDs. 

The QEP provides a number of different ways to look 

at performance. As a result of the program, most areas 

of specialization showed a significant improvement in 

performance proficiency. Facility QA staff expected that, 

since performance is directly tied to operational errors 

and deviations (OE/ODs), improvement in individual 

and team performance would help reduce the number 

of OE/ODs.

The cornerstone of this program was the establishment 

of two Quality Evaluation Supervisor (QES) positions in 

the facility. These supervisors are assigned to the Quality 

Assurance Office and conducted QEP assessments of 

controller performance independent of technical assessments conducted by the individual employee’s Operations 

Supervisors. The QES adhere to a strict standard of evaluation for each assessment. Items assessed include altitude 

assignment and confirmation, monitoring of readbacks, 

traffic advisories, route clearances, speed control, vectoring, beacon assignments, and point outs. An evaluation 

of the process showed that the QES assessments were 

accurate, consistent, and objective.

All deficiencies are addressed as quickly as possible by 

the first-level supervisor. In the event that a supervisor 

was unable to address a particular assessment within 15 

days, notification was given to the Quality Assurance 

Office so that tapes and/or SATORI re-creations could 

be made and retained for review. 

An improved system for tracking technical performance 

trends was also developed. The QESs maintained a master 

schedule of TTAs and TTDs and an electronic folder for 

each employee. The master schedule ensured that each 

employee received a TTA every other month and a TTD 

every six months. Any necessary follow-up assessments 

were accomplished during the intervening months. In 

an “After Care” program, the QESs performed random 

TTAs for employees involved in recent operational errors/operational deviations/incidents. 

Formal organizational recognition of good performance 

was also part of the program. If an employee performed 

without a deficiency during a TTA, formal recognition was 

awarded. The recognition program included certificates, 

monthly posters, and inclusion on a Regional Honor 

Roll. “Fantastic Feats” was the first program employed 

for this purpose; however, to maintain interest and spark 

involvement, the program changed periodically. The 

“Chain of Excellence” Program recognized all deficient 

free performance as a “Link in the Chain.”

The QES supervisors and the Quality Assurance (QA) 

Manager met with the OMs on a regular basis to discuss all 

aspects of the program and make adjustments as needed. 

The QES assessments were tracked so that results could 

be examined at the level of the individual, specialty, and 

facility. All performance deficiency trends were incorporated into current and future training programs. 

In addition to the finding that most specialties showed a 

significant improvement in controller performance, TTA 

data indicated that technical performance deficiencies 

decreased an average of 16.6% in the facility and that 

deficiency-free performance increased by an average of 

50.8% over the same period. 

These data can also be used to produce various types 

of facility information. For example, although the majority of specialties showed a significant improvement 

in performance during the time period reported, one 

specialty showed an increase in deficiencies. However, 

this was attributed to an increase in the number of items 

being assessed and the presence of heavy weather for the 
C-9

majority of one assessment period. The overall number of 

deficiency-free performances increased for the same time 

period. Also, individual performance found to be deficient 

during other assessment periods showed improvement 

in later assessments. In sum, the improved system for 

tracking technical performance trends permitted performance deficiencies to be addressed through quality 

training in a timely manner both for the individual and 

for the facility. 

Kansas City Center’s Performance Management Program. In 

late 2002, Kansas City ARTCC initiated a Performance 

Management Program (PMP). The PMP program was 

based on the belief that the best OE initiative is directed 

at the level of the individual rather than at groups. The 

program elements included a Quality Assurance Review 

(QAR), a direct observation module, and a TTD. The 

QAR is a process conducted for other types of system 

deficiencies unrelated to OEs. One element of a QAR 

is a review of the employee’s performance, along with 

other factors, such as procedures or equipment, which 

may have contributed to a deficiency in the control of 

traffic. QAR activities may range from a discussion of 

the situation with the involved controllers to reviewing 

recorded radar and voice communications data.

All forms used for these processes are automated and 

linked to a reporting feature. For example, the QAR 

module and the direct observation module automatically 

feed the TTD program. This process resulted in very few 

TTDs being conducted without including individual, 

specific performance information in the review. This 

information is powerful when reviewed by the supervisor 

with the controller. For example, the controller and the 

supervisor can review a sample of voice tapes representing 

a controller’s communications on position over a period 

of several weeks. They provide a means for the controller 

to review normal phraseology habits. It is an opportunity 

for the controller to determine whether good or poor 

habits have been developed. This assists the controller 

to enhance his/her communications skills.

After initiating the program in April of 2003, the center’s 

OE/ODs declined for the next three years, resulting 

in a 2002 “commendable” rating from the national 

evaluation team. Table 1 (Truelove, Hatem, & VanDyne, 

2005) shows the number of OEs by year of program 

implementation.

Computer-Based Instruction in Human Factors. Responding to the push to reduce OEs, personnel in the Great 

Lakes Region’s Air Traffic QA office distributed a threemodule program on human factors. FAA headquarters 

staff provided the program to interested facilities nationwide. The material covered topics such as attitude, 

cooperation, communication, organization, efficiency, 

perception, memory, and attention. Most of the material 

appeared to be based on generally accepted psychological 

and physiological theories. 

An individual knowledgeable in both human factors and 

ATC reviewed the material and expressed the opinion that 

controllers would find the lessons entertaining but that 

the material would have limited training value. Without 

supporting data we don’t know whether the materials can 

be applied directly to controller performance. Take, for 

example, statements such as “controllers who recognize 

and maximize situational awareness skills automatically 

distance themselves from high-risk situations and errors.” 

While this statement might be true, it is not clear whether 

this observation about automatic behaviors was based 

on any data and, if so, whether they were collected from 

operational controllers. Similarly, statements suggesting 

that controllers select, process and remember only those 

perceptions on which attention has been focused (i.e., in 

focal attention) ignores what is known about how people 

also process information that is not the focus of attention or information of which they are not fully aware. 

This emphasis on focal attention might lead controllers 

to discount other processes that can also lead to human 

error. Suggestions that controllers have a physiological 

structure that allows them to know the difference between 

important and unimportant information seem to be irrelevant or misleading. Examples given of important and 

unimportant information are of a fire versus a lawn mower. 

The authors suggest that somehow the physiological brain 

structure knows the difference between what is important 

and what is trivial information. Such broad statements 

risk giving controllers unwarranted confidence to rely on 

their physiological structure as a form of performance 

“backup” system. Finally, the statement that the complex 

tasks of air traffic control require faster processing speed 

to complete operations should also include a discussion 

of experience, skill, etc. and their effect on performance. 

Thus, there are a number of concerns associated with the 

development and use of this training module.

Table 1. ZKC OEs by year of PMP implementation. 

Program year Date # OEs 

Prior year 1/1/01 – 12/31/01 29 

First year 1/1/02 – 12/31/02 27 

Second year 1/1/03 – 12/31/03 19 

Third year 1/1/04 – 12/31/04 20 

Fourth year 1/1/05 – 7/31/05 10 
C-10

Sheridan’s Analysis of OE Reduction Efforts. Sheridan, 

(1997) analyzed the effectiveness of OE reduction efforts 

conducted in the Great Lakes Region during fiscal year 

1996 (FY96). At the beginning of FY96, several initiatives were initiated by regional headquarters’ managers in 

an attempt to reduce OEs. These included assigning an 

OE quota to field managers, not to be exceeded within 

each air traffic manager’s area of responsibility. Air traffic 

managers were held accountable through negative performance evaluation ratings. For his analysis, Sheridan 

used data from both OEs and operational deviations 

(which accounted for approximately ten percent of the 

total error rate). Results showed that intervention efforts 

had a mixed effect on OE rates. 

The en route OE rate exhibited a modest correlation with traffic volume (r = .497). Traffic explained 

only approximately one-quarter of the variance in 

the OE rate. 

The terminal OE rate also exhibited a modest 

correlation with traffic volume (r = .460). Traffic 

explained roughly 21% of the explained variance 

in the OE rate.

Error rates rose marginally in the en route environment 

while a modest improvement was realized in the terminal 

environment. Data revealed that en route OE rates rose 

from 8.50 per month to 9.83 per month. By contrast, 

tower OEs on average fell from 5.78 per month to 5.0 

per month. One possibility for the differences lies in the 

changes in traffic volume experienced during the different 

reporting periods. However, analyses revealed that traffic 

volume explained a modest percentage of the explained 

variance in OE rates.

Runway Safety Initiatives

In October 2002, after an accident involving arriving and 

departing aircraft at a municipal airport, the FAA adopted 

NTSB recommendations (FAA, 2002b) and required air 

traffic control tower facility managers to include local 

procedures in their facility directives to assist the ground 

and local controllers in maintaining awareness of aircraft 

positions on the airport. Procedures were adjusted to address the needs of a specific facility or airport.

The Runway Safety Office identified several initiatives 

for controllers covering topics related to communications 

and teamwork. CBI Modules for ATC were developed 

by the Runway Safety Office on topics of awareness of 

conditions, procedures, rules, and tools. The Memory 

•

•

Enhancement Project convened a workgroup to review 

availability of off-the-shelf programs and to identify 

memory aids currently used at facilities. Originally, the 

results were intended to be distributed to other facilities 

for their consideration as potentially useful tools. However, 

distribution was not completed.

In 2001, a booklet entitled “Runway Safety: It’s Everybody’s Business” was distributed (Cardosi, 2001). Half 

of the material was directed at controllers to describe 

what they could do to improve surface safety. The other 

half of the material was directed at pilots and their role 

in surface safety. Topics included causes of OEs, memory 

strategies, how to improve pilot-controller communications, the importance of teamwork, and strategies to 

mitigate fatigue. Additional recommendations include 

the following. Only recommendations with an ATC 

focus are listed here. 

Controller Communications. Three recommendations 

were made. 

Minimize controller-pilot voice communications, 

e.g., eliminate unnecessary elements of controller 

transmissions and explore expanded use of standard 

(classified) taxi routes. 

Develop operational protocols for ATC communications in airport operations, e.g., a requirement for 

pilots or ground personnel to use standard phraseology when communicating with ATC. 

Implement improved communication technology to 

reduce blocked or partially blocked transmissions.

AT Procedures. One recommendation was made.

Improve procedures for taxiing aircraft into position, 

holding on the runway, and crossing intersecting 

runways.

Controller Training. Three recommendations were 

made.

Promote training in communications and coordination, such as cross-operational training for pilots and 

controllers on the others’ duties and concerns.

Provide controllers with refresher training (e.g., 

phraseology, memory enhancement, scanning) for 

surface operations. 

Implement teamwork training to improve communications, cooperation, and operations planning 

between AT and airport users.

•

•

•

•

•

•

•
C-11

Regional Runway Safety Initiatives. Several regional air 

traffic and runway safety offices developed additional 

materials and programs to reduce OEs, including a number of initiatives implemented by the Regional Runway 

Safety Program Managers, such as pilot briefings, Runway Safety Action Team (RSAT) meetings with airport 

operators, tenants, and tower personnel, and initiatives 

with AOPA, such as issuance of publications on runway 

safety, videos, etc. All of these efforts were geared to 

minimize the possibility that pilot actions would result 

in incidents, including OEs. 

A real-time, human-in-the-loop simulation study (called 

ACTIVE-1) was conducted at Anchorage Tower by the 

FAA Alaskan Region Runway Safety Office to determine 

whether a high fidelity tower simulator could be useful to 

enhance controller training (Racine & Sierra, 2004). In 

particular, the study investigated whether a high fidelity 

tower simulator would improve controller recognition and 

management of common factors leading to runway incursions (RI), and improve their skill in taking appropriate 

actions to mitigate unanticipated events. The study used a 

series of operationally challenging scenarios and a state-ofthe-art tower simulator to determine if fully qualified and 

current tower controllers would benefit from simulator 

practice. Performance improvement from pretest to posttest was based on measures of efficiency and safety. Results 

showed significant improvements in detection of specific 

scripted errors as well as general trends of improvement 

in other system performance measures.