Taste 5
PERCENTAGES OF ERRORS THAT WERE MODERATELY SEVERE BY Causat Factor AND ATCS AwakeENess

OVERALL FLs290 FL>290
Yo N % N % N
MODERATE MODERATE MODERATE
Factors Involved with Greater Severity
Misuse - Conflict Alert 52.0 25 60.0 20 20.0 5
Communications 36.9 379 45.0 300 6.3 79
Readback 40.8 206 47.9 169 8.1 37
Readback - Altitude 41.1 151 46.5 127 12.5 24
Coordination 38.5 161 42.7 110 28.0 51
Inter-facility Coordination 53.8 26 60.0 20 33.3 6
Factors Involved with Lesser Severity
Misuse of Displayed Data 22.0 473 27.9 323 9.3 750
(excluding conflict alert)
ATCS Aware Error was Developing 21.0 267 26.0 177 11.1 90
Note: All of the causal factors above were significant at ps.01 on chi-square tests.
N= the number of OEs that were related to each factor.
%= the percentage of each N that was moderate (compared to minor) in severity.
FL=_ flight level, 290=29,000 feet.
Taste 6
Dirrerences tN HoriZONTAt SEPARATION BeTWetw AIRCRAFT WHEN OPERATIONAL ERROR
Invotveo Certain Causal Factors AND ATCS Awareness
HORIZONTAL SEPARATION
OVERALL FL<290 FL>290
MILES N MES N MILES N
Factors Involved with Greater Severity
Misuse - Conflict Alert -.31 25 -.34 20 -.15 5
Communications -.25 379 -.28 300 16 79
Readback -.30 206 -.27 169 -.39 37
Readback - Altitude -.29° 151 ~.25 127 -.43 24
Coordination -.25 161 -.20 110 -32 51
inter-facility Coordination ~42 26 -.43 20 -.41 6
Factors Involved with Lesser Severity
Misuse of Displayed Data 34 473 33 323 37° «150
(excluding conflict alert)
ATCS Aware Error was Developing .28 267 33 177 13 90

Note: The values presented are mean differences in horizontal separation in miles for errors
that involved a given causai factor versus all other errors. Negative values represent less
separation at occurrence while positive values indicate greater separation.

N=number of O€s that were related to each factor.

16


conditions during the OEDs were found across the
ARTCCs. Some degree of variability would be expected,
given that the workload measures are reported by
many different investigators. Although the report form
and associated information provides derailed
information concerning the evaluation process, certain
determinations remain subjective. In particular, the
criteria for categorizing air traffic complexity seers to
require additional study. Ic is possible that an analysis
of comments provided in the section of the report
devoted to “explanation of traffic complexity” could
provide some insight into differences in the rating
procedures. Also, to what extent and how consistently
are the “number of aircraft being worked at the time of
occurrence” reported as the number handled at the
time of actual loss of separation or as the number
handled during a specified time period surrounding
the OED?

Some causal factors were found to occur more often
under certain workload levels. The strongest associa-
tion between workload and the type of causal factor
contributing to the OEDs was for data posting - which
involves computer entry and processing of flight
progress strips. However, these relationships were small
and tended to be evident only at the extremes of the
workload conditions. These findings offer some sup-
port to Sperandio (1971), who found that alterations
in workload led ATCSs to adopt different modes of
operation. Also, the results are consistent with recent
research regarding how air traffic controllers har ‘le
board management under varying traffic scenarios
{Vortac, O.U., et al). A question for future study is
whether, and to what extent, air traffic controllers
implemen: less effective a) procedures in response to
changes in workload, and ) strategies as a conse-
quence of high workload?

The number of incidents at ARTCCs during the
calendar years 1985-88 was found to be positively
associated with total number of facility operations, but
inversely related to the ARTCC’s proportion of air
traffic controllers who were at full performance level.
This relationship is consistent with previous research
on operational errors at ARTCCs for 1987 (Splawn,
Edwards, & Chin, 1988). In the current study, it was
found that after accounting for the total number of
operations and FPL ratio variables, a facility’s average

air traffic complexity level (reported at the time of its
OEDs) was also a significant predictor of to:al OEDs,
with 84% of the variance associated with the OED
data subsumed by those three variables. Thus, al-
though most OEDs occurred under average or lower
air traffic complexity, this more qualitative measure of
workload helped explain the variation in the number
of OEDs at en route facilities.

REFERENCES

Kershner, A. M. (1968). Air traffic control system error
data for 1965 and 1966 as related to age, workload,
and time-on-shift of involved controller personnel.
Atlantic City, New Jersey: FAA, NAFEC Report
No. NA-68-32.

Kinney, G. C., Spahn, J., & Amato, R. A. (1977). The
human element in air traffic control: Observations
and analyses of the performance of controllers and
supervisors in providing ATC separation services.
McLean, VA: METREK Division of the MITRE
Corp., MTR-7655.

Schroeder, D. J. The loss of prescribed separation be-
tween aircraft: How does it occur? Proceedings (P-
114), Behavioral Objectives in Aviation Automated
Systems Symposium, Society of Automotive Engi-
neers (SAE), October, 1982, 257-69.

Sperandio, J.C. (1971). Variation of operator’s strategies
and regulating effects on workload. Ergonomics, 14,
571-77.

Splawn, W. j., Edwards, C. N., & Chin, K. M. (1988).
Profile of operational errors in the National Airspace
System - calendar year 1987. Washington, D. C.:
Federal Aviation Administration, Office of Avia-
tion Safery.

Stages, P. & Hameluck, D. (1990). Ergonomics in air
traffic control. Ergonomics, 33, 493-99.

Vortac, O.U., Edwards, M.B., Jones, j.P., Manning,
C.A., & Rotter, A.J. (1992). Ex Reute Air Traffic
Controller's Use of Flight Progress Strips: A Graph-
Theoretic Analysis, Washington, D.C.: Office of
Aviation Medicine Report, DOT; FAA/AM-92/31.


Taste

16

Prorite OF AircRAFT in MoperaTety Severe OEs at En Route Faciuiies

ALL DESCEND ALL LEVEL LEVEL ALL
FACILITY CLIMB CLIMB DESCEND CLIMB DESCEND LEVEE
ZAB 0 Oo 22.2 0 77.8 0
ZAU 0 6.5 0 22.6 61.3 9.7
ZBW 0 15.8 5.3 21.1 52.6 5.3
ZDC 2.9 2.9 5.9 20.6 58.8 8.8
ZDV 0 0 9.1 36.4 45.5 9.1
ZFW 0 0 0 37.5 43.8 18.8
ZHU 7.7 0 23.1 30.8 30.8 7.7
ZID 4.5 13.6 45 31.8 13.6 31.8
ZjX 0 5.0 5.0 20.0 60.0 10.0
ZKC 0 0 0 55.6 33.3 11.1
ZLA 0 0 0 33.3 46.7 20.0
ZLC 0 25.0 25.0 -0 25.0 25.0
ZMA 9.1 18.2 -0 27.3 36.4 9.1
ZME 0 0 0 50.0 50.0 0
ZMP -O ae) 20.0 40.0 20.0 20.0
ZNY 3.8 7.7 7.7 30.8 46.2 3.6
ZOA 0 20.0 0 20.0 40.0 20.0
ZOB 5.3 5.3 5.3 31.6 42.1 10.5
ZSE 0 0 16.7 33.3 50.0 0
ZTL 0 11.1 1141 22.2 44.4 W441
Overal! 1.7 6.6 8.0 28.2 43.9 11.6

Note: The data are percentages of a facility’s moderately severe OEs that occurred under the

various aircraft profiles.

awareness (Quadrant 3), while ZNY and ZOA re-
ported relatively low ATCS workload combined with
greater awareness (Quadrant 4).

There was also considerable variability in the fre-
quency of the causal factors previously found to be
related to severity. For example, the percentages of
QEs that involved Misuse of Displayed Data ranged
from 11% 10 44% across facilities (Table 9). Similarlv.
while Readback was involved in 26% of all moderately
severe OEs, at the facility level, between 0% and 46%
of the OEs involved Readhack as a factor.

Table 10 illustrates the percentages ct moderately
severe OEs thar occurred under the various aircraft

profile categories described previously. The most fre-

20

quent profile - aircraft leve! and aircraft descending -
characterized over 40% of the OEs with a range of
13.6% to 77.8% across facilities. The “All level” air-
craft profile was present between 0% and 31.8% de-
pending on facility.

Overall these results illustrate why future research
regarding operational errors should measure facility-
level differences to estimate the extent to which find-
ings are generalizable among ARTCCs. It would be
interesting to investigate the relationships that may
exist between these facility-icvel differences and facil-
ity differences in traffic density, local procedures. and
airspace configuration.


TABLE OF CONTENTS

An Examination of the Workload Conditions Associated with
Operational Errors/Deviations at Air Route Traffic Contro} Centers
David J. Schroeder and

Lendell G. Nye

sesavesnersnasansssssnes socaerecensetenasessesouassnenessteoesseseresesoesesscarsdsnecsareceoreranearecens 1
Factors Associated with the Severity of Operational

Errors at Air Route Traffic Control Centers

Mark D. Rodgers and

Lendel] G. Ny@ .csccssssessessssesssscsesscssessssstcnscecessesssescsuesessecsesusscaneseseeeeueseaceneaeeeeneass 11


Taste 7
PERCENTAGES OF Errors THAT WERE Moperatety SEVERE 8Y Pgorite OF AIRCRAFT

OVERALL FLs290 FL>290
%o N % N Yo N
MODERATE MODERATE MODERATE

Profile of Aircraft
All Climbing 21.9 32 23.3 30 0.0 2
Level & Climbing 26.3 323 34.4 218 9.5 105
Descending & Climbing 28.0 75 31.1 61 7.7 13
Level & Descending 30.3 469 37.6 346 9.8 123
All Descending 34.5 58 36.5 52 16.7 6
All Level 43.0 86 52.9 51 28.6 35
Note:
N= the number of OEs that occurred in each aircraft profile
%= the percentage of each N that was classified as moderately severe
FL= flight level, 290=29,000 feet

jer that the error was developing. Twenty-two percent
of the 473 OEs that involved Misuse of Displayed
Data resulted in a moderately severe error, and simi-
larly, 21% of the 267 OEs in which the ATCS was
aware of the developing situation resulted in errors of
moderate severity. By contrast, other causal factors
were more likely to result in moderate severity. In
particular, over 50% of errcrs that involved Misuse of
Conflict Alert or Inter-facility Coordination were clas-
sified as moderate. Another factor related to a greater
loss of separation was Readback (a communication
during which the ATCS fails to detect a pilot’s incor-
rect response to a clearance provided by the ATCS)
and more specifically, “Readback invoiving altitude
information.” When Readback was a causal factor,
over 40% of those errors were classified as moderately
severe.

Table 6 lists the differences in average horizontal
separation (overall, and at FL 290- or FL 290+) be-
tween those errors that involved each of the causal
factors in Table 5 versus those that did not. For
example, for the errors in which Misuse of Displayed
Data (excluding use of Conflict Alert) was involved,
the average horizontal separation between aircraft was
.34 miles greater than the OEs that did aot involve this
causal factor. By contrast, in the OEs in which Readback

was a factor, the resultant horizontal separation was
-30 miles less than the OEs that did not involve
Readback. We found that vertical separation was not
significantly related to any of the factors. Thus, the
impact of these factors in terms of OE severity cat-
egory, as illustrated in Table 5, was found to be related
primarily to the horizontal separation parameter.
Table 7 examines the relationships between aircraft
profile, in conjunction with flight levels, and OE
severity. Tests of significance were not conducted, due
to low expected values in some cells; however, the
results are presented for descriptive purposes. Most
OEs in the overall sample occurred when one or more
aircraft were level and a) others were climbing (N=323,
31.0%), orb) others were descending (N=469, 45.0%).
Surprisingly, the greatest likelihood for moderate se-
verity occurred when all aircraft were at “level” flight
(43.0% overall}; 52.9% at FL 290-, and 28.6% at FL
290+. A‘so, flight level was related to severity for each
aircraft profile; i.c., a greater percentage of OEs were
moderately severe at FL 290-, compared with FL 290+.

17


Characteristics of OEs of Major Severity

Six of the 15 OEs categorized as “major” had 2
complexity rating of “above average,” while four were
classified as having occurred under “average”
conditions, rwo had a “below average” rating, and 3
had an “easy” rating; producing a mean complexity
rating of 2.87. The average number of aircraft being
worked was 8.47 with the range between 2 and 15.
These findings illustrate the sizable variability in ATCS
workload conditions found during the OEs classified
as “major” in this study.

Regarding the flight characteristics of the aircraft in
the major OEs, three of the OEs occurred at FL 290+.
In five cases, ali of the aircraft involved were in a level
profile, in seven the aircraft were level and descending.
in one OE the profile was level and climbing, and two
occurred when aircraft were descending and climbing.

Table 11 shows the frequencies of the causal factors
that resulted in OEs that were rated “major” in sever-
ity. The Daa Posting factor was involved in five OEs,
Radar Display in nine, Communications in six. Goor-
dination in five, and Relief Briefing in one OE. (As
noted previously, an OE can involve multiple causal
factors).

DISCUSSION

General Findings
A counter-intuitive finding of this study was that
neither a quantitative measure of air traffic controller
workload (number of aircraft being worked) nor a
rating of air traffic complexity was found to be related
to OE severity. Future research could examine the
correspondence between these “objective” workload
measures and “subjective” workload assessments made
with instruments such as the National Aeronautics and.
Space Administration Task Load Index (Hart &
Staveland, 1988) in order to expand the understanding
of the workload conditions of air traffic controllers.
Causal factors that relate to ATCS awareness would
likely include those that are not easily corrected if not
caught immediately, allowing the error situation to
build until it is too late to avoid. In general, the causal
factors that resulted in a greater loss of separation were
those which would likely reduce awareness of the
situation by the air traffic controller (i.e. inter-facility

coordination, incorrect readback of altitude by pilot,
etc.). This assertion is supported by the finding that if
the ATCS was aware the error was developing, the
errors tended to be less severe. Additionally, the misuse
of displayed data (excluding conflict alert) was associ-
ated with less severity, which suggests that the factors
associated with severe errors have an insidious nature
that is not characteristic of errors that involve the
misuse of displayed data.

Itis interesting to note that, for each factor, only the
horizontal separation parameter, not vertical separa-
tion, was significantly affected, This may be because
altitude. used to provide vertical separation, is re-
ported numerically in the data block. Thus, altitude
information is likely to be much more salient than the
information used to make a judgment of the extent of
horizental separation between aircraft. Judgment of
horizontal separation is based on the visual distance in
separation between two targets on the plar view dis-
play and judgments of their relative speed. However,
ATCSs may also prefer zo use the horizontal parameter
to achieve separation, therebv resulting in a difference
that is based primarily on frequency of usage.

Although most operational errors occur when at
least one aircraft is in level flight and at least one
aircraft is either descending or ascending, the “All
Level” aircraft profile was found to be associated with
a greater percentage of moderately severe operational
errors. Perhaps less attention is directed to aircraft in
the “All Level” profile, resulting in a grearer likelihood
of severe errors. Additionally, moderately severe errors
were more likely to occur at FL < 290. However, at FL
s 290, when aircraft are in an “All Level” profile, they
typically are flying at faster speeds and thus. separation
can be lost more rapidly, potentially resulting in a
greater loss of separation between aircraft.

It should be borne in mind that these are national
averages and the values are different for certain facili-
ties. Differences in traffic density. local procedures,
and airspace configuration may limit the ability to
generalize these results across different facilities. This
is evidenced by the results of the facility level analyses
presented earlier. Furthermore, without additional
information concerning the percentage of time ATCSs
at the various facilities spend controlling traffic under

23


Taete 9
Causat Factors Associatep with Moperatety Severe OEs at En Route Facuumies (Y% OF Errors)

DISPLAYED CONFLICT READBACK- INTER-FACILITY
FACILITY DATA ALERT READBACK —_ALTITUDE COORDINATION
ZAB 22 11 33 22 00
ZAU 32 03 32 26 00
ZBW 32 16 21 WV 05
ZDC 43 00 20 14 00
ZDV 27 00 27 27 00
ZFW 38 06 25 25 3
ZHU 23 038 46 31 00
ZID 35 oo 30 30 09
ZX 25 15 2 20 05
ZKC 11 30 44 22 oo
ZLA 33 30 27 13 07
ZLC 100 00 00 90 00
ZMA 27 09 09 09 00
ZME 20 00 30 20 20
ZMP 40 20 0G 00 00
ZNY 23 04 19 15 12
ZOA 27 00 45 09 18
ZOB 26 00 32 26 00
ZSE 33 oo 33 33 oo
ZTL 44 co 33 11 00
Overall 33 05 26 18 04

Note: The data are the percentages of a facility’s moderately severe OEs that involved these

causal factors.

To examine ATCS workload in conjuaction with
awareness that the error was developing, multidimen-
sional scaling (MDS) was applied, with the results
shown in Figure 1. First, the two workload measures
and ATCS awareness daia were aggregated at the
facility level. Then the “z” score values for number of
aircraft being worked and traffic complexity were
summed to compute 2 composite measure represent-
ing the average ATCS workload at the time of OE
occurrence for each facility. A dissimilarity matrix
among facilities was created based on Euclidean dis-
tances using the composite workload and error aware-
ness variables. The matrix was then analyzed using the
classical MDS approach, in which the cases (en route
facilities) were plotted in two dimensions. Thus, MDS

provided 2 “geometric” representation of the facilities
sumilarity/dissimilarity in terms of workload and ATCS
awareness. Dimension | geometrically represents ATCS
workload, while dimension 2 is ATCS awareness that
the error was developing. The intersection of the
dimensions produced four quadrants into which facili-
ties were grouped, based on their relative similarity/
dissimilarity among each other. Quadrant 1 was de-
fined by relatively high ATCS workload combined
with greater than average error awareness. This high
workload- greater awareness condition was illustrated
best by ZDV, ZLC, and ZKC. The high workload-less
awarer.ess facilities (Quadraat 2) included ZFW, ZTL,
and ZME. ZAU and ZBW were characterized by
relatively low ATCS workload combined with less


(1)
(2)
(3)
(4)

b. Flight progress strip

(1) Not prepared

(2) Not updated

{3) Posted incorrectly ..

(4) Reposted incorrectly

(5) Updated incorrectl

(6) Sequenced incorrectly .

(7) Resequenced incorzectly
(8) interpreted incorrectly .

(9) Premature r.-noval

(10) Other ..

2. Radar display
a. Misidentification

q)
(2)

(3)
(4)

(5)
(6)

1. Data posting
a. Computer entry

Tagie 2

Percentaces OF “Minor” or “Moperate” Errors at EN Route Facitities
THAT INVOLVED Each Causat Factor

Incorrect inpu'
Incorrect update .
Premature termination of data
Other ou...

WONDORNDOWOON AANA

Overlapping data blocks.
Acceptance of incoraplete or
difficult to correlate position

info vel
Improper use of identifying turn ...0
Failure to reidentify aircraft when
accepted target identity becomes
questionable vee
Failure to confirm aircraft identity
after accepting a radar handoff ....0
Othe oe ceeceeeeeeesseereneneeteneeeeee 11

1

b. inappropriate use of displayed data ..47

(1)
(2)
(3)
(4)
(5)

3. Communications error ..
a. Phraseology..
b. Transposition

Conflict alert
Quick look
Mode C.....
MASAW/EMSAW
Other

Note: 0 indicates < .5%

14

a9

e.
f

5. Position relief briefing deficiencies noted

. Other...

Misunderstanding
Readback...
(1) Altitude
(2) Clearance
(3) Identification
Acknowledgment
Other

fob tpl
bhWROR

4. Coordination 0... ceescsecsesecseesses ceneenee 15
a

Area of occurrence
(1) Inter-position
(2) Intra-position
(3) Inter-sector
(4) Inter-facility ..
An aircraft penetrated designated
airspace of another position of operation
or facility without prior approval ........ 4
Coordination was effected and
controller(s) did not utilize informa-

tion exchanged
(1) Aircraft identification
(2) Altitudes/Flight Level
(3) Route of flight
(4) Clearance limit
(5) Speeds
(6) APREQS
(7) Special in
(8) Other

NOCCO=NOH

w

Employee did not use position relief
checklist .
Employee being relieved gave
incomplete briefing.....
Relieving employee did not make use
of pertinent data exchanged at

briefing



Taste 4
Percentace or OEDs Occurring UNpeR EACH OF THree WorkLOAD Catecories For EACH ARTCC

NUMBER OF AIRCRAFT
1-6 7-11 ¥Z OR MORE
ARTCC AIRCRAFFE AIRCRAFT AIRCRAFT N
ZAB - Albuquerque 22.7% 50.0% 27.3% a4
ZAN - Anchorage 23.1% 11.5% 65.4% 26
ZAU - Chicago 31.0% 43.7% 158
ZBW- Boston 45.2% 43.7% 435
ZDC - Washington DC 25.9% 49.7% 185
ZDV - Denver 15.4% 44.9% 78
ZFW - Fort Worth 33.7% 51.8% &3
ZHU - Houston 38.8% 43.5% 85
ZID - Indianapolis 29.7% 58.1% 155
ZJX - Jacksonville 21.8% 43.7% 87
ZKC - Kansas City 30.6% 55.6% 108
ZLA - Los Angeles 53.2% 39.7% 126
ZLC - Salt Lake City 13.5% 32.4% 37
ZMA- Miami 16.7% 66.7% 42
ZME - Memphis 19.7% 55.3% 76
ZMF - Minneapolis 22.5% 49.3% 71
ZNY - New York City 39.0% 32.6% 172
ZOA - Oakland 29.6% 36.6% 71
ZOB- Cleveland 30.3% 41.0% 122
ZSE - Seattle 41.9% 32.3% 31
ZSU - San Juan 15.2% 36.4% 33
ZTL - Atlanta 32.5% 39.5% 114
Combined 31.1% 44.8% 2039

Note:X7(42)=199.45; p<.001

spend controlling traffic under the various complexity
or workload conditions it is difficult to determine the
primary factors associated with these outcomes.
Differences in percentages of OEDs occurring un-
der different reported workload conditions were also
evident in ihe three categories of “number of aircraft
worked” (Table 4). Six facilities (ZAN, ZDV, ZJX,
ZLC, ZOA, and ZSU) had more than 30% of their
OEDs occur when “12 or more” aircraft were being
worked. ZBW, ZFW, ZID, ZKC and ZLA had less
than 15% of their OEDs occur under this higher
workload condition. Three ARTCCs (ZBW, ZLA,
and ZSE) had more than 40% of their OEDs occur

w

under the lower traffic condition (6 aircraft or less).
Once again, additional information is needed to clearly
understand the basis for these differences. To what
extent are the differences related to how determina-
tions are made concerning the aumber of aircraft
handled or to aspects of the underlying traffic condi-
tions?

Correlations between che two workload measures
“number of aircraf:” (#WRKD) and complexity
(CMPLXTY) and the five causal faccor categories are
presented in Table 5. Z#WRKD refers to the value of
the “number of aircraft” measure, following transfor-
mation into a 2 score, standardized within each facil-


Il.

Il.

Iv.

APPENDIX A
Causat Factor DeFiniTions*
Data Posting - A data posting error is any error of calculation, omission, or incomplete data, erroneous

entries, handling, or subsequent revisions to this data. This includes errors in posting and recording data.
It does not include errors involved in receiving, transmit

ing, coordinating, or otherwise forwarding this
information.

Radar Display

A. Misidentification - Radar misidentification means a failure to properly identify the correct target and
includes subsequent errors committed after the original identification was properly accomplished.
Indicate the listed item(s} most closely describing the reason for misidentification.

B. Inappropriate Use of Displayed Data - A data or display information error occurs due to a failure to
maintain constant surveillance of a flight data display or traffic situation and to properly present or
utilize the information presented by the display or situation.

Communications - A communications error is a causal factor associated with the exchange of information

between two or more people (e.g., pilots and specialists). It refers to the failure of human communication

not communications equipment.

A. Phraseology - Use of incorrect or improper phraseology.

B. Transposition - Errors due to transposition of words, numbers, or symbols by either oral or written
means. This involves writing/saying one thing while thinking/hearing something else.

C.  Misunderstanding - The failure to communicate clearly and concisely so that no misunderstanding
exists for any actions contemplated or agreed upon.

D. Readback - The failure to identify improper or incorrect readback of information.

E. Acknowledgment - The failure to obtain an acknowledgment for the receipt of information.
Coordination - Any factor associated with a failure to exchange requirement information. This includes
coordination between individuals, positions of operation, and facilities for exchange of information such
as APREQ's, position reports, forwarding of flight data, etc.

Position Relief Briefing - Relief briefing errors are special errors of both communication and coordination
that occur as the result of position relief. They include such things as failure to give a relief briefing, failure

to request a briefing, incomplete or erroneous briefing, etc.

* These definitions were extracted from FAA Form 7210-3.

Al BUS. GOVERNMENT PRINTING OFFICE: 1996 - S60-063/x0801



function of a) primarily, the total number of opera-
tions and b) secondarily, the percentage of ATCSs at
the facility who have reached full performance level
(FPL). This finding for calendar year 1987 was evalu-
ated for consistency over the entire 1985-88 time
period and was expanded to include the facility’s
average complexity of air traffic reported during the
OED occurrence as an additional potential predictor
of the number of OEDs for the ARTCC facilities.

METHOD

GEDS Data Base

Information was extracted from the data base for all
1985-88 ARTCC OEDs. The “number of aircraft”
variable refers to the number of aircraft that were being
worked (radar identified and/or established radio
communication) at the time of the OED occurrence.
Traffic complexity involves the use, by the quality
assurance specialist, of a five-point rating scale (traffic
was 1- “easy” to 5- “complex”) to indicate the “overall
difficulty of the controller’s task considering weather,
traffic mix, variety of operations, limited use of altitudes,
airspace available for radar vectoring, coordination
requirements, etc.” (FAA Form 7210-3). [t should be
noted that the instructional guide for the form differed
from the OED report form in referring to the initial
end point as “not complex” and the final as “extremely
complex.” Workload information was provided by the
facility in the finai reports. However, final reports were
not required for all OEDs that were classified as
“minor.” Thus, only the final reports with complete
controller workloac. nformation were included in this
study.

The OED data base lists the types of “causal factors”
involved in each incident. The causal factors for
ARTCCs were categorized as: a) radar display factor -
inappropriate use of displayed data, misidentification,
eic.; b) communication factor - improper phraseology,
readback, etc.; c) coordination factor - between sec-
tors, facilities, etc.; d) data posting - computer entry
and processing flight progress strips; and e) position
relief briefings - deficiencies in giving or utilizing
briefings. An error could involve more than one em-
ployee and have multiple contributing causes; there-
fore, for this study, an error was considered to have

involved a given factor if it was identified as a contrib-
uting factor for one or more employees. Appendix A
provides the definitions of the causal factors from the
Final Operational Error/Deviation Report (FAA Form
7210-3).

Analyses

Means and standard deviations were determined for
the two workload measures for each year. For
comparisons across facilities, the “number of aircraft”
measure was separated into three categories; 1-6 aircraft,
7-11 aircraft, and 12+ aircraft. Percentages of OEDs
that occurred under three levels of complexity and
“number of aircraft worked” conditions were
determined for each ARTCC. Causal factors were
coded as “0” if the factor was not involved in the error
and “1” if the factor was listed as having been involved
one or more times in the report. Analyses were confined
to those OED records with at least one ATCS employee
causal factor and non missing data for (a) the number
of aircraft being worked and (b) traffic complexity.
Correlation coefficients were computed between the
workload measures and error factors. Multiple
regression analyses were performed to determine the
relationships of several average facility-level conditions
with the number of OEDs for the ARTCCs.

RESULTS

The average number of aircraft worked during an
OED declined slightly from 1985 to 1988: M=9.6,
SD=4.7 (1985); M=9.2, SD=4.5 (1986); M=8.6,
SD=4.1 (1987); and M=8.7, SD=4.2 (1988). This was
true despite the general increase in the overall number
of aircraft operations handled by the ARTCCs, from
approxima.ely 32.27 million in 1985 to 35.91 million
in 1988. Percentages of OEDs occurring under the five
complexity levels for each year are presented in Table
1.

While the percentages of occurrences for the differ-
ent complexity levels were similar for 1985 and 1986,
in the latter two years, there was a slight increase in the
percentage of errors that occurred under “easy” condi-
tions (9.8% in 1987 and 10.7% in 1988 versus 4.8%
and 4.6% in 1985 and 1986, respectively). Also, across
the four years, from 21% to 29% occurred under less



AN EXAMINATION OF THE WORKLOAD CONDITIONS

ASSOCIATED WITH OPERATIONAL ERRORS/DEVIATIONS
AT AIR ROUTE TRAFFIC CONTROL CENTERS

David J. Schroeder and Lendell G. Nye

INTRODUCTION

In addition to the analysis of aircraft accidents, several
other aspects of aviation activity are monitored and
evaluated by the Federal Aviation Administration (FAA)
to assess the overall safety of the National Airspace
System (NAS). This study involves the analysis of data
for one such measure, Operational Errors/Deviations
(OEDs). An operational error (OE) takes place when
an air traffic controller allows less than applicable
minimum separation criteria between aircraft (or an
aircraft and an obstruction). Standards for separation
minima are described in the Air Traffic Control (ATC)
Handbook (7110.65, and supplemental instructions).
In the context of the Operational Error/Deviation
(OED) reporting system, an operational deviation
occurs when the appropriate separation standards are
maintained, but an aircraft enters airspace assigned to
another contro!ler's position of operation or air traffic
control facility without prior approval.

Since 1985, the FAA has operated the OED report-
ing system to track the operational errors/deviations
reported across the nation’s ATC system. An auto-
matic OE detection system was implemented in do-
mestic Air Route Traffic Control Centers (ARTCCs)
in 1984. Under this system, an automatic alarm is
triggered whenever minimum separation standards are
violated by radar-tracked aircraft. When an OED
occurs, the associated facility is responsible for gather-
ing data and completing a report, in accordance with
FAA Order 7210.3 (Facility Operation and Adminis-
tration). Details concerning the air traffic control
situation and the involved controller{s) are gathered
and summarized on the reporting form. Data gathered
via this reporting prograr) are then coded and entered
into the OED computerized data base, under the
purview of the Office of Aviation Safety. The current
OEDS data base contains data from the time of its

implementation in 1985. This data base contains OED
data from Terminal facilities (radar approach control
and towers) and ARTCCs.

This study will focus on data gathered from
ARTCCs, also called En Route facilities; Air Traffic
Control Specialists (ATCSs) at those facilities prima-
rily handle aircraft traveling between the terminal
facilities across the nation. The study was designed to
examine relationships between workload (number of
aircraft and complexity of air traffic) and causal factors
associated with ARTCC OEDs. First, this study
examined the relationship between the two workload
measures for ail ARTCCs combined and determined
whether there were significant differences between
facilities in either the average number of aircraft being
worked or the complexity level reported at the time of
the OED occurrence. Using previous OED reporting
systems, the results of analyses of 1965 and 1966 data
(Kershner, 1968), data from Kinney, Spahn and Amato
(1977), and subsequent operational error data
Schroeder (1982), indicated a general trend for an
increased percentage of errors under “Light” and “Mod-
erate,” as compared to “Heavy” workloads. Stager and
Hameluck (1996) called for an examination of the
causes of operational errors in terms of the conditions
existeat for the air traffic controller during the opera-
tive time-frame of the incident. These conditions
included “fatigue, distraction, attitudes, excessive
workload, and procedural knowledge.” Second, this
study determined if certain categories of operational
errors occur more or less frequently under the different
workload conditions. Sperandio (1971) found that air
traffic controllers tended to adopt different methods of
operation in response to alterations in workload. Third,
the number of OEDs az a given ARTCC has been
found (Splawn, Edwards. & Chin, 1988) to be 2


Tasie 3
Aur Trarric CONTROLLER WORKLOAD AND SEVERITY OF OPERATIONAL ERRORS

SEVERITY
MINCR N MODERATE N
# of Aircraft Worked 8.35 737 8.50 315
FL s 290 8.10 487 8.25 281 Fit. Level F, joes 13-9; P<.GO1
FL > 290 8.83 250 1059 34 Severity F, jo4g=2-2; NS
Traffic Complexity Rating 3.02 737 3.09 315
FL s 290 3.03 487 3.01 281 Fit. Level F, ,g4,=-06; NS
FL > 290 3.06 250 3.29 34 Severity F, .°,,=-93; NS
Note: FL = ffight level, 290 = 29,000 feet
Taste 4
Severity CLASSIFICATION WITHIN EACH Compiexity RATING
MODERATE MINOR
TRAFFIC COMPLEXITY SEVERITY % SEVERITY % N
Easy 27.2 72.8 103
Below Average 29.9 70.1 197
Average 28.5 71.5 376
Above Average 32.7 67.3 306
Complex 31.0 69.0 71
Overall 30.0 70.0 1053
Note: X?,,=1.89; NS. The N represents the number of OEs that occurred in each complexity

category.

not significantiy different between severity levels
(F, :o:72-2, NS), while the number of aizcraft being

1048
worked was significantly greater (F, ,.,.=13.9, p<.001)

at FL 290+. The average air traffic complexity rating
was not significantly different between the severity
categories (F, 447-93, NS) or flight levels CF, soag=-96,
NS). Similarly, a chi-square test suggested no significant
difference in air traffic complexity rating across OE
severity categories, (X*,=1.89, NS). As indicated in
Table 4, the percentages of errors that were moderate
(30.0% overall), compared to minor, were relatively
consistent, regardless of the traffic complexity.
Specifically, 27.2% of the OEs that occurred under the

“easy” complexity rating were classified as moderate in

15

severity, while 31.0% of the OEs that were evaluated
as eccurring under “complex” conditions resulted in a
moderately severe OE.

The error causa! factors that were differentially
related to OE severity are shown in Table 5. The
number of cases in each category in Table 5 equals the
total number of errors (both minor and moderate in
severity) that involved either a given causal factor or
ATCS awareness that the error was developing. The
percentages represent the proportion of the OEs that
were moderately severe for each causal factor. The
factors that were associated (determined by using
XX? =p<.01 as the criterion level) with a lower percent-
age (compared to 30% of all OEs) of moderate errors
included: a) the Misuse of Displayed Data - excluding
use of Conflict Alert, and b) awareness by the control-


Taste 3
PercentaGe oF OEDs Occurrinc Unper THree Comptexity CONDITIONS FOR EACH ARTCC

COMPLEXITY RATING

LESS THAN AVERAGE ABOVE
ARTCC AVERAGE COMPLEXITY AVERAGE N
ZAB - Albuquerque 27.3% 40.9% 31.8% 44
ZAN - Anchorage 15.4% 34.6% 50.0% 26
ZAU - Chicago 29.1% 45.6% 25.3% 158
ZBW- Boston 18.5% 53.3% 28.1% 135
ZDC - Washington DC 21.1% 29.7% 49.2% 185
ZDV - Denver 17.9% 29.5% 52.6% 78
ZFW - Fort Worth 16.9% 39.8% 43.4% 83
ZHU - Houston 23.2% 37.6% 34.1% 85
ZID - ianapolis 20.0% 49.0% 31.0% 155
ZX - Jacksonvilic 26.4% 44.8% 28.7% 87
ZKC - Kansas City 27.8% 30.6% 41.7% 108
ZLA - Los Angeles 29.4% 24.6% 46.0% 126
ZiC - Salt Lake City 27.0% 35.1% 37.8% 37
ZMA- Miami 23.8% 54.8% 21.4% 42
ZME - Memphis 22.4% 44.7% 32.9% 76
ZMP - Minneapolis 26.8% 49.3% 23.9% 71
ZNY - New York City 27.9% 36.6% 35.5% 172
ZOA - Oakland 25.4% 40.8% 33.8% 71
ZOB - Cleveland 31.1% 32.0% 36.9% 122
ZSE - Seattle 29.0% 29.0% 41.9% 31
ZSU - San Juan 36.4% 21.2% 42.4% 33
ZTL - Atlanta 25.4% 46.5% 28.1% 114
Combined 25.0% 39.1% 35.9% 2039

Note:X*(42)=98.42; p<.001

was not included, due to the small numb=r of OEDs
(less than 10) in the database. There were marked
differences across facilities in the percentages of OEDs
occurring under the three complexity levels. While
25% of the OEDs occurred under “less than average”
complexity categories, the percentages across facilities
ranged from 15.4% at the Anchorage ARTCC to
31.1% at Cleveland and 36.4% at San Juan. Similar
differences were noted for “average” complexity (rang-
ing from 21.2% at San Juan to 54.8% at Miami).
Percentages of OEDs under “above average” complex-
ity ranged from 21.4% at Miami to 52.6% at Denver.
Of the 22 facilities, four (ZAN, ZDC, ZDV, and ZLA)
had more than 45% of their OEDs occur under “above

average” complexity conditions. In contrast, three
(ZAU, ZMA, and ZMP) had 26% or less of their
OEDs occur under “above average” complexity condi-
tions. The extent to which these differences reflect
actual differences in complexity of the sectors at the
various ARTCCs, or to differences in how the com-
plexity ratings are assigned, is not determinable from
the available data. Certainly within each facility, it is
reasonable to expect that some sectors present more
complex and challenging air traffic control situations
than others. Also, a sector considered “complex” at one
facility might be rated as “average” at another ARTCC.
Thus, without additional information concerning the
percentage of time ATCSs at the various facilities


Ficure 1
MULTIDIMENSIONAL SCALING OF EN Route FAciLitiEs BASED ON WorKLOAD
AND SITUATION AWARENESS CHARACTERISTICS AT THE TIME OF THE OES

Low Awareness High Workload
LC
Low Workload High Awareness
Note:
Quadrant! = high workload - high awareness
Quadrant I= high workload - low awareness
Quadrant #1 = low workload - iow awareness
Quadrant !V = low workload - high awareness
Workload = standardized levels for number of aircraft being worked plus air

traffic complexity at the time of the OEs
Awareness = ATCS awareness that the OE was developing

21


DOT/EAA/AM-93/22

Office of Aviation Medicine
Washington, D.C. 20591

AD-A275 98

ih
t
i

TE

94-05960
Ct

An Examination of the
Operational Error Database
for Air Route Traffic

Control Centers

Mark D. Rodgers, Editor

Civi! Aeromedical Institute
Federal Aviation Administration
Oklahoma City, Oklahoma 73125

This document is available to the public
through the National Technical Information
Service, Springfield, Virginia 22161.

U.S. Department
of Transportation

Federal Aviation
Administration

Gas

NL

a
‘4
Fe


Taste 6
Causat Factor Tyres AND Numser ct Aircrart Workep -
Catecories STANDARDIZED Wititn ARTCC Facturties

CAUSAL FACTOR # OF CEDS
# DF AIRCRAFT DATA RELIEF RADAR COMMUNI- COORDIN- IN WORKLOAD
(Z VALUES) POSTING BRIEFING DISPLAY CATION ATION LEVEL
s-1.0 15.9% 24% 49.3% 31.0% 42.1%
n=46 n=7 n=143 n=90 n=122 290
-.99-+.99 18.6% 4.0% 58.3% 239.7% 27.5%
n=268 n=57 n=839 n=427 n=395 1438
1.0-1.99 30.3% 6.1% 56.7% 28.6%
n=70 n=14 n=131 n=66 231
20and> 37.1% 10.1% 56.2% 29.2% 37.1%
n=33 n=y n=50 n=26 n=33 89
Overall 20.4% 4.2% 56.8% 29.7% 29.6%
n=417 n=87 n=1163 n=609 n=608 2048
Chi-square 35.7 12.1 8.6 0.4 29.3
(d&3) p<.001 p<.01 p<.05 p<.001

Note: Results of separate chi-square tests of OEDs that involved a given causa! factor versus
those that did not involve that factor (i.e. al! others). Row percentage totals exceed 190%
because an OED can involve multipie causal factors. n=number of OFDs that occurred under
each combination of workload level and causal factor type. %=percentage of OEDs within a
workload level involving that causal factor.

greater. Even though the relationships between workload conditions. The Radar Display factor was
margi 0 occur I
condit voived in from 49% zo 58% of
the OEDs, Coordinaticn tended to be involved more
frequently at both low and high workload conditions.

compared to moderate or average ccnditions. Thus.

workload measures and the causal factors were low, the nally less likely the lowest workload

results were consistent with expectations. The percent- ms, and wa:
ages shown in Table 6 represenz, within a given
workload condition, the number of OEDs in which a

causal factor was involved, divided by the total number

of OEDs for that workload condition. I: was possible
for multiple causal factors to be listed fora single OED
and thus the number of causa! factors exceed the
number of OEDs. Overall, 20.4% of the OEDs in-
volved data posting. When a relatively large number of
aircraft were being handled (z score of 2.0 or more},
data posting causa! factor was reported in 37.1% of
those OEDs, compared to 15.9% of the OEDs ia the
lowest workload condition. The Relief Briefing factor
was also more likely to be involved at

while causal factors were not highly influenced by
either of the ewo workload conditions (“compiexity”
and “number of aircraft"), there was some evidence
that

her percentages of certain error factors oc-

curred at the extremes of the workload distribution.

rather than at average to moderate workload cendi-
ticns.

Table 7 contain
which the criterion - average num
ARTCC facilities from 1985-88 (EA

ts of 2 regression analysis in
of GEDs &

> was evali-



Taste 5
INTERCORRELATIONS BeTweeN Worktoap Measures AND Causal Factors

Z#WRKG #WRKD CMPLXTY DATAP RADARD COMM COORD
Z#WRKD -
#WRKD 32"* _
CMPLXTY -65** -61** —
DATAP .11** -16** .06* _
RADARD .02 -.01 .06* -.13** _—
COMM -.01 -.01 +01 -.10** -.40** _
COORD -.05 -.04 -.05* -16** -.23** -.15** —
RBRIEF -05 .09** -03 .15** -.06* -.06* .08**
Note:
n= 2042, 2-tailed significance levels
= p<.01
“te p<.001
Z#WRKD = number of aircraft being worked-standardized within each facility
#WRKD = _—_ actual number of aircraft being worked
CMPLXTY = air traffic complexity rating
DATAP = ___ data posting was a causal factor
RADARD = radar display
COMM = — communication
COORD = coordination
RBRIEF = __ relief briefing

iy, rather than on all facilities combined. The five
causal factors were: data posting (DATAP); radar dis-
play (RADARD); communications (COMM); coordi-
nation (COORD); and relief briefing (RBRIEF). As
already indicated, complexity and “number of air-
craft” are highly correlated (7=.61). “Number of air-
craft” is most closely related to the data posting causal
factor (r=.16, p<.001). A statistically significant posi-
tive correlation was also noted berween “number of
aircraft” and the relief briefing factor (7=.09, p<.J01).
Thus, OEDs involving a greater number of aircraft
were somewhat more likely to involve a problem asso-
ciated with dara posting. They were also slightly more
likely to involve problems associated with the relief
briefing. Complexity also appears ta have very small
statistical relationships with data posting (7.06, p<.G1)
and use of the radar display (r=.06, p<.01). A small! but
statistically significant negative correlation was noted

between complexity and the coordination factor
(r=-.05, p<.01). Thus, an error involving coordination
was somewhat more likely to occur under lower, rather
than higher, complexity conditions. While statistically
significant, the practical significance of these results is
limited, due to the low magnitude of the correlation
coefficients. Significant patterns of intercorrelations
were found among types of causal factors within the
OED incidents. In particular, when the radar display
was involved, communication and coordination fac-
tors tended not to be involved (-.40, p<.001 and rs-
.23, p<.001 respectively).

To provide a better understanding of the relation-
ship between the “number of aircraft” workload mea-
sure and the various causal factors, the workload
measure for “number of aircraft” was categorized on
the basis of the four z score ranges: low through -1.0;

-.9© through .99; 1.0 through 1.99; and 2.0 and


Taste 1
PercentaGes oF GEDs Occurrinc Unper faci oF Five Comptexity Conpittons, BY YEAR

YEAR EASY

1985 46 18.3
1986 4.6 16.7
1987 9.8 15.6
1988 10.7 18.6
Overall 74 17.3

Note:X?(12)}=30.83; p<.01

AVERAGE COMPLEX
39.7 28.6 8.6
40.1 29.9 8.6
37.8 28.2 8.5
38.0 27.3 5.5
38.9 28.5 7.9

Taste 2
Numaer of Aircrart Worked Durinc OEDs ror Trarric Comptexity CATEGORIES
N OF MEAN
TRAFFIC COMPLEXITY OEDS # AIRCRAFT sD MINIMUM MAXIMUM
Easy 155 4.01 2.0 1 13
Below Average 356 6.03 2.3 1 17
Average 800 8.66 3.2 1 25
Above Average 576 11.41 4.1 3 35
Complex 161 14.30 5.6 5 40
Totals 2048 9.07 45 1 40

than average complexity conditions. Furthermore,
nearly two-thirds of the errors occurred under average
or lower complexity. The interrelationship between
the two measures is demonstrated in Table 2. Even
though there is a gradual increase in the average num-
ber of aircraft worked under the five complexity levels,
from 4.01 under “easy” conditions to 14.30 under
“complex” conditions, the range within each of the
categories is considerable. “Easy” conditions involved
from 1 to 13 aircraft and “complex” conditions in-
volved from 5 to 40 aircraft. (Note: involvement of 2
single aircraft occurred in only 11 of the incidents, a
majority of which were operational deviations). Dif
ferences across the five categories and the general
overiap would seem to raise some questions concern-

ing the criteria used to detesmine air traffic complexity
and whether those criteria are consistently applied
across facilities. Under what circumstances would a
situation involving 3 aircraft be of “above average”
complexity for a particular sector? Likewise, how cana
situation involving 25 aircraft be considered to be of
only “average” complexity? Additional information
and analyses are required to determine the factors most
closely related to application of the complexity ratings
and whether those ratings should be modified.
Percentages of CEDs occurring under “less than
average” (“easy” and “below average complexity”),
“average,” and “above average” complexity (“above
average” and “complex”) for each of 22 ARTCCs are
presented in Table 3. The Honolulu (ZHN) ARTCC


Tape &
Averace ATCS Worxtoap Duginc Mooeratety Severe OEs at EN Route Facttimies

# Aircraft

Facility N Worked
ZAB 9 10.00
ZAU 31 7.68
ZBW 19 5.79
ZDC 35 9.09
ZDV 1 9.27
ZFW 16 9.31
ZHU 13 8.92
ZID 23 9.30
ZJX 20 8.20
ZKC 9 10.00
ZLA 15 7.00
ZLC 4 11.25
ZMA 11 8.64
ZME 10 10.60
ZMP 5 8.60
ZNY 26 6.88
ZOA 11 7.45
ZOB 19 9.26
ZSE 6 7.33
ZTL 9 8.33
Overall

Mean 8.65
SD 1.35

Traffic % ATCS
Complexity Awareness
3.33 (2) 22
2.77 (1) 0
2.74 (1) WW
3.71 (2) 23
3.73 (2) 36
3.31 (2) 6
3.31 (2) 15
3.13 (2) 4
2.50 (1) 20
3.78 (2) 33
2.60 1) 20
3.25 (2) 75
3.09 (2) 18
3.30 (2) 10
3.20 (2) 0
2.77 @) 31
2.82 (1) 36
3.00 Q) 21
2.67 qa) 17
3.78 (2) 11
3.14 21

40

Note: The number in () represe: ‘+ a classification of each facility as either relatively high
ATCS workload (2) or low workload (1) reported at the time of the moderately severe OEs.
Awareness= the percentage of each facility’s moderately severe OEs in which the ATCS was

aware that the error was developing.

Facility-level characteristics of moderately
severe OEs

Tables 8 through 10 illustrate the results of analyses by
facility for the moderately severe errors at ARTCC
facilities located in the continencal U.S., i.e. excluding
ZSU, ZHN, and ZAN. Only moderately severe errors
{excluding minor OEs) are reported here because of
their greater salience for aviation safety and our finding
of nonsignificant associations between workload and
OE severity. Table 8 lists the avezage number of
aircraft being worked and air traffic complexity rating.
The results of a series of cluster analyses produced two
interpretable groups of facilities based on reported
ATCS workload. Each of the relatively low workload
facilities (ZAU, ZBW, ZJX, ZLA, ZNY, ZOA, and

18

ZSE) was characterized by an average air traffic
complexity rating of less than 3.0 (“average”) combined
with 8.2 or fewer aircraft being worked. It should be
noted that some degree of variability could be expected,
given potential differences in reporting standards and
practices, not only between facilities but within a
facility as well. Table 8 also illustrates facility differences
in the percentages of moderately severe errors in which
the ATCS was aware that the error was developing.
Specifically, ZAU and ZMP reported that no ATCS
was aware of the developing error in any of their OEs.
By contrast, several facilities (ZDV, ZKC. ZLC, and
ZOA) reporied tha: the ATCS was cognizant of the
situation prior to loss of separation in one-third or
mote of their moderately severe OEs.


the various complexity cr workload conditions it is
difficulr to determine the primary factors associated
with these outcomes.

Facility Level Findings

Although neither the number of aircraft being worked
nor a rating of air traffic complexity was found to be
related to OE severity

a series of cluster analyses

produced two interpretable groups of facilities based
on reported ATCS workload. This measure, which for
low ATCS workload was characterized by an average
air traffic complexity rating of less than 3.0 (“average”)
combined with 8.2 or fewer aircraft being worked, was
used to examine ATCS workload in conjunction with
situation awareness.

Several hypotheses of the effect of workload on
situarion awareness are offered. The firs: suggests that
awareness decreases as workload increases. This might
be due to difficulty in maintaiai
f

g an accurate mental
picture of the air traffic situation as its complexity and/
or information load increases. This phenomena is
more likely to operate under very high workload con-
ditions. Another possibility suggests that, as workload
shifts from a high to low level, awareness decreases.
This situation is likely to exist after a busy traffic
“push”. Additionally, awareness may decrease if low
workload conditions are sustained for an extended
petiod of time. This is more likely to occur during very
low workload conditions. Furthermore, a fatigue ef-
fect could be operating under sustained periods of high
workload. resulting in a decrease in awareness during
later periods of a high workload watch.

It appears that reduced situation awareness might
occur 2s an effect of very different workload related
causes. In this research, several clusters of facilities
distinguished based on differences along the
dimensions of ATCS workload and situation aware-
ness. Facilities were classified as either high workload/

wer

high awareness. high workload/low awareness. low
workload/high awareness, or low workload/low aware-
ness. The data are not available to determine which of
the four hypotheses mentioned above might have been
eperating during each of the errors analyzed in this
study.

tw
Be

Two fac

s best exemplified the relatively low
workicad/low awareness combination (ZAU,ZBW).
They accounted for 17% of ail the moderately severe
errors. However, it was not possible to determine the
extent to which these errors were a result of traffic
density, airspace configuration, local operating proce-
dures, workstation design, poor technique, or any of a
number of other factors. In fact, since normative
information is not available on variables such as traffic
density, one must settle for a description of the factors
associated with operational irregularities.

“Major” OE Severity Findings

Fortunately, over the 31/2 years of data analyzed and
presented in this report, only 15 of the 1053 OEs were
rated “majoz” in severity. Causal factor frequencies for
errors proportionately matched the
frequencies of the “minor” and “moderate” causal
factor categories. Additionally. the average number of
aircraft being worked did nox differ between severity

the “majo

categories. Furthermore, the profile in which the
majority of “minor” and “moderate” errors occurred
(level & descending) was also the profile of the majority
of the “major” errors. Although 40% of the “major”
exrors were rated “above average” in complexity, the
mean complexity for the “major” errors (2.87) was less
than tht for the “minor” and “moderate” errors (3.05).
However, given the small sampie of “major” errors this
findiag was not statistically significant.

CONCLUSION

The types of analyses presented in this report allow for
the identification of factors more likely to precipitate
air traffic control operational errors. However, as
mentioned earlier, due to the lack of normative data,
one must settle fora description of the factors associated
the
of
the investigators, may affect the extent to which these
relationships can be determined. For the investigative
process to allow fora more definitive determination of
the factors involved in operational irregularities,

with these operational irregularities. Additionall

reporting process, including the reporting reliabil

investigators musi be able to review the dynamics
associated with the air traffic situation.



ACKNOWLEDGMENT

The authors gratefully acknowledge the assistance of
Gena K. Drechsler

for her effort in the preparation of this manuscript.

Accession For

BTIS RAs

pric TAB Q
Unannounced a
Jestifieation —___—_—-}

By

Distribution/

_ Availability
i

iii



FACTORS ASSOCIATED WITH THE SE*/ERITY OF OPERATIONAL
ERRORS AT AIR ROUTE TRAFFIC CONTROL C: NTERS

Mark D. Rodgers and Lendell G. Nye

INTRODUCTION

In the history of the Federal Aviation Administration,
no aircraft have collided while under positive control
in en route airspace. However, aircraft have violated
prescribed separation minima and approached in close
proximity. This event can occur as a result of either a
pilot deviation, or an operational error (OE). In this
study, analyses were conducted of data gathered
concerning OEs. An operational error cakes place
when an air traffic controfler allows less than applicable
minimum separation criteria between aircraft (or an
aircraft and an obstruction). Standards for separation
minima are described in the Air Traffic Control (ATC)
Handbook (7110.65F, and supplemental instructions).
While there is considerable complexity in those
standards, ar flight levels between 29,000 and 45,000
feet, Air Traffic Control Specialists (ATCSs) at en
route facilities are required to maintain either 2,000
feet vertical separation or 5 miles horizontal separation
between aircraft. At flight levels below 25,000 feet
with aircraft under IFR conditions, ATCSs are required
to maintain either 1,000 fect vertical separation or 5
miles horizontal separation. This study focused on
data gathered from Air Route Traffic Contro! Centers
(ARTCC), also called en route facilities.

The first purpose of this study was to examine the
relationship between the level of severity (degree of
loss of vertical and horizontal separation between
aircraft) and air traffic controller workload, measured
by the number of aircraft being worked and complex-
ity level reported at the time of the occurrence. The
analyses of 1965 and 1966 data by Kershner (1968),
data from Kinney, Spahn, and Amato (1977), and
analyses of subsequent operational error data
(Schroeder, 1982, Stager and Hameluck, 1990, and
Schroeder and Nye, 1993) found a general trend for an
increased percentage of errors under “Light” and “Mod-
erate.” as Compared to “Heavy” workloads. Schroeder
and Nye (1991) also found two measures of workload

(number of aircraft being worked and situation com-
plexity) to be highly correlated with each other. Al-
though OEs tend to occur more often under light and
moderate workloads, ic was hypothesized that the
more severe OEs occur under heavy workload condi-
tions.

The second purpose was to determine which of the
“causal factors” attributed to the air traffic controlfer
tended to be associated with OE severity. The general
cazegories of causal factors include: Data Posting.
Radar Display, Coordination, Communication, and
Position Relief Briefing. Each of these factors is sub-
divided into more specific categorie , with 5-15 sub-
categories under each main factor. It was hypothesized
that the OEs involving the causal factor categories
more closely related to the ATCS’s situation awareness

y, that is,
more serious sepatation errors. Another variable (not
defined 2s 2 causal factor) reported on the OE form
addzesses whether cr not the ATCS was aware the error
was developing. This factor was included in the analy-
ses, since it relates directly to the hypothesis,

The third purpose of this study was to determine the
relationship between the severity of an OE and two of
the flight characteristics of the aircraft involved in the
errors: aircraft profile and altitude. It was hypoth-
esized that aircraft in a climbing or descending profile
are more likely to be involved in the more severe OEs,
due to the rapidly changing dynamics of the centro}
situation. [twas alse ypothesized that aircraft at lower
altitudes, where less separation is required, are more
likely to be involved in the more severe OEs,

The fourth purpose of this study wes to determine
how these variables are distributed among individual
ARTCCs. i: was hypothesized that clusters of facilities
could be determined in regard to certain factors related
r0 OE severity.

were more likely to result in greater severi



Taste 1
Derinttion oF Severity CATEGORIES

VERTICAL SEPARATION POINTS

HORIZONTAL SEPARATION POINTS

If occurrence was at or below
altitude of 29,000 ft. (FL 290):
Less than 500 feet...
500 feet to 600 feet
600 feet to 700 feet
700 feet to 800 feet
800 feet to 900 feet
900 feet to 1,000 feet .......eeceeeeeeeeeee

lf occurrence was above altitude
of 29,000 ft. (FL 290):

Less than 500 feet ..
500 feet to 700 feet
700 feet to 1,000 feet
1,000 feet to 1,500 fee
1,500 feet to 2,000 feet

NUaAwoe

Less than '/2 mile
‘2 mile to 1 mile
i mile to 1'/2 miles
1'/2 miles to 2 miles
2 miles to 2'/2 miles
2'/2 miles to 3 miles
3 miles to 3'/2 miles
31/2 miles to 4 miles
4 miles to 5 miles.

SEVERITY :
20 points = Major .
- 14-19 pcints = Moderate i

: 13 or less points = Minor i
! i

bo

The causal factors attributed to the ATCS com-
prise a hierarchy of specific elements within more
general categories, as shown in Table 2. For example,
if a computer entry had been incorrectly updated
(causal factor 1a.1) then that error also involved the
less specific category of Data Posting. An operational
error can involve multiple causal factors. For this
study, an errcr was coded as 1 if a factor was recorded,
and 0 ifit was not. Table 2 also includes the percentage
of the 1053 errors that involved each factor. It should
also be mentioned that sometimes the more global
categories were not recorded, while at other times,
only the global categories were recorded. For example,
in this study, if computer entry error was reported as a
factor in an OE, that occurrence was also coded as
having involved data posting as a causal factor.

Aircraft Profile Characteristics

One of six possible combinations of aircraft flight
characteristics was identified for each error in the
sample. The profiles were as follows:

1) all aircraft were climbing,

2) all aircraft were descending,

3) all aircraft were at level flight,

4) one (or more) aircraft was descending and one
(or more) aircraft climbing,

5) one (or more) aircraft was at level flight and one
{or more) aircraft climbing,
or

6) one {or more) aircraft was at level flight and one
(or more) aircraft descending.

RESULTS

Factor Relationships to OE Severity

Since OE severity is differentially defined for aircraft
above or below 29,000 feet, the following results are
presented in terms of that distinction. As shown in
Table 3, che average number of aircraft being worked at
the time of the OEs ranged from 8.10 for minor errors at
flight levels of 29,000 feet or below (FL 290-) to 10.59
for moderate errors at flight levels above 29,000 feet
(FL 290+). The number of aircraft being worked was

13



Tecknical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Cataiog No.
DOT/FAA/AM- 93/22

4. Title and Subtitie 5. Report Date

AN ANALYSIS OF THE OPERATIONAL ERROR DATABASE FOR December 1993
AIR ROUTE TRAFFIC CONTROL CENTERS

7. Authors) &. Performing Organization Report No.

Mark D. Rodgers, Ph.D., Editor

9. Performing Organization Name and Address TO. Work Unit No. (TRAIS)
FAA Civil Aeromedical Institute
P.O. Box 25082

Oklahoma City, OK 73125 7. Contract or Grant No.

TZ. Speasoring Agency name and Address 13. Type of Report and Period Covered
Office of Aviation Medicine
Federal Aviation Administration
800 Independence Ave., S.W.

Washington, D.C. 20591 14. Sponsoring Agency Code

15. Supplemental Notes
This work was performed under task A-C-92-HRR-115.

16. Abstract

Monitoring the frequency and determining the causes of operational errors - defined as the loss
of prescribed separation between aircraft - is one approach to assessing the operational safety
of the air traffic control system. The Federal Aviation Administration (FAA) refers to the
loss of separation standards between aircraft as an operational error /[OB). The extent to
which separation is lost determines the severity of the error.

The first study examined the relationships between error occurrence, controller workload
(number of aircraft and traffic complexity) and causal factors involved. The FAA's Final
Operational Error/Deviation Reports for ARTCC facilities during calendar years 1985-88
comprised the data base. A majority of the errors occurred under conditions of below average
(25%) or average (39%) complexity. Complexity and number of aircraft were highly correlated.
However, there was a significant difference across facilities in average workload during an
event. Improved guidelines for quality assurance personnel are needed to insure a more
standardized determination and reporting of workload dimensions. Results suggested that the
frequency of some of the causal factors varied in response to changes in number of aircraft
worked and traffic complexity.

The second study analyzed the workload and causal factors related to the severity of OEs at
ARTCCs during 1988-91. Neither the number of aircraft being worked nor air traffic complexity
were significantly associated with severity. In general, the causal factors that resulted in
greater severity likely involved reduced situation awareness by the controller. The
relationship of aircraft profiles and flight levels with OE severity were examined. Facility
level differences were reviewed regarding controller workload and awareness of the developing
error. More in depth information is needed to determine precisely the manner in which
alterations in workload influence the nature of the error process. Both studies point to the
need for increasing the level of objectivity in the operational error investigation process.

17. Key Words: 18. Distribution Statement
Air Traffic Control, Operational Decument is available to the public
Errors, Human Performance, Human through the National Technical
Factors, Workload Information Service, Springfield, VA
22161
19. Security Classif. (of this report) 20. Sacurity Classif. (of this page: 21. No. of Pages 22. Price
Unclassified Unclassified 30

Form DOT F 1700.7 (8-72) "eproduction of completed page authorized



Taste 14
Frequency of Causa: Factor Catecoris tHa? Resurrep in “Major” Errors at En Routt Facnomes

1. Data posting 0... ceeceseseeceecesesessaeenetere 5 c. Misunderstanding
a. Computer entry ... ad d. Readback....
(1) Incorrect input . 2 (1) Altitude
(2) incorrect update 1 (2} Clearance
(3) Premature termination of data......1 (3) identification

(4) Other
b. Flight progress stri
{1} Not prepared
(2) Not updated .
(3) Posted incorrectly
(4) Reposted incorrect!
(5) Updated incorrectly
(6) Sequenced incorrectly .
(7) Resequenced incorrectly
{8} Interpreted incorrectly .
(9) Premature removal... a airspace of another position of
(AQ Otne reece cceeseeeeeceeeeeeee renee operation or facility without prior
approval!
¢. Coordination was effected and
controtlerts} did not utilize

e. Acknowledgment
f. Other

2

4

ie]

3 4. Coordination .
0 a. Area of occurrence
ie) {1} Inter-positio:
2 (2) Intra-position
0 (3) Inter-sector
QO (4) Inter-facili
3 b. An aircraft penetrated designated
e

2. Radar display ...
a. Misidentificaiion .
(1) Overlapping data blocks...

information exchanged 5
(2) Acceptance of incomplete or {1) Aircraft identification 1
difficult to correlate position info. 1 (2) Altitudes/Flight Levei 1

(3) {mproper use of (3) Route of flig
identifying turn ...... {4} Clearance limi
(4) Failure to reidentify aircraft when 45) Speeds ....

accepted target ide becomes (6) APREQS

questionable - (7) Special instructions 3
{5} Failure to confirm aircraft identity (8) Other 2
after accepting a radar handoff ....0
{6) Other 2 3. Position relief briefing deficiencies noted 1
b. Inappropriate use of displayed data ....7 a. Employee did net use position relief
(1) Conflict alert 2 checklist ..
(2) Quick look te) b. Employee being
(3) Mode C... 1 incomplete briefi serene T
(4) MSAW/EMSAW 0 c. Relieving employee | did rot make use
(5) Other oo. eeseeeseeeeeeeenesceneneee 7 Of pertinent data exchanged at
briefing .
3. Communications error .. 6 d. Other
a. Phraseology 3
B. Transposition... escent t

as


NOTICE

This document is disseminated under the sponsorship of
the U.S. Department of Transportation in the interest of
information exchange. The United States Government
assumes no liability for the contents or use thereof.



Currently, Quality Assurance (QA) specialists re-
view multiple printouts of data that detail aircraft
locations and ATCS/Host Computer System interac-
tions, as well as multiple audic recordings of pilot/
controller and interphone communications to deter-
mine events surrounding the occurrence of an opera-
tional irregularity. The task of piecing together the
“big picture” from multiple sources of data isa difficult
and time-consuming part of the QA specialist’s job.
The reliability of the findings of the QA investigation
affect the reliability of the report filed for each irregu-
larity. To allow the QA team to spend more time
identifying the sources of problems associated with an
incident and less time piecing together the dynamics of
the air traffic control situation, a method is required
that allows fora re-creation of incidents and integrates
the various data sources into a coherent picture. Such
a system has been developed that would allow for the
collection of normative data and assist in determining
more definitively the location and likelihood of break-
downs in the human-machine system (Rodgers &
Duke, 1993).

It may be possible to quantify characteristics of the
air traffic control situation such as workload, complex-
ity, severity, as well as others as a by-product of the re-
creation of incidents using the system mentioned above.
This would reduce the variability that appears to exist
between investigators and facilities. Standardization
of the investigation process in this manner would allow
for an improved determination of the factors involved
in operational irregularities. Additionally, such a sys-
tem would provide the capability to collect normative
information and porentially allow for more than a
descriptive view of these occurrences.

REFERENCES

Federal Aviation Administration. (1989). Air Traff-
Control (FAA Order 7110.65F). Washington, DC:
U.S. Department of Transportation.

Federal Aviation Administration. (1991). Facility Opera-
tion and Administration (FAA Order 7210.3). Wash-
ington, DC: U.S. Department of Transportation.

Hart, S. G. & Straveland, L. (1988). Development of the
National Aeronautics & Space Administration
(NASA) task load index (TLX): Results of empiri-
cal and theoretical research. In P. A. Hancock & N.
Meshati (Eds.), Human Mental Workload pp. 139-
183). Amsterdam, North Holland.

Kershner. 4. M. (1968). .4ir traffic control system error
data for 1965 and 1966 as related to age, workload,
and time-on-shift of involved controller personnel.
Atlantic City, New Jersey: FAA, NAFEC Report
No. NA-68-32.

human element in air traffic control: Observations
and analyses of the performance of controllers and
supervisors in providing ATC separation services.
McLean, VA: METREK Division of the MITRE
Corp., MTR-7655.

Rodgers, M. D. & Duke, D. A. (1993). SATORI: Situa-
tion Assessment through the Re-creation of Incidents.
Washington, D.C: Federal Aviation Administr--
tion Report No. DOT/FAA/AM-93/12.

Schroeder, D. J. & Nye, L. G. (1993). An Examination
of the Workload Conditions Associated with Op-
erational Errors/Deviations at Air Route Traffic
Control Centers. In M. D. R. Agers, (Ed.) Ax
Examination of the Operational Errcs Database for
Air Route Traffic Control Centers. Washington, D.C:
Federal Aviation Administration Report No. DOT/
FAA/AM-93/22.

Schroeder, D. J. (1982, October). The loss of prescribed
separation bztween aircraft: How does it occur?
Proceedings (P-114), Behavioral Objectives in Avia-
tion Automated Systems Symposium, Society of Auto-
motive Engineers (SAE) (pp. 257-69). J. (1982,
October}.

Stager, P & Hameiuck, D. (1990). Ergonomics in air
traffic control. Ergonomics, Voi. 33, No. 4, pp.
493-9.


Tage 7
Muttiere REGRESSION FOR Operationat Errors/DEviations AT ARTCCs From 1985-88

VARIABLE MEAN SD VARIABLE LABEL

EALL 69.8 94.9 Avg. Annual ARTCC Error:
OPSAVG 1,632,680.29 442,995.13 Avg. Facility Operations
FPLRATIO 66 11 Ratio FPL to Total ATCSs
CMPLXAVG 3.12 16 Avg Complexity Reported at

Occurrence of OEDs

Dependent Variable = EAL. - Avg. Annual ARTCC Errors

Multiple & “914
R Square -836
Adjusted R Square -807
BETA
VARIABLE WEIGHT
OPSAVG 949
FPLRATIO ~.332
CMPLXAVG -301

F, ,,=28.92; p<.001

PARTIAL
CORR. F
-909 81.41
-.627 11.00
571 8.22

ated as a function of a) average annual facility opera-
tions (OPSAVG), b) the facility’s ratio of full perfor-
mance level (FPL) controllers co total ATCS workforce
(FPLRATIO). and c) the facility’s average traffic com-
plexity reported at the time of the occurrence of the
OEDs (CMPLXAVG). The data for the ratio of FPL/
total ATCSs were obtained from the 1987 calendar
year reporz by Splawn, Edwards, & Chin (1988),
which approximates the mid-point of the time-frame
of the current study.

The set of predictor variables accounted for ap-
proximately 84% of the variance in number of ARTCC
OEDs, with each measure contributing significantly
to the regression equation. The standardized beta
coefficients and partial correlations were greatest for
the cuantitative measure of facility workload (B=.95,
7.91) and similar in magnitude fos the FPL experi-
ence level (B=-.33, r=-.63) compared with average
wraffic complexity (B=.30, ~.57). As was found for
1987, the average number of OEDs during 1985-88
tended to he strongly and positively related to total
operations and inversely related to the ratio of FPL air

traffic controllers. The measure of the qualitative na-
ime of the OEDs
was shown to bea significant additicnal component in
explaining facility differences in the number of OEDs.
This last finding was the result of the inverse reiation-
ship (r=-.34) berween an ARTCC’s average operations
and average complexity (i.e., ARTCCs with a greater
number of annual ATC operations tended to report

ture of a facility’s workload at the

that their OEDs occurred under less complex traffic
conditions). The determination of whether or not
comparable facility differences exist in air traffic com-
plexity during non-error conditions requires further
evaluation.

DISCUSSION

This study found that both the average number of
aircraft being worked and che level of the air traffic
complexity during OEDs declined slightly during 1985-
88. As expected, there wasa strong r

onship between
the rwo workload measures reported in the OED
database. Differences in the average workload


The fifth purpose of the study was to determine
what factors operated during a “major” OE. Since
there were only 15 major OEs in the database during
the time-frame of this study, only frequencies of the
factor categories occurring during a major error were
reviewed.

METHOD

Operational Error Data Base

Since 1985 the FAA has operated an Operational Error
Data Base to track the operational errors reported
across the nation’s ATC system. Quality Assurance
personnel at each facility are responsible for gathering
data and completing a report in accordance with FAA
Order 7210.3 (Facility Operation and Administration).
Data are coded and entered into the operational error
computerized data base, which is under the purview of
the Office of Aviation Safety. Since 1988, the data base
has includea a more detailed encoding of the “causa!
factor” information. Therefore, for the purposes of
this report, the sample includes the error reports for
the years 1988, 1989, 1990, and approximately two-
thirds of the OEs from 1991, which included
information on ATCS causal factors and workload,
along with OE severity.

Error Severity

The severity of an OE is categorized and reported by
the FAA according to the closest proximity of aircraft,
in terms of both horizontal and vertical distances.
Table 1 shows the separation parameters and
corresponding point assignments, which are added to
determine whether the severity was “major,”
“moderate,” or “minor.” This calculation is made by a
sub-routine of the OE data base. Of the 1053 errors in
our sample, only 15 were coded as “major” errors,
defined as being less than 0.5 miles horizontal separation
AND less than 500 feet vertical separation. Meeting
this criterion is the only way that an error can be rated
“major,” whereas there are many ways that an error can
be rated “minor” or “moderate.” Some of the key
points regarding the separation standards can be
summarized as follows:

12

a) a “major” error is defined as less than 0.5 miles
horizontal separation AND less than 500 feet
vertical separation between aircraft at the time of
occurrence;

b) the designation of “moderate” or “minor” error
is based on the determination of the altitude of
occurrence:

1) at 29,000 ft (FL 290) or below, a moderate
error can involve a range of separation param-
eters from less than 0.5 miles horizontal com-
bined with up to 900 feet vertical at one
extreme, compared to almost 3.0 miles hori-
zontal combined with less than 500 feet verti-
cal. A minor error can involve less than 0.5
miles horizontal separation only if vertical
separation is 900 feet or greater. Also, an error
can be classified as minor with less than 500
feet vertical separation only if the aircraft were
3.0 miles or farther apart.

2) above FL 290, a moderate error can involve
less than 0.5 miles horizontal combined with
up to 1000 feet vertical (compared to 900 ft.
at FL 290 or lower). At this flight level, while
severity points are added for vertical separa-
tion up to 2,000 feet, all errors occurring with
vertical separation of 1,000 ft. or greater are
classified as minor.

Of the 15 “major” errors, 7 resulted in the filing of

a near mid-air collision report by at least one of the
involved pilots. Since, for the above reasons, the FAA
has categorized these errors as qualitatively different
from the other errors, and because there were so few of
them, they were analyzed separately, with only fre-
quency data presented in this report.

Air Traffic Controller Workload and Causal
Factors

The two measures of workload reported in the OE data
base are the number of aircraft being worked by the air
traffic controiler at the time of the error and a rating of
the air traffic complexity, which is an estimation by the
quality assurance specialist of the difficulty of the job
tasks, based on factors such as weather, airspace
restrictions, and variery of duties. The complexity
rating is made on a 5-point scale with anchors 1 =
“easy,” 3 = “average,” and 5 = “complex.”