DOT/FAA/AM 97/13 Distribution of Attention, Situation

Awareness, and \X/orkload in a Passive

Office of Aviation Medicine

Washington, DC 20591 AirTraffic Control Task: Implications

for Operational Errors and Automation

Mica R. Endsley

Texas Tech University

Lubbock, Texas 79409

Mark D. Rodgers

Civil Aeromedical Institute

Federal Aviation Administration

Oklahoma City, Oklahoma 73125

July 1997

Final Report OtocuBogJfejawMd

This document is available to the public

through the National Technical Information

Service, Springfield, Virginia 22161.

O

U.S. Department

of Transportation

Federal Aviation

Administration

19970912 018

DTIC QUALITY INSPECTED 5
NOTICE

This document is disseminated under the sponsorship of

the U.S. Department of Transportation in the interest of

information exchange. The United States Government

assumes no liability for the contents or use thereof.
Technical Report Documentation Page

1. Report No.

DOT/FAA/AM-97/13

2. Government Accession No. 3. Recipient's Catalog No.

4. Title and Subtitle

Distribution ofAttention, Situation Awareness, and Workload in a

Passive Air Traffic Control Task: Implications for Operational Errors

and Automation

5. Report Date

July 1997

6. Performing Organization Code

7. Author(s)

Endsley M.R., Department of

Industrial Engineering, Texas Tech

University, Lubbock, TX 79409

9. Performing Organization Name and Address

FAA Civil Aeromedical Institute

P.O. Box 25082

Oklahoma City, OK 73125

8. Performing Organization Report No.

Rodgers, M.D.*

FAA Civil Aeromedical Institute

*Now at:

Office of System Architecture

and Investment Analysis

FAA Headquarters, ASD-130

Washington, DC 20591

10. Work Unit No. (TRAIS)

11. Contract or Grant No.

12. Sponsoring Agency name and Address

Office ofAviation Medicine

Federal Aviation Administration

800 Independence Ave., S.W.

Washington, DC 20591

13. Type of Report and Period Covered

Final Report

14. Sponsoring Agency Code

15. Supplemental Notes

This work was performed under Contract DTFA-02-95P35476.

16. Abstract

A study was conducted to investigate factors underlying operational errors (OEs) in en route air traffic control.

Twenty active duty controllers watched re-creations of OEs and were asked to report on their situation

awareness and workload on two occasions during the re-creations. A total of 14 OEs were examined.

Responses were analyzed to determine how subjects allocated their attention while viewing the scenarios.

While observed patterns probably reflect necessary prioritization schemes, attention strategies identified in this

study can be linked to data on factors underlying OEs. Both objective taskload, as indicated by the number of

aircraft being controlled, and subjective workload were found to be related to controllers' ability to report

situation awareness information. Workload was found to be higher at the time of the OE than at the other

stop during the re-creation. During high workload, controllers appeared to reduce attention paid to certain

aircraft and variables to maintain awareness of more important information. Implications ofthis research are

drawn for potential problems in situation awareness under passive monitoring conditions that may be present

if certain forms of automation are introduced in the future air traffic control system.

7. Keywords

Attention, Situation Awareness, Workload

Operational Error, Air Traffic Control,

Human Error

18. Distribution Statement

Document is available to the public through the

National Technical Information Service

Springfield, Virginia 22161

19. Security Classif. (of this report)

Unclassified

20. Security Classif. (of this page)

Unclassified

21 No. of Pages

.25

22. Price

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized
ACKNOWLEDGMENTS

The authors thank the controllers and management at the Atlanta Air Route Traffic Control Center

for their participation and support ofthis study, in particular Al Guice for his coordination efforts. In

addition, our appreciation is extended to Dennis Rester, of the Civil Aeromedical Institute, for his

programming efforts, and to Cindy Fox, also ofthe Civil Aeromedical Institute, andAshok Sundaram

at Texas Tech University, for their assistance in data analysis.
DISTRIBUTION OF ATTENTION, SITUATION AWARENESS, AND WORKLOAD IN A

PASSIVE AIR TRAFFIC CONTROL TASK: IMPLICATIONS

FOR OPERATIONAL ERRORS AND AUTOMATION

INTRODUCTION

In the history ofthe Federal Aviation Administration (FAA), no aircraft have collided while under

positive control in en route airspace. However, aircraft have violated prescribed separation minima and

approached in close proximity. This event can occur

as a result ofeither a pilot deviation from clearances or

an operational error (OE) by an air traffic controller.

An OE takes place when the controller allows less than

applicable minimum separation between an aircraft,

another aircraft, or an obstruction. Standards for

separation minima are described in the Air Traffic

Control (ATC) Handbook (FAA Order 7110.65J)

and supplemental instructions. While there is considerable complexity in those standards, the criteria are

established as 2,000 feet of vertical separation or 5

miles of horizontal separation between aircraft operating at altitudes between 29,000 feet and 45,000 feet.

For aircraft operating under instrument flight rules

(IFR) at flightlevels below 29,000 feet, a minimum of

1,000 feet ofvertical separation, or 5 miles ofhorizontal separation are required. These separation standards provide tolerance zones ensuring that aircraft

pass well clear of one another.

A relatively small number ofOEs occur nationwide

each year. In 1993,430 OEs were recorded at en route

air traffic control centers in the US, with 37,170,000

aircraft handled. In an effort to ensure flight safety,

there is a desire to reduce the number of OEs that

occur. Doing so requires an understanding of why

these errors occur and the factors that are likely to

increase the probability of an operational error.

Rodgers andNye (1993) investigated causal factors

associated with (minor and moderate) OEs occurring

at en route ATC facilities over a three and one-half

year period, based on the FAA's Operational Error

Data Base. This data base records circumstances associated with OEs, as identified by quality assurance

(QA) investigators following the OEs' occurrence.

They found that 36% ofOEs involved problems with

communications (including 20% that were specifically readback errors), 15% involved coordination

problems, 3% involved deficiencies in position relief

briefings, 13% were associated with problems in data

posting, and 59% were related to the radar display

(including 14% that involved misidentification of

information and 47% that involved inappropriate use

of displayed data). Some errors are attributed to

multiple causal factors.

A number ofresearch studies have sought to investigate the relationship between estimates ofcontroller

workload and the incidence of OEs. Operational

errors have been found to occur under both high and

low workload conditions, with more errors occurring

under low and moderate levels of workload than

under high levels of workload (Kinney, Spahn, &

Amato, 1977; Schroeder, 1982; Stager & Hameluck,

1990). Itis unclear from these data, however, whether

this reflects a decreased tendency to make errors under

high workload conditions or a lower frequency ofhigh

workload conditions occurring overall.

A recent study by Schroeder and Nye (1993) found

a positive correlation between the number of aircraft

under the Air Traffic Control Specialist's (ATCS's)

control (normalized for the average number ofaircraft

per ATCS in that center) and the occurrence of OEs

involving data posting, position relief briefings, and

misuse of displayed radar data. They also found an

association between OEs involving coordination problems and both a lower than average number of aircraft

and a higher than average number of aircraft. There

was no association found between number of aircraft

and OEs involving communications problems.

It should be noted that these studies rely on estimates of controller workload that were made following the OE by FAA QA investigators who used a

simple workload scale (1 to 5, non-anchored) that is
not clearly defined. The investigators typically receive

little or no training on what factors to include in their

workload estimates. Thus, non-standardization, as

well as potential inaccuracy of estimates made after

the fact are limitations affecting the workload measures used in these studies.

There has also been an interest in determining to

what degree OEs involve a problem with controller

situation awareness (SA) — their mental picture ofthe

constantly changing air traffic situation. Formally

defined, SA is the "perception of the elements in the

environment within a volume oftime and space, the

comprehension oftheir meaning, and the projection

oftheir status in the near future" (Endsley, 1988). It

encompasses not only an awareness of specific key

elements in the situation (Level 1 SA), but also an

integration and comprehension of that information

in light of operational goals (Level 2 SA), along with

an ability to project future states ofthe system (Level

3 SA). These higher levels of SA (Levels 2 and 3) are

felt to be particularly critical for effective functioning

in complex environments, such as air traffic control.

In air traffic control, SA involves the continuous awareness ofthe location ofeach aircraft, along with pertinent

aircraft parameters (speed, heading, communications,

etc.) and their projected future locations relative to

each other, among many other pieces ofinformation,

to provide minimum separation and efficient aircraft

movement. A complete delineation ofSA information

requirements for en route ATC is provided in Endsley

and Rodgers (1994).

Endsley (1995a) developed a taxonomy of SA errors, describing causal factors associated with the

occurrence of SA errors (shown in Table 1). This

study found that 88% of major air carrier accidents

associated with pilot error involved a problem with

situation awareness. Of these, 72% involved problems with Level 1 SA , 22% involved problems with

Level 2 SA, and 6% involved problems with Level 3 SA.

Table 1 lists general causal factors in the SA error

taxonomy associated with a lack of situation awareness at each of its three levels. Problems with SA at

Level 1 (perception of the elements in the environment) can occur when needed information (1) is not

available through a failure of the system design or

communications process, (2) is available but is difficult to detect or perceive, (3) is not observed or

monitored, often due to distractions, attentional narrowing, high taskload, or failures in the scanning

process, (4) is misperceived, frequently due to erroneous expectations, or (5) is initially perceived, but is

forgotten due to failures in working memory, and

thus is not taken into account.

Table 1. SA Error Taxonomy (modified from Endsley, 1995a)

LEVEL 1: FAILURE TO CORRECTLY PERCEIVE INFORMATION

• Data not available

• Data difficult to detect or perceive

• Failure to monitor or observe data

• Misperception of data

•Memory failure

LEVEL 2: FAILURE TO COMPREHEND SITUATION

• Lack of or poor mental model

• Use of incorrect mental model

• Over-reliance on default values in mental model

• Other

LEVEL 3: FAILURE TO PROJECT SITUATION INTO THE FUTURE

• Lack of or poor mental model

• Overprojection of current trends

• Other
Problems with SA at Level 2 (comprehension ofthe

situation) can occur when information is correctly

perceived, but its significance or meaning is not comprehended. This may be because (1) a good mental

model for combining and processing perceived information is not available, (2) an incorrect mental model

is selected, leading the person to improperly interpret

perceived information, (3) there is an overreliance on

default values (general expectations about how parts

of the system will function) in the mental model, or

(4) other factors, such as limited working memory

with which to process information or lapses in usual

cognitive processes.

Problems with SA at Level 3 (projection of future

behavior of elements in the environment) may occur

ifa person understands what is going on in the current

situation but has trouble projecting what that means

for the future. This may occur because (1) a good

mental model that provides for predictions ofsystem

dynamics and behavior is not available, (2) there is a

tendency to project future behavior as a linear function of current systems dynamics when the dynamics

may change in non-linear ways, or (3) other lapses in

cognitive processing.

The SA error taxonomy was applied in a recent

study of 146 incidents involving reported problems in

SA among both pilots and controllers in NASA's

voluntary Aviation Safety Reporting System (ASRS)

(Jones & Endsley, 1996). (Only incidents involving

air traffic control are discussed here.) Of the 33

incidents involving air traffic controllers, 69% involved problems with Level 1 SA, 19% involved

problems with Level 2 SA, and 12% involved problems with Level 3 SA. Of the Level 1 SA errors, the

most common problem was a failure to monitor or

observe data (51.5%). This was most frequently due

to task distraction (53% of these cases), followed by

problems with high workload (17.6%), vigilance

(11.8%), and other miscellaneous causes (17.7%),

such as a failure to scan the runway, failure to notice

an aircraft overshoot, and failure to notice traffic on

the runway. Other causal factors were also related to

Level 1 SA errors: 18.2% involved cases where needed

data were not available, 18.2% involved cases where

controllers forgot important information (frequently

under high workload), 6.1% involved data that were

hard to discriminate or detect, and 6.1% involved the

misperception of information.

Level 2 SA errors were attributed to an incomplete

or inaccurate mental model (22.2%), the use of an

incorrect mental model (22.2%), over-reliance on

default values (22.2%), and other miscellaneous factors (33.3%). Level 3 SA errors were attributed to

over-projection of current trends (33.3%) and other

miscellaneous factors (66.7%). There were no obvious cases ofLevel 3 errors due to poor mental models.

It should be noted that the errors by air traffic

controllers examined in the Jones and Endsley study

involved voluntarily reported information from a

variety of ATC facilities including air route traffic

control centers (ARTCC), TRACONs, and towers

(both local and ground control). As such, while the

data do provide some information about the types of

errors that may occur across different types of ATC,

this cannot be viewed as a truly representative, or

completely unbiased, sample of controller errors. In

addition, it is often difficult to ascertain exactly why

some ofthe errors occurred from the limited information available in such reports.

Most information about errors is based on the

analysis of available historical reports. These reports

are often not developed with the objective ofexamining detailed causal factors, are usually based on afterthe-fact interviews that may be incomplete or biased,

and frequently suffer from problems ofinconsistency,

as different people usually conduct each investigation. The objective ofthe present study was to collect

more detailed data about OEs than what is available in

such accounts, providing for a better understanding

of factors that may contribute to their occurrence.

This study focused on data gathered on OEs from the

Atlanta ARTCC.

To gain more insight into the nature of OEs, the

Systematic Air Traffic Operations Research Initiative

(SATORI) system was developed (Rodgers & Duke,

1993). SATORI graphically recreates a visual display

of the radar data recorded during actual air traffic

control (based on computer tapes routinely recorded

at each air traffic control facility), synchronized with

the recorded audio tapes ofcommunications between

controllers and between controllers and pilots. Prior

to the development ofthe SATORI system, it was not
possible for the FAA QA team investigating errors to

review the control situation in a format like the one

presented to the controller when the OE occurred.

That is, the dynamics ofthe situation (the interaction

between control actions and displayed data) were

unavailable for review, not only by the QA team

investigating the irregularity, but also by the controller who committed the error. This limited not only

the extent to which reliable and accurate determinations of causal factors could be made for an error, but

also the extent to which the effects of the dynamic

situation on controller SA could be determined. The

SATORI system allows QA specialists and controllers

to view an accurate, dynamic representation of the

ATC data associated with an OE.

In the present study, the re-creation of OEs using

SATORI was combined with a modification of the

Situation Awareness Global Assessment Technique

(SAGAT) (Endsley, 1988). SAGAT is a technique

used during simulations in which the simulation is

frozen at random, unexpected intervals with all display screens blanked and the operator of the simulation is queried about the state ofthe current situation.

The operator's perceptions are then compared to the

actual state ofthe environmentto provide an objective

assessment ofthe operator's situation awareness. The

use of SAGAT to measure situation awareness in

aircraft simulations has been extensively validated

(Endsley, 1990a; 1990b).

In this study, SAGAT was modified to include

queries that pertain to major factors associated with

SA in en route ATC, based on an analysis by Endsley

and Rodgers (1994). As a second modification, the

technique was employed in conjunction with SATORI,

which involves the passive viewing of a situation, as

opposed to an interactive simulation in which the

subject is involved. While it is not clear how the SA of

a passive observer differs from that ofan active participant, this measure should still provide an indication

ofthe way in which controllers distribute their attention to various factors involved in the scenarios. As

thisstudy involves currently certified controllers viewing re-creations of real OEs, the combined use of

SATORI and SAGAT may provide unique insight

into factors affecting OEs in operational settings. Due

to their rare occurrence and the limited conditions

usually involved in simulations, observing OEs in

simulations can prove to be quite difficult.

In addition, it may be considered that SA under

passive viewing conditions may be analogous to actual

controller tasks if the ATC system ever becomes

highly automated. While many ofthe tasks a controller currently performs support the acquisition and

maintenance of SA, the task environment under conditions of high automation may change such that the

requirement to perform many routine tasks is eliminated. In this case, the controller would become a

monitor of the air traffic situation, which would be

controlled by an automated system. Since the scenarios recreated for subjects using SATORI similarly

involve the passive monitoring of a situation that is

actually controlled by another, this study may provide

some insights into SA with a hypothetical highly

automated system. This is a concern, as there is some

indication that SA may be compromised under highly

automated systems (Endsley & Kiris, 1995).

METHOD

Subjects

Twenty volunteersubjects participated in the study.

All were experienced, full performance level (FPL) air

traffic control specialists at Atlanta ARTCC. The 20

subjects included 4 subjects viewing OEs in each of 5

areas of specialization in the facility, as shown in

Figure 1. All subjects were certified in the area of

specialization for the re-created errors that they observed during the study. Subjects were relieved from

their duties on the air traffic control room floor to

participate in the study. Once subjects completed

their participation, they returned to their assigned

duties.

Procedure

Fifteen OEs that occurred in the Atlanta Air Route

Traffic Control Center (ARTCC) in 1993 and 1994

were recreated using SATORI. These errors were

selected from errors involving a single ATC sector,

based on the availability of complete and legible

computer data tapes and audio recordings. Three

errors in each of five areas of specialization of the

centerwere selected. (One error was eliminated during
S1

S2

Ü

CO

S3

S4

error 1

error 2

error 3

error 1

error 3

error 2

error 2

error 1

error 3

error 3

error 2

error 1

S5

error 4

error 5

error 6

S6

error 4

error 6

error 5

S7

error 5

error 4

error 6

S8

error 6

error 5

error 4

Area of Specialization

3

S9

S10

S11

S12

error 7

error 8

error 9

error 7

error 9

error 8

error 8

error 7

error 9

error 9

error 8

error 7

error 10

S13 error 11

error 12

error 10

S14 error 12

error 11

error 11

S15 error 10

error 12

error 12

S16 error 11

error 10

S17

error 13

error 14

error 15

S18

error 13

error 15

error 14

S19

error 14

error 13

error 15

S20

error 15

error 14

error 13

Figure 1. Experimental Design

testing due to problems with the data tapes, leaving 14

errors to be included in the analysis.)

Subjects were provided with a set of instructions

and signed a voluntary subject consent form. They

were then shown scenarios involving three errors from

sectors in the area ofspecialization on which they were

certified. Each scenario consisted of a re-creation of

the 10 minutes immediately prior to the occurrence of

the OE. Twice during each scenario, the re-creation

was halted and the screen blanked. The first freeze

occurred two minutes prior to the occurrence of the

error and the second freeze occurred at the time ofthe

OE in each scenario. Although subjects were informed that freezes would occur, they were not informed ofthe timing ofthe freezes or the occurrence

of the error.

During each freeze, subjects were provided with a

map ofthe sector. Sector boundaries, navigation aids,

airways and intersection markings were shown on the

map; however, no aircraft were included. Subjects

were asked to indicate the location of all known

aircraft on the map, and, for each aircraft, to indicate

or make a judgment of:

(1) if the aircraft were:

(a) in the displayed sector's control,

(b) other aircraft in the sector not under sector

control, or

(c) would be in the sector's control in the next

two minutes,

(2) aircraft call sign,

(3) aircraft altitude,

(4) aircraft groundspeed,

(5) aircraft heading,

(6) the next sector the aircraft would transition to,

(7) whether the aircraft was climbing, descending

or level,

(8) whether the aircraft was in a right turn, left turn

or straight,

(9) which pairs of aircraft had lost, or would lose

separation if they stayed on their current (assigned) courses,

(10) which aircraft would be leaving the sector in the

next two minutes,
(11) which aircraft had received clearances that had

not been completed and, for those, whether the

aircraft received its clearance correctly and

whether the aircraft was conforming to its clearance, and

(12) which aircraft were currently being impacted by

weather or would be impacted in the next five

minutes.

Of these, queries 1, 2, 3, 4, 5, 7, and 8 can be

regarded as pertaining to Level 1 SA, and queries 6, 9,

10, 11, and 12 can be regarded as pertaining to Levels

2 and 3 SA.

Following the completion of the questionnaire,

each subject completed a NASA-TLXsubjective workload rating (Hart & Staveland, 1988), indicating the

amount ofworkload they felt they would be under if

they were controlling the traffic in the scenario presented. Following completion ofthe NASA-TLX questionnaire, the scenario was resumed until the second

freeze. Then, the SA queries and NASA-TLX questionnaire were again presented in the same order,

following which the scenario was terminated and the

next scenario presented. After all three scenarios had

been presented, subjects completed a NASA-TLX

workload paired comparison ranking form, allowing

each subject's ratings on each NASA-TLX subdimension to be weighted based on the subjective

importance of the subdimension to each subject.

Apparatus

SATORI re-creations were presented on a DEC

3000-300 Alpha computer system using dual Sony

19-inch high-resolution (1280 x 1024) color monitors. NASA-TLX ratings were obtained using

HyperCard on a Macintosh Powerbook.

RESULTS AND DISCUSSION

SA Questionnaire

Subjects' responses to each question were scored

for accuracy based on computer data for each aircraft

at the time ofeach freeze. Subjects' indications ofeach

aircraft's location on the map were matched to the

closest aircraft actually present in the sector at the

time ofthe freeze and the distance error recorded. The

percentage of aircraft present that were reported by

the subject was calculated. Following that, scoring for

each subsequent question was calculated as the number of correct responses compared to the number of

aircraft that the subject reported knowing about (e.g.,

percent correct for altitude was calculated as the

number of correct aircraft altitudes reported, divided

by the total number of aircraft reported.) Subjects'

responses for each question were scored as either

correct or incorrect, based on operationally determined tolerance intervals (as listed in Table 2). Missing responses were scored as incorrect. It should be

noted that the selected sample of errors had a higher

number of aircraft (Mean = 12.9, S.D. = 5.6) than

typically occur at ZTL (Mean = 8.43, S.D. = 1.86);

however, given the focus of the study on controller

situation awareness in a future automated ATC system, it was deemed appropriate. Automation and

higher traffic levels have been associated with the

future operations of NAS. The study employed a

passive ATC task involving situation monitoring,

although this task is quite different than the current

ATC task, which involves active control, it generally

represents the cognitive task environment that might

be associated with excessive automation.

Means and standard deviations for subject response

accuracy are shown in Table 2. On average, 12.8

aircraft were present at the time of the freezes (range

4 to 23) across scenarios. Ofthese, subjects on average

reported 8.0 aircraft or 67.1% of the aircraft present.

Mean distance error was 9.6 miles (.68 inches) from

the aircrafts' reported location to their actual location. This may reflect aircraft movement occurring

during the visual scan (while the controller was viewing other aircraft) or may also be an artifact of the

passive viewing procedure used in this study.

For the aircraft reported, the correctness ofsubject

responses on the remaining questions was calculated.

Subjects correctly identified the control level of the

aircraft (in sector control, other aircraft in sector, will

be in sector control in the next 2 minutes) for 73.8%

of the aircraft reported. Aircraft callsigns were often

incomplete. The initial alphabetical part ofthe callsign

(indicating airline company, military, or civil aircraft

designation) was reported correctly 73.8% of the

time. The numerical part of the callsign (the aircraft

identification number) was reported correctly for only
Table 2. Awareness of Situation Across all Subjects and Scenarios

VARIABLE

Actual aircraft present (number)

Aircraft reported (%)

Distance error (miles)

Control level (% correct)

Call sign: alphabetic (% correct)

Call sign: numeric (% correct)

Altitude (+/- 300 feet) (% correct)

Change in altitude (% correct)

Speed (+/-10 knots) (% correct)

Heading (+/-15 degrees) (% correct)

Turn (% correct)

Separation problems (% correct)

Transition to next sector (% correct)

Assigned clearances complete (% correct)

Assigned clearance correct (% correct)

Assigned clearance conformance (% correct)

Weather impact (% correct)

EAN STD. DEV

12.9 5.6

67.1 18.0

9.6 4.5

73.8 17.3

79.9 23.4

38.4 32.0

59.7 22.1

66.4 25.6

28.0 25.6

48.4 30.6

35.1 40.2

86.2 32.3

63.5 45.1

23.2 22.9

74.4 43.9

82.9 37.9

60.7 49.1

38.4% of the aircraft. It should be noted that other

studies have found that, in general, 4% ofOEs involve

readback errors associated with aircraft identification

(Rodgers & Nye, 1993. The low level of accuracy in

recall knowledge ofaircraft callsigns is probably highly

indicative ofthese readback errors, as it indicates that

controllers may not attend to or retain much information on aircraft callsign in working memory, particularly the identification number.

Aircraft altitude was correctly reported (± 300 ft)

for 59.7% ofthe aircraft (mean error of 655 ft). The

aircraft were correctly identified as ascending, descending, or level 66.4% ofthe time. Correct groundspeed (±10 knots) was reported for only 28.0% ofthe

aircraft (mean error 21.8 knots). Correct aircraft heading (±15 degrees) was reported for 48.4% of the

aircraft (mean error 15.6 degrees). Only 35.1% ofthe

aircraft were correctly identified as being in a leftturn,

right turn, or proceeding straight ahead. These results

indicate that subjects were fairly poor at keeping up

with the dynamics of the aircraft in the scenario, at

least for many of the aircraft.

An argument can be made that perhaps subjects

simply did not retain this type ofdetailed information

about each aircraft (Level 1 SA) and instead, maintained awareness ofhigher level situation comprehension and projection issues (e.g., aircraft separation

and future projections of actions). Previous research,

however, indicates that people do maintain task relevant information about Level 1 SA elements that can

be recalled reliably under SAGAT testing when actively performing in a simulation (Endsley, 1990a).

There is also evidence that this measurement technique is reflective ofsubject attention allocation across

sources of information (Fracker, 1990. It is more

likely, therefore, that these measures do provide some

indication ofthe ways in which subjects in this study

were deploying their attention across displayed information, at least on a relative basis.

The subjects' higher level of understanding of the

scenarios was also evaluated. The aircraft pairs they

identified as having "lost or will lose separation ifthey

stay on their current (assigned) courses" was compared to those aircraft that actually had lost or would

lose separation (in the following two minutes) at the

time ofthe freeze. Subjects correctly identified 86.2%

of these aircraft pairs. (Aircraft pairs that the subject

identified as having potential separation problems,

but did not, were not scored.) Subjects correctly

identified 63.5% of aircraft that would be leaving the

sector in the next two minutes. Thus, they did not appear

to be fully aware of upcoming sector transitions.
Subjects correctly identified only 23.2% of aircraft

that had not yet completed control assignments. Of

those that they identified as not having completed an

assignment, subjects correctly determined in 74.4%

of the cases if the aircraft had correctly received its

assignment and correct in 82.9% of the cases if the

aircraft was conforming to its assignment. Overall,

subjects did not attend well to an aircraft after a

clearance was given, in terms of monitoring for compliance or progress in completing the control action,

most likely because they may have been concentrating

on other traffic present.

Subjects were incorrect in identifying weather as a

current impact (or impact in the next 5 minutes) in

39.3% of the scenarios. This is perplexing, in that

even though light and heavy weather symbols were

displayed in some scenarios, poor weather did not

impact traffic in any of the scenarios presented. This

finding most likely indicates that controllers have

difficulty estimating the impact of weather on air

traffic based on available data, an issue which has

previously been raised by controllers.

The frequency ofcorrect responses on each variable

provides some insight into the trade-offs that controllers make in allocating their limited attention across

multiple aircraft and pieces ofinformation that compete for that attention. This analysis is not meantto be

critical regarding the information controllers did not

attend to or retain in working memory. Attention

allocation strategies, such as those indicated here, are

needed and are effective most of the time in dealing

with the demands of controlling air traffic, as can be

demonstrated by the effective daily performance of

controllers and relatively low nationwide error rates.

A point that can be made, however, is that these

strategies may lead to a lack ofsituation awareness that

occasionally (due to a probabilistic link between SA

and performance [Endsley, 1995b]) results in errors.

This point is reinforced in that the patterns of attention demonstrated here can be correlated with certain

systematic characteristics of OEs.

It should also be noted that a fairly high degree of

variability was present on many ofthe variables, across

aircraft, subjects, freezes, and scenarios. Possible

sources of these variations will be examined more

closely.

Analysis of Freeze Number

An analysis was conducted to ascertain it there was

a difference between the first and second freezes in the

subjects' ability to correctly identify what was happening in the scenarios. The first freeze always occurred two minutes before the OE and the second

freeze always occurred at the time of the error. A

multivariate test was performed on the accuracy of

subjects' responses across the queries to examine differences between these two freeze times. (All measures, expressed as percent correct, were subjected to

an arcsine transformation prior to analysis to meet the

conditions of ANOVA.) The MANOVA was not

significant, F(13,98) = 1.554, p>.05. Therefore, subjects' recall of the situation was not significantly

different as a function of the presence of an OE.

An analysis was also conducted to determine whether

subjects reported a different level of subjective workload between the two freezes, as there is a concern that

higher workload may be associated with the occurrence of OEs. The NASA-TLX ratings were weighted

based on each subject's rankings and a combined

workload score was calculated. An ANOVA was conducted on the combined workload ratings to test for

differences between the two freezes. The overall NASATLX workload rating was significantly higher at the

time ofthe second freeze (during the OE), F(l,109) =

24.08, p<.001, as shown in Figure 2.

To investigate further, ANOVAs were performed

on each ofthe subscale ratings (performance, temporal demand, frustration, mental demand, effort, and

physical demand), revealing that ratings were significantly higher atthe second freeze for all ofthe subscales

(see Table 3) except for physical demand, which is as

would be expected. This supports the contention that

higher workload is associated with OEs; however, it is

unclear whether higher workload caused the error, or

whether the higher workload ratings were the result of

the error (i.e., higher ratings on the subscales ofstress,

frustration, demand, effort, and performance could have

been an outcome ofthe fact that an error occurred).

Analysis ofWorkload Impact

Subjective workload. Since subjective workload

was higher at the time ofthe OE, further analysis was

conducted to determine ifthere was a direct relationship
Freeze 1

Freeze 2

NASA - TLX Physical Demand Performance

Figure 2. Mean Workload Ratings at Each Freeze

Frustration

Table 3. ANOVAs: Impact of Freeze Number on

NASA- TLX Rating and Subscale Scores

VARIABLE df F P

NASA-TLX (overall) 1, 109 24.079 .001

- Performance 1,109 41.705 .001

- Temporal 1,109 9.580 .002

- Frustration 1,109 30.094 .001

- Mental 1, 109 5.722 .018

- Effort 1, 109 5.231 .024

- Physical 1.109 2.193 .142

' significant at a = .05 level

'* significant at a = .01 level
Table 4. Regressions: Relationship between NASA- TLX Rating and Awareness

of the Situation

VARIABLE df

Distance error (miles) .004

Aircraft reported (% correct) .064

Control level (% correct) .006

Call sign: alphabetic (% correct) .017

Call sign: numeric (% correct) .000

Altitude (% correct) .001

Change in altitude (% correct) .015

Speed (% correct) .018

Heading (% correct) .048

Turn (% correct) .018

Separation problems (% correct) .023

Transition to next sector (% correct) .022

Assigned clearances complete (% correct) .065

- all measures expressed in percentages were subjected to an arcsine transformation prior

to analysis

* significant ata = .05 level

** significant at a = .01 level

1,109 .475 .492

1, 109 7.418 .008

1, 109 .674 .413

1, 109 1.882 .173

1,109 .039 .843

1,109 .120 .730

1, 109 1.715 .193

1,109 2.049 .155

1,109 5.484 .021

1,109 1.956 .165

1,109 2.618 .109

1,109 2.495 .117

1,109 7.573 .007

Table 5. ANOVAs: Relationship between NASA- TLX Rating and

Awareness of the Situation

VARIABLE df

Assigned clearance correct (y/n) 1,79 5.705 .019

Assigned clearance conformance (y/n) 1, 79 .388 .535

Weather impact (v/n) 1,109 2.807 .097

' significant at a = .05 level

10
between the NASA-TLX combined rating and subjects' accuracy in their awareness of the situation.

Regressions were performed on the subjects' scores (in

terms of percentage correct) in comparison to their

workload ratings at the time, shown in Table 4. Those

variables scored as correct or incorrect were subjected

to an analysis ofvariance to determine whether workload ratings were associated with accuracy of recall,

shown in Table 5. A MANOVA showed a significant

relationship between these variables and the NASATLX score, F (14,96) = 1.93, p <.05. (Assignment

correct and assignment conformance were notincluded

in the multivariate analysis as theywould significantly

reduce the degrees of freedom in the test. These two

questions had a lower sample size because they were

only asked for aircraft that had not yet completed their

assigned clearances.)

The regressions revealed that subjects reported significantly fewer aircraft present (Figure 3) and identified aircraft heading correctly significantly less often

for those aircraft (Figure 4) as subjective workload

level increased. In addition, with increasing levels of

workload, subjects were significantly less likely to

correctly identify whether aircraft had completed their

assigned clearances (Figure 5) and were significantly

less likely to identify whether an aircraft had received

its assigned clearance correctly (Figure 6).

Number of aircraft. An objective measure of

taskload was also examined to determine its impact on

subject awareness and perceived workload. The number of aircraft present at the time of each freeze was

calculated and regressions performed to examine its

relationship to accuracy on each question (in terms of

percentage correct), shown in Table 6. Those variables scored as correct or incorrect were subjected to

an analysis of variance to determine whether number

of aircraft was related to accuracy, shown in Table 7.

A MANOVA showed a significant relationship between these variables and the number of aircraft

present, F (14,97) = 9.50, p <.001.

100

90

80

70

•a

S 60

o 50

oc

* 40

30

20

10

0

• • •

.... • i

20 40 60 80 100

NASA-TLX

Figure 3. Impact of Subjective Workload

on Aircraft Reported

100 - • •

90

80 • •

70

« 60

1 50 - •

* 40 •

30 -

20 m •

10

0

"

J. . I . . .1. -L20 40 60 80 100

NASA-TLX

Figure 4. Impact of Subjective Workload

on Awareness of Aircraft Heading

11
#-• • •

-u

20 40 60 80 100

NASA-TLX

Figure 5. Impact of Subjective Workload on

Awareness of Aircraft with Clearances

Not Completed

Correct Incorrect

Figure 6. Relationship Between Subjective Workload Rating and Awareness

of Correct Receipt of Assigned Clearance

12
As the number ofaircraft increased, the percentage

of aircraft that subjects reported as present significantly decreased (Figure 7). For those aircraft that

subjects did report, subjects were also significantly

less accurate on most ofthe other factors about those

aircraft as the number of aircraft present increased.

Theywere significantly more erroneous in their awareness of the location of the aircraft (Figure 8), and

correct less frequently regarding the aircrafts' control

level (Figure 9), both the alphabetic portion and

numeric portion ofthe call sign (Figures 10 and 11),

altitude (Figure 12), change in altitude (Figure 13),

airspeed (Figure 14), and heading (Figure 15) as the

number of aircraft increased. They were also correct

significantly less frequently in their awareness ofwhich

aircraft would transition out ofthe sector in the next

two minutes (Figure 16), which aircraft had completed

their assigned clearances (Figure 17), and if weather

would be a factor (Figure 18) as the number of aircraft

increased. Interestingly, the number of aircraft present

did not affect subjects' accuracy in reporting which

aircraft had a potential or current separation problem.

Table 6. Regressions: Relationship between Number of Aircraft and Awareness

of Situation

VARIABLE r2 df F D

Distance error (miles) .052 1 110 6.091 .015 '

Aircraft reported (% correct) .331 1 110 54.409 .001 '

Control level (% correct) .108 1 110 13.324 .001 '

Call sign: alphabetic (% correct) .075 1 110 8.945 .003 '

Call sign: numeric (% correct) .103 1 110 12.619 .001 "

Altitude (% correct) .080 1 110 9.556 .003 '

Change in altitude (% correct) .057 1 110 6.660 .011 '

Speed (% correct) .150 1 110 19.400 .001 '

Heading (% correct) .037 1 110 4.187 .043 '

Turn (% correct) .006 1 110 .695 .406

Separation problems (% correct) .003 1 110 .386 .536

Transition to next sector (% correct) .128 1 110 16.200 .001 "

Assigned clearances complete (% correct) .072 1 110 8.479 .004 '

- all measures expressed in percentages were subjected to an arcsine transformation prior

to analysis

* significant at a= .05 level

** significant ata = .01 level

Table 7. ANOVAs: Relationship between Number of Aircraft and

Awareness of Situation

VARIABLE df

Assigned clearance correct (y/n) 1,80 .003 .960

Assigned clearance conformance (y/n) 1, 80 2.789 .099

Weather impact (v/n) ' 1,110 8.898 .004

' significant at a= .05 level

'* significant at a = .01 level

13
1QQ

90

•

80

•

•

•

70

sr

^^|w •

60 •

•

* . '

^ •

• •

SO

• «

a

oc • •

*

40

30

20

10

-

•

•

•

0 ^..1 . . , i , ul_

30

0 S 10 15 20 25

Number of Aircraft

Figure 7. Impact of Number of Aircraft on

Aircraft Reported

20 -

S

<

.:•::.;.!' ;••':•.

I

* * • ' ■■■■! • I .... I

0 5 10 15 20 25

Number of Aircraft

Figure 8. Impact of Number of Aircraft on

Aircraft Placement Accuracy

100 r

90

80

70

| 80

8 50

* 40

30

20

10

0

i* il

« « t i i i i i i . . , , |

10 15 20 25

Number of Aircraft

Figure 9. Impact of Number of Aircraft on

Awareness of Aircraft Control Level

tS 60

e

£ 50 h

Ü

5? 40

30

20

10

0 . . . 4 I I • I » I.I

10 15 20 25

Number of Aircraft

Figure 10. Impact of Number of Aircraft on

Awareness of Aircraft Callsign (Alphabetic)

14
100 • • • *

100

90 • • • 90

80 •

• •

•

• •

• • 80

70

60

•

• •

1

•

•

•

t. •

I

o

70

60

50

40 • •

•

1 •

•

•

O

*

SO

40

30

20

•

•

• •

•

• •

• •

•

•7^

30

20

10 ». •

•

•

• • •

• 10

0

<

• * • ' 1 1 0

> 5 10 15 20 25

Number of Aircraft

Figure 11. Impact of Number of Aircraft on

Awareness of Aircraft Callsign (Numeric)

•t

• • • •

• • • • • • •

* • •

* _ •

i . . i .... t.... i ... . i

5 10 15 20 25

Number of Aircraft

Figure 12. Impact of Number of Aircraft on

Awareness of Aircraft Altitude

100

90

80

70

« 60

1 50

* 40

30

20

10

0

• •••• • • •••

• •

• •

-

• •

0 5 10 15 20 25

Number of Aircraft

Figure 13. Impact of Number of Aircraft on

Awareness of Aircraft Change in Altitude

Number of Aircraft

Figure 14. Impact of Number of Aircraft on

Awareness of Aircraft Speed

15
100 p

00 -

80

70 r60

50 5

* 40

30

20

10

0

• • • •

• • t •

• •

Ill|l|«ll4ltl|l||l<l||lj

10 15 20 25

Number of Aircraft

Figure 15. Impact of Number of Aircraft on

Awareness of Aircraft Heading

» » 4 I —'»'ll I I ' I I U ' I I 4 I I

5 10 15 20 25

Number of Aircraft

Figure 16. Impact of Number of Aircraft on

Awareness of Aircraft Transitioning Out of Sector

25 n

i i i» i iiiiiiji «ill i 11 i t

10 15 20 25

Number of Aircraft

Figure 17. Impact of Number of Aircraft on

Awareness of Aircraft Assignment Completion

Correct Incorrect

Knowledge of Weather

Figure 18. Impact of Number of Aircraft on

Awareness of Weather

16
Correlation between measures. An analysis was

also conducted to determine the relationship between

the number ofaircraft present, as an objective measure

of taskload, and subject's reported subjective workload. A regression showed a significant relationship

between these two measures ofload, F(l,109) = 6.45,

p = .013, as shown in Figure 19. It appears that the

number ofaircraft present in a scenario is a significant

driver of perceived workload that negatively impacts

subjects' ability to keep an accurate mental picture of

the situation, independent of any other factors that

might also drive subjective workload. An increase in

the number of aircraft present not only was related to

a tendency to attend to fewer aircraft, but also to the

tendency to know significantly less about these aircraft. This load-related shedding of information attended to appears to reflect some prioritization of

tasks, as awareness of those aircraft with current or

potential separation problems was not significantly

impacted by workload.

Analysis of Operational Errors

A closer examination of the nature of the OEs

included in this study was made. Table 8 provides a

summary of each error. Each error was classified in

terms of the SA Error Taxonomy (Endsley, 1995a).

Classifications were made by two independent raters

based on a description of each OE contained in the

Final Operational Error/Deviation Report completed

by internal FAA quality assurance investigators after

the OE, an analysis ofeach OE from the SATORI recreation, and verbal comments made by the subjects

in this study.

Of the 14 errors investigated, 5 clearly involved

task distractions in which the controller was distracted by the need to attend to other aircraft in the

sector. Three involved the misperception of some

information (due to expectations, workload, or task

distraction). Two OEs involved memory loss (associated,

with task distraction and high workload) in which the

controller forgot about an aircraft or a previous action.

Two OEs involved an over-reliance on defaults, expecting aircraft to behave as they usually do. Four involved

problems with inadequate projection ofthe dynamics of

the aircraft to anticipate separation problems. High

workload was specifically cited as a problem in one OE.

100

•

•

•

•

•

•

•

•

80 - •

•

•

•

•

• • •

•

•

3

60 -

• •

*

*~T*>

<

CO

•

*

J

"t"T» •

•

• V

<

z 40 — • •

•

•

i

•

• •

•

•

•

•

•

20

I

i

i".

•

•

n i-a-l

•

Ui-A

0 5 10 15 20 25

Number of Aircraft

Figure 19. Relationship Between Number of

Aircraft and Subjective Workload Ratings

Because controllers routinely must cope with competing task demands and high workload situations, an

analysis was made to determine whether the situations

in which these OEs occurred were intrinsically likely

to induce errors. Each of the investigated errors is

listed in Table 9, along with a listing ofthe number of

aircraft in the sector at the time ofthe OE and a rating

of the complexity of the scenario (as reported in the

Final Operational Error/Deviation Report). (This

information was not included in the reports for Scenarios 6, 12 and 13.) Scenarios 1, 2, and 14 had the

highest level of complexity and highest number of

aircraft in the sector at the time ofthe error, followed

by scenarios 9 and 11. Scenarios 3, 4, 7, and 10 had

lower than average complexity and number ofaircraft

at the time of the error. This agrees with previous

findings that many OEs occur during low workload

situations.

Table 9 also lists whether the controller making the

OE was aware that the OE was occurring at the time.

(This information was not available for two of the

errors.) In 4 out ofthe remaining 12 OEs, the controller was aware than an error was building but was not

able to avoid it. In the remaining eight, the controller

was not aware that an OE had occurred.

17
Table 8. Summary of Operational Errors Investigated

Scenario Description of Error SA Error Type

1 Clearance to wrong aircraft Level 1 - misperception

Level 1 - failure to monitor

(task distraction)

Confused two aircraft

Attending to other

aircraft

2 Descended aircraft into other aircraft Level 2 - inadequate mental model

Level 3 - lack of projection

3 difference in climb/closure rates of

two aircraft

Level 3 - lack of projection

4 difference in climb/closure rates of

two aircraft

Level 3 - lack of projection

5 Expected aircraft to descend faster

than it did

Level 2 - over-reliance on defaults

6 Readback error Level 1 - misperception

(expectations)

7 Delay in turning aircraft to accomodate

slow descent of other aircraft

Level 3 - lack of projection Didn't turn soon enough

8 Forgot aircraft - provided inadequate

clearance

Level 1 - memory loss

task distraction

9 Did not separate aircraft Level 1 - failure to monitor

task distraction Other radio calls

10 Readback error - aircraft at different

altitude

Level 1 - misperception

task distraction

11 Gave wrong heading command

Didn't monitor compliance

Level 1 - failure to monitor

12 Descended aircraft into other aircraft Level 1 - failure to monitor

Level 2 - over-reliance on defaults

13 Climbed aircraft into other aircraft

didn't judge separation

Level 1 - task distraction

14 Issued clearance to aircraft off frequency

Lost track of slow aircraft climb rate

Level 1 - memory loss - workload

Level 1 - failure to monitor- workload

Table 9. Subject Awareness of Error

Scenario Complexity

Rating

Number of

Aircraft in

Sector

Controller

Aware of Error

atOccurance

Subjects Aware of

Involved Aircraft

at Stop 1 (°/4

Subjects Aware of

Involved Aircraft

at Stop 2 f/4

Subjects Aware

of Error at

Stcp1(0/9

Subjects Aware

of Error at

Stop 2 (°/<)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

5 14 no 75 87 - 75

5 12 no 87 87 - 75

2 7 no 87 100 25 75

1 2 yes 100 100 - 100

3 5 yes 100 87 - 75

- - no 100 100 - 100

1 3 no 100 100 50 100

3 - no 75 87 - 75

4 10 yes 87 100 50 100

2 5 no 100 100 - 100

4 8 no 100 87 - 75

- - - 87 100 - 75

- - - 67 100 - 50

5 12 yes 75 100 25 100

18
As a comparison, the responses of the subjects in

this study were examined to determine whether they

were aware that separation errors were developing.

The percentage of the four subjects viewing each

scenario who reported the existence ofthe two aircraft

involved in the OE at each ofthe two freezes is listed

in Table 9. While for many of the scenarios both

aircraft were reported by all four subjects, at least one

of the involved aircraft was not reported at all by at

least one subject in approximately one-half of the

scenarios at the first freeze and approximately onethird of the scenarios at the second freeze.

In their response to the question regarding which

aircraft had lost, or would lose separation in the next

two minutes, in only four of the scenarios did even

one of the four subjects list the two aircraft involved

in the OE at the freeze, which occurred two minutes

before the OE. At the time ofthe second freeze, when

the OE occurred, at least one ofthe four subjects did

not identify the separation error in eight ofthe fourteen scenarios. In three of these cases, they reported

both aircraft but did not indicate a separation problem. In the remaining five scenarios, they also failed to

report at least one ofthe involved aircraft, indicating

it was outside of their focus.

Based on this analysis, it would appear that the

factors causing the controller to make an error in

many of these OEs were significant enough to have

been a problem for other trained controllers. This

result should be viewed with caution, however, as the

method employed in this study involved passive viewing

of the scenarios, instead of being under actual operational control.

CONCLUSIONS

This study reveals many interesting findings on the

role of situation awareness and workload in operational errors. Significant deficiencies in the ongoing

situation awareness ofthe subjects were presentin this

study. They had a fairly low ability to report on the

existence of many aircraft, or accurately recall their

location or many oftheir parameters. Their accuracy

was significantly impacted by the number of aircraft

present in the scenario and, to a lesser degree, by

perceived workload. After the number of aircraft

present exceeded approximately eight to ten, the ability ofsubjects to report on each aircraft declined quite

rapidly. Even for those aircraft they did report on,

their awareness of the relevant parameters for these

aircraft also declined when there were more than

approximately ten aircraft present. In the face of a

high number ofaircraft, subjects tended to attempt to

maintain their awareness of aircraft separation; however, this was still less than perfect (86.2% correct on

average). Other tasks, such as follow through on

clearances given to aircraft, also appeared to suffer

under increases in the number of aircraft and perceived workload.

While it is difficult to say that controllers need to

be able to remember aircraft parameters as long as they

know about aircraft separation, the pattern of attention represented by the accuracy scores in this study

are indicative ofmany ofthe OEs that occur. Readback

errors, for instance, may be directly related to the

tendency for subjects in this study to have a fairly low

awareness of whether aircraft given clearances had

received the clearance correctly. (Subjects were only

correct in 23.2% ofthe cases in knowing if an aircraft

had completed its assigned clearance, and ofthese, in

only 74.4% ofthe cases did they know ifthe aircraft

had received its assigned clearance correctly.) Problems in reporting the numeric portion of callsigns is

also reflective ofOE error patterns. Many ofthe OEs

included in this study involved over-reliance on expectations about how fast aircraft would travel or

should ascend or descend. Low accuracy in reporting

on the speed and heading of aircraft and being able to

report whether aircraft were turning, ascending, or descending are most likely indicative of this type of error.

An importantissue is why these SA problems occur.

It is a mistake to infer that these subjects (or the

controllers involved in the errors) were simply inattentive. Everyone has a limited amount ofattention to

distribute in any situation (Wickens, 1992). In complex activities, such as air traffic control, there is a

great deal of information to process. The pattern of

errors in this study suggests that, even ifsubjects made

as much use oftheir attention as possible in keeping

up with the scenarios, they may have had to limit

attention to some information in order to keep up

with the need to ensure that all aircraft were separated.

19
Probabilistically, this strategy is effective much ofthe

time, but is likely to produce occasional errors ofthe

type described here.

Information regarding the role of workload in the

occurrence of OEs is also present here. Subjective

workload, as measured by NASA-TLX, was significantly higher at the time ofthe operational error than

it was two minutes prior in the scenarios. (The number of aircraft was not higher, however.) While the

NASA-TLX score was correlated with the number of

aircraft present, the number of aircraft present was

more closely related to decreases in subjects' awareness of the situation (beyond a certain number of

aircraft). (What other factors are included by subjects

in their subjective workload ratings is unknown.)

It is important to note, however, that low and

moderate levels of SA were also present on many

variables even when the number of aircraft was relatively low. This finding concurs with other studies

that have found that SA and workload can operate

independently for various reasons (Endsley, 1993).

OEs were also found to occur under high-, moderate-,

and low-load conditions (as indicated by the number of aircraft present). Thus, while workload and

in many of these cases, momentary task distraction, can be associated with loss of SA and OEs,

errors can also occur for other reasons at lower

levels of workload.

One limitation of this study is that it involved

subjects who, although current, experienced controllers, passively viewed the ATC scenario re-creations.

While this procedure provided a great deal ofinsight

into factors affecting actual OEs (which can be difficult to produce under simulated conditions), one can

speculate as to whether their situation recall accuracy

and subjective workload ratings are the same as they

would be for controllers actively working the same

scenarios. While it is probably difficult to stipulate

that the absolute levels of SA and workload reported

would be the same, the general patterns presented

here are probably valid as reflections of subjects'

attention distribution.

It is somewhat likely, however, that levels of SA

may be lower under passive viewing conditions, rather

than active decision making, such as has been demonstrated under higher levels of automation in recent

research (Endsley & Kiris, 1995). In this light, difficulties in accurately identifying aircraft separation

problems shown by subjects in this study may be at

least partially reflective of the difficulties associated

with passive monitoring. This possibility needs to be

seriously investigated with regard to systems being

developed for automating future air traffic control.

As a basis of comparison, Mogford and Tansley

(1991), using a procedure similar to SAGAT during

actual simulations ofair traffic control with controller

trainees, found that subjects were able to report aircraft position with 86% accuracy, heading 82%, altitude 73%, callsign 55% and speed 53%. In comparison

with the data we obtained during passive viewing, the

attention devoted to each type of information is

similarly distributed; however, the Mogford and

Tansley study showed much higher levels of SA, even

though they studied controller trainees who would be

theorized to have lower SA than Full Performance

Level controllers. This supports the contention that

SA may be compromised under passive viewing, which

should be a significant concern for automation systems designed to place the controller in the role of

passive monitor.

In conclusion, this study may indicate several

sources leading to operational errors. The degree to

which these results are generalizable to controllers

involved in actively controlling air traffic needs to be

investigated, since the method employed in this study

involved passive viewing of the scenario, instead of

actual operational control ofthe aircraft. Comparable

data need to be examined during simulations of air

traffic control to verify similar attention allocation to

situation variables and responses to workload. At the

very least, issues regarding SA demonstrated in this

study may be indicative of problems that can be

expected if air traffic control ever becomes highly

automated, relegating the controller to a monitor.

Alternate automation designs that keep the controller

in the active decision making loop need to be explored

to prevent such an outcome.

20 -&U.S. GOVERNMENT PRINTING OFFICE: 1997 - 559-079/60001
REFERENCES

Endsley, M.R. (1988). Design and evaluation for situation awareness enhancement. In Proceedings ofthe

Human Factors Society32ndAnnualMeeting (pp. 97-

101). Santa Monica, CA: Human Factors Society.

Endsley, M.R. (1990a). A methodology for the objective measurement ofsituation awareness. In SituationalAwarenessinAerospace Operations (AGARDCP-478) (pp. 1/1 -1/9). NeuillySur Seine, France:

NATO-AGARD.

Endsley, M.R. (1990b). Predictive utility ofan objective

measure ofsituation awareness. In Proceedings ofthe

Human Factors Society 34thAnnualMeeting (pp. 41-

45). Santa Monica, CA Human Factors Society.

Endsley, M.R. (1993). Situation awareness and workload: Flip sides ofthe same coin. In R.S. Jensen &

D. Neumeister (Eds.), Proceedings ofthe Seventh

International Symposium on Aviation Psychology

(pp. 906-911). Columbus, OH: Department of

Aviation, The Ohio State University.

Endsley, M.R. (1995a). A taxonomy ofsituation awareness errors. In R. Fuller, N. Johnston, & N.

McDonald (Eds.), Human Factors inAviation Operations (pp. 287-292). Aldershot, England:

Avebury Aviation, Ashgate Publishing Ltd.

Endsley, M.R. (1995b). Toward a theory of situation

awareness. Human Factors, 3_Z(1)> 32-64.

Endsley, M.R., & Kiris, E.O. (1995). The out-of-theloop performance problem and level of control in

automation. Human Factors, 21(2), 381-394.

Endsley, M.R., & Rodgers, M.D. (1994). Situation

awareness information requirementsfor en route air

traffic control (DOT/FAA/AM-94/27). Washington, DC: Federal Aviation Administration Office

of Aviation Medicine. Available from: National

Technical Information Service, Springfield, VA

22161; ordering no. ADA289649.

Fracker, M.L. (1990). Attention gradients in situation

- awareness. In Situational Awareness in Aerospace

Operations (AGARD-CP-478) (Conference Proceedings #478) (pp. 6/1-6/10). Neuilly Sur Seine,

France: NATO - AGARD.

Hart, S.G., & Staveland, L.E. (1988). Development of

NASA-TLX (Task Load Index): Results ofempirical and theoretical research. In P.A. Hancock &

N. Meshkati (Eds.), Human mental workload {pp.

139-183). Amsterdam: North-Holland.

Jones, D.G., & Endsley, M.R. (1996). Sources ofsituation awareness errors in aviation. Aviation, Space

and EnvironmentalMedicine, 6_Z(6), 507-512.

Kinney, G.C., Spahn, M.J., & Amato, R.A. (1977). The

human element in air traffic control: Observations

and analyses ofthe performance of controllers and

supervisors in providing ATC separation services

(MTR-7655). McLean, VA: METREK Division,

MITRE Corporation.

Mogford, R.H., & Tansley, B.W. (1991). The importance ofthe air traffic controller's mental model

Paper presented at the Human Factors Society of

Canada Annual Meeting, Canada.

Rodgers, M.D., & Duke, D.A. (1993). SATORI: Situation assessment through the re-creation of incidents.fournalofAir Traffic Control, 25.(4), 10-14.

Rodgers, M.D., & Nye, L.G. (1993). Factors associated

with the severity ofoperational errors at Air Route

Traffic Control Centers. In M.D. Rodgers (Ed.)

An analysis ofthe operational error databaseforAir

Route Traffic Control Centers (DOTIFAAIAM-931

22). Washington, DC: Federal Aviation Administration Office of Aviation Medicine. Available

from: National Technical Information Service,

Springfield, VA 22161; ordering no. ADA275986.

Schroeder, D.J. (1982). The loss of prescribed separation between aircraft: How does it occur? In Proceedings ofthe Behavioral Objectives in Aviation

Automated Systems (pp. 257-269). Anaheim, CA:

Society ofAutomotive Engineers (SAE).

Schroeder, D.J., & Nye, L.G. (1993). An examination

ofthe workload conditions associated with operational errors/deviations at air route traffic control

centers. In M.D. Rodgers (Ed.) An analysis ofthe

operationalerrordatabase forAirRoute Traffic Control Centers (DOTIFAAIAM-93122). Washington,

DC: Federal Aviation Administration Office of

Aviation Medicine. Available from: National Technical Information Service, Springfield, VA 22161;

ordering no. ADA275986.

Stager, P., & Hameluck, D. (1990). Ergonomics in air

traffic control. Ergonomics, 3_3_(4), 493-499.

Wickens, CD. (1992). Engineering Psychology and

Human Performance (2nd ed.). New York:

Harper Collins.

21