DOT/FAA/AM-01/20

Larry L. Bailey

Ben F. Willems*

Linda M. Peterson

Civil Aerospace Medical Institute

Federal Aviation Administration

Oklahoma City, OK 73125

*William J. Hughes Technical Center

Federal Aviation Administration

Atlantic City International Airport, NJ 22161

December 2001

Final Report

This document is available to the public

through the National Technical Information

Service, Springfield, VA 22161.

The Effects of Workload and

Decision Support Automation on

Enroute R-Side and D-Side

Communication Exchanges

Office of Aerospace Medicine

Washington, DC 20591

U.S. Department

of Transportation

Federal Aviation

Administration
NOTICE

This document is disseminated under the sponsorship of

the U.S. Department of Transportation in the interest of

information exchange. The United States Government

assumes no liability for the contents thereof.
i

Technical Report Documentation Page

1. Report No. 2. Government Accession No. 3. Recipient's Catalog No.

DOT/FAA/AM-01/20

4. Title and Subtitle 5. Report Date

The Effects of Workload and Decision Support Automation on December 2001

En Route R-Side and D-Side Communication Exchanges

6. Performing Organization Code

7. Author(s) 8. Performing Organization Report No.

Bailey, L. L.1, Willems*, B.F.2, and Peterson, L.M.1

9. Performing Organization Name and Address 10. Work Unit No. (TRAIS)

1

FAA Civil Aerospace Medical

Institute

P.O. Box 25082

Oklahoma City, OK 73125

2

FAA William J. Hughes Technical

Center

Atlantic City International

Airport, NJ 22161

11. Contract or Grant No.

12. Sponsoring Agency name and Address 13. Type of Report and Period Covered

Office of Aerospace Medicine

Federal Aviation Administration

800 Independence Ave., S.W.

Washington, DC 20591 14. Sponsoring Agency Code

15. Supplemental Notes

This report was performed under Task HRR518.

16. Abstract

The Federal Aviation Administration (FAA) is introducing new decision aid technology, called decision support tools

(DSTs), into the air traffic control (ATC) workforce. Although considerable research has focused on the effects that

DSTs will have on pilot-controller communications, relatively little research has been conducted on how DSTs will affect

controller-controller communications. In this study, we examined the effects that aircraft density and different types of

DSTs have on the communication exchanges occurring within en route ATC teams. Two hypotheses guided the

research. Hypothesis 1: More communication exchanges will occur under high workload conditions, as compared with

low workload conditions. Hypothesis 2: More communication exchanges will occur when using DSTs, as compared with

not using DSTs. Method. Eight 2-person teams, consisting of certified ATC specialists from an en route center,

participated in a decision support automation research experiment. In the experiment communication exchanges of team

members were assessed within a 2 (aircraft density) X 3 (type of DST) repeated measures design. Communications were

analyzed by an ATC subject matter expert using an FAA ATC communication taxonomy consisting of three categories:

the topic of communication, the grammatical form of communication, and the mode of communication. Results. A total

of 3,194 communication events were coded. Partial support was achieved for hypothesis 1. When communications were

analyzed as a composite number, no statistically significant results were observed. However, when the categories were

analyzed separately, main effects for aircraft density were observed for two communication topics: general

communications about a specific aircraft, F(1,5) = 11.25. p < .05, and communications involving altitude changes,

F(1,5) = 10.66, P < .05. In both cases, there were more communications associated with the high-aircraft-density

condition. No statistical support was achieved for hypothesis 2. However, trend data suggested partial support for the

dominant topics of communication while performing under low aircraft density conditions. Conclusion. Controller-tocontroller communications may be differentially affected by varying levels of aircraft density and technology employed.

However, further replications are needed before a more definitive statement can be made.

17. Key Words 18. Distribution Statement

Document is available to the public through the

National Technical Information Service;

Air Traffic Control, Communication, Teamwork

Springfield, VA 22161

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price

Unclassified Unclassified 20

Form DOT F 1700.7 (8-72) Reproduction of completed page authorized

1

 THE EFFECTS OF WORKLOAD AND DECISION SUPPORT AUTOMATION ON

EN ROUTE R-SIDE AND D-SIDE COMMUNICATION EXCHANGES

Federal Aviation Administration (FAA) forecasts

indicate that the number of passengers carried on

commercial aircraft will double over the next 15 years

reaching one billion by 2015 (FAA Plan to Modernize, 1998). To handle the projected increase in air

traffic, the FAA is introducing new technology to aid

air traffic control specialists (ATCSs) in their tactical

and strategic decision making. Research previously

focused on the effects of these decision aids on pilotto-controller communication; however, relatively little

research is available concerning the impact of decision

aids on controller-to-controller (CTC) communication (Kanki & Pinzo, 1996). In this study, we examined the effects of aircraft density and automated

decision aids on communication exchanges between

en route air traffic control teams.

The National Civil Aviation Review Commission

states that the expected growth in aviation cannot be

safely accommodated without significant breakthroughs in air traffic modernization. The commission also cites air traffic communications as critical

components requiring modernization in the aviation

system (Aviation Financing, Air Traffic Control,

1999). Several emerging air traffic control systems,

including the 21st Voice Switching and Control System (VSCS), Display System Replacement (DSR),

and the User Request Evaluation Tool (URET) are in

differing stages of implementation. Researchers at the

Civil Aerospace Institute (CAMI) and the William J.

Hughes Technical Center (WJHTC) are jointly conducting investigations to assess the impact of modernizing ATC communications. This research considers both

hardware and software aspects of ATC modernization.

The current joint program of research focuses on

CTC communications as they relate to the coordination of job-related tasks. The present study examines

communication exchanges between En route Sector

Teams (ESTs), also known as R-side and D-side air

traffic controller teams, performing under varying

levels of workload and using various decision support

tools (DSTs). ESTs are composed of two individuals,

a radar controller (R-side) and a data controller (Dside). The radar controller’s responsibilities include

monitoring aircraft, maintaining aircraft separation,

and communicating via radio with the aircraft in their

airspace (i.e., sector). The data controller’s duties

include maintaining flight progress strips, entering

National Airspace System (NAS) data, coordinating

sector activities, and conducting intra-facility and

inter-facility communication. The above functions

are coordinated through CTC communication.

Operational Error Data Base

Tracking the number of operational errors is one

way of evaluating the quality of intra-EST coordination. Broadly speaking, an operational error occurs

when a deviation from the standard operational procedures (SOPs) leads to a loss of aircraft separation.

For example, in the en route environment, aircraft at

the same altitude must be separated by at least five

horizontal miles (FAA Order, 2000). Any time a

verified loss of separation occurs, an operational error

(OE) is recorded in the FAA’s Operational Error

database, and an investigation is conducted to determine the factors contributing to the loss of separation.

During an operational error investigation, an operational error reporting form is used to help determine the factors contributing to the error. As shown

in Table 1, the reporting form is divided into five error

categories. These include errors associated with: (1)

Data posting, (2) Radar display, (3) Communication

(primarily pilot to controller), (4) Coordination, and

(5) Position relief briefings.ÐW: Each error category

is further divided into varying levels of specificity.

Of interest in this report are the “intra-position”

errors associated with “Coordination.” In the en route

environment, intra-position coordination primarily

refers to problems with EST coordination. Since

communication is the means by which the R-side and

D-side coordinate their respective tasks, errors in

coordination include errors in CTC communications. Table 2 shows the number and percentage of

coordination related OEs recorded in the FAA’s OE

database. For the period from January 1998 through

October 2000 approximately 126 out of 505 (25%)

OEs were associated with problems of coordination.

Of the 126 coordination errors (CEs) 45 (36%) occurred in the en route environment. A breakdown in

intra-position coordination accounted for 13 out of

45 (30%) en route coordination errors (ECEs).
2

Table 1.

Structure of FAA ATC Operational Error Reporting Form

1. Data Posting c. Misunderstanding

a. Computer entry d. Readback

(1) Incorrect input (1) Altitude

(2) Incorrect update (2) Clearance

(3) Premature termination of data (3) Identification

(4) Other e. Acknowledgment

b. Flight progress strip f. Other

(1) Not prepared

(2) Not updated 4. Coordination

(3) Posted incorrectly a. Area of incident

(4) Reposted incorrectly (1) Inter-position

(5) Updated incorrectly (2) Intra-position

(6) Sequenced incorrectly (3) Inter-sector

(7) Resequenced incorrectly (4) Inter-facility

(8) Interpreted incorrectly b. An aircraft penetrated designated 

(9) Premature removal airspace of another position of operation

(10) Other or facility without prior approval.

c. Coordination was effected and

2. Radar display controller(s) did not utilize information

a. Misidentification exchange.

(1) Overlapping data blocks (1) Aircraft identification

(2) Acceptance of incomplete or (2) Altitudes/Flight level

difficult to correlated position info. (3) Route of flight

(3) Improper use of identifying turn. (4) Clearance limit

(4) Failure to reidentify aircraft when (5) Speeds

accepted target identity becomes (6) APREQS

questionable. (7) Special instructions

(5) Failure to confirm aircraft identity (8) Other

after accepting radar handoff.

(6) Other 5. Position relief briefing deficiencies noted

b. Inappropriate use of displayed data a. Employee did not used position relief

(1) Conflict alert Checklist

(2) Quick look b. Employee being relieved gave

(3) Mode C incomplete briefing

(4) MSAW/EMSAW c. Relieving employee did not make use

(5) Other of pertinent data exchanged at briefing

d. Other

3. Communication error

a. Phraseology

b. Transposition
3

Based on the results of Table 2, it may appear that

intra-position coordination problems are not a major

factor in the creation of operational errors. However,

it should be noted that operational error investigations occur after the fact. Since intra-EST communications are not recorded on tape, an account does not

exist of the actual communication exchanges between

R-side and D-side positions. Controllers must rely on

their memory to reconstruct both the content and

timing of the communication exchange being investigated. Thus, intra-EST communications may be a

factor in the creation of OEs, but the current reporting system is not sensitive enough to capture the

details. Additionally not all intra-position coordination problems produce an operational error. That is,

a breakdown in coordination may occur but the

problem is resolved in time to prevent an operational

error. Although there may be no loss of aircraft separation, from a safety perspective, even a potential loss

of separation is a matter of concern. A research initiative revising the OE reporting form to address these

and other human factors related to the systematic

analysis of OEs is underway at CAMI (Scarborough &

Pounds, 2001).

Even if intra-EST communications prove not to be

a major factor in the development of OEs, the need to

conduct research is not diminished. Empirical knowledge of the daily task-related communication exchanges between R-side and D-side controllers is

limited. Considering that the new DSTs are designed

to enhance the tactical and strategic decision making

capabilities of ESTs, further intra-EST communication research is essential. What effect might these new

technologies have on existing patterns of intra-EST

communications? Without an understanding of the

current baseline patterns of intra-EST communications,

there is no empirical way to answer this question.

Field Study

Given the lack of understanding of intra-EST communications, Peterson, Bailey & Willems (in press)

conducted a field study to determine the kinds and

frequency of intra-EST communication exchanges

associated with routine ATC performance. The field

study was conducted over a three-day period at one of

the FAA’s en route centers. Trained ATC specialists

using the FAA’s Controller-to-Controller Communication/Coordination Taxonomy (C4

T) recorded a

total of 24 hours of intra-EST communication. Coding occurred in 30-45 minute blocks of time between

the hours of 0700 and 1900. Coders chose to observe

the most active sectors to ensure a sufficient number

of coding events. Thus, rather than obtaining a statistical sample, coders chose a sample of convenience.

The C4

T has three communication categories: the

topic of communication, the grammatical form of

communication (e.g. question, answer), and the mode

of communication (e.g. verbal, nonverbal). Thus the

C4

T captures the “what” (topic) and “how” (form and

mode) of communication. Tables 3-5 respectively

describe the C4

T sub levels within each of the three

categories. For further information on the development and operational validation of the C4

T, the

reader is referred to Peterson, Bailey and Willems (in

press).

A frequency analysis of the coding results indicated

the following observations. The topic of most intraEST communications was related to aircraft traffic

(40%) and route of flight (15%), with the least communications involving inter-sector coordination approvals (1%). R-side and D-side controllers

demonstrated no statistical differences in the topic of

their communication. However, they differed in the

grammatical form of their communication. Whereas

the D-side had a higher percentage of statements and

Table 2.

Breakdown of En route Coordination Errors (1998-2000)

Year

All

ATC

OEs

All ATC

Coordination

Errors (CEs)

All En Route

Coordination

Errors (ECEs)

Inter-Position

ECE

Intra-Position

ECE

Inter-Sector

ECE

Inter-Facility

ECE

# %OE # %CE # %ECE # %ECE # %ECE # %ECE

1998 517 154 30% 47 31% 24 51% 11 23% 0 0 12 26%

1999 627 167 27% 59 35% 30 51% 21 35% 0 0 8 14%

2000* 371 58 16% 29 50% 19 66% 8 27% 0 0 2 7%

Average 505 126 25% 45 36% 24 53% 13 30% 0 0 7 16%

* Through October 1, 2000
4

Table 4.

Controller-to-Controller Coordination Communications Taxonomy (C4

T): Coordination-Communication

Grammatical Form.

ATC CoordinationCommunication

Grammatical Form

Definitions

Question A direct inquiry about the state or status of sector events.

Answer A response to a direct or implied question

Statement Providing information, without being asked, about the state or status of sector events.

Command A direct order to perform a specific act

Table 3.

Controller-to-Controller Coordination Communications Taxonomy (C4

T):

Coordination-Communication Topic

ATC CoordinationCommunication

Topic

Definitions

Approval Communications about intersector control/approval requests (“Get me control for descent

on that aircraft.” “APREQ N1234 climbing to FL330.”).

Handoff Communications relating to the transfer of radar identification of a particular aircraft

(“Handoff N1234.” “Did you handoff N1234?”).

Point Out Communications relating to the transfer of radar identification of a particular aircraft when

radio communications will be retained (“Point out N1234 to 22.”).

Traffic

Communications about a traffic situation involving a specific aircraft. Includes conflict,

spacing, other protected air space or terrain and the resolution of that situation (“Are you

watching that aircraft?”).

Altitude Communications about altitude not in relation to traffic (“N1234 is requesting flight level

220.”).

Route Communications regarding headings and/or amendments to route, not in relation to traffic

situations (“N1234 is on a 330 heading.” “Next sector, 27, wants N1234 over WEVER.”).

Speed Communications about speed not in relation to traffic situations (“These three aircraft are

slowed to 250 knots.”).

Weather

Communications about weather display or weather updates (Often communicated

nonverbally by passing written information: “Sector 22 says continuous moderate

turbulence above FL290.”).

Frequency Communications about an aircraft’s radio communications transfer or frequency

assignment (“Have you switched N1234 yet?” “Tell them to switch to N1234.”).

Flow Messages Communications about traffic flow restrictions not referring to a specific aircraft (“The

next sector is requesting 25 miles in trail.”) (due to radar outage).

Flight Strips Communications about flight progress strips (“Where is that strip?”) Often communicated

nonverbally.

Equipment Communications about any ATC hardware (The radar is out of service.”).

Aircraft ID Communications involving identifying a specific aircraft (Who was that who called?”

“That was N1234 who called.”).
5

Table 5.

Controller-to-Controller Coordination Communications Taxonomy (C4

T): Coordination-Communication

Expression

ATC CoordinationCommunication Form Definitions

Verbal Use of voice only communication.

Nonverbal Use of only body movement communication.

Mixed Communication that contains both a verbal and non verbal component.

Electronic Not used. Communication that is electronically transferred.

observations (56% vs. 30%), the R-side had a higher

percentage of answers (43% vs. 35%). From this, it

appeared that, compared with the R-side, the D-side

controller was the initiator of more communication.

Finally, there was no intra-EST difference in the

method of communication. The most frequent method

of intra-EST communication was verbal only (70%).

The remaining 30% of communications contained a

mixture of verbal and non-verbal expressions. This

latter finding suggested that any changes affecting the

line of sight between R-side and D-side positions

could disrupt the adaptive use of intra-EST nonverbal

communication.

Peterson et al. (in press) concluded their report by

highlighting the need to conduct controlled experiments to determine how changes in workload and

technology affects intra-EST communications. Although field studies preserve operational realism, with

a complex task such as air traffic control, many variables come into play, making it difficult to determine

which factors affect performance. For example, one

might wish to know the effects of more or less CTC

communication on sector safety and/or efficiency.

However, CTC communication is the result of the

interaction between the EST members and the environment (sector complexity and traffic volume) in

which they work. Within a laboratory environment,

researchers have greater control over the environmental setting (e.g., equipment, sector complexity, traffic

volume, work duration) and, thus, can better understand the effects of the experimental manipulation.

Collaborative Research

Although researchers have greater control in a laboratory environment, sometimes the experimental environment does not adequately represent operational

reality. When this happens, it is difficult to generalize

from experimental outcomes to the operational environment. One way of addressing the artificial nature

of a laboratory setting is to ensure that the equipment,

participants, and stimulus material reflect real world

conditions. This requires significant financial resources. The hardware and software support necessary

to simulate a high fidelity en route air traffic control

environment costs in excess of $500,000. In addition,

the cost for each ATC participant is $2500/week.

Given that research in the area of intra-EST communication is at the descriptive stage of empirical

research, large expenditures devoted solely to describing CTC communications are difficult to justify. To

help reduce the cost of research, researchers from

CAMI and the WJHTC developed a collaborative

initiative whereby intra-EST communication data are

collected on all experiments related to the WJHTC’s

Decision Support Automation Research (DSAR) program (Willems, 1999). The research objective of DSAR

is to evaluate ATC human performance under varying

levels of workload and DSTs. Thus, DSAR experiments provide an ideal setting for the study of intraEST communications.

Research Hypotheses

During the descriptive phase of research, objectives

focus more on systematically collecting and organizing data than it does on hypothesis testing. However,

a body of literature exists on controller-to-pilot communication. Using this literature as a starting point,

some initial hypotheses can be generated about the

effects of workload and technology on communications.

Workload

A consistent finding in controller-to-pilot voice

communications research is that workload affects the

quality and quantity of communication exchanges
6

(e.g., Prinzo & Britton, 1993). In this literature,

workload is primarily measured by the number of

aircraft at a given time (i.e., aircraft density) under the

control of a R-side ATCS. As aircraft density increases, there is a corresponding trend toward an

increasing number of communication errors (Morrison

& Wright, 1989; Morrow, Lee, & Rodvold, 1990).

Research suggests that, as ATCS and pilots become

overburdened the clarity of their communications

(e.g., incomplete phraseology, mispronunciation, and

rapid speech) begin to suffer. This, in turn, places

ATCS and pilots at a higher risk of committing

readback/hearback errors (Morrison & Wright, 1989).

Standard ATC protocol requires pilots and ATCS to

repeat what they heard. The sender then knows that

the message was accurately received. A readback/

hearback error occurs when an incorrect pilot or

ATCS readback of information goes uncorrected.

In addition to communication errors, changes in

workload also affect the kinds of communication

exchanges that occur. For example, researchers at

Human Technology Incorporated (1991) examined

the effects that ATC communications had on system

performance. They found that high performance teams

issued more communication reports to pilots than did

lower performance teams. Furthermore, compared to

low performance teams, high performance teams issued shorter messages as a means of insuring accuracy.

Ratings of team performance were based on over-theshoulder ratings conducted by ATC subject matter

experts. However, it remains to be seen whether these

results will generalize to the broader ATC population

or are specific to a given experimental manipulation.

Automation

Technology is sometimes considered to be the

answer for life’s problems. However, technology itself

can become a burden. In studies of the use of automation in the cockpit, Hart and Sheridan (1984) found

that pilot workload shifted from a physical burden to

a more cognitive one. This was evident when the

automation required pilots to serve as monitors of a

system such as monitoring cockpit flight management

system displays (Sarter & Woods, 1994). In studies of

the use of automation in commercial airlines, Wiener

(1985) noted that pilots found the additional task of

monitoring to be troublesome especially during high

workload. Not only did monitoring add to the fatigue

but there were times when the automation needed to

be turned off or ignored during critical portions of a

flight (Wiener, 1988).

In addition to the added burden that some technologies create, operators may not use the technology

as designed because they simply do not trust it (Riley,

1994). Technology promises much, but the reality is

that it sometimes falls short on delivery. For example,

as part of its ATC modernization program, the FAA

developed new data display terminals and keyboard

configurations. It was expected that the new system

would help to improve system capacity. However,

when the equipment was tested in a field setting, users

found that they were having problems making the

transition to the new system (Allendoerfer, Galushka,

& Mogford, 2000). Compared with performance

under the old system, ATCSs were slower and tended

to make more data entry errors. Rather than viewing

this as simply a matter of adjustment, users attributed

the problems that they were having to poor system

design.. Although some sort of an adjustment period

was expected, problems such as these emphasize the

need to switch from technology centered approaches

to more human centered approaches to modernization. Whereas the former requires the human to adapt

to the technology, the latter requires the technology to

adapt to the needs of the user (Billings, 1988).

Although the use of technology is related to

workload, its effects on communication are unclear.

From the concept of monitoring, an operator has no

need to communicate unless an event occurs requiring

the actions of another. As far as whether a given

technology is used or trusted, one might conjecture

that communications would increase during the transition period of adjusting to the new technology. This

would be true if there were problems with the technology and team members had to determine the level of

trust or confidence in the system. However, once the

transition period is completed (and assuming that the

system is reliable), one would expect communications

to return to a previous baseline.

Considering the above discussion, the following

hypotheses were derived.

Hypothesis 1: More intra-EST communication

exchanges will occur under high workload conditions, as compared with low workload conditions.

Hypothesis 2: More intra-EST communication

exchanges will occur when using DSTs, as compared

with not using DSTs. This hypothesis is based on the

assumption that there will be insufficient training

time for participants to feel fully comfortable with the

new system procedures.
7

Study Limitations

One of the inherent difficulties with conducting

applied research is working within limitations. Two

limitations that affect this current research are a lack

of statistical power and financial resources. Statistical

power refers to the minimal sample size necessary for

a given effect to reach statistical significance. As cell

sizes fall below this minimum, a the ability to adequately test hypotheses is reduced.

Prior to conducting the experiments, researchers

conducted a power analysis using software from the

Methodologist’s Toolchest (Brent & Thompson,

1996). The procedure is derived from Kraemer and

Theimann (1987) for related samples balanced designs on pair-wise comparisons with a two-tailed test

of significance. Assuming a mean difference of 2, an

average standard deviation of 10, an alpha of .10, and

a beta of .50, the results produced a minimum sample

size of 18 cases (or ESTs) or a total of 36 ATCSs.

However, in this study, the available financial resources could only cover the cost of 8 ESTs or a total

of 16 ATCSs. As is often the case in applied settings,

economic realities become the limiting factor in the

quality of the studies conducted.

Given the above limitations, one might ask, is there

any real value in studying intra-EST communications

in a laboratory setting? The answer, of course, depends on the value that is placed on inferential statistics. There exists the concern among part of the

scientific community that, unless results achieve statistical significance, they are not meaningful or are

just artifacts of sampling error (e.g., Branch, 1999).

What gets lost is that statistical significance does not

guarantee that the results will generalize to the broader

population. Generalization depends on how the sample

was selected to ensure representation (i.e. generalization back to the population). Continuing with the

preceding example, let us assume that we have a population of 10,000 enroute ATCSs or 5,000 ESTs. According to sampling theory, we would have to draw a random

sample of 357 ESTs to ensure that our sample represents

the enroute population. Sample computations were

made using the website calculator at

http://ebook.stat.ucla.edu/calculators/sampsize.phtml.

Note that an additional 321 ATCSs are needed to

ensure representation as compared to statistical significance. If achieving statistical power is cost prohibitive, representation is even more so.

One way of dealing with the problem of inadequate

sample size is evaluating intra-EST communication

patterns across numerous studies. Over time, a communication profile develops based on known contextual factors (i.e., experimental conditions). To the

extent that similarities in the communication profile

are observed, this would suggest a phenomena that is

robust and thus indicative of the universal nature of

R-side and D-side communication. However, to the

extent that the communication profiles are different,

this would suggest that situation-specific conditions

dictated the nature of intra-EST communications.

Thus, regardless of the outcome, the systematic collection of data has the potential to provide insight into

the nature of intra-EST communications and to provide guidelines for conducting field studies.

METHOD

The research described in this report is a subset of

a broader experiment associated with the WJHTC’s

DSAR program. Only the methods directly related to

the assessment of CTC communications are reported

in this study. For additional information on the

experimental design, see Willems (1999).

Participants

Eight 2-person teams, consisting of certified ATCSs

from an en route center participated in a two-week

DSAR experiment. Participants were paid their regular salary and were on government per diem throughout the duration of the experiment. Following training,

participants were randomly assigned to an experimental

condition.

Equipment

The equipment used in this experiment was functionally equivalent to the workstations used by the R-side

and D-side ATCSs in an en route center. An ATC highfidelity simulator was used to model the airspace used in

this study. The experimental environment included full

DSR workstations with full operational functionality. A

2,000 by 2,000 pixel, 29" video display unit represented

the DSR radarscope. Also included were a DSR flight

strip bay, an en route keyboard and trackball, and a DST

terminal that included both a conflict avoidance tool

(CAT) and a flight path planning tool (FPPT). Prior to

the experiment, none of the participants had used the

CAT or FPPT.

The CAT used the trajectory of aircraft to predict

potential loss of aircraft separation. For example, all

things being constant, the CAT determined whether

the trajectory of two aircraft would intersect any time

within a 20-minute period. Conflict information was

presented in several display windows that depicted

various flight data and conflict information. One

display window presented a list of all aircraft inbound

to the sector. The list showed who was the controlling
8

sector of a particular aircraft; flight data such as flight

route, aircraft type, speed, altitude and beacon code,

and conflict indication. The conflict indication displayed red for predicted violations of less than 5

nautical miles and displayed yellow when less than 12

nautical miles, but more than 5 nautical miles. A

graphic-plan-display window graphically depicted

aircraft and resembled the DSR display.

By a D-side entering a revised altitude or route

change, the FPPT enabled the EST to determine the

best flight plan for resolving a potential intra- and/or

inter-sector conflict. Thus, instead of resolving conflicts as tactical decisions, the FPPT enabled ATCSs

to choose a strategic resolution to a given problem.

Within an EST, information from the DSTs was processed by the D-side and communicated to the R-side.

Stimulus Material

An ATC supervisor on detail to the WJHTC developed air traffic control scenarios for use in the training

and experimental conditions. Each scenario was 45

minutes long. For the training condition, six scenarios placed participants under a moderate workload

defined as the amount of air traffic that could be

comfortably handled by a R-side/D-side team, as

perceived by a typical ATC supervisor. For the experimental conditions, six low-and seven high-workload

scenarios were developed. The low-workload scenarios were defined as the least amount of air traffic in

which a typical ATC supervisor added a D-side to

assist the R-side controller. The standard for developing high workload scenarios was the greatest amount

of air traffic that a typical ATC supervisor allowed a

R-side / D-side team to manage.

Measures

All task-related R-side/D-side communications were

videotaped and coded by an ATC subject matter expert

(SME) using a computerized version of the C4

T. The

SME was a retired ATCS who had spent the last five years

providing contract support in the area of ATC communications research. This involved providing subject matter expertise in the development of coding taxonomies

for both controller- to-pilot communications and controller-to-controller communications.

Training

Prior to participating in the experiment, participants

received four days of training on the air space, scenario

flow and traffic type, DSR workstations, and DST

equipment. After completing the familiarization phase

of training, participants performed ATC tasks in response

to six 45-minute air traffic scenarios. Each scenario was

calibrated by a SME to represent a moderate amount of

workload. The decision aids used during training followed the experimental design.

Design and Procedures

Communication exchanges of team members were

assessed within a 2 (workload) X 3 (type of DST)

doubly repeated measures design. The two levels of

workload were low and high, as previously described

in the Stimulus Material section. DST levels were: (1)

Tech 1- only paper flight strips, (2) Tech 2 - electronic

flight strips and a CAT, and (3) Tech 3- electronic

flight strips, CAT, and a FPPT.

RESULTS

Field Comparison

Prior to hypothesis testing, the frequency data

(collapsed across all trials) were analyzed by topic,

grammatical form, and communication mode so that

a direct comparison could be made with the field

study conducted by Peterson, et al (in press). The

purpose of this comparison was to determine if differences existed in the patterns of communication operating within the two settings. Although the experiment

was not modeled after a particular en route center, the

scenarios were designed to reflect real-world events.

Thus, one might expect similarities between intraEST communications within a field and experimental

setting. However, differences are also likely because

communications, as a whole, are the product of various person (e.g., personality) and environmental factors (sector/scenario demands).

Table 6 shows the comparisons for the percentage

of R-side and D-side communications related to the

topic of communication, its grammatical form, and

the mode of expression. Although Table 6 shows that

field and laboratory settings differed in the percentage

of total communications that were attributed to a

given topic, both laboratory and field assessments

identified the same top three topics. These included

communications about “Traffic,” “Route,” and “Altitude.” Compared to the field the most noticeable

difference in the experiment was the lack of communications about “Weather (not part of the experiment), “Point-Outs,” and “Traffic flow.” Additionally,

there were only minimal amount of communications

concerning “Flight Strips.”

The grammatical form of communications also

differed between the two environments. The field

results show a strong tendency for the D-side to make

statements (55.9%) and the R-side (42.8%) to provide
9

Table 6.

Contrasting percentage comparisons of R-side and D-side communications in field and laboratory

settings.

Enroute Center Laboratory Setting

R-side% D-side% R-side% D-side%

Communication Topic

Traffic 41.0 37.9 53.7 51.2

Route of flight 14.2 15.6 13.1 11.7

Altitude 7.1 8.0 16.0 21.1

Weather 5.5 6.8 0.0 0.0

Point-out 5.0 6.1 0.0 0.0

Traffic flow 5.2 5.6 0.0 0.0

Frequency 5.9 4.7 3.5 2.7

Flight Strips 5.6 4.5 0.7 1.0

Equipment 3.3 4.0 4.5 4.9

Hand-off 3.6 3.1 2.9 1.8

Speed 2.6 2.8 4.4 2.8

Approval 1.0 0.9 1.0 1.8

Communication Format

Statement 29.7 55.9 58.0 77.3

Answer 42.8 25.1 18.3 10.4

Question 12.2 16.4 22.9 11.3

Command Answer 5.8 0.3 0.0 0.0

Command 0.5 2.4 0.8 1.0

Communication Mode

Verbal 77.1 69.3 93.9 69.0

Verbal & Nonverbal 14.7 16.8 5.0 24.7

Nonverbal 13.9 13.9 0.5 2.8

Equipment 0.0 0.1

Equipment & Verbal 0.6 3.4

Equipment & Nonverbal 0.0 0.1
10

answers. In contrast, the experiment’s results show

both R-side and D-side predominately making statements (58% vs. 77.3%).

Verbal communication is the method of choice for

R-side and D-side controllers. However, as Table 6

shows, the D-side had a stronger tendency to use a

mixture of verbal and nonverbal expressions in the

experiment than did the R-side (24.7% vs. 5%). For

the field setting, the percentage of mixed messages was

similar for both the R-side and D-side (14.6% vs.

16.8%). Another difference between the two settings

is the percentage of nonverbal communications that

were used. In the field, 13.9% of the communications

was solely nonverbal for both the R-side and D-side.

This is in contrast to the lower percentages recorded

during the experiment (R-side 0.5%, D-side 2.8%).

Hypothesis Testing

During the experiment, problems occurred with

the computer algorithm used to assign participants to

a given condition. This created missing data for three

teams in time 1 and three different teams in time 2.

Because the communication component of the experiment was primarily descriptive in nature, the

decision was reached not to replace missing data with

mean substitutions, but instead to drop the cases from

further analysis. Furthermore, since time 1 and time

2 data involved missing data for six different teams,

the doubly repeated measure design was dropped and

changed to a simple 2 x 3 repeated measure design

using only time 1 data. From these data, 3,194 communication events were coded. Descriptive statistics

for these data are presented in Table 7. These data are

also graphically displayed in Figures 1-3.

A 2 x 3 repeated measures analysis of variance

(ANOVA) was used to examine the effects that aircraft density (low vs. high) and level of technology

(Tech1, Tech2, and Tech3) had on the topic of intraEST communication, its grammatical form, and the

method of expression.

Hypothesis 1 stated that there would be greater

amounts of communication under high workload

conditions, as compared to low workload conditions.

When communications were analyzed as a composite

number, no statistically significant results were observed. Each of the C4

T categories was then analyzed

separately to determine if the experimental conditions

affected any of the subcategories. No significant results were observed for any of the subcategories within

the grammatical form of communication or the mode

of communication. However, main effects for

workload were observed for two communication

topics: (1) communications identifying a specific aircraft, F(1,5) = 11.25. p < .05, and communications

involving altitude changes, F(1,5) = 10.66, p < .05. In

both cases the high aircraft density condition was

associated with more communication exchanges between the R-side and D-side positions.

Hypothesis 2 stated that more communications

would occur in the higher tech condition as compared

to the lower tech conditions. Although statistical

significance was not attained, Figure 1a shows that

under the low workload condition, the highest degree

of technology (tech 3) recorded the most communication events about Traffic and Altitude.

CONCLUSION

In this study we examined the effects of aircraft

density and different kinds of automated decision

aids on communication exchanges between R-side

and D-side air traffic controller team members. Collapsing across all technologies, the results showed

more total communication exchanges under high aircraft density conditions as compared to low aircraft

density conditions. This result was driven by the top

two topics of communication: (1) communications

about the identification of a specific aircraft, and (2)

altitude changes. The effects of automation decision

aids on communications, however, were not as clear.

There was some evidence that under the low aircraft

density condition, more communication exchanges

occurred (about the identification of a specific aircraft, altitude changes, and route of flight) using the

highest level of automation, as compared with the two

lower levels of automation.

The overall trend of the C4

T profile of the experiments (collapsing across all conditions) compared

favorably with data collected from the field. In both

cases, the top three topics of communication were

Traffic, Route, and Altitude. Also, in both cases,

verbal communication was the dominant mode. Despite these similarities there were notable differences

in the grammatical form of communication between

R-side and D-side controllers. Whereas in the field it

appeared that the R-side was primarily responding to

statements made by the D-side, in the experiment, it

appeared that both R-side and D-side controllers were

issuing statements. In addition, more nonverbal communications were observed in the field as compared to

the laboratory environment. Given that communication is a function of the characteristics of team members and the situations that they face, it is not surprising

that differences occurred. It remains for future research
11

Table 7.

C4

T Descriptive Statistics of Experimental Conditions.

Tech 1 Tech 2 Tech 3

Aircraft

Density Measure n M SD n M SD n M SD

Low

Total Communication 5 63.20 23.27 5 62.60 27.63 5 61.00 27.07

Topic of Communication

Approval 1 1.0 1 1.0 3 1.33 0.58

Handoff 3 3.33 6.43 2 5.0 5.66 3 5.0 4.58

Pointout 0 0 0

Traffic 5 24.40 6.43 5 26.20 21.71 5 28.60 14.28

Altitude 5 8.80 6.53 5 8.80 9.26 5 12.40 5.94

Route of Flight 5 7.4 3.36 5 5.80 4.55 5 9.00 9.80

Speed 5 3.2 2.68 5 4.8 5.93 2 2.50 .71

Weather 0 0 0

Frequency 5 3.4 1.94 3 2.67 0.58 4 2.75 0.96

Traffic Flow 0 0 0

Flight Strips 2 6.0 4.24 0 0

Equipment 2 8.5 10.60 5 5.0 3.87 4 5.0 2.45

Aircraft ID 0 0 0

Grammatical Form of

Communication

Question 5 9.00 4.36 5 10.20 3.56 5 12.20 7.16

Answer 5 9.60 8.99 5 8.80 4.18 5 8.80 5.22

Statement 5 36.20 11.12 5 42.40 19.63 5 39.60 16.85

Communication Mode

Verbal 5 46.40 18.30 5 53.60 24.09 5 51.60 23.55

Mixed Verbal Nonverbal 5 6.8 7.05 5 6.60 3.51 5 7.80 4.66

High

Total Communication 5 69.00 31.63 5 72.60 5.43 5 60.60 23.07

Topic of Communication

Approval 1 5.00 2 3.0 1.41 1 6.00

Handoff 2 2.0 3 2.33 1.52 2 1.50 .71

Pointout 1 1.50 0 0

Traffic 5 30.40 10.55 5 35.40 13.18 5 36.40 13.16

Altitude 5 20.00 10.86 5 13.60 6.19 5 11.40 5.94

Route of Flight 5 8.20 9.88 5 13.60 10.11 5 5.40 3.78
12

Table 7. (Continued)

C4

T Descriptive Statistics of Experimental Conditions.

Tech 1 Tech 2 Tech 3

Aircraft

Density Measure n M SD n M SD n M SD

High

Speed 3 3.67 1.53 4 3.50 1.00 4 4.75 5.56

Weather 0 0 0

Frequency 4 3.75 3.59 4 1.75 0.50 4 3.25 1.26

Traffic Flow 0 0 0

Flight Strips 2 4.0 1.41 0 0

Equipment 2 5.5 6.36 3 4.33 1.53 3 6.33 3.51

Aircraft ID 0 0 0

Grammatical Form of

Communication

Question 5 8.40 2.97 5 13.60 2.88 5 13.40 5.03

Answer 5 7.60 6.43 5 11.00 4.95 5 9.40 6.91

Statement 5 51.80 23.99 5 45.80 6.38 5 42.20 13.03

Communication Mode

Verbal 5 52.80 24.10 5 55.20 8.76 5 56.40 17.90

Mixed Verbal Nonverbal 5 13.80 8.14 5 14.60 7.06 5 8.40 8.14
13

0 10 20 30 40

Approval

Handoff

Pointout

Traffic

Altitude

Route

Speed

Weather

Frequency

Traffic Flow

FlightStrips

Equipment

A/C ID

Tech 1 Tech 2 Tech 3

Figure 1a.

Topic of Communication Comparison of Three Decision Support Technologies Under Low

Workload Conditions.
14

0

10

20

30

40

50

Question Answer Statement

Tech1 Tech2 Tech3

Figure 2a.

Comparison of the Grammatical Form of Communication for Three Decision Support

Technologies Under Low Workload Conditions.

0

10

20

30

40

50

60

Question Answer Statement

Number of Communication Events

Tech1 Tech2 Tech3

Figure 2b.

Comparison of the Grammatical Form of Communication for Three Decision Support

Technologies Under High Workload Conditions.
15

0

10

20

30

40

50

60

Verbal Mixed

Tech1 Tech2 Tech3

Figure 3a.

Comparison of the Mode of Communication for Three Decision Support

Technologies Under Low Workload Conditions.

0

10

20

30

40

50

60

Verbal Mixed

Tech1 Tech2 Tech3

Figure 3b.

Comparison of the Mode of Communication for Three Decision Support

Technologies Under High Workload Conditions.
16

to determine whether and how intra- or inter-team

variability affects system outcomes of safety, efficiency, and effectiveness.

Future Directions

Objective measures of ATC intra-team coordination

have remained an elusive goal for the FAA. Despite the

number of training initiatives targeted at improving

ATC team performance, none of them has empirically

demonstrated that they lead to improvements in intrateam coordination within the job setting.

Recently, the FAA funded a cooperative research

program with Kansas State University to develop a

method of identifying expert performance: the

Cochran-Weiss-Shanteau statistic (CWS) (Shanteau,

2001; Thomas, 2001; Weiss, 2001). In its current

form, CWS is a summative index that differentiates

between levels of expertise residing within individuals. However, a dynamic measure of performance

expertise is in the developmental stages. By extending

this measure to the team level of analysis, researchers

will accomplish a number of objectives. These include: (1) objectively classifying teams based on their

levels of expertise, (2) determining the role that intrateam communication plays in developing team expertise, and (3) determining how levels of team expertise

relate to sector outcome measures such as the average

amount of fuel burn or time in sector.

Whether intra-EST communication research takes

advantage of advances in the measurement of team

expertise, it is important to explore the impact that intraEST communications has on system safety and efficiency outcomes. To the extent that a relationship exists,

researchers will be in a better position to develop more

sophisticated models of ATC performance.

REFERENCES

Allendoerfer, K. R., Galushka, J., Mogford, R. H. (2000).

Display system replacement baseline research report.

(DOT/FAA/CT-TN00/32). Atlantic City International Airport, NJ: Federal Administration William J. Hughes Technical Center.

Aviation Financing, Air Traffic Control Modernization, and Safety and Security, Statement of the

Honorable Jane F. Garvey, Federal Aviation Administrator, before the House Committee on Appropriations, Subcommittee on Transportation,

(March 9-10, 1999).

Billings, C. E. (1988). Toward human centered automation. In S. D. Norman & H. W. Orlady (Eds.),

Flight deck automation: Promises and realities (pp

167- 90). Moffett Field, CA: NASA-Ames Research Center.

Branch, M. N. (1999). Statistical inference in behavior

analysis: Some things significance testing does and

does not do. Behavior Analyst, 22, 87-92.

Brent, E., & Thompson, A. (1996). Methodologist’s

toolchest: User’s guide and reference manual. Columbia, MO: Idea Works, Inc.

FAA Plan to Modernize the Air Traffic Control System,

Statement of the Honorable Jane F. Garvey, Federal Aviation Administrator, before the Committee on Commerce, Science, and Transportation,

Subcommittee on Aviation, (February 26, 1998).

Federal Aviation Administration. (2000). Air Traffic

Control (DOT/FAA/Order 7210.65M). Washington, DC: Federal Aviation Administration.

Hart, S. G., & Sheridan, T. B. (1984). Pilot workload,

performance, and aircraft control automation. In:

Human Factors Considerations in High Performance Aircraft (AGARD-CP-371) (pp. 18/1-18/

12). Neuilly Sur Seine, France: NATO-AGARD.

Human Technology, Inc. (1991). Analysis of controller communication in en route air traffic control.

Report to the Federal Aviation Administration.

McLean, VA.

Kanki, B.G. & Prinzo, O.V. (Eds.) (1996). Methods and

metrics of voice communications. (DOT/FAA/AM96/10) Department of Transportation, Federal

Aviation Administration, Office of Aerospace

Medicine, Washington, D.C.

Kraemer, C., & Thiemann, S. (1987). How many subjects?: Statistical power analysis in research. Beverly

Hills: Sage.

Morrison, R. & Wright, R.H. (1989). ATC control and

communications problems: An overview of recent

ASRS data. In R. S Jensen (Ed.) Proceedings of the

Fifth International Symposium on Aviation Psychology (pp. 901-7). Columbus, OH: The Ohio

State University.

Morrow, D.G., Lee, A., & Rodvold, M. (1990).

Analysis of routine pilot-controller communication. In Managing the Modern cockpit: Third

Human Error Avoidance Techniques Conference Proceedings. Warrendale PA: Society of

Automotive Engineers, Inc.
17

Peterson, L.M., Bailey, L. L., & Willems, B. (in press).

Controller-to-controller communication and coordination taxonomy (C4

T) Department of Transportation, Federal Aviation Administration, Office of

Aerospace Medicine, Washington, DC.

Prinzo, O. V., & Britton, T. W. (1993). ATC-pilot voice

communications - A survey of the literature. (DOT/

FAA/AM-93/20) Department of Transportation,

Federal Aviation Administration, Office of Aerospace Medicine, Washington, DC.

Riley, V. (1994). A theory of operator reliance on automation. In M. Mouloua & R. Parasuraman (Eds.),

Human performance in automated systems: Current research and trends (pp. 8-14). Hillsdale, NJ:

Erlbaum.

Sarter, N. B. & Woods, D. D. (1994). Pilot interaction

with cockpit automation II: An experimental study

of pilots’ model and awareness of the flight management system. The International Journal of Aviation Psychology,2, 1-28.

Scarborough, A., & Pounds, J. (2001). Retrospective

human factors analysis of ATC operational errors.

Presented at the 11th International Symposium of

Aviation Psychology, Columbus, Ohio.

Shanteau, J. (2001). Results from tests of the CWS method

against existing data sets. Presented at the 11th

International Symposium of Aviation Psychology,

Columbus, Ohio.

Thomas, R. P. (2001). The adaptation of CWS to

dynamic stimuli. Presented at the 11th International Symposium of Aviation Psychology, Columbus, Ohio.

Weiss, D. J. (2001). The derivation of CWS based on

findings from studies of expert performance. Presented at the 11th International Symposium of

Aviation Psychology, Columbus, Ohio.

Wiener, E. L. (1985). Cockpit automation: In need of a

philosophy. In Proceedings of the 1985 Behavioral

Engineering Conference (pp. 369-75). Warrendale,

PA: Society of Automotive Engineers.

Wiener, E. L. (1988). Field studies in automation. In S.

D. Norman & H. W. Orlady (Eds.), Flight deck

automation: Promises and realities (pp 37-55).

Moffett Field, CA: NASA-Ames Research Center.

Willems, B.F. (1999). Test plan for decision support

automation research (DSAR) in the en route air

traffic control environment.(Draft). Atlantic City

International Airport, NJ: Federal Administration

William J. Hughes Technical Center.