# Air Traffic Control in Airline Pilot Simulator Training and Evaluation

Judith Bürki-Cohen and Andrew J. Kendra

Much airline pilot training and checking occurs entirely in the
simulator, and the first time a pilot flies a particular airplane, it
may carry passengers. Simulator qualification standards, however, focus
on the simulation of the airplane without reference to the air traffic
environment. This paper describes research examining the question of
whether simulator pilot training and evaluation would benefit from
improved simulation of radio communications. First, existing radio
communication simulation practices were investigated. Second, opinions
from instructors/evaluators were solicited. Third, the pertinent
literature was reviewed. Fourth, the effectiveness of current practices
was evaluated by surveying the Aviation Safety Reporting System.
Finally, recent efforts to improve radio communication simulation were
examined. The paper concludes that there is much evidence that
increasing the realism of radio communications would improve simulator
training and evaluation of airline pilots, but that finding effective
ways to do so will depend on collaboration of government, industry,
military and academia.

## introduction

### Simulator use in airline pilot training

Air carriers have been using airplane simulators for pilot training
since the mid 1950s. Procedures training, systems knowledge, and
navigation could be demonstrated and checked in a simulator, but stick
and rudder skills as well as judgment had to be demonstrated in the
airplane.

This changed with the introduction of the United States (U.S.) Federal
Aviation Administration's (FAA) Advanced Simulation Plan (ASP) \[FAA,
1980\], which specified technical requirements for simulators. Total
training and checking of airline pilots were now permitted in a
simulator that was qualified for this purpose by the FAA. This means
that certain types of training and certification of airline pilots can
be completed entirely in the simulator. This is followed by supervised
Initial Operating Experience (IOE) in revenue service, i.e., in the
airplane carrying paying passengers, for consolidation of knowledge and
skills acquired in the simulator. The FAA requires no further official
training and no additional check in the airplane \[FAA, 1996\].

The introduction of the ASP has eliminated training-related accidents at
major airlines that have access to the high-fidelity simulators required
for total training and checking in the simulator\--"it's a long time
since I lost a buddy in a training accident" \[Rolfe, 1996\].
Training-related accidents, however, continue to occur at smaller
airlines that still conduct some of their training in the airplane.
Also, the lack of access to high-fidelity simulators deprives the
smaller airlines of some of the training opportunities provided by
simulators, such as carefully constructed scenarios containing
emergencies.

In the interest of one level of safety for all airlines, a requirement
that came into effect in 1997 \[FAA, 1997\], the FAA is considering a
new rule mandating that all airline training and checking occur in the
simulator. At the same time, the FAA is also considering an extension of
the credit allowed in the highest-level simulators, which may lead to a
reduction in the duration of supervised IOE.

In light of these initiatives, it becomes all the more critical that the
skills acquired in the simulator fully transfer to the airplane when
using the simulator for training. Just as important, pilots' in-air
skills must be accurately reflected in the simulator when using the
simulator for pilot certification. In other words, a valid simulator
will stimulate the same pilot response as the airplane, so that both
performance and behavior (control strategy and inputs) of pilots in the
simulator and in the airplane are similar.

To achieve full transfer between simulator and airplane, simulators need
to be validated. Today, such validation still mainly focuses on the
fidelity of parameters such as flight performance, handling qualities,
model confirmation, and adequacy of training and checking maneuvers.
While this ensures the validity of simulators to train and check stick
and rudder skills, it does not ensure their validity for training and
checking the whole range of piloting skills, which more and more include
cognitive skills (see below).

### Shift in airline pilot training needs towards cognitive skills

Major changes, however, have occurred in the work environment of airline
pilots in the past twenty years. First, although the workload reduction
achieved by automation on the so-called glass cockpit has allowed the
elimination of the third crew member position of the flight engineer, it
has also added a fourth task, "manage systems," to the pilot's task
hierarchy to first aviate, then navigate, then communicate \[Schutte and
Trujillo, 1996\]. Second, with the increase in congestion in the global
air space, coordination with entities outside the cockpit such as air
traffic control (ATC) and company dispatchers has become more and more
critical. Third, with the increase in demand for airline pilots, the
prior jet-airplane experience of new hires at airlines has been
significantly reduced \[Carey, 1998\]. Moreover, newly hired pilots tend
to be a much more heterogeneous group then prior recruits from the
military, and it is harder to develop a curriculum that suits
everybody's needs \[Kirijan, 2001\].

Both the industry and the FAA have recognized these changes in airline
pilots' work environment and the concomitant shift of training needs
towards sharpening of cognitive skills in addition to stick and rudder
skills. By establishing the Advanced Qualification Program (AQP) \[FAA,
1990\], the FAA explicitly attempted to increase aviation safety by
responding to changes in aircraft technology, operations, and training
methodologies \[Longridge, 1997\]. AQP represents an alternative means
for training and checking of airline pilots. In contrast to traditional
training that applies equally to crew positions across airlines and
fleets (Code of Federal Regulations, Subparts N & O), AQP takes into
account both the training needs and proficiency level of a particular
airline and fleet. The training need is established via a front-end
analysis of a particular airline and fleet. After training to a set
proficiency standard within an approved curriculum, proficiency
evaluation may consist of a sampling of proficiency objectives presented
in a full mission simulation in a full cockpit crew environment. AQP
particularly demands the integration of Crew Resource Management (CRM)
skills in both training and evaluation. CRM refers to the effective use
of all available resources, including human resources, hardware, and
information. The Advisory Circular (AC) on CRM explicitly mentions the
participation of "other groups routinely working with the cockpit crew,"
such as company dispatchers, flight attendants, maintenance personnel,
and air traffic controllers, in the CRM process \[FAA, 1998, AC
120-51C\].

### Implications for simulator fidelity requirements

Both the need for full mission scenarios including emergencies in AQP
and the impending requirement to use simulators for all airline pilot
training and evaluation underscore the necessity for all airlines to
have access to effective flight simulator training and evaluation. As
mentioned above, current simulator validity checks mainly focus on a
faithful physical simulation of the airplane. Given the shift in
training needs, however, it may well be that a faithful representation
of the cognitive aspects of the flying task imposed by the air traffic
environment is just as important as the faithful representation of the
airplane. While high-level simulators generally provide some weather
information via motion and the out-the-window view, and perhaps even
some visual traffic, any representation of entities outside the cockpit
such as radio communications from ATC, other airplanes talking to ATC on
the same radio frequency, and communications from company
representatives (dispatch, maintenance, and flight attendants) is
presumably left to the creativity of each individual
instructor/evaluator in the simulator. In the framework of an overall
review of simulator fidelity requirements, the FAA has tasked the Volpe
Center with an examination of the need for providing realistic radio
communications during airline pilot simulator training and evaluation.

The following pages will summarize the results of an investigation of
current radio communication practices during simulator training and
evaluation, report the opinions solicited from airline pilot
Instructors/Evaluators (I/Es) regarding the effectiveness of these
practices, and present the findings from a literature review. The
results of an examination of Aviation Safety Reporting System (ASRS)
reports on IOE for evidence on the effectiveness of current radio
communication simulation practices will also be presented. Finally,
several airline and industry efforts to improve radio communication
simulation will be described.

## Current Radio Communication Simulation Practices

The findings summarized below are based on responses from 29 I/Es of
seven major, one cargo, four regional airlines participating in the
Advanced Qualification Program and two non-U.S. airlines (see
\[Bürki-Cohen, et al., 2000\] for more details on the study).
Respondents averaged some 10 years of pilot instruction and evaluation.
All but one of them indicated that they conducted full mission training,
i.e., Line Oriented Flight Training (LOFT). Twenty reported conducting
full mission evaluations, i.e., Line Operational Evaluations (LOEs).
I/Es were queried on their company's current practices of simulating ATC
communications and company communications to own aircraft and, for ATC,
to and from other aircraft (the so-called party line), during these
events. They were also queried regarding their company's visual and
Traffic Alert and Collision Avoidance System (TCAS) simulation of the
traffic environment.

### Radio Communications to and from own airplane

#### Communications with Air Traffic Control. During both training and evaluation events, all I/Es reported simulating ATC communications from all tower and Terminal Radar Control (TRACON) positions. This was true also for en route communications from the Air Route Traffic Control Centers (ARTCC), with the exception of two I/Es who do not simulate ATC instructions en route during training events. Slightly fewer I/Es reported simulation of Automatic Terminal Information Service (ATIS) and pre-departure clearance delivery (PDC) (89 and 93 percent during training and 95 and 89 percent during evaluation). 

#### Communications with Company. For company communications, the percentage of I/Es attempting communication simulation is lower. Company dispatch is most consistently simulated, by 93 percent of I/Es during training and 95 percent of I/Es during evaluation. Radio communications with maintenance personnel are simulated by 67 percent during training and 84 percent during evaluation. Sixty-three and 70 percent of I/Es, respectively, simulate Ramp/Gate and cabin personnel communications during both events. 

### ATC Communications to and from other aircraft

#### Simulation of other aircraft. Before querying I/Es about their simulation of communications between ATC and other aircraft on the same frequency, they were asked how other traffic is represented in their company's out-the-window view and TCAS simulation. Only 59 percent indicated that their simulators provided some out-the-window view of traffic, mostly on the airport surface (48 percent). Some indicated that they also simulated emergency vehicles on the surface. Ten I/Es (34 percent) each reported simulating visual traffic in the terminal and/or en-route environment. Ten I/Es also reported representation of traffic via TCAS.

#### Communications to and from other aircraft. Only eleven (38 percent) of all I/Es reported simulating some communications to or from other aircraft or vehicles, at least on the surface. Even fewer provide communications both ways, i.e., to and from other traffic (28 percent). Only two I/Es (less than 7 percent) provide communications to or from airborne aircraft.

### Methods of simulating radio communications

Most radio communications have to be role-played by the individual I/Es.
About a fifth of the I/Es indicated the availability of Aircraft
Communications Addressing and Reporting System for communications from
the company dispatcher and PDC. Fifty-six percent reported the use of
audio recordings or printouts for the distribution of ATIS information.

Three of seven respondents from one airline indicated that occasionally,
two I/Es may participate in a simulator session, at least when
instructing a three-person crew. One of these specified that in this
case, the I/E for the Captain and First Officer would role-play the ATC
communications, whereas the I/E for the flight engineer would provide
the company communications.

Finally, two I/Es indicated the availability of recorded or synthetic
controller voice for communications to other vehicles in the terminal
environment. Follow-up discussions with one of these revealed that he
was referring to the availability of GATES (Ground and Air Traffic
Environment Simulation) on some simulators, a new technology that will
be discussed in the section discussing industry efforts of providing
realistic radio communications below.

### Instructor/Evaluator workload in the simulator

Given that the burden of providing realistic radio communications lies
mainly with the individual I/Es, I/Es were also asked to indicate their
perception of the percentage of time and effort spent running the
simulation, simulating radio communications, instructing and observing.
As can be seen in Figure 1, I/Es spend about half of their time and
effort observing. Even during LOFT training, the time and effort spent
instructing is surprisingly small, 8 percent vs. 4 percent during LOE
evaluation (for Special Purpose Operational Training, SPOT, however, the
time and effort spent instructing shoots up to 25 percent). The rest of
I/Es' time and effort is more or less equally divided between managing
the simulator systems and providing radio communications. One I/E
indicated that his time and effort spent filling out forms and taking
notes is similar to his involvement in radio communications and
simulator management. Although this activity had not been explicitly
mentioned in the question, it is probably safe to assume that all I/Es
spend some time and effort with paper work, which may further detract
from their ability to provide a realistic radio communication
environment.

\-\-\-\-\-\-\-\-\-\-\-- Insert Figure 1 here \-\-\-\-\-\-\-\-\-\-\--

Figure 1. I/E reports of percentage of time and effort spent during Line
Operational Evaluation (LOE), Line Oriented Flight Training (LOFT) and
Special Purpose Operational Training (SPOT).

### Summary of current practices

In summary, I/Es spend about a fifth of their time and effort during
pilot training and evaluation providing radio communications, which they
almost exclusively simulate by role play. This effort is mainly spent in
the terminal environment and with ATC communications to own aircraft
that are necessary for a particular scenario. "Company communications,"
one I/E added, "are not normally used \[in simulation\]; too
time-consuming." With regard to communications to other aircraft,
another mentioned, "some instructors simulate \[them\], but none of our
formal training documentation requires it."

## oPINIONS OF iNSTRUCTOR/eVALUATORS

The same I/Es asked about airlines' current radio communication
simulation practices were queried on their perceptions of the effect of
role-playing radio communications on their own workload and on the
workload of the pilots during simulator training and evaluation. They
also offered their opinions on the importance of simulating radio
communications realistically for training and evaluation effectiveness.

### Effect of radio communication role play on I/E and pilot workload

I/Es rated their workload consistently higher in the simulator than for
training and evaluations in the actual aircraft. This applied to all ATC
environments and to all communications with company. Moreover,
role-playing radio communications "divides \[I/E's\] attention," one I/E
added. This is especially difficult for new I/Es, another mentioned. A
third added that the "I/E can become task-saturated when crew works two
VHF radios and/or communications with cabin simultaneously." The highest
discrepancy in I/E workload was indicated for simulation of
communications with other aircraft.

Pilot workload in the simulator, however, was rated consistently lower
than in the actual aircraft for all ATC environments and company
communications, and also for listening to the party line. "I/E
\[communication simulation\] is less than actual, therefore it reduces
pilot workload," one I/E explained. Another I/E alluded to the fact that
even the manual workload of pilots is reduced by I/E role play of radio
communications, because "\[p\]ilots are not normally given a chart
frequency, nor do they need to redial a new frequency to communicate."

### Importance of radio communications for training and evaluation effectiveness

I/Es were asked about the importance of radio communications for
training and evaluation effectiveness in two contexts, first in the
context of their company's communication practices, then in the context
of specific training/evaluation goals.

Some I/Es may have downgraded the importance of radio communications in
the context of their own company's practices, because they feel that
their company's communication simulation is "not very effective during
simulation, because the instructor must cover all bases himself," as one
I/E explicitly stated. Nevertheless, as many as 73 percent of
respondents rated the overall importance of ATC communication simulation
in the context of their company's practices as high or very high (for
communications with TRACON). (The corresponding percentages for the
other ATC communications were, in descending order, 68 for tower ground,
65 for tower local, and 61 percent for ARTCC.) No more than 8 percent of
the respondents rated any of the ATC communications as of low
importance. None of the ATC communications were rated as of very low
importance.

Fewer I/Es rated the importance of company communications in the context
of their company's practices high or very high. Communications with
cabin personnel achieved the highest rating with 57 percent considering
it highly or very highly important. (The corresponding percentages for
dispatch and ramp/gate were 43 and 27, respectively.) Up to 15 percent
of I/Es rated any of the indicated company communications as of low or
very low importance.

Despite the limitations of their own company's practices, as many as 84
percent of the I/Es considered ATC communications to other aircraft
medium to highly important. Sixteen percent gave it low importance in
the context of their own practices. However, several I/Es commented on
the importance of radio communications to other aircraft for the realism
of the simulation. "I believe the 'simulator mind-set syndrome' must be
fought with realism. How can we expect crews to 'treat the sim\[ulator\]
like the aircraft' when the audio environment belies the condition so
often?" one I/E asked. "Party line enables CRM elements such as workload
and distraction to be assessed more effectively," added another. One
last I/E mentioned that the simulation of ATC communications to other
aircraft is his "biggest concern, so pilots are listening."

These opinions that radio communications are important were confirmed in
the context of specific training (or evaluation) goals. I/Es indicated
how often they relied on radio communications and how important they
found radio communications for effective training (and evaluation).

I/Es indicated relying most often on radio communications for training
CRM and non-routine ATC, such as pilot-ATC coordination, where 93
percent of the respondents indicated that they use it at least
sometimes. The importance of radio communications for effectively
training CRM and non-routine ATC was rated as high or very high by 89
and 86 percent of the I/Es, respectively. "If CA\[ptain\] does not
delegate duties, my technique is to load the crew with B.S. \[*sic*\]
radio transmissions," one I/E added.

Radio communications are also very important for training and evaluating
distraction management skills. Eighty-two percent of I/Es rely on radio
communications for this purpose at least sometimes, with 78 percent
rating their importance as high or very high. They are used to train and
evaluate situation awareness skills, where 88 percent of I/Es reported
using them at least sometimes, with 68 percent rating their importance
as high or very high. For training and evaluation of new ATC procedures,
such as simultaneous approaches to closely spaced parallel runways, 92
percent of I/Es responding to this question indicated relying on radio
communications at least sometimes, and 62 percent rated the importance
of radio communications as high or very high.

I/Es overall concern with simulating radio communications may have best
been summarized by the I/E who stated: "Without communication
simulation, when the pilot trainee finally arrives in the 'real world,'
he must add another component, which was not learned during training.
This new (additional) component can really complicate line flying."

## LITERATURE REVIEW

This section presents the results of a review of the Advanced
Qualification Program/Crew Resource Management and the task management
training literature. It was found that many of the subject matter expert
opinions found in the previous section are confirmed in the literature
(for more detail, see \[Bürki-Cohen et al., 2000\]).

### Advanced Qualification Program and Crew Resource Management

CRM training and evaluation is an integral part of AQP, which emphasizes
both technical *and* cognitive skills \[Longridge, 1997\]. CRM skills
include leadership, communication skills, time management, situational
awareness, and attitudes in flight operations \[FAA, 1990, AC 120-35B\].
As mentioned in the introduction, its importance has been growing over
the last decades due to the increase in flight deck automation and the
concurrent reduction in crew size, as well as the increase in congestion
of the airspace.

The founding principle of AQP is that training and evaluation of pilots
should be based on the activities encountered on the job. Both the AQP
and CRM literatures underscore the importance of realistic scenarios.
AQP requires a thorough analysis of all tasks a pilot needs to perform
during actual operations, which then guides curriculum and scenario
development. The AQP task-listing example found on the FAA AQP
Management Website \[FAA, 2000\] clearly shows that coordination with
company and ATC over the radio frequencies is an integral part of line
operations and that frequency monitoring is important for maintaining
traffic and weather situation awareness.

The Advisory Circular on CRM training explicitly not only lists onboard
flight deck and cabin personnel, but also ground-based maintenance
personnel, aircraft dispatchers, and air traffic controllers as part of
the CRM process \[FAA, 1998, AC 120-51C, Crew Resource Management
Training, CRM\]. In a section on extending training beyond the flight
deck, the AC highlights the benefits of using real air traffic
controllers, dispatchers, and maintenance personnel during full mission
simulation training.

A 1996 Air Line Pilot article concluded that pilot confusion can "best
be prevented through continuing emphasis on crew performance, with the
understanding that ATC is a key member of the flight team" \[Rosenthal
et al., 1996\]. The authors had surveyed 100 ASRS reports explicitly
mentioning pilot confusion, many of them involving amended ATC
clearances, and recommend that CRM place greater emphasis on crew-ATC
interactions.

Many ASRS surveys, research investigations, and aviation magazine
articles discuss the role of communications in incidents and accidents.
The Flight Safety Foundation Approach and Landing Accident (ALA)
Reduction Task Force report found that "incorrect or inadequate ATC
instruction/advice/service" was a causal factor in 33 percent of the 76
ALAs and serious incidents analyzed (Khatwa and Helmreich, 1999). It
ranked eleventh among the most common causal factors, long before
"interaction with automation" in seventeenth place. "\[D\]emanding ATC
clearances" are also explicitly mentioned in context with even higher
placed causal factors such as the eighth placed "press-on-itis." In many
cases of "press-on-itis," "a breakdown in CRM between the flight crew
and ATC" was observed.

Other ATC-communications-related causal factors in the ALA reduction
task force report that are not included in the 33 percent mentioned
above are misunderstood or missed communications such as missed read
backs, call sign confusions, and simultaneous transmissions (12
percent). Instances of controllers and crews using non-standard
phraseology are also mentioned. This can become especially problematic
when non-native English speakers/listeners and an emergency situation
are involved, as shown by the 1990 Avianca Airlines crash on Long Island
and the 1995 American Airlines crash near Cali, Columbia \[National
Transportation Safety Board, 1991; Simmon, 1998\].

In summary, the ALA Reduction Task Force recommends that operators
"\[i\]nclude training scenarios that allow crews to experience overload,
task saturation, loss of situational awareness, out-of-control and
too-far-behind-the-aircraft situations, and communications in stressful
circumstances." Joint training should be held between pilots and air
traffic controllers including scenarios that "promote mutual
understanding of issues on both the flight deck and in the ATC
environment, and foster improved communications during emergency
situations."

### Cockpit task management training

The need for cockpit task management (CTM) training has been documented
not only using incident and accident reports, but also experimentally.
Chou, Madhavan, and Funk \[1996\] elicited the CTM errors found in an
accident and incident review in a controlled simulator experiment and
confirmed that task prioritization is greatly degraded by the number of
concurrent tasks. Another study found that ATC interruptions
significantly increased procedure performance errors as well as
flight-path management workload of commercial airline pilots
\[Latorella, 1996\].

The question then is whether CTM training in the simulator is effective.
Gopher, Weil, and Bareket \[1994\] showed that task management trained
even in a very low physical-fidelity computer game did transfer to
flight. Gopher et al. report that the computer game was perceived as
such a successful auxiliary training tool that the Israeli Air Force
incorporated it into their curriculum.

Given that CTM training can be effectively trained in a synthetic
environment, the next question is what are the best methods for such
training. One of the I/Es queried in the study presented earlier
indicated that communications are "most important at the end of
training," implying that pilots may not be able to handle radio
communications in addition to aviating during early training. He seems
to favor part-task training, where trainees are only gradually
introduced to the full complexity of aviating, navigating,
communicating, and managing systems.

There is much theoretical and experimental evidence, however, that
whole-task training in a fully loaded environment is superior to
part-task training in an incomplete environment that may induce a false
sense of operational simplicity akin to tunnel vision. The Gopher et al.
study mentioned above also compared different training regimes, one a
hierarchical part-task technique where trainees worked their way up
through part-task games of increasing complexity to exposure to the full
game, the other a whole-task technique. For the whole-task training,
emphasis was shifted between different aspects of the game in subsequent
training trials. Although the part-task group outperformed the
whole-task group when tested on the game, there was no difference
between the groups in actual flight performance. Fabiani, Buckley,
Gratton, Coles, Donchin, and Logie \[1989\], extended these results by
finding that the whole-task group outperformed the part-task group even
in the game when additional tasks were added, but only when trained with
the shifting-emphasis technique.

These results bolster the theoretical argument that a complete mental
model of the task needs to be acquired during training \[Bransford and
Franks, 1976; Mangold and Eldredge, 1993\]. Learning is an active
process, and practice can lead to either activation or inhibition of
cognitive pathways. If pilots are consistently exposed to an
impoverished environment during training compared to the real world,
they may end up unprepared for the complexity of flying in the air,
where "he must add a new (additional) component" that "can really
complicate line flying," as one of the I/Es in the study presented
earlier admonished.

## Effect on initial Operating Experience in the air

A review of the Aviation Safety Reporting System (ASRS) was conducted to
see whether the concerns expressed both by I/Es and in the literature
would be confirmed by reports of line check airmen (LCA) and pilots on
events during their initial operating experience in the air.

### Method

A search of the 205,070 reports in the ASRS database, which was
established in 1988, up to October 1999 \[ASRS, 2001\] using the terms
Initial Operating Experience, IOE, and Operating Experience yielded 423
reports after exclusion of duplicates. After exclusion of reports
referring to incidents that had occurred prior to 1993 or not during
IOE, or reports covering non-flying issues, 93 reports remained for
analysis. The authors determined types of errors and primary and
contributing factors based on the ASRS reporters' narrative. Most of the
errors involved several factors.

### Types of errors reported

More than a third (32) of the 93 reports involved altitude deviations or
crossing restriction violations, including two near controlled flights
into terrain. Course deviations were mentioned in 12 of the reports.
Legal separation between airplanes was lost in six of the reports,
including three near-midair collisions.

Landings without clearance were the subjects of 11 reports. Seven
reporters approached the wrong runway, including one who actually
landed. Four crews, of which one landed, approached the wrong airport.
Two crews took off without clearance. Runway incursions were reported
six times.

Communications were lost three times. The remaining 10 errors were
reported each only once.

### Factors contributing to errors

#### General factors. Several reports appeared to indicate that training is not completed when pilots transition from the simulator to the airplane. In almost a quarter of the reports, the fact that the LCA was administering remedial training at least contributed to the incident. One LCA explained a course deviation by his becoming "preoccupied with instructing the first officer to give passenger briefing."

Insufficient stick and rudder skills played a role in 15 percent of the
reports. "\[The pilot's\] unfamiliarity with the actual aircraft
performance and the quickness of the events" was given as the reason for
an altitude deviation.

Violation of standard operating procedures played a role in almost 8
percent of the reports. A pilot who had failed to establish contact with
ATC as requested by the standard taxi clearance explained that the
"controller's workload was very high with a lot of frequency
congestion."

Other factors cited included equipment failures, which contributed to 8
percent of the reports. Problems with the automation contributed to 11
percent. Weather contributed to 20 percent of the reports, and fatigue
to 9 percent.

#### Radio Communication Factors. Radio communications contributed to as many as 87 percent (81) of the reports, and was the primary factor in 72 percent (67). This clearly demonstrates the importance of radio communication training. Figure 2 shows the number and types of errors where radio communication problems played a primary role.

\-\-\-\-\-\-\-\-\-\-\-- Insert Figure 2 here \-\-\-\-\-\-\-\-\-\-\--

Figure 2. Number and types of errors with radio communications as
primary factor.

The radio communication factors contributing to errors are shown in
Figure 3. Demanding, inadequate, or even erroneous ATC instructions were
implicated most often as a primary or (23) or contributing (15) factor.
Reporters often cited amended clearances requiring reprogramming of the
automation or erroneous expectations raised by the controllers. After an
instruction to "expect no delays," e.g., the crew "perceived that there
would be no delay at the end" and taxied on an active runway.

\-\-\-\-\-\-\-\-\-\-\-- Insert Figure 3 here \-\-\-\-\-\-\-\-\-\-\--

Figure 3. Radio communication factors contributing to reports.

Inadequate CRM or CTM involving radio communications played a role in 40
reports (21 primary, 19 contributing). "The cause \[...\] was my
inexperience with the quick pace of an airline environment and its
associated distractions," explained a pilot after deviating from the
assigned altitude. Another explained a near-midair collision during an
approach to the wrong runway by the crew being "so busy that we were not
paying attention to what the controller was saying." An LCA after
landing without clearance blamed the fact that IOE is comparable to
"flying 'single pilot with a distraction'," explaining the error by
being distracted by the trainee who didn't contribute much else to
flying.

ATC interruptions including traffic calls were mentioned in 14 reports,
and in eight of these they appeared to be the primary reason. After a
crew missed an instruction to clear the runway, the pilot complained
"tower controllers \[...\] give instructions \[...\] while the aircraft
is still in a critical phase."

Frequency congestion, stuck microphones blocking an entire frequency, or
pilots stepping on an ongoing conversation played a role in 14 reports
as well, but were the primary reason in only five reports. After a
course deviation due to a misunderstanding, the pilot reported, "several
aircraft were stepping on each other's radio calls."

Problems with tuning the radio played a role in four reports and was the
suspected primary reason for each of the errors, such as in a runway
incursion after loss of communications where the "F\[irst\] O\[fficer\]
possibly moved \[the\] radio select switch from tower to \[the\] other
side in \[an\] attempt to contact ground control prior to selecting
\[the\] frequency."

Phraseology and/or accent contributed to seven reports. They appeared to
be the primary reason in three of these, such as in an approach to the
wrong airport in Mexico where "the transmissions and comm\[unication\]s
from the tower were exceedingly hard to understand and \[we\] had to ask
several times for clarification."

Interruptions from the cabin, be it from flight attendants or
passengers, played a role in seven reports, being the primary reason in
three. One pilot reported a near-midair collision and mentioned that "as
\[the\] clearance was coming off the printer, \[the\] F\[light\]
A\[ttendant\] entered \[the\] cockpit for meal orders."

No comparison has been made to determine whether radio communication
related problems occur more frequently in IOE than during non-IOE
flights (see literature review). Also, the overall incidence of such
occurrences cannot be determined from ASRS reports, which are naturally
biased towards cases where something did happen. Nevertheless, improved
realism of radio communication simulation during simulator training may
have better prepared pilots for many of these IOE occurrences.

### Effect of pilot experience on radio communication problems during IOE

One of the I/Es queried in the study presented earlier in this paper
expressed his opinion that experienced airline pilots have "proven their
abilities" to handle radio communications, implying that airline pilots
may require radio communication training and evaluation only early in
their career. The ASRS reports do not consistently provide a direct
means to test this hypothesis, such as flight time of the pilot flying,
but may indirectly indicate the experience level of crews by providing
the weight class of the airplane involved. The assumption is that in
general, pilot experience increases with airplane weight.

If radio communication problems decrease with experience
disproportionately to other problems, then the weight class distribution
in the IOE reports with communications problems should be different from
the weight class distribution of reports involving airline pilots in the
overall database, resulting in a low correlation between the two
distributions.

In fact, the opposite was found. At a ratio of .97, the airplane weight
class distribution in the sample of IOE ASRS incidents where
communications were the primary factor correlated very highly with the
airplane weight class distribution in the overall airline reports.
Similarly high were the weight class distribution correlations of all
airline reports with all IOE reports as well as all IOE reports where
communications played at least a contributory role (.95 each).

This analysis shows that there is no difference between the
distributions of airplane types represented in the overall ASRS database
and in the samples reporting communication problems. Therefore, prior
experience with communications in different airplanes does not seem to
protect airline pilots from experiencing problems with radio
communications during IOE.

## Industry initiatives

The previous sections have shown that subject matter expert opinions,
accident and incident reports, and the AQP, CRM and academic literatures
concur that realistic radio communications are important. It is thus not
surprising that airlines and simulator manufacturers have made several
efforts to improve the realism of radio communications in simulator
events. Although the following pages will describe only examples of
airline efforts to include radio communications into training and
evaluation of pilots, it is interesting to note that even the
manufacturers of personal-computer (PC) flight-simulator systems see a
market value in offering realistic radio communication add-ons to their
products. Typical features of these systems include recorded natural
speech ATC instructions to own airplane (with limited response
capability) and communications to and from invisible other aircraft
consisting of random chatter with no relevance for traffic or weather
awareness (see Bürki-Cohen et al., \[2000\], for more detail on PC and
other supporting technologies).

### United Airlines' Interactive Real Time Audio System (IRAS) 

The most comprehensive effort to provide automated realistic radio
communications in the simulator is United Airlines' (UAL) Interactive
Real Time Audio System (IRAS), also known as "Chatter Program." It is an
in-house development with very high operational realism. It is based on
field recordings of actual ATC communications on UAL routes. By dubbing
ATC with the respective I/E voice, the I/E can intervene without the
pilot trainees realizing it. The engineers at United attempted to
include many subtle nuances of audio communications into the
environment, including demanding timing imposed by ATC, frequency
congestion, foreign and regional accents, stuck microphones blocking an
entire frequency, and meaningful communications to and from other
aircraft to train pilots to listen.

The system, however, encountered many technical difficulties. Algorithms
coded to trigger radio communication recordings at appropriate times did
not always function adequately, forcing an embarrassed instructor to
intervene, thus increasing instructor stress and workload. Operations in
dense terminal areas also taxed the scenario algorithms, with normal
variations in crew response causing additional timing problems and
workload for the instructor.

The IRAS interface was often not well integrated into the instructor
station and was hard to use. Different instructor interfaces across
simulator models often made it even harder for instructors to operate
the system. The cost and difficulty of system-wide implementation was
further hampered by different simulator designs with different visual,
audio, and navigation models.

All of the above in combination with expensive scenario development
including field recording, transcribing, dubbing, database maintenance,
as well as costly route, sector map, and simulator-interface code
development contributed to the program gradually losing support. As of
the end of the 1990s, the program had been scaled back to use only one
generic ATC voice, and it was only used for new-hire screenings in a
B-727 flight-training device. The applicants were presented with
relevant communications to and from other aircraft to evaluate their
planning abilities during an approach, as well as automated frequency
changes. New-hire evaluators reportedly complained when the system was
down because of the increased workload of providing frequency changes.

The lessons learned from IRAS are that for a system to be successful, it
must be flexible, transparent, easy to use, easy to implement and
maintain, and easy to integrate with different scenarios, simulators,
and simulator systems (e.g., visual, audio, etc.). There may be a
cost/benefit trade-off for the different aspects of realism required for
different training and evaluation events that must be examined in this
research. For instance, it may not be necessary to conceal instructor
intervention from the pilots, a capability achieved by IRAS, but at a
high price.

### Ground and Air Traffic Environment System (GATES)

Airlines have advanced the state of the art by specifying requirements
to the simulator manufacturers. The Canadian simulator manufacturer's
CAE product GATES was developed after a request from an airline to
provide a visual representation of traffic in the airport terminal
environment. It soon became obvious to the developers and the airline
that correlated and meaningful radio communications would have to be an
essential component of such traffic representation.

Several U.S. airlines and training facilities as well as non-U.S.
airlines and military are currently installing GATES-equipped
simulators. The product as it is today provides simulated aircraft
traffic and associated relevant communications to and from aircraft on
the airport and in the terminal environment. The product also provides
visual simulations of ground traffic such as emergency vehicles.

GATES-simulated vehicles do not follow scripted scenarios. A continuous
flow of arriving and departing traffic is generated. The traffic
elements are aware of and will react to each other and the simulated own
airplane. The I/E has limited ability to control specific situations
like runway incursions and emergency vehicles. The I/E still provides
all ATC communication to own airplane, however.

The main reason a U.S. airline cited for choosing GATES was to add
realism and increase distraction factors for LOFT sessions.
Specifically, they wanted to enhance TCAS scenarios, ground hazard
avoidance, and runway incursions. A senior check airman responsible for
GATES implementation stated that the improved overall sense of realism
alone justified its purchase and support costs. Workload increase for
the I/E is reportedly minimal. He added that interactive/reactive
communications capability would enhance usability and realism.

### Lufthansa's Joint Operational Incidents Training (JOINT)

A last effort to expose pilots to realistic radio communications in the
simulator to be described here is the German airline Lufthansa's Joint
Operational Incidents Training (JOINT) with the German ATC organization,
Deutsche Flugsicherung (DFS) \[Hensel, 2000; Jung, 1999; Lexen, 1999;
Strassburger and Novack, 1997\]. JOINT started in 1996 with the
connection of one DFS ATC simulator representing one control sector with
pseudo traffic to one B-737 simulator at Lufthansa Flight Training's
facilities in Frankfurt. The purpose was to expose crews to abnormal
procedures in a realistic ATC environment.

The program was expanded in 1998 to encompass two DFS ATC simulators
representing two control sectors with pseudo traffic that can be
connected to up to eight full flight simulators representing the entire
Lufthansa fleet (B-737, B-747, A310/300, A320, A340). Each ATC simulator
consists of a controller work station with a radar display showing the
simulated airplanes flown by the flight simulator crews as well as other
airplanes sharing the same airspace operated by a pseudo pilot sitting
at a connected computer station. The flight crews and the pseudo pilots
communicate with the sector controllers via two VHF frequencies.

JOINT training takes place during recurrent training. Two controllers
providing approach/departure and/or area ATC communications and two
pseudo pilots providing pseudo traffic that can be seen on the
radarscope and heard on the respective frequencies represent DFS.
Lufthansa is represented by an average of four crews and their
instructors. While the DFS controllers and pseudo pilots provide
approach/departure and/or area communications to and from other
aircraft, Lufthansa instructors still have to provide tower
communications. Lufthansa also maintains the simulators.

For the past two years, JOINT sessions took place weekly during a
four-hour simulator session (due to a controller shortage at DFS, this
has been reduced to once a month in January 2001) \[Hensel, 2001\].
Lufthansa and DFS jointly define the scenarios, which include abnormal
procedures and emergencies such as fuel leaks, jammed flaps, engine
failures on take-off, degraded navigation capabilities, hydraulic
failures, etc. Each simulator is presented with one or more of these
problems at a predefined point in time unknown to the participating
flight and controller crews. While one simulator crew may be in the
take-off phase, two may be controlled by departure and one or two more
by arrival, all while pseudo traffic is sharing the same frequencies and
airspace. Each scenario lasts about two hours and is followed by a
thorough joint debriefing of air traffic controllers and flight crews.

JOINT was conceived as part of Lufthansa's shift in training from pilot
flying oriented maneuver training towards CRM oriented LOFT, where CRM
involves not only the flight crew, but also coordination between pilots
and controllers and controllers and controllers. Both controllers and
pilots are enthusiastic and feel that JOINT realistically prepares them
for the coordination and concentration demands of real-life emergencies
with all their uncertainties and distractions. Moreover, both find the
insight into the realities of each other's jobs invaluable.

Given a Lufthansa/DFS shared development cost of about \$110,000
\[Lexen, 1999\], Lufthansa anticipates important economic benefits from
its pilots' increased understanding of ATC management and thus optimized
operational decision making capabilities \[Hensel, 2001\]. Lufthansa/DFS
are discussing further enhancements of the system, such as
synchronization of the auditory traffic environment with the
out-the-window view and the TCAS displays. This would permit increased
training credit in the simulator.

## Summary and Conclusions

The principal issue examined in this paper was whether realistic radio
communication simulation is necessary to ensure full transfer of
performance and behavior from the simulator to the airplane (for
training) and from the airplane to the simulator (for evaluation). This
issue is especially pertinent in view of the impending regulatory
changes making the simulator the sole tool for airline pilot training
and evaluation. Other factors contributing to its relevance are the
growing complexity of the global airspace, the shrinking pool and
increasing heterogeneity of new-hires, and the increase in automation
and concomitant reduction in crewmembers. All this dramatically
increases especially the cognitive demands on the remaining crew. It is
therefore critical that the simulator environment poses the same task
demands pilots encounter in the air to ensure the safety of their future
passengers.

A review of current airline practices, however, showed that radio
communications transmitted over the ATC and company frequencies are not
consistently simulated, thus reducing the workload of pilots during
training and evaluation. This is especially true for company
communications and for communications to other aircraft (party line).
This is mainly because the simulation of radio communications is left to
the individual I/Es, who have at most one fifth of their time left to
role-play radio communications after observing, operating the simulator,
instructing, etc. The consequences of this minimal consideration of
radio communications during training and evaluation are that close to 90
percent of incidents during IOE reported to the ASRS system involve
radio communications at least as a contributing factor.

The I/Es themselves and the literature emphasize the importance of
realistic radio communications to achieve full transfer of performance
and behavior between the simulator and the airplane and thus ensure
safety. Even if expanded data link capabilities should succeed in
replacing some voice communications, the basic issue of the importance
of simulating realistic communications-related cognitive demand remains
unchanged. In some ways, data link may actually increase the challenges
to the pilot in that arena (see, e.g., \[Latorella, 1998\]).

The initiatives by UAL, CAE, and Lufthansa/DFS demonstrate that both the
airline and the simulator industry have recognized the potential
increase in safety and thus economic value of providing realistic radio
communications to some degree, especially for training and evaluating
line-oriented full-flight and operational incident scenarios.

Despite of all this evidence that increasing the realism of radio
communications may improve transfer of performance and behavior between
simulator and airplane and thus overall training and evaluation
effectiveness of simulators, the authors know of no official regulations
that would render the simulation of realistic radio communications
mandatory. Much collaboration of government, industry, military and
academia will be required, both in the area of improving technologies
supporting realistic radio communication simulation and in demonstrating
an objective safety and economic value before either market forces or
regulatory requirements will lead to a general improvement of radio
communication realism in airline pilot simulator training.

## Acknowledgments

This work was performed as part of an ongoing research program at the
U.S. Department of Transportation's John A. Volpe National
Transportation Systems Center (Volpe Center) investigating fidelity
requirements for simulators used in training and evaluation of airline
pilots. The research is supported by the FAA Office of the Chief
Scientific and Technical Advisor for Human Factors where we thank Dr.
Eleana Edens, the program manager, for her effective guidance and
assistance. Dr. Thomas Longridge of the Advanced Qualification Program
determined the need for this work and provided many insights.

We would like to thank the instructors/evaluators of the Advanced
Qualification Program Instructor/Evaluator Focus Group who shared their
experiences and opinions with us. Captain Bill Hamman and colleagues
provided demonstrations and discussions on United Airlines' Interactive
Real Time Audio System (IRAS). CAE's Mr. Jean-Sébastien Lavoie provided
information on CAE's Ground Air Traffic Environment System (GATES). We
thank all of them. Errors and opinions, however, are ours.

## acronyms and abbreviations

AC Advisory Circular

AQP Advanced Qualification Program

ASP Advanced Simulation Plan

ALA Approach and Landing Accident

ARTCC Air Route Traffic Control Center

ASRS Aviation Safety Reporting System

ATC Air Traffic Control

ATIS Automatic Terminal Information Service

CRM Crew Resource Management

CTM Cockpit Task Management

DFS Deutsche Flugsicherung

FAA Federal Aviation Administration

GATES Ground and Air Traffic Environment System

I/E Instructor/Evaluator

IOE Initial Operating Experience

IRAS Interactive Real Time Audio System

JOINT Joint Operational Incidents Training

LCA Line Check Airman

LOE Line Operational Evaluation

LOFT Line Oriented Flight Training

LOS Line Oriented Simulation

PC Personal Computer

PDC Pre-departure Clearance

SPOT Special Purpose Operational Training

TCAS Traffic Alert and Collision Avoidance System

TRACON Terminal Radar Control

UAL United Airlines

U.S. United States

VHF Very High Frequency

## references

Aviation Safety Reporting System (2001), *National Aeronautics and Space
Administration,* Moffett Field, CA \[On-line\]. Available:
[http://nasdac.faa.gov/asp/asy_asrs.asp]{.underline}.

Bransford, J. and Franks, J. (1976), "Towards a Framework for
Understanding Learning," *The Psychology of Learning and Motivation,*
10: 93-127.

Bürki-Cohen, J., Kendra, A., Kanki, B., and Lee A. (2000*), Realistic
Radio Communication in Pilot Simulator Training.* Final Report No.
DOT/FAA/AR-00/62, Washington, DC.

Carey, S. (1998), "Demand for Pilots is Soaring as Old-Timers Take Off,"
*The Wall Street Journal,* June 4.

Chou, C., Madhavan, D., and Funk, K. (1996), "Studies of Cockpit Task
Management Errors," *The International Journal of Aviation Psychology,*
6: 307-320.

Fabiani, M., Buckley, J., Gratton, G., Coles, M., Donchin, E., and
Logie, R. (1989), "The Training of Complex Task Performance," *Acta
Psychologica,* 71: 259-299.

Federal Aviation Administration (1980), *Advanced Simulation Plan,* 14
C.F.R. Part 121, Appendix H, Department of Transportation, Washington
DC.

Federal Aviation Administration (1990), *Advanced Qualification
Program,* 14 C.F.R. Part 121, SFAR 58, Department of Transportation,
Washington DC.

Federal Aviation Administration (1990), *Crew Line Operational
Simulations: Line Oriented Flight Training, Special Purpose Operational
Training, Line Operational Evaluation,* Advisory Circular No. 120-35B,
U.S. Department of Transportation, Washington DC.

Federal Aviation Administration (1996), *Crewmember Qualifications,* 14
C.F.R. Part 121, Subpart O, Department of Transportation, Washington DC.

Federal Aviation Administration (1997), "FAA One Level of Safety
Commuter Rule Deadline Marks Safety Improvements, Decline in Accident
Rate," *FAA News,* APA 44-97, March 21 \[On-line\]. Available:
[http://www.dot.gov/affairs/1997/apa4497.htm]{.underline}.

Federal Aviation Administration (1998), *Crew Resource Management
Training*. Advisory Circular No. 120-51C, U.S. Department of
Transportation, Washington, DC.

Federal Aviation Administration (2000), *Advanced Qualification
Program,* AFS-230 Web site: <http://www.faa.gov/avr/afs/aqphome.htm>.

Gopher, D., Weil, M., and Bareket, T. (1994), "Transfer of Skill from a
Computer Game Trainer to Flight," *Human Factors,* 36: 387-405.

Hensel, D. (2000), "The A340 Recurrent Training: A New Philosophy,"
*Proceedings of the International Conference Flight Simulation\--The
Next Decade, Royal Aeronautical Society,* London, UK*.*

Hensel, D. (2001), e-mail, February 2001.

Jung, S. (1999), "Gemeinsam den Notfall trainieren," *transmission,*
DFS, März.

Kirijan, F. (2001), Personal Communication, January 24, Tempe, Arizona.

Khatwa, R. and Helmreich, R. (1999), "Analysis of Critical Factors
During Approach and Landing in Accidents and Normal Flight," *Flight
Safety Digest*, Flight Safety Foundation, Alexandria, VA.

Latorella, K. (1998), "Effects of Modality on Interrupted Flight Deck
Performance: Implications for Data Link," *Proceedings of the Human
Factors and Ergonomics Society 42nd Annual Meeting,* Santa Monica, CA.

Latorella, K. (1996), "Investigating Interruptions: An Example from the
Flightdeck," *Proceedings of the Human Factors and Ergonomics Society
40th Annual Meeting,* Santa Monica, CA.

Lexen, G. (1999), "Flight Simulator and ATC Simulator Coupling," *5th
Annual International Flight Simulator Engineering and Maintenance
Conference,* Denver, Colorado \[On-line\]. Available:
http://www.arinc.com/fsemc/99_fsemc_presentations.html.

Longridge T. (1997), "Overview of the Advanced Qualification Program,"
*Proceedings of the Human Factors and Ergonomics Society 41^st^ Annual
Meeting,* Santa Monica, CA.

Mangold, S. and Eldredge, D. (1993), "An Approach to Modeling Pilot
Memory and Developing a Taxonomy of Memory Errors," *Proceedings of the
Ohio State University Seventh International Symposium on Aviation
Psychology,* Columbus, OH.

National Transportation Safety Board (1991), *Aircraft Accident
Report-Avianca, The Airline of Colombia, Boeing 707-321B, HK 2016, Fuel
Exhaustion, Cove Neck, New York.* Report PB91-910404, Washington DC.

Rolfe, John (1996). Twenty-five Years of Flight Simulation - or - It\'s
a Long Time Since I Lost a Buddy in a Training Accident. The Royal
Aeronautical Society \[On-line\]. Available:
<http://www.raes.org.uk/fl-sim/fsg25yer.htm>.

Rosenthal, L., Chamberlin, R., and Matchette, R. (1996), "Flight Deck
Confusion: A Survey of ASRS Reports Indicates That Better Communication
Between Cockpit Crews and ATC Could Help Prevent Confusion-Related
Incidents," *Air Line Pilot.* Vol. 65.

Schutte, P. and Trujillo, A. (1996), Flight Crew Task Management in
Non-Normal Situations, *40^th^ Annual Meeting of the Human Factors and
Ergonomics Society*, Philadelphia, PA.

Simmon, D. (1998), "Boeing 757 CFIT Accident at Cali, Colombia Becomes
Focus of Lessons Learned," *Flight Safety Digest*, Flight Safety
Foundation, Alexandria, VA.

Strassburger, A., and Nowack, S. (1997), "Training Connection,"
*transmission,* DFS, März.

## biographies

**Judith Bürki-Cohen** is the manager of the Flight Deck Human Factors
Program of the Operator Performance and Safety Analysis Division of the
U.S. Department of Transportation\'s Volpe Center. Her program focuses
on both flight simulator fidelity requirements and flight deck
technology. Previously, she has worked in the areas of air traffic
control, high-speed rail, and speech perception. She is a member of The
National Academies Transportation Research Board Committee on Simulation
and Measurement of Vehicle and Operator Performance and the American
Institute of Aeronautics and Astronautics Modeling and Simulation
Technical Committee. She is also a member of the Association of Aviation
Psychologists and the Human Factors and Ergonomics. She holds a Ph.D. in
Experimental Psychology and post-baccalaureate degrees in German,
French, English and Italian.

**Andrew J. Kendra** is a computer specialist at the Volpe Center
Operator Performance and Safety Analysis Division's Flight Deck Human
Factors Research Laboratory. He conducts experiments and develops
software and hardware for part-task flight simulations. He has a
commercial pilots certificate with flight instructor instrument airplane
ratings. Before coming to the Volpe Center he worked as a flight
instructor and research engineer.

![](media/image1.wmf)Figure 1.

![](media/image2.wmf)Figure 2.

Figure 3.

![](media/image3.wmf)