AIAA-2001-4425

# Flight Simulator Fidelity Considerations for Total Air Line Pilot Training and Evaluation

Judith Bürki-Cohen

Volpe National Transportation Systems Center

U.S. Department of Transportation Research and Special Programs
Administration

Cambridge, Massachusetts

Tiauw H. Go

Massachusetts Institute of Technology, Cambridge, Massachusetts

Thomas Longridge

Federal Aviation Administration, Washington, D.C.

## ABSTRACT[^1]

This paper presents the FAA/Volpe Center's Flight Simulator Fidelity
Research Program, which is part of the Federal Aviation
Administration\'s effort to promote the effectiveness, availability and
affordability of flight simulators. This initiative will become
increasingly critical with the anticipated regulatory changes mandating
the use of simulators in airline pilot training and evaluation,
dramatically reduced pilot new-hire experience levels and growing
operational complexity. Two research areas with high pay-off potential
for this effort are radio communications and platform motion simulation.
Initial results suggest that for fully effective training and evaluation
of the cognitive and workload management skills associated with radio
communications, significant improvements in radio communications realism
are needed. Initial research on the training effectiveness of a
fixed-base simulator with a wide field-of-view visual system compared to
a like system having platform motion failed to find an operationally
significant effect of motion. Follow-up work will examine whether this
result was a function of the motion characteristics or the maneuvers
tested. No changes in regulatory requirements can be expected without
absolute confidence in the reliability and validity of the results,
requiring considerable additional research in both areas.

## Notations

AC Advisory Circular

AQP Advanced Qualification Program

ASP Advanced Simulation Plan

ASRS Aviation Safety Reporting System

ATC Air Traffic Control

CRM Crew Resource Management

DFS Deutsche Flugsicherung (German ATC)

DOF Degrees Of Freedom

FAA Federal Aviation Administration

FOV Field Of View

GATES Ground Air Traffic Environment System

I/E Instructor/Evaluator

IOE Initial Operating Experience

IRAS Interactive Real Time Audio System

JOINT Joint Operational Incidents Training

NAS National Airspace System

PF Pilot Flying

PNF Pilot Not Flying

NPRM Notice of Proposed Rulemaking

PTS Practical Test Standards

RTO Rejected Take-Off

SFAR Special Federal Aviation Regulation

SME Subject Matter Expert

TCAS Traffic Alert and Collision Avoidance System

V~1~ Take-off decision speed; the minimum speed in the take-off,
following a failure of the critical engine, at which the pilot can
continue the take-off and achieve the required height above the take-off
surface within the take-off distance.

V~1~ cut Engine failure at or above the take-off decision speed, V~1~,
with continued take-off

V~R~ cut Engine failure at or above the rotation speed, V~R~, with
continued take-off

## Introduction

This paper gives an overview of the Federal Aviation Administration\'s
(FAA) initiative towards promoting the effectiveness, availability and
affordability of flight simulators. It discusses simulator use in
airline pilot training and evaluation as well as recent changes in
training and evaluation needs and their implications for simulator
fidelity requirements. It then explains the FAA's rationale and
objectives for tasking the Volpe Center with a scientific review of
simulator requirements that examines both the fidelity of the simulation
of the environment in which the airplane operates and the fidelity of
the airplane simulation itself. Finally, it presents the current focal
points of the investigation, namely, the needor lack of needfor
realistically simulating radio communications and airplane motion.
Findings to date and future plans are discussed.

## Simulator Use in Airline Pilot Training and Evaluation 

Simulators have been used in air-carrier pilot training and evaluation
since the 1950s. These early simulations were effective for training and
checking of procedures, systems knowledge, and navigation skills. Not
until the introduction of the FAA's Advanced Simulation Plan (ASP),
however, which established technical requirements for simulators, could
stick and rudder skills be trained and evaluated in the simulator. Major
airlines are currently not only conducting their recurrent training
entirely in the simulator, but also their initial, transition, and
upgrade training and certification. This is followed by carefully
structured Initial Operating Experience (IOE) carrying passengers in the
air, under supervision of a Check Airman.

The control and flexibility afforded by flight simulators has greatly
enhanced pilot training as well as practically eliminated training
accidents at major airlines that have access to the high-fidelity
simulators required for total training and checking. Training accidents,
however, continue to occur at smaller airlines that still conduct some
of their training in the airplane. Also, the lack of access to
high-fidelity simulators deprives the smaller airlines of some of the
training opportunities provided by simulators, such as carefully
constructed scenarios containing emergencies that require cognitive
skills such as crew coordination, task management, and decision making
in addition to motor flying skills. These training opportunities are
particularly important in view of recent changes in the airline
operational environment, effecting a shift in training and evaluation
needs.

## Training Challenges in Today's Operational Environment

Managers of airline pilot training programs today are increasingly being
faced with some unprecedented challenges. Of these, changing
demographics in the available new-hire population is clearly the most
prominent. Many of the U.S.'s largest airlines have found it necessary
to dramatically decrease their entry-level requirements in terms of
flight hours and prior experience. This in turn has placed even greater
pressure on smaller airlines, where entry levels have been lowered to
bare minimums, while turnover among pilots leaving for positions with
major airlines is at an all time high.

All of this is occurring in a backdrop of increased congestion within
the National Airspace System (NAS), associated short-term strategies
(such as Land and Hold Short) to manage capacity, the acquisition of
newer aircraft with increasingly automated cockpit systems, the merging
of airlines which may differ in their operating procedures and their
corporate cultures, and near term plans for new ways of operating within
the NAS, such as free flight and Air Traffic Control (ATC) data link.

These developments clearly have implications for the design and content
of pilot training. Pilot training curricula, which were based on certain
entry-level assumptions that are no longer valid, must be revised to
incorporate training in areas either not previously required for airline
new hires, or not previously addressed in depth. In some cases this will
require a substantial increase in the footprints allocated to training
and assessment of basic knowledge and hands-on flying skills. In all
cases it will require that flight operations training include specific
emphasis on building proficiency in the integration of cognitive and
motor flying skills. While these needs are particularly pertinent to
initial qualification curricula, they may also necessitate certain
changes in recurrent training curricula, in order to assure that
requisite proficiency is maintained, especially for rarely practiced
skills, or those with very low prior experience histories.

Similarly, it becomes increasingly important for each airline to assure
that pilot training for both new-hire and existing pilot populations
appropriately reflect its dynamically changing environment, whether that
be in terms of aircraft systems, flight procedures, or corporate
culture.

With regard to the regulatory environment pertinent to these
developments, there is both good and bad news. On the positive side, the
FAA has established a voluntary regulatory program for airlines that is
well suited to meet the training challenges of today and tomorrow. The
Advanced Qualification Program (AQP), which was established as Special
Federal Aviation Regulation (SFAR) 58,^2^ is specifically designed to
assure that pilot training programs remain responsive to changing needs,
and that the graduates of such programs not only possess the requisite
knowledge and hands-on skills, but that in particular, they can
demonstrate proficiency in the integration of cognitive and motor skills
in operationally realistic scenarios that test both.

On the negative side, the traditional pilot training regulations,
Subparts N & O of Part 121 of the Code of Federal Regulations, which
constitute the only alternative to AQP, are sorely out of date relative
to today\'s needs. The FAA is presently in the process of rewriting
Subparts N & O. A Notice of Proposed Rulemaking (NPRM) detailing
proposed changes to traditional pilot training, testing, and checking
requirements is anticipated at a future date.

AQP requires airlines to employ a systematic instructional-design
process in determining the content of pilot training, testing, and
checking, as well as in the training equipment allocation for those
purposes to curricula. It specifically requires the use of entry-level
analysis to assure that training content is appropriately matched to an
airline\'s pilot population, and it mandates the implementation of a
continuing quality control process for monitoring the effectiveness of
curricula. While it allows flexibility in equipment selection, AQP is
not ordinarily intended for use in curricula that employ the aircraft
for training. Rather, AQPs typically employ a suite of training
equipment, which includes flight training devices and full flight
simulators.

The revised Subpart N & O rules are expected to incorporate a similar
philosophy. This is in contrast to the existing rules, which still
permit training, testing, and checking to be conducted entirely in the
aircraft. It can reasonably be expected, therefore, that within a few
more years, all airline pilot training in the U.S. will require the use
of full flight simulators for certain training, testing, and checking
tasks, while allowing for the use of flight training devices for many
but not all such tasks. It can also be expected that pilot training
addressing both cognitive and motor skills in simulated scenarios that
require both sets of skills will be a requirement for all curricula,
both in AQP and otherwise.

A related regulatory development is the pending issuance of an NPRM on
simulator qualification. Whereas presently U.S. simulator qualification
procedures and standards are detailed only in an FAA Advisory Circular
(AC),^3^ the proposed new rule is intended to establish a regulatory
basis for those procedures and standards. If issued as a final rule,
airlines and training centers that do not maintain their flight
simulators in accordance with the procedures and standards on which
basis the FAA originally qualified the equipment could be subject to FAA
enforcement action.

## FAA/Volpe Flight Simulator Fidelity Research Program

The net result of these developments is that ultimately all pilot
training in the U.S. will be conducted in FAA qualified training
equipment (full flight simulators, and, where permitted, flight training
devices) rather than in aircraft. Airlines will be required by
regulation to maintain the fidelity of such equipment in accordance with
FAA qualification criteria. Furthermore, it can be expected that
regardless of whether an airline is conducting pilot training under an
AQP or under revised traditional rules, all curricula will require
training in operationally realistic scenarios.

While these developments will be largely transparent to major airlines
under AQP, they will be especially challenging to smaller airlines that
do not presently employ simulators, or that have limited access to such
equipment. Depending on the particular aircraft make, the availability
of flight simulators may be problematic, and for some operators, cost
could be prohibitive.

In anticipation of these regulatory changes, the FAA is committed to
ensure that flight simulators effectively prepare pilots to carry
passengers in the air. First, the simulator must faithfully represent
both the cognitive and the motor challenges encountered in today's
operational environment. Second, simulator fidelity requirements must be
sufficient to ensure full transfer of behaviors and performance between
airplane and simulators. In other words, for evaluation purposes, the
skills, behaviors, and performance that a pilot demonstratesor fails to
demonstratein the air must be accurately reflected in the simulator. For
training purposes, the skills, behaviors, and performance that a pilot
acquires in the simulator must be available to him or her in the air.

To ensure access of all airlines large and small to the benefit of
simulator training, however, the FAA must also ensure that the required
means and level of fidelity for cue representation are necessary. That
means that fidelity requirements must result in a demonstrable,
operationally relevant transfer benefit, rather than being based on user
preferences and technological availability alone.

The FAA has therefore tasked the Volpe Center with developing a
scientific basis to assure that FAA requirements promote full transfer
of pilot performance between simulator and airplane without
unnecessarily driving up cost. A thorough review of the literature and
systematic consultation of subject matter experts (SMEs) has shown that
while the effectiveness of flight simulators for initial and recurrent
training of airline pilots is well recognized, there are two areas that
warrant further examination in light of the preceding discussion: the
fidelity of the representation of radio communications and the
requirement for platform motion cues during full flight simulations.

## Realistic Radio Communications Simulation Fidelity Requirements

One of the widely recognized deficiencies in the current state of the
art in full flight simulation is realistic radio communications. The
simulation of such communications is typically accomplished by the pilot
instructor/evaluator (I/E), who is also tasked with operating the
simulator while observing trainee performance. This mode of simulation
is a highly simplified representation of radio communications, and does
not even begin to approach the challenges of hearing, acknowledging, and
appropriately responding to radio communications within a real world
environment. It follows that scenario-based training, testing, and
checking that relies on such an unrepresentative means of simulation is
not likely to be fully effective in developing the cognitive and
workload management skills associated with radio communications. While
this is obviously a deficiency of long standing, it may assume greater
significance in light of the changing demographics of the pilot new-hire
population. It follows that research is needed to measure the impact of
realistic radio communications on training effectiveness, and to make an
assessment of whether there may be an affordable means of better
simulating this function considering developing technology.

Volpe, in collaboration with NASA Ames, has therefore examined 1)
airlines' current methods of simulating radio communications, 2) the
effect of these practices on training and evaluation according to SMEs,
literature, and reports on IOE, and 3) industry efforts to improve the
current situation. A summary of this work follows.^4^

### Current Practices

The findings summarized below are based on information collected from 29
I/Es from 14 AQP airlines, including seven major, one cargo, four
regional, and two non-U.S. airlines. I/Es were queried about their
simulation of different events, including ATC (tower,
approach/departure, en route) and company communications (dispatch,
ramp, maintenance, flight attendants) to own aircraft, ATC
communications to and from other aircraft or ground vehicles (the
so-called party line), as well as visual representation of other
traffic.

A first finding was that the method of simulating radio communications
is indeed almost exclusively I/E role play, where the I/E issues
instructions and responds from his station directly behind the crew.
This was found both for company and ATC radio communications.

With regard to communications to own airplane, all I/Es reported
simulating ATC clearances in the terminal environment, and all but two
provide communications en route. Fewer I/Es have time to role-play
company communications, between 63 (ramp/gate) and 94 percent
(dispatch).

I/Es were also queried about representation of other traffic. Only 59
percent indicated that their simulators provide some out-the-window view
of traffic, mainly on the airport surface. Ten reported simulation of
traffic via the Traffic Alert and Collision Avoidance System (TCAS).
With regard to the party line, only 38 percent of all I/Es reported
simulating any communications to and/or from other aircraft or vehicles,
mainly on the airport surface.

In summary, it appears that although I/Es provide their best effort
simulating radio communications, without additional support they have to
focus their limited resources on role-playing communications to own
airplane in the terminal environment.

### Instructor/Evaluator Opinions

I/Es were asked to indicate their allocation of time and effort between
running the simulation, simulating radio communications, instructing and
observing. I/Es spend about 50 percent of their time and effort
observing, twenty percent each role-playing radio communications and
operating the simulator, and less than ten percent instructing.

Not surprisingly, I/Es rated their workload consistently higher for
training and checking in the simulator than in the actual aircraft with
real communications and no simulator to run. On the other hand, they
feel that the communications workload of the pilots is significantly
reduced. One I/E mentioned that even the manual workload of pilots is
reduced, because "\[p\]ilots are not normally given a chart frequency,
nor do they need to redial a new frequency to communicate."

I/Es stressed the importance of radio communications simulation for
teaching such skills as (new) ATC procedures, Crew Resource Management
(CRM) and situation awareness effectively. The overall importance of
radio communications is perceived highest in the terminal environment.
I/Es concern with simulating radio communications may have best been
summarized by the I/E who stated: "Without communication simulation,
when the pilot trainee finally arrives in the 'real world,' he must add
another component...This new (additional) component can really
complicate line flying."

### Literature Review

Many of the I/E opinions found in the previous section are confirmed in
the literature.^4^ FAA materials on AQP and CRM stress that coordination
with ATC, company, and flight attendants is an integral part of line
operations and that frequency monitoring is important for maintaining
traffic and weather situation awareness.^5,6^

Incident and accident investigations highlight the importance of radio
communications. The Flight Safety Foundation Approach and Landing
Accident Reduction Task Force recommends that operators "\[i\]nclude
training scenarios that allow crews to experience overload, task
saturation, loss of situational awareness, out-of-control and
too-far-behind-the aircraft-situations, and communications in stressful
circumstances." Joint training should be held between pilots and air
traffic controllers including scenarios that "promote mutual
understanding of issues on both the flight deck and in the ATC
environment, and foster improved communications during emergency
situations. "^7^

There is much theoretical and experimental evidence that whole-task
training in a fully loaded environment is superior to part-task training
in an incomplete environment that may induce a false sense of
operational simplicity akin to tunnel vision.^8^ Learning is an active
process, and practice can lead to either activation or inhibition of
cognitive pathways.^9^ If pilots are consistently exposed to an
impoverished environment during training, they may end up unprepared for
the "quick pace of an airline environment and \[its\] associated
distractions," [^2] where their inexperience may prove fatal.

### Effect on Initial Operating Experience

A search of the Aviation Safety Reporting System (ASRS) database from
January 1993 to October 1999, resulting in 93 reports related to flight
events during IOE, confirmed some of the concerns expressed by I/Es and
the literature.^10^ Most of the errors reported involved altitude
deviations or course deviations. Other multiple errors were landing or
take-off without clearance, approach to or landing on the wrong runway
or airport, runway incursions, and loss of communications.

Radio communications played a major role in 72 percent of the reports.
Demanding, inadequate, or even erroneous ATC instructions spearheaded
the factors leading to events deemed worthy of a report. Amended
clearances requiring reprogramming of the automation or erroneous
"expect" instructions were often cited. This was followed by inadequate
CRM or task management related to radio communications, ATC
interruptions including traffic calls and frequency congestion, stuck
microphones blocking an entire frequency, or pilots stepping on an
ongoing conversation. Next were radio-tuning problems and unfamiliar
phraseology and accent, the latter usually in non-English speaking
territory.

### Industry Initiatives

Given all the evidence presented in the previous section, it is not
surprising that both airlines and simulator industry are striving to
improve radio communications realism. Only efforts known to have been
used in airline training will be discussed.^4,10^

Among the most operationally realistic efforts is United Airlines'
Interactive Real Time Audio System (IRAS). It is based on field
recordings of actual ATC communications, including both communications
to own and to and from other aircraft with controller voices dubbed with
the individual instructors' voices. The system, however, is no longer
operational due to high scenario-development, integration and
instructor-training costs.

The Canadian simulator manufacturer CAE has developed the Ground Air
Traffic Environment System (GATES) after a request from a foreign
airline to provide a visual representation of traffic in the airport
terminal environment. It soon became obvious that correlated and
meaningful radio communications would have to be an essential component
of such traffic representation. The I/E still provides all ATC
communications to own airplane, however. Several domestic airlines and
training facilities as well as foreign airlines and military are
currently equipping their simulators with GATES.

Lufthansa and the German ATC organization Deutsche Flugsicherung (DFS)
are collaborating on Joint Operational Incidents Training (JOINT). Up to
eight Lufthansa simulators can be linked to two DFS ATC control sector
simulators, resulting in highly effective recurrent training of both
pilots and controllers. Each ATC simulator consists of a controller work
station with a radar display showing the simulated airplanes flown by
the flight simulator crews as well as other airplanes sharing the same
airspace operated by a pseudo pilot sitting at a connected computer
station.

## Simulator Motion Fidelity Requirements

While all FAA qualified full flight simulators are required to have
platform motion, a number of questions yet remain to be empirically
addressed through appropriate research. Specifically, the question to be
answered is whether the training conducted in a fixed-base simulator
with a wide Field-Of-View (FOV) visual system produces a result
equivalent to that obtained in a like system having platform motion
cueing. An additional question from a regulatory perspective is whether
proficiency checks conducted in a visually equipped fixed-base simulator
provide an equivalent opportunity to verify the line-operational
readiness of air-carrier pilots.

If the answers to these questions can reliably and validly be obtained,
the FAA may be better able to determine what level of equipment should
be required for initial or recurrent training programs in the future,
and whether changes to future qualification criteria for such equipment
are warranted. These decisions could significantly affect the cost and
availability of flight training equipment in light of future regulatory
plans, particularly for small operators.

### Subject Matter Expert Opinion

The focus on simulator motion originated from a series of joint
FAA-Industry symposia that were part of an FAA-sponsored review of the
simulator requirements outlined in AC120-40B.^3^ This effort is
summarized in Ref. 11. The intent was to systematically examine the most
costly aspects of airplane simulation for potential cost-savings without
loss in training effectiveness. A first symposium on the simulation of
the aeromodel focusing on flight data collection and predictive modeling
resulted in recommendations that would lower the cost of the simulator
data package by 50%.^12^

Regarding simulator motion, the SMEs from industry, academia, and FAA
participating in the discussions generally perceived that the absence of
platform motion cueing in fixed-base devices is likely to have a
detrimental effect on pilot control performance, particularly in
maneuvers entailing sudden motion-onset cueing with limited visual
reference. It was also noted, however, that there was no scientific
evidence that training in a fixed-base device would lead to degraded
control performance in the actual aircraft. This issue is especially
pertinent in a device equipped with a wide FOV visual system, which can
generate a relatively strong illusion of motion (vection).^13^

### Past Work

An extensive literature review suggested that platform motion in the
simulator might improve the acceptability of the simulator, at least
when the pilots were aware of the motion manipulation.^14,15^ Motion
also improved pilot performance and control behavior in the simulator,
especially for disturbance tasks and tracking tasks of aircraft with low
dynamic stability.^16,17,18^ Some of the benefits of platform motion
have also been shown to transfer to a higher fidelity simulator.^19^
However, the benefits of platform motion have not been proven in the
case of transfer of training to the airplane.^20^

Driven by the above findings and the FAA commitment to improve the
availability and affordability of simulators, Volpe has been tasked to
obtain objective data on simulator motion requirements. Questions to be
answered include, but are not limited to, the following: Are there any
flight tasks for which a measurable difference in simulator training
effectiveness can be found with and without platform motion? What is the
relationship in motion cueing effectiveness for a wide FOV visual
display versus platform motion? Are existing platform motion
qualification criteria optimal? Is there a relationship between pilot
experience level and the effectiveness of platform motion for training?

### Results of First Study

A first experiment was performed to answer some of these questions for a
simulator that is routinely used for pilot training and evaluation. It
used an "as-is" FAA qualified Level C simulator with a six
Degree-Of-Freedom (DOF) synergistic motion system and a wide FOV
high-quality visual system, simulating a 30-passenger turboprop airplane
with twin wing-mounted engines.^21,22^ Deficiencies identified in a
review of prior research designs were avoided by measuring both pilot
stimulation and response, testing both maneuvers and pilots that are
diagnostic of a need for motion, preventing pilot and instructor bias,
and ensuring sufficient statistical power to capture operationally
relevant effects. Experienced airline pilots were evaluated and trained
in the simulator, half of them with and the other half without motion.
Then the transfer of skills acquired by both groups during this training
was tested in the simulator with the motion system turned on as a
stand-in for the airplane (quasi-transfer). The test maneuvers selected
were engine failures on take-off with either rejected take-off (RTO) or
continued take-off (V~1/R~ cut) in low visibility, which satisfied the
criteria described in the literature as diagnostic for the detection of
a motion requirement. These criteria included 1) closed loop, to allow
for motion to be part of the control feedback loop to the pilot; 2)
disturbance maneuver, to highlight an early alerting function of
motion,^18,23^ 3) high gain, to magnify any motion effects and to reduce
the stability of the pilot/airplane control loop,^18^ 4) high workload,
to increase the need for redundant cues such as provided by motion,
out-the-window view, instruments and sound; and 5) short duration, to
prevent pilots from adjusting to a lack of cues. Both subjective (I/E
grades, questionnaires) and objective data recorded from the simulator
were collected from the experiment.

The results of the study indicate that the motion provided by the test
simulator did not, in an operationally significant way for the tasks
tested, affect evaluation and training progress in the simulator, or
transfer of training acquired in the simulator with or without motion to
the simulator with motion. It also didn't consistently affect the
Pilots' Flying (PF), Pilots' Not Flying (PNF), and I/Es' subjective
perception of the PFs' performance, workload, and training, or of their
own comfort in the simulator. Neither did it affect the acceptability of
the simulator to the PF and the PNF.

Note, however, that because the current simulator qualification
procedures do not provide a means to objectively assess the quality of
the produced motion,^24^ the motion provided by the test simulator may
not be typical of other FAA qualified Level C simulators. Specifically,
the observed lack of lateral acceleration produced by the simulator
shortly following the engine failure compared to the lateral
acceleration from the aircraft mathematical model suggests that the
simulator used in the study may not have provided sufficient motion
stimulation to the pilot.

To validate and generalize the previous results, two follow-up studies
are underway. One examines the typicality of the test simulator used in
the previous experiment, by gathering data from other Level C and D
simulators and then comparing these data to the test simulator. The
second study will be similar to the previous one, while eliminating any
possible causes of not having found an effect of motion.

### Comparison Study

This effort is expected to provide information on whether the earlier
results may apply to other FAA qualified simulators, at least for the
maneuvers and pilot population tested. The main purpose of the
comparison is to examine whether the level of lateral accelerations
generated by the motion system and the equations of motion of the test
simulator for the V~1~ cut maneuver are typical. Data from nine level C
and D simulators, representing turboprop or turbofan airplanes with twin
wing-mounted engines, were obtained for comparison. Due to simulator
time constraints, the requested data were obtained by running the
maneuver using the automatic testing feature of the simulators.

There are several difficulties in performing the comparison. First of
all, the simulators came from a range of manufacturers and as such the
capability of accessing the data at various points in the simulation
loop varied from one simulator to another. Therefore, not all the data
requested were obtained from each simulator. Also, sometimes there was
no way of telling whether values of certain variables recorded from the
simulator computer had passed through some scaling. Second, the engine
failure profiles used in the automatic testing vary greatly from one
simulator to another, and these profiles can affect the resulting
lateral acceleration. Thus, absolute comparison is not possible with the
data gathered.

Within the above constraints, however, initial analysis of the data
suggested that the level of lateral acceleration produced by the test
simulator during V~1~ cut maneuvers is not atypical.

### Current Research

As mentioned earlier, a study is currently underway to validate and
generalize the previous results. As in the first study, the current
effort is maximized for finding an effect of motion if there is one. An
effort is made to eliminate any causes of not finding an effect of
motion in the previous experiment. For example, the motion system of the
test simulator will be tuned to provide the best possible performance
within its operational envelope. Also, test subjects will fly different
maneuvers considered to be diagnostic for detecting an effect of motion.
Finally, the use of a different airplane simulation will automatically
result in the use of a different pilot population.

This study will be performed in collaboration with the NASA Ames
Research Center and will utilize its B747-400 simulator, which is an FAA
qualified level D simulator. In collaboration with the manufacturer
(CAE), some modifications on the motion system are performed to
guarantee that the maximum capability of the motion actuators be
utilized for flying the test maneuvers.

A between-subjects design is used for the experiment. Subjects will be
divided into two groups, where one group will be evaluated and trained
in the simulator with motion and the other in the simulator without
motion. Both groups will then be tested for transfer of training in the
simulator with motion as a stand-in for the real airplane
(quasi-transfer).

For the test maneuver, the selection criteria remain the same as in the
previous study, with some relaxation of the short-duration criterion in
favor of maneuvers that require a more complex correction by the pilot.
The current plan is to combine a continued take-off with an engine
failure and a high-workload landing task into one complete scenario. The
candidates for the landing tasks are a sidestep landing with vertical
gust disturbance on rollout and a precision instrument approach with
varying crosswind disturbance. Dependent on the maneuver selection, the
other settings in the simulator, such as airplane configuration and
visibility, will be adjusted so that the motion stimulation and the
pilot workload are maximized, while still complying with aviation
regulations. Subjective (questionnaires, I/E grades) and objective
(measurements, simulator recordings) data to assess motion stimulation
of pilots, pilot performance, and pilot workload/behavior will be
collected. Unlike the previous experiment, I/E grades for the current
experiment may have to be obtained offline due to cost and availability
constraints.

Further preparation for the study and fine-tuning of the experiment
design are currently underway. Results of this study will help determine
whether the benefits of tighter motion standards, as currently
considered, would justify the potential reduction in simulator
availability due to an inevitable increase in acquisition, maintenance,
as well as enforcement efforts.

## conclusionS

Given the anticipated regulatory changes mandating the use of simulators
in airline pilot training and evaluation, dramatically reduced pilot
new-hire experience levels and growing operational complexity, the twin
issues of training effectiveness and affordability of flight simulators
for use by U.S. airlines will become increasingly critical. In that
regard, two research areas with high pay-off potential are realistic
radio communications and platform motion.

Initial FAA-sponsored research on the simulation of radio communications
for U.S. airline pilot training strongly suggests that in order to be
fully effective in developing the cognitive and workload management
skills associated with radio communications, significant improvements
are needed in the resources available to the pilot instructor for that
purpose. The present practice of relying on IOE to compensate for the
deficiencies in simulator resources for training radio communications
skills may become increasingly inadequate in view of the changing
demographics of the pilot new-hire population, as well as the
anticipated future regulatory requirement for all airlines to conduct
training in operationally realistic flight simulator scenarios. Although
there have been a number of promising developments in this arena,
considerable additional research is needed to reduce the cost and labor
requirements associated with simulating radio communications in an
operationally realistic fashion.

Initial research on the training effectiveness of a fixed-base simulator
with a wide FOV visual system compared to a like system having platform
motion failed to find an operationally significant effect of motion
using FAA qualified equipment. Objective measurement of the motion
characteristics of the FAA qualified simulator employed in the
experiment, however, indicated that lateral acceleration stimulation was
minimal. Subsequent comparisons with objective data acquired from other
FAA qualified simulators indicated that the motion characteristics of
the test simulator may be typical for the equipment routinely used for
total airline pilot training and evaluation. A second training
effectiveness study is planned using a test simulator tuned to provide
the best possible platform motion performance and maneuvers selected to
be especially diagnostic. However, considering the controversial nature
of this issue, the safety implications, and the fact that no changes in
regulatory requirements can be expected without absolute confidence in
the reliability and validity of the results, it is likely that
considerable additional research in this area will still be needed.

## ACKNOWLEDGMENTS

This work was funded by the Office of the Chief Scientific and Technical
Advisor for Human Factors of the Federal Aviation Administration,
AAR-100. We are greatly indebted to Dr. Eleana Edens, the FAA program
manager, for her excellent direction. The late Mr. Edward M. Boothe
guided our steps throughout this project. We miss his knowledge, wisdom,
insight, and friendship every day. The opinions expressed are those of
the authors and not necessarily those of the Department of
Transportation, the Federal Aviation Administration, or the U.S.
Government.

## References

1.  Longridge, T., Bürki-Cohen, J., Go, T. H., and Kendra, A. J.,
    "Simulator Fidelity Considerations for Training and Evaluation of
    Today's Airline Pilots," Proceedings of the 11^th^ International
    Symposium on Aviation Psychology, The Ohio State University Press,
    Columbus, OH, March 2001.

2.  Federal Aviation Regulation, "Advanced Qualification Program," 14
    C.F.R., Part 121, SFAR 58, Washington, DC, 1990.

3.  Federal Aviation Administration, "Airplane Simulator Qualification,"
    Advisory Circular No. 120-40B, U.S. Department of Transportation,
    Washington, DC, 1991.

4.  Bürki-Cohen, J., Kendra, A., Kanki, B., and Lee, A., "Realistic
    Radio Communication in Pilot Simulator Training," Report No.
    DOT/FAA/AR-00/62, Washington, DC, 2000.

5.  Federal Aviation Administration, Advanced Qualification Program,
    AFS-230, 2000, web site: <http://www.faa.gov/avr/afs/aqphome.htm>

6.  Federal Aviation Administration, "Crew Resource Management
    Training," Advisory Circular No. 120-51C, U.S. Department of
    Transportation, Washington, DC, 1998.

7.  Khatwa, R. and Helmreich, R., "Analysis of Critical Factors During
    Approach and Landing in Accidents and Normal Flight," *Flight Safety
    Digest*, Flight Safety Foundation, Alexandria, VA, 1999.

8.  Fabiani, M., Buckley, J., Gratton, G., Coles, M., Donchin, E., and
    Logie, R., "The Training of Complex Task Performance," *Acta
    Psychologica*, 71, 1989, pp. 259-299.

9.  Bransford, J. and Franks, J., "Towards a Framework for Understanding
    Learning," *The Psychology of Learning and Motivation*, 10, 1976,
    pp. 93-127.

10. Bürki-Cohen, J., and Kendra, A.J., "Air Traffic Control in Airline
    Pilot Simulator Training and Evaluation," *Air Traffic Control
    Quarterly*, accepted for publication pending revision.

11. Longridge, T., Ray, P., Boothe, E., and Bürki-Cohen, J., "Initiative
    Towards More Affordable Flight Simulators for U.S. Commuter Airline
    Training," Paper presented at the Royal Aeronautical Society
    Conference on TrainingLowering the Cost, Maintaining Fidelity,
    London, UK, May 1996.

12. Transcript[^3] of the Joint FAA/Industry Symposium on Level B
    Airplane Simulator Aeromodel Validation Requirements, Washington
    Dulles Airport Hilton, March 13-14, 1996.

13. Transcript of the Joint FAA/Industry Symposium on Level B Airplane
    Simulator Motion Requirements, Washington Dulles Airport Hilton,
    June 19-20, 1996.

14. Reid, L. D. and Nahon, M. A., "Response of Airline Pilots to
    Variations in Flight Simulator Motion Algorithms," *Journal of
    Aircraft*, Vol. 25, No. 7, 1988, pp. 639-646.

15. Bussolari, S. R., Young, L. R., and Lee, A. T., "The Use of
    Vestibular Models for Design and Evaluation of Flight Simulator
    Motion," in AGARD CP-433, Paper presented at the Aerospace Medical
    Panel Symposium on Motion Cues in Flight Simulation and Simulator
    Induced Sickness, Brussels, Belgium, 1988.

16. Hosman, R. J. A. W. and van der Vaart, J. C., "Effects of Vestibular
    and Visual Motion Perception on Task Performance," *Acta
    Psychologica*, 48, 1981, pp. 271-287.

17. Hall, J. R., "Motion versus Visual Cues in Piloted Flight
    Simulation," in AGARD CP-249, Paper presented at the Flight
    Mechanics Panel Specialists Meeting on Piloted Aircraft Environment
    Simulation Techniques, Brussels, Belgium, 1978.

18. Hall, J. R., "The Need for Platform Motion in Modern Piloted Flight
    Training Simulators," Technical Memorandum FM 35, Royal Aerospace
    Establishment, London, UK, 1989.

19. Levison, W. H., "Effects of Whole-Body Motion Simulation on Flight
    Skill Development," Report AFOSR-TR-82-006, Air Force Office of
    Scientific Research, Washington, DC, 1981.

20. Waag, W. L., "Training Effectiveness of Visual and Motion
    Simulation," Report AFHRL-TR-79-72, Air Force Human Resource
    Laboratory, Brooks Air Force Base, Texas, 1981.

21. Go, T. H., Bürki-Cohen, J., and Soja, N. N., "The Effect of
    Simulator Motion on Pilot Training and Evaluation," AIAA Paper
    2000-4296, Proceedings of AIAA Modeling and Simulation Technologies
    Conference, Denver, CO, August 2000.

22. Bürki-Cohen, J., Boothe, E.M., Soja, N.N., DiSario, R.D., Go, T.,
    and Longridge, T., "Simulator Fidelity--The Effect of Platform
    Motion," Proceedings of the Royal Aeronautical Society conference on
    Flight SimulationThe Next Decade, London, UK, May 2000.

23. Gundry, J., "Man and Motion Cues," Paper presented at the Third
    Flight Simulation Symposium, London, April, 1976.

24. Lahiri, A., "Significance of a Qualitative Handling Assessment
    Approach Towards Motion System Requirements for Flight Simulators,"
    Proceedings of the Royal Aeronautical Society conference on Flight
    SimulationA Decade of Regulatory Change: Has the Situation Changed
    for the Better? London, UK, 2000.

[^1]: ^\*^ Portions of this work have been published as Ref. 1.

    Engineering Psychologist, Member AIAA.

    Postdoctoral Associate, Department of Aeronautics and Astronautics,
    Member AIAA.

    Manager, Advanced Qualification Program, AFS-230.

    This paper is declared a work of the U.S. Government and is not
    subject to copyright protection in the United States.

[^2]: ^^ From an ASRS report

[^3]: ^^ Available in electronic format from Dr. Thomas Longridge, tel.
    (703) 661-0275