Technical Report 1092

Optimizing Simulator-Aircraft Mix for U.S. Army

Initial Entry Rotary Wing Training

John E. Stewart III and John A. Dohme

U.S. Army Research Institute

Robert T. Nullmeyer

Air Force Research Laboratory

March 1999 1 999 0 414 0 17

United States Army Research Institute

for the Behavioral and Social Sciences

Approved for public release; distribution is unlimited.

VALITY¢oarraz~rt
U.S. Army Research Institute

for the Behavioral and Social Sciences

A Directorate of the U.S. Total Army Personnel Command

EDGAR M. JOHNSON

Director

Technical Review by

Gavan Lintern, Defense Science and Technology Organization

Peter Crane, Air Force Research Laboratory

NOTICES

DISTRIBUTION: Primary distribution of this Technical Report has been made by ARI.

Please address correspondence concerning distribution of reports to: U.S. Army

Research Institute for the Behavioral and Social Sciences, Attn: TAPC-ARI-PO, 5001

Eisenhower Ave., Alexandria, VA 22333-5600.

FINAL DISPOSITION: This Technical Report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research for the Behavioral and Social

Sciences.

NOTE: The findings in this Technical Report are not to be construed as an official

Department of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-w) 2. REPORT TYPE 3. DATES COVERED (from... to)

March 1999 Final March 1998-March 1999

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Optimizing Simulator-Aircraft Mix for U.S. Army Initial Entry Rotary Wing Training

5b. PROGRAM ELEMENT NUMBER 62785

6. AUTHOR(S) 5c. PROJECT NUMBER A790

John E. Stewart I1 (ARI), John A. Dohme (ARI), and Robert T. Nullmeyer (AFRL)

5d. TASK NUMBER 206

Se. WORK UNIT NUMBER C02

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

U.S. Army Research Institute Air Force Research Laboratory

Rotary Wing Aviation Research Unit Human Effectiveness Directorate

Bldg. 5100 Warfighter Training Research Division

Fort Rucker, AL 36362-5354 6001 S. Power Rd., Bldg. 561

Mesa, AZ 85206-0904

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Avenue ARI

Alexandria, VA 22333-5600 11. MONITOR REPORT NUMBER

Technical Report 1092

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

14. ABSTRACT (Maximum 200 words): Early fixed wing research demonstrated that potential cost and training benefits could be derived

from simulation-augmented primary flight training. Unfortunately, more recent research in this area has been the exception, not the rule.

This is especially true in the case of rotary wing (helicopter) aircrew training research. The present report reviewed the research

literature on military aviation transfer of training (TOT) research, and examined the current U.S. Army Initial Entry Rotary Wing (IERW)

Program of Instruction. An in-depth review was also conducted on the recent IERW simulation research performed by the Army

Research Institute (ARI) Rotary Wing Aviation Research Unit (RWARU). Review of the IERW TOT research showed that a

combination of synthetic flight simulation and criterion-based training had the potential for saving training time and costs in the aircraft.

Adaptive training aids such as the ARI RWARU Intelligent Flight Trainer, also showed promise. A research program, focusing on

revising the current IERW program to optimize the use of simulation, was proposed. This program would include (a) criterion-based

instructional strategies, (b) low-cost simulation, and (c) investigation of different combinations of simulator vs. aircraft training events, in

order to determine the optimal simulator/aircraft training "mix"

15. SUBJECT TERMS Primary flight training, Simulation, Transfer of training research, Aircrew training programs, Helicopter primary training

:.:.::.::: 19. LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

ABSTRACT OF PAGES (Name and Telephone Number) 16. REPORT 17. ABSTRACT 18. THIS PAGE Dennis C. Wightman, Ph.D.

Unclassified Unclassified Unclassified Unlimited 52 (334) 255-2834/9091

i
Technical Report 1092

Optimizing Simulator-Aircraft Mix for U.S. Army

Initial Entry Rotary Wing Training

John E. Stewart III and John A. Dohme

U.S. Army Research Institute

Robert T. Nullmeyer

Air Force Research Laboratory

Rotary-Wing Aviation Research Unit

Dennis C. Wightman, Chief

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

March 1999

Army Project Number Personnel Systems and Performance

20262785A790 Technology

Approved for public release; distribution is unlimited.

iii
FOREWORD

The Aircrew Performance Team of the U.S. Army Research Institute for the Behavioral

and Social Sciences (ARI) Rotary Wing Aviation Research Unit is located at Fort Rucker,

Alabama. The ARI Aircrew Performance Team is committed to enhancing aviation training in

the Army. The focus of the present report is on the optimization of simulation-augmented Initial

Entry Rotary Wing (IERW) training programs. A program of research in this area was deemed

necessary because little data existed for developing benchmarks as to how best to use simulation

in the IERW environment.

Currently, the Army uses simulators only for instrument training in its IERW training

syllabus. ARI has conducted transfer of training research which has demonstrated that

simulation can be effectively employed in both the primary (non-instrument) as well as the later

instrument phase of IERW training. Building on lessons learned from the completed research,

ARI has proposed a program of research to determine the best "mix" of simulator and aircraft

training in the Army's IERW program. The results of this program of research will provide

objective data by which the rotary wing training community (i.e., private operators as well as the

Army) will be able to improve the efficiency and effectiveness of IERW training. The principal

research plan, based upon this report, was briefed to the Commander, Aviation Training Brigade,

on 16 February 1999.

A M. SIMUTIS

hnical Director

v
ACKNOWLEDGEMENT

The authors are indebted to Dale S. Weiler of CAE, Inc. for his invaluable support as a

subject matter expert. Mr. Weiler was assigned the task of bringing up to date the documentation

on the Army's Initial Entry Rotary Wing Program of Instruction. This he accomplished with

great skill, giving the authors a better understanding of how the Army currently administers

primary aviation training. This in turn provided a benchmark for the design of future transfer of

training experiments, in which the authors plan to address alternative instructional strategies for

this type of training.

vi
OPTIMIZING SIMULATOR-AIRCRAFT MIX FOR U.S. ARMY INITIAL ENTRY ROTARY

WING TRAINING

EXECUTIVE SUMMARY

Research Requirement:

Early fixed wing research demonstrated that simulators can be both cost- and trainingeffective. More recently, programmatic research on simulation-augmented aircrew training has

been the exception, not the rule. This can be stated even more emphatically in the case of rotary

wing (helicopter) aircrew training research. Because so little research has been conducted in this

area, a systematic knowledge base does not exist. Consequently, training developers and other

decision-makers have little substantive foundation for determining what such a training system

(or even a simulator) should look like.

The U.S. Army presently does not employ simulation for the primary, visual (contact)

phase of Initial Entry Rotary Wing (IERW) training. It does, however, provide 30 hours of

training time in a UTH-1 helicopter simulator without a visual display system. It is not known

how much training time can be offloaded to simulation without degrading the quality of training.

Research is required to determine the optimal mix (i.e., tradeoff) of aircraft for simulator training

time for IERW training.

Another, related research issue to be investigated in this context is the use of criterionbased as opposed to class-based (i.e., lock-step) training. In criterion-based training, all student

pilots do not train for a preset number of hours. Instead, each student trains until he or she has

reached a criterion for the performance of a specific task (e.g., three consecutive successful

stationary hovers). Once having met the criterion, the student moves on to the next task or set of

tasks. Although previous research has shown some advantages for criterion-based training, the

Army's IERW program remains class-based.

Procedure:

The present report reviewed the limited research literature on military aviation transfer of

training (TOT) research, examined the current IERW Program of Instruction, and analyzed in

some detail more recent IERW simulation research. On the basis of these efforts,

recommendations were made for a programmatic research effort to determine ways of enhancing

both the efficiency and effectiveness of IERW training.

Findings:

Review of aviation transfer of training research. The review revealed that what little

research had been conducted, showed that simulation had the potential for making current ab

initio training more efficient. An examination of simulator visual display research indicated

that, though research has provided useful data on visual display requirements unique to

helicopter simulation, these data have not been exploited by the designers. A review of Army

Research Institute (ARI) TOT research showed that a combination of synthetic flight simulation

vii
and criterion-based training had the potential of saving training time and costs in the aircraft.

ARI research on the Intelligent Flight Trainer, an adaptive training aid incorporating artificial

intelligence, was also promising.

Proposed research program. A research program, demonstrating the advantages of a

simulation-augmented IERW training system, was proposed. The research would be

programmatically focused, in that it would concentrate on the total IERW training system, and

not just the low-cost simulators that are part of it. Instructional strategies would be criterionbased, not classed-based as they are now. Phase I would concentrate on the instrument phase of

IERW training, in which some simulation is currently employed. It would examine the effects of

varying the mix of simulator and aircraft hours beyond the current 30 simulator-20 aircraft mix.

It would also compare the TOT effects of a low-cost simulator with those of the Army's current

instrument trainer simulator. Phase II would concentrate on the contact (i.e. visual) phase of

IERW, which currently does not employ simulation. The research would draw upon previous

ARI studies that investigated the effects of pretraining in a low-cost simulator on trainee

performance in the aircraft.

Utilization of Findings:

The insights gleaned from the present review and planning effort will form the basis for a

research program to investigate the use of simulation in a model TERW training system. This

research will provide an empirical basis for determining the configuration of such a training

system for the next millennium. Results of the planned research should provide the Army with

much-needed guidance in the design of training systems that are more efficient and effective than

those of today.

ViIl
CONTENTS

Page

Intro du ctio n ............................................................................................ 1

B ack g ro u n d ........................................................................................ 1

Aviation Transfer of Training Research ........................................................ 1

Improving Initial Entry Rotary Wing (IERW) Training Efficiency ............................. 5

Army IERW Training Program .................................................................. 5

Requirements for Enhanced Training Effectiveness ........................................ 7

Measures of Training Effectiveness and their Meaning ..................................... 9

An Optimal Mix of Aircraft and Simulation Training ...................................... 12

Training M anagem ent ............................................................................ 14

Trainee Performance Measurement ............................................................ 15

Tradeoffs and Recommendations .............................................................. 18

Army Research Institute Ab Initio Training Effectiveness Research .......................... 19

B ackground ........................................................................................ 19

O verv iew ...................................................... ............................. ....... . 19

E xperim ent 1 ............................................................................... . ...... 19

E xperim ent 2 ....................................................................................... 2 1

E xperim ent 3 ....................................................................................... 22

E xperim ent 4 ....................................................................................... . 22

Summary and Conclusions: Experiments 1-4 ................................................ 23

TH-67 Skills Trainer Transfer of Training Experiment ......................................... 23

Introduction ................................................................................ . ...... 23

M etho d ........................................................................................ ..... 24

R esults ...................................................................................... . ...... 25

Future Research Directions ..................................................................... 26

Automated, Adaptive, Simulator-Based Training ................................................. 27

O v erview ........................................................................................... 2 7

The Automated Hover Trainer ................................................................. 27

Automated Hover Trainer Quasi-Transfer Experiment ...................................... 28

Transfer of Training from the Automated Hover Trainer to the Aircraft .................. 28

Conclusions based on Automated Hover Trainer Findings ................................. 30

Implications of ARI IERW Experimental Findings ............................................... 30

ix
CONTENTS (Continued)

Page

General Conclusions and Recommendations for Future Research .............................. 31

Program m atic Issues ............................. ............................................ 31

R esearch P ossibilities ............................................................................ 31

Contact IERW Training Research .............................................................. 34

Implications of Proposed Research ............................................................ 35

R eferences ........................................................................................ ..... 37

Appendix A: IERW Common Core Outline ...................................................... A-1

B: Abbreviations and Acronyms ..................................................... B-1

LIST OF TABLES

Table 1. Quantitative Relationship between CTER, ITER, and Number of Trials

Saved in the A ircraft .................................................................... 9

2. Transfer Effectiveness Ratios for Transfer of Training Experiments ............... 20

3. Iterations Transfer Ratios for Transfer of Training Experiments .................. 21

4. Time to Flight Evaluation Checkride and End of Pre-Solo Phase Flight

G rade ............................................................................... . ...... 25

5. Mann-Whitney U/Wilcoxon Rank Sum Test Comparing Time to Complete

Successful Checkride for the Experimental (Simulator) and Control (NonSim ulator) G roups ......................................................................... 25

6. Mann-Whitney U/Wilcoxon Rank Sum Test Comparing Time to Complete

Unsupervised Solo for the Experimental (Simulator) and Control (NonSim ulator) G roups ......................................................................... 26

7. Time to Completion of Unsupervised Solo Flight, For Experimental and

Control G roups ................... ........................................................ 26

8. Percent of Students Meeting Criterion for Maneuver Training and Passing

Subsequent Checkride ........................................ ............................ 28

9. Mean Maneuver Iterations to Criterion for Experimental (n=6) and Control

(n= 33) Groups ............................................................................ 29

x
CONTENTS (Continued)

Page

Table 10. Quasi-Experimental Design for Investigating the Functional Requirements

for an IERW Training Simulator ..................................................... 33

11. Experimental Design for Investigating Optimal Mix of Simulator vs.

A ircraft H ours .......................................................................... 34

LIST OF FIGURES

Figure 1. Effects of Pretraining in the Simulator on Transfer Effectiveness

R atio (TE R ) .............................................................................. 10

2. Hypothetical Relationship Between the Amount of Simulator Pretraining

and Subsequent Required Aircraft Training (Adapted from Bickley, 1980) ...... 13

xi
OPTIMIZING SIMULATOR-AIRCRAFT MIX FOR U.S. ARMY INITIAL ENTRY

ROTARY WING TRAINING

Introduction

Background

Objectives of the present report. The main objectives of this report are to (a) summarize

representative research that has been done, as well as ongoing research in the area of rotary wing

simulation and training, (b) discuss the evolution and current status of the U.S. Army primary

rotary wing training course, (c) identify limitations and problem areas in simulation-augmented

primary flight training research, with an emphasis on rotary wing training systems, and (d)

propose a program of research with the goal of demonstrating the feasibility and effectiveness of

low-cost simulation in a primary helicopter training environment, with the goal of specifying the

optimal mix of simulator and aircraft training for the Army's primary training syllabus.

Assumptions underlying simulation research. Some assumptions related to optimizing

the training capabilities of simulators are: maximizing the number of task iterations per training

hour, and maximizing the rate of learning progression to integrated, whole task performance. A

much larger number of assumptions pertain to optimization of the cost-effectiveness and

efficiency of training. Many approaches for accomplishing this objective have been proposed

(e.g., trading aircraft for simulator time, reducing student training time, and maximizing

instructor effectiveness in the simulator). The challenge is to control program costs without

diminishing, and preferably, proving, product quality. However, with a few exceptions (e.g.,

Derrick, Tomczak, Nullmeyer & Burright, 1992), existing research addresses only a few of these

assumptions in limited training contexts, most of which are outdated. As we shall see, most of

the transfer of training (TOT) research concerns fixed wing rather than rotary wing flight

training. What little flight training research exists has been limited in scope.

We will now briefly review some of the empirical research. The emphasis will be on

helicopter-specific training and training device research. We seek to provide a general

framework for conveying the rationales and training strategies that, according to the literature,

should be considered in the development of a primary helicopter flight training system.

Aviation Transfer of Training Research

Evidence of training effectiveness. Taylor, Lintern & Koonce (1993) define simulator to

aircraft TOT simply and concisely.

A flight simulator is effective if the skills that a pilot learns in the simulator can be

performed in the aircraft; that is, if the skills transfer from the simulator to the aircraft.

The effectiveness of training in a flight simulator is a function of the amount of skill that

transfers. Its cost-effectiveness in a pilot-training program depends on the amount of

skill that transfer to the aircraft as well as the ratio of simulator to aircraft operating costs.

In this age of increasing reliance on simulators, it is ironic that so little research has been

done to optimize the effectiveness of aviation training systems. In spite of the assumption that a

1
simulator should "fly" like an aircraft, little research has been conducted to determine how many

of them actually do (Stewart, 1994). Furthermore, little is known as to how much fidelity is

required for effective training. Simulators that "fly" poorly can still be effective training devices,

in some environments. For example, Grimsley (1969) has demonstrated that even low fidelity

simulators can produce good TOT. Procedural and flying tasks are so interrelated, however, that

it is often difficult to determine which of these contributes to the positive TOT. This makes it

difficult for the researcher to determine from TOT results, how closely the flying characteristics

of the simulator and aircraft should match.

Even today, simulators are frequently integrated into training systems without evaluating

their training effectiveness. One possible explanation for this was proposed by Caro (1973), who

observed that most personnel who design and integrate simulators are engineers, not behavioral

scientists. Caro also expressed concern that much more attention has been paid to the

development of the simulator itself, than to the training program which supports it. Caro (1976)

stated that those who integrate and employ simulators in aviation training systems are best

described as artisans. They have shown considerable skill in getting simulators, even bad ones,

to perform their training missions. However, their efforts do not provide a common base of

constructs that describe the functional requirements for effective simulators. This leaves training

system developers without much guidance. Caro's observations of two decades ago are echoed

by Salas, Bowers & Rohodenizer (1998), who state that though simulation technology has

undergone incredible evolution, the evolution of training is at a virtual standstill. In short,

evidence of training effectiveness is lacking for most military aviation simulation and training

systems, because little research has been conducted.

Early fixed wing transfer of training research. Pioneering TOT research (e.g., Mahler &

Bennett, 1950; Williams & Flexman, 1949) demonstrated that pretraining in training devices

produced payoffs for later training in the aircraft. Examples of payoffs included fewer trials,

fewer flight hours, fewer errors to qualify in the aircraft (Williams & Flexman, 1949), fewer

serious student errors and fewer total errors in the aircraft (Mahler & Bennett, 1950). Early

fixed wing studies demonstrated that simulators can be both cost and training-effective

(Flexman, Roscoe, Williams, & Willeges, 1972; Williams & Flexman, 1949). Another,

somewhat more recent research emphasis has been on optimizing the mix of simulation and

aircraft time, and associated cost-benefits tradeoffs (e.g., Povenmire & Roscoe, 1973).

More recent fixed wing transfer of training research: an example. Edwards and Hubbard

(1991) conducted a controlled experiment to evaluate the training effectiveness of the Air

Intercept Trainer, a low-cost simulator for the F-16. It is also a good example of in-simulator

transfer, or quasi-transfer. The experimental group practiced air intercept tasks in the Air

Intercept Trainer, while the control group spent the same number of hours training in

conventional academics. Next, both groups performed these tasks in the Operational Flight

Trainer, a high-fidelity, full-mission simulator for the F-16. Edwards and Hubbard found that

mean proficiency ratings of participants in the experimental group were significantly higher.

Further, proficiency was obtained by these participants on more types of intercepts than the

controls (2.7 vs 1.5); this difference was also significant. They concluded that the Air Intercept

Trainer had the potential of significantly reducing training time in the simulator (whose

operational costs are higher).

2
Early helicopter transfer of training research. Much of this early research was conducted

by Paul Caro and his associates of theHuman Resources Research Organization division at Fort

Rucker, Alabama. The Fort Rucker division was the predecessor of the current Army Research

Institute for the Behavioral and Social Sciences (ARI) Rotary Wing Aviation Research Unit at

Fort Rucker. Some of the research took place at Fort Wolters, Texas, before all Army aviation

primary training was transferred to Fort Rucker. Caro and Isley (1966) reported the results of an

Army TOT experiment involving the Whirlymite, a lightweight, single-place helicopter tethered

by an articulated arm to a ground-effects machine. Its handling characteristics were the same as

those of a lightweight free-flying helicopter. Caro and Isley pretrained student pilots for 3.75 or

7.25 hours on the device, and compared their flight training performance with a non-pretrained

control group. Results indicated that the two experimental groups were ready to solo

significantly earlier than the control participants, though training time for subsequent evaluation

flights did not differ. Differences due to checkride grades were not statistically significant, nor

were differences between the 3.75 vs. the 7.25-hour experimental groups. However, Whirlymite

training was found to produce a significant reduction in eliminations of students from flight

training.

Valverde (1973) acknowledged that the results of some aviation TOT studies, both fixed

and rotary wing, have been inconsistent, offering several possible reasons. First, criterion

measures of performance (e.g., flight grades) were often subjective. Secondly, questions have

been raised as to the homogeneity of subjects and the use of paper-and-pencil tests as matching

criteria. Thirdly, instructor effects needed to be acknowledged. Finally, studies have differed in

their basic designs, especially with regard to the ordering of blocks of simulator and aircraft

training events. These are important issues that affect the comparability of studies, whether they

were conducted in the 1950s or the 1990s. These studies have evaluated the training device, not

the overall program. Scientific, or even systematic, evaluation of simulation-augmented training

programs remains the exception, rather than the rule. Hays, Jacobs, Prince, & Salas (1992) were

able to locate only seven helicopter simulation experiments. Their meta analysis revealed

negligible differences between simulator plus aircraft training vs. training in the aircraft alone.

Consequently, an adequate knowledge base upon which to anchor constructs for rotary wing

TOT research does not exist. A review of two more recent studies, which follows, will

demonstrate the advantages of a programmatic approach to training system research.

The MH-53J training system evaluation. Selix (1993) evaluated the effectiveness of a

rotary wing training program, based upon an integrated suite of training devices. This study

shows the advantages of a holistic, systems approach to training program development and

evaluation. Selix addressed the question of simulator/aircraft mix for the U.S. Air Force MH-53J

Pave Low Combat Crew Qualification Course. This investigation was driven by the high hourly

operational costs of the MIH-53J, an extremely complex multimission helicopter. In 1986, the

MH-53H aircraft qualification course was almost entirely aircraft-based. The increasing

complexity of aircraft systems imposed additional training demands when the MH-53H was

replaced by the MH-53J in 1990.

This became the Air Force's longest aircraft qualification course (150 training days) at a

time when flying hours were being reduced. Thus, it was decided to offload as many training

hours as possible to simulators and a suite of part task trainers. Each part task trainer was

3
dedicated to a specific sensor/ avionics subsystem. Students were trained to proficiency in the

least sophisticated training device on which a given task could be satisfactorily trained (this had

been determined through task analysis). Students did not proceed to the next level in the

syllabus until after having demonstrated proficiency. Once these skills were acquired on part

task trainers, they were integrated through crew-level practice in the Weapon System Trainer, a

high-fidelity, full-mission simulator for the MH-53J. Selix describes the 1993 course as

approximately a 50:50 mix of simulation and aircraft hours, and even more simulation-intensive

for the tactical and sensor phases of the course.

The Pave Low phase, in which integrated sensor operations were taught, was the most

demanding. In 1990 it comprised 18 two-hour aircraft sorties. Selix states that the Weapon

System Trainer's advanced capabilities allowed this phase to be restructured to 12 simulator and

three aircraft sorties. In brief, the MH-53J course is a good example of how the use of

simulators and PTTs can effectively save training time and costs in the aircraft. The savings in

this case were quite real: 1993 hourly costs for the MH-53J were $3100 vs. $800-1000 for the

Weapon System Trainer.

The training effectiveness of the MH-53J Pave Low training system is further

underscored in a study by Rakip, Kelly, Appler & Riley (1993), who conducted a survey of

experienced Pave Low crewmembers and their commanders, who were assigned to operational

units. The respondents were asked to evaluate new crews assigned to their units on the basis of

their knowledge, understanding, and execution of critical mission functions (e.g., operational

qualifications, tactical understanding, mission planning/execution). They rated crewmembers

with whom they had flown, who were trained during a time when the system was being upgraded

(i.e., non-simulator group). They also rated crewmembers who were trained during a time that

the Pave Low Weapon System Trainer was integral to the curriculum (i.e., simulator group).

The survey measured perceptions of proficiency and not actual performance-based

proficiency. Rakip et al. found that new crewmembers trained in the simulator were rated as

superior to their aircraft-only counterparts on all criteria except Night Vision Goggles abilities,

for which ratings were virtually the same. Respondents commented that crewmembers trained

in the simulator took 2-3 months to be brought up to operational standards vs. up to one year

for the non-simulator group. Likewise, the simulator group combat qualification time was

approximately 20 flight hours vs. at least 50-60 hours for the aircraft-only group. The only area

where crewmembers trained only in the aircraft showed superior performance was in their ability

to fly the aircraft, due to greater accumulated flight time. It should be noted, however, that most

of the skills required for the MH-53J course are procedural, and that respondents thought the

tradeoff was worthwhile. Rakip et al. concluded from these post hoc, quasi-experimental results

that course graduates trained in the simulator exhibited better, more highly-integrated missionrelated skills than did those who trained in the aircraft alone. Simulator-trained graduate crew

members required less time for full mission qualification than did those who only trained in the

MH-53J, when they reported to their operational units. The most important point of this study is

that the simulator need not be merely a substitute for the aircraft, but a superior training

alternative, for some mission tasks and elements.

4
ARI AH-64A Aircraft Qualification Course proof-of-concept project. Like the Pave

Low project, the AH-64A project was driven by cost considerations. The AH-64A is a complex,

expensive aircraft to operate, so major reductions in training costs should be possible if simulator

time were substituted for time in the aircraft. In this project, conducted by Wightman and

Wright (in preparation), students were required to practice and demonstrate proficiency in the

simulator on maneuver tasks from the Aircraft Qualification Course Program of Instruction

(POI). Students practiced these tasks in either the AH-64 Combat Mission Simulator, or ARI's

Simulator Training Research Advanced Testbed for Aviation, which also simulated the AH-64A.

Following these simulator sessions, students demonstrated performance of the same tasks in the

AH-64A. On the basis of this demonstration, the amount of helicopter flight training required

for proficiency was determined for each student pilot. The results were compared to those of

classmates trained within the normal aircraft-based POI. The results demonstrated that

offloading training to simulation can substantially reduce the time it takes to qualify in the

aircraft. Mean total helicopter flight training time was reduced by 20 hours. The combined

reduction in actual aircraft flight training time resulted in estimated (1997 cost basis) savings of

$70,000 per student. The simulation-based POI required approximately 56 hours in the

simulator and 25 hours in the aircraft; the traditional POI required approximately 28 hours in the

simulator and 45 in the aircraft.

The research and development efforts leading to the simulation-focused Pave Low

course, and the AH-64A proof of concept research, were in response to external pressures, both

economic and technological. Both of these systemic factors are also affecting the conduct of

Army helicopter training. The next millennium will undoubtedly see more intensive use of

simulators and part task trainers at all levels of training. This will necessitate restructuring of the

training courses to increase the use of simulators and other training devices. The primary phase

of Army aviation training is at present heavily aircraft-focused, employing simulation only for

instrument training. Consequently, it is an excellent candidate for systematic training

effectiveness research which could specify the optimal mix of simulator and aircraft training.

Improving Initial Entry Rotary Wing (IERW) Training Efficiency

Army IERW Training Program

Background. The POI is the official statement of what is to be trained. The IERW POI is

developed and administered by the U.S. Army Aviation Center at Fort Rucker, Alabama. The

purpose of the IERW course is to provide ab initio helicopter training for neophyte trainees. The

challenge faced by the aviation training community is to train both college graduates (the

Commissioned Officer Corps) and high school graduates (the Warrant Officer Corps) efficiently

and effectively. Training involves not only "hands and feet" flight skills, but also Army rotary

wing missions, within the budget and scope of the IERW POI. This has become more of a

challenge in recent years, because of a decrease in funding, particularly for flight hours.

Historical overview of the IERW POI. The IERW POI has evolved from the 175/40

(175 hours aircraft; 40 hours simulator) POI jointly developed by ARI and the Directorates of

Evaluation and Standardization and of Training and Doctrine, (presently called the Directorate of

Training, Doctrine and Simulation) at Fort Rucker, AL. In Fiscal Year (FY) 77, the 175/40

5
program was introduced to add more simulator-based training to the POI and to introduce a

second training track for Aeroscout aviators (U.S. Army Aviation Center, 1979). The 175/40

POI instituted several changes in the training regimen:

1. The number of aircraft training hours was reduced from 180 to 175 while the number

of simulator (instrument) training hours was increased from 20 to 40.

2. A second training "track" was added to the course. The 180/20 course trained only

plenary (i.e., fully rated) aviators in the utility mission. The 175/40 course added the

Aeroscout mission (flown in the OH-58A aircraft) to the POI, thus, creating a

multitracked POI.

3. An assignment algorithm was empirically developed and validated to identify

students for assignment to the demanding Aeroscout Track.

4. A procedure was developed, using the Aeroscout Algorithm, to identify students for

"turnaround" training in the AH-I Cobra Aircraft Qualification Course immediately

following IERW graduation.

Since the introduction of the 175/40 POI, the number of IERW training tracks has been

changed to four and then back to two, while the primary training aircraft has been changed from

the piston-powered TH-55 to the UH-1, and most recently, to the TH-67. However, only one of

these changes to the POI was based on research. When the Multi-Track POI was first

introduced, bringing additional training tracks into the IERW curriculum for the UH-60

Blackhawk and the AH-64 Apache, a requirement existed to expand the Aeroscout assignment

procedure to include the additional training tracks. ARI developed an algorithm to make track

assignments based on probability of success in each track. A battery of tests was developed and

data collected on students' performance in the IERW course. The assignment algorithm was

designed using discriminant analysis. This empirically-based assignment procedure was

successful in improving the throughput of the IERW course but it was abandoned when funding

for the IERW POI fell to the point where multiple mission-based training tracks were no longer

viable.

Description of the current Program of Instruction. In FY 95, Army aviation revised the

IERW POI. The lead agency was the Aviation Training Brigade at Fort Rucker. The redesign

effort, entitled Flight School 2000, was largely motivated by a substantial cut in training funds

available for the IERW course. The goal was not solely to reduce training costs. The stated

program objectives were to "make flight school more efficient" and to "increase warfighting

capability of graduates." ARI reviewed the existing POI to ascertain whether research data were

available to guide improvements in the efficiency of ab initio training. Particular attention was

paid to the use of simulation to lower training costs without a corresponding decrement in

training quality.

The POI was changed to reflect a reduction in flight and simulator hours, resulting in the

149-hour aircraft and 30-hour simulator mix. The Common Core was designed to provide the

student with basic rotary-wing operator skills and knowledge for qualification in the TH-67

aircraft. At the outset, students attend a series of classes designed to bring warrant and

commissioned officers to a common starting point. Warrant officers attend the Warrant Officer

Entry Level Course; commissioned officers attend the Officer Basic Course. Both courses teach

6
a general development curriculum and give orientation to students on their careers in aviation

and the Army.

The IERW Aviator Common Core begins after completion of these classes. The group

of students is divided into two sections, of which one becomes the AM (i.e., morning) and the

other the PM (i.e., afternoon) section. For the remainder of the training cycle, the two groups

alternate AM and PM training sessions. One group attends academics (covering topics such as

fundamentals of flight, aeromedical science, and weather) in the AM session while the other

attends flight line training. AM and PM groups switch activities during the midpoint of the

training day. On Training Day 1, classes cover aeromedical subjects emphasizing effects of

rotary wing training on the human body. Other topics concern hazardous conditions in the

aviation environment, and basic flight systems of the TH-67. Students begin TH-67 cockpit

procedure training on or about Training Day 7 and continue through Training Day 10.

On Training Day 11, they attend the flight line for the first flight session and continue through

Training Day 26. They fly 1.3 hours each day until their stage I flight exam at the 18.5 flight

hour level. Students then progress to the 22-25 flight-hour level. At that time, they are allowed

to perform solo-flights of 1.3 hours in duration with their "stick-buddy" (i.e., fellow student

pilot) acting as copilot.

Next, student pilots advance to more complex maneuvers and also refine their skills at

basic maneuvers. That training cycle continues until they reach the 58-hour level where the

stage II (end of primary phase) flight evaluation is administered. If the student pilot

demonstrates knowledge and proficiency in all maneuvers, he or she is phased into the next level

of training: Instruments.

In Basic Instruments all instruction takes place in the UBH-1H Synthetic Flight Training

System (i.e., UH-1 simulator). The student pilot receives 29.2 hours of instruction in the

simulator. At that point, the Basic Instruments (simulator) checkride/evaluation of .8 hours is

administered. If proficiency is demonstrated in all maneuvers, the student pilot progresses to

Advanced Instruments where all instruction takes place in the TH-67. The student pilot flies

18.5 hours in the aircraft, practicing all basic instrument maneuvers, after which the 1.5 hour

Advanced Instruments checkride/ evaluation is administered. Upon completion of Instruments,

the student pilot moves on the next phase, Combat Skills, where tasks more pertinent to the

tactical mission are mastered. The Aircrew Training Manual maneuver tasks which comprise the

POI (including the Combat Skills Phase) appear in Appendix A. A glossary of relevant

acronyms appears in Appendix B. A listing of Army Aviation training courses for Fiscal Year

1998 can be found on the World Wide Web at: www-rucker.army.mil.

Requirements for Enhanced Training Effectiveness

Past research demonstrated increased training efficiencies as a result of the application of

low-cost simulators and automated, adaptive trainers to the IERW POI. These findings were not

adopted by Army Aviation as modifications to the training curriculum. Prior to the mid '90s, the

Army aviation training community did not acknowledge a requirement for greater training

efficiencies. The general rule was to retain the same number of "blade hours" in the curriculum

and to resist attempts to increase reliance on simulation. A related issue is that increased use. of

7
flight simulation in training requires that additional simulators be procured. In financially

austere times, it is difficult to obtain funds for simulator development and fielding. The austerity

of the current training environment is driving the need for increased efficiency and effectiveness.

Thus, Army decision makers are increasingly interested in research supporting greater use of

simulation and other low-cost training techniques in the IERW course.

Optimization of the IERW POI. The IERW course has continued to be successful in that

a high percentage of trainees have graduated and won their wings. Still, there is little guidance

regarding the minimum number of flight hours required to graduate a capable Army aviator.

Similarly, additional questions have emerged, regarding whether increasing the use of simulation

could lower training costs without affecting the quality of the course graduate. To address these

research questions, ARI proposes an overall analysis of the IERW POI to include:

"* The current training objectives.

"* The measurement of trainee performance.

"* The mix of aircraft and simulator (including other training devices) training.

"* The integration of academic classwork and flight training.

"* The costs of each training phase and each instructional method.

"* The effect of instructor pilot attitudes and beliefs upon training effectiveness.

"* The effects of trainees' individual differences, e.g., personality, prior flight experience,

attitude toward training, specific strengths and weaknesses, learning style and disposition

toward feedback, upon training effectiveness.

" The structure of the curriculum, i.e., the limitations inherent in a fixed-time POI with

students remaining in flight classes, versus the feasibility of a self-paced curriculum in

which students' progress based on completion of training objectives rather than on a

fixed-time curriculum.

Issues to be addressed in future experiments. Based on that analysis of the current POI

and the training objectives which must be met, a series of experiments will be designed to

address the key questions in developing a more cost-effective mix of simulator and aircraft flying

training events in the IERW POI. The focus will be on empirically determining methods of

increasing training efficiency through instructional strategies such as:

"* Criterion referenced (competency-based) training.

"* Individually-paced instruction.

"* Increased use of flight simulation.

"* Increased use of artificial intelligence such as neural networks to augment training.

"* Matching students' and instructors' personalities and learning styles.

"* Implementing self-paced, self-evaluated computer-based academic instructional modules

"* Replacing fixed flight-hour training with objective-based training with continuous

evaluation to signal the instructor pilot when the student has met the training objective.

"* Functional requirements/design tradeoffs for a low-cost, training effective simulator for

both visual and instrument flight training.

8
The results of these experiments will provide the most cost-effective training milieu for

the IERW POI. However, it will not necessarily provide the optimal means of implementing

such a program. ARI personnel will need to cooperate closely with the Army Aviation Center,

as well as the Training and Doctrine, and Simulation, Training, Research and Instrumentation

Commands, to develop an approach meeting the Army's training objectives. The development

and procurement of new training devices and simulators has been a costly endeavor, though

emerging, PC-based technologies may moderate these costs. A cost-effectiveness model will

need to be developed to determine whether up-front training development costs will be

amortized within a reasonable time frame.

Measures of Training Effectiveness and their Meaning

ARI has conducted demonstration experiments, in preparation for a larger, more

integrated program of IERW research. These experiments were also performed in conjunction

with the development and refinement of research simulator testbed equipment. At this point, it

would seem appropriate to review these research efforts, in order to acquire a perspective on the

research that is planned.

In the reporting of the research that follows, and in later sections of this paper, metrics of

training effectiveness will be used with which the reader may not be familiar. At the outset, the

reader will be presented with a verbal description of the rationale behind each of three metrics.

These are the Transfer Effectiveness Ratio (TER), Cumulative Transfer Effectiveness Ratio

(CTER) and Incremental Transfer Effectiveness Ratio (ITER). Conceptually, they are quite

similar. They are, however, different tools for different jobs. What tool is used depends on what

we want to know about the consequences of training. Do we want to know the end effect of

simulator-based training, the cumulative effects at a point on the learning curve, or do we want a

"snapshot" of simulator-based training at a particular point on the curve?

Table 1 presents some notional training data, to illustrate the rationale behind these

measures of transfer effectiveness. Note that although the total number of iterations saved in the

aircraft increases with increased training, the efficiency of training diminishes. In this example,

pretraining for ten trials in the simulator would save ten iterations in the aircraft, but to save

another ten, pilots would have to train for 20 more trials, for a total of 30 iterations. These same

relationships appear graphically in Figure 1.

Table 1.

Quantitative Relationship between CTER, ITER, and Number of Trials Saved in the Aircraft

Simulator Trials 10 20 30 40 50

CTER 1.0 .80 .67 .55 .46

ITER 1.0 .60 .40 .20 .10

Trials Saved in Aircraft 10 16 20 22 23

9
1.2 25

1" " "20

0.8

15•

S0.6- EA

0.4

0.2-

0-I I I0

0 -0

10 20 30 40 50

Trials in the Simulator

-4-- CTER - ITER - * - -Trials Saved

Fig. 1. Effects of pretraining in the simulator on Transfer Effectiveness Ratio (TER) and

number of trials (iterations) saved in the aircraft.

Again, Table 1 and Figure 1 provide strictly hypothetical data to illustrate training

effectiveness metrics. The TER is a snapshot measure of the effectiveness of simulator

pretraining expressed as the ratio of pretraining in the simulator to the savings achieved in the

aircraft. It is calculated using the following equation (adapted from Roscoe & Williges, 1980):

Ci - El

TER=

EI (sim)

where C, is the number of control group (no simulator training group) iterations to criterion in

the aircraft, E, is the number of experimental group (simulator pretraining group) iterations, and

El (.sim) is the number of experimental group iterations in the simulator. TER is the ratio of

savings in aircraft maneuver iterations to the number of iterations performed in the simulator by

the experimental students (which is a training cost in that operation of the simulator and student

pilot time expend economic resources). Examining the curves in Figure 1 reveals that successive

sets of simulator iterations yield diminishing reductions in flight iterations.

10
Roscoe (1971) proposed a more dynamic measure of transfer effectiveness, the CTER:

(Yo- Yi)

CTER =

where xi represents the number of iterations performed in a simulator, yi is the number of

iterations needed in the aircraft to demonstrate criterion performance after xi simulator training,

and yo is the number of iterations that would be required in the aircraft if no simulator were

available. Each point on the CTER function represents a TER. For the first point on the CTER

function, ten simulator iterations reduced iterations needed for students to demonstrate criterionlevel behavior in the aircraft by ten in flight iterations for a TER of 1.0. CTER is similar

conceptually to TER. TER measures the effects of simulator pretraining for a particular number

of trials or iterations; CTER is concerned with the cumulative effects (see Figure 1). The main

distinction is temporal. The training researcher is concerned with the CTER when he or she is

interested in the effects of training along a continuum of simulator trials as illustrated in Figure

1. The CTER conveys more information than the TER in that it illustrates the actual shape of the

transfer effectiveness curve, whereas the TER simply conveys the effects of training at one

particular point (in the illustration, after 40 simulator trials, 22 trials are saved in the aircraft, so

TER = .55).

By contrast to CTER, the ITER isolates the savings in expensive aircraft training as a

consequence of each successive increment set of lower cost simulator training iterations (Roscoe,

1971). The ITER is defined as:

Yt-k- Yt

ITER =

Xt - Xt-k

where:

x is time (or iterations) on the simulator

y is aircraft time (or iterations) to criterion

t is the te measurement point

k is the range incrementer such that t-k gives the

range over which the ITER is calculated, 10 in this

example.

The ITER is expected to be a negatively decelerating function and its parameters are

expected to vary widely from maneuver to maneuver (Povenmire & Roscoe, 1973). A researcher

would use ITER to predict the amount of savings expected in the aircraft, if the student were to

train for x more iterations (or trials) in the simulator. The fundamental questions he or she would

ask are: would it be worthwhile to practice for one more trial, and when do we reach the point of

diminishing returns? ITER is separately calculated for each block of maneuvers to determine

the overall cost savings attributable to that additional set of simulator pretraining. As illustrated

in Figure 1, increasing the number of simulator iterations from 20 to 30 (providing ten additional

11
simulator iterations) produces an additional savings of four iterations (or trials) in the aircraft,

yielding an ITER of .4 at this point of the function.

The applications of these variables to models of simulation and training efficiency and

cost-effectiveness will be discussed in more detail in a subsequent section. The purpose of

presenting these metrics at this point was to help the reader to understand the rationales

underlying them, and why more than one is needed.

An Optimal Mix of Aircraft and Simulation Training

The current section will take a more in-depth look at the metrics of training effectiveness

previously introduced; namely, the TER, CTER, and ITER. Prior allocations of aircraft and

simulation training resources to IERW training have been made by experienced training

managers who have used expert judgment to structure the PO. This method will produce a

successful training program, especially if the training schedules for each training phase are

adjusted iteratively in response to changing demands and changes in normative performance over

time. However, a judgmentally-based method will not necessarily produce the most cost

effective curriculum. Mathematical models have been developed to quantify the increased

training effectiveness and efficiency that results from the increased use of simulation in flight

training. These models derive from the more general cost-effectiveness analysis approach. In

the current application, the purpose of the model is to identify the most cost-effective allocation

of aircraft and simulator training hours.

Determination of the most cost-effective training schedule requires that the tradeoff

between the lower cost of simulator-based training, especially automated simulator-based

training, be balanced against the greater number of training hours required to meet the same

proficiency criterion. The assumption that the overall TER for a training simulator will be less

than 1.00 is supported by numerous empirical research efforts. Accepting this assumption, the

most time-efficient way to train students to criterion is 100% aircraft training while the most

cost-efficient POI will involve mostly simulator-based training. There are exceptions, however;

optimized training programs may yield greater TERs (see Diehl & Ryan, 1977).

The reader should recall that CTER is a ratio of savings that results from simulator

pretraining to the "cost" of that pretraining in the simulator. It is the same concept as the TER,

except that Roscoe views CTER as a cumulative function relating the savings in aircraft training

to stepwise increases in simulator training costs. The TER values were based on training

students to the same performance criterion in the simulator and the aircraft, and comparing them

with an aircraft-only control group. Instead of looking at the cumulative or incremental effect of

varying amount of simulator pretraining, the TER approach pre-established the criterion for

acceptable performance and viewed only the effect of simulator pretraining to that level of

proficiency. Diehl and Ryan (1977) found that, with a properly designed training syllabus,

simulation-based training programs yielded a median CTER of 1.00 vs. .43 for those programs

without a simulator syllabus. These findings lend cogency to the argument advanced by training

researchers (Nullmeyer & Rockway, 1984; Salas, Bowers & Rhodenizer, 1998) that effective

training depends as much (or more) on the training program as on the simulator.

12
yo

•r :I

4- ,i 

En M

Amount of simulator training

Fig. 2. Hypothetical relationship between the amount of simulator pretraining (x)

and subsequent required aircraft training (y) (Adapted from Bickley, 1980).

As Roscoe points out, the CTER is not constant across varying amounts of simulator

training but will decrease as the value of n increases. Thus, the empirically determined value

for CTER will only be valid for a small range of values in the region close to a given NI. As one

can see in Figure 2, there is a point where continued training in the simulator will not increase

training savings in the aircraft. Bickley (1980) recognized this limitation and noted that, while

the a measure of the training effectiveness of all the x values could be calculated by regressing

CTER on various experimentally-determined values of x, a simpler and more direct approach is

to regress y on x. That is, a prediction model can be developed that will relate the relatively lowcost units of training in the simulator (x) to the relatively high-cost units of training in the aircraft

(y) as required to meet the training objective. When x = 0 (no simulator pre-training), Y will be

equal to the CTER's yo. As x increases, y is expected to decrease but not at a constant rate. That

is, it has been empirically demonstrated that increasing amounts of simulator training result in

decreasing training effectiveness in that the amount of aircraft training time (or iterations) saved

is less as simulator pre-training increases. Since current simulators are not perfect replicas of the

aircraft, there will always be some amount of aircraft training required to meet the training

criterion, regardless of how much simulator pre-training is used. Bickley (1980) termed this

amount of aircraft training c.

Figure 2 presents the hypotheticalrelationship between x, y, and c, depicted as a

curvilinear function. As simulator pretraining increases, the aircraft training required to meet the

criterion decreases, thus, yo - __ represents the potential savings in aircraft costs that will be

realized as the result of simulator pre-training. Bickley made the assumption that, as x increases,

13
the rate at which y decreases is a constant proportion of y - c. Bickley (1980) went on to apply

this model to a reanalysis of the fixed wing simulator training effectiveness data reported above

(Povenmire & Roscoe, 1973). The model showed that 44.3 hours were required for the average

student to reach the criterion with zero simulator pre-training and that 37.6 aircraft hours are

required to meet the criterion even if more than ten hours of simulator training were provided.

These two cost-analysis tools, the CTER and ITER, were presented by Povenmire and

Roscoe (1973) for students working toward their Private Pilot Ratings in a Piper Cherokee fixedwing aircraft. In their analysis, the simulator cost of operation of the (GAT-1) simulator was .73

times the cost of the aircraft. Their research found that the simulator was cost effective until

approximately the fourth training hour at which point the ITER fell below 0.73. In IERW

training, the cost of simulator operation is a much smaller proportion of the aircraft cost...more

like $75.00 vs. $400.00 per hour so the ITER could drop to a value below 0.20 and still result in

cost-effective simulator-based training.

Training Management

One means of improving training effectiveness and efficiency is through managing

trainees individually rather than as members of a flight class. A computer implemented Training

Management System could be developed using artificial intelligence to optimize training

efficiency. By incorporating individually managed training schedules, each trainee can progress

through the curriculum at his or her optimum rate. There are costs associated with flexible

scheduling, such as assigning aircraft, training devices and instructors. These costs, however,

should be modest when using computer based databases. The goal behind the Training

Management System is to provide the instruction required for each student pilot to meet training

objectives at the least cost. One requirement underlying its implementation is to develop

measures to determine when the student pilot has met the criterion for each flight training

objective. This suggests the value of criterion-referenced training using objective performance

metrics which are sensitive enough to determine when the objective has been met. This

performance evaluation could be accomplished automatically in the simulator but would

probably have to be judged against published benchmark criteria by the instructor pilot, for

training evaluation in the aircraft. If the student pilot's daily performances were entered into the

system, then each student pilot's current training status would be known. The system could then

schedule students, instructor pilots, and aircraft efficiently. Similarly, academic objectives could

be scheduled and managed based on individual student pilot requirements. Students could attend

classroom presentations together but could be scheduled by the Training Management System

for remedial tutoring when daily performance evaluations indicated that they had not mastered

the academic assignments.

The general instructional philosophy behind the managed training concept would be to

train academic and procedural skills to criterion before the student pilot would begin flight

training dependent on these skills. Procedural and basic flight skills training would be conducted

in the part task trainer or simulator first, then validated in the aircraft. For those skills for which

the student pilot demonstrated proficiency in flight, no further aircraft training would be

required. For those skills in which the student pilot was not fully proficient, additional training

would be prescribed by the Training Management System, based upon empirical results with.

14
previous students. It would select the training alternative that had historically demonstrated the

greatest probability of successful remediation.

Trainee Performance Measurement

Measurement challenges. An empirically-based program to improve the effectiveness

and efficiency of aviator training is contingent upon reliable and valid performance

measurement. Conventional measures of student performance during training are: course

pass/fail, setbacks in the course, academic grades, flight training grades, checkride grades,

instructor pilot "putup" grades (the checkride grade the student pilot is expected to receive on an

end-of-phase checkride), and class standing. Training system and trainee proficiency measures

which are possible but are not currently in use in the IERW course consist of flight hours to

criterion (in a criterion-referenced course, not a lockstep course); maneuver iterations to

criterion; objective aircraft performance measures such as flight data recordings; similar

objective data gathered during "flight" in a simulator; peer and student pilot self-progress ratings.

Whatever measures are selected, they must be demonstrated to be reliable, valid, and sufficiently

sensitive to identify relatively small changes in the efficiency of training. Such marginal

increases in training efficiency may represent substantial dollar savings when multiplied across

thousands of trainees.

The potential of criterion-based, empirically-derived performance measures was

championed in a study by Nullmeyer and Rockway (1984), in which the researchers evaluated

the effectiveness of the C-130 Weapon System Trainer, a full-mission simulator with a visual

display system. Although the researchers found the Weapon System Trainer superior overall to

an earlier, non-visual instrument simulator, they were nevertheless forced to conclude that it was

"employed in a less than optimal manner." Its potential as a training device was unknown. Their

recommendations were directed not at changes in its design, but in the training program

supporting it. Among these recommendations were the abandonment of a highly-structured

training regimen, and the adoption of a proficiency-based training program in which students

would demonstrate proficiency in the simulator for selected tasks. Along with requiring

proficiency, training would be tailored to the needs of the individual student. The student pilot

would not be required to train in the simulator for a preset number of hours, but to demonstrate

proficiency on the training syllabus in the simulator, and, once having done so, to proceed to

flight line to demonstrate what had been learned.

Flight grades as predictors. There is even some question as to whether the performance

of the student pilot during formal school training is the best criterion for the evaluation of the

POI. The ultimate criterion for IERW training is not to produce course graduates, it is to

produce aviators capable of safely and effectively executing Army missions in the field. The

question can be asked whether the best performers during IERW training are also the best

subsequent aviators after being assigned to aviation units in the field. Research by Bale, Rickus

& Ambler (1973) which followed Naval aviators after graduation, showed that flight training

grades did not effectively predict later operational performance. The conclusion reached by Bale

et al. (1973) was that training grades do not assess all of the mission-oriented aviator skills and

abilities that determine success in the field. In the current effort, it is proposed that IERW

training developments be referenced against criteria in the "schoolhouse" (IERW) and in the.

15
field. However, it must be realized that the latter performance measures will be more costly to

obtain, both in dollar costs and time costs, since they will involve mission-related tasks such as

gunnery, troop insertions, lift operations, and coordination of battle plans with other units. Given

the high costs of IERW training, it is likely that field performance measures will be costeffective.

A research project should be performed to investigate these two classes of performance

measures to determine whether training for success in the schoolhouse is commensurate with

training for success in the field. Specifically, if any measures of student pilot performance are

negatively related to subsequent mission-performance in the field, they need to be identified and

studied further to identify the underlying behavioral differences between schoolhouse criteria

and field criteria. This research will necessarily be longitudinal and time-consuming.

Using initial entry grades as performance measures is problematic for several reasons. In

the case of the Army IERW course, the daily training grade is a letter (A, B, C, or U) while the

putup and checkride score are numerical, based on a 100 point scale. The modal daily grade is a

B. If an A or a "downgrade" (C, or U) is given, the instructor pilot is required to explain the

grade with a write-up on the grade slip. Thus, the grade of B is parsimonious with regard to the

instructor pilot's time. Past research efforts have shown that there is too little variance in the

distribution of daily training grades (the "swarm of Bs") to make them effective as either

criterion measures or as predictors of later performance.

Basic Qualities. Army IERW training uses the concept of Basic Qualities to reference

the individual student pilot's underlying knowledge, skills, and abilities that are judged by the

instructor pilot to predicate the assignment of an upgrade or downgrade. Each student pilot is

evaluated on each of the six Basic Qualities during each training flight. Those that are postulated

to underlie IERW training performance during Primary Phase are:

* Flight Safety

0 Attitude

0 Knowledge of Procedures

0 Co-Ordination (sic)

Planning and Judgment

Division of Attention

These are hypothetical constructs that are postulated to explain an student performance

during training. They are not empirically derived from trainee performance using a quantitative

procedure, but instead reflect traditional beliefs in Army aviation. Previous analyses of the

distribution of Basic Qualities grades has not shown them to be effective as criterion measures or

as predictors of future training performance (Dohme, 1979). They are worthy of further analysis,

however, in that empirically derived Basic Qualities may provide a basis for the knowledge,

skills, and abilities that contribute to IERW training performance. Other researchers have

identified additional weaknesses in the use of IERW training grades as criterion measures.

Pilot Performance Description Record (PPDR). Researchers have long been cognizant of

the deficiencies mentioned above. This has led to attempts to objectify and standardize the

16
performance evaluation process. Greer, Smith & Hatfield (1962), pretested the PPDR, a

performance rating system consisting of scales referenced to the Aircrew Training Manual

standards for basic flight maneuvers. The rationale behind the PPDR was to minimize the

subjectivity and variability between instructor pilots that the investigators had found to be

inherent in the use of conventional flight grades and Basic Qualities. Results of the pretesting

were promising. The use of the PPDR showed evidence of improved reliability when compared

to grades, and that it was demonstrated to be a good tool for diagnosing specific sources of an

student pilot's flight deficiencies. Furthermore, check pilots highly familiar with the PPDR

showed higher reliability in their scoring of students than did those not so familiar. The PPDR

was used operationally at Fort Wolters, Texas, for five years. Despite its demonstrated

superiority to conventional flight grades, its use was discontinued when all IERW training was

consolidated at Fort Rucker.

Basic Flight Instruction Tutoring System. Koonce, Moore & Benton (1995), describe

the Basic Flight Instruction Tutoring System as a PC-based system that has both a tutorial and

performance measurement function. It is criterion-referenced in that it requires the student to

meet a pre-set standard for one maneuver before proceeding to the next one. It is tutorial,

teaching the student how to perform a given maneuver, then demonstrating the maneuver,

measuring performance while the student performs it, and presenting feedback when the student

exceeds certain pre-set parameters. The feedback (e.g., airspeed low) instructs the student pilot

how to get back into the specified performance envelope while performing the maneuver. After

completion of the maneuver, the student pilot is told whether or not performance was

satisfactory. The student pilot must perform all maneuvers in a particular lesson module to

standard before being allowed to move on to the next, more advanced, module. The system

captures error scores for each variable comprising a particular maneuver. In this sense it is a

performance measurement system.

Koonce et al. demonstrated the effectiveness of the Basic Flight Instruction Tutoring

System by pretraining ten entering students in an FAA Part 141 training school on it, and

comparing their performance on the same maneuvers in the final checkride in a Cessna 152.

Seventeen control students progressed through the course unassisted. Performance of the

pretrained students was superior to that of the controls on a number of dimensions, notably flight

hours to solo (p <. 001). Instructor pilots noted also that pretrained students required fewer

attempts to land the airplane, and seemed to impact the ground more softly than the controls.

The investigators concluded that the Basic Flight Instruction Tutoring System is a criterion-based

training system that minimizes the cost of instruction, and has also demonstrated TOT to the

aircraft. It was developed under a Small Business Innovative Research grant, and has been

adopted and operationally employed by a commercial flight training academy. It could possibly

alleviate problems inherent in the subjectivity of evaluating performance in the simulator.

The Semi-Automated Flight Evaluation System was derived from the Basic.Flight

Instruction Tutoring System (Benton, Corriveau & Koonce, 1993). It is PC-based on-board

performance evaluation and tutoring system that is intended to be easily installable and

removable. This system should be of great interest to researchers such as the current authors.

Performance measurement systems have been tested in simulators, but there are no operational,

on-board systems in aircraft. Since the system is configured from commercial, off-the-shelf.

17
components, it should be a cost-effective alternative to more expensive hardware, namely, digital

flight recorders. Devising a common set of performance measurement parameters for both

simulator and aircraft, and demonstrating their reliability and validity, would be a significant

milestone in the development of aviation training technology.

Tradeoffs and Recommendations

Several authors have demonstrated the utility of designing a flight training curriculum for

optimal cost-effectiveness using mathematical models to calculate the most economical mix of

aircraft and simulation time (Derrick, Tomczak, Nullmeyer & Burright, 1992). This approach is

recommended for the TERW POI. Westra and Lintern (1995) recommend the application of an

exponential decay model to determine the ITER for the optimal use of simulation. This type of

mathematical model requires that the amount of aircraft training (to criterion) for each maneuver

be known, and that the operational costs of the aircraft and the simulator be calculated. In the

case of an experimental training simulator, the operational costs are unknown and must be

estimated. Similarly, the operational cost of the training aircraft is dependent upon the

assumptions that go into the cost model. For example, does the model include:

"* Cost of the instructor pilot?

"* Cost of operating the stage (training) field?

"* Cost of providing emergency medical and fire personnel?

"• Cost of providing aviation weather services?

"• Cost of bussing students to the stage field where the aircraft is located?

Calculation of the cost of training to criterion in the aircraft with no simulator pretraining

is difficult to determine in the case of the IERW course since the current lockstep curriculum

does not allow relating training costs to proficiency. The curriculum is fixed for each training

day and so the cumulative training time to criterion for a given maneuver is not known. ARI has

surmounted this problem in the past by counting maneuver iterations to criterion (three

successive maneuvers within published standards). While this criterion measurement is

arbitrary, it is, at least, repeatable and provides for an empirical estimate of training costs. Costeffectiveness data will need to be estimated based on research results and then refined based on

the accumulation of daily training data to precisely determine the most cost-effective mix of

simulator and aircraft for each specific training maneuver.

The sections to follow review prior ARI research on increasing the cost-effectiveness of

flight training. We propose new research leading to a validated IERW POI that will meet

increasingly demanding training objectives while reducing training costs. A research master

plan is proposed for FY 98, supporting empirical research to be conducted in FY 99. The goal

of the research is development of an empirically-based IERW curriculum by FY 2000.

18
Army Research Institute Ab Initio Training Effectiveness Research

Background

As we have learned, research has addressed the training effectiveness of entry level flight

training programs. Most of this research was conducted over twenty years ago; simulators have

evolved dramatically since that time. Most of it concerned fixed wing ab initio students and may

not apply to rotary wing training. This section reviews ARI research for experimental rotary

wing training programs. We will attempt to draw conclusions regarding the contributions of

prior research to the development of a more efficient IERW POI. In addition, benchmarks

established by prior research will serve as heuristic devices for the research program under

design. We will also examine what is known and what remains to be determined in the design of

an optimally cost-effective IERW training POI.

In the late 1980s, ARI began to answer the questions posed by the Army Aviation Center

regarding simulation as a "training efficiency multiplier" by developing a low-cost simulator

specifically designed to support IERW training. Since the newly developed simulator had

unknown training capabilities, it was necessary to determine whether training in the UH-1

Training Research Simulator provided any TOT to the UIH-1 aircraft before it was evaluated as a

substitute for the aircraft. The TOT paradigm was chosen as the best metric to evaluate the

effectiveness of the simulator since it provided a separate estimate of training effectiveness for

each maneuver and could serve to estimate the cost-effectiveness of simulator-based training.

Overview

Four TOT experiments were performed, using the Training Research Simulator. These

experiments were embedded in the IERW course structure, and in all of them IERW students

served as participants. In the four experiments that follow, eight Primary Phase maneuvers were

selected for evaluation: takeoff to hover; hover taxi; hovering turns; hovering autorotation;

normal takeoff; traffic pattern; normal approach; and landing from hover. There were specific

variations among these experiments; hence, they are reported separately. In overview,

Experiment 1 was a process evaluation of the simulator, to investigate TOT to the UH-1 aircraft.

Participants were student pilots who had completed Primary Phase training in the TH-55 aircraft.

Experiments 2 and 3 were conventional TOT experiments, employing neophyte trainees.

Experiment 4 was a substitution experiment in which seven hours of UH- I1 aircraft time was

replaced with nine hours of simulator time. Weather conditions on the flight line reduced the

number of aircraft flight hours from the originally planned nine to seven.

Experiment I

Introduction. The UH-1 Training Research Simulator was evaluated by selecting a

random sample often flight students who had already completed the Primary Phase of IERW

training in the TH-55 helicopter. All students had soloed the aircraft and had demonstrated

proficiency in a variety of emergency procedures as well as in the common rotary wing flight

maneuvers. It was hypothesized that all these student pilots, who had an average 46 hours flight

training, would demonstrate positive TOT from the simulator to the UH-1 since they had already

19
demonstrated basic helicopter piloting skills. Students who had also graduated from TH-55

Primary Phase of IERW served as controls. They went directly to the UH-1 aircraft without any

intervening training. The hypothesis was that the simulator-trained students would meet the

criteria for soloing the UH-1, as written in the Contact (UH-1) Phase POI, in fewer maneuver

iterations than would the control group.

Method. Instructor pilots were all qualified to teach Primary Phase and were given two

students each for 1.0 hours of training per day in the simulator to the same standards they would

use on the flight line. The control group students were selectively matched to the experimental

students on factors known to influence performance in the IERW course: Flight Aptitude

Selection Test scores, Primary Phase flight grades, rank, and age. The matched pair flew as stick

buddies in the UH-1 with the same instructor, to reduce variance in the experimental design. To

ensure a fair and unbiased evaluation, instructor pilots were not informed which of the paired

students had been pretrained in the simulator.

Findings. There were three possible outcomes for each of the eight maneuvers: the

simulator could demonstrate: (1) positive TOT; (2) no TOT; or (3) negative TOT. The actual

results of the experiment provided all three outcomes depending upon the particular maneuver.

Given the primitive nature of the visual imagery from the low-cost image generators used at that

time, this result was not surprising. Overall, the evaluation showed an average TER of .22.

TER values from Experiments 1 through 4 are presented in Table 2. The data show that

six of the eight maneuvers demonstrated a moderate positive TOT, one (normal approach) was

essentially zero, and one (normal takeoff) demonstrated a negative TOT. The average TER

across all eight maneuvers was .22.

Table 2

Transfer Effectiveness Ratios for Transfer of Training Experiments

Aircrew Training Manual Maneuver Tasks

Experiment T/O to Hover Hover Hover Normal Pattern Approach Land from

Hover Taxi Turns Auto. T/O Hover

1 .21 .41 .24 .31 -. 13 .42 .07 .24

2 .15 1.0 -.03 -.35 .54 .38 .48 .67

3 .25 .73 .70 .48 .43 .68 .31 .40

4 .32 -.06 .61 .21 .00 .09 .24 .49

Combined .23 .52 .38 .16 .21 .39 .28 .45

20
Table 3

Iterations Transfer Ratios for Transfer of Training Experiments

Aircrew Training Manual Maneuver Tasks

Experiment T/O to Hover Hover Hover Normal Pattern Approach Land from

Hover Taxi Turns Auto. T/O Hover

1 4.8 2.5 4.2 3.2 n/a 2.4 14.3 4.2

2 6.7 1.0 n/a n/a 1.9 2.6 2.1 1.5

3 4.0 1.4 1.4 2.1 2.3 1.5 3.2 2.5

4 3.1 n/a 1.6 4.8 n/a 11.1 4.2 2.0

Combined 4.7 1.6 2.4 3.4 2.1 4.4 6.0 2.6

The same data are presented in another form in Table 3. Iterations Transfer Ratio (ITR)

values are the reciprocals of the TERs. They provide the number of simulator iterations required

to save one iteration in the aircraft. Thus, ITRs serve as a practical index of the amount of TOT

expected in an applied training scenario.

The performance metric entailed iterations to criterion rather than the more customary

flight hours to criterion. This was because the exact number of hours each student pilot required

to meet each published training objective was unknown. The IERW POI follows a fixed-hour

curriculum in which every student pilot gets the same training regimen. If the POI calls for 0.5

hours of hover training on a given training day, each student pilot gets that amount of training.

Therefore, it is impossible to review the students' logbooks and determine how many hours were

required to meet the training standard for a given maneuver.

Experiment 1 demonstrated that the low-cost simulator produced positive TOT to the

aircraft in students already familiar with the basic Primary Phase IERW flight maneuvers. Based

on these results, ARI decided to upgrade the image generators and to make improvements to the

aerodynamic simulation model before conducting additional TOT research.

Experiment 2

Additional research evaluations were completed, following the TOT paradigm outlined

above. A random sample often students with no prior flight experience was selected. The nonselected student pilots in the same flight class served as controls. The experimental

participants' training schedule was altered to provide for two weeks simulator-based training

prior to beginning in-aircraft training on the flight line. The students were trained on the same

eight target maneuvers. Training was conducted in the UH-1TRS by qualified Primary Phase

instructor pilots to the published maneuver criteria. An average of 8.7 training hours was

required per student to meet the criteria (contrasted with 6.3 hours per student pilot for the

students in Experiment 1 who had prior training). The TERs for experiment 2 are presented in

the second row in Table 2. The related ITRs are presented in Table 3.

The results from Experiment 2 revealed an average TER of .36 with two maneuvers

(takeoff to hover and hovering turns) displaying essentially zero TOT and one (hovering

21
autorotation) displaying moderate negative TOT. It was not known if the improvement in

average TER in Experiment 2 was the result of improvements to the UIH-1TRS or the result of

using neophytes as research subjects. With neophytes, the degree of improvement in flight skills

would be expected to be large for the early training hours compared to the trainees from

Experiment 1 who had already completed the Primary Phase of IERW training.

Experiment 3

Further improvements were made to the UH-1 Training Research Simulator. The image

generators were upgraded to improve the quality of the out-the-windows imagery. The frame

update rate was increased from 30 to 50 Hz, and greater level of detail management and

texturing capabilities were added. A visual database was developed to model the Fort Rucker,

AL training Area of Operations. The same experimental procedure was followed as that used in

Experiment 2.

TER values for Experiment 3 are presented the third row in Table 2 and the derived ITRs

in row three of Table 3. In this experiment, a positive TOT was observed for all eight training

maneuvers. The overall average TER for Experiment 3 was 0.45. The average ITR for these

maneuvers was 2.32:1 which suggests that, under these conditions, two and one third maneuvers

in the low-cost simulator will save one maneuver on the flight line. Given the differential in

hourly costs of operating the aircraft and the simulator, approximately $700 vs. $75 respectively,

the simulator was considered to be an effective trainer.

Experiment 4

Background. At the request of the Commanding General at Fort Rucker, ARI performed

a fourth experiment to evaluate whether the low-cost simulator could substitute for aircraft flight

training. Since the results of Experiments 2 and 3 suggested that the average student pilot meets

the POI training criterion for all eight target maneuvers in approximately nine flight hours,

Experiment 4 (The Substitution Experiment) replaced nine hours of "blade time" with nine

hours of simulator time. The simulator was configured the same as it was for Experiment 2.

Dependent measures. The dependent measures of interest in Experiment 4 were not

TERs. They were instead measures of student pilot progress in training. It had already been

demonstrated that the Training Research Simulator produced positive TOT to the aircraft. The

issue in Experiment 4 was whether training in it could substitute for aircraft training on an hourfor-hour basis. The simplest measure of equivalency is whether any students in either the

experiment or control group were eliminated from or set back to an earlier phase of training.

There were no setbacks or eliminations in Primary Phase for either group. When flight training

grades were compared at the end of Primary Phase, the control group had higher training grades

than the experimental group; the difference was not statistically significant.

Findings. TERs and ITRs were calculated for the ten experimental students. These are

presented in Tables 2 and 3. A positive TOT was achieved on six of the eight target maneuvers;

there was essentially no transfer on normal takeoff, and a slightly negative transfer on the hover

taxi maneuver. The average TER for Experiment 4 was .23. Overall, the substitution

experiment was considered a success because of the following:

22
1. No experimental group students were set back or eliminated.

2. There was no significant difference in Primary Phase grades.

3. Net savings in training cost were approximately $36,000 for ten trainees.

4. Overall positive TOT was demonstrated.

Summary and Conclusions: Experiments 1-4

The four TOT experiments used random samples of 40 Army students as they progressed

through IERW flight training. The remaining students in the same classes served as control

participants. Conclusions should be generalizable to current and future Army trainees. The last

row in Table 3 displays average ITR values for the four combined experiments. While each

individual experiment evidenced considerable variability across the eight target maneuvers, the

combined results show that, overall, the simulator produced positive TOT on all maneuvers.

Variability across maneuvers and experiments was probably attributable to changes in simulator

configuration, differences in instructor pilot experience, skill level, and attitude toward

simulation, and to differences in student pilot abilities.

In summary, a low-cost training simulator like the Training Research Simulator has

considerable potential for training IERW students in the basic flight skills at greatly reduced

costs. The simulator was dismantled when the Army changed the Primary Phase training

helicopter from the UH-1 to the TH-67.

The technology of the Training Research Simulator is dated. A training simulator like it

can be constructed at relatively low cost using PC technology. At this writing, a high-end PC

with a visual image accelerator card can produce high quality imagery for out-the-window

scenes at a fraction of the cost of the image generators employed in these experiments. ARI is

experimenting with this technology to stay abreast of developments that support the future

generation of low-cost trainers. A TOT experiment employing a contemporary low-cost, PCbased simulator follows.

TH-67 Skills Trainer Transfer of Training Experiment

Introduction

The TH-67 Skills Trainer (Frasca International, Inc., May 1995) is the low-cost

simulator used in the present experiment. Its cockpit is modeled after that of the TH-67

helicopter, with dual flight controls, flight instruments, and system indicators. The avionics can

be used to train instrument flight rules (IFR) operations. A variety of environmental conditions,

including wind speed and direction, turbulence, visibility, cloud ceiling, and day/dusk/night

illumination can be simulated. Various engine, electrical, hydraulic and mechanical failures can

also be simulated. The simulator provides helicopter sound cues, including engine, transmission,

main rotor, and wind noise, plus warning tones. The out-the-window view is projected onto a

screen in front of the cockpit. The visual field, as configured for this experiment, was 330

horizontal by 25' vertical. All locations in the visual database (e.g., airfield, helipad, taxi lanes)

are accurately modeled and internally consistent in terms of latitude, longitude, and magnetic

compass orientation.

23
Method

Overview. A subset of the flight tasks was selected from the IERW PO. The maneuver

tasks were: takeoff/landing to a hover, hovering flight, hovering turns, normal takeoff, normal

approach, traffic pattern flight, and hovering autorotation. Instructor pilots trained these tasks to

a criterion of three consecutive successful iterations in the simulator. Then instructor pilots

trained the same tasks to criterion for the same students in the aircraft. In short, a modified TOT

design with two groups (simulator/non-simulator) was used. Participants were students from

primary phase IERW classes. They were matched on Flight Aptitude Selection Test scores. One

member of each matched pair was assigned randomly to the simulator group, the other to the

control group. The simulator group trained to criterion in the flight simulator, then transferred to

the flight line for training to criterion in the TH-67 helicopter. The control group received no

simulator training prior to being transferred to the flight line for training to criterion in the

TH-67. Performance measurement and data collection took place at the flight line during Phase

II primary flight training.

Evaluation. Because of technical difficulties in collecting data in the aircraft, the actual

number of iterations to criterion were not measured in the aircraft. Instead, the number of hours

required for the student pilot to attain proficiency at critical milestones comprised the principle

dependent variable.

Simulator phase. Training took place at the ARI. Each of the six simulator group

participants received one 45-minute simulator training session per day for the first ten training

days. This is 7.5 hours total of simulator training time. Each was instructed individually by his

or her assigned instructor pilot. Each instructor pilot trained his two student pilots in consecutive

sessions. Simulator sessions were partially counterbalanced across training days so that no

instructor pilot or student pilot consistently got the late time slots. During training days 1

through 10 members of the control group were occupied with traditional preflight academic

activities. This was a necessary part of the TOT paradigm.

Flightline phase. Six contract evaluators were selected to serve as training monitors and

data collectors during the flightline phase of the experiment on training days 12 through 17:

three evaluated members of the simulator (experimental) group; the other three evaluated

members of the control group. Contract evaluators monitored training during each flight period.

They recorded the amount of time spent demonstrating and practicing each specified flight task,

and the number of demonstration and practice trials for each flight task. The criterion for any

particular task is defined as performing it to standards three consecutive times.

Results

Pre-solo phase. The tvo major dependent variables were the overall grade the student

pilot received on his or her checkride, and the total flight hours to complete a satisfactory

checkride. Table 4 presents the means and standard deviations (SDs) for these two measures of

performance.

24
Table 4

Time to Flight Evaluation Checkride and End of Pre-Solo Phase Flight Grade

Measure Mean SD

Experimental group

Time 17.50 .50

Gradej 87.50 3.73

Control group

Time 19.22 2.86

Grade 82.50 6.98

Close examination of Table 4 reveals the difference in variability on these two criteria

between the experimental and control groups. For both, there was much more variation in

performance in the latter group. For this reason, a non-parametric test, the Mann-Whitney U

Test, was used. The Experimental-Control comparison for final grade was not statistically

significant (p < .20). The comparison for time to complete successful checkride was significant

(p_< .04), and is presented in Table 5.

Table 5

Mann-Whitney U/ Wilcoxon Rank Sum Test Comparing Time to Complete Successful

Checkride for the Experimental (Simulator) and Control (Non-Simulator) Groups

Group Mean Rank Sum of Ranks z

Experimental 4.33 26.00 -2.10 .04

Control 8.67 52.00

These results imply that pretraining in the low-cost simulator saved training time in the

aircraft. It seems, then, that time to the criterion (i.e., successful completion of an evaluation

checkride) was a sensitive measure of training effectiveness. The data also suggest that

pretraining in the simulator reduced performance variability in the experimental group. Flight

grades did not appear to be a highly sensitive index of the effects of simulator training.

Nonetheless, the experimental group did outperform the control group on one maneuver task,

hovering flight (J = 5.00, Z = -2,38, p < ,02). No other differences were significant.

Unsupervised solo phase. The two groups were followed up, and total flight hours at the

completion of the unsupervised solo were collected for both groups of students. As before, data

were analyzed via a Mann-Whitney Uj-test. The results of the analysis, significant at p < .007,

are presented below in Table 6. The means and standard deviations for the two groups are

presented in Table 7.

25
4

Table 6

Mann-Whitney U/ Wilcoxon Rank Sum Test Comparing Time to Complete Unsupervised Solo

for the Experimental (Simulator) and Control (Non-Simulator) Groups

Group Mean Rank Sum of Ranks z P_

Experimental 3.67 22.00 -2.72 .007

Control 9.33 56.00

Table 7

Time to Completion of Unsupervised Solo Flight, for Experimental and Control Groups.

Group N Mean hours SD

Experimental 6 28.70 1.716

Control 6 36.40 4.469

Mean total hours at the end of the unsupervised solo were significantly fewer for the

experimental group. The SD was also substantially lower, which necessitated the use of the nonparametric test. The lower variability could be indicative of more uniformity in performance

among those students pretrained in the simulator. Interestingly, there was no variation in the

number of hours to supervised solo for the experimental group; all were ready by 22 total flight

hours. However this was not the case for the control group (M = 23.98 hours- SD = 3.31).

Experimental group participants trained to criterion in the simulator. Time to criterion

did not correlate significantly with total time to completion of the unsupervised solo (rs = -.46).

However, the Spearman correlation between total simulator grade and the same criterion variable

was significant (r, =-.81, p < .05). The small sample size makes interpretations of these

correlations difficult.

Future Research Directions

Students used the Skills Trainer as an adjunct to the traditional IERW training syllabus;

nothing else was changed. In future experiments, the use of simulation will be evaluated as an

integral part of the total IERW training system. For example, some of the issues that ARI plans

to investigate are the use of simulation to integrate and enhance those skills already learned in

the aircraft, and the degree to which practice in the simulator can substitute for practice in the

aircraft. Likewise, breakthroughs in PC technology have made possible new self-monitoring,

adaptive training devices. ARI is currently exploring one development, based upon artificial

intelligence technology.

26
Automated, Adaptive, Simulator-Based Training

Overview

The experiments outlined above demonstrated that low-cost simulation can be an

effective tool in ab initio training. This finding led the ARI research staff to propose further

inroads by capitalizing upon the ever-increasing power of low-cost PCs. Specifically, we were

interested in the possibility that PCs and intelligent software applications could be used to train

neophytes in lieu of the traditional requirement for an instructor pilot in the cockpit. Since a

simulator cannot "crash" and hurt the trainee, there is no requirement for an instructor pilot to

serve as a "safety pilot." As the authors know from experience, learning to hover, the most basic

helicopter flight skill, involves much frustration. During the earliest hours of hover training, the

student continues to overcontrol the aircraft, lose control, and relinquish the controls to the

instructor who then brings the helicopter from its dangerous attitude back into control. The

frustration, danger, and inefficiency of early hover training served as a heuristic stimulus that led

to the concept of an automated, adaptive, computer-managed hover trainer.

The Automated Hover Trainer

Background. The Automated Hover Trainer (Dohme, 1995) was developed to evaluate

the idea that students would learn to hover more quickly and with less frustration if they didn't

have to experience the instructor pilot snatching away the controls to save the aircraft. From a

human factors perspective, students should learn more rapidly and with less negative affect if

they could continue to have control of the machine with varying degrees of help provided by an

augmented set of flight controls that would synthesize an easy-to-fly helicopter. It was designed

jointly by ARI personnel and engineers from the University of Alabama Departments of

Aerospace and Electrical Engineering to test that idea. The UH-1 Training Research Simulator

was used as the training vehicle. Modifications were made to add the computer artificial

intelligence to the control loop. In addition, synthesized voice feedback was provided to students

to provide verbal feedback regarding their errors and to provide guidance toward improving their

hovering flight skills.

Rationale. The premise behind the development of the Automated Hover Trainer was to

synthesize a training simulator that would continuously review student pilot performance against

the IERW POI training objectives and adaptively augment control inputs such that the demand

characteristics of the device would accommodate the trainee's ability to successfully hover. The

Optimal Control Model (Kleinman, Baron, & Levinson, 1970) was applied to the trainer to

create an adaptive trainer providing "inner-loop stability augmentation" to the man-in-the-loop

simulation. The mathematical model periodically compares ongoing trainee performance with

performance norms established by an expert pilot flying textbook maneuvers. As the student

pilot's performance approaches the norm, the computer augmentation reduces the level of control

augmentation until the student pilot is able to fly the maneuver with the unaugmented helicopter

aerodynamic model.

27
Automated Hover Trainer Quasi-Transfer Experiments

Approach. The same approach was used that was successful with the TOT experiments,

i.e., a random sample of lERW students served as research subjects and the experiment was

embedded in the IERW course. A preliminary evaluation of the Automated Hover Trainer was

accomplished using what Taylor, Lintern & Koonce (1993) termed a quasi-transfer experiment.

Students were trained to the IERW POI standards in the Automated Hover Trainer. Transfer was

evaluated by an instructor pilot using the device itself as the criterion vehicle. Students trained

to IERW standards on the five basic hovering maneuvers: stationary hover, hover taxi, hovering

turn, land from hover, and takeoff to hover. Criterion performance was defined as two

successive minutes at autohelp level zero (no control augmentation).

The initial evaluation, using the quasi-transfer research design, was undertaken as an

initial feasibility study of the Automated Hover Trainer concept. As such, it was not deemed

necessary to include a control group.

Results. In-simulator checkrides were administered by a Standardization Instructor Pilot

who was not privy to the training data from the research subjects (e.g. how much time or how

many iterations were required by each student pilot to meet the criterion). Students were graded

as though they were being evaluated for authorization to perform their first solo flight, a

checkride that was administered at approximately the 14 hour level in the aircraft. At the time of

the checkride, participants had spent an average of 3.1 hours each over four days in the

Automated Hover Trainer (total training time was limited by the schedule imposed upon the

research subjects by their Army assignments). The data from the quasi-transfer experiment are

presented in Table 8, showing the percent of trainees passing each maneuver during the timelimited training and the percent passing the checkride.

Table 8

Percent of Students Meeting Criterion for Maneuver Training and Passing Subsequent Checkride

Percent Passing Aircrew Training Manual Maneuver Tasks

Hover Hover Hover T/O to Land to

Taxi Turns Hover Hover

Training Program 96 100 96 87 96

Checkride 100 100 100 96 96

Transfer of Training from the Automated Hover Trainer to the Aircraft

Approach. Although the quasi-transfer experiment demonstrated the Automated Hover

Trainer to be an effective training device, it remained to be seen whether training would transfer

to the aircraft. Two TOT experiments were conducted using a random sample of Army student

pilots. The experiments followed the same procedure and are combined for this review. It was

necessary to replicate the experimental design since only ten participants per day could be

trained in the device during the time the students were available.

28
The twenty officer students, drawn at random from their respective flight classes, learned

the five basic hovering maneuvers in the Automated Hover Trainer. When they went to the

flight line, their progress was assessed using a research evaluation form. Each time hovering

flight was trained, the UH-1 instructor pilot recorded the training on the form on his/her

kneeboard. There were four possible outcomes to each maneuver iteration:

"* DEMO: instructor pilot demonstrates the maneuver.

"• ASSIST: student pilot attempts maneuver, instructor pilot takes controls to assist.

"* ATTEMPT: student pilot completes maneuver without help but does not meet IERW

POI standards.

"* STANDARD: student pilot completes maneuver within standards.

Successful completion of training for each maneuver was operationally defined as three

successive maneuver iterations in the STANDARD category. The dependent measure was the

total number of iterations of each maneuver to meet the criterion.

Transfer of training was evaluated by comparing the hover training performance of the

experimental group with classmates who served as a control group. The criterion was the same as

that used in the simulator (three successive STANDARD grades). The hypothesis under

evaluation was that the experimental students would meet the criterion for the target maneuvers

in the aircraft with significantly fewer maneuver iterations than would the control students.

Results. The results of the experiment are presented in Table 9, which compares the

iterations to criterion for the two groups by maneuver. In each case, the experimental students

met the criterion in significantly fewer iterations than did the control group. The experimental

group met the training criterion in the aircraft with an average of 15.2 iterations per maneuver

and a total of 75.9 iterations across the maneuvers. The control group required an average of

18.9 iterations per maneuver with a total of 94.7 iterations across the maneuvers. This difference

reflects a savings of 19.9% of the control group's maneuver iterations on the flight line.

Table 9

Mean Maneuver Iterations to Criterion for Experimental (n = 16) and Control (n = 33) Groups

Groups Aircrew Training Manual Maneuver Tasks

Hover Hover Taxi Hover T/O to Land from

Turns Hover Hover

Experimental 14.3 14.1 17.6 15.3 14.6

Control 18.3 19.0 21.8 18.2 17.3

X 2 (df l; p•< 10.4(.01) 14.9(.001) 9.5(.01) 5.2(.05) 4.7(.05)

29
Conclusions based on Automated Hover Trainer Findings

This research demonstrates that the automated, adaptive simulator-based training concept

has merit. Beyond that conclusion, it shows that the Automated Hover Trainer was an effective

training device capable of producing positive TOT in the IERW curriculum. In these

experiments, the IERW POI was not changed from the usual class-based approach except for the

addition of the automated hover training before flight line training. Since the curriculum was not

developed to optimize the effect of this specialized training, the experimental results are

considered to be a conservative estimate of its potential.

Implications of ARI IERW Experimental Findings

The series of experiments described above were conducted to evaluate the feasibility and

practicality of training IERW students in ab initio helicopter piloting skills using low-cost

simulation and computer-automated training. Since the research was conducted using random

samples of Army IERW trainees as research participants with training embedded into the IERW

POI at Fort Rucker, the results are considered to be directly generalizable to that training

program. Although the course structure has changed somewhat since this research was

completed, the following conclusions are considered to be supported by this research:

1. Low-cost simulation is effective in training neophyte students in the basic flight

control skills underlying helicopter pilotage.

2. Training in low-cost visual simulators can substitute for in-aircraft training with no

significant loss in trainee performance. However, it may be necessary to provide

more maneuver iterations in the simulator than in the aircraft to meet the same

standards. The combined research data from Experiments 1-4 suggest that

approximately 2 3⁄4 iterations in the low-cost simulator are required to replace one

iteration on the flight line.

3. Training in a low-cost simulator can show positive TOT to the aircraft, provided that

the visual out-the-window scene and the aerodynamic flight model offer the trainee

at least moderate fidelity.

4. An automated, adaptive, simulator-based trainer can provide significant benefit to

the training of hovering flight skills at very low cost.

5. Improvements in the quality of the out-the-window visual scene such as more

polygons displayed, textured surfaces, and faster scene update rates resulted in

greater TOT.

Based on these conclusions, we recommend that the Army conduct further experiments to

evaluate the most effective application of low-cost simulation and automated, adaptive,

simulator-based training to the IERW POI. At the present time, ARI is developing a low-cost

training simulator based on the Automated Hover Trainer. The Intelligent Flight Trainer will go

beyond automated, adaptive hover training to include other IERW Primary Phase maneuvers. It

uses a low-cost simulation of the TH-67 and is being developed to provide automated, adaptive

training of traffic pattern flight as well as in the basic hovering maneuvers.

30
General Conclusions and Recommendations for Future Research

Programmatic Issues

The research discussed above promises to improve the efficiency of IERW training by

augmenting training on the flight line with lower cost training using simulation. This is a

mechanism to lower training costs and it should be empirically evaluated and implemented.

Other cost-savings training amplifiers should be considered in addition to simulation. These are

pertinent to broader programmatic issues.

The current IERW POI uses the concept of the flight training class. All students are

assigned to a class with each class following a fixed POI providing multiple modes of instruction

such as the classroom, procedures training devices, simulation, and flight training, all on a fixed

schedule. Experience shows that students do not all learn the material and meet the training

objectives at the same rate. In the current POI, these differential learning rates are handled by

mechanisms such as the setback in which a student is reassigned to another class at an earlier

stage of training so as to repeat a portion of the curriculum. There are advantages of structuring

the IERW course into classes. For example, students are encouraged to develop an esprit de

cor1p by wearing a specific color of cap to identify their flight class. Likewise, students have

classmates at the same stage of training with whom they can work cooperatively to master the

material.

There are also disadvantages to the flight class-based (lockstep) POI. A student who is

learning rapidly and is ahead of the fixed curriculum is nonetheless required to stay with the

curriculum. The authors know of numerous students who have already met the training

objectives of a given training phase and yet continue to fly in order to meet the flight time

requirement to complete that training phase. Students and instructor pilots sometimes refer to

this training as " boring holes in the sky." While additional flight time has positive

consequences such as improving the students' confidence and honing overall flight skills, it is

expensive. In short, the class-based curriculum is optimized to benefit the slowest learners. A

critical test would be a direct comparison of the current time-driven system with the proficiencybased, objective-driven system.

Research Possibilities

Class-based vs. criterion-based training. First, we suggest that research be conducted to

evaluate self-paced, criterion-referenced training (i.e., practice until a performance criterion is

met) in IERW. An experiment could reveal whether students who train to meet POI training

objectives (such as passing an academic test or passing an in-flight checkride on the target

maneuvers) meet the criteria in fewer total training hours and/or at less training expense than

under the fixed-curriculum flight class system. The same training resources could be retained,

e.g., classroom instruction, procedures training device instruction, in-simulator instruction, and

in-flight instruction. However, students could be administered self-paced academic and handson-performance exams (such as aircraft systems mastery). With increased use of visual flight

simulation, the focus of "blade hour" flight training could be on evaluation. The aircraft could be

used to "fine tune" training already begun in the simulator until the student pilot displayed

31
mastery on a specified training objective. This method would obviate the need for a

conventional in-aircraft checkride. For quality control purposes, students could be randomly

selected and administered a formal checkride by an Army Standardization instructor pilot to

validate the training POI. In the next section, we will set forth in detail an example of such an

experiment. This experiment will actually be two studies in one; the first will address questions

regarding the fuinctional requirements for IERW simulators, while the second will manipulate

several combinations of simulator vs. aircraft hours.

Exploring simulation technology and simulator/aircraft mix. Thus far we have reviewed

past ARI research efforts, and discussed the advantages of a proficiency-based, simulation

augmented, IERW training program, along with optimizing the mix of aircraft/simulator training

time. We would like to present an example of an experiment that would be employed to

investigate the optimal simulator/aircraft mix, for the Instrument Phase of IERW.

The rationale for initiating the IERW research program with the Instrument Phase is

based upon two factual situations. First, simulation is currently employed in the Instrument

Phase, though to a limited extent. Secondly, advances in simulation technology have provided

more cost-effective, PC-based alternatives to the current UH- 1-based simulators currently in use.

The TH-67 Skills Trainer was developed from a Bell 206 instrument trainer and has much

commonality with the TH-67s now in service. Research can be initiated with minimal disruption

in the training syllabus, as simulation is currently a part of the POI. Consequently, the first

research effort will examine different approaches to employing simulation for IERW instrument

training.

Current allocation of IERW Instrument Phase hours. The reader should recall that the

duration of the Instrument Phase is 50 hours over a 40-day training period. Basic Instruments

(Stage I) consists of 30 hours training in the simulator, including the .8 hour checkride

administered in the simulator. Advanced Instruments (Stage II) consists of 18.50 hours in the

aircraft, and a 1.5-hour end of phase checkride.

Limitations of current training program. One limitation is in the technology and

configuration of the UH-1 simulators. They represent an aircraft that has been phased out of the

primary IERW Common Core training syllabus. Another limitation concerns the many

differences between the UH-1 and the TH-67. One would expect some degree of negative TOT

between two cockpits with very different layouts. This may be the reason why there is no

simulation-based training at Stage II of the Instrument Phase. Before the most recent revision to

the POI, simulator hours were divided between both stages (12.8 hours, Stage I, and 17.2 hours,

Stage II). Furthermore, the current simulators use complex, older technology. They are more

expensive to operate and maintain than the present generation of PC-based training devices.

Although they have a five degree of freedom motion platform, they do not have out-the-window

visuals. This obviously limits their use to instrument training alone, and precludes training in

tasks involving transition to visual reference landing (e.g., after breaking out under a low cloud

ceiling). Whatever advantages are lent to TOT by motion cueing have not been demonstrated

empirically.

32
Proposed research approaches. The research that is proposed here will have two basic

dimensions. This is dictated by the obvious fact that, before determining the optimal training

mix, one must determine the optimality of the training device itself. Phase I will compare the

Skills Trainer and UH- 1 simulator as alternative training devices, while Phase II will be

concerned with the substitution of Skills Trainer for aircraft hours.

Phase I will seek to answer the question of how similar (different) are these two devices

in terms of training effectiveness. Some configurational differences will have to be confounded.

The Skills Trainer has no motion system but has out-the-window visuals; the UIH-1 simulator has

motion but no visuals. Comparisons can be made within each simulator, however; the Frasca

Trainer can be employed as an instrument trainer with or without visuals, the UHI-1 simulator

with or without motion. This experiment will establish the similarities and differences between

the two training devices, and give the researcher and training developer insight into the

functional requirements for an effective primary flight trainer for IERW. For both phases, a task

analysis will determine what specific flight maneuvers can be realistically and safely trained in

the simulator. Table 10 presents a notional four-group quasi-experimental design for

determining the effects of the alternative configurations on student pilot performance. It would

be a quasi-experiment because, as was previously stated, motion and visual display

configurations are confounded with simulator type. One important question this study would

answer is whether the TH-67 Skills Trainer is a viable low-cost alternative to the UH-1

simulator.

Table 10

Quasi-Experimental Design for Investigating the Functional Requirements of an IERW Training

Simulator.

Experimental Groups Simulator Perform to Criterion Evaluation

1. "Motion On" UH-1 Instrument tasks Instruments Checkride

2. "Motion Off" UH-1 Instrument tasks Instruments Checkride

3. "Visuals On" Skills Trainer Instrument tasks Instruments Checkride

4. "Visuals Off" Skills Trainer Instrument tasks Instruments Checkride

In the above design, four independent groups of students would perform identical Basic

Instruments tasks in either the UII-1 simulator or the TH-67 Skills Trainer. The dependent

measures would entail ratings of Basic Instrument Phase checkride performance, time to

proficiency in the instrument tasks, and performance in the aircraft during the Advanced

Instruments phase, which would include the final checkride. The design of the study and the

measurement of performance would reflect a proficiency-driven rather than a traditional timedriven training program.

Phase II will explore the optimal mix, and will employ the Skills Trainer at ARI as part of

a model training system test bed. The primary manipulation will be in the number of simulator

hours. This will necessitate the use of three simulator groups, with the baseline, or "status quo"

group, adhering to the current hourly requirements as set forth in the POI. In addition to the

status quo group, three other groups will be employed in order to provide four complete data.

33
points for different amounts of simulator hours. The proposed experimental design for Phase II

of the research appears in Table 11, below. This experiment would be a four-group design,

comparing the offloading of aircraft hours to the simulator from the current 30 to the maximum

of 50 hours.

Table 11

Experimental Design for Investigating Optimal Mix of Simulator vs. Aircraft Hours

Experimental Groups Simulator Aircraft Evaluation

Hours Hours

1. Baseline (Status Quo) 30 20 Checkrides; Time to Proficiency

2. Alternative One 40 10 Checkrides; Time to Proficiency

3. Alternative Two 45 5 Checkrides; Time to Proficiency

4. Alternative Three 50 0 Checkrides; Time to Proficiency

Performance measures. Training outcomes would be assessed using the same

performance measures previously discussed (e.g. checkride performance, iterations to criterion,

time to training milestones, flight grades, setbacks). The same dependent measures can be used

for both phases of the proposed research.

Expected results and payoffs. For Phase I, it would be reasonable to expect that students

trained in the Frasca device would show more rapid acquisition (i.e. earlier proficiency) of basic

instrument skills than those trained in the UH-l-based simulator. Performance during the end of

Phase I checkride should be superior for students trained in the Skills Trainer. Students trained

in the Skills Trainer should also reach critical training milestones earlier than those trained in the

UH- 1 simulator. Further, it would seem reasonable to expect that transfer from the Skills Trainer

to the aircraft would be facilitated by similarity of crew stations and flight controls, as well as the

presence of a visual display system. Because of a paucity of data, it is difficult to specify in

advance the transfer properties of the UH-1 simulator with and without the motion system

operative.

Phase II will closely parallel the methodology employed by Povenmire and Roscoe

(1973). The three levels of simulator training will help specify the relationship between hours of

pretraining and performance in the in-simulator and in-aircraft checkrides. This will give

researchers some information as to whether there is a point of diminishing returns similar to that

found by Povenmire and Roscoe, in which students' performance in the aircraft diminishes with

increasing time in the simulator. This could specify a point of optimality, beyond which it is not

advisable to offload more hours to the simulator.

Contact IERW Training Research

After completion of the Instrument Phase IERW research, the authors plan to address the

same simulator/aircraft mix questions in the context of contact primary flight training. Recall

that a quasi-experiment was previously conducted on a small sample of students and controls,

with promising results. The third phase of the research program will be to conduct a similar.

34
experiment, using the Skills Trainer, or alternative TH-67 simulators with better visual display

systems. Phase III will incorporate the following refinements: (1) The amount of pretraining in

the TH-67 Skills Trainer will be varied from an eight hour baseline to 16 to 24 hours; (2) display

field-of-view (FOV) will be varied; (3) training in the simulator will not be conducted on a noninterference basis (simulation training will not be in addition to the total hours prescribed in the

syllabus, but will be integral to it). The main variables investigated, then, will be the number of

hours students train in the simulator, and the complexity of the simulator, in terms of FOV of the

visual display system.

Implications of Proposed Research

Enhancing IERW training effectiveness. The authors have attempted to demonstrate the

current status of the need for simulation-augmented IERW training, and its potential advantages,

especially with regard to savings in training effectiveness and efficiency. We have also

presented evidence to underscore the urgency of the need; it is difficult to find definitive data

that could provide the guidance needed to develop and validate an effective simulation-based

IERW training system. However, existing research appears promising, in that advantages of

simulation for rotary-wing primary flight training were demonstrated. Unfortunately, these

studies were conducted at different times with different simulator configurations, making

generalizability difficult. Comparisons between different configurations were all made after the

fact. Nonetheless, they do demonstrate that further research, with large enough sample size and

adequate pre-experimental planning, could provide Army aviation with an empirical database

regarding the functional requirements of an IERW simulator and how it can be integrated into a

training program to improve training quality and save flight hours and costs.

Technology transfer possibilities. If it can be shown that a low-cost IERW training

device will save training costs and possibly enhance the effectiveness of training, would civilian

helicopter training operators be interested? According to an unpublished market survey

conducted in 1993 by Zacharias and Dohme, the private sector helicopter training community

has a high level of interest in enhancing the efficiency and effectiveness of its own primary

training programs. Zacharias and Dohme performed a target market analysis to determine

whether the civilian training community perceived any market potential for a low-cost helicopter

flight trainer. They visited six California-based operators, ranging in size from an annual

throughput of 6 to 300 students.

They found that attitudes toward the use of simulation were positive; all believed that a

cost-effective trainer would positively impact student throughput. All agreed that a trainer

costing the same as the aircraft, with half the operational costs, would be justified. At that time,

a training device at the price of an R-22 primary training helicopter was not feasible. As a result

of the market survey, they concluded that the civilian helicopter training community would

cooperate in the test and evaluation of such a device. One reason to expect a high level of

interest is the fact that operational and maintenance costs for a light training helicopter are

greater than those for a light training fixed wing aircraft. A modem, PC-based trainer like the

Frasca Trainer should therefore not only increase student throughput, but reduce overhead costs

as well.

35
As part of the technology transfer demonstration, the Skills Trainer could be beta tested

at a civilian rotary-wing training site. This would take place subsequent to the LERW

demonstration research at Fort Rucker. The research proposed here can be followed up by just

such a test. The bottom line for any commercial operator is cost savings, and if simulationaugmented primary helicopter training can save training costs, and possibly improve training

outcomes, for the Army, it can undoubtedly do the same for the commercial operator.

36
REFERENCES

Bale, R.M., Rickus, G.M. & Ambler, R.K. (1973). Prediction of advanced level aviation

performance criteria from early training and selection variables. Journal of Applied

Psychology, 58(3), 347-350.

Benton, C.J., Corriveau, P. & Koonce, J.M. (1993). Concept development and design of a semiautomated flight evaluation system (SAFES). (Tech. Rep. No. ALiHR-TR-1993-1024).

Brooks AFB, TX: Armstrong Laboratory Human Resources Directorate Manpower and

Personnel Research Division.

Bickley, W.R. (1980). Training device effectiveness: Formulation and evaluation of a

methodology. (Res. Rept. No. 1291). Alexandria, VA: U.S. Army Research Institute for

the Behavioral and Social Sciences.

Caro, P.W. (1973). Aircraft simulators and pilot training. Human Factors, 15(6), 502-509.

Caro. P.W. (1976). Some factors influencing transfer of simulator training. (Prof. Pap. 1-76).

Alexandria, VA: Human Resources Research Organization.

Caro, P.W. and Isley, R.N. (1966). Helicopter trainee performance following synthetic flight

training. Journal of the American Helicopter Society, 11(3).1 3 8-44.

Derrick, W., Tomczak, W., Nullmeyer, R. & Burright, B. (1992). Cost and training effectiveness

impacts of contracted C- 130 aircrew training. Proceedings of the 14tb Interservice/Industry

Training Systems and Education Conference (pp. 550-562). Washington, D.C.: American

Defense Preparedness Association

Diehl, A. and Ryan, L. (1977). Current simulator substitution practices in flight training. (TAEG

Rep. No. 43). Orlando, FL: Training Analysis and Evaluation Group.

Dohme, J.A. (1979). Validation of the aeroscout selection process. ARI Working Paper FR/FU

79-4. Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social

Sciences.

Dohme, J.A. (1995). The military quest for flight training effectiveness. In Larsen, W.E.,

Randle, R.J., & Popish, L.N. (Eds.) Vertical Flight Training. NASA Reference Publication

1373 (DOT/FAA/CT-94/83). Moffett Field, CA: National Aeronautics and Space

Administration.

Edwards, B.E. and Hubbard, D.C. (1991). Transfer of training from a radar intercept part-task

trainer to an F-16 flight simulator (Tech. Rep. AL-TR-1991-0038). Williams AFB, AZ:

Aircrew Training Research Division, Armstrong Laboratory.

37
Flexman, R.E., Roscoe, S.N., Williams, A.C., & Williges, B.H. (1972). Studies in pilot training:

The anatomy of transfer. Aviation Research Monographs, 2(1). Champaign, IL: University

of Illinois Aviation Research Laboratory.

Frasca International, Inc. (May, 1995). Technical Specification Frasca TH-67 Helicopter Flight

Simulator. Frasca Doc. No. 114288-1.

Greer, G.D., Smith, W.D. & Hatfield, J.L. (1962). Improving flight proficiency evaluation in

Army helicopter flight training. (Tech. Rep. 77). Alexandria, VA: Human Resources

Research Organization.

Grimsley, D.L. (1969). Acquisition, retention, and retraining: Effects of high and low fidelity in

training devices. (Tech. Rep. No. 69-1). Alexandria, VA: Human Resources Research

Organization.

Hays, R.T., Jacobs, J.W., Prince, C. & Salas, E. (1992) Flight simulator training effectiveness: A

meta-analysis. Military Psychology, 4 (2), 63-74.

Kleinman, D.L. Baron, S. & Levinson, W.H. (1970). An optimal control model of human

responses. Part I: Theory and validation. Automatica, vol. 6.

Koonce, J.M., Moore, S.L. & Benton, C.J. (1995). Initial validation of a basic flight instruction

tutoring system (BFITS). Proceedings of the Eighth International Symposium on Aviation

Psychology (pp. 1037-1040). Columbus, OH: The Ohio State University.

Mahler, W.R. and Bennett, G.K. (1950). Psychological studies of advanced Naval air training:

Evaluation of operational flight trainers. (SDC-991-1-1). Navy Special Devices Center,

Port Washington, NY.

Nullmeyer, R.T. and Rockway, M.R. (1984). Effectiveness of the C-130 Weapon System

Trainer for Tactical Aircrew Training. Proceedings of the 6th Interservice/Industry Training

Systems and Education Conference (pp. 431-440). Washington, D.C.: American Defense

Preparedness Association

Povenmire, H.K. and Roscoe, S.N. (1973). Incremental transfer effectiveness of a ground-based

general aviation trainer. Human Factors, 15(6), 534-542.

Rakip, R., Kelly, J., Appler, S. & Riley, P. (1993). The role of the MH-53J HII E Pave Low

weapon system trainer/mission rehearsal system (WST/MRS) in preparing students for

Operation Desert Storm, and future operations. Proceedings of the 1 5th

Interservice/Industry Training Systems and Education Conference (pp. 432-438).

Washington, D.C.: American Defense Preparedness Association

Roscoe, S.N. (1971). Incremental transfer effectiveness. Human Factors, 13, 561-567.

38
Roscoe, S.N. and Williges, B.H. (1980). Measurement of transfer of training. In Roscoe, S.N.

(Ed.) Aviation Psychology. Ames, IA: Iowa State University Press

Salas, E., Bowers, C.A. & Rhodenizer, L. (1998). It is not how much you have but how you use

it: Toward a rational use of simulation to support aviation training. International Journal

of Aviation Psychology, 8(3), 197-208.

Selix, G.A. (1993). Evolution of a training program: The effects of simulation on the Mf-53J

Pave Low Combat Crew Qualification Course. Proceedings of the 1 5th

Interservice/Industry Training Systems and Education Conference (pp. 422-431).

Washington, D.C.: American Defense Preparedness Association

Stewart, J.E. (1994). Using the backward transfer paradigm to validate the AH-64 Simulator

Training Research Advanced Testbed For Aviation. (Res. Rep. No. 1666). Alexandria, VA:

U.S. Army Research Institute for the Behavioral and Social Sciences.

Taylor, H.L., Lintern, G., & Koonce, J.M. (1993). Quasi-transfer as a predictor of transfer from

simulator to airplane. Journal of General Psychology, 120, 257-276.

U.S. Army Aviation Center (1979). Evaluation of the 175/40 Initial Entry Rotary-Wing Flight

Training Program. (Training Report 79-02.) Fort Rucker, AL.

Valverde, H.H. (1973). A review of flight simulator transfer of training studies.

Human Factors, 15(6 , 510-523.

Westra, D.P. and Lintern, G. (1995). Transfer of training and cost-effectiveness.

In Larsen, W.E., Randle, R.J., & Popish, L.N. (Eds.) Vertical Flight Training. NASA

Reference Publication 1373 (DOT/FAA/CT-94/83). Moffett Field, CA: National

Aeronautics and Space Administration.

Wightman, D.C. and Wright, R.H. (in preparation). Simulator-based training in the AH-64

Aircraft Qualification Course: Proof of concept. ARI Research Note.

Williams. A.C. and Flexman, R.E. (1949). An evaluation of the Link SNJ operational trainer as

an aid in contact flight training (SDC-71-16-3) Navy Special

Devices Center, Port Washington, NY.

Zacharias, G. L. and Dohme, J.A. (1993). Business Plan. Unpublished market survey.

39
APPENDIX A

IERW COMMON CORE OUTLINE

PRIMARY PHASE

Basic Contact: Stage I

Hover Take Off to a Hover Land from a Hover

Hovering Turns Fly Straight/Level Climbs

Descents Turns Accel/Decel

VMC Take Off Traffic Patterns VMC Approach

Hovering Auto Standard Auto SEF/Altitude

SEF/Hover Low Level Auto Shallow App-R/O Landing

Go-Around Emergency Procedures Eval I

Advanced Contact: Stage II

Max Performance T/O VMC App (shallow) Simulated Anti-torque Malf

VMC App (steep) Slope Operations Simulated Hydraulic Malf

Student Solo Standard Auto w/Turn Low Level Auto

Emergency Procedures Eval IH

INSTRUMENT PHASE

Basic Instruments: Stage I

Straight/Level Standard Rate Turns Compass Turns

Timed Turns Climbing Turns Descending Turns

Steep Turns Accel/Decel SEF/Altitude

Climbs/Descents Emergency Procedures Unusual Attitude Recovery

Trim Control Eval I

Advanced Instruments: Stage 11

Instrument Take Off Radio Navigation Instrument Approach:

Radio Communication Lost Communication ADF, PAR, ILS, LOC,

Missed Approach Holding: NDB, VOR, LOC Emergency Procedures

Eval II

A-1
COMBAT SKILLS PHASE

Basic Combat Skills: Stage I

SEF/In Gnd Effect Slope Operations Simulated Anti-Torque Malf

VMC Approach Hovering Auto Standard Auto with Turn

SEF/Altitude SEF/Hover Shallow App-RIO Landing

Low Level Auto Go-Around Simulated Hydraulic Malf

Standard

Auto Traffic Patterns Low Level/Low Airspeed

Auto

Wire Obstacles NOE Decel Refueling Operations

Emergency Procedures Eval I

Advanced Combat Skills: Stage II

Zone Reconnaissance Terrain Flight NAV:Pilotage/Dead

Reckoning

Area Reconnaissance Terrain Flight T/O Negotiate Wire Obstacle

Route Reconnaissance Terrain Approach Call/Adjust Indirect Fire

Security Mission Tactical Report Recon LZ, PZ, Hold Area

Emergency Procedures Hover/OGE Select Combat Position

Evasive Maneuvers Mask/Unmask Target Handover to Atk Hel

Eval II

A-2
APPENDIX B

ABBREVIATIONS AND ACRONYMS

Accel/Decel Acceleration/Deceleration

ADF Automatic Direction Finder

App Approach

Atk Attack

Auto Autorotation

CTER Cumulative Transfer Effectiveness Ratio

Eval Evaluation

Gnd Ground

Hel Helicopter

IERW Initial Entry Rotary Wing

ILS Instrument Landing System

ITER Incremental Transfer Effectiveness Ratio

Lnd Landing

LOC Localizer Radio Course

LZ Landing Zone

Malf Malfunction

NAV Navigation

NDB Nondirectional Radio Beacon

NOE Nap-of-the-Earth

OGE Out of Ground Effect

PAR Precision Approach Radar

POI Program of Instruction

PPDR Pilot Performance Description Record

PZ Pick-up Zone

Recon Reconnaissance

SEF Simulated Engine Failure

TER Transfer Effectiveness Ratio

T/O Take Off

TOT Transfer of Training

VMC Visual Meteorological Conditions

VHF Very High Frequency

VOR VHF Omnidirectional Receiver

B-1