U.S. Army 

Special 

Research 

Report 

Institute 

#56 Enhancing U.S. Army

May, 2003 Aircrew Coordination

0 0) Training

Lawrence C. Katz

U.S. Army Research Institute

Gary N.Grubbh

Dynamics Research Corporation

DISTRIBUTION STATEMENT A:

Approved for Public Release -

.Distribution Unlimited
Enhancing U.S. Army

Aircrew Coordination

Training

ARI Special Report # 56

Lawrence C. Katz

U.S. Army Research Institute

Gary N. Grubb

Dynamics Research Corporation

May 2003

U.S. Army Research Institute for the Behavioral and Social Sciences

5001 Eisenhower Avenue, Alexandria, VA 22333-5600

Internet site: http//www.ari.army.mil
ii
Foreword

This special report describes objectives and outcomes of ongoing team training research and

development (R&D) under the guidance of the U.S. Army Research Institute for the Behavioral

and Social Sciences, Rotary-Wing Aviation Research Unit (ARI-RWARU) at Fort Rucker,

Alabama. The Aircrew Coordination Training Enhancement (ACTE) program is an applied

research project that employs experience, innovation, and technology to address the operational

issue, "Can interactive multimedia courseware using web-based distribution provide the realism

and relevance necessary for effective behavior-based team training and evaluation?"

The goal of the ACT Enhancement effort is to make available a web-delivered, interactive aircrew

coordination training system that provides Army aircrews with the knowledge and skill-sets

needed to increase flight safety and mission effectiveness in daily operations. Focus areas for the

applied research were identified by Army leadership and the Aircrew Coordination Training

Working Group. They emphasized -

"* Current ACT program revitalization and enhancement without repeating

previous research

"* Automation and aircraft configuration issues

"* Course length reductions

"* Simplified assessment and evaluation procedures

"* Adult learning-based presentation, feedback, and discussion

The products from the first phase of research are research reports on the development and

evaluation of computer-based Aircrew Coordination Training. They include as attachments two

interactive multimedia courses of instruction with supporting training, the ACTE Aircrew Course

and the ACTE Instructor Course. Both courses contain a fully integrated Data Management

System that tracks student demographics, provides graphic feedback displays during evaluation

exercises, and facilitates electronic course critiques. User testing and validation results indicate

high levels of acceptance for both the training and performance evaluation components. Initial

testing of the prototype courseware on the Army's distance learning suite supports both the webbased and instructor facilitated delivery strategies. The products have been presented to the U.S.

Army Aviation Center for consideration for Army-wide implementation.

Initial data suggest that web-delivered, interactive multimedia courseware provides effective

realism and relevance for team training and evaluation. Ongoing ARI research activities include

developing aircraft-specific training support packages and a program to meet the particular training

needs of non-rated crewmembers. The ACTE serves as a model for adult learning-based training

supported by distance learning technologies.

ZITA M. SIMUTIS

Director

iii
iv
Table of Contents

Page

The Aircrew Coordination Training (ACT) Challenge

Aircrew Coordination Training Defined ................................................................ I

Aircrew Training Program ................................................................................... 4

Enhancing Behavior-based Training and Evaluation

Aviation Safety Investment Strategy Team (ASIST) ........................................... 6

Aircrew Coordination Training Master Plan ......................................................... 6

Aircrew Coordination Training Enhancement (ACTE) Research ......................... 7

Enhancement Prototype Products

Simplified Performance Evaluation System ....................................................... 9

Web-based Courseware Development .............................................................. 10

A ircrew C ourse ................................................................................................ 10

Instructor C ourse ............................................................................................... II

Distance Leaming Delivery .............................................................................. 12

Performance and Effectiveness Results

Usability A ssessm ent ........................................................................................ 13

F ield S tudy ..................................................................................................... . . 15

Operational Field Testing ................................................................................. 17

Meeting the Challenge

Fielding Effective Training Products .................................................................. 19

Sustaining Performance Improvements .......................................................... 20

R efe re n ce s ................................................................................................................... 2 1

Appendix A Objectives, Basic Qualities, and Risk Management ...................... A-1

Appendix B Performance Evaluation Checklist .................................................. B-1

V
vi
The Aircrew Coordination Training (ACT) Challenge

The crew of the UH-60 Black Hawk helicopter was conducting night training. The four-man crew

was wearing night vision goggles in an extremely dusty environment. CW2 Stanley was a little

nervous, having only 600 hours in this aircraft,

but he was very aware that his Instructor Pilot

(IP), CPT Fulford, had over 3,000 hours offlight

time and was highly respected in the unit for his

abilities. CPT Fulford was on the controls when

they took off into a dust cloud Stanley and both of

the crew chiefs sitting in the back seat sensed that

the aircraft was in a left turn, but none of them

said anything, not wanting to question CPT

Fulford. The aircraft had indeed been in a left

turn, which changed a stiff right crosswind into a

tailwind and prevented it from climbing out of the

dust cloud. It contacted the ground, rolled over,

and was destroyed.

The results of the Accident Review Board were conclusive. Crew

error, specifically on the part of the instructor pilot, was the prime

contributing cause of this accident. CPT Fulford allowed himself to

exceed his capabilities by attempting to perform a maneuver that he

had not performed in the aircraft for an extendedperiod of time. But

"there was a deeper, more disturbing element present here as well.

Three other crewmembers just sat there and let it occur. All sensed

that something was not quite right, but they made no attempt to say

anything during the maneuver that may have prevented the error that

was about to occur.

Aircrew Coordination Training Defined

The US Army defines aircrew coordination as a set of principles, attitudes, procedures, and

techniques that transforms individuals into an effective crew. The stated objective of Aircrew

Coordination Training (ACT) is to provide aircrews the knowledge, skills and attitudes necessary

to increase their mission effectiveness, while decreasing the errors that lead to accidents.

ACT and Crew/Cockpit Resource Management (CRM) programs were instituted in the 1980's,

first in commercial aviation and later in military aviation, to address adverse mishap rate trends

that showed the inability of many aviators to work well together in periods of high stress or

workload (Helmreich, Merritt, & Wilhelm, 1999). Minor aircraft malfunctions were resulting in

fatal accidents with alarming frequency. While aviators generally displayed excellent knowledge

and understanding of aircraft systems, operating procedures, rules and regulations and other

technical information, they often displayed a glaring inability to communicate effectively,

distribute workload, maintain or regain situational awareness and make sound decisions. Military

aviation took note of the success of CRM in the civilian sector and instituted similar training

programs (Orlady & Foushee, 1987).

1
ACT/CRM programs have been structured in various ways and continue to evolve as the

perspective changes as to what constitutes effective team coordination training. Most programs

include the following basic elements:

"* A discussion of the core behaviors or basic skill sets that make up ACT. Each

program structures these core behaviors differently, but all contain common

elements.

"* An examination of the applicability of ACT behaviors in the "real world." This

typically takes the form of one or more case studies of real-world incidents or

accidents and includes an analysis of where or when proper ACT behaviors

could have been employed.

"* Some type of role-playing or practice of ACT behaviors in a simulated mission

setting, e.g., line-oriented flight training (LOFT) or its equivalent.

"* Some form of assessment of the learning or changes in attitudes and behaviors

that have taken place as a result of the training, and the evaluation of the

training by the students.

During fiscal years 1984-1989, crew coordination failures identified by the Army Safety Center

directly contributed to the loss of 147 lives and $292 million in aviation accident costs. In 1990,

Army leadership directed the Army Aviation Center to focus its aviation training and evaluation on

crew performance.

Beginning in 1988, the US Army Research Institute (ARI) began conducting a program of training

research that responded to the Army need for better crew coordination training. This program of

research conducted in close cooperation with the US Army Aviation Center (USAAVNC) in

support of the effort to revise aviation training standards to incorporate increased emphasis on

crew-level performance. To assist in the effort to refocus aircrew training on the crew level of

performance, ARI contracted with Dynamics Research Corporation (DRC) for a two-phase

research effort. First, develop and test measures of crew coordination, then develop and validate an

Aircrew Coordination Training course using field aviators. Evaluation testing of aircrew

coordination measures by unit aviators produced favorable results (Pawlik, Simon, Risser, &

Zeller, 1990). The US Army Aviation Center then formed an Aircrew Coordination Working

Group (ACWG) to review measures, research results, and companion training and evaluation

materials. The ACWG revised the behavior-based training and evaluation designs for development

and validation testing of an aircrew coordination course.

ARI worked closely with Army aviation training, evaluation, and safety personnel to develop,

validate, and field an ACT Exportable Training Package. The test bed demonstrated and validated

the program for training and evaluating crew coordination skills. Test results showed that the

crews performed their missions significantly more effectively and safely after the training than

before the training (Simon & Grubb, 1993).

The Army ACT program suite of methods and measures included behaviors and performance (see

Table 1). Results of the Army ACT validation test showed significant improvement in aircrews'

overall mission effectiveness between pre- and post-training evaluation scenarios. During this test,

all aircrews made errors in completing the complex, tactical simulator missions. Pre- to post2
training results improved markedly for navigation, instrument flight recovery, and missionthreatening error performance measures. Like the mission effectiveness increases, results of the

Army ACT validation test showed significant improvement in aircrew flying safety between preand post-training evaluation scenarios.

Table 1. 1992 ACT Validation Test.

Methods and Measures

ACT behaviors or Basic Qualities evaluated with supporting

behaviorally anchored rating scales

Aircrew Training Manual task performance

Mission performance of two flight simulator scenarios similar in

difficulty in terms of time stress, navigational demands, quantity

and capabilities of simulated threat

Mission Effectiveness Results

Overall mission effectiveness +20%

Navigation accuracy +38%

Mission objectives +44%

Error Management Results

Course deviations -45%

Arrive at correct landing zone +38%

No early descent below minimums +32%

Manage mission threatening error +27%

Flying Safety Results

Unexpected weather recovery +25%

Emergency detection +38%

Number of aircraft crashes -43%

Instructor evaluators rated crew performance of the set of 13 Army ACT behaviors (called Basic

Qualities) during each test mission. There was improvement between the pre-training and posttraining evaluations in every Basic Quality with statistically significant improvements on 12 of 13.

The USAAVNC approved the Aircrew Coordination Exportable Training Package (ETP) (Pawlik,

Simon, Grubb, & Zeller, 1992) late in 1992, and formed a training cadre in August, 1993. This

cadre, made up of attack, cargo, observation, and utility subject matter experts (SMEs), was tasked

to field the training and evaluation system to both the Army Active and Reserve Components by

the end of May 1995 (Pawlik, Simon, Grubb, & Zeller, 1993).

3
Aircrew Training Program

The Army's Aircrew Training Program (ATP) specifies that aircrew coordination be emphasized

during readiness level progressions and will be evaluated during all evaluation flights. The

inclusion of aircrew coordination in Aircrew Training Manual (ATM) task descriptions reflects the

"crew concept" philosophy that generally no task is an individual undertaking. Each task can be

performed more effectively and safely by the coordinated efforts of the entire crew. ATM revisions

will include crew actions in the task descriptions, as appropriate. Crew actions define

responsibilities, whether individual or crew, by describing the parts of a task that an individual or

group of crewmembers will perform. Knowledge of the crew actions for tasks being performed

will help crewmembers perforum their individual actions more effectively and enhance crew

coordination.

Research and studies conducted by USAAVNC, ARI, and the US Army Safety Center (USASC)

show the importance of good aircrew coordination. A 1990 analysis of US Army aviation

accidents revealed that a significant percentage of these accidents resulted from one or more crew

coordination errors committed before or during the flight. Often an accident was the result of a

sequence of undetected crew errors that combined to produce a catastrophic result. Additional

research by ARI showed that even when accidents are avoided, these same errors can result in

degraded performance. A systematic analysis of these error patterns identified specific areas where

crew-level training could reduce the occurrence of such errors and break the error chains leading to

accidents and poor performance (Peusch & Hicks, 2001).

Broadly defined, aircrew coordination is the interaction of crewmembers necessary for the safe,

efficient, and effective performance of tasks. Working with this definition, the USAAVNC and

ARI translated crew coordination concepts into a set of 5 crew coordination objectives and 13

Basic Qualities (see Appendix A). Crew coordination objectives are the on-going crew mission

responsibilities that form the organizing structure for aircrew coordination training and evaluation

behaviors. Basic Qualities are the behaviors and skills that can be trained, observed, and evaluated

relative to crew coordination performance. Both Crew Coordination Objectives and Basic

Qualities have been incorporated into the USAAVNC Aircrew Coordination ETP. Each basic

quality is defined in terms of observable behaviors that represent superior, satisfactory, or

unsatisfactory levels of crew coordination. These Basic Qualities and goals are summarized in

each aircraft ATM. The Aircrew Coordination ETP contains detailed guidance and performance

descriptions for use by commanders to evaluate ACT performance and qualify crewmembers.

Description of the Initial Army ACT Program

The U.S. Army implemented its version of ACT in 1994 (Department of the Army, 1992). As a

result of the initial ACT training program, Army aircrews learned behavioral skills and team

coordination techniques that helped them to remain focused and ready to deal with emergencies

and unforeseen problems so they were able to better concentrate on mission objectives.

4
Implementation

Crew coordination training provides the knowledge, skills, and attitudes to aircrews that increase

their mission effectiveness, while decreasing the cockpit errors that contribute to accidents.

Following the implementation of the initial Army ACT in the mid-nineties, the Class A accident

rate dramatically dropped (see Figure 1). Army ACT was presented as "one-time training" without

annual continuation or sustainment training. Though other variables may have contributed to the

decrease, the Class A aviation accident rate increased when ACT was no longer emphasized.

Figure 1. Impact on accident rate.

Immediate positive impact 50 percent

reduction Army aviation class A accidents

2.5'

2.0 1.52 

1.2 !26 13

C 0.5

F0 9 FY9 .-.4'CA 

FY 93 FY 94 FY 9A FY 9 FY 97 FY 98 FY 9q

When Aircrew Coordination Training no longer

emphasized Class A accident rate increased

Commanders and aircrews alike acknowledged the benefit of the mandatory, one-time training that

was received by all aviators within the Army aviation community. The initial program did not

address sustainment issues and did not package the training in a program that would facilitate such

training. Therefore, sufficient funds were not provided for developing a program to sustain this

important training. Funding issues notwithstanding, significant personnel turbulence associated

with downsizing the force since the 1994 program inception has potentially resulted in an erosion

of the safety gains initially realized as a result of ACT. Finally, the atrophy of skills and the

decline of experience levels that occurred during successive years of limited defense funding

coincided with a sharp increase in accident and incident rates (Allman, 2002).

Lack of effective aircrew coordination continues to be cited as a definite or suspected contributing

factor in aviation flight accidents, and it is a factor limiting attainment of the full mission

effectiveness of Army aviation. For example, the Director of Army Safety reported in the

December 1999 issue of Flightfax, "In fact, FY99 produced Army aviation's worst safety

performance since Desert Shield/Desert Storm." The ACT program has not been updated since its

original introduction. Currently, ACT is conducted in the classroom via eight hours of instruction,

with no mandatory follow on training. Instructors responsible for evaluating and reinforcing this

academic training receive four hours of academic training with no exam to determine competency.

Temporary measures such as awareness videos, assistance visits, safety newsletter articles, and a

web-based training support package have been ineffective substitutes for focused ACT training.

5
Enhancing Behavior-based Training and Evaluation

Aviation Safety Investment Strategy Team (ASIST)

Lack of effective aircrew coordination continues to be cited as a definite or suspected contributing

factor in aviation flight accidents, and it is a factor limiting attainment of the full mission

effectiveness of Army aviation. The Aviation Safety Investment Strategy Team (see Figure 2) was

chartered in 1994 to define measurable accident prevention goals and identify the most important

Army-wide investments needed to achieve them. The ASIST study reported that a crew

coordination sustainment-training program would help attain the Army accident reduction goals at

the least investment cost (Hicks & Peusch, 2000).

Figure 2. Aviation Safety Investment Strategy Team (ASIST).

SMisson~ffect :Flying Safety

> Personnel Turbulence and Operations Tempo

) Increasing Reliance on Active-Reserve Unit Mix

> Aviation Safety Investment Strategy Team (ASlST)

"* In-depth analysis of accident experience

"* All force-modernized aircraft during FY 94-98

"* Eight of top ten hazards (290 total hazards) are crew

coordination related

"* Number four of top ten controls (249 total controls) is

crew coordination related

Develop, monitor, and evaluate a Crew Coordination Sustainment Training

program Integrated into aviation tasks (Control Number 4 in "One to Goal List")

Aircrew Coordination Training Master Plan

The ACTE program is being developed under the guidelines established by the ARI-produced

Army Aircrew Coordination Training Master Plan. The plan provides a proactive, multi-phased

course of continuous improvement to maximize Army aviation modernization investments and

complement leadership training initiatives (see Figure 3). The now completed initial research task

was to establish a logical master plan that provides a business process for achieving the desired

ACTE program end-state. Aircrew training programs currently provided by Army, other

Department of Defense (DoD) agencies, and commercial air carriers were reviewed as benchmarks

for the training and evaluation of aircrew and team coordination in both the institution and unit

environments. Issues relevant to both rated and non-rated aircrew members in rotary wing and

fixed wing environments were identified. Insights and operational trend data provided by the

aviation safety, standards and evaluation, and training communities were examined.

6
Figure 3. Aircrew Coordination Training Master Plan.

t' Begin Divestment of Legacy >c Begin Flight School XXI 0• Fully Implement Flight School XXI 1. Divest LegacylTransitional Systems

Systems (UH-1, AH-) > Begin Aviation Combined Arms B- Begin UH-60M Production > Convert All Corp and Division

Tactical Trainer Deployment 1, 204 Distance Leaming Sites Brigade (AC-RC) to Transitional

>. Begin Distribution of Trans- Operational Force Structure

formation Aircraft to RC > Aviation Combined Arms Tactical

ACTE Master Plan Trainer Operational

ACTE Phase I ACTE Phase CTE Phase III

* Establish Interim Aircrew Coordination Training * Establish Permanent Aircrew Coordination Integrate I to include all aircrew members

Working Group Training Working Group Implement Accident Investigative Tool

* Review Current Aircrew Coordination Training * Develop Separate Aircrew Coordination Training * Train Accident Investigators

Programs Policy * Develop Advanced Aircrew Coordination

* Develop Master Plan * Develop Refresher Training Support Packages Training Scenarios

Include Aircrew Coordination Training in Aviation * Train Unit Leaders and Instructors * Develop Aircrew Coordination Training Website

Leader Conferences I incorporate Aircrew Coordination Training (Resources Repository)

* Recommend as Senior Readiness Oversight Evaluation into Readiness Level and Aviation * Establish O&M Recurring Funding Plan

Council Interest Item Proficiency and Readiness Test * Establish Team Training Research and

* Develop Aircrew Coordination Training Integrate Aircrew Coordination Training into Flight Development Program

Performance Evaluation System School XXI

* Develop Core Training Modules * Develop Aircrew Coordination Training Website

SPre-test, Dem-Val, and Field Test Courses * Develop Accident Investigative Tool

Integrate Aircrew Coordination Training into

Advanced Aircraft Simulators and Distributive

Interactive Simulation Scenarios

* Provide Non-rated Crewmembers Training

* Develop Evaluation-based Feedback System

* Provide Site to Host ACTE Courseware

Policy Training Policy Training

AR 95-1 Advanced

ACT ETP1 FS XXI AR 385 Series SIMNET ATEPTC 1-200 AVCATT POI

Leadership _Leadership

FM 1-0 Advanced Leadership

FM 3-h Ceurses FM 25-i00 OBO, DAC. WOAC. ANCOC

ASIST Ann Mod Plan Webs!t AVN Mod Plan ACTE Dist. Leamning Plan Accident Investigation Tool SME simulation Plan SME

EvaluationlRisk Mgmt Plans Evaluation/Risk Mgmt Plans

Aircrew Coordination Training Enhancement (ACTE) Research

The objective of the research effort to enhance Army ACT is to improve the crew and team

coordination effectiveness of Army aircrews in their day-to-day mission planning and flight

operations. The enhanced ACT program builds on the original exportable training package,

revitalizing it from a one-time training event and enhancing it to a dynamic, relevant program that

is continuously updated and improved (see Figure 4). Establishing and maintaining a unit-level

command climate that promotes the use of team coordination behaviors will realize this objective

and places equal emphasis on technical and team coordination skills in daily flight operations.

Instructor pilots and ACT facilitators in aviation units are key to the institutionalization of a

successful ACT program.

ARI's Rotary-Wing Aviation Research Unit convened an Aircrew Coordination Working Group

(ACWG) with recognized knowledge and experience in aircrew coordination training, standards

and evaluation, safety, and human factors to review measures, methods, and training materials

prior to inclusion in the prototype ACTE courses. The ACWG is composed of designated

representatives from the USAAVNC, the USASC, and the Army National Guard. Other agencies

that have information or resources that contribute to planning or developing the program or have a

role in implementing or evaluating the program are included.

7
Figure 4. Old versus enhanced ACT.

OLD Revitalized

"* Initial, one-time qualification training 9 Sustainment training with annual updates

"* Facilitated discussion of paper-based * Advanced adult learning technologies in

courseware with generic video segments electronic classroom environment

"* Basic qualities of ACT behaviors * ACT skills level of instruction

"* Research-based evaluation methods * Simplified evaluation tools and techniques

and measures & Reduced course length

a Reduce instructor requirements for units to

implement Tailored to aircraft and unit

missions

- Hands-on applications based on current

operational situations

* Enhanced error management tools and

techniques

* Unit-level training packages with webbased feedback and support

* Ongoing program evaluation and

management, i.e., training, safety,

standards and evaluation

The current enhancement program managed by the ARI is a multi-year, multiphase program of

applied research structured in three major phases - upgrade and sustain the existing ACT program,

refresh and maintain the upgraded ACT program, and deploy advanced ACT applications. Phase I

of the research effort to upgrade and sustain the current ACT program is completed. Two ACTE

prototype courseware programs have been delivered to the Army, the Aircrew Course and the

Instructor Course. Each course of instruction is the integration and end product of:

"* Analysis of the current aircrew coordination training program

"* Definition of evaluation tools and techniques for assessing along specific

behavioral proficiency dimensions overall crewmember team performance

"* Development of prototype focused interventions for training and evaluating

team coordination behaviors and for managing risk

"* Validation of prototype team training and evaluation techniques in selected

aviation units

"* Field-testing of prototype training, evaluation, and technology products

8
Enhancement Prototype Products

When the Phase I prototype products are fully developed and deployed, the Aircrew Coordination

Training Enhancement (ACTE) program will provide mission-oriented sustainment training and

provision for web-accessible updates.

Simplified Performance Evaluation System

The products necessary to provide a simplified performance evaluation system consisted of a

practical assessment methodology and a suite of quantitative, field-usable measures to allow

across-platform and across-crew configurations evaluation of ACT behaviors and skills.

Implementing the evaluation system required a tool for recording ACT performance evaluations to

support facilitation of team performance improvement during the after-action review.

The central product is a set of observable measures of individual and collective behavior, the

Behaviorally Anchored Rating System (BARS). The BARS is gleaned from extensive research

across DoD and commercial aviation communities (e.g., Leedom & Simon, 1993; Simon, 1990)

and served as a precursor to courseware content design and development. The BARS evaluation

criteria are based on issues relevant to both rated and non-rated aircrew members in rotary wing

and fixed wing environments and serve as the benchmarks against which crew team behaviors are

evaluated.

The measurement of aircrew coordination behavior is a critical component of the aircrew

coordination program and is central to the training content design and delivery. The vehicle for

documenting these evaluations is the ACT Performance Evaluation Checklist (see Appendix B)

which is based on the 5 Crew Coordination Objectives (CCO) and 13 Basic Qualities (BQ)

accepted by the Army as descriptors of aircrew coordination behavior. ACT behaviors and skills

are organized by CCO and are rated using a seven-point scale with values ranging from 1, Below

Standards to 7, Exceeds Standards (see Figure 5). In an electronic classroom setting, learners

observe a training vignette and are given the opportunity to evaluate aircrew behaviors and skills

using an on-line electronic ACT Performance Evaluation Checklist. Their individual responses are

aggregated as a histogram and displayed to the class to serve as the basis for instructor-facilitated

discussion to achieve learning objectives.

Figure 5. BARS numeric rating scale.

Below Meets Exceeds

Standard Standard Standard

1 2 3 4 5 6 7

Written descriptions are provided for the ACT behaviors and skills and levels of performance for

rating aircrews at the values of 1, 4, and 7. These descriptions serve as behavioral "anchors" and

are designed to assist in determining how well an aircrew performs ACT behaviors and skills in

relation to a well-defined set of performance criteria. The anchors are used as the standard for

evaluating ACT performance. This avoids the trap of norm referencing, i.e., comparing one

aircrew's performance with that of another. An aircrew's performance is always rated solely in

relation to the anchors. This has long-term implications for the objective measurement of aircrew

coordination improvement.

9
Web-based Courseware Development

ACTE courses are designed for access via an Internet browser. Courseware web application

components are compliant with Army Distance Learning software requirements and graphics are

produced with a suite of Macromedia Flash and Dreamweaver UltraDev authoring tools. A

Microsoft Access hosted Data Management System (DMS) that is fully compatible with emerging

learning management system (LMS) standards is used to manage course administration tasks, track

student demographics, and collect student critique data. A key feature of the DMS is its ability to

capture student responses from their use of the ACT Performance Evaluation Checklist and

provide graphic feedback displays of practical exercise results. Student reports can be printed

directly from a web browser. Critique items were developed to collect participant data on the

following information areas: Course structure and delivery, length of lesson, instructor, equipment,

and course effectiveness. A five-point response scale with values ranging from 1 (Strongly

Disagree), 3 (Neither Agree nor Disagree), and 5 (Strongly Agree) were presented for each item in

the electronic course critique.

Aircrew Course

The Aircrew Course consists of five modules of instruction: Introduction, an ACT Principles and

Risk Management Review module, a Case Study, a Problem Solving Exercise, and a Conclusion

(see Figure 6).

Figure 6. Aircrew Course design.

Course Map - ACT Aircrew Course

Module I Module 2 Module 3 Module 4 Module 5

Course Ai rcewbmNnn

Welcome Coordina, untion se Conclusion

Overview

Course Risk

Overview Management Course Critique

CCOs

and BQs

* Contains 5 Modules with Mutliple Lessons

• Modules 1, 2 and 5 Web-Based, Self Paced

* Modules 3 and 4 Instructor Facilitated

* One day in length, Distance Learning Deliverable

Both the Principles Review and Risk Management modules employ extensive use of narrated

audio video files. The narration guides the user through the course material while the audio video

files provide support at the exact time of cognitive need. Comprehension checks provide learners

the opportunity to check their retention and feedback relating to their understanding of important

information. A linked list of related topics is provided on each topic header page for those aircrew

members who desire a more in-depth review.

10
The case study module provides an opportunity for aircrew members to become familiar with the

performance evaluation system and the BARS for their use in promoting and evaluating team

coordination behaviors. Once they have reviewed the CCOs and BQs, students are presented a

current case study. A central feature of the case study is an audio video vignette, based on a

mishap taken from actual USASC files, that focuses on automation or crew configuration related

accident or incident trends. Once students observe the vignette, they are given the opportunity to

evaluate aircrew behaviors using the on-line electronic ACT Performance Evaluation Checklist.

Their individual responses are aggregated as a histogram and displayed to the class to serve as the

basis for instructor-facilitated discussion to achieve learning objectives. Finally, the learners are

presented a problem solving exercise in which they are given the opportunity to apply their ACT

knowledge and evaluation skills to a simulation derived from actual events and experiences in the

field.

Instructor Course

The Instructor Course consists of six modules of instruction: Introduction, ACT Instructional

Setting module, Evaluation Tools and Techniques, Observation and Evaluation Exercise, a module

on Facilitation Skills, and a Conclusion (see Figure 7).

Figure 7. Instructor Course Design.

Course Map - ACT Instructor Course

Module I Module 2 Module 3 Module 4 Module 5 Module 6

SetogTehrqes xres

Note: Self-paced lessons are shown in

green. Instructor facilitated lessons are

shown in blue.

"* Contains 6 Modules with Multiple Lessons

"* Modules 2 through 5 Instructor Facilitated

"* One day in length, Distance Learning Deliverable

Completion of the ACTE Aircrew Course by unit instructor pilots is a prerequisite to attending the

Instructor Course. Instructor pilots must be highly proficient in all ACT behaviors and skills

learned during the Aircrew Course and they must be able to apply and evaluate them in their

organization's mission environment. As the role model of ACT behaviors and skills for other

aircrew members, they must "walk the walk" every flight and every simulation period. The ACT

Instructional Setting module emphasizes the ACT Instructor's role in an operational unit, relates

the ACT Crew Coordination Objectives (CCOs) and Basic Qualities (BQs) to the Army Risk

Management Process, and demonstrates their use as control measures to mitigate risk.

11
Presentation approaches and features similar to those used in the Aircrew Course provide in-depth

instruction and practice exercises on applying the ACT Performance Evaluation System tools and

techniques to achieve reliable ratings of observed aircrew situations. The Instructor Course

provides a module consisting of two separate lessons on detailed evaluation tools and evaluation

techniques to equip unit instructor pilots as experts in identifying, evaluating, and facilitating ACT

performance improvement. A second module devoted to practice exercises provides three mission

related situations for instructors to improve their use of the performance evaluation system and

establish a pattern of rating reliability.

Inherent to the ACT performance evaluation process is the ability to identify and apply the CCOs

and BQs to operational and simulated mission settings. Courseware support materials for the

Instructor Course include an Instructor Guide and ACT event driven scenario outlines for use in

simulator or aircraft evaluations of aircrew ACT performance.

Additionally, to ensure that the ACT lessons learned during unit operational or training missions

do not go unheeded, students will be taught the facilitation skills necessary to ensure that unit

aircrews do, in fact, identify, apply and assimilate such lessons learned into future missions on a

daily basis. This training requires well-developed observational and evaluative skills. The use of

vignettes and the electronic Performance Evaluation Checklist supports practice exercises in the

Facilitation Skills module.

Distance Learning Delivery

Both the Aircrew Course and Instructor course are accessible on the World Wide Web. A number

of distribution tests, or evaluations of the courseware deliverability, were conducted throughout the

testing and pre-fielding periods of Phase I. In each test, the courseware ran without significant

error and the few technical issues presented were easily remedied with minor technical adjustments

to the underlying programming. Importantly, the distribution tests confirmed the utility of the

ACTE courseware for simultaneous consumption by geographically dispersed audiences.

Additionally, distribution of the courseware in both a private local area network (LAN) and CD

ROM format were tested and confirmed (see Figure 8). The outcome of the testing of courseware

components ensures that the Army aviation community can make ACTE available to aviators

worldwide.

Figure 8. Local Area Network and Classroom XXI Delivery

12
Performance and Effectiveness Results

After the board findings were released, the unit sat

down and conducted a sincere self-assessment of

their crew coordination performance. They were all

qualified in aircrew coordination training, either

from the original team training in 1994-95, Initial

Entry Rotary Wing training, or the Exportable

Training Package. What had happened to their

skills? What could they do to regain or enhance

these skills? By chance, ARI was bringing a new

ACT enhancement prototype program to their

installation for usability assessment. Could this

enhanced program help to revive their behaviors

and skills that they once had? They had little to lose

by participating in the program. In just under two months they were due to deploy to the National

Training Center (NTC), and there they would encounter the same type of environmental conditions

that created the situation for the accident. The unit decided to sign on for the evaluation of the

research program, and was able to refresh 24 of their crewmembers in ACT. At the conclusion of

this training, all participants felt that their skills were enhanced, and were ready to apply these

skills on the NTC rotation.

The effectiveness of the ACTE training and evaluation components was measured at increasing

levels of fidelity and scope. The ACWG actively participated in the courseware design and

development process, approved release of prototype products for unit testing, and reviewed test

results and approved subsequent products as they progressed through the prototype courseware

development and testing process. The sequence of prototype product design, development, review,

approval, testing, improvement began with user pretest of core modules, continued through

usability assessment and field study, and concluded with operational field testing of the final

prototype products. The demonstration and validation of the effectiveness of the Aircrew and

Instructor Courses occurred as a two-part process consisting of a usability assessment and a field

study followed by a final operational field test.

Usability Assessment

For the usability assessment, use of the courseware was observed and assessed in selected Army

National Guard (ARNG) aviation units that represented cargo and observation aircraft, missions,

and operating conditions. A type of formative evaluation, the usability assessment employed the

prototype courseware in a field setting using representative groups of the target training audience.

Data were gathered on Aircrew and Instructor course usability, effectiveness, and areas of needed

interface and content improvement.

Using unit classrooms and facilities, contractor instructors presented the Aircrew and Instructor

Courses in two four-hour periods each using the contractor provided LAN of notebook computers.

ARI reviewed and approved the automated data collection items. Units identified and scheduled 20

participants to provide a cross section of highly experienced aircrew members and instructors.

Participant ratings for the Aircrew Course averaged 4.0 or higher on a 5-point scale (see Table 2).

13
Table 2. Usability Assessment Participant Ratings.

Aircrew Instructor

Rating Items Course Course

Courseware structure and navigation logical and 4.0 3.8

understandable

Comprehension checks understandable 4.1

Amount of information, pace and time good 3.0 3.1

(On 5-point scale, 1= "too little", 5="too much"; 3 is optimal)

Vignettes allowed observation & related ACT to risk 4.0 3.8

management

Instructor knowledge, preparation, and clarity 4.6 4.4

Student workstation equipment 4.6

Performance Evaluation System and Aircrew Guide helpful 4.2

Content relevant 4.3

Positive effect on flying safety 4.4 4.1

Effect as refresher to previous Aircrew Coordination Training 4.2

Positive effect on mission effectiveness 4.0

Lessons helped understand how to identify ACT behaviors 4.1

Lessons helped understand how to evaluate ACT 4.0

performance

Lessons helped understand how to facilitate ACT performance 4.1

improvement

Instructor Evaluator Guide helpful for implementing ACT in 4.3

daily ops

The subsequent analysis of usability assessment ratings and comments indicated wide and

enthusiastic acceptance of the courseware as necessary and beneficial to Army aviation flight

safety. Example participant comments include:

* "It was quick and to the point and covered applicable material without dragging

on for a week or two." (Aircrew Course)

* "I enjoyed the interaction of the problem solving exercises. I think this is where

the real learning takes place." (Aircrew Course)

0 "New information and examples should be added at least annually for

continuation training to be effective." (Aircrew Course)

* "The interaction of other Instructor Pilots on an important topic that can model

the attitude of a unit by helping mold the attitudes of individuals in the unit and

make them function as a team." (Instructor Course)

14
Field Study

The field study was designed to compare units immediately before and after receiving the

prototype ACTE training and evaluation instruction. Field study participants were aviation units

that represented utility aircraft, missions, and operating conditions. The study provided evidence of

relative improvement due to training. It also supported descriptive, graphical, and user comments

analysis (see Table 3).

Table 3. Field Study Measures.

Measurement Area Measurement Instruments

ACT Behaviors Performance Evaluation Checklist

Behaviorally Anchored Rating System (BARS)

ATM Task Performance Scenario Worksheets

Mission Effectiveness Scenario Worksheets

Crew Related Errors Scenario Worksheets

Course Critique Data Management System Scalar Critique Items

Data Management System Open-ended Items

Prototype courseware for the Aircrew and Instructor Courses was used to train observerevaluators, establish inter-rater reliability, and standardize the field study scenarios. Once

observer-evaluators were trained, a total of 35 unit leaders, instructor pilots, and aircrews

participated in structured training and evaluation events (see Figure 9).

Figure 9. Field study events.

AKey Personnel Observer Evaluator Aircrew Cre- averag

hA- ntce (See Vsta 4 Team Training Classes Evaluation scena I1I Oct 01 29-31 Oct 01 5-6 Nov 01

Ke esne Alrcrew Post-TrainingAiceAC

Out Brief Evaluation Missions Enhancement Trainn

15 Nov 01 13-14 Nov 01 78Nv0

As in the usability assessment, field study data were analyzed and the findings reported to the

Aircrew Coordination Working Group. Participant ratings for the Aircrew Course averaged 4.0 or

higher on a 5-point scale (see Table 4). Results of the simulator-based evaluation scenarios

indicated measurable improvement between pre and post training mission effectiveness and flying

safety, despite the fact that all participants have been previously ACT qualified (see Table 5).

15
Table 4. Field Study Participant Ratings.

Aircrew Instructor

Rating Items Course Course

Tutorial provided knowledge to navigate courseware 4.3

Courseware structure and navigation logical and 4.6 4.0

understandable

Comprehension checks understandable 4.2

Amount of information, pace and time good 3.0 3.0

(On 5-point scale, 1= "too little", 5="too much"; 3 is optimal)

Vignettes allowed observation & related ACT to risk 4.1 4.0

management

Instructor knowledge, preparation, and clarity 4.8 4.3

Student workstation equipment 4.9

Performance Evaluation System and Aircrew Guide helpful 4.3

Content relevant 4.4

Positive effect on flying safety 4.4 4.0

Effect as refresher to previous Aircrew Coordination Training 4.6

Positive effect on mission effectiveness 4.3

Lessons helped understand how to identify ACT behaviors 4.2

Lessons helped understand how to evaluate ACT 4.2

performance

Lessons helped understand how to facilitate ACT performance 3.8

improvement

Instructor Evaluator Guide helpful for implementing ACT in 4.3

daily ops

Example field study participant comments include:

"* "The actual interaction between instructor and student; being evaluated and

seeing the relevance." (Aircrew Course)

"* "The multi-media usage was a very good combination of resources and kept me

interested throughout the entire lesson. It was very easy to navigate. The

scenarios and role-play were very helpful." (Aircrew Course)

"* "It provides a solid foundation for all ACT Instructors to work from. It should

standardize how ACT is trained and evaluated throughout the Army."

(Instructor Course)

16
Table 5. ACT Versus ACTE Pre-Post Training Effectiveness.

Comparison Points ACT (1992) ACTE (2001)

Validation Test 16 UH-60 aircrews 8 UH-60 aircrews

18 hours classroom 8 hours web-based

academics 2 simulator training CBT Courseware

missions

Measures and Metrics 1993 final report 2002 final report

Descriptive statistics and Descriptive statistics

significance test data

ACT Behaviors +34% (13 Basic Qualities) +14% (5 Crew Objectives)

ATM Tasks +33% +11%

Mission Effectiveness +20% +20%

Crew Related Errors

- Manage Threatening Errors +27% +37%

- Number of aircraft crashes -43% -67%

Operational Field Test

For the final operational field test, use of the courseware was observed and assessed in both Army

National Guard (ARNG) and Army Active Component (AC) aviation units that represented cargo,

attack, and utility aircraft, missions, and operating conditions. The field test employed the

prototype courseware in a field setting using representative groups of the target training audience.

Contractor instructors used unit classrooms and facilities to present the Aircrew and Instructor

Courses to a cross section of 31 highly experienced aircrew members and instructors. Data were

gathered on Aircrew and Instructor Course usability, effectiveness, and areas that needed interface

and content improvement. Participant rating results are shown in Table 6.

17
Table 6. Operational Field Test Participant Ratings Results

Rating Items Aircrew Instructor

Course Course

Tutorial provided knowledge to navigate courseware 4.0

Courseware structure and navigation logical and 4.0 3.9

understandable

Comprehension checks understandable 3.5

Amount of information, pace and time good (3.0 optimal) 3.1 3.3

Mission vignettes allowed observation and related ACT to RM 3.8 4.1

Instructor knowledge, preparation, and clarity 4.4 4.5

Student workstation equipment 4.2

Performance Evaluation System and Aircrew Guide helpful 3.9

Content relevant 3.8

Positive effect on flying safety 4.0 4.0

Effect as refresher to previous Aircrew Coordination Training 4.1

Positive effect on mission effectiveness 3.9

Lessons helped understand how to identify ACT behaviors 3.9

Lessons helped understand how to evaluate ACT 4.2

performance

Lessons helped understand how to facilitate ACT performance 4.0

improvement

Instructor Evaluator Guide helpful for implementing ACT in 4.0

daily ops

Example operational field test participant comments include:

"* "The combination of self-paced computer and classroom instruction, as well as

the facilitated discussion helped to enhance absorption of material." (Aircrew

Course)

" "Working through scenarios and evaluating the crews in those scenarios [was

most helpful]." (Instructor Course)

" "It got you thinking about ACT. Also, the case studies were good but could

have been much better by having more in-depth detailed study." (Instructor

Course)

18
Meeting the Challenge

The NTC rotation was just what the unit

had expected Very dusty conditions flown

at high gross weights and under day and

night vision goggle scenarios. Under these

conditions, the unit aircrews applied the

behaviors and skills they had re-learned

during their participation in the previous

ARIA CTE usability assessment program.

They rigorously applied the crew

coordination objectives (CCOs) and basic

qualities (BQs), and conducted thorough,

well-documented after-action reviews

(AAR). After each mission they ensured

that the lessons learned from AARs were

disseminated to all aircrew members in

the unit. As a result of this meticulous application ofACT, the unit had a productive and incidentfree NTC rotation. The crews were especially proud of being able to apply their enhanced skills to

trap developing errors in the execution phase of each mission, and mitigate the risk of these errors

to manageable levels. However, the challenge of maintaining these skills was still a significant

concern. The unit Commander stated that ACT, as a perishable skill, deteriorates without

command emphasis and sustainment training. The solution he suggested was for the command to

emphasize, and the Army to provide sustainment-training products to ensure that crew

coordination errors are minimized.

Fielding Effective Training Products

Can interactive multimedia courseware using web-based distribution provide the realism and

relevance necessary for effective behavior-based team training and evaluation? The results of this

applied research effort lead to the conclusion that the prototype Aircrew Course and Instructor

Course provide operational situations that effectively convey ACT information to multiple mission

aircraft audiences. User testing and validation results indicate high levels of acceptance for both

the training and performance evaluation components. Initial testing of the prototype courseware on

the Army's distance learning suite supports both the web-based and instructor facilitated delivery

strategies for Army-wide implementation.

The prototype products from the first phase of research include two interactive multimedia courses

of instruction with supporting training materials. The Aircrew Course and Instructor Course

include a fully integrated Data Management System that tracks student demographics, provides

graphic feedback displays during evaluation exercises, and facilitates electronic course critiques.

The prototype courseware designed and developed to support ACTE research used the collective

experience of the Aircrew Coordination Working Group and the contractor, innovative application

of proven processes and procedures, and Army and training industry preferred technology and

products. ACTE training effectiveness results from systematic evaluation and testing by aviation

units in the field serve to validate the application of experience, innovation, and advanced

19
instructional technology in the development of this prototype courseware. Furthermore, prototype

courseware products are-

* Authored using Army and industry accepted authoring software

* Tested and demonstrated on the Army's distance learning system

* Manageable using approved Army course management systems

* Upgradeable to comply with Shareable Content Object Reference Model

(SCORM) guidelines

* Documented in sufficient detail to readily meet certification requirements

Sustaining Performance Improvements

The persistent challenge is how to sustain and advance the cultural and team performance

improvements achieved by initial ACT training and enhancements. Key elements for an effective

long-range strategy include actions to:

"* Fully integrate ACT into the organizational structure, command climate, rules,

and regulations that set the stage for daily flying operations

"* Institute ongoing ACT program evaluation and sustainment activities to keep

training realistic and relevant to operational missions and conditions (Abell,

2000)

"* Correlate accident investigation and accident data analysis to the ACT program

structure (behaviors) so that accident investigation data can be used to target

specific areas of the ACT training program for increased emphasis

"* Include instruction on strategies, tools, and techniques that apply ACT

behaviors and skills to manage risk and avoid, trap, or mitigate aircrew error

ACTE Phase II research is currently underway and will further advance the upgraded program by

designing and delivering a comprehensive train-the-trainer course; developing training support

packages for specific aircraft and missions; integrating ACT into Non-rated Crewmember and

Flight School XXI institutional courses; developing a prototype evaluation-based feedback system;

and including ACT in distance learning developments.

Phase III research will focus on incorporating the products of Phase I and II as part of normal

operations and deploying advanced ACT applications. It will include the development of ACT

event-driven scenarios for multiple aircraft missions and extending ACT scenarios to advanced

simulators and distributed interactive simulation training exercises.

20
References

Abell, M. (2000). Soldiers as distance learners: what army trainers need to know. Retrieved from

the world wide web: http://www.tadlp.monroe.army.mil/abell/20paper.htm

Allman, P. (2002, July). Why so many aviation accidents? Flightfax 30 (7) 13.

Department of the Army (1992). Aircrew coordination exportable training package (Vol. 1-3). Fort

Rucker, AL: U.S. Army Aviation Center.

Helmreich, R., Merritt, A., & Wilhelm, J. (1999). The evolution of crew resource management

training in commercial aviation. The International Journal of Aviation Psychology, 9 (1), 19-32.

Hicks, J., & Peusch, I. (2000). Protecting the force through risk management. Systems Engineering

and Risk Analysis 2000, Vol. 10. edited by W. W. Doerr, November 2000. American Society of

Mechanical Engineers, New York, NY.

Lacoste, G. (1999, December). Improving aviation safety performance. Flightfax 27 (12) 1-3.

Leedom, D. K., & Simon, R. (1993). U.S. Army Crew Coordination Training and Evaluation.

Proceedings of the 7th International Symposium on Aviation Psychology, vol. 1, 527-531.

Orlady, H. W., & Foushee, H. C. (Eds.) (1987). Cockpit resource management training:

Proceedings of NASA/MAC workshop (NASA CP 2455). Moffett Field, CA: NASA Ames

Research Center.

Pawlik, E.A., Simon, R., Risser, D. & Zeller, J. (1990). Technical report: Development of

measures of crew coordination (E-16870U). Andover, MA: Dynamics Research Corporation.

Pawlik, E.A., Simon, Grubb, & Zeller (1992, October). Crew Coordination Exportable Evaluation

Package for Army Aviation. Andover, MA: Dynamics Research Corporation.

Pawlik, E.A., Simon, R., Grubb, G. & Zeller, J. (1993). Technical report: Development of

candidate crew coordination training methods and materials (#-21983U). Andover, MA:

Dynamics Research Corporation.

Peusch, I., & Hicks, J. (2001, September). Army safety investment strategy team (ASIST)

translating aviation accident information to hazards and controls. Proceedings of the 19 th

International System Safety Conference, USA, 2001. Systems Safety Society.

Simon, R. (1990). Development of measures of crew coordination. Wilmington, MA: Dynamics

Research Corporation.

Simon, R., & Grubb, G. (1993). Validation of crew coordination training and evaluation methods

for army aviation (Tech. Rep. No. E-785U). Wilmington, MA: Dynamics Research Corporation.

21
22
Appendix A

A~6 T] E 

(V 0

4- L22-j f

C..2

ca0

Cr 0

CY =- 0 0 ifz Of W m

m E t Cm(~ U M 0 .9 - 2 cm c 0 .2 -r) 0E 0: 0 E m E

E 0)

cu c :U

m

-F - -9 o 0 0 5 r 0 

0., CL - 0 0a .2 o CCF 

CC ~ E Eo

U) 0 

.0 m 0 c- 3. C

CL 1 c 8 CL Q

&~~~~~E 

CL D. ;''''~ 

"-

9OU.)LUJr OJJ~td 101!IU)fOv 'JJ.

o ~~~~UOIj3WJOjUI U0ISS!IAJ quBLJ)xý...............]

Z;C SICACOl PeOpNJOA UlelU!BM puL- L~qeSl CL aL Ul) C

0 iesjeaiieH PUB 5~uiluuned UO!SSiVJC

sdoijsuoijeje~j u1801aj UIeLUIeMA PUB LfslIqEIS3 a

A-1
Appendix B

ACT Performance Evaluation Checklist

For use of this form, see the ACT Aircrew Guide

CCO BQ Crew Coordination Objectives/Basic Qualities Rating

1 Establish and Maintain Team Relationships

1 Establish and Maintain Team Leadership and Crew Climate

2 Mission Planning and Rehearsal

2 Premission Planning and Rehearsal Accomplished

3 Application of Appropriate Decision Making Techniques

3 Establish and Maintain Workload Levels

4 Prioritize Actions and Distribute Workload

5 Management of Unexpected Events

4 Exchange Mission Information

6 Statements and Directives Clear, Timely, Relevant, Complete

and Verified

7 Maintenance of Situational Awareness

8 Decisions and Actions Communicated and Acknowledged

9 Supporting Information and Actions Sought from Crew

5 Cross-Monitor Performance

10 Crewmembers Actions Mutually Cross-Monitored

11 Supporting Information and Actions Offered by Crew

12 Advocacy and Assertion Practiced

13 Crew/Flight After-Action Reviews Accomplished

Remarks: (Use continuation sheet[s] if necessary)

Notes:

Consult the ACT Aircrew Guide evaluation procedures and guidelines. Enter a summary rating

(1 - 7) in the rating block for each ACT Crew Coordination Objective (CCO). Refer to the rating

scale below.

Below Meets Exceeds

Standard Standards Standards

1 2 3 4 5 6 7

B-i
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy) 2. REPORT TYPE 3. DATES COVERED (from... to)

May 2003 Final September 2000 - March 2002

4. TITLE AND SUBTITLE 5a. CONTRACT OR GRANT NUMBER

Enhancing U.S. Army Aircrew Coordination Training N61339-01-C-0120

5b. PROGRAM ELEMENT NUMBER

6. AUTHOR(S) 5c. PROJECT NUMBER

Lawrence C. Katz (U.S. Army Research Institute), and 63007A792

Gary N. Grubb (Dynamics Research Corporation)

5d. TASK NUMBER

252

5e. WORK UNIT NUMBER

Col

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER

Dynamics Research Corporation E-5502U

60 Frontage Road

Andover, MA 01810

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. MONITOR ACRONYM

U. S. Army Research Institute for the Behavioral & Social Sciences ARI

ATTN: Rotary-Wing Aviation Research Unit

5001 Eisenhower Avenue 11. MONITOR REPORT NUMBER

Alexandria, VA 22333 Special Report 56

12. DISTRIBUTION/AVAILABILITY STATEMENT

Approved for public release; distribution unlimited.

13. SUPPLEMENTARY NOTES

Contracting Officer's Representative: Dennis C. Wightman

14. ABSTRACT (Maximum 200 words):

This report summarizes the objectives and outcomes of ongoing team training research and development under the guidance of

the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI). The Aircrew Coordination Training Enhancement

(ACTE) program is an applied research project that applies experience, innovation, and technology to research the operational

issue, "Can interactive multimedia courseware using web-based distribution provide the realism and relevance necessary for

effective behavior-based team training and evaluation?" The report briefly describes prototype products from the first phase of

research as two interactive multimedia courses of instruction with supporting training materials for usability assessment,

evaluation, and validation testing by aviation units in the field. User testing and validation results indicate high levels of

acceptance for both the training and performance evaluation components. Initial testing of the prototype courseware on the

Army's distance learning suite supports both the web-based and instructor facilitated delivery strategies. The final prototype

courseware and support materials are ready for refinements to meet certification requirements and subsequent fielding. Training

effectiveness results suggest research into applying the ACTE courseware design and delivery model to accelerate the fielding of

priority training systems to meet the Army's critical training needs.

15. SUBJECT TERMS

Aircrew coordination, training, web-distributed, distance learning, flight safety, crew coordination objectives, Army aviation,

computer-based training (CBT), interactive media instruction (IMI)

SECURITY CLASSIFICATION OF 19. LIMITATION OF 20. NUMBER 21. RESPONSIBLE PERSON

ABSTRACT OF PAGES (Name and Telephone Number)

16. REPORT 17. ABSTRACT 18. THIS PAGE Unlimited Dennis C. Wightman

Unclassified Unclassified Unclassified (334) 255-2834