Training for Effective Human Supervisory Control of Air 

and Missile Defense Systems 

by John K. Hawley, Anna L. Mares, and Cheryl A. Giammanco 

ARL-TR-3765 March 2006 

Approved for public release; distribution unlimited. 
NOTICES 

Disclaimers 

The findings in this report are not to be construed as an official Department of the Army position 

unless so designated by other authorized documents. 

Citation of manufacturer’s or trade names does not constitute an official endorsement or 

approval of the use thereof. 

Destroy this report when it is no longer needed. Do not return it to the originator. 
Army Research Laboratory 

Adelphi, MD 20783-1197 

ARL-TR-3765 March 2006 

Training for Effective Human Supervisory Control of Air 

and Missile Defense Systems 

John K. Hawley, Anna L. Mares, and Cheryl A. Giammanco 

Human Research and Engineering Directorate, ARL 

Approved for public release; distribution unlimited. 
ii 

REPORT DOCUMENTATION PAGE Form Approved 

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently 

valid OMB control number. 

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YYYY)

March 2006 

2. REPORT TYPE 

Final 

3. DATES COVERED (From - To) 

1 October 2004 to 30 September 2005 

5a. CONTRACT NUMBER 

5b. GRANT NUMBER 

4. TITLE AND SUBTITLE 

Training for Effective Human Supervisory Control of Air and Missile Defense 

Systems 

5c. PROGRAM ELEMENT NUMBER 

5d. PROJECT NUMBER 

5e. TASK NUMBER 

6. AUTHOR(S) 

John K. Hawley, Anna L. Mares, and Cheryl A. Giammanco 

5f. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

U.S. Army Research Laboratory 

ATTN: AMSRD-ARL-HR-ME 

Fort Bliss Field Element 

Fort Bliss, TX 79916

8. PERFORMING ORGANIZATION 

 REPORT NUMBER 

ARL-TR-3765

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S) 

U.S. Army Research Laboratory 

Human Research and Engineering Directorate 

Aberdeen Proving Ground, MD 21005 

11. SPONSOR/MONITOR'S REPORT 

 NUMBER(S) 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution unlimited. 

13. SUPPLEMENTARY NOTES 

14. ABSTRACT 

One of the defining properties of the next generation of air and missile defense (AMD) command and control (C2) systems is 

an increasing reliance on automation. This report is the third in a series of three dealing with human performance and training

issues in the development and effective use of automated AMD C2 systems. The first report (Hawley, Mares, & Giammanco, 

2005) discusses the impact of automation on air defense operators and the consequences of their role change from traditional 

operators to supervisory controllers. The second report (Hawley & Mares, 2006) expands upon that background material and 

addresses the issue of developing effective human supervisory control (HSC) in AMD C2 systems. The focus of this report is 

training for AMD operators and the battle staff. In the words of the Army Board of Inquiry (BOI) investigating the Patriot 

fratricides that occurred during Operation Iraqi Freedom, it is necessary to re-look the level of expertise required to operate

such lethal systems on the modern battlefield. Together, these reports are intended as a primer on automation, supervisory 

control, and effective human performance for commanders, concept developers, system designers, trainers, and other 

personnel involved with decision-making and operations for the next generation of AMD C2 systems. 

15. SUBJECT TERMS

Automation, supervisory control, training, and Patriot air defense missile system 

16. SECURITY CLASSIFICATION OF:

19a. NAME OF RESPONSIBLE PERSON

John K. Hawley 

a. REPORT 

Unclassified 

b. ABSTRACT 

Unclassified 

c. THIS PAGE 

Unclassified 

17. LIMITATION 

OF ABSTRACT 

SAR 

18. NUMBER 

OF PAGES 

42 19b. TELEPHONE NUMBER (Include area code) 

(915) 568-2896 

Standard Form 298 (Rev. 8/98) 

Prescribed by ANSI Std. Z39.18 
iii 

Contents 

List of Figures iv

Executive Summary v

1. Background 1

1.1 Overview .........................................................................................................................1 

1.2 Concepts and Terms ........................................................................................................1

1.3 The Patriot Vigilance Project ..........................................................................................2 

1.4 The Rest of the Report.....................................................................................................4 

2. Core Training Issues: Revisiting the Basics 5

2.1 Training Preliminaries.....................................................................................................5 

2.2 The Development of Expertise........................................................................................7 

2.3 Training Demons: The Enemies of Good Training........................................................9 

3. AMD Training Dilemmas: Issues for Research and Experimentation 11

3.1 Overview .......................................................................................................................11 

3.2 Initial Skill Development ..............................................................................................13 

3.3 Time to Train and Job Progression................................................................................14 

3.4 Training for Unreliable Automation..............................................................................17 

3.5 Team Training ...............................................................................................................18 

3.6 Training for Adaptability...............................................................................................20 

3.7 Mindfulness...................................................................................................................21 

4. A Path Forward for AMD 23

4.1 Overview .......................................................................................................................23 

4.2 Training for Routine Expertise......................................................................................23 

4.3 Training for Adaptive Expertise....................................................................................25 

4.4 The Training Challenge.................................................................................................27 

5. References 28

Distribution List 31
iv 

List of Figures 

Figure 1. Scenario-based training cycle......................................................................................... 7 

Figure 2. The development of organizational competence.......................................................... 20 
v 

Executive Summary 

One of the defining properties of current and future air and missile defense (AMD) command 

and control (C2) systems is an increasing reliance on automation. Technological opportunities 

and an increasingly complex operating environment have created a situation where AMD 

operators must be provided with automated decision support to meet mission objectives. There 

is a tendency among system developers with little background in human performance issues to 

assume that automation is innately beneficial. Research in a number of areas suggests, however, 

that such is not always the case. To begin, automation elevates operators into system monitors 

rather than active controllers. Operators are thus removed from moment-to-moment, active 

control and become monitors and managers of subordinate automated processes. It is a wellestablished fact that humans make very poor system monitors. Beyond classical vigilance, 

research and operational experience indicate that automation does not replace human operator 

tasks. Rather, it changes the nature of the work that operators do, and it does this in ways that 

are often unanticipated by system developers and users. Moreover, the preponderance of theory 

and empirical evidence suggests that the job of supervisory controller is quite different from that 

of a traditional operator. In a similar vein, it has been shown that automation does not reduce 

training requirements; it changes training requirements and often makes job preparation more 

demanding. 

Other human performance problems associated with automation generally fall into one of two 

categories: (1) loss of situational awareness (SA) and (2) skill impairment. SA is important 

because it has been shown to be a key determinant of decision quality in battle command. 

Automation in and of itself does not prevent operators from establishing or maintaining SA or 

contribute to skill impairment. However, improper implementation coupled with inadequate or 

inappropriate training can make it more difficult for operators to establish and maintain SA and 

contribute to skill development and retention problems. 

Much of contemporary automation applied to real-time C2 brings into play what has been called 

the Catch-22 of human supervisory control: Automation has been introduced because it is 

thought to be able to do the job better than human controllers, but humans have been left in the 

control loop to “monitor” that the automated system is performing correctly and to override the 

automation when it is “wrong.” The unstated assumption is that human operators can properly 

decide when the automation’s decisions should be overridden. Humans are expected to 

compensate for machine unreliability, but they suffer from a variety of cognitive limitations and 

vulnerabilities that make it nearly impossible to meet this expectation. A number of automation 

researchers have thus concluded that while the risks associated with automation unreliability can 

never be eliminated entirely, they can be managed more effectively through a number of positive 

actions directed at supporting and enhancing effective human supervisory control (HSC). These 
vi 

actions generally fall into one of two categories: (1) design to support effective HSC, and (2) 

training to support effective HSC. This report addresses training to support effective HSC. 

Section 1 begins with a brief review of results from the Patriot Vigilance project. This effort was 

concerned with the human performance contributors to fratricides involving the Patriot air 

defense missile system during Operation Iraqi Freedom. Recommendations from the Patriot 

Vigilance project indicated that the AMD community must address two primary problems 

associated with automation as applied in current Patriot operations: (1) developing effective 

HSC, and (2) the level of operator expertise required to employ a highly automated system such 

as Patriot on the modern battlefield. These issues are also relevant to AMD systems under 

development and to other Army systems being fielded to support network-centric warfare 

concepts. 

Section 2 opens with a review of basic steps in the development of a good instructional program. 

In present context, a good instructional program is one that supports relevant practice with 

feedback leading to desired warfighting competencies. This section also includes discussion of 

relevant background research on the development of human expertise. The section concludes 

with a discussion of Training Demons, the enemies of good training. Training Demons are 

training outcomes and practices that occur when the rules of proper instructional design are not 

followed. These enemies of good training include: (1) inadequate job and task analysis, (2) 

misplaced training planning emphases, (3) lock-step versus criterion-referenced instruction, (4) 

content-based versus skill- and job-oriented training, (5) a misplaced emphasis on free-play 

versus deliberate practice, (6) inadequate or poorly-defined performance standards, (7) lack of 

emphasis on performance feedback, and (8) various forms of training folk wisdom that 

negatively impact training delivery. 

Section 3 addresses six training dilemmas or issues that remain unresolved and stand in the way 

of moving AMD training away from its current focus on crew-drills in the direction of regarding 

operators as managers of lethal and complex systems. These issues include: (1) initial skill 

development, (2) time to train and job progression patterns, (3) training for unreliable 

automation, (4) team training, (5) training for adaptive expertise, and (6) mindfulness—

developing crews and C2 teams that are capable of sustained high reliability in a complex and 

uncertain operational setting. 

Finally, section 4 outlines a path forward for AMD. This section is developed in two parts: (1) 

training for routine expertise and (2) training for adaptive expertise—coping with automation 

unreliability and the inevitable fog of war. The subsection on training for routine expertise 

addresses six focus areas for improvement in AMD training for routine operations—when things 

work as they should. These focus areas are: (1) job and task analysis, (2) scenario content and 

training delivery, (3) performance standards, (4) time to reach desired competency levels, (5) 

qualified instructors, and (6) performance feedback—the after action review (AAR) process. 
vii 

Training for adaptive expertise is intended to foster the ability to think “outside the box” defined 

by routine crew drills. Adaptive expertise is built on a solid foundation of the basics of AMD 

operations—training for routine expertise. Section 4 addresses the basic requirements for the 

development of adaptive individuals, crews, and teams. The discussion ends negatively by 

noting that three major roadblocks stand in the way of developing adaptive expertise: (1) 

training time and job progression practices, (2) training quality (aspects of the AMD branch’s 

traditional training practices), and (3) trainee motivation to develop as AMD professionals. 

Unless these roadblocks are confronted and remedied, adaptive crews and C2 teams are likely to 

be the exception rather than the rule. As the Defense Science Board has cautioned, the risk in 

leaving things mostly as they are is that training failures can negate promising technology and 

hardware. 
viii 

INTENTIONALLY LEFT BLANK
1 

1. Background 

1.1 Overview 

One of the defining properties of the next generation of air and missile defense (AMD) command 

and control (C2) systems is an increasing reliance on automation. This report is the third in a 

series of three dealing with human performance and training issues in the development and 

effective use of automated AMD C2 systems. The first report (Hawley, Mares, & Giammanco, 

2005) discusses the impact of automation on air defense operators and the consequences of their 

role change from traditional operators to supervisory controllers. The second report (Hawley & 

Mares, 2006) expands upon that background material and addresses the issue of developing 

effective human supervisory control (HSC) in AMD C2 systems. The focus of this report is 

training for AMD operators and the battle staff. In the words of the Army Board of Inquiry 

(BOI) investigating the Patriot fratricides that occurred during Operation Iraqi Freedom (OIF), it 

is necessary to re-look the level of expertise required to operate such lethal systems on the 

modern battlefield. Together, these reports are intended as a primer on automation, supervisory 

control, and effective human performance for commanders, concept developers, system 

designers, trainers, and other personnel involved with decision-making and operations for the 

next generation of AMD C2 systems. 

1.2 Concepts and Terms 

Prior to introducing the major topics in the report, it is necessary to define and clarify several 

terms that are used in the text. These definitions and clarifications are given in the paragraphs to 

follow. 

Sheridan (1992, p. 3) defines automation as “the automatically controlled operation of an 

apparatus, a process, or a system, by mechanical or electrical devices that take the place of 

human organs of observation, decision, or effort.” In contemporary AMD C2, the combination 

of operational complexity and technical advances have led to a situation in which functions—

perception, decision-making, response selection and implementation—assigned to the human 

subsystem in previous generations of AMD systems are now assigned to the machine subsystem. 

Supervisory control is defined as a situation in which “one or more operators are continually 

programming and receiving information from a computer that interconnects through artificial 

effectors and sensors to the controlled process or task environment” (Sheridan, 2002, p. 115). 

Under a supervisory control regimen, operators do not interact with the controlled process 

directly, as they previously did in manual or less automated systems. Rather, the operators 

receive information from and provide input to a computer, which, in turn, directs the controlled 

process. The operator’s role is thus changed from direct, on-line process control to supervisor of 

a mostly computer-directed process. Their job is to supervise the computer controller. The 
2 

consequences of this role transformation—though often subtle—must be reflected in system 

design, performance support feature (i.e., job aiding), and operator-controller1 training and 

professional development. 

Effective HSC means that AMD operator-controllers are able to carry out their explicit role, 

which is to supervise or direct subordinate automated control systems. Effective HSC implies 

meaningful human oversight, where meaningful human oversight means that operator-controller 

supervision of subordinate automated control systems is actual and not theoretical or abstract. A 

third related term, positive human control, is a specific aspect of effective HSC and meaningful 

human oversight. Positive human control means that firing decisions—to shoot or not to shoot—

are based on conscious problem solving and discernment and not merely the result of automation 

bias, or uncritical acquiescence following the system’s recommendation. 

Knowledge refers to background information required in the performance of a job or task. 

Knowledge is sometimes described as “knowing about.” Skill refers to a task or group of tasks 

performed to a specific level of proficiency, which often use motor functions and typically 

require the manipulation of instruments or equipment. Some skills are knowledge- and attitudebased and do not require equipment manipulation. Skills are sometimes characterized as 

“knowing how to do.” A competency is a set of skills performed to a specific standard under 

specific conditions. Competencies are sometimes characterized in terms of the level of knowing 

how to do. Competencies differ from task standards in that they are whole-job-referenced rather 

than individual task referenced. 

1.3 The Patriot Vigilance Project 

Personnel from the Army Research Laboratory’s Human Research and Engineering Directorate 

(HRED) began looking into Patriot and AMD performance and training issues at the invitation of 

the then Ft. Bliss Commander, MG Michael A. Vane. MG Vane was interested in operator 

vigilance and situational awareness (SA) as they relate to the performance of automated AMD 

battle command systems. The generally accepted definition of SA is from Endsley (1996) who 

defines it as the perception of elements in the environment, the comprehension of their meaning, 

and the projection of their status in the near future. MG Vane was particularly concerned by 

what he termed a “lack of vigilance” on the part of Patriot operators along with an apparent “lack 

of cognizance” of what was being presented to them on situation displays with an ensuing 

“absolute trust in automation.” His request for human engineering support was prompted by an 

unacceptable number of fratricide incidents by Patriot units during OIF. 

 1To differentiate traditional manual operators from operators performing in a HSC setting, these latter personnel are referred 

to as “operator-controllers.” 
3 

The project staff spent most of the summer and fall of 2004 performing a root cause analysis of 

the OIF fratricide incidents—reading documents, interviewing knowledgeable personnel in the 

Ft. Bliss area, and observing Patriot training and operations. An initial report was delivered to 

MG Vane in October 2004. A discussion of the results of what was termed the Patriot Vigilance 

project is presented in Hawley and Mares (2006) and is not repeated here, other than in a 

summary manner. HRED’s report to MG Vane in October 2004 recommended two primary 

actionable items to redress the performance problems identified during the Patriot Vigilance 

effort: 

1. Re-define the operators’ roles to provide meaningful human oversight of system 

operations, and 

2. Develop more effective Engagement Operations (EO) personnel, particularly the air battle 

operations (ABO) slice of EO—re-look the level of expertise required to operate such a 

lethal system on the modern battlefield. 

The Defense Science Board (DSB) (DSB, 2004) reinforced HRED’s recommendations with the 

following comments. Although the full DSB report on Patriot system performance is classified, 

these extracts are not. 

The Patriot system should migrate to more of a “man-in-the-loop” philosophy versus a 

fully automated philosophy—providing operator awareness and control of engagement 

processes. 

and 

Patriot training and simulations should be upgraded to support this man-in-the-loop 

protocol including the ability to train on confusing and complex scenarios that contain 

unbriefed surprises. 

The key notion in the first DSB recommendation is captured in the phrase, “providing operator 

awareness and control of engagement processes.” Simply put, Soldiers and not the automated 

system must be the ultimate decision makers in AMD engagements. Decisions to shoot or not to 

shoot must be made by crews having adequate SA and the expertise to understand the 

significance of the information available to them. The DSB’s first recommendation is 

synonymous with HRED’s first actionable item concerning establishing meaningful human 

oversight of Patriot and other AMD systems. 

The second DSB recommendation having major significance for human performance in 

contemporary AMD operations concerns operator-controller training and professional 

development. Here, the DSB supported HRED’s conclusion that it is necessary to re-look the 

level of expertise necessary to operate such a lethal system on the modern battlefield. To 

highlight the importance of the training issue, the fratricide BOI stated bluntly “The system 

[Patriot] is too lethal to be placed in the hands of crews trained to such a limited standard.” 
4 

The BOI examining the OIF fratricides specifically criticized Patriot training for emphasizing 

“rote drills” versus the “exercise of high-level judgment.” The essence of this criticism is that 

the user community approaches training for a complex, knowledge-based function like ABO in 

much the same manner as linear, skill- and rule-based actions like March Order and 

Emplacement or System Set-up. The emphasis is on mastering routines rather than adaptive 

problem solving. However, the range of actions required in routine drills is narrower and more 

predictable than those encountered in combat operations. 

The U.S. Navy faced a similar reconsideration of training practices in the aftermath of the shootdown of the Iranian airbus by the USS Vincennes in 1988. After more than 10 years of research, 

the Navy reached several conclusions that are also relevant to the contemporary AMD setting. 

First, the Navy’s research concluded that Aegis operator-controller training must emphasize the 

development of adaptive decision-making skills. Adaptive decision-making skills, or the ability 

to think outside the box defined by routine crew drills, are a key aspect of effective operatorcontroller performance in ambiguous situations. The second major conclusion was that 

shipboard (i.e., unit) training must address team in addition to individual performance. 

Competent crews are the basis of effective unit performance, and crews are more than the sum of 

their individual members. Both of these issues are addressed explicitly in the sections to follow. 

The DSB’s recommendation to include unbriefed surprises in training does not mean that it is 

sufficient merely to insert anomalous events like those encountered in OIF into training 

scenarios. In advanced AMD training, the scenario is the curriculum. To properly prepare 

operator-controllers for combat, scenario designers must bear in mind that the surprises of OIF 

are representative of a class of potential anomalies. Selected anomalies occurred then; others—

some similar, some different—will occur on future battlefields. It is thus necessary that 

operator-controllers be imbued with a sense of mindfulness that automated battle command 

systems are fallible. The system’s recommendations will be correct most but not all of the time. 

Training must foster the development of the expertise essential to recognize potential anomalies 

and the skills necessary to determine an appropriate course of action. Operator-controllers must 

walk a fine line between blind faith and wholesale mistrust. AMD decision makers must not 

underestimate the difficulties associated with adequately meeting this training challenge. 

1.4 The Rest of the Report 

The idea for this report grew out of a conversation with the Army Training and Doctrine 

Command (TRADOC) System Manager-Lower Tier (TSM-LT), during which he suggested 

follow-on reports addressing two topics beyond those addressed initially in Hawley, Mares, and 

Giammanco (2005): 

• Design for effective HSC of AMD systems 

• Training for effective HSC in AMD operations 
5 

The present report addresses the second follow-on topic: Training for effective HSC. In the next 

section, we begin this treatment with a discussion of core training issues in AMD. A section 

addressing various unresolved training issues for research and experimentation follows this 

opening discussion. Finally, we discuss a path forward for AMD. The path forward addresses 

elements of training to develop both routine (that required for normal operations) and adaptive 

expertise (that required for situations that are not typically encountered during routine 

operations). In keeping with the focus on effective HSC, the discussion to follow emphasizes 

training for AMD ABO—activities performed by Patriot crewmembers in the battery-level 

Engagement Control Station (ECS) and at the battalion-level Information and Coordination 

Central (ICC). 

2. Core Training Issues: Revisiting the Basics 

2.1 Training Preliminaries 

Training is a process by which job-relevant knowledge, skills, and competencies are acquired by 

individuals, crews, and multi-echelon units. In a military setting, its purpose is to deliver warfare 

competence where and when it is needed (DSB, 2003). The DSB report further notes that 

training consists of relevant practice with feedback. In essence, therefore, training is a process 

by which learning at various levels (individual, crew, or unit) is facilitated. 

Training is not simulations, networks, virtual environments, or games even if they are labeled as 

training devices. All of these mechanisms can be used to craft an environment in which learning 

might take place, but they are not training per se. This is an important distinction: Various 

methods and technologies such as those mentioned above can be used to support training if they 

provide an environment in which job-relevant practice with appropriate feedback can occur. 

Training developers must, however, first know what must be practiced and the appropriate form 

of essential feedback. 

A good instructional program is defined as one that supports relevant practice with appropriate 

feedback leading to desired warfare competencies. Developing a good instructional program is a 

relatively simple process. However, the devil can be in the details of how the various steps in the 

process are carried out. The steps in proper instructional design are listed as follows (Whitmore, 

2002): 

1. Specify the job performances we want trainees to learn to do. 

2. Break those job performances down successively into their component activities (including 

mental activities) until we reach the level of skill already possessed by the trainees. 
6 

3. Develop situations that lead trainees through those things we want them to learn to do. 

4. Lead trainees through practicing the new activities, beginning with the lowest level and 

successively assembling higher-level activities from lower-level components. 

5. Provide trainees with sufficient practice—with feedback—at each level to reach desired 

competency levels. 

Steps 1 and 2 comprise what is often called a Job and Task Analysis (JTA), or Front-End 

Analysis (FEA). Step 3 is an aspect of course design, specifically scenario design. Steps 4 and 5 

describe proper instructional delivery. Taken together and without all of the detail, this process 

is conceptually similar to what is called for in the Instructional Systems Development (ISD) 

process or its Army variant, the Systems Approach to Training (SAT). 

Before leaving the present topic, we want to comment on the importance of the JTA. The 

information obtained from a JTA must be based on a system’s contemporary operating 

environment and not simply reflect a subjective update of job and task analysis material from a 

previous era. Training developers are often tempted to pursue this latter course of action because 

comprehensive job and task analysis can be an expensive and time-consuming undertaking. A 

thorough job and task analysis must be a high priority for current systems being used in mission 

settings different from that originally intended (e.g., Patriot against a tactical ballistic missile 

[TBM] threat) and for all follow-on systems. Competent JTA involves much more than simply 

updating an existing task list along with supporting documentation. In their 2003 report, the 

DSB remarked that a recurring problem with Army training is that a JTA is rarely performed 

properly, and this “original sin” returns to haunt Army training over the life cycle of most 

systems. 

It has been noted that for a dynamic, performance-based setting like AMD ABO, the scenario is 

the curriculum (Cannon-Bowers, Burns, Salas, & Pruitt, 1998). The relevant practice and 

feedback essential to developing warfare competence takes place within a simulation-based 

learning environment. In this setting, training scenarios provide the structure within which the 

stimulus and response aspects of learning are presented and evaluated. The process by which 

effective scenario-based training is developed and implemented is depicted in figure 1. 
7 

Source: Adapted from Cannon-Bowers, Burns, Salas, & Pruitt (1998) 

Figure 1. Scenario-based training cycle. 

From figure 1, results from the JTA are used to prepare a skill inventory, or list of job skills to be 

trained. These job-relevant skills are used to develop a list of learning objectives and associated 

competencies. Learning objectives drive the development of scenario events, or scripts. If you 

want a trainee to learn how to do something, then the training scenario must provide the stimuli 

necessary to elicit that behavior. Next, the training developer must define performance measures 

and standards. A performance measure is a behavioral description that permits an evaluator to 

unambiguously decide whether a performance took place. The performance standard provides a 

way of judging whether that performance was acceptable. 

In performance-oriented training, feedback is just as important as performance evaluation. For 

learning to occur, trainees must be provided with feedback regarding their performance along 

with corrective guidance. Accordingly, the next step in the Scenario-Based Training Cycle is 

performance diagnosis: If the performance was not acceptable, why not? Performance 

diagnostic information provides the basis for the feedback and debrief stage, or After Action 

Review (AAR). Performance diagnosis must be objective and specific to the performances 

involved. Moreover, for best effect, feedback should be immediate with an opportunity to 

practice less-than-successful performance elements. 

2.2 The Development of Expertise 

Throughout this series of reports we have referred repeatedly to operator-controller expertise. In 

present usage, the term expertise refers to a capability for consistently superior performance on a 

specified set of representative tasks for a domain (Ericsson & Lehmann, 1996). Expertise in 
8 

AMD ABO is derived from all aspects of operator-controller job preparation: (1) traditional 

training (institutional and unit), (2) professional development (self-directed study and 

professional military education), and (3) relevant on-the-job experience. 

We have also argued that SA is the key factor determining decision quality in battle command 

(Hawley, Mares, & Giammanco, 2005). SA is built upon in-depth technical and tactical 

expertise. The primary implication of this conclusion is that marginally-skilled or apprentice 

operator-controllers cannot develop the SA necessary for effective supervisory control, 

regardless of the sophistication of the battle command hardware suite provided to them. 

Technology can support and amplify human expertise, but cannot substitute for it. 

Given the centrality of operator-controller expertise to effective AMD C2, an obvious follow-on 

question is, “How is such expertise developed?” Norman (1993) notes that there are at least 

three phases of learning leading to expertise as defined above. These are (1) accretion, 

(2) tuning, and (3) restructuring. Accretion is the accumulation of facts (knowledge in our 

terminology). When the learner has a proper conceptual framework or mental model, accretion 

is easy, painless, and efficient. Without a good conceptual framework, accretion is slow, tedious, 

and error-prone. 

Tuning refers to the process of translating knowledge into skill. This requires a hands-on 

learning environment and hours and hours of practice under the supervision of a coach or 

mentor. How many hours are necessary? Norman (1993) asserts that for any complex activity, a 

minimum of 5,000 hours of practice (two years of full-time effort) are required to turn a novice 

into an entry-level expert. Other researchers such as Ericsson and Charness (1994) put the time 

estimate for transitioning from novice to expert at 10,000 hours of practice. It all depends on 

one’s definition of an expert. 

The final stage of learning is restructuring, or forming and reforming the proper conceptual 

structure for performing as an expert. Norman (1993) remarks that accretion and tuning are 

primarily experiential—they take place actively in an experience-based learning environment. 

Restructuring is reflective. It involves exploring the domain in depth, forming comparisons, and 

integrating across related domains. Accretion and tuning can be done within a traditional 

training setting. Restructuring is a professional development activity; it is best done as selfdirected study or as part of formal professional military education. 

Norman also comments that one of the problems associated with the development of expertise is 

making trainees want to do the hard work necessary for restructuring. Ericsson and Charness 

(1994) support Norman’s view and assert that a trainee’s motivation to attend to the task and 

exert effort to improve his or her performance is critical to development as an expert performer. 

Training managers delude themselves if they believe that skilled performance is easy and can 

come about without extended effort on the part of trainees. 
9 

The bottom line for the previous discussion is clear: Practice is the major independent variable 

in skill acquisition and the development of expertise—but not just any kind of practice. Ericsson 

and Charness (1994) assert that skilled performance is not an automatic consequence of simply 

having more experience with an activity. Skilled performance is developed through what is 

referred to as “deliberate practice.” Deliberate practice is a long period of active learning during 

which job performers refine and improve their skills under the supervision of an instructor or 

coach. It requires a commitment to hours and hours of relevant practice with expert feedback. 

Several times throughout the previous discussion, we have alluded to the importance of a proper 

conceptual framework or mental model in learning and job performance. A mental model is a 

job performer’s internal understanding of how a performance situation works or fits together. It 

permits job performers to make sense of a performance situation and his or her role in successful 

performance. Norman (2002) remarks that a good mental model permits equipment users to 

predict the effects of their actions. Without a good model, users perform as they are told without 

really knowing why. As long as things work, they can manage. However, when things go 

wrong or when the unexpected happens, users frequently are at a loss as to how to proceed. 

Mental models also can be an aid to effective instruction. We noted above that a proper mental 

model facilitates accretion. Without one, trainees are forced to engage in a rote memorization 

drill that is tedious and error-prone. Also, material learned in this manner is not easily retained 

or generalized. Further, in the tuning phase of learning, mental models serve a similar purpose. 

They help trainees make sense of what they are being asked to do. It is easier to learn and retain 

something that makes sense as opposed to something that does not. A word of caution is in order 

here: Trainees will develop mental models on their own if not provided with them as part of the 

instruction—and these may not be accurate or entirely appropriate. Inappropriate mental models 

can work against the objectives of the training program. 

2.3 Training Demons: The Enemies of Good Training 

To close the present section, we provide a list of Training Demons, or enemies of good training. 

These are things that we have observed routinely in AMD and Army training. They illustrate 

what can happen when training development and delivery are conducted without regard for the 

basics as described in the previous subsections. 

1. Inadequate job and task analysis: Not understanding what to train. A competent JTA is the 

primary input to the instructional development process. Training is only as good as its 

analytical basis. To the extent this analysis is incomplete or inadequate, trainers run the 

risks associated with the GIGO phenomenon: Garbage In-Garbage Out. 

2. Misplaced training planning emphases: Media dreams, training fads, and silver bullets. 

We remarked earlier that exotic media, games, simulations and other “enhanced” training 

environments are not training per se. They can be used to support learning, but their 

effective use is predicated on a comprehensive understanding of (1) what must be trained, 
10 

(2) a proper performance setting for relevant practice, and (3) feedback requirements. In 

training, there are no silver bullets. 

3. Lock-step, time-based versus criterion-referenced instruction. Much of AMD and Army 

training is lock-step (everyone moves through the curriculum at the same pace and finishes 

at the same time) and time-based (shoe-horned into a fixed allotment of time). Training-tostandard, or criterion-referenced instruction, does not fit well into such a straight jacket. 

People do not learn at the same pace. Course lengths must be based on a consideration of 

what must be trained, competency requirements, and how long that learning process is 

expected to take. Arbitrarily fixing course lengths simply means that some knowledge and 

skills will not be learned to standard, and this will likely be reflected down-stream in 

inadequate on-the-job performance. Exit criteria from training must be some level of 

proficiency, not a fixed number of hours of class time. 

4. Content-based training. Content-based training is most often encountered in academic 

instruction. Its purpose is to teach students about something, not to teach them how to do 

something. Classroom training is not an adequate approach to training skills—how to do 

something. People learn to do by doing. Content-based training is often a workaround for 

an inadequate JTA. We do not know in detail what the job requirements will be, so we 

teach about the job rather than how to do it. 

5. Not understanding that free-play is not deliberate practice. As noted earlier, deliberate 

practice is essential to the development of job competence. This is training conducted 

under the close supervision of a master performer with frequent constructive feedback. 

From our observation, much unit-based training in AMD is more like free play. Trainees 

perform within the context of a scenario without close supervision and the relevant 

feedback that goes with it. Based on the JTA, scenarios must be written to provide the kind 

of structured training and practice required for skill development. Further, there must be 

multiple scenarios for the practice of any particular skill. Hence, scenario construction, 

based on the JTA, is a high priority for effective training. Free play affords few 

opportunities for effective learning and skill development. 

6. Inadequate or poorly-defined standards. A review of AMD and Army training publications 

reveals page after page of tasks, conditions, and standards, sometimes very detailed. The 

task-standard approach to assessing training success might be appropriate for a linear 

function like March Order and Emplacement or System Set-Up. But, for a complex, 

knowledge-based job like AMD ABO, individual task performance standards are not 

enough. The ABO job is not performed as a series of individual, sequential tasks. It is 

entirely possible that a trainee could successfully meet each task performance standard in 

sequence, yet be “clueless” about overall job requirements—fighting the air battle. The 

OIF BOI alluded to this possibility with its remark that “The system [Patriot] is too lethal 

to be placed in the hands of crews trained to such a limited standard.” In a knowledge-
11 

based setting like ABO, standards must address performance requirements for the whole 

job or for meaningful job segments as opposed to individual tasks. Individual task 

standards might be useful during early phases of ABO training, but they are not suitable for 

later tactical training. The ABO job is more than the sum of its individual task parts. 

7. Lack of emphasis on the AAR process. Proper feedback or knowledge of results is 

essential to learning. With respect to AMD ABO, research indicates that proper feedback 

is a critical factor in improving decision accuracy (Kozlowski, 1998). In a discussion of 

air-defense-specific results, Bisantz, Kirlik, Gay, Phipps, Walker, and Fisk (1997) assert 

that knowledge-oriented training for complex judgment tasks must address the skills 

required to consistently execute judgments based on that knowledge. Trainees must be 

taught why their decisions were good or bad. This requires close and continuous contact 

with a master performer acting as coach. 

8. Training folk wisdom. Folk wisdom refers to beliefs about a subject area that are based on 

personal experience uninformed by theory or broader empirical results. In the training 

domain, folk wisdom is exhibited in statements like “Everybody’s a trainer” or “This is the 

way we’ve always done it. It worked for me, and it’ll work for them.” Roth (1998) 

remarks emphatically that job performance and unit experience do not imply instructional 

competence. Training and learning are serious, technical activities. Not to regard them as 

such invites many of the problems alluded to in the first seven of these Training Demons. 

3. AMD Training Dilemmas: Issues for Research and Experimentation 

3.1 Overview 

If the AMD ABO job setting were like most others in the Army or like those that dominated in 

the past, it would be possible to stop our training discussion here. The basics described in the 

previous section, if properly applied, would be sufficient to provide competent job performers. 

However, that is not the case. Compared with routine Army jobs, the AMD ABO job setting has 

more of the following characteristics: 

1. Complex 

2. Ambiguous and uncertain 

3. Non-linear 

4. Knowledge-intensive 
12 

Earlier, we commented on the importance of a comprehensive and up-to-date JTA in developing 

good training for a job like AMD ABO. Two of the characteristics listed above make conducting 

a JTA for such jobs more complex than with routine jobs. First, in most Army jobs (including 

many AMD jobs) task sequences are linear. This means that tasks are executed in a designated 

sequence: A to B to C to D, and so on. Branches in the task sequence are governed by relatively 

well-defined rules: If (Situation), Then (Task E), Else (Task F). Conventional time and error 

metrics can be used as the basis for performance standards. 

In a non-linear job, the task sequence is not so well defined. Instead of linear task sequences, 

there are non-linear task networks. Moreover, for knowledge-intensive jobs, conscious problem 

solving and decision-making—based on cues from the operating environment—dictate which 

actions are performed and in what order. Non-linear, knowledge-based performances also 

introduce varying degrees of equifinality into the task network. Equifinality means that there is 

more than one path through the task network that leads, essentially, to the same outcome. These 

aspects of the AMD ABO job do not fit conveniently into traditional JTA concepts. Moreover, 

traditional task time and error metrics can be problematic in the presence of equifinality. 

The job characteristics listed above also require that training for the ABO job be approached 

differently than more routine activities. A failure to do so on the part of the AMD training 

community was the principal reason for the OIF BOI criticism that Patriot training emphasized 

rote drills rather than flexible problem solving. Jobs characterized by the features listed above 

cannot be adequately trained outside of a problem-solving context. Such jobs also place a 

premium on jobholder expertise. Preparation for them will require an expansion of the Branch’s 

views regarding training in the direction of what has been more typically regarded as education. 

Traditionally, training has been viewed as preparation for a job that can be fully characterized by 

the results of a conventional JTA. Education, on the other hand, has been viewed as appropriate 

for jobs that cannot be so neatly circumscribed. Under a supervisory control regimen, operatorcontroller jobs have evolved away from traditional operator in the direction of system manager. 

What the Army has traditionally called training also must evolve to reflect this change in job 

orientation. 

Within the AMD ABO performance domain, we have identified six training-related dilemmas or 

issues for follow-on research and experimentation. These are issues that remain unresolved and 

thus stand in the way of moving ABO training away from its current crew drill orientation in the 

direction of regarding operator-controllers as managers of lethal and complex systems. These 

dilemmas are: 

1. Initial skill development 

2. Time to train and job progression 

3. Training for unreliable automation 

4. Team training 
13 

5. Adaptability 

6. Mindfulness 

Each of these issues is addressed separately in the subsections to follow. 

3.2 Initial Skill Development 

It is a fact of life that future AMD C2 systems will involve considerable automation and 

accompanying HSC. More than two decades ago, Bainbridge (1983) first commented on one of 

the apparent ironies of automation: An automated performance environment often does not 

provide a good training setting. From a skill development perspective, automation introduces a 

special problem not encountered in less complex systems. In manual systems, operators can 

develop their skills progressively as they move from simple operating situations to more 

complex tasks. Automated systems, on the other hand, often represent an all-or-nothing 

situation. They do not require—and sometimes prevent—direct operator participation in the 

control process. Therefore, an automated performance environment might not support 

progressive skill development. This also means it might not be suitable to use actual equipment 

with embedded training for all phases of operator training. 

Klein (2003) discusses a second skill development problem associated with automated systems. 

He remarks that “smart” technology (like that associated with automation) can make operatorcontrollers “stupid” (ineffective as decision makers) in three ways: 

1. By disabling the expertise of controllers who are already skilled. 

2. By slowing the rate of learning so users do not develop appropriate levels of expertise. 

3. By reinforcing dysfunctional skills that will interfere with users’ ability to achieve 

expertise in the future. 

In the first case, Klein notes that the information screening and filtering often associated with 

automation prevent operator-controllers from finding the information they need to make 

decisions. Operator-controllers thus are denied access to some or all of the decision-cueing 

information essential to effective decision making. One might say that operator-controller 

decision making can be inadvertently short-circuited through data display and access decisions 

made by system developers, who often are not familiar with tactical performance requirements. 

Second, the same information screening and filtering alluded to in the previous paragraph can 

deny novice or journeyman controllers access to the data necessary to form the associations 

between cue sets and environmental patterns that are essential to effective decision making. In 

the first case, information filtering prevents current experts from performing as experts. That 

same filtering can prevent new operator-controllers from ever developing into experts. 

Klein’s third aspect of decision-making impairment involves reinforcing dysfunctional coping 

practices within the automated environment. Here, operator-controllers become passive 
14 

recipients of information from the machine. Klein notes that this passivity tends to make new 

users reluctant to work around problems or strike out on their own to become true experts. They 

lack the background to do so. He further remarks that new users lose or never develop their 

ability to look and search critically within the tactical environment. Smart technology has made 

them passive and ineffective as problem solvers. 

Rasmussen (1986) remarks that in an automated setting the skills required for successful 

performance during extreme situations are not always developed or maintained during normal 

training or operations. He cites three reasons for this potential problem: 

1. Unknown but latent system faults cannot be simulated. 

2. System behavior may not be known for faults that can be predicted but have not been 

experienced. 

3. What was abnormal becomes routine. 

The first two bullets above are self-explanatory: Things will happen that have not been 

anticipated or experienced and thus cannot be explicitly trained. In the third situation, what was 

once considered abnormal enters the training program and becomes routinized over time and 

repetition. The tendency here is not to regard the abnormal situation as representative of a 

potential class of abnormal situations that might occur, but just another isolated fault that has 

been identified and corrected. This illustrates the danger of simply incorporating unbriefed 

surprises in training scenarios without emphasizing these anomalies within a context of 

adaptability and mindfulness. 

With respect to initial skill development, the issue to be resolved going forward is simply stated 

as follows: Is it necessary to learn first in a less automated performance setting? Can jobholders 

adequately master their ABO role if training is conducted solely in an automated practice 

environment? The research results in this area suggest that automation itself might be an 

impediment to the development of the level of expertise required for effective ABO 

performance. 

3.3 Time to Train and Job Progression 

Traditionally, one of the primary articles of faith in Army training is that personnel must be 

trained effectively. Trainees must acquire all of the skills needed to perform all of the tasks 

needed to meet the missions or goals and objectives of the system. Not to do so is to open the 

door to poor on-the-job performance and possibly to court military disaster. In section 2, we 

noted that deliberate practice is the major independent variable in skill acquisition. Moreover, 

the amount of time devoted to training must be based on a consideration of what must be trained 

and standards for acceptable on-the-job performance. Arbitrarily fixing school course lengths or 

the duration of on-the-job training (OJT) means that some knowledge and skills will not be 
15 

learned adequately, and this deficiency will be reflected downstream in inadequate on-the-job 

performance. 

Section 2 also raises another controversial topic: The amount of time in deliberate practice 

required to progress from novice to introductory master performer. A figure of 5,000 hours, or 

two years of full-time instruction, was cited as the minimum amount of time required to quality 

as an entry-level expert. Those figures being on the table, let us consider how much training 

time is currently allocated for U.S. Army Patriot personnel to progress from novice status (no 

Patriot-specific training) to qualification as a Tactical Control Officer (TCO) or Tactical Control 

Assistant (TCA). Estimates vary, but following conversations with subject matter experts 

(SMEs), we arrived at the following times. For officers, the estimate is about eight months. This 

includes approximately four months of Patriot Weapons Track Training as part of the Officers’ 

Basic Course followed by as little as four months of OJT. For enlisted personnel (MOS 14E), 

the Patriot Fire Control Enhanced Operator-Maintainer course during Advanced Individual 

Training (AIT) lasts 19 weeks. However, more than half of 14E AIT is taken up with 

maintenance topics. The other portion is taken up with ABO training. Following the nearly five 

months of AIT, a new 14E could qualify as a TCA with as little as four months of OJT at his or 

her assigned unit—nine months total. These estimates are clearly short of the two years cited in 

the literature for developing in-depth job expertise. 

As part of our exploration of the time-in-job-preparation issue, we also examined training 

patterns for two benchmark cases: 

1. Israeli Patriot 

2. FAA en route air traffic controllers (ATCs) 

Israeli Patriot operators first undergo four months of initial training, which qualifies them as a 

low-level TCA. As a low-level TCA, trainees are allowed to observe in the ECS under 

supervision and perform other battery duties. Trainees remain low-level TCAs for four months 

to one year, depending on their performance-based skill progression. After completing their stint 

as a low-level TCA, trainees undergo an additional two months of high-level TCA training. 

Upon successful completion, they become high-level TCAs and perform as a squad leader. 

The “best” sergeants to complete the high-level TCA course and other leadership courses are 

selected as officer candidates. Selection as an officer candidate is followed by a three-month 

officer’s basic course and a four-month air defense (AD) officer’s course (seven months total). 

Trainees then return to their unit to serve as a low-level TCO. As a low-level TCO, they are not 

assigned to an active crew that responds to real-world alerts. 

After an additional year in the unit with extensive simulation-based training, trainees can qualify 

for promotion to first lieutenant (1LT). Promotion to 1LT is followed by two additional months 

of training in AD tactics. Successful completion of the tactics course qualifies the trainees as 

high-level TCOs. As a high-level TCO, the officer can serve during alerts as part of a three-
16 

officer ECS crew consisting of 1LTs and captains. In total, a minimum of 31 months of training 

interspersed with unit experience is required for qualification as a TCO. 

Additional relevant features of Israeli Patriot training include the following: 

• All TCA and TCO training is strictly on ABO 

• ICC Tactical Directors are selected from tactical ECS crews and undergo an additional six 

weeks of training 

• OJT is simulation-based 

• Periodic hands-on exams are required following certification 

• Disqualification from alert status is automatic upon failure of an examination 

The job of an FAA en route ATC is conceptually similar to a Patriot operator-controller, albeit 

with different ends in mind. To begin the process of becoming an en route controller, job 

candidates undergo a battery of pre-training selection instruments looking at both trainability and 

aptitude for the ATC job. Candidates who pass the pre-training screening phase move on to the 

FAA Academy in Oklahoma City, OK, for four months of intensive resident instruction on en 

route operations. Much of the Academy’s training is simulation-based with a roughly 1:2 

student-to-instructor ratio. One instructor serves as a real-time mentor, while the other performs 

discrete skill checks. Essentially, academy hands-on training is tutorial, with the tutors being 

retired ATCs. 

Following successful completion of Academy training, controller candidates return to their 

assigned center for an additional two to three years of apprenticeship, depending on the 

complexity of the air sectors at that center. Certification—being able to sit at the scope and 

direct traffic alone and without supervision—is performance-based and also depends upon a 

mentor’s assessment of readiness for full certification. Total time to full performance 

certification as an en route ATC is 28-40 months. Should a controller move from one center to 

another, the OJT process starts over. However, once qualified, an experienced controller may 

progress through the OJT sequence faster than a new trainee. 

Job preparation times in each of these cases are considerably longer than the time allocated for 

training in U.S. Army Patriot. Both benchmark situations also are consistent with the literature 

in that more than two years of preparation precedes being declared qualified for an operational 

position. The benchmark situations also illustrate the time-honored preparation sequence for a 

high-skill job: trainee to apprentice to journeyman, with extensive job sample testing along the 

way. Both are also characterized by spending a considerable amount of time in the operational 

setting apart from formal training. The FAA has observed, for example, that spending time in 

the job setting prior to the onset of formal training has a positive impact on performance during 

training. Trainees who have spent time in the organization as an observer-apprentice appear to 

be better able to take advantage of formal training when it is offered. They progress through the 
17 

training sequence faster and academic attrition is lower. Spending time in the operational setting 

apart from formal training permits trainees to acquire what Sternberg et al. (2000) refer to as 

“tacit” knowledge. Tacit knowledge is job-relevant information that is personally important to 

the learner. It has been found to enhance “practical intelligence,” or the ability to find the best fit 

between trainees and their work environment. 

We are not saying that either the FAA or the Israelis have the “right” answer for how long 

training should take for a job like that of Patriot operator-controller. It is worth noting, however, 

that in both benchmark instances the organizations in question keep their job candidates in a 

preparatory status far longer than the U.S. Army—more than three times as long. Given the 

negative comments on Patriot training coming out of OIF, the issue of training time and 

preparation sequence for Patriot operator-controllers must be re-examined. Training time is 

clearly an issue for research and experimentation. It must not be dictated solely by bureaucratic 

constraints or administrative considerations. 

3.4 Training for Unreliable Automation 

One of the unmistakable conclusions from OIF combat operations is that the Patriot system’s 

automation is not perfectly reliable (see Hawley & Mares, 2006). Reliability is defined as being 

dependable or capable of being relied upon. Extending this basic definition to real-time C2 as in 

AMD battle command, a reliable automated system is one in which the functions assigned to the 

machine are performed accurately and appropriately. 

Research indicates that if operator-controllers have a choice, trust in automation determines 

usage. Simply put, operator-controllers will elect not to use a system they do not trust (Lee & 

Moray, 1992). This same body of research also suggests that trust in automation is affected by 

the same factors that influence trust between individuals: effectiveness and reliability (Muir, 

1988). In the case of AMD C2, operator-controllers do not have a choice. They must use the 

system they are provided. So let us next consider the issue of training for situations in which the 

automation is not perfectly reliable. 

Rovira, McGarry, and Parasuraman (2002) remark that one of the true “ironies” of automation is 

that the more reliable the automation, the greater its detrimental effects when it does fail. 

Repeated successful use lulls operator-controllers into a false sense of security or complacency 

regarding the automation’s performance. These authors also report results indicating that 

operators become over-reliant on automation when it provides decision and action choices for 

them and do not check underlying information choices as carefully as when such choices are not 

provided explicitly. Near-perfect automation will lead to a situation in which operator-controller 

decision making is biased in favor of uncritically accepting the automation’s recommendations. 

Wickens, Dixon, and Ambinder (2005) present results indicating that imperfect automation is 

manageable, but users must be pre-warned of the nature and source of the automation’s 
18 

imperfections. These authors also caution that reliabilities less than 75% are worse than no 

automation at all, and can provide users with what they term a “concrete life preserver.” 

Cohen, Parasuraman, and Freeman (1997) and Masalonis and Parasuraman (1999) argue that 

trust in automation should not be all-or-none, but graded and differentiated according to the 

operational context. These authors refer to this as situation-specific trust. The automation may 

work very reliably in certain contexts, in which the operator should use it and trust it. But in 

certain other cases, that the operator-controller has been trained to look for, the automation’s 

recommendations may be suspect. Operators should be told to assess the situation and take the 

action that best suits the context, in their judgment. If operator-controllers can be trained to 

recognize the appropriate context, then they can know when to trust the automation and when its 

recommendations should be discounted. 

In a similar vein, Lee and See (2004) assert that automation should be designed for appropriate

as opposed to greater trust—based on the automation’s expected reliability in handling various 

functions. These authors go on to state that in situations involving imperfect automation, 

operator-controller training must emphasize: 

1. Expected system reliability 

2. The mechanisms underlying potential reliability problems 

3. How usage situations interact with the automation’s technical characteristics to affect 

reliability 

The implications of this brief discussion of automation reliability and its impact on training are 

clear: System developers and users must be brutally honest regarding automation reliability. 

Extensive tests must be performed to determine those situations in which the automation does 

not meet design criteria for reliability. Boundaries of successful system performance must be 

pushed. Moreover, the mechanisms underlying unreliable performance must also be explored. 

Commanders and operator-controllers must then be apprised of system unreliability patterns and 

trained in situations that will expose them to system imperfections. Meeting this challenge will 

be a tall order. It is not clear that even well prepared operator-controllers will be able to make 

the decisions necessary to realize situation-specific trust. Much research and experimentation on 

this issue is required. A good starting point for this work might be the Lens Model approach to 

assessing Dynamic Decision Making performance advocated by Bisantz and her colleagues 

(Bisantz, Kirlik, Gay, Phipps, Walker, & Fisk, 2000; Jha & Bisantz, 2001). 

3.5 Team Training 

It has been noted that units, not individuals are the basis of warfare competence (DSB, 2003). 

This is not to say that individual performance is not important. Individual performance is an 

important component of crew and unit performance. But for the most part, individuals do not 

perform missions. Crews and units do. 
19 

It is also an accepted fact that a crew is more than the sum of its individual parts, and melding 

the disparate parts into a functioning unit takes time working together. Proper performance at 

the crew level requires that individual crewmembers be aware of their interdependencies 

(Kozlowski, 1998). A variety of recent research and operational results suggests that managing 

crew interdependencies may require team process training similar to the Crew Resource 

Management (CRM) programs prevalent in the aviation community. Once almost exclusively an 

aviation program, CRM is now broadly viewed as the use of all available human, informational, 

and equipment resources toward effective and efficient operations in operational domains 

dependant on crew or team performance (Helmreich, Merritt, & Wilhelm, 1999). In the aviation 

world, CRM skills are viewed as a primary line of defense against human error and its 

consequences. The FAA also uses CRM principles to foster cooperation and coordination 

among the controllers in ATC centers. 

Several additional points regarding the development of effective crews are listed as follows: 

1. Effective team training is based on solid individual technical training 

2. Time to train is a key neglected issue in developing effective crews 

3. Leaders are central to the crew development process 

In the first case, it goes without saying that a team is no more able than its least effective 

member. Hence, competent crews first require competent individuals. Second, as we noted at 

the outset, developing effective crew performance takes time working together. It is essential 

that training plans be based on an understanding of how organizations acquire competence. As 

shown in figure 2, competence is developed from the ground up: individuals, then crews, and 

finally multi-echelon units. Inadequate training at any point in the performance chain can shortcircuit the process. 
20 

Figure 2. The development of organizational competence. 

Recent research also emphasizes the importance of leaders in the development of effective 

crews. Kozlowski (1998) notes that leaders foster effective crew performance by (1) melding 

individuals into a coherent team, (2) fostering the development of an adaptive network of roles, 

and (3) assisting the crew in becoming a self-learning system. These three features—coherency, 

an adaptive network of roles, and a self-learning system—differentiate effective teams from a 

simple aggregation of individuals. Effective teams work together, learn as a group, and adjust 

their intra-team roles to reflect changing operational circumstances. 

3.6 Training for Adaptability 

One of the human dimension enablers for the Army’s future force is individual and team 

adaptability. The primary reason for wanting adaptive individuals and teams is to cope with the 

uncertainty that is expected to characterize future operations. Individuals and teams must be able 

to make the necessary modifications to meet emergent challenges. In the AMD domain, crews 

must expect to modify or replace plans. They must expect to improvise. 

Despite the recent emphasis on adaptability, little has been written about what it means to be 

adaptive, and even less has been done to foster the development of adaptive individuals and 

teams (Klein & Pierce, 2001). These authors also remark that much of what takes place during 

routine military training and operations has an effect counter to what is desired. That is, much of 

current training and operations acts to produce non-adaptive individuals and crews. Specifically, 

these aspects of traditional training include: 

1. Training for mastery of task routine to the exclusion of problem solving 

2. Using the crawl, walk, run method of training while avoiding trainee mistakes 
21 

In the first case, the usual emphasis of training is on mastering routines rather than adaptive 

problem solving. Klein and Pierce (2001) refer to the results of this practice as 

“experiosclerosis.” Crews believe they are experts and combat ready because they are good at 

the routines, but the routines can prove to be a strait jacket during combat. Traditional individual 

and unit evaluation practices reinforced this mistaken belief on the part of crews and 

commanders at all levels by focusing only on satisfactory performance of routine drills. OIF 

BOIs specifically criticized Patriot training for its focus on crew drills while not addressing 

adaptive problem solving. 

The crawl, walk, run method of training often means that training is not challenging or 

threatening. However, trainees learn from their mistakes. If they are not allowed to make and 

learn from mistakes, trainees do not develop the mental models that underlie effective problem 

solving performance. Klein and Pierce note that learning from both successes and failures is 

rapid. Accordingly, advanced scenarios must require trainees to face ambiguities and to 

reconsider options and re-plan. Crews must rehearse familiar scenarios of failure and strive to 

imagine novel ones. Failure, in present usage, means situations in which intended results were 

not achieved or adverse incidents occurred. Fratricides like those that occurred during OIF are 

examples of adverse incidents. While this approach might sound straightforward, it is not easily 

put into practice. 

Klein and Pierce caution that while most crews can become adaptive, most will not achieve that 

level of expertise. The most likely reason for not becoming adaptive has to do with the 

requirement for long hours of deliberate practice and reflection discussed in section 2. Aside 

from trainee motivation, there also are other roadblocks to individuals and crews becoming 

adaptive. These have to do with the availability of master performers to serve as mentors, 

training equipment availability, adequate training scenarios, and a variety of routine bureaucratic 

constraints and administrative practices. It can be argued that in the current training 

environment, adaptive crews are likely to be the exception rather than the rule. 

3.7 Mindfulness 

The challenge facing the AMD community is developing units that are capable of sustained high 

reliability in a complex, unpredictable operational setting. Daunting as this challenge might 

seem, there is a class of organizations that have managed to do just that—maintain high 

performance in complex and unpredictable environments. Weick and Sutcliffe (2001) refer to 

such organizations as high-reliability organizations (HROs). Examples of HROs include air 

traffic control facilities, nuclear submarines, and aircraft carrier deck operations. As one FAA 

ATC manager put it: “99.9% operational reliability is not good enough for us. That would mean 

we would have several dozen ‘noticeable’ incidents each working day. And that would bring our 

operations to a complete halt” (M. Morrison, personal communication, August, 30, 2005). 
22 

Weick and Sutcliffe remark that HROs are characterized by ways of acting and leadership styles 

that enable them to manage the unexpected better than most kinds of organizations. These 

authors note that HROs manage the unexpected through five processes: 

1. Preoccupation with failures rather than successes. Performance lapses and near misses are 

treated as symptoms that something is wrong with the system—something that could have 

severe consequences if a combination of lapses were to coincide at the same point in time. 

Consequently, HROs study incidents and near misses for what can be learned. They are 

wary of the potentially disastrous consequences of complacency and a drift into automatic 

processing. 

2. Reluctance to simplify interpretations. HROs are reluctant to accept simplified 

explanations of potential trouble spots. They encourage critical inquiry across the 

organization, and are skeptical of “received wisdom” from outside sources who are often 

less knowledgeable about line conditions. 

3. Sensitivity to operations. HROs are characterized by sensitivity to line operations. This is 

defined as a focus on understanding the situational aspects of organizations operations. 

They do not permit a disconnect between operations as viewed from the top and operations 

as implemented on the front line. 

4. Commitment to resilience. HROs accept that no system is perfect. They expect that errors 

will occur and train their crews to recognize and recover from them. Following adverse 

incidents or near misses, HROs focus on system-wide fixes rather than item-by-item 

repairs. This requires a deep knowledge of the technology, the system, crews, and 

individuals. Consequently, HROs place a premium on expertise: personnel with extensive 

experience, recombination skills, and training. 

5. Deference to expertise, as exhibited by encouragement of a fluid decision-making system. 

HROs recognize that rigid hierarchies are vulnerable to error. Rigid C2 structures are 

suitable for a stable world, but can be an impediment when adaptability is required. HROs, 

on the other hand, push decision making down and around in the organization. Decisions 

are made on the front line; authority migrates to the people with the most expertise, 

regardless of rank. 

Together, these five processes produce an attitude of mindfulness or “intelligent wariness.” To 

be mindful is to have an awareness of detail and an enhanced ability to identify and prevent 

errors that could escalate into an adverse event. How can an organization manage the 

unexpected? Weick and Sutcliffe assert that organizations can manage the unexpected by acting 

more like a HRO. 

Desirable as it might be, acting more like a HRO is neither simple nor easy. For example, Weick 

and Sutcliffe caution that it is difficult for individuals and crews to remain chronically wary 

about their operations. Moreover, several of the HROs noted in the previous discussion (e.g., 
23 

nuclear submarines and air traffic control) have had to create distinct supporting subcultures that 

sometimes put them at odds with their parent organizations (see Bierly, 1995). They have had to 

take and hold to the position that effective and reliable line operations come first and must be 

emphasized above broader organizational policies and practices. Their imperative is to make 

their operations as effective and robust as practical in the face of sometimes unpredictable 

engineering, human, and operational vulnerabilities. 

4. A Path Forward for AMD 

4.1 Overview 

The final section of the report concerns a path forward for AMD. In developing this section, it is 

not our intent to tell others their business. That is, it is not our intent to get deeply into Army or 

AMD training design and development processes, other than to suggest areas where 

improvements to increase effectiveness and organizational reliability are indicated. These 

suggestions for improvement are taken from the human performance and training literature as 

well as from lessons learned in parallel organizations. A parallel organization is one engaged in 

operations and training for jobs similar to AMD operator-controllers. Again, the focus of the 

following discussion is training for ABO. 

The discussion to follow is also presented in two parts: (1) training for routine operations and 

(2) training for adaptability. Training for routine operations addresses enhancements directed at 

improving effectiveness and reliability in situations involving little ambiguity or uncertainty. 

The intent here is to develop personnel and crews well practiced in the basics of AMD 

operations. Training in advanced aspects of AMD operations such as coping with unreliable 

automation must be based on a firm foundation in AMD and system fundamentals. 

Training for adaptability is based upon competent baseline performance by individuals and 

crews. It might be termed advanced training for situations that are non-routine or out-of-theordinary. As the name implies, it is intended to foster the adaptability and mindfulness necessary 

to cope with unreliable automation and the inevitable surprises of combat operations. 

4.2 Training for Routine Expertise 

Our observations of AMD training and operations plus material from the literature cited in the 

previous sections suggest six areas for improvement in AMD training for routine operations. 

These points are listed and discussed as follows. 

1. Job and task analysis. We remarked earlier that training design can be no better than the 

JTA on which it is based. Training developers must understand the performance context 
24 

and what trainees must learn to perform competently in that environment. This goal cannot 

be met if the JTAs upon which training are based are simply updates of previous JTA 

material. The first and second reports in this series (Hawley, Mares, & Giammanco, 2005; 

Hawley & Mares, 2006) emphasize that automated operations present a different 

performance setting from that associated with traditional manual control. These 

differences must be reflected in training and job aiding. To the extent these performance 

changes are not addressed during the JTA and subsequent training development, the 

resulting training will be inappropriate and sometimes ineffective vis-à-vis actual 

performance requirements. 

2. Scenario content. We noted previously that in training for a job like AMD operatorcontroller, the scenario is the curriculum. This means that training scenarios must be 

carefully scripted to cue the decisions and performances indicated in the JTA as being 

critical. Training managers must also be aware that training scenarios have a limited shelf 

life. After more than about three uses with the same group of trainees, they lose their 

training value. Trainees recognize the scenario and begin to “game” it. Gaming scenarios 

destroys their learning value, since gaming rather than learning becomes the objective of 

the exercise. 

3. Performance standards. Training managers must attend to the development of operational 

performance standards for scenarios and for success at the various certification levels. 

Skill checks for individual tasks or portions of tasks are important in skill development, but 

they are not sufficient for judging job competence. Standards must be objectively stated in 

terms of successful performance against benchmark scenarios having stated performance 

objectives and known difficulty levels. In ABO training, the whole defined as competent 

job performance is more than the sum of its individual task parts. Standards must also be 

applied uniformly in institutional and unit training. Trainers and commanders must know 

where the bar defining competent job performance is set. 

4. Time to reach desired competency levels. Time to train to desired competency levels is 

one of recurring problems in Army and military training in general (DSB, 2003). Training 

times must be determined on the basis of how long individuals take to reach desired 

competency levels. A situation must not be allowed to exist in which training is 

shoehorned into available times or is fixed arbitrarily on the basis of bureaucratic or 

administrative considerations. If such considerations are permitted to dictate training 

times, the result will be incomplete training and inadequate on-the-job performance. In 

training, there is no way to escape the fact that you get what you pay for. 

5. Qualified instructors. There is an old adage that it is hard for an apprentice to learn if there 

are no masters. Personnel assigned to training jobs in the institution and in units must be 

expert job performers (EJPs). An EJP is a SME who also has recent and relevant on-thejob experience. Their job is to teach trainees how to do the job, not just to teach them 
25 

about the job. Consideration must also be given to the teaching ability of potential 

instructors. Roth (1998) cautions, for example, that system experience is not a guarantor of 

effective instructional capability. Also, Norman (1993) cites results indicating that a 

course’s reputation is a major determinant of trainee motivation. If a course has a good 

reputation that word will spread and trainees will come to the course motivated and 

prepared to learn. On the other hand, if a course has a bad reputation, that fact will be 

reflected in an unwillingness to come or attend to the material. Trainees will use any 

excuse to get out of attending. Requiring attendance will not always result in effective 

learning. 

6. The AAR process. A generally accepted axiom of experimental psychology is that no 

learning takes place in the absence of feedback. This simple fact underlies much of the 

success attributed to the Army’s training at Combat Training Centers (CTCs) such as the 

National Training Center (NTC). At the CTCs, considerable effort goes into the postexercise AAR. Training design can be exceptional; scenarios can be great; but if the AAR 

process is deficient, learning will be inadequate. The same relationship applies to lower 

echelon training conducted in institutions and units outside the CTC context. Good AARs 

go well behind a mere replay or recap of what went on. They consist of a penetrating 

behavioral critique that is designed to change the way trainees think about the 

performances in question. Trainees must be told not only that a particular performance was 

right or wrong, but also why that performance was right or wrong. Good AARs are critical 

to tuning the mental models upon which future performance will be based. 

4.3 Training for Adaptive Expertise 

As noted previously, training for adaptability is intended to foster the ability to think outside the 

box defined by routine crew drills. It is intended to provide operator-controllers with the ability 

to cope with unreliable automation and the inevitable unforeseen events that characterize combat 

operations. In one sense, the mechanisms underlying training for adaptability are 

straightforward. As the DSB recommended, trainees should face confusing and complex 

scenarios that contain unbriefed surprises. Such scenarios must push trainees and crews outside 

their comfort zone and stress problem solving over routine operational processes. Crews 

working together must recognize when a situation is not ordinary and requires problem-solving 

intervention. If there is a key to achieving this result, the operative word is expertise. Not to 

confront issues associated with expertise as defined in this report invites a drift toward automatic 

processing and the kinds of performance problems that led to the Patriot Vigilance effort. 

As desirable and straightforward as training for adaptive expertise might seem, producing 

adaptive individuals and crews will not be a simple undertaking. We cite again Klein and 

Pierce’s (2001) caution that most crews can become adaptive, but most will not. Why not? In 

our observation, three roadblocks stand in the way: 
26 

1. Time and job progression practices 

2. Training quality 

3. Trainee motivation 

First, achieving adaptive expertise will require more time for training than the Army has 

traditionally allocated for AMD operator-controller job preparation. Simply put, there is no way 

to avoid the 5,000 hour rule that applies in other high-skill situations. Further, intra-unit job 

progression patterns will have to change. Operator-controller trainees will have to spend an 

extended period in an apprentice status while they acquire the skills and experience necessary for 

effective job performance. Unit metrics regarding qualified crews will have to change, and it is 

not certain that current personnel management practices and concerns can accommodate such a 

requirement. 

Second, many qualitative aspects of operator-controller training will have to change. To begin, 

training for what we have termed routine operations will have to be more rigorous and 

performance-oriented than at present. Training content and scenarios must reflect job 

requirements, and standards must be rigorously applied across the board. Introductory, baseline 

training will have to be followed by crew-oriented training that emphasizes fluid decisionmaking within an adaptive network of roles. This will require intact crews, special training such 

as CRM, competent instructors, and—above all—time to form this collective expertise. Crew 

leadership also is important to adaptive team performance. 

Third, trainees must be motivated to develop the deep expertise in AMD technology, weapons 

systems, and operations necessary to inform the decision processes that characterize being 

adaptive. This is the reflective aspect of job preparation. As we noted in the previous section, 

the proper setting for such preparation is self-directed study and professional military education. 

Trainees, commanders, and the general Army culture must accept that preparation for a job like 

operator-controller involves no less professionalism than preparation for command or any other 

high-skill job. The topics and focus are different, but the preparatory requirements are similar. 

We have discussed them extensively in this report. 

An example from the Army Aviation community might clarify and illustrate this third point. 

Army Aviation uses a special Warrant Officer career progression track to increase 

professionalism and expertise among enlisted aviators. Warrant Officer aviators are officers, are 

highly selected, are granted honor and deference by all concerned, and are expected to be highly 

trained professionals who operate at the highest levels of expertise. Moreover, it is anecdotally 

reported that instructor pilots demand more of Warrant Officer students than of Commissioned 

Officer students (D. M. Johnson, personal communication, January 10, 2006). Commissioned 

Officer students eventually grow up and become staff officers, administrators, and commanders; 

but Warrant Officer students develop as expert aviators and remain in this role throughout their 

career. The emerging job of an AMD operator-controller is no less demanding than that of an 
27 

Army Aviator, and skill and professional development for these personnel must be approached in 

a similar manner. 

4.4 The Training Challenge 

Achieving the objectives cited in the previous sections will not be easy. To begin, commanders 

and training managers must recognize that the Army’s crew drill mentality is a major part of the 

problem associated with preparing Soldiers for knowledge-intensive jobs. As Klein and Pierce 

(2001) caution, the crew drill mentality discourages adaptive problems solving and almost 

guarantees a drift toward automatic, unthinking processing. Its mantra is, “This is the way we do 

it, and any other way is wrong.” Crew drills are appropriate for some job situations, but 

inappropriate in others. They are not suitable for ABO training. 

In the report Training for Future Conflicts, the DSB asserts that the future will require that more 

of our people do new and more complicated things (DSB, 2003). That same report also remarks 

that meeting this challenge amounts to a “qualitative change in the demands upon our people that 

cannot be supported by traditional training practices” (p. 6). The DSB report concluded that 

training transformation to support warfare transformation will be a challenging undertaking. 

Our assessment of the path forward for AMD training is consistent with the DSB’s conclusion. 

Technology and the operating environment are driving changes in weapons systems, and these 

combined changes require a change in training concepts and practices. Not to change means that 

AMD training will become increasingly unsuited to operational requirements. The 2003 DSB 

report cautions that training failures can negate promising hardware and technologies. That 

warning applies directly to the technology-intensive AMD situation. 
28 

5. References 

Bainbridge, L. Ironies of automation. Automatica 1983, 19, 775–779. 

Bierly, P. E., III. Culture and high reliability organizations: The case of the nuclear submarine. 

Management Review 1995, Winter, 1–17. 

Bisantz, A.; Kirlik, A.; Gay, P.; Phipps, D.; Walker, N.; Fisk A. Specifying training needs in 

dynamic decision tasks using a Lens Model approach. In Proceedings of the 1997 IEEE 

Transactions on Systems, Man and Cybernetics, 1997. 

Bisantz, A.; Kirlik, A.; Gay, P.; Phipps, D.; Walker, N.; Fisk, A. Modeling and analysis of 

dynamic judgment tasks using a Lens Model approach. IEEE Transactions on Systems, Man 

and Cybernetics 2000, 30 (6), 1–12. 

Cannon-Bowers, J. A.; Burns, J. J.; Salas, E.; Pruitt, J. S. Advanced technology in scenariobased training. In J. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress: 

Implications for individual and team training. Washington, DC: American Psychological 

Association, 1998. 

Cohen, M. S.; Parasuraman, R.; Freeman, J. T. Trust in decision aids: A model and its training 

implications; Technical Report; Arlington, VA: Cognitive Technologies, Inc, 1997. 

Defense Science Board. Training for future conflicts. (Final Report of the DSB Task Force on 

Training for Future Conflicts). Washington, DC: Office of the Undersecretary of Defense 

for Acquisition, Technology, and Logistics, 2003. 

Defense Science Board. Patriot system performance. (Final Report of the DSB Task Force on 

Patriot System Performance). Washington, DC: Office of the Undersecretary of Defense for 

Acquisition, Technology, and Logistics, 2004. 

Endsley, M. R. Automation and situation awareness. In R. Parasuraman & M. Mouloua (Eds.), 

Automation and human performance: Theory and applications. Hillsdale, NJ: Erlbaum, 

1996. 

Ericsson, K. A.; Charness, N. Expert performance: Its structure and acquisition. American 

Psychologist 1994, 49 (8), 725–747. 

Ericcson, K. A.; Lehmann, A. C. Expert and exceptional performance: Evidence of maximal 

adaptation to task constraints. Annual Review of Psychology 1996, 47, 273–305. 

Hawley, J. K.; Mares, A. L.; Giammanco, C. A. The human side of automation: Lessons for air 

defense command and control; ARL-TR-3468; U.S. Army Research Laboratory: Adelphi, 

MD, 2005. 
29 

Hawley, J. K.; Mares, A. L. Developing effective human supervisory control for air and missile 

defense systems; ARL-TR-3742; U.S. Army Research Laboratory: Adelphi, MD, 2006. 

Helmreich, R. L.; Merritt, A.C.; Wilhelm, J. A. The evolution of Crew Resource Management in 

commercial aviation. International Journal of Aviation Psychology 1999, 9 (1), 19–32. 

Jha, P.; Bisantz, A. Modeling fault diagnosis in a dynamic process control task using a 

multivariate lens model. In Proceedings of the Human Factors and Ergonomics Society 45th

Annual Meeting. Santa Monica, CA: Human Factors and Ergonomics Society, 2001. 

Klein, G. Intuition at work. New York: Doubleday, 2003. 

Klein, G.; Pierce, L. Adaptive teams. In Proceedings of the 6th International Command and 

Control Research and Technology Symposium. Annapolis, MD, 2001. 

Kozlowski, S.W.J. Training and developing adaptive teams: Theory, principles, and research. 

In J. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress: Implications for 

individual and team training. Washington, DC: American Psychological Association, 1998. 

Lee, J. D.; Moray, N. Trust, control strategies, and allocation of function in human-machine 

systems. Ergonomics 1992, 35, 1243–1270. 

Lee, J. D.; See K. A. Trust in automation: Designing for appropriate reliance. Human Factors 

2004, 46 (1), 50–80. 

Masalonis, A. J.; Parasuraman, R. Applying the trust concept to the study of human-automation 

interaction. Washington, DC: Cognitive Sciences Laboratory, The Catholic University of 

America, 1999. 

Muir, B. M. Trust between humans and machines and the design of decision aids. In E. 

Hollnagel, G. Mancini, & D. D. Woods (Eds.), Cognitive engineering in complex dynamic 

systems. London: Academic Press, 1988. 

Norman, D. A. Things that make us smart. Cambridge, MA: Perseus Books, 1993. 

Norman, D. A. The design of everyday things. New York: Basic Books, 2002. 

Rasmussen, J. Information processing and human-machine interaction: An approach to 

cognitive engineering. New York: North Holland, 1986. 

Roth, J. T. Improving decision-making skills through on-the-job training: A roadmap for 

training shipboard trainers. In J. Cannon-Bowers & E. Salas (Eds.), Making decisions under 

stress: Implications for individual and team training. Washington, DC: American 

Psychological Association, 1998. 
30 

Rovira, E.; McGarry, K.; Parasuraman, R. Effects of imperfect automation on decision making 

in command and control. In Proceedings of the Human Factors and Ergonomics Society 46th

Annual Meeting (pp. 428-432). Santa Monica, CA: Human Factors and Ergonomics 

Society, 2002. 

Sheridan, T. B. Telerobotics, automation, and human supervisory control. Cambridge, MA: 

MIT Press, 1992. 

Sheridan, T. B. Humans and automation: System design and research issues. New York: 

Wiley, 2002. 

Sternberg, R. J.; Forsythe, G. B.; Hedlund, J.; Horvath, J. A.; Wagner, R. K.; Williams, W. M. 

Practical intelligence in everyday life. New York: Cambridge University Press, 2000. 

Weick, K. E.; Sutcliffe, K. M. Managing the unexpected: Assuring high performance in an age 

of complexity. San Francisco: Jossey-Bass, 2001. 

Whitmore, P. G. How to make smart decisions about training: Save time, money & frustration. 

Atlanta, GA: CEP Press, 2002. 

Wickens, C. D.; Dixon, S. R.; Ambinder, M. S. Workload and automation reliability in 

unmanned air vehicles; Draft Technical Report; U.S. Army Research Laboratory: Adelphi, 

MD, 2005. 
31 

Distribution List

ADMNSTR 

DEFNS TECHL INFO CTR 

ATTN DTIC-OCP (ELECTRONIC COPY) 

8725 JOHN J KINGMAN RD STE 0944 

FT BELVOIR VA 22060-6218 

DARPA 

ATTN IXO S WELBY 

3701 N FAIRFAX DR 

ARLINGTON VA 22203-1714 

OFC OF THE SECY OF DEFNS 

ATTN ODDRE (R&AT) 

THE PENTAGON 

WASHINGTON DC 20301-3080 

US ARMY TRADOC 

BATTLE LAB INTEGRATION & TECHL 

DIRCTRT 

ATTN ATCD-B 

10 WHISTLER LANE 

FT MONROE VA 23651-5850 

SMC/GPA 

2420 VELA WAY STE 1866 

EL SEGUNDO CA 90245-4659 

US ARMY ARDEC 

ATTN AMSTA-AR-TD 

BLDG 1 

PICATINNY ARSENAL NJ 07806-5000 

COMMANDING GENERAL 

US ARMY AVN & MIS CMND 

ATTN AMSAM-RD W C MCCORKLE 

REDSTONE ARSENAL AL 35898-5000 

US ARMY INFO SYS ENGRG CMND 

ATTN AMSEL-IE-TD F JENIA 

FT HUACHUCA AZ 85613-5300 

DIRECTOR 

DIRECTORATE OF COMBAT 

DEVELOPMENTS 

ATTN COL H L COHEN 

5800 CARTER RD 

FT BLISS TX 79916-7001 

DIRECTOR 

DIRECTORATE OF TRAINING, DOCTRINE, 

AND LEADER DEVELOPMENT 

ATTN COL R BURKE 

2 SHERIDAN RD BLDG 2 

FT BLISS TX 79916-7001 

OFFICE, CHIEF OF AIR DEFNS ARTILLERY 

ATTN LTC J H JENKINS III 

2 SHERIDAN RD BLDG 2 

FT BLISS TX 79916-7001 

TRADOC SYSTEM MANAGER-LOWER 

TIER 

ATTN COL R JASSEY 

BLDG 12, PERSHING RD 

FT BLISS TX 79916-7001 

TRADOC SYSTEM MANAGER-UPPER TIER 

ATTN COL S PETERS 

BLDG 12, PERSHING RD 

FT BLISS TX 79916-7001 

COMMANDING GENERAL 

US ARMY AIR DEFNS ARTILLERY CTR 

AND FT BLISS 

ATTN BG R P LENNOX 

FT BLISS TX 79916-7001 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-HR-ME 

 A MARES (1 HC, 1 ELEC) 

ROOM 143 

FT BLISS TX 79916 

US ARMY SIMULATION TRAIN & 

INSTRMNTN CMND 

ATTN AMSTI-CG M MACEDONIA 

12350 RESEARCH PARKWAY 

ORLANDO FL 32826-3726 

US GOVERNMENT PRINT OFF 

DEPOSITORY RECEIVING SECTION 

ATTN MAIL STOP IDAD J TATE 

732 NORTH CAPITOL ST., NW 

WASHINGTON DC 20402 
32 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-CI-OK-TP 

 TECHL LIB T LANDFRIED (2 HC) 

ABERDEEN PROVING GROUND MD 21005-

5066 

US ARMY RSRCH LAB 

ATTN AMSRL-ARL-HR-MB 

 C GIAMMANCO (4 HC, 1 ELEC) 

BLDG 459 

ABERDEEN PROVING GROUND MD 21005-

5425 

DIRECTOR 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-RO-EV W D BACH 

PO BOX 12211 

RESEARCH TRIANGLE PARK NC 27709 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-HR-ME 

 J HAWLEY (10 HC, 1 ELEC) 

FT BLISS TX 79916 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-D J M MILLER 

ATTN AMSRD-ARL-CI-OK-T 

 TECHL PUB (2 HC) 

ATTN AMSRD-ARL-CI-OK-TL 

 TECHL LIB (2 HC) 

ATTN IMNE-ALC-IMS 

 MAIL & RECORDS MGMT 

ADELPHI MD 20783-1197