Looking Back at 20 Years of MANPRINT on Patriot: 

Observations and Lessons 

by John K. Hawley 

ARL-SR-0158 September 2007 

Approved for public release; distribution unlimited. 
NOTICES 

Disclaimers 

The findings in this report are not to be construed as an official Department of the Army position 

unless so designated by other authorized documents. 

Citation of manufacturer’s or trade names does not constitute an official endorsement or 

approval of the use thereof. 

Destroy this report when it is no longer needed. Do not return it to the originator. 
Army Research Laboratory 

Adelphi, MD 20783-1197 

ARL-SR-0158 September 2007 

Looking Back at 20 Years of MANPRINT on Patriot: 

Observations and Lessons 

John K. Hawley 

Human Research and Engineering Directorate, ARL 

Approved for public release; distribution unlimited. 
ii 

REPORT DOCUMENTATION PAGE Form Approved 

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently 

valid OMB control number. 

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YYYY)

September 2007 

2. REPORT TYPE 3. DATES COVERED (From - To) 

5a. CONTRACT NUMBER 

5b. GRANT NUMBER 

4. TITLE AND SUBTITLE 

Looking Back at 20 Years of MANPRINT on Patriot: Observations and 

Lessons 

5c. PROGRAM ELEMENT NUMBER 

5d. PROJECT NUMBER 

5e. TASK NUMBER 

6. AUTHOR(S) 

John K. Hawley 

5f. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

U.S. Army Research Laboratory 

ATTN: AMSRD-ARL-HR-ME 

2800 Powder Mill Road 

Adelphi, MD 20783-1197

8. PERFORMING ORGANIZATION 

 REPORT NUMBER 

ARL-SR-0158

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S) 

11. SPONSOR/MONITOR'S REPORT 

 NUMBER(S) 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution unlimited. 

13. SUPPLEMENTARY NOTES 

14. ABSTRACT 

During the combat operations phase of Operation Iraqi Freedom (OIF), Patriot air and missile defense units were involved in 

two fratricide incidents. Patriot’s unacceptable fratricide rate during OIF (18% of engagements) prompted the commanding 

general of the Army Air and Missile Defense Center to request a human-performance-oriented assessment of the fratricide 

incidents to complement the official Army board if inquiry investigation. This report summarizes the results and 

recommendations from that assessment. Recommendations for a solution to the fratricide problem involved both command 

and control and training modifications. The paper’s primary focus is MANPRINT observations and lessons from the Army’s 

25-year developmental effort with Patriot. Specific observations on the Patriot MANPRINT program gleaned from a review of 

assessments and test reports going back 20 years are presented and discussed. This material is followed by a discussion of 

broader lessons from the Patriot MANPRINT program. These broader lessons include: (1) going-in concepts really matter, (2) 

training issues really matter, (3) testing must be more comprehensive and rigorous, and (4) lessons must be learned. 

Implications of the specific observations and broader lessons for the MANPRINT program going forward also are presented 

and discussed. The observations and lessons discussed in the report are presented in the spirit of action research, defined as 

research that any community of practice can do to improve its methods. In this sense, the spirit of the report is institutional

learning and practice improvement rather than after-the-fact criticism. 

15. SUBJECT TERMS

Patriot, training, adaptive expertise, simulation 

16. SECURITY CLASSIFICATION OF:

19a. NAME OF RESPONSIBLE PERSON

John K. Hawley 

a. REPORT 

Unclassified 

b. ABSTRACT 

Unclassified 

c. THIS PAGE 

Unclassified 

17. LIMITATION 

OF ABSTRACT 

SAR 

18. NUMBER 

OF PAGES 

28 19b. TELEPHONE NUMBER (Include area code) 

915-568-2896 

Standard Form 298 (Rev. 8/98) 

Prescribed by ANSI Std. Z39.18 
iii 

Contents 

List of Figures iv

Introduction 1

Background...............................................................................................................................1 

The Patriot Vigilance Project ...................................................................................................2 

Follow-On Work, Implementation, and Current Status ...........................................................6 

MANPRINT Observations and Broader Lessons 8

Observations on Patriot MANPRINT ......................................................................................8 

Broader Lessons from Patriot.................................................................................................10

MANPRINT Going Forward 15

References 18

Distribution List 20
iv 

List of Figures 

Figure 1. Patriot vigilance logic model.......................................................................................... 3 

Figure 2. Ladder of sociotechnical concerns. .............................................................................. 17 
1 

Introduction 

Background 

During the combat operations phase of Operation Iraqi Freedom (OIF), Army Patriot units were 

involved in two fratricide incidents. In the first, a British Tornado was misclassified as an antiradiation missile (ARM) and subsequently engaged and destroyed. The second fratricide 

incident involved a Navy F/A-18 that was misclassified as a tactical ballistic missile (TBM) and 

also engaged and destroyed. Three flight crew members lost their lives in these incidents. OIF 

involved a total of 11 Patriot engagements by U.S. units. Of these 11, nine resulted in successful 

TBM engagements; the other two were fratricides. 

Patriot is the Army’s first-line air and missile defense (AMD) system. The system has been in 

the active force since the early 1980s. Initially, Patriot was intended as a defense against 

conventional air-breathing threats (ABTs). However, since Operation Desert Storm (ODS) in the 

early 1990s, the system has been used primarily against TBMs. Future usage scenarios envision 

the system being used against a spectrum of air threats including TBMs, conventional ABTs, 

cruise missiles, and various categories of unmanned aerial vehicles (UAVs). The range of 

potential air threats in the contemporary operating environment has significantly increased the 

complexity of the battle command problem for Patriot and other AMD systems. 

Since Patriot is an existing system and has been in the Army’s inventory since the early 1980s, 

what do lessons from Patriot tell us about other systems, particularly those at the concept stage or 

under development? As Patriot has evolved over the past two decades, the system has acquired 

features and characteristics that are more typical of systems the Army will employ in the future 

than those in the current inventory. Terms that are now used to describe Patriot include (1) joint, 

(2) network-centric, (3) complex, and (4) knowledge-intensive. First, command and control (C2) 

for the Patriot system is joint—involving both the Army and Air Force, and sometimes the Navy. 

Second, effective employment of system assets is dependent on a robust network. Third, the 

system as broadly defined is complex in that it consists of a large number of interacting 

components. And fourth, Patriot is knowledge-intensive in terms of the amount of information 

required to characterize and comprehend the system. It can be argued that Patriot is a relevant 

data point regarding human performance issues the Army is likely to face with emerging high 

technology and network-centric systems. Moreover, this glimpse into the future is tangible and 

real and not abstract or hypothetical. The lessons discussed in this paper are from the crucible of 

combat operations and not based solely on the results of operational tests or simulated exercises. 

The central focus of the paper is, however, human system integration (HSI) or MANPRINT 

(Manpower and Personnel Integration) observations and lessons from the Army’s 25-year 

developmental experience with Patriot. [Note: MANPRINT is the Army’s HSI initiative. The 
2 

program was started several years after Patriot was initially fielded.] This retrospective case 

study assessment of HSI practices, successes, failures, and lessons was made possible because of 

a unique research opportunity that resulted from the Patriot fratricides during OIF: The Patriot 

Vigilance project. The next section provides an overview of the Patriot Vigilance project and its 

results. Prior to proceeding, it should be noted that the discussion to follow is presented in the 

spirit of action research, defined as self-directed research that any community of practice can do 

to improve its methods with the aim of improving its strategies, methods, and knowledge of the 

environment in which it practices (Schein, 2004). The intent of the report is institutional 

learning and HSI practice improvement rather than simple after-the-fact criticism. 

The Patriot Vigilance Project 

Personnel from the Army Research Laboratory’s Human Research and Engineering Directorate 

(ARL HRED) began looking into Patriot and AMD performance and training issues at the 

invitation of the then Ft. Bliss Commander, Major General (MG) Michael A. Vane. MG Vane 

was interested in operator vigilance and situational awareness (SA) as they relate to the 

performance of automated AMD battle command systems. [Note: The generally accepted 

definition of SA is from Endsley, Bolte, and Jones (2003) who define it as the perception of 

elements in the environment, the comprehension of their meaning, and the projection of their 

status in the near future.] MG Vane was particularly concerned by what he termed a “lack of 

vigilance” on the part of Patriot operators along with an apparent “lack of cognizance” of what 

was being presented to them on situation displays and a resulting “absolute trust in automation.” 

His request for human factors support was prompted by the unacceptable rate of fratricidal 

engagements by Patriot units during OIF—two out of a total of 11 engagements, or 18%. MG 

Vane’s reference to lack of vigilance by Patriot operators led to the effort being called the Patriot 

Vigilance project. 

Following general approaches to human error investigations and case study research outlined in 

Dekker (2002) and Yin (2003), respectively, the project staff spent most of the summer and fall 

of 2004 performing a human-performance-oriented critical incident assessment of the OIF 

fratricides. This involved activities such as reading documents from the fratricide boards of 

inquiry (BOIs), interviewing knowledgeable personnel in the Ft. Bliss area, and observing Patriot 

training and operations. An initial assessment briefing was delivered to MG Vane in October 

2004. The project staff also prepared a supporting technical report describing the human 

performance problems associated with automation and supervisory control (Hawley, Mares, & 

Giammanco, 2005). 

The logic model (see Yin, 2003) resulting from HRED’s critical incident assessment of the OIF 

fratricides is presented in figure 1. In developing this logic model, the project staff was 

responding to MG Vane’s request to explain “how we got to the OIF incidents.” He was not 

interested in a further dissection of the specifics of the incidents, since those details had been the 

focus of the various BOIs convened to examine the fratricides. Rather, MG Vane was interested 
3 

in understanding how the branch got into a situation in which those incidents were almost 

inevitable. His observations concerning Patriot operator performance cited above speak directly 

to the General’s view of the situation. The logic model was constructed with this intent in mind. 

It is explanatory in a broad, conceptual manner rather than in a narrow, technical sense (e.g., a 

step-by-step dissection of operator actions). HRED’s intent also was to point the way to 

actionable solutions rather than to lay further blame. 

The first block in the causal network leading to the OIF fratricides is termed “undisciplined 

automation,” defined as the automation of functions by designers and subsequent implementation 

by users without due regard for the consequences for human performance (Parasuraman & Riley, 

1997). Undisciplined automation tends to define the operators’ roles as by-products of the 

automation. Operators are expected to “take care of” whatever the system cannot handle. 

However, in the case of Patriot, little explicit attention was paid during design and subsequent 

testing to determining (1) what these residual functions were, (2) whether operators reasonably 

could be expected to perform them, (3) how operators should be trained, or (4) the impact on the 

overall system’s (hardware plus operators) decision-making reliability. 

Figure 1. Patriot vigilance logic model. 

The downstream impact of undisciplined automation was exacerbated by two additional factors: 

(1) unacknowledged system fallibilities, and (2) a “fascination with and blind faith in 

technology.” [Note: Several terms presented in quotes without reference citations are taken from 

the classified BOI reports.] A series of Patriot operational tests indicated that the system’s 
4 

automated engagement logic was subject to air target (i.e., track) misclassification problems—

system fallibilities. However, these sources of automation unreliability were not fully addressed 

during system software upgrades, nor did information about them find its way into operator 

training; battle command practices; tactics, techniques, and procedures (TTPs); or Tactical 

Standing Operating Procedures (TSOPs). System developers continued to pursue technologycentric solutions to automation reliability problems (e.g., increased use of artificial intelligence, 

non-cooperative target recognition, etc.). But the basic problem remained: The total system 

(hardware plus crew) was not sufficiently reliable in critical functional areas, most notably track 

classification and identification. Users were not informed regarding these problems, or if they 

were informed, little effective responsive action was taken. 

In the aftermath of the first Gulf War (ODS), the AMD user community (combat developer, 

training developer, operational units, etc.) acquiesced to the developmental community’s 

apparent lack of urgency regarding problems with Patriot’s track classification accuracy. 

Emboldened by Patriot’s seeming success in engaging the Iraqi SCUD threat during ODS, 

Patriot’s organizational culture emphasized “Reacting quickly, engaging early, and trusting the 

system without question.” Users were allowed to persist in their belief that the system would not 

confuse air-and non-air-breathing threats such as ARMs and TBMs. 

The cultural norm of unwarranted trust in automation was exacerbated by the AMD branch’s 

traditional training practices, which were criticized in BOI reports as emphasizing “rote drills 

versus the exercise of high-level judgment.” The Patriot user community continued to approach 

training for air battle operations in much the same manner as march order and emplacement or 

system set-up. The emphasis was on mastering routines rather than active thinking and adaptive 

problem solving. Klein and Pierce (2001) refer to the result of this practice as 

“experiosclerosis.” Crews believe they are competent and “combat ready” because they are good 

at the routines, but the routines can prove to be a straitjacket during combat. Traditional 

individual and unit evaluation practices reinforced this mistaken belief on the part of crews and 

commanders at all levels by focusing only on satisfactory performance of routine drills. The 

Army BOI investigating the OIF fratricides stated bluntly that “the system (Patriot) is too lethal 

to be placed in the hands of crews trained to such a limited standard.” 

A second detrimental factor was the branch’s traditional personnel assignment practices which 

tended to place inexperienced personnel in key crew positions in the C2 chain: the battery-level 

Engagement Control Station (ECS) and battalion-level Information and Coordination Central 

(ICC). Before the first round was fired during OIF, the stage was thus set for what Parasuraman 

and Riley (1997) refer to as “automation misuse,” specifically automation bias on the part of 

Patriot operators. Automation bias is defined as unwarranted over-reliance on automation, and 

has been demonstrated to result in failures of monitoring (vigilance problems) and accompanying 

decision biases (an absolute and unthinking trust in automation—let’s do what the machine 

recommends). Recall that these are the very concerns expressed by MG Vane in his kick-off 

discussion with the Patriot Vigilance staff. 
5 

One must be careful, however, not to lay too much blame for these shortcomings at the feet of 

the Patriot operators or the supporting battle staff. As suggested in figure 1, the roots of these 

human shortcomings can be traced back to systemic problems resulting from decisions made 

years earlier by concept developers, software engineers, procedures developers (government and 

vendor), trainers, and commanders. In one sense, the OIF Patriot operators did what they had 

been trained to do and what Patriot’s culture emphasized and reinforced. It should also be noted 

that system developers and users edged into the OIF situation by degrees. As Patriot evolved 

over its 25-year life, its dominant control mode changed, by degrees, from traditional manual 

control to supervisory control as increasing levels of automation and other technical features 

were added to cope with increasingly sophisticated threats and an increasingly complex 

operating environment. 

Hardware-wise, Patriot evolved into a very lethal system. It can be argued, however, that the 

system was not properly managed during OIF. Driven by technology and mission expansion, the 

Patriot crew’s role changed from traditional operators to supervisory controllers whose primary 

role is supervision of subordinate automatic control systems. But this role change was not 

reflected in the AMD culture, design and evaluation practices, battle management concepts, 

operational procedures, training practices, or personnel usage patterns. Moreover, system 

management issues (doctrine, battle command concepts, TTPs, TSOPs, etc.) and crewmembers’ 

ability to execute them were not addressed with the same rigor during development and 

evaluation as hardware and software capabilities. As the lessons of OIF suggest, these aspects of 

the total “system” are as important to operational effectiveness as hardware and software 

capabilities. 

HRED’s briefing to MG Vane in October 2004 described the human performance circumstances 

that contributed to the fratricides and recommended two primary actionable items to address the 

problems thus identified: 

1. Re-examine automation concepts, operator roles, and C2 relationships in AMD battle 

command systems to emphasize effective human supervisory control (HSC); and 

2. Develop more effective missile crews and C2 teams, or in the words of the Army BOI 

report “re-look the level of expertise required to operate such a lethal system on the modern 

battlefield.” 

In present usage, the term effective HSC refers to a situation in which soldiers and not the 

automated system are the ultimate decision makers in AMD firing decisions. Uncritical 

acquiescence to the automated system’s recommendations is not effective HSC. 

A month following HRED’s report to MG Vane, the Defense Science Board (DSB) (DSB, 2004) 

reinforced HRED’s conclusions with the following recommendations. Although the full DSB 

report on Patriot system performance is classified, these extracts are not. 
6 

“The Patriot system should migrate to more of a ‘man-in-the-loop’ philosophy 

versus a fully automated philosophy—providing operator awareness and control 

of engagement processes.” 

and 

“Patriot training and simulations should be upgraded to support this man-in-theloop protocol including the ability to train on confusing and complex scenarios 

that contain unbriefed surprises.” 

A summary of the DSB report on Patriot system performance is available for download on the 

DSB’s web site. 

Follow-On Work, Implementation, and Current Status 

After reviewing initial project results, the Army Training and Doctrine Command (TRADOC) 

Capability Manager for Lower Tier AMD systems (TCM-LT), requested that the Patriot 

Vigilance project continue into a second phase. The TCM specifically requested that HRED’s 

project staff expand on the material presented in Hawley, Mares, and Giammanco (2005) and 

prepare two, more-detailed reports, one concerned with design for effective human supervisory 

control and a second addressing training for the emerging class of automated AMD battle 

command systems. In the TCM’s words, the intent of these reports was to inform the AMD 

community on “what right looks like” in each of these topic areas. The results of the second 

phase of the effort were the technical reports Developing Effective Human Supervisory Control 

for Air and Missile Defense Systems (Hawley & Mares, 2006) and Training for Effective Human 

Supervisory Control of Air and Missile Defense Systems (Hawley, Mares, & Giammanco, 2006). 

Both reports contain a summary and discussion of the technical state of the art in each of the 

topic areas. In addition, supporting informational briefings were developed for use across the 

AMD community. The project staff also worked with various elements in the AMD system 

development (combat and materiel developer), training, and user communities on operationally 

defining and implementing Patriot Vigilance recommendations. Phase two formed the 

theoretical basis for what later were to be turned into actual design and training modifications. 

In the late summer of 2005 after MG Vane had left Ft. Bliss for another assignment, the project 

staff briefed his replacement, MG (then Brigadier General) Robert P. Lennox, on the status and 

results of the Patriot Vigilance project. Based on this presentation and subsequent urging from 

the TCM-LT, MG Lennox formally requested that the project be continued for at least another 

year so that the technical staff could continue to work with the AMD community on 

implementing selected results. HRED’s project staff also would participate as the MANPRINT 

evaluator during an operational test of the Post-Deployment Build 6 (PDB-6) software suite for 

the Patriot system. PDB-6 was developed to address many of the Patriot system’s operational 

deficiencies that had surfaced during OIF and were generally considered to have contributed to 

the unacceptable fratricide rate. 
7 

It also turned out that in addition to wide-ranging software fixes, the PDB-6 operational test was 

expanded to address a number of changes consistent with HRED’s first actionable item 

concerning a re-examination of automation concepts, operator roles, and C2 relationships in 

AMD battle command systems to emphasize effective HSC. The centerpiece of these changes 

was the integration of a Fire Coordination Center (FCC) into the Patriot battalion command post. 

The FCC represents an enhanced C2 entity similar in concept to the combat information center 

on Navy Aegis cruisers. If the FCC concept proved successful, it rather than the traditional ECSICC combination would become the “trigger-puller” for Patriot units. With the introduction of 

the FCC, the branch was implicitly recognizing that “two people inside a van (the ECS) 

conducting engagement operations is no longer viable.” The FCC potentially represents a 

significant step forward in addressing the SA problem that contributed to the C2 failures of OIF. 

First, however, it would have to be demonstrated (1) that the FCC provided the incremental SA 

essential for more accurate engagement decision making and (2) that the new C2 configuration 

could do so in a timely manner. Engagement decision time lines for Patriot against TBMs are 

very short—less than 10 seconds in the case of the fratricide involving the British Tornado 

during OIF. Decision cycle time is a significant issue in AMD battle command. 

From the fall of 2005 through the summer of 2006 during the New Equipment Training (NET) 

and unit train-up period for the PDB-6 test, the HRED project staff’s observations regarding the 

progress of training for the test unit sounded an alarm. PDB-6 training was not progressing 

according to plan. Training events were being completed, but individual and crew performance 

objectives were not being met. In addition, many of the training issues identified and discussed 

in Hawley, Mares, and Giammanco (2006) were re-surfacing and were not being addressed 

adequately by the NET process or follow-on collective training by the test unit. These included 

but were not limited to (1) an emphasis on training events to the exclusion of test player 

performance capabilities, (2) lack of focus on the unit’s core test mission—air battle operations, 

(3) inadequate standards, (4) inappropriate training methods, and (5) inadequate performance 

feedback—the after-action review (AAR) process. 

The project staff viewed these deficiencies as a serious problem because inadequate test player 

training would compromise the validity of test results and undermine the basis for evaluating the 

value added of PDB-6 software changes, the FCC concept, and other C2 modifications (see 

Hawley, 2007a). Even more serious was the fact that fratricides and fratricide-inducing 

conditions (e.g., dropped or improperly correlated tracks, loss of network connectivity, etc.) 

similar to those that occurred during OIF were still all-too-frequent during the test itself. Many 

of the human performance problems that had shown up during OIF were apparent again, with 

similar results. 

The Patriot and AMD C2 situation is not all gloom and doom. As a result of the Patriot 

Vigilance project and HRED’s participation in the PDB-6 LUT, beneficial changes are in the 

offing. These include (1) new concepts for effective HSC for Patriot and follow-on AMD 

systems (e.g., the FCC and a follow-on AMD integrated battle command system); (2) 
8 

performance-oriented concepts for NET and unit train-up prior to testing for Patriot 

enhancements (e.g., PDB-6.5 and -7.0); (3) modified concept evaluation and test and evaluation 

practices; and (4) a revision of institutional and unit training practices to emphasize deliberate 

practice, active thinking, and sensemaking over traditional crew drills. An overview of these 

developments is provided in Hawley (2007b). 

MANPRINT Observations and Broader Lessons 

The idea for a MANPRINT observations and lessons assessment using Patriot as an exemplar 

grew out of a conversation with the TCM-LT prior to the conclusion of the PDB-6 LUT. After 

discussing emerging results from the test and recommendations from the Patriot Vigilance 

project, the TCM said something to the effect that “We’ve been doing MANPRINT work on 

Patriot for more than 20 years. How did we miss all of the things you wrote about and that we’re 

now seeing? Certainly all of these things are not new.” These questions prompted the project 

staff to ask the obvious follow-on questions “How did we miss these high-driver issues? Or, did 

we note these things and nothing came of them?” These latter questions led to a review of the 

available Patriot-related MANPRINT assessments and test reports going back to the start of the 

formal MANPRINT initiative in the mid-1980s. Prior to MANPRINT’s formalization in 1985, 

human factors concepts were different. HSI as we understand the term today was not explicitly a 

goal in system development. Rather, the concern in those days was a rather narrow view of what 

is now termed human factors engineering (HFE): Properly engineering the interface between 

users and the hardware system, or fitting the person to the system. 

The next section presents the project team’s observations concerning the Patriot MANPRINT 

program that evolved after 1985. Readers should note that the operant word in the previous 

sentence is “evolved.” MANPRINT concepts as applied to Patriot evolved from a nearly strict 

preoccupation with HFE to include a broader concern for other nominal MANPRINT domains: 

Manpower, Personnel, Training, System Safety, Health Hazards, and Soldier Survivability. The 

project staff has tried to avoid hindsight bias and judging earlier MANPRINT activities by the 

conceptual standards of today. Again, the intent is practice improvement rather than mere 

criticism of what went on with Patriot. 

Observations on Patriot MANPRINT 

A review of MANPRINT documents and test reports going back to the mid- to late-1980s and 

into the 1990s indicated that the term evolved might actually be a misnomer when describing the 

Patriot MANPRINT program. Early assessments focused on what might be termed “1472 

issues” (control and display features, symbology concerns, etc.) and user-jury type evaluations. 

The term 1472 issues refer to the concerns addressed in MIL-STD-1472, Human Engineering. 
9 

User-jury type assessments refer to the extensive use of questionnaires and interviews during 

system evaluation. User-jury data typically are collected from a panel of recognized experts. 

There is nothing inherently wrong with the assessment methods alluded to in the previous 

paragraph. They are a part but not all of most contemporary MANPRINT assessments. 

Problems began to arise, however, when our MANPRINT methods did not evolve as the system 

evolved. Over the course of its 25-year life, Patriot evolved from a system that was quite 

literally just one-step more advanced than its predecessor, the Hawk missile system, to the 

complex system that exists today. As more and more technology and software capability was 

added to the Patriot system, the operators’ roles changed from traditional manual control to 

supervisory control, bringing with it everything that goes with that control mode. Our 

MANPRINT methods did not advance in concert with system enhancements. We continued to 

look at the system as if it were no different than a traditional manual system. The MANPRINT 

assessments performed for later system upgrades were not substantially different from those 

performed early in the system’s development. Moreover, many of the human performance 

problems that became apparent during OIF and later during the PDB-6 LUT were starting to 

show up in earlier test results. But the MANPRINT community did not raise a red flag, and 

potential show-stoppers either were ignored or allowed to be downplayed. The most common 

form of downplaying was to dismiss human performance problems as isolated training issues 

that could be fixed later during institutional or unit training. It should be noted that most of the 

human performance problems thus identified and categorized as training problems were never 

actually addressed. 

The bottom line with respect to the 20-year history of Patriot’s MANPRINT program is that we 

as a community missed the elephants in the middle of the room. These were (1) an eroding 

potential for effective human supervisory control, and (2) the need for increased levels of 

operator expertise—not simply additional training of the traditional kind, but training with a 

focus on developing operator expertise. It should be noted that the DSB’s post-OIF review of 

Patriot system performance focused quickly on these issues, as did the Patriot Vigilance 

assessment requested by MG Vane. Furthermore, similar problems were apparent in related 

domains going back to near the time the Patriot system was initially fielded. These domains 

include nuclear power operations and the 1979 Three Mile Island incident; Navy anti-air 

operations and the 1988 USS Vincennes-Iranian airbus incident; and the FAA’s $3.5 billion 

Advanced Automation System (AAS) “meltdown” in the 1990s. [See Bar-Yam (1997) for a 

discussion of the events leading to the “safety veto” of the AAS.] All of these incidents were 

reported and discussed widely in the human factors literature, yet no mention of them is found in 

Patriot MANPRINT documents—despite the fact that Patriot’s operational concept has many 

features in common with these systems. 

In all fairness, the MANPRINT community was not alone in failing to emphasize the potentially 

serious impact of the control and training problems on Patriot system performance. We are part 

of a team, and the other members of the system development team—the system’s prime 
10 

contractor, the materiel developer, the combat developer, the TCM, and the Army test and 

evaluation (T&E) community—all overlooked or downplayed these issues. These observations 

regarding Patriot mostly would be moot were it not for the fact that this insensitivity to the 

downside potential associated with human performance issues continues to be prevalent. It was 

observed during the Patriot Vigilance project; it was apparent during the PDB-6 LUT and its 

aftermath; and it continues to be a problem in AMD systems other than Patriot. The natural 

tendency of most of the other players is to downplay human performance issues as isolated 

incidents or as inconvenient, troublesome problems with no clean (i.e., technology-oriented) 

solution. 

In the case of Patriot with PDB-6, a strong case was made regarding the potential seriousness of 

the control and training issues. Based on these conclusions and supporting material, headway 

toward developing and implementing a training fix also was made in the Air Defense Artillery 

(ADA) School (see Hawley, Mares, Fallin, & Wallet, 2007). However, the request for a PDB-6 

conditional materiel release contained the usual “all is well with the system” comments and 

mostly ignored the human performance problems and training inadequacies that showed up 

clearly in test results and were similar to those that were judged by the Army BOI to have 

contributed to the Patriot fratricides during OIF. 

Broader Lessons from Patriot 

Beyond the specific observations stemming from our review of Patriot MANPRINT materials 

going back some 20 years, there are a series of lessons from the Patriot experience that 

generalize to present and future system development efforts other than Patriot. Several of the 

more prominent of these broader lessons from Patriot are now briefly discussed. 

Going-In Concepts Really Matter. The first general lesson to be taken away from the Patriot 

MANPRINT experience is that going-in concepts really matter. If a flawed system concept is 

allowed to proceed into development, it may prove impossible to modify that concept later to 

obtain necessary levels of performance. Faulty going-in concepts pursued into development can 

create a cul-de-sac from which there might not be a graceful exit. A good example of this 

situation is the safety veto of the FAA’s AAS for air traffic control in the late 1990s after more 

than $3.5 billion had been spent on its development. Judging from available documentation on 

the system’s demise, it was undone by a combination of controller reluctance to accept the 

system coupled with operational test results suggesting that the system might not be reliable 

enough to deploy as an operational air traffic control system (Bar-Yam, 1997). 

In the case of Patriot, the system’s going-in concept flew in the face of several well-known (even 

for that time) dilemmas concerning automated systems. The first of these has been termed the 

brittleness problem of automata (Woods, 2002). The brittleness dilemma refers to breakdowns 

in handling atypical or off-nominal situations—similar to those that occurred during OIF. 

Patriot’s developers disregarded the brittleness issue and proceeded under the assumption that 

automation would eliminate many of the human performance problems associated with earlier 
11 

AMD systems and also reduce training requirements. Both assumptions proved to be unfounded. 

As noted previously, automation applied to real-time C2 changes the operators’ roles from 

traditional operators to supervisory controllers. And that subtle role shift brings with it what has 

been termed the Catch-22 of human supervisory control (Reason, 1990). Consequently, Patriot 

finds itself in somewhat the same situation as the FAA’s AAS. The system’s developers have 

committed to a system concept that demonstrates patterns of performance unreliability that might 

prove difficult and expensive to correct. More attention to proper operator-system function 

allocation and the potential downside of automation as the system evolved might have resulted in 

a product that exhibited fewer of these problems. 

Training Issues Really Matter. The second general lesson emerging from the Patriot 

MANPRINT experience is that training issues really matter. As noted previously, while the 

Patriot system evolved over its 25-year life, system enhancements (new hardware features, new 

software drops like PDB-6, etc.) the system became more complex. Increased complexity 

coupled with the operator role shift from traditional operator to supervisory controller added to 

the system’s training burden. Yet this increase in training requirements was not matched by 

corresponding changes in institutional or unit training. Hawley, Mares, and Giammanco (2006) 

and Hawley and Mares (2007) argue that changes in system complexity must drive quantitative 

and qualitative changes in the associated training program. Training must more rigorous and 

qualitatively different in the sense that it emphasizes the development of mental models and 

knowledge structures and not simple habit transfer, what the Army BOI referred to as “rote 

drills.” This view is supported by other observers such as the Defense Science Board. In the first 

of two reports on training for future conflicts, the DSB (2001) cautioned that an increasing risk 

exists that training failures will negate hardware promise. Their 2003 follow-on report further 

remarked that the future will require that more of our people do new and more complicated 

things, and “meeting this challenge amounts to a qualitative change in the demands placed on our 

people that cannot be supported by traditional training practices” (DSB, 2003, p. 38). 

These observations regarding the human performance impact of system complexity are not 

particularly new or unique to AMD. For example, in her classic work In the Age of the Smart 

Machine, Shoshanna Zuboff (1988) remarks that computer-mediated work like that found in 

many new systems brings with it an increase in “intellective skill requirements.” Commenting 

on what they had observed during Operation Desert Storm in the early 1990s, Cordesman and 

Wagner (1996, p. 25) note that technical advances are used to demand more from operators, and 

meeting these demands often requires “exceptional human expertise.” More recently, an early 

operational assessment of DARPA’s Command Post of the Future (CPOF) currently being used 

by the Army in Iraq remarks that in order to take advantage of the features provided by this new 

capability there is a “need for a soldier with a wider ‘intellectual bandwidth,’ where management 

and assimilation of information from many sources is a necessity” (Center for Army Lessons 

Learned, 2005, p. vii). In a case study assessment of the impact of net-centric operations using 

the Stryker Brigade Combat Team (BCT) as an exemplar, Gonzales, Johnson, McEver, Leedom, 
12 

Kingston, and Tseng (2005, p. 35) concluded that “training is more important than ever in the 

Stryker brigade and other digitized units because the networking and battle command systems 

employed are more complex than those used in analog-equipped brigades. If soldiers and 

commanders are not adequately trained on the NCW [network-centric warfare] systems and are 

not proficient in their use in stressful battlefield conditions, then these NCW systems can be a 

hindrance rather than a help in combat.” Finally, in a post-test briefing to the Vice Chief of Staff 

of the Army concerning the Army Battle Command and Enablers (ABCE) system of systems, the 

Army Test and Evaluation Command (ATEC) concluded “There is no indication that units can 

dedicate the time, resources, or personnel to adequately train on the digital C4I systems and 

allow the unit to adequately comprehend the system’s capabilities, much less exploit these 

systems as a force multiplier” (ATEC, 2006). 

So what does all of this have to do with MANPRINT? The answer is captured in the two DSB 

comments cited above. First, risk exists that training failures will negate hardware promise, and 

second, the notion that new systems may impose a training burden that cannot be supported by 

traditional training practices. When viewed from these perspectives, training issues are indeed a 

system development problem. For example, in the case of Patriot with PDB-6, the Army 

Evaluation Center’s System Assessment Report concluded that operator performance 

requirements for PDB-6 exceeded the current Army training standard. This conclusion is 

significant with respect to downstream system performance. The implication is that if training 

practices remain unchanged, the Patriot system with PDB-6 has become too complex for its 

operators to employ effectively. The DSB’s warning that training failure might negate hardware 

promise will have become a reality. 

MANPRINT analysts and assessors must forcefully advocate quantitative and qualitative training 

changes when assessment or test results support a conclusion like that put forward above for 

ABCE or that emerging from the PDB-6 LUT. The operational demonstration of new training 

capabilities and methods for AMD unit training reported in Hawley, Mares, Fallin, and Wallet 

(2007) provides an example of the kinds of positive change that can result from effective 

advocacy by the MANPRINT community. Admittedly, the AMD operational demonstration is a 

tentative first step in the direction of implementing the changes that will be necessary to bring 

AMD unit training into line with system requirements, but it is a first step. 

Testing Must be More Comprehensive and Rigorous. The third general observation emerging 

from the review of MANPRINT practices with the Patriot system is that operational testing must 

be more comprehensive and rigorous, particularly where the impact of human performance on 

system performance is concerned. During the PDB-6 LUT, for example, the MANPRINT 

support team encountered a strong bias on the part of testers to look at the system primarily from 

a hardware and software perspective. The impact of operator performance on system 

performance was regarded as a secondary issue—one that could be taken care of mostly with 

user-jury type questionnaires. User-jury type questionnaires are potentially useful in system 

evaluation, but they should not be the sole source of data regarding the operators’ impact on 
13 

system performance. Operators who are not familiar with new system functions and 

requirements often “don’t know what they don’t know,” thus their subjective assessments may 

lack critical information regarding the system’s performance potential. 

Looking back on the situation with the PDB-6 LUT and other tests, the fact that MANPRINT 

(i.e., Soldier performance) is classified as a supportability issue and not a system performance 

issue during test design and conduct undoubtedly contributes to the hardware-software bias noted 

above. Dekker (2002) argues that complex systems like Patriot and others coming into the Army 

inventory require an “overwhelming human contribution” for their effective operation. He goes 

on to state that “people are the only ones who can hold together the patchwork of technologies in 

their world; the only ones who can make it work in actual practice” (p. 103). Given this view 

backed up with the observations reported in the previous subsection, it is hard to justify the 

notion that Soldier performance is a secondary issue during T&E. Soldier impact on system 

performance cannot continue to be viewed as a secondary issue. Continuing to do so opens the 

door to total system reliability problems like those that contributed to the Patriot fratricides 

during OIF. For Patriot, it was assumed that operators would be able to compensate for known 

hardware-software reliability problems in the area of track classification accuracy, but this 

assumption was never verified empirically during testing. Later events during OIF indicated that 

the assumption was unwarranted. 

Meaningfully putting the human component into the testing equation for human-machine 

systems will require a significant change in the way the Army prepares test players to participate 

in operational tests (see Hawley, 2007a). In present usage, the term “test players” refers to the 

individuals, crews, or units participating in a test event. Both the Defense Science Board and the 

Government Accountability Office (GAO) have criticized the Department of Defense in general, 

and the Army in particular, for not properly preparing test players to participate in operational 

tests (e.g., DSB, 1999, 2003; GAO, 2000). The DSB has stated bluntly that the “Army continues 

to field new equipment without adequate training” (DSB, 2003, p. 44). Training deficiencies for 

new systems are a problem for receiving organizations, but they can be remedied with use over 

time. However, for T&E, pre-test training deficiencies can have more serious consequences—

because T&E frequently is a one-shot affair. The GAO notes that testing is the primary means 

used to gauge the progress being made when a concept is translated into an actual product for 

users; it is the basis for determining fitness for use before the product is provided to users (GAO, 

2000). 

Failures in the training domain would not be particularly of interest to testers were it not for the 

serious downstream impact of inadequate training on test integrity. The GAO and DSB both 

caution, for example, that inadequate training of test participants seriously undermines the 

validity of test results. Following a review of a representative sample of operational tests, 

Hawley and Frederickson (1990) concluded that inadequate test player preparation was one of 

the most frequent reasons for test failure. Failure, in present usage, refers to an inability to 

cleanly address test issues using test data. These authors cautioned that if test player capabilities 
14 

are uncertain, test results are likely to be compromised. The most common form of compromise 

is confounding between test outcomes and pre-test proficiency levels. It is not possible to state 

unambiguously that test outcomes reflect system capabilities and features, test player 

proficiency, or some combination of the two. 

In GAO’s terminology, poor test player preparation almost inevitably results in a “hollow test” 

(GAO, 2000, p. 5). A hollow test is one that satisfies the requirement to hold a testing event, but 

does not advance system-related knowledge. Consequently, test results provide little insight into 

the system’s performance potential and cannot be used to debug or improve the system. Test 

planners often attempt to compensate for an emerging hollow test by restricting or scripting test 

events as a workaround for test player performance deficiencies. A great deal of this “rescripting” was observed during the PDB-6 LUT. But such tactics do not really address the faulty 

train-up issue. Regardless of the camouflage used, the test is still hollow. Clever statistics and 

sophisticated post-test analyses cannot compensate for an intrinsically flawed test. For test 

results to be meaningful, test planners must verify that test players have been trained to the 

competency levels required by test events. 

Lessons Must be Learned. Former Chief of Staff of the Army, General Eric Shinseki, is 

credited with the observation that lessons are not learned until a resulting change occurs. If 

change does not occur, the lessons are merely “observed.” The same might be said of 

MANPRINT lessons and results. In the case of Patriot and PDB-6, the MANPRINT team 

encountered enormous pressure to pass the system on and support a conditional materiel release 

for PDB-6 with the promise that problems emerging from the test would be fixed later. This 

occurred in spite of the fact that similar problems showing up in earlier tests or as lessons from 

combat operations during OIF had not yet been acted upon. Hard and unpleasant facts must be 

faced and acted upon explicitly. In the absence of action, problems typically do not get better 

with time; and ignoring problems does not make them go away. 

One interesting observation emerging from the PDB-6 test concerns the handling of feedback 

from operational tests, specifically feedback concerning training, doctrine, or procedural 

inadequacies impacting overall system performance. Materiel developers (e.g., Program 

Managers) routinely get feedback from T&E and are accustomed to addressing system 

inadequacies based on test results. These “wickets” in the system development process are 

routinely used and well developed. Resource or other constraints may prevent the timely 

implementation of essential materiel fixes, but the mechanism for their acknowledgement is well 

known. 

Such is not the case for training, doctrine, or procedural inadequacies—the stuff of many 

MANPRINT assessments. In the case of training, for example, feedback regarding training 

inadequacies such as those found in the PDB-6 LUT would have to go to TRADOC for 

institutional training fixes and to various Major Commands (e.g., Forces Command—

FORSCOM) for unit training modifications. The Army Independent Evaluator judged that it had 
15 

no authority to task TRADOC or FORSCOM for any mitigation plan addressing training fixes. 

These commands could be apprised of potential training problems, but no formal 

acknowledgement of the problem or plan for its remedy is required in response. (Apparently, 

only the Army G3 has the authority to task the MACOMS.) As noted, such is not the case with 

respect to materiel deficiencies. PMs are routinely required to provide so-called “get well” plans 

in response to system deficiencies (per AR 700-142). The get well plan is a condition for 

materiel release, and the progress of its implementation is monitored by the Independent 

Evaluator. There is an old adage that what gets measured gets done. Until doctrine, tactics, 

procedures, training methods, training support (training devices), and other parts of the total 

system package (e.g., the DOTLM-PF components, minus the M) are put on the same plane as 

materiel, only part of the feedback from T&E has any potential for long-term beneficial impact. 

MANPRINT Going Forward 

The previous section noted that one of the primary motivators for this report was the TCM-LT’s 

musing concerning how during 20 years of MANPRINT work on Patriot we missed many of the 

human performance and training problems that contributed to the fratricides during OIF. It 

cannot be stated with any certainty that the issues of effective human supervisory control and 

inadequate training were completely missed. However, it is certain based on the review of 

available MANPRINT assessments and test reports that a discussion of these issues never made 

it into those documents. Hence, in that sense, the issues were missed. In retrospect, one must 

conclude that the TCM’s implied criticism of the Patriot MANPRINT program was justified. 

This conclusion raises the broader question of why these issues were missed. Part of the answer 

lies in the previous observation that our MANPRINT methods did not evolve along with the 

system. An eroding potential for effective HSC and the concomitant problem of inadequate 

training came in on the coattails, so to speak, of technical enhancements to the Patriot system 

over time. However, we kept doing what we had always done, but that something became 

increasingly inadequate. A contributor to this problem lies in the way the MANPRINT program 

was set up originally. Back in the 1985-86 timeframe when the program was codified in AR 

602-2, the breakout of the MANPRINT program into semi-independent domains—HFE, 

Manpower, Personnel, Training, Soldier Survivability, Health Hazards, and Soldier 

Survivability—preserved the stakeholder “rice bowls” that existed at that time. However, the 

all-important issue of domain integration was mostly given lip service. Platitudes concerning the 

importance of domain integration were put into the Regulation and supporting documents, but no 

mechanism was put in place to bring it about. The lesson from Patriot is that MANPRINT 

domain integration involves more than a simple concatenation of assessments from the 

component domains. An effective MANPRINT program is more than simple the sum of its 
16 

domain parts. In Patriot, operator performance issues that emerged at the intersection of domains 

such as automation, supervisory control, and their impact on operator performance demands and 

training requirements simply fell through the cracks. 

A second observation regarding MANPRINT going forward is the need for a broader analytical 

and programmatic perspective. As a combined work program, the Patriot Vigilance project, the 

PDB-6 LUT, and subsequent efforts at implementation such as the training operational 

demonstration, support and illustrate this view. Vicente (2006) and others advocate what might 

be termed a sociotechnical systems (STS) approach to MANPRINT and HSI. An STS approach 

to MANPRINT begins with the premise that people (the social part of the system) and 

technology must be viewed as an integrated system, and that both the social and technical parts 

of the system must be optimized equally and as a unit. Nothing of this sort happened with 

Patriot as it evolved, and nothing of the sort is happening with follow-on AMD systems. In spite 

of General Max Thurman’s admonitions about the dangers of manning systems versus equipping 

people and organizations to accomplish a mission, we still mostly build systems and then put 

people into them—and then put those systems into existing organizations. (Max Thurman is 

generally considered to be the “Godfather” of the Army’s MANPRINT initiative.) It should be 

noted that the structure of the Army’s system development process almost guarantees that things 

will work this way. 

Vicente’s (2006) notion of what comprises an STS approach to MANPRINT and system 

development is shown in figure 2. All of the steps on the sociotechnical ladder are necessary for 

the initiative to succeed and contribute to system and organizational effectiveness. Based on the 

Patriot experience, figure 2 also suggests that issues become more problematic as one ascends 

the ladder, but all the steps are necessary—all the way to the top. In the case of Patriot, for 

example, the project staff is discovering that issues on the top two steps—Organization and 

Political—seriously impact solutions to issues at lower levels. Implementing the training fixes 

indicated by the OIF BOIs, the Patriot Vigilance project, and follow-on work is impeded by 

Patriot and AMD’s organizational culture (the way we routinely do things) and political issues 

(e.g., TRADOC and MACOM policies, budgetary pots, regulations at various levels, stakeholder 

rice bowls, etc.). As MANPRINT practitioners, we cannot change culture or affect politics 

directly, but we must apprise decision makers of the potential impact of these factors on system 

and organizational effectiveness. 
17 

Figure 2. Ladder of sociotechnical concerns. 

Physical: 

• Size 

• Shape 

• Location

• Weight 

• Color 

• Material 

Cognitive: 

• Information 

 content/structure

• Cause/effect 

 relations 

Crew/team: 

• Authority 

• Communication

 patterns 

• Roles 

Organizational: 

• Culture 

• Rewards 

• Structure and

 staffing 

Political: 

• Policy 

• Budget 

• Regulations 

(Operability) 

Source: Adapted from Vicente (2006)

Moderately Hard 

Very Hard/ 

Impossible 

Challenge 
18 

References 

Army Test and Evaluation Command. [Army Battle Command and Enablers]. Unpublished test 

summary briefing, 2006. 

Bar-Yam, Y. Dynamics of complex systems; Reading, MA: Perseus, 1997. 

Center for Army Lessons Learned. Command post of the future (CPOF) early operational 

assessment (Initial Impressions Report 05-21). Ft. Leavenworth, KS: TRADOC System 

Manager – Battle Command, 2005. 

Cordesman, A. H.; Wagner, A. R. The lessons of modern war—Vol.4: The gulf war; Boulder, 

CO: Westview Press, 1996. 

Defense Science Board. Test and evaluation. (Final Report of the DSB Task Force on Test and 

Evaluation). Washington, DC: Office of the Undersecretary of Defense for Acquisition, 

Technology, and Logistics, 1999. 

Defense Science Board. Training superiority and training surprise. (Final Report of the DSB 

Task Force on Training Superiority and Training Surprise). Washington, DC: Office of the 

Undersecretary of Defense for Acquisition, Technology, and Logistics, 2001. 

Defense Science Board. Training for future conflicts. (Final Report of the DSB Task Force on 

Training for Future Conflicts). Washington, DC: Office of the Undersecretary of Defense 

for Acquisition, Technology, and Logistics, 2003. 

Defense Science Board. Patriot system performance. (Final Report of the DSB Task Force on 

Patriot System Performance). Washington, DC: Office of the Undersecretary of Defense for 

Acquisition, Technology, and Logistics, 2004. 

Dekker, S. The field guide to human error investigations; Burlington, VT: Ashgate, 2002. 

Endsley, M. R.; Bolte, B.; Jones, D. G. Designing for situation awareness: An approach to usercentered design; New York: Taylor & Francis, 2003. 

Government Accountability Office. Best practices: A more constructive test approach is key to 

better weapon system outcomes; (GAO/NSAID-00-199); Washington, DC: U.S. 

Government Accountability Office, 2000. 

Hawley, J. K. Training and testing: A complex and uneasy relationship. ITEA Journal of Test 

and Evaluation 2007a, 27 (4), 34-40, December 2006/January 2007. 

Hawley, J. K. Patriot after Operation Iraqi Freedom: A case study in constructive change; 

Draft ARL Special Report; U.S. Army Research Laboratory: Adelphi, MD, 2007b. 
19 

Hawley, J. K.; Frederickson, E.W. MANPRINT guidelines for soldier performance evaluation in 

early user test and experimentation. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences, 1990. 

Hawley, J. K.; Mares, A. L. Developing effective human supervisory control for air and missile 

defense systems; ARL TR 3742; U.S. Army Research Laboratory: Adelphi, MD, 2006. 

Hawley, J. K.; Mares, A. L. Developing effective adaptive missile crews and command and 

control teams for air and missile defense systems; ARL SR 149; U.S. Army Research 

Laboratory: Adelphi, MD, 2007. 

Hawley, J. K.; Mares, A. L.; Fallin, J. I.; Wallet, C. Reconfigurable tactical operations 

simulator (RTOS) operational demonstration in 5-52 air defense artillery; Draft ARL SR; 

U.S. Army Research Laboratory: Adelphi, MD, 2007. 

Hawley, J. K.; Mares, A. L.; Giammanco, C. A. The human side of automation: Lessons for air 

defense command and control; ARL TR 3468; U.S. Army Research Laboratory: Adelphi, 

MD, 2005. 

Hawley, J. K.; Mares, A. L.; Giammanco, C. A. Training for effective human supervisory 

control of air and missile defense systems; ARL TR 3765; U.S. Army Research Laboratory: 

Adelphi, MD, 2006. 

Klein, G.; Pierce, L. Adaptive teams. In Proceedings of the 6th International Command and 

Control Research and Technology Symposium, Annapolis, MD, 2001. 

Parasuraman, R.; Riley, V. Humans and automation: Use, misuse, disuse, abuse. Human 

Factors 1997, 39 (2), 230-252. 

Reason, J. Human error; Cambridge: Cambridge University Press, 1990. 

Schein, E. H. Organizational culture and leadership; San Francisco: Jossey-Bass, 2004. 

Vicente, K. The human factor: Revolutionizing the way people live with technology; New York: 

Routledge, 2006. 

Woods, D. D. Steering the reverberations of technology change on fields of practice: Laws that 

govern cognitive work. Proceedings of the 24th Annual Meeting of the Cognitive Science 

Society (Plenary Session), 2002. 

Yin, R. K. Case study research: Design and methods (Third Edition); Thousand Oaks, CA: 

Sage, 2003. 

Zuboff, S. In the age of the smart machine: The future of work and power; New York: Basic 

Books, 1988. 
20 

Distribution List

ADMNSTR 

DEFNS TECHL INFO CTR 

ATTN DTIC-OCP (ELECTRONIC COPY) 

8725 JOHN J KINGMAN RD STE 0944 

FT BELVOIR VA 22060-6218 

DARPA 

ATTN IXO S WELBY 

3701 N FAIRFAX DR 

ARLINGTON VA 22203-1714 

OFC OF THE SECY OF DEFNS 

ATTN ODDRE (R&AT) 

THE PENTAGON 

WASHINGTON DC 20301-3080 

ARL HRED AMEDD FLD ELMT 

ATTN AMSRD-ARL-HR-MM V RICE-BERG 

BLDG 4011 RM 217 1750 GREELEY RD 

FT SAM HOUSTON TX 78234-5094 

ARL-HRED AMCOM FLD ELMT 

ATTN AMSRD-ARL-HR-MO J MINNINGER 

BLDG 5400 RM C-242 

REDSTONE ARSENAL AL 35898-7290 

US ARMY RSRCH DEV AND ENGRG CMND 

ARMAMENT RSRCH DEV AND ENGRG CTR 

ARMAMENT ENGRG AND TECHNLGY CTR 

ATTN AMSRD-AAR-AEF-T J MATTS 

BLDG 305 

ABERDEEN PROVING GROUND MD 21005-5001 

ARMY G1 

ATTN DAPE-MR B KNAPP 

ARMY G1 MANPRINT DAPE MR 

300 ARMY PENTAGON RM 2C489 

WASHINGTON DC 20310-0300 

ARMY RSCH LAB-HRED JFCOM JOINT 

EXPERIMENTSATION J9 JOINT FUTURES LAB 

ATTN AMSRD-ARL-HR-MJK J HANSBERGER 

115 LAKEVIEW PARKWAY STE B 

SUFFOLK VA 23435 

ARMY RSRCH LAB-HRED 

ATTN AMSRD-ARL-HR-MU M SINGAPORE 

6501 E 11 MILE RD MS 284 BLDG 200A 2ND FL 

RM 2104 

WARREN MI 48397-5000 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MQ M R FLETCHER 

AMSRD-NSC-WS-E BLDG 3 RM 343 

NATICK MA 01760-5020 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-ML J MARTIN 

MYER CENTER RM 2D311 

FT MONMOUTH NJ 07703-5601 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MG R SPINE 

BUILDING 333 

PICATINNY ARSENAL NJ 07806-5000 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MK J REINHART 

10125 KINGMAN RD 

FT BELVOIR VA 22060-5828 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-M M STRUB 

6359 WALKER LANE SUITE 100 

ALEXANDRIA VA 22310 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MN R SPENCER 

DCSFDI HF 

HQ USASOC BLDG E2929 

FT BRAGG NC 28310-5000 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MW E REDDEN 

BLDG 4 RM 332 

FT BENNING GA 31905-5400 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MT J CHEN 

12423 RESEARCH PARKWAY 

ORLANDO FL 32826-3276 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MD T COOK 

BLDG 5400 RM C242 

REDSTONE ARSENAL AL 35898-7290 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MH C BURNS 

BLDG 1467B RM 336 THIRD AVE 

FT KNOX KY 40121 
21 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MP D UNGVARSKY 

BATTLE CMD BATTLE LAB 

415 SHERMAN AVE UNIT 3 

FT LEAVENWORTH KS 66027-2326 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MV HQ USAOTC 

S MIDDLEBROOKS 

91012 STATION AVE RM 111 

FT HOOD TX 76544-5073 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MY M BARNES 

2520 HEALY AVE STE 1172 BLDG 51005 

FT HUACHUCA AZ 85613-7069 

ARMY RSCH LABORATORY - HRED 

AVNC FIELD ELEMENT 

ATTN AMSRD-ARL-HR-MJ D DURBIN 

BLDG 4506 (DCD) RM 107 

FT RUCKER AL 36362-5000 

US ARMY TRADOC 

BATTLE LAB INTEGRATION & TECHL 

DIRCTRT 

ATTN ATCD-B 

10 WHISTLER LANE 

FT MONROE VA 23651-5850 

COMMANDANT USAADASCH 

ATTN AMSRD-ARL-HR-ME A MARES (5 

COPIES) 

ATTN ATSA-CD 

5800 CARTER RD 

FT BLISS TX 79916-3802 

DIRECTOR 

DIRECTORATE OF COMBAT DEVELOPMENTS 

ATTN COL H L COHEN 

5800 CARTER RD 

FT BLISS TX 79916-7001 

DIRECTOR 

DIRECTORATE OF TRAINING, DOCTRINE, & 

LEADER DEVELOPMENT 

ATTN COL R K CARL 

2 SHERIDAN RD, BLDG 2 

FT BLISS TX 79916-7001 

OFFICE, CHIEF OF AIR DEFNS ARTILLERY 

ATTN LTC J H JENKINS III 

2 SHERIDAN RD BLDG 2 

FT BLISS TX 79916-7001 

PM TIMS, PROFILER (MMS-P) AN/TMQ-52 

ATTN B GRIFFIES 

BUILDING 563 

FT MONMOUTH NJ 07703 

SMC/GPA 

2420 VELA WAY STE 1866 

EL SEGUNDO CA 90245-4659 

TRADOC CAPABILITY MANAGER-LOWER 

TIER 

ATTN COL R L DELGADO 

BLDG 12, PERSHING RD 

FT BLISS TX 79916-7001 

COMMANDING GENERAL 

US ARMY AIR DEFNS ARTILLERY CTR AND 

FT BLISS 

ATTN MG R P LENNOX 

BLDG 2 SHERIDAN RD 

FT BLISS TX 79916-7001 

US ARMY INFO SYS ENGRG CMND 

ATTN AMSEL-IE-TD F JENIA 

FT HUACHUCA AZ 85613-5300 

COMMANDER 

US ARMY RDECOM 

ATTN AMSRD-AMR W C MCCORKLE 

5400 FOWLER RD 

REDSTONE ARSENAL AL 35898-5000 

US ARMY RSCH LABORATORY 

ATTN AMSRD-ARL-CI-OK-TP S FOPPIANO 

BLDG 459 

ABERDEEN PROVING GROUND MD 21005 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-CI-OK-TP TECHL LIB T 

LANDFRIED 

BLDG 4600 

ABERDEEN PROVING GROUND MD 21005-5066 

US GOVERNMENT PRINT OFF 

DEPOSITORY RECEIVING SECTION 

ATTN MAIL STOP IDAD J TATE 

732 NORTH CAPITOL ST., NW 

WASHINGTON DC 20402 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-HR-S L PIERCE 

ATTN AMSRD-ARL-HR-SE D HEADLEY 

ATTN AMSRD-ARL-HR-SE K COSENZO 

BLDG 459 

ABERDEEN PROVING GROUND MD 21005 

US ARMY RSRCH LAB 

ATTN AMSRL-ARL-HR F PARAGALLO 

BLDG 459 

ABERDEEN PROVING GROUND MD 21005-5066 
22 

DIRECTOR 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-RO-EV W D BACH 

PO BOX 12211 

RESEARCH TRIANGLE PARK NC 27709 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-CI-OK-T TECHL PUB (2 

COPIES) 

ATTN AMSRD-ARL-CI-OK-TL TECHL LIB (2 

COPIES) 

ATTN AMSRD-ARL-D J M MILLER 

ATTN IMNE-ALC-IMS MAIL & RECORDS 

MGMT 

ADELPHI MD 20783-1197