Reconfigurable Tactical Operations Simulator (RTOS) 

Operational Demonstration in 5-52 Air Defense Artillery 

by John K. Hawley, Anna L. Mares, John I. Fallin, and Chad Wallet 

ARL-SR-0156 August 2007 

Approved for public release; distribution unlimited. 
NOTICES 

Disclaimers 

The findings in this report are not to be construed as an official Department of the Army position 

unless so designated by other authorized documents. 

Citation of manufacturer’s or trade names does not constitute an official endorsement or 

approval of the use thereof. 

Destroy this report when it is no longer needed. Do not return it to the originator. 
Army Research Laboratory 

Adelphi, MD 20783-1197 

ARL-SR-0156 August 2007 

Reconfigurable Tactical Operations Simulator (RTOS) 

Operational Demonstration in 5-52 Air Defense Artillery 

John K. Hawley and Anna L. Mares 

Human Research and Engineering Directorate, ARL 

John I. Fallin and Chad Wallet 

U.S. Army Air Defense Artillery School 

Directorate of Training, Doctrine, and Leader Development 

Approved for public release; distribution unlimited. 
ii 

REPORT DOCUMENTATION PAGE Form Approved 

OMB No. 0704-0188

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the 

data needed, and completing and reviewing the collection information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the 

burden, to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302. 

Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently 

valid OMB control number. 

PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS. 

1. REPORT DATE (DD-MM-YYYY)

August 2007 

2. REPORT TYPE 3. DATES COVERED (From - To) 

5a. CONTRACT NUMBER 

5b. GRANT NUMBER 

4. TITLE AND SUBTITLE 

Reconfigurable Tactical Operations Simulator (RTOS) Operational 

Demonstration in 5-52 Air Defense Artillery 

5c. PROGRAM ELEMENT NUMBER 

5d. PROJECT NUMBER 

5e. TASK NUMBER 

6. AUTHOR(S) 

John K. Hawley, Anna L. Mares, John I. Fallin, and Chad Wallet 

5f. WORK UNIT NUMBER 

7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 

U.S. Army Research Laboratory 

ATTN: AMSRD-ARL-HR-ME 

2800 Powder Mill Road 

Adelphi, MD 20783-1197

8. PERFORMING ORGANIZATION 

 REPORT NUMBER 

ARL-SR-0156

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR'S ACRONYM(S) 

11. SPONSOR/MONITOR'S REPORT 

 NUMBER(S) 

12. DISTRIBUTION/AVAILABILITY STATEMENT 

Approved for public release; distribution unlimited. 

13. SUPPLEMENTARY NOTES 

14. ABSTRACT 

During the combat operations phase of Operation Iraqi Freedom (OIF), Patriot air and missile defense units were involved in 

two fratricide incidents involving coalition aircraft. Patriot’s unacceptable fratricide rate during OIF (18% of engagements) 

prompted the commanding general of the Army Air and Missile Defense Center to request a human-performance-oriented 

assessment of the fratricide incidents to complement the official Army board if inquiry investigation. This report summarizes 

the results and recommendations from that assessment. Recommendations for a solution to the fratricide problem involved 

both command and control and training modifications. The paper’s primary focus is the demonstration and evaluation of 

modified training practices in an operational Patriot unit, 5-52 Air Defense Artillery (ADA). Specific aspects of these 

modified training practices included (1) the use of a less-than-full fidelity training device to supplement the unit’s organic 

embedded training capability (the Reconfigurable Tactical Operations Simulator, or RTOS) coupled with (2) a performanceoriented instructional strategy focused on deliberate practice. The conduct of the operational demonstration in 5-52 ADA is 

described and results are reported. Results obtained from the demonstration included (1) an RTOS utility assessment, (2) preversus post-training results for trial instructional modules, and (3) participant comments. Demonstration results are discussed

in terms of the literature on simulation and training effectiveness. Recommendations for extending the scope of the 

demonstration to include a more intensive study of training effectiveness and transfer of learning also are provided. 

15. SUBJECT TERMS

16. SECURITY CLASSIFICATION OF:

19a. NAME OF RESPONSIBLE PERSON

John K. Hawley 

a. REPORT 

Unclassified 

b. ABSTRACT 

Unclassified 

c. THIS PAGE 

Unclassified 

17. LIMITATION 

OF ABSTRACT 

SAR 

18. NUMBER 

OF PAGES 

48 19b. TELEPHONE NUMBER (Include area code) 

(915) 568-2896 

Standard Form 298 (Rev. 8/98) 

Prescribed by ANSI Std. Z39.18 
iii 

Contents 

List of Figures v

List of Tables v

1. Introduction 1

1.1 Background .....................................................................................................................1 

1.2 The Reconfigurable Tactical Operations simulator (RTOS) Operational Demonstration3 

1.3 Demonstration Objectives ...............................................................................................5 

2. Method 6

2.1 Participants ......................................................................................................................6 

2.2 RTOS Layout ..................................................................................................................6 

2.3 Air Battle Operations Tasks ............................................................................................6 

2.4 Training Method and Instructional Approach.................................................................8 

2.5 Procedure.........................................................................................................................9 

3. Results 12

3.1 Utility Evaluation ..........................................................................................................12 

3.2 Pre- Post-Test Results ...................................................................................................14

3.3 Buy-In Results...............................................................................................................14 

3.4 Open-Ended Comments ................................................................................................15 

3.5 Hotwash Comments ......................................................................................................16 

4. Discussion 17

5. References 21

Appendix A. RTOS Patriot PDB-5 Tactical System Representation 23

Appendix B. ARM Classification Knowledge Test 29

Appendix C. Utility Assessment Survey 31
iv 

Appendix D. RTOS Demo Questionnaire 33

Appendix E. Open-Ended Survey Responses 35

Distribution List 37
v 

List of Figures 

Figure 1. Reconfigurable Tactical Operations Simulator (RTOS) layout for the operational 

demonstration.......................................................................................................................... 7 

Figure A-1. RTOS Patriot system battalion representation. ........................................................ 24 

Figure A-2. The RTOS system. ................................................................................................... 25 

Figure A-3. Latest RTOS Reconfigurable Console Display........................................................ 26 

List of Tables 

Table 1. Commonly used simulation terms and definitions. ......................................................... 5 

Table 2a. Tactical Control Assistant (TCA) and Tactical Director Assistant (TDA) tasks used in 

the operational demonstration................................................................................................. 8

Table 2b. Tactical Control Officer (TCO) and Tactical Director (TD) tasks used in the 

operational demonstration....................................................................................................... 9 

Table 3. Operational demonstration training support timeline: 16 to 27 April 2007. ................ 10 

Table 4. Daily training schedule for days 1 and 2. ...................................................................... 11 

Table 5. Daily training schedule for day 3................................................................................... 11 

Table 6a. Utility evaluation ratings for Tactical Control Assistant (TCA) and Tactical Director 

Assistant (TDA) tasks. .......................................................................................................... 13 

Table 6b. Utility evaluation ratings for Tactical control Officer (TCO) and Tactical director 

(TD) tasks.............................................................................................................................. 13 

Table 7. Pre- and post-training results for the ARM classification training module.................. 14 

Table 8. Buy-In survey results..................................................................................................... 15 

Table 9. RTOS issues and modifications resulting from participant hotwash comments. .......... 17 
vi 

INTENTIONALLY LEFT BLANK. 
1 

1. Introduction 

1.1 Background 

During the combat operations phase of Operation Iraqi Freedom (OIF)—March-April 2003, U.S. 

Army Patriot air and missile defense (AMD) units were involved in two fratricide incidents. In 

the first, a British Tornado was misclassified as an anti-radiation missile (ARM) and 

subsequently engaged and destroyed. The second fratricide incident involved a Navy F/A-18 

that was misclassified as a tactical ballistic missile (TBM) and also engaged and destroyed. 

Three flight crew members lost their lives in these incidents. OIF involved a total of 11 Patriot 

engagements by U.S. units. Of these 11, nine resulted in successful TBM engagements; the 

other two (18%) were fratricides. 

In the spring of 2004, a team from the Army Research Laboratory’s Human Research and 

Engineering Directorate (ARL HRED) began looking into Patriot performance and training 

issues at the invitation of the then Ft. Bliss Commander, Major General (MG) Michael A. Vane. 

After reviewing the conclusions of the various boards of inquiry formed to look into the OIF 

fratricides, MG Vane was convinced that human performance issues were part of the problem 

leading to those incidents. He was particularly concerned by what he termed a “lack of 

vigilance” on the part of Patriot operators along with an apparent “lack of cognizance” of what 

was being presented to them on situation displays with a resulting “absolute trust in automation.” 

Accordingly, he requested that HRED conduct a human-performance-oriented critical incident 

assessment to complement the board of inquiry investigations and report back to him regarding 

potential problems and solutions. 

HRED’s project team spent most of the summer and fall of 2004 performing the requested 

human-performance-oriented critical incident assessment of the OIF fratricides. An initial 

assessment was delivered to MG Vane in October 2004. HRED’s report to MG Vane 

recommended two primary actionable items to redress the performance problems identified 

during the Patriot Vigilance effort: 

1. Re-examine automation concepts, operator roles, and command and control (C2) 

relationships in AMD battle command systems to emphasize effective human supervisory 

control; and 

2. Develop more effective missile crews and C2 teams, particularly with respect to Air 

Defense Operations—re-look the level of expertise required to operate such a lethal system 

on the modern battlefield. 
2 

In present usage, the term effective human supervisory control refers to a situation in which 

Soldiers and not the automated system are the ultimate decision makers in AMD firing decisions. 

Uncritical acquiescence to the automated system’s recommendations is not effective human 

supervisory control. 

In a report on Patriot system performance requested by the Under Secretary of Defense for 

Acquisition, Technology, and Logistics, the Defense Science Board reinforced HRED’s 

recommendations with the following comments (Defense Science Board, 2004). Although the 

full report is classified, these extracts are not. 

The Patriot system should migrate to more of a “man-in-the-loop” philosophy 

versus a fully automated philosophy—providing operator awareness and control 

of engagement processes. 

and 

Patriot training and simulations should be upgraded to support this man-in-theloop protocol including the ability to train on confusing and complex scenarios 

that contain unbriefed surprises. 

A summary of the report on Patriot system performance is available for download on the 

Defense Science Board web site. Readers should note that the report’s recommendations closely 

parallel HRED’s actionable items, and both address the need for AMD training modifications. 

In addition to the briefing to MG Vane, the initial phase of the Patriot Vigilance project also 

resulted in an ARL technical report titled The Human Side of Automation: Lessons for Air 

Defense Command and Control (Hawley, Mares, & Giammanco, 2005). After reviewing results 

from the first phase of the project, the TRADOC Capability Manager (TCM) for Lower Tier 

AMD systems requested that the Patriot Vigilance project continue into a second year. The 

TCM specifically requested that HRED’s project staff expand on the material presented in 

Hawley, Mares, and Giammanco (2005) and prepare two, more-detailed reports, one concerned 

with design for effective human supervisory control and a second addressing training for the 

emerging class of AMD systems. The intent of these reports was to inform the AMD community 

on “what right looks like” in each of these topic areas. The results were the two reports: 

Developing Effective Human Supervisory Control for Air and Missile Defense Systems (Hawley 

& Mares, 2006), and Training for Effective Human Supervisory Control of Air and Missile 

Defense Systems (Hawley, Mares, & Giammanco, 2006). 

In the late summer of 2005 after MG Vane had left Ft. Bliss for another assignment, the project 

staff briefed his replacement, MG (then Brigadier General) Robert P. Lennox, on the status and 

results of the Patriot Vigilance project. MG Lennox formally requested that the project be 

continued for at least another year so that the technical staff could work with the AMD 

community on implementing selected results. HRED’s project staff also would participate as the 

MANPRINT (Manpower and Personnel Integration) evaluator during a Limited User Test of the 
3 

Post-Deployment Build 6 (PDB-6) software suite for the Patriot system. PDB-6 was developed 

to address many of the Patriot system’s operational deficiencies that had surfaced during OIF and 

were generally considered to have contributed to the unacceptable fratricide rate. 

From the Fall of 2005 through the Summer of 2006 during the New Equipment Training (NET) 

and unit train-up period for the PDB-6 Limited User Test, the HRED project staff’s observations 

regarding the progress of training for the test unit sounded an alarm bell loudly. PDB-6 training 

was not progressing as it should have. Training events were being completed, but individual and 

crew performance objectives were not being met. Many of the training issues identified and 

discussed in Hawley, Mares, and Giammanco (2006) were surfacing and were not being 

addressed adequately by the NET process or follow-on collective training by the test unit. 

Moreover, inadequate unit training showed up clearly in the ensuing test results. 

The Army Evaluation Center’s System Assessment Report resulting from the PDB-6 Limited 

User Test concluded that the level of expertise required to employ the Patriot system properly 

with PDB-6 software had exceeded the current Army training standard. An earlier Army board 

of inquiry report on the OIF fratricides had reached a similar conclusion, noting that “the system 

[Patriot] is too lethal to be placed in the hands of crews trained to such a limited standard.” This 

convergence of evidence supported the conclusion that modifications to current AMD training 

practices were required. Moreover, these modifications would require not simply more 

“traditional” training, but performance-oriented strategies focused on deliberate practice (Hawley 

& Mares, 2007). Deliberate practice denotes a hands-on instructional regimen focused on 

specific instructional objectives accompanied by immediate and expert feedback (Ericsson & 

Charness, 1994). Frequent deliberate practice has been found to satisfy a primary condition for 

the development of high levels of job expertise (Ericsson, Krampe, & Tesch-Romer, 1993). 

1.2 The Reconfigurable Tactical Operations simulator (RTOS) Operational 

Demonstration 

After reviewing results from the OIF boards of inquiry, OIF training-related lessons learned, 

conclusions from HRED’s Patriot Vigilance project, and results from the PDB-6 Limited User 

Test, the Army Air Defense Artillery (ADA) School’s Directorate of Training, Doctrine, and 

Leader Development (DOTD-LD) concurred that a reexamination of AMD training practices and 

strategies was required. In addition to general agreement that a change in training rigor and 

instructional strategies was necessary, DOTD-LD identified a further training capability gap 

across AMD forces. This gap concerned the organic simulation capability available to AMD 

units. Their conclusion was that units required a capability to train fire control crews across the 

entire AMD kill chain that: 

1. Supplemented units’ organic embedded training (ET) capability; 

2. Did not require use of tactical equipment; and 
4 

3. Could be conducted in an environment conducive to teambuilding, coaching, teaching, and 

mentoring. That is, a capability that should be used to support a performance-oriented 

instructional strategy focused on deliberate practice. 

The School’s interest in an enhanced simulation capability to supplement ET on tactical 

equipment was initiated by an Operational Needs Statement produced by the 35th ADA Brigade 

in Korea. 

The ADA School’s concern for an enhanced unit training capability was heightened by that fact 

that upcoming Base Realignment and Closure (BRAC) moves of AMD units away from Ft. Bliss 

will amplify the training capability gap by denying units ready access to Ft. Bliss’ institutional 

(such as the Patriot Conduct of Fire Trainers, or PCOFTs) and post-wide training resources (such 

as the Drive Up Simulation and Training facility). 

DOTD-LD identified an existing device, the Reconfigurable Tactical Operations Simulator 

(RTOS), as potentially fulfilling the need for a simulation capability to supplement units’ organic 

ET capability, while not requiring the use of tactical equipment. RTOS hardware is commercial 

off the shelf (COTS), with the software being owned by the Army. The device has been in use 

since the late 1970s to support major air defense exercises as well as higher level 

experimentation and analysis. However, prior to the current project, the RTOS had not been 

used as a training device. 

As a trainer, RTOS is an exemplar of a medium- to high-fidelity, part-task training device for 

AMD air battle operations. The term medium-to high-fidelity means the RTOS is not an exact 

physical replica of the Patriot system, and that its functional characteristics are not identical to 

those of the tactical system. Table 1 presents and defines a set of terms commonly used to 

describe simulators and simulations. Part-task means that the RTOS does not support all AMD 

air battle operations tasks. The term reconfigurable means that the RTOS’ physical features 

(e.g., displays and controls) are not hard-manufactured or hard-wired into the device. Rather, 

displays are software driven and controls are implemented either using a mouse or a touchsensitive screen. The original intent behind the RTOS was that any potential AMD system or 

system modification could be mocked-up readily and used in demonstrations or experiments. A 

more complete description of the RTOS and its capabilities is provided in appendix A. 

Less-than-full-fidelity training devices such as the RTOS are commonly used in aviation and in 

other training applications (see, for example, Johnson & Stewart, 2005). Because of their 

relatively low cost versus full-fidelity training equipment, devices such as the RTOS have been 

found to be particularly useful in supporting essential skills integration (e.g., learning decision 

processes that underlie weapons system use) before moving to more costly and limited fullfidelity training settings (Alexander, Brunye, Sidman, & Weil, 2005). Other research discussed 

in Stewart, Johnson, and Howse (2007) suggests that less-than-full-fidelity training devices 

might even be beneficial for novice to mid-level trainees. Novice and mid-level trainees who are 

still making mistakes and learning through feedback may benefit from a simpler, more focused 
5 

training environment, more suitable for training primary skills. For trainees at the early- to midlevel of learning, high fidelity may provide no advantage for transfer to the target performance 

setting, and may actually introduce too much complexity into an already novel and stressing 

environment. In other words, the sheer complexity of the actual equipment setting in AMD may 

impede training for novice to mid-level trainees by making it more difficult to concentrate on the 

tasks at hand. 

Table 1. Commonly used simulation terms and definitions. 

Physical fidelity The degree to which the physical simulation looks, sounds, and feels like the operational 

environment in terms of visual displays, controls, and audio as well as the physical 

models driving each of these variables. 

Functional fidelity The degree to which the simulation acts like the operational environment in reacting to 

the tasks executed by the trainee. 

Buy-In The degree to which a person recognizes that an event or experience is useful for 

training. Acceptance of the simulation as “relevant,” and a willingness to use the 

environment to practice the behaviors targeted by the training program. 

Source: Alexander, Brunye, Sidman, & Weil (2005) 

1.3 Demonstration Objectives 

The RTOS Operational Demonstration (OpDemo) was organized as a joint project involving the 

ADA School’s DOTD-LD and the 5th Battalion 52nd ADA Regiment (5-52 ADA). 5-52 ADA is 

part of the 11th ADA Brigade. Personnel from ARL’s HRED participated with DOTD-LD and 

5-52 ADA in a technical advisory capacity. Technical support to the RTOS and other technical 

aspects of the OpDemo was provided by the simulator’s developer, Science Applications 

International Corporation (SAIC). 

DOTD-LD’s objectives in the RTOS OpDemo were to: 

1. Assess whether the RTOS could supplement existing training in the areas of (a) problemsolving for reaction to off-nominal engagement situations and (b) advanced Joint Air 

Defense Operations. 

2. Assess the utility of the RTOS for unit-level crew training in air defense operations in 

preparation for PDB-6 New Equipment Training. 

5-52 ADA’s objective in the OpDemo was to train fire control crews in an environment 

conducive to coaching, teaching, and mentoring using RTOS consoles and multiple engagement 

scenarios. Their interest in the OpDemo was motivated by a desire to conduct additional unitlevel crew training in preparation for PDB-6 NET. 

In essence, the OpDemo was focused on evaluating the feasibility, utility, and potential 

effectiveness of a new approach to AMD unit training emphasizing deliberate practice and using 

the RTOS as a training vehicle. This is an important point and bears repeating. The OpDemo 

was not simply about the RTOS. It was focused on two separate but related objectives: (1) the 
6 

demonstration and evaluation of modified instructional methods for use in AMD unit training, 

and (2) an assessment of the potential utility of the RTOS as a device to support AMD unit 

training. 

2. Method 

2.1 Participants 

Forty-two Soldiers from 5-52 ADA participated in the OpDemo during the period 16-27 April 

2007. Time in service for the participants ranged from 18.5 years to one year, with a median 

length of service of two years. In terms of time in their current job position, participant 

responses ranged from a high of seven years to a low zero (right out of Advanced Individual 

Training [AIT]). The median time in their current job position was 11.5 months. 

Personnel from 5-52’s Alpha (A), Bravo (B), Charlie (C), and Delta (D) batteries formed the fire 

unit crews used in the OpDemo—Engagement Control Station (ECS) Tactical Control Officers 

(TCOs) and Tactical Control Assistants (TCAs). Personnel from 5-52’s Headquarters and 

Headquarters Battery (HHB) formed the Information and Coordination Central (ICC) crews—

Tactical Director (TD) and Tactical Director Assistant (TDA). Instructor personnel used in the 

OpDemo were from 5-52’s EMMO (Electronic Missile Maintenance Officer) team and the ADA 

School’s DOTD-LD. In Patriot units, battalion EMMO personnel provide training and 

evaluation support to component batteries. 

2.2 RTOS Layout 

The RTOS stations used in the OpDemo were set up as a suite of eight consoles representing 

three Patriot fire unit (FU) ECS stations and one ICC. An additional workstation representing an 

external Patriot battalion was used as the Air Defense Artillery Fire Control Officer (ADAFCO) 

display. Additionally, a simulation and support workstation area was set up to allow the 

instructors to both create and distribute scenarios. Figure 1 proves a graphic depiction of the 

RTOS layout used in the Demonstration. The site for the OpDemo was 5-52 ADA’s Tactical 

Fire Direction Facility located at the Tobin Wells range area on Ft. Bliss. 

2.3 Air Battle Operations Tasks 

Twenty-three AMD air battle operations tasks were chosen for evaluation during the OpDemo. 

All of these are critical Patriot operator tasks for Military Occupational Specialties (MOSs) 14E 

(enlisted) and 14E/140E (officer/warrant officer). In addition, four high-priority operator 

activities associated with specific scenarios used during the OpDemo but not formally defined as 

tasks were added to the task list. The 27 tasks included in the OpDemo (23 “standard” plus four 

special) are listed in tables 2a and 2b. In tables 2a and 2b, tasks are categorized by job position: 
7 

TCA/TDA or TCO/TD. Tasks 24-27 are the extra activities included in the OpDemo. These 

tasks pertain to ARM classification procedures and FEZ-JEZ-MEZ operations. (Note: FEZ 

refers to a Fighter Engagement Zone; JEZ refers to a Joint Engagement Zone; and MEZ refers to 

a Missile Engagement Zone.) ARM classification procedures were selected for inclusion in the 

OpDemo because they represent an off-nominal performance situation that contributed to the 

Tornado fratricide during the combat operations phase of OIF (see Hawley & Mares, 2006). 

FEZ-JEZ-MEZ procedures are an important aspect of advanced Joint Air Defense Operations. 

The FEZ-JEZ-MEZ exercise permitted the trainees to participate in the complete kill chain for 

Joint Air Defense Operations from the ADAFCO located at the Air Force Controlling Authority 

down through the Patriot ICC to the ECS. Other unit-level training capabilities such as the 

embedded trainers in Patriot tactical shelters or the PCOFT at the ADA School do not currently 

support training on joint kill chain operations. 

Figure 1. Reconfigurable Tactical Operations Simulator (RTOS) layout for the operational demonstration. 

S3S S3S

S3S

S

S

S

[ FU 1 ] [ FU 2 ] 

[ADAFCO ] RT

[ VIDEO SCREEN ] 

[VIDEO 

PROJECTOR ] 

[ WHITE BOARD – Hard copy ] 

[ICC ] 

S

S

S

[ FU 3 ] 

S3 S3S

[Simulation and Support Workstations ] 

S3S S3S

S3S

[ FU 4 ] 
8 

2.4 Training Method and Instructional Approach 

Beyond the potential use of the RTOS in a unit setting, the OpDemo was intended to illustrate 

several modified approaches to AMD air battle operations training advocated in Hawley and 

Mares (2007). The first of these involved using the simulator in an open instructional 

environment conducive to coaching and mentoring. As trainees went through the various RAL 

(Reticule Aim Level—a proxy for scenario difficulty) scenarios used in the OpDemo, instructor 

cadre were immediately available to answer questions and provide corrective guidance. Each 

scenario was followed by a scenario-specific after action review (AAR) during which the 

trainees’ performance was critiqued. It was also possible to stop the scenario at any point to 

review an important concept or critique and correct participant performance to that point. The 

stop-action feature was used several times during the OpDemo. 

Several segments of the OpDemo also were used explicitly to illustrate and explore the benefits 

of a deliberate practice approach to training selected air battle operations tasks. As noted 

previously, deliberate practice denotes a hands-on instructional regimen focused on specific 

instructional objectives accompanied by immediate and expert feedback (Ericsson & Charness, 

1994). The deliberate practice portion of the OpDemo was implemented using what might be 

termed a “chunk and simulate” format. Trainees were first provided with a brief (15-20 minute) 

slide presentation addressing the specific instructional objective at hand (the “chunk” portion). 

This was followed by a relatively short, simulation-based practical exercise (30 minutes) 

focusing on the concepts covered in the presentation. Trainees were thus immediately allowed to 

practice the concepts covered in the up-front presentation. The simulation portion of the exercise 

was followed by a focused AAR that reviewed the concept under consideration, critiqued the 

trainees’ performance, and provided corrective guidance. Deliberate practice exercises were 

developed by DOTD-LD for ARM classification procedures and FEZ-JEZ-MEZ operations. 

Table 2a. Tactical Control Assistant (TCA) and Tactical Director Assistant (TDA) tasks used in the 

operational demonstration. 

TCA/TDA Tasks

1 441-083-1478 Engage Targets on the Engagement Control Station (ECS) 

2 441-083-1479 Evaluate Preengagement Data at the Engagement Control Station 

3 441-083-1480 Evaluate Preengagement Data at the Information and Coordination Central 

4 441-083-1482 Initiate Target Engagement at the Information and Coordination Central 

5 441-083-1486 Perform Friendly Protect at the Engagement Control Station (ECS) 

6 441-083-1487 Perform Friendly Protect on the Information and Coordination Central (ICC) 

7 441-084-1407 Perform Engagement Control Station (ECS) Initialization 
9 

Table 2b. Tactical Control Officer (TCO) and Tactical Director (TD) tasks used in the operational demonstration. 

TCO/TD Tasks

8 441-EW1-1084 Perform Engagement Control Station (ECS) Initialization 

9 441-EW1-1085 Perform Engagement Control Station (ECS) Reinitialization 

10 441-EW1-1087 Direct Firing Battery System Reorientation 

11 441-EW1-1088 Perform Tactical Ballistic Missile (TBM) Engagement Operations 

12 441-EW1-1089 Perform Air Defense Mission in all Modes of Control for Engagement Control Station 

(ECS) 

13 441-EW1-1090 Identify Targets at Engagement Control Station (ECS) 

14 441-EW1-1091 Supervise Engagement of Targets 

15 441-EW1-1093 Implement Firing Doctrine Changes 

16 441-EW1-1094 Define Engagement Control Station (ECS) Tabular Displays 

17 441-EW1-1095 Implement Air Breathing Threat (ABT) and Tactical Ballistic Missile (TBM) Search using 

Expanded Search Sectors 

18 441-EW1-1099 Supervise the Process of Airspace Control Orders (ACO) 

19 441-EW1-1113 Supervise Engagement Operations 

20 441-EW1-1121 Analyze the Process of Evaluation, Decision, and Weapons Assignment (EDWA) 

21 441-EW1-1122 Perform Friendly Protection 

22 441-EW1-1123 Perform Duties and Responsibilities as the Tactical Control Officer (TCO) and Tactical 

Control Assistant (TCA) 

23 441-EW1-1124 Perform Air Breathing Threat (ABT) and Tactical Ballistic Missile (TBM) Search Using 

Expanded Search Sectors 

24 Identify Mis-Classified Tracks that are displayed as ARM, TBM, or ABT symbology 

25 Identify when crew duties and responsibility should be modified 

26 Use of a banded FEZ,JEZ, MEZ TTPs 

27 Performing De-Lousing TTPs 

2.5 Procedure 

An overview of the structure of the OpDemo is provided in table 3. Each of 5-52 ADA’s 

batteries (A, B, C, and D) received a full day of training each week, for a total of two days. The 

Fridays of each week were shared by the batteries and were focused on battalion-specified 

training activities. Tables 4 and 5 provide a break-out of OpDemo activities for each day. Day 1 

refers to each battery’s training day during the first week; Day 2 refers to that battery’s training 

day during the second week. Table 5 shows the planned break-out of activities for the Fridays 

shared by the batteries, denoted as Day 3. 

Day 1 began with a presentation that provided an overview of the RTOS and discussed the intent 

of the OpDemo. Progressive RAL scenarios then commenced. Each scenario was followed by a 

Hotwash during which each of the fire unit crews, the ICC crew, the EMMO team, and anyone 

else participating in the exercise (e.g., the simulated ADAFCO) was permitted to express their 

opinions both pro and con on what had transpired. The DOTD-developed FEZ-JEZ-MEZ 

exercise took place during the afternoon of Day 1. 

Day 2 opened with a written pretest addressing the knowledge component of the ARM 

Classification exercise. The pretest was followed by the ARM Classification chunk and simulate 

training module. Following the training module, the ARM Classification knowledge test was 
10 

administered again as a posttest. The ARM Classification knowledge test used in the OpDemo is 

shown in appendix B. At the end of Day 2, demonstration participants were asked to complete 

two surveys. The first survey asked them to evaluate the RTOS in terms of how well it 

supported the 27 air battle operations tasks included in the OpDemo. Participants were asked to 

provide one evaluation for each task, with three possible levels of evaluation: Does Not Support

(N/S), Partially Supports (P/S), or Fully Supports (F/S). The first survey addresses what Bell 

and Waag (1998) and Johnson and Stewart (2005) refer to as the simulator’s utility for training 

the tasks under consideration. The Utility survey is shown in appendix C. 

The second survey administered at the end of Day 2 addressed the issue of trainee Buy-In 

(reference table 1); it also permitted the participants to provide open-ended, written comments on 

the OpDemo, the instructional procedures employed, or the RTOS as a training device, along 

with suggestions for improvements in any of those categories. This survey, denoted the RTOS 

Demo Questionnaire, is shown in appendix D. 

Although the OpDemo began with the schedule of activities outlined in tables 4 and 5, deviations 

from the plan occurred. For example, some of the battery crews were more advanced than others 

and progressed more rapidly, thus requiring the use of more difficult RAL scenarios near the end 

of their scheduled time. Similarly, some of the battery crews were not as far along in their 

training sequence as had been assumed, and thus required more practice on less complex 

scenarios. The schedule was also adjusted to accommodate visits to the OpDemo site by VIPs. 

These schedule adjustments permitted the VIPs to view and comment on salient aspects of the 

OpDemo such as the ARM Classification and FEZ-JEZ-MEZ training modules. VIPs visiting 

the OpDemo site included the Ft. Bliss Commanding General and members of his staff, the 

Commander of the 32nd Army AMD Command (AAMDC), various AMD brigade and battalion 

commanders and their staffs, and representatives from the AMD Lower Tier Project Office. 

Patriot is classified as an AMD Lower Tier system. 

Table 3. Operational demonstration training support timeline: 16 to 27 April 2007. 

April 

MON TUE WED THUR FRI 

16 ALPHA (A) 17 BRAVO (B) 18 CHARLIE (C) 19 DELTA (D) 20 BRAVO (Two 

Crews) 

 CHARLIE (One 

Crew) 

23 ALPHA 24 BRAVO 25 CHARLIE 26 DELTA 27 ALPHA (One 

Crew) 

 DELTA (Two 

Crews) 
11 

Table 4. Daily training schedule for days 1 and 2. 

DAY 1 

0900-0930 RTOS Overview 

0930-0950 Crew in brief 

0950-1000 Break 

1000-1045 1st Air Battle (RAL 5) 

1045-1050 Break 

1050-1105 Hotwash 

1105-1150 2nd Air Battle (RAL 9) 

1150-1200 Hotwash 

1200-1300 Lunch 

1300-1330 Crew in brief (DOTD JEZ) 

1330-1400 3rd Air Battle (DOTD JEZ) 

1400-1415 Break 

1415-1450 Hotwash with discussion 

1450-1500 Break 

1500-1550 4th Air Battle (RAL 11) 

1550-1600 Break 

1600-1630 AAR (for both scenario and day) 

DAY 2 

0900-0930 Written Test 

0930-0950 ARM Presentation 

0950-1000 Break 

1000-1045 ARM Scenario with discussion 

1045-1100 Written Test 

1100-1115 Break 

1115-1200 Test review and Hot Wash 

1200-1300 Lunch 

1300-1315 Crew in brief 

1315-1350 5th Air Battle 

1350-1400 Break 

1400-1430 Hotwash 

1430-1440 Break 

1440-1500 Crew in brief 

1500-1550 6th Air Battle 

1550-1600 Break 

1600-1630 AAR/ Written test (for both scenario and day) 

Table 5. Daily training schedule for day 3. 

DAY 3 

0900-0915 Crew in brief 

0915-0950 8th Air Battle (BN Focus) 

0950-1000 Break 

1000-1030 Hotwash 

1030-1115 9th Air Battle (BN Focus) 

1115-1130 Break 

1130-1200 Hotwash 

1200-1300 Lunch 

1300-1315 Crew in brief 

1315-1350 10th Air Battle (BN Focus) 

1350-1400 Break 

1400-1430 Hotwash 

1440-1600 Unit Focused Events 

1600-1630 AAR/ RTOS Surveys 
12 

3. Results 

3.1 Utility Evaluation 

Data from the Utility assessment of air battle operations tasks are presented in tables 6a and 6b. 

Table 6a presents results for TCA and TDA tasks, while table 6b presents results for TCO and 

TD tasks. Responses to the Utility survey were first screened to ensure that respondents held the 

proper MOS. This initial screening resulted in five participants being eliminated from the 

analysis (all 25Fs or 14Js). The remaining surveys were then screened again to ensure that 

respondents had the proper background (job assignment, time in the job position, etc.) to provide 

a valid assessment of the RTOS’ utility for training the task in question. Screenings were done 

by Patriot subject matter experts (SMEs) from DOTD-LD. The secondary screening resulted in 

the numbers given in the table columns labeled Total. 

The utility assessment results shown in tables 6a and 6b indicate that a large majority of the 

participants judged the RTOS to have utility for training the air defense operations tasks used in 

the OpDemo. In most cases, a large majority of the survey responses were for fully supports 

(F/S) or partially supports (P/S). The only exceptions to this general observation are for Tasks 7, 

8 and 9. These tasks all concern System Initialization and Reinitialization. At present, the 

RTOS does not support these tasks fully. 

After reviewing results from the Utility survey, Patriot SMEs from DOTD-LD cautioned that 

because of their lack of experience some participants might not have fully understood the 

complete scope of the tasks involved or what the survey questions implied. Consequently, while 

the Utility results support a general conclusion that the RTOS has value for training air defense 

operations tasks, further inquiry is necessary before making definitive conclusions regarding the 

device’s specific strengths or weaknesses. Clearly, however, most of the participants came away 

from the OpDemo with a favorable impression of the RTOS and its potential use as a unit-level 

training device. 
13 

Table 6a. Utility evaluation ratings for Tactical Control Assistant (TCA) and Tactical Director Assistant (TDA) 

tasks. 

TCA/TDA Tasks Total N/S P/S F/S 

1 Engage Targets on the Engagement Control Station (ECS) 20 0 5 15 

2 Evaluate Preengagement Data at the Engagement Control Station 21 1 4 16 

3 Evaluate Preengagement Data at the Information and Coordination 

Central 

12 0 1 11 

4 Initiate Target Engagement at the Information and Coordination 

Central 

12 0 1 11 

5 Perform Friendly Protect at the Engagement Control Station (ECS) 22 1 5 16 

6 Perform Friendly Protect on the Information and Coordination 

Central (ICC) 

18 1 1 16 

7 Perform Engagement Control Station (ECS) Initialization 20 3 8 9 

Table 6b. Utility evaluation ratings for Tactical control Officer (TCO) and Tactical director (TD) tasks. 

TCO/TD Tasks Total N/S P/S F/S 

8 Perform Engagement Control Station (ECS) Initialization 17 0 6 11 

9 Perform Engagement Control Station (ECS) Reinitialization 17 0 9 8 

10 Direct Firing Battery System Reorientation 19 0 6 13 

11 Perform Tactical Ballistic Missile (TBM) Engagement Operations 17 0 0 17 

12 Perform Air Defense Mission in all Modes of Control for 

Engagement Control Station (ECS) 

17 2 4 11 

13 Identify Targets at Engagement Control Station (ECS) 19 0 1 18 

14 Supervise Engagement of Targets 19 0 1 18 

15 Implement Firing Doctrine Changes 19 0 2 17 

16 Define Engagement Control Station (ECS) Tabular Displays 19 0 1 18 

17 Implement (ABT) and Tactical Ballistic Missile (TBM) Search using 

Expanded Search Sectors 

19 0 2 17 

18 Supervise the Process of Airspace Control Orders (ACO) 19 3 2 14 

19 Supervise Engagement Operations 19 0 3 16 

20 Analyze the Process of Evaluation, Decision, and Weapons 

Assignment (EDWA) 

19 0 4 15 

21 Perform Friendly Protection 19 0 3 16 

22 Perform Duties and Responsibilities as the Tactical Control Officer 

(TCO) 

18 0 1 17 

23 Perform Air Breathing Threat (ABT) and Tactical Ballistic Missile 

(TBM) Search Using Expanded Search Sectors 

19 0 1 18 

24 Identify Mis-Classified Tracks that are displayed as ARM, TBM, or 

ABT symbology 

19 0 1 18 

25 Identify when crew duties and responsibility should be modified 19 0 2 17 

26 Use of a banded FEZ,JEZ, MEZ TTPs 19 0 2 17 

27 Performing De-Lousing TTPs 19 0 2 17 
14 

3.2 Pre- Post-Test Results 

Bell and Waag (1998) list three categories of approach for evaluating the training effectiveness 

of simulators. Utility evaluations are the easiest and quickest to obtain, and are reported in the 

previous subsection. The second category of evaluation is in-simulator learning. Trainees 

practice tasks in the simulator and thereby show learning through an improvement in 

performance. Typically, the method is one of pre-test, practice, and then post-test. This general 

approach was used to assess the training effectiveness of the RTOS coupled with the chunk-andsimulate instructional approach for the ARM Classification module. Bell and Waag’s third 

category, transfer of training to the job setting, was beyond the scope of the OpDemo. The 

project staff will, however, continue to track the transfer of training issue during subsequent 

training and performance certification activities involving 5-52 ADA. 

Pre- and post-training results for the ARM Classification module are summarized in table 7. The 

mean score on the pre-training knowledge test was 18.78. After the chunk-and-simulate session, 

the mean score was 67.92. The mean improvement across participants was 49.14 points. 

Difference scores for each of the 37 MOS-qualified participants completing both the pre- and 

post-tests were evaluated using a paired-sample t-test. The t-test resulted in an observed t 

statistic of 12.91 with 36 degrees of freedom (p < .001). The differences in knowledge test 

performance pre- and post-training module are large and statistically significant. 

Table 7. Pre- and post-training results for the ARM classification 

training module. 

 

Preliminary 

Test 

PostTraining 

Test Change Score 

N=37 

Mean 18.78 67.92 49.14 

SD 16.73 23.95 23.15 

3.3 Buy-In Results 

Recall from table 1 that Buy-In is defined as “the degree to which a person recognizes that an 

event or experience is useful for training: Acceptance of the simulation as ‘relevant,’ and a 

willingness to use the environment to practice the behaviors targeted by the training program.” 

Buy-In is an important consideration with respect to the selection of any simulator. Research has 

shown (e.g., Salas, Bowers, and Rhodenizer, 1998) that trainee buy-in, or acceptance of a 

simulator or instructional setting as a legitimate training experience, is an important determinant 

of whether they will willingly use it and take the training experience seriously. 

Buy-In was assessed using the five statements listed as follows: 
15 

1. The RTOS unit training environment is more conducive to coaching, teaching, and 

mentoring than the current method used to train. 

2. I learned new advanced Joint Air Defense Operations techniques. 

3. I learned techniques to evaluate and react to off-nominal situations 

(misclassifications, anomalies, etc.). 

4. All things considered, this demonstration was a worthwhile training event. 

5. I think the RTOS will be a useful tool for unit training. 

Participants were asked to agree or disagree with each statement. 

Response totals for each statement are given in table 8. For each statement, a large majority of 

participants indicated agreement with the statement. For statements 4 and 5, participant 

agreement was unanimous (100% Agree): Participants uniformly agreed that the Demonstration 

was a worthwhile training event and that the RTOS would be a useful tool for unit training. The 

lowest percentage of favorable response was for statement 1. Here, only 37 of the 42 (88.1%) 

participants expressed agreement that the RTOS as used in the OpDemo was more conducive to 

coaching, teaching, and mentoring than the current method used to train. 

Table 8. Buy-In survey results. 

Statement Agree Pct (%) Disagree Pct (%) Total 

1 37 88.10% 5 11.90% 42 

2 40 95.24% 2 4.74% 42 

3 40 95.24% 2 4.76% 42 

4 42 100.00% 0 0.00% 42 

5 42 100.00% 0 0.00% 42 

3.4 Open-Ended Comments 

The RTOS Demo Questionnaire also permitted participants to provide open-ended comments 

concerning their experiences during the OpDemo. Two open-ended questions were posed to 

OpDemo participants: 

1. Comments? (Environment, Training, RTOS Issues, General Impressions) 

2. What does the RTOS need to do that it does not currently do? 

A list of all participant responses to item 1 is provided in appendix E. Participant responses to 

the second question concerning RTOS limitations generally tended to reflect the same issues and 

opinions obtained during the post-scenario Hotwash sessions. These results are addressed in the 

next subsection on Hotwash comments. For each comment in appendix E, the respondent’s rank 

and current job position or MOS are provided as an identifying header. 
16 

A review of participant responses in appendix E indicated an overall favorable opinion of the 

OpDemo, the RTOS, and the way it was used—the instructional approach focusing on deliberate 

practice. Participant comments were consistent with the Utility and Buy-In results presented 

previously. Many participants picked up on and commented favorably on the fact that the 

instructional experience provided during the OpDemo was different from standard training 

practices in AMD units. They liked the way training was done during the OpDemo. For the 

several respondents remarking that they liked the RTOS better than the PCOFT, follow-on, oneon-one interviews indicated that their unfavorable comments regarding the PCOFT resulted from 

the manner in which the PCOFT is currently used rather than any negative impressions of the 

simulator itself. 

3.5 Hotwash Comments 

At the conclusion of each scenario session a Hotwash was conducted. In present usage, the term 

Hotwash refers to a debrief session during which each group of participants was permitted to 

comment on the conduct of the exercise, their experiences with the RTOS, or any other topic 

they wished to bring up. A total of 30 Hotwash sessions were conducted over the two weeks of 

the OpDemo. For purposes of analysis, positive comments and criticisms from all 30 Hotwash 

sessions were aggregated by common themes. A total of three positive and four negative themes 

ran through participant Hotwash comments. 

The three positive Hotwash themes that emerged from participant comments were: 

1. Participants liked the overall training approach characterized by an open instructional 

environment with ready access to instructor cadre for feedback and corrective guidance. 

2. Participants liked the chunk-and-simulate instructional approach used for the JEZ-FEZMEZ and ARM Classification training modules. 

3. Participants viewed the OpDemo as an enjoyable, motivating instructional experience. 

With respect to the third Hotwash theme, one relatively experienced participant remarked that an 

instructional setting like that used in the OpDemo might make personnel “now avoiding the vans 

want to be in the vans.” The implication here is that training inside the tactical shelters (the vans) 

using the embedded trainers is a somewhat unpleasant experience, one that many AMD 

personnel avoid whenever possible. 

The four negative Hotwash themes reported by OpDemo participants were: 

1. Frequent and recurring RTOS “glitches,” or technical problems experienced during the 

Demonstration. 

2. RTOS physical fidelity and ergonomic issues—differences between the RTOS and Patriot 

tactical equipment or near-full-physical-fidelity trainers such as the PCOFT. 
17 

3. RTOS functional fidelity issues—RTOS functional limitations or differences between the 

way the system performs actions and the way these same functions currently are performed 

using the RTOS. 

4. The way ECS-ICC communications were handled during the Demonstration—the lack of 

headsets for ECS and ICC crews resulting in a somewhat noisy and chaotic training setting. 

Many of the RTOS features underlying these negative themes are already being addressed. For 

example, the RTOS developer and support contractor, SAIC, has already responded to a number 

of the participants’ comments, and others are under investigation for correction. A summary of 

RTOS modifications currently performed, underway, or under consideration as a result of the 

OpDemo is given in table 9. In all fairness, it should be noted that the support contractor had 

only a two-week preparatory period to install and integrate the RTOS workstations, develop and 

test scenarios, and refine the training environment. 

Table 9. RTOS issues and modifications resulting from participant hotwash comments. 

• First time used as a trainer and fully integrated with 8 consoles 

• Simulation running prototype software (TACI/K7/OB) 

• 70 hours of training, 13 simulation aborts (3 hrs downtime) = 96% available 

• Fire Platoon TAB 54 entries - 3 instances (software fixed) 

• Fire Platoon TAB 01 entries – 5 instances (software fixed) 

• Battalion TAB 55 entries - 2 instances (software fixed) 

• Battalion TAB 14 entries - 2 instances (software fixed, being tested) 

• One console loss of power during simulation (moved power strip) 

• Software Problems – 18 software problems were identified, 10 were corrected, 8 under investigation 

• Simulation Differences from Tactical System – 17 differences between the tactical system and the 

simulation were identified by the operators. 11 were corrected and 6 under investigation 

• Requested Simulation Enhancements Identified by Operators and Trainers 

• Remote Tracks at ECS 

• Ability to perform Battalion Initialization and Command Plan 

• Ability to perform Mapping 

• Ability to change TABS 70, 71, and 72 during tactical operations 

• Ability to perform clutter map update 

• Emulation of tactical communications networks between FPs, ICC, and ADAFCO 

• FU-FU capabilities 

4. Discussion 

As emphasized throughout the report, the RTOS OpDemo was an exercise intended to illustrate 

and evaluate the feasibility and acceptability of a new training approach for possible use in AMD 

units using the RTOS as a training vehicle. The new training approach emphasized deliberate 
18 

practice within an open instructional environment conducive to teaching, coaching, and 

mentoring. 

It should be noted that the Demonstration was not a training experiment. No novice fire unit 

crews were trained to criterion using the RTOS coupled with the new instruction instructional 

strategy and then compared to a comparable control group. It was more similar to a 

demonstration and case study than a controlled experiment. Consequently, any generalizations 

of results from the OpDemo are limited and must be made cautiously. However, despite these 

limitations and cautions, selected conclusions can be stated. 

First, the OpDemo results indicated that the RTOS has potential utility to support AMD unit 

training for air battle operations. Results from the Utility assessment strongly support this 

conclusion. In addition, results from the Buy-In survey indicate that participants viewed RTOSbased training as a legitimate training experience and were willing to accept it as a training tool. 

Both of these conclusions are further supported by participant responses to the Open-Ended 

survey questions and Hotwash remarks. 

With targeted modifications, the results suggest that the RTOS has sufficient physical and 

functional fidelity to be useful in the early to mid-ranges of the AMD training sequence. 

Potential categories of enhancements to the RTOS include: 

1. Ergonomic enhancements to increase physical fidelity. 

2. Functional modifications and add-ons. 

3. Instructional support features. 

Potential ergonomic and functional modifications were addressed in the previous section. 

Instructional support features are simulator modifications added to enhance the instructional 

process the device is intended to support. These include features such as stop-action, rewind, 

fast-forward, real-time replay from a fast-forward or rewind point, performance evaluation 

assists, and the like. It is also essential that the RTOS support the latest version of Patriot tactical 

software and that software upgrades be accomplished in a timely manner. 

In spite of the favorable review given the RTOS, AMD decision makers must bear in mind that 

simulators by themselves do not train. They are tools used by good instructors operating within 

a well-designed instructional strategy to achieve training objectives. Research on training 

effectiveness has consistently shown that how a simulator is used is more important than specific 

simulator features or simulation technologies (Salas, Bowers, & Rhodenizer, 1998). Moreover, a 

large base of training research indicates that more fidelity is not always better (Alexander, 

Brunye, Sidman, & Weil, 2005). These latter authors note that “a compromise must be made 

between physical fidelity, related costs, and training effectiveness, such that an adequate match 

can be made between training and real environmental elements and the logical structure of [the] 

tasks [to be trained]” (p. 7). They go on to note that the relationships among simulation fidelity, 
19 

user acceptance, and training effectiveness are not well understood. Each case must be explored 

heuristically based on its own merits, requirements, and knowledge base. The term heuristically 

means exploration using experimental, trial-and-error methods. The OpDemo was an example of 

a heuristic approach to problem exploration. 

In spite of the fidelity conundrum, the literature on simulators and simulations is clear on one 

point (Salas, Bowers, & Rhodenizer): Developers must avoid the blind pursuit of fidelity for its 

own sake. These authors (p. 200) note that realism (i.e., physical and functional fidelity) as a 

sole simulator design criterion “has led us to the point where in the quest for more realistic 

simulation, we have lost sight of the true goal—a more effective training device in terms of both 

training outcomes and cost. The two are not the same.” 

Second, the training strategy focusing on deliberate practice and characterized by an open 

instruction setting conducive to teaching, coaching, and mentoring was well received by 

OpDemo participants. Buy-In results, Open-Ended survey results, and Hotwash comments all 

support this conclusion. Participants liked the instructional approach used in the OpDemo and 

thought it should be further applied in a unit setting. In addition, pre- and post-test results from 

the ARM Classification training module indicate that the approach was effective. Pre- versus 

post-instruction test results showed significant positive improvement. Similar anecdotal results 

were observed for the JEZ-MEZ-FEZ instructional module. 

Third, there was remarkable consistency across the categories of data obtained over the course of 

the OpDemo. Utility results, Buy-In Survey results, Open-Ended comments, and Hotwash 

remarks all paint a consistent picture of user acceptance and potential training effectiveness. 

Such consistency suggests that in spite of the limitations on interpretation and generalization 

noted previously, the ADA School has a “green light” to pursue further applications of the RTOS 

(or an RTOS-like device) and the modified instructional strategies used during the OpDemo. In 

this respect, the RTOS OpDemo must be view as a success. Moreover, the training set-up used 

during the OpDemo (RTOS plus modified instructional approach) represents a partial prototype 

for a solution to the training deficiencies that contributed to the OIF fratricides and that showed 

up again during the PDB-6 Limited User Test. 

These latter two conclusions have considerable support within the training effectiveness 

literature. For example, Salas, Bowers, and Rhodenizer (1998) argue that the training 

community must emphasize training systems that use technology to promote learning. As 

mentioned previously, this requires a shift from designing simulation solely for realism—and 

hoping that learning occurs—to the design of human-centered training systems that support the 

acquisition of complex skills. These authors go on to assert that achieving this goal will require 

a paradigm shift from a focus on simulation to a broader consideration of the entire training 

system, of which the simulator is but one component. In essence, this comment summarizes the 

approach used in the OpDemo and that met with an overwhelmingly favorable response from 

participants. 
20 

In spite of the success observed during the OpDemo, it would be a mistake to deploy the RTOS 

or any such device to AMD units without an extensive training support package and the requisite 

command acceptance, support, and emphasis. Without these dual elements of support, the 

present initiative to improve AMD unit training and enhance unit performance will fail. For the 

RTOS or any similar simulator, essential training support requirements must include: 

1. An overall AMD training strategy into which such a device fits. 

2. Enhanced instructional methods (e.g., vignettes supporting progressive instructional 

objectives and deliberate practice versus simple free-play in a scenario-based 

environment). 

3. Pedagogical modifications (e.g., changes in the way instructors conduct unit training). 

4. Courseware support (scenario templates, training and evaluation guides, etc.). 

5. Technical support to the simulator itself. 

All of these support elements are critical to the successful application an instructional regimen 

emphasizing deliberate practice within an open learning environment, regardless of the simulator 

selected for use. 

Success in an AMD training reform effort also may require a re-look at the issue of what must be 

trained. Current job and task analysis products for Patriot and other AMD systems may not 

support the instructional approach used during the OpDemo without extensive modifications and 

clarifications. For example, the JEZ-MEZ-FEZ and ARM Classification modules used during 

the OpDemo are not included in official Critical Task Lists for Patriot MOSs, yet these modules 

involve skills that are essential to successful job performance in the contemporary Patriot 

operational environment. 
21 

5. References 

Alexander, A. L.; Brunye, T.; Sidman, J.; Weil, S. A. From gaming to training: A review of 

studies on fidelity, immersion, presence, and buy-in and the effects on transfer in PC-based 

simulations and games. DARWARS Training Impact Group. Defense Advanced Research 

Projects Agency: Arlington, VA, 2005. 

Bell, H. H.; Waag, W. L. Evaluating the effectiveness of flight simulators for training combat 

skills: A review. International Journal of Aviation Psychology 1998, 8 (3), 223-242. 

Defense Science Board. Patriot system performance. (Final Report of the DSB Task Force on 

Patriot System Performance). Office of the Undersecretary of Defense for Acquisition, 

Technology, and Logistics: Washington, DC, 2004. 

Ericsson, K. A.; Krampe, R. T.; Tesch-Romer, C. The role of deliberate practice in the 

acquisition of expert performance. Psychological Review 1993, 100, 363-406. 

Ericsson, K. A.; Charness, N. Expert performance: Its structure and acquisition. American 

Psychologist 1994, 49 (8), 725-747. 

Hawley, J. K.; Mares, A. L. Developing effective human supervisory control for air and missile 

defense systems; ARL-TR-3742; U.S. Army Research Laboratory: Adelphi, MD, 2006. 

Hawley, J. K.; Mares, A. L. Developing effective adaptive missile crews and command and 

control teams for air and missile defense systems; ARL-SR-149; U.S. Army Research 

Laboratory: Adelphi, MD, 2007. 

Hawley, J. K.; Mares, A. L.; Giammanco, C. A. The human side of automation: Lessons for air 

defense command and control; ARL-TR-3468; U.S. Army Research Laboratory: Adelphi, 

MD, 2005. 

Hawley, J. K.; Mares, A. L.; Giammanco, C. A. Training for effective human supervisory 

control of air and missile defense systems; ARL-TR-3765; U.S. Army Research Laboratory: 

Adelphi, MD, 2006. 

Johnson, D. M.; Stewart, J. E. III. Utility of a personal computer-based aviation training device 

for helicopter flight training. International Journal of Applied Aviation Studies 2005, 5 (2), 

287-305. 

Salas, E.; Bowers, C. A.; Rhodenizer, L. It is not how much you have but how you use it: 

Toward a rational use of simulation to support aviation training. International Journal of 

Aviation Psychology 1998, 8 (3), 197-208. 
22 

Stewart, J. E.; Johnson, D. M.; Howse, W. FASS-C ARI Q&A. Unpublished manuscript, U.S. 

Army Research Institute for the Behavioral and Social Sciences, Rotary Wing Aviation 

Research Unit, Ft. Rucker, AL, 2007. 
23 

Appendix A. RTOS Patriot PDB-5 Tactical System Representation 

The Reconfigurable Tactical Operations Simulator (RTOS) system began its development in 

1977 under contract with the U.S. Army Missile Command (MICOM). In 1981, the 

responsibility of the system was transferred to the U.S. Army Air Defense Artillery School 

(USAADASCH) at Ft. Bliss, Texas. The RTOS has undergone extensive enhancements to 

replicate the functions of the deployed Patriot Tactical System and Support simulation models of 

other Air Defense systems. The U.S. Army renamed the Patriot Tactical Operations Simulator 

(PTOS)-to-RTOS in 1989 because of the additional air defense simulations added, the use of 

reconfigurable consoles, and the incorporation of Distributed Interactive Simulation (DIS) into 

the RTOS architecture. These capabilities enable the RTOS to be used as a multiple-system air 

defense simulator with growth capabilities to incorporate additional weapon or command-andcontrol systems and participate in exercises and training events. 

RTOS Patriot simulation provides a high fidelity representation of the PATRIOT software in key 

functional areas that support studies, exercises and training. Since its initial development, the 

RTOS Patriot model was based on Data Processing System Requirements (DPSR) documents of 

the tactical Patriot system and provides algorithm representation of the functional areas 

supporting its role as an analysis tool for the U.S. Army’s Directorate of Combat Developments 

(DCD). The RTOS Patriot simulator uses two levels of modeling: algorithmic and probabilistic. 

During the early development of the Patriot Missile System, the RTOS was used by 

USAADASCH to implement, verify, and validate the proposed Patriot tactical software. Impacts 

on established tactics and doctrine were identified and resolved before the tactical system 

software became operational. A major engineering revision to the tactical software could be 

identified and tested before implementation of the system software, avoiding costly and timeconsuming modifications to the tactical Patriot software before fielding. 

The RTOS Patriot is capable of representing a full battalion configuration of up to six Fire 

Platoons made up of single or dual console representations of ICC and ECSs. The RTOS ECS 

representation can either be interactive with Reconfigurable consoles or they can be represented 

in batch (non-interactive) with internal operator representations. 
24 

Figure A-1. RTOS Patriot system battalion representation. 

A-1. RTOS Users 

The RTOS plays an important role in the U.S. Army’s participation in multilevel air defense 

demonstrations and exercises. From the earliest days of the PPO, through the deployment of the 

first operational Patriot software revision and refinement to the present configuration, the RTOS 

has been an important Soldier-in-the-loop air defense tool supporting studies and exercises. 

Although the proponent and owner of the RTOS system is USAADASCH, there are other users 

that support the RTOS. The German Air Force (GAF) used the RTOS system for more than 17 

years. The German Patriot Project Office (GEPO), located at Ft. Bliss, Texas, uses the system in 

conjunction with the Army to produce studies and recommendations for future Patriot system 

changes as part of bi-lateral studies. The RTOS Patriot system was procured by the Japan 

Defense Agency (JDA) with five RTOS consoles in 1996 and installed at Iruma Air Base, Japan 

and includes both PDB-3 and PDB-4 representations. 
25 

Figure A-2. The RTOS system. 

In addition to the U.S. Army, the German and Japanese Air Force, other users and supporters of 

the RTOS include the Missile Defense Agency (MDA), Joint Tactical Air and Missile Defense 

Office (JTAMDO), Space and Missile Defense Command (SMDC), and AMRDEC-Software 

Engineering Directorate with permanent RTOS systems located in their facilities (Ft. Bliss 

Warfighting Center, AMRDEC-SED-Redstone, JNIC-Colorado Springs, JASDF- Japan, SMDC 

Battle Lab - Redstone, and Virtual Warfare Center-St. Louis). 

A-2. Model Configuration 

RTOS supports both Standalone and Distributed simulations. Stand-alone simulations are 

supported with the RTOS support programs that use a scenario generated by the internal 

Scenario Generation Program supporting threat development of Air Breathing Threats (ABT), 

Tactical Ballistic Missiles (TBM), and Cruise Missiles. Distributed simulations are supported by 

the RTOS DIS capability which allows the RTOS to be driven by other models and interact with 

other non-RTOS models either in the RTOS facility or externally over data/phone lines. This 

capability allows other models to provide higher fidelity threat interjection. The characteristics 

of RTOS are the following: 

• Supports real-time execution on large size scenarios 

• Supported by user-definable parameters and event scheduling 

• Supports customization via its modular design to allow new air defense models to be 

incorporated into the existing models. 
26 

RTOS is supported by Interactive Support Programs that support the users’ ability to use RTOS 

as a standalone trainer or as a exercise support tool. Support programs include: Scenario 

Generation Program, Namelist Program, Event Calendar File Program, Data Reduction Program 

and the capability to supports individual console replay. The RTOS system allows the 

Integration of Multiple RTOS systems locally through Ethernet connections. This capability 

allows different RTOS models (for example Patriot and SLAMRAAM) to execute together using 

the same scenario. Using the DIS standard and defined protocol data units, RTOS models can be 

linked together with other models in other locations 

A-3. Hardware 

The RTOS system is composed of off-the-self hardware components. The simulation and 

console computer systems are supported by multiple processor platforms to include Silicon 

Graphics, SUN and PC. The reconfigurable console system provides a display for the air 

defense systems during the simulation and is composed of two 30 inch high resolution LCD 

touch screens with replica keyboards and joy-stick. The reconfigurable console is modular and 

easily transported. 

Figure A-3. Latest RTOS Reconfigurable Console Display. 
27 

A-4. Software Architecture 

The RTOS is a highly flexible and realistic simulator of multiple air defense weapon systems. 

Its design concept was driven by the goal to develop an analysis tool with an open system 

architecture that can be adapted to growing needs without costly redesign. The very extensive 

software strictly follows actual system specifications, has been coded in a high-order language, 

and consists of over 150,000 lines of code. A modular design structure allows for easy and lowrisk modifications, extensions, and even the addition of complete new system models. This 

flexibility supports rapid and inexpensive responses to changes in weapons system software. It 

also supports the analyst's need for specific modifications to the simulation models necessary to 

enforce a very particular system performance within specific analyses. The majority of the 

model areas of the RTOS are supported by user-modifiable parameters and probabilities. These 

areas include the following: 

• Surveillance and radar 

• Missile 

• Tracking 

• Identification Friend of Foe (IFF) 

• Time thresholds 

• Identification 

• Selection 

• Data links 

• Display. 

A-5. RTOS Patriot Simulation Software 

The Patriot software of the RTOS is periodically upgraded to reflect changes in the fielded 

Patriot system and currently maintains baselines for PDB-3, PDB-4, and PDB-5 and initial 

development of PDB-6. The RTOS Patriot ICC controls up to six Patriot Engagement Control 

Stations (ECS). The reconfigurable consoles can be configured as ICC and/or ECS. 

The Fire Unit-level model in RTOS Patriot is represented by the major functions of: 

Surveillance/Track, Classification / Identification, Threat Assessment/Engagement Decision, 

Weapon Assignment/Missile Guidance, and Kill Assessment/Attacker Penetration. The 

Engagement Decision and Weapon Assignment (EDWA) function is an example of an 

algorithmic level of modeling, which models the tactical operations processing for the Patriot 

FUs. The EDWA function provides threat evaluation of detected targets, ID, target 

classification, threat assessment and engagement eligibility evaluation, engagement processing 
28 

and response to ICC messages and operator actions. Processing to accomplish these tasks during 

a scenario is based upon the algorithms outlined by the Patriot EDWA DPSR. 

Track management at the FP is an example of a probabilistic level of modeling. It models the 

detection of threat groups and maintenance of the track position, velocity and selected track 

status information. The detection and tracking process is based on a surveillance procedure that 

includes a radar model of the Patriot FP radar characteristics and probability factors that can be 

set by the user. Masked terrain effects on the radar visibility are supported. High-fidelity 

software modules, which replicate established Patriot data link, Patriot Data Interface Language 

(PADIL), are employed to model actual data communications behavior. 

The RTOS Patriot model replicates the Command and Control function of the Patriot battalion 

and provides target evaluation, engagement-control, and processing-command input from the 

operator or higher echelon (HE). Evaluation consists of target Identification (ID) and 

Identification Friend or Foe (IFF) processing classification, threat assessment and protection of 

assets, engagement eligibility, and selection of a FP to conduct the engagement. Engagement 

control includes pre-engagement processing of threats, computation of engagement data, and 

maintenance of the to-be-engaged queue (TBEQ), engagement release to subordinate units, and 

the monitoring and kill assessment of ongoing engagements. Command input provides 

processing of commands from the operator, via switch action, or from HE via data link message. 

These include engage, hold-fire, cease-fire, and engage-hold. Operators may release hold-fire, 

cease-fire, and engage-hold commands. Major portions of data communications between 

subordinate units and the battalion C2 element are included in the RTOS. The communications 

model uses PADIL and external tactical higher echelon TADIL-J data link. The TADIL-J 

datalink allows the RTOS Patriot ICC to directly connect with any higher echelon element. 

SAIC has supported over 50 simulation and training exercises with the TADIL-J data link. 
29 

Appendix B. ARM Classification Knowledge Test 

Date________________ 

Soldier Rank and Name:_______________ ______________________ 

Unit:____________ Crew Position______________ Age:__________ 

Military Education: _____________________________________________________ 

Civilian Education______________________________________________________ 

Do you have a PC at Home?__________ Do you play video games?________ 

PC or Console?_______________ 

1. What are the three tests that the system uses to determine if a track is an ARM 

1. 

2. 

3. 

2. What are some general characteristics that the system uses to determine if a track 

 is an ARM? 

3. What does the Config 3 Radar add as a requirement for the system to classify a track as 

an ARM? 

4. What is the primary discriminant for classifying a track? 

5. Target classification is NOT reassessed after a classification is ____________. 

6. The default tab setting in Tab 76 page A for range is ___________. 
30 

7. The default tab setting in Tab 76, page A for minimum speed is __________. 

8. If you have an AS-11 threat, your minimum speed tab setting is ___________ as per the 

PDB 5.5.2 Speed Values for ARM Threats white paper. 

9. If you have an AS-17a threat, your minimum speed tab setting is ___________ as per the 

PDB 5.5.2 Speed Values for ARM Threats white paper. 

10. What tab setting should you authorize to improve ARM detection performance as per the 

PDB 5.5.2 Improving ARM Detection Performance White Paper? 
31 

Appendix C. Utility Assessment Survey 

Date________________ 

Soldier Rank and Name:_______________ ______________________ 

Unit:____________ Crew Position______________ Age:__________ 

Military Education: _____________________________________________________ 

Civilian Education______________________________________________________ 

Do you have a PC at Home?__________ Do you play video games?________ 

PC or Console?_______________ What Genre (RPG,RTS,FPS):______ 

 Task N/S P/S F/S 

TCA/TDA Tasks 

1 441-083-1478 Engage Targets on the Engagement Control Station (ECS) 

2 441-083-1479 Evaluate Preengagement Data at the Engagement Control Station 

3 441-083-1480 Evaluate Preengagement Data at the Information and Coordination 

Central 

 

4 441-083-1482 Initiate Target Engagement at the Information and Coordination 

Central 

 

5 441-083-1486 Perform Friendly Protect at the Engagement Control Station (ECS) 

6 441-083-1487 Perform Friendly Protect on the Information and Coordination Central 

(ICC) 

 

7 441-084-1407 Perform Engagement Control Station (ECS) Initialization 

 

TCO/TD Tasks 

8 441-EW1-1084 Perform Engagement Control Station (ECS) Initialization 

9 441-EW1-1085 Perform Engagement Control Station (ECS) Reinitialization 

10 441-EW1-1087 Direct Firing Battery System Reorientation 

11 441-EW1-1088 Perform Tactical Ballistic Missile (TBM) Engagement Operations 

12 441-EW1-1089 Perform Air Defense Mission in all Modes of Control for Engagement 

Control Station (ECS) 

 

13 441-EW1-1090 Identify Targets at Engagement Control Station (ECS) 

14 441-EW1-1091 Supervise Engagement of Targets 

15 441-EW1-1093 Implement Firing Doctrine Changes 

16 441-EW1-1094 Define Engagement Control Station (ECS) Tabular Displays 

17 441-EW1-1095 Implement (ABT) and Tactical Ballistic Missile (TBM) Search using 

Expanded Search Sectors 

 

18 441-EW1-1099 Supervise the Process of Airspace Control Orders (ACO) 

19 441-EW1-1113 Supervise Engagement Operations 

 

20 441-EW1-1121 Analyze the Process of Evaluation, Decision, and Weapons 

Assignment (EDWA) 

 
32 

21 441-EW1-1122 Perform Friendly Protection 

22 441-EW1-1123 Perform Duties and Responsibilities as the Tactical Control Officer 

(TCO) 

 

23 441-EW1-1124 Perform Air Breathing Threat (ABT) and Tactical Ballistic Missile 

(TBM) Search Using Expanded Search Sectors 

 

24 Identify Mis-Classified Tracks that are displayed as ARM, TBM, or ABT symbology 

25 Identify when crew duties and responsibility should be modified 

26 Use of a banded FEZ,JEZ, MEZ TTPs 

27 Performing De-Lousing TTPs 
33 

Appendix D. RTOS Demo Questionnaire 

Primary MOS ______________________ Duty MOS ___________________ Rank _______ 

Job/Position ______________________ # Yrs Service ___________ # Yrs in Position ___ 
34 

Please comment on any response of (1) Disagree to (2) Agree 

Statement Disagree 

(1) 

Agree 

(2) 

1. The RTOS Unit training environment is 

more conducive to coaching, teaching and 

mentoring then the current method used to 

train. 

 

2. I learned new advanced Joint Air Defense 

Operation techniques. 

 

3. I learned techniques to evaluate and react to 

off nominal situations (misclassifications, 

anomalies) 

 

4. All things considered, this demo was a 

worth while training event. 

 

5. I think the RTOS will be a useful tool for 

unit training. 

 

6. Comments: (Environment, Training, RTOS issues, General Impressions). 

_____________________________________________________________________________ 

_____________________________________________________________________________ 

______________________________________________________________________________ 

______________________________________________________________________________ 

______________________________________________________________________________ 

______________________________________________________________________________ 

______________________________________________________________________________ 

7. What does the RTOS need to do that it does not currently do? 

______________________________________________________________________________ 

______________________________________________________________________________ 

______________________________________________________________________________

______________________________________________________________________________ 

______________________________________________________________________________ 

______________________________________________________________________________ 

______________________________________________________________________________ 
35 

Appendix E. Open-Ended Survey Responses 

SPC/TDA: RTOS does not have the capability to surpass training on actual equipment. 

However, RTOS would be great tool to deployed units where system equipment is not readily 

available for training. RTOS would be good for use with a tactical seminar situation as a 

demonstration tool for specific tactical situations and negative Patriot situations like fratricide. 

PV2/TDA: I think it [provides] a great learning and teaching environment. I learned more here 

in two weeks than in most of AIT. It was fun yet very productive—good people, great teaching, 

and good tools. Should be put in all Patriot batteries for further teaching and learning. 

CW3/TD: Very useful tool for S/I and Tab learning. Useful for maintaining crew proficiency. 

Good for giving classes and showing what is being taught to large groups. Another tool for 

crews to training and get familiar with the system. 

CW3/TD: Great training tool. Training with RTOS resulted in increased system knowledge and 

understanding. Needs to support training of complete crews—incorporate 25F and 

communications. 

SPC/TDA: Environment conducive to giving FUs situation awareness (ICC sit display on 

projector). I think it should be at the battery level for system knowledge and low intensity air 

battles. Screens seem to get too cluttered for high intensity. 

PV2/TDA: I liked it. I got what the FU and ICC go through. Need more commo realism—but 

if it did that it wouldn’t be as informational as it is. Talk between FU and ICC in the same room. 

PV2/14E: It was a well constructed program and system, and I hope that I can have more work 

with it. I definitely learned new things and gained more information, and I just left the AIT 

environment. Need to add fix or fight decision making and FUFU operations. 

PFC/TDA: Very good training tool. 

SSG/14E: Good and could be a useful training tool at the battalion or battery level to training 

soldiers with a different perspective than the PCOFT or in Van by yourself. 

SPC/TCA: Isolated training (the ARM Classification and JEZ-MEZ-FEZ modules) excellent. 

Outstanding training system. 

2LT/14E: Good training tool. Gives batteries greater access to hands-on practice as well as 

ability to accentuate training lessons immediately after a verbal class. 

SPC/14E: Train units at least every other day to make all units battle ready. 

SPC/14E: Good for training—no need to set up. 
36 

SPC/TCA: Minus some minor system glitches, the system is very useful and would be a very 

valuable asset. 

2LT/TCO: I like it! It would be better than the PCOFT. I think it would be helpful at the battery 

level. 

SPC/TCA: Good system—great training tool. 

2LT/TCO: This was excellent training because the focus was on training and not on evaluating. 

We were allowed to ask as many questions as we could ask and there were plenty of people to 

give an answer. The hands-on is the best method of learning, but it is much better when you 

have a presentation precede it the way it was done here. I hope to see the RTOS in the near 

future as part of the unit’s equipment. It will facilitate training as often and as simple as 

possible. For once we can focus on actual training and not on passing “spinning wheels” (an 

11th ADA Brigade term which means they are doing an in depth preparation for a certain event 

usually a table certification). Our job is air battles and not mechanics. 

2LT/TCO: I believe the RTOS system is the ideal place to teach additional skills and info 

(initially). Once crews have been exposed to it, they should refine it in the van. 

2LT/TCO: Should have somebody knowledgeable by the TCOs and TCAs position to help 

mentor and coach them during the decision making process throughout the air battles. That way 

we can see what we are doing right and wrong. I think the RTOS is a very complete program 

and covers all aspects of the ECS. A good thing is that you have the ICC right there answering 

the TCO’s questions. 

2LT/14E: The specified tasks, scenarios and training prior to and during RALs (scenarios) were 

very useful. I felt I had a better understanding of my purpose and understand the situation, 

threat, and capabilities of the system better now. We need to do some ACO (Airspace Control 

Order) training. 

PV2/TCA: I liked the hands-on training. I think it would be priceless to have one at the battery 

level. 

SSG/14E30: This system would be good training at the platoon level. 

2LT/TCO: It was good in all areas. I think it’s better than the PCOFT. 

2LT/14A: Good training. Helps before going to the van. 

SGT/14E: Great tool for training. 

SPC/14E: A few glitches, but is a good training device. 
37 

Distribution List

ADMNSTR 

DEFNS TECHL INFO CTR 

ATTN DTIC-OCP (ELECTRONIC COPY) 

8725 JOHN J KINGMAN RD STE 0944 

FT BELVOIR VA 22060-6218 

DARPA 

ATTN IXO S WELBY 

3701 N FAIRFAX DR 

ARLINGTON VA 22203-1714 

OFC OF THE SECY OF DEFNS 

ATTN ODDRE (R&AT) 

THE PENTAGON 

WASHINGTON DC 20301-3080 

COMMANDING GENERAL 

US ARMY AIR DEFNS ARTILLERY CTR 

AND FT BLISS (5 COPIES) 

ATTN MG R P LENNOX 

BLDG 2 SHERIDAN RD 

FT BLISS TX 79916 

ASSISTANT COMMANDANT 

US ARMY AIR DEFNS ARTILLERY CTR 

AND FT BLISS (5 COPIES) 

BLDG 2 SHERIDAN RD 

FT BLISS TX 79916 

COMMANDER 

108TH ADA BDE 

ATTN AFZX-CO (5 COPIES) 

FT BRAGG NC 28310-5100 

COMMANDER 

11TH ADA BDE 

ATTN AFVJ-CO (5 COPIES) 

FT BLISS TX 79916 

COMMANDER 

263RD AAMDC (5 COPIES) 

1 NATIONAL GUARD RD 

COLUMBIA SC 29201 

COMMANDER 

31ST ADA BDE 

ATTN AFVR-CO (5 COPIES) 

FT BLISS TX 79916 

COMMANDER 

32ND AAMDC 

ATTN AFVL-CO (5 COPIES) 

FT BLISS TX 79916 

COMMANDER 

35TH ADA BDE 

ATTN EAAD-H, PSC3 (5 COPIES) 

APO AP 96266 

COMMANDER 

69TH ADA BDE 

ATTN AETV-GW-CO, CMR 475 (5 COPIES) 

PO BOX 1402 

APO AE 09036 

COMMANDER 

94TH AAMDC 

ATTN AMPC-CG (5 COPIES) 

MAIL STOP 577 

FT SHAFTER HI 96858 

ARL HRED AMEDD FLD ELMT 

ATTN AMSRD-ARL-HR-MM V RICE-BERG 

BLDG 4011 RM 217 1750 GREELEY RD 

FT SAM HOUSTON TX 78234-5094 

ARL-HRED AMCOM FLD ELMT 

ATTN AMSRD-ARL-HR-MO J MINNINGER 

BLDG 5400 RM C-242 

REDSTONE ARSENAL AL 35898-7290 

US ARMY RSRCH DEV AND ENGRG CMND 

ARMAMENT RSRCH DEV AND ENGRG CTR 

ARMAMENT ENGRG AND TECHNLGY CTR 

ATTN AMSRD-AAR-AEF-T 

ATTN AMSRD-AAR-AEF-T J MATTS 

BLDG 305 

ABERDEEN PROVING GROUND MD 21005-

5001 

ARMY G1 

ATTN DAPE-MR B KNAPP 

ARMY G1 MANPRINT DAPE MR 

300 ARMY PENTAGON RM 2C489 

WASHINGTON DC 20310-0300 
38 

ARMY RSCH LAB-HRED JFCOM JOINT 

EXPERIMENTSATION J9 JOINT FUTURES 

LAB 

ATTN AMSRD-ARL-HR-MJK 

 J HANSBERGER 

115 LAKEVIEW PARKWAY STE B 

SUFFOLK VA 23435 

ARMY RSRCH LAB-HRED 

ATTN AMSRD-ARL-HR-MU 

 M SINGAPORE 

6501 E 11 MILE RD MS 284 BLDG 200A 2ND 

FL RM 2104 

WARREN MI 48397-5000 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MQ 

 M R FLETCHER 

AMSRD-NSC-SS-E BLDG 3 RM 341 

NATICK MA 01760-5020 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-ML J MARTIN 

MYER CENTER RM 2D311 

FT MONMOUTH NJ 07703-5601 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MG R SPINE 

BUILDING 333 

PICATINNY ARSENAL NJ 07806-5000 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MK J REINHART 

10125 KINGMAN RD 

FT BELVOIR VA 22060-5828 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-M M STRUB 

6359 WALKER LANE SUITE 100 

ALEXANDRIA VA 22310 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MN R SPENCER 

DCSFDI HF 

HQ USASOC BLDG E2929 

FT BRAGG NC 28310-5000 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MW E REDDEN 

BLDG 4 RM 332 

FT BENNING GA 31905-5400 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MT J CHEN 

12423 RESEARCH PARKWAY 

ORLANDO FL 32826-3276 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MD T COOK 

BLDG 5400 RM C242 

REDSTONE ARSENAL AL 35898-7290 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MH C BURNS 

BLDG 1467B RM 336 THIRD AVE 

FT KNOX KY 40121 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MP 

 D UNGVARSKY 

BATTLE CMD BATTLE LAB 

415 SHERMAN AVE UNIT 3 

FT LEAVENWORTH KS 66027-2326 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MV 

 HQ USAOTC S MIDDLEBROOKS 

91012 STATION AVE RM 111 

FT HOOD TX 76544-5073 

ARMY RSRCH LABORATORY-HRED 

ATTN AMSRD-ARL-HR-MY M BARNES 

2520 HEALY AVE STE 1172 BLDG 51005 

FT HUACHUCA AZ 85613-7069 

ARMY RSCH LABORATORY - HRED 

AVNC FIELD ELEMENT 

ATTN AMSRD-ARL-HR-MJ D DURBIN 

BLDG 4506 (DCD) RM 107 

FT RUCKER AL 36362-5000 

US ARMY TRADOC 

BATTLE LAB INTEGRATION & TECHL 

DIRCTRT 

ATTN ATCD-B 

10 WHISTLER LANE 

FT MONROE VA 23651-5850 

COMMANDANT USAADASCH 

ATTN AMSRD-ARL-HR-ME A MARES 

ATTN ATSA-CD 

5800 CARTER RD 

FT BLISS TX 79916-3802 
39 

DIRECTOR 

DIRECTORATE OF COMBAT 

DEVELOPMENTS 

ATTN COL H L COHEN 

5800 CARTER RD 

FT BLISS TX 79916-7001 

DIRECTOR 

DIRECTORATE OF TRAINING, DOCTRINE, 

& LEADER DEVELOPMENT 

ATTN COL R K CARL (5 COPIES) 

ATTN J FALLIN 

ATTN C WALLET 

2 SHERIDAN RD, BLDG 2 

FT BLISS TX 79916-7001 

PROJECT MANAGER 

IAMD PEO MS 

ATTN SFAE-MSLS-SS R THOMAS 

PO BOX 1500 

HUNTSVILLE AL 35807-3801 

PROJECT MANAGER 

LTPO PEO MS 

ATTN SFAE-MSLS-LT E EDWARDS 

PO BOX 1500 

HUNTSVILLE AL 35807-3801 

DIRECTOR 

OFFICE, CHIEF OF AIR DEFENSE 

ARTILLERY 

ATTN ATSA-OCADA 

FT BLISS TX 79916 

PM TIMS, PROFILER (MMS-P) AN/TMQ-52 

ATTN B GRIFFIES 

BUILDING 563 

FT MONMOUTH NJ 07703 

PRODUCT MANAGER TIMS 

ATTN SFAE-IEWS-NS TIMS 

ATTN SFAE-IEWS-NS TIMS B GRIFFIES 

BLDG 563 

FT MONMOUTH NJ 07703 

DIRECTOR 

QUALITY ASSURANCE OFFICE 

ATTN ATSA-QAO 

FT BLISS TX 79916 

SMC/GPA 

2420 VELA WAY STE 1866 

EL SEGUNDO CA 90245-4659 

TRADOC CAPABILITY MANAGER-LOWER 

TIER 

ATTN COL R L DELGADO 

BLDG 12, PERSHING RD 

FT BLISS TX 79916-7001 

TRADOC CAPABILITY MANAGER-UPPER 

TIER 

ATTN ATSA-TSM-UT 

FT BLISS TX 79916 

US ARMY INFO SYS ENGRG CMND 

ATTN AMSEL-IE-TD F JENIA 

FT HUACHUCA AZ 85613-5300 

COMMANDER 

US ARMY RDECOM 

ATTN AMSRD-AMR W C MCCORKLE 

5400 FOWLER RD 

REDSTONE ARSENAL AL 35898-5000 

US ARMY RSCH LABORATORY 

ATTN AMSRD-ARL-CI-OK-TP S FOPPIANO 

BLDG 459 

ABERDEEN PROVING GROUND MD 21005 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-CI-OK-TP 

 TECHL LIB T LANDFRIED 

BLDG 4600 

ABERDEEN PROVING GROUND MD 21005-

5066 

US GOVERNMENT PRINT OFF 

DEPOSITORY RECEIVING SECTION 

ATTN MAIL STOP IDAD J TATE 

732 NORTH CAPITOL ST., NW 

WASHINGTON DC 20402 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-HR-S L PIERCE 

ATTN AMSRD-ARL-HR-SE D HEADLEY 

ATTN AMSRD-ARL-HR-SE K COSENZO 

BLDG 459 

ABERDEEN PROVING GROUND MD 21005 

US ARMY RSRCH LAB 

ATTN AMSRL-ARL-HR F PARAGALLO 

BLDG 459 

ABERDEEN PROVING GROUND MD 21005-

5066 
40 

DIRECTOR 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-RO-EV W D BACH 

PO BOX 12211 

RESEARCH TRIANGLE PARK NC 27709 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-HR-ME J HAWLEY 

FT BLISS TX 79916 

US ARMY RSRCH LAB 

ATTN AMSRD-ARL-D J M MILLER 

ATTN AMSRD-ARL-CI-OK-T 

 TECHL PUB (2 COPIES) 

ATTN AMSRD-ARL-CI-OK-TL 

 TECHL LIB (2 COPIES) 

ATTN IMNE-ALC-IMS 

 MAIL & RECORDS MGMT

ADELPHI MD 20783-1197