Research Product 2003 - 04

Research Observations and Lessons

Learned for the Future Combat Systems 

Charlotte H. Campbell 

Human Resources Research Organization 

May H. Throne 

Barbara A. Black 

Carl W. Lickteig 

U.S. Army Research Institute 

June 2003 

Armored Forces Research Unit

U.S. Army Research Institute for the Behavioral and Social Sciences 

Approved for public release; distribution is unlimited. 
U.S. Army Research Institute 

for the Behavioral and Social Sciences 

A Directorate of the U.S. Total Army Personnel Command 

ZITA M. SIMUTIS 

 Director 

Technical Review by 

Margaret S. Salter, U.S. Army Research Institute 

Dee Andrews, U.S. Air Force Research Laboratory 

NOTICES 

DISTRIBUTION: Primary distribution of this Research Product has been made 

by ARI. Please address correspondence concerning distribution of reports to: 

U.S. Army Research Institute for the Behavioral and Social Sciences, Attn: 

TAPC-ARI-PO, 5001 Eisenhower Ave., Alexandria, VA 22333-5600. 

FINAL DISPOSITION: This Research Product may be destroyed when it is no 

longer needed. Please do not return it to the U.S. Army Research Institute for 

the Behavioral and Social Sciences. 

NOTE: This Research Product is not to be construed as an official Department 

of the Army position, unless so designated by other authorized documents.
REPORT DOCUMENTATION PAGE

1. REPORT DATE (dd-mm-yy)

 June 2003 

2. REPORT TYPE 

 Final 

3. DATES COVERED (from. . . to) 

 29 Jul 02 – 15 May 03 

5a. CONTRACT OR GRANT NUMBER 

 DASW01-99-D-0012, DO 0018 

4. TITLE AND SUBTITLE

Research Observations and Lessons Learned for the Future

Combat Systems 5b. PROGRAM ELEMENT NUMBER

 0603007A

5c. PROJECT NUMBER

 A792

5d. TASK NUMBER

 212 

6. AUTHOR(S) 

Charlotte H. Campbell (Human Resources Research 

Organization), May H. Throne, Barbara A. Black and 

Carl W. Lickteig (U.S. Army Research Institute)

5e. WORK UNIT NUMBER

 Human Resources Research U.S. Army Research Institute for the 

 Organization (HumRRO) Behavioral and Social Sciences (ARI) 

 66 Canal Center Plaza, Suite 400 ATTN: TAPC-ARI-IK 

 Alexandria, VA 22314 2423 Morande Street 

 Fort Knox, KY 40121-5620

8. PERFORMING ORGANIZATION

REPORT NUMBER 

10. MONITOR ACRONYM

 ARI 

9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES)

 U.S. Army Research Institute for the Behavioral and Social Sciences

 ATTN: TAPC-ARI-IK 

 5001 Eisenhower Avenue 

 Alexandria, VA 22333-5600

11. MONITOR REPORT NUMBER

Research Product 2003-04 

12. DISTRIBUTION/AVAILABILITY STATEMENT

 Approved for public release; distribution is unlimited.

13. SUPPLEMENTARY NOTES

Contracting Officer’s Representative: Dr. May H. Throne

This brief summarizes some of the important observations and lessons learned from training research and 

development conducted by the U.S. Army Research Institute for the Behavioral and Social Sciences (ARI) over

the past 15 years. Its purpose is to assist Future Combat Systems (FCS) designers and developers as they

formulate plans for the systems and for the training that will accompany the systems. The topic summaries

cover the need for a Master Training Plan, training development and implementation concerns, training and 

performance support and system issues, and a few non-training and non-system observations. The reference 

list (Appendix B) gives the complete citation for readers who need more information, and the reference matrix at 

Appendix C shows which documentation sources each topic summary is based on. In selected cases, more

complete summaries on specific techniques or principles are provided in Appendix D. Finally, Appendix E 

contains a compendium of research issues that remain to be investigated. 

15. SUBJECT TERMS 

 Embedded Training Lessons Learned 

Future Combat Systems Systems Approach to Training (SAT)

SECURITY CLASSIFICATION OF

16. REPORT

 Unclassified

17. ABSTRACT

 Unclassified

18. THIS PAGE

 Unclassified 

19. LIMITATION OF 

ABSTRACT

 Unlimited 

20. NUMBER

 OF PAGES 

21. RESPONSIBLE 

PERSON

 Dr. May H. Throne 

 502/624-3450 

i 
ii 
iii 

Research Product 2003 - 04 

 

Research Observations and Lessons Learned 

for the Future Combat Systems 

Charlotte H. Campbell 

Human Resources Research Organization 

May H. Throne 

Barbara A. Black 

Carl W. Lickteig 

U.S. Army Research Institute

Armored Forces Research Unit 

Barbara A. Black, Chief 

U.S. Army Research Institute for the Behavioral and Social Sciences 

5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600 

June 2003 

Army Project Number Personnel, Performance 

2O363007A792 and Training 

Approved for public release; distribution is unlimited. 
iv 
RESEARCH OBSERVATIONS AND LESSONS LEARNED FOR THE FUTURE

COMBAT SYSTEMS 

FOREWORD

The U.S. Army has initiated transformation to an Objective Force equipped with Future

Combat Systems (FCS) to be more responsive, deployable, agile, versatile, lethal, survivable, 

and sustainable to meet the full spectrum of future missions. Development and fielding of the 

FCS is critical to the success of the Army’s Transformation. Training is recognized as an 

essential component of the FCS Program, and considerable thought and analysis have been given 

to the driving requirements for training. One of the key drivers concerns the reality that FCS and 

the Objective Force are new – revolutionary rather than evolutionary. The design and 

development of the evolving System of Systems is broad and multifaceted, and the scope of

work involved in developing collective and individual training is correspondingly complex. 

For many years, the U.S. Army Research Institute for the Behavioral and Social Sciences

(ARI) has been involved in the development of embedded training concepts and methods. As 

initial planning and acquisition documents indicate that the primary method for implementing

Objective Force training will be embedded training, ARI, at its Armored Forces Research Unit at

Fort Knox, undertook to create a compendium of observations and lessons learned from the 

many research reports and products generated over the past 15 years. 

This Research Product presents the findings from the effort. Nearly 100 reports were 

reviewed, and interviews were conducted with a dozen experts in training who had participated 

in the research. Almost 400 individual “nuggets” of information were collected, sorted, and 

summarized. The summaries are succinct – rarely will they provide sufficient information for a

designer or developer who does not already understand something about training principles to go

design some training. For those who are interested in more information, documentation sources 

for each topic as well as more complete summaries on specific techniques and principles are 

provided in the appendixes. 

The preliminary results of this effort were presented to Mr. Jim Shiflett, the FCS training

leader for Science Applications International Corporation (SAIC), which is the Lead Systems 

Integrator (LSI) for FCS, on 16 December 2002. The purpose of this product is to assist FCS

designers and developers as they formulate plans for the systems and for the training that will 

accompany the systems. 

 MICHAEL G. RUMSEY 

 Acting Technical Director 

v 
ACKNOWLEDGEMENTS

On a report such as this, the credit for any value conferred to the reader goes to the many

researchers and military thinkers whose work inspired this effort. The reference section gives a

list of all of the reports and articles reviewed. Special credit is due to all of those authors. We

also need to recognize Ms. Deborah Marcum (HumRRO), who served as manager of the 

database containing all of the research report data, observations, and lessons learned. 

We are especially indebted to those individuals who contributed additional thoughts and insights 

as the report was being written, notably: 

Dr. John Barnett, U.S. Army Research Institute for the Behavioral and Social Sciences 

(ARI), Simulation Systems Research Unit (SSRU), Orlando. 

Dr. Billy L. Burnside, ARI, Armored Forces Research Unit (AFRU), Fort Knox. 

Mr. Daniel E. Deter, SAIC, Fort Knox. 

Dr. Jean L. Dyer, ARI, Infantry Forces Research Unit (IFRU), Fort Benning. 

Mr. Michael R. Flynn, Northrop Grumman Information Technology, Fort Knox. 

Dr. James R. Gossman, HumRRO, Advanced Distributed Training Program (ADTP), 

Fort Knox. 

Mr. Christopher Graves, HumRRO, ADTP, Fort Knox. 

Mr. Charles Heiden, HumRRO, ADTP, Fort Knox.

Mr. William T. Holden, Jr., HumRRO, ADTP, Fort Knox. 

Dr. Angela I. Karrasch, ARI, AFRU, Fort Knox

Dr. James W. Lussier, ARI, AFRU, Fort Knox. 

Dr. Larry Meliza, ARI, SSRU, Orlando. 

Dr. Kathleen A. Quinkert, Chief, TRADOC Scientific Coordination Office, Fort Monroe. 

Ms. Margaret S. Salter, ARI, IFRU, Fort Benning. 

Dr. Bruce S. Sterling, U.S. Army Research Laboratory (ARL), Fort Knox. 

vi 
RESEARCH OBSERVATIONS AND LESSONS LEARNED 

FOR THE FUTURE COMBAT SYSTEMS 

Page

Introduction....................................................................................................................................1 

Method ............................................................................................................................................2 

Organization of the Report ...........................................................................................................2 

Discussion........................................................................................................................................3 

The Big Picture: The FCS Master Training Strategy ...............................................................4 

Define the master training plan ................................................................................................4 

Integrate the master plan ..........................................................................................................4 

Information...............................................................................................................................5 

Resources and activities ...........................................................................................................5 

Big part of big picture: the Training Design Team (TDT)......................................................6 

Analysis...........................................................................................................................................8 

Training audience .....................................................................................................................8 

Tasks and standards..................................................................................................................8 

Conditions ................................................................................................................................8 

Modeling expert performance ..................................................................................................9 

Analysis methods .....................................................................................................................9 

Objectives – What to Train.........................................................................................................10 

Digital skills ...........................................................................................................................10 

Back-up skills.........................................................................................................................10 

Individual competencies.........................................................................................................11 

Staff and team skills...............................................................................................................12 

Training Design............................................................................................................................13 

Training focus.........................................................................................................................13 

Learner styles .........................................................................................................................13 

Skill acquisition and retention................................................................................................14 

Expert skill training................................................................................................................15 

Training sequence...................................................................................................................15 

Large-scale v. small-scale exercises.......................................................................................15 

vii 
Training support packages......................................................................................................16 

Training Development.................................................................................................................18 

Use of user and expert input...................................................................................................18 

Development approaches .......................................................................................................18

Developer tools.......................................................................................................................19 

Training Implementation............................................................................................................20 

Access issues..........................................................................................................................20 

Multiple delivery modes.........................................................................................................20 

Analysts and O/Cs..................................................................................................................21 

Flexible training support ........................................................................................................21 

Managing TSPs ......................................................................................................................21 

Simulation environments........................................................................................................22 

Sustain and manage content ...................................................................................................23

Training management system.................................................................................................23 

Evaluation.....................................................................................................................................25 

Formative evaluation..............................................................................................................25 

Summative evaluation ............................................................................................................25 

Performance Measurement.........................................................................................................27 

Measuring performance standards..........................................................................................27 

Automated measurement protocols........................................................................................27 

Non-automated protocols.......................................................................................................28 

Skill indicators........................................................................................................................28 

Feedback .......................................................................................................................................30 

Focused feedback ...................................................................................................................30 

Timely feedback .....................................................................................................................30 

Tailored feedback ...................................................................................................................32 

Performance Support Systems....................................................................................................33 

Information management and support....................................................................................33 

Feedback.................................................................................................................................33 

Operator surrogate or backup .................................................................................................34

Training mechanism...............................................................................................................34 

System Acquisition and Related Early Decisions......................................................................35 

Early collaboration .................................................................................................................35 

viii 
Planning ahead .......................................................................................................................35 

Soldier-in-the-loop .................................................................................................................35 

Doctrine development ............................................................................................................36 

System Characteristics and Features.........................................................................................37 

Growth potential.....................................................................................................................37 

Integration ..............................................................................................................................37 

Reliability ...............................................................................................................................37 

Training support .....................................................................................................................38 

Presentation features...............................................................................................................38 

Workload management...........................................................................................................39 

User needs ..............................................................................................................................39 

Performance Improvement – Factors Beyond Training ..........................................................41 

Personnel challenges ..............................................................................................................41 

Human factors challenges ......................................................................................................41 

Formal information sharing....................................................................................................42 

Miscellaneous.........................................................................................................................42 

Appendixes 

Appendix A. Definitions of Acronyms and Jargon ................................................................... A-1 

Appendix B. References .............................................................................................................B-1 

Appendix C. Matrix of Topics and References ..........................................................................C-1 

Appendix D. Explication of Selected Methods and Findings.................................................... D-1 

D.1 Proposed Roadmap for Multi-Skilled Soldier (MSS) Achievement................................ D-2 

D.2 Development of Structured Simulation-Based Training Programs.................................. D-3 

D.3 Expanded Procedure for Development of Structured Simulation-Based Training 

Programs........................................................................................................................... D-4 

D.4 Development of Small Group Exercises.......................................................................... D-5 

D.5 The Staff Performance Analysis Process and Results...................................................... D-6 

D.6 Digital Skill Sets............................................................................................................... D-7 

D.7 Digitization Applied to Unit Performance Problems....................................................... D-8 

D.8 Method for Analysis of Back-Up Requirements.............................................................. D-9 

D.9 Training Objectives for Battalion and Brigade Sections and Leaders ........................... D-10 

ix 
D.10 Performance Objectives Methodology........................................................................... D-11 

D.11 Train-the-Trainer: Developing Guidance for Observers, Controllers, 

and Interactors................................................................................................................ D-12 

D.12 High Impact Transfer Technology ................................................................................. D-14 

D.13 Training Support Code Methodology............................................................................. D-15 

D.14 Evaluating Virtual Environment Training Capabilities.................................................. D-16 

D.15 Identifying Core Set Exercises....................................................................................... D-18 

D.16 Training Program Conversion........................................................................................ D-19 

D.17 Refined Training Analysis and Feedback Center of Excellence 

Concept........................................................................................................................... D-20 

D.18 Strategies to Reduce O/C Workload .............................................................................. D-23 

D.19 Evaluating TSP Components.......................................................................................... D-25 

D.20 Training Effectiveness Analysis Methods...................................................................... D-26 

D.21 Diagnostic Indicators of Digital Proficiency.................................................................. D-27 

D.22 Factors Impacting Digital Proficiency ........................................................................... D-29 

D.23 Usage Considerations for Embedded Training and Performance Support Systems...... D-30 

D.24 Design Guidelines for Embedded Training and Performance Support Systems............ D-31 

D.25 AAR Aids....................................................................................................................... D-32 

D.26 Cognitive Learning Strategies........................................................................................ D-33 

D.27 System Improvement Recommendations....................................................................... D-34 

D.28 Leader Insights on Digitization and Transformation ..................................................... D-36 

Appendix E. Research Issues.......................................................................................................E-1 

Performance issues...............................................................................................................E-1 

System issues........................................................................................................................E-1 

Training issues......................................................................................................................E-2 

Methodological issues..........................................................................................................E-2 

x 
RESEARCH OBSERVATIONS AND LESSONS LEARNED 

FOR THE FUTURE COMBAT SYSTEMS 

Introduction

The U.S. Army has initiated transformation to a lighter, more mobile Objective Force that can 

operate readily within joint and coalition environments. This force will be responsive, deployable, 

agile, versatile, lethal, survivable, and sustainable. These characteristics will enable Objective 

Force units to see first, understand first, act first, and finish decisively. Objective Force units will

possess knowledge-based capability to respond rapidly and decisively across the full spectrum of 

military operations through deployment with the Future Combat Systems (FCS). The FCS is 

designed to be a system of systems that is fully networked to ensure rapid and complete sharing of 

information. The FCS will be fielded initially in mobile combined arms Units of Action, roughly

equivalent to today’s brigades. 

Throughout its history, the Army has undergone transformation frequently, if not continuously. 

Over the last 10-15 years, the pace of transformation has quickened as the Army has engaged in 

more diverse, non-traditional missions, while at the same time becoming more digitally equipped. 

Previous research and development (R&D) efforts have examined training, performance support, 

and feedback methods for these new missions and environments. There is a potential for much 

useful information that can be gathered from these past efforts that will have an impact on training

Units of Action equipped with FCS. 

Given that the Army is in the early stages of transformation, it is essential to review previous 

relevant R&D efforts and collect the lessons learned as well as the observations of the key

researchers involved. These research observations and lessons learned will serve as a starting point

for training Units of Action, helping to ensure that available findings are applied appropriately and 

previous mistakes are not repeated. This report summarizes some of the important observations 

and lessons learned from training research and development conducted by the U.S. Army Research

Institute for the Behavioral and Social Sciences (ARI) over the past 15 years. Its purpose is to

assist Army designers and developers as they formulate plans for the Future Combat Systems 

(FCS) and for the training that will accompany the systems. 

ARI’s long history and reputation as a leader in training research and development (R&D) is 

evidenced in thousands of research reports, technical reports, study reports, and research products. 

ARI scientists and sponsored contractors have published findings and analyses in the open 

literature and presented at professional conferences for both military and industry audiences. The 

domain is not restricted to conventional training methods, but draws on ARI expertise in human 

factors, cognitive processes, organizational theories, military operations and doctrine, individual 

and collective performance improvement, and technological system design. The body of 

information has long been a source of ideas and building blocks for researchers and decisionmakers, but had not been assembled into a coherent summary or searchable database. 

The intent of this project is to provide a very basic compilation of research observations and

lessons learned to inform FCS developers and Unit of Action (UA)1

 training developers. It is not a 

comprehensive database of all ARI work, nor does it incorporate information from non-ARI

sources. However, it provides training developers with a succinct collection of recommendations

and guidelines for use as training for the FCS goes forward. 

 

1

 Underlined terms are defined in Appendix A.

1 
Method 

A number of sources were examined. For the most part, we relied on documentation in the form of 

research reports, technical reports, and study reports published by ARI. One of those sources 

(Leibrecht, Johnston, Black, & Quinkert, 2002) discusses a “knowledge elicitation” process for use 

in eliciting tacit knowledge from experts. We employed a modification of the process for this

report that was more focused on published documentation than on interviews. We also, however, 

interviewed a dozen of the principal researchers who had performed and directed work on multiple

projects, both ARI and contractor, to gather more global observations that would not be found in 

the separate reports. Nearly 400 discrete “nuggets” were entered into the database from these 

sources. 

The nuggets were then grouped to form topics and specific subject areas within the broader topics. 

Topic definitions were written and refined to focus the content, although it should be noted that the 

integrated nature of training development requirements precludes production of non-overlapping

topic areas. Summaries were prepared to present the best wisdom from the nuggets – 

recommendations, suggestions, opinions, strong research findings, and alternatives are all included.

Several readers reviewed and refined each summary, ensuring that the nuggets were adequately

represented. 

At the same time, all of the information was entered into a relational database using Microsoft 

Access®. At the conclusion of the project, the database, with data-entry forms, queries, and report

templates, was added to ARI’s collection of research and study products. This database is 

expandable and searchable, and should be maintained and used as FCS development continues and

as additional ARI research is completed. 

As we were gathering and analyzing the observations and lessons learned, we encountered one that

encouraged us to continue the process. It is presented here for your edification.

As the number of ‘lessons learned’ continues to grow, we are continually reminded to 

learn old lessons better, even as we amass new ones. There is a need to review and

analyze those lessons in the course of development projects. While all of the 

information is available in the separate reports, it would be a benefit to developers to 

have a single compendium or database containing lessons and even the larger lessons 

that emerge from content analysis of the individual lessons. (Campbell, Deter, et al., 

1999, p. 33) 

Organization of the Report 

The topic summaries are presented in an order that made sense to the authors. The first topic

addresses the need for a Master Training Plan – not to imply that it has not been thought of, but to 

reinforce the need for such a plan and to offer a few reminders of what should be included, based 

on the research. The next several topics are related to training development and implementation. 

We then discuss training and performance support and system issues. The final topic summary

touches on a few non-training and non-system observations – they are not training, but they are too 

important to leave out. 

Each topic summary contains a short introductory paragraph that defines the issue. The subject 

areas within each topic also begin with introductory paragraphs, followed by a cautionary note: 

what will happen if the issue is not addressed. The remaining portion of the subject area discussion 

presents the imperatives or suggestions drawn from the reviews. Appendix A contains definitions 

2 
of acronyms and terms that may not be familiar to all readers; such terms have been underlined like 

this in the topic summaries.

The summaries are succinct – rarely will they provide sufficient information for a designer or 

developer who does not already understand something about training principles to go design some

training. However, the reference list (Appendix B) gives the complete citation for readers who 

need more information, and the reference matrix at Appendix C shows which documentation 

sources each topic summary is based on. In selected cases, more complete summaries on specific 

techniques or principles are provided in Appendix D. Finally, Appendix E contains a compendium

of research issues culled from the sources, issues that remain to be investigated. 

Discussion 

This is by no means a comprehensive summary of all R&D findings related to training, nor is the 

list of reports in Appendix B exhaustive. For the most part, we fixated on embedded training (ET)

2

and lessons learned that may have an impact on training Units of Action equipped with FCS. 

Despite all of the obstacles that remain, we can more readily visualize such training occurring in

vehicles than in equipment carried by dismounted soldiers (e.g., the Land Warrior systems). 

Additionally, the development of simulation systems for dismounted troops is currently less mature 

than other development, so that deriving lessons learned is more straightforward in R&D related to 

maneuver units. 

Many of the outstanding issues are presented in Appendix E. Although the list is short (two pages)

it is quite dense, and yet only scratches the surface. These issues were identified in published 

reports; a team of military trainers could easily identify many more. 

 

2

 For a brief review and definitions of ET, see Throne and Burnside (2003). 

3 
The Big Picture: The FCS Master Training Strategy

A recurring theme in training research and development concerns the need for planning and 

analysis. This is a requirement not only for discrete training support packages (TSPs) or training 

programs, but also for the Big Training Program. That is, there is a need for an overarching 

strategy that specifies the TSPs that will be needed. To all appearances, the FCS team is investing 

significant energy and thoughts in developing an overall training plan that covers individual and 

collective training, multiple training locales (schoolhouse, new equipment training (NET), unitbased), and an assortment of training environments

Define the

master 

training 

plan 

The FCS Master Training Strategy must address the full array of training approaches 

that have been shown to be effective in previous training programs, specify how and 

when they are to be used, and designate the role of the institution, the units, and the 

individual. 

If the Master Plan does not cover alternative approaches, a strategy for when to train 

with what, and details of who does what, then it is not yet ready for use.

The integrated training strategy should address (at a minimum) ET, structured 

training, synchronous and asynchronous distance learning, distributed and fixed 

site simulation based training, and live training. 

The training approach should clearly distinguish between training, concept 

exploration, and mission rehearsal, and address all three. 

Train leaders first, because leaders are the trainers for their units and so must be 

expert in the training methodology and doctrine specific to the Objective Force. 

Before a significant bank of TSPs is available, the strategy should prescribe 

what types of training should be developed for what purposes. Once the TSPs 

begin to become available, the strategy should designate specific TSPs for 

specific purposes. 

A number of roles need to be assigned to the institution, the unit, and the 

individual, including the roles of training developer, distributor, evaluator,

manager, and trainer. 

The training management system needs to be connected to every element of the 

strategy.

Training issues include the training burden. One hope (not a method) is that

institutional training will not be lengthened and the training burden will not fall 

on field commanders. Rather, soldiers will arrive at their assignments with all 

of the skills necessary to perform competently, or will train themselves using

distributed training approaches once they leave initial training. This is not 

going to happen by itself, but will require support materials and resources. 

Integrate 

the master 

plan 

Even bigger than the training plan is the whole FCS concept, and the pieces need to be 

connected explicitly. 

Training is not the only way to ensure expert performance, but it is essential, and 

cannot succeed unless it is an integrated part of the whole FCS concept, including 

systems development and personnel issues. 

4 
A range of non-training issues is discussed in the section on Performance Improvement.

Only two are described here. 

The strategy needs to address technology insertion plans – not only for the 

initial FCS systems but also for the training, and not only to cover training

upgrades to match FCS upgrades but also to match emerging training

technologies.

For the multi-skilled soldier (MSS) concept, the training design is one of several 

aspects, including force design, military occupational specialty (MOS) design, 

implementation, and sustainment. It will be necessary to balance and make

tradeoffs between constraints with respect to training requirements, training

resources, and training technology. Appendix D, section D.1, contains a 

proposed roadmap for achieving the MSS goal. 

Information One component of the training strategy will be information and education for potential 

users of the training. 

If developers of the Master Training Strategy do not anticipate and recognize 

expectations, and manage them through information and education, widespread 

adoption of the strategy will be very difficult. 

Education can be thought of as “fear reduction,” as a way to overcome reluctance or 

resistance due to ignorance (which we mean in the nicest possible way to indicate 

absence of knowledge, not absence of intelligence). Some of the elements of this 

component of the plan will be: 

Clearly define what can (and cannot) be trained or partially trained in each type 

of training (e.g., ET, distributed training). This should be a required product

from the collaborative efforts of the system development team (SDT) and 

training design team (TDT).

Include education for trainers and leaders on the nature and value of each type 

of training, and on how to make training decisions that are based on unit 

proficiency and identification of training objectives. 

Provide training management enablers and decision-making aids to training

managers and UA leaders. 

Acknowledge and work to dispel the perceptions that constructive and virtual 

simulations are unrelated to live simulation. 

Incorporate the same cognitive strategies in the education component as you 

would in the training itself.

Resources 

and 

activities

As the Master Training Strategy is constructed, elements that require resourcing must be 

identified. 

Calling for training support activities without resourcing them is like any vision without 

funding – it is a fantasy, not a vision. 

The nature of the resourcing and funding will vary among components of the strategy,

of course, but some things to keep in mind include: 

5 
The Army needs a unified training/training management/training support 

strategy that synchronizes training and training support resources demanded by

the new operational environment and pattern of operations. 

If the burden of training will fall to the unit, there will need to be a realistic

appraisal of preparation and delivery resource requirements. 

Training development will take advantage of technologies that will automate 

many of the processes, incorporate reusable learning objects and components, 

and allow for streaming data collection to permit training evaluation. But the 

basic Instructional Systems Design/Systems Approach to Training (ISD/SAT) 

process of analyze – design – develop – implement – evaluate remains as an 

effective and comprehensive approach. 

A more effective formalized link between institution and units must be

constructed and embraced. Whether this is done through focused integrated 

process teams (IPTs), communities of practice (COPs), or some other 

collaborative mechanism, the sharing of information and plans is essential to 

effective growth. 

Incorporate activities for sustaining and updating training. 

Ensure that the strategy attends to a range of audience mixes, including: 

• Individual, team, and collective training. 

• Single echelon and multiechelon training. 

• Skill training for units equipped with Legacy (or current) systems, 

Stryker (the Army’s medium-weight system that serves as the interim

between Legacy and FCS), and FCS, separately and together. 

• Institutional, unit, and professional self-development training. 

• NET.

• Integrated training for active component (AC) and reserve component 

(RC) units and for joint and multinational coalition forces. 

Big part of

big picture:

the Training

Design 

Team (TDT) 

The training within the strategy is going to be designed and developed by – whom? 

This is not just a matter of reading proposals and reviewing prior experience – it also 

involves knowing what you want – defining standards and criteria – before you go

looking for it. 

Without a clear understanding and vision of what the TDT should do and how it should 

operate, much time will be lost in forming a plan and beginning substantive work. 

Experience over the past 25 years (for many of the experts interviewed) suggests the 

following:

There needs to be only one government decision-maker monitoring and 

directing the team. This is the person who should be able to interpret and 

enforce or modify the Statement of Work (SOW) provisions, and he/she should 

be given some autonomy and flexibility.

Integrated teams of government and contractors are necessary, working closely

and sharing information openly. 

6 
Teams working on the training should be multidisciplinary, to include 

instructional design, psychology and behavioral scientists, and multimedia 

programmers in addition to military experts. 

It may be advantageous to draw team members from multiple companies, to 

bring together more expertise and cover a greater breadth and depth of

knowledge and experience. 

The teams should not be insulated from other teams or from on-going decisionmaking. 

7 
Analysis

Repeatedly, studies and reports have stressed the importance of early and thorough analysis, 

documentation of decisions and rationale, and concurrence by all stakeholders. Five specific focus

areas for analysis were identified. 

Training

audience

When we accept that the foundation of UA training is ET (UAMBL, 2002), we need to 

also accept that there will be less direct management and facilitation of the actual 

training by observer/controllers (O/Cs) or contractors. The UA personnel will be 

largely self-directed, often without humans to help the user through the selection and 

execution process. 

If we do not have accurate definitions of the multiple training users in mind, the training 

will not be used or, if used, will not be optimally effective. 

To arrive at those accurate definitions of the users, the TDT needs to clearly and 

thoroughly identify the intended audience for each training component. 

The learning styles within target training audience need to be identified. 

Information on experience and entry skills will be needed. 

Methods of access to the training environments and capabilities must be 

documented. 

Tasks and 

standards 

Performance standards define what we want performers to be able to do. To the extent

that the standards are clear and correct, training can lead users to mastery. 

If performance standards are not identified and clearly addressed in the training design, 

the training flounders and has no purpose. 

For FCS UA tasks, the TDT must identify standards for individual and collective 

performance during the course of analysis. It is essential that stakeholders examine and 

agree on performance standards. 

Maintain crosswalks of objectives and training component content.

Ensure the stated objectives are addressed as measure of performance (MOPs) 

and measure of effectiveness (MOEs) in the measurement and feedback

mechanisms. 

Make use of concept exploration capabilities offered by simulation 

environments to capture task processes and standards. 

Conditions The Objective Force may be operating in any of a vast array of environments – jungle, 

desert, urban, mountainous – and under various combinations of conditions – 

nuclear/biological/chemical (NBC), joint, interagency, multiservice. Analysis must 

examine and document the combinations. 

Tasks are condition-specific; without explicit definition of how they are performed in 

different conditions, the developed TSPs will be inadequate. 

Specifying the task conditions is as important as specifying tasks and standards. 

In effect, a matrix of tasks x conditions is required as a definition of the training

domain. 

8 
It will be impossible to develop and expect units to use TSPs for every task

under every possible condition; ratings of uniqueness, probability, and 

criticality will also be needed. 

On the other hand, the TDT can and should continue to expand the domain 

matrix and prepare for emerging missions. As automated development tools 

become available and deployment-immediate training becomes feasible, the 

need for comprehensive TSP libraries will diminish. 

Modeling 

expert 

performance 

With ET (or any computer-delivered adaptive training), there is a great opportunity to 

build training components that will lead novices to perform tasks like experts do, at least 

at a journeyman level. 

Unless we understand expert performance and task structures, we will fail in designing 

expert training. 

The TDT needs to capture both the novice and the expert user process, 

including how they perform tasks in the system, codify the performance 

hierarchy, group tasks, and understand the task sequencing. 

Provide input to SDT so that the system itself is designed to facilitate how 

experts perform.

Analysis 

methods 

In addition to reviewing doctrinal guidance on a regular basis and incorporating the 

latest information, the TDT should select one (or more, as appropriate) tested and 

proven methods for job, task, needs, and work place analyses. Several are summarized 

in Appendix D (sections D.2 to D.5). 

Unless a deliberate method for analysis is adopted and followed, the foundations on

which to build the training will be weak and will not support growth. 

Different methods are appropriate for different situations or types of training; 

the TDT will need to study and understand the methods prior to adopting one or

elements of one. 

Certain information about tasks will always be needed: criticality, frequency, 

learning difficulty, performance latency, consequences, consequence latency,

intrinsic feedback, and amount of supervision. 

There is a divergence of opinion on how widely the analytic process can be 

distributed. While geographical distribution is becoming a less serious problem

with more sophisticated means of communication, there is still a risk of 

disconnects if the methods and products are not synchronized. 

9 
Objectives – What to Train 

We are hopeful that job and task analysis will tell us what to train (see Analysis). In advance of

having that information, prior research already points to several areas that will need to be 

addressed. They are presented below, grouped into four general classes of objectives. 

Digital skills The Army is already becoming digital, and the FCS should, by all rights, be a 

continuation and acceleration of the current trend. Training objectives should continue

to include digital skills. 

The extensive research on battle command and control in digital environments must be 

considered in designing the training, or we will find ourselves relearning old lessons. 

We know that user skills go beyond the ability to simply create digital products. They

also include the ability to employ digital products to enhance mission performance. 

Analyses have resulted in definition of four classes of digital skills, and have also

identified specific performance problems that could be addressed through training in 

these skills (see Appendix D, sections D.6 and D.7). 

Networking skills are concerned with maintaining connectivity throughout a 

mission. 

Basic operator skills are concerned with using the network to transmit and 

manage information. 

Basic user skills are tactical skills performed using digital systems, where 

digitization does not offer a distinct tactical advantage – that is, the digital 

system is no more than an alternative mechanism. 

Exploitation skills are tactical skills performed using digital systems, where 

digitization offers a distinct tactical advantage – that is, the digital system is a 

powerful tool that clearly enhances performance. 

Back-up skills Digitization and other technology solutions do not remove the requirement to train 

conventional and back-up skills. 

Back-up analysis must become part of the mindset of decision-makers from 

conceptualization of systems through implementation. 

Back-up considerations must be integrated into procurement and development 

processes, doctrine development, task analysis, and training development. 

Analysis to identify conventional and back-up tasks and to determine 

appropriate mix of conventional, back-up, and digital training required for the 

UAs employing FCS should use the methodology developed for Force XXI

analyses (summarized in Appendix D, section D.8). 

The principal factor in successful application of the methodology is for training

analysts and subject matter experts (SMEs) to be extremely knowledgeable 

about FCS operation and employment at multiple echelons. 

Further analysis should include update of existing doctrine identifying relevant 

tasks as FCS or back-up; a comprehensive needs assessment for tasks requiring

repair or workaround back-ups; risk assessment on tasks with back-up 

requirements (probability and impact). 

10 
Whether it is thought of as back-up or as the new operational environment, UAs 

will likely need to operate alongside Legacy and Stryker forces, and need to 

train for those situations. 

Individual

competencies

Task analysis on not-yet-developed systems will be challenging, to say the least (see 

Analysis). In the process, however, it will be important to not lose sight of the non-task

skills and abilities that contribute to success. 

We know enough about non-task skills and their role in expert performance to include 

them in the bank of training objectives. 

Those non-task-specific skills cut across tasks, and can be trained in the context of taskand mission specific training. Some of the prime areas include: 

Competencies: Defined as the ability to get things done, whatever they are, by

adaptively employing existing inventories of knowledge, abilities, attitudes, and 

skills, competency-based training is an alternative (or complement) to 

traditional task-based training. Competency-based training should enable UA 

leaders to transition rapidly from one kind of mission to another. 

Stress resistance: Researchers have found that, under high stress, leaders use 

their experience but misuse their intelligence; under low stress, they use their

intelligence but misuse their experience. Experience in stressful conditions

lowers vulnerability to stress and increases task structure and therefore increases

the leader's situational control. Such leadership experience improves selfefficacy, and also interpersonal relations with subordinates. 

Situational awareness: Leaders and soldiers must be trained to trust the

situational awareness support system integrated in FCS but to avoid overreliance on it. Instead, they must integrate all sources and forms of information 

and use that information to make better decisions quickly, not deferring

decisions while waiting for more perfect situational awareness. 

Information management: Cognitive strategies for information management 

such as techniques to filter, sort or group, prioritize and interpret incoming

information can be trained, and leaders must have practice in integrating the 

information received in order to synchronize actions. They will need to define 

information requirements, apply appropriate information filters, and formulate 

and execute information search strategies. 

Learning how to learn: The multi-skilled soldiers of the UA must be 

perpetually learning, and must be trained and motivated to deepen and broaden 

the skills they acquire in initial entry training (IET), largely through selfeducation by means of distributed training and distance learning (DL). 

Adaptive thinking: Training and situational practice in adaptive thinking, 

coupled with identification and training of minimum essential tasks, can extend 

the benefits of structured training. Soldiers will need to be able to identify and 

adjust to information technology requirements and quickly master individual 

and collective learning requirements. 

System management: Leaders and staffs must thoroughly understand how 

robotic forces work and what rules they follow in order to effectively employ 

11 
 them. They must acquire simple operational knowledge as well as more 

advanced understanding of how to exploit capabilities and knowledge, and must 

master conceptual as well as mechanical aspects of command and control (C2). 

Staff and 

team skills 

In addition to individual competencies, we also have the ability to work on staff and 

team skills, effectively multiplying the utility of individual proficiency.

If training for staffs and teams is no more than individual training in groups, the result

will be proficiency of a group of independently working individuals, rather than team 

proficiency. 

Analysis and research have already identified a set of staff skills that can be trained and 

have also provided information on how they should be trained:

Monitor, process, analyze/evaluate, communicate, coordinate, integrate,

recommend, disseminate, synchronize – these skills can be trained in small staff 

groups using scenario-based training (one such use is summarized in section 

D.9, Appendix D). 

Shared mental models: Leaders and their staffs need to learn how to share their 

knowledge and understanding quickly in order to form a common vision of the 

situation, work as a group on making decisions, and understand, individually

and collectively, the actions required. 

Training of stability operations and support operations (SASO) missions should 

consist primarily of leader training at brigade and higher, while lower echelons 

should continue to focus on warfighting operations. The resulting discipline,

team building, and familiarization with chain of command will strengthen 

small-unit capability on all types of missions. 

There are strong recommendations for how leaders should acquire and develop 

proficiency in leadership skills: 

• Increase experiential learning. 

• Ensure stress (i.e., battle friction) in advanced learning.

• Use chain-of-command training. 

• Develop team cohesion and coherence. 

• Fine-tune decision-making processes. 

• Establish common tactical scenarios. 

12 
Training Design

Training has a goal – to improve performance – and a target audience – the trainee, the learner. 

During analysis, the training audience was defined in terms of experience, prior learning, styles,

and job situation, among other things (see Analysis). That information, plus results of research on 

learning and human factors, guides training design decisions. The observations and lessons learned

are presented in the context of seven topics. 

Training

focus 

While training for the UAs is still at the start of the overall analysis process, it is critical

that the analysis include not only task analysis, but also a clear definition of the focus 

for each training component (as well as a vision of how the components themselves fit 

together, see Big Picture). 

If the focus for each component is not stated, understood, and addressed in design, the 

resulting training will be both incomplete and duplicative, and users will be unable to

determine what component to conduct. 

Throughout the design process for the system and the training, the TDT needs to prepare 

and publish a clear and thorough statement of the focus of each training component. 

Identify the full range of tasks covered for each component, based on a learning

hierarchy of minimum essential skills through high-level exploitation of

capabilities and organizations. Multiple tasks and training objectives may be 

addressed in a single component, if it increases the efficiency of the training. 

Ensure the purpose of the training is clearly understood by training managers, 

training developers, training audience members, and leaders. 

Use the information on training objectives to design and conduct training

evaluations (see Evaluation). 

Learner 

styles 

The effectiveness of the training will depend on how well it addresses learner needs and 

matches how learners learn. Although there are a number of relatively universal 

learning principles, it has been shown that different individuals learn differently, and 

groups learn differently from individuals. 

If training design decisions do not both address common learning principles and also 

account for a substantial portion of different learning styles, they will be inappropriate 

for a substantial portion of learners. 

There is a significant body of research on learning styles, far too much to cover here. 

Some of the widely appropriate principles for design include: 

Adopt a model where the learner constructs the new knowledge through 

interactions; avoid passive reception of data. 

Given an analysis of the training audience (see Audience), build on their prior 

knowledge to introduce new concepts or procedures. 

Make content relevant and meaningful. Generic situations that are deliberately

not relevant to the learner environment are rarely effective. 

Vary the scenario situations, presentations, and demands to increase flexibility

and keep learner interest up. Scenario nesting should be kept reasonable to

minimize the risk of leaving some participants behind. Ensure realism at all 

levels, but also ensure active participation opportunities for all participants. 

13 
Give the learner and learner teams choices and autonomy over the pace and 

sequence of the instruction, when feasible. 

Incorporate adaptable content for different prior skill sets. The embedded 

system should have the capability to detect learner needs and provide the 

appropriate presentation. 

Problem-solving strategies may slow acquisition but are effective in supporting

skills that are adaptable to new situations. 

Include deliberate practice to identify shortcomings in staff processes and 

conduct staff integration training. 

Provide repeated practice, performance feedback, and increasingly difficult and 

complex situations. 

Use short tactical vignettes to provide standardized iterative training. 

A behaviorist approach is more useful for basic skill learning, while a 

constructivist approach should be used as learners are refining their skills and 

focusing on application.

When there are instructors, they can effectively act as co-learners by designing

tasks wherein learners solve real world problems, reflect on skills used to

manage their own learning, address misconceptions in their thinking, categorize

problems around themes and concepts, and generally take ownership for their 

own learning. 

As the system itself is being designed, the TDT must provide input to the SDT 

so that the information on learning styles is addressed, allowing the training to 

leverage system capabilities. 

Skill

acquisition 

and retention 

Design decisions should include training principles and approaches that support swift 

skill acquisition, longer retention, and more efficient relearning. 

Many of the tasks and skills will be new, and training approaches that do not reinforce 

learned skills in order to resist decay would be a waste of resources. 

There are specific training principles and approaches that contribute to more efficient 

skill acquisition, longer retention, and faster re-acquisition. For example: 

Especially for more basic operator tasks and other procedural tasks, covering

the underlying principles as well as operational steps yields more flexible 

knowledge that is more resistant to skill decay. 

Combining instructional principles has a multiplicative effect. 

In addition to step-by-step basic training, give skills training and drills, with 

feedback opportunities, to build automaticity. Maximize initial learning to 

require practice beyond meeting of minimal standards.

Optimize scheduling of refresher training based on prediction of the rate of skill 

decay. 

Optimize effectiveness of refresher training by using technologies (e.g., video 

previews) to reestablish context. 

Test the skills being trained.

14 
Provide spaced rather than massed practice. 

Use task-oriented context-relevant training. 

Use peer tutoring. 

Expert skill 

training 

Most formal training will aim to develop journeyman skill levels, but the long-term

training goal is to produce hyper-proficient expert individuals and teams who can fully

exploit technology.

If the training is not designed to model and shape expert performance, based on the 

processes of expert performers, we will have pedantic training and minimally competent 

users.

Continually, show the bridge between the expert process and the learner’s current state. 

Design the training to mimic expert performance. 

Group tasks for training to encourage the grouping that makes sense to experts 

in performance. 

Address minimum essential skills with drills and procedural training; use a 

performance objectives approach for higher-level skills. This will allow

soldiers to learn-transfer-adapt to achieve hyper-proficiency. A method for 

developing performance objectives training materials is shown in Appendix D, 

section D.10. 

Identify and highlight similarities and differences between the learner’s current 

proficiency and expert performance to facilitate learning, retention, and

adaptability. 

Training

sequence 

The training has to be done in some sequence – it can’t all be done simultaneously. 

If some sequences are better than others, designers must incorporate the optimal 

sequences in the overall training plan, rather than leaving it totally random. 

Provide users with a general understanding of system capabilities and train-up 

expectations during the initial training. 

Incorporate progressive skill building approach by attention to enabling and 

terminal skills. 

Provide multiple entry points and paths to ensure adequate practice while 

crediting prior experience or readiness.

Allow time and provide support for drills to facilitate skill consolidation, 

automaticity, and the benefits of cumulative practice. 

Large-scale v. 

small-scale 

exercises 

During design, there will be decisions on whether to create large-scale or small-scale

exercises. This is not an “either/or” situation – it depends on the training objectives. 

Building only large-scale or only small-scale exercises for training ignores the reality 

that different training structures are suited for different purposes. 

Small-scale exercises are less resource intensive to implement; large-scale exercises can 

give trainers more bang for the buck. Designers should use both approaches, as well as

other models (small group, dyad/triad/quadad) as appropriate. Just as training builds 

15 
from enabling objectives to terminal objectives, exercise scale should build as part of 

the overall training strategy (see Big Picture). 

Provide small-scale focused training as well as multiechelon training with full

inclusion of assets. 

The difficult but necessary effort is to keep smaller scale exercises focused on a 

small group and a selected, limited event. Without this focus, the exercise

grows into a “little war” and the purpose of having easily implemented focused 

exercises is defeated

Large-scale exercises may be cost-effective if they are reserved for use after 

participants have learned their own jobs. 

Provide training components and guidance on sequencing to allow UAs to work

from individual skills to small groups to staffs and leaders, rolling in 

subordinate staffs and subordinate units along the way. 

Incorporate multiechelon programs, one-up/one-down. 

Reserve “a-way” approaches for training at lower echelons. 

Consider and include a variety of approaches to allow for a more human-centric

approach. 

Training

support

packages

Whether training is embedded, computer- or simulation-supported outside the FCS, or 

platform, TSPs will be required. 

The technology can take us a long way on the road to training that is delivered

seamlessly, transparently, and effortlessly – but only if the TSPs, the learning 

management system (LMS), and the unit training management system (UTMS) are

designed in concert to do that. 

Allocate resources for designing and developing resilient and robust TSPs that

cannot only tolerate tailoring but also encourage it. 

Ensure the TSP elements (e.g., scenario, training audience materials) are 

tailored to the users and purpose, rather than to a predefined structure. 

Use structured training, which calls for a focus on specified training objectives, 

opportunities for practice, and performance feedback on those objectives. 

All of the TSP contents – including information on how to select the appropriate 

training, instructions on how to access and implement the training, and content 

– must be immediately understandable and usable by all potential users. 

Provide tools for use of self-assessment techniques that link user observations to 

the training objectives. 

Don’t assume that O/Cs can reliably provide feedback without some guidance 

from developers. Section D.11 in Appendix D describes train-the-trainer 

materials to include in a TSP. 

Take advantage of materials and instructional techniques that reside in existing, 

proven, structured training products, and convert them to new needs as 

appropriate (see Development). 

16 
Disseminate existing TSPs to all potential users, and also to potential

developers. Virtual dissemination, via a centralized accessible repository, must 

be supported by active and aggressive information pushes. 

Knowing the accessibility obstacles (e.g., bandwidth), make sure that the 

intended users will be able to access the training. 

17 
Training Development 

Training development begins with analysis and design, discussed in separate sections. It contains

continual evaluation activities, which are also covered in a separate section. That said, there 

remain many observations and lessons learned related to the portions of development that involve 

the design and actual construction of the training materials or infrastructure. They are grouped 

under three general headings. 

Use of user 

and expert 

input

However careful we are in ensuring that training is ideally suited to the user, at some

point we actually need to talk to users – both those who will be using the training and 

those who will be their leaders. Their input and reactions are essential to ensuring that

the products will be accepted, used, and useful. 

If we fail to involve users and experts in planning and decision-making, we operate in a 

vacuum and have little hope of producing training that will be adopted. 

Obtain early input to task analysis, training designs, and training products from diverse 

and representative users and innovative thinkers. 

Analyze and evaluate breadth and depth of experience before using the input. 

Unexamined input is no better than no input at all. Look for logical thinking,

foundations in reality, linkages to other accepted models, and hard data. 

Get users’ input at multiple points in the process. The advantage of using the 

same users and experts all along the way is that they will be familiar with prior 

thinking and decisions. However, there is advantage in tapping fresh minds, to 

ensure that the decisions are critically reviewed and the design process 

unbounded. 

Input must reflect not only how things are done now, but also visions of how the

future will look. 

Remember that users will also include the people who have to manage and 

sustain the training, as well as facilitators (O/Cs and the like) and 

contractor/trainers. Using their input will ensure they will be able to support the 

training. 

Development

approaches 

A deliberate and methodical approach to development can be swift and flexible while 

still ensuring attention to detail. The TDT should use the ISD model or one of the many

variations that have been documented (summarized in Appendix D, sections D.12

through D.16; also see sections D.2 through D.4). 

Beginning training development without a plan and a method guarantees a haphazard 

and minimally useful product. 

In the case of FCS, and in fact in any case of simultaneous system and training

development, the deliberateness of the ISD model may appear cumbersome and 

unworkable; in fact, it is even more critical that the approach be deliberate, in order to

ensure the linkage between current status and decisions. Be prepared: There is a clear 

need to incorporate features of rapid prototyping and flexible component-based 

construction to facilitate continual updates.

Use existing products and a conversion approach when the changes in systems, 

organizations, or doctrine are variations on existing conditions. 

18 
For any such conversion effort, document the changes and the rationale so that 

the next wave of conversions can be executed rapidly and accurately. Be aware 

that conversion is not a short cut to full development. 

Likewise, use a conversion approach to keep up with changes in delivery modes 

and training technologies. 

However, do not be wedded to conversion, forsaking development of innovative 

training solutions. 

Use the core exercise set identification process in the case of new organizations 

or missions. This process is shown in Appendix D, section D.15. 

In general, for training that is going to be linked vertically (i.e., mutually

supporting across echelons in terms of scenarios or tasks), try to design down 

and develop up. That is, design beginning with the higher echelon, but then 

build the training and the TSPs, based on that design, from the lower echelon. 

Exercises and training elements will need a built-in mechanism for 

accommodating modifications. By means of reusable and sharable components

within flexible instructional programs and TSPs, automated tools to conjoin 

those components into remixed instructional programs and TSPs, and larger 

scale automated tools to configure the elements of the training system into a 

remastered system, training developers and units may have a chance of staying

abreast of technology and other developments. 

Developer 

tools 

In the long run, it will be more efficient to create and distribute tools for modifying or 

converting existing training products than to continue to create new training. Giving

thought and resources to developing those tools right now will pay off down the road. 

Without tools to facilitate and simplify the efforts of training developers, we will be

continuously spending time looking up methods and existing products, rather than being 

able to leverage those items, and we will continue to deny users the opportunity to tailor 

their training. 

Tool development should parallel training development. Any process that the TDT

finds itself doing more than once should be a candidate for tooling.

During the early training and tool development process, the TDT should use 

scripting and programming methods that support flexible modifications and thus

are amenable to tooling. 

The UAs will need tools to modify training for specific circumstances and 

needs. 

Training managers will need tools to modify measures and feedback and 

generate appropriate evaluation and feedback materials.

19 
Training Implementation 

Methods for implementing training will generally be part and parcel of the training component and 

the associated TSP. But efficient and effective training will need to be supported by a well-thoughtout training infrastructure, embodying strategies and plans as well as a range of resources and 

support mechanisms. Instructors, surge teams (personnel who assist units with first time

implementation of more complex exercises), O/Cs, and other experts will continue to be an integral

part of the implementation plan. Given a continuing structured program of support, from

education and information through exercise preparation to conduct of the exercise and feedback 

facilitation, units are able to participate in a wide range of training elements. Similarly, instructors 

will be available for students in distributed individual instruction situations. The lessons learned

concerning the implementation of training are presented within eight topic areas. 

Access issues Successful and effective implementation of high-tech training, such as embedded or 

simulation-based training, requires that the target users have all of the equipment and 

TSP materials needed to conduct the training. 

The best TSP in the world is useless if users are unable to use it. 

The requirements are different for different types of training: 

Distance learning approaches require up-to-date computer hardware, software, 

and high-speed Internet access with sufficient bandwidth for the intended TSP.

Embedded training requires users to be in their vehicles or on their network

stations, unless there are mechanisms for replicating the training in other 

environments. 

Virtual and constructive simulations at fixed sites require some amount of colocation, even if several simulation sites are linked with distributed systems. 

Until automated measurement systems are sufficiently mature to independently

handle all of the analysis, feedback, and coaching duties, O/Cs will be needed 

(but see Analysts, below). 

Access requirements include making time available for using the TSP, and 

being able to fence that time off from other requirements. 

Multiple 

delivery 

modes 

If access problems are associated with availability of training equipment, a partial 

solution is to provide a range of options for training implementation. 

If each component of the training program for FCS is available via just one gate, a 

significant number of the intended users may be shut out. 

Include, in addition to ET, other modes of delivery: 

Distance learning, via Internet or computer-based. There is a high rating of 

student satisfaction with these methods, and training effectiveness has been 

demonstrated for certain applications. 

Delivery via hand held systems (e.g., personal digital assistant [PDA]). These 

have also been rated as satisfactory by students, especially for declarative and 

procedural tasks. Greatest effectiveness comes with voluntary use. 

Use of a variety of training aids, devices, simulators, and simulations (TADSS) 

in live training allows users to get the value of added training time and repeated 

practice. 

20 
Pay attention to how TSPs are delivered in order to maximize use. 

Analysts and 

O/Cs 

The access problem may concern availability of O/Cs. Using O/Cs with expert military

credentials and comprehensive TSP guidance can be the key to effective collective

training, but it can also be very costly and difficult to staff. 

Given that expert O/Cs are needed but too costly and rare to support training 

requirements, training designers may need to “do more with less.” 

One potential solution (described in slightly more detail in Appendix D, sections D.17 

and D.18) links two approaches in order to do more with less. 

Use Combat Training Center (CTC) O/Cs as centralized core groups of

analysts, linked electronically to the UA in training. This approach increases 

the ratio of units trained to analysts, and bridges the gap from homestation to 

CTC by using the same analysts. 

Provide instrumentation and automated data collection and analysis for the 

analyst. 

Whether centralized analyst approach is used or the more common on-site O/C/analyst 

approach, several elements are critical: 

O/C expertise and experience, both as a warfighter and as a trainer. 

Support, in terms of TSP guidance and time for adequate preparation. 

Instruction on what is available and how to use it. 

Flexible 

training 

support

Recent experience and observation have indicated that, for individual and collective 

task-oriented training, a structured training approach is effective. Experience also

shows, however, that the users will almost always want to change the training – to 

better address their training need or simply to put their own thumbprint on the exercise. 

Fortunately, structured training can be very flexible. 

Rejecting structured training approaches as rigid or inflexible is shortsighted and illadvised, resulting in loss of a powerful and adaptable tool for skill acquisition. 

The strength of structured training is in providing events that ensure the unit will have

to practice selected tasks, regardless of their battle plan. 

Training managers need to be educated, by means of the TSP itself, on how to

modify the TSP without losing the advantages of structure. 

The structure allows and supports the possibility that users can themselves 

develop training exercises along with the TSP materials. 

Managing 

TSPs

As UAs modify existing TSPs or even create their own training, there will be 

apprehensions that the training is not doctrinally correct or is not employing good 

training techniques. 

No matter how good the training development tools are, unit-developed training that is

not reviewed and approved will be regarded with suspicion and may not be used again. 

Expert tools can help to ensure appropriate products (see Development), and 

repositories of approved products can be constructed. But the perceived need for 

centralized review will still exist (particularly, of course, if the expert tools are not

21 
provided to or used by units). The research in this area provides recommendations but 

no proven solutions. 

Even with a repository, the Army will need a mechanism for proponent review

and approval of TSPs (both proponent- and user-developed) before storing

them in the repository. 

Stored TSPs will require maintenance to preserve doctrinal currency. 

The repository for TSPs needs to be centralized but still accessible and secure. 

A virtual private network (VPN) could provide a high level of security while 

satisfying the Army Training Information Architecture (ATIA) principle that

data be logically centralized, but physically dispersed.

Units will need ongoing education and information about what is available and 

how to get it. 

Simulation 

environments

Virtual and constructive simulations will remain useful for some time. However, the 

TDT needs to make intelligent choices about when and how to use them. 

Simulations – whether fixed site, distributed, or embedded – are not the universal 

training solution (the silver bullet) but they do have some specific utility. 

A good simulation in a well-executed environment offers control, realism,

instrumentation for reliable measurement and replay, efficiency, and effectiveness. 

Everyone wants efficiency and effectiveness, but control, realism, and instrumentation 

requirements are relative. 

Control, or standardization, permits carefully designed replays of scenarios 

with the same or different conditions, allowing deliberate practice and skill 

building. For simple instruction or for tasks where job aiding is prevalent, 

multiple practice opportunities may be superfluous. 

Realism is required to permit users to see outcomes of decisions and actions. If

the focus for the training will be on the decision rather than a dynamic 

outcome, simulation environments are not necessary. If the outcome is a long

time in developing, the simulation may not be a good use of time. 

Likewise, realistic simulation systems can represent systems or troops for 

tactical realism and cues to participants, but should only be used if the realism

is essential to the training objective. 

If there is nothing to measure in the course of an exercise, that is, if the training

is process- rather than product-oriented, then instructional features such as

measurement and replay will be of scant utility.

Until simulation environments provide agents for all non-live roles, developers 

will need to allow for role-players, especially in SASO exercises. More than 

move and shoot training will be needed, and simulation environments are 

currently less adequate outside of move and shoot. 

Desktop or other types of reconfigurables allow wider participation for 

somewhat less cost and space requirements. They are also vastly useful for 

concept exploration. 

Synthetic theatres of war, with linked simulation environments, have distinct

training potential for certain tasks (e.g., battle damage assessment, indirect fire

22 
support, commander’s recon, battle monitoring, recon tracking, and land

management). 

To be useful, the simulation environments should also have integrated tools 

such as automated message generation, eavesdropping, exercise flags, and

multiple observation perspectives. 

Sustain and 

manage

content 

Once training is available for implementation, it will still require sustainment – to add 

options, maintain currency, incorporate new technologies, and so on. 

Marketable shelf life of TSPs, without sustainment, can be as short as 6 months;

without sustainment, it will be perceived as irrelevant and outdated. 

If TSPs are not updated frequently, users will perceive that no one is attending to the 

training. If updates are too frequent, however, users will be confused by the 

appearance of multiple versions. The change may be purely cosmetic, such as

changing the “last updated” date. More likely, the change will include new training

opportunities. Not only does the sustainment need to happen, but it needs to be 

systematic, controlled, and communicated. 

The Army must design and implement mechanisms for fielding, sustaining, and 

updating programs, just as it does for operational systems and training devices. 

The Army should designate how and by whom the training is managed and 

sustained (currency). 

As the TSPs are built, they should be tested to ensure they are manageable by

the designated managers, with a centrally managed universal training support 

system. 

Embeddedness should permit training to be tethered to its source/maintainer, 

who must also ensure that alternate versions are changed in concert. 

Users will tolerate irregularities for a short time, but they should be fixed in a

timely fashion. 

Program resources must be allocated for feedback, reviews, and updates. 

It will be necessary to stabilize systems for some periods of time – UAs cannot 

train on software that is always changing. Instead, the Army should institute 

block changes, where the system stays the same for a period of time, then all

upgrades are done in a 2-week period and everyone gets them. 

Training

management

system 

As individuals and UAs participate in various training activities, a centralized tracking

and management system will be needed. 

If the training gets used but no one knows what has been done, it will be impossible to

plan additional training or training development. 

The training management system will serve at least two functions: 

Managing training implementation: The system should keep track of all the 

pieces, what the soldier or UA has done, how well, what’s next, and links 

among repositories for records. 

Managing training development: Data on the training that is being used will 

inform training developers on what implementation models are succeeding and 

23 
what TSPs are being ignored. The information should be useful in maintaining

the overall training program for FCS UAs. 

24 
Evaluation 

The concept of evaluation has a plethora of applications in training and system design. “Test and 

evaluation” is a familiar requirement for systems developers, and the requirement is rarely waived. 

By the same token, evaluation of training should be required. There are a number of facets to 

evaluation in the context of training, here presented under two general headings. 

Formative

evaluation 

Formative evaluation involves the testing and reviewing that is done during the 

development process, which then leads to revisions in the training prior to fielding for 

operational use. It means doing our best to ensure that the training is accurate, 

understandable, useful, accessible, and effective – before putting it in the hands of users. 

If we allow training to be fielded for use without formative evaluation, we have no idea 

whether the training can be understood or used by the target audience – the real

customer. 

The TDT needs a full battery of formative evaluation tools and needs to understand

when to use them. 

Have training materials reviewed by training experts as well as military content 

experts. 

Try out the training with both experts and novices who represent the training

audience. 

Conduct rigorous system trials and user trials, but maintain flexibility in 

introducing modifications on the fly; document modifications and reasons. 

In pilot testing, ensure that users can understand the implementation plan for the 

training and seem enthusiastically willing to conduct the training. If not, use the

collected data to identify obstacles, and remove them. 

In pilot testing, ensure that users are able to perform the training objectives to

standard. If not, use the collected data to redesign the training, refine the 

materials, or make corrections so that objectives will be achieved. 

Take advantage of the existing methods for evaluating the training support 

materials (summarized in Appendix D.19, section D.19). 

Get input from multiple sources and weight the input in accordance with your 

judgment of the credibility of the user. Do not base decisions on data from only

one segment of the domain.

Summative 

evaluation 

Summative evaluation is usually conducted as the training is fielded for operational use. 

More importantly, it refers to those activities and objectives related to the completion of 

the training: whether the users are able to use their learning on the job and whether the 

performance of the organization overall is improved because of the training provided to 

individuals or groups within the organization. 

Without summative evaluations, there is no evidence of return on investment for the 

training. 

Because major decisions could be made on the basis of the summative evaluation, 

planning and execution should be rigorous and thorough. 

Identify the evaluation goals and issues and select reliable measures to address 

those areas. 

25 
Use the measures of performance (MOP) and measure of effectiveness (MOE) 

(see Performance Measurement) also for evaluation of the training programs

themselves. 

Maintain scientific rigor in data collection and analysis; flexibility and 

modifications during the evaluation are acceptable and appropriate in the

formative evaluation but not in summative evaluation. 

Take advantage of the methods for training effectiveness analysis (summarized 

in Appendix D, section D.20). 

26 
Performance Measurement

In order for training to lead to improved performance or certified proficiency, there must be

evaluative feedback; in order to provide that feedback, there must be measurement. For FCS, 

automated measures are expected to be the most common type, although other approaches may 

also be needed. One important observation is that measurement needs to address what should be 

measured, not only what can be measured. Prior research yields performance measurement 

imperatives in four areas.

Measuring 

performance 

standards 

Performance standards are what we measure performance against; to tell us whether or

not the user has gained mastery of the training content. It is not sufficient to state the 

standards – they also have to be measurable. To the extent that the standards are clear 

and correct, and performance is measured during training and assessed against those 

standards, we can be sure that the training is appropriate and that the FCS leaders,

soldiers, and UAs have achieved their training objectives. 

If performance standards are not identified, stated, and measured, training becomes 

“check-the-block” and gives us no assurance of competency, much less proficiency. 

For FCS and UA tasks, standards for individual and collective performance will have

been identified during the course of analysis and will be stated as the training objectives 

(see Analysis and Design). The TDT must construct metrics for assessing achievement 

of the objectives (see Evaluation).

Ensure MOPs and MOEs are clear and can be reliably assessed. 

Ensure the MOPs and MOEs are valid measures of the performance that they

are meant to address. 

Ensure the MOPs and MOEs are acceptable to users and training managers and 

appropriate for the performance standards they address.

Automated

measurement 

protocols 

With embedded and other computer-based training, automated and objective 

measurement of activities is a given. Translating the raw data into useful feedback

presentations or summary indicators of proficiency is a necessary next step. 

If automated measures are not developed beyond raw data to evaluative feedback and 

understandable presentations that are linked to the established standards, the data will

be clutter and of no use to users or training managers.

The TDT needs to attend to analysis and feedback in consideration of automated 

measures as much as to data collection. Take full advantage of the system’s capability

to get the data, analyze it to produce summary measures, show the results, and make

assessments of the quality of the performance. 

Establish the link between MOPs and MOEs to tell why things happened, not 

only what happened. 

Measures/data must be clearly linked to the battlefield conditions and to the 

performer, and must address human and system behaviors related to C2 

performance. For example, measurement of four objectives used for staff 

training – monitor, process, analyze, and communicate – should be supported by

the system. 

Evaluate and validate the measures and relationships, and work with individuals

and units to ensure the feedback is effective. 

27 
The system must provide not only data recording, but also reduction, 

management, and analysis capabilities. 

Ensure automated measures used in training are closely aligned with any

performance support systems (see Performance Support Systems).

Early and continual communication with the SDT will be essential to ensuring

that the FCS itself will support the automated measuring and analysis designed. 

Nonautomated

protocols 

Even though intelligent agent technologies are evolving rapidly, they are not yet ready

to replace expert observation and interpretation of performance. Automated measures 

are not the only answer. 

If we discontinue use of expert observers, analysts, and coaches in training, we will lose 

input that is not available any other way, and will also lose the opportunity to allow 

experts to analyze their own behaviors. 

Don’t hesitate to use traditional methods in addition to automated measurement to

ensure broad coverage of performance assessment. 

Trained observers are currently the best way to put automated measures outputs 

into a meaningful context, but they will require training and guidance to ensure

the reliability and validity of their output. 

For more complex decision making tasks, using one-on-one active coaching or 

mentoring is effective. 

Self-reports are also valid: Information about a unit’s progress can be gained by

simply asking questions of unit leaders and members. Other techniques include 

think aloud verbal protocols, query and probe techniques. Both objective and 

subjective measures have been developed and investigated. 

Keep an eye on the physical environments in which the UA soldiers will operate 

– it may require more space than is available to have observers at the user’s

shoulder. Look for ways to employ surrogate measures – those that indicate 

performance – rather than requiring the observer to have eyes on the user.

Don’t rely on just one type of data – think broadly about collecting information 

besides button pushes (e.g., audio and video recording, eye-tracking). 

Skill

indicators The body of work with digital skill indicators provides examples of MOPs and MOEs 

for individual and collective measures. 

Training developers can start from scratch and reinvent methods for identifying 

appropriate MOPs and MOEs, or they can work smart and consult the collected 

information amassed on digital skill definition. 

The real value of considering the findings from digital work is in how the lessons were 

gathered, analyzed, and used, as much as what the lessons were. 

A scaleable, searchable repository for collecting and cataloging lessons should 

be designed and set up early, before experience with FCS begins to yield 

information, so that information is adequately catalogued. 

28 
Topics for the repository can begin with the topics that came out of the digital 

experience. Examination of these findings and how they were garnered 

(summarized in Appendix D, sections D.21 and D.22) can help lead the TDT to 

the appropriate MOPs and MOEs for UAs. 

Experiments in simulation using early conceptual models of FCS capabilities 

should be targeted for gathering and formally cataloguing lessons learned. 

Deliberate knowledge elicitation methods should be employed to capture 

experience as FCS is used in initial testing and experimentation. 

29 
Feedback 

Feedback is the essential element to allow training to produce change in performance. It is more 

than a scorecard; rather, it is the essential ingredient for planning, productive use of lessons 

learned, and learning by sharing. Its purpose is to facilitate the learning process for the individual 

or group. Research observations and lessons learned concerning feedback fall into three categories.

Focused

feedback 

Feedback should be tailored to focus on key selected performance issues, not on an

entire operation. This type of performance feedback will be difficult, if not impossible, 

in very large-scale exercises. 

If the feedback is expected to cover the entire gamut of performances exhibited in an 

exercise, it will overwhelm the learner and result in no learning at all. Conversely, If

feedback is properly held to focus on a selected set of objectives, limited or scaled 

exercises will be sufficient; large-scale exercises will embody lots of wasted effort and 

draw the focus to events rather than performance processes. 

During analysis, the TDT identified the training objectives, and designed MOPs and 

MOEs to address them. Now, during conduct of the training, the feedback should be 

focused on the specific objectives for which the training was designed. 

Scaled exercises, with 5-8 objectives for a 2-4 hour block of performance, can 

be very effective in providing iterative practice and improvement.

Feedback should be learner-centric and go beyond discussion of what was 

accomplished and the processes used. It should go on to a comparison with 

what the users intended to accomplish and the processes they intended to use, 

determining if they need to make changes to their processes, and describing

how they intend to accommodate those changes in future operations.

The structure supporting this kind of after action review, as well as the 

mechanisms for collecting the data and presenting the analyzed data as 

feedback, should be a standard part of any training component.

Large-scale exercises are costly in many respects and frequently do not have

training value for a significant portion of the participants. Scale the exercises to

ensure that all participants are engaged.

Large-scale exercises should be conducted only when individuals and UAs have

been certified as proficient by means of scaled exercises and training.

Use agent technologies to identify performance weaknesses and to provide 

automated coaching and feedback.

Use the MOPs and MOEs to direct users to additional training. 

The TDT will also need to develop measures of performance, so that leaders can

tell whether users have in fact met the standards and, if not, what they should do

to further develop proficiency.

For non-procedural tasks, using a coach or mentor for each individual should 

supplement, or even replace, the group-style after action review. 

Timely

feedback 

With ET and computer-assisted training, the timing of feedback can be adjusted for 

maximum effectiveness. Sometimes, immediate corrective feedback is appropriate, 

30 
while at other times it makes better sense to allow the learner to explore and evaluate 

solutions with only intrinsic feedback. 

If we do not adapt the timing of feedback to the training need, we will either fail to

allow the learner to learn, or fail to correct incorrect performance. 

The nature of the performance and the proficiency level of the training participants will

be among the prime determinants of feedback timing.

For novice participants, initial learning will be more effective with more 

frequent and directive feedback that shapes behavior along expert lines. 

For individuals and groups who have mastered the basics, exploration learning

can be more effective, as long as intrinsic feedback occurs as appropriate. 

For tasks where there is no intrinsic feedback, or where that feedback can be 

misleading (e.g., not totally a result of the performance), extrinsic feedback

should be provided. 

Withholding feedback so users learn from their mistakes should only occur in

training if there is no feedback, intrinsic or extrinsic, in operational settings. 

During exercises, short informal performance feedback sessions can help

learners to consolidate lessons learned and recognize cues to appropriate 

performance. Whether this is a programming chore or an O/C chore, tools will

be needed to speed up the process. 

Performance feedback after an exercise is completed should occur with as little

elapsed time as possible. Again, whether this is a programming chore or an O/C 

chore, tools to speed up the process will be needed. 

Real-time feedback should be a selectable feature. 

31 
Tailored

feedback 

Tailorable feedback will be a useful feature. With the right tools, the feedback can be 

customized to show exactly why particular things happened, to focus on one slice of the 

training audience, or (most simply) to flag events and viewpoints to be shown later. 

There is no reason to present feedback as “one size fits all” – the technology will allow 

customization, and our knowledge of individual differences and cognitive load enables 

us to design appropriate options. 

Different users have different preferences for how the feedback is presented, and 

allowing them to customize (within limits) increases the probability that they will pay

attention to – and use – the information. 

Make it easy for commanders, users, and trainers to customize feedback by

providing intuitively understandable tools, performance support on how to use 

the tools, filters, masks, templates, and wizards to assist them in finding the 

appropriate information. 

The tools should enable them to 

• customize feedback formats to match personal preferences. 

• look at samples of various feedback outputs. 

• adapt amount of feedback and presentation to training needs 

• flag events and viewpoints. 

Designate tools appropriate for before-during-after the exercise. 

Incorporate safeguards to prevent displays of misleading information or 

presentation. 

32 
Performance Support Systems 

By definition, a performance support system (PSS) is a system that provides on-demand access to 

information, guidance, advice and assistance, training, and tools to enable high-level job 

performance with a minimum of support from other people. For electronic PSS, we add the

qualifier that it is an integrated computer-based system. The primary function is to provide 

information or assistance to soldiers and teams in performing a broad range of tasks quickly and 

completely while minimizing sensory, cognitive, and physical demands. Research observations on 

this topic focus on four specific uses of performance support capabilities. 

Information 

management

and support

One of the most obvious uses of a PSS is to simplify the information flow. 

The users needs have to be at the center of the PSS design, so that the tools and reports

actually support the user. 

The PSS, embedded as an element of the FCS software, will have several roles in

information management to assist the user: 

Reach: In some cases, information should be readily available on user demand 

with an intuitive and reliable search engine. Other information should be 

offered without any overt user action. Certain information should be

automatically posted to a central repository. 

Source fusion: The PSS should be designed to examine, analyze, and assess 

information from a variety of sources, in order to report highly reliable and 

accurate information. 

Knowledge management: Users should have the opportunity and support (in 

terms of agents and tools) for constructing and sharing knowledge, both in 

training and in operations. 

Some Objective Force performance needs are not fully addressable through 

training, requiring some other form of support to enable performance. For 

example, a prompt for a next step in performing a task may be far more 

effective and efficient than remedial training. 

Feedback The PSS should seamlessly support both training and performance. The SDT, as it

designs software to capture and present data for training, must also explore the utility of 

feedback during rehearsals, concept exploration experimentation, and actual operations. 

Although the same tools and feedback that are available for training can also be 

available for non-training situations, users must retain the option to tailor the feedback 

to match their roles, personal preferences, and the situation. 

Operational presentations of feedback will not be the same as training presentations, 

even though they are derived from the same data collection protocols. 

Ensure operational feedback presentations are informative, not instructional. 

The tools for customizing information and feedback presentations must be 

intuitively understandable (see Feedback). 

The capability to sharply reduce or even suppress feedback must be an option. 

Users must be able to turn “training opportunities” off during operations. 

33 
Operator

surrogate or

backup 

The PSS can itself perform certain operator tasks or back-up operators as they perform. 

If careful analysis of expert performance and information requirements forms the basis

for designing the PSS, it can serve as an expert system to support operators or even take

over certain tasks for them.

For example: 

Monitoring: A significant portion of the operator’s time and effort will involve 

monitoring automated systems. The PSS can assist in monitoring by attending

to selected information requirements or threshold conditions. 

Vigilance: Closely related to the requirement to monitor will be the need to stay

alert. The PSS can be designed with features that change in simple ways

periodically, which has been shown as a way to remind the operator to attend to 

the displays and information. 

Job performance aids: Simple procedures can be annotated and be available 

either automatically or on demand. Such approaches have been shown to 

reduce errors and performance time. 

Training

mechanism 

The PSS information management capability and integration with the global network

can easily be programmed to support training. 

In as many ways as possible, the training role within FCS should mirror the 

performance environment, and the PSS can help do that. 

Recently, an initial set of guidelines was prepared that, if followed, would lead to a 

more functional ET and electronic performance support system (EPSS) capability

(outlined in Appendix D, sections D.23 and D.24). 

As the FCS is further developed, the initial guidelines need to be revised and 

expanded to keep up with current capabilities. 

If the PSS can provide real-time information on a situation, it can also generate

scenario-specific information to drive training or rehearsals. 

The PSS, integrated with the training management system, should be 

programmed to adapt scenarios “on the fly” to match user readiness and training

needs. 

34 
System Acquisition and Related Early Decisions 

The FCS team has taken the position that training plans and system plans need to run on parallel

and frequently touching paths, and that doctrine and tactics, techniques, and procedures (TTP) will

be formulated while FCS is still in the design and early development stages. The observations in 

this section may be already reflected in the current approach. They are nonetheless presented here

to reinforce the decisions and courses of action that the FCS team is already espousing. 

Early 

collaboration 

Early collaboration between system designers and training designers is essential to

ensure that individual, crew, and collective training requirements are addressed. 

Bringing doctrine developers and training development teams on board after a system 

design is set eliminates the flexibility required to maximize the potential capability of 

the system. 

The advantages of early collaboration are compelling: 

When systems cannot fully support training and workarounds are required, 

early collaboration can produce intelligent and valuable workarounds rather

than those that negatively impact training. 

Permanent, formal, and continuing early collaboration can be valuable in 

ensuring consideration of manpower and personnel integration (MANPRINT)

aspects and integration of concept exploration. 

Early collaboration can shorten the time for training program development and 

integration into unit training plans and provides valuable feedback to system

designers. 

Planning

ahead 

Early collaboration is easier said than done. Planning how to bring all participants to 

the table to share ideas, plans, activities, concerns, and problems needs to be addressed

even earlier than the early collaboration itself. 

Early collaboration is not going to happen without concrete plans for how to 

accomplish it.

The FCS team needs a template against which training and system

development proposals will be judged to ensure integration among individual, 

self-development, and unit training. 

Soldier-in-theloop 

Modeling and simulation are useful and valuable for concept development – they allow 

for multiple examinations of an issue under various conditions, with carefully

standardized control of external variables. But at some point, soldiers must also be

involved in the examination. 

Until we can perfectly model the soldier, we need to use real soldiers in conjunction 

with system simulations. 

The soldier-in-the-loop approach remains essential for, at a minimum:

Proofing of MANPRINT design considerations. 

Developing and confirming the effectiveness of system and non-system

training devices. 

Developing and proofing crew and unit TTPs. 

35 
Doctrine

development

It is essential that doctrine and training be fielded at the same time as the systems. 

An Objective Force and FCS without doctrine, even for a short period of time, will

have nothing to train, no standards for performance, and little chance of being a 

potent force in the full spectrum of operations. 

This should go without saying, but it didn’t. 

Address concept of fighting doctrine early in acquisition process. 

Use modeling and simulators/simulations to first push doctrinal envelope then

proof fighting doctrine. 

Ensure that doctrine development positions are filled by personnel with 

experience in digital, conventional, and back-up operations. 

 

36 
System Characteristics and Features 

Much of the research performed over the past decade has addressed the characteristics of digital or 

information systems. Those observations are largely as relevant today for FCS as they were for the 

emerging forerunners of FCS. General observations about the characteristics needed for the FCS 

have been organized into seven topic areas. 

Growth

potential

Design and development of FCS constitute a sort of “living” process that will never 

stop changing or growing. Technology not only permits frequent upgrades, but 

demands the upgrades by virtue of its own fast pace of change. 

If FCS is designed and built to exploit only current and near-term potential, it will be 

obsolete before it is fielded.

As design decisions are made, they need to consider not only the best that inventors 

and programmers can offer today, but also the “hooks” for what the future will offer 

(see Big Picture). 

Design all hardware, software, and otherware for growth and reconfiguration. 

Use modular software to support rapid reconfiguration.

Make iterative refinements based on soldier-in-the-loop testing to identify

reconfiguration requirements. 

Integration The FCS is conceived as an “integrated” system – the concept is laudable and must

apply to more than just the hardware and software found on one vehicle. 

If integration as a goal is not applied to multiple systems, users, and functions, the 

system of systems will never achieve the envisioned potential. 

Integration needs to be designed in, not jury-rigged after the fact. 

System designers must devise ways to integrate exercise support, training

management, automated measurement, and performance support. 

Command and control systems should be integrated across echelons, 

platforms, and “players” (human, semi-automated, robotic). 

Reliability Reliability is always a goal, but it is even more necessary with high tech systems that 

cannot be repaired with duct tape and pantyhose (chewing gum, soap, etc.).

Soldiers will lose trust in nonreliable systems with lightning speed and will never quite

recover. 

One obvious solution is ultra-reliable systems. Other actions must also be taken: 

Manage user expectations concerning reliability and timelines. 

Provide training on back-up tasks as well as on troubleshooting and corrective 

actions. 

Communicate clearly any upgrades that are made. 

Make upgrade transitions painless by testing the upgrade process thoroughly

and providing support. 

37 
Training

support

Embedded training is going to be the primary mode of training, and the command,

control, communications, computers, intelligence, surveillance and reconnaissance

(C4ISR) network will be the backbone of training as well as operations. The FCS 

capabilities have to include training support as a nonnegotiable requirement. 

To see FCS as a weapon system (of systems) that also supports training is no better 

than seeing it as a training device that also shoots – both functions are essential. 

The TDT and SDT will need to work collaboratively as early as possible to ensure that

user-training needs are as well supported, as are user operational requirements. 

Automation must be exploited to support feedback sessions (see Appendix D, 

section D.25 for summaries of strategies for automated feedback sessions). 

The tools for customizing training exercises and feedback should be integrated 

into the training subsystem.

Capabilities to update training materials must be integrated and the process 

should be effortless for users; it should not be transparent: Users should be

told when updates are posted, and the nature of the update should be described. 

Training management features should include training delivery (support for 

preparation and implementation activities and streaming of training scenarios), 

monitoring training-in-progress to provide feedback, tracking training progress 

for individuals and units, and generating training plans to ensure continuous 

improvement. 

Presentation 

features 

The TDT and SDT need to work together beginning with the first training design

decisions, to make the presentations appeal to the audience. 

If training is presented only with linear, passive formats, learners will not stay 

engaged and will not retain the information. 

There are some specific features of computer-based training (including ET) that have

been shown to be effective:

Use multimedia previews that are accurate, colorful, concise, and mentally

stimulating to facilitate immersion. 

Make the presentations intuitively understandable. If detailed explanations are 

required for users to understand the purpose of a presentation (whether

feedback, information management, or PSS), the system will be of little or no

value to users. 

Leaders must be able to perceive immediately and understand unequivocally

what they are seeing on their screen (e.g., aggregated v. disaggregated unit

icons, friendly v. threat forces). 

The tactical display and its controls should perceptually reflect the task and 

subtask structure of a user's hierarchical perception of task requirements. The 

key to mastery of a complex task is in the immediate perceptual processes 

where the task is structured. 

Build the system with commands with specific names to support retention. 

For initial skill acquisition, use direct manipulation or windows-like approach 

rather than command-based. 

38 
Incorporate instructional principles such as advance organizers, metaphors, 

rehearsals, and framing (described in Appendix D, section D.26). 

Create training with a focus on pattern recognition for situations. 

Screens should be designed to be similar to what soldiers are used to – 

“common look, common feel” should be universal, not exclusive within the 

Army. 

However: 

Don’t reject traditional job aids (paper-based). Users in some situations may

want something “always on top.” 

Workload 

management 

Technological advances rarely simplify jobs without also increasing complexity, as

well as cognitive and sensory load. 

If results of job and task analysis in simulation are not fed back to system design and 

job design, performance of UA soldiers could quickly be degraded due to work 

overload. 

The management of soldier workload must be addressed in terms of various aspects of 

their jobs: 

Control of robotic and semi-autonomous entities is affected by such variables 

as the design of the interface, attentional resources, the number and type of

concurrent tasks, the reliability of the systems, and environmental stressors, all 

of which must be monitored and managed. 

Deliberate procedures are needed to control the analysis and flow of digital 

data. Not only should information management be a primary topic in the 

standard operating procedure (SOP), but the FCS itself should be able to 

perform some of the SOP requirements and tasks. This assumes, of course, 

that the SOP/TTP for FCS UAs will be standardized to some noticeable 

degree. 

Both information amount and information relevance significantly impact the 

ability of soldiers to manage information received from C2 systems, and must 

be monitored and controlled. 

User needs The design of a C2 system must be based on the information demands and needs of 

users. 

For the designers of FCS to produce a system that meets user needs, they need to pay 

attention to user needs and not get trapped into a technology-driven design process. 

Designers need to review and work with recommendations from users to identify and 

meet battle command information requirements. 

Battle lab experiments have already provided user recommendations to serve 

as a starting point (see Appendix D, section D.27). 

Instead of a shotgun blast of information, commanders want information sent

to them tailored, so they only receive what they need. 

Instead of raw intelligence, commanders expect grounded intelligence 

summaries that explain the enemy's intent. The digital [and FCS]

commander's questions are more specific than his analog counterpart's. 

39 
Commanders’ priority information request (PIRs) are more focused when the 

systems exist to support them. As a result of information sharing, technology

makes PIRs available at all levels. 

40 
Performance Improvement – Factors Beyond Training

Despite what the training contractors would have you believe, it isn’t all about training. But it isn’t 

all about the systems, either. Some of the threats to success are presented here. 

Personnel

challenges

Training cannot be the answer for all of the UA performance hopes and dreams. 

If we provide brilliant training but don’t address the rest of the picture, we will fail. 

It is vital that the Army conduct analyses to determine role of training, selection and 

classification, and personnel assignments in the Objective Force UAs. Related nontraining personnel actions include: 

Upgrading the personnel selection process, classification and promotion 

procedures, and utilization and assignment policies. We cannot simply ask

soldiers to perform a broader range of skills while at the same time specifying

that no increase in aptitudes and experience should be required to operate, 

maintain, or sustain the FCS UA.

Taking another hard look at the notion that FCS UA soldiers will be multiskilled, a keystone for the UA, especially as it operates in concert with legacy

and Stryker forces. 

Examining leader development, which depends on the formation of in-depth 

expertise across a range of command-related areas, both vertical and horizontal. 

Without major changes in the way officers and non-commissioned officers 

(NCOs) are recruited, selected, trained, and mentored, it is highly unlikely that 

seasoned adaptive leaders can be produced easily and rapidly. 

Human

factors 

challenges

The new FCS capabilities will also introduce problematic situations that did not arise in 

the older legacy systems, and somewhere, somehow, the FCS team needs to address the 

issues. Such threats to excellence include: 

Information overload and resultant fatigue. 

Overcontrol of subordinates. 

Vulnerability to countermeasures. 

Incompatible technologies among coalition forces. 

Failure to adapt organization structure to new doctrine and procedures. 

Additionally: 

The suggestion that we reduce manpower requirements without reducing

combat effectiveness through use of smaller crews augmented by new

technology may be naive. Our experience is that information technology (IT)

can flatten organizational structure, reduce the need for centralized operations, 

and change the nature of jobs, sometimes making them more complex, but 

rarely does it reduce workloads. Instead, the workload is shifted and the nature

of work loading is changed. Low skill tasks are automated, leaving soldiers to 

perform more complex tasks that cannot be easily automated. In the end, there 

are limited degrees of freedom: To have smaller teams, we will need increased 

aptitude and/or training. 

41 
Formal

information

sharing 

As FCS enters the Army and units begin to develop TTP, information flow will be

critical if we want to continue to share observations and lessons learned. 

If we cannot devise and maintain an information sharing mechanism, progress toward 

readiness will be handicapped. 

There are a number of entities that need to be nodes in and/or users of the information 

network.

Proponent interaction is essential, to interface and share lessons learned on 

important aspects of training, how-to-fight doctrine, and back-up requirements. 

Permanent working groups among proponents, as well as joint and coalition 

groups need to be formed to focus on specific shared issues. 

As UAs are forming and learning, crosstalk and feedback among them is needed

to formulate TTP.

Not only should we continue to review lessons learned from leaders, we should 

be prepared to obtain the same type of information from Objective Force 

leaders. A summary of key findings distilled from leaders' insights on Force 

XXI transition was prepared – it is shown in Appendix D, section D.28. 

We need to institutionalize and formalize linkages between early-fielding units

and later fielding units, and between units and proponents. 

Miscellaneous Having categorized the vast majority of “nuggets” from the literature and interviews

into topics and subtopics, there remained a few that didn’t fit anywhere else but were 

too good to just discard. They include the following:

The Army needs to be a "learning organization" and needs to encourage creative

thinking, balanced with standardized ways of doing things. 

On the topic of span of control: Commander and staff at corps, division, and 

brigade (and most company level) report more severe negative impact from the 

complexity of the environment and absence of assistance from their 

organizational structure than from an increased span of control. 

Additionally, lack of habitual relationship with supported units seemed to have

at least as much negative impact as the number of subordinate units. 

A consistent finding for force projection and SASO stressed the importance of

avoiding ad hoc structures. The Joint Task Force (JTF) should be built on an

existing DoD unit, and be augmented as necessary by specially trained cells. 

As military strategists and tacticians ponder and plot unit organizational 

structures, they will also need to explicitly define the role of the homestation 

operations center (HSOC) in training, and the relationship between the HSOC 

and forward deployed units. 

If any training is to be effective, there will need to be a strong commitment by

senior leaders to protect training time and resources. 

42 
Appendix A. Definitions of Acronyms and Jargon

Acronyms

A3ARM advanced after action review media 

AAR after action review

AC active component 

ADTP Advanced Distributed Training Program 

AFRU Armored Forces Research Unit 

AIT advanced individual training

ARI U.S. Army Research Institute for the Behavioral and Social Sciences 

ARL Army Research Laboratory

ARTEP Army Training and Evaluation Program 

ATESC advanced tactical engagement simulation concepts 

ATIA Army Training Information Architecture 

BC Battle Captain 

BCT basic combat training

BF battlefield function 

BLUFOR blue forces 

BOS battlefield operating system

C2 command and control 

C4I command, control, communications, computer, and intelligence

C4ISR command, control, communications, computers, intelligence, surveillance and 

reconnaissance 

CALL Center for Army Lessons Learned 

CCTT Close Combat Tactical Trainer 

CEP concept experimentation program 

COP communities of practice

CRIOT cognitive requirements for information operations training

CTC Combat Training Center

CVC combat vehicle command

DL distance learning

EPSS electronic performance support system 

ET embedded training

FBCB2 future battle command brigade and below 

FCS Future Combat Systems

FRAGO fragmentary orders 

FSO fire support officer 

HSOC homestation operations center 

A-1 
IDEF0 Integrated Definition 

IET initial entry training

IFRU Infantry Forces Research Unit

IPT integrated process team

IS instrumentation system

ISD Instructional Systems Design

IT information technology

JIM joint, interagency, and multi-national 

JTF Joint Task Force

LMS learning management system 

LOS line-of-sight 

LSI Lead Systems Integrator

MANPRINT manpower and personnel integration 

METL mission essential task list

METT-TC mission, enemy, terrain, troops, time, and civilian considerations 

MOE measure of effectiveness 

MOP measure of performance

MOS military occupational specialty

MSS multi-skilled soldier

MTOE modified tables of organization and equipment 

MTP Mission Training Plan 

NBC nuclear/biological/chemical

NCO non-commissioned officer 

NET new equipment training

NLOS non line-of-sight 

O/C observer/controller 

OIS objective instrumentation system

OPFOR opposing force 

OPTEMPO operational tempo 

PDA personal digital assistant 

PIR priority information request

POI program of instruction 

PSS performance support system 

q.v. quod vide, meaning to go look it up

R&D research & development 

RC reserve component 

S2 intelligence officer

S3 operations officer 

SA situational awareness 

A-2 
SAIC Science Applications International Corporation 

SAT Systems Approach to Training

SDT system development team

SIMNET simulation networking

SME subject matter expert 

SOP standard operating procedures 

SASO stability operations and support operations 

SOW Statement of Work 

SPA Staff Performance Analysis

SSRU Simulation Systems Research Unit 

STRAP system training plan 

STRIVE Satisfying Training Requirements in Virtual Environments 

TAAF-X Training Analysis and Feedback Center of Excellence 

TADSS training aids, devices, simulators, and simulations 

TAF tactical analysis facility

TDT training design team 

TES tactical engagement simulation 

TRADOC U.S. Army Training and Doctrine Command 

TSC training support code

TSP training support package

TTP tactics, techniques, and procedures 

UA Unit of Action 

UAMBL Unit of Action Mounted Battlespace Laboratory

UTMS unit training management system 

VPN virtual private network 

XO Executive Officer 

Jargon 

Access or accessibility: Includes downloading, working on-line, logging in to a discussion 

group, reading – however the user and the component are supposed to connect. 

Aggregated: With tactical displays, refers to how entities (units, soldiers, etc.) are represented, 

and could be, for example, one icon or symbol per vehicle, per squad, per company, or

per UA. When the entities are “rolled up” they are said to aggregate; when broken out, 

they are disaggregated. The tradeoffs are in terms of information precision vs. big picture 

displays of information. 

Asynchronous: In distributed learning arenas, communications among students and instructors 

that involve some delay, such as by means of email where the receiver retrieves messages 

on his/her own timeline and can respond later. 

Automaticity: In task performance, the quality of being able to perform complex activities, 

comprising a long series of steps or motions, without conscious attention to the sequence 

A-3 
or process. Common (non-military) examples include using a keyboard, playing piano, 

driving a car. 

A-way: As opposed the [only] way, indicates that the subject training or process will work but is 

not necessarily the only way to do the training or process. 

Behaviorist: Characterizing behaviorism theories of instruction, where objectives are determined 

by analysts before training is provided and are the focal point for developing the 

instructional materials; training participants learn how to execute tactical actions; and 

AARs are led by the instructors. See also Constructivist. 

Cognitive strategies: Ways of learning (and therefore of instructing) that reflect the way

individuals process information. A key feature of these strategies is that the learner is an 

active participant in the learning process. 

Component: See training component.

Concept exploration: The use of simulation, ranging from low-fidelity enactments to high

fidelity technology-supported exercises or live exercises, to try out and refine ideas on 

organizations, operations, doctrine, etc. 

Constructivist: Characterizing constructivism theories of instruction, where instructional 

objectives emerge from decisions made by the training participants; there are authentic

instructional conditions; the participants develop an understanding of the principles of 

tactics; and AARs are led by the students. 

Disaggregated: See aggregated. 

Distributed training: Training of teams, small groups, or units where the participants are not colocated. They may be in their own vehicles or tactical locations, or may even be at 

dispersed learning centers or posts. 

Dyad/triad/quadad: Groups of two, three, or four or more (quadad being a term invented to mean 

“four or more”) who are designated as the training audience for targeted exercises. 

Enabling Skills: Those skills that are more basic, learned early (or earlier), and may not of

themselves be important but are important as the building blocks for more complex

critical or terminal tasks (see terminal skills). 

Enabling objectives: Training objectives for enabling skills (q.v.). 

Extrinsic feedback: Feedback that occurs because a device that is appended or external to the 

operational system is triggered, or because a designated O/C detects a particular behavior. 

Force XXI: During the years after the end of the cold war and into the new century, the Army

has been focused on change. Training and Doctrine Command was charged with the 

mission of transforming the tactical Army. The TRADOC focused on doing that across 

all the Army’s mission areas: doctrine, training, leader development, equipment, and 

force design, ensuring quality in each area. The process of change is called Force XXI.

Integrated Definition (IDEF0) system analysis: IDEF0 modeling is used to analyze the functions 

a system performs and to record the mechanism (or means) by which these functions are 

done. The IDEF0 modeling is composed of a hierarchical series of diagrams that 

gradually display increasing levels of detail describing functions and their interface with 

the context of a system. 

A-4 
Intrinsic feedback: Feedback that occurs within a system or environment as the natural outcome

of a particular action; not delivered by an O/C or generated by some appended system. 

Journeyman: More skilled than novice or apprentice, but not yet expert; able to perform most 

tasks with minimal supervision. 

MANPRINT: Generally, a set of standards and guidelines for systems that defines the minimal 

requirements for manpower and personnel, including training and ergonomics. 

Mission rehearsal: A relatively high-fidelity and realistic opportunity to practice and improve 

methods for a specific operation under specific conditions.

Multi-skilled soldiers (MSS): The MSS concept is an important part of the UA requirements 

discussions. For the UA, the MSS is 

(1) additionally skilled with cross-training from the institution, to provide greater 

depth and redundancy of specific skill sets within the UA; 

(2) generic in terms of MOS non-specialization, with focus on principles that would 

generalize; 

(3) adaptable, exhibiting creative and problem-solving qualities attributed to the

emerging notion of the adaptive leader, with training focused on mindset of 

adaptability, self-education, and problem-solving; 

(4) perpetually learning, must be trained and motivated to deepen and broaden the 

skills they acquire in IET through never-ending self-education, largely through 

distributed training and DL; requires significant professional development 

incentives, resources, and mentoring. 

Performance objectives: Statements and descriptions that focus first on what should be 

accomplished, and then move on to describe suggested ways to do it. Where tasks tend to

be procedural, performance objectives describe the desired end state and only assist the 

user on performance steps peripherally.

Procedural training: Training in how to perform tasks that have set steps and sequences. 

Scenario nesting: A method for developing the scenarios underlying training exercises where 

different individuals, teams, and units at multiple echelons all work within a common 

underlying tactical scenario. Each user or user group sees only the appropriate slice of 

the larger scenario. 

SIMNET: Simulation Networking, a virtual simulation technology used principally in training

mounted maneuver forces and mounted infantry. 

Structured training: Training that is focused on specific training objectives in a deliberatelyconstructed training package, and that takes full advantage of instructional design

principles and available simulation capabilities to provide training that is both efficient 

and effective. 

Synchronous: In distributed learning arenas, communications that occur in real time among

students and instructors; direct interactions, whether by voice or instant messaging-like

methods. 

Technology insertion: The situation in the design and develop cycle for a system (any system –

training, hardware, software, etc.) in which a technology (new or not previously included) 

A-5 
is deemed important and useful enough that the already-developed system will be adapted 

to accommodate use of the new technology.

Terminal objectives: Training objectives for terminal skills. 

Terminal skills: Those skills that are the end state for a block of training, the culmination and 

raison d’etre for the training of a set of enabling skills (see enabling skills). 

Training component: Information items that make up the TSP, as opposed to the content of the 

TSP and its components. 

Training domain: The complete set of tasks, conditions, and standards that should be the content 

of training; may be restricted to a particular audience, force type, etc. 

Training support package: A complete, exportable package integrating training products, 

materials, and/or information necessary to training one or more critical 

[collective/individual] tasks (definition from TRADOC Reg 350-70, Systems Approach to 

Training Management, Processes, and Products [Department of the Army, 1999]). 

Unit of Action: The command and control unit of the Objective Force, roughly equivalent to 

brigade-level in the Legacy Force.

Workarounds: Deliberate and planned methods for representing elements or activities or the 

outcomes of activities in simulation environments when they cannot be presented 

directly. 

A-6 
Appendix B. References

Anderson, L. B., Begley, I. J., II, Arntz, S. R., & Meliza, L. L. (2000). Training Analysis and 

Feedback Center of Excellence (TAAF-X) (ARI SR 2001-01) [DTIC ADA384030]. 

Alexandria, VA: U.S Army Research Institute for the Behavioral and Social Sciences. 

André, C. R., Wampler, R. L., & Olney, G. W. (1997). Battle Staff Training System in Support 

of Force XXI Training Program: Methodology and Lessons Learned (ARI RR 1715) 

[DTIC ADA338728]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Atwood, N. K, Quinkert, K. A., Campbell, M. R., Lameier, K. F., Leibrecht, B. C., 

Doherty, W. J. (1991). Combat Vehicle Command and Control Systems: Training 

Implications Based on Company-Level Simulations (ARI TR 943) [DTIC ADA246460]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Atwood, N. K., Winsch, B. J., Quinkert, K. A., & Heiden, C. K. (1994). Catalog of Training 

Tools for Use in Distributed Interactive Simulation (DIS) Environments (ARI RP 94-12) 

[DTIC ADA282841]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Barnett, J. S. & Holmquist, J. P. (2001). Human Sergeants, Robot Soldiers: Soldier-Machine 

Integration Issues with Semi-Autonomous Systems. Proceedings of the 

Interservice/Industry Training, Simulation and Education Conference (CD-ROM) (I/ITSEC 

Paper HS-016). Orlando, FL: National Training Systems Association. 

Barnett, J. S., Meliza, L. L., & McCluskey, M. R. (2001). Defining Digital Proficiency

Measurement Targets for U.S. Army Units (ARI TR 1117) [DTIC ADA396899]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Bonk, C. J. & Wisher, R. A. (2000). Applying Collaborative and E-Learning Tools to Military

Distance Learning: A Research Framework (ARI TR 1107) [DTIC ADA389681]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Brown, B. R., Nordyke, J. W., Gerlock, D. L., Begley, I. J., II, & Meliza, L. L. (1998). Training 

Analysis and Feedback Aids (TAAF Aids) Study for Live Training Support (ARI SR 98-04) 

[DTIC ADA351107]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Burnside, B. L. (1990). Assessing the Capabilities of Training Simulations: A Method and 

Simulation Networking (SIMNET) Application (ARI RR 1565) [DTIC ADA226354]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Campbell, C. H., Campbell, R. C., Sanders, J. J., Flynn, M. R., & Myers, W. E. (1995). 

Methodology for the Development of Structured Simulation-Based Training (ARI RP 95-08) 

[DTIC ADA296171]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Campbell, C. H. & Deter, D. E. (1997). Guide to Development of Structured Simulation-Based 

Training Programs (ARI RP 97-14) [DTIC ADA328666]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

B-1 
Campbell, C. H., Deter, D. E., Ford, L. A., Graves, C. R., Campbell, R. C., Pratt, D. M., 

Jenkins, S. N., & Quinkert, K. A. (1999). Combined Arms Operations at Brigade Level, 

Realistically Achieved Through Simulation III (COBRAS III): Report on Development and 

Lessons Learned (ARI RR 1736) [DTIC ADA359920]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

Campbell, C. H., Deter, D. E., & Quinkert, K. A. (1997). Report on the Expanded Methodology 

for Development of Structured Simulation-Based Training Programs (ARI RR 1710) 

[DTIC ADA328671]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Campbell, C. H., Ford, L. A., Campbell, R. C. & Quinkert, K. A. (1998). A Procedure for 

Development of Structured Vignette Training Exercises for Small Groups (ARI RP 98-37) 

[DTIC ADA350021]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Campbell, C. H., Graves, C. R., Deter, D. E., & Quinkert, K. A. (1998). Combined Arms 

Operations at Brigade Level, Realistically Achieved Through Simulation II (COBRAS II): 

Report on Development and Lessons Learned (ARI RR 1726) [DTIC ADA352029]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Campbell, C. H. & Holden, W. T., Jr. (2001). Reflections on the Structure of the Future 

Training System (ARI RN 2001-12) [DTIC ADA393431]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

Campbell, C. H., Pratt, D. M., Deter, D. E., Graves, C. R., Ford, L. A., Campbell, R. C., &

Quinkert, K. A. (1999). The COBRAS Synthetic Theater of War Exercise Trial: Report on 

Development, Results, and Lessons Learned (ARI RR 1734) [DTIC ADA359923]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Campbell, C. H., Pratt, D. M., Graves, C. R., & Quinkert, K. A. (2000). Summary of 

Recommendations for Successful Fielding of Force XXI Training Products (ARI RR 1758) 

[DTIC ADA378082]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Clagg, R. A., Detrani, R. L., Burnside, B. L., & Finley, D. L. (1999). Integration of Training 

Development Among Schools and Distributed Training Environments (ARI SR 2000-01) 

[DTIC ADA371931]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Collins, J. W., III & Throne, M. H. (1996). Annotated Bibliography of Training Technologies 

and Methods for Teaching the Use of Advanced Technology (ARI RP 96-02) 

[DTIC ADA308831]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Cox, J. A., Holder, L. D., Leibrecht, B. C., & DeRoche, L. (2002). Vertical Teams in the

Objective Force: Insights for Training and Leader Development (ARI RR 1798) 

[DTIC ADA408924]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Deatz, R. C. & Campbell, C. H. (2001). Application of Cognitive Principles in Distributed 

Computer-Based Training (ARI RP 2001-03) [DTIC ADA392929]. Alexandria, VA: U.S. 

Army Research Institute for the Behavioral and Social Sciences. 

B-2 
Deatz, R. C., Forrest, D., Holden, W. T., Jr., Sawyer, A. R., Britt, D. B., & Gray, R. (1998). 

Follow-On Development of Structured Training for the Close Combat Tactical Trainer

(ARI RR 1725) [DTIC ADA350545]. Alexandria, VA: U.S. Army Research Institute for 

the Behavioral and Social Sciences. 

Deatz, R. C., Greene, K. A., Holden, W. T., Jr., Throne, M. H., & Lickteig, C. W. (2000). 

Refinement of Prototype Staff Training Methods for Future Forces (ARI RR 1763) 

[DTIC ADA383380]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Department of the Army. (1999, March 9). Systems Approach to Training Management, 

Processes, and Products (TRADOC Reg 350-70). Fort Monroe, VA: Headquarters, U.S. 

Army Training and Doctrine Command. 

Dierksmeier, F. E., Johnston, J. C., Winsch, B. J., Leibrecht, B. C., Sawyer, A. R., 

Quinkert, K. A., & Wilkinson J. G. (1999). Structured Simulation-Based Training Program 

For a Digitized Force: Approach, Design, and Functional Requirements, Volume I

(ARI RR 1737) [DTIC ADA361534]. Alexandria, VA: U.S. Army Research Institute for 

the Behavioral and Social Sciences. 

Dudley, M. G., Johnston, J. C., Jones, W. S., Strauss, C. P., & Meliza, L. L. (2001). Making the 

Transition From Analog to Digital Warfighting: Changes in Unit Behavior and Knowledge

(ARI RR 1785) [DTIC ADA397575]. Alexandria, VA: U.S. Army Research Institute for 

the Behavioral and Social Sciences. 

Elliott, G. S., Sanders, W. R., & Quinkert, K. A. (1996). Training in a Digitized Battalion Task

Force: Lessons Learned and Implications for Future Training (ARI Research Report 1695) 

[DTIC ADA313293]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Endsley, M. R., Holder, L. D., Leibrecht, B. C. Garland, D. J., Wampler, R. L., &

Matthews, M. D. (2000). Modeling and Measuring Situation Awareness in the Infantry 

Operational Environment (ARI RR 1753) [DTIC ADA372709]. Alexandria, VA: U.S. 

Army Research Institute for the Behavioral and Social Sciences. 

Fallesen, J. J., Lussier, J. W., & Michel, R. R. (1992). Tactical Command and Control Process

(ARI Research Product 92-06) [DTIC ADA255036-FICHE]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

Finley, D. L., Sanders, M. G., & Ryan, A. J., III. (1996). Application of Training Transfer

Principles in Developing the High Transfer Training (HITT) Methodology. Innovations in 

Education and Training International, 33, 232-239. [DTIC ADA319115]. 

Finley, D. L., Shlechter, T. M., & Lavoie, M. C. (2000). Combined Arms Structured SimulationBased Training Program: Reflections of Key Developers (ARI RR 1756) 

[DTIC ADA376715]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Flynn, M. R., Campbell, C. H., Myers, W. E., & Burnside, B. L. (1998). Structured Training for 

Units in the Close Combat Tactical Trainer: Design, Development and Lessons Learned

(ARI RR 1727) [DTIC ADA354171]. Alexandria, VA: U.S. Army Research Institute for 

the Behavioral and Social Sciences. 

B-3 
Ford, J. P., Mullen, W. J., III, & Christ, R. E. (1998). Effective Span of Command and Control 

by Echelon in Training and Operational Environments (ARI RN 99-03) [DTIC 

ADA355164]. Alexandria, VA: U.S. Army Research Institute for the Behavioral and 

Social Sciences. 

Ford, J. P., Mullen, W. J., III, & Keesling, J. W. (1997). Analysis of Command and Control 

Battlefield Functions Performed in the Armored Brigade (ARI RR 1711) 

[DTICADA328661]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Ford, L. A., Campbell, R. C., & Cobb, R. M. (1998). Analysis of Emerging Digital and Back-up 

Training Requirements (ARI SR 98-07) [DTIC ADA354190]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

Gossman, J. R., Beebe, M. E., Bonnett, Michael, Forrest, D., Shadrick, S. B., Dannemiller, 

M. A., Mauzy, R. P., & Bonnett, Mitch (1999). The Commanders’ Integrated Training 

Tool for the Close Combat Tactical Trainer: Design, Prototype Development, and Lessons 

Learned (ARI RR 1739) [DTIC ADA364066]. Alexandria, VA: U.S. Army Research 

Institute for the Behavioral and Social Sciences. 

Gossman, J. R., Burnside, B. L., Flynn, M. R., Dannemiller, M. A., & Mauzy, R. P. (2002). 

Command Group Training in the Objective Force (ARI RR 1793) [DTIC ADA406176]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Gossman, J. R., Graves, C. R., Mauzy, R. P., & Clagg, R. A. (2001). Assessing and Managing 

User-Produced Training Support Packages (ARI RR 1772) [DTIC ADA390420]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Graham, S. E. & Matthews, M. D. (1999). Infantry Situation Awareness: Papers from the 1998 

Infantry Situation Awareness Workshop (ARI UD160.I53 1999) [DTIC ADA371869]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Graves, C. R., Campbell, C. H., Deter, D. E., & Quinkert, K. A. (1997). Combined Arms 

Operations at Brigade Level, Realistically Achieved Through Simulation I (COBRAS I): 

Report on Development and Lessons Learned (ARI RR 1719) [DTIC ADA343683]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Graves, C. R. & Myers, W. E. (1997). An Expansion of the Virtual Training Program: History 

and Lessons Learned (ARI RR 1703) [DTIC ADA328416]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

Graves, C. R., Pratt, D. M., Campbell, C. H., Allen, J. D., Thorson, K. G., Jenkins, S. N., &

Quinkert, K. A. (1999). Force XXI Training Program - Digital Project: Report on 

Development and Lessons Learned (ARI RR 1748) [DTIC ADA371065]. Alexandria, VA: 

U.S. Army Research Institute for the Behavioral and Social Sciences. 

Greess, M. (1994). Combat Vehicle Command and Control System Architecture Overview

(ARI RP 95-05) [DTIC ADA286259]. Alexandria, VA: U.S. Army Research Institute for 

the Behavioral and Social Sciences. 

Heiden, C. G. (1994). Integrating SIMNET into Heavy Task Force Tactical Training (ARI

RP 95-01) [DTIC ADA285953]. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

B-4 
Hoffman, R. G. (1997a). A Description of Multimedia Presentation of COBRAS Vignette 

Training Support Package Information (ARI RP 97-16) [DTIC ADA336703]. Alexandria, 

VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Hoffman, R. G. (1997b). Combat Support and Combat Service Support Expansion to the Virtual 

Training Program SIMNET Battalion Exercise: History and Lessons Learned (ARI

RR 1717) [DTIC ADA341201]. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Hoffman, R. G., Graves, C. R., Koger, M. E., Flynn, M. R., & Sever, R. S. (1995). Developing 

the Reserve Component Virtual Training Program: History and Lessons Learned (ARI

RR 1675) [DTIC ADA296153]. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Holden, W. T., Jr., Throne, M. H., & Sterling, B. S. (2001). Prototype Automated Measures of 

Command and Staff Performance (ARI RR 1779) [DTIC ADA397634]. Alexandria, VA: 

U.S. Army Research Institute for the Behavioral and Social Sciences. 

Holland, V. M. & Wisher, R. A. (1990). Computerized Hand-Held Instructional Prototype 

(CHIP) (ARI RR 1557) [DTIC ADA226450]. Alexandria, VA: U.S. Army Research 

Institute for the Behavioral and Social Sciences. 

Jenkins, S. N., Graves, C. R., Deter, D. E., & Quinkert, K. A. (1999). Development of the 

COBRAS III Performance Objectives for the Brigade and Battalion Staff Exercise (ARI

RP 99-03) [DTIC ADA364558]. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Jones, C. [Cpt.] (1999). Digital Instruction Experiment [DIE], 26 Oct - 6 Nov 1998: Assessment 

Results and Conclusions (Final Draft 5 Feb 1999). 3rd Squadron, 16th Cavalry Professional 

Development Division. 

Kern, R. P., Wisher, R. A., Sabol, M. A., & Farr, B. J. (1994). Reacquisition of Skills by Combat 

Engineers Mobilized from the Individual Ready Reserve (ARI RR 1667) 

[DTIC ADA286244]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Koger, M. E., Long, D. L., Sanders, J. J., Broadwater, T. W., Brewer, J. D., & Britt, D. B.

(1996). Simulation-Based Mounted Brigade Training Program: History and Lessons 

Learned (ARI RR 1689) [DTIC ADA309752]. Alexandria, VA: U.S. Army Research 

Institute for the Behavioral and Social Sciences. 

Koger, M. E., Quensel, S. L., Sawyer, A. R., Sanders, J. J., Crumley, K. A., Brewer, J. D., &

Sterling, B. S. (1998). Staff Group Trainer: Development of a Computer-Driven, 

Structured, Staff Training Environment (ARI RR 1718) [DTIC ADA347074]. Alexandria, 

VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

LaJoie, A. S. & Sterling, B. S. (1999). A Review and Annotated Bibliography of the Literature 

Pertaining to Team and Small Group Performance (1989 to 1999) (ARI RP 2000-01) 

[DTIC ADA371864]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

B-5 
Leibrecht, B. C., Johnston, J. C., Black, B. A., & Quinkert, K. A. (2002). Managing Force XXI 

Change: Insights and Lessons Learned in the Army's First Digital Division (ARI

SR 2002-04) [DTIC ADA400644]. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Leibrecht, B. C., Winsch, B. J., Ford, L. A., Sawyer, A. R., Meade, G. A., Ainslie, F. M., Smith, 

P. G., Sever, R. S., & Doherty, W. J. (1993). Battalion Evaluation of the Combat Vehicle 

Command and Control System in Distributed Interactive Simulation: Preliminary Findings

(ARI TR 992) [DTIC ADA275912]. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Lickteig, C. W. & Collins, J. W., III. (1995). Combat Vehicle Command and Control System 

Evaluation: Vertical Integration of an Armor Battalion (ARI TR 1021) [DTIC 

ADA292718]. Alexandria, VA: U.S. Army Research Institute for the Behavioral and 

Social Sciences. 

Lickteig, C. W. & Emery, C. D. (1994). Information Management Performance of Future 

Platoon Leaders: An Initial Investigation (ARI TR 1000) [DTIC ADA282908]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Lickteig, C. W. & Quinkert, K. A. (2001). Applying Digital Technologies to Evaluation: A 

Focus on Command and Control (ARI TR 1115) [DTIC ADA392994]. Alexandria, VA: 

U.S. Army Research Institute for the Behavioral and Social Sciences. 

Lickteig, C. W., Sanders, W. R., Shadrick, S. B., Lussier, J. W., Holt, B. J., & Rainey, S. J.

(2002). Human-System Integration for Future Command and Control: Identifying 

Research Issues and Approaches (ARI RR 1792) [DTIC ADA405044]. Alexandria, VA: 

U.S. Army Research Institute for the Behavioral and Social Sciences. 

Lickteig, C. W., Sterling, B. S., Elliott, G. S., Burns, J. E. & Langenderfer, J. E. (1998). 

Assessing Battle Command Information Requirements and the Military Decision Making 

Process: An Exploratory Investigation for the Mounted Battlespace Battle Lab (ARI RR 

1731) [DTIC ADA359897]. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Lickteig, C. W. & Throne, M. H. (1999). Applying Digital Technologies to Training: A Focus 

on Pictorial Communication (ARI TR 1097) [DTIC ADA369262]. Alexandria, VA: U.S. 

Army Research Institute for the Behavioral and Social Sciences. 

Lussier, J. W., Ross, K. G., & Mayes, B. (2000). Coaching Techniques for Adaptive Thinking. 

Proceedings of the Interservice/Industry Training, Simulation and Education Conference 

(CD-ROM). Orlando, FL: National Training Systems Association 

Meliza, L. L. (1996). Standardizing Army After Action Review Systems (ARI RR 1702) [DTIC 

ADA322044]. Alexandria, VA: U.S. Army Research Institute for the Behavioral and 

Social Sciences. 

Meliza, L. L. (In Preparation). Digital Skills and Indicators of Overall Digital Proficiency. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Meliza, L. L. & Knerr, B. W. (1991). Early Training Strategy Development for Individual and 

Collective Training (ARI TR 936) [DTIC ADA242753]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

B-6 
Morrison, J. E. & Hammon, C. (2000). On Measuring the Effectiveness of Large-Scale Training 

Simulations (IDA Paper P-3570) [DTIC ADA394491]. Alexandria, VA: Institute for 

Defense Analyses. 

Moses, F. L. (2001). Training Challenges for Digitization (ARI SR 47) [DTIC ADA395401]. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Nelson, J. T., II & Akman, A. (2002). The Multi-Skilled Soldier Concept: Considerations for 

Army Implementation (ARI SR 2002-06) [DTIC ADA402901]. Alexandria, VA: U.S. 

Army Research Institute for the Behavioral and Social Sciences. 

Olmstead, J. A. (1992). Battle Staff Integration (IDA Paper P-2560) [DTIC ADA248941]. 

Alexandria, VA: Institute for Defense Analyses. 

Quensel, S. L., Myers, W. E., Koger, M. E., Nepute, J. T., Brewer, J. D., Sanders, J. J., 

Crumley, K. A., & Sterling, B. S. (1999). Development of a Refined Staff Group Trainer

(ARI RR 1735) [DTIC ADA359918]. Alexandria, VA: U.S. Army Research Institute for 

the Behavioral and Social Sciences. 

Salter, M. S. & Black, B. A. (1998). Back-Up Training Requirements for the Digitized

Battlefield: An Overview (ARI SR 98-05) [DTIC ADA354022-FICHE]. Alexandria, VA: 

U.S. Army Research Institute for the Behavioral and Social Sciences. 

Sanders, W. R. (1999). Digital Procedural Skill Retention for Selected M1A2 Tank InterVehicular Information (IVIS) Tasks (ARI TR 1096) [DTIC ADA368212]. Alexandria, VA: 

U.S. Army Research Institute for the Behavioral and Social Sciences. 

Sanders, W. R. (2001). Cognitive Psychology Principles for Digital Systems Training (ARI RR 

1773) [DTIC ADA391035]. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Sanders, W. R. & Burnside, B. L. (2001). Assessment of Initial Delivery of the Armor Captains' 

Career Course (Distance Learning) (ARI RR 1775) [DTIC ADA393635]. Alexandria, VA: 

U.S. Army Research Institute for the Behavioral and Social Sciences. 

Sanders, W. R. & Guyer, C. W. (2001). Commanders' Survey: Armor Captains' Career Course 

(Distance Learning) (ARI RR 1771) [DTIC ADA390166]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

Shlechter, T. M., Bessemer, D. W., Nesselroade, K. P., Jr., & Anthony, J. (1995). An Initial 

Evaluation of a Simulation-Based Training Program for Army National Guard Units (ARI

RR 1679) [DTIC ADA297271]. Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Shlechter, T. M., Burnside, B. L., & Thomas, D. A. (1987). Issues in Developing and 

Implementing Computer-Based Instruction for Military Training (ARI RR 1451) [DTIC 

ADA189479]. Alexandria, VA: U.S. Army Research Institute for the Behavioral and 

Social Sciences. 

Shlechter, T. M. & Finley, D. L. (2000). Structured Simulation-Based Training Programs: 

History and Lessons Learned (ARI RR 1755) [DTIC ADA376389]. Alexandria, VA: U.S. 

Army Research Institute for the Behavioral and Social Sciences. 

B-7 
Smith, M. L., Holden, W. T., Jr., & Starry, H. M. (2002). Training Model for Contingency 

Operations (ARI SR 2002-08) [DTIC ADA408556]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

Sterling, B. S. & Quinkert, K. A. (1998). Review of Battle Staff Training Research at Brigade 

and Battalion Levels (ARI RR 1730) [DTIC ADA359259]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

Sticha, P. J., Campbell, R. C., & Knerr, C. M. (2002). Training Concepts for Virtual 

Environments: Final Report (ARI SR 2002-05) [DTIC ADA402315]. Alexandria, VA: 

U.S. Army Research Institute for the Behavioral and Social Sciences. 

Sulzen, R. H. (1996). National Training Center Research Element, Fort Irwin, California, 1986-

1996 (ARI SR 30) [DTIC ADA326056]. Alexandria, VA: U.S. Army Research Institute 

for the Behavioral and Social Sciences. 

Throne, M. H. & Burnside, B. L. (2003). Integrated Training and Performance Support For the 

Objective Force (ARI RR 1801). Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Throne, M. H., Deatz, R. C., Holden, W. T., Jr., Campbell, C. H., Sterling, B. S. &

Lickteig, C. W. (1999). Prototype Staff Training and Evaluation Methods for Future 

Forces (ARI RR 1745) [DTIC ADA369663]. Alexandria, VA: U.S. Army Research 

Institute for the Behavioral and Social Sciences. 

Throne, M. H., Holden, W. T., Jr., & Lickteig, C. W. (2000). Refinement of Prototype Staff 

Evaluation Methods for Future Forces: A Focus on Automated Measures (ARI RR 1764) 

[DTIC ADA384027]. Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Throne, M. H. & Lickteig, C. W. (1997). Training Computer Skills for the Future Battlefield: A 

Review and Annotated Bibliography (ARI RP 97-15) [DTIC ADA337314]. Alexandria, 

VA: U.S. Army Research Institute for the Behavioral and Social Sciences.

Unit of Action Maneuver Battle Laboratory. (2002). Operational Requirements Document for 

the Future Combat Systems (Draft, 30 Aug 02). Fort Knox, KY: Author, U.S. Army Armor 

Center and Fort Knox. 

Wampler, R. L. & Livingston, S. C. (1998). Battle Staff Training System II: Computer-Based 

Instruction Supporting the Force XXI Training Program (ARI RR 1729) [DTIC 

ADA359252]. Alexandria, VA: U.S. Army Research Institute for the Behavioral and 

Social Sciences. 

Wigginton, D. & O'Brien, L. H. (1991). Task Analysis for the Combat Vehicle Command and 

Control (CVCC) System (ARI RP 91-13) [DTIC ADA240292]. Alexandria, VA: U.S. 

Army Research Institute for the Behavioral and Social Sciences. 

Wisher, R. A., Champagne, M. V., Pawluk, J. L., Eaton, A., & Curnow, C. K. (1999). Training 

Through Distance Learning: An Assessment of Research Findings (ARI TR 1095) [DTIC 

ADA368592]. Alexandria, VA: U.S. Army Research Institute for the Behavioral and 

Social Sciences. 

B-8 
Wisher, R. A. & Kincaid, J. P. (1989). Personal Electronic Aid for Maintenance: Final 

Summary Report (ARI RR 1516) [DTIC ADA210348]. Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

B-9 
Appendix C. Matrix of Topics and References 

In the matrix below, each of the reviewed articles and reports is shown with the topics that it

addresses. Readers who want more information on particular topics can thus limit their search to 

the most relevant sources. References that are bolded are particularly relevant, and are also 

summarized in Appendix D. 

REFERENCES 

(See Appendix B for full citation) 

 Theme StrategyAnalysisObjectivesDesign DevelopmentImplementation Evaluation MeasurementFeedbackPSS Acquisition FeaturesNontraining Research

Anderson et al. (2000) X

Andre et al. (1997) X X

Atwood et al. (1991) X

Atwood et al. (1994) X X

Barnett & Holmquist (2001) X X X

Barnett et al. (2001) X

Bonk & Wisher (2000) X X

Brown et al. (1998) X X

Burnside (1990) X

Campbell & Deter (1997) X X X X X

Campbell & Holden (2001) X X X X

Campbell et al. (1995) X X X X X

Campbell et al. (1997) X X X X X X

Campbell et al. (2000) X X X

Campbell, Deter, et al. (1999) X X X

Campbell, Ford, et al. (1998) X X X

Campbell, Graves, et al. (1998) X X X X

Campbell, Pratt, et al. (1999) X X

Clagg et al. (1999) X X X X X

Collins & Throne (1996) X

Cox et al. (2002) X X

Deatz & Campbell (2001) X X X

Deatz et al. (1998) X

Deatz et al. (2000) X X X

Department of the Army (1999) X X X X X

Dierksmeier et al. (1999) X X X X X X

Dudley et al. (2001) X

Elliott et al. (1996) X

Endsley et al. (2000) X

Fallesen et al. (1992) X

Finley et al. (1996) X

Finley et al. (2000) X X X X X X X

Flynn et al. (1998) X

Ford, J. P. et al. (1997) X

Ford, J. P. et al. (1998) X X X

Ford, L. A. et al. (1998) X X X X X

Gossman et al. (1999) X

Gossman et al. (2001) X X

Gossman et al. (2002) X X X X X

Graham & Matthews (1999) X X X

C-1
RE

FERENCES 

(See Appendix Bfor fullcitation) 

Implementation MeasurementDevelopmentNontraining Acquisition ObjectivesEvaluation FeedbackResearchFeaturesStrategyAnalysis Theme Design PSS 

Graves & Myers (1997)X

Graves et al. (1997)XXXXXX

Graves et al. (1999)XXXXX

Greess (1994)XX

Heiden (1994)X

Hoffman (1997a)XX

Hoffman(1997b)X

Hoffman et al. (1995)XX

Holden et al. (2001)XXXX

Holland &Wisher (1990)X

Jenkins et al. (1999)X

Jones (1999)XXX

Kern et al.(1994)X

Koger et al. (1996)X

Koger et al. (1998)XXXX

LaJoie &Sterling (1999)X

Leibrecht et al. (1993)XXX

Leibrecht et al. (2002)XX

Lickteig & Collins (1995)XX

Lickteig & Emery (1994)X

Lickteig & Quinkert (2001)XXX

Lickteig & Throne (1999)X

Lickteig et al. (1998)X

Lickteig et al. (2002)XXX

Lussier etal.(2000)X

Meliza (1996)X

Meliza (in press)XX

Meliza &Knerr (1991)X

Morrison & Hammon (2000)X

Moses (2001)X

Nelson &Akman (2002)XX

Olmstead(1992)XX

Quensel et al. (1999)XXXX

Salter &Black (1998)XXXX

Sanders (1999)XX

Sanders (2001)XX

Sanders & Burnside (2001)X

Shlechter & Finley (2000)XXXXXXXX

Shlechter et al.(1987)X

Shlechter et al. (1995)X

Smith et al. (2002)X

Sterling & Quinkert (1998)XXX

Sticha et al. (2002)XXX

Sulzen (1996)X

Throne & Burnside (2003)X

Throne & Lickteig (1997)XX

Throne et al.(1999)X

Throne et al. (2000)XX

C2
REFERENCES 

(See Appendix B for full citation) 

Implementation MeasurementDevelopmentNontraining Acquisition ObjectivesEvaluation FeedbackResearchFeaturesStrategyAnalysis Theme Design PSS 

Wampler & Livingston (1998) X X

Wigginton & O'Brien (1991) X X

Wisher & Kincaid (1989) X

Wisher et al. (1999) X

C-3
Appendix D. Explication of Selected Methods and Findings 

In the following pages, 24 methods, processes, and summaries of findings are presented.

The presentations are brief, providing only outlines of the information. They are included

to give the reader basic information on what direction and guidance can be found in

selected reports and publications. These summaries are not intended to give complete 

guidance or data, but rather to let the reader know what to expect when the full reference 

is consulted. 

D-1
D.1 Proposed Roadmap for Multi-Skilled Soldier (MSS) Achievement 

In order to transform the current soldier to a multi-skilled soldier, the Army is faced with making a series 

of major decisions and initiating a set of key planning and implementing actions. This set of decisions 

and actions is captured in an MSS Roadmap (Nelson & Akman, 2002), which indicates pivotal steps 

required in the short-, near- and long-term to emplace a functioning MSS system by 2008. The Roadmap 

actions are associated with three umbrella organizational groupings: the Army Leadership, the Personnel 

Community, and the Training Community. In the context of the Roadmap, the Training Community

refers primarily to U.S. Army Training and Doctrine Command (TRADOC) Headquarters (East and 

West), the training centers, and the subordinate system of branch and functional schools. That portion of

the Roadmap, taken from Nelson and Akman, is outlined below. 

Near-Term (2002-2004) Mid-Term (2004-2005) Long-Term (2006-2008) 

Training Policy Actions

• Develop concept for future training

paradigm (and scope). 

• Determine changes to leadership 

development paradigm. 

• Consider impact on RC. 

• Determine future role of branches. 

• Determine changes to training base. 

• Determine master transition

concept for Training Community. 

• Begin implementing master

transition plan in earnest. 

• Adjust training base as planned.

• Review Objective Force 

requirements. 

• Adjust master plan as required. 

• Prepare Army as a whole for 

training changes.

• Publish supporting training

documents. 

Training Planning and Development Actions 

• Redesign basic combat training

(BCT) and advanced individual

training (AIT).

• Determine additional skill modules. 

• Design unit-of-assignment training. 

• Develop DL support concept.

• Determine transition resources. 

• Develop construction plan. 

• Develop re-stationing plan. 

• Develop transition plan for the

Training Community.

• Develop plan to acquire training 

developers in sufficient numbers. 

• Develop DL support plan. 

• Test DL methodology. 

• Conduct behavioral research in

selected supporting topics. 

• Program transition resources. 

• Start required construction.

• Develop supporting DL courses.

• Start implementing transition plan

for training base.

• Test new training paradigm courses 

with control groups.

• Develop initial entry base courses. 

• Develop additional skill modules. 

• Develop unit of assignment

courses.

• Develop DL courses. 

• Develop plan to modify leadership

courses.

• Increase number of training

developers. 

• Continue behavioral research. 

• Position training personnel, 

materials, and equipment for 

full-scale execution.

• Refine plans for eventual 

NCO training of MSS. 

• Fully resource training base 

for execution. 

• Complete required

construction.

• Expand/test DL courses and

methodology. 

• Adjust leadership courses to 

prepare for MSS. 

• Continue behavioral research. 

Transition Training Actions 

• Develop Training transition plan. • Prepare initial training base cadre. 

• Set up required courses. 

• Test all courses with larger 

and larger control groups. 

Reference:

Nelson, J. T., II & Akman, A. (2002). The Multi-Skilled Soldier Concept: Considerations for Army 

Implementation (ARI SR 2002-06). Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

D-2
D.2 Development of Structured Simulation-Based Training Programs 

One of the earliest training development efforts for simulation networking (SIMNET) was also one of the 

earliest examples of large-scale development of structured training. Campbell, Campbell, Sanders, Flynn, 

and Myers (1995) assembled the lessons learned and numerous examples from the project into a

methodology, based on the ISD method, for developing the program, associated TSPs, and training

guidance for exercise observers and support personnel. A separate section of the methodology report 

addresses transitioning existing training for new requirements. Numerous job aids, in the form of 

developer worksheets, are also included. The development process for the training itself is described 

briefly below; the process for developing support personnel guidance is outlined in the following annex. 

Phase Activities Evaluation

1. Document

Initial Decisions 

Decision areas: 

• Target training audience. 

• Training context – mission type, enemy, terrain/locale, unit 

type.

• Simulation technology. 

• Other, including exercise time, trainer resources, entry points...

Proponent review

2. Design

Training

Objectives

2.1 Identify task sources, tasks, and standards.

2.2 Refine task list according to simulation support. 

2.3 Select tasks that support the mission. 

Proponent reviews;

internal review

3. Design

Scenario and 

Exercise

Structure 

3.1 Design the scenario – mission and higher echelon mission.

3.2 Prepare exercise context and specifications, including

sequencing and table structures. 

3.3 Outline events and build exercises. 

Map exercise; 

simulationcontrolled trials; 

pilot test with

knowledgeable 

personnel

4. Develop TSP 4.1 Prepare TSP components for exercise support and O/Cs.

4.2 Prepare TSP components for the unit in training. 

Tryouts (pilot tests 

and trials), expert 

reviews

Reference:

Campbell, C. H., Campbell, R. C., Sanders, J. J., Flynn, M. R., & Myers, W. E. (1995). 

Methodology for the Development of Structured Simulation-Based Training (ARI RP 95-08). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-3
D.3 Expanded Procedure for Development of Structured 

Simulation-Based Training Programs 

During construction of battalion staff exercises for Force XXI, Campbell and Deter (1997) updated and 

modified an earlier methodology (Campbell, Campbell, Sanders, Flynn, & Myers, 1995) to make it 

suitable for use in developing a broad range of exercises in live, virtual, or constructive simulation 

environments. The expanded guidance also contains additional examples of TSP components. The 

procedure is outlined below.

Phase Activities Evaluation

1. Document

Initial Decisions 

Decision areas: 

• Target training audience. 

• Training context – mission type, enemy, terrain/locale, unit 

type.

• Simulation technology. 

• Other, including exercise time, trainer resources, and entry

points.

Proponent review

2. Design

Training

Objectives

2.1 Identify task sources, tasks, and standards: 

2.2 Refine task list according to simulation support. 

2.3 Select tasks that support the mission. 

Proponent reviews;

internal review

3. Design

Scenario and 

Exercise

Structure 

3.1 Design the scenario. 

3.2 Prepare exercise context and specifications. 

3.3 Outline events and build exercises. 

Map exercise; 

simulation-controlled

trials; pilot test with

knowledgeable personnel

4. Develop TSP 4.1 Design TSP structure.

4.2 Prepare TSP materials.

Tryouts (pilot tests and

trials), expert reviews 

References:

Campbell, C. H. & Deter, D. E. (1997). Guide to Development of Structured Simulation-Based 

Training Programs (ARI RP 97-14). Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

Campbell, C. H., Campbell, R. C., Sanders, J. J., Flynn, M. R., & Myers, W. E. (1995). 

Methodology for the Development of Structured Simulation-Based Training. (ARI RP 95-08). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-4
D.4 Development of Small Group Exercises 

In the process of developing 24 small group exercises, or vignettes, for brigade staff members, Campbell, 

Ford, Campbell, & Quinkert (1998) also delineated the procedure followed, which can be used in creating

additional exercises. The procedure is a special application of the broader methodology found in 

Campbell and Deter (1997), which addresses development of structured training exercises in general. The

report contains examples and explanations that specifically address the construction and evaluation of 

small group structured exercises. The procedure is outlined briefly below. 

Phase Activities Evaluation

1. Prepare Initial 

Design

1.1 Document initial scope decisions. 

1.2 Select vignette event and participants. 

1.3 Outline vignette architecture. 

Internal review, 

proponent review

2. Design Vignette 

Scenario 

2.1 Analyze event context and participant performance 

requirements. 

2.2 Determine vignette scope and primary participants. 

2.3 Determine vignette inputs and methods for delivery.

2.4 Describe specifications of tactical products.

Proponent reviews;

SME reviews and

walkthroughs

3. Prepare Detailed 

Training

Objectives

3.1 Identify sources for performance requirements and 

performance standards.

3.2 Draft specific vignette training objective and performance 

requirement statements.

3.3 Draft after action review (AAR) aids. 

Internal reviews,

proponent reviews 

4. Develop TSP 4.1 Prepare TSP components.

4.2 Assemble the TSP.

4.3 Evaluate vignette and TSP.

Proponent review, 

tryouts (pilot tests and

trials), expert reviews 

References:

Campbell, C. H., Ford, L. A., Campbell, R. C. & Quinkert, K. A. (1998). A Procedure for 

Development of Structured Vignette Training Exercises for Small Groups (ARI RP 98-37). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Campbell, C. H. & Deter, D. E. (1997). Guide to Development of Structured Simulation-Based 

Training Programs (ARI RP 97-14). Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

D-5
D.5 The Staff Performance Analysis Process and Results 

Campbell, Ford, Campbell, & Quinkert (1998) present a process for discovering staff tasks not included 

in standard doctrinal references. In the Staff Performance Analysis ([SPA], diagrammed below), project 

staff systematically explored the performance of brigade staff activities required in the conduct of the 

three selected missions. Activity exploration was conducted through mission enactment, or roleplay

events, by SMEs in brigade operations, followed by introspective probing after mission enactment. Thus, 

both immediate and briefly delayed recollections of physical and cognitive processes served as the 

foundation of task identification. The SMEs identified and constructed staff products to be used in 

refining the scenario storyline and preparing the AAR structure and materials. 

1. Select scenario 

events

2. Observe the event

3. Construct an

event timeline 

4. Specify interaction 

nodes

5. Identify decision 

points

6. Identify products 

7. Conduct probes 

Mission analysis: Starts with receipt of

order, ends with briefing to commander. 

Detailed written record of observed behaviors 

each event.

0019: S2, S3, and FSO meet to discuss

identification.

0132: Staff decides on course of action 

recommendation to commander. 

Product: Overlay of the course of action and 

decision support template. 

Task lists for selected brigade staff officers,

cues and products.

Timeline

0000 0019 0110

0010 0030 0132 

STEPS EXAMPLE OUTCOMES

Note: S2 – Intelligence Officer; S3 – Operations Officer; FSO – Fire Support Officer

Reference:

Campbell, C. H., Ford, L. A., Campbell, R. C. & Quinkert, K. A. (1998). A Procedure for 

Development of Structured Vignette Training Exercises for Small Groups (ARI RP 98-37) . 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-6
D.6 Digital Skill Sets

The following table presents 22 digital skills divided into four basic categories (adapted from Table 5 in 

Barnett, Meliza, & McCluskey, 2001, p. 13). Barnett et al. identified these skills, which cut across 

specific digital systems and software versions, by reviewing available documentation on how digital 

systems were to be operated. Their report also provides a description of each skill and the source of the 

concept for each. 

Digital Skill 

Categories Skills 

Network Skills 1. Prepare for, and recover from, system crashes or other periods of non-availability.

2. Establish and check communications links and network connections. 

3. Protect network from operator error and malfunctions. 

4. Perform periodic checks of digital systems.

Basic Operator Skills 5. Prepare and update plans, reports, and other messages. 

6. Exchange data with external databases. 

7. Create, modify, and employ overlays, templates, and graphics. 

Basic User Skills 8. Assess completeness of information on the tactical situation.

9. Assess currency of information on the tactical situation.

10. Assess completeness and clarity of planning products.

11. Coordinate with others to acquire information.

12. Identify situations where a physical terrain reconnaissance is required. 

13. Monitor changes in planning products.

Exploitation Skills 14. Maintain awareness of own unit relative to threats. 

15. Compare expected and actual status of friendly units.

16. Maintain awareness of trigger events and events addressed by execution matrices. 

17. Use situational awareness data to move to a vehicle or control measure location.

18. Use situational awareness data and terrain analysis tools to select routes and 

positions. 

19. Use situational awareness data to control unit movement and deconflict routes. 

20. Use situational awareness data and terrain analysis tools to predict contact variables

and support battlefield operating system (BOS) integration.

21. Monitor timing of planning activities. 

22. Define rehearsal objectives. 

Reference:

Barnett, J. S., Meliza, L. L., & McCluskey, M. R. (2001). Defining Digital Proficiency 

Measurement Targets for U.S. Army Units (ARI TR 1117). Alexandria, VA: U.S. Army Research 

Institute for the Behavioral and Social Sciences. 

D-7
D.7 Digitization Applied to Unit Performance Problems 

The following table presents eight categories of unit performance problems and how they can be

addressed, at least partially, through digitization (adapted from Table 4 in Barnett, Meliza, & McCluskey,

2001, p. 10). Barnett et al. also present over 200 specific problems that fall into these eight categories and 

whether improved battlefield visualization, increased operational tempo (OPTEMPO), or both will

address each problem. In general, the benefits of digitization that are likely to address a problem in a 

category will also address other problems in the same category.

Problem Categories Digitization Potential

Lack of awareness of some aspect of the tactical 

(friendly or threat) situation

Battlefield visualization (increased situational awareness). 

Lack of synchronization (within or across BOSs)

in terms of time, space, or activities

Battlefield visualization (increased situational awareness 

and wargaming tools) and increased OPTEMPO (sharing

of evolving plan). 

Lack of awareness of some aspect of the plan or 

lack of input to the plan by a BOS or sub-unit

Increased OPTEMPO (sharing of evolving plan). 

Details missing from plan Increased OPTEMPO (sharing of evolving plan). 

Lack of understanding of the tactical situation Battlefield visualization (increased situational awareness 

and wargaming tools). 

Key elements of the plan produced late Battlefield visualization (increased situational awareness) 

and increased OPTEMPO (sharing of evolving plan). 

Inadequate mission preparation Battlefield visualization (increased situational awareness) 

and increased OPTEMPO (sharing of evolving plan). 

Unit is highly vulnerable or lacks lethality Battlefield visualization (increased situational awareness 

and wargaming tools). 

Reference:

Barnett, J. S., Meliza, L. L., & McCluskey, M. R. (2001). Defining Digital Proficiency 

Measurement Targets for U.S. Army Units (ARI TR 1117). Alexandria, VA: U.S. Army Research 

Institute for the Behavioral and Social Sciences. 

D-8
D.8 Method for Analysis of Back-Up Requirements 

Ford, Campbell, and Cobb (1998) discuss training issues associated with digitization, specifically those 

that address a need to train and maintain back-up skills along with digital skills. Their study investigated 

issues applicable to a wide range of digital applications and training conditions and examined individual 

and collective tasks that support tank operations. A summary of their method for analysis of digital and 

back-up requirements is provided below. 

Define the digital performance 

domain

Delineate tasks in terms of digital or nondigital performance; types of tasks 

include – 

a. Individual tasks

b. Leader tasks

c. Crew tasks

d. Platoon tasks

e. Company/team tasks 

f. Battalion/task force tasks 

Categorize performance type Categories include – 

a. Communicate information – exchange of information

b. Equipment operation – button-pushing procedures

c. Gunnery – acquiring and destroying targets

d. Maintenance (diagnostics only) – identification of vehicular faults 

e. Navigation and movement – locating a point or determining a course to

a position and moving to that location

f. Situational awareness – seeing and understanding battlefield

information

Analyze back-up requirements Identify documented tasks that can serve as back-up and those back-up 

requirements that have not yet been documented for – 

a. Individual tasks

(1) Conventional

(2) Workaround

(3) Repair/replace system

b. Leader, crew, and collective tasks

Reference:

Ford, L. A., Campbell, R. C., & Cobb, R. M. (1998). Analysis of Emerging Digital and Back-up 

Training Requirement (ARI SR 98-07). Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

D-9
D.9 Training Objectives for Battalion and Brigade Sections and Leaders 

Quensel et al. (1999), in developing training for brigade and battalion commanders and staffs, adapted 

work of Olmstead (1992) to compile training objectives focused on cognitive competencies, as described 

below. 

Training Objective Description

Monitor unit operations Each section actively seeks information about higher, adjacent, support, and subordinate units.

Each section acquires information by listening to reports and asking for needed information. 

Process information and Each section collates, transforms, and organizes information. 

messages Each section stores information on maps, situation boards, journals, and files. All information can 

be retrieved and used.

Analyze/evaluate

information

Each section attaches meaning (speculative or confirmed) to acquired information.

Communicate mission

critical information

Each section transmits information or intelligence to those who must make decisions about or act on

it. This includes initial transmittal of sensed information; relaying; and disseminating throughout the

staff, command posts, subordinate units, supporting units, and higher headquarters.

Coordinate information

and intelligence

Each section exchanges and discusses information and intelligence with others outside the section to

clarify meaning and determine implications. 

Integrate staff input The Executive Officer (XO)/Battle Captain (BC) aids the commander’s battlefield awareness by: 

combining information and intelligence from all staff sections, 

putting information and intelligence into a useable format, and 

passing information and intelligence to the commander. 

The XO/BC identifies areas requiring staff sections to combine efforts to support the commander’s 

intent. 

Recommend a course of

action

XO/BC and staff sections develop and analyze courses of action. 

XO/BC recommends a course of action to the commander.

Disseminate

commander’s decision

The staff prepares and issues orders or fragmentary orders (FRAGOs) to inform units and staff of

commander’s decision. 

Synchronize activities of

units

The XO/BC and each section monitor subordinate and supporting unit and BOS assets to ensure

their efforts are aligned to execute the commander’s intent or direction.

Direct assets to support 

commander’s intent

The XO/BC and each section track activities of BOS assets and intervene, if required, to ensure their 

activities support the commander’s intent. 

References:

Olmstead, J. A. (1992). Battle Staff Integration (IDA Paper P-2560). Alexandria, VA: Institute for 

Defense Analyses. 

Quensel, S. L., Myers, W. E., Koger, M. E., Nepute, J. T., Brewer, J. D., Sanders, J. J., 

Crumley, K. A., & Sterling, B. S. (1999). Development of a Refined Staff Group Trainer (ARI RR 

1735). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-10 
D.10 Performance Objectives Methodology

To prepare for a CTC or other large-scale exercise, a unit must identify the collective activities to focus

on that will give them the greatest return on the time and effort invested in their preparations. The 

performance objective effort (Jenkins, Graves, Deter, & Quinkert, 1999) focused on critical techniques 

and procedures and provided flexibility to tailor the exercises to address specific unit training needs. The 

objectives target high-payoff performance and procedures, collective behaviors that require interaction 

and synchronization, and the techniques and procedures that are known to experts but not contained in 

published doctrine. They are usually multiechelon, and cover whole series of events and activities that

occur over an entire exercise or a significant part of an exercise as the situation is developed. The basic 

process for developing performance objectives is outlined below. 

Step Activities

1. Identify Candidate 

Topics

Review sources:

• Lessons learned from CTCs, discussions with CTC personnel and unit leaders, TTP 

publications, battlefield function (BF) analyses, etc. 

• Down-select based on criticality, acceptability, feasibility, and nonredundancy.

• Outline selected objectives in terms of who, what, when, what, how, why.

• Clearly define objective and supporting tasks.

• Indicate appropriate training participants. 

• Observable within scenario and simulation environment. 

• Consistent with doctrine. 

2. Prepare techniques

and procedures 

Prepare guidance on how the objective can be achieved, based on literature review, 

roleplay enactments, expert input. 

3. Integrate 

techniques and 

procedures in TSP 

Prepare event sheets to identify prompts, cues, messages and instructions to cue 

activities. 

Prepare performance objective observation guidance with assessment guide (indicators

of performance). 

Reference:

Jenkins, S. N., Graves, C. R., Deter, D. E., & Quinkert, K. A. (1999). Development of the COBRAS 

III Performance Objectives for the Brigade and Battalion Staff Exercise (ARI RP 99-03). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-11 
D.11 Train-the-Trainer: Developing Guidance for Observers, Controllers, 

and Interactors 

Campbell, Campbell, Sanders, Flynn, & Myers (1995) documented a process for preparing guidance for 

observers, controllers, and interactors supporting structured simulation-based training programs. 

Phase Focus Areas

1. Analysis Training requirements – based on

• Role (observer, controller, interactor, or combined). 

• Content and intent of the exercise. 

• Training objectives. 

• TSP materials for preparation, execution, and feedback. 

• Simulation.

Audience 

• Level of experience re training, simulation, unit type, and mission. 

• Organizational structure of training support personnel (if any). 

• Numbers of support personnel training participants. 

Resources 

• Time allotted for train-the-trainer sessions. 

• Delivery media and methods. 

• Instructional staff personnel. 

2. Design Training outline

• Objectives. 

• Sequence and time allotted to each objective. 

• Training events.

Observer/Controller/Interactor TSP outline. 

• Handbook and job aids. 

Training media and methods plan

• Simulator/simulation and multimedia support.

• Classroom, classroom plus hands-on, or distributed. 

• Individual or group. 

3. Development Prepare TSP materials.

Pilot test the training with representative support personnel; revise as needed. 

4. Implementation Plan

• Prepare agenda (aka program of instruction [POI]). 

• Select facilities and instructors. 

• Identify training material quantities needed. 

• Notify participants. 

Prepare 

• Reserve and set up facilities. 

• Rehearse instructors.

• Copy materials. 

• Confirm training schedule with participants. 

Execute 

• Do it. 

D-12 
Phase Focus Areas

• Reevaluate agenda and training plan continually (daily). 

• Consolidate loose ends daily.

• Observe training for evaluation purposes. 

5. Evaluation Formative Evaluation

• Interviews, questionnaires, and observations on the Train-the-Trainer sessions.

Summative Evaluation

• Interviews, questionnaires, and observations on the implementation of the 

structured training program.

Reference:

Campbell, C. H., Campbell, R. C., Sanders, J. J., Flynn, M. R., & Myers, W. E. (1995). 

Methodology for the Development of Structured Simulation-Based Training (ARI RP 95-08). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-13 
D.12 High Impact Transfer Technology

The process below, derived from Finley, Sanders, and Ryan (1996, p. 235) describes a two-phased 

training development process that incorporates training transfer principles in developing courses of 

instruction. 

Phase Procedure

1. Task generalization:

Analysis phase 

a. Prepare objects and verb lists. 

b. Develop generalized components and objects lists. 

c. Describe generic action statements. 

d. Describe knowledges and skills groups associated with the generic action

statements. 

2. Generic design a. Develop and sequence generic terminal learning objectives. 

b. Develop learning specifications. 

c. Develop syllabus. 

d. Develop lesson materials.

Reference:

Finley, D. L., Sanders, M. G., & Ryan, A. J., III (1996). Application of Training Transfer Principles

in Developing the High Transfer Training (HITT) Methodology. Innovations in Education and 

Training International, 33, 232-239. 

D-14 
D.13 Training Support Code Methodology

Burnside (1990) developed a rule-based method for assessing Army Training and Evaluation Program

Mission Training Plan (ARTEP-MTP) standards that can be met and subtasks and tasks that can be

performed in training devices and simulations. The training support code (TSC) method was applied to

assess the performance capabilities of the Simulation Networking (SIMNET) system, and results 

indicated that the method could be used to identify key tasks for training effectiveness and to assess the 

effects of system enhancements in terms of additional tasks to be trained. The method has also been 

applied to identify tasks for training in the Close Combat Tactical Trainer (CCTT). A summary of the 

assessment method is provided below. 

Analysis Step Procedure 

1. Task Framework Determination of the tasks to be considered

2. Standard Criteria Rating of degree to which each task element or standard can be performed or 

met in the simulation: 

a. Highly supported – standard can be met entirely.

b. Partially supported – standard can be met to a large extent. 

c. Minimally supported – standard can be met to a limited extent. 

d. Outside support required – standard can be met, but at least half of

required actions must be performed outside the simulation.

e. Not supported – standard cannot be met.

3. Decision Rules Consolidation of ratings into subtask and task assessments through application 

of decision rules (see Burnside, 1990 for list of decision rules)

4. Review and

Coordination of 

Ratings 

Submission of ratings to SMEs for review

Reference:

Burnside, B. L. (1990). Assessing the Capabilities of Training Simulations: A Method and 

Simulation Networking (SIMNET) Application (ARI RR 1565). Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

D-15 
D.14 Evaluating Virtual Environment Training Capabilities 

Sticha, Campbell, and Knerr (2002) developed a methodology for estimating whether the capabilities of 

virtual environment technology are sufficient to allow training of a specified set of activities. The 

methodology, known as Satisfying Training Requirements in Virtual Environments (STRIVE), modifies 

and extends the TSC method by incorporating a simple behavioral analysis that is focused on the aspects 

of military tasks that are likely to present difficulty to virtual environment training. 

The STRIVE methodology was presented in the context of the Integrated Definition (IDEF0) system

analysis. The methodology is described below in terms of activities, what they produce, what they use for 

input, and how they are constrained by other information. 

Activity Description

Select Analysis

Level 

This activity determines whether the elements rated in the analysis should be tasks, task

steps, performance measures, or some combination of these three. 

Input: List of tasks, each broken into task steps and performance measures. 

Activity: SME decides, based on task documentation, which tasks can be rated as

single units, which tasks can be rated at the task step level, and which steps need to be

broken down further to individual performance measures. 

Output: List of tasks, task steps, and performance measures that will be rated.

Select Relevant 

Technology 

Dimensions 

In this activity, the SME answers some general questions about the training domain. 

The answers are used to select the technology dimensions that will be evaluated. 

Input: Eight dimensions that describe capabilities of virtual environment technology: 

• Visual display resolution.

• Manipulation of terrain. 

• Manipulation of equipment. 

• Scene complexity (if operating dismounted). 

• Gesture recognition (if operating dismounted). 

• Tactile/force cues (if operating dismounted). 

• Ability to operate both mounted and dismounted.

• Speech recognition.

Activity: Describe performance requirements of the tasks.

Output: Decision on relevant capabilities needed. 

Determine 

Available 

Capability

The purpose of this activity is to determine whether there are any design constraints

that might affect the performance that is possible using virtual environment technology. 

Input: Relevant capabilities (from previous activity)

Activity: Describe design constraints in terms of speech recognition, transportability,

and reconfigurability. 

Output: Identification of available capabilities 

D-16 
Activity Description

Rate Task-Element 

Coverage 

In this step the SME answers several questions regarding each of the task elements to 

be rated. Each question relates to one technology dimension. A score is calculated that 

represents the extent to which the capabilities of that dimension support the activities

conducted in the rated task element. 

Input: Task elements, relevant and available capabilities. 

Activity: Answer questions specific to the relevant dimensions for each task element. 

Output: Rating of performance support for each task element. The rating is the 

minimum for all of the dimensions rated for the element. 

Aggregate Ratings

to Task Level 

The method uses a rule system to determine numeric scores for each task, task step, and 

performance measure based on the ratings computed in the previous step. SME input is

then applied to refine the score. 

Input: Rating of performance support for each task element.

Activity: SMEs indicate, for each task step, the importance of training that step in a 

virtual environment; these judgments are used as weights in computing task trainability

ratings. 

Output: Task trainability ratings, consisting of weighted and aggregated support

ratings. 

Reference:

Sticha, P. J., Campbell, R. C., & Knerr, C. M. (2002). Training Concepts for Virtual Environments

(ARI SR 2002-05). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social

Sciences. 

D-17 
D.15 Identifying Core Set Exercises 

Gossman, Graves, Mauzy and Clagg (2001) examined the management and assessment of user-produced 

TSPs to provide the basis for design of TSP development tools. In addition to describing TSP

components and TSP assessment, distribution, and maintenance requirements, they also specify a 

methodology for identifying exercises that would comprise a core set for different user audiences. The 

methodology is described below. 

Step Description

1. Specify Unit Type Identify the unit for which the core set is being specified, in terms of branch and

echelon. 

2. Select Viable

Training Methods 

Determine viable training methods, in terms of specific exercise types and training

environments, for the unit. Figure 4 in the report shows the domain of exercise

types and environments, and Table 1 demonstrates rationales for determining

viability.

3. Identify Training

Focus

Beginning with a domain list of possible training content (e.g., the appropriate 

MTP), filter the list by deciding if each task should be the focus of an exercise or

trained in the context of another task. Group tasks into combinations that can be

sequenced into realistic battlefield storylines by identifying how tasks actually occur 

and identifying tasks that can be trained together. 

4. Identify 

Specifications for 

Collective Training 

Analyze storylines (Step 3) and viable methods (Step 2) to identify the full set of

possible types of exercises by determining whether each method supports the tasks

and whether the resulting training would be efficient. 

5. Develop a Core

TSP Set

Identify the specific exercises that comprise the core set, based on the analysis in

Step 4. by selecting from among the possible methods and tasks. (The eventual goal 

is to have TSPs for all storylines and methods identified in Step 4, but it would be

impractical to develop all of the exercises at one time.) 

Reference:

Gossman, J. R., Graves, C. R., Mauzy, R. P., & Clagg, R. A. (2001). Assessing and Managing 

User-Produced Training Support Packages (ARI RR 1772). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences. 

D-18 
D.16 Training Program Conversion 

Graves et al. (1999) took another look at the basic methodology for developing structured simulationbased training (Campbell, Campbell, Sanders, Flynn, & Myers, 1995), modifying it to address the need 

for converting existing analog training to digital. The conversion approach was then implemented for 

three specific conversion instances; the report provides details on the process, descriptions of the 

products, and lessons learned. The basic process, as further revised to address conversion efforts in 

general, is outlined below.

Step Activity

1. Conduct Front-End 

Analysis

Define the “old” (e.g., conventional) training in terms of –

• Environment (simulated mission, enemy, terrain, troops, time, and civilian

considerations [METT-TC]). 

• Purpose of the training.

• Training audience. 

• Training conditions (specific training environment). 

Define the “new” (e.g., FCS) training in terms of – 

• Environment (simulated METT-TC). 

• Purpose of the training.

• Training audience. 

• Training conditions (specific training environment). 

2. Develop the 

Conversion Plan

Document 

Identify procedures and conditions

• Identify areas for content changes.

• Identify components to be modified. 

• Identify procedures for specifying and implementing conversions.

3. Design and Develop

New Product

Execute the Conversion Plan

• Specify design and structure. 

• Develop TSP.

• Conduct formative evaluation.

References:

Campbell, C. H., Campbell, R. C., Sanders, J. J., Flynn, M. R., & Myers, W. E. (1995). 

Methodology for the Development of Structured Simulation-Based Training (ARI RP 95-08). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

Graves, C. R., Pratt, D. M., Campbell, C. H., Allen, J. D., Thorson, K. G., Jenkins, S. N., &

Quinkert, K. A. (1999). Force XXI Training Program - Digital Project: Report on Development 

and Lessons Learned (ARI RR 1748). Alexandria, VA: U.S. Army Research Institute for the 

Behavioral and Social Sciences. 

D-19 
D.17 Refined Training Analysis and Feedback Center of Excellence 

Concept 

The refined Training Analysis and Feedback Center of Excellence (TAAF-X) (Anderson, Begley, Arntz,

& Meliza, 2000) helps clarify CTC and homestation responsibilities. This refined concept assigns 

functions that the TAAF-X TAF analysts are responsible for, clarifies the roles of the analysts at the local 

tactical analysis facility (TAF), specifies the echelons supported at CTCs and homestations, and 

establishes assumptions necessary to make the TAAF-X viable. The division of labor is presented below, 

followed by a summarized list of potential problems that would prevent implementation of TAAF-X and 

their solutions (both adapted from Anderson et al.). 

Division of Labor in the Refined TAAF-X Concept

TAAF-X Analysts • Collect TAF analyst requirements for AAR products. 

• Prepare and transmit AAR products to TAF analysts for O/C presentation.

• Produce take home package based on input from TAF analysts and O/Cs. 

• Publish and deliver take home package to rotational units. 

• Provide Center for Army Lessons Learned (CALL) access to rotation data. 

CTC/Homestation 

TAF Analysts

• Perform all necessary exercise control functions. 

• Coordinate with O/Cs for AAR requirements from AAR products list and forward 

AAR products requirements to TAAF-X for construction. 

• Review and send final AAR products from TAAF-X to O/Cs for use in AAR 

presentation.

• Compile O/C and analyst comments and observations then forward to TAAF-X for 

inclusion in take home package. 

O/Cs • Assist the local analyst in exercise control functions. 

• Notify the supporting analysts what type of AAR product they need. 

• Provide input to changes and refinements to AAR products list as required. 

• Mentor and coach unit players.

Potential TAAF-X Implementation Problems and Possible Solutions 

Potential Problem Possible Solution Strategy

Analysts lack time to support

additional units

Strategy 1: Implement automation strategies described in the advanced

tactical engagement simulation concepts (ATESC), cognitive requirements

for information operations training (CRIOT), and advanced after action

review media (A3RM) reports. 

Differences in AAR systems 

among sites

Strategy 1: This problem will be addressed with the fielding of the CTC 

objective instrumentation system (OIS). 

Delays in communication with 

TAAF-X 

Strategy 1: Provide an automated system at the CTCs with limited AAR 

production capability, such as for statistical aids and for text slides. 

Strategy 2: Require each O/C to create his own AAR product for inclusion in

the AAR. Equip him with a laptop, presentation software and a way to 

send/receive e-mail. He would create his products and send them to the AAR

facility for senior O/C review and inclusion/deletion.

Selection of AAR aids from

standard list reduces O/C 

Strategy 1: The automated system for preparing “standardized” menus of

AAR aids should be given an editing capability that matches the variety of 

D-20 
Potential Problem Possible Solution Strategy

flexibility changes O/Cs are likely to make. 

Strategy 2: Be prepared to have TAAF-X analysts spend a certain portion of

their time preparing requested, unique AAR aids for an exercise. (The O/C

requesting the aid could possibly identify a new type of aid that TAAF-X

adopts if it meets an information need not being met by existing aids.) 

Distractions in homestation 

training will lead to inefficient 

use of TAAF-X 

Strategy 1: TAAF-X can use down time resulting from cancellation of

exercises to prepare lessons learned reports in cooperation with CALL. 

Strategy 2: Reinforce the chain of commands commitment to reduce training

distractions at all levels within units. Leaders from the Division Commander 

level down to Squad Leader level should put more emphasis on training,

ensuring nothing interferes with planned training. 

Unit leaders may not want

information about their unit’s

performance going to a central

site

Strategy 1: Encrypt all data from the instrumentation system (IS), tactical 

engagement simulation (TES), and CTC TAF/Homestation TAF 

workstations. 

Strategy 2: Run dedicated lines from TAAF-X to all supported installations. 

Strategy 3: Use both Strategy 1 and Strategy 2. 

The need to consider differences

in range, resource constraints,

and operational equipment 

among training sites may 

overwhelm TAAF-X analysts

Strategy 1: Restrict the number of training areas with which each analyst 

normally works.

Strategy 2: Prepare database with training area specific information

(locations normally used for defensive positions, etc.). 

Strategy 3: Limit the number of exercises an analyst supports during a given

period to one. 

Procedures and AAR techniques 

vary greatly among sites

Strategy 1: Coordinate with CTC and homestations to determine a standard 

presentation sequence for AARs. Train all O/Cs and TAF analysts in AAR

presentations in the same manner. 

Strategy 2: Re-establish AAR presentation guidelines and doctrinal formats 

to ensure specific procedural requirements are maintained. 

Difficult to adjust TAAF-X 

staffing to match workloads 

Strategy 1: Create a TAAF-X cell for each exercise, coordinating local 

analyst duty times and manning with TAAF-X duty times and manning.

Strategy 2: Man the TAAF-X around the clock, using a three or four-shift

rotation of personnel with equal numbers of analysts in each shift. 

Terminology for AAR aids and

organization of TAF cells vary 

among CTCs

Strategy 1: The standardization of the terminology and organizations of the 

TAF cells at the CTCs and homestations will improve the TAAF-X’s ability

to support all units. 

Potential lack of homestation 

analyst training

Strategy 1: Homestations should provide continuity in terms of the personnel 

tasked to serve as local analysts to reduce the costs associated with analyst 

training.

Potential lack of homestation O/C 

training

Strategy 1: At a minimum, homestation O/Cs should receive a brief

introduction to the types of AAR aids available from TAAF-X and their 

utility (e.g., traces of unit movement provide a fast way to show a unit 

wandered or backtracked during movement). 

Lack of an O/C and homestation

analyst habitual relationship 

Strategy 1: Assigning personnel to work as local analysts for extended 

periods will help to alleviate this problem. 

D-21 
Potential Problem Possible Solution Strategy

Lack of a TAAF-X analyst and

homestation analyst habitual

relationship 

No strategy identified. 

Potential lack of TAAF-X analyst 

training

No strategy identified. 

TAAF-X Analysts may be

overwhelmed with AAR product 

requests 

Strategy 1: Disperse start times for exercises assigned to a TAAF-X analyst 

or TAAF-X analyst cell to minimize overlap during the most frantic AAR aid 

preparation periods.

Strategy 2: Provide tools to help TAAF-X analysts track AAR aid

preparation activities for multiple exercises concurrently. 

Strategy 3: Provide some level of AAR preparation capabilities at local 

training sites (local analysts will need workstation to perform exercise control 

functions).

Communication of AAR 

requirements to TAAF-X will be 

difficult if voice is the only

communication media

Strategy 1: Video teleconferencing capabilities would improve the 

communication capabilities among O/Cs, local analysts and TAAF-X

analysts. 

Local analysts will be distracted 

by requirement to act as link

between O/Cs and TAAF-X

Strategy 1: Video teleconferencing capabilities would improve the 

communication capabilities among O/Cs, local analysts and TAAF-X

analysts. 

Inexperienced O/Cs will impose a

heavy AAR aid preparation

requirement on TAAF-X

No strategy identified. 

Reference:

Anderson, L. B., Begley, I. J., II, Arntz, S. R., & Meliza, L. L. (2000). Training Analysis and 

Feedback Center of Excellence (TAAF-X) (ARI SR 2001-01). Alexandria, VA: U.S. Army

Research Institute for the Behavioral and Social Sciences. 

D-22 
D.18 Strategies to Reduce O/C Workload 

Brown, Nordyke, Gerlock, Begley, and Meliza (1998) studied the requirements for exercise control and 

training feedback at CTCs and homestation. They found that the workload for trainers is becoming

increasingly intensive, and identified 13 strategies for reducing or managing O/C and Training Analysis 

Facility workload. 

Strategy Description

1 – Automate NLOS

battle damage

assessment

Providing a non line-of-sight (NLOS) simulation modeling capability would require 

installation of sensors on actual indirect fire platforms to determine the tube

azimuth/elevation and the type ammunition fired, but would remove the burden of

observation and simulation manipulation from the analyst. 

2 – Pair Designator to 

Target Designated 

Similar to Strategy 1, for remotely designated engagements. 

3 – Pair Shooter to

Misses 

Combining the pairings of hits/kills collected by the instrumentation system with the 

misses collected by the line-of-sight (LOS) simulation model can assist TAF analysts

in assessing the distribution and massing of blue forces (BLUFOR) direct fires.

4 – Reduce 

pyrotechnics

expended for NLOS 

battlefield effects

Extends Strategy 1 to a model that depicts virtual images of indirect fire on the live

battlefield; integrates a heads-up virtual visor into the combat vehicle command

(CVC) helmet to portray virtual 3D visual effects and incorporates sound as well. 

5 – Overcome

limitations of laser 

technology 

Virtual mirroring could produce virtual firing signatures and terminal effects for LOS 

and NLOS engagements using the virtual visor and headphones. This will eliminate 

laser transmitters and receivers, most pyrotechnic requirements, and reduce feedback

requirements. 

6 – Provide a Virtual

opposing force 

(OPFOR) 

Inserting a virtual OPFOR on a live battlefield has great potential for homestation live

training, where maneuver area restrictions and live units to serve as OPFOR severely

limits the scope.

7 - Provide tactile 

feedback

Participants need devices that provide tactile feedback in response to the varying

degrees of energy received form non-lethal weaponry. 

8 – Automate C4I

data collection and 

control 

To provide extrinsic command, control, communications, computer, and intelligence 

(C4I) feedback, the instrumentation system must capture all digital traffic transmitted

and received by every player node and filter the information for significant BLUFOR 

digital actions or inactions. 

9 – Automate tracking

of player activities

and expended 

resources

By tracking the use and consumption of simulated resources from the support area to

final destination, O/Cs and analysts are able to track activity and use of resources. 

10 – Automate TES

system monitoring

Automating TES system testing will provide more consistent testing, reduce O/C 

workload, and enhance the credibility of the exercise. 

D-23 
Strategy Description

11 – Automate AAR 

preparations

By determining typical O/C and analyst subjective assessments and identifying typical 

AAR aids that support each assessment, a knowledge base can be developed that 

contains algorithms and AAR formats. 

12 – O/C control,

observation, and

future battle 

command brigade 

and below (FBCB2)

Workstation 

O/Cs need a mobile workstation to support their control and observation requirements. 

The workstation should support the O/C’s control and coordination needs and provide 

the O/C the capability to monitor the unit’s digital activities.

13 – TAF analyst 

workstation

An instrumentation system that archives 2D and 3D computer-generated imagery,

video, audio, C4I digital data, and all email and digital reports exchanged among O/Cs

and analysts are needed. 

Reference:

Brown, B. R., Nordyke, J. W., Gerlock, D. L., Begley, I. J., II, & Meliza, L. L. (1998). Training 

Analysis and Feedback Aids (TAAF Aids) Study for Live Training Support (ARI SR 98-04). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-24 
D.19 Evaluating TSP Components

When units produce TSPs, either by creating a new one or by modifying an existing one, several levels of

assessment may be needed. Gossman, Graves, Mauzy, and Clagg (2001) present an approach to 

assessment that defines the levels and the criteria for assessment at each level. The first three levels

should be conducted for any user-produced TSP; the last two come into play only if the TSP is a 

candidate for distribution to other units. The approach is outlined below. 

Assessment

Level Criteria 

Unit assessment

– conducted by

commander or 

parent 

organization 

Is the exercise ready to run?

• Does the exercise match the unit’s training needs?

• Is the exercise tactically and doctrinally correct?

• Are the mission essential task list (METL) tasks and supporting collective and individual

tasks addressed adequately?

• Does the exercise utilize the training environment best suited to the unit’s training needs?

• Are the resources adequate to conduct the training?

• Are safety and environmental issues addressed?

• Are sound training principles incorporated?

Site assessment

– conducted by

simulation site

personnel or

system operators

Is the exercise technologically correct?

• Are initialization conditions correct?

• Are timings correct?

• Is entity count within bounds?

• Do events and effects occur as planned?

Execution 

assessment –

conducted by

commander and 

other users

How did the exercise work when executed?

• Did events and effects occur as planned?

• Did the TSP have all necessary information?

• Were cues adequate to require unit performance as planned? 

Proponent 

assessment –

conducted by

TRADOC 

organization

responsible for 

distribution

Does the exercise meet criteria for inclusion in the centralized database/repository?

• Is the exercise tactically and doctrinally correct?

• Are sound training principles incorporated?

• Is this exercise substantially different from others in the database?

• Are all necessary TSP components included?

• Is the exercise packaged properly for electronic distribution?

Selection 

assessment –

conducted by

unit commander 

prior to

adopting an

exercise from

the repository 

Does the exercise significantly meet unit training needs?

• Does the exercise match the unit’s training needs?

• Is the exercise tactically and doctrinally correct?

• Are the METL tasks and supporting collective and individual tasks addressed adequately?

• Does the exercise utilize the training environment best suited to the unit’s training needs?

• Are the resources adequate to conduct the training?

• Are safety and environmental issues addressed?

Reference:

Gossman, J. R., Graves, C. R., Mauzy, R. P., & Clagg, R. A. (2001). Assessing and Managing 

User-Produced Training Support Packages (ARI RR 1772). Alexandria, VA: U.S. Army Research

Institute for the Behavioral and Social Sciences. 

D-25 
D.20 Training Effectiveness Analysis Methods 

Morrison and Hammon (2000) classify training effectiveness analysis methods by the research operations 

used and the types of data collected. The three broad categories are described below; the report also 

includes examples of the use of each method. 

Method Description

Training Surveys Usual method for obtaining user reactions, where users include not only trainees, but

also managers, unit commanders, and trainers. 

Empirical 

Performance-Based 

Research Designs 

Most highly respected approach for measuring training effectiveness but also the most

resource intensive. Three basic types:

1. Randomized (true) experiments – require random assignment of participants to

experimental treatments (e.g., training approaches), as well as pre- and posttraining measurements. Allows the researcher to make strong inferences about 

cause-effect relationships between the training conditions and the performance 

measures. Includes: 

a. Learning experiments – used to determine the amount of learning that occurs

as a direct result of training on the simulation system, by comparing pre- and 

post-training test scores. 

b. Transfer experiments – used to assess the degree to which learning on the 

training system transfers to and improves performance in the operational 

setting.

2. Quasi-experimental designs – involves systematic use of intact comparison groups

rather than random assignment to experimental conditions. 

3. Correlational studies – involves passive observation or assessment of levels of

training, rather than control of experimental conditions. 

Training Analyses Method for estimating the training effectiveness of training systems. A key 

characteristic is reliance on nonemperical data (i.e., data other than those derived from

actual performance or user input). Includes: 

1. Training system estimation models – based on the assumption that a training

system can be evaluated in terms of its elements and that the benefits of a training

system can be understood by studying those elements. These methods are usually

complex and costly. 

2. Modeling – refers to the analyst’s attempts to represent accurately and 

economically the key characteristics of the training system. 

3. Analogy – by which the analyst infers the effectiveness of a training system by

examining the effectiveness of a similar system. 

4. Extrapolation – prediction based on some understanding about how a process

works.

5. Task list analyses – evaluation of training devices in terms of the tasks they can

train.

Reference:

Morrison, J. E. & Hammon, C. (2000). On Measuring the Effectiveness of Large-Scale Training

Simulations (IDA Paper P-3570). Alexandria, VA: Institute for Defense Analyses. 

D-26 
D.21 Diagnostic Indicators of Digital Proficiency

Meliza (In Preparation) presents 15 diagnostic indicators of digital proficiency that cut across specific 

digital skills. For each indicator, Meliza also provides several tasks that are indicative of level of

proficiency. The indicators, along with brief descriptions, are provided below. 

Indicators of Basic Digital 

Proficiency Description 

1. Taking actions to ensure 

connectivity. 

Maintaining a high level of connectivity requires command emphasis

down through the ranks. A unit that falls too low with respect to this 

activity is digitized in terms of equipment but not in terms of linkages 

or behaviors.

2. Displaying confidence in the 

robustness and value of digital 

systems. 

The unit will take actions to allow a rapid return to digital operations if

there are crashes if the effort required for digitized operations is

viewed as being worth the value obtained from the system. 

3. Using digital systems to ensure 

timely distribution and review

of planning products. 

The amount of effort a unit puts forth to put planning products in front

of decision makers and unit leaders as soon as possible is an indicator 

of digital proficiency.

4. Appreciating the value of 

digital capabilities. 

The greater the perceived contributions to tactical performance, the 

more likely that individual systems are ensured to be operational. 

5. Using digital systems to apply 

tactical task triggers. 

A potential benefit is the capability to employ trigger events that can be

observed using situational awareness (SA) displays. If a unit does not

employ digital systems during rehearsals, it is unlikely that the 

rehearsal will reveal any information about possible digital triggers. 

6. Ensuring adequate monitoring

of digital displays. 

Strategies for periodically checking SA displays and/or delegating the 

responsibility are an important indicator of digital proficiency.

7. Tailoring views of the battlespace to fit visualization

objectives. 

Systems need to be tailored to meet leaders’ information requirements. 

Indicators of Advanced Digital

Proficiency Description 

8. Using terrain analysis tools to 

support tactical decisions. 

A critical variable appears to be understanding what decisions can be 

supported by software and what decisions require a physical terrain

reconnaissance. 

9. Integrating digital systems

into rehearsals. 

As a unit gains more experience, digital systems may provide a means of

showing what the tactical situation will look like during mission execution.

10. Using digital displays to 

monitor/control unit 

movement. 

Leaders can use digital displays to decide whether their subordinates are 

moving to the correct location; using the appropriate formation, at an

appropriate speed; maintaining an appropriate degree of dispersion; and 

moving and halting when they should.

11. Actively maintaining

awareness of the threat 

situation. 

As a unit progresses digitally, digital contact reports with geo-referenced

icons can be expected. The more a unit employs digital capabilities to track

the threat situation, the more work a unit has to perform.

D-27 
Indicators of Advanced Digital

Proficiency Description 

12. Using digital tactical SOPs to

facilitate communication and 

visualization. 

Without SOPs to guide file-naming conventions, it will be difficult to

manage information, maintain a network, and create a common picture of

the tactical situation.

13. Using effective mixes of 

reporting modes. 

In many cases, units should use digital messages in combination with voice 

messages.

14. Delegating digital tasks to 

ensure adequate employment

of digital systems. 

Delegating responsibility of tasks that use digital systems is a good 

indication that a unit is digitally proficient. 

15. Organizing digital data to 

facilitate communication. 

Until units progress to the point where they push electronic copies of

planning products down to company and platoon levels, there is little data to

organize.

Reference:

Meliza, L. L. (In Preparation). Digital Skills and Indicators of Overall Digital Proficiency. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-28 
D.22 Factors Impacting Digital Proficiency

Meliza (In Preparation) describes five variables that influence how far a unit progresses in terms of 

digitization. Changes in the variables below are expected to impact digital proficiency.

Variables Affecting Progress in 

Unit Digitization Questions to Answer Before Training

1. Changes in digital system capabilities 

a. Increased robustness

b. Features better designed to meet users’

needs

Are system operators aware of changes in capabilities and

application problems between the current version and last version

of the digital system?

Does the version of the digital system used in the training 

situation match the current operational version used by the unit?

2. User understanding of and appreciation 

for digital capabilities 

c. Greater tactical benefits

d. Requirement to maintain high rate of

connectivity among digital systems 

e. Requirement for scenario situation that 

sets the stage in order to gain certain

benefits

What capabilities do users plan to employ?

What capabilities are users unlikely to employ?

3. Digital training strategies What training have leaders and soldiers had regarding system

capabilities?

What capabilities have been used and rewarded in the context of

an exercise?

4. Digital SOPs Does the unit have SOPs addressing conventions for naming

planning products?

Does the unit have SOPs addressing setting filters and other 

digital options?

Does the unit have SOPs regarding the delegation of digital tasks 

to specific duty positions or individuals?

5. Legacy behaviors Do unit leaders use digital line-of-sight tools?

At what point does the staff begin including planning products

within the digital network?

Reference:

Meliza, L. L. (In Preparation). Digital Skills and Indicators of Overall Digital Proficiency. 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-29 
D.23 Usage Considerations for Embedded Training and Performance 

Support Systems 

Although many researchers discuss use of ET or EPSS, there is no comprehensive listing of when these 

methods are appropriate. Throne and Burnside (2003) present a consolidated list of considerations, 

gathered from various sources. Since there are numerous similarities between these two types of

performance enablers, suggestions gathered from EPSS and ET writings are likely to apply to both fields. 

Embedded training and EPSS are most useful when some or all of the following conditions exist: 

• Computers will be used regularly in task performance. 

• A majority of the work is mental rather than physical. 

• Personnel turnover is high.

• Current training is insufficient.

• Training costs need to be cut. 

• Job performance needs improvement.

• Supporting information is difficult to access. 

• System users are geographically dispersed. 

• On-the-job training is important. 

• The user will be engaging in complex tasks. 

• The users are not closely supervised. 

• Mistakes in task performance are costly. 

• The organization is downsizing. 

• Users have diverse learning styles.

• Users need access to experts to perform their jobs. 

• Information and technology explosions have occurred. 

• Expectations for performance are high. 

Reference:

Throne, M. H. & Burnside, B. L. (2003). Integrated Training and Performance Support for the 

Objective Force (ARI RR 1801). Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

D-30 
D.24 Design Guidelines for Embedded Training and Performance Support 

Systems 

Once the decision has been made to use ET and EPSS, training developers will need a set of guidelines

for building appropriate performance enablers. The guidelines below, compiled by Throne and Burnside 

(2003) from writings on ET and from the FCS system training plan ([STRAP] draft, August 2002), are 

based on capabilities that should exist for success in both EPSS and ET efforts. Embedded training and 

EPSS capabilities should have the following characteristics:

• Be fully compatible with the prime system. 

• Be as fully integrated with the prime system as possible. 

• Not endanger personnel, equipment, or data through incorrect operator actions.

• Not compromise security of the system or its data. 

• Have minimal impact on the prime system’s reliability, availability, or maintainability.

• Not lead to negative transfer in the operational mode. 

• Enable combined arms and joint, interagency, and multi-national (JIM) proficiency.

• Leverage UA network architecture.

• Allow on-demand access (reach) to doctrine and TSP repositories. 

• Provide collaborative wargaming tools, supporting mission planning and rehearsals, while deployed or 

deploying.

• Provide required tactical engagement simulation for the full range of weapons, including electronic 

warfare. 

• Provide CTC-like instrumentation: data capture, management, and analysis, along with rapid feedback. 

References:

Throne, M. H. & Burnside, B. L. (2003). Integrated Training and Performance Support for the 

Objective Force (ARI RR 1801). Alexandria, VA: U.S. Army Research Institute for the Behavioral 

and Social Sciences. 

Unit of Action Maneuver Battle Laboratory. (2002). Operational Requirements Document for the 

Future Combat Systems (Draft, 30 Aug 02). Fort Knox, KY: Author, U.S. Army Armor Center and 

Fort Knox. 

D-31 
D.25 AAR Aids

Brown, Nordyke, Gerlock, Begley, and Meliza (1998), in their study of requirements for exercise control

and training feedback at CTCs and homestation, developed concepts for a number of AAR aids. The 

automated AAR system would allow the analyst to access the aids in the database that support the O/C’s 

subjective assessment. As the library of aids grows in number and sophistication, the system could 

categorize assessments by tactical mission, BOS, operational activity (e.g., planning, execution). Nine

such aids are illustrated in the report (pp. 80-86) for a battalion task force assault, and are described 

below. 

AAR Aid Description 

1 – Task Standards The performance measures for the task or task steps that most closely matches

the subjective assessment. 

2 – Mission and Execution

Plan

A short statement of the mission and a graphic (top-down) depiction, with

graphic control measures, of the plan for execution.

3 – Battle Outcome Bar graph showing BLUFOR and OPFOR operational systems at the

beginning and end of the battle. 

4 – Direct Fire Distribution Shows BLUFOR direct fire distribution during assault on the objective, along

with candidate questions the O/C may ask to generate discussion. 

5 – Voice Fire Commands Replays clips of the BLUFOR assault from a top-down view, with

synchronized audio from the tactical voice net, isolating fire commands issued 

by selected commanders. 

6 – Indirect Fire Distribution Graphic (top-down view) of distribution of smoke and casualty-producing

munitions, with candidate AAR questions. 

7 – Video of Smoke Graphic (out-the-window) portrayal of smoke distribution, revealing whether

BLUFOR smoke obscured the vision of the OPFOR vehicles, with candidate

AAR questions. 

8 – Effectiveness of Support

by Fire Elements

Line graph showing effectiveness in suppressing OPFOR fires and protecting

the assaulting elements, with candidate AAR questions. 

9 – Coaching Guide Coaching points to support more directive assistance from the O/C on how to

improve performance. 

Reference:

Brown, B. R., Nordyke, J. W., Gerlock, D. L., Begley, I. J., II, & Meliza, L. L. (1998). Training 

Analysis and Feedback Aids (TAAF Aids) Study for Live Training Support (ARI SR 98-04). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-32 
D.26 Cognitive Learning Strategies 

Sanders (2001) provides a summary of cognitive learning strategies identified in the literature on 

cognitive processes. A key feature of these strategies is that the learner is an active participant in the 

learning process. By offering multiple learning strategies within a training program, students can better

achieve their learning goals. Listed below are the nine cognitive strategies (adapted from Sanders, p. 12). 

Learning Strategies Description 

Chunking Rational ordering, classifying, or arranging of complex arrays preparatory to deeper 

processing. 

Frames 1 A matrix, grid, or framework for representing knowledge organized by a spatial strategy.

Facilitates comparisons and contrasts. Presents holistic cognitive structure for material. 

Frames 2 A matrix organized by some lawful principle, allows use of logic or inference to fill in

missing information. 

Concept Map Graphics reveal the structural pattern in the material, provide the “big picture.” Aids the 

recall of detail. Computers good in allowing flexible presentation of material and

relationships. 

Advance Organizer Presented prior to new material. A brief transition statement identifying connections

between prior learning and new material. 

Metaphor Sensitizes student to similarities between something known and something new. Helps 

learner transfer prior knowledge to another topic. Helps sensitize learner to similarities 

across knowledge arrays. 

Rehearsal Activities to keep material active in consciousness to support transition into short-term

memory. Supports deeper processing for long-term recall. Gets learner actively,

intellectually engaged in processing information, not just passively exposed to information.

Imagery Cognitive strategy that aids recall. Mental pictures learner forms to aid recall. A key way

of storing information in the mind, images can hold large amounts of information and great 

detail. 

Mnemonics Artificial aids to memory used with material low in structure until meaningful mental

structures developed. 

Reference:

Sanders, W. R. (2001). Cognitive Psychology Principles for Digital Systems Training. (ARI RR 

1773). Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-33 
D.27 System Improvement Recommendations 

In 1998, a concept experimentation program (CEP) was conducted by the Mounted Battlespace Battle 

Lab (now the Unit of Action Maneuver Battle Laboratory) to assess battle command information 

requirements and military decision making in the 2010-2015 timeframe. Lickteig, Sterling, Elliott, Burns, 

and Langenderfer (1998) collected recommendations on how to improve C4I systems to better meet battle 

command information requirements from the CEP participants; they are presented below. 

Mission Component Recommendations on Information Requirements 

Warning Order • Standardized format with more detail. 

• Ability to manipulate the detail provided. 

Operations Order • Ability to extract or parse most relevant portions of a higher echelon order. 

• Wider dissemination of the brigade level operations order. 

Operations Overlay • Standard library of icon symbology. 

• More integrated C4I tool set to fuse information sources. 

• Integration of all mission related information into dynamic map overlays and displays. 

Commander’s Intent • Clear delineation of priorities and end state. 

• Formats that provide clear understanding of the intended end state of the mission. 

Course of Action • Information formats that delineate and coordinate the sequence of actions. 

• More continuous linkage for Whiteboard-type communications to clarify and adjust

activities during the course of operations. 

Fragmentary Orders • More continuous linkage for whiteboard-type communications. 

Enemy Component Recommendations on Information Requirements 

Location • More extensive feed from higher echelons. 

• Dynamic linkage and match of enemy situation templates to depicted enemy locations. 

Composition • Additional information on vehicle and unit types. 

• Manual labeling capability to indicate composition.

Disposition • Better information on enemy location and composition.

Ability to See • Graphic depiction of “dead space” associated with enemy vehicles or icons provided as

required. 

Ability to Move • More valid information on enemy’s ability to move. 

Ability to Shoot • Information on enemy weapon ranges. 

Ability to

Communicate 

• Information to identify vehicles with multiple antennas. 

• Information on estimated radio ranges adjusted for terrain. 

Ability to Sustain • Mission design entailing combat service support.

• More extended mission operations. 

Terrain Component Recommendations on Information Requirements 

Observation • Database to support requirements for observation.

Cover • Cover information in more challenging relief.

D-34 
Terrain Component Recommendations on Information Requirements (continued) 

Concealment • Accurate terrain databases. 

Obstacles • Depiction of information required on manmade obstacles. 

Key Terrain • Capability to explicitly identify or mark key terrain features.

Approach Avenues • Provision of approved graphic symbols for approach avenues.

• Ability to annotate and delete graphics as required. 

• Automatic analysis and identification of approach avenues, adaptable to unit size. 

Weather • Scenario designs other than clear daylight conditions. 

Troop Component Recommendations on Information Requirements 

Location • Distinctive color scheme for friendly icons and symbols to ease tracking.

Organization • Labels indicating unit affiliations. 

Ability to See • Higher magnification for terrain database manipulation.

Ability to Move • Information to improve understanding of trafficability.

Ability to Shoot • Ability to directly observe firings by direct and indirect systems on the tactical display

(i.e., engagement updates).

Ability to

Communicate 

• Ability to analyze radio ranges and assess need to retransmit. 

• More immediate communication, particularly for e-mail messages. 

Ability to Sustain • Inclusion of logistics. 

• Inclusion of combat service support.

Time Component Recommendations on Information Requirements 

Plan • Timelines for task force operations. 

• Capability to analyze time requirements or monitor progress in relation to timeline. 

Wargame • Automated capability to wargame potential courses of action.

Prepare • Established timeline. 

Rehearse • Automated capability for unit to rehearse. 

Execute • Decision point tool. 

• Input and display decision points on tactical display.

• Decision points coupled with automated check and feedback before designated decision

points reached. 

Synch Execute • Digital version of synchronization matrix with dynamic format and automated updates. 

Reference:

Lickteig, C. W., Sterling, B. S., Elliott, G. S., Burns, J. & Langenderfer, J. (1998). Assessing Battle

Command Information Requirements and the Military Decision Making Process: An Exploratory 

Investigation for the Mounted Maneuver Battlespace Laboratory (ARI RR 1731). Alexandria, VA: 

U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-35 
D.28 Leader Insights on Digitization and Transformation 

Leibrecht, Johnston, Black, and Quinkert (2002) conducted a series of structured interviews with senior

leaders in the Army’s First Digital Division and used the information to prepare guidance on managing

change in a number of focal areas. A brief summarization of selected insights, addressing the process of 

managing Force XXI transition as well as the transition’s impact on tactics and doctrine, is shown below 

(adapted from pp. 24-44). 

Managing Transition Summary of Insights

Command 

Climate – 

Setting the 

Unit Tone 

Create the unit as a learning organization where discovery and new ideas are prized. 

Accept risk as a positive factor, encourage people to think differently.

Employ systems thinking: achieve efficiencies by setting priorities, rethinking all aspects of

change – doctrine, leadership, soldiers, budgets, etc. 

Command 

Climate – 

Motivating

Leaders and 

Soldiers 

Leaders need to mentor and empower their key people, demonstrate acceptance of

digitization.

Let people know that input is valued and feedback is being used. 

Manage expectations – early versions of systems are forerunners of the better ones. 

Major training events can be forcing functions, giving the unit anchor points for concentrating 

energy. 

Command 

Climate – 

Setting and 

Protecting Unit

Priorities

By establishing main and supporting axes, leaders can balance needs of the unit with needs of

system developers, experimentation proponents, etc. 

Responding to the stress of high workload during transition by neglecting organizational

management functions can cause major problems down the line. 

Command 

Climate – 

Working With 

Others

During transition, there will be many stakeholders and interested observers, and the leaders 

need to be able to team with those outside the unit in order to remain knowledgeable about

and responsible for solders being trained and ready.

Support organizations and agencies need guidance from unit leaders on how they can best

support the unit. Routine collaboration can help. 

Contractors who are working to introduce systems and operators who need to learn the 

systems need to work closely to facilitate transition.

With tests and experimentation, there can be conflicts and misunderstandings about how

training events are conducted.

Exploiting

Digital 

Technologies

Although the initial learning curve for digitization can be steep, sound tactical skills and

leadership talent remain the leader’s greatest assets. 

Leaders must push for the system functionality they need. 

Committing to successful transition causes leaders and soldiers to find ways to make new

doctrine and systems work. 

Forceful methods may be needed to jump-start transition, such as removing the means for 

doing things “the old way.”

To rein in the impact of system evolution, forceful control of software changes is imperative;

leaders must communicate how software change impacts training, scheduling, and competing

requirements. 

D-36 
Managing Transition Summary of Insights

Infrastructure 

for Change – 

Army Culture 

Groups with less history of change are slower to accept newer systems and will experience

more frustration, which can be mitigated by experience-sharing among units.

Preserving the “discovery” environment by delaying the rush to performance standards can

actually help soldiers address the challenge of transition.

Infrastructure 

for Change – 

Enablers

Managing and implementing transition demands full-time dedicated personnel and

transformational leaders who will focus on new ways of doing things. 

Innovative options for training and professional development will be necessary to keep up

with rapid change. 

Personnel Leaders must continue to be warfighters, not technicians, and will need progressive

assignments within the new environment.

Mentoring for junior staff officers can encourage their excitement and development of vision. 

Young officers and soldiers, with high levels of comfort in computer environments, are 

crucial to transformation, but it should not be assumed that they will be comfortable with

change. 

Long term (greater than a year) personnel stabilization can be effective in ensuring sufficient

train-up before major events and an orderly transition afterwards. Additionally, continuous

infusions of personnel generate a greater training and assimilation burden as constants such as

standard equipment and consistent SOPs are less common.

With a streamlined force, 100% fill is indispensable. 

Training –

Fundamentals 

Large-scale training events can stress soldiers and leaders in ways that force them to exploit 

their new systems.

Training in a transitional unit must address basic tactical (legacy) tasks as well as tasks on the 

new systems.

Four factors that determine the rate of unit learning: frequency of repetitions, amount of

enabling learning between repetitions, personnel stability, and consistent focus. 

In early stages of transition, the train-assess-train cycle must be repeated to identify what 

needs to be learned – especially with new equipment and organizational structures. 

Major transition events, including experimentation and testing, must be used as training

events to advance readiness.

Training –

Special Groups

During transition, special efforts are needed to train NCOs because many senior NCOs are not 

comfortable with the technology, and if they do not understand about skill acquisition and

decay; it can hinder the transformation process.

System training should be tailored to the position and responsibilities of the user but should

also be conducted in the system-of-systems context. 

Unit 

OrganizationReorganization

Process 

Reorganization, inevitable in transition, is a high-risk enterprise for the initial units because

they have to determine how well the new structure works, and their feedback may lead to

additional changes. 

There will be times during transition when reorganization is overlapping system changes,

doctrine changes, and training developing, and the confusion is unavoidable. It does pass,

however.

A reorganized unit needs to invest energy in establishing new SOPs, and their need to be

checks that the procedures are working.

D-37 
Managing Transition Summary of Insights

Unit 

Organization – 

Impact of 

Reorganization

People-oriented leaders find ways to implement reorganization without creating negative

impact on individual warfighters (e.g., phased and sequenced company transitions with ample 

information). 

Fielding New

Systems 

Experimentation brings non-standard equipment that evolves into modified tables of

organization and equipment (MTOE) property, at which point support and maintenance 

become the unit’s responsibility; this transition needs to be anticipated and planned for. 

Documenting

Critical

Knowledge 

Sharing information is critical, especially because discovery learning is the norm. 

Units undergoing transition play the pivotal role in developing TTP and doctrine. 

Obtaining valid assessment data and lessons learned demands a reasonable level of individual

and collective proficiency on the new systems, which in turn requires a systematic training

plan and a cutoff for system upgrades.

An audit trial of unit transition issues, decisions, and successful and failed attempts to effect

transition is indispensable if new leaders are to benefit from the experience of their 

predecessors. 

Transitional units tend to invest their lessons-learned energy in SOPs, and lessons about 

managing change and measuring progress may be lost.

Impact on

Tactics and 

Doctrine 

Summary of Insights

Impact of 

Technology 

While the new digital tools were welcomed with enthusiasm, they are not magic. New

systems provide better tools, but will not make up for a lack of basic warfighting skills. 

Digital tools’ precision for visualizing enemy elements boosts the ability to recognize patterns

on the battlefield and to set the conditions needed to destroy the enemy.

Enhanced situational awareness enables bold and aggressive maneuver by giving the 

commander the confidence to move quickly and decisively. 

Signal operations may be the Achilles’ heel of digital operations, and redundant signal 

companies may be required to enable repairs and restore connectivity.

Degraded

Operations

Redundancy among systems can enable units to sustain operations even when specific 

technologies fail; when redundancy is not apparent, units will resort to conventional methods. 

It is not difficult to overrun the capacity of emerging systems, which can cause a virtual 

meltdown.

Reference:

Leibrecht, B. C., Johnston, J. C., Black, B. A., & Quinkert, K. A. (2002). Managing Force XXI 

Change: Insights and Lessons Learned in the Army’s First Digital Division (ARI SR 2002-04). 

Alexandria, VA: U.S. Army Research Institute for the Behavioral and Social Sciences. 

D-38 
Appendix E. Research Issues 

The list of what we know, or think we know, is extensive, as evidenced even by the limited

summary provided in this paper. What we don’t know is yet more extensive, and we know

it. Every experiment, study, observation, and program evaluation yields another set of

interesting questions. A sampling of the more immediate needs, and some research-based

guidance on addressing the needs, is presented here. 

Performance

issues 

The current decision-making model may be obsolete. What is the appropriate 

model in FCS that allows decisions to be made more quickly, within the enemy’s 

decision-making cycle? What information does it use? How will it be trained? 

Distributed forces with better information at lower levels will both enable and 

require decision-making at lower levels. How will leaders learn to not only allow

but empower subordinates to adapt to changing situations? 

With distributed teams, leaders may not come face-to-face with subordinate unit 

leaders for days. What will be the effect on command and control, and how can it 

be overcome? 

Command and control of robotic forces will increase the workload. Can one

person handle both live and robotic subordinates? Or, if the robots are largely

autonomous, what will be the effect on command and control?

What methods can the soldier use to control off-board entities (robots)? Are 

multiple control methods necessary? Are they linked to levels of automation? 

Commanders and staffs must thoroughly understand how robotic forces work and

what rules they follow in order to effectively employ them. What level of 

understanding will be sufficient? What is the best human-human and humanautomation authority structure? How will automation affect the human leadership

structure? 

Decision or performance aids will be developed. How will commanders and staffs

develop a balance of trust and logical skepticism concerning the use of these aids? 

What decision tools will commanders need and want? 

“No increase in force structure” for FCS is not a useful mandate. The proper 

context for the discussion is the position that any increase in force structure must 

be justified by increases in the UA's relative combat effectiveness vis-à-vis a 

comparable baseline organization (legacy or Stryker). How will this be judged? 

System issues The FCS will have the capability to assume some decision-making responsibility. 

What decisions can be made by the FCS (if any) and what must be made by the 

human? How does this relate to levels of automation? Does evidence show that 

the FCS features will, in fact, facilitate the soldier’s decision-making? 

Vast amounts of information will be available to leaders, and they may need vast 

amounts for decision-making. Does the volume of information needed exceed the 

soldier’s ability to process the information (information overload)? 

Does the workload under expected combat conditions exceed human capabilities?

How will workload be affected by partial system failures? 

Information input could be diversified across human sensory modalities (e.g., 

visual, auditory, and possibly tactile). Will that help? How well can soldiers

E-1 
integrate multiple sensor feeds from multiple positions and orientations? How can 

displays be designed to complement human sensory and information processing?

Training issues Structured training, both individual and collective, is frequently constructed using

“nested” scenarios. Is there any real value to this approach? How damaging are 

the negative effects? 

How effective is distance learning? Do the benefits (if any) justify the costs? 

How disruptive are the negative effects? 

How should training requirements analyses change for digital tasks and for hybridconventional environments to determine the training needed for producing and 

sustaining high performance soldiers? 

What training methods work best to prepare soldiers for frequent software or 

equipment changes combined with ad hoc and varied missions (including combat

and special assignments)? 

What methods work best to prepare leaders and to train widely dispersed soldiers 

linked as teams and units through digital networks? 

What methods work best to prevent skill decay? To retrain after skill decay? 

What timing of interventions works best? 

What measurement/assessment and feedback tools are needed to ensure that

required skill levels are attained and retained? 

What training should be done how, when, and where using soldier-centered 

distributed/digital environments? 

Methodological 

issues 

Large-scale simulations, for purposes of concept explorations or system

experimentation, are too resource intensive and unwieldy to keep up with the pace

of technological changes. Instead, what is needed is a mix of demonstrations, 

experiments, simulations, exercises, and formal tests. 

Testing and evaluation are hamstrung by the limited availability of soldiers and 

units with which to research and validate training, operational concepts, systems, 

organizational structures, and doctrine. Using simulation cuts down on some of 

the costs, and to be optimally useful the simulations should be reconfigurable to 

permit rapid prototyping.

What is also needed is a designated experimental unit for research, in order to be

able to focus on exploration and discovery of performance techniques. The 

formation of a shadow cadre to do concept exploration and testing would add 

value to the use of user juries. The approach could cycle unit members in and out,

or a best-of-the-best model could be used. 

Investigate the extent to which change-driven knowledge gathered from one 

position in a particular unit can apply to other positions and units (or even be 

useful to others in same position/unit). 

Fully map knowledge requirements of change managers in various positions, 

echelons, and units. 

Apply the knowledge elicitation methodology to establish comprehensive 

multiechelon knowledge bases for other transition arenas (e.g., Stryker). 

E-2