Technical Report 788

!FIL.E COp" i

Design of Battle Simulations for

Command and Staff Training

IC) Robert E. Solick and James W. Lussier

C,_ 

D TC,.!.

'p.

JUN 23 1988

0%,

ARI Field Unit at Fort Leavenworth, Kansas -Z

Systems Research Laboratory

U. S. Army

Research Institute for the Behavioral and Social Sciences

April 1988

Approved for public release; distribution unlimited. 1,],

%

&17

Z."*Z-IZZ
I.i

I

U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

WM. DARRYL HENDERSON

EDGAR M. JOHNSON COL, IN

Technical Director Commanding

Technical review by ..

H. F. Barber NTIS CAj! I.

R. E. Christ [)TIC TAB U S

R. Wright U.enan ¢',d U J ;t". C'.tJ ''

L- -..----- -.

Oric t;h4'v~o ' /

It PE ,~h id , - 5

NOTICES

copy

and ATT Soc .PERI-P,, ences 1 Esenhowr Ave. Ale.adria V

NOTICES "

STalBUTI ' rpirnary r~.ibution of t i report hasj~ made b &RI. Please address . :

and Socil . nc ,-T E IP ieh wrAeAeadiV a2 600.'

FINAL DISPOSITION: This report may be destroyed when it is no longer needed. Please do not

return it to the U.S. Army Research Institute for the Behavioral and Social Sciences. "

NOTE: The findings in this report are not to be construed as an official Department of the Army 54

position, unless so designated by other authorized documents.

.5

-,'- .- ; .-. ., -. , .- ... , , _ , ._. . ..- , ._.- : ." .- '. -'. ,-. -,.. - , .- . . .
UNCLASSIFIED

SSECURITY CLASSIFICATION OF THIS PAJEp REPORT DOCUMENTATION PAGE

Ia. REPORT SECURITY CLASSIFICATION 1b. RESTRICTIVE MARKINGS

Unclassified__________________________

2a. SECURITY CLASSIFICATION AUTHORITY 3. DISTRIBUTION /AVAILABILIT'Y OF REPORT %1

Approved for public release; 2b. DECLASSIFICATION IDOWNGRADING SCHEDULEdstiuon nlmed

4. PERFORMING ORGANIZATION REPORT NUMBER(S) S. MONITORING ORGANIZATION REPORT NUMBER(S)

ARI Technical Report 788 -

6a. NAME OF PERFORMING ORGANIZATION 6b. OFFICE SYMBOL 7a. NAME OF MONITORING ORGANIZATION

ARI Field Unit j (if applicable)

Ft. Leavenworth I PERI-SL -

6c. ADDRESS (City, State. and ZIP Code) 7b. ADDRESS (City, State, and ZIP Code)

P.O. Box H

Ft. Leavenworth, Kansas 66027-0347 -

8a. NAME OF FUNDING ISPONSORING I8b. OFFICE SYMBOL 9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBER

ORGANIZATION (if ?oplicable)

Army Research Institute PERI-S -

ac. ADDRESS (City, State, and ZIP Code) 10. SOURCE OF FUNDING NUMBERS

5001 Eisenhower Avenue PROGRAM PROJECT' TASK ~ WORK UNIT

Alexandria, VA 22 333-5600 ELEMENT NO. NO. 2Q26 3 NO. IACCE

NO.

263744 744A79 1.3.3.

I1I TITLE (includie Security Classification)

Design of Battle Simulations for Command and Staff Training

12. PERSONAL AUTHOR(S)

Robert E. Solick and James W. Lussier

13a TYPE OF REPORT 113b. TIME COVERED 114. DATE OF REPORT (Year, Month. Day) 15. PAGE COUNT

Final Report IFROM __Q/85 TO 08/j86 11988, April i 46

16 SUPPLEMENTARY NOTATION..

17. COSATI CODES 18. SUBJECT TERMS (Continuie nrvre if necessary and Identify by block numnber)

FIELD GROUP SUB-GROUP Command and Control, 4t ,.. .n

Computers, --NBattleSsimulation,

Design- Operatid'ns research, (Continued)

19 ABSTRACT (Continue on reverse if necessary and identify by block number) V5:

* This report summarizes findings from 1() y ears of research at the Armv Research Institute Fort Leavenworth Field Unit on command and staiff training with automated battle SiMLI-

[at ions. Topics include design issues- related to train ing objectives, perf ormance

measurement, simulat ion reqIuiremei~ts, user intert sees,, and workload est imat ion. This

assessment of lessons, learned p rov t- guidae f, Idve pe rs of- t le n~ext ve nerat ion of

hat tle25i~~lt 115

20 DISTRISIUTION /AVdAILABILITY OF ABSTRACT 21 ABSTRACT SECURITY CLASSIFICATION

El UNCLASSIFIED/UNLIMITED 0 SAME AS RPt 0 DTIC USERS lI I 1,0 I ie d

22a NAME OF RESPONSIBLE INDIVIDUAL 22b. TELEPHONE (include Area Code) 22c. OFFICE SYMBOL

Robert 1E- Solick (91 1) 084i-49 H 1 PE.I-SI.

DO FORM 1473. 84 MAR 83 APR edition mnay be used until exhausted. SECURITY CLASSIFICATION OF THIS PAGE

All other editions are obsolete.IN2AS 11

~ p.
UNCLASSIFIED

'SECURIITY CLASSIFICATION OF THIS PAGE

ARI Technical Report 788

18. Subject Terms (Continued)

Performance measurement

Command Group training

t!NCLASS I Ft El)

SSECURITY CLASSIFICATION OF THIS Pk,,E

, ' % / /-'%*% - 4

S54.% % 4 4 %i . ' : :- s '
Technical Report 788

Design of Battle Simulations for

Command and Staff Training

Robert E. Solick and James W. Lussier ,

9%

ARI Field Unit at Fort Leavenworth, Kansas

Stan!ey M- Halpin, Chief

Systems Research Laboratory

Robin L. Keesee, Director

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333-5600

Office, Deputy Chief of Staff for Personnel

Department of the Army-J

April 1988

Army Project Number Manpower, Personnel, and Training

20263744A795

0%

Approved for publi(, release; distribution unlimited.

.
I' p

FOREWORD

The Fort Leavenworth Field Unit of the Army Research Institute for the

Behavioral and Social Sciences (ARI) supports the Combined Arms Center with

research and development on combined arms operations and command group trail:-

ing. With the fielding of the first fully automated systems for command aijn

staff training, the simulation development community is engaged in assessin,

lessons from previous efforts while embarking on the automation of staff

training at high echelons.

As part of Research Task 1.3.3., "Improved Methods for Command Group

Training," the report summarizes 10 years of research on issues from which design criteria may be derived for the new generation of training systems. The

report was prepared under a Memorandum of Understanding, dated 30 May 1985,

between the Combined Arms Training Activity (CATA) and the Army Research Institute. Reccmmendations from the report, which were briefed to the Training

Simulations System Manager of CATA on 30 January 1987, will be incorporated in

requirements for new simulations.

EDGAR M. JOHNSON

Technical Director

'p

A1

,'';., z',,, ,'. ,': .." z,' ... . ,, "n , x .w, .",; .(- ": ' 'i ; '' "" , '' ,' "; ';'; '( ":'.? ,;" ( ' p.,
The authors wish to acknowledge the considerable contributions of Dr. Jon -

J. Fallesen.

a'.

'-

'.a,.

vi a,

a,
--- A

DESIGN OF BATTLE SIMULATIONS FOR COMMAND AND STAFF TRAINING

EXECUTIVE SUMMARY

Requirement:

To provide design criteria for the next generation of automated battle

simulation based on behavioral research on command and staff training.

Procedure:

A systems-approach-to-training model was adapted to the context of command and staff training with automated battle simulations. Results of research on each of the major components of the model were discussed and related

to design goals and criteria that should be considered in the development of

future training systems.

Findings:

Excessive support staff requirements, lack of system support for scenario

development, lack of system control over information and intelligence and lack -

of performance measurement capabilities were found to be recurrent deficiencies in previously developed systems. A new requirement for training with

tactical data systems may ameliorate some of these deficiencies while adding

to the complexity of system software. Proposed solutions include developing a

systemic model to minimize support requirements and using on-line data capturing techniques for performance measurement.

Utilization of Findings:

Application of these lessons learned will allow developers of training

systems to avoid previous problems in user acceptance of systems. Incorporation of the recommendations in the new generation of training systems will

reduce support costs and increase the ability of such systems to assess and

address training needs.

N.

vii

**t 1~ . 7' -a r- . % % . AZ.~. 
DESIGN OF BATTLE SIMULATIONS FOR COMMAND AND STAFF TRAINING

CONTENTS

Page

Section 1. INTRODUCTION .. ......................... 1

Background .. .......................... 1

Organization of Report .. .................... 3

Section 2. TRAINING OBJECTIVES. ...................... 6

ARTEP: Tasks, Conditions, and Standards. ............ 6

Training Audience. ....................... 7

Scenario Development .. ..................... 8

Section 3. DEVELOP PERFORMANCE MEASUREMENT. ............... 10

Battle Outcome Statistics .................... 10

Products and Procedures ..................... 12

Knowledge and Decisions ..................... 13

Section 4. DEFINE SIMULATION REQUIREMENTS. ................. 16

Model Fidelity. ......................... 16

Software Maintenance. ...................... 17

Exercise Preparation. ...................... 18

Control of Training Conditions. ................. 18

Section 5. DEFINE CONTROLLER TASKS.......... ........... 20

Controlling The Exercise. .................... 20

Controlling The Model ...................... 22

Role Playing. .......................... 24

A Summary ............................. 26

Section 6. ESTIMATE CONTROLLER WORKLOAD .. ................ 27

Section 7. CONCLUSIONS AND SUMMARY RECOMMENDATIONS. ........... 32

Training Objectives ....................... 32

Performance Measurement ..................... 32

Simulation Requirements ..................... 33

Define Control Tasks. ...................... 33

Estimate Support Staff Requirements. ............. 33 $ Final Remarks .......................... 34

ix
CONTENTS (Continued)

Page

REFERENCES. ................................ 35

LIST OF FICURES

Figure 1. A typical command staff training configuration ... ....... 2

2. Steps in designing a battle simulation ..... ........... 4

3. Steps in determining staffing requirements . ......... . 28

x

- - i'"' ~ . ~- 5. 5,.-s- S.
DESIGN OF BATTLE SIMULATIONS FOR COMMAND AND STAFF TRAINING

SECTION 1. INTRODUCTION

Simulations designed to train Army commanders and staffs in planning and

conducting milit-iry operations have reached a turning point in their development. Manual simulations at lower echelons (battalion and brigade) have been

supplemented and to some extent supplanted by computer-assisted simulations.

Currently, the Army is fielding the first operational fully-automated simulations. The availability of low-cost high-performance computer systems and

high-fidelity combat models has made it possible to design the next generation

of training devices based upon the desired performance capabilities of the

system rather than being overly constrained by limitations in technology. The

simulation development community is now engaged in assessing the lessons

learned in previous development efforts while embarking upon a much more ambitious plan to automate staff training at high echelons.

The Army Research Institute (ARI) Field Unit at Fort Leavenworth has been

participating in battle simulation research and deyelopment for ten years,

including work on CAMMS, CATTS, ARTBASS, and MACE. During that decade, an

accumulation of experience and research findings has developed that can be

brought to bear upon the design of the next generation of simulations. This

report documents some of the major training issues and design goals that should

be considered in battle simulation development.

Background

The basic features of a systems approach to training are depicted in Figure

1. The training objectives are comprised of a list of tasks that the trainees

should perform, a list of conditions under which they should be performed, and S

a set of standards for performance. In the case of battle simulations, the

training system consists of a simulated battlefield situation that should be

designed to require performance of the tasks included in the training objectives (or a subset of those tasks) while simulating the conditions under which

those tasks would be required. Figure 1 emphasizes the role of feedback in the

conduct of training and in determining which objectives require additional

training.

Feedback is critical for effective learning. A major reason to adopt a

simulation-based training approach is to increase the amount of task feedback

obtained by the trainees. Command groups receive this feedback from the simulation, from subordinates, from superiors, and from other members of the staff.

The Combined Arms Map Maneuver System, the Combined Arms Tactical Training

Simulation, and the Army Training Battle Simulation System, respectively. MACE

(not an acronym) has been renamed BABAS Battalion Battle Simulation.
TRAINING~ SCNA

OBJECTIVE

CONTRL COMANDE

PERSONEL &

STAFF

AI

SIMUATIO TACICA

MODEL DAT

SYSTEM

MEASUEMEN FEDBAC & 

Figue co ma.'d 1.A saff tpica raiing onfguraion

21

or I le FM - .14 's~i'd
Unfortunately, the feedback received is not as immediate nor as accurate as in

simulations designed to train simple judgmental and motor skills, such as

flight simulations. Instead, the root causes of problems are often unknown.

In addition, inappropriate decisions and actions may sometimes lead to success, S

while appropriate behavior may occasionally lead to failure. For these reasons, the intrinsic feedback must be supplemented with an extrinsic assessment

of performance. Furthermore, this measurement requirement may require modifications in the design of the training system.

Figure 1 shows how the general training model is typically applied to command staff training. A set of training objectives, based upon a prior assessment of the training needs of a command group, is used to select and embellish

a scenario which will require the staff to perform the tasks in the set. The

training objectives are also used to develop a plan for observation and measurement of staff performance. The scenario consists of a mission, resources

sufficient to accomplish the mission, an opponent, a recent history of the con- •

flict, a piece of terrain, and an intelligence build-up giving clues to the

size, location, and probable course of action of the opponent. The basic features of the scenario are used by the control personnel to prepare an opera- V

tions order for delivery to the training audience (the commander and staff

being trained), to establish initial conditions in the simulation model and, in

units equipped with tactical data systems, to initialize their data bases as S

well.

During an exercise, the control personnel simulate higher, lower, and adja- r

cent headquarters; they translate command group directives into computer input

formats; they translate computer output into tactical messages; they keep themselves apprised of the tactical situation and of the status of resources in

their functional area; they supply additional model control input for the opposing force; and they participate in observation and measurement of staff

performance.

The commander and staff of the group being trained analyze their mission,

develop a plan, deliver an operations order, and execute their plan. With the

advent of tactical data systems, the staff consults the system for information,

files information from subordinate elements, and provides standard operating

procedure (SOP) reports via digital transmission. The control personnel can % 2

also file SOP reports using the tactical data systems, and, if properly equipped, can access the information residing there to determine what picture of the

battle is currently available to the members of the training audience.

Organization of Report

Figure 2 shows some of the steps involved in developing a simulation-based

training system. The organization of this report is aligned with the steps in

the figure. Although other characterizations are possible, the steps in- •

dicated affect the most basic training decisions - who is trained, what they

are trained to do, how their progress is measured - and the most basic design

decisions - what modeling techniques are used, how much automated support is

provided, how large a support staff is required, and what hardware capabilities

are needed to accommodate the model and the users of the system.

3 %p.

V ... 
DEF INE TRAI N ING AUD IENCE

DETERMINE TRAIN ING OBJECT IVES

DEVELOP PERFORMANCE MEASURES DEFINE SIMULATION REQUIREMENTS

DESIGN MEASUREMENT TECHNIQUES

NIr

DEFINE

CONTROL

TASKS

I r

ESTIMATE APPLY

CONTROLLER CONSTRAINTS

WORKLOAD

Figure 2. Steps in designing a battle simulation.

4

41
The determination of who should be trained in a battle simulation and of

what they should be trained to do is discussed in Section 2 of this report.

The commonplace assumption that every member of the staff group should be

trained simultaneously on all of their combat tasks is critically examined.

The influence of performance assessment techniques upon system design is

emphasized in Section 3. Performance measurement has been neglected in existing simulations. Usually, the more promising approaches to objective, automated measurement and feedback cannot be retroactively incorporated.

Simulation design and fidelity issues are considered in Section 4. A comprehensive analysis of modeling, system usage, and support requirements is

recommended.

The determination of control personnel tasks and estimation of controller

workload are examined in Sections 5 and 6, respectively. The design process is

viewed as iterative, with the key constraints being upon the availability and

qualifications of control personnel.

".

'I"

-'.

75
SECTION 2. TRAINING OBJECTIVES

The basic goal of simulation-based training is to allow members of the

command group to acquire the skills necessary to operate smoothly within the

time and resource constraints they should expect in battle. Ideally, the conduct of each training exercise is guided by explicit training objectives which

specify who is to be trained, what tasks they are being tzained to perform,

under what conditions the tasks must be performed, and what criteria or standards indicate successful task performance. The development of simulation

systems for command and staff training must be guided by an analysis of the

range of training objectives for which the system might be employed.

The systems-approach-to-training (SAT) focuses on the critical tasks that

must be performed to achieve minimal proficiency. SAT is sometimes criticized

as being an incomplete approach to training because of this focus: it does

not emphasize the factors that distinguish true expertise from acceptable competence. For any given group of people who achieve acceptable task performance

in a SAT-based training system, there will be a few individuals who are considered outstanding at their job, a larger group who are considered average, and

some who are marginal. It has been contended that close study of how exceptional performers do things differently in command and control may lead to the

identification of a teachable set of battle management skills that could provide much richer training objectives. Command and control researchers are

beginning to investigate these issues. It remains to be seen whether the results could best be applied as additional selection criteria, as improved

training techniques, or as decision aids. In the near term the SAT approach is

the only viable alternative.

In this section, several implications of the analysis of training objectives are addressed. First, the analyses produced by the Army Training and

Evaluation Program (ARTEP) are discussed, and suggestions are made for developing additional detail about objectives. Next, some considerations are presented about the definition of the training audience. Finally, the view is

presented that training objectives should be considered during scenario development, and that scenario specification should be facilitated by the simulation

system.

ARTEP: Tasks, Conditions, and Standards

The ARTEP manuals contain lists of training objectives which include tasks

to be performed, conditions under which the tasks are performed, and standards

which should be achieved. The lists have been developed for commanders and

staffs at various echelons and for various unit types.

The tasks specified by the command-staff ARTEP manuals focus on the products that should be produced by various staff elements (e.g., estimates, plans,

orders, reports) and to a lesser extent upon the procedures they should follow

in producing the products. Often, the ARTEP manuals list tasks that the command group must accomplish, without indicating which staff section or which

individual should carry out the tasks.

6%
The conditions listed for performance of ARTEP tasks focus almost exclusively on missions and tactical situations. Several aspects of command group

training which a simulation should be designed to accommodate are neglected. A

few examples include training to deal with incomplete, unreliable, contradictory, and incorrect information and intelligence; to operate under time pressure; to conduct extended operations requiring the use of separate shifts; to

continue operations with the loss of one or more key staff members; to continue

operations with the loss of a tactical data system; to recognize and deal with

deception; and to make decisions under conditions of fatigue, noise and discomfort. Some obvious implications of this partial listing of additional conditions are that the training system should be designed to operate for extended

periods of time, and that it should provide for filtering a.-,d Medification of

ground truth information from the simulation. From these examples, it should be

clear that an expanded view of training conditions can provide useful design

criteria. Conditions will be discussed further in sections 4 and 5.

The standards for performance comprise the third aspect of training objectives. ARTEP standards for command and staff functions tend to be vague and

somewhat general (e.g., accurate, timely, responsive, etc.). In order to provide for measurement of progress and assessment of training needs, these standards must be supplemented. This issue will be considered further in Section

3.

Despite their limitations, the command staff ARTEP manuals are the best

available source of information on what the training objectives of command

group training should be, although they include some objectives which are

better suited to training in field exercises than in command post exercises.

To be of use to simulation training system developers, the ARTEP task

lists must be screened for inappropriate objectives. They also must be expanded to include detailed descriptions of the procedures involved in task

accomplishment and an assignment of individual and group responsibility. It

would be useful to trace the interrelationships among tasks and subtasks, as

this information will facilitate the subsequent development of measurement

procedures. The actual conditions under which the task might have to be performed still need to be specified because the ARTEP analysis is limited to

general situations, i.e., missions, in which the tasks are performed. Finally,

the standards must be made more precise and measurable.

Training Audience

A typical error which occurs when determining who must be in the training

audience is to include too many people. Attempting to train the entire command

group tends to include personnel in the training audience simply to simulate

the performance of minor functions that are rarely required.

In research involving 12 battalion command staffs (Kaplan, 1984), ARTBASS

participants were asked to select training objectives from a list of 52 potential objectives and, after the exercise, to rate how much they had learned

about each of the objectives selected. ARTBASS proved to be an effective

trainer for the S3, commander, S2 and FSO, but trained only half of the objectives selected by the S4, while the SI reported that he learned little about

7.
any of his objectives. The relative inequality of training across staff positions is by no means limited to ARTBASS but is a general feature of battlefield

simulations. Partly the problem is the result of exercises which begin with a

scenario in which the forces are not yet engaged and the units are fully manned

and equipped. Those in combat and combat support functions can begin work immediately while those in combat service support (CSS) functions have little to

do during the initial period. Even beyond this source of inequality, however,

the simulation systems usually provide less challenging tasks, less varied activity, and fewer problems for the CSS sections.

Previous attempts to deal with this inequality of training opportunity

include the development of off-line manual simulations to be played in conjunction with the automated system, e.g., ADMINMOD and LOGMOD for CAMMS, the use of

probe messages to stimulate activity in administrative and logistics areas, and

the development of medical and maintenance simulations in BABAS. In the case

of the FSO, expansion of the training audience to include fire direction center

personnel dramatically improved the realism of his training.

When determining the training audience, the quality of the activity the

players perform must be considered. Simply because a player is kept active

during the exercise does not guarantee a training benefit. A division chaplain

may receive messages about soldiers who need counseling or a veterinarian about

meat which may be contaminated. This will give them something to report at the

end of the day but does not provide them training in how to counsel soldiers or

inspect meat. In contrast, the G3, during the exercise, is able to make plans

and practice his job. When deciding who will be included in the training

audience, particularly in high echelon simulation systems, it should be determined whether the person will perform his function in a simulated exercise or

will merely simulate performance of his function.

Scenario Development

The scenario used in command group training is the principal means of control over the exercise. The basic situation, the activity of the opposing

force, the intelligence available and the resources under the control of the

training audience must be carefully selected to insure that all training objectives selected for a particular exercise are addressed. Ideally, a battle

simulation system should be flexible enough to exercise any scenario consistent

with the training objectives. In practice, current automated systems are limited by the available graphics and terrain modeling data (either videodisk

images or Defense Mapping Agency data), and further limited in terms of the

types of opposing forces for which data exists. Although a small set of standardized scenarios is usually developed in conjunction with a battle simulation, primarily for operational testing purposes, existing simulation systems

provide very limited support for scenario development or modification. The

support that does exist consists primarily of databases describing typical U.S.

and opposing force (OPFOR) units, along with software for creating modified

unit descriptions and task organizations. Since scenario development is extremely labor intensive and requires a high level of expertise in large unit

operations, the lack of support in automated systems for this process has had

the unfortunate side effect of concentrating most command staff training on a

few well known scenarios.

8

Va.~~~~~~~ Fa *5* - -. ~ . . . .
Although development of training scenarios is currently something of an art

form, with little existing documentation on what should be included or on how

the situation can be modified to meet particular training objectives, enough is

known about the process to recommend the incorporation of tools to relieve the

scenario developer of some routine labor. In particular, terrain analysis can

be relieved of much tedium by incorporating automated line-of-sight fans and

three dimensional perspective views of the terrain. Similarly, sensor placement, weapons positioning, and indirect fire coverage for the OPFOR can be

assisted by the use of range fans. Doctrinal templates of OPFOR scheme of

maneuver and defensive tactics can be provided to assist the scenario developer

in choosing initial positions and avenues. Many similar tools have already

been developed for use with analytical wargames (McGrew, 1980). Scenario development would be greatly facilitated if the simulation is based upon a "systemic" wargame, i.e., one which is capable of operation for extended periods

without controller input. The systemic model could be run to test scenarios

to insure that appropriate force ratios are used. Further, the wargaming capability could be used to develop information on how well a unit might be expected to perform under the given conditions (SAIC, 1985). U.

9N

,I

.

-. -~ .AUM * -. 4 SU 4~ A4 .J 4.4.--. -.. - - .\('U~ -

-s.. *~4 ~ -4 '.'. .U U % 4

4~. % *. . -- UV.~ N %~U. 'U
SECTION 3. DEVELOP PERFORMANCE MEASUREMENT

Performance assessment has tended to receive insufficient attention when

battle simulation systems are designed and when they are employed. This is

especially unfortunate because the value of the training received is inextricably linked with the quality of feedback presented. Research by Thomas,

Barber, and Kaplan (1983) shows that explicit incorporation of measurement and

feedback in simulation-based command and control training exercises is

essential for effective training.

The requirements of performance measurement and feedback need to be considered early in the design process or even minimal system support may be precluded. The system must provide the hardware for collecting, storing, processing, and disseminating performance data. Also, the software should be designed

in such a way as to facilitate the collection of data on battle outcomes,

status, resource usage, and other information associated with training objectives. With the advent of operational tactical data systems, the opportunity

for on-line collection and analysis of staff performance data has increased

dramatically, provided that the linkage of the simulation system to the tactical data system is appropriately designed.

Performance measurement can be addressed at several levels of the command

staff environment. One classification scheme is to consider heirarchial levels

of: (a) command post or system, (b) major subsystem, (c) staff element, and

(d) individual. Different indicators or "observables" are characteristic of

measurement at each level. Statistics summarizing the outcomes of the simulated battles are useful at the system and major subsystem (e.g., maneuver,

fire support, air) level. Products and procedures are associated with evaluation at the staff element level. Knowledge and decisions require individual

level measurements.

This discussion will cover basic measurement considerations and review

successful measurement techniques for each type of measure. Automated measurement techniques and their associated data and resource requirements will be

emphasized.

Battle Outcome Statistics

This category includes most of the standard operations research techniques

for judging the outcomes of analytical wargames, such as loss-exchange ratios

(LERs), surviving maneuver force ratio differentials (SMFRDs), and combat power

ratios. It also includes indicators of the efficiency of an operation such as

consumption of supplies in various logistics categories. Research performed by

Thomas (1983) showed that, of the most widely used measures of this type, only

SMFRDs accounted for a significant proportion of the variance in expert judgment of mission accomplishment for a set of training exercises.

In subsequent research (Thomas & Cocklin, 1983), a panel of experts evaluated outcome measures from a large number of covering force operations and

assigned mission accomplishment scores. A weighted linear combination of

SMFRDs, measures of territory lost, time the enemy was delayed, and accuracy of

10I
intelligence estimates at the conclusion of the mission accounted for a much

higher proportion of the variance in mission accomplishment judgments than

SMFRD's alone. The regression model was applied to a new set of training

exercises, and was extremely accurate in predicting the mission accomplishment

judgments of the same panel of experts. Further development, including extension to other missions and validation against a larger sample of expert judgments, is required before the regression models could be used to replace human

evaluation of mission accomplishment.

Despite the experimental demonstratic of how battle outcome statistics can be used to provide reliable indicators of performance, they are not suffi- cient to evaluate staff performance. There are a number of reasons for this.

First, there exists considerable doubt as to the accuracy of the underlying

models of combat which provide the attrition results. This is particularly

true of models which use firepower score methods to estimate the relative ability of a unit to inflict casualties on an opponent, because firepower scores

ignore the differential effectiveness of various weapons systems against targets of varying "hardness". To some extent, there is also distrust of more

sophisticated attrition methodologies if they are being used in conjunction

with unclassified weapons effects data.

Second, the raw numbers provided by battle outcome statistics are meaningless in themselves. They must be compared to standards of performance which

describe how well a unit would be expected to perform a similar mission with

similar assets against a comparable opponent. These data do not currently

exist; they cannot be reliably obtained from training environments due to the

variability of such exercises in unit composition, training goals and methods,

and scenarios played; and they cannot be generated using current analytical

wargames due to the inherent speed limitations and lack of replicability of

man-in-the-loop simulations.

Third, such data are not a pure reflection of the performance of a commander or staff. Outcomes are also influenced by the behavior of friendly

controllers, data entry personnel, and the opposing force controllers. Current

training simulations do not automate the decision making of subordinate, support or opposing elements, but rely upon very detailed model inputs provided by

training support personnel who are often too busy to provide appropriate inputs (Thomas & Solick, 1982).

Finally, measurement based on battle outcome statistics is not well-suited

to the primary purpose of command-staff training, which is to diagnose and

correct performance deficiencies. This purpose is better served by measurement

approaches which directly examine the tasks a staff performs, the way in which

they do it, and the products resulting from their effort.

As a design issue, the use of battle outcome statistics as performance

measures increases requirements for simulation fidelity, storage and analysis

of detailed records, and development of supporting techniques to acquire

normative data against which training results can be compared.

11

*] ,-- 
Products and Procedures

The staff element is the intermediate level in the process heirarchy, and

indices of staff performance primarily involve either products or procedures.

Products include items such as estimates, plans, orders and reports. Measures

of the products can be on dimensions of timeliness, accuracy, understandability, relevancy and completeness. Procedures involve whether actions and products were developed in accordance with prescribed doctrine or Field SOP.

Observation has been the only technique commonly employed for evaluation

of staff products and procedures during simulated battle. It is usually done

informally, relying upon the participants and training support personnel to

take note of key events, apparent misunderstandings, and failures to coordinate

plans. These impressions are commonly reviewed in a debriefing session at the

end of the zxercise (ENDEX), known as an after action review (AAR). The AAR is

usually supplemented by a brief summary of the status of forces at ENDEX. In

simulation systems which provide replay capabilities, locations, status, and

activities at key points in the battle are reviewed as the participating commander and chief controller remember them. Occasionally, these informal techniques are augmented by the presence of one or more special observers who may

record their observations and evaluations using an ARTEP task list or other

mnemonic such as a list of key information to be presented in the operations

order (Olmstead, Baranick & Elder, 1978b). Some simulation centers have also

instrumented their simulated Tactical Operations Center (TOC) facilities with

video monitors and centralized communication systems to improve the effectiveness of observers.

There are at least two opportunities for designing battle simulations to

improve this rather haphazard approach to performance assessment. First, the

computer system can be given a list of key events to look for which indicates

common failures in planning and coordination. This list can be used to focus

an automatic replay on instructive situations or be provided as input to a more

traditional AAR. Examples of such events include routing supplies through your

own minefield (indicating failure to coordina e resupply and barrier plans),

sealing off one of your own units with FASCAM (indicating failure to coordinate maneuver and fire support), and cancellation of air strikes due to smoke

or dust over the target area (failure to coordinate air support and indirect

fires). These and other more subtle failures in planning or coordination can

be noted by simulation software without relying upon an observer being in the

right place or recalling events that happened several hours earlier.

The second area of opportunity exists when the staff being trained in a

battle simulation environment is equipped with a tactical data system (TDS) or

with equipment which simulates the functionality of such a system. The TDS can

be equipped with a "watchdog" program to note and record usage of the system

for later comparison with normative or doctrinal data stored on the training

device. The types of events that might be of interest include the number and

time of submission of reports required by SOP, the categories of information

2

Family of scatterable mines. FASCAM may be implaced by artillery, by air, or

by surface vehicles.

12

ON ~ !. "o. %.~ 
T

 - -- ---- - WN - - - - - - - - -

accessed by each staff section during preparation of plans and annexes to the

operations order, the frequency with which status reports are requested, and

the delay in entering information received via voice communications.

There has been some experimental use of testing methods with staff products

and procedures. The testing involved variations of probe methodology, where a

message is inserted into the information flow of the exercise specifically to

generate some required staff action so that the type, quality, and timeliness

of the staff response can be noted (Carter & Patton, 1985). This differs from

purely observational methods in that the situation is deliberately altered,

sometimes in rather artificial ways, to generate behavior that would not ordinarily occur in a "free play" exercise. Examples of such probes include jamming of communication, report of a chemical weapons attack, looting reports

etc. A number of such probes were developed for use in manual simulations.

Battle simulations may support the probe technique by storing a list of

probe messages and prompting controllers to insert them into the message flow

to the TOC at specific times or in conjunction with the occurrence of specific

events in the simulation. If the probes are designed to generate some required

staff activity on a tactical data system (TDS), the technique can be tied in

with the previously suggested watchdog program to provide automated assistance

in observation of responses and response times.

Staff products and procedures are the major emphasis of command post training exercises. As such, they should also be the primary focus for automated

measurement procedures.

Knowledge and Decisions

The individual performance level is divided into knowledge and decision

categories. Since knowledge is not directly observable and only the results of

decision making can be directly observed, measurement of individual performance

must rely upon: (a) test techniques to determine whether the individuals have t

the knowledge they should, and (b) judgments of the quality of decisions. The

assessment of decisions requires speculation about how alternatives would have

worked out; the effectiveness of a set of decisions is seldom determined.

Typically observers will only note when a decision varies from accepted practice or from what they would have done in the same circumstances.

The most successful application of testing to determine knowledge is based

on information flow methodology (Kaplan, 1980, 1981). This is a technique for

tracing the spread of specific items of information throughout a staff group

during planning, preparation and delivery of an operations order. It requires

a prior analysis of the types of information that each staff section or subordinate element should receive on the plans, resources, and activities of other

staff elements. When the mission is briefed to the staff group undergoing

3

Probes used with manual simulations were intended only to exercise staff areas,

e.g., administrative and logistics functions, that were not well supported by

the simulation techniques employed. When combined with an observation plan,

they are also useful for measurement.

13

% N Ar
%

training, these test elements of information are presented individually to

different staff sections. After the staff plans and presents its operations

order, they and subordinates are given a written test on the items they received initially, the items they should have received through coordination or

in the operations order, and on other items that were inserted but not directed

to them, i.e., items that they were not required to know to perform their individual functions.

Attempts have been made in experimental situations to extend the information flow method to the mission execution phase of exercises (Thomas, Barber, &

Kaplan, 1984; and Thomas, Kaplan, & Barber, 1984). Knowledge was tested in

each staff section by requests from appropriate controllers. Specific queries

were directed to key staff members in the experiments by Thomas, et al, rather

than introducing significant events and tracing the spread of information

throughout the staff as proposed by Carter and Patton (1985). The limitation

on this technique is on the number of seemingly random queries that can be

instituted without interfering with the attempt to maintain a realistic battlefield situation. In the cited studies, 5-6 queries per staff element were

inserted. This number is too low for any statistical conclusions to be drawn

about individual performance, but the numbers aggregated across staff elements

could give fairly reliable indications of a staff group with problems in inter- I

nal coordination.

Another approach to determining the accuracy and timeliness of distribution

of information was explored in several exercises. It was assumed that the information posted on the various situation maps maintained by different staff

sections was an indicator of the knowledge held by the staff. By sampling that

information periodically and comparing it to the "ground truth" information

taken directly from the battle simulation, one could obtain data on the timeliness and accuracy of information dissemination. In practice, the technique

did not work. The basic practical difficulty was in obtaining information W

posted on the maps and analyzing it in time to be useful for feedback to the

command group. Another difficulty was in interpreting the results obtained.

There is currently no doctrine for how much timeliness or how much accuracy is

enough. In results from a division level exercise, it was clear that the working answer varied widely across staff sections. It would be costly to assemble

enough normative data using manual techniques to permit interpretation of these

numbers.

Though the testing methods used to evaluate knowledge were successful in

experimental tests, they are impractical to implement due to the amount of

manpower required. This suggests another design criterion for battle simulations: they should provide automated assistance in initiating performance

tests, in collecting performance data, and in analytically developing standards

for performance, particularly in the area of evaluating decisions. Two techniques are suggested here that require attention in the design of new battle

simulations: a comparison of the state of the model with the state of the

staff's knowledge, as represented in the data base of a TDS, and the development of wargaming methods for assessing alternative decisions.

14
The first technique proposed is essentially an automation of the comparison

of the "ground truth" in the model with information "posted" by the staff in

their TDS. If implemented appropriately, however, it can go well beyond a

simple comparison of location and status information. For example, if the

staff is required to formally post estimates, these can be analyzed computationally and compared to the "real world" in the model.

The second technique, analytical wargaming, is an extension of the previous

proposal for generation of normative data on outcome measures. If wargames are

to be used for comparison of alternative decisions, they must provide an explicit representation of decision making. To minimize the influence of controller activities on the training exercise results, the model used to drive

the training system should also operate in a systemic fashion. Thus we recommend that the same basic model should be used for both purposes to simplify the

comparison of alternative decisions.

Very fast systemic models of the type needed are currently under development by Training and Doctrine Command Analysis Center (TRAC) (viz. the VIC --

Vector In Commander -- model) and under the auspices of Defense Advanced Research Projects Agency (DARPA), where Corps Battle Analyzer (CORBAN) is being

implemented on an experimental parallel processor. Current results indicate

that VIC will run ten times faster than real time, in contrast to slower than

real time operation for man-in-the-loop versions of analytical models. The

first parallel version of CORBAN is sixty times faster than its implementation

on a standard serial architecture. These results indicate that this technology

may soon become a practical design alternative for training simulations.

N

15.

;-e

U,
I..

SECTION 4. DEFINE SIMULATION REQUIREMENTS

There are two fundamental questions to be answered in defining requirements .4

for a training simulation: (a) what to simulate, and (b) how to simulate it.

Ideally, the answer should flow directly from an analysis of training objectives; specifically, who is to be trained, on what tasks, under what conditions

and to what standards. -i practice, neither the state-of-the-art in modeling,

nor in task analysis have permitted such a straightforward translation of requirements. Previous simulations for training have been designed upon the

basis of what modeling techniques and computer resources were available, with

an implicit assumption that controllers could use manual simulation or roleplaying techniques to fill in the gaps. This evolutionary approach to system

design has had some unfortunate side effects. These will be discussed under

four categories: model fidelity, software maintenance, exercise preparation,

and control of training conditions. Suggestions will be presented for more

closely approximating the desired relationship between training objectives and

system design.

Model Fidelity

General requirements for fidelity in a training simulation include the

ability to portray those missions that might be realistically assigned to the

unit being trained; the availability of all major resources that might be

under the control of the unit; the ability to portray realistic restrictions on

resources and activities imposed by terrain, weather, competition for scarce

assets, equipment and personnel limitations, and the ability to portray realistic timing of activities, for example, movement, coordination, set-up and

breakdown times, planning and implementation time required by subordinate and

support elements. Comprehensive listings of the resources and activities that

should be portrayed in staff training are available in Miller and Bonder (1982) -.

and Michel and Solick (1983).

Discussion of some common omissions in developing systems is provided in

Thomas and Solick (1982). For battalion and brigade level training systems,

nuclear and chemical contamination, sensor systems, movement of artillery and

air defense assets, counter battery and counter mortar fire, electronic warfare, construction time for obstacles, maintenance of equipment, and medical

problems were usually not included in models. Since then various features from

this list have been added to developing simulations, but none of the current

training systems have incorporated all of the previous omissions.

Two additional fidelity issues impinge directly upon the scope of the model

and the size and type of equipment needed to accommodate it. First is the

level at which to simulate subordinate units. The standard answer is two echelons below the level of the training audience. While this answer is adequate

for maneuver units, it can lead to an underestimate of the number of entities

to be modeled, and thus potentially to a selection of inadequate equipment and

to a bottleneck in data entry. In the development of specifications for the

Corps Battlefield Simulation (SAIC, 1985) it was determined that certain highly

specialized support elements required modeling at the company or platoon level.

This roughly doubled the number of entities to be modeled.

16
A second basic fidelity issue is whether the training system is to operate

on classified or unclassified data. Cost to incorporate electronic shielding

and to provide secure communications is usually contrasted with the perceived

benefit of obtaining realistic combat results, particularly if the results are

to be used for performance evaluation. Briefly, there is no evidence that this

factor makes a difference in training. Other considerations, such as whether

the system is intended to exercise classified scenarios, will override training

benefit issues.

The basic lesson learned in the development of models to support training is

that maneuver and direct fire conflicts do not constitute a combined arms

training system. The modeling effort and system resources required for indirect fires, air and air defense, intelligence and logistics, is at least as

great as that required for maneuver. If medical or maintenance functions are

to be included wi:h sufficient fidelity to provide training in the performance

of those functions, each of those areas will require similar resources to the

foregoing. Failure to take these factors into account has led repeatedly to a

system design that required extensive modification (e.g., MACE/BABAS) or to an

unworkable concept (e.g., the distributed CATTS system and CAMMS II were both

incapable of accommodating the required volume of messages over phone lines to

a central processor). For the systems that have been fielded despite a neglect

of combat support and combat service support functions, there have been consistent complaints from the field that they require an unrealistically large number of controllers to manually portray the neglected functions (Thomas &

Solick, 1982; Barber & Solick, 1980; Kaplan & Barber, 1979; and Barber, McGrew,

Stewart, & Andrews, 1979) and that they fail to address many of the training

objectives of the principal staff members (Kaplan, 1984; Kaplan, 1985; and

Kaplan & Fallesen, 1986)

Software Maintenance

The evolutionary approach to the design of battlefield simulations has

tended to neglect many issues that affect the system's success as a home station training device.

Continual changes in equipment, system capabilities, and doctrine for force

design require that a training system be designed for continual revision. It

is fairly straightforward to include in the initial design of the training

system provisions for programmatic changes that will occur during its design

life, such as new weapon systems. However, incremental improvements in speed,

range, accuracy or resource consumption of systems frequently require data

base changes after a training system is developed and fielded. The need for

change also arises as new data become available on lethality or vulnerability

and as new means are developed for employing existing equipment. Forecasting

such changes based upon current experience with design modifications can minimize the disruption in fielding training devices by providing excess computational and storage capacity in the original design. It is essential to have

comprehensive documentation of the system for such cases, including documentation of the relationship between software models of processes and associated

control staff activities, such as data entry, model output interpretation and

role playing tasks. Model maintenance is considerably easier if a structured

17
IF.

design technique, e.g., JSD/JSP (Jackson, 1983), is used to prepare a simulation for implementation and if modular programming (e.g., ADA, Modula-2) or

object-oriented programming (e.g.,ROSS, Smalltalk, LOOPS, Flavors) is used for

the implementation (Pressman, 1982).

Exercise Preparation

Preparation for a training exercise includes development or selection and

modification of a scenario tailored to the unit's training objectives, adaptation of a generic data base to the specific forces included in the scenario,

and preparation to carry out the plan developed by the training audience.

Preparations also include training of support staff to operate the equipment,

to plan the exercise in accordance with training objectives, and to play the

roles of higher, subordinate, and adjacent headquarters. Current simulations

usually provide one or more standard scenarios, a suggested program of instruction for controllers, and user manuals which list control functions. The most

recently fielded training system, ARTBASS, also includes computer-assisted

instruction on operation of the computer system. In general, however, little

is provided in the nature of tools for scenario development for data base modification in accordance with doctrinal templates, for planning an exercise to

meet training objectives, nor for instruction on playing the role of other

headquarters elements (Kaplan, 1985).

A complete system design should include these considerations, since the

cost of such preparations approach the cost of conducting the actual training r

exercise (Kaplan & Barber, 1979). Furthermore, it may not be most effective to

consider them as low-cost add-ons to be designed later, as the equipment selected for the exercise may not provide the most appropriate input and output

devices for these other system functions. For example, mouse input devices are

useful for high-speed menu selection, but not as good as graphics tablets for

entry of detailed graphics such as avenues of advance, phase lines, or control

measures. Similarly, optical scanning or bar-code readers might be best for

setting up a unit data base, but not for initiating resupply actions (Monk,

1984).

Control of Training Conditions

Design of simulations to accommodate varying training conditions is usually

interpreted strictly in tactical terms, e.g., varying geography/terrain, varying weather conditions, accommodating different unit types, or varying missions

and combat ratios. While these factors have been shown to strongly influence

simulated combat results (Thomas & Cocklin, 1983) they are not closely related

to measures of the products (plans, estimates, reports) that staff members

produce nor to measures of the extent to which they follow accepted procedures

(Thomas, Barber & Kaplan, 1983). In order to directly manipulate the difficulty of staff tasks, a training simulation must be able to provide control of

message loads, amount and accuracy of information and intelligence, integrity

of communications links, and other factors directly influencing information

processing and decision making. Control of such factors is important if training is to be tailored to the level of skill of the staff since a major goal of

the training program is to create command groups capable of functioning smoothly under adverse conditions. A "How to Train" manual should be produced which

documents the objectives that the system can support, the resources needed to

18
conduct training exercises, the ways in which exercises can be tailored to

specific deficiencies, and the ways in which the system supports performance

measurement and feedback. This manual should address the composition of the

intended training audience and the ways in which supplementary training can be

provided to those members whose function is only marginally supported by the

simulation. The "How to Train" manual should be included as on-line documentation for the system to synchronize the maintenance of software and documentation, to permit more rapid dissemination of changes, and to provide on-line

assistance in resource calculations and scheduling of training events.

11

5,%

19
SECTION 5. DEFINE CONTROLLER TASKS

Defining the tasks performed by the training support personnel lies at the

heart of the design process. Controller tasks must be explicitly defined in

order to: (a) delimit the performance requirements of the simulation, (b)

design the interface between the controller and the simulation and the interface between the controller and the players, (c) estimate the system manning

requirements, and (d) develop the supporting documentation and training manuals.

In current simulations there are three fairly distinct classes of training

support personnel: the controllers, the interactors, and the role players.

Controllers include dedicated simulation center personnel to handle functions

that require detailed knowledge of the training system and, in some cases,

members from the training audience's parent headquarters to oversee the unit

training. Interactors perform primarily clerical tasks, entering data at the

direction of controllers and role players and calling up displays and status

reports. They provide the training to role players on the system capabilities

and limitations. Role players portray elements with whom the commanders and

staff must interact. They often serve in their actual duty position; translating orders into activities that can be entered into the computer; directing

their own (simulated) subordinates; and providing orders, information and intelligence to the training audience.

The discussion here will focus on the control functions that must be performed and the system requirements for interfacing with these functions. The

emphasis will be upon identifying and making design provisions for control

tasks that minimize the required number and qualifications of support personnel. Three general areas of controller functions will be discussed in this

section: controlling the exercise, controlling the model, and role playing.

Controlling The Exercise

The exercise control function encompasses the planning, preparation, monitoring, and evaluation of the exercise. Control personnel function as trainers

insofar as they control the conduct of the exercise, the conditions to which

the training audience is subjected, and the planning of performance evaluation

and feedback. To respond to observed strengths or deficiencies of the trainees,

controllers need to adjust the conditions and course of the battle to present

appropriate training opportunities. Also, controllers may be responsible for

performance measurement, using probes, measures, observing staff performance,

marking key events for emphasis during after action reviews and coordinating

with other controllers.

Setting Training Conditions. It has been observed that insufficient provisions are made for adjusting the conditions under which the command staffs se

train (Fallesen, Lussier, & Garlinger, 1986). In a given exercise, controllers

may wish to change the difficulty of the exercise by increasing the tempo of

battle, removing key personnel suddenly, or jamming communication equipment.

Command groups need both to train on conducting a battle which follows prepared

plans and to react to sudden events just as in actual battle.

20
Sometimes the command group should receive clear and accurate intelligence

from above and below, and at other times their information should be vague and

distorted. In short, simulation systems need to be flexible enough to allow

play under a wide variety of conditions (Olmstead, Baranick, & Elder, 1978a).

The choice of conditions should depend on the entrance level of training of the

unit, the specific training objectives of the exercise, and on the course of

play during the exercise. System developers need to be aware of this controller function and provide methods to put the control of training conditions in

the hands of the controllers. Systems should be adaptive to the performance of

the command group, recognize areas in which more training is necessary and

guide the play of the exercise in a manner which presents the appropriate

training opportunities (Shaket, Ben-Basset, Madni, & Leal, 1979). Training

systems must be designed with sufficient flexibility to allow adaptive control

of conditions.

Evaluating Performance. Another overall responsibility of controlling the

exercise is evaluation. Evaluation tasks have been performed by interactors as

well as controllers and infrequently by independent observers without other

control tasks. The measurement of the performance of the command staff was

covered in section 3 of this report, but it cannot be overemphasized; the presentation of useful feedback is critical to the success of the exercise. Not

only must the feedback be accurate, it must be accepted by the training audience. This means that controllers participating in evaluations must be able to

support their conclusions, ideally through an objective system of performance

measurement.

An after action replay function and combat outcome summaries are both techniques designed to assist in providing feedback. Neither, however, explicitly

identifies the specific strengths or deficiencies of the training unit. Without further development of a performance measurement technology, these methods

do not allow an effective assessment of the command group. However, the incorporation of performance measurement techniques will also impose additional

tasks upon the training support staff, additional requirements for training

controllers to perform these tasks, and additional control staff coordination

requirements. These considerations must be included in the workload and system manning analysis.

ARI is conducting research in the area of command staff performance measurement and will be able to provide more specific assistance in the future. At

present a few basic suggestions can be made. First, the controller workload

should not be so heavy as to prohibit the possibility of measurement and evaluation functions. Second, there should be some provision made for the presence

of evaluators, to include the ability to view events occurring in the model,

traffic on the communication nets, and activity of the training audience.

Third, no matter how rudimentary, a start must be made on providing automated

assistance to the measurement problem. The system could be designed to identify and flag occurrences from a previously identified set of typical errors at

a given echelon, to request staff products, to prompt controllers to insert

specific behavioral probes, and to request that the controllers input evaluative information at critical points during the exercise.

21

. ! 
Controlling The Model

Model control involves the translation of the directives of the training

audience into commands for entry to the computer and control of the operation

of the battle models. The tasks can be as tedious as arranging for the movement of a battalion, platoon by platoon, and as creative as simulating the

effects of resources which are not modeled, e.g., achieving the same effects as

a demolition team through "phantom" airstrikes.

The controllers must be able to implement the orders of the commander and

staff without major discrepancies or omissions and must do it without lengthy

delays. Imagine, for example, that troop movements were achieved in an accurate and timely manner, but either engineer operations or fire missions were

greatly delayed or altered. Such a situation frustrates the training audience

as it destroys the synchronization of their forces and the effectiveness of

their plans.

A primary purpose of any training exercise is to identify deficiencies in

the behavior of the training audience. The commander and staff, however, typically find it very easy to see the deficiencies of the system and very difficult to see their own faults. Some tolerance for interactor error exists and

can be considered as "fog of battle." It would be better, however, if this

"fog of battle" effect was intentional and was guided by the training objectives. As indicated above it is sometimes desirable to conduct training under

the conditions of a perfectly efficient set of simulated subordinate units

while, in other cases, a less efficient group of "subordinates" would be better

to train the monitoring and directing aspects of command and control. In any

event, when the controllers cannot effectively control the model, it is almost

certain that the training audience will see the model/interactor difficulties

as an excuse for all the problems of the exercise.

In part, the solution involves greater attention to the design of the

interactor-computer interface, and a strong commitment to human factors engineering (Monk, 1984). Prior to design of the user interface, system developers

should develop a list of criteria and consult with human factors experts on how

the desired performance may be attained. Basic issues to consider include

sharing of displays, color versus monochrome display requirements, multiple

displays versus multiple windows, and use of iconic and nomographic techniques.

Attention should be given to:

* Providing a clear, easy-to-learn (self teaching) network of commands and

menus without employing abstract coding.

" Using formats and terminology which are natural to the military user.

" "Bombproofing" data entry techniques so that the user is not allowed to

make irreversible errors.

" Allowing recovery from error which entails a minimum of reentry.

" Extensive employment of feedback, acknowledgements of instructions,

prompts, and warnings.

22
" Use of an appropriate set of input devices and creatively designed software to minimize workload.

" Highlighting of critical or urgent messages and actions.

A second avenue of approach to the goal of reducing the model control tasks

is through more extensive automation. Using operations research or artificial

intelligence (AI) techniques, the computer can assume increasing amounts of the

control workload (Shannon, 1975, Barr & Feigenbaum, 1982). An example is the

current ability of some systems to employ route optimization routines. Control

of the opposing force is particularly open to AI application because there is

no requirement for communication or coordination with members of the training

audience.

The design goal must be to reduce the model control tasks to an absolute

minimum, in terms of both physical and mental workload. Requirements for computer expertise must be reduced to the point that controllers drawn from the

unit being trained can operate the equipment and can translate computer output

into tactical messages after a brief training session. Workload reduction will

reduce the number of controllers required by eliminating those who perform only

data entry functions, will reduce controller training time, and will free controllers to perform other valuable tasks. Realistic goals must be set for the

size of the control staff and for the amount of exercise preparation required.

The additional analysis and software development costs this approach entails

will be recovered through increased efficiency in conducting training.

Because units must train as they would fight, future battle simulation

exercises will need to incorporate tactical data systems such as the Maneuver

Control System, which is being fielded at echelons from battalion through

corps. The introduction of tactical data systems to command and staff exercises has the potential to significantly reduce the controller's model control

workload. Information will need to flow between the players and the computer

in both directions, with the results of the computer modeling process automatically updating the tactical data system's database, and command group directives helping to control the operation of the model. The role playing workload

may also be reduced by automatic generation and digital transmission of periodic reports required by field SOP (SAIC, 1985).

In summary, the tasks for controlling the model need to be reduced to a

minimum. If the tasks are too numerous, time-consuming or complex, the training effectiveness of the system will be greatly impaired. Representation of

battle events should not be at a level of detail below that necessary to pres- ent a realistic picture to the training audience. If the representation is

much beyond the minimum necessary, controller workload may suffer. Finally,

simulations need to be designed so that the link with tactical data systems at

least eases the controller workload rather than intensifies it. Efforts to enhance the human factors design or to automate the controller functions are not

frills but are vital given the conditions under which the systems are used in

the field.

23
p

Role Playing

Controllers must play the roles of key personnel in units which are higher,

subordinate, and adjacent to the training unit. The role playing function P

serves to insulate the training audience from the computer and to allow training in required communications. Typically, subordinate unit commanders are

played by people brought to the exercise by the training unit and are often the

actual subordinate unit commanders. Occasionally, the higher headquarters

assign personnel to the exercise to act in various positions from that headquarters. More often, however, controllers from the simulation center portray

the roles of higher headquarters. While adjacent units need to be included in

the scenario, it is rare to have role players in these positions; information

normally obtained from adjacent unit commanders can be obtained from the higher

headquarters role players without a significant loss of realism.

Subordinate Role Playing. Subordinate unit role players must keep themselves apprised of the tactical situation, manage simulated subordinate and

support units in accordance with the plans and orders of the command group, and

generate an information stream of realistic tactical messages based on the

output of the simulation. The latter function is not well supported by the

current generation of battle simulations. In particular, the output of automated simulations appears to compromise realism of the message traffic by providing too much military intelligence and not enough detail. For example, a

sensor report might include a complete opposing force unit designation and an

exact center-of-mass location, whereas a realistic report might provide an

estimate of a number of vehicles sighted, type of vehicles, approximate location and direction. Role players are required to perform this translation.

Preliminary results from work in progress at Fort Leavenworth ARI Field Unit

indicates that this translation task is difficult to perform. The difficulty

is compounded in the context of an exercise with the demands of the role

player's other duties and the obligation to report to one's actual superiors.

One possible solution would be to automate the production of message traffic. This would insure that input to the training audience would accurately

portray events in the battle, would not reveal unrealistic amounts of intelligence, and would conforia to standard military format. The controller workload

would also be reduced. It has been argued that the role players are receiving

training benefit by composing the messages themselves, however, in the current

circumstances this argument seems to have dubious merit. Taking ground-truth

information and dirtying it up for transmission to the next higher echelon is

not a task in which Army officers need training.

An alternative solution to the translation problem is to redesign the interface between the subordinate unit commander and the computer. Displays of

ground-truth information could be avoided. Instead, at the subordinate unit

c-mmander stations (or other functional area stations such as log/admin, air,

oi fire support), the role player would have available only that information

which he could normally be expected to have, either through the activity of his

staff and subordinates or through first-hand observation. Current simulations

have terrain appreciation capability that portray a 3-dimensional view from any

location (McGrew, 1980), given that digitized terrain data is available. The

24
computer would need to supply a further description of what battle events the

role player should see from the ground, and to provide displays of the information which should be available to the role player.

Although replacing ground-truth computer representations of the battle with

a realistic interface between the role player and the computer would require a

major software development effort, there would be many significant advantages:

* The training audience could be supplied with a realistic message stream

which is generated in a natural manner.

* Errors by role players in portraying the situation would be reduced.

* Role players would receive practice in skills which more closely match

their usual military assignments.

* The interface could be used to establish an effective simulation of the

unit commander "going forward" to see the battle and personally direct subordinates.

Many of the controllers are in a position to receive considerable training

benefit from the exercise. This potential can be realized by purposely designing the functions performed by subordinate elements of the command group to

resemble, as much as possible, the tasks on which they are to be trained.

Higher Echelon Role Players. The role players portraying higher echelon V.

personnel must work on the scenario development, present the initial briefings

and operations order, coordinate with the various functional areas, and make

decisions regarding the distribution of higher echelon assets and the dissemination of intelligence. Higher headquarters role players are usually one or

more staff officers from the training unit's actual higher headquarters or are

controllers who work at the simulation center. In the latter case, they may be "

called upon to portray positions with which they have little experience.

There are several areas in which the system might support the higher role

players, for example, by aiding in scenario development, preparing specialized

functional information packages, and by generating messages to be inserted

during play. One observation concerning higher echelon play is that the role

players provide much of the support of the higher echelon, i.e., providing

intelligence and combat assets, but rarely make the demands of the higher echelon, i.e., additional taskings such as primary information requirements (PIR),

other information requirements (OIR), or required standard reports (Thomas &

Solick, 1982). The load on the training audience would be more realistic if

additional requirements were placed by the higher echelon. The simulation

could incorporate such taskings or prompt the role players to demand scheduled

reports, increasing the requirements on the command group.

In summary, role play is an important aspect of the exercise and can mean

the difference between a command group training in a realistic situation under

realistic constraints and a command group which is merely playing a battle game

against a computer. Neither the design of current sir.alations nor the training

currently given to the controllers provides sufficient support for this important function.

25

S-. ., .b ~ ~ ~ ' - W- ~. . -
Role playing must be more realistic than current practice. The messageI

traffic to the training audience can be enhanced by automated production. An

improved computer interface needs to present more realistic information to the

role players in a more usable form. The higher echelon needs to be enhanced so

all required reports and additional taskings are included.

Summary

In the past the model control task has received the major portion of attention in the design process. Not only is it necessary for the operation of the

system but it is the function which designers naturally associate with the

controllers. Role playing is recognized during the design phase, but it is

generally given little or no automated support. Other controller functions

such as controlling the exercise and performance evaluation typically are ignored completely until operational testing of the system. For battlefield

simulation systems to be truly effective trainers, there must be a shift to

increased emphasis on interacting with the training audience rather than constrained attention to the model.

264

4" or
op

SECTION 6. ESTIMATE CONTROLLER WORKLOAD

Simulation systems are developed in accordance with performance goals,

which specify the required operating capabilities of the system, and staffing

goals, which indicate the desired limit of operator and controller personnel. Working within the performance and staffing constraints, the system designer should compile a complete description of all the tasks performed by

controllers, should apportion these tasks between individual controller positions, and specify the skill requirements for each position. The establishment

of manning requirements is normally an iterative process, involving redesign

until both the skill requirements and workload levels for each proposed controller position are reasonable.

The resulting estimates of the number of controllers needed and the description of tasks they must perform drive the hardware design requirements for

controller workstations. In existing simulations, several controllers are

assigned to each workstation with data entry devices and situation map displays

usually shared by two or three controllers. The tradeoff between cost of

equipment and efficiency of control must be considered. Cost has been the determining factor for existing simulations, but equipment costs have been reduced to the point that efficiency may now predominate. If model control tasks

are distributed to the role players, it is preferable to equip each individual

with his own input and display devices. It may be necessary to develop pre- r

liminary cost and staffing estimates for both approaches. -

Complete, well-documented analysis of control personnel requirements is the

exception rather than the rule in battle simulation development, primarily

because of the evolutionary approach of incremental refinement to existing

simulations. A notable exception is the requirements analysis performed ror 1

the Corps Battlefield Simulation (SAIC, 1985). The recommendations made in a

this report are primarily based upon that effort and upon lessons learned in A

the MACE Concept Evaluation Program (Thomas & Solick, 1982). 1

The process of determining manning requirements is presented in Figure 3.

The initial step involves an analysis of the component processes of the model,

in which a large and detailed list is compiled of all the functions, tasks and

subtasks which must be accomplished in the training systems. Examples of such

an analysis may be found in Miller & Bonder (1982). The task list should include all functions of the simulation, including the model control, role play,

and evaluation functions.

Each of the subtasks must be allocated to either the controllers or the

computer. In general, this allocation should be guided by a comparison of the

human and computer strengths and weaknesses. The ability to make complex inferences, to find similarities, and to hear, speak, and understand are human

strengths. The ability to store and retrieve large amounts of information and

make fast and accurate calculations are computer strengths (Woodson, 1981).

Further analysis of the tasks may be helpful in making the allocation decision

(TRADOC PAM 351-4(T)). Some aspects of the tasks which may be considered are:

27 '1

-U -
COMPILE

TASK LIST

ANALYZE TASKS/

COLLECT DATA

ALLOCATE TASKS COMPARE MANNING SCHEME

TO MAN OR MACHINE TO STAFFING GOAL

IDENTIFY

CONTROL POSITIONS

DISTRIBUTE TASKS

TO CONTROL STAFF

4 1 4

COLLECT ESTIMATE WORKLOAD LEVELS AND

WORKLOAD DATA SKILL REQ'MENTS FOR EACH POSITION

Figure 3. Steps in determining staffing requirements.

28

I"

-p]' .~ ,-~' . 4*~ *,4~. p~~.
* The action performed - abstract the essential actions required.

* Skills required - differentiate between clerical activities and those '

which require judgment or perspective.

9 Knowledge required - indicate the military knowledge required and what

functional specialties might be necessary.

Input required - identify the characteristics of the input requirements

and source rather than the specifics of current practice. Indicate whether the

information is gained "shortly" before or "long before" the task period begins. .°

o Output required - as with input, aim for the characteristics of requirements and destinations.

Preceding activities - distinguish between the required events which

precede task performance and those which make logical sense or are preferred

but are not strictly necessary.

9 Timing - tell when the task should occur, what kind of delays may occur,

what time constraints exist, what are realistic delays. How frequently do

tasks occur? Note tasks which must be performed concurrently.

0 Potential difficulties - machine breakdown or backlog, controller overloading, etc.

e Potential errors - identify potential sources and the impact of errors.

Additionally, the functions concerned with evaluation of the training

unit's performance need special consideration. The evaluation tasks are difficult and seem to require much human assistance. They also are tasks which can

be ignored without "crashing" the exercise but are vital to attainment of the

goals of the exercise. Evaluation tasks need to be given sufficient machine

support to insure that they receive the controllers' attention.

Once the tasks to be accomplished by controllers have been analyzed, a

preliminary list of controller positions can be established based on the staffing goal, on the functional specialities for role playing, and on the essential

model control functions. The preliminary manning scheme is completed by taking

each of the tasks r-quired and assigning it to one of the proposed controller

positions.

The manning scheme must be analyzed to discover weak points, where either

the skill or workload requirements are too high. Two general principles should

guide the analysis. First it is better to conduct experiments, observe current

exercises, or use survey or historical data than it is to simply guess. Second, the estimates should be conservative, leaving sufficient margin for error.

There will be tasks that have been overlooked, activity at peak levels which

was not anticipated, and personnel who work with less than 100 percent efficiency or who lack skills.

29

I
One aspect of workload estimation which needs to be addressed is the frequency of occurrence of the tasks. This includes both the number of times the

task is performed over the exercise (or per day) and the peak frequency, the

highest rate at which the task must be performed. Estimation can be assisted

by analyzing doctrinal material or unit SOPs or by observing current simulations. For example, message traffic for a controller position can be estimated

by determining the number of SOP reports per transmission mode (voice, digital,

or written) for that position and inflating the totals to account for non-SOP

queries and replies. Observing actual exercises at that echelon may help to

determine the percentage of SOP to non-SOP reports, the average duration of

each transmission per mode, and the peak message rate. A similar analysis can

be done for other events such as movement orders or fire missions. When using

data from current simulations, it should be kept in mind that the data may not

be ideal. They represent the best that can be done in the current generation

of simulations and do not necessarily reflect appropriate levels for an accurate representation of battle.

Estimates must be made of the time required for controllers to perform each

task. For data base entry and retrieval tasks, one of the best ways to accomplish this is to build a mock-up or prototype of the man-machine interface

early in the design process. Experiments with the interface prototype will

serve to both improve the design efficiency and provide workload estimates for

the individual tasks. Additionally, potential data entry bottlenecks will be

revealed. The interface tests should always be conducted with inexperienced

personnel to insure that excessive controller requirements are not built into

the system.

Besides contributing to the estimates of controller workload, measures of

the time required to perform the various tasks are necessary to assess adequacy

of the simulation. Most of the tasks can be considered time-critical, in that

excessive delays will seriously impair the realism of the exercise. For example, when indirect fires are called for, either too responsive or too sluggish

artillery action will provide unrealistic training conditions. In general,

realistic delays can be built into the simulation where needed. Data entry

bottlenecks, or other input delays in time-critical tasks, however, indicate

that the interface must be redesigned for more efficient operation, more personnel must be assigned to the task, or that greater portions of the task must

be automated.

The manning scheme can be assessed by considering each position, determining whether the average and peak workload levels and the skill and knowledge

requirements are reasonable. The number of control personnel can be reduced by

shifting tasks, combining role play with model control and evaluation tasks,

"doubling up" of roles or making other adjustments to the extent that military

knowledge requirements permit. The staffing goal needs to be met, of course,

and also some survey of the units which will ultimately use the system must be .

made. It should be determined whether the units are willing to provide sufficient numbers of persons with appropriate skill levels to support the exercise.

If it is determined that planned controller levels cannot be reached, the process must be iterated, starting back at the stage where tasks are allocated to a4

the computer or the controllers and testing alternative allocations and task I

at

30

L ut .e a.%
AI,

a.-

S.

A.

designs. The analyses previously conducted, for example the prototype interface A.

tests, should indicate which tasks are the most time consuming and where the

greatest potential saving can be made.

S

4,.

.1'

~'JA

V

VVS

-A

S

S

.1'

-A

I

t

.1*

-A

'p

I

A.

A'.

A'

I

4

S

'I.

Na

S

31

-A
SECTION 7. CONCLUSIONS AND SUMMARY RECOMMENDATIONS -.

This section recapitulates the main points of the previous sections. Most

of the recommendations presented are consistent with the current evolutionary

development approach, though increased emphasis is placed on analysis of the

tasks of both the training audience and the training support staff.

Training Objectives

Collection and analysis of tasks, conditions and standards is a necessary

precursor to development of a training system. The ARTEP manuals are a good

starting place but further development of training objectives is necessary.

Some personnel typically included in the training audience are not provided

adequate training opportunities. It is recommended that a way to provide adequate training be found or these people be removed from the training audience.

Tools should be provided for development and modification of scenarios.

Included with the tools should be on-line help facilities and documentation of

how the tools should be used. If systemic wargaming is incorporated as proposed, the model should be adapted for testing and modification of scenarios.

Performance Measurement

Numerous recommendations were made in section 3 for various levels of support for diagnosis of deficiencies and provision of feedback. These include:

9 Develop templates for instituting the information flow methodology, to

be filled in with specific items of information from the scenario. Document

the methodology and the sources of information in the data base that are to be

used.

* Develop a list of probes, along with a means for automatically notifying

appropriate controllers to insert them, either at pre-set times or in response

to simulation events.

" Provide automatic detection of events based on common errors that indicate failures in staff planning or coordination.

" Develop normative data from model runs for interpretation of mission

accomplishment data.

9 Develop a watchdog program for the staff's tactical data system to track

preparation and delivery of reports.

* Develop procedures to compare the ground truth data in the training

system data base with the staff's picture of the battle as reflected in the

tactical data system.

* Develop analytical wargaming procedures to evaluate alternative decisions.

32

**r
Simulation Requirements

Beyond basic considerations of resources to portray, resource limitations,

and level of detail (i.e., model fidelity ) these additional areas were

discussed:

a. Software maintenance was discussed as a basic factor in a system's

success as a home station training device. Comprehensive documentation of the

system and use of modular programming techniques were recommended.

b. The training system should be designed to minimize exercise preparation

effort. Scenario development, controller training, and data base modification

are amung the most labor-intensive aspects of exercise preparation.

c. Emphasis should be placed on training to overcome obstacles to effective information processing and decision making. This implies that control is

needed over message loads, accuracy of information, integrity of communications

and other factors which directly affect the difficulty of staff work. A

"How-to-Train" manual should be included as on-line documentation.

Define Control Tasks

Three general areas of controller function were discussed: exercise control, model control, and role play. Control functions required by various

performance measurement techniques were discussed. These impose system requirements for communication among controllers, observation of the training

audience and interfacing with various automated measurement techniques.

Minimization of model control functions was recommended, either through development of a systemic model, through close attention to human factors engineering of the user interface, or through automation of low level, repetitive

controller decision making functions. Additional support for role playing was

discussed under two alternatives: automate production of message traffic, or

design the simulation interface to supply the information that subordinate unit

commanders would ordinarily have through observation and communication with

their staff and their own subordinate commanders.

Estimate Support Staff Requirements

This principally involves setting realistic goals based upon availability

of personnel at the field locations where the training system is to be used,

analyzing both the workload imposed by the system and the military knowledge

requirements for role playing functions and successively refining the system

until it can be operated by the available personnel.

A preliminary design based upon current training systems can be used to

detail all of the functions to be portrayed. These functions can be further

described in terms of how the current design is intended to operate and in

terms of what control functions will be required for the current design for

model control, role playing, and evaluation.

33

Rt
Data on the number of simulation events (e.g., movements, fire missions, and

air strikes), to be portrayed per unit time can be derived from current command

post or field training exercises. Similarly, data on the number of voice and

digital transmissions per unit time can be obtained. Average time per message S

can be directly obtained from the transmission data to estimate the average and

peak communications workload for controllers.

To estimate model control workload, initial values can be obtained analytically, but for greater accuracy, it is recommended that a prototype of the

interface to the simulation be created as one of the first steps in system

development. Analysis of time and errors in the use of the interface can re- N,

veal potential data entry bottlenecks and areas of model representation that

demand further automation before the simulation software is built. Performance

tests on the prototype interface should use inexperienced personnel to insure

that excessive controller training requirements are not built into the system.

Workload estimates from this preliminary analysis can be used to determine

the minimum number of controllers required to perform role playing functions

and the minimum number of additional personnel needed to meet model control

requirements. This number added to the training management and evaluation

staff is the minimum required by the preliminary design. To reduce the abso- -

lute number of people required, "doubling-up" of functions should be examined,

adding model control tasks to role play and evaluation personnel and giving

multiple roles to role players to the extent that military knowledge requirements will permit. A comparison to staffing goals will then probably indicate

that further reductions are needed. The data from the prototype interface can

be used to target the most time consuming functions for redesign.

Final Remarks r%

We recommend two radical departures from incremental improvement of existing systems. These are (1) base the simulation model upon a systemic analytical wargame and (2) begin development of on-line performance measurement

techniques based upon the interaction of the training audience with a tactical

data system. These steps are believed to be necessary to cope with the most

critical shortcomings of current command staff training systems: excessive

control personnel overhead and the lack of performance measurement capabilities.

34

2V
- - - - - *j,-wvwyw-Y

REFERENCES

Barber, H. F., McGrew, J. F., Stewart, S. R., & Andrews, R. S. (1979). The

computer-based map maneuver system: A preliminary examination of its

training effectiveness and suitability for use as a research vehicle, ARI

Research Memorandum 79-9. AD A077 981

Barber, H. F., & Solick, R. E. (1980). MILES training and evaluation test

USAREUR: Battalion command group training, ARI Research Report 1290. AD

A109 192

Barr, A., & Feigenbaum, E. A. (1982). The handbook of artificial intelligence, Vol III, William Kaufman: Los Altos, CA.

Carter, C. F., & Patton, M. S. (1985). Command group behaviors: their identification, quantification, and impact on collective output in automated

and non-automated environments, ARI Working Paper FLvFU 85-3.

Fallesen, J. J., Lussier, J. W., & Garlinger, D. K. (1986). Trip report.

(Available from Fort Leavenworth Field Unit, ARI Leavenworth, Kansas.)

Jackson, M. (1983). Systems Development. Englewood Cliffs, NJ: PrenticeHall.

Kaplan, I. T. (1980). Information flow in battalion command groups. ARI

Technical Report 499. AD A109 469

Kaplan, I. T. (1981). Communication patterns in decision-making groups: Information received versus information required. Paper presented at the

meeting of the Eastern Psychological Association, New York.

Kaplan, I. T. (1984). Strengths, weaknesses, and recommendations for improvement of the Army Training Battle Simulation System (ARTBASS), ARI Working

Paper FLvFU 84-1.

Kaplan, I. T. (1985). Lessons learned in research on command group training,

ARI Technical Report 751, USARI, 5001 Eisenhower, Alexandria, VA.

AD A186 496

Kaplan, I. T., & Barber, H. F. (1979). Evaluation of a computer-assisted map

battle simulation: CAMMS versus a CPX, ARI Technical Paper 355.

AD A068 014

Kaplan, I. T., & Fallesen, J. J. (1986). After action review (AAR) guide for

ARTBASS, ARI Research Product 86-32, USARI. AD A180 797

McGrew, J. F. (1980). Computer generated 3-D terrain graphics for military

training. Paper presented at the 46th MORS Conference, Newport, Rhode

Island.

Michel, R. M., & Solick, R. E. (1983). Review of literature on the effects of

selected human performance variables on combat performance. ARI Working

Paper FLvFU 83-4.

351

,* _
-NQN VW MMI717 -1 777 1-77 777:7 2- -w -X71 .- V

I

Miller, G., & Bonder, S. (1982). Human factors representations for combat

models, ARI Technical Report 571. AD A133 351

Monk, A. (ed) (1984). Fundamentals of Human-Computer Interaction. London: p

Academic Press.

Olmstead, J. A., Baranick, M. J., & Elder, B. L. (1979). Research on training

for brigade command groups: factors contributing to unit combat readiness, ARI Technical Report TR-78-A18. AD A076 916

Olmstead, J. A., Baranick, M. J., & Elder, B. L. (1979). A training feedback

system for brigade command groups. ARI Technical Report TR-78-A19.

AD A071 116

Pressman, R. S. (1982). Software Engineering: A Practitioner's Approach.

New York: McGraw-Hill.

SAIC (1985). Corps Battle Simulation (CORSIM) Hardware Requirements Design

Document. Final Report under contract number DAEAI8-81-G-0017, D.O.

6001, SAIC-85/1126.

Shaker, E., Ben-Basser, M., Madni, A., & Leal, A. (1979). Applications of

adaptive programming technology (APT) to command group training and

performance measurement, ARI Technical Report TR-79-A22. AD A076 234

Shannon, R. E. (1975). Systems Simulation. Englewood Cliffs, NJ: PrenticeHall.

Thomas, G. S. (1983). Battle simulation outcomes as potential measures of

battalion command group performance in CATTS exercises, ART Working Paper

FLvFU 83-1. V

Thomas, G. S., Barber, H. F., & Kaplan, I. T. (1983). Improvement in command

group performance as a result of diagnostic feedback and CATTS training,

ARI Working Paper FLvFU 83-3.

Thomas, G. S., Barber, H. F., & Kaplan, I. T. (1984). The impact of CATTS

system characteristics on selected measures of battalion command group

performance, ARI Technical Report 609. AD A140 231

Thomas, G. S., & Cocklin, T. G. (1983). A model of mission accomplishment in -%

simulated battle. ARI Technical Report 599. AD A148 234

Thomas, G. S., Kaplan, I. T., & Barber, H. F. (1984). Command and control

training in the Combined Arms Tactical Training Simulator, ARI Technical

Report 615. AD A142 742

Thomas, G. S., & Solick, R. E. (1982). Training and human factors issues in

the MACE concept evaluation program, ARI Working Paper FLvFU 82-2.

TRADOC PAM 351-4(T). Job and task analysis handbook.

Woodson, W. E. (1981). Human Factors Design Handbook. New York: McGrawHill.

36

AL% L% 5_1d A