- -

~~~ 

—

1

~ / ~~~) -

-

~~~~~~ Thchnlcal R.ps,I

~~ 3 — ‘. 

,

- .~~~~~~~~~~

e0MBAT OPERATIONS TRAINING )

/~ FFECTWENESS ANALYSIS MODEL~1

/

1979 PERSPECTIVE . I

4

~

. L J

1:”.’

Steven M. MedIj n

ENGAGEMENT SIMULATION TECHNICAL AREA

~~ ‘

1 ~~—~~~ ~~ 

~~ 2 ~

“ 

~/ 7

~ 

‘

~~

/

D DC ~~~~~~~~~~~~~~~~~~~~~~~~~~~

10 1919

~~~ij ; 

~~ , 

U

U. S. Army

Research Institute for the Behavioral and Social Sclei,ces

2 // ul

fll!J / ~~~

Approv ed ~

er public riles..; dist ributi on unlimited. /

,

~~~ 

6, 79 ~ 2 7 0 65

I

____________________________ -— —
__ ~ ---- —~~ 

U. S. ARMY R ESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENC ES

A Field Operating Agency under the Jurisd iction of th e

Deputy Chief of Staff for Personn el

WILLIAM L. HAUSER

JOSEPH ZEIDNER Colonel . U S Army Technical Direct or Command er

- .

Ti LA !

NOTICES

DISTRIB UTI ON: Pr.msry distribution of this report Pt~ bsen ni~~ by ARI. Pt.... eddrus corrsst,ondsn~

. concerning distribution Of sports to: U. S. Army Pa~ srch Instilut. f*r the Behavior•I snd Soci.l Scisnces. ATTN. PER1.P, 5001 Eis.nhoocr Avenue. Aio~andrii Virgi ni. 22333.

FINAL DISPOSITION This re oqt my be dsstroyid ssfien it is no longer needed . PIN do not return It to

the U. S. Army N.s,.rcsi Institut, for the Schevior.I end Soci l Sciences.

~

QIL Th. findings in this report ev, not to bi construed as an offlci.I Ospo,tvnent of the Army position, unless ce des~nItsd by other .uthoriled documents.

LI ~~~~~~~~~~~~~~~~~~ L ___ _ _______
- -

Unclassified

SECURITY CLASSIFICATION OF THIS PAGE (Wh en 0.,. ftnt.r.d)

~~~~~~~~~~ r

~~~,-. i &a~~ ki~~~A,9r~IJ DA 1

~~~ 

READ INSTRUCTION S i

~~~~~FVl~~ I U’JS..VMCI~~ U ~~ I itJ1~ ~ BEFORE COMPLETIN G FORM

F~ REPORT NUMBER 2. GOVT ACCESSION NO 3. RECIPIENTS CATALOG HUMBER

Technical Report 393 ____________________________

4. TITLE (end Subtiil.) S. TYPE OF REPORT & PERIOD COVERED

COMBAT OPERATIONS TRAINING EFFECTIVENESS -- ANALYSIS MODEL: 1979 PERSPECTIVE S. PERFORMING OR G. REPORT NUMBER

7. AUTHOR(s) S. CONTRACT OR GRANT NUMS ER(a)

Steven M. Medlin --

C. PERFORMING ORGANIZATION NAME AND ADDRESS to. PNOGRAM ELEMENT PROJECT . TASK

U.S. Army Research Institute for the Behavioral

and Social Sciences 2Q762722A764

5001 Eisenhower Avenue , Alexandria, VA 22333 ___________________________

II. CONTROLLING OFFICE NAME AND ADDRESS t2. REPORT DATE

Deputy Chief of Staff for Personnel ju1

~’ 1979

IL NUMBEROFPAGES Washington , DC 20310 20 14 MONITORING AGENCY NAME C ADDRESS(U dltl.,.n t from Cantroltin4 Olfic.) *8. SECURITY CLASS. (of thu ?.port)

—— Unclassified

IS.. DECLASSIFICATION/DOWNG RADING —

SCHEDULE

15. DISTRIBU TiON STATEMENT (of hi. R.port)

Approved for public release; distribution unlimited .

I?. DISTRICUTION STATEMENT (of ha ab.tr.ct ant.r.d in Block 20, i f diff.ronl from Report)

IS. SUPPLEM ENTARY NOTES

II. I

~EY WORDS (Continu. on ever.. aid. if n.c...my end fd.nif~ ’ by block ncenb.r)

Army Training and Evaluation Program (AI~

rEP) Evaluation system

Combat Operations Training Effectiveness Expected values subsystem

~ Analysis Model (COTEAM)

.

, 

Comparison subsystem ‘

~~ gagement simulation

25 ~~~~~ RACT~ (— ~~~~~~~~~~~ — orvoree .5* N .

~~~~—-—, ~~~ IdaevSfr by block bar)

~ The Army Research Institute has been involved in a systematic program

of research on unit evaluation. The goal of this research is to develop a

criterion—referenced system for evaluation of unit tactical performance . A

model for an evaluation system is developed using the current ARTEP evaluation

system as a starting point. The mode l also provides (a) a realistic simulated

combat environment , in which units perform tactical operations and objective ~

(Continued)

DO ~~ W3 £Drnos or ’N ovss ls o.soi.ErE 

i. Unclassified

$EcumrY CLAUW1CAThON oF Tht1 PAGE (Shari Data EIISt.d) 

~

:. :TY1. ~~~ _ i . - 1

.

_

~~. ~~—
Unclassified

sECURITY CLAUIPICATION OF THIS PAGE(Whmi Data Zni.ssd)

tern 20 (Continued)

performance data can be obtained ; (b) procedures for defining standards

against which unit performance can be compared; and (c) techniques by which

training deficiencies and training level/combat readiness can be assessed .

A review of the current status of the COTEAM evaluation system m di- f

cates that research is being conducted on all aspects of the model: (a) The

ARTEP manual is being modified to provide empirically determined behavioral

objectives, a list of training diagnostic behaviors, objective standards,

and a program of instruction for ARTEP evaluators. (b) The COTEAM simulated

combat environment exercises are superior to ARTEP field exercises in pro—

moting tactical proficiency , terrain appreciation , and use of cover and

concealment in providing immediate , valid feedback to individuals and

weapon systems; in enhancing troop motivation ; and in allowing ample opportunity for collection of objective data on unit performance. (c) A set of

systematic methods for defining specific performance criteria against which

unit performance can be compared are being developed , validated , and implemented . (d) A comparison process, in which behaviors observed in field •

exercises are compared to performance criteria, is being explored. This

comparison subsystem allows for decisions concerning the training level !

combat readiness of the unit being evaluated .

Unclassified

SECURITY CLASSIFICATION OF THIS PAGE(Whe n Data Rntar•d)

I. - .

~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~ -~~-- L . _ ~~~~~~~~~~~~~~ -~~~~~~~~~~ 
— ~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

— ----

Technical Report 393

COMBAT OPERATIONS TRAINING

EFFECTIVENESS ANALYSIS MODEL:

1979 PERSPECTIVE

Steven M. Medlin

Angelo Mirabella, Team Chief

Submitted by:

Robert T. Root. Acting Chief

ENGAGEMENT SIMULATION TECHNICAL AREA

Approved by:

Frank J. Harris. Acting Director

ORGANIZATIONS AND SYSTEMS

RESEARCH LABORATORY

U.S. ARMY RESEAR CH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 EIsenhowe r Avenue. Alexandria . VirgIn ia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

July 1979

Army Project Numb r Training and Education

2Q7e2122A754

A4Whrossd for public rblssss; distribution unlimited. 
- . - . -

~

- _~ j

~ J~~T T:-.--

~

- -

~ —~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ‘

~~~~~ 

.---- - - -

ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready for implementation at the time pf publication are presented in the last part of the Brief. Upon completion of a malor phase of the task, formal recom- mendations for official action normally are conveyed to appropriate military agencies by briefing or Disposition Form.

~

iv

—— --‘.-. ~~~~~~~~~~~~~~~~~~~~ . .~-- -
FOREWORD

The mode l presented in this paper was developed under the Training

and Education Project in the Engagement Simulation Technical Area of

the Army Research Institute for the Behavioral and Social Sciences (ARI).

The goal of this project was to provide quantitative methods for evaluating unit proficiency. The means for achieving this goal included

basic research in criterion—referenced test methodology , measurement

and scaling model s, and decisionmaking implications of test score

interpretation .

i hc Combat Operations Training Effectiveness Analysis Model ( cOTEAM )

is a criterion-re ferenced evaluation system based on the Army Training

and Evaluation Program (ARTEP). The model integrates aspects of all

parts of the Technical Area work program : evaluation of small combat

units in a simulated combat environment (ARTEP, REALTR.AIN), ARTEP evalu—

~tor training , methods for developing performance criteria , and improved

training—diagnostic feedback procedures . Research from all of these

areas is related to the COTEAM framework .

Antic ipated future research under the Training and Education Project includes the development of a computer model for performance evaluation and development of measurement , scaling , scoring , decisionmaking ,

and quality-control models for use in performance evaluations when

criterion-referenced testing procedureS are employed .

ARI research in this area is conducted as an in-house research

effort . The research program is responsive to the requirements of Army

Project 2Q762722A764.

S H ZEID~~1

ec ical Director

V

____ _____________
- -

COM~AT OPERATION S TRAININ G EFFECT IVENESS ANALYSIS MODEL :

1979 PERSPECTIVE

BRIEF

Requirement:

To present the current status of theory and resaarch relevant to

the Combat Operations Training Effectiveness Analysis Model (COTEAM),

an evaluation system for small combat units.

Description of the Model :

The current Army Training and Evaluation Program , ARTEP , possesses

several weaknesses. COTE.AM is based on the ARTEP framework but corrects its shortcomings by providing (a) a realistic simulated oombat

environment in which units perform tactical operations and objective

data èan be obtained , (b) procedures for defining standards against

which unit performance can be compared , and (c) techniques for assessing training deficiencies and levels of training or combat readiness.

Findings :

A review of the current status of the COTEAM evaluation system indicates that research is being conducted on all aspects of the model :

(a) The ARTEP manual is being modified to provide empirically determined

behavioral objectives , a list of training diagnostic behaviors, objective standards , and a program of instruction for ARTEP evaluators.

(b) The COTEAM simulated combat environment exercises are superior to

ARTEP field exercises in promoting tactical proficiency, terrain appreciation , and use of cover and concealment; in providing immediate , valid

feedback to individuals and weapon systems; in enhancing troop motivation; and in allowing ample opportunity for collection of objective data

on unit performance . Cc) A set of systematic methods for defining specific performance criteria against which unit performance can be compared

are being deve loped , validated, and implemented. (d) A comparison process, in which behaviors observed in field exercises are compared to per—

formance criteria, is being explored . This comparison subsystem allows

decisions to be made concerning the training level and combat readiness

of the unit being evaluated.

vIl
Utilization of Findings :

The findings of this presentation are being used to further refine

the evaluation system to be implemented in the Army and to foster fur- - ther research in the assessment of unit performance in field exercises.

3:

t

viii

I ________

I. ____

—~ -- — - . — -—— - --~~~- - —-

~~ --

~ a
CC~48AT OPERATIONS TRAINING EFFECTIVENESS ANALYSIS MODEL:

1979 PERSPECTIVE

CONTENTS -

Page

INT~~DUCr ION 1

COTEAM 4

Modifications to the ARTEP Manual 6

~~gagement Simulation Field Exercise Test Bed 7

Expected Values Subsystem 9

DELPHI 9

Board Games 11

Computer Simulations 12

Sua~nary of Expected Value Generation 12

Comparison Subsystem for Training Diagnosis 13

SUMMARY 14

REFERENCES 17.

DISTRIBUTION 19

LIST OF FIGURES

Figure 1. ARTEP evaluation system 2

2. COTEAM evaluation system

ix

________________ -.~~~~ —-—-“.—~--—.—~

- _

~~ $

~—.~

-—-_-~~~~.

- 

~~~~~~~~~~~ .— 

-

_________ —- -- ---- ---- - .- ~~~~~~~~~~~~~~~~~~~~~ -

~~~~~ . ~~~~~~~~~~~~ ~~~~“
COMBAT OPE RATIONS TRAININ G EFFECTIVENESS

ANALYSIS MODEL : 1979 PERSPECTIVE

INTRODUCTION

In recent years , the Army Research Institute for the Behavioral and

Social Sciences (ARI) has been involved in a systematic program of research on unit evaluation . The goal of the research is to develop a

criterion-referenced system for evaluation of unit tactical performance .

Cons iderab le progress has been made in developing evaluation systems

that provide (a) a realistic simulated combat environment in which units

perform tactical operations and in which objective performance data can

be obtained , (b) procedures for defining standards against which unit

performance can be compared , and (C) techn iques for assessing training

defic iencies and training/combat level readiness. One of the current

model s, the Combat Operations Training Effectiveness Analysis Model

(COTEAM) , uses the current Army evaluation system—-the Army Training and

Evaluation Program (ARTEP)--as a starting point but also includes the

aforementioned critical features.1

As curren tly used , the ARTEP includes an ARTEP manual , a field

exercise evaluation , and a training program based on the evaluation .

Figure 1 presents a flow chart of the system. As the figure suggests,

the ARTEP manual drives the evaluation system. Based on doctrine and

military expertise , the ARTEP manual was designed to be a training and

evaluation guide . Evaluation requirements are stated in terms of specific unit performance objectives and focus on whether a unit can perform specified missions. Task statements are written at an integrated ,

functional , mission-oriented level. Conditions for performing each

task have been specified to allow greater standardization of the evalu—

ation . Training and evaluation standards provide a relatively objective

basis on which the evaluator can judge the unit’s performance on a par—

ticular task; the standards also can be used to develop training programs.

The external conditions are factors that influence the ARTEP field

exercise and that can be controlled by the evaluator . The ARTEP manual

and doctrine determine mission , posture , personnel , weapons , ammunition ,

and petroleum , oils , and lubricants (POL) . The evaluation team determines whether the terrain and weather are suitable , and the time of day

model was originally outlined by Eugene Johnson , K. I. Epstein ,

and Angelo Mirabella as part of ARI basic research on criterion—

referenced evaluation . Its features have been described in Epste in and

Johnson , 1976 , and Mirabella , 1977. Related evaluation concepts have

been discussed in Erwin , 1976 ; Root , Knerr , Severino , and Word , 1978;

Sulzen ( undated draft report) , Sulzen and Root , 1976; and Sulzen , Root,

and Epstein , 1976.

1

— - -

~~
- --4 ~

4

C

~

0 ~ C

~~ --4 ~~ __________________________________

C

0

C ~~ --4

~ 0 ‘

J

- - -4 r~~~(

~ .—~~0 - --4 0) ~- -

~~

~

0)

C) c~

-

. )

~~--4 C-. ~ 

— --4

0)

--4cL.

0)

U

—4 

C 4-4 ~

—4

0) E-’ ~ --4

i-4 C0) CCO ~~~~~~.r-4 (I) ~ Q~

~

-4 ~~~~~~~~ - - 40).-4 C

r~ ~~~~~~~~ 

4 X ~ - .-4 fl --40)~~~

U) 4.) ~

24 O ~44J c1) Z > —4 C~~~ a

~ ‘

~~~E-~ --- ~~‘- -‘

~~~.-4 0 ) d

0 ~~

0

J

~~

CC>,

2 

~~— - - -~~~ -

~~~-~~~~~~~~~~
and intelligence for the exercise . (These are examples of external

conditions , not an exhaustive list.)

Af ter con sider ing the external conditions , the evaluation team

des igns a field exercise based on the unit performance objecti ves in

the ARTEP manual. Many characteristics of the unit being evaluated

affect its performance in the field exercise ; for example , the individual skills of the unit’s members , the unit’s training level , and the

unit ’s leadership. During the field exercise , the evaluation team uses

the ARTEP manual as a guide in the evaluation process. Based on the

unit’s performance in the field exercise , the evaluation team makes decisions on training level/combat readiness and training remediation .

A training program then is designed to strengthen the unit ’s deficiencies and maintain its proficiencies . As Figure 1 shows , this training

program primarily affects the characteristics of tne unit, which in

turn affect the unit’s performance in its next field exercise .

The ARTEP has several major weaknesses that make an improved corn—

bat evaluation system necessary. The first weakness lies in the lack p

of standardized or scientific procedures for determining the tasks, V

subtasks, and standards in the Training and Evaluation Outlines (T&EOs)

of the ARTEP manuals. The current content of the T&EOs was chosen by

military experts without the benefit of available procedures for in—

suring consensus among different teams of experts.

The second major weakness in current ARTEP use is that the field

exercises are often unrealist4c and do not provide objective data for

the evaluation team. If the field exercises do not provide a realistic

combat environment, then the evaluation of combat units in this setting

has little or no validity.

Third , the ARTEP manual provides little or no guidance to evaluators on how to design exercises , measure unit performance, determine

training proficiencies/deficiencies , or evaluate the observed perform—

ance . Effective and efficient measurement methodology is lacking , particularly with regard to objective measures of performance and optimal

utilization of limited evaluation personnel and equipment . Current

practice relies heavily on subjective evaluations and the ability of

the evaluation team to anticipate critical events during an exercise

and to insure that those events are observed . At best, such evaluation

yields good qualitative judgments , but there is no assurance that the

judgments will not be more or less arbitrary.

The ARTEP is currently dominated by unsystematized , unaided , idiosyncratic human judgments. The standards tend to be open to cons-.der—

able interpretation . The complexities of combat are too great to be

handled by an evalua tion system based solely on an intuitive , subjective ,

ascientific approach. Although human judgment must play a significant

role in the evaluation process , it must be systematized , aided by

information—processing technology , and made less idiosyncratic. The

lack of objective data and an explicit criterion data base make it

3 

-~ - - ‘1

~ 

~~~~~~~~~~~ ~~~~~~ - -
difficult to interpret the results of an evaluation , to compare dif f er—

out units , or to compare the same unit at different times or locat ions.

The evaluation system needs to be driven by objective , quantitative

data . Systematized , aided human jud gment should be supported by quantitative measures as much as possible. Because of the complex nature

of field exercises, the system should incorporate multiple measures at

several levels ot resolution .

Thi’ fourth weakness in currenL ARTEP use is the lack of guidance

on how users are to dea l with the partially stochastic nature of combat . Estimates must be made of the degree to which chance event s reduce the accuracy of effect iveness measures and therefore reduce the

validity of the evaluation decisions . L~ w emp irical correlations between variables that would be expected to be hi ghly related on loqical

qroundu can occur for many reasons . Such reasons inc lude stochastic

effects , uniquely effective or i n e f fec t i ve behavior by the opposing team ,

nontraLning sources of variance that may n~

isk learning effects, and the

suppressor variable effects of events that intervene between a process

and a later outcome . This suggests that the evaluation system should

conside r the processes leading up to the final outcomes , as well as the

final outcomes.

Finally, the ARTEP d~~’s not provide guidance in how to develop

rain inq programs from the field exerc ise evaluat ion . Although the

ARTEP is a t r a in i n g and evaluation guide , units typically train to pro—

par e for taking an ARTEP evaluat ion , or test. After an ARTEP exercise ,

feedback to the unit is provided , but the feedback is neithe r timely

enough nor in a form that is useful for developing a training program.

~~zrthermore , many aspects of lower echelon performance are neglected

during the larger ARTEr t it’ hl exercises; this violates the ARTEP concept

of mult —e chelon training and evaluat ion and prevents the integrat ion of

pert i nent information from lower echelons into the training program .

Thus , ci it i

~-ai aspects of training within the ~

.RTEP are lost (Havron ~ Wanschura , 1979).

L’CTEAM

The ~OTEJ~

M evaluation system uses the ARTF.P as a starting point

and systematically addresses the ARTEP ’s weaknesses. Figure 2 present s

a flow chart of COTEAM . A comparison of Fi ures 1 and 2 shows that

cOTF.AM is based on the ARTEP evaluat ion system . The model modifies the

current ARTEP manual and adds these main components: (a) an engagement

simulation field exercise test bed , (b) the expected values subsystem,

and (c) a comparison subsystem for training diagnosis.

4

-- - - - A ~~~~~~~~ -

~ - ~~~—~~~--~~~~~~~~~~~~~~ ~~~~~ , ~ - - - - - —~
E s

o e o

0

0 0 041 _____ --4 t

~~~’~

U, ) • .-4 0) ’e .-4 C) ~~~~~~~~~~~ S

0 44.4 C) _____

0V41 1F%

0

U)

‘—4 0)

gz w0 )

0_ o x ..

0 0 0) 0)5

0 0 a

U

41

U)

______ U,

C

.2 C 41 o v

415 -4 ( U 6 ~

U, , .-4 0) 0)

—I o~~ ( > S 0 0 ) 1)

4J U1 C U > , ~) U) d ,-4~~ ( t

~

’1 c14 _o

U

~ 

L~~~~~~~~~~ d IL

~ E-i~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ < >0) )< O - . 0 ww e 0 4) e 0 4 1 ~~ ~~~0 Z~~~~~~~~~~~~ Ai &4~~~~H U .

~ ~~~~~~~~~~~~~~~~~~~~~~ __ ____________ 

~~~e

C

~~~~~ .4 --4

‘~~~s ~~ ‘U Des

5

___ L -

~~~~~~~~~

‘

~~

-

~~~

—- - -
Modifications to the ARTEP Manual

Traditionally, the ARTEP manual and the front-end analysis of behavioral objectives, performance variables, and measurements have re- lied heavily on expert military judgments. At ARI, efforts have been

made to use standardized , scientific procedures based on empirical data , as a way to increase experts’ consensus and as a means of testing and

verifying expert opinion . These methods also provide a basis for re- ducing many expert opinions down to a manageable set of variables that are most important in assessing unit performance in field exercises .

One research effort has focused on identifying variables that appear to be useful in discriminating among units differing in tactical

proficiency. Using a large data base from combined arms field exercises, critical incidents within exercises have been identified, as well as the causes or precipitating factors of particularly damaging or helpful

tactical events. These critical events include particularly long delays

between acquiring targets and delivering indirect fire , units that ex- pose themselves for long periods of time, and breakdowns in conununications. A prototype ARTEP manual is being developed with a set of diag- nostics based on these critical inc idents. These diagnostics are “enabling behaviors”; that is , a unit that performs the behavior is more likely to complete its task or mission than a unit that does not . Evaluators also can use these diagnostics to help develop training programs based on the unit ’s ARTEP evaluation. By noting which tasks were not

completed successfully and , with the aid of the diagnostics, tracing

back to determine why a tack was not completed , the evaluator can assess the uni t ’s training deficiencies and develop a training program to cor- rect them.

A second research effort is aimed at defining the major dimensions or factors that military judges use to assess unit performance in field

exercises. Multidimensional scaling (MDS) techniques are being explored as a means of identifying these important factors . Using written nar- ratives of 15 different armor/anti-armor field exercises , military offi— cers were asked to rate each unit’s performances. One of the 15 narra- tives was used as a “target” ; the other narratives were compared to the

target along certain specific criteria . For instance , the officers were

asked to “judge how similar each narrative is to the target narrative

with respect to the performance of the combat unit in the narratives. ” Each of the 15 narratives was used as the target ; comparisons were made

between all the other narratives and the target . A similarity matrix

composed of all the comparisons was used as input to an MDS computer

program , which mathematically determined how-many factors, or dimensions , the judges used in their comparisons . The results yielded a threedimensional space ; that is , the judges used only three factors to evalu- ate unit performance.

6 

~---~~~--- --—— ——— -

—‘~~ -

- - -

-

-.-

-- 
The next step in this research is to define or label these dimensior~s. To do this , a list of possible dimension—defining attributes

is composed , a new set of military judges then ranks the narra tives

with respect to how much each narrative is characterized by the attributes , and these comparisons are used as input to a second MDS analysis.

The set of three attributes that defines a three—dimensional space

most similar to the original is the set of factors that military judges

consider to be important in assessing unit performance in field exercises.

A third area in which the current ARTEP is being improved is in

the training given to the evaluation team. A detailed program of instruction (P01) (Havron , McCullough , McFarling, & Wanschura , 1979) for

evaluators is being developed and field tested . The P01 teaches the

evaluators how to design and set up field exercises , obtain observations ,

assess unit performance , and provide feedback to the units. The guide

includes the rationale for the development of the P01, which stresses

the roles and importance of the evaluators; a command-planning module,

which presents the planning necessary to conduct an evaluator training

session; and the P01 itself.

The P01 provides structured guidance for the evaluators in all aspects of their duties , including each evaluator ’s role and the required

coordination of roles , methods for controlling field exercises , use of

pyrotechnic simu lators, terrain reconnaissance for exercise lanes, data

analysis, report preparation , and postexer cise critiques. The P01 constitutes the first significant attempt to develop guides for persons

who evaluate unit performance in field exercises.

Engagement Simulation Field Exercise Test Bed

Although some engagement simulation (ES)- technology has been fielded

by the Army , most current ARTEP field exercises do not use objective

methods for simulating and assessing casualties. Some question exists ,

therefore , of whether the behavior displayed in these exercises represents true combat performance . The evolution of engagement simulation

from SCOPES and REALTRAIN (TC 71-5) to the Multiple Integrated Laser

Engagement System (MILES) has increased dramatically the realism of the

combat environment and the objectivity of casualty assessment. The

National Training Center (NTC) will move this evolution a step farther

by providing a computer—controlled , instrumented range with unique weapon

signatures for each weapon system in field exercises up to battalion size .

Engagement simulation exercises are superior to the ARTEP field exercises in promoting tactical behaviors , terrain appreciation, cover ,

and concealment (Havron , McFarling, Hill , & Wanschura , l979b; Scott,

Meliza et al., 1979; Banks et al.,, 1977; Meliza et al., 1979 ; Scott ,

Banks et al., 1979). Engagement simulation provides immediate and valid

feedback to individuals and weapon crews and enhances troop motivation

and presumably readiness to learn (Sulzen & Bleda , 1979). The after

action review (AAR) gives the unit , weapon crews, and individual soldiers

7 

~~~~~~~~~ ~~

— ~ _ -

~~ —--.-..- -~

---- - -

~~~~~ ------ - -- - —
- ~

-

~ 

--—--—- --—~ - --- - ----- ~~ -- ---—-—-- -- —~~~- - -

~- - —~~

--- - --~~ 

an opportunity to learn from their mistakes and benefit from the experiences of the other participants.

In addition to providing a realistic combat environment, ES exercises allow ample opportunities for collection of objective data on unit

performance . In March 1978, ARI produced and field tested at Fort Carson , Cob ., a low—cost tactical operations measurement system that is

computer-supported (Epstein , 1978). The system includes methods and

measurement forms for recording tact~.cal processes , position location

da ta , maneuver routes , and tactical outcomes without instrumented data

collection . The data are reduced and coded Using machine—scorable optical scanning forms and are organized by the computer to provide a

coherent data base suitable for a wide variety of analyses. The NTC

will improve this data collection effort by using instrumentation that

automatically records high-resciution position location , time , and

engagement data.

In an effort to implement ES in the field , ~.RI has conducted an -

~~~

ongoing research program designed to incorporate the ES methodology into

the ARTEP framework. This incorporation may not be a trivial , mechanical problem. ARTEP currently consists almost entirely of procedural

statements of tasks , condi tions , and standards for one side in a twosided game ; that is , it is formatted as a procedural checklist . ES,

however , was designed to improve training by increasing realism , particularly with respect to casualty assessment. The problem is to combine

the high-fidelity outcome data available with ES with the procedural

formats related to critical- processes included in ARTEP .

The modified ARTEP discussed above attempts to relate the highfidelity casualty data obtained using ES methods to specific ARTEP proc—

esses and more global task statements. For example , REALTRAIN data

from the combined arms test at Fort Carson are being analyzed to estab—

lish their connection with ICRTEP-type task statements. What has evolved

to date is an experimental ARTEP format (Mirabella , 1977; Sulzen &

Root , 1976) including mission-derived tasks and conditions supported

by process and product measures of performance . This experimental format provides initial attempts to reduce limitations on performance

measures and casualty assessment characteristic of the original ARTEP

model.

Although the modified ARTEP relates objective ES outcome data to

ARTEP tasks and processes , the standards problem has not been addressed

except in a norm-referenced sense ; for example , what is the performance

level of Team A relative to some earlier point in training, or relative

to Team B? In the current operational system , the ARTEP evaluators

carry around in their heads both the scales of performance and a set

of performance criteria for applying these scales. Current ARTEPs do

not provide an appropriate set of criteria . More often than not , ARTEP

standards are either procedural statements or undefined cut points;

e.g., “casualties shall not be excessive .” Also , the current standards

are not suitable for two—sided interactive behavior . With regard 

to8
_

scales of performance , current ARTEPs provide little if any guidance .

ES operationalizes some of the scales of performance , although it does

not , in and of itself , provide the criteria for interpreting the performance data; a new model feature is required .

Expected Values Subsystem

The expected values subsystem of the COTEAM evaluation system is

designed to provide performance criteria , standards , or benchmarks

against which ES field exercise outcomes can be compared. One objective of the subsystem is to provide a set of systematic methods for defining specific behavioral and attrition expected values for ES exer—

cises based on values of external variables in a specific exercise and

the assumption of combat-ready opposing forces. The evaluation staff

can use these expected values in making evaluation decisions regarding

unit performance and in training diagnosis.

To systematize and standar- ize the evaluation of unit performance

in ES exercises , performance beuchzuarks or standards must be defined.

The dynamics of two—sided field exercises do not permit exact , deterministic standards , however . The standards must be in the form of

probability distributions , tolerance limi ts , or principle—derived sets

of correct solutions to tactical problems. To establish one or more

of these types of standards , large amounts of ES outcome data must be

obtained so that the characteristics of the distributions of ES outcomes

can be determined . Because replication of field exercises is difficult

and costly , the distribution of ES outcomes cannot be generated in field

exercises. As part of the COTEAM research program , therefore , inexpensive simulations of ES exercises are being developed to provide large

amounts of valid ES outcome data.

The concept of situation-specific forecasting is being used to

develop an ES outcome data base from which the characteristics of the

distributions of ES outcomes can be determined . situation-specific

(Root et al., 1978) means that the forecasting procedures replicate

the particular field exercise conditions as closely as possible; a

forecasting exercise uses the same force ratios, weapons mix , terrain,

weather conditions , and missions as the corresponding field exercise .

Given a valid data base of ES outcomes, expectations about tactical

processes and casualties can be derived for units participating in ES

field exercises. The methods being developed or adapted for generating

the data base include (a) military experts ’ DELPHI, (b) board war

games , and Cc) computerized ES models.

DELPHI

The DELPHI technique is designed to exploit and enhance experts’

judgments in analysis, evaluation , and forecasting. In its simplest

form , the DELPHI technique is a carefully designed series of individual

9

- ~~~

j ---——--— ~---.-. I ~~~~~~~~~~~~~~~~~~~~~~ - —~~-- 
r

interrogations (usually written quest ionnaires) coupled with in formation and opinion feedback.

For example , suppose a panel of military experts is convened to

estimate the number of casualties a small combat unit would sustain

during a specific mission . Each panel u~ember responds individually

with an initial estimate. The person in charge of the panel collects

the data and computes the median and the interquartile range (the interval containing the middle 50% of the responses). Each panelist receives

the results and is asked to make a new estimate . In addit ion , if the

panelists ’ estimates are outside the interquartile range , panelists are

asked to indicate why their responses were different from the majority

jud gment . Thus , panelists holding extreme views are forced either to

defend their positions with explicit reasons or to move toward the

majority opinion . Results of this second iteration are tabulated and

given to the panelists along with a summary of reasons presen ted by

those who hold extreme positions. This process continues for four or

five rounds of voting . The median of the final set of responses is

then assumed to approximate the group judgment , and the range of responses may be presented , representing an ordered , weighted series of

jud gments , i.e., a set of answers with associated priorities .

Pi variation of the DELPHI technique that may have particular use

in COTEAM combines DELPHI with scaled or comparative judgments across

a set of stimuli. (See Wheaton and Mirabella,. 1972 , for an ilbustra—

tive application in training effectivencss research.) In this case ,

empirical data on-reference training units might be used as anchors

for judgments of other assumed training levels.

The DELPHI technique typically leads to a convergence of opinion.

Other benef its gained from using DELPHI are that assumptions are made

explicit , a broad range of alternatives is produced , and judges are

sensitized to the complexities and interactions that influence alter—

nat ive outcomes. The success of the DELPHI technique , however , depends

on several factors——the “exper tn ess” of the panelists , the type and 

amount of feedback prov ided , and the quality of the questionnaires

used .

A pilot study to determine whether military personnel could make

the judgments required for the DELPHI procedure was conducted in 1977

at Fort Ord , Calif., as part of a REALTRAIN rifle squad validation

test (Mirabella , 1977). Exercise participants were asked how they

would expect infantry squads to perform on a number of variables pertinent to a movement to contact mission , assCiming four levels of training : BC’F; Level 2 of infant- rv \RTEP ; Level 2 , plu s 3 or more day s of

SCOPES training; and combat ~

.perienced . Forecasts were made for a

set of process and product variables. Results suggested that military

personnel can describe differences in expected performance as i function

of hypothesized level of training and that expected values form a consistent ordinal scale across forecasters.

I0

-

~~~~~

- ~- ~~~~~~ 

- ~~~~M t - I
F - -- - - - - ——-

~~~~~~~~~

- -.——-

~~

—_

——-. .

-—-- . .

~

- 

1

Board uames

A second method of producing accurate forecasts was explored during

the armor Combined Arms Test (CATEST) at Fort Carson , Cob ., from January to April 1978 (Medlin , l979a). Using scenarios identical to those

used during the actual field exerci~.es ( i .e., force mix , OPORD , weather ,

terrain, etc. ), data were collected using the Fort Carson Forecasting

Game , a board war game . The q ame w~~ ~

l veleped sp ecit ~ically tor the

l~ort Carson exercises. A 1:3,125 pictom ap of the exercise lanes served

as the qame board . A hexagonal qrid overlay , wi th each hex corresponding to 50 m of terrain, was placed over the map to help standardize

movement rates , detection distances , and range of weapon effectiveness .

In the two—player version of the game, each player maneuvers hi s forces

on a separate board , out of sight of the other player . Play is controlled by a single controller , who insures observance of the rules ,

keeps exercise time movinq , delivers indirect fire , provides detections ,

and assesses casualties. Although the controller may seem to have considerable power , most of his functions are clearly and precisely expli—

cated in the rules. The only subjective decision involves detections ;

if the controller is unbiased and has had some exper ience with the actual

exercise terrain , hi s decisions on detections can be fair , accurate , and

easy to make.

Resul ts of the study indicate a few small differences between forecast and field exercise data ; in genera l , hcweve r , the data from the two

different sources are quite comparable . Only two differences of any

magnitude were encountered——the greater casualty-inflicting ability of

the tank in the forecasting exercises and the greater lethality of artillery in the field exercises. These discrepancies do not seem to be

related to the use of different skills in the field and on the game

board . Rather, they seem to be due mainly to slight differences in the

way the ES casualty rules were enforced in the field and on the game

board .

A second study was conducted to evaluate the validity of board war

gaming as a forecasting technique to determine performance benchmarks

(Medlin , 1979b). Using data from the Combined Arms Test at Fort Carson

for March 1978 and from the Fort Carson Forecasting Game , military

judges were asked to distinguish between field exercise data and forecast data. Judgments were made on maps of exercise maneuver routes and

tables summarizing data on the casualties suffered and the weapon system which inflicted each casualty. The military judges were not able

to distinguish between field and forecast maneuver routes , and they

tended to classify forecast casualty data as field exercise casualty

data. Game board exercises , therefore , generate data similar enough

to field exercise data for use in determining characteristics of the

distribution of ES outcomes. In turn , these data can be used to define

performance benchmarks or standards for units participating in ES field

exercises.

11
Computer Simulations

Ultima tely, a computer simulation can be used to generate ES outcome data . Input parameters can be varied to assess the effects of

external conditions and/or unit characteristics on field exercises.

To develop the algorithms necessary for a computer simulation of ES

exercises , the nature of the exercises , the processes involved in reaching the final outcomes, and the relationships among the variables affecting the e,cercises must be understood much better than at present.

An exploratory effort at building a cámputer simulation to generate a

computerized data base is now in progress. The simulation is designed

to generate hypothetical casualty effects~~ s--a function—of tr~ Th±ng

level.

Suimsary of Expected Value Generation

I

Md of the above methods for expectation generation may be thought

of as combat simula tions , or theoretical models of combat . The methods

must be situation—specific , replicating the field exercise battlefield

as closely as possible. Convergent and empirical validity are establ ished for the models if the expectations generated by them agree with

each other and with the observed values from the field validation exercises, using units at assumed levels of proficiency (for example ,

“combat-ready”).

If the expectations and observations do not agree, the expectation

generation methodology is not necessarily defic,~ent . One or both of

the field exercise units may not be at the assumed level of training .

This confounding of possible explanations can be minimized by using the

leaders of the field exercise units to generate expectations about their

own field exercises; that is, by having the same leaders forecast outcome s and conduct identical field exercises in which outcomes are

recorded .

Because of the probabilistic relationship between behavior and attrition , and because of measurement error , discrepancies between expected

and observed values for these two major classes of combat dependent variables will not always be consistent across replications of the same

evaluation exercise . However , large divergence from expected values

for either class of variables should warrant additional substantiating

analyses, a reassessment of the expected—values generation methodology ,

or additional replications of the evaluation exercise . The value of

the foregoing approach is not that it immediately resolves ambiguities

associated with evaluating combat units ; rather , the value of the approach is that it provides a systematic and scientific framework linking combat theory and combat unit evaluation. With repeated application ,

the framework will help resolve ambiguities associated with both combat

theory, includ ing doctrine , and combat. unit evaluation .

12 

- - —-- ~~ —.- —. - ~~~~~~~~~~~~ - 

~~

.-

~ --—-.-—~~ .-.-.~~~-.-
Comparison Subsystem for Training Diagnosis

In addition to the diaqnostics listed in the modified ARTEP and

the ARTEP evaluators ’ P01, the COTEAM system introduces new techniques

for assessinq training deficiencies and proficiencies. The approach

follows the research methodology for model testing and building outlined by Rapoport ( 1975) . After method s for generating expected values

have been val idated , the expectations are used as “ ideal , ’ optima~~ or

baseline performance , and discrepancies between observed and expectei~ performance are explored . It is assumed that discrepancies will be

neithe r so barge nor so unsystematic as to make the compaiison meaningless or useless. If system atic discrepancies ire found , they may be

interpreted in terms of unit characteristics , external conditions, doe—

trifle , or serendi pitous; events. If proper cont rol is exercised , the

interpretations may rest solely on unit characteristics and , in particular , on the traininq leve l of the unit. If the unit performs better

than expected on some task , then it is proficient in that area . If the

unit performs worse than expected , it has a deficie ncy t hat can be addressed in subsequent t r iin i iiq .

In the initial expect at ion generation process , it is assumed that

both units are combat-ready. This assumption makes it possible to obt am an “ideal’ or optimal I-.erfornun c . reference point . For example ,

suppose that a reliable discrepancy has i-e ’ -n observed between the expected value and the observed value for att iit ion in a two-sided (Blue

versus Red) , free-play simulated combat exercise . What inferences

about the training leve l of the Blue and Red forces can be made? Is

Blue attrition lower than expected because Blue tiaining is suprastan—

dard or because Red training is substandard or some combination of the

two?

To make inferences about the training level of both Blue and Red

based on relative attrition , some reference points or baselines are

needed. A minimum set of reference points are expected values for attrition under the following conditions: (a) Blue trained but Red not

trained , (b) Red trained but Blue not trained , and (c) neither Bl ue

nor Red trained . These additional expected values for attrition under

these three conditions provide anchor points on a scale that can be

used to interpret observed values of attrition for the Blue and Red

forces and to relate observed attrition to training level. Additional

reference points may be generated by representing assumed Blue and Red

training levels with other than extreme parameter values. A similar

set of reference points may be generated for quantifiable behavioral

dimensions.

Judging unit performance on the basis of a comparison between

what is expected of a unit and what it actually does is an attractively

simp le idea. The comparison provides the framework needed to evaluate

units in a context broader than a specific exercise or series of exercises , yet it can be made sensitive to the particular conditions under

wh ich a unit is observed . Depending on the purpose of a particular

13

— — .- . —j -——- —.-

~.*—-..~~

—~~~~~~~_ — ,—~~~~—-- - - —— .—.---———-—- ‘— --—~~ .

~~ .__ _~~~~ ___. _. __ I ~~~~~~ ~~~~~~~~~~~~~~~
evabuation exercise , the comparison process can address specific training diagnostic questions, provide data for training management comparable to grade-equivalent scores , answer doctrinal questions , or

indicate readiness for combat .

SUMMAR Y

The Combat Operations Training Effectiveness Analy sis Model (

~~~ EAM )

is an evaluation system for combat unit performance that systematically

addresses the weaknesses inherent in the current ARTEP . ARI developed

the mode l and is conducting research on all aspects of the system. Using

the ARTEP as a starting point , the L~

’OTEP

~

M model proposes a modified ARTEP

manual that includes task/subtask diagnostics that facilitate the evaluation and training feedback processes and an ev~

iluator program ~t inStruc- 3

tion (P01) that teaches evaluators roles , duties , and how t

~~ conduct a

field exercise . Furthermore , the task ’subtasks listed in the manual are

being scientifically and systematically analyzed to determine the critical events that must occur for a unit to perform its mission .

The rest of the CJrEAM evaluation system is driven by the im~

x 1ified

ARTEP manual. The evatuation environment is defined by the modified

ARTEP manual (mission , posture , personnel , weapon s, etc .) and the field

conditions (weather , terrain , time of day , etc.). The unit to be evaluated enters this environment with certain unit characteristics (individual skills , unit training level , leadersh ip , etc.). A realistic combat environment should be provided to assess the combat unit’s performance on the A.RTEP tasks/subtasks. Engagement simulation (ES) is a

two—sided , free—play, tactical maneuver/field exercise that allows immediate and realistic casualty assessment. ES fie ld exercises provide

a realistic combat environment in which combat units can be evaluated .

(Although a high-fidelity combat environment maximizes the usefulness

of COTEAM , the model can be applied where high-fidelity technology is

not available if suitable methods are used to generate expectations.)

The expected values subsystem of COTEAM is a means of ev aluating

performance in tactical field exercises. Given the external conditions

as defined by the ARTEP manual , the field environment , and the ES ex—

ercise rules , expected values or expectations are generated for the

process and product outcomes of the field exercises. These expected

values provide a baseline , benchmark , or standard with which observed

unit performance can be compa r ed . Considerable research has been directed at the expected values subsystem of the COTEAM model. The DELPHI

technique , board war gaming , and computer simulations are being deve loped as methods of generating expectations. Although the feasibility

~f the DELPHI technique and computer simulations is sti ll being explored ,

board war gaming has been es tab li shed as a means of generating expected

values for process and product outcomes extremely similar to those observed in field exercises.

14

‘-

~
A comparison of the expected and observed outcomes facilitates

evaluation of unit performance and decisions concerning training level

and/or com bat readiness. This compar ison process , in conjunction with

the ARTEP manual diagnostics and the evaluator training provided in

the P01, provides information for feedback to the unit , particularly

feedback concern in g training deficiencies and proficiencies. This feedback can be used to structure the unit ’s training programs.

The COTEAM evaluation system is one of the models being developed

as part of ARI ’s research on unit evaluation . The system introduces a

realistic simulated combat environment in which units perform tactical

opera tions and in which objec tive da ta can be obtained , specific procedures for determining standards against which unit performance can be

compared , and techniques for assessing training level , combat readiness ,

and training deficiencies c~ n be refined . Research is being conducted

on all aspects of the model , and modifications to COTEAM will incorporate the results of these efforts. The ultimate goal of this research

is to develop a criterion—referenced system for evaluating unit tactical

performance .

15
REFER~~CES

Banks, J . H., Har dy , G. D., Scott, T. D., Kress , G., & Word , L. E.

REALTRAIN validation for rifle squads : mission accomplishment.

ARI Research Report 1192, October 1977 .

Epstein , K. I. Automated tactical operations measurement system—-ATOMS .

Paper presented at the Military Operations Research Society 41st

Symposium, July 1978 .

Epstein , K. I., & Johnson , E., III. Team performance assessment. Paper

presented at the Amer ican Educational Research Association , San

Francisco , Calif., 1976 .

Erwin, D. E. Evaluative criteria for comparing models of unit performance . Paper presented at the Military Testing Association , Orlando , Fla., 1976.

Havron , D., McCullough , T. J., McFarling, L. H., & Wanschur a , R. G.

Improved Army training and evaluation program (ARTEP) methods for

unit evaluation . Volume IV: Guidance for planning and conduct of

company—level field exercises. ARI Research Product RP79—7 ,

April 1979.

Havron, M. D., McFarling, L. H., Hill ,- H., & Wanschura , R. G. Improved Army training and evaluation program (ARTEP) methods for

unit evaluation . Volume V: Analysis of alternative training

settings. ARI Technical Report TR-79-23 , April 1979.

Havron , M. D., & Wanschura , R. G. In~

proved Army training and evaluation program (ARTEP ) methods for unit evaluation. Volume VII:

Executive summary. ARI Technical Report TR-79-A25, April 1979.

Medlin , S. H. Behavioral forecasting for REA.LTRAIN combined arms.

ARE Technical Paper 365 . May 1979. (a)

Medlin , S. M . A partial validation of forecast engagement simulation

exercise outcomes. ARI Technical Paper 382. 1979. (b)

Meliza , L. L., Scott , T. 0., & Epstein , K. I. REALTRAIN validation for

rifle squads II: tactical performance . ARI Research Report 1203,

March 1979.

Mirabella , A. Criterion-referenced system approach to evaluation of

combat units. Paper presented at the 19th Military Training Association, San Antonio , Tex., October 1977 .

• Rapoport , A. Research paradigms for studying dynamic decision behavior.

In D. Wendt & C. Viek (eds.), Utility, probability, and human

decision making . Dorducht , Holland : 0. Reidel , 1975.

7 -

~~~ 

— — -—-,- --— -

~~~~~~

. 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -~~~-

1 

~~~~~~~ ~~~~~~~~~~ ~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~

— --‘ .

- ~~- — -

~ — t ~~~~~~~~~~ ~~~~~~~~~~~~~~~~ —U--
Hoot , H. T., Knerr , C. S., Severino , A. A., & Word , L. E. Tactical

engagement simulation : a method for learning the re~ lities of

combat. Paper presented at the American Educational Research

Association , Toronto , Canada , 1978.

Scott , T. 0., Banks , J. H., Hardy , G . D., & Sulzen, R . H. REALTRAIN

validation for rifle squads Ill: tactical performance during

movement-to-contact. ARI Research Report 1213 , July 1979.

Scott , T. 0., Mel iza , L. L., Hardy, G. D., Banks, .3. H ., & Word , L. E.

REALTRA 1N validation for armor/anti-armor teams . ARI Research

Report 1204, March 1979. r

Sulzen , h. H. Combat actions: a process focus for an engagement

simulation ARTEP. Undated draft report .

Sul,,-t: , H. H., & Bleda , P. R. Effects of combat simulation on the

work-related motivation/satisfaction of participants . ARI Tech—

ni ii Paper 351 , March 1979. - 

-

Suizen , R. H., & Root, R. T. Development of an evaluation model and

training program for the multiple integrated laser engagement

simulation (MILES) : Phase II , ARI draf t report , 1976.

Suizen , R. H. , Root, R. T., & Epstein , K. I. Development of an evalu—

ation model and training prograxft for the multiple integrated laser

engagement simulation (MILES) : Phase I. ARI Research Problem

Review 76—7 , Octobe r 1976.

Wheaton , G ., & Mirabella , A. Effects of task index variations on

training effectiveness criteria. N ava l Tra ining Equipment Center

Technical Report 71—C—0059—l , October 1972, Contract No. N61339—

7l—C—0059. -

18

• t—•~ — -—--.~ —--—-.-—---• .-.•~ 

~— - • • ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ~~~~~~~~~~
DISTRIBUTION

AHI O,~ iliut,oii I. ist

4 OASO (M& HA I 7 HOUSACOEC , ti Ord, ATTN: Library

/ -IODA l[)AMI CS/I 1 )IOUSACDEC , Ft Ord, ATTN; ATEC - IX -E Hum Factors

IIIIDA IDAPE PBR) 2 USAEE C . Ft Benja min Harrison , A TTN Liliia iy

~~~~~ ~

o~~

v r

~ ~~~ I USAPACDC , Ft Beiij amur Harrison ATTN A’ICP HR

1 HOIIA IOAP~ HAl II)) I USA Comm-E lect 5th, Ft Moninouth , ATTN ATSN IA

I HOOA ISCRO ID) 1 USAEC , Ft Monmo uth , ATTN AMSEL CT HOP

I HUDA )IJAM I DOT C) 1 USAEC Ft Monmout h , ATTN~ AMSEL -PA P

I HODA (DAPC PM/ A) 1 USAIC, Ft Monmout in , ATTN: AMSEL SI—CS

I l-IOOA IDACII PP.’ A) 1 USAEC , Ft Monmouth , ATTN; C, Fact Des Br

I HODA IDAPE I-IRE) 1 USA Materials Sys Anal Agcy, Aberdeen , ATTN AMXSY P

1 HODA IDAPE MPO C) 1 Edgewood Ars enal . Aberdeen , ATTN : SAREA BL. H

I HODA )OAI’ [ OW) 1 USA Ond Ct, & Sch, Aberdeen ATTN: ATSL-TEM - C

1 HODA (DAPE HA L) 2 USA Hum Engr Lab . Aberdeen , ATTN . Library/Die

I HODA (DAPE CI’S) 1 USA Combat Arms Tng Sd, Ft Benrtt ng, ATTN. Ad Supervisor

1 HODA IDAFO MFA) 1 USA Infantry Hum Rsch Unit . Ft Benning, ATTN: Chief

1 F-IOOA (DARO ARS- P) 1 USA Infantr y Sd, Ft Bennir ig, ATTN: STEBC - TI - T

I HODA (DAPC PAS A) 1 USASMA . Ft Bliss , ATTN: ATSS - -LRC

I HODA IDUSA 0)11 1 USA Air Def Sch Ft Bliss ATTN: ATSA CTD ME

I I-IUDA IOAM() RQF1 ) 1 USA Air Oct Sch, Ft Bliss , AT TN- Tech Lib

I IlOI)ts )IIA’ ,C) I USA Air Del Bd • Ft Bliss , ATTN: FILES

I HO[)A IDA IO-PI) I USA Air Del Sd . Ft BlIss , ATTN STEBD-PO

I (lI~~t Con,s,,II Div (DA-OTSG) . Anlellihi . MD 1 USA Cmd & General StI College , Ft Leavenwor th , ATTN~ Liti

1 Md Ass t. Hugo Res , O1)OIt&E . OAO IE&LS) 1 USA Cmd & General 55$ College . Ft Leavenworth , ATTN: ATSW—SE-L

1 lID IISARAL AP O Sc ’a t t li ’. ATTN AAAGP it I USA Cmcl & General Stf College , Ft Leav enwort h , AT TN: Ed Advisor

1 HO First Ar rri y, ATIN: AFK A - OI TI 1 LISA Ciimlilned Ariri s Cmbt 0ev Act , Ft Leavi ’ i iwo rr h . ArTN: DepCd,

2 HO Fifttr Army. Ft Sam Houst on 1 USA Coenbinsd Arms Cmbt Des Act. Ft Leavenworth . AT’T’N: CCS

1 Dir . Army SIt Studies Otc . ATTN : OAvCSA loSt’ ) I USA Combin.d Arms Cn*A Des Act . Ft Leavenworth, ATTN: ATCASA

I DI,- Chniel ,,l SrI St ud ies Ofc 1 USA 1. iiiII,nvd Arm s Crnbt Dew Act . Ft Leavenworth , ATIN: ATCACO --E

I l)CSI’ER ATTN CI’S/OCI’ I USA Coin li ini ’d Ar m s Cmbt Des Act , Ft Leavenw orth . ATIN: ATCAC C—&:I

I the Army Lil, . Peritamtor i. ATTN ASH ChiIcl • I USAFCOM . Night Visio n Lab , Ft Belvoir , ATTN: AMSEL—NV—SO

1 The Army Li b Perrtaqoi i A TTN ANRAL 3 USA Computer Spy Cmd , Ft Belv oir , ATTN : Tech Library

1 Oh .. A~~

t Sect ol the Army lR&Dl 1 USAMERDC . Ft Belvoir . ATIN: STSFB-’DO

I Tech Support Ofc . OJCS I USA E rng Sch , Ft Belvoir , ATTN Library -

I USASA . Ar l ir,,t to n , ATT N IARO-T 1 USA Topo graphic Lab . Ft Belvoi r , ATTN: ETL ID—S

1 USA AscIi Ole , Durham . AT TN Life Scie nces Dir 1 USA Topograp hic Lab , Ft Belvoir , ATIN: STINFO Cerrter

7 USARIEM , Nati rk , ATTN SGRD uE CA 1 USA Topographic Lab . Ft Balvoir , ATTN: ETL GSL

I IISAT rc It CI.ip io n ’ . Al tF’l ~

; tt fl : Ml) A I lISA liiti ’ll iijv ne.’ Ctr & Sct i. Ft Hii.ic hrmn j , ATTN : CTD MS

I USAIMA . Ft Bragg, At UN- AISU-ClO -OM 1 USA Intelligence Cii & Sch , Ft Huachuca , ATTN: ATS—CTD—MS

1 USAIMA , Ft Bragg, ATTN: Marquat Lib 1 USA Intelligence Ct r & Sch, Ft Huachuca , ATTN: ATS I—TE

1 US WAC Ctr & Sch , Ft McClellan , ATTN : Lii , 1 USA Intelligence Ctr & Sch, Ft Huachuca , ATTN: ATSI—T EX—GS

1 US WAG Ctr & 5th, Ft McClellan , ATTN Trig Di~ 1 USA Intelligence Ctr & Sch , Ft Huachuca . ATTN: ATSI—CTS—O R

1 USA Quartermaster ScPr , Ft Lee , ATTN: ATSM TE 1 USA Intellig ence Ct, & 5th , Ft Huachuc a . ATTN: ATSI—CID--DI

1 Intellig ence Material Dcv Ofc , EWL , Ft Holabird 1 USA Intellig ence Ctr & Sch , Ft Huachuc*, ATTN: A1’SI—CTD—GS

I USA SE Signal Sch . Ft Gordon , ATTN : ATSO EA 1 USA Intellig ence Ct r & Sch, Ft Huachuca . ATTN : I)AS/S RI)

I USA Chaplain Ct, & Si’h. Ft Hamilton , ATTN; ATSC•TE RIT I USA Intellig ence Ctr & Sets , Ft Huachuca . ATTN: ATSI—TEM

1 USATSCH , Ft Eustis, ATTN . Educ Advisor 1 USA Intellig ence Cs , & Sch, Ft Huachuca , ATTN; Libra ry

1 USA War College . Carl isle Barracks , ATT N: Lb I CDR . HO Ft Huachuca . ATTN: Tech Ref Div

2 WRAIR , N~iiri ,psych iat ry Div 2 COR , USA Electronic Prvg Grd , ATTN STEEP MT—S

I DLI , SDA . Mo nterey 1 HO, TCATA , ATT N: Teds Library

1 USA Concept Anal Agcy . Bethesda, ATTN: MOCA-MR 1 HO. TCATA . ATTN: AICAT.OP’O, Ft Hood

1 USA Concept Anal Agcy , Bethesda , ATTN. MOCA-JF I USA Recruiti r

~ Cmd , Ft Sheridan . An N: USARCPM.P

1 USA Arctic Teti Ctr , APO Seattle , ATTN: STEAC-PL.MI - I Senior Army Mv. , USAFAGODFTAC , Elgi n AF Arms FId No 9

1 USA Arctic Test Ctr . APO Seattle, A T1’N: AMSTE.PL.1’S 1 HO, USARPAC , DCSPER . APOSF 96558, ATTN: GPPE-SE

1 USA Armament Cmd . Redtton s Arsenal , A’fl N: ATSK .TEM 1 Stim son Lib , ~~ademy of Health Sc iences , Ft Sam Houston

1 USA Armament Crnd , Rock Island . ATT N AMSA R.TDC I Marine Corps Inst .. ATTN: Oean.MCI

• 1 FAA.NAFEC , Atlantic City . ATTN: Library 1 HO. USMC. Commandant. AT1’N: Code MTMT

1 FAA NAFEC . Atlantic City , ATTN: Human Engr Br 1 HO. USMC, Commandant. ATTh : Code MPI-20’ 2S

-1 FAA Aeronautical Ctr , Oklahoma City, ATTN: AAC-44 0 2 USCG Academy. New London. ATTN: Admission

2 USA Fid Arty Scfs . Ft Sill , ATTN: Library 2 USCG Acade my. New London . ATTN : Library

• I USA Armor Sets , Ft Knox , ATTN: Libra ry 1 USCG Training Ctr. NV , ATTN: CO

1 (ISA Armor Sets . Ft Knos , ATTN: ATSB.DI F I USCG Training Ctr . NV . ATTN: Ecluc Sec Ole

I USA Armor Sets , Ft Knos , ATTN: ATSB DI TP 1 USCG. Psyctr ol Res Br , DC, ATTN GP 1/62

1 USA Armoe Sch, Ft Knox . ATYN: ATSB -CO-AD 1 HO Mid.Range Br , MC Oat . Quantico. ATTN : P&S Div

19

• , - .IU.~

_ .g - ~~~~~~

L _,,. ..,.

~

,a. ~~~~~~ .iS .a.. &. ‘

.. .... ,, ... — • In ..a ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —_ - I
~ ~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~-- - -

I US Min i’,’ I ,,m ps I. iJ,unii I )I, - AtniIC. Alevanm.Ir a, A T UN AMC&,5 i I l)r I h

~ U, I b nist 01 1 m w,, g i M.diy’,mie ( amn wl,m

I [ISAI l ’I AI ) t I ( I n 55 ,,,, ,,’ AJ’IN A tAD It) I AIR t RESS . Kensingt~

nii ATTN l,,t,i Spy Ar

i LISAIRAOO( . I M,,m ir, m,

’ - Al IN AIPH At) I Md it iwrpsykm i logi sk 1Ipn.sln ’ , Ci.p.nn hamp n

I USATF1ALX )C Ft M i,,,. AT TN AIlS F A I Military Attache . F npr,54h F mhasyy ATTN Dow Sew

1 USA I d i l ‘s t ’,rn,l Ft M~ Plmeis,gi, All N I nl ,n,mn I Mn’dr~

, iii (.I’n.t I’ U RI’ A Aiierwal, T,,,,l,n,i N.,val F

1 I iSA Avi.,t’,m’i tes t It’l. F I Amy .- All N 1; IF H(

~ P0 1 Pn io I,i

~eor ‘I i i - (lit Atitnl Hi,,o U sgs Ru In I),r M,nr mst ,

I (ISA A~~g l,

~ Av,aln,,in i-im P- iy I i Hivkr ’n - Al 1 N t , Is~ m ,

~ ‘‘I I),’lenmss . New t)ellii

I USA Agcy f u n Asma tmon Sahety, It RI im k,, Al IN Educ Advisor I P~~

5 Hwh Ole L Iver y AKA . Israe l ().l,nw l,i,,,’s

I USA Avijt,,i,, 5th. ti Humi ’kr ’n . ATT N P0 Drawer ~ I M,n,ister,w warm D.$,’rnymp 000P/KE 41,1 Sum ‘a.. I

I HOUSA Aw’ ,nr “I Sym CinwI , St LoIm’i ATTN AMSAV 20) 1 I’,

~ , li,ili,giw Ii.’ .‘

ak ,’u, Th. Hague N.ttsemharsnts

• / (ISA Av, ,,i.,ii, 5,’, lu’s) Art - I dw,un’is Al l), Al TN SAV It

I I.l’wA Am, Dcl ‘,, Ii I- 1 1111,- Al T N A TSA TIM

I II’

,A I’

pi M.rIwI,Is Rs,li (I, h r ’s Lii’ . IrI,,Ibi’ll Fti I , Al IN: :,AV lil Ai-’

I I T A Av,.,n,g m,, St-tn Hi’ s lirq MgI . El Himckr’r , ATTN A 1’ST I RIM

I USA Avnat,,i,, 5,-i, (N,) It Hucker ATTN AIST - D- A

$ Ill) I;AFt c:(lM. Al,naiimlnla . AT UN AMXCI) IL

I ilL) DARCI 1W Alenandrna . Al TN CUR

1 US M’lntary Academy, West Pouun r. ATTN Serials I,Inut

I lb S Militaig Academy. West Pimirr n ATTN Ole ol M,ll Lilt slip

I (‘5 Mnlmte’~ A, ,i Im’iru’, W.’sI Point ATtN MAOI(

I USA SIsnml.,n,li,’at iwsn tip, UK FP() NV . ATT N MASt - DC

I Oh,- of Naval Rsi’fn . Ai Ii,u,jt im,i ATT N Comic 452

3 111,-al Naval AscIi ,

~

,I,,,,j r,,,, ATT N Code 4511

I (It, of Naval Rsm:h . An I, ingtoni , ATTN : Cmm,le 450

• 1 ill, if Naw, ml AscIi, Arliumm 1t ium n , ATTN C,,mlnr 441

1 N.mv,,l Ar ‘‘“I” Mr’ut Res I m l, Pensam il, ATTN Ac uirms Set, Div

i N,uv.,l A. ii ’SI’, Mu’, I Il,’s I il’ t’ennsj rm,Ij , A TIN C ii, ’ L 51

I N,mv.mI Aan, nst’n M,’,l Hr’s LaIr , P~m- ,sac,nl,i A TTN Coile 15

I CII it N, mv h ’ ,-,s . A TTN Pets OR

1 NAVA I R S SA Nim b lE , ATT N: Safety Ctr

I Njv llu’eainu,i, ,gmhmc . DC . ATT N Code 6751 , Charts & Tech

1 (- 1cm of Naval A,nal , All N- Dm5,’ Ct ,

I N.msAir SysConrs , ATT N AIR 53131’

I N.m BuMed , ATT N 11 3

I NavHe llcoptenSubSqua 7, l’PO SF 96601

• I AFHRI. (Fl) Willi ams AFB

I AFHRI. ITTI Lowr y AFB

1 AFHRL IASI WPAFB . OH

2 AFHRL (00)/I Brooks AFB

1 AT HAL IOOJN) Ljckl and Ale

I HQUSAF IINVSDI

I HOUSAF (DPXXA I

1 AFV TG IRDI Randolph AFB

3 AMR1~~HEIWPAFB , OH

7 At Int l of T i -I’ WPAFR . OH, ATTN: ENE/SL

1 AIC IXPTD I Randol ph Al IT

I (ISAF Acr’,Mmt L , l n , Brooks AIB ISUL 41, ATTN : DOC SEC

I Al’ OSR (NI I , An Irr s q to n r

I AF log CmmI , Mr’CleII ,imn A l ’S, ATTN AIC/DPCRB

1 Am n Fm,rc. Academy , CO. ATTN: Dept of Sal Sen

Ii Nav Pers & Des Ct, , Sari DIego

2 Navy Mail Nrrunopsychmatric AscIi Unml , San Diego

I N.m Electron ,,’ LaIn , San Diego , ATTN. Res lab

I Nay Tr ,mqC i’n , San Diego , ATTN Coil e 9000’ Lili

1 NavPostG raSch , M mr ntm’n r’y, ATTN : Code 5BAa

I NavP ur,tC,naSch . Monter ey. ATTN : Code 7124

I Nav trr ng Equm lnCtr , Orlando , ATT N Ted , m b

I US Dept aI labor . DC, ATTN Manpower Admmm n

I US Dept of Justice , DC. AITN: Drug Enlorc e Admin

I Nat Bti ’ ol Standard s , DC, ATTN : Computer Into Section

1 Nat Cleaemn q Hot,se for MR Info , Rock vi lle

I Denver Federal Ct, , lak,wood, ATTN: BLM

• 17 D,I.ni, Docum entation Center

4 Dir Psych , Army Hq. Russell Ofcs , Canberra

1 SciC nlil ic Advsr. Mul Sd , Army Hg. Russell Ofcs , Canberra

1 Md md Air Attache , Austri an Embassy

I Crnrlie de Rechm,rcbe Des Factaues , Humaine dv Ii, Oetenst.

Nat mnm rs alq . Br,icscl ~ 7 Cari,sd ian Joint Statt Was hrrrgto n

I C/Arr Staff , Royal Canaml nar s AF , ATTN Pert Slit Anal Br

3 Cbn net , Camnadi an Del Rsc pm St ill ATTN : CICROS(W)

4 Bi,t ,s h Del St aff Ilnilnsbs [mbassy . Was hm rsgt on 

~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -