Technical Report 502

- A METHODOLOGY FOR ESTIMATING RELATIVE

COST-BENEFITS OF ALTERNATIVE

PRETESTING PROCEDURES

Jack H. Hiller

ARI FIELD UNIT AT PRESIDIO OF MONTEREY, CALIFORNIA

DTI 'ELEC TE

STE

F

LL U. S. Army

Research Institute for the Behavioral and Social Sciences

November 1980

Approved for public relese, distribution unlimited.

82 C 2 056)
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

L. NEALE COSBY JOSEPH ZEIDNER Colonel, IN Technical Director Commander

NOTICES

DISTRIBUTION: Primary distribution of this report has been made by ARI. Please address correspondence concerning distribution of reports to: U.S. Army Research Institute for the Behavioral and Social Sciences, ATTN: PERI-TST, 5001 EIsenhower Avenue, Alexandria, Virginia 22333.

1I:JAL DISPOITION: This report may be destroyed when it is no longer

needed. Please do not return it to the U.S. Army Research Institute for

he Behavioral and Social Sciences.

NOTE: The findings in this report are not to be construed as an offIcl1

Department of the Army position, unless so designated by other authorIzed

documents.
Unclassified

SECURITY CLASSIFICATION OF THIS PAGE (Whin Does Entered)

READ INSTRUCTIONS REPORT DOCUMENTATION PAGE BEFORE COMPLETING FORM

1. REPORT NUMBER 2. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

Technical Report 502 A't+-A\i -77

4. TITLE (and Subiftle) S. TYPE OF REPORT & PERIOD COVERED

A METHODOLOGY FOR ESTIMATING RELATIVE COSTBENEFITS OF ALTERNATIVE PRETESTING PROCEDURES

6. PERFORMING ORG. REPORT NUMBER

7. AUTHOR(o) 8. CONTRACT OR GRANT NUMBER(s)

Jack H. Hiller

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT, PROJECT, TASK

AREA & WORK UNIT NUMBERS

ARI Field Unit

Presidio of Monterey, CA 2Q162722A777

I. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

U.S. Army Research Institute for the Behavioral November 1980

and Social Sciences PERI-IO 13. NUMBER OF PAGES

5001 Eisenhower Avenue, Alexandria, VA 22333 37

14. MONITORING AGENCY NAME & ADDRESS(i different from Controlling Office) IS. SECURITY CLASS. (of thle report)

Unclassified

15.. DECLASSIFICATION/DOWNGRADING

SCHEDULE _-

16. DISTRIBUTION STATEMENT (of this Report)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of the habtract entered in Block 20, If different from Report)

IS. SUPPLEMENTARY NOTES

It. KEY WORDS (Continue on reveree aide If necesary and Identify by block number)

Pretesting Cost-benefit

Cost-effectiveness Instructional system development

Cost-efficiency

'V20. A§! CT (Coat~aea m reverse ebb Nf rneeeemy ad identify by block nue,&er)

"The purpose of the research reported here was to develop a methodology

for measuring the relative benefits of alternative pretesting procedures so

that an optimal procedure may be selected. The research was accomplished as

follows: Alternative pretest procedures were formulated. Variables that affect the amount of time saved or lost by employing pretests were identified

and defined. Algebraic models which take into account measurement accuracy,

pretesting time, and training time were constructed so that the amount of time

(Continued)

DD) W3a ffOITON OF I NOV 6s IS ODSOLETIE D JON Unclassified

SECURITY CLASSIFICATION OF THIS PAGE (Moen Dot Entered)

,- ... - - . ,
unclassified

SEUIYCLASSIFICATION OF THIS PA@St(Whmi Da amteft

Item 20 (Continued)

saved (or lost) by pretesting could be estimated. A limited sample of empirical data was gathered and analyzed by applying these cost-benefit models.

It was shown how these cost-benefit models can be employed to identify the

best procedure in a specified set of competing procedures

Disc

:2

VN!-, GIRA&I )

2TAB f

io ~unced

By--

Distribut iOn/

A vla Iab Iiit y Codes

jAvail SnAor

Dist Special

'I _____________________'211_

Unclassified

SECURITY CLASSIFICATION OP THIS PAOE(ft.,f Data It.,.re
Technical Report 502

A METHODOLOGY FOR ESTIMATING RELATIVE

COST-BENEFITS OF ALTERNATIVE

PRETESTING PROCEDURES

Jack H. Hiller

Submitted by:

James A. Thomas, Chief

ARI FIELD UNIT AT PRESIDIO OF MONTEREY, CALIFORNIA

Approved by:

E. Ralph Dusek, Drco

PERSONNEL AND TRAINING

REERHLABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Off ice, Deputy Chief of Staff for Personnel

Department of the Army

November 1980

Army Project Number Individual Training 1120162722A777 and Evaluation

Approvied forw public ire. istibution unlimrIted.
ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phase of the task, formal recommendations for offidial action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

i1

" i
FOREWORD

The Presidio of Monterey Field Unit has conducted a broad program of

research on problems of training in Army field units since its establishment in 1974. This program has resulted in a number of cost effective

solutions torAty-trainin-gproblems in a number of different areas. The

Unit Training Programs Team concentrates on issues relating to efficient

use of training resources, integration of individual and collective training

and the development and execution of cost effective training programs by

units.

Current training doctrine in units as embodied in the Battalion Training

Management System, directs trainers to conduct proficiency testing before

individual or collective training is executed (i.e., pretesting). However,

pretesting is rarely conducted in the field, and trainers typically challenge

the cost effectiveness of such a procedure as a prerequisite for individual skill training.

This report describes a methodology that was developed for determining

the cost-effectiveness of any form of pretesting procedure. The methodology

was tested through its application to empirical data and found to be workable.

The report describes how this methodology can be employed to improve the

efficient use of time and resources for unit training through pretesting.

The methodology can be easily used by training managers in FORSCOM, USAREUR,

and other organizations responsible for conducting training.

3 SE HiZEI rER

T hni al irector

v
!

A METHODOLOGY FOR ESTIMATING RELATIVE COST-BENEFITS OF ALTERNATIVE . PRETESTING PROCEDURES

EXECUTIVE SUMMARY

Requirement:

To develop a quantitatively based methodology which will enable objective selection of an optimally effective pretesting procedure from a set of

alternative procedures.

Procedure:

The research was accomplished as follows: Alternative pretesting procedures were formulated. Variables that affect the amount of time saved or

lost by employing pretests were identified and defined. Algebraic cost/

benefit models which take into account measurement accuracy, pretesting time,

and training time were constructed sQ that the amount of time saved (or lost)

by pretesting could be estimated. A limited sample of empirical data was

gathered and analyzed to illustrate the possible applications of the cost/

benefit models.

Findings:

It was found possible to construct generalized algebraic models that can

be used to calculate the time saved or lost (i.e., a benefit value) when any

particular pretesting procedure is employed in a group training situation.

By simple comparison of the benefit values for each of the alternative procedures, an optimal procedure can be identified. Analysis of a small sample

of data for infantry tasks implied that considerable improvements in time

utilization may be achieved through use of carefully designed pretesting

procedures.

Utilization of Findings:

This cost/benefit methodology will be used to identify an optimal pretesting procedure for the common tasks required by MOS lB at Skill Level 1.

Based on the logic of the cost/benefit analysis, and the preliminary empirical resuTts, the use of pretesting, where appropriate, is being advocated by

the Army Training Board.

v

j vii
A METHODOLOGY FOR ESTIMATING RELATIVE COST-BENEFITS OF

ALTERNATIVE PRETESTING PROCEDURES

CONTENTS

Page

INTRODUCTION ............... ............................. 1

DESIGN OF ALTERNATIVE PRETESTING PROCEDURES ...... ............. 2

COST-BENEFIT ANALYSIS MODEL FOR SQUAD MEMBER TIME .... .......... 3

Pretesting Variables ............ ....................... 8

Primary Time Saved by Pretesting ........ ................ 9

Pretesting Cost ............ ........................ 9

Adjustment of Pretest Cost .......... ................... 9

Time Lost When a Pretest Fails To Identify a "GO" Soldier . 10

Complete Cost-Benefit Model ......... .................. 10

SPECIFIC COST-BENEFIT MODELS ....... ..................... 11

Equation for Procedure A (Figure 1) ..... ................ 11

Equation for Procedure B (Figure 2) ..... ................ 12

Equation for Procedure C (Figure 3) ..... ................ 12

Equation for Procedure D (Figure 4) ..... ................ 12

Equation for No Pretest ........ ...................... 13

Modification to the Equations for Simultaneous Checkouts ..... . 13

AN EFFICIENT DATA COLLECTION PROCEDURE ..... ................ . 14

ILLUSTRATIVE APPLICATION OF THE COST-BENEFIT MODEL .. .......... . 14

A COST-BENEFIT ANALYSIS FOR SQUAD LEADER (TRAINER) TIME ....... . 20

COMBINED BENEFIT MODELS FOR THE SQUAD LEADER AND HIS MEN ........ . 21

CONCLUSIONS ............ ............................. 25

REFERENCES ............ .............................. 29

DISTRIBUTION ............ ............................. 31

LIST OF TABLES

Table 1. Empirically based data for procedures permitting false

"NO GO" classifications ...... .................. 18

2. Benefit values in minutes for 9 man squads .......... . 19

ix
CONTENTS (Continued)

Page

Table 3. BL benefit values for trainer time in minutes ........ .. 22

4. BWLD benefit values for trainer time in minutes . ...... .. 23

5. Mean and rank order benefit values for all models and

procedures .......... ......................... ... 26

LIST OF FIGURES

Figure 1. Pretesting Procedure A ......... .................. 4

2. Pretesting Procedure B ......... .................. 5

3. Pretesting Procedure C ......... .................. 6

4. Pretesting Procedure D ......... .................. 7

5. Data collected for the Soldier's Manual task--Encode/Decode

KAL 61 ........... .......................... ... 15

6. Results for task--Encode/Decode KAL 61--needed to calculate

the benefit value for Procedure A .... ............. ... 16

7. Results for task--Encode/Decode KAL 61--needed to calculate

the benefit value for Procedure B ...... ............. 17

8. Results for task--Encode/Decode KAL 61--needed to calculate

the benefit value for Procedure C ... ............. ... 17

x
A METHODOLOGY FOR ESTIMATING RELATIVE COST-BENEFITS OF

ALTERNATIVE PRETESTING PROCEDURES

INTRODUCTION

The costs of education and training are one of the largest and most visible aspects of both civilian and military budgets. In the press to economize,

the U.S. Army has been searching for methods which may achieve increased

training efficiency as well as higher standards of effectiveness. One recently adopted feature of Army training doctrine aims at both efficiency and

effectiveness. Specifically, this feature is the requirement to conduct performance-based diagnostic testing before training on a task is given, i.e.,

pretesting.

In theory, pretesting should yield the following advantages:

a. Training time will be effectively gained by eliminating unnecessary

training sessions.

b. Planning and scheduling of training will be facilitated when pretests are used to identify common deficiencies;

c. Testing (and training) standards will be maintained at high levels

when the pretests are performance-based (i.e., actually performing

a task instead of talking or writing about it);

d. Infrequently used skills will be reinforced by the practice gotten

from taking the performance-based pretests;

e. Job climate will be improved by avoiding unnecessary, boring training.

The purpose of the research reported here was to develop a methodology for

estimating the time savings produced by various forms of pretesting so that the

most cost-effective form could be identified.

Pretesting has been incorporated into an on-the-job training system for

infantry currently undergoing research and development by the Army Research

Institute (Bialek et al., 1978). In this experimental system, the rifle squad

leader is designated as the key trainer, and it is his responsibility to conduct the pretests. However, the amount of time that may actually be saved by

having squad leaders conduct pretests is uncertain.

Consider the following hypothetical situation. A squad leader with a

nine-man group intends to have all of his men become proficient in performing

some individual task. It happens that he has insufficient information concerning any of his men's ability to perform the task, so he conducts a pretest. The pretest chosen is a performance test that may be administered to

only one man at a time and it requires about 10 minutes to conduct for each

man. Altogether then, conducting this pretest for the entire squad consumes

1 hours of the trainer's time, and during this period the squad members may

not be spending their time usefully. If it turned out that every squad member failed the pretest and needed substantial training, then no time was saved,

and 1 hours were lost.
Given that pretesting may not necessarily yield time savings and that

different forms of pretesting may also differ in their cost-effectiveness, a

need becomes evident for research to develop a methodology for measuring and

predicting the cost-effectiveness of alternative pretesting procedures (including the procedure which deletes pretesting of any kind, i.e., the null

pretest) so that an optimal procedure may be selected for a given situation.

The research that was performed to meet this need will be described according

to the following stages:

a. Alternative pretesting procedures were designed.

b. Variables which may affect the time saved or lost by employing the

pretesting procedures (including the null pretest) were identified.

c. Cost-benefit models were formulated in terms of training time saved

or lost.

d. An efficient data collection procedure was devised.

e. The cost-benefit models were applied to sample data.

Each of these stages is described below.

DESIGN OF ALTERNATIVE PRETESTING PROCEDURES

The following procedures were considered for possible use as pretests:

a. Each squad member estimates his own task proficiency;

b. The squad leader estimates the task proficiency of his squad's

members;

c. Paper-pencil criterion-referenced tests are used;

d. Computer or equipment intensive simulation is used for testing;

e. Hands-on performance testing is used.

Self-estimates, although of uncertain validity, are fast and easy to acquire. Given the high levels of turbulence found in operational units, the

infrequent occurrence of many infantry tasks, and the fallibility of memory,

estimation by the trainer (i.e., squad leader) was rejected for this research

effort. Paper-pencil tests are relatively quick and easy to administer, but

have uncertain validity for infantry troops due to their verbal requirements.

Simulated testing was rejected due to resource requirements beyond the scope

of this project. Hands-on performance testing is typically time consuming,

especially for process testing which requires observation of each task step.

However, the results of performance testing provide the criterion measure of

task proficiency for all practical purposes. In summary, a decision was made

to explore the use of self-estimation, paper-and-pencil criterion-referenced

tests, and performance tests.

2
One possible approach to using the three candidate pretests would be to

employ each one by itself. For example, a soldier could be asked if he is

able to perform a task to standard, and then either be placed in training if

he said "no," or assigned to some other activity if he said "yes." Another

possible approach is to arrange the candidate pretests in a systematic order

to capitalize on their virtues while minimizing the effects of their weaknesses. Figure 1 shows an ordering of the pretests which may provide an optimal procedure. This procedure provides the easiest and fastest pretest as

the first step, the second easiest to administer as the second step, and the

most accurate, but time consuming, pretest (the hands-on performance test)

as the last step for anyone who is not already eliminated.

Requiring soldiers to take the performance test as the last step insures

that no one will falsely be considered proficient, since the soldier must

demonstrate his proficiency before being cr-.dited with a "Go." Theoretically, the only error that can be made with the avove ordering of pretests is to assign a soldier to training when he does not need it. Such errors may occur

either because the soldier misjudges his true ability, or because he fails

the paper-and-pencil test.

In addition to the pretesting procedure shown in Figure 1, several other

possible procedures were defined for this research. These were:

a. Self-estimate followed by performance test (see Figure 2);

b. Paper-and-pencil test followed by performance test (see Figure 3);

c. Performance test alone (see Figure 4);

d. No pretesting, everyone enters training.

COST--BENEFIT ANALYSIS MODEL FOR SQUAD MEMBER TIME

Having defined alternative pretesting procedures, it is necessary to devise a method for identifying an optimal pretesting procedure for various

training situations. The approach taken was to identify variables which may

influence time saved or lost when squad members are pretested, and then to

construct an algebraic equation which may be used to calculate the time saved

or lost by using the diagnostic information produced by testing. In addition

to the time saved or lost for the squad's members, the use of pretesting may

also affect the time available to the squad's leader. However, since squad

leader time has a different value than squad member time, and since there are

approximately 10 times more squad members, the cost-benefits of pretesting

for each kind of soldier will be dealt with separately. In the following

portions of this paper, the cost-benefit analysis for squad member time will

be developed, after which the analyses for the squad leader will be presented.

It should be stressed that the analysis presented below reflects the requirement that a soldier must be performance tested before he can be classified "GO." As a consequence, the only testing error is to classify a GO soldier NO GO during a pretest. Assuming the performance test is accurately

scored, a NO GO soldier will not be misclassified GO.

3
SWHAT\

SOLDIERS 

NO

ADMINISTER WRITTEN

TEST

SOLDI>ER N.:

ASS?.

. "YES

EMPLOY SSE

CO CONDUCT

PERFORMANCE

TEST

4I1*

SODR DID NO TRAIN

EMPLOY AS PEER

INSTRUCTOR OR CONDUCT

NEXT PRETEST

FIGURE I

PRETESTING PROCEDURE A

-Ni
WHAT

NO

NEXTIPRETEST

PREESTINGTROCEDUB

ASS

r, F

EMPLOYAS PEE
DNMRUTORORCONUC

\WRITTEN/.

NX PTEST

NO

• YES

CONDUCT

PERFORMANCE

TEST

SOLIE TRAIN

6I

EMPLOY AS PEER INSTRUCTOR OR CONDUCT

NEXT PRETEST

FIGURE 3

PRETESTING PROCEDURE C

°'
CONDUCT

PERFORMANCE

TEST

DID

SOLDIER TRAIN

ASS?

Y

EMPLOY AS PEER

INSTRUCTOR OR CONDUCT

NEXT PRETEST

FIGURE 4

PRETESTING PROCEDURE D

:167
Ph,, t-ra ini g iode I under Iy inq the anial ys is i~is an IIows.

I . pre test of somne sort is nIivtjn (al thoughi for 0trt-t ir. ti5 7

cons idera tions rul tO 0 t prof est irig

or most tasks to be tauoiht, ai complete., iiiilot,- Qpt 1ndori.

of t he task is (liVeni; then each task st,'I 1 i 1 ~ t t 11ow 1:. x

plai ned--step by step. Where practical , th _ iIi. I .;,

pe tnorm i ng each-I step.

T1-o tra Lno-is then atteumpt to per form th l t ,siflic~ci.;a~

I bIe; tr ainers aru to st on the t~ raineo ,,h itark c o

'11"I s;how/exr la irl so t hat 5r r t r t i 1(, 1 .orli 1 ut

r;I thu t td OOO trainei bel i(ev(s- th, ti -iinee has "'Ieac.

t ) e orm- t ho, t.as k to stanidard , a pet formance test f or the recuor

i.; liyen (train ins, records aro kept. to ail t raining5 mana~lomelt -

i=s lo is) . I' thte tra r nee fails, his; etrors ate corrected, ind doeen tinq onl the? lve 1 of performance hec is ei ther retested, or loo)pe;

0 tok to step, 3 to) conitinlue practicing.

Th'le key variables that have been employe-', to perform a cost-beniefit ari,fvf prtest ins ar ye 'ofilled be 1 o)w:

B Boe :it-, defined as the total number of man-minutes save 3i

hy applying any pretesting procedure.

The proportion of squad members (excludling thy squad leader

who serves as the trainer/tester) who are able to perform

the task (i.e. , who are "Go"') before traini no is gliven.

e The proportioni of men in a squad who are able to perform

the task but are erroneously classified by a pretestinq

proc~edurte as ''NO 0

:I Ce N The proportion of men in a squad who are correctl y classili ed by a pro to st io procedure as "G'O' onl a task (hits')

The- time that- it takes for an entire squa I to be pret-ested.

T 'he time it take(s for onle seldie r to unidol so .i pot-formani11e

test which is termed a"ceku

L)Time fo r the demon s t ra t i n i/e(p x)11 aat ion 1phos-,e o t r i11 i1 n 11 (1

Th e , lemon.-s t rot io()n i s i cmij ,1 ,' to oUr i I te0r i-Upt ed taIs;k 1orI for I

manco'. The exp)lana tion con 1ist s of a - st ej-thv-fnte(p most ration with itarn stop explained.

N The nu mbe r of men in a given s;quad (exc nd Ins the sqo. 0~

leadeor who serves, as the t r ainei/tes;tr).
Primary Time Saved by Pretesting. The principle segment of time that

may be saved by conducting a pretest is the time that it takes to demonstrate

and explain how a given task is performed (D). Soldiers who are found by a

pretest to be proficient in a task do not have to watch the demonstration/

explanation, but may instead engage in a more productive activity, such as

serving the trainer as a peer instructor or demonstrator, or working on another task, or taking care of a personal need.

For the purpose of constructing a model describing how time is saved or

lost for squad members, it is assumed here that any soldier may take a performance test immediately after the task demonstration is finished. (This

assumption is clearly consistent with the envisioned training system but represents a worst case situation for estimating the positive value of pretesting, since in reality, soldiers may have to wait to be tested while the squad

leader coaches soldiers who need help.) In this case, the length of the

demonstration (D) determines the amount of time that can be saved by an accurate pretest. The time that the pretest costs will be dealt with later.

Assume for the sake of argument that two soldiers who could both perform

a task had to sit through a demonstration because a pretest was not given, or

it was given but falsely found them to be "NO GO's." Each soldier woul& have

lost D time. In addition to the D time lost for each soldier, it may be seen

that the second soldier to be tested after the demonstration has finished has

also lost the time that it took for the first soldier to be performance tested

immediately after the demonstration. This time loss that is incurred as one

or more soldiers wait for their turn to be tested occurs in the same way for

both pretesting and posttesting. We have accordingly adjusted the time cost

of pretesting when constructing the cost-benefit model, and will explain how

after the pretest cost is defined below.

Algebraically, the total amount of time in man-minutes that is saved by

having proficient soldiers avoid a demonstration that they do not need is

found by multiplying the demonstration time, D, times the total number of men

in a given squad who have been correctly classified as "GO" by the pretest,

Nh, or finally:

Primary Savings = NhD

Pretesting Cost. The direct time cost for pretesting is simply the

span of time that it takes the squad leader or trainer to pretest his entire squad, times the number of men in the squad, or:

Uncorrected Pretest Cost = -NP

Adjustment of Pretest Cost. In the neu training system that underlies

the analysis being developed here, a performance test must be administered to

each soldier to establish if he is "GO" or not when the soldier or his trainer

believes he has mastered the task. In other words, a performance test, called

a checkout, must be administered to each soldier as part of the training procedure in the new system. Thus, the cost of the performance portion of the

pretest procedure for a soldier who passes it is not uniquely attributable to

the pretest procedure. The reason is this cost would have been incurred anyway when the performance test was given after the task demonstration. Therefore, the time cost for successful performance of the checkout during the

9
pretest procedure is deducted from the overall cost of the pretest procedure.

The analysis of this time credit is explained below.

The number of men in a squad who are correctly measured to be "GO" on a

task is defined above as Nh.

The time cost in man-minutes when one member of this group is administered the checkout, C, is the time C multiplied by the group size, Nh, since

everyone in this group is held up while the trainer administers the checkout

to one man. This time cost is:

CNh

Since each member of this "GO" group must take the performance test, the

total time involved in man-minutes is found by multiplying the group size by

the cost for administering one checkout:

2 2

ChCNh, or CN h

Thus, the cost of pretesting, adjusted for the posttraining checkouts that

would have been given even if pretesting were not accomplished is:

2 2 2

Pretest Cost = -(NP - CN h ), or -N9P - NCh2 ).

Time Lost When a Pretest Fails To Identify a "GO" Soldier. When a proficient soldier is misclassified as "NO GO" by a pretest, for whatever reason, he will then be required to attend a task demonstration/explanation ses- sion that he does not need. By spending his time on the unnecessary

demonstration, other tasks which could have been attended to go undone, and

thus time in this sense is wasted.

It might be argued that the time spent on the demonstration is not entirely wasted, since the proficient soldier's ability to perform well and to

retain his skill are incremented to some degree. However, there are several

reasons for believing that any such increment is generally negligible. These

reasons derive from considerations of motivation and cognitive theory. A

soldier who perceives that he already knows how to perform a task that is

being demonstrated is not likely to watch closely, if at all. Rather, he may

experience dissatisfaction and bcredom, with consequent inattention and daydreaming. From a learning theory point of view, the relatively passive

demonstration/explanation session has a serious limitation--the cognitive

requirements for watching someone else do something are different from those

for actively, smoothly performing a task, so that transfer of learning from

watching to doing is problematical; furthermore, this problem will be compounded by inattention.

The proportion of squad members who are mistakenly classified as "NO GO"

is -N so that the total number of soldiers involved is NeN. The time wasted

by -roviding these soldiers with the demonstration they do not need is simply:

Wasted Time = -Ne D.

N

Complete Cost-Benefit Model. Collecting the terms established above

for:

10
a. The primary time savings from pretesting;

b. The cost of pretesting adjusted for the checkouts that would be

given anyway; and

c. The time wasted when a pretest fails to identify proficient soldiers,

we have:

B = NhD - N(P - NCh 2 ) - NeND

By factoring out "N," and rearranging terms we get:

General Cost-Benefit Model: B = N(D(h - eN) - P + NCh 2 )

SPECIFIC COST-BENEFIT MODELS

Having constructed a generalized benefit model, the next step is to derive a specific form of the equation for each of the specified alternative

pretesting procedures.

Equation for Procedure A (Figure 1)

Procedure A involves all three pretest elements--self-estimate, written

test, and checkout. It is necessary to define the following new variables to

account for these elements.

y = The proportion of squad members who say, "Yes, I can perform the

task to standard."

A = The time it takes a squad to be asked for their self-estimates plus

the time to give the squad leader the answers.

w = The proportion of squad members who pass the written test.

K = The time it takes to administer and score the written knowledge

test for a squad.

Two forms of the benefit equation need to be defined to reflect two possible outcomes for the self-estimates.

a. Equation for Procedure A Where y = 0. The time cost, P, involves

only the time it takes to obtain the self-estimate. Since the checkout will

not be given to any squad member, no possible "GO" members can be identified.

Therefore h = 0.

Substituting A for P, and 0 for h yields:

For y 0, B N(D(O eN) A+NCO) N(-De-A) jN
Inspection of this equation reveals the following points:

1. The benefit value cannot be positive since both terms are negative.

2. The least costly outcome is the case where the error term is zero

(i.e., all soldiers correctly estimated that they couldn't perform

the task) in which case the only cost stems from A.

b. Equation for Procedure A Where y > 0. The time required for pretesting, P, has three components, one for each of the three pretest elements:

P- A + K + NwC

The term K mainly reflects the time that it takes for the slowest squad

member to take the written test, where scoring is quickly done with a key or

from memory. The term NwC reflects the number of squad members who have

passed the written test, Nw, multiplied by the time required to give each

one the checkout, C.

Substitution for P in the general model yields:

For y > 0, B = N(d(h - eN) - (A + K + NwC) + NCh2).

Equation for Procedure B (Figure 2)

The term P has two components--one for the self-estimate, A, and one for

the time that it takes to give the checkout to each soldier who estimated he

could perform the task, NyC. Substitution for P yields:

B = N(D(h - eN) - (A + NyC) + NCh2 ).

It may be observed here that acquisition of positive values for h is crucial

for generating positive values for B. If h is zero (which may occur when no

one can perform a task before training, or none of the soldiers who can perform the task estimate they can do it) then NCh 2 drops out, and the remaining

terms cannot rise above zero. In addition, the largest possible benefits

arise when the demonstration time is large, as well as h equaling one.

Equation for Procedure C (Figure 3)

The term P has two components, one for K and one for C. Substituting

for P yields:

B = N(D(h - eN) (K + NwC) + NCh2 ).

Equation for Procedure D (Figure 4)

The value for P is derived solely from the size of the squad and the

time for conducting each squad member's checkout:

12
2

B N(D(h - eN) - NC + NCh2 ), or

N(D(h - eN ) - NC(l - h2)).

Equation for No Pretest

When no pretest is given, P is zero. Furthermore, there is no possibility of identifying any proficient soldier before the demonstration is

given, so the proportion of hits, h, is zero.

The resultant equation is:

B = -NDeN '

An equivalent alternative equation is based on the fact that the magnitude of the error term eN equals the porportion of squad members who are

"GO," g, before the demonstration is given. The alternative equation is:

B - -NDg.

Therefore, when no pretest is given no time can be saved, and time is

lost as a direct function of the length of the demonstration/explanation and

the number of soldiers who are proficient without further training.

Modification to the Equations for Simultaneous Checkouts

Performance testing for the majority of tasks covered by the new training

system requires observation of the step-by-step process used by each soldier,

since there is no clear-cut product. In some cases, even where evaluation of

a product would suffice for a checkout, the value of the diagnostic information that may be gotten from observation of performance is so high that a

process test is preferred. However, when a product-oriented test is preferred, then the cost/benefit equations need to be modified.

The definition for "checkout" used thus far assumed that only one soldier could be tested at a time. If a product test is employed, then it may

be possible, where resources are sufficient, to test all squad members simultaneously, just as when a written knowledge test is used. Wherever a time

cost for the checkout was previously involved, the cost was calculated by

multiplying the number of test takers by C. However, for a simultaneously

given product test, the cost is the amount of time that represents the upper

bound of the slowest acceptable test performance.

As before, we term the time taken by the entire pretest as P. However,

in formulating the credit adjustment for passing a checkout in the pretest

that would have cost time after the demonstration, a change must be made.

In this case the time cost of the checkout at either time is simply the proficient group size (i.e., Nh) multiplied by C. To avoid possible confusion,

the checkout time for a simultaneous test is denoted C'.

Pretest Cost = -(NP - NhC').

The corresponding general model is:

13

-. -
B = NhD - NP + NhC' - Ne ND, or N(D(h - eNN ) P + hC').

AN EFFICIENT DATA COLLECTION PROCEDURE

The alternative pretesting procedures described above need to be comparatively evaluated from the standpoint of their cost-effectiveness. To do

this, a task domain must first be specified, as for example the 56 common

or basic tasks included in the Soldier's Manual for Military Occupational

Specialty (MOS) lIB10. Given the fact that a data collection effort involving the four different pretesting procedures when applied to lIB tasks would

be time-consuming and difficult to accomplish, it seemed important to devise

an efficient data collection procedure. The solution adopted was as follows.

Participating soldiers are asked to read a description of the task, conditions, and standards for each task that is sampled from the Soldier's Manual.

Each soldier is then asked first to estimate his ability to perform it, second to take a paper-and-pencil test regardless of his estimate, and finally

to take the performance test regardless of his written test result. The

full procedure is explained in advance so that soldiers who might otherwise

overestimate their capability realize that a hands-on performance test will

be given. The data collected by this procedure may then be distributed to

all four active pretesting alternatives by means of a logic tree analysis,

thereby effecting a sizeable economy in data collection requirements. The

analytical technique is illustrated by the results for a sample task in

Figures 5 through 8.

ILLUSTRATIVE APPLICATION OF THE COST-BENEFIT MODEL

Reported here is a deE iption of an application of the general costbenefit model to the four alternative pretesting procedures that have been

described earlier in this paper in Figures 1 through 4. In addition, the

model is applied to the procedure of placing all soldiers into training

without pretesting. This application is based on data that were collected

from infantry soldiers in units of the 7th Infantry Division at Fort Ord,

Calif., during 1977. The data were specifically collected on a sample of

five Soldier's Manual tasks included in the set for Military Occupational

Specialty (MOS) lB. These five tasks were selected for study because they

provide a sample which covers a fairly wide spectrum of activities--leadership skills (Organize, and Employ a Tank Hunter-Killer Team), cognitive

(Encode/Decode, and Authenticate Messages with a KAL 16 Coding Device), and

"hands-on" (Emplace/Recover an MI6AI Anti-Personnel Mine). This small sample of tasks clearly cannot be used to estimate accurately possible results

for all tasks in the lIB series, but it does serve as a test bed.

The data were collected according to the procedure described in the previous section. Values for all variables except A (the time to obtain selfestimates) are shown in Table 1; the value for A was approximately 1 minute.

The logic tree analysis was used to obtain the values for model-specific

variables for a hypothetical squad having nine members in addition to the

leader.

By employing the sample data shown in Table 1 to the alternative pretesting procedures, it is possible to generate the benefit values in minutes

for each of the sample tasks, as shown in Table 2.

14
44)

0

c

tva

4

4)

4-

Ca.

44

L.

41)

0 U,

4-h 

0

00)

7-a
I.L

0 -- 0

.D 0

aL.J

00)

4)

.0

41)

0% 0 - '.0 75

16L
n -35

Self-Estimate 19 16

Performance71242

Error

FIGURE 7

Results for task -- Encode/Decide KAL6I -- needed to calculate the benefit value for

Procedure B.

n = 35

Written 16

Performance51 63

Error

FIGURE 8

Results for task-- Encode/Decode KAL61 -- needed to calculate the benefit value for

Procedure C.

17 L
C&L wLO 0 0 0 0-

V)

1 6 Cl '3 4i N N C I".s 

d Z 24 0 0 0 0

2: Z CD Cl (5 Cs Co4 CD

CL CD . . C)

C: 0n 00 0 0 ON. " .4 C4 

14 C) C u

(N

C4 C) 2: GoJ r- %.e

rN <N 2:4 4ON 0 00 C' to L 

0L . ~ crV . CJ)* 

V7 0 J 0 -0 C)O V)0 0 0 E~C.

C) C 3 CC ( u Cd 

x/~ 0 NC o, E

r* m CD - C) )C

ci~4- ) b C/V 0

Cl Cl C i

ci CL. Z CC C ( o~ 0-"4 

0 -C-

< >4 C) "' C) ' .-3 '3 N ~ u--0 C -0 

(N (N (N (N (N V

Cs 00 (N 0 0 - -~ E , > ) L- C

04 %- u" -o -to~N 000 Co0 L0 Cs (N C C a< Cs 

N I-m C0 u 0o

20 ID. E4. L. s E 4.. e

CCL

co C) %. ri z 1111Ei11 C1 3 3 :34 > + -- rj-' 0
U r

00

o x4 I- C1I

-0.

00

L6 ON 00 50 -0 -6 0' 

0 r i

.

U -

z V Q W' V% N 00 LL j - 4-

00

>j 'Q

D~~ o00 '

0 xi t - -x 

z ~ 19 I 
It may be seen from the summary line for Table 2 that Procedure B minimized the time lost (i.e., it appears to be the most cost-effective procedure),

while automatically placing all soldiers into training appeared to be the most

costly procedure. Procedure B happens to be the pretesting procedure selected

by the new training system, but, of course, the data reported here are insufficient to establish the validity of this selection. With a sufficiently

large sample of tasks, the kind of raw data shown in Table 2 could be analyzed by a one-way analysis of variance test to estimate if the alternative

procedures reliably differ, and, if so, then analyzed by a test like the

Newman-Keuls to estimate exactly which procedures are distinctly different

in their benefit values.

A COST-BENEFIT ANALYSIS FOR SQUAD LEADER (TRAINER) TIME

Conducting a pretest obviously consumes the squad leader's own time as

well as the squad member's time. In this section, cost-benefit models for

the squad leader's time in his role of trainer/tester are developed.

The time that a pretest, P, takes is a cost for the trainer which needs

to be adjusted for the checkout time, C, that would be consumed anyway after

the demonstration/explanation even if a pretest weren't given. The adjustment is simply the number of hits, Nh, multiplied by the time for each checkout. The tentative benefit equation for the squad leader is:

B = -(P - NhC). L

In the relatively unusual case where all squad members are hits, then

the time that would have been spent on the demonstration, D, is saved. The

appropriate equation is:

BLD =-(P - NhC) + D.

Given the infrequent applicability of this model, it will not be discussed further here.

Since the squad leader's actions affect all of his men, it may be appropriate to weight his time by the number of his men, N. The benefit equation then becomes:

BWL = -N(P - NhC).

The specific formulation of BWL for each of the alternative pretest procedures simply requires substituting for P the expressions previously explained. The equations for each procedure are as follows:

Procedure A, y > 0: BWL -N(A + K + NwC - NhC), or

-N(A + K + NC(w - h))

20

L
Procedure B: B = -N(A + NyC - NhC), or WL

-N(A + NC(y - h))

Procedure C: B -N(K + NwC - NhC), or WL

-N(K + NC(w - h))

Procedure D: B -N(NC - NhC), or

-N 2 C(l - h)

Procedure E, No Pretest: BWL = 0.

To illustrate the behavior of BWL, the data from Table 1 have been used

to generate results, which are shown in Table 3. It can be seen from these

results that the least costly procedure is of course to skip the pretest.

Conversely, the data show that the most expensive procedure would be to give

all squad members a checkout.

The experimental training system places a strong emphasis on the use of

peer trainers to motivate acquisition of specific task and leader skills by

squad members, and to increase the squad leader's ability to spend his time

where he can be most productive. Therefore, it is possible that squad leaders may generally assign one of their squad's members who has passed a pretest to conduct the task demonstration. In this case, the previously defined

model for BLD applies, since the time that the demonstration takes is saved

by the squad leader. Table 4 displays sample weighted squad leader benefit

values for BWLD where the general equation is:

B WL D = -N(P - NhC) + D.

The actual application of this equation to infantry squads for tasks with

relatively short demonstration times may well be limited by the squad leader's

desire to provide supervision. However, it is apparent from the sample BWLD

values shown in Table 4 that rather dramatic time savings for the squad

leader may be generated by his use of peer instructors.

In the next section, various benefit models are constructed by combining

the benefit equations for squad member and squad leader time.

COMBINED BENEFIT MODELS FOR THE SQUAD LEADER AND HIS MEN

A combined general equation for B and BWL is:

B + BWL = N(D(h - eN ) - P + NCh - P + NCh), or N(D(h - eN) - 2P + NCh(h + 1)).

21
L.

U

0

LJ .

z

U

N COD

0• 4. 0% % N.

< 0

04

Jz L.

V (N as@ e~0

~~N4

-4 4

0 41 4 " .

LL c% 0 ~ D- -4 . c <4

0U Ix w ce X be a

&2
Co C

lu

u

0

C) 41,

C)U

0

V) 00~( ' L.

.)

0%

L LI

z

401

I ca

D 00~

o co >~ U

* CL. o

e* u

C m) l C C J (0L v Z~ 0i- w.- 3 C) 0 X . 0 5 4.C IX 

23
Since the overall terms which reduce P (to reflect checkouts that would

have been given after the demonstration even if the pretest were not used)

constitute only a fraction of P, this combined model provides a less favorable outlook for pretesting.

The specific equations for each of the experimental alternative pretesting procedures are presented below:

Procedure A, for y > 0:

B + B = N(D(h - eN) - 2(A + K + Nwc) + NcH(h + 1))

Procedure B:

B + BL = N(D(he N ) - 2(A + NyC) + Nch(h + 1))

Procedure C:

B + B = N(D(he N) - 2(K + Nwc) + Nch(h + ))

Procedure D:

B + B N(D(h - eN ) + NC(-2 + h + h)) WL N

Procedure E: B + BWL = 0.

The corresponding equations for (B + BWLD) are found simply by adding ND

to the (B + BWL) values for Procedures A, B, C, and D.

For certain possible applications of this scheme of analysis, it may be

thought inappropriate to be giving the trainer's time a weight equal to the

trainee group size. For example, if the trainer and trainees are working

full time at training, then multiplying the trainer's pretesting time cost

by the trainee group size may lead to an awkward result. Consider that when

the pretesting benefit for the squad members is calculated, the time that the

pretest takes is multiplied by the group size, N. Now when BWL is calculated,

the pretest time is also multiplied by N. In this way, when B and BWL are

combined, the benefit (or cost) of the pretest time for the group is effectively counted twice. Therefore, unweighted equations for BL and BLD may

also be desirable for some situations. The combined general equation for

BL + B is: 

2

B + BL = N(D(h - eN) - P + NCh 2 ) - P + NCh, or

N(D(h - e N ) - P(N + 1)/N + Ch(Nh + 1))

The equation for BLD + B is:

24
N~h 2 )

B + B = N(D(heN ) - P + NCh - P + Nch + D, or

N(D(h - eN) - P(N + 1)/N + ch(Nh + 1)) + D.

To conveniently illustrate the relative behavior of all benefit models,

the sample data from Table 1 have been used to generate results (i.e., mean

benefit values, and rank ordered scores for the means from each model) which

are shown in Table 5. It may be seen that Procedure B gave the best overall

performance ( = 4 man-minutes saved) and Procedure E the worst (F = -76 manminutes, lost).

If the sample data were reliable, then the following conclusions could

be made for the infantry tasks that were studied. When only the squad leader's time and not squad member time is considered, and when the squad leader

will either conduct or supervise the task demonstration (Models BL and BWL),

then no pretesting should be given. But, for all seven remaining models,

Procedure B ranks first five times and second twice. However, these conclusions are meant only to illustrate how the models behave and may be analyzed,

but cannot be regarded as reliable because of the small samples of tasks,

soldiers, and testing locations.

CONCLUSIONS

A methodology designed to enable selection of an optimally efficient pretesting procedure for use in group training/testing situations has been explained and illustrated with sample results. This methodology consists of:

a. A generalized set of benefit models,

b. Specified alternative pretesting procedures,

c. Specific algebraic equations which permit calculation and comparison of the benefits accruing from the various alternative procedures,

d. A suggested efficient data collection plan for the specified alternative pretesting procedures.

One salient limitation of the specified alternative pretesting procedures and their benefit models derives from ah ideal requirement imbedded

in the proposed on-the-job infantry training system underlying this research--

namely, the requirement that task training may officially terminate for a soldier only when he has demonstrated task proficiency by passing a performance

test. As a consequence of this requirement, all analyses were predicated on

the assumption that no soldier would falsely be classified proficient by a

pretest. This point is illustrated by the diagram below, where the validity

of both pretest outcomes is described as a function of the soldier's true

proficiency:

25
000

< C

4)~0 0 0 '3 -30

0 cv C

E2 I- 2

U

C.C 0

OLU

4-J

C0

o4 C' l C (%4 . rq .- 4

< V 0

v CO N(N4 c 4 CD(, N(N( co~~ c% \

a L. -C4

z 0

*t r

< 0

lz 6

.. + +o co t o a c c o c Ix Ix z,

26
IPRETEST OUTCOME

GO NO GO

GO VALID INVALID

A B

TRAINEE'S TASK PROFICIENCY

NO GO INVALID VALID

C D

All pretesting procedures were constructed so that cell C could not occur; the only mistakes logically possible stem from cell B. However, the

ideal requirement that training always be capped with a performance test is

sometimes perceived as a time-consuming luxury when training is conducted by

operational units in field settings. Thus in reality, the invalid outcome

depicted by cell C may sometimes occur. It is necessary therefore to perform

an additional analysis which will enable estimation of the costs or benefits

that arise when performance testing is not reliably practiced in a training

system and the required research is underway.

A final point to consider is the potential scope of application offered

by the quantitative approach toward cost-benefit analysis presented here.

Although only time costs have been discussed, financial costs may also be

handled.

Time may be directly transltted into money by multiplying it by the rate

of pay. Once time is translated into dollars, then other financial costs may

be introduced into the benefit equations. For example, if expensive simulation equipment needs to be purchased, then its ammortized cost may be included in the cost-benefit equation which represents the appropriate alternative pretesting (or testing) procedure. Another example concerns the use of

expensive ammunition which will be expended during performance testing.

The potential annual dollar equivalent of the time that the Army may

gain by the use of pretests can be illustrated with an example from infantry

training. There are a minimum of 56 basic, common infantry tasks that any

soldier in a rifle or weapons squad must be able to perform. It is reasonable to assume that each squad would be required to train on each of the 56

tasks at least once a year. The mean benefit values shown in Table 5 may be

multiplied by 56 to estimate the time involved for a single squad in 1 year,

and then multiplied by the number of squads in the Army. Finally, the time

involved for squad leaders and members may be multiplied by their estimated

hourly costs to find the dollar equivalent. As an example of the annual

benefit that may be achievable, the results for Procedure B may be compared

with Procedure E using the benefit model B + BLD. In this comparison, Procedure B was found to gain about 430,000 man-hours which has a value of approximately 6.5 million dollars, based on salary costs for productive training time (estimated at 30% of paid time by Bialek, 1977).

14 More important than the annual dollar differences between the various

|* pretesting procedures is the percentage of time that may be freed or lost,

given that combat readiness depends on the amount of training time that can

be used. A comparison of Procedure B with Procedure E for Model B shows the

27
difference between the mean benefit values to have been (-42-(-136)), or

94 minutes. For a single squad member this amounted to 94/9, or about

10 minutes. If the average time for task training (i.e., task demonstration, plus skill practice until mastery is achieved, plus the checkout) were

1 hour, then the time saved by Procedure B over Procedure E would have been

about 17%. Thus if the magnitude of this sample result is accurate, then

training efficiency and effectiveness may be significantly enhanced by application of this quantitative approach toward estimating pretesting costs

and benefits.

28

28

. I
REFERENCES

Bialek, Hilton M. "Personnel turbulence and time utilization in an infantry

division," HumRRO Professional Paper 3-77, 1977.

Bialek, Hilton, M., Taylor, John E., & Melching, William H. Human Resources

Research Organization, and Hiller, Jack H., & Bloom, Richard D., U.S.

Army Research Institute. "Continuation of a development of an individual extension training system for managing training in the Army unit,"

ARI Technical Report TR 78-BI, 1978.

29
DISTRIBUTION

I US AmMY WLSILRN LOMMANI AITN: AVPL

I L)EPAr4IMENI OF liL NAVY TRIIlqu ANALYSIS ANU EVALUATION bROUP

I HUI)A AT TN: UAA~.,-ED ... ____

I HU. iCATA ATTN: ATCAT-uP-w

2 HUIIA RESEARCH AN) STUDIES OFC

I MILT IARY OCCUPATIDNAL UEVL-LOiPMEN( UIV I)PC-MSF1Q,-P4RM~--ew'~--tl-r-----....-

4 QASI) (MRA AND L)

I NHi)A ATTN: iUAMJ-q(JR

I HU rLATA TECHNICAL LIbRAHY ________ __

I HU0jA ODCSPER

I USHAuC09 SIC

I HUIJA AlIN: DAMI-151___

I USA mYIATION SYSTEMS CUNIU AflrN: UHSAY-LDN

I USA LONAUCOM ATrI4: AMSEL-PA-RH

I USA iaRRAUCOM AfTN: AltE-LU-AC .-- -- ---.-----

I HEA~wUARTERS, US 4ARINL CORPS AITN: CODE MPI-20

2 US A"MY EUROPE AN) SLVLNTH- ARMY

I 1ST iNFANTRY DIVISION ANU FT. RILEV AlTri: AFIN-tPT4;-----_

I (154 iNTELLIDLNCL AND SrCUKITY COUMMAND ATTN. IAOD'S-1N6-T

~? HG Ti".AUOC TtCHNiICAL LIKHAHY

1 NAVAL IRAININLI EuJIPMLNT LtN AITNI TELHNICAL uLtIwARTr-- - ---

I MILIIARY UCCUPATIDNAL ULVLLOPMENI UIRECTORATE ATTN: ATZI-NCR-MS-M, RM 3N33 HOFFMAN SLOG 11

I DATA ANALYSIb DIVISION ATTN: AILI-NCR-MO. HOFFMAN BLDG 11

1 UiSA mdLPE-RCEN ATTN: DAPL-P'OO-T . -- 1.-- - .. -- I

I US4iF.CFA CMlIv~ JkbANlZAIiOFJAI LFHECTIVE4E5S JWRANCH I AMH i NFANTRY DIVISIOi4

I HUOA ARMY FURL. 'OIIRNILAIION LOUNUI4ATION OFPICE --- ~--*__

I NAVAI AIR SYSTLm COMMAND /

I DMsOwS (U1ST 4) A(TN: LAMU-RjI

1 1230 USARCOM HrSrPVt CEN1LR -. ...

I OS AWAMY 50LIEt, tjPJWT LLIjTL,4

1 ) 1. E T ukat T ,oF a 4 AV I A I ION Al IN: AILK-AAU

1 USAAH"C + FT. NNOK AVi~iON DIViSIUN..........----.-.-.-.-.-

I LISA PORCES COMMA143 AFIN - UEPulY L OF s FOR INTLLLIGENCE

1 USA iORCES COMMAND AFUP - DEPuIY C.HIEF OF STAFF FOR OPERATIONS

I US A.tMY AIR DE~kNSE SCHOOL ATTN; ATSA-rlTn

I DIN(Et-OWATb OF TRAININU AIIM: AILU-T

I 1ELTOHATE UF CU~IbAI ULVLLUI-MINIS AWN: AFZ(WO

I HU')ANCOM MARINI: CORPS LIAISON OItC ---.-------- .--.

I DEPAwlMENT OF lHL ARMY OS ARmy INTELLIGENCE + SLCURITY COMMAND

I US A"MY SAFETY CEVTfk ATIN: LldHAV,4 PILDG 4905

1USA mISSILF COMIA14J ATTN: OHSt:I-NTN...........----------- .

I UjS A"~MY CICOM AIN: DI-0LL-Aluil

1 USA rORCtbS CU*M',IN)

I PMM 7"AOL /

I US MiLITARY DISTRICT OF WASHINOJUN UFC OF EQUAL OPPORTUNITY

I NAVAL CIVILIAN PE4SONNLL LCUML SOUTHE4N FLU UIlY

22 ANtI IAISUN OFF ICE

1 7TH mHMY TRAINI'iu CUMMANU

I HU UaARLUR ATIJq: UC.Ol'S

I HUILA. OCS STUDY 3IiIKt. ..- ~ .......---..

I U.S. NAVY THAINIJD ANALYSIS LVALUAIIJN GROUP

1 (SACuEC ATTN: ATEC-EX-E HUMAN I-ACTORS

I ISAFA(OS/TAC bFNI0H A~v4Y ADVISUN

I Ir4ILK-UNIV StMlINAR ON ARMLUj OrULS + SOC

I US%~ tLECTRONIC ,WHJVINb7 7UNU AITNI 51EEP-MT-ES

1 r)A A (RDA) DEPuTY FUR SLILNLL ANU ltE;HNOLOOY

I OFC %)F NAVAL RLSLAHCil/

I AF'IR./LPIT

1 AF'IWL/LRLG

I AIR t.ORCE HUMAN mNbSUCES LAB AITN: Atr4L/TSH

31

PRJW PA= DBJUNON FLM
I FEOEI'AL AVIATION 4DI41NISTRATION CLNTRAL REGIUNO LIORA"y. AL6---..--

I AF AMrL/I3B

* 1 AFa~M.4L/Hk

I NAVAL PE.RSONNEL K ANU U CLNTLR L.UMMAN1J AW UPA4569

I NAVY PERSONNEL R AND 0 CENTER/

I NAVY PERSONNEL R AND D) CENTER ULkCTOf( OF PROORAMS

)NAVY PERSONNEL R AND 0 CLNrER / - .- _____I. --

I US AHtMY AVN LNWIN-E.RINb LWtHI ALrIVITY %iTTN: UAVTE-TO

? OFC uF NAVAL kL)LAHCH PLHSUNNFL ANU TkAlidlINU RESEARCH PNO(,RAMS

*~ NAVAL PERSONNEL k * U CLNILR / __

I OFC vF NAVAL RLSEARCH PRUJECI OtFICE~v ENVIRONMENTAL PHYSIOLOGY

t DEPf. OF NATIONAL UEFENCt DEFENCE AND) CIVIL INSTITUTE OF ENVIR MEL)

) NAVAL AEROSPACE MEDICAL RSLH LAd AEOPC-S~o,6-E.RMN

I USA INAVOC SYSIEMS ANALYSJb ALIIVITY ATTN: ATAA-TCA

I HLADwUARIERS9 LUAST UUARU CHIFF9 I"SY :HULuGILAL HSCH HR

I USA v4ESEARCH ANU T-LCH1NULU(3Y LAb /

1 USA tNGINEER TOPOGRAPHIC LAOS ATTN*. ET LGSL

I USA tNGINEER TOPOGRAPHIC LAOS ATTN: ST INFO CENTER

I USA .oNGINEER TOPU5(APhIL LAOS ATTN-: Lk-10mS -

I USA -40BILITY EOLJIPMENT H ANI) u CUMJ ATTNI. IJHME-.TGI (SCHOOL)

I NiGHA VISION LAb ATiN: UHSLL-NV-blUU

I USA sWAININi alUAkO ATTN: ATTCawAaIA

I USA riUMAN ENGINEERiN LAO

I USAHrL LIAISON REP, USAAVNC/

I IJSA P4ATERIEL SYSTEMS ANALYSIS ACTIVITY ATTN. DRXSV-C--.

I lUSA 4ESEARCH OFC /

1 NAFEt. HUMAN EN(,LJLEEINU dkANCH

I HAITtFLLE-COLUAb,)S LA9ORAT~kIL4 4AC;TICAL -T-Er.*4ICAL- Q.'iI USA ARCTIC TEST CEN ATTN: AMSIE"PPL-TS

I USA i.OLO REGIONS TEST CEN ATTN: SIECR-Op

I USA tIONCEPTS ANALYSIS AG.LY ATTN4 C;SCA-.RQP-- ____________

I USA LONCEPTS ANALYSIb At3LY ATTNZ LSCA-JF

I HU wkAIR nIV UF 4EUHOPSYCHIATwY

I &SACACDA ATTN: ATLL-CAC-iG-- -- ____________

I USACACDA ATTN; ATZL-CAC-iM

I USACAC ATTN: ATZL-CAC-IA

I USACoCUA ATT14: ATZL-CAC-AI USA tLECTRONIC WARFARE LAO~ CHILo INTE.LLIGENCE MATER nEVEL *SUPP OFF

IUSA HSCH UEVEL # STANDANDiLA kiW9 U*K*

I USA kL~SLARCH ANOD LVELUJPMLNT LAb CHLF kF.EWA hrtgNec Ul 9091 cO~c'--ILAla

I TRAJmNA ATTN: SAJS-OR

I NAVAL AIR SYSTEMS COMMAND ATTN: AIR-5313

IECUM ATTN: AKSEL-CT-O . __________

I USACiEC TECHNICAL INFORMATION CENTER

I USAA.4L L18RARY - --

IHUA4, RESOURCES NSCH 0145 IaUMR4W) d

I SEVILLE RESEARCH CORPORATION

I USA IRAQOC SYSTEMS ANALYSIS ACTIVITY ATTN: ATAA-SL (TECH LISRARY)

)UNIFuRMEi) SERVICES UNIT OI THE IILALTH SdI -EAlrTn-lSt4AT3

I USA uOA4PUTER SYSTEMS CUMMANU ATTN! COMMAND TECHNICAL LIBRARY H-9

I HUlAwi RLSOURCES kSCH O~o (hsUMHRQ)

I HUMRmO LIBRARY -- _____________

1 EUSfiS DIRECTORATE, USAAMRUL TErCHNICAL LISRARV

I RA40 CORPORATION / .

I RAND CORPORATION ATTN: LISRARy U- ----

I FEDEoAL AVIATlION ADMINISTRATION ATTN: CAMI LIVRARY ACC-44DI

I NAFEL LIBRARY, ANA-4

I GRONiNGLH LIBRARY ATTN: AIZF-RS-L bLt~b 1313

I CENr.tR FOR NAVAL 4NALYSIS

I NAVAL. HEALTH RSCH CEN LIdRAWY. ___-

I NAVAL ELECTRONICS LAB ATIN: RWEARCH LIBRARY

I NAVAl- PERSONNEL R AND U CLN LIKHAkY ATTNt CODE P196

3?
I AIR ,4)RCE HUMAN RESOURCES LAb ATTN: AFIIRL/OTS.___

1 HO. 1T. HUACHUCA ATTNZ TLL.H 10 LIV

I USA ALAUEMY OF HE.ALTH $CIt.NCES STIMSJN LIBRARY (OOCUMFNTSJ

I SCHOuL OF SYSTLMS ANU LObISTICS/

I USAMItRDC TECHNICAL LIbHAHY

I DEPANTMENT 0F THE NAVY IHAINlNtb ANALYSIS AND EVALUATION SP

1 NAfluNAL CENTER F)R HEALTH STAIISTICS / - _____

I USMA DEPT OF BEHAVIURAL bLI ANU LLAOENSHIP

I OLD- iOMINION UNIVERSITY #J.RFOo4MANLL AbSFs3S,.NT LAdURATORY

I USA LOMMAN) ANU t7ENEMAL STAFF CULLLUbE ATTN; LtONANY -

I USA iRANSP0RTAI'4 SCHOOL USA IhANSP TECH INFO ANU RSCH CEN

1 NAiA HQ

I NMRDt; PROGRAM MANAGER FON HUMAN PERFOHr4ANCE -_ -

I NAVAL MEDICAL H AND U CUMMANU ('4)

1 uS~k ,DMINCEN IEC'INILAL kt.%,kARCI bkANCH LIBRARY

?HOUA USA MED HS(A AND DEYLL COMMAND- -- - - __ -

IUSA 1-IELU ARTY BUD /

INAT LLEAR1NGiHOUSL FOR mLNIAL HEALTH INFO PAWKLAWN BLDG

I U OF TEXAS CEN FOR COMMUNICAT ION RSCr( ------ _______

I IN'3TiTUTL FOR DEFENSL ANALYSES

IUSA IRAINING SUPIJ:)HT CLNTLH Al 11: Allt-I)ST-IA

I AFrHkL TECMNOLOt3Y OFC (H) _________-

I PURDuE UNIV DEPT OF PSYCHUL~bICAL SCIENCES

1 USA MOBILITY EWUIPMENT R AND U COMMANJ ATTN: DROME-ZG

I HQ. wSA MO)W AfTN: ANPE-Ot- -- __ ________

I D)A Ua ARMY RETHAIJINb. tiuL RtSF.AHCt * LVALUAIIN DIVISION

I CALS#'AN HUMAN FAIrOWS ANU TRMI"'INb CENTER

I UiSA ALROMLI)ICAL RESEARCHI LAW SlCENTIFiIC INFU14AitfN-Cvtw1tW

I USA~F SCHOOL OF AEROSPACE MEDICINt AERUMEOICAL LIBRARY (TSK(-4)

I US MLLITARY ACAot4Y DEPT. OF HISTORY, BLDG 601

I USA INTELLIGENCE ZCEN AND SLH ATTN: SCHOOL Lt0AftAy--

I USA iNTLLLIGENCf CEN ANu S)LlliI~IN' ATSI-oP

I MA41i'E CORPS INSTITWlE

1 NAVAi- SAFETY CENTER /-_______

I USAAvNC AND FT. RJCKER ATTN: AIlZU-ES

I US Ai.MY AVN lhlk L18RARY ATTN: CHItF LIBRARIAN

I IUSAAVNC ATTN: ATZWU

1 US MiIiTARY ACM)E4Y DIRLLIOR tu1 INSTITUTIONAL RSCH

1 USA~ AIR ULENSL SCHOOL AIIN: A1bA-CU-mS

I USAAu!3-LibRAHY-DUCUMtNTS ____________

1 USA AIR DEFENSE 83ARD ATTN: FILLS REPOSITORY

I USA iNFANTRY BOAH) ATIN: ATZH-IU-AE

1 USA~ iNTELLIGENCE CEN AND 5C,1 ATTN: AT5-Tft----

I USA uNDNANCE CLN' AND SCM ATIN: AISL-TU-TAC

I USA ARMOR SCHOOL ATTN: AILK-Ti)

I USA ARMOR CENTLO VIRECTOMATt Ol' LOM8At -DEVEL0PftN?5-'---

1 NAVAu POSIGRADUAIE SCH ATIN: ULEY K(NOX LIbRARY (CODE 1424)

I USA iRANSPORTATIUI SCHOOL DEPUTY ASST* COMMANDANT EDuCA. TECHNOLOGY

1 UISA sIGNAL SCHOOL AND FT. GORDO~N ATTN': 'ATZM-E?--

I IUSA ARMOR CENTER 4 Ff. $KNUX OUFCL OF ARMOR FORCE MGT + bTANDARDIZATION

I CHIEt OF NAVAL LDJCAIIUN AND T'~v

I USA b16NAL SCHOOL # fT. iUUUN tJUCATAIJNAL-lt"flOt0 DIV191014

I HU AIC/XPTD THA1INN SYSTEMS DtVELOP4ENT

5 USA iNTELLIGENCE CE.N AND SCM AITNI ATSI-ERM

I US AmMY ARMOR CENTER ATTN: A17K-TU-P*M.O

1 USA wUARTLRMASfkR SCHOOL DIREtORUATE UF TRAININU DEVELOPMENTS

*I US1 C~jASr OLAHD ACADEI4Y /

1 lISA IRANSPORTAIT UN SCHOOL UIHF.LIUHATE OF TRAINING *DOCTRINE

* I USA INFANTRY SICHOOL LIbHAHV

1 lISA INFANTRY SCHOUOL AINI AISH-L-V. ______

I US AxMY INFANTKY SCHOOL AITNt A[SH-C)

I UjSi INFANTRY SLHOOL AimN: Atsii-DOT-LRU
I IJSA INFANTRY SLH03)cL AimN: AlSH~-LV

I USn .4P + CHEM !:10l/TNo, LLN * FT. mCLLEL.LAN AITN: ATLN-PTS

I 'ISA 4) + LHEM SCH/Tlo. CE'4 +H, 14LCLE6.LAN uix, CQM8AJ LEVI.LUPMENJT

I USA .4P + CHLH c)t1.v1Nt CL~q + I-I. MCLL..LAN UIHl, IWAININ(. JtV4i"T.--

I USA m.4 CHEM bCH/TNu. CLN + FT. MC(.LEL..LAN ATTN: ATZN-mP-ACE.

I USA iNSTITLTE UF AUMINISII(ATIUN ATTN: HE51ULNT TRAINING MANAGEmENT

I USA iIELU ARIILLE4Y bChOUL MURkl Sw~Tr LI8fCARY

I o~i iNSU TIITE U ~4±i'ibTHA1 IUl ALAU .MIC LWHiARY

I 'IS% mAk CULLt.(jL ATTN: LIIHMNY

I USA c-lbiNLEW SLriuZIL LIWiI'MY AiLJ LLAN ViniG NE.ISUUNLES CE.NT~M-.k

I USA Ad4MUR SCHOUL (USAkmS) Alrf,: LlbRARY

I UR64,AZATIONAL HE'*tClIvtLNtSS CE~N +' SCrI ATTN: LI'6RARIAN

I US AtqMY LNTFLLit3L\I4CE CLNrtk SLH-IUUL ATTN: ATSI-70

I Wi AmmY INTELLilot4C Clmd\It + scr I~UUL A I TI: ATSI-R'4-M

I uIS AMY INTELLIbt'J~t CLi4TLH + .(;iUUL A T 1,: AFSI-TU-Pml

I US A"MY INIELLiu'LJCL CLNr-x +' ttrUUL A f7N: AT$I-D- CS

I US AtiMY INTELLliA, CE Cb,41L + SCIIOUL ATTN: ATSI-ES

1 lEPAtFMLNT OF r'iL AIR IUHtt AIR UNIVERSITY LIWRAHY (ATC)

1 HQ TqAL)UC TRAIN1\K, LEVELUIPMLNI LNSTIIUTF

? riNIT H tMHASSY -441lSrl OLFL14CL. SIAFF

SCAJAuiIAN JOINT 'SI4H

I CULS (W) LWH3AKY

I Fkr.NLH ARMY Al IAC~i

1 AUSoIAN LMHASS)Y DEFENStL, MILLIANY AW1 AIN ATTACHE-

~I CANA&JIAN LII4ENILF LIALSUN olAF ATIN: LURINSLLORv OEFENCE P. ANU 0)

I POY~i. Nt-1FFLAN)t tM'ASSY MIt-lAHMY AITACH-E

I CAviAuIAN F(flCE.' dibE LURNAALLIS AITN: Pi-PNINEL SELLCTION

e CA.'AuIAN FOkLLt) PJLSUNNLL APPL KtiSf U'411

i AR4Y PENSUNNtL kLSLACH- Lt~lAaLISHMLNT

I NE1Ht.NLANUS ktft$A~SY 014 ltL U1 iIHL Alq ATTACHE

s~ LliR..RY OF CUNtNLSS EACtiANbE AIVUI bIH1 w1V

I J)ELIE..St TLC'INIL;L jINF UNKI 1' LL'W ATT~N: iT IC-LJVA-L?

14to LliRmRY 01 CUNur't.S UNII UULUI L04d t.l'itto(i PHOJL(CT

I US (3uVEkNMtNJ Pki'.j UtL LltKAKkY, P9L~iC UUCUMENrS D)EPAWlI .-- -

I US GuYEkNMENT Fod'J1IN~G UFL Llt<NARY A'JL STATUTORY, LIR DIV (SLL)

ITHE ANMY LII4HAKY ATTN: ARMY STUUILS StC

NUMHER %A ALURLtSSLEZS 2 ?.

To)TAL ojutjLN3t OF CLt'-S 4Ub

1