Technial Repr 418 LEVEK

C A COMPARISON OF EXPERT RATINGS OF ,

TASK DIFFICULTY WITH AN INDEPENDENT

CRITERION

"IIV

David L. Ryan-Jones

WTIC

EL ge

*i ARI FIELD UNIT, USAREUR

b U.S. Army

k - Research Institute for the Behavioral and Social Sciences

S- November 1979

Approved for public retse; distribution unlimited.

o80 18 008
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A field Operating Agency under the Jurisdiction of the -

Deputy Chief of Staff for Personnel

FIR-ANKN A. HMT~

JOISEIi ZEIDNR Colonel, US Aimy

Technical Directnt Commander

%CTICES

* ~ ~~'I P0t ho b~m mwM b, AAI1 P'..m 0gdrg corWor'fftz

A?" jPfP TV j ~ ~ Aex~r V~rgpns. 22333

S ~ ~ ~ %A* ,-O. 's ,,vb 0'PLr'o,, 000" ftO 104 "Wde P4Wm do not nt"f at t0

* - I .-. ~ et*~.~o 9 o'Q%. S .o, e"d SW.gd sconma

.tto0orvej0 wo'11 01^A\W " m ~~ - ' i~ w' ., t oo ~ OI~A OS Ce Oo Y otOAm t
Unclassified SECURITY CLASSIFICATION OF THIS PAGE (Ohm. Date _ _ __W_

RaD IUTUUCTIOU REPORT DOCUMENTATION PAGE Br= CjN7LqMc 7om

1. REPORIIT NUMBER 2. GOVT ACCESSION NO 3. RECIPIENTS CATALOG NUMIER

Technical Report 418

4. TITLE(Ma. w4itJ .. S. TYPE OF REPORT & PEmOO COVEREO

A COMPARISON OF EXPERT RATINGS OF TASK DIFFICULTY

WITH AN INDEPENDENT CRITERION& S. PERFORMING ORG. REPORT NUMBER

7. AUTHORINsJ 6S. CONTRACT OR GRANT NUM1E9ife)

V David L. Ryan-Jones

. PERFORMING ORGANIZATION NAME AND ADDRESS t0. PROGRAM ELEMENT. PROJECT, TASK

t. dUS Army Research Institute for the Behavioral RE ISoORaUNITiUnUER

and Social Sciences

5001 Eisenhower Avenue, Alexandria, VA 22333 ( 2762722A764

,t. CONTROLLING OFFICE NAME AND ADDRESS ____.0 ____--

Deputy Chief of Staff for Personnel November 1079

r!ashington, DC 20310 NUMBER OF' FAGES

10

14. 0071nIN AGENCY NAME & ADORIESS(iI di fferent from Controiliaj Office) IS. SECURITY CLASS. (of this report)

Unclassified

q / IS. DECLASSIFICATION/DOWNGRADING

SCHEDULE

I. OISTRIOUTION STATEMENT (of this Repott)

Approved for public release; distribution unlimited.

S 17. Do I GN STATEMENT (of the abstract entered in Block 20. If diffterenit Mrm Report)

IS. SUPPLEMENTARY NOTES

IS. KEY WORDS (Contia en rewer*e side it neesseary am dentify Iy block nomber)

Subjective judgement

Ratings

Training Management

Task Difficulty

Criterion-referenced testina SO. ASTRACT (Cenit e um evered. Os If aeo0. - 1W1t1W 6F Week 1mbr)

-0 This report evaluates the possibility of utilizing ?expertf ratings of

task difficulty as a substitute for actual measurement or determination of

task difficulty. Task difficulty information exists for only a small nuwber

of tasks in the US Army inventory of tasks. Accurate ratings of task difficulty would reduce the logistical problem of actually measuring difficulty

for each task in the inventory.

Expert ratings of task difficulty were compared with the results of a

DI . ." 1473 EDITION OF .0oV 6 1 OSOLETE Unclassified

ISCUITV CLA OICATM oTI. PA magiM, Sm. n.
7 E. "M.

Unclassified

89~CUOUTV CLASSFICATIOM OF This PAOCE4Uh Dad* Ealwaa

criterion-referenced test of the same tasks. There was a non-significant

correlation between the expert ratings and the independent weasure of

difficulty. There is a need for further research into methods of iuproving

expert ratings of task difficulty.

STA B

Dist E)C, al

I N . Unclassi fied

IiSECURITY CL.ASSI PIC ATIO11 OF THIS P AG9(hM, DO* Eftm
Techicael Roped 418

A COMPARISON OF EXPERT RATINGS OF

TASK DIFFICULTY WITH AN INDEPENDENT

CRITERION

David L. Ryan-Jones

John F. Hayes, Team Chief

Submitted by: William W. Haythorn, Chief

ARI FIELD UNIT, USAREUR

Approved by:

Milton S. Katz, Acting Director

ORGANIZATIONS AND SYSTEMS

RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue. Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

* Department of the Army

November 1979

I. .Army Project Number Training and Education

20782722A764

Approve for Public releae; distribution unlimited.
.7 -WJ I , r 7

AR! Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

iv

V.' lv

J- -**--- ,-
FOREWORD

The Army Research Institute for the Behavioral and Social Sciences

. (ARI) maintains a field unit with the U.S. Army, Europe (USAREUR) to

conduct research to meet the special needs of USAREUR and to evaluate

* other research projects and products under front-line operational

readiness requirements, with feedback leading to modification and

refinements.

Sustainment training is receiving increasing emphasis by the Army as the

only viable approach to maintenance of performance of critical combat unit

tasks. One of the requirements of the sustainment approach to training

management is a detailed knowledge of both the rates of degradation of performance over time and the time required to train an individual on critical

tasks. This information is ultimately related to some measure of task

difficulty.

This report evaluates the possibility of using expert ratings of task

difficulty to provide the data base on task difficulty required by the sustainment model. The research was conducted under Army Project 2Q762722A764,

"Training and Education".

Shnia Drcor

V.

.
A COMPARISON OF EXPERT RATINGS OF TASK DIFFICULTY WITH AN INDEPENDENT CRITERION

BRIEF

Requirement:

To evaluate the possibility of using "expert" ratings of task difficulty

to assess actual task difficulty.

i Procedure:

Expert ratings of task difficulty, obtained from squad leaders and platoon leaders, were compared with Skill Qualification Test (SQT) results for

soldiers at Skill Level One (SQT 2) and Skill Level Three (SQT 4). Additionally,

SQT 2 results were compared with SQT 4 results as well as SQT 2 results for

the SQT given five quarters later. The percent of soldiers missing the

written component of a particular SQT task was used as the independent measure of task difficulty.

Findings:

There was no significant correlation between the expert ratings of

0difficulty and difficulty as indicated by SQT results. There was a high

correlation between SQT 2 and SQT 4 results and between results for successive administrations of the SQT. The conclusion was that expert ratings

of difficulty may not be representative of actual task difficulty.

Utilization of Findings:

The findings suggest that it may be necessary to investigate methods

oZ improving expert ratings of task difficulty in order to derive ratings

representative of actual task difficulty.

Vii

, -. . ' - - a * -. 9
A COMPARISON OF EXPERT RATINGS OF TASK DIFFICULTY WITH AN INDEPENDENT CRITERION

CONTENTS

Page

INTRODUCTION ............................................................ 1

PURPOSE ................................................................. 1

METHOD .................................................................. 2

RESULTS ................................................................. 6

fDISCUSSION...............................................................6

REFERENCES .............................................................. 8

LIST OF TABLES

Table 1. Percent of soldiers missing task on SQT 2, percent of judgements

in each category, and weighted difficulty. 3

2. Percent of soldiers missing task on SQT2 and SQT4, and weighted

difficulty. 4

3. Percent of soldiers missing task on successive administrations

of SQT2. 5

I

IL

L.° ix

I I I --. --
A COMPARISON OF EXPERT RATINGS OF

TASK DIFFICULTY WITH AN INDEPENDENT CRITERION

INTRODUCTION

tTraining management in the US Army requires the training manager to

possess detailed information regarding individual tasks for planning and

scheduling purposes. This information must include task difficulty, which

serves as a guide for setting priorities when planning training and as

an indicator of the training time needed when scheduling training. The

most common method of determining task difficulty is to ask for "expert"

opinion regarding the difficulty of training an individual on a particular

individual task. The expert is usually someone, such as a first line

supervisor, who is thought to have an insight into the training process.

In the absence of empirical information, these experts must be considered to be making judgements about task difficulty under conditions

of uncertainty. This may lead to systematic errors of judgements resulting from erroneous intuitions about the nature of the factor being judged.

This means that the judges may have personal conceptions about what constitutes a difficult task, which may or may not be representative of actual

difficulty (Kahneman and Tversky, 1977; Tversky and Kahneman, 1977).

Expert opinion is rarely tested against an unrelated criterion measure of

the factor being judged. This must be done to insure that there is an

empirical foundation for such judgements of task difficulty.

PURPOSE

The research reported here was conducted to evaluate expert ratings

of task difficulty. Ratings of task difficulty represent a potential

data base for developing predictive models of retention for specific

tasks. Such a model is needed if a sustainment approach to training is

to be eventually adopted by Army battalions. The sustainment approach

requires that the training manager have access to the degradation rates

for all tasks on which the battalion must train. From this information

the training manager can determine the training frequency on each task

required to produce an acceptable level of sustainment.

Ratings of task difficulty, obtained from squad leaders and platoon

leaders, were compared with Skill Qualification Test (SQT) results for

soldiers in grades El-E4. The SQT is a performance-based measure of job

proficiency consisting of a number of tests of tasks which are constructed

using behaviorally derived scoring standards. The SQT may have a hands-on

component, a performance certification component, and a written component. I. The written component consists of a number of tests of tasks, each represented by a set of items designed to measure essential behaviors or steps

in performing the task. The exact nature of the SQT varies with the responsibilities of the military grade, and thus skill level, of the subpopulation being tested. Research has shown that performance on written tests

N. ... 1
correlates highly with the level of performance on actual performance tests

if the written tests are criterion referenced to the same set of criteria

as the actual performance test (Osborn and Ford, 1978). The SQT is criterion

referenced and each task on the written component is validated against actual

performance on the task (Maier and Hershfield, 1978; Osborn et al., 1977).

The results of the written component of the SQT should relate to task

difficulty, with more difficult tasks being missed more frequently than less

difficult tasks. This is indirectly implied in that the percent of individuals missing a question on the written component of the SQT is a direct

measure of item difficulty of the particular question (Steinheiser et. al,

1978). If the item is validated as discriminating between performers and

non-performers, then the item difficulty is related to task difficulty,

with performance on a set of items being a relative measure of the actual

difficulty of any particular task. The assertion that SQT results are

representative of difficulty might not hold if soldiers were trained to a

high level of performance on difficult tasks and a low level of performance

on easy tasks. This would normally not be the case, since one trial performance is the US Army criterion. Additionally, there is such a variety

of training methods and training priorities within the US Army that the

likelihood of a systematic bias in training would be small. This would mean

that in the best case all tasks would be trained to criterion, and in the

worst case there would be random training of tasks to criterion (Yates, 1979).

With this in mind, the written component of SQT2 was selected as the independent criterion against which the expert ratings of task difficulty were

evaluated. SQT2 was selected because of the large number of soldiers and

organizations represented, limiting the introduction of a systematic training

bias. Also, SQT2 is for low ranking soldiers who have been in the service

for a relatively short period of time. This helps to insure that overlearning on any set of tasks has not occurred, and that most soldiers do not

pass most tasks.

The real advantages of using ratings of difficulty as opposed to actually measuring difficulty or obtaining SQT results are numerous. There are

hundreds of tasks in the US Army inventory of tasks and the logistics of

collecting data on each task is enormous. Obtaining ratings from experts

is a relatively simple and low-cost process. The SQT results could provide

part of the answer, but there are no SQTs yet for many Military Occupational

Specialities (MOSs). Additionally, many of the tasks for any particular MOS

having an SQT are never tested, leaving the data base incomplete for the MOS.

METHOD

Judgements were obtained, during a previous research effort, from

sixty-eight (68) randomly selected squad leaders and platoon leaders in

mechanized infantry units regarding the difficulty of eighteen (18) Skill

Level One (SL1) individual tasks for El-E4 soldiers in MOS lIB. The judges

were asked to rate the difficulty of each task in terms of easy, moderately

difficult and extremely difficult (Bonner, 1978). These data were transposed

into a form more amenable to analysis. Using the values of easy=O, moderately

difficult=50, and extremely difficult=f00, the rated task difficulty was

2
V.

converted into a scale of 0-100, with the category frequency serving as

the weight for each value.

Each major command receives a quarterly SQT report listing the percent

of men failing to pass each task on the SQT. This SQT report was obtained

and data extracted for the same eighteen (18) tasks of interest for MOS lUB

soldiers in grades EI-E4, and from the same population of mechanized infantry

units as the squad leaders and platoon leaders. For each of the individual

tasks of interest, the percent of men missing the written component task

was taken as a direct measure of task difficulty. The assumption of this

report is that the less the task was missed, the less difficult t'e task.

The SQT data were derived from the written component results for 2,003

EI-E4s. Table 1 shows the category frequencies and weighted difficulty

for the expert judgements, and the percent missing for the SQT data.

TABLE 1

Percent of Soldiers Missing SQT2 Task, Percent of Judgements in

Each Category, and Weighted Difficulty

SQT2* -JUDGEMENTS**

%Soldiers Moderately Very Weighted

Task Missing Easy Difficult Difficult Difficulty

071-11A-0001 51 66 30 4 19

071-11A-0150 41 66 28 6 20

071-IIA-0502 26 49 41 10 31

071-11A-0511 42 66 24 10 22

071-11A-0704 25 38 43 19 41

071-11A-0705 63 44 43 13 35

071-11A-0703 47 51 40 9 29

071-IIA-0801 49 81 13 6 13

071-11A-0960 53 54 28 18 32

071-11A-2003 52 82 18 0 9

071-11A-2004 52 63 33 4 21

071-11B-2006 30 69 28 3 17

071-IIA-2104 46 50 43 7 29

071-11A-2401 58 57 33 10 27

071-IIA-1501 73 65 29 6 21

071-IIA-4402 69 71 23 6 18

071-11A-4502 37 65 28 7 21

* 071-11A-4503 25 37 40 23 43

*N=2,003

**N=68

-. 375, df=16, p > .10

3

rI

]'?,.
Since it is possible that the judges rated task difficulty in terms of

their own ability and personal experience with the task, SQT written component results were obtained for SQT4 (N=349). This is the SQT that a

squad leader would be expected to take. Although these data are for squad

leaders, similar results could be expected from platoon leaders since they

have a similar degree of competence on the tasks as squad leaders. This

similar competence is due to the nature of the branch training platoon

leaders receive as they complete their basic officer training. Therefore,

SQT4 results should be representative of task difficulty for both squad

leaders and platoon leaders. There were nine (9) written component tasks

in common between SQT2 and SQT4, and seven (7) written component tasks

in common between SQT4 and the difficulty ratings. These tasks, with

the percent of soldiers missing the tasks, and the weighted difficulty,

are shown in Table 2.

TABLE 2

Percent of Soldiers Missing Task on SQT2 and SQT4, and Weighted Difficulty

SQT2* SQT4*

% Soldiers % Soldiers Weighted ***

Task Missing Missing Difficulty

071-11A-0001 51 41 19

071-1A-0705 62 47 35

071-11A-0960 53 51 32

071-1A-2003 52 52 9

071-1IA-2104 46 38 29

071-1IA-2304 88 77 no data

071-1IA-2401 58 30 27

071-11A-1501 73 57 21

071-11A-4505 74 51 no data

*N=2,003

**N=349

***N=68

r=+.774, df=7, p < .02 (between SQT2 and SQT4)

r=--.264, df=5, p > .10 (between SQT4 and Weighted Difficulty)

I4

.

i4
There is also a possibility that SQT results are not stable in terms

of relative task difficulty. This could result from differences in the

nature or wording of particular questions on successive forms of the

written component of the SQT. As a check on this stability, the SQT2

results (SQT2A) which were correlated with the ratings of task difficulty

were compared with the SQT2 results for the SQT (SQT2B) given to the same

population of mechanized infantry battalions five quarters later. The

fourteen (14) tasks in common between the two SQTs and the percent of

soldiers missing the tasks are shown in Table 3.

TABLE 3

Percent of Soldiers Missing Task on Successive Administrations of SQT2

SQT2A SQT2B**

% Soldiers % Soldiers

TASK Missing Missing

071-11A-0150 41 34

071-11A-0502 26 25

071-IA-0511 42 36

071-11A-0704 25 29

071-11A-0705 63 60

071-11A-0703 47 41

071-11A-0801 49 48

071-11A-2003 52 49

071-11A-2004 52 75

071-IIB-2006 30 33

071-11A-2104 46 52

071-11A-2401 58 48

071-11A-1501 73 69

071-11A-4503 25 29

*N-2,003

**NWI,337

r=+.854, df=12, p < .001

I.

I Ii 5 . .. .. . ..
RESULTS

The percent of soldiers missing the task on SQT2 and the weighted

task difficulty, for each of the eighteen (18) tasks in Table 1, were

correlated by the Pearson method. The correlation was found to be

knon-significant (r=-.375, df=16, p>.10).

The percent of soldiers missing a task on SQT2 was correlated by the

same method with the percent of soldiers missing the same task on SQT4

for each of the nine (9) tasks in Table 2. The correlation was significant

(r=+.774, df=7, p< .02).

The percent of soldiers missing an SQT4 was correlated by the same

method with the weighted task difficulty for each of the seven (7) tasks

in Table 2. The correlation was nonsignificant (r=-.264, df=5, p>.10).

1% The percent of soldiers missing a task on the earlier SQT2 (SQT2A)

was correlated by the same method with the percent of soldiers missing

the same task on the later SQT2 (SQT2B) for the fourteen (14) tasks in

Table 3. The correlation was highly significant (r=+.854, df=12, p<.001).

DISCUSSION

The results indicate that the expert ratings of task difficulty may

not be representative of the actual task difficulty. The low correlation

(r=-.375, p>.10) between the weighted ratings of difficulty and the SQT2

results suggests that the squad leaders and platoon leaders may have been

guessing at the task difficulty without relying on some common conception

of jus+- what constitutes task difficulty. This is, of course, dependent

upon whether or not the written component of the SQT is representative of

actual task performance. The SQT is criterion referenced and validated

against actual performance, as previously mentioned. The high correlation

(r=+.854, p<.001) between two administrations of the SQT, as well as the

high correlation (r=+.774, p<.02) between SQT2 and SQT4, lends support

to this idea and indicates that is is unlikely that the earlier SQT was

unrepresentative of SQTs in general.

The possibility exists that the criteria for task difficulty utilized

by the squad leaders and platoon leaders does not apply to or is not represented by the SQT2. The weighted rating of task difficulty might then

be expected not to correlate with the SQT2 results, but to be representative

of difficulty with respect to SQT4 only. The low correlation (r=-.264,

p>.10) between SQT4 and weighted rating of difficulty suggests that this is

not the case.

The negative, though insignificant, correlation between the ratings of

task difficulty and SQT results for both inexperienced soldiers (SQT2, r=-.375)

and experienced soldiers (SQT4, r=-.264) is interesting, and is suggestive

SN. 6

Ii-.. .
of a trend. This apparent trend should only be attributed to a random

variance due to the nonsignificant r, but there may actually be a bias

in the ratings that result in the negative correlation. The most likely

reason for a trend, if such a trend exists, is that the raters tended to

rate all tasks as easy regardless of the relative difficulty. The fact

that many of the difficult tasks were rated as being easy suggests a

I lack of familiarity with difficult tasks.

The results do not imply that expert ratings cannot be used as

estimates of task difficulty, but only that new methods must be devised

to insure the use of relevant criteria for task difficulty across raters.

Crocker et al. (1977) suggests that one of the major sources of judgemental

uncertainty is a random and unpredictable environment which produces

random and unpredictable information, making it difficult for a judge to

discern an information pattern. This may be the case in the turbulent

military environment. One possible approach to future ratings of task

difficulty is to define for the rater just what factors or elements

contribute to task difficulty. However, there is a problem in defining

the components of task difficulty for the rater. Task difficulty can be

described in terms of task complexity, and can be resolved into functional

and process components. Functional complexity describes the number of subtasks comprising a task and process complexity the information processing

requirements of a subtask (Teichner, 1974). The relevant components of the

task described to the rater are inheretly dependent upon the task taxonomy

used to provide the terminology, and the level of detail required to enable

the rater to discriminate between tasks along the dimension of difficulty.

Judgemental biases of multidimensional complexity descriptions could possibly

be reduced through the techniques of sensitivity analysis (Fischhoff et al.,

1978).

47

4

I

I. -.

-" r - i i . . . " . . . . ... . . . .
REFERENCES

Bonner, B. A Survey of USAREUR Entry Level Skills for the 11B Infantryman

(Draft Technical Report). Alexandria, VA: US Army Research Institute

for the Behavioral and Social Sciences, April 1978

Crticker, 0., and Mitchell, T. Sources of Judgemental Uncertainty (TR 77-11).

Seattle, WA: University of Washington, September 1977. (AD A049 745).

Fischhoff, B., Slovic, P., and Lichtenstein, S. Improving Intuitive

Judgements by Subjective Sensitivity Analysis (PTR-1060-78-7). Eugene

OR: Decision Research, July 1978. (AD A058 998).

Kahneman D., and Tversky, A. Inituitive Prediction Biases and Corrective

Procedures (PTR-1042-77-6). Eugene, OR: Decision Research, June 1977.

(AD A047 747).

Maier, M., and Hershfield, S. Criterion Referenced Job Proficiency

Testing: A Large Scale Application (RR 1193). Alexandria, VA:

US Army Research Institute for the Behavioral and Social Sciences,

February 1978. (AD A055 664).

Osborn, W., Campbell, R., Ford, J., Hirshfield, S., and Maier, M. Handbook

for the Development of Skill Qualification Tests (SP-77-5). Alexandria,

VA: US Army Research Institute for the Behavioral and Social Sciences.

Osborn, W., and Ford, J. Knowledge Tests of Manual Task Procedures (in

HumRRO pp-4-78). Paper presented at the 19th Conference of the Military

Testing Association, San Antonio, October 1977. (AD A054 953).

Steinheiser, F. Jr., Epstein, K., Mirabella, A., and Macready, G. CriterionReferenced Testing: A Critical Analysis of Selected Models (TP 306).

Alexandria, VA: US Army Research Institute for the Behavioral and

Social Sciences, August 1978. (AD A061 569).

Teichner, W. W. Quantitative Models for Predicting Human Visual/Perceptual/

Motor Performance (NMSU-ONR-TR-74-3). Las Cruces, NM: New Mexico

State University, October 1974.

Tversky, A., and Kahneman, D. Casual Schemata in Judgements Under Uncertainty

(PTR-1060-77-10). Eugene, OR: Decision Research October 1977.

(AD A056 667).

Yates, L. The Estimated Impact of SQT on USAREUR Infantry Units: Surve?

Results (Draft Research Problem Review). Alexandria, VA: US Army

Research Institute for the Behavioral and Social Sciences, March 1979.

I8

1-. 8
DISTRIBUTION

AHt I,.rt~ i Lost

4 OASO (M&RA) 2 HOUSACOLLG. Ft I d. ATTN: Library

7 HODA IIIAMf CSZ1 I HOUSACOEC. Ft Orri. ATTN: ATEC EX F Hom Fartors

1 HODA lOAPF PORI 21 USAEFC. Ft Sienjamin Ht-iio. ATTN Llv.-,y

I flOFOA ITAMA ARtI I ISAPACOC, Ft Berian.rrHari.,,, ATTN A1CP HR

IHIOl'A IDA)'F HRE t'OI USA Cimin Elect SeliFt MonririruthATTN ATSN [AI

I H HUA A IDAMI SRllIUAF.tMoiiih DOT Ct 1 uSAFC. Ft Montmouthi. ATTN: TNAMLCTHP AMSEL PA 

1 HODA IDAP(: PM? A) I IJSAEC, Ft Monmerurt. ATTN AMSEL SI CH

I HODA IDACH PP?' A) I USAEC, Ft Monmouth. ATTN C, Fact Dev 8.

IHODA IDAPI HAFI I USA Materials Sys Aiial Agev. Abeidepti, ATTN AMXSY P

I NODA (DAPE MPU (j I Erlqewncrl Arsenal. Aberdeen, ATTN: SARFA IIlL H

I HODA IDAPn DWI I USA Did Ct. II Sch. Aberdeen, ATTN: ATSL TEM C

I HODA (DAPE HRL) 2 USA Hum Enqe Lab. Alii'eteeei, ATTN Lrlbrary'Dir

IHODA IDAPE CPS) 1 USA C-nilat Arms Tng Sd. Ft Bersinoil. ATTN Ad Sutrervisor SI HODlA (DAFD MFA) 1 USA Infinty Hum Fiscl. Unit. Ft Oenninq. ATTN: Chief

I HODA WOARD ARS P) I USA tInfanry 8d, Ft Senning. ATTN: STE8C TF T

I HODA IDAPC PAS Al 1 IISASMA. Ft Slls. ATTN: ATSS I.RC

I HOD IDUS OR)I USA Ai, Del Sch. Ft 81tys. ATTN! ATSA CTD MF

I1 it0tA IDAMO ROIO~ I LISA Air nlit Sdt,, FtI H81s ATTN Tact, Lili

I H00li/ IDIA(.1 I USA Ali Del Bd FtI RIr~s, ATTN. F ILES

1,0010 ID)AIOI 10Fl USA AirDe OcSd. Ft Blis. ATTN STEBO PO

(, fi Ci...... it liv IL'i 01501. ArWlit, MID I USA Cird & General Sit College., Ft Leavenwoith. ATTN, lI.

I '.1,1 A,,t H..-. Ate t)IH&I OAT) IF 8.151 I [ISA Cmd &General SriCotlrirte. FtLeavenworth. ATTN ATSW -SE I

II 10iSAf4AL APOS-t.iI. ATTN ARAGP a I UlSA Cirif& Gorwral St Crler. Ft Lcertiawit,. ATTN. EdlArierso,

I HO first Ariry.A FIN AF KA (ITI I I i"A C, -irri ,,ed At usi tCiebt Div Art. Ft Lierv tiwvo t1b. ATTN: Drep(t

2 NO F itt. Army, F t Sam Houiston 1 USA Combined Arms Cmnbt 0ev Act. Ft Leavenworth, ATTN: CCS

I It,, Amy Sit Sli~le, Oft. ATTN' OAVCSA (DSPI 1 USA Combined Arms Crvtbt Dan Act, Ft Leavenworth, ATTN: ATCASA

1 (11, t I ,f S1Ii~' Of, 1 USA Cirilirid Arms Cmbt Dcv Act, Ft Leavenworth. ATTN: ATCACO -E

Ii 1I'1Ft. At TN CP%0tCP I UISA Coveinr, d Aimrs Cmst Dev Air,. Ft Leavenwiorth. ATTN: ATCACC - CI1

I iii. Airiry 1 .1, ATTN HsH .tl I USAFCCOM. Night Vision Lati, Ft 0-hviir, ATTN: AMSFL-NV -SD

I Th.. Army LI., P itaw-ii ATTN ANRAL :1 USA Computter Sys Cid. Ft Belvoir. ATTN Tech Lobrary

I (it, Ast-ir itreAn RV I USAMERDC Fir evoir. ATTN4 SFSFR DO

I ti-i-t Suiptt n Of, OJCS 1 USA Erirrich, Ft Mevoir. ATTN- Library

)%tIASA, A, iiir,, ATTN tARt) T 1 IJSA Topographic Lab. Ft Belvoir. ATTN (IL TO S

I USA Rs~h li, Dun..n,At TN Le~tr S.-,irsDi I t ISA TistraptiI bati. Ft Belveir, ATTN STINFO Ceite

tr7IiSARIFM.Nl,,AfTN ';GRD UFCtA I LISA Tisiqrtrpic Lab, Ft Belvsrir. ATTN: E L 651

I 1V,At ir. ii, . Al Ift '.1111 MitA I ISlA l~-ttiCo*~r A cfi,. Ft Ibrvli~ A TN CTI) M101

II iAI MA.I linisi AlIN AISUtjtUtM I USA lrt,tewrsi Ct ASet. Ft Huirchiva, ATTN ATS-CT0-MS

F t USAIMA Fr Rivi AT TN M..ritiat Ifi t USA l..tel,,enc Ct. & Sch. Ft Hsuaet-ira. ATTN ATSI - TE

II ItS WAC Cr. &i f.i. FtI Mi-Cittari, AT rN Lit I U)SA lntettstience Ct. & Sch. Ft Huach,,ca. ATTN ATSI - TF X GS IS ) AC Ctr & Sr-I tFr McC~lellan. ATTN F,,,I Di I U!;A imietlvtr-nce Ct. & Sch, Ft Hsurhutic. ATTN: ATSI CTS- OR

I It)%A Ouafterm,ni ,rt .tfi te, ATTN ATSM TIT t IJ',A lntelliqence Ctt & Scb. Ft Huactica. ATTN ATSI- CTD OT

rrIrmneMt.iityit EWLFt Hrtali,d 1 USA Into. ltiqenci, Cii & Set.. Ft Huachuca. ATTN ATSt- Cli CS

I 0SA SE S.,p Itt tG-l,, ATTN ATtO FA I USA Initelligence- Cii & Set'. Ft Huachuca. ATTN DASISRO

I I 1'A Chap~.,.. 1. p.., t,, t, ATTN ATSC TF Fin I 0%SA lietellit..rscr Ct & Sch. Ft Huachic... ATTN ATM TEM

I U;A I10 1ii li fisis Al I N Fit,, Aii,sur 1 USA lirtr-ltqerce ii A Set.. Ft Huchrnj. ATTN I ibiarit

I lAWrcr rirI,.a ATNLi. I CDRt HO Ft HurstiiierA ATTN Tech Ret Div

WHAIR, 0.i.ii'., 2 CDR, USA ElectrIric Prey Ori. ATTN STTTP MT-S -

I) DI ',DA M. -. ,I HO. TCATA. ATTN Tedh Library

I rt'A Ci,ie-ii Aiji An, v Wthesictla. ATTN MOCA MR 1 HO. TCATA, ATTN- AT CAT-OPO0. Ft HWinrd

UA % Crow, tilo Airat AIiy Nentiasli. ATTN. MOCA JF I USA Recruiting Cmd rt Sheridan. ATTN USARCPU-P

iA Arcir, Test (,Ir. APU 'ie.vtIrl. ATTN STEAC PL fil 1 Spnior Army Artv., USAFAGO'ITAC. Elgirn AF Auxe Fill No 9

1 lISA Areric Test Ctr APT) Settle. ATT N AMSTE PbL TS I HO. USARPAC. 0CSPER. APO SF 9655. ATTN: GPWE St

tItSA Armament CirId Rertsmrore Arsenal. ATTN ATSK-TEM 1 Sitirvson Lib, Academy of Health Sciences. Ft Sam Hourston

II ISA Armamewnt Crrsd. Rock tslarnd, ATTN AMSAR TDC 1 Marine Corps Inst. ATTN: Doen-MCI

IFAA NAFEC. Atlantic City. ATTN Library 1 HO, USrIC. Commandant. ATTN: Code MTMT

1 F AA NAF FC, Arlantic Cryv ATTN Hurman I Nr Br I HO. USMC. Commandant. ATTN -Code, MPI 202X

I FAA Awenaurtical Cii (kIai,rna City. ATTN AAC 440 2 USCG Academy. New London, ATTN: Admission

21JSA FId Aty Sch, Ft Silt, ATTN library 2 UqCG Academty. New London, ATTN: Library LI lISA Arrir... Sit.. Ft Kiwe, ATTN Library 1 USCG Training Cit. NY. ATTN- CO IU I1SA Arenrrr st. rt K .... ATTN ATSR Of I 1 USCG Trining Ct. NY, ATTN Erluc Stc OftSI ISA Anrir- %,i i,1t V,-.rr, A T TN A ISR OT IP 1 MSCC. Psychust is St. DC. ATTN GP 1/62

1 USA Areir S, hiS fr .... ATTN AISRCUAI) I HO MidRllange OrMC Doi,Ouanreu, ATTN P&S Drv
1 US Marine Corps Liaison Of c, AMC. Al exandria, ATTN AMCGS-r 1 Oct &QCvil Inst of EnvroM Mediine, Canada

I USATRADOC. Ft Monroe, ATTN: ATRO-ED I AIR CRESS. Kensington, ATTN: Info Sys O

S USATRADOC, Ft Monroe. ATTN: ATPR - AD I Miletaeupeyktolgmdr Tfonees. Copenhagen

1 UISATRADOC, Ft Monroe. ATTN: ATTS-EA 1 Military Attache, French Embssy. ATTN: Doc Sec

I UISA Forces Crid. Ft McPhserson, ATTN: Litrary I Medfecin Chesf.C.E.R.PA.-Arunavl, TouonNival France

2 USA Aviation Test Sd. Ft Rucker, ATTN: STEBG-PO 1 Prin Scientific Off. Appi Humn Engr Rscti Div, Ministry

I A Ay for Aviation Safety. Frt Rucker. ATTN: Library of Doees, New Delhi

IUAAgcy for Aviation Safety. Ft Rucker, ATTN: Educ Advisor I Peirs Rich Ofc Library, AKA, Israel Defense Forces

IUAAviation Sch, Ft Rucker. ATTN: PO Drawer 0 1 Ministeris van Defenele, DOOPIKL Afd Soceaml

IHOUSA Aviation Sys Cmd. St Louis. ATTN: AMSAV-ZDR Psychoogitcher Zakren, The Hague, Nertherlnd

2 USA Aiatiou Sys Tntf Act.. Edwards AFB. ATTN: SAVTE--T

I USA Air DeO Srch, Ft lllis%, ATTN: ATSA TEM

1 UISA Air MiilsIity Rscli A Dims Lali, MoffettI Flit. ATTN: SAVOL -AS

1 USA Aviation Sch. Res Trig Mgt. Ft Rucker, ATTN: ATST-T-RTM

1 USA Aviation Sds. CO, Ft Rucker, ATTN: ATST-D-A

I HtO, DARCOM. Alesandria. ATTN: AMXCD -TL

1 HO. DARCOM. Alexandria. ATTN: CDR

I US Military Academy. West Point. ATT'N: Serials Unit

1 US Military Academy. West Point. ATTN: Ofc: of Milt Ldishp

1 US Military Academny. West Point, ATTN: MAOR

I USA Standardiiation Gp. UK. FP0 NY. ATT'N: MASE--C

1 Otc of Naval Rich. Arlington. ATTN: Code 452

3 Ofc of Naval Rich. Arlington. ATTN: Code 458

1 Ofc: of Naval Rich, Arlington. ATTN: Code 450

1 Ofe of Naval Rich, Arlington, ATTN: Code 441 lob I Naval Aerospc Mod Res Lab. Pensacota. ATTN: Acours Sch Div

I Naval Amostice Med Res Lab. Pensacola. ATTN: Code LSI

1 Naval sherosc Med Res Lob, Pensacota, ATTN: Code L5

1 Chiet of New~rs. ATTN: Pars-OR

1 NAVAIRSTA. Norfolk, ATTN: Safety Ctr

1 Nair Oceanographic, DC, ATTN: Code 6251, Charts & Tech

I Center of Naval Anal. ATTN: Doc Ctr

1 NavAmrSvsCom, ATTN: AIR--5313C

1 Nov BuMed. ATTN: 713

1 NareompterSubiquis 2. FPO SF O560

I AFHRL fFT) William AFS

I AFHRL (TT) Lowry AFS

1 AFHRL (AS) WPAFB, OH

2 AFHRI. EDOJZ) Brookts AFB

1 AFHRL (00.14) Lactrlarid AFB

1 HOUISAF (INYSD)

1 HOUSAF EDPXXA)

1 AFVTG (RD) Randolph AFB

3 AMRL (HE) WPAFB. OH

2 AF Inst of Tech, WPAFB, OH. ATTN: ENE/SL

1 ATC (XPTDI Randolph AFB

1 USAF AeroMeri Lih, Brooks AFS fSUL- 4). ATTN: DOC SEC

I AFOSR (NI. Arlington

1 AF Log Cmd, McClettlm AFS. ATTN: ALC/DPCRS

1 Air Force Academry, CO. ATTN: Dept of Bel Scn

5 Navers & Dev Ctr, San Diego

2 Navy Med NeuroPsychistric Rach Unit, San Diego

1 Nav Electronic Lab. San Diego. ATTN: Res Lab

1 Nov Trigren. San Diergo, ATTN: Code 9000- Lib

1 NavPostGraScti. Monterey. ATTN: Code 55A@

1 NavPostGreSch. Monterey, ATTN: Code 2124

1 NavTrngEquipCtr, Orlanido, ATTN: Tech Lib

I US Dept of Labor. DC, ATTN: Manpower Adin

1 US Dept of Justice. DC. ATTN: Drug Enforce Admin

1 Nat Our of Standards, DC. ATTN: Consputer Info Sectioni

1 Nat Clearing House for MH-lnfo. Rockrville

I Denver Federal Ctu. Laktewood, ATTN: IBLM

12 Defense Documnentation Center

0 4 Dir "sch Army Hq, Rusell Ofics. Canberra

I Scientific Admir, Mil Sd, Army Hq, Ruseell Ofca, Canberr

I1 Mil and Air Attache, Austrian Embassy

1. ~1 Centre do Recheirchet Des Facteirsn. Humnalne de ta Defene

I Nationale, Brussels

2 Canadian Joint Staff Wshington

I I CIAir Staff. Royal Canadian AF, ATTN: Pare Std Anal Sr

3 Chief, Canadian Def Rich Staff. ATTN: C/CftOSMW

4 Brtish Oct Staff, British Enubase, Washingtoo

10