If;.

Technical Oaper 384

ENHANCING qUALITY CONTROL IN THE TESTING

OF MILITARY APPLICANTS.

/ M. A./Fischl Md Robert M./fRoss

PERSONNEL UTILIZATION TECHNICAL AREA

DTIC f~JD~iE S E.CT E)

4 >U. S. Army

.. J Research Institute for the Behavioral and Social Scie. es

" /AprfliSrl , (_

Approved for public release; distribution unlimited. 1/ / 6 /

p

* f:5li

,.'. . l l
U. S. ARMY RESEARCH INSTITUTE 

FOR THE BEHAVIORAL AND SOCIAL SCIENCES 

A Field Operating Agency under the Jurisdiction of the 

Deputy Chief of Staff for Personnel 

JOSEPH ZEIONER 

Technical Director 

NOTICES 

FRANKLIN A. HART 

Colonel, US Army 

Commander 

D ISTRI BU TIOI\i Ptom8r y drsHrbut•on o f thos r~por t h .. been m1~ by ARI Pleew eddren correspon~n;t 

Cl)nCernong d ostrobutton o f reports to U S Army AltelrCh lnstolute for the 8eh8Yrorel 1n d SOCrll Screne91, 

ATTN: PERl TP. 5001 ErsenhoWI!r Avenue. AleJCandrra. V irgin ra 223;,3 . 

.:.F.~.:IN.!.;A:::,L. ...... Pu...Sii!P:.IrOt,~StLITu!J.O~N Thrs ••POrt mey be destroyed when ot rl no 10091r needed. Pie•• do notl'9turn rt to 

the U S Army R .. erch lnl!rlute for the 81h8Yrorel end Socrll Scrences. 

~ The ftnd•ngs .,, th 11 repa t r ere not to be construed 11 en offrcrel Olpertment of thl A rmy POI•tron. 

un leu to delrQnlted by other euthorrzed documenu. ·, 

t 

jl 

i\ 

"~ 
Unclassified

SECURITY CLASSIFICATION OF THIS PAGE (hen Data Entered)

DOCUMENTATION PAGE READ INSTRUCTIONS REPORT DBEFORE COMPLETING FORM

1. REPORT NUMBER 2GC IT ACCESSION NO. 3. RECIPIENT'S CATALOG NUMBER

Technical Paper 384 , [ / 4. TITLE (and Subtitle) S. TYPE OF REPORT & PERIOD COVERED

ENHANCING QUALITY CONTROL IN THE TESTING OF

MILITARY APPLICANTS 6. PERFORMING ORG. REPORT NUMBER

7. AUTHOR(.) 8. CONTRACT OR GRANT NUMBER(.)

M. A. Fischl and Robert M. Ross

S. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT, PROJECT. TASK

U.S. Army Research Institute for the Behavioral AREA & WORK UNIT NUMBERS

and Social Sciences

5001 Eisenhower Avenue, Alexandria, VA 22333 2QI63101A768

11. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

Deputy Chief of Staff for Personnel April1980

Washington, DC 20310 13. NUMBER OF PAGES

8

14. MONITORING AGENCY NAME & ADDRESS(If different from Controlling Office) IS. SECURITY CLASS. (of this report)

Unclassified

1S.. DECL ASSI FICATION/DOWNGRADING

SCHEDULE

16. DISTRIBUTION STATEMENT (of this Report)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of the absttact entered In Block 20, If different from Report)

IS. SUPPLEMENTARY NOTES

Based on a presentation to the 19th annual Military Testing Association

Conference, 1977.

19. KEY WORDS (Continue on reverse aide If necessary and Identify by block nurmber)

Test verification

Measurement error

Armed Services Vocational Aptitude Battery (ASVAB)

20. A"ACT teia;rs..1IFfn AfMW wn"IdaUilybboknubf)

Diverse sources of error must be controlled for aptitude tests to have " substantial validity. This paper describes a highly cost effective procedure for immediate verification of the veridicality of operational test

0 scores.

A continuous need exists to maintain the high quality of testinq procedures and of operational test scores used in selecting and classifying

5ji (Continued)

DO J 1473 ."'.O P o s IS OOLETE Unclassified

" SECURITY CLASSI1FICA T

ION OF IrmS PAGE (When Dl Entered)

t J
Unclassified

SECURITY CLASSIFICATION OF THIS PAGE(Whtn Date Entered)

Item 20 (Continued)

nlisted personnel. In a large-scale testing program such as the one that

uses the Armed Services Vocational Aptitude Battery (ASVAB), the risk of

test compromise is always present. A cost effective procedure for detecting

the incidence of spurious scores was developed, consisting of (a) comparison

of scores on two ASVAB subtests to detect any large differences between

them, (b) administration of a 10-minute retest to examinees showing the

large difference, and (c) comparison of original and retest scores to verify

the incidence of likely test compromise. Tryout of the procedure indicated

that the 10-minute retest of fewer than 20% of all examinees could detect

approximately 70% of all cases of test compromise.-

Accession For

NTIS GRAI

DTIC TAB 0

Unannounced

Justification

By.-

iAvaiiiility Codes

iAvail and/or

D Special

iUnclassified

SECURITY CLASSIFICATION OF THIS PAGE(Wihen Des Entered)

6r9
Technical Paper 384

ENHANCING QUALITY CONTROL IN THE TESTING

OF MILITARY APPLICANTS

M. A. Fischl and Robert M. Ross

Submitted by:

Joyce L. Shields, Chief

PERSONNEL UTILIZATION TECHNICAL AREA

Approved by:

E. Ralph Dusek

PERSONNEL AND TRAINING

RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel Department of the Army

April 1980

Army Project Number Enlisted

20163101 A768 Selection

Approwd for public release; distribution unlimited.

- - -I-,..
~ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

ivI

:o1

j1 i I
FOREWORD

This research is related to ongoing ARI efforts to make the most efficient use of the Armed Services Vocational Aptitude Battery (ASVAB) for selection and classification of enlisted recruits. The report describes a

quality-control technique that can be used to detect cases of possible ASVAB

compromise. The technique is based on psychometric properties of ASVAB subtests. The ASVAB research is responsive to requirements established by the

Deputy Chief of Staff for Personnel, Department of the Army, and was conducted under Army Project 2Q163101A768.

i~

OEftZ ER

~hnical Director

' r

9i

I

- _' . .. . r. " ' "

. ... -.
ENHANCING QUALITY CONTROL IN THE TESTING OF MILITARY APPLICANTS

BRIEF

Requirement:

The Armed Services Vocational Aptitude Battery (ASVAB) is the principal

test battery used to select and classify recruits. A test cannot provide accurate information if it has been compromised (that is, if an examinee learns

the questions and answers in advance). This research provides an operational

method of detecting a substantial proportion of individual cases of ASVAB

compromise.

Strategy:

The basic strategy analyzed the statistical relationships between the

ASVAB subtest most vulnerable to compromise and other subtests, using ASVAB

scores from 1,000 enlistment applicants nativn-ide to determine the range of

normal and abnormal score patterns. The Word Knowledge (WK) subtest is the

most likely to be compromised, in part because vocabulary words are fairly

easy to remember, look up, and discuss. The Arithmetic Reasoning (AR) subtest, on the other hand, is not easily compromised. Most people score in the

same range on both tests. Cases in which the WK score is more than 10 points

higher than the AR score are suspect. Retests with the 10-minute WK subtest

from the 1973 Army Classification Battery (ACB-73), which is no longer in use

for the active Army and therefore unlikely to be compromised, show that a

difference of ii to 14 raw score points between the two WK subtests would

confirm cases of test compromise.

Field Tryout:

Several months after the nationwide data collection, a sample of 111 enlistees whose ASVAB scores had been recorded in that collection were retested

with the ACB-73 at the Fort Jackson, S.C., Reception Station. Comparing

their recorded ASVAB WK and AR scores flagged 20 cases; comparing the ASVAB

WK and ACB WK scores for these 20 cases identified 9 as highly suspect.

Both sets of WK scores were then compared for the entire sample, and 13

highly suspect cases were identified in all. That is, retesting 18% of the

sample (20 out of 111) identified about 70% of the compromise cases (9 out

of 13).

Utilization of Findings:

This quality-control procedure for aptitude testing is highly cost effective because of its simplicity, short testing time, and screening

effectiveness.

vii

1 - W ?. ii •
ENHANCING QUALITY CONTROL IN THE TESTING OF MILITARY APPLICANTS

CONTENTS

Page

INTRODUCTION ............. .............................. 1

APPROACH .............. ................................ 2

IMPLEMENTATION .............. .. ............................. 2

EMPIRICAL TEST .............. .. ............................. 4

SUMMARY AND CONCLUSIONS ........... .. ........................ 5

D'STRIBUTION ............... .............................. 7

LIST OF TABLES

Table 1. Statistical description of AFQT subtests of ASVAB-6 .. ..... 3

2. Statistical description of Word Knowledge subtest

in ACB-73 (WK 2) ........... ....................... 4

3. Results of empirical test ......... .................. 5

ix

j.__ __ _.____ ___ " -" ... . .... .
ENHANCING QUALITY CONTROL IN THE TESTING OF MILITARY APPLICANTS

INTRODUCTION

All testing is subject to influences, lasting and temporary, general and

specific, that cause the aptitude test score an individual attains to vary

from the theoretical true score. For purposes of prediction in selection and

classification through the use of testing, all reasons that would increase

this variance over a group may be considered error.

Such semipermanent influences as the ability to deal with instructions

on tests, or general examinee strategies for answering test questions, vary

widely with individuals. The services have used several methods in attempts

to reduce error attributable to this "test wiseness." Instructions are easy

to understand and are targeted to low levels of reading ability, and sample

test items and sample instructions are provided in an information pamphlet

intended to familiarize everyone concerned with the nature of the test.

Temporary influences on test scores may also affect measurement. A person's physical and emotional condition and the physical testing environment

may cause variation from true scores. To reduce these temporary effects that

add to measurement error, care is taken to excuse from the testing session

persons who are clearly ill or excessively fatigued, or persons who are disturbing others; regulations prohibit testing for long periods without breaks,

or testing in places without proper lighting and temperature conditions.

Scoring and recording errors occur either as transitory human errors

or, at times, as semipermanent conditions when, for example, an undetected

malfunction develops in equipment used to score tests. Generally, the variety of scoring aids now used in Armed Forces Examining and Entrance Stations

(AFEES), including optical scanning equipment, not only reduces errors but

saves time as well.

Another source of measurement error is test compromise. These measurement errors, rather than being randomly distributed, usually operate in one direction--to yield overestimates of qualifications. Although compromise

probably would not affect the measurement of very large numbers of enlistees

as could other measurement errors, its nonrandom character makes test security of great importance.

In the past, the most common means of coping with test compromise has

been by use of alternate test forms. There are two types of alternate forms,

and they differ in cost of production and in the kind of protection they provide. One type uses the same items, but arranged in different sequences in

different test booklets. This type remedies situations in which the compromise has taken the form of examinees being provided with a key to the correct

answers, but not the content of those answers (for example: la, 2c, 3d, etc.).

This type of compromise is believed to be relatively uncommon. The other type

of alternate test form is very much more costly to produce but also very much

more comprehensive in its protection. It consists of two tests with similar

(but not identical) content, matched in difficulty and other statistical

properties. The protection afforded is not just for cases that include 1) I .,. -
applicants having the key, but for applicants having the full answers to one A

of the forms. Both of the two types of alternate test form are now used in

the test quality control programs of the services.

The parallel forms approach provides reasonable protection, but at an

extremely high cost of production. That approach also does not, in and of r itself, identify cases of suspect scores.

This paper describes an alternative approach to test quality control

that involves minimum test development costs as well as minimum examining

time onsite.

APPROACH

The objective of this development was to provide an operational tool to

detect a substantial percentage of enlistment qualification test compromise

cases. The general strategy was to capitalize on ;hat is known or can be deduced logically concerning the differential compromise vulnerability of the

various parts of the battery (ASVAB), and to combine that information with

known statistical relationships among the subtests so as to "flag" highly

unusual score patterns for subsequent followup.

Operational experience has shown that the main target for compromise

has been the AFQT portion of the test battery. AFQT has been in joint services use the longest; for some of the services, AFQT is the principal selection standard. The nature of its contents--vocabulary, arithmetic problems, and geometric figures--are generally the best known of all military

te Sts.

Within the AFQT portion of the battery, experience has indicated that,

if compromise takes place, the compromise usually involves the vocabulary

items. This is not surprising because vocabulary words are easy to remember

and to look up after the examination. The other two subtests do not lend

themselves to this kind of compromise: the arithmetic problems are relatively

long prose paragraphs, and there is no readily available source of the right

answers; and the totally pictorial test of spatial relations is nearly impossible to compromise through memory.

-Given (a) that Word Knowledge is probably the key ASVAB subtest compromised, that (b) the other components are relatively hard to compromise, and

that (,:) the psychometric relationships among these subtests are stable and

known: Likely compromise can be detected by comparing discrepancies in

score between the Word Knowledge subtest and one or both of the other AFQT

components (Arithmetic Reasoning, Space Perception).

IMPLEMENTATION

The numeric values needed to begin to implement the logic of this approach were derived from a national sample of 1,000 AFEES applicants drawn

in January 1976. These 1,000 cases were stratified on AFQT to conform to

the standard mobilization reference population, and the statistics shown

in Table 1 were obtained. As may be seen, the correlation of Word Knowledge

4 (WK) with Space Perception (SP) is 0.43. This means that fairly sizable

2 I

'p 'A
_ - -

score discrepancies between WK and SP can be expected just by chance. On the

other hand, the correlation of WK with Arithmetic Reasoning (AR) is high

enough to be usable, 0.68. As a result, development focused on use of the

WK/AR discrepancy.

Table 1

Statistical Description of AFQT Subtests of ASVAB-6

Correlation

Subtests Mean SD with WK

Word Knowledge (WK), 30 items 17.5 7.5 -

Arithmetic Reasoning (AR), 20 items 11.7 4.8 0.68

Space Perception (SP), 20 items 10.3 4.1 0.43

Standard error of estimate of WK on AR 5.5

Regression line: WK = 5.07 + 1.06 (AR)

Note. N 1,000 AFEES applicants tested in January 1976.

The intention was to develop an initial screening procedure that would

"flag," as suspicious, cases in which the WK raw score exceeded the AR raw

score by some amount greater than chance expectation. The regression line

of WK on AR can predict the expected WK score from any AR score (Table 1).

The prediction has confidence limits defined by the standard error of estimate of WK on AR and the confidence level selected. A somewhat low one-tailed

confidence level of p < 0.80 was chosen in consideration of maximizing detectability for subsequent followup. Using the regression formula and standard

error of estimate it was found that a difference of 10 raw score points between

WK and AR is unlikely to occur by chance, i.e., outside the confidence interval. The 10-point difference is appropriate through the score range AR < 15

since the regression coefficient was so close to 1.0. Therefore, 15% to 20%

of cases that have a difference equal to or greater than 10 points and with

AR less than 15 are flagged as unusual cases. These cases will be the only

ones used in further screening for possible compromise detection.

A group exhibiting the unusual score pattern consists of two types of

individuals: (a) those for whom the abilities measured by the WK subtest

are truly well in excess of their abilities in the domains measured by AR,

and (b) those whose WK scores are artificiaily inflated through some breach

of test security. The next step, then, is to separate these types.

The simplest way to sort the compromise cases from the genuine, though

unusual, ones is to administer a 10-minute retest consisting of WK items

known to be secure, and to compare performance on the WK retest with performance on the original WK. For some cases, the original WK score (WK 1)

will replicate, plus or minus a calculable chance error effect; for others, )

I3
the second WK score (WK 2) will be so much lower as to be virtually unexplainable through normal chance variation.

Just as the initial screen utilized the values shown in Table 1 to define the critical WK/AR difference, values in the same table plus those in

Table 2 were used to set the limits for the WK I/WK 2 difference. For this

step a confidence level of p <0.95 was set to minimize the risk of false accusation and to identify cases virtually unexplainable by the hypothesis of

chance variation.

Table 2

Statistical Description of Word Knowledge Subtest in ACB-73 (WK 2)

Correlation with

Number of items Mean SD ASVAB-6 WK 1

20 11.8 4.6 0.76

Standard error of estimate of ASVAB-6 WK on ACB-73 WK = 4.87

Regression line: WK 1 = 2.88 + 1.24 (WK 2)

A difference of ii to 14 raw score points (depending on the level of the

WK 2 score) between WK 1 and WK 2 is the critical difference, i.e., beyond

that point score differences are probably not due to chance. Individuals exhibiting a larger difference are identified as most likely having received

improper pretest assistance.

EMPIRICAL TEST

In the spring of 1976 a sample of I1 enlistees who had been tested with

ASVAB at AFEES was retested at the Fort Jackson, S.C., Reception Station with

ACB-73. ACB-73 contains WK and AR suLL-<ts and was the Army's basis for computing AFQT scores until it was replaced by ASVAB-6 and -7 in January 1976.

At the time the test sample was drawn, ACB-73 was no longer operational, and

hence its WK subtest could be considered as completely secure.

4 The first step in the test was to calculate the one-sided difference of

4 ASVAB-6 WK minus AR and to refer it to the specified critical difference of

10 points. This step identified 20 cases.

The second step was to calculate the one-sided difference of ASVAB-6 WK

minus ACB-73 WK and refer that difference to the specified critical difference of 11 to 14 raw score points. This procedure identified 9 of the 20

flagged cases as highly suspect compromise cases. These and other important

relationships are summarized in Table 3. As may be seen, when the retest

scores of the entire sample were examined, 13 cases were identified as highly

Li Af 4
suspect. Under operational conditions, only 18%--the 20 flagged cases--would

have been retested, and only 9 of the 13 highly suspect cases caught; that

is, retest of less than 20% of the sample caught about 70% of the compromise

cases.

Table 3

Results of Empirical Test

Flagged by WK-AR Passed by WK-AR Total

"Clean" 11 87 98

Highly suspect 9 4 13

T... 42 91 i1

A final empirical test was performed to assure maximum certainty of the

percentage of the input which would have to be retested under the rule of

WK-AR 1 10 points. It may be recalled that 10 points implements a confidence

level of 0.80--i.e., about 15% to 20% of the population flagged for retesting--

and one sample, at Fort Jackson, yielded 18% so flagged. In mid-1976, another

sample of AFEES data was drawn, of size 500, and the WK minus AR criterion

was again applied. Results in this sample flagged 17% of the cases.

SUMMARY AND CONCLUSIONS

In recognition of the fact that the Word Knowledge subtest is the most

vulnerable to compromise of all the tests in the selection and classification

battery, a simplified procedure was developed to detect WK compromise. The

procedure has two steps:

1. At the time of scoring the AFQT portion of the battery, separate

those papers in which the AR raw score is less than 15, and the

WK raw score is 10 or more points greater than the AR score. This

step will flag, as potentially suspect, some 15 to 20% of the cases.

2. To only those flagged by step one, administer a 10-minute retest

consisting of a completely secure WK and separate those papers in

which the WK retest score is at least 11 to 14 raw score points

lower than the original WK score (checklist tables can easily be

prepared to accomplish all conversions and all comparisons with

critical differences). This combination of steps will identify,

as highly suspect, approximately 70% of all cases of likely test

compromise.

.!f 5
An alternative to the two-step procedure is to administer the WK retest

to everyone and apply the rule of an 11 to 14 raw score point drop. This

will detect more compromise cases, but at five to seven times the cost (that

is, retesting 100% of AFEES applicants instead of between 15% and 20%).

Another alternative is to enlarge the requisite WK/AR difference so as

to retest 10% of the input. In the Fort Jackson sample, this detected about

40% of the likely compromise cases.

For any of these alternatives, the conclusion may be drawn that a simple

and cost effective procedure for enhancing quality control in the testing of

military applicants has been developed.

A..

. . . . .. .. . . . . . . . . l I I I I . ...
DISTRIBUTION

I US ARMY CINCPAL S)JPPukf UMOUIl VLkbONJLl- ULVISLON

I 110,1A ATTN: PMAMC

I TA6/IAUCtN ATTN: UAu(6-LUi

I HU. ICLLTA ArIN: AICAI-UP-U

I HQ- UbmFYCOM ArINi: MLHLI-H

e HU0A WLSLAkCH sAND STUUIL OFC

1 MILITAPY UCCUPAIA)NAL ULVLLOFPMLNT UIV UAPC-M$P-0, RM 85dL HIOFFM4AN BLOG I

I HQi)A I)IRLCTUH9 P.-ANS), P UHAMS ANLD PJLICIES

I MI1) A ATTN: DAMU-k.,i

I HUI)A (:HIALF , HOUIM\J' H UUNtL~S ULVLLUPMtNT uIV

I -IUiJA AT114 : DAMT-IT

I ISA AVIATION SYS)IEMS COMUl Ali: UHSAV-Z(iR

I PACT RFLAIJUNS A14 LuUUL QvIPUkTUNIIY dHANLH ATTNV: AF7T-PA-dtR

I WSA AHWAULOM MIN: ATFE-LIU-AL

I HE'WUUAkTERS Ob r-IARINt LUMP~S ATTN: C)Lk MIMI

I rlEpIUUARILRb, U--, 'lAN±NL LUMPS AIIN: CUu MPA-e8

C LIS ARMY EUROPE, ANj hLVtidN ARMY

I 1ST INFANTRY O1VlhjON ANU I-T. kil-Y A]TrN: AAF2N-DIT.T

I CMLLF9 SUkVE! ONMAJLh' ATIN UAwL-M-S- HoFFMAN OL~b 11

i UJSA INTtLLI(,tNCL ANO) SLLUKITY CUMMANU ATTN: IAOPS-TNG

e H-o, rNA;'uL TtCri hiAL LIbHAHY

IMILIARY OCCUPAfTUNAL UtVLLOPM'ldI UIVl ATTN: TN:AP-SS. UAP C-MSP-D, HM 852C9 B2 HOFFMAN UFANLb 8LUGILTYUCPA)NL

I

I MILITARY UCCUPAJILNAL LVtLUl~mt-1T I JV ATTN: UAPC-MSP-T, KM 85C HOFFMAN 8LO6 I

I USA ONI)NANCE * CM:-MILAL LtidTLi, ANU SCTIUUL ATTN: ATSL-DTC-P

I HU0A IAN"' FOHLi-b MANMLLMLNT Uo.L

A H Qo)A ATTN: UA.'>-;)Tb

1 1230. USARLom1 HLtNVt CL.N~tR

I F r. t~tt\iJAMIP J j~I~ IN 4btflb

I UJSA FUWCLS CoMMs.'J) A~ek - ULPTY LIIIEF OF ST0F FOR IPERSUNNLL

I IT-t ANmY TRAINI,\LJ COMMANU

I 1 UI'00H~A1E OF 1k~laiANIN ATTN: AlLU-i

i l)A4'LoHAIF OF CUvrAtil IAvtLuPmi-iNTS Ai',: ATZA.I)

I MUs)ANom P-ARINt COMPS, LIMI5UN U L

I LLA'HTr4ENl OF ],,t AR~MY US AkMy INTEL616LNCE * SLCURIIY COMMAND

A ANi ,TnN MNMY LOll.JI Urd'MNlLAIIuNAL EFFLCTIVLNEStS

I ANTADS AJIrJ: ULLM-ITV>-II

A tJS. F UwLLS) CUM-,A'4J

A PM IHA,)E ATTN; UCPM-lNU-HL (,iH. LHNtlO-1LM)

I US MILrIAKY UlSTIALT OF *MA-HN(,IUN OFL OF EUUAL QPFURTUNITY

A NAVAL CIVIL IAN Pt-i5UiNNILL kuMU oUUTHER~N FLU UIdV

2U AR! LIAISON OFf l(.;

I lSiCL * ATTN' Al;C-L.A-t hUMAN tFALIOMS

I SACNRAMFNTO ALL/QPCNRs

I 114TH-NIV SEmijAA-( 01\ AHMLU F'ULt-t + SUC

A COjS0) (H ANIJ U) tuEF'UI t10R SCAlrNLL ANi) TEC.MNULUGY

A UC NAAL O L ACHATT(N: LUL455 (OR. MARITIN A. TOLCUTTI

A APtIL/TT

I APHRIL/Ab £~A. oRLMJ4MJMAUNL ~ TTN: RL

I I

1tAy PFWUNNLL H w L) LLNi ATTN: (LoIOL 30b) tDMUN) U. THOMAS

i NAVY PFNSUNNLL k ANU 0 CLNIFJH uiRLCTJH OF PROGIRAMS

I NAVY PFNSONNLL R ANI) 1) CLNI4(Fm ATTN: UN. H. HIMLANU

e UFc OF NAVAL RL :LAHCH PLNSONN.L ANt) INATNING RESEARCH PROGhAMS

I OFC. OF NAVAL HLSLAHCri Abbl. uNltLIUH PLWS +TRAINING RSCH IPROrjS

I oFr OF NAVAL kt5CH (,PNIIUTIUNAL LFFLrfVENLSS PRO.

AI NAVAL AFLNOSPACL MECLICAL Hak.H LAO ATTN; (COUL L51)

I NAVAL 'LHUSPACL mEUICAL NS1LH LAO AIF~4URNt RANGER RESEARCHI

I HU4LAU OF NAVAL PEHSUNNLL SLLAF,411IC AUVISUR (PER$-UR)

7

IZ7s7~ 7. 
INAVAL ~afUHPACL - 'u1LAL HSL?1 L40I AER)SPACE PSYCHOLOGY LPARTME141

i "JSA IIIAUUI. -llrM1S ANALY .I-' 4 (jVIIY ATJlj: A7AA-TCA

I HIAuUUA~lt)~, (LoA' QUAHU ChII-rw PSY H$ULOGICAL HSCH HP

I 1SA tN(;INL#-k TUP-U ,PHAI, LAbS AM,.~: .IL-.Tll- i

i OSi M4U-AlLIrY 'J-~1H milol b LUMU ATTN: UHUt4E-TU

I1 USA 1HIN "IN(, iU, H -)

I USJ IATUWIFL SYT~AWAsLT I1S .-IlIY AITN: UtRASY-M

I NAtI-C HUM~AN~ -0d.,I'tjEjN~,7 tHANL~i

I HAI IL~i tz-Lo(tu,1dLjb LAtUIHIUH1L S IALTI.ML rECh-NICAL UFC

I USA AH~CCI I'li~~ Alliv AMSTL-I-L-TS

I obA AiclicIC tsl C;-N AInN:* SIL,%L-IL-M1

I US.1 LCLcT-rS AIVALY5IS W:,0Y Al 104 MOCA-c,

I HkJ VJAIR 01V Ut-\t l.rr~,(

I U'ACALoA Al ,n: AILLLA-CI1-A

I LJSACAUI A A7lJ,: NILL-A-CA

L 11SA tLfCTK0flNC vl~-A~ L~4" 4tifLI't J1TLLLAIft.NCt MATER~ fEVtL + SUPP ul'F

IUS,, IH()PIL ri Sl CZNTtW At IN: ILLHNt-ICAL LISRANY

I tiIS, N4SCH L*VEL + SIANUMNULLA W9U .

I NA~VY Po-NSUNNI-L (<'):e #- (JtI~L ( jltH At IN: (CUut J307)

I UJSA NAT iCr AL~t NLU uLVE.LU#-MLNl CJMmat)D CHILFo 6EHAV SLILNCE~l DIV9 FUOD SCI LAS

I ()ALU. F AN)i L It- 4NU L :) mILITAM)' ASSI I-P INUi # PRS TECHNOL

I "W~toA 4T1)8 (4'-1 Llu MJCHA,4U t-. IAH17ELL)

I UbAAHL LIMAHY

I HIJ4AAN wlt U1 NCL-j Ht(.Cf Uhlo CnUM)- gU LIJHARY

I s tvLL NSt1ANqh U-'UNPAIlUN

I tJbA INALJ%)( SYSIt"'S ANA1L1I )Z, ACTAvilY All-q: AIAA-SL (TFCH LLHHAWY)

I i;NIFO~t.I %LHvi( ' S U'NII Ut T1t ttALTHr '3Cl VtPARTMENr OF P-SYCHIATRY

I ou)A CumpuTFHN 5YSI;7Mb CUMMMI4U JIN: C;JMMAtIO TECH-NICAL L18HNY

I H-4.IAN WbULKLL KSud U1-u Lr1UMU)

I HU P~kU *LbTFH-(. LIMNAky

I HAT ILLI L LPKI I IiAky

I WAj)T CoRIIUNMfu'.,j AlII N: Ljr RAi4-1IA I TL ANU

I RA.~jU iP) L~k-u N Au,. AT IN; LitPANy U

INAVtC I IHMANRY AI'JA-tl

I(,RUjNNbL _Im,-)i4 Allh': mlIZI--L tLJ), Ij1

I icr~jt.H FUH NAVAL AlvALYtIS

L NAVAL sILALIH kt5Cr1 CtN LIohAkY

I NAVAL PtRSONNI-L l< NU u LLN LI-4HMNI' 41 rv CUIt 1020L

I AI' t-ONLt HOM'AN I'b ,y~t I- Ab A IIN: I

I t-U. ti-. tUACHULA MIIN: ILCLH N.I UIV

I ISC~i)L 01- bY$T)Lm:- ANU LUUi IIC. AITN'. AFIT/LSCM

I IHIC PfVOCtS5n-U AN~L) fI-LNCc-ICL t-AL ALiUI'!ITIUNS LWARTIAN I OLPAK) lLNI flII 111L NAVY IHmlINL ANALYSIS ANU EVALUATION (,P

I NArIONAL -Nfch I-JH -IIMLIrI SIIS)IC AIIN: H. UUY (IPSYCI10L0G1CAL AUVISUR)

I 1IA l ~I)I 01'- rLHA10HVUL ai-I Amu- LtAUr-HSHIP

1 01-1 uU'ul,.4(JN UNjv;HSJIY PtRI'W01ANtCL bi SI-SSMLNT LA80RA&1ONY

I WS.% (UM1L'.U ANU (-i~tAL -'IoiF- t.YLLh-At AT TN: LIBHARY

IUS,) IHAWSPOIA1UN'3LHCUjL UZA IHAVSP- ItFCH INt-I ANt) RSCH- CEN

I US A AL)" INLF N It-C-0L'CAL HLbEA14kti tINANCLI LIOIAKY

I uJS, FLI'LIl ARIY, IL' Aliv: AIZH-9lUl1 (KJLSI-4OM)

I NAT LLFANNjHUUSL t-UH- MLNIiLL i-AL~T- I'JI-U IPAHISLAVN 13LfC,

1 13 OF- Ii-AMZ) LCL" t' Jn CUMMUNiCA I IU14 tHSCri

I I NI 111tL I-OH Ut -Wt .,ALYSLb)

I os, THI0,jN,!> SU,i-) Ct-WTLH W-VLL SybIL-S INb + DEVICES Jt)LCTOWAIe.

I 4'FH-HL TtCI1NOLULY JI-C (h-)

FI PULNILUL UNIV ULPI Ut- WSYCMlULUuILAL SCL't tS

i U'3 M04L fI IY LUjlIDMkN1 M NU U LUMI'ANJ ATTN' URUME-LG

L114, Ub". MUW Alliq: ANIpL-uL ~ 4VTO

I 0A4 U5 AkMY Wt THAI'NINU tmyI RtbFM(. +tCLAIO I

I t4U4AfJ kt-AULHC- '-,"\AUMtNI Lt.No SAN Ult~k3U

IS IJAFA ULPT Ut- L~t ANL' 6L"l b~L

4 8
IUS MiL ITARY ACAUL4Y LIbHARY

I USiA INTt.LLIGENLE CLN ANU bCM ATTN: SCHOOL L1t:RARY

L MARINE CORPS INSTITUTE

I US COAST t*LAHU INdG CEN ATTN: t.UCATIJNAL SC FIE

I USAAVNC AN[) F-T. kjCIR AFIN: AIL(w-Es

I LISA AIR ULFENSL SCHOUL ATTN: A ISA-OT

I USAAVNC ATTN< ATZU-U

I US MILITARY ACAOL4Y Ut-C Ut- MILITARY LEAI)ERSIIIR

I US MILITARY ALAit->~y UitLIUk oF INSTITUTIONAL RSCH

I USA AIP UtFENSL SCHOOL AIrN: ArS'A-CO-MS

I UbAAUS-LfLhWAHYOUCUMLNiS

I USA Alk ULFE14SL ?4)ANL AIIN: FjLtS REPUSITORY

I H-lu USA SLR(,EANTb MAJOk UI.AULMY ATTN: LEARNINd- RESOURCES CENTER

I USA INTELL16tNLt ChN ANU bLI1 FuLATIJNAL- AD)VISOR

I USiA Okf)NANcE Cti'N ANU SCti ATN: ATbL-TLMt-C

I ObA AR~IUH SCHOUL ArTTN: ATSd-UT-IP

i NAVAL POSIGHAUUArE SCH AT IN: j~uLLY t NUX LIbRARY (CODE 1424)

1 USA THANbPORTAIIO'N SCHUUL ULLuIY ASSI. COMMANUANT EL)UCA. TLCHNOLOG;Y

1 USA SiGNAL SCHUOL ANU I-I. UOHOON AT7'4: AIZH-E.T

I USA MILITARY PULICE iUUL ATTly L18~RARY

I USA AHMOR SCHOUL EVAL 15HANCrl, uIRH.CTJHATL Of- INSTRUCTION

I USASI(bs STAFF ANj FACULIY UtV ANU IN5 DIV

I Hu AIC/XPIO THA1'INNti bYSILMS iiLVL.LOP~iLNT

I USA INSTIT~Lt I-OR fMILIlAkY ASSISIANCIE ATTN: ATSU-TU-TA

I US ARMY ARMOR SCH-JOL ULRtLTORATL OF TNAIiqNb.

I US4 UUARILPMA5[tH SCHUIUL UIRLCIORATEi OF- IRAININ(- UEVELOPMENTS

I US COAs T LUARU ACAL)tmY AIIN: CAU.1 CJUNSLLOH (DICK SLIMAK)

I USA IHA14SPOR1AIUN\ SCHOOL DIRELtOR OF TPAININS

I LISA CHAPLAIN CLNIEH ANU b'L1OL ANIN: LIkWARIAN

I USA LNFANTRY SLriUOL LIbRARY ATINI MISS LIVLNL3STUN

I USA INFAMNTRY SLH-U'JL AliN; AT!SH-i-\f

I US ARMY INFANTHY SCHUOL AITN: AISII-CO

I USA INFANIRY SLt1uDL ATTN; ATSH-UUT

I USA INFANTRY SCHODL ATTN: AISs--LV

I USA MILITARY PULICE SCHUOL/THAININb CENTr-H ATTN: ATZN-PTS

I USA MILITARY POLICE bCHOOL/TRAININU CENTFR Ul~o CUMBAT UEVELOPMENT

I LISA MIt IT ARY PUL ~t SCm-UL/THAINiNb cENTEH DIR. TRAINING Dk.VELOPMENT

I USA MI IARY POLICE bCUL/TRAININ, CENTFW AT TN: ATZN-ACk

I USA INSTII(-TE OF AUMIN151ATIUKl ATTN: RESIULNT TRAINING MANAbED4ENT

I USA I-IFLV ARTILLERY zCHUUL MUWHjIS SWIT L16RAMY

I USA INPTITUTE OF ADMINISIHAT1O0,I ACAU MIC L1)jRARY

1 USA WAk CULLE(UL A1TN: LIOtkAky

I USA LN(;INLER SLH-UOL LIHHARY AUM LEARNING RESOURCES CENTER

I USA~ ARMOR SCH-OU-L (USAHMb) AlIN; LIBRARY

I US CUAST UUAARU ACADEMY LibkANY

I USA IRANSPORTATION SCHUUL TkA14SW-ON1AIUr, SCHUOL LIBRARY

I UR(,AIATIONAL t.FFtCIIVI-NLSS3 TNb- CLN * SCMH ATTN: LIBRARIAN

1 0S ARMY IN~tLLIuENCE CL.IIH *SCHOOL ATT14: AISI-TU

I US ARMY IN~t-LLlvL4JE CLNILH * SCHOOL ATTN: ATSI-TULD

4 RIIS5H EMHASSY -hiWISH ULFENCL ST AFF

e CANAUIAN JOINT STAFF

I CULS (w) LIBRAKY II -RINCH MILITARY ATIAACHE

AUSTRIAN EMHASSY MILI]AHI ANU AIR ATTACHE

j - CAAAUAN OFFE-NCE LiAISOIN bTAII ATTN: COUINSELLOR, DEFFNCE R AND D

1 ROYAL NtTHFWLAN:b EMbASy MILIIARY AtTACHE

1 CANIADIAN IORCEb HASE CUHNWALLIS AI'TN: PFRSONNEL SLCTION

e CANAOIAN P-ORCEh P-EkSUNNtL APPL kbCm u.'4T

I AR,4Y RI-RSONNLL kLSLiAHCH LhiAbLl!),HMLNT

1 ANAY PF1RSONNEL RESLANCH L.SIAuLIShMNT AWI SCIENTIFIC COORDINATION OFFICE

b L1RANY OF CUN'tHt.SS E.XCHANbL ('NU bIFt UIVf

I DEFENSI- TICHNILAL INPORMAION CL1N ATTN: ~jTIc-TC
15J LINWARY U CU;,0HSh UJ1I UULLJUAL4N> E'APtLVTIN6 PHOJECT

jI-b HiTiH, R ANLJ U I-I4LALIIhL ATIN: UHLUL-LN

I 1S uLVI H14MF NT li- 1,41 iI. uH t LJ,4HAHI PUi IC UOCUMENTS DEPAH IMENT

I 1S tO~lfkNMENI PN\j4TIi'jU uH.. LI N A'JU sTA]UIORY, LIH DIV (SLL)

I Ijr AH14Y LIt1KAXM

10