- — _- ~~~~~~ — — — I., *_~

_ ~~~~~~~~~~~~~~~~~~~~~~

:

T

~~~~~~~~~

323 1

~ qi

1i ~ 4 ~~ iJ ~~

A BAYESIAN METHOD FOR EVALUATING

TRAINEE PROFICIENCY

Kenneth I. Epstein

0. . and

C..) Frederick H. Stelnhelser, Jr.

uJ

.-J

____ UNIT TRAINING & EVALUATION SYSTEMS TECHNICAL AREA

~

~

1 1IEIi:

~~

.

~

J

~~~~ j U~i L ~

L

~

J

U. S. Army

Research Institute for the Behavioral and Social Sciences

S pt.mb r 1878

Appsovsd for public rstsus; distribution unlimltsd, 

~

?8 12 18 .00 9

~~~~~~~~~~~~~~~ ~

IthL~~L~ :. 

~~~~~~~
~~~~‘~~~~~T ~~~~~~~~~~~ ~~~~~~~~~~~~~~~~ 

- . -

~~ ________ r “V- ~~~~~~~~~~~~~~~~~~~~ _

~~~~~~~

~~.~~~~•*1..~~~~~~~~~ . ._~~_=_._st ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisd iction of the

Deputy Chief of Staff for Personnel

WILLIAM L. HAUSER

JOSEPH ZEIDNER Cofonel, US Army

Techmcal Directot Commander

1

~~

I

NoTicEs

DISTRIBUTI ON: Pri nary distribution of th is riport his bssn madi by ARI. Pu N. sddress corrsioon*ncu

concerning distribution of sports to: U. 5. Army Rssssrdi Institute for the Behavioral and Social Sclsncss.

ATTh: PERt-P . 5001 Eiatnhos*r Avsnu , Aleeandrls, Virginia 22333.

FIMAI. 0 POIITION: This nspoct .

~ sy be d.ssmysd *hen It I. no IongS~ needed. Plsite do rot iiturn It to

this U. S. Army Research Institute for thi Riheeloral and Socisi $chen s.

~~~ 

The findings In this ripen are not to be eons*rusd as sn of fIn al Department of the Army position.

unleSs as disignated by othlr authorized documsnts.

~~~~~~ ‘

~~ ________
____I_”- —.-, —~~~~~ 

-.-—..-- —.- - -.- 

~~~~~~

-, . -.-.-—, ~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~

Unclassified

‘TV CLASSIFICATION OP THIS PAGE (

~~

isi Dat. &,rc,m~ ________________________________

meSS#1~~ r WI is&~~~rrAvU~~I D Ai~ 

- READ DI$?RUCUON$ rs~~. tp , ~ u ~~~~~~~~~~ u ~~ i i~~~’ r BEFORE COSIPLETmG FO~~ ST I RKPORT NUMSE _.‘ 

~

2. GOVT ACCESSION NO 3. RECIPIENT’S CATALOG NUMSER

chnical ~aper

~ 23 “

- Ti (and s wrn•i - S. TYPE OF REPORT a PERIOD COVERED

( SIAN HOD FOR ~~ AWATIWG ~~~~INEE -- ____________________________________ 

S. PERFORMING ORG. REPORT NUMPER

7. AUTHO - CONTRACT OR GRANT NUMSE*(.)

enneth I.)EPstein 9Sd~ Frederic /Steinheiser. Jr~ ‘/

3. PERFORMING ORGANIZATION NAM E AND ADDRESS ,~ 10. PROGRAM ELEMENT. PROJECT . TASK

Army Research Institute for the Behavioral and V ...

~~ AREA & WORK UNIT NUMPERS

Social Sciences ( Io 2Q762722A764

5001 Eisenhower Avenue, Alexandria, VA 22333

II. CONTROLLING OFFICE NAME AND ADDRESS

Army Deputy Chief of Staff for Personnel ~A

Se 1 . ~~

’

~~78 ~

Washington , DC 20310 ii. NUMUER~ )F PAGES

Ii. MONITORING AGENCY NAM E & AODRESS(tt diff .omt from ConiroiUn 01Db.) II. SECURITY CLASS. (of thu. uport)

Unclassified

ISa. DECLASSI FICAT1ON/OOWNGRADING

SCHEDULE 

IS. DIS TRIIUTI ON STATEMENT (of this R.pp,t) —— Approved for public release; distribution unlimited.

a : _ _

17. DIST ijripN S ATEMENT (of A. ab.SracS sit.rod in Stock 30, if atff.twt from R.porf)

IS. SUPPLEMENTARY NOTES

1$. KEY WORDS (Ccntinu. an rvr•• aid. it n.cs..m~ and td.ntity by block nomb.r)

Decisionmaking

Bayes’ Theorem

ry

Classification error

\ it. ASITRACT (Conefaee , ro, .i

~~ If anc..wy and Id~ ItSt~ by kiosk .&.~

)

\ __—7 xn any testing or evaluation program, there will be some percentage of

false positives and false negatives, i.e., misciassifications will’occur. A

decisionmaker therefore needs to make a best estimate about the true level of

proficiency of an examinee . A multiparameter , prograimnable model was devel—

oped to examine the interactive influence of certain parameters on the probebility of deciding that an examinee had attained a specified degree of mastery

through a program of instruction . The parameters, readily obtainable from —

(continued)

DO ~~~~~~~ 153 Eenso eF Mov

e7

~~~~LETE 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
--~~~~~~~~~~~~~ -~~~ ~~~~ - , - --

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

—

~~~~~~~~~~~~~~~~~ 

- - - -

_... . .te~

.. 

~~~~~~~~ S ... . ~~~~~~ .-- - -

( S OweTY C uspsps1msi.sp~w*f-rinrr

~ pat. 

~~ ., 

~~~ - ~~~~

20. 

1 I idecisionaikers, includes 

~

$. ths~n~~~.r of assumed mastery states - (“master,

“intermediate ,” “nonaa4er”-) ,. ~~~ the prior distribution of scores from similar

- - - examinee groups, and. the nt

~~ er of test trials or items that could be

given .

Results of several simulation. showed that- the degree of confidence that

a decisionmaker can have in his decision (e.g. , “x%” certainty that an examinee

is a master-) is markedly affóôt*d- by - values - -for- the- above.ssi~

ticned parameters.

A key feature of a Bayesian -mo4 1 is that t sti

~g time, manpàwer, expense , and

test length can b.-- reduced if the “prior” information i. accurate and valid for

the particular tested gr~~p. ‘U not, little can be gained from a - Bayesian

model. Simulated test results - also - showed that a test can be too - short to be

of any decisionsaking valne~

I 7

~~~~~

t Unclassified

I ISCUNITY CL*$SIPICA?ION OP TWIt PASL(Phil~ Des. tnIsro~~

_________ - —-—— 

4~- ~ - I -

- - — T’~~ ~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~
~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

. ~~~~~~~~~~~~~~~~~~~~~~~ 

—- 

~~~~~~~~

- - --

~ 

—- P 

~~~~~~~~~~~~~~~~~~ ~ 

- 

~~

._ ..- -‘-- ,

--

- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - -

.

—

TNhnIcaIPap r 323 -

A BAYESIAN METHOD FOR EVALUATING

TRAINEE PROFICIENCY

Kennet h I. Epstein

and

Frederick H. Steinheiser, Jr.

UNIT TRAINING & EVALUATION SYSTEMS TECHNICAL AREA

Submitted U complete and Approvsd By:

technically accurate, by:

Frank J. Harris A.H. Bimbaum, Acting Dirsctor

Technical Area Chief ORGANIZATIONS AND SYSTEMS

RESEARCH I.ABORATORY

J

~~sph Zsldnsr f TECHNICAL DIRECTOR

I

U.S. ARMY RESEARCH INSTITUTE FOR ThE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

Sept.mb.r 1918 -

Army Project Number Unit Training Standards

20752722A764 and EvaluatIon

*,

~

vreed for puatic r

~s.se; msuItbitiSDUnSIet~~ .
~ ~~ .

~~~ -~‘v ’~” 

~~- -r- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~ 

- 

- 

~~~~~~~~~~~~~~~~~~~ -‘I

-ARI Research Reports and Technical Papers are intended for spo nsors of - - R&D tasks and other research and mihtary agencies. Any findings ready for

implementation at the time of-pubhcation are presented in the latter part of

the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

-agencies by br4eIki

~ 

or- Diepouhion - Form. ,

~ : -

~~~

- 

_ _J -__-1 u - --

a

I 

______ 

-, :- :I

~~ ____ 

____________ ~~~~~~~ ~~~ -- ~~~~~ -
- ~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - --—.-.-~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -

~

._ ~~~~~~~~~~~~~~ -.- ,--

~~~~

FOREWORD

The research presented in this report was conducted under Project

METTEST (Methodological Issue s in Criterion-Referenced Testing ) , under

the auspices of -the Unit Training and Evaluation Systems (UTES ) Technical Area of the Army Research Institute for the Behavioral and Social

Sciences (ARt). The goal of Project METTEST is to provide quantitative

methods for evaluating unit proficiency . The means for achieving this

goal include basic research in test construction methodology , measurement and scalin g models , and decisioninaking implications of test score

interpretation . ARI Technical Paper 306 is the initial publication on

the project. -

Related , ongoing programs within the UTES Technical Area include

evaluation of small combat units under simulated battlefield conditions

(REALTRAIN) , qualification of tank gunnery crews and revision of Table

VIII (IDOC) , and improving the standardization and reliability of the

Army Training and Evaluation Program (ARTEP) .

Anticipated future research under Project METTEST includes the

development of a computer-programed model for performance evaluat ion

and several additiona l 6.1 basic research grants for the development

of measurement , scaling , shoring , decisionmaking , and quality control

models for use in performance evaluations when criterion-referenced

testing procedures are employed .

The present research was conducted by personnel of the UTES Technical Area as an in-house research project , under Army Project

2Q762722A764. G. Gary Boycan supplied a key creative insight into the

“misclassification problem. ” An earlier version of this paper has been

printed in the Proceedings of the October 1976 Naval Training Equi pment

Center (NTEC ) Conference .

t -

JOS PH ZEIDI(ER

‘Pec ical Director

~

- . - - - - - -

-~
- 

-

A BAYESIAN METHOD FOR EVALUATING TRAINEE PROFICIENCY

BRIEF

Requirement:

The educational decisionmaker typically wants to know if a student

can perform a-job at some prespecified level of acceptability . If the

student’s test score is above the minimal passing standard , the m dividual may be classified as a master——otherwise , as a norunaster . The

present paper describes a mathematical model that provides maximal

classification accuracy with the least number of test items or trials.

Classification Model :

Estimates of several variables must be provided as input to the

model, which is derived from Bayes’ Theorem. Two of these variables

are probability estimates: the prior expectation of selecting a master

from the student population and the conditional probability that a known

master would answer a randomly selected test item correctly. Two other

variables--the minimal passing standard and the number of test items--

are under some degree of control by the tester. Furthermore, the effect

of the latter two variables is an interaction , because the model shows

that classification accuracy is not invariant over different test lengths

when the same percent correct score is attained by examinees.

Findings:

A computer simulation of the model demonstrated the effects of -

simultaneously varying five variables on classification accuracy . The

arbitrary nature of defining the criterion for mastery as a percent - -

correct test score was critically evaluated. Testing may be irrelevent in situations where the test length is less than the minimal num-

— ber of items.

Utilization of Find~.nqs: - I

The m~~e1 shows explicitly the risks involved in using a given

length of test once the tolerance for misclassif ication error has been

specified by the examiner. 

- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ‘‘------ -

~- -- ——-~~~--—— — —
— -~~~~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~

r

A BAYESIAN METHOD FOR EVALUATING TRAINEE PROFICIENCY

CONTENTS

Page

INTRODUCTION 1

TRAINING TO MASTERY 1

CONSTRUCTION OF THE MODEL 3

Bayes’ Theorem 3

Variables of Interest in the Present Simulation 5

Changes in p (MIT) , Assuming Two Mastery States 5

Elaboration to Three Mastery States 11

Flow-Chart Analysis of How the Bayesian Model Was Developed . 18

TEST LENGTH AND MISCLASSIFICATION ERROR 21

SU?O4ARY AND CONCLUSIONS 25

APPENDIX . A COMPUTATIONAL EXAMPLE FOR THREE MASTERY STATES . . 27

DISTRIBUTION 31

LIST OF FIGURES

Figure 1. Conditional probability of mastery when there are two

very distinct prior states of mastery 7

2. Conditional probability of mastery when there are two

relatively distinct prior states of mastery 8

3. Conditional probability of mastery when there are two

equally probable prior states of mastery 9

4. Conditional probability of mastery when there are three

prior states of mastery and three conditional

probabilities of answering an item correctly 12

5. Conditional probability of mastery when there are three

prior states of mastery and three conditional

probabilities (same values in different order

from Figure 4) of answering an item correctly 13

-

~ 

-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Page

Figure 6. Conditional probability of mastery when there are three

prior states of mastery (with mastery and nonmastery

being the least probable) and three conditional

probabilities of answering an item correctly 14

7. Conditional probability of mastery when there are three

prior states of mastery (values from Figure 6) and

three conditional probabilities of answering an

item correctly (values from Figure 5) 15

8. Conditional probability of mastery as a function of

percent correct using the sante parameter values as

in Figure lA 22

9. Conditional probability of mastery as a function of

percent correct using the same parameter values as - -

in Figure lD 23

Flow Chart 1. Required inputs and sequence of steps in order to

obtain the conditional probability of mastery

given a test score 19

Li -_____ - - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

--
~ 

A BAYESIAN METHOD FOR EVALUATING

TRAINEE PROFICIENCY

INTRODUCTION

No instructional system is complete without a strong testing component. Any student who begins an instructional program should be

able to achieve all the objectives that the program was designed to

teach. However, some students may require remedial or other supplementary instrut-tion to master all of the objectives, even though the

program was carerully developed. Furthermore, during the development

of the instruction , test data from prospective students are required ,

first to revise and later to validate the instruction. To support the

instructional development activities and to make decisions about the

abilities of students who have completed instruction, a powerful testing program is necessary . -

The final desired output of a test for a given examinee is information that can pinpoint ability to do whatever is required by an objective. That is, the examiner observes a test score and then infers

the ability of the examinee. This paper outlines a “Bayesian” method

for drawing such inferences. It also discusses and illustrates the

adequacy of the method as a function of the number of test items administered and the effects of the tester’s beliefs about the quality

of the examinee population on the inferences drawn.

Using the Bayesian method , the testers hypothesized varying numbers of ability groups so that the classification of examinees into

these ability groups is most useful to the overall instructional system. For example, the simplest case is to classify examinees into two

groups, the first group containing those who have mastered the objective, and the second containing those who have not. Alternatively, one

could hypothesize three groups, consisting of masters, nonmasters, and

an intermediate group containing people whose skills are almost satisfactory and who could be brought up to the mastery level with relatively

little additional instruction . The Bayesian model presented in this

paper explores up to three levels of mastery, although this number

could easily be expanded. The model also explores the effects on decisionmaking (correctly classifying masters and nonmasters) if more

than two ability levels have been hypothesized but are then collapsed

to form just two groups——masters and nonmasters.

TRAINING TO MASTERY

Ideally, the educational decisionmaker wants to know if a person

(student, trainee) can do a job at some prespecified level of acceptability. A student who scores above the minimal passing standard on a

test may be classified as a master; if the score is below the minimal

1

- i - -—_ -L

Li
- - - ~~~~~~~~~~ - -~~- -~~~~~~~~ -~~ -- 

passing score , the student would be termed a norunaster . But since data

always have some error variability, misciassifications are likely to

occur .

True

competency state

NonMaster master

True False

Master . . -

positive pos:LtiVe

Classification based __________ _________

on test score False True

Nonmaster negative negative

Ideally , the probability of a true positive should be much greater than

that for a false positive, and the probability for a true negative

should be much greater than that for a false negative.

To evaluate how well our testing program achieves this goal, we

want to be able to infer as accurately as possible the conditional

probabi 1it~’ of the mastery (or nonmastery) state, given the test score

data, p (M1IT) , p(M2I T). Our first problem is what amount of data is

this probabilistic inference based upon? Suppose that the passing

standard was 80% of the test items correct. A student with 33 out of

40 items correct would pass and would be classified as a master. Now

suppose that on another form of the test (or a test given over the same

material by another instructor) , another student gets 25 out of 30 test

items correct. This student would also have met the 80% correct cri—

ten on and would be classified as a master. The model presented in

this paper will show that the p(M1IT) varies systematically with the

number of test items, along with the minimal percentage correct for

passing.

We may also ask: How is the accuracy of inference about mastery

affected by postulating more than two states (mastery and nonmastery) ?

and can the data from various states be combined without seriously affecting the final p (M1 I T) inference? For example, suppose that there

are intermediate states of partial mastery. The following decision

model shows that p (M1IT) can be more validly estimated when the mastery

states are processed independently , but that educational decisionmakers

will not sacrifice very much classification accuracy if indeed they do

dichotomize multichotomous data. We suggested that defining an intermediate group which required minimal remediation might be useful for

some instructional systems. The model shows that the probabilit~ of

P being in the mastery group when indeed the datum was a test score

2
1

~ obtained by a master will be increased if the other data are processed

independently. The concept of “independent processing” requires that

all nonmastery groups maintain their integrity, rather than being aggregated into one generalized nonmastery group.

- 

CONSTRUCTION OF THE MODEL

Bayes’ Theorem

The statistical model which we have applied for classifying students

into mastery and nonmastery groups, given their test scores, is based

upon a form of Bayes’ Theorem:

Ml = p(TIM1)p(Nl) p( T) [p(TIM 1)p(Ml ) + p(TIM2)p(M2)]

Here we assumed that the two states of nature (master and nonmaster)

are mutually exclusive and collectively exhaustive , and that T is the

test score observed . We also assume that the test is dichotomously

scored and that the items are independent. A correct response is denoted “1,” an incorrect response is denoted “0,” and the total test

score is simply the number of correct responses. What we seek to find

is the term on the left, the probability that a given student is a master, having been given his test score . To find it, we need an estimate

of the prior probability of mastery (p (Ml) ) in the population of students from which this student was drawn. The prior probability of mastery can be considered the proportion of students in the exantinee population we think are masters. For example, if our instruction were very

good, the prior probability of mastery would be high, and most of the

students who completed the instruction should have mastered the objective. The actual number specified for the prior probability of mastery

may be an informed guess based on experience , or it may be based on the

empirical results of tests given to previous classes of similar students.

We must also estimate the conditional probability of a certain

test score , given that the student who receives that score is a master.

For example, if only one item is administered, the conditional probability of a score of one correct, given that the student was a master,

is simply the probability that a master responds correctly . We may

estimate this conditional probability empirically based on previous

student groups, or we way provide a best guess as to how well masters

perform, or this conditional pr~bability may reflect a minimal standard

of achievement. We shall show how the p (MI T) will vary as a function

of the prior expectations of th. tester, number of test items, and conditional probabilities, p(T1M), after an example to illustrate the

computations. — 

- - - — - ~~~~~~~ - - ___
_____________ _________ ____ 

_ _

~~

_

~~ 7~ ~~

- - - - - -‘.‘-

~~~~~~

Suppose that a student chosen at random from a trainee population

is given a criterion-reference test, and that he passes the test.

Given the results of the test, what is the probability that the stu—

dent is indeed a master of that particular course of instruction? To

calculate the probability, we obtain the following information from

the educational expert who administered the CR?: The probability that

a master would obtain a passing score — .90, (p(’tjMl) .90) ; the probability that a nonmaster would obtain a passing score — .05, (p(T~M2)

.05) ; and the prior probability of randomly selecting a master trots

this trainee population is equal to .70 , that is, we believe that 70%

of this and siailar previous trainee populations may be assumed to be

composed of masters. substituting these values into the formula

.9 x .7

p (Ml T) ~~~ ~ -, +- 

~

os x .3

equals .977. Hence, before- the test score was available , the probabi].ity that this student wag a- master was .70, but after a passing score

was observed, the probability that this person is a waster has increased

to .977. (The probability of this student’s being a nonmaster , given

the same passing score, p (M2 ~~), would be equal to 1 — .977 or .023.)

Tb generalize the Bayesian approach to~ a wide weriety of applications in evaluating training e’ffe’ctiveness’, twoS additions must be made

to the basic formula. These mdditiø~~ are the number of trials or - items

on the test (N) ,

- and the niimbez o~ h~~~thesizSdf mastery states (S). The

derivation of the general Ba~~eiem foren-la for this purpose was originally presented by HersIlmal?:

N - 11 P (MiIt~ )

p(MiI’fl ~ N

II p(Mi~t4)

r N-l -

i~I p (Mi)

In this foriwila, p-(Kt ~

‘t~’) equals the coitditi’om*]. probability of a ~~rson in the ith mastery state getting the jth- test itets correct; p ~

Mi)

is the prior probability of the r.presentatii~~ ~f the ith mastery state

in the student pcpulatisn (the ~~~e.ntag. at st~~.nt who are .atimat.d

1

0.rshiiian, R. L. A Rule for the Integration of Bayesian Opinions.

Human FactcIs, 1971, 13, 255—259.

4

______ ___ 

-i’- 

~~~~~~~~~~~~~~ ~~~~~~~~~~ -~~~~~--~ -— ~~~ 

—- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- -~~~-

~~~~“~~~~~ --,--, ~~~ - ---,-—----. ~ -~~~~~~- - -

to be in the ith mastery state); and p(Mi IT) is the conditional probability of a particular student being in the ith mastery state given his

total test score. A computational example showing how the formula is

applied for three mastery states is given in the appendix.

Variables of Interest in the Present Simulation

In the typical situation for evaluating training proficiency, the

tester has some control over the number of items or trials that he will

include on a test. In a performance—based test, each trial may be rather

expensive (such as tank gunnery or field artillery , where each shell

costs over $100), and so the tester will be obliged to use a minimum

number of trials to meet his decisionmaking requirements . Consequently ,

we examined the effect on p(MIT) when N took on values of 5, 10, 20, and

— 40 trials.

The tester also has responsibility for assigning reasonable values

to the prior probabilities of mastery, denoted as p(Mi), and to the conditional probabilities of a known master (or nonmaster ) getting a randomly selected item correct, denoted as p(t~Mi). Values for both the

H prior and conditional probabilities were systematically manipulated in

the present simulation.

The number of mastery states is a variable which the trainer and/or

tester may also set. In some measurements of trainee proficiency it

may be most appropriate to dichotomize on an all-or-none basis, whereas

other training evaluation contexts may suggest a “pass, give refresher

training, recycle failures through complete training” trichotomy. More

than three mastery states may of course be hypothesized , but the contputations in the present and all other models of proficiency evaluation

become extremely complex. (However , we are developing a computer program

that will handle up to five states of mastery.)

The dependent variable of main interest is the percent of items

answered correctly . The tester may decide that 70% is a passing score.

But the 70% value is not an absolute standard , since it is dependent

upon the number of test items and the prior and conditional probability

estiuates. In the present simulation, three values of percent correct

observed scores were used: 60%, 70%, and 80%.

Changes in p (M IT), Assuming Two Mastery States

The fundamental purpose of the present study was to investigate how

the probability of mastery classification changes as a function of the

simultaneous manipulation of up to four parameters (independent variables). The scope of the study is not exhaustive, since only several

values of each of the four variables were used. However, some general

trends do seem to emerge, as can be seen in the following figures.

5 

~~~~ -~~ - - —
-

~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ,

~~~ rffr -- - - - -

Figures 1, 2 , and 3 show the results of applying the model to a

situation in which only two mastery groups (mastery and nonmastery)

have been hypothesized . The data points represent the probability that

a trainee is a master, given (conditional upon) his total test score ,

P (M IT) . The lines show how the P (M T) changes as a function of variations in the four parameters: prior expectation of mastery , the percentage correct items observed , the conditional probabilities of both

.1 master and a nonmaster responding correctly to an item, and the nunber of items comprising the test.

Figure 1 represents a testing situation in which the training was

of extremely high quality, since the proportion of masters in the trainee population was assumed to equal 0.9. That is, p(Ml) = 0.9. Fig -

— ure 1A portrays the situation in which both masters and nonmasters have

attained a rather high degree of proficiency, since the probability of

a master responding correctly to any given item is 0.9 , and the probability of a nonmaster responding correctly is 0.6. If a person scores

80% on a 5-item test, the probability that he is a master is approzimately .91. This probability drops to .65 if a 60% score on 5 items

(3 out of 5 correct) is obtained. Mote that when the test length is

increased to 40 items, an 80% score (32 correct) produces a .99 probability of mastery . However, a score of 60% (24 correct> ~

,ie1ds an essentially zero probability of mastery . The effect of the test length

• variable on classification accuracy is dxamatic: If the p(M~T) had to

be at least 0.5 for a person to be called a master, then scores of 60%

on a 5-item test would lead to stery classification . But a 60% score

on a 40—item test would lead to ocemastery classification.

Figure 1A also illustrates the .ffect of “prior beliefs” on p(MIT) .

One might suppose intuitively that the chances were much higher that a

person who obtained a score of 60% (even fro. a S—item test) came from

a population whose probability of corr.ctly answering an item was 0.6

than from a population whose probability of answering an item correctly

was 0 • 9. However, the relative proportions of the two groups (expressed

as prior belief in mastery and noumastery , ox p(Ml) — .9 and p (M2 ) — .1,

respectively) are such that the probability of a person being in the

mastery state is appromimately 0.65 for a score of 3 correct (60%) o~

:

a 5-item test. Only by increasing th. number of t.st items can the

strong prior bias in favor of th. mastery decision be reversed. Fiqures 2A and 3A show what happens when prior beliefs are not so heavily

— biased in favor of mastery . In neither oés. is the probability of being

in the mastery state above 0.5 for scores of less than 80% • But Figure

].A suggests that when prior beliefs heavily favor one group over the

other, longer length teats shou~ 1 be used. Otherwise , the emount of

data may not be sufficient to force a chang. in th. originally held

prior beliefs.

6

• ______ 

______________

________ 

~~~~~~~~

—- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 
________________________________________________________________________ — - - 

~

1

0

0

~~~~ 

0 

~~~~~~~~~~~~~~~~~~

i

i ~~~~ 

~~

‘ 

~~~

~

I

. 

~ t

I

~ 

~ ___________________ .~ 

i_

~
r’ ~~~~~~~~~~~~~~~~~~~ 

-

1

- - -

~ 

-- -- --I

-

i

~

5. 0. -

~~~ 0 — I’,

‘B L

ur 0

~~

’

~~~

’

’0” 

~~~~~~~~~~~~~~~~~~~~~~~ II

U. )

~

,

~‘•I

o

~~~~~~I

H 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~ 

I

‘I

ku .. 

~ .

,

• 

-g

I ~~ I I r

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

’

-

# 2 0

I I ’

- 8

L.. 

• ____

__________ 

~~~~~~

—- - —- --— - 

~~~~~~~~~~~~~
__________________________________________ - -

~~~-~~~~ ~~~~~~~~ -.--~~ - -.- - -.---~ .—~~--r-- w -.

-• -- — -- ——- _- ‘,c--

_ -

. . 

-?

-g

I

1 I 

‘4 0

i ~~~~~~~~ 0 ‘4 

,

~~

. 

— c ~

l

I :!

/ j::! ii ‘

II ’ S

I _4•’ 

-

~ I _SSl0_

_

~

- 00 :

H -

I’) ‘4 1

Iu • u

• ‘4 — g

‘4

i

In.

-In

2 0

~J. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
— —~ —~~~ — -‘-—-wP.r-, -

~----- - -- ,-- ~-..

~

- . . r —•-—-~- 

~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~

The effect of changing the prior beliefs concerning the proportion

of masters and nonmasters in the exazainee population, while holding all

other parameters constant, can be seen by comparing corresponding Graphs

A , B, C, and D in Figures 1, 2, and 3.

The impact of prior information on classification accuracy is very

significant : positively so, if the pr-iors are accurate; and unfavorably, if the priors are inaccurate . Novick and Lewis2 claim that if

the criterion level for mastery is kept constant, then low priors will

require high test scores to convince the (skeptical) decisionnaker that

the examinee has attained the criterion level for mastery . Further,

high priors will allow lower test scores to convince a (less skeptical)

decisionmaker that the exaninee had attained the same criterion level

for mastery. In su~~Ary, if prior information is strong but inaccurate,

then longer tests will be needed to overcome this bias; but if the

prior information is strong and accurate, then test lengths can be reduced (by 50%, for example) relative to the number of items that would

be required to reach the same decision with no prior information.

The effect of changing the probability of a correct response ,

p ( lI M i), can be seen by comparing Graphs A, B, C , and D for Figures 1,

2 , and 3. For example, the only difference between Figure 1A and Fig—

tire lB is that the p(lf 141.) changes from 0.9 to 0.8, all other parameters

being held constant. (This change might reflect a lower level of required proficiency and, hence , less training, for Graph B than for A.

Or perhaps previous test results indicate that masters of the instruction respond to items with a probability of correct response equal to

0.8 rather than 0.9.) In any case , the effect of this small change in

the p (11141) on the p (M IT) is readily apparent. For any test length or

observed test score, the probability of being in the mastery state is

greater in Graph B than in A. This shift is most obvious for the 70%

• observed correct curve. Notice that p(M~T) on Graph A for an observed

score of 70% (28 out of 40 correct) is approximately 0.04. However ,

the value for p(MJT) in Graph -B for 70% of a 40-item test correct is

0.87.

• The main reason for this abrupt change from Graph A to B (in Fig—

ures 1, 2 , and 3) is the lowered requirement for mastery, from 0.9 to

0.8. The probability that “0.9 persons” score only 70% correct on long

tests is relatively low. But when masters are defined as those trainees

who come from a population with a probability of responding correctly

equal to 0.8 , the probability of their scoring 70% on a long test is

high. One of the most difficult jobs for an instructional designer is

2

Novick, M. R., & Lewis, C. Prescribing Test Length for CriterionReferenced Measurement. In C. W. Harris, 14. C. Alkin, & W. 3. Pophain

(Eds.), Center for the Study of Evaluation Monoqraph Series in Evaluation , III: Problems in Criterion—Referenced Measurement. Los Angeles:

U.C.L.A. Center for the Study of Evaluation , 1974.

10

_______
F~~~~~~~

’ T “~~~~~ —~~~‘.,~~,

~~~~~~~~~~~ __ __ 

~~~~~~~~~~~~~ ~~~~ - - -

~~~~~~~~~~~~ - _

~•_.~~~~~ _ _ ••,-_•-- _ •_

~~~~~

- -—,

~~~~~~

to describe the level of capability required of graduates and the level

of capability actually achieved. Comparison of these graphs indicates

the magnitude of the effect that these specifications can have on the

classification of trainees.

Graphs C and D of Figures 1, 2, and 3 further illustrate the effect of variations in the probability of correct responses. The only

difference between Graphs B and C is that the probability of a correct

response from a nonmaster decreases from 0.6 to 0.5. The effect of

this decrease in correct response probability from a nonmaster is to

increase the probability that someone with a score of 70% or 80% will

be a master. Note that the 70% and 80% curves are higher in Graph C

than in B. Not evident from the graphs is the additional result that

nonmasters are less likely to achieve a high score in C than in B, since

p( lIM2 ) = .6 in B, and p(lIM2 ) = .5 in C. Finally, Graph D portrays an

extreme case in which neither masters nor nonmasters are responding at

particularly high levels. However , the level of performance for nonmasters is so low (0.4) , that even for observed scores of 60% the probability of being in the mastery state exceeds 0.8 for all test lengths,

except for 5 and 10 items in Figure 2 , and 5, 10, and 20 items in

Figure 3.

Further detailed analysis of these figures is not included in this

paper. In comparing the 12 graphs against each other , note the magnitude of the changes in p(M~T) when small changes have been made in the

prior beliefs , in the correct response probabilities, and in the percent

correct observed responses. The implication is that extreme care must

be taken when specifying parameters in a Bayesian approach to testing

and decisionmaking . If the parameters are realistic, great savings in

testing time and expense, and increased confidence in decisionmaking

are possible (Novick & Lewis, 1974). However, if the parameters are

not realistic, there is a very real danger of misclassifying many examinees. The next section of this paper deals with an elaboration of

• the model to three mastery states, thus helping to quantify sources

of classification error.

Elaboration to Three Mastery States

Figures 4 , 5, 6 , and 7 represent cases for which three mastery

states have been hypothesized. In Figures 4 and 6 the probability of

a correct response for a person assumed to be in mastery state Ml

equals 0.8; for mastery state M2 this probability is 0.6; and for mastery state M3, it is 0.5. These values could correspond to the situation in which the nonmastery group was divided in half. That is, those

persons whose probability of getting any given item correct is 0.5

(comprising mastery state M3) would need extensive retraining; whereas

those whose probability is 0.6 (comprising mastery state M2) would

merely need selective retraining. People in mastery state Ml have a

probability of 0.8 for making a correct response and may therefore be

considered as “masters” who have successfully passed training.

j 

11

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
— - - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - —~•w,•~— -

~-‘-‘•.--- - ~~~~~~~~~~~~~~~~~~ ..- ---~~~~ e.

0 

~~~~~

• •i —?

I I I

I •

I

I “ r r

I ~~ / 5.0

‘ k /~~~~~~~~ —~~~~

‘°

E I-. 

— ~~~~~~~4J

I 

_.

I 

/

,-

~ ~~~~~~~~~~ Z

H 

I j -H ,-?

! 

~~~~~~~ 

I /

0

12 

—••---—‘,- - - 

- ~~~~~~~ -1
-- -.--

~~~

,.—--

— 

-

p ~~-?

/

/ 

-g

A

’

I / 

•

‘~~

‘

I / — 

~~~~

,

S 

-U,

00

i

i ~~~~~

I 

,/

i

~~~~

H 

~~~~~

-

~

-

W . 00

I

Ow

S W 

•.-4

p

- 

I ~~~~~~~~~~~~

H in

0 0

• 

~,

.p4

Q) raw

-;

/

/

.1) 

-

~~~

I 

I

I 

I

-t t 

~

°

0 0 

o

~~l3

a.

___ 1
— - - ‘ _

~~~~~~ _ ‘~~“~ ‘_

a 

r .

I __?

I I —g __ i U

~~~~~~~~~~~~~~ 

‘I 

II I .

‘

_

“

1 

~ 

I J’j_~

U) I ~

J /4 _

~~~~~~~~~~~~~~ I J U, _

~

0 bu. w w 0 ~~

I • 5.54

I ~~ a .

.

\ \ ,~ —g

-

~~~~ S

I ~

.,dV’ —

~~~~ I I

i

~ 

I 

S

7 III

4~ , I

~ 

~~ 

1 ~~ 

I

,

,

,

, 

~~ 

I

~~~~~~~~~l4 

0 

— — —-- — — - --- - - - -— -- - ---——- -— —------------ - - -
- -—-----

• — ?

I / 0

I ~~~ — “

I

, 

~

I I 0

P — I w e 0. I

up

,

W I.’

U — ?

/

1 

-; 

~

; !

~

e LU

U

j ml

‘

~NI,I 

~~~~~~ ~~ w tu’ N - .•4~~~~~)

2 ‘ 0~~~ 0

a 0

0 2 p N

I-. 0

• - a 0. I . if I

5 I

-0.

~ 

I I — 

/ ~~~!

15
~~~- -~~~-~~~- -— - - --- -~~ - ~~~~~ - - -~~~~~ - - - ---- -~~ -j------ ~~~-

~~~~- - -~~~~~~~~~~~~ — - ---

For Figures 5 and 7 , the corresponding probabilities of a correct

response for people in mastery states Ml , M2 , and 143 are 0.9 , 0.8 , and

0.6 , respectively. These probabilities might describe a situation in

which the mastery group was dichotomized, perhaps in an attempt to identify those students who had achieved an exceptionally high level of proficiency, i.e., p(lIM l) = 0.9.

In Figures 4 and 5 , the prior probabilities (or assumed proportions)

of examinees in each mastery state are: p (Ml) = 0.5 , p (M2) = 0.3 , and

p(M3) = 0.2. In Figures 6 and 7, the corresponding prior probabilities

are 0.25, 0.50, and 0.25, respectively . The prior values in Figures 4

and 5 display a bias toward higher levels of mastery (50% of the cxaminees are assumed to be type Ml masters), whereas the bias in Figures

6 and 7 is toward the intermediate level of mastery (50% of the examinees

are assumed to be type M2 masters) .

A detailed analysis of Figures 4 and 5 provides the basis for an

interpretation of Figures 6 and 7 , which is an exercise left to the

reader. The three graphs labeled A , B , and C represent the probability

that an individual is in mastery state Ml , M2 , and M3 , respectively.

Graph D represents the probability that a porson is i’i mastery state

Ml after mastery states M2 and M3 have been combined into one composite

state.

Graph A of Figure 4 shows the probability that an individual is

in mastery state Ml, given observed scores of 60%, 70%, and 80% correct

on 5— , 10— , 20— , and 40—item tests. Thus, for an observed score of 4

out of 5 correct, the probability that this person is in mastery state

Ml is about 0.65. But if this same person scores 32 out of 40 (still

80% correct), the probability that he is an Ml master jumps to 0.98.

These results are similar to those obtained when two mastery groups

were hypothesized , and again illustrate the effect of increasing test

length on the level of confidence in the mastery classification p(MIT).

The probability of being in mastery state M2, given observed

scores, is plotted in Graph B. If a person got 4 out of 5 correct, the

probability of being in state M2 is about 0.25. However, if he got 32

out of 40 correct (still 80% correct), this probability pluinmets to

0.02. Finally, using these same test score values, Graph C shows that

the probability of being a type 143 master is 0.10 for 4 out of 5 carrect, and nearly zero for 32 out of 40 correct. This result makes in—

a tuitive sense, because there is only 20% of type M3 (non)masters in

the examinee population , and the probability of their getting any item

correct is only 0.50, which is a long way from 80% observed correct.

Notice that for any given test length and percent correct, the

sum of the probabilities of being in states Ml, M2, and 143 equals 1.0.

Comparison of Graphs A , B, and C shows that when either 70% or 80% of

the items for any test length are correctly answered, the probability

of being in state Ml is greater than the probability of being in either

16

:—::- ~ z

~Li~ i:TT z~~~~i ~~~~~~~~~~~~~ ~ •, 

-
__________________________ -.-- - -

-,-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ “~~~~~~-—-~

—•.----

~

•—- --

~~

---— --

_-

~~

-- , —-

~~

“---•- -

~~ 

— 

,I1

state 142 or 143. That is, both the 70% and 80% curves are higher in

Graph A than in either Graph B or C. For an observed score of 60% ,

the probability of being in state 142 is greater than for Ml or M3.

The probability of being in state M3 is rather low for all. values of

test length and percent correct observed in this particular example.

Graph D depicts the probability that a person is in mastery state

Ml, as opposed to a new nonmastery state composed of both M2 and 143.

It can be seen that when states 142 and 143 have been thus combined , the

probability of being in state Ml is greater than when all three states

were analyzed independently. For observed scores of 70% or 80% correct ,

there is slight difference in the decisions that would be made under the

“independence ” versus “composite” conditions. However , if a score of

60% were observed , the possibility of distinguishing between M2 and M3

would be lost when thos• states were combined . This loss of information may be very important if there is a large difference in cost between the selective training required for people in the M2 state and

the extensive retraining needed for those in M3. This example also

illustrates the potential significance of maintaining the integrity of

the various nonmas~~ry States. If the instructional decisionmaker knew

the p (Ml) with great accuracy and also knew that there were two nonxnastery states, but decided to combine the two states of nonmastery into

just one state , he or she would be throwing away potentially valuable

information. We shall return to this point in the discussion of Figure 5.

The interrelationship between test length and three hypothesized

mastery states becomes even more apparent in Figure 5. For example ,

Graph A shows that the probability of being in state Ml for 80% correct

on a 5—item test is about 0.48. The probability of being in state M2

(shown in Graph B) for 80% correct on a 5—item test is about 0.36.

There is thus a greater chance that a person whose score is 4 out of 5

is in Ml (p U4IIT) = 0.48), instead of 142 (p (M2IT) = 0.36) or M3

(p(143 IT) = 0.16). However, if a score of 80% correct were observed

on a 40-item test, the graphs indicate that a much different decision

would be appropriate. In this case, p(M1I T) equals 0.21, p(M2IT) =

.78, and p (M3 I T) = 0.01. Hence, people scoring 32 out of 40 correct

should be classified as type 142 masters. Also note that a score of

60% for any test length implies that these people should be placed in

the M3 state.

For the data used in Figure 5, the probability of finding Ml type

masters is overall quite low. Instead, for the levels of achievement

demonstrated by obtained scores of 60%, 70%, or 80%, it is more likely

a that such scores were produced by people in mastery states 142 (p(11M2 ) =

0.8) and M3 (p(lIM3) 0.6).

17

• - -- --- - —~~~~~~--~ -—-~~~

__________ ~---— ~~ -- ~~

,-.•--— ••-.-~

— —
~~~~~~~~~~~~~~~~ 

~

— -r -

~~ —--~~~--,- --— —..,- - — ~~~~~~~~~~~ .- -, -----. .~~~- ,-

Graph D in this figure also represents th. probability that a person is in mastery state Ml as opposed to the new (non)aastery state

formed by combining states 142 and 143. In this example , most of the

probabilities in Graph D are lower than in Graph A. A glance beck at

Figure 4 , Graphs A and D , reveals that the combination of states 142

and M3 increased the probability of classifying a person with a given

test score as a type Ml master. Inspection of the trends in Graphs A

and D of Figures 4 , 5, 6, and 7 suggests that the effect of combining

mastery states is to enhance the trend of the uncombined state . That

is, if the probability of being in state Ml is high when the three

states are treated independently , the p (M1IT) will increase after M2

and 1(3 are combined . Conversely , if p (Ml IT) is low when the three

states maintain their integrity, then combining states 142 and 143 tends

• to decrease the pOIlI T) .

Flow-Chart Analysis of How the Bayesian —

Model Was Developed -

The impact of adding a third mastery state to the development of

the model can be illustrated by tracing the logic that is required in

formulating a description of the examinee population . (Refer to accompanying flow chart for a schematic s

~~~ary of this discussion.) The

first question the decisionmaker must ask (and which we considered )

is: Are there two or three states of mastery inherent in the examinee

population (Step A)? If two •tate~ are posited, parameter estimates

for p (Ml) , p(M2), p(lf Ml) , and p(lIM2’ are specified, along with p].ausible test lengths and values for the percent correct (Step B). The output of the Bayesian processing is the probability that a particular

person is in the mastery state , p (Ml IT) (Step D). A unique graph for

each of Figures 1, 2 , and 3 was obtained by holding the prior and conditional probabilities constant while simultaneously varying the test

lengths and percent correct that would plausibly be observed (Step E).

If three states are hypothesized~ parameter estimates for p(Ml) , p(M2) ,

p (M3) , p(l 1141) , p ( 1IM2) , and pUIM3} need to be specified , along with

values for test lengths and percent correct (Step F).

Now if three states are postulated , a second decision must be

made (Step G). It would seem to be usually desirable to determine the

probabilities of a person’s being in each of the three states (Step I).

Having obtained these probabilities for selected values of prior and

conditional probabilities and over a range of test lengths and percent

correct scores, Graphs A , B, and C can be drawn such as those shoWn in

Figures 4 , 5, 6 , and 7 (Step J).

However, in some instances it may be more convenient to comb~

ine

the information known about two of the three mastery states. For example , even though one mastery state and two nonmastery states are hypothesized , the decisionmaking process may require that people be

divided into only two groups--”mastery” and “nonmastery.” In the

present example, states M2 and M3 were combined (Step K). The result

18 -

~~~~~~~~~~~~~

LI ., 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~ - ~~~~~~~~~~~~~~~~~~~~~~~~~ -
- - - -

.-——-~- --. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- 3 ‘

~~~~~~~states? 2

Specify pUll) , p(M2) , p(l IMl) ,

- ; p(11142), test lengths, and

values for percent correct .

C

Bayesian processing .

D I -

~~~~~~

j 

p(MIT) assuming two mastery

states. -

E I a

[Pr epare curves (Figures, 1, 2, 3). J

F _________

Specify p(Ml), p(M2), p043), p(1IM1),

p (l1142), p(1I113) , test lengths, and

value s for percent correct.

i

_ s 

AL

~~~ ~

.4iiree statei\ NO

to be

H 

analyzed? 

K

Bayesian processing . 1 Combine two maste ry I states.

I I

p (Mi IT) assuming three

mastery states and ana- Bayesian processing .

lyzing three mastery

states. “ -

-• 1 p (Ml’ T) assumi ng three

mastery states but analyzPrepare curves (Graphs ing two mastery states.

A , B, C , Figures 4, 5,

6, 7). N1

Prepare curves (Graph D ,

Figures 4, 5 , 6, 7).

- 4 Flow Chart 1

19
--—_—-‘-_—..—,‘--—-—

~

-_-_ _--—.— ‘-,——‘

~~~

——- - -

of Bayesian processing on these combined data is the probability that

a person is in the new mastery state (Step 14) . Iteration of this procedure for various test lengths and percent correct scores over the

same prior and conditional probabilitieS yields Graph D curves, such

as those of Figures 4 , 5, 6, and 7 (Step W i .

The diff erences that result from following each of the three paths

in the flow chart can be seen by comparing Figures 3k , 5A , and 5D. In

each case the prior probability of being in mastery states Ml was set

equal to 0.50 , and the conditional probability that a type Ml master

would make a correct response to an item was set equal to 0.90. Figure 3A corresponds to path A ,B,C,D,E in the flow chart . Figure 5k

corresponds to path A ,F,GH, I,J, and Figure SD corresponds to path

A,P,G,X ,L,M ,N .

In Figure 3A , p(lIM2) — 0.6, that is, a nonmaster has a 60% chance

of correctly responding to an item. However, in Figure SD the nonmastery state is the combination of states 142 and $3, with probabilities

of responding correctly to an item of 0.8 and 0.~~, respectively . The

effect of combining 142 and 1(3 is to create a new (nøn) irastery state ,

where the probability of a correct response is a weighted average of

the values for the uncombined groups. By defining a relatively high

ability intermediate state and then c~~~ining it with a relatively low

state , the probability of being in the highest mastery state is lower

than if that intermediate statS remained undefined . Xn fact, if the

Figure 5 values of the prior and conditional probabilities are valid

representations of the “real” states of mastery , but the values of Figure 3 (which are a simplification of the Figure 5 values) are used for

decisionmaking, then people achieving scores of 80% will be falsely

classified as type Ml masters.

The differential trend between Graphs A and D of Figure 5 is noteworthy, although the absolute magnitude of the trend is rather small.

For different parameter estimates (of prior and conditional probabilities), the effect of combining group. may be much more extensive. Note

also that the information provided in Graph 0 refers only to the probability of a person ’s being in the mastery state and does not directly

show the loss of information about the two discrete nonmastery states

that have been combined . Furthermore, when two mastery states are

combined and contrasted to a third nonmastery state, the changes in

the probability of being in the newly defined mastery state will often

be quit. different from the probability of being in the original mastery state.

It must be emphasized that unrealistic descriptions of the exaininee

population (in terms of number of mastery group.) can cause severe distortions in classification accuracy . For e~~~ple, had the decision—

maker hypothesiz.d only two states when , in fact, training had produced

three fairly distinct states of proficiency, th. results of his analysis

could be highly misleading. Thu., note that the 80% line of Figure 3k

20

a; --

____ - -- 

- 

~~~~~~

--—--—----~ --—-—-——--—- .-- ~~~~~~~-~~~- .- - ~~ Y— — 

-.-- _.-.---- ~~~~~~~~~
._p ~__ ~~ - ~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~ 

ascends ~a more item. are added (i.e., p(M 1IT) increases), whereas the

80% line of Figure SD descends (i.e., p (M 1IT) decreases) as more items

are add.Ø.

caution must also be observed in the opposite case , where one

might be t pted to specify more states of mastery than are actually

pr.s.nt, in an effort to extract more information than i. justified by

the test data.

Th. present 3i~

yesian model is not limited to three mastery states.

Exploratory analyse. have been conducted with up to five mastery states, a 

and it is also hoped that the model can be generalized to deal with continuous distributions.

TEST LENGTH MD MISCLASSIFICATION ERROR

One of the most important questions that must be - answered in designing a training evaluation program is “What is the probability of

F - falsely classifying a person on the basis of a given observed score?”

[ It is also possible to turn the question around and ask “How long must

P a test b., and what score is required for classification decisions to

be made with some specified lower limit of misclassification?”

Figures 8 and 9 demonstrate how the Bayesian model can be used to

answer these two questions. Assuming that the prior and conditional

t probabilities are realistic and f ixed , the important variables are then

test length and cutting score. Suppose that p (Ml) — 0 • 9, p (142) — 0.1,

p (lIMl) — 0.9, and p(1~M2) — 0.6 as in Figures 8 and 1k. In this example , the prior belief that an untested trainee is a master is very

high, p (M1) 0.9. A reasonable question might therefore be “What

score must be observed such that a nonmastery decision can be made with

at least 90% confidence?” (In other words, what data are required to

force a reversal in the prior belief?)

To be 90% confident of a nonmastery decision, p(M2IT) must be

equal to at least 0.90. Sinc* the sum of p(Ml~T) and p (M2~T) equals

1.0, p(M1IT) must therefore not be greater than 0.10. Referring to

Figure 8, a horizontal line crossing the ordinate at 0.10 can be drawn .

This line crosses the curve for a 5—item test at a point corresponding

to 26% correct. The next lowest possible test score is one correct

(20%), so the decision rule is that all persons scoring one correct or

less should be considered nonmasters. The point on the ordinate corresponding to 20% correct on the 5-item test is about 0.05. Hence,

the final decision rule statea that nonmastery decisions based on an

observed score of 1 correct out of 5 can be made with 95% confidence

(1.00 — 0.05 — 0.95). For observed scores lower than the cutoff score,

the confidence in making a correct decision must increase. Continuing

with the present example, the p(M1IT) if zero correct are observed is

___ -

~~ 

-- 

21 

- -

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

-
- 

- - ~~—~~

-- -

~

- -.--—~~~~~~~~~ 

-‘—

~~~~

-.,

~~~~ ~

--

~~

- --,- 

~~~~~~ _ _~~~~~~~~~~~~ -

~~~-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~ ~~~~~~~~~~~~~

- --- ~~~~~~~~~~~ --— .----- - - -—--—. -.-—— -

~

- - - --

~c - - -~

- -

~

-

~ ~

- - - --

~

- - - --- - -‘

- - —--

—~~

I S

t —~~ 

2

I

H ~~~~~~~~~ ‘4‘4

‘4 C. ,

lu -‘

‘4 —~

\

~ II ~

0 ‘- H •

• t - 1 2

I _____________

J

ft. Is ~ ~~~~~ a

I I I

A a ~~

.

d Ed d d a t 2

E 22

- 

-

~k ~ 

~~~~ -~~-~~ ~~~~~~~~~ - -—
r ~uuI _

~~~;;Ir;:— -•’•• - - - ‘

~~~—‘. -,~ —--~~ -- , 

a— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~r”’ —

~~

- ‘—-a-—

~

-- -—-a- - 

~~~~~ - -- - -‘---a--- - -

__

r 

-

~~~~~~~ I ‘ p I I

- I I

I (a -~~~~

-: i

I - 11 -

~~~~

I C)

I — o 4.4

0

Ui

I 

\

~

44.

I 

-

~~

I ‘4

I - 

‘

4

s

\ - R ilL

- 

.

- 

~~~~ I I 

I

~~~~~~23

f ~~~~~~~~

-‘ 

~~~~ ~

liii _________________________________ 

_

~~-t~ 
F __i - - - 

~~~~~~ 

- ~~~~~~~ -

virtually equal to zero . Hence , those persons who get no items riyht

may be classified as type M2 nonmasters with nearly 100% confidence .

A similar analysis applied to the 40—item test curve indicates

that the cutting score should be about 73% correct. The next lowest

possible score to 73% is 28 correct out of 40 items, or 70%. The probability of mastery, given an observed score of 28 correct, is about 0.04.

At such a low value of p(Ml~T) the chances for misclassification using

a 5-item test and a 40-item test are almost the same. However, the observed percent correct at which the nonmastery decision is made for the

two tests is 20% on the 5—item test and 70% on the 40-item test. Superficially, two tests of different lengths would seem to produce the same

decision outcome, and longer tests may not really be necessary for reducing classification error.

To appreciate the benefits gained from using longer tests, we

must examine the entire curve. Note that at 80% correct, the 5—item

test yields a p(M1IT) equal to 0.92. This result means that, on the

average, 8% of the mastery decisions will be in error, since p(M2IT)

equals 0.08. For the 40-item test, the probability of mastery , given

80% correct, is about 0.99. That is, there is only a 1% chance that

an examinee of nonmastery competence would be incorrectly classified

as a master.

A test that distinguishes sharply between masters and nonmasters

is one in which the probability of mastery is close to either 0.0 or

a 1.00 for most obtained scores. On such tests there is only a small

region in which classification error is large. For example, in Figure 8, for the 40-item test the region where p(MlfT) is greater than

0.1 and less than 0.9 extends from 71% to 77% correct. This means that

the probability of misclassification (calling a true master a “nonmaster ,” and vice versa) will exceed 0.10 only when observed scores range

from 71% to 77% correct. In contrast, the region of the 5-item test

curve for which p (M 1IT) is greater than 0.10 and less than 0.90 extends

from about 26% to about 79% correct. Hence, there is a much larger

region for which the probability of misclassification exceeds 0.10.

Therefore , if classification accuracy is to be maximized over the entire range of possible test scores , longer tests are required. Ideally ,

a very long test would produce a step function, for which the probability of a given mastery state would be very close to either zero or

one.

Figure 9 can be analyzed in a manner similar to that for Figure 8.

However , Figure 9 has one outstanding characteristic that merits special

4 attention. If nonmastery decisions must be made with 90% confidence,

and a horizontal line at p (MIT) — 0.1 is drawn , the line does not intersect the curve for the 5-item test. This means that it is not possible to classify a nonmaster with 90% confidence if a 5-item test is

used , given the parameters used in Figure 9. If resource or time constraints are such that no more than five items may be given, and if the

24

- 1

___________________ 

- S~ ~~~~~~ L _ ~~~~~ ,-- ~~~~ - _____________________ ____
— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - a--—— a-~~ - -

parameter values used in Figure 9 are realistic, and if 90% confidence

for mastery decisions are required, then there is no reason to test.

Testing is irrelevant because no matter what score is observed , in- -

cluding zero correct , the decision rule compels a mastery decision to

be made. In fact, for the present values, the probability of mastery,

given zero correct, is equal to 0.21. This simply means that if persons obtaining a score of zero are classified as nonmasters, 21% of

them will be misclassified , on the average.

The implication of these results for performance testing is obvious. Since performance tests are often rather short, it is essential

to recognize the magnitude of misclassification error that can be incurred with such tests. Designing tests that have clear and direct

relation to actual performance is certainly a worthwhile and much-needed

effort. However, reasonable levels of confidence in classifying trainees must not be sacrificed merely for the sake of using conveniently

short tests.

SUMMARY AND CONCLUSIONS

The present simulation study highlights some very pertinent issues

for test developers and educational decisionmakers. The simulated results demonstrate explicitly the effects that changes in the estimates

of the examinee population quality, number of assumed mastery states,

criteria required for mastery classification , and test length can have

on the probability of correctly classifying a particular examinee.

Furthermore, the simultaneous manipulation of combinations of these

parameters can produce drastic and complex changes in the probability

of correctly classifying a specific examinee.

A unique feature of any Bayesian model is the need for “prior ” information . In the present context, this is the estimate of the proportion of masters and nonmasters in the examinee population. The more

accurately that such an estimate can be made, the greater the value in

using a Bayesian approach: “It is this increment in information that

is equivalent to prior observations which permits a reduction in test

length when a Bayesian procedure is used” (Novick & Lewis, 1974, p.

149, italics added). If the number of items or trials that can be

given on a test is constrained (such as the cost associated with firing

live anisunition in tank gunnery or field artillery), then a Bayesian

model may be desirable.

The simulation results also demonstrate that a criterion for mastery (usually expressed as a percent correct of all possible test items —

that could be given) is not invariant across various test lengths. The

significant implication is that the probability of correct classification varies as a function of test length , mastery criterion, and their

interaction . Classification accuracy improves with longer length tests

and with stricter mastery criteria. However , there is a point of

25

-4 ~~~~~~~~

—

c

-

~

~-
-. - ——- - ------ -—--- — ------- --- - -- -- - - - —-- — — — -.----- ----r------— .— .. ‘/-,-_-

~-~~~~~--a— ,a- —-—a-- ‘ ~~~~~~~~~~~~~~~~~~ Vifl ’ ’, r•a-~~~~~~ Tfl , $-a- ,._- . ~~~~~~~~~~~~~~~~~~~~~~~~~ - -- -- - - - -

diminishing returns, for which increases in test length or criterion

strictness yield successively smaller increments in classification

accuracy.

Another unique feature of the Bayesian approach is that it yields

the probability of a mastery state, given or conditional, upon a specific e~*minee ’s test score. Since the mastery state is probabilisti~

cally inferred and not assumed , it is not possible to compute false

positive and false negative error rates. However , the model seems to

be asking the correct question : “What is the probability that a given

examinee is a master, given his test score?” An alternative binomial

model does give the false positive and false negative error rates but

does not give explicit information about a specific examinee. This

is because it assumes a certain mastery state and then works “backwards”

to complete the misclassificatio n rates for that hypothesized mastery

state, instead of using prior data to infer the unobservable mastery

state.

-. 

Hersbman ’s (1971) original formulation of the Bayesian model combined several states of nature into a smaller number of states, under

the assumption that the prior probabilities of the new states were

equal. This assumption leads to the conclusion that it is generally

undesirable to combine states of nature (mastery ) because of the severe

distortions in classification accuracy that arise. In contrast, our

approach was to simply combine the prior probabilities, but not to

equate then as Hershman did. Hence, p041) .25, p (M2) , = .3, and

• p (M3) - .45 would be combined into the values p (Ml) — .25 and p (M2 , 3) =

.75. The effect of this method of combining prior probabilities caused

relatively little change in classification accuracy , compared to the

case where the mastery states were processed distinctly. Our approach

of combining prior information seems more reasonable, since one would

expect that the probability of one state which is not combined with

any other should not be affected when the others are combined . This

may be called an “ independence of states of nature” assumption .

The final rather significant insight to be gleaned concerns the

issue of minimal test lengths that are required when limits for the

probability of misclassification have been specified by the examiner.

It has been analytically shown that a test can be too short to be of

any value in decision3naking , depending upon the misclassification rate

that the examiner is willing to tolerate. What this model does is to

show explicitly the risks involved in using a given length of test ,

once the tolerance for misclassification error has been specified by

the examiner.

26 

-

_________ ________ 

— 

—-—- —-~~~~~~~ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ 
-r -

~~~ ~~~~~~~~~~~ ~~~, -- -

APPENDIX

A COMPUTATIONAL EXAMPLE FOR THREE MASTERY STATES

The following example illustrates the computations necessary for

processing data with the Bayesian model. The values chosen for this

example correspond to Figure 4. Assume that there are three states of

mastery, and unequal prior probabilities for these three states. The

educational decisionmaker must provide estimates for the prior probabilities of master , p (Mi). For this example let us assume the values

to be p 00.) .5; p(M2) = .3; and p(M3) — .2. The decisionmaker must

also provide estimates for the conditional probability of getting any

given test item right, given each mastery state. Use the following

values as the conditional probability of getting an item right, given

a mastery state: p(l~Ml) = .8; p (l~M2) = .6; p(lIM3) = .5. The conditional probabilities of getting an item wrong given a mastery state

are p(OIMl) = .2; p (0 1M2 ) — .4; and p(01M3 = .5.

First we need to calculate the probability that an item is answered

correctly. For the overall population,

S

p(tj = correct) = E p(Mi)p(tj = correct Mi) = (.5) (.8)

i—l -

+ (.3)(.6) + (.2)(.5) = .68.

Likewise ,

S

p(tj = wrong) = E p(Mi)p(tj = wrong i Mi)

i=l

— (.5)(.2) + (.3)(.4) + (.2)(.5) = .32.

We also need to obtain the set of conditional probabilities for the

different mastery states, given that an individual item was responded

to either correctly or wrongly. The general equation is

p(M i

~tj ) = p (Mi)p(tj j Mi)

27

- ——- - ____

- - I

- - - - -.. - - -.- -

~~

- -—~~

— -- - -

~~~~~

--— -— - --

~~~~~~ •~~~ . 
- - 

Substituting the above values yields

p(M l tj = correct) = (.5) (.8) .68 = .588;

p (M2 tj = correct ) (.3) (.6) * .68 = .265; and

p043 tj correct) (.2) (.5) * .68 = .147.

(Note that the sum equals 1.0.) Finally,

p Oll tj wrong) — (.5) (.2) ÷ .32 .3125;

p042 tj wrong) (.3)( .4) + .32 .375; and

p(M3 tj = wrong) — (.2) (.5) + .32 .3125.

If 6 items were answered correctly on a 10—item criterion referenced

N

test , the following it p(Miltj) values result:

i—i

M1 3.9 x lO s

; M2= 6.8 x i0 6

; M3* 9.6 X io 8

.

Finally, the general Bayesian formula yields the conditional probability

for each mastery state given the total test score. For example,

Mi T) (3. 9 x l0~~ ) = 272

(

~

5) 9 [~

(

~

3

~~ x lOj + 

(6.8 x 10 j_ + 

(9.6 x 10

.1 (•5)9 (~3)9 (. 2)~

Similar calculations yield p (M2IT) — .473 and p(M3~T) — .254 .

In order to combine mastery states 112 and 143 into a single mastery

state (which could represent c~~~ining th. two degrees of nonmastery,

Figure 4, Graph D), the following calculations are required. The values

N

for pOll) and ii p(M1~tj) remain the same, .5 and 3.9 x iO~~ , respectively .

j l

The new nonnastery state (M2’) occurs as a result of combining the previous states 112 and 113. Hence ,

j p(M2’) p(M2) + p(143) — .3 + .2 = .5,

~~~~~~~- p (P12’ tj = correct) — p (142 1 tj = correct) + p (113 tj correct) - -

— .265 + .147 .412, and

p (112’ tj — wrong) = p (112 1 tj wrong) + p (113 tj wrong)

= .375 + .3125 — .6875.

28 

~~~~~~~~~ — ---—- —_- -

~~~~ — - _____________
- — ~~~~~~ —,- -- --~ ~~— •.,—- -,--.- fl~~~~ 7~~~ -) ~~~~~~~~~~~~~~~~~~~~~~ __—~

,,---_- - T~~ m~~ ~ - 

~~~~~~~~~~~~ - -. t

~

uuI~~

N

Calculation of -ir r (M2 ’ tj) yields

j=l 

1.09 x l0~~ .

Entering these new values into the general Bayesian Formula , the following values of p (Ml’ IT) and p(M2’ IT) are obtained :

p(Ml’ 

~ = - 

r3.9 x ~~ = .264,

(~5)9 

~~~~~ 

x 10 ) + (1.09 x 

~ 1 (.5) (.5) J -

1.09 x p(M2 IT) = r 4 = .736.

(~ 5) 9 

~~~ x 10 ~ + 

(1.09 x 10

~ (. 5) (.5)

Some interesting properties of the model emerge when an alternative

procedure for combining mastery groups is used . Note that to combine

two mastery states it is not necessary to calculate new values for

p(llM2’) and p(01M2’). However, it is possible to show that these values are weighted averages of p11~142) and p(1~M3), and p(O~M2) and

p(01M3), respectively , where the weights are the relative proportions

of the new state accounted for by each of the previous states. The

calculations follow.

Since p (142) = .3 and p (143) = .2 , state M2 accounts for 60% and M3

accounts for 40% of the new state M2’. Hence , the value of

p(lIM2’) = (.6)p( 11M2) + (.4)p(11M3) = (.6) (.6) + (.4) (.5) = .56 and

p (01M2’) (.6)p(01M2) + (.4)p(01M3) = (.6)(.4) + (.4)(.5) = .44.

Using these new values,

p(tj = correct ) = p(Ml’)p(lIMl’) + p (M2’)p(lIM2’)

= (.5) (.8) + (.5) (.56) = .68 and

p(tj = wrong) = p(Ml’)p(OIMl’) + p(M2 ’)p(0 1M 2 ’)

= (.5)(.2) + (.5)(.44) = .32 .

Finally, p (M2 ’ Il) and p (M2’ 10) may be calculated.

29

- - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ • ~~~~—-~~~
F

p(M2’ Ii) = ~ (M2’)p (iIM2’) (.5)(.56) .412,

and

p0(2’ (0) = p (M2’)p(01M2 ’) = (.5)(.44) — .6875.

These values are the same as those obtained by the simple addition procedure shown above.

This exercise serves to illustrate the effect of combining two mastery states. combining states 112 and 143 creates, in effect , a new description of the examinee population in which only two mastery states

are hypothesized. The parameter estimates for the new states in this

example , are

p~ 41) .5 p042) = .5

p (lIMl) .8 p( 1(M2) = .56.

In choosing to combine groups, the decisionmaker must consider whether

a two—state description of the population with parameter estimates such

as those above is a better representation than the original three-state

descriptions with parameter estimates.

p (Ml) = .5, p042) .3, p(M3) — .2 ,

p( 1IM1) = .8, p( 1(M2 ) = .6 , p(11143) — .5.

30 

- - --- -

~~~~~~ 

~ ~~~~~~ -
— — - - - - .—~- -— - - — -P-- 

~~~~~~~~~~~ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -

DISTRIBUTION

ARI Distribution List

4 OASD (M&RA) 2 IIQUSACDEC. Ft Ord. ATTh: Library

2 HQDA (DAMI-CSZ) I HOUSACOEC. Ft Ord. ATTN : A’TEC—EX—E —Hwn Factors

1 HQOA IDAPE-P$R 2 LSAEEC. Ft Benj amin Harrison , ATTh: Library

1 HODA (DAMA.AR) I USAPACOC, Ft Benjamin Harrison, ATTN : ATCP—HR

I HODA (DAPE-HRE-PO) I USA Comm—El.ct Sch , Ft Monmouth, ATTh: ATSN—EA

1 HODA (SGRD-ID) I USAEC, Ft Monmouth. ATTN: AMSEL—CT-HDP

1 HODA (OAMI-DOT-C) 1 USAEC. Ft Monmouth. ATrN: AMSEL-PA—P

1 HODA (DAPC-PMZ-A) 1 USAEC, Ft Monmouth, ATTN: AMSEL—Sl-CB

1 HODA (DACH-PPZ•A) I USAEC, Pt Monmouth, ATTN: C. F.d Div Br

1 HQDA (DAPE-HRE) 1 USA Materials $ys Anal Agcy. Aberdeen, ATIN: AMXSV-P

1 HODA (DAPE.MPO-C) I Edgewood Anenal, Aberdeen, ATTN: SAREA—BL—H

1 HODA (DAPE-DW) I USA Ord Cv & Sch, Aberdeen , AT1N: ATSL-TEM-C

1 HODA (DAPE.HRL) 2 USA Hum Engr Lab. Aberdeen, ATTN: Library/Dir

1 HODA (DAPE.CPS) I USA Combat Arms Tng Sd. Ft Benning. ATTN: Ad Supervisor

1 HODA (DAFD.MFA) 1 USA Infantry Hum Rich Unit, Ft Banning, ATTh: Chief

I HODA (DARD.ARS.P) I USA Infantry Sd, Ft Benning, ArrN: STEBC—TE-T

1 HODA (DAPC.PAS-A) I USASMA. Ft Bliss, ATTN: ATSS—LRC

1 HODA (DUSA-OR) I USA Air Oaf Sch, Ft Bliss , ATTh: ATSA-CTD—ME

1 HODA (DAMO-ROR) I USA Air Def Sch, Ft Bliss, ATTN: Tech Lib

I HODA (DASG) 1 USA Air Def Sd, Ft Bliss, ATTN: FILES

I HODA (DAIOPl) 1 USA Air Def Sd, Ft Bliss, AIiM: STEBO—PO

1 Chief. Consult Div (OA-OTSG), MeiphI, MD I USA Cmd & General Stf College, Ft Leavenworth, ATTN: Lib

1 Nil Aast. Hum Rec, ODDR&E. OAD (E&L$) I USA Cmd & General Stf College. Ft Leavenworth, ATTN: AT W—SE-L

1 HO USARAL, APO Seattle, ATTN: ARAGP-R 1 USA Ond & General stf Coffegs . Ft Leavenworth, ATIN: Ed Advisor

I HO First Army. ATTN: AFKA-Ol-Tl 1 USA Combined Arms Cmbt Des Act. Ft Leavenworth. ATIN: DepCdr

2 HO Fifth Army. Ft Sam Houston I USA Combined Arms Cmbt Des Act. Ft Leavenworth. ATTN: CCS

1 Dir. Army Stf Sesdies Ofc. Aim: OAVCSA (DSP) 1 USA Combined Arms Cmbt Des Act, Ft Leavenworth. ATIN: ATCASA

1 Ofc Chief of Stf, Stief lee Ofc 1 USA Cambined Arms Cmbt Div Act. Ft Leavenworth, ATTN: ATCAC O-E

1 DCSPER, ATTh: CPS/OCP I USA Combined Arms Cmbt Des Act, Ft Leavenworth. ATm: ATCACC—C’

1 The Army Lib, Pentagon, ATTN: RSB Chief I USAECOM, Night Vision Lab, Ft Belvoir, AIiM: AMSEL—NV—SD

I The Army Ub, Pentagon, A IiM: ANRAL 3 USA Computer Sys Cmd, Ft Beivolr, ATm: Tech Library - 

1 Ofc. AceS Sect of the Army (R&D) I USAMERDC. Ft Belvoir, AiiM: STSFB—DQ

I Tech Support Ofc, OJCS I USA Eng Sch, Ft Belvoir. ATtN: Library

1 USASA. Arlington. ATTYI: lARD-I I USA Topographic Lab. Ft Balvoir, ATIN: Eli—TO—S

1 USA Rich Ofc, Durham, AIiM: Life Sciences Dir 1 USA Topographic Lab. Ft Belvoir, A IiM: STINFO Center

2 USARIEM. Natick, AIIM: SGRD4JE-CA I USA Topographic Lab. Ft Belvolr, AiiM: ETL—GSL

I USATTC, Ft Clayton. ATTN: STETC-MO-A I USA Intelligence Cu & Sch, Ft Huechuce, ATTN: CTD—MS

I USAIMA, Ft Bragg. ATrN: ATSU-CiD-OM I USA intelligence Cu & Sch, Ft Huachuca, Al-rN: ATS-CTD-MS

1 USAIMA. Ft Bragg. ATTN: Marquat Lib 1 USA intelligence Cv & Sch, Ft Huachucs. ATTN: ATSI—TE

1 US WAC Ctr & Sch, Ft McClellan, AflN: Llb 1 USA intelligence Cv & Sch, Ft Huachuca, ATrN: ATSI—TEX--GS

1 US WAC Cu & Sch, Ft McClellan, AIIM: Trig Dir I USA Intelligence Cu & Sch. Ft Huachuce. ATTN: ATSI-CTS-OR

1 USA Duartarme,ter Sch, Ft Lee, ATTN: ATSM-TE I USA Intelligence Cv & Sob , Ft Huachuca, ATTN: ATSi—CTD—DT

1 Intelligence Material Div Ofc, EWL, Ft Holabird 1 USA Intelligence Ctr & Sch. Ft Huechuca, Al-IN: ATSI—CTD -CS

1 USA SE Signal Sth, Ft Gordon. AIIM: ATSO-EA I USA intelligence Cv & Sch, Ft Huachuca. ATIN: DAS/SRD

1 USA Chaplain Cv & Sob, Ft Hamliton, Al-TN: ATSC-TE-RD 1 USA Intelligence Ct.’ & Sob, Ft Huachuca. AIIM: ATSI—TEM

— 1 USATSCH, Ft Eustis, AIiM: Educ Mvieor 1 USA Intelligence Cu & Sch, Ft Huachuce, Al-rN: Library

1 USA War College, Carlisle Barracks, ATTN: Lib I COR. HO Ft Huachuce. Al-TN: Tech Ref Div

2 WRAI R, Neuropeychiatry Dlv 2 CDR, USA Electronic Prvg Grd. AITh: STEEP—MT—S

1 OLI,SDA,Monterey I CDR,PTOjectMASSTER,ATrN:TeCh lnfO Centet

I USA Concept Anal Agcy. Bethesda, Al-IN: MOCA.WGC 1 H~ MASSTER, USATRADOC. LNO

1 USA Concept Anal Aqcy, Bethesda, Al-TN: MaCA-MR 1 Research Institute, HO MASSTER, Ft Hood

— 1 USA Concept Anal Agcy. Bethesda, AiiM: MOCA-JF I USA Revuiting Cmd. Ft Sherdien , AiiM: USARCPM-P

I USA Arde Test Ct.’, APO Seattle, Al-IN: STEAC.MO.ASL I Senior Army Mv., USAFAGOD/TAC. Elgln AF Aux Fin No. 9

1 USA Artic Test Cu, APO Seattle, Al-iN: AMSTE.PL-TS 1 HO USARPAC, DCSPER, APO SF 06568, AIIM: GPPE—SE

1 USA Armament Cmd, Redstone Arsenal, A ITN: AT$K.TEM I $tlmson Lib, Academy of Health Sciences, Ft Sam Houston

USA Armamant Cmd, Rock Island, ATI’N: AMSAP.TOC 1 MarIne Corps Inst. ATTN: Dean—MCI

FAA-NAFEC, Adi ntic City, ATIN: Library 1 HOUSMC. Commandant, Al-IN: Code MIMI 51

I FAA-NAFEC, Atlantic City, AIIM: Hum Engr Sr I HOUSMC. Commandant, Al-TN: Code MPI --20

1 FAA Aeronautical Cv, Oklahoma City, ATTN: AAC-44D 2 USCG Academy, New London, AIM : Admi

~ ion

2 USA Fin Arty Sob, Ft Sill, ATTN: Library 2 USCG Academy, New London, Al-TN: Library

I USA Armor 5th, Ft Knox, ATTN: Library 1 USCG Training Cv, NV, ATTN: CO

I USA Armor Sch. Ft Knox, Al-TN: AtUDl-E I USCO Training Cu, NV, Al-TN: Educ Svc Ofc

1 USA Armor Sch, Ft Knox , Al-TN: ATSB DT.TP 1 USCG, Psychol Re, Br DC, AiiM: GP 1/62

1 USA Armor Sch, Ft Knox, AIIM: AISS-CO-AD 1 HO Mid—Range Br. MC Det, Quantlco, AiiM: P55 Div

31

— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -

~~~~~i~~~~~~~~~~ T- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
________ — —- -—- — 

~———~ r - m ~~~’ -

~

-H

I US Marine Corps Lideion Ofc, AMC. Alexandria, Al-TN: AMOOS—P I Quf I CMI I

~~ si-Su*s 1L$~~T ~~~~~~~~~

I USAT RADOC. FtMoiwos. AIIM: ATRO-ED I CI1lI 1*lgt.i MTh2MI0~0~~

S I*ATRAOOC . 

Ft Monroe, Al-TN: ATPR-AD I WLUU1 iSI ‘~~ IIN ~~~~~~~~~~~~~

I USATRADOC. Pt Monroe, Aim: A fl’S -EA I NIBeY l

~

Ns*t Pei

~~~ L.J. AThS Dsg $0

1 USA Forces Cud Ft Mafherion. ATTN library M*abl ~~~~~~,C,LI?.A ~ A. Tsi sJ

~*1~

$1anae

2 USA Aviation Teat Sd, Ft Rucisar, AIIM: SIESO-PO I 

~~U~ 1iIL~~ 

I

~ ~~~~~~ 681. 65IV

1 USA Agoy for Aviation Safety, Ft Rusher, Al-TN: Library

I USA Agey for Adadon Safety, Ft Rudssr , ATm: Ides Adeleer I PIN I

~~~OIo Uleny. *11*. brad Defense FIIN

I USA ~~~~~~~~~~ 5th. Pt Rusher. ATIM: P0 Oi~~~ 0 i V ~~~~~~~~~~~ 000P/KL AN Seu~~

f

1 H

~~*A Asbaleis lye Cud. It Lassie, ATTN IM$AV-’Z$0 ~

1i

~*~ -1l1~~ 1

~5M~

. TIN ifepar. N4UieddS

2 USA Aviadass $0 lINt Ad., Esiv.vds AlL A’VNI MV65-.T

1 USA Air 014 $0, Pt Ilba, Al-TN: *15* TIM

1 USA Air Mebility Radii & Des l.d’, MoNett FId, ATIRa $*NI4.-A$

I USA Aviation Sali. Ras Trig Mgt, Ft Ru, Al-TN: *lIT -T-4l~~ I USA Aviation Salt, CO Ft Stacker, *11W: *1ST-P-- A 

-

1 HO, DARCOM, Alexandria, Aim: *MXCD-4L

1 140, DARCOM . Alexandria, Aim: COR

I US Military Academy, ~~at Point, Al-TN: Ieriel*&dS

1 US Military Academy, Weet Point. Al-TN: OfcofM~

k l.

~~~~~~~

I US Military Academy, West Po nt, Aim: MAQR

1 USA Standardization Gp, OX, FF0 NY, Al-TN: M5 -GC

1 Ofc of Naval aid’. Arlington, Aim: Cede 462 -

3 Ofc of Naval Rash, Arlington, AIIM: Code 468

1 01⁄4 of Naval Rich, Arlington, ATTN: Code 480

1 Of c of Naval Nsdi, Arlington, Aim: Code 441 - -

1 Nasal Asroapc Mad Rex Lab, Pensacola, Aim: Acoue $0 Dlv

1 Naval A.roepc Med Ret Lab. Pensacola, Aim: Code -1.51

I Naval Aeroipc Med Rex Lab, Pensacola, Al-TN: Code 1.5

I Chief of NawPerv, AIIM: Pest-OR

1 NAVAIRSTA. Norfolk, Al-TN: Safety Car

1 Na’, Oceanographic, DC, Aim: Code 6211, CIwta & Tech

1 Center of Naval Anal. Aim: Doc Ct.’

1 NevAirSysCom, AIM: AlR—5313C

INavBuMed ATTN: 713

1 NavI4e4ieepiei’$ub6~ua 2, FF0 SF 96601

1 AFHRL (PT) William APS

1 AFHRL (TT) Loeev AFB

1 AFHRL (AS) WPAFS. 044

2 APHRL IDOJZ) Sroolu API

1 AFHRL (OOJN) Ladihen d AFB

I HOJSAF (INYSO)

1 HOUSAF (DPXXA)

I AFVIG (PD) Randolph API

3 AMRL (HE) ~~ AFS, OH

2 AF Inst of Teds, 

~~‘AFB. OIl, Al-TN: ENE/SL

1 ATC (XPTD) R.ndolpb AFB

1 USAF AesiMed Lila, Irooka API (SUL—4), ATflI: DOC SOC

1 AFOSR (ML) Arlington

- - 1 AF Log Cud, Md iuilan API. Al-TN: ALC/DPCR$

I Al, Farce fi nij, CO. Al-TN: Dept .f Sal Ian

b NvFW &OevCte, Sen Diego

2 Nary Mad Naucegovdui~~ic Nash Unit, Son Diego

1 Na, EI~te~~.,c L~~, Son Diego, Aim: Rae Lab

1 Ne, TrngCan, Sen Diego, Al-TN: Code S000-’Ub

I NesPosiGraddi . Ne.*.,,,i. AIIM: Code SM.

1 NavPoetGrdich . M..nasr,

~~ Al-TN: Cods 2134

1 NevlmgEqulpCtr. Orlando, Aim: Task Lila

I US Dept of Labor, DC, Aim: Menpossar Adnin

I US Dupe of ,haat*ee. DC. AIIM: Ossig Enforce Adam

I Nat Our of 5s...dade, DC, Aim: CoeiguIsr Info leaden

I Nat Clearing Hoase for MM—Info, Nashville

1 Denver Fuderal Car, L.k,-a’rd, AiiM: SLM

12 Defemo Doasasuentedon Career

4 DIr Psych, Army Hq, Rueedl Ofee, Canberra

1 lelanillIc Aduer, MU Id, Army Hg, fluseeil ONe, Canberra

I Nil and Air Attach., Auaarlan Embaisy

I Canire da Nadsetalie Dee Pacleuss, I4umelne di Ia Defense

National.. Snaseels

2 Canadian Joint Staff Washington

I C/Air Staff, Royal Canadian Al. Al-TN: Per, lid Anal It

3 Chief, Canadian Def Rich Staff, Aim: C/CRO$4W)

4 BritIsh Oaf Staff , British Emt,.py, Washington

32

i - I

- ~~~~ti,- ~

—---- -- ——-- — .----—-— 

- - - - -

Li - - __________

~

--

~ 

______________