H

~~~~~~~ TP1y •‘~ 

‘

~

1 Tec haical Paper 280 FAD 

4

~~~~~~~ 

-

A STR A TEG Y FOR SELECTING A W ORT H

ASSESS MENT TECHNIQUE.

( /

~ 

\ NorwoodP/KnepPreth Wilham~~oesseI ~~~ David H ~~ustafson

~~~ t-i~~---i 

~

(S.~ n.in

Edgar M.~~ohnson

Army Research Institute for the Behavioral and Social Science:

~

z 7

~~ 1fl ,

~~~;~~~;iTT i 

~~~

BATTLEFIELD INFORMATION SYSTEMS TECHNICAL AREA

U. S. Army

H

Research Institute for the Behavio ral and Social Sciences

______ 

/ - —- - . ~-r /

Approved for public releas e; distribution unlimited.

L 78 06 09 Q5~

4 L i . ___________________ ___
Unclassified ARI TI’-. ~O SECURITY CLASSIFICATION OF T pSiS PAGE (W~

,w 0.,. EnV.~.d)

DE~~~~~ rIflrIIAAEkJTATIS’SU DAr .

E READ INSTRUCTIONS

‘

.#r~ U ~~~~~~~ r’i U r

~~’ i’.a; BEFORE COMPLETrNG FORM

IT REPORT NUMBER 2. GOVT ACCESSION NO 3. RECIPIENT ’ S CATALO G NUMBER

Techn ical Paper ~‘

~0

4. TITL E ( i d SubeIl.) S. TYPE OF REPORT & PERIOD COVERED

A STRATEGY FOR SELECTING A WORTh ASSESSMENT Final

TECHNiQU E 4. PERFORMING ORG. REPORT NUMBER

7. AUTHOR(.) C. CONTRACT OR GRANT P4UNBCR(a)

N. P. Kneppreth , W. Hoessel , D. H. Gustafson ,

and B. M. Johnson DAMC19-73-G-0008

5. PERFORMING ORGANIZATION NAME AND ADDRESS IC. PROGRAM ELEMENT, PROJECT , TASK

AREA C WORK UNIT NUMBERS

University of Wisconsin ~~ O1A

Mad ison , Wisconsin ‘53706 2

~~621~1A75J4

II. CONTROLLING OFFICE NAME AND ADDRESS IS. REPORT DAT E

February 1978 ~~~~~

U . S. Army Intelligence Center and School ~~ NUMB EROF WAGES

Fort Huachuca , AZ ~~b13 38

14. MONITORING AGENCY NAME 6 AOOR ESS,Udiff.r.i f from Conf,olfln4 Qific.) 15. SECURITY CLASS. (of U.S. r.porl)

U. S . Army Research Institute for the Behavioral

and Social Sciences , ‘001 Eisenhower Avenue Unclassified

Alexandria , Virginia 2’ ~33 “a OECLASS IFICATION /OOWNORAOING

¶6 . DISTRIBUTION STATEMENT (of U.S. R.pofl)

Approved for public release , distribution unlimi ted .

17. DIST RISUTION STATEMENT (of U.. ab.ir.ct .nt. ,.d I., Stock 30, If dtlf.,.nt Sroo, R.po~

t)

II. SUPPLEMENTARY NOTES

I. KEY WORDS (Co&Snu. on rave,., aid. If n.c... y mid ld,niliy by block numb.,)

Worth assessment techniques Selection strategies

Decisionmaking strategies Subjective evaluation

Selection of worth assessment techniques Judgment

Delphi procedure Utility theory

- 

l&. A~~~T~~AC1 ~~~~~~~~ ,.v~~mi .1~~ S ~~~~~~~~~ d Sdmif lip b

~ block n. ba,)

A variety of techniques ejcist for evaluating and assessing subjective

worth. The present Technical F~per develops and presents a strategy for

selec ting the most appropriate worth assessment technique for a given situation

A three-round Delphi procedure was used to systematically solicit and

collate informed judgments from practitioners and experts in the field of

worth assessment. —~~~,

DO IJ*N 71 W3 P~~~~ o.s WOV8I lSO O~ ETE Unclassified SECUMTY CLAUIFICATIOR OF ThIS RAGE (Ubmi O~~a

78 oe 09 0
ARI TP-28’O

k Unclassified SECURITY CLA SSIFICATION OF THIS PAGE(W ~,w Data Sni.r.d)

20. -

.—~~-The first Delph i questionnaire identified 41 factors influencing the selection and successful -

~

ise of worth assessment techniques.

The second Delphi>~uestionnaire was used to determine the relative importance of the 41 factors identified in the first round of the Delphi. Ten

composite factors, four dichotomous and six continuous , were developed which

appeared to account for the 41 factors and their relative weights.

The third Delphi questionnaire was used to jdevelop a prof ile for m4ee / /

different worth assessment techniques on the s~3,continuous factors . A -ehr~e—

~~

- step strategy, based on the 10 composite factors, was developed for selecting a

specific worth assessment technique for a given situation .

The strategy developed for selecting a worth assessment technique should ~.

-

serve to increase the successf ul application of wor th assessment t4chniques to

specific situations , such as occur in the appraisal of intelligence information .

•

0’

White Section ~ B f Section 0

BY ——

OI~

J i ~~J

~ ~

?

~TY r9~

ES

Unclassified

SECURITY CLASSIFICATION OF THIS PAGE(Wb.n Data Enf.,.d) 

— .

_- __ 
ARI T1’-.’

~°

T.chIcaI Paper 280

A STRATEG Y FOR SELECTING A WORTH

ASSESSMENT TECHNIQUE

- - Norwood P. Kneppreth , William I-b essel , and David H. Gustatson

University of Wisconsin

and

Edgar M. Johnson

Army Research Institute for the Behavioral and Social Sciences

Robert S. Andrews, Supervisory Project Director

BA ITLEFIELD INFORMATION SYSTEMS TECHNICAL AREA

Robert S. Andrews , Chief

Submitted by: Approved By:

Joseph Zeidner , Director J. E. Uhlaner

ORGANIZAT IONS & SYSTEMS TECHNICAL DIRECTOR

RESEARCH LABORATORY

U. S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office , Deputy Chief of Staff for Personnel

Department of the Army

February 1978

Army Project Number Intelligence Systems

20062703A754

- tspp, ss.d to, public reissue; disirtliutlon uniin~ted. 

.

..-~~~~~~~~~~~ ____
.:—~~

:

~~~— :‘ 

~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~ ~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ :-

~~~~-~~~~~~~~~~~~~~~~~~~ 

‘.

~~~~~~~~~~~

ARI TP-2~0

ARI Research Reports and Technical Papers are intended for sponsors of

R&D tasks and other research and military agencies. Any findings ready for

implementation at the time of publication are presented in the latter part of

the Brief. Upon completion of a major phase of the task, formal recommen

dations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

U: -.. - -
ARI TP 280

FOREWORD

The Battlefield Information Systems Technical Area of the U. S. Army Research Institute for

the Behavioral and Social Sciences (AR I) is concerned, in part, with human information analysis ,

interpretation functions , and subsequent utilization of the products in intelligence systems.

The objective is to provide technological advances in human/machine- aided tactical intelligence

information processing and the translation of these advance s in support of intelligence systems

requirement and design decisions, and formulation of doctrine and procedures.

Achievement of the required technology is often inhibited by a lack of understanding of

fundamental principles. Where this is true, the requirement is to increase the scientific basis

underlying the state of the art.

One such area is the assessment of subjective value or worth, which has been addressed in ARI

Technical Paper 254, “Techniques for the Assessment of Worth .” A functional assessment

technology is needed to support efforts such as determination of the value of intelligence data and

improvement of intelligence collection procedures.

The present publication prese nts a strategy for selecting the most appropriate worth assessment

technique for a given situation. The research effort is responsive to requirements of RDTE Project

2006210IA754. “Intelligence Information Processing.” and to special requirements of the U. S.

Army Intelligence Center and School.

ARI research in this area is conducted as an in-house research effort augmented by contracts

with organizations selected as having unique capabilities and facilities for research in a specific

area. The present study was conducted jointly by personnel of the Army Research Institute and

the Industrial Engineering Department of the University of Wisconsin at Madison.

/ - Technical Director

- 

. ; _- ~~~~~~~~~~~~~~~~~~~~~ - 
— —.. .—,.-..-—-.- ., _-_-—,.- ——--v -. -— ~~

—..-.-, 

~~~~~~~~~~~~~~~~~~~~~~~~~~

- -~~~—.

~ - .

P

ARI TP 2MO

A STRATEGY FOR SELECTING A WORTH ASSESSMENT TECHNIQUE

BRIEF

Requirement

To determine the factors influencing the outcome of worth assessment and to develop a

strategy for selecting the most appropriate worth assessment technique for a specific

decisionmaking problem.

Procedure

A large number of techniques of worth assess ment a term applied to the analysis of subjective

value judgments and their reduction to a quantitative scale . have been developed. In the present

analysis, a strategy was developed for selecting a worth assessment technique for specific

application based on a 3

~

round Delphi procedure.

In the first round, 41 factors influencing the selection and successful use of worth assessment

techniques were identified by a questionnaire distributed to analysts active in the worth

assessment field.

The second Delphi questionnaire was used to determine the perceived relative importance of

the factors. Ten composite factors, four dichotomous and six continuous, appeared to account for

the 41 factors and their relative weights.

Reference profiles for nine techniques were developed for the six continuous factors , based on

a third Delphi questionnaire.

Findings

The research product is a three~step strategy for selecting a worth assessment technique fot a

given situation, based on the ten composite factors.

The first step is the analysis of the four dichotomous factors v ’ srt...

~ • ‘~~

t i.’diniques not

suited to the situation. The second step is an evaluation of the siIi, ~~t

~,wi ~ I i

’

,iti,,i,rnents (attributes )

for a technique, based on the six continuous factois

Finally, the situation profile ~ eisnt,

aii’iI *ttt ’ the reference profiles to determine which of the

nine techniquss ~ rn~ st mp$WIISH at. tot me given situation.

p . _________________________________________

_____________ 

- . .—-— - - 

- - 

- -
___________ 

~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~

-

P

ARI TP 280

Utilization of Findings

Worth assessment is a vital ingredient of advanced intelligence information processing. The

strategy developed cannot be said to insure selection of the perfect technique. However , such

strategy should serve to increase likelihood of success in worth assessment application by leading

the user to consider situational factors and attributes of available worth assessment techniques.

~~~~~~~~~~~~~~~~~~~~~~~ . 

~~~~~~~~~~~~~~~~~~~~ . . - .
ARI TP 280 

A STRATEGY FOR SELECTING A WORTH ASSESSMENT TECHNIQUE 

Title 

METIIOD 

TilE DELPHI PROCEDURE • 

Round 1 

Round 2 

Round 3 

TABLE OF CONTENTS 

A STRATEGY FOR SELECTING A ' WORTH ASSESSMENT TECHNIQUE 

PROCEDURE FOR SELECTING A WORTH ASSESSMENT TECHNIQUE 

Step 1 

Step 2 

Step 3 

DISCUSSION AND SUMMARY . 

APPENDIX A. EVAWATION QUESTIONNAIRES FOR ASSESSING SI'ruATIONAL 

REQUIREMENTS FOR ASSESSMENT TIME, TRAINING TIME, 

FACE VALIDITY, ACCURACY, FLEXIBILITY, AND 

1 

2 

3 

4 

5 

7 

7 

9 

11 

13 

COMPLEX! TY A -1 

DISTRIBUTION LIST 

LIST OF FIGURES 

1. Brief Descriptions of Nine Worth Assessment Techniques 

2 . The Relative Weights of Critical Factors in 

Worth Assessment · · · · · • · · · · · · 

3. Attributes Used in the Selection Strategy 

4. Flow Chart Summarizing the Strategy for Selecting a 

Worth Assessment Technique 

5. Blank Situation Profile ..... . 

B-1 

15 

l6 

17 

18 

19 
ARI TP 2 

FIGURES (continued) 

Reference Profile of Constant Probability Wagers . 

1· Reference Profile of Variable Probability Wagers 

Reference Profile of the Indifference Curve 

Method . . • . . . . . ... . . 

Re ference Profile of Ranking Methods 

10. Refe rence Profile o.f Method of Paired 

Comparisons . • . . . . . . . 

11. Reference Profile of Equivalence Grouping 

12 . Reference Profile of Scaled Reaponse (Double 

Anchor) Method . . . . . . . . . • . • . . . 

13. Reference Profile of Scaled Response (Single 

Anchor) . . . • • . . . . • . . . • . 

14 . Reference Profile of Graphic Method 

15. Instructions for Weighting the -Method Attribute s 

LIST OF TABLES 

1 . Dichotomous Attributes of Assessment Techniques 

2 . Sample Calculations for Weighted Aggregate Approach 

Page 

1 

2 

2) 

2 

2 

10 

1 
-

~~~-~

AR 1 TP 28O

A STRATEGY FOR SELECTING A WORTh ASSESSMENT TECHNiQU E

Subjective worth forms the basis for the selection of future alter-

• native s and the evaluation of pas t actions. Techniques for measuring

subjective worth extend back to the eighteenth century .

In the last ~C y e ars , there has been an exp losive growth in the

research on the theory of subjective worth and a proliferation of assessment or scaling techniques in this field . This Technical Paper

addresses two related questions.

Firs t, what specific factors are like ly to affect the success of a

worth assessment techni que ? Second , wha t strategy can be used to select

an appropriat e wor th assessment technique for a specif ic app lication ?

There are fac tors which not only differen tiate one worth assessment

technique from another, bu t also determine the relative success of

techniques in a specific application . An extensive literature search

yielded li ttle information on why a particular technique was used in a

given situation or hew the techni que was chosen.~~

The idea that ~ne technique would be more appropriate in a given

situation while a :sother would be better in a different situation has

not been extensively considered .

METhOD

Given the paucity of information in the literature , the remaining

source of information was the collective knowledge of the practitioners

- and experts in the field of worth assessment , gathered by a Del phi

procedure . The Delphi procedure involves the sequential individual

questioning usually by questionnaire) of a set of experts , interspersed

wi th information and opinion feedback derived from earlier par ts of a

questionnaire program .*~~

Edwards, W . The Theory of Decision Making. Psychological Bulletin,

, 

~~~~~~ ~ , ~70— 1~17 -

‘‘

~

Kneppreth , N. P., Gustafson , D. H ., Leifer , R, P ., and Johnson , E. M.

Techniques for the Assessment of Worth . ARI Technical Paper L’ -.t.,

Augus t 1)’~~ . M~ ~~~

‘‘‘Dalkey, N ., and Helmer , 0. An Experimental Applica tion of the Delphi

Method to the Use of Experts . Management Science, 1

~)t ’~~, ~~~ , 

2- . - -- __________ ______ 

~~- ‘ ‘ —-----—-t~r —----

~-- -,- -
- ._.- .-_ _.- _ . - -

~~~~~~~~~~~~ .~~~~~ ‘~~~~~! —;- - -~~

-

~~ - ---—--- ---- -.~~ - - - .- - - -

p

AR! TP 280

In the present protect , a th ree—round Delphi procedure was us,..~d to

systematically solicit and collate informed ludgnents on factors

influencing the selection of a worth assessment technique .

To keep the stud y within reasonable bounds , nine worth assessment

techniques we re selected for partici pan t consideration . F.ach of the

techniques has several variations and each represents a group of

techni ques wi th common attributes .

A brief descri p tion of each technique is given in Figure 1, while

more comple te descrip tions are available in Kneppreth et al , 1974. In

the presen t d iscussion , the role of worth assessor is differentiated

from the role of decisionmaker (114).

Although in many cases the worth assessor and the PM are the same

person , the two roles represent different perspectives of a worth assessment problem.

The analysis of a worth assessment problem to select a techni que for

a given evaluation is the role of an assessor/analyst. The evaluation

of the worth of a specific level of an attribute is the PM ’s role. Thus ,

from the assessor/analyst ’s viewpoint , the characteristics cf the PM

are factors in a worth assessment problem.

In the following sections , “factor ” and “attribute ” have sligh tly

different meanings . Factor re fe rs to a characteristic or peculiarity

of th e entire decision environment . Attribute is used in the more

restrictive sense , re ferring to characteristics of variables being

assessed — in this case , the assessment techniques . An attribute is a

factor by this definition , but a factor is not necessarily an attribute.

TH E DELPH I PROC EDURE

A three—round Delphi was used to obtain the informe d judgments and

op inions of 48 experts and practitioners in the field of worth

assessment.

Round 1

The first Delphi questionnaire was sent to 48 people and was designed

to determine what factors were important in choosing a worth assessment

technique . The 48 persons in the sample were asked to respond to the

following two ques tions :

2

- 

-

~ ~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~ :~~~~~~:~~~=-~

::-:r.
ARI TP 280

What factors would you consider important in deciding

which worth assessment technique to use and why?

What specific factors would likely affect the success of a

worth assessment session? Give an example of what you mean .

Responses were received from 16 individuals who collectively listed 41

different factors (Figure 2). Anything considered important by a

respondent was included as a factor.

Factors include characteristics of several widely different areas :

the assessor , the problem , the decisionmaker, and the techniques . The

diversity of opinion and the number of factors identified underscore the

complexity of the worth assessment process .

Round 2

The second Delphi questionnair e , sent to the same 48 people who

received Round 1, was designed to determine relative importance of the

factors identified in Round 1. Individuals were provided a list of the

41 factors separated into five categories . They were asked to:

Choose the nine most important factors from the entire list

and indicate (in the space for comments) why you feel these

are importan t , and how they affect selection .

Rank order the nine factors , assigning a value of “1” to

the mos t important and “9” to th e least important .

Individuals were also asked to indicate the two or three most importan t

factors within each of the five categories . Responses were received

from 23 individuals , including 14 of the respondents to the first

questionuaire .

The relative importance of each factor was determined by adding the

inverse rankings that each factor received and then multiplying this sum

by the number of respondents who indicated the factor was important.

The relative weights normalized over all 41 factors on a 0 (least important) to 100 (most important) scale are shown in Figure 2.

The 4 factors in Figure 2 with a 0 relative weight were not

considered critical by any respondent . The 37 critical factors can be

useful in themselves to the practitioner as they elaborate what individuals in the field consider important when assessing worth . Tne knowledge that certain factors may adversely affect the outcome should help

the practitioner anticipate and alleviate problems encountered in an

assessment session.

The diversity of factors considered important Indicates the need for

further research in the area. Many of the factors identified are interdependent , and their empirical relationship to assessment success is

unknown .3
ii,

ARI TP -

It would be usefu l to know the exact dependencies among factors and

the exact relationshi ps between the factors and assessment success. This

information would increase not only the likelihood of successfu l worth

assessmen t , but would enable the development of more precise selection

and assessment strategies.

Round 3

F The p rocedu re was based on the assumption that there are attributes

which differentiate various worth assessment technique s in the context

of a particular decision situation . The ~L1 factors identified in Round

are not satisfactory in this regard .

Whil e some of the factors do relate to attributes of assessment

technique s, they do not differentiate between techni ques. Many relate

to more than one attribute , and several do not relate at all to attributes of the techniques.

Therefore , a set of attributes was inferred to account for the

concerns expressed in the first two rounds of the Delphi and to differentiate among the techniques. By successively partitioning the ~ fac tors Figure 2), six continuous and four dichotomous technique

attributes appeared most appropriate Figure ~

)

These ten attributes appear to~ account for nearly all the factors

listed in Figure :~~, with the principal exception of several relating to

the competence of the assessor . These ten technique attributes are the

basis of the selection strategy developed later. While this set of

attributes is not unique . it accounts for selection differences stated

in the Delphi responses.

The four dichotomous attributes are easy to evaluate , as the leve l

appropriate for each of the nine technique s Figure I) is fairly obvious .

The continuous attributes are more difficult to evaluate and were the

subject of Delphi Round c~

The third Delphi questionnaire was designed to develop a profile of

the six continuous attributes for each of the nine techniques . The

questionnaire was sent to 17 individuals , chosen for their expertise

and their familiarity with all nine techniques. Respondents were asked

to rate each method on each of the continuous attributes. For instance.

in the case of “Assessment Time” they were asked to:

-- — 

• .

~ ~~~~~~~~~~~~~~~~~~~~~~~~~-- -mL~~~~~~__ • 

- 

- 

-
-- _ : _ 7 ~~~~~.

ARI TP 280

Please rate the following techniciues on the relative time

it takes to complete an assessment of a given attribute with

a decisionmaker trained in using the technique . This is a

measure of the face—to—face time an assessor and decision—

maker would spend in assessing a given attribute. It does

not include training time of the decisionmaker. Please

place a 10 next to the technique that you feel takes the

least amount of time , and a 90 next to the technique which

would take the most t ime . Rate the other techniques in

relation to these two.

Responses were received from nine individuals and the resulting profiles

are given in Figures 6 through 14.

A STRATEGY FOR SELECTING A WORTH ASSESS~~NT TECIINIOUR

The 10 decision situation attributes developed in the Delphi

(Figure 3) form the basis of a strategy for selecting a worth assessment

technique . The four dichotomous attributes , summarized in Table 1,

represent characteristics of the decision attributes in the assessment

problem .

The six continuous attributes summarized in the reference profiles - - (Figures 6—14) represent characteristics of the setting of the worth

assessment problem .

The proposed selection strategy Is designed to guide the user

through an analysis of the decision problem in terms of these 10 attributes and to insure that these factors , which might determine the

success of the application of worth assessment , are considered by the

user (Figure 4).

There are three steps in the proposed strategy . The first step

(Block 1 In Figure 4) is the analysis of the dichotomous attributes

to screen out techniques not suitable for the situation .

As an example, if one of the attributes to be assessed in the

decision is discrete and cannot be approximated as continuous , this

difficulty will rule out several of the techniques : the graphic ,

constant probability , and indifference curve methods .

The four dichotomous attributes are analyzed using a series of

questions for the assessor/analyst. After each question , one or more

techniques may be ruled out from further consideration . In some casas

only one feasible technique will remain at the end of this step , and

the analyst need go no further.

S

~~~~~~~~~~

______ 

-

~ 

-- — ~~~~~~~~~~~~~~
~~~~~~~~

•-

~

-.-—

~~

- 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- ARI TP 280

In other cases, It will be apparent that more than one technique will

be needed . For instance , one technique may be needed for dependent

decision attributes , another for the remaining attributes .

If feasible to use more than one assessment technique, subsets of

similar decision attributes should be handled separately in the remaining

two steps of the selection strategy .

The second step (Block 2 of Figure 4) is to determine what values a

technique should have for the six continuous attributes . The different

factors in the situation that define the required levels of each of the

continuous attributes are considered.

For instance , the time pressure of the study , the amount of time

the decisionmaker has available, and the number of attributes involved • will interact to define how quick the technique chosen must be. Each

of the six attributes is a weighted composite of the situational

factors developed in the Delphi that influence the value of the

attribute.

This step yields a profile of the situation and defines a minimally

acceptable technique. By comparing this situation profile with the

profile of each of the techniques , it will be possible to reduce the

• subset of feasible techniques still further.

In the third step (Block 3 of Figure 4), the profile of the situation

is compared with the normative reference profiles of each of the

techniques remaining from the first step . Techniques which match or

• exceed the minimally acceptable profile on each attribute are considered

feasible.

From these techniques , the one that best satisfies other criteria

not yet considered in the analysis can be selected . An example of other

criteria would be the amount of faith that the assessor/analyst has in

the technique.

The main strength of the strategy is not that it insures a choice

of the “perfect” technique. The chief strength is rather that it guides

an analysis of the decision situation to force consideration of factors

important in the choice of a technique .

All too of ten, the choice of an assessment technique is made on only

one or two factors . Even then, the factors considered are not always

the most critical. In addition , the strategy forces the assessor to

consider all nine of the techniques rather than some restricted subset.

6

— .

~L.._. - .~
F ~~~~ - —,..

~~~

-— -, —-- —

~

ARI TP 280

PROCE DURE FOR SELECTING A WORTH ASSESSMENT TF.CHNIOTT E

Step 1

The four dichotomous attributes are listed below as a series of

questions . If the answer to any one of them is “no ,” go on to the next

• questior ; If the answer is “yes ,” continue with the explanations

following the question .

~1e4tLa~I 1. Are any of the decision attributes dependent?

If the value of an attrib ute depends on the level of another attri—

bute , then the two attributes are said to be dependent. Stated another

way , if the decisionmaker cannot specify values for an attribute without knowing the value of another attribute then dependencies are

probably present.

• The indifference curve method is the only method frequently used to

handle dependent factors . This method could be the immediate choice for

assessing dependent variables . However, the method has several short—

¶ comings which often make it advantageous to modify the situation and use

a different method . A series of procedures may make possib le the use of

another method .

Can the dependent attributes be combined into a single factor?

Combining decision attributes may enable the composite attribute to be

treated as independent. For instance , in a decision to purchase a car ,

fuel capacity (gallons) and fuel economy (miles/gallon) might be

dependent attributes .

Combining them mathematically by multiplication gives a new attribute , fuel range (miles/tankful), that can be asaessed as a single

Independent attribute. If possible to combine dependent attributes In

the situation at hand , combine them and go to question 2.

Can the dependent attributes be assessed as pairs , trip lets . “ r

other combinations? If the set of all meaningful comhlnattc’ns .‘t t he

dependent attributes is of reasonable length , then fi ~‘L ’ he .‘dvantageous

to assess the attribute pairs as a single atr rtF ’nte .

If color and make of car ar~ .ii.pvni~

ent and , for instance , blue is

con side red the best ~~~~ hi at I ’rtl and green the best color for a

Chevrolt~t , a;~

d th ree e,’ It ~rs are being considered for each make of car ,

then the ’ ~ t n eiIt ~r make combinations may be assessed as a single

r ~

b tit t

~ It thi. combination is feasible in the situation , then go t o

lit~~... I I .‘i~ .

~

7

~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ —~ 

- . 

~~

— J
-

~~~~

ARI TP 280

Cart independence he assumed? Even if the attributes are dependent ,

it might be useful to assume they are Independent . If the dependencies

are such that independence is not an unreasonable assumption , e.g., the

assumption will not affect the decision to any great degree , t hen simply

assume they are independent and go to question 2.

If the decision attributes are dependent , and none of the above

p rocedures is possible , then the indifference curve method mus t be

employed. However, in practice this techniciue is limited to the assessment of two continuous monotonic attributes at a time . If more than two

attributes are dependent pairwise , then some more sophisticated technique

than the nine considered here has to be used .

For the situation with only pairwise dependent attributes , the

indifference curve method must be used , and the analyst need go no

further. However , if one or more interdependent pairs , p lus one or more

independent attributes are in the situation , continue to question 2.

~

)tLe4-tion 2. Does the decision situation involve risk?

Some analysts feel that a decisionma ker ’s s

~~ ri~~~e ’ n r ttt .Illte r s

between risky and riskless situat ions . Pi e. . ~

. i.r d14 1

~~ton alt ernative is selected and the outcome ~~~ net kn i ’w~

a . ~~~~~~~~ analvst’ pre fer to - use an assessment technique t h :i ~~~~~~~~~ .oiit’ assessment of risk . If

the situation Invo1 ve~

. ~~~~~ I

~~~~ the two gamble methods should be

favorab ly con~~

fd,~r u e e ct e d onl y if they cl early do not meet the

other rif t’ , I. .

‘t ~~~: 3 . Are any of the attributes nonmonotonicallv related to value ?

An attribute is nonmonotonically related to value when an increase

in the attribute over one part of the attribute ’s range increases the

value to the decisionmaker , but an increase over another part of the

attribute ’s range decreases the value to the decisionmaker. A nonmono—

tonic curve has one br more peaks or humps .

If any attribute is suspected of being rtonmonotonicallv related to

value , then it is advisable to limit the use of the gamble techniques

and to eliminate the indifference curve method for consideration . The

indifference curve method requires monotonicity , and problems in

constructing wagers may arise with the gamble methods.

8

I I

____:

~

i

~~~~ 

.

~~~~~~~~~
ARI TP 280 

Que6tion 4. Are any of the decision attributes discrete? 

If any factors are discrete and cannot be approximated by a continuous function, then the graphic method, indifference curve method, and 

constant probability wagers should be ·rejected. 

The dichotomous attributes of the nine aAsessment techniques are 

summarized in Table 1. 

Step2 

At this point, the analyst should have an indication of how many 

different techniques might be required in the particular situation. For 

instance, it might be known that both the indifference curve method for 

the dependent attributes and one other technique to handle discrete 

factors would be necessary in the assessment process. 

Each subset of attributes must be analyzed separately in steps 2 and 

3. One technique, the indifference curve method, is known. The rest 

of the process would therefore be needed only to determine the beat 

technique for the discrete attributes. 

If there are time constraints, some of the subsets might be combined 

at this point to reduce the time needed for trainin~ deciaionmakers in 

several different techniques. Training can take a great deal of time and 

place a heavy burden on the decisionmaker. If there is only one continuous attribute, it may be useful to assess it by the same method selected 

for the discrete attributes. 

Based on the critical factors in a decision situation, questionnaires 

were developed (Round 3 of the Delphi) to evaluate the situational 

requirements for a worth assessment technique on the six continuous 

factors. 

The Evaluation Questionnaires are shown in the appendix. The 

resulting values of these continuous factors obtained from the questionnaires form a profile of the minimally acceptable worth assessment 

technique which matches the decision situation. The same six evaluation 

questionnaires should be completed by the analyst in order to develop a 

meaningful comparison profile of the particular situation. 

9 
ARI TP :

~o

Table 1. Dichotomous Attribute s of Assessment Techniques

Methods Dependence Risk Nonmonotonicity Discreteness

GAMBLE METHODS

• Constant

Probability ? YES ? NO

Variable

Probability ? YES ? YES

MULTIVARIATE METHOD

Ind ifference

Curve YES NO NO NO

ORDINAL METHODS

Ranking ? NO YES YE S

Pair Comparisons ? NO YES YES

Equivalence

Grouping ? NO YE S YE S

DIRECT METHOD S

Double Anchor ? NO YES YES

Single Anchor ? NO YES YES

Graphical NO YES NO

~~~g, The entries represent answers to the questions: Will the tevhnique handle dependent attributes; does the method involve

risk; will the method handle attributes nonmonotonically related to worth; and, will the method handle disc rete

attributes.

c

10
- 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -

~~

-•- --

~

---—----

~~

-•-•-- -

ARI TP 280

To illustrate the profile of a minimally acceptable assessment

technique , transfer the total scores developed from the questionnaiit~ to the scales shown in Figure 5. Draw vertical lines at the points

corresponding to the total scores , and shade the portion of the scale

to the left of these marks . In this way , the situation profile can

easily be compared with the graphic reference profiles of the nine

assessment methods (Figures 6—14).

)

Figures 6—14 give the reference profiles of the nine worth assessment

techniques considered in the present development . The shaded portion

on the left of each bar on the profiles extends to the mean value

assessed during the Delphi , and constitutes the acceptable portion of

the scale .

The crosshatched portion extends one standard deviation from the

mean, and gives an indication of the relative agreement among the

judges . In this step , the situation profile develooed in Step 2 is

compared to these profiles , and the most suitable technique is chosen .

This step in the strategy is divided into three parts .

P~vt.t I c’~

ç S~te.p 3. The output of the comparison part of Step 3 will

be a siibset of techniques that matches or exceeds the situation profile

on all factors . This subset is referred to as the feasible subset and

stay contain no techniques , or one or more techniques . If any techniques

were rejected in Step 1, disregard these techniques throughout this step .

Compare the situation profile with the profiles in Figures 6 through

14. In each comparison , the technique should be rejected if it does not

match or exceed the situation profile on all six factors . If the technique matches or exceeds the profile for all factors , the technique is

considered part of the feasible subset. Only the shaded part of the

profile is considered in the comparisons .

When this process is completed , the feasible subset may contain only

one technique : This technique is the one considered most appropriate by

the proposed strategy for the decision process under study , and the

formal strategy ends here. However, the analyst should go back and

check the selection against intuition . In some cases , there may be no

techniques in the feasible subset , all the techniques having at least

one factor which does not match or exceed the theoretical profile . In

that case , fur ther analysis is necessary . Omit part 2 and continue with

part 3 of the step . If the feasible subset contains more than one

technique, go to part 2.

__ _ 

___ _ 

11 

______________ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~ ~~~~

‘

~~~~~~~~~~

-- - .. _

—-
- 

~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~

ARI TP 280

P~

rtt 2 ~~ Step 3. This part is for selecting from a group of

feasible techniques . If the feasible subset contains several techniques ,

the fi nal choice of te chn ique mu st b e made on the basis of additional

criteria , since any one of the feasib le techniques in the subset is

probab ly appropriate for the situation. As an example of such criteria , the analyst mi gh t choose the technique that :

1. he has the most faith in and is most familiar with, or

2. will provide the DM wi th the bes t feedback , or

3. the DN has used before , or

4. allows for easy experimenting and value changing by the DM.

Possible criteria are numerous , and the choice can he made on any

criteria that the assessor deems important. If no additional criteria

are deemed important , and if a choice still cannot be made, part 3 of

Step 3 can be followed to make a final selection .

Pcvtt 3 ~~ Step 3. A method of choosing a technique on the basis of

a weighted aggregate model is appropriate in situations in t’thich either

no feasible technique was found in part 1 of Step 3 or no choice among

feasible techniques could be made.

First , the method attributes must be rated to determine their relative importance in the given decision situation . Figure 15 gives direc—

• tions for developing attribute weights .

When weights have been determined for each of the attributes , one

may determine a score for each technique by computing the weighted sum

of the deviations of each reference profile from the situation profile .

For each technique being considered , determine the deviation from

the situation profile , multiply each of these deviations by the appropriate attribute weight , and finally sum these across all the attributes

of each technique .

A deviation is considered positive if the technique is rated higher

than the situation profile . A deviation is considered negative if the

technique is rated lower than the situation profile .

Table 2 shows an example of these calculations for one technique .

The technique that has the highest score is chosen since this result

indicateti the least undesirable technique .

12
~~• T T

~ 

T

~~~~:T- z- --~ ~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~

• ART TP 280

Table 2. Sample Calculations for Weighted Aggregate Approach

Method Profile Ratings

Att ributes Situation Techni que ‘A ” Deviation Weight Product

Assessment

Time 80 90 +10 10 100

Training

Time 80 75 —5 3 —15

Flexibility 25 25 0 1 0

H Face

Validity 55 20 — 35 2 70

Complexity 75 95 +20 9 180

Accuracy 70 75 +5 6 30

SCORE FOR TECHNIQUE ‘A’ +225

DISCUSSION AND SUMMARY

The strategy for selecting a worth assessment technique structures

the analysis of an assessment problem in terms of ten composite attributes . The underlying assumption is that certain factors of the total

assessment problem are critical to success of the worth assessment

process.

The 10 attributes of the strategy represent. a synthesis of 41

factors identified in the Delphi procedure (Figure 2). The success of

the strategy depends on the extent to which the critical factors were

captured in the Delphi and incorporated into the analysis.

A relatively small number of those queried responded to the Delphi

questionnaire . However, even the small sample of respondents in round 3

(9) represented many years of experience in worth assessment by active

practitioners in the field.

~~~~~~~~~~~~~~~ 

-

~~~~~~~~~~

. • •

~~ ~

-

~~~~~~~

- _______
_______________ - ~~~~~~~~ • -.-. —- -- -

~~ - 7

- _ : - - .

L

• ART TP 280

The nine techniques included in the anal\’sfs (Figure 1) represent

the most frequently used assessment techniques . Each of the techniques

actually represents a family of techniques with similar characteristics

(Kneppreth et al, 1974 , op. cit.). Considering additional techniques

would not require a change in the strategy , but would require extenc ing

the procedure to develop further reference profiles .

The 10 attributes of the techniques represent two types of factors

in an assessment problem . The first type , the four dichotomous attributes summarized in Table 1, focuses on the characteristics of the

decision attributes in the assessment problem.

The analysis of the dichotomous attributes is in terms of the

limitations inherent in the application of a worth assessment technique

in any situation : Is the technique feasible for a specific problem?

The second type , the six continuous attributes , focuses on the

setting of the worth assessment problem. The analysis of the continuous

attributes is in terms of the constraints governing the application of

any worth assessment technique in a situation : Is the technique acceptable for a specific situation ?

The 10 attributes do not represent an exhaustive synthesis c f

factors determining the successful application of a worth assessment

• technique . The goal was not completeness , but rather inclusion of the

• critical factors .

This strategy represents an initial attemp t to structure the problem of selection of a worth assessment technique . The strategy is

designed to guide the user through an analysis of the assessment problem

in terms of factors critical in ~he successful application of worth

assessment .

The relative contribution of the diffe rent factors to assessment

success , as well as the scales used for depicting situations in terms of

these factors , represents the authors ’ analysis and insights derived

from the Delphi process and a review of worth assessment techniques

rather than being empirically derived.

Further definition of the relationships noted in the analysis ,

validation , and refinement of the strategy is required. A strategy

that focuses analysis of the assessment problem on factors accepted by

experts as being important in applying worth assessment techniques should

increase the likelihood of successful decisionmaking.

14

______ - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ---, ••-••• —
ARI TP 2 - J

GAMBLE METHODS

CONSTANT PROBABILITY WAGERS involve constructing wagers (usually

with probabilities of 1/2) and then varying the factor levels

until the dm is indifferent between the wager and an alternative ‘sure thing ’. Yields interval data.

VARIABLE PROBAB iLITY WAGERS involve constructing wagers and then

varying the probabilities until the dm is indifferent between

the wager and an alternative ‘sure thing ’ . Yields interval

data.

MULTIVAR IATE METHODS

IND I FFERENCE CURVE METHOD involves the joint assessment of two

attributes by con structin g a p lane of the possible combinations

and then determining indifference points and curves. Can be used

to asse ss two dependent or two independent attributes at a t ime .

Yie lds ordinal data.

ORD INAL METHODS

RANKIN G involves p lacing rank numbers on a form or sorting cards

• into an order. Yields ordinal data.

PAIRE D COMPARISONS involve choosing the preferred attribute levels

in each of a series of pairs . Yields ordinal data.

EQUIVAL ENCE GROUPING involves placing factor levels into a discrete

number of classes representing varying degrees of worth. It may

involve forms or sorting cards into piles. Yields partial

rank order with some interval information .

DIRECT ME THODS

SCALED RESPONSE (DOUBLE ANCHOR) involves identifying the two

extremes of the attribute scale and then anchoring the extremes .

Intermediate attribute levels are then assessed by giving

numbers or drawing marks on a scale. Yields interval data.

SCALED RESPONSE (SINGLE ANCHOR) involves anchoring one attribute

level to a scale and then rating the other levels in relation

to this point (usually by ratios). May involve eliciting

numbers or marking a scale. Yields interval data.

GRAPHICAL METHOD involves drawing directly onto a graph , with one

axis for worth and the other for the attribute levels, a line

which represents the worth function of the attribute. Some

points are usually anchored . Yields interval data.

Figure 1. Brief Descriptions of Nine Worth Assessment Techniques 15

-- - - _

~i~~~~~~_ - ~~1l~
ARI TP 280 

Relative 

Weipt 

Decision Problem/S i~uation and Purpose 

100 The r equired sensiti vity of the results 

2 The decision model 

56 The size of the decision problem 

48 Whether the problem involves risk or not 

41 Whether the worths are invariant over time 

22 Wh~ her the situation is static or dynami c 

18 Whelhe r the purpose is normative or descriptive 

1 rhe degree of accuracy desired in the results 

14 The number of decisionmakers to be assessed 

4 Whether past choice data is available 

2 Whe ther one is modelling choices or processes 

1 Whether alternative s can be compared to a standard 

0 Probability of a unique final choice 

0 The order of processing the attributes 

0 The location of the worth assessment session 

Assessor/Analyst 

5 The expertise of the assessor 

3 The amount of faith that the assessor/analyst has in the t echnique 

14 The assessor's familiarity with the technique 

6 Previous relationship betwP.en the assessor and decisi onmaker , DH) 

4 Motiva tion of the assessor 

Decision Attributes 

100 The number of decision attributes 

3 Whether or not the decision attributes are independent 

15 Whether attributes are nonmonotonica l ly related t o worth 

14 Whether attributes are continuous or discrete and have de finite end points 

7 Whether attributes can be reso lved into a single dimens1on 

2 Whether some combinations of a ttribute levels a re not possible 

Decisionmaker(s) 

75 The ease with which the DM can understand the technique 

70 The i~rtance of che decision to the decisionmaker 

4 5 The infor.nation demanded from the decisionmaker 

28 The value of time expended by the decisionmaker 

23 Whether the DHwill benefit froa giving his/her worths 

1 The sophistication and vocabulary of the DH 

1 (For Group Decisions) Whether the DH will feel that his/her expertise will be recognizP.d 

0 (For Group Decisions) Personality, attitude , and sta us of the DM 

Techniques 

56 Known biases of the technique 

4 5 The cost of administering the technique 

29 The face validity of the technique 

21 The amount of education required to use the tet:hnique 

21 Whether or not the technique allows feedback 

10 Whether the results of the technique can be cross-checked 

6 Whether the technique allows immediate lue changing or experimenting 

Figure 2. i'he Relative Weights of Critical Factors in Worth ~t 

16 
------ ------· .......__._--=-

ARI TP 280 

Dichotomous Attributes 

Independence: A decision attribute is independent if its value 

does not depend on knowing the level of another attribute. 

Risk: A decision problem involves risk if the decision is 

related to probabilistic outcomes. 

Honotonicity: A decision attribute is monotonically related 

to value if an increase (or decrease) in the attribute 

increases (or decreases) the value to the OM over its entire 

range. 

Discreteness: A decision attribute is discrete if only a 

limited number of levels exist. 

Continuous Attributes 

Assess.ant Time: The relative amount of time required to 

coaplete an assessment of a given attribute with a DM trained 

in using the technique. 

Training Time: The relative amount of time required to trai~ 

a DH, with no previous knowledge of the technique, in the use 

of the technique. 

Face Validity: The relative appearance to the nM that his/her 

responses yield useful and meaningful information. 

Accuracy: The relative precision with which the technique 

elicits a mt' s true worth perspective as well as the technique 's 

sensitivity to variations in the OM's worth function. 

Flexibility: The relative ease with which the tec'mique can 

be modified to suit different situations and DHs, and the 

ease with which a DH can modify his/her worth function. 

Complexity: The relative complexity of t he mt' s t ask in using 

the technique; including the abstractness of required responses 

and the number and order of diffe rent activities. 

Figure 3. Attributes Used in the Selection Strategy 

17 
-

~~ —-‘------‘

• ART TP ‘

~~~~~

1 +

Evaluate the dichotomous

method attributes and

determine which of the — — — 

Subset of feasible

methods are feasible for methods.

the situation at hand. 1

2* 3 $

Ana lyz e the sit u ation Compa re f easible met hod

in order to determine profiles with the profile

what att ribute leve ls _____ 

developed f or the situation

are needed in the situation at hand , choosing those

for the continuous method methods that exceed the

attributes, requirements developed .

L No technique More t h.~~ .‘nc Ott. i i ’ t hin Eque’

meets the [CCIiI1 I t ime ste .’! a. ’ t ’t S the

• requirement s ~ •~. a r equirements

Weight the .tt’ t

.1110 tn .t k s t hit ’ t~ ’t. Choose among this

.‘~i t i

~~

’ hi .; s .‘t Nc’ choice group on the basis Done

..t;i ~h it~ d - ‘ ~~ re- possible of personally relevant criteria

Choice is

possible

e

Figure 4. Flow Chart Summarizing the Strategy for Selecting a Worth Assessment Technique

18

- : ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —~~

--- .. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

‘
ARI TP 280 

•••• ... .,, 

ASSISS.IIT 

, .•. I i I i I I I I I I 

•••• 10 so 50 70 90 sllert 

TIAIIIII 

, .•. I I I ' ' I I 

••• 10 30 50 70 90 ...... fACE 

VAll lilY I I I I I I 

••• 10 so 50 70 90 ...... ACCUIACY 

I I I I ' I 

••• 10 30 50 70 90 ...... fLIIIIILITY 

I I I I I I ...... 10 30 50 70 90 

• •• 

CO.PLIIITY I I ' I I I 

10 30 50 70 90 

Figure 5. Blank Situation Profile 

19 
- --_—_-, — .----_ -- --- - - 

— ~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~ 

.-

~

ARI TP 0

CONSTANT PROBABILITY WAGERS

These techniques involve constructing wagers (usually with probabilities of 1/2) and then varying the fac tor levels until the dm is

indiffe rent between the wager and an alternative ‘su r e thing ’ . The

technique yields interval data.

DICHOTOMO US FACTORS :

Risk? The technique involves risk and probabilities .

Dependence? The technique will not handle dependent attributes .

Norimonotonicity? While the technique can theoretically handle

nonmonotonically related attributes , there may be problems

in varying the factor levels and constructing wagers .

Discrete? The technique will not handle discrete attributes.

CONTINUOUS FACTORS :

• The crosshatched portion extends one standard deviation from

the mean (shaded portion) .

• long s hort

ASS E SSMENT ________________________________________________

TIM E I

long 10 30 50 70 90 

short

T R AINING ________________________________________________

TIM E I I

low 10 30 50 70 90 high

FACE 1’

VALID ITY 1 I I t 1

low 10 30 50 70 90 hig h

ACCURACY 

1 I 1

low 10 30 50 70 90 high

F L EXIBILITY

I I I I I

high 10 30 50 70 90 low

CO MPLEXITY

I I I I I

10 30 50 70 90

Figure 6. Reference Profile of Constant Probability Wagers

20

— ~~

- .

~~~~~~~- —‘a—— - —— ~~~~~ - - -. — —~~._ , ‘-_~~_

~~~~;_~~

____.____ — -
1” 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

‘11 - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

ARI TP 280

VARIABLE PROBABILITY WAGERS

This technique involves constructing wagers and then varying the

prob abilities until the dm is indifferent between the wager and an

• alternative ‘sure thing ’ . The technique yields interval data.

DICHOTOMOUS FACTORS :

Risk? Th e technique involves risk and probabilities.

• Dependence? The technique will not handle dependent attributes.

Nonmonotonicity? The technique can theoretically handle

attributes that are nonmonotonically related to worth . However ,

problems may arise in constructing the wagers .

Discrete? The technique will handle discrete attributes .

CONTINUOUS FACTORS :

The crosshatched portion extends one standard deviation from

the mean (shaded portion) .

long short

• ASSESSMENT a—— .

TIME I I

long 

10 30 50 70 90 sh ort

TR A INING ______________________________________________

TIME I f I I I I I I I 1

low 10 30 50 70 90 high

FA C E —a—-

V AL IDITY I I I

low 10 30 50 70 90 high

ACCURACY 

I I I 1

low 10 30 50 7 0 90 hig h

FLEXIBILITY

I I 1 I I I I I

high 10 30 50 10 90 low

C OMPLEXITY

I I I I I I I

10 30 50 70 90

Figure 7. Reference Profile of Variable Probability Wage rs

21 

————

~~~~~~~

. a -
______ 

~~ — -

~~~~

--

~~~~~~~~~

----

~~~~

—..-——•—

~~

-

F

ARI TP ~~0

INDIFFERENCE CURVE METhOD

This method involve s the joint assessment of two attributes , and

is accomp lished by con structi n g a plan e of the possib le combina tions

of the two attributes and then determining indifference points and

curves on this plane . The technique y ields ordi nal data.

DICHOTOMOUS FACTORS :

Risk? This technique does not involve risk or probabilities.

Dependence? This technique handles dependencies however , it is

realistically limited to assessment of on ly two dependent

• attributes.

Nonmonotonicity ? This technique cannot assess attributes which

are nonmonotonically related to worth .

Discrete? This technique is inappropriate for discrete attributes .

CONTI NUOUS FACTORS :

The crosshatched portion extends one standard deviation from the

mean (shaded portion).

long short

ASSESSMENT ________________________________________________

TIME I I I I I I

long 

10 30 50 7 0 90 s hort

TRAINING ________________________________________________

TIME I -I I 1 I I

10 30 50 70 90 low

FACE ______________________________________________________

VALIDITY 1 I I I

low 10 30 50 70 90 hig h

ACCURACY

I 1 I 1 1 I I

10 30 50 70 90 high

FLEXIBILITY 

I I I ~~~~~~~~~~~~~ 1 I I

high 10 30 50 7 0 90 low

COMPLEXITY 

I I I I I I I

10 30 50 70 90

Figure 8. Reference Profile of the Indifference Curve Method

22

I

~~—-~ - —~

--

~~

-- ..~~~ . 

_

.---. _- .-. :
ARI TP *“-

~

0

RANKING METHODS

These techniques involve placing rank numbers on a form or sorting

ca rds in to rank order. The technique yields ordinal data.

DICHOTOMOUS FACTORS :

Risk ? These techniqu es do not involve risk or probabilities.

Dependence ? The technique will not handle dependent attributes.

Nonmonotonicity ? Th e techniq ue s can assess attributes which are

notunonotonically related to worth.

Disc rete? The technique will handle discrete attributes.

CONTINUOUS FACTORS :

The crosshatched portion extends one standard deviation from the

mean (shaded portion) .

long s hort

ASSESSMENT __________________________________________________

TIME I I I I I I I 1 I

long 

10 30 50 70 90 short

TRAINING ________________________________________________

TIME I I I 1 1

10 30 50 7 0 90 low nig

FACE ____________________________________________________

VA L IDITY I I I I I I

10 30 50 7 0 90 low

ACCU R ACY 

F I I 1 I~~~~~~~~~~~~~~~~~~ I I 1

low 10 30 50 70 90 hig h

FLEXIBILITY 

I I 1 1 1 I I ~~ 1

high 10 30 50 70 90 low

CO MP L E X ITY 

I I 1 1 I I 1 1

10 30 50 70 90

Figure 9. Reference Profile of Ranking Methods

— ~ ~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ -
-

~~~~

p

AR1 TI~ .

PAIRE D COMPARISONS

This technique Involves choosing the preferred attribute levels

in each of a series of pairs. The technique yields ordinal data.

DICHOTOMOUS FACTORS :

Risk ? The technique does not involve risk or probabilities.

Dep enden ce? Th e tech n ique will not handle dependent attributes.

No nmonotonicitv? The technique can assess attributes which

ar e nonmonotonicallv related to worth.

Disc rete? The technique will handle discrete attributes.

CONTINu OUS FACTORS :

The crosshatched portion extends one standard deviation from the

mean (shaded portion).

long short

ASSESSMENT ____________________________________________________

TIME ~~ I I I ~~~~~~~~~~~~~~~~~ 1

long 10 30 50 70 90 short

TRAINING __________________________________________________

TIME I I I I 1 I I

low 10 30 50 70 90 high

VALIDITY 

low 1

~

0 

1 

5

’

O 90 h11h

ACCURACY 

1 ~~~~~~~~~~~~~~~~~ 1 1 1

low 10 30 50 70 90 hi gh

FL E X I BPL I TY 

I I 1 I I I I

high 10 30 50 70 90 low

COMPLEX IT Y

I I 1 I I I I

10 30 50 70 90

Figure 10. Reference Profile of Method of Paired Comparisons

24

- -~~ - ~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~ - . ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~ —

~~~~~~~~~

‘

~~~~‘ -:
--~~-

~~~~~~~~~ •~~~~• ~~~~~~~~~ .

~~~~~~~~~~~~~~~~~~~~~

.

p

ARI TP 2~O

11~~l

~ I \‘AI.I - N ( )- GROIJ1’ I

These techniques involve p lacing factor levels into a discrete

number of classes representing varying degrees of worth . They may

involve fo rms or sorting cards into piles . The technique yields

pa rtial rank order with some interval information .

DICH o TOMOUS FACTORS :

Risk? The technique does not involve risk or probabilities.

Dependence? The technique will not handle dependent attributes.

Nonmonotonicity ? The technique can handle attributes which are

nonmonotonically related to worth.

Discrete? The technique will handle discrete attributes.

CONTINUOUS FACTORS :

The crosshatc hed portion extends one standard deviation from

the mean (shaded portion).

long short

ASSESSMENT ________________________________________________

TIME 1 1 1 1 I 1 1

tong 10 30 50 70 90 s hort

TRAINING ________________________________________________

TIME I I 1 I

10 30 50 70 90 L low uI9~

FACE ________________________________________________

VAL IDITY I I I I

10 30 50 70 90 L low ~

ACCURACY 

1 1 ~~~~~~~~~~~~~~ I I I I

10 30 50 70 90 h• h

FLEXIBILITY 

I 1 I 1 I I I 

~

1

hig h 10 30 50 70 90 low

CO MPLEXITY 

I I J I I i

~

o 

~~~ 

I

Fi gure 11. Reference Profile of Equivalence Grouping

25

~~~- ~~~~~~~~~~~~~~ - -

~~~~~~~ ---,~ •
~~~~

-

~~

-

~~: 

-

. •— - --

~~ 

—

-

~—-~~ - 

~~~~~~~~~~~-r-—---.~ 

-- -.------------- — - -

ARI TP 2

~

-U

SCALED RESPONSE (DOUBLE ANCHOR) METHOD

The technique involves anchoring the extremes of the worth scale

to the extreme attribute levels and then assessing intermediate levels

in relation to these two points either by giving numbers or drawing

marks on a scale. The technique yields interval data.

D ICHOTOMOUS FACTORS :

Risk? The technique does not involve risk or probabilities.

Dependence? The technique will not handle dependent attributes .

Nonmonotonicity? The technique will handle attributes which

are nonmonotonically related to worth .

Discrete? The technique will handle discrete attributes .

CONTINU OUS FACTORS :

The crosshatched portion extends one standard deviation from

the mean (shaded portion). -

long sh ort

ASSESSMENT ________________________________________________

TIME I I I I I .r

~

Iv

~~~ 1 I

long 10 30 50 70 90 short

T RAINING __________________________________________________

TIME I 1 I 1

low 10 30 50 70 90 high

FACE ____________________________________________________

VALIDIT Y I I I 1 I I

10 30 50 70 90 low hIgh

A C CURACY

I I I I I 1

low 10 30 50 70 90 high

FLIXISILI T Y

I I I I I 1 I I

hig h 10 30 50 70 90 low

COMPLEXI T Y

I I I I I 1 1

10 30 50 7 0 90

Figure 12. Reference Profile of Scaled Response (Double Anchor) Method

26

— .• • —w - ,, - -
r

~~

-

~~ ’~ - - r ~~~~~~~ T’

~ 

-- 

~~~~~~~~~~~T~~T :—-- -- --

ARI TP 7-~

o

SCALED RESPONSE (SINGLE ANCHOR)

This technique involves anchoring one attribute level to a scale and

then rating the other levels in relation to this point (usually by

ratios). May involve eliciting numbers or marking on a scale . The

technique yields interval data.

D ICHOTOMOU S FACTORS :

Risk? The technique does not involve risk or probabilities .

Dependence? The technique will not handle dependent attributes .

Nonmonotonicity? The technique will handle attributes which are

nonmonotonically related to worth .

Discrete? The technique will handle discrete attributes .

CONTINUOUS FACTORS :

The crosshatched portion extends one standard deviation from the

mean (shaded portion) .

long short

ASSESSMEN T I

TIME I I I I I I I 1

long 

10 30_ 50 70 90 short

TRAINING 1

TIME I 1 I 1 I I

low 10 30 50 70 90 high

FA CE 1

V A L IDITY I I I I I I

10 30 50 70 90 ~ L t ow nign

ACCURACY 1

I I I I 1

10 30 50 70 90 L

~

ow nig

F LEXIBILI T Y

I I 1 1 ~ I

high 10 30 50 70 90 low

COMPLEXITY 

10 30 

I I I

Figure 13. Reference Profile of Scaled Response (Single Anchor) •

27
• •

p

ARI LP -

GRAPHIC MFTUOD

This technique involves drawing directly onto .i graph , with one axis

for worth and the other for t-he attribute levels , a l ine wh ich repr esen ts

the worth function of the attribute. ~

- erne points are usually anchored .

The technique y ields interval ~

I a t a .

DICHOTOMOU S FACTORS :

Risk? The technique does not involve risk or probabilities .

Dependence? The technique will not handle dependent attributes .

Nonmonotonicity ? The techniq ue will handle attributes which are

• nonmonotonicall v related to worth.

Disc rete? The technique will handle discrete attrIbutes .

CONTINUOUS FACTORS :

The crosshatched portion extends one standard deviation from the

mean (shaded portio n).

long • short

• ASSESSMENT _________________________________________________________

TIME I I I I I I I

long 10 30 50 70 90 short

TRAINING __________________________________________________

TIME I I I I I I I

low 10 30 50 70 90 high

FA C E _________________________________________________________

VALIDITY I I 1 1 I I 1 1

10 30 50 70 90 low high

ACCURACY 

I I I I I I

~

88~

88

~~ 1

low 10 30 50 70 90 high

FLEXIBILITY

I I I I I

high 10 30 50 70 90 tow

COMPLEXITY

I I I 1 I 1 I I

10 30 50 70 90

Figure 14. Reference Profile of Graphic Method

28

-

~~~

-.

~~~~~ i~~TI ~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~ —
F

~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

— -

• 

-

~~~~~~

-

~~~~~ ~~~

- 

~

— -

~

— - •.

~~~~~~

—

~~

-- - •• ... -

~

—_-

~~~

- -- - -- 

~~~~~

- - - - •

ARI TP 280

Read the list of attributes below and choose the one that you feel

is least important in your decision situation . Assign a score of 1 to

• this attribute. Now choose the one which seems to be most important and

assign a 10 to it. These two numbers form the endpoints of a scale of

importance . Please rate the other four attributes in relation to the

two already specified . These numbers become the weights for the attri-- • butes for part 3 of the strategy .

SCORE ATTRIBUTE

_____ 

Assessment Time — the amount of time available for assessment

sessions.

_____ 

Training Time — amount of time available for training the dm.

_____ 

Face Validity — the amount of faith the dm must have in the

method .

_____ 

Accuracy of Results — the required specificity of the results.

_____ 

Flexibility — the number of variations required .

_____ 

Complexity — difficulty of the din’s task.

Figure 15. Instructions for Weighting the Method Attributes

29

‘ 1 

___ ___________ ____ ____ _________
—~~~~~~~~~~~ -

~~ 

-.-- 

_ -

~

. - . -

~ 

-•-

H •

ARI TP 2~O

APPENDIX A

EVALUATION QUESTIONNAIRES FOR ASSESSING SITUATIONAL REQUIREMENTS

FOR ASSESSMENT TIME , TRAINING TIME , FACE VALIDITY ,

ACCURACY , FLEXIBILI TY, AND COMPLEXITY

ASSESSMENT TIME

Score

Time Pressure : Ra te on a zero to 20 scale the relative time

horizon of the decision process. Rate as zero a situation

in which there is no time pressure to make a decision.

Rate as 20 a situation in which the decision must be made

as quickly as possible _____

Decisionmaker (DM) Time Available : Rate on a zero to 30 scale

the amount of DM time available. Rate as zero when there

is an unlimited amount of DM time. Rate as 30 when there

is little time available. Time available will depend not

only on the physical constraints of the DM but also on

the DM ’s motivation. If the DM is unmotivated , he/she may

not be willing to give much time to the assessment . _____

Number of Decisionrnakers: Rate on a zero to 20 scale the number

of DMs to be used in the assessment process. Rate as zero

when there is only one DM. Rate as 20 when there are five

or more DMs . If more than one DM are going to be treated

as a group and there will be one assessment session for

• the group , rate this as a 7 _____

Number of Attributes: Rate on a zero to 30 scale the number of

attributes to be assessed . Rate as zero when there is

only one attribute to be assessed . Rate as 30 when there

are 10 or more attributes to be assessed _____

TOTAL SCORE (sum of the above) _____

Other Considerations: The total score can be modified slightly depending

on several considerations. For instance , if a decisionmaker is not

sophisticated and likely to be slow with worth assessment , then

increase the total score. Or, if there is little money to pay the

DMs for their time (if this is the case), then the total score also

should be increased . If any other characteristics of the situation

tend to make time critical, Increase the total score and if any

characteristic makes time less critical, decrease the total score.

A .1

________________ — -

~~ -- — -~~~~-~~~~~ . - -

• 

• ~~~~~~
AR! TI’ ~~~~~

TRAINING TIME

Score

Time P ressure : Rate on a zero to ;

~~) scale the relative time

ho rizon of the decision process. Rate as zero a

sit uation in which there is no time pressure . Rate as

a situation in which the decision must be made as

quick ly as possible - - - -

Decisionmaker Time Available: Rate on a zero to fl~

) scale the

amount of t ime that the DMs have available. R’ite as zero

a situation where the DMs have an unlimited amount of

t ime to spend . Rate as 20 a situation where the DMs have

ve ry little time available . If the I)Ms ar e very unmotivated , t raining may take more time and t h e value here

should be increased 

Numbe r of Decisionmakers: Rate on a zero to ~~~~ scale the rela tiv e

numbec of decisionmakers . Rate as zero a situation where

only one DM need be trained . Rate as 30 where five or

more DMs are to be trained , if the DM s are to be t rained

as a group then rate th is factor a 10 

• Numbe r of Me thods: Rate on a zero to ~

O scale the relative

number of methods that will be emp loyed in the assessment

process. This is a measure of the number o techniques to

be emp loyed for the different types of attributes. Rate

as zero a situation where only one technique will be

used . Ra te as ~~ a si tuation where four or more techni ques w ill he used _____

TOTAL SCORE (sum of the above)

Othe r Considerations : The total score can he modified based

on seve ral other considerations. If the DM is not familiar :11

all with worth assessment or if he/she is not sophisticated in

these types of techniques , training may take longer and the ’

value of the total score should be increased . If there are

factors which , in your situation , will decrease’ the time for

training, decrease the total score appropriatel y.

A- ;’

________________________ —- ~~~~ .

— — 

- —-

- - ~~~~~ ~~~~~~~~~~~~~~~ — 

~~~~~~~~~~~~~~~~~~~~~~ -_ .— ~~

.—.—• • ~~• :_ __• _ -
ARI TP 280

FACE VALIDITY

Scor e

Past Contac t Between DM and Assessor: Rate on a zero to 50

scale the relative degree of faith that the DM has in - the

assessor. It will depend on past contacts with the DM ,

and the reputation of the assessor as perceived by the DM.

Rate as zero a situation in which the DM has complete

faith in the assessor. Rate as 50 a situation in which the

DM distrusts the assessor. Rate as 20 if the DM and the

assessor have not met -

~~~~~~~~~~~~

Importance of the Decision to the DM: Rate on a zero to 30 scale

the relative importance of the decision to the DM. Rate

as zero a situation in which the DM does not consider the

decision important. Rate as 30 a situation in which the I)M

feels that the decision is very important and wants to

ensure that his/her value is assessed validly 

— Status of the DM: Rate on a zero to 20 scale the relative status

of the DM. Rate as zero a situation in which the DM is of

low status and has been “ordered” to take part in the

assessment. Rate as 20 a situation in which a high status

• person who is taking part in the assessment of his/her own

free will 

TOTAL SCORE (sum of the above)

Other Considerations : The education and sophistication of the

DM might affect this factor. A highly sophisticated PM

4 may require the technique to have a higher face validity

(or possibly a lower face validity~ . In any case if it

is felt that a higher face validity is needed for one

reason or another , the total score may be increased .

I

I

I

I

A- 3

______________ • -—-~~~~~~~~ 

—•_&_.~~~~~~~ -- ~~~~~~~~~~~~~ -; • • 
- 

~~~

=

~~

— 

~

-- - -

~~

---- -

•

H

ACCURACY

Score

Cost of Making an Incorrect Decision : Rate on a zero to 30

scale the relative importance of the decision. Rate as

zero a situation in which the decision is not important

and no adverse effects would occur from making an incorrect

decision. Rate as 30 a situation in which the decision is

extremely critical and the cost of making an incorrect

decision is very high _____

The Model: Rate on a zero to 20 scale the type of measurements required by the decision model to be used . Rate as

zero a model which requires anything less than full rank

order , and rate as 20 a model which requires strict

interval data _____

Number of DMs: Rate on a zero to 30 scale the number of DMs

to be used in the assessment process. If there are many

decisionmakers whose worth functions will somehow be

averaged , then the individual assessments will be less

critical. Rate as zero a situation in which there are many

DMs over which worths would be averaged . Rate as 30 a

situation in which there is only one DM _____

Number of Attributes : Rate on a zero to 20 scale the number

of activities to be assessed . As the decision is based

on more attributes , the effects of any single one will be

reduced . Rate as zero a situation in which there are very

many attributes . Rate as 20 a situation where there are

few attributes _____

TOTAL SCORE (sum of the above) _____

Other Considerations: There are many other considerations which

will affect the required sensitivity of the worth assessment

technique . For instance, if the purpose of the assessment

is to determine how values change over time, it may be

Important to use a more sensitive method . If the situation

requires a more accurate method , increase the total score.

A-4

~~~I~L -- - ,‘ 

- —
____ - 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~

i7iE~

I----~~~~- - ~~~~~~~~~~~~~~ =-- ~~

—• 

~~~~~~~~ “1

AM TP 280

FLEXIBILITY

Score

The Number of Different Variations : Rate on a zero to 40

scale , the number of different variations of assessment

techniques that will be required for diff erent decision

attributes . This is an indication of the number of different types of attributes that the method will encounter.

Rate as zero a situation where the technique will not have

to be modified to suit different attributes _____

The Number of Different DM8: Rate on a zero to 60 scale the

• number of variations of the selected method that will be

required to suit different decisionmakers. Rate as zero a

situation in which no variations will be necessary . Rate as

60 a situation where there are five or more different DMs

• who will require variations _____

TOTAL SCORE (sum of the above) _____

- 

J 

Other Considerations: In general this will not be an

important consideration in the selection of a worth

assessment technique. However, in some situations it is

important to tailor the technique to meet various conditions .

In this type of situation the total score should be increased .

A-5

_____ -- ~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~ - - 

________
• ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

ARI TP 280

COMPLEXITY

Score

Sophistication of the DM: Rate on a zero to 50 scale the

educational level and quantitative abilities of the DM.

Rate as zero a situation in which the DM is highly educated and unlikely to have trouble with any of the worth

assessment techniques. Rate as 50 a situation in which

the education level is low and there is a good possibility that the DM will have much trouble _____

Motivation of the DM: Rate on a zero to 50 scale the

relative level of motivation of the DM. Rate as zero a

situation in which the DM feels the decision is important

and is highly motivated. Rate as 50 a situation in which

• the DM does not feel the decision is important and is not

• motivated to cooperate with the assessment _____

TOTAL SCORE (sum of the above) . . _____

A-6

_____

_

_ _ 

~~~~1
.T ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~

— - • 7

I

ARI TP 280

DISTRIBUTION

• ARI Dist ribution List

4 OASO IM&RA) 2 HQUSACDEC, Ft Ord , ATTN: Lib rary

2 HODA (DAMI-CSZ) 1 HOUSACOEC. Ft Ord , ATTN: ATEC—EX --E -Hum Factors

I HQOA (DAPE-PBR 2 USAEEC , Ft Renlamln Harrison, ATTN: Library

1 HQDA IDAMA -AR) 1 USAPACDC , Ft Benjamin Harr iso n ATTN: ATCP—HR

1 HODA (DAPE-HRE-PO) 1 USA Comm-Elect ScIs , Ft Monmouth, ATTN: ATSN-EA

1 HODA ISGRD- ID) 1 USAEC . Ft Monmouth , ATTN: AMSEL —CT- HOP

1 HQDA (DAMI-DOT-C) I USAEC. Ft Monmouth , ATIN: AM5EL-PA--P

1 HODA IDAPC-PMZ.A) 1 USAEC . Ft Monmouth , AT1’N: AMSEL—SI--CB

1 HODA (DACH-PPZ-A) t USAEC, Ft Monmouth, An N: C, Fact Dee Br

1 HQDA (DAPE-HRE) 1 USA Material s Sys Anal Agcy, Aberc een , ATTN : AMXSY—P

1 HODA IDAPE-MPO-Cl I Edgewood Arsena l . Aberdeen , A rTN: SAREA- -BL-- H

1 HQDA (DAPE-OW) 1 USA Ord Ctr Si Sch. Aberdeen , ATTN: ATSL—TEM—C

• 1 HQDP IDAPE-HRL) 2 USA Hum Engr Lab , Aberdeen, ATIN: Library /Dir

1 HQDA (DAPE-CPS) I USA Combat Arms Tng Sd . Ft Benning, ATTN: Ad Super,isor

1 HQDA (DAFD-MFA) 1 USA Infantry Hum Rsch Unit . Ft Banning, ATTN : Chief

1 HQDA )DARD-ARS-P) 1 USA Infantry Bd . Ft Banning, ATTN: STEBC—TE--T

1 HODA (DAPC-PAS-A) 1 USASMA . Ft Blis s , ATTN: ATSS—LRC

1 HODA (DUSA-OR) 1 USA Air Oct Sch, Ft Bliss , ATTN : ATSA CTD--ME

1 HODA (DAMO-ROR) 1 USA Air Def Sch, Ft Bliss . ATTN: Tech Lib

1 HQOA IDASG) 1 USA Air Def Bd , Ft Bliss . ATrN: FILES

1 HQDA (DA b -Pt) 1 USA Air Oct Bd , Ft Bliss , ATTN : STEBD—PO

1 Chief . Consult Div (DA-OTSG). Adelphi , MD 1 USA Cmd & General Sti College , Ft Leavenworth . ATTN: Lib

1 Mu Asit. Hum Ret , ODDR&E , OAD I E&LS) 1 USA Cmd & General Stf College . Ft Leavenw ort h , ATTN : ATSW—SE—L

1 HO USARAL , APO Seattle , ATm: ARAGP -R 1 USA Cmd 8i General Sd College , Ft Leavenworth . ATTN: Ed Advisor

1 HO First Army, ATTN: AFKA -OI-TI 1 USA Combined Arms Cmbt 0ev Act , Ft Leavenworth , ATTN: DepCdr

2 HO Fifth Army, Ft Sam Houst oa 1 USA Combined Arms Cmbt Dcv Act . Ft Leavenworth , ATTN: CCS

1 Dir . Army Stf Studies Ofc , ATTN: OAv CSA (DSP) 1 USA Combined Arms Cmbt Den Act , Ft Leavenworth . ATTN : ATCASA

1 Ofc Chief of Stf . Studies Ofc 1 USA Combined Arms Cmbt 0ev Act , Ft Leavenworth , ATTN: ATCAC O—E

1 DCSPER , ATTN CPS/OCP 1 USA Combined Armt Cmbt 0ev Act . Ft Leavenwo rth , ATTN: ATCACC—Cl

I The Army Lib, Pentagon , ATTN: RSB Chief 1 USAECOM , Night Vision Lab , Ft Belvoir , ATTN: AMSE L—NV—SD

1 The Army Lib , Pentagon , ATTN: ANRAL 3 USA Computer Syt Cmd , Ft Belvo ir , A’l’TN: Tech Library • 1 Ofc . Asat Sect of the Army (R&D) 1 USAMERDC, Ft Belvo ir , ATIN: STSFB--DQ

1 Tact, Support Ofc . OJCS 1 USA Eng Sch, Ft Belvoir , ATTN : Library

I USASA, Arlington, ATTN: lARD-I I USA Topographic Lab , Ft Belvoir , ATIN: ETL -TO—S

1 USA Rsch (Sc, Durham , ATIN: Life Sciences Dir 1 USA Topographic Lab , Ft Belvoir , ATTN: STINFO Center

2 USARIEM , Natick , AT1’N: SGRD-UE.CA I USA Topographic Lab , Ft Belvo ir , A’fl’ N: ETL—GSL

1 USATTC, Ft Clayton , ATt N: STETC-MO-A 1 USA Intel ligence Ct r & Sch , Ft Huachuca , ATTN: CTD—MS

1 USAIMA. Ft Bragg, ATIN: ATSU-CTO-OM 1 USA Intelligence Ctr & Sch , Ft Huachuca , ATTN: ATS—CTD--MS

I USAIMA . Ft Bragg, ATIN: Marguat Lib 1 USA Intelligence Ct r & 5th, Ft Huachuca. ATTN: ATSI-~

-TE

I US WAC Ctr & Sch, Ft McClellan , ATTN: Lib 1 USA Intelligence Cr, & Sch, Ft Huactsuca , ATTN: ATS I—TEX—GS

I US WAC Ctr & 5th , Ft McClellan , ATTN: Tng Dir 1 USA Intellig ence Ctr & Sch , Ft Huachuc a . ATTN: ATSI—CTS --OR

1 USA Quartermaster Sch , Ft Lee, ATTN: ATSM.TE 1 USA Intelligence Ctr & Sth, Ft Huac)wca, ATTN: ATSI—CTD--DT

1 Intell igence Material Div Ofc . EWL . Ft Holab ird I USA Intelligence Ctr & Sets , Ft Huachuca , ATTN: ATSI—CTD—CS

1 USA SE Signal Sch, Ft Gordon , ATTN: A1’SO-EA I USA Intelli gence Ctr & Sch. Ft Husctsuca, ATTN: DAS/SRD

I USA Chaplain Ctr & Set,, Ft Hamilton , ATTN : ATSC~

TE-R D 1 USA Intelligen ce Ctr & Sch , Ft Huachuca , ATTN: ATSI—TE M

I USATSCH . Ft Eustis , ATTN: Educ Advisor I USA Intelligence Ctr & Sets , Ft Hu.chuca. ATTN: Library

1 USA War College . Carlisle Barracks , ATTN : Lib 1 CDR , HO Ft Huachuca , ATIN: Tech Ref Div

2 WRAIR , Neuropsychiatry Div 2 COR , (iSA Electronic Prvg Gi’d, ATIN: STEEP—MT—S

1 DLI , SDA. Monterey I CDR , Project MASSTER , ATTN: Tech Info Canter

1 USA Concept Anal Agcy. Bethesda, ATm: MOCA~

WGC I Hg MASSTE R . USATRADOC, LNO

1 USA Concept Anal Agcy. Betheade, AT1’N: MOCA-MR 1 Researc h Institute , HO MASSTER , Ft Hood

1 USA Concept Anal Agcy , Bethetda, ATTN: MOCA’JF 1 USA RecruIting Cmd. Ft Shertl ian , ATI’N: USARCPM—P

1 USA Arti c Test Ct,, APO Seattle, ATIN: STEAC.M0.ASL 1 Senior Army Ads.. USAFAGOD/TAC, Elgin AF Aux FId No. 9

1 USA Art ic Test Ctr , APO Seattle, ATTh. AMSTE.Pt..TS 1 HO USARPAC . DCSPER , APO SF 96558. ATTh: GPPE—SE

I USA Armament Cmd, Redstone Arsenal , ATTN: ATSK-TEM 1 Stimson Lib , Academy of Health Sciences, Ft Sam Houston

1 USA Armament Cmd, Rock Island . ATTN : AMSAR.TDC 1 Marine Corps Inst., ATTN: Dean—MCI

1 FAA-NA FEC , Atlantic CIty , ATTN: Lib rary 1 HQUSMC, Commandant , ATTN: Code MTMT 51

I FAA .NAFEC , Atlantic CIty , ATTN: Hum En~ Br 1 HQUSMC, Commandant, ATTN: Code MPI—20

1 FAA Aeronautical ctr, Oklahoma City. ATIN: AAC-440 2 USCG Academ y, New London , A’fl ’N: Admission

2 USA FId Arty Scti , Ft Sill , ATTN: Library 2 USCG Acad emy . New London , ATIN: Library

1 USA Armo r $ch, Ft Knox , ATTN: Library 1 USCG Training Ctr , NY . AT1’N: CO

1 USA Armor 5th, Ft Knox, ATIN: ATSB-Dl-E 1 USCG Training Ctr , NY , ATIN: Educ Svc Ofc

I USA Armor Sets. Pe Knox , ATTN: ATSB.DT.TP 1 USCG, Piychol Re, Br . DC, AT1’N: GP 1/62

1 USA Armo r 5th, Ft Knox , ATIN: ATSB.CD’AD 1 HO Mid—Range Br , MC Des , Ouantico. ATTN: P&S Di’

B— i
~~~~~~~~

-

--~~~ - -

~ -—-~

— -

— - - ——s -

~

-•-- --—- ~~~~~ • - --.------

-

~— —- ,--

H

ARI TP 280

1 US Marine Corps Liais io n OIc , AMC, Alesandria , ATTN: ANCGS—F 1 Def & Civil Inst of Enviro Medicine. Canada

1 USAT RADOC , Ft Monroe . ATTN : ATRO—ED I AIR CRESS. Kensington, ATTN: Info Sys Br

6 USATRADOC , Ft Monroe , ATTN : ATPR—AD 1 Milltasr psyltologisk Tjeneste , Copehigen

1 USATRADOC , Ft Monroe , ATTN: ATTS—EA 1 Military Attache, French EmIiissy, ATTN: Doe Sec

I USA Forces Cmd, Ft McPfierson, ATTN: Library 1 Medecin Chef , C.E.R.P.A.—Art enal . Tou lo n/Naval France

2 USA Aviat ion Test Bd , Ft Rucker , ATIN: STEBG—PO 1 Prin Scientific Off , AppI Hum Engr Rs ch Div . Ministry

I USA Agcy for Aviation Safety , Ft Rucker , ATIN: Library of Definse. New Delhi

1 USA Agc y for Aviation Safety, Ft Rucker , ATTN: Educ Advisor 1 Pars Rich Ofc Lib rary. AKA . Israel D&ense Forces

1 USA Aviation 5th, Ft Rucker , ATTN: P0 Drawer 0 1 Ministe ris van Defensie , DOOP/KL Afd Sociul

1 HOUSA Aviation Sys Cmd , St Louis , ATTN: AMSAV—ZDR Psyclsologuscfse Zaken , The Hague , Netherl ands

2 USA Aviation Sys Test Act., Edwards AFB . ATTN: 5AVTE— T

1 USA Air Del Sch, Ft Bliss , ATIN: ATSA TEN

I USA Air Mobility Rsc h & Dee Lab, Not/itt FIt AIIM: SAVDL—A$

1 USA Aviation Sets . Rev Tng Mgt. Ft Rucker , AT1’N: ATST—T—RTM

1 USA Aviation Set, , CO. Ft Rucker , ATTN: ATST— D—A

1 HO. USAMC , Alexandria . ATTN: AMXCD—TL

1 HO, USAMC . Alexandria , AT’TN: COR

I US Military Academy, West Point , ATTN: Serials Unit

I US Military Academy, Watt Point , AITN: Ofc of Milt Ldrshp

1 US Milita ry Academy, Wets Point , ATIN: MAOR

1 USA Standardization Gp. UK , FPO NY, AIIM: MASE-.GC

1 (Sc ot Naval Rich, Arlin gton , AIIM: Code 452

3 (Sc of Naval Rich , Arlington , A’fl ’N: Code 458

1 (Sc of Naval Rich. Arlington , ATTN: Code 450

1 Oft of Naval Rich. Arlington, AIIM: Code 441

1 Naval Aarotpc Ned Rev Lab, Pensacola, ATTN: Acous Sets Div

1 Naval Aerospc Med Ret Lab , Pensacola. A’fl N: Cods L51

1 Naval Aerospc Med Rat Lab , Pensacola , AIIM: Code L5

I Chief of NavPers , AIIM: Pert’OR

1 NAVAIRSTA , Norfolk , ATIN: Safety Ctr

1 Nay Oceanographic , DC, ATTN: Code 6251, Charts & Tech

1 Center of Naval Anal . ATIN: Doe Ctr

I NevAirSys Com, ATIN: AIR—63 13C

1 Na, BuMed, ATTN: 713

1 NavHelicoptsrSubSqua 2, FPO SF 96601

1 AFHRL (FT( William AFB

1 AFHRL ITT) Loswy AFB

1 AFHRL (AS) WPAFB , OH

2 AFHRL (DOJZ) Brooks AFB

1 AFHRL (DOJN( Lickland AFB

1 HOUSAF (INYSO)

I HQUSAF (OPXXA)

1 AFVTG (AD) Randolph AFB

3 AMRL (HE) WPAFB, OH

2 AF Inst of Tech, WPAFB. OH. AIIM: ENEISL

I AIC (XPTD) Rando lph AFB

I USAF AeroM.d Lib , Broo ks AFB (SUL—4). ATTN: DOC SEC

I AFOSR (ML). Arlingt on

1 AF Log Cmd, McClellan AFB . ATTN: AL C/DPCRB

1 Air Force Academy, CO. AIIM: Dept of Bal Sen

5 NavPsrs & Des Ctr, San Diego

2 Navy Med Neuropaychlatrlc Rach Unit. San Diego

1 Na, Electronic Lab, Sen Diego, ATTN: Ass Lab

I Na, IrngC.n, San Diego , ATTN: Code 9000—Lib

I NavPostGraSch. Monterey, ATT’N: Cods S5Aa

1 NevPovtGraSch. Monterey, ATTN: Cod, 2124

1 NavlmgEqulpCtr, Orlando, AiiM: Tech Lib

1 US Dept of Labor. DC, ATTN: Manpower Admmn

I US Dept of Justice. DC, ATTN: Drug Enforce Admln

1 Nat Bur of Standard,. DC. ATIN: Computer Info Section

I Nat Clearing House fo r MH—Info, Rockvllle

I Denver Federal Ct,. Lak,wood . ATTN: BLM

12 Defense Documentation Cents,

4 DIr Psych , Army Hg, Ass.,ii Ofca, Canberra

I Scientific Advir , MII Sd, Army Hg, Russell Ofce. Canberra

I Nil and Air Attache , Austrian Embassy

I Centre da R,ct’i erche Des Facfsur s, l’fumalne da Ii Oefsn.a

N.tio nala. Brussels

2 CanadIan Joint Stiff Washington

I C/AIr Staff , Royal Canadian AF, ATIM: P.nStd Anal Br

3 Clsiaf, Canadian Def Rich Stiff , ATTN: C/CRDS(W)

4 BritIsh Oaf Staff, British Embaeay, WashIngton

B-2

- ~~~~~~~~~~~~~~~