Technical Report 406 2)

AN ALGORITHM FOR COMPUTERIZED

\ADAPTIVE DECISION ANALYSIS

James L. Raney

:1

PERSONNEL AND MANPOWER TECHNICAL AREA

U. S. Army

Research Institute for the Behavioral and Social Sciences

.. September 1979

Approved for Public release; distribution unlimited.

804 9 034
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

WILLIAM L. HAUSER

JOSEPH ZEIDNER Colonel, U S Army Technical Director Commander

NOTICES

DISTRIBUTION: Priryr distribution of this report has been Made by ARI. Islam address correspondence concerning distribut ion of reports to: U. S. Army Research Institute for the Behavioral anid Social Sciences. 'I ATTN. PERI.P, 5001 Eisenhovier Avenue, Alexandria, Virginia 22333.

FIA ISPOSITION: This report rmy be destroyed when itis no longer needed. Pleae do not return it to

she U. S. Army Research Institute for the Behavioral and Social Sciences.

Pi :Thefind " ngsin this report re not to be construed a s an official Departmentl of he Armyv position,

unlssgo esgnaedby other outhorilled dotumeonts.
Unclassified

SECURITY CLASSIFICATION OF THIS PAGE (Ue., Deis Ent.,ecO

REPOT DCUMNTATON AGEREAD INSTRUCTIONS

REPOR DOCMENTTIONPAGEBEFORE COMPLETING FORM_

IREPORT NUMBER - 462. GOVT ACCESSION No. 3. RECIPIENT'S CATALOG NUMBER

Technical Report 406n

4. TITLE (and Subettie) -TVPE OF REP'ORT & PERIOD COVERED

final i-epti

~ALGORITHM FORSPMPUTERIZEDADAPTIVE DECISION Mn"PV~f-Sepg1b*W76

PERFORMN4G %SG PORT ,wmSiuR

7._AUTHOR(e) RaeS. CONTRACT OR GRANT NUMBER(&)

S. PERFORMING ORGANIZATION NAME AND ADDRESS 10 RGRAM ELEMENT. PROJECT, TASK

U.S. Army Research Institute for the Behavioral, 1 4EAa WORK UNITNUMBS

and Social Sciences -- f2T611AB

5001 Eisenhower Avenue, Alexandria, VA 22333 ?

It. CONTROLLING OFFICE NAME AND ADDRESS 11 REPORT DATE ii SepSN 79 .7 Deputy Chief of Staff for Personnel 4 MBE OPAE Washington, DC 20310 3

14. MONITORING AGENCY NAME & ADDRESS(iI different from Controlig Office) I5. SECURITY CLASS. (of this report)

f Unclassified

Is. DECLASSI FICATION/ DOWNGRADING S- SCHEDULE

1S. DISTRIBUTION STATEMENT (of tisl Report)

Approved for open release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of t. .bstract entered In Block 20, If different from Repot)

I8. SUPPLEMENTARY NOTES

Research was performed under the In-house Laboratory Independent Research

(ILIR) program (Reference AR 70-55).

I9. KEY WORDS (Continue on revere side If necessar ad Identify by block number)

Measurement Scaling

Model Additivity

Algorithm . Conjoint

*1 2& AETRAcr (Vked, m ,evenw stf N neeemy end identify by. black n&=60br)

This research investigated an application of recent results in conjoint

measurement theory research with the aim of developing a new methodology for

(1) modeling a decision process used to evaluate preferences for complex

choice alternatives, and (2) producing measurement scales for choice component factors and composite choice alternatives based on the decision model.

An algorithm for interactive conjoint measurement (ICM) was developed to k

minimize the problems of redundancy and random error in testing the

(Continued)

D FCAON 147 EDITION OF I NOV065 IS OUSOLETE Unclassified a/~i~ 1.K 1 SECURITY CLASSIFICATION OF THIS PAG2E (When~ Des Efneered) /1

4c
Unclassified

SECURITY CLASSIFICATION OF THIS pAF ena, Dole En rd)

Item 20 (Continued)

additive-independence model (AIM) in pair-comparisons designs with fallible

data. No provisions were made for handling the systematic error problem or

for accommodating more than two choice component factors. The ICM algorithm

was tested in error-free data and in data with random error. The results

showed that this ICM algorithm performed rather poorly.

'.1

Unclassified

ii SECURITY CLASSIFICATION OF THIS PAGE(f1te Data nter"l)
Technical Report 406

AN ALGORITHM FOR COMPUTERIZED

ADAPTIVE DECISION ANALYSIS

James L. Raney

Submitted by: M. A. FIschI, Acting Chief

PERSONNEL AND MANPOWER TECHNICAL AREA

Approved by:

E. Ralph Dusek

PERSONNEL AND TRAINING

RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCE5

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

September 1979

Army Project Number Basic Research

2T611O1AO1B

Approved for public release; distribution unlimited.

iiii
I

ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

iv

V C61...... ~ a. ~, .
-FOREWORD

This technical report is a product of basic research performed

under the In-house Laboratory Independent Research (ILIR) program. The

ILIR program provides to R&D centers and laboratories the financial

means to support, in addition to the regularly assigned program, work

judged to be important or promising, provided it contributes toward the,.

solution of a problem that is included within the mission assigned to

the laboratory. This research contributes toward development of new

methods for analysis of decision-making behavior, which is a major

topic of interest in the Personnel and Manpower Technical Area.,-

( SEP IDNER

chnical Director

Dist

v

'..1-
AN ALGORITHM FOR COMPUTERIZED ADAPTIVE DECISION ANALYSIS

BRIEF

Requirement:

To develop new methods for analysis of decision-making behavior--

specifically, new methods for (1) modeling a decision process used to

evaluate preferences for complex choice alternatives, and (2) producing

measurement scales for choice component factors and composite choice

alternatives based on the decision model.

Procedure:

Recent results in conjoint measurement theory research were applied

to develop an algorithm for minimizing the problems of redundancy and

random error in testing the additive-independence model in pair-comparisons

designs with fallible data. No provisions were made for handling the

systematic error problem or for accommodating more than two choice component factors.

Findings:

The algorithm was tested in error-free data and in data with random

error. A reduction of approximately 44% in the number of pair-comparisons

necessary to determine all model constraints was obtained in a 5 x 5 factorial design with error-free data. This reduction was decreased to 33%

when a moderate amount of random error was introduced. Various deficiencies in algorithm performance were noted. In general, the results showed

that this algorithm performed rather poorly.

Utilization of Findings:

This research demonstrates some limitations of a direct application

of the generalized cancellation condition for testing the additiveindependence model. Further research is required to develop an analytical approach which can ameliorate these limitations.

F

4 Vii
Ij AN ALGORITHM FOR COMPUTERIZED ADAPTIVE DECISION ANALYSIS

CONTENTS ______

INTRODUCTION............................. . .. . ..... . ... .. .. .... 1

General Statement of the Problem ...............

Specific Statement of the Problem.................................

INTERACTIVE CONJOINT MEASUREMENT..................... ..

Measurement--Theoretical Basis ................

Constraints Matrix........................ . . . ... . .. .. .....

Algorithm Logic..........................

ALGORITHM EVALUATION........................ . . . ...... . . .. .. .....

Error-Free Data........................

Data With Random Error...................... .. . . .... .. .. . ...

CONCLUSIONS......................... ........... .. .. ........ 17

REFERENCES..................................19

APPENDIX................................. . .. . ..... . .. .. .. .....

DISTRIBUTION....................................33

Table 1. Additive constraints matrix......................6

2. Nonadditive constraint-, matrix....................6

LIST OF FIGURES

Figure 1. PSI matrix of composite scale values................10

2. CONSTRAINTS matrix after first-order cancellation

in error-free data........................11

3. TIMES matrix after first-order cancellation in

error-free data........................12

ix

IL * , ;g ., -
CONTENTS (Continued)__________ _________

Fiqure 4. CONSTRAINTS matrix after second-urder crTI1tt:

in error-fret c data . . . . . . . . . . . . . . . . . .

5. TIMES matrix after seododrcance]lhAtr in1

error-free data..................... .. . . . ... .. . ...

6. TIME~S matrix after secondl-order c2a:cellation in

data with randomn error .. .. .. .. . . .. . . . .

7. CONSTRAINTS matrix after second-order canceilatior,

in data with random error...............

x
AN ALGORITHH FOR COHPUTERIZED ADAPTIVE DECISION ANALYSIS 

INTRODUCTI ON 

General Statement of the Problem 

In Army research it is often desirable to inves tigate the nature o f 

a de cisi on p rocess used by Army personne l in making some important cho i c e. 

In many i ns tances the choice alternatives may represent composites of 

several component factors wh h may·be identified and considered separately . For example , i n Army career planning research it may be o f intere st to investi~ate the nature of the decision process used by officers 

in indic ating their preferences for various assignments. In this examp l e the c hoice alternatives may .be d e scribed as composite s of compone nt 

factors such as assignment l ocation (e . g. , EUROPE , CONUS) , type (e . g . , 

COHHAND , STAFF) , and duty specialty (i. e . , PRIMARY , ALTERNATE) . 

The additive-independence model (AIM) is one partie~ ar model of the 

dec ision process which has considerable appeal because of its simplicity 

in compari son with other possible models and because of its general applica bility in other substantive areas of psychology. In this model 

component factor levels are assigned specific scale values relative to 

one another. The scale values assigned to levels of a component factor 

are designated independently for each component factor. The scale value 

attached to a particular composite choice alternative is determined by 

summing the scal e values assigned to levels of the component facto rs 

whic h are present in the composite . The relative mag11itude of scale 

values for composite choice alternatives serves as a basis for the preference dec isi0n . 

, 

For example , EUROPE l and CONUS = 2 may be the scale values assig ned to assignment location; COHMAND = 2 and STAFF = 3 rna:' be the s c ale 

values assig ned to assignment type; and PRIMARY = 5 and ALTERNATE = l may 

be the scal e values assigned to assignment duty specialty . If an individual uses the AIM in evaluating p r eferences for various assignments , 

the scal e val ue attached to a PRIMARY specialty COHMAND assignment in 

EUROPE is 5 + 2 + l = 8; the scale value attached to a PRIMARY specialty 

STAFF assignment in CONUS is 5 + 3 + 2 = 10. Since the composite scale 

value for the latter choice alternati ve e xceeds the composile scale v a lue 

for the former choice a l ternative, the individual would indic ate a pref -

erence for the latter choice alternative . If composite s c ale values for 

the c ho i ce a lternatives are equ a l , an individual may indic ate no 

preference . 

In r esearch on the nature of a decision process two specific object ives may be identified. The first objective is to elop a model of 

the decision process used to eval uate preferences f o r c omplex choic e 

alternatives. If individual differen ces in the d eci s ion process a re 

discovered , an individual-specific mode l of the decisic 1 p rocess .may be 

l 
required. The second objective is to produce measurement scales for 

choice component factors and composite choice alternatives based on the 

dec ision mode l. The general purpose of this study was to investigate 

an application of recent results in conjoint meas urement theory research 

with the aim of dev e loping a new methodology for accomplishing these two 

researc h objectives. 

Specific Statement 0 f the Problem 

The conjoint measurement problem is that of obtaining measurement 

scales for component stimuli ~nd composite stimuli simultaneously bas ed 

on a specified composition model when only the rank order of the composite stimulus effects is known. The order constrain ts generate a 

finite system of homogeneous equations and inequalities in the composition model. If the specified composition model is valid, t he r esulting 

system should be consistent and measurement scales for component s t imuli 

and composite stimuli may be derived by solving the system for the unknown parameters. In the case of the AIM , the derived measurement scales 

constitute "multidimensional ordered-metric scales" (Krantz, Luce , Suppes, 

and Tversky, 1971, Chapter 9) . 

Although conjoint measurement theory has been particularly we l ldeveloped for the additive-independence composition model, practica l applications with individual subjects have been few . Three formidabl e 

problems remain to be solved for applications of con joint measurement 

theory in fallible ordinal data. First, the magnitude of the paircomparison task may exceed the capability of the individual subject when 

the number of composite stimuli is large. If the composite stimulus set 

contains N elements, N(N-1)/2 nontrivial pair-comparisons must be made 

to provide a complete ordering of the composite stimuli. Since t he number of pair- comparisons usually greatly exceeds the number of par ameters 

to be estDnated for the composition model, it is clear that a large amount 

of redundancy of effort is inherent in the complete pair-comparisons 

method. Second, the presence of random experimenlal error may introduce 

ordinal inversions in the ordering of composite stimuli. The result is 

an inconsistent system of equations and inequalities which provides only 

a n approximation of the latent measurement structure. Third, the possible presence of systematic error may be difficult to detect in the presence of random error. Systematic error may result when the latent composition model differs from the additive model. 

The specific purpose of this study was to develop an algorithm f or 

interactive conjoint measurement (ICM) to minimize the problems of redundancy and random error for applications of additive conjoint measurement theory 1n fallible ordinal data. Although the algorithm presently 

has no provision for handling the systematic error problem and is limited 

to two component factors, extensions of the algorithm to test several 

models simultaneously with multifactor composite stimuli may be possible. 

In the next section the conjoint measurement-theoretical basis for the 

algorithm is presented. The notion of a constraints matrix is defined 

2 
and an ICM algorithm is described. An evaluation of the performance of

the algorithm is provided in the last section.

INTERACTIVE CONJOINT MEASUREMENT

Measurement-Theoretical Basis

Let us represent component stimulus variables A and P as the finite

sets {a,b,c,....} and {p,q,r .... }, respectively, where the elements of a

set are the particular component stimuli included in an experiment. Each

composite stimulus may be represented as an element in the Cartesian

product set A x P. For example, composite stimulus (a,p) represents the

combination of stimulus component a in A with stimulus component p in P.

Suppose all possible pairs of composite stimuli are presented to a subject whose task is to indicate the preferred composite stimulus in each

pair. The experiment gives rise to a binary preference relation (denoted ) on A x P. The two sets of component stimuli A and P together

with the binary relation u defined on A x P constitute an empirical relational structure (denoted <A, P,,)).

The empirical relational structure <A, P,> may be said to satisfy

the additive-independence model (AIM) if there exist real-valued component stimulus-scale functions 4A on A and p on P such that the additive combination of the component stimulus-scale values preserves at

least the rank ordering of the composite stimuli for all a, b in A and

for all p, q in P:

(1) (a,p) t (b,q) iff A(a) + p(p) A (b) + #p(q).

The symbol "iff" is an abbreviation for "if and only if." Component

stimulus-scale functions OA and p map elements in sets A and P, respectively, into the set of real numbers (denoted Re). When such homomorphisms exist, the empirical relational structure <A, P, is said

to be mapped into the numerical relational structure <Re, Re, > in the

sense that OA maps A into Re, pp maps P into Re, and % is mapped into _

defined appropriately on Re x Re. In practice, the functions OA and

Sp may be constructed by solving the finite system of homogeneous linear

equations and inequalities generated by on A x P.

In order for solutions 'A and p to exist, Scott (1964) proved that

the following conditions were necessary and sufficient:

Connectedness. Either (a,p) (b,q) or (b,q) (a,p) for all a,b

in A and for all p, q, in P.

Cancellation. For all sequences a0 , al, ..., an in A, for all

sequences p0 , Pl, ..- , Pn in P, and for all permutations 7 and a of

{0, 1, ..., n), where n > 0, if for i = 1, 2, .... n (ai, Pi) "

(a (i),P ai), then (a r (0), )) (a0, p0).

3

... .. .. ..!

-2%
The connectedness axiom simply requires that all composite stimuli

must be comparable. Although the connectedness axiom may be assumed to

hold trivially in many experimental applications, its validity may be

questioned in others (Tversky, 1967; Krantz et al., 1971, p. 17). A

more general. axiomatization of the AIM which does not require connectedness is presented by Tversky (1967). For the purposes of this study,

the axiom is assumed to hold in the latent psychological composition

process.

Since a sequence of elements in a set may contain repetitions of

elements (as opposed to a subset of elements which may not contain repetitions), the cancellation axiom actually defines a countably infinite

set of cancellation axioms indexed by n. Each nth-order cancellation

axiom asserts that n inequalities imply an additional inequality via

the AIM provided that identical terms may be canceled from each side of

the inequalities until only one term from each component stimulus set

remains on each side (Krantz et al., 1971, p. 427). Thus, if the nthorder cancellation axiom fails from some n, than a fortiori the mth-order

cancellation axioms must also fail for all m > n. Similarly, if the

nth-order cancellation axiom holds for some n, then a fortiori the mthorder cancellation axioms must hold for all m < n.

The 1CM algorithm used in this study was based on testing successive

cancellation axioms (i.e., n = 1, 2, ... ). The algorithm minimized the

redundancy problem by selecting for presentation to the subject only those

composite stimulus pairs which are critical for determining AIM constraints.

All other (redundant) constraints were derived via the AIM from knowledge

of previously obtained ordinal constraints. The algorithm minimized the

random error problem by detecting ordinal inconsistencies in real-time

and attempting to rectify the discrepancies by repeating critically important pair-comparisons. An extension of the 1CM algorithm based on

Tversky's (1967) irreflexivity axiom may be possible for testing multifactor polynomial composition models of specified degree.

Constraints Matrix

A central concept in the development of the 1CM algorithm is the

notion of a constraints matrix which indicates the preference relationships among composite choice alternatives. A constraints matrix describes the empirical binary preference relation defined over the Cartesian product-set A x P. If the row composite stimulus is preferred to

the column composite stimulus, enter "+" in the matrix. If the column

composite stimulus is preferred to the row composite stimulus, enter

~-'in the matrix. If no preference is indicated between the two composite stimuli, enter "0" in the matrix. A constraints matrix is skewsymmetric with zeroes along the main diagonal. Thus, the matrix contains

M =N(N-l)/2 nontrivial terms, where N is the number of elements in the

Cartesian product-set. A total of 3M constraints matrices may exist for

a specified Cartesian product-set. some of these matrices may be fit

with an additive model, while others may not.

44
In order to illustrate the concept of a constraints matrix, let us

consider the simplest case of a 2 x 2 factorial design. The following

two sets of levels of component factors may be considered:

Assignment = (a, b) = {COMMAND, STAFF)

and

Place = {p, q} = fEUROPE, CONUS).

The corresponding Cartesian product-set is given by

A x P = {(a,p) , (a,q) , (b,p) , (b,q) },

where

(a,p) = ap = COMMAND assignment in EUROPE,

(a,q) = aq = COMMAND assignmdnt in CONUS,

(b,p) = bp = STAFF assignment in EUROPE,

(b,q) = bq = STAFF assignment in CONUS.

An example of a constraints matrix which may be fit with an additive

model is shown in Table 1. The finite system of homogeneous linear equations and inequalities determined by this constraints matrix is:

a + p - a - q < 0 or p - q < 0

a + p - b - p > 0 or a - b > 0

a + p - b - q > 0 or a - b > q - p

a + q - b - p > 0 or a - b > p - q

a + q - b - q > 0 or a - b > 0

b + p - b - q < 0 or p - q < 0.

This system may be solved for component scale values a, b, p, and q

which constitute "ordered-metric" scales (i.e., the resulting scales

have properties better than mere ordinal scales but not so good as interval scales). Note the redundancy of the equations and inequalities

in the system. Only the first three constraints are necessary to completely describe the system.

An example of a constraints matrix which may not be fit with an

additive model is shown in Table 2. The finite system of homogeneous

linear equations and inequalities determined by the constraints matrix

is:

a + p - a - q < 0 or p - q < 0

a + p - b - p < 0 or a - b < 0

a + p-b - q <0 or a-b < q -p

a + q -b-p > 0 or a-b> p - q

5
a + q - b - q > 0 or a-b > 0

b + p - b - q < 0 or p - q < 0.

Note the inconsistency of the equations and inequalities in the system.

This system may not be solved for component scale values which satisfy

the AIM.

Table 1

Additive Constraints Matrix

ap aq bp bq

ap 0 - + +

aq + 0 + +

bp - - 0 -

bq - - + 0

Table 2

Nonadditive Constraints Matrix

ap aq bp bq

ap 0 - - -

aq + 0 + +

bp + - 0 -

bq + - + 0

I

Algorithm Logic

The objectives of the ICM algorithm are twofold: (1) to minimize

the number of comparisons between choice alternatives required to generate

a complete constraints matrix for the AIM; and (2) to rectify discrepancies from the AIM due to random error. A method of accomplishing these

objectives will be presented below.

] 6
The ICM algorithm consists of three successive stages. In the first

stage the first-order cancellation axiom is used to construct a constraints matrix that satisfies the first-order cancellation properties

of the AIM. In the second stage the second-order cancellation axiom is

used to construct a constraints matrix that satisfies the second-order

cancellation properties of the AIM as well as the first-order cancellation properties. In the third stage the resulting constraints matrix

may be passed to a linear programming subroutine which tests the constraints matrix for additivity. If a solution exists, the subroutine

may generate scale values for the component factors of the composite choice

alternatives based on the AIM.

First-order cancellation conditions consist of two forms of singlecomponent-wise cancellation. The A-component-wise cancellation form is:

If (a,p) (a,q),

then (b,p) (b,q)

for all a, b in A and for all p, q, in P. The P-component-wise cancellation form is:

If (a,p) (b,p),

then (a,q) % (b,q)

for all a, b, in A and for all p, q in P. Since the antecedent inequality

implies the conclusion inequality, only the antecedent constraint must be

ascertained by querying the subject. The conclusion constraint may be

derived since first-order cancellation is a necessary condition for the

AIM.

Second-order cancellation conditions consist of two forms of doublecomponent-wise cancellation. In the simple transitivity form, the Acomponent and P-component of one composite stimulus in each of the two

antecedents are identical:

If (a,p) ' (bq)

and (b,q) Z (c,r),

then (a,p) > (c,r)

for all a, b, c, in A and for all p, q, r in P. In the generalized

transitivity form, the A-component and P-component of each composite

stimulus in the antecedents are different:

7

V P
If (a,p) ^U (b,q)

* and (b,r) . (c,p),

then (a,r) > (c,q)

for all a, b, c in A and for all p, q, r in P. Since the antecedent inequalities imply the conclusion inequality, only the antecedent constraints

must be ascertained by querying the subject. The conclusion constraint

may be derived since second-order cancellation is a necessary condition

for the AIM.

In the general case first- and second-order cancellation properties

are necessary but not sufficient for a constraints matrix to satisfy the

*AIM since higher-order cancellation properties which are not tested may

fail to hold. However, some recent results by Arbuckle and Larimer (1976)

indicate that the double cancellation requirement becomes stronger as the

size of the factorial design increases. For large factorial designs (e.g.,

5 x 5, 6 x 6), the study showed that the chance probability of observing

a constraints matrix which satisfies both first- and second-order cancellation is very small. Since the ICM algorithm may be most useful with

large factorial designs, and since the computational requirements in testing third- and fourth-order cancellation were found to require large

amounts of computer time, only the first- and second-order cancellation

tests were implemetted in the present version of the algorithm. The test

for sufficiency of the constraints matrix for the AIM may be performed by

a linear programming subroutine.

The antecedent constraints ascertained by querying the subject may

be incorrect when random error is present in an experiment. Any conclusion constraints derived from cancellation conditions based on an incorrect antecedent constraint will also be incorrect. The ICM algorithm

must include a mechanism to detect ordinal inconsistencies in the constraints matrix and to rectify those discrepancies from the AIM when they

are discovered.

The mechanism for detecting ordinal inconsistencies involves simply

checking the conclusion constraint for each possible set of antecedent

constraints generated on the basis of Scott's (1964) cancellation axiom.

When a particular set of antecedent constraints implies a conclusion constraint, the following logic is used:

1. If no conclusion constraint has been entered in the constraints

matrix, the implied constraint is entered and the algorithm

proceeds to test the next set of antecedent constraints.

2. If the correct conclusion constraint has been entered in the

constraints matrix, the constraints are consistent and the

algorithm proceeds to test the next set of antedecent

constraints.
3. If an incorrect conclusion constraint has been entered in the

constraints matrix, an ordinal inconsistency has been detected.

An error-correction strategy must be adopted to rectify the

discrepancy from the AIM. An ordinal inconsistency detected

in this manner may stem from one of two sources: (1) one or

more of the antecedent constraints may be incorrect; (2) the

conclusion constraint may be incorrect. For the purposes of

this study, a simple error-correction strategy was adopted.

Each antecedent constraint was queried in turn until a constraint different from the table entry was obtained. If all

the antecedent constraints were found to correspond to the

table entries, the table value for the conclusion constraint

was changed to the derived value. Since any constraints which

were altered may have been used in previously tested conditions,

it was necessary to restart the cancellation condition testing

process from the beginning each time an ordinal inconsistency

was corrected.

The ICM algorithm logic described above was implemented on a CDC 3300

computer. A listing of the FORTRAN program is provided in the appendix.

Some internal program documentation is provided by means of comment statements. Program MAIN is the driver program which initializes the algorithm

and successively calls up the first- and second-order cancellation test

subroutines CAN1 and CAN2, respectively. Subprograms ASK, OUTPUT, INDEX,

and XNORM perform a variety of utility functions as indicated by comments

in the listing. Since the linear programming subroutine LPSUB was taken

* directly from Davisson (1972), the listing is not included in the appendix.

The results of testing the first- and second-stages of the algorithm produced discouraging results, so the third-stage of the algorithm represented by LPSUB was not fully implemented.

ALGORITHM EVALUATION

Error-Free Data

The purpose of testing the ICM algorithm in error-free data was to

determine the degree of success in minimizing the redundancy problem.

In a complete pair-comparison experimental design with N composite stimuli,

M = N(N-l)/2 nontrivial pair-comparisons are made to generate a complete

constraints matrix. The degree of success in minimizing the redundancy

problem may be measured by contrasting the number of queries of the subject required by the ICM algorithm to generate a complete constraints

matrix in error-free data with the theoretical maximum number M.

A 5 x 5 factorial design was chosen for the test. The 25 composite

stimuli yield a constraints matrix with 25(25-1)/2 = 300 constraints.

The five values of each component stimulus were generated in accordance

with a random-effects model by sampling randomly from independent normal

distributions with means equal to zero and variances equal to unity. The

25 values of composite stimuli were generated in accordance with a

9

I, ! 'I
strict additive-independence model by summing appropriate component stimulus scale values. The resulting 5 x 5 matrix of composite stimuli is the

PSI matrix in the ICM algorithm computer program listing. The PSI matrix

generated for the error-free data test is shown in Figure 1.

PSI MATRIX

P N S I

A 1.3290 .4t4l .5348 .323- .6083

-0,5b55 -100864 -1.3597 -10370*1 -1,2d62

C 1.9932 1*478J 1.1990 . a 1.2725

) 1.0116 .4961 .2174 *0064 92909

.0796 -094353 -U.7146 -0.9251r -0.6411

Figure 1. PSI matrix of composite scale values.

In the first stage of the ICM algorithm, the test of first-order

cancellation required only 20 queries in order to fill in 100 constraints

in the matrix. These 100 constraints are shown as 1 or -1 above the main

diagonal in the CONSTRAINTS matrix in Figure 2. The table value of 9 indicates that a constraint was neither queried nor derived in the firstorder cancellation stage. The 20 constraints which were queried in the

first-order cancellation stage are shown as 1 in the TIMES matrix in

Figure 3. The table value of 0 indicates that a constraint was not

queried in the first-order cancellation stage.

In the second stage of the ICM algorithm, the test of second-order

cancellation required 149 queries in order to fill in the remaining 200

constraints. The complete CONSTRAINTS matrix is shown in Figure 4. The

149 queries required by second-order cancellation plus the 20 queries required by first-order cancellation are shown as 1 in the TIMES matrix

in Figure 5. The table value of 0 indicates that a constraint was not

queried in either cancellation stage.

In total, 169 queries were required by the ICM algorithm to fill in

all 300 constraints for a reduction of approximately 44%. Although the

savings were substantial, more than three minutes of CDC 3300 computer time were required to compile, load, and execute the program to implement

the first two stages of the algorithm. If the third stage of the algorithm (i.e., linear programming for scaling solutions) had been implemented for this test, the computer time and core memory requirements would

have been much more substantial. Thus, the computational requirements of

the ICM algorithm based on a direct application of Scott's (1964) can- cellation condition may be considered rather excessive. A more sophisticated

10
r

COmSTRAIN1S MATRIN

- r B SI as AT OF W' I MF on a 31 o C c CS CT op 00 on u So IF IU zw 9 UK '

AP * 1 1.1 1 £ 9 9 9 9 -1 9 9 9 9 1 9 9 9 9 1 9 9 9 9

&a -1 - 1 1 1 9 9 9 9 9 9 -1 9 9 9 9 1 9 9 9 9 1 1 9 9

Aft -1 0 1 -1 9 9 1 ' 9 9 9 -&oi 9 9 9 9 1 9 v IF 9 -1 IF

A* -S -1 -1 * -1 9 9 9 1 9 9 9 9 -1 9 9 9 9 1 9 9 9 9 1 9

AT -1 -1 1 S v 9 9 v 9 1 9 9 9 9 -1 9 9 9 9 1 9 9 9 9 1

-- -- t- * t-9 0 1 1 1 A 1 9 9 -1 9 9 9 v wl_ W - - V

s 9 -1 9 9 9 -1 6 1 1 1 9 -1 9 9 9 9 -o 9 9 9 9 -1 9 9 9

n 9 9 -1 9 9 -1 -1 S 1 1 9 9 -A 9 9 9 9 -1 9 9 9 9 -1 9 9

9 9 9-t 9 -1 -1 -1 0 -1 9 9 1 -1 9 9 9 9-1 v v9 99 -1 9

61 9 9 9 9 -1 -1 -1 1 1 S 9 9 9 9 -1 9 9 9 9 -1 9 9 9 9 -A

cp 1 9 9 9 9 1 9 9 9 9 0 1 1 1 9 9 9 9 1 9 9 9 9

O9 1 9 9 9 9 I 9 9 9 -i * 1 9 1 9 9 9 1 1 9 9 9

ca 9 9 1 9 9 9 9 1 9 9 -1 -1 0 1 -1 9 9 1 9 9 9 9 1 9 9

CS 9 9 9 1 9 9 9 9 1 9 -1 -a - 0 *1 9 9 9 1 9 9 9 9 1 9

c 4 9I 9 v 1 9 9 9 9 A -a a a 9 9 9 9 1 9 9 9 9 2

OF -1 9,9 9 9 1 9 9 9 9 -1 1 9 9 9 0 1 1 1 1 1 9 9 9 9

o v -1 9 9 9 9 1 9 9 9 9 -1 9 9 9 -1 a t 1 1 9 1 9 9 9

4w 4 9 -1 9 9 9 9 1 9 9 9 9 -0 9 9 -1 -1 6 1 -1 9 9 1 9 9

D$ 9 9 9 -1 9 V 9 9 1 9 9 9 9 -a 9 -1 -1 -1 S -1 9 9 9 1 9

D v 9 9 v -1 9 9 v 9 1 9 9 9 9 -1 - -1 t 1 1 S 9 9 9 9 1

eP - 9 9 9 9 ! v 9 v -1 9 9 9 9 -1 9 9 9 9 S 1 1 1 1

Ea 9 01 9 9 9- 9 a 9 9 9 -1 9 9 9 9 1 9 9 9 -1 1 1 1

Ea 9 9 01 9 9 9 9 1 9 9 9 9 -a 9 9 9 9 -1 9 9 -1 -1 a 1 -1

--- a 9 9 -1 9 9 l 9 1 v 9 9 9 -a 9 9 9 9 -1 9 -1 -1 -1 0 -1

ET 9 9 9 9 -1 9 9 9 9 A 9 9 9 ,-a 9 9 9 9 -1-1- a 1 0

Figure 2. CMSTUr i utrix after first-order c¢mwelLafiou in

error-free data.

11
TI

TIMES MATRIX

WA *Q AR AS AT 1W "Ma O 85 "1 ce ce ci - CT- VP O u R gs +T-_ -U - 1t° -i

AP 0 1 1 1 1 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0

AQ 0 0 1 1 1 0 0 0 0 0 0 0 0 0 a 0 0 0 0 0 0 0 0 0 0

AR a 0 0 1 1 0 a 0 0 a 0 0 0 0 a t 0 0 -- v o 1 1 0- 0

AS 0 0 0 0 1 0 0 0 0 0 0 S 9 0 0 0 0 0 0 0 0 0 0 0 0

AT 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a 0 0 0 0

OP 0 0 0 0 0 0 0 0 0 o 1 0 0 a v 1 0 0 0 O V 1 0 0 11

sQ 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

BR 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a

0 0 0 0 0 0 0 0 0 0 0 SI 0 0 U-U 0 a U 0 04 -

OT 0 0 0 0 0 0 0 0 0 0 0 S 0 0 0 0 0 0 0 0 0 0 0 0 0

CD 0 0 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

CU 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

C14 6 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

CS 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

CT 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 a 0 0 0 0 0 0 0 0

OP 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0

O 0 0 0 U 0 0 0 0 0 0 0 S 0 0 0 0 0 0 0 0 0 0 0 0 0

OR 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

1 U 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

(D 0 0 0 a 0 0 U 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

U 0 0 0 0 0 a 0 a0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 a 0

T 0 0 0 0 0 U 0 U 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

EN 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ESG 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

E1T 0 0 0 0 a 0 0 0 a 0 0 a a 0 0 0 0 0 0 0 0 0 0 0 0

Figure . TIMES matrix after first order cancellation in

error-free data.

12
CONSTRAINTS MATRIX

.... ) A AT1 B1 P as DR S NT CP CQ C8 CS CT UP DQ DR aS OT EP EQ ER ES ET

aP 0 I 1 I I I I I 1 1 -I -I I I I I I

"A - 0 1 1 1 1 1 1 1 a -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 1 1 1

AS -1 --1 -1 0 -1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 1 1

AT -Ii-I I- 1 0 1 1 1 1 1 -1 -1 -1 -1 -1 -1 1 1 1 1 1 1 1 1 1

Ap -t -1 -1 -1 .1 0 1 I 1 -1 -I - I - 1 -1 m 1 -1 "1 -1 "1 -1 1 1 1

So -1 -1 -- 1 1 0 1 1 1 1-I - 1 -1 1 --1 1-1 -1 1- 1 -1 1 -1 -1

SR -a -I - -a -1 I - 1 0 1 -a -1 -1 -I - -I .1 I I-1 -1 -1 1 -1 -1 -1 -1

-S -1 -1 -t -t -1 -1 -1 -1 0 -A -1 -b -1 -1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1

CP 1 1 1 1 a 11 a 1 0 a a 1 1 1 1 1 1 1 1 1 1 1

- i 1 " 1 1 1 1 a 1 I- -- 1 1 -1 1 1 1 1 1 1 1 1 1 1

Cs -1 1 1 1 1 1 1 1 1 1 -1 -1 -1 1 -1 1 1 1 1 1 1 1 1 1 1

-C+---*--I I I - I I I 1 1 -1 -k1 -1 0 1 1 1 1 1 1 1 1 1 1 1

Cf I t I I I I I I I I 0 I I a I I I I I I I I I I

UP -1 1 1 1 1 1 1 1 1 1 -1 1 1 1 -1 0 1 1 1 1 1 1 1 1 1

Do -1 -I -1 a -a a 1 1 1 1 -I -1 -I -1 1 I 1 1 1 1 1 1

---- '--- I -i--- t 1 1-I-i 1 1 -1 1 -1 1 1 1 1 1

D -1 -1 -1 -1 -1 1 1 1 1 1 -1 -1-1 -1 -1 1 0 1 1 1 1 1

-P- -1-I " - --t 1 1 -1-1 -1-" -I-I-1 -1 1 -1 0 1 1 1 1

EQ -1 -1 -1 -1 -1 1 1 1 1 1 -1 -a -1 -! -1 -1 -1 -1 -1 -1 -1 0 1 1 1

ER -1- 1--11 1 1 11 -1 -1 -1 -1 -1 11 -1 -1 0 1 1 11

- i 1I -I -1 -1 1 1 1 1 -1 -i 1 -1 1 1 -1 -1 -1 -1 -1 -1 1 0 1 --e - -1 -1 -1 -1 -1 1 1 1 1 -1 -i -1 -1 -1 -1 -1 -1 -1 -1 -a -1 -0 1 -a

ET -1 -1 -1 -1 - a 1 a -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 1 1 0

Figure 4. CONSTRAINTS matrix after second-order cancellation in error-free data.

13

ir
__- b a' -" z - ' i, t~ Ok .* ' 4 Wt 1 ( - z £' I. t*

8 . I, . - . J L A. 4 .i £ a - - - I 4 1 - ± 4 1 J.

* - * * a 1i I * * 4 A * A .I i A 4 1 - £ 4 I

eS A A A 4 * I A. 4 z 4 T t 1 &4• A - -

7V A D1 A T T , I . A I . 4 L z a .

-a ~ a 4 1 A 4 4 4 A 1 4 2 2

.z- i A. .* . I 3 T - I L -i - - - - . - - -

-. a -L 4 A 4 . 4 A - - - -t a A, - I - -

S I A.' 4 j. i: A. 4l . 4' , A I , i 4 I. I 1 . 1 4" 1

l I 4 4 . 4 1 i 4 ± 4, . L . 1 1 1 1 i- A. 2 2 2 * I - 1
mathemati cal approach seems required to provide a more e ffi c i ent solution 

to the redundancy problem. The principal disadvantage of the present approach seems to be the inherent rigidity in c onstruc t ing the second - order 

cancellation tests based on specific sequences for testing all a, b , c, 

in A and a ll p , q, r in P. In order to further r educe the number of 

quer~es required in the second-order cancellation stage of the a l gorithm , 

it seems necessary to abandon thi s rigid rocedure in favor of an a daptive procedure based on the nature of all the known constraints . Just 

how to accomplish this goal in the present context o f Scott's (1964) 

cancellation condition is not c lear. Perhaps an adaptation of the general algorithm of McClelland and Coombs (1975) may be feasible for this 

purpose of interactive conjoint measurement. 

Data With Random Error 

The purpose of testing the I~M algorithm in data with random error 

was to determine the degree of success in minimizing the problem of random error. This degree of success may be assessed in terms of (1) the 

~umber of additional queries required by the ICM algorithm to resolve 

dete c ted discrepancies from the AIM, ~nd (2) the extent ~o which the 

estimated constraints matrix corresponds to ~he true constraints matrix 

(i . e., the extent to which the obtained measurement structure approximates the true latent measurement structure) . 

The 5 x 5 PSI matrix of composite choice alternative scale values 

generated for the test of the ICM algorithm in error- free data was also 

used for the test of the algorithm in data with random error. Whenever 

the algorithm queried the simulated "subject," the true absolute difference in composite choice alternative scale values was perturbed by adding an error component prior to ascertaining the preference relation. 

The error components were sampled independently from a n ormal distribution with mean equal to zero and varian ce equal to . The error variance was set at a moderate value of . 25. 

The results of this test showed that a t otal of 200 queries were 

required to generate a co~:1 lete constraints matrix which satisfied both 

first- and second-order· cancellation conditions. Thus, the 44% reduction 

obtained in error-free data was decreasea to 33% reduction in data with 

a moderate amount of random erLor . The number of querie s made for each 

pair of composite choice alte rnatives is shown in the TIMES matrix in 

Figure 6. Note that comparisons (a,r) v e rsus (a,s) and (a,q) versus (a,t) 

were made nin-~ times and eight times, respectively. The large numbers 

of queries required for these comparisons may be attributed to the sm ll 

absolute differences in scale values (approximately .21) which make l hese 

comparisons especially sensitive to random error (s tandard d eviation .50). 

The fact that the algorithm made repeated queries of these comparisons 

rather than queries 0f equivalent comparisons [e.g., (b,r) versus (b,s) ; 

(b,q) versus (b,t)] indicates another deficien y of this algorithm which 

is due to the inherent rigidity of constructing cancellation conditions 

based on spec ific sequences of elements in the component stimulus sets. 

15 
TiMES MATRiX

AP AQ AR AS AT b so8R 3 8T CP CQ C8 CS CT LP UU Uk US UT L I-E Ek ES L

P o 5 2 1 1 1 1 1 1 1 2 3 1 1 1 1 1 1 1 1 1 1 C I

AQ 0 0 1 1 8 1 U 0 0 0 1 0 1 1 1 0 0 U 0 1 0 0 0 0

AR 0 0 0 0 0O 0 0 0 1 0 T 1 1 0 D 0

AS U 0 0 0 1 0 0 0 0 00 o 0 1 1 0 0 U Q 0

AT 0 0 0 0 U U 0 0 0 0 0 0 0 0 0 1 1 0 0 0 V L U

OP 0 0 1 1 1 0 0 0 0 0 1 0 0 0 0 I I i I 1

OW J 0 11 1 1 0 0 0 0 1 0 1 1 1 1 0 1 1 1 1

e 0 1 0 1 1 0 0 0 0 - 1 1 * . 1 1 1 0 1 1 1 1 0 1 1

as 0 1 1 0 1 C 0 0 0----O I 1 0 1 1 1 1 0 1 1 1 1 G I

BY 0 l 1 1 0 0 0 0 0 1 1i 1 0 1 1 1 1 0 1 1 2 1 0

0 0 b 0 0 0 0 0 0 0 0 0 0 0 1 0 U 0 0 1 0 0 i U

C-P 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 1 I 1 1 0 1 1 1

Ci0 o o0 0 0 0 0 0 0 0 0 0 o 2 I 1 0 I I I 0 1

-Cs 0 0 0 0 u U 0 0 0 0 0 0 0 0 0 2 1 2 0 1 1 1 1 o 1

CT 6 0 0 0 0 0 0 U 0 0 0 0 0 0 0 11 1 1 0 1 1 1 1 0 0

OP o0 0 0 0 0 b -00 d--0 0 0 0 00 0 0 0 1 0 0 0 0

44 - 0 0 0 0 0 0 0 0 0 0 0 0 o 0 0 0 0 0 1 0 1 1 1

DR4 0 0 0 0 0 0 U 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 0 1 1

SoS 0 00 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 1 1 1 0 1

OT 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 0

rEP 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

ER 0 0 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 0 0 o 0o

t o 0 0 -0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 zs o 000o o oo i o i O o io o 0 0o 0 0 0 0 0o o

Figure 6. TIMES matrix after second-order cancellation in

_ d __ I th rnd error.

16

.... • ,.r. .,.r.....
The estimated CONSTRAINTS matrix obtained in this test is shown

in Figure 7. Note that this constraints matrix corresponds to the true

constraints matrix shown in Figure 4 with the exception of three constraints: (a,r) versus (d,q), (b,p) versus (e,q), and (c,r) versus

(d,p). Since perfect correspondence was not obtained, these results

imply a need to test higher-order cancellation conditions prior to

passing the constraints matrix to the linear programming third stage

of the algorithm. More than six minutes of CDC 3300 computer time were

required to compile, load, and execute the program for this test with

random error. Again, the computational requirements of the ICM algorithm based on a direct application of Scott's (1964) cancellation condition seem excessive. A more computationally efficient algorithm seems

required to serve as a basis for computerized adaptive model testing.

CONCLUSIONS

This study investigated an application of recent results in conjoint measurement theory research with the aim of developing a new

methodology for (1) modeling a decision process used to evaluate preferences for complex choice alternatives and (2) producing measurement

scales for choice component factors and composite choice alternatives

based on the decision model. An algorithm for interactive conjoint

measurement (ICM) was developed to minimize the problems of redundancy

and random error in testing the additive-independence model (AIM) in

pair-comparisons designs with fallible data. No provisions were made

for handling the systematic error problem or for accommodating more than

two choice component factors.

The results showed that a reduction of approximately 44% in the

number of pair-comparisons necessary to determine all model constraints

was possible in a 5 x 5 factorial design with error-free data. This reduction was decreased to 33% when a moderate amount of random error was

introduced. Although only first- and second-order cancellation conditions were built into the algorithm for this test, the test in error-free

data required more than three minutes of computer time and the test in

data with a moderate amount of random error required more than 6 minutes

of computer time. The main conclusion from the study was that the ICM

algorithm based on a direct application of Scott's (1964) cancellation

axiom performs rather poorly, and that a more computationally efficient

algorithm seems required to serve as a basis for computerized adaptive

model testing.

17

- - . - -1111111 Iff
CONSTRAINTS MATRIX .....

AP AQ AR AS AT OP 8V bR bS OT CP CO CS C CT OP OQ DR US OT LP EQ ER ES ET

AP 0o 1 1- 1 11 1 1 1 01 -1 1 1 1 1 1 1 1 1 1 1

-I ~~~AQ -1 0 1i 1 --- ~ 1-- i i a a

AR-i-1 0 1-1 1 1 1 1 -1-1-1-1 -1 -1-1 1 1 1 1 1 1 1

AS.--1 -1 0 -1 1 1 1 1 1 -1 -1 -1-1 -1-1 -1 1 1 j 1 1 1 1 1

AT -1 -1 1 10 1 1 1 1 1 -1 - -' -1 -1 -- 1 1 1 1 1 1 1 1 1 1

OP -1. -1 -1 -1 -1 0 1 1 1 1 -1 -I -1 -1 -1 -1 -i -1 -1 -1 -1 1 1 1 1

AT - 1 -I - 1 - 1 -1 - 1 0 1 I - - -L - 1 - - -1 - - - 1 1 -1 - -1

CP1- 1-1- - 1 - - 1 -1 1 1 1 - -1 -1 -1 -1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1

c5 .1 - 1 - -1 - -1 -1 -1 1 -1 -1 -0 .1 -1 -1 -1 -1 -1 1 -1 1 -1 -1 1 1

8 CA-I -1 -1 1 1 -1 1 1 01-1-01 0 1-1- 1 -1 1 -1 1 1 -1 -1 -1 -1

CS€-1 1 1 1 1 1 1 1 1-011 1 1-11 1 i 1 1 1 1 1 1 1

Cf-i 1 1 1 1 1 1 1 1 1-1-0 1 1 0 1 1 1 1 1 1 1 1 1 1

Co -1 1 1 1 1 1 1 1 1 1 -1 01 1 1 1 1 1 1 1 1 1 1 1 1 1

GO - 1 1 1 1 1 1-1-1-1.0-1-1 1 1 1 1 1 1 1 1 1

Ol-11 1 1 1 1 1 1 -1 1-1-1 1 -1-11 1 -1 1 1 1 1

Do -1 -1 1 1 -1 1 1 a I 1 -1 -1 -1 -1 -1 -1 0 1 1 1 1 1 1 1 1

OD -1 -1 -1 -1 -1 1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 1 1 " 1 1 1 1 1

Of -1 -1 -1 -1 -1 1 1 1 1 1 -1 -1 -1 - -1 --1 -1 -1 1 -1 .0 1 1 1 1 EP -1 -1 -1 -1 -1 .1 1 1 1 1 -1" 1- 1-1" 1 1-1 II1 1 1

EQ -1 -1 -1 -1 -1 -1 1 1 1 1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 *1 0 1 1 1

ER -1 -1 -1 -1 -1 -1 1 1 1 1 -1 -1 -1. -1 -1 -1 -1 -1 -1 -1 -1 -1 1 -1

ES -1 -1 -1 -1 -1 -1 1 1 1 1 -1 -1 -1 -1 1 *1 1 -1 -1 -! *1 -I -1 0 -1

*ET -1-11-11-1 -1 1 1 1 1-11 i-1 -1.101-1 1 - 1 -1-1 1 1 0

Figure 7. CUSIM3ATS matrix after second-order cancellatioa

18

SE *1

I - *
REFERENCES

Arbuckle, J., and LarLer, J. The number of two-way tables satisfying

certain additivity axioms. Journal of Mathematical Psychology, 13,

89-100, 1976.

Davisson, C. M. LPSUB: A FORTRAN subroutine for solving any standard

linear programming problem of a size compatible with the computer

being used. Naval Research Laboratory Memorandum Report 2383/

Computer Bulletin 27, January 1972.

Krantz, D. H., Luce, R. D., Suppes, P., and Tversky, A. Foundations of

measurement. New York: Academic Press, Volume I, 1971.

McClelland, G. H., and Coombs, C. H. ORDMET: A general algorithm for

constructing all numerical solutions to ordered metric structures.

Psychometrika, 40, 269-290, 1975.

Scott, D. Measurement structures and linear inequalities. Journal of

Mathematical Psychology, 1, 233-247, 1964.

Tversky, A. A general theory of polynomial conjoint measurement. Journal of Mathematical Psychology, 4, 1-20, 1967.

19

i- -
APPENDIX

,4SFORTRAN (4*3) / MSUS So5.1s7 AG 0

PROGRAM 4AIN

DIMENSIDI RUWA(b),COLP(b)9ITE4C?)

INTEGLR CONMAT,TIMESgSUMIT

INTEGERI APtFLAG

CHARACTER AAPPAST

COMMON /DATA/ NiANAMINAM2,NAM3,NAM4NPNPM,NP42,NPM3NP14,

IA (5) P (5) ,FLAGAA (5) ,PP (5) ,AST ,PSI (5,5) ,CONt4AT (625) ,TIMES (6'd5) SE

29IX

C INITIALIZATIONS

C

AA(1 =21

AA(2) = 2203

AA(3) = 23B

AA(4) = 483

AA(S) z 258

PP(1) = 4713

PP(2) = 5083

PP(3) a 5113

PP(4) =s?

PP (5). =638

AST =543

IX= 4321567

SA =1.

Sp= 1.

SE =0.

NA 5

NP 5

NAMi a NA - 1

NAM2 z NA - 2

NAM3 = NA - 3

NAM4 x NA - 4

NPM1 z=4 - 1

NPM2 = ND 2

NPM3 2 NP - 3

NPM4 = ND - 4

MAXIT = 150)

SumIT = o

FLAG = u

Do lol IzI92

ITERMI a

101 CONTINUE

I = (44)*

Do Io2 Jx19 1

CONMAT(J) = 9

TIMES(J) z=

102 CONTINUE *1 Do 103 I=1,NA UO lInJ J=19NP

IND) a IN)EX(IJq19J)

CONMAT(IJD) z

103 CONTINUE

Do 10 I=1,NA I.104 ROWA(I) aXNOHM(LX,0o9SA)

CONTINUE

Do lob Jz1,NP 21
NSFORTRAN (4.3) / MSUS 5.1 07/25/79 PAGE 002

COLP(J) a XNORM(tX,0*,SP)

105 CONTINUE

00 106 IulNA

DO lO6 JU1,NP

PSI(IJ) a ROWA(M) + COLP(J)

106 CONTINUE

C

C TEST CANCELLATION AXIOMS

C.

201 CALL CANI

IF (FLAG .NE9 0) 60 TO 401

CALL OUTPUT (NANPSUMITITE~,PSICONMATTIMESAAPP)

CALL CAN2

IF (FLAG NE. 0) GO TO 401

301 CALL OUTPUT (NAoNPSUNMT,1TE~,PSICONMAToTIMESAAPP)

GO TO 501

401 SUMIT a SUMIT * 1

ITER(FLAG) a ITEH(FLAG) * 1

IF (SUMIT *GT. MAXIT) GO TO301

FLAG B 0

GO TO 201

S01 CONTINUE

STOP

END

'2

221

' I 4
"SFORTRAN (4.3) / MSOS 5.1 07/25/79 PAGE 001

SUBROUTIE CANI

INTEGER CONMATTIMES

INTEGER AjA29A39A4qPI9P2tP39P4

INTEGER AqP9FLAG

CHARACTER AAPP9AST

COMMON /DATA/ NANAM1,NAM2,NAM3.NAM4,NPNPMINPM2,NPM3,NPM4,'

IA(5)P(5)FLAGAA(5)tPP(5)tASTtPSI(595)tCONMAT(625),TIMES(625)'hSE

EQUIVALENCE (A(1),IA)t(A(2)tJA),(A(3)tKA)

(P(I)vIP)9(P(2)#JP)q(P(3)9KP)

C

C CANI TESTS FIRST ORDER CANCELLATION

C

IF (NA oLTe 2 *ORo NP oLTo 2) GO TO 9999

C

C IN FIRST ORDER CANCELLATION, THE SEQUENCE LOGIC IN SCOTT*S THEOREM

C MAY BE REPLACED OY SUBSET LOGIC WITH NO LOSS OF GENERALITY AND WITH

C SUBSTANTIAL SAVINbS IN COMPUTER TIME,

C

C TEST EACH SUBSET OF 2 ELEMENTS IN A

C IA = 0

1 IA = IA * 1

IF (IA *EQ. NA) bO TO 9'99

JA = I

2 JA = JA + I

IF (JA o3T. NA) (O TO 1

IF (IA *EQ. JA) GO TO e

C

C TEST tAC SUBSET UF 2 LLEMENTS IN P

C

IP s 0

3 IP Z IP + 1

IF (IP .EU. NP) WU TO 2

JP a IP

4 'JP 2 Jp I

IF (JP o3T. NP) GO TO 3

IF (IP ,U. JP) GO TO 4

C

C TEST EACH PERMUTATION OF ELE4ENTS IN SUdSET OF A

C

00 102 lzl,2

DO 101 J=1,2

IF (I oE3o J) GO TO 101

C

C TEST EACH (NONTRIVIAL) PERMUTATION OF ELEMENTS IN SUBSET OF P

C

ll = J

JJ= I

C

C TEST FIRST ORDER CANCELLATIOn'

C

Al 2 IA

A2 = A(I)

A3 m A(J)

A4 a JA 23
WSORTRAN (4.3) /MSOS 5.1 07/25/79 PAGE 002

P1: a P

P2 a P(II)

P3 u PJ)

INDi a 140EX(A1,PLA2,P2)

IORI * C3?NMAT(INL1L)

IF (IORI *EU. 9)

1CALL ASK (IORIA1,PIA29P2)

IND2 xI41EX(A39P.3,A4,ID4)

10R2 aCONMAT(INUe)

IF (I0R2 *NE, 9) Go To bi

CONMAT(1402) = LORI

IND a INDEX(A4#P4#A3rPJ)

CONMAT(I1401 -IURI

Go TO 101

51 IF (IORZ .EU. 10kl) GO TO 101

FLAG =

WRITE (61,1001) AA(A1),PP(PI),ASTAA(A2),PP(P2,,

1 AA(A3)tPP(P3)tASTgAA(A4ioPP(P4)

100i FORMAT (//(1X#2AIIXvAlIXt2A1))

WRITE (6191002) LORI102

1002 FORMAT (/1X9215)

CALL ASA~ (IORAIP1,A29'2)

IF (lOW *NE* IoHl) GO TO 9999

CONMAT(I'4D2) =IOkI

IND = INDEX(A49P49A3,P3)

CONMAT(I'JD) =-IORI

Go TO 9999

101 CONTINUE

102 CONTINUE

Go TO 4

9999 RETURN

END

244
MSFORTRAN (4*3) / MSOS 5o, 07/25/79 PAGE 001

SUBROUTINE CAN2

INTEGER CONMATsTIMES

INTEGER AlA2,A3,A4,ASA6,PlP2.P3,P4,P5,P6

INTEGER APFLAb

CHARACTER AAPP9AST

COMMON /DATA/ NANAMINAMZNAM3,NAN4NPtNPMItNPM2NPM3.NPM4t

IA(5) P(S)tFLAGAA(5)tPP(5)tASTJPSI(5,5)tCONMAT(625)oTIMES(62S)tSE

EQUIVALENCE (A()I~tA)t(A(2)9JA)q(A(3)tKA)

EQUIVALENCE (P(1),IP)9(P(2)vJP)9(P(3),KP)

C CAN2 TESTS SECONU ORDER CANCELLATION

C

IF (NA i., 3 *OR, NP oLT. 3) GO TO 9999

C

C TEST EACH (NONREPEATIN6) SEQJENCE OF 3 ELEMENTS IN A

C

IA = 0

1 IA = IA + 1

IF (JA o3T. NA) WO TO 9999

JA = 0

2 JA = JA + 1

IF (JA *GTo NA) 60 TO I

IF (JA *E . JA) 60 TOd

KA = 0

3 KA = KA + I

IF (KA oGT. NA) GO TO 2

IF (IA .Q. KA .ORo JA .EQ. KA) GO TO 3

C

C TEST EACH (NONREPEATINO) SEQJENCE OF 3 ELEMENTS IN P

C

IP = 0

4 Ip = Ip

IF (IP .37. NP) GO TO 3

Jp U 5 ,JP = JP

IF (JP .3T. NP) WO TO 4

IF UP .E o JP) 6O TO b

KP = 0

6 KP = KP + 1

IF (KP *GTo NP) 5O TO b

IF (IF oE*o KP oUk. JP *EQ. KP) GO TO 6

C

C TEST EACH PERMUTATION OF ELE4ENTS IN SEQUENCE OF A

C

00 lOb 1=1,3

DO 105 J=1,3

IF (I oE~o J) GU TO 105

00 104 K.13

IF I *EQ* K oORo J ,Ewo K) 30 TO 104 I C

C TEST LACH (NONTRIVIAL) PERMUTATION OF ELEMENTS IN SEQUENCE OF P

C

DO 103 II1.3

IF (1 .El. 1 oANU, 1i oEQ. 1) GO TO 103

DO 10 JJz1l3

25

- I * r
mSFORTRAN (4.3) / tS05 S.1 07/25/79 PAGE 002

IF (11 .EQ. Jj) (i0 TO 102

IF (J *E~. 2 oAN~o JJ sEQ* 2) GO TO 102

DO 101 KK2193

IF (11 .EQ. KK *ORo JJ .EQo KK) GO TO 101

IF (K .EQ- 3 .ANU). KK .EQ- 3) GO TO 101

IF (K oE~o 1 wANUe KK .EQ. 1 9ANO. I .EQ* 3 *AND. 11 *EQ. 3)

IF (K *EQ, 2 eANUo KK *E~Q* 2 *AND. J sEQ. 3 *AND. JJ *EO. 3)

IGo To 101

C TEST SECOND ORL)ER CANCE.LLATION

C

Al =IA

A2 =A(I)

A3 =JA

A4 -A(J)

AS A(K~)

A6 = KA

P1 =IP

P2 = P(I13

P3 z JP

P4 = P(Jj)

P5 c P(KjK)

P6 = KP

INDI 2 14DEX(AlPjA29P2)

IDRI a C3NMAT(INU1)

IF (IDR1 oEU. 9)

1CALL ASK (IOHIAl9PlA29P2)

IND2 UNEX(A3vP3qA4Ip43

IOR2 z CONMAT(INUi2)

IF (I0R2 oEU9 9)

1CALL ASK (IOR29A3,P3,A49P4)

IF (IOR2 oNEo lORI,) Go TO. 101

IN03 z I4OEMA5vP~,A6,Pb)

10R3 - CONMAT(INU3)

IF (IUR3 9NE. 9) (i0 TO 51

CONMAT(1N03) z 10R2

IND z INDEX(A69PbA59Pb)

CONMAT(I1ID) = -10R2

Go TO 101

51 IF (I0H3 oE(J. 10HZ) GO TO 101

FLG =

vWRITE (6191001) AA(Al) ,gPP(Pl) ,ASTAA(A2) ,PP(P2),

I AA(A3)OPP(P3)IASTAA(A4),PP(P4),

AA(A5)9P'P(P5)9ASTAAIA6)OPP(P6)

1001 FORMAT (//U1X92AI1X9A191XZA1W)

WRITE (6191002) IOR19IUN29IOR3

1002 FORMAT (/1X9315)

CALL ASK IIOR9A1,P1,A29P2)

IF (IOR *NE. IORL) 60 TO 9999

CALL ASK (1OR9A39V39A49'4)

IF (JOe .NEo IORe) GO TO 9999

CONMAT(14.3) a IH2

IND a IN)EX(A69PbvASP5)

CONMAT(14DO) x -IUR2

Go TO 9999 26

i t
NSFORTRAN (4.3) /MSUS 5.1 07/25/79 PAGE 003

101 CONTINUE J102 CONTINUE

103 CONTINUE

104 CONTINUE

105 CONTINUE

106 CONTINUE

Go To 6

9999 RETURN

END

27

I W4L 
MSFORTRAN (4,3) /MSOS 5.1 07/25/79 PAGE 001

SUBROUT14E ASK (IORoIJtKtL)

INTEGER APqFLAG

INTEGER CONMAT, TIMES

CHARACTER AAPPAST

COMMON /DATA/ NANAM1,NAM2,NAM3,NAM4,NP9NPM1,NPM42,NPM39NPM49

JA5V(5) FLAGAA (5) PP (5) AST ,PSI (595) ,CCNMAT (625)9 TIMES (625) .SE

2,IX

C

C ASK QUERIES THE sUBJLCr CONCERNING THE PREFERENCE RELATION

C FOR STIMJLUS (1,J) RELATIVE ro SIMULUS (KL).

C

PREF a PSI(IJ) - PSI (KtL)

PREF a PREF # XNOHM(1X9O.09SE)

IF (PREF) 19293

1 bOR = -1

GO TO4

2 IOR = 0

Go TO 4

3 IOR = I

4 CONTINUE

IND = INDEX(IJKqL)

CONMAICIND) x IOR

TIMES(IND) zTIMESCINO) *1

IND = INDEX(KtLglgJ)

CONMAT(IJu) -IUR

RETURN

END

28
mSFORTRAN (493) /MSOS 5.1 07/25/79 PAGE 001

SUBROUT1~4E OUTPUT (NANPSIJMIT, TERPS1,CONMATTIMESAAPP)

DIMENS104 ITER(2),PSI(595)

INTEGER SUMITCONMAT(beb)9Tl(ES(625),TEMP(595)

CHARACTER AA(5)tPP(5)

C

C OUTPUT ITERATIONS

C

WRITE (6191001) SUMIT, (ITER(I),I=192)

1001 FORMAT dlHl//1IH ITERATIONS

I //6H SUmITteX,13

2 /b ITER1,2X913

3 //6Ht ITER2,eX913)

C OUTPUT CONSTRAINTS MATRIX

WRITE (6191002) ((AA(I)9PPCJ)vj=1,NP),121,NA)

DO 101 I=1,NA

DO 101 J=1,NP

Do 1 K=lNA

0O 1 L=19NP

IND = INDEX(IjKgL)

TEMP(IL) = CONMAT(IND)

1 'CONTINUE

WRITE (6191003) AA(I),PP(J),((TEMP(KL)L1,NP),K=1,NA)

1003 FORMAT (/2A,2Al,2X925(1292X))

101 CONTINUE

C

C OUTPUT TIMES MATHLX

C

WRITE (6191004) ((AA(I)9PPCJ)9J.1,NP)qI1,9NA)

1004 FORMAT (1HI//13H TIMES MATRIX//6X925(2A1,2X))

Do 102 I=1,NA

DO 102 J=1,NP

DO 2 K=1,NA

(JO 2 L=1,NP

IND = INOEX(1,JtKtL)

TEMP(K9L) =TIMES(IND)

V CONTINUE -

WRITE (6191003) AA(I),PIe(J),((TEMP(KL),L=1,NP),K=1,NA)

102 CONTINUE

C

C OUTPUT PSI MATRIX

C

WRITE (61,1005) (PP(J),JxlNP)

1005 FORMAT (1H1//11H PSI MATRIX//bX,5(5X*A1,4X))

Do 1o3 1=1,NA

WRITE (61v1006) AA(I),(PSI(IJ),J:1,NP)

10OA FORMAT (/3XAl,2X95F10.4)

103 CONTINUE

RFTURN

ENO

29

'77

-~ ~~ ~~ Z$t-6'j~dS(h
WSFORTRAN (4,3) / MSOS 5ej 07/25/79 PAGE 001

FUNCTION INUEX(lgJIKL)

COMMON /DATA/ t4,LUUM(4),N

C

C INDEx cO4PuTES THE LOCATION OF ELEMENT (IvJK,LI IN A LINEAR

C F()UR-t0MENSIONAL ARRAYV OF SIZE (NNNN)o

C

INDEx I *(J-13*M + (K-1)*NON *(L-1)#4*N*M

RETURN

END

30
MSFORTHAN (4.3) / MSUS 5.1 07/25/79 PAGE 001

FUNCTION XNORM(IXtXMtXS)

C

C XNORM CO4PUTES A NORMALLY DISTRIBUTED RkNDOM VARIABLE wITH

C MEAN XM AND STANUARD DEVIATION XS.

C

XNORM = 0.

DO 101 1=1,12

IY = Ix 0 4093

IF (IY) 1292

I IY = IY + 838b607 + I

2 YFL = IY

YFL = YFL O.119209LE-U6

Ix = ly

XNORM = XNORM * YFL

101 CONTINUE

XNORM = (XNORM - b.) * AS * AM

RETURN

END

31

mamma"
DISTRIBUTION

- ARI DUttibutioi List

4 QASO (M&RAI 2 HOUSACOEC, Ft Ord, ATTN: Library

2 HODA IflAMI CSZI 1 HOUSACOEC. Ft Ord, ATTN: ATEC-EX-E -Hum Factors

?HODA IDAPE PBRI 2 USAEEC, Ft Benjamin Harrison, ATTN: Libiaiv

I PODA (DAMA AR) 1 USAPACDC, Ft Benj~amini Harrison, ATTN: ATCP -HR

I HtO0A IDAPE HRE-POI 1 USA Comm-Elect Sch. Ft Monimouth, ATTN: ATSN-EA

IHODA ISGRD-ID) 1 USAEC, Ft Monmouth, ATTN: AMSE L -CT- HOP

IHUDA IOAMI-DOT-C) I USAEC, Ft Monmiouth, ATTN: AMSEL-PA P

I HOVA (DAPC PMZ-A) 1 USAEC, Ft Monmouth, ATTN: AMSEL-SI-CB

I HODA (DACH-PPZ A) I USAEC. Ft Monmouth, ATTN: C, Fact Dev Dr

I HODA (DAPE-HRE) 1 USA Materials Sys Anal Agcy, Aberdeen, ATTN: AMXSY -P

1 HODA IOAPE.MPO-CI I Edgewood Arsenal, Aberdeen, ATTN: SAREA BL- H

1 HODA IDAPE DW) 1 USA Ond Ctr & Sch. Aberdeen, ATTN: ATSL-TEM-C

1 HODA (DAPE-HRL) 2 USA Hum Engr Lab, Aberdeen, ATTN: Library/Dir

I HODA IDAPE-CPS) 1 USA Combat Arms Tnig Rd, Ft Benning, ATTN: Ad Supervisor

I HODA 1DAFD-MFA) 1 USA Infantry Hum Rsch Unit, Ft Benning, ATTN: Chief

IHOA WDARD-ARS-PI 1 USA Infantry 9d, Ft Banning. ATTN: STEBC -TE-T

I HODA (DAPC-PAS-AI 1 USASMA. Ft Bliss, ATTN: ATSS--LRC

IHODA (DUSA-OR) 1 USA Air Def Sch, Ft Bliss, ATTN: ATSA CTO ME

I H-IDA (DAMO-RQRI 1 USA Air Def Sch. Ft Bliss, ATTN! Tech Lib

I 0(A (DASGI 1 USAaiAir Del Rd. Ft Bliss, ATTN: FILES

I HO)A IDA I -PI) 1 USA Air Def Bd. Ft Bliss. ATTN: STE BD-PO

I Chuel. Consult Div (DA-OTSG). Adelphi, MD 1 USA Cmd & General Stf College, Ft Leavenworths, ATTN: Lils

1 Mud Asst, Hum Res, ODDR&E, OAD IE&LSI 1 USA Cmii & General St College, Ft Leavenworth. ATTN: ATSW-SE-L

I lHO USARAL. APO Seattle. ATTN: ARAGP-R 1 USA Cmii & General 5sf College, Ft Leavenwosrthi, ATTN: Ed Advisor

I HO First Army, ATTN: AFKA-01 TI 1 USA Combuined Arms Cmbt Dev Act, Ft Lravreiiworth, ATTN: DepCdr

2 HO Fifth Army. Ft Sam Houston 1 USA Combined Arms Cmbt Dev Act. Ft Leavenworfth, ATTN: CCS

1 Dii. Army Sit Studies Ofc. ATTN: OAVCSA IDSP) 1 USA Combinred Arms Cnsbt Dev Act. Ft Leavenworlth. ATTN: ATCASA

I ofi- Chief iif Sit, Studiei Ofe 1 USA Combined Arms Cmbt Dev Act. Ft Leavenworth, ATT'N: ATCACO-E

I DCtSl'ER, ATTN: CPSIOCP I USA Cnim.uined Arms Cmbt Dew Act. Ft Leavenworth, ATTN: ATCACC-Cl

I The Army Lili. Priusarioii ATTN: RSB Chief 1 USAECOM., Night Vision Lab. Ft Belerjir, ATTN: AMSEL-NV-SD

1 The Army Lilt. Pentagon, ATTN: ANRAL 3 USA Computer Sys Cmd. Ft Belvoir. ATTN : Tech Library

1 Of c, Asti Sect of the Army (R&D) 1 USAMERDC, Ft Belvoir, ATTN: STSFB-00

I Tech Support Ofc. OJCS 1 USA Eng Sch, Fit Belvair, ATTN: LibiratV

1 USASA. Arlington, ATTN: IARD-T 1 USA Topographic Lab, Ft Belvoir. ATTN: ETL TO-S

1 USA Rich Of c, Durham,. ATTN: Life Sciences Dir 1 USA Topographic Lab, Ft Relvoir, ATTN: STINFO Center

2 LJSARIEM, Naick, ATTN: SGRD-UE-CA 1 USA Top~ographic Lab, Ft Belvoir, ATTN: ETL GSL

I USAi rc., I I CI.iyioii Al I IN: 'frf TC MO A I USA liitiIlijcno. Ctr & Scu. Fi Hiiuhiica. ATTN: CTD MS

I USAIMA. Ft Bragg. AFN: AISU-CTO-OM I USA Intelligence Ctr &Sch, Ft Huachuca, ATTN: ATS-CTD-MS

1 USAIMA. Ft Bragg, ATTN: Marquat Lib 1 USA Intelligence Ctr & Sels, Ft Huachuca, ATTN: ATSI-TE

1 US WAC Ctr & Sch, Ft McClellan, ATTN: Lib 1 USA Intelligence Ctr & Sch, Ft Huachuca. ATTN: ATSI-TEX-GS

I US WAC Ctr & Sch. Ft McClellan, ATTN: Tnq Dir 1 USA Intelligence Ctr & Scti, Ft Huachuca, ATTN: ATSI-CTS-OR

1 USA Quartermaster Sch, F t Lee. ATTN: ATSM.TE I USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI-CTO--OT

I Intelligence Material Dev Ofc, EWL, Ft Holabird 1 USA Intelligence Ctr & Sch, Ft Huactiuca, ATTN: ATSI-CTD-CS

1 USA SE Signal Sch. Ft Gordon. ATTN: ATSO-EA 1 USA Intelligence Ctr & Sch, Ft Huachuca. ATTN: DAS/SRD

I USA Chaplain Ctr & Sch. Ft Hamilton, ATTN: ATSC-TE-RD 1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI-TEM

1 USATSCH, F t Eustis. ATTN: Educ Advisor 1 USA Intelligence Ctt & Seft, Ft Huachuca, ATTN; Library

1 USA War Colleqe. Carlisle Barracks. ATTN: Lib 1 CDR, HO Ft Huachuca, ATTN: Tech Ref Div

2 WRAIR. Neinrusychiatiy Div 2 CDR, USA Electronic Prvg Grd, ATTN : STEEP--MT-S

1 DLI, SOA. Monterey I HO, TCATA. ATTN: Tech Library

1 USA Concept Anal Agcy, Bethesda, ATTN: MOCA-MR 1 HO. TCATA. ATTN: AT CAT.OP-a, Ft Hood

1 USA Concept Anal Agcy, Bethesda, ATTN: MOCA-JF I USA Recruiting Cmd. Ft Sheridan. ATTN: USARCPMI-P

1 USA Arctic Test Ctr, APO Seattle. ATT'N: STEAC-PL-MI 1 Senior Army Adv., USAFAGOD/TAC. Elgin AF Aux Fid No.09

1 USA Arctic Test Ctr, APO Seattle, ATTN: AMSTE-PL-TS I HO, USARIPAC. DCSPER. APO SF 9655% ATTN: GPPE SE

1 USA Armament Cmd, Redstone Arsenal, ATTN; ATSK.TEM I Stimason Lib. Academy of Health Sciences. Ft Sam Houston

- II USA Armament Cmii, Rock Island. ATTN: AMiSAR.TDC 1 Marine Corps Inst., ATTN: Dean-MCI

I FAA.NAFEC, Atlantic City. ATTN: Library I HO. USMC. Commandant. ATTN: Code MTMT

1 FAA-NAFEC, Atlantic City, ATTN: Human Engr Br I HO. USMC, Commandant. ATTN: Code MPI-20-26

1 FAA Aeronautical Ctr, Oklahoma City, ATTN: AAC-44D 2 USCG Academy. New London. ATTN: Admission

2 USA Ei Arty Sch, Ft Sill, ATTN: Library 2 USCG Academy. New London, ATTN: Library

I USA Armor Sch, Ft Knox. ATTN: Library 1 USCG Training Ctr, NY, ATTN: CO

1 USA Armror Sch. Ft Knox. ATTN: ATSB-DI-E I USCG Training Ctr, NY, ATTN: Educ Svc Otc

I USA Armor Seft. Ft Knox, ATTN: ATSB-DT-TP 1 USCG, Psychol Ret Br, OC, ATTN: GP 1/62

1 USA Armor Sch. Ft Knox. ATTN: ATSO-CO-AD I HO Mid.Rmnge Br. MC Dot, Ousintito, ATTN: PgS Div

33 I------
IUS Marine Corps Liaison Ofc. AMC. Alextandria. ATTN AMCGs-r I Def &Civil Inst of Enviro Medicine. Can11ada

1I STAO.F ore TN TOE I AIR CRESS, Kaqtsngtori, ATTN: Info Sys Or

'.I USATRADOC, Ft Monrroe ATTN: ATPROAD I Militarpykofogik Tianete. Copenhagn

1 USATRADOC. Ft Monroe, ATTN! ATS EAD 1 Military Attache. French Embassy. ATTN: Doe Sac

1 USATRFosCmii Ft Mcnroen, ATTN: Librar EA Medecin Chef, C.E.R.P.A.-Arsenal. Toulon/Naval France

2 tiSA Aviation Test Od. Ft Rucker, ATTN: STEOG-PO 1 Drfen, No fAPIDelhieinst

I USA Agcy for Aviation Safety, FN Rkicket. ATTN: Library 0fDfne e eh

1 USA Agcy for Aviation Safety, Ft Rucker, ATTN: Educ AdIvisor 1 Pert Rich Dfc Library. AKA. Isral Defense Forces

I USA Aviation Sch. Ft Rucker, ATTN: PO Drawer 0 1 Ministeris wan Defanuia. DOOP/KL_ Afd Sociaal

1 HOUSA Aviation Sys Cmii. St Louis. ATTN: AMSAV-ZDR Psychologische Zaken. The Hae. Netherlands

2 USA Aviationi rys Test Act., Edwards AFB, ATTN: SAVTE--T

I USA Air Del Sch, Ft Bliss, ATTN: ATSA TEM

I LISA Air Miiliility Rsclr & Crev Lal,, Moffett FirI. ATTN: SAVDL -AS

IUSA Aviation Scfr. Res Trig Mgt, Ft Pucker. ATTN: ATST-T-RTM

IUSA Aviation Sch. CO. Ft Pucker, ATTN: ATST-D-A

1 HO, DARCOM. Alexandria, ATTN: AMXCD-TL

1 HO. DARCOM. Alexandria, ATTN: CDR

1 US Military Academy, West Point, ATTN: Serials Unit

1 US Military Academy. West Point. ATTN: Ofc, of Milt Ldrshp

IUS Militaiy Academy. West Point. ATTN: MAOR

I USA Standardization Gp, UK, FPO NY. ATTN: MASE-GC

1 Ofc of Naval Fisch, Arlington, ATTN: Code 452

3 Of c of Naval Psch. Arlington. ATTN: Code 458

I Of c of Naval Psch, Arlington, ATTN: Code 450

1 Qfc of Naval Rich, Arlington, ATTN: Code 441

1 Naval Aerospc Med Ret Lah, Pensacola, ATTN: Acous Sch Div

I Nival Aernslic Med Res Lab, Pensacola, ATTN: Code L51

I Naval Aerostpc Med Res Lab. Pensacola, ATTN: Code LS

1 Chief of NavPrers, ATTN: Pers.OP

1 NAVAIRSTA. Norfolk, ATTN: Safety Ctr

1 Nav Oceanographic. DC. ATTN: Code 6251, Charts & Tech

1 Center of Naval Anal, ATTN: Doc Ctr

I NavAirSysCom, ATTN: AIP--5313C

I Nayr BuMed. ATTN: 713

1 NavHellcopterSubSqua 2. FPO SF 96601

I AFHRL [FT) Williams AFB

1 AFHRL (TT) Lowry AFB

1 AFHPL (AS) WPAFB, OH

2 AFHPL fDOJZ) Brooks AFS

I AFHRL (DOJN) Lackland AFO

1 HOUSAF (INYSD)

I HOUSAF (DPXXAI

1 AFVTG (RD) Randolph AFB

3 AMPL (HE) WPAFO, OH

2 AF Inst of Tech, WPAFB, OH. ATTN: ENE/SL

1 ATC IXPTD) Randolph AFB

1 UJSAF AeroMed Lb., Brooks AFB (SUL-4). ATTN: DOC SEC

1 AFOSR (NLI. Arlington

I AF Log Cmd, McClellan AFB, ATTN: ALC/DPCR8

1 Air Force Academy. CO, ATTN: Dept of Sol Scri

5 NavPers & Dew Ctr. San Diego

2 Navy Med Neuroipsychiatric Rsch Unit. San Diego

I Nay Electronic Lab. San Diego. ATT'N: Res Lab

1 Nay TrirgCen, San Diego. ATTN: Code 9000--Lib

1 NavPostGraSclt, Montere!y. ATTN: Code 55Aa

1 NavPostGraSch, Monterey. ATTN: Code 2124

1 NavTfrigEquipiCtr. Orlando. ATTN: Tech Lib

1 US Dept of Labor, DC, ATTN: Manpower Admin

1 US Dept of Justice. DC. ATTN: Drug Enforce Admin

INat Biur of Standards, DC. ATTN: Computer info Section

1 Nat Clearing House for MH-* Info, Rockville

1 Denver Federal Ctr, Lakewood. ATTN: BLM

12 Defense Documentation Canter

4 Dir Psych. Army Eq, Russell Dfcs. Canberra

1 Scientific Advsr, Mil Bd, Army Hg. Russell Ofcs, Canberra

1 Mil and Air Attache. Austrian Embassy

1 Centie de Recherche Des Facteurt. Humaine de Is Defense

Nationale, irtistels

2 Caniadfian Joiiit Stall Washington

1 C/Air Staff, Royal Canadian AF, ATTN: Pars; Std Anal Br

3 Chief. Canadian Def Rsch Staff. ATTN: C/CRDSIW)

4 Bits.h Del Staff, British Embassy, Washington

34

-am-~rs v ~