rD

Research Note 86-41

DATA COLLECTION VIA A QUASI-EXPERIMENTAL SIMULATION TECHNOLOGY: I. *

MULTIPLE MEASUREMENT OF PERFORMANCE EXCELLENCE IN COMPLEX

AND UNCERTAIN MANAGERIAL TASKS

Siegfried Streufert, Rosanne M. Pogash. and

0') Mary T. Piasecki ,

Milton S. Hershey Medical Center

Pennsylvania State University

cc

Contracting Officer's Representative

George Lawrence

BASIC RESEARCH

Milton S. Katz, Director

CLECTE

U. S. Army

.Research Institute for the Behavioral and Social Sciences

April 1986

Approved for public relese; distribution unlimited.

86 5 16 07
tJ

U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

Ij

WM. DARRYL HENDERSON

EDGAR M. JOH4NSON COL, IN

Technical Dirctor Commanding

Research accomplished under contract for

the Department of the Army

Milton S. Hershey Medical Center

Pennsylvania State University

Technical review by

Steve Kronheim

(DTIC) to comply with regulatory requirements. It he% been given no primary distribution other than to DTIC

and wvill be available only through DTIC or other reference services such as the National Technical Information

Service (NTIS). The vitwi. opinions, and/or findings contained in this report ore those of the author (sl and

should not be construej as an official Depaitnri of the Army position, policy. or decision, unless so designated by other official documentation.
# REPORT DOCUMENTATION PAGE READ INSTRUCTIONS

REPORT NUMBER 2. GOVytCCESSIONO CIP EFR. ENTS CATALOG NUMBER 0.

ARI Research Note 86-41 A

A6 ftINK

4. TITLE (and Subtitle) 5. TYPI OF REPORT & PERIOD COVERED

Data Collection Via a Quasi-Experimental Interim Report %

Simulation Technology: I. Multiple Measurement __

of Performance Excellence in Complex and 6. PERFORMING ORG. REPORT NUMBER

Uncertain Managerial Tasks _ _ _ _

7. AUTHOR(e) 8. CONTRACT OR GRANT NUMBER(s)

Siegfried Streufert, Rosanne M. Pogash, and

Mary T. Piasecki MDA 903,83-C-0106

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT. TASK AREA & WORK UNIT tDUM8ERS

Milton S. Hershey Medical Center V

Pennsylvania State University 2Q161102B74F

Hershey, PA 17033

It. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

US Army Research Institute for the Behavioral April 1986

and Social Sciences, 5001 Eisenhower Ave., 13. NUMBER OF PAGES

Alexandria, VA 22333-5600 71 14. MONITORING AGENCY NAME 4 AOORESS(It different tre'. Controlling Olfice) IS. SECURITY CLASS. (of this report)

Unclassified

ISa. DECL ASSI FI CATION/ DOWNGRADING

SCHEDULE

16. DISTRIBUTION STATEMENT (of this Report)

Approved for public release; distribution unlimited.

17. DISTRIBUTION STATEMENT (of the abstract entered In Block 20. It different from Report)

II. SUPPLEMENTARY NOTES

• -.

COR • C T.rpnrc"-

19. KEY WORDS (Continua on reverse aide if necasary and Identify by block number) M %

)quasi-experimentalj decision-making)

simulation, complex and uncertain tasks,

performance excellence- multiple measures.

time-event matrix.

2G. ABSTRACT (C aum am ,everse ag Ift wecoweaay and idantif by block number)

A simulation technique is used to determine whether complexity (multidimensionality) of task performance is complex managerial tasks is

trainable. The present report is specifically concerned with measurement.

Previous simulation based measurement (cf. Streufert, 1983) which had

included sixteen measures was extended to thirty-seven primary measures and

twelve derived measures. Information-is provided on the characteristics

and purpose of each of those measures. In addition formulas or related

statements that allow calculation of performance scores by other researchers

DO FOM 143 EDITION OF I NOV 6S IS OBSOLETE UNCLASSIFIED

SECURITY CLASSIFICATION OF THIS PAGE (When Date Entered)

*.' . . . .
- . -Tr- -- VW

Item 20 (Continued)

and/or in other settings is provided. Further, this report considers the

Time-Event Matrix on which measurement is based. :

ccesion 1

Uacceioil NTIS CRAIFor

Unannouwsced

By . .......

r VI~b~t Codes

iD UtNCLASSIFIED101

.1..
Background and Purpose

A contract between the U.S. Army Research Institute for the

Behavioral and Social Sciences and Pennsylvania State University,

The Milton S. Hershey Medical Center, (S. Streufert, principal

investigator) stipulates that the contractor will use a simulation technique to determine whether complexity (multidimensionality) of task performance in complex managerial tasks is

trainable. For that purpose, the contractor developed and/or

improved a computer based simulation system during the first year

of effort. The simulation was to be pretested during the second

year of the contract.

Initial development was completed on Apple computers which

were, however, too limited in capacity to effectively operate

the simulation system. To overcome this problem, IBM AT enhanced

computers were ordered. Delivery, as scheduled by IBM, would

have resulted in a three months delay in scheduled tasking.

Unfortunately, IBM subsequently cancelled all delivery dates

because of a faulty component of the AT system and rescheduled

delivery for some 8 months later. As a result, final simulation

development, design of measurement techniques and simulation

pretesting was delayed for periods up to 12 months. With IBM AT

systems finally available during the late summer of 1985,

efforts to finalize the simulation technology and to develop

measurement techniques was initiated. Data collection on those

segments of the contracted efforts are now completed. The

obtained data and the associated methodology will be considered
2

in three reports.

The present report is specifically concerned with measurement. Previous simulation based measurement (cf. Streufert,

1983) which had included sixteen measures was extended to thirtyseven primary measures and twelve derived measures. This report

provides information on the characteristics and purpose of each

of those measures. In addition, it will provide formulas or

related statements that allow calculation of performance scores

by other researchers and/or in other settings. Further, this

report considers the Time-Event Matrix on which measurement is

based.

A second, subsequent report will be concerned with measurement reliability. Any procedure which requires test and retest

of performance functions must assure the reliable measurement of

some phenomenon of interest on two separate occasions. For

present purposes, reliability across two measurements from two

parallel simulations with diverse scenarios is required. With

training interspersed, measurement across simulations must be

comparable if changes in performance are to be discovered.

Variable error must be kept to as low a value as possible. The

second re-ort in this series will focus on the question of

,vw

inter-simulation reliability of the measures that are considered

in this report.

A third report will be based on data analysis that is now

in progress and/or in the planning stage. This last report in

the series devoted to simulation technology will be concerned

I
• i. i T: ', AX W A;.. S.VK S : ,- - . rW2'V ' WAr* . .. pr J .rr , - cr.. .. - . -r. ,

3

with the interrelationship among the various measures (based on

factor analytic techniques) and with measurement validity.

Validity analysis will be achieved by calculating correlations

among measures that load highly on obtained factor scores with

several secondary indicators of performance, e.g. job level (at

age), number of promotions, income (at age) and so forth.

Reports that follow the present series will be concerned with

measured training effects per se.

Discovering Performance Characteristics

A few executives are fabulously successful. Others fail

early in their careers. Most do their jobs adequately. What

makes the difference?

The majority of executives, especially at senior levels,

are highly intelligent. They have gained ample experience.

They have been well trained. Yet, if we observe what they

understand, what they know and what decisions they make we still

cannot distinguish those who are great from those who are

mediocre. Many observers of executive performance have watched

and recorded what great executives do. Unfortunately, their

descriptions of excellence would hold in one organizational

setting but may not apply in another.

The difference between excellence and mediocrity is not

generally found in what an executive may know, think or decide.

The difference is not just determined by the choice of correct

and appropriate actions. It is not based on the capacity to

follow a preconceived strategy toward some fixed goal. It is "

.5
- 4WV4

not found in the tendency to amass all relevant information that

can be obtained. Even mediocre executives can do all those

things - and most can probably do them well.

Excellence in executive performance depends on how an

experienced executive thinks, plans and responds to task demands.

It is inherent in the creative alternatives that are considered.

It is evident in the development of multiple strategies that

proceed from step to step, always adaptive, with goals that are

general and become more and more defined as they are approached.

It is found in conclusions that are checked, reconsidered and

reassessed as more information arrives. It is found in the "'1

adaptive flexibility that permits different approaches to

different people and tasks, that permits the executive to switch

from strategic planning to immediate action when emergencies

arise and to return to a planning mode when the emergency has

been conquered.

How can one measure that executive excellence? It requires

a view of an executive's actions over time, in different situations, relevant to diverse task demands. It requires assessing

how executives develop, interrelate and apply their ideas. It

requires an analysis of the steps which executives take to

approach and attain their goals. A simple test, an interview or

even a short task cannot provide us with adequate information

about executive capacity. Just as a photographic snapshot

cannot tell us what a person will do in a few hours, the usual

measurement techniques and well intentioned observations tell us

:.':
-~I . R W V .

-.°

5 5

little about executive performance across a variety of task

demands.

There have been very few efforts that have analyzed execu- VA

tive performance through a number of potentially interrelated

events or through a series of decisions that are part of a

strategic effort. If we wish to discover how executives function, such a time oriented analysis is exactly what is needed.

Observations that are more or less casual ('e.g. those of Peters

and Waterman, 1982, Peters and many other recent observers of

executive functioning) are indeed valuable, but they rarely

generate the necessary depth of understanding. Much more useful

are, for example, the careful efforts of Dan Isenberg (1984) who

has recorded both the actions and the rational for consecutive

executive actions across considerable time periods. The present

approach is quite similar to that of Isenberg, but uses a A

simulation methodology to (1) automate data collection procedures, (2) to obtain multiple independent measures of performance, (3) to maintain control over relevant independent variable

manipulation across time and participants and, last not least,

(4) to explore lengthy periods of performance during which

strategic efforts might develop. The simulation permits this

kind of performance measurement in reduced time periods via the .-

compression of time experience. The technique is described in

the next section of this report.

Data Collection via a Quasi-Experimental Simulation

The simulation technique which was developed in part for

%".
6

present research effort is best defined as a "quasi experimental

simulation" (Fromkin and Streufert, 1983; Streufert and Swezev,

1985). The methodology allows the introduction of controlled

independent variables not only at the beginning of the simulation

task but maintains control over independent variables throughout

the simulation. The considerable length of time (several hours)

during which participants deal with their simulated environment

permits the simultaneous and/or successive manipulation of

several experimental variables.

Continued control over experimental variables throughout

the simulation permits the application of two kinds of performance measurement: (1) repeated collection of data throughout the

simulation period to record the effects of conditions that are

experimentally modified as the simulation progresses (for

example one might measure the effects of different load levels,

e.g., Streufert and Driver, 1965; Streufert and Schroder, 1965;

Streufert, 1970), and (2) one-time collection of data that are

relevant to some specific preprogrammed single event which

occurs in the simulation. The former measurement techniques are

particularly useful to assess the (structural, "style of thinking" that underlies task performance. The latter can capture •

structural thought processes only to some extent, but is especially useful to obtain information about the appropriateness,

speed and accuracy of responses to specific task demands.

The development of measurement in our quasi-experimental

simulation technique is best described as an ongoing process. '.

. . ......
Since the computer system which operates the simulation technology records a large variety of actions, plans and response

characteristics of participants, newly developed measures can

often be derived from data collected at a much earlier date.

New measures may or may not be entirely independent of previous

measures, (overlap of meaning), others may be entirely independent (orthogonal). The precise interrelationships of the various

measures will be evaluated in a factor analytic procedure based

on data from approximately 80 simulations. The results will be

reported some time later this year (likely in the summer of

1986).

At present, the computerized simulation system records

performance characteristics in terms of interrelationship

vectors and calculates scores for more than fifty performance

measures. Thirty-seven of those measures are more or less

complex calculations of performance frequencies. The remaining

measures relate these frequencies to the number of actions taken

(decisions made) to obtain proportions of performance characteristics as they relate to general activity.

Data collection for the simulation technique is reflected

in a graphic procedure that has become known as a "Time-Event

Matrix." The rationale for data collection and measurement

procedures are best explained via that matrix system. The

following section of this paper will focus on the matrix procedure. Subsequently, the measures themselves will be explained in

some detail.
The Time-Event Matrix

The task of an individual or a task-oriented group operating

in the world outside of the simulation laboratory is rarely

limited to deciding on a single event within a limited context.

For example, most decision makers in applied settings must

respond to an ongoing series of inputs from their environment.

The resulting output is usually a sequence of actions determined

in part by some plan and in another part by the necessities of

dealing with current events. The output may consist of primaril.

"respondent" actions or it may reflect some degree of "strategy,"

i.e., decisions which are interrelated with each other and occur

in a planned sequence to achieve some kind of goal. Whether

or not individual or group actions do reflect pure respondent

behavior, whether they reflect some kind of strategy (and the

level and/or characteristics of that strategy) may be of considerable importance for the outcome of the task effort. The

majority of previous researchers have not focused on measuring

or describing such differences. To alleviate that problem,

Streufert and associates have developed a time-event matrix to

help researchers or observers to identify different kinds of

actions and their frequencies, as they occur in naturally

complex task settings. Reliability and validity for some of the

measures derived from time-event matrices have been established .

in previous efforts. Additional reliability and validity data

will be reported in a subsequent report.

Performance quality, particularly in complex tasks, is
determined by at least two major components of individual or

group efforts: (1) appropriate knowledge about what responses

are potentially correct or incorrect (where possible) and (2)

the ability to develop plans, and to respond at the right time

with optimal combination of responses, i.e., the use of strategy.

The time-event matrix was designed to measure the latter of the

two components. In many cases, the first component, i.e.,

appropriate content knowledge and understanding of the task

situation can be assumed, as long as sufficient training and

experience is available. Nonetheless, specific quality performance measures are collected as well, even though they may not be

directly captured by the time-event matrix procedure.

Time-event matrices can be used to measure a variety of

task performance activities, depending on the interests and

orientations of the researcher or observer. This paper cannot

cover all of the purposes for which the matrix can or has been

employed. For greater ease of communication, let us focus on

decision-making matrices as an example for all matrix possibilities. It should be remembered, however, that most other performance areas, aside from decision-making, could have been selected

equally well.

The time-event matrix technique was initially developed

to measure the interrelationships among actions across time and

the effects of information flow which precedes those actions.

Details about the construction of time-event matrices will be

discussed below. At this point it is merely important to be

t.-.

k ;"'' 'g'' -. . , .:",: -".. '- ,"" "--' ","--.' -'-" . " " -' "" ,,- .' --. . 7.""
IO 4

aware that these matrices capture all data about incoming

information, about decisions and other actions based on that

information, about interrelationships among information and

decisions as well as interrelationships among decisions (e.g.,

strategy). The matrices may be used to collect data on measures

which reflect how task oriented individuals or groups process

information and how that information processing determines or

affects observed performance. Measures based on the time-event

matrices may be considered "intermediate" assessments of performance quality. They provide a necessary vehicle for estimating

and defining important action antecedents of performance (criter- %

ion based) quality, particularly in complex tasks. Their

predictive validity for success (e.g., managerial excellence) may W

be accessed in research designs that obtain the predictive

validity would hamper each measure (to be reported subsequently).

Tasks and their requirements differ. The same strategy is

not necessarily useful in all task settings (aside from differences in knowledge content). As a result, the measures derived

from the time-event matrix should be carefully validated against

each general performance task. Many tasks will, of course,

produce quite similar patterns of "optimal" measurement levels

to criterion. As a matter of fact, a number of validations have

shown that specific score levels for the measures that will be

presented later tend to be quite robust across a number of tasks

and a number of performance environments. Where adequate

training and/or sufficient experience is likely to result in few

- -. - - - - - - - - - - -
(if any) content errors in performance, predictions of quality

task performance made on the basis of matrix measurement is of

substantial value. Where training, task familiarity, knowledge

and experience with task requirements is minimal, the scores LU

would provide information about the capacity to acquire excellence after training.

The dimensions of a matrix are time and action (here

decision) types. Each will be discussed in turn:

(1) Time

Time in the matrix is plotted horizontally. There are no A

particular restrictions on the gradations to be used (no matter

whether time proceeds normally or is - as in some simulations -

expanded or condensed), except that events which occur sequentially and independently of each other must appear on different

time points. The time dimension moves from the left to the

right. The units of the scale used are not of significance,

except that decision-making sequences which are to be compared

must contain the same scale units (since the formulas should

calculate comparable values).

(2) Decision Categories

Decision-making tasks and settings differ. Consequently

decisions employed differ as well. For example, executives

dealing with the potential purchase of another corporation may

be concerned with such action areas as establishing the value of

the other company, determining potential duplication of effort,

etc. On the other hand, military decision makers may be

I4

. . . . . . . . . . . . .-..... ,
12

concerned with troop movements, air support decisions and so

forth. In other words, groupings of decisions (decision categories) must be established separately for each general group of

decision-making situations. Selection of predefined decision

categories is best accomplished by experts in the field. The

selected categories should be inclusive, where possible of

approximately equal breadth, and conceptually meaningful and

.4.

consistent. They should clearly differ from each other in

activity, method, meaning, etc. Decision categories should

provide the potential for use by decision makers. While some

decision makers would likely use one group of decisions, others

may focus on a different group, of course with considerable

overlap.

While there is no restriction on the number of potential

decision categories that might be represented in a time-event

matrix, the inclusiveness of decision categories should be

selected so that decision makers utilize, on the average,

somewhere between ten and fifty different categories of

decisions1 in any time sequence that lasts for several hours.

Note, however, that these suggestions are ideal requirements and

do not supercede the practical characteristics of any particular K

task situation. For example, if a decision task requires only

one kind of decision, one cannot "manufacture" other decision

categories by hook or crook. In effect, the use of the decision

'Since decision makers would rarely employ all available

decision categories, the potential for considerably more than 50

categories may be provided.

• ° ° ° ••-•' ° = -,-° ',-,-. % " °.•°• o . - ° o . . " % % °% . • - . .,.•% "° 7 
matrix in such simple situations would have little value. For

example, if all actions reflected troop movements, then splitting

decisions by the unit moved may not be useful.

(3) Decision Points

Once time is plotted horizontally and decision categories

(as selected, for example, bv an expert panel) are plotted

vertically, each decision made by an individual or a group of

decision makers (as desired by the researcher or observer) can

be represented by a point placed vertically beneath the time

when that decision was made (or announced, or transmitted, again

depending on the intent of the researcher or observer) and

horizontally next to the decision category represented. All

decisions can thus be placed in the matrix.

(4) Information Input

Information input that is received during the simulation (a

potential carrier for independent variable manipulations), is

considered as it relates to decision output (this limitation was

chosen for convenience and is not necessary). Any unit of input

which leads to an output is marked (e.g., by a *) under its

appropriate (input) time and in front of (on the same

decision-type line as) any decision made as a consequence of that

input. The input is placed in advance of each output which it

produced, i.e., the same input may occur on more than one

horizontal (decision-category) line. The distance on the

horizontal between the input * and the decision point is marked

with a dashed line. It reflects the time elapsed between receipt ]

.............- ....'.'......'.-.-..-..................- "..............
14

of information and relevant response.2

(5) Diagonals

As stated above, we are interested in the relationships

among decisions as they reflect, for example, the development of

plans or strategies. Consequently we wish to know whether a

decision made at one time is related (leads) to a decision at a

later time. Where a decision in one category is made to make a

later decision representing another category possible, the two

decisions are connected across time with a diagonal line. An

arrow-head points forward toward the later decisions. 3 If two

decisions show an isolated relationship to each other, a single

arrow is drawn. If, on the other hand, the decision maker(s)

decide(s) to engage in decision types A and B at time one to

allow for action C in the future, and wants to accomplish C to

allow D to occur even later, and if all these decisions are

actually made in time, a longer chain of diagonal connections

is established:

2Su e the section on Integration Time Weight (below) for a discussion of time measurement. ...

II

3Such diagonal connections in the matrix will later be"..

referred to as "integrations.""i "
* -a'

Z4

,qq 526

.4.

15

TIME

122

312

333

S415 *--

R 526

z

FIGURE 1. Developing a Time-Event Matrix

Number, length and interconnectedness of forward diagonals will

be of importance for several of the measures that will be

discussed later. Diagonals are sometimes drawn with arrowheads

facing backwards. If, for example, a decision maker or a

decision-making group engages in action A without considering a

future action, but later finds that action A is of use when a

later action is decided upon, a backward arrow diagonal between

the later action and the previous action may be drawn. As a

rule, interconnectedness among backward diagonals does not

occur with great frequency.

e-- e- °-

...............................

a....
16

(6) End Effects

Whether or not a diagonal is drawn depends, of course, on

whether a planned later decision is indeed produced as a followup to an earlier decision. Where a decision task ends abruptly,

the opportunity to carry out planned subsequent decisions may

not exist. Ending a simulation could arbitrarily limit the

number of diagonals produced by decision makers as reflected in

obtained measures (see below). Randomization experimental

events mav, in some cases, be employed to avoid a constant

error. Alternatively, the probabilities of scores that should

have been obtained during latter time periods in the simulation

may be calculated by comparing earlier and late performance

characteristics. This latter method is especially useful if a

fixed order of simulated events is needed for purposes of data

collection.

(7) Establishing Relationships in the Matrix

It is important to establish clear relationships (a)

between inputs and subsequent output decisions, and (b) among -

decisions which are causally or strategically related (represented by diagonals). Of course, the only perfect representation

of these interrelationships exists in the brain of the decision

maker(s) at the moment relevant decisions are made. Any measure

of those relationships can, consequently, be subject to some V-S

error. Clearly, it is important to opt for the least amount

of error in any experimental or observational setting. Certainly, the error levels of obtained data would likely be much

• .1

. ., - " ... ' .- . - j - -. .' - -.. '.. "." . , j . - . j- '. -. - . ." "- ". . - .' . -. '. -. . ". -. . -• - - -."

- k. - . -. . , ik. i. --..-- - ........................................................-............---.. ¢ v -.. -. .-'.-..-..-
smaller in a well designed complex experimental simulation than

in research based on observation in an ongoing free environment.

For example, in experimental simulations, records of planning

can be obtained directly from participants during the planning

process. In real-world task environments, less precise techniques such as post-hoc interviews are required.

Ideally, decision maker(s) should be asked immediately

(upon making a decision) to indicate (a) any information received

upon which the decision is based, and (b) any planned subsequent '.- '

decisions that they might employ as a follow-up to a current

decision. That kind of data can be obtained in complex experimental simulations (the participants may have to be persuaded, ..

however, that indicating previously received information and

indicating planned future decisions would be of value to them in

terms of long range outcome). In many free simulations (particularlv if interrupt control is lacking) or in the observation of

real-world decision-making environments these kinds of questions

cannot be asked. Collecting those data after completion of their

task often introduces serious bias. One might require experts to

consider all decisions that were made and to judge whether these

decisions represent responses to previous information and/or

were part of decision-making sequences that should be represented

by diagonal interconnections. Hopefully, interjudge reliability for such a task would be high. Previous experience has shown

that judges produce little variable error in making these

judgments. As long as the judges have no particular biases for

............ .- . .- .C ,2 -. I
or against certain decision makers they are evaluating, constant

,.

errors across various samples may not be excessive. Nonetheless, "-

the immediate data collection method that can be employed in

experimental simulations has considerable advantages over other

procedures.

Establishing connections between inputs and decisions on

the basis of expert judgments is relatively easy. Respondent

decisions are typically directly related to the verbal content

of input information or describe the same location or information

source contained in the input. When such commonalities are

seen, a connection may be assumed to exist. More difficult is

the interpretation of connections among decisions. Obviously,

where one decision directly refers to a previous decision

("Order tne unit which we moved to quadrant X5 to fire on...") a

diagonal connection is directly established. However, is this a

forward or a backward diagonal? If we were able to ask the

decision maker(s) about future decisions, when the original

decision to move the unit was made, then we would now know. If

we were not able to ask (in free simulations of the type described above or in real-world applied decision-making settings),

then we cannot be certain. In these cases forward and backward

diagonals cannot be distinguished and arrowheads cannot be

drawn.

Whenever no clear relationship is stated by the decision

maker(s), aids must be used to determine whether relationships

may exist. Such commonalities among decisions as addressee,

" " "." . " ."."." .. ., " .' .' .... r -. ,... .o .. " .. -. ..- - .. . . ... -. . .. .. .. . . ... - . . .... .. . .
location, action etc. can be useful for that purpose. The most

reliable indicator of interrelationship is probably location.

In a military setting, to give a relatively simple example,

moving artillery to quadrant X5, asking it to fire on Y4, moving

infantry to Y5 and finally ordering the infantry to attempt to

take Y4 would reflect a minimal series of interrelated decisions

across time.

It should be noted here that moving troops to Y5 and

another troop unit (both infantry) to X5 (at a later time) would

not result in a diagonal connection: both actions are included

within the same decision category: repetitious action is not

necessarily representative of strategic action. If, on the

other hand, both units are later asked to attack Y4, separate

diagonal connections between each of the two movements and the

later attack would be drawn.

Some decision sequences may be difficult to judge in terms

of their potential interconnectedness. To the degree to which

the judge can develop a picture of the strategy decision makers

used (or if the judge can obtain advance information about their

plans), the determination of strategic relationships will be

considerably easier. In any case, if, after considerable

thought, a judge is uncertain whether two decisions are or are

not related to each other, it is better to err by omission.

Uncertain relationships (interconnections) should not be scored

to avoid artificially inflating some of the measures (below)

which can show near quadratic effects of erroneously scored

-'f - "L . . "L ' .' L ' L.£ :.""."/ ;'" 2' " ", ','-- ,. -' ,".-.--'-'-. . .''.'.- "" .'.': "-"---"-"-°'. .".-4"-'
- iI

* 20

relationships. Of course, the use of direct questioning that is

possible in experimental and quasi-experimental simulation

techniques would avoid such quandary altogether.

An example of two decision matrices is provided on page 21.

The figure shows decision matrices generated by two groups of

participants which differed in their decision-making styles

(complexity).

MEASURES .

A considerable number of measures are presented and explained in this report. Additional measures can be developed if they

are useful for any specific task at hand. Calculation of the

measures assumes either that a time-event matrix has been drawn

(by computer, if obtained from an experimental simulation, or by

judges, if obtained from a free simulation or a real-world 7Ndecision-making environment) or that computer generated vectors

for task performance events are availabli--. The various measures

were developed to reflect different kinds of task performance.

In-and-of-themselves, each measure cannot be considered a

reflection of "good" versus "bad" performancc with regard to any

particular criterion without considering momentary demands (e.g.,

environmental conditions). Without question, there are situI ations where complex strategic planning (as, for example,

reflected in the QIS and WQIS measures, below) is of considerable

value. On the other hand, there are situations where such "I'

planning would be superfluous and inappropriate since task

demands may require immediate (e.g., respondent, see below)

= % • . .f • t - °.- . . . . . . . .
* F- - - ..

TIME 21

-L -0'a~.~*

~. A W

4:-* .

........

-- -- --- -

PONT INTIE

EACH DIGNLRPEET ESRTGCITERTO FDFEETDCSOSA

DEISO INEVL

EACH POISNT REPRESENTS P A LFS ELTE EI IGUE FIFRNIAE 

EACH EGRCAS LNCONNECTS D BL ECSOSMDEA TESMEPIN NIE
22j

actions. Measures for both kinds of these activities (and man%

relevant others) are included. Each measure and its purpose will

be discussed below.4 Where necessary, calculation examples will

be provided.

For convenience of communication, we will again focus on

time-event matrices involving decision making. The measures

will be presented in the order in which they are printed out by

the computer program, i.e., the order in which they were developed. Note, that similar measures may not be located adjacently

to each other.

1. Number of Decisions

This measure reflects the amount of decision making activitv. It consists of a count of the number of decisions made,

i.e., the number of points in the matrix during any period of

time that is of interest. The formula for this measure may be

written as

p

where I through p indicates the time period(s) of interest and d

is a decision made within that time.

2. Number of Respondent Decisions

This measure indicates the number of information items that

4 Some measures, as the reader will note, are based on

factorial combinations of specific values. To reduce artifactual

spread in the distribution of higher values (increased artificial

distances from regression lines) some measures are calculated via

natural logarithm transformations.

..... . ....-....-. ,....... . ..-...-.. .- .. ... .-...... .. . ... . _... .. . . . . . . . . . . . . . . . . . .
were utilized in responses to environmental information. Since

any one respondent decision may be responsive to multiple

information, the score for this measure may exceed the score for

number of decisions.

P

Ln r i

where

r i is any response to information within a given time

period (see below) after receipt of relevant information. A

response (respondent decision) in this measure is counted

more than once if it responds to two or more items of

relevant information that was received previously.

Whether a decision responds to previously received information should ideally (as discussed earlier) be determined by a

verbal indicator from the decision maker(s). That is the case

in all experimental or quasi experimental simulations. For other

research settings, it may have to be determined by competent

judges as long as appropriate access to the decision maker(s)

for questioning is not available.

Different decision-making situations require diverse time

frames for the processing and accessing/communicating of information and development of subsequent decisions. Respondent

decisions (as defined here, see some potential modifications

below) are most often made quickly after receipt of incoming

* information. The decision output is usually not extensively

% I _%A 1.2
W!k 1. Wn .. VA^%

24

pondered and is rarely considered in terms of existing or

emerging plans (strategy). For example, a respondent decision

to the intrusion of enemy aircraft into friendly airspace may

involve immediate defensive action. Certainly the reasons for

that intrusion may be considered subsequently and may be reflected in future activities that may or may not be strategic (reflect

planning). Nonetheless the initial action, occurring as immediately as possible, represents (often quite appropriate) one-toone responding to information. In situations where any meaningful response to information must occur quickly, a time restriction for scoring actions as responsive might be introduced. The

time limitation between receipt of information and a response

which determines whether a decision follows receipt of that

information with sufficient rapidity to qualify as a respondent

decision must depend on the constraints of the decision-making

situation. In other words, that time frame must be determined

individually for each group of decision-making settings (or

scenarios) that are of interest. However, that time frame

cannot subsequently be changed from one person or group to the

next, if meaningful comparisons are to be made.

It should be noted that two variants of the respondent

decision-making measure have been used with success. One is

"retaliatory decision making" (cf. Measure 8). In this measure,

responses to information receipt are not included in the value of

"r" if the relevant decision is part of an integrated decision

sequence (i.e., is counted in the Number of Integrations

.7-

... -.-- .. -...,-i '-2... ."; ,- -?..--2 i-i,.--.- -.', . .-i -. ,i .2-..-i-2-.- -2 i. :. -.- : -- : --... ,-z: ---- '.. ,
25

measure). The assessment of retaliatory deci'sion making provides

an estimate of non-strategic respondent behavior. A second

modification of the respondent decision-making measure eliminates

any time constraints for the information-decision sequence. Here

all decisions made in response to information (no matter how much

delayed) are counted. The resulting score reflects the total

amount of respondent activity. Note, however, that these

modifications of the respondent decision-making measure are not

statistically independent (orthogonal) of each other. Nonetheless they can be quite useful for specific research or observation intents.

3. Number of Decision Categories

This measure is a simple count of the number of decision

categories decision makers use during any specific time period.

It may be conceived as an approximate estimate of the degree of

differentiation in decision making. Any category which is part

of the count may have been used once or more than once. The

measure reflects whether the decision maker(s) would be likely

to select smaller or larger numbers of action types. In addition, further analysis could reveal whether decision maker(s)

are likely to select certain specific actions and eliminate

others from consideration. The basic measure may be written as

PV

1

where C is the number of categories employed and I through p is

OL
26

the time period of participation in the simulation that is of

interest for analysis and interpretation.

4. (and 7). Number of Integrations (Forward, Backward or Total)

The score for Number of Integrations reflects the frequency of the application of single steps of strategy.

p p p p

if or ib) or (ib+if)

,1 1 1 1•

where Lj

if are forward integrations (relationships, i.e., connections among decision-making points with diagonal

arrows pointing forward), (Measure 4),

ib are backward integrations (relationships, i.e., connections among decision-making points with diagonal arrows

pointing backward), (Measure 7),

i are integrations, i.e., relationships where directionality cannot be established, (Measure, 4, modified).

As discussed earlier, some decision-making tasks (particularly real-world decision-making settings where the researcher

or observer cannot interfere) may not lend themselves to questioning the decision maker(s) about the intent of decisions.

Consequently, it may be impossible to determine whether a

connection (relationship) among decisions reflects forward

integrations (planning a later decision at the time an earlier

decision has been made), or reflects backward integration,

(using a previous decision to advantage although the connection

IG,

* .- .
was not considered at the time the earlier decision did occur).

However, whenever possible, forward diagonals in the matrix

should be counted as forward integrations and backward diagonals

should be counted as backward integrations. Translation of

diagonals into integration scores is achieved on a one-to-one

basis: Counting the number of diagonals of a specific type

produces the relevant integration score. Where no distinction

between forward and backward diagonals can be made, integrations

are counted without concern for the direction of arrowheads.

Example

For simplicity's sake, let us return to the example matrix

in Figure 2, page 21. The upper matrix contains two forward '.1

diagonals, i.e., a score of 2 for if (forward integrations). It

contains three backward diagonals, i.e., a score of 3 for ib

(backward integrations). The score for i (if + ib) would be 5.

Obviously the score for the lower matrix in Figure 2 is considerably higher.

The measures concerned with number of integrations, integration time weight and QIS depend on the diagonals connecting

earlier and subsequent decisions, indicating that an earlier

decision made a later decision possible (strategic time

sequence). Where an entire matrix is analyzed, simple counting

or statistical processing of the number of diagonals is sufficient. However, if an experimenter or observer is concerned .1

with a limited time period as part of a larger decision time

sequence (e.g., if different experimental conditions are

,' - -, - -. .° .. - .-,. .. .- . - . . . . . . . .. . . a . . .-- . ::- 
2

introduced into an experimental simulation or if artificial or

natural probes are utilized in a free simulation), diagonals

will often cross the time lines that describe a period of

interest. In that case, diagonals are scored for the time

period during which they originate. If distinctions between

backward and forward integrations (diagonals with backward and

forward arrows) can be made, then backward integrations will be

credited to the period of the subsequent (of two) decisions.

Forward integrations will be credited to the period of the

initial (of two interconnected) decisions. If no distinctions

between forward and backward diagonals can be made, all diagonals

are credited to the time period of the initial decision. If

specific experimental manipulations are utilized in experimental

or quasi-experimental simulations, the manipulations (if manipulated "within" across time) should, where possible, be randomized

in order to replace constant error with variable error. Adjustments to reduce error produced by the end of a measurement

period are discussed below.

5. Multiplexity F

The multiplexitv F measure is concerned with forward

planning only. It considers the QIS value (see below) that

would be calculated if only diagonals in the Time-Event Matrix

that connect two points ahead in time are considered. As a

" result, the measure focuses on the degree of strategy that is

presently applied and ignores strategic components that have

already been completed. It should be noted, that Multiplexity F

. .. . . . .. . . . . .. . . . .*
29

is especially sensitive to end effects of data collection or

observation. The formula for Multiplexitv F is similar to the

formula for QIS (with earlier integrations omitted) and may be

written as

P

Z W (1+nf)

i

For an explanation of terms, see Measures 6 and 9.

A related measure that is not currently printed by the

simulation program but is related to Multiplexity F is called L

Multiplicity. The Multiplicity measure is also closely akin to

the QIS measure (Measure 9). It does, however, not take time

between decisions in an integration sequence (See measure 6,

below) into account. While Multiplicity is not orthogonal to

QIS, it is considered to be supplemental. This measure is

potentially more meaningful than QIS in situations where responding (including strategic integrated responding) must occur quite

rapidly or where the time delay between an original and a later

decision in strategic sequences is more a function of task

demand than of planning or task performance. The formula for

multiplicity can be derived directly from the QIS formula

(Measure 9) by removing the term for Weight (Measure 6):

p

(1 + np + nf) ',

For an explanation of terms, see Measure 9.

1%

t' i . . . " c D. _. ". ,. '. . .. .,... . . . .... . .. .. ' ..' ' . -"; . .., .'. . .. ..' " .... - ." .; .... .... . ' ..- ' .[ " .-.... . .; -.' l -i . i
3U

6. Integration Time Weight

The Time Weight measure calculates the length of time for

which plans and/or strategies are developed.

p

Ln Z W

Where the measure for number of integrations was merely

concerned with the frequency of interconnections (strategic

relationships) among decisions, the time weight measure considers

the length of time involved in future planning. The me sure

utilizes the same individual integrations discussed for the

integration measure (diagonals) but measures each diagonal on

the time dimension (in fixed units chosen by the experimenter or

observer, see above) and replaces the value of 1 (for the

occurrence of the diagonal) with the value of length of time

which the diagonal spans between decisions. Consider, for

example, the matrix example in Figure 3, below:

7 .

.

w ,'.%

A %° 

I£ ' ..-' Z C. b_,' ,.-.-..2' - ""r-'-'-,-,-. ,..- ' '.... ..... ,,. ... . . . .. . .• ... .. .. .- ..
-a V. Lwwsw j w~. W W1 W) j WJw WV j W w W.. j .. . .. WVV W~ 7Wj W -4 z

3 1

1 2 3 4 5 6 7 8 9 10 11 12 TI ME

171

203

w 221

R 223 0

310 ,.

451

461 S

488

639 FIGURE 3

l,°, *,'.

The time weight for the forward integrations (diagonal

connections) between the initial decisions 203 and 488 which are

connected to decision 221 represents two time units each (remember that time units are selected by the experimenter/observer but

must be held constant if numerical comparisons among scores for

different decision makers are to be made). The connection

between decisions 221 and 223 represents four time units. The

connection between 223 and 221 represents one time unit, and

finally the connection between 461 and 223 represents two time

units. The total score for integration time weight in this

matrix is then Ln (2 + 2 + 4 + 1 + 2) - the natural log of 11.

7. Backward Integrations

The rationale for scoring backward integrations and a

formula for that measure have already been discussed (cf.

Measure 4) and need not be repeated at this point.

- °° •
32 '

.0.

8. Unintegrated Respondent Decisions (Retaliatory Decisions)

Decisions or other actions which occur in response to

information may be part of an ongoing strategy and may reflect

an understanding of the interrelationship between events,

actions and anticipated outcomes. Alternatively, respondent

actions may be retaliatory rather than strategic, i.e., they may

be one-to-one responses to incoming information that do not

consider the overall impact of information on plans, strategies

and so forth. The latter actions are considered in the measurement of unintegrated respondent decision making. Returning to

the Time-Event Matrix, all respondent decisions that are not

interconnected with other decisions by diagonals would be

included in this measure. A formula for this measure may be

written as

p .

ru

where

ru are all decisions that are made in response to incoming

information which are not part of an integrated

sequence.

9. QIS (Quality of Integrated Strategies)

This measure provides an indication of the complexity of

sequential strategic efforts. Similar data for special applica- ,.

tions are represented by the Multiplexity of Integration and the

Weighted QIS (WQIS) measures.

i% %.o-
p 33

Ln ZO W~+ n+ nf

Where W represents the length of the time dimensions for any

forward integration (or any integration, if distinctions between forward and backward integrations cannot

be made). W is the last measure discussed above

(integration time weight)

n is the number of other forward integrations (or any

integration, if distinctions between forward and

backward integrations cannot be made) connecting to

the decision point representing the initial decision

in a diagonal connection between two decisions and

nf is the number of forward integrations (or any integrations, if distinctions between forward and backward

integrations cannot be made) connecting to the decision

point representing the subsequent decision in a

diagonal connection between two decisions.

The number of integrations np and nf here include only those

integrations which are directly connected to either the initial

(np) or subsequent (nf) decision points.

The QIS measure assesses the degree to which planning

(strategic behavior) follows an overall pattern versus a number

of separate unrelated plans. While the score for number of

integrations may, for example, be the same in either case, an

overall plan connecting all components of a decision-making

sequence in a combined strategy would result in a higher QIS

score; separate strategic plans would result in lower QIS scores.

QIS scores tend to distinguish between excellent and mediocre

-7
34

decision-making quality where decision makers must operate at

advanced decision-making levels. QIS are identical to Integration Time Weight scores where all integrations remain independent

of each other, i.e., where an overall strategic plan does not

exist.

Example

Let us again return to Figure 3, page 31. A QIS value

would be established for each diagonal in the matrix. Let us

initially take the diagonal which is connecting 203 and 221. We

already concluded that its not yet transformed weight (W) score

is 2 There are no diagonals connecting to its beginning point.

On the other hand, there are two diagonals connecting directly

to its end point. The score would be

2(1 + 0 + 2) 6

The same value of 6 would also be obtained for the 488 to 221

diagonal. The 221 to 223 diagonal with a W value of 4 connects

to two other diagonals at its beginning point and one other

diagonal at its end point. Its score would be

4(1 + 2 + 1) = 16

In turn the 223 to 221 connection would be

1(1 + I + 0) = 2

Finally, the 461 to 223 diagonal maintains its W value since

there are no diagonals connected to either the initial nor the

subsequent decisions: ":

2(1 + 0 + 0) = 2 NFor this matrix the total QIS score then would be

rI
Ln (6 + 6 + 16 + 2 + 2) the natural log of 3"2

10. Weighted QIS

Weighted Quality of Integrated Strategies (WQIS) is an

extension of the QIS measure to obtain scores for the sequential

chain of interconnections among integrated decisions over long

periods of time (i.e., multiple long-term strategic actions that

are coordinated). Where the QIS formula calculated the time

weight for an integration (diagonal connection between decision

points differing in time) and multiplied that weight value with

the number of other diagonals connected directly to the beginning

point (initial decision) and the end point (later integrated

decisions) of an integration, the WQIS measure considers all

integrations (diagonals) which lead in chain sequences to the

decision which begins the integration, as well as all integrations (diagonals) which follow the later integrated decision, as

long as there is no interruption in diagonal (integration)

links. Because of the multiplicative nature of this measure,

quite high scores can be obtained whenever additional links are

added in any strategic chain of decisions. Where no more than

three decision points (differing in time) are connected with

diagonals (integrations), the WOIS measure will not differ from

the QIS measure. Where four decision points (three sequential

diagonals) are involved, the measure hill not differ for the

middle integration, but will differ fcr the outer two integration

diagonals. With even greater number- cf diagonal connections

in chain sequence, the score for W01- would considerably exceed

4 V€
36

the QIS score. The formula for WQIS can be written as

p

Ln W(1 + npp + nff)

where M

npp is the number of forward integrations reflected in the

term np for the QIS measure plus all other forward

integrations connecting to these integrations, and so

forth, until all integrations (diagonals in the

matrix) which connect to each other and can be traced

without interruption to the beginning point of the

forward integration of interest have been exhausted.

nff is the number of forward integrations reflected in the

term nf for the QIS measure plus all other forward

integrations connecting to these integrations, and so

forth, until all integrations (diagonals in the

matrix) which connect to each other and can be traced

without interruption to the later decision have been

exhausted.

All other terms are the same used in previous formulas.

For the example in Figure 3 on page 31 the WQIS score would

be

2(1+0+3) + 2(1+0+3) + 4(1+2+1) + 1(1+3+0) + 2(1+0+0) = the

natural log of 38

11. Average Response Speed

The measure reflects the typical time delay between receipt

p-

.:-.-.s<--.---- -*,-
3-

of information and decision responses to that information.

p

Ln tr

rp

where

tr is the elapsed time between information receipt and

any subsequent respondent decision, and

r is the number of respondent decisions made in the time

period between 1 and p.

The response speed measure simply reflects the rapidity

with which decision maker(s) respond to incoming information.

The time length between each input and the subsequent decision

is measured; the sum of those measures is divided by the number

of observed responses to information. For this measure it is V.

worthwhile to consider a value for r (number of respondent

decisions) which is not constrained by a time limitation between

information receipt and subsequent decision.

If no responses to information should occur (a rather

unlikelv event) the score for Measure 11 must be set to a value

that is higher than all other scores in a sample.

12. Serial Connections

The serial connection measure is similar to the Number of

Integrations measure but counts interconnections between decisions that are placed into the same decision category. For

example, if decision makers decide to move troop unit A and plan
38

to subsequently move troop unit B. a forward serial connection is %

established: Both decisions fall into a single decision

category: troop movement. They are, by themselves, not likely

to reflect an ongoing strategy unless they are also interconnected with other decisions located in different categories (to

which they would be connected with diagonals in the matrix).

Serial connections without integrations often reflect a stagnating series of moves that frequently fail to take the complexities of a task or environment into account. If associated with

strategic moves (as reflected in high scores on such measures as

Number of Integrations or QIS), they may be part of a general

(e.g., in the military, an encircling) strategy.

Serial connections may be measured (as were number of

integrations) in terms of forward, backward or general connections between decisions of a single category:

p p p p

if or Zisb or (isf + isb) = Z s

1 1 1 1

where

isf are forward serial connections, and isb are backward

serial connections.

13. Planned Integrations

Not all actions, here decisions, which are planned as

sequels to current actions will actually be carried out. Time

demands, changed situations, forgetfulness, new strategies and

more may be the cause of lacking follow-up actions. Where

A
39

multiple strategies toward goals were followed, planned actions

that are part of strategy may be dropped as unnecessary when

some other parallel strategy is about to reach its successful

conclusion. In other cases, however, an incomplete connection

between a current action and a planned future action may indicate

poor implementation of strategy. The Planned Integrations

measure reflects the number of times decision makers fail to

carry out subsequent future actions that had been previously

planned. The formula for planned integrations can be written as

___ipf

where

ipf is a planned forward integration which was not carried

out in the future.

Planned integrations that did not come to fruition may be

compared with the number of integrations which were completed to

obtain an estimate of the degree to which decision makers do

operationalize their plans. This score would be reflected by

the formula

p P ,a

if - pf

1 1 ¢

Finally, the planned integration measure may be utilized to

estimate the assumed time value for number of integrations where

that measure is truncated by the end of a measurement or

VI

................................
J.- - - .7777 .- 75 VC'7K qv - -1-

40

observation sequence (e.g., at final participation periods in

experimental simulations or at the retirement of an executive or -

officer prior to final completion of a task). Under these

conditions, it may not have been possible to complete all future

decisions which were planned when a previous action was initiated. As a result the uncorrected measure for number of

integrations would underestimate the strategic planning of the

decision maker. This correction may be calculated as:

p p

ipf

i f

where 1 through pp is any prior time period (or periods) to which

a time period under analysis is to be compared.

The obtained value of this correction is then multiplied

with the total number of intended integrations:

P p

iipf + if 1.

1

to obtain the estimated value of corrected Number of Integrations. Unless the corrected value is less than the actually

obtained score for Number of Integrations, the Number of Integrations score may be replaced by the corrected score. A measure
discussed later in this paper is specifically derived for

calculating estimated values on the basis of earlier simulation

performance (Measure 36). Similar calculations may be employed

to correct other measures which are based on forward or backward

integrations.

14. General Unintegrated Decisions

Decisions which are neither part of any integrated strategy

nor are made in response to information are defined as General

Unintegrated Decisions. In most cases, these decisions are not

particularly valuable unless they are part of a tentative probe

for preliminary information, e.g., in the form of "I wonder what

would happen if...": In a more or less defined task setting

with the potential for strategic goal directed actions, an

excess of General Unintegrated Decisions tends to reflect poor

performance. The formula for this measure may be written as

p

Zu

1

where u is an unintegrated respondent decision.

15. Spread across Decision Categories 5

This measure, as well as the next measure (Average Spread

Across Decision Categories), considers the degree to which

decision makers favor certain kinds of actions over other kinds.

5 This measure has been slightly modified compared to

previous applications.
42

p

/7 2(dCa dCb) + (dCd - dCe)

1

where, d is the number of decisions

dCa is the number of decisions from the category or

categories representing the upper ten percent of

decision frequency,

dCb is the number of decisions from the category or

categories representing the lowest ten percent of

decision frequency,

dCd is the number of decisions from the category or

categories representing the upper fifty percent of

decision frequency, and

dCe is the number of decisions from the category or

categories representing the lower fifty percent of

decision frequency.

A high value of this measure suggests that the decision maker(s),

while not necessarily totally ignoring other potential decision

categories, nonetheless concentrate major effort on a limited

number of activities. For example, an executive who selects

most decisions because they specifically relate to current

profit or a military commander who lets the infantry do nearly

all of the fighting without support by other units would score

high on this measure. A low score, on the other hand, would

reflect a more well rounded approach to more or less complex

tasks. The measure is not meaningful if only one decision
42

category is utilized.

Example

Let us assume that decision maker(s) made a total of fifty

decisions during a given period of time. These decisions

represented the following decision types: I-".

Decision Category Number of Decisions

111 5

122 10

312 20

333 3

415 8

526 1

527 3

Total 50

The upper and lower ten percent would represent a value of

5 each (10% of fifty) while the upper and lower 50% would

represent values of 25 each.

Decision category 312 is included in the upper 10%, providing a value of 20 (the number of decisions in that decision tvpe

categorv) for dCa.

Decision categories 333, 526 and 527 are included in the

lower ten percent providing a value of 7 for dCb.

Decision categories 122 and 312 are included in the upper

50%, providing a value of 30 for dCd.

Decision categories 122, 415, 111, 333, 526 and 527 are

included in the lower 50% providing a value of 30 for dCe.

',',,, ." ¢ ,'- '4""," _. " ,,-' .,,- 'o'- .". .'%,.' _ -._.,.--..r . ".-.''.., " '," "- " - .,.. ,, .- ,, "- .- .- -. 4 ,,, " " .- -." " ., - - ; ". ." .. - .- .- ". " - .- .. -. -. • - - - '. -
44

The resulting calculation for spread across decision

categories is then: 2(20 - 7) + (30 - 30) = 26.

16. Average Spread Across Decision Categories

The formula for Spread Across Decision Categories is to

some degree affected by the number of categories used (and/or

available). To correct for potential errors, particularly when

many decision categories are available and utilization differs

widely among individuals or groups, an additional measure mav be

introduced. This measure divides the score for spread across

categories by the number of categories utilized. The average

spread measure is not assumed (or demonstrated) to be orthogonal

from Spread Across Decision Categories. It may be calculated as

p

2(dCa dCb) + (dCd - dCe)

1P

Cr

For the example of decision types and decision frequencies

on page 43 the obtained numerical score would be 26 (the value

obtained for the Spread Across Decision Categories measure)

divided by the Number of Categories measure, here 7 = 3.7143.

17. Number of Decisions in Response

Where Measure 2 was concerned with the number of information

items which lead to responses, Measure 17 focuses only on the
.4.

number of responses themselves that were, at least in part,

generated by information. For example, Measure 2 counts each

time that a previous message is listed as one antecedent. For

example, if an executive makes three decisions and claims that

each of those decisions was based on three previous messages,

the resulting score for Measure 2 would have been 9. In contrast, Measure 17 counts only the decisions themselves if they

are claimed to be in response to previous information, irrelevant

of the number of messages to which those responses were made.

In other words, the same person's score on Measure 17 would

be 3. Measure 17 may be represented by the formula

p

r

where r equals the number of decisions that listed previous

messages as the basis of decision making. It differs from

Measure 2 in the term r which, in that case, was ri .

18. Most Recent Response Speed

Measure 11 had assessed the mean response speed during any

one particular period of interest. The score would be a reflection of both immediate and delayed responding to information.

However, some situations do not lend themselves to respondent

action until some critical event has occurred. Once that event

is observed, an immediate response might only then become

optimal. Measure 11 might, in some situations, confound the

assessment of the latter response with elapsed time between

d Rb
46

earlier, partially relevant information and the arrival of later

critical information. That time span is, however, under control

of the simulation system and is not affected by any actions of

participants. Consequently, it seemed useful to develop a

measure that would assess the rapacity of response to the last

relevant message received. Measure 18 was designed for that

purpose:

Z

1

rp

where

trr is the time elapsed between receipt of the most

recent relevant information and the subsequent respondent decision and

rp is (as it was in Measure 11 ) the number of respondent

decisions during the time between 1 and p. 1*

If no respondent decisions occur, the score for Measure 18

should be set to a value higher than all other scores obtained

from the sample.

19. Total Integrative Activity (TIA)

Total Integrative Activity is a measure of strategic I'.

effort. It includes a count of attempted integrations that did

not reach fruition and completed integrations. The latter

category includes forward integrations that are based on planning
- * - -. - - 777 - 7 7 -. -. -

47

as well as backward integrations that derive from retrospective

pragmatic insight. TIA scores are calculated as the simple sum

of Measure 4 (Forward Integrations), Measure 7 (Backward Integrations), Measure 12 (Serial Connections) and Measure 13 (Planned

Integrations). Persons scoring high on Measure 4 are most

likely to score high on this measure. However, many persons

scoring high on TIA fail to score high on Measures 4 or Measure

12.

20. Total Forward Integrative Activity (TFIA)

Total Forward Integrative Activity is similar to Measure

19, but considers only planned integrative actions. The measure

is calculated as a simple sum of Measures 4 and 13.

21. Integrations Across Categories

Earlier, the method of assigning three digit numbers to

decision categories was discussed. The reader will remember

that the first digit represents a general characteristic (e.g.,

economic, military, etc.). Measure 21 was designed to assess

the degree to which integrations (strategic actions), as discussed in Measure 4, extend beyond the first digit level of decision

categories. For example, if one economic decision was

strategically integrated with another economic decision, the

score for that integration on Measure 21 would be zero. However,

a score of one would be generated if the economic decision was

integrated with a military decision, and so forth. In other

words, Measure 21 estimates the degree to which strategic

(integrative) planning is more generalized or more limited in its

.,r
. . .. . "

45"

range. The formula for Measure 21 has some similarity to the

formula for Measure 4:

p

II fc

with the term if of Measure 4 replaced by the term ifc

ifc is a forward integration (relationship, i.e., connection between decision making points in the time-event

matrix with arrowheads pointing forward) if this

connection interrelates decisions from categories with

a different first numerical digit.

22. Integrations Within Categories

This Measure calculates the frequency of integrative

interrelationships that interconnect decision points that are

located within the same first digit decision category. While a p.

high score in this category does not suggest a lesser capacity

for integrative strategic thought than would be implied by a

high score in Measure 21, a disproportionately high score in

Measure 22 compared to Measure 21, may represent a lower level

of strategic thinking. Where such a lower level is either . -

inappropriate or less effective, considering the task at hand a

I diminished proportion of scores in Measure 21 over Measure 22

ft" would imply lower levels of performance. Measure 22 can be

Icalculated as

p '
1. VT.- - -W I- MU T-L 

P9

pp

f "w. IwC. .--.-

where

ifw are forward integrations that interconnect decisions

located within the same first digit decision category.

Alternately, Measure 22 may be calculated by subtracting

Measure 21 from Measure 4.

p P P --.

i - fc

23. Proportion of Category Integrations

As already suggested, the proportion of Measure 21 divided

by Measure 22 may, if tasks are fluid, multifaceted and complex,

reflect the overall quality of integrative activity. Measure 23

assesses that quality. It is calculated as

1 + Measure 21

1 + Measure 22

24. Shift 1

This measure assesses the capacity of decision maker(s) to

shift their emphasis from an integrative strategic mode to a

more respondent mode when task demands require. For the ..

simulations employed by the present research team, an emergency ",,

is introduced at the beginning of the fourth playing period in

the simulation (approximately after four hours of participation).

That emergency is best handled by immediate respondent actions v',

............. ,, • ° - . ° - .%
50

and without further strategic planning (for the time being).

The calculation procedure for the Shift I measure would differ,

depending on scenario characteristics. In our simulations, no

emergencies occur throughout Periods 1, 2, 3 and 6. Periods 4

and 5 present a disaster situation which requires respondent

activity. In the case of our simulation design, the formula for

Shift I would be:

X Measure 4 (for Periods 1,2,3) X Measure 4 (Periods 4,5)

divided by "__

3 2

A high score would indicate the presence of strategic integrative

activity before the disaster event and a shift to little integrative activity throughout the disaster.

A number of potential problems may emerge for this measure

(and some subsequent measures) that must be controlled bv the

introduction of default values. For example, the complete

absence of integrative actions during Periods 4 and 5 would

result in a division of 0 + 0 by 2, resulting in zero. The

subsequent division of the first part of the formula by zero

would not be meaningful. Similarly, some (especially the

second) part of this formula may generate a quotient between

zero and 1, generating a meaningless increase in the score for

this measure. To eliminate these problems, a minimum default

value of 1.0 is set for both the first and the second part of

the formula.

25. Shift 2

This measure corrects Measure 24 for the actual amount of

q . ". ', . . -j . " . . . " •. .- .. . .. , . - " .. - . . .. . • .. a .
51

activity a person has expended on respondent actions throughout

proportion of integrative activity and respondent activity as

that proportion shifts from time periods where strategic planning

should be a priority to time periods where respondent activity

should be a priority. For the simulation procedures employed in

this research (with planning appropriate in periods 1 through 3

and respondent actions most appropriate in periods 4 and 5) the

formula should be

X Measure 4 X Measure 4

(for periods 1,2,3) divided by (periods 4,5,' a

X Measure 17 X Measure 17

Similarly to other measures which are based on changes in S.

situational demands, the formula for Shift 2 must be modified

with changes in the introduction and duration of emergency

conditions. Default values must again be introduced if Measure 4

or Measure 17 values are equal to zero resulting in division

by zero or division by a fraction.

26. Recovery 1

If stress or emergency situations require a priority shift

from strategic to respondent activity, planning and strategy

must often re-emerge once immediate problems have been resolved.

However, some persons may be overwhelmed by the emergency stress

experience and may fail to recover, i.e., may fail to reinstitute

strategic and planning activities when appropriate. Those

persons would not perform adequately in task settings that

contain intermittent stress periods. To assure that assessment
... . A... . . ..-- - - .. ,, - -. v -j -r - ~ - ,-T w r7

52

of performance does not miss the potential detrimental impact of

stressor experience, two recovery measures were developed.

Recovery 1 compares the number of integrations after an emergency

has been resolved (period 6 in the present simulations) with the

number of integrations which were evident prior to the stressor

experience (before the emergency occurred). For the present

simulations, the score is calculated as follows: 6

Measure 4 (Period 6)

X Measure 4 (Periods 1, 2 and 3)

Where the Mean value of Measure 4 for periods I through 3 equals

zero, the score for this Measure is equal to the value of

Measure 4 in Period 6.

27. Number of Information Search Decisions

This measure simply counts the number of information search

activities in which decision makers engage during a time period

of interest:

p

5

Where

s represents a decision with the purpose of obtaining

additional information.

28. Number of Integrated Information Search Decisions

This measure is concerned with the purpose of information

search. Where information search was initiated for strategic

reasons, the search action will typically precede subsequent

-.;.-.-.; . . .., :.. _. ; .., ...., . .. .-., ,..,. . ., , , ,, .. . .. ,,, ,., . .... ,... ,,-..._. . - .. ::-. - ,:; .. ._ . _.:/.- I-. I::-
I'I

53

related actions, i.e., it will be integrated with those later

actions. In the time-event matrix, a forward arrow will appear

between the search decision and subsequent related decision(s).

The formula may be written as

p

ifris

Where

ifis are the number of forward integrations that originate

in information search decisions.

29. Disaster Response Speed

Measures 29 through 35 focus on the introduction of an

emergency that is an intricate part of our own simulation

procedure. Where emergencies are not introduced, Measures 29

through 35 are meaningless. The advent of an emergency in our

simulations is coded by a message with the number M300. The

specified time location of M300 within the simulation allows the

calculation of values for emergency responses. The first of

these, Disaster Response Speed, measures the elapsed time

between the advent of the emergency message M300 and the first

respondent decision to that message. The formula may be written

as follows:

tdl trl - tM300

Where

tdl is the obtained disaster response speed

trl is the time of the first respondent decision relevant
a"3 54

to the emergency message, and

tM300 is the time of arrival of Message 300 (the emergency

message) I

If no response to the emergency message M300 is made, the

score for Measure 29 must be set to a value that is higher than

all other scores in the sample.

30. Average Disaster Response Speed

Most decision makers respond to the emergency message with

a number of actions. The present measure calculates the mean

time distance between the arrival of the emergency message and

the various responses to that message:

txd X(tr - tM300)

Where

t~d is the obtained Mean Disaster Response Speed

tr is the time of any decision which responds to Message

300, and

tM300 is the time of arrival of the emergency message

(M300)

If no response to the emergency message M300 is made, the

score for Measure 29 must be set to a value that is higher than

all other scores in the sample.

31. Integrated Disaster Responses

This measure considers the degree to which the potential of

an emergency was included in previous strategic planning. If

this measure is to be meaningful, the simulation must be designed

so that emergency (or equivalent) problems are viewed as possible

NO

. . ... .

-.-
55

by participants and can be, to some extent, anticipated in their

potential characteristics. This is the case for the simulations

we have designed. The measurement for integrated disaster

response counts the number of forward diagonals that end (terminate) in any decisions that list the disaster message (M300) as a

•7

preceding relevant message:

ifrM300

Where

ifrM300 are all integrations that terminate in respondent

decisions based (entirely or in part) on Message 300.

32. Applied Disaster Strategy

While strategic actions in response to emergency situations

are not likely and often are not appropriate, some integrated

strategies might reasonably originate from responses to an

emergency event. The Applied Disaster Strategy measure was

designed to assess the number of strategic integrations (forward

diagonals in the matrix) that originate from respondent decisions

which list Message 300 as a relevant preceding message. The

formula may be written as:

i / ifriM300

Where

ifriM300 are all integrations that originate from respondent -eq

decisions that list the disaster message M300 as a

preceding message.

,=°- .,
. . .1 .

56

33. Number of Disaster Decisions

We are here concerned with the number of respondent actions

to the advent of an emergency. The measure simply counts the

number of respondent decisions (whether or not they are integrated) that list M300 as a relevant preceding message. The

formula may be written as follows:

dM 3 0 0

Where

dM300 is any decision made in response to Message 300.

34. Disaster Backward Integrations

Actions taken at an earlier time, while not previously

planned to deal with potential future emergencies, might nonetheless be useful once an emergency arises. Backward integrations

(i.e., pragmatic post-hoc application of strategy) that are

applied to actions which respond to emergencies are counted in

this measure. The score is obtained with the following formula:

ibM300

Where

ibM300 are all backward integrations connected to respondent

decisions that list Message 300 as a preceding message.

35. Disaster TIA ' •m

We have already encountered TIA in Measure 19. In this

case we are only concerned with Total Integrative Activity as it

applies to the emergency situation. Disaster TIA calculates a

........... ... ... ...... :...: ..-... .. - :... , .
..

57 ,4

TIA value only for those respondent decisions that list M300 as

a preceding message. The formula for Disaster TIA is calculated

as

friM300 +bM300 + ipfM300 + isfM300

where the first two terms are applied as previously discussed and

ipfM300 are planned forward integrations that were not

realized and were planned to originate in a respondent

decision to Message 300.

isfM300 are serial connections within the same three digit

decision category that were realized and originated in

respondent decisions to Message 300.

36. Adjusted Number of Forward Integrations

This measure was designed to correct for a potential end

effect that might diminish the number of integrations obtained

in later periods of any controlled decision making task (e.g., a

simulation) or that may be due to the termination of observations

in an executive setting. Measure 36 calculates the proportion

of realized and planned integration in earlier periods of time

and applies that proportion to generate expected values during

later time periods. This formula must, of course, be adapted to

the length of any simulation or period of observation. For ..

present purposes, the first two periods of participation in our .4.

simulations are viewed as a performance baseline. Corrections

are applied to all subsequent periods. To obtain the value for

- .
56

the baseline proportion, the value Y is calculated as

Y X of Measure 4 (for periods I and 2)

Measure 20

To obtain a correction coefficient Z, Y is multiplied by Measure

20 for each of the subsequent periods. The value of Measure 36

for each time segment (simulation period) is the larger of

either Measure 4 or the calculated value of Z. Measure 36

obtains this score for periods 3-6 of the simulation. As stated

earlier, scores for periods 1 and 2 are set to be identical to

scores obtained by Measure 4.

Occasionally, the formula may call for decision by zero.

Where that is the case the value divided by zero is set to zero.

37. Recovery 2

This measure is similar to Recovery 1 (Measure 26) but

employs Measure 36 rather than Measure 4 as the numerator. In

other words, it adjusts the obtained score via the adjusted

(calculated) number of forward integrations, if they exceed the

actual number of integrations in later periods of observation or

task performance. For the simulations employed in our research,

the formula for Recovery 2 is" .

Measure 36 (period 6)

X Measure 4 (for periods 1, 2 and 3)

If the numerator in the formula falls below a value of 1.0, the

latter will be substituted to avoid an unwarranted increase in

the obtained score for the Recovery 2. In other words, if X

Measure 4 for periods 1, 2 and 3 = 3.0, the value of Measure 37
59

will be equal to Measure 36.

Additional Measurement

(1) Frequency Proportions

To some extent, performance characteristics of a person or

a decision making group may be limited by the overall decision

or action output in a task. These limitations could be due

either to lack of attention, to a lack of opportunity to respond

or to task characteristics that may limit actions. Consequently,

the proportion of performance score values as part of total

relevant activity may, in some cases, be a preferred measure of

performance. For that purpose, the quotient of relevant measures

divided by the number of actions taken (e.g., decisions made)

could represent an important estimate of actual performance.

That quotient is calculated in our computerized measurement

system for Measures 2, 4, 5, 7, 8, 19, 20, 21, 22, 27, 28 and 36.

(2) Time Periods and Overall Performance

Performance values on most measures is calculated by the

IBM AT based software program for six separate time periods in "'.I

the simulation. In addition, it is summarized for the entire

simulation. All measures are included in the score summary.

Measure values that are relevant to only one event in the

simulation (e.g., the occurrence of an emergency) are, of

course, calculated only once and are provided only as overall

values in the final score summary. Scores that are continuously

relevant are provided for each period and in the final summary

table. An example of such a printout is provided in the Appendix. --!
60

(3) Options

Since the underlying simulation software was designed to be

flexible, other scenarios may be designed to utilize that program .

at will. Such scenarios, while they might require modifications

of certain measures (as indicated) would also permit the

development of additional measures of interest that might be

based on data that are automatically collected by the simulation

system.

The simulation technique permits the introduction of .

measures that would assess the correctness and timeliness of the

content of decisions and other actions by participants. Those

measures would, of course, have to be specifically designed for

each particular scenario and would likely require additional

programming to permit computer based data collection.

':a'

i<...
61

APPENDIX

The following example of a computer printout of Measures

from a simulation (taken from the reliability sample) contains

information on six playing periods and a final Measure summary.

Please note that natural log transformations on the measures are L

not provided by the computer system and must be performed

separately.

P.%

.'..

,' ...-...-: ; -: -:--2'--'. ' . :- " -" .i:b -i-..: ki"": , ., ".. -- -- --- -- --- .- -',"- - -"- . - "- -" "-".. .-'.-v
62

PERIOD 1 9009DIS EMCTRL

1-MEASURE= 18 (# OF DECISIONS)

2-MEASURE= 19 105 % (# OF RESPONDENT DEC.)

3-MEASURE= 15 (# OF DEC. CATEGORIES)

4-MEASURE= 12 66 % (# OF FWD INTEGRATIONS)

5-MEASURE= 28 155 % (MULTIPLEXITY F)

6-MEASURE= 451 MINUTES (WEIGHT)

7-MEASURE= 6 33 % (# OF BlD INTEG)

8-MEASURE= 9 50 % (# OF UNINTEG.RES.DEC.)

9-MEASURE= 1294 (QIS)

1O-MEASURE= 1937 (WEIGHTED QIS)

11-MEASURE= 4.968422 (AVE.RESPONSE SPEED)

12-MEASURE= 2 (SERIAL CONNECTIONS)

13-MEASURE= 25 (PLANNED INTEGRATIONS)

14-MEASURE= 3 (GENERAL UNINTEGRATED DEC.)

17-MEASURE= 15 83 % (# OF DECISIONS IN RESPONSE)

18-MEASURE= 3.946667 (MOST RECENT RESPONSE SPEED)

19-MEASURE= 45 250 % (TOTAL INTEGRATIVE ACTIVITY=TIA)

20-MEASURE= 37 205 % (TOTAL FORWARD INTEGRATIVE ACTIVITY=TFIA)

S21-MEASURE= 10 55 % (INTEGRATIONS ACROSS CATEGORIES) "

22-MEASURE= 2 11 (INTEGRATIONS WITHIN CATEGORIES)

23-MEASURE= 3.666667 (PROF.CAT.INTEG.)

27-MEASURE= 14 77 % (# INFO SEARCH DEC.)

28-MEASURE= 8 44 % (# INFO SEARCH DEC. INT.)

36-MEASURE= 12 67 % (ADJUSTED # OF FWD.INTEG.)

,. -
63

PERIOD 2 9009DIS EMCTRL- - - -

1-MEASURE= 14 (# OF DECISIONS)

2-MEASURE= 14 100 % (# OF RESPONDENT DEC.)

3-MEASURE= 9 (# OF DEC. CATEGORIES)

4-MEASURE= 2 157 % (# OF FWD INTEGRATIONS)

5-MEASURE= 145 1035 % (MULTIPLEXITY F)

6-MEASURE= 184 MINUTES (WEIGHT)

7-MEASURE= 4 28 % (# OF BID INTEG)

8-MEASURE= 6 42 % (# OF UNINTEG.RES.DEC.)

9-MEASURE= 1598 (OIS)

1-MEASURE= 5288 (WEIGHTED QIS)

11-MEASURE= 10.19286 (AVE.RESPONSE SPEED)

12-MEASURE= 7 (SERIAL CONNECTIONS)

13-MEASURE= 26 (PLANNED INTEGRATIONS)

14-MEASURE= 1 (GENERAL UNINTEGRATED DEC.)

17-MEASURE= 13 92 1% (# OF DECISIONS IN RESPONSE)

18-MEASURE= 8.976923 (MOST RECENT RESPONSE SPEED)

19-MEASURE= 59 421 %X (TOTAL INTEGRATIVE ACTIVITY=TIA)

20-MEASURE= 48 342 "% (TOTAL FORWARD INTEGRATIVE ACTIVITY=TFIA)

21-MEASURE= 19 135 % (INTEGRATIONS ACROSS CATEGORIES)

22-MEASURE= 3 21 % (INTEGRATIONS WITHIN CATEGORIES)

23-MEASURE= 5 (PROP.CAT.INTEG.)

27-MEASURE= 1 7 % (# INFO SEARCH DEC.)

2-MEASURE= 0 0 ( (# INFO SEARCH DEC. INT.)

6-MEASURE= 22 157 % (ADJUSTED # OF FWD.INTEG.)

_0 .1
64

====================PERIOD 3 9009DIS EMCTRL-----------------------

1-MEASURE= 13 (# OF DECISIONS)

2-MEASURE= 22 169 % (# OF RESPONDENT DEC.)

3-MEASURE= 10 (# OF DEC. CATEGORIES)

4-MEASURE= 25 192. % (# OF FWD INTEGRATIONS)

5-MEASURE= 220 1692 % (MULTIPLEXITY F)

6-MEASURE= 303 MINUTES (WEIGHT)

7-MEASURE= 12 92 % (# OF BKD INTEG)

8-MEASURE= 10 76 % (# OF UNINTEG.RES.DEC.)

9-MEASURE= 4243 (QIS)

1C-MEASURE= 12838 (WEIGHTED QIS)

11-MEASURE= 7 045455 (AVE.RESPONSE SPEED)

12-MEASURE= 3 (SERIAL CONNECTIONS) a

13-MEASURE= 28 (PLANNED INTEGRATIONS)

14-MEASURE= 2 (GENERAL UNINTEGRATED DEC.)

17-MEASURE= 13 100 % (# OF DECISIONS IN RESPONSE)

18-MEASURE= 4.400001 (MOST RECENT RESPONSE SPEED)

19-MEASURE= 68 523 % (TOTAL INTEGRATIVE ACTIVITY=TIA)

20-MEASURE= 53 407 . (TOTAL FORWARD INTEGRATIVE ACTIVITY=TFIA)

21-MEASURE= 17 130 % (INTEGRATIONS ACROSS CATEGORIES)

22-MEASURE= 8 61 % (INTEGRATIONS WITHIN CATEGORIES)

23-MEASURE= 2 (PROF.CAT.INTEG.)

27-MEASURE= 2 15 % (# INFO SEARCH DEC.)

' 8-MEASURE= 0 0 % (# INFO SEARCH DEC. INT.)

36-MEASURE- 25 192 14 (ADJUSTED # OF FWD.INTEG.)

L\

;h
65

PERIOD 4 9009DIS EMCTRL-----,-

1-MEASURE= 23 (# OF DECISIONS)

2-MEASURE= 4Z 186 % (# OF RESPONDENT DEC.)

3-MEASURE= 18 (# OF DEC. CATEGORIES)

4-MEASURE= 25 108 % (# OF FWD INTEGRATIONS)

5-MEASURE= 129 560 % (MULTIPLEXITY F)

6-MEASURE= 79 MINUTES (WEIGHT)

7-MEASURE= 42 182 % (# OF Bi:".D INTEG)

8-MEASURE= 28 121 % (# OF UNINTEG.RES.DEC.)

9-MEASURE= 734 (DIS)

1O-MEASURE= 2162 (WEIGHTED QIS)

11-MEASURE= 16.75814 (AVE.RESPONSE SPEED)

12-MEASURE= 1 (SERIAL CONNECTIONS)

13-MEASURE= 19 (PLANNED INTEGRATIONS)

14-MEASURE= 2 (GENERAL UNINTEGRATED DEC.)

17-MEASURE= 23 100 % (# OF DECISIONS IN RESPONSE)

18-MEASURE= 7.'30434 (MOST RECENT RESPONSE SPEED)

19-MEASURE= 87 378 % (TOTAL INTEGRATIVE ACTIVITY=TIA)

20-MEASURE= 44 191 % (TOTAL FORWARD INTEGRATIVE ACTIVITY=TFIA)

21-MEASURE= 17 73 % (INTEGRATIONS ACROSS CATEGORIES)

22-MEASURE= 8 34 % (INTEGRATIONS WITHIN CATEGORIES)

2-MEASURE= 2 (PROF.CAT.INTEG.)

27-MEASURE= 0 0 % (# INFO SEARCH DEC.)

28-MEASURE= 0 0 % (# INFO SEARCH DEC. INT.)

.6-MEASURE= 25 109 % (ADJUSTED # OF FWD. INTEG.)

'*(

• " -. .. ,,,. .- *.-.*.-.....-¢,.... .,.....-...-.... .,.... .. °.'..... ..... ,.-..,-. .,'. -.- ,,
66

F'ER IOD 5 9?009D15 EMCTRL-------------

1-MEASURE- 15 (# OF DECISIONS)N

2-MEASURE- 7 46 %. (# OF RESPONDENT DEC.)

3-MEASURE= 12 (# OF DEC. CATEGORIES)

S4-MEASURE- 13 %. (# OF FWD INTEGRATIONS)

5-MEASURE= 2 13 %. (MULTIPLEXITY F)

6-MEASURE= 2 MINUTES (WEIGHT) -

S7-MEASURE- 4 26 %. (# OF BKD INTEG)

B -MEASURE- 6 40 %h (# OF UNINTEG.RES.DEC.).

-. 9-MEASURE= 2 (DIS)

S 10-MEASURE= 2 (WEIGHTED QIS)

11-MEASURE= 13.21428 (AVE.RESPONSE SPEED)

:12MEASURE- 4 (SERIAL CONNECTIONS)

13-MEASURE- 2 (PLANNED INTEGRATIONS)

S 14-MEASURE= 9 (GENERAL UNINTEGRATED DEC.)

S 17-MEASURE- 7 46 %h (# OF DECISIONS IN RESPONSE)

S 16-MEASURE- 13.21428 (MOST RECENT RESPONSE SPEED)

19-MEASURE- 12 80 %. (TOTAL INTEGRATIVE ACTIVITY=TIA)

* 20-MEASURE- 4 26 %. (TOTAL FORWARD INTEGRATIVE ACTIVITY=TFIA)

21-MEASURE- 2 137. (INTEGRATIONS ACROSS CATEGORIES) .

2:-MEASURE- 0 0 %. (INTEGRATIONS WITHIN CATEGORIES)

23-MEASURE- 3 (PROF.CAT.INTEG.)

.27-MEASURE- 4 26 % (# INFO SEARCH DEC.)

28-MEASURE- 0) () % (# INFO SEARCH DEC. INT.)

* 36-MEASURE- 2 13 7.(ADJUSTED * OF FWD.INTEG.)
Ll lv9. m a

67

PERIO 6 9009DIS EMCTRL

I-MEASURE= 16 (*OF DECISIONS)

2-MEASURE= 9 56 % (# OF RESPONDENT DEC.)

3-MEASURE= 12 (# OF DEC. CATEGORIES)

4-MEASURE= 5 31 % (# OF FWD INTEGRATIONS)

5-MEASURE= 9 56 % (MULTIPLEXITY F)

6-MEASURE= 7 MINUTES (WEIGHT)

7-MEASURE= 7 43 %~ (# OF BKD INTEG)

B-MEASURE= 5 31 % (# OF UNINTEG.RES.DEC.)

9-MEASURE= 17 (01S)

10-MEASURE= 17 (WEIGHTED 015)

11-MEASURE= 28.76667 (AVE.RESPONSE SPEED)

12-MEASURE= 4 (SERIAL CONNECTIONS)

13-MEASURE= '2 (PLANNED INTEGRATIONS)

14-MEASURE- 5 (GENERAL UNINTEGRATED DEC.)

17-MEASURE= 9 56 X (# OF DECISIONS IN RESPONSE)

18-MEASURE- '28.76667 (MOST RECENT RESPONSE SPEED)

19-MEASURE= 18 112 %. (TOTAL INTEGRATIVE ACTIVITY-TIA)

20-MEASURE= 7 43 ;. (TOTAL FORWARD INTEGRATIVE ACTIVITY=TRIA)

21-MEASURE= 3. 18 % (INTEGRATIONS ACROSS CATEGORIES)

22-MEASURE- 2 1'2 %/ (INTEGRATIONS WITHIN CATEGORIES)

'2 3MEASURE= 1. 3333;Z (PROP. CAT. INTEG.)

27-MEASURE= 7 43 % (# INFO SEARCH DEC.)

.28-MEASURE= 1 6 /% (# INFO SEARCH DEC. INT.)

36-MEASURE= 5 31 %. (ADJUSTED # OF FWD.INTEG.)
MEASURE 15- (SPREAD ACROSS DEC.CAT.)

5316

513 5 68

614 4

771 4

7,31 4

431 4 *=t ,. .='.5

121

5323

714

772

621 ;

611 2 A..

612 2

752 2

7 22" '"

7 5 3 " ,-,

721 .. 313 2'=

625 1 It

639 1

651 1.

615 1

633 1 'K..

6161,

911 1

699 1

622 1 ,

2.4 1 .

732 1 'P

641 1

661 1 ?24 1

411 1

511 1

7121

751 1

711 1

211 1

763 1

5611

781

774 1

741 1

775 1

311 1

671 1

461 1

617 1

691 1

692 1

623 1

251 1

762 1

4.* --a-

*" - . . -. p . .t #,' .• . " ". . .: . . " ." s. .. .: ".. ." ".'." -". -"b .. ; '. . -". " ." " :'s"""t" "•" """ .
69

OVERALL MEASURES 9009DIS EMCTRL-----....--

1-MEASURE= 99 (# OF DECISIONS)

2-MEASURE= 114 110 % (# OF RESPONDENT DEC.)

3-MEASURE= 1 (# OF DEC. CATEGORIES)

4-MEASURE= 91 95 % (# OF FWD INTEGRATIONS)

5-MEASURE= 53 585 % (MULTIPLEXITY F)

6-MEASURE= 1026 MINUTES (WEIGHT)

7-MEASURE= 75 67 % (# OF BKD INTEG)

8-MEASURE= 64 60 % (# OF UNINTEG.RES.DEC.)

9-MEASURE= 7888 (0IS)

10-MEASURE= 22244 (WEIGHTED QIS)

11-MEASURE= 13.49097 (AVE.RESPONSE SPEED)

12-MEASURE= 21 (SERIAL CONNECTIONS)

13-MEASURE= 102 (PLANNED INTEGRATIONS)

14-MEASURE=- 2.2 (GENERAL UNINTEGRATED DEC.)

15-MEASURE= 1 (SPREAD ACROSS DEC.CAT.)

16-MEASURE= 1.785714E-02 (AVERAGE SPREAD)

17-MEASURE= 80 80 % (# OF DECISIONS IN RESPONSE)

18-MEASURE= 11.20583 (MOST RECENT RESPONSE SPEED)

19-MEASURE= 289 294 % (TOTAL INTEGRATIVE ACTZVITY=TIA)

20-MEASURE= 193 202 % (TOTAL FORWARD INTEGRATIVE ACTIVITY=TFIA)

21-MEASURE= 68 71 % (INTEGRATIONS ACROSS CATEGORIES)

22-MEASURE= 2 - 23 % (INTEGRATIONS WITHIN CATEGORIES)

23-MEASURE= 2.833333 (PROP.CAT.INTEG.)

24-MEASURE= 1.45679 (EMERGENCY SHIFT 1)

25-MEASURE= 2.144425 (EMERGENCY SHIFT 2)

26-MEASURE= .2542373 (RECOVERY 1)

27-MEASURE= 28 28 % (# INFO SEARCH DEC.)

28-MEASURE= 9 8 % (# INFO SEARCH DEC. INT.)

29-MEASURE= 2.599999 (DISASTER RESP.SPEED)

30-MEASURE= 11.48461 (AVERAGE DISASTER RESPONSE SPEED)

31-MEASURE= 31 (INTEGRATED DISASTER RESPONSE)

.2-MEASURE= 15 (APPLIED DISASTER RESPONSE)

33-MEASURE= 13 (# OF DISASTER DECISIONS)

34-MEASURE= 8 (DISASTER BACKWARD INTEGRATIONS)

35-MEASURE= 26 (DISASTER TIA)

3-MEASURE= 91 95 % (ADJUSTED # OF FWD.INTEG.)

37-MEASURE= .2542373 (RECOVERY 2)

. ..
°V

70

REFERENCES

Fromkin, H. L. and Streufert, S. (1983). Laboratory

experimentation. In M. Dunnette (Ed.), Handbook of

Organizational and Industrial Psychology. New York:

Wiley.

Isenberg, D. J. (1984). How Senior managers think. Harvard

Business Review, November/December, 81-90.

Peters, T. J. and Waterman, R. H., Jr. (1982). In Search of

excellence. New York: Harper and Row.

Streufert, S. (1970). Complexity and complex decision making:

Convergences between differentiation and integration

approaches to the prediction of task performance. Journal

of Experimental Social Psychology, 6, 494-509.

Streufert, S. (1983). Measurement of task performance on the

basis of the time-event matrix: An extension of methods.

Pennsylvania State University College of Medicine:

Technical Report ONR #12.

Streufert, S. and Driver, M.J. (1965). Conceptual structure,

information load and perceptual complexity. Psychonomic

Science, 3, 249-250.

Streufert, S. and Schroder, H.M. (1965). Conceptual structure,

environmental complexity and task performance. Journal of

Experimental Research in Personality, 1, 132-137.

•°1

::: $,___ -___ , .,.- .-... ,.;..v;,,-. -. ...- ............... . -.- '. ' .,.-,7