Technical Report 407 

tIp~I

~~~ ~

k

I APPLICATION OF COMPUTER SIMULATION

TECHNIQUES IN MILITARY EXERCISE CONTROL

SYSTEM DEVELOPMENT: I.

NETMAN Model Sensitivity Test and Validation

Arthur I. Siegel, W. Rick Leahy

and J. Jay Wolf

Applied Psychological Services, Inc. ~~,

1..f ~~~~

• ~~~ .—‘- ~~~~~ and ~~~~~ • ~~~~~~~

• Thomas G. Ryan

U.S. Army Research Institute for the C Behavioral and Social Sciences ~

ENGAGEMENT SIMULATION TECHNICAL AREA

1’-• I Thia dcx ument hae been cippt~vea~

• 

~~ 

i r

~ 

i 

~ 

I

:

~ :

~

..: U. S. Army

Research Institute for the Behavioral and Social Sciences

September 1979

.

~~~J ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AN D SOCIAL SCIENCES

A Field Operating A gency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

/ ‘

WILLIA M L. HAUSER

• JOSEPH ZEIDNER Colone l , U S Army

Technical Director Commander

I

I Research accomplished f or the 4

Department of the Army

Applied Psychological. Services , Inc

• NOTICES

•

1

DISTRIBUTION: Prims,y distribution of this report h been med. by API. PIes~ sd~ sss cocrupondvws

concerning distribution of report, to: U. S. Army RSies,ch Instêtute for the Behiviarsl and Social Sciences. ATTN: PERI-P, 5001 Eisenho~

ver Avenue. Alex.ndri.. Virgini• fl333.

FINAL DI~~OSITION: This report ‘ywy be destroyed when it is no Iongev needed. Pleuce do not return it to

the U. S. Army Research Institute for the Bsltsviovel and Sociel Sciences.

~

QIL The findings in thus report eve not to be construed esan officiel Deportment of the Army position. unItes so designated by other authorized documents

L 

L _________________________________

- - •. - •~~~ .

“

~~~~~~‘~~~ -~~~~~ ‘ h1:- ~~~~~~~~~~~~~~
SECURITY CLA SSIFICATION OF T HIS PA GE (When Data Entered) - ___________________________________

READ INSTRUCTIONS • R____ BEFORE COMPLETIN G FORM • 

~~ 

~~~~~~~~ 

IN 

10 ______ NO. 3. REC7~~~~~ 5ATA LOG NUMBER

TIT L(td SiblUl.) —~~~~~~~~~~~~~ •

,

~~~~~ - • .- •~~~~~~~~~~~ - — ~ 

~~~~~~~~~~~~~

£2 ~~PPLICATION OF ~OMPUTER ~ IMULATION~~ECHNIQUES IN) 22 Jur ~~78-26 June 079 “

~~~~~‘~~ WLITARY ON OL SYSTEM~~EVELOPMENT. I. S —

( MODEL SENSITIVITY TEST AND VALIDAT ION 6 PERFORMING 0

—~~~ .— -•,

~~~~~ - - • .. - • • -.

• .

--.

~~~~~- —~~~~~

—---.•—• —---

THOR(.) _______________________________ B. • TRACT OR GRANT NUMBER(I)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

Ja~~~~~~~~~APS 

~~~3;8~~~~~~~~~~~~~~~

9. PERFORMING ORGANIZATION NAM E AND ADDRESS ¶ 0. PROGRAM ELEMENT. PROJECT , TASK

. . - • - AREA & WORK UNIT NUMBERS Applied PsYcholo~ icayServ1ces1 Inc.

II. CONTROLLING OFFICE NAME AND ADDRESS ~~ • ~~~~~~ -*2~ WEF8IIT I

~~~~~

TRADOC Training Devices Directorate /~

/ Septrcsi~eC 079 1

U.S. Army Training Support Center 17.

Fort Eustis , VA 23604 100

14 MONITORING AGENCY NAME & ADDRESS(I1 different free, Controfling OWcq) ‘l

~~ SECURITY CLA SS. (of thi. report)

U.S. Army Research Institute for the Behavioral ’—- -

and Social Sciences Unfclassified

5001 Eisenhower Avenue , Alexandria , VA 22333 ISa . DECLASSI FICATION/DOWNGRA OING

~SCHEDULE

¶ 6. DISTRIBUTION STATEMENT (of this Report) 

~~1, (_~

t

~7 ‘ Approved for public release ; distribu

17. DISTRIBUTION STATEMENT (of the abstract entered in Block 20. If different from Report)

15. SUPPLEMENTA R Y NOTES

IC. KEY WOR DS (Continue on revere, aid. it n.c.aaa,y end identify by block number)

Computer simulation Human Factors Performance evaluation

Exercise control Performance analysis Model validation

System analysis Network analysis Model sensitivity

Trade-of fs Modeling Simulation method

~“ 20. ABSTRACT fCoeUma. revere. .(

~~ if n.c. ~ .aIy d Identify by block nu

~~b.r)

A stochastic simulation model , NETMA N , had been developed to simu late

the information collection and scoring control systems used in tactical war—

fare training exercises. The model enables its users to obtain information

such as person nel requirements, train ing requ irements , and workload alternatives in systems such as the Mar ine Corps ’ Tactical Warfare Simulation Analysis and Evaluation System (TWSEAS) . The model had received prel iminary

sensitivity testing, and a more thorough program was requ ired . The Arm y .._.——

~~~

(Continued)

DO 

~ ~

473 EDITION OF I NOV 6S IS OBSOLETE Unclassified

j SECURITY CLASSIFICATION F TN?S P*GE (BIeen Dee. En ter

___ 

~~~~73/~

/ _

• — - . .- •

.

. - • — ;‘. 

~~~~~~~~~~~~ -I

~~u~~~~.

•

~1•• ~~
—~~--- 

~~~~~ -,--.-~

- - 

-, ‘

Ufl.- ~~ __-,~~

_•

~

_______________ _______________ 

Unclassified

SECURITY CLASSIFICATION OF THIS PAGE(IThen Data Kaf.r.d)

Item 20 (Continued )

~~ Resea rch Institute instituted a program to: (1) complete a forma l model

sensitivity test and validation , and (2) define the role of such exercise

control models in system design and redesign . - 

-

This report describes the methods, procedures , and results of the

sensitivity test. This summary of the first year ’s effort also provides

the model validation results , including selection and collection of criterion data from actual military exercises and the comparison of model

results against these measures. Conclusions indicate that the model can

be used with increased confidence to predict the performance of training

exercise control systems up to battalion level .)

- I

I

~ una~

sno1u1c8

~

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

By~~~~~~~~~~~~1

~~~~~~~~~~~~~~~~~~~~~~~~~~~

~~ va~1 and / or

Dist 8peC~

ial

I

I

Unclassified

SECURITY CLASSIFICATION OF THIS PAGE (Wh.n Oat. £nter.d)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .

~-•-Ji •

~~~~——-

Technical Report 407

APPLICATION OF COMPUTER SIMULATION

TECHNIQUES IN MILITARY EXERCISE CONTROL

SYSTEM DEVELOPMENT: I.

NETMAN Model Sensitivity Test and Validation

Arthur I. Siegel, W. Rick Leahy

and J. Jay Wolf

Applied Psychological Services, Inc.

and

Thomas G. Ryan

• U.S. Army Research Institute for the

Behavioral and Social Sciences

Robert T. Root, Team Leader

Submitted by: Frank J. Harris, Chief

ENGAGEMENT SIMULATION TECHNICAL AREA

Approved by:

Milton S. Katz, Ph.D., Acting Director

ORGANIZATIONS AND SYSTEMS

RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria , Virginia 22333

• Office , Deputy Chief of Staff for Personnel

Department of the Army

September 1979

Army Project NUmber Tactical Skill Acquisition :

1

2Q783743A780 end Retention

Approved for public release; distribution unlimited.

iii

L - 

- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~ ~~~~~~
-~~ _~

•••_~

_ - - - • .

~~~~ - — 

~ -r ~~~~~~~- - ~~~~~~~~~~ ¶•~ •-~•

~~~~ -•r

ARI Research Reports and Technical Reports are intended for sponsors of

= R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief . Upon completion of a major phase of the task , formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

V

iv
APPLICATION OF COMPUTER SIMULATION TECHNIQUES

IN MILITARY EXERCISE CONTROL SYSTEM DEVELOPMENT I:

NETMAN MODEL SENSITIVITY TEST AND VALIDATION

FOREWORD

Recent experience in military aviation and aerospace systems development has demonstrated computer modeling to be a well grounded and

cost—effective design technique , especially for defining personnel

requirements, i.e., manning levels, job descriptions, training, man—

machine interfaces and operating procedures. Cognizant of this experience, the Army Research Institute for the Behavioral and Social Sciences (ARI) is conducting a research program investigating, in part, the

application of computer simulation techniques in the design and development of Army field exercise management systems, from the standpoint of

the man in the system.

The design of a field training and evaluation management control

system focuses on field exercise management group responsibilities in

staging unit training and/or evaluation exercises. Prominent in this

research has been the development of a simulation model called NETMAN —— a stochastic digital computer model for simulating pertinent information

throughput in a field exercise management system, with the emphasis on

the people—portion of the process. Processing taxonomies simulated by

• NETMAN emerged from analyses of the Army Tactical Operations System

(TOS), and , subsequently, the Marine Corps’ Tactical Warfare Analysis

and Evaluation System (TWAES).

The research reported here involving sensitivity testing, calibration and field validation of the NETMAN model was part of the larger

research program designed to enhance field exercise management. As

such, this research is part of Army Project 2Q763743A780, Training

Development for Battlefield Effectiveness, and is responsive to the

TRADOC Training Devices Directorate of the U.S . Army Training Support

Center , Fort Eustis, Virginia.

OSEPH Z~~~~ ER

echnical Direc tor

V
~ 

- - -• - 

EXECUTIV E SUMMARY

Problem

When large scale combat exercises are conducted, they are

• coordinated by a technologically advanced field exercise control system . To this end , correct and timely information must be obtained

• from the field by the control system in order to develop troop performance evaluations. Detailed information concerning the field situation

is also necessary to control the insertion of scenario events such as

artillery, aircraft maneuvers, enemy contacts, and other combat related situations.

= The information in the system is also of value for deriving

training requirements and for maintaining realism within the scenario.

For example, casualties should coincide with the accuracy of an ordered artillery. in order for this realism to occur on the simulated

battlefield , the field exercise control system must hav e current information from which to insert simulated casualties into the battle program.

According ly, optimization of the exercise control system is important

to achieving maximal benefit from costly field exercises.

The conceptu al design of control systems is often carefully

worked out. How ever, design test before system implement ation is

prohibit ively expensive. Computer simulation represents a viable , relatively inexpensive alternative to field testing of such designs. A cornputer simulation model allow s economical test of alternate design concepts.

It is generally found that , although modification of design concepts as a

result of computer simulation testing does not guarantee a perfect final j system, it allows the elimination of many possibilities which will not

work we ll in the f ield . It follows, then , that a reliable comput er model

which can be used to “try out ” different exercise control system concepts

and compare them in a quantified evaluation has strong appeal. Such a

model could be used to help develop optimal field exercise control configurations and , thereby, maximize the training and troop evaluation

benefits from field exercises.

Background

The Army Research Institute for the Behavioral and Social Sciences has developed a family of exercise control system oriented cornputer simulation models under contract with the Applied Psychological —

vii

— — -

~~~~~~~~

___
Services. One model was developed to simulate message processing

in the Army’s Tactical Operations System (TOS). In the TOS system,

messages composed by action officers are delivered to users of inputoutput devices for transmittal to a central computer and subsequent

automatic updating of a battlefield information data bank. The cornpute r model which simulates TOS is called MANMOD. MAN MOD has

been successively implemented and calibrated. Throughout its use,

MANMOD has been found to be useful and reliable.

More recently, MANMOD was modified and expanded to allow

simulation of an entire message processing network of the type used

by a military field exe rcise control system. The resulting new computer model was called NET MAN. NET MAN was organized with the

design of semiautornated military control systems specifically in mind.

NET MAN assumes four levels of message processing. The message

generation level is the exercise referee. He re, messages are generated for transmittal to the control center. The second l.evel is me ssage transmittal, in this case by the radio ope rator. The third level

is a computer capable of decoding a message received from the field

and presenting this message to the fourth level- -a situation evaluator

called a controller.

The NETMAN model is based on the prior highly tested MANMOD and has been restructured to accommodate message handling

networks. Although some testing and model assessment was previously performed , further test, calibration, and validation analysis

were necessary in order to assure confidence in the model in its

present form.

Objectives

The overall objective of the current effort was to assess the

current status of the NET MAN computer model. Determination was

required of the degree of confidence which can be placed in the capability of the current configuration of the NETMAN model to evaluate

exercise control systems. In addition to confidence assessment, an

assessment of cost of use and the ease of use of the model was necessary. A final objective of the present work was to investigate and - 

-

identif y areas of NETMAN program modification so as to: (1) increase

ease of use , (2) increase the fidelity of the model, (3) reduce the cornplexity of the input data preparation, (4) simplify the output so as to

provide answers to specific que stions, and (5) improve the utility of

the model.

Viii

________ -• --- -
~~ -

~~~~~~ — ~~~~~~~~~ -

~~-— —~~ -~~ - ___

Method s and Results

= A variety of computer runs was completed to test the performance

of the NETMAN computer model under many different simulated cond i-

= tioris. The effect of these condition changes was then carefully evaluated from the viewpoints of rationality of output , cost analysis, ease

of use, and program logic error .

The first NETMAN aspect investigated was output stability as

a function of number of iterations. In a stochastic computer model

like NETMAN , which can simulate numerous combinations of likely

and unlikely events, a number of simulations of the same mission is

required in order to arrive at a stable estimate of the output parameter.

If an insufficient number of repetitions is used , the output will be sensi-

= 

tiv e to unlikely occurrences and may be biased . With more repetit ions,

how ever, the effect of unlikely events tend s to balanc e out. The more

complex the mission simulated , the larger the number of repetitions

which are required to produce stable output. A relatively basic mission

scenario was used in these t ests, The results indicated considerable

results stability with a limited number of iterations.

Parameters varied in the major sensitivity tests were : operator speed , operator precision, operator level of aspiration , operator

stress threshold , operator fatigue level, number of operator networks,

number of referee/radio operator teams per network , und etected error

probability, message frequency, message length , transmission delay ,

and task difficulty. The results which were analyzed from such points

of view as reasonableness, meaningfulness, utility, dependability, and

reliability generally indicated support f or the structural logic and internal valid ity of the model.

To determine the predictiv e valid ity of the NETMAN model, the

Marine Corps ’ Tactical Warfare Simulation, Evaluation, - and Analysis System (TWSEAS) was observed during control of a full battalion field exercise.

Field observers, assigned to troop units, made measurements of

message processing time and frequency. Additional observers collected

lJ data in the control center on message handling. In this manner, data

were obtained concerning the quality of the operation of the TWSEAS

system. Prior to the exercise, the control system personnel involved

in the TWSEAS operation during the field exercise were tested to provide

personnel operating characteristics for input to the model.

The NETMAN computer model was run to simulate the TWSEAS

3 performanc e during the field exercise and the NETMAN-generated data

were compared with the criterion data (actual TWSEAS operation). In

ix

*

~~~~~~~~~~~~~~~~~ —t

_~ •- -

~~
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ — •

general, quite acceptable agreement was found between the model’s

predictions and actual TWSEAS operation . The predictions of the

model fell , almost without exception, within one standard deviation

of the exercise based data.

Implications

Due to the model response to the parametric variations of the

sensitivity tests and the agreement between the model and the TWSEAS

criterion data , a substantial degree of confidence may be placed in

indications derived from the NETMAN computer model. Moreover,

the model was implemented at a relatively low cost . Future exercise

control system design would benefit from early test through the use

of NETMAN. Moreover , the NETMAN model may be used to determine or confirm personnel allocation, effects of personnel proficiency,

effec ts of va riou s operator characteristics, network configuration,

and the like in present network oriented exercise control systems.

I

x

——....— — - ‘•‘- ~~~~~~~~~~~~~~~~~
_____ -• =• - - — ••- - •,---- -

~

-

~

-

~~~

-,-

~

—— —•—••

~ 

—,— ~--.

~

•

~~~—-. -~

•- -—• •-- —• --•—-••.-----,--

~~

-•-,— - ,

~

--—.-—-.,

~~ • -—.—- -

‘~~~~~~~~ --~ 

~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .— — ~~~~~~~~~~~~~ -* ~~~~~~~~~~~~~~~~~~

TABLE OF CONTENTS

Page

I. INTRODUCTION 1

Simulation Models vs. Actual System Test 1

Overview of the NETMAN Model 2

Prior Message Processing Models 5

NETMAN Panel Review and Recommendations 6

Comments Concerning the Programming 7

Comments Concerning Mathematical Aspects 8

Comments Concerning General Utility 8

Suggestions Concerning the User Interface (Ease of Input and

Out put Interpretation) 9

Suggestions for Logic Changes 10

Panel Priority Weighting s 10

Model Preparation for Sensitivity Tests 12

II. SENSITIVITY TESTS 15

Sensitivity Test Objectives 15

Simulation Results Rationality Confirmation 15

Cost to Run 15

Ease of U se 16

Program Error 16

Sensitivity Tests Perfor med 16

Va riables and Lev els 16

Number of Iterations 19

Results - - Rationality Confirmation 19

Operator Proficiency 20

Percentage Time Busy 24

Level of Aspiration 24

Stress Threshold 27

Fatigue 27

Number of Networks 31

Nu mber of Teams per Network 36

Undetected Error Probability 39

Message Frequency 39

Field Message Length 43

Controller Message Length 45

Transmission Delay 48 3

Task Difficulty 48 -

-

Cost to Run 50 •

xi
F.—. -

TABLE OF CONTENTS (cont. )

Page

Ease of Use 53

Program Error 54

III. MODEL VALIDATION 55

Criterion Data Attributes 55

Criterion Variables Selected 55

Mod el Validation Concepts 59

Model Validat ion 64

Individual Parameter Data Development . - . 

- 65

Task Performance Time for Referees 65

Task Performance Time for Radio Operators 70

Stress Threshold Det ermination 72

•

~ Level of Aspiration Determination 74

Ind ividual Precision Determination 74

Ind ividual Speed Factor 76

Background Information 79

Message Frequency and Frequency of Message Type 79

Computer Runs Performed 81

Observer Training 81

Field Exercise 83

Field Exercise Data Collection 83

Discussion of Field Exercise Data 85

Validation Analysis 86

Thoroughness 86

Responsiveness 86

Overall Effectiveness 87

Total Message Processing Time 88

Operator Processing Time 88

Transmission Delay 89

Integration of Validation Data 89

Discussion 90

IV. SUMMARY AND CONCLUSIONS 95

H E F ER EN C ES 99

xii
~~ ~~~--

- ~~9’

~~~~~~~ • ~~~~~~~ .-• ,~

_-~—-. _—. -•—‘—---_- -- •

• ---,—----—.

- —-•— -•--- - —— •• -------—— — —

T LIST OF TABLES

Table Page

1-1 Panel Priority Rankings of Major Issues 11

2-1 Sensitivity Tests Parameters 17

2-2 Summary of Variance Analyses of Operator Speed and

Precision 21

2-3 Percentage of Time Busy as a Function of Operator

Speed and Precision 26

2-4 Effects of Stress Threshold on Simulation Results 28

2-5 Effects of Fatigue Variable on Mean Times per Message and

on Thoroughness 30

2-6 Summary of Analyses of Variance of Fatigue Effects 32

2-7 Effects of Number of Networks Parameters on Simulation

Results. 33

2-8 Summary of Analyses of Variance for Network Size Effect

on Message Completion 35

2-9 Summary of Analysis of Variance for Network Size Effect

on Time . 35

2-10 Effect of Number of Teams Per Network on Effectiveness

Measures • • 38

2-11 Summary of Analysis of Variance for Undetected Error

Probability Effect 38

2-12 Summary of Analysis of Variance of Message Frequency

Effect on Overall Effectiveness 42

2 - 13 Summary of Analysis of Variance for Task Difficulty Effect

on MessagesProcessed Per Hour . . . . . . . 42

2-14 Summary of Analysis of Variance for Task Difficulty Effect

on Message Processing Time 52

2-15 Summary of Analysis of Variance for Task Difficulty Effect

on Overall Effectiveness . . . . . . . . . . . . . . . . . . 52

xiii

- 

_ 

_ _
• •

-- ---.

~

-•-•-•—. - -

•-.•-•.- 

~~~~~~~~~~~~~ ‘

~~~‘‘‘ ~~~~~~~~~~~~~~~~~ ‘

~~~~~

‘— -‘— ‘---—— . -• 

~~~~~

LIST OF TABLES (cont. )

Table Page

3-1 Desirable Criterion Attributes 56

3-2 Criterion Variable Description and Method of Measurement 57

3-3 Criterion Evaluation 58

3-4 Criteria for Evaluating the Utility of a Computer Model . 60 -

‘

3-5 Referee Task Analyses 69

3-6 Radio Operator Task Analytic Time 73

3-7 Stress Threshold of Each Referee and Radio Operator. . 73

3-8 Level of Aspiration of Each Referee and Radio Operator 75

3-9 Success Probability and Precision of Each Referee and

Radio Operator 77

3-10 Mean Messages Processing Time and Speed Factor for

Each Referee and Radio Operator 78

3-11 Field Message Frequency per Hour 80

3-12 Message Type Frequency 80 -:

3-13 Consolidation of Message Types 82

3-14 Summary of Validation Statistics 92

I

I 

j

xiv

_______ -

L . -~~~~~~~~~~ 

• .
_ _

, 

~~~~~~~~~~~~~

LIST OF FIGURES

Figure Page

1-1 Schematic of message processing flow 3

2-1 Operator level of proficiency predictions 22

2-2 Mean time per message as a function of referee (a), radio

operator (b) , and controller (c) speed and precision . . . . 23

2-3 System responsiveness (a) and thoroughness (b) as a

function of simulated personnel speed and precision~ . . . 25

2-4 Messages completed as a function of number of teams

per network 34

2-5 Effect of number of teams on time spent by controller on

message processing 37

2-6 Effect of undetected error probability on the accuracy

component of effectiveness 40

2-7 Effect of number of messages per hou r on overall effectiveness and on effectiveness components 41

2-8 Effect of message length on messages completed and mean

time per message for referee and radio operator 44

2-9 Effect of message length on messages completed and mean

time per message for controller 46

2-10 Effect of transmission delay on processing time 47

2-11 Effect of task element success probability on message

processing times 49

2- 12 Effect of task element success probability on effectiveness

elements and on overall effectiveness 51

3-1 Steps in model development 62

3-2 information provided to referees for composing one TWSEAS

message 66

I —

~~~~~~~~~ 

_

xv 

-

~~~~~~

• ~— - ,- - t _ • —~

•—‘-‘- - j-— rc .-‘ va. rrw.J .P
-• - -

~~~~~~~~~~~~~~~~~~~~~~~~~---~ -~

— - =1 ~~~~~~~~~~~~~~~~~~~~~~~~~~ - •- — -

~~ ---

LIST OF FIGURES (cont,)

Figure Page

3-3 Task analytic data collection form for referees 67

3-4 Task analytic data collection form for radio operators , 71

3-5 Field data collection form 84

3-6 Model predictions versus field data 91 -

1) :

~

• 

t 

- :

1

~~~~~

II

xvi -

- ~~~~~ ~~ -

~~~~~~ ,
- -- 

- 

—-—--

~~~~~~~~ 

~~~~~~~~~~~~

— ‘-

~~~~~~~~~~~~~~~

I. INTRODUCTION

The NETMAN model was developed to provid e a basis for stochastic

simulation of information throughput in a military field exercise control

system. This model , described briefly later in this chapter , permits

its users to evaluate such effects as personnel distribution, varying

sy st em configurations, training, and workload on system performance

in tactical military exercise control systems.

-

• . The NETMAN model is a second generation message-handling

simulation model . Its predecessors are summarized later in this chapter .

Although the prior work provided preliminary sensivity testing of NETMAN ,

a more thorough program of sensitivity testing and validation was determined t o be required .

Simulation Models vs. Actual System Test

Inherent in any computer simulation concept is the understanding

that considerable savings can be achieved by substituting computer simulated exercises for actual system test . Such savings, of course, are

predica t ed on a demonstration that adequat e agreement can be achieved

between a model , such as NETMAN , and actual system operation . A second advantage of the use of a model is that the relative time required to

yield visable results is less than that of an actual system exercise. The

initial da t a collection and preparation for a model simulation may be

extensive but , once complete , the results of parametric variations may

be obtained in very short times (minutes/hours) as is shown in the

sensitivity runs of Chapter 2 . Other advantages of mod els in general over

• ac tu al syst em exercises are :

• ex ercising a model is less costly

• fewer personnel are involved

— • models are independent of uncontrollable conditions

• models do not expose personnel to danger or accidents

• models do not expose equipment to damage

• convenienc e

Not e also that the more complex , costly or large scale the operational

system, the more dominent these relativ e advantages of simulation mod-

~~ eling become.

‘,

~

P Last , simulation offers the capability to consider and evaluate

the impact of new anticipated equipment , d ifferent speeds and numbers •

of communication lines, as yet unauthorized operator sequences/proced ures, and system loads.

_______________________________________________________ __________________________ • ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Overview of the NETMAN Model

The NETMAN model simulates each person and each message involved in the data acquisition required for evaluating performanc e during

field ex er cises, These personnel includ e up to 27 referees, 27 radio operators, and 3 controllers interacting in a fixed network of communication

lines linked with a Central Computing Center (CCC).

The field exercise data acquisition situation simulated may be

viewed as a message processing network configured as shown in Figure

1-1. This figure symbolically displays 27 simulated referees (Ri through

H 27) receiving simulated symbolic input messages from independent sources

as well as from one of three simulated controllers (CON 1, CON 2 , or CON

3). Messages are indicated by the symbol pj .

Military field ex ercises of some forms ar e observed by referees,

who complete evaluative and situational reports which are transmitted to

a computer via a radio operator . Messages introduced into the system are

processed by various personnel and the CCC and then delivered to controllers

• for evaluation on CRT terminals. The field exercise data acquisition is

simulated through random message generation based on pertinent values

such as message length , type , and arrival time. Each generated message

is then processed through the referee-’- radio operator-’- CCC -

~control1er

network and processing time is determined along with a number of other

descriptiv e indices.

Each simulated referee in the left to right message processing flow

of Figure 1-i performs some appropriate procedure on the simulated message(s) received . This is shown symbolically by a circle in which TAR

(Task Analysis, Referee) appears. A message then passes to the corresponding radio operator (one of HO 1 through HO27) for processing in accordance with some specified task analysis procedure , circle TAO .

Up to 27 simulated messages, each processed by a different radio

operator , could then be read y for entry into the CCC. Entry into the simulated CCC for any given message is made over the communication line

for the three networks shown . Accordingly, in a given network , there may

be up to nine simulated messages competing for the one available CCC input line.

Telecommunication lines are designated in Figure 1-1, and the resultant queues await ing CCC actions are designated by ~

j ... ~~~~~~~

On a first-in, first-out basis, the CCC processes messages from :

all of the three networks in accordance with its task analysis procedure- -

depicted by circle TAC (Task Analysis, Computer) . These messages then

enter another queue awaiting action by one of three controllers. Each

2
I

/f\ \

~~

z

~

—~ ~~ -

~~ / I ~~ \ ~~~~~~~~~~~~~~~~ 

.

~~ ~

i/ /I \ -

~~~ -

-.

-

--------

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~~~~~~

0 ~~~~~~~~~~~ s...

~~~~~ 

: 

~~4~~ 444 ~~E4 __ 

~~~~

~~~~~~~~ 

U U

3
~~~~t ~~~~~~~~~~ - -• .,

_______________ -

~~~~~~~~~~

-

~~~~~~~~~~~

-

~~~~~

simulated controller then assesses and operates on the oldest message

from his network in the queu e and performs in accordance with the task

analysis for controllers as symbolized by circle TAN.

The loop is closed by the simulated generation of new messages

by the controller for input to one of the referees in his nine-way network

as a function of a parameter input to the model. Besides the link from

the controller to his referees, the referees are also interconnected and

may generate new messages as a result of their interactions with one

another . Up to 5,400 messages may be included in a given simulation run .

During NETMAN ’s processing of message information, the model places

special emphasis on certain human performance features considered to be

important in a field operational system of this sort . These include operator

stress, fa tigue, and level of aspiration.

The overall man-machine performance measure calculated by the

model , effec tiven ess, is calculated for each day of the simulated exercise.

It is composed of four independent factors- -thoroughness, complet eness,

accuracy, and r esponsiveness.

Output from the model is presented in the form of computer tabulations and terminal displays. These prov id e data which promot e insight for

evaluating alternatives both in terms of absolute and relative value. Th~ printed and displayed output is designed and organized so as to provid e r

suits that answer questions of practical importance , such as:

1. What is the average processing time for a message

through the network as a function of the frequency

of input messages, operator capabilities, and network

configuration?

2 . How do changes of input parameters affect predicted

total man-machine effectiveness values?

3. What is the loading situation relative to idle vs. busy

time for referees, radio operators, the CCC, and

the controller? —

4. How great a stress is placed on the operators? What

is the fatigue profile over the mission for each type of

personnel? - =

5. Would changes in the task organization of one or more

operators materially affect average processing time - •

and system effectiveness?

6. What are some effects of operator commission and

omission errors under variou s conditions? 

• . 

4
I

7. How would increased personnel training or improved

personnel selection affect system performance?

The program presents detailed message processing time and error

information, if desired , as well as hourly summary and run summary outputs. The detailed message processing outpu t shows the fine grain of the •

results of the simulation of each task in the processing of messages.

The hourly summary presents a consolidation of the results of a

simulated hou r’s work across all iterations and includes items such as:

number of messages completed , time spent working, end of hou r stress

level, performance and aspiration, time spent performing various processes, and average time per message.

The simulation run summary, produced after N iterations of the

exercise, includes manpower utilization, message processing times,

overall effectiveness indicators, and workload summary information.

NETMAN is programmed in FORTRAN IV for the Univac 1108

system. It is organized to allow the user to conduct various numerical

experiments relative to the field exercises. Each computer run of the

mod el represents a simulation of a field session up to 10 hours in duration conducted under conditions as specified by input parameters. Examples of exercise input parameters include the frequency of messages

entered to the system, the number of operators, and the speed and aspiration levels of these operators.

A model description is contained in Siegel, Leahy and Wolf (1977)

together with a discussion on model utilization, program flow charts , subroutine definitions, user input-output formats, and task analyses.

Prior Message Processing Models

Several of the NETMAN model concepts are based on a prior

operational computer model , developed by Applied Psychological Services

in collaboration with the U. S. Army Research Institute for the Behavioral * 

and Social Sciences, for simulating the U. S. Army ’s Tactical Operations

System (TOS). The earlier model , called MANMOD (Siegel , Wolf , & Leahy,

1972) simUlates the behavior and performance of up to six men who function —

as action officers and input-output device operators in the TOS system.

These men perform tasks similar to those simulated in NETMAN. The

mechanism for task-by-task performanc e evaluation in NETMAN is basically the same as in MANMOD .

MANMOD was originally designed for batch run processing in

FORTRAN IV on the CDC 3300 computer. In this form, original sensitivity

~

.

- ~~~~~~~~
F ~~~~~~~~~~~~

- -- --

~~~~~~~~~

- -•• -—- -

~~~~~~~ -- -• -~~~~ -

- ~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~ -

~~

-•- - •

and validation runs were made , in the validation, a high degree of

correspondence was found between the model’s output and a set of error

data collected from an independent source .

In a follow-on effort (Siegel, Wolf , Leahy, Bearde, and Baker, 1973)

the MANMOD mod el was modified to operate in an interactive computer timesharing mode. This feature allows the experimenter (mission analyst) to

interact in a “conversational ” mod e with the model and to enter data “on

line. ” This interaction is performed through a computer terminal and

greatly increases the ease with which simulations can be performed . NETMAN

possesses this same type of interactive capability.

A variant of the MANMOD was also developed (Leahy , Lautman ,

Bearde , and Siegel , 1974) which ailows collection of data during an experiment in which one or more actual operators perform a part of the process

and the computer simulates the remainder of the TOS activity.

More recently, MANMOD was adapted for the Univac 1108 computer

and several new capabilities were added which increase the realism of the

simulation. It was modified to exchange data wit h two other independent

computer models in such a way as to maximize the strong points of each

of the models (Leahy, Siegel , and Wolf , l975a , 1975b). Other area s of

similarity between MANMOD and NETMAN are:

1. the message generation technique is similar

2 . the operator performance and asp iration determination

technique is the same

3. some of the parameters and two of the four effectiveness

= factors are the same

4. the basic nature of both models is stochastic. As a

result , a number of repetitions is required to produce a

st able result

5. both models provide lists of inputs, optional detailed

output , hourly sum maries, and run summaries.

NETMAN Panel Review and Recommendations

Before proceding with the NETMAN sensitivity tests and validation, •

it seemed proper to obtain an independent review of the model. A review

panel of inde pende nt personnel, expert in various areas of digital modeling

and related field s, was brought together for this purpose. The panel members

made suggestions relative to: model improvement, additional features to

be add ed , features which should be deleted , features which may yield misimpressions, errors in the work , and attributes which are useful, Particularly important were suggestions relative to improvements concerning

documentation, sensitivity test considerations, and validation.

6

I.--
_____ — • • -.-•- •. -~~~~~ • • - --• 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In preparation for the panel meeting, panelists were advised of

the following areas of emphasis:

• How the programming, its architecture and organization,

the documentation, and the programming structure might be

improved.

• How the mathematics could be improved , any errors, any -•

discontinuities, any area over embellished, any “better”

approaches.

• What is the general utility of the model fcr achieving its

purpose, how the model compares with other models, how

the documentation compares with documentation of physical

models and obvious positive and negative aspects.

• The value of the user-model interface, the value of the

mod el, and the output formats as aid s in decision making

along with display characteristic, output interpretation, and

system architecture improvements.

• How the Army user will use such a model, where he will

find difficulty with it, where he will find its information useful, what can be done to improv e its utility, what are the user

interests and characteristics, and what input problems face

the user.

A summary of the panel ’s conclusions in each major area fo~

llow s.

Comments Concerning the Programming

The panel found the model to be highly portable, being written in

FORTRAN IV with almost no machine specific aspects. The exception

is the random number generator. Such routines are usually uniquely

developed for each computer system.

The program was held to follow the rules of structured programming.

Each major function in the program is separated into its own subroutine.

This allow s program changes to be quickly made.

The subroutines were said to be well organized and the interfaces

between subrortines to be clearly identified. Internal documentation,

including comment cards, was held to be adequate.

The fixed format input was said to be difficult for the user and

substitution of free format input or more identifying information was

-

~~~~~~~~~~ 

____ 

7

S.-

L

~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~
- •r,~

. 

- •

suggested. The panel also said that the program is relatively large. This

feature may limit the number and type of model enhancements. Additionally,

the panel contended that some data are specified for input but not used

internally (for example, operator factors for the computer are not used).

A minor discrepancy in the program was noted by the panel. This

discrepancy appears to hav e no effect on the processing! If the random

number generator is called in a certain way, an erroneous output would

result. How ever, the random number generator is never called in this

way. Accordingly, a problem could only arise in the case of model changes

which involved calling the random number generator in the specific error

application. -

Comments Concerning Mathematical Aspects

The stress function has a strong discontinuity at 95 percent of time

worked . This may cause an unrealistic instability in simulated performance

when the percentage time worked is around 95%. It was noted that only part

of the originally developed stress function was incorporated into NETMAN

and that inclusion of the entire stress function should be considered .

The message length variable is simulated in the model with a

normal distribution, while the real distribution would probably be positively

skewed . It was suggested that a log normal distribution or a triangular

distribution might be substituted .

Comments Concerning General Utility

The panel held the model to be flexible and to be usefu l for simulating

exercise control systems. Becau se the model is totally compartmentalized ,

it can be extended to other goals not originally anticipated .

NETMAN is a research tool and as such is primarily intended for

research psychologists and system analysts. It offers a way of trying out

new ideas, pr oced ures, equipment configurations, and personnel combina tions . Generally, such models have value in that “11 it works in the mod el,

it may wor k in the field . If it does not work in the model , then it will not

work in the field.” NETMAN is not suitable for a casual user and needs more

user choice of output if it is to be directly useful for system engineers.

The input is a formidable obstacle for first-time users, according to

• the panel. The values to assign to the input data are also a problem. There

• are little data available from field exercises, and laboratory data have limitations.

8 

- 

- -
Suggestions Concerning the User Interface

(Ease of Input and Outpu t Interpretation)

The panel’s recommendations for the user interface included:

1. User worksheets should be developed which would lead a user

through the input data collection and preparation in a step-bystep manner .

2 . Examples of how to set up input data should be included in

the U ser ’s Manual .

3. Automatic edits of input data to identify illegal or erroneous

entries should be provided .

4. More data default options are needed to facilitate the case in

L which the u ser only wants to enter a .

~mall set of new data.

5. A capability is needed which would allow the user to enter data

in the form that is normally available and have the computer

automatically perform any transformations required. For

r example, the program now requires the probabilities of related

events to be entered as cumulative probabilities.

6. An option to print as results only the data type which has been

changed ,instead of all of a cat egory,is needed .

7. In the interactive mod e, the need exists for more computer

prompting and options for da ta display.

8. A more positive action is required to initiate execution. - At

present , a blank or zero entry starts the processing.

9. The operator should be able to display only selected output data

elements or only the data elements that change between • 

. simulations.

10. To facilitate understanding, graphic output displays should be

provid ed . .

11. An easy interface to statistical program packages which -

perform data analysis might be helpful .

I

____ • •

•

~ ___

~~~

• • •

~ 

___

_____________________________________________________
- . -

~~~~~~~

- - - -• •- 

~~

• •

~~~~~~~~ 

-

~~~

- • --• •

~~~

- -•• -- - -

~~~~~~~~~~w .----— -

~~~~

Suggestions for Logic Changes

The panel ’s suggestions for logic changes included the need for

entering human parameters for each man ind ividually (at present , these

parameters are entered as constant for each type of personnel) and for

the inclusion of stress, aspiration, and fa tigue in the error generation

formula. The panel also suggested different fatigue curves for different

personnel types since it felt that the referees and the radio operators in

the field may fatigue more quickly than controllers.

The panel also suggested that logic might be added to simulate

referee mobility, simultaneous observation of multiple situations, and

decision making functions. At present the referee is primarily simulated

only in his message originating aspect .

Fluctuation of level of asp ir ation due t o local effects such as

presence of officers and situ ational variables was indicat ed by the panel

t o be an additi onal desirable fea ture along w ith an increase in the simulation duration capability beyond 10 hours.

Panel Priority Weightings

A list was compiled of the 22 major issues raised during the

panel discussion and each of the six panel participants was asked to jud ge

each item on the following five point scale:

5 = High priority

4 Considerable priority

3 = Moderate priority

2 = Low priority

1 = Negligible priority -

After priorities were assigned by each panel expert to each item ,

the mean priority assigned w as calcula t ed for each item. The mean priority

rating s were ranked. The items, as w ell as the ranking of the priorities,

are shown in Table 1-1. The three items tied for the highest priority were

user manual enhancements, programming changes for improved ease of

use , and discrepancies between manual and program. The fourth ranked

item was utility testing, which refers to use of the model in such a way as

to draw out any problems, limitations, or other unknown characteristics of

• the program. This item is largely satisfied by the sensitivity tests and validation runs, reported in later chapters of this report.

• Tied for fift h place were norm development , data base development ,

and abbreviated user mode initialization. Norm development refers to

the availability of the necessary input data for the model, and abbreviated

user mod e includes the reduction of data input and outpu t by the user to a =

level most convenient for his application.

- - 10
____ - ~~~~~~~~~~~ .

“—- ————- - - ---—-‘-- - —-- - ~~~~ -~~~~~--—~~~~~~ — __ __ ___-_- _ _P_

~

-__ -~

- • ~~ -_________ • --- --~.--•---_--.S——-_-.-.-———

Table 1-1

Panel Priority Rankings of Major Issues

Issu e Issue

No. Description of Issue Rank

1. User manual enhancement 2

2 . Programming changes for improved ease of use 2

3. Discrepancies between manual and program 2

4. Utility testing 4

5. Norm development, data base development 5

6. Abbreviated user mod e, initialization 5

7. Human effects on error rate incorporation 8

8. Inquiry mod e for output and output d ifference analysis 8

9. Overall processing 8

10. Better simulation of referee activ ity 10

11. Grap hic output provision ii

12 . Window package display : controller-referee interface 11

13. Effectiveness measures 13

• 14. Fatigue curve ind ividualization 13

15. Stress function-error frequency modification 17

16. Catastrophic failure indicator 17

17. Sensitivity to local stressors 17

18. • 

Statistical package incorporation 17

19. Human submodel validation 17

20 . Valid ity 20

21. Geographic representation 21

22. Stress threshold test 22

I

’

11

L - . - 

.• ~~~~~~~~~~~~ ~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ •

~~~~~~~~
Tied for eighth place were human effects on error rate , inquiry

mod e for output , and overall processing. The human effects which might

be expected to affect error generation are stress, aspiration, and fatigue.

The inquiry mode would allow an on-line experimenter to request the

specific sets of data or analyses which are relevant for his use . Overall

processing refers to the entire through put and the event oriented sequencing

of message handling.

Tent h ranked was the expanded simulation of referee activity described earlier under logic changes.

Lower priority issues are considered next . These include: (a)

graphic output to allow the capability of automatically transforming the

mod el output into bar graphs or line graphs, (b) a window package to allow

a special type of out put display, (c) fatigue curv e individualization to allow

different decrement curves for different levels of the network as well as

different decrements for d ifferent operations at the same level of the network , (d) stress function error frequency modification to allow stress to

affect the number of errors, (e) catastrophic failure indication to allow

consideration of those factors, su ch as radio failu re, which would actually

shut down or greatly reduce the effectiveness of the system, (f) sensitivity

increase as the result of local stressors such as level of aspiration change J

in the presence of officers, (g) statistical package interface so as to

facilitate data reduction, (h) separate testing , perhaps through laboratory

measurement, of the component subroutines which simulate human activities,

(i) incorporation of area and movement effects on personnel in the simulation , and (j) investigation and possible incorporation of a fuller stress

function in the NETMAN model.

Model Preparation for Sensitivity Tests

A full description of the NETMAN program is found in Leahy,

Siegel, and Wolf (1975). To facilitate the sensitivity and validation runs,

a series of fine tuning and readiness-type adjustments was mad e to the

program. These are summarized here :

1. the prior NETMAN program was copied to a new file (NETVAL)

and an intact copy of NETMAN was retained so as to allow

separation of the two developments. J

2. new control language files were prepared to facilitate

simulations.

3. the inpu t format was expand ed to allow entry of the day number

for fatigue calculation for each simulated person rather than a

single value for all individuals.

12

p

— .

-~~~,- — ~~~~~~~~~~~
____ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~

-

~~~~~~~

-

: 

— 

~~~~~~~~~~~~~~~~~~~~~~~~~ 

—- -

4. the input format was changed to allow input of different

value s of speed , precision, aspiration, and stress threshold for each person simulated.

5. message origin identifi cation was revised so that messages

originating from the same stimulus message are keyed to

that source message.

6. a new type of task element, random walk task, was added

to better simulate decision tasks. Associated with this a

s new subroutine , IN RAN , was added to control random walk

dat a input.

7. corrections were made in the random number generation

function to prevent its misuse as discussed in the preceding section.

8. variable dimension statements were modified to accommodate larger data arrays for the following~ fatigue , speed ,

precision, aspiration, and stress threshold.

9. modification to allow simulation of networks which include

one r feree, one radio operator , and one controller--a case

not previously ha ndled by NETMAN.

ing, and printing elements. These changes facilitate perfo rmance of multip le simulation runs.

~~ 10. set up of new files for input , output , and mapping, compil11 . enhancements to enable the analyst to enter task difficulty

and task duration data from the terminal in an interactive

mode.

L ____ 

13
LU. IL-J, - 

~~~~~~~~~ 

_

-- ~~~~~~~~~

II. SENSITIVITY TESTS

- 

As the first step in evaluating the utility of the NETMAN simulation

model , a comprehensiv e set of sensitivity tests was defined and conducted .

The purpose of these tests was to evaluate the internal logic of the model,

as a prerequisite to a meaningful validy study, presented in Chapter III.

Sensitivity Test Objectives

Fou r inherent aspects of the NETMAN program were considered

in the sensitivity tests: simulation results rationality confirmation, cost

analysis, ease of use, and logic error.

Simulation Results Rationality Confirmation

One of the major goals of the sensitivity test of any model is confirmation that the model ’s output is sensitiv e to variation in model input data.

• Not only should the output reflect changes in input but the change should

possess appropriate directionality and magnitude. Moreover, the output

should , in most cases, be reasonably smooth over a change in input

L values. Parametric values were selected for the sensit ivity tests which

would allow determining whether or not the model exhibits these properties.

Sensitivity test results often suggest a need for model changes.

Within a model, constants and equations are employed to detail relationships.

When an illog ical r esu lt is obt ained , the need for modifying such constants

and relationships is suggested .

In addition to errors of commission, errors of omission in the

program were sought as the result of sensitiv ity tests. There may be

~ combinations of conditions for which additional logic is required or the

need for additional variables may be suggested . When developing a model,

it is difficult to anticipate all possible interactions . Sensitivity tests were

made with the thought that the model would be calibrated as required .

Cost to Run

A byproduct of sensitivity tests is information concerning the cost

for running the program . This was a secondary goal of the tests. Although

the cost of computer time is generally slight in comparison with the cost

- 

- 

of oth er method s of simul ation , computer time cost may be a limiting factor

in model use. Knowledge about run costs as a function of inpu t parameters

was therefore sought.

- • -

-—••• - - - 

• 

-— 

• 

15 

~~~~~~ ~~~~~:

- - . - - —-

~

-

-‘- - •L . ~~ - ‘

~~~~T~~
— ~~~~~~~~ — —— —~ - • ~~~~~~ —•_——— —, ~‘ — — - - ‘ - (r~

— — — - ~~~~~~~~~~~~ —.—.—— _---—— —-—-.— .————

Ease of Use

Information about the ease of use of the program is another byproduct of the process of organizing and performing sensitivity tests.

Problems relative to input data organization and program execution

become evident. Additionally, estimates of the time required to prepare the input data are made available.

Program Error

The NETMAN program was previously tested and is believed to

contain minimum logic error. However, one goal of the sensitivity testing is identification of any overlooked program errors. Program error

correction is an ongoing process which starts in the initial programming,

continues on through the sensitivity testing, and often continues eve n into

f inal product use, with the return from tests becoming smaller and smalle r

as additional te sts are performed. Sensitivity te sts allow examination of

the model in many diverse situations and elimination of program errors

as they are identified.

Sensitivity Tests Performed

In order to meet the sensitivity test objectives described above , a

comprehensive set of sensitivity tests was performed . The parameters

varied in these tests are shown in Table 2-1. The program was run w ith

the parameters shown on a Univac 1108 Computer System. Both the ter-

, mirial and the batch modes of operation were employed .

- 

• Variables and Levels

The tests included a large number of variable s, and each variable

was tested at two or more levels including the limits as specified in the

model. Generally, when a single input parameter was varied across its

practical range , all of the other variable s in the modeiwere held constant.

In addition to the parametric variation, a number of selected combinations

of variable s was te sted.

The primary input parameters were selected for variation in the

sensitivity te sts:

1. number of iterations of the simulation run

2. operator speed measure ~> operator proficiency 3. operator precision level~ 4. operator aspiration level

5. operator stress level -

6. operator fatigue level

7. number of operator networks

8. number of operator te ams per network

16

L 

- _________
— -

Table 2-1

Sensitivity Test Parameters

Assigned Values

C. S S S .

~~ Q. D5 o .

~~ .4 C.. — a 0 ~ C. ~~~~~ .! ~~

~

0 -

~ — 6 2 e, -

~ .

~~ 

-

~~ ~~ ~ .~ ~ u

~ a. ... ~— ~~~~ ~~~~ ~ C. ~

~ ~~~ 

-

~~ ~~~

~~~~~ ~ ~~~ce ~~~~~~~~~~~~~~~~~~~~ CeO 0~~ ~ ~~~~~ O~~ de s.. C, ~~~~~~ ~~~~~~~~~~~~~~~~~~~ .~~~~~~~~ ~~

Number 

Parameter ~~~. ~~~. ~~~. ~~ ~~.

ta Iterations 5 1.0 1.0 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

lx N 1.0 1.0 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

2 Operator N .90 .90 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

3 Proficiency N .95 .90 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

~ (speed and N 1.00 .90 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

r ~ - ~~ N 1.05 .90 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

6 p ec 10 N 1.10 .90 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

7 N .90 .95 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

8 N .95 .95 .95 2.3128 .01 5 22 300 2.0 .9524

9 5 1.00 .95 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

10 5 1.05 .95 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

11 N 1.10 .95 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

12 N .90 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

13 N .95 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

14 N 1.00 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

15 N 1. 05 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

16 N 1.10 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

17 N .90 1.05 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

18 N .95 1.05 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

r 19 5 1.00 1.05 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

F: 20 N 1.05 1.05 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

21 N 1 .10 1.05 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

22 N .90 1.10 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

23 N .95 1.10 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

24 N 1.00 1.10 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

25 N 1.05 1.10 .95 2.3 1 2 8 .01 5 22 3u0 2.0 .95 2 4

2b N 1.10 1.10 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

27 Operator N 1.00 1.00 .97 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

28 Aspiration N 1.00 1.00 .90 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

29 5 1.00 1.00 .98 2.3 1 2 8 .01 10 22 300 2.0 .95 2 4

30 N 1.00 1.00 .95 2.3 1 2 8 .01 10 22 300 2.0 .95 2 4

.95

31 Operator N 1.00 1.00 .95 2.0 1 2 8 .01 5 22 300 2.0 .95 2 4

32 Stress N 1.00 1.00 .95 2.0 1 2 8 .01 15 22 300 2.0 .95 2 4

33 ThreshoLd N 1.00 1.00 .95 3.0 1 2 8 .01 5 22 300 2.0 .95 2 4

34 51.00 1.00 .95 3.0 1 2 8 .01 15 22 300 2.0 .95 2 4

• 17

— —— ~

—- ~~~ ---— ---—- - •———-—--

-

. 

•~~~~.. .-.--~-~ - --- ~ ~~~~~~~~~~~~~~~~~~~
~--——-~

—,- -

~~- w .•- _ ~~~~_~~~~- •—-----—

____________ _________________ ~~~~~ ---— •--- -

Table 2- 1 Continued

35 Fatigue N 1.00 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 10

36 5 1.00 1.00 .95 2.3 5 2 8 .01 5 22 300 2.0 .95 2 10

37 5 1.00 1.00 .95 2.3 9 2 8 .01 5 22 300 2.0 .95 2 10

38 Number of N 1.00 1.00 .95 2.3 1 1 8 .01 10 22 300 2.0 .95 2 4

39 Networks N 1.00 1.00 .95 2.3 1 2 8 .01 10 22 300 2.0 .95 2 4

40 N 1.00 1.00 .95 2.3 1 3 8 .01 10 22 300 2.0 .95 2 4

41 Number of N 1.00 1.00 .95 2.3 1 2 1 .01 5 22 300 2.0 .95 2 4

42 Teams per N 1.00 1.00 .95 2.3 1 2 5 .01 5 22 300 2.0 .95 2 4

~~ Network N 1.00 1.00 .95 2.3 1 2 - 9 .01 5 22 300 2.0 .95 2 4

44 Undetected N 1.00 1.00 .95 2.3 1 2 8 .1 5 22 300 2.0 .95 2 4

45 Error N 1.00 1.00 .95 2.3 1 2 8 1.0 5 22 30C 2.0 .95 2 4

46 Message N 1.00 1.00 .95 2.3 1 2 8 .01 15 22 300 2.0 .95 2 4

47 Frequency N 1.00 1.00 .95 2.3 1 2 8 .01 25 22 300 2.0 .95 2 4

47A N 1.00 1.00 .95 2.3 1 2 8 .01 30 22 300 2.0 .9~ 2 4

48 Field N 1.00 1.00 .95 2.3 1 2 8 .01 5 10 300 2.0 .95 2 4

49 Message N 1.00 1.00 .95 2.3 1 2 8 .01 5 100 30G 2.0 .95 2 4

SO Length 5 1.00 1.00 .95 2.3 1 2 8 .01 5 1000 300 2.0 .95 2 4

51 ControLler s 1.00 1.00 .95 2.3 1 2 8 .01 5 22 50 2.0 .95 2 4

52 Message N 1.00 1.00 .9D 2.3 1 2 8 .01 5 22 100 2.0 .95 2 4

53 Length N 1.00 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .95 2 4

54 Transmis — N 1.00 1.00 .95 2.3 1 2 8 .01 5 22 300 0 .95 2 4

55 sion N 1.00 1.00 . 95 2.3 1 2 8 .01 5 22 300 10.0 .95 2 4

56 Delay N 1.00 1.00 .95 2.3 1 2 8 .01 5 22 300 100.0 .95 2 4

57 Task s 1.00 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .99 2 4

58 Difficulty 5 1.00 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .80 2 4

59 (Success N 1.00 1.00 .95 2.3 1 2 8 .01 5 22 300 2.0 .60 2 4

Prob .)

4

18

• • ,~
9. undetected error probability

10. message frequency (number per hour)

11. message length (characters)

12 . controller message length (characters)

13. duration of transmission delay (seconds)

14. task difficulty (task success probability)

Number of Iterations

One major consideration in the cost of employing a stochastic model

is the number of simulations or iterations required to produce a stable estimate of output parameters. If the sample is too small , chance combinations

of low probability events may bias the resulting simulation summaries. In

order to avoid this random error a sufficiently large number of iterations

must be completed . The minimum size of this sample is a function of the

variation in the various random processes embedded in the simulation .

The data used to evaluate stability of simulation outputs were the

mission segment times which identif y completion points at various stages of

message processing. There are 19 segment times given for each of seven

message types. The resulting 133 time values provide a sufficient N to make

stability comparisons between computer runs, i. e., between sets of computer

iterations of the simulated exercise. Pairs of simulations were run with increasing numbers of iterations. The results of the simulations were intercorrelat ed and the mea n differe nce betw een runs was ca lculat ed . The criteria

for sufficient iterations was the point at which the correlation between simulations was .90 or greater and b r the mean difference between runs was five

percent or less.

t The Pearson product moment correlation was calculated between the

segment time data for each mission segment from two independent fiveiteration simulations. Each such run was initiated with identical input parameters except for different starting pseudo-random numbers. The resultant

correlation coefficient was . 9999. This indicates a very high degree of

stability for the simulation involved. Such high stability might not be attained

for more complex scenarios which involve more branching. The mean difference between runs was less than five percent.

Results - - Rationality Confirmation

The major result of the sensitivity tests was confirmation that the

model produces results which vary with input variation in a reasonable

manner. The results of various simulation runs which were completed

relative to this issue are presented below.

19

t 

_ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ,, - - •,

~~j’--—.---

— 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~b. ~~~~~ • _________________
~ - -- - ---- r -----~~- -- . • •

~~~~ 

Operator Proficiency

Operator proficiency is reflected in the NETMAN model through

two variables- -operator precision and speed . Operator precision affects

the probability of error ‘while operator speed affects the amount of time

required to perform tasks. These variables were combined factorially

to allow analysis of the main effects as well as the interactions. The value

of each of these two parameters was varied as follows: 0. 90, 0. 95 , 1. 00,

1.05, and 1.10. These runs are listed in Table 2-1 as runs 2 through 26.

Other exploratory runs were made with a setting of 0. 5 for operator

speed . Three separate two-way analyses of variance were performed on the

segment time data from this set of 25 runs, for referee, radio operator

and controller. The results of these three variance analyses are shown

together in Table 2-2 . The results of these analyses showed that the

following parameters produced statistically significant effects:

speed of referee

speed of radio operator

speed of controller

precision of referee

precision of radio operator

precision of controller

interaction of speed and precision for referee

interaction of speed and precision for controller

The anticipated direction of each prediction is indicated in Figure 2-1

-

• 

by an arrow head . Prediction number 1, as shown by this Figure is that

simulated operators with a precision of . 90 and a speed of . 90 will perform

better than simulated operators with a precision of . 90 and a speed of . 95.

Similarly, prediction 27 is that simulated operators with a speed of . 95 and

a precision of . 95 will do better than operators with a speed of . 95 and a

precision of 1. 00.

There were 120 predicted differences (40 each for referee, radio

operator, and controller). 01 these, 113 were in the correct direction, six

were tied and one was in the wrong direction. A normal curv e approximation yielded a statistical significance level of 0.001 for this result. . -

~

Figure 2-2 (a), (b), and (c) show the mean time in second s required

per message for each of the three types of system personnel as a result of

the variation of the operator speed and precision parameters over the range

0. 90 to 1. 1. As expected , in all cases, improved speed and precision

resulted in reduced message processing time. For speed , the value 0. 9

represents fast persons and 1. 1 represents slow personnel. Precision

refers to the operator tendancy to produce errors which require repeat of

20

C .

- - - .

- - - _ _ . _ &•____ _ • ~~~~~~~ ~‘

.~~~~~~ -• - O •~ - -.
~IC 

~~~~~~~~~~~~ — ~~~~~ 

— — -. 

ill ~~~~~~~ ~~~~ ~~~~~~ ..wa...-T w—p ~~~1 •~~~~ ._ — —-——— ——————•——•—————•— _— — •—.~~~———— .—— —,—-.——-—— .———~~ _— —

I

* * * z .

~

C.

* * * * * .

~~ * * ILl 145 C’~ 0) 03 03 N

• -

~~~~~~~~~~ -

0)03 ~~

, CO

03

‘-I

0

~~‘ 0) ~ o co 0 c

~

.i t- 0) 03 CO 0)1 C’I 03 03 03 ~~j l N 03 C) C— Co Lf) C)

0) Cd ~~ ~~~ ~~~ ~~ 03 .-4 03 CC) N CO Co

~ 

I CO ‘-4 ‘-4 CC) 

~ , ~ ~~ uS CC) N ri C- C/) I • • — — • •0 -‘c ~ , C-. ,..4 ~~~,..4

a) a)

0

Cl) ~~j4 .

~~4 C) C) C) I_c) ,_

~

4 03 03 145

4 c.—, U) CC) 0) 145 ‘~ 4 N C- CO ‘

~~‘ CC) ii) 0 143 CO C’1

2 09 ~~~~LC5 I45~~~~~ ’

~~ 3 cd ~ Cd 1 0) 145 ~~ 4 CO C\1 CO 0 CC) N Lc) 145 -4 0 0) ‘d~

~ 

I 0) 0 0 C~

1 CO C— ‘d~ 0) CC) C) 0) —4 0) C’-I CO

C ~ &~~~] Cr314 0 ~~~~~Lc)~~~~ ‘ C’1 145 C0

0 I CC) LIS 03 N

c.

~

1

0

— CO cC~ •~~ ~~ 0 • (0

f >~ (00 0) V 0) ‘d~ CO I_f) C) CO I_C) 0) ~~ CO (15

0) —‘ N 0) ,-l ~~ 0) ‘-4 C-

~~

a) a) 114

o

C

Cd

ICCd — -

~~~ —-

~~

0 ..-..C .—. C 0 IC ~~~~ •

~~~~(fl

C ~~~~~~~~~~~~~~~~~~~~~~~~~

.C-C a) a)~~~4 IC •+~~ 0)a) >4 IC .4C a) a)~~~~~~ IC IC 0 r3, IC IC 0 0, IC IC

UI

- I_C) 

~~~ P 0 0 IC • Z .2 v i V I

a) Cd 0

0

21

. 

•

~~~~~~~~~~~~~~~~~~~~ —‘—.

~

;

~~~ _~

-_‘- ,— _ .L’

-

.

z:.—
, —‘--~~~~— -—---- - ——-- -- ,———-,—-—..-—--——--—~-.--——-- ,—--- —--—‘——--——. —--‘-. .— -~ - _.-- - -—-—--— -- — - _ - -

~~ - - — -________________________________________________ 

- ~~~~~~~~~~~~~~~~~~~~ 

- __

OPERATOR PRECISION

SPEED 90 95 100 105 ~ 0 - -

.90 _ _ _ _ _ _ _ _

H

l.00 E® E-® E® ~~~~® I - ~

. 

I 05 _ _ _ _ _

____ ____ ____ ____ ___ ____ . ____

— Figure 2-1. Operator level of proficiency predictions. -

‘1:

22

~~

ii .

..

. ~~~~~~~~~~~~~~~~~ ~~~~~~
—- 

~~~~ -~~- - - ~~~~~~~L 

150 

LIO PRECISION

IS.

REPEREE

MEAN TIME : 90 PRECISION

t.,,

__

~~__ , . -

~~~

110 PRECISION

ISO -

RADIO OPERATOR ‘

~~ ____

_._ 1.05 PRECISION

MEAN TIME PER ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

MESSAGE

•50 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 11 0 PRECISION

I23

MEAN TIMEPER

J (SECS) (c)

~~~ ,~ ~~~ (~~ SPEED

Figure 2-2. Mean time per message as a function of referee (.1. radio opera tor (b).

and controller (c) speed and precision.

23

~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~ ~~~~~~~~~~ ~~~~ ~~w
— ---- ---,--_ - - -—-.----.-,.--—-.-- ,—- ._-.---..-.

----.--——,—.,,.— 

some or all of a task. Improvement of precision below 1. 0 had v ery small

effects but quite significant changes in message processing time were

produced when precision values were above 1. 0. Mean time per message

increased 60 to 70 percent as a result of a 10 percent decrement in precision

above the nominal value (1 . 00), and about 10 percent for the corresponding

decrease in operator speed below the nominal.

The effect of operator speed and precision on two of the four effectiveness components, responsiveness and thoroughness, was also marked. Figure

2-3 (a) and (b) displays this affect for the system thoroughness and responsiveness indices. The thoroughness measure is the ratio of the number of - -

message blocks completed during an hour by a controller to the numbe r of

messages arriving during the hour . Responsiveness is determined as the

average message processing time (referee start to controller end) divided

by the average total elapsed message time .

Percentage Time BusX

The effect of operator speed and precision on percentage time busy

for the referee and radio operator is presented in Table 2-3. Table 2-3

shows a strong trend for the percentage time busy to increase as the simulated personn el beca me slowe r and less precise. Regardless of speed

or precision valu e, the simulated controller was busy 100 percent of the

time. Percentage time busy for the referee varied betweeen 14 and 30

percent over the range of values simulated ; for the radio operator the

range ‘was 16 to 34 percent.

Level of Aspiration

The effect of level of aspiration of the simulated personnel depend s,

in the model, on their level of performance and the stress level. There are

four conditions which affect performar~

ce:

1. the simulated personnel are performing below their level

of aspiration and are, only mildly stressed. The result

is a positive pace adjustment factor due to aspiration.

2. the simulated personnel are performing above their level - - -

of aspiration and stress is low. This will tend to cause -

~~~ -

-

the operators to decrease their level of aspiration.

3. the simulated personnel are, performing below their level

of aspiration but are under high stress. In this case,

the level of aspiration tends to be reduced and the -

speed of performance is degraded .

— 

24 

-
__ 

~~~~~~~~~~ -

~~~~~~~~~

I

I

~~~~~~~. 

- 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ as~N:Los

~~

-. 

PRECISION: 1.05 i- s

PRECISION: 1 . 10

Ib)

.S 00 400 101 LIO

OPERATOR SPEED

Figure 2-3. System responsiveness (a) and thoroughness (b) as a function

of si mulated personnel speed and precision.

25

15 —-- - —--..
- —--.- — - 

~~

— -- - 

~~~~~~~~~~~~~~~~ 

- 

~~~~~~~~~~~~~~~~~~~~ - ----- - - T~~~~- j -

~~~~

Table 2-3

Percentage of Time Busy as a Function of

Operator Speed and Precision

Precision

Speed .90 .95 1.00 1.05 1.10

.90 Referee 15 14 14 18 24

Radio Op. 17 17 16 20 28

ControlLer 100 100 100 100 100

.95 Referee 14 15 15 19 25

Radio Op. 16 17 17 22 29 -~

Controller 100 100 100 100 100

1.00 Referee 16 16 16 20 27

Radio Op. 18 18 18 23 31

Controller 100 100 100 100 100

1.05 Referee 16 17 17 21 28

Radio Op. 19 19 19 23 32

Controller 100 100 100 100 100

1. 10 Referee 17 17 18 22 30

Radio Op. 19 20 20 24 34

Controller 100 100 100 100 100

•

.

~~~~~~~~~~~~~~~~

-

26

_____ -

- -S ~~~~~~~~~~~ ______
____ _ 

~~~~~ 

4. the personnel are performing at their level of aspiration but are under high stress. The result will be a

slight reduction in stress.

The sensitivity test results indicated no noticeable affect by level of

aspiration values over the . 90 to . 97 rang e on model outputs such as the

effectiveness components or performance. The interrelations are such that

the initial level of aspiration values are believed to be short lived in the

model and dominated by other conditions. That is, regardless of the

initial setting of aspiration values in the range , a final aspiration value

of . 96 was reached before the end of the first mission hour 1 after which

initial aspiration values exerted no affect.

Stress Threshold

Within the NETMAN model , each simulated person t s stress threshold is a variable which affects his response to stress. As stress increases,

performance improves up to a point called the stress threshold . When

stress passes the threshold , the facilitation is removed . Stress, w ithin the

model, is a function of message processing workload (i. e. , percentage

time worked). Stress begins to build beyond a nominal value of 1. 0

when operators are working more than 66 . 6 percent but less than 95 percent of the time. Run 31 through 34 results displayed almost no affect of

the operator stress threshold parameter alone over its tested range of

values from 2 to 3. The results are shown in Table 2-4. Table 2-4 also -

indicates that the percentage of time the operators and computer were busy

did not change as the stress threshold was increased from 2 to 3 on the stress - tolerance scale . The threefold increase in percentage busy time due to a

threefold increase in messages per hour input was insufficient to generate a

stress cond ition because the percentage remained below 66 percent for the

referee and radio operator and above 95 percent for the controller. The

marked affects of changes in workload s, i. e. , number of messages to be

processed per unit time , are analyzed and presented in the section of this

chapter which deals with message load effects.

Fatigue

Two factors determine fatigue as simulated in the NET MAN model. During

each simulated work day, a performanc e decrement representing effects of

- 1

—— 

—‘ ~~~ .tflIC’.

’

- kr

.

j.r nr
~~~~~~~ - -— — - - —‘—-

~ —-~ -------- ----------.

Table 2-4

Effects of Stress Threshold on Simulation Results

Run Information

Run Number 31 32 33 - 

34

Stress Threshold 2 2 3 3

Messages per Hour 5 15 5 15

Results C

Mean Time per Message

Referee 85 86 85 85

Radio Operator 98 99 97 98

Controller 89 88 88 88

Effectiveness Component

Thoroughness 0. 75 0.25 0.75 0.25

Completeness 0. 97 0. 97 0. 96 0. 97

Responsiveness 0. 85 0. 85 0. 86 0. 86

Accuracy 0. 99 0.99 0. 99 0. 98

Percent Time Busy

Referee 16 48 16 48 *

Radio Operator 18 54 18 55

Computer 15 43 15 44

Controller 100 100 100 100

Final Stress, Each Operator 1 1 1 1

28

______ 1-
I

fatigue is applied with increasing influence over a 10 hour period . Second ,

on succeeding work days, the decrement is applied earlier in the work day.

For the purpose of determining the sensitivity of the mod el~s fatigue effects,

three fatigue conditions were simulated . The three conditions were: first

day of work , fifth day of continuous work, and ninth day of continuous work.

Within each day, a 10 hou r work period was simulated with workload in each

hou r held constant. Within each of three cond itions, decrement was predicted to increase over the 10 hour working period . Moreover, the performance on the fifth day was anticipated poorer than on the first simulated day ,

and performance on the simulated ninth day was predicted to be worse than

performance on the fifth day.

= The principal results of the tests of model sensitivity to the fatigue

vaHable are shown in Table 2-5. Operator performanc e, as measured by

changes in the average time required per operator to process a message,

degr aded (i. e., time increased) as expected with longer mission periods.

The percentage time increase from hour 1 to hour 9 were:

MISSION DAYS

Qperator Type ! ~~ .

Referee 9. 3 14. 0 19. 8

Rad io Operator 8.2 13.3 19.4

Controller 10. 1 14. 6 19. 3

- Average Percentage

t Time Increase - 9.2 14.0 19.5

To evaluate further the impact of fatigue, two correlational

analyses and an analysis of variance were completed for each operator

type. First, product moment correlation coefficients were calculated

for each of the three operator types and for three time periods ( day 1,

5, and 9) by hou r. The resultant values, shown below , indicate a high

correlation between hour of work and simulated operator processing

time per message. One would anticipate that as time on the job increased

(fatigue increase), the processing time would also increase. Each correlation is based on an N of 10 hours.

Referee Radio Operator Controller

1 .85 .84 .72

5 .94 .96 .89

9 .98 .97 .97

_ 

I_
- ,- ----—

to co C) ~~ IC) N 0 CO IC)

ssauq~noJoq,,L N t- to t- N to N to to to N

J~

jTTOJ1UO ”

~~~I to to .—I ~-4 ~~

4 CON —I .4 It) ~~ 4

II •r C.,F C) C) C) 0) C) 0 0 0, C)

.—4 .-I ~~~I

b

0 . I to N 0 to C’1 IC) to -4 0 N d~~ otp’e~j 1 C) C) 0 00 0 0 .- I —4 0

o .-i ,.4 ....4 ~~

.4 .4 ....4

~~~~~~ 

to It) N C) ~I to It) NO to CC) J J H ~ ~~ to to C) 0) 0) C) 0 0 0) -4 _4

cd

C)

to IC) It) IC) C) C’~

1 ‘-4 —4 0) 10 to

ssou~~~noJo~~~~~I N N N N ~~~~~~~N N .

Wto I’-

.1 8T~OJ ‘U O’’ I C) N N C) N C\I C’

~

1 ~~ N C”

~

I C’-1 IL ? s—’ I CO to CO to to C) C) C) C) 0 C)

—I

C)

~ cd c’1

I4~ ~

J) .A I to to to C) 0 to IC) CO I:’— —I CV)

~ ~

.O0TP~~T~ C) C) C) C) 00 0 0 0 —4 0

c” 1 ‘ - -

C)

I to IC) IC) IC) CO 0 ‘-4 C’l to CO 0

cd ~ CO to to to to 0) 0) C) C) C) C)

0 to N ~~

4 It) it) N ~~ to to N C) sseuqOnoJoT.~J, N N N N N N N N N to N.

-4 S • ~ ~ • . . .

Cd

‘-4

Cd C) to to CO CO to C) 1 to to 0 J

~ TTOJWoD to to to cc to to CO C) C) C) C)

C) ~ “ j Cd’-

~

• d I CO CO to N N to 0 to to to 0 - = ~ OTp~~H I C) C) C) C) C) 0) 0 0 0 0 0

Cd 4 ,I .4 . 1

I to CD IC) to N N 0 Cs3 ~~

4 to —

~~J9Ja~J 1 to CO to to to CO to C) 0) C) to

Cl)

S

CC) ~~

4 IC) CO N CO 0) 0 Cd 0 - ~

30

—~

-

~~~~~~~-~ 

-

~~~~~~

_ - 

.

-

~ — — -

~~ — c-. - ~~~~~~~~~~~~~~~~ 

— 

~~~~~~~~~~~
~~~~~~~~~~~~~~~~~ - . - .. — - ~~~~~~~~ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~. .

~~~~ ~~~~~~~~~~~~ 

-

An analyses of variance was completed for eachoperator type using

the time per message per operator data of Table 2-5. Here the mission

time was divided into three levels of days worked , and the 10 hourly

results were div ided into three periods of three hours each (the first

hour was not used). The summary of the analysis is shown in Table

2-6. Statistical significance at the 0. 001 Level of significance was

observed for all three operator types in both the effect of fatigu e on

performance for the mission time periods and for the hourly (day)

periods. This indicates the clear and significant impact of fatigue on

operator performance.

The effectiveness measure called thoroughness also showed a

consistent reduction in operator performance over time. For the 10

hou r simulated mission, the decrease in this measure was almost 16

percent .

Number of Networks

A network is composed of radio operator/referee teams who

report to a common controller . The model can simulate as many as

three such networks simultaneously. All networks are processed

through the same simulated central computer and the resulting time

sharing has an affect on message processing time. For the sensitivity

test of the effect of number of networks, three levels were used--one ,

two , and three networks. A fairly heavy workload was imposed on the

system in order to draw out the effect of the number of networks on

overall processing time. It was predicted that increasing the number of

networks would increase the time required to transmit to the computer

and that increased waiting time within the computer would result from

the increase in the number of networks being hand led simultaneously.

Within the simulation, each network had the same workload . Accordingly,

the workload factor was held constant.

These predications were only partially fulfilled under the cond itions of runs 14, 38, 39 , and 40. Table 2-7 shows a negligible effect of

number of networks on: the effectiveness components, mean message

processing time , and proportion of time busy. Table 2-7 also show s the

reason for this small effect . In the section of Table 2-7 dealing with

proportion of time busy, the computer is predicted to be busy only 7 percent of the time for one network , 14 percent for two , and 22 percent for

three networks. Accordingly, the workload assumed for the computer in

these runs did not approach an overload situation. This effect was confirmed in field tests, described in Chapter III of this report , in which ~

. -

the computer was never a bottleneck in message processing. In terms of ~

- 

-

- ~~~~~~~~~~~~~~~ - — , t

~~Ec~~~~ ~~~~~~~~~~~~~~~~~~~~~
— ~=.—---- - _ - - - --.------------~

-

~~

.

-----—--‘———._-—_,

C)

C)

0 ~~ ~~ —4 .—I ~~

-4 ,—4

00 00 0 0

00 00 0 0

C) . . S S S S 

-

‘

V V V V ‘I V

CO

C) 0 ‘-4 ~

- IC) N N C’)

~ 

I 10 C- ‘-4 CO to 10 C) CO to

Ci) ~~~ C ) C ~ - IC) O’-~ to

a)

a)

~~ It) 0 0 C) N (0 C’

~

1 IC) I- to C)

Cd~~~I ‘40

C) (“1 0 N CC) (‘s) CO CO it) N 0 0 CD toN. 

- 

C’~

I N C O N -I

)

~~~~ ~~~~~~~~~~~~~~ t o C ’ .1C ’ ~~~ 4 CO t o’ 1 C’ 1

~~~~~ to

C) Cd ai C) Cl .—i C”l ‘-I (“l ‘-4

—Cd 0

Cl)

0)

Cl)

C~ ~~I Cl 0 0 ‘l~ 0 ~~ CC) N

~~ I ~~~~~~~~~ C) P- I C) C- Q

~~~~~~ t oto O~ to~~~~ 0 t o t o E ~I 0 to ~ 4 CC) CO CC) ~~4 IC) CC) C) C- N ~~4 ~~I

N “

~~ 4 N ‘~~4 ~-I N 

rl 0 t/) C/) ~ — — — Cd p- —.

~~~ C)~~~~~~~~~ C)

CO ~~E4

~~~E-~~~ ~~E-4~~~E-4 r1

~ ~~~~~~~~~~

-

~

-‘I ~ 

0 Cdl C) ‘4-’ 0

0cd

C) . 4 ~~~~ C_ Il c4-4 “O Q)

‘1 ~ cd Q. 0 “ Z ~~0 U

32 

~~~~~~~~~ ~~~,— - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~

- J— 7——~~ —- 

—--- --—- — ——--------

Table 2-7

Effects of Number of Networks Parameters on Simulation Results

V

Run Information

Run Number 38 39/14 40

No. of Networks 1 2 3

No. of Referees 8 16 24

No. of Radio Operators - 8 16 24

No. of Controllers 1 2 3

No. of Computers 1 1 1

Results

Effective Measures

Thoroughness .79 .77 .76

Completeness . 97 . 96 . 97

Responsiveness . 86 . 85 . 86

Accuracy .99 .99 .99 -

Overall .90 .89 .89

Messages Completed Per Hour

Referee 1081 1073 583

Radio Operator 2132 2113 1655

Controller 3243 3214 2487

Mean Time Per Operator Per Msg.

Referee 86 86 84

- - Radio Operator 98 98 97

Controller 86 87 87

Proportion of Time Busy

Referee . iS . 16 . 16

Radio Operator . 18 . 18 . 18

Controller 1.00 1.00 1.00

Computer .07 .14 .22

I 

*Note : Messages completed is a total across four hours, five iterations, and

eight referees per network.

33

_____________ _________ !- — - - -

V ’ . -

~~~~~~~~~~~C/- -
p.- - - - - ‘ - - - _-‘ . 

— --- —-.. ——----_-.-- .‘- .-- - - -~~~ - *.___~~~~~~~~~~~~ ___,_.__

- ~~~~~~~~~~~~ —- —. -.-. — ~~~~~~~~~~~~~~~~~~~~~ - -,.. , . _________________________________________________ — - — -

600 -

CONTROLLER 500 - 

REFEREE

MEAN RADIO

NUMBER OPERATOR

M ESSAGES

COMPLETED

PER HOUR (419.25) 4OO~~

CONTROLLER

(339.25)

300 .

200 -

100 .

(68.75)

NUMBER OF TEAMS PER NETWORK

Figure 2-4. Messages completed as a function of number of teams per network.

34

L 

‘

iii ____~~_, — 

— 

— —
I 

~~~!,

~ C - C cd~ .i: 

~

i

~~I -

~~~

~~~ 

C-It)

‘-4 Cl Cl

0 to C-)

4 .-4 

CO 4-’ C)

cc .

~ ~~ Z

C Cl 0

I Cl 0 ~~

o

C) ~~~ ClC) ~~~C l C) ~~~C lC) W~~~ ~~~~~~~~

-

~~~ C4-4

ci 0)

C)

4-4 ci 0 “4 IC) 0 10 N ~~ N 0 N. 10 C) 0 If) Cl) j Cl 0 Cl to Cl ~~ C) CO Cl ., C-- 0 Cl

0 01 CI) c._. ~I . . ~~ I 0 ‘Ct4 CD CO N —4 0 Cl 03 o ~ I Ct4 Cl Cl ~~ cii —I C) .1 ‘I N Ct~ CO it) Cl 

~ 

I ~~ ‘ to

0 j ~~~~~~~ r - - w CO CO I C’

-) C) Cl ‘--4 0 ‘—4 to IC) Cl 

~~ Cl 0) IC) I.C) 0 N to to CO N. N CO ~~ C) Cl) Cl Cl If) it) 10 (C) 4-’ ~~4 CC) 0

toto

--4 —s

.

~

E ~

4 ~~~~~~~ _~~~~~.c:~ ~~~~~~~~

~

~~ 

‘

~~ c~~~~ ~~~~~~~~~~~~~~ ~~~i ~

E E-~ E4

CO CO E - ‘

0) 0 cil 0 C) ‘4-’

~

.,l ~

., 0 ” 0)1 4-’ C) ‘0 0) -

51 0 0)

‘U Z ~~0 -

35

- _ _

~~~~

----.

~~~

- ---‘ - ‘ - - ____

— — — 

Al \4
F -—.-

——— “ — —‘-~~~~-~~ — — — — —~~ --— ---- —

messages processed, however , adding networks produced an effect

directly proportional to the number of networks added. That is, two

networks processed about twice as many messages as one network,

and three networks processed about three times as many messages

as one network. This proportional relationship holds true at the

referee, radio operator, and controller levels of the system.

Number of Teams per Network

Within the model, up to nine radio operator/referee teams

may be assigned to each controller. An increased number of teams

could be expected to increase the number of messages process ed.

Three levels of teams per network were te sted: one team per network, five teams per network, and nine teams per network.

Figure 2-4 shows the effect of the three team levels on the

number of messages per hour which were completed by each personnel type per hour over the four hour mission simulated. As expected ,

increasing number of teams per network resulted in an increased number of messages processed. However, for the controller, the increase

was at a declining rate. For one team network, the result was 68. 75

messages per hour completely processed (

~. e., processed by the controller). Through the five team situation there was essentially no . - -

degradation since these teams processed 339. 25 messages per hour--

only about 1 percent below the single team rate. But, in the five to

nine team range , the controller became a bottleneck. For the nine

team situation, only 4l9 . 25 messages were processed as comparedwith

the potential (9 x 68. 75 = 618 . 75). The other operators’ message

processing was almost linear with the number of teams per network;

a one -way analysis of variance (Table 2-8) for three networks was calculated independently for each operator type . For all three types,

diffe rences between number of networks showed significance at the

• 001 level.

Figure 2-5 displays the relationship between the number of

teams per network and the logarithm of mean time involved in the

reception of messages by the controller. The between teams variance

(1, 5, 9 teams) was determined to be statistically significant (p <. 001).

Table 2-9 summarizes the pertinent analysis of variance. The data

used in this analysis comprised the time spent in segment 15, which

corresponds to delay to controller. Since the re are seven message

type s and the re was no diffe rence between message type s, there was

an N of seven within each cell.

36

I— - - — - - - .— —‘ ---.- .—---‘~~.— --.-?-. -, - 4~~~f~-

~ -
7—

(3.2731 ) 

/

+3 .

I

+2

LOG OF

MEAN TIME

t

! SPENT IN

RECEPTION

By

CONTROLLER +1 -

t

(0.2755)

0

(-0.6990)

____________________________________________ I

S S

NUMBER OF TEAMS PER NETWORK - -

Figure 2-5. Effect of number of teams on time spent by control ler on message processing.

1

- —

~~ ~~~~~~~~~~~~~ ~~~~~~~~~ ---- -~~~~ - - ---—----—--—~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - - - — - -~~~~~~~~~ -- - ~~~ - -
— ~~——,--—-—- -—— -

~~~~~~~

——- 

~~

—,—— - -

~~

- _ __~~ c - - — -

~~~~~ — -~ -~~~~~~~~~~~ 

- - - ------

~~~~-—.~~~~~~~~~-

—

~

-— - - - - - - - - - - —

~

-—-----—-— -- 

-4

to ~~~ N 

-

~~ 0) C) CO

5. . . •

O 0)

C)

O c i

-4

0 l

~ J ~~~~~~~~ 0

~~~~~C ) C )C)

ci

0) --4

--4

CI) .

~~ 0 C) I C) CI) N -

a) p-s C)

--4 ~

.4 -4-, C) ~

.4

° ~. 0

ci CO CO CO

U) t o to CO ~

-l

I

—

‘0 ci

~~~ I

0 CI) CI) C) ~~ I N O Cl) 4-, C)Q

0)

0 ~ Z 0) 03

‘

~~ 0 4-’

—4 0)

~~ I V Cl ~~

. ~~ Cl C) C) ~

— . Z ~~ 0) 4-i CI)

Cl) NN.N 0

~~~C) ~~~~C lC)

ci .

~-4

C) I

0 ci

IU) 0) 03 —s

ci ~~I LC) IL3 O

Z ~~cdI

C l C O CO

4-, 0) C) to 

~

‘

~

I

0

0) ~4

0 4-4 ci

E

E

CO ~

)

— 4-. ~ o ~~~~~~~~ 0 ~~

~

1) ‘ it) C) —

i

~~~ 0) ‘-4 p-I

38 

-- -

- ~~~~~~~~~~~~ -~~~~~ - -
For the one team per network case, 0. 2 second s were spent

in message reception out of a total message time of 285 seconds.

This increased to an average of 0. 8 second s for the five team per

network case out of a total time of 288 second s. How ever, in simulation

run 43 (nine teams per network),the average time spent by the controller

in message reception was 1875 second s out of a total of 2172 seconds.

Again, this suggest overloading of the controllers for this case. -:

Table 2-10 shows that only one of the effectiveness measures,

thoroughness, was impacted by variation of the number of teams per

network. After the overload cond ition set in, the anticipated reduction in thoroughness was indicated .

Undetected Error Probability -

Many of the errors which occur in the simulated message handu ng are detected and corrected; others pass through and contaminate

the simulated data base. The probability of an undetected error is

specified by NETMAN model input in terms of the probability of a low

importanc e error and in terms of the probability that a significant error

will enter the data base. These error probabilities are expressed in

terms of probability per message. These probabilities are used in the

model in the calculations of information loss and effectiveness. The

error probabilities selected for test were: . 01, . 1, and 1.0 (i. e., one

message in 100 will contain undetected errors, one message in 10 will

- contain undetected errors , and one undetected error for every message

t 

proce ssed .)

j Figure 2-6 shows a plot of the resultant effectiveness measure

Accuracy (1... total information loss ) as a function of undetected

messages completed

error probability. The other three effectiveness components were

- essentially unaffected by the undetected error probability over the

range .01 to 1.0; the resulting out put ranges were:

Thoroughness - .74 - .77

Completeness — . 96 - . 97

Responsiveness - .85

A one-way analysis of varianc e indicated the effect on accuracy

to be statistically significant at the 0. 001 level. The analysis is summarized in Table 2-11.

Message Frequency

j 

The frequency with which the simulated referees send messages,

whether originated by themselves or by the controller, is a prime

39

~~~~~~~~~~~~~~ ~‘L ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
- - - - - - - 

~~~ --- ~~~~~~~~~~~ -- 

- — ~~~~~~~~~~~~~~ - - 

~~~~~~~~

I.0

- UNDETECTED ERROR PROBABILITY

Figure 2-6. Effect of undetected error probability

on the accuracy component of effectiveness.

40

- -

~ _ 4 ~

- 

-

.

---- c’c- 
- - . _:

~~ ~~~:

~~~~~~~5ii~~~~

-

~~ 

-

~~~~

-‘

~

---

~~~~~

—

1.00

I

90

Ii -

OVERALL EFFECTIVENESS

\

\

~~~~ 50

z

14)

> \

U

w

IL.

Il.. 40 -

U)

\

\ .30 \

\

20 -

- - 

‘

~~~ ~~~ 

THOROUGHNESS

2 0_ I I I I

S IS 25 30

MESSAGE FREQUENCY = NUMBER OF INPUT MESSAGES

PER HOUR PER TEAM

j Figure 2-7. Effect of number of messages per hour on overall effectiveness

and on effectiveness components .

41

_______ __________

- - • :~~~~~

- : -

-

~~~~~~ ~~~~~~~~~ - -

- -- —-- ,‘ J_ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
~ -

____ - 

-

0)’ 0)1

CI) ‘ C ) l 0

~ 0~~~J ~~ 0) ‘-4 ci ‘—4

~~~~~~~~~~~~~~

~~ I~~~~~~~

~ ~~

CflI 

~ ~~CIDI C’) 00 C1) ,—4 

~

-4 -4 ‘4-’ ,-4

I I I F -

Cl Cl

0) 0) 4

-4 _-4

.0 Cd .0 ~~ ci Cl) ci ~~ I E--s ~~ ~~~ .

~~

I-. U) ~

-‘ I

~ COO ~ C O O .~ C ) ’0 r. C ) ’ 0

0 ~ ‘-~~ ~‘ ~~ 1’ Q) 0) ‘ Cl C)

~

- 4C ) —4 —4 

~

-I ~~~~~~ 0) bI)~~

.s 0 bI)~~ ! :; -

~~~ ~~~Cil ‘

4 .-4 0 

~~ 

~~~~ Cd

~ ~~~~~ U) ~~ • I.-. 

~ J O)C )

~~~~~CO C~ 1

OOO CI) 

~~~1 ClCl

.cr~

‘4-4

0

‘

4-4

0 Ci

C) C) ~

. 0.)

C c. ci ~

~~~ ~~~~~~~~~~~~~~~ ~~~ 

-: _±__
~ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ . 

~~~~~~~~~~~~~~ u,_~ ~~~~~~~~~~~~

determinant of the workload on the entire system. Message frequency

varied over four levels: 5, 15, 25 , and 30 messages per hour per

team. This range represents a variation from light workload , through -

heavy workload , to a work overload .

Figure 2-7 show s (solid line) the effect of input message load - per unit time on the overall operator effectiveness. This relationship

was shown below to be significant at the . 001 level as a result of a

two way analysis of variance (effectiveness measure by message frequency). This variance analysis is summarized in Table 2-12 . This

finding is largely due to the very high changes in the thoroughness cornponent. Much smaller changes were noted in the other effectiveness

components.

The delay time between computer processing and controller

start as a function of message frequency is shown below :

Message Controller Delay Proportion of Time Busy

Frequency Mean Sigma R eferee Radio Operator Controller

5 949 98 . 16 .18 . 14

15 11, 333 225 .49 .56 .44

25 11, 181 364 .69 .79 .71

30 11, 663 574 .72 .82 .86

As anticipated, delay increased as workload increased and the proportion of time busy rose even more rapidly as a function of workload.

Field Message Length

The length of each message is specified in the model in terms

of the number of characters it contains. The number of characters to

be processed is directly related to the time required to process the

entire message, assuming that the time to process each character is

held constant . The levels of message length tested were: 10, 22 , and

100 characters.

The expected increase was observed in message processing

as a function of number of characters. Figure 2-8 presents the

obtained results. The Pearson product moment coefficient of correlation between processing time (mean time per message) and message

length was calculated to be . 9996 for the radio operator and . 9995

43 F 

-a - -- - - - -. - -•~~ —~~-~~~~~ -- -

-

- 

—~~~, --—. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ — . - -. —
—‘7--

-

Cl)

U)

0

2

i

- j ‘0 I

I a, 0

V V 14) I I

V U) I I

V V I 1 E V \ 14J

~~~~~—I’.-’ I

V ‘

~. J ~~~O ( j o Li IO\ \ 

~ 

I l

~~ U!

~ I

~~~~~ Q U)

0 \ V I

~~ O~~~~~

-

-

I “

.

~ V LU .- l

~~ 

\ CD

!i\ \

V V / ‘ % , I’ 

~

. ‘

~ .

V V

\\

1’ ~~

—- ~~~~~~~~~~~~~~
for the refe ree. It was also confirmed that field message length had

a marked effect on effectiveness components, particularly responsiveness and accuracy, as shown below:

Message

Length Thoroughness Completeness Responsiveness Accuracy Overall

10 .76 . 96 . 89 1. 00 . 90

;

~~ 22 .77 .96 . 85 . 99 . 89

100 . 76 - 97 - 47 . 74 . 72

- - - - - Similarly, Figure 2-8 shows a corresponding affect, in the an- ‘

‘

. tic ipated d irect ion, of message length on message s completed. As message length increased from 10 to 100 characters, the number of messages completed decreased by about five percent .

Controller Message Length

A message which is sent by the simulated field teams is automatically decoded by the simulated compute r before it is displayed to

the controller. Messages to the controller are typically longe r, in

terms of number of characters, than the messages composed in the

field . For the purpose of these sensitivity tests, the range of message lengths used was 10, 22 , and 100 characters.

The increase in mean time per message as a function of increased message length was as pronounced for the controller as it was

shown to be for the other personnel. The pe rtinent data are displayed

in Figure 2-9 , and the corresponding triserial correlation between controller message length and mean time per message was . 99.

Correspondingly, the total number of messages completed by

the controller was reduced as message length increased , as shown in

— the dashed curve of Figure 2-9 .

— The effects of controller message length on the effectiveness

components were :

Message

Length Thoro ughness Completeness Responsiveness Accuracy Overall

50 . 97 . 97 . 91 . 99 . 96

100 . 98 . 97 - 90 . 99 - 96

300 - 77 - 96 . 85 - 99 - 89

i

45

— __a nt~

a

~~~~. c.e.na - A - 

~tC’’-~ - ~
F’ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

-7-,.-—-,’. ’__ —r-—- ~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -__

_____________________________________________________________________________ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ . _ -

Li

8 ~~~~~~~ 8

_______________________________________________________ ____

c

,

~~ 

.

\ /

/ U

\ / OW -

\/ 4 -E

~~ U

/ ‘- ~~~0 ..

~~

/

I V

E

I

I Li.. I V 0 -

.

I \ ~~ o LU V 0 CD ‘0

\ IJJ

~~~~~~~ -J I V _D a, I V p-~~~~ 4 I - I

it

0

I -I 0 0 C - 0

LU~~

46 -

___________ - 

~~~~~~~-

-.
- -

~ 

- - - - 

- - -- 

_

200 -

, 

RADIO OPERATOR

ISO -

MEAN

PROCESSING

TIME

(SECONDS )

03 - 

50 100

TRANSMISSION DELAY (SECONDS)

Figure 2 10 Effect of transmission delay on processing time

47 

- -

— 4 .

~~ --- - _______________________________________________________ 

~~~~~~~~~~~~~~~~~~~~~~~ .LI ‘

~‘
-—

~~~~~

Transmission Delay

Before a radio operator sends a message, he selects a transmission frequency and list ens to determine whether or not the frequency

is clear. Depending on the traffic , the message transmittal may be delayed . As predicted , simulating this delay added to the average message

processing time for the simulated radio operator and caused a reduction

in the number of messages which he handled per hou r.

Figure 2-10 show s these effects for the three levels of delay in

the sensitivity tests: no delay , 10 second delay , and 100 second delay.

No effect was to be expected for the controller or the referee because

this variable should unly operate on the radio operator processing time.

This consistency is also shown in Figure 2-10. The triserial correlation

between radio operator processing time and the duration of the transmission delay was calculated to be 0. 974. The effect of this delay on

the effectiveness components is show n by the following data to be very

modest - - less than a 5 percent chang e over the entire range of transmission delays.

Transmission

Delay

(sees) Thoroughness Completeness Responsiveness Accuracy Overall

0 .76 .96 .87 .99 .89

10 .77 .96 .85 .99 .88

100 .76 .97 .72 .99 .85

Task Difficulty

Task (mission) difficulty within the model is indicated by the

value of task element success probability assigned to each of the

tasks in the sequence for each person simulated. Typically, system

personnel will correct the error which caused failure on a task element

immediately. The additional time spent in correction and to’tch up,

however, increases the time required for task completion and would

be expected to reduce the overall performance effectiveness. Task

element success probabilities were varied over three levels. Under

the low difficulty cond ition, success probabilities were 0. 99 on each

task element. In the moderate difficulty condition, the success probability was 0. 80, while in the high difficulty condition a success probability of 0. 60 was entered for each task element. It was predicted

that increased task difficulty would produce a strong decrement in

the number of messages processed , an increase in the time per

message, and a decrease in overall effectiveness.

48
— -

~ 

—

0 

~~ 0 Q~

. 0

S

- - ~~~~~~~~~

— 

-

~~

/ I ~~~~~~~~~~~~

~I / I .

— 2

0 f / I0.

~ I o

~~ \ I a

~ 

ii a,

\

>~.1~LI

/ \//

Ii .;

p

4 (l) - (flu

w 0I~

a. -

49

~ 

,-~~~-

- ~~~~~~~~~~ — - -i ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
— 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

——--7—,-,-—,

- - - - — - —.- ———-‘—,.-—‘—~~ ——-———-~~ ---——-——.——‘-.---—-—- ---

Figure 2-11 confirms the anticipated effects of changes in 7 task difficulty on the number of messages per hour. Here the effect of

increased messages processed as tasks become less difficult is clearly

noted for all operators. In addition the number of messages processed

increased about 29 percent for the controller who is overloaded but not

at all for the referee and radio operators who are only mildly loaded.

A one-way analysis of varianc e (Table 2-13) indicated significance

at the . 001 level for messages processed per hour for the controller.

Figure 2-11 also show s the expected effect of decreased message

processing times for the controller as the difficulty of the task increased .

An analysis of variance (Table 2-14) also indicated this effect to be statistically significant. However, the referee and the radio operator showed

virtually no change because these personnel were very lightly loaded . Also

not e that the total number of messages processed by the radio operators

never quite equalled the number processed by the referees due to the sequence of message handling .

Several effectiveness components indicated degradation as

d ifficulty increased (Figure 2-12). An analysis of variance calculated

for the overall effectiveness measure indicated statistical significance

(p < .001). This analysis is summarized in Table 2-15.

Cost to Run

The second consideration in the sensitivity tests was cost to run.

Cost to run was examined from both the direct cost and computer time

used point of view . The U1108 facility at Edgewood , like many computer

V facilities, charges d ifferent rates depending on the time of day of use.

Computer runs mad e between 0800 and 1600 hours are most costly; use

of the facility between 1600 and 2400 is discounted and use between 0000

to 0800 is further discounted . The rate is also a function of memory,

input , and output requirements. Similarly “batch ” runs are much cheaper

than “demand ” runs. A typical fiv e iteration simulation in the “batch ” mode cost $. 45 and used 6. 848 second s of central processing unit (CPU) time.

A terminal set up for another run which also involved an aborted

simulation run cost $2 . 88. The terminal was connected to the computer

from 1046 to 1132 hours and used 37. 47 second s of CPU time. As an

example of one of the higher costs of running the model, a simulation was

run and both lengthy set up and print out were required . This run was

mad e from a terminal which was on line with the computer from 1411 to

1556 hours. This run cost $24 . 79 and required 65 . 63 second s of CPU time .

50 

-
r r -- 

______________

100

ACCURACY

COMPLETENESS

OVERALL EFFECTIVENESS 90 -

~~~~~~~~~~~~~~~~~~~~

EFFECTIVENESS

THOROUGHNESS 90 -

10

60 . -

0.6 0.I

TASK SUCCESS PROBABILITY

Figure 2-12. Effect of task element success probability on effectiveness elements

and on overall effectiveness.

51

I

_f - 

~

I7I 

- UIIIIIIII
—-.~~~~

-

-—--. -..----_.--,-—,------ - -

-~~ ~~I

_ _ •

E~ ~~~ ~ 

i.

~I

bJ ~ -

~~ ,~~~~~~~~ v

U) -

~ C)

: ~

C\1 U

0 Cl) o I Cl) 4.4

o C)’ Cl)~~~4 C) ~~

I

.2~ ~~~~ ~ ~~ -

~

l c~. Cd cdl • Cd cdl cl) O -

~ ~

.

~ . ,.,

-4-, — IC)

‘‘ C.)

C.) I

C’~ C\~ c.~

C) C) _I 

~ — .0 ‘—4 .0 ~~ cd ~~ cd ~ E-’ U) F-’ Cd

Cd c...~ F’ 0

~~~ ~ ,

~

-, (1) 0 0 U)O 0 C)0 (

4-’ C ) ’ O c~ C) C) 7-” 

~ C) C) C) 7-’

C) ~

- 4C ) ‘-4 0 ~~~~~~~~

~~ 

~~ Cd

Cd

Cd

Q

C4-4 ~~ c,-, ~ I

0 OC ) • . 0 C)I Cl)

~

-4 ‘d’ ‘~S’ ~ 

U) ~., I ‘—4 CO 0

U) ~ Cd 03 ‘d’ ~~‘ --‘ 

~ cdl 0 CO 0

~ U) 

~ U) I o co o ,. — — . • c

~~~ ‘

~~ 

Cl)~~~~ Cd ~~~~

44.4

(

-4 0 0

Cd

Cd C) C) C)~~~ C) C) ~~~~.2

~ ~ C/) 

~~~~~~~~~~~~~~ 

i-I

52

— .- -.~~~~ fl, .Xr .fl ‘ar -
- _ - ---—--~~~~~~~~~~~~•w-,-— — ~~---,~~~~~---,~~~~~~~~~~~~~~~~ - - — - - —

~

---- ——— ___

Small program or file changes typically required only a few

second s of CPU time and cost less than $. 25.

Due to the size of the program, long delays were often experienced while waiting for sufficient computer space to become available. In the case of batch runs, this delay was as high as several

- hours. In the case of terminal runs, when delays exceeded a few

- - minutes, the run was aborted due to the relative preponderance of

telephone line costs compared to computer costs.

Ease of Use -

Although the NETMAN model cannot be called “easy to use ”,

the input is organized in a systematic manner which promotes ease

of use. Some complexities in the preparation of the input data are

described in this section.

An example of the volume of data required is show n in data

prepared for the D-Day simulation run described in the next chapter

of this report . This input required 92 cards including over 600 values.

The NETMAN model requires fixed format entry. That is,

where data are called for , the exact card column(s) or spacing on a

terminal is necessary. Any deviation from the indicated spacing w ill

cause either a discrepancy in the input or, more likely, the program

will reject the input as invalid .

The computer program does not double check the consistency — of the input data. For example, if two networks are specified in

the input , the computer will read in the number of men in each from

the appropriate card columns. If entries in these columns are

omitted , a zero is entered . This will cause the program to terminat e execution once the simulation has begu n and it may be quit e

-: - difficult to trace back to the source of the proble m .

Although the organization of the initial input data to the NETMAN

model can be quite time consuming, changing small amounts of input

is quite easily accomplished in the experimenter interactive mode.

Such variables as personnel characteristics and message characteristics

can be temporarily changed and the effect on output determined . Normaily, a great many variations on the inpu t data are run in order to
r 

r

~ ~~~~~~~~~ :

~~~~~~~~~~~~~ -

-

~~~~~~~~~~~~~~~~~ I

study the range of the simulated system ’s performance. Accordingly, the time for initial input data preparation, when amortized over a

number of runs under various conditions, is not exc essive.

Program Error

Two “errors ” wer e found during the sensitivity tests. These “errors” provide hitherto unknown limitations of the NETMAN program. First, the model cannot be used to simulate a full 12 hour mission as

was initially intended. A 10 hour mission was simulated without prob- lem. Secondly, a message load over 30 messages per hour per referee

cannot be handled. It is believed that both of these limits are due to

the present size of data storage arrays. In extreme cases the program execution is automatically terminated by the computer, probably due

to attempts to make illegal transfers of data. The changes required to

remove these limitations are believed to be relatively minor.

J

j 

-

I

’

54

_____-- ~ T~~~4

-wul
- - ~~~~~~~~~~~ - —----—---—----—-——-. --——— ___ _ __ _ ___ _ _____*t _____ 1

~~~~~

’

!j — _

~

-

~ 

- --

III. MODEL VALIDATION

Field validation is another step in completing the NETMAN development cycle. The purpose of the validation is to derive indices of the

model’s precision in estimating the system and human ettects it was designed to predict. A field exercise management environment from which

appropriate criterion data might be derived, i. e., one employing information processing characteristics similar to those simulated by the netman

model, is essential to such validation. The Marine Corps’ Tactical Warfare Simulation, Evaluation, and Analysis System was considered to be

an ideal system for validation purposes. TWSEAS’ predecessor, TWAES,

was one of the systems from which the NET1VLAN simulation was developed.

This chapter summarizes the planning for and collection of actual

field data suitable for evaluating the validity of the NETMAN computer

model. By way of overview, the validation was based on comparing simulation model results with direct observational data obtained from the

Tactical Warfare Simulation, Evaluation, and Analysis System (TWSEAS).

Data were collected at Camp LeJeune, North Carolina.

Criterion Dat a Attributes

The purpose of the present validation was to compare the model’s

predictions with the performance of an actual system. When such a validation is performed, the actual performance dat a, called criterion dat a,

must be caref ully collected and must represent as close an approximation as possible to the situation that model was built to simulate. The

TWSEAS meets this requirement.

In evaluating the TWSEA S as a vehicle against which the NETMAN

may be considered , the attributes of an “ide al” criterion must be held

in mind. A list of such attributes is presented in Table 3-1.

Criterion Variable s Selected

The eight TWSEAS criterion variable s selected for assessing the

validity of the NETMAN model are shown in Table 3-2 , together with the

associated model variables.

Table 3-3 presents a formulation whereby the relative goodness

of the eight criteria may be determined. It shows a matrix of evaluation

scores for each criterion against each of the criterion aspects which was

identified in Table 3-1. The maximum total score that any one of the

field criteria could achieve is 48 (columns of Table 3-3). No criterion

rece ived this score. The minimum possible score is 12. By this yardstick, no criterion received a minimal evaluation. Transmission delay

55

_________ - - -

~~~~-t.Waa .m.,,n4-.~ e44, -S- -fl-~~~~~~.. - - -,-~~~~~~~- .—

-

~~~ . 

~

r

_

~~~
——- r -

Table 3-1

Desirable Criterion Attributes

Criterion Attribute Notes on Acceptability of Criteria

Data Availability An acceptable criterion is one which is backed by

substantial empirical data for the range of populations

to be modeled .

Data Reliability An acceptable criterion is one for which the error

range of the available data is known and minimum.

Relevance to An acceptable criterion is one which is critical to

Situation and possesses obvious saliency to the acts and

behaviors of ind ividuals and groups involved in the

simulation.

Sensitivity An acceptable criterion is one which will vary as the

result of various conditions expected to be changed 3 -

during simulation runs.

Objectivity An acceptable criterion is one which is based on

objective information.

Uniqueness A preferable criterion is one which is associated with —

uniqu e output variance. Each variable should contribute to the richness and completeness of validation.

Suitability of Form That criterion is preferred for which the available

data do not require excessive transformation, rescaling,

preprocessing, or translation .

Generality That criterion is preferred which is applicable to a

wide range rather than only a limited number of

situations.

Comprehensibility A criterion should be easily understood by the users

of the model’s output .

Utility That criterion is best which is most important to the

questions the planner and model’s user wishes to ask.

Freedom from A criterion which will reflect only minor effects

Triviality should be avoided .

Heuristic Value The comparison of criterion data with model output

should raise questions relative to the mod el as well

as to answer questions.

56 

— —~~~
C) C) C) C) C) C) U) U) U) U) U) U) -.4 -.4

o 0 0 0 0 C)

C C) 

C) C) C) C) 

C)

>4 >4 >4 >4 >4 C) C) C) C) C) C) -

~~~

C 

c

~;~ C C C C C ‘.4 ‘.4 ..-4 -_4

C/) U) 0 -~

~~ C -o -o 

‘

~~ 

-

~~

F-’ C) C) C) C) C) C)

C 

~-.i Cd ~ ~~ Cd ~ ~~ C) U) U) Cl) Cl) U) Cl)

C) Cd ~~~C) Cd Cd Cd Cd Cd

U) 

~

4U) C) 

.~~~~~Cl) C) C) C) C) C)

Cd ~~ ~~~~~~~~~ ~~ 

E

C C) ~ +-~~ C) 

~

C >4 

~ 0 C) - ..i 0 C) . .4 

• .-l ..-4 .4

C 0 F’ 

C) F-” F-’ F-’ F-” F’ — C)

C)

Cl) Cl) (4) Cd _ C) C) U) U)

C) 

~~~

-, 0 U) Cd CC) ~~~ 4-’ C) 0 ~~~~ C ._~ ~~‘

~ ~~- 0. 

,

~~ Cd

-

~~ C)~~~

-. 0 U) Ci)

0 U)~~~ ~~~~~~~~~ 0 Cl)

-~~ 

~~~0 Cd C) 4-’ C)c+-~ 4-’ 0 0 C) C Cd C) C) Cd~~~ ~~ C) U)

~~ 

0

~~~ ~~~~ D 

~1)

~~

.0 

~~~

0. ~~~Cd ~~ C)

C .—4 C) ,-4 — 

C

Cl) Cd ~~ 

~~~~~

-

~~4 

~~~~~~ 

C~~~’ 

r

~~~o ~ 

C) C)c d -.4 0 C ’-’ ~ C C) C C ~> ~~~~~~~ .0Cd ~~~~~~ 0. Cd — 0 ‘ CdCd .0 C~~~ “' 4~ ..4 b~ cC ~-. -, — 4-’ C) 4-’ U) U) C 0 C) C) - Cd ~~ - U) . .4U) 0. F-’ -.-~ Cl) C)~~~ ~~~ ~~ o ~~~ 0 ~-i cC ~~~~~~~~~ CC ~~ o ~ -4-’ ~~~Cd ~~~~ C - cC -i-’ 0 o C)~~~ 0 ‘-~~~~~~ ~~ .4 U) ~,_ , U) 0— I .~~ I — ‘~~‘ -—4 ~-4 ~-l U)

C) o~

~° 

4-’ C) U)

~~ 

0

~~~ cdU ) C) C) 0. ‘

,-I

C ~ ~ — a —‘

C)~~~ 

0

~~.. C) r r C Cd C) 

0 to 0 ~

—

., U) , ~~~~ ~~~~~~ C) 4-’ C

-

~~ ~~~0 ~~~~ ~~~~~~~ C)~~~” 

‘

4.4 C

._4 ~~~~ Cd ., ~~~~~ C C) cC 

0 ~~~~~~ ~‘C ~~~~ -4-’

~~

Cd 

- 

.0 ~~~~ — C C C 

C “ ~~~ Cd ~~~~0. cC~~ - c ‘ ~ ~~~~~~. --‘ ~~~~~~~ c

~ CC Cd Cd

C ~ ~~~o C) C) 

o

’

~ C) C) C) C)

g Z F- ~~~ F’~~ ~

C)

I ~~ .

_ U) ~~~ C) 4-, 

t~ Cl) 

0~

~ ’ 

~‘-~~~

-“-‘ 

C

C c,j Cl) C) c,. ~~~~~ ~ Cl) ~~~C C o C ~ ‘4-’ CI) -—

.C C) C) C) U) C) C$) 

U)

~~‘ ~ C C) ., U) Cl)

C C) C) 

0 Q)

0 0 cC 

_4 C) ~~~~ ~~~Q 

~~~0 U) 0. Cd~~~ C) C .40 ~‘0 C 0 Cl) C) ~-. ,.., c.-~~~~~ O~~~ cC C) ~ 0~~ . C) 4.. cC0. 0 0. ~- F-’ Z 0 F” C) F’

57 

______________________________

- ~~~~~~~~~~~~~~~~~~~~~~
r - --—-—-- —--———--- ---- ~_ ‘

~~

- -

~ 

-- - ~~ - ‘

~~ 

Table 3-3

Criterion Evaluation

Message

Processing Time

Ci)

U)

Cd C — C) C)

CI) -

~~~ s-I CI) ~_, 4_4

C) 0 ~, C

C C) 0 C) C) ~~ C~ C ‘s-’

0 ~~ 0. C)

0 tO U) ~~ 0 CO ~~~ ~ C — C) 0 —l 0 0 Cd Criterion — 0

Cd ~~ ~~ Cd C) - - C

0 U) C) ~-‘ - ‘0 C Cd Attribute 0 C C) 0 C) Cd 0

F’ F’ ~ 0 F-’ ~ C) F’

Data Availability 29 E G F E E E E E

Data Reliability 23 G G F G G G G G

Relevance to Situation 28 E E E E G G G G

Sensitivity 31 E E E E E E E G

Objectiv ity 29 E E P E E E E E

Uniqueness 29 E E G G E E E G

Stability of Form 28 E E P E E E E G

Generality 23 E E E G F F F F

Comprehensibility 31 E E E E E E E G

Utilit y 27 G G E E G G G E

Freedom from Triviality 31 E E E E E E E G

Heuristic Value 32 E E E E E E E E

Totals 46 45 37 45 43 43 43 39

Score:

4 = E - Excellent

3 - G = Good

2 = F - Fair

1 = P = Poor

58

______

- 

-- - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ C_ - _~I
~~-

~~~~~~-- - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

_________ 

—UT —- — --- ‘a - ,

~~~~ - - --

received a low evaluation score largely because it depends on equipment

characteristics and transient effects. Overall effectiveness received

the lowest rating due to its lack of objectivity and stability. Thoroughness and responsiveness received the highest scores.

From the point of view of attributes across criteria, heuristic

value, - comprehensibility, and freedom from triviality received the

highest scores (32, 31, and 31 respectively), while data reliability

and generality received the lowest scores (23 in each case ). The pos-

• sible score range for attributes was 8 to 32 . Most attribute s scored

relatively high in this range .

Model Validation Concepts

The need to validate a computer simulation model was recognized early. Siegel and Wolf (1959) spoke about the need to investigate the properties of new models, to apply a new model to more than

one test, to evaluate the effectiveness and consistency of a model, and

about the desirability and feasibility of validating a model against

outside criterion data. The desirability of model validation by comparison of model results against those obtained through a controlled

laboratory test was also recognized early (Siegel, Wolf , & Sorenson,

1962). The suitability of the Campbell and Fiske (l959)technique for

test validation of models was shown in Siegel, Lautman, and Wolf (1972 ).

Emshoff and Sisson (1970) in a discussion on model validity

concluded: “the only possible evidence of validity for a simulation

model that has been developed specifically for a situation is that the

model has made satisfactory predictions in the past. ” The y suggested five “preliminary criteria for evaluating first time models” as

described by Hermann (1967). These five are identified by an aster isk

in the more comprehensive list of 15 criteria for evaluating a simulation model which are displayed in Table 3-4 . These criteria are not

necessarily mut ually exclusive. Some are overlapping, but all are

considered im portant in some sense and/or for some classes of models.

In order to place these criteria into some perspective and to • - view the sequential steps through which a model might pass, consider

Figure 3-1, which attempts to tie together the various model development/validation phases with these 15 criteria for model evaluation (per Table 3-4). Figure 3-1 displays the major steps (large

rectangles) from concept and model requirements derivation through

the situation in which a model can be considered for decision aiding

and eventually for decision making. The 15 numbered vertical arrows,

represe nting the 15 criteria, show that each step in the process yields

some measure of utility, feasibility, cost, reasonableness or validity.

__________ 

- 

59

- ~ - _ - ~~~~~~~~~~~~~~~~~~~ - - - ~

t ,Oe- _

~~~~~Ua. S4-’ - —

~~~~~ -‘~~~~~ ~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ ~~~
‘

1

Table 3-4

Criteria for Evaluating the Utility of a Computer Model

Criterion - Definition

1. Interna l consistency Extent to which the constructs of the

model are marked by coherence and

similarity of treatment

2. Indifference to trivial aggregation Potential of the model to avoid major

changes in output when input groupings

or conditions undergo insignificant

flu ctuations

3. Correct prediction in the extreme Extent of agreement (correctness of

(predictive or empirical valid ity) predictions) between model and actual

performance at very high/low values

of conditions ~

- 

-

4. Correct prediction in mid range Like above for middle ranges values

(predictive or empirical validity) of conditions

5. Construct valid ity Theoretic adequacy of the model

constructs

6 . Cont ent (variable parameter) Extent to which the model’s variables/ t -

validity (Fidelity) parameters match real life conditions

7. Realism or “face validity ” Extent to which selected content matches

each attr ibute modeled

8. Richness of output Number and type of output variables

and forms of presentation

9. Ease of use Extent to which an analyst can read ily - 

- -

prepare data for, apply, and extract

understandable results from the model

10. Cost of development Value of effort to conceive, develop , L -

test , document and support

11. Transportability-generality Extent to which different systems,

missions, and configurations can be

simulated

60
~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - _______ ____________

Table 3-4 (continued)

12 . Cost of use Value of all effort involving use of

model includ ing data gathering, input,

data processing, and analysis of results f

13. Internal validity * Extent to which outputs are repeatable

when inputs are unchanged

- . 14. Event or time series validity * Extent to which simulation predicts j

events and event patterns

15. Hypothesis validity * Extent to which model relationships

¶ correspond to similar relationships

in the observable universe

* Approaches to validation defined by Hermann (1967)

61

_____________ 

I
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ F-

-7 

____

Q (.1)

~~~uJ * 

_j o, I

S __ 

__ 

• 1

ILl J~~~ 

~~~ 

‘4 ,

‘)4,f’

- —~~~~~t 

_ ~ 

L ~~~ ‘4 

4

~~4~~ 

~~ 

I

~~~~~~ I— () 2

___ 

~~~~~~~~~~~~~~~~~~~~ (I)

4 

0 ~~~~~~~~~~~~~~~~~~ 

u~~

ILl C.!) .f~~~4

)~ CDW 

IL a: _j i— C

--I o ~~~~ -4 CI)

o. LLJ 5

8

~~~~~~~~~~~~~ _ _

~ 

‘

~~~ ;

~4 OZOCL. Lb I— z

( 98

j~~~ ~~~co c, 4 C~) z ,

~ co~~~~~w - W ~-‘

~~~~~ Co F - W W~~~

62 

~

HD

• 

— — -

- 

— — ~~~~~~~~~~~~~~~~~~~~~~~ — 4. -
____

0 —

LI I-

(.) Cl) Li I.LJ ~ 

0

LL cr ~~ ~~~~~.ci

~

L) 0 Li

L. 4

\

\ ~~

-

~~~~‘h~~~ IF— ___

I- cD— -

~~~

~~~~~ 

~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~

g3~~ j

___ ~~~ I ~~ 

>. ~~~ o - ~~~~ z

2:~~~~~Q E A A A ~~~~~3

-

~~~~~~~~ 2 ~ ~

Q t3

~~~ 

I I T

ci W c.~ u z 

~ _______ 

~~

- z

\ ~~3 

Li 

0 ~~~ W 0 C)

z\

~ 

8 ~~~~ 

— 

LiJ

V /N 

‘

/ 

0c~

I- __

/ 

~

—

~~~~~~~~~~~~~ --‘

A

~~~~~~~~~~~~~ 

4

- 

:

~ _ _ ~~~ . 

C,)

<< 0

—Jcn z I- (

~

<ILl 

W I~ J

~~ 2

~ O ld

I-I — - 

Q_

— - 

- -_________ -. 

63

—-~~~~~~~~~~~~~ -

~~~ ,-~~~~•

—- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~~~~~~~~~~~

It is suggested that a model whose design meets the criteria emanating from the model design box be said to be “suitable ” (see lowest

oval). A model which is programmed and debugged enters a state

h er e ca lled “testable. “ After sensitivity testing, (and the implementation of corrections to the model as required), a model is said to

be “reasonable. ” Following adequate validation testing, a model is

termed ~

t

valid hI or “u seable ” for decision aiding and , after the experience of use , a model is “operative, “ “proven, “ or “effective. “ The

various types of data and information required as inputs to each phase

are shown entering from the left with the resulting documentation outputs exiting to the right.

In the present study both the sensitivity testing and validation

testing stages apply. Figure 3-1 shows that the three primary modes

of validation testing are:

• compare model output against real mission

criterion data

• compare model output against expert opinion

• compare model output against laboratory data

An alternate, expensive approach exists for these three modes of

validation for the case in which real mission data acquisition is impossible or too dangerous. That is, the potential to develop two

(or more) simulation models by independent groups of analysts. Validation can be accomplished then by comparing them against each

other or against a single set of expert opinion relative to a preselected

set of criteria. No example of such a case is currently known.

Model Validation

The model validation proceeded in several stages. First, a

number of tests were designed to measure the operator parameters

required for input to the model. These tests were administered to the

personnel who employed the TWSEAS system during the field exercise

on which the validation was based. Second , the TWSEAS performance during the field exercise was measured in order to collect the

data required for the development of criterion information. Based on

the input data describing the characteristics of the TWSEAS personnel

who man the system during an actual field exercise and scenario for

the exercise, the model was run to simulate the TWSEAS performance

during the field exercise, Finally, the output data from the model and

the TWSEAS criterion data from the field exercise were compared for

similarity.

64 

-

~~ — - .

_

~~ fl

—; 

—•- 

~~~ -~~~~~~~~~~ ~~~~~~~~~~~~ -~~

-•

• -- .
,

_ _ ______ _______ — - - -- - -- - —

I

Ind ividual Parameter Data Development

The initial step in the work involved participation in a briefing

with the Marine Corps officers and noncommissioned officers who

were scheduled to participate in the TWSEAS operation during an

anticipated field exercise. In this briefing , the overall scenario and

each participant’s ind ividual responsibilities were discussed . The

group was also addressed by a representative who described the purposes and goals of the NETMAN validation. It was emphasized that the

intention was to collect data to evaluate the NETMAN model and not to

evaluate the performance of these ind ividuals. The participants were

informed that information concerning any given ind ividual and his performance would remain completely confid ential. It is believed that

the acceptance and cooperation of all personnel was achieved. Then ,

the collection of ind ividual data , for entry into the ultimate simulation

of these persons, commenced . A total of 20 TWSEAS referees and radio

operators was involved.

Task Performance Time for Referees

Within NETMAN , simulation of the performance of each task

by the control system personnel simulated is partially based on the

time it requires the personnel to complete the elements of the task.

These data are called t ask analytic information within the model and are

supplied as input information. Because simulation of the characteristic s

of the referees who participated in the field exercise was required , the

basic task performance information required by NETMAN was collected

from the same referees who participated in the actual exercise. During

this data collection, the basic time required to process a message was

measu red . Each referee was given 21 scenarios, one at a time, and

instructed to compose a TWSEAS message using the available information

and the TWSEAS coding book . The scenarios consisted of the actual decoded format for the messages which constituted correct answers. Figure 3-2 show s an example of a problem scenario. Th” referee was asked

to compose each message as accurately and as quickly as he could .

Figure 3-3 shows the data collection form used for recording

the referee task performance information as required by NETMAN.

The times were recorded on the data sheet in the form of cumulative

elapsed time . Timing began as soon as the message was given to

the referee . When the referee stopped leafing through the TWSEAS

message book, the time was recorded under “Message type selection. ”

If the referee changed his mind and moved to another message type ,

an “F” was recorded under “Success/Fail ” and the time of final selection of a message type was recorded under “Touch up.” If the

65

- - ______ - --~ - -~ ~~~~ .
It -

Scenario 1

You are assigned to unit 12 and decide to report the evacuation of casualties by motor transport. It is day D + 2 of the exercise

and the time is 1045. Evacuation of the casualties was requested at

0945, and field medical cards were prepar ed for all casualties at

that time. At 0800, one Marine was killed in action and 16 were

wounded requiring evacuation. The casualty sorting was “above average ” and evacuation procedures were effective.

Figure 3-2. Information provided to referees for composing one

TWSEAS message.

66 - 

- - • • ,

_ ,._•_

& - ~~~~~~~~~—- t

~

- 

—-

,

~~~~~~~~ - - — —~~~~~~~~~~~~~~~~~~~~~--~~ ~~-~~- -~~~ --
- - 

- - 

- --

APS Project NAM E - Task Analytic Dat a Collection for Referee

Identification No. _____________________ 

Date ______________________

Start time _______ ______ 

Message No. _______________

(hours) (mm .,)

Stop time _______ ______ 

Observer __________________

(hours) (mm .,)

Duration Success/Fail Touchup

(secs.) (secs.)

Message type selection _____________

Message encoding ____________

Visual check of entries _____________ - _________

Delivery to radio operator _____________

Comments :

Figure 3-3 Task analytic data collection form for referees.

67

- —I- 

— 1

~

—~ ~~~~~ . - --- ._ _ 

- — — -

- -
————--- -•---- —

referee was satisfied with his initial selection, an “S” was entered

under “Success/Fail” and the “Touch up” time entry was left blank. If the referee selected a wrong message type , the error was not

brought to his attention until the end of the complete data collection.

The time of w riting the last digit of the TWSEAS code was

recorded under “Message encoding.” If the referee changed an

entry, an “F” (otherwise “S”) was entered under “Success/ Fail” and

the time of completing the final touchup correction was entered .

Similarly, the time of completion of the final check of the

message was recorded under “Visual Check. ” This end point was

taken to be the point at which the referee looked up from the message

or the time at which the referee finished writing the time of message

completion.

The timing ended when the referee returned the completed

message and the scenario to the test administrator. This time was

recorded under “Delivery to radio operator .”

During the message preparation, the referee was not allowed

to interupt for questions and the timing , once started , was carried

through to completion. Any delay in the message handling process,

such as dropp ing a pencil , was considered to be a possible occurrence in the field and was included in the timing.

Mean time, across referees, was calculated for each task

element of each message type. The message types are described in

Table 3-13. The resulting means and standard deviations are shown

in Table 3-5. Currently, NETMAN is limited to a total of eight task

analyses in one simulation. Since at least one task analysis must be

assigned to each level of the system, three are accounted for by the

controller, computer, and radio operator. This allowed an allocation a maximum of fiv e task analyses for use by referees. Message

types 5 and 2~ as shown in Table 3-5, were omitted from this simulation. This omission seems justified on the basis that no messages

of type 5 and only two messages of type 2 were ultimately involved in

the field exercise. Utilization of “Success/ Fail” data is presented

below in the discussion of precision determination (see Table 3-9).

Task elements 1 through 4 in Table 3-5 correspond to the fou r tasks

in Figure 3-3.

68

____________________ 

‘

~~~~~~~~~~~~~~~~~ , 

I
- Table 3-5

Referee Task Analyses

Message Task Mean Task Duration Standard Deviation

Type Element (sees.) (See s.)

-

~~

1 1 45.0 40.0

t • 2 117. 1 4 1.4

3 5.8 9.5

4 3. 1 3. 2

2* 1 37.8 23.0

2 137.5 74. 9

3 5.2 9.6

4 3.0 3.9

3 1 39.4 21.8

2 135.4 70.1

3 5.7 9.4

4 2.2 2.8

4 1 44.9 18.8

2 203. 2 75 .9

3 2.4 4.2

4 3.1 3.4

— 5* 1 43.0 29.9

2 161.7 43.2

3 2.6 3.4

4 2 .3 2.6

6 1 38.2 14.3

2 153.3 58.2

3 6.0 12.5

4 7.0 10.7

7 1 34.9 10.5

2 182. 3 39 .5

t 3 5.6 12.6

4 4.7 7.4

* Omitted from simulation.

~~ 69

~

- _______________________________

. -- 

~ t tt~~i~ 
r ~~__ . .,~~~_ 

~~~~~~~~~~~~~~~~~~~~~~~ —-—-—---~~ ---- —

_____________________________________________________________________________

Task Performance Time for Radio Operators

NETMAN requires the same type of task anal ytic data for radio

operators as for referees. The procedure for collecting task element performance time relative to radio operators paralleled the procedure followed for referees. Each radio operator was sequentially given 21

messages and was asked to transmit the message using a DMED (a

message transmittal device) box . As for the referees, the time to complete variou s message segments, as required by the NETMA N , was recorded for the radio operator using the form shown in Figure 3-4.

Time was recorded in the form of cumulative elapsed time. Timing started whe n the radio operator first received the message. When the radio operator put the message down or placed it in posi- tion on the top of the DMED box, as some of them did , this time was recorded under “Visual inspection of message. “ If the radio opera- tor decided to revise the message, an “F” was entered under “Suc~ cess/ Fail ” and the revision time was entered under “Touch up. “ When

the radio operator finished dialing the final digit of the 22 digit TWSEAS

message, the time was recorded under “Visual check. “ If the radio

operator went back to change any dialed in digit , a failure was recorded and the final time for completion was recorded under “Touch up. “

Alter setting up the DMED box, the radio operator set the radio

to the proper transmittal frequency. The time of completion of radio

set up, usually indicated by reaching for the headset, was recorded under “Radio set up. “ After setting up the radio, the radio operator pressed a button to transmit the message and then waited for a beep signal which indicated acknowledgment by the computer . In actual operation, a message is transmitted and acknowledged twice . However, there was no actual computer acknowled gment during this data

acquisition pe riod. The radio operators were told to wait a few seeonds---as if they we re waiting for an actual acknowled gment. The

time at which they completed the transmittal was recorded unde r “tran sm ittal.

The final task to be performed by the rad io operator was

recording the message information in his log book . The time of

completion of this recording was entered under “Storage of message” and this entry re presented the final time of the message processing.

In the case of the radio operator , only one task analysis was used in the ultimate simulation. The mean and standard deviation

___________ — -- -— - -• 

70 

- - - 

—

—&~

.I.A~~
:

~

- 

~~~~~ 

— — - —- —---- — — ~~~~~~~~~~~ - - .-•----—--

~ -,-----.----- •

- —--— -

APS Project NAME - Task Analytic Data Collection for Radio Operator

Identification No. - Date ____________________

-: Start time _______ ______ 

Message No. ____________

(hours) (mm .,)

Stop time _______ ______ 

Observer _______________

(hours) (mm .)

Duration Success/ Fail Touchup

(secs.,) (sees.) —

Visual inspection of message ______________ ______________ _________

Visual check _____________ _____________ _________

Radio set up ____________ ____________ ________

Transmittal _____________ _____________ _________

Storag e of message ____________ ____________ ________

Comments:

I

Figure 3-4 Task analytic data collection form for radio operators.

71

~

- - - - - ~~~~~~~~-- -—~~~~ —--____________

L -
- --- -.- •~~~~~~~~•-~ -— ~~~~~~~~~~~~~---~~~ -

~~~~~~- - -- -

~~ - ---- ~~~~—--~~~~~~~~~~~~~ - -~~~~~~~~~~~~-~~~~-- -

of each task element, as derived from the above described test data ,

is shown in Table 3-6. These times and standard deviations were

used during the validation simulations for all message types.

Other input information required by NETMAN for each person

included in the simulation included the stress threshold , level of

aspiration, a speed (proficiency) factor, and a precision factor. The

ba sis f or these data is desc ribed below .

Stress Threshold Determination

Within NETMAN , an ind ividual stress threshold is employed to affect the success experience and response time of each person

simulated. To derive the str ess threshold for each individual, a

mirror tracing task was employed . This task, due to the fact that the

visual feedback is disorient ed by the mirror reversal, is stress

producing for most people. Stress was measured through use of

electrodes attached to the fingers of the nonpreferred hand while

the preferred hand was used in the mirror tracing task. This test

requir ed approximately 20 minutes to administer.

Skin conductance was measured through the use of an Autogenics Model 3000 Skin Conductance Dermograph. After a practice -

trial, the conductance level change was measirred throughout a formal test.

The maximum change in skin conductance level was converted

to a stress threshold for each referee and radio operator through use

of the following equation:

STRM - - .0909 STR + 2 . 9091

Where STRM = stress threshold

STR = stress level chang e in skin

conductance level

This equation was derived by translating the range and direction of conductance level changes derived from the test data to the

stress threshold range as scaled within the model. The result was

a linear transform with a high threshold reflecting a low conductance change. The resulting stress thresholds are shown in Table

3-7. 

72

- - - .- -- fl.— — r —‘-— 
I

Table 3-6

Radio Operator Task Analytic Time

Mean Task Duration Standard Deviation

Task Element (sees.) (sees.)

Visual Inspection 8. 3 7. 4

Visual Check 43.0 14. 0

Radio Setup 7 . 6 7. 3

Transmittal 11 . 1 6. 1

Storage 38.0 11.3

I Table 3-7

Str ess Threshold of Each Referee and Radio Operator

Subject Skin Conductance

Identification Level Change Stress

Number Threshold Referees

1 8.10 2.17

2 3.83 2.56

3 4.58 2.49

4 9.67 2.03

5 7.52 2.23

14 4.79 2.47

15 5.12 2.44

16 8.83 2.11

17 10.00 2.00

18 4.75 2.48

Radio Operators

6 6.92 2.28

7 7.75 2.20

8 3.00 2.64

9 8.58 2.13

10 3.18 2.62

11 4.33 2.52

12 5.67 2.39

13 3.83 2.56

; 19 2.96 2.64

20 10.00 2.00

73
.- .—.- —. — ,.— -—.-.-~~•-. ~- — --- .--••---—-•.-------- - •L.~ ----~

.

~ —

Level of Aspiration Determination

Level of aspiration is a variable within the NETMAN which

affects each simulated person ’s response to success or failure. Level of aspiration is generally an indication of an individual’s need

for achievement. It determines the level of success which he antic - ipates and strives towards. Within the model , the effect of level of

aspiration is a somewhat complex function of the stress level present, the individual ’s “Success/Fail ” history, and his initial level of aspiration. The result is an effect on success probability. )

To obtain an estimate of the level of aspiration of each of

the persons to be simulated , each referee or radio operator was

told that most persons can mark 30 “x’s” in a matrix of square boxes

in 30 second s. He was then asked how many he could do in 30 second s.

To derive the required level of aspiration, the following

equation was employed.

- Estimate Level of Aspiration = 30

The upper limit of level of aspiration is set at 1. 0. Table 3-8 presents the resulting level of aspiration indices.

Individual Precision Determination

Operator precision is another individual input required by the

NETMAN model. Precision is used within the model to affect the number of mistakes which are made by a simulated operator. Table 3-9

shows the number of errors made by each referee and radio opera- tor in the message preparation (described earlier). The smaller

number of errors mad e by radio operators is probably a function of

the more basic task performed by radio operators. The mean number of errors was calculated as a percentage of the number of —

possible errors. In the case of the referees, there were four task

elements which might be failed and 21 messages. ThiE means that

there were 84 possible errors and , since 11 errors resulted on the

average, the overall probability of success across all referees was

.869 . For the radio operators, there were fiv e task elements resulting in 105 possible errors. The average radio operator success

probability was . 984. The success probability of each man was

calculated in the same way. The ind ividual precision factors were

calculated as shown below .

— .._ _.._c—-_•-_-.—.----.- -- —

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
_________ - ——--- — — — -•--- -— —.- -•--- — .— — — —

t 

__

Table 3-8

Level of Aspiration of Each Referee and Radio Operator

Subject

Identification - Level of

Number Referee Aspiration

1 1.00

2 1.00

3 1.00

4 1.00

5 .67

14 1.00

15 1.00

16 1. 00

17 1.00

18 1. 00

Radio Operator

6 l. 0f

7 1.0

8 - 

-- 9 1.

10

11 1

12

13

19

- 1 20

75 

- 

. I

~~~~~~~~~~~~ 1~_ .ZT ~~~T . - _______ ~~~~~~

- ‘ ‘

~~~ 

-
- ~ - — - ~~~~~~~~~~-- -~~~~~~~~~~~~ ,

~~~~~~~~ -- --~

-“

For individuals with fewer mistakes than the average :

PREC = 1 - 

[

PROB - AVPRB )/[5 (1 - AVPRB)]]

For individuals with more mistakes than the average :

PREC = [6- (PROB/AVPHB)]/5

Where PREC = individual precision factor

PROB = individual success probability AVPRB = mean group success probability

The se equations are the same as the equations which determine the effects of precision within the model. Precision within

the model is scaled the same as speed. A 1. 0 precision level is the

nominal condition and has no effect on success probability. A precision level of. 9 produces a highe r success probability and fewer,

• errors. A precision level of 1. 1 produces reduced success probabilities and more errors.

The resulting precision factors are shown in Table 3-9.

individual Speed Factor

W ithin NETMAN , performance time for various task elements is partially based on a speed factor for persons simulated.

This speed factor is another required input .

The speed factor input for each referee and radio operator

was calculated on the basis of individual message processing time

in comparison with the overall message processing time for the

respective group. Table 3-10 shows the mean time for each referee

and each radio operator to process a message. These times are

means across the 21 message s processed by each man. The time

for the referees ranged from a low of 118. 2 seconds to a high of

261 . 4 seconds. The mean time to process a message across all

referees was 187. 5 seconds. The formula used to compute each

referee ’s speed factor, F, was:

F- Individual Mean Message Processing Time

• 

- 

Group Mean Message Processing Time

For the radio operator, processing time ranged from 85. 1 seconds to 130. 8 second s with a mean of 108 . 7 seconds. The speed factor

for each radio operator was calculated in the same manner as for referees.

76 

—--—-- - ~~~ -- ~~~~~~~~~~~~~~~

— ,

~ i_ _ _ ~

- i - ~~~

-j
- _______________

- ~~~~~~~

- - - - -- -

~~~~~~~~~ 

- Table 3-9

success Probability and Precision of Each Referee and Radio Operator

Subject

Identification No. of Success Operator

Number Errors Probability Precision

Referees

• 1 1 .988 .818

2 9 .893 .963

3 9 .893 .963

4 10 .881 .982

- 5 12 .857 1.003

- 14 24 .714 1.036

— 15 7 .917 .927

- 16 17 .798 1.016

17 14 .833 1.008

18 7 . 917 . 927

- Average 11.0 .869 1.000

Radio Operators

- 6 1 .990 .925

7 5 952 1 007

- -‘

~~-:- 8 0 1.000 .800 - 

-

~~ 9 0 1.000 .800

10 0 1. 000 . 800

- 11 4 .962 1.004

12 1 .990 .925

13 3 . 971 1. 003

• 19 2 .981 - 1. 001

20 1 .990 - 

. 925

- Average 1.7 .984 1.000

-ç 

-

~~~~~~~~~~ 

77

rr ‘c ~- ~~~~~~~~~~~~~~~~~
_________ ~~~~~~~~~~ -

~~~~~~~ -.

-

~~~~~~~~ ~~~~~~~~~~~~~ - “ 

- - - -- -

____________________________________________ ______.______.•__,-__=_.___, ‘—if- -- - — -———--- -- --— - _ - -

~~

—

_________________________________________ 

‘

I

—

Table 3-10

Mean Messages Processing Time and Speed Factor

for Each Referee and Radio Operator

Subject Mean

Identification Processing Time Speed

Number (sees.,) Factor

Referees

1 185. 3 . 99

2 207.0 1.10

3 132.4 .71

4 186. 1 .99

5 205.3 1.09

14 220 . 8 1. 18

15 134. 9 . 72

16 261.4 1.39

• 17 223.2 1.19

18 118.2 .63

Average 187.5

Radio Operators

6 110.3 1.01

7 126.3 1.16 — -f

8 98.2 .90

9 115.3 1.06

10 105.5 .97

11 111 .0 1.02

12 130.8 1.20

13 85. 1 .78

19 106.2 .98

20 97. 9 . 90

Average 108.7

78 

_____________

)

- - - - — - - ——-- ---- —~~~~-
—

The resultant speed factor , for each referee and radio

operator is shown in Table 3-10.

Background Information

All personnel tested were asked to complete a background

data form. The purpose of this form was to collect information

about the experience and background of the referees and the radio

operators involved in the field exercise. The referees included

fiv e first lieutenants and fiv e sergeants. Their time in the Marine

Corps ranged from 31 months to 174 month s. The mean service

time in the Marine Corps was 84 months. The level of education

was four years of college for all of the lieutenants and high school

graduate or equivalent for the sergeants. Only one of the refer ees,

a sergeant, had worked with the TWSEAS system before.

The radio operator group consisted of one sergeant, seven

corporals, and two privates, first class. Their Marine Corps

• service ranged from 18 months to 102 months with a mean of 38 —

months. Eight of the 10 were high school graduates; one did not

graduat e from high school, and one had one year of college. Only

one radio operator, a private first class, had worked with the

TWSEAS system before.

Message Frequency and Frequency of Message Type

, 

~ Message frequency is a nonpersonnel input information requirement of the model. This input is the mean number of messages

generated per referee per hour and its standard deviation . It is

presently required that these numbers be entered into the model in

- - integer form (i.e., no decimal point). Table 3-11 show s the number

of TWSEAS messages which were actually tracked during the field

exercise of the validation process.

These data were employed to provid e the input information

necessary for simulating the field exercise message frequency. The

frequency of each message type in the field was computed so that

this required input information could be used in the model ’s simulation of the field exercise. Table 3-12 shows the actual frequency

of each of the listed TWSEAS message types over two day s of the

field exercise. These data form~d the basis for the message type

input requirement of the simulation aspect of the validation process.

&

I 

________________________________ 

_________

-—_ ~~~~~~-- - - — - I- - 

-~~~~~~~

~~~~~~~~~~~~~~~~~~~~~~~
Table 3-11

Field Message Frequency per Hour

Hour Day l Day 2

1 4 4

2 2 3

3 4 7

4 0 4

5 1 4

6 2 4

7 8 4

8 3 3

9 3 3

10 1 1

Total 28 38

Table 3-12

Message Type Frequency

TWSEAS

Message

Type Frequency Description

7 1 Scenario Event Report

10 1 Personnel Replacement Report

11 11 Personnel/Equipment/Small Arms

Ammo Initial/ Update

30 31 Situation/ Location Report

31 4 Planned Move Report

32 5 Contact Report

35 3 Attach Plan Report

36 4 Defense Status Report

38 2 Attachment/Detachment Report

41 1 Casualty Assessment Report

42 1 Casualty Evacuation Report

80 2 Assault Wave Landing Report

80 - 1

_ _-- -- - - -- ~~~ -- -

~~~~~~~~~~~ -~~~~~~~~~~~ •- ---- -~~~

__--- 

.

_ -

~&
To reduce the number of message types to the model ’s capacity,

the number of message types was collapsed as shown in Table 3-13,

which show s the correspondence between the 12 TWSEAS message types

(Table 3-12) and the five message types used in this model application.

Computer Runs Performed

The input data , as derived in the prior sections, were entered in

the required format for simulation within the NETMAN model. One run

was organized to simulate TWSEAS operation during the first day (D -

• day) of a battalion level field exercise and another run was complet ed

to simulate D - day plus one. Ten hours of operation were simulated

each day to provid e a total of 20 hours of model predictions which could

be compared with TWSEAS performance.

Observer Training

Seven Marine Corps observers collected the required TWSEAS

performanc e information. The group was composed of four Marine

Corps’ Second Lieutenants, one First Lieutenant, and two retired Marine

Corps’ Sergeants Major .

The observers were trained during the two days prior to the

exercise. The training included a briefing on the overall objectives

of the present work , practice in filling out TWSEAS messages, and

practice in timing simulated referee tasks and radio operator tasks. The observers completed the same 21 TWSEAS messages which were

used to test the referees. The results were reviewed and errors were

corrected . The observers were instructed on the behaviors which signaled various aspects of the message processing. For example, referee

message processing started when the referee exhibited an initial behavior which involved the message. Frequently, picking up the TWSEAS

message code book was the starting point ; but , sometimes, a referee

filled out part of the message, such as unit identification and time, before picking up the TWSEAS booklet. The observers were also instructed to comment on any unusual occurrences.

After thorough discussion of the behaviors which formed the basis

for timing and how to complete the data collection form , the observers

received practice in form completion in a simulation situation. During

• the simulation, the observers recorded the appropriate data. Approximately 10 messages were involved . The data forms, as completed by

each observer, were individually reviewed for accuracy and completeness. Further instructions were given, as required , unt il each observer was proficient in all aspects of the data collection.

81
I

Table 3-13

Consolidation of Message Types

k

Mes sage Consolidated

Type Description To Type

30 Situation/ Location Report 1

41, 42 Stream messages - request and delivery. 2

7 , 31, 32 Stream messages - ground engagement. 2

35, 36 Other messages requiring controller ac- 3

tion and/or complex encod ing by the

referee, with a high probability of occurrence.

None Other messages requiring controller ac- Not Used

tion and/or complex encoding by the

r efe ree, with a low probability of occurrence.

10, 11 Other messages requiring no controller 4

action and normal encoding by the referee, with a high probability of occurrence.

80, 38 Other messages requiring no controller 5

action and normal encoding by the referee, with a low probability of occurrence.

82
I _ _

Field Exercise

The logic for the validation process called for comparing the

results of a field exercise with the NETMAN predictions of these results when the performance of the same personnel that are included

in the field exercise is simulated. Prior sections described the

collection of the required personnel descriptive data for the model.

The present section describes the field exercise which provid ed the

basis for the results comparison.

In the field exercise , a full Marine Corps battalion landing team

assaulted a beach with armor and mechanized units. The objective of

the exercise was to rescue an abducted member of congress. The details

of the exercise included numerous searches, engagements, and move- j ments. The exercise started w ith an assault on a beach and terminated with the withdrawal of troops back to the ships. The exercise

was observed for two successive day s (10 hours each day). The

withdrawal took place on the third day and was not observed or simulat ed.

Field Exercise Data Collection

Two types of field exercise data were collected. The first

type was provided by observers who traveled with mobile units and

collected data concerning referee and radio operator message processing. The second type of field data ~as collected at the control van

and concerned the computer and controller message processing.

The observers assigned to the control van were one psychologist and one ex-army Captain.

Figure 3-5 shows the field data collection form used by all

observers to record data. The identification number was used to relat e test data to field data. Since the same form was used for referees,

radio operators, and controllers, the position entry was required .

The TWSEAS message type and the time entered on the message were

entered to allow tracing the message throu gh the system - - from the

referee through the control van . The times entered were the message start and stop times. The delay for transmission was entered

only for radio operators. Comments of all types were included to

promot e greater insight into the functioning of the system.

~~~ To provid e a measure of overall effectiveness, the controllers

were asked to rate the overall system information handling effectiveness

83
r

APS Project NAME - Field Data Collection Form

Identification No. _______________________ Date

Position (R , RO, or C) _________________ 

Message No. ___________

Unit ______________________ Observer _______________ • 

4

Message Type (TWSEAS) _______________

Message Time Tag ___________________

Clock Time

Start ________ _______ 

Stop ________

(hours) (mm .) (hours) (mm ..)

Elapsed Time ______________ 

Delay for Transmission ___________

(s ee.) (sec .)

Comments:

-

Fi~~ re 3-5 Field data collection form
on a magnitude estimation scale. Referenc e points and an example

were provided on the effectiveness rating sheets.

f

Discussion of Field Exercise Data

When selecting the exercise for use as a validation exercise,

the representativeness and appropriateness of the exercise were carefull y considered. While the field exercise was fully realistic, the

statu s reporting system (TWSEAS) was limited by radio failures, insufficiently trained (in the use of TWSEAS equipment) personnel , variations in message handling procedures, and the relatively small number of messages measured in their entirety.

Radio problems, while send ing the TWSEAS messages, provided many delays, changes in procedure , or attempts to solve these

problems. Radio operators changed batteries, changed antennas,

moved their radios to other locations, and used the voice frequency

to verbally relay the TWSEAS messages. Often , the temporary radio

transmission problems caused the almost complete cessation of attempts to send TWSEAS messages. Other techniques~ such as messengers and voice relays,were substituted . These non TWSEAS techniques

are not simulated or accounted for in the present structure of the

NETMAN model.

Additionally, deviations from prescribed procedures occurred

on a number of occasions. For example, occasionally a referee

would compose a message on the DMED box and transmit it himself.

Such deviations in procedure are not simulated within the NETMAN.

Such problems are to be expected in any quantitative data

collection effort which takes place in a real field situation. Neverthe-less, the affect of such problems on criterion data quality may

not be trivial.

_ 

85 

___

k
Validation Analysis

The agreement between the model’s predictions and the TWSEAS

performance during the field exercise was examined according to the

eight criterion variables described earlier. Agreement was measured

for the followi!lg measures as computed by the model: thoroughness,

responsiveness, overall effectiveness, total message processing time,

referee message processing time, radio operator message processing

time, controller message processing time, and transmission delay.

These eight aspects will be examined individually in turn.

Thorou ghness

Thoroughness within the model is calculated as the number of

messages completely processed divid ed by the number of messages

available to be processed in each hour. The criterion data were highly variable and responsive to factors which are not simulated within

the model. The data from the field were therefore considered to be

a range of data into which the model’s output should fall. A tally was

made of the number of times that the model predictions fell within one

standard deviation of the field data or within one half standard deviation of the field data. There were 20 comparisons based on 10 hours

per day for two days. Out of the twenty comparisons, nineteen of

the models hourly predictions fell within . 5 standard deviations of

the field hourly data. According to the sign test (two tailed), this agreement ratio (19 out of 20) is statistically significant at the .001 level.

Since the thoroughness index is fu ndamentally a measure of

network throughput, this finding seems particularly important. System planners and evaluators are interested in throughput because of

the obviou s relationship of throughput to system efficiency.

Responsiveness

Responsiveness, within the NETMAN model , is calculated as

the ratio of human message processing time to total message processing time. Human message processing time includes the sum of

the referee, radio operator, and controller time spent working on a

message. Total message processing time includes human time plus

other time such as delays or waiting time. Model responsiveness

predictions fell within one standard deviation of the actual TWSEAS

hourly performance data 18 times out of 20. This agreement ratio

is indicated to be statically significant by a two tailed sign test at

tLe .001 level of confidence.

86

A
I

I

The indication of validity for this responsiveness index

will be of value to system designers, planners, and users because

it provides a basis for knowing whether or not the human is contributing most to system throughput time. In a sense, the responsiveness index complements the thoroughness index . The latter provides

a time oriented measure while the former prov ides a mes’

~

age load

oriented index . Quite obviously, to gain insight into the reasons fo~ a high or a low responsiveness ind ex, the user will want more detailed , valid information on individual processing time . The validity of such data, as yielded by the model, is discussed later under

the Referee Message Processing Time, Radio Operator Processing

Time , and Controller Message Processing Time headings.

Overall Effectiveness

Overall effectiveness, within the model, is calculated as a

composite of fou r individual effectiveness components - - thoroughness, accuracy, completeness, and responsiveness. The relative

weights for each component are entered by the model’s user. In the

simulation of the TWSEAS, equal weights were assigned to each component . The overall effectiveness equation also takes into account the

correlations between the effectiveness components. The values entered to describe these intercorrelations were all 0. 5.

For the TWSEAS performance criterion, the opinion of the

controllers was used as the measure of overall effectiveness. One

problem with this criterion is that the controllers based their evaluations on the performance of the total complex while the NETMAN

simulation only included those messages which were successfully

transmitted using the normal TWSEAS system.

The NETMAN model provides hourly data for overall effectiveness. The model’s predictions were compared with plus or minus one

standard deviation of the TWSEAS criterion in a two tailed sign test.

The result was not statistically significant. The controller evaluations

of overall system effectiveness were substantially lower than the model’s prediction. This lcwer rating by the controllers may be due to

t the presence of equipment and radio transmission difficulties, which

were bypassed through the use of alternative delivery methods. These

difficulties were not simulated in the model. Only messages which

are able to pass through the normal TWSEAS channel are considered

in the simulation.

Alternatively, the weights which were assigned to the four

effectiveness components may have not adequately reflected the subjective values of the controllers. Or, the assigned intercorrelations

may have been erroneou s.

87

~~. 

-

~ 

-—-
Total Message Processing Time

Total message processing time begins when a referee starts

working on a message and end s when the controller finishes working

on the message. Total message time includes referee working time,

radio operator working time, and controller working time as well as

all delays and interruptions which occur between working times. The

measurement of time required to process a message completely is

possibly the best single measure of system performance. Total message processing time was obtained for each hour of the TWSEAS operation. Total message processing time is also an output function from

the NETMAN model. The TWSEAS criterion data were compared with

the NETMAN output by means of a matched pairs t -test. The resulting t value was . 381. With an N of 20, this value is not statisically

significant (i. e., the model versu s TWSEAS time difference was not

significantly different). This find ing supports a contention of reasonable correspondance between predicted and actual TWSEAS total message processing time. As a result, confidence in the model’s ability

to predict total message processing time is significantly increased;

and , since total message processing time is an especially important

aspect of the simulation, confidence in the use of the NETMAN model

can be increased .

Operator Processing Time

During the field exercise, the time spent working on each message was measured for each ind ividual (of those who were under observation). These data were averaged across operator type to produce

mean time per message per hour for each operator type (referee , radio

operator, or controller). The NETMAN produces parallel data. Independent comparisons were performed for referee processing time,

radio operator processing time,- and controller processing time. In

each comparison , the frequency of the NETMAN prediction data falling

within plus or minus one standard deviation of the mean of the TWSEAS

observational data was determined . In each case, 20 out of 20 of the

model’s predictions fell within one standard deviation of the TWSEAS

criterion. For each position, the two tailed sign test was statistically

significant at the . 001 level of confidence. This finding indicates considerable agreement between the NETMAN’s predictions and the TWSEAS

time criterion at all levels of message processing. The agreement between predicted and criterion time is especially important since modeling manipulations of operator time (for example, by varying operator

proficiency level or through task revision) can provide important information to system designers concerning the required number of personnel

and the required training of personnel in new systems. Such simulations

88
would also provide information concerning the workload imposed on

personnel, and, therefore, the availability of such personnel for ad-

~~~ ditional responsibilities. Finally, the individual time data may now

be employed as a basis for gaining additional insight into the responsiveness measure, as described above.

Transmission Delay

• Transmission delay in the TWSEAS is represented by the time

between a first attempt to transmit a message and the time record ed

by the computer as the time of message entry. In a “perfect ” communication network, this time would be nil. Within NETMAN , transmission delay is defined similarly. Transmission delay can occur for

a variety of reasons. Equipment problems are one of the most common.

Another reason for delay is that a line of sight radio transmission frequently induces difficulty when used under less than ideal conditions. A

third , less prevalent, reason for transmission delay in TWSEAS is line

or channel congestion.

Only one overall transmission time is available per day from

the NETMAN computer model. This one prediction time was compared

with the data for each hour from the TWSEAS criterion. Comparison

of the predictions of the NETMAN model with the TWSEAS performance

indicated that the model data were within .5 standard deviations of the

criterion data in 19 out of the 20 comparisons. This agreement is

statistically significant (sign test) at the . 001 level. Such a finding is

consistent with the agreement between the model’s prediction and overall processing time in the TWSEAS. Since transmission delay is an

- r - ingredient of overall message processing time, one would not expect

-? agreement in the overall unless the subaspects also agree.

The NETMAN model does not simulate equipment problems as

such. NETMAN only simulates the resulting time delays. Accordingly,

the obtained level of agreement is especially encouraging.

Integration of Validation Data

Figure 3-6 shows the correspondence between the NETMAN

model’s predictions and the TWSEAS data in terms of deviation from

the TWSEAS data. The “x ’s” represent the mean model predictions

across all hours, while the mean and standard deviation of the horizontal axis represent the mean and standard deviation of all field data

points. Two of the predictions agreed almost exactly with the respective criterion- -thoroughness and total processing time. Five of the

predictions - - responsiveness, referee processing time, radio operator

processing time, controller processing time, and transmission delay ,

89

:-

~~~

-

~~~.

- 

~~~~~~~~
~~~~~~~~~~~~

_ _

were within one standard deviation of the mean of the field data. Only

one of the indices investigated, overall effectiveness, was more than

one standard deviation from the mean of the field data. In this case, the

deviation exceeded two standard deviations. As was indicated , there

was a lack of correspondence between the TWSEAS overall effectiveness

as rated by the controllers and overall effectiveness as measured by

the model.

Table 3-14 summarizes the statistics and results of the eight

validation analyses. The sign tests were all structured to test the

degree of agreement between model and field data. Agreement to

within one standard deviation was found in all cases except for overall effectiveness. A t -test was performed to compare model and field

total processing time. No statistically significant difference was found

between model predictions and TWSEAS data in this case.

Discussion

The results of the validation support contentions favoring the

validity of the NETMAN model relative to all but one of the diverse

TWSEAS criteria employed . The degree and magnitude of the diverse

events involved in field exercises in which a TWSEAS like system is

involved brings out clearly the need for computer models in this area.

Where variable occurrences overshadow design variables, the need for

improvements is very hard -to evaluate . Each field exercise, by its

nature, is expensive and provides only a small sample of possible conditions. Within a model, however, the effects of such events can be

assessed by combining the effects of many low probability events

through stochastic iteration.

The criterion quality is believed to be reasonably acceptable.

One problem was the limited amount of data , i. e., the number of

messages that the referees were able to transmit from the field over

the normal TWSEAS system. The small message sample size represents the primary limitation on the adequacy of the criterion data.

The measurement of the TWSEAS performance, as performed

by the field observers, is believed to be highly reliable. Although no

formal independent measures were available, the use of discrete time

points and message identifiers when coupled with the training that was

administered, allow a high degree of confidence in the reliability of

the data. The exercise data , as collected , were highly objective. Discrete, predefined starting and stopping points were identified for time

measure ments. Subjective evaluations were used. only in the case of 

the controller 4 ’s overall effectiveness evaluation.

90

_____ __ _ L ,

~~~~~

. 

~~~~~- - 

~~~~~~~~~
-

~~~~~

-- -

~~~~

Field Reference

SD -3 -2 -i MEAN 4-1 +2 +3 SD

Thoroughness ‘ X

- S I

Responsiveness X i

• I I

Effectiveness , , X

Total Time X

Referee X I I

Radio Operator I x I I

Controller x I I

Trans. Delay i X s

— - - I I

, S

S I

‘4

Figure 3 -6. Model predictions versus field data. 

—- -. - --

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

L

I . —— 

- ~~~~~~~~~~~~~~~~ . . —
* *

* * * * * *

a) --4 0 -

~~~ ~~~~ 

__

4.- ~~~. -

~~ -

~~~ (I,

bZ~~~~~~

~~~~c d O ~

J

~

1 C)

--4 4

4-. .-’ 4 1

cdl

•~ 

a —~

-

~~~ ~~~~ bil — C) ~

)

U) .4] ~~~~~~~~~~~~~~

C.) Cl) p a)

a) ÷~~~a) --I .4-. c4_4

Cd

4..) 

~~ a) (ID 4-? U)

b

~ ~O 4-’ bi) bfl b o ~ ‘~~ ‘~~ I ~~ a) ~ Cd Cd

~~ ~~ E~ 

C/) Cl) (ID Ct) Cl) C/) C/)

C~

) Cd ~~~cd a) Cd -

~~ 

U)~~._ 4_. a) -.4 Cd Cd — -

~~ a) C.) cd ~~~~~~~~~~~~~~ - Cd

C~.4 a) .

~~~ o U)

U) ..4 a ) ( )

E • -

Cd ~~~~~~~~~~~ ~ 4-’

- ~-I U’ Cd a )4- ’

C” ~~ - ~ Cd — ~ .

~~~ U’ C) C/) U) C) ~~~~~~~~ ~~~~~~~~~~~ U) U) Cd •-s U) U) a) 0 C’) U) C) C) C) ~~

‘ U’ C) ~~ a) ~~~~~Cd 0 —l 

~ 0 a) b~ Cd 4-’

E ~ Cd ~~ Cl) ~~ .

~~ .C~ ~~~~~~~~~~

Cd U)~~~~~~~ Q~~~~~~~U ) 0 C)

~~ 

-

~~ E .~~~ ~~~

— ~~

. C) Ci) ~ Ci) — ~‘ c. -—‘ * * * Cl) —

— ~ ~ o ~c. ~

~ a) C) O C) o o Cd ~ ~ o~~ ~~~ CI)

Cd a) “ o ~~~

~~ C) 4.? C4..~ Cd

C) ~ 0 C) Cd 0

0 F’

92

______ _____________ ___ 

(‘

I

~~
TWSEAS performance data required little or no transformation, rescaling, preprocessing, or translation in order to be used as criten a . The calculations and processing described in the early part of

this chapter were involved in the calculation of input data for the mod -

I el, not for the transformation of criteria data.

The high variability of the TWSEAS performance, confou nded

I as it was by radio and other equipment problems greatly limits its

j sensitivity to operator performance factors, fatigue and other factors

to which the model is sensitive. The higher sensitivity of the model

to these effects is one of the primary advantages of the model.

___ 

4. ________________________________
- - ,- - - -

~~~-~~~~-~ -----—-~~

_______- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ —

~

-

~

IV. SUMMARY AND CONCLUSION S

• Extensive testing of the NETMAN computer model was cornpleted. This testing included about 60 different simulations for sensitivity test purposes and one simulation of a real system for validational purposes. In the sensitivity tests, the effects of a variety of

personnel variables, workload variables, manpower configurations,

and task variable s were examined for consistency, reliability, indeperidence from trivial effects, and rationality. In general, the results

were found to be reasonable, appropriate, and useful. The most important variable s were operator speed, operator precision, and network configuration. The psychological factors, stress threshold and

level of aspiration, exerted a much less powerful affect on output.

Another result of the sensitivity tests was an evaluation of

the ease of use and the cost of use of the NETMAN computer model.

In terms of cost of use , the NETMAN program was found to be very

efficient. In most cases, an extensive number of iterations is not

required. The findings relative to ease of use were equally satisfactory. Setup for a new simulation is now estimated as not more

than 40 man hours, for an experienced analyst. This time involve -

— ment will depend , to some extent, on the complexity of the system

being simulated. Once a simulation has been organized , however,

changes and rerunning the simulation are readily accomplished.

The test vehicle chosen for evaluating the NETMAN model’s

validity was the Marine Corps’ Tactical Warfare Simulation, Evaluation and Analysis System (TWSEAS). Message processing information

on eight diffe rent aspects of the system performance was collected to

serve as the criteria in the validation analysis. Personneiworking in

F the TWSEAS were pretested to provide the required personnel input

data for the NETMAN model. The predictions from the simulation

were compared with the criterion data during actual TWSEAS use. The

model’s predictions were within plus or minus one stand ard deviation

of the criterion dat a 96 percent of the time for “thoroughness, “ 90 percent of the time for “responsiveness, ” and 100 percent of the time

for refe ree message processing time, radio operator message processing time , and controller message processing time. On the negative side , the model’s prediction of overall system effectiveness was - 

-

over two standard deviations from the mean of the “overall effective -

ness ” criterion measure. The discrepancy in this case is believed to be

due to the controllers who provided the TWSEAS “overall effectiveness”

t critcrion measure. The controllers evaluated the entire system while

~~~~~~~ 

95 

— — 

~~

• —
F - ‘

~~-~~~~ ~~~~~~ - - ~~~~w”~~~-n-~ -~~ -,. ..

~ - -

in the model overall effectiveness is based on messages that are pro- - cessed in the normal TWSEAS condition. Radio problems during the

validation data collection often caused messages to bypass the normal -

radio link and take other routes to the control van . The radio diffi- - - .

culties and indirect message routes caused the controller to rate overall system effectiveness lower than was predicted by the model.

Overall, the results of the sensitivity and validation tests suggest substantial confidence in the NETMAN simulation model. How ever, a number of areas were identified for model improvement . Presently,

the model computes means over all iterations but does not compute

the standard deviation or variance of the data. Variance data would be

useful in evaluating model output. Among other things, it would provide a way to estimate the number of iterations which are necessary

to obtain model output stability.

At the outset of the current effort , a panel of experts was convened to discuss the present statu s of the model. A number of areas

for model improvement were identified . Increasing the ease of use of

the model was considered to be especially important by the panel. A

number of factors which must presently be computed manually by the

analyst preparing the input data for the model could be computed automatically as part of the computer program. As an example , the input

data for message type frequency must be entered in the form of cumulative proportions. The model could be changed so that actual frequencies are entered with the model itself calculating the cumulative proportions. Similarly, there are a number of types of error which might

be made in the assembl y of input data. The program could be extended

to check for many of these errors before simulation execution. As an

example , when two networks are specified in the input , the number of

men in each (some number greater than zero) , must be specified . The

program could very easily be modified to check that the entry in both

cases is nonzero .

The NETMAN computer program already allows a number of

changes to be made to the input data in an interactiv e mode. After

calling the program from a terminal, the user is able to select from

different categories of input , make changes, and then initiate a new

simulation. This interactive mode is alread y usefu l but a number of

additional method s for prompting and assisting the model user in this

mode might be made available. There are also a number of variables

in the model which cannot currently be changed in the interactiv e mode.

Capability might be expanded in this regard .

96 ) —
- 

~~~~~~~~~~

The following conclusions appear warranted: -

4

1. The NETMAN model is adequately sensitive over

a wide range of parametric variation .

2. The parametric vari ation introduced during the

current sensitivity tests produced results which

are reasonable and which possess proper directionality.

3. The results of the validity tests indicate that

NETMAN can provide information about exercise control system operation comparable to

that available in a field exercise environment

such as TWSEAS.

4. Several areas for model improvement are indicated.

97

-
r- 

-

I

REFERENCES j

Campbell, D.T. & Fiske, D.W. Convergent and discrixninant validation by J

the multitrait-multimethod matrix. Psychological Bulletin, 1959,

56 , Si—lOS.

I Emshoff , J.R. & Sisson, R.L., Design and Use of Computer Simulation J Models , London, MacMillan Co., 1970.

Hermann, C. Validation Problems in Games and Simulations , Behavioral

Science, 1967 , 13 , pp. 216—30.

• Leahy, W .R., Lautman, M.R., Bearde , J.L., & Siegel , A.I. A digital

simulation model of message handling to the tactical operations

system. III. Further extensions of the model for increased interaction. Research Memorandum 74—11. U.S. Army Research Institute

for the Behavioral and Social Sciences , August 1974.

Leahy, W.R., Siegel , A.I., & Wolf , J.J. A digital simulation model of

message handling in the tactical operations system . IV. Model

integration with CASE and SAMTOS . Wayne , Pa.: Applied Psychological Services, November 1975.

Leahy , W.R., Siegel , A .I., & Wolf , J.J. A digital simulation model of

message handling in the tactical operations system . V. User ’s

guide to the integratal MANMQD/CASE/SAMTOS computer simulation .

Wayne , Pa: Applied Psychological Services, November 1975.

Siegel, A.I., Lautman, M.R., & Wolf , J.J., A Multimethod—Multitrait

Validation of a Digital Simulation Model . Wayne , Pa.: Applied

Psychological Services, 1972.

I Siegel , A.I., Leahy , W .R., & Wolf, J.J. A computer model for simulation - of message processing in military exercise control and evaluation

systems . ARI Technical Report TR—77-A22 . U.S. Army Research

Institute for the Behavioral and Social Sciences, October 1977.

Siegel , A.I., & Wolf , J.J. Techniques for Evaluation Operator Loading

in Man—Machine System: Application of a Previously Derived Model

to the Launching of an Air—to—Air Missile. Wayne , Pa.: Applied

• Psychological Services, 1959.

Siegel , A.I., Wolf , J.J., & Leahy, W.R. A digital simulation model of

- message handling in the tactical operations system. I. The model ,

its sensitivity, and user ’s manual. Research Memorandum 73—5. U.S.

Army Research Institute for the Behavioral and Social Sciences,

November 1972 .

Siegel, A. !., Wolf , JJ ., Leahy, W.R ., Bearde, J.L., & Baker , J.D. A : 

digital simulation model of message handling in the tactical operations system. II. Extensions of the model for interacting with

subject and experimenters. Research Memorandum 73—6. U.S . Army

Research Institute for the Behavioral and Social Sciences, December

1973.
r - - 

~

-—- - 

,

~~~ !_ •__ ~

__ ~~~ ,_,•___,__•____ ________ 

•,

~

- - --——•— ••——--—_. —---- - -—•-—— --— •__—_ __-_~

___- --_-- - _____ .,-~ -

~ 1~~ -

Siegel, A. !., Wolf , J.J., & Sorenson, R. T. Techniques for Evaluating - Operator Loading in Man-Machine System: Evaluation of a One or

Two-Operator System Evaluative Model through a Controlled La borator y Test s Wayne, Pa.: Applied Psychological Services, 1962.

1 

100 /

L ~~~~~~~~~~ 

- 

-~
- - - -- - -

~~~~

DISTRI IUTION

ARI Distribution Liit

4 0*50 (M&RA) 2 HOUSAC0EC, Ft Ord. ATTN: Library

2 HODA (DAMI-CSZ ) 1 HOUSAC0EC, Ft Ord, ATTN: ATEC-EX—E -Hum Factors

t HODA IDAPE -PBR) 2 USAEEC. Ft Benjamin Har~

ason, ATTN: Library

1 1400* ~DAMA-A R( 1 USAPACDC, Ft Benuamin Hamson, ATTN : ATCP —HR

1 HODA IDAPE-HRE-PO) 1 USA Comm—Elect ScIs, Ft Moninoutli, ATTN : ATSN—EA

1 HODA (SGRD.lD) I USAEC, Ft Monmouth, ATTN: AMSEL--CT -- HOP

I HODA (DAMP-DOT-C) I USAEC. Ft Monmouth, A fTN: AMSEL—PA - P

1 HODA (DAPC-r’MZ-A) I USAEC, Ft Monmouth, ATTN: AMSEL—Sl—C8

1 1400* IDACH-PPZ-AI I USAEC, Ft Monmouth, ATTN: C, Fed Div Br

1 HQOA IDAPE-HREP I USA Materia ls Sys Anal Agcv. Aberdeen, ATTN : AMXSY--P

1 HOOA (DAPE-MPO~

C) I EdgeWood Artinal, Aberdeen, ATTN: SAREA-- -BL--ll

1 HODA (DAPE-OW) I USA Ord Ctr & Sch, Aberdeen, ATIPI: ATSL—1’EM--C

1 1400* (DAPE-HRL) 2 USA Hum Engr Lab, Aberdeen, ATTN: Library/Dir

1 1400* (OAPE.CPSI 1 USA Combat Arms Trig Bd, Ft Benning, ATTN: Ad Supervisor

I 1400* (DAFD-MFAI 1 USA Infantry Hum Rich Unit , Ft Banning, ATTN: Chief

1 HODA (DARO-ARS-PI I USA Infantry Sd, Ft Binning, ATTN: STEBC—TE—T

1 HODA (DAPC-PAS-A) 1 USASMA. Ft Bhss. AT1N: ATSS—LRC

1 HODA (DUSA-OR) I USA Air Del ScIs, Ft Bliss, ATTN: ATSA CTD - ME

1 1100* IDAMO-ROR) 1 USA Air Del Sch, Ft Bliss , ATTN: Tech Lib

I HODA (DASG) 1 USA Air Del Bd, Ft 8lis~, ATTN: FILES

1 IIODA (DAIOPI) 1 USA Air Oaf Bd, Ft Bliss, ATTN: STESO—PO

1 Chief, Consult Div (DA-OTSG). Adelphi, MD 1 USA Cmd & General Stf College, Ft Leavenworth, ATTN: Lib

1 Mil Asit. Plum Ret, 000R&E, 0*1) )E&LS) 1 USA Cmd & General 5sf College, Ft Leavenworth, ATTN: ATSW—SE—L

1 HO USARAL , APO Seattle , ATTN: ARAGP-R 1 USA Cmd & General Ssf College, Ft Leavenworth , ATIN: Ed Advisor

I HO First Army, AIiM: AFKA-O I-TI I USA Combined Arms Crnbt 0ev Act , Ft Leavenwor th, ATTN: DepCdr

2 HO Fifth Army. Ft Sans Houston 1 USA Combined Arms Cmbt Des Art. Ft Leaveriwseth , A1tN CCS

I Dir . Army Stf Studies Oft, ATTN: OAVCSA (DSP) 1 USA Combin ed Arms Cntht Div Act , Ft Leavenwo rth, AiTh: ATCASA

I (Mc Chief of 5ff , Studie s Otc 1 USA Combined Arms Cmbt Des Act , Ft Leavenw orth, ATTN: *TCACO—E

1 I)CSPER, AIIM: CPS/OCP 1 USA Combin ed Arms Cmtn Dew Act , Ft Leavenworth, AIIM: ATCACC—4 ,

1 The Ar my Lii,, Pentagon , ATTN: RSB Chief I USAECOM. Night Vision Lab, Ft Belvoir , ATTN: AMSEL—NV.--SO

1 The Army Lib , Pentagon , A’flN: ANRAL 3 USA Computer Syt Cmd, Ft Belvoic. ATTN: Tech Library

I Ofc , Aert Sect of the Army (R&D) 1 USAMERDC. Ft Belvo ir , AIIM: STSFB--OO

1 Tech Support Oft , OJCS I USA Eng 5th, Fl Belvoir , ATTN: Library

1 USASA, Arlington, AIIM: lARD-I I USA Topogr aphic Lab, Ft Belvoir . ATTN: EU TD—S

I USA Rich Of c, Durham, ATTN: Life Sciences Dir 1 USA Topograp hic Lab. FtB elvoir , AIIM: STINFO Center

2 USARIEM, Natick . AIIM: SGRO-UE-CA 1 USA Topographic 1.b. Ft Belvoin . ATTN: EU- GS%.

I USA! IC, It Cl~yto, s , Al IN: ~;It It; MO A I USA tistelIigenc. ~ Ctr & 5th, Fl I-lu.L’PHIea, ATTN: CTO MS

I USAIMA, Ft Bragg. ATIN: ATSU-CTD-OM 1 USA Intelligence Ctr & Sds , Ft Huichuca, ATTN: ATS—CTD-MS

1 USAIMA . Ft Bragg, AIIM: Marguat Lib 1 USA Intelligence Ctr & 5th, Ft Huachuca , ATTh: AISI—TE

1 US WAC Ctr & Sch, Ft McClell an, ATTN: Lib I USA Intelligence Ctr & Sch, Ft Huachuc., ATTN: ATSI—TFX --GS

I US WAC Ct, & Scb, Ft McClellan, ATTN: log Dir 1 USA Intelligence Ctr & Set, Ft Huechuca, AITh: ATSI—CIS-OR

1 USA Quartermaster Sch, Ft Lee, AIIM: ATSM TE 1 USA Intellig ence Ctr & Sds , Ft Huachuca, ATTN: ATSI—CTD--OT

1 Intelligenc e Material 0ev Ofc , EWL. Ft Holabird I USA Intelligence Ctr & Sch, Ft Huachuca, AIIM: ATSI—CT0—CS

1 USA SE Signal Sen. Ft Gordon, AT’T N: ATSO-EA I USA Intelligence Ctr & Sets , Ft Huachuca, ATTN: 0*S/SRO

I USA Chaplain Cir B Sch , Ft Hamilton , AIIM: ATSC-TE-RD 1 USA Intelligence Ctr & Sets. Ft Huadsuca . ATIN: ATS I—TEM

1 USATSCH, II Euttis. ATTN: Edoc Advisor 1 USA Intelligence Cr t & Sets , Ft Huadsuca , ATIN: Libemy

• 1 USA Wir College, Carlis le Barracks , ATTN: Lib 1 CDR , HO Ft Huachuca, ATm: Tacit Ret Div

2 WRAIR , Neiur opsych iatiy Div 2 CDR, USA Electronic Prig Grd, A1’TP4: STEEP MT—S

I DLI. SDA. Monterey 1 HO, ICATA, ATTN: Teds Library

1 USA Concept Anal Agcy . Bethesda. AIIM: MOCA-MR I HO, TCATA, ATTN: AT CAT OPO, Ft Hood

1 USA Concept Anal Agcy, Bethesda, AIIM: MOCA-JF 1 USA Reciuitii~ Cmd, Ft Sheridan . ATIN: USARCPM P

1 USA Arctic hit Ct,, APO Seattle, AIIM: STEAC-PL.MI 1 Senior Army Mv., USAFA000/TAC. Elgin AF Aux FM No. B

1 USA Arct ic Test Ct,, APO Seattle, ATTN: N.ISTEPL.TS 1 HO, USARPAC. OCSPER. *PO SF ISBGB, AIIM: GPPE-SE

1 USA Armernent Crnd, Redstone Arsenal, ATTN: ATSK TEM 1 Stimion LIb, Academy of Health ScIences. Ft Sam Houston

1 USA Armament Cmd, Roth Idend, AIIM: AMSAR.TOC 1 MarIne Corps Inst., AIIM: 0.anMCI

1 FAA.NAFEC, Atla ntic City, *11W: LIbrary I HO, USMC, Commandant, ATm: Code MiMI

1 FAA-NAFEC, Atlintic City, AIIM: Human Engr Br 1 HO, USMC, Commandant. AIIM: Cod. MPI 20 2S

1 FAA Acconautical Ctr , Oklahoma City , AIIM: AAC-440 2 USCG Academy, New London, ATTN Admission

2 USA FId Arty Sds, Ft Sill , ATTI’4: LIbrary 2 USCG Academy, New London, AIIM: Library

I USA Armor Sets, Ft Knox, AIIM: Library I USCG TralnInS Ct,, NY , ATTN: CO

I USA Armor Sets, Ft Knox, AIIM: AISB-DI E I USCG Trainin Ctr. NY, AiIM: Educ Sec Ole

I USA Armor Sch, Ft Knox, AITN: ATSB-OT-IP 1 USCG, Piychd Res Br, DC, AIIM: OP 1/12

1 USA Armor Set. Ft Knox, AIIM: ATSB-CD-AD I 140 Mid.Rangs Br, MC Ost, Ouantlco. *11W: PIbI Dlv

Ht - 

101

__________________ ______ - -:- - - ~~~~~~~~~~~~

‘-
— 

~4 -‘~

-

~LL,

-- ~~~~~~~~~~~~~~~~~~~~ I ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~Li

~~ j —-

1 US Marine Corps Liaison Of C. ANC. Alexandria, AIIM: AMCGS-F 1 Del & Civil lnit of Enviro Medicine, Canada

I USATRAI)OC, Ft Monroe, ATIN: ATRO—ED 1 AIR CRESS, Kanalngto.s, AIIM: Info Sys Br

B USAIRADOC. Ft Monroe, AIIM: ATPR--AQ I Militaerpeykofogi* Tjeneste, Copenhagen

1 USAIRADOC, Ft Monroe AIIM: ATIS-EA 1 Military Attatte. French Embissy, ATTN: Dcc Sec

1 USA Forces Cind, Ft McPherson, AIIM: Libra ry 1 Medecin Chef, C.E.R?.A.—Arsenst, Toulon/Navil France

2 USA Aviation Test Sd, Ft Rocker , AIIM: STEBG—PO I Prin Scientific Off , A~

spI Hum Engr Rich Dir, Miniitry

1 USA Agry fat Aviation Safety, Ft Rocker , AIIM: Libr ary of Defence, New Delhi

I USA Agey for Aviat ion Safety, Ft Rock er, AIIM: Educ Advisor I Part Rich Ole Library . AKA , Israel Defense Forces

I USA Avialio ~ Sets, Ft flutte r, AIIM: P0 Drawer o 1 Minitteri s wan befansi., 000P/KL Al d Sociaal

1 HOUSA Aviation Sys Cmd . St Louis, AIIM: AMSAV--ZDR Psyctsoksgische Zaken, lii Hague , Netherlinds

2 USA Aviahio ~ Syt Test Act., Edwaids AFB , AIIM: SAVTE—T

1 USA Alt Del Sets, Fl Slits, AIIM: ATSA TEN

I USA Air Mobil ity Rscl a & Dew Lab, Moff etf Fist , AIIM: SAVDI. —AS

1 USA Aviation Sets, Ret log Mgt , Ft Ruciter. ATIN: *1ST-I—RIM

1 USA Aviation Sets , CO. Ft Plucker, AIIM: ATST-D—A

1 HO, DARCOM, Alexandria. AIIM: AMXCD—IL

1 HO, DARCOM. Alexandria, AIIM: COR

1 US Military Academy, West Point, AIIM: Serials Unit

1 US Military Academy, West Point . AIIM: Ofc of Milt Ldrshp

I (iS Military Academy , West Point , AIiM: MAOR

1 USA Standardisatiovs GD, UK, FPO NY, AIM: MASE-GC

1 Ole of Naval Rich, Arlington, AIIM: Code 452

3 (Mc of N.val Rich , Arlington , AIIM: Code 458

1 Ole of Naval Rsch, Arlington , AIIM: Code 450

1 Ole of Naval Rich . Arlington , ATIN: Code 441

1 Naval Aerospc Med flee Lab, Pensacola , AIIN: Acous Sets Div

1 Naval Aetospc Med Ret Lab , Pensacola , AIIM: Code LIl

I Naval Aero~

pc Med Plea Lab, Pensacola. AIIM: Code LI

I Chief of N.vPers, AIIM: Pen -OR

1 NAVAIRSTA , Norfolk , AIIM: Safety Ctr

1 Mae Ocaessographic, DC, AIIM: Code 6251, Chvts & Tech

1 Center of Naval Anal, AIIM: Doe Ctr

I NavAänSysCom, AIIM: AIR—5313C

1 N.e BuMed, AIIM: 713

I NavHelicopt erSubsqua 2. FPO SF 96601

1 AFHRL (Fl) William s AFB

I AFHRI. (TI) Lowry AFB

1 AFI4RL (AS) WPAFB.OH

2 AFHRL (OOJZ) Broths AFB

1 AFHRL (DOJN) Lackland AFB

1 HOUSAF (INYSD)

I HOUSAF (DPXXA)

I AFVTG (RD) Randolph MB

3

2 AF Inst of Tech , WPAFB, OH. AIIM: ENE/SL - I AIC (XPID) Randolph AFB

1 USAF AeroMed Lib, Bronic s MB ISUL--4), AIIM: DOC SEC

1 AFOSR (NI). Arli ngton

1 AF Log Cmd, McClellan AFB , AIIM: ALC/DPCRB

1 Air Force Academy, CO. AIIM: Dept of Sal Sen

5 NavPens&Dev Ctr, San Diego

2 Navy Med Neuropsychiat ric fitch Unit . San Diego

1 Mae Electronic Lab, San Dieqo, AITN: Res Lab

1 N.e IrngCen. S.n Diego, AIIM: Code 9000.-Lib

1 NavPostGraSch. Monterey, AIIM: Code 55*.

1 NavPostGreScts, Monterey, AIIM: Code 2124

1 NavlrngEquipCtr, Orlando, AIIM: Tech LIb

I US Dept of Labor. DC. AIIN: Manpower Admin

1 US Dept of Justice, DC, AIIM: Drug Enforce Admin — 1 Nat Bur of Standards, DC, AIIM: Consputer Info Section

I Nat Clearing House for MN--Info, Rockvifle

I Denver Federal CIt, LaIcew~od, AIIM: BIN

12 Defense Document.tion Center

4 Di, Psyds , Army Hg, Rusiell Ofcs. Canberra

1 Scientific Advsn, Nil Sd, Army 14g. Russell Ole,, Canberra

I Mt and Air Attache, Austrian Embassy

1 Centre do Reclsarcts. Des Facteurs, Humain. do I. Defense

National,, Brussels

2 Canadian Joint Staff Washington

1 C/Air Staff , Royal Canadian A F, AIIM: Per, Std Anal Br

3 Chief, Canadian Del Rich Staff, AIIM: C/CRDS(W)

4 Brit ish Del Staff , British Embassy, Washington 

$

1c2

-

~~~~ - .

“—C